# Analysis and Applications of Class-wise Robustness in Adversarial Training

Qi Tian1, Kun Kuang1†, Kelu Jiang1, Fei Wu1, Yisen Wang2† 1 College of
Computer Science and Technology, Zhejiang University  2 Key Lab of Machine
Perception (MoE), School of EECS, Peking University  tianqics, kunkuang,
jiangkelu<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2021)

###### Abstract.

Adversarial training is one of the most effective approaches to improve model
robustness against adversarial examples. However, previous works mainly focus
on the overall robustness of the model, and the in-depth analysis on the role
of each class involved in adversarial training is still missing. In this
paper, we propose to analyze the class-wise robustness in adversarial
training. First, we provide a detailed diagnosis of adversarial training on
six benchmark datasets, i.e., MNIST, CIFAR-10, CIFAR-100, SVHN, STL-10 and
ImageNet. Surprisingly, we find that there are remarkable robustness
discrepancies among classes, leading to unbalance/unfair class-wise robustness
in the robust models. Furthermore, we keep investigating the relations between
classes and find that the unbalanced class-wise robustness is pretty
consistent among different attack and defense methods. Moreover, we observe
that the stronger attack methods in adversarial learning achieve performance
improvement mainly from a more successful attack on the vulnerable classes
(i.e., classes with less robustness). Inspired by these interesting findings,
we design a simple but effective attack method based on the traditional PGD
attack, named Temperature-PGD attack, which proposes to enlarge the robustness
disparity among classes with a temperature factor on the confidence
distribution of each image. Experiments demonstrate our method can achieve a
higher attack rate than the PGD attack. Furthermore, from the defense
perspective, we also make some modifications in the training and inference
phases to improve the robustness of the most vulnerable class, so as to
mitigate the large difference in class-wise robustness. We believe our work
can contribute to a more comprehensive understanding of adversarial training
as well as rethinking the class-wise properties in robust models.

adversarial training; adversarial robustness; class-wise properties;
adversarial examples

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2021; Virtual Event, Singapore.††booktitle: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), August
14–18, 2021, Virtual Event, Singapore††price: 15.00††isbn:
978-1-4503-8332-5/21/08††doi: 10.1145/3447548.3467403††ccs: Computing
methodologies Machine learning††ccs: Security and privacy

##  1\. Introduction

Deep learning has achieved great success in many applications (such as image
classification (He et al., 2016), video processing (Zhang et al., 2020),
recommender systems (Zhang et al., 2021)). Unfortunately, the existence of
adversarial examples (Szegedy et al., 2013) reveals the vulnerability of deep
neural networks, which hinders the practical deployment of deep learning
models. Adversarial training (training on adversarial examples) (Madry et al.,
2018) has been demonstrated to be one of the most successful defense methods
by Athalye et al. (2018). While it can only obtain moderate robustness even
for simple image datasets like CIFAR-10, a comprehensive understanding for
adversarial training is critical for further robustness improvement.

Previously, some works tried to analyze adversarial training from robust
optimization (Wang et al., 2019a), robustness generalization (Raghunathan et
al., 2019; Wu et al., 2020), training strategy (Madry et al., 2018; Zhang et
al., 2019; Wang et al., 2019c; Pang et al., 2020; Bai et al., 2021). However,
in these works, they all focus on the averaged robustness over all classes
while ignoring the possible difference among different classes. In other
fields, there are some works revealing the class-bias learning phenomenon in
the standard training (training on natural examples) (Tang et al., 2020; Wang
et al., 2019b), in which they found that some classes (“easy” classes) are
easy to learn and converge faster than other classes (“hard” classes).
Inspired by this, a natural question is then raised here:

> Does each class perform similarly in the adversarially trained models? Or is
> each class equally vulnerable? If not, how would the class-wise robustness
> affect the performance of classical attack and defense methods in
> adversarial learning?

22footnotetext: Corresponding Authors.

In this paper, we investigate the above questions comprehensively.
Specifically, we conduct a series of experiments on several benchmark datasets
and find that the class-bias learning phenomenon still exists in adversarial
training which is even severe than standard training. For this finding, we
have the following questions to explore:

(a) MNIST

(b) CIFAR-10

(c) CIFAR-100

(d) SVHN

(e) STL-10

(f) ImageNet

Figure 1. Class-wise robustness at different epochs in the test set

  * 1)

What is the relation between the unbalanced robustness and the properties of
the dataset itself?

  * 2)

Can we use the class-wise properties to further enlarge the differences among
classes?

  * 3)

Are there any ways to improve the robustness of vulnerable classes so as to
obtain a more balance/fairer robust model?

We conduct extensive analysis on the obtained robust models and summarize the
following contributions:

  * $\bullet$

Analysis on class-wise robustness

    1. 1)

We systematically investigate the relation between different classes and find
classes in each dataset can be divided into several groups, and intra-group
classes are easily affected by each other.

    2. 2)

The relative robustness between each class is pretty consistent among
different attack or defense methods, which indicates that the dataset itself
plays an important role in the class-wise robustness.

  * $\bullet$

Applications for stronger attack

    1. 1)

We make full use of the properties of the vulnerable classes to propose an
attack that can effectively reduce the robustness of these classes, thereby
increasing the disparity among classes.

  * $\bullet$

Applications for stronger defenses

    1. 1)

Training phase: Since the above group-based relation is commonly observed in
the dataset, we propose a method that can effectively use this relation to
adjust the robustness of the most vulnerable class.

    2. 2)

Inference phase: We find that the background of the images may be a potential
factor for different classes to be easily flipped by each other. Our
experiments show that the robustness of the most vulnerable class can be
improved by simply changing the background.

##  2\. Related Work

Class-wise analysis. Class-wise properties are widely studied in the deep
learning community, such as long-tailed data (Tang et al., 2020) and noisy
label (Wang et al., 2019b). The datasets for these specific tasks are
significantly different in each class. i.e., in long-tailed data task, the
tail-class (with few training data) usually achieves lower accuracy since it
cannot be sufficiently trained. In the asymmetric noisy label task, classes
with more label noise usually have lower accuracy. However, in the adversarial
community, few people pay attention to class-wise properties because all
benchmark datasets seem to be class-balanced. Recently, we notice two parallel
and independent works (Nanda et al., 2021; Benz et al., 2020) also point out
the performance disparity in robust models, but none of them explore the
relation between class-wise robustness and the properties of the dataset
itself, and our work takes the first step to investigate this problem.

(a) MNIST

(b) CIFAR-10

(c) CIFAR-100

(d) SVHN

(e) STL-10

(f) ImageNet

Figure 2. Confusion matrix of robustness in the test set

Attack. Adversarial attacks are used to craft adversarial examples by adding
small and human imperceptible adversarial perturbations to natural examples,
which mainly include white-box attacks and black-box attacks. In white-box
settings, the attackers know the parameters of the defender model and generate
adversarial noise by maximizing the loss function (e.g., Fast Gradient Sign
Method (FGSM) (Goodfellow et al., 2015) and Projected Gradient Descent (PGD)
(Madry et al., 2018) attack maximize cross-entropy loss, while Carlini-Wagner
(C&W) (Carlini and Wagner, 2017) attack maximize hinge loss). In black-box
settings, there are transfer-based and query-based attacks. The former attacks
a substitute model and the generated noise can transfer to the target model
(Wu et al., 2019; Wang et al., 2021). The latter crafts adversarial examples
by querying the output of the target model (Li et al., 2019; Bai et al.,
2020). In this paper, we analyze the class-wise robustness performance of
different attacks and propose an attack to illustrate that the unbalanced
robustness can be enlarged by carefully using the information of vulnerable
classes.

Defense. Adversarial training (Madry et al., 2018) is known as the most
effective and standard way to against adversarial examples. A range of methods
have been proposed to improve adversarial training, including modifying
regularization term (Zhang et al., 2019; Wang et al., 2019c; Pang et al.,
2020), adding unlabeled data (Carmon et al., 2019) and data augmentation (Song
et al., 2019). Since adversarial training is more time-consuming than standard
training, Wong et al. (2019) propose some solutions to accelerate model
training. On the other hand, some researchers (Raff et al., 2019; Pang et al.,
2019; Bai et al., 2019) try to improve model robustness by preprocessing the
image in the inference phase, and these methods are usually complementary to
adversarial training. However, none of these methods consider the difference
in class-wise robustness, and we have proposed some methods that can improve
the robustness of the most vulnerable class so as to obtain a fairer output.

##  3\. Preliminary

In this section, we first introduce the formula and notations in adversarial
training, then give several definitions about robust/non-robust example and
robust/vulnerable/confound class used through this paper.

Vanilla adversarial training. Madry et al. (2018) formalize the adversarial
training as a min-max optimization problem. Given a DNN classifier
$h_{\bm{\theta}}$ with parameters $\bm{\theta}$, a correctly classified
natural example $\bm{x}$ with class label $y$, cross-entropy loss
$\ell(\cdot)$ and an adversarial example $\bm{x}^{\prime}$ can be generated by
perturbing $\bm{x}$, then the objective of adversarial training is:

(1)
$\displaystyle\textstyle\min\limits_{\bm{\theta}}\frac{1}{n}\sum_{i=1}^{n}\max\limits_{\left\|{\bm{x}}_{i}^{\prime}-{\bm{x}}_{i}\right\|_{p}\leq\epsilon}\ell\left(h_{\bm{\theta}}\left({\bm{x}}_{i}^{\prime}\right),y_{i}\right),$

where the inner maximization applies the Projected Gradient Descent (${\rm
PGD}$) attack to craft adversarial examples, and the outer minimization uses
these examples as augmented data to train the model. Since the adversarial
perturbation should not be observed by humans, these noises are bounded by
$L_{p}$-norm
$\left\|{\bm{x}}_{i}^{\prime}-{\bm{x}}_{i}\right\|_{p}\leq\epsilon$.

TRADRS. Another popular adversarial training method (TRADRS (Zhang et al.,
2019)) is to add a regularization term to the cross-entropy loss:

(2)
$\displaystyle\textstyle\min\limits_{\bm{\theta}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(h_{\bm{\theta}}\left({\bm{x}}_{i}\right),y_{i}\right)+\beta\max\limits_{\left\|{\bm{x}}_{i}^{\prime}-{\bm{x}}_{i}\right\|_{p}\leq\epsilon}\mathcal{K}\left(h_{\bm{\theta}}\left({\bm{x}}_{i}\right),h_{\bm{\theta}}\left({\bm{x}}_{i}^{\prime}\right)\right),$

where $\mathcal{K}(\cdot)$ represents Kullback-Leibler divergence and $\beta$
can adjust the relative performance between natural and robust accuracy.

In addition, we define some concepts for the convenience of the following
expressions.

###### Definition 3.1.

(Robust Example) Given a natural example $\bm{x}$ with ground truth class $y$
and a DNN classifier $h_{\bm{\theta}}$ with parameters $\bm{\theta}$, if this
example does not exist adversarial counterpart in bounded $\epsilon$-ball
$\left\|{\bm{x}}^{\prime}-{\bm{x}}\right\|_{p}\leq\epsilon$:
$h_{\bm{\theta}}({\bm{x}}^{\prime})\equiv y$, the example $\bm{x}$ is defined
as a robust example.

###### Definition 3.2.

(Non-Robust Example and Confound Class) Given a natural example $\bm{x}$ with
ground truth class $y$ and a DNN classifier $h_{\bm{\theta}}$ with parameters
$\bm{\theta}$, if the prediction of the model is $y^{\prime}$ after adding a
bounded $\epsilon$-ball
$\left\|{\bm{x}}^{\prime}-{\bm{x}}\right\|_{p}\leq\epsilon$ :
$h_{\bm{\theta}}({\bm{x}}^{\prime})=y^{\prime}\neq y$, the example $\bm{x}$ is
defined as a non-robust example and $y^{\prime}$ is defined as confound class
of example $\bm{x}$.

Table 1. Adversarial robustness (%) (under popular attacks) on CIFAR-10.

Defenses(Attacks) | Tot. | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | CV | MCD
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Madry(FGSM) | 65.5 | 73.7 |  81.21 | 51.9 | 41.52 | 54.2 | 49.4 | 73.9 | 72.5 | 78.5 | 78.5 | 191.9 | 39.7
TRADES(FGSM) | 66.9 | 77.5 | 85.9 | 49.7 | 41.9 | 55.8 | 52.8 | 73.0 | 76.8 | 80.7 | 75.3 | 211.2 | 44.0
MART(FGSM) | 67.4 | 73.7 | 84.9 | 54.5 | 45.7 | 50.1 | 51.6 | 76.9 | 75.2 | 83.9 | 77.7 | 206.1 | 39.2
HE(FGSM) | 68.4 | 71.9 | 84.6 | 52.0 | 42.2 | 57.0 | 57.9 | 76.5 | 77.8 | 83.4 | 80.9 | 200.5 | 42.3
Madry($\rm CW_{\infty}$) | 57.1 | 67.5 | 79.5 | 43.0 | 37.7 | 41.5 | 41.0 | 57.5 | 60.0 | 71.5 | 72.0 | 212.5 | 41.8
TRADES($\rm CW_{\infty}$) | 59.4 | 69.5 | 85.5 | 39.0 | 38.5 | 43.0 | 46.5 | 57.0 | 67.0 | 77.5 | 70.5 | 258.5 | 47.0
MART($\rm CW_{\infty}$) | 58.8 | 65.5 | 80.5 | 43.0 | 39.5 | 41.0 | 41.0 | 63.0 | 67.5 | 76.5 | 71.0 | 232.7 | 41.0
HE($\rm CW_{\infty}$) | 63.6 | 71.2 | 87.5 | 47.1 | 44.4 | 49.8 | 50.1 | 61.4 | 71.2 | 81.5 | 72.7 | 210.8 | 43.1
Madry(${\rm PGD}$) | 52.1 | 63.8 | 71.6 | 39.1 | 25.3 | 36.7 | 38.6 | 57.4 | 59.5 | 63.1 | 66.8 | 224.3 | 46.3
TRADES(${\rm PGD}$) | 56.3 | 67.8 | 80.6 | 37.8 | 29.4 | 40.6 | 43.9 | 59.3 | 66.9 | 71.8 | 65.6 | 263.6 | 51.1
MART(${\rm PGD}$) | 58.2 | 64.5 | 78.0 | 45.1 | 35.4 | 37.7 | 43.5 | 65.3 | 67.5 | 76.3 | 69.5 | 235.1 | 42.6
HE(${\rm PGD}$) | 60.7 | 64.9 | 79.3 | 41.0 | 34.5 | 47.9 | 51.5 | 67.6 | 70.5 | 76.9 | 73.2 | 224.8 | 44.8
Madry(Transfer-based attack) | 80.2 | 84.5 | 87.7 | 71.0 | 68.3 | 78.9 | 69.2 | 86.3 | 82.9 | 87.6 | 86.4 | 56.1 | 19.4
TRADES(Transfer-based attack) | 82.0 | 87.7 | 92.3 | 70.9 | 68.0 | 78.2 | 70.0 | 87.8 | 87.6 | 90.8 | 86.8 | 78.1 | 24.2
MART(Transfer-based attack) | 82.9 | 87.4 | 94.7 | 74.0 | 66.7 | 76.0 | 68.8 | 89.9 | 88.0 | 93.7 | 90.0 | 99.2 | 28.0
HE(Transfer-based attack) | 84.5 | 90.1 | 95.9 | 75.6 | 60.8 | 77.4 | 76.7 | 91.1 | 92.1 | 93.4 | 92.2 | 115.3 | 35.1
Madry($\mathcal{N}$ atacck) | 56.1 | 67.5 | 77.7 | 43.7 | 31.4 | 42.7 | 49.0 | 53.7 | 60.1 | 64.4 | 71.1 | 190.5 | 46.3
TRADES($\mathcal{N}$ atacck) | 64.4 | 73.1 | 87.4 | 46.4 | 44.4 | 49.1 | 61.7 | 56.9 | 71.6 | 79.5 | 74.1 | 200.0 | 43.0
MART($\mathcal{N}$ atacck) | 67.5 | 72.3 | 83.4 | 55.3 | 49.0 | 54.1 | 61.2 | 67.1 | 72.9 | 82.3 | 77.6 | 133.6 | 34.4
HE($\mathcal{N}$ atacck) | 69.7 | 75.9 | 88.3 | 52.7 | 44.7 | 65.4 | 62.6 | 70.1 | 76.0 | 84.5 | 77.5 | 168.6 | 43.5

  * 1

The underscore indicates the most robust class.

  * 2

The bold indicates the most vulnerable class.

Table 2. Superclasses in CIFAR-10 and STL-10.

Dataset | Transportation
---|---
CIFAR-10 |  Airplane(01) | Automobile(1) | Ship(8) | Truck(9)
STL-10 | Airplane(0) | Car(2) | Ship(8) | Truck(9)
Dataset | Animals
CIFAR-10 | Bird(2) | Cat(3) | Deer(4) | Dog(5) | Frog(6) | Horse(7)
STL-10 | Bird(1) | Cat(3) | Deer(4) | Dog(5) | Horse(6) | Monkey(7)

  * 1

The number in brackets represents the numeric label of the class in the
dataset.

###### Definition 3.3.

(Robust Class and Vulnerable Class) A class whose robustness is higher than
the overall robustness is called a robust class. In contrast, a class whose
robustness is lower than the overall robustness is called a vulnerable class.

##  4\. Class-wise Robustness Analysis

In this section, we focus on analyzing the class-wise robustness, including
class-biased learning and class-relation exploring on six benchmark datasets.
Moreover, we investigate the class-wise robustness with different attack and
defense models.

We use six benchmark datasets in adversarial training to obtain the
corresponding robust model, i.e., MNIST (LeCun et al., 1998), CIFAR-10 &
CIFAR-100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), STL-10
(Coates et al., 2011) and ImageNet (Deng et al., 2009). Table 2 highlights
that the classes of CIFAR-10 and STL-10 can be grouped into two superclasses:
Transportation and Animals. Similarly, CIFAR-100 also contains 20 superclasses
with each has 5 subclasses. For the ImageNet dataset, the pipeline of
adversarial training follows Wong et al. (2019), while the training methods of
other datasets follow Madry et al. (2018). See Appendix A.1 for detailed
experimental settings.

###  4.1. Class-biased Learning

Figure 1 plots the robustness of each class at different epochs in the test
set for six benchmark datasets with adversarial training, where the shaded
area in each sub-figure represents the robustness gap between different
classes across epochs. Considering the large number of classes in CIFAR-100
and ImageNet, we randomly sample 12 classes for a better indication. From
Figure 1, we surprisingly find that there are recognizable robustness gaps
between different classes for all datasets. Specifically, for SVHN, CIFAR-10,
STL-10 and CIFAR-100, the class-wise robustness gaps are obvious and the
largest gaps can reach 40%-50% (Figure 1(b)-1(e)). For ImageNet, since the
model uses the three-stage training method (Wong et al., 2019), its class-wise
robustness gap increases with the training epoch, and finally up to 80%
(Figure 1(f)). Even for the simplest dataset MNIST, on which model has
achieved more than 95% overall robustness, the largest class-wise robustness
gap still has 6% (Figure 1(a)).

Hence, we can conclude that the class-bias learning phenomenon is also common
in adversarial learning, and there are remarkable robustness discrepancies
among classes, leading to unbalance/unfair class-wise robustness in
adversarial training. Inspired by the phenomenon in Figure 1, we next conduct
the analysis to the robust relation between classes and the impact of
different attacks and defenses on class-wise robustness in the following
subsections.

###  4.2. The relations among different classes

We first systematically investigate the relation of different classes under
robust models. Figure 2 shows the confusion matrices of robustness between
classes on all the six datasets. The X-axis and Y-axis represent the predicted
classes and the ground truth classes, respectively. The grids on the main
diagonal line represent the robustness of each class, while the grids on the
off-diagonal line represent the non-robustness on one class (Y-axis) to be
misclassified to another class (X-axis).

Observations and Analysis. From the results reported in Figure 2, we have the
following observations and analysis: (i) The confusion matrices on all six
benchmark datasets roughly demonstrate one kind of symmetry (i.e, the
highlight colors of off-diagonal elements are symmetrical about the main
diagonal), which indicates that some classes-pair could be easily
misclassified between each other.

(a)

(b) MART’s output for image 127 (class 3) with iteration steps 1

(c) MART’s output for image 127 (class 3) with iteration steps 10

(d) MART’s output for image 127 (class 3) with iteration steps 20

Figure 3. Analysis of output confidence Table 3. Adversarial robustness (%)
under Temperature-PGD20 attack on CIFAR-10.

Defense | 1/T | Tot. | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | CV | MCD
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Madry | 2 |  51.8(-0.3)1 |  63.4(-0.4) |  72.0(+0.4)2 |  38.8(-0.3) |  25.2(-0.1) |  33.9(-2.8) |  38.5(-0.1) |  56.6(-0.8) |  59.8(+0.3) |  63.0(-0.1) |  66.9(+0.1) |  235.6(+11.3) |  46.8(+0.5)
TRADES | 5 |  54.6(-1.7) |  66.8(-1.0) |  80.0(-0.6) |  36.7(-1.1) |  26.2(-3.2) |  35.6(-5.0) |  43.0(-0.9) |  56.0(-3.3) |  66.0(-0.9) |  70.8(-1.0) |  64.9(-0.7) |  291.6(+28.0) |  53.8(+2.7)
MART | 5 |  54.3(-3.9) |  62.7(-1.8) |  77.1(-0.9) |  41.5(-3.6) |  26.3(-9.1) |  27.5(-10.2) |  41.5(-2.0) |  60.8(-4.5) |  66.1(-1.4) |  72.8(-3.5) |  67.3(-2.2) |  311.3(+76.2) |  50.8(+8.2)
HE | 5 |  57.3(-3.4) |  62.4(-2.5) |  74.8(-4.5) |  38.4(-2.6) |  29.4(-5.1) |  43.1(-4.8) |  47.8(-3.7) |  62.9(-4.7) |  69.1(-1.4) |  74.5(-2.4) |  70.8(-2.4) |  240.8(+16.0) |  45.4(+0.6)
HE | 50 |  50.4(-10.3) |  58.2(-6.7) |  71.8(-7.5) |  33.3(-7.7) |  17.6(-16.9) |  23.0(-24.9) |  41.6(-9.9) |  56.2(-11.4) |  66.9(-3.6) |  69.9(-7.0) |  65.7(-7.5) |  363.5(+138.7) |  54.1(+9.3)

  * 1

“-” represents the robustness reduction compared with the corresponding
element of PGD attack in table 1.

  * 2

“+” represents the robustness improvement compared with the corresponding
element of PGD attack in table 1.

(ii) The symmetry classes-pair in Figure 2 are always similar to some degree,
such as similar in shape or belonging to the same superclasses, hence, would
be easy misclassified to each other. Specifically, for SVHN, digits with
similar shapes are more likely to be flipped to each other, e.g., the number 6
and number 8 are similar in shape and the non-robustness between them (number
6 is misclassified to be number 8 or vice versa) is very high as shown in
Figure 2(d). For CIFAR-10 and STL-10, Figures 2(b) and 2(e) clearly show that
the classes belonging to the same superclass have high probabilities to be
misclassified to each other, for example, both class 3 (cat) and class 5 (dog)
in CIFAR-10 belong to the superclass Animals, the non-robustness between them
is very high in Figure 2(b). (iii) Few misclassifications would happen between
two classes with different superclasses. For example, in STL-10, the class 5
(dog) belongs to superclass Animals, while class 9 (truck) belongs to
Transportation, and their non-robustness is almost 0 as shown in figure 2(e).

For CIFAR-100 and ImageNet, we can also observe symmetry properties of
confusion matrix in Figure 2(c) and Figure 2(f), which is consistent with the
above analysis. Overall, Figure 2 demonstrates that the classes with similar
semantic would be easier misclassified (with higher non-robustness) to each
other than those with different semantics (e.g., the classes belong to
different superclasses).

###  4.3. The class-wise robustness under different attacks and defenses

The above analysis mainly concentrates on the performance under ${\rm PGD}$
attack. In this subsection, we investigate the class-wise robustness of state-
of-the-art robust models against various popular attacks in the CIFAR-10
dataset.

The defense methods we chose include Madry training (Madry et al., 2018),
TRADES (Zhang et al., 2019), MART (Wang et al., 2019c) and HE (Pang et al.,
2020). We train WideResNet-32-10 (Zagoruyko and Komodakis, 2016) following the
original papers. White-box attacks include FGSM (Goodfellow et al., 2015),
${\rm PGD}$ (Madry et al., 2018) and $\rm CW_{\infty}$ (Carlini and Wagner,
2017), and the implementation of $\rm CW_{\infty}$ follows (Carmon et al.,
2019). Black-box attacks include a transfer-based and a query-based attack.
The former uses a standard trained WideResNet-32-10 as the substitute model to
craft adversarial examples, and the latter uses $\mathcal{N}$ attack (Li et
al., 2019). All hyperparameters see Appendix A.2.

In order to quantitatively measure the robustness unbalance (or discrepancy)
among classes, we give the definition of two statistical metrics: class-wise
variance (CV) and maximum class-wise discrepancy (MCD) as follows

(a) Misclassified confusion matrix

(b) Homing confusion matrix

(c) Adjust class 3 robustness

Figure 4. Case study about adjusting class 3 robustness in the training phase

###### Definition 4.1.

(Class-wise Variance ,CV) Given one dataset containing $C$ classes, the
accuracy of each class $c$ is $a_{c}$, the average accuracy over all classes
is $\overline{a}=\sum_{c=1}^{C}a_{c}/C$, and then CV is defined as:

$CV=\frac{1}{C}\sum_{c=1}^{C}(a_{c}-\overline{a})^{2}.$

###### Definition 4.2.

(Maximum Class-wise Discrepancy, MCD) Given one dataset, let $a_{max}$ and
$a_{min}$ represent the maximum and minimum accuracy of class, then MCD is
defined as:

$MCD=a_{max}-a_{min}.$

The insight of these two metrics is to measure the average discrepancy and the
most extreme discrepancy among classes. Intuitively, these metrics will be
large if there are huge class-wise differences.

Observations and Analysis. Based on the CIFAR-10 dataset, we check the class-
wise robustness of different attack and defense models and report the results
in Table 1. From the results, we can have the following observations and
analysis: (i) In all models and attacks, there are remarkable robustness gaps
between different classes, and class 1 and class 3 are always the most robust
and vulnerable class in all settings, which suggests the relative robustness
of each class has a strong correlation with the dataset itself. (ii) Stronger
attacks in white-box settings are usually more effective for vulnerable
classes. For example, comparing FGSM and PGD of the same defense method, the
robustness reduction of the vulnerable classes (e.g., class 3) is obviously
larger than that of robust classes (e.g., class 1), resulting in larger class-
wise variance (CV) and maximum class-wise discrepancy (MCD). (iii) In black-
box settings, the main advantage of the query-based attack over the transfer-
based attack is also concentrated in vulnerable classes. One explanation is
that many examples of these classes are closer to the decision boundary,
making it easier to be attacked.

In addition, we have also checked the CV and MCD of the adversarial training
are significantly larger than the standard training in all datasets. For
example, in terms of the CIFAR-10 dataset, the CV of adversarial training is
28 times that of standard training, and the MCD of adversarial training is 5
times that of standard training. This shows that class-wise properties in the
robustness model are worthy of attention. See Appendix B for more details.

##  5\. Improving adversarial attack via class-wise discrepancies

Although Section 4.3 have shown the class-wise robustness discrepancies are
commonly observed in adversarial settings, we believe that this gap can be
further enlarged if the attacker makes full use of the properties of
vulnerable classes. Specifically, since the images near decision boundary
usually have smooth confidence distributions, popular attacks cannot find the
effective direction in the iterative process, and Figure 3-3 clearly show an
example of the failed attack with PGD (i.e., the bar for ground truth class 3
is always the highest). To solve this problem, we propose to use a temperature
factor to change this distribution, so as to create virtual power in the
possible adversarial direction.

For a better formulation, we assume that the DNN is $f$, the input example is
$x$, the number of classes in the dataset is $C$, then the softmax probability
of this sample $x$ corresponding to class $k$ ($k\in C$) is

(3)
$\displaystyle\mathbb{S}(f(x))_{k}=\frac{e^{f(x)_{k}/T}}{\sum_{c=1}^{C}e^{f(x)_{c}/T}}.$

Using this improved softmax function, the adversarial perturbation crafted at
$t^{\rm th}$ step is

(4)
$\displaystyle\textstyle\delta^{t+1}=\prod_{\epsilon}(\delta^{t}+\alpha\cdot{\rm
sign}(\nabla\ell(\mathbb{S}(f(x+\delta^{t})),y))).$

(a) Numbers after adding background

(b) Confusion matrix for natural images

(c) Confusion matrix for adversarial images

Figure 5. Experiments about adding background on MNIST

Where $\prod_{\epsilon}$ is the projection operation, which ensures that
$\delta$ is in $\epsilon$-ball. $\ell(\cdot)$ is the cross-entropy loss.
$\alpha$ is the step size. $y$ is the ground truth class.

The bar corresponding to Temperature-PGD (1/T=5) in Figure 3-3 is a good
example of how our proposed method works. To better understand the impact of
our method on different defense models, the class-wise variance of confidence
(CVC) is proposed to measure the smoothness of the confidence output of these
models.

###### Definition 5.1.

(Class-wise Variance of Confidence, CVC) Assume that there are $C$ classes in
the test set, class $k$ has $N$ images, the confidence output of one image $i$
is $\bm{p}=(p_{1},\cdots,p_{c},\cdots,p_{C})$ and the average confidence of
this image is $\overline{p}^{i}=\sum_{c=1}^{C}p_{c}^{i}/C$, then CVC of class
$k$ is defined as

$CVC_{k}=\frac{1}{NC}\sum_{i=1}^{N}\sum_{c=1}^{C}(p_{c}^{i}-\overline{p}^{i})^{2}.$

Figure 6. Changing background of class 3 images in CIFAR-10

Intuitively, this value will be small if the output confidence of one class is
smooth. From the results of Figure 3(a) and Table 2, we can find that in all
defense models, the CVC of superclass Animals is smaller than that of
superclass Transportation, and the CVC of class 4 and class 3 is the smallest
and second-smallest. Combined with the information of Table 1, the class with
low robustness is closer to the classification boundary, so the confidence
distribution is smoother, which is consistent with our previous analysis. On
the other hand, the overall CVC of HE is much smaller than other defense
methods, which means that popular attacks (i.e., PGD) may be very inefficient
for this defense model.

Overall results. In practice, we perform a grid search on the hyperparameter
$\rm{1/T}\in[2,5,10,50]$, and report the best performance. The results of
Table 3 verify the effectiveness of our proposed method. Specifically, (i) The
CV and MCD of all defense models have become larger, which means that the
disparity between the classes is enlarged. (ii) Since the confidence output of
the vulnerable classes is smoother (Figure 3(a)), our method can significantly
reduce the robustness of these classes. (e.g., class 3 and class 4). (iii) The
Madry model has a steeper confidence distribution, so it is not sensitive to
Temperature-PGD attack. On the contrary, because the confidence outcomes of
the HE model is extremely smooth, increasing the temperature factor to make
the outcome steeper can significantly improve the attack rate, i.e., total
robustness reduces 10.3% (when 1/T=50), of which class 3 and class 4 reduce
16.9% and 24.9% respectively. Overall, the success of Temperature-PGD is
effective evidence that stronger attackers can further increase the class-wise
robustness difference.

##  6\. Improving the robustness of the vulnerable class

In this section, we propose two methods to mitigate the difference in class-
wise robustness. Specifically, our goal is to improve the robustness of the
most vulnerable subgroup in CIFAR-10 (i.e., class 3), because it has the
lowest robustness as described in the previous analysis.

###  6.1. Adjust robustness at the training phase

Figure 2 analyzes the relation of class-wise robustness in detail. Here we
further explore the more fine-grained relation between these classes by
removing the confound class (Definition 3.2). Specifically, for the example
$\bm{x}$ from class $y$ is attacked to the confound class $y^{\prime}$, we are
curious if we remove confound class $y^{\prime}$ (i.e., remove all examples of
ground truth class $y^{\prime}$ in the training set) and re-train the model,
will example $\bm{x}$ become a robust example WITHOUT being maliciously
flipped to a new confound class111We usually think that there are many
decision boundaries in bounded $\epsilon$-ball, so a new confound class is
likely to appear even if one decision boundary is removed.?

###### Definition 6.1.

(Homing Property) Given an adversarial example $x^{\prime}$ from class $y$
which is misclassified as the confound class $y^{\prime}$ by a model, this
example satisfies homing property if it becomes a robust example after we re-
train the model via removing confound class $y^{\prime}$.

Table 4. Robust model prediction results for images of class 3 in the test set
(1000 images) under Temperature-PGD20 attack.

Line number | Test set | Class 0 | Class 1 | Class 2 | Class 3(correct) | Class 4 | Class 5 | Class 6 | Class 7 | Class 8 | Class 9
---|---|---|---|---|---|---|---|---|---|---|---
1 | Original image (Natural) |  181 | 4 | 43 | 710 | 38 | 88 | 64 | 15 | 6 | 14
2 | \+ white background (Natural) |  39(+21)2 |  10(+6) |  48(+5) | 742(+32) |  13(-25) |  59(-29) |  63(-1) |  5(-10) |  3(-3) |  18(+4)
3 | \+ training adjustment method (Natural) |  35(+17) |  15(+11) |  56(+13) | 758(+48) |  23(-15) |  24(-64) |  61(-3) |  6(-9) |  8(+2) |  14(+0)
4 | Original image (Adversarial) | 34 | 16 | 82 | 262 | 113 | 254 | 139 | 53 | 14 | 33
5 | \+ white background (Adversarial) |  76(+42) |  22(+6) |  72(-10) | 403(+141) |  80(-33) |  136(-118) |  116(-23) |  38(-15) |  7(-7) |  50(+17)
6 | \+ training adjustment method (Adversarial) |  93(+59) |  31(+15) |  80(-2) | 435(+173) |  72(-41) |  84(-170) |  111(-28) |  35(-18) |  3(-11) |  56(+23)

  * 1

The number represents how many images in the corresponding test set are
predicted to be the corresponding class.

  * 2

“+” represents the increase in the number of images compared with the
corresponding element of the original image test set, and “-” is vice versa.

To explore the above question, we conduct extensive experiments and the
results are reported in Figure 4. Figure 4(a) and Figure 2(b) are similar, and
the difference is that the values in Figure 4(a) represent the number of
examples instead of percentage, and the main diagonal elements (the number of
examples correctly classified) are hidden for better visualization and
comparison. Thus this figure is called the Misclassified confusion matrix. To
check the homing property, we alternatively remove each confound class to re-
train the model and plot the results in Figure 4(b), where the element in the
$i^{\rm th}$ row and $j^{\rm th}$ column (indexed by the classes starting from
$0$) indicates how many adversarial examples with ground truth class $i$ and
confound class $j$ that satisfy homing property (i.e., these examples will
become robust examples after removing the confound class $j$), so this figure
is defined as the Homing confusion matrix.

Figure 4 clearly shows homing property is widely observed in many
misclassified examples. For example, we can focus on the 3th row and the 5st
column of Figure 4(a) and 4(b). 200 in Figure 4(a) means that 200 examples of
class 3 are misclassified as class 5, and 119 in Figure 4(b) means that if we
remove class 5 and re-train the model, 119 of 200 examples will home to the
correct class 3 (i.e., become robust examples). This suggests that changing
the robustness of class 3 only needs to carefully handle the relation with
class 5. Interestingly, these group-based relations are commonly observed in
CIFAR-10, e.g., class 1 (automobile)-class 9 (truck) and class 0
(airplane)-class 8 (ship).

The proposed method. Based on the above discovery, we try to use this group-
based relation to adjust the class-wise robustness. Our method is based on
TRADES (Zhang et al., 2019) as shown in the Equation (5). Specifically, Zhang
et al. (2019) set $\beta$ as a constant to adjust natural accuracy and robust
accuracy, while we modify $\beta$ to a vector
$\bm{\beta}=(\beta_{1},...,\beta_{c},...,\beta_{C})$ to adjust class-wise
robustness, where $c\in C$ is the class id, thus the loss function of class
$c$ is

(5)
$\displaystyle\textstyle\min\limits_{\bm{\theta}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(h_{\bm{\theta}}\left({\bm{x}}_{c}^{i}\right),y_{c}^{i}\right)+\beta_{c}\max\limits_{\left\|{\bm{x}}_{c}^{\prime,i}-{\bm{x}}_{c}^{i}\right\|_{p}\leq\epsilon}\mathcal{K}\left(h_{\bm{\theta}}\left({\bm{x}_{c}^{i}}\right),h_{\bm{\theta}}\left({\bm{x}}_{c}^{\prime,i}\right)\right).$

Since class 3 and class 5 have an obvious one-to-one relation in Figure 4(b).
We only change the $\beta_{c}$ of class 5 to adjust the robustness of class 3,
while fixes the $\beta_{c}$ of other classes. The result is shown in Figure
4(c). Each line represents the class-wise robustness under the Temperature-PGD
attack. The title in the figures represents the $\beta_{c}$ value of each
class $c$, that is, ’66666X6666’ stands for $\beta_{c}=6$ ($\forall c\in C$
and $c\neq 5$), and the number 6 is chosen to be comparable to the experiment
of Zhang et al. (2019) ($\forall c\in C,\beta_{c}=6$).

Overall results. Figure 4(c) demonstrates that the robustness of class 3 can
be improved or reduced by adjusting the value of $\beta_{5}$. Specifically,
when $\beta_{5}=6\rightarrow\beta_{5}=0.5$, the robustness of class 3 changes
from 26.2% to 33.6% and the robustness of class 5 changes from 43.0% to 34.1%,
which shows that our method can effectively adjust the robustness of the most
vulnerable class 3, thereby reducing the class-wise disparity. Intuitively,
Other group-based relations can also be used to further balance the overall
robustness.

###  6.2. Adjust robustness at the inference phase

MNIST and CIFAR-10 are the most commonly used datasets for adversarial
training. However, the overall performance of robust models in MNIST usually
exceeds 95%, while this is only 50%-60% in CIFAR-10. We speculate that the
unified background of MNIST is one of the potential reasons why its
performance is better.

To verify our assumption, we add different backgrounds to each class of images
in MNIST to explore the role of the background. Specifically, we first modify
the original images into three-channel images and then add two sets of
background colors to the training set and test set of each class, as shown in
Figure 5(a). In the training and inference phase, $\epsilon$ is set to 0.5 to
highlight the robust relation between classes, and other settings are
consistent with Section 4. In addition, we have also verified that this
background-changing dataset has almost no effect on the accuracy of standard
training. Therefore, we only report the confusion matrices of adversarial
training as shown in Figure 5(b) and Figure 5(c).

The confusion pattern in Figure 5(c) is completely consistent with the
background relation of each class in Figure 5(a), which is the evidence that
the class-wise robust relation can be changed through the background. One
possible explanation is that the model mistakenly learned the spurious
correlation (Shen et al., 2018) between the foreground and the background
during the training process, e.g., the model may think that the number 2 and
the number 3 are more similar since they have the same background, while the
number 2 and the number 5 are vice versa. However, from the perspective of
causality (Shen et al., 2018), the intrinsic feature to judge whether numbers
are similar should be the shape rather than the background. In fact, Shen et
al. (2018) has proved that this phenomenon has a negative impact on model
prediction, but comparing the results of Figure 5(b) and Figure 5(c), it is
clearly demonstrated that the influence of the background on the adversarial
examples is much greater than that on the natural examples, which makes this
factor very important in adversarial settings. To the best of our knowledge,
this is the first step to explore the connection between background and model
robustness.

The proposed method. Inspired by the above phenomenon, we believe that the
complex background in the CIFAR-10 dataset may affect the robustness of each
class and we can use this property to adjust class-wise robustness at the
inference phase. To check this, we first select the images with the ground
truth class 3 in the test set and then record the confidence of adversarial
prediction corresponding to the class 3 of each image (i.e.,
$\mathbb{S}(h_{\bm{\theta}}({\bm{x}}_{i}^{\prime}))_{y=3}$, where $\mathbb{S}$
is the softmax function) and visualize images according to the confidence from
high to low. Surprisingly, we find that the backgrounds of the highly robust
images in class 3 are pure white color. Figure 6 shows the most robust image
(ID: 2261) in class 3 has this white background.

Therefore, we manually extract the mask that can locate the background from
one non-robust image (ID: 3634) of class 3 in the test set, and then replace
the original background with a white background to investigate the change of
prediction. As shown in Figure 6, the boxes represent the natural prediction
and robust prediction of the corresponding image. ‘Rank 1’ and ‘Rank 2’
represent the classes with the highest and the second-highest confidence, and
the value in brackets represents the specific confidence. The result indicates
that this non-robust image can become a robust one by replacing the
background, while it slightly affects the natural prediction.

We apply the above image processing method to all images of class 3 (1000
images) in the test set to verify whether the above phenomenon can be
generalized. As shown in Table 4. The number in each row represents how many
images in the corresponding test set are predicted to be the corresponding
class. Since the ground truth of all test images is class 3, the column
corresponding to class 3 is the number of images that are correctly predicted.

Overall results. As illustrated in Line 4 and Line 5 of Table 4, many non-
robust examples become robust after adding a white background (i.e., the
robustness changed from 26.2% to 40.3%), while Line 1 and Line 2 indicate
natural predictions are not sensitive to the background, which proves that the
background mainly has a great influence on the model’s adversarial prediction.
Furthermore, we combine the modified training method mentioned in Section 6.1,
and the robustness of class 3 becomes 43.5% (Line 6), which means that the
robustness of the most vulnerable class in CIFAR-10 has been greatly improved.

##  7\. Conclusion

In this paper, we have a closer look at the class-wise properties of the
robust model in benchmark datasets based on the observation that robustness
between each class has a recognizable gap. We conduct systematic analysis and
find: 1) In each dataset, classes can be divided into several subgroups, and
intra-group classes are easily flipped by each other. 2) The emergence of the
unbalanced robustness is closely related to the intrinsic properties of the
datasets. Furthermore, we make full use of the properties of the vulnerable
classes to propose an attack that can effectively reduce the robustness of
these classes, thereby increasing the disparity among classes. Finally, in
order to alleviate the robustness difference between classes, we propose two
methods to improve the robustness of the most vulnerable class in CIFAR-10
(i.e., class 3): 1) At the training phase: Modify loss function according to
group-based relation between classes. 2) At the inference phase: Change the
background of the original images. We believe our work can contribute to a
more comprehensive understanding of adversarial training and let researchers
realize that the class-wise properties are crucial to robust models.

##  8\. Acknowledgements

This work is supported in part by Key R&D Projects of the Ministry of Science
and Technology (No. 2020YFC0832500), National Natural Science Foundation of
China (No. 61625107, No. 62006207), National Key Research and Development
Program of China (No. 2018AAA0101900), the Fundamental Research Funds for the
Central Universities and Zhejiang Province Natural Science Foundation (No.
LQ21F020020). Yisen Wang is partially supported by the National Natural
Science Foundation of China under Grant 62006153, and CCF-Baidu Open Fund
(OF2020002).

## References

  * (1)
  * Athalye et al. (2018) Anish Athalye, Nicholas Carlini, and David Wagner. 2018\.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In _ICML_.
  * Bai et al. (2019) Yang Bai, Yan Feng, Yisen Wang, Tao Dai, Shu-Tao Xia, and Yong Jiang. 2019\.  Hilbert-Based Generative Defense for Adversarial Examples. In _ICCV_.
  * Bai et al. (2020) Yang Bai, Yuyuan Zeng, Yong Jiang, Yisen Wang, Shu-Tao Xia, and Weiwei Guo. 2020\.  Improving query efficiency of black-box adversarial attack. In _ECCV_.
  * Bai et al. (2021) Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. 2021\.  Improving Adversarial Robustness via Channel-wise Activation Suppressing. In _ICLR_.
  * Benz et al. (2020) Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. 2020\.  Robustness may be at odds with fairness: An empirical study on class-wise accuracy.  _arXiv preprint arXiv:2010.13365_ (2020).
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Towards evaluating the robustness of neural networks. In _S &P_.
  * Carmon et al. (2019) Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, and John Duchi. 2019.  Unlabeled data improves adversarial robustness. In _NeurIPS_.
  * Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee. 2011.  An analysis of single-layer networks in unsupervised feature learning. In _AISTATS_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _CVPR_.
  * Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015\.  Explaining and harnessing adversarial examples. In _ICLR_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.
  * Li et al. (2019) Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. 2019.  Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. In _ICML_.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018\.  Towards Deep Learning Models Resistant to Adversarial Attacks. In _ICLR_.
  * Nanda et al. (2021) Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. 2021.  Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning. In _FAccT_.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 2011\.  Reading digits in natural images with unsupervised feature learning.  (2011).
  * Pang et al. (2019) Tianyu Pang, Kun Xu, and Jun Zhu. 2019.  Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks. In _ICLR_.
  * Pang et al. (2020) Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Jun Zhu, and Hang Su. 2020\.  Boosting adversarial training with hypersphere embedding. In _NeurIPS_.
  * Raff et al. (2019) Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. 2019.  Barrage of random transforms for adversarially robust defense. In _CVPR_.
  * Raghunathan et al. (2019) Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. 2019\.  Adversarial training can hurt generalization.  _arXiv preprint arXiv:1906.06032_ (2019).
  * Shen et al. (2018) Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peixuan Chen. 2018.  Causally regularized learning with agnostic data selection bias. In _MM_.
  * Song et al. (2019) Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E Hopcroft. 2019.  Robust Local Features for Improving the Generalization of Adversarial Training. In _ICLR_.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013\.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ (2013).
  * Tang et al. (2020) Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. 2020\.  Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect. In _NeurIPS_.
  * Wang et al. (2021) Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. 2021\.  A unified approach to interpreting and boosting adversarial transferability. In _ICLR_.
  * Wang et al. (2019a) Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019a.  On the Convergence and Robustness of Adversarial Training.. In _ICML_.
  * Wang et al. (2019b) Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. 2019b.  Symmetric cross entropy for robust learning with noisy labels. In _ICCV_.
  * Wang et al. (2019c) Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. 2019c.  Improving adversarial robustness requires revisiting misclassified examples. In _ICLR_.
  * Wong et al. (2019) Eric Wong, Leslie Rice, and J Zico Kolter. 2019.  Fast is better than free: Revisiting adversarial training. In _ICLR_.
  * Wu et al. (2019) Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. 2019.  Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets. In _ICLR_.
  * Wu et al. (2020) Dongxian Wu, Shu-Tao Xia, and Yisen Wang. 2020.  Adversarial Weight Perturbation Helps Robust Generalization. In _NeurIPS_.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ (2016).
  * Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael I Jordan. 2019.  Theoretically Principled Trade-off between Robustness and Accuracy. In _ICML_.
  * Zhang et al. (2020) Shengyu Zhang, Ziqi Tan, Zhou Zhao, Jin Yu, Kun Kuang, Tan Jiang, Jingren Zhou, Hongxia Yang, and Fei Wu. 2020\.  Comprehensive information integration modeling framework for video titling. In _KDD_.
  * Zhang et al. (2021) Shengyu Zhang, Dong Yao, Zhou Zhao, Tat-Seng Chua, and Fei Wu. 2021.  CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation. In _SIGIR_.

##  Appendix A Appendix: hyperparameters for reproducibility

###  A.1. Hyperparameters for defenses in Section 4

MNIST setup. Following Zhang et al. (2019), we use a four-layers CNN as the
backbone. In the training phase, we adopt the SGD optimizer with momentum
$0.9$, weight decay $2\times 10^{-4}$ and an initial learning rate of $0.01$,
which is divided by 10 at the 55th, 75th and 90th epoch (100 epochs in total).
Both the training and testing attacker are 40-step PGD (${\rm PGD^{40}}$) with
random start, maximum perturbation $\epsilon=0.3$ and step size $\alpha=0.01$.

CIFAR-10 & CIFAR-100 setup. Like Wang et al. (2019c) and Zhang et al. (2019),
we use ResNet-18 (He et al., 2016) as the backbone. In the training phase, we
use the SGD optimizer with momentum $0.9$, weight decay $2\times 10^{-4}$ and
an initial learning rate of $0.1$, which is divided by 10 at the 75th and 90th
epoch (100 epochs in total). The training and testing attackers are ${\rm
PGD^{10}}/{\rm PGD^{20}}$ with random start, maximum perturbation
$\epsilon=0.031$ and step size $\alpha=0.007$.

SVHN & STL-10 setup. All settings are the same to CIFAR-10 & CIFAR-100, except
that the initial learning rate is 0.01.

ImageNet setup. Following Wong et al. (2019), we use ResNet-50 (He et al.,
2016) as the backbone. Specifically, in the training phase, we use the SGD
optimizer with momentum $0.9$ and weight decay $2\times 10^{-4}$. A three-
stage learning rate schedule is used as the same with Wong et al. (2019). The
training attacker is ${\rm FGSM}$ (goodfellow2015explaining) with random
start, maximum perturbation $\epsilon=0.007$, and the testing attacker is
${\rm PGD^{50}}$ with random start, maximum perturbation $\epsilon=0.007$ and
step size $\alpha=0.003$.

###  A.2. Hyperparameters for attacks in Section 4.3

FGSM setup. Random start, maximum perturbation $\epsilon=0.031$.

PGD setup. Random start, maximum perturbation $\epsilon=0.031$. For RST model,
step size $\epsilon=0.01$ and steps $\alpha=40$, following Carmon et al.
(2019). For other models, step size $\epsilon=0.003$ and steps $\alpha=20$.

CW∞ setup. Binary search steps $b=5$, maximum perturbation times $n=1000$,
learning rate $lr=0.005$, initial constant $c_{0}=0.01$, $\tau$ decrease
factor $\gamma=0.9$. Similar to Carmon et al. (2019), we randomly sample 2000
images to evaluate model robustness, and 200 images per class.

Transfer-based attack setup. All settings are the same to PGD for the
substitute standard model.

$\bf{\mathcal{N}}$ attack setup. Random start, maximum perturbation
$\epsilon=0.031$, population size $n_{pop}=300$, noise standard deviation
$\sigma=0.1$ and learning rate $lr=0.02$. Similar to Li et al. (2019), we
randomly sample 2000 images to evaluate model robustness, and 200 images per
class.

##  Appendix B Appendix: Discussion with the class-wise accuracy in standard
training

Table 5. Statistical metric: class-wise variance(CV) and maximum class-wise discrepancy(MCD) of the benchmark datasets Dataset |  | Standard training
---
(natural accuracy)
| Adversarial training
---
(robustness accuracy)
CV(%) | MCD(%) | CV(%) | MCD(%)
MNIST | 0.11 | 0.98 | 2.71 | 6.00
SVHN | 2.78 | 5.47 | 110.25 | 37.10
CIFAR-10 | 10.75 | 10.80 | 284.43 | 52.80
STL-10 | 151.98 | 36.13 | 250.06 | 51.00
CIFAR-100 | 115.62 | 30.00 | 212.82 | 44.00
ImageNet | 501.70 | 68.00 | 637.21 | 78.00

The body part of this paper mainly focuses on the adversarial robustness of
the model. One possible concern is that even with standard training, the
class-wise accuracy may not be exactly the same. Indeed, in order to highlight
the difference of this phenomenon between adversarial training and standard
training. We also report the CV and MCD of these two training methods since
these metrics can quantitatively reflect the discrepancy between classes.

In our experiment, the pipeline and hyperparameters of the standard training
are consistent with the adversarial training, except that the adversarial
examples are not added to the training set. From the results in Table 5, it
can be found that in all datasets, the class-wise variance (CV) and maximum
class-wise discrepancy (MCD) of the adversarial training are significantly
larger than the standard training, especially for the most commonly used
dataset in the adversarial community, CIFAR-10, the CV of adversarial training
is 28 times that of standard training, and the MCD of adversarial training is
5 times that of standard training. This shows that class-wise properties in
the robustness model are worthy of attention.


# 3D Printed Graded Porous Sensors for Soft Sensorized Insoles with Gait Phase
& Ground Reaction Forces Estimation

Nick Willemstein
Department of Biomechanical Engineering
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>
Saivimal Sridar
Department of Biomechanical Engineering
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>
Herman van der Kooij
Department of Biomechanical Engineering
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>
Ali Sadeghi
Department of Biomechanical Engineering
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>

###### Abstract

Sensorized insoles provide a tool to perform gait studies and health
monitoring during daily life. These sensorized insoles need to be comfortable
and lightweight to be accepted. Previous work has already demonstrated that
sensorized insoles are possible and can estimate both ground reaction force
and gait cycle. However, these are often assemblies of commercial components
restricting design freedom and flexibility. Within this work, we investigate
the feasibility of using four 3D-printed porous (foam-like) piezoresistive
sensors embedded in a commercial insole. These sensors were evaluated using an
instrumented treadmill as the golden standard. It was observed that the four
sensors behaved in line with the expected change in pressure distribution
during the gait cycle. In addition, Hammerstein-Wiener models were identified
that were capable of estimating the vertical and mediolateral ground reaction
forces (GRFs). Their NRMSE fits were on average 82% and 73%, respectively.
Similarly, for the averaged gait cycle the $R^{2}$ values were 0.98 and 0.99
with normalized RMS errors overall below 6%. These values were comparable with
other insoles based on commercial force sensing resistors but at a
significantly lower cost (over four times cheaper). Thereby indicating that
our 3D-printed sensors can be an interesting option for sensorized insoles.
The advantage of 3D printing these sensors is that it allows for significantly
more design freedom, reduces assembly, and is cheaper. However, further
research is needed to exploit this design freedom for complex sensors,
estimate the anteroposterior GRF, and fully 3D print the entire insole.

##  1 Introduction

With the current boom in the use of advanced health monitoring technologies
for both medical and personal use, wearables have become a large part of day-
to-day lives 1. The use of smartwatches and heart rate monitors for sports,
and recreational activities, as well as monitoring of health conditions, is
steadily on the rise to maintain optimal physical performance2. Biological
markers such as cardiac rhythm, blood pressure, and respiratory variables have
been investigated using smartphones and -watches 3, 4 to predict the onset of
ailments such as sleep apnea and stroke. Although useful, these technologies
are limited to the estimation of a hand full of bio variables only restricted
by the type of sensors available for use5.

In the field of kinesiology, the estimation of moments about the joints in the
human body has been vastly studied for understanding human biomechanics as
well as applying them to sports science and wearable robotics. For the
estimation of biological joint torques, the use of motion capture systems in
conjunction with force plates has been widely utilized 6. This setup is highly
accurate but cannot be utilized in outdoor scenarios. Therefore, wearable
sensors offer several benefits over commercially available smart devices such
as phones and watches. Typically, inertial measurement units have been
utilized to estimate the kinematics of the human body during outdoor tasks 7.
However, several assumptions such as person-specific limb mass and the
location center of mass have to be made and can be tedious to obtain due to
the required complex equipment such as 3D scanners8.

Therefore, several groups have worked on designing insole sensors for
measuring ground reaction forces via the use of several types of sensors.
Commercially available load cells have been utilized for sensing insoles 9 but
have the downside of being bulky. Hence, soft sensors could be a good
candidate for force sensing as demonstrated by Kong et. al10 via air pressure
sensors embedded in the shoe. Furthermore, several insoles sensors that use
capacitive sensors embedded in textiles11 as well as piezo-resistive sensors12
have been researched and are only limited to measurement of normal force and
the center of pressure, not shear. Typically, the aforementioned sensors also
require careful and manual manufacturing processes such as moulding13 and
manually coiling a tube 14. Therefore techniques such as 3D printing can
reduce the complexity of manually manufacturing these devices as well as human
error.

With the advances in additive manufacturing, it is now possible to 3D print
compliant materials such as thermoplastic elastomers15. Furthermore, the
printing of these materials is not limited to the material properties but
several groups have demonstrated the printing of conductive polymers16 and
their applications in sensors. Moreover, 3D printing has been utilized to
customize shoe insoles for diabetic feet using porous structures 17. These
insoles could be further printed using a sensorized material and used to
measure contact forces at the foot during various activities. This work aims
to bridge the gap literature by presenting a sensorized insole using 3D
printed graded porous sensors to estimate GRFs as well as gait phases during
walking as shown in Fig. 1. Compared to other soft insole sensors in
literature, the presented insoles leverages additive manufacturing to
fabricate the sensing elements. Furthermore, the presented sensors can be
manufactured at a cost four times lower than insoles assembled with off-the-
shelf force sensing resistors.

Figure 1: Illustration of a user walking over the ground wearing gait sensing
insoles. A schematic of the insole sensor with the individual sensing elements
is presented. Further, the measured ground reaction forces and individual
sensing element outputs are depicted in the plot.

This paper is organized as follows. Section II details the design of the soft
sensorized insole, its fabrication method, and the test procedure used to
evaluate the insole. Section III, presents the results from the estimation of
gait cycles during walking, vertical ground reaction forces, and estimation of
mediolateral GRFs using system identification. The conclusions from this work
and future work are detailed in Section IV.

##  2 Materials and Methods

###  2.1 Soft Sensorized Insole Design & Force Estimation Method

The insole consists of four individual layers: a soft foam layer that embeds
the sensor, two insulating (top and bottom) layers, and a layer with grounding
electrodes (Fig. 2(a)). The purpose of the insulating layers was to prevent
the user from creating an additional conductive path. In addition, they were
used to package the entire structure.

Figure 2: The four layers of the sensorized insole (a) with sensor location
toe (TO), metatarsal (MT), midfoot (MF), and heel (HL), and pipeline from
individual sensor’s output voltage through Hammerstein-Wiener model to ground
reaction forces (b) and a sensor embedded in foam (c).

The sensing was implemented using the other two layers by adding
piezoresistive porous (foam-like) sensors. These piezoresistive sensors 18,
19, 20 change their resistance when subjected to a load. The resistance of the
sensors was measured using a voltage divider (see Fig. 2(b)). To realize the
voltage dividers a common grounding layer was added inside the insole. A small
strip was left at the edge of the common ground layer to provide a place
outside the user’s shoe for connecting the grounding electrode.

To estimate the gait cycle and ground reaction force, four sensors were added.
These sensors were distributed over the entire length of the foot.
Specifically, the sensors were added at the heel(HL), midfoot(MF),
metatarsal(MT), and toes(TO) (Fig. 2(a)). These locations allow for the
reconstruction of aspects of the gait cycle such as heel strike and toe-off.

The four piezoresistive porous sensors change resistance when subjected to a
force. Within this work, the change in resistance $\Delta R$ is used for force
estimation. The change in resistance is defined as (with $R_{0}$ the
resistance when the user is standing still at the start)

$\Delta R(t)=\frac{R(t)-R_{0}}{R_{0}}\cdot 100\%$ (1)

This change in resistance can be exploited to estimate the gait cycle
behavior. Unfortunately, the change in resistance of piezoresistive porous
sensors is nonlinear and exhibits hysteresis. This behavior needs to be
compensated for to estimate the ground reaction forces more accurately. In
literature, people solve this by using a model-based compensation, such as a
Hammerstein-Wiener (HW) model21, 18, training a neural network22 and/or other
machine learning tools 23. Within this work, the HW model is employed as it is
simple yet able to provide good force estimates, as seen in 21. This model
class is shown in Figure 2(b) and consists of three components: two nonlinear
functions ($f_{1},f_{2}$) and a linear transfer function ($G(j\omega)$) in the
middle, which gives

$F(t)=f_{2}\left(G(j\omega)\left(f_{1}\left(\Delta R(t)\right)\right)\right)$
(2)

Within this equation, the variable $F(t)$ represents the ground reaction
forces that need to be estimated. Note that the nonlinear input function
$f_{1}$ consists of four nonlinear functions (one for each sensor). The HW
models can be identified using MATLAB’s (The MathWorks, USA) System
Identification Toolbox. Within this work, piecewise linear activation
functions were used as these gave good results for strain18 and force
estimation21. The number of breakpoints was determined experimentally by
trying different combinations in the range of five to ten per activation
function. Combining the four sensors and an HW model allows for both gait
cycle estimation and ground reaction force estimation.

###  2.2 Insole Prototype

Fabrication of the insole was done in five steps. The first step was to print
the porous sensors. The sensors were printed using our InFoam method 24. This
method exploits the liquid rope coiling effect to print structures with a
graded porosity. Specifically, it uses the empty space of the coils to create
pores leading to a structure with programmable porosity. Within this work, a
modified Ender 5 Plus (Shenzhen Creality 3D Technology Co., Ltd., China) with
a custom screw extruder was used. The screw extruder used TC7OEX-BLCK pellets
(Kraiburg TPE, Germany) that have a Shore Hardness of 70A and volume
resistivity of 10 $\Omega$cm. These were printed at 195∘C with a 0.6 mm
nozzle. Using this material three blocks of 60x60x8 mm3 were printed that were
cut to 60 mm by [20,25,20,30] mm blocks for the toe, metatarsal, midfoot, and
heel sensor, respectively. The blocks were printed with three levels of
porosity [84,71,64]%. Other works indicate that the introduction of a
stiffness gradient can enable a larger range of measurement 25.

In the second step, a sheet of polyethylene foam (thickness of 5 mm) was cut
to shoe size 43 (EU). Four pockets of appropriate size (see above) were cut at
the toe, metatarsal, midfoot, and heel for embedding the sensors.
Subsequently, the 3D printed sensors were placed in each pocket, as seen in
Fig. 2(c). An electrode (a copper wire) was attached to every sensor. These
electrodes were soldered to the sensor by using the conductive TPE itself
(Fig. 2(c)). Lastly, the wires were guided through the insole’s edge by using
tape to have a single point of entry on the side of the foot.

Afterwards, the common grounding layer was added. using a woven sheet of
conductive textile (Adafruit, USA). Lastly, to finalize the insole, an
insulating layer was added on top and the bottom of the insole by adding
insulating tape.

###  2.3 Sensorized Insole Testing

The sensorized insole was embedded in a shoe (Fig. 3(a)) for testing. The
wiring of the insole was attached to the subject by using elastic bands with
velcro (Fig. 3(a)) to ensure the wires do not influence the walking motion.
The subject, who does not have an abnormal gait pattern, walked on an
instrumented treadmill (Qualisys AB, Sweden) with split belt force plates
(AMTI, USA). The instrumented treadmill was used to record the ground reaction
forces of both feet. Walking trials were performed on the treadmill with
speeds of 1, 1.5, and 2 m/s for approximately 40 seconds each. This walking
trial was performed three times. During these trials, the changes in
resistance were recorded using the measurement setup of Fig. 3(b). The first
walking trial was used for identification. Whereas the other two trials are
used as validation datasets for the identified models. The subject was one of
the authors who consented to participate in the study. Within this study, we
only performed technical tests during simple walking and no personal data was
collected. Therefore, no review was performed.

Figure 3: The subject wearing the insole on an instrumented treadmill and
embedded in a shoe(a), and the measurement system of a single insole (b).

The measurement setup used a voltage divider to measure the resistance of the
sensor using a bias resistor $R_{b}$ of 560$\Omega$. The output voltage was
measured using a 16-bit analog-to-digital converter ADS1115 (Texas
Instruments, USA) on a breakout board (Adafruit, USA) that is connected to an
Arduino Uno (Arduino AG, Italy). The Arduino streamed data over USB to a
laptop for further processing. The measured voltages were then converted to
resistance by

$R=R_{b}\frac{1}{V_{in}/V_{out}-1}$ (3)

Within this equation, the variables $V_{in}$ and $V_{out}$ represent the input
(5 V from Arduino) and measured voltage, respectively. The resistance changes
were then computed by using $R$ and Eq. 1. The changes in resistance were then
used to identify a linear and Hammerstein-Wiener model. The model with the
highest normalized root mean square fit (NRSME) kept, which was defined as
(using the two norm):

$\textrm{NRMSE fit}=100\left(1-\frac{||F-\hat{F}||}{||F-\bar{F}||}\right)$ (4)

Within this equation, the variables $F$, $\hat{F}$, and $\bar{F}$ represent
the measured force, predicted force, and average (measured) force. In
addition, the models were also compared in terms of their RMS error scaled by
the maximum force. Lastly, the coefficient of determination $R^{2}$ was
computed by comparing the measured and estimated GRF.

##  3 Results and Discussion

###  3.1 Gait Cycle Estimation

The vertical ground reaction force (GRF) versus gait cycle is shown in Fig.
4(a,b) in error bar format. A typical gait cycle can be observed with a clear
heel strike ($\approx 5$%), mid stance ($\approx 30$ to 50%), and toe-off
($\approx 75$%)6. The resistance change curve (Fig. 4(b)) has an overall
similar shape as the gait cycle but is inverted. The inverted shape is to be
expected as an increased force means a decrease in resistance. Interestingly,
there seems to be some overlap with regions of higher variance in the force
transients (i.e. between heel strike and mid stance and after toe-off) in both
the GRF and resistance change, which implies a correlation between the force
magnitude and resistance change that can be beneficial in force estimation.

Figure 4: Behavior during the gait cycle of vertical forces (a), resistance
change (b), and gait phase (c).

Besides similarities in the general shape, the sensors show the change in
pressure distribution over the gait cycle, which corresponds with the gait
phase (Fig. 4(c)). During heel strike and towards midstance the heel, midfoot,
metatarsal, and toe sensors activate successively. Indicating a gradual
(expected) change in force distribution from the heel to encompassing the
entire foot during mid-stance. Within the mid-stance, all sensors measure
something. This behavior indicates that the entire foot is on the ground.
However, one can see an increase in force on the midfoot, metatarsal, and toe
sensor indicating that the subject is shifting weight towards his toes (as
expected). Afterward from mid stance towards toe off, all sensors show
relaxation in the same order as the heel strike event (i.e. from heel to toe).
This successive relaxation indicates that an increasing number of sensors do
not experience a load anymore. Moreover, the shift in force towards the
metatarsal/toe should be right before the toe off, as also seen in our sensor.
These results imply that the sensors correctly capture the force distribution.

###  3.2 Vertical Ground Reaction Forces

The vertical (V) and mediolateral (ML) GRFs were estimated using identified HW
and linear models based on the changes in resistance. The HW and linear models
were identified separately for the V and ML GRFs for both feet. The averaged
GRFs of the identified models for the V GRF are shown in Fig. 5(a-d) for both
feet. These figures show the identified models’ performance for the
identification (5(a,c)) and validation dataset (5(b,d)). In general, it can be
observed that the linear model is better at capturing the two peaks whereas
the low forces are better captured by the HW model. However, both seem to
follow the trajectory of the force plate.

Figure 5: Averaged behavior of the identified models for the vertical(a-d)
ground reaction forces with (a,c)/(b,d) the results for the identification and
validation dataset of the left and right foot, respectively, and (e/f) the
averaged identification/validation datasets for the left foot mediolateral
(ML) ground reaction forces.

The overlap is confirmed by the NRMSE (normalized root mean square) fit of the
time-series data (computed using MATLAB’s compare-function) of 87/81% for the
identification set for the HW/linear model of the left foot, respectively.
Similar to our previous work 18 the HW model outperforms the linear model
significantly with an error reduction of 31.5%. The capability of the HW model
to better estimate force is also seen for the validation datasets with NRMSE
fits of [81,75]% and [74,67] % for the HW and linear model for the left foot,
respectively. Thereby providing evidence that the HW model also generalized
well with the other datasets (as its fit was reduced slightly less). Similar
results were obtained for the right foot with identification and validation
NRMSE fits for the HW and linear model averaging 84.1% and 78.0%,
respectively. Indicating that this model type is suitable for the estimation
of vertical GRFs. For visualization of the estimation performance, the
estimated and real GRFs during gait are visualized in the supplementary movie.
The estimated and real GRF were evaluated for its $R^{2}$-value (based on the
averaged gait cycle), which gave an average of 0.976 and 0.982 for the linear
and HW model, respectively. Thereby indicating that both models correlate
quite well with the GRF. Similarly, the RMS error normalized by the maximum
force also indicated good performance with an average of 6.24 and 3.72% for
the linear and HW model, respectively. Indicating again that the HW model is
better overall. Although it is interesting to note that the shape of the
linear model is better but overall it seems less capable of estimating the
GRF. A more in-depth comparison of this estimation performance with other
insoles is provided in Section III-D.

###  3.3 Mediolateral Ground Reaction Forces

The mediolateral (ML) GRFs were also predicted using the changes in resistance
and identified models. Their result is shown in Fig. 5(e) and (f) for the
identification and validation dataset of the left foot, respectively. Similar
to the vertical GRFs the estimated models behave similarly to the ground truth
showing similar peak-valley-peak behavior. Although the HW model has better
captured this trajectory than the vertical GRF. The first peak seems to be
captured better than the second peak in terms of magnitude but the overall
behavior is consistent with the force plate data.

The same observation as for the vertical GRFs can be made namely: the linear
model is less accurate than the HW model. This observation is made for both
the identification (NRMSE fit of the time-series data of 80% vs. 74%) and the
validation datasets ([72,69]% and [67,63]%) of the left foot. Similarly, the
linear and HW model for the ML GRF were also identified for the right foot.
The NRMSE fits of these models gave an average of 73.7 and 68.1 % for the HW
and linear models, respectively. Thereby indicating comparable results to the
left foot. Similarly, the $R^{2}$-values were 0.990(HW) and 0.983(linear)
implying that the HW correlates slightly better. Lastly, even with the
(relatively) lower NRMSE fits the range (maximum minus minimum) normalized RMS
error was 3.9% (HW) and 5.1% (linear), respectively. Thereby implying a
reasonable estimation of the mediolateral GRF. Similar to the vertical GRFs,
the predicted and real ML GRFs are visualized in the supplementary movie as
well. It is important to note that these fits are lower than the vertical GRFs
in general.

###  3.4 Insole Comparison

The developed insoles are capable of estimating the gait cycle and the ground
reaction forces (GRF). To put the acquired fits into context the insole will
be compared to several force-sensing insoles. The focus will be on force
sensing resistors (FSRs) as these are the most comparable technology. Firstly,
the cost of fabricating (customizable) sensorized insoles in-house is much
lower than assembling them using off-the-shelf FSRs. As both are
piezoresistive sensors the required electronics are assumed to be equivalent.
For printing both insoles completely around 85 grams of material would be
necessary. Assuming a cost of 5-100 euro/kg for elastomeric pellets, the total
price would range from 0.43 to 8.50 euros for both insoles. In contrast, if
one would assemble the insole using commercial off-the-shelf FSRs with the
cheapest around 3.87 euros on Farnell (The Netherlands) then two insoles with
four sensors would cost 30.96 euros. Thus, printing the insoles is 73.7-3.64
times cheaper while also providing more design freedom compared to assembling
off-the-shelf FSRs. Lastly, the inherent soft foam-like nature of our sensors
could make the insole more comfortable for the user. In terms of performance,
the presented insoles are capable of estimating the phases of gait during
walking making them on par with other insoles such as26, 11. Similar to those
works our insoles can provide gait cycle data. However, our insoles can also
estimate ground reaction forces during gait. A table comparing our results
with three other insoles is shown in Table 1 (averaging the left and right
foot results). The absolute RMS error is the RMS error without scaling with
the maximum force. The most notable aspect is that all works use different
models for compensation and/or multiple subjects. This difference makes
absolute comparisons of the metrics difficult. It can be observed that our
insoles perform comparably in terms of the coefficient of determination
$R^{2}$ for the vertical forces ($V$). The insole of 14 comes near our $R^{2}$
but was only based on their repeatability testing, not GRFs. The RMSE values
(based on the averaged dataset) vary significantly but are in the same order
of magnitude. Normalizing by the maximum force shows similar values.

Table 1: Table with selection of sensorized insoles ($V$,$ML$, ANN= Vertical, Mediolateral, artificial neural network) Metric | Our Insole | FSR-122 |  FSR-2 23 |  Pneumatic 14
---|---|---|---|---
No. sensors | 4 | 6 | 6 | 4
Principle | Piezoresistive | Piezoresistive | Piezoresistive | Pressure
Model | HW | ANN | Random Forest | Hysteresis Compensator
$R^{2}$ ($V$, $ML$) (%) | 0.98, 0.99 | 0.96 | 0.91,0.59 | 0.98*
Absolute RMSE ($V$, $ML$) (N) |  $\pm$37.3, $\pm$2.26 |  $\pm$32.7, n/a |  $\pm$65.7, $\pm$10.7 |  $\pm$194, n/a
Rounded RMSE ($V$, $ML$) (%) | 4, 4 | 5, n/a | 8, 13 | 32, n/a

The mediolateral ($ML$) forces were also estimated in 23. Their $R^{2}$ is
much lower than ours just like their RMS error. Indicating that our sensor is
quite capable of estimating $ML$ forces. Overall our insole sensors are at
least comparable to other insoles. However, our insole uses two sensors less
than the other two FSR-based insoles. With its comparable performance, it does
indicate that for gait analysis our porous sensors can work similarly to FSR-
based insoles. However, the other two works also investigated the center of
pressure (both) and the ante posterior GRF 23. These were not investigated
and, therefore, not compared.

##  4 Conclusion

Sensorized insoles can provide insight into aspects such as gait studies and
health monitoring. For such insoles to be used in daily life they need to be
discrete, lightweight, and non-intrusive. Within this work, we showcased a
soft sensorized insole with 3D-printed porous (foam-like) sensors embedded.

The resistance changes versus the gait cycle indicate that the gait pattern
can be deduced from the sensor data. Moreover, the identified models for the
vertical GRF with NRMSE fits of 87-75% for the HW models and slightly lower
for the linear (81-67%). A similar result is seen for the mediolateral GRF
albeit a bit lower (80-66% and 74-63%, respectively). The estimation
capabilities of our insoles are comparable to three other insoles (two FSRs
and a pneumatic). Thereby indicating that our 3D printed sensors can reach
similar performance at a much lower price point (more than 4x as cheap).

Future research on these insoles should include investigating their resilience
over longer timespans. In addition, the full 3D printing of the sensorized
insole can reduce cost and increase design flexibility. The full 3D printing
of the insole also allows for the integration of more complex sensors and/or
electrode placement, which can improve GRF estimation (mediolateral and
vertical). Furthermore, these sensors could enable the estimation of the
centers of pressure and ante-posterior GRFs. This addition would allow for a
full reconstruction of the GRFs during gait. Also, other movements could be
investigated to evaluate their performance in other scenarios.

## References

  * 1 M. N. K. Boulos, S. Wheeler, C. Tavares, and R. Jones, “How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from ecaalyx,” _Biomedical engineering online_ , vol. 10, no. 1, pp. 1–14, 2011.
  * 2 M. Pobiruchin _et al._ , “Accuracy and adoption of wearable technology used by active citizens: a marathon event field study,” _JMIR mHealth and uHealth_ , vol. 5, no. 2, p. e6395, 2017.
  * 3 F. Michard, “Toward smart monitoring with phones, watches, and wearable sensors,” _Anesthesiology Clinics_ , vol. 39, no. 3, pp. 555–564, 2021.
  * 4 T. Liang and Y. J. Yuan, “Wearable medical monitoring systems based on wireless networks: A review,” _IEEE Sensors Journal_ , vol. 16, no. 23, pp. 8186–8199, 2016.
  * 5 T.-C. Lu, C.-M. Fu, M. H.-M. Ma, C.-C. Fang, and A. M. Turner, “Healthcare applications of smart watches,” _Applied clinical informatics_ , vol. 7, no. 03, pp. 850–869, 2016.
  * 6 M. W. Whittle, _Gait analysis: an introduction_. Butterworth-Heinemann, 2014.
  * 7 A. Karatsidis, G. Bellusci, H. M. Schepers, M. De Zee, M. S. Andersen, and P. H. Veltink, “Estimation of ground reaction forces and moments during gait using only inertial motion capture,” _Sensors_ , vol. 17, no. 1, p. 75, 2016\.
  * 8 P. L. Davidson, S. J. Wilson, B. D. Wilson, and D. J. Chalmers, “Estimating subject-specific body segment parameters using a 3-dimensional modeller program,” _Journal of biomechanics_ , vol. 41, no. 16, pp. 3506–3510, 2008\.
  * 9 P. H. Veltink, C. Liedtke, E. Droog, and H. van der Kooij, “Ambulatory measurement of ground reaction forces,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 13, no. 3, pp. 423–427, 2005.
  * 10 K. Kong and M. Tomizuka, “A gait monitoring system based on air pressure sensors embedded in a shoe,” _IEEE/ASME Transactions on mechatronics_ , vol. 14, no. 3, pp. 358–370, 2009.
  * 11 Q. Zhang, Y. L. Wang, Y. Xia, X. Wu, T. V. Kirk, and X. D. Chen, “A low-cost and highly integrated sensing insole for plantar pressure measurement,” _Sensing and Bio-Sensing Res._ , vol. 26, p. 100298, 2019.
  * 12 R. de Fazio, E. Perrone, R. Velázquez, M. De Vittorio, and P. Visconti, “Development of a self-powered piezo-resistive smart insole equipped with low-power ble connectivity for remote gait monitoring,” _Sensors_ , vol. 21, no. 13, p. 4539, 2021.
  * 13 K. Kim, S. Shin, and K. Kong, “An air-filled pad with elastomeric pillar array designed for a force-sensing insole,” _IEEE Sensors Journal_ , vol. 18, no. 10, pp. 3968–3976, 2018.
  * 14 S. Wachtel, B. Hassett, Z. Qiao, P. T. Chinimilli, and W. Zhang, “Design and characterization of shoe embedded pressure sensors for gait analysis and rehabilitation,” in _Frontiers in Biomedical Devices_ , vol. 40672\. American Society of Mechanical Engineers, 2017, p. V001T05A005.
  * 15 M. Saari, M. Galla, B. Cox, P. Krueger, A. Cohen, and E. Richer, “Additive manufacturing of soft and composite parts from thermoplastic elastomers,” in _2015 International Solid Freeform Fabrication Symposium_. University of Texas at Austin, 2015.
  * 16 M. Criado-Gonzalez, A. Dominguez-Alfaro, N. Lopez-Larrea, N. Alegret, and D. Mecerreyes, “Additive manufacturing of conducting polymers: Recent advances, challenges, and opportunities,” _ACS Applied Polymer Materials_ , vol. 3, no. 6, pp. 2865–2883, 2021.
  * 17 Z. Ma _et al._ , “Design and 3d printing of adjustable modulus porous structures for customized diabetic foot insoles,” _International Journal of Lightweight Materials and Manufacture_ , vol. 2, no. 1, pp. 57–63, 2019\.
  * 18 N. Willemstein, H. van der Kooij, and A. Sadeghi, “3d printed proprioceptive soft fluidic actuators with graded porosity,” _arXiv preprint arXiv:2302.13141_ , 2023.
  * 19 S. Nakamaru, R. Nakayama, R. Niiyama, and Y. Kakehi, “Foamsense: Design of three dimensional soft sensors with porous materials,” in _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ , 2017, pp. 437–447.
  * 20 S. P. Murali Babu _et al._ , “Sensorized foam actuator with intrinsic proprioception and tunable stiffness behavior for soft robots,” _Advanced Intelligent Systems_ , vol. 3, no. 6, p. 2100022, 2021.
  * 21 M. Y. Saadeh and M. B. Trabia, “Identification of a force-sensing resistor for tactile applications,” _Journal of Intelligent Material Systems and Structures_ , vol. 24, no. 7, pp. 813–827, 2013.
  * 22 H. S. Choi, C. H. Lee, M. Shim, J. I. Han, and Y. S. Baek, “Design of an artificial neural network algorithm for a low-cost insole sensor to estimate the ground reaction force (grf) and calibrate the center of pressure (cop),” _Sensors_ , vol. 18, no. 12, p. 4349, 2018.
  * 23 B. Oubre, S. Lane, S. Holmes, K. Boyer, and S. I. Lee, “Estimating ground reaction force and center of pressure using low-cost wearable devices,” _IEEE Transactions on Biomedical Engineering_ , vol. 69, no. 4, pp. 1461–1468, 2021.
  * 24 N. Willemstein, H. van der Kooij, and A. Sadeghi, “3d printing of soft fluidic actuators with graded porosity,” _Soft matter_ , vol. 18, no. 38, pp. 7269–7279, 2022.
  * 25 A. Mohammadi, Y. Tan, P. Choong, and D. Oetomo, “Flexible mechanical metamaterials enabling soft tactile sensors with multiple sensitivities at multiple force sensing ranges,” _Scientific reports_ , vol. 11, no. 1, pp. 1–9, 2021.
  * 26 M. Beccatelli _et al._ , “All-polymeric pressure sensors based on pedot: Pss-modified polyurethane foam,” _ACS Applied Polymer Materials_ , vol. 3, no. 3, pp. 1563–1572, 2021.