# DEDGAT: Dual Embedding of Directed Graph Attention Networks for Detecting
Financial Risk

Jiafu Wu∗, Mufeng Yao  ,  Dong Wu, Mingmin Chi  Fudan UniversityShanghaiChina
and  Baokun Wang$\dagger$, Ruofan Wu, Xin Fu
Changhua Meng, Weiqiang Wang  Tiansuan Lab,Ant GroupShanghaiChina

(2023)

###### Abstract.

Graph representation plays an important role in the field of financial risk
control, where the relationship among users can be constructed in a graph
manner. In practical scenarios, the relationships between nodes in risk
control tasks are bidirectional, e.g., merchants having both revenue and
expense behaviors. Graph neural networks designed for undirected graphs
usually aggregate discriminative node or edge representations with an
attention strategy, but cannot fully exploit the out-degree information when
used for the tasks built on directed graph, which leads to the problem of a
directional bias. To tackle this problem, we propose a Direc-ted Graph
ATtention network called DGAT, which explicitly takes out-degree into
attention calculation. In addition to having directional requirements, the
same node might have different representations of its input and output, and
thus we further propose a dual embedding of DGAT, referred to as DEDGAT.
Specifically, DEDGAT assigns in-degree and out-degree representations to each
node and uses these two embeddings to calculate the attention weights of in-
degree and out-degree nodes, respectively. Experiments performed on the
benchmark datasets show that DGAT and DEDGAT obtain better classification
performance compared to undirected GAT. Also. the visualization results
demonstrate that our methods can fully use both in-degree and out-degree
information.

Graph Attention Network, Financial Risk Control, Directed Graph, Dual
Embedding

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; April 30–May 4, 2023; Austin, USA††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††copyright: none††ccs: Computing
methodologies Machine learning Figure 1. Importance score distribution of in-
degree and out-degree edge weights in a financial risk control dataset with
our proposed methods DGAT and DEDGAT compared to GAT on a directed graph. The
out-degree edges are far less important than the in-degree ones for GAT. This
is defined as a directional bias. Our methods can solve this problem and
aggregate in-degree and out-degree represetations in a balanced manner.

##  1\. Introduction

Financial risk control is the core of Internet financing and one of the main
applications of artificial intelligence (li2009network, ; aziz2019machine, ;
ma2019financial, ; yang2020construction, ). In risk control scenarios, there
are often various relationships between entities, such as transfer
relationship, device relationship, regional connection, etc. Using graph
structure to modeling the relationship can help us discover abnormal objects
in the financial system.

Graph-based methods can often be divided into shallow graph methods and graph
neural network (GNN) methods. The shallow methods represented by DeepWalk
(perozzi2014deepwalk, ), node2vec (grover2016node2vec, ), LINE (tang2015line,
) can only learn the shallow represetation information of the graph, so the
classification performance is limited. Different from shallow methods, GNN can
automatically learn high-level features of graphs in a supervised manner and
utilize the features of nodes and edges to improve the performance of
downstream tasks such as node or edge classification. Graph Convolutional
Network (GCN) (kipf2016semi, ) uses convolution to aggregate node information.
GraphSage (hamilton2017inductive, ) uses neighbor sampling to aggregate the
neighborhood information of nodes, making training on large-scale graph data
possible. Graph Attention Network (GAT) (velivckovic2017graph, ) aggregates
the information of each node and its neighbor nodes according to the weighted
attention, which is widely used in the task of graph node classification.
While these methods achieve a better performance compared to shallow methods
(weber2019anti, ), most of them are designed for undirected graphs.
Accordingly, the direction information of the graph cannot be fully exploited.
DGCN (ma2019spectral, ) extends graph convolution from undirected graphs to
directed graphs by introducing the concepts of 1-st neighbors and 2-nd
neighbors to compute Lapalacian matrix. MagNet (zhang2021magnet, ) adopts
Hermitian magnetic Laplacian to decouple the direction of the graph. These
methods are explicitly represented for directed graphs in a transductive way,
but do not fully consider the distinction between in-degree and out-degree
information.

In financial risk control, the distinction between in-degree and out-degree
information is very important. For example, the information transmitted from
bank account A to account B and B to A are significantly different. In
addition, although the transfer from A to B is one-way, the transfer
relationship implies a two-way social relationship between A and B. Therefore,
the graph network for financial risk control should explicitly distinguish in-
degree and out-degree information, and adequately aggregate their information
in a balanced manner. However, GAT has a strong directinal bias as shown in
Fig. 1, where the out-degree weights are far less important than the in-degree
ones. This limits the performance on financial risk control tasks.

To tackle this problem, we propose a directed graph attention network (called
DGAT), which calculates the in-degree and out-degree information separately
during the feature aggregation process. To better distinguish the in/out-
degree represetation on directed graph, we further propose a dual embedding of
directed attention network (referred to as DEDGAT) to sovle the directional
bias. Specifially, a graph node is divided to in-degree and out-degree nodes
respectively and then graph representation is sepratedly computed for in/out-
degree nodes. The proposed models achieve SOTA performance on three financial
risk control datasets by reducing directional bias verified by GNNExplainer
(ying2019gnnexplainer, ).

##  2\. Proposed Models

As shown in Fig. 2(a), the undirected GAT can only aggregate in-degree
information, resulting in insufficient out-degree information. This leads to
the problem of directional bias as shown in Fig. 1. To explicitly exploit
directed information of graph nodes, we propose a directed graph attention
network (DGAT) to calculate both in-degree and out-degree attentions,
respectively, as shown in Fig. 2(b). To further distinguish in-degree and out-
degree informaton, we propose a dual embedding in terms of DGAT (as referred
to DEDGAT) to extract discriminative features for different directions of
graph nodes, as shown in Fig. 2(c).

###  2.1. Directed Graph Attention Networks (DGAT)

(a) GAT

(b) DGAT

(c) DEDGAT

Figure 2. Different Aggregation processes. (a) GAT only aggregate in-degree
features in one layer. (b) DGAT utilize both in-degree and out-degree
features. (c) DEDGAT seperates a node to in-degree (triangles) and out-degree
(squares) nodes and computes two different embeddings for in- and out-degree
feature represntations, respectively.

Given the representation $h_{i}^{(l)}$ of the node $i$ at the layer $l$, the
representation of its neighbors $j_{1},j_{2},\cdots$ is
$h_{j_{1}}^{(l)},h_{j_{2}}^{(l)},\cdots$, and the linear transformation before
propagation is given by $\hat{h}_{i}^{(l)}=W^{(l)}h_{i}^{(l)}$ and
$\tilde{h}_{i}^{(l)}=U^{(l)}h_{i}^{(l)}$ for in-degree and out-degree
respectively. Then, two map functions
$\mathcal{F}:\hat{h}\in\mathbb{R}^{2d}\rightarrow\hat{e}\in\mathbb{R}^{1}$ and
$\mathcal{G}:\tilde{h}\in\mathbb{R}^{2d}\rightarrow\tilde{e}\in\mathbb{R}^{1}$
are used to calculate the attention weights of the neighbors as follows,

(1) $\displaystyle\hat{e}_{i,j}^{(l)}$
$\displaystyle=\sigma((a_{t}^{(l)})^{T}[\hat{h}_{i}^{(l)}\|\hat{h}_{j}^{(l)}]),$
(2) $\displaystyle\tilde{e}_{i,j}^{(l)}$
$\displaystyle=\sigma((a_{s}^{(l)})^{T}[\tilde{h}_{i}^{(l)}\|\tilde{h}_{j}^{(l)}]),$

where $\|$ represents the concatenation operator, $\sigma$ represents the
softmax function, and $a_{t}^{(l)}$ and $a_{s}^{(l)}$ are the parameters of
the inner and outer neighbor attention mechanisms. After softmaxing the
attention weights of neighbors, the representation of node $i$ at level $l+1$
is given by Eq. 3, where $\mathcal{T}(i)$ and $\mathcal{S}(i)$ represent the
set of inner and outer neighbors, respectively.

$\displaystyle h_{i}^{(l+1)}=$
$\displaystyle\hat{h}_{i}^{\text{'(l)}}+\tilde{h}_{i}^{\text{'(l)}}$
$\displaystyle=$
$\displaystyle\sum_{j\in\mathcal{T}(i)}\hat{e}_{i,j}^{(l)}\hat{h}_{j}^{(l)}+\sum_{j\in\mathcal{S}(i)}\tilde{e}_{i,j}^{(l)}\tilde{h}_{j}^{(l)}$
(3) $\displaystyle=$
$\displaystyle\sum_{j\in\mathcal{T}(i)}\hat{e}_{i,j}^{(l)}W^{(l)}h_{j}^{(l)}+\sum_{j\in\mathcal{S}(i)}\tilde{e}_{i,j}^{(l)}U^{(l)}h_{j}^{(l)}.$

###  2.2. Dual Embedding of DGAT (DEDGAT)

DGAT adequately adopts in-degree and out-degree information. Nevertheless, the
directivity of feature is still not taken into account. We argue that single
feature cannot fully reflect the difference between input and output
information. Therefore, we propose a directed attention with dual embedding
model DEDGAT. As shown in Fig. 2(c) , each node consists of two
representations $t_{i}^{(l)}$ and $s_{i}^{(l)}$ to extract node features of
in-degree and out-degree, respectively. Before computing attention, there are
two linear transformation defined as $\hat{t}_{i}^{(l)}=w^{(l)}t_{i}^{(l)}$
and $\hat{s}{(l)}=u^{(l)}s_{i}^{(l)}$ for $\quad i=1,2,\cdots,n$ where
$w^{(l)}$ and $u^{(l)}$ are the weights of linear layer. Second, the attention
weights of in-degree and out-degree are calculated based on Eq.(4-5), where
$a_{t}^{(l)}$ and $a_{s}^{(l)}$ are the parameters of the inner and outer
neighbor attention mechanisms.

(4) $\displaystyle\hat{e}_{i,j}^{(l)}$
$\displaystyle=\sigma((a_{t}^{(l)})^{T}[\hat{t}_{i}^{(l)}||\hat{s}_{j}^{(l)}]),$
(5) $\displaystyle\tilde{e}_{i,j}^{(l)}$
$\displaystyle=\sigma((a_{s}^{(l)})^{T}[\hat{s}_{i}^{(l)}||\hat{t}_{j}^{(l)}]).$

Then, the two branches of the in-degree and out-degree attentions are
aggregated respectively to generate two representations for the next layer. On
the last layer $l$, the in-degree representation $t_{i}^{(l)}$ and the out-
degree representation $s_{i}^{(l)}$ are added to generate the representation
of node $i$. The added features are used to predict the network output.

###  2.3. Node Classification

We use two DEDGAT layer following a linear layer to conduct node
classification on three financial graph datasets. Given the embedding
generated by the last DEDGAT layer $L$, the prediction for node classification
is given by:

(6) $\displaystyle\hat{y}_{i,c}=\text{softmax}_{c}(Vh_{i}^{(L)}),$

where $V$ denote the final linear layer.

##  3\. Experiments

###  3.1. Datasets

We evaluate our approach on three directed graph datasets. Among them, the
Elliptic dataset(weber2019anti, ) is to detect money laundering from
transaction data, the Ant dataset is to detect financial fraud from transfer
records, the DGraphFin dataset (huang2022dgraph, ) is to detect financial
fraud from social networks. The Ant dataset is self-collected and others are
open-sourced. Each dataset contains two categories, 0 represents normal nodes
and 1 represents abnormal nodes. Optimizing node classification tasks on these
datasets can help us detect financial risk. Table. 1 shows the statistics of
three datasets.

Table 1. Financial Risk Control Datasets - | Elliptic | Ant | DGraphFin
---|---|---|---
Nodes | 203,769 | 1,296,486 | 3,700,550
Edges | 234,355 | 1,785,844 | 4,300,999
Labeled Nodes | 203,769 | 1,296,486 | 1,225,601
Features | 93 | 59 | 17
Classes | 2 | 2 | 2

###  3.2. Baselines and Parameter Settings

We select GAT(velivckovic2017graph, ), GCN(kipf2016semi, ),
GraphSage(hamilton2017inductive, ), DGCN(ma2019spectral, ),
MagNet(zhang2021magnet, ) methods for comparison. Among them, GAT, GCN and
GraphSage are basic graph models, DGCN, MagNet are latest graph models
designed for directed graphs.

To compare openly and transparently, we provide Tab. 2 which shows the network
parameters of each GNN model used in our experiments.

Table 2. Numbers of Network Parameters | Elliptic | Ant | DGraphFin
---|---|---|---
GCN | 98,306 | 63,490 | 20,482
GAT | 113,696 | 78,880 | 35,872
SAGE | 98,818 | 64,002 | 20,094
DGCN | 620,82 | 57,730 | 52,354
MagNet | 134,917 | 121,861 | 89,842
DGAT | 57,892 | 40,484 | 89,160
DEDGAT | 114,756 | 79,940 | 88,200

###  3.3. Implementations

We use AGL (aglpaper, ), a scalable, fault-tolerance and integrated system,
with fully-functional training and inference for GNNs to train our networks.
The AGL system consists of three parts, i.e., GraphFlat, GraphTrainer,
GraphInfer, respectively. GraphFlat is an efficient and distributed generator
based on message passing to generate K-hop neighborhoods that contains
information complete subgraphs of each targeted nodes. GraphTrainer combines
pipeline, pruning, and edge-partition modules to alleviate the overhead on I/O
and optimize the floating point calculations during the training of GNN
models. GraphInfer is a distributed inference module that splits K layer GNN
models into K slices, and applies the message passing K times based on
MapReduce. The three computing logics above are built on the distributed file
system, CPU cluster, and MapReduce computing framework. Our AGL system,
implemented on mature infrastructures, can finish the training of a 2-layer
graph attention network on a directed graph with billions of nodes and hundred
billions of edges in 14 hours.

###  3.4. Classification performance

Table. 3 shows the experimental results of each method on the node
classification task. Our DEDGAT effectively improves the classification
performance, achieves the best results on all three datasets.

Table 3. Node classification performance(AUC) | Elliptic | Ant | DGraphFin
---|---|---|---
GCN | 91.09±0.28 | 85.18±0.57 | 70.78±0.23
GAT | 95.06±0.45 | 97.43±0.24 | 76.24±0.81
SAGE | 95.79±0.33 | 97.49±0.19 | 77.61±0.18
DGCN | 94.48±0.60 | 94.71±0.85 | 79.58±0.15
MagNet | 92.84±1.01 | 91.95±0.20 | 79.71±0.27
DGAT | 95.79±0.43 | 97.92±0.31 | 80.72±0.11
DEDGAT | 96.18±0.23 | 98.16±0.09 | 81.37±0.06

(a) GAT

(b) DGAT

(c) DEDGAT

Figure 3. Comparison of interpretation of target node 7102 by GNNExplainer.
GAT ignores four out-degree edges $($7102 $\to$ 30934$)$, $($ 7102 $\to$ 30933
$)$, $($ 30933 $\to$ 1576$)$ and $($ 30933 $\to$ 35572$)$, DGAT resorts to two
out-degree edges $($7102 $\to$ 30934$)$ and $($ 7102 $\to$ 30933 $)$, and
DEDGAT leverages all four edges which are ignored by GAT.

###  3.5. Visualization results

When GAT (velivckovic2017graph, ) is applied to undirected graphs, the nodes
only aggregate information from the in-degree direction. In a directed graph,
edge ${E_{ij}}$ represent the out-degree edge of node i and the in-degree edge
of node j. GAT implicitly uses out-degree information based on the directional
symmetry. In the directional context of digraph, GAT-like methods which
implicitly fuse out-degree information cannot generalize well to the
recognition scene of directed graphs. However, Our method explicitly separates
the out-degree and in-degree information for calculation, and introduces dual
embedding in DEDGAT to emphasize the idea of separation. The purpose of
discrete modeling is to alleviate the in-degree bias that exists in GAT.

GNNExplainer (ying2019gnnexplainer, ) is a graph interpreter, it outputs the
importance score of each edge in graph model. To verify the validity of the
directionality we introduced, we use GNNExplainer to visualize the aggregation
process of three methods : GAT, DGAT and DEDGAT. As Fig. 3 shows (Darker edges
mean higher importance): GAT underutilizes the four out-degree edges $($7102
$\to$ 30934$)$, $($ 7102 $\to$ 30933 $)$, $($ 30933 $\to$ 1576$)$ and $($
30933 $\to$ 35572$)$; In contrast, our DGAT resorts to two out-degree edges
$($7102 $\to$ 30934$)$ and $($ 7102 $\to$ 30933 $)$ ; DEDGAT steps futher and
leverages all four edges which ignored by GAT.

We traverse the Ant dataset to provide global statistics result(Fig. 1) for
${S_{in}}$ and ${S_{out}}$,where ${S_{in}}$ and ${S_{out}}$ are defined as
follows:

(7)
$\begin{gathered}{S_{in}}=\left\\{{{w_{ij}}\in\left.S\right|k\left(i\right)>k\left(j\right)}\right\\},\hfill\\\
{S_{out}}=\left\\{{{w_{ij}}\in\left.S\right|k\left(i\right)<k\left(j\right)}\right\\},\hfill\\\
\end{gathered}$

where $k\left(i\right)$ represents the number of hops from node i to the
central risk node. S is defined as
$S=\left\\{{\left.{{w_{ij}}}\right|I\left({i,j}\right)=1}\right\\}$, where
${w_{ij}}$ is from GNNExplainer, i.e.,
${w_{ij}}=GNNExplr\left({{E_{ij}}}\right),{E_{ij}}\in G\left({E,V}\right)$.
Here, $I\left({i,j}\right)$ is the indicative function

(8)
$I\left({i,j}\right)=\left\\{{\begin{array}[]{*{20}{c}}{1,{A_{ij}}=1\wedge{A_{ji}}=1},\\\
{0,{A_{ij}}=0\vee{A_{ji}}=0},\end{array}}\right.$

where A is the adjacency matrix of the directed graph G.

(a) GAT

(b) DGAT

(c) DEDGAT

Figure 4. The proportion of in-degree and out-degree edge weights when their
importance scores are larger than 0.5. The proportion of the in-degree edge
weights of GAT is 1.73 times than that of the out-degree ones, while the
proportion of the two directional edge weights in the proposed DGAT and DEDGAT
is almost the same.

As shown in Fig. 1, GAT relies excessively on the in-degree information, but
out-degree edges are under exploited; Benefiting from DGAT and DEDGAT’s
separate calculation of in-degree and out-degree, both in-edges and out-edges
are assigned approximate weights. We set threshold $\lambda$ to 0.5 and count
those in-degree and out-degree edges whose weights are larger than the
threshold(these edges are important in the graph convolution). Statistics
results are shown in Fig. 4: GAT emerges a sharp in-degree information bias.
But DGAT and DEGAT keep the in-degree and out-degree information on a balance
level.

##  4\. Conclusion

In this paper, we have proposed a directed graph attention network (referred
to as DGAT) to explicitly calculate in-degree and out-degree representation of
a directed graph to tackle the problem of directional bias, such as the weight
important distribution shown in Fig. 2(a) for graph attention network (GAT)
(velivckovic2017graph, ). In order to better distinguish the in-degree and
out-degree node representation of the directed graph, we split each node into
in-degree and out-degree nodes. A dual embedding of DGAT model (named as
DEDGAT) is further proposed to separately calculate the node representations
of the incoming and outgoing degrees, respectively. Both DGAT and DEDGAT can
effectively solve the problem of the directional bias of GAT. Also, DEDGAT
achieves the best performance on the three financial risk control datasets.
Visualization results by GNNExplainer (ying2019gnnexplainer, ) demonstrate the
importance of directionality for financial risk control tasks.

## References

  * (1) Hamilton, William L and Ying, Rex and Leskovec, Jure.  Inductive representation learning on large graphs..  Proceedings of the 31st International Conference on Neural Information Processing Systems, 1025–1035, 2017.

  * (2) Yang, Bo.  Construction of logistics financial security risk ontology model based on risk association and machine learning..  Safety Science. Vol123, pages 104437, 2020.
  * (3) Ma, Xiaomeng and Lv, Shuliang.  Financial credit risk prediction in internet finance driven by machine learning..  Neural Computing and Applications. Vol 31, pages 8359–8367, 2019.
  * (4) Aziz, Saqib and Dowling, Michael.  Machine learning and AI for risk management..  Disrupting finance. pages 33–50, 2019.
  * (5) Li, Nan and Liang, Xun and Li, Xinli and Wang, Chao and Wu, Desheng Dash.  Network environment and financial risk using machine learning and sentiment analysis..  Human and Ecological Risk Assessment. vol15, pages 227–252, 2009.
  * (6) Perozzi, Bryan and Al-Rfou, Rami and Skiena, Steven.  Deepwalk: Online learning of social representations..  Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701–710, 2014.
  * (7) Grover, Aditya and Leskovec, Jure.  node2vec: Scalable feature learning for networks..  Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * (8) Tang, Jian and Qu, Meng and Wang, Mingzhe and Zhang, Ming and Yan, Jun and Mei, Qiaozhu.  Line: Large-scale information network embedding..  Proceedings of the 24th international conference on world wide web, pages 1067–1077, 2015.
  * (9) Weber, Mark and Domeniconi, Giacomo and Chen, Jie and Weidele, Daniel Karl I and Bellei, Claudio and Robinson, Tom and Leiserson, Charles E.  Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics..  arXiv preprint arXiv:1908.02591, 2019.
  * (10) Ma, Yi and Hao, Jianye and Yang, Yaodong and Li, Han and Jin, Junqi and Chen, Guangyong.  Spectral-based graph convolutional network for directed graphs..  arXiv preprint arXiv:1907.08990, 2019.
  * (11) Zhang, Xitong and He, Yixuan and Brugnone, Nathan and Perlmutter, Michael and Hirn, Matthew.  Magnet: A neural network for directed graphs..  Advances in Neural Information Processing Systems, vol 34, pages 27003–27015, 2021.
  * (12) Veličković, Petar and Cucurull, Guillem and Casanova, Arantxa and Romero, Adriana and Lio, Pietro and Bengio, Yoshua.  Graph attention networks..  arXiv preprint arXiv:1710.10903, 2017.
  * (13) Kipf, Thomas N and Welling, Max.  Semi-supervised classification with graph convolutional networks..  arXiv preprint arXiv:1609.02907, 2016.
  * (14) Ying, Zhitao and Bourgeois, Dylan and You, Jiaxuan and Zitnik, Marinka and Leskovec, Jure.  Gnnexplainer: Generating explanations for graph neural networks..  Advances in neural information processing systems, vol 32, 2019.
  * (15) Huang, Xuanwen and Yang, Yang and Wang, Yang and Wang, Chunping and Zhang, Zhisheng and Xu, Jiarong and Chen, Lei and Vazirgiannis, Michalis.  DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection..  Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * (16) Zhang, Dalong and Huang, Xin and Liu, Ziqi and Zhou, Jun and Hu, Zhiyang and Song, Xianzheng and Ge, Zhibang and Wang, Lin and Zhang, Zhiqiang and Qi, Yuan.  AGL: A Scalable System for Industrial-Purpose Graph Machine Learning..  Proc. VLDB Endow. pages 3125–3137, 2020.


# Revisiting the Adversarial Robustness-Accuracy Tradeoff in
Robot Learning

Mathias Lechner1, Alexander Amini2, Daniela Rus2, Thomas A. Henzinger1
1Institute of Science and Technology Austria (IST Austria) {mlechner,
tah}@ist.ac.at)2Massachusetts Institute of Technology, Computer Science and
Artificial Intelligence Laboratory, Cambridge, MA, 02139 USA {amini,
<EMAIL_ADDRESS>
###### Abstract

Adversarial training (i.e., training on adversarially perturbed input data) is
a well-studied method for making neural networks robust to potential
adversarial attacks during inference. However, the improved robustness does
not come for free, but rather is accompanied with a decrease in overall model
accuracy and performance. Recent work have shown that, in practical robot
learning applications, the effects of adversarial training do not pose a fair
trade-off but inflict a net loss when measured in holistic robot performance.
This work revisits the robustness-accuracy trade-off in robot learning through
systematically analyzing if recent advances in robust training methods and
theory in conjunction with adversarial robot learning, are capable of making
adversarial training suitable for real-world robot applications. We evaluate
in a wide variety of robot learning tasks ranging from autonomous driving in a
high-fidelity environment amenable to sim-to-real deployment, to mobile robot
gesture recognition. Our results demonstrate that, while these techniques make
incremental improvements on the trade-off on a relative scale, the negative
side-effects caused by adversarial training still outweigh the improvements by
an order of magnitude. We conclude that more substantial advances in robust
learning methods are necessary before they can benefit robot learning tasks in
practice.

##  I Introduction

Adversarial attacks are well-studied vulnerabilities of deep neural networks
[1, 2]. These norm-bounded input perturbations make the network completely
change its decision compared to the unaltered input and can have catastrophic
impact in practical robotics applications. Furthermore, the critical issue of
adversarial attacks is that the perturbed inputs are barely distinguishable
from the original input by humans. At the same time, the attacker can make the
networks produce arbitrarily incorrect outputs and decision. For instance,
Figure 1 shows such adversarial attacks norm-bounded by various magnitudes on
images. Most realistic-sized compute vision networks can be fooled by
perturbations that change each pixel by a maximum of 4% (i.e., a
$l_{\infty}$-norm less or equal to 8) while being barely noticeable by humans.

The vulnerability of neural networks to adversarial attacks raises critical
concerns about their deployment in robotic tasks where physical damage and
safety are at risk. Notably, the existence of adversarial examples raises the
following question:

> _If a human’s decision is not affected by adversarial attacks, then why
> should we trust a robot that can be fooled by such attacks?_

From a pure safety point of view, we may require image processing networks
deployed in safety-critical applications to be robust regarding manipulating
each pixel by up a few percent as such a change would not impair the decision
of a human controlling the system.

Original sample $x$$\left\lVert x-\tilde{x}\right\rVert_{\infty}\leq
1$$\left\lVert x-\tilde{x}\right\rVert_{\infty}\leq 2$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 4$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 8$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 16$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 32$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 64$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 128$$\left\lVert
x-\tilde{x}\right\rVert_{\infty}\leq 256$ Figure 1: Visualization of a sample
image $x$ from the visual gesture recognition dataset [3] and corresponding
images $\tilde{x}$ attacked by FGSM [2] and different perturbation norms.
Attacks with a norm less or equal of 8 are barely visually recognizable.

Robust learning aims to tackle the problem by training networks that are
immune to adversarial or other types of attacks [4, 5, 6, 7, 8, 9, 10, 11].
One of the most dominant approaches for training robust models is adversarial
training which adds adversarial perturbations to the training data online
during and throughout the learning procedure [4, 8]. Adversarial training
methods improve the test-time robustness on adversarial examples at the
critical cost of lower nominal accuracy [12, 13, 14]. For instance, the
advanced adversarial training algorithm of [15], which won the NeurIPS 2018
Adversarial Vision Challenge, yielded a robust network with an accuracy of 89%
on the CIFAR-10 dataset. In contrast, standard training algorithms can easily
produce non-robust networks with an accuracy above 96% on this dataset [16].
This dilemma of choosing between an accurate but vulnerable and a robust but
less accurate model is known as the robustness-accuracy trade-off [14, 15,
12].

Recent work [3] has investigated this trade-off specifically in the context of
robot learning applications where both accuracy and robustness are critical as
the system is ultimately deployed into physical, safety-critical environments.
The authors observed that this trade-off is not fair trade but poses a net
loss when evaluating the robots’ overall performance and concluded that
adversarial training is not ready for robot learning. However, recent work has
shown that multiple factors contribute to the reduction in accuracy of robot
learning methods. In particular, these works underline that larger models are
necessary for robustly fitting the training data. Moreover, they emphasize
that a more careful selection of the neural network architecture and
hyperparameters is needed when replacing standard training with adversarial
training methods. However, there remains a critically important and open
question on if these recent advances are sufficient to quell the costs of
adversarial training for robotics.

In this work, we assess whether the conclusion of [3] that adversarial
training is not ready for robot learning remains true or is challenged by
these recent advances to the field. In particular, we evaluate if
overparametrized models, vision transformers, and smooth curvature activation
functions can provide acceptable accuracy and robustness on robot learning and
autonomous driving tasks.

Our results provide a systematic framework for evaluating the effect of these
algorithms in the context of robot learning and demonstrate that while the
methods from the literature make single digits improvements on the robustness-
accuracy Pareto front, i.e., improving both accuracy and robustness, the
negative side-effects of adversarial training methods still outweigh these
advances by an order of magnitude. Nonetheless, we observed that type of
learned representation has a great impact in determining whether a good
accuracy and robustness can be achieved. In particular, our results indicate
that further research on large-scale pre-trained and attention-based
representations are promising paths for learning robust and accurate models in
practice.

We summarize our contributions as:

  * •

We assess several theoretical and experimental advances on the robustness-
accuracy trade-off for their suitability in real-world robot learning tasks.

  * •

We perform an extensive evaluation by training a total of 435 networks with
adversarial training using various attack budgets, and evaluating across
multiple robot learning tasks including mobile robot driving and gesture
recognition.

  * •

We demonstrate strong empirical evidence that, while robustness can be
improved by the methods from literature, the negative side-effects of
adversarial training still outweigh the improvements by an order of magnitude.

The remainder of this paper is structured as follows. In section II, we
recapitulate robustness of neural networks, adversarial training, and the
robustness-accuracy trade-off. In section III, we describe related work on
improving the robustness of neural networks and avoiding the reduced clean
accuracy of adversarial training. Finally, in section IV, we experimentally
evaluate these improvements on our robot learning tasks, ranging in complexity
.

##  II Background and Related Work

A neural network is a function $f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$
parameterized by $\theta$. In supervised learning, the training objective is
to fit the function to a given dataset in the form of
$\\{(x_{1},y_{1}),\dots(x_{n},y_{n})\\}$ assumed to be i.i.d. sampled from a
probability distribution over $\mathcal{X}\times\mathcal{Y}$. This fitting
process is done via empirical risk minimization (ERM) that minimizes

$\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(f_{\theta}(x_{i}),y_{i})$ (1)

via stochastic gradient descent. The differentiable loss function
$\mathcal{L}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$ characterizes
how well the network’s prediction $f_{\theta}(x_{i})$ matches the ground truth
label $y_{i}$.

An adversarial attack is a sample $(x_{i},y_{i})$ from the data distribution
and a corresponding attack vector $\mu$ with
$\left\lVert\mu\right\rVert\leq\varepsilon$ such that $f(x_{i})\neq
f(x_{i}+\mu)$ with $\varepsilon$ being a thresholds. As illustrated in Figure
1, for image data small thresholds $\delta$ are usually not recognizable or
appear as noise for human observers. It has been shown that most neural
networks, irrespective of network types, input domains, or learning setting,
are susceptible to adversarial attacks [2, 17, 18, 19, 20, 21, 22].

Typical norms used in adversarial attacks are the $\ell_{1}$, $\ell_{2}$, and
the $\ell_{\infty}$ norm. In this work, we focus on the $\ell_{\infty}$ norm.
A network is robust on a given sample if no such attack $\mu$ exists within a
threshold $\varepsilon$. The robust accuracy is the standard metric for
measuring the robustness of a network aggregated over an entire dataset
$\\{(x_{1},y_{1}),\dots(x_{n},y_{n})\\}$ by counting the ratio of correctly
classified samples that are also robust.

In practice, deciding whether a network is robust for a sample is an NP-hard
problem [23, 24, 25] and, therefore, cannot be computed for typically sized
networks in a reasonable time. Instead, the robustness of networks is often
studied with respect to empirical gradient and black-box-based attack methods.
The fast gradient sign method (FGSM) [2] computes an attack by

$\mu=\varepsilon\
\text{sign}\Big{(}\frac{\partial\mathcal{L}(f_{\theta}(x_{i}),y_{i})}{\partial
x_{i}}\Big{)}.$ (2)

Despite its simplicity, adversarial training often uses the FGSM method due to
its speed. The iterative fast gradient sign method (I-FGSM) [26] is a more
sophisticated generalization of the FSGM. It computes an attack iteratively in
$k$ steps starting from $\mu_{0}=\mathbf{0}$ and updating it by

$\mu_{i}=\frac{\varepsilon}{k}\
\text{sign}\Big{(}\frac{\partial\mathcal{L}(f_{\theta}(x_{i}+\mu_{i-1}),y_{i})}{\partial
x_{i}}\Big{)}.$ (3)

DeepFool [27], the C&W method [28], and projected gradient descent [8] are
other common iterative attack methods that are used for evaluating robustness
but are too computationally expensive to incorporate in adversarial training.
DeepFool [27] linearizes the network in each iteration of updating $\mu_{i}$.
Projected gradient descent [8] applies unconstrained gradient descent but
divides each $\mu_{i}$ by its norm and multiplies the results with
$\varepsilon$ to project it back into the given threshold. The C&W method [28]
avoids such projection by parametrizing the attack vector $\mu$ by another
variable and a transformation that already normalizes the attack to stay
within a given threshold. It has been experimentally shown that any network of
non-trivial size is, at least in parts, vulnerable to such attacks [8].

Robust learning methods aim to train networks that are robust [4, 5, 6, 7, 8,
9, 10]. One of the most common robust learning methods is adversarial training
which changes the standard ERM objective to the min-max objective

$\frac{1}{n}\sum_{i=1}^{n}\max_{\mu:\left\lVert\mu\right\rVert\leq\varepsilon}\mathcal{L}(f_{\theta}(x_{i}+\mu),y_{i}),$
(4)

where $\varepsilon>0$ is some attack budget controlling how much each input
can be perturbed. Due to the computation overhead by this training objective,
fast attack generating methods are typically used for computing the $\max$ in
Equation 4, e.g., the FGSM or I-FGSM.

Alternative approaches to adversarial training make minor modifications to the
objective term in Equation 4. For instance, the TRADES algorithm [15] replaces
the label $y_{i}$ in Equation 4 with the network’s prediction of the original
input, i.e., $f_{\theta}(x_{i})$, and optimizes a joint objective of the
standard ERM term and the robustness term. The approach of [29] removes the
overhead imposed by the maximization step in Equation 4 by pre-computing $\mu$
in the previous gradient descent step. Although, such pre-computed $\mu$ can
become inaccurate, i.e., stale, [29] showed that it improves robustness in
practice.

The major limitation of adversarial training methods is that they negatively
affect the network’s standard accuracy (or other performance metrics). For
example, medium-sized networks achieve an accuracy of 96% on the CIFAR-10
dataset when trained with standard ERM [16]. However, in [15] the best
performing network trained with the TRADES algorithm could only achieve a
standard accuracy of 89% on this dataset. This phenomenon of an antagonistic
relation between accuracy and robustness was first studied in [14] and is
known as the accuracy-robustness trade-off. The trade-off was studied in the
context of robot learning in [3] by investigating whether the gained
robustness is worth the reduction in nominal accuracy in real-world robotic
tasks. The authors observed that the adversarially trained networks resulted
in a worse robot performance than by using a network trained with the standard
way.

The concept of adversarial training and the min-max objective of robust
learning has been adopted to other task-specific types of specifications, such
as safety. For example, [3] has introduced safety-domain training by replacing
the norm-bounded neighborhoods of labeled samples with arbitrary sets and
corresponding labels, i.e., a min-max training objective over labeled sets.
Some modifications of the min-max objective have been studied in feedback
systems with closed-loop safety and stability specifications. For instance,
[30, 31, 32] propose to learn a safety certificate via a learner-verifier
framework where the maximization step is replaced by a verification module
that provides formal guarantees on the certificate.

##  III Methods

In this section we describe three directions from the literature that point to
paths of how to improve robustness without sacrificing standard accuracy.

###  III-A Smooth activations and bag of tricks

Recent work suggests that the common ReLU activation function, i.e.,
$\max\\{0,x\\}$, is not well suited for adversarial training methods [33].
Instead, the authors observed that activation functions with smooth curvatures
provide better robustness at roughly the same standard accuracy. Specifically,
the sigmoid-weighted linear unit (SiLU) activation function [34], i.e.,
$x\cdot\frac{1}{1+\exp(-x)}$, was highlighted as having a smooth second
derivative and observed to improve robustness compared to alternative
activations. We note that the SiLU activation was concurrently proposed as
swish activation function in [35].

The work of [36] investigated how hyperparameters of the learning process
affect adversarial training compared to standard ERM. For example, the authors
experiment with learning rate schedules, early stopping, and batch size, among
other settings. The authors observed that adversarial training benefits from a
higher weight decay factor than standard training. Moreover, the authors
confirmed that a smooth activation function improves robustness over the ReLU
activation.

Summer(in-distribution)Winter(in-distribution)Autumn(out-of-
distribution)Night(out-of-distribution) Figure 2: Test conditions of our
closed-loop driving experiment using a data-driven simulation environment
[37]. The training data are collected in summer and winter conditions
(separated from the testing data).

###  III-B Robustness requires overparametrization

Theoretical contributions to the robustness-accuracy tradeoff recently
discovered that overparametrization is necessary for smoothly fitting the
training data. While empirical results already suggested that the accuracy of
larger models suffers less from adversarial training than for small models,
the critical insight is that such large models are necessary. In particular,
the authors proved that for a dataset of $n$ samples with $d$-dimensional
features, a model with $n$ parameters can fit the training samples but cannot
smoothly interpolate between them. Moreover, the authors show that a model
needs at least $nd$ parameters to fit the training data and interpolate them
smoothly. The authors also demonstrated that contemporary models for standard
datasets do not contain enough parameters with respect to their proven
results.

###  III-C Vision transformers are more robust than CNNs

The vision transformer (ViT) [38] is a powerful machine learning architecture
that represents an image as a sequence of patches and processes this sequence
using a self-attention mechanism [39]. Detailed experimental comparisons
between vision transformer and convolutional neural networks suggest that ViTs
are naturally more robust with respect to object occlusions and distributions
shifts [40]. Concurrent work on comparing ViTs to CNNs with respect to
adversarial attacks has found that vision transformers seem to be naturally
more robust to adversarial attacks as well.

All advances on the robustness-accuracy tradeoff discussed above are either
theoretical or were evaluated on static image classification tasks. Moreover,
the methods are typically evaluated on research datasets such as CIFAR and
ImageNet. While these datasets allow studying machine learning models’ general
performance, they significantly differ from real-world robot learning tasks.
For example, the CIFAR datasets consist of very low-resolution images, i.e.,
32-by-32 pixel, whereas robotic vision processing systems handle images with
much higher resolution, e.g., 256-by-256 pixels in [3]. Although the samples
of the ImageNet dataset have a realistic image resolution, typical robot
learning datasets consist of multiple orders of magnitude fewer samples than
the ImageNet dataset. Moreover, experiments on research datasets often report
static test metrics, whereas learned robotic controllers are deployed in a
closed-loop on a robot.

The next section evaluates the methods described above on multiple real-world
robot learning tasks, including open-loop training and closed-loop evaluation
on an autonomous driving task.

##  IV Experiments

In this section we study the advances in adversarial training methods on three
robot learning tasks.

###  IV-A End-to-end driving

Our first experiment considers an autonomous driving task. In particular, a
network is trained to predict the curvature of the road ahead of a car from
images received at a camera that is mounted on top of the vehicle. The
training data is collected by a human driver who maneuvers the car around a
test track. The networks are then trained on collected data using supervised
learning. Finally, we deploy the networks in a closed-loop autonomous driving
simulator. We use the VISTA simulation environment [37] for this purpose.

Model | Adversarial training budget $\varepsilon$
---|---
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CNN | 0 | 0 | 0 | 1 | 2 | 6 | 10 | 14 | 12
CNN-Large | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 4 | 19
ViT | 0 | 0 | 0 | 0 | 0 | 1 | 9 | 4 | 4
CNN (SiLU) | 0 | 0 | 2 | 16 | 25 | 42 | 46 | 71 | 77
CNN (wd+) | 0 | 0 | 0 | 0 | 7 | 13 | 27 | 34 | 31
TABLE I: Number of crashes out of 100 simulation runs using data collected in summer (in-training distribution data).  Model | Adversarial training budget $\varepsilon$
---|---
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CNN | 0 | 0 | 4 | 9 | 12 | 35 | 41 | 40 | 51
CNN-Large | 0 | 0 | 0 | 2 | 2 | 5 | 9 | 13 | 20
ViT | 0 | 2 | 1 | 0 | 4 | 2 | 6 | 5 | 5
CNN (SiLU) | 3 | 5 | 19 | 38 | 42 | 59 | 57 | 78 | 82
CNN (wd+) | 0 | 0 | 0 | 2 | 9 | 20 | 41 | 51 | 55
TABLE II: Number of crashes out of 100 simulation runs using data collected in winter (in-training distribution data).  Model | Adversarial training budget $\varepsilon$
---|---
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CNN | 1 | 30 | 40 | 61 | 61 | 64 | 91 | 89 | 85
CNN-Large | 1 | 14 | 42 | 55 | 67 | 83 | 81 | 80 | 86
ViT | 0 | 43 | 65 | 67 | 74 | 91 | 69 | 84 | 78
CNN (SiLU) | 1 | 45 | 60 | 86 | 87 | 91 | 80 | 89 | 91
CNN (wd+) | 7 | 32 | 59 | 62 | 59 | 70 | 88 | 89 | 87
TABLE III: Number of crashes out of 100 simulation runs using data collected in autumn (out-of-training distribution).  Model | Adversarial training budget $\varepsilon$
---|---
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CNN | 36 | 78 | 78 | 90 | 93 | 86 | 94 | 97 | 87
CNN-Large | 52 | 90 | 88 | 92 | 87 | 93 | 95 | 94 | 94
ViT | 68 | 69 | 80 | 73 | 81 | 79 | 88 | 91 | 64
CNN (SiLU) | 71 | 79 | 79 | 95 | 90 | 93 | 95 | 94 | 94
CNN (wd+) | 62 | 60 | 68 | 69 | 91 | 95 | 93 | 92 | 95
TABLE IV: Number of crashes out of 100 simulation runs using data collected
during the night (out-of-training distribution).

We compare the performance of a baseline CNN with four variations. First, we
compare with an enlarged variant of the baseline CNN to validate the necessity
of overparametrization for robustness empirically. Next, we equip the baseline
with the smoother SiLU activation and increase the weight decay. Finally, we
test a vision transformer model. The baseline model consists of 440k, the
enlarged CNN of 7.7M, and the tested vision transformer of 2.0M trainable
parameters. The input of all architectures are 160-by-48 RGB images that are
normalized per-image to have zero mean and unit standard deviation. The
architecture details of the two convolutional networks are listed in Appendix
A in Table VIII and Table IX. Our vision transforms splits the input image
into non-overlapping patches of 16-by-12 pixels, uses a latent dimension of
256, with 4 attention heads, 384 feed-forward dimension, and 4 layers in
total. For the training we use the Adam optimizer [41] with a learning rate of
0.0003 and a batch size of 64. The weight decay is set to $10^{-5}$, except
for the wd+ variant, which is trained with a decay factor of $5\cdot 10^{-5}$.
We train all networks for a total of 900,000 steps. We train all models with
standard and adversarial training with increasing attack budget
($\varepsilon=0,1,\dots 8$) and I-FGSM as attack methods.

For each model and attack budget pair, we run a total of 400 simulations,
split into 200 in-training distribution, and 200 out-of-training distribution
condition runs. The in-training distribution data were collected in summer and
winter and were separated from the training data, i.e., there is no overlap
between the training data and the evaluation data. The out-of-training
distribution data were collected in autumn and during the night, with no such
condition present in the training data. The four conditions are visualized in
Figure 2. As an evaluation metric, we report the number of crashes during the
simulation, i.e., when the vehicle leaves the road.

Table I and Table II show the crashes during the summer and winter simulations
respectively. The best values are highlighted in bold. The results show that
an overparametrized model and a vision transformer indeed provide better
performance at a larger adversarial training budget. An increased weight decay
improved the performance only at lower attack budget training, while the
networks with SiLU activation performed worse in the closed-loop tests. At
larger attack budgets, no model could drive the car safely, while most models
learned by standard ERM could drive all 200 runs flawlessly.

The out-of-training distribution simulation results for autumn and night
conditions are shown in Table III and Table IV. We observe that adversarial
training significantly hurt the out-of-distribution performance of all models,
i.e., especially in the autumn data. A video demonstration of the simulated
runs is available at https://youtu.be/TQKP7l9PfNo. In summary, the best
driving performance across all four tested conditions was observed with
networks trained with standard ERM.

Model | Adversarial | Validation | Validation accuracy under I-FGSM [26] attack | Test
---|---|---|---|---
| training budget | accuracy | $\varepsilon=1$ | $\varepsilon=2$ | $\varepsilon=4$ | $\varepsilon=8$ | accuracy
ResNet50 | $\varepsilon=0$ |  99.5% $\pm$ 0.0 | 85.3% $\pm$ 2.1 | 49.5% $\pm$ 6.6 | 8.7% $\pm$ 2.0 | 2.3% $\pm$ 1.9 |  93.5% $\pm$ 3.8
$\varepsilon=1$ |  99.0% $\pm$ 0.4 |  98.1% $\pm$ 0.4 |  93.6% $\pm$ 1.0 | 72.4% $\pm$ 3.4 | 18.9% $\pm$ 3.2 |  86.1% $\pm$ 3.1
$\varepsilon=2$ |  98.7% $\pm$ 0.6 |  97.7% $\pm$ 0.7 |  96.0% $\pm$ 0.5 | 89.5% $\pm$ 1.9 | 55.3% $\pm$ 3.1 | 76.0% $\pm$ 2.5
$\varepsilon=4$ |  96.8% $\pm$ 0.8 |  95.4% $\pm$ 1.3 |  93.9% $\pm$ 0.9 |  90.7% $\pm$ 2.0 | 77.0% $\pm$ 3.8 | 68.6% $\pm$ 3.2
$\varepsilon=8$ | 70.3% $\pm$ 5.1 | 68.9% $\pm$ 2.4 | 67.9% $\pm$ 4.2 | 66.3% $\pm$ 5.0 | 60.7% $\pm$ 2.7 | 51.8% $\pm$ 6.6
ResNet101 | $\varepsilon=0$ |  99.2% $\pm$ 0.4 |  90.1% $\pm$ 2.9 | 54.1% $\pm$ 7.6 | 10.9% $\pm$ 2.3 | 3.8% $\pm$ 1.3 |  85.9% $\pm$ 7.3
$\varepsilon=1$ |  99.5% $\pm$ 0.0 |  97.8% $\pm$ 0.5 |  93.6% $\pm$ 1.2 | 74.4% $\pm$ 4.0 | 20.7% $\pm$ 2.2 |  82.9% $\pm$ 4.4
$\varepsilon=2$ |  98.7% $\pm$ 1.1 |  96.2% $\pm$ 1.4 |  94.7% $\pm$ 1.0 | 87.3% $\pm$ 3.0 | 54.8% $\pm$ 1.4 | 76.1% $\pm$ 2.6
$\varepsilon=4$ | 44.2% $\pm$ 0.4 | 44.4% $\pm$ 0.5 | 43.9% $\pm$ 0.5 | 44.2% $\pm$ 0.7 | 44.4% $\pm$ 0.2 | 41.5% $\pm$ 1.2
$\varepsilon=8$ | 44.4% $\pm$ 0.5 | 44.1% $\pm$ 0.6 | 43.6% $\pm$ 0.5 | 43.9% $\pm$ 0.5 | 43.9% $\pm$ 0.5 | 41.7% $\pm$ 0.0
ResNet152 | $\varepsilon=0$ |  98.8% $\pm$ 0.8 | 86.8% $\pm$ 4.0 | 57.3% $\pm$ 4.8 | 14.8% $\pm$ 6.2 | 4.4% $\pm$ 4.4 |  87.2% $\pm$ 4.2
$\varepsilon=1$ |  99.0% $\pm$ 0.4 |  97.6% $\pm$ 0.4 |  95.2% $\pm$ 1.4 | 83.0% $\pm$ 1.4 | 29.6% $\pm$ 5.7 |  82.1% $\pm$ 7.0
$\varepsilon=2$ |  98.1% $\pm$ 0.8 |  97.6% $\pm$ 0.6 |  95.5% $\pm$ 0.7 |  90.9% $\pm$ 1.6 | 66.7% $\pm$ 4.2 | 75.8% $\pm$ 4.4
$\varepsilon=4$ | 60.5% $\pm$ 18.9 | 60.3% $\pm$ 18.4 | 60.0% $\pm$ 18.0 | 58.4% $\pm$ 16.4 | 52.6% $\pm$ 9.8 | 50.8% $\pm$ 12.4
$\varepsilon=8$ | 44.4% $\pm$ 0.4 | 44.3% $\pm$ 0.2 | 44.4% $\pm$ 0.4 | 43.9% $\pm$ 0.4 | 44.5% $\pm$ 0.4 | 41.7% $\pm$ 0.0
TABLE V: Accuracies on the visual gesture recognition dataset of various pre-trained residual neural networks fine-tuned via standard and adversarial training. Validation accuracies greater than 90% and test accuracies greater than 80% are highlighted in bold. The different column show the standard accuracy and accuracy-under-attack with respect to the I-FGSM method and different attack budgets. Model | Adversarial | Validation | Validation accuracy under I-FGSM [26] attack | Test
---|---|---|---|---
| training budget | accuracy | $\varepsilon=1$ | $\varepsilon=2$ | $\varepsilon=4$ | $\varepsilon=8$ | accuracy
ViT-Small/16 | $\varepsilon=0$ |  99.0% $\pm$ 0.3 | 84.5% $\pm$ 4.7 | 50.4% $\pm$ 6.7 | 15.6% $\pm$ 8.6 | 0.8% $\pm$ 1.5 |  94.1% $\pm$ 2.1
$\varepsilon=1$ |  99.5% $\pm$ 0.0 |  97.7% $\pm$ 1.2 |  91.8% $\pm$ 2.0 | 70.3% $\pm$ 1.6 | 21.3% $\pm$ 5.6 |  84.6% $\pm$ 6.0
$\varepsilon=2$ |  99.4% $\pm$ 0.2 |  98.9% $\pm$ 0.2 |  97.6% $\pm$ 0.9 | 89.2% $\pm$ 2.3 | 48.0% $\pm$ 4.8 |  83.3% $\pm$ 4.7
$\varepsilon=4$ |  99.3% $\pm$ 0.5 |  98.8% $\pm$ 0.4 |  97.0% $\pm$ 0.8 |  94.4% $\pm$ 1.4 | 72.3% $\pm$ 5.0 | 75.2% $\pm$ 3.1
$\varepsilon=8$ | 44.1% $\pm$ 0.4 | 44.0% $\pm$ 0.4 | 43.6% $\pm$ 0.2 | 44.1% $\pm$ 0.4 | 43.9% $\pm$ 0.4 | 41.1% $\pm$ 1.9
ViT-Base/16 | $\varepsilon=0$ |  98.4% $\pm$ 1.1 | 89.4% $\pm$ 4.7 | 67.0% $\pm$ 6.2 | 28.7% $\pm$ 5.2 | 11.7% $\pm$ 4.5 |  82.3% $\pm$ 5.7
$\varepsilon=1$ |  98.9% $\pm$ 0.6 |  99.2% $\pm$ 0.5 |  92.1% $\pm$ 5.9 | 72.8% $\pm$ 11.8 | 37.9% $\pm$ 8.4 | 79.4% $\pm$ 1.7
$\varepsilon=2$ |  99.5% $\pm$ 0.0 |  99.5% $\pm$ 0.3 |  97.9% $\pm$ 0.9 |  92.5% $\pm$ 1.0 | 61.4% $\pm$ 3.8 |  84.5% $\pm$ 4.6
$\varepsilon=4$ |  97.4% $\pm$ 2.3 |  94.5% $\pm$ 4.8 |  93.5% $\pm$ 5.4 | 86.7% $\pm$ 7.5 | 67.1% $\pm$ 11.7 | 76.0% $\pm$ 1.9
$\varepsilon=8$ | 53.4% $\pm$ 7.3 | 53.2% $\pm$ 7.6 | 52.2% $\pm$ 7.2 | 50.0% $\pm$ 5.1 | 48.0% $\pm$ 3.7 | 54.6% $\pm$ 10.2
ViT-Large/16 | $\varepsilon=0$ |  98.7% $\pm$ 0.5 |  92.1% $\pm$ 1.6 | 74.7% $\pm$ 4.3 | 44.1% $\pm$ 11.0 | 20.8% $\pm$ 11.0 |  85.9% $\pm$ 2.5
$\varepsilon=1$ |  99.3% $\pm$ 0.2 |  98.1% $\pm$ 0.0 |  95.0% $\pm$ 0.7 | 77.6% $\pm$ 0.7 | 35.3% $\pm$ 1.2 |  89.9% $\pm$ 3.6
$\varepsilon=2$ |  99.5% $\pm$ 0.0 |  98.6% $\pm$ 0.5 |  97.6% $\pm$ 1.0 |  92.3% $\pm$ 2.4 | 67.1% $\pm$ 9.9 |  89.6% $\pm$ 3.2
$\varepsilon=4$ |  98.6% $\pm$ 1.0 |  97.6% $\pm$ 1.0 |  96.0% $\pm$ 1.6 |  93.3% $\pm$ 2.8 | 77.4% $\pm$ 6.6 | 71.0% $\pm$ 15.9
$\varepsilon=8$ | 64.6% $\pm$ 22.9 | 64.4% $\pm$ 22.3 | 63.8% $\pm$ 23.0 | 63.0% $\pm$ 21.3 | 58.0% $\pm$ 17.5 | 47.2% $\pm$ 9.8
TABLE VI: Accuracies on the visual gesture recognition task of various pre-
trained vision transformer fine-tuned via standard and adversarial training.
Validation accuracies greater than 90% and test accuracies greater than 80%
are highlighted in bold. The different column show the standard accuracy and
accuracy-under-attack with respect to the I-FGSM method and different attack
budgets.

###  IV-B Visual gesture recognition

Our second experiment concerns training an image classifier that controls the
operating modes of a mobile robot as reported in [3]. The dataset consists of
2029 sample 256-by-256 pixel images corresponding to three classes, i.e., idle
(905 samples), enable (552 samples), and disable (572 samples), which are
split into a training and a validation set with a 90%:10% ratio. The
experiments on the physical robot in [3] suggest that that a validation
accuracy of above 90% is necessary for an acceptable robot performance. Due to
the small size of the dataset, we resort to transfer learning of a pre-trained
classifier using the big-transfer (BiT) fine-tuning protocol of initializing
the output layer with all zeros and training all layers [42].

In this experiment, we test the theoretical necessity of overparametrization
in practice. We train networks of different sizes using adversarial training
with increasing attack budget ($\varepsilon\in\\{1,2,4,8\\}$) and report the
clean and robust validation accuracy under I-FGSM attacks with various attack
budgets.

As a proxy for the real-world test accuracy, we collect a new datasets
comprising of 190 idle samples, 129 enable samples, and 140 disable samples.
An example visualizing how the two data sources differ is shown in Figure 3.
Our motivation for collecting this dataset is that the original validation set
of [3] is temporally and spatially correlated with the training data, i.e.,
collected at the same time and location. Consequently, the original validation
set may not capture the actual real-world accuracy of our trained models due
to robust overfitting [43]. We use the clean accuracy of the new set as our
test metric to estimate the real-world performance.

Original data [3]Test data Figure 3: Comparison of sample images of the
original visual gesture recognition dataset [3] and the newly collected data.
The new data was collected with a different time and location than the
original data, making it capture real-world performance more accurately.

For increasing the size of the model, we test a ResNet50 (24M), ResNet101
(43M), and ResNet152 (58M) with the number of trainable parameters reported in
parenthesis [44]. We also evaluate the vision transformer models ViT-Small
(22M), ViT-Base (86M), and ViT-Large (304M) that process the images in the
form of 16-by-16 pixel patches [38]. For the training we use the Adam
optimizer [41] with a learning rate of 0.00005 and a batch size of 64, except
for the ResNet152 where a batch size of 32 is used due to out-of-memory
errors. We repeat each training run with 5 random seeds and report the mean
and standard deviation.

The results in Table V and Table VI show that adversarial training with a
small attack budget, e.g., $\varepsilon\in\\{1,2\\}$ can have positive effects
on the robustness while maintaining a good clean accuracy compared the
networks trained with standard training. Interestingly, the models trained
with such small attack budget express a non-trivial robust accuracy even for
larger attack budgets under I-FGSM attacks. However, the performance drop
significantly to an unacceptable performance when the networks are trained
with a larger attack budget of $\varepsilon=8$. A surprising observation is
that our results contradict that larger models provide a better robustness in
practice. In particular, out of all adversarially trained convolutional
networks, the ResNet50 trained with $\varepsilon=1$ provides the best test
accuracy. Moreover, the ResNet50 also achieves the best test and validation
accuracy under stronger training attack budgets, e.g.
$\varepsilon\in\\{4,8\\}$. In contrast, the vision vision transformer
architecture seems to benefit from a larger model. The largest ViT provides
the best test accuracy under adversarial training with an attack budget of
$\varepsilon\in\\{1,2,\\}$ out of all tested models. These results suggest
that the type of learned representation, e.g. convolutional vs attention-
based, plays role whether overparameterization improves robustness or not. In
both convolutional and attention-based models, we observed a double descent
phenomenon with respect to the model size [45]. Particularly, both the mid-
size ResNet101 and ViT-Base/16 performed worse than their smaller and larger
counterparts.

Safety |  | Validation accuracy
---|---|---
level |  | Width 1 | Width 2 | Width 3 | Width 4
0 | Baseline |  83.2% $\pm$ 0.8 |  84.7% $\pm$ 1.6 |  83.9% $\pm$ 1.9 |  85.2% $\pm$ 0.8
ELU | 73.3% $\pm$ 1.5 | 72.5% $\pm$ 3.3 | 73.3% $\pm$ 0.8 | 71.3% $\pm$ 1.3
wd+ |  82.5% $\pm$ 2.0 |  84.0% $\pm$ 2.1 |  85.7% $\pm$ 1.3 |  85.7% $\pm$ 1.2
1 | Baseline | 75.1% $\pm$ 2.6 | 78.6% $\pm$ 3.7 | 77.4% $\pm$ 2.1 | 78.7% $\pm$ 3.4
ELU | 53.1% $\pm$ 0.6 | 53.5% $\pm$ 0.4 | 52.9% $\pm$ 0.6 | 52.3% $\pm$ 0.8
wd+ | 74.2% $\pm$ 3.4 | 75.0% $\pm$ 1.8 | 65.9% $\pm$ 10.7 | 67.4% $\pm$ 12.0
2 | Baseline | 76.3% $\pm$ 3.1 | 76.8% $\pm$ 4.9 | 76.1% $\pm$ 2.8 | 78.5% $\pm$ 3.2
ELU | 53.6% $\pm$ 0.3 | 53.1% $\pm$ 0.3 | 53.2% $\pm$ 0.4 | 52.9% $\pm$ 0.6
wd+ | 72.9% $\pm$ 3.3 | 75.5% $\pm$ 2.1 | 68.4% $\pm$ 8.6 | 70.7% $\pm$ 10.0
3 | Baseline | 51.8% $\pm$ 0.9 | 52.8% $\pm$ 0.5 | 53.3% $\pm$ 0.1 | 53.9% $\pm$ 0.3
ELU | 53.2% $\pm$ 0.8 | 53.8% $\pm$ 0.5 | 53.1% $\pm$ 0.1 | 53.2% $\pm$ 0.4
wd+ | 51.4% $\pm$ 1.1 | 52.8% $\pm$ 0.7 | 52.8% $\pm$ 0.6 | 53.4% $\pm$ 0.4
TABLE VII: Validation accuracy on the robot follow dataset [3] of
1D-convolutional NNs with various hyperparameters and trained with standard
and safety-domain training. Values greater than 80% are highlighted in bold.
Safety level 0 corresponds to standard training, while the network trained
with safety level 1 and above provide formal safety guarantees of never
crashing the robot into an obstacle. The columns show networks with different
widening factor and consist of an increasing amount of learnable parameters,
i.e., width 1 (360k), width 2 (1.4M), width 3 (3.2M), and width 4 (5.7M).

###  IV-C Certified safety-domain training

Adversarial training methods do not ensure robustness but provide only
empirical improvements over common attack methods. Certified training methods
such as the interval bound propagation [46] can learn networks with formal
robustness or safety guarantees. In this experiment, we study the safety-
domain training of LiDAR-based mobile robot navigation controller [3]. The
objective of the learned controller is to map 541-dimensional laser range
scans to 7 possible categories, i.e., stay, straight forward, left forward,
right forward, straight backward, left backward, and right backward. The
dataset consists of 2705 training and 570 validation samples uniformly
distributed across the seven classes. Using safety-domain training, we want to
ensure that the robot never crashes into an object in front of it. This is
achieved by training an abstract interpretation representation of the network
to never output a forward locomotion class in case the LiDAR input indicates
an obstacle. There are four safety levels with different strictness of what
accounts for an obstacle, e.g., several consecutive rays or just a single ray,
defined in [3]. Safety level 0 corresponds to standard training, while safety
level 3 represents the strictest level.

We test the overparametrization, increased weight decay (from $0$ to
$10^{-5}$), and smooth activation function methods on this task. As a
baseline, we use the 1D-CNN from [3], which is comprised of 360k parameters.
Our overparametrized models increase the width of the network to obtain CNNs
with 1.4M, 3.2M, and 5.7M parameters respectively. We use the exponential
linear unit (ELU) activation function [47] to represent a smooth activation
due to the non-monotonically of SiLU being incompatible with the used abstract
interpretation domains. We train all models with the Adam optimizer [41] with
a learning rate of 0.0001 and a batch size of 64. The safety level 0 models
are trained for 20 epochs, while the networks trained using safety-domain
training for 2000 epochs. The network architectures are shown in Appendix A in
Table X.

We report the validation accuracy as an evaluation metric. The experiments on
the physical robot in [3] suggest that a validation accuracy above 80% is
necessary to achieve an acceptable real-world performance. Note that all
models, except those trained with safety level 0, provide some form of formal
safety guarantees. Therefore, this experiment studies how much validation
accuracy is traded for the ensured safety. We repeat each training run with 5
random seeds and report the mean and standard deviation.

The result in Table VII shows that safety-domain training benefits from an
increased number of parameters (width). However, the improvement over the
baseline is rather incremental and accounts only for a few percentage. In
contrast, the accuracy reduction caused by the safety-domain training is
several times more significant, e.g., around 10%, and no network trained with
safety-domain training exceeds the threshold of 80% accuracy. The networks
with smooth activation function and increased weight decay performed worse
than the baseline when using safety-domain training. This suggests that
certified training methods such as safety-domain training may require
different hyperparameters and learning settings than adversarial training.

##  V Conclusion

Adversarial training (i.e., training on adversarially perturbed input data) is
a well-studied method for making neural networks robust to potential
adversarial attacks during inference. However, the improved robustness does
not come for free but rather is accompanied by a decrease in overall model
accuracy and performance. Recent work has shown that, in practical robot
learning applications, the effects of adversarial training do not pose a fair
trade-off but inflict a net loss when measured in holistic robot performance.
This work revisits the robustness-accuracy trade-off in robot learning by
systematically analyzing if recent advances in robust training methods and
theory in conjunction with adversarial robot learning can make adversarial
training suitable for real-world robot applications. We evaluate a wide
variety of robot learning tasks ranging from autonomous driving in a high-
fidelity environment amenable to sim-to-real deployment, to mobile robot
gesture recognition. Our results demonstrate that, while these techniques make
incremental improvements on the trade-off on a relative scale, the negative
side-effects caused by adversarial training still outweigh the improvements by
an order of magnitude. We conclude that more substantial advances in robust
learning methods are necessary before they can benefit robot learning tasks in
practice.

## Acknowledgment

This work was supported in parts by the ERC-2020-AdG 101020093, National
Science Foundation (NSF), and JP Morgan Graduate Fellowships. We thank
Christoph Lampert for inspiring this work.

## References

  * [1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [2] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [3] M. Lechner, R. M. Hasani, R. Grosu, D. Rus, and T. A. Henzinger, “Adversarial training is not ready for robot learning,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [4] A. Wald, “Statistical decision functions which minimize the maximum risk,” _Annals of Mathematics_ , pp. 265–280, 1945.
  * [5] P. J. Huber, “Robust estimation of a location parameter,” _The Annals of Mathematical Statistics_ , vol. 1, no. 35, 1964.
  * [6] C. G. Atkeson and S. Schaal, “Robot learning from demonstration,” in _International Conference on Machine Learning (ICML)_ , 1997.
  * [7] H. Xu, C. Caramanis, and S. Mannor, “Robustness and regularization of support vector machines.” _The Journal of Machine Learning Research (JMLR)_ , vol. 10, 2009.
  * [8] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [9] C. Song, K. He, L. Wang, and J. E. Hopcroft, “Improving the generalization of adversarial training with domain adaptation,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [10] B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” in _International Conference on Machine Learning (ICML)_ , 2012\.
  * [11] N. Konstantinov and C. Lampert, “Robust learning from untrusted sources,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [12] A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang, “Adversarial training can hurt generalization,” _arXiv preprint arXiv:1906.06032_ , 2019\.
  * [13] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel, “Efficient neural network robustness certification with general activation functions,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2018\.
  * [14] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [15] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [16] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _British Machine Vision Conference (BMCV_ , 2016.
  * [17] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer, “Adversarial patch,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [18] J. Uesato, B. O’Donoghue, A. v. d. Oord, and P. Kohli, “Adversarial risk and the dangers of evaluating against weak attacks,” in _International Conference on Machine Learning (ICML)_ , 2018.
  * [19] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” _arXiv preprint arXiv:1802.00420_ , 2018.
  * [20] L. Schönherr, K. Kohls, S. Zeiler, T. Holz, and D. Kolossa, “Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding,” _arXiv preprint arXiv:1808.05665_ , 2018.
  * [21] M. Giacobbe, T. A. Henzinger, and M. Lechner, “How many bits does it take to quantize your neural network?” in _International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS)_ , 2020.
  * [22] Y. Bai, J. Mei, A. L. Yuille, and C. Xie, “Are transformers more robust than cnns?” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [23] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient smt solver for verifying deep neural networks,” in _International Conference on Computer Aided Verification (CAV)_ , 2017.
  * [24] M. Sälzer and M. Lange, “Reachability is np-complete even for the simplest neural networks,” in _International Conference on Reachability Problems_ , 2021.
  * [25] T. A. Henzinger, M. Lechner, and Đ. Žikelić, “Scalable verification of quantized neural networks,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [26] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * [27] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [28] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy_ , 2017.
  * [29] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [30] Y. Chang, N. Roohi, and S. Gao, “Neural lyapunov control,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [31] M. Lechner, D. Zikelic, K. Chatterjee, and T. A. Henzinger, “Infinite time horizon safety of bayesian neural networks,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [32] M. Lechner, Đ. Žikelić, K. Chatterjee, and T. A. Henzinger, “Stability verification in stochastic control systems via neural network supermartingales,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , 2022.
  * [33] V. Singla, S. Singla, S. Feizi, and D. Jacobs, “Low curvature activations reduce overfitting in adversarial training,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [34] S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” _Neural Networks_ , vol. 107, pp. 3–11, 2018.
  * [35] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” _arXiv preprint arXiv:1710.05941_ , 2017.
  * [36] T. Pang, X. Yang, Y. Dong, H. Su, and J. Zhu, “Bag of tricks for adversarial training,” in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [37] A. Amini, T.-H. Wang, I. Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus, “Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles,” _arXiv preprint arXiv:2111.12083_ , 2021.
  * [38] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [40] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Intriguing properties of vision transformers,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [42] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, “Big transfer (bit): General visual representation learning,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [43] L. Rice, E. Wong, and Z. Kolter, “Overfitting in adversarially robust deep learning,” in _International Conference on Machine Learning (ICML)_ , 2020\.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [45] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [46] S. Gowal, K. D. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli, “Scalable verified training for provably robust image classification,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [47] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” _arXiv preprint arXiv:1511.07289_ , 2015.

##  Appendix A Network architecture details

Here we describe the details of the neural network architectures used in our
experiments.

Layer | Parameter
---|---
Conv2D | F=32, K=5, S=2, ReLU
Conv2D | F=64, K=5, S=1, ReLU
Conv2D | F=96, K=3, S=2, ReLU
Conv2D | F=128, K=3, S=1, ReLU
GlobalAveragePool2D |
Fully-connected | 1000 units, ReLU
Fully-connected | 100 units, ReLU
Fully-connected | 1 unit
TABLE VIII: Convolutional neural network baseline architecture for our autonomous driving experiment (440k parameters). F refers to the number of filters, K to the kernel size, and S to the stride. Layer | Parameter
---|---
Conv2D | F=32, K=5, S=2
BatchNorm2D | ReLU (post BN)
Conv2D | F=128, K=5, S=1
BatchNorm2D | ReLU (post BN)
Conv2D | F=256, K=3, S=2
BatchNorm2D | ReLU (post BN)
Conv2D | F=512, K=3, S=1
BatchNorm2D | ReLU (post BN)
Conv2D | F=1024, K=3, S=1
GlobalAveragePool2D |
Fully-connected | 1024 units, ReLU
Fully-connected | 256 units, ReLU
Fully-connected | 1 unit
TABLE IX: Enlarged neural network architecture for our autonomous driving experiment (7.7M parameters). F refers to the number of filters, K to the kernel size, and S to the stride.  Layer | Parameter
---|---
Conv1D | F=w*32, K=5, S=1, ReLU
Conv1D | F=w*96, K=5, S=2, ReLU
Conv1D | F=w*96, K=5, S=2, ReLU
Conv1D | F=w*96, K=5, S=2, ReLU
Conv1D | F=w*96, K=5, S=2, ReLU
Conv1D | F=w*96, K=5, S=2, ReLU
Flatten |
Fully-connected | w*128 units, ReLU
Fully-connected 7 | softmax
TABLE X: Network architecture of the 1D-CNN trained with safety-domain
training (F= number of filters, K = kernel size, S = stride). w is the
widening factor.