$\kappa(x):=s(x)/t(x)={x^{2p}}/{\exp(2rx)}$ is non-increasing and convex for
all $x$ sufficiently large. Since $\nu_{k+1}\leq\nu_{k}+T$ for all $k$
sufficiently large, by setting ${\sf B}:=T$ and $\beta:=1-2c_{\vartheta}r>0$
in Lemma 3.10, it follows for all $x>0$ that

${\sf
B}[s^{\prime}(x)-\kappa(x)t^{\prime}(x)]=0-\frac{Tx^{2p}}{\exp(2rx)}\cdot\frac{c_{\vartheta}(2rx-2p)}{T}\frac{\exp(2rx)}{x^{2p+1}}=-2c_{\vartheta}r+2p/x\geq-1+\beta.$

Moreover, we have $\sum_{k=0}^{\infty}{1}/{s(\nu_{k})}=\infty$. Thus, invoking
Lemma 3.10, we obtain $\limsup_{k\to\infty}y_{k}\cdot g(\nu_{k})^{2}<\infty$
and mimicking the discussion in Step 3.3, we can conclude

${\limsup}_{j\to\infty}\|F^{\lambda}_{\mathrm{nat}}(x^{j})\|^{2}\cdot
g({\gamma_{j}})^{2}\leq\lambda\,{\limsup}_{j\to\infty}\|F^{\lambda}_{\mathrm{nor}}(z^{j})\|^{2}\cdot
g({\gamma_{j}})^{2}<\infty$

and $\limsup_{j\to\infty}|\psi(x^{j})-\psi^{*}|\cdot
g({\gamma_{j}})^{2}<\infty$. In order to derive the rate for
$\\{x^{k}\\}_{k}$, we apply Eq. 46 with $\vartheta=\frac{1}{2}$. Using the
bound for $\\{\varepsilon_{n}\\}_{n}$ and $\limsup_{k\to\infty}y_{k}\cdot
g(\nu_{k})^{2}<\infty$, this yields
${\sum}_{k=n}^{\infty}d_{k}\leq{10\sqrt{2}\hat{c}}{\xi^{-1}}\sqrt{y_{n}}+\varepsilon_{n}=\mathcal{O}(g(\nu_{k})^{-1})$.
Mimicking discussions in Step 3.4, we can conclude
$\limsup_{j\to\infty}\|x^{j}-x^{*}\|\cdot g({\gamma_{j}})<\infty$.

This finishes the proof of 3.5. ∎

We continue with several concluding remarks. Our general proof flow and the
proof of 3.5 differ from the derivations in [92, 9, 30] and are closer to
classical KL-based analyses in optimization. In a nutshell, our overall
analysis is built upon the core components “approximate descent $\to$
asymptotic global convergence $\to$ KL-based strong limit convergence” and
allows to effectively incorporate time window techniques, step size dynamics,
and error estimates.

ApproximateDescent PropertyLemma 2.13Asymptotic Global
Convergence$-$$F^{\lambda}_{\mathrm{nat}}(\bm{x}^{k})\to 0$,
$\psi(\bm{x}^{k})\to\bm{\psi}^{*}$2.15 & 2.16Strong Limit
Convergence$-$$\bm{x}^{k}\to\bm{x}^{*}$$-$Rates for
$\\{\psi(\bm{x}^{k})\\}_{k}$, $\\{\bm{x}^{k}\\}_{k}$3.5 & Corollary 3.6Time
window /time scalesDiminishingstep sizesError estimatesLemma 2.8 & 3.9KL
property

To illustrate this statement, we may consider the basic error condition
$\sum_{k=0}^{\infty}g(\nu_{k})^{2}\bm{s}_{k}^{2}=\sum_{k=0}^{\infty}\beta_{m_{k}}^{2}\bm{s}_{k}^{2}<\infty$
that is used in the definition of the master event $\mathcal{U}$. This bound
plays a central role in our analysis and allows to establish global
convergence and to obtain fast rates of convergence in 3.5. In [92], Tadić
works with a weaker error condition of the form
$\limsup_{k\to\infty}g(\nu_{k})\bm{s}_{k}<\infty$. However, in order to ensure
convergence, the accumulated error terms $\\{\bm{u}_{k}\\}_{k}$ still need to
be controlled in a suitable way. This ultimately affects the order of the
estimates and explains the different convergence results and rates in [92]
(cf. Remark 3.8). As Tadić mostly utilizes indirect (contradiction-based)
proof techniques that are more tailored to $\mathsf{SGD}$, it is not clear
whether a better control of $\\{\bm{s}_{k}\\}_{k}$ (as in our case) can be
effectively exploited or if extensions of his analysis to other algorithms and
the nonsmooth setting are possible.

By contrast, we believe that the proof framework and analysis techniques
presented in this work can be likely transferred to other stochastic
optimization methodologies and are applicable in a broader context. We plan to
investigate such a potential and more general stochastic KL-based analysis
framework in future work.

(a) news20(b) rcv1(c) epsilon

Convergence

Sparsity

Figure 2: Convergence results and sparsity information epochs for solving the
binary classification problem Eq. 57. (Averaged over $10$ runs).

##  4 Preliminary Numerical Experiments

In this section, we compare the proposed algorithm
$\mathsf{norM}\text{-}\mathsf{SGD}$ with $\mathsf{prox}\text{-}\mathsf{SGD}$
on two large-scale learning tasks. In the experiments, we strictly control the
number of stochastic gradient evaluations so that the overall computational
complexity of both algorithms is identical at each iteration. In particular,
both $\mathsf{norM}\text{-}\mathsf{SGD}$ and
$\mathsf{prox}\text{-}\mathsf{SGD}$ use the same type of stochastic oracle to
approximate gradient information. Our primary goal is to illustrate that
$\mathsf{norM}\text{-}\mathsf{SGD}$ does not perform worse than
$\mathsf{prox}\text{-}\mathsf{SGD}$ and has a similar practical behavior.

###  4.1 Nonconvex Binary Classification

We first consider a sparse nonconvex binary classification problem, [94, 67],
of the form:

$\min_{x\in\mathbb{R}^{d}}~{}\psi(x)=\frac{1}{N}\sum_{i=1}^{N}f_{i}(x)+\varphi(x):=\frac{1}{N}\sum_{i=1}^{N}[1-\tanh(b_{i}\cdot
a_{i}^{\top}x)]+\nu\|{x}\|_{1},$ (57)

where $a_{i}\in\mathbb{R}^{d}$ denotes the training sample, $b_{i}$ is the
associated binary label, and $\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$ is
the hyperbolic tangent function. In the numerical comparison, we use the
popular datasets111All datasets are available at
www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets news20 ($N=19\,996$,
$d=1\,355\,191$), rcv1 ($N=20\,242$, $d=47\,236$), and epsilon ($N=400\,000$,
$d=2\,000$) and the $\ell_{1}$-regularization parameter $\nu$ is set to $1/N$
for all datasets. At iteration $k$, we randomly pick indices from
$[N]:=\\{1,2,\dots,N\\}$ without replacement to form a subset
$S_{k}\subseteq[N]$. We then use $g^{k}:=\frac{1}{|S_{k}|}\sum_{i\in
S_{k}}\nabla f_{i}(x^{k})$ as standard stochastic approximation of $\nabla
f(x^{k})$.

Implementational details. In our test, we use the initial point
$x^{0}=\mathds{1}/d\in\mathbb{R}^{d}$ and the size of the mini-batches $S_{k}$
is set to $|S_{k}|=256$ for all $k\in\mathbb{N}$. We apply step sizes of the
form $\alpha_{k}=\alpha/({\sf L}+k)$ and choose $\lambda=\alpha$ in
$\mathsf{norM}\text{-}\mathsf{SGD}$, where $\alpha$ lies in the range
$[10^{0},10^{5}]$ and ${\sf L}$ is an approximate Lipschitz constant of
$\nabla f$. Specifically, based on the structure of $\nabla^{2}f$, we utilize
the estimate ${\sf L}=4\|A\|^{2}/(5N)$, where
$A=[a_{1}^{\top},\dots,a_{N}^{\top}]^{\top}$. For this example, the proximity
operator is the well-known shrinkage operator

$\mathrm{prox}_{\lambda\nu\|\cdot\|_{1}}(z)=\mathrm{sgn}(z)\odot\max\\{0,|z|-\lambda\nu\\},$

where all operations are understood componentwisely. Notice that
$\mathsf{prox}\text{-}\mathsf{SGD}$ computes
$\mathrm{prox}_{\alpha_{k}\nu\|\cdot\|_{1}}$ in each iteration, while we work
with $\mathrm{prox}_{\lambda\nu\|\cdot\|_{1}}$ in
$\mathsf{norM}\text{-}\mathsf{SGD}$.

In Figure 2, we first depict the number of epochs required to satisfy the
relative accuracy criterion “$\psi(x^{k})-\psi^{*}\leq
0.01\max\\{1,\psi^{*}\\}$” vs. different step size parameters
$\alpha\in[10^{0},10^{5}]$. The average performance (over $10$ independent
runs) is shown using a darker color and thicker line style. Here, one single
epoch corresponds to $N$ iterations of $\mathsf{prox}\text{-}\mathsf{SGD}$ or
$\mathsf{norM}\text{-}\mathsf{SGD}$ and the approximate optimal value
$\psi^{*}$ is obtained by running the deterministic proximal gradient method
starting from three different initial points until
$\|F_{\mathrm{nat}}^{1}(x^{k})\|<10^{-5}$. The results shown in the top row of
Figure 2 indicate that $\mathsf{norM}\text{-}\mathsf{SGD}$ is more robust with
respect to the choice of the parameter $\alpha$ and that it generally achieves
faster convergence on news20 and rcv1. In the bottom row of Figure 2, we plot
the sparsity level $100\%\cdot|\\{i:|x_{i}^{k}|\leq 10^{-8}\\}|/d$ of the
iterates $\\{x^{k}\\}_{k}$ vs. the number of epochs for
$\alpha\in\\{10^{2},10^{3},10^{4}\\}$ (again averaged over 10 runs). Overall,
$\mathsf{norM}\text{-}\mathsf{SGD}$ tends to recover sparser solutions than
$\mathsf{prox}\text{-}\mathsf{SGD}$ which reflects the different scalings of
the $\ell_{1}$-norm in the proximity operators.

(a) Training loss(b) Training error(c) Test error

ResNet-18

VGG-16

Figure 3: Numerical results for the model Eq. 58 on CIFAR-10 using ResNet-18
(top) and VGG-16 (bottom) architecture.

###  4.2 Deep Learning

In this subsection, we study the performance of
$\mathsf{norM}\text{-}\mathsf{SGD}$ and $\mathsf{prox}\text{-}\mathsf{SGD}$ on
a multi-class image classification task for the dataset CIFAR-10 [48] which
contains $10$ classes of images and each class contributes $6\,000$ images. We
split the dataset into $N_{\text{train}}=50\,000$ training samples and
$N_{\text{test}}=10\,000$ test samples. To classify the images, we apply the
standard ResNet-18 [43] and VGG-16 [89] architectures with the cross-entropy
loss function [97, 93] and elastic net regularizer [98]:

$\min_{x\in\mathbb{R}^{d}}-\frac{1}{N}\sum_{i=1}^{N}\log\left(\frac{\exp(\mathcal{T}(x,a_{i})^{\top}e_{b_{i}})}{\sum_{j=1}^{10}\exp(\mathcal{T}(x,a_{i})^{\top}e_{j})}\right)+\nu_{1}\|{x}\|_{1}+\nu_{2}\|{x}\|^{2},\quad\nu_{1},\nu_{2}>0.$
(58)

The training sample $(a_{i},b_{i})$ contains the color image
$a_{i}\in\mathbb{R}^{32\times 32\times 3}$ and the corresponding label
$b_{i}\in\\{1,\dots,10\\}$. The operator
$\mathcal{T}(x,\cdot):\mathbb{R}^{32\times 32\times 3}\mapsto\mathbb{R}^{10}$,
which maps an input image to a ten dimensional vector, represents the
architecture with weights $x$. We utilize the elastic net regularizer
$\varphi(x)=\nu_{1}\|{x}\|_{1}+\nu_{2}\|{x}\|^{2}$ with $\nu_{1}=10^{-6}$ and
$\nu_{2}=10^{-4}$.

Implementational details. We use step sizes with adaptive decay222We apply the
scheduler $\mathsf{ReduceLROnPlateau}$ (default setting) in PyTorch [75] with
the initial step size $\alpha=0.1$. in both algorithms and the parameter
$\lambda$ in $\mathsf{norM}\text{-}\mathsf{SGD}$ is set to $\lambda=10^{-2}$.
We train both ResNet-18 and VGG-16 for $100$ epochs with batch size $128$ and
run each algorithm $5$ times independently. The overall performance is shown
below in Figure 3. The proximity operator for the elastic net regularizer is
given by
$\mathrm{prox}_{\lambda\varphi}(z)=(1+{1}/{(1+2\lambda\nu_{2})})\mathrm{prox}_{\lambda\nu_{1}\|\cdot\|_{1}}(z)$.

Comparison. Though the deep learning task Eq. 58 does not fully fit the
structure of problem Eq. 1 (the operator $\mathcal{T}$ can contain nonsmooth
activation components), our results help to illustrate the numerical
performance of $\mathsf{norM}\text{-}\mathsf{SGD}$ and
$\mathsf{prox}\text{-}\mathsf{SGD}$ on this more challenging class of learning
problems. In Figure 3, besides the training and test error, we also report the
change of the training loss with respect to the number of epochs. Similar to
our previous experiment, $\mathsf{norM}\text{-}\mathsf{SGD}$ manages to
achieve a significantly lower training loss compared to
$\mathsf{prox}\text{-}\mathsf{SGD}$. In terms of training and test error,
$\mathsf{norM}\text{-}\mathsf{SGD}$ performs similar to
$\mathsf{prox}\text{-}\mathsf{SGD}$ achieving slightly lower errors on both
neural network architectures.

##  5 Conclusion

In this paper, we develop a novel normal map-based stochastic proximal
gradient method, $\mathsf{norM}\text{-}\mathsf{SGD}$, for nonconvex composite-
type optimization problems and analyze its convergence. Utilizing the
unbiasedness of the stochastic normal map steps, we show that the iterates
generated by $\mathsf{norM}\text{-}\mathsf{SGD}$ satisfy a time window-based
approximate descent property which allows to establish asymptotic global
convergence results and non-asymptotic complexity bounds under standard
assumptions. We then provide strong limit convergence and convergence rates
for the objective function values $\\{\psi(\bm{x}^{k})\\}_{k}$, the
stationarity measure
$\\{\|F^{\lambda}_{\mathrm{nat}}(\bm{x}^{k})\|^{2}\\}_{k}$, and the iterates
$\\{\bm{x}^{k}\\}_{k}$ under the KL inequality. The obtained rates depend on
the step size dynamics $\\{\alpha_{k}\\}_{k}$ and the KL exponent
$\bm{\theta}\in[0,1)$ of $\psi$. Specifically, in the popular case
$\alpha_{k}=\alpha/(\beta+k)^{\gamma}$, $\gamma\in(\frac{2}{3},1]$, our
results imply (trajectory-based) convergence of $\\{\bm{x}^{k}\\}_{k}$ with
asymptotic rates that are faster than related and existing convergence rates
for $\mathsf{SGD}$ and that can beat the complexity estimates. To the best of
our knowledge, $\mathsf{norM}\text{-}\mathsf{SGD}$ seems to be the first basic
stochastic proximal algorithm for nonconvex composite optimization with this
broad range of stochastic convergence properties. In addition, we believe that
the stochastic KL-based techniques studied in this work have general potential
and can be applied to other families of stochastic and simulation-type
algorithms.

## References

  * [1] P.-A. Absil, R. Mahony, and B. Andrews, Convergence of the iterates of descent methods for analytic cost functions, SIAM J. Optim., 16 (2005), pp. 531–547.
  * [2] Y. F. Atchadé, G. Fort, and E. Moulines, On perturbed proximal gradient algorithms, J. Mach. Learn. Res., 18 (2017), pp. 1–33.
  * [3] H. Attouch and J. Bolte, On the convergence of the proximal algorithm for nonsmooth functions involving analytic features, Math. Program., 116 (2009), pp. 5–16.
  * [4] H. Attouch, J. Bolte, P. Redont, and A. Soubeyran, Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality, Math. Oper. Res., 35 (2010), pp. 438–457.
  * [5] H. Attouch, J. Bolte, and B. F. Svaiter, Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods, Math. Program., 137 (2013), pp. 91–129.
  * [6] H. H. Bauschke, P. L. Combettes, et al., Convex analysis and monotone operator theory in Hilbert spaces, vol. 408, Springer, 2011.
  * [7] M. Benaïm, A dynamical system approach to stochastic approximations, SIAM J. Control Optim., 34 (1996), pp. 437–472.
  * [8] , Dynamics of stochastic approximation algorithms, in Seminaire de probabilites XXXIII, Springer, 1999, pp. 1–68.
  * [9] , On strict convergence of stochastic gradients.  arXiv preprint, arXiv:1610.03278v1, 2018.
  * [10] A. Benveniste, M. Métivier, and P. Priouret, Adaptive algorithms and stochastic approximations, vol. 22 of Applications of Mathematics (New York), Springer-Verlag, Berlin, 1990.  Translated from the French by Stephen S. Wilson.
  * [11] D. P. Bertsekas, Nonlinear programming, Athena Scientific Optimization and Computation Series, Athena Scientific, Belmont, MA, third ed., 2016.
  * [12] D. P. Bertsekas and J. N. Tsitsiklis, Gradient convergence in gradient methods with errors, SIAM J. Optim., 10 (2000), pp. 627–642.
  * [13] C. M. Bishop, Pattern recognition and machine learning, Information Science and Statistics, Springer, New York, 2006.
  * [14] J. Bolte, A. Daniilidis, and A. Lewis, The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems, SIAM J. Optim., 17 (2006), pp. 1205–1223.
  * [15] , A nonsmooth Morse-Sard theorem for subanalytic functions, J. Math. Anal. Appl., 321 (2006), pp. 729–740.
  * [16] J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota, Clarke subgradients of stratifiable functions, SIAM J. Optim., 18 (2007), pp. 556–572.
  * [17] J. Bolte, A. Daniilidis, O. Ley, and L. Mazet, Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity, Trans. Amer. Math. Soc., 362 (2010), pp. 3319–3363.
  * [18] J. Bolte, S. Sabach, and M. Teboulle, Proximal alternating linearized minimization for nonconvex and nonsmooth problems, Math. Program., 146 (2014), pp. 459–494.
  * [19] S. Bonettini, I. Loris, F. Porta, M. Prato, and S. Rebegoldi, On the convergence of a linesearch based proximal-gradient method for nonconvex optimization, Inverse Probl., 33 (2017), p. 055005.
  * [20] V. S. Borkar, Stochastic approximation: a dynamical systems viewpoint, vol. 48, Springer, 2009.
  * [21] L. Bottou, F. E. Curtis, and J. Nocedal, Optimization methods for large-scale machine learning, SIAM Review, 60 (2018), pp. 223–311.
  * [22] L. Breiman, Probability, vol. 7 of Classics in Applied Mathematics, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1992\.  Corrected reprint of the 1968 original.
  * [23] D. L. Burkholder, B. J. Davis, and R. F. Gundy, Integral inequalities for convex functions of operators on martingales, in Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability (Univ. California, Berkeley, Calif., 1970/1971), Vol. II: Probability theory, 1972, pp. 223–240.
  * [24] A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), pp. 120–145.
  * [25] E. Chouzenoux, J.-B. Fest, and A. Repetti, A Kurdyka-Lojasiewicz property for stochastic optimization algorithms in a non-convex setting.  arXiv preprint, arXiv:2302.06447v2, 2023.
  * [26] K. L. Chung, On a stochastic approximation method, Ann. Math. Statistics, 25 (1954), pp. 463–483.
  * [27] P. L. Combettes and J.-C. Pesquet, Proximal splitting methods in signal processing, in Fixed-point algorithms for inverse problems in science and engineering, vol. 49 of Springer Optim. Appl., Springer, New York, 2011, pp. 185–212.
  * [28] D. Davis and D. Drusvyatskiy, Stochastic model-based minimization of weakly convex functions, SIAM J. Optim., 29 (2019), pp. 207–239.
  * [29] D. Davis, D. Drusvyatskiy, S. Kakade, and J. D. Lee, Stochastic subgradient method converges on tame functions, Found. Comput. Math., 20 (2020), pp. 119–154.
  * [30] S. Dereich and S. Kassing, Convergence of stochastic gradient descent schemes for Łojasiewicz-landscapes, arXiv preprint arXiv:2102.09385, (2021).
  * [31] J. Duchi and Y. Singer, Efficient online and batch learning using forward backward splitting, J. Mach. Learn. Res., 10 (2009), pp. 2899–2934.
  * [32] J. C. Duchi and F. Ruan, Stochastic methods for composite and weakly convex optimization problems, SIAM J. Optim., 28 (2018), pp. 3229–3259.
  * [33] M. Duflo, Random iterative models, vol. 34 of Applications of Mathematics (New York), Springer-Verlag, Berlin, 1997.  Translated from the 1990 French original by Stephen S. Wilson and revised by the author.
  * [34] F. Facchinei and J.-S. Pang, Finite-dimensional variational inequalities and complementarity problems, Springer Series in Operations Research, Springer-Verlag, New York, NY, USA, 2003.
  * [35] I. Fatkhullin, J. Etesami, N. He, and N. Kiyavash, Sharp analysis of stochastic optimization under global Kurdkya-Łojasiewicz inequality, in Thirty-sixth Conference on Neural Information Processing Systems, 2022.
  * [36] P. Frankel, G. Garrigos, and J. Peypouquet, Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates, J. Optim. Theory Appl., 165 (2015), pp. 874–900.
  * [37] S. Gadat and F. Panloup, Optimal non-asymptotic bound of the Ruppert-Polyak averaging without strong convexity, arXiv preprint arXiv:1709.03342, (2017).
  * [38] S. Ghadimi and G. Lan, Stochastic first- and zeroth-order methods for nonconvex stochastic programming, SIAM J. Optim., 23 (2013), pp. 2341–2368.
  * [39] S. Ghadimi, G. Lan, and H. Zhang, Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization, Math. Program., 155 (2016), pp. 267–305.
  * [40] R. Gower, O. Sebbouh, and N. Loizou, SGD for structured nonconvex functions: Learning rates, minibatching and interpolation, in International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 1315–1323.
  * [41] T. H. Gronwall, Note on the derivatives with respect to a parameter of the solutions of a system of differential equations, Ann. Math., (1919), pp. 292–296.
  * [42] T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning, Springer Series in Statistics, Springer-Verlag, New York, 2001.  Data mining, inference, and prediction.
  * [43] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [44] S. J Reddi, S. Sra, B. Poczos, and A. J. Smola, Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization, Advances in neural information processing systems, 29 (2016).
  * [45] H. Karimi, J. Nutini, and M. Schmidt, Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition, in Machine Learning and Knowledge Discovery in Databases, P. Frasconi, N. Landwehr, G. Manco, and J. Vreeken, eds., Cham, 2016, Springer International Publishing, pp. 795–811.
  * [46] A. Khaled and P. Richtárik, Better theory for SGD in the nonconvex world.  arXiv preprint, arXiv:2002.03329, 2020.
  * [47] T. Ko and X. Li, A local convergence theory for the stochastic gradient descent method in non-convex optimization with non-isolated local minima.  arXiv preprint, arXiv:2203.10973v3, 2022.
  * [48] A. Krizhevsky, Learning multiple layers of features from tiny images, Master’s thesis, University of Toronto, 2009.
  * [49] K. Kurdyka, On gradients of functions definable in o-minimal structures, Ann. Inst. Fourier (Grenoble), 48 (1998), pp. 769–783.
  * [50] H. Kushner and G. G. Yin, Stochastic approximation and recursive algorithms and applications, vol. 35, Springer Science & Business Media, 2003\.
  * [51] H. J. Kushner and D. S. Clark, Stochastic approximation methods for constrained and unconstrained systems, vol. 26 of Applied Mathematical Sciences, Springer-Verlag, New York-Berlin, 1978.
  * [52] H. J. Kushner and H. Huang, Rates of convergence for stochastic approximation type algorithms, SIAM J. Control Optim., 17 (1979), pp. 607–617.
  * [53] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, 521 (2015), pp. 436–444.
  * [54] Y. Lei, T. Hu, G. Li, and K. Tang, Stochastic gradient descent for nonconvex learning without bounded gradient assumptions, IEEE Transactions on Neural Networks and Learning Systems, 31 (2019), pp. 4394–4400.
  * [55] G. Li and T. K. Pong, Global convergence of splitting methods for nonconvex composite optimization, SIAM J. Optim., 25 (2015), pp. 2434–2460.
  * [56] , Douglas–rachford splitting for nonconvex optimization with application to nonconvex feasibility problems, Math. Program., 159 (2016), pp. 371–401.
  * [57] X. Li and A. Milzarek, A unified convergence theorem for stochastic optimization methods, in Advances in Neural Information Processing Systems, vol. 35, Curran Associates, Inc., 2022, pp. 33107–33119.
  * [58] X. Li, A. Milzarek, and J. Qiu, Convergence of random reshuffling under the Kurdyka-Łojasiewicz inequality, arXiv preprint arXiv:2110.04926, (2021).
  * [59] J. Liu and Y. Yuan, On almost sure convergence rates of stochastic gradient methods, in Proceedings of Thirty Fifth Conference on Learning Theory, vol. 178, PMLR, 2022, pp. 2963–2983.
  * [60] L. Ljung, Analysis of recursive stochastic algorithms, IEEE Trans. Autom. Control, 22 (1977), pp. 551–575.
  * [61] , Strong convergence of a stochastic approximation algorithm, Ann. Statist., 6 (1978), pp. 680–696.
  * [62] L. Ljung, G. Pflug, and H. Walk, Stochastic approximation and optimization of random systems, vol. 17 of DMV Seminar, Birkhäuser Verlag, Basel, 1992.
  * [63] S. Łojasiewicz, Ensembles semi-analytiques, Institut des Hautes Etudes Scientifiques, 1965.
  * [64] S. Łojasiewicz, Sur la géométrie semi- et sous-analytique, Annales de l’institut Fourier (Grenoble), 43 (1993), pp. 1575–1595.
  * [65] S. Majewski, B. Miasojedow, and E. Moulines, Analysis of nonsmooth stochastic approximation: the differential inclusion approach.  arXiv preprint, arXiv:1805.01916v1, 2018.
  * [66] P. Mertikopoulos, N. Hallak, A. Kavis, and V. Cevher, On the almost sure convergence of stochastic gradient descent in non-convex problems, arXiv preprint arXiv:2006.11144, (2020).
  * [67] A. Milzarek, X. Xiao, S. Cen, Z. Wen, and M. Ulbrich, A stochastic semismooth newton method for nonsmooth nonconvex optimization, SIAM J. Optim., 29 (2019), pp. 2916–2948.
  * [68] J.-J. Moreau, Proximité et dualité dans un espace hilbertien, Bull. Soc. Math. FR., 93 (1965), pp. 273–299.
  * [69] E. Moulines and F. Bach, Non-asymptotic analysis of stochastic approximation algorithms for machine learning, in Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, eds., vol. 24, Curran Associates, Inc., 2011.
  * [70] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2008), pp. 1574–1609.
  * [71] A. Nitanda, Stochastic proximal gradient descent with acceleration techniques, Advances in Neural Information Processing Systems, 27 (2014).
  * [72] P. Ochs, Y. Chen, T. Brox, and T. Pock, iPiano: inertial proximal algorithm for nonconvex optimization, SIAM J. Imaging Sci., 7 (2014), pp. 1388–1419.
  * [73] F. Orabona, Almost sure convergence of SGD on smooth non-convex functions, https://parameterfree.com/2020/10/05/almost-sure-convergence- of-sgd-on-smooth-non-convex-functions/, (2020).
  * [74] W. Ouyang and A. Milzarek, A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization, arXiv preprint arXiv:2106.09340, (2021).
  * [75] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems, 32 (2019).
  * [76] V. Patel, Stopping criteria for, and strong convergence of, stochastic gradient descent on Bottou-Curtis-Nocedal functions, Math. Program., 195 (2022), pp. 693–734.
  * [77] A. Patrascu and P. Irofti, Stochastic proximal splitting algorithm for composite minimization, Optim. Lett., 15 (2021), pp. 2255–2273.
  * [78] M. Pelletier, On the almost sure asymptotic behaviour of stochastic algorithms, Stochastic Process. Appl., 78 (1998), pp. 217–244.
  * [79] B. T. Polyak, Introduction to optimization, Translations Series in Mathematics and Engineering, Optimization Software, Inc., Publications Division, New York, 1987.  Translated from the Russian, With a foreword by Dimitri P. Bertsekas.
  * [80] H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statistics, (1951), pp. 400–407.
  * [81] S. M. Robinson, Normal maps induced by linear transformations, Math. Oper. Res., 17 (1992), pp. 691–714.
  * [82] R. T. Rockafellar and R. J.-B. Wets, Variational analysis, vol. 317 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], Springer-Verlag, Berlin, 1998.
  * [83] L. Rosasco, S. Villa, and B. C. Vũ, Convergence of stochastic proximal gradient algorithm, Appl. Math. Optim., 82 (2020), pp. 891–917.
  * [84] S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms, Cambridge University Press, 2014.
  * [85] S. Shalev-Shwartz and A. Tewari, Stochastic methods for $\ell_{1}$-regularized loss minimization, J. Mach. Learn. Res., 12 (2011), pp. 1865–1892.
  * [86] O. Shamir and T. Zhang, Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes, in International Conference on Machine Learning, PMLR, 2013, pp. 71–79.
  * [87] A. Shapiro, D. Dentcheva, and A. Ruszczyński, Lectures on stochastic programming, vol. 9 of MOS-SIAM Series on Optimization, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA; Mathematical Optimization Society, Philadelphia, PA, second ed., 2014.  Modeling and theory.
  * [88] L. Simon, Asymptotics for a class of non-linear evolution equations, with applications to geometric problems, Ann. Math., (1983), pp. 525–571.
  * [89] K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, in International Conference on Learning Representations, May 2015.
  * [90] D. W. Stroock, Probability theory, Cambridge University Press, Cambridge, second ed., 2011.
  * [91] V. B. Tadić, Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema.  arXiv preprint, extended version, arXiv:0907.1020v5, 2009.
  * [92] , Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema, Stochastic Process. Appl., 125 (2015), pp. 1715–1755.
  * [93] J. Tang, M. Liu, N. Jiang, H. Cai, W. Yu, and J. Zhou, Data-free network pruning for model compression, in 2021 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, 2021, pp. 1–5.
  * [94] X. Wang, S. Ma, D. Goldfarb, and W. Liu, Stochastic quasi-newton methods for nonconvex stochastic optimization, SIAM J. Optim., 27 (2017), pp. 927–956.
  * [95] S. Wojtowytsch, Stochastic gradient descent with noise of machine learning type part I: discrete time analysis, J. Nonlinear Sci., 33 (2023), pp. Paper No. 45, 52.
  * [96] L. Xiao and T. Zhang, A proximal stochastic gradient method with progressive variance reduction, SIAM J. Optim., 24 (2014), pp. 2057–2075.
  * [97] M. Yang, A. Milzarek, Z. Wen, and T. Zhang, A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization, Math. Program., 194 (2022), pp. 257–303.
  * [98] H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B-Stat. Methodol., 67 (2005), pp. 301–320.

##  Appendix A Stochastic Estimates and Derivations

###  A.1 Measurability of $L$

In order to establish measurability of the mapping $\bm{L}$ — defined in Eq.
13 — it suffices to verify ${\bm{L}}^{-1}([a,\infty])\in\mathcal{F}$ for all
$a\in[-\infty,\infty]$. Here, we have

$\displaystyle{\bm{L}}^{-1}([a,\infty])$
$\displaystyle=\\{\omega\in\Omega:{\bm{L}}(\omega)\in[a,\infty]\\}=\\{\omega\in\Omega:\exists~{}\bar{x}\in\mathrm{cl}(\mathrm{conv}(\\{\bm{x}^{k}(\omega)\\}_{k}))\,\text{such
that}\,\mathrm{lip}\,\nabla f(\bar{x})\geq a\\}$
$\displaystyle=\\{\omega\in\Omega:\Gamma_{a}(\omega)\neq\emptyset\\}=\mathrm{dom}(\Gamma_{a}),$

where the multifunction $\Gamma_{a}:\Omega\rightrightarrows\mathbb{R}^{d}$ is
given by
$\Gamma_{a}(\omega):=\mathrm{cl}(\mathrm{conv}(\\{\bm{x}^{k}(\omega)\\}_{k}))\cap\\{x\in\mathbb{R}^{d}:\mathrm{lip}\,\nabla
f(x)\geq a\\}$. Since $\mathrm{lip}\,\nabla f$ is upper semicontinuous, the
set $\mathcal{C}_{a}:=\\{x\in\mathbb{R}^{d}:\mathrm{lip}\,\nabla f(x)\geq
a\\}$ is closed for all $a$, see, e.g., [82, Chapters 1 and 9]. In addition,
using
$\Gamma(\omega):=\mathrm{cl}(\mathrm{conv}(\\{\bm{x}^{k}(\omega)\\}_{k}))=\mathrm{cl}(\mathrm{conv}(\mathrm{cl}(\\{\bm{x}^{k}(\omega)\\}_{k})))$
and applying [82, Theorem 14.5, Exercise 14.12, and Proposition 14.2], the
multifunction $\Gamma$ can be shown to be closed-valued and measurable. Thus,
by [82, Proposition 14.11], this establishes measurability of
$\Gamma_{a}=\Gamma\cap\mathcal{C}_{a}$ and $\bm{L}$.

###  A.2 Proof of Lemma 2.8

###### Proof.

Let us first define the filtration
$\mathcal{U}_{\ell}:=\mathcal{F}_{n_{k}+\ell}$ and let us introduce the
sequence $\\{\bm{y}^{\ell}\\}_{\ell}$ as follows: $\bm{y}^{0}:=0$,

$\bm{y}^{\ell}:={\sum}_{i=n_{k}}^{n_{k}+\ell-1}\alpha_{i}\beta_{i}\bm{e}^{i},\;\text{for}\;\ell\in\\{1,\dots,n_{k+1}-n_{k}\\},\quad\text{and}\quad\bm{y}^{\ell}:={\sum}_{i=n_{k}}^{n_{k+1}-1}\alpha_{i}\beta_{i}\bm{e}^{i},\;\text{for}\;\ell>n_{k+1}-n_{k}.$

Then, each of the functions $\bm{y}_{\ell}$ is $\mathcal{U}_{\ell}$-measurable
and we have

${\mathbb{E}[{\bm{y}^{\ell+1}\mid\mathcal{U}_{\ell}}]={\sum}_{i=n_{k}}^{n_{k}+\ell}\alpha_{i}\beta_{i}\mathbb{E}[{\bm{e}^{i}\mid\mathcal{U}_{\ell}}]=\bm{y}^{\ell}+\alpha_{n_{k}+\ell}\beta_{n_{k}+\ell}\mathbb{E}[{\bm{e}^{n_{k}+\ell}\mid\mathcal{F}_{n_{k}+\ell}}]=\bm{y}^{\ell}.}$

(similarly for $\ell\geq n_{k+1}-n_{k}$). Thus, $\\{\bm{y}^{\ell}\\}_{\ell}$
defines a $\\{\mathcal{U}_{\ell}\\}$-martingale. By the Burkholder-Davis-Gundy
inequality, (A.2), and setting $\bar{\bm{r}}_{k}:=\max_{n_{k}<j\leq
n_{k+1}}\|{\sum}_{i=n_{k}}^{j-1}\alpha_{i}\beta_{i}\bm{e}^{i}\|$, it follows

$\displaystyle\mathbb{E}[{\bar{\bm{r}}_{k}^{2}}]=\mathbb{E}\left[{{\sup}_{\ell>0}\|\bm{y}^{\ell}\|^{2}}\right]$
$\displaystyle\leq
C_{2}\mathbb{E}\left[{{\sum}_{\ell=1}^{\infty}\|\bm{y}^{\ell}-\bm{y}^{\ell-1}\|^{2}}\right]$
$\displaystyle=C_{2}{\sum}_{i=n_{k}}^{n_{k+1}-1}\alpha_{i}^{2}\beta_{i}^{2}\mathbb{E}[{\|\bm{e}^{i}\|^{2}}]\leq
C_{2}{\sum}_{i=n_{k}}^{n_{k+1}-1}\alpha_{i}^{2}\beta_{i}^{2}\sigma_{i}^{2}.$
(59)

Next, for all $j\in\\{n_{k}+1,\dots,n_{k+1}\\}$ and similar to [92, Lemma
6.1], we have

$\displaystyle\sum_{i=n_{k}}^{j-1}\alpha_{i}\bm{e}^{i}$
$\displaystyle=\frac{1}{\beta_{j-1}}\sum_{i=n_{k}}^{j-1}\alpha_{i}\beta_{i}\bm{e}^{i}-\frac{1}{\beta_{j-1}}\sum_{i=n_{k}}^{j-2}\alpha_{i}\beta_{i}\bm{e}^{i}+\sum_{i=n_{k}}^{j-2}\alpha_{i}\bm{e}^{i}$
$\displaystyle=\dots=\frac{1}{\beta_{j-1}}\sum_{i=n_{k}}^{j-1}\alpha_{i}\beta_{i}\bm{e}^{i}+\sum_{\ell=n_{k}}^{j-2}\left[\frac{1}{\beta_{\ell}}-\frac{1}{\beta_{\ell+1}}\right]\sum_{i=n_{k}}^{\ell}\alpha_{i}\beta_{i}\bm{e}^{i}.$

Since $\\{\beta_{k}\\}_{k}$ is non-decreasing for all $k$ sufficiently large,
this yields

$\beta_{n_{k}}\bm{r}_{k}=\beta_{n_{k}}\max_{n_{k}<j\leq
n_{k+1}}\left\|{\sum}_{i=n_{k}}^{j-1}\alpha_{i}\bm{e}^{i}\right\|\leq\max_{n_{k}<j\leq
n_{k+1}}\beta_{n_{k}}\left[\beta_{j-1}^{-1}+\sum_{\ell=n_{k}}^{j-2}(\beta_{\ell}^{-1}-\beta_{\ell+1}^{-1})\right]\cdot\bar{\bm{r}}_{k}=\bar{\bm{r}}_{k}$

for all $k\geq K^{\prime}$ and some $K^{\prime}\in\mathbb{N}$. Furthermore,
using (A.4), the monotone convergence theorem and Eq. 59 imply

$\displaystyle\mathbb{E}\left[{{\sum}_{k=0}^{\infty}\beta_{n_{k}}^{2}{\bm{r}_{k}^{2}}}\right]={\sum}_{k=0}^{\infty}{\mathbb{E}[{\beta_{n_{k}}^{2}\bm{r}_{k}^{2}}]}\leq{{\sum}_{k=0}^{K^{\prime}-1}\mathbb{E}[\beta_{n_{k}}^{2}\bm{r}_{k}^{2}]+{\sum}_{k=K^{\prime}}^{\infty}\mathbb{E}[\bar{\bm{r}}_{k}^{2}]<\infty.}$

and consequently, we obtain
$\sum_{k=0}^{\infty}\beta_{n_{k}}^{2}{\bm{r}_{k}^{2}}<\infty$ almost surely. ∎

##  Appendix B Derivation of Preparatory Lemmas

###  B.1 Proof of Lemma 2.10

###### Proof.

For $\omega\notin\mathcal{L}$, we have $\bm{L}(\omega)=\infty$ and the
expression on the right sight of Eq. 22 equals “$\infty$”. The estimate Eq. 22
is obviously satisfied in this case. Next, let us consider an arbitrary sample
$\omega\in\mathcal{L}$ with associated realizations
$\\{x^{k}\\}_{k}=\\{\bm{x}^{k}(\omega)\\}_{k}$,
$\\{z^{k}\\}_{k}=\\{\bm{z}^{k}(\omega)\\}_{k}$, $L=\bm{L}(\omega)$, etc. Based
on Eq. 17, we can deduce that $f$ is Lipschitz smooth on
$\mathrm{cl}(\mathrm{conv}(\\{x^{k}\\}_{k}))$ with modulus $L$. This yields

$\displaystyle\psi(x^{n})-\psi(x^{m})\leq\langle\nabla
f(x^{m}),x^{n}-x^{m}\rangle+\frac{L}{2}\|x^{n}-x^{m}\|^{2}+\frac{1}{\lambda}\langle
z^{n}-x^{n},x^{n}-x^{m}\rangle,$ (60)

where we utilized
$\nabla\mathrm{env}_{\lambda\varphi}(z^{n})=\lambda^{-1}(z^{n}-x^{n})\in\partial\varphi(x^{n})$
in the last inequality. Moreover, based on the definition of
$F^{\lambda}_{\mathrm{nor}}$, we can reformulate Eq. 60 as follows

$\displaystyle\psi(x^{n})-\psi(x^{m})$ $\displaystyle\leq\langle\nabla
f(x^{m}),x^{n}-x^{m}\rangle+\frac{L}{2}\|{x^{n}-x^{m}}\|^{2}+\langle
F^{\lambda}_{\mathrm{nor}}(z^{m})-\nabla f(x^{m}),x^{n}-x^{m}\rangle$
$\displaystyle\hskip 17.22217pt+\lambda^{-1}\langle
z^{n}-x^{n},x^{n}-x^{m}\rangle-\lambda^{-1}\langle
z^{m}-x^{m},x^{n}-x^{m}\rangle$
$\displaystyle=\left({\frac{L}{2}-\frac{1}{\lambda}}\right)\|{x^{n}-x^{m}}\|^{2}+\langle
F^{\lambda}_{\mathrm{nor}}(z^{m}),x^{n}-x^{m}\rangle+\frac{1}{\lambda}\langle
z^{n}-z^{m},x^{n}-x^{m}\rangle,$

which finishes the proof. ∎

###  B.2 Proof of Lemma 2.11

###### Proof.

As before, we only need to consider samples $\omega$ in $\mathcal{L}$. Let
$\omega\in\mathcal{L}$ be such an arbitrary sample and let us set
$x^{k}=\bm{x}^{k}(\omega)$, $z^{k}=\bm{z}^{k}(\omega)$, $L=\bm{L}(\omega)$,
$e\equiv e^{m,n}=\bm{e}^{m,n}(\omega)$, $\tau\equiv\tau_{m,n}$, etc. Using the
definition of $F^{\lambda}_{\mathrm{nor}}$ and Eq. 16, we can expand
$F^{\lambda}_{\mathrm{nor}}(z^{n})$ as follows

$\displaystyle F^{\lambda}_{\mathrm{nor}}(z^{n})$ $\displaystyle=\nabla
f(x^{n})+\frac{1}{\lambda}\left({z^{n}-x^{n}}\right)=F^{\lambda}_{\mathrm{nor}}(z^{m})+[\nabla
f(x^{n})-\nabla
f(x^{m})]+\frac{1}{\lambda}\left({z^{n}-z^{m}+x^{m}-x^{n}}\right)$
$\displaystyle=(1-\tau\lambda^{-1})F^{\lambda}_{\mathrm{nor}}(z^{m})+[\nabla
f(x^{n})-\nabla f(x^{m})]-\lambda^{-1}[x^{n}-x^{m}]+\lambda^{-1}e.$ (61)

Applying the (trajectory-based) Lipschitz continuity of $\nabla f$, see Eq.
17, we can then infer

$\displaystyle\|F^{\lambda}_{\mathrm{nor}}(z^{n})\|^{2}$
$\displaystyle=\left({1-\frac{\tau}{\lambda}}\right)^{2}\|F^{\lambda}_{\mathrm{nor}}(z^{m})\|^{2}+\|\nabla
f(x^{n})-\nabla f(x^{m})\|^{2}-\frac{2}{\lambda^{2}}\langle
e,x^{n}-x^{m}\rangle$ $\displaystyle\hskip
17.22217pt+2\left({1-\frac{\tau}{\lambda}}\right)\langle
F^{\lambda}_{\mathrm{nor}}(z^{m}),[\nabla f(x^{n})-\nabla
f(x^{m})]-\lambda^{-1}[x^{n}-x^{m}]+\lambda^{-1}e\rangle+\frac{1}{\lambda^{2}}\|e\|^{2}$
$\displaystyle\hskip 17.22217pt+2\langle\nabla f(x^{n})-\nabla
f(x^{m}),\lambda^{-1}e-\lambda^{-1}[x^{n}-x^{m}]\rangle+\frac{1}{\lambda^{2}}\|x^{n}-x^{m}\|^{2}$
$\displaystyle\leq\left({1-\frac{\tau}{\lambda}}\right)^{2}\|F^{\lambda}_{\mathrm{nor}}(z^{m})\|^{2}+\frac{2}{\lambda}\left({1-\frac{\tau}{\lambda}}\right)\langle
F^{\lambda}_{\mathrm{nor}}(z^{m}),\lambda[\nabla f(x^{n})-\nabla
f(x^{m})]-[x^{n}-x^{m}]+e\rangle$
$\displaystyle\quad+\left({L^{2}+\frac{2L}{\lambda}+\frac{1}{\lambda^{2}}}\right)\|x^{n}-x^{m}\|^{2}+\frac{2}{\lambda}\langle\nabla
f(x^{n})-\nabla
f(x^{m}),e\rangle+\frac{1}{\lambda^{2}}\|e\|^{2}-\frac{2}{\lambda^{2}}\langle
e,x^{n}-x^{m}\rangle,$

which completes the proof. ∎

##  Appendix C Global Convergence Under Lipschitz Continuity

###  C.1 Proof of 2.16

###### Proof.

Given ${\sf T}$, we can define the associated time indices
$\\{\hat{m}_{k}\\}_{k}$ recursively via $\hat{m}_{0}:=0$ and
$\hat{m}_{k+1}:=\varpi(\hat{m}_{k},{\sf T})$ for all $k\in\mathbb{N}$. Similar
to the discussion in subsection 2.4, we can again guarantee $\frac{4}{5}{\sf
T}\leq{\sf
T}-\alpha_{\hat{m}_{k+1}}\leq\tau_{\hat{m}_{k},\hat{m}_{k+1}}\leq{\sf T}$ for
all $k\geq K_{0}$ and some $K_{0}\in\mathbb{N}$. We now first discuss the
behavior of the error terms $\\{\bm{s}_{k}\\}_{k}$ under the general variance
bound (B.2) and the weaker step size condition
$\sum_{k=0}^{\infty}\alpha_{k}^{2}\max\\{\sigma_{k}^{2},{\sf A}\\}<\infty$.
Following the proof of Lemma 2.13, we have

$H_{\xi}(\bm{z}^{\hat{m}_{k+1}})-H_{\xi}(\bm{z}^{\hat{m}_{k}})\leq-\frac{\xi{\sf
T}}{5}\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}+\frac{5}{{\sf
T}}\bm{s}_{k}^{2}\quad\forall~{}k\geq K_{0},\quad\forall~{}\omega\in\Omega,$
(62)

where $\xi$ is defined in subsection 2.5.2 and we set
$\bm{s}_{k}:=\bm{s}_{\hat{m}_{k},\hat{m}_{k+1}}=\max_{\hat{m}_{k}<j\leq\hat{m}_{k+1}}\|\sum_{i=\hat{m}_{k}}^{j-1}\alpha_{i}\bm{e}^{i}\|$.
Mimicking the application of the Burkholder-Davis-Gundy inequality (2.5) in
the proof of Lemma 2.8 (by essentially setting $\beta_{k}\equiv 1$) and
applying (B.2), we obtain

$\mathbb{E}[\bm{s}_{k}^{2}]\leq
C_{2}{\sum}_{i=\hat{m}_{k}}^{\hat{m}_{k+1}-1}\alpha_{i}^{2}(\sigma_{i}^{2}+{\sf
A}\mathbb{E}[\psi(\bm{x}^{i})-\bar{\psi}]),\quad{C_{2}\leq 4}.$ (63)

Using $\tau_{\hat{m}_{k},i}\leq{\sf T}\leq\lambda$ and
$\bar{\tau}_{\hat{m}_{k},i}\leq\bar{\lambda}$ for all
$i=\hat{m}_{k}+1,\dots,\hat{m}_{k+1}$, the estimate Eq. 34 in the proof of
2.15 is valid, i.e., we have

$\psi(\bm{x}^{i})\leq\psi(\bm{x}^{\hat{m}_{k}})+[{\tfrac{1}{2}+({\sf
L}\lambda+1)}(1+\bar{\lambda}\lambda)^{2}]\lambda\cdot\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}+{({\sf
L}+\lambda^{-1})}(1+\bar{\lambda}\lambda)^{2}\bm{s}_{k}^{2}$

for all $i=\hat{m}_{k}+1,\dots,\hat{m}_{k+1}-1$ and all $\omega$.
Consequently, defining $\Lambda_{1}:=[1+2({\sf
L}\lambda+1)(1+\bar{\lambda}\lambda)^{2}]\xi^{-1}$ and $\Lambda_{2}:=({\sf
L}+\lambda^{-1})(1+\bar{\lambda}\lambda)^{2}$, this yields
$\mathbb{E}[\psi(\bm{x}^{i})-\bar{\psi}]\leq\Lambda_{1}\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]+\Lambda_{2}\mathbb{E}[\bm{s}_{k}^{2}]$.
Setting
$\hat{\gamma}_{k}:={\sum}_{i=\hat{m}_{k}}^{\hat{m}_{k+1}-1}\alpha_{i}^{2}\max\\{\sigma_{i}^{2},{\sf
A}\\}$ and using the last bound in Eq. 63, it follows

${\mathbb{E}[\bm{s}^{2}_{k}]\leq
4\hat{\gamma}_{k}+4\Lambda_{1}\hat{\gamma}_{k}\cdot\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]+4\Lambda_{2}\hat{\gamma}_{k}\cdot\mathbb{E}[\bm{s}^{2}_{k}]}.$
(64)

By assumption $\\{\hat{\gamma}_{k}\\}_{k}$ converges to zero and hence, there
is $K_{1}\geq K_{0}$ such that $\hat{\gamma}_{k}\leq 1/(8\Lambda_{2})$ for all
$k\geq K_{1}$. Thus, taking expectation in Eq. 62, we can conclude

$\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k+1}})-\bar{\psi}]\leq{\left(1+\frac{40\Lambda_{1}}{{\sf
T}}\cdot\hat{\gamma}_{k}\right)}\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]-\frac{\xi{\sf
T}}{5}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]+{\frac{40}{{\sf
T}}}\cdot\hat{\gamma}_{k}$ (65)

for all $k\geq K_{1}$. Due to $\sum_{k=0}^{\infty}\hat{\gamma}_{k}<\infty$,
this shows that
$\\{\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]\\}_{k}$ satisfies a
quasi-Féjer-type condition and thus, by [11, Proposition A.31] there is
$\rho^{*}\in\mathbb{R}$ such that
$\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})]\to\rho^{*}$ and we have
$\sum_{k=0}^{\infty}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]<\infty$.
Hence, the sequence
$\\{\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]\\}_{k}$ is bounded
and using Eq. 64, we can infer
$\mathbb{E}[\bm{s}_{k}^{2}]=\mathcal{O}(\hat{\gamma}_{k})$. Applying the
monotone convergence theorem, this implies
$\sum_{k=0}^{\infty}\bm{s}_{k}^{2}<\infty$ almost surely.

At this point, the steps and derivations in the proof of 2.15 can be fully
mimicked and repeated which establishes (a). Next, invoking the estimate Eq.
32 and taking total expectation after squaring both sides, we obtain

$\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]\leq
2(1+\lambda\bar{\lambda})^{2}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]+2\bar{\lambda}^{2}\mathbb{E}[\bm{s}_{k}^{2}]\quad\forall~{}i=\hat{m}_{k}+1,\dots,\hat{m}_{k+1}.$
(66)

Using
$\lim_{k\to\infty}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]=0$
and $\lim_{k\to\infty}\mathbb{E}[\bm{s}^{2}_{k}]=0$, this readily implies
$\lim_{k\to\infty}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{k})\|^{2}]=0$.
In addition, it holds that

$\mathbb{E}[\psi(\bm{x}^{\hat{m}_{k}})]=\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})]-\frac{\xi\lambda}{2}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]\to\rho^{*}.$

Similar to Eq. 34 and Eq. 35, we can derive upper and lower bounds for
$\mathbb{E}[\psi(\bm{x}^{i})]$ in terms of
$\mathbb{E}[\psi(\bm{x}^{\hat{m}_{k}})]$,
$\mathbb{E}[\psi(\bm{x}^{\hat{m}_{k+1}})]$, $\mathbb{E}[\bm{s}_{k}^{2}]$, and
$\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]$. This
allows to show $\lim_{k\to\infty}\mathbb{E}[\psi(\bm{x}^{k})]=\rho^{*}$. The
convergence of
$\\{\mathbb{E}[\|F^{\lambda}_{\mathrm{nat}}(\bm{x}^{k})\|^{2}]\\}_{k}$ follows
from
$\mathbb{E}[\|F^{\lambda}_{\mathrm{nat}}(\bm{x}^{k})\|^{2}]\leq\lambda^{2}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{k})\|^{2}]\to
0$. ∎

###  C.2 Proof of 2.17

###### Proof.

By assumption, the step sizes $\\{\alpha_{k}\\}_{k}$ satisfy
$\alpha_{k}\in(0,\frac{{\sf T}}{5}]$ for all $k$. Hence, the conditions stated
in (T.1) and (T.2) hold for $\beta_{k}\equiv 1$ and $\delta=\frac{4}{5}$ and
we have $\frac{4}{5}{\sf T}\leq\tau_{\hat{m}_{k},\hat{m}_{k+1}}\leq{\sf T}$
for all $k\in\mathbb{N}$. In addition, utilizing the definition of ${\sf T}$,
we can infer $K_{0}=K_{\delta}=0$. Thus, Eq. 62 holds for all
$k\in\mathbb{N}$. Moreover, due to ${\sf A}=0$, the estimates Eq. 63 and Eq.
65 reduce to $\mathbb{E}[\bm{s}_{k}^{2}]\leq
C_{2}\sum_{i=\hat{m}_{k}}^{\hat{m}_{k+1}-1}\alpha_{i}^{2}\sigma_{i}^{2}\leq
4\hat{\gamma}_{k}$ and

$\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k+1}})-\bar{\psi}]\leq\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{k}})-\bar{\psi}]-\frac{\xi{\sf
T}}{5}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]+{\frac{20}{{\sf
T}}}\cdot\hat{\gamma}_{k}\quad\forall~{}k\geq 0.$ (67)

Let $j\in\mathbb{N}$ be given with
$\hat{m}_{j}\leq\mathsf{K-1}<\hat{m}_{j+1}$. Multiplying Eq. 66 with
$\alpha_{i}$, using $\tau_{\hat{m}_{k},\hat{m}_{k+1}}\leq{\sf T}$, and summing
the resulting inequalities, we obtain

${\sum}_{i=\hat{m}_{k}}^{\hat{m}_{k+1}-1}\alpha_{i}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]\leq
2(1+\lambda\bar{\lambda})^{2}{\sf
T}\cdot\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]+8\bar{\lambda}^{2}{\sf
T}\cdot\hat{\gamma}_{k}\quad\forall~{}k\geq 0.$ (68)

Similarly, applying the derivations in the proof of Lemma 2.9 with
$m\equiv\hat{m}_{j}$ and $n\equiv\mathsf{K}-1$,
$\tau_{\hat{m}_{j},\mathsf{K}-1}\leq{\sf T}\leq\lambda$, and mimicking our
earlier steps, we can show

${\sum}_{i=\hat{m}_{j}}^{\mathsf{K}-1}\alpha_{i}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]\leq
2(1+\lambda\bar{\lambda})^{2}{\sf
T}\cdot\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{j}})\|^{2}]+8\bar{\lambda}^{2}{\sf
T}\cdot{\sum}_{i=\hat{m}_{j}}^{\mathsf{K}-1}\alpha_{i}^{2}\sigma_{i}^{2}.$
(69)

Here, the bound $\mathbb{E}[\bm{s}_{\hat{m}_{j},\mathsf{K}-1}^{2}]\leq
4{\sum}_{i=\hat{m}_{j}}^{\mathsf{K}-1}\alpha_{i}^{2}\sigma_{i}^{2}$ again
follows from the Burkholder-Davis-Gundy inequality and (A.2). Combining Eq. 68
and Eq. 69, summing Eq. 67 for $k=0,\dots,j-1$, and invoking the definition of
$H_{\xi}$, this yields

$\displaystyle{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}\cdot\min_{i=0,\dots,\mathsf{K}-1}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]$
$\displaystyle\leq{\sum}_{k=0}^{j-1}{\sum}_{i=\hat{m}_{k}}^{\hat{m}_{k+1}-1}\alpha_{i}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]+{\sum}_{i=\hat{m}_{j}}^{\mathsf{K}-1}\alpha_{i}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]$
$\displaystyle\leq 2(1+\lambda\bar{\lambda})^{2}{\sf
T}\cdot{\sum}_{k=0}^{j}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{\hat{m}_{k}})\|^{2}]+8\bar{\lambda}^{2}{\sf
T}\cdot{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}^{2}\sigma_{i}^{2}$
$\displaystyle\leq{10(1+\lambda\bar{\lambda})^{2}}{\xi^{-1}}\left\\{H_{\xi}(z^{0})-\bar{\psi}-\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{j}})-\bar{\psi}]+\frac{20}{{\sf
T}}{\sum}_{i=0}^{j-1}\hat{\gamma}_{i}\right\\}$ $\displaystyle\hskip
17.22217pt+{4(1+\lambda\bar{\lambda})^{2}{\sf
T}}{(\xi\lambda)^{-1}}\mathbb{E}[H_{\xi}(\bm{z}^{\hat{m}_{j}})-\bar{\psi}]+8\bar{\lambda}^{2}{\sf
T}\cdot{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}^{2}\sigma_{i}^{2}.$

Finally, rearranging the terms and using
$\mathbb{E}[\|F^{\lambda}_{\mathrm{nat}}(\bm{x}^{i})\|^{2}]\leq\lambda^{2}\mathbb{E}[\|F^{\lambda}_{\mathrm{nor}}(\bm{z}^{i})\|^{2}]$,
we can infer

$\min_{i=0,\dots,\mathsf{K}-1}\mathbb{E}[\|F^{\lambda}_{\mathrm{nat}}(\bm{x}^{i})\|^{2}]\leq\frac{10(1+\lambda\bar{\lambda})^{2}\lambda^{2}\cdot[H_{\xi}(z^{0})-\bar{\psi}]}{\xi\cdot{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}}+\max\left\\{\frac{200(1+\lambda\bar{\lambda})^{2}\lambda^{2}}{{\sf
T}\xi},8\lambda^{2}\bar{\lambda}^{2}{\sf
T}\right\\}\frac{{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}^{2}\sigma_{i}^{2}}{{\sum}_{i=0}^{\mathsf{K}-1}\alpha_{i}}.$

This finishes the proof. ∎

##  Appendix D Lemmas for Strong Convergence

###  D.1 Proof of Lemma 3.9

###### Proof.

By (ii), the mapping $g$ is monotonically increasing on $I$. Thus,
$\\{\beta_{k}\\}_{k}$ is monotonically increasing for all $k$ sufficiently
large and using (i), this verifies (A.4). Let $\\{m_{k}\\}_{k}$ denote the
time indices associated with the time window $T$. Then, the observations
(T.1)–(T.2) are applicable and there exists $K_{\delta}>0$ such that

$\delta Tk\leq\gamma_{m_{n+k}}-\gamma_{m_{n}}\leq
Tk,\quad\forall\;k\in\mathbb{N},\quad\forall\;n\geq K_{\delta}.$

Let us set $\nu_{k}:=\gamma_{m_{k}}$ and notice that
$x\mapsto\rho_{\vartheta}(x):=g(x)^{-2\vartheta}$ is monotonically decreasing
on $I$ as $g$ is increasing on $I$. Since $\nu_{k}\to\infty$ (by (A.3)), there
exists $N_{\delta}\geq K_{\delta}$ such that $\nu_{k}\geq\iota$. Thus, using
the integral test for convergence and the relation $\nu_{n}+\delta
Tk\leq\nu_{n+k}$, we have for all $k\geq N_{\delta}$,

$\displaystyle\sum_{k=n}^{\infty}\rho_{\vartheta}(\nu_{k})$
$\displaystyle=\sum_{k=0}^{\infty}\rho_{\vartheta}(\nu_{n+k})\leq\sum_{k=0}^{\infty}\rho_{\vartheta}(\nu_{n}+\delta
Tk)$
$\displaystyle\leq\rho_{\vartheta}(\nu_{n})+\int_{0}^{\infty}\rho_{\vartheta}(\nu_{n}+\delta
Ts)\,\mathrm{d}s=\rho_{\vartheta}(\nu_{n})+(\delta
T)^{-1}\int_{\nu_{n}}^{\infty}\rho_{\vartheta}(t)\,\mathrm{d}t,$

where we used the substitution $t=\nu_{n}+\delta Ts$ in the last step. This
completes the proof. ∎

###  D.2 Proof of Lemma 3.10

Before presenting the proof, we first restate Lemma 3 in [79, Section 2.2]

######  Lemma D.1.

Let the sequences $\\{p_{k}\\}_{k}$, $\\{q_{k}\\}_{k}$, and $\\{u_{k}\\}_{k}$
satisfy $a_{k}\geq 0$, $0\leq q_{k}<1$,

$u_{k+1}\leq
q_{k}u_{k}+p_{k},\quad{\sum}_{k=0}^{\infty}(1-q_{k})=\infty,\quad\text{and}\quad\lim_{k\to\infty}p_{k}/(1-q_{k})=0.$

Then, $\limsup_{k\to\infty}u_{k}\leq 0$.

###### Proof.

By assumption, there exists $K_{b}\in\mathbb{N}$ such that $b_{k+1}\leq
b_{k}+\mathsf{B}$, $b_{k}\in I$, and $s(b_{k})>1$ for all $k\geq K_{b}$.
Hence, using the monotonicity and convexity of $\kappa(x):=s(x)/t(x)$, it
follows

$\kappa(b_{k+1})-\kappa(b_{k})\geq\kappa(b_{k}+\mathsf{B})-\kappa(b_{k})\geq\mathsf{B}\kappa^{\prime}(b_{k})=\mathsf{B}[s^{\prime}(b_{k})-\kappa(b_{k})t^{\prime}(b_{k})]\cdot\frac{1}{t(b_{k})}\geq\frac{\beta-1}{t(b_{k})}$
(70)

for all $k\geq K_{b}$. This yields

$a_{k+1}-\frac{1}{\beta}\kappa(b_{k+1})\leq\left(1-\frac{1}{s(b_{k})}\right)a_{k}+\frac{1}{t(b_{k})}-\frac{1}{\beta}\kappa(b_{k})-\frac{\beta-1}{\beta
t(b_{k})}=\left(1-\frac{1}{s(b_{k})}\right)\left(a_{k}-\frac{1}{\beta}\kappa(b_{k})\right).$
(71)

Suppose now there is $n\geq K_{b}$ such that
$a_{n}\leq\frac{1}{\beta}\kappa(b_{n})$. Thanks to Eq. 71 and $s(b_{k})>1$,
$k\geq K_{b}$, this implies $a_{k}\leq\frac{1}{\beta}\kappa(b_{k})$ for all
$k\geq n$. If such $n$ does not exist, then we can infer
$a_{k}-\frac{1}{\beta}\kappa(b_{k})\geq 0$ for all $k\geq K_{b}$. Dividing Eq.
70 by $\kappa(b_{k})$, we further obtain

$\frac{\kappa(b_{k+1})}{\kappa(b_{k})}\geq
1+\frac{\beta-1}{s(b_{k})}=\frac{s(b_{k})-(1-\beta)}{s(b_{k})}.$

Consequently, multiplying Eq. 71 with $1/\kappa(b_{k+1})$, it follows

$\displaystyle\frac{a_{k+1}}{\kappa(b_{k+1})}-\frac{1}{\beta}$
$\displaystyle\leq\left(1-\frac{1}{s(b_{k})}\right)\frac{\kappa(b_{k})}{\kappa(b_{k+1})}\left(\frac{a_{k}}{\kappa(b_{k})}-\frac{1}{\beta}\right)$
$\displaystyle\leq\left(1-\frac{1}{s(b_{k})}\right)\frac{s(b_{k})}{s(b_{k})-(1-\beta)}\left(\frac{a_{k}}{\kappa(b_{k})}-\frac{1}{\beta}\right)=\left(1-\frac{\beta}{s(b_{k})-(1-\beta)}\right)\left(\frac{a_{k}}{\kappa(b_{k})}-\frac{1}{\beta}\right).$

Applying Lemma D.1 with $u_{k}:=\frac{a_{k}}{\kappa(b_{k})}-\frac{1}{\beta}$,
$p_{k}:=0$, and $q_{k}:=1-\frac{\beta}{s(b_{k})-(1-\beta)}$ and using
$\sum_{k=0}^{\infty}\frac{1}{s(b_{k})}=\infty$, we can finally infer
$\limsup_{k\to\infty}\frac{a_{k}}{\kappa(b_{k})}-\frac{1}{\beta}\leq 0$ which
completes the proof. ∎


# A Lightweight Multi-Attack CAN Intrusion Detection System on Hybrid FPGAs

Shashwat Khandelwal & Shanker Shreejith  Department of Electronic & Electrical
Engineering, Trinity College Dublin
Dublin, Ireland
Email: {khandels<EMAIL_ADDRESS>
###### Abstract

Rising connectivity in vehicles is enabling new capabilities like connected
autonomous driving and advanced driver assistance systems (ADAS) for improving
the safety and reliability of next-generation vehicles. This increased access
to in-vehicle functions compromises critical capabilities that use legacy in-
vehicle networks like Controller Area Network (CAN), which has no inherent
security or authentication mechanism. Intrusion detection and mitigation
approaches, particularly using machine learning models, have shown promising
results in detecting multiple attack vectors in CAN through their ability to
generalise to new vectors. However, most deployments require dedicated
computing units like GPUs to perform line-rate detection, consuming much
higher power. In this paper, we present a lightweight multi-attack quantised
machine learning model that is deployed using Xilinx’s Deep Learning
Processing Unit IP on a Zynq Ultrascale+ (XCZU3EG) FPGA, which is trained and
validated using the public CAN Intrusion Detection dataset. The quantised
model detects denial of service and fuzzing attacks with an accuracy of above
99% and a false positive rate of 0.07%, which are comparable to the state-of-
the-art techniques in the literature. The Intrusion Detection System (IDS)
execution consumes just 2.0 W with software tasks running on the ECU and
achieves a 25% reduction in per-message processing latency over the state-of-
the-art implementations. This deployment allows the ECU function to coexist
with the IDS with minimal changes to the tasks, making it ideal for real-time
IDS in in-vehicle systems.

###### Index Terms:

Controller Area Network, Machine Learning, FPGA, Intrusion Detection System

##  I Introduction & Related Works

Most high-end vehicles today integrate over 50 electronic computing units
(ECUs) interconnected through different network standards for incorporating
safety-critical, comfort and automation capabilities in a cost and energy-
efficient manner. Controller Area Network (CAN) and its variants continue to
be the most widely used network protocol in automotive electric/electronic
systems owing to their low cost, flexibility, and robustness. CAN is a
broadcast protocol with implicit priority using the _ID_ field and support for
multiple data rates (from 125 Kbps to 1 Mbps), allowing their use in critical
and non-critical network segments in the vehicle [1, 2]. CAN (and most other
automotive networks) do not have native support for message (sender)
authentication or encryption; however, due to the physical localization of
earlier vehicular systems and the design of (critical) applications as silos
(paritioned into domains), integrating them over unprotected CAN networks was
considered secure. With increasing connectivity enforcing bridged access
between previously siloed domains, security gaps and vulnerabilities in these
networks have been exposed further through numerous exploits [3]. For
instance, passive eavesdropping attacks enabled through malicious code or
aftermarket extensions can lead to a complete loss of user privacy [4], while
more enhanced attacks like Denial of Service (DoS) can fully disrupt normal
vehicle functioning.

Intrusion detection systems (IDSs) for high-speed CAN networks address these
challenges, often leveraging hardware accelerators to analyse and identify
attacks like DoS in real-time [5]. Early rule-based IDS schemes used inherent
network and message properties like message frequency, voltage profiles of
ECUs, or timing of remote request frames as features to detect abnormal
operations [6, 7, 8]. However, the rule-based approach limits their ability to
scale and generalise towards new attack vectors, while also incurring
incremental storage and computational overheads for new attack modes/vectors.
More recently, both classical and deep-learning based IDSs have proved
extremely efficient in detecting CAN bus attacks [9, 10, 11, 12, 13]. In [12],
the authors use a deep-convolutional neural network (CNN) based on inception
net architecture as an IDS taking the CAN IDs as the input features to achieve
an average detection accuracy of 99% on both the DoS and Fuzzing attacks.
Alternative network architectures like Generational Adversarial Networks (GAN)
[14], temporal convolutional networks with global attention [15] and a
combination of CNN and long short-term memory (LSTM) as an unsupervised
learning network [16] have been explored in the literature to improve
detection accuracy, using only CAN IDs or complete data frame as inputs. Most
such models rely on GPU acceleration to deploy the IDS at an ECU. Dedicated
IDS-ECUs have also been proposed, where stacked models are executed on a
Raspberry Pi-3 device that acts as an IDS-ECU [17]. An intrusion prevention
ECU has also been proposed where an anomaly detection algorithm is executed on
a Raspberry Pi device (acting as a standalone ECU) to detect fuzzing and
spoofing attacks [18]. This approach drops any suspected messages from the
network by triggering an error response on the bus; however, this can cause a
large number of critical messages to be affected in case of false positives or
DoS attacks. In all the above cases, the scalable and generalisable nature of
ML models are deployed through specialised accelerators (GPUs) or dedicated
ECUs and can adapt to new threats post-deployment, making them more attractive
than traditional rule-based methods.

An alternative architecture approach is through the use of hybrid FPGAs as
ECUs, allowing a machine learning accelerator to be closely coupled with the
application(s) running on the ECU for a software-controlled hardware IDS.
Hybrid FPGA devices like Zynq Ultrascale+ combine automotive-grade ARM
processors and a large programmable fabric closely integrated on the same die,
allowing custom machine learning accelerators to be integrated on the same
die. Prior research has explored FPGA-based ECUs that enable custom
accelerators and specialised network capabilities to be tightly integrated as
a system-on-chip module while maintaining full AUTOSAR compliance [19] and for
the acceleration of complex algorithms like advanced driving-assistance system
(ADAS) using network interface extensions to augment ECU functionality and
security [20]. Machine learning on hybrid FPGAs has also shown tremendous
promise in applying deep learning in multiple applications through both vendor
tool flows and academic efforts [21, 22, 23, 24, 25, 26, 27].

In this paper, we present a lightweight deep-learning model for multi-attack
(MA) intrusion detection and its deployment using a hybrid-FPGA based ECU
architecture modelling a standard software-controlled accelerator framework in
ECUs. Our deep-CNN model is quantised and integrated using Xilinx’s Vitis-AI
flow using a Deep-learning Processing Unit (DPU) accelerator IP which is
controlled through Python APIs for our test cases [28]. We use an Ultra96-V2
board as a prototype ECU platform which uses a ZynqMP ZU3EG device running
Linux on the ARM cores with the PYNQ libraries. Our model is trained to
simultaneously detect DoS and fuzzing attacks using the public CAN attack
dataset that is widely used in literature. Our tests show that the tightly
coupled IDS achieves comparable detection accuracy as the state-of-the-art
detection accuracy with 25.08% reduction in the per-message processing latency
over state-of-the-art Raspberry Pi based approach while consuming just 2.0 W
in operation. We believe that the proposed IDS model and its deployment
architecture will pave the way for seamless integration of generalised IDS in
future vehicular network architectures.

##  II System Architecture

###  II-A Hybrid-FPGA based IDS capable ECU

Figure 1 shows an overview of the proposed hybrid-FPGA based distributed ECU
architecture on a Xilinx Zynq Ultrascale+ device. The Zynq Ultrascale+ device
integrates a quad-core ARM processor and a dual-core ARM real-time core within
the processing system (PS) section of the device, along with integrated
peripherals and interfaces. The programmable logic (PL) section of the device
enables the addition of specialised custom logic blocks or accelerators that
can be accessed using the Advanced eXtensible Interface (AXI) protocol from
the PS. In this architecture, standard ECU function(s) are mapped as software
tasks onto the ARM cores on the PS, on top of a standard operating system like
linux or a real-time operating system. The operating system provides relevant
drivers and APIs for accessing the PS peripherals and the PL accelerators,
abstracting away low-level details of these blocks to create an AUTOSAR
compliant architecture [19].

In our approach, we propose to use the integrated CAN interface on the PS to
handle the interfacing of ECU to the CAN bus, as shown in Figure 1. Any
received packets at the interface are read by the software tasks on the PS and
further processed based on the ECU’s task specification. In tandem, the IDS
task extracts relevant packet feature information from the received packet and
forwards it to the IDS accelerator through the integrated APIs. For this
evaluation, we have used the off-the-shelf DPU accelerator as the IDS
inference engine, which executes our TensorFlow ML model. DPU is an
instruction-based array of programmable processing engines (PEs) for
accelerating deep-learning inference on FPGAs. DPU is instantiated as a
memory-mapped IP, by configuring the number of PEs and associated resources
for configurable levels of parallelism based on resources available on the
target FPGA. The Vitis AI design flow compiles the TensorFlow model to
executable instructions for the DPU’s PE. Standard Vitis AI Runtime (VART)
APIs are used to configure and communicate with the DPU-based IDS engine. The
model uses interrupts to indicate the completion of tasks to the PS, allowing
the software tasks to run in a non-blocking fashion. This approach allows for
seamless integration of the IDS API calls with the other event/time-triggered
tasks to be executed by the ECU. For high-performance networks like Automotive
Ethernet, the IDS model can be configured to directly read the packets from
the interface (also in PL), eliminating the bottleneck caused by the software-
based data movement. In this case, the IDS task would enable the accelerator
operation at the startup phase of the network and would be interrupted only
when an anomaly is observed in the network traffic.

Figure 1: Proposed system architecture of the IDS-ECU. The ML model is
accelerated on the PL part of the FPGA device.

###  II-B Dataset & Training

We use the CAR Hacking dataset for training and testing our model [12, 29].
The dataset provides a labelled set of normal and attack messages which were
captured via the Onboard Diagnostic (OBD) port in an actual vehicle, with
attack messages injected in real-time. The dataset includes generalised DoS
and Fuzzing message injections (as well as other targeted attacks like gear-
ECU spoofing) allowing us to validate the detection accuracy across these
different attacks. DoS attacks flood the bus with a higher priority message
when injected while a fuzzing attack transmits messages with random IDs and
payloads at random intervals to disrupt the normal flow of messages. An
extract from the dataset is shown in table I, showing the CAN ID (ID field),
control field, data length code (DLC) and the actual data segment. We ignored
targeted attacks in the dataset since they are modelled by a constant message
sequence (specific ID and payload) and are not generalisable to other ECUs.

As our input feature, we use a window approach to capture adjacent sequences
of CAN messages, which are fed as input to the ML model to capture the
correlation between messages under different operating conditions. Through
design space exploration, we observed that a timeseries window of 4 successive
CAN-IDs provided optimal performance and was hence chosen for our
implementation. The dataset is pre-processed to mimic byte-wide binary CAN
data which can be read from the CAN interface controller. The ID bits are
extracted and stacked to form a block shape of {4,11}, which is then time-
spliced to generate the {2,2,11} dimension that is fed as input to the model
[13]. During implementation, this can be mapped as a FIFO style buffer
stacking sequential CAN IDs from the CAN controller (in PL) to form the input
tensor for the IDS.

To train the model, we used the adam optimizer with the binary cross-entropy
loss function. The learning rate was set at 0.0001 during the entire training
process to reduce any performance degradation when quantising the network
post-training as observed in [30]. We first train the model on DoS attack and
ensure optimal performance; subsequently, this trained model is trained on the
fuzzing dataset to improve generalisation across the two attack modes through
inductive transfer. This model file is then tested in both the DoS & Fuzzing
attacks to ensure there was no performance degradation. The model saves
intermediate results at each epoch allowing us to progressively track and
integrate early stopping in case of a significant drop in the accuracy. The
model was trained for 25 epochs with a batch size of 64 and the dataset was
split as 80:15:5 for training, validation and testing respectively for both
attacks to allow a sufficient amount of information for training and fine-
tuning the quantised network.

TABLE I: An extract from the open Car hacking dataset which is used for our
testing and evaluation.

Time | ID | DLC | Data
---|---|---|---
$\ldots$ |  |  |
1478198376.389427 | 0316 | 8 | 05,21,68,09,21,21,00,6f
1478198376.389636 | 018f | 8 | fe,5b,00,00,00,3c,00,00
1478198376.389864 | 0260 | 8 | 19,21,22,30,08,8e,6d,3a
$\ldots$ |  |  |

###  II-C Proposed Model

To determine the model configuration with minimal computational complexity, we
explored different network architectures that combine the layers supported by
the Vitis-AI flow. Each model was defined in TensorFlow (TF) using standard TF
functions and nodes. We evaluated different models comparing their inference
latency and detection accuracy across both attack datasets when mapped using
the Vitis-AI to determine our lightweight network model to implement. Our
chosen model is composed of 5 Conv2D layers implemented with 40, 80, 120, 160
& 200 filters at each layer, each filter having a dimension of 3x3. This is
followed by a Flatten and 2 Dense layers implemented with {32, 1} units
respectively. The final dense layer consists of sigmoid activation to predict
the probability of an attack message. In addition, batch normalisation and
dropout layers were used between the convolutional layers to prevent
overfitting and to improve learning efficiency during the inductive training
phase.

##  III Deployment and Experimental Results

Once the model was chosen, it was trained using an Nvidia RTX A6000 GPU using
the training split of the dataset(s), starting with the DoS attacks and
subsequently moving to the fuzzing attacks. The pre-quantisation accuracy is
then verified using the test split of the dataset and is shown in Table II.
The exported model file is passed to the Vitis-AI tool flow which quantises
the model, weights, biases, and activations to 8-bit precision, followed by an
optional post-quantisation fine-tuning step to account for any performance
degradation from the quantisation. In our case, the post-quantisation fine-
tuned model performance showed no visible degradation in inference performance
measuring using Precision, Recall, F1 score, False Negative Rates (FNR) &
False Positive Rate (FPR) metrics, as shown in Table II.

TABLE II: Inference accuracy metrics of the proposed MA-QCNN model, pre and
post quantisation on the two attacks.

Attack | Model | Precision | Recall | F1 | FPR | FNR
---|---|---|---|---|---|---
DoS | Pre-Q | 0.9992 | 1 | 0.9996 | 0.04% | 0%
MA-QCNN | 0.9992 | 1 | 0.9996 | 0.04% | 0%
Fuzzy | Pre-Q | 0.9966 | 0.9875 | 0.992 | 0.1% | 1.25%
MA-QCNN | 0.9966 | 0.9878 | 0.9922 | 0.1% | 1.22%

The model is then packaged as an ‘xmodel’ file which can be executed using a
DPU accelerator in the PL with the VART libraries installed on the OS running
on the PS. For our experiments, we use a petalinux based image for the
Ultra-96 V2 board on which the PYNQ and VART libraries were installed to use
the standard VART/PYNQ APIs. Subsequently, the bitstream corresponding to the
DPU accelerator is loaded to the PL from the OS (as an overlay). For our
experiments, we use the B1152 DPU for most of the performance measurements
(unless specified) with the DPU using a 150 MHz interface clock and 300 MHz
DSP core clock.

We compare our model performance in terms of inference accuracy metrics and
processing latency with the state-of-the-art IDSs and IPSs described in the
literature. Performance metrics such as Precision, Recall, F1 score, and FNR
are used to compare the inference accuracy of other models where reported and
use per-message latency to capture the true latency of the detection approach
starting from the arrival of a new CAN message at the interface. In case of
schemes where inference is performed on a block of CAN messages, this is
captured using the block size in the comparison. We further determine the
power consumption of the ECU while executing the IDS and compare it against
the execution of the full-precision model on a GPU. Note that none of the ML-
based IDS approaches in the literature report the actual power consumption of
their implementations; however, since most models are significantly more
complex than our lightweight model, we assume that our GPU implementation
would be a good baseline for comparison.

###  III-A Inference Accuracy

Table IV compares the proposed MA-QCNN with the state-of-the-art approaches
discussed in the research literature: GIDS [14], DCNN [12], iForest [18], MTH-
IDS [17], TCAN-IDS [15] & NovelADS [16]. In the case of MTH-IDS, the authors
only report an average accuracy, precision & recall of 99% across all attacks
which is identical to our model on the three metrics. In case of the DoS
attack, our quantised hardware accelerator performs almost identical to the
DCNN, TCAN-IDS, NovelADS, MLIDS across all metrics and has superior
performance than GIDS across all metrics. In case of the fuzzing attack, we
observe that DCNN, MLIDS & NovelADS perform marginally better, with 0.6%, 0.7%
& 0.8% better F1 scores respectively. Our model is almost identical to TCAN-
IDS in case of fuzzing attacks and offers a marginal improvement over iForest
and GIDS by 1.7% and 0.9% respectively (F1 scores). Note that in case of
competing techniques, the models are trained afresh for each attack, whereas
our model uses an induced learning approach during training and hence, uses
the same weights, biases and activation values for both attacks.

Table III shows the confusion matrix of MA-QCNN on the two attacks. We also
plot the receiver operating characteristic (ROC) curve for the two attacks
when using our model on the Ultra-96 V2, which is shown in Figure 2, and
further determine the area under curve (AUC) values for our model based on the
test datasets. We observe that our model achieves an AUC value $>$0.99 in both
attacks with a slightly lower performance in the fuzzing attack. We believe
that this could be improved by fine-tuning the training and post-quantisation
tuning steps.

TABLE III: Confusion matrix capturing the classification results of the MA-
QCNN (on DPU).

Attack | Message Type | Predicted Normal | Predicted Attack
---|---|---|---
DoS | True Normal | 33282 | 13
True Attack | 0 | 16705
Fuzzy | True Normal | 38784 | 38
True Attack | 136 | 11042

TABLE IV: Accuracy metric comparison (%) of our quantised FPGA accelerator
(MA-QCNN) against the reported literature.

Attack | Model | Precision | Recall | F1 | FNR
---|---|---|---|---|---
DoS | GIDS [14] | 96.8 | 99.6 | 98.1 | -
DCNN [12] | 100 | 99.89 | 99.95 | 0.13
MLIDS [31] | 99.9 | 100 | 99.9 | -
NovelADS [16] | 99.97 | 99.91 | 99.94 | -
TCAN-IDS [15] | 100 | 99.97 | 99.98 | -
| iForest [18] | - | - | - | -
| MA-QCNN (DPU) | 99.92 | 100 | 99.96 | 0
Fuzzing | GIDS [14] | 97.3 | 99.5 | 98.3 | -
DCNN [12] | 99.95 | 99.65 | 99.80 | 0.5
MLIDS [31] | 99.9 | 99.9 | 99.9 | -
NovelADS [16] | 99.99 | 100 | 100 | -
TCAN-IDS [15] | 99.96 | 99.89 | 99.22 | -
| iForest [18] | 95.07 | 99.93 | 97.44 | -
| MA-QCNN (DPU) | 99.66 | 98.78 | 99.22 | 1.22

Figure 2: ROC curve of the MA-QCNN on the two attacks.

###  III-B Latency, Power consumption & Resource utilisation

We measure the cold-start setup time of IDS to quantify the time from power on
to the model being ready. This is an important consideration since many
attacks could be initiated during the startup phase of the network and ECUs.
In case of our test platform, we observe this cold-start latency as 8 seconds
when averaged across multiple runs. It should be noted that our test platform
uses a slower boot device (SD image) and full Linux OS, whereas typical ECUs
rely on non-volatile flash memory and real-time operating system which could
reduce the cold-start latency by up to 6$\times$. Next, we evaluate the
processing latency of the inference datapath from the arrival of a new message
and compare it against competing approaches in the literature. As mentioned
before, some approaches perform block-based detection and hence require the
block of messages to be available before it is fed as input to the inference
engine. This reduces average inference latency per message leveraging the
parallelism on the GPUs. However, accumulating 64 messages (for example)
consumes nearly 8.3 ms on a 1 Mbps CAN network with 8-byte messages, which
should be factored into their worst-case detection latency when averaging. The
results are shown in table V, also capturing the execution platform and the
block size in case of block-based inference. We observe the lightweight model
on our approach takes 0.43 ms from the arrival of a message from the CAN
interface, when measuring the processing time to the arrival of the last
message of the block, achieving a 1.3x speedup over state-of-the-art CAN IDSs.
Finally, our approach does incur additional latency due to message looping
through the ECU, and could be reduced further using a CAN interface in the PL
feeding the DPU through a FIFO. Also, on a more capable hybrid FPGA, a larger
DPU could be configured to have multiple execution units and/or multiple
execution threads to reduce latency further for higher speed CAN networks at
the expense of slightly higher power consumption and device cost.

We also measure the power consumption of the IDS-ECU while performing
inference, using the PYNQ-PMBus package to monitor the power rails directly.
All power measurements are averaged over 100 inference runs, each evaluating
the test set lasting nearly 10 seconds. We observe that the Ultra-96 ECU
consumes 2 W when the CAN messages are been analysed using the DPU block on
the PL. In comparison, our lightweight floating-point model running on the
A6000 consumes 54 W while performing inference at similar CAN data rates,
which is 27$\times$ higher than the entire IDS-ECU on the Ultra-96 device.
Other IDS implementations in the literature do not report power measurements;
however, our lightweight model on the GPU should be a good baseline and more
complex models discussed in the literature should consume higher power during
inference on GPUs. Table VI shows the resource utilisation on the PL, with the
design consuming 45% DSP blocks and 57% BRAMs on the XCZU3EG device, leaving
sufficient resources for other custom logic. Thus, our IDS-ECU on a hybrid
FPGA like an Ultra-96 device offers an ideal mix of detection accuracy,
latency, power consumption, while offering software-driven control of IDS and
seamless GPU-like deployment with minimal change to the ECU application,
making it an ideal choice for in-vehicle deployment.

TABLE V: Per-message latency comparison against state of the art IDSs
(GPU/Raspberry Pi) reported in literature.

Models | Latency | Frames | Platform
---|---|---|---
MLIDS | 275 ms | per CAN frame | GTX Titan X
NovelADS | 128.7 ms | 100 CAN frames | Jetson Nano
GIDS | 5.89 ms | 64 CAN frames | GTX 1080
DCNN | 5 ms | 29 CAN frames | Tesla K80
TCAN-IDS | 3.4 ms | 64 CAN frames | Jetson AGX
MTH-IDS [17] | 0.574 ms | per CAN frame | Raspberry Pi 3
MA-QCNN (DPU) | 0.43 ms | per CAN frame | Zynq FPGA

TABLE VI: Resource utilization breakdown for PL Accelerator of our proposed
CAN IDS (XCZU3EG).

Node | LUT | FF | BRAM | DSP
---|---|---|---|---
DPU | 30726 | 48400 | 123 | 164
Overall | 32648 | 51347 | 123 | 164
(% usage) | (46.2%) | (36.3%) | (56.9%) | (45.5%)

##  IV Conclusion

In this paper, we present a lightweight multi-attack machine learning model
integrated as an IDS accelerator on a hybrid FPGA-based ECU architecture that
can successfully detect multiple attack models on a CAN bus. The proposed
lightweight model has comparable detection performance (almost identical or
within 1%) to the state of the art complex machine learning models for
detecting DoS and fuzzing attacks, almost all of which require dedicated GPU
acceleration or dedicated IDS-ECUs for line-rate detection. Our model also
achieves a 25% reduction in latency of processing per message while consuming
less than 2 W of active power when executing the model. We believe that the
proposed learning model and integration approach can be adapted for emerging
in-vehicle networks like Automotive Ethernet and also for enabling a
distributed intrusion detection framework for vehicular networks.

##  V Acknowledgement

This research was supported by grants from NVIDIA and utilised NVIDIA RTX
A6000 GPU.

## References

  * [1] R. B. GmbH, “CAN Specification, Version 2.0,” 1991.
  * [2] F. Hartwich et al., “CAN with flexible data-rate,” in Proc. iCC, pp. 1–9, Citeseer, 2012.
  * [3] C. Miller and C. Valasek, “Remote exploitation of an unaltered passenger vehicle,” Black Hat USA, vol. 2015, no. S 91, 2015.
  * [4] M. Enev, A. Takakuwa, K. Koscher, and T. Kohno, “Automobile Driver Fingerprinting.,” Proc. Privacy Enhancing Technology, vol. 2016, no. 1, pp. 34–50, 2016.
  * [5] C.-W. Lin and A. Sangiovanni-Vincentelli, “Cyber-security for the controller area network (CAN) communication protocol,” in Proc. Intl. Conf. on Cyber Security, pp. 1–7, IEEE, 2012.
  * [6] K.-T. Cho and K. G. Shin, “Fingerprinting electronic control units for vehicle intrusion detection,” in USENIX Security Symposium (USENIX Security), pp. 911–927, 2016.
  * [7] K.-T. Cho and K. G. Shin, “Viden: Attacker identification on in-vehicle networks,” in Proc. Conf. on Computer and Communications Security, pp. 1109–1123, 2017.
  * [8] H. Lee, S. H. Jeong, and H. K. Kim, “OTIDS: A novel intrusion detection system for in-vehicle network by using remote frame,” in Proc. Conf. on Privacy, Security and Trust (PST), pp. 57–5709, IEEE, 2017.
  * [9] S. N. Narayanan, S. Mittal, and A. Joshi, “Using data analytics to detect anomalous states in vehicles,” arXiv preprint arXiv:1512.08048, 2015.
  * [10] A. Alshammari, M. A. Zohdy, D. Debnath, and G. Corser, “Classification approach for intrusion detection in vehicle systems,” Wireless Engineering and Technology, vol. 9, no. 4, pp. 79–94, 2018.
  * [11] L. Yang, A. Moubayed, I. Hamieh, and A. Shami, “Tree-based intelligent intrusion detection system in internet of vehicles,” in Proc. IEEE Global Communications Conference (GLOBECOM), pp. 1–6, IEEE, 2019.
  * [12] H. M. Song, J. Woo, and H. K. Kim, “In-vehicle network intrusion detection using deep convolutional neural network,” Vehicular Communications, vol. 21, p. 100198, 2020.
  * [13] S. Tariq, S. Lee, and S. S. Woo, “CANTransfer: transfer learning based intrusion detection on a controller area network using convolutional LSTM network,” in Proc. ACM Sym. on Applied Computing, pp. 1048–1055, 2020\.
  * [14] E. Seo, H. M. Song, and H. K. Kim, “GIDS: GAN based intrusion detection system for in-vehicle network,” in Proc. Conf. on Privacy, Security and Trust (PST), pp. 1–6, IEEE, 2018.
  * [15] P. Cheng, K. Xu, S. Li, and M. Han, “TCAN-IDS: Intrusion Detection System for Internet of Vehicle Using Temporal Convolutional Attention Network,” Symmetry, vol. 14, no. 2, p. 310, 2022.
  * [16] K. Agrawal, T. Alladi, A. Agrawal, V. Chamola, and A. Benslimane, “NovelADS: A Novel Anomaly Detection System for Intra-Vehicular Networks,” IEEE Transactions on Intelligent Transportation Systems, 2022.
  * [17] L. Yang, A. Moubayed, and A. Shami, “MTH-IDS: A Multitiered Hybrid Intrusion Detection System for Internet of Vehicles,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 616–632, 2021.
  * [18] P. F. De Araujo-Filho, A. J. Pinheiro, G. Kaddoum, D. R. Campelo, and F. L. Soares, “An Efficient Intrusion Prevention System for CAN: Hindering Cyber-Attacks with a Low-Cost Platform,” IEEE Access, vol. 9, pp. 166855–166869, 2021.
  * [19] F. Fons and M. Fons, “FPGA-based automotive ECU design addresses AUTOSAR and ISO 26262 standards,” Xcell journal, vol. 78, p. 20, 2012.
  * [20] S. Shreejith and S. A. Fahmy, “Smart network interfaces for advanced automotive applications,” IEEE Micro, vol. 38, no. 2, pp. 72–80, 2018\.
  * [21] Xilinx, “Vitis AI User Guide,” 2021.
  * [22] Y. Gorbachev, M. Fedorov, I. Slavutin, A. Tugarev, M. Fatekhov, and Y. Tarkan, “Openvino deep learning workbench: Comprehensive analysis and tuning of neural networks inference,” in Proc. Intl. Conf. on Computer Vision Workshops, pp. 0–0, 2019.
  * [23] E. Wang, J. J. Davis, P. Y. Cheung, and G. A. Constantinides, “LUTNet: Rethinking inference in FPGA soft logic,” in Proc. Intl. Sym. on Field-Programmable Custom Computing Machines (FCCM), pp. 26–34, IEEE, 2019.
  * [24] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott, P. Leong, M. Jahre, and K. Vissers, “Finn: A framework for fast, scalable binarized neural network inference,” in Proc. Intl. Sym. on Field-Programmable Gate Arrays (FPGA), pp. 65–74, 2017.
  * [25] S. Soltani, Y. E. Sagduyu, R. Hasan, K. Davaslioglu, H. Deng, and T. Erpek, “Real-time and embedded deep learning on FPGA for RF signal classification,” in Proc. IEEE Military Communications Conference (MILCOM), pp. 1–6, IEEE, 2019.
  * [26] H. Nakahara, M. Shimoda, and S. Sato, “A demonstration of FPGA-based you only look once version2 (YOLOv2),” in Proc. Intl. Conf. on Field Programmable Logic and Applications (FPL), pp. 457–4571, IEEE, 2018.
  * [27] D. Mahajan, J. Park, E. Amaro, H. Sharma, A. Yazdanbakhsh, J. K. Kim, and H. Esmaeilzadeh, “Tabla: A unified template-based framework for accelerating statistical machine learning,” in Proc. Intl. Sym. on High Performance Computer Architecture (HPCA), pp. 14–26, IEEE, 2016.
  * [28] Xilinx, “Zynq DPU v3.2,” 2020.
  * [29] CAR Hacking Dataset, “https://ocslab.hksecurity.net/datasets/can-intrusion-dataset,” 2020.
  * [30] S. Wu, G. Li, F. Chen, and L. Shi, “Training and inference with integers in deep neural networks,” arXiv preprint arXiv:1802.04680, 2018.
  * [31] A. K. Desta, S. Ohira, I. Arai, and K. Fujikawa, “MLIDS: Handling Raw High-Dimensional CAN Bus Data Using Long Short-Term Memory Networks for Intrusion Detection in In-Vehicle Networks,” in Proc. Intl. Telecommunication Networks and Applications Conference (ITNAC), pp. 1–7, IEEE, 2020.