# Stochastic Deep Model Reference Adaptive Control

Girish Joshi and Girish Chowdhary *Supported by the Laboratory Directed
Research and Development program at Sandia National Laboratories, a multi-
mission laboratory managed and operated by National Technology and Engineering
Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell
International, Inc., for the U.S. Department of Energy’s National Nuclear
Security Administration under contract DE-NA-0003525.Authors are with
Coordinated Science Laboratory, University of Illinois, Urbana-Champaign, IL,
USA girishj2<EMAIL_ADDRESS>
###### Abstract

In this paper, we present a Stochastic Deep Neural Network-based Model
Reference Adaptive Control. Building on our work “Deep Model Reference
Adaptive Control”, we extend the controller capability by using Bayesian deep
neural networks (DNN) to represent uncertainties and model non-linearities.
Stochastic Deep Model Reference Adaptive Control uses a Lyapunov-based method
to adapt the output-layer weights of the DNN model in real-time, while a data-
driven supervised learning algorithm is used to update the inner-layers
parameters. This asynchronous network update ensures boundedness and
guaranteed tracking performance with a learning-based real-time feedback
controller. A Bayesian approach to DNN learning helped avoid over-fitting the
data and provide confidence intervals over the predictions. The controller’s
stochastic nature also ensured ”Induced Persistency of excitation,” leading to
convergence of the overall system signal.

##  I Introduction

Deep Model Reference Adaptive Control (DMRAC) is a control architecture that
seeks to learn a high-performance control policy in the presence of
significant model uncertainties while guaranteeing stability [1, 2, 3]. DMRAC
leverages deep neural networks, which have demonstrated their learning power
in many supervised learning applications in computer vision, natural language
processing, etc.[4]. However, using deep networks for learning control
policies can lead to system instability, as is well known in the adaptive
control literature [5, 6]. Furthermore, learning the parameters of a multi-
layer neural network can be computationally intensive and hence cannot be
updated in real-time. DMRAC addresses both issues through an asynchronous
weight update law for a deep neural network model of the uncertainty. DMRAC
temporally separates the slower learning of the inner layers, which form
features for the fast learning output layer weights. The outer-most layer
weights are adapted in real-time using Lyapunov-based Adaptive laws that have
known stability guarantees [5, 7, 8, 9]. This online learning of the last
layer ensures minimization of system tracking error and provides the necessary
robustness and guaranteed tracking performance. The slower learning of the
inner-layer weights of the DNN can be done over batches of data, either online
or offline, over judiciously collected data sets. The inner layer features can
thus be optimized for the final layer, which otherwise can be very difficult
to be designed [10]. Thereby, the DMRAC controller circumvents the entire
feature design problem for complex nonlinear control problems, much like in
other deep neural network applications.

DMRAC takes one critical step towards verifiable deep learning-based adaptive
control and improves the power of the learning models for adaptive control. In
[2] DMRAC was demonstrated to provide stable controller in the presence of
severe faults on a real quadrotor and showed to have a higher level of
performance than shallower methods such as Gaussian Process-based MRAC (GP-
MRAC) [11, 12]. The asynchronous framework used in DMRAC also shows much
promise for safety-critical applications and for other learning-based control
methods, such as Deep Reinforcement Learning (D-RL). However, DMRAC or
traditional DNN based learning methods cannot provide confidence bound on
their predictions, and are in general susceptible to the critical issue of
overfitting, leading to loss of generalizability which could be detrimental to
long-term learning. Also, it is well known that when applied to supervised
learning problems, deep networks are often incapable of correctly assessing
the uncertainty in the training data and often make overly confident decisions
about the correct class or prediction.

To address this issue of over-fitting of deep neural networks in estimating
model uncertainty in DMRAC, we introduce in this paper a Stochastic-DMRAC
(S-DMRAC) architecture using Bayesian neural networks. Apart from the benefits
of Bayesian methods in adding a natural regularization to feature learning, we
show that using finitely exciting training data, a desirable property of
”Induced Persistency of Excitation” of the features can be obtained. The
persistency of excitation property of the deep features enables the outer
layer weights to converge to their ideal values and therefore ensuring a
desirable property of convergence of all signals. We empirically demonstrate
the controller performance on the wing-rock system and compare the results
against the DMRAC controller. We show both empirically and theoretically, the
S-DMRAC network weights are well behaved and demonstrate boundedness given the
induced persistency of excitation property of stochastic deep features.

##  II Background

###  II-A Deep Networks and Feature spaces in machine learning

The key idea in machine learning is that a given function can be encoded with
weighted combinations of feature vector $\Phi\in\mathcal{F}$, s.t
$\Phi(x)=[\phi_{1}(x),\phi_{2}(x),...,\phi_{k}(x)]^{T}\in\mathbb{R}^{k}$, and
$W^{*}\in\mathbb{R}^{k\times m}$ a vector of ‘ideal’ weights s.t
$\|y(x)-W^{*^{T}}\Phi(x)\|_{\infty}<\epsilon(x)$. Instead of hand picking
features, or relying on polynomials, Fourier basis functions, comparison-type
features used in support vector machines [13, 14] or Gaussian Processes [15],
DNNs utilize composite functions of features arranged in a directed acyclic
graphs, i.e.
$\Phi(x)=\phi_{n}(\theta_{n-1},\phi_{n-1}(\theta_{n-2},\phi_{n-2}(...))))$
where $\theta_{i}$’s are the layer weights. The universal approximation
property of the DNN with commonly used feature functions such as sigmoidal,
tanh, and RELU is proved in the work by Hornik’s [16] and shown empirically to
be true by recent results [17, 18, 19]. Hornik et al. argued the network with
at least one hidden layer (also called Single Hidden Layer (SHL) network) to
be a universal approximator. However, empirical results show that the networks
with more hidden layers show better generalization capability in approximating
complex function. While the theoretical reasons behind better generalization
ability of DNN are still being investigated [20, 21], for the purpose of this
paper, we will assume that it is indeed true, and focus our efforts on
designing a practical and stable control scheme using DNNs.

##  III System Description

This section discusses the formulation of Deep Model Reference Adaptive
Control [1]. We consider the following system with uncertainty $\Delta(x)$:

$\dot{x}(t)=Ax(t)+B(u(t)+\Delta(x)).$ (1)

where $x(t)\in\mathbb{R}^{n}$, $t\geqslant 0$ is the state vector,
$u(t)\in\mathbb{R}^{m}$, $t\geqslant 0$ is the control input,
$A\in\mathbb{R}^{n\times n}$, $B\in\mathbb{R}^{n\times m}$ are known system
matrices and we assume the pair $(A,B)$ is controllable. The term
$\Delta(x):\mathbb{R}^{n}\to\mathbb{R}^{m}$ is matched system uncertainty and
be Lipschitz continuous in $x(t)\in\mathcal{D}_{x}$. Let
$\mathcal{D}_{x}\subset\mathbb{R}^{n}$ be a compact set and the control $u(t)$
is assumed to belong to a set of admissible control inputs of measurable and
bounded functions, ensuring the existence and uniqueness of the solution to
(1).

The reference model is assumed to be linear and therefore the desired
transient and steady-state performance is defined by selecting the system
eigenvalues in the negative half plane. The desired closed-loop response of
the reference system is given by

$\dot{x}_{m}(t)=A_{m}x_{m}(t)+B_{m}r(t).$ (2)

where $x_{m}(t)\in\mathcal{D}_{x}\subset\mathbb{R}^{n}$ and
$A_{m}\in\mathbb{R}^{n\times n}$ is Hurwitz and $B_{m}\in\mathbb{R}^{n\times
r}$. Furthermore, the command $r(t)\in\mathbb{R}^{r}$ denotes a bounded, piece
wise continuous, reference signal and we assume the reference model (2) is
bounded input-bounded output (BIBO) stable [22].

###  III-A Deep MRAC: Total Controller

The aim is to construct a feedback law $u(t)$, $t\geqslant 0$, such that the
state of the uncertain dynamical system (1) asymptotically tracks the state of
the reference model (2) despite the presence of matched uncertainty.

A tracking control law consisting of linear feedback term $u_{pd}=k_{x}x(t)$,
a linear feed-forward term $u_{crm}=k_{r}r(t)$ and an adaptive term
$u_{ad}(t)$ form the total controller

$u=u_{pd}+u_{crm}-u_{ad}.$ (3)

The baseline full state feedback and feed-forward controller is designed to
satisfy the matching conditions such that $A_{m}=A-Bk_{x}$ and $B_{m}=Bk_{r}$.
For the adaptive controller ideally we want $u_{ad}(t)=\Delta(x(t))$.

The true uncertainty $\Delta(x)$ in unknown, but it is assumed to be
continuous over a compact domain $\mathcal{D}_{x}\subset\mathbb{R}^{n}$. A
Deep Neural Networks (DNN) have been widely used to represent a function when
the basis vector is not known. Using DNNs, a non linearly parameterized
network estimate of the uncertainty can be written as
$f_{\theta}(x)\triangleq\theta_{n}^{T}\Phi(x)$, where
$\theta_{n}\in\mathbb{R}^{k\times m}$ are network weights for the final layer
and
$\Phi(x)=\phi_{n}(\theta_{n-1},\phi_{n-1}(\theta_{n-2},\phi_{n-2}(...))))$, is
a $k$ dimensional feature vector which is function of inner layer weights,
activations and inputs. The basis vector
$\Phi(x)\in\mathcal{F}:\mathbb{R}^{n}\to\mathbb{R}^{k}$ is considered to be
Lipschitz continuous to ensure the existence and uniqueness of the solution
(1). Therefore the adaptive part of the controller can be written as

$u_{ad}(t)=W^{T}\Phi(x(t)).$ (4)

Where the network weights $\boldsymbol{\theta}$ are updated minimizing a loss
function $\ell(\boldsymbol{Z_{i},\theta})$ over a batch of carefully collected
locally exciting data $(\boldsymbol{Z_{i}})$ using Model Reference Generative
Network (MRGeN) [11, 1].

$\boldsymbol{\theta}=\arg\max_{\theta}\left(-\frac{1}{M}\sum_{i=1}^{M}\ell(\boldsymbol{Z_{i},\theta})\right).$

And $W$ are the fast updated weights, which are learned according to MRAC
rule.

$\dot{W}=-\Gamma\Phi(x(t))e(t)^{T}pB.$

The weight $W$ will replace the final layer network weights $\theta_{n}$ in
$f_{\theta}$ to form the adaptive control element (4) in the total controller
(3).

Developing on this idea of a Lyapunov-based method for adaptation laws of the
output-layer weights of a DNN model in real-time while a data-driven
supervised learning algorithm is used to update the inner-layer weights of the
DNN; we present in this paper a stochastic deep MRAC architecture using
Bayesian learning (S-DMRAC).

##  IV Bayesian Deep Neural Networks

Bayesian Neural Networks(BNN) are comprised of a Probabilistic Model and a
Neural Network. The intent of such a design is to combine the strengths of
Neural Networks and Stochastic modeling. Neural Networks exhibit universal
approximation property over continuous functions [23]. Probabilistic models
allows us to combine prior and likelihood to produce probabilistic guarantees
on the model predictions through generating distribution over the parameters
learned from the observations. Therefore the network can specify the
confidence interval over the prediction. Thus BNNs are a unique combination of
neural network and stochastic models.

The layer weights in the Bayesian neural networks are represented by
probability distributions over possible values, rather than a single fixed
value as in the traditional neural networks. The trained deep features and
therefore adaptive element will be a stochastic quantity. The amount of
perturbation each weight exhibits is learnt in a way that coherently explains
variability in the training data. Thus instead of training a single network,
the Bayesian neural network trains an ensemble of networks, where each network
has its weights drawn from a shared, learnt probability distribution.

A neural network can be viewed as a probabilistic model
$P(\mathcal{Y}|\mathcal{X},\boldsymbol{\theta})$. We assume $y$ is a
continuous variable and $P(\mathcal{Y}|\mathcal{X},\boldsymbol{\theta})$ is
gaussian likelihood. Give the data
$Z^{M}=\\{\\{x_{i},y_{i}\\}\in\mathcal{X}\times\mathcal{Y}\\}_{i=1}^{M}$. The
total likelihood can be written as,

$p(Z|\boldsymbol{\theta})=\prod_{i}\left(p(y_{i}|x_{i},\boldsymbol{\theta})\right),$
(5)

which is function of network parameters $\boldsymbol{\theta}$. Assuming a
gaussian likelihood, performing maximization of negative log likelihood leads
to least square regression which can suffer over-fitting. Using a prior belief
on the parameters $p(\boldsymbol{\theta})$ and likelihood of the data given a
deep neural network model, we can obtain a updated posterior belief on the
parameter using Bayes rule as follows,

$p(\boldsymbol{\theta}|Z)=\frac{p(Z|\boldsymbol{\theta})p(\boldsymbol{\theta})}{\int_{\theta}p(Z|\mathbf{\theta})p(\boldsymbol{\theta})d\boldsymbol{\theta}}.$
(6)

Maximizing the posterior $p(\boldsymbol{\theta}|Z)$ leads to Maximum-a-
posterior (MAP) estimate of parameter $\boldsymbol{\theta}$. Computing the MAP
estimate has a regularizing effect and can prevent overfitting. The
optimization objectives here are the same as maximizing likelihood plus a
regularization term coming from the log prior. If we had a full posterior
distribution over parameters we could make predictions that take weight
uncertainty into account by marginalizing the parameters as follows,

$p(y|x,Z)=\int_{\theta}p(y|x,\boldsymbol{\theta})p(\boldsymbol{\theta}|Z)d\boldsymbol{\theta}.$
(7)

This is equivalent to averaging predictions from an ensemble of neural
networks weighted by the posterior probabilities of their parameters
$\boldsymbol{\theta}$.

##  V Adaptive Control Using Bayesian Deep Neural Networks

###  V-A Bayesian Feature Learning: Variational Inference

This section provides the details of the DNN training over data samples
observed over n-dimensional input subspace
$x(t)\in\mathcal{X}\in\mathbb{R}^{n}$ and m-dimensional target subspace
$y\in\mathcal{Y}\in\mathbb{R}^{m}$. The sample set is denoted as $Z$ where
$Z\in\mathcal{X}\times\mathcal{Y}$.

We are interested in the function approximation task for DNN. Let the function
$f_{\boldsymbol{\theta}}(x)=\theta^{T}_{n}\phi_{n}(\theta_{n-1},\phi_{n-1}(\theta_{n-2},\phi_{n-2}(...))))$
s.t $f_{\boldsymbol{\theta}}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ be the network
approximating the model uncertainty with parameters
$\boldsymbol{\theta}\in\boldsymbol{\Theta}$, where $\boldsymbol{\Theta}$ is
the space of parameters. We assume a training data buffer $\mathcal{B}$ can
store $p_{max}$ training examples, such that the set
$Z^{p_{max}}=\\{Z_{i}|Z_{i}\in\mathcal{Z}\\}_{i=1}^{p_{max}}=\\{(x_{i},y_{i})\in\mathcal{X}\times\mathcal{Y}\\}_{i=1}^{p_{max}}$.
A batch of samples can be drawn independently from the buffer $\mathcal{B}$
over probability distribution $p$ for DNN training.

Unlike the conventional DNN training where the true target values
$y\in\mathcal{Y}$ are available for every input $x\in\mathcal{X}$, in S-DMRAC
true system uncertainties as the labeled targets are not available for the
network training. We use the part of the network itself (the last layer) with
pointwise weight updated according to MRAC-rule as the generative model for
the data. The S-DMRAC uncertainty estimates
$y_{i}={W}^{T}\Phi(x_{i},\theta_{1},\theta_{2},\ldots\theta_{n-1})$ along with
inputs $x_{i}$ make the training data set
$Z^{p_{max}}=\\{x_{i},y_{i}\\}^{p_{max}}$. We use kernel Independence test
[24] to qualify a data point to be stored in the buffer, to ensure local
persistently exciting data is collected. The main purpose of DNN in the
adaptive network is to extract relevant features of the system uncertainties,
which otherwise is very tedious to obtain with unstructured uncertainty and
unknown bounds of the domain of operation. For the details of generative
network and data qualification methods refer [1, 11] and reference therein.

Unlike Gaussian Processes [10], analytical solution for posterior
$p(\boldsymbol{\theta}|Z)$ for a multi-layer neural network is intractable. We
therefore approximate the true posterior with a variational distribution
$q(\boldsymbol{\theta}|\boldsymbol{\zeta})$. The variational distribution is a
known function parameterized by variational parameters $\boldsymbol{\zeta}$
which are estimated online. The variational parameters are estimated online by
minimizing the Kullback-Leibler (KL) divergence between variational
distribution $q(\boldsymbol{\theta}|\boldsymbol{\zeta})$ and true posterior
$p(\boldsymbol{\theta}|Z)$. The KL objective can be written as,

$\displaystyle
KL\left(q(\boldsymbol{\theta}|\boldsymbol{\zeta})\|p(\boldsymbol{\theta}|Z)\right)$
$\displaystyle=$ $\displaystyle\int
q(\boldsymbol{\theta}|\boldsymbol{\zeta})\log\left(\frac{q(\boldsymbol{\theta}|\boldsymbol{\zeta})}{p(\boldsymbol{\theta}|Z)}\right)d\boldsymbol{\theta},$
(8) $\displaystyle=$
$\displaystyle\mathbb{E}_{q(\boldsymbol{\theta}|\boldsymbol{\zeta})}\log\left(\frac{q(\boldsymbol{\theta}|\boldsymbol{\zeta})}{p(\boldsymbol{\theta}|Z)}\right).$

The KL-divergence objective can be written as,

$KL\left(q(\boldsymbol{\theta}|\boldsymbol{\zeta})\|p(\boldsymbol{\theta}|Z)\right)=-\mathcal{L}(Z,\boldsymbol{\theta,\zeta})+\log(p(Z)).$
(9)

The term $\mathcal{L}(Z,\boldsymbol{\theta,\zeta})$ is known as evidence lower
bound since it defines lower bound on the evidence term $p(Z)$ as follows,

$\mathcal{L}(Z,\boldsymbol{\theta,\zeta})=\log(p(Z))-KL\left(q(\boldsymbol{\theta}|\boldsymbol{\zeta})\|p(\boldsymbol{\theta}|Z)\right).$
(10)

Since the KL-divergence term is always positive semi-definite we can write,

$\mathcal{L}(Z,\boldsymbol{\theta,\zeta})\leq\log(p(Z)).$ (11)

Therefore, the KL divergence between the variational distribution
$q(\boldsymbol{\theta}|\boldsymbol{\zeta})$ and the true posterior
$p(\boldsymbol{\theta}|Z)$ is minimized by maximizing the evidence lower
bound,

$\boldsymbol{\zeta}\xleftarrow[]{}arg\max_{\boldsymbol{\zeta}}\mathcal{L}(Z,\boldsymbol{\theta,\zeta}).$
(12)

And the deep feature parameters are drawn from this variational distribution
as follows,

$\boldsymbol{\theta}\sim q(\boldsymbol{\theta}|\boldsymbol{\zeta)}.$ (13)

We use Bayes by Backprop [25] algorithm to learn the latent parameters of
variational distribution using stochastic gradient descent algorithm.

###  V-B S-DMRAC Parameter Update using Bayesian Deep Features

In [1] we have presented the DMRAC update rule and its stability
characteristics using the standard deep features. In this paper we will extend
the results to stochastic deep features. The deep feature vector is defined as
$\Phi^{\sigma}_{n}(x)=f_{\boldsymbol{\theta}}\backslash\theta_{n}$, where
$f_{\boldsymbol{\theta}}$ is the entire network and $\sigma$ is the switching
signal. Due to stochastic nature of the network, the feature vector
$\Phi^{\sigma}_{n}(x)$ can be considered as a draw over multivariate normal
distribution

$\Phi^{\sigma}_{n}(x)\sim\mathcal{N}\left(\bar{\Phi}^{\sigma}_{n}(x),\left[G_{\sigma}(x)G_{\sigma}(x)^{T}\right]|\boldsymbol{\theta}\right).$
(14)

Where $\bar{\Phi}^{\sigma}_{n}(x),\left[G_{\sigma}(x)G_{\sigma}(x)^{T}\right]$
are mean and covariance for the feature.

Using the outer layer weights $W$, updated according to MRAC rule and the deep
features from Bayesian deep neural network, we can write adaptive element
$u_{ad}(t)$ in the total controller as,

$u_{ad}(t)=W^{T}\Phi^{\sigma}_{n}(x(t)).$ (15)

We will show using stochastic stability theory that the following weight
update rule leads to mean square uniform ultimate boundedness almost surely.
The weight update rule for the final layer weights using MRAC rule is given as

$\dot{W}(t)=-\Gamma\bar{\Phi}^{\sigma}_{n}(x)e(t)^{T}PB.$ (16)

Where $\bar{\Phi}^{\sigma}_{n}(x)$ is mean of the features given as

$\bar{\Phi}^{\sigma}_{n}(x)=\mathbb{E}_{q(\boldsymbol{\theta}|\boldsymbol{\zeta})}\left(\Phi^{\sigma}_{n}(x)\right).$
(17)

However in case the true expectation is intractable, we can estimate the
expectation empirically as follows,

$\hat{\Phi}^{\sigma}_{n}(x)=\frac{1}{N}\sum_{i=1}^{N}\left(\Phi^{\sigma}_{n,i}(x)\right).$
(18)

Where $\Phi^{\sigma}_{n,i}(x)$ is the $i^{th}$ draw from distribution (38) for
every given $x(t)$. Therefore we can modify the weight update law as,

$\dot{W}=proj\left(-\Gamma\left(\frac{1}{N}\sum_{i=1}^{N}\left(\Phi_{n,i}^{\sigma}(x)\right)\right)e(t)^{T}PB,W\right).$
(19)

However in the further section we will show that using a stochastic feature
presents the property of Induced Persistency of Excitation, which ensures the
online updated weights converge to true weights neighborhood for all switching
signal “$\sigma$”.

##  VI Stability Analysis for Stochastic-DMRAC

In this section we will introduce the stochastic stability proof for S-DMRAC
controller. Using the above stochastic adaptive element (15) in the total
controller (3) we can rewrite the system dynamics (1) as follows,

$\dot{x}(t)=Ax(t)+B(-k_{x}x(t)+k_{r}r(t)-u_{ad}+f(x)).$ (20)

Using the model matching condition we can write,

$\dot{x}(t)=A_{m}x(t)+B_{m}r(t)+B(f(x)-u_{ad}).$ (21)

Using the expression for stochastic adaptive element we can rewrite the above
expression as

$\dot{x}(t)=A_{m}x(t)+B_{m}r(t)+B\left(f(x)-W^{T}\Phi_{n}^{\sigma}(x)\right).$
(22)

Since we know the the feature vector instance is draw over the distribution of
parameters $\boldsymbol{\theta}\sim
q(\boldsymbol{\theta}|\boldsymbol{\zeta})$, we can write the random draw of
the feature vector as

$\Phi_{n}^{\sigma}(x)=\bar{\Phi}_{n}^{\sigma}+G_{\sigma}\xi,$ (23)

where $\bar{\Phi}_{n}^{\sigma}\in\mathbb{R}^{k}$ is the mean of feature
distribution and $G_{\sigma}\in\mathbb{R}^{k}$ is variance and
$\xi\sim\mathcal{N}(0,I)$. Using the feature instance we can write the closed
loop system as

$\displaystyle\dot{x}(t)$ $\displaystyle=$ $\displaystyle A_{m}x(t)+B_{m}r(t)$
(24) $\displaystyle+$ $\displaystyle
B\left(f(x)-W^{T}\bar{\Phi}_{n}^{\sigma}(x)-W^{T}G_{\sigma}\xi\right).$

We assume for every $\bar{\Phi}_{n}^{\sigma}$ that there exists an ideal
weight $W^{*}$, such that,

$f(x)=W^{*T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon_{m}^{\sigma}(x).$ (25)

Where $\epsilon_{m}^{\sigma}(x)=\Delta(x)-W^{*T}\bar{\Phi}_{n}^{\sigma}(x)$ is
network approximation error due to universal approximation theorem, and this
error can be bounded as,

$\bar{\epsilon}^{\sigma}=\sup_{x\in\mathcal{D}}\left\|f(x)-W^{*T}\bar{\Phi}_{n}^{\sigma}(x)\right\|.$
(26)

Using (2), we can write the error dynamics as,

$\dot{e}(t)=A_{m}e(t)+B\left(\tilde{W}^{T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon_{m}^{\sigma}(x)\right)+BW^{T}G_{\sigma}\xi.$
(27)

Where $\tilde{W}=W^{*}-W$. We can write the above tracking error dynamics as
diffusion process as follows,

$\displaystyle de(t)$ $\displaystyle=$ $\displaystyle
A_{rm}e(t)dt+B\left(\tilde{W}^{T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon_{m}^{\sigma}(x)\right)dt$
(28) $\displaystyle+$ $\displaystyle BW^{T}G_{\sigma}d\xi(t),$

where $d\xi(t)$ is zero mean Wiener process.

###### Theorem 1

Consider the system in (1), the control law of (3), and assume that the
uncertainty $f(x):\mathbb{R}^{n}\to\mathbb{R}^{m}$ is bounded and Lipschitz
continuous on a compact set $x(t)\in\mathcal{D}_{x}$. Let the reference signal
$r(t)$ is such that the state $x_{m}(t)$ of the bounded input bounded output
reference model (2) remains bounded in the compact ball
$\mathcal{B}_{m}=\\{x_{m}:\|x_{m}(t)\|\leq\mathcal{B_{\delta}}\\}$, then the
adaptive control $u_{ad}=W^{T}\Phi^{\sigma}_{n}(x)$, guarantee that the system
is mean square uniformly ultimately bounded in probability a.s.

###### Proof:

Let,

$V(e_{t},\tilde{W}(t))=\frac{1}{2}e_{t}^{T}Pe_{t}+\frac{1}{2}tr\left(\tilde{W}(t)^{T}\Gamma^{-1}\tilde{W}(t)\right),$
(29)

be stochastic lyapunov candidate. The error term $e_{t}$ is a realization of
drift diffusion process, we assume $t$ as index and not function of time,
whereas $W(t)$ is a deterministic process $W(t)$ is a function of time. Note
that the lyapunov candidate is bounded above below by

$\underline{\zeta}\left(e_{t},\tilde{W}\right)\leq
V(e_{t},\tilde{W})\leq\bar{\zeta}\left(e_{t},\tilde{W}\right),$ (30)

where
$\underline{\zeta}\left(e_{t},\tilde{W}\right)=\frac{1}{2}\underline{\lambda}(P)\|e_{t}\|^{2}+\frac{1}{2}\underline{\lambda}\left(\Gamma^{-1}\right)\|\tilde{W}\|_{F}^{2}$
and
$\bar{\zeta}\left(e_{t},\tilde{W}\right)=\frac{1}{2}\bar{\lambda}(P)\|e_{t}\|^{2}+\frac{1}{2}\bar{\lambda}\left(\Gamma^{-1}\right)\|\tilde{W}\|_{F}^{2}$
and $\underline{\lambda}$ and $\bar{\lambda}$ is the minimum and maximum Eigen
value operator and $\|\|_{F}$ is the Frobenius norm.

The Ito derivative of the Lyapunov candidate along the error dynamics (28) for
$\sigma^{th}$ system,

$\displaystyle\mathcal{L}V\left(e_{t},\tilde{W}\right)=-\dot{W}\Gamma^{-1}\tilde{W}-\frac{1}{2}e^{T}Qe+\tilde{W}\bar{\phi}_{n}^{\sigma}(x)e^{T}PB$
$\displaystyle+e^{T}PB\epsilon_{m}^{\sigma}(z)+\frac{1}{2}[(BW^{T}G_{\sigma})(BW^{T}G_{\sigma})^{T}P].$
(31)

Using the weight update law (16), we can simplify the above expression as
follows

$\displaystyle\mathcal{L}V\left(e_{t},\tilde{W}\right)$ $\displaystyle=$
$\displaystyle\frac{1}{2}e^{T}Qe+e^{T}PB\epsilon_{m}^{\sigma}(z)$ (32)
$\displaystyle+$
$\displaystyle\frac{1}{2}[(BW^{T}G_{\sigma})(BW^{T}G_{\sigma})^{T}P].$

Let $c_{1}=1/2\|P\|\|W\|_{F}^{2}\|BG_{\sigma}\|^{2}$ and $c_{2}=\|PB\|$, then

$\displaystyle\mathcal{L}V\left(e_{t},\tilde{W}\right)$ $\displaystyle\leq$
$\displaystyle-\frac{1}{2}\lambda_{min}(Q)\|e\|^{2}+c_{2}\|e\|\|\epsilon_{m}^{\sigma}(z)\|+c_{1}$

We know
$\sup_{x\in\mathcal{D}}\|\epsilon_{m}^{\sigma}(x)\|\leq\bar{\epsilon}^{\sigma}$,
and using the projection operator the outer layer weights $W(t)$ are bounded
$\|W\|\leq\mathcal{W}_{b}$. Using this bound in above expression we can write,

$\mathcal{L}V\left(e_{t},\tilde{W}\right)\leq-\frac{1}{2}\lambda_{min}(Q)\|e\|^{2}+c_{2}\|e\|\bar{\epsilon}^{\sigma}+c_{1}.$
(33)

Let $c_{5}^{\sigma}=c_{2}\bar{\epsilon}^{\sigma}$.

We can define an set outside which the
$\mathcal{L}V\left(e_{t},\tilde{W}\right)\leq 0$

$\Theta^{\sigma}=\left\\{\|e\|\geq\frac{c_{5}^{\sigma}+\sqrt{\left(c_{5}^{\sigma}\right)^{2}+2*\lambda_{min}(Q)c_{1}}}{\lambda_{min}(Q)}\right\\}.$
(34)

Therefore, outside the set $\Theta^{\sigma}$,
$\mathcal{L}V\left(e_{t},\tilde{W}\right)\leq 0$ a.s. Using BIBO property of
reference model (2), when $r(t)$ is bounded, $x_{m}(t)$ remains bounded within
$B_{m}$. Solution to (1) $x(t)\in\mathcal{D}_{x}$ a.s. Since this is true for
all $\sigma$, and because Algorithm guarantees that $\sigma$ does not switch
arbitrarily fast,(28) is mean square uniformly ultimately bounded inside set
$\Theta^{\sigma}$ a.s. ∎

##  VII Persistency of Excitation for S-DMRAC

In multi-layer architecture for uncertainty approximation, it is challenging
to ensure the Persistency of Excitation, which result in non P.E deep features
and therefore non convergence of the outer-layer weights. One straightforward
way to remedy this problem is to inject exogenous perturbations into the
dynamics of the gradient descent algorithm so that even the parameters in the
hidden layers receive persistent excitation during training. The traditional
procedure to introduce robustness for linear models is the addition of
regularization term into the loss function. However the regularization cannot
be re-framed as a method to ensure persistent excitation but can bound model
parameters.

However, we will show that Bayesian-deep neural network is a natural way of
adding regularization and the stochastic nature of network weights add
randomness in the network prediction and therefore lead to desirable property
of induced persistency of excitation.

The KL-objective for posterior inference given data in training Bayesian-deep
neural network is follows,

$\mathcal{L}(Z,\boldsymbol{\theta,\zeta})=\mathbb{E}_{q(\boldsymbol{\theta}|\boldsymbol{\zeta})}\left(\log\left(p(Z|\boldsymbol{\theta})\right)\right)-KL\left(q(\boldsymbol{\theta}|\boldsymbol{\zeta})\|p(\boldsymbol{\theta})\right).$
(35)

While maximizing $\mathcal{L}(Z,\boldsymbol{\theta,\zeta})$, we are maximizing
the model likelihood given data $Z$ and minimizing KL-divergence between
posterior and prior on parameters, which is akin to adding regularization.

Uncertainty in predictions arise from two sources; uncertainty in weights
called Epistemic uncertainty and the variability coming from the finite
excitation in training data know as Aleatoric uncertainty. Epsitemic
uncertainty can grows smaller, if we have more training data. Consequently,
epistemic uncertainty is higher in regions of no or little training data and
lower in regions of more training data. Epistemic uncertainty is covered by
the variational posterior distribution. Aleatoric uncertainty is covered by
the probability distribution used to define the likelihood function. The
Aleatoric uncertainty cannot be reduced if we get more data, it is the
inherent variability in the training data. The total uncertainty or randomness
in the feature vector due to Epistemic and Aleatoric uncertainty, lead to
favorable property of induced persistency of excitation of $\Phi^{\sigma}_{n}$

###### Theorem 2

Let the Bayesian deep network $f_{\boldsymbol{\theta}}$ be trained over data
$Z^{p_{max}}=\\{\\{x_{i},y_{i}\\}\in\mathcal{X}\times\mathcal{Y}\\}_{i=1}^{p_{max}}$,
such that data is collected using kernel independence test. Then the features
$\Phi^{\sigma}_{n}=f_{\boldsymbol{\theta}}\backslash\theta_{n}$ is
persistently exciting.

###### Proof:

We use autocovariance argument to prove the that the stochastic features
$\Phi^{\sigma}_{n}$ are persistently exciting i.e iff

$R_{\Phi^{\sigma}_{n}}(0)>0,$ (36)

Implies the integral for any given $t\geq t_{0}$ and over any interval
$[t,t+T]$ for any $T>0$ satisfies,

$\int_{t}^{t+T}\Phi^{\sigma}_{n}(\tau)\Phi^{\sigma}_{n}(\tau)^{T}d\tau\geq\gamma
I.$ (37)

To prove the fact that deep features of S-DMRAC are PE, we show that the
autocovariance $R_{\Phi^{\sigma}_{n}}(0)$ exists and
$R_{\Phi^{\sigma}_{n}}(0)>0$.

Since the stochastic feature vector are realization of distribution defined as

$\Phi^{\sigma}_{n}(x)\sim\mathcal{N}\left(\bar{\Phi}^{\sigma}_{n}(x),\left[G_{\sigma}(x)G_{\sigma}(x)^{T}\right]|\boldsymbol{\theta}\right).$
(38)

Therefore the autocovariance of feature vector $\Phi^{\sigma}_{n}$ can be
defined as,

$\displaystyle
R_{q_{\zeta}(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}(x)|x\right)=\mathbb{E}_{q_{\theta}}\left[\left(\Phi^{\sigma}_{n}-\mathbb{E}(\Phi^{\sigma}_{n})\right)^{2}\right]$
(39) $\displaystyle=$
$\displaystyle\mathbb{E}_{q_{\theta}}\left(\Phi^{\sigma}_{n}\Phi^{\sigma_{n}^{T}}\right)-\mathbb{E}_{q_{\theta}}\left(\Phi^{\sigma}_{n}\right)\mathbb{E}_{q_{\theta}}\left(\Phi^{\sigma}_{n}\right)^{T},$

Where $q_{\boldsymbol{\zeta}}(\Phi^{\sigma}_{n}|x,\theta)$ is the variational
predictive posterior distribution defined as,

$\displaystyle q_{\boldsymbol{\zeta}}(\Phi^{\sigma}_{n}|x,\theta)$
$\displaystyle=$ $\displaystyle
p(\Phi^{\sigma}_{n}|x,\theta)q(\boldsymbol{\theta}|\boldsymbol{\zeta}).$ (40)

and $p(\Phi^{\sigma}_{n}|x,\theta)$ is the posterior distribution on feature
given a input $x$ and network parameters $\boldsymbol{\theta}$

$p(\Phi^{\sigma}_{n}|x,\theta)=\mathcal{N}\left(\bar{\Phi}^{\sigma}_{n}(x),\left[G_{\sigma}(x)G_{\sigma}(x)^{T}\right]|\boldsymbol{\theta}\right).$
(41)

Using the above definitions of conditional distributions, the autocovariance
term (39) can be decomposed into Aleatoric and Epistemic uncertainties as
follows [26]:

$\displaystyle
R_{q}\left(\Phi^{\sigma}_{n}(x)|x\right)\mathbb{E}_{q}\left(\Phi^{\sigma}_{n}\Phi^{\sigma_{n}^{T}}\right)-\mathbb{E}_{q}\left(\Phi^{\sigma}_{n}\right)\mathbb{E}_{q}\left(\Phi^{\sigma}_{n}\right)^{T}$
$\displaystyle\Rightarrow\int_{\Theta}diag\left(\mathbb{E}_{p(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}\right)\right)$
$\displaystyle-\mathbb{E}_{p(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}\right)\mathbb{E}_{p(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}\right)^{T}q_{\boldsymbol{\zeta}}(\boldsymbol{\theta})d\boldsymbol{\theta}$
$\displaystyle+\int_{\Theta}\left[\mathbb{E}_{p(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}\right)-\mathbb{E}_{q_{\zeta}(\Phi^{\sigma}_{n})}\left(\Phi^{\sigma}_{n}\right)\right]\times$
$\displaystyle\left[\mathbb{E}_{p(\Phi^{\sigma}_{n}|x,\theta)}\left(\Phi^{\sigma}_{n}\right)-\mathbb{E}_{q_{\zeta}(\Phi^{\sigma}_{n}})\left(\Phi^{\sigma}_{n}\right)\right]^{T}q_{\boldsymbol{\zeta}}(\boldsymbol{\theta})d\boldsymbol{\theta}.$
(42)

The first term of the predictive variance of the variational posterior
distribution (42) in known as Aleatoric uncertainty. The expectation is with
respect to predictive posterior distribution $p(\Phi^{\sigma}_{n}|x,\theta)$
integrated over all parameters $\boldsymbol{\theta}$. The second term of the
predictive variance of the variational posterior distribution (42) is the
epistemic uncertainty, variance in predictions due to uncertainty in weights.

Figure 1: Bayesian Neural Network: Shaded area is non zero
Epistemic(prediction) uncertainty and Black dots are training points chosen to
ensure non zero Aleatoric uncertainty (kernel independence test)

If we chose the training data such that Aleatoric uncertainty is non zero,
then we can show that the autocovariance of features $\Phi^{\sigma}_{n}$ is
non zero everywhere. We take benefit of finite excitation property, and using
kernel independence test [24] and qualify the training data points, such that
are sufficiently distinct from the stored data in buffer
$\\{Z:x,y\in\mathcal{X},\mathcal{Y}\\}_{i=1}^{p_{max}}$ hence ensuring
Aletoric uncertainty is non zero and due to fact that the network weights are
stochastic the epistemic/prediction uncertainty is non zero everywhere else.

Hence using stochastic Bayesian network and training the network over data
points chosen according to kernel independence test we can ensure the total
variance $R_{q}\left(\Phi^{\sigma}_{n}(x)|x\right)>0$ and therefore
$\Phi^{\sigma}_{n}$ is P.E ∎

###  VII-A Uniform boundedness of the Outer-Layer parameters

Consider the problem of parameter estimation using a linear estimator with
stochastic deep features. We learn a given function by minimizing its squared
loss error.

Let us assume the function to be estimated be
$f:\mathbb{R}^{n}\to\mathbb{R}^{m}$. Let the true function $f$ be of form

$f(x)=W^{*T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon(x),$ (43)

Where the unknown function be linearly parameterized in unknown ideal weight
$W^{*}\in\mathbb{R}^{k\times m},\forall\sigma$ and
$\bar{\Phi}_{n}^{\sigma}(x):\mathbb{R}^{n}\to\mathbb{R}^{k}$ such that
$\bar{\Phi}_{n}^{\sigma}(x)=\mathop{\mathbb{E}}_{\theta}\left({\Phi}_{n}^{\sigma}(x)\right)$.
Where ${\Phi}_{n}^{\sigma}(x)$ is stochastic nonlinear continuously
differentiable basis function which is drawn from distribution (38)

Let the $W(t)$ be the online estimate of true ideal weights $W^{*}$, therefore
the online estimate of $f(x)$ can be represented by mapping
$\nu:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that,

$\displaystyle\nu(x)$ $\displaystyle=$ $\displaystyle
W^{T}{\Phi}_{n}^{\sigma}(x)=W^{T}\left(\bar{\Phi}_{n}^{\sigma}(x)+G_{\sigma}\xi\right).$
(44)

Let $e(t)$ be defined as,

$\displaystyle e(t)$ $\displaystyle=$ $\displaystyle f(x)-\nu(x)$ (45)
$\displaystyle=$ $\displaystyle
W^{*T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon(x)-W^{T}\left(\bar{\Phi}_{n}^{\sigma}(x)+G_{\sigma}\xi\right)$
$\displaystyle=$
$\displaystyle\tilde{W}^{T}\bar{\Phi}_{n}^{\sigma}(x)+\epsilon(x)-W^{T}\left(G_{\sigma}\xi\right).$

the parameter update using gradient descent algorithm is given as

$\dot{W}=-\Gamma\bar{\Phi}_{n}^{\sigma}(x)e(t)$ (46)

The parameter error dynamics can be found by differentiating $\tilde{W}$ and
using equation (46) as,

$\dot{\tilde{W}}=-\Gamma\bar{\Phi}_{n}^{\sigma}(x)e(t),$ (47)

This is a linear time varying differential equation in $\tilde{W}$.
Furthermore, note that if PE Condition using Theorem-2 is satisfied, then
$\tilde{W}$ is asymptotically bounded near zero solution.

###### Theorem 3

Consider the system model given by equation (43), the online estimation model
given by equation (44), the stochastic deep feature gradient descent weight
update law (47), and assume that the regressor function
${\Phi}_{n}^{\sigma}(x)$ is continuously differentiable and that the
measurements ${\Phi}_{n}^{\sigma}(x)\in\mathcal{D}$ where
$\mathcal{D}\in\mathbb{R}^{m}$ is a compact set. If the
${\Phi}_{n}^{\sigma}(x)$ satisfy the PE condition, then the zero solution of
the weight error dynamics of equation (44) is globally uniformly ultimately
bounded.

###### Proof:

Consider a quadratic Lyapunov function

$V(\tilde{W})=\frac{1}{2}tr\left(\tilde{W}^{T}\Gamma^{-1}\tilde{W}\right).$
(48)

Such that $V(0)=0$ and $V(\tilde{W})>0,\forall\tilde{W}\neq 0$. Since
$V(\tilde{W})$ is quadratic, letting $\lambda_{min}(.)$ and $\lambda_{max}(.)$
denote the operators that return the minimum and maximum eigenvalue of a
matrix, we have:

$\lambda_{min}\left(\Gamma^{-1}\right)\left\|\tilde{W}\right\|_{F}^{2}\leq
V(\tilde{W})\leq\lambda_{max}\left(\Gamma^{-1}\right)\left\|\tilde{W}\right\|_{F}^{2}.$

Taking a time derivative along the trajectory (47), we have

$\displaystyle\dot{V}(\tilde{W})$ $\displaystyle=$ $\displaystyle
tr\left(\tilde{W}^{T}\Gamma^{-1}\dot{\tilde{W}}\right).$ (49)

Using (47), (45) and the definition of empirical estimate of
$\bar{\Phi}_{n}^{\sigma}(x)$ in (18), the above expression cn be rewritten as,

$\displaystyle\dot{V}(\tilde{W})$ $\displaystyle=$
$\displaystyle-\tilde{W}^{T}\bar{\Phi}_{n}^{\sigma}(x)\bar{\Phi}_{n}^{\sigma}(x)^{T}\tilde{W}-\bar{\Phi}_{n}^{\sigma}(x)\tilde{W}^{T}\epsilon(x)$
(50)
$\displaystyle-\tilde{W}^{T}\bar{\Phi}_{n}^{\sigma}(x)\left(G_{\sigma}\xi\right)^{T}\tilde{W}$
$\displaystyle\leq$
$\displaystyle-\lambda_{max}(\Omega)\left\|\tilde{W}\right\|^{2}-\left\|\bar{\Phi}_{n}^{\sigma}\right\|_{\infty}\mathcal{W}_{b}\bar{\epsilon}^{\sigma},$
(51) $\displaystyle\leq$
$\displaystyle-\frac{\lambda_{max}(\Omega)}{\lambda_{min}\left(\Gamma^{-1}\right)}V(\tilde{W})-\left\|\bar{\Phi}_{n}^{\sigma}\right\|_{\infty}\mathcal{W}_{b}\bar{\epsilon}^{\sigma}.$
(52)

Where $\Omega$ is defined as follows,

$\Omega=\frac{1}{N}\sum_{i,j=1}^{N}\left(\Phi_{n,i}^{\sigma}(x)\bar{\Phi}_{n,j}^{\sigma}(x)^{T}+\Phi_{n,i}^{\sigma}(x)(G_{\sigma}\xi_{j})^{T}\right).$
(53)

Using P.E condition of $\Phi_{n,i}^{\sigma}(x)$, we know that
$\lambda_{max}(\Omega)>0$. Therefore using Lyapunov stability theory if

$\left\|\tilde{W}\right\|\geq\sqrt{\frac{\left\|\bar{\Phi}_{n}^{\sigma}\right\|_{\infty}\mathcal{W}_{b}\bar{\epsilon}^{\sigma}}{\lambda_{max}(\Omega)}},$
(54)

we have $\dot{V}(\tilde{W})\leq 0$. Therefore the set
$W_{\delta}=\left\\{\left\|\tilde{W}\right\|\leq\sqrt{\frac{\left\|\bar{\Phi}_{n}^{\sigma}\right\|_{\infty}\mathcal{W}_{b}\bar{\epsilon}^{\sigma}}{\lambda_{max}(\Omega)}}\right\\}$
is positively invariant, hence $\tilde{W}$ approaches the bounded near zero
solution exponentially fast and remain are ultimately bounded. ∎

If Theorem-3 holds, then the adaptive weights $W(t)$ will approach
exponentially fast and remain bounded in a compact neighborhood of the ideal
weights $W^{*}$.

##  VIII S-DMRAC evaluation: Wing-Rock System

This section evaluates the S-DMRAC controller on the Wing-Rock system [27,
28]. The aim of the controller is to follow the step reference command under
unknown nonlinear uncertainties.

Let $\phi$ denote the roll angle of an aircraft, $p$ denote the roll rate,
$\delta_{a}$ denote the aileron control input, then a simplified model for
wing rock dynamics is given by

$\displaystyle\dot{\phi}$ $\displaystyle=$ $\displaystyle p,\hskip 11.38109pt$
(55) $\displaystyle\dot{p}$ $\displaystyle=$
$\displaystyle\delta_{a}+\Delta(x).$ (56)

The system uncertainty is defined as

$\Delta(x)=W_{0}+W_{1}\phi+W_{2}p+W_{3}|\phi|p+W_{4}|p|p+W_{5}\phi^{3}$ (57)

Where $x=[\phi,p]$. The true parameters are as follows
$[W_{0}=1.0,W_{1}=0.2314,W_{2}=0.6918,W_{3}=-0.6245,W_{4}=0.1,W_{5}=0.214]$.
The task of the controller is to follow a reference step command $r(t)$. The
reference model chosen is a stable second order linear system with natural
frequency of $2rad/s$ and damping ratio of $0.5$. The linear control gains are
given by $k=[-4,-2]$ and $k_{r}=-4$ and the learning rate is set to
$\Gamma=10$. Initial conditions for the simulation are arbitrarily chosen to
be $\phi=1deg$, $p=1deg/s$. The tolerance threshold for kernel independence
test is selected to be $\epsilon_{tol}=0.1$ for updating the basis vector
$\mathcal{BV}(\sigma)$. Using high bias ”$W_{0}$”, varied step size in
tracking command and higer learning rate, we push the controller to their
limits, to test the robusteness of S-DMRAC vs DMRAC controller.

Figure-2 show the closed-loop system performance in tracking the reference
signal for the S-DMRAC controller. As we can see, the DMRAC has a larger
oscillation in state tracking compared to S-DMRAC. The presented controller
S-DMRAC achieves better uncertainty estimate as shown in Fig-3 compared to the
DMRAC controller. Due to the Bayesian property of including prior information
to parameter update, which introduces natural regularization, for the same
learning rate, S-DMRAC estimates are less noisy compared to DMRAC. Due to the
stochastic nature of the S-DMRAC controller, we also derive the confidence
bound around the estimates Fig-3. We can Observe around the region of the plot
where the tracking signal has a variation (finitely exciting), the prediction
confidence is high. The prediction confidence is low around the non P.E region
of the reference signal; therefore, a large variance Fig-3. Due to this
overall variation (Aleatoric and Epistemic), the deep features are
characterized by the induced persistency of excitation, resulting in network
weights $W$ for S-DMRAC evolve steady and display convergence-like properties
compared to DMRAC Fig-4.

Figure 2: Wind-Rock Roll and Roll-rate for step command following using
S-DMRAC controller Figure 3: Uncertainty Estimate S-DMRAC vs DMRAC Figure 4:
Outer-Layer weights $W$ evolution with time for S-DMRAC vs DMRAC

##  IX Conclusion

This paper presented an S-DMRAC adaptive controller using Bayesian neural
networks to address feature designing for unstructured uncertainty. The
proposed controller uses Bayesian DNN to model significant uncertainties
without knowledge of the system’s domain of operation. We provide theoretical
proofs that, given the network predictions’ stochastic nature, ensures
”Induced persistency of excitation” of the features leading to convergence of
both state and network final layer weights. Numerical simulations with the
Wing-rock model demonstrate the controller performance in achieving reference
tracking in the presence of significant matched uncertainties. We compared the
results with a state-of-the-art DMRAC controller and demonstrated the network
weights have less transient and show convergence. We conclude that both DMRAC
and S-DMRAC are highly efficient architectures for high-performance control of
nonlinear systems with robustness and long-term learning properties.

## References

  * [1] Girish Joshi and Girish Chowdhary.  Deep model reference adaptive control.  In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 4601–4608. IEEE, 2019.
  * [2] Girish Joshi, Jasvir Virdi, and Girish Chowdhary.  Asynchronous deep model reference adaptive control.  arXiv preprint arXiv:2011.02920, 2020.
  * [3] Girish Joshi, Jasvir Virdi, and Girish Chowdhary.  Design and flight evaluation of deep model reference adaptive controller.  In AIAA Scitech 2020 Forum, page 1336, 2020.
  * [4] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [5] Anuradha M Annaswamy and Kumpati S Narendra.  Adaptive control of simple time-varying systems.  In Decision and Control, 1989., Proceedings of the 28th IEEE Conference on, page 1014?1018 vol.2, 12 1989.
  * [6] Zachary T Dydek, Anuradha M Annaswamy, and Eugene Lavretsky.  Adaptive control of quadrotor uavs in the presence of actuator uncertainties.  AIAA Infotech@ Aerospace, pages 20–22, 2010.
  * [7] Petros A Ioannou and Jung Sun.  Robust Adaptive Control.  Prentice-Hall, Upper Saddle River, 1996.
  * [8] J.-B. Pomet and L Praly.  Adaptive nonlinear regulation: estimation from the Lyapunov equation.  Automatic Control, IEEE Transactions on, 37(6):729–740, 6 1992\.
  * [9] Girish Chowdhary and Eric N Johnson.  Concurrent Learning for Convergence in Adaptive Control Without Persistency of Excitation.  In 49th IEEE Conference on Decision and Control, pages 3674–3679, 2010.
  * [10] Miao Liu, Girish Chowdhary, Bruno Castra da Silva, Shih-Yuan Liu, and Jonathan P How.  Gaussian processes for learning and control: A tutorial with examples.  IEEE Control Systems Magazine, 38(5):53–86, 2018.
  * [11] Girish Joshi and Girish Chowdhary.  Adaptive control using gaussian-process with model reference generative network.  In 2018 IEEE Conference on Decision and Control (CDC), pages 237–243. IEEE, 2018.
  * [12] G Chowdhary, H A Kingravi, J P How, and P A Vela.  Bayesian Nonparametric Adaptive Control Using Gaussian Processes.  IEEE Transactions on Neural Networks and Learning Systems, 26(3):537–550, 3 2015.
  * [13] Bernhard Scholkopf, Ralf Herbrich, and Alex Smola.  A Generalized Representer Theorem.  In David Helmbold and Bob Williamson, editors, Computational Learning Theory, volume 2111 of Lecture Notes in Computer Science, pages 416–426. Springer Berlin / Heidelberg, 2001.
  * [14] Bernhard Schölkopf and Alexander J Smola.  Learning with kernels: Support vector machines, regularization, optimization, and beyond.  MIT press, 2002.
  * [15] Carl Edward Rasmussen and Christopher KI Williams.  Gaussian process for machine learning.  MIT press, 2006.
  * [16] K Hornik, M Stinchcombe, and H White.  Multilayer Feedforward Networks are Universal Approximators.  Neural Networks, 2:359–366, 1989.
  * [17] Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio.  Learning Functions: When Is Deep Better Than Shallow.  arXiv e-prints, page arXiv:1603.00988, Mar 2016.
  * [18] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review.  International Journal of Automation and Computing, 14(5):503–519, Oct 2017.
  * [19] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  arXiv e-prints, page arXiv:1611.03530, Nov 2016.
  * [20] Matus Telgarsky.  Benefits of depth in neural networks.  arXiv e-prints, page arXiv:1602.04485, Feb 2016.
  * [21] Shiyu Liang and Rayadurgam Srikant.  Why deep neural networks for function approximation?  arXiv preprint arXiv:1610.04161, 2016.
  * [22] P Ioannou and J Sun.  Theory and design of robust direct and indirect adaptive-control schemes.  International Journal of Control, 47(3):775–813, 1988.
  * [23] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural networks, 4(2):251–257, 1991.
  * [24] Girish Chowdhary, Tansel Yucelen, Maximillian Mühlegg, and Eric N Johnson.  Concurrent learning adaptive control of linear systems with exponentially convergent bounds.  International Journal of Adaptive Control and Signal Processing, 27(4):280–301, 2013.
  * [25] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural networks.  arXiv preprint arXiv:1505.05424, 2015.
  * [26] Yongchan Kwon, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik.  Uncertainty quantification using bayesian neural networks in classification: Application to ischemic stroke lesion segmentation.  2018\.
  * [27] Jia Luo and C Edward Lan.  Control of wing-rock motion of slender delta wings.  Journal of Guidance, Control, and Dynamics, 16(2):225–231, 1993\.
  * [28] Mogen M Monahemi and Miroslav Krstic.  Control of wing rock motion using adaptive feedback linearization.  Journal of guidance, control, and dynamics, 19(4):905–912, 1996\.