# Controllability-Constrained Deep Network Models
for Enhanced Control of Dynamical Systems

Suruchi Sharma1, Volodymyr Makarenko1, Gautam Kumar2, Stas Tiomkin1,∗
Charles W. Davidson College Of Engineering, San Jose State University, CA, USA
{suruchi.sharma, volodymyr.makarenko, gautam.kumar<EMAIL_ADDRESS>This work was funded by Charles W. Davidson College Of Engineering (CoE), San
Jose State University (SJSU)1 Computer Engineering Department, CoE, SJSU 2
Chemical and Materials Engineering Department, CoE, SJSU ∗ Corresponding
Author

###### Abstract

Control of a dynamical system without the knowledge of dynamics is an
important and challenging task. Modern machine learning approaches, such as
deep neural networks (DNNs), allow for the estimation of a dynamics model from
control inputs and corresponding state observation outputs. Such data-driven
models are often utilized for the derivation of model-based controllers.
However, in general, there are no guarantees that a model represented by DNNs
will be controllable according to the formal control-theoretical meaning of
controllability, which is crucial for the design of effective controllers.
This often precludes the use of DNN-estimated models in applications, where
formal controllability guarantees are required. In this proof-of-the-concept
work, we propose a control-theoretical method that explicitly enhances models
estimated from data with controllability. That is achieved by augmenting the
model estimation objective with a controllability constraint, which penalizes
models with a low degree of controllability. As a result, the models estimated
with the proposed controllability constraint allow for the derivation of more
efficient controllers, they are interpretable by the control-theoretical
quantities and have a lower long-term prediction error. The proposed method
provides new insights on the connection between the DNN-based estimation of
unknown dynamics and the control-theoretical guarantees of the solution
properties. We demonstrate the superiority of the proposed method in two
standard classical control systems with state observation given by low
resolution high-dimensional images.

##  I INTRODUCTION

In recent years, the development of dynamical control models from high-
dimensional data, such as images, using deep neural networks (DNNs) has
garnered substantial interest within various fields, ranging from robotics to
neuroscience [1, 2]. Such data-driven models allow for the design of optimal
control strategies for intricate dynamical control systems. However, these
data-driven DNN-based models solely rely on the quality of input-output data
acquired from a system. Often, this data lacks the desired control-theoretic
properties, such as controllability, necessary for effective system control.
Consequently, designing optimal controllers using these models can lead to
sub-optimal performance. In this paper, we address this issue by explicitly
incorporating a controllability metric into the development of latent models
derived from high-dimensional imaging data. By using these models within a
model predictive control (MPC) framework, we demonstrate significant
improvement in the MPC performance on two classical control systems: the
inverted pendulum and a cart-pole system.

One of the widely adopted methods for capturing dynamical features in a low-
dimensional latent space from high-dimensional data is Variational
AutoEncoders (VAEs) [3]. In essence, a VAE comprises an encoder model and a
decoder model, both trained on the input data to minimize the reconstruction
error between the encoded-decoded data and the input data [3, 4]. Unlike
standard autoencoders, which encode input to a single point, VAEs encode
inputs as probability distributions across the latent space. VAEs have found
applications in various domains, including anomaly detection [5], image
regeneration [6], data compression [7, 8]

Notably, VAEs have also emerged as valuable low-dimensional latent models for
designing and controlling dynamical systems in recent research endeavors [4,
9]. In this study, state information is represented as images. Then
Convolution Neural Networks are used for processing these images and creating
latent models. Convolution Neural Networks excel in handling image data, as
demonstrated in [10, 11, 12]. These latent representations are used to acquire
an understanding of the dynamics of the environment, executing control tasks
similar to the approach in [13, 14, 4, 9]. Diving deeper into this concept
reveals that this approach not only helps in predicting short horizons but
also longer horizons such as in [15, 16].

None of the above approaches explicitly include controllability constraints in
their latent space. We aim to induce controllability constraints in the latent
space of the Variational AutoEncoder (VAE) model[3]. The Model Predictive
Control techniques [17, 18] are used to evaluate the performance of the
models. Since VAEs are trained using the input-output (control-observation)
data from the system, the data used for training VAEs may limit the extent to
which VAEs exhibit system controllability in response to exogenous inputs.
Formally, the VAE’s objective [3] is to minimize the prediction error between
the next state observation, generated by a dynamics model, given the current
state observation and current control action. As we show in this work, this
prediction error itself does not guarantee the controllability of a dynamics
model, leading to the sub-optimal performance of controllers derived with such
a model.

Incorporating a controllability constraint, such as controllability Gramian
[18], during VAE training could significantly improve the intrinsic
controllability of VAEs and lead to the development of efficient control
strategies for complex systems. In this work, we conceptualize this idea by
developing an approach to enhance the controllability of estimated models by
imposing a controllability constraint to the standard VAE objective. In
particular, we augment the standard VAE objective for dynamics learning with
the degree of controllability of the latent dynamics. For our best knowledge,
this is the first attempt to explicitly incorporate controllability
constraints in designing latent models for dynamical systems estimated from
data.

Thus, this work establishes new connections between the state-of-the-art
machine learning methods, (VAE), for the dynamics estimation from high-
dimensional data (images/Lidar/etc) and the essential control-theoretical
properties of dynamical control systems (controllability), leading to more
reliable data-driven models and more efficient data-driven controllers. The
main contributions of this work are as follows:

  * •

Augmentation of the standard VAE objective for the estimation of unknown
dynamics from high-dimensional data with the controllability constraint.

  * •

An efficient algorithm for the optimization of the controllability-constrained
VAE objective.

  * •

Demonstration of the effectiveness of the proposed method on the classical
dynamical control systems on the tasks of long-term prediction and model-
predictive-control.

This paper is organized as follows. In Section II, we introduce the notations
and necessary definitions for the main components such as state, state
observation, action, original dynamics, $f$, and a dynamics model,
$f_{\theta}$, (latent dynamics), degree of controllability and controllability
Grammian, $W$, and feature extractor and image generator, (encoder, $h_{\phi}$
and decoder, $g_{\psi}$, respectively). Also, we overview the standard
objective for learning a dynamics model from high-dimensional state
observation in the formalism of VAE. In Section III we formulate a novel
controllability-constrained objective for VAE, which comprises a prediction
error term and a penalty term for a reduced degree of controllability of an
estimated dynamics model. In Section IV we demonstrate the advantage of the
controllability-augmented dynamics models in comparison to the standard
(baseline) model. Finally, in Section V we conclude the current work and
discuss future research directions and eventual applications. An overveiw of
the method is provided at Fig. 1.

##  II Problem Formulation

###  II-A Original and Latent Dynamics

Consider a dynamical control system in discrete time, $f$:

$\displaystyle f\;:\;x_{t}\times a_{t}\rightarrow x_{t+1}$ (Original Dynamics)
(1)

with the state and action at time $t$, denoted by $x_{t}\in\mathcal{X}$ and
$a_{t}\in\mathcal{A}$, respectively.

We assume the original dynamics function, $f$, is unknown, and the state,
$x_{t}$ is partially observed through the state observation,
$o_{t}\in\mathcal{O}$. The latter is given by a signal from a particular
sensor, such as a camera, lidar, proprioceptive sensor, etc. The original
dynamics can be examined by input-output triples,
$\\{o_{t},a_{t},o_{t+1}\\}_{t=1}^{N}$, which are available or can be
collected.

The goal of this work is to improve the quality of dynamics models estimated
from data, $f_{\theta}$, denoted by ‘latent dynamics’ in the sequel, and the
efficiency of controllers, $\pi$, derived with latent dynamics models.

The latent dynamics is described by the following mapping:

$\displaystyle f_{\theta}\;:\;z_{t}\times a_{t}\rightarrow z_{t+1}$ (Latent
Dynamics) (2)

where $z_{t}\in\mathcal{Z}$ is the latent state at time $t$, and
$\theta\in{\Theta}$ is a model parameterizing dynamics in the model class,
${\Theta}$.

A model, $\theta$, of latent dynamics, $f_{\theta}$, is estimated from data
$\mathcal{D}=\\{o_{t},a_{t},o_{t+1}\\}_{t=1}^{N}$, by standard techniques [4,
9], as overviewed below. In this work we assume that $\forall t\;:\;o_{t}$ is
the state observation given by a low-resolution image, rendered from the
original state, $x_{t}$.

Figure 1: Method overview: the current and next time states, $x_{t}$ and
$x_{t+1}$, in the original dynamics are rendered to the corresponding state-
observations, given by low resolution images, $o_{t}$ and $o_{t+1}$,
respectively. The former, $o_{t}$, is encoded to the latent state $z_{t}$ by
the encoder, $h_{\phi}$. The next latent state, $z_{t+1}$, is predicted by the
parametrized latent dynamics, $f_{\theta}$, and the corresponding predicted
image, ${\color[rgb]{1,0,0}\hat{o}_{t+1}}$, is generated by the decoder,
$g_{\psi}$. The prediction error between the decoded and original state-
observations, $\hat{o}_{t+1}$ and $o_{t+1}$, is minimized as in the standard
VAE framework. We enhance the standard VAE framework with the controllability-
constraint, which explicitly penalizes latent dynamics with low degree-of-
controllability. The resulted controllability-constrained latent dynamics (in
red) is superior to the standard baseline models (in blue) both in long-term
prediction and control tasks. The former is visualized by a larger deviation
of the standard model (in blue) from the original dynamics (in black), and the
latter is depicted by a large control cost in the original dynamics driven by
a controller derived with the standard model for $\beta=0$ in comparison to a
controller derived with the controllability-constrained model for $\beta>0$.

###  II-B Estimation of Latent Dynamics

Deep Neural Networks have been proven as a powerful model class, ${\Theta}$,
for the representation of latent dynamics in this setting with an appropriate
training objective [15, 16]:

$\displaystyle\ell(\theta,\phi,\psi$
$\displaystyle)=\frac{1}{N}\sum_{t=1}^{N}d\bigl{(}{o}_{t+1},\hat{o}_{t+1})$
(3) $\displaystyle\mbox{with
}\hat{o}_{t+1}=g_{\psi}\bigl{(}\underbrace{f_{\theta}(\overbrace{h_{\phi}({o}_{t})}^{z_{t}},a_{t})}_{z_{t+1}}\bigr{)},$

where $N$ is the number of training input-output pairs, $h_{\phi}$ and
$g_{\psi}$ are trainable mappings to and from the latent space, respectively:

$\displaystyle h_{\phi}\;:\;$ $\displaystyle o_{t}\rightarrow z_{t}$ (Encoder)
(4) $\displaystyle g_{\psi}\;:\;$ $\displaystyle z_{t}\rightarrow o_{t}$
(Decoder) (5)

and $d(\cdot,\cdot)$ is a given similar function between the original next
state observation, $o_{t}\in\mathcal{O}$ and the reconstructed one,
$\hat{o}_{t}\in\mathcal{O}$:

$\displaystyle d\;:\;$ $\displaystyle o_{t}\times\hat{o}_{t}\rightarrow
R^{+}.$ (6)

In our experiments, we used the commonly-used cross-entropy similarity between
the original image, $o_{t}$, and the reconstructed image, $\hat{o}_{t}$, of
the original dynamics. There exist various architectures and training methods
for the derivation of $f_{\theta}$, $h_{\phi}$, and $g_{\psi}$, [15, 16].

Our method, (Section III) applies to arbitrary architectures and training
methods, but in the experiments, we chose a particular latent dynamics model,
’Embed-To-Control’, [4] due to its simplicity and transparency.

An optimal solution to the problem in Eq.(3) is derived by solving the
following optimization problem [4, 9]

$\displaystyle\theta^{*},\phi^{*},\psi^{*}=\underset{\theta,\phi,\psi}{\mbox{
argmin }}\ell(\theta,\phi,\psi;\mathcal{D}),$ (7)

where $\mathcal{D}=\\{o_{t},a_{t},o_{t+1}\\}_{t=1}^{N}$ is the training data.
There exist efficient techniques to solve the optimization problem in Eq.(7)
with high-dimensional observation and latent spaces [17, 19]. In particular we
used the ’ADAM’ optimizer [20].

A solution to Eq.(7) does not guarantee, in general, controllability in
estimated latent dynamics. In the next section, we introduce a controllaiblty
constraint to the optimization problem in Eq. (7), using degree of
controllability as a metric [18].

###  II-C Degree of Controllability

In linear dynamics, $x_{t+1}=A_{n\times n}x_{t}+B_{n\times m}a_{t}$,
controllability is given by a binary test, where the rank of the
controllability matrix [18], ${C}=[B,AB,A^{2}B,\dots,A^{n-1}B]$, is equal to
the state dimension, $n$, in a controllable system, and it is less than $n$ in
an uncontrollable system. A more flexible controllability criterion is the
degree of controllability, which is identified by the eigendecomposition of
the controllability Gramian, $W_{n\times n}=C\cdot C^{T}$ [18]111The transpose
is denoted by $T$, e.g., $M^{T}$ is the transpose of $M$. More controllable
directions, (eigenvectors of $W$), correspond to larger eigenvalues of $W$
[18], and the total degree of controllability can be identified by the
controllability volume given by the determinant of the controllability
Grammian.

In non-linear dynamics, $x_{t+1}=f(x_{t},a_{t})$, the local degree of
controllability can be estimated by the Grammian of locally-linearized
dynamics around the state, $x$, which gives the local degree of
controllability by the eigendecomposition of $W(x)=C(x)\cdot C(x)^{T}$. We use
this observation in the development of the differentiable controllability
constraint in the next section.

##  III Proposed Method for Controllability-Enhanced Latent Dynamics

In this section we propose a novel method to explicitly induce controllability
into latent dynamics, $f_{\theta}$, by augmenting the loss in Eq.(3) with the
controllability Grammian of locally-linearized latent dynamics around the
latent state, $z$:

$\displaystyle W_{\theta}(z)=$ $\displaystyle C_{\theta}(z)C_{\theta}(z)^{T}$
(Controllability Grammian) (8) $\displaystyle C_{\theta}(z)=$
$\displaystyle[B_{\theta}(z),A_{\theta}(z)B_{\theta}(z),A^{2}_{\theta}(z)B_{\theta}(z),\dots$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\dots,A^{n-1}_{\theta}(z)B_{\theta}(z)]$

where $A_{\theta}(z)=\nabla_{z}f_{\theta}(z)$ and
$B_{\theta}(z)=\nabla_{a}f_{\theta}(z)$ are the Jacobians of the latent
dynamics, $f_{\theta}$, with regard to the latent state, $z$, and real action,
$a$. And, $C_{\theta}(z)$ is the corresponding parametrized controllability
matrix [18].

In general, the pair $\bigl{(}A_{\theta}(z),B_{\theta}(z)\bigr{)}$ is not
necessarily controllable, which may deteriorate the quality of control. In the
next section, we augment the standard VAE-type loss for data-driven
estimations of dynamics with the parametrized controllability Grammian, which
explicitly penalized for a low degree of controllability.

###  III-A Controllabilty-Enchanced Loss

The parametrized Gramian, Eq. (8), in the latent space, $W_{\theta}(z)$,
allows us to formulate the controllability-enhanced loss, $\hat{\ell}$, for
learning latent dynamics, $f_{\theta}$, as follows:

$\displaystyle\hat{\ell}(\theta,\phi,\psi,\beta;\mathcal{D})=\ell(\theta,\phi,\psi;\mathcal{D})-\beta\ell_{\omega}(\theta;\mathcal{D}).$
(9)

The second term in Eq.(9) explicitly increases the total degree of
controllability, (the eigenvalues of $W_{\theta}$), evaluated by the
determinant of the controllability Grammian [21]

$\displaystyle\ell_{\omega}(\theta;\mathcal{D})\doteq\frac{1}{N}\sum_{t=1}^{N}\ln\det
W_{\theta}(z_{t})$ (Constraint) (10)

The interpretation of this constraint is the time-average controllability-
volume of the linearized latent dynamics. Importantly, this constraint is
differentiable, which makes it suitable for gradient-based optimization [20,
22].

The solution to Eq.(9) can be derived by the same methods as in Eq. (7), with
the essential difference that now $f_{\theta}$ has a desired degree of
controllability set by $\beta$.

$\displaystyle\theta^{*}_{W}(\beta),\phi^{*}(\beta),\psi^{*}(\beta)=\underset{\theta,\phi,\psi}{\mbox{
argmin }}\hat{\ell}(\theta,\phi,\psi,\beta;\mathcal{D}).$ (11)

Notably, in Eq. (11), the optimal parameters for latent dynamics, encoder, and
decoder, are parametrized by $\beta$, which produces a family of solutions, as
shown in Fig. 2, and explained in the next sections.

The derivation of the optimal value of $\beta$ can be done by dual
optimization (cf., Eq. (9) in [23] or in general [24]), which we defer to
future work (cf., Section V).

###  III-B Controllability-Prediction Trade-off

Figure 2: Relative Change, in the MPC Cost (red) and Degree of Controllability
(blue). The left y-axis represents the Percentage change in the MPC cost for
$\beta\in[0,\dots,5.0]$. The MPC cost without controllability constraint is
shown for $\beta=0$. There is $9\%$ decrease in the MPC cost for $\beta\approx
0.5$. The right y-axis shows the minimal eigenvalue of the controllability
Grammian, $\ln\lambda_{\mbox{min}}(W(\beta))$, for the range of $\beta$
values, which reflects the degree-of-controllability,. There are $3$ orders of
magnitude in the increase of the degree of controllability between $\beta=0$
and $\beta\approx 0.5$, which corresponds to the improvement in the efficiency
of the MPC controller. As expected, for large values of $\beta$, the model
predictability term in Eq. (10) is less effective, resulting in a poor
dynamical model for fitting the data.

The trade-off between the degree of controllability of latent dynamics,
$z_{t+1}=f_{\theta}(z_{t},a_{t})$, and the quality of prediction of $o_{t+1}$
from $z_{t+1}=f_{\theta}(h_{\phi}(o_{t}),a_{t})$ is controlled by a non-
negative parameter, $\beta\geq 0$. This trade-off characterizes the relative
importance between the first term in Eq. (9) and its second term. The former
controls the overall prediction quality of the next image, $o_{t+1}$, from the
current image, $o_{t}$, and the current action, $a_{t}$.The latter directly
affects the degree of controllability of the parametrized latent dynamics,
$f_{\theta}$.

This way the optimal latent dynamics, $\theta^{*}_{W}(\beta)$, in Eq. (11),
both possess a desired controllability degree and represents the mapping
$\forall t\;:\;\\{o_{t}\times a_{t}\rightarrow o_{t+1}\\}$.

This trade-off produces a family of solutions, $f_{\theta^{*}(\beta)}$
parameterized by $\beta$, where the original solution is given by $\beta=0$.
This additional degree of freedom in the optimization problem allows for
better solutions with a particular property of interest in this work -
controllable latent dynamics. We demonstrate this by numerical simulations of
standard control benchmark systems in both tasks: planning with latent
dynamics and deriving a more efficient controller for original dynamics.

A typical trade-off, (family of solutions), is demonstrated in Fig. (2), where
the solution without the controllability constraint in Eq. (10) appears for
$\beta=0$, and there is $9\%$ of the decrease in the Model-Predictive-Control
cost for $\beta=0.5$. For larger $\beta$ values the MPC cost increases as
expected because the first term in Eq. (10) does not succeed in estimating the
dynamics model. The MPC cost is estimated on control of the original dynamics,
$x=f(x,a)$, while the optimal control actions are derived with the latent
dynamics, $z=f_{\theta}(z,a)$. The full experiment settings are explained in
the next section.

Pseudo-Code: Summary of the Proposed Method

1: Stage I: Training controllability-enhanced models

2:$\mathcal{D}=\\{o_{t},a_{t},o_{t+1}\\}_{t=1}^{N}$ \- training data; $\beta$
\- value of the trade-off parameter; $\theta$, $\phi$ and $\psi$ \- initial
DNN-parameters for the dynamics model, the encoder, and the decoder,
respectively.

3:repeat

4:
$\\{\theta,\phi,\psi\\}\leftarrow\\{\theta,\phi,\psi\\}-\nabla\hat{l}(\theta,\phi,\psi;\mathcal{D})$$\triangleright$
Eq. (9)

5:until convergence

6:return optimal ${f_{\theta^{*}}},h_{\phi^{*}},g_{\psi^{*}}$.

7: Stage II: Derivation of the optimal controller

8:${f_{\theta^{*}}}$, $h_{\phi^{*}}$, $x$, $x_{g}$, $H$

9:// Encoding of original initial and target states

10:$z=h_{\phi^{*}}\bigl{(}\mbox{RenderImage}(x)\bigr{)}$$\triangleright$
initial latent state

11:$z_{g}=h_{\phi^{*}}\bigl{(}\mbox{RenderImage}(x_{g})\bigr{)}$$\triangleright$
target latent state

12:// Main Loop

13:repeat

14: $a^{*}=\mbox{MPC}(\mbox{COST},{f_{\theta^{*}}},z,H)$$\triangleright$
optimal action

15: $x=f(x,a^{*})$$\triangleright$ original dynamics

16: $z={f_{\theta^{*}}(z,a^{*})}$$\triangleright$ latent dynamics

17:until $x=x_{g}$

18:return optimal control trajectory
$\\{a_{1}^{*},a_{2}^{*},\dots,a_{k}^{*},\dots\\}$

19: /* Utility functions: MPC and COST */

20:function MPC(COST, ${f_{\theta^{*}}}$, $z$, $H$)

21: $\vec{a}\doteq\\{a_{1},a_{2},\dots,a_{H-1}\\}=\mbox{RandomActionSequence}$

22: $\vec{a}^{*}=\underset{\vec{a}}{\mbox{minimum}}\mbox{
COST}(\vec{a},{f_{\theta^{*}}},z,H)$

23: return $\vec{a}^{*}_{1}$$\triangleright$ Return the first action

24:end function // MPC

25://Inputs: latent state, action sequence, planning horizon

26:function COST($\\{a_{1},a_{2},\dots,a_{H-1}\\},{f_{\theta^{*}}},z,H$)

27: $C=0$

28: for $k\in(1,\dots,H-1)$ do

29: $e=z_{g}-z$$\triangleright$ latent space error

30: $C=C+e^{T}\cdot e+a^{T}_{k}\cdot a_{k}$$\triangleright$ accumulated cost

31: $z={f_{\theta^{*}}}(z,a_{)}$$\triangleright$ latent dynamics update

32: end for

33: return $C$

34:end function // COST

###  III-C Pseudo Code of the Method

The proposed method is summarized by the Pseudo-Code above, wherein the ’Stage
I’ a model, $f_{\theta}$, is estimated from data with the controllability-
constrained objective (cf., Eq. (9)), while in ’Stage II’, this model is used
for the derivation of the optimal controller to control an original dynamics,
$f$, Eq. (1). In both cases, state observations are given by low-resolution
images, rendered from the original state.

##  IV Experiments

In this section, we demonstrate the method represented in Section III on two
classical control benchmarks, used for the evaluation of new algorithms for
image-based estimation of dynamics and control. We evaluate the
controllability-enhanced models, $f^{*}_{\theta}$, in two important tasks:
long-term planning and model predictive control with the estimated model222All
the experiments and results can be reproduced by our code
repository:https://github.com/suruchi1997/ControlledVAE.git.

This section is organized as follows. Firstly, we explain the data collection
and the evaluation metrics, then we provide the numerical simulations for
’Long-Term Planning’ and ’Control’ in two different dynamics: Inverted
Pendulum and Cart Pole.

####  IV-1 Data Collection

Data for training controllability-constrained models, $f_{\theta}$, is
collected with the Open AI simulator for the classical control environments
[25, 14, 26]: ’Inverted Pendulum’ and ’Cart Pole’.

In data collection, the action $a_{t}$ is applied to the simulator of an
environment, denoted by ’Step’, which produces (’Renders’) an image of the
resulting state, $o_{t+1}$:

$\displaystyle\forall
t\in[0,\dots,N]\;:\>o_{t+1}=\mbox{RenderImage}(\mbox{Step}(a_{t})).$ (12)

This way the training data, $\mathcal{D}$, comprises of action and state-
observation trajectories, $\mathcal{D}=\\{a_{t},o_{t}\\}_{t=0}^{N}$.

####  IV-2 Training

The training procedure is based on the standard stochastic gradient descent
(SGD) optimization [20], summarized by ’Stage I’ in the Pseudo-Code.

In the experiments, the encoder, $h_{\phi}$, the latent dynamics,
$f_{\theta}$, and the decoder, $g_{\psi}$, are implemented by a similar neural
networks’ architecture as in the baseline system [4], which is a SOTA in the
field. The architecture details are provided in Appendix APPENDIX for
completeness, and the exact implementations are accessible at the the link.
Remarkably, our method requires only a small modification, (addition of the
controllability constraint), to baseline systems, and can be eventually
applied to arbitrary systems.

####  IV-3 Evaluation Metrics

To examine the effectiveness of the controllability constraint Eq. (10) on the
quality of the latent dynamics, $f_{\theta}$, we used two metrics:

-Qualitative comparison (cf., Fig. 3 and Fig. 4) between state observation trajectories generated by the ground truth dynamics in Eq. (1) and by the trained latent dynamics, $f_{\theta}$, for different values of $\beta$.

-Quantitative comparison (cf., Fig 2 and in the text below) between control cost by a controller derived with the latent dynamics, $f^{*}_{\theta}$, trained with the controllability-constraint, $\beta>0$ and without it $\beta=0$. The MPC controller [27, 17]

$\displaystyle\pi:z_{t}\rightarrow a_{t}$ (13)

is derived with the quadratic cost defined in the latent space, $\mathcal{Z}$,
as summarized by ’Stage II’ in the Pseudo-Code (cf., lines 27-29). The optimal
actions, $a^{*}_{t}=\pi(z_{t})$, are applied to the original dynamics,

$\displaystyle x^{*}_{t+1}=f(x^{*}_{t},a^{*}_{t})$ (14)

and the quadratic cost for the optimal action sequence
$\\{a^{*}_{1},a^{*}_{2},\dots,a^{*}_{k},\dots\\}$ is calculated in the
original space $\mathcal{X}$. These evaluation metrics address the two
important properties of dynamics: ’Long-Term Planning’ and ’Control’,
respectively.

###  IV-A Inverted Pendulum

####  IV-A1 Original Dynamics

Swing up and stabilization of the inverted pendulum is one of the classic
problems in control theory [4, 28, 14, 29]. The goal of this experiment is to
examine whether the controllability-constrained dynamics reflects better the
intrinsic properties (long-term prediction and controllability) of the

original dynamics.

In this experiment, we train the latent dynamics with low-resolution images,
representing state observation, $o_{t}$, and real actions, $a_{t}$. This task
is challenging because the model $f_{\theta}$ should learn a dynamics model
from low-resolution high-dimensional images.

The ’Inverted Pendulum’ environment is schematically represented in the figure
to the left. It comprised of a pivot point (the motor), where the limited
control action (torque),
$|a_{t}|<2($\mathrm{N}\mathrm{/}\mathrm{m}\mathrm{)}$$, can be applied, and a
pole with a point mass at the end.

The state space of this system is two-dimensional, consisting of angle,
$\theta_{1}=\theta($\mathrm{rad}$)$, and angular velocity,
$\theta_{2}=\dot{\theta}($\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$)$. The
dynamics of the pendulum is given by:

$\displaystyle\dot{\theta}_{1}(t)=$ $\displaystyle{\theta_{2}}(t)$
$\displaystyle\dot{\theta}_{2}(t)=$
$\displaystyle\frac{g}{l}\sin(\theta_{1}(t))+\frac{a(t)}{ml^{2}}$ (15)

where $g=9.8$\mathrm{m}\mathrm{/}\mathrm{s}^{2}$$ is gravity,
$m=1$\mathrm{kg}$$ is the mass of the point mass, and $l=1$\mathrm{m}$$ is the
length of the pole.

####  IV-A2 State Observation

The dynamics in Eq. (15) is

simulated in discrete time with $dt=0.1$\mathrm{s}$$. The original state,
$x_{t}$, in the time $t$ is rendered (cf., Eq. (12)) to a corresponding low-
resolution image with dimension $48\times 48$ pixels, which represents the
state observation $o_{t}$, as shown to the left.

The qualitative comparison is shown in Fig. 3. The original dynamics (black),
the controllability-constrained latent dynamics (red), and the baseline latent
dynamics (blue) [4] start from the same initial state. However, in the three
last rows, corresponding to the time steps $t_{7}$, $t_{8}$ and $t_{9}$, it is
visible that the controllability-constrained dynamics better predict future
states in comparison to the baseline dynamics. Additional details appear in
the caption of Fig. 3.

To validate the effectiveness of the proposed method for control, we derived
an MPC controller in Eq. (13) with the controllability-constrained latent
dynamics for $\beta\in[0,\dots 5]$. The change in the MPC cost for different
values of $\beta$ is shown in Fig 2. Control cost in the original dynamics by
the controller derived with baseline latent dynamics for $\beta=0$ is the
reference. The MPC cost is improved by $\sim 9\%$ by the controller derived
with the controllability-constrained latent dynamics for $\beta\approx 0.5$.
As expected, the MPC cost increases for larger values of $\beta$, because the
controllability-constraint becomes dominant in the training objective Eq. (9),
which deteriorates the prediction quality of the latent dynamics. This new
controllability-prediction trade-off shows that estimated models can be
enhanced by explicitly augmenting them with the controllability feature. This
trade-off connects machine learning methods for sample-based model estimation
with the fundamental concepts in optimal control theory.

Original |
---|---
Dynamics |  Constrained
Dynamics |  Unconstrained
Dynamics |

$t_{0}$
$t_{2}$
$t_{7}$
$t_{8}$
$t_{9}$

Figure 3:  Qualitative comparison between the long-term prediction by the
latent dynamics, $f_{\theta}$, with the controllability-constraint (the 2nd
column in red) and without it (the 3rd column blue). The first column
represents the image trajectory by the original dynamics. The pivot point is
at the center of the image. The pendulum starts at
$\theta(t_{0})=\frac{\pi}{4}\hskip 1.42271pt{$\mathrm{rad}$}$ from the bottom
and $\dot{\theta}(t_{0})=0\hskip
1.42271pt{$\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$}$. The rows correspond to
the time steps shown to the left. Visible changes appear after $7$ times
steps. And, as shown at the bottom row, after $9$ time steps, there is a
significant deviation of the latent dynamics trajectory for $\beta=0$ (blue)
from the original dynamics trajectory (black); while the latent dynamics
trajectory for $\beta=0.5$ (red) closely follows the original dynamics. The
deviation increases in further steps.

###  IV-B Cart Pole

To validate the effectiveness of the proposed method in a more complicated
system, we conducted the same experiment with the ’Cart Pole’ dynamics.

####  IV-B1 Original Dynamics

Balancing the pole over the cart is another classic problem in control theory.

The cart pole system consists of a pole attached to a cart that moves over the
frictionless track[30]. The state space dimensionality is four. The components
of state space include, cart position, ${x}$, cart linear velocity, $\dot{x}$,
pole-angle, $\theta$, and pole angular velocity, $\dot{\theta}$.

The action space, $\vec{a}(t)\in\mathcal{A}$, is discrete, comprising of two
force values +1$\mathrm{N}$ and -1$\mathrm{N}$. The original dynamics is given
by

$\displaystyle\ddot{x}(t)=$
$\displaystyle\frac{m\sin\theta(t)(\ell\dot{\theta}^{2}(t)+g\cos\theta(t))+a(t)}{M+m\sin^{2}\theta(t)},$
(16) $\displaystyle\ddot{\theta}(t)=$
$\displaystyle-a(t)\cos\theta(t)-m\ell\dot{\theta}^{2}(t)\cos\theta(t)\sin\theta(t)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\quad-(M+m)g\sin\theta(t),$

where $x(t)$, $\theta(t)$, $m$, $M$, $\ell$, $g$, $|a(t)|\leq 1$ are the $x$
coordinate of the center of mass of the cart, the angle of the pole, the pole
mass, the cart mass, the pole length, the free-fall acceleration, and the
force applied to the cart.

####  IV-B2 State Observation

In the same procedure as in the Inverted Pendulum, the dynamics in Eq. (16) is

simulated in discrete time, and the state $x_{t}$ at the time, $t$, is
rendered to a low-resolution image with dimension $80\times 80$, shown to the
left.

Similarly to the ’Inverted Pendulum’ environment, the performance is evaluated
on both long-term prediction and control tasks. Fig. 4 shows the qualitative
comparison between the future state predictions by the latent dynamics and the
original dynamics. The black images, in the 1st column, represent the image
trajectory by the original dynamics. The red images, in the 2nd column, show
the predictions by the controllability-constrained models, and the blue
images, in the 3rd column, show the predictions by the baseline models. All
three dynamics start from the same initial conditions, the 1st row. The images
generated by the controllability-constrained model (red) and the baseline
model (blue), are similar at the first two steps. In the further steps, it is
visible that controllability-constrained models accurately follow the original
dynamics, while the baseline models (trained without the controllability
constraint) deviate from the original dynamics. This deviation increases in
further steps.

The control cost in the original dynamics driven by the controller derived
with the controllability-constrained latent dynamics for $\beta=0.005$ is
lower than that derived with the baseline latent model for $\beta=0.0$.

Original |
---|---
Dynamics |  Constrained
Dynamics |  Unconstrained
Dynamics |

$t_{0}$
$t_{2}$
$t_{7}$
$t_{8}$
$t_{9}$

Figure 4:  Qualitative comparison in Cart Pole. The column and row meaning is
the same as in Figure (3). The pole starts at
$\theta(t_{0})=\frac{\pi}{10}\hskip 1.42271pt{$\mathrm{rad}$}$ from the top
and $\dot{\theta}(t_{0})=0\hskip
1.42271pt{$\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$}$. In this more complicated
environment, there is a significant deviation of the baseline solution (blue)
from the original dynamics (black) after 7 times steps, while the proposed
method (red) closely follows the original dynamics in further steps as well.

##  V Conclusion and Future Work

In this work, we conceptualize the idea of the enhancement of data-driven
models with the controllability property. The implementation of this idea
boils down to adding the controllability constraint to the standard VAE
objective for learning a dynamics model from data. The resulting
controllability-enhanced VAE objective has the same number of parameters to
optimize as in the original VAE. That is because the constraint, (the degree
of controllability, expressed via the controllability Grammian), is given with
the same parameters as in the original unconstrained model. We discovered the
controllability-prediction tradeoff, which is regulated by a scalar parameter
$\beta$. This proof-of-the-concept study shows that data-driven models may
have a small prediction error, which is not enough to guarantee the
controllability of the model. The main observation and conclusion are that
there exists a positive value of $\beta$, when the performance of a derived
controller with a learned model is superior to that for $\beta=0$, as shown in
Figure (2). We defer the optimization of the trade-off parameter $\beta$ to
future work, where the optimal value of $\beta^{*}$ will be explored with the
dual optimization techniques. A full-fledged method for the derivation of
data-driven dynamical models with control-theoretic guarantees for
controllability will allow to design of efficient model-based controllers for
various critical mission applications such as, neuro-stimulation feedback,
robotic surgery, etc.

## ACKNOWLEDGMENT

The authors thank the graduate students at the CI2 Lab, CoE, SJSU, Tristan
Shah and Himaja Papala for proofreading the manuscript.

## References

  * [1] David Daniel Cox and Thomas Dean  “Neural networks and neuroscience-inspired computer vision”  In _Current Biology_ 24.18 Elsevier, 2014, pp. R921–R929
  * [2] Harry A Pierson and Michael S Gashler  “Deep learning in robotics: a review of recent research”  In _Advanced Robotics_ 31.16 Taylor & Francis, 2017, pp. 821–835
  * [3] Diederik P Kingma and Max Welling  “Auto-encoding variational bayes”  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2014
  * [4] Manuel Watter, Jost Springenberg, Joschka Boedecker and Martin Riedmiller  “Embed to control: A locally linear latent dynamics model for control from raw images”  In _Advances in neural information processing systems_ 28, 2015
  * [5] Rong Yao, Chongdang Liu, Linxuan Zhang and Peng Peng  “Unsupervised Anomaly Detection Using Variational Auto-Encoder based Feature Extraction”  In _2019 IEEE International Conference on Prognostics and Health Management (ICPHM)_ , 2019, pp. 1–7  DOI: 10.1109/ICPHM.2019.8819434
  * [6] Kuan Han et al.  “Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex”  In _NeuroImage_ 198 Elsevier, 2019, pp. 125–136
  * [7] Lei Zhou et al.  “Variational autoencoder for low bit-rate image compression”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 2617–2620
  * [8] Lahiru D Chamain, Siyu Qi and Zhi Ding  “End-to-End Image Classification and Compression with variational autoencoders”  In _IEEE Internet of Things Journal_ 9.21 IEEE, 2022, pp. 21916–21931
  * [9] Ershad Banijamali, Rui Shu, Hung Bui and Ali Ghodsi  “Robust locally-linear controllable embedding”  In _International Conference on Artificial Intelligence and Statistics_ , 2018, pp. 1751–1759  PMLR
  * [10] Alexey Dosovitskiy, Jost Tobias Springenberg and Thomas Brox  “Learning to generate chairs with convolutional neural networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1538–1546
  * [11] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “Imagenet classification with deep convolutional neural networks”  In _Advances in neural information processing systems_ 25, 2012
  * [12] Liu Hui and Song Yu-Jie  “Research on face recognition algorithm based on improved convolution neural network”  In _2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA)_ , 2018, pp. 2802–2805  IEEE
  * [13] Marzuki Khalid and Sigeru Omatu  “A neural network based control scheme with an adaptive neural model reference structure”  In _[Proceedings] 1991 IEEE International Joint Conference on Neural Networks_ , 1991, pp. 2128–2133  IEEE
  * [14] Nir Levine et al.  “Prediction, consistency, curvature: Representation learning for locally-linear control”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2020
  * [15] Danijar Hafner, Timothy Lillicrap, Jimmy Ba and Mohammad Norouzi  “Dream to control: Learning behaviors by latent imagination”  In _Proceedings of International Conference on Learning Representations_ , 2020
  * [16] Danijar Hafner et al.  “Learning latent dynamics for planning from pixels”  In _International conference on machine learning_ , 2019, pp. 2555–2565  PMLR
  * [17] Yuval Tassa, Tom Erez and William Smart  “Receding horizon differential dynamic programming”  In _Advances in neural information processing systems_ 20, 2007
  * [18] Steven L Brunton and J Nathan Kutz  “Data-driven science and engineering: Machine learning, dynamical systems, and control”  Cambridge University Press, 2022
  * [19] Weiwei Li and Emanuel Todorov  “Iterative linear quadratic regulator design for nonlinear biological movement systems”  In _First International Conference on Informatics in Control, Automation and Robotics_ 2, 2004, pp. 222–229  SciTePress
  * [20] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ , 2015
  * [21] Tohru Katayama  “Subspace methods for system identification”  Springer, 2005
  * [22] Iman Ahmadianfar, Omid Bozorg-Haddad and Xuefeng Chu  “Gradient-based optimizer: A new metaheuristic optimization algorithm”  In _Information Sciences_ 540 Elsevier, 2020, pp. 131–159
  * [23] Xue Bin Peng et al.  “Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow”  In _International Conference on Learning Representations, (ICLR)_ , 2019
  * [24] Yatin Nandwani, Abhishek Pathak and Parag Singla  “A primal dual formulation for deep learning with constraints”  In _Advances in Neural Information Processing Systems_ 32, 2019
  * [25] Greg Brockman et al.  “Openai gym”  In _arXiv preprint arXiv:1606.01540_ , 2016
  * [26] Javier Arroyo, Carlo Manna, Fred Spiessens and Lieve Helsen  “An Open-AI gym environment for the Building Optimization Testing (BOPTEST) framework”  In _Building Simulation 2021_ 17, 2021, pp. 175–182  IBPSA
  * [27] J.B. Rawlings  “Tutorial overview of model predictive control”  In _IEEE Control Systems Magazine_ 20.3, 2000, pp. 38–52  DOI: 10.1109/37.845037
  * [28] Nenad Muskinja and Boris Tovornik  “Swinging up and stabilization of a real inverted pendulum”  In _IEEE transactions on industrial electronics_ 53.2 IEEE, 2006, pp. 631–639
  * [29] Marvin Bugeja  “Non-linear swing-up and stabilizing control of an inverted pendulum system”  In _The IEEE Region 8 EUROCON 2003. Computer as a Tool._ 2, 2003, pp. 437–441  IEEE
  * [30] Andrew G. Barto, Richard S. Sutton and Charles W. Anderson  “Neuronlike adaptive elements that can solve difficult learning control problems”  In _IEEE Transactions on Systems, Man, and Cybernetics_ SMC-13.5, 1983, pp. 834–846  DOI: 10.1109/TSMC.1983.6313077

## APPENDIX

###  V-A Network Architectures and Hyperparameters

####  V-A1 Pendulum

Input: Two $48\times 48$ images, 15000 training samples of the form
$(o_{t},a_{t},o_{t+1})$, Action Space: 1-dimensional, Latent Space:
2-dimensional, Encoder: 4 layers: Convolution: $16\times 3\times 3$; stride
(2,2) - Convolution: $32\times 3\times 3$; stride (2, 2) - 256 units - 4 units
(2 for mean and 2 for variance), Decoder: 5 layers: 256 units - 4608 units -
Convolution Transpose: $16\times 3\times 3$; stride (2, 2) - Convolution
Transpose: $2\times 3\times 3$; stride (2, 2) - Sigmoid Layer, Transition
Dynamics: 3 layers: 100 units - 100 units - 8 units, MPC Prediction Horizon
(H): 10, Control Horizon: 10, Number of Models (q): 20, Learning Rate: 0.0003,
Optimizer: ’ADAM’ [20], Cost Matrices: $Q=I_{n\times n}$, $R=0.01$.

####  V-A2 CartPole

The architecture is inspired by. [14] Input: Two $80\times 80$ images. 15000
training samples of the form $(o_{t},a_{t},o_{t+1})$, Action Space:
1-dimensional, Latent Space: 4-dimensional, Encoder: 6 layers: Convolution:
$32\times 5\times 5$; stride (1,1) - Convolution: $32\times 5\times 5$; stride
(2, 2)-Convolution: $32\times 5\times 5$; stride (2, 2)-Convolution: $10\times
5\times 5$; stride (2, 2) - 200 units - 8 units (4 for mean and 4 for
variance), Decoder: 7 Layers: 200 units - 1000 units - Convolution Transpose:
$32\times 5\times 5$; stride (1, 1) - Convolution Transpose: $32\times 5\times
5$; stride (1, 1) - Convolution Transpose: $32\times 5\times 5$; stride (1, 1)
- Convolution Transpose: $2\times 5\times 5$; stride (1, 1)-Sigmoid Layer,
Transition Dynamics: 3 layers: 100 units - 100 units - 24 units, MPC
Prediction Horizon: 10, Control Horizon: 10. Number of Models (q): 8, Learning
Rate: 0.0001, Optimizer: ’ADAM’ [20], Cost Matrices : $Q=I_{n\times n}$,
$R=1$.

###  V-B Additional Training Details

To generate Fig 2 for every $\beta$ value ’q’ sets were created. To compute
the final original planning cost for the baseline as well as controllability-
constrained models, an average of all the ”original total cost” values for
every (q,$\beta$) model was taken.