# A Simple Baseline for Efficient Hand Mesh Reconstruction

Zhishan Zhou*, Shihao Zhou*, Zhi Lv , Minqiang Zou, Yao Tang, Jiajun Liang †
Jiiov Technology
{ zhishan.zhou, shihao.zhou, zhi.lv, minqiang.zou, yao.tang,
<EMAIL_ADDRESS>

http://simplehand.github.io

###### Abstract

Hand mesh reconstruction has attracted considerable attention in recent years,
with various approaches and techniques being proposed. Some of these methods
incorporate complex components and designs, which, while effective, may
complicate the model and hinder efficiency. In this paper, we decompose the
mesh decoder into token generator and mesh regressor. Through extensive
ablation experiments, we found that the token generator should select
discriminating and representative points, while the mesh regressor needs to
upsample sparse keypoints into dense meshes in multiple stages. Given these
functionalities, we can achieve high performance with minimal computational
resources. Based on this observation, we propose a simple yet effective
baseline that outperforms state-of-the-art methods by a large margin, while
maintaining real-time efficiency. Our method outperforms existing solutions,
achieving state-of-the-art (SOTA) results across multiple datasets. On the
FreiHAND dataset, our approach produced a PA-MPJPE of 5.8mm and a PA-MPVPE of
6.1mm. Similarly, on the DexYCB dataset, we observed a PA-MPJPE of 5.5mm and a
PA-MPVPE of 5.5mm. As for performance speed, our method reached up to 33
frames per second (fps) when using HRNet and up to 70 fps when employing
FastViT-MA36. Code will be made available.

††* Equally contribution. † Corresponding author. Figure 1: Trade-off between
Accuracy and Inference Speed. Our technique surpasses non-real-time
methods($\leq$ 40 fps) in both speed and precision. Compared to real-time
methods ($\geq$ 70 fps), it offers a substantial boost in accuracy while
preserving comparable speeds. For fair comparison, all speed evaluations were
conducted on a 2080ti GPU with a batch size of one.

##  1 Introduction

The field of hand mesh reconstruction has seen rapid advancements, with
various types of mesh decoders being proposed. Despite their commendable
performance, these methods often suffer from high system complexity, involving
unnecessary components that may hinder efficiency. To facilitate a clear
discussion, we decompose the mesh decoder into two primary components: the
token generator and the mesh regressor.

The token generator serves a crucial role by integrating prior information
with image features to extract task-specific features. For instance, FastMETRO
[5] employs a strategy to predict weak-perspective camera parameters, which
aggregates image features. MobRecon [4] develops a stacked encoding network to
obtain gradually refined encoding features, and applies a technique known as
pose pooling to suppress features that are unrelated to joint landmarks.
PointHMR [7] on the other hand, proposes to use features sampled at positions
of vertices projected from 3D to 2D spaces as intermediate guidance. These
approaches collectively provide informative and discriminating features that
enhance the overall performance of the system.

The mesh regressor, the second component, decodes the tokenized features
obtained from the token generator into mesh predictions. FastMETRO [5] takes a
set of learnable joint tokens and vertex tokens as input and masks self-
attentions of non-adjacent vertices according to the topology of the triangle
mesh during training. MobRecon [4] employs a strategy of 2D-to-3D lifting and
Pose-to-vertex lifting to gradually approximate meshes. MeshGraphormer [11]
uses a coarse template mesh for positional encoding and then applies a linear
Multi-Layer Perceptron (MLP) to sample the coarse mesh up to the original
resolution. These methods aim to alleviate training difficulties due to
heterogeneous modalities.

Through investigation on existing methods, we found a interesting phenomenon
that, although some methods shares same performance, they differs in specific
failure cases. Namely, methods with coarse sampling strategy lack perceptual
ability for fine-grained gestures such as pinch. Methods with limited upsample
layers struggles in generating reasonable hand shapes. This observation
prompts us to question: How different structures make effect on mesh decoder?
By answering the question, we can streamline the process, eliminating
excessive computation and complex components, to complete mesh prediction in a
simple and efficient way. To design concise experiments, we start from the
simplest structure for the aforementioned two modules, then gradually add and
optimize the most commonly used components abstracted from the state-of-the-
art (SOTA) methods.

Through extensive ablation experiments, we discovered that the important
structure of token generator is to sample discriminating and representative
points, while the important structure of mesh generator is to upsample sparse
keypoints into dense meshes. For implicitly, in the following paper, we define
each of these structure as core structure. In the model design process,
provided that the core structure’s functionality is fulfilled, high
performance can be achieved with minimal computational resources.

Based on these observations, we propose a simple baseline that surpasses the
SOTA methods by a significant margin and is computationally efficient.
Referring to Figure 1, our proposed technique delivers state-of-the-art
performance on various datasets. On the FreiHAND [28] dataset, it recorded a
PA-MPJPE of 5.8mm and PA-MPVPE of 6.1mm. When tested on the DexYCB [1]
dataset, these metrics were further refined to a PA-MPJPE of 5.5mm and a PA-
MPVPE of 5.5mm. Our method is also advantaged in efficiency, achieving 33
frames per second (fps) on HRNet[23] and an impressive 70 fps on FastViT-MA36
[21]. Our contributions can be summarized as follows:

  1. 1.

We abstract existing methods into token generator and mesh regressor modules,
and reveal the core structures of these two modules respectively.

  2. 2.

Based on these core structures, we developed a streamlined, real-time hand
mesh regression module that excels in both efficiency and accuracy.

  3. 3.

Our method has achieved PA-MPJPE of 5.7mm and PA-MPVPE of 6.0mm on FreiHAND,
and achieved SOTA results on multiple datasets, demonstrating its
effectiveness and generalizability.

Figure 2: An illustration demonstrates various designs of token generators.
The grids colored in red represent the sampled points. a) Global feature; b)
Grid sampling; c) Keypoint-guided sampling on the original feature map; d)
Keypoint-guided sampling with 4x upsampling, resulting in an enhanced feature;
e) Keypoint-guided sampling with 4x upsampling, where the feature is further
improved by convolution; f) Coarse-mesh-guided point sampling with 4x
upsampling.

##  2 Related Work

In this section, we briefly review existing methods of hand mesh
reconstruction which usually include two main components: a token generator
and a mesh regressor. Token generator processes the backbone image feature and
generates tokens fed to the decoder. Mesh regressor decodes the input tokens
into 3D mesh directly or parametric hand model coefficient.

###  2.1 Hand Mesh Reconstruction

Estimating the 3D hand mesh from a single image has been widely researched.
[27] proposes an end-to-end framework to recover hand mesh from a monocular
RGB image. They use the 2D heatmap as input tokens and fully convolutional and
fully connected layers to regress the MANO [17] parameters.

Transformer [22] has shown powerful performance in language and vision tasks
which could model long range relation among input tokens. MetaFormer [26]
argues that the general architecture of transformers instead of the specific
token mixer is the key player. They replace the self-attention module with a
simple spatial pooling operator and achieve competitive performance with fewer
parameters and less computation. METRO [11] extracts a single global image
feature with a convolutional neural network and performs position encoding by
repeatedly concatenating the image feature with 3D coordinates of a mesh
template. A multi-layer transformer encoder with progressive dimensionality
reduction regresses the 3D coordinates of mesh vertices with these input
tokens. Due to the constraints of memory and computation, the transformer only
processes a coarse mesh by sub-sampling twice with a sampling algorithm [16],
and Multi-Layer Perceptrons (MLPs) are then used to upsample the coarse mesh
to the original mesh.

Graph convolutional neural network(GCNN) [9] is good at modeling the local
interaction between neighbor vertices, thus it is very appropriate for mesh
reconstruction. Pose2Mesh [6] designs a cascaded architecture to regress 3D
mesh vertices from 2D pose directly using GCNN. MeshGraphormer [11] combines
the ability of transformer and GCNN presenting a graph-convolution-reinforced
transformer to model both local and global interactions.

Instead of extracting a global feature from the input image, pointHMR [7]
argues that sampling features guided by vertex-relevant points could better
utilize the correspondence between encoded features and spatial positions.
They conduct feature sampling by element-wise multiplication of image feature
and 2D heatmap trained by projection of 3D mesh vertices. These sampled
features are then fed into the transformer encoder with progressive attention
mask as the form of vertex token. The progressively decreased local connection
range realized by constraining the attention mask encourage the model to
consider the local relationship between neighbor vertices. They also use
linear projection to reduce the dimension of the encoded token and upsampling
algorithm [16] to expand the sparse vertices into original dense vertices.

###  2.2 Lightweight Networks

To achieve real-time hand mesh reconstruction, many lightweight networks have
been studied for years. FastViT [21] is a hybrid vision transformer
architecture which obtains state-of-the-art latency-accuracy trade-off by
structural reparameterization and train-time overparametrization techniques.
MobRecon [4] designs multiple complicated modules to improve efficiency,
including a stacked 2D encoding structure, a map-based position regression
2D-to-3D block and a graph operator based on spiral sampling [10]. FastMETRO
[5] identifies the performance bottleneck of encoder-based transformers is
caused by token design. They propose an encoder-decoder architecture to
disentangle the interaction among input tokens which reduces the parameter.

##  3 Method

In our research, we dissected the existing methods into two key components: a
token generator and a mesh regressor. However, defining the optimal core
structure for each of these modules remains a challenging task. For each
module, we start with a fundamental, intuitive structure, and then
progressively incorporate the most commonly used components, which we have
abstracted from state-of-the-art (SOTA) methods.

Given that these two modules, the token generator and the mesh regressor,
operate in tandem, it’s important to keep one constant when analysing the
other. In practical terms, we first conduct experiments on the mesh regressor
while keeping the token generator, as depicted in Figure 2-B, constant. Then,
we apply the mesh regressor configuration that demonstrated the best
performance to the token generator in subsequent experiments.

###  3.1 Token Generator

Given a single image of dimensions $\\{H,W\\}$, our model utilizes a backbone
to extract image features $X_{b}^{\in\frac{H}{32}\times\frac{W}{32}\times C}$.
The token generator $T$ takes $X_{b}$ as input and produces tokenized mesh
feature $X_{m}^{\in N\times C}$, where $N$ denotes the number of sampled
points. Thus, we can express this as $X_{m}=T(X_{b})$.

Starting with the simplest implementation, we apply a single spatial pooling
(Figure 2-A). This approach establishes a surprisingly competitive baseline,
comparable to the Fastmetro [5]. Changing spatial pooling to point sample
(Figure 2-B) improves the performance. To further improve the quality of the
feature, we follow the MobRecon [4] model and conduct keypoint-guided point
sampling(Figure 2-C). However, this modification did not yield any noticeable
improvement.

Upon visual inspection, it appears that a $7\times 7$ resolution is not
sufficiently discriminating. Consequently, we apply deconvolution on $X_{b}$
to sample the feature map to $14\times 14$ then $28\times 28$ (Figure 2-D),
respectively. This approach results in progressive improvement, but it does
not work for $8\times$ deconvolution or larger.

Models such as MobRecon [4] and PointHMR [7] report improvements by enhancing
features, for example, using a FPN-like structure or stacked blocks. In our
study, we tested different $4\times$ upsample schemes, including double
$2\times$ upsampling, directly $4\times$ upsampling, and adding more
convolution layers during the upsampling process(Figure 2-E). Although these
schemes vary in computational complexity, their performance remains
consistent.

We also tested the coarse mesh sampling method proposed by FastMETRO [5]. This
method (Figure 2-F) generates denser points compared to keypoint-guided
sampling but does not offer any significant advantages. Detailed results are
shown in table 5. These experiments suggest that keypoint-guided point
sampling at an appropriate resolution is a crucial structure for the token
generator. As such, feature enhancement and exhaustive point sampling are not
as necessary as initially thought.

###  3.2 Mesh Regressor

The mesh regressor $R$ takes the tokenized mesh feature $X_{m}^{\in N\times
C}$ as input and outputs predicted meshes Figure 3. [5] [4] adopts a multi-
stage approximation approach and proposes various methods to formulate the
topology relationship between joints and mesh. Finding their intersection
components, we construct a cascading upsampling mesh regressor $R$ using a
series of decoder layers:

$R=H_{k}H_{k-1}...H_{0}$ (1)

Each decoder layer $H_{k}$ takes the calculated tokens $T_{k}$ as input, then
subsequently processes these using a dimension reduce layer, metaformer, and
upsample layer:

$H_{k}(X_{k})=U_{k}(MF_{k}(P_{k}(X_{k})))$ (2)

where $U_{k}$, $P_{k}$ denotes the $k_{th}$ upsample layer and dimension
reduce layer respectively, each composed of a single-layer MLP. The upsample
layer increases token numbers, while the dimension reduce layer modifies
channel shapes. $MF_{k}$ denotes the $k_{th}$ metaformer block, $T_{k}$ is the
$k_{th}$ output token where $X_{k+1}=H_{k}(X_{k})$, $X_{0}=X_{m}$.

Let $d_{k}$ be the output dimension of $U_{k}$ and token numbers of $MF_{k}$,
$n_{k}$,$c_{k}$ and $m_{k}$ are the block number, tokenmixer and block
dimensions for $MF_{k}$. We start with the first layer $MF_{0}$ to demonstrate
its operation. For the ${N\times C}$ shaped tensor $T_{0}$, $P_{0}$ projects
it to ${N\times c}$, which is then processed by $MF_{0}$ and outputs a tensor
of the same shape. Subsequently, $U_{k}$ upsamples it to ${d\times c}$. The
following decoder layers repeat this procedure to output $X_{k}$.

We began from a baseline where
$\\{k=1,n=\\{1\\},d=\\{778\\},m=\\{identity\\}\\}$, which yielded competitive
performance despite its simplicity. We then increase flops by enlarge $n$ but
observe no improvement. Inspired by [4], We sequentially add blocks with an
increasing value of $d$. When $k\leq 3$, Significant performance improvements
are observed. However, as $k$ continues to increase beyond this point, no
further gains are detected. Moreover, Modifying the token mixer from $ide$ to
$attn$ also beneficial. However, for fixed $d$, simply increasing $n$ did not
improve performance. According to our experimental findings, the core function
of each decoder layer is to incrementally elevate the number of tokens from an
initial quantity of 21 up to 778. Additional strategies like augmenting
computational workload or altering intricate specifics of the network appear
to have minimal impact. In our best practice, parameters were set to
$\\{k=3,n=\\{1,1,1\\},d=\\{21,84,336\\},m=\\{attn,attn,attn\\}\\}$.

Existing hand joints and mesh topology modulation approaches stand out due to
their ability to incorporate spatial information. However, their heuristic
design is heavily reliant on hyperparameters and can be labor-intensive.
Recognizing these strengths, we propose a novel method that modulates spatial
relations without requiring manual design or additional computational
resources. We achieve this by introducing learnable position embedding
parameters $emb_{k}$ to each output tensor $X_{k}$ where

$X_{k}=X_{k}+emb_{k}$ (3)

Figure 3: Architecture of decoder layer in mesh regressor. It is composed of
sequentially connected dimension reduce layer, metaformer block and upsample
layer. Figure 4:  Overview of our architecture. The architecture of our model
proceeds as follows: Firstly, the image feature $X_{b}$ is extracted via a
backbone network. These features are then passed to our token generator
module, responsible for predicting 2D keypoints and performing point sampling
on the upsampled feature map, thus generating joint tokens. Next, these joint
tokens are input into our mesh regressor module, which carries out the mesh
prediction to get the final coordinates.

###  3.3 Framework Design

As discussed above, the image feature extracted by the backbone is
sequentially processed by both the token generator and the mesh regressor. The
overall framework can be simply computed by $R(T(X_{b}))$.

The core structures form the basis of the overall structure, which is depicted
in Figure 4. Given an input image of size $\\{H,W\\}$, we conduct point
sampling guided by the predicted 21 keypoints at a resolution of $H/8,W/8$.
For image classification style backbones like Fast-ViT, we apply a 4x upsample
deconvolution to its final layer. However, for segmentation style backbones
like HRNet, we directly use the feature on the corresponding resolution. In
the mesh regressor, we apply position encoding before each MetaFormer block.
Although this is not regarded as a core structure, it serves as a beneficial
addition.

###  3.4 Loss Functions

The method proposed in this paper is trained with supervision for vertices, 3D
joints, and 2D joints. In our implementation, both the 2D joints, denoted as
$J_{2d}$, and the vertices, denoted as $V_{3d}$, are directly predicted by the
model’s output. The 3D joints, represented as $J_{3d}^{{}^{\prime}}$, are
calculated using the equation $J_{3d}=J\times V_{3d}$, where $J$ signifies the
regression matrix. All of these components utilize the L1 loss to compute the
discrepancy between the ground truth and the predictions. The losses for the
vertex, 3D joint, and 2D joint, denoted as $L_{vert}$, $L_{J_{3d}}$, and
$L_{J_{2d}}$, are respectively formulated as follows:

$L_{J_{3d}}=\frac{1}{M_{J_{3d}}}\|J_{3d}-J_{3d}^{{}^{\prime}}\|_{1}$ (4)

$L_{J_{2d}}=\frac{1}{M_{J_{2d}}}\|J_{2d}-J_{2d}^{{}^{\prime}}\|_{1}$ (5)

$L_{vert}=\frac{1}{M_{V_{3d}}}\|V_{3d}-V_{3d}^{{}^{\prime}}\|_{1}$ (6)

Here, $J_{3d}\in R^{M\times 3}$ represents all the ground truth points, and
the symbols annotated with primes denote the predicted values. The overall
loss function is defined as:

$L=w_{3d}L_{J_{3d}}+w_{2d}L_{J_{2d}}+w_{vert}L_{vert}$ (7)

Given that the primary objective of this study is mesh prediction, 2D
keypoints only affect point sampling and thus do not need to be highly
accurate, we have accordingly adjusted the coefficients $w_{3d}$, $w_{2d}$,
and $w_{vert}$ to 10, 1, and 10, respectively.

##  4 Experiments

###  4.1 Implementation Details

Our network is implemented based on Pytorch [15]. We use HRNet64[23] and
FastViT-MA36 [21] as our backbones, with their initial weights pre-trained on
ImageNet. We use the AdamW [8] optimizer to train our network, with a total of
100 epochs. The learning rate is initially set to 5e-4, and then adjusted to
5e-5 after 50 epochs. We train the network with eight RTX2080Ti GPUs, with a
batch size of 32 per GPU. It costs 7 hours training with FastViT-MA36 backone
and 11 hours with HRNet. The features of intermediate layers are directly fed
to the Token Generator without extra upsampling layer when the backbone is
HRNetw64. The Mesh Regressor has three Encoder Layers, with the corresponding
input token numbers being [21, 84, 336], output token numbers being [84, 336,
778], and feature dimensions being [256, 128, 64] respectively. We adopt
Attention as the default token mixer, as its performance is slightly better.

###  4.2 Datasets

Our primary experiments and analyses are conducted on the FreiHAND [28]
dataset. In order to validate the generalization of our method, we also do
experiments on the large-scale 3D hand-object dataset, DexYCB [1]. The
FreiHAND dataset contains 130,240 training samples and 3,960 testing samples.
DexYCB contains 406,888 training samples and 78,768 testing samples.

###  4.3 Evaluation Metrics

Method | Backbone | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | F@05 $\uparrow$ | F@15 $\uparrow$ | FPS
---|---|---|---|---|---|---
I2L-MeshNet [13] | ResNet50 | 7.4 | 7.6 | 0.681 | 0.973 | 72
CMR [3] | ResNet50 | 6.9 | 7.0 | 0.715 | 0.977 | -
I2UV [2] | ResNet50 | 7.2 | 7.4 | 0.682 | 0.973 | -
Tang et al. [20] | ResNet50 | 6.7 | 6.7 | 0.724 | 0.981 | 47
MobRecon [4] | DenseStack | 6.9 | 7.2 | 0.694 | 0.979 | 80
METRO [27] | HRNet | 6.7 | 6.8 | 0.717 | 0.981 | 27
MeshGraphomer [11] | HRNet | 6.3 | 6.5 | 0.738 | 0.983 | 24
FastMETRO [5] | HRNet | 6.5 | 7.1 | 0.687 | 0.983 | 28
Deformer [25] | HRNet | 6.2 | 6.4 | 0.743 | 0.984 | -
PointHMR [7] | HRNet | 6.1 | 6.6 | 0.720 | 0.984 | -
FastViT [21] | FastViT-MA36 | 6.6 | 6.7 | 0.722 | 0.981 | 84
Ours | HRNet | 5.8 | 6.1 | 0.766 | 0.986 | 33
Ours | FastViT-MA36 | 5.7 | 6.0 | 0.772 | 0.986 | 70

Table 1: Results on the FreiHAND dataset. Our results are shown in bold. “-”
indicates not reported. Our results surpass all existing methods in terms of
accuracy metrics.

To evaluate the accuracy of 3D Hand Mesh Reconstruction methods, we adopt five
metrics: Procrustes-aligned mean per joint position error (PA-MPJPE),
Procrustes-aligned mean per vertex position error (PA-MPVPE), mean per joint
position error (MPJPE), mean per vertex position error (MPVPE), and F-Score.
PA-MPJPE and PA-MPVPE refer to the MPJPE and MPVPE after aligning the
predicted hand results with the Ground Truth using Procrustes alignment,
respectively. These two metrics do not consider the impact of global rotation
and scale.

###  4.4 Results

Comparison with previous methods To validate our proposed modules., we adopted
HRNet and FastViT-MA36 as backbones for non-real-time and real-time methods
respectively, following established models [27] [11] [5] [21]. For fair
comparison, we provide performance metrics without Test-Time Augmentation
(TTA) and FPS without TensorRT optimization. Table 1 shows that our method,
despite being slightly slower than FastViT, improves PA-MPJPE by 0.9mm.
Compared to transformer-based methods, our approach demonstrates superior
speed and performance, while only requiring 10% of parameters, as shown in
Table 2.

The qualitative comparison results are shown in the figure 5. Compared to
previous methods, our method produces more accurate hand reconstruction
results.

Method |  $\\#$Params | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---|---
METRO [27] | 102M | 6.7 | 6.8
MeshGraphormer [11] | 98M | 6.3 | 6.5
FastMETRO [5] | 25M | 6.5 | 7.1
Ours | 1.9M | 5.8 | 6.1

Table 2: Comparison of transformer-based approaches. $\\#$Params refer to the
network parameters that are not included within the backbone structure of the
model. Our approach not only surpasses existing benchmarks in key metrics but
also achieves a parameter reduction of one to two orders of magnitude.

Evaluation on DexYCB We employed the large-scale hand-object dataset DexYCB to
validate our method’s effectiveness and generalizability. As shown in Table 3,
our model outperforms existing single-image input methods on all metrics.
Significantly, we surpassed previous benchmarks by 1.5mm and 0.8mm on the
MPJPE and MPVPE measures respectively, thereby setting new standards and
demonstrating our method’s broad applicability.

(a) Input

(b) MobRecon [4]

(c) METRO [27]

(d) MeshGraphormer [11]

(e) Ours

(f) GT

Figure 5: Qualitative comparison between our method and other state-of-the-art
approaches.

Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | MPJPE $\downarrow$ | MPVPE $\downarrow$
---|---|---|---|---
METRO [27] | 7.0 | - | - | -
Spurr et al. [19] | 6.8 | - | - | -
Liu et al. [12] | 6.6 | - | - | -
HandOccNet [14] | 5.8 | 5.5 | 14.0 | 13.1
MobRecon [4] | 6.4 | 5.6 | 14.2 | 13.1
H2ONet [24] | 5.7 | 5.5 | 14.0 | 13.0
Ours | 5.5 | 5.5 | 12.4 | 12.1

Table 3: Results on DexYCB. Our method shows advantages on Procrustes-Aligned
metrics and surpassed the previous methods by a large margin on non-
Procrustes-Aligned metrics.

###  4.5 Ablation Study

To thoroughly validate the various parameter combinations, a large number of
ablation experiments were conducted. For efficiency, all ablation experiments
were implemented on smaller models (e.g., Hiera-tiny). After identifying the
optimal parameter combination, it is then applied to the standard models to
facilitate a fair comparison with existing methods.

The state-of-the-art backbone, Hiera-Tiny [18], is utilized in our study as a
strong baseline. We conduct a series of ablation experiments on the FreiHAND
dataset with the aim of examining the efficacy of the structure we propose.

Effectiveness of Our Token Generator and Mesh Regressor. In order to evaluate
the effectiveness of our Token Generator and Mesh Regressor, we initially set
up a standard baseline model. This model’s Token Generator is constructed
based on global features, while its Mesh Regressor is designed as a Multilayer
Perceptron (MLP). We subsequently substitute these components with our
proposed structures individually. The results of these experiments, detailed
in Table 4, confirm that both modules, when incorporated in place of the
original structures, contribute positively towards enhancing overall
performance. When implemented together, these modifications lead to even
further improvements.

Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---
Simple Baseline | 6.9 | 7.2
\+ mesh regressor | 6.5 | 6.8
\+ token generator | 6.6 | 7.1
\+ both | 6.2 | 6.5

Table 4: Ablation study of our proposed modules. Each of these methods brings
about enhancements when utilized individually. However, when these strategies
are integrated, they yield an even more substantial improvement.

Analysing Core Structure of Token Generator. As shown in Table 5, performance
using only global features is competitive. The grid sampling and point
sampling on a $7\times 7$ feature map show similar efficiencies. Increasing
the resolution of the feature map to $28\times 28$ through a single four-fold
deconvolution improves performance. However, further optimization is not
achieved by replacing single four-fold deconvolution with two layers of two-
fold deconvolutions or adding more convolutions. Similarly, no improvement is
observed when changing from point sampling to coarse mesh sampling.
Qualitative comparison of different point sampling strategies is shown in Fig.
6

Sample Method | Resolution | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---|---
Global | 1x1 | 6.5 | 6.8
Grid | 7x7 | 6.3 | 6.6
Point | 7x7 | 6.3 | 6.6
Point | 14x14 | 6.3 | 6.6
Point | 28x28 | 6.2 | 6.5
Point | 28 x28 enhanced | 6.2 | 6.5
Coarse mesh | 28x28 | 6.2 | 6.5

Table 5: Ablation study of Our Token generator A point sample at a resolution
of 28x28 achieves optimal efficiency. Contrarily, increasing the number of
sampled points or incorporating additional convolutional layers do not lead to
any further improvements.

Analysing the Core Structure of the Mesh Regressor. As shown in Table 6, for a
single encoder layer, adding an extra encoder layer with a larger token number
sharply increases performance by 0.3mm. The optimal setting consists of three
encoder layers, with token numbers progressively multiplied by 4. As the layer
number increases further, the marginal benefit becomes inconsequential and
sometimes even decreases. Furthermore, as shown in Table 7, given a fixed set
of token numbers, increasing computational complexity produces negligible
differences in either block numbers or block dimensions in the encoder layer.
A middle-sized block dimensions setting is optimal. Qualitative comparison of
different upsample layers is shown in Figure 7.

Layer Nums | Token Nums | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---|---
1 | [21] | 6.6 | 7.1
2 | [21, 256] | 6.3 | 6.6
2 | [21, 384] | 6.3 | 6.6
3 | [21, 256, 384] | 6.2 | 6.5
3 | [21, 84, 336] | 6.2 | 6.5
4 | [21, 128, 256, 384] | 6.2 | 6.5
4 | [21, 63, 126, 252] | 6.3 | 6.6

Table 6: The Number of Upsampling Layers and Corresponding Token Numbers in
Encoding Layers. Three encoding layers yield optimal efficiency.

Dimensions | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---
64, 32, 16 | 6.5 | 6.9
128, 64, 32 | 6.3 | 6.6
256, 128, 64 | 6.2 | 6.5
512, 256, 12 | 6.2 | 6.5
1024, 512, 256 | 6.4 | 6.7
Block Nums | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
1, 1, 1 | 6.2 | 6.5
2, 2, 2 | 6.2 | 6.6
3, 3, 3 | 6.3 | 6.6

Table 7: Dimensions and block nums. Single layer blocks with middle sized
dimensions are optimal.

Position Encoding and Attention Mixer. We utilized position encoding layer and
attention mixer during ablation experiments because they are intuitively
helpful. Based on our SOTA result, we remove position encoding layers to
observe a slightly 0.1mm degrade. Similar thing happens when we substitute
attention mixer to identity mixer, see Tab. 8.

Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$
---|---|---
hiera-tiny sota | 6.2 | 6.5
Identity mixer | 6.3 | 6.6
w/o position emb | 6.3 | 6.6

Table 8: Token mixer and position embedding. When substitute attention mixer
to identity mixer, or remove position encoding layer, the performance dropped
sligntly by 0.1mm.

Limits and Failure cases. As mentioned earlier, our work is dedicated to
summarizing and abstracting from existing work. Since no targeted optimization
was performed, some failure cases present in previous work remains
challenging. These cases are concentrated in scenes with self-occlusion and
object occlusion, see Fig. 8.

(a) Input

(b) global

(c) coarse

(d) upsampled

(e) GT

Figure 6: Qualitative Comparison of Different Point Sampling Strategies. The
global/coarse feature fails in scenarios with detailed finger interactions,
where upsampled feature works well.

(a) Input

(b) one layer

(c) two layer

(d) three layer

(e) GT

Figure 7: Qualitative Comparison of the Number of Layers of Mesh Decoder. When
constrained to one, the reconstructed mesh tends to corrupt into unnatural
shapes. Performance improves as the number of layers increases.

(a) Input

(b) ours

(c) gt

(d) Input

(e) ours

(f) gt

Figure 8: Typical Failure Cases. Failure cases are concentrated in scenes with
self-occlusion and object occlusion. Some are difficult to discern due to the
small area of exposure, while others present ambiguities caused by the
occlusion.

##  5 Conclusion and Future Work

We observed shared advantages and disadvantages of typical structures. Based
on these observations, we introduce the concept of the core structure. Through
experiments, we revealed that a framework with the core structure could
achieve high performance with limited computational load. We evaluated our
approach quantitatively and qualitatively to demonstrate its effectiveness.

However, our method is explicitly designed to reconstruct single hand
gestures. Other scenarios, such as extreme lighting, occlusion, interactions,
or out-of-distribution cases, showed no improvement over existing methods.
Such cases require specifically designed methods.

## References

  * Chao et al. [2021] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al.  Dexycb: A benchmark for capturing hand grasping of objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9044–9053, 2021.
  * Chen et al. [2021a] Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan.  I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12929–12938, 2021a.
  * Chen et al. [2021b] Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng.  Camera-space hand mesh recovery via semantic aggregation and adaptive 2d-1d registration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13274–13283, 2021b.
  * Chen et al. [2022] Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo.  Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [5] Junhyeong Cho, Kim Youwang, and Tae-Hyun Oh.  Cross-attention of disentangled modalities for 3d human mesh recovery with transformers.
  * Choi et al. [2020] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  _Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose_ , page 769–787.  2020\.
  * Kim et al. [2023] Jeonghwan Kim, Mi-Gyeong Gwon, Hyunwoo Park, Hyukmin Kwon, Gi-Mun Um, and Wonjun Kim.  Sampling is matter: Point-guided 3d human mesh reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12880–12889, 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kipf and Welling [2016] Thomas Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv: Learning,arXiv: Learning_ , 2016.
  * Lim et al. [2019] Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt.  _A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3D Meshes_ , page 349–362.  2019\.
  * Lin et al. [2021] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Liu et al. [2021] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang.  Semi-supervised 3d hand-object poses estimation with interactions in time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14687–14697, 2021.
  * Moon and Lee [2020] Gyeongsik Moon and Kyoung Mu Lee.  I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16_ , pages 752–768. Springer, 2020.
  * Park et al. [2022] JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee.  Handoccnet: Occlusion-robust 3d hand mesh estimation network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1496–1505, 2022.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Ranjan et al. [2018] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black.  _Generating 3D faces using Convolutional Mesh Autoencoders_ , page 725–741.  2018\.
  * Romero et al. [2017] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: modeling and capturing hands and bodies together.  _ACM Transactions on Graphics_ , page 1–17, 2017.
  * Ryali et al. [2023] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al.  Hiera: A hierarchical vision transformer without the bells-and-whistles.  _arXiv preprint arXiv:2306.00989_ , 2023.
  * Spurr et al. [2020] Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz.  Weakly supervised 3d hand pose estimation via biomechanical constraints.  In _European conference on computer vision_ , pages 211–228. Springer, 2020.
  * Tang et al. [2021] Xiao Tang, Tianyu Wang, and Chi-Wing Fu.  Towards accurate alignment in real-time 3d hand-mesh reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11698–11707, 2021.
  * Vasu et al. [2023] PavanKumarAnasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan.  Fastvit: A fast hybrid vision transformer using structural reparameterization.  2023\.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Neural Information Processing Systems,Neural Information Processing Systems_ , 2017.
  * Wang et al. [2020] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(10):3349–3364, 2020.
  * Xu et al. [2023] Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu.  H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17048–17058, 2023.
  * Yoshiyasu [2023] Yusuke Yoshiyasu.  Deformable mesh transformer for 3d human mesh recovery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17006–17015, 2023.
  * Yu et al. [2022] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Zhang et al. [2019] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng.  End-to-end hand mesh recovery from a monocular rgb image.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Zimmermann et al. [2019] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 813–822, 2019.


# Physics Embedded Neural Network Vehicle Model and Applications in Risk-Aware
Autonomous Driving Using Latent Features

Taekyung Kim∗, Hojin Lee∗, and Wonsuk Lee† The authors are with the Ground
Technology Research Institute, Agency for Defense Development, Daejeon 34186,
Republic of Korea. {ktk1501, hojini1117<EMAIL_ADDRESS>authors
contributed equally to this work†Corresponding author

###### Abstract

Non-holonomic vehicle motion has been studied extensively using physics-based
models. Common approaches when using these models interpret the wheel/ground
interactions using a linear tire model and thus may not fully capture the
nonlinear and complex dynamics under various environments. On the other hand,
neural network models have been widely employed in this domain, demonstrating
powerful function approximation capabilities. However, these black-box
learning strategies completely abandon the existing knowledge of well-known
physics. In this paper, we seamlessly combine deep learning with a fully
differentiable physics model to endow the neural network with available prior
knowledge. The proposed model shows better generalization performance than the
vanilla neural network model by a large margin. We also show that the latent
features of our model can accurately represent lateral tire forces without the
need for any additional training. Lastly, We develop a risk-aware model
predictive controller using proprioceptive information derived from the latent
features. We validate our idea in two autonomous driving tasks under unknown
friction, outperforming the baseline control framework.

##  I Introduction

In recent years, Model Predictive Control (MPC) has shown promising results
for autonomous driving tasks under complex and nonlinear environments [1, 2].
Typically, MPC schemes optimize an open-loop control sequence and then execute
it until the next optimization update. With such a strategy, acquiring
accurate models of dynamical systems is critical. Inaccurate long-horizon
predictions from the model will result in an incorrectly optimized trajectory
and eventually lead to inappropriate motions.

Extensive studies have sought to create analytic models of non-holonomic
vehicle motion, and such works can be divided into two broad categories:
kinematic models and dynamic models. Kinematic models that assume no wheel
slip are insufficient to represent the motions of a high-speed driving vehicle
[3]. Despite attempts by dynamics models to describe the complex wheel/ground
interactions by utilizing a linear tire model, they reveal their natural
limitations when the environments are highly nonlinear [4].

To this end, recent robotics studies have been widely employing machine
learning models owing to their universal function approximation properties.
Most standard machine learning techniques to learn dynamics models include a
Gaussian Process and a deep-learning-based approach using neural networks [5].
In particular, current neural networks use different architectures to
approximate the model, e.g., deterministic models [1, 6], stochastic models
[7], and ensemble models [8]. However, these black-box learning approaches
completely ignore the existing knowledge of well-known physics and create the
mapping between inputs and states from scratch. This strategy not only makes
the learning problem unnecessarily more difficult but also cannot ensure
physical plausibility.

In this paper, we seamlessly combine deep learning with a fully differentiable
physical vehicle model to endow the neural network with available prior
knowledge. We also present a control scheme that can be applied in conjunction
with our neural network model to perform risk-aware autonomous driving tasks
with unknown friction.

Figure 1: Overview diagram of our risk-aware autonomous driving strategy using
the latent proprioceptive information obtained from the physics-embedded
neural network dynamics.

###  I-A Related Work

While model learning for dynamic systems has been extensively studied, only a
few approaches use physics knowledge to improve learning performance outcomes.
One study [9] proposed the use of knowledge of the physical model to become
part of the mean function or of the kernel in a Gaussian Process setting. In
this approach, however, the representation of the model still remains a black-
box.

A more recent development is the Physics-Informed Neural Network (PINN) [10],
which structures the learning problem by involving the knowledge of a specific
differential equation, leading to lower sample complexity. This method can
directly solve a specific set of differential equations after training.

One line of research shares the motivation of the PINN but focuses instead on
model learning problems [11, 12, 13]. These methods approach the model
learning problem using a structure that infers equations of motion. These
approaches do not require to be engineered according to physical assumptions
specific to each system embodiment and are generally applicable to a wide
range of mechanical systems. On the other hand, we focus on the fact that if
there are existing physical priors extensively addressed by prior work,
explicitly embedding them into the neural network could yield extra benefits.

###  I-B Contributions

In this paper, we propose to embed a well-known physics-based model into a
neural network to capture the underlying vehicle dynamics. We add the Pacejka
magic formula [14] and the nominal dynamics of the bicycle model as the last
hidden layers. In the middle of forward propagation of this model, we can
extract latent variables representing the lateral tire forces that are
difficult to measure in the real world. We leverage these latent features as
hints to be given for the risk-aware MPC. To the best of the authors’
knowledge, there has never been a control scheme in which an MPC adapts to the
latent variables of a physics-inspired neural network. An overall diagram of
the proposed control scheme is illustrated in Fig. 1.

In summary, the advantages of using the proposed framework are as follows:

  * •

The model learning problem is entailed by physics priors to yield a faster
learning speed, higher accuracy, and better generalization.

  * •

The proposed model can accurately estimate lateral tire forces with zero
effort without the need for any additional training with ground truth data.

  * •

The latent proprioceptive information, which can be acquired from the forward
propagation of the model during the MPC optimization process, can be used as
valuable hints when designing a risk-aware autonomous driving controller with
unknown friction. 111Our video can be found at: https://youtu.be/rPN0hGCW0R4

##  II Background

Figure 2: A diagram of the bicycle model. The dynamic state variable ${\bf x}$
is depicted in blue. The input variable ${\bf u}$ is depicted in red.

In the following, we provide a brief overview of the Pacejka tire model [14].
To model the motions of a vehicle driving at high speeds, the dynamic bicycle
model is commonly employed because it provides a good trade-off between model
accuracy and simplicity for real-time implementation [2]. The bicycle model is
illustrated in Fig. 2. The dynamic state variable ${\bf
x}=\left(v_{x},v_{y},r\right)^{\top}$ consists of the longitudinal velocity
$v_{x}$, the lateral velocity $v_{y}$, and the yaw rate $r$. The kinematic
state variable in the global coordinate frame consists of the position
$\left(p_{x},p_{y}\right)^{\top}$ and the heading angle $\theta$. The inputs
to the system ${\bf u}=\left(\delta,v_{des}\right)^{\top}$ are the steering
angle $\delta$ and the desired speed $v_{des}$. Note that, unlike the electric
vehicles used in recent MPC-based autonomous driving studies [15, 2], the
majority of current vehicles shift gears automatically. The throttle is
undesirable to be used as an action in these vehicles because the dynamic
characteristics change dramatically as the gears shift. Therefore, we here use
$v_{des}$ as the input and append a low-level Proportional-Integral (PI)
velocity controller to manage the throttle and brake.

The distances between the center of gravity and the front and rear tires are
denoted as $l_{f}$ and $l_{r}$, respectively. The tire sideslip angles can be
obtained as follows:

$\displaystyle\alpha_{f}$
$\displaystyle=\arctan\left(\frac{v_{y}+l_{f}{r}}{v_{x}}\right)-\delta,$ (1)
$\displaystyle\alpha_{r}$
$\displaystyle=\arctan\left(\frac{v_{y}-l_{r}{r}}{v_{x}}\right),$

where the notations $(\cdot)_{f}$ and $(\cdot)_{r}$ denote correspondingly the
front and the rear of the vehicle. Then, the dynamics of the tire-road
contacts can be formulated into nonlinear equations:

$\displaystyle F_{y,f}=F_{z}{D_{f}}$
$\displaystyle\sin({C_{f}}\arctan(B_{f}{\alpha_{f}}-$ (2) $\displaystyle
E_{f}(B_{f}{\alpha_{f}}-\arctan(B_{f}{\alpha_{f}})))),$ $\displaystyle
F_{y,r}=F_{z}{D_{r}}$ $\displaystyle\sin({C_{r}}\arctan(B_{r}{\alpha_{r}}-$
$\displaystyle E_{r}(B_{r}{\alpha_{r}}-\arctan(B_{r}{\alpha_{r}})))),$

where $F_{z}$ denotes the vertical load of the tire. The model parameter $B$
refers to the stiffness factor, $C$ refers to the shape factor, $D$ denotes
the peak value, and $E$ is the curvature factor [3].

Conventional engineering approaches would determine these parameters through a
system identification approach with real-world driving tests. Unfortunately,
the test setup for these measurements is complicated and costly, and the data-
collection procedures by manual driving are time-consuming tasks. Furthermore,
they are determined under a specific set of conditions and naturally change as
environmental factors such as road friction change. These factors complicate
the use of this tire model in an autonomous driving task under unknown
friction. This problem is also related to the fact that the lateral tire
forces $F_{y}$ are difficult to be obtained analytically.

##  III Physics Embedded Neural Network Vehicle Model

###  III-A The Baseline Model

Our research goal is to control an Unmanned Ground Vehicle (UGV) so that it
drives successfully under unknown surfaces with high and low friction.
Accurate dynamics modeling must be accompanied to achieve such control, but
the complex nonlinear motions in this environment are difficult to model.
Depending on the friction level of the road surface, the same lateral force
applied to the wheel at the same corner will result in completely different
vehicle motions. There is one more factor that adds complexity to the modeling
problem, which is the time-varying characteristics due to the automatic gear
shifting.

To overcome these issues, we use a neural network in which historical state-
action pairs $\bm{\mathcal{H}}$ are used as inputs to capture time-varying
behaviors [16]:

$\bm{\mathcal{H}}_{t}=\left[{\bf x}_{t},\,{\bf u}_{t},\,{\bf x}_{t-1},\,{\bf
u}_{t-1},\,\dots\bf x_{t-H+1},\,{\bf u}_{t-H+1}\right].$ (3)

The length of the history is denoted as $H$ and the discretized time variable
is denoted as $t$. This strategy enables the network to accurately approximate
nonlinear and complex dynamics without explicitly encoding the automatic gear
shifting mechanism [17]. Then, we can employ a standard fully-connected neural
network ${\bf F}$ to predict the discrete change in states over the time step
duration of $\Delta t$ [7]:

${\bf x}_{t+1}-{\bf x}_{t}={\bf F}(\bm{\mathcal{H}}_{t}).$ (4)

The idea behind estimating the residual difference in states is proposed to be
effective when using a small discretized time interval [6].

Following prior work [18], we design a baseline feedforward neural network
(abbreviated as “FF-NN”) consisting of four fully-connected hidden layers with
$H=4$. The input to each activation is denoted using ${\bf z}_{\diamond}$ and
the output of the activation is indicated with ${\bf o}_{\diamond}$, where
$\diamond$ represents the layer number. The activation function is denoted
using $\operatorname{act}({\bf z}_{\diamond})$. ${\bf W}_{\diamond}$ and ${\bf
b}_{\diamond}$ are correspondingly the weights and biases of the network. The
baseline model then takes the following form:

$\displaystyle{\bf z}_{1}$ $\displaystyle={\bf
W}_{1}^{\top}\bm{\mathcal{H}}_{t}+{\bf b}_{1}$ (5) $\displaystyle{\bf o}_{1}$
$\displaystyle=\operatorname{act}\left({\bf z}_{1}\right)$ (6)
$\displaystyle\qquad\dots$ $\displaystyle{\bf o}_{3}$
$\displaystyle=\operatorname{act}\left({\bf z}_{3}\right)$ (7)
$\displaystyle{\bf z}_{4}$ $\displaystyle={\bf W}_{4}^{\top}{\bf o}_{3}+{\bf
b}_{4}$ (8) $\displaystyle{\bf x}_{t+1}$ $\displaystyle={\bf x}_{t}+{\bf
z}_{4}.$ (9)

###  III-B Deep Pacejka Model

In this section, we introduce our Deep Pacejka Model (DPM). We propose to
explicitly embed the prior knowledge of the vehicle physics model into the
neural network. We add the Pacejka tire model as a layer between the last
hidden layer of the baseline network and its output layer. Here, each node in
the last hidden layer (8) predicts the parameters of the physics model in (2).
Note that the longitudinal force $F_{x}$ applied to the vehicle is an
essential component of the vehicle’s lateral dynamics. However, unlike an
electric vehicle, it is undesirable to model it as a linear relationship with
throttle commands in a vehicle with an automatic transmission. Therefore, we
also let the neural network predict this difficult-to-model force:

${\left[F_{x},B_{f},C_{f},D_{f},E_{f},B_{r},C_{r},D_{r},E_{r}\right]}^{\top}={\bf
z}_{4}.$ (10)

Then, the physics model (2) is followed by the last hidden layer to compute
the lateral forces of both the front and rear tires. We denote this layer as
the Pacejka layer. Subsequently, the output of the Pacejka magic formula is
forward propagated with the longitudinal force into the nominal dynamics of
the bicycle model [4]:

$\dot{{\bf
x}}_{t+1}=\left[\begin{array}[]{c}\frac{1}{m}\left(F_{x}-F_{y,f}\,{\sin(\delta(t))}+m{v_{y}(t)}{r(t)}\right)\\\
\frac{1}{m}\left(F_{y,r}+F_{y,f}\,{\cos(\delta(t))}-m{v_{x}(t)}{r(t)}\right)\\\
\frac{1}{I_{z}}\left(F_{y,f}\,{\cos(\delta(t))}\,{l_{f}}-F_{y,r}\,{l_{r}}\right)\end{array}\right],$
(11)

where $m$ and $I_{z}$ denote the vehicle’s mass and yaw moment of inertia,
respectively, and they are determined by the vehicle. We denote this layer as
the bicycle model layer. Finally, we can obtain the next state using the state
derivative:

${\bf x}_{t+1}={\bf x}_{t}+\dot{{\bf x}}_{t+1}\,{\Delta{t}}.$ (12)

Here, we emphasize that the DPM model still outputs the residual difference in
states, identical to the baseline network. In addition, the Pacejka formula
(2) and the nominal dynamics (11) are fully differentiable. This allows the
DPM to be trained in an end-to-end manner by minimizing precisely the same
objective used in the baseline network, such that $\left|\left|{\bf
x}_{t+1}-{\bf x}_{t}-{\bf F}(\bm{\mathcal{H}}_{t})\right|\right|^{2}$. Lastly,
our method is a general solution that can be utilized in other types of model
architectures, including Recurrent Neural Networks (RNN). Furthermore, it can
be formulated as a stochastic model or an ensemble model. An ablation study of
these considerations is, however, out of the scope of this paper.

In one line of research, the model learning problem is structured generally
with physical priors, but it does not require knowledge of the specific
physical embodiment [11, 12, 13]. Conversely, we explicitly embed system-
specific physics into the neural network as layers, inspired by certain ideas
introduced in vehicle trajectory prediction studies [19, 20]. The advantage of
taking this difference is that the forward propagation of the model can
produce latent variables that are typically difficult to obtain in the real
world. In our setting, we can implicitly acquire the lateral tire forces as
the output of the Pacejka layer. Note that directly measuring these forces
requires wheel force transducers, which are very expensive sensors. An
alternative way is to design a dedicated observer to estimate them [21, 22].
In contrast, the proposed method can obtain them with zero effort.

###  III-C Estimation Results

Throughout this paper, we use IPG Carmaker, a high-fidelity vehicle simulator
that solves nonlinear dynamics in real time, to validate our idea. We
collected $70$ minutes of training data by manually driving under various
conditions, with a data rate of $10$ Hz ($\Delta t=0.1$ s). The dataset
consists of four different types of maneuvers: 1) zig-zag driving at low
speeds, 2) high-speed driving, 3) driving to slide as much as possible, and 4)
random motions. 1) and 2) were collected on a race track with six sharp
corners. 3) and 4) were collected on flat ground. The maximum vehicle speed
$v_{max}$ was set to $40$ km/h. These maneuvers were performed on seven
different road surfaces, each with a distinct friction coefficient:
$\mu\in\\{0.4,0.5,\dots 1.0\\}$. A Volvo XC90 was used as the control vehicle.

We trained the FF-NN and the DPM for comparison with a fixed random seed.
Except for the physics embedding, the same settings were applied to both
models. We split the data into the training and test sets with a 7:3 ratio. A
Rectified Linear Unit (ReLU) was used for activation. The Mean Squared Error
(MSE) loss was used as the loss function and the networks were trained with
the Adam optimizer. We applied an early stop on the test set.

Figure 3: The test RMSE curves for the compared methods during training.

We use two error metrics for the performance evaluation: the root mean square
error (denoted as RMSE) and the max error (denoted as $E_{max}$). The
resulting test error curves for both methods are shown in Fig. 3. The results
show that the proposed method converged much faster than the FF-NN because the
physics prior had guided the learning problem. Our method also converged to a
better solution than the FF-NN. Concretely, the test errors for each state
element are shown in Table I.

TABLE I:  Estimation results using test dataset after training. |  $v_{x}$ [m/s] |  $v_{y}$ [m/s] |  $r$ [rad/s]
---|---|---|---
| RMSE | $E_{max}$ | RMSE | $E_{max}$ | RMSE | $E_{max}$
FF-NN | 0.0293 | 0.4451 | 0.0355 | 0.6793 | 0.0289 | 0.5956
DPM | 0.0299 | 0.3765 | 0.0214 | 0.4889 | 0.0169 | 0.4727

We also collected validation data to demonstrate the superiority of our method
in terms of generalization performance. The data were collected on the same
race track used to obtain the training data, but with adjusted friction
levels. The following values were assigned to the friction coefficients of
each corner, with none included in the training data:
$[0.45,0.55,0.65,0.75,0.85,0.95]$. The other regions were set to $\mu=1.0$.
Table II shows the validation errors for each state element in detail. It’s
worth noting that lateral dynamics estimations ($v_{y}$ and $r$) are more
critical for safe vehicle control; this demonstrates that the DPM has greater
generalization capabilities than the FF-NN by a large margin.

TABLE II:  Estimation results using validation dataset after training. |  $v_{x}$ [m/s] |  $v_{y}$ [m/s] |  $r$ [rad/s]
---|---|---|---
| RMSE | $E_{max}$ | RMSE | $E_{max}$ | RMSE | $E_{max}$
FF-NN | 0.0224 | 0.3136 | 0.0265 | 0.2358 | 0.0225 | 0.2179
DPM | 0.0234 | 0.3281 | 0.0128 | 0.1106 | 0.0122 | 0.1220
Figure 4: The prediction results of lateral tire forces on the validation
data. Figure 5: The estimation results of the cornering stiffness on the
validation data (in orange). The ground truth friction coefficient of the road
surface (in green) and the vehicle slip angle (in blue) are also visualized.
Note that we apply logarithmic scaling to the cornering stiffness.

Lastly, we visualize the latent features of lateral tire forces $F_{y}$ while
the DPM estimates with the validation data (see Fig. 4). The results show that
the extracted latent features closely match the ground truth values from the
simulator. It is important to point out that the model was never trained with
respect to $F_{y}$. Despite the absence of ground truth data for supervised
learning, our method can successfully estimate the parameters of the physics
model, resulting in accurate predictions on difficult-to-model forces.

###  III-D Extracting Latent Information for Risk-Aware Control

In this section, we describe the method of leveraging the latent variables to
provide useful information to the controller. The relationship curve between
the lateral tire force and sideslip angle under different friction levels is
shown in Fig. 6. As long as the tire slip angle is small, the lateral tire
force equation can be approximated by linearizing (2) as $F_{y}\approx
C_{\alpha}\,\alpha$, where $C_{\alpha}$ is the product of the tire parameters
$B\cdot C\cdot D$ and generally referred to as the cornering stiffness.
Although this approximation is effective for normal driving tasks in urban
environments, it is obvious that when the tire sideslip angle is large, the
lateral force begins to slip into the nonlinear area.

Figure 6: Lateral tire force models with different environments.

We aim at being sensible of situations in which the controlled vehicle may
lose its stability if the controller outputs more aggressive commands at the
critical moment. Here, we assume the following two cases: 1) situations in
which the lateral tire force is saturated into nonlinear regions despite a
high friction level (see Fig. 6b), and 2) situations in which the cornering
stiffness decreases due to a low friction level, regardless of the slip angle
(see Fig. 6c). Both of these cases can be characterized by a decrease in
$F_{y,r}/\alpha_{r}$. It can be seen in (1) that the equation of the front
slip angle contains the steering command, which can cause the resulting value
to be noisy. Therefore, we use the rear axle to define this information.

The estimated cornering stiffness $F_{y,r}/(\alpha_{r}+\epsilon)$ while the
DPM estimates with the validation data can be observed in Fig. 5 ($\epsilon\to
0$). The result shows that the estimated cornering stiffness drops at corners
with low friction coefficients. We propose to use this variable as a latent
risk metric. It is important to point out that the slip angle is not an
alternative to the proposed metric. The proposed risk metric reacts
differently depending on the road surface even at the same slip angle and
therefore it can quickly gain foresight into risky situations even with a
small slip angle (See Fig. 5a and Fig. 5b).

Taking cornering stiffness variations into account, the linearized tire model
can be expressed as follows:

$F_{y,r}=\left(C_{\alpha,th}-\Delta C_{\alpha}\right)\alpha_{r},$ (13)

where $\Delta C_{\alpha}$ is the uncertainty variables of the cornering
stiffness [23], and $C_{\alpha,th}$ is a threshold variable determined
empirically. Finally, we define the latent proprioceptive information yields
by the DPM as shown below:

$\Delta C_{\alpha}=\begin{cases}0,&\mbox{if
}C_{\alpha,th}<\left(\frac{F_{y,r}}{\alpha_{r}+\epsilon}\right)\\\
C_{\alpha,th}-\left(\frac{F_{y,r}}{\alpha_{r}+\epsilon}\right),&\mbox{otherwise}.\end{cases}$
(14)

##  IV Risk-Aware Controller Using Sampling-Based MPC

###  IV-A Smooth Model Predictive Path Integral

In this section, we provide a brief overview of the Smooth Model Predictive
Path Integral (SMPPI) algorithm [17]. Sampling-based MPC has been proposed to
optimize both convex and non-convex objectives using the forward dynamics
${\bf F}$. In particular, the Model Predictive Path Integral (MPPI) algorithm
introduced a powerful technique for evaluating thousands of sample
trajectories in a parallel manner using Graphic Processing Units (GPUs) [1].
Then, an optimal control sequence is produced by calculating the weighted
average of the sampled trajectories, with the weights defined by the cost of
each trajectory rollout. Due to its appealing characteristics, MPPI has been
successfully applied in various robotic applications.

However, the stochastic nature of such a sampling-based algorithm inevitably
causes chattering in the resulting commands. This problem becomes more
prominent when the MPPI attempts to respond to a rapidly changing environment.
SMPPI is an extended study of the MPPI algorithm, seamlessly combined with an
input-lifting strategy. This framework not only mitigates undesirable
chattering but can also effectively adapt to situations in which the optimal
control distribution deviates significantly from the previous iteration [17].

Following our prior work [17], the state-dependent cost function is defined as
follows:

$c({\bf x})={\text{Track}({\bf x})}+{\text{Slip}({\bf x})}+{\text{Speed}({\bf
x})}.$ (15)

The track cost guides the vehicle to drive inside the track boundary by
imposing an impulse-like penalty. The given two-dimensional cost map M returns
1 if the vehicle position is on the outside boundary of the track, and
otherwise returns 0.

$\text{Track}({\bf x})={10000\textbf{M}(p_{x},p_{y})}.$ (16)

The slip cost imposes both soft and hard costs on the slip angle to encourage
the controller to plan a stable future trajectory. $\alpha_{th}$ is an
empirically determined threshold.

$\displaystyle\text{Slip}({\bf x})$
$\displaystyle=\alpha^{2}+10000I\,(\\{\left|\alpha\right|>\alpha_{th}\\})$
(17) $\displaystyle\alpha$
$\displaystyle=-\text{arctan}\left(\frac{v_{y}}{\lVert v_{x}\rVert}\right),$

where $I$ is an indicator function. The speed cost is a standard quadratic
function to achieve the reference vehicle speed:

$\text{Speed}({\bf x})=(v_{x}-v_{ref})^{2}.$ (18)

###  IV-B Velocity-Dependent Control

To design a general risk-aware controller for autonomous driving tasks under
unknown friction, we leverage the latent proprioceptive information $\Delta
C_{\alpha}$ introduced in Section III-D. We found that the SMPPI algorithm is
an effective baseline for this purpose. The sampling-based MPC offers
significant advantages in terms of objective function flexibility, allowing
for easy encoding of high-level behavior. We formulate a velocity-dependent
controller, based on the SMPPI algorithm, that adaptively adjusts the
reference speed $v_{ref}$ according to the risk metric. This is a simple yet
effective strategy and it offers intuitive engineering parameters [24, 25].
The SMPPI controller was demonstrated in our prior work [17] to be capable of
generating optimal trajectories despite rapid changes in optimal distribution.
Concretely, the reference speed decreases from a maximum speed of $v_{max}=40$
km/h according to the following update law:

$v_{ref}=v_{max}-w_{1}\log{\left(w_{2}\,\Delta C_{\alpha}+w_{3}\right)},$ (19)

where the parameters $w_{(\cdot)}$ are empirically tuned. Except for this
risk-aware design, the same cost functions and control parameters were used
across all the experiments. It is important to point out that the SMPPI
controller can directly use either the FF-NN or the DPM as its forward
dynamics model.

##  V Experiments

###  V-A Experimental Setup

The purpose of this section is to validate our idea of applying the latent
proprioceptive information extracted from the physics-embedded neural network
to the controller, enabling more stable driving even in challenging
conditions. We designed two types of experiments.

####  V-A1 Circular track

First, we considered a path tracking experiment on a single-lane circular
track of radius $20$ m to the center of the lane. The width of the lane was
$16$ m. It was designed to evaluate quantitatively whether each controller can
track the reference trajectory well when continuous slipping occurs on a low-
friction road. We set the center line of the track as the target trajectory
and added accordingly a quadratic tracking error cost to the track cost
function:

$\text{Track}({\bf
x})=\left(\sqrt{{p_{x}}^{2}+{p_{y}}^{2}}-20\right)^{2}+{10000\textbf{M}\left(p_{x},p_{y}\right)}.$
(20)

####  V-A2 Race track

Second, we designed an autonomous driving experiment on a novel race track to
validate the effectiveness of our risk-aware strategy. We built a new race
track with a length of $790$ m in the simulator (see Fig. 9). The track
consisted of two types of road surfaces, one with a friction coefficient of
1.0 and the other with a friction coefficient of 0.4.

###  V-B Experimental Results

Figure 7: The vehicle trajectories taken by three different control methods.
The friction coefficient of the circular track was $0.45$. The black dashed
lines in the center of the tracks indicate the reference trajectories.

####  V-B1 Circular track

We evaluated the control performances of the MPC using the FF-NN and the DPM.
In addition, the DPM used with the risk-aware strategy (hereinafter referred
to as “DPM-R”) was also evaluated. Each controller was deployed to complete 10
laps around the circular track. We ended the trial if the vehicle lost control
and collided with the track boundary.

We observed that all controllers were able to follow the reference trajectory
perfectly on the roads with friction levels above $0.7$. Therefore, we
conducted experiments in detail with friction levels of
$\mu\in\\{0.65,0.6,0.55,0.5,0.45,0.4\\}$. As an evaluation metric, the RMSE of
tracking error was employed, which is the average distance the vehicle
deviates from the reference trajectory. We also measured how many laps the
vehicle completed without colliding with the track boundary. We referred to
this as course progress rate (CPR). The experimental results for each
controller are shown in Table III.

TABLE III:  Experimental results on the circular track. Average tracking errors (in [m]) and the course progress rate (CPR) with different control methods are displayed. | FF-NN | DPM | DPM-R
---|---|---|---
| RMSE | CPR | RMSE | CPR | RMSE | CPR
$\mu=0.65$ | 0.654 | 10/10 | 0.421 | 10/10 | 0.342 | 10/10
$\mu=0.6\,\,\,$ | 0.777 | 10/10 | 0.402 | 10/10 | 0.344 | 10/10
$\mu=0.55$ | 0.754 | 10/10 | 0.608 | 10/10 | 0.342 | 10/10
$\mu=0.5\,\,\,$ | 2.575 | 5/10 | 1.569 | 10/10 | 0.338 | 10/10
$\mu=0.45$ | N/A | 0/10 | 2.971 | 2/10 | 1.616 | 10/10
$\mu=0.4\,\,\,$ | N/A | 0/10 | N/A | 0/10 | 3.612 | 1/10

The results show that the controller using the DPM had lower tracking errors
than the one using the FF-NN at all friction levels. This is due to the fact
that the physics-based model provided strong guidance at the last hidden layer
of the DPM, enabling better generalization with limited training data. In
particular, the DPM outperformed the FF-NN on low-friction road surfaces by a
large margin. As described in Section III-C, the DPM has a greater lateral
dynamics estimation accuracy than the vanilla model, which accounts for this
observation.

Even though the modeling accuracy has improved, the performance of the
controller would inevitably deteriorate if the target speed remains constant
despite the low-friction levels. Alternatively, the DPM-R adjusted the target
speed according to the proprioceptive hints, resulting in lower tracking
errors than others. It also showed a robust result in the test with a friction
level of $\mu=0.45$. This strategy is effective in cases where safety is the
first priority. In the tests with $\mu=0.4$, the friction level was too low
that all controllers failed to retain their stabilities. We visualize the
trajectories taken by each controller in the tests with $\mu=0.45$ (see Fig.
7). The controller using the DPM traversed a longer distance than the one
using the FF-NN, but eventually collided with the track boundary. On the other
hand, the DPM-R reduced the vehicle speed and successfully followed the given
trajectory for 10 laps.

Figure 8: The vehicle trajectories taken by the compared controllers. The
friction coefficient of the gray region was $0.4$, and the remainder was
$1.0$.

####  V-B2 Race track

Here, we conducted an ablation study on the DPM-R method. We visualize the
vehicle trajectories taken by the controllers using the DPM and the DPM-R,
respectively, while passing through the first sharp corner of the track (see
Fig. 8). The friction coefficient was suddenly changed at the corner entry
($\mu:1.0\to 0.4$). The sampling-based controller with the DPM attempted to
respond to this corner by applying a large steering angle, but it failed
because the controller’s own deceleration commands were insufficient to get
through this slippery corner. On the contrary, the DPM-R was informed by the
latent proprioceptive information to reduce the target speed for minimizing
the risky sliding motions. As a result, the proposed risk-aware controller
completed 10 laps around the race track successfully (see Fig. 9). The
cornering stiffness uncertainty increased as the vehicle entered the corners
with low coefficients of friction, the controller then lowered the desired
speed to prevent significant sliding. Subsequently, after the vehicle entered
straight roads with high coefficients of friction, the controller increased
the target speed again as the uncertainty variable decreased. In other words,
the proposed control scheme was able to achieve both high-speed driving on
high-friction surfaces and conservative driving on low-friction surfaces, by
adaptively modifying the target speed based on the latent proprioceptive
information.

Figure 9: The vehicle trajectory taken by the controller using the DPM-R. The
friction coefficients of the white regions were $0.4$, and the remainder were
$1.0$.

##  VI Conclusion

We presented a physics embedded neural network vehicle model, which seamlessly
combines deep learning with physics-based models, structuring the model
learning problem by involving the well-known prior knowledge. The proposed
model outperformed the baseline neural network model by a large margin,
showing a faster learning speed, higher accuracy, and better generalization.
In addition, our model is capable of accurately estimating lateral tire
forces, which can be acquired from the forward propagation of the model,
without the need for any additional training with ground truth data. Based on
these latent features, we develop a risk-aware autonomous driving control
scheme designed to achieve both high-speed driving and conservative driving
depending on the environmental conditions. We conducted extensive experiments
in two autonomous driving tasks with unknown friction, demonstrating that the
proposed control framework shows much more robust performances than the
baseline framework. Future research should focus on transferring the proposed
model to a real-world vehicle using the Sim-to-Real approach.

## References

  * [1] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic MPC for model-based reinforcement learning,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 1714–1721.
  * [2] J. Kabzan, L. Hewing, A. Liniger, and M. N. Zeilinger, “Learning-based model predictive control for autonomous racing,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3363–3370, 2019.
  * [3] R. Rajamani, _Vehicle dynamics and control_. Springer Science & Business Media, 2011.
  * [4] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and dynamic vehicle models for autonomous driving control design,” in _IEEE Intelligent Vehicles Symposium (IV)_ , 2015, pp. 1094–1099.
  * [5] D. Nguyen-Tuong and J. Peters, “Model learning for robot control: a survey,” _Cognitive Processing_ , vol. 12, no. 4, pp. 319–340, 2011.
  * [6] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, “Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 7559–7566.
  * [7] N. O. Lambert, D. S. Drew, J. Yaconelli, S. Levine, R. Calandra, and K. S. J. Pister, “Low-Level Control of a Quadrotor With Deep Model-Based Reinforcement Learning,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 4224–4230, 2019.
  * [8] M. Janner, J. Fu, M. Zhang, and S. Levine, “When to Trust Your Model: Model-Based Policy Optimization,” in _Neural Information Processing Systems (NIPS)_ , 2019.
  * [9] D. Nguyen-Tuong and J. Peters, “Using model knowledge for learning inverse dynamics,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2010, pp. 2677–2682.
  * [10] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” _Journal of Computational Physics_ , vol. 378, pp. 686–707, 2019.
  * [11] M. Lutter, C. Ritter, and J. Peters, “Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [12] S. Greydanus, M. Dzamba, and J. Yosinski, “Hamiltonian neural networks,” in _Neural Information Processing Systems (NIPS)_ , vol. 32, 2019.
  * [13] M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho, “Lagrangian Neural Networks,” in _ICLR Workshop on Integration of Deep Neural Models and Differential Equations_ , 2020.
  * [14] H. B. Pacejka and E. Bakker, “The Magic Formula Tyre Model,” _Vehicle System Dynamics_ , vol. 21, pp. 1–18, 1992.
  * [15] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 1433–1440.
  * [16] N. A. Spielberg, M. Brown, N. R. Kapania, J. C. Kegelman, and J. C. Gerdes, “Neural network vehicle models for high-performance automated driving,” _Science Robotics_ , vol. 4, no. 28, 2019.
  * [17] T. Kim, G. Park, K. Kwak, J. Bae, and W. Lee, “Smooth Model Predictive Path Integral Control without Smoothing,” _IEEE Robotics and Automation Letters_ , 2022.
  * [18] T. Kim, H. Lee, S. Hong, and W. Lee, “TOAST: Trajectory Optimization and Simultaneous Tracking Using Shared Neural Network Dynamics,” _IEEE Robotics and Automation Letters_ , 2022.
  * [19] C. Tang, J. Chen, and M. Tomizuka, “Adaptive Probabilistic Vehicle Trajectory Prediction Through Physically Feasible Bayesian Recurrent Neural Network,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 3846–3852.
  * [20] H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, J. Schneider, D. Bradley, and N. Djuric, “Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 10 563–10 569.
  * [21] G. Baffet, A. Charara, and G. Dherbomez, “An Observer of Tire–Road Forces and Friction for Active Security Vehicle Systems,” _IEEE/ASME Transactions on Mechatronics_ , vol. 12, no. 6, pp. 651–661, 2007\.
  * [22] M. Acosta and S. Kanarachos, “Tire lateral force estimation and grip potential identification using Neural Networks, Extended Kalman Filter, and Recursive Least Squares,” _Neural Computing and Applications_ , vol. 30, no. 11, pp. 3445–3465, 2018.
  * [23] G. Baffet, A. Charara, and D. Lechner, “Estimation of tire-road forces and vehicle sideslip angle,” _Advances in robotics, automation and control_ , 2008.
  * [24] H. Du, N. Zhang, and F. Naghdy, “Velocity-dependent robust control for improving vehicle lateral dynamics,” _Transportation Research Part C: Emerging Technologies_ , vol. 19, no. 3, pp. 454–468, 2011.
  * [25] G. Kahn, A. Villaflor, V. Pong, P. Abbeel, and S. Levine, “Uncertainty-Aware Reinforcement Learning for Collision Avoidance,” in _arXiv preprint arXiv:1702.01182_ , 2017.