11institutetext: Niantic
https://nianticlabs.github.io/handdgp/

# HandDGP: Camera-Space Hand Mesh Prediction
with Differentiable Global Positioning

Eugene Valassakis Now at Synthesia. Work done while at Niantic. Guillermo
Garcia-Hernando

###### Abstract

Predicting camera-space hand meshes from single RGB images is crucial for
enabling realistic hand interactions in 3D virtual and augmented worlds.
Previous work typically divided the task into two stages: given a cropped
image of the hand, predict meshes in relative coordinates, followed by lifting
these predictions into camera space in a separate and independent stage, often
resulting in the loss of valuable contextual and scale information. To prevent
the loss of these cues, we propose unifying these two stages into an end-to-
end solution that addresses the 2D-3D correspondence problem. This solution
enables back-propagation from camera space outputs to the rest of the network
through a new differentiable global positioning module. We also introduce an
image rectification step that harmonizes both the training dataset and the
input image as if they were acquired with the same camera, helping to
alleviate the inherent scale-depth ambiguity of the problem. We validate the
effectiveness of our framework in evaluations against several baselines and
state-of-the-art approaches across three public benchmarks.

###### Keywords:

camera-space hand mesh estimation hand and body pose shape from RGB images
3D-to-2D scale ambiguity differentiable solver

##  1 Introduction

Predicting 3D hand meshes from single-view RGB images has become an
increasingly popular research area due to its potential in augmented and
virtual reality applications, such as virtual try-on experiences [49], human
digitization [16], gaming [22], and teleoperation [1, 18]. Despite recent
progress, challenges remain [3, 52] due to the hand’s articulated structure,
self-occlusions, annotation difficulty, and 2D-to-3D scale and depth
ambiguity.

Because of these challenges, most previous work have focused on predicting
quality root-relative hand meshes, _i.e_. 3D hand meshes in coordinates
relative to a pre-defined root joint, such as the wrist [14], as opposed to
predicting in the global camera space. Root-relative predictions with a camera
projection model [29, 54, 8, 4] can be sufficient in applications that end up
displayed on 2D images, such as virtual try-on experiences. However, camera-
space predictions are critical for interactions in 3D virtual and augmented
worlds, _e.g_. in applications such as gaming, and office work when using
mixed-reality headsets [37, 2].

Figure 1: Method overview: Through our Differentiable Global Positioning
module (DGP) which predicts the root translation of the hand, our method is
able to back-propagate through the root-finding operation, enabling an end-to-
end solution.

For tasks that need 3D camera-space hand meshes, the dominant approach is to
take those root-relative hand meshes and then lift those predictions to the
camera-space 3D coordinates in a separate, independent process. For example,
Iqbal  _et al_. [28] and Zhang  _et al_. [55] output predictions in a relative
2.5D space and infer the global coordinates analytically up to scale. With a
similar relative representation, Moon  _et al_. [40] and Tang  _et al_. [49]
predict camera-space coordinates with the aid of an independent network known
as RootNet [39]. Exploiting 2D-to-3D correspondences, CMR [15] and MobRecon
[14] first predict both 2D keypoints and 3D root-relative meshes, and then
find the 3D rigid transformation that best explains the mesh projection via an
independent test-time optimization process. Similarly, HandOccNet [41] also
uses a test-time optimization to predict scale and root translation by
minimizing a 2D projection loss. All these methods confine the learning stage
to the relative space, yielding state-of-the-art relative meshes with high
efficiency, but falling short when it comes to placing the hand in the camera
space, as shown in our experiments.

We propose to achieve the best of both worlds by simultaneously learning root-
relative meshes and the 3D lifting function in an end-to-end manner. To this
end, we propose a Differentiable Global Positioning (DGP) module, a modern
take on the classical Direct Linear Transform algorithm [23]. DGP enables the
backpropagation of gradients directly from camera space outputs to the 2D-3D
correspondences defined by the 2D keypoint predictions and the root-relative
3D hand meshes. Thanks to being differentiable, DGP could potentially also be
included in the learning of any hand mesh prediction neural network that
predicts 2D hand keypoints and root-relative 3D hand meshes such as CMR [15]
and MobRecon [14]. In our experiments, we show that allowing gradients to flow
from the camera-space through the root-relative network results in better
global hand mesh predictions compared to disjoint two-step approaches, _e.g_.
using RootNet [39] or test-time optimizations, as well as an end-to-end
regression baseline that predicts both camera-space translation and relative
predictions. Further, inspired by recent metric 3D scene geometry prediction
work [51], we found in our experiments that a simple image and camera
parameter rectification step helps to alleviate the inherent 2D-to-3D depth
and scale ambiguity problem. The core idea is to conduct all learning as
though the images were captured using the same camera model, thereby reducing
the ambiguity that the network must resolve. While this approach leads to
improved camera-space predictions, it incurs a slight decline in performance
for relative-space predictions. We thoroughly examine the impact of this
rectification step on both our method and on baseline approaches. In summary,
our contributions are as follows:

  * •

We propose HandDGP, a flexible framework that unifies the learning of both
root-relative and camera-space hand meshes in an end-to-end manner.

  * •

We identify and highlight the importance of performing image rectification in
alleviating some of the 2D-to-3D depth and scale ambiguity in the camera-space
hand mesh prediction problem.

  * •

We conduct an extensive experimental evaluation of various design choices for
addressing the problem of camera-space hand mesh prediction, aiming to
encourage research in this direction. This area has been somewhat overlooked
in previous work in favor of root-relative predictions.

##  2 Related Work

#### Camera-Space 3D Hand Mesh Prediction.

Most previous works in monocular RGB-based camera-space 3D hand mesh and pose
estimation follow a two-stage approach: (1) a first stage for predicting hand
mesh/pose in root-relative coordinates, and (2) a lifting stage to recover
camera-space coordinates from root-relative ones. For monocular 3D hand pose
estimation, Iqbal _et al_. [28] predicts in 2.5D root-relative space, and then
lifts those predictions to 3D camera space predictions using an analytical
solution. This result is up to a scaling factor, however, and to resolve the
scale ambiguity an extra scale parameter is required, which is assumed to be
provided [48] or globally estimated from data. I2L-MeshNet [40] proposes a
regression approach to recover root-relative 2.5D meshes and subsequently
lifts them to camera-space using a separate network RootNet [39] which uses
prior anthropometric knowledge to reduce scale ambiguity [34]. NFV [26]
proposes a neural voting approach with a 3D implicit function that directly
regresses 3D hand poses in camera-space from full images. NFV uses a Marching
Cubes post processing to predict meshes, which degrades efficiency and is at
the cost of not having semantic mappings for their predicted vertices, which
are crucial for some applications. Hasson  _et al_. [24] predicts both object
and hand camera-space translations using both hand and object cues, but makes
assumptions on the geometry of the object which facilitate the scale recovery.
Closest to our work, CMR [15], MobRecon [14] and HandOccNet [41] first predict
both 2D keypoints and 3D root-relative meshes, while 3D camera-space
coordinates are obtained with a test-time registration function that estimates
the root position. This function typically aims to find the 3D rigid
transformation for the hand mesh that best projects into the 2D input image
given correspondences between 2D hand keypoints and 3D mesh keypoints. Our
method builds on top of such 2D-3D paradigm with the key difference of
leveraging a differentiable registration function, enabling us to directly
learn our mesh network directly in the camera-space in an end-to-end manner.
We compare our work with [40, 39, 26, 24, 15, 14, 41] in Section 4.3 and find
that our method outperforms these state-of-the-art methods in camera-space
predictions.

#### Root-Relative 3D Hand Mesh Prediction.

Different methods have been proposed for RGB-based monocular hand mesh
reconstruction, with various hand output representations, such as parametric
models, voxels, vertices, implicit functions or UV maps [13]. Parametric
models [25, 8, 4, 5, 53, 15, 17, 10, 21] typically regress hand shape and pose
coefficients of the MANO parametric hand model [46]. These methods are
intrinsically limited by the expressiveness of the model used. Model-free
methods circumvent these limitations by working on high dimensional
representations. This can be true in voxel-based methods, such as I2L-MeshNet
[40], which represents volumetric hand data in a 2.5D manner at the cost of
efficiency. Also typically constrained by efficiency but at the benefit of
high resolution are implicit function methods [31, 30, 26], which inherit from
the trend started by human body digitization [47, 38, 6, 43, 26]. Vertex-based
methods [19, 32, 35, 15, 36, 14, 41] aim to directly predict 3D vertex
coordinates. In this work we build upon MobRecon’s framework [14] by adopting
their pipeline: predicting 2D keypoints following an encoder-decoder approach
then leveraging those keypoints to grid-sample features that are lifted to
root-relative 3D by a graph neural decoder. We extend this framework by
instead learning in camera-space owing to our proposed differentiable global
positioning function. In Section 4.1 we show that our method improves [14]’s
camera-space predictions significantly.

#### Differentiable Correspondence Solvers.

Solving for unknown geometric quantities using 2D-2D or 2D-3D correspondences
has long been a central subject in computer vision [44, 23]. With the
emergence of deep learning, several studies have attempted to integrate these
well known geometric and algebraic solutions to deep learning pipelines [45,
11, 9, 7, 50, 12]. Notable examples include (1) Chen _et al_. [11] which
propose a differentiable perspective-n-point (PnP) solver and validate it in
various problems such as pose estimation, or camera calibration, and Remelli
_et al_. [45] that use a differentiable Direct Linear Transform (DLT)
implementation to perform multi-view body pose estimation. We derive a DLT
solution to the root finding problem in the context of hand-mesh inference,
and show that it can be implemented differentiably and integrated to an end-
to-end pipeline for camera-space hand-mesh prediction.

Figure 2: HandDGP Framework Overview. Rectified images are passed through our
framework, which predicts camera-space coordinates using our proposed DGP
module.

##  3 Proposed Framework

Overview. Shown in Figure 2, the core idea behind our approach is to exploit
the geometry of the problem to integrate hand root finding in a differentiable
pipeline that can predict a hand mesh directly in 3D camera-space coordinates.

Predicting root-relative hand meshes. Starting from one RGB image
$\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, we first predict a set of 2D
keypoints $\mathcal{\mathbf{K}}^{2D}=\\{k^{2D}_{i}\\}_{i=1}^{N_{K}}$, that can
be joints or other landmarks, a set of root-relative 3D vertices
$\mathbf{V}^{rel}=\\{v_{i}^{rel}\\}_{i=1}^{N_{V}}$, and a set of weights
$\mathbf{W}=\\{w_{i}\\}_{i=1}^{N_{K}}$, that represent the confidence in the
predictions of each keypoint. We then obtain
$\mathbf{K}^{3D}=\\{k_{i}^{3D}\\}_{i=1}^{N_{K}}$, a set of root-relative $3D$
keypoints on the hand model that correspond to the 2D keypoints
$\mathbf{K}^{2D}$. To obtain $\mathbf{K}^{3D}$, we assume having access to a
3D keypoint regressor
$\mathbf{J}_{reg}:\mathbf{V}^{rel}\rightarrow\mathbf{K}^{3D}$.
$\mathbf{J}_{reg}$ usually comes in the form of a matrix, which defines
keypoints on the hand as a linear combination of mesh vertices, and is
typically provided with popular mesh models such as MANO [46] for hands and
SMPL [42] for full body meshes.
Finding the hand root with HandDGP. Our key innovation is our differentiable
global positioning (DGP or positioning module, for brevity), which uses the
root-relative 3D keypoints $\mathbf{K}^{3D}$, the 2D keypoints
$\mathbf{K}^{2D}$ and the weights $\mathbf{W}$ in order to predict a global
translation $\mathbf{t}\in\mathbb{R}^{3}$ in camera-space which we can use to
obtain the camera-space vertex predictions
$\mathbf{V}^{cs}=\\{v_{i}^{cs}\\}_{i=1}^{N_{V}}$, with

$v_{i}^{cs}=v_{i}^{rel}+\mathbf{t}.$ (1)

The camera-space vertex predictions can finally be used to project the mesh
into 2D using a pinhole camera perspective projection. This pipeline allows us
to include the global root translation and the resulting mesh projections as
part of our network training. Incorporating root prediction as part of the
training this way has the benefit of avoiding the accumulation of errors that
can occur when using two independent processes for root-relative predictions
and root-finding. We finally note that (1) our method is agnostic to the
particular method used to obtain the predictions for $\mathbf{V}^{rel}$,
$\mathbf{K}$ and $\mathbf{W}$ and (2) the task of recovering the root of the
hand is equivalent to finding the hand’s global translation in camera space.
Therefore, we will use both terms interchangeably throughout.

###  3.1 Differentiable Global Positioning

At the core of our method lies the differentiable global positioning module,
which takes in $\mathbf{K}^{3D}$, $\mathbf{K}^{2D}$, $\mathbf{J}_{reg}$, and
the camera intrinsics $\mathbf{\Lambda}$ as input, and outputs the global
camera-space translation $\mathbf{t}$ in a differentiable manner. Although our
full approach also considers the keypoint confidences $\mathbf{W}$, for
clarity, in this section we describe how we obtain the global translation
assuming equally confident keypoints. We explain how we incorporate keypoint
confidences in Section 3.2.

To obtain the global translation $\mathbf{t}=[\tau_{x},\tau_{y},\tau_{z}]^{T}$
in a differentiable way, we derive a solution based on the Direct Linear
Transform (DLT) [44, 23], adapted to our specific problem. Firstly, by design
$\mathbf{K^{3D}}$ and $\mathbf{K^{2D}}$ give us a set of 2D-3D correspondences
$\mathcal{M}=\\{(k_{i}^{3D},k_{i}^{2D})\\}_{i=1}^{N_{K}}$, with
$k_{i}^{3D}=[x_{i},y_{i},z_{i}]^{T}$ and $k_{i}^{2D}=[u_{i},v_{i}]^{T}$.
Additionally, it is important to note that the 3D keypoints $\mathbf{K}^{3D}$
are expressed in a frame that shares the same orientation as the camera frame,
with only the global root translation missing to map root-relative keypoint
coordinates to camera-space coordinates. Assuming a pinhole camera model with
intrinsic parameters $\mathbf{\Lambda}$, we can express the projection
equation as:

$\displaystyle d_{i}\begin{bmatrix}u_{i}\\\ v_{i}\\\ 1\end{bmatrix}$
$\displaystyle=\mathbf{\Lambda}\begin{bmatrix}1&0&0&\framebox{}&\tau_{x}\\\
0&1&0&\framebox{}&\tau_{y}\\\ 0&0&1&\framebox{}&\tau_{z}\\\
\end{bmatrix}\begin{bmatrix}x_{i}\\\ y_{i}\\\ z_{i}\\\ 1\end{bmatrix},$ (2)

with $d_{i}$ the depth value of keypoint $i$. Expanding and re-arranging, this
gives a system of linear equations that can be written in the following form:

$\displaystyle\begin{bmatrix}-1&0&u_{i}^{\prime}\\\ 0&-1&v_{i}^{\prime}\\\
\end{bmatrix}\begin{bmatrix}\tau_{x}\\\ \tau_{y}\\\ \tau_{z}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}x_{i}-z_{i}u_{i}^{\prime}\\\
y_{i}-z_{i}v_{i}^{\prime}\\\ \end{bmatrix},$ (3)

where

$\displaystyle\begin{bmatrix}u_{i}^{\prime}\\\ v_{i}^{\prime}\\\ 1\\\
\end{bmatrix}$ $\displaystyle=\mathbf{\Lambda}^{-1}\begin{bmatrix}u_{i}\\\
v_{i}\\\ 1\\\ \end{bmatrix}.$ (4)

Since Equation 3 is obtained considering a single keypoint correspondence, and
since we have three unknowns, it is under-constrained. Using all the keypoints
in our correspondence set, Equation 3 can be re-written as

$\displaystyle\begin{bmatrix}-1&0&u_{1}^{\prime}\\\ 0&-1&v_{1}^{\prime}\\\
&\vdots&\\\ 0&-1&v_{N_{K}}^{\prime}\\\ \end{bmatrix}\mathbf{t}$
$\displaystyle=\begin{bmatrix}x_{1}-z_{1}u_{1}^{\prime}\\\
y_{1}-z_{1}v_{1}^{\prime}\\\ \vdots\\\
y_{N_{K}}-z_{N_{K}}v_{N_{K}}^{\prime}\\\ \end{bmatrix},$ (5)

which has the form $\mathbf{At}=\mathbf{B}$. To solve for $\mathbf{t}$, we
consider the least-squares solutions
$\mathbf{t}^{*}=\operatorname*{arg\,min}_{\mathbf{t}}||\mathbf{A}\mathbf{t}-\mathbf{B}||^{2}$
[44], which can be obtained in closed-form:

$\mathbf{t^{*}}=(\mathbf{A}^{T}\mathbf{A})^{-1}\mathbf{A}^{T}\mathbf{B}.$ (6)

The DGP module first uses the network outputs to build the matrices
$\mathbf{A}$ and $\mathbf{B}$, and then uses the linear least-squares solution
to solve for the translation. Since all the operations involved are
differentiable, we can use this translation to obtain and backpropagate
through camera-space vertex predictions, fully incorporating the root-finding
task in an end-to-end training pipeline.

###  3.2 Keypoint Selection

While the solution presented in Section 3.1 allows us to incorporate root
finding into an end-to-end differentiable pipeline, it does not provide for
any outlier filtering or keypoint selection mechanism that could help filter
out more uncertain correspondences. This can be problematic in cases such as
occlusion or self-occlusion of parts of the hand. In those instances, occluded
parts would presumably result in more uncertain keypoint placements, they
would be considered equally to visible keypoints when computing the global
translation. To address this issue, we consider a weighted variant to our
approach. Assuming each keypoint correspondence has a confidence score $w_{i}$
associated with it it – in practice obtained from our weight decoder – we
first construct a weight matrix by duplicating each weight once and placing
them in a diagonal matrix
$\mathbf{W}=\operatorname{diag}([w_{1},w_{1},w_{2},w_{2}\ldots
w_{N_{K}},w_{N_{K}}])$. Then, we consider a weighted least-squares
minimisation
$\mathbf{t}^{*}=\operatorname*{arg\,min}_{\mathbf{t}}||\mathbf{W}(\mathbf{A}\mathbf{t}-\mathbf{B})||^{2}$,
with closed-form solution:

$\mathbf{t^{*}}=(\mathbf{A}^{T}\mathbf{W}^{2}\mathbf{A})^{-1}\mathbf{A}^{T}\mathbf{W}^{2}\mathbf{B}.$
(7)

###  3.3 Input Image Rectification

Inspired by the recent work on monocular 3D geometry estimation from Yin _et
al_. [51], the main idea is to establish a canonical camera space and
transform all the training data to that space. During inference, the image is
rectified just before entering the network, and the predictions are then
mapped back to the original camera space. The original set of camera
parameters is defined by $\\{f,u_{0},v_{0}\\}$ where $f$ represents the focal
length (we assume $f_{x}=f_{y}=f$) and $u_{0}$ and $v_{0}$ be the principal
points. We resize the input image $\mathbf{I}$ with the ratio $\omega_{r}$ =
$\frac{f^{c}}{f}$ which converts the camera parameters to
$\\{f^{c},\omega_{r}u_{0},\omega_{r}v_{0}\\}$. Different to [51] and motivated
by the hand being the object of interest, we further rectify the principal
point to be the center of the hand crop, resulting in the final canonical
intrinsics matrix $\mathbf{\Lambda}$ defined by $\\{f^{c},H/2,W/2\\}$ where
$f^{c}$ is the canonical focal length and the rectified principal point is the
center of the input image. It is important to note that this does not affect
the 3D geometry, thus root $\mathbf{t}$ remains unchanged.

###  3.4 Architecture and Training Details

Network Overview. In practice, our network first takes an image
$\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ as an input to a convolutional
encoder to produce a feature map $\mathbf{F}\in\mathbb{R}^{H/32\times
W/32\times C}$. The feature map $\mathbf{F}$ is then input to three separate
decoder heads: a 2D decoder outputting a set of $N_{K}$ 2D keypoints
$\mathbf{K}^{2D}$, a 3D decoder outputting the root-relative vertices
$\mathbf{V}^{rel}$, and a weights decoder outputting a set of confidence
weights $\mathbf{W}$. Using $\mathbf{J}_{reg}$, we obtain the root-relative 3D
keypoints $\mathbf{K}^{3D}$, forming a set of 2D-3D correspondences
$\mathcal{M}=\\{(u_{i},v_{i},x_{i},y_{i},z_{i})\\}_{i=1}^{N_{K}}$. Using
$\mathcal{M}$, we then construct the matrices $\mathbf{A}$ and $\mathbf{B}$,
and obtain $\mathbf{t}$ using Equations 6 and 7. Finally, we use $\mathbf{t}$,
$\mathbf{V}^{rel}$ and $\mathbf{K}^{3D}$ to obtain the camera-space vertices
$\mathbf{V}^{cs}$ and camera-space keypoints, following Equation 1, and use
all of the mentioned network outputs to construct our training losses.

Figure 3: Weight decoder head.

Weights decoder. While our 2D and 3D decoders follow MobRecon [14], our
weights decoder is illustrated in Fig. 3. Starting from the feature map
$\mathbf{F}\in\mathbb{R}^{H/32\times W/32\times C}$, we perform a series of
$1\times 1$ convolutions to obtain a new feature map
$\mathbf{F_{W}}\in\mathbb{R}^{H/32\times W/32\times D}$. We then use the 2D
positions provided by $\mathbf{K}^{2D}$ in order to grid sample a set of
$N_{K}$, $D$-dimentional features, which we concatenate in $D\times N_{K}$
dimensional latent vector $\mathbf{Z_{w}}$ which is then processed through a
set of dense layers with leaky ReLU activations, and the final output is
processed through a sigmoid function, forcing the confidence weights to be in
$[0,1]$.
Losses. We distinguish between two sets of losses: Relative-space losses that
are applied to any outputs that precede our global positioning module, and
camera-space losses that follow the global positioning module. Relative-space:
we follow MobRecon [14], and use (1) a root-relative 3D Vertex $L_{1}$ Loss
between the root-relative ground truth vertices and the outputs of the 3D
decoder, (2) a 2D keypoint $L_{1}$ loss on the outputs of the 2D decoder, (3)
a consistency loss that enforces consistency of outputs between inputs with
different visual augmentations, (4) a $L_{1}$ edge loss on the length of the
predicted mesh edges, and (5) a normal loss on the predicted mesh normals. For
our camera-space losses, we use (1) a root mean squared error loss on the
translation, (2) a keypoint consistency loss, which is a $L_{1}$ loss between
the outputs of our 2D decoder and the projection of the predicted camera-space
3D keypoints, and (3) a 2D vertex $L_{1}$ loss on the 2D projection of the 3D
camera-space vertices.

Implementation details and additional method information, including losses and
network architecture details, are available in the supplementary material.

##  4 Experiments

Datasets. We report our experiments on the following datasets:

  * •

FreiHAND [56]. We follow the evaluation protocol by [15, 26] using the dataset
for our experiments. The dataset consists of images, 3D hand poses and MANO
[46] fittings. It provides 130,240 training and 3,960 test images.

  * •

HO3D-v2 [20]. This dataset comprises real images capturing 3D hand-object
interactions, with 66,034 images in the training set and 11,524 in the test
set with MANO [46] model hand mesh annotations. Hands in this dataset suffer
from severe occlusions caused by the manipulated object. The test set is not
publicly available, and the evaluation is conducted on a public server. The
server provides results in camera-space, root-relative, and aligned formats.
However, participants are given ground truth camera-space hand translation
values, and previous work typically reports results using this ground truth.

  * •

Human3.6M [27] This dataset is a large-scale 3D body pose benchmark containing
3.6 million frames with annotations of 3D joint coordinates and SMPL [42]
meshes. We follow existing evaluation protocols [15], but do not use common
aligned metrics and measure errors in camera-space. We adapt our framework to
predict body meshes by just swapping MANO by SMPL.

Metrics. We report the following metrics:

  * •

CS-MJE / CS-MVE: Measures the error, in terms of Euclidean distance, between
the predicted joints (MJE) / vertices (MVE) and the ground truth in camera-
space (CS) coordinates. Both are average errors over the test set in mm. In
some experiments, we compute the Area-Under-the-Curve (AUC) of Percentage of
Correct Keypoints (PCK) vs. error thresholds.

  * •

RS-MJE / RS-MVE: This measures the error between Procrustes-aligned predicted
and ground truth joints and vertices. This metric serves as a measure of root-
relative reconstruction quality.

###  4.1 Baselines and Method Ablations: Descriptions

Method |  | Root-Relative | Camera-Space
---|---|---|---
| Image
---
Rectification
| End-to-End
---
Training
RS-MJE$\downarrow$ | RS-MVE$\downarrow$ | CS-MJE$\downarrow$ | CS-MVE$\downarrow$
B1. Baseline + PnP | ✗ | ✗ | 6.8 | 6.9 | 50.1 | 50.2
B2. Baseline + DLT | ✗ | ✗ | 6.8 | 6.9 | 50.0 | 50.0
B3. Baseline + RootNet [39] | ✗ | ✗ | 6.8 | 6.9 | 62.6 | 62.5
B4. Baseline + Optimization [14] | ✗ | ✗ | 6.8 | 6.8 | 50.2 | 50.3
B5. Root regression | ✗ | ✓ | 7.2 | 7.6 | 81.3 | 81.8
BR1. Baseline + DLT + Rect. | ✓ | ✗ | 7.4 | 7.5 | 48.4 | 48.8
BR2. Baseline + Optimization [14] \+ Rect. | ✓ | ✗ | 7.4 | 7.5 | 48.9 | 49.0
BR3. Root regression + Rect. | ✓ | ✓ | 7.8 | 7.7 | 52.6 | 55.4
A4. Ours - w/o Rect. | ✗ | ✓ | 6.8 | 6.9 | 49.4 | 49.4
Ours (Full framework) | ✓ | ✓ | 7.4 | 7.6 | 46.3 | 46.3

Table 1: Baseline and ablation experiments on FreiHAND dataset [56]. The
‘Image Rectification’ column indicates whether the training images are
rectified with our proposed approach. ‘End-to-end Training’ denotes whether
gradients flow through the global positioning function during training. The
task we care about for 3D interactions is quality in the camera space where we
outperform all the baselines and variants.

Baselines. Our root-relative module is combined with different methods for
predicting camera-space root translation. Methods labeled as ‘Baseline + {root
prediction method}’ incorporate the root-relative module (see Fig. 2) without
any global positioning system during training. Therefore, only the relative-
space losses are applied and the positioning of the hand mesh in camera space
occurs only at test time. In ‘B1. Baseline + PnP’, the global translation is
obtained by solving the perspective-n-point (PnP) problem [44], using the
outputs of our 2D decoder and the 3D root-relative keypoints as 2D-3D
correspondences. ‘B2. Baseline + DLT’ refers to our baseline wherein our
formulation of the DLT module is applied at test time only during a forward
pass, i.e., without gradient propagation through the network. In ‘B3. Baseline
+ RootNet’, the global translation is determined using RootNet [39]. ‘B4.
Baseline + Optimization’ corresponds to our reimplementation of MobRecon[14],
where the global translation is identified via an optimization-based process
that minimizes the re-projection errors of predicted keypoints. Finally, ‘B5.
Root Regression’ is the only end-to-end baseline that employs a decoder,
similar to our weight decoder shown in Fig.3, but for regressing the camera-
space root value $\mathbf{t}$ and incorporating a translation loss. We also
implemented some of these baselines with our image rectification step, denoted
as BR1-3. Results are summarized in Table 1.
Ablations. We conduct several ablation studies on our full pipeline, as shown
in Table 2. Ours (Full Framework) is our proposed method, which includes
performing image rectification on the input image and conducting
differentiable global positioning during training, with gradients back-
propagating through the DGP module. ‘A1. Ours - w/o DGP - w/o Rectification’
represents our framework without the proposed DGP and rectification steps,
following the setup of B4. ‘A2. Ours - w/o DGP’ evaluates the impact of our
image rectification step in comparison to A1. In the ‘A3. Ours - w/o Keypoint
Weights $\mathbf{W}$’ experiment, we utilize our full framework but, instead
of predicting keypoint confidences using our weight decoder, manually set all
the weights to $1.0$. Lastly, in A4. No Rectification + DGP as detailed in
Table 1, we implement the full DGP module but exclude the image rectification
from the pipeline.

Method | FreiHAND | HO3D-v2 | Human 3.6M
---|---|---|---
CS-MJE$\downarrow$ | CS-MVE$\downarrow$ | CS-MJE$\downarrow$ | CS-MVE$\downarrow$ | CS-MJE$\downarrow$ | CS-MVE$\downarrow$
A1. Ours - w/o DGP - w/o rectification | 50.2 | 50.3 | 121.7 | 121.6 | 336.9 | 342.8
A2. Ours - w/o DGP | 48.9 | 49.0 | 85.3 | 85.4 | 164.7 | 179.3
A3. Ours - w/o keypoint weights $\mathbf{W}$ | 46.6 | 46.6 | 50.4 | 50.4 | 159.9 | 174.0
Ours (Full framework) | 46.3 | 46.3 | 50.3 | 50.3 | 147.6 | 162.0

Table 2: Method ablation experiments. We outperform all ablations on camera-
space predictions across all datasets, validating our design choices and use
of HandDGP.

###  4.2 Baselines and Method Ablations: Results Discussion

Learning the 3D Global Positioning Function. In this evaluation, we assess the
effect of back-propagating gradients from the DGP to the network, as proposed
in Section 3.1. In Table 1 we observe that DGP outperforms all the other root-
finding mechanisms. Non-learning approaches such as PnP, DLT and the
optimization from Chen  _et al_. [14] (B1, B2, B4) offer better predictions
than learning-based methods such as RootNet [39] and our regression baseline
(B5). These results still hold when image rectification is applied (BR1-3).
Notably, the DGP formulation, even without training (forward pass), achieves
comparable results to more sophisticated optimization methods, validating our
design choice. In Table 2 we observe a significant reduction in error in all
the datasets when learning the global positioning function with the proposed
DGP (A2). For example, the camera-space vertex error is reduced by $2.7$ mm,
$35.1$ mm and $17.3$ mm in FreiHAND, HO3D-v2 and Human3.6M datasets
respectively. This last result shows that HandDGP also transfers well to full
body mesh predictions.

Figure 4: Keypoint selection. Effect of keypoint selection with our weight
decoder. Test-set images on FreiHAND and HO3D-v2 with the 2D keypoints
overlaid: The brighter the keypoint, the higher the weight.

Image Rectification Effect. We observe that rectifying input images enhances
the learning of scale-sensitive features, as we remove one source of ambiguity
by keeping the camera intrinsics constant during training. Interestingly, we
note that while rectifying images aids in camera space predictions, it impacts
root-relative predictions (Ours vs A4). We hypothesize that the primary reason
for this degradation is that rectifying training images de facto reduces the
amount of data augmentations that the network encounters during training,
affecting the prediction of both 2D keypoints and 3D mesh geometry. This
suggests a potential trade-off between root-relative and absolute prediction
quality, depending on how the training data is processed, exacerbated by the
2D-to-3D depth and scale ambiguity. The network may generate incorrect hand
shapes to compensate for errors in translation prediction and vice versa. In
all cases, better camera-space positioning is achieved when the input image is
rectified.

Keypoint Selection. In Table 2 we observe that learning the keypoint selection
weights $\mathbf{W}$ introduced in Eq. 7, instead of leaving them fixed
further helps reducing the prediction error (Ours vs. A3). We illustrate this
effect qualitatively in Fig. 4. For images in the test set, we draw the
keypoints predicted by our 2D decoder onto the input image, color-coding each
keypoint using its associated weight, as output by our weight decoder. The
brighter the keypoint, the higher the weight associated with it. We also
illustrate the weights themselves associated with each image in the bottom row
of the figure. The first observation is that the weights associated with each
keypoint clearly vary from image to image, indicating that our weight decoder
actively contributes to our overall pipeline. Next, we notice that occluded
keypoints generally tend to be associated with lower weights, suggesting that
our method tends to focus on higher confidence keypoints.

Qualitative Results. In Fig. 6 we present qualitative results using our full
framework on test images from different datasets. In most cases, the hand
projections appear accurate, as also quantitatively demonstrated in our
relative results in Table 1. Additionally, we showcase some rare failure cases
where high ambiguity arises due to blurring, changes in viewpoint, and self-
occlusion. Merely displaying projected 2D meshes might not convey the complete
truth, as predictions are made in the actual 3D camera space. To address this,
we include predicted 3D meshes from rotated viewpoints to illustrate the
actual translation gap between the predictions and the ground truth in Fig. 5
(b). While the hand posture is generally correct, we observe instances of
translation errors, sometimes overcompensating with larger or smaller hand
shapes.

Figure 5: (a) 3D PCK for camera-space hand mesh prediction on FreiHAND. (b)
Camera-space hand mesh predictions rotated for illustration purposes. All
meshes project correctly in the image, however some predictions display a 3D
error offset. (c) Root-relative vs camera-space errors. Selected FreiHAND
images with average camera-space (CS) and root-relative (RS) errors and ground
truth (mesh in white).

Root-relative results discussion. While not the focus of this work, Table 1
shows our method is outperformed by some baselines in the root-relative task
by an average of 0.6 mm (RS-MVE). However, our improvements on camera-space
errors (CS-MVE) are about 4 mm. To assess if this RS-MVE increase is an
acceptable trade-off, Fig. 5 (c) compares our baseline and HandDGP on the
FreiHAND test set. We selected images with a CS-MVE decrease of about 4 mm and
an RS-MVE increase of at least 0.6 mm for MobRecon over HandDGP. The CS-MVE
improvement is more evident when rotating the 3D viewpoint (columns 2-3). Our
meshes are visibly closer to the ground truth, crucial for applications
requiring interaction with real and digital objects. Higher RS-MVE (0.84-1.68
mm) are harder to visualize. Notably, a better RS error doesn’t always result
in a better image projection (last row) and can hide scale errors due to
Procrustes alignment. We believe the 4 mm CS-MVE improvement is significant,
and our visualizations suggest a 0.6 mm increase in RS errors is an acceptable
trade-off.

Framework Generalizability. In the supp. material we show the benefits of
applying our framework to a different state-of-the-art root-relative method
[15].

###  4.3 Comparison with State of the Art

Method | CS-MJE$\downarrow$ | CS-MVE$\downarrow$
---|---|---
ObMan [25] | 85.2 | 85.2
MANO CNN [56] | 71.3 | 71.5
I2L-MeshNet [40] | 60.3 | 60.4
NVF [26] | 47.2 | n/a†
CMR-SG-RN18 [15] | 49.7 | 49.8
CMR-SG-RN50 [15] | 48.8 | 48.9
MobRecon-RN50 [14] | 50.2 | 50.3
Ours | 46.3 | 46.3

Table 3: State of the art comparison on FreiHAND. †can only be evaluated for
keypoints.

FreiHAND: In Table 3 and Fig. 5 (a) we compare our proposed framework with
state-of-the-art camera-space methods. Notably, our method achieves the lowest
camera-space errors among all methods. We achieve a 2.6 mm error improvement
over CMR [15], despite also using segmentation masks in their root finding. We
also compare with the ResNet50 variant of MobRecon [14] which is the same as
our baseline B4 from the previous section. For MobRecon, we report our results
based on our implementation, which closely aligns with the one provided by the
authors, with slight changes in the data processing that are also shared with
our method. A similar trend is observed in Figure 5 where our method achieves
the highest AUC of vertex PCK’s among all methods closely followed by both CMR
and MobRecon-RN-50. We also compare favorably with NVF [26], the only
available method that, similar to us, predicts directly in camera space. Note
that NVF does not directly predict hand meshes, making them not easily
comparable due to the loss of the MANO topology as their meshes are generated
using Marching Cubes. We observe that the root-relative + 2D-3D global
positioning paradigm (ourselves, CMR and MobRecon) performs significantly
better than other methods, followed by I2L-MeshNet [40] that uses [39] for
root positioning, similar to our baseline B3.

Method | CS-MJE$\downarrow$ | CS-MVE$\downarrow$
---|---|---
HandOccNet [41] | 156.4 | 156.2
MobRecon-RN50 [14] | 121.7 | 121.6
Hasson _et al_. [24] | 55.2 | 55.1
Ours | 50.3 | 50.3

Table 4: State of the art comparison on HO3D-v2.

HO3D-v2: In Table 4 we present quantitative results of our method compared to
state-of-the-art methods in camera-space coordinates using the public
submission server. It is important to note that previous work typically
reports their results in relative coordinates after an aligning step and often
uses provided ground-truth root values. We do not have a way to know which
participants on the leaderboard used this ground-truth; because of this, we
had to recompute the scores. We report results of the available subset of
methods that: i) have publicly available code, ii) provide a trained model,
and iii) include a global coordinate prediction stage. This dataset is more
challenging than FreiHAND as it is object-focused, and several occlusions are
present. We observe that our method compares favorably to HandOccNet[41],
which is currently the state of the art in relative pose predictions. To
compute these predictions, we run their code that performs test-time
optimization to predict the root translation. Despite their mesh projections
looking good, the 3D predictions are often incorrect, with errors in the order
of 15 cm. We also compare favorably to the work of Hasson  _et al_. [24] that
predicts both object and hand global translations. Given that object size is
constant—compared to hands—predicting object roots is likely to help with the
scale/depth ambiguity problem. We show qualitative comparisons both in the
supplementary material and in the video presentation.

Figure 6: Qualitative visualizations of our camera-space mesh prediction
framework on FreiHAND, HO3D-v2, Human3.6M and in-the-wild web images [33].
Meshes are projected into the image plane using perspective projection,
including failure cases. Further qualitative results and comparisons are
available in the supplementary material.

##  5 Conclusion

We presented a framework for camera space hand mesh prediction, enabling
learning directly in camera space. Our baseline and ablation studies validated
our design choices, showing our method surpasses state-of-the-art approaches
that predict hand meshes in camera-space coordinates. Estimating absolute 3D
geometry from a single RGB image is inherently ill-posed. Rectifying images
and predicting in camera space help reduce errors.

Our experiments show that while root-relative error is in the low single-digit
millimeters, likely lower than annotation error, camera space error is 6 to 7
times larger. This is visually illustrated in Fig. 5 (b) and (c), suggesting a
significant portion of total errors stems from 2D-to-3D depth ambiguity. We
conjecture that isolating the hand from its context will soon reach a
performance ceiling. Further research in new datasets and context-aware
approaches, such as using scene geometry or objects, is needed to advance
camera-space mesh inference.

## Acknowledgements

We would like to thank Filippo Aleotti for his help with baseline experiments
and infrastructure; Jamie Watson, Zawar Qureshi, and Jakub Powierza for their
help with infrastructure; Axel Laguna for his insightful discussions on
minimal solvers and network architectures; Daniyar Turmukhambetov for valuable
technical discussions; and Gabriel Brostow, Sara Vicente, Jessica Van
Brummelen, and Michael Firman for their valuable feedback on different
versions of the manuscript.

## References

  * [1] Antotsiou, D., Garcia-Hernando, G., Kim, T.K.: Task-oriented hand motion retargeting for dexterous manipulation imitation. In: ECCV Workshop (2018)
  * [2] Apple: Vision Pro. https://www.apple.com/apple-vision-pro/, [Online; accessed 7-March-2024]
  * [3] Armagan, A., Garcia-Hernando, G., Baek, S., Hampali, S., Rad, M., Zhang, Z., Xie, S., Chen, M., Zhang, B., Xiong, F., et al.: Measuring generalisation to unseen viewpoints, articulations, shapes and objects for 3D hand pose estimation under hand-object interaction. In: ECCV (2020)
  * [4] Baek, S., Kim, K.I., Kim, T.K.: Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In: CVPR (2019)
  * [5] Baek, S., Kim, K.I., Kim, T.K.: Weakly-supervised domain adaptation via gan and mesh model for estimating 3D hand poses interacting objects. In: CVPR (2020)
  * [6] Bhatnagar, B.L., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. In: NeurIPS (2020)
  * [7] Bhowmik, A., Gumhold, S., Rother, C., Brachmann, E.: Reinforced feature points: Optimizing feature detection and description for a high-level task. In: CVPR (2020)
  * [8] Boukhayma, A., Bem, R.d., Torr, P.H.: 3d hand shape and pose from images in the wild. In: CVPR (2019)
  * [9] Brachmann, E., Krull, A., Nowozin, S., Shotton, J., Michel, F., Gumhold, S., Rother, C.: Dsac-differentiable ransac for camera localization. In: CVPR (2017)
  * [10] Cao, Z., Radosavovic, I., Kanazawa, A., Malik, J.: Reconstructing hand-object interactions in the wild. In: CVPR (2021)
  * [11] Chen, B., Parra, A., Cao, J., Li, N., Chin, T.J.: End-to-end learnable geometric vision by backpropagating pnp optimization. In: CVPR (2020)
  * [12] Chen, H., Wang, P., Wang, F., Tian, W., Xiong, L., Li, H.: Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation. In: CVPR (2022)
  * [13] Chen, P., Chen, Y., Yang, D., Wu, F., Li, Q., Xia, Q., Tan, Y.: I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling. In: ICCV (2021)
  * [14] Chen, X., Liu, Y., Dong, Y., Zhang, X., Ma, C., Xiong, Y., Zhang, Y., Guo, X.: Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In: CVPR (2022)
  * [15] Chen, X., Liu, Y., Ma, C., Chang, J., Wang, H., Chen, T., Guo, X., Wan, P., Zheng, W.: Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration. In: CVPR (2021)
  * [16] Chen, X., Wang, B., Shum, H.Y.: Hand avatar: Free-pose hand animation and rendering from monocular video. In: CVPR (2023)
  * [17] Chen, Y., Tu, Z., Kang, D., Bao, L., Zhang, Y., Zhe, X., Chen, R., Yuan, J.: Model-based 3d hand reconstruction via self-supervised learning. In: CVPR (2021)
  * [18] Garcia-Hernando, G., Johns, E., Kim, T.K.: Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning. In: IROS (2020)
  * [19] Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., Yuan, J.: 3d hand shape and pose estimation from a single rgb image. In: CVPR (2019)
  * [20] Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: A method for 3D annotation of hand and object poses. In: CVPR (2020)
  * [21] Hampali, S., Sarkar, S.D., Rad, M., Lepetit, V.: Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In: CVPR (2022)
  * [22] Han, S., Liu, B., Cabezas, R., Twigg, C.D., Zhang, P., Petkau, J., Yu, T.H., Tai, C.J., Akbay, M., Wang, Z., et al.: Megatrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM TOG (2020)
  * [23] Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
  * [24] Hasson, Y., Tekin, B., Bogo, F., Laptev, I., Pollefeys, M., Schmid, C.: Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In: CVPR (2020)
  * [25] Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M.J., Laptev, I., Schmid, C.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)
  * [26] Huang, L., Lin, C.C., Lin, K., Liang, L., Wang, L., Yuan, J., Liu, Z.: Neural voting field for camera-space 3D hand pose estimation. In: CVPR (2023)
  * [27] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI (2013)
  * [28] Iqbal, U., Molchanov, P., Breuel Juergen Gall, T., Kautz, J.: Hand pose estimation via latent 2.5D heatmap regression. In: ECCV (2018)
  * [29] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: CVPR (2018)
  * [30] Karunratanakul, K., Spurr, A., Fan, Z., Hilliges, O., Tang, S.: A skeleton-driven neural occupancy representation for articulated hands. In: 3DV (2021)
  * [31] Karunratanakul, K., Yang, J., Zhang, Y., Black, M.J., Muandet, K., Tang, S.: Grasping field: Learning implicit representations for human grasps. In: 3DV (2020)
  * [32] Kulon, D., Guler, R.A., Kokkinos, I., Bronstein, M.M., Zafeiriou, S.: Weakly-supervised mesh-convolutional hand reconstruction in the wild. In: CVPR (2020)
  * [33] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV (2020)
  * [34] Li, Z., Liu, J., Zhang, Z., Xu, S., Yan, Y.: Cliff: Carrying location information in full frames into human pose and shape estimation. In: ECCV (2022)
  * [35] Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: CVPR (2021)
  * [36] Lin, K., Wang, L., Liu, Z.: Mesh graphormer. In: ICCV (2021)
  * [37] Meta: Quest 3. https://www.meta.com/us/quest/quest-3/, [Online; accessed 7-March-2024]
  * [38] Mihajlovic, M., Zhang, Y., Black, M.J., Tang, S.: Leap: Learning articulated occupancy of people. In: CVPR (2021)
  * [39] Moon, G., Chang, J.Y., Lee, K.M.: Camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. In: ICCV (2019)
  * [40] Moon, G., Lee, K.M.: I2L-MeshNet: Image-to-lixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In: ECCV (2020)
  * [41] Park, J., Oh, Y., Moon, G., Choi, H., Lee, K.M.: Handoccnet: Occlusion-robust 3d hand mesh estimation network. In: CVPR (2022)
  * [42] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3D hands, face, and body from a single image. In: CVPR (2019)
  * [43] Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q., Bao, H., Zhou, X.: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: CVPR (2021)
  * [44] Prince, S.J.: Computer vision: models, learning, and inference. Cambridge University Press (2012)
  * [45] Remelli, E., Han, S., Honari, S., Fua, P., Wang, R.: Lightweight multi-view 3D pose estimation through camera-disentangled representation. In: CVPR (2020)
  * [46] Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM TOG (2017)
  * [47] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: CVPR (2019)
  * [48] Spurr, A., Iqbal, U., Molchanov, P., Hilliges, O., Kautz, J.: Weakly supervised 3d hand pose estimation via biomechanical constraints. In: ECCV (2020)
  * [49] Tang, X., Wang, T., Fu, C.W.: Towards accurate alignment in real-time 3D hand-mesh reconstruction. In: ICCV (2021)
  * [50] Wei, T., Patel, Y., Shekhovtsov, A., Matas, J., Barath, D.: Generalized differentiable ransac. In: ICCV (2023)
  * [51] Yin, W., Zhang, C., Chen, H., Cai, Z., Yu, G., Wang, K., Chen, X., Shen, C.: Metric3D: Towards zero-shot metric 3D prediction from a single image. In: ICCV (2023)
  * [52] Yuan, S., Garcia-Hernando, G., Stenger, B., Moon, G., Chang, J.Y., Lee, K.M., Molchanov, P., Kautz, J., Honari, S., Ge, L., et al.: Depth-based 3D hand pose estimation: From current achievements to future goals. In: CVPR (2018)
  * [53] Zhang, X., Huang, H., Tan, J., Xu, H., Yang, C., Peng, G., Wang, L., Liu, J.: Hand image understanding via deep multi-task learning. In: ICCV (2021)
  * [54] Zhang, X., Li, Q., Mo, H., Zhang, W., Zheng, W.: End-to-end hand mesh recovery from a monocular RGB image. In: ICCV (2019)
  * [55] Zhou, Y., Habermann, M., Xu, W., Habibie, I., Theobalt, C., Xu, F.: Monocular real-time hand shape and motion capture using multi-modal data. In: CVPR (2020)
  * [56] Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M., Brox, T.: FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In: ICCV (2019)


# Neuroscheduling for Remote Estimation

Marcos M. Vasconcelos and Yifei Zhang M. M. Vasconcelos and Y. Zhang are with
the Department of Electrical and Computer Engineering, FAMU-FSU College of
Engineering, Florida State University, Tallahassee, FL 32306, USA. E-mails:
<EMAIL_ADDRESS>yz23r<EMAIL_ADDRESS>
###### Abstract

Many modern distributed systems consist of devices that generate more data
than what can be transmitted via a communication link in near real time with
high-fidelity. We consider the scheduling problem in which a device has access
to multiple data sources, but at any moment, only one of them is revealed in
real-time to a remote receiver. Even when the sources are Gaussian, and the
fidelity criterion is the mean squared error, the globally optimal data
selection strategy is not known. We propose a data-driven methodology to
search for the elusive optimal solution using linear function approximation
approach called neuroscheduling and establish necessary and sufficient
conditions for the optimal scheduler to not over fit training data.
Additionally, we present several numerical results that show that the globally
optimal scheduler and estimator pair to the Gaussian case are nonlinear.

##  I Introduction

A fundamental question in networked systems is: if an agent has more data than
what can be transmitted over a link at any given time, how does it choose what
to transmit and what to discard? The transmitter’s choice depends on what the
receiver wants and what it is capable of doing with the data transmitted. We
approach this problem by first formulating it in its simplest form, shown in
Fig. 1. One transmitting agent has access to two variables $X_{1}$ and
$X_{2}$, and must choose one of them to send to a receiving agent according to
a function, $\gamma$. The receiving agent will only have access to either
$X_{1}$ or $X_{2}$, but it wants to recover both. To that effect, the receiver
uses a function, $\eta$. The goal of the designer is to optimize an
appropriate performance metric (mean squared error, probability of error,
etc.) over all possible functions $\gamma$ and $\eta$.

To the best of our knowledge, the optimal solution to this problem is unknown
even in the canonical case of Gaussian sources under the mean-squared error
distortion. In this paper, we propose searching for the optimal scheduler
using data-driven linear function approximation approach called
neuroscheduling. Our main contributions are: 1. we provide a condition that
needs to be satisfied by the basis functions in our approximation to avoid
over-fitting; 2. we show via numerical results that linear estimators in the
Gaussian case are not optimal.

###  I-A Related Literature

Data selection is a topic with a rich history rooted in statistics, where the
goal is choose a subset of the original feature set without making any
transformation on the attributes [1]. The conventional approach to this is to
use informativeness metrics reminiscent of Estimation theory, such as
functions of the error covariance matrix and select a subset of predetermined
size $K$ from a larger set with $N$ features [2]. This optimization problem is
known to be computationally intractable and the alternative is to use
suboptimal algorithms that have an acceptable performance and can produce a
solution in reasonable time [3, 4, 5]. However, this approach is done before
the data is collected based on the probabilistic model of the data and the
observation noise. Moreover, the strategies obtained using such approaches are
static, e.g. “if sensor A is more informative than sensor B, then sensor A is
prioritized.”

Information theory provides alternative ways to measure informativeness, that
can also be used to perform data selection [6]. However, information theoretic
quantities, such as entropy and mutual information, do not depend on the data
content at all and were designed to capture features related to the
probabilistic model. Recent developments have established a connection between
information and moments of the underlying random variables [7], but their use
to select informative data on the basis of their realizations remains an open
question. Existing approaches require a full knowledge of the probabilistic
model of the system and provide policies that do not capture situations where
a less informative data source may produce a very informative data sample.

Figure 1: Observation-driven sensor scheduling system with two sensors, one
scheduler and one receiver.

Control theory, on the other hand, is based on the notion that data is
measured first and then processed using policies – therefore, the results are
naturally adapted to data. In the control theory literature, data selection
appear under the moniker of event-triggered estimation/control/learning [8, 9,
10, 11, 12]. The premise is that in an event-triggered policy, the data is
transmitted to a remote decision-maker if the observed variable exceeds a
certain threshold. The idea is that observations that are small (in an
appropriate sense) are predictable and redundant – they are not informative. A
large body of literature based on this principle exists. In particular, the
area of remote estimation has emerged as an area of interest, where smart
sensors independently make decisions to transmit (or not) their observations
to a fusion center [13, 14, 15, 16, 17, 18]. The literature reveals that in
existing applications, each sensor observes a single variable/vector that is
either fully kept or discarded. However, to the best of our knowledge, a
similar systematic event-based approach to subset selection where multiple
variables are simultaneously observed at the same location and a subset of
them are selected has not been developed until now.

Another body of literature closely related to our work pertains to the Value
of Information (VoI), a concept originally introduced in the field of
economics. Historically, VoI has been considered in making decisions in
optimal control settings [19, 20, 21, 22], and it is similar to what we
propose here: transmit the data (or not) based on the level of performance
improvement that can be achieved using that additional data. In certain cases,
scheduling based on VoI may coincide with the scheduling of Extremum
Information [23]. It has been well-documented that the decision not to
communicate an observation may convey implicit information [24, 13, 15, 25,
26]. Such implicit communication leads to substantial gains in performance,
which should also be taken into consideration when computing the VoI. However,
the framework introduced in [20, 21] deals only with a single data source,
whereas our focus in this paper is on identifying a one of multiple sources
that have the largest VoI for a particular task.

##  II Problem setup

Consider two Gaussian random variables observed by a scheduler, which is
interested in sending one of them to a receiver jointly distributed according
to $(X_{1},X_{2})\sim f_{X_{1}X_{2}}$, and assume that all sources have
stationary statistics. At a given time, the scheduler observes one realization
of $X_{1}$ and $X_{2}$ decides using a scheduling policy, whether $X_{1}$ or
$X_{2}$ will be transmitted to the destination. Communication happens in real
time, i.e., we do not make our decisions based on observing blocks of data,
which would necessarily incur communication delay [27]. A scheduling policy is
a map from the observation space $\mathbb{R}^{2}$ to the set $\\{1,2\\}.$ The
decision variable $U\in\\{1,2\\}$ is computed according to

$U=\gamma(X_{1},X_{2}).$ (1)

Based on $U$, the information sent over the channel is determined as follows:

$Y=\begin{cases}(1,X_{1}),&\textup{if}\ \ U=1\\\ (2,X_{2}),&\textup{if}\ \
U=2,\end{cases}$ (2)

where the index in front of the information variable is important since the
receiver needs to know which information source generated the real number
observed in the packet, i.e., it indicates the origin of the communication
packet.

At the destination, the receiver implements an estimation policy, which
attempts to reconstruct both sources based on the observation received over
the link, $Y$. We define $\eta$ as such estimation policy, and the estimates
are computed as follows:

$\begin{bmatrix}\hat{X}_{1}\\\ \hat{X}_{2}\end{bmatrix}=\eta(Y).$ (3)

From the designer’s perspective, the optimization problem that we are
interested in solving is the following:

$\min_{(\gamma,\eta)\in\Gamma\times\mathrm{H}}\mathbf{E}\left[(X_{1}-\hat{X}_{1})^{2}+(X_{2}-\hat{X}_{2})^{2}\right],$
(4)

where $\Gamma$ and $H$ are the spaces of all admissible scheduling and
estimation policies, respectively. By looking at Problem (4), we are searching
for an optimal scheduling-estimation pair. A conceptualy simpler version of
this problem is: given an estimator pair $(\eta_{1},\eta_{2})$, determine the
scheduling policy that is optimal for this fixed estimator. The answer is the
following scheduling policy:

$\gamma^{\star}_{\eta_{1},\eta_{2}}(x_{1},x_{2})=\begin{cases}1,&\text{if}\ \
|x_{1}-\eta_{1}(x_{2})|\geq|x_{2}-\eta_{2}(x_{1})|\\\
2,&\text{otherwise,}\end{cases}$ (5)

where $\eta_{1}$ and $\eta_{2}$ are functions used to estimate $X_{1}$ and
$X_{2}$, respectively, based on the information revealed to the receiver by
the scheduler. Assuming that this scheduling policy is used, then the next
step is to find pair of optimal estimators $(\eta_{1}^{*},\eta_{2}^{*})$,
which are nonlinear functions, in general. The scheduling policy in Eq. 5 is
called max-scheduling [28]. In many cases, it is possible to show that certain
estimation policies are person-by-person optimal [29] with this policy,
leading to locally optimal solutions. However, such results rely on a guess-
and-verify approach, and there is no systematic way of obtaining locally nor
globally optimal solutions.

###  II-A Optimality of Estimator Pairs

Generally, a scheduler is a measurable function on $\mathbb{R}^{2}$ that
induces a partition of $\mathbb{R}^{2}$. Let
$\gamma:\mathbb{R}^{2}\rightarrow\\{1,2\\}$, where
$\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=1\\}\cup\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=2\\}=\mathbb{R}^{2}$
and
$\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=1\\}\cap\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=2\\}=\varnothing$.
We are interested in obtaining the optimal estimator pair when the scheduler
follows the max-scheduling policy of Eq. 5. For this purpose, let us define
the error vector on $\mathbb{R}^{2}$ for a particular estimator pair
$(\eta_{1},\eta_{2})$:

$\epsilon_{\eta_{1},\eta_{2}}\operatorname{\overset{def}{=}}\begin{cases}X_{2}-\eta_{2}(X_{1}),&U=1\\\
X_{1}-\eta_{1}(X_{2}),&U=2.\end{cases}$ (6)

For a given pair $\eta_{1},\eta_{2}$, the quantity
$\epsilon_{\eta_{1},\eta_{2}}$ is a random variable on $\mathbb{R}^{2}$.
Whenever we change the estimator function pair $(\eta_{1},\eta_{2})$, its
distribution changes. Ideally, the following estimator pair should be the
optimal one we are after:

$\eta_{1}^{\star}(x_{2})=\mathbf{E}\big{[}X_{1}\mid X_{2}=x_{2},U=2\big{]},$
(7)

and

$\eta_{2}^{\star}(x_{1})=\mathbf{E}\big{[}X_{2}\mid X_{1}=x_{1},U=1\big{]}.$
(8)

It is important to notice that, in general, $\mathbf{E}\big{[}X_{1}\mid
X_{2}=x_{2},U=2\big{]}\neq\mathbf{E}\big{[}X_{1}\mid X_{2}=x_{2}\big{]}$ and
$\mathbf{E}\big{[}X_{2}\mid
X_{1}=x_{1},U=1\big{]}\neq\mathbf{E}\big{[}X_{2}\mid X_{1}=x_{1}\big{]}$. Take
the Gaussian case as an example, suppose $(X_{1},X_{2})$ follows a standard
bivariate normal distribution with nonzero correlation coefficient $\rho$,
then define

$\hat{\eta}_{1}(x_{2})=\mathbf{E}\big{[}X_{1}\mid X_{2}=x_{2}\big{]}=\rho
x_{2}$ (9)

and

$\hat{\eta}_{2}(x_{1})=\mathbf{E}\big{[}X_{2}\mid X_{1}=x_{1}\big{]}=\rho
x_{1}.$ (10)

Computing the expectation of $\epsilon_{\hat{\eta}_{1},\hat{\eta}_{2}}$,we get

$\mathbf{E}[\epsilon_{\hat{\eta}_{1},\hat{\eta}_{2}}]=\frac{1}{2}\int_{\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=1\\}}-\rho
x_{2}\hskip 5.0pt\mathrm{d}\mu_{X}\\\
+\frac{1}{2}\int_{\\{(x_{1},x_{2})\in\mathbb{R}^{2}\mid\gamma(x_{1},x_{2})=2\\}}-\rho
x_{1}\hskip 5.0pt\mathrm{d}\mu_{X},$ (11)

where $\mu_{X}$ is the measure induced by $(X_{1},X_{2})$ on $\mathbb{R}^{2}$.
Generally $\mathbf{E}[\epsilon_{\hat{\eta}_{1},\hat{\eta}_{2}}]$ can never be
0 for a nonzero $\rho$. However, if we compute
$\mathbf{E}[{\epsilon_{\eta_{1}^{\star},\eta_{2}^{\star}}}]$, by law of total
expectation, we have

$\mathbf{E}[\epsilon_{\eta_{1}^{\star},\eta_{2}^{\star}}]=\frac{1}{2}\Big{(}\mathbf{E}[X_{2}]-\mathbf{E}\big{[}\mathbf{E}[X_{2}\mid
X_{1}=x_{1},U=1]\big{]}\Big{)}\\\
+\frac{1}{2}\Big{(}\mathbf{E}[X_{1}]-\mathbf{E}\big{[}\mathbf{E}[X_{1}\mid
X_{2}=x_{2},U=2]\big{]}\Big{)}=0,$ (12)

which implies that even for Gaussian distributions, the optimal estimator pair
$(\eta_{1}^{\star},\eta_{2}^{\star})$ corresponding to the max-scheduling
policy can be nonlinear. The max-scheduling policy defines the random variable
$U$ in the following sense:

$U=\begin{cases}1,&\big{(}x_{1}-\eta_{1}(x_{2})\big{)}^{2}\geq\big{(}x_{2}-\eta_{2}(x_{1})\big{)}^{2}\\\
2,&\big{(}x_{1}-\eta_{1}(x_{2})\big{)}^{2}<\big{(}x_{2}-\eta_{2}(x_{1})\big{)}^{2}.\end{cases}$
(13)

For any estimator pair $(\eta_{1},\eta_{2})$, the scheduler always chooses to
send out the variable that would give the larger estimation error if not
transmitted. If we plug Eq. 13 into Eq. 6, it is immediate to see that
minimizing $\mathbf{E}[\epsilon_{\eta_{1},\eta_{2}}]$ is equivalent to
minimizing the expectation of the error which has a smaller magnitude among
those two. Thus, we define the optimality of estimators corresponding to the
max-scheduling policy in the following proposition.

###### Proposition 1

Suppose the scheduler is desinated to apply the max-scheduling policy as in
Eq. 13. Then the corresponding optimal estimators $\eta_{1}^{\star}$ and
$\eta_{2}^{\star}$ are:

$(\eta_{1}^{\star},\eta_{2}^{\star})\in\underset{(\eta_{1},\eta_{2})\in\mathrm{H}_{1}\times\mathrm{H}_{2}}{\arg\min}\mathcal{J}(\eta_{1},\eta_{2})\\\
\operatorname{\overset{def}{=}}\mathbf{E}\left[\min\bigg{\\{}\big{(}X_{1}-\eta_{1}({X}_{2})\big{)}^{2},\big{(}X_{2}-\eta_{2}({X}_{1})\big{)}^{2}\bigg{\\}}\right].$
(14)

In practice, given a set of independent and identically distributed data
samples $\\{(x_{1}^{i},x_{2}^{i})\\}_{i=1}^{M}$ drawn from $f_{X_{1}X_{2}}$,
we evaluate the performance of $(\eta_{1},\eta_{2})$ using the empirical mean:

$\bar{\mathcal{J}}(\eta_{1},\eta_{2})=\frac{1}{M}\sum_{i=1}^{M}\min\Big{\\{}\big{(}x_{1}^{i}-\eta_{1}(x_{2}^{i})\big{)}^{2},\big{(}x_{2}^{i}-\eta_{2}(x_{1}^{i})\big{)}^{2}\Big{\\}}.$
(15)

##  III Approximation of Nonlinear Estimators

In general, Problem (14) does not admit a solution in closed form [30]. There
are no systematic ways of solving this problem in its current description.
Instead, we need to use some form of approximation to obtain a candidate pair
of optimal solutions $(\eta_{1}^{\star},\eta_{2}^{\star})$. Before we proceed
with the search for optimal estimators, one natural question arises: – What
are the appropriate spaces $\mathrm{H}_{1}$ and $\mathrm{H}_{2}$ of admissible
estimators?

We assume that the admissible estimators belong to the sets of continuously
differentiable functions. So ideally $\mathrm{H}_{1}=\mathcal{C}^{1}[a,b]$ and
$\mathrm{H}_{2}=\mathcal{C}^{1}[c,d]$, where

$\displaystyle
a\operatorname{\overset{def}{=}}\min\\{x_{2}^{1},\dots,x_{2}^{M}\\},\
b\operatorname{\overset{def}{=}}\max\\{x_{2}^{1},\dots,x_{2}^{M}\\}$ (16)
$\displaystyle
c\operatorname{\overset{def}{=}}\min\\{x_{1}^{1},\dots,x_{1}^{M}\\},\
d\operatorname{\overset{def}{=}}\max\\{x_{1}^{1},\dots,x_{1}^{M}\\}.$ (17)

This means that we will ignore the performance of estimator pairs on
$\mathbb{R}^{2}\setminus[a.b]\times[c,d]$ since there are no samples on this
set. For $\mathrm{H}_{1}=\mathcal{C}^{1}[a,b]$ and
$\mathrm{H}_{2}=\mathcal{C}^{1}[c,d]$, Problem (14) is still non-convex, and
infinite-dimensional. To solve it in practice, we must turn it into a finite
dimensional optimization problem first. In this section, we will discuss how
to approximately solve Problem (14) by performing optimization over a vector
of weight parameters.

###  III-A Principles of Linear Function Approximation

We are going to use a linear combination of functions
$\\{\phi_{n}\\}_{n=1}^{\infty}$ to approximate the optimal estimator pair
$(\eta_{1}^{\star},\eta_{2}^{\star})$. It is known that, rigorously, the
cardinality of the set of basis functions for $\mathcal{C}^{1}[a,b]$ is
$\aleph_{1}$. Therefore, we can not define a countable basis for all
differentiable functions. To make the approximation meaningful, we should
assume that a unique linear combination of $\\{\phi_{n}\\}_{n=1}^{\infty}$
approaches $\eta_{1}^{\star}$ and $\eta_{2}^{\star}$ in a Schauder basis sense
[31]. That is, there exist two unique sequences of weight parameters
$\\{w_{1}^{n}\\}_{n=1}^{\infty}$ and $\\{w_{2}^{n}\\}_{n=1}^{\infty}$ such
that

$\eta_{1}^{\star}(x)=\lim\limits_{K\to\infty}\sum_{n=1}^{K+1}w_{1}^{n}\phi_{n}(x),\
\ x\in[a,b]$ (18)

and

$\eta_{2}^{\star}(x)=\lim\limits_{K\to\infty}\sum_{n=1}^{K+1}w_{2}^{n}\phi_{n}(x),\
\ x\in[c,d].$ (19)

Here, the convergence is with respect to the sup norm equipped on the space of
continuously differentiable functions $\mathcal{C}^{1}$. Formally, consider a
set of differentiable functions denoted by $\Phi$, such that:

$\Phi(x)=\begin{bmatrix}\phi_{1}(x)\\\ \vdots\\\ \phi_{K+1}(x)\\\
\vdots\end{bmatrix}.$ (20)

Let $\phi_{1}(x)=1,$ $x\in\mathbb{R}$. A linear approximation for our
estimator functions is given by:

$\eta_{j}(x)=\bm{w}_{j}^{\mathsf{T}}\Phi(x),$ (21)

where $\bm{w}_{j}$ is a sequence of real numbers, $j\in\\{1,2\\}$. Then, the
optimization is performed over the weight sequences
$\bm{w}\operatorname{\overset{def}{=}}(\bm{w}_{1},\bm{w}_{2})$, as follows:

$\min{\mathcal{J}}(\bm{w})=\mathbf{E}\left[\min\bigg{\\{}\big{(}X_{1}-\bm{w}_{1}^{\mathsf{T}}\Phi({X}_{2})\big{)}^{2},\right.\\\
\left.\big{(}X_{2}-\bm{w}_{2}^{\mathsf{T}}\Phi({X}_{1})\big{)}^{2}\bigg{\\}}\right].$
(22)

For computational tractability, we must truncate sequences $\bm{w}_{1}$ and
$\bm{w}_{2}$ to have $(K+1)$ components, where $K$ can be interpreted as a
measure of complexity for the approximation architecture. This reduces the
dimension of the original problem to $(2K+2)$. With samples
$\\{(x_{1}^{i},x_{2}^{i})\\}_{i=1}^{M}$, Problem (22) is approximated by:

$\min_{\hat{\bm{w}}\in\mathbb{R}^{2K+2}}\hat{\mathcal{J}}(\hat{\bm{w}})=\frac{1}{M}\sum_{i=1}^{M}\min\Big{\\{}\big{(}x_{1}^{i}-\hat{\bm{w}}_{1}^{\mathsf{T}}\Phi({x}_{2}^{i})\big{)}^{2},\\\
\big{(}x_{2}^{i}-\hat{\bm{w}}_{2}^{\mathsf{T}}\Phi({x}_{1}^{i})\big{)}^{2}\Big{\\}}.$
(23)

The choice of $\phi_{n}$ functions and the dimension must be on a case-by-case
analysis. A smart choice of $\\{\phi_{n}\\}_{n=1}^{\infty}$ would allow for a
shorter truncation, which significantly reduce the computational cost. By
solving Eq. 23 instead of solving Eq. 14, we lose accuracy with respect to two
aspects. First, after the truncation, the search is restricted to the space
$\mathrm{span}\\{\phi_{1},\dots,\phi_{2K+1}\\}$, which is finite-dimensional.
Second, the resulting estimators from (23) can be tailored to the specific
data set $\\{(x_{1}^{i},x_{2}^{i})\\}_{i=1}^{M}$. We consider this as a over-
fitting: the resulting estimator may perform extremely well on this specific
data set, but it is far from the optimum when considering the whole
distribution. The former aspect of losing accuracy is inevitable since we do
not have infinite computational resources. The later aspect, however, has some
patterns. Through our numerical experiments, we observed that, to avoid over-
fitting, it is necessary to require resulting estimators to preserve the
distribution type. That is, if $X_{1}$ and $X_{2}$ are continuous random
variables, then $\eta_{2}(X_{1})$ and $\eta_{1}(X_{2})$ should have continuous
distributions. We have the following assumption.

###### Assumption 1

Let $(X_{1},X_{2})$ be a continuous random vector supported on
$\mathbb{R}^{2}$ with $\mathrm{Cor}\\{X_{1},X_{2}\\}\neq 0$. The conditional
expectations, $\mathbf{E}\big{[}X_{1}\mid X_{2},U=2\big{]}$ and
$\mathbf{E}\big{[}X_{2}\mid X_{1},U=1\big{]}$, when regarded as random
variables $\eta_{1}^{\star}(X_{2})$ and $\eta_{2}^{\star}(X_{1})$, have
continuous distributions supported on $\mathbb{R}$. That is, the probability
measure on $\mathbb{R}$ induced by $\eta_{1}^{\star}(X_{2})$ and
$\eta_{2}^{\star}(X_{1})$ should be absolutely continuous with respect to the
Lebesgue measure [32].

The reason for 1 is very natural: as $\eta_{1}^{\star}(X_{2})$ and
$\eta_{2}^{\star}(X_{1})$ are estimations for $X_{1}$ and $X_{2}$, we should
require this comparison happens in the same family of distributions. Unless
$X_{1}$ and $X_{2}$ are independent, then $X_{1}$ carries some information
about $X_{2}$ and vice versa. The information can be incomplete, but when it
comes to the conditional expectation and change the distribution
correspondingly, we assume that the resulting distribution remains continuous.

The following theorem provides a basic qualification for functions
$\\{\phi_{n}\\}_{n=1}^{\infty}$ based on the previous observation.

###### Theorem 1

Let $(X_{1},X_{2})$ be a random vector supported on $\mathbb{R}^{2}$ with
continuous distribution. Denote Lebesgue measure on $\mathbb{R}$ by
$\lambda(\cdot)$. The following two statements cannot be simultaneously true:

$\text{1.}\hskip
20.0pt\lambda\Big{(}\bigcap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\}\Big{)}>0$

2. There exist unique sequences $\\{w_{1}^{n}\\}_{n=1}^{\infty}$ and $\\{w_{2}^{n}\\}_{n=1}^{\infty}$ such that

$\eta_{1}^{\star}(x)=\lim\limits_{K\to\infty}\sum_{n=1}^{K+1}w_{1}^{n}\phi_{n}(x)\hskip
10.0pt\text{and}\hskip
10.0pt\eta_{2}^{\star}(x)=\lim\limits_{K\to\infty}\sum_{n=1}^{K+1}w_{2}^{n}\phi_{n}(x).$

###### Proof:

We take random variables $X_{1}$ and $\eta_{1}(X_{2})$ as an example. The
other pair would be similar thus it is omitted here. The optimal estimator
$\eta_{1}^{*}(\cdot)$ is the conditional expectation
$\mathbf{E}\big{[}X_{1}\mid X_{2},\mathbf{U}=2\big{]}$, it induces a
probability measure $\mu^{\prime}$ for sets of $x_{2}$:
$\mu^{\prime}\operatorname{\overset{def}{=}}\mu_{X_{1}}(\eta_{1}^{*-1}(\cdot))$,
here $\mu_{X_{1}}$ denotes the probability measure induced by $X_{1}$ and
$\eta_{1}^{*-1}(\cdot)$ is the pre-image of a set through $\eta_{1}^{*}$. Note
that by our assumption 1, $\mu^{\prime}$ is a measure induced by a continuous
random variable supported on $\mathbb{R}$, by Radon-Nikodym theorem,
$(\exists)\hskip 5.0ptf,g:\mathbb{R}\rightarrow\mathbb{R}$ positive almost
everywhere, such that $(\forall)$ measurable set $A$,

$\mu_{X_{1}}(A)=\int_{A}f\mathrm{d}\lambda\hskip 5.0pt\text{and}\hskip
5.0pt\mu^{\prime}(A)=\int_{A}g\mathrm{d}\lambda$ (24)

such $f$ and $g$ are probability density functions. As they are positive
almost everywhere on $\mathbb{R}$, functions $\frac{1}{f}$ and $\frac{1}{g}$
are positive and exist almost everywhere on $\mathbb{R}$. We can write:

$\lambda(A)=\int_{A}\frac{1}{f}\mathrm{d}\mu_{X_{1}}\hskip
5.0pt\text{and}\hskip
5.0pt\lambda(A)=\int_{A}\frac{1}{g}\mathrm{d}\mu^{\prime}$ (25)

This equation indicates equivalence between measures $\mu_{X_{1}}$,
$\mu^{\prime}$ and $\lambda$.
Now consider $\bar{\eta}_{1}=\sum_{n=1}^{\infty}w_{1}^{n}\phi_{n}$ as the
optimizer of problem 23 when $K\rightarrow\infty$. Similarly we define
$\mu^{\prime\prime}\operatorname{\overset{def}{=}}\mu_{X_{1}}(\bar{\eta}_{1}^{-1}(\cdot))$.
$\mu^{\prime\prime}$ is the probability measure on $\mathbb{R}$ induced by
random variable $\bar{\eta}_{1}(X_{2})$. On the set
$\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\}$,
$\phi_{n}$ can only take a constant value, and the weight parameter sequence
$\\{w_{1}^{n}\\}_{n=1}^{\infty}$is unique. This implies that
$\bar{\eta}_{1}=\bar{c}$ on
$\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\}$, where
$\bar{c}$ is a constant or $\pm\infty$.
On the one hand,
$\mu^{\prime\prime}(\\{\bar{c}\\})\geq\mu_{X_{1}}(\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\})$
by its definition, and we can compute the right hand side by

$\mu_{X_{1}}(\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\})=\int_{\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\}}f\mathrm{d}\lambda$
(26)

Suppose
$\lambda(\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\})>0$.
The integration in (26) is strictly positive since we are integrating a
positive a.e. function with respect to Lebesgue measure on a set of positive
measure. This shows that $\mu^{\prime\prime}(\\{\bar{c}\\})>0$ if
$\lambda(\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\})>0$.
On the other hand, we have $\lambda(\\{\bar{c}\\})=0$. Measure
$\mu^{\prime\prime}$ disagrees with $\lambda$ at least on one $\lambda$-null
set, thus they can not be equivalent. But $\mu^{\prime}$ is equivalent with
$\lambda$. By looking at the definition of $\mu^{\prime}$ and
$\mu^{\prime\prime}$, the only explanation is: $\eta_{1}^{*}(\cdot)$ and
$\bar{\eta}_{1}(\cdot)$ are not the same function. The approximation fails in
this case even when $K\rightarrow\infty$. ∎

###### Corollary 1

Theorem 1 suggests that simple, wavelets and Rectified Linear Unit (ReLU)
functions are not good choices for $\\{\phi_{n}\\}_{n=1}^{\infty}$.

###### Remark 1

Theorem 1 is a necessary condition for the choice of $\phi_{n}$, whose purpose
is to avoid over-fitting. If we approximate the estimator without taking this
condition into consideration, we would obtain an oscillating function that
tries to fit every sample point in the data set. Such estimators fail to pass
validation tests: If we generate another set of data from the same
distribution and apply our previous estimator, the performance is degraded
with high probability. Neural networks are popular architectures for
approximation of unknown functions and one of the most commonly used
activation functions is the ReLU. Unfortunately, using an architecture based
on ReLUs violates the condition proposed in Theorem 1. This means the
approximation generated by a ReLU network would lead to over-fitting. Thus, we
must resort to other activation functions when applying a Neural Network on
Problem (23).

##  IV Numerical Results

As pointed out in the previous section, we should avoid to use a set of
functions such that their derivatives being zero on a set of positive Lebesgue
measure. We tried different types of functions, it turns out that the family
of softplus functions and polynomials return extrodinary and stable results.

###  IV-A Approximation Using Soft-Plus Functions

With a slightly abuse of terminology, we call the function

$\phi_{\alpha,\beta}(x)\operatorname{\overset{def}{=}}\frac{1}{\alpha}\log\Big{(}1+\exp\big{(}1\pm\alpha(x-\beta)\big{)}\Big{)}$
(27)

as a soft-plus function since it is a smooth approximation to the ReLU
function without violating the principle proposed in Theorem 1. Although the
linear combination of $\phi_{\alpha,\beta}(x)$ can not cover $\mathcal{C}^{1}$
for arbitrary countable collection of $(\alpha,\beta)$, it gives us stable and
promising results, especially when $(X_{1},X_{2})$ is a bivariate Gaussian
vector. As shown in [33], the density function of the minimum of two
correlated Gaussian random variables has an explicit analytical form, which is
related to the Gaussian error function $\mathrm{erf}(\cdot)$.

Figure 2: SoftPlus estimator, along with its corresponding scheduling policy,
and a validation analysis for a symmetric Gaussian distribution across
different correlation values.

###  IV-B Approximation Using Polynomials

The Weierstrass Approximation Theorem states that on a closed interval, we can
approximate a continuous function with vanishing error by increasing the
dimension of approximation. The convergence is even uniform for every element
on the designated interval. This gives us a theoretical guarantee of
convergence if we choose $\phi_{n}=x^{n}$. It is also worth mentioning that
the polynomials $\\{x^{n}\\}_{n=1}^{\infty}$ never violate the condition
$\lambda(\cap_{n=1}^{\infty}\\{x\in\mathbb{R}\mid\phi^{\prime}_{n}(x)=0\\})=0$.
This makes polynomials a great choice. However, in practice, we find out that
it requires a relatively large amount of polynomials to get a decent
performance on the resulting estimator pair. The idea of polynomial
approximation definitely worth more investigation, since it may leads to
convergence speed and sample complexity results in the future.

Figure 3: Polynomial estimator, along with its corresponding scheduling policy, and a validation analysis for a symmetric Gaussian distribution across different correlation values. TABLE I: Validation cost of four different estimators for the symmetric Gaussian with different correlation values. $\rho$ | MMSE | Linear | SoftPlus | Polynomial
---|---|---|---|---
$0$ | 0.363 | 0.363 | 0.303 | 0.334
$0.25$ | 0.360 | 0.345 | 0.285 | 0.315
$0.5$ | 0.337 | 0.288 | 0.230 | 0.261
$0.75$ | 0.254 | 0.173 | 0.140 | 0.159

##  V Conclusion and Future Work

We have considered the problem of remote estimation of a bivariate Gaussian
source when only one of the variables is revealed by a scheduler to the
estimator. The optimal scheduler for this seemingly simple problem is unknown.
Therefore, we propose to look for good solution using a single-layer neural
network, called neuro-schedulers. We establish a condition to determining
which activation functions are cannot be used if we are trying to avoid data
over-fitting. We also show that the the optimal scheduler is a nonlinear in
general. Future work consists of using deep neural-networks and obtaining
performance guarantees for specific architectures as well as optimization
algorithms that exploit the inherent structure of the scheduling problem.

## References

  * [1] C. Villa-Blanco, C. Bielza, and P. Larranaga, “Feature subset selection for data and feature streams: a review,” Artificial Intelligence Review, vol. 56, pp. 1011–1062, 2023.
  * [2] A. Hashemi, M. Ghasemi, H. Vikalo, and U. Topcu, “Randomized greedy sensor selection: Leveraging weak submodularity,” IEEE Transactions on Automatic Control, vol. 66, no. 1, pp. 199–212, 2021.
  * [3] S. Joshi and S. Boyd, “Sensor selection via convex optimization,” IEEE Transactions on Signal Processing, vol. 57, no. 2, pp. 451–462, 2008.
  * [4] J. Moon and T. Başar, “Static optimal sensor selection via linear integer programming: The orthogonal case,” IEEE Signal Processing Letters, vol. 24, no. 7, pp. 953–957, 2017.
  * [5] S. Takahashi, Y. Sasaki, T. Nagata, K. Yamada, K. Nakai, Y. Saito, and T. Nonomura, “Sensor selection by greedy method for linear dynamical systems: comparative study on fisher-information-matrix, observability-gramian and kalman-filter-based indices,” IEEE Access, 2023\.
  * [6] Z. Wang, M.-A. Badiu, and J. P. Coon, “A framework for characterizing the value of information in hidden Markov models,” IEEE Transactions on Information Theory, vol. 68, no. 8, pp. 5203–5216, 2022.
  * [7] W. Alghamdi and F. P. Calmon, “Measuring information from moments,” IEEE Transactions on Information Theory, vol. 70, no. 2, pp. 763–802, 2024.
  * [8] A. Molin and S. Hirche, “Event-triggered state estimation: An iterative algorithm and optimality properties,” IEEE Transactions on Automatic Control, vol. 62, no. 11, pp. 5939–5946, 2017.
  * [9] P. Tabuada, “Event-triggered real-time scheduling of stabilizing control tasks,” IEEE Transactions on Automatic Control, vol. 52, no. 9, pp. 1680–1685, 2007.
  * [10] F. Solowjow and S. Trimpe, “Event-triggered learning,” Automatica, vol. 117, p. 109009, 2020.
  * [11] J. Yun, A. Eryilmaz, J. Moon, and C. Joo, “Remote estimation for dynamic IoT sources under sublinear communication costs,” IEEE/ACM Transactions on Networking, 2023.
  * [12] M. Salimnejad, M. Kountouris, and N. Pappas, “Real-time reconstruction of Markov sources and remote actuation over wireless channels,” IEEE Transactions on Communications, pp. 1–1, 2024.
  * [13] G. M. Lipsa and N. C. Martins, “Remote state estimation with communication costs for first-order lti systems,” IEEE Transactions on Automatic Control, vol. 56, no. 9, pp. 2013–2025, 2011.
  * [14] A. Nayyar, A. Mahajan, and D. Teneketzis, “Decentralized stochastic control with partial history sharing: A common information approach,” IEEE Transactions on Automatic Control, vol. 58, no. 7, pp. 1644–1658, 2013.
  * [15] M. M. Vasconcelos and N. C. Martins, “Optimal estimation over the collision channel,” IEEE Transactions on Automatic Control, vol. 62, pp. 321–336, January 2017.
  * [16] M. M. Vasconcelos and N. C. Martins, “Optimal remote estimation of discrete random variables over the collision channel,” IEEE Transactions on Automatic Control, vol. 64, no. 4, pp. 1519–1534, 2019.
  * [17] Y. Ni, X. Ren, S. Dey, and L. Shi, “Remote state estimation with a strategic sensor using a Stackelberg game framework,” IEEE Transactions on Control of Network Systems, vol. 8, no. 4, pp. 1613–1623, 2021.
  * [18] A. S. Leong, D. E. Quevedo, and W. Liu, “Stability enforced bandit algorithms for channel selection in remote state estimation of Gauss-Markov processes,” IEEE Transactions on Automatic Control, vol. 68, no. 12, pp. 8308–8315, 2023.
  * [19] F. Chiariotti, A. E. Kalør, J. Holm, B. Soret, and P. Popovski, “Scheduling of sensor transmissions based on value of information for summary statistics,” IEEE Networking Letters, vol. 4, no. 2, pp. 92–96, 2022.
  * [20] T. Soleymani, J. S. Baras, and S. Hirche, “Value of information in feedback control: Quantification,” IEEE Transactions on Automatic Control, vol. 67, no. 7, pp. 3730–3737, 2022.
  * [21] T. Soleymani and D. Gündüz, “State estimation over broadcast and multi-access channels in an unreliable regime,” arXiv preprint arXiv:2308.16085, 2023.
  * [22] A. Zancanaro, G. Cisotto, and L. Badia, “Modeling value of information in remote sensing from correlated sources,” Computer Communications, vol. 203, pp. 289–297, 2023.
  * [23] M. M. Vasconcelos and U. Mitra, “Observation-driven scheduling for remote estimation of two gaussian random variables,” IEEE Transactions on Control of Network Systems, vol. 7, no. 1, pp. 232–244, 2020.
  * [24] O. C. Imer and T. Başar, “Optimal estimation with limited measurements,” International Journal of Systems, Control and Communications, vol. 2, no. 1-3, pp. 5–29, 2010.
  * [25] X. Zhang and M. M. Vasconcelos, “Robust one-shot estimation over shared networks in the presence of denial-of-service attacks,” arXiv preprint arXiv:2302.14689, 2023.
  * [26] E. D. Santi, T. Soleymani, and D. Gunduz, “Remote estimation of markov processes over costly channels: On the benefits of implicit information,” arXiv preprint arXiv:2401.17999, 2024.
  * [27] E. Akyol, K. B. Viswanatha, K. Rose, and T. A. Ramstad, “On zero-delay source-channel coding,” IEEE Transactions on Information Theory, vol. 60, no. 12, pp. 7473–7489, 2014.
  * [28] M. M. Vasconcelos and N. C. Martins, “A survey on remote estimation problems,” Principles of Cyber-Physical Systems: An Interdisciplinary Approach, pp. 81–103, 2020.
  * [29] S. Yüksel and T. Başar, Stochastic networked control systems: Stabilization and optimization under information constraints.  Springer Science & Business Media, 2013.
  * [30] M. M. Vasconcelos and U. Mitra, “Data-driven sensor scheduling for remote estimation in wireless networks,” IEEE Transactions on Control of Network Systems, vol. 8, no. 2, pp. 725–737, 2021.
  * [31] F. Albiac and N. J. Kalton, Topics in Banach space theory, vol. 233.  Springer, 2006.
  * [32] E. Cinlar, Probability and stochastics, vol. 261.  Springer, 2011.
  * [33] S. Nadarajah and S. Kotz, “Exact distribution of the max/min of two Gaussian random variables,” IEEE Transactions on Very Large Scale Integration Systems, vol. 16, no. 2, pp. 210–212, 2008.