††thanks<EMAIL_ADDRESS>
# Non-Classical Kernels in Continuous Variable Systems

Roohollah Ghobadi  Institute for Quantum Science and Technology and Department
of Physics and Astronomy, University of Calgary, Calgary, Alberta, Canada T2N
1N4

###### Abstract

Kernel methods are ubiquitous in classical machine learning, and their formal
similarity with quantum mechanics has recently been established. Understanding
the properties of the nonclassical kernel functions can provide us with
valuable information about the potential advantage of quantum machine
learning. In this paper, we derive a nonclassicality witness for kernel
functions in continuous-variable systems. We discuss the implication of our
witness in terms of the resulting distribution of data points in the feature
space.

Small scale quantum systems are a new testbed for understanding the potential
of quantum enabled technologies Google ; Zhong ; Xanadu ; RL . A promising
application of the small-scale quantum systems is in machine learning, where
the goal is to reveal patterns in data biamonte2017quantum ; RMP19  or
designing intelligent agents for reinforced learning RL . One scenario for
taking advantage of these intermediate devices in machine learning is using
them to estimate a classically expensive kernel function, a similarity measure
between data points. A classical computer can then find the best performing
hypothesis schuld2019quantum ; NatureIBM .

The quantum kernel methods provide a framework for understanding the quantum
advantage in machine learning. For example, the generalization performance in
supervised machine learning, which quantifies how accurate a learning model
can classify unseen data, has recently been studied in this framework Huang2 ;
and a necessary condition for achieving a quantum advantage in generalization
has been derived Huang2 . Also, encoding data in a quantum system is captured
by the resulting kernel function Schuld21 . These works highlight the
importance of understanding the difference between quantum and classical
kernel functions to assess the quantum advantage in machine learning.

Photonic hardware is a promising path for implementing and scaling up the
quantum algorithms Bourassa . Some groundbreaking demonstrations such as
Gaussian boson sampling RL ; Xanadu , photonic neural network Steinbrecher
and quantum advantage in reinforcement learning RL  have recently been
reported based on photonic hardware. From a theoretical standpoint, an
appealing property of quantum optical platforms compare to qubit systems is
their well-understood phase space structures. At the conceptual level, this
makes it possible to distinguish the genuine quantum effects from classical
ones using quasiprobability distributions Spekkens  such as the Wigner
function and to identify the enabling resource for quantum computing Howard .

In this paper, building on a recently proposed phase-space inequality
BohmanPRL ; BohmanQuantum ; Biagi , we drive a nonclassicality witness for
kernel functions. We use our proposed witness to show that the nonclassical
kernel leads to better separation of data points in feature space. Since the
nonclassicality of a quantum state can be converted to quantum entanglement
Kim ; Killoran , our result can be seen as an indirect study of the role of
quantum correlation in machine learning.

Supervised machine learning. Given a labeled dataset
$\\{(x_{k},f(x_{k})),x_{k}\in\mathbb{R}^{N},k=1,...M\\}$, the task is to find
a mapping $f(x)$ for new (unlabeled) data point. In the case of classification
$f(x_{k})=\pm 1$ and for regression $f$ is a continuous function of $x$.
Formally the learned function $h^{*}(x)$ is the solution of the following
equation scholkopf2001generalized ; hofmann2008kernel

$h^{*}(x)=\text{arg
min}_{h\in\mathcal{H}}\frac{1}{M}\sum_{i=1}^{M}\mathcal{L}(h(x_{i}),y_{i})+\lambda||h||^{2}$
(1)

where $\mathcal{L}$ is a loss function that characterizes the performance of
the learned function, and the $\mathcal{H}$ is the space of hypothesis. The
second term is known as the regularization term with $\lambda$ as the
regularization coefficient. The regularization term in Eq.(1) penalizes
complex hypotheses that best match the training data and favors a smooth one
with better prediction performance.

The kernel method in machine learning is based on mapping(encoding) data into
a Hilbert space by the feature map $\Phi:\mathcal{X}\rightarrow\mathcal{H}$
where $\mathcal{H}$ is a Hilbert space. One can then use the inner product
$\langle.,.\rangle_{\mathcal{H}}$ over $\mathcal{H}$ to define the kernel
function scholkopf2001generalized ; hofmann2008kernel

$K(x,x^{\prime})=\langle\Phi(x),\Phi(x^{\prime})\rangle_{\mathcal{H}}$ (2)

a measure of similarity between for $x,x^{\prime}\in\mathcal{X}$. The kernel
is symmetric $K(x,x^{\prime})=K(x^{\prime},x)$ and positive semidefinite
$\forall x_{i},x_{j}\in\mathcal{X}$, i.e.
$\sum_{i,j}c_{i}c_{j}K(x_{i},x_{j})\geq 0$. A useful family of hypothesis,
known as Reproducing kernel Hilbert space (RKHS), can be constructed from a
positive kernel function as

$\mathcal{R}_{K}=\\{h:\mathcal{X}\rightarrow\mathbb{C}|h(x)=\langle
w,\Phi(x)\rangle_{\mathcal{H}},w,\Phi\in\mathcal{H}\\}$ (3)

with the reproducing property $h(x)=\langle h,K_{x}\rangle_{\mathcal{H}}$ for
$h\in\mathcal{H}$. Therefore $K_{x}\in\mathcal{H}$ is the evaluation
functional which maps $h$ to $h(x)$. The advantage of RKHS is best shown using
the representer theorem that grantees that the solution of Eq.(1) in the RKHS
is given by scholkopf2001generalized ; hofmann2008kernel

$h^{*}(x)=\sum_{x_{k}\in\mathcal{X}}c_{k}K(x,x_{k})$ (4)

with $c_{k}\in\mathbb{R}$ to be determined. For the square loss function
$\mathcal{L}(h(x_{i}),y_{i})=(h(x_{i})-y_{i})^{2}$ from Eqs.(1,4) one obtains
$C=(K+\lambda M\mathbb{I})^{-1}Y$.

Generalization error bound. For a class of bounded functions
$\\{f\in\mathcal{R}_{K}:||f||\leq B\\}$ the generalization error bound is
given by kernelbook

$\mathbb{E}_{x\in\mathcal{D}}|h^{*}(x)-f(x)|\leq\frac{2B}{\sqrt{M}}$ (5)

where the average is with respect to a fixed distribution $\mathcal{D}$ over
input data and
$B=||f||^{2}=C^{T}KC=Y^{T}(K+M\lambda\mathbb{I})^{-1}K(K+M\lambda\mathbb{I})^{-1}Y$.
From Eq.(5), one can see that the kernel function controls the generalization
error bound thorough $B$.

Non-classicality in phase space. The phase-space approach to quantum mechanics
would allow distinguishing the quantum mechanics from the classical
statistical mechanics. To see this consider the coherent state which is
defined as $|\alpha\rangle=D(\alpha)|0\rangle$ where $D(\alpha)=e^{\alpha
a^{\dagger}-\alpha^{*}a}$ is the displacement operator and $|0\rangle$ is the
vacuum state. We can expand arbitrary quantum state in the basis of the
coherent state as Glauber ; Sudarshan

$\rho=\int d^{2}\alpha P(\alpha)|\alpha\rangle\langle\alpha|$ (6)

where $P(\alpha)$ is known as the Glauber-Sudarshan $P$ distribution. In the
case of classical state $P(\alpha)$ is a valid (density) probability
distribution i.e. $P(\alpha)\geq 0$. Accordingly, violation of $P(\alpha)\geq
0$ signals the nonclassical nature of state states. Instead of working with
$P$ distribution, it is more convenient to work with quasiprobability
distributions such as the Wigner function, which are non-singular and
experimentally accessible Leonhardt . In this paper, we are especially
interested in the following inequality, which holds for classical states
BohmanPRL ; BohmanQuantum

$W_{\rho}(\bm{\alpha})W_{\rho}(\bm{\alpha}^{\prime})-e^{-|\bm{\alpha}-\bm{\alpha^{\prime}}|^{2}}\\\
W_{\rho}(\frac{\bm{\alpha}+\bm{\alpha^{\prime}}}{2})^{2}\geq 0$ (7)

where $W_{\rho}$ is the Wigner function of density matrix $\rho$ 111The Wigner
function of state $\rho$ is defined as $W_{\rho}(q,p)=\frac{1}{2\pi}\int
dy\langle q-\frac{y}{2}|\rho|q+\frac{y}{2}\rangle e^{-ipy}$. The Wigner
function is the joint probability distribution for position and momentum. For
a quantum system, the Wigner function could become negative. The negativity of
the Wigner function, therefore, can serve as the nonclassicality witness.,
$\bm{\alpha}=(\alpha_{1},...,\alpha_{N})$ are arbitrary points in the phase
space. The violation of inequality Eq.(7) for
$\bm{\alpha}\neq\bm{\alpha^{\prime}}$ signals the nonclassicality of the
state. Notably, (7) can capture the quantum nature of states with non-negative
Wigner functions BohmanPRL ; BohmanQuantum .

Quantum machine learning. A remarkable similarity between kernel methods and
quantum mechanics is revealed by defining the feature map as
$x\rightarrow\rho(x)$ with $\rho$ as a quantum state in the Hilbert space of
the quantum system schuld2019quantum ; NatureIBM . The kernel function is then
defined as

$K(x,x^{\prime})=\frac{\text{tr}(\rho(x)\rho(x^{\prime}))}{\sqrt{\text{tr}(\rho^{2}(x))}\sqrt{\text{tr}(\rho^{2}(x^{\prime}))}}$
(8)

Note that we normalize the kernel such that $K(x,x)=1$. A potential path for
using small-scale quantum devices in machine learning could be to devise
computationally expensive kernels that can not be estimated efficiently on
classical computers schuld2019quantum ; NatureIBM ; Ghobadi . We are now in a
position to drive the main result of this paper. We start with the following
the data encoding Chatterjee

$\rho(x)=D^{\dagger}(x)\rho D(x)$ (9)

where the input data point, $x$, is encoded in the amplitude of displacement
operator. This choice for encoding is intuitive as the displacement operator
is a passive operation; therefore, it does not alter the quantum nature of our
input state, as opposed to the squeezing operator, for example. Next in Eq.(7)
we make the following substitutions $\bm{\alpha}\rightarrow\bm{\alpha}-\bm{x}$
and $\bm{\alpha^{\prime}}\rightarrow\bm{\alpha}-\bm{x^{\prime}}$ for
$\bm{x}\neq\bm{x^{\prime}}$ and use
$W_{\rho}(\bm{\alpha-x})=W_{\rho(\bm{x})}(\bm{\alpha})$,
$W_{\rho}(\bm{\alpha-x^{\prime}})=W_{\rho(\bm{x^{\prime}})}(\bm{\alpha})$
Leonhardt  to obtain

$W_{\rho(\bm{x})}(\bm{\alpha})W_{\rho(\bm{x^{\prime}})}(\bm{\alpha})-e^{-|\bm{x}-\bm{x^{\prime}}|^{2}}W_{\rho(\frac{\bm{x+x^{\prime}}}{2})}^{2}(\bm{\alpha})\geq
0$ (10)

Next we integrate over entire phase space and use the overlap formula
Leonhardt  $\text{tr}(AB)=2\pi\int
d^{2}\bm{\alpha}W_{A}(\bm{\alpha})W_{B}(\bm{\alpha})$ where $W_{A}$ and
$W_{B}$ are the Wigner functions of operators $A$ and $B$, respectively, and
obtain

$K_{\rho}(\bm{x},\bm{x^{\prime}})\geq e^{-|\bm{x}-\bm{x^{\prime}}|^{2}}$ (11)

We note that although in our derivation (11) we used the Wigner function, the
final result is independent of the Wigner function. On the virtue of the
Eq.(7), we call a kernel to be a nonclassical kernel if the inequality Eq.(11)
is violated. In the following, we give a few examples of quantum states and
their kernels.

Coherent state. The kernel of the coherent state is a Gaussian function

$K_{|\alpha\rangle\langle\alpha|}(x,x^{\prime})=e^{-|x-x^{\prime}|^{2}}$ (12)

which its width $\sigma=\frac{1}{\sqrt{2}}$ dictated by the vacuum
fluctuations of the optical field Leonhardt . One can conclude that the
coherent state satisfies the lower bound in (13), which is consistent with the
fact that the coherent state is the closest quantum state to a classical
field. The effect of dissipation on the coherent state is to reduce the
amplitude to $\sqrt{\eta}\alpha$ where $0\leq\eta\leq 1$ and $1-\eta$ is the
dissipation coefficient. From this observation, it follows that the kernel
function for the damped coherent state remains the same as in Eq.(12). Using
(12) and (13), one can define the following witness for the kernel’s
nonclassicality

$\mathcal{W}(\bm{x},\bm{x^{\prime}})=K_{\rho}(\bm{x},\bm{x^{\prime}})-K_{|\alpha\rangle\langle\alpha|}(\bm{x},\bm{x^{\prime}})$
(13)

which is positive for classical kernels and while can take negative values for
nonclassical kernels. Note that $\mathcal{W}(\bm{x},\bm{x})=0$ for
$\bm{x}=\bm{x^{\prime}}$.

Squeezed state. The squeezed state is defined as $|S\rangle=S(r)|0\rangle$
where $S(r)=e^{r(a^{\dagger 2}-a^{2})}$ is the squeezing operator and $r$ is
squeezing parameter. The action of squeezing operator in phase space can be
understood by the transformations $q\rightarrow e^{r}q$ and $p\rightarrow
e^{-r}p$. The Wigner function for squeezed state reads
$W_{\text{sq}}(q,p)=\frac{1}{\pi}e^{-(e^{2r}q^{2}+e^{-2r}p^{2})}$. After some
straightforward calculation, the kernel function of the damped squeezed state
is

$K_{|S\rangle\langle
S|}(x,x^{\prime})=e^{-\frac{(x_{1}-x^{\prime}_{1})^{2}}{1-\eta(1-e^{-2r})}}e^{-\frac{(x_{2}-x^{\prime}_{2})^{2}}{1-\eta(1-e^{2r})}}$
(14)

Single photon. The single photon state is defined as
$|1\rangle=a^{\dagger}|0\rangle$ with the Wigner function $W_{|1\rangle\langle
1|}(q,p)=\frac{2(q^{2}+p^{2})-1}{\pi}e^{-\frac{q^{2}+p^{2}}{2}}$. The single
photon state is a non Gaussian state which in contrast to squeezed state it
can take negative values. The single photon kernel function then reads

$K_{|1\rangle\langle
1|}(x,x^{\prime})=\frac{e^{-|x-x^{\prime}|^{2}}}{(1-2\eta+2\eta^{2})}\\\
[1-2\eta(1-\eta)+\eta|x-x^{\prime}|^{2}(2-\eta(4-|x-x^{\prime}|^{2}))]$ (15)

In Fig.(1) (upper row) we have plotted the kernel function for squeezed (we
set $x_{2}=x^{\prime}_{2}=0$ in Eq.(14)) and single-photon state,
respectively, for different value of loss. As can be seen, the effect of loss
for the squeezed state is to broaden the width of the kernel function and to
shift the minimum of the $\mathcal{W}$ to right; see also (14). From Fig. (1),
the same trend can be seen for a single-photon state with the difference that
the minimum of $\mathcal{W}$ is now shifted to left. In Fig.(1) (lower row),
the $\mathcal{W}(x,x^{\prime})$ is shown for the squeezed state (lower left)
and single-photon state (lower right). From Fig.(1), it is also clear that the
kernel’s nonclassicality will survive for the loss as high as $80\%$ for the
squeezed state and $50\%$ for the single-photon state.

Figure 1: The Kernel function for the squeezed state (upper left) and single
photon state (upper right) for different loss as shown in the inset. In the
lower row we show the non-classicality condition $\mathcal{W}(x,x^{\prime})$.
For the squeezing parameter we used $r=1$ and $x_{2}=x^{\prime}_{2}=0$.

Data distribution in feature space. In the following we discuss some
implications of relation (13). Firstly, consider arbitrary points
$x,x^{\prime}\in\mathcal{X}$, and their distance in feature space
$d_{K}^{2}(x,x^{\prime})=||\rho(x)-\rho(x^{\prime})||^{2}=\langle\rho(x)-\rho(x^{\prime}),\rho(x)-\rho(x^{\prime})\rangle=2(1-K(x,x^{\prime}))$.
From (13) it follows that

$d^{2}_{\text{NC}}(x,x^{\prime})>2(1-e^{-|x-x^{\prime}|^{2}})\geq
d^{2}_{\text{C}}(x,x^{\prime})$ (16)

where we have introduced $d_{\text{NC}}$ and $d_{\text{C}}$ are the distance
induced by non-classical and classical kernel, respectively, and used
$K(x,x)=K(x^{\prime},x^{\prime})=1$. From Eq.(16) one can conclude that a
nonclassical kernel leads to a better separation between data points.

Secondly, defining the center of mass of a data set by
$\bar{\rho}=\frac{1}{M}\sum_{i=1}^{M}\rho(x_{i})$, from (11) it follows that
$||\bar{\rho}_{\text{C}}||^{2}>||\bar{\rho}_{\text{NC}}||^{2}$ with
$||\rho(x)||^{2}=\langle\rho(x),\rho(x)\rangle$. As a consequence nonclassical
kernels results in a greater average distance from the center of mass of
training data. Another observation is that

$\frac{1}{M}\sum_{i=1}^{M}||\rho_{\text{NC}}(x_{i})-\bar{\rho}_{\text{NC}}||^{2}>\frac{1}{M}\sum_{i=1}^{M}||\rho_{\text{C}}(x_{i})-\bar{\rho}_{\text{C}}||^{2}$
(17)

therefore non-classical kernel results in a greater expected distance from the
centre of mass.

We would like to emphasize that the kernel functions in Eqs. (14,15) are easy
to estimate classically and are shown as simple applications of (13). Examples
of circuits leading to computationally expensive kernels for qubit systems are
given in NatureIBM ; Liu . A potential route for engineering computationally
expensive kernels could be in the optical circuit, including nonlinear
elements Gu ; schuld2019quantum .

Discussion and conclusion. One direction to extend the results of our paper is
to seek similar witnesses for discrete systems. One challenge in doing so is
that the phase space approach for qubits is still not quite understand
Raussendorf . An intermediate task could be to start with odd-dimensional
Hilbert space where a well-defined phase space formalism already exists
Gibbons .

It is also interesting to consider more complicated encoding such as
$x\rightarrow\rho^{n}(x)$ which leads to more extensive kernel
$K(x,x^{\prime})=\text{tr}(\rho^{n}(x)\rho^{n}(x^{\prime}))$. This task
requires generalizing (7) for $n+1$-points correlation functions.

We presented an approach for assessing the role of nonclassicality of the
quantum state in machine learning, focusing on continuous-variable systems. We
believe that quantifying the nonclassical nature of kernels could lead to a
better understanding of the potential quantum advantage in machine learning.

Acknowledgement. I acknowledge discussions with Christoph Simon, Wilten
Nicola, and Hsin-Yuan Huang. This work was supported by the Alberta Major
Innovation Fund (MIF) Quantum Technologies project and by the Natural Sciences
and Engineering Research Council (NSERC) of Canada.

## References

  * (1) A. Frank, et al. Nature 574 505 (2019).
  * (2) Zhong, Han-Sen, et al. Science 370 1460 (2020).
  * (3) J. M. Arrazola et al. Nature 591, 54(2021).
  * (4) V. Saggio et al. Nature 591 229 (2021).
  * (5) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549 7691 195 (2017).
  * (6) G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Rev. Mod. Phys. 91, 045002 (2019).
  * (7) M. Schuld, and N. Killoran, Phys. Rev. Lett. 102 040504 (2019).
  * (8) V. Havlíček, A. D. Córcoles, K. Temme, Kristan, A. W. Harrow, A. Kandala, Abhinav, J. M. Chow, J. M. Gambetta, Nature, 567 209 (2019).
  * (9) H.-Y. Huang, M. Broughton, M. Mohseni, R. Babbush, S. Boixo, H. Neven, J. R. McClean, Nat. Commun. 12, 2631 (2021).
  * (10) M.Schuld, arXiv:2101.11020 (2021).
  * (11) J. Eli Bourassa et al. Quantum 5, 392 (2021).
  * (12) G.R. Steinbrecher, J.P. Olson, D. Englund, J. Carolan, npj Quantum Inf. 5, 1 (2019).
  * (13) R. W. Spekkens, Phys. Rev. Lett. 101, 020401 (2008).
  * (14) M. Howard, J. Wallman, V. Veitch and J. Emerson Nature. 510 351 (2014).
  * (15) Martin Bohmann, Elizabeth Agudelo, Phys. Rev. Lett. 124, 133601 (2020).
  * (16) M. Bohmann, E. Agudelo, J. Sperling, Quantum 4, 343 (2020).
  * (17) N. Biagi, M. Bohmann, E. Agudelo, M. Bellini, A. Zavatta, Phys. Rev. Lett. 126, 023605 (2021).
  * (18) M. S. Kim, W. Son, V. Buzek, and P. L. Knight, Phys. Rev. A 65, 032323 (2002).
  * (19) N. Killoran, F.E. S. Steinhoff, M. B. Plenio, Phys. Rev. Lett. 116, 080402 (2016).
  * (20) B. Schölkopf, R. Herbrich, A. J. Smola, International conference on computational learning theory, 416, (2001).
  * (21) T. Hofmann, B. Schölkopf, and A. J. Smola, Alexander, The annals of statistics, 1171 (2008).
  * (22) J. Shawe-Taylor, and N. Cristianini. Kernel methods for pattern analysis. Cambridge university press, (2004).
  * (23) R. J. Glauber, Phys. Rev. 131, 2766 (1963).
  * (24) E. C. G. Sudarshan, Phys. Rev. Lett. 10, 277 (1963).
  * (25) U. Leonhardt, Measuring the Quantum State of Light, Cambridge University Press, Cambridge, 1997.
  * (26) R. Ghobadi, J.S Oberoi, E. Zahedinejhad, arXiv preprint, arXiv:1905.01390 (2019).
  * (27) R. Chatterjee and T. Yu, Quantum Information and Communication 17, 1292 (2017).
  * (28) Y. Liu, S. Arunachalam, and K. Temme, arXiv preprint, arXiv:2010.02174 (2020).
  * (29) M. Gu, C. Weedbrook, N. C. Menicucci, T. C. Ralph, P. van Loock, Phys. Rev. A 79 , 062318 (2009).
  * (30) R. Raussendorf, J. Bermejo-Vega, E. Tyhurst, C. Okay, M. Zurel, Phys. Rev. A 101, 012350 (2020).
  * (31) K. S. Gibbons, M. J. Hoffman, and W. K. Wootters, Phys. Rev. A 70, 062101 (2004).


#  HandBooster: Boosting 3D Hand-Mesh Reconstruction by
Conditional Synthesis and Sampling of Hand-Object Interactions

Hao Xu1 Haipeng Li2 Yinqiao Wang1 Shuaicheng Liu2 Chi-Wing Fu1
1The Chinese University of Hong Kong
2University of Electronic Science and Technology of China
<EMAIL_ADDRESS>{lihaipeng<EMAIL_ADDRESS>
###### Abstract

Reconstructing 3D hand mesh robustly from a single image is very challenging,
due to the lack of diversity in existing real-world datasets. While data
synthesis helps relieve the issue, the syn-to-real gap still hinders its
usage. In this work, we present HandBooster, a new approach to uplift the data
diversity and boost the 3D hand-mesh reconstruction performance by training a
conditional generative space on hand-object interactions and purposely
sampling the space to synthesize effective data samples. First, we construct
versatile content-aware conditions to guide a diffusion model to produce
realistic images with diverse hand appearances, poses, views, and backgrounds;
favorably, accurate 3D annotations are obtained for free. Then, we design a
novel condition creator based on our similarity-aware distribution sampling
strategies to deliberately find novel and realistic interaction poses that are
distinctive from the training set. Equipped with our method, several baselines
can be significantly improved beyond the SOTA on the HO3D and DexYCB
benchmarks. Our code will be released on
https://github.com/hxwork/HandBooster_Pytorch.

11footnotetext: Department of Computer Science and Engineering; Institute of
Medical Intelligence and XR (IMIXR).

##  1 Introduction

The task of reconstructing 3D hand mesh from a single image facilitates a wide
range of applications,  _e.g_., in AR/VR and human-computer interactions.
Recently-proposed data-driven methods show promising results in hand-object
interaction scenarios. Yet, their performance is largely limited by the
training data, since existing datasets typically lack diversity in hand
appearances/poses, views,  _etc_.

Existing real-world hand-object datasets are collected in laboratory or in-
the-wild scenes. For laboratory-captured datasets such as DexYCB [8] and HO3D
[22], they offer a large quantity of hand-object interaction samples with
relatively accurate 3D annotations. However, the variations in the samples are
still limited, since the data collection process is typically very tedious
with the expensive MoCap system. Conversely, for in-the-wild datasets,
_e.g_., YouTube-Hands [40], the data has richer variations, but they provide
only pseudo labels, without accuracy guarantee.

Figure 1: HandBooster significantly improves several baselines, making them
SOTA again on both HO3D and DexYCB.

Synthetic data, both rendering- and generative-based, is a workaround to avoid
tedious data collection while ensuring annotation precision. ObMan [23], YCB-
Afford [17], and ArtiBoost [71] use Blender [16], Maya [1], or PyRender [47]
to produce hand renderings. However, the results are unrealistic due to
simulated hand appearance and inconsistency in foreground/background lighting.
Also, they need extensive extra data such as HDR environment maps for lighting
and background augmentation. On the other hand, generative methods,  _e.g_.,
HOGAN [31], can alleviate these issues. Yet, they consider only novel view
synthesis but not other aspects of data diversity. Also, there is no evidence
that the hand-mesh reconstruction performance can be improved consistently on
existing methods. To sum up, the key challenge is how to generate realistic
and diverse hand-object interaction images with reliable annotations.

In this paper, we present HandBooster to uplift the data diversity and boost
3D hand-mesh reconstruction by training a conditional generative space on
hand-object interactions and purposely sampling the space to produce effective
data samples. In short, we first construct versatile content-aware conditions
to guide a diffusion model to produce realistic image samples. By this means,
we can controllably produce images of diverse hand appearances, poses, views,
and backgrounds, where precise 3D annotations are available for free. Further,
we design a novel condition creator based on our similarity-aware distribution
sampling strategies to find novel interaction poses distinct from the existing
ones, thus maximizing the training data quality.

First, directly mapping an input 3D mesh to 2D RGB values is challenging. So,
we decompose this process into two steps: (i) project the 3D mesh into a more
interpretable 2D image form while preserving its geometric information; and
(ii) utilize the 2D results as conditions on the diffusion model to enable
controllable generation of realistic images. Among the candidates for the 2D
conditions, inspired by [31], we empirically choose the comprehensible and
informative normal map and object texture. Further, observing that small
changes in 3D hand orientation can hardly be seen in 2D images, we embed 3D
hand orientation as an additional condition to ensure orientation-aware
generation.

Second, to enhance the reconstruction performance and generalizability, solely
utilizing existing hand-object grasping poses is inadequate. The synthesized
hand-object interaction samples should be: (i) realistic, ensuring natural
grasping poses; (ii) diverse, encompassing various grasping poses; and (iii)
novel, encouraging unseen poses. To achieve (i), we employ an optimization-
based method [63] to simulate grasping poses, followed by our validation
strategy to find natural poses. To promote (ii) and (iii), we further propose
similarity-aware sampling strategies, including an intra-distribution farthest
pose sampling strategy to avoid repeated or similar poses within both the real
and synthetic data and a cross-distribution sampling strategy to encourage the
likelihood of sampling novel-generated poses. Last, we train several baselines
using our generated samples together with real-world data, showing that their
performance can be significantly improved over the SOTA; see Fig. 1.

Our main contributions are summarized as follows,

  * •

We propose HandBooster, a new generative framework that utilizes content-aware
conditions to synthesize realistic hand-object images with diverse hand
appearances, poses, views, and backgrounds, as well as accurate annotations,
to boost the reconstruction performance.

  * •

We design the novel condition creator to effectively produce realistic and
diverse novel views and grasping poses, by designing the intra-distribution
farthest pose sampling strategy and the cross-distribution sampling strategy.

  * •

Extensive evaluations on two widely-used datasets show that HandBooster
achieves consistent performance gain on several methods, setting new SOTA
performance.

##  2 Related Works

#### 3D Hand-Mesh Reconstruction.

Extensive research has been conducted on 3D hand-mesh reconstruction from RGB
images. Most of them tackle the problem by regressing MANO coefficients [76,
79, 69, 23, 77, 5, 2, 81, 13, 78, 7, 3, 34, 44, 74, 60]. Others mainly regress
voxels [32, 50, 51, 70], implicit functions [48], and vertices [21, 40, 11,
42, 43, 12]. Despite careful network designs, reconstructing accurate 3D hand
meshes from monocular images remains challenging, particularly under severe
occlusions. Hence, some recent works [68, 9, 66, 65, 20] attempt to leverage
multi-frame information. A very recent work [72] uses prior knowledge in a
diffusion model to render object geometries to improve everyday object
reconstruction. Our approach is orthogonal to these methods and achieves top
performance when partnered with several recent baselines [52, 12, 66].

Figure 2: Our HandBooster framework. (i) The Novel Condition Creator prepares
and samples diverse and novel grasping poses against real-world and synthetic
distributions to create conditions. (ii) The Hand-Object Image Synthesizer
follows the conditions to generate image samples. (iii) The synthesized
samples can then be employed to effectively train different types of
reconstruction models.

#### Hand-Object Interaction Image Synthesis.

So far, few works have explored image synthesis of hand-object interactions.
These works are either rendering- or generative-based. The former employs
rendering tools such as Blender [16] and Maya [1] to produce synthetic data.
To generate grasping poses, some [17, 71, 33] design specific algorithms,
while others [23] use off-the-shell tools such as GraspIt [49] and recent
works [63, 67, 62]. However, when applied to 3D hand-mesh reconstruction,
these synthetic data inevitably introduce noticeable domain gaps compared to
real-world data, leading to inferior performance.

Generative-based methods can produce more realistic images. HOGAN [31]
synthesizes novel views using the target posture as guidance; yet, it cannot
effectively generate images of novel grasping poses and lacks other aspects of
diversity. Another work [73] generates hand graspings on RGB images that
contain objects by using a diffusion model conditioned on a generated hand
orientation mask. However, its diversities are still restricted by the input
images and the lack of annotations severely limits its usage in downstream
applications. In this work, we are able to synthesize realistic and diverse
hand-object interactions that encompass various appearances, grasping poses,
object types, and camera views, significantly enhancing the hand-mesh
reconstruction performance, as shown in Fig. 1.

#### Conditional Diffusion Model.

Diffusion model is a type of generative model that adopts the stochastic
diffusion process in thermodynamics [55]. Besides, it can also be formulated
as a score-based generative model [57, 58]. Recently, DDPM [28] models complex
data distributions through discrete steps. In this work, we focus mainly on
conditioned generation, including classifier-guided [18, 45] and classifier-
free [27] methods. Though LDM [53], ControlNet [75], or other related methods
[64, 30, 56, 37, 46, 4, 54, 29, 41, 35] appear as potential tools for our
work, they are primarily designed for producing high-resolution images or
exploiting data priors in pre-trained models, lacking task-specific designs to
meet the needs of 3D hand-mesh reconstruction,  _e.g_., plausible hand-object
interactions and hand orientation awareness. In this work, we propose various
content-aware conditions to guide the DDPM, demonstrating controllability and
fine-grained realistic results.

##  3 Method

###  3.1 Overview

Fig. 2 shows the overall HandBooster framework. First, we formulate the hand-
object image synthesizer, which utilizes content-aware conditions to generate
compatible hand-object images (Sec. 3.2). Next, to maximize its efficacy, we
design the novel condition creator, aiming to prepare and sample realistic,
diverse, and novel grasping poses as conditions (Sec. 3.3). Combining their
strengths, we can effectively generate realistic hand-object images with
annotations to train various reconstruction models (Sec. 3.4.)

###  3.2 Hand-Object Image Synthesizer

We first describe the process of generating realistic images from a given 3D
hand mesh $\mathbf{m}$ using our hand-object image synthesizer. To tackle the
challenge of constructing a mapping between the 3D coordinate space and the 2D
image space, we decompose the process into two steps, as inspired by [53, 27].
First, we perform a 3D-to-2D projection while retaining sufficient information
to construct content-aware conditions. Then, we adopt them to guide a
conditional diffusion model to generate realistic hand-object images.

#### Content-aware Conditions.

To minimize the information lost during the 3D-to-2D projection process and
reduce the learning difficulty, we aim to choose informative and interpretable
conditions for synthesizing hand-object images. We have considered several
candidates in 2D format, including skeleton, segmentation, texture, depth, and
normal maps, which can be divided into two categories: depth and normal maps
contain more topology information, while others are more semantic. To strike a
balance, a sound solution is to choose one from each category. As Fig. 3
shows, the texture map contains both shape and color knowledge, so it is
selected due to its high informativeness. Also, the normal map is selected for
its ease of interpretation, as different parts of the hand and object are more
distinguishable compared to the depth map. However, relying solely on 2D
images cannot capture small changes in the 3D hand orientation, leading to
large regression errors in the camera space during the reconstruction. Hence,
we design another condition to encourage orientation-aware generation, where
the hand orientation is represented using quaternion, embedded into latent
space, and incorporated into several stages of the diffusion model. This
inclusion allows for better handling of changes in hand orientation and gives
hints to generate arms (not included in conditions) in the image synthesis.

#### Controllable Realistic Image Synthesis.

We first utilize the classifier-free diffusion model [28, 27] for a
controllable generation of hand-object images. The forward diffusion is a
Markov chain of diffusion steps, in which Gaussian noise is added
progressively to a real-world data sample,
$\mathbf{x}_{0}\\!\\!\sim\\!\\!q(\mathbf{x})$, producing a noisy transition
sequence $\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T}$. To facilitate the
training, we use the parameterization method following [38] to sample
$\mathbf{x}_{t}$ at arbitrary timestamp. Then, we adopt a UNet-like
classifier-free denoising model $f_{\theta}(\cdot)$ to learn the reverse
diffusion, taking our content-aware conditions to control the generation of
the hand-object images from an isotropic Gaussian noise. After the training,
we can obtain a controllable model for synthesizing high-quality images with
fine-grained details, following our content-aware conditions.

Figure 3: Condition candidates. Normal and texture maps are more informative
than others; see the red/yellow bounding boxes.

During training, $\mathbf{x}_{0}\\!\in\\!\mathbb{R}^{128\\!\times
128\\!\times\\!3}$ are the same image used for 3D hand-mesh reconstruction. To
improve the quality of the generated data and employ them in our task, we
incorporate our content-aware conditions $\mathbf{y}$ with $\mathbf{x}_{t}$
and $t$ to generate samples $\hat{\mathbf{x}}_{0}$ using the denoising model
$f_{\theta}(\cdot)$,

$\small\hat{\mathbf{x}}_{0}=f_{\theta}(\operatorname{Concat}(\mathbf{n},\mathbf{c}),\mathbf{h}_{q},\operatorname{PE}(t)),\vspace{-2mm}$
(1)

where $\operatorname{Concat}(\cdot)$ denotes the channel-wise concatenate
operation; $\mathbf{n}$ and $\mathbf{c}$ indicate the normal and texture maps,
respectively; and $\mathbf{h}_{q}$ is the hand orientation, represented using
quaternion to ensure continuity; and the timestamp $t$ is encoded through
positional embedding (PE) [61]. Following [37], we let the denoising model
$f_{\theta}(\cdot)$ predict RGB values and calculate L2 loss with
$\mathbf{x}_{0}$. Formally, we have

$\small\mathcal{L}_{DM}=\mathbf{w}\frac{\bar{\alpha}_{t}}{1-\bar{\alpha}_{t}}||\hat{\mathbf{x}}_{0}-\mathbf{x}_{0}||_{2}^{2},\vspace{-2mm}$
(2)

where $\mathbf{w}$ is a pixel-wise weighting map to enhance the foreground,
since the generation quality of the background is less important. We use the
hand-object segmentation map as the mask and set the weight to 0.1 for the
background.

In the reverse diffusion, synthetic samples $\hat{\mathbf{x}}_{0}$ are
generated from randomly-sampled Gaussian noise
$\mathbf{x}_{t}\\!\in\\!\mathcal{N}(\mathbf{0},\mathbf{I})$ with the content-
aware conditions $\mathbf{n}$, $\mathbf{c}$, and $\mathbf{h_{q}}$. After
certain timestamps, the generated images $\hat{\mathbf{x}}_{0}$ would become
realistic and the input 3D meshes $\mathbf{m}$ are exactly the corresponding
ground truths, forming a training pair $(\hat{\mathbf{x}}_{0},\mathbf{m})$.

#### Edge Cases Filtering.

Though we can effectively generate realistic images, it is unlikely to fully
avoid undesired artifacts, so filtering is necessary. To mitigate the negative
effects of training the reconstruction models with such image samples, we
propose a selection process to filter out edge cases. Specifically, we adopt a
3D hand-mesh reconstruction model pre-trained on the same training set to
evaluate the generated images $\hat{\mathbf{x}}_{0}$. Here, we calculate the
3D joint and vertex errors between the predicted results and the corresponding
ground truths $\mathbf{m}$. Considering that the model should yield reasonable
outcomes on eligible synthetic images that align with the distribution of
real-world data, we thus exclude edge-case samples that exhibit significant
errors over predetermined thresholds in the selection.

###  3.3 Novel Condition Creator

To effectively generalize the reconstruction models to handle unseen scenarios
while avoiding potential over-fitting, it is essential to incorporate diverse
hand-object interaction poses. However, simply augmenting existing conditions
can only produce novel views. Doing so is insufficient to boost the
reconstruction performance. Hence, we design the novel condition creator to
construct novel and diverse conditions by finding unseen poses and combining
them with the existing ones via our distribution sampling strategies.

#### Pose Preparation.

We first try to fully utilize the grasping poses in the current training set
by augmenting their hand orientations. Since not all frames involve object
grasping, the data is split into “grasping” and “non-grasping” parts. Only
“grasping” poses are augmented to synthesize novel views, while others remain
unchanged to preserve realistic motions. Yet, existing datasets [8, 22] do not
provide labels for grasping status, so we derive the grasping status from the
object pose automatically. We calculate the isotropic Relative Rotation Error
(RRE) and Relative Translation Error (RTE) between the initial and current
poses:

$\small\mathrm{RRE}\\!=\\!\arccos(\frac{\mathrm{trace}({\mathbf{o}_{r}^{t}}^{T}\mathbf{o}_{r}^{0}\\!-\\!1)}{2}),\;\mathrm{RTE}\\!=\\!||{\mathbf{o}_{t}^{t}-\mathbf{o}_{t}^{0}}||_{2},$
(3)

where $\mathbf{o}_{r}$ and $\mathbf{o}_{t}$ denote the object rotation matrix
and translation vector, respectively; and the superscripts $0$ and $t$ denote
the first and $t$-th frames, respectively, in the same sequence. If the
difference exceeds a pre-defined threshold (empirically setting $5^{\circ}$ on
RRE and $10mm$ on RTE), the frame is labeled as “grasping”. To produce novel
views, the orientations of the “grasping” poses are perturbed in a certain
range and then rendered as conditions.

Next, we focus on generating novel hand-object grasping poses. Specifically,
we use the same YCB [6] objects as in DexYCB [8] and obtain initial poses by
simulating the process of falling from a random height to a plane. To generate
a grasping pose, we employ the recent work [63] for its fast convergence and
high success rate. Please refer to [67] for the details. We perform this
simulation 10,000 times on each object. Though penalties have been applied to
encourage realism, undesired poses still exist. Thus, we design a validation
process to select high-quality poses. A qualified pose must meet three
criteria: (i) the hand must make contact with the object (it may fail to grasp
small/thin objects); (ii) no obvious hand-object intersection (the volume of
the intersection part should be smaller than a pre-defined threshold); and
(iii) no self-penetration for the hand. Finally, to avoid introducing domain
gaps, we align the orientation of the generated pose to a random one from the
training set and apply perturbation as augmentation.

Data: Set of poses $\mathbf{P}$, number of sampled poses $M$

Result: Sampled set of poses $\mathbf{Q}$

$\mathbf{Q}\leftarrow\varnothing\cup\\{\mathbf{v}_{0}\in_{\scriptstyle{R}}\mathbf{P}\\}$;

while  _$|\mathbf{Q}| <M$_ do

$\mathbf{v}_{max}\leftarrow\text{None}$;

$d_{max}\leftarrow-\infty$;

for  _$\mathbf{v}_{i}\in\mathbf{P},\;\mathbf{v}_{i}\notin\mathbf{Q}$_ do

$d_{i}\leftarrow\min_{\mathbf{v}_{j}\in\mathbf{Q}}D_{c}(\mathbf{v}_{i},\mathbf{v}_{j})$;

if  _$d_{i} >d_{max}$_ then

$d_{max}\leftarrow d_{i}$;

$\mathbf{v}_{max}\leftarrow\mathbf{v}_{i}$;

end if

end for

$\mathbf{Q}\leftarrow\mathbf{Q}\cup\\{\mathbf{v}_{max}\\}$;

end while

Algorithm 1 Farthest Pose Sampling Algorithm

#### Intra-distribution Sampling.

The grasping pose partly depends on the initial object pose. Yet, the number
of feasible grasping poses is inherently limited by the shape of the object,
_e.g_., a bowl can only be placed upright or upside down on a plane, each
poses having very limited ways of grasping. Though it is possible to grasp the
same object multiple times, the resulting poses could be too similar to one
another, as different grasping poses do not occur with equal probability. To
promote pose diversity and avoid cases that dominate the distribution,
inspired by [19], we propose a farthest-pose-sampling (FPS) strategy to select
poses as evenly as possible for each object category.

For real-world and synthetic grasping pose sets $\mathbf{P}_{r}$ and
$\mathbf{P}_{s}$, we perform this intra-distribution sampling separately to
obtain the sampled sets $\mathbf{Q}_{r}$ and $\mathbf{Q}_{s}$. The subscript
is ignored for brevity. The pose vector $\mathbf{v}$ is constructed as

$\small\mathbf{v}=\operatorname{Concat}(\mathbf{h}_{s},\mathbf{o}_{q},\mathbf{o}_{t}),\quad\mathbf{v}\in\mathbb{R}^{n}$
(4)

where $\mathbf{h}_{s}$ and $\mathbf{o}_{q}$ denote the quaternion
representation of the hand pose and object rotation, respectively. Note that
the global orientation is removed from the grasping pose, which means we
compute the distance at the canonical pose. As described in Alg. 1, FPS begins
by initializing the sampled pose set $\mathbf{Q}$ using a random sample
$\mathbf{v}_{0}$ from the input pose set $\mathbf{P}$. While the number of
samples in $\mathbf{Q}$ is less than $M$ ($M$ and $N$ for real-world and
synthetic data, respectively), it traverses the remaining poses in
$\mathbf{P}$ and selects $\mathbf{v}_{max}$ with the largest nearest distance
$d_{max}$ to $\mathbf{Q}$ iteratively. The distance between two pose vectors
$\mathbf{v}_{i}\\!\in\\!\mathbf{P},\mathbf{v}_{i}\\!\notin\\!\mathbf{Q}$ and
$\mathbf{v}_{j}\\!\in\\!\mathbf{Q}$ is computed using the cosine distance
$D_{c}(\cdot,\cdot)$, _i.e_.,

$\small
D_{c}(\mathbf{v}_{i},\mathbf{v}_{j})\\!=\\!\frac{\mathbf{v}_{i}\cdot\mathbf{v}_{j}}{||\mathbf{v}_{i}||||\mathbf{v}_{j}||}\\!=\\!\frac{\sum_{k=1}^{n}v_{i}^{k}v_{j}^{k}}{\sqrt{\sum_{k=1}^{n}{v_{i}^{k}}^{2}}\\!\cdot\\!\sqrt{\sum_{k=1}^{n}{v_{j}^{k}}^{2}}}.\vspace{-1mm}$
(5)

As Fig. 4 (left) shows, the sampled poses are more distinctive from each other
when applying our FPS strategy.

Figure 4: Visualizations in sampling. For three object categories in HO3D, we
show the similarity of sampled real-world poses with our FPS or random
sampling (left) and the histograms of similarity between sampled real-world
and synthetic poses (right).

| Methods | _Procrustes Alignment_ | _Root-relative_
---|---|---|---
| J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ | J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$
| S2Hand [13] | 7.3 | 85.5 | - | - | - | - | - | - | - | - | - | -
| VIBE [39] | 6.4 | 87.1 | - | - | - | - | - | - | - | - | - | -
| TCMR [15] | 6.3 | 87.5 | - | - | - | - | - | - | - | - | - | -
| H2ONet [66] | 5.3 | 89.4 | 5.2 | 89.6 | 80.5 | 99.3 | 13.7 | 74.8 | 12.7 | 76.6 | 52.1 | 92.3
Multi-frame | Deformer [20] | 5.2 | 89.6 | - | - | - | - | - | - | - | - | - | -
| METRO [42] | 7.0 | - | - | - | - | - | 15.2 | - | - | - | - | -
| Spurr  _et al_. [59] | 6.8 | 86.4 | - | - | - | - | 17.3 | 69.8 | - | - | - | -
| MeshGraphormer [43] | 6.4 | 87.2 | - | - | - | - | 15.2 | - | - | - | - | -
| Liu  _et al_. [44] | 6.6 | - | - | - | - | - | 15.3 | - | - | - | - | -
| Tse  _et al_. [60] | - | - | - | - | - | - | 16.1 | 72.2 | - | - | - | -
| HandOccNet [52] | 5.8 | 88.4 | 5.5 | 89.0 | 78.0 | 99.0 | 14.0 | 74.8 | 13.1 | 76.6 | 51.5 | 92.4
| \+ Our HandBooster | 5.2 | 89.6 | 5.0 | 89.9 | 81.3 | 99.2 | 11.9 | 77.8 | 11.5 | 78.5 | 55.6 | 93.6
| MobRecon [12] | 6.4 | 87.3 | 5.6 | 88.9 | 78.5 | 98.8 | 14.2 | 73.7 | 13.1 | 76.1 | 50.8 | 92.1
| \+ Our HandBooster | 6.2 | 87.6 | 5.4 | 89.3 | 79.2 | 99.1 | 13.2 | 74.9 | 12.3 | 76.6 | 51.8 | 92.5
| H2ONet [66] | 5.7 | 88.9 | 5.5 | 89.1 | 80.1 | 99.0 | 14.0 | 74.6 | 13.0 | 76.2 | 51.3 | 92.1
Monocular | \+ Our HandBooster | 5.1 | 89.8 | 5.1 | 89.8 | 81.3 | 99.2 | 12.9 | 76.0 | 12.5 | 76.5 | 52.1 | 92.2

Table 1: Results on “S0” (default) data split of DexYCB. -: unavailable
results. Our method consistently boosts all three baselines.

Methods | _Procrustes Alignment_ | _Root-relative_
---|---|---
J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ | J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$
[52] | 6.6 |  | 86.8 |  | 6.3 |  | 87.3 |  | 72.1 |  | 98.4 |  | 17.4 |  | 67.6 |  | 16.8 |  | 68.5 |  | 40.7 |  | 86.2 |
\+ Ours | 6.1 | +0.5 | 87.8 | +1.0 | 5.9 | +0.4 | 88.3 | +1.0 | 75.4 | +3.3 | 98.7 | +0.3 | 15.8 | +1.6 | 70.4 | +2.8 | 15.3 | +1.5 | 71.3 | +2.8 | 44.1 | +3.4 | 88.5 | +2.3
[12] | 6.9 |  | 86.2 |  | 6.4 |  | 87.2 |  | 72.1 |  | 98.4 |  | 18.3 |  | 66.0 |  | 17.4 |  | 67.5 |  | 39.7 |  | 85.6 |
\+ Ours | 6.6 | +0.3 | 86.7 | +0.5 | 6.1 | +0.3 | 87.9 | +0.7 | 74.4 | +2.3 | 98.6 | +0.2 | 17.5 | +0.8 | 67.7 | +1.7 | 16.7 | +0.7 | 69.1 | +1.6 | 41.1 | +1.4 | 86.8 | +1.2
[66] | 6.2 |  | 87.7 |  | 6.4 |  | 87.2 |  | 72.4 |  | 98.6 |  | 17.6 |  | 67.9 |  | 17.2 |  | 68.4 |  | 39.4 |  | 86.2 |
\+ Ours | 6.0 | +0.2 | 88.1 | +0.4 | 6.0 | +0.4 | 88.1 | +0.9 | 75.4 | +3.0 | 98.7 | +0.1 | 17.0 | +0.6 | 68.9 | +1.0 | 16.4 | +0.8 | 69.7 | +1.3 | 41.5 | +2.1 | 87.2 | +1.0

Table 2: Results on “S1” (unseen subjects) data split of DexYCB. Our method
consistently improves all baselines for all metrics.

#### Cross-distribution Sampling.

Likewise, the synthetic grasping poses should also be novel and diverse.
Simply blending them with real-world poses leads to uneven distribution, as
shown in Fig. 4 (right), due to the presence of similar samples. While our FPS
strategy can mitigate this issue, it can not ensure sufficient inclusion of
real-world poses, since $M\\!\ll\\!N$, _e.g_., $M\\!=\\!10$ and $N\\!=\\!500$
for HO3D. Additionally, the number of possible unique poses is unknown, making
it challenging to define precise criteria for identifying different pose
categories. Thus, we leverage the similarity relationship between the real-
world and synthetic poses to adaptively control the sampling probability of
the synthetic data. Specifically, for each object category, we calculate the
similarity matrix and convert it into a discrete probability distribution $P$
for the sampling process:

$\small\vspace{-1mm}P(X\\!=\\!\mathbf{v}_{i}^{s})=\operatorname{Norm}(\sum_{j=1}^{M}(1\\!-\\!D_{c}(\mathbf{v}_{i}^{s},\mathbf{v}_{j}^{r}))),$
(6)

where $\mathbf{v}_{i}^{s}\\!\in\\!\mathbf{Q}_{s}$ and
$\mathbf{v}_{j}^{r}\\!\in\\!\mathbf{Q}_{r}$. $\operatorname{Norm}(\cdot)$
denotes the min-max normalization. The sampling probability of a synthetic
pose is inversely related to its similarity to all real-world poses, yielding
a more balanced distribution.

###  3.4 Hand Mesh Reconstruction

In the task of 3D hand-mesh reconstruction from a single RGB image, the goal
is to estimate the 3D hand pose and shape. To demonstrate the effectiveness of
our approach, we adopt it to train two commonly-used pipelines,  _i.e_., MANO-
based and vertex-based methods. The former represents the hand using a
predefined template and utilizes MANO coefficients to control its pose and
shape, which are regressed directly from the extracted features of the input
image. The latter typically regresses the 3D coordinates of hand vertices in a
coarse-to-fine manner. In this work, we select three recent methods as our
baselines: HandOccNet (MANO-based), MobRecon (vertex-based), and H2ONet
(vertex-based). They not only employ different hand-mesh representations but
also utilize unique network architectures and scalable input resolutions. By
applying our synthesized data to these baselines, we can effectively evaluate
the generalizability and versatility of our approach.

##  4 Experiments

###  4.1 Experimental Settings

#### Datasets.

We employ the following commonly-used hand-object benchmark datasets in our
experiments. (i) DexYCB [8] (a real-world dataset): we use the default “S0”
split (train/test: 406,888/78,768 samples) and the more challenging “S1” split
with unseen subjects (train/test: 7/2 subjects). (ii) HO3D (version 2) [22] (a
real-world dataset): note that evaluation can only be done by submitting
results to the official server. (iii) ObMan [23] (a rendering-based synthetic
dataset): we select it as the competitor of our generated data. Note that
other rendering/generative-based candidates are not available, including YCB-
Afford [17] (download link not accessible), ArtiBoost [71] (data not open-
sourced), and HOGAN [31] (missing essential files for training). To show the
improvement in generalizability, we additionally use MOW [7] as the in-the-
wild test set, which provides 512 annotated samples.

#### Evaluation Metrics.

We adopt common metrics following [8, 22, 12, 52, 66]. J-PE/V-PE denotes the
root-relative joint/vertex position error, measuring the average Euclidean
distance in $mm$ between the predicted and ground-truth 3D hand joint/vertex
coordinates. J-AUC/V-AUC computes the area under the percentage of correct
keypoints (PCK) curve in several error thresholds for joint/vertex. F@5 and
F@15 measure the harmonic mean of the recall and precision for vertex with
$5mm$ and $15mm$ thresholds. Procrustes Alignment (PA) aligns the orientation,
translation, and scale of the estimated results to match the ground truths.
Fréchet Inception Distance [26] (FID) measures the image fidelity.

#### Implementation Details.

Adam optimizer [38] is applied to train the diffusion model and the
reconstruction methods. 2D conditions are 256$\times$256 and rendered using
PyRender. The diffusion model is trained on 16 NVIDIA 2080Ti GPUs with a batch
size of 256 and a learning rate of 0.00008 for 700,000/200,000 iterations on
DexYCB/HO3D. Please refer to our supp. material for other details.

| Methods | J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$
---|---|---|---|---|---|---|---
| Hasson  _et al_. [24] | 11.4 | 77.3 | 11.4 | 77.3 | 42.8 | 93.2
| Hasson  _et al_. [25] | - | - | 14.7 | - | 39.0 | 88.0
| S2Hand [13] | 11.4 | 77.3 | 11.2 | 77.7 | 45.0 | 93.0
| Liu  _et al_. [44] | 9.8 | - | 9.4 | 81.2 | 53.0 | 95.7
| VIBE [39] | 9.9 | - | 9.5 | - | 52.6 | 95.5
| TCMR [15] | 11.4 | - | 10.9 | - | 46.3 | 93.3
| TempCLR [80] | 10.6 | - | 10.6 | - | 48.1 | 93.7
| Deformer [20] | 9.4 | - | 9.1 | - | 54.6 | 96.3
Multi-frame | H2ONet [66] | 8.5 | 82.9 | 8.6 | 82.8 | 57.0 | 96.6
| Pose2Mesh [14] | 12.5 | - | 12.7 | - | 44.1 | 90.9
| I2L-MeshNet [50] | 11.2 | - | 13.9 | - | 40.9 | 93.2
| ObMan [23] | 11.1 | - | 11.0 | 77.8 | 46.0 | 93.0
| HO3D [22] | 10.7 | 78.8 | 10.6 | 79.0 | 50.6 | 94.2
| METRO [42] | 10.4 | - | 11.1 | - | 48.4 | 94.6
| Liu  _et al_. [44] | 10.2 | 79.7 | 9.8 | 80.4 | 52.9 | 95.0
| I2UV-HandNet [10] | 9.9 | 80.4 | 10.1 | 79.9 | 50.0 | 94.3
| Tse  _et al_. [60] | - | - | 10.9 | - | 48.5 | 94.3
| AMVUR [36] | 8.3 | 83.5 | 8.2 | 83.6 | 60.8 | 96.5
| HandOccNet [52] | 9.1 | 81.9 | 9.0 | 81.9 | 56.1 | 96.2
| HandOccNet* [52] | 9.6 | 80.8 | 9.6 | 80.7 | 52.4 | 95.4
| \+ Our HandBooster | 8.5 | 82.9 | 8.6 | 82.9 | 57.7 | 97.2
| MobRecon [12] | 9.4 | 81.3 | 9.5 | 81.0 | 53.3 | 95.5
| MobRecon† [12] | 9.2 | 81.6 | 9.4 | 81.2 | 53.8 | 95.7
| \+ Our HandBooster | 8.7 | 82.6 | 8.8 | 82.5 | 56.1 | 97.0
| H2ONet [66] | 9.0 | 82.0 | 9.0 | 81.9 | 55.4 | 96.0
Monocular | \+ Our HandBooster | 8.2 | 83.6 | 8.4 | 83.2 | 58.5 | 97.2

Table 3: Results on HO3D (_Procrustes Alignment_). ∗: reproduced results using
its official code. †: complement data is used. -: unavailable results. Our
method brings improvement consistently.

###  4.2 Comparison with State-of-the-art Methods

#### Evaluation on DexYCB.

We first conduct quantitative comparisons on DexYCB. To demonstrate the
effectiveness of our method, we evaluate metrics before and after performing
PA, as presented in Tab. 1 and Tab. 2 for “S0” and “S1” data splits,
respectively. Additionally, Fig. 5 (left) visualizes the root-relative mesh
PCK/AUC comparison for the more challenging “S1” split. Our synthetic data
significantly improves the performance of HandOccNet [52] and the monocular-
based H2ONet [66], enabling them even surpass some multi-frame methods, such
as Deformer [20]. MobRecon [12] exhibits relatively small performance gains
due to its mobile-friendly designs. Its capability is limited by its lower
FLOPs (0.46G) and the number of parameters (8.23M) compared to H2ONet
(0.74G/25.88M) and HandOccNet (15.47G/37.22M). Our HandBooster consistently
boosts the performance of all three baselines across all metrics and two data
splits, giving strong evidence of its effectiveness. It even outperforms
multi-frame methods (see Tab. 1), by taking only a single view as its input.

Fig. 6 (a-b) and (d) show qualitative results on DexYCB. Note that we directly
use models pre-trained on DexYCB’s “S0” split to test on MOW. Comparing the
results w/ and w/o using our data, the performance gains clearly show
HandBooster’s efficacy and generalizability. Fig. 7 shows some of the
generated samples with novel views/poses, where the fine-grained alignment
with conditions and the realistic appearance show the controllability of
HandBooster. More results are shown in the supp. material.

#### Evaluation on HO3D.

We perform the same experiments on HO3D. Since the ground truths of the test
set are not publicly available, some results are from previous papers and the
official evaluation server. Tabs. 3 and 4 show results before and after PA,
respectively. To facilitate comparison, we also provide the mesh PCK/AUC after
PA, visualized in Fig. 5 (right). It is clear that all our baselines
consistently achieve improved precision on all the metrics when boosted by our
generated data, showing the great effectiveness and robustness of our method.
For an intuitive comparison, we present visual comparisons before and after
applying our method in Fig. 6 (c). Benefiting from the diverse and novel
samples in our generated data, all these baselines become able to produce
plausible shapes and estimate accurate hand orientations, even under severe
occlusions in the inputs.

Methods | J-PE $\downarrow$ | J-AUC $\uparrow$ | V-PE $\downarrow$ | V-AUC $\uparrow$ | F@5 $\uparrow$ | F@15 $\uparrow$
---|---|---|---|---|---|---
[52] | 24.9 |  | 53.9 |  | 24.2 |  | 55.1 |  | 26.0 |  | 72.9 |
[52]* | 25.1 |  | 53.3 |  | 24.5 |  | 54.4 |  | 25.6 |  | 72.8 |
\+ Ours | 21.1 | +4.0 | 59.4 | +6.1 | 20.5 | +4.0 | 60.4 | +6.0 | 28.7 | +3.1 | 77.9 | +5.1
[12] | 25.2 |  | 53.7 |  | 24.4 |  | 55.0 |  | 26.4 |  | 72.0 |
\+ Ours | 23.4 | +1.8 | 56.5 | +2.8 | 22.6 | +1.8 | 57.9 | +2.9 | 27.7 | +1.3 | 75.3 | +3.3
[66] | 26.3 |  | 52.3 |  | 25.5 |  | 53.5 |  | 24.9 |  | 71.5 |
\+ Ours | 24.0 | +2.3 | 56.7 | +4.4 | 23.3 | +2.2 | 57.7 | +4.2 | 26.6 | +1.7 | 74.4 | +2.9

Table 4: Results on HO3D (_Root-relative_). ∗: reproduced results using its
official code. Our method boosts baselines significantly.  Figure 5: The mesh
AUC comparison under different thresholds. All baselines are improved
consistently after applying our data. Figure 6: Qualitative comparison of [66,
52, 12] w/ and w/o our HandBooster. (For each case, left: predicted results,
right: error map.) Figure 7: Generated examples for novel views/poses with
realistic hand appearance and backgrounds.

###  4.3 Ablation Studies

We conduct ablation studies on DexYCB to evaluate the effectiveness of
HandBooster, as shown in Tab. 5. MobRecon is selected as the baseline due to
its fast training speed.

#### Rendering-based Synthesis.

We show the importance of realism in data generation by comparing our data
with ObMan, where hand appearance, grasping pose, and background are randomly
selected for rendering. Comparing Rows (a) and (b), introducing ObMan only
slightly improves the root-relative metrics after PA. Correspondingly,
comparing Rows (a) and (f), our data enhances performance on all metrics even
without the adoption of our other techniques, revealing the importance of
realism in the data generation. Further, we compute the FID score between our
data/ObMan and the training set of DexYCB. The scores of 2.91/8.52 reflect the
realism of our generated data.

#### Content-aware Conditions.

To better understand the influence of different conditions for 3D hand-mesh
reconstruction, we evaluate various candidate combinations. Due to the lack of
shape information, the skeleton and foreground segmentation are not used in
this analysis. The results, shown in Rows (d-f) of Tab. 5, reveal that
incorporating both normal and texture yields the highest performance.
Substituting either component with alternative candidates leads to a
significant decrease in performance, thus supporting our idea of constructing
conditions to be both informative and easy to understand. Further, by
introducing embedded hand orientation as an additional control, our model
shows noticeable improvements in root-relative metrics, further validating the
effectiveness of our design.

#### Novel Condition Creator.

We also investigate the impact of components in our novel condition creator.
Comparing Rows (f) and (g), though only utilizing novel grasping poses boosts
root-relative performance, certain poses may dominate the entire distribution
and limit the performance. Comparing Rows (h-i) with (g), performing the
intra- and cross-distribution sampling sequentially brings boosts across all
metrics clearly, demonstrating their necessities. Further experimental results
are presented in the supp. material.

| Models | _Root-relative_ | _Procrustes Align._
---|---|---|---
| J-PE $\downarrow$ | V-PE $\downarrow$ | J-PE $\downarrow$ | V-PE $\downarrow$
(a) | Baseline | 14.20 | 13.05 | 6.36 | 5.59
(b) | w/ ObMan | 14.00 | 13.00 | 6.40 | 5.60
(c) | w/ Depth & Segment. | 14.91 | 13.94 | 6.64 | 5.85
(d) | w/ Normal & Segment. | 14.82 | 13.85 | 6.60 | 5.81
(e) | w/ Normal & Texture | 13.93 | 12.92 | 6.37 | 5.53
(f) | \+ Embedded Orientation | 13.79 | 12.78 | 6.33 | 5.49
(g) | \+ Novel Conditions | 13.51 | 12.57 | 6.34 | 5.55
(h) | \+ Intra-dist. Sampling | 13.42 | 12.42 | 6.28 | 5.46
(i) | \+ Cross-dist. Sampling | 13.25 | 12.34 | 6.20 | 5.36

Table 5: Ablation study on major components.

##  5 Conclusion

We presented HandBooster, a new generative method to boost 3D hand-mesh
reconstruction by enhancing the data diversity. First, we create a conditional
generative space, from which we can controllably produce realistic and diverse
hand-object images with reliable 3D annotations. Then, we explore this space
to produce novel and diverse training samples by formulating a novel condition
creator and two similarity-aware sampling strategies. Extensive experiments on
three baselines and two common benchmarks demonstrate our effectiveness and
SOTA performance.

#### Acknowledgments

This work was supported by the Research Grants Council of the Hong Kong
Special Administrative Region, China (Project No. T45-401/22-N and No. CUHK
14201921) and the National Natural Science Foundation of China under grant No.
62372091. Hao Xu thanks for the care & support from Yutong Zhang & his family.

## References

  * Autodesk, INC. [2018] Autodesk, INC.  Maya.  https:/autodesk.com/maya, 2018.
  * Baek et al. [2019] Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim.  Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering.  In _CVPR_ , pages 1067–1076, 2019.
  * Baek et al. [2020] Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim.  Weakly-supervised domain adaptation via GAN and mesh model for estimating 3D hand poses interacting objects.  In _CVPR_ , pages 6121–6131, 2020.
  * Bao et al. [2022] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.  Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.  _arXiv preprint arXiv:2201.06503_ , 2022.
  * Boukhayma et al. [2019] Adnane Boukhayma, Rodrigo de Bem, and Philip H.S. Torr.  3D hand shape and pose from images in the wild.  In _CVPR_ , pages 10843–10852, 2019.
  * Calli et al. [2015] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  The YCB object and model set: Towards common benchmarks for manipulation research.  In _ICRA_ , pages 510–517. IEEE, 2015.
  * Cao et al. [2021] Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik.  Reconstructing hand-object interactions in the wild.  In _ICCV_ , pages 12417–12426, 2021.
  * Chao et al. [2021] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al.  DexYCB: A benchmark for capturing hand grasping of objects.  In _CVPR_ , pages 9044–9053, 2021.
  * Chen et al. [2021a] Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, and Xiaohui Xie.  Temporal-aware self-supervised learning for 3D hand pose and mesh estimation in videos.  In _WACV_ , pages 1050–1059, 2021a.
  * Chen et al. [2021b] Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan.  I2UV-HandNet: Image-to-UV prediction network for accurate and high-fidelity 3D hand mesh modeling.  In _ICCV_ , pages 12929–12938, 2021b.
  * Chen et al. [2021c] Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng.  Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration.  In _CVPR_ , pages 13274–13283, 2021c.
  * Chen et al. [2022] Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo.  MobRecon: Mobile-friendly hand mesh reconstruction from monocular image.  In _CVPR_ , pages 20544–20554, 2022.
  * Chen et al. [2021d] Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan.  Model-based 3D hand reconstruction via self-supervised learning.  In _CVPR_ , pages 10451–10460, 2021d.
  * Choi et al. [2020] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  Pose2Mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose.  In _ECCV_ , pages 769–787, 2020.
  * Choi et al. [2021] Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee.  Beyond static features for temporally consistent 3D human pose and shape from a video.  In _CVPR_ , pages 1964–1973, 2021.
  * Community [2019] Blender Online Community.  Blender.  http://www.blender.org, 2019.
  * Corona et al. [2020] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez.  GANHand: Predicting human grasp affordances in multi-object scenes.  In _CVPR_ , pages 5031–5041, 2020.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Eldar et al. [1997] Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi.  The farthest point strategy for progressive image sampling.  _IEEE TIP_ , 6(9):1305–1315, 1997.
  * Fu et al. [2023] Qichen Fu, Xingyu Liu, Ran Xu, Juan Carlos Niebles, and Kris M Kitani.  Deformer: Dynamic fusion transformer for robust hand pose estimation.  _arXiv preprint arXiv:2303.04991_ , 2023.
  * Ge et al. [2019] Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan.  3D hand shape and pose estimation from a single RGB image.  In _CVPR_ , pages 10833–10842, 2019.
  * Hampali et al. [2020] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  HOnnotate: A method for 3D annotation of hand and object poses.  In _CVPR_ , pages 3196–3206, 2020.
  * Hasson et al. [2019] Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In _CVPR_ , pages 11807–11816, 2019.
  * Hasson et al. [2020] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid.  Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction.  In _CVPR_ , pages 571–580, 2020.
  * Hasson et al. [2021] Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev.  Towards unconstrained joint hand-object reconstruction from RGB videos.  In _3DV_ , pages 659–668, 2021.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  In _NeurIPS_ , 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , pages 6840–6851, 2020.
  * Hoogeboom et al. [2023] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  Simple diffusion: End-to-end diffusion for high resolution images.  _arXiv preprint arXiv:2301.11093_ , 2023.
  * Hu et al. [2022a] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _ICLR_ , 2022a.
  * Hu et al. [2022b] Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li.  Hand-object interaction image generation.  In _NeurIPS_ , pages 23805–23817, 2022b.
  * Iqbal et al. [2018] Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz.  Hand pose estimation via latent 2.5D heatmap regression.  In _ECCV_ , pages 118–134, 2018.
  * Jian et al. [2023] Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu.  Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose.  In _ICCV_ , pages 14713–14724, 2023.
  * Jiang et al. [2021] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang.  Hand-object contact consistency reasoning for human grasps generation.  In _ICCV_ , pages 11107–11116, 2021.
  * Jiang et al. [2023a] Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, and Shuaicheng Liu.  Low-light image enhancement with wavelet-based diffusion models.  _ACM TOG_ , 42(6):1–14, 2023a.
  * Jiang et al. [2023b] Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams.  A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image.  In _CVPR_ , pages 758–767, 2023b.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _arXiv preprint arXiv:2206.00364_ , 2022.
  * Kingma and Ba [2015] P. Diederik Kingma and Lei Jimmy Ba.  Adam: A method for stochastic optimization.  In _ICLR_ , 2015.
  * Kocabas et al. [2020] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  VIBE: Video inference for human body pose and shape estimation.  In _CVPR_ , pages 5253–5263, 2020.
  * Kulon et al. [2020] Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou.  Weakly-supervised mesh-convolutional hand reconstruction in the wild.  In _CVPR_ , pages 4990–5000, 2020.
  * Li et al. [2024] Haipeng Li, Hai Jiang, Ao Luo, Ping Tan, Haoqiang Fan, Bing Zeng, and Shuaicheng Liu.  DMHomo: Learning homography with diffusion models.  _ACM TOG_ , 2024.
  * Lin et al. [2021a] Kevin Lin, Lijuan Wang, and Zicheng Liu.  End-to-end human pose and mesh reconstruction with transformers.  In _CVPR_ , pages 1954–1963, 2021a.
  * Lin et al. [2021b] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In _ICCV_ , pages 12939–12948, 2021b.
  * Liu et al. [2021] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang.  Semi-supervised 3D hand-object poses estimation with interactions in time.  In _CVPR_ , pages 14687–14697, 2021.
  * Liu et al. [2023] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell.  More control for free! image synthesis with semantic diffusion guidance.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 289–299, 2023.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _arXiv preprint arXiv:2206.00927_ , 2022.
  * Matl. [2019] Matthew Matl.  Pyrender.  https://github.com/mmatl/pyrender, 2019.
  * Mescheder et al. [2019] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3D reconstruction in function space.  In _CVPR_ , pages 4460–4470, 2019.
  * Miller and Allen [2004] Andrew T Miller and Peter K Allen.  Graspit! a versatile simulator for robotic grasping.  _IEEE Robotics & Automation Magazine_, 11(4):110–122, 2004.
  * Moon and Lee [2020] Gyeongsik Moon and Kyoung Mu Lee.  I2L-MeshNet: Image-to-lixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image.  In _ECCV_ , pages 752–768, 2020.
  * Moon et al. [2020] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee.  Interhand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image.  In _ECCV_ , pages 548–564, 2020.
  * Park et al. [2022] JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee.  HandOccNet: Occlusion-robust 3D hand mesh estimation network.  In _CVPR_ , pages 1496–1505, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , pages 10684–10695, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  pages 2256–2265, 2015.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _NeurIPS_ , 32, 2019.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Spurr et al. [2020] Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz.  Weakly supervised 3D hand pose estimation via biomechanical constraints.  In _ECCV_ , pages 211–228, 2020.
  * Tse et al. [2022] Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang.  Collaborative learning for hand and object reconstruction with attention-guided graph convolution.  In _CVPR_ , pages 1664–1674, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wan et al. [2023] Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang.  Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning.  _arXiv preprint arXiv:2304.00464_ , 2023.
  * Wang et al. [2023a] Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang.  Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation.  In _ICRA_ , pages 11359–11366. IEEE, 2023a.
  * Wang et al. [2023b] Yinhuai Wang, Jiwen Yu, and Jian Zhang.  Zero-shot image restoration using denoising diffusion null-space model.  _ICLR_ , 2023b.
  * Wen et al. [2023] Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, and Wenping Wang.  Hierarchical temporal transformer for 3D hand pose estimation and action recognition from egocentric RGB videos.  In _CVPR_ , pages 21243–21253, 2023.
  * Xu et al. [2023a] Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu.  H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction.  In _CVPR_ , pages 17048–17058, 2023a.
  * Xu et al. [2023b] Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al.  Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy.  In _CVPR_ , pages 4737–4746, 2023b.
  * Yang et al. [2020a] John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak.  Seqhand: RGB-sequence-based 3D hand pose and shape estimation.  In _ECCV_ , pages 122–139, 2020a.
  * Yang et al. [2020b] Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, and Cewu Lu.  BiHand: Recovering hand mesh with multi-stage bisected hourglass networks.  In _BMVC_ , 2020b.
  * Yang et al. [2021] Linlin Yang, Shicheng Chen, and Angela Yao.  SemiHand: Semi-supervised hand pose estimation with consistency.  In _ICCV_ , pages 11364–11373, 2021.
  * Yang et al. [2022] Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu.  Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis.  In _CVPR_ , pages 2750–2760, 2022.
  * Ye et al. [2023a] Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani.  Diffusion-guided reconstruction of everyday hand-object interaction clips.  In _ICCV_ , pages 19717–19728, 2023a.
  * Ye et al. [2023b] Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu.  Affordance diffusion: Synthesizing hand-object interactions.  In _CVPR_ , pages 22479–22489, 2023b.
  * Zhang et al. [2021a] Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang.  Interacting two-hand 3D pose and shape reconstruction from single color image.  In _ICCV_ , pages 11354–11363, 2021a.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , pages 3836–3847, 2023.
  * Zhang et al. [2019] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng.  End-to-end hand mesh recovery from a monocular RGB image.  In _ICCV_ , pages 2354–2364, 2019.
  * Zhang et al. [2021b] Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, and Ji Liu.  Hand image understanding via deep multi-task learning.  In _ICCV_ , pages 11281–11292, 2021b.
  * Zhao et al. [2021] Zimeng Zhao, Xi Zhao, and Yangang Wang.  TravelNet: Self-supervised physically plausible hand motion learning from monocular color images.  In _ICCV_ , pages 11666–11676, 2021.
  * Zhou et al. [2020] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In _CVPR_ , pages 5346–5355, 2020.
  * Ziani et al. [2022] Andrea Ziani, Zicong Fan, Muhammed Kocabas, Sammy Christen, and Otmar Hilliges.  TempCLR: Reconstructing hands via time-coherent contrastive learning.  In _3DV_ , pages 627–636. IEEE, 2022.
  * Zimmermann et al. [2019] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.  FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images.  In _ICCV_ , pages 813–822, 2019.