Feng Shuang et al.

mode = title]3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario

1]Feng Shuang[style=chinese,

[1]organization=Guangxi Key Laboratory of Intelligent Control and Maintenance of Power Equipment, School of Electrical Engineering, Guangxi University,
addressline=100 Daxuedong Rd.,

1]Wenbo He[style=chinese]

1]Shaodong Li[style=chinese

[1]

[cor1]Corresponding author

Recently, 3D hand reconstruction has gained more attention in human-computer cooperation, especially for hand-object interaction scenario. However, it still remains huge challenge due to severe hand-occlusion caused by interaction, which contain the balance of accuracy and physical plausibility, highly nonlinear mapping of model parameters and occlusion feature enhancement. To overcome these issues, we propose a 3D hand reconstruction network combining the benefits of model-based and model-free approaches to balance accuracy and physical plausibility for hand-object interaction scenario. Firstly, we present a novel MANO pose parameters regression module from 2D joints directly, which avoids the process of highly nonlinear mapping from abstract image feature and no longer depends on accurate 3D joints. Moreover, we further propose a vertex-joint mutual graph-attention model guided by MANO to jointly refine hand meshes and joints, which model the dependencies of vertex-vertex and joint-joint and capture the correlation of vertex-joint for aggregating intra-graph and inter-graph node features respectively. The experimental results demonstrate that our method achieves a competitive performance on recently benchmark datasets HO3DV2 and Dex-YCB, and outperforms all only model-base approaches and model-free approaches.

3D hand reconstructionhand-object interaction scenarioMANO modelmutual graph-attention

§ INTRODUCTION

3D hand reconstruction has been widely used in human-machine interaction [1][2], virtual and augmented reality [5][3][6][4], sign language recognition and translation [7], and has made great progress in recent years. But it is still challenging to robust to occlusion, particularly when the hand is severely occluded under hand-object interaction scenario.

The current approaches of 3D hand reconstruction focus on recovering highly accurate and plausible 3D hand meshes. Those approaches can be generally categorized into model-based and model-free approaches.  Model-based approaches [11][9][8][10] refer to utilize a parametric model with strong prior knowledge of hand and regress parametric representation in terms of shape and pose. Among the approaches in recent years, MANO [12] as statistical parametric model is the most widely used and shows great advantages in reconstructing physical plausible hand model when facing the challenges of object-occlusion, self-occlusion and low resolution. However, the regression of accurate MANO parameters from RGB image is a highly nonlinear mapping process, and will be limited since the MANO parameter model is constructed using limited hand samples. In contrast, Model-free approaches [14][16][13][15] directly regresses to the coordinates of 3D hand mesh vertices and joints, and no longer employ predefined parameterized model. Model-free approaches can recover highly accurate 3D hand meshes and joints because of measuring an average joint or vertex position error directly. However, model-free approaches can not robust to meet physical plausibility because of without prior information of hand shape and pose when facing challenging environment variations.

Therefore, it is necessary to combine the benefits of model-based and model-free approaches. Most recently, Jiang et al. [17] propose a probabilistic model that estimate the prior probability distribution of joints and vertices with incorporating a model-based network as a prior-net. Yu et al. [18] leverage direct mesh fitting for alignment accuracy and guidance from MANO for plausibility. Motivated by the above observations, our ultimate goal is to reconstruct both highly accurate and physical plausible hand model with the benefits of model-based and model-free methods for challenging hand-occlusion. In this work, we propose a 3D hand reconstruction network cascading MANO and Model-free method that consists of initial stage and refinement stage. In the initial stage, MANO with 3D pose and shape priors is used to reconstruct physical plausible hand meshes, but it is challenging to regress accurate MANO parameters with severe hand-occlusion from RGB image. Then in the refinement stage, the model-free method guided by MANO is used to further improve the accuracy of the reconstruction.

In the initial stage, we aim to obtain the high-quality MANO model in hand-object interaction scenario. In the regression of MANO parameters, the previous works [11][19][10] use CNN to encode the extracted hand image features and 2D heatmaps as a latent feature, and then transfer the latent feature to the pose decoder to obtain MANO parameters. However, those methods don’t distinguish the pose and shape parameters and the mapping from the abstract image feature to the MANO parameter space is highly non-linear. Some approaches [21][20][18] employ MLP or Transformer to regress MANO parameters from concrete 3D joints, but those methods depend on the accurate 3D joints estimation in the previous stage. Due to the severe hand-occlusion and inherent depth ambiguity of monocular RGB images, it is not easy to obtain accurate 3D joints. In this work, we propose a novel MANO pose parameters regression module based on SemGCN that can achieve the mapping of MANO pose parameters directly from 2D joints. Previous methods [20][24][23][22] have demonstrated that Graph Convolutional Network(GCN) can finish the encoding of 3D pose space features exploiting the topological structure information of 2D joints. Compared with ordinary GCN, SemGCN [22] can further adaptively model connection strength to capture semantic information among joints,  which strengthen the connection between occluded and unoccluded joints. We employ SemGCN to complete the encoding of 3D MANO pose parameters space through exploiting 2D joints information, which avoids the process of highly nonlinear mapping from abstract image feature and no longer depends on accurate 3D joints.

In the refinement stage, we aim to effectively refine the hand meshes and joints using model-free method guided by MANO from the initial stage. For enhancing the hand mesh and joint feature representation, the previous works [8][26][25] employ GCN to aggregate adjacent mesh vertex and joint feature through modeling neighborhood vertex-vertex and joint-joint interactions respectively. However, those methods ignore the correlation between vertex and joint that weaken feature enhancement. To capture the correlation between vertices and joints, the common methods [9][15][10] use the pre-defined regression matrix from MANO, but the matrix is designed to only map MANO-derived meshes to joints. Recently, the methods [17][15] use 2D spatial attention mechanism to capture the correlation of vertex-joint to improve feature representation. Motivated by the above observation, we propose a vertex-joint mutual graph-attention model to jointly enhancing mesh vertex and joint feature from the initial stage. Specifically, our proposed refinement model is made up of a stack of basic blocks of the similar structure, which consists of a GCN layer and a mutual attention. GCN is employed to model the dependencies of vertex-vertex and joint-joint for aggregating intra-graph node features. And mutual attention  is devised to capture the correlation between vertices and joints for aggregating inter-graph node features.

In summary, our contributions are as follows:

* We propose a 3D hand reconstruction pipeline combining the benefits of model-based and model-free approaches for hand-object interaction scenario, which balance accuracy and physical plausibility to improve the precision of hand meshes and joints estimation.

* We present a novel MANO pose parameters regression module from 2D joints directly, which avoids the process of highly nonlinear mapping from abstract image feature and no longer depends on accurate 3D joints.

* We propose a vertex-joint mutual graph-attention model guided by MANO that can jointly refine hand mesh vertices and joints, which model the dependencies of vertex-vertex and joint-joint and capture the correlation of vertex-joint for aggregating intra-graph and inter-graph node features.

* We demonstrate the effectiveness of our method by conducting comprehensive experiments on two benchmark datasets: Dex-YCB [44] and HO3Dv2 [40] that contain severe hand occlusion. Compared with the state-of-the-art methods, our method achieves a competitive performance. Moreover, we further perform ablation studies to highlight the effectiveness of our proposed model and its major components on the HO3DV2 dataset.

§ RELATED WORK

§.§ Model-based Methods
In recent years, many methods have been proposed to reconstruct the 3D hand by regressing the MANO parameters. Boukhayma et al. [11] firstly regress the MANO parameters in single-hand reconstruction via a deep neural network. Zhang et al. [27] propose an architecture of iterative regression module based on CNN to obtain the MANO parameters from 2D joint heat-maps. For hand-object interaction scenario, some approaches focus on modeling interactions between hands and objects. Liu et al. [19] devise a contextual reasoning module for modeling hand-object interaction. A most recent approach [37] propose a harmonious feature learning network that can extract hand and object harmonious features. Some approaches based on 2D spatial attention has achieved an impressive performance. Park et al. [10] introduce an feature injection mechanism that fully exploits the information at occluded regions to enhance image features. However, it is challenging to regress accurate MANO parameters that limited their reconstruction accuracy. Model-based approaches with pose and shape priors show great advantages for hand reconstruction. However, most previous methods focus on feature extraction and enhancement and ignore the highly nonlinear mapping process of MANO parameters. In this work, our proposed MANO pose parameters regression module from 2D joints directly is able to avoid the process of highly nonlinear mapping from abstract
image feature and no longer depend on accurate 3D joints.

§.§ Model-free Methods
To relax this heavy reliance on the MANO parameter space, some approaches directly fit the mesh vertices. Ge et al. [8] and Choi et al. [14] use Graph-CNN to model neighborhood vertex-vertex interactions for regressing the hand meshes in a coarse-to-fine manner. Kulon et al. [28] apply spiral convolutions for neighbourhood selection. Tang et al. [26] utilize a small Graph-CNN to predict an offset mesh to quickly align the rough mesh with AR image space. Tse et al. [25] use GCN with attention-guided to iteratively refine results from dense hand and object graphs. But GCN can only capture the local interactions between neighboring vertices, Lin et al. [15] use transformer to capture global interactions between the vertices and joints. Wang et al. [29] introduce a dense mutual attention to model fine-grained dependencies between the hand and the object. Different from model methods, models-free methods can improve average accuracy of reconstruction but are usually weak to robust to occlusion and low resolution. In this work, our proposed vertex-joint mutual graph-attention model gudided by MANO are used to effectively refine 3D mesh vertices and joints.

§.§ GCN-based Methods
Graph convolution network (GCN) allow enhancing representations of the relationships between the nodes of graph-based data. GCN have been wildly used in 3D hand pose estimation and mesh reconstruction since hand mesh and joints naturally form a graph. Zhao et al. [22] proposed a SemGCN for capturing complex semantic relationships among human body joints for 3D human pose estimation. Doosti et al. [30] proposed an adaptive Graph-UNet jointly estimating 3D hand joints and object bounding box corners from 2D coordinates. Huand et al. [20] encode initial 2D poses with GCNs for the following 3D reconstruction in Transformer. Most recently, Wang et al. [31] propose an node-offset graph convolutional cascadeing transformer and GCN, and devise a Topology-aware head based on SemGCN.

Overview of our method that consists of three stages. The Backbone extracts the hand feature $I$ and finish the 2D joint locations ($J_{2D}$) from the input single RGB image. At the initial stage, two separate branches are used to regress the MANO pose parameters and shape parameters respectively. Then, the rough 3D hand mesh vertices ($V$) and joints ($J_{3D}$) are obtained by forwarding the MANO parameters to MANO layer. In the refinement stage, we firstly construct mesh vertex and joint graphs according to the MANO model. In the refinement stage, we use a stack of basic block of the similar structure to generate the refinement hand meshes and joints, where a basic block consists of stack of GCN layers followed by the proposed mutual attention.

§ METHOD

In this section, we introduce the overall pipeline of our method as shown in <Ref>. Our model consists of three stages. At the backbone, we firstly extract image feature and obtain hand 2D joint locations from an input RGB image $I\in R^{H\times W\times 3}$ (Section 3.1). At the initial stage, we then regress the MANO model to obtain rough hand mesh vertices and 3D joints (Section 3.2). At the refinement stage,we jointly refine the mesh vertex and joint feature using the proposed vertex-joint mutual graph-attention model (Section 3.3). The final outputs of the refinement stage are the 3D vertex and joint coordinates. We train the proposed model end-to-end and show the details of overall loss function (Section 3.4).

§.§ Backbone

The backbone extracts feature $F$ and localize 2D joints $J_{2D}$ from a hand RGB image $I\in R^{256\times 256\times 3}$. We firstly feed the hand image to ResNet50 [32] -based FPN [33], which produces the feature map $F\in R^{32\times 32\times 256}$. After that, a SET [10] model is adopted to refine the feature $F$ by referencing the distant information  with self-attention mechanism. Then, a stacked Hourglass Network takes enhanced feature as input and output 2D heatmaps $H$ for each joint. At last, we use COM-based decoder [34] to regress 2D joints coordinates $J_{2D}$ from 2D heatmaps $H$.

§.§ Initial Stage

§.§.§ Preliminaries

MANO as a low dimensional parametric hand model is devised to cope with low-resolution, occlusion, and noise, which is learned from around 1000 high-resolution 3D scans of hands of 31 subjects. The final 778 vertices and 16 joints of one hand are obtained by forwarding the pose parameters $\theta \in R^{48}$ and shape parameters $\beta \in R^{10}$ to MANO layer. And $\theta$ is joint rotations representation in axis-angle and $\beta$ is coefficients of a shape PCA. The MANO hand model is defined as follows:

\begin{equation} \left\{\begin{aligned}
&M(\beta ,\theta )=W(T_P(\beta ,\theta ),J(\beta),\theta,\omega ) \\
&T_P(\beta ,\theta)=\overline{T} +B_s(\beta)+B_p(\theta)
\end{aligned}\right.
\end{equation}

where $W$ is a linear blend skinning (LBS) function, $T_P$ corresponds to the articulated mesh template to blend, consisting of $K$ joints, $J$ represents the joint locations learned from the mesh vertices via a sparse linear regressor, and $\omega$ indicates the blend weights. $\overline{T}$ is a mean hand template, $B_s$and $B_p$ are shape blendshapes and pose blendshapes respectively.

§.§.§ MANO parameters Regression

It is challenging to directly regress MANO parameters from image feature, since the mapping from an image to the MANO parameters space is highly non-linear. In this work, we propose a novel MANO pose parameters regression module basded on SemGCN achieving the mapping of the MANO pose parameters from 2D joints directly. 2D joints can be considered as a kind of graph structure data, in which each joint can be regarded as a node. Node features contain rich location information and the neighboring nodes also provide useful features to estimate the relative offsets which can play a critical role for invisible joints. The topological hand joints structure plays an essential role in predicting accurate hand pose especially in severe occlusion. GCN naturally introduce the prior of hand kinematic topology and aggregates information about the nodes and their corresponding neighbor nodes under the guidance of topology. Compared with ordinary GCN, SemGConv [22] adds a learned weighting matrix $M$ to adaptively model connection strength between joints, which is written as:

\begin{equation}
X^{(l+1)}=\sigma (\textbf{W}X^{(l)}\rho _i(\textbf{M}\otimes (\textbf{A}+\textbf{I}))
\end{equation}

where $\sigma$ is the activation function, $\textbf{W}$ is a transformation matrix, $\rho$ is the Softmax non-linearity which normalizes the weight of connections between a node and the neighboring nodes, $\otimes$ denotes element wise multiplication, $\textbf{A}$ is the adjacency matrix and $\textbf{I}$ is the identity matrix. We use SemGCN to encode the latent pose parameters space $\Theta \in R^{16\times 128}$ from 2D joints, which consists of ResGCN based on SemGconv. In hand-object interaction Scenario, SemGCN can capture semantic information about whether adjacent joints are occluded through utilizing the learned weighting matrix $\textbf{M}$, which strengthen connection between adjacently occluded and unoccluded joints. After that, we build a decoder to regress the pose parameters from the latent pose space, which consists of one SemGconv layer and MLP. The illustration of the module is shown in <Ref>.

For the shape parameters $\beta$ of MANO, we continue the method of previous works [10][19]. The network with four residual blocks takes a concatenation of the enhanced hand feature $F$ and the 2D heatmaps $H$ as input and output 2048 dimensional vector, which is then passed to fully-connected layers to predict  $\beta$. Finally, the MANO model are obtained by forwarding the MANO parameters to MANO layer.

The SemGCN-based MANO pose parameters Regression module.

§.§ Refinement Stage

Given the rough but physical plausible 3D hand mesh vertices $V\in R^{778\times 3}$ and joints $J_{3D} \in R^{21\times3}$ from the initial stage, we propose a vertex-joint mutual graph-attention model that can jointly refine hand mesh vertices and joints. To this end, we regard hand mesh and joints as two graphs and use GCN to capture the intra-graph dependencies. To further model the inter-graph interaction, we use mutual attention that allows feature aggregation between two graphs. The illustration of the model is shown in <Ref>.

§.§.§ Graph Construction and GCN Layer

The hand meshes and joints are modeled by separate graphs with vertices and joints as nodes, and their connections are defined in the MANO model mesh structure and joint topological structure. To construct the 3D vertex and 3D joint graph node features, we firstly extract global feature $F_{global}$ from the backbone FPN. Inspired of previous work[17], we attach the initial MANO 3D coordinates of joints $J_{3D}$ and mesh vertices $V$ to the image-level global feature vector $F_{global}$ to obtain the new joint feature matrix $F_j\in R^{2051\times 21}$ and new vertex feature matrix$F_v\in R^{2051\times 778}$. After initializing the joint and vertex features, we then update the joint features $\hat{F} _j$ and vertex features  $\hat{F} _v$ via Gconv and SemGconv respectively. Intuitively, the graph convolutional layers can effectively model intra-graph dependencies with exploit neighboring correlation from the topology of the meshes and joints.

§.§.§ Mutual Attention Layer

Following one or several graph convolutional layers, we model vertex-joint interaction in the mutual attention layer. For each node from one graph, our mutual attention layer aims to aggregate features from the other graph via the attention mechanism. Specifically, for every node feature in the mesh graph, we first use three 1D convolutional layers to extract the query, key, and value, and collect all queries, keys, and values as $Q_v \in R^{778\times C}$, $K_v \in R^{778\times C}$ and  $V_v \in R^{778\times C}$ respectively, where each row of them is the query, key or value of a particular node. Similarly, we have the query, key and value for the joint graph as $Q_j \in R^{21\times C}$, $K_j \in R^{21\times C}$ and  $V_j \in R^{21\times C}$ respectively. We then compute the vertex-to-joint attention between the queries from the joint graph and the keys from the vertex graph following  as:

\begin{equation}
A^{v\rightarrow j}= softmax(\frac{Q_j K_v^T}{\sqrt{C} } )
\end{equation}

where $A^{v\rightarrow j}\in R^{21 \times 778}$ is the vertex-to-joint attention map, with the $i$-th row denoting the expected contribution proportion of all vertices nodes to the $i$-th joint node. We can then aggregate vertex node features weighted by the vertex-to-joint attention as:

\begin{equation}
F^{v\rightarrow j}= A^{v\rightarrow j} V_v
\end{equation}

where $F^{v\rightarrow j}\in R^{21\times C}$ is the aggregated features from the vertex graph. Similarly, we can compute the joint-to-vertex attention as:

\begin{equation}
A^{j\rightarrow v}= softmax(\frac{Q_v K_j^T}{\sqrt{C} } )
\end{equation}

where $A^{j\rightarrow v}\in R^{778 \times 21}$. And we can compute the joint-to-vertex aggregated feature as:

\begin{equation}
F^{j\rightarrow v}= A^{j\rightarrow v} V_j
\end{equation}

where $F^{j\rightarrow v}\in R^{778\times C}$. We finally fuse the aggregate feature with the original feature in each node as:

\begin{equation} \left\{\begin{aligned}
&\widetilde{F}_j=f_j(\hat{F_j}\oplus F^{v \rightarrow j}) \\
&\widetilde{F}_v=f_v(\hat{F_v}\oplus F^{j \rightarrow v})
\end{aligned}\right.
\end{equation}

where $\widetilde{F}_j$ and $\widetilde{F}_v$ are the refined joint and vertex feature, $f_j$ and $f_v$ are independent fusion units consisting of MLP. Finally, $\widetilde{F}_j$ and $\widetilde{F}_v$ are passed to MLP to predict the final 3D hand mesh vertices $\widetilde{V}\in R^{778\times 3}$ and joints $\widetilde{J}_{3D}\in R^{21\times 3}$ respectively.

The MANO pose parameters Regression module based on SemGCN.

§.§ Overall Loss Function

To effectively train the proposed model, the multi-task loss function employed here is defined as:

\begin{equation}
L=\lambda _1L_{MANO}+\lambda _2L_{r}+\lambda _3L_{e}+\lambda _4L_{n}
\end{equation}

where $L_{MANO}$ denote the MANO loss to supervise backone and initial stage, $L_{r}$ denote the refinement loss in the refinement stage, $L_{e}$ and $L_{n}$ denote the edge loss and normal loss that are used to penalize outlier vertices and irregular surfaces. Finally, $\lambda _1$, $\lambda _2$, $\lambda _3$ and $\lambda _4$ are coefficients that balance the weight of each loss function.

§.§.§ MANO Loss

We adopt an L2 loss to supervise the regression of 2D joint location and the MANO model, and the $L_{MANO}$ is defined as:

\begin{equation}
\begin{split}
L_{MANO}=&\left \| V-V^{gt} \right \|_2 +\left \| J_{3D}-J_{3D}^{gt} \right \|_2
+\left \| \theta -\theta ^{gt} \right \|_2  +\\&\left \| \beta-\beta ^{gt} \right \|_2+\left \| H-H ^{gt} \right \|_2+\left \| J_{2D}-J_{2D} ^{gt} \right \|_2
\end{split}
\end{equation}

where $V^{gt}$, $J_{3D}^{gt}$, $\theta ^{gt}$, $\beta ^{gt}$, $H ^{gt}$ and $J_{2D} ^{gt}$ represent the ground truth of mesh vertices, 3D joints, MANO pose parameters, shape parametesr,2D heatmaps and 2D joints respectively.

§.§.§ Refinement Loss

The final outputs in the refinement stage are the coordinates of 3D joint $\widetilde{J}_{3D}$ and the hand mesh vertex $\widetilde{V}$ . We also adopt an L2 loss to supervise the refinement stage and the $L_r$ is defined as:

\begin{equation}
L_r=\left \|\widetilde{V}-V^{gt}\right \|_2 + \left \|\widetilde{J}_{3D}-J_{3D}^{gt}\right \|_2
\end{equation}

§.§.§ Edge Loss and Normal Loss

We employ the edge loss $L_e$ and the normal loss $L_n$ to penalize flying vertices and irregular surfaces following [26], which is defined as :

\begin{equation}
L_e=\left \| \left | \widetilde{e}_i\right |-\left | e^{gt}_i\right |  \right \|_1
\end{equation}

\begin{equation}
L_n=\left \|\left \langle \widetilde{e}_i,n^{gt}_i  \right \rangle  \right \|_1
\end{equation}

where $\widetilde{e}_i$ denote the $i$-th mesh edge vector of the refined hand meshes, $e^{gt}_i$ and $n^{gt}_i$ are the ground truth of the edge vector and the normal of the corresponding edge respectively.

§ EXPERIMENTS

In this section, we firstly introduce the benchmark datasets for 3D hand reconstruction to train our proposed model and define the evaluation metrics. Then we present the implementation details and the quantitative and  qualitative results of comparing with state-of-the-art methods. Finally, we present the results of ablation studies on each component to demonstrate the effectiveness of our proposed modules.

§.§ Datasets

The following recently hand reconstruction benchmark datasets are exploited to train and evaluate our proposed model: HO3DV2 and Dex-YCB, which contain significant occlusions in hand-object interaction scenario. The HO3DV2 is a challenging dataset of hand-object interaction,which consists of 77K images from 68 video sequences and are split into 66,034 images for training and 11,524 images for testing. And we submit the test results to the official website to report performance. Dex-YCB is a latest large scale dataset that includes 582K images from over 1000 video sequences that covers 10 subjects and 20 objects. And we present the results according to the official s0 splitting protocol. Specifically, Dex-YCB is a recently large scale dataset and our model is pre-trained on Dex-YCB for a few epochs to alleviate the overfitting on HO3DV2 following [35].

§.§ Evaluation Metrics

In order to evaluate our proposed model and consistently compare the results with state-of-the-art methods, we adopt the evaluation metrics on each benchmark dataset that are commonly used for 3D hand reconstruction and majorly reported by related works[10][17]. These are the mean per joint/vertex position error(J-PE/V-PE), mean per joint/vertex position error after procrustes alignment(PA-J-PE/PA-V-PE) and vertices at distance thresholds of 5mm and 15mm after procrustes alignment (PA-F@5 and PA-F@15), where the position error is the mean 3D Euclidean distance between the predicted coordinates and the groundtruth coordinates, and procrustes alignment is employed to ignor the global rotation and scale differences.

Hand reconstruction results on the HO3Dv2 dataset compared with SOTA methods. -: the results are unavailable from previous papers. We mark the best results in bold for better comparison.

Method                Category         J-PE     PA-J-PE     V-PE    PA-V-PE    PA-F@5     PA-F@15

ArtiBoost [38]         Model-based       -        11.4      -         10.9      0.488     0.944

Hasson et al. [39]     Model-based       -        11.0      -         11.2      0.464      0.939

Hampali et al. [40]    Model-based       -        10.7      -         10.6      0.506     0.942

I2UV-HandNet [41]      Model-based       -        9.9      -         10.1       0.500      0.943

Liu et al. [19]        Model-based       30.0       9.9      28.9         9.5      0.528     0.956

HandOccNet [10]         Model-based       24.9     9.1      24.2         8.8      0.564      0.963

HFL-Net [37]            Model-based       -        8.9      -         8.7          0.575      0.965

Pose2Mesh [14]           Model-free       -        12.5      -         12.7      0.441     0.909

I2L-MeshNet [23]      Model-free        26.8      11.2      -         13.9       0.409      0.932

Keypoint Trans [16]      Model-free     -        10.8         -         -          -           -

METRO [15]              Model-free        -      10.4      -         11.1       0.484      0.946

H2ONet [35]              Model-free        23.0      9.0      22.4         9.0       0.554      0.960

Jiang et al. [17]              combined        19.2      8.3      18.3         8.2       0.608      0.965

Ours                   combined         19.7      8.8      19.4         8.7       0.555      0.970

Results on the Dex-YCB dataset compared with SOTA methods. -: the results are unavailable from previous papers. We mark the best results in bold for better comparison.

Method                Category                                 V-PE    PA-V-PE    J-PE     PA-J-PE

Spurr et al. [42]         Model-based      -         -     17.3     6.8

Liu et al. [19]            Model-based          -         -       15.3        6.6

HandOccNet [10]        Model-based      13.1         5.5      14.0        5.8

HFL-Net [37]             Model-based       -         -      12.7       5.5

METRO [15]                Model-free              -         -        15.2       6.7

MobRecon [43]             Model-free          13.1        5.6     14.2        6.4

H2ONet [35]              Model-free             13.0        5.5      14.0        5.7

Ours                         combined                   12.4        5.4      13.2        5.6

§.§ Implementation Details

We implement our method in PyTorch. All experiments are run on NVIDIA RTX 2080Ti GPU and Inter xeon gold 6226 CPU.We train the network using the Adam [36] optimizer and set the batch size as 24. The initial learning rate is set as 1e-4. For Dex-YCB, we train the model in 50 epochs and the learning rate decay by 0.9 after every 10 epochs. For HO3DV2, we firstly pre-train the model on Dex-YCB for 20 epochs. Then we train the model in 100 epochs and the learning rate decay by 0.7 after every 10 epochs. To makes hands in the center of the input image, we use the hand ground truth 2D joints to crop each sample image and resize it to 256×256. And common data augmentation strategies are adopted including random scaling([0.9,1.1], rotation([-180,180]), translation([-10,10]), and color jittering.

§.§ Comparison with State-of-the-art Methods

To evaluate the hand reconstruction quality from a single RGB image, we compare our method with the state-of-the-art methods quantitatively on the HO3DV2 and Dex-YCB. As the ground truths of HO3DV2 test set are not publicly available, we obtain the test results from the official evaluation server. Besides, we follow [35] to pre-train the model on Dex-YCB for a few epochs to alleviate the overfitting on HO3DV2. We report the performance before and after the procrustes alignment(PA) to better show our method effectiveness. The experimental results on HO3DV2 and Dex-YCB are shown in Table 1 and Table 2 respectively.

As shown in Table1, comparing with all state-of-art only model-based and model-free methods, our proposed network cascading MANO and Model-free method achieve the best performance on majority evaluation metrics on HO3DV2, and only second to the most latest Jiang et al. [17] that propose a probabilistic model that estimate the prior probability distribution of joints and vertices with incorporating a model-based network as a prior-net to. Especially on mean per joint/vertex position error without procrustes alignment metrics(J-PE/V-PE), we have made considerable progress compared with most previous methods, and we are slightly better than Jiang et al. [17] on the PA-F@15. As shown in Table2, our method also achieve competitive performance on Dex-YCB and only second to the most latest HFL-Net [37]. We attribute the advantage of our proposed network to combine the benefits of model-based and model-free approaches, which MANO is used to reconstruct coarse but physical plausible hand meshes and pose in the initial stage and model-free method guided by MANO is used to further improve the accuracy of the reconstruction in the refinement stage.

Qualitative results of our method on the HO3D and Dex-YCB datasets.

Analysis of the MANO pose parameters regression module.

Exp                Set         PA-J-PE     PA-V-PE

1)                B       10.4        10.0

2)                B+OG       9.7        9.7

3)                B+SG       9.5        9.4

4)               B+SG+G       9.3        9.4

5)               B+SG+G+MA       9.0        9.1

6)                B+SG+G+MA+P       8.8        8.7

§.§ Ablation Studies

We perform ablation studies on HO3DV2 to study the effectiveness of our proposed model and its major components. As <Ref> shows, we denote Baseline (B) as our model after removing the following major components: SemGCN(SG) in MANO pose parameters regression module,GCN layer(G) and mutual attention layer(MA). Besides, OG denotes the ordinary GCN is used in MANO pose parameters regression module and P denotes that the model is pre-trained on Dex-YCB for a few epochs. Besides, we also provide 3D joints and mesh vertices PCK after procrustes aligned of our method on HO3DV2 dataset in <Ref>.

3D joints and mesh vertices PCK after procrustes aligned of our method on HO3DV2 dataset.

§.§.§ Effectiveness of MANO Pose Parameters regression Module

We follow the previous methods [19][10] to regression MANO pose and shape parameters on Baseline(B), which use CNN to encode the MANO parameters space from the concatenation of hand feature and 2D heatmaps. Comparing the first three rows in Table 3, we can see that GCN can more effectively encode MANO pose parameters space from 2D joint with alleviating the process of highly nonlinear mapping than Baseline and SemGCN achieve the best performance. SemGCN adds a learned weighting matrix M to adaptively model connection strength between adjacent joints, that is more suitable for our regression task comparing with ordinary GCN.

Qualitative comparison of with state-of-art method on HO3DV2 dataset.

§.§.§ Effectiveness of Vertex-Joint Mutual Graph-Attention Module

Our proposed vertex-joint mutual graph-attention module contists two main components: GCN layer (G) and mutual attention (MA). Comparing Rows 3), 4) and 5) in <Ref>, we can see that only GCN layer with modeling the dependencies of vertex-vertex and joint-joint can improve the precision of joints from the initial stage. When the mutual attention  (MA) is included, we observe the best performing result that demonstrate our proposed mutual graph-attention layer can effectively further aggregate inter-graph node features through capture the correlation between the vertices and joints. Besides, we follow [35] to pre-train the model on Dex-YCB for a few epochs to alleviate the overfitting on HO3DV2 and the comparison result of Rows 5) and 6) show its necessity.

§.§ Qualitative Results

To further evaluate the hand mesh reconstruction quality, we visualize 3D hand meshes on camera view and another view, and we show the qualitative results on HO3DV2 and Dex-YCB in <Ref> and <Ref> respectively. <Ref> show the qualitative comparison results of the initial stage and final output. And the results of initial stage show that our proposed MANO regression module can provide the initial MANO model that robust to object-occlusion for hand-object interaction scenario.Then the results of final output show that our proposed vertex-joint mutual graph-attention model guided by MANO can effectively refine hand mesh vertices and joints jointly. Moreover, <Ref> show the qualitative comparison results between our method and state-of-art HandOccNet on HO3DV2. It can be seen from the figure that our method makes more accurate hand meshes estimation than HandOccNet on camera view.

§ CONCLUSION

In this paper, we propose a 3D hand reconstruction pipeline combining the benefits of model-based and model-free approaches for hand-object interaction scenario. Our proposed MANO pose parameters regression module based on SemGCN can effectively encode MANO pose parameters space from 2D joint directly and achieve better performance than the common methods using CNN to regress MANO pose parameters from image feature. Moreover, we further propose a vertex-joint mutual graph-attention model guided by MANO that can jointly refine hand mesh vertices and joints, which model the dependencies of vertex-vertex and joint-joint and capture the correlation between the vertex-joint. Compared our method with the state-of-the-art methods on recently benchmark HO3DV2 and Dex-YCB datasets, our method achieves competitive performance. On HO3Dv2, our method outperforms all only model-base approaches and model-free approaches and achieves second performance on majorly metrics, especially we have made considerable progress on J-PE/V-PE. On Dex-YCB, our method also achieve best performance on V-PE/PA-V-PE and second best performance on J-PE/PA-J-PE. Moreover, ablation studies on HO3DV2 demonstrate the effectiveness of our proposed model and its major components. Besides, the qualitative results on camera view and another view further show the hand reconstruction quality of our method in challenging environments.

§ ACKNOWLEDGMENTS

This work was supported in part by the Natural Science Foundation of Guangxi (Grant No. 2022JJB170009), and in part by the funding of basic ability promotion project for young and middle-aged teachers in Guangxi's colleges and universities (Grant No. 2022KY0008). Feng also acknowledges support by the Bagui Scholars Program of Guangxi Zhuang Autonomous Region.

[1]
N. Conci, P. Ceresato, and F. G. De Natale, “Natural human-machine interface using an interactive virtual blackboard,” in 2007 IEEE International Conference on Image Processing, vol. 5.1em plus 0.5em minus 0.4emIEEE, 2007, pp. V–181.

[2]
R. Yin, D. Wang, S. Zhao, Z. Lou, and G. Shen, “Wearable sensors-enabled human–machine interaction systems: from design to application,” Advanced Functional Materials, vol. 31, no. 11, p. 2008936, 2021.

[3]
S. Jung and C. E. Hughes, “Body ownership in virtual reality,” in 2016 international conference on collaboration technologies and systems (CTS).1em plus 0.5em minus 0.4emIEEE, 2016, pp. 597–600.

[4]
J. Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. A. Otaduy, D. Casas, and C. Theobalt, “Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video,” ACM Transactions on Graphics (ToG), vol. 39, no. 6, pp. 1–16, 2020.

[5]
S. Han, B. Liu, R. Cabezas, C. D. Twigg, P. Zhang, J. Petkau, T.-H. Yu, C.-J. Tai, M. Akbay, Z. Wang et al., “Megatrack: monochrome egocentric articulated hand-tracking for virtual reality,” ACM Transactions on Graphics (ToG), vol. 39, no. 4, pp. 87–1, 2020.

[6]
F. Mueller, M. Davis, F. Bernard, O. Sotnychenko, M. Verschoor, M. A. Otaduy, D. Casas, and C. Theobalt, “Real-time pose and shape reconstruction of two interacting hands with a single depth camera,” ACM Transactions on Graphics (ToG), vol. 38, no. 4, pp. 1–13, 2019.

[7]
X. Liang, A. Angelopoulou, E. Kapetanios, B. Woll, R. Al Batat, and T. Woolfe, “A multi-modal machine learning approach and toolkit to automate recognition of early stages of dementia among british sign language users,” in Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 278–293.

[8]
L. Ge, Z. Ren, Y. Li, Z. Xue, Y. Wang, J. Cai, and J. Yuan, “3d hand shape and pose estimation from a single rgb image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 833–10 842.

[9]
Y. Chen, Z. Tu, D. Kang, L. Bao, Y. Zhang, X. Zhe, R. Chen, and J. Yuan, “Model-based 3d hand reconstruction via self-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 451–10 460.

[10]
J. Park, Y. Oh, G. Moon, H. Choi, and K. M. Lee, “Handoccnet: Occlusion-robust 3d hand mesh estimation network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1496–1505.

[11]
A. Boukhayma, R. d. Bem, and P. H. Torr, “3d hand shape and pose from images in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 843–10 852.

[12]
J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” arXiv preprint arXiv:2201.02610, 2022.

[13]
N. Kolotouros, G. Pavlakos, and K. Daniilidis, “Convolutional mesh regression for single-image human shape reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4501–4510.

[14]
H. Choi, G. Moon, and K. M. Lee, “Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 769–787.

[15]
K. Lin, L. Wang, and Z. Liu, “End-to-end human pose and mesh reconstruction with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 1954–1963.

[16]
S. Hampali, S. D. Sarkar, M. Rad, and V. Lepetit, “Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 090–11 100.

[17]
Z. Jiang, H. Rahmani, S. Black, and B. M. Williams, “A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 758–767.

[18]
Z. Yu, C. Li, L. Yang, X. Zheng, M. B. Mi, G. H. Lee, and A. Yao, “Overcoming the trade-off between accuracy and plausibility in 3d hand shape reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 544–553.

[19]
S. Liu, H. Jiang, J. Xu, S. Liu, and X. Wang, “Semi-supervised 3d hand-object poses estimation with interactions in time,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 687–14 697.

[20]
L. Huang, J. Tan, J. Meng, J. Liu, and J. Yuan, “Hot-net: Non-autoregressive transformer for 3d hand-object pose estimation,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3136–3145.

[21]
C. Gao, Y. Yang, and W. Li, “3d interacting hand pose and shape estimation from a single rgb image,” Neurocomputing, vol. 474, pp. 25–36, 2022.

[22]
L. Zhao, X. Peng, Y. Tian, M. Kapadia, and D. N. Metaxas, “Semantic graph convolutional networks for 3d human pose regression,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3425–3435.

[23]
G. Moon and K. M. Lee, “I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 752–768.

[24]
V.-T. Le, T.-H. Tran, V.-N. Hoang, V.-H. Le, T.-L. Le, and H. Vu, “Sst-gcn: Structure aware spatial-temporal gcn for 3d hand pose estimation,” in 2021 13th International Conference on Knowledge and Systems Engineering (KSE).1em plus 0.5em minus 0.4emIEEE, 2021, pp. 1–6.

[25]
T. H. E. Tse, K. I. Kim, A. Leonardis, and H. J. Chang, “Collaborative learning for hand and object reconstruction with attention-guided graph convolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1664–1674.

[26]
X. Tang, T. Wang, and C.-W. Fu, “Towards accurate alignment in real-time 3d hand-mesh reconstruction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 11 698–11 707.

[27]
X. Zhang, Q. Li, H. Mo, W. Zhang, and W. Zheng, “End-to-end hand mesh recovery from a monocular rgb image,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 2354–2364.

[28]
D. Kulon, R. A. Guler, I. Kokkinos, M. M. Bronstein, and S. Zafeiriou, “Weakly-supervised mesh-convolutional hand reconstruction in the wild,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 4990–5000.

[29]
R. Wang, W. Mao, and H. Li, “Interacting hand-object pose estimation via dense mutual attention,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 5735–5745.

[30]
B. Doosti, S. Naha, M. Mirbagheri, and D. J. Crandall, “Hope-net: A graph-based model for hand-object pose estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6608–6617.

[31]
Y. Wang, L. Chen, J. Li, and X. Zhang, “Handgcnformer: A novel topology-aware transformer network for 3d hand pose estimation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 5675–5684.

[32]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

[33]
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.

[34]
X. Zhang and F. Zhang, “Differentiable spatial regression: A novel method for 3d hand pose estimation,” IEEE Transactions on Multimedia, vol. 24, pp. 166–176, 2020.

[35]
H. Xu, T. Wang, X. Tang, and C.-W. Fu, “H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 048–17 058.

[36]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv e-prints, 2014.

[37]
Z. Lin, C. Ding, H. Yao, Z. Kuang, and S. Huang, “Harmonious feature learning for interactive hand-object pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 12 989–12 998.

[38]
L. Yang, K. Li, X. Zhan, J. Lv, W. Xu, J. Li, and C. Lu, “Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2750–2760.

[39]
Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid, “Learning joint reconstruction of hands and manipulated objects,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11 807–11 816.

[40]
S. H. Shivakumar, M. Rad, M. Oberweger, and V. Lepetit, “Honnotate: A method for 3d annotation of hand and object poses,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition: CVPR 2020, 2020, pp. 3193–3203.

[41]
P. Chen, Y. Chen, D. Yang, F. Wu, Q. Li, Q. Xia, and Y. Tan, “I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 929–12 938.

[42]
A. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, and J. Kautz, “Weakly supervised 3d hand pose estimation via biomechanical constraints,” in European conference on computer vision.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 211–228.

[43]
X. Chen, Y. Liu, Y. Dong, X. Zhang, C. Ma, Y. Xiong, Y. Zhang, and X. Guo, “Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 544–20 554.

[44]
Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield et al., “Dexycb: A benchmark for capturing hand grasping of objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9044–9053.

[45]
L. Huang, C.-C. Lin, K. Lin, L. Liang, L. Wang, J. Yuan, and Z. Liu, “Neural voting field for camera-space 3d hand pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8969–8978.