# RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose
Estimation

Lijun Li Corresponding author<EMAIL_ADDRESS>Alibaba Group  Shanghai
Artificial Intelligence Laboratory  Linrui Tian  Alibaba Group  Xindi Zhang
Alibaba Group  Qi Wang  Alibaba Group  Bang Zhang  Alibaba Group  Mengyuan Liu
Key Laboratory of Machine Perception, Shenzhen Graduate School, Peking
University  Chen Chen  Center for Research in Computer Vision, University of
Central Florida

###### Abstract

The current interacting hand (IH) datasets are relatively simplistic in terms
of background and texture, with hand joints being annotated by a machine
annotator, which may result in inaccuracies, and the diversity of pose
distribution is limited. However, the variability of background, pose
distribution, and texture can greatly influence the generalization ability.
Therefore, we present a large-scale synthetic dataset –RenderIH– for
interacting hands with accurate and diverse pose annotations. The dataset
contains 1M photo-realistic images with varied backgrounds, perspectives, and
hand textures. To generate natural and diverse interacting poses, we propose a
new pose optimization algorithm. Additionally, for better pose estimation
accuracy, we introduce a transformer-based pose estimation network, TransHand,
to leverage the correlation between interacting hands and verify the
effectiveness of RenderIH in improving results. Our dataset is model-agnostic
and can improve more accuracy of any hand pose estimation method in comparison
to other real or synthetic datasets. Experiments have shown that pretraining
on our synthetic data can significantly decrease the error from 6.76mm to
5.79mm, and our Transhand surpasses contemporary methods. Our dataset and code
are available at  https://github.com/adwardlee/RenderIH.

##  1 Introduction

Figure 1: Randomly selected samples from RenderIH dataset. The rendered hands
are realistic and varied, capturing a variety of poses, textures, backgrounds,
and illuminations.

3D interacting hand (IH) pose estimation from a single RGB image is a key task
for human action understanding and has many applications, such as human-
computer interaction, augmented and virtual reality, and sign language
recognition. However, obtaining 3D interacting hand pose annotations from real
images is very challenging and time-consuming due to the severe self-occlusion
problem. Some previous works [12, 18] have collected some real hand
interaction data using a sophisticated multi-view camera system and made
manual annotations, but the amount of data is limited. Synthetic 3D annotation
data has become increasingly popular among researchers because of its easy
acquisition and accurate annotation [27, 22, 3, 7, 15, 24, 41]. However, there
remain two main challenges: the validity of the generated 3D hand poses and
the diversity and realism of the generated images. Therefore, in this paper,
we present a high-fidelity synthetic dataset of 3D hand interaction poses for
precise monocular hand pose estimation.

Dataset | Type | Data size | MT | AP | background | illumination | Hand type | IH Size
---|---|---|---|---|---|---|---|---
NYU [31] | real | 243K | - | ✗ | lab | uniform | SH | -
STB [40] | real | 36K | - | ✗ | lab | uniform | SH | -
H2O-3D [12] | real | 76K | - | ✗ | lab | uniform | HO | -
H2O [38] | real | 571K | - | ✗ | indoor scenes | uniform | HO | -
MVHM [3] | synthetic | 320K | ✗ | ✗ | static scenes | uniform | SH | -
ObMan [15] | synthetic | 147K | ✓ | ✗ | static scenes | uniform | HO | -
DARTset [7] | synthetic | 800K | ✓ | ✗ | static scenes | manual | SH | -
IH2.6M [26] | real | 2.6M | - | ✗ | lab | uniform | IH | 628K
Ego3d [22] | synthetic | 50K | ✗ | ✗ | static scenes | random | IH | 40K
RenderIH (Ours) | synthetic | 1M | ✓ | ✓ | HDR scenes | dynamic | IH | 1M
Table 1: Comparison of the related hand datasets. “MT” is short for multi-
textures and means whether the hand models in the dataset are assigned with
diverse textures, AP is short for anti-penetration, “Hand type” means which
interaction type the dataset focus on (SH-single hand, HO-hand to object, IH-
hand to hand), and “IH Size” means the proportion of IH poses. “HDR” is short
for High Dynamic Range. Static scenes refer to the use of randomly selected
images as the background.

Firstly, ensuring the validity of the generated 3D interacting hand poses is a
crucial challenge for a synthetic hand system. For example, the pose of Ego3d
[22] is randomized which means a significant portion of the data is not valid.
To ensure effective hand interactions, the generated two-hand poses must be
proximal to each other, while increasing the risk of hand interpenetration.
Therefore, we design an optimization process that considers the constraints of
hand attraction and anti-penetration in the meantime, to ensure the proximity
of two interacting hands and prevent the occurrence of hand penetration
(Section 3.1). In addition, the plausibility of hand poses must also be
considered. Hence, we introduce anatomic pose constraints and apply
adversarial learning to ensure that the generated hand poses adhere to
anatomical constraints and realism. Benefiting from pose optimization, our
generated dataset contains a rich set of validated two-hand interaction poses
as shown in Figure 1.

Secondly, most existing 3D synthetic hand images lack diversity in terms of
backgrounds, lighting, and texture conditions, which prevents them from
capturing the complex distribution of real hand data [22, 3, 15]. Most
existing datasets for hand gesture recognition, such as Ego3d [22], Obman
[15], and MVHM [3], do not consider the quality and diversity of the images.
For instance, Ego3d [22] uses the same texture as the MANO model [29], which
is unrealistic and monotonous. In contrast, our rendering system introduces
various textures, backgrounds, and lighting effects that can produce vivid and
realistic synthetic hand images (see Section 3.2). By combining HDR
background, dynamic lighting, and ray-tracing renderer, we obtain 1M high-
quality gesture images (see Figure 1).

To assess the performance of our proposed dataset, we carried out
comprehensive experiments on it. We demonstrate how much we can reduce the
dependency on real data by using our synthetic dataset. Then we contrast our
proposed RenderIH with other 3D hand datasets, such as H2O-3D [12] and Ego3d
[22], by training a probing model for each of them and testing on a third-
party dataset. Finally, we train a transformer-based network on a mixed
dataset of RenderIH and InterHand2.6M (IH2.6M) and achieve state-of-the-art
(SOTA) results on 3D interacting hand pose estimation. Our main contributions
are as follows:

  * •

We propose an optimization method to generate valid and natural hand-
interacting poses that are tightly coupled and avoid interpenetration. For
image generation, we design a high-quality image synthesis system that
combines rich textures, backgrounds, and lighting, which ensures the diversity
and realism of the generated images.

  * •

Based on our data generation system, we construct a large-scale high-fidelity
synthetic interacting hand dataset called RenderIH, which contains 1 million
synthetic images and 100K interacting hand poses. To the best of our
knowledge, this is the largest and most high-quality synthetic interacting
dataset so far.

  * •

We conduct extensive experiments to verify the effectiveness of our proposed
dataset-RenderIH. The results show that with the help of our synthetic
dataset, using only 10% of real data can achieve comparable accuracy as the
models trained on real hand data. We also propose a transformer-based network
that leverages our dataset and achieves SOTA results.

##  2 Related work

###  2.1 Realistic hand dataset

Establishing a realistic hand dataset is a tedious and challenging procedure,
most realistic data are collected by different sensors [26, 13, 11, 42, 40,
30, 20] including multiple cameras and depth sensors. STB dataset [40]
obtained 3D annotations of a single hand (SH) via 2D manual labels and depth
data. Since manual annotations are time-consuming [26], some researchers [30,
26, 13, 42] utilized semi-automatic methods to make annotations. Moon et al.
[26] captured hand interactions with hundreds of cameras. They manually
annotated the 2D keypoints of both hands on a few images and utilized a
machine detector to help annotate the rest data. While some researchers [11,
1, 31] proposed automatic methods to make annotations, Hampali et al. [11]
collected hand-object (HO) interactions and jointly optimized 2D key points on
multiple RGB-D images to estimate 3D hand poses. Some researchers [8, 38, 9]
obtain the 3D annotations of hands via some special equipment. Ye et al. [38]
captured hand poses via multiple joint trackers. Due to the limitation of the
data collection scene, most realistic datasets are in simple scenarios, e.g.
lab [40, 30, 11] or green screen [42, 26, 38, 1, 32]. Most realistic datasets
focus on SH or HO interactions and very few papers [26, 32] collect
interacting hand data.

###  2.2 Synthetic hand dataset

To obtain precise annotations and increase the dataset’s diversity, several
papers [27, 22, 3, 7, 15, 24, 41] established synthetic hand dataset by
applying multiple backgrounds [41] or different hand textures [7]. Most
datasets [3, 7, 27, 41] focus on SH pose data. DARTset [7] introduced a shaped
wrist and rendered hand images with different skins and accessories. But the
dataset did not contain IH. To simulate the HO interactions, Hasson et al.
[15] utilized physics engine [25] to generate object manipulation poses, but
their rendered images are not photo-realistic. Although some datasets [22, 24]
provide poses of both hands, the rendered images are not natural enough and
lack diversity. Those poses of Ego3d [22] were randomized, which leads to
severe interpenetration between hands and the pose is relatively strange.
Based entirely on the pose annotations of IH2.6M [26], AIH [24] produced a
synthetic interacting hand dataset, but only hand masks were created and other
annotations were missing.

We summarize some representative hand datasets and compare them to ours in
Table 1. While most datasets focus on SH or HO interactions, they are
deficient in handling mesh collision, maintaining high-quality annotations,
and providing pose diversity to some extent.

Figure 2: The distribution of anchors and hand subdivision. Purple points
denote the anchors.

##  3 RenderIH dataset

One of the main contributions of our paper is the interacting hand pose
optimization method that can generate valid and natural poses. In our paper,
valid poses are non-penetration hands and conform to the anatomic constraint
outlined in Table 2. The natural poses not only conform to the anatomy but
also frequently occur in daily life. We uniformly combine generated poses with
a variety of hand textures, high dynamic range (HDR) backgrounds, and camera
views. All collections are sampled independently to create images as diverse
as possible. In Section 3.1, we introduce our new hand pose generation
algorithm. After hand pose generation, how to render the synthetic image is
demonstrated in Section 3.2. In Section 3.3, we briefly introduce some
statistics about our RenderIH dataset.

###  3.1 Interacting hand pose optimization

Hand model. Based on the widely used parametric hand model MANO [29], Yang et
al. [37] proposed A-MANO, which assigns the twist-splay-bend Cartesian
coordinate frame to each joint along the kinematic tree and fit the natural
hand better. Therefore, we adopt A-MANO to make our optimization more
biologically plausible.

Initial pose generation. To produce massive valid and natural IH interaction
poses, we derive raw poses from the IH2.6M [26] and then augment the raw poses
by assigning random rotation offsets to hand joints. The augmented poses are
shown in Figure 3, after augmentation, the rotation of the $j_{th}$ finger
joint can be expressed as:

$\displaystyle\\{R_{ji}\in
SO(3)\\}_{i=1}^{I}=\\{R_{j}R_{b}(\theta_{i}^{b})R_{s}(\theta_{i}^{s})\\}_{i=1}^{I},$
(1)

where $I$ is the number of augmentation, $R_{b/s}(\theta)$ denotes the
rotation along the bend/splay axe, the angle offset
$\theta^{b}\in[-90^{\circ},90^{\circ}]$ and
$\theta^{s}\in[-30^{\circ},30^{\circ}]$. $SO(3)$ is a group of 3D rotations.
$\theta^{s}=0$ when the joint is not the finger root joint. To avoid abnormal
gestures, each augmented joint is restricted according to Table 2. As the
augmented poses are totally random, most of them suffer from serious mesh
penetration and their gestures are unnatural, it is necessary to optimize the
poses.

Figure 3: Visualization for the effect of different components in
optimization.

Anti-penetration. Inspired by [17], we adopt multi-person interpenetration
loss to interacting hands and propose to divide the hand region into 16 parts.
Let $\Omega$ be the modified Signed Distance Field (SDF) [14] for each hand.
The SDF is defined on a voxel grid of dimensions $N\times N\times N$. It is
defined as follows:

$\displaystyle\Omega(x,y,z)=-min(SDF(x,y,z),0),$ (2)

where $\Omega$ states that its value within a hand is positive and
proportional to the distance from the surface, and it is simply zero outside.
The penetration loss for a single hand is calculated as follows:

$\displaystyle L_{p}^{s}=\sum_{v\in\\{V\\}}\Omega_{\hat{s}}(v).$ (3)

$V$ means the hand vertices, $s$ is the side of the hand, and $\hat{s}$ is the
side of the other hand. While the hand is highly articulated with a complex
pose and shape, basic hand mesh SDF is not accurate enough. We propose to
divide the hand into 16 parts based on its joint position and compute a
separate $\Omega$ function for each hand submesh which is divided according to
the hand subdivision in Figure 2. After applying for each submesh, the
penetration loss is defined as:

$\displaystyle
L_{p}^{s}=\sum_{i=1}^{N}\sum_{j=1}^{N}(\sum_{v\in\\{M_{sj}\\}}\Omega_{{\hat{s}}i}(v)),$
(4)

where $M_{sj}$ means the $j^{th}$ submesh of the hand. The total loss of this
part is $L_{p}=L_{p}^{right}+L_{p}^{left}$. The detailed visualization
comparison between basic SDF loss and our penetration loss is shown in the
supplementary material (SM).

Interhand attraction. When the IH is in close contact, severe hand occlusion
may occur, making it difficult to make annotations. Additionally, the
available close contact data are limited. To address this problem, it is
recommended to ensure the IH remains in tight contact.

To create contact between the hands, simply bringing the closest vertices
together would suffice. However, to reduce the optimization’s time complexity,
we adopt anchors to guide the position and pose of both hands. As shown in
Figure 2, to downsample the hand vertices as anchors, we traverse IH2.6M to
assess the contact frequency of each vertex with the other hand. We selected
the vertices with the highest contact frequency as the initial anchors and
proceeded to sample the remaining vertices sequentially. Subsequently, we skip
the 2-hop neighbors and then continue to sample the yet-to-be-selected ones.
Finally, we obtained 108 anchors.

If anchor $a_{j}^{l}$ on the left and the anchor $a_{i}^{r}$ on the right hand
are the closest, they will establish an anchor pair, and the loss of anchor
pairs is defined as:

$\displaystyle L_{ij}^{A}=\frac{1}{2}k_{ij}{\Delta{d_{ij}}}^{2},$ (5)

where $\Delta{d_{ij}}=||a_{i}^{r}-a_{j}^{l}||_{2}$. And
$k_{ij}=0.5*cos(\frac{\pi}{s}\bar{\Delta{d_{ij}}})+0.5$, in which
$\bar{\Delta{d_{ij}}}$ is the initial distance between anchors pair. This
definition means the initially close anchors tend to keep in contact.
Especially the factor $s$ is set to $0.02m$, and we set $k_{ij}=0$ if
$\bar{\Delta{d_{ij}}}>s$. The anchor pairs connection and $k_{ij}$ will be
rebuilt during the optimization to adapt to dynamically changing IH poses.

However, only these constraints cannot keep interacting poses valid with
random joint angles, we further introduce anatomic optimization.

Anatomic Optimization. The finger comprises joints, namely the Carpometacarpal
joint (CMC), the Metacarpophalangeal joint (MP), and the Interphalangeal joint
(IP). According to the coordinates systems of A-MANO, each finger has three
joints, and we denote them as root (CMC of thumb, MP of the others), middle
(MP of thumb, Proximal IP of the others), and end joint (IP of thumb, Distal
IP of the others). Each of them theoretically has 3 DOF. We define the hand
pose in Figure 2 as the T-pose, where all rotation angles are zero. The
constraints are defined as follows:

finger$\backslash$joint | root (B,S) | middle (B) | end (B)
---|---|---|---
thumb |  $[-20,40]$,$[-30,30]$ | $[-8,50]$ | $[-10,100]$
index |  $[-25,70]$,$[-25,15]$ | $[-4,110]$ | $[-8,90]$
middle |  $[-25,80]$,$[-15,15]$ | $[-7,100]$ | $[-8,90]$
ring |  $[-25,70]$,$[-25,15]$ | $[-10,100]$ | $[-8,90]$
pinky |  $[-22,70]$,$[-20,30]$ | $[-8,90]$ | $[-8,90]$

Table 2: Joint rotation limitations. The values are in degrees. ’B’/’S’
denotes whether the joint can bend/splay.

  * •

Available rotation directions. Middle and end joint can only rotate
$\theta_{i}^{b}$ around the B (Bend) axe, while the root can also rotate
$\theta_{i}^{s}$ around S (Splay) axe. Always keep $\theta_{i}^{t}=0$ around
the T (Twist) axe.

  * •

Angle limitations. According to hand kinematics [10, 19], the joint rotation
limitations are presented in Table 2.

The anatomic optimization objective for each hand is defined as:

$\displaystyle
L_{a}=\sum_{i=1}^{15}\sum_{a\in\\{b,s,t\\}}(\beta(\theta_{i}^{a}))^{2},$ (6)

where
$\beta(\theta_{i}^{a})=max(\theta_{i}^{a}-\hat{\theta_{i}^{a}},0)+min(\theta_{i}^{a}-\check{\theta_{i}^{a}},0)$
is the deviation of the rotation angle from its range, and
$\hat{\theta_{i}^{a}}$/$\check{\theta_{i}^{a}}$ is the max/min value of
$\theta_{i}^{a}$’s range.

Natural discriminator. After anatomic optimization, the poses become valid.
However, as shown in Figure 3(e), some optimized poses would not be natural
enough. To get the natural poses, we further employ a discriminator
$\mathcal{D}$. The detailed structure of the discriminator is illustrated in
Figure 4. The single-hand pose $\Theta$ is given as input to the multi-layer
discriminator. The output layer predicts a value $\in[0,1]$ which represents
the probability of belonging to the natural pose. The objective for
$\mathcal{D}$ is:

$\displaystyle
L_{\mathcal{D}}=\mathbb{E}_{\Theta\sim{P_{R}}}[(\mathcal{D}(\Theta)-1)^{2}]+\mathbb{E}_{\Theta\sim{P_{G}}}[\mathcal{D}(\Theta)^{2}],$
(7)

where $P_{R}$ represents a hand pose from real datasets, such as IH2.6M [26]
and Freihand [42], $P_{G}$ is a generated pose. The adversarial loss that is
backpropagated to pose optimization is defined as:

$\displaystyle
L_{adv}=\mathbb{E}_{\Theta\sim{P_{G}}}{[(\mathcal{D}(\Theta)-1)^{2}]}.$ (8)

The discriminator is pre-trained before optimization. We extract 63K natural
single-hand poses from Freihand [42], DexYCB [2], and IH2.6M [26], their
”natural” probabilities $p_{n}$ are labeled as 1. To get unnatural poses, we
follow the methods in ”Initial pose generation” to randomly add offsets to the
poses, and calculate their probabilities according to the offsets (the higher
the offsets, the closer the $p_{n}$ is to 0). The qualitative and quantitive
improvements brought by $\mathcal{D}$ could be seen in SM. Since the natural
standard may vary from person to person, we also conducted a user study to
confirm the discriminator’s effect in SM.

Figure 4: The architecture of the discriminator.

Poses Optimization. In IH optimization, for each hand, it has 15 joints
rotation $\Theta=\\{R_{i}\in SO(3)\\}_{i=1}^{15}$, hand root rotation
$R_{r}\in SO(3)$ and hand root translation $T_{r}\in\mathbb{R}^{3}$, we take
$\psi=\\{\Theta,T_{r}\\}$ as the optimization parameters and the total IH loss
is denoted as:

$\displaystyle\mathop{argmin}\limits_{\psi^{r},\psi^{l}}(w_{1}\sum_{i=1}^{A_{r}}\sum_{j=1}^{A_{l}}L_{ij}^{A}+w_{2}L_{a}+w_{3}L_{adv}+w_{4}L_{p}),$
(9)

where $A_{r}$/$A_{l}$ is the anchor numbers of right/left hand,
$L_{a}=L_{a}^{r}+L_{a}^{l}$, and $w_{*}$ is the weight hyperparameter.

###  3.2 Rendering

Our dataset offers various benefits, including high-resolution hand textures
that create a more natural appearance. Additionally, we simulate natural
lighting and environments to address limited diversity in studio settings.
Furthermore, our dataset covers a wide range of poses and camera positions,
bridging the gap between real-world applications and synthetic data.

Figure 5: Same hand with different hand textures.

Texture. To enhance the variety of skin textures we present a broad selection
of hues as illustrated in Figure 5. Color tones include white, light-skinned
European, dark-skinned European, Mediterranean or olive, yellow, dark brown,
and black. A total of 30 textures are available. In addition, random skin tone
parameters can be superimposed on these base skin tones in the shaders to
adjust brightness, contrast, and more. Apart from that, these textures also
depict wrinkles, bones, veins, and hand hairs to cope with differences in
gender, ethnicity, and age.

Figure 6: Same hand under diverse illumination.

Lighting and background. It is widely accepted that high-quality synthetic
data should resemble real-world scenes as much as possible. For instance, the
authors mixed their synthetic hands images with diverse real-world background
photographs when creating IH synthetic data [22]. However, simply pasting the
rendered hands on the background images is unnatural due to differences in
lighting conditions and light angles. Since creating a large number of various
synthetic 3D background models is time-consuming, we composite synthetic hands
with various real-world scenery panoramic images. We collect 300 high-dynamic-
range (HDR) photography with realistic indoor and outdoor scenes with
appropriate lighting for rendering purposes. They enable our hand models to
blend seamlessly with diverse settings resulting in highly photorealistic
rendered scenes (see Figure 6).

Figure 7: Different viewpoints from the camera track.

Camera Settings. We defined a spherical camera arrangement that can contain
both viewpoints, enhancing the generalization of the model to different
viewpoints. The center of the two-handed model is first computed and placed at
the center of the world, and the camera track is placed around the center with
the camera pointing to the center. Figure 7 shows the layout of our simulation
environment. For each pose, we define four 360-degree circular tracks, which
can be averaged by the number of samples to define dense or sparse viewpoints.
For sparse sampling, 10 viewpoints were selected for each track.

Figure 8: Network architecture. We use the global features extracted by the
encoder to predict the left-hand features and right-hand features. After that,
our model gradually regresses the hand vertices from 3 identical correlation
encoder blocks by fusing multi-resolution image features with hand features.
Each correlation encoder contains two transformer encoders and lateral
connection from the other hand feature.  Figure 9: TSNE visualization for IH
poses distribution. Our data not only contain the raw poses of IH2.6M but also
fill the vacancy by augmentation, resulting in a broader distribution.

Render quality. Our major objective is to improve the photorealism of the
synthetic dataset. Therefore, we render the scene in Blender based on the ray-
tracing rendering engine Cycles. When creating the hand mesh, we used custom
shader settings to adjust the base color, subsurface, and roughness to make
the skin more realistic. The resolution of the image is 512$\times$334 pixels
and the color depth is 8 bits.

###  3.3 Analysis of RenderIH dataset

For distribution diversity comparison, we project the hand pose in IH2.6M and
RenderIH into the embedding space using TSNE [34]. Figure 9 clearly shows that
our data has a broader pose distribution than IH2.6M. Examples of synthetic
images are depicted in Figure 1 and the rendering video can be found in the
SM. More visualization effects of different optimization modules and
statistical information can be found in the SM.

##  4 TransHand

We propose a transformer-based network, TransHand, for 3D interacting hand
pose estimation and conduct extensive experiments on it.

As the transformer blocks are effective in modeling global interactions among
mesh vertices and body joints [23, 35], we propose a transformer-based IH
network. Our system contains two parts: the encoder and the decoder. Given an
image with size 256$\times$256, the encoder outputs a global feature vector
$G_{F}$ and the intermediate feature maps {$F_{i},i=1,2,3$} where $i$
indicates the feature level. After that, we map $G_{F}$ to the left vertex
feature $L_{F}$ and the right vertex feature $R_{F}$ by using fully connected
layers. Since the global feature does not contain fine-grained local details,
we concatenate different level features $F_{i}$ with the hand vertex feature
as input to the decoder blocks.

As shown in Figure 8, the decoder consists of 3 identical blocks. Each block
consists of 2 sub-modules, each sub-module is a typical transformer encoder
composed of a multi-head attention module and an MLP layer. Each block is made
up of two transformer encoders. As there is usually mutual occlusion in IH, it
is natural to combine the other hand feature to improve the estimation
precision. Inspired by Slowfast [6], we use a symmetric structure to
incorporate the other hand feature by adding it, which is the lateral
connection in the Correlation Encoder (CE) shown in Figure 8. Each block has
three inputs including the left vertex feature, right vertex feature, and
image feature. The blocks gradually upsample the coarse mesh up to refined
mesh and finally to the original dimension with 778 vertices.

Figure 10: Qualitative results of our method on IH2.6M test set.

Loss Function. For training, we apply $L_{1}$ loss to 3D mesh vertices and
hand joints, and $L_{1}$ loss to 2D projected vertices and hand joints.

$\displaystyle
L_{joint}=\sum_{s=0}^{1}\sum_{i=0}^{M-1}\sum_{d\in\\{3D,2D\\}}{\|J_{s,i}^{d}-J_{s,i}^{d,GT}}\|_{1},$
(10) $\displaystyle
L_{mesh}=\sum_{s=0}^{1}\sum_{i=0}^{N-1}\sum_{d\in\\{3D,2D\\}}{\|V_{s,i}^{d}-V_{s,i}^{d,GT}}\|_{1},$
(11)

where $s$ represents the hand side, $i$ represents the number of joints or
vertices, and $d$ denotes whether the computation is for 3D or 2D. To
guarantee the geometric continuity of the predicted vertices, smoothness loss
is applied which regularizes the consistency of the normal direction between
the predicted and the ground truth mesh:

$\displaystyle
L_{smooth}=\sum_{s=0}^{1}\sum_{f=0}^{F-1}\sum_{j=0}^{2}{\|e_{f,j,s}\cdot{n_{f,s}^{GT}}\|_{1}},$
(12)

where $f$ means the face index of hand mesh, $j$ means the edge of face $f$
and $n^{GT}$is the GT normal vector of this face.

##  5 Experiments

###  5.1 Experiment setup

Dataset. IH2.6M [26] is the largest real dataset with interacting hand (IH),
and most of our experiments are conducted on this dataset. As we are only
focused on IH, we selected only the IH data with both human and machine
annotations. After discarding single-hand samples and invalid labeling, we
obtain 366K training samples and 261K testing samples. Tzionas dataset [33] is
a small IH dataset. We only use it for generalization ability evaluation by
using the models trained from different datasets. H2O-3D [12] is a real
dataset with 3D pose annotations for two hands and an object during
interactions. It contains 60K samples. Ego3d [22] provides 50K synthetic
images and corresponding labels of two hands, in which 40K samples are IH and
the poses are randomized.

Implementation details. The input images are resized to $256\times 256$ and
fed to TransHand encoder to generate the global feature and image feature
maps. ResNet50 [16] is selected as the encoder. For all experiments, the
networks are implemented using Pytorch [28]. We train all models with IH
images using Adam optimizer. The initial learning rate is $1e^{-4}$ and the
batch size is 64. All experiments are performed on 1 NVIDIA Ampere A100 GPU.
To demonstrate the usefulness of our RenderIH, we train three mainstream IH
pose estimation methods on IH2.6M and a combination of IH2.6M and RenderIH,
InterNet111Since InterNet and DIGIT are trained on the IH subset of IH2.6M
v0.0, we train them on v1.0 to make fair comparisons. [26], DIGIT1 [4] and
state-of-the-art method IntagHand222All the training codes have been open-
sourced by the authors. [21].

Evaluation metrics. To evaluate these methods, we report results by two
standard metrics: Mean Per Joint Position Error (MPJPE) and the Mean Per Joint
Position Error with Procrustes Alignment (PAMPJPE) in millimeters (mm).
Additionally, to ensure a fair evaluation with prior research [21, 39], we
select the MCP joint of middle finger as root joint and also report SMPJPE
which performs scaling to the ground truth bone length. To evaluate the
accuracy of estimating the relative position between the left and right hand
roots during interaction, we utilize the mean relative-root position error
(MRRPE) [4] and hand-to-hand contact deviation (CDev) [5] metrics. More
results are presented in SM with wrist as root joint for future comparison.

###  5.2 Results and analysis

With $\mathcal{D}$ | No $\mathcal{D}$ | Raw poses | Augmented poses
---|---|---|---
81.25% | 54.68% | 90.82% | 32.92%
Table 3: User study on natural rate. The higher the number, the more natural
it is.

User study for naturalness. Since the perceptions of ”natural” may differ from
human to human, We conduct experiments to prove the discriminator’s effect. We
invited 20 persons with/without computer technical background, their ages are
from 20 to 60, and the proportion of male to female was approximately 2:1. For
each of them, we show 120 pictures (including 30 of augmented poses, 30 of
optimized poses, 30 of optimized without discriminator, and 30 of raw poses
from IH2.6M) of the IH poses, they are asked to determine whether the shown
poses are natural, we count the NR (natural rate) of each category. The
results are presented in Table 3, the “Raw poses” are those from IH2.6M[26],
they are performed by humans and have high NR, however, some serious mesh-
penetration caused by annotation mistakes might make the testers hardly to
determine the “natural”. The “Augmented poses” are augmented from the raw
poses by assigning random rotation offsets to hand joints, they follow the
joint limitation but have randomness, and some of them are in mesh
penetration, the NR is low in this category. Optimizing the augmented poses
without $\mathcal{D}$ solves the penetration, and the poses are valid, but the
poses are not natural enough. It is clear that $\mathcal{D}$ improves the
naturalness of the poses.

Effectiveness of correlation encoder. Table 4 shows the performance of models
with and without the CE. The baseline method fuses the left-hand feature and
right-hand feature with the image feature independently through a transform
encoder. The result indicates CE can improve performance by fusing the
correlation between hands. Our model is used as default model for subsequent
experiments.

method$\backslash$metric |  PAMPJPE/MPJPE/SMPJPE(mm)$\downarrow$
---|---
Baseline | 7.32/11.12/10.82
Baseline+CE | 6.76/10.6/9.63
Table 4: Effect of correlation encoder (CE) on IH2.6M test set (PAMPJPE/MPJPE/SMPJPE(mm)$\downarrow$). It is shown that CE helps reduce the error by a clear margin.  method$\backslash$train set | IH2.6M | Mixed
---|---|---
InterNet | 18.28 | 17.19
DIGIT | 15.48 | 14.28
IntagHand | 10.9 | 9.72
Ours | 10.6 | 10.06
Table 5: Comparison between models trained from IH2.6M and a mixture of
RenderIH and IH2.6M in MPJPE(mm)$\downarrow$. The methods are reproduced using
their official training code. Figure 11: Comparing MPJPE by the degree of
occlusion on Tzionas dataset. The IoU between groundtruth left/right masks
measures the degree of interaction. The left (yellow) and right (blue) hand
masks provide interaction examples in each IoU range

Mixing synthetic with real images. To demonstrate the usefulness of RenderIH,
we test InterNet, DIGIT, IntagHand, and our TransHand on the IH2.6M test set
under the setting of training with or without using the full 1M data from the
RenderIH dataset. As shown in Table 5, RenderIH is helpful to further reduce
the estimation error. For example, the error can be greatly reduced from
10.9mm to 9.72mm for the SOTA IntagHand method. The results prove that our
RenderIH has great complementarity with real data. Meanwhile, when hand-hand
occlusion is severe, training with our synthetic dataset can handle those
cases better than IH2.6M only which is shown in Figure 11.To quantify the
impact of interaction and occlusion, we use the IoU between left and right
hand ground truth masks following DIGIT [4]. The higher IoU implies more
occlusion and half-length of the error bars correspond to 0.5 times of MPJPE
standard deviation. With minimal occlusion, the MPJPE is similar between the
mixed image model and IH2.6M only. As occlusion increases, the mixed image
model reduces MPJPE more substantially than IH2.6M alone. This highlights the
value of our RenderIH data.

Figure 12: Results of training IH2.6M with different number of RenderIH images
on MPJPE(mm)$\downarrow$.

Synthetic data size influence. During the training phase that involved various
combinations of synthetic data and the IH2.6M training set, an obvious decline
in the error is observed initially, followed by a gradual decrease after the
incorporation of 900K synthetic images, as illustrated in Figure 12. The trend
indicates that beyond a certain volume of synthetic data, the benefits of
incorporating additional data become marginal. To balance the cost of training
and accuracy, we select 1M as the optimal size for RenderIH.

Training strategy comparison. The training strategy of synthetic data and real
data is studied in this section. From Figure 13, both data mix training and
pretraining from synthetic data can lead to significantly higher accuracy.
Compared to dataset mixing, pretraining on the synthetic followed by fine-
tuning on real images led to better precision. In contrast to dataset mixing,
our results suggest that pretraining on synthetic data followed by finetuning
on real images offers a more effective approach for reducing error.

Figure 13: Comparison between training with RenderIH only, with part of IH2.6M only, the combination of the two, with pretraining on RenderIH and finetuning on IH2.6M.  Method | PAMPJPE$\downarrow$ | MPJPE$\downarrow$ | SMPJPE$\downarrow$ | MRRPE$\downarrow$ | CDev$\downarrow$
---|---|---|---|---|---
InterNet∗ [26] | 11.72 | 18.28 | 16.68 | - | -
DIGIT∗ [4] | 9.72 | 15.48 | 13.43 | - | -
InterShape [39] | - | - | 13.07 | - | -
HDR [24] | - | 13.12 | - | - | -
IntagHand [21] | 6.10 | 10.30 | 8.79 | 12.1 | 25.1
IntagHand∗ | 7.16 | 10.90 | 10.47 | 13.6 | 29.6
Ours | 6.76 | 10.66 | 9.63 | 12.98 | 27.9
Ours# | 5.79 | 9.64 | 8.18 | 11.95 | 24.6
Table 6: Comparing with SOTA methods on IH2.6M test set ($*$ means official
code reproduction, $\\#$ means RenderIH pretraining)

Real data size influence. We study how the real data size affects the
estimation precision in Figure 13. We use all the samples from RenderIH in
this section. For real data, we sample the number of data ranging from 3663 to
366358, which takes 1%, 5%, 10%, 30%, 50%, 70%, and 100% of the real data.
Although training only on RenderIH performs poorly, the MPJPE can be greatly
reduced from 27.73mm to 12.6mm by finetuning on only 1% of real data. With
finetuning on 10% of real data, the MPJPE can be almost the same as training
on the full real data. When finetuning on all real data, the error can be
0.96mm lower than training only on all real data.

train set$\backslash$test set | IH2.6M | Tzionas
---|---|---
H2O-3D | 47.81/48.19 | 35.61/34.01
Ego3d | 58.40/57.48 | 56.90/54.89
RenderIH | 43.35/42.17 | 32.80/28.70
Table 7: Generalization ability comparison between H2O-3D, Ego3d, and RenderIH on MPJPE/SMPJPE (mm)$\downarrow$. The number of samples is 40K and fixed for each dataset.  train set$\backslash$test set | IH2.6M | Tzionas
---|---|---
H2O-3D+IH2.6M | 11.05/9.91 | 12.03/12.02
Ego3d+IH2.6M | 10.66/9.60 | 11.13/11.06
RenderIH+IH2.6M | 10.58/9.52 | 10.63/10.56
Table 8: Training on the mixture of datasets with all IH2.6M data on
MPJPE/SMPJPE (mm)$\downarrow$. The number of samples is 40K for each dataset.

Comparison with H2O-3D dataset, Ego3d dataset and RenderIH subset. In Table 7
and Table 8, we compare the generalization ability of these datasets with the
same number of 40K samples. The model pretrained on RenderIH reaches lower
error than other models pretrained on H2O-3D and Ego3d in Table 7, which
proves that our artificial data is realistic and the knowledge is more easily
transferable. The model trained on RenderIH performs better, possibly because
all images have objects that interfere with two-handed interaction in H2O-3D.
When training TransHand on RenderIH and IH2.6M, the estimation error is the
lowest both in the IH2.6M and Tzionas dataset which is shown in Table 8.
Especially the result on Tzionas dataset shows our varied pose distribution,
background, and texture is helpful for improving generalization.

Comparison with SOTA methods. As is shown in Table 6, our TransHand can
outperform SOTA IntagHand method trained from its official code. Furthermore,
their method involves multitask learning and their network comprises of
complex graph transformer modules. In comparison, our method is simpler yet
highly effective. When pretraining on RenderIH and finetuning on the IH2.6M
data, our method can further reduce the MPJPE by about 1mm. Better hand-hand
contact (CDev) and better relative root translation (MRRPE) can be observed in
this table. Moreover, it is shown in Table 9 that training on our dataset in
addition to IH2.6M can lead to obviously lower error on the Tzionas dataset
compared with training on IH2.6M alone. Results that are computed with wrist
as root is shown in Section 3.3 of SM.

Metrics |  MPJPE/MRRPE/CDev$\downarrow$
---|---
Training set$\backslash$Test set | Tzionas
RenderIH | 22.11/25.8/47.7
IH2.6M | 11.38/11.1/19.9
IH2.6M+RenderIH | 10.49/9.37/19.5
Table 9: The comparison of training with or without our dataset and test on
Tzionas dataset.

Qualitative results. Our qualitative results are shown in Figure 10. We can
see our method can generate high-quality IH results in IH2.6M images. More in-
the-wild results can be found in the SM.

##  6 Conclusion

In this paper, we propose a new large-scale synthetic dataset for 3D IH pose
estimation. Various experiments are conducted to study the effectiveness of
RenderIH. With the whole synthetic hand images and only 10% of real hand
images, we can achieve precision that is comparable to the same method which
is trained on all the real hand images. We hope that this dataset could be a
meaningful step towards developing 3D IH pose estimation models that do not
depend on real data and adaptable to to various scenes.

RenderIH: A large-scale synthetic dataset for 3D interacting hand pose
estimation (Supplementary Material)

This supplementary material contains additional information that could not be
included in the main manuscript due to space limitations. We will begin by
providing more detailed information about the dataset. Following that, we will
briefly discuss the pose optimization details in our approach. Then we will
then present additional visualization results from our qualitative
experiments. Finally, we will discuss the broader impacts and limitations of
our dataset.

##  1 More details on RenderIH

RenderIH is composed of 1 million synthetic images by varying the pose, camera
view, and environment (texture, lighting, and background). By collecting
annotations from IH2.6M, we removed samples of similar poses resulting in 3680
distinctive poses. For each distinctive pose, we augment $I=30$ poses. After
augmenting and optimization, we filter out those IH poses that still have
notable penetration or exceed joint limits, the remaining data accounts for
93% of the total, and we produce approximately 100K natural and non-
interpenetration IH poses. Then we apply 10 camera viewpoints to each pose and
produce 1M synthetic images in total. For each image, we randomly pick from a
collection of 300 HDR images to illuminate the hand and provide the background
together with a hand texture map. The rendering process took more than 200
hours using 4 NVIDIA A100 GPUs. As for the corresponding annotation, we
provide pose and shape parameters, 3D joint coordinates, 2D joint coordinates,
and camera intrinsic and extrinsic parameters. It is worth noting that the
synthetic data labels can be freely extended based on the user’s preferences,
such as generating hand parts segmentation masks. The automatically generated
annotations are free of noise and are more flexible than the traditional
labels of the real dataset. Some rendering examples to illustrate our photo-
realistic effect are provided in the video demo.

##  2 Pose optimization details

Penetration loss comparison. Hands are highly articulated and have a curved
and concave shape as a whole object. It makes the original multi-person
penetration loss [17] hard to detect the correct penetrated positions.
However, by dividing the hand into 16 almost convex parts, we can perform SDF
on each part and calculate the penetration, which can lead to more accurate
penetration detection results. The visual difference between our algorithm and
multi-person penetration loss [17] can be seen from Figure 14.

Figure 14: Visual comparison of penetration loss between ours and multi-person
penetration loss [17]. The penetration position of each hand can be obtained
by utilizing the penetration loss. The green color and red color are used to
denote where the right hand and left hand are penetrated respectively.  Figure
15: Building anchor pairs between IH. a) Nearest anchors are connected as the
pairs, they tend to make more contacts between both hands. b) Using face
normals to avoid abnormal pairs. c) The proper pairs are hard to build when
the hand parts are in serious overlaps.  Figure 16: Results for different
components of optimization. a) The poses augmented from the IH2.6M. b) After
being optimized by the proposed method, the poses become valid and natural. c)
Using vertices instead of anchors to make IH attraction has no significant
differences. d) Poses optimized without discriminator are valid but not
natural. e) Poses optimized without IH attraction have fewer contacts.  Figure
17: Qualitative comparison with our method and IntagHand [21] on InterHand2.6M
under a variety of viewpoints and different levels of inter-hand occlusion.
Red circles are used to highlight the positions where our methods can generate
better results. In the first row, our result can be even better than the
ground truth, where the middle, ring, and little fingers of the right hand are
curved.

Optimization details. As shown in Figure 15 a), The anchor pairs are built
between the closest anchors on both hands, making the IH has more contact. As
shown in Figure 15 b), to avoid abnormal anchors pairs, the pair can only be
established when $\bar{n_{i}^{a}}\cdot\bar{n_{j}^{a}}<0$, in which
$\bar{n^{a}}$ is the mesh face normal vector of the anchor. However, the IH
attraction might cause a negative influence when the parts are in serious
overlaps, as shown in Figure 15 c), there are conflicts between pairs, making
the mesh hard to separate, the simple way to solve this problem is separating
the hands at first so that we could have better anchor pairs.

$\displaystyle\mathop{argmin}\limits_{\psi^{r},\psi^{l}}(w_{1}\sum_{i=1}^{A_{r}}\sum_{j=1}^{A_{l}}L_{ij}^{A}+w_{2}L_{a}+w_{3}L_{adv}+w_{4}L_{p}),$
(13)

In our implementation, we optimize the loss function in Equation 13 which is
defined in the main paper in 215 iterations, we assign a larger weight $w_{4}$
for $L_{p}$ and a smaller weight $w_{1}$ for $L^{A}$ at the beginning to
separate the hands, $w_{1}$ will increase while $w_{4}$ decrease during the
optimization until 165th iteration. The anchor pairs will be rebuilt every 40
iterations to adapt to dynamically changing IH. The learning rate is set to
0.01 and will reduce after 20 no-loss-decaying iterations. Adam solver is
utilized for optimization.

Figure 18: Qualitative comparison of the interacting hand pose estimation with
our method and IntagHand [21] on in-the-wild images. The first three columns
display images from the Tzionas dataset [33], while the last three columns
showcase images from the RGB2Hands dataset [36]. Red circles are used to
highlight the positions where our methods can generate better results. From
the visualization results, we can clearly see that our model can generalize
better for in-the-wild images.

##  3 More visualization results

###  3.1 Results for different optimization components

Visualization of the effect of different components. We define multiple
optimization loss functions to get valid and natural IH poses. As shown in
Figure 16, the “Augmented Pose” is randomly augmented from the raw poses in
IH2.6M, the joint poses are restricted according to Table 2 in the main paper.
After being optimized by the full constraints, we get natural and non-
interpenetration poses. Comparing Figure 16(b) and Figure 16(c), we can see
that adopting anchors to make IH attraction has no significant differences
from employing full vertices while reducing the time complexity. Furthermore,
as demonstrated in Figure 16(d), the natural discriminator $\mathcal{D}$ could
make the IH more natural, the natural poses are defined in the main paper,
they not only conform to the anatomy but also frequently occur in daily life.
Additionally, as shown in Figure 16(e), IH attraction enhances hand contact,
which is hard to annotate in reality due to inter-occlusion.

Figure 19: Qualitative comparison of the interacting hand pose estimation with our method and IntagHand [21] on InterHand2.6M. Red circles are used to highlight the positions where our methods can generate better results.  Figure 20: Qualitative comparison of our models trained on different training sets which test on in-the-wild images.  Training set$\backslash$Metrics |  PAMPJPE/MPJPE/SMPJPE/MRRPE$\downarrow$
---|---
RenderIH | 13.50/47.73/49.42/32.08
IH2.6M | 6.76/16.78/13.97/14.63
IH2.6M+RenderIH | 5.79/15.78/12.16/14.15
Table 10: The comparison of training with or without our dataset and test on
IH2.6M dataset. Wrist joint is used as root.

###  3.2 Qualitative results comparison

Comparison with IntagHand. To better demonstrate the superiority of our data
and method, we compare our result with the existing state-of-the-art method
IntagHand [21] (Their models is also trained on the combination of IH2.6M [26]
and synthetic images). Some qualitative comparisons with IntagHand are shown
in Figure 19. By directly projecting 3D hand mesh onto the image, we can see
our result is closer to the pose in the raw image. Additionally, the results
of these images from various views are also presented (see Figure 17). In the
first row of Figure 17, our result can be even better than the ground truth,
where the middle, ring, and little fingers of the right hand are curved. To
further compare our generalization ability, we compare with IntagHand on in-
the-wild images (see Figure 18). The results show that our method can clearly
achieve less inter-penetration of two hands and more accurate finger
interactions.

Impact of synthetic data. When only RenderIH is used for training, the
performance is worse than when only IH2.6M is used, in part because the
background variation in Tzionas is limited. The trend can be seen in the
qualitative result in Figure 20. However, as a synthetic dataset, the function
of our dataset is to largely reduce the number of real data needed for
training instead of replacing real data entirely.

###  3.3 Quantitive results with wrist joint as root joint

For convenient future comparison, we report our model’s performance using
wrist joint as root joint following common practice. As shown in Table 10, the
model trained on a mixture of RenderIH and IH2.6M demonstrates consistent
improvement across all metrics compared to training on IH2.6M alone.

##  4 Broader impacts and limitations

Broader impacts. In this paper, we introduce a synthetic 3D hand dataset,
RenderIH, with accurate and diverse poses. Since there are no large-scale
synthetic interacting hand datasets, RenderIH will be impactful for the
community, due to its unprecedented scale, diversity, and rendering quality.
Moreover, the dataset not only can be used to improve the generalization
ability in real scenes but also can be used for domain adaptation.

Limitations. The hyperparameters of pose optimization are chosen on the basis
of experimental results, such as factor $k$ and $s$ in Interhand attraction
and weights in the final optimization loss. In the future, we may set them as
learnable parameters that can be automatically learned from data.

## References

  * [1] Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, and James Hays.  Contactpose: A dataset of grasps with object contact and hand pose.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 361–378, Cham, 2020. Springer International Publishing.
  * [2] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox.  Dexycb: A benchmark for capturing hand grasping of objects.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9040–9049, 2021.
  * [3] Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, and Xiaohui Xie.  Mvhm: A large-scale multi-view hand mesh benchmark for accurate 3d hand pose estimation.  pages 836–845, 01 2021.
  * [4] Zicong Fan, Adrian Spurr, Muhammed Kocabas, Siyu Tang, Michael J. Black, and Otmar Hilliges.  Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation.  In International Conference on 3D Vision, 3DV 2021, London, United Kingdom, December 1-3, 2021, pages 1–10. IEEE, 2021.
  * [5] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges.  Arctic: A dataset for dexterous bimanual hand-object manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023.
  * [6] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 6201–6210. IEEE, 2019.
  * [7] Daiheng Gao, Yuliang Xiu, Kailin Li, Lixin Yang, Feng Wang, Peng Zhang, Bang Zhang, Cewu Lu, and Ping Tan.  Dart: Articulated hand model with diverse accessories and rich textures, 10 2022.
  * [8] Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim.  First-person hand action benchmark with rgb-d videos and 3d hand pose annotations.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 409–419, 2018.
  * [9] Francisco Gomez-Donoso, Sergio Orts-Escolano, and Miguel Cazorla.  Robust hand pose regression using convolutional neural networks.  In Anibal Ollero, Alberto Sanfeliu, Luis Montano, Nuno Lau, and Carlos Cardeira, editors, ROBOT 2017: Third Iberian Robotics Conference, pages 591–602, Cham, 2018. Springer International Publishing.
  * [10] Verónica Gracia-Ibáñez, Margarita Vergara, Joaquín L Sancho-Bru, Marta C Mora, and Catalina Piqueras.  Functional range of motion of the hand joints in activities of the international classification of functioning, disability and health.  Journal of Hand Therapy, 30(3):337–347, 2017.
  * [11] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3193–3203, 2020.
  * [12] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit.  Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090–11100, 2022.
  * [13] Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang.  Megatrack: Monochrome egocentric articulated hand-tracking for virtual reality.  ACM Trans. Graph., 39(4), aug 2020.
  * [14] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black.  Resolving 3d human pose ambiguities with 3d scene constraints.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2282–2292, 2019.
  * [15] Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 04 2019.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016.
  * [17] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588, 2020.
  * [18] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh.  Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10873–10883, 2019.
  * [19] Derek G Kamper, T George Hornby, and William Z Rymer.  Extrinsic flexor muscles generate concurrent flexion of all three finger joints.  Journal of biomechanics, 35(12):1581–1589, 2002.
  * [20] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys.  H2o: Two hands manipulating objects for first person interaction recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138–10148, October 2021.
  * [21] Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu.  Interacting attention graph for single image two-hand reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2761–2770, 2022.
  * [22] Fanqing Lin, Connor Wilhelm, and Tony Martinez.  Two-hand global 3d pose estimation using monocular rgb.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2373–2381, 2021.
  * [23] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 12919–12928. IEEE, 2021.
  * [24] Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo.  3d interacting hand pose estimation by hand de-occlusion and removal.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 380–397, Cham, 2022. Springer Nature Switzerland.
  * [25] Andrew Miller and Peter Allen.  Graspit!: A versatile simulator for robotic grasping.  Robotics & Automation Magazine, IEEE, 11:110 – 122, 01 2005.
  * [26] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee.  Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image.  In European Conference on Computer Vision, pages 548–564. Springer, 2020.
  * [27] Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt.  Real-time hand tracking under occlusion from an egocentric rgb-d sensor.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1163–1172, Oct 2017.
  * [28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024–8035, 2019.
  * [29] Javier Romero, Dimitrios Tzionas, and Michael Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, 36, 11 2017.
  * [30] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4645–4653, 2017.
  * [31] Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin.  Real-time continuous pose recovery of human hands using convolutional networks.  volume 33, 08 2014.
  * [32] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall.  Capturing hands in action using discriminative salient points and physics simulation.  International Journal of Computer Vision, 118(2):172–193, 2016\.
  * [33] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall.  Capturing hands in action using discriminative salient points and physics simulation.  International Journal of Computer Vision, 118:172–193, 2016.
  * [34] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [35] Ziniu Wan, Zhengjia Li, Maoqing Tian, Jianbo Liu, Shuai Yi, and Hongsheng Li.  Encoder-decoder with multi-level attention for 3d human shape and pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13033–13042, 2021.
  * [36] Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt.  Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video.  ACM Transactions on Graphics (ToG), 39(6):1–16, 2020.
  * [37] Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu.  Cpf: Learning a contact potential field to model the hand-object interaction.  pages 11077–11086, 10 2021.
  * [38] Ruolin Ye, Wenqiang Xu, Zhendong Xue, Tutian Tang, Yanfeng Wang, and Cewu Lu.  H2o: A benchmark for visual human-human object handover analysis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15762–15771, 2021.
  * [39] Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang.  Interacting two-hand 3d pose and shape reconstruction from single color image.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 11334–11343. IEEE, 2021.
  * [40] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang.  3d hand pose tracking and estimation using stereo matching.  arXiv preprint arXiv:1610.07214, 2016.
  * [41] Christian Zimmermann and Thomas Brox.  Learning to estimate 3d hand pose from single rgb images.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 4913–4921, 2017.
  * [42] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max J. Argus, and Thomas Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 813–822, 2019.


11institutetext: Google Research
11email<EMAIL_ADDRESS>
# ICDAR 2023 Competition on Hierarchical Text Detection and Recognition

Shangbang Long  Siyang Qin  Dmitry Panteleev  Alessandro Bissacco  Yasuhisa
Fujii  Michalis Raptis

###### Abstract

We organize a competition on hierarchical text detection and recognition. The
competition is aimed to promote research into deep learning models and systems
that can jointly perform text detection and recognition and geometric layout
analysis. We present details of the proposed competition organization,
including tasks, datasets, evaluations, and schedule. During the competition
period (from January 2nd 2023 to April 1st 2023), at least $50$ submissions
from more than $20$ teams were made in the $2$ proposed tasks. Considering the
number of teams and submissions, we conclude that the HierText competition has
been successfully held. In this report, we will also present the competition
results and insights from them.

###### Keywords:

OCR Text Detection and Recognition Layout Analysis.

##  1 Introduction

Text detection and recognition systems [11] and geometric layout analysis
techniques [12, 13] have long been developed separately as independent tasks.
Research on text detection and recognition [14, 15, 16, 17] has mainly focused
on the domain of natural images and aimed at single level text spotting
(mostly, word-level). Conversely, research on geometric layout analysis [12,
13, 18, 19], which is targeted at parsing text paragraphs and forming text
clusters, has assumed document images as input and taken OCR results as fixed
and given by independent systems. The synergy between the two tasks remains
largely under-explored.

Recently, the Unified Detector work by Long et al. [20] shows that the
unification of line-level detection of text and geometric layout analysis
benefits both tasks significantly. StructuralLM [21] and LayoutLMv3 [27] show
that text line grouping signals are beneficial to the downstream task of
document understanding and are superior to word-level bounding box signals.
These initial studies demonstrate that the unification of OCR and layout
analysis, which we term as Hierarchical Text Detection and Recognition (HTDR),
can be mutually beneficial to OCR, layout analysis, and downstream tasks.

Given the promising potential benefits, we propose the ICDAR 2023 Competition
on Hierarchical Text Detection and Recognition. In this competition, candidate
systems are expected to perform the unified task of text detection and
recognition and geometric layout analysis. Specifically, we define the unified
task as producing a hierarchical text representation, including word-level
bounding boxes and text transcriptions, as well as line-level and paragraph-
level clustering of these word-level text entities. We defer the rigorous
definitions of word / line / paragraph later to the dataset section. Fig. 1
illustrates our notion of the unified task.

Figure 1: Illustration for the proposed unified task: Hierarchical Text
Detection and Recognition (HTDR). Given an input image, the unified model is
expected to produce a hierarchical text representation, which resembles the
form of a forest. Each tree in the forest represents one paragraph and has
three layers, representing the clustering of words into lines and then
paragraphs.

We believe this competition will have profound and long-term impact on the
whole image-based text understanding field by unifying the efforts of text
detection and recognition and geometric layout analysis, and furthering
providing new signals for downstream tasks.

The competition started on January 2nd 2023, received more than $50$
submissions in $2$ tasks in total, and closed on April 1st 2023. This report
provides details into the motivation, preparation, and results of the
competition. We believe the success of this competition greatly promotes the
development of this research field. Furthermore, the dataset except the test
set annotation and evaluation script are made publicly available. The
competition website111https://rrc.cvc.uab.es/?ch=18 remains open to submission
and provides evaluation on the test set.

##  2 Competition Protocols

###  2.1 Dataset

The competition is based on the HierText dataset [20]. Images in HierText are
collected from the Open Images v6 dataset [28], by first applying the Google
Cloud Platform (GCP) Text Detection
API222https://cloud.google.com/vision/docs/ocr and then filtering out
inappropriate images, for example those with too few text or non-English text.
In total, $11639$ images are obtained. In this competition, we follow the
original split of $8281/1724/1634$ for train, validation, test sets. Images
and annotations of the train and validation set are released publicly. The
test set annotation is kept private and will remain so even after the end of
the competition.

As noted in the original paper [20], we check the cross-dataset overlap rates
with the two other OCR datasets that are based on Open Images. We find that
$1.5\%$ of the $11639$ images we have are also in TextOCR [29] and $3.6\%$ in
Intel OCR [30]. Our splits ensure that our training images are not in the
validation or test set of Text OCR and Intel OCR, and vice versa.

Figure 2:  Example of hierarchical annotation format of the dataset.

The images are annotated in a hierarchical way of word-to-line-to-paragraph,
as shown in Fig. 2. Words are defined as a sequence of textual characters not
interrupted by spaces. Lines are then defined as space-separated clusters of
words that are logically connected and aligned in spatial proximity. Finally,
paragraphs are composed of lines that belong to the same semantic topic and
are geometrically coherent. Fig. 3 illustrates some annotated samples. Words
are annotated with polygons, with $4$ vertices for straight text and more for
curved text depending on the shape. Then, words are transcribed regardless of
the scripts and languages, as long as they are legible. Note that we do not
limit the character sets, so the annotation could contain case-sensitive
characters, digits, punctuation, as well as non-Latin characters such as
Cyrillic and Greek. After word-level annotation, we group words into lines and
then group lines into paragraphs. In this way, we obtain a hierarchical
annotation that resembles a forest structure of the text in an image.

Sample 1

Sample 2

Sample 3

Sample 4

Figure 3:  Illustration for the hierarchical annotation of text in images.
From left to right: word, line, paragraph level annotations. Words (blue) are
annotated with polygons. Lines (green) and paragraphs (yellow) are annotated
as hierarchical clusters and visualized as polygons. Images are taken from the
train split.

###  2.2 Tasks

Our challenge consists of $2$ competition tracks, Hierarchical Text Detection
and Word-Level End-to-End Text Detection and Recognition. In the future, we
plan to merge them into a single unified Hierarchical Text Spotting task that
requires participants to give a unified representation of text with layout.

####  2.2.1 Task 1: Hierarchical Text Detection

This task itself is formulated as a combination of $3$ tasks: word detection,
text line detection, and paragraph detection, where lines and paragraphs are
represented as clusters of words hierarchically.

In this task, participants are provided with images and expected to produce
the hierarchical text detection results. Specifically, the results are
composed of word-level bounding polygons and line and paragraph clusters on
top of words. The clusters are represented as forests, as in Fig. 1, where
each paragraph is a tree and words are leaves. For this task, participants do
not need to provide text recognition results.

Figure 4: Illustration of how hierarchical text detection can be evaluated as
$3$ instance segmentation sub-tasks. The coloring of each column indicates the
instance segmentation for each sub-task.

As illustrated in Fig. 4, we evaluate this task as $3$ instance segmentation
sub-tasks for word, line, and paragraph respectively. For word level, each
word is one instance. For line level, we take the union of each line’s
children words as one instance. For paragraph level, we aggregate each
paragraph’s children lines, and take that as one instance. With this
formulation, all the 3 sub-tasks will be evaluated with the PQ metric [31]
designed for instance segmentation, as specified in [20]:

$PQ=\frac{\sum_{(p,g)\in TP}IoU(p,g)}{|TP|+\frac{1}{2}|FP|+\frac{1}{2}|FN|}$
(1)

where $TP,~{}FP,~{}FN$ represent true positives, false positives, and false
negatives respectively. We use an IoU threshold of $0.5$ to count true
positives. Note that the PQ metric is mathematically equal to the product of
the Tightness score, which is defined as the average IoU scores of all TP
pairs, and the F1, score which is commonly used in previous OCR benchmarks.
Previous OCR evaluation protocols only report F1 scores which do not fully
reflect the detection quality. We argue that tightness is very important in
evaluating hierarchical detection. It gives an accurate measurement of how
well detections match ground-truths. For words, a detection needs to enclose
all its characters and not overlap with other words, so that the recognition
can be correct. The tightness score can penalize missing characters and
oversized boxes. For lines and paragraphs, they are represented as clusters of
words, and are evaluated as unions of masks. Wrong clustering of words can
also be reflected in the IoU scores for lines and paragraphs. In this way,
using the PQ score is an ideal way to accurately evaluate the hierarchical
detection task.

Each submission has $3$ PQ scores for word, line, and paragraph respectively.
There are $3$ rankings for these $3$ sub-tasks respectively. For the final
ranking of the whole task, we compute the final score as a harmonic mean of
the $3$ PQ scores (dubbed H-PQ) and rank accordingly.

####  2.2.2 Task 2: Word-Level End-to-End Text Detection and Recognition

For this task, images are provided and participants are expected to produce
word-level text detection and recognition results, i.e. a set of word bounding
polygons and transcriptions for each image. Line and paragraph clustering is
not required. This is a challenging task, as the dataset has the most dense
images, with more than $100$ words per image on average, $3$ times as many as
the second dense dataset TextOCR [29]. It also features a large number of
recognizable characters. In the training set alone, there are more than $960$
different character classes, as shown in Fig. 5, while most previous OCR
benchmarks limit the tasks to recognize only digits and case-insensitive
English characters. These factors make this task challenging.

For evaluation, we use the F1 measure, which is a harmonic mean of word-level
prediction and recall. A word result is considered true positive if the IoU
with ground-truth polygon is greater or equal to $0.5$ and the transcription
is the same as the ground-truth. The transcription comparison considers all
characters and will be case-sensitive. Note that, some words in the dataset
are marked as illegible words. Detection with high overlap with these words
(IoU larger than $0.5$) will be removed in the evaluation process, and ground-
truths marked as illegible do not count as false negative even if they are not
matched.

Figure 5:  Character set in the training split.

###  2.3 Evaluation and Competition Website

We host the competition on the widely recognized Robust Reading Competition
(RRC) website333https://rrc.cvc.uab.es/ and set up our own competition page.
The RRC website has been the hub of scene text and document understanding
research for a long time and hosted numerous prestigious competitions. It
provides easy-to-use infrastructure to set up competition, tasks, and carry
out evaluation. It also supports running the competition continuously, making
it an ideal candidate.

###  2.4 Competition Schedule

We propose and execute the following competition schedule, in accordance with
the conference timeline:

  * •

January 2nd, 2023: Start of the competition; submissions of results were
enabled on the website.

  * •

April 1st, 2023: Deadline for competition submissions.

  * •

April 15th, 2023: Announcement of results.

###  2.5 Other Competition Rules

In addition to the aforementioned competition specifications, we also apply
the following rules:

  * •

Regarding the usage of other publicly available datasets: HierText is the only
allowed annotated OCR dataset. However, participants are also allowed to do
self-labeling on other public OCR datasets as long as they don’t use their
ground-truth labels. In other words, they can use the images of other public
datasets, but not their labels. They can also use non-OCR datasets, whether
labeled or not, to pretrain their models. We believe they are important
techniques that can benefit this field.

  * •

Usage of synthetic datasets Synthetic data has been an important part of OCR
recently [22, 23, 24, 25, 26]. Participants can use any synthetic datasets,
whether they are public or private, but are expected to reveal how they are
synthesized and some basic statistics of the synthetic datasets if they are
private.

  * •

Participants should not use the validation split in training their models.

  * •

Participants can make as many submissions as desired before the deadline, but
we only archive the latest one submission of each participant in the final
competition ranking.

###  2.6 Organizer Profiles

Authors are all members of the OCR team at Google Research. In addition to
academic publications, authors have years of experience in building industrial
OCR systems that are accurate and efficient for a diversity of image types and
computation platforms.

##  3 Competition Results

In total, the competition received $30$ submissions in Task 1 and $20$
submissions in Task 2. Note that, we encourage participants to submit multiple
entries using different methods, for example, to understand the effect of
applying different techniques such as pretraining and synthetic data. To
produce the final leaderboard in compliance with the ICDAR competition
protocols, we only keep the latest $1$ submission from each participants. The
final deduplicated competition results are summarized in Tab. 1 / Fig. 6 and
Tab. 2 / Fig. 7. In total, the competition received $11$ unique submissions in
Task 1 and $7$ in Task 2.

|  |  |  | Task 1
---
metric
Word | Line | Paragraph
User | Method | Rank | H-PQ | PQ | F | P | R | T | PQ | F | P | R | T | PQ | F | P | R | T
YunSu Kim | Upstage KR | 1 | 76.85 | 79.80 | 91.88 | 94.73 | 89.20 | 86.85 | 76.40 | 88.34 | 91.32 | 85.56 | 86.48 | 74.54 | 86.15 | 87.40 | 84.94 | 86.52
DeepSE x Upstage |  | DeepSE hierarchical
---
detection model
2 | 70.96 | 75.30 | 88.49 | 93.50 | 83.99 | 85.10 | 69.43 | 82.43 | 82.65 | 82.21 | 84.23 | 68.51 | 81.39 | 81.69 | 81.10 | 84.17
zhm |  | hiertext_submit_0401
---
curve_199_v2
3 | 70.31 | 76.71 | 88.18 | 92.71 | 84.08 | 86.99 | 71.43 | 83.32 | 89.32 | 78.07 | 85.73 | 63.97 | 74.83 | 81.25 | 69.35 | 85.48
Mike Ranzinger | NVTextSpotter | 4 | 68.82 | 73.69 | 87.07 | 95.10 | 80.29 | 84.63 | 67.76 | 80.42 | 93.87 | 70.35 | 84.25 | 65.51 | 78.04 | 81.82 | 74.60 | 83.94
ssm |  | Ensemble of three
---
task-specific
Clova DEER detection
5 | 68.72 | 71.54 | 92.03 | 93.82 | 90.31 | 77.74 | 69.64 | 89.04 | 91.75 | 86.49 | 78.21 | 65.29 | 83.70 | 84.17 | 83.23 | 78.01
xswl |  | Global and local instance
---
segmentations for
hierarchical text detection
6 | 68.62 | 76.16 | 90.72 | 93.45 | 88.16 | 83.95 | 68.50 | 82.22 | 80.24 | 84.31 | 83.31 | 62.55 | 75.11 | 74.00 | 76.25 | 83.28
Asaf Gendler |  | Hierarchical Transformers
---
for Text Detection
7 | 67.59 | 70.44 | 86.09 | 88.47 | 83.83 | 81.82 | 69.30 | 85.23 | 87.83 | 82.78 | 81.31 | 63.46 | 78.40 | 77.84 | 78.97 | 80.94
JiangQing | SCUT-HUAWEI | 8 | 62.68 | 70.08 | 89.58 | 89.79 | 89.37 | 78.23 | 67.70 | 86.20 | 90.46 | 82.33 | 78.53 | 53.14 | 69.06 | 74.03 | 64.72 | 76.96
Jiawei Wang | DQ-DETR | 9 | 27.81 | 61.01 | 77.27 | 80.64 | 74.17 | 78.96 | 26.96 | 35.91 | 26.81 | 54.39 | 75.07 | 18.38 | 24.72 | 15.99 | 54.41 | 74.36
ZiqianShao | test | 10 | 21.94 | 27.45 | 41.75 | 51.82 | 34.95 | 65.76 | 25.61 | 39.04 | 51.50 | 31.43 | 65.59 | 16.32 | 24.52 | 35.61 | 18.70 | 66.57
Yichuan Cheng | a | 11 | 0.00 | 0.00 | 0.00 | 0.24 | 0.00 | 53.62 | 0.01 | 0.01 | 0.25 | 0.01 | 51.29 | 0.01 | 0.02 | 0.21 | 0.01 | 50.89

Table 1:  Results for Task 1. F/P/R/T/PQ stand for F1-score, Precision,
Recall, Tightness, and Panoptic Quality respectively. The submissions are
ranked by the H-PQ score. H-PQ can be interpreted as Hierarchical-PQ or
Harmonic-PQ. H-PQ is calculated as the harmonic means of the PQ scores of the
$3$ hierarchies: word, line, and paragraph. It represents the comprehensive
ability of a method to detect the text hierarchy in image. We omit the $\%$
for all these numbers for simplicity.  Figure 6:  Figure for the results of
task 1.

|  |  | Word
---|---|---|---
User | Method | Rank | PQ | F | P | R | T
YunSu Kim | Upstage KR | 1 | 70.00 | 79.58 | 82.05 | 77.25 | 87.97
DeepSE x Upstage | DeepSE End-to-End Text Detection and Recognition Model | 2 | 67.46 | 77.93 | 88.05 | 69.89 | 86.57
ssm | Ensemble of three task-specific Clova DEER | 3 | 59.84 | 76.15 | 77.63 | 74.73 | 78.59
Mike Ranzinger | NVTextSpotter | 4 | 63.57 | 74.10 | 80.94 | 68.34 | 85.78
JiangQing | SCUT-HUAWEI | 5 | 58.12 | 73.41 | 74.38 | 72.46 | 79.17
kuli.cyd | DBNet++ and SATRN | 6 | 51.62 | 71.64 | 82.76 | 63.15 | 72.06
LGS | keba | 7 | 44.87 | 54.30 | 68.37 | 45.03 | 82.64

Table 2:  Results for Task 2. F/P/R/T/PQ stand for F1-score, Precision,
Recall, Tightness, and Panoptic Quality respectively. The submissions are
ranked by the F1 score. We omit the $\%$ for all these numbers for simplicity.
Figure 7:  Figure for the results of task 2.

###  3.1 Submission Validation

In the final leaderboard, each participant is only allowed to have one
submission. We validate each submission and examine the number of submissions
from each team. If a team has more than one submission, we keep the latest one
and remove the rest from the leaderboard. Note that these removed submissions
will remain on the RRC portal for reference, since they also provide important
aspects into this research field. We adopt the following rules to determine
the authorship of each submission:

  * •

user_id: If two submissions have the same user_id field, it means they are
submitted by the same RRC user account and thus should be from the same team.

  * •

method description: Participants are asked to provide descriptive information
of their submissions, including authors, method details, etc. If two
submissions have strictly almost identical author list and method description,
we consider them to be from the same team.

###  3.2 Task 1 Methodology

Task 1 in our competition, i.e. Hierarchical Text Detection, is a novel task
in the research field. There are no existing methods that participants can
refer to. Even the previous work Unified Detector [20] can only produce line
and paragraph outputs but no word-level results. Among the $8$ submissions in
Task 1 which have disclosed their methods, we observed that $5$ of them
develop ‘multi-head plus postprocessing’ systems. These methods treat words,
lines, and paragraphs as generic objects, and train detection or segmentation
models to localize these three levels of text entities in parallel with
separate prediction branches for each level. In the post-processing, they use
IoU-based rules to build the hierarchy in the post-processing step, i.e.
assigning words to lines and lines to paragraphs. The most of the top ranking
solutions belong to this type of methods. One submission (from the SCUT-HUAWEI
team) adopts a cascade pipeline, by first detecting words and then applying
LayoutLMv3 [27] to cluster words into lines and paragraphs. The Hierarchical
Transformers for Text Detection method develops a unified detector similar to
[20] for line detection and paragraph grouping and also a line-to-word
detection model that produces bounding boxes for words. Here we briefly
introduce the top 2 methods in this task:

Upstage KR team ranks 1st place in Task 1, achieving an H-PQ metric of
$76.85\%$. It beats the second place by almost $6\%$ in the H-PQ metric. They
implemented a two-step approach to address hierarchical text detection. First,
they performed multi-class semantic segmentation where classes were word,
line, and paragraph regions. Then, they used the predicted probability map to
extract and organize these entities hierarchically. Specifically, an ensemble
of UNets with ImageNet-pretrained EfficientNetB7[9] / MitB4 [8] backbones was
utilized to extract class masks. Connected components were identified in the
predicted mask to separate words from each other, same for lines and
paragraphs. Then, a word was assigned as a child of a line if the line had the
highest IoU with the word compared to all other lines. This process was
similarly applied to lines and paragraphs. For training, they eroded target
entities and dilated predicted entities. Also, they ensured that target
entities maintained a gap between them. They used symmetric Lovasz loss [10]
and pre-trained their models on the SynthText dataset [25].

DeepSE X Upstage HK team ranks 2nd in the leaderboard. They fundamentally used
DBNet [7] as the scene text detector, and leveraged the oCLIP [6] pretrained
Swin Transformer-Base [5] model as the backbone to make direct predictions at
three different levels. Following DBNet, they employed Balanced Cross-Entropy
for binary map and L1 loss for threshold map. The authors also further fine-
tuned the model with lovasz loss [10] for finer localization.

###  3.3 Task 2 Methodology

Task 2, i.e. Word-Level End-to-End Text Detection and Recognition, is a more
widely studied task. Recent research [16, 2] focuses on building end-to-end
trainable OCR models, as opposed to separately trained detection and
recognition models. It’s widely believed that end-to-end models enjoy shared
feature extraction which leads to better accuracy. However, the results of our
competition say otherwise. The top 2 methods by the Upstage KR team and DeepSE
End-to-End Text Detection and Recognition Model team are all separately
trained models. There are two end-to-end submissions. The unified_model team
applies a deformable attention decoder based text recognizer and ranks 3th
place. Here we briefly introduce the top 2 methods in this task:

Upstage KR team uses the same task 1 method for detecting words. For word-
level text recognition, they use the ParSeq [1] model but replace the visual
feature extractor with SwinV2 [4]. The text recognizer is pretrained with
synthetic data before fine-tuning it on the HierText dataset. They use an in-
house synthetic data generator derived from the open source SynthTiger [26] to
generate word images using English and Korean corpus. Notably, they generate
5M English/Korean word images with vertical layout, in addition to 10M English
/ Korean word images with horizontal layout. For the final submission, they
use an ensemble of three text recognizers for strong and stable performance.

DeepSE End-to-End Text Detection and Recognition Model team also uses the
ParSeq [1] model as their recognizer. They point out that, in order to make
the data domain consistent between the training and inference stages, they run
their detector on training data, and then crop words using detected boxes.
This step is important int adapting the training domain to the inference
domain. This trick essentially improves their model’s performance.

##  4 Discussion

In the Hierarchical Text Detection task, the original Unified Detector [20]
can only achieve PQ scores of $48.21\%$, $62.23\%$, $53.60\%$ on the words,
lines, and paragraphs respectively. The H-PQ score for Unified Detector is
only $54.08\%$, ranking at 10th place if put in the competition leaderboard.
The winning solution exceeds Unified Detector by more than $20\%$. These
submissions greatly push the envelope of state-of-the-art Hierarchical Text
Detection method. However, current methods are still not satisfactory. As
shown in Fig. 6, we can easily notice that for all methods, word PQ scores are
much higher than line PQ scores, and line PQ scores are again much higher than
paragraph PQ scores. It indicates that, line and paragraph level detections
are still more difficult than word detection. Additionally, Fig. 8 shows that
layout analysis performance is only marginally correlated with word detection
performance, especially when outliers are ignored. We believe there’s still
hidden challenges and chances for improvement in layout analysis. Furthermore,
winning solutions in our competition rely on postprocessing which can be
potentially complicated and error-prone. It’s also important to improve end-
to-end methods.

Figure 8:  Correlation between text levels. Each dot is a submission in the
Task 1. Left: Correlation between word PQ and line PQ. Right: Correlation
between word PQ and paragraph PQ.

The task 2 of our challenge is a standard yet unique end-to-end detection and
recognition task. While it inherits the basic setting of an end-to-end task,
it is based on a diversity of images which has high word density, and it has
an unlimited character set. For this task, we see most of the submissions are
two-stage methods, where the detection and recognition models are trained
separately, and there’s no feature sharing. These two-stage methods achieve
much better performances than end-to-end submissions. This contrasts with the
trend in research paper that favors end-to-end trainable approaches with
feature sharing between the two stage. Therefore, we believe the HierText
dataset can be a very useful benchmark in end-to-end OCR research. Another
interesting observation for Task 2 is that, while most submissions achieve a
tightness score of around $80\%$, the correlation between tightness scores and
F1 scores and very low, with a correlation coefficient of $0.06$. It could
indicate that recognition is less sensitive to the accuracy of bounding boxes
after it surpasses some threshold. This would mean that the mainstream
training objective of maximizing bounding box IoU might not be the optimal
target. For example, a slightly oversized bounding box is better than a small
one which might miss some characters. With that said, a precise bounding box
is still useful itself, which indicates the localization. Another potential
reason is that bounding box annotation is not always accurate – it’s always
oversized because text are not strictly rectangular.

##  5 Conclusion

This paper summarizes the organization and results of ICDAR 2023 Competition
on Hierarchical Text Detection and Recognition. We share details of
competition motivation, dataset collection, competition organization, and
result analysis. In total, we have $18$ valid and unique competition entries,
showing great interest from both research communities and industries. We keep
the competition submission site open to promote research into this field. We
also plan to extend and improve this competition, for example, adding
multilingual data.

## References

  * [1] Bautista, Darwin, and Rowel Atienza. "Scene text recognition with permuted autoregressive sequence models." Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII. Cham: Springer Nature Switzerland, 2022.
  * [2] Ye, Maoyuan, et al. "DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting." arXiv preprint arXiv:2211.10772 (2022).
  * [3] Kim, Seonghyeon, et al. "DEER: Detection-agnostic End-to-End Recognizer for Scene Text Spotting." arXiv preprint arXiv:2203.05122 (2022).
  * [4] Liu, Ze, et al. "Swin transformer v2: Scaling up capacity and resolution." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
  * [5] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
  * [6] Xue, Chuhui, et al. "Language matters: A weakly supervised vision-language pre-training approach for scene text detection and spotting." Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII. Cham: Springer Nature Switzerland, 2022.
  * [7] Liao, Minghui, et al. "Real-time scene text detection with differentiable binarization." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 07. 2020.
  * [8] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in Neural Information Processing Systems 34 (2021): 12077-12090.
  * [9] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." International conference on machine learning. PMLR, 2019.
  * [10] Berman, Maxim, Amal Rannen Triki, and Matthew B. Blaschko. "The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  * [11] Long, Shangbang, Xin He, and Cong Yao. "Scene text detection and recognition: The deep learning era." International Journal of Computer Vision 129.1 (2021): 161-184.
  * [12] Lee, Joonho, et al. "Page segmentation using a convolutional neural network with trainable co-occurrence features." 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019.
  * [13] Yang, Xiao, et al. "Learning to extract semantic structure from documents using multimodal fully convolutional neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  * [14] Ronen, Roi, et al. "GLASS: Global to Local Attention for Scene-Text Spotting." arXiv preprint arXiv:2208.03364 (2022).
  * [15] Long, Shangbang, et al. "Textsnake: A flexible representation for detecting text of arbitrary shapes." Proceedings of the European conference on computer vision (ECCV). 2018.
  * [16] Qin, Siyang, et al. "Towards unconstrained end-to-end text spotting." Proceedings of the IEEE/CVF international conference on computer vision. 2019.
  * [17] Kittenplon, Yair, et al. "Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
  * [18] Liu, Shuang, et al. "Unified Line and Paragraph Detection by Graph Convolutional Networks." International Workshop on Document Analysis Systems. Springer, Cham, 2022.
  * [19] Wang, Renshen, Yasuhisa Fujii, and Ashok C. Popat. "Post-ocr paragraph recognition by graph convolutional networks." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.
  * [20] Long, Shangbang, et al. "Towards End-to-End Unified Scene Text Detection and Layout Analysis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
  * [21] Li, Chenliang, et al. "StructuralLM: Structural Pre-training for Form Understanding." arXiv preprint arXiv:2105.11210 (2021).
  * [22] Long, Shangbang, and Cong Yao. "Unrealtext: Synthesizing realistic scene text images from the unreal world." arXiv preprint arXiv:2003.10608 (2020).
  * [23] Liao, Minghui, et al. "SynthText3D: synthesizing scene text images from 3D virtual worlds." Science China Information Sciences 63 (2020): 1-14.
  * [24] Jaderberg, Max, et al. "Synthetic data and artificial neural networks for natural scene text recognition." arXiv preprint arXiv:1406.2227 (2014).
  * [25] Gupta, Ankush, Andrea Vedaldi, and Andrew Zisserman. "Synthetic data for text localisation in natural images." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [26] Yim, Moonbin, et al. "SynthTIGER: synthetic text image GEneratoR towards better text recognition models." Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV 16. Springer International Publishing, 2021.
  * [27] Huang, Yupan, et al. "LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking." arXiv preprint arXiv:2204.08387 (2022).
  * [28] Kuznetsova, Alina, et al. "The open images dataset v4." International Journal of Computer Vision 128.7 (2020): 1956-1981.
  * [29] Singh, Amanpreet, et al. "TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
  * [30] Krylov, Ilya, Sergei Nosov, and Vladislav Sovrasov. "Open images v5 text annotation and yet another mask text spotter." Asian Conference on Machine Learning. PMLR, 2021.
  * [31] Kirillov, Alexander, et al. "Panoptic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.