# ABC Easy as 123: A Blind Counter for Exemplar-Free
Multi-Class Class-agnostic Counting

Michael A. Hobley, Victor A. Prisacariu
Active Vision Lab, University of Oxford
<EMAIL_ADDRESS>
###### Abstract

Class-agnostic counting methods enumerate objects of an arbitrary class,
providing tremendous utility in many fields. Prior works have limited
usefulness as they require either a set of examples of the type to be counted
or that the image contains only a single type of object. A significant factor
in these shortcomings is the lack of a dataset to properly address counting in
settings with more than one kind of object present. To address these issues,
we propose the first Multi-class, Class-Agnostic Counting dataset (MCAC) and A
Blind Counter (ABC123), a method that can count multiple types of objects
simultaneously without using examples of type during training or inference.
ABC123 introduces a new paradigm where instead of requiring exemplars to guide
the enumeration, examples are found after the counting stage to help a user
understand the generated outputs. We show that ABC123 outperforms contemporary
methods on MCAC without the requirement of human in-the-loop annotations. We
also show that this performance transfers to FSC-147, the standard class-
agnostic counting dataset. Project page: ABC123.active.vision

(a) Exemplar-Based Counting

(b) Our Exemplar-Free Counting

Figure 1: ABC123 counts objects of multiple unseen types. Our method not only
does not need exemplars to define the type to count but also finds examples of
each type it has counted.

##  1 Introduction

Given an image and told to ‘count’, a person would generally understand the
intended task and complete it with accuracy even if the kind of objects
present has not been previously observed. However, in cases with a large
number of objects, this is likely very slow.

This natural human ability to count arbitrarily has not been modelled by
today’s datasets or methods. Current automated methods can count with a fair
level of accuracy, so long as they have either an exemplar image as a prior on
the type to count or an image with only one class of object.

Since real-world situations will likely include objects of multiple classes,
methods need to accurately count in these environments. We first introduce
MCAC, a new synthetic multi-class class-agnostic counting dataset, and show
that methods previously assumed to work in multi-class settings perform poorly
on it. We then propose ABC123, a multi-class class-agnostic counter which does
not need exemplars during training or inference. ABC123 significantly
outperforms prior works on MCAC while also generalising to other datasets,
mirroring the arbitrary nature of human counting abilities.

Our contributions are:

  * •

We introduced MCAC, the first multi-class class-agnostic counting dataset

  * •

We propose ABC123, the first exemplar-free multi-class class-agnostic counter

  * •

We show that prior methods do not perform as expected in multi-class settings
and that ABC123 tackles multi-class counting effectively

Figure 2:  An example image from the training set of MCAC. All objects have
associated instance labels, class labels, bounding boxes, centre points, and
occlusion percentages.

##  2 Related Work

Class-specific counting methods aim to enumerate the instances of a single or
small set of known classes [1, 5, 8, 23]. These methods struggle to adapt to
novel classes and need new data and retraining for each type of object. To
address these issues, Lu et al. [12] proposed class-agnostic counting, a
framework without inference-time classes during training. Still most class-
agnostic methods [16, 25, 19], including this one, require an exemplar images
of the object class at test time. These methods generally work by creating a
sufficiently general feature space and applying some form of matching to the
whole feature map [25, 18] or to proposed regions of interest [16, 19].

Recent works, RepRPN [15], CounTR [11], ZSC [24] and RCC [6] do away with
exemplar images at inference-time, removing the need for intervention during
deployment. RepRPN is a two-step method which proposes regions likely to
contain an object of interest and then uses them for an exemplar-based density
map regression method. It proposes more than one bounding box and enumerates
them separately. ZSC [24] uses a multi-stage process in which a text input is
used to generate an generic image of the type to be counted that is then used
to find exemplar patches. These exemplar patches then function as the input to
an exemplar-based method [18]. CounTR uses a large vision-transformer encoder-
decoder to regress a density map of instance locations. It is trained in a
mixed few/zero-shot way, applying understanding gained from exemplar-based
examples to exemplar-free cases.

It has been assumed that the above methods can function in multi-class
settings. However, this has not been proven rigorously because the main
dataset for class-agnostic counting (FSC-147/133 [16, 6]) contains only one
labelled class per image. In fact, we show in Sec. 5 that these methods
perform poorly in contexts with multiple types present. FSC-147 being single-
class has also explicitly motivated work such as RCC, which regresses a single
scalar value from an image. It is trained without exemplar images and uses
only scalar supervision instead of density maps. Even with the constraint of
only counting one kind of object and with no further direction on the type to
count, RCC achieves competitive results with exemplar-based methods on
FSC-147, showing the limitations of this dataset.

While large models with image inputs [14] like SAM [9] would seem to be able
to effectively count objects of arbitrary types, in fact these methods have
poor numerical understanding [13], and SAM performs unsatisfactorily on all
counting tasks but especially on images with small objects or a high density
of objects.

##  3 MCAC Dataset

###  3.1 Motivation

There are currently no datasets suitable for multi-class class-agnostic
counting problems. This significantly impacts the research into methods
addressing these tasks. The use of FSC-147 as the main class-agnostic counting
dataset has caused many problems, the lack of multi-class cases being one of
the most significant. This has lead to many methods being wrongly assumed to
work in multi-class settings. It has also limited the development of exemplar-
free multi-class methods. To this end we introduce MCAC, the first multi-class
class-agnostic counting dataset.

While the deployment query scenario, counting given an unlabelled image of
objects, is natural, the training and quantitative evaluation of methods to
address it is not. To facilitate training and evaluation of methods in multi-
class settings, we need images with multiple objects of multiple types. To
evaluate a methods generalisability to unseen object types, the classes
present in the images need to be mutually exclusive between training,
validation and testing. It is infeasible to gather natural images with (a) a
wide variety of classes, (b) a wide variety of the number of times an object
appears in an image, and (c) no repetition of the types of object between the
train, test, and validation splits. Using synthetic images allows the above
constraints to be satisfied while also providing a high level of precision and
accuracy in the labels for each image. As shown in Sec. 5.3, the understanding
gained from training on synthetic data is general enough to apply to the
standard photographic counting dataset, FSC [16].

###  3.2 Ambiguity

Both exemplar-based and exemplar-free methods bump into problems of ambiguity.
If there are objects of varied levels of generality, which boundary should be
used? For example, on a chess board with a single white pawn as the exemplar,
should the count be of all the pieces, all the white pieces, all the white
pawns, all the pawns, and so on?

Given the infeasability of defining every possible way of grouping the objects
present in an image, we define a single way of grouping the objects: an
identical mesh and texture, independent of size or orientation. We do,
however, acknowledge the existence of other valid-but-unknown counts, the
unlabelled ways of grouping the objects.

###  3.3 Dataset

MCAC contains images with between 1 and 4 classes of object and between 1 and
400 instances per class. The distribution of classes per image and instances
per class are shown in Fig. 3. MCAC has three data splits: training with 4756
images (8298 counts) drawn from 287 classes; validation 2413 images (3640
counts) drawn from 37 classes, and testing with 2114 images (4286 counts)
drawn from 19 classes. Each instances in an image have associated class
labels, model labels, center coordinates, bounding box coordinates,
segmentation maps, unoccluded segmentation maps, and occlusion percentages.
The occlusion percentage is calculated as $1-\frac{A_{0}}{A_{1}}$, where
$A_{0}$ is the number of pixels in the final image and $A_{1}$ is the number
of pixels that would be seen if the object was unoccluded and was completely
within the bounds of the image.

Objects are ‘dropped’ into the scene, ensuring random locations and
orientations. As objects in real setting often vary in size, we vary the size
of objects by $\pm$50% from a random nominal size. We also vary the number,
location, and intensity of lights present. Models and textures are drawn from
ShapeNetSem [17]. The exact data splits by class name are in Appendix A.

1234$0$$1{,}000$$2{,}000$$3{,}000$$4{,}000$ |  050100150200251298$10^{0}$$10^{1}$$10^{2}$$10^{3}$
---|---

Number of Classes

(a) *

Number of Instances

(b) *

Figure 3: Distributions of the number of classes in an image and of the number
of instances of each class per image across the dataset.

###  3.4 Using MCAC

Here we lay out our suggested usage of MCAC, which we used to generate our
results. We will also release code for a pytorch dataset to enable easy
adoption.

Objects that are more than 70% occluded by either other objects or the edge of
the frame should be excluded. The ground-truth density map has a Gaussian
centred on the center pixel of each object with $\sigma=8$. As is standard
[16, 11, 25, 10], when the Gaussians from different instances overlap, we sum
them rather than taking the maximum value. We then scale the whole density map
so the sum over it is the correct value. This adjustment improved our results
by $\sim 5\%$.

When training exemplar-based methods, we take bounding boxes randomly from
instances with less than 30% occlusion. We evaluate these methods using the
bounding boxes of the three least occluded instances. In cases with more than
three equally occluded instances, we use those with the lowest instance IDs.

##  4 Method

Figure 4:  The ABC123 pipeline. Our method learns to count objects of multiple
novel classes without needing exemplar images. During training and
quantitative evaluation, the matcher aligns the unguided predictions to the
ground truth labels. The example prediction stage locates instances associated
with each generated count.

Our method, ABC123, takes an image with multiple instances of objects of
multiple types and regresses the count of each type. This is achieved blind,
i.e. on objects of arbitrary classes with no requirement to have seen the
object class during training or to have an exemplar image to define the type
during inference. We achieve this by first regressing density maps for each
type then enumerating the instances using integration. To facilitate training
and evaluating ABC123 in an exemplar-free way, we propose a matching stage. To
further increase the interpretability of the outputs of ABC123, we design an
example discovery stage which finds specific instances of the counted object.

###  4.1 Density Map Regression

For each image, there are $m$ classes present, each with an associated ground
truth count $y$ and density map $d$. We regress $\hat{m}$ counts and density
map predictions, $\hat{y}$ and $\hat{d}$ respectively.

$\hat{y}=\sum_{h,w}{\hat{d}_{(h,w)}}$ (1)

where $\hat{d}_{(h,w)}$ denotes the density value for pixel (h, w). We achieve
this by using $\hat{m}$ convolutional up-sampling heads on top of a vision
transformer backbone [4]. We use a vision transformer backbone due to its
globally receptive field and self-attention mechanism, which Hobley and
Prisacariu [6] showed is crucial to generating a complex understanding in
counting settings. Each head regresses a single pixel-wise density map
prediction and count prediction from a patch-wise low-resolution high-
dimensional feature space.

Similar to other contemporary methods [16, 11, 25, 10], we use the pixel-wise
error $||d-\hat{d}||_{1}$ as our loss, where $d$ and $\hat{d}$ are the ground
truth and predicted density maps. This produced more accurate results than
using the L2 distance between the $d$ and $\hat{d}$; see Sec. 6.1 for the full
comparison.

###  4.2 Matching

In single-class or exemplar-based settings, there is a single prediction-label
pair. However, in multi-class exemplar-free settings, there are multiple
predictions as well as multiple labels, without a clearly defined pairing.
This resembles class-discovery [21, 26] and clustering [7, 22] problems, where
the number and cardinality of new classes is not necessarily known.

In keeping with these fields and to facilitate training and evaluate accuracy,
we find correspondences between the set of $m$ known counts and the set of
$\hat{m}$ predicted counts. The correspondence matrix is defined as
$\mathcal{X}=\\{0,1\\}^{m\times\hat{m}}$ where $\mathcal{X}_{i,j}=1$ iff
prediction $i$ is assigned to label $j$. A problem instance is described by an
$m\times\hat{m}$ cost matrix $\mathcal{C}$, where $\mathcal{C}_{i,j}$ is the
cost of matching prediction $i$ and ground truth label $j$. The goal is to
find the complete assignment of predictions to labels of minimal cost.
Formally, the optimal assignment has cost

$\min_{\mathcal{X}}\sum_{i=0}^{m}\sum_{j=0}^{\hat{m}}\mathcal{C}_{i,j}\cdot\mathcal{X}_{i,j}$
(2)

Specifically, our cost function is defined as the the pixel-wise distance of
the normalised ground truth density map $d_{i}$ and the predicted density map
$\hat{d}_{j}$.

$\mathcal{C}_{i,j}=\left|\left|\frac{d_{i}}{||d_{i}||_{2}}-\frac{\hat{d}_{j}}{||\hat{d}_{j}||_{2}}\right|\right|_{2}$
(3)

The normalisation ensures the matching is done on the locality of the counted
objects rather than the magnitude of the prediction itself. We use the
Hungarian algorithm, specifically the Jonker-Volgenant algorithm outlined in
Crouse [3], to solve for $\mathcal{X}$ robustly.

The supervision loss for each image is the sum of the L1 difference of the
ground truth density maps and their matched predictions as:

$\mathcal{L}=\sum_{i,j}^{m,\hat{m}}||d_{i}-\hat{d}_{j}||_{1}\cdot\mathcal{X}_{i,j}$
(4)

It should be noted that every label has an associated prediction, but the
inverse is not the case as generally $\hat{m}>m$. This means we do not impose
a loss on the unmatched density maps. This allows the network to generate more
nuanced count definitions as it does not punish valid-but-unknown counts which
are likely present in any counting setting.

As is usual [21, 26, 7, 22], we use the same matching procedure to evaluate
our performance at inference-time. It should be noted that as this matching
uses the ground-truth density maps, it could be used to significantly benefit
a method’s quantitative results without improving its deployment capabilities.
Specifically, a method could in principle predict all possible density maps
and use the matching stage to pick the correct one. We limit ourselves to
generating 5 density maps to minimise this behaviour. We explore the effect of
this further in Sec. 6.3. We also normalise the density maps between 0 and 1
to ensure we are only matching to the locations of objects rather than the
counts themselves.

###  4.3 Example Discovery

While exemplar-free counting saves a user time, as no manual intervention is
required, it does require the user to interpret the results. A set of scalar
count values can be unclear as it is not always obvious which count
corresponds to which type of object in the input image. Density maps can also
often be difficult to interpret, especially in high density situations.

To aid the user in understanding to which class a generated count corresponds,
we propose flipping the usual exemplar-based paradigm. Instead of using
exemplar images to define the type to count, we find examples of the type
counted. We achieve this by using the peaks from the predicted density map as
seed inputs for a pre-trained segmentation method [9]. While this segmentation
method is often not accurate enough to segment a singular whole object given
only a singular point, they are generally good enough that an expanded
bounding box around them contains enough information for a user to understand
the class of object that has been counted; see Fig. 7 for examples.

Figure 5:  Comparison to other methods on MCAC. ABC123 produces more accurate
results than the exemplar-based methods without using exemplar images. The
ground truth (GT) and predicted counts are shown in the top right corner of
their respective density maps.

Figure 6:  Results of our method on images with 4 classes. ABC123 is able to
generate accurate counts and meaningful density maps from images with four
novel classes. MCAC has between one and four classes of object per image.

Figure 7:  Example Finding. Using our density map predictions and a SAM [9]
network, we can generate meaningful bounding boxes to aid a user in
understanding what has been counted.

###  4.4 Implementation

We use ViT-Small [4] due to its lightweight nature and for comparison to
methods that use the resnet-50 backbone, such as FamNet [16, 18]. ViT-S has a
similar number of parameters (21M vs 23M), throughput (1237im/sec vs
1007im/sec), and supervised ImageNet performance (79.3% vs 79.8%) as ResNet-50
[20]. ABC123 is trainable in less than eight hours using two 1080Tis. It takes
less than two hours to train just the head with a frozen backbone (ABC123❄).

Since vision transformers typically demand substantial training data, we
initialised our transformer backbone with weights sourced from Caron et al.
[2]. This self-supervised pre-training process endows the network with an
understanding of meaningful image features prior to exposure to our dataset
and without supervision. This approach reduces the risk of overfitting when
the model is then trained on our dataset.

Our counting heads, comprised of 3 Conv-ReLU-Upsample blocks, increases the
patch-wise resolution of the trained counting features from $k\times(28\times
28)$ to a pixel-wise density map prediction of $\hat{m}\times(224\times 224)$,
where $k$ is the dimensionality of the transformer features and $\hat{m}$ is
the number of predicted counts. For ABC123, $k=384$. We set $\hat{m}=5$ to
ensure that the method has the capacity to generate a count per defined class
in the dataset and at least one valid-but-unknown count. Our choice of ViT-S
limits the resolution of our input image to (224$\times$224) as opposed to the
$\geq$(384$\times$384) resolution used by contemporary methods with ResNet-50
or larger ViT backbones. For the example discovery stage, we use a frozen
pretrained ViT-B SAM model [9].

##  5 Results

###  5.1 Benchmarking Methods

We evaluate our method against two trivial baseline methods, predicting the
training-set mean or median count for all inference images. As there are no
previous multi-class exemplar-free class-agnostic counting methods, we compare
ABC123 to exemplar-based methods using separate exemplars from each of the
classes present. We compare our method to FamNet [16], BMNet [18] and CounTR
[11] on MCAC. Additionally, we compare to them and also RCC [6] and CounTR in
its zero-shot configuration on MCAC-M1, the subset of MCAC with only a single
type of object present per image.

To ensure a good comparison, we follow the suggested procedure laid out in
Sec. 3.4, evaluating the exemplar-based methods using the bounding boxes of
the three least occluded instances of a given class.

As in Xu et al. [24], we use Mean Absolute Error (MAE), Root Mean Squared
Error (RMSE), Normalized Absolute Error (NAE), and Squared Relative Error
(SRE) to evaluate the performance of each method.

$MAE=\frac{1}{nm}\sum_{j=1}^{n}\sum_{i=1}^{m_{j}}|y_{i}-\hat{y_{i}}|$
$RMSE=\sqrt{\frac{1}{nm}\sum_{j=1}^{n}\sum_{i=1}^{m_{j}}(y_{i}-\hat{y_{i}})^{2}}$
$NAE=\frac{1}{nm}\sum_{j=1}^{n}\sum_{i=1}^{m_{j}}\frac{|y_{i}-\hat{y_{i}}|}{y_{i}}$
$SRE=\sqrt{\frac{1}{nm}\sum_{j=1}^{n}\sum_{i=1}^{m_{j}}\frac{(y_{i}-\hat{y_{i}})^{2}}{y_{i}}}$

where $n$ is the number of test images, $m_{j}$ is the number of classes in
image $j$, and $y_{i}$ and $\hat{y_{i}}$ are the ground truth and the
predicted number of objects of class $i$ in image $j$ respectively.

###  5.2 MCAC

We achieve significantly better result than FamNet, BMNet, and CounTR on MCAC
both quantitatively and qualitatively without needing exemplars; see Tab. 1
for results and Fig. 5 for comparative examples. As seen in Fig. 5, FamNet
often fails to discriminate between objects of different kinds when they are
visually similar or in high density applications. Both quantitatively and
qualitatively, BMNet and CounTR outperform FamNet. However, in many cases,
they appear to count the ‘most obvious’ objects in the image regardless of the
provided exemplar images. Our method performs well on images with 4 classes
even when they have high intra-class appearance variation, such as having
different colours on different sides, and low inter-class variation; see Fig.
6. A downside to current exemplar-based class-agnostic counting methods is
that while they have some multi-class capabilities, they all take a single
exemplar at a time and produce only one count. This is slow and inefficient as
compared to our method which generates all counts simultaneously.

As would be expected, the performance of all methods improves when evaluating
on MCAC-M1, the images from MCAC with only a single class present; see Tab. 2.
This is due to a lack of ambiguity as to the type to be counted. This was more
significant when the methods were trained on MCAC-M1 instead of MCAC. In this
training configuration, the methods generally learnt a broader definition of
similarity as there was no chance they would accidentally combine classes or
count instances from another class. RCC performs well on MCAC-M1, showing the
strength of the simple count-wise loss in cases where there is little
ambiguity as to what is to be counted. In contrast to other methods, ABC123
trained on MCAC-M1 has similar performance to when it is trained on the full
MCAC dataset, demonstrating that it avoids the issues with other methods
concerning intra-class variance and combining classes. Training ABC123 with
only a single head ($\hat{m}=1$) and no matching stage has very similar
performance to using its default ($\hat{m}=5$) configuration with a matching
stage. This increases our belief that the matching head does not provide an
unfair advantage to our method’s quantitative results.

###  5.3 Applicability to FSC-147/133

ABC123 trained on only MCAC, a synthetic dataset, produces accurate results
when applied to FSC-133/147, a photographic dataset. However, it often finds
valid-but-unknown counts. As seen in Fig. 8, the generated counts are correct
for the type of object counted, but the type counted is often not aligned with
the labels in the original dataset. Classes are often divided into sub-
classes, and unlabelled classes are discovered.

There are two clear discrepancies between MCAC and FSC:

Cross domain image differences. As has been noted in many papers, applying a
method solely trained on synthetic data to real data is likely to have
negative results. Since images in the MCAC dataset are synthetic and simple,
they do not reflect all of the variation found in FSC. For example, all images
in MCAC are taken from above with no camera noise or harsh shadows. That being
said, as is seen in Fig. 8, the generalised counting ability gained from
training on MCAC clearly translates into the more varied and complex images in
FSC.

Inter and Intra class differences. As discussed in Sec. 3, the labels in MCAC
associate a count with objects of the same mesh and texture. FSC, however, is
labelled by hand using high-level semantic understanding. This often leads to
grouping of objects of significantly different geometries, colours, or
textures under the same label. As seen in Fig. 8, when applying ABC123 trained
on MCAC to images from FSC, we find that it often counts unlabelled classes,
generates multiple identical counts because it enumerates different parts of
the same objects, or subdivides classes by colour or geometry such as
splitting ‘chairs’ into ‘red chairs’ and ‘grey chairs’. Novel class-discovery
and duplicate part counts do not affect quantitative results on FSC as the
matching stage removes them. However, sub-class counting causes erroneous
numerical results. As our matching stage does not combine sub-class counts,
only one of the sub-class counts is associated with the human labelled count.
This generates a high numerical error and leads to poor quantitative results
in the standard benchmark tests.

|  | Val Set | Test Set
---|---|---|---
Method | Shots | MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
Mean | N/A | 39.87 | 53.56 | 3.07 | 11.40 | 42.67 | 59.68 | 2.79 | 10.93
Median | N/A | 36.25 | 58.15 | 1.51 | 6.70 | 39.81 | 65.36 | 1.38 | 6.73
Exemplar-based |  |  |  |  |  |  |  |  |
FamNet+ [16] | 3 | 24.76 | 41.12 | 1.12 | 6.86 | 26.40 | 45.52 | 1.04 | 6.87
BMNet+ [18] | 3 | 15.83 | 27.07 | 0.71 | 4.97 | 17.29 | 29.83 | 0.75 | 6.08
CounTR [11] | 3 | 15.07 | 26.26 | 0.63 | 4.79 | 16.12 | 29.28 | 0.67 | 5.71
Exemplar-free |  |  |  |  |  |  |  |  |
ABC123 ❄ | 0 | 14.64 | 23.67 | 0.46 | 2.97 | 15.76 | 25.72 | 0.45 | 3.11
ABC123 | 0 | 8.96 | 15.93 | 0.29 | 2.02 | 9.52 | 17.64 | 0.28 | 2.23
Table 1: Comparison to SOTA methods on MCAC. We significantly outperform methods which use exemplar images and test-time adaptation without requiring them. ABC123❄ denotes our method trained with a frozen pre-trained backbone.  |  |  |  | Val Set | Test Set
---|---|---|---|---|---
Method | Multi-Class Training | Shots | $\hat{m}$ | MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
Mean | N/A | N/A | N/A | 53.36 | 67.14 | 3.53 | 13.46 | 58.54 | 75.58 | 3.37 | 13.27
Median | N/A | N/A | N/A | 45.98 | 76.64 | 1.08 | 6.68 | 51.35 | 86.61 | 1.03 | 7.00
Exemplar-based |  |  |  |  |  |  |  |  |  |  |
FamNet+ [16] | ✓ | 3 | 1 | 24.97 | 48.63 | 0.36 | 3.79 | 28.31 | 54.88 | 0.35 | 3.97
FamNet+ [16] | ✗ | 3 | 1 | 12.54 | 24.69 | 0.37 | 4.71 | 13.97 | 26.19 | 0.25 | 2.12
BMNet+ [18] | ✓ | 3 | 1 | 11.70 | 23.08 | 0.26 | 2.39 | 11.57 | 22.25 | 0.24 | 1.96
BMNet+ [18] | ✗ | 3 | 1 | 6.82 | 12.84 | 0.25 | 2.95 | 8.05 | 14.57 | 0.19 | 1.43
CounTR [11] | ✓ | 3 | 1 | 11.44 | 21.37 | 0.33 | 2.36 | 10.91 | 21.70 | 0.29 | 2.01
CounTR [11] | ✓ | 0 | 1 | 13.57 | 25.53 | 0.30 | 2.48 | 13.09 | 25.72 | 0.29 | 2.41
CounTR [11] | ✗ | 3 | 1 | 9.00 | 16.91 | 0.41 | 3.56 | 9.96 | 18.92 | 0.38 | 2.93
CounTR [11] | ✗ | 0 | 1 | 9.16 | 17.13 | 0.42 | 3.56 | 10.10 | 19.10 | 0.40 | 3.02
Exemplar-free |  |  |  |  |  |  |  |  |  |  |
CounTR†[11] | ✗ | 0 | 1 | 11.46 | 21.24 | 0.35 | 2.78 | 12.54 | 23.84 | 0.31 | 2.38
RCC [6] | ✗ | 0 | 1 | 7.78 | 15.40 | 0.24 | 2.71 | 8.81 | 16.92 | 0.19 | 1.73
ABC123 ❄ | ✗ | 0 | 5 | 10.78 | 18.83 | 0.28 | 1.97 | 13.23 | 24.57 | 0.29 | 2.39
ABC123 ❄ | ✗ | 0 | 1 | 11.38 | 19.73 | 0.40 | 3.51 | 14.31 | 25.40 | 0.37 | 2.79
ABC123 ❄ | ✓ | 0 | 5 | 10.98 | 18.85 | 0.30 | 1.93 | 13.13 | 23.93 | 0.29 | 2.18
ABC123 | ✗ | 0 | 5 | 5.82 | 11.74 | 0.15 | 1.22 | 7.54 | 15.30 | 0.21 | 1.87
ABC123 | ✗ | 0 | 1 | 5.85 | 12.91 | 0.24 | 3.37 | 7.53 | 15.69 | 0.22 | 2.19
ABC123 | ✓ | 0 | 5 | 6.08 | 12.62 | 0.16 | 1.22 | 6.82 | 14.70 | 0.16 | 1.51
Table 2: Comparison to SOTA methods on MCAC-M1. MCAC-M1 is the subset of MCAC
with only one class present per image. Methods are either trained on the full
multi-class dataset (✓) or MCAC-M1 (✗). ‘$\hat{m}$’ denotes the number of
predictions the method generates per query and ‘Shots’ denotes the number of
exemplar images per query at inference time. CounTR†is an exemplar-free
adaption of CounTR as in Hobley and Prisacariu [6]. ABC123 outperforms other
methods when trained in both single or multi-class settings.  Figure 8:
ABC123 method trained on MCAC applied to FSC-147. ABC123 generates accurate
counts for the classes it locates. However, these classes are often not
aligned with the human annotations. ABC123 often discovers unlabelled classes,
divides labelled classes into sub classes by colour or shape, or counts the
same object multiple times by subdividing each object into component parts.
Centre: the ground truth (GT) density map is overlaid in green, Right:
predicted density maps are overlaid in red, green, and blue, with the
associated counts in the the top corner.

##  6 Ablations

In this section, we explain and validate various design decisions including
our loss and matching stage scaling. We also show the effect that generating
large numbers of predictions can have on quantitative benchmarks.

###  6.1 Validating Our Loss

Hobley and Prisacariu [6] showed that using image-wise count loss functions
like the absolute count error could be beneficial over pixel-wise loss
functions as they allow a network to learn its own idea of positional
salience. We test our loss, the pixel-wise error $||d-\hat{d}||_{1}$, against
both pixel-wise error squared $||d-\hat{d}||^{2}_{2}$, as in [16, 11, 24], and
the image-wise count percentage error $\frac{|y-\hat{y}|}{y}$ [6], where $d$
and $\hat{d}$ are the ground truth and predicted density maps, and $y$ and
$\hat{y}$ are the ground truth and predicted counts respectively.

Using an image-wise count loss creates poor results, with somewhat arbitrary
density maps and incorrect counts. This is likely due to the matching stage.
Whereas it usually operates on the density map to which the loss is directly
applied, it now operates over an unconstrained latent feature. This, along
with the added complexity of multi-class images, makes the problem
significantly harder to understand from purely integer supervision. We also
found that using a pixel-wise L2 slightly degraded performance as compared to
using the L1 distance between the ground-truth and predicted density maps.
Still, the predicted density maps were qualitatively meaningful. This
discrepancy is likely due to the increased significance the higher-density
edge-cases have on the training. See Tab. 3 for the quantitative comparison.

|  | Val Set | Test Set
---|---|---|---
Method | Loss | MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
ABC123 ❄ | Count-MAPE | 19.88 | 38.20 | 0.48 | 3.62 | 25.32 | 49.18 | 0.52 | 4.28
ABC123 ❄ | Pixel $L_{2}$ | 16.46 | 26.54 | 0.78 | 4.79 | 17.92 | 28.88 | 0.75 | 4.77
ABC123 ❄ | Pixel $L_{1}$ | 14.64 | 23.67 | 0.46 | 2.97 | 15.76 | 25.72 | 0.45 | 3.11
ABC123 | Count-MAPE | 15.95 | 29.71 | 0.50 | 3.54 | 16.28 | 29.82 | 0.49 | 3.58
ABC123 | Pixel $L_{2}$ | 10.76 | 19.31 | 0.49 | 3.84 | 11.91 | 21.42 | 0.51 | 4.21
ABC123 | Pixel $L_{1}$ | 8.96 | 15.93 | 0.29 | 2.02 | 9.52 | 17.64 | 0.28 | 2.23
Table 3: Effect of different loss functions during training. Pixel-wise
$L_{1}$ loss produces significantly better results than the pixel-wise $L_{2}$
loss and the count-wise MAPE.

###  6.2 Validating Our Matching Scaling

We found that normalising the density maps during training and testing was
beneficial because it forced the network to generate the correct localisation
as well as the correct magnitude. This normalisation also decreases the issue
of the matcher cheating to find the correct count. Instead of selecting the
correct count from a large set of predicted counts, the network must generate
a density map with a correct localisation. Using the L2 normalisation over the
L∞ has a slight performance increase for training the network and during
evaluation. See Tab. 4 for qualitative comparison of the normalisations during
the matching stage.

| Normalisation | Val Set | Test Set
---|---|---|---
Method | Train | Eval | MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
ABC123 ❄ | None | None | 20.31 | 32.40 | 0.78 | 5.40 | 22.67 | 36.63 | 0.92 | 7.25
ABC123 ❄ | None | L2 | 21.38 | 35.08 | 0.86 | 6.43 | 25.46 | 43.37 | 1.05 | 8.67
ABC123 ❄ | None | L∞ | 20.13 | 32.15 | 0.81 | 5.86 | 22.76 | 36.78 | 0.95 | 7.64
ABC123 ❄ | L2 | None | 41.20 | 66.04 | 0.87 | 6.22 | 45.04 | 73.41 | 0.87 | 6.51
ABC123 ❄ | L2 | L2 | 14.64 | 23.67 | 0.46 | 2.97 | 15.76 | 25.72 | 0.45 | 3.11
ABC123 ❄ | L2 | L∞ | 17.21 | 27.94 | 0.51 | 3.28 | 17.97 | 29.36 | 0.48 | 3.30
ABC123 ❄ | L∞ | None | 22.40 | 35.80 | 0.73 | 4.89 | 23.93 | 38.59 | 0.80 | 5.86
ABC123 ❄ | L∞ | L2 | 17.92 | 28.98 | 0.67 | 5.12 | 20.40 | 33.39 | 0.77 | 6.24
ABC123 ❄ | L∞ | L∞ | 19.33 | 30.75 | 0.67 | 4.77 | 20.84 | 33.61 | 0.74 | 5.91
ABC123 | None | None | 15.46 | 28.49 | 0.60 | 5.12 | 16.15 | 29.82 | 0.69 | 6.50
ABC123 | None | L2 | 14.83 | 27.25 | 0.68 | 6.15 | 17.04 | 31.88 | 0.84 | 8.06
ABC123 | None | L∞ | 14.75 | 27.41 | 0.65 | 5.92 | 16.54 | 30.80 | 0.77 | 7.45
ABC123 | L2 | None | 35.29 | 61.12 | 0.67 | 5.55 | 38.15 | 67.89 | 0.64 | 5.78
ABC123 | L2 | L2 | 8.96 | 15.93 | 0.29 | 2.02 | 9.52 | 17.64 | 0.28 | 2.23
ABC123 | L2 | L∞ | 9.95 | 18.16 | 0.32 | 2.22 | 9.89 | 18.55 | 0.29 | 2.20
ABC123 | L∞ | None | 15.72 | 28.83 | 0.59 | 4.69 | 16.11 | 30.22 | 0.67 | 6.16
ABC123 | L∞ | L2 | 12.17 | 22.81 | 0.57 | 5.09 | 13.97 | 27.66 | 0.71 | 7.05
ABC123 | L∞ | L∞ | 12.46 | 23.00 | 0.56 | 4.90 | 13.96 | 27.12 | 0.66 | 6.37
Table 4: Effect of using normalised density maps for the matching stage.
Normalisation improves the accuracy of ABC123. $L_{2}$ normalisation is more
effective than $L_{\infty}$ during training and evaluation.

###  6.3 Validating Our Number of Predictions

As predicted, generating much larger numbers of predictions increases the
quantitative performance of the network; see Tab. 5 for the complete results.
This is because the network can generate more diverse counts and use the
matching stage to select the best one. We believe, however, that this does not
align with a more useful network in a deployment situation. This numerical
gain derives purely from the matching stage, which is not present during
deployment. In fact, during deployment, this would correspond to a much more
difficult to interpret output as a user would have to figure out which of the
many outputs was most relevant. We found that when high numbers of predictions
were generate, fewer than half were used, i.e. the outputs of some heads were
never picked. This is likely due to these heads not being matched frequently
during training so the loss is rarely propagated back through them. There is
also significant redundancy between the heads. The predictions of certain
heads over the whole dataset were clearly similar and could be grouped. Of the
39 utilised heads, there were three groups of, respectively, 13, 6, and 4
heads that were very similar, lowering the effective number of utilised heads
to 19.

$\hat{m}$ | Val MAE | Channels Utilised | % Channel Utilisation
---|---|---|---
5 | 8.96 | 5 | 100%
10 | 8.39 | 10 | 100%
20 | 7.78 | 17 | 85%
50 | 7.26 | 34 | 68%
100 | 7.11 | 39 | 39%
Table 5: Effect of using more prediction heads. Increasing the number of pre-
matching predictions improves the quantitative results of our method. However,
as the number of heads increases, the percentage of heads that are frequently
($>0.4\%$) matched decreases.

##  7 Conclusion

In this work, we present ABC123, a multi-class exemplar-free class-agnostic
counter, and show that it is superior to prior exemplar-based methods in a
multi-class setting. ABC123 requires no human input at inference-time, works
in complex settings with more than one kind of object present, and outputs
easy to understand information in the form of examples of the counted objects.
Due to this, it has good potential for deployment in various fields. We also
propose MCAC, a multi-class class-agnostic counting dataset, and use it to
train our method as well as to demonstrate that exemplar-based counting
methods may not be as robust as previously assumed in multi-class settings.

## References

  * Cao et al. [2018] Xinkun Cao, Zhipeng Wang, Yanyun Zhao, and Fei Su.  Scale aggregation network for accurate and efficient crowd counting.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 734–750, 2018.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9650–9660, 2021.
  * Crouse [2016] David F. Crouse.  On implementing 2d rectangular assignment algorithms.  _IEEE Transactions on Aerospace and Electronic Systems_ , 52(4):1679–1696, 2016.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Go et al. [2021] Hyojun Go, Junyoung Byun, Byeongjun Park, Myung-Ae Choi, Seunghwa Yoo, and Changick Kim.  Fine-grained multi-class object counting.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 509–513. IEEE, 2021.
  * Hobley and Prisacariu [2023a] Michael Hobley and Victor Prisacariu.  Learning to count anything: Reference-less class-agnostic counting with weak supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023a.
  * Hobley and Prisacariu [2023b] Michael A Hobley and Victor A Prisacariu.  Dms: Differentiable mean shift for dataset agnostic task specific clustering using side information.  _arXiv preprint arXiv:2305.18492_ , 2023b.
  * Hoekendijk et al. [2021] Jeroen Hoekendijk, Benjamin Kellenberger, Geert Aarts, Sophie Brasseur, Suzanne SH Poiesz, and Devis Tuia.  Counting using deep learning regression gives value to ecological surveys.  _Scientific reports_ , 11(1):1–12, 2021.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Lin et al. [2021] Hui Lin, Xiaopeng Hong, and Yabin Wang.  Object counting: You only need to look at one.  _arXiv preprint arXiv:2112.05993_ , 2021.
  * Liu et al. [2022] Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie.  Countr: Transformer-based generalised visual counting.  _arXiv preprint arXiv:2208.13721_ , 2022.
  * Lu et al. [2018] Erika Lu, Weidi Xie, and Andrew Zisserman.  Class-agnostic counting.  In _Asian Conference on Computer Vision_ , 2018.
  * Ma et al. [2023] Zhiheng Ma, Xiaopeng Hong, and Qinnan Shangguan.  Can sam count anything? an empirical study on sam counting.  _arXiv preprint arXiv:2304.10817_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Ranjan and Hoai [2022] Viresh Ranjan and Minh Hoai.  Exemplar free class agnostic counting.  _arXiv preprint arXiv:2205.14212_ , 2022.
  * Ranjan et al. [2021] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai.  Learning to count everything.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3394–3403, 2021.
  * Savva et al. [2015] Manolis Savva, Angel X. Chang, and Pat Hanrahan.  Semantically-Enriched 3D Models for Common-sense Knowledge.  _CVPR 2015 Workshop on Functionality, Physics, Intentionality and Causality_ , 2015.
  * Shi et al. [2022] Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao.  Represent, compare, and learn: A similarity-aware framework for class-agnostic counting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9529–9538, 2022.
  * Sokhandan et al. [2020] Negin Sokhandan, Pegah Kamousi, Alejandro Posada, Eniola Alese, and Negar Rostamzadeh.  A few-shot sequential approach for object counting.  _arXiv preprint arXiv:2007.01899_ , 2020.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pages 10347–10357. PMLR, 2021.
  * Troisemaine et al. [2023] Colin Troisemaine, Vincent Lemaire, Stéphane Gosselin, Alexandre Reiffers-Masson, Joachim Flocon-Cholet, and Sandrine Vaton.  Novel class discovery: an introduction and key concepts.  _arXiv preprint arXiv:2302.12028_ , 2023.
  * Xie et al. [2016] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In _Proceedings of The 33rd International Conference on Machine Learning_ , pages 478–487, New York, New York, USA, 2016. PMLR.
  * Xie et al. [2018] Weidi Xie, J Alison Noble, and Andrew Zisserman.  Microscopy cell counting and detection with fully convolutional regression networks.  _Computer methods in biomechanics and biomedical engineering: Imaging & Visualization_, 6(3):283–292, 2018.
  * Xu et al. [2023] Jingyi Xu, Hieu Le, Vu Nguyen, Viresh Ranjan, and Dimitris Samaras.  Zero-shot object counting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15548–15557, 2023.
  * Yang et al. [2021] Shuo-Diao Yang, Hung-Ting Su, Winston H Hsu, and Wen-Chin Chen.  Class-agnostic few-shot object counting.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 870–878, 2021.
  * Yang et al. [2010] Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang.  Image clustering using local discriminant models and global integration.  _IEEE Transactions on Image Processing_ , 19(10):2761–2773, 2010.

##  A Class Splits

###  A.1 Training Classes

’1Shelves’, ’2Shelves’, ’3Shelves’, ’4Shelves’, ’5Shelves’, ’6Shelves’,
’7Shelves’, ’AAABattery’, ’AABattery’, ’AccentTable’, ’AirConditioner’,
’Airplane’, ’ArcadeMachine’, ’Armoire’, ’Ashtray’, ’Backpack’, ’Bag’, ’Ball’,
’BarCounter’, ’BarTable’, ’Barstool’, ’BaseballBat’, ’Basket’, ’Bathtub’,
’Battery’, ’BeanBag’, ’Bear’, ’Bed’, ’BeerBottle’, ’Bicycle’, ’Bidet’, ’Bird’,
’Blender’, ’Board’, ’Bottle’, ’Bookcase’, ’Booth’, ’Bowl’, ’Broom’, ’Bucket’,
’Bus’, ’Butterfly’, ’Cabinet’, ’Cabling’, ’Cage’, ’Camera’, ’CanOpener’,
’Canister’, ’CanopyBed’, ’Cap’, ’Carrot’, ’Cassette’, ’Cat’, ’CeilingFan’,
’CerealBox’, ’Chaise’, ’ChessBoard’, ’ChestOfDrawers’, ’ChildBed’,
’Chocolate’, ’Clock’, ’Closet’, ’Coaster’, ’CoatRack’, ’CoffeeMaker’,
’CoffeeTable’, ’Coin’, ’Compass’, ’Controller’, ’Cookie’, ’Couch’, ’Counter’,
’Courtyard’, ’Cow’, ’Cradle’, ’Credenza’, ’Cup’, ’CurioCabinet’,
’CuttingBoard’, ’DSLRCamera’, ’Dart’, ’DartBoard’, ’Desk’, ’Desktop’,
’Detergent’, ’DiningTable’, ’DiscCase’, ’Dishwasher’, ’Dog’, ’Donkey’, ’Door’,
’DoubleBed’, ’DraftingTable’, ’Dresser’, ’DresserWithMirror’, ’DrinkBottle’,
’DrumSet’, ’Dryer’, ’Easel’, ’Elephant’, ’EndTable’, ’Eraser’, ’Fan’, ’Fan
CeilingFan’, ’Faucet’, ’FileCabinet’, ’Fireplace’, ’Fish’, ’FlagPole’,
’Flashlight’, ’Folder’, ’Futon’, ’GameTable’, ’Gamecube’, ’Giraffe’, ’Guitar’,
’GuitarStand’, ’Gun’, ’Hammer’, ’HandDryer’, ’Hanger’, ’Harp’, ’Hat’,
’Headboard’, ’Horse’, ’Ipad’, ’Ipod’, ’IroningBoard’, ’Kettle’, ’Key’,
’KingBed’, ’LDesk’, ’Ladder’, ’LampPost’, ’Laptop’, ’Lectern’, ’Letter’,
’LightBulb’, ’LightSwitch’, ’Lock’, ’LoftBed’, ’Loveseat’, ’Magnet’, ’Marker’,
’Mattress’, ’MediaChest’, ’MediaDiscs’, ’MediaPlayer’, ’MediaStorage’,
’Microscope’, ’Microwave’, ’MilkCarton’, ’Mirror’, ’Monitor’, ’Motorcycle’,
’Mouse’, ’MousePad’, ’Mug’, ’Nightstand’, ’NintendoDS’, ’Notepad’, ’Orange’,
’Ottoman’, ’OutdoorTable’, ’Oven’, ’OvenTop’, ’PS’, ’PS2’, ’PS3’, ’PSP’,
’Palette’, ’Pan’, ’Paper’, ’PaperBox’, ’PaperMoney’, ’Pedestal’, ’Pen’,
’Pencil’, ’Phone’, ’Phonebooth’, ’Piano’, ’PianoKeyboard’, ’PicnicTableSet’,
’Picture’, ’PillBottle’, ’Pizza’, ’Planet’, ’Plate’, ’Plotter’, ’PosterBed’,
’PotRack’, ’PowerSocket’, ’PowerStrip’, ’Purse’, ’QueenBed’,
’QueenBedWithNightstand’, ’Rabbit’, ’Rack’, ’Radio’, ’Recliner’,
’Refrigerator’, ’RiceCooker’, ’Ring’, ’Rock’, ’Room’, ’RoundTable’, ’Rug’,
’Ruler’, ’Sandwich’, ’Scale’, ’ScrewDriver’, ’Sectional’, ’Shampoo’, ’Ship’,
’Shirt’, ’Shoes’, ’ShoppingCart’, ’Shower’, ’Showerhead’, ’Sideboard’,
’SingleBed’, ’Sink’ ’Sleeper’, ’Snowman’, ’SoapBar’, ’SoapBottle’, ’SodaCan’,
’Speaker’, ’StandingClock’, ’StaplerWithStaples’, ’Statue’, ’Suitcase’,
’SupportFurniture’, ’Sword’, ’TV’, ’Table’, ’TableClock’, ’Tank’, ’Tape’,
’TapeDispenser’, ’TapeMeasure’, ’Teacup’, ’Telescope’, ’TelescopeWithTripod’,
’Tent’, ’Thermostat’, ’Thumbtack’, ’TissueBox’, ’Toaster’, ’ToasterOven’,
’Toilet’, ’Toothbrush’, ’TrashBin’, ’Truck’, ’Trumpet’, ’Trundle’, ’TvStand’,
’USBStick’, ’Umbrella’, ’Vanity’, ’Vase’, ’VideoGameConsole’, ’Stapler’,
’VideoGameController’, ’Violin’, ’WallClock’, ’WallLamp’, ’WallUnit’,
’Wallet’, ’Wardrobe’, ’Washer’, ’WasherDryerSet’, ’WebCam’, ’Whiteboard’,
’Wii’, ’Window’, ’WineBottle’, ’WineGlass’, ’WineRack’, ’Wood’, ’Xbox’,
’Xbox360’, ’Candle’, ’Car’, ’Blind’, ’Spoon’, ’Fork’, ’Knife’

###  A.2 Validation Classes

’Book’, ’Books’, ’Chair’, ’Stool’, ’Donut’, ’ChairWithOttoman’, ’Bench’,
’SideChair’, ’OfficeChair’, ’KneelingChair’, ’OfficeSideChair’, ’AccentChair’,
’CeilingLamp’, ’ComforterSet’, ’Computer’, ’Copier’, ’Curtain’,
’DecorativeAccessory’, ’DeskLamp’, ’FloorLamp’ ’Lamp’, ’PaperClip’,
’PersonStanding’, ’PictureFrame’, ’Pillow’, ’Printer’, ’Scissors’,
’TableDeskLamp’, ’TableLamp’, ’Teapot’, ’Throw’, ’WallArt’, ’Poster’,
’WallDecoration’, ’WallArtWithFigurine’, ’Painting’, ’SkateBoard’,
’ToiletPaper’

###  A.3 Testing Classes

’Animal’, ’Keyboard’, ’Calculator’, ’CellPhone’, ’ComputerMouse’,
’DrinkingUtensil’, ’FoodItem’, ’Headphones’, ’Helicopter’, ’Plant’,
’PottedPlant’, ’RubiksCube’, ’Telephone’, ’ToyFigure’, ’Watch’, ’Sheep’,
’Glasses’, ’Fruit’, ’FruitBowl’, ’Apple’


# 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal

Hao Meng 113*3* 0000-0001-8136-0080 Sheng Jin 223*3* 0000-0001-5736-7434
Wentao Liu 3344 0000-0001-6587-9878 Chen Qian
Mengxiang Lin 33 0000-0002-8761-5563 11 Wanli Ouyang 4455 0000-0002-9163-2761
Ping Luo
1 Beihang University 2 The University of Hong Kong 3 SenseTime Research and
Tetras.AI 4 Shanghai AI Lab 5 The University of Sydney
<EMAIL_ADDRESS>22 0000-0002-6685-7950 {jinsheng  liuwentao
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Estimating 3D interacting hand pose from a single RGB image is essential for
understanding human actions. Unlike most previous works that directly predict
the 3D poses of two interacting hands simultaneously, we propose to decompose
the challenging interacting hand pose estimation task and estimate the pose of
each hand separately. In this way, it is straightforward to take advantage of
the latest research progress on the single-hand pose estimation system.
However, hand pose estimation in interacting scenarios is very challenging,
due to (1) severe hand-hand occlusion and (2) ambiguity caused by the
homogeneous appearance of hands. To tackle these two challenges, we propose a
novel Hand De-occlusion and Removal (HDR) framework to perform hand de-
occlusion and distractor removal. We also propose the first large-scale
synthetic amodal hand dataset, termed Amodal InterHand Dataset (AIH), to
facilitate model training and promote the development of the related research.
Experiments show that the proposed method significantly outperforms previous
state-of-the-art interacting hand pose estimation approaches. Codes and data
are available at https://github.com/MengHao666/HDR.

††* Equal Contribution.

###### Keywords:

3D Interacting Hand Pose Estimation, De-occlusion, Removal, Amodal InterHand
Dataset

##  1 Introduction

Figure 1: State-of-the-art hand pose estimation models often struggle to
estimate the 3D poses of interacting hands, due to severe hand-hand occlusion
and appearance ambiguity of two hands. In this example, we observe erroneous
pose estimation of the occluded part and significant uncertainty between the
left and right wrist. Our HDR framework tackles these two challenges via hand
de-occlusion (recovering the appearance content of the occluded part) and
removal (removing the other distracting hand). It transforms the challenging
interacting hand image into a simple single-hand image, which can be easily
handled by the hand pose estimator.

Estimating the 3D hand pose from a monocular RGB image is critical in many
real-world applications, e.g. human-computer interaction, augmented and
virtual reality (AR/VR), and sign language recognition. Although significant
progress has been made for single-hand pose estimation, analysis of hand-hand
interactions remains challenging. Estimating 3D interacting hand pose from a
single RGB image has attracted increasing research attention in recent years.

In this paper, we propose to decompose the challenging interacting hand pose
estimation task, and predict the pose of the left and the right hand
separately. However, solving single-hand pose estimation in close two-hand
interaction cases is non-trivial, because of two major challenges. One of the
main challenges is the severe hand-hand occlusion. Considering hands in close
interactions, the occlusion patterns are complex. Many areas of the target
hand can be occluded, making it very challenging to infer the pose of the
invisible parts. Another challenge is that the homogeneous and self-similar
appearance of hands (i.e. the left and the right hand) may cause ambiguities.
And the hand pose estimator may be confused by the other visually similar
distracting hand.

To tackle these challenges, we propose a simple yet effective Hand De-
occlusion and Removal (HDR) framework. Specifically, our HDR framework
comprises three parts, Hand Amodal Segmentation Module (HASM), Hand De-
occlusion and Removal Module (HDRM), and the Single Hand Pose Estimator
(SHPE). HAS segments both the complete (amodal) and visible parts for both two
hands. The resulting segmentation masks not only contain information to
localize the rough position of the two hands, but also provide cues for
subsequent de-occlusion and removal process by HDRM. _De-occlusion_ targets at
predicting the appearance content of the occluded part. _Removal_ targets at
removing the distracting part in the image. In our case, when estimating the
pose of the right hand, the left hand becomes the distracting part and should
be removed. As shown in Fig. 1, recent state-of-the-art hand pose estimation
methods suffer from severe hand-hand occlusion and the homogeneous appearance
of two hands, resulting in inferior pose estimation results. Thanks to our
proposed HDR framework, we can transform the challenging scenario of hand-hand
interactions into a common single-hand scenario, which can be easily handled
by an off-the-shelf SHPE.

However, to the best of our knowledge, there exist no datasets that contain
both the amodal segmentation and appearance content ground-truths of
interactive hands. To fill in this blank, we synthetically generate a large-
scale Amodal InterHand dataset, namely AIH dataset. The dataset contains over
3 million interacting hand images along with ground-truth amodal and modal
segmentation, de-occlusion and removal ground-truths. The dataset consists of
two parts, i.e. AIH_Syn and AIH_Render. AIH_Syn is obtained by simple random
copy-and-paste. It retains detailed and realistic appearance information.
However, it may generate implausible interacting hand poses that violate the
biomechanical structure of the human body. AIH_Render is generated by
rendering the textured 3D interacting hand mesh to the image plane. The inter-
dependencies between two hands are fully considered to avoid physically
implausible configurations, e.g. intersecting fingers. However, it may suffer
from the appearance gap because the rendered texture is synthetic. Combining
the advantages of both, we make a large-scale 3D hand-hand interaction dataset
with large pose and appearance variety. We empirically validate the
effectiveness of the synthetic dataset through extensive experiments. We
envision that our proposed dataset will foster the development of the related
research, e.g. interacting hand pose estimation, amodal or modal instance
segmentation, de-occlusion, etc.

Our proposed Hand De-occlusion and Removal (HDR) framework is simple,
flexible, and effective. Extensive experiments on the well-known InterHand2.6M
benchmark [21] show that our method significantly outperforms the state-of-
the-art 3D interacting hand pose estimation systems. Our framework builds upon
the latest research progress of amodal segmentation [33], de-occlusion [38,
36, 19], and 3D single-hand pose estimation [39]. Note that, we do not perform
complete comparisons with previous amodal segmentation, de-occlusion, and SHPE
approaches. We also do not claim any algorithmic superiority concerning model
architecture design. Because our aim is to propose a framework to solve the
challenges of 3D interacting hand pose estimation. And designing powerful
modules to improve the performance of amodal segmentation, de-occlusion, and
SHPE is not the focus of this paper.

Our contributions are summarized as follows:

  * •

We propose a novel Hand De-occlusion and Removal (HDR) framework to tackle the
challenging task of 3D interacting hand pose estimation.

  * •

We propose to explicitly handle the challenges of self-occlusion by hand de-
occlusion and the homogeneous appearance ambiguity by distractor removal. To
the best of our knowledge, we are the first to apply de-occlusion techniques
to improve the downstream pose estimation accuracy.

  * •

We propose the first large-scale synthetic Amodal InterHand Dataset (AIH) to
settle the task of hand de-occlusion and removal. We envision that AIH will
foster the development of the related research.

##  2 Related Work

###  2.1 Amodal Instance Segmentation and De-occlusion

Amodal Instance Segmentation. Unlike modal instance segmentation, which aims
at assigning labels to visible parts of instances, amodal instance
segmentation targets at producing the amodal (integrated) masks of each object
instance involving its occluded parts. Li and Malik [16] proposed the first
amodal instance segmentation model which iteratively expands the bounding
boxes and recomputes the heatmaps. Zhu et al. [40] proposed COCOA dataset for
amodal instance segmentation and presented AmodalMask model as the baseline.
Zhan et al. [36] propose a method to reason about the underlying occlusion
ordering and recover the invisible parts in a self-supervised manner.

De-occlusion. De-occlusion aims at recovering the appearance content of the
invisible occluded parts. SeGAN [6] adopts a residual network based model for
mask completion and inferring the appearance of the invisible parts of indoor
objects. Yan et al. [34] presented an iterative multi-task framework for
amodal mask completion and de-occlusion of vehicles. Zhou et al. [38] built
upon a well-known inpainting approach [19] and proposed to reason about the
occluded regions and recover the appearance content of humans. Baek et al. [2]
presents a weakly-supervised method to adapt from hand-object domain to single
hand-only domain. However, its image generation module and the pose estimator
are deeply coupled together, limiting its generalization ability to adapt to
different hand pose estimators and resulting in low-quality restored image.

Our approach differs from previous works in three major aspects. First,
previous works mostly focus on improving the quality of image content
recovery, while we aim to improve the performance of the downstream task, i.e.
3D interactive hand pose estimation. Second, compared with common rigid
objects, recovering the appearance content of the interacting hands is more
challenging because of larger pose variations, severe hand-hand occlusion, and
self-similar appearance of hands and fingers. Third, besides de-occlusion, our
proposed HDR framework also performs distracting hand removal to reduce the
ambiguities caused by the homogeneous appearance of hands.

###  2.2 Monocular RGB-based Hand Pose Estimation

Isolated hand pose estimation. RGB-based single (isolated) hand pose
estimation has made significant progress in the past few years. Zimmermann et
al. [41] introduced one of the first deep learning models to estimate hand
poses from monocular RGB images. It first uses HandSegNet to localize hand
regions, then uses PoseNet to estimate 2D hand poses, and finally maps 2D
poses into 3D space. Iqbal et al. [11] proposed to encode hand joint locations
with 2.5D heatmap representation to address the depth ambiguity problems and
improve localization precision. Spurr et al. [29] proposed a VAE-based
generative model to regress 3D hand joint locations. Zhou et al. [39] proposed
to fully exploit non-image MoCap data to improve model generalization and
robustness. Recently, many works also attempt to estimate 3D hand meshes from
monocular RGB images. Most of them [1, 4, 35] are model-based, which train a
convolutional neural network to estimate the MANO parameters [26]. Others are
model-free, which directly regress 3D vertices of the human hand using mesh
convolution [15], graph neural networks [5], or transformers [18].

Interacting hand pose estimation. Most works conduct interacting two-hand pose
estimation by utilizing multi-view RGB images [3, 9], depth data [23, 22, 30],
and tracking strategy [23, 28, 31]. Only a few existing works have considered
estimating 3D poses of two hands from a single RGB image, which is challenging
due to severe occlusion and close interactions. Lin et al. [17] employed a
synthetic egocentric hand dataset to learn to estimate two-hand poses from a
single RGB image. Moon et al. proposed a large-scale interacting hand dataset,
termed InterHand2.6M dataset [21], and designed the InterNet model to predict
2.5D hand poses. Zhang et al. [37] designed a hand pose-aware attention module
to address the self-similarity ambiguities and leveraged a context-aware
cascaded refinement module to improve pose accuracy. Kim et al. [13]
introduced an end-to-end trainable framework to jointly perform interacting
hand pose estimation. Rong et al. [27] presented a two-stage framework to
generate precise 3D hand poses and meshes with minimal collisions from
monocular single RGB images. Fan et al. [7] proposed DIGIT (DIsambiGuating
hands in InTeraction) to explicitly leverage the per-pixel probabilities to
reduce the ambiguities caused by self-similarity of hands.

In this work, we empirically show that existing hand pose estimators often
suffer from extreme self-occlusions and appearance ambiguity. To this end, we
propose a novel Hand De-occlusion and Removal (HDR) framework to explicitly
handle these two challenges, which significantly outperforms prior arts.

##  3 Method

Figure 2: Illustration of our Hand De-occlusion and Removal (HDR) framework
for the task of 3D interacting hand pose estimation. We first employ HASM
(Hand Amodal Segmentation Module) to segment the amodal and modal masks of the
left and the right hand in the image. Given the predicted masks, we locate and
crop the image patch centered at each hand. Then, for every cropped image, the
HDRM (Hand De-occlusion and Removal Module) recovers the appearance content of
the occluded part of one hand and removes the other distracting hand
simultaneously. In this way, the interacting two-hand image is transformed
into a single-hand image, and can be easily handled by SHPE (Single Hand Pose
Estimation) to get the final 3D hand poses.

###  3.1 Overview

As shown in Fig. 2, we propose a three-stage framework for interactive hand
pose estimation. The first stage segments the complete and visible part for
both two hands. The second stage recovers the RGB values of the occluded hand
and the background behind the distracting hand at the same time. The third
stage predicts the 3D pose of each hand separately.

###  3.2 Hand Amodal Segmentation Module (HASM)

As shown in Fig. 2, given an interacting two-hand image, we first obtain the
modal and amodal masks of both hands using the Hand Amodal Segmentation Module
(HASM). We simply adapt the off-the-shelf instance segmentation model, i.e.
SegFormer [33], to fit in our two-hand amodal segmentation tasks.
Specifically, we increase the number of decode heads from one to four to
predict four kinds of segmentation masks, namely the right hand amodal mask
$M_{ra}$, the right hand visible mask $M_{rv}$, the left hand amodal mask
$M_{la}$ and the left hand visible mask $M_{lv}$. These segmentation masks
contain (1) spatial localization information to roughly localize the
left/right hand, and (2) rich cues about the occluded regions for de-occlusion
and the distractor regions for removal.

We apply the binary cross entropy losses $\mathcal{L}_{BCE}\left(*\right)$ to
supervise the segmentation model. The final segmentation loss functions are
formulated as follows:

$\displaystyle\mathcal{L}_{HAS}$
$\displaystyle=\mathcal{L}_{BCE}\left(M_{ra},M_{ra}^{*}\right)+\mathcal{L}_{BCE}\left(M_{lv},M_{lv}^{*}\right)+$
(1) $\displaystyle\quad\
\mathcal{L}_{BCE}\left(M_{la},M_{la}^{*}\right)+\mathcal{L}_{BCE}\left(M_{lv},M_{lv}^{*}\right),$

where $M_{ra}$, $M_{lv}$, $M_{la}$, and $M_{lv}$ are predicted segmentation
masks; $M_{ra}^{*}$, $M_{lv}^{*}$, $M_{la}^{*}$, and $M_{lv}^{*}$ are the
corresponding ground-truth masks.

###  3.3 Hand De-occlusion and Removal Module (HDRM)

Figure 3: Illustration of the HDRNet. The input of HDRNet includes 4 kinds of
data: (a) the image erased on occluded portion of the right hand $I_{D}$, (b)
the modal mask of the right hand $M_{rv}$, (c) the image erased on redundant
portion of the distracting hand $I_{R}$, and (d) the modal mask of background
$M_{bv}$. HDRNet recovers the appearance content of the occluded parts and
inpaints the distracting hand to avoid ambiguity.

Hand De-occlusion and Removal Module (HDRM) aims at transforming a previously
challenging case of hand-hand interactions into a common single-hand case,
which can be easily solved by an off-the-shelf single-hand pose estimator.
Specifically, given amodal and modal masks, _De-occlusion_ is responsible for
recovering the appearance content or RGB values of the occluded regions, while
_Removal_ targets at inpainting the distracting regions in the image, reducing
the ambiguities caused by the homogeneous appearance of two hands.

For clarity, in the following sections, we will focus on the right hand only
and regard the left hand as the distractor. Note that, the left-hand centered
image can be flipped horizontally before performing hand de-occlusion,
removal, and pose estimation, thus following the same pipeline.

As shown in Fig. 2, for the right hand, we first use the amodal mask $M_{ra}$
to locate the right hand. Then we crop the original image and the segmentation
masks at the center of the right hand. The newly cropped image and masks are
denoted as $I_{s}^{crop}$, $M_{ra}^{crop}$, $M_{rv}^{crop}$, $M_{la}^{crop}$
and $M_{lv}^{crop}$ respectively. We will omit the superscript ${crop}$ in
subsequent sections for simplicity.

We use $M_{D}$ to denote the region where the target hand is occluded by the
other hand and $M_{R}$ to denote the region where the distracting hand
occupies. They are computed as follows:

$\displaystyle M_{D}=M_{ra}\cdot\left(1-M_{rv}\right),$ (2) $\displaystyle
M_{R}=(1-M_{ra})\cdot M_{lv}.$

$I_{D}$ and $I_{R}$ are the original image $I_{s}$ erased by the mask $M_{D}$
and $M_{R}$ respectively. They can inform the HDRNet where to focus and how to
inpaint these two regions with partial convolution [19]. In addition, the
modal mask of the right hand $M_{rv}$ and the modal mask of the background
$M_{bv}$ point out where the HDRNet can refer to for de-occlusion and removal
respectively. Formally, $I_{D}$, $I_{R}$ and $M_{bv}$ are computed as follows:

$\displaystyle I_{D}=I_{s}\cdot(1-M_{D}),$ (3) $\displaystyle
I_{R}=I_{s}\cdot(1-M_{R}),$ $\displaystyle M_{bv}=(1-M_{ra})\cdot(1-M_{la}).$

$I_{D}$, $M_{rv}$, $I_{R}$ and $M_{bv}$ are concatenated together as the
input, as shown in Fig. 3. HDRNet then uses these data to recover the
appearance content of the occluded parts and inpaints the distracting hand to
avoid ambiguity. For model architecture choice, we follow [38, 36] to adopt
the network of Liu et al. [19] and further improve it by adding a few
transformer blocks [32]. The transformer block enhances image feature
interactions, enlarges the receptive fields, and focuses more on important
image regions. Finally, the HDRNet outputs a recovered image $I_{o}$. We
follow [38] to employ an image discriminator [12] $D$ to enhance the image
recovery quality through adversarial training. The loss function of HDRNet is
as follows:

$\displaystyle\mathcal{L}_{HDR}=$
$\displaystyle\lambda_{1}(\mathbb{E}_{I_{o}}[\mathrm{log}(1-D(I_{o}))]+\mathbb{E}_{I_{o}^{*}}[\mathrm{log}(D(I_{o}^{*}))])+$
(4) $\displaystyle\lambda_{2}\mathcal{L}_{\ell
1}(I_{o},I_{o}^{*})+\lambda_{3}\mathcal{L}_{prec}(I_{o},I_{o}^{*})+\lambda_{4}\mathcal{L}_{style}(I_{o},I_{o}^{*}),$

where $\mathcal{L}_{prec}(*)$ denotes the perceptual loss [8], and
$\mathcal{L}_{style}(*)$ denotes the style loss [19]. $I_{o}$ is the recovered
image, while $I_{o}^{*}$ is its corresponding ground truth. $\lambda_{1}$,
$\lambda_{2}$, $\lambda_{3}$, and $\lambda_{4}$ are hyper-parameters to
balance the losses.

###  3.4 3D Single Hand Pose Estimation (SHPE)

Our de-occlusion and removal framework can be applied to any off-the-shelf
pose estimators. However, designing a more powerful hand pose estimation
network architecture is not the focus of this paper. In this work, we choose
the DetNet of MinimalHand [39] as our baseline SHPE for its simplicity and
good performance. MinimalHand [39] comprises two modules, i.e. DetNet and
IKNet. DetNet predicts the 2D and 3D hand joint positions. IKNet then takes as
input the predicted 3D hand joint positions and maps them to the joint angles.
In our implementation, we simply discard the IKNet and re-train the DetNet on
the InterHand2.6M dataset [21]. The loss function of SHPE is as follows:

$\displaystyle\mathcal{L}_{SHPE}=\mathcal{L}_{heat}+\mathcal{L}_{loc}+\mathcal{L}_{delta}+\mathcal{L}_{reg},$
(5)

where $\mathcal{L}_{heat}$ is the 2D heatmap loss. $\mathcal{L}_{loc}$ and
$\mathcal{L}_{delta}$ are location map loss and delta map loss respectively.
$\mathcal{L}_{reg}$ is a $\ell_{2}$ weight regularizer to avoid overfitting.
Please refer to Zhou et al. [39] for more training details.

##  4 Amodal InterHand (AIH) Dataset

Existing amodal perception datasets [40, 24, 10] mostly focus on amodal
segmentation of common objects (e.g. , vehicles, buildings, and indoor
objects). To the best of our knowledge, there exists no dataset that targets
at amodal segmentation and appearance content recovery of interactive hands.
To fill in this blank, we synthetically generate the first large-scale Amodal
InterHand dataset, namely AIH dataset. We envision that the proposed dataset
will boost the related research, e.g. amodal perception, de-occlusion, and
hand pose estimation.

Our AIH dataset is constructed based on the well-known InterHand2.6M V1.0
dataset [21]. As shown in Fig. 4, our proposed Amodal InterHand (AIH) dataset
consists of two parts: AIH_Syn and AIH_Render. In total, AIH dataset consists
of about 3 million images, where AIH_Syn contains 2.2M samples and AIH_Render
contains over 0.7M samples. AIH_Syn is generated by simple 2D image-level copy
and paste, i.e. copy the left single-hand image and paste it on the right
single-hand image; AIH_Render is obtained by rendering the textured
interacting hand mesh to the image plane. Both AIH_Syn and AIH_Render contain
the amodal and modal segmentation masks as well as the appearance content
ground-truths.

AIH_Syn We first get the hand mesh with the ground-truth MANO parameters of
the single-hand samples from the InterHand2.6M V1.0 dataset, and then project
it into the 2D image plane to get the amodal segmentation mask. Then, we
filter out some bad samples in which MANO parameters or the corresponding
image are not valid. As a result, we get over 250K cropped single-hand images
with masks for each side hand. To generate the interacting two-hand samples,
we randomly pick two hands with similar texture from both sides. Then we crop
the left hand region given its amodal mask and paste it on the right hand
image. Random scaling, rotation, and color jittering are applied to increase
diversity.

AIH_Render Although AIH_Syn provides plenty of amodal data, such 2D level
copy-and-paste can not generate mutual occlusion cases which are very common
for interacting hands. Therefore, based on MANO parameters of InterHand2.6M
V1.0 dataset, we decorate the corresponding hand mesh with random hand texture
[25], augment it with random translation and rotation, and finally render it
to a random background image from the dataset.

Figure 4: Visualization of our proposed Amodal InterHand (AIH) dataset.
AIH_Syn is obtained by simple 2D copy-and-paste, while AIH_Render is generated
by rendering the textured 3D interacting hand mesh to the image plane.

##  5 Experiments

###  5.1 Implementation Details

All experiments are conducted on 8 NVIDIA Tesla V100 GPUs. Training mini-batch
size is set as 48 and Adam [14] is adopted for model parameter tuning. Details
of HASM. Our HASM is trained for 200k iterations with a learning rate of
$2.5\times 10^{-3}$ following the training setting of SegFormer [33]. Details
of HDRM. Our HDRM has an input resolution of $256\times 256$. And the loss
weights are set as $\lambda_{1}=0.1$, $\lambda_{2}=3.0$, $\lambda_{3}=0.1$,
$\lambda_{4}=250.0$. We first train HDRM with ground-truth masks for 100k
iteration, and then fine-tune it with segmentation masks for another 100k
iteration. The learning rate of these two stages are $1.5\times 10^{-3}$ and
$1\times 10^{-3}$ respectively. Details of SHPE. Our SHPE has an input
resolution of $256\times 256$. We train the network for 300k iterations with
an initial learning rate of $1\times 10^{-3}$. The learning rate is decayed to
$1\times 10^{-4}$ and $1\times 10^{-5}$ at the 100k and 200k iterations
respectively. Other training settings are kept the same as those of
MinimalHand [39].

###  5.2 Datasets and Evaluation Metrics

Datasets. The experiments are conducted on InterHand2.6M V1.0 [21] dataset and
Tzionas dataset [30]. _InterHand2.6M V1.0 dataset_ [21] is a publicly
available large-scale realistic pose estimation dataset for two-hand
interactions. The dataset provides RGB images with semi-automatically
annotated 3D poses, and MANO [26] parameters obtained from NeuralAnnot [20].
In the experiments, we follow the common practice [21] to use the downsized
$512\times 334$ image resolution at 5 frames-per-second (FPS) version of the
released dataset. Following the official configurations, the dataset is split
into three branches, namely ‘H’ for the human annotation branch, ‘M’ for the
machine annotation branch, and ‘ALL’ for all data. The ‘M’ branch data
contains many unseen poses and more diverse sequences, which makes it more
similar to real-world scenarios. Moreover, we notice that the ‘H’ branch data
contains missing or incomplete mesh annotations. In the experiments, we
majorly conduct experiments on the ‘M’ branch, but also report the results on
‘ALL’ branch for comparisons. To focus on the interacting hands, the original
dataset [21] divides the whole dataset (IH26M-ALL) into single-hands subset
(IH26M-SH) and interacting-hands subset (IH26M-IH). Following [27], we further
select samples from the original “IH26M-IH” test set, and generate a more
challenging subset called “IH26M-Inter”. “IH26M-Inter” contains samples with
more than 30 valid ‘ground-truth’ 3D hand keypoints. Since InterHand2.6M [21]
is captured in a lab environment, its background diversity is relatively
limited. To evaluate the generalization ability, we perform qualitative and
quantitative experiments on the _Tzionas dataset_ [30]. Since Tzionas dataset
does not provide a separate training set, we directly use it as the testing
set to evaluate the model trained on the InterHand2.6M dataset [21].

Evaluation Metrics. For InterHand2.6M [21], we report 3D Mean Per Joint
Position Error (MPJPE). MPJPE is defined as the mean Euclidean distance
between ground truth and predicted 3D joint locations, calculated after
aligning the root joint for each left and right hand separately. The
measurements are reported in millimeters (mm). For Tzionas dataset [30], we
follow the common practice [21, 4, 13] to use 2D end point error (EPE) for
evaluation.

###  5.3 Comparisons with state-of-the-art methods

We compare with previous state-of-the-art pose estimation methods on the ‘ALL’
branch and the ‘machine_annot (M)’ branch of InterHand2.6M V1.0 dataset [21].
MPJPE (mm) is adopted to evaluate the 3D hand pose estimation accuracy. For
fair comparisons, the AIH dataset is only used to train HASM and HDRM for
amodal segmentation and de-occlusion. No pose annotations in AIH are used to
train the pose estimator (SHPE). Table 1 summarizes the experimental results.

We first compare performances of three single-hand methods, i.e. Boukhayma et
al. [4], Pose2Mesh [5] and BiHand [35]. Our approach significantly outperforms
all the state-of-the-art single-hand approaches. On the “IH26M-ALL” split,
compared with BiHand [35], our model reduces MPJPE from $25.10$mm to
$10.97$mm, resulting in as much as $56\%$ error reduction. And in the more
challenging “IH26M-Inter” split, our approach obtains about $47\%$ accuracy
improvement. This shows existing single-hand pose estimators do not handle
heavy hand-hand occlusions and are easily confused by the other distracting
hand.

We also compare with recent two-hand pose estimation approaches, i.e. InterNet
[21], Rong et al. [27], and DIGIT [7]. We show superior performance over these
3D interacting hand pose estimation systems. For example, our approach
significantly improves upon Moon et al. [21]’s state-of-the-art results from
$14.21$mm to $10.97$mm (about 23% error reduction) on the “IH26M-ALL” split.
The clear performance gap validates the effectiveness of our framework.
Overall, our approach consistently ranks the first across all evaluation
protocols.

Table 1: Comparisons with state-of-the-art methods on the ‘ALL’ branch and the
‘machine_annot (M)’ branch of InterHand2.6M V1.0 Dataset. MPJPE (mm) is
adopted to evaluate the 3D joint estimation accuracy. The results marked with
‘*’ are from [27].

Methods | InterHand2.6M - ALL branch | InterHand2.6M - M branch
---|---|---
IH26M-SH | IH26M-IH | IH26M-ALL | IH26M-Inter | IH26M-SH | IH26M-IH | IH26M-ALL
*Boukhayma et al. [4] | _ | _ | 27.14 | 31.46 | _ | _ | _
Pose2Mesh [5] | _ | _ | 27.10 | 32.11 | _ | _ | _
BiHand [35] | _ | _ | 25.10 | 28.23 | _ | _ | _
Rong et al. [27] | _ | _ | 17.12 | 20.66 | _ | _ | _
DIGIT [7] | _ | 14.27 | _ | _ | _ | _ | _
InterNet [21] | 12.16 | 16.02 | 14.21 | 18.04 | 12.52 | 18.04 | 15.28
HDR (Ours) | 8.51 | 13.12 | 10.97 | 14.74 | 8.52 | 14.98 | 11.74

Table 2: Comparisons with state-of-the-art methods on Tzionas dataset [30].
The results of other algorithms are from [13]. 2D EPE is adopted to evaluate
pose results.

Model | Boukhayma et al. [4] | Wang et al. [31] | InterNet [21] | Kim et al. [13] | SHPE | SHPE+HDR
---|---|---|---|---|---|---
EPE$\downarrow$ | 12.91 | 13.31 | 17.61 | 12.42 | 14.88 | 8.70

We also follow [13] to report hand pose estimation results (EPE) on Tzionas
dataset [30] in Table 2. Our method (SHPE+HDR in the table) significantly
improves upon the baseline SHPE, and outperforms the prior arts.

###  5.4 Effect of Hand De-occlusion and Removal (HDR) Framework

As shown in Table 3, and Table 4, we conduct experiments on the ‘machine_annot
(M)’ branch and the ‘ALL’ branch of InterHand2.6M V1.0 dataset respectively.
We compare the results with or without using our HDR framework. We notice that
the recent state-of-the-art single-hand pose estimation (SHPE) method
(MinimalHand [39]) struggles with occlusions and appearance ambiguity in
interacting hand scenarios (IH26M-IH). To tackle these challenges, we propose
a novel Hand De-occlusion and Removal (HDR) framework to perform hand de-
occlusion and distractor removal. In Table 3, we show that our approach
significantly improves upon the SHPE baseline in interacting hand scenarios,
e.g. from 40.98mm to 25.45mm (M, IH26M-IH). We find that adding AIH dataset
for training will further improve the performance of SHPE, which validates the
effect of AIH dataset. Experiments on the ‘ALL’ branch have a similar
phenomenon.

Table 3: Effect of HDR Framework. Experiments are conducted on the
‘machine_annot (M)’ branch of InterHand2.6M V1.0 dataset. MPJPE (mm) is
adopted to evaluate the 3D joint estimation accuracy.

Methods | Train (M, IH26M-SH) | Train (M, IH26M-SH +AIH)
---|---|---
IH26M-IH | IH26M-ALL | IH26M-IH | IH26M-ALL
SHPE [39] | 40.98 | 25.78 | 32.27 | 21.66
+HDR (Ours) | 25.45 | 17.98 | 24.59 | 17.80

Table 4: Effect of HDR Framework. Experiments are conducted on the ‘ALL’
branch of InterHand2.6M V1.0 dataset. MPJPE (mm) is adopted to evaluate the 3D
joint estimation accuracy.

Methods | Train (ALL, IH26M-SH) | Train (ALL, IH26M-SH +AIH)
---|---|---
IH26M-IH | IH26M-ALL | IH26M-IH | IH26M-ALL
SHPE [39] | 39.96 | 25.90 | 30.23 | 20.93
+HDR (Ours) | 25.93 | 18.39 | 23.99 | 17.58

###  5.5 Ablation Study

Table 5: Ablation Studies. Experiments are conducted on the ‘machine_annot
(M)’ branch of InterHand2.6M V1.0 dataset. We use MPJPE (mm) to evaluate the
3D joint estimation accuracy. $\Delta$ means the absolute (and relative)
difference compared with our final model #8. ‘w/o’ is short for ‘without’.

| Methods | MPJPE (mm) | $\Delta$
---|---|---|---
#1 | SHPE [39] only | 25.78 | +7.80 (43.4%)
#2 | w/o Removal | 24.16 | +6.18 (34.4%)
#3 | w/o De-occlusion | 19.69 | +1.71 (9.5%)
#4 | w/o Discriminator | 18.11 | +0.13 (0.7%)
#5 | w/o Transformer Block | 18.85 | +0.87 (4.8%)
#6 | AIH_Render only | 18.10 | +0.12 (0.7%)
#7 | AIH_Syn only | 18.35 | +0.37 (2.1%)
#8 | Ours | 17.98 | -

In this section, we conduct ablation studies to evaluate the effectiveness of
the key components of our approach on the ‘machine annot (M)’ branch of
InterHand2.6M V1.0 dataset [21]. For fair comparisons, in all the ablation
experiments, we use the same SHPE [39] trained on the IH26M-SH set.

Analysis of Hand De-occlusion and Removal (HDR) Module. There are two major
challenges of hand pose estimation in interacting scenarios, i.e. severe self-
occlusion, and ambiguity caused by the homogeneous appearance of hands. As
shown in Table 5, #1, #2, #3, and #8, we conduct ablative experiments to
quantitatively evaluate the effect of De-occlusion and Removal. Comparing #2
and #8, we observe that disabling ‘Removal’ will dramatically increase the
MPJPE by 34.4%. Comparing #3 and #8, we see that disabling ‘De-occlusion’
increases the MPJPE by 9.5%. If we only apply SHPE [39] without HDR, the
errors are further increased. These results clearly show that (1) state-of-
the-art SHPE [39] is sensitive to self-occlusions and inter-hand ambiguities
(2) HDRM is effective in handling the aforementioned two major challenges.

Analysis of Model Design Choices. We empirically validate the model design
choice of HDRNet, especially _Discriminator_ and the _Transformer_ block.
Discriminator is applied to enhance the image recovery quality by adversarial
training. Comparing #4 and #8 in Table 5, we observe that although
Discriminator helps in improving the quality of the recovered image, its
influence on the final results is only marginal (0.7%). The Transformer block
enhances image feature interactions, enlarges the receptive fields, and
focuses more on important image regions. Comparing #5 and #8 in Table 5, we
see that without using the Transformer block impacts the final results by a
clear margin (4.8%).

Analysis of AIH_Syn and AIH_Render. Our proposed AIH dataset is composed of
two subsets, namely AIH_Syn and AIH_Render. Both have their own advantages and
disadvantages. AIH_Syn retains more detailed and realistic appearance
features, while AIH_Render considers the inter-dependencies between two hands
to avoid physically implausible configurations. Using a combination of these
two sets to train the HDRNet will achieve the best performance. In Table 5,
comparing #6, #7, and #8, we compare different training settings for HDRNet.
We notice that it already achieves reasonably good results even if we only use
one of the two sets. For example, using “AIH_Render only” to train HDRNet, we
can achieve $18.10$ MPJPE (mm), which is only marginally worse than the final
model #8. We also empirically find that “AIH_Render” seems to have a larger
impact on the final results than “AIH_Syn” does.

###  5.6 Time Complexity Analysis

We analyze the time cost on one Tesla P40 GPU in a single thread. On average,
HASM, HDRM, and SHPE take 12.6 ms, 0.6 ms, and 34.0 ms per frame (including
two hands) respectively. The time cost of HDRM (our major contribution) is
only a small proportion of the total time cost (0.6 vs 47.2 ms).

###  5.7 Qualitative Results

Figure 5: Qualitive results of how HDR helps in handling severe hand-hand
occlusion and appearance ambiguities of two hands. Best viewed in color.

In Fig. 5, we provide qualitative analysis on InterHand2.6M [21] and Tzionas
dataset [30] to illustrate how HDR helps in handling severe hand-hand
occlusion and the homogeneous appearance of hands. We see that HDR recovers
the appearance in the occluded region and removes the distractor in
challenging hand-hand occlusion cases. The results on Tzionas dataset [30]
further validates the generalization ability of our proposed framework.

##  6 Conclusions and Limitations

Interacting hand pose estimation is important but challenging due to severe
hand-to-hand occlusion and ambiguity caused by the other distracting hand. In
this paper, we propose to decompose the task into two relatively simple sub-
tasks, i.e. (1) Hand De-occlusion and Removal (HDR), (2) Single Hand Pose
Estimation (SHPE). Through HDR, we can simplify the case, which an off-the-
shelf SHPE can handle. We empirically verified the effectiveness of our HDR
framework on the InterHand2.6M and Tzionas dataset. Our limitations mainly lie
in artifacts produced by HDRM. Improving the image recovery quality requires
efforts in various research fields, which we will explore in the future.

Acknowledgement. We would like to thank Wentao Jiang, Wang Zeng, Neng Qian,
Yumeng Hu, Lixin Yang, Yu Rong, Qiang Zhou and Jiayi Wang for their helpful
discussions and feedback. Mengxiang Lin is supported by State Key Laboratory
of Software Development Environment under Grant No SKLSDE 2022ZX-06. Ping Luo
is supported by the General Research Fund of HK No.27208720, No.17212120, and
No.17200622. Wanli Ouyang is supported by the Australian Research Council
Grant DP200103223, Australian Medical Research Future Fund MRFAI000085, CRC-P
Smart Material Recovery Facility (SMRF) – Curby Soft Plastics, and CRC-P ARIA
- Bionic Visual-Spatial Prosthesis for the Blind.

## References

  * [1] Baek, S., Kim, K.I., Kim, T.K.: Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1067–1076 (2019)
  * [2] Baek, S., Kim, K.I., Kim, T.K.: Weakly-supervised domain adaptation via gan and mesh model for estimating 3d hand poses interacting objects. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
  * [3] Ballan, L., Taneja, A., Gall, J., Gool, L.V., Pollefeys, M.: Motion capture of hands in action using discriminative salient points. In: Eur. Conf. Comput. Vis. pp. 640–653. Springer (2012)
  * [4] Boukhayma, A., Bem, R.d., Torr, P.H.: 3d hand shape and pose from images in the wild. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 10843–10852 (2019)
  * [5] Choi, H., Moon, G., Lee, K.M.: Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In: Eur. Conf. Comput. Vis. (2020)
  * [6] Ehsani, K., Mottaghi, R., Farhadi, A.: Segan: Segmenting and generating the invisible. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 6144–6153 (2018)
  * [7] Fan, Z., Spurr, A., Kocabas, M., Tang, S., Black, M., Hilliges, O.: Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation. In: International Conference on 3D Vision (3DV) (2021)
  * [8] Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2414–2423 (2016)
  * [9] Han, S., Liu, B., Cabezas, R., Twigg, C.D., Zhang, P., Petkau, J., Yu, T.H., Tai, C.J., Akbay, M., Wang, Z., et al.: Megatrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM Trans. Graph. 39(4), 87–1 (2020)
  * [10] Hu, Y.T., Chen, H.S., Hui, K., Huang, J.B., Schwing, A.G.: Sail-vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3105–3115 (2019)
  * [11] Iqbal, U., Molchanov, P., Gall, T.B.J., Kautz, J.: Hand pose estimation via latent 2.5 d heatmap regression. In: Eur. Conf. Comput. Vis. pp. 118–134 (2018)
  * [12] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1125–1134 (2017)
  * [13] Kim, D.U., Kim, K.I., Baek, S.: End-to-end detection and pose estimation of two interacting hands. In: Int. Conf. Comput. Vis. pp. 11189–11198 (2021)
  * [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [15] Kulon, D., Guler, R.A., Kokkinos, I., Bronstein, M.M., Zafeiriou, S.: Weakly-supervised mesh-convolutional hand reconstruction in the wild. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 4990–5000 (2020)
  * [16] Li, K., Malik, J.: Amodal instance segmentation. In: Eur. Conf. Comput. Vis. pp. 677–693. Springer (2016)
  * [17] Lin, F., Wilhelm, C., Martinez, T.: Two-hand global 3d pose estimation using monocular rgb. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2373–2381 (2021)
  * [18] Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1954–1963 (2021)
  * [19] Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Eur. Conf. Comput. Vis. (2018)
  * [20] Moon, G., Lee, K.M.: Neuralannot: Neural annotator for in-the-wild expressive 3d human pose and mesh training sets. arXiv preprint arXiv:2011.11232 (2020)
  * [21] Moon, G., Yu, S.I., Wen, H., Shiratori, T., Lee, K.M.: Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In: Eur. Conf. Comput. Vis. pp. 548–564. Springer (2020)
  * [22] Mueller, F., Davis, M., Bernard, F., Sotnychenko, O., Verschoor, M., Otaduy, M.A., Casas, D., Theobalt, C.: Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Trans. Graph. 38(4), 1–13 (2019)
  * [23] Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Tracking the articulated motion of two strongly interacting hands. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1862–1869. IEEE (2012)
  * [24] Qi, L., Jiang, L., Liu, S., Shen, X., Jia, J.: Amodal instance segmentation with kins dataset. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3014–3023 (2019)
  * [25] Qian, N., Wang, J., Mueller, F., Bernard, F., Golyanik, V., Theobalt, C.: HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization. In: Eur. Conf. Comput. Vis. Springer (2020)
  * [26] Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 (2022)
  * [27] Rong, Y., Wang, J., Liu, Z., Loy, C.C.: Monocular 3d reconstruction of interacting handsvia collision-aware factorized refinements. In: International Conference on 3D Vision (2021)
  * [28] Smith, B., Wu, C., Wen, H., Peluse, P., Sheikh, Y., Hodgins, J.K., Shiratori, T.: Constraining dense hand surface tracking with elasticity. ACM Trans. Graph. 39(6), 1–14 (2020)
  * [29] Spurr, A., Song, J., Park, S., Hilliges, O.: Cross-modal deep variational hand pose estimation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 89–98 (2018)
  * [30] Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. Int. J. Comput. Vis. 118(2), 172–193 (2016)
  * [31] Wang, J., Mueller, F., Bernard, F., Sorli, S., Sotnychenko, O., Qian, N., Otaduy, M.A., Casas, D., Theobalt, C.: Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Trans. Graph. 39(6), 1–16 (2020)
  * [32] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media 8(3), 1–10 (2022)
  * [33] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inform. Process. Syst. 34 (2021)
  * [34] Yan, X., Wang, F., Liu, W., Yu, Y., He, S., Pan, J.: Visualizing the invisible: Occluded vehicle segmentation and recovery. In: Int. Conf. Comput. Vis. pp. 7618–7627 (2019)
  * [35] Yang, L., Li, J., Xu, W., Diao, Y., Lu, C.: Bihand: Recovering hand mesh with multi-stage bisected hourglass networks. In: Brit. Mach. Vis. Conf. (2020)
  * [36] Zhan, X., Pan, X., Dai, B., Liu, Z., Lin, D., Loy, C.C.: Self-supervised scene de-occlusion. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3784–3792 (2020)
  * [37] Zhang, B., Wang, Y., Deng, X., Zhang, Y., Tan, P., Ma, C., Wang, H.: Interacting two-hand 3d pose and shape reconstruction from single color image. In: Int. Conf. Comput. Vis. pp. 11354–11363 (2021)
  * [38] Zhou, Q., Wang, S., Wang, Y., Huang, Z., Wang, X.: Human de-occlusion: Invisible perception and recovery for humans. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3691–3701 (2021)
  * [39] Zhou, Y., Habermann, M., Xu, W., Habibie, I., Theobalt, C., Xu, F.: Monocular real-time hand shape and motion capture using multi-modal data. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5346–5355 (2020)
  * [40] Zhu, Y., Tian, Y., Metaxas, D., Dollár, P.: Semantic amodal segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1464–1472 (2017)
  * [41] Zimmermann, C., Brox, T.: Learning to estimate 3d hand pose from single rgb images. In: Int. Conf. Comput. Vis. pp. 4903–4911 (2017)

## Appendix

##  Appendix 0.A Video Demo

To justify the generalization ability and the potential of our proposed method
in real-world applications, we run our approach on several video clips from
the Tzionas dataset [30]. Note that our models are only trained on
InterHand2.6M V1.0 dataset [21] and our proposed AIH dataset. Tzionas dataset
[30] is _unseen_ during training.

In the video demo111Our video demo can be downloaded from
https://connecthkuhk-
my.sharepoint.com/:v:/g/personal/js20_connect_hku_hk/EW_S3kZu97xPlMk_HQLAJVMBtizU48sGh4jXwvUuyugFRw?e=u2GB6I.,
we compare our approach with ‘Baseline’ which is a Single-Hand Pose Estimator
(SHPE). For fair comparisons, both ‘Ours’ and ‘Baseline’ employ the same SHPE
[39] trained on the ‘ALL’ branch of InterHand2.6M V1.0 dataset [21]. Note that
this model is the same as the one used in Sec. 5.6 (Fig. 5) of the main paper.
The pose results are directly obtained from the output of the SHPE model
without temporal smoothing.

We first visualize the predicted amodal/visible mask of both hands. Given the
segmentation mask, we obtain the corresponding single-hand box (‘red’ for the
right hand, and ‘green’ for the left hand). We also demonstrate the results of
Hand De-occlusion and Removal Module (HDRM). In order to tackle the severe
hand-hand occlusion problem, HDRM applies the hand de-occlusion technique to
recover the appearance (texture) in the occluded region. In the meanwhile, it
also removes (inpaints) the distracting hand to handle the ambiguity caused by
the homogeneous appearance of hands. Finally, our approach obtains better 3d
hand pose estimation results.

Although the quality of the image recovery is satisfactory in most cases,
there are still some problems in some difficult situations. For example, the
boundary of the hand segmentation can be over-smoothed, leading to undesirable
artifacts around the hand. This problem can be mitigated by applying an
advanced amodal/visible mask segmentation model, which we will explore in the
future.

##  Appendix 0.B More Examples of AIH Dataset

In this section, we present more examples of our proposed Amodal InterHand
(AIH) dataset. Our AIH dataset consists of two parts: AIH_Syn and AIH_Render.
AIH_Syn is constructed by copy-and-paste while AIH_Render is constructed by
rendering the textured interacting hand mesh to the image plane. As shown in
Fig A1, both AIH_Syn and AIH_Render have great diversity in hand poses,
textures, occlusion and interaction types.

Figure A1: Top: More examples of our proposed AIH_Syn dataset. Bottom: More
examples of our proposed AIH_Render dataset.