#  TEMP3D: Temporally Continuous 3D Human Pose Estimation Under Occlusions

Rohit Lal1,∗ Yash Garg1,∗ Arindam Dutta1,∗ Calvin-Khang Ta1 Dripta S.
Raychaudhuri2
M. Salman Asif1 Amit K. Roy-Chowdhury1
1University of California, Riverside $\quad\quad{}^{2}$AWS AI Labs
{rlal011, ygarg002, adutt020, cta003, drayc001, sasif<EMAIL_ADDRESS>
###### Abstract

Existing 3D human pose estimation methods perform remarkably well in both
monocular and multi-view settings. However, their efficacy diminishes
significantly in the presence of heavy occlusions, which limits their
practical utility. For video sequences, temporal continuity can help infer
accurate poses, especially in heavily occluded frames. In this paper, we aim
to leverage this potential of temporal continuity through human motion priors,
coupled with large-scale pre-training on 3D poses and self-supervised
learning, to enhance 3D pose estimation in a given video sequence. This leads
to a temporally continuous 3D pose estimate on unlabelled in-the-wild videos,
which may contain occlusions, while exclusively relying on pre-trained 3D pose
models. We propose an unsupervised method named TEMP3D that aligns a motion
prior model on a given in-the-wild video using existing SOTA single image-
based 3D pose estimation methods to give temporally continuous output under
occlusions. To evaluate our method, we test it on the Occluded Human3.6M
dataset, our custom-built dataset which contains significantly large (up to
100%) human body occlusions incorporated into the Human3.6M dataset. We
achieve SOTA results on Occluded Human3.6M and the OcMotion dataset while
maintaining competitive performance on non-occluded data.
https://sites.google.com/ucr.edu/temp3d

Figure 1: Current single image-based 3D pose estimation methods fail to
consistently produce clean, accurate human body pose predictions in cases of
heavy occlusion. This discrepancy arises from a lack of temporal contextual
information, as illustrated in the third videoframe of the figure. In this
frame, the pose produced by the single image-based method [3] is temporally
inconsistent with the previous poses. Indeed, when the image contains heavy
human body occlusion which render the person almost entirely invisible, such
as in the fourth and fifth frames, the single image based method fails to
predict any pose at all. However, the incorporation of temporal continuity
enables the refinement of noisy, inconsistent poses as well as accurate pose
predictions on frames where single image based methods fail to perform.

##  1 Introduction

Accurate 3D pose estimation [49] is an important problem in computer vision
with a variety of real-world applications, including but not limited to action
recognition [20], virtual and augmented reality [1], and gait recognition
[51]. Single image-based 3D pose estimation methods [37, 34] have shown
impressive performance gains in inferring the three-dimensional pose of the
subject by only leveraging the information contained in a single image frame.
Naturally, such algorithms struggle under scenarios of occlusions. Video based
pose estimation algorithms [45, 50] leverage temporal continuity to infer
poses even in frames that are heavily occluded. However, these algorithms are
extremely slow, impeding their deployment in scenarios necessitating real-time
performance. The success of current state-of-the-art algorithms [3] can
primarily be attributed to supervised training on datasets captured under
controlled settings [3]. Naturally, the performance of these algorithms on in-
the-wild videos is glaringly sub-optimal. This sub-optimal performance can be
attributed to two reasons: firstly, in-the-wild videos exhibit distribution
shifts [23] from the training data for these algorithms and secondly, in-the-
wild videos may contain prolonged instances of natural occlusions. While the
recent work by Black et. al. [3], largely addresses the problem of
distribution shifts by training models like HMR [14] and CLIFF [26] on large
amounts of synthetic data, the problem of sub-optimal performance under
prolonged occlusions still persists. Thus, we propose to leverage temporal
continuity to improve the performance of the models proposed in [3] under
scenarios of prolonged occlusions in in-the-wild captured videos.

To illustrate the issue of prolonged occlusions, consider a monocular video
scenario in which an individual who is walking in a forest undergoes complete
occlusion by a tree. A single image-based 3D pose estimation model [3] fails
to estimate the pose of the occluded person, due to the aforementioned lack of
temporal context [32, 33, 42, 31]. This is qualitatively illustrated in Fig.
1.

To effectively exploit this temporal motion information, numerous video-based
pose estimation methods have been developed [45, 50]. However, two common
limitations of these approaches are the heavy dependence on annotated data
sets and the assumption that occlusion only sporadically affects single frames
[36, 16, 10, 24]. In reality, occlusion events can persist across multiple
frames, presenting a considerable challenge for accurate pose estimation [5].
Another drawback of these existing video-based pose estimation methods is the
limited effectiveness of their performance on in-the-wild videos, which arises
due to domain shifts between training datasets and testing videos [21, 22,
23]. Moreover, methods such as [8, 9] assume access to the entire set of
labelled training data during inference. This assumption introduces challenges
related to privacy and storage issues.

As a result, we surpass the performance of single-image-based 3D pose
estimation methods by incorporating temporal context into the 3D pose
estimation process. We also outperform video based methods as all of these
algorithm use controlled datasets for training, thereby failing to generalize
to in-the-wild videos. Unlike video-based approaches such as [8, 9], our
approach operates under the constraint of not having access to labelled
training data during inference time, making it privacy [38] and storage-
friendly. Our setting is similar to the setup in Deep Video Prior [25] which
only trains on video after independently applying an image-level algorithm to
each frame of an input video directly to bring in temporal continuity.

TEMP3D particularly outperforms other pose estimation methods when presented
with large occlusions. It accommodates scenarios which contain up to 100%
human body occlusions spanning across a significant number of consecutive
frames. The versatility of our method is further underscored by its inherent
agnosticism to pose estimation models. This compatibility enables TEMP3D to
utilize the most updated pose estimation models, thereby improving temporal
continuity without necessitating additional training. Recent state-of-the-art
methods [52, 3] report results on datasets which contain sporadic or no
occlusions at all. To address this dataset occlusion deficiency, we created an
artificial dataset, Occluded Human3.6M, based on Human3.6M [13], which is
specifically designed to challenge the existing state-of-the-art methods in
handling continuous occlusions. This dataset allows us to comprehensively
evaluate and compare the performance of current state-of-the-art techniques
against our proposed methods, providing insights into their robustness in
scenarios where occlusion is persistent and prevalent.

To summarise, in this paper we make the following key contributions:

  1. 1.

We introduce TEMP3D, a method which achieves temporally continuous 3D pose
estimation under heavy occlusions. We do not assume any access to labelled
data and our method works for a single in-the-wild video.

  2. 2.

TEMP3D seamlessly integrates temporal continuity into existing state-of-the-
art single image-based 3D pose estimation methods during test time, thereby
enhancing their performance in occluded video data.

  3. 3.

We introduce Occluded Human3.6M, a human motion capture dataset built off of
Human3.6M which contains significant occlusions on human subjects. This
dataset is specifically designed to challenge the existing state-of-the-art
methods in handling continuous occlusions.

  4. 4.

We perform experiments on challenging datasets such as Occluded Human3.6M and
OCMotion [46, 11], demonstrating state-of-the-art performance, and achieve
comparable results on non-occluded data.

Figure 2: The presented figure illustrates our temporally continuous pose
estimation or TEMP3D pipeline. Initially, we pre-train a motion prior model,
denoted as $\mathcal{M}$, using a diverse set of 3D pose data sourced from
various public datasets. The primary objective of this motion-prior model is
to generate a sequence of poses that exhibit temporal continuity when provided
with a sequence of initially noisy poses. Moving into the single video
training stage, we acquire a sequence of noisy poses using a 3D pose
estimation model, $\mathcal{P}$. The weights of $\mathcal{P}$ are held
constant during this phase. Subsequently, we pass this noisy pose sequence
through the motion prior model $\mathcal{M}$ and retrain it using various
supervised losses, as outlined in Eq. 10. The end result of this training
process is a model capable of producing temporally continuous 3D poses for
that specific video.

##  2 Related Works

Monocular 3D Pose Estimation. Monocular 3D Pose Estimation is a fundamental
and challenging problem in computer vision which involves the localisation of
3D spatial pose coordinates from single images. The problem is inherently
complex due to the diversity of body shapes, clothing, self-occlusions, etc.
Despite these bottlenecks, recent deep learning based methods have shown
impressive performance on challenging academic datasets [3, 49]. The
pioneering work by Mehta et. al. [29] proposed the first CNN based approach to
regress 3D joints from a single RGB image in an end-to-end fashion. Since
then, numerous works such as [37, 34] have improved upon these ideas by using
additional information such as multi-view constraints and depth information.
More recent works such as [43] employ kinematic constraints for improved pose
estimation, while [44] uses anatomical constraints and data augmentation for
obtaining state-of-the-art results on academic datasets. However, it is worth
noting that methods are trained under supervised settings and often fail to
generalize under distribution shifts. This motivated works such as [21, 22,
23] to propose self-supervised algorithms for 3D human pose estimation.
However, these methods also failed to generalise under occluded cases and lack
temporal continuity.

2D-3D Human Pose Lifting. Modern 3D human pose estimation encounters
significant challenges in generalization due to limited labeled data for real-
world applications. Martinez et. al. [28] addressed this issue by breaking
down the problem into 2D pose estimation and 2D to 3D lifting. Subsequent
studies improved on this, including self-supervised geometric regularization
by Chen et. al. [4], synthetic data usage Zhu et. al. [53], grid convolutions
Kang et. al. [15], spatio-temporal transformers Zheng et. al. [48], and
frequency domain analysis Zhao et. al. [47]. Zhu et. al. [52] achieved state-
of-the-art results by modelling motion priors from a sequence of 2D poses.

Video Based 3D pose estimation. Video based 3D human pose estimation have
demonstrated impressive performance gains on challenging datasets. Authors of
[45] proposed a mixed spatio-temporal approach for 3D human pose regression
which alternated between spatial consistency and temporal consistency. [2]
exploited temporal consistency for improved human pose estimation in the wild.
[50] proposed direct regression to 3D human poses by employing consistency
between 3D joints and 2D keypoints, and [35] proposed the use of temporal
convolutions for 3D human pose estimation in videos. Despite their success,
these are supervised learning algorithms, which necessitate large quantities
of labelled data.

##  3 Proposed Methodology

###  3.1 Problem statement

Given a single in-the-wild video potentially containing heavy occlusions, our
objective is to extract temporally continuous 3D poses. Additionally, we are
provided with a state-of-the-art monocular 3D pose estimation model, denoted
as $\mathcal{P}$. However, this model generates a sequence of poses that is
noisy, primarily due to occlusion or domain gap issues.

The goal of this research is to refine the noisy pose sequences produced by
$\mathcal{P}$. We aim to provide cleaned-up, temporally continuous 3D poses
that closely resemble natural human motion. Importantly, this task is
undertaken without assuming any availability of training data. Fig. 2
illustrates the entire pipeline.

Our proposed method consists of two stages of training. Namely, a Large Pre-
training (LPT) stage and a Single Video Training (SVT) stage, elaborated as
follows:

  * •

In the large pre-training stage, we train a self-attention-based motion prior
model $\mathcal{M}$ that takes noisy 3D joints $X$ as inputs and outputs a
sequence of temporally continuous 3D joints. The large pre-training is done
with BERT style [7, 52] approach using labelled data so that the model is able
to learn a human motion prior.

  * •

For the second stage, we first obtain a sequence of 3D poses for a video by
using $\mathcal{P}$ [3]. This sequence is inherently noisy due to the presence
of occlusions and domain gap. Next, we use the motion prior model
$\mathcal{M}$ and align it in an unsupervised fashion to obtain temporally
continuous pose estimates for the given video.

In Section 3.2, we have described the architecture used for pertaining and
then introduced the self-supervised lossses used for self-training.

###  3.2 Network Architecture

We represent the input sequence of 3D poses each of dimension $J\times 3$,
$X\in\mathbb{R}^{T\times J\times 3}$. Here $T$ represents a time and $J$
represents a number of joints. We modify the DSTFormer [52] architecture for
sequence-to-sequence modelling. The main role of the DSTFormer is to convert a
sequence of noisy poses to temporally continuous poses. Originally DSTFormer
was trained for uplifting 2D poses to 3D poses. In our case, we change the
input dimension to incorporate the 3D motion of the sequence. The DSTFormer
architecture hence outputs sequences $\hat{X}\in\mathbb{R}^{T\times J\times
3}$ when the input is sequence of noisy 3D poses $X\in\mathbb{R}^{T\times
J\times 3}$

#### Spatial Block.

The Spatial Multi-Head Self-Attention (S-MHSA) mechanism, denoted as
$\text{S-MHSA}(\mathbf{Q}_{\text{S}},\mathbf{K}_{\text{S}},\mathbf{V}_{\text{S}})$,
is a key component aimed at capturing relationships among joints within the
same time step in a given sequence. The formulation, as expressed in Equation
1, involves multiple attention heads ($h$) and a projection parameter matrix
$\mathbf{W}_{\text{S}}^{P}$. Each attention head, denoted as
$\text{head}_{i}$, applies the softmax function to the scaled dot-product of
query $\mathbf{Q}_{\text{S}}^{i}$ and key $\mathbf{K}_{\text{S}}^{i}$,
followed by multiplication with the corresponding value
$\mathbf{V}_{\text{S}}^{i}$.

$\begin{split}\small\text{S-MHSA}(\mathbf{Q}_{\text{S}},\mathbf{K}_{\text{S}},\mathbf{V}_{\text{S}})=[\text{head}_{1};...;\text{head}_{h}]\mathbf{W}_{\text{S}}^{P},\\\
\small\text{head}_{i}=\text{softmax}(\frac{\mathbf{Q}_{\text{S}}^{i}(\mathbf{K}_{\text{S}}^{i})^{\prime}}{\sqrt{d_{K}}})\mathbf{V}_{\text{S}}^{i},\end{split}$
(1)

We apply S-MHSA [40] to features of different time steps in parallel. The
output undergoes further processing, including residual connection and layer
normalization (LayerNorm), followed by a multilayer perceptron (MLP).

#### Temporal Block.

The Temporal Multi-Head Self-Attention (T-MHSA) focuses on capturing
relationships across time steps for a body joint. It operates similarly to
S-MHSA but is applied to per-joint temporal features parallelized over the
spatial dimension. The computation is expressed as:

$\begin{split}\small\text{T-MHSA}(\mathbf{Q}_{\text{T}},\mathbf{K}_{\text{T}},\mathbf{V}_{\text{T}})=[\text{head}_{1};...;\text{head}_{h}]\mathbf{W}_{\text{T}}^{P},\\\
\small\text{head}_{i}=\text{softmax}(\frac{\mathbf{Q}_{\text{T}}^{i}(\mathbf{K}_{\text{T}}^{i})^{\prime}}{\sqrt{d_{K}}})\mathbf{V}_{\text{T}}^{i},\end{split}$
(2)

#### Dual-Stream Spatio-temporal Transformer.

We then use the dual-stream architecture employing spatial and temporal Multi-
Head Self-Attention mechanisms. These mechanisms capture intra-frame and
inter-frame body joint interactions, necessitating careful consideration of
three key assumptions: both streams model comprehensive spatio-temporal
contexts, each stream specializes in distinct spatio-temporal aspects, and the
fusion dynamically balances weights based on input characteristics.

###  3.3 Large Pre-training Stage

We draw inspiration from the success of large-scale pre-training in Natural
Language Processing (NLP) models, such as BERT [7]. In our work, we extend
this approach to the domain of 3D pose estimation. Similar to [6, 52], our
model is pre-trained to predict temporally continuous poses when provided with
a sequence of noisy and masked input poses.

The pre-training phase plays a crucial role in achieving efficient parameter
initialization. This initialization expedites convergence because the model
has the human motion prior. This is extremely valuable when we are given only
a single in-the-wild video. We can observe that stage 2 training (Section 3.4)
converges stably and quickly to a lower loss value. The benefits of pre-
training are thus evident not only in enriching the model’s understanding of
human motion but also in enhancing its efficiency and performance in real-
world applications.

We employ both joint-level and frame-level masks with specific probabilities.
These corruptions mimic scenarios where a pre-trained model produces imperfect
3D predictions or encounters detection issues due to occlusion. Subsequently,
we feed this noisy $\mathbf{X}$ through the DSTFormer [52] and obtain the
reconstructed 3D motion $\mathbf{\hat{X}}$. The training process involves
minimizing the joint loss $\mathcal{L}_{\text{3D}}$ between $\mathbf{\hat{X}}$
and the ground truth (GT) 3D motion $\mathbf{X}$. Additionally, we incorporate
a velocity loss $\mathcal{L_{O}}$ following established methodologies [35,
45]. The 3D reconstruction losses are thus expressed by

$\mathcal{L}_{\text{3D}}=\sum\limits_{t=1}^{T}\sum\limits_{j=1}^{J}\parallel\mathbf{\hat{X}}_{t,j}-\mathbf{X}_{t,j}\parallel_{2}$
(3)
$\mathcal{L}_{\text{O}}=\sum\limits_{t=2}^{T}\sum\limits_{j=1}^{J}\parallel\mathbf{\hat{O}}_{t,j}-\mathbf{O}_{t,j}\parallel_{2},$
(4)

where $\mathbf{\hat{O}}_{t}=\mathbf{\hat{{X}}}_{t}-\mathbf{\hat{X}}_{t-1}$,
$\mathbf{O}_{t}=\mathbf{X}_{t}-\mathbf{X}_{t-1}$.

Please note that we only use the same motion prior model $\mathcal{M}$ as an
initialisation for any in-the-wild video and hence we do not need the training
data anymore.

###  3.4 Training for Single Video

Once we have a pre-trained motion prior model $\mathcal{M}$ that takes noisy
3D poses and gives a sequence of temporally continuous poses, we aim to use
any existing 3D pose estimation model $\mathcal{P}$ [3] to obtain a sequence
of 3D poses. These models will produce a sequence of noisy and temporally
inconsistent outputs as the video may contain occlusions and suffer from the
domain shift problem. To obtain a cleaner sequence of poses, we pass the noisy
3D poses to the motion prior model. This model then outputs a sequence of
clean poses as it was trained to give temporally continuous poses. Even though
just denoising motion using motion prior $\mathcal{M}$ gave competitive
results (Table 3), there were still some pose inconsistencies and loss of
temporal motion smoothness. We, therefore, propose self-training of
$\mathcal{M}$ that further improves this temporal inconsistency by using some
regularization techniques. We propose to do this self-training during the test
time for a single video to bring in geometric and physics-based constraints.
This is similar to works like Deep Video Prior [25] that use internal learning
for one video. In this work we experiment with three such losses that regulate
(1) the velocity of joints, (2) the size of limbs, and (3) the smoothness of
poses in missing frames. Please note that we only update the motion prior
model $\mathcal{M}$ and keep the weights of 3D pose estimation model
$\mathcal{P}$ fixed throughout.

#### Limb Loss:

Limb length consistency is an important aspect of anatomically plausible 3D
human pose predictions. This loss encourages the model to produce temporally
stable limb lengths, contributing to more realistic and physically plausible
pose estimations. The idea is to penalize variability in limb lengths across
frames. If the limb lengths exhibit large variations, it may indicate
inconsistency or instability in the predicted poses. The Limb Loss function
$\mathcal{L}_{lim}$ can be expressed below

$\mathcal{L}_{lim}=\frac{1}{J}\sum_{j=1}^{J-1}\underbrace{\frac{1}{T}\sum_{t=1}^{T}\left(\mathcal{J}_{t,j}-\frac{1}{T}\sum_{t^{\prime}=1}^{T}\mathcal{J}_{t^{\prime},j}\right)^{2}}_{\text{Variance
of joints lengths across time}}$ (5)

Here $\mathcal{J}\in\mathbb{R}^{T\times(J-1)}$ represents a matrix of the
normalised length of limb $j<(J-1)$ at any time $t<T$. By calculating the
variance of limb lengths and taking the mean, the loss encourages the model to
produce more consistent and stable limb lengths across the entire sequence.
This can be beneficial in applications where maintaining anatomical
consistency in the predicted 3D poses is crucial.

To further regularise for the cases where the 3D pose estimation model
$\mathcal{P}$ misses to detect any pose, we use linear interpolation between
joints. Let’s say the video consists of $N$ frames out of which the model
fails to predict anything for $q$ frames. The linear extrapolation and
interpolation function $L:\mathbb{R}^{(N-q)\times J\times
3}\rightarrow\mathbb{R}^{N\times J\times 3}$ fills in the missing inputs. This
acts as pseudo-labels while training for two of our loss function. This
pseudo-label also helps to ensure temporal continuity in the predicted poses.

#### Mean Per Joint Position (MPJP) Loss:

This loss focuses on the accuracy of the pose estimation by penalizing
deviations in the spatial position of individual joints. This loss function
computes the mean Euclidean distance between the predicted $\hat{X}$ poses and
pseudo-poses $\tilde{X}=L(X)$ where $X$ is the noisy sequence of poses
obtained with model $\mathcal{M}$. It measures the average distance between
corresponding joints in the predicted and pseudo labels. The
$\mathcal{L}_{mpjp}(\hat{X},\tilde{X})$ is defined in Eq. 6.

$\mathcal{L}_{mpjp}=\frac{1}{T\cdot J\cdot
3}\sum_{t=1}^{T}\sum_{j=1}^{J}\sum_{d=1}^{3}\lVert\hat{X}_{t,j,d}-\tilde{X}_{t,j,d}\rVert_{2}$
(6)

#### Normalized MPJP (N-MPJP) Loss:

This loss function introduces a normalization step to address scale variations
between the predicted and target poses. It calculates the scale factor based
on the norms of the predicted and target poses and then applies this scale
factor to the predicted poses before computing the MPJPE. The normalization in
$\mathcal{L}_{nmpjp}$ aims to make the model more robust to variations in
absolute pose values. It is particularly useful when the scale of the poses in
the training and testing data may differ. By incorporating scale information,
$\mathcal{L}_{nmpjp}$ addresses scale-related issues during training,
potentially improving the model’s generalization to different scenarios.

$\mathcal{L}_{nmpjp}=\mathcal{L}_{mpjp}(s\hat{X},\tilde{X})$ (7)

Here the scale factor $s$ is defined by

$s=\frac{\sum_{t=1}^{T}\sum_{j=1}^{J}\sum_{d=1}^{3}\left\|\tilde{X}_{t,j,d}\cdot\hat{X}_{t,j,d}\right\|_{2}}{\sum_{t=1}^{T}\sum_{j=1}^{J}\sum_{d=1}^{3}\left\|\hat{X}_{t,j,d}\right\|_{2}^{2}}$
(8)

The combination of both $\mathcal{L}_{nmpjp}$ and $\mathcal{L}_{mpjp}$ losses
allows the model to simultaneously optimize for accurate joint positions
($\mathcal{L}_{mpjp}$) and address scale variations ($\mathcal{L}_{nmpjp}$).
This is important because accurate joint positions are crucial, but the model
also needs to handle scenarios where the scale of poses may differ between
training and testing data.

#### Velocity Loss:

We optimize velocity loss similar to Eq. 4, but instead of ground truth, we
use pseudo-labels. The velocity loss is defined in Eq. 4.

$\mathcal{L}_{vel}=\frac{1}{N\cdot(J-1)}\sum_{t=1}^{T-1}\sum_{j=1}^{J}\sum_{d=1}^{3}\left\|\hat{V}-\tilde{V}\right\|_{2}$
(9)

where, $\hat{V}=\hat{X}_{t+1,j,d}-\hat{X}_{t,j,d}$ and
$\tilde{V}=\tilde{X}_{t+1,j,d}-\tilde{X}_{t,j,d}$ represent velocities of
predicted poses and pseudo label poses respectively. The velocity loss helps
in smoothing the movement and removing unwanted jittering occurring across
frames.

In summary, by combining all the above-mentioned losses into one final loss
function as shown in Eq. 10, the model is trained to produce accurate joint
positions, maintain anatomical consistency, handle scale variations, and
exhibit smooth temporal transitions. Please note that we only train the model
$\mathcal{M}$ with single video during test time without any supervision. The
final loss function uses self supervision to improve temporal continuity.

$\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{mpjp}+\lambda_{2}\mathcal{L}_{vel}+\lambda_{3}\mathcal{L}_{lim}+\lambda_{4}\mathcal{L}_{nmpjp}$
(10)

Here $\lambda_{i}$ where $i\in{1,2,3,4}$ are loss-weighing hyper-parameters
which remain constant for all evaluations.

Frame 1

Frame 6

Frame 9

Frame 10

Frame 16

Frame 19

Frame 20

Frame 26

Frame 29

Figure 3: Samples of Occluded Humans3.6M dataset. We add artificial occlusions
on Human3.6M dataset that persist both spatially and temporally, covering the
subject up to 100%. Here the person between the Frame 10 to Frame 19 remains
occluded.

##  4 Experiments and Results

In this section, our primary objective is to provide a comprehensive
understanding of our approach. We elaborated on the datasets employed and
conducted a thorough comparison with state-of-the-art methodologies.
Furthermore, we analyzed the qualitative results, pinpointing areas where
existing methods may falter. As a conclusive step, we performed an ablation
study to assess the impact of pre-training and different loss functions,
shedding light on their contributions to our experimental framework.

Method | PA-MPJPE | Accel
---|---|---
PoseFormerV2∗ [47] | 126.3 | 28.5
MEVA† [27] | 59.9 | 29.0
VIBE† [17] | 58.6 | 44.5
SPIN† [19] | 56.7 | 47.0
OOH† [46] | 55.0 | 48.6
PARE† [18] | 52.0 | 43.6
ROMP† [39] | 48.1 | 57.2
BEDLAM∗ [3] | 47.1 | 49.0
TEMP3D (ours) | 46.2 | 47.8

Table 1: 3D pose estimation results on OCMotion. ∗ based on our reproduction.
† results produced by [12].

Method | PA-MPJPE | MPJPE | Accel
---|---|---|---
CLIFF† [26] | 39.4 | 62.9 | -
CLIFF w/o H3.6M† [26] | 56.1 | 89.6 | -
BEDLAM-HMR† [3] | 51.7 | 81.6 | -
BEDLAM-CLIFF† [3] | 50.9 | 70.9 | 39.14
MotionBERT∗ [52] | 64.15 | 95.8 | 14.8
TEMP3D (ours) | 50.4 | 69.7 | 37.1

Table 2: Human3.6M results. ∗ based on our reproduction.
† results are taken from [3] as it uses the same testing protocol as ours.

Prior | $\mathcal{L}_{mpjp}$ | $\mathcal{L}_{vel}$ | $\mathcal{L}_{lim}$ | $\mathcal{L}_{nmpjp}$ | MPJPE | PA-MPJPE
---|---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | ✗ | 179.5 | 98.9
✓ | ✗ | ✗ | ✗ | ✗ | 106.5 | 80.2
✓ | ✓ | ✗ | ✗ | ✗ | 82.1 | 60.4
✓ | ✓ | ✓ | ✗ | ✗ | 81.4 | 59.6
✓ | ✓ | ✓ | ✓ | ✗ | 81.1 | 59.6
✓ | ✓ | ✓ | ✓ | ✓ | 80.7 | 59.0

Table 3: Ablation study Analyzing the effects of the pre-trained motion prior
model and various loss functions.

###  4.1 Ablation study

###  4.2 Datasets

  * •

Human3.6M [13]: An indoor-scene Human3.6M dataset is a pivotal benchmark for
3D human pose estimation from 2D images. Captured with a 4-camera setup, it
includes 11 subjects, each with 15 different actions, annotated in 17
keypoints format. Following [3], we keep only 1 in 5 frames in the test split
comprising the S9 and S11 sequence. We showed experiments on a non-occluded
Human3.6M dataset to show that our method achieves comparable performance with
other state-of-the-art methods.

  * •

Occluded Human3.6M We curate the Occluded Human3.6M dataset to evaluate our
method, specifically designed for assessing human pose estimation under
significant occlusion, unlike existing datasets such as Human3.6M, MPI-
INF-3DHP[30], and 3DPW[41]. To accomplish this, we use random erase occlusions
to Human3.6M videos, completely covering a person up to 100%. These occlusions
persist spatially and temporally for 1.6 seconds within 3.2 seconds of the
frame sequence. The dataset is visualized in Fig. 3. We will make the code to
generate this dataset publicly available.

  * •

OCMotion [12, 11]: OCMotion is a video dataset that extends the 3DOH50K image
dataset [46], incorporating large natural occlusions. The dataset comprises
300K images captured at 10 FPS, featuring 43 sequences observed from 6
viewpoints. It includes annotations for 3D motion using SMPL, 2D poses, and
camera parameters. The sequences {0013, 0015, 0017, 0019} are designated for
testing. Our method does not require supervised training, so we have only used
the test split while performing all experiments.

###  4.3 Quantitative Results

Method | PA-MPJPE | MPJPE | Accel
---|---|---|---
PoseFormerV2 [47] | 193.9 | 260.2 | 38.7
BEDLAM [3] | 179.5 | 98.9 | 39.1
MotionBERT [52] | 76.1 | 112.8 | 28.7
TEMP3D (ours) | 59.0 | 80.7 | 26.6

Table 4: Quantitative comparison of 3D Pose estimation methods on Occluded
Human3.6M. Results show that TEMP3D significantly outperforms existing state-
of-the-art methods.

Figure 4: Qualitative 3D pose estimation results on OCMotion dataset. This
figure shows how our method brings temporal continuity to the video sequences.
The second row represents TEMP3D results. The translucent blue color in the
second row represents the ground truth predictions.

We conduct evaluations on three datasets with varying levels of occlusion:
Human3.6M, representing scenarios without occlusion; OCMotion, portraying
medium occlusion; and Occluded Human3.6M, illustrating heavy occlusion. The
metrics assessed include Procrustes-aligned mean per joint position error (PA-
MPJPE), mean per joint position error (MPJPE), and acceleration error (Accel),
measured as the disparity in acceleration between ground-truth and predicted
3D joints. We report the metrics in (mm).

Our method is most effective when there is heavy occlusion. We outperform
other state-of-the-art methods on the Occluded Human3.6M dataset as shown in
Table 4. Ours TEMP3D performs significantly better than BEDLAM even though our
method uses pseudo-labels from BEDLAM. BEDLAM fails to give any poses when
there is heavy occlusion; hence, the evaluation results fall significantly.
However, since we are using temporal information to fill these gaps in the
video, we predict reasonable poses even in case of heavy occlusions and
improve the result of BEDLAM by a significant margin. Since these results are
on an artificial occlusion, we also experimented on the OCMotion dataset,
which has real-world natural occlusion. Table 1, shows that our approach
TEMP3D attains state-of-the-art results on the OCMotion dataset. Since we
obtained good pseudo-labels from BEDLAM under no-occlusion and partial
occlusions, we observe the proximity of our results to BEDLAM. For the same
reason, we get a similar performance to BEDLAM in Table 2 for the original
Human3.6M dataset. However, we still performed better than our baseline
BEDLAM-CLIFF for MPJPE by a margin of 1.2. Furthermore, we exhibit superior
temporal continuity, outperforming the BEDLAM-CLIFF baseline by a factor of 2
in the acceleration metric. It is important to highlight that methods such as
[39, 18] are supervised and trained on the training split of OCMotion. In
contrast, our approach does not assume access to any labeled training dataset.
A more detailed analysis of results and temporal smoothness is added in the
supplementary materials.

To further investigate our approach and its dependence on various loss
functions, we perform an ablation study where we add $\mathcal{L}_{mpjp}$,
$\mathcal{L}_{vel}$, $\mathcal{L}_{lim}$ loss one by one and observe its
error. In Table 3 we first begin inference using CLIFF with BEDLAM weights. We
see these results in Row 1 of Table 3. These pose sequences are inherently
noisy, hence we refine them using our pre-trained motion prior model
$\mathcal{M}$ which improves the PA-MPJPE error rate by a significant margin
of 18.7 . However, using only the motion prior model to refine poses isn’t
sufficient alone, hence we finetune our model with various self-supervised
losses for the given video to achieve optimal results.

Additional implementation details can be found in the supplementary material.

###  4.4 Qualitative Results

In Fig. 5 we compare our method against an existing state-of-the-art 3D pose
estimation method named PoseFormerV2 [47]. We observe that TEMP3D’s skeleton
is best aligned with the actual ground truth pose, even when there is
significant occlusion. Fig. 4 shows the qualitative results of temporal
continuity achieved using TEMP3D. Additional qualitative results and video
comparisons can be found in the supplementary material.

(a) OCMotion Image

(b) Ground Truth

(c) PoseFormerV2

(d) TEMP3D (ours)

Figure 5: This figure shows how our method works when tested in natural
occlusion cases. The translucent blue color in the second column, third
column, and fourth column represents the ground truth. Blue, red, and green
similarly represent Ground Truth, PoseformerV2 and TEMP3D results,
respectively.

##  5 Conclusions

In conclusion, while existing 3D human pose estimation methods excel in
various scenarios, they struggle with handling significant occlusions. Our
research introduces TEMP3D, an unsupervised approach which utilizes large-
scale pre-training, self-supervised learning, and temporal context to enhance
3D pose estimation for single in-the-wild videos which contain occlusions.
TEMP3D achieves state-of-the-art results in occluded scenarios on datasets
which contain significant human body occlusions, demonstrating improved
occlusion robustness. Future work involves the incorporation of temporally
continuous pose estimates for enhanced downstream tasks such as action
recognition, mesh recovery, and gait recognition.

## References

  * Anvari and Park [2022] Taravat Anvari and Kyoungju Park.  3d human body pose estimation in virtual reality: A survey.  In _2022 13th International Conference on Information and Communication Technology Convergence (ICTC)_ , pages 624–628, 2022.
  * Arnab et al. [2019] Anurag Arnab, Carl Doersch, and Andrew Zisserman.  Exploiting temporal context for 3d human pose estimation in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3395–3404, 2019.
  * Black et al. [2023] Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang.  Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion, 2023.
  * Chen et al. [2019] Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Rohith Mv, Stefan Stojanov, and James M Rehg.  Unsupervised 3d pose estimation with geometric self-supervision.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5714–5724, 2019.
  * Cheng et al. [2019] Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T Tan.  Occlusion-aware networks for 3d human pose estimation in video.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 723–732, 2019.
  * Ci et al. [2022] Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang.  Gfpose: Learning 3d human pose prior with gradient fields, 2022.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Guan et al. [2021a] Shanyan Guan, Jingwei Xu, Michelle Z. He, Yunbo Wang, Bingbing Ni, and Xiaokang Yang.  Out-of-domain human mesh reconstruction via dynamic bilevel online adaptation, 2021a.
  * Guan et al. [2021b] Shanyan Guan, Jingwei Xu, Yunbo Wang, Bingbing Ni, and Xiaokang Yang.  Bilevel online adaptation for out-of-domain human mesh reconstruction, 2021b.
  * Hossain and Little [2018] Mir Rayat Imtiaz Hossain and James J. Little.  _Exploiting Temporal Information for 3D Human Pose Estimation_ , page 69–86.  Springer International Publishing, 2018.
  * Huang et al. [2022a] Buzhen Huang, Yuan Shu, Jingyi Ju, and Yangang Wang.  Occluded human body capture with self-supervised spatial-temporal motion prior.  _arXiv preprint arXiv:2207.05375_ , 2022a.
  * Huang et al. [2022b] Buzhen Huang, Tianshu Zhang, and Yangang Wang.  Object-occluded human shape and pose estimation with probabilistic latent consistency.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022b.
  * Ionescu et al. [2014] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1325–1339, 2014.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose, 2018.
  * Kang et al. [2023] Yangyuxuan Kang, Yuyang Liu, Anbang Yao, Shandong Wang, and Enhua Wu.  3d human pose lifting with grid convolution.  _arXiv preprint arXiv:2302.08760_ , 2023.
  * Katircioglu et al. [2018] Isinsu Katircioglu, Bugra Tekin, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua.  Learning latent representations of 3d human pose with deep neural networks.  _International Journal of Computer Vision_ , 126, 2018.
  * Kocabas et al. [2020] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In _CVPR_ , 2020.
  * Kocabas et al. [2021] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black.  Pare: Part attention regressor for 3d human body estimation.  In _ICCV_ , 2021.
  * Kolotouros et al. [2019] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In _ICCV_ , 2019.
  * Kong and Fu [2022] Yu Kong and Yun Fu.  Human action recognition and prediction: A survey.  _International Journal of Computer Vision_ , 130(5):1366–1401, 2022.
  * Kundu et al. [2020] Jogendra Nath Kundu, Siddharth Seth, MV Rahul, Mugalodi Rakesh, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty.  Kinematic-structure-preserved representation for unsupervised 3d human pose estimation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11312–11319, 2020.
  * Kundu et al. [2021] Jogendra Nath Kundu, Siddharth Seth, Anirudh Jamkhandi, Pradyumna YM, Varun Jampani, Anirban Chakraborty, et al.  Non-local latent relation distillation for self-adaptive 3d human pose estimation.  _Advances in Neural Information Processing Systems_ , 34:158–171, 2021.
  * Kundu et al. [2022] Jogendra Nath Kundu, Siddharth Seth, Pradyumna YM, Varun Jampani, Anirban Chakraborty, and R Venkatesh Babu.  Uncertainty-aware adaptation for self-supervised 3d human pose estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 20448–20459, 2022.
  * Lee et al. [2018] Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee.  Propagating lstm: 3d pose estimation based on joint interdependency.  In _Computer Vision – ECCV 2018_ , pages 123–141, Cham, 2018. Springer International Publishing.
  * Lei et al. [2022] Chenyang Lei, Yazhou Xing, Hao Ouyang, and Qifeng Chen.  Deep video prior for video consistency and propagation, 2022.
  * Li et al. [2022] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan.  Cliff: Carrying location information in full frames into human pose and shape estimation, 2022.
  * Luo et al. [2020] Zhengyi Luo, S Alireza Golestaneh, and Kris M Kitani.  3d human motion estimation via motion compression and refinement.  In _ACCV_ , 2020.
  * Martinez et al. [2017] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little.  A simple yet effective baseline for 3d human pose estimation.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2640–2649, 2017.
  * Mehta et al. [2017a] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt.  Monocular 3d human pose estimation in the wild using improved cnn supervision.  In _2017 international conference on 3D vision (3DV)_ , pages 506–516. IEEE, 2017a.
  * Mehta et al. [2017b] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt.  Monocular 3d human pose estimation in the wild using improved cnn supervision.  In _3D Vision (3DV), 2017 Fifth International Conference on_. IEEE, 2017b.
  * Moreno-Noguer [2016] Francesc Moreno-Noguer.  3d human pose estimation from a single image via distance matrix regression, 2016.
  * Newell et al. [2016] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation, 2016.
  * Newell et al. [2017] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping, 2017.
  * Nie et al. [2017] Bruce Xiaohan Nie, Ping Wei, and Song-Chun Zhu.  Monocular 3d human pose estimation by predicting depth on joints.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 3467–3475. IEEE, 2017.
  * Pavllo et al. [2019] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.  3d human pose estimation in video with temporal convolutions and semi-supervised training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7753–7762, 2019.
  * Radwan et al. [2013] Ibrahim Radwan, Abhinav Dhall, and Roland Goecke.  Monocular image 3d human pose estimation under self-occlusion.  In _2013 IEEE International Conference on Computer Vision_ , pages 1888–1895, 2013.
  * Rhodin et al. [2018] Helge Rhodin, Jörg Spörri, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer, Erich Müller, Mathieu Salzmann, and Pascal Fua.  Learning monocular 3d human pose estimation from multi-view images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8437–8446, 2018.
  * Schwartz and Solove [2011] Paul M Schwartz and Daniel J Solove.  The pii problem: Privacy and a new concept of personally identifiable information.  _NYUL rev._ , 86:1814, 2011.
  * Sun et al. [2021] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J Black, and Tao Mei.  Monocular, one-stage, regression of multiple 3d people.  In _ICCV_ , 2021.
  * Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * von Marcard et al. [2018] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * Walker et al. [2017] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert.  The pose knows: Video forecasting by generating pose futures, 2017.
  * Xu et al. [2020] Jingwei Xu, Zhenbo Yu, Bingbing Ni, Jiancheng Yang, Xiaokang Yang, and Wenjun Zhang.  Deep kinematics analysis for monocular 3d human pose estimation.  In _Proceedings of the IEEE/CVF Conference on computer vision and Pattern recognition_ , pages 899–908, 2020.
  * Xu et al. [2021] Yuanlu Xu, Wenguan Wang, Tengyu Liu, Xiaobai Liu, Jianwen Xie, and Song-Chun Zhu.  Monocular 3d pose estimation via pose grammar and data augmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(10):6327–6344, 2021.
  * Zhang et al. [2022] Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, and Junsong Yuan.  Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 13232–13242, 2022.
  * Zhang et al. [2020] Tianshu Zhang, Buzhen Huang, and Yangang Wang.  Object-occluded human shape and pose estimation from a single color image.  In _CVPR_ , 2020.
  * Zhao et al. [2023] Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen.  Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8877–8886, 2023.
  * Zheng et al. [2021] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding.  3d human pose estimation with spatial and temporal transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11656–11665, 2021.
  * Zheng et al. [2023] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah.  Deep learning-based human pose estimation: A survey.  _ACM Comput. Surv._ , 56(1), 2023.
  * Zhou et al. [2016] Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Konstantinos G Derpanis, and Kostas Daniilidis.  Sparseness meets deepness: 3d human pose estimation from monocular video.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4966–4975, 2016.
  * Zhu et al. [2023a] Haidong Zhu, Wanrong Zheng, Zhaoheng Zheng, and Ram Nevatia.  Sharc: Shape and appearance recognition for person identification in-the-wild, 2023a.
  * Zhu et al. [2023b] Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang.  Motionbert: A unified perspective on learning human motion representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15085–15099, 2023b.
  * Zhu and Picard [2022] Yue Zhu and David Picard.  Decanus to legatus: Synthetic training for 2d-3d human pose lifting.  In _Proceedings of the Asian Conference on Computer Vision_ , pages 2848–2865, 2022.


# Complementary datasets to COCO
for object detection

Ali Borji
Quintic AI,
San Francisco, CA,
<EMAIL_ADDRESS>

###### Abstract

For nearly a decade, the COCO dataset has been the central test bed of
research in object detection. According to the recent benchmarks, however, it
seems that performance on this dataset has started to saturate. One possible
reason can be that perhaps it is not large enough for training deep models. To
address this limitation, here we introduce two complementary datasets to COCO:
i) COCO_OI, composed of images from COCO and OpenImages (from their 80 classes
in common) with 1,418,978 training bounding boxes over 380,111 images, and
41,893 validation bounding boxes over 18,299 images, and ii) ObjectNet_D
containing objects in daily life situations (originally created for object
recognition known as ObjectNet; 29 categories in common with COCO). The latter
can be used to test the generalization ability of object detectors. We
evaluate some models on these datasets and pinpoint the source of errors. We
encourage the community to utilize these datasets for training and testing
object detection models. Code and data is available at
https://github.com/aliborji/COCO_OI.

##  1 Introduction

Object recognition is believed, although debatable, to be solved in computer
vision witnessed by the “superhuman” performance of the state-of-the-art
(SOTA) models ($\sim$3% _vs_ $\sim$5% top-5 error rate human _vs_ machine on
ImageNet, see [8, 6, 14]). Unlike object recognition, however, object
detection remains largely unsolved and models perform far below the
theoretical upper bound [4]. The best performance on _COCOval2017_ and
_COCOtest-dev2017_ are 63.2% and 63.3%,
respectively111https://paperswithcode.com/task/object-detection..

A critical concern is that maybe detection datasets are not big enough to
capture variations in object size222The median scale of the object relative to
the image in ImageNet _vs._ COCO is 554 and 106, respectively. Therefore, most
object instances in COCO are smaller than 1% of the image area [15].,
viewpoint, occlusion, and spatial relationships among objects. In other words,
scaling object detection seems to be much more challenging compared to scaling
object recognition.

The bulk of research on object detection has been primarily focused on
developing network architectures or designing new loss functions; less
attention has been devoted to data. A notable exception is OpenImages dataset.
Despite being orders of magnitude larger than COCO [11], however, OpenImages
[10] has been less adopted. Motivated by studies (e.g., [6, 9, 12]) that have
used very large datasets (e.g., JFT-300M [16]) to achieve SOTA accuracy in
object recognition (about 90% top-1 acc; 10% improvement over models trained
on ImageNet), here we extend the COCO dataset building on top of its
ecosystem. Our approach is orthogonal to works that have employed data
augmentation to increase data (e.g., [18, 7]). We construct a new validation
set to evaluate detectors and diagnose model errors. It is important to have
multiple evaluation sets with different characteristics to gauge
generalization and robustness of object detectors.

##  2 Complementary datasets to COCO

Figure 1: Samples from validation sets of ObjectNet_D and COCO_OI.

###  2.1 COCO_OI

We selected images from all 80 categories of OpenImages that are in common
with COCO, except person, car, chair classes which are 3 most frequent classes
in COCO (Fig. 3; Appx 6). This is to a) avoid making the dataset highly
imbalanced333When discarding some object classes from OpenImages, any image
with any object instance from these classes, even though it had some objects
from other classes of interest, was discarded., and b) keep the number of
samples and computation for model training under control. Some classes from
OpenImages fall under the same COCO category (e.g., bear, brown bear, polar
bear all are mapped to the bear class in COCO). The class mappings are given
in Appx 5. To keep the image size in the range of COCO image size, images from
OpenImages are resized such that the larger dimension is 640 pixels while
preserving the aspect ratio. The train set of COCO_OI has 1,418,978
annotations in total across 380,111 images which is larger than
_COCOtrain2017_ with 860K boxes and 118,287 images. From these, 261,824 images
are selected from the train and test sets of OpenImages containing 558,977
boxes. As part of this dataset, we also create a validation set with 41,893
bounding boxes across 18,299 images chosen from validation set of OpenImages.
See Fig. 1.

###  2.2 ObjectNet_D

We annotated the object bounding boxes in the ObjectNet dataset originally
introduced by Barbu et al. [1]. This dataset contains objects in daily life
situations (https://objectnet.dev/) and its images (50,000 across 313
categories) are pictured by Mechanical Turk workers using a mobile app in a
variety of backgrounds, rotations, and imaging viewpoints.

Barbu et al. reported a dramatic performance drop of the SOTA models on
ObjectNet compared to their accuracy on ImageNet ($\sim$ 40-45% drop). We
revisited the Barbu et al. ’s results and found that applying object
recognition models to the isolated objects, rather than the entire scene as is
done in the original paper, leads to 20-30% performance improvement [3]. It
would be interesting to see how well object detectors perform on this dataset.
In total, 5875 bounding boxes across 5875 images (one annotated object per
image) from 29 categories in common with COCO are annotated. Please see Appx 6
for more details on this dataset.

##  3 Model performance and error analysis

We tested two SOTA models, Efficientdet [17]444efficientdet-d7.pth from
https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch and DetectoRS [5]
on the new validation sets. Models have been trained on COCO dataset. As you
can observe in Table 1, both models perform lower on ObjectNet_D and COCO_OI
datasets compared to COCO. In particular, over ObjectNet_D, performance has
been severely degraded partially because objects in this dataset are pictured
in various, and sometimes odd, viewpoints and backgrounds (Fig. 1). Also, APS
and APM on this dataset are both very low since almost all objects in
ObjectNet_D are large, per the object size definitions in COCO.

Dataset | Model | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
COCO | Efficientdet | 51.2 | 70.4 | 55.1 | 35.7 | 55.8 | 64.8
| DetectoRS | 49.1 | 67.7 | 53.4 | 29.9 | 53 | 65.2
COCO_OI | Efficientdet | 44 | 52.7 | 45.7 | 9 | 23.6 | 51.5
| DetectoRS | 51.5 | 51 | 45.1 | 11.4 | 25.5 | 50.9
ObjectNet_D | Efficientdet | 23.9 | 41.6 | 24.5 | 0 | 5.7 | 24.7
| DetectoRS | 20.3 | 37.8 | 19.6 | 0 | 4.7 | 21.8
Table 1: Model performance over the validation sets of datasets.

To examine whether models behave similarly, we compare their error patterns
over COCO and COCO_OI validation sets. To this end, we employ the recently
proposed error diagnosis and analysis tool known as TIDE by Bolya et al. [2].
According to Fig. 2, both models behave the same on each dataset. Their
behavior across datasets, however, is drastically different. On COCO, errors
are almost equally distributed over 4 major error types (Loc, Cls, Bkg, and
Miss), whereas over COCO_OI, Bkg (i.e., classifying background as an object)
significantly outweighs other error types. This result signifies the impact of
distribution shift on models.

Figure 2: Error analysis over validation sets of COCO and COCO_OI datasets.

##  4 Conclusion

Object detection models have come a long way but performance is still low
compared to other vision tasks such as object recognition. Heavy emphasis on
the COCO dataset for model building has increased the risk of overfitting to
this dataset, as is the case over ImageNet [13]. Lower performance of object
detectors on ObjectNet_D dataset (Table 1) indicates that, similar to
recognition models, object detectors also suffer from domain shift.

We foresee the following directions for future research in this area: b)
employing COCO_OI for training models and comparing the results with models
trained on COCO, and c) testing models over validation sets of the datasets,
in particular ObjectNet_D dataset, and c) using the proposed techniques to
generate even larger training datasets which can open up more research
opportunities for robustness, transfer learning, and semi-supervised learning
in object detection,

## References

  * [1] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  Advances in neural information processing systems, 32:9453–9463, 2019.
  * [2] Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman.  Tide: A general toolbox for identifying object detection errors.  2020\.
  * [3] Ali Borji.  Contemplating real-world object classification.  arXiv preprint arXiv:2103.05137, 2021.
  * [4] Ali Borji and Seyed Mehdi Iranmanesh.  Empirical upper-bound in object detection and more.  arXiv preprint arXiv:1911.12451, 2019.
  * [5] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [7] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  arXiv preprint arXiv:2012.07177, 2020.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  arXiv preprint arXiv:1912.11370, 6(2):8, 2019.
  * [10] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  arXiv preprint arXiv:1811.00982, 2018.
  * [11] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [13] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  arXiv preprint arXiv:1902.10811, 2019.
  * [14] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [15] Bharat Singh and Larry S Davis.  An analysis of scale invariance in object detection snip.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3578–3587, 2018.
  * [16] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [17] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  arXiv preprint arXiv:1911.09070, 2019.
  * [18] Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of freebies for training object detection neural networks.  arXiv preprint arXiv:1902.04103, 2019.

##  5 List of object categories in new datasets

The list of object categories present in the COCO along with their
corresponding classes in the OpenImages dataset is given below. The mapping
between COCO to ObjectNet$\\_D$ datasets is also presented. Note that some
classes in COCO correspond to more than one class in the destination dataset
(i.e., one to many relationship).

COCO to OpenImages category mapping:

    COCO_to_OI_dict = {’Baseball glove’ : ’Baseball glove’,
    ’Parking meter’ : ’Parking meter’,
    ’Skateboard’ : ’Skateboard’,
    ’Car’ : ’Car’,
    ’Cup’ : [’coffee cup’, ’Measuring cup’],
    ’Toilet’ : ’Toilet’,
    ’bicycle’ : ’bicycle’,
    ’sink’ : ’sink’,
    ’cow’: [’Cattle’, ’Bull’],
    ’cat’: ’cat’,
    ’Motorcycle’ : ’Motorcycle’,
    ’Bed’ : ’Bed’,
    ’boat’ : ’boat’,
    ’Cake’ : ’Cake’,
    ’refrigerator’ : ’refrigerator’,
    ’Airplane’ : ’Airplane’,
    ’Carrot’ : ’Carrot’,
    ’Stop sign’ : ’Stop sign’,
    ’Zebra’ : ’Zebra’,
    ’Tie’ : ’Tie’,
    ’Pizza’ : ’Pizza’,
    ’Banana’ : ’Banana’,
    ’Traffic light’ : ’Traffic light’,
    ’Tennis racket’ : ’Tennis racket’,
    ’Microwave’ : ’Microwave oven’,
    ’Apple’: ’Apple’,
    ’donut’: ’Doughnut’,
    ’Bear’: [’Bear’, ’Brown bear’, ’Polar bear’],
    ’Teddy bear’ : ’Teddy bear’,
    ’clock’: [’Alarm clock’, ’Clock’, ’Digital clock’],
    ’sports ball’ : [’Ball’, ’Football’, ’Volleyball’ , ’Tennis Ball’,
                          ’Rugby ball’, ’golf ball’,  ’Cricket ball’],
    ’horse’: ’horse’,
    ’Hair drier’: ’Hair dryer’,
    ’Fire hydrant’: ’Fire hydrant’,
    ’Bus’: ’Bus’,
    ’Dining Table’ : [’Table’, ’Kitchen & dining room table’],
    ’Couch’: [’Couch’, ’Studio couch’],
    ’mouse’: ’computer mouse’,
    ’Remote’: ’Remote control’,
    ’Bench’: ’Bench’,
    ’Scissors’: ’Scissors’,
    ’Truck’: ’Truck’,
    ’Wine glass’: ’Wine glass’,
    ’Oven’: ’Oven’,
    ’Backpack’: ’Backpack’,
    ’Bird’: ’Bird’,
    ’Handbag’: ’Handbag’,
    ’Sheep’: ’Sheep’,
    ’Spoon’: ’Spoon’,
    ’Tv’:’Television’,
    ’Frisbee’ : ’Flying disc’,
    ’Keyboard’ : ’computer Keyboard’,
    ’Sandwich’ : [’Sandwich’, ’Submarine sandwich’],
    ’Cell phone’ : ’Mobile phone’,
    ’Giraffe’ : ’Giraffe’,
    ’Chair’ : ’Chair’,
    ’Bottle’ : ’Bottle’,
    ’Potted plant’ : ’Houseplant’,
    ’Broccoli’ : ’Broccoli’,
    ’Umbrella’ : ’Umbrella’,
    ’Orange’ : ’Orange’,
    ’Hot dog’ : ’Hot dog’,
    ’Knife’ : ’kitchen knife’,
    ’Vase’ : ’Vase’,
    ’Surfboard’ : ’Surfboard’,
    ’Toaster’ : ’Toaster’,
    ’Bowl’ : [’bowl’, ’Mixing bowl’],
    ’Suitcase’ : ’Suitcase’,
    ’Fork’ : ’Fork’,
    ’Skis’ : ’Ski’,
    ’Book’ : ’Book’,
    ’Kite’ : ’Kite’,
    ’Person’ : ’Person’,
    ’Dog’ : ’Dog’,
    ’Laptop’ : ’Laptop’,
    ’Elephant’ : ’Elephant’,
    ’Toothbrush’ : ’Toothbrush’,
    ’Baseball bat’ : ’Baseball bat’,
    ’Snowboard’ : ’Snowboard’,
    ’Train’ : ’Train’}

COCO to ObjectNet_D category mapping:

    COCO_to_ObjNet_dict = {’Baseball glove’ : ’baseball_glove’,
    ’Cup’ : [’drinking_cup’, ’measuring_cup’],
    ’bicycle’ : ’bicycle’,
    ’Tie’ : ’tie’,
    ’Banana’ : ’banana’,
    ’Tennis racket’ : ’tennis_racket’,
    ’Microwave’ : ’microwave’,
    ’clock’: ’alarm_clock’,
    ’Hair drier’: ’hair_dryer’,
    ’Dining Table’ : ’coffee_table’,
    ’mouse’: ’computer_mouse’,
    ’Remote’: ’remote_control’,
    ’Bench’: ’bench’,
    ’Backpack’: ’backpack’,
    ’Tv’:’Tv’,
    ’Keyboard’ : ’keyboard’,
    ’Cell phone’ : ’cellphone’,
    ’Chair’ : ’chair’,
    ’Bottle’ : [’water_bottle’, ’cooking_oil_bottle’, ’beer_bottle’, ’wine_bottle’],
    ’Umbrella’ : ’Umbrella’,
    ’Orange’ : ’Orange’,
    ’Knife’ : [’butchers_knife’ , ’bread_knife’],
    ’Vase’ : ’Vase’,
    ’Toaster’ : ’Toaster’,
    ’Bowl’ : [’mixing_salad_bowl’, ’soup_bowl’],
    ’Suitcase’ : ’briefcase’,
    ’Book’ : ’book_closed’,
    ’Laptop’ : ’laptop_open’,
    ’Baseball bat’ : ’Baseball_bat’}

##  6 Dataset statistics

Some statistics of the proposed datasets are shown here. In building
$COCO\\_OI$ dataset, Fig. 3, we have avoided to include samples from Person,
Car, and Chair categories from the OpenImages dataset. Including these
categories will make the $COCO\\_OI$ dataset highly imbalanced. Our code for
creating this dataset ()link, however, is very general and allows discarding
arbitrary object categories to make a more balanced dataset.

Figure 3: Distribution of the number of objects per category in training sets
of COCO and COCO_OI datasets. Figure 4: Distribution of the number of objects
across the validation sets of the used datasets. Figure 5: Distribution of the
number of objects and normalized object size across the used datasets.

###  6.1 Details on ObjectNet_D dataset

ObjectNet dataset [1], is build with the purpose of having less bias than
other recognition datasets. It consists of indoor objects that are available
to many people, are mobile, are not too large, too small, fragile or
dangerous. This dataset is supposed to be used solely as a test set and comes
with a licence that disallows the researchers to finetune models on it.

ObjectNet images are pictured by Mechanical Turk workers using a mobile app in
a variety of backgrounds, rotations, and imaging viewpoints. ObjectNet
contains 50,000 images across 313 categories, out of which 113 are in common
with ImageNet categories. Astonishingly, Barbu et al. found that the state of
the art object recognition models perform drastically lower on ObjectNet
compared to their performance on ImageNet (about 40-45% drop).

The 113 object categories in the ObjectNet dataset, overlapped with the
ImageNet, contain 18,574 images in total. On this subset, the average number
of images per category is 164.4 (min=55, max=284). We drew a bounding box
around the object corresponding to the category label of each image. If there
were multiple nearby objects from the same category (e.g., chairs around a
table), we tried to include all of them in the bounding box. Some example
scenes and their corresponding bounding boxes are given in Fig. 6.

Figure 6: Sample images from the ObjectNet dataset along with our manually
annotated object bounding boxes from Chairs, Teapots and T-shirts categories.
The leftmost column shows three chair examples from the ImageNet dataset.
ImageNet scenes often have a single isolated object in them whereas images in
the ObjectNet dataset contain multiple objects. Further, ObjectNet objects
cover a wider range of variation in contrast, rotation, scale, and occlusion
compared to ImageNet objects (See arguments in [1]). In total, we annotated
18,574 images across 113 categories in common between the two datasets. This
figure is modified from Figure 2 in [1]. Figure 7: Error analysis over the
ObjectNet_D dataset.

##  7 Dataset licenses

Below we list each dataset’s license, as provided either in the paper
proposing the dataset or on the dataset website.

  1. 1.

MSCOCO: Creative Commons Attribution 4.0 License,
https://cocodataset.org/#termsofuse

  2. 2.

OpenImages: CC BY 4.0 license, the images are listed as having a CC BY 2.0
license https://storage.googleapis.com/openimages/web/factsfigures.html

  3. 3.

ObjectNet: Creative Commons Attribution 4.0 with only two additional clauses:
a) ObjectNet may never be used to tune the parameters of any model, and b) Any
individual images from ObjectNet may only be posted to the web including their
1 pixel red border. See https://objectnet.dev/download.html.