capbtabboxtable[][]

# CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators

Harry Zhang, Luca Carlone
Massachusetts Institute of Technology
Cambridge, MA 02139, USA
{harryz<EMAIL_ADDRESS>
###### Abstract

We introduce CHAMP, a novel method for learning sequence-to-sequence, multi-
hypothesis 3D human poses from 2D keypoints by leveraging a conditional
distribution with a diffusion model. To predict a single output 3D pose
sequence, we generate and aggregate multiple 3D pose hypotheses. For better
aggregation results, we develop a method to score these hypotheses during
training, effectively integrating conformal prediction into the learning
process. This process results in a differentiable conformal predictor that is
trained end-to-end with the 3D pose estimator. Post-training, the learned
scoring model is used as the conformity score, and the 3D pose estimator is
combined with a conformal predictor to select the most accurate hypotheses for
downstream aggregation. Our results indicate that using a simple mean
aggregation on the conformal prediction-filtered hypotheses set yields
competitive results. When integrated with more sophisticated aggregation
techniques, our method achieves state-of-the-art performance across various
metrics and datasets while inheriting the probabilistic guarantees of
conformal prediction.

##  1 Introduction

Learning to estimate 3D human poses from videos is an important task in
computer vision and robotics. Typical 3D human pose estimators learn to
predict a single 3D human pose from an RGB image, either in an end-to-end
manner or by relying on pre-existing 2D keypoint detection methods applied to
the RGB images. Recent advances in sequence-to-sequence modeling enable
learning 3D human poses in sequence by using RGB videos, which significantly
improves the flexibility and efficiency of such estimators (Zhang et al.,
2022b, a). While promising, such approaches only focus on single-hypothesis
predictions, which is an inherently ill-posed problem when trying to
reconstruct a 3D human pose given inputs collected from one viewpoint.

Thus, a more recent line of work focuses on learning multi-hypothesis 3D human
pose estimators from 2D inputs. Instead of learning one deterministic target,
such methods model the 2D-to-3D pose learning problem as learning a
conditional distribution, which describes the distribution of the 3D poses
given 2D inputs. This change in problem formulation prompts the use of
generative models such as GANs, VAEs, and Diffusion models (Li & Lee, 2020;
Sharma et al., 2019; Shan et al., 2023), which can propose multiple hypotheses
of 3D poses given a single 2D input. For practical uses, one should also
consider aggregating the hypotheses to generate one single output prediction.
However, due to the imperfection of the trained generative models, using all
hypotheses when aggregating could be inefficient and suboptimal if some are
exceedingly inaccurate.

To counter the aforementioned problems, we take inspiration from an important
tool in statistical learning, conformal prediction (CP) (Shafer & Vovk, 2008;
Angelopoulos & Bates, 2021), which uses a post-training calibration step to
guarantee a user-specified coverage: by allowing to predict confidence sets
$C(X)$, CP guarantees the true value $Y$ to be included with confidence level
$\alpha$, i.e. $P(Y\in C(X))\geq 1-\alpha$ when the calibration examples
$(X_{i},Y_{i})\in I_{\text{cal}}$ are drawn exchangeably from the test
distribution. There are typically two steps involved in the CP process: In the
calibration step, the conformity scores on the calibration set are ranked to
determine a cut-off threshold $\tau$ for the predicted values guaranteeing
coverage $1-\alpha$, usually via quantile computation. In the prediction step,
conformity scores, which measure the conformity between the output and
possible ground-truth values, are computed to construct the confidence sets
$C(X)$ by using the calibrated threshold $\tau$.

Figure 1: CHAMP sample results obtained on in-the-wild videos collected from
TikTok. Having observed 2D keypoints, CHAMP proposes multiple hypotheses of
the 3D human skeleton poses, and then a conformal predictor trained end-to-end
with the pose estimator refines the confidence set by filtering out low-
conformity-score hypotheses. This smaller set will be used in downstream
aggregation for a single output prediction.

CP is highly flexible as it can be applied to any machine learning model.
However, since it is applied post-training, learning the model parameters is
not informed about the post-hoc conformalization step: the model is not tuned
towards any specific CP objective such as reducing expected confidence set
size (inefficiency). To bias the training towards lower inefficiency, ConfTr
(Stutz et al., 2021) proposes a fully differentiable CP operation for
classification tasks, which is applied end-to-end with the prediction model
and optimizes a differentiable inefficiency loss in tandem with the original
class loss for the classifier. This operation allows the model to learn
parameters to reduce the inefficiency of the confidence set size during
conformalization. Inspired by this, we wish to learn a scoring model that
scores the “quality” of each hypothesis by measuring its conformity to the
ground truth. We can then use this scoring model to simulate CP and optimize
for the inefficiency in a differentiable manner, as done in (Stutz et al.,
2021). During test time, we propose a large number of hypotheses and do
regular CP to filter out the low-score ones before feeding into the downstream
aggregation steps. The learned CP procedure can be applied to any multi-
hypothesis estimator training process to refine the hypotheses confidence set
size during test time for better aggregation results. With this learned CP
wrapper, we present CHAMP, a Conformalized 3D HumAn Multi-Hypotheses Pose
Estimator. To summarize, the contributions of this work include:

  * •

A novel sequence-to-sequence, multi-hypothesis 3D human pose estimator from 2D
keypoints.

  * •

A novel method to conformalize 3D human pose estimates during training by
learning a score function to rank the quality of the proposed hypotheses.

  * •

A novel method to aggregate the multiple hypotheses from the estimator output
using conformalization based on a learned score function.

  * •

Quantitative and qualitative results that demonstrate the state-of-the-art
results of our method on a variety of real-world datasets.

##  2 Related Work

Diffusion Models are a family of generative models that gradually corrupt data
by adding noise, and then learn to recover the original data by reversing this
process (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2020;
Song et al., 2020). They have achieved success in generating high-fidelity
samples in various applications such as image generation (Ho et al., 2022;
Saharia et al., 2022; Nichol et al., 2021; Batzolis et al., 2021; Rombach et
al., 2022; Zhang et al., 2020), multi-modal learning (Levkovitch et al., 2022;
Kim et al., 2022; Huang et al., 2022; Avrahami et al., 2022; Zhang et al.,
2021) and pose estimations in 3D vision problems (Gong et al., 2023; Wu et
al., 2023; Devgon et al., 2020). We use Diffusion models to learn 3D human
poses from 2D keypoints.

3D Human Pose Estimation is an important problem in computer vision and
robotics. With deep learning, various end-to-end approaches have been proposed
(Tekin et al., 2016; Pavlakos et al., 2017; Elmquist et al., 2022). However,
with the maturity of 2D human keypoints detection, (Ho et al., 2022; Cao et
al., 2017; Ma et al., 2022; Zhang, 2016; Sim et al., 2019), more robust
approaches focus on uplifting 2D keypoints to 3D, resulting in better
performance (Martinez et al., 2017; Zhou et al., 2017; Xu & Takano, 2021; Zhao
et al., 2019; Lim et al., 2021, 2022), where one typically deals with a frame-
to-frame problem, or a sequence-to-sequence problem. Being able to predict a
3D keypoints sequence directly from a 2D keypoint sequence is highly desirable
as it has higher efficiency and flexibility. In this scheme, deterministic
methods learn to predict one single 3D output from the 2D input (Pavllo et
al., 2019; Zeng et al., 2020; Zheng et al., 2021; Shan et al., 2021; Jin et
al., 2024; Hu et al., 2021; Zhan et al., 2022; Eisner et al., 2022). However,
with a single viewpoint, such a formulation is ill-posed because there could
be many possible 3D poses satisfying the same 2D keypoint configuration. Thus,
it is desirable to model the problem as a conditional probability
distribution. Deep generative models have shown great potential in modeling
such distributions. Specifically, mixed-density network (Li & Lee, 2019; Pan
et al., 2022, 2023), VAE (Sharma et al., 2019; Zhang et al., 2023),
normalizing flows (Wehrbein et al., 2021), GAN (Li & Lee, 2019), and Diffusion
models (Feng et al., 2023; Choi et al., 2023; Rommel et al., 2023; Avigal et
al., 2021; Yao et al., 2023; Shen et al., 2024) have all been applied to
modeling such conditional distribution. In our work, we use (Zhang et al.,
2022b) as the backbone for generating 3D keypoints sequences from 2D keypoints
sequences. To infer multiple hypotheses, we follow the frameworks in (Shan et
al., 2023; Gong et al., 2023; Zhou et al., 2023) to learn to recover 3D pose
hypotheses from Gaussian noises.

Conformal Prediction (CP) is a powerful and flexible distribution-free
uncertainty quantification technique that can be applied to any machine
learning model (Angelopoulos & Bates, 2021; Shafer & Vovk, 2008). Assuming the
exchangeability of the calibration data, CP has desirable coverage guarantees.
Thus, it has been applied to many fields such as robotics (Huang et al., 2023;
Avigal et al., 2020; Sun et al., 2024), pose estimation (Yang & Pavone, 2023),
and image regression (Teneggi et al., 2023; Angelopoulos et al., 2022). More
sophisticated CP paradigms have been also proposed to tackle distribution
shift and online learning problems (Angelopoulos et al., 2024; Gibbs & Candès,
2022; Bhatnagar et al., 2023; Feldman et al., 2022). Since CP is applied post-
training, the learned model is not aware of such conformalization during
training. To gain better control over the confidence set of CP, learning
conformal predictors and nonconformity score end-to-end with the model has
been proposed in (Stutz et al., 2021; Fisch et al., 2021; Bai et al., 2022).
More closely related to our proposal is the work of (Stutz et al., 2021), but
instead of using raw logits as the conformity score for classification, we
learn an extra scoring model as the score to rank the hypotheses and simulate
CP during training.

##  3 Problem Formulation

We are interested in the problem of learning a sequence of 3D human poses from
a sequence of 2D human pose keypoints. We assume the 2D keypoints are
available to us, which could be detected from the RGB images using well-
established methods such as (Li et al., 2022b). Formally, given the input 2D
keypoints sequence $\bm{x}=\\{\bm{p}^{2d}_{n}|n=1,\dots,N\\}$, where
$\bm{p}^{2d}_{n}\in\mathbb{R}^{J\times 2}$, our goal is to learn a conditional
distribution for the possible corresponding 3D positions of all joints
$p_{\theta}(\bm{y}|\bm{x})$, where the sequence
$\bm{y}=\\{\bm{p}^{3d}_{n}|n=1,\dots,N\\}$ and
$\bm{p}^{3d}_{n}\in\mathbb{R}^{J\times 3}$. Here, $N$ represents the number of
input and output frames and $J$ is the number of human joints in each frame.
With the learned distribution $p_{\theta}(\bm{y}|\bm{x})$, we are able to do
inference and generate hypotheses of 3D poses
$\bm{H}_{\bm{y}}\in\mathbb{R}^{H\times N\times J\times 3}$, where $H$ is the
number of the hypotheses. We then conformalize the hypotheses to select the
higher-quality ones and aggregate the latter in order to obtain a single
estimate $\tilde{\bm{y}}$ as the final output.

##  4 Methods

We use a Diffusion model (Ho et al., 2020) to learn the conditional
distribution $p_{\theta}(\bm{y}|\bm{x})$ due to its capability of modeling
complex distributions and producing high-quality samples. With the trained
diffusion model, we are able to generate many hypotheses sequences and
aggregate them to produce a robust single prediction sequence for the final
output. Moreover, we would like to make the model itself aware of the post-hoc
hypotheses aggregation step so that the aggregation step does not get skewed
by low-quality hypotheses. Thus, there should be an extra component in the
model to shape the hypotheses confidence set. To achieve this, we learn an
extra scoring model to rank the hypotheses and optimize the size of the
hypotheses confidence set thresholded by the 90% quantile score, effectively
simulating conformal prediction during training, illustrated in Fig. 2.

Figure 2: CHAMP Overview. CHAMP takes as input a sequence of 2D keypoints
detected on a series of input RGB video frames. The 2D keypoints sequence gets
fed into a Diffusion model to produce 3D keypoints hypotheses for the
sequence. The output of the Diffusion model is supervised via a Pose Loss.
Then we apply differentiable CP end-to-end during training on the hypotheses
sequences, resulting in a smaller confidence set. The confidence set is used
to calculate an Inefficiency Loss during training. Note that we show one frame
in the sequence and hard assignment for the confidence set during training for
better interpretability.

###  4.1 Learning 3D Human Poses with a Diffusion Model

The use of Diffusion models in 3D human pose estimation has been shown
effective in various previous works (Feng et al., 2023; Gong et al., 2023;
Shan et al., 2023; Choi et al., 2023), and we adapt the same paradigm in
modeling the forward and the reverse process for the Diffusion model.

Forward Process. In the forward process, the Diffusion model takes as input a
ground-truth 3D human pose sequence $\bm{y}_{0}\in\mathbb{R}^{N\times J\times
3}$ and a sampled time step $t\sim\text{Unif}(0,T)$, where $T$ is the maximum
number of diffusion steps. Then the forward process gradually diffuses the
input by adding independent Gaussian noises
$\bm{\epsilon}\sim\mathcal{N}(0,\bm{I})$ at each step. The nice property
reported in (Ho et al., 2020) shows that the process can be succinctly written
as:

$q(\bm{y}_{t}|\bm{y}_{0})=\sqrt{\bar{\alpha}_{t}}\bm{y}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$
(1)

where $\bar{\alpha}_{t}$ is a constant depending on the cosine variance
schedule.

Reverse Process. To train such a Diffusion model, in the reverse process, we
denoise the corrupted 3D pose sequence $\bm{y}_{t}$. While the derivation in
(Ho et al., 2020) simplifies the ELBO objective to minimizing the distance
between the injected noise $\bm{\epsilon_{t}}$ and a learned noise
$\bm{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}\bm{y}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{t},t)$,
we frame the problem slightly differently, following (Shan et al., 2023).
Instead of training a network to learn the injected noise, we train a denoiser
model $D_{\theta}$ that outputs the predicted pose $\bar{\bm{y}}_{0}$
directly:

$\bar{\bm{y}}_{0}=D_{\theta}(\bm{y}_{t},\bm{x},t)$ (2)

where the denoiser model takes as input the corrupted 3D pose sequence, the
input 2D keypoints sequence, and the diffusion step to recover the uncorrupted
input pose sequence $\bar{\bm{y}}_{0}$. We then supervise the predicted
sequence with the ground-truth sequence using the mean per-joint MSE:

$\mathcal{L}_{\text{pose}}=\frac{||\bm{y}_{0}-\bar{\bm{y}}_{0}||^{2}_{2}}{N\cdot
J}$ (3)

Denoiser Model. We build upon the MixSTE model (Zhang et al., 2022b) as the
denoiser model in the reverse process of the Diffusion model. MixSTE uses two
separate attention mechanisms, effectively learning spatial and temporal
relationships of the keypoints sequence in a modular way by combining several
spatial and temporal attention blocks. To condition the MixSTE network on the
corrupted 3D keypoints sequence, we change the network input by concatenating
2D keypoints and noisy 3D poses and we add a diffusion timestep embedding
after the input embedding in the first attention block. A detailed description
of the denoiser model is given in Appendix B and D.

Generating Hypotheses. Using the denoiser model, which models the conditional
distribution $p_{\theta}(\bm{y}|\bm{x})$, we are able to generate a number of
hypotheses during inference. Following the reverse process of the Diffusion
model, we obtain the hypotheses set
$\bm{H}_{\bm{y}}=\\{\bar{\bm{y}}^{1},\bar{\bm{y}}^{2},\dots,\bar{\bm{y}}^{H}\\}$
of size ${H}$ by sampling standard normal noises
${\bm{y}}^{h}_{T}\sim\mathcal{N}(0,\bm{I})\in\mathbf{R}^{N\times J\times 3}$,
where the superscript $h=[1,2,\dots,H]$ indexes the hypotheses, and feeding
them into the denoiser model $D_{\theta}$:

$\bar{\bm{y}}^{h}=D_{\theta}({\bm{y}}^{h}_{T},\bm{x},T)$ (4)

###  4.2 Learning Conformalization for the Hypotheses Confidence Set

While we are able to generate arbitrarily large numbers of hypotheses during
test time with the distribution learned by the Diffusion model, we still need
to aggregate the proposed hypotheses into a single prediction. The correct way
to aggregate the hypotheses remains an open problem. Taking a naive average
would be suboptimal as the existence of outliers in the proposed hypotheses
might skew the average. Other approaches improve the aggregation step by
selecting the hypotheses that result in closer distances to 2D keypoints after
projection via camera matrices, but still retain a large set of hypotheses
before the aggregation step (Shan et al., 2023).

Conformal Prediction. A possible solution is to conformalize the hypotheses
using conformal prediction (CP). Conformal prediction produces a set of
predictions which covers the ground truth with high probability: given a
trained prediction model $f_{\theta}$, conformal prediction first calibrates a
cutoff value $\tau$ by ranking a calibration set $I_{\text{cal}}$ given each
component’s conformity score $\phi(\bm{y}_{\text{cal}},\bm{y}_{\text{GT}})$
and setting the cutoff value as the user-specified
$(1+1/|I_{\text{cal}}|)\alpha$-quantile of the calibration set scores. After
the calibration step, the generic threshold conformal predictor constructs the
conformal prediction set as follows:

$C(\bm{x},\tau)=\\{\bm{y}:\phi(\bm{y},f_{\theta}(\bm{x}))\geq\tau\\}$ (5)

With a properly selected conformity score function $\phi$, as well as a user-
specified coverage parameter $\alpha$, the post-training conformal prediction
guarantees the true value to be included in $C(\bm{x},\tau)$ with confidence
level $\alpha$, i.e. $P(\bm{y}_{\text{GT}}\in C(\bm{x},\tau))\geq\alpha$. A
key metric for a conformal predictor is the size of the confidence set
$C_{\theta}(\bm{x},\tau)$, or _inefficiency_ : a good confidence set should be
large enough to cover the ground truth, and small enough to be informative
(low inefficiency). A very large confidence set could cover the ground truth
with high probability, but it might not be necessarily useful as it is
inefficient to consider a lot of possible values (high inefficiency). Given
${H}$ hypotheses of 3D human pose sequences, we wish to use the threshold
conformal predictor above to refine the hypotheses set, resulting in
${H}^{\prime}$ “good” hypotheses, where ${H}^{\prime}\leq H$. Essentially, we
wish to have a low-inefficiency conformal predictor, which selects a smaller
number of elite hypotheses from the set for downstream aggregation. CP is
intended to be used as a “wrapper” around the prediction model, and we wish to
achieve better, fine-grained control over the inefficiency of the CP wrapper
for downstream aggregation, conditioned on the input 2D and output 3D human
pose sequence data. Thus, inspired by (Stutz et al., 2021), we integrate CP
into the training, optimizing for the inefficiency of the confidence set
online by simulating CP during training in a differentiable manner.

Learning CP via Inefficiency Optimization. We build on top of the ConfTr
paradigm proposed in (Stutz et al., 2021). We first design a conformity score
function for poses. Unlike (Yang & Pavone, 2023), where the score function
$\phi$ is hand-designed, we wish to learn a conformity score function
$\phi_{\theta}$ that measures the distance between the proposed hypothesis and
the ground truth. During training, we propose ${H}_{\text{train}}$ predicted
output sequences as the hypotheses for each input 2D keypoints sequence in the
mini-batch. To implement the score function, we reuse the input embedding
layer of the modified MixSTE and further project the embeddings into two
lower-dimensional tensors with an extra MLP for the hypothesis and the ground-
truth 3D pose sequence respectively and then calculate the cosine similarity
between the two embeddings. Thus, a higher score function output means the
hypothesis is more similar to the ground-truth 3D pose sequence in the
embedding space. Formally, the conformity score between the hypothesis 3D pose
sequence $\bar{\bm{y}}^{h}$ and the ground-truth 3D pose sequence
$\bm{y}_{\text{GT}}$ is measured as:

$\phi_{\theta}(\bar{\bm{y}}^{h},\bm{y}_{\text{GT}})=\frac{s_{\theta}\left(\text{IE}_{\theta}(\bar{\bm{y}}^{h})\right)^{\top}s_{\theta}(\text{IE}_{\theta}(\bm{y}_{\text{GT}}))}{||s_{\theta}\left(\text{IE}_{\theta}(\bar{\bm{y}}^{h}))||\cdot||s_{\theta}(\text{IE}_{\theta}(\bm{y}_{\text{GT}})\right)||}$
(6)

where $s_{\theta}$ is the extra MLP in the conformity score model and
$\text{IE}_{\theta}$ is the input embedding layer of the denoiser model. With
the conformity score model output, we perform differentiable CP on each mini-
batch of size $B$ during training. Similar to (Stutz et al., 2021), we split
each set of $H_{\text{train}}$ hypotheses of the mini-batch in half: the first
half, $\bm{H}_{\text{cal}}$, is used for calibration, and the second half,
$\bm{H}_{\text{pred}}$, for prediction and loss computation. On
$\bm{H}_{\text{cal}}$, we calibrate $\tau$ by computing the
$(1+1/|\bm{H}_{\text{cal}}|)\alpha$-quantile of the conformity scores in a
differentiable manner. This is done using a smooth sorting operation, which
has been widely studied in (Blondel et al., 2020) and (Cuturi et al., 2019).
On $\bm{H}_{\text{pred}}$, we calculate the soft inefficiency score by using
the soft assignment of hypothesis $\bar{\bm{y}}^{h}$ being in the prediction
set given the quantile threshold $\tau$:

$C_{\theta}(\bar{\bm{y}}^{h},\tau):=\sigma\left(\frac{\phi_{\theta}(\bar{\bm{y}}^{h},\bm{y}_{\text{GT}})-\tau}{\mathcal{T}}\right)$
(7)

where $\sigma$ here is the sigmoid function and $\mathcal{T}$ is the
temperature hyperparameter. When $\mathcal{T}\rightarrow 0$, we recover the
hard assignment and $C_{\theta}(\bar{\bm{y}}^{h},\tau)=1$ if
$\phi_{\theta}(\bar{\bm{y}}^{h},\bm{y}_{\text{GT}})\geq\tau$ and 0 otherwise.
Thus, we are able to measure the size of the CP set of the hypotheses by
$\sum_{h}C_{\theta}(\bar{\bm{y}}^{h},\tau)$. On $\bm{H}_{\text{pred}}$, we
compute $C_{\theta}(\bar{\bm{y}}^{h},\tau)$ and then calculate the
differentiable inefficiency score:

$\Omega\left(C_{\theta}(\bm{H}_{\text{pred}},\tau)\right)=\max\left(\sum_{h=1}^{|\bm{H}_{\text{pred}}|}C_{\theta}(\bar{\bm{y}}^{h},\tau)-\kappa,0\right)$
(8)

where $\kappa$ is a hyperparameter that avoids penalizing singletons. With the
differentiable inefficiency score defined, we are able to optimize for the
expected inefficiency across batches during training:

$\mathcal{L}_{\text{ineff}}=\log\mathbb{E}[\Omega(C_{\theta}(\bm{H}_{\text{pred}},\tau))]$
(9)

Full Training Objective. The computational graph of this loss involves the
denoiser model input embedding, and since every operation is made
differentiable by construction, we are able to backpropagate the inefficiency
loss into the model, further shaping the model on top of the original pose
loss. The final training objective of our model is thus:

$\mathcal{L}=\mathcal{L}_{\text{pose}}+\lambda\mathcal{L}_{\text{ineff}}$ (10)

where $\lambda$ is a hyperparameter representing the weight of the
inefficiency loss.

###  4.3 Conformalized Inference

During inference, to retain the coverage guarantee of CP, any CP method can be
applied to re-calibrate $\tau$ on a held-out calibration set $I_{\text{cal}}$
as usual, i.e., the thresholds $\tau$ obtained during training is not kept
during test time. Given a trained pose estimation denoiser model $D_{\theta}$
and the conformity scoring model $\phi_{\theta}$ trained end-to-end with
$D_{\theta}$, for each testing 2D keypoints sequence, $\bm{x}$, we wish to
construct a hypotheses confidence set for the downstream aggregation as
follows:

$C_{\theta}(\bm{y},\tau)=\\{\bm{y}:\phi_{\theta}(\tilde{\bm{y}},\bm{y})\geq\tau\\}$
(11)

where $\tilde{\bm{y}}$ is obtained via eq. (4) and
$\bm{y}_{T}\sim\mathcal{N}(0,\bm{I})$. In practice, we take the mean of 20
$\tilde{\bm{y}}$. To generate candidate hypotheses for the hypotheses
confidence set, we sample from the learned distribution
$p_{\theta}(\bm{y}|\bm{x})$ $H$ times using (4) and refine them using (11),
resulting in a smaller hypotheses set $\bm{H}_{\bm{y}}$ of size $H^{\prime}$.
We then aggregate the set $\bm{H}_{\bm{y}}$ to obtain a single output for
practical use.

##  5 Experiments

To evaluate our method, we train and test on standard human pose estimation
datasets and provide quantitative results. We also provide qualitative results
on in-the-wild videos.

Human3.6M (Ionescu et al., 2013) is the standard and the largest indoor
dataset for 3D human pose estimation. The dataset collects videos from 11
actors engaging in 15 activities and the pose sequence videos are captured by
4 synchronized and calibrated cameras at 50Hz. Similar to (Shan et al., 2023;
Zhang et al., 2022b), we train on 5 actors (S1, S5, S6, S7, S8) and evaluate
on 2 actors (S9, S11). Quantitatively, following the standard evaluation
scheme, we report the mean per joint position error (MPJPE), which is often
referred to as Protocol #1, which computes the mean Euclidean distance between
estimated and ground truth 3D joint positions in millimeters. We also provide
Protocol #2 results in Appendix E.

MPI-INF-3DHP (Mehta et al., 2017) is a more challenging dataset consisting of
indoor and outdoor activities, from 14 camera views which cover a greater
diversity of poses. The training set contains 8 activities, and the test set
contains 6. We preprocess the dataset with the valid frames provided by the
authors for testing, following (Zhang et al., 2022b; Shan et al., 2023; Gong
et al., 2023). Quantitatively, we report MPJPE, the percentage of correct
keypoints (PCK), which describes the percentage of keypoints with Euclidean
error less than 150mm, as well as the AUC score for this percentage.

For both of the datasets, in our setting, we hold out 2% of the training
dataset for conformal calibration during test time. The held-out calibration
set is not seen by the network during training. We compare 4 variations of our
method. The backbone (weights) of the variations and the training process
remain the same, but the key difference lies in the aggregation step during
test time:

  * •

CHAMP-Naive: Trained with inefficiency loss, but for multi-hypothesis
scenarios, during test time, we do not refine the hypotheses set with CP, and
we use all hypotheses with a mean aggregation. In single-hypothesis scenarios,
we only propose 1 hypothesis as the output.

  * •

CHAMP: Same backbone as CHAMP-Naive, but only aggregates refined hypotheses
confidence set in eq. (11) via mean aggregation.

  * •

CHAMP-Agg: Instead of taking the average, we use the J-Agg method from (Shan
et al., 2023) on the set in eq. (11), which uses known or estimated intrinsic
camera parameters to reproject 3D hypotheses to the 2D camera plane and
selecting the joint hypotheses with the minimum reprojection error.

  * •

CHAMP-Best: We use J-Best method in (Shan et al., 2023) on the set in eq.
(11), which selects the joint hypothesis that is closest to the ground truth,
and then combines the selected joints into the final 3D pose. This is the
upper bound of J-Agg performance.

###  5.1 Results on Human-3.6M

We discuss the quantitative results on the Human-3.6M dataset. In the single-
hypothesis setting, we set $H=1$ to compare with other deterministic methods.
While our method primarily focuses on multi-hypothesis scenarios, we propose 1
hypothesis and use the CHAMP-Naive variation, without using the conformal
prediction pipeline. As the results suggest in Table 1, our method achieves
performance on par with the current state-of-the-art methods in the single-
hypothesis case.

In the multi-hypothesis setting, we set $H=80$. CHAMP variants achieve SOTA
results, especially when combined with more sophisticated aggregations
proposed in D3DP (Shan et al., 2023). On average, without the conformal
prediction pipeline, CHAMP-Naive is able to achieve 40mm MPJPE error and
improves by 1.5mm when using conformal prediction and mean aggregation. Using
J-Agg and J-Best, CHAMP gets further improved by 1.7mm and 3.6mm respectively.

Mean Per-Joint Position Error (Protocol # 1) - mm
---
| AVG. | Dir | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT.
Single Hypothesis
Ray3D (Zhan et al., 2022) | 49.7 | 44.7 | 48.7 | 48.7 | 48.4 | 51.0 | 59.9 | 46.8 | 46.9 | 58.7 | 61.7 | 50.2 | 46.4 | 51.5 | 38.6 | 41.8
STE (Li et al., 2022a) | 43.6 | 39.9 | 43.4 | 40.0 | 40.9 | 46.4 | 50.6 | 42.1 | 39.8 | 55.8 | 61.6 | 44.9 | 43.3 | 44.9 | 29.9 | 30.3
P-STMO (Shan et al., 2022) | 42.8 | 38.9 | 42.7 | 40.4 | 41.1 | 45.6 | 49.7 | 40.9 | 39.9 | 55.5 | 59.4 | 44.9 | 42.2 | 42.7 | 29.4 | 29.4
MixSTE (Zhang et al., 2022b) | 41.0 | 37.9 | 40.1 | 37.5 | 39.4 | 43.3 | 50.0 | 39.8 | 39.9 | 52.5 | 56.6 | 42.4 | 40.1 | 40.5 | 27.6 | 27.7
DUE (Zhang et al., 2022a) | 40.6 | 37.9 | 41.9 | 36.8 | 39.5 | 40.8 | 49.2 | 40.1 | 40.7 | 47.9 | 53.5 | 40.2 | 41.1 | 40.3 | 30.8 | 28.6
D3DP (Shan et al., 2023) | 40.0 | 37.7 | 39.9 | 35.7 | 38.2 | 41.9 | 48.8 | 39.5 | 38.3 | 50.5 | 53.9 | 41.6 | 39.4 | 39.8 | 27.4 | 27.5
CHAMP-Naive (H=1) | 40.2 | 39.5 | 40.3 | 35.8 | 38.7 | 42.2 | 48.7 | 40.2 | 38.2 | 50.3 | 53.2 | 41.5 | 39.2 | 40.2 | 26.9 | 27.3
Multiple Hypotheses
MHFormer (Li et al., 2022b) | 43.0 | 39.2 | 43.1 | 40.1 | 40.9 | 44.9 | 51.2 | 40.6 | 41.3 | 53.5 | 60.3 | 43.7 | 41.1 | 43.8 | 29.8 | 30.6
GraphMDN (Oikarinen et al., 2021) | 61.3 | 51.9 | 56.1 | 55.3 | 58.0 | 63.5 | 75.1 | 53.3 | 56.5 | 69.4 | 92.7 | 60.1 | 58.0 | 65.5 | 49.8 | 53.6
DiffuPose (Choi et al., 2023) | 49.4 | 43.4 | 50.7 | 45.5 | 50.2 | 49.6 | 53.4 | 48.6 | 45.0 | 56/9 | 70.7 | 47.8 | 48.2 | 51.3 | 43.1 | 43.4
DiffPose (Gong et al., 2023) | 36.9 | 33.2 | 36.6 | 33.0 | 35.6 | 37.6 | 45.1 | 35.7 | 35.5 | 46.4 | 49.9 | 37.3 | 35.6 | 36.5 | 24.4 | 24.1
DiffPose (Feng et al., 2023) | 43.3 | 38.1 | 43.3 | 35.3 | 43.1 | 46.6 | 48.2 | 39.0 | 37.6 | 51.9 | 59.3 | 41.7 | 47.6 | 45.5 | 37.4 | 36.0
D3DP-Agg (Shan et al., 2023) | 39.5 | 37.3 | 39.5 | 35.6 | 37.8 | 41.4 | 48.2 | 39.1 | 37.6 | 49.9 | 52.8 | 41.2 | 39.2 | 39.4 | 27.2 | 27.1
CHAMP-Naive | 40.0 | 39.4 | 40.1 | 35.4 | 38.2 | 42.0 | 48.5 | 40.0 | 38.1 | 50.1 | 52.9 | 43.4 | 39.1 | 39.9 | 27.7 | 28.2
CHAMP | 38.5 | 36.5 | 38.9 | 34.2 | 37.1 | 40.1 | 46.9 | 38.2 | 36.5 | 47.9 | 50.7 | 40.1 | 37.6 | 37.6 | 25.7 | 26.4
CHAMP-Agg | 36.0 | 36.2 | 36.1 | 30.9 | 31.8 | 39.5 | 43.6 | 35.1 | 33.9 | 43.1 | 47.2 | 39.9 | 34.9 | 36.1 | 25.1 | 25.9
D3DP-Best∗ (Shan et al., 2023) | 35.4 | 33.0 | 34.8 | 31.7 | 33.1 | 37.5 | 43.7 | 34.8 | 33.6 | 45.7 | 47.8 | 37.0 | 35.0 | 35.0 | 24.3 | 24.1
CHAMP-Best∗ | 34.9 | 35.9 | 35.2 | 30.3 | 31.4 | 37.4 | 42.9 | 34.4 | 33.5 | 41.5 | 46.9 | 37.9 | 34.8 | 34.8 | 24.5 | 24.6

Table 1: MPJPE ($\downarrow$) results on Human-3.6M dataset. Red: lowest
error. Blue: second lowest error. In the single-hypothesis setting, our method
runs without CP. In the multi-hypothesis setting, we compare four different
variants of CHAMP as discussed in the main text. ∗Upper-bound performance,
uses ground truth.

| PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$
---|---|---|---
Single Hypothesis
P-STMO (Shan et al., 2022) | 97.9 | 75.8 | 32.2
MixSTE (Zhang et al., 2022b) | 96.9 | 75.8 | 35.4
D3DP (Shan et al., 2023) | 97.7 | 77.8 | 30.2
CHAMP-Naive | 97.5 | 77.7 | 30.2
Multiple Hypotheses
MHFormer (Li et al., 2022b) | 93.8 | 63.3 | 58.0
DiffPose (Gong et al., 2023) | 98.0 | 75.9 | 29.1
DiffPose (Feng et al., 2023) | 94.6 | 62.8 | 64.6
D3DP-Agg (Shan et al., 2021) | 97.7 | 78.2 | 29.7
CHAMP-Naive | 97.5 | 78.1 | 29.9
CHAMP | 97.9 | 78.4 | 29.1
CHAMP-Agg | 98.0 | 78.6 | 28.6
D3DP-Best (Shan et al., 2021) | 98.0 | 79.1 | 28.1
CHAMP-Best | 98.2 | 79.1 | 28.0

Table 2: Results on the 3DHP dataset.

###  5.2 Results on MPI-INF-3DHP

We discuss the results on the MPI-INF-3DHP dataset in Table 2. In the single-
hypothesis setting, CHAMP-Naive achieves results on par with other
deterministic methods for the three metrics.

In the multi-hypothesis setting, we again set $H=80$. CHAMP-Naive again yields
fairly competitive results and when combined with conformal prediction and
mean, J-Agg, and J-Best aggregation techniques, the performance continuously
gets improved across the three metrics, with the CHAMP-Agg metric being the
most competitive one, achieving SOTA results. These results here on this
dataset follow the same trend as those on the Human3.6M dataset.

###  5.3 In-the-Wild Videos

To show the generalizability of our method to in-the-wild videos, we collect
videos from YouTube, TikTok, and 3DPW dataset (Von Marcard et al., 2018). To
construct 2D keypoints input, we use Cascaded Pyramid Network (Chen et al.,
2018) fine-tuned on the Human3.6M dataset as the default weights are trained
on MSCOCO (Lin et al., 2014), which has a slightly different skeleton
structure. We directly apply the CHAMP model trained on the Human3.6M dataset
to test on in-the-wild videos. Sample results are shown in Fig. 1, where the
input videos are collected from TikTok. Qualitative results suggest that CHAMP
is able to filter out outlier hypotheses using the conformity score function
trained end-to-end with the pose estimation model. More results are shown in
Appendix F.

###  5.4 Ablation Studies with Human3.6M Dataset

subfloatrowsep=qquad [3]

Figure 3: Your Figure Caption Here

| E2E | Sep. | Fix.
---|---|---|---
MPJPE | 34.9 | 35.1 | 35.5

Figure 4: Comparison of conformity scores. Top: filtered hypotheses of two
joints using the three scoring functions. Bottom: MPJPE (mm) values.

Figure 5: Comparison of #hypotheses in training and inference with 4 variants
of CHAMP. Red: Naive, Yellow: CHAMP, Brown: Agg, Green: Best.

Figure 6: Comparison of $\lambda$ in loss definition $\mathcal{L}$ (Eq. 10)
with 4 variants of CHAMP. Red: Naive, Yellow: CHAMP, Brown: Agg, Green: Best.

Choice of the Conformity Function.  We compare our proposed end-to-end learned
conformity score function with a separately trained conformity score function
(Corso et al., 2022), and a hand-designed conformity score function. We
compare the average performance across all actions on the Human3.6M dataset.
For the separately trained conformity score function, we first only train the
pose estimation backbone and generate 100 hypotheses per input sequence over a
small subset of the training dataset. We then train the score function to
predict if the proposed hypothesis conditioned on the input 2D keypoints
results in an MPJPE of less than 25mm and use the ground-truth pose sequence
as supervision. For the hand-designed conformity score, we use
$\phi_{\text{peak}}$ from (Yang & Pavone, 2023), measuring the maximum MPJPE
with respect to the ground truth across output frames. Note
$\phi_{\text{peak}}$ measures “nonconformity”, so we negate such values to fit
our setup. All three variants use the J-Best aggregation method. From Fig. 6,
E2E score function yields the best performance and the two learned variants
supersede the hand-designed version. In the two examples shown in Fig. 6, the
filtered hypotheses seem to be more concentrated for learned score functions.
It’s worth noting that all three variants result in competitive performance,
demonstrating the importance of CP.

Number of Hypotheses.  Another interesting ablation study to conduct is to
ablate on the number of hypotheses during training ($H_{\text{train}}$) and
inference ($H$). We compare the effects of $H_{\text{train}}$, $H$ in Fig. 6,
on the four variants of CHAMP. To compare the effect of $H_{\text{train}}$, we
fix $H=80$ and train a model for each value of hypotheses proposed during
differentiable CP. Similarly, to compare the effect of $H$, we fix
$H_{\text{train}}=20$ during training and use the same model with different
$H$ during inference for CP. Results suggest that a good performance can be
achieved with $H_{\text{train}}=20$ during training, while any number higher
than 20 brings marginal improvement and requires more GPU memory. Moreover,
$H=80$ during inference is sufficient for all four variants.

Strength of the Inefficiency Loss. Finally, we ablate on the value of
$\lambda$ in the overall training objective $\mathcal{L}$. This is an
important ablation in that we can find suitable strength of the inefficiency
loss $\mathcal{L}_{\text{ineff}}$ to make sure it does not conflict
$\mathcal{L}_{\text{pose}}$. Results in Fig. 6 suggest that $\lambda=0.05$ is
the most efficient strength across all values, as smaller $\lambda$ does not
train the scoring model sufficiently and higher $\lambda$ conflicts with the
pose loss. This corroborates the smaller-scale hyperparameter sweep
experiments we conducted before training the models.

###  5.5 Exchangeability and CP Guarantee

The inherent assumption of CP requires the calibration dataset to be
exchangeable, which is weaker than asking them to be independent (Angelopoulos
& Bates, 2021). This assumption typically fails when the calibration set is a
single video sequence, where the image frames are temporally correlated. We
learn a sequence-to-sequence model, where the input and output are long
sequences of image frames from different videos, instead of single frames from
the same video. This would drastically decrease the temporal correlation
across the calibration dataset. Moreover, as shown in (Yang & Pavone, 2023),
the video frames tend to be more independent if the videos are taken by
multiple evenly spaced cameras, which is the case for our datasets of
interest. We also investigate the empirical coverage of our method in Human
3.6M test set in Appendix A. Results suggest that we are able to inherit the
coverage guarantee from CP with the learned conformity score even if the
dataset is not fully exchangeable.

###  5.6 Implementation and Training Details

CHAMP’s denoiser model uses Adam optimizer with a weight decay parameter of
0.1 and momentum parameters $\beta_{1}=\beta_{2}=0.99$. For the training
objective in eq. (10), we use $\lambda=0.05$. We train the CHAMP model using
an NVIDIA V100 GPU, where the training process consumes an amortized GPU
memory of 18GB, for 300 epochs with a batch size of 8 and a learning rate of
5e-5 and reduce it on plateau with a factor of 0.5. Following previous work
(Zhang et al., 2022b; Shan et al., 2021, 2023), we use input pose sequence of
243 frames ($N=243$) of Human3.6M universal skeleton format. During training,
the number of hypotheses is 20, and #DDIM iterations is set to 1. During
inference, they are set to 80 and 10. The maximum number of diffusion steps is
$T=999$.

##  6 Limitations

While our method shows a promising improvement over the current 3D human pose
estimation methods, it does have several limitations. First, the theoretical
coverage guarantee of CP cannot be fully justified due to the input videos’
temporal correlation. Moreover, many hypotheses need to be proposed during
training to improve the learned conformity score, which consumes a lot more
GPU memory. Lastly, CHAMP only learns single-human 3D skeleton estimation,
without considering dense, multi-human scenarios.

##  7 Conclusion

This work presents CHAMP, a novel method for learning multi-hypotheses 3D
human poses with a learned conformal predictor from 2D keypoints. We
empirically show that CHAMP achieves competitive results across multiple
metrics and datasets, and when combined with more sophisticated downstream
aggregation methods, it achieves state-of-the-art performance. Future work
includes using more recent CP techniques that relax the exchangeability
assumption (Barber et al., 2023), using more efficient sequence-to-sequence
models such as Mamba (Gu & Dao, 2023), and scaling the pipeline up to dense,
multi-human pose-shape joint prediction scenarios (Loper et al., 2023).

## References

  * Angelopoulos et al. (2024) Anastasios Angelopoulos, Emmanuel Candes, and Ryan J Tibshirani.  Conformal pid control for time series prediction.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Angelopoulos & Bates (2021) Anastasios N Angelopoulos and Stephen Bates.  A gentle introduction to conformal prediction and distribution-free uncertainty quantification.  _arXiv preprint arXiv:2107.07511_ , 2021.
  * Angelopoulos et al. (2022) Anastasios N Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano.  Image-to-image regression with distribution-free uncertainty quantification and applications in imaging.  In _International Conference on Machine Learning_ , pp. 717–730. PMLR, 2022.
  * Avigal et al. (2020) Yahav Avigal, Samuel Paradis, and Harry Zhang.  6-dof grasp planning using fast 3d reconstruction and grasp quality cnn.  _arXiv preprint arXiv:2009.08618_ , 2020.
  * Avigal et al. (2021) Yahav Avigal, Vishal Satish, Zachary Tam, Huang Huang, Harry Zhang, Michael Danielczuk, Jeffrey Ichnowski, and Ken Goldberg.  Avplug: Approach vector planning for unicontact grasping amid clutter.  In _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_ , pp. 1140–1147. IEEE, 2021.
  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18208–18218, 2022.
  * Bai et al. (2022) Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Efficient and differentiable conformal prediction with general function classes.  _arXiv preprint arXiv:2202.11091_ , 2022.
  * Barber et al. (2023) Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani.  Conformal prediction beyond exchangeability.  _The Annals of Statistics_ , 51(2):816–845, 2023.
  * Batzolis et al. (2021) Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann.  Conditional image generation with score-based diffusion models.  _arXiv preprint arXiv:2111.13606_ , 2021.
  * Bhatnagar et al. (2023) Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai.  Improved online conformal prediction via strongly adaptive online learning.  In _International Conference on Machine Learning_ , pp. 2337–2363. PMLR, 2023.
  * Blondel et al. (2020) Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga.  Fast differentiable sorting and ranking.  In _International Conference on Machine Learning_ , pp. 950–959. PMLR, 2020.
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7291–7299, 2017.
  * Chen et al. (2018) Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7103–7112, 2018.
  * Choi et al. (2023) Jeongjun Choi, Dongseok Shim, and H Jin Kim.  Diffupose: Monocular 3d human pose estimation via denoising diffusion probabilistic model.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 3773–3780. IEEE, 2023.
  * Corso et al. (2022) Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola.  Diffdock: Diffusion steps, twists, and turns for molecular docking.  _arXiv preprint arXiv:2210.01776_ , 2022.
  * Cuturi et al. (2019) Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert.  Differentiable ranking and sorting using optimal transport.  _Advances in neural information processing systems_ , 32, 2019.
  * Devgon et al. (2020) Shivin Devgon, Jeffrey Ichnowski, Ashwin Balakrishna, Harry Zhang, and Ken Goldberg.  Orienting novel 3d objects using self-supervised learning of rotation transforms.  In _2020 IEEE 16th International Conference on Automation Science and Engineering (CASE)_ , pp. 1453–1460. IEEE, 2020.
  * Eisner et al. (2022) Ben Eisner, Harry Zhang, and David Held.  Flowbot3d: Learning 3d articulation flow to manipulate articulated objects.  _arXiv preprint arXiv:2205.04382_ , 2022.
  * Elmquist et al. (2022) Asher Elmquist, Aaron Young, Thomas Hansen, Sriram Ashokkumar, Stefan Caldararu, Abhiraj Dashora, Ishaan Mahajan, Harry Zhang, Luning Fang, He Shen, et al.  Art/atk: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineering.  _arXiv preprint arXiv:2211.04886_ , 2022.
  * Feldman et al. (2022) Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano.  Achieving risk control in online learning settings.  _arXiv preprint arXiv:2205.09095_ , 2022.
  * Feng et al. (2023) Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma, and Hyung Jin Chang.  Diffpose: Spatiotemporal diffusion model for video-based human pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 14861–14872, 2023.
  * Fisch et al. (2021) Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay.  Few-shot conformal prediction with auxiliary tasks.  In _International Conference on Machine Learning_ , pp. 3329–3339. PMLR, 2021.
  * Gibbs & Candès (2022) Isaac Gibbs and Emmanuel Candès.  Conformal inference for online prediction with arbitrary distribution shifts.  _arXiv preprint arXiv:2208.08401_ , 2022.
  * Gong et al. (2023) Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu.  Diffpose: Toward more reliable 3d pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13041–13051, 2023.
  * Gu & Dao (2023) Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  _arXiv preprint arXiv:2312.00752_ , 2023.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. (2022) Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _Journal of Machine Learning Research_ , 23(47):1–33, 2022.
  * Hu et al. (2021) Wenbo Hu, Changgong Zhang, Fangneng Zhan, Lei Zhang, and Tien-Tsin Wong.  Conditional directed graph convolution for 3d human pose estimation.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 602–611, 2021.
  * Huang et al. (2023) Huang Huang, Satvik Sharma, Antonio Loquercio, Anastasios Angelopoulos, Ken Goldberg, and Jitendra Malik.  Conformal policy learning for sensorimotor control under distribution shifts.  _arXiv preprint arXiv:2311.01457_ , 2023.
  * Huang et al. (2022) Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pp. 2595–2605, 2022.
  * Ionescu et al. (2013) Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(7):1325–1339, 2013.
  * Jin et al. (2024) David Jin, Sushrut Karmalkar, Harry Zhang, and Luca Carlone.  Multi-model 3d registration: Finding multiple moving objects in cluttered point clouds.  _arXiv preprint arXiv:2402.10865_ , 2024.
  * Kim et al. (2022) Heeseung Kim, Sungwon Kim, and Sungroh Yoon.  Guided-tts: A diffusion model for text-to-speech via classifier guidance.  In _International Conference on Machine Learning_ , pp. 11119–11133. PMLR, 2022.
  * Levkovitch et al. (2022) Alon Levkovitch, Eliya Nachmani, and Lior Wolf.  Zero-shot voice conditioning for denoising diffusion tts models.  _arXiv preprint arXiv:2206.02246_ , 2022.
  * Li & Lee (2019) Chen Li and Gim Hee Lee.  Generating multiple hypotheses for 3d human pose estimation with mixture density network.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9887–9895, 2019.
  * Li & Lee (2020) Chen Li and Gim Hee Lee.  Weakly supervised generative network for multiple 3d human pose hypotheses.  _arXiv preprint arXiv:2008.05770_ , 2020.
  * Li et al. (2022a) Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, and Wenming Yang.  Exploiting temporal contexts with strided transformer for 3d human pose estimation.  _IEEE Transactions on Multimedia_ , 25:1282–1293, 2022a.
  * Li et al. (2022b) Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool.  Mhformer: Multi-hypothesis transformer for 3d human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13147–13156, 2022b.
  * Lim et al. (2021) Vincent Lim, Huang Huang, Lawrence Yunliang Chen, Jonathan Wang, Jeffrey Ichnowski, Daniel Seita, Michael Laskey, and Ken Goldberg.  Planar robot casting with real2sim2real self-supervised learning.  _arXiv preprint arXiv:2111.04814_ , 2021.
  * Lim et al. (2022) Vincent Lim, Huang Huang, Lawrence Yunliang Chen, Jonathan Wang, Jeffrey Ichnowski, Daniel Seita, Michael Laskey, and Ken Goldberg.  Real2sim2real: Self-supervised learning of physical single-step dynamic actions for planar robot casting.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pp. 8282–8289. IEEE, 2022.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Loper et al. (2023) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pp. 851–866. 2023.
  * Ma et al. (2022) Xianzheng Ma, Hossein Rahmani, Zhipeng Fan, Bin Yang, Jun Chen, and Jun Liu.  Remote: Reinforced motion transformation network for semi-supervised 2d pose estimation in videos.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 1944–1952, 2022.
  * Martinez et al. (2017) Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little.  A simple yet effective baseline for 3d human pose estimation.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2640–2649, 2017.
  * Mehta et al. (2017) Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt.  Monocular 3d human pose estimation in the wild using improved cnn supervision.  In _2017 international conference on 3D vision (3DV)_ , pp. 506–516. IEEE, 2017.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Oikarinen et al. (2021) Tuomas Oikarinen, Daniel Hannah, and Sohrob Kazerounian.  Graphmdn: Leveraging graph structure and deep learning to solve inverse problems.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–9. IEEE, 2021.
  * Pan et al. (2022) Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held.  Tax-pose: Task-specific cross-pose estimation for robot manipulation.  _arXiv preprint arXiv:2211.09325_ , 2022.
  * Pan et al. (2023) Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held.  Tax-pose: Task-specific cross-pose estimation for robot manipulation.  In _Conference on Robot Learning_ , pp. 1783–1792. PMLR, 2023.
  * Pavlakos et al. (2017) Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis.  Coarse-to-fine volumetric prediction for single-image 3d human pose.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7025–7034, 2017.
  * Pavllo et al. (2019) Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.  3d human pose estimation in video with temporal convolutions and semi-supervised training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 7753–7762, 2019.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Rommel et al. (2023) Cédric Rommel, Eduardo Valle, Mickaël Chen, Souhaiel Khalfaoui, Renaud Marlet, Matthieu Cord, and Patrick Pérez.  Diffhpe: Robust, coherent 3d human pose lifting with diffusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3220–3229, 2023.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 conference proceedings_ , pp. 1–10, 2022.
  * Shafer & Vovk (2008) Glenn Shafer and Vladimir Vovk.  A tutorial on conformal prediction.  _Journal of Machine Learning Research_ , 9(3), 2008.
  * Shan et al. (2021) Wenkang Shan, Haopeng Lu, Shanshe Wang, Xinfeng Zhang, and Wen Gao.  Improving robustness and accuracy via relative information encoding in 3d human pose estimation.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 3446–3454, 2021.
  * Shan et al. (2022) Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao.  P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation.  In _European Conference on Computer Vision_ , pp. 461–478. Springer, 2022.
  * Shan et al. (2023) Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang, Siwei Ma, and Wen Gao.  Diffusion-based 3d human pose estimation with multi-hypothesis aggregation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 14761–14771, 2023.
  * Sharma et al. (2019) Saurabh Sharma, Pavan Teja Varigonda, Prashast Bindal, Abhishek Sharma, and Arjun Jain.  Monocular 3d human pose estimation by generation and ordinal ranking.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 2325–2334, 2019.
  * Shen et al. (2024) Sitian Shen, Zilin Zhu, Linqian Fan, Harry Zhang, and Xinxiao Wu.  Diffclip: Leveraging stable diffusion for language grounded 3d classification.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3596–3605, 2024.
  * Sim et al. (2019) Khe Chai Sim, Françoise Beaufays, Arnaud Benard, Dhruv Guliani, Andreas Kabel, Nikhil Khare, Tamar Lucassen, Petr Zadrazil, Harry Zhang, Leif Johnson, et al.  Personalization of end-to-end speech recognition on mobile devices for named entities.  In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 23–30. IEEE, 2019.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pp. 2256–2265. PMLR, 2015.
  * Song & Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Stutz et al. (2021) David Stutz, Ali Taylan Cemgil, Arnaud Doucet, et al.  Learning optimal conformal classifiers.  _arXiv preprint arXiv:2110.09192_ , 2021.
  * Sun et al. (2024) Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Nobel, Mykel J Kochenderfer, and Mac Schwager.  Conformal prediction for uncertainty-aware planning with diffusion dynamics model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Tekin et al. (2016) Bugra Tekin, Artem Rozantsev, Vincent Lepetit, and Pascal Fua.  Direct prediction of 3d body poses from motion compensated sequences.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 991–1000, 2016.
  * Teneggi et al. (2023) Jacopo Teneggi, Matthew Tivnan, Web Stayman, and Jeremias Sulam.  How to trust your diffusion model: A convex optimization approach to conformal risk control.  In _International Conference on Machine Learning_ , pp. 33940–33960. PMLR, 2023.
  * Von Marcard et al. (2018) Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 601–617, 2018.
  * Wehrbein et al. (2021) Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt.  Probabilistic monocular 3d human pose estimation with normalizing flows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 11199–11208, 2021.
  * Wu et al. (2023) Yue Wu, Yongzhe Yuan, Xiaolong Fan, Xiaoshui Huang, Maoguo Gong, and Qiguang Miao.  Pcrdiffusion: Diffusion probabilistic models for point cloud registration.  _arXiv preprint arXiv:2312.06063_ , 2023.
  * Xu & Takano (2021) Tianhan Xu and Wataru Takano.  Graph stacked hourglass networks for 3d human pose estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16105–16114, 2021.
  * Yang & Pavone (2023) Heng Yang and Marco Pavone.  Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8947–8958, 2023.
  * Yao et al. (2023) Yupu Yao, Shangqi Deng, Zihan Cao, Harry Zhang, and Liang-Jian Deng.  Apla: Additional perturbation for latent noise with adversarial training enables consistency.  _arXiv preprint arXiv:2308.12605_ , 2023.
  * Zeng et al. (2020) Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin.  Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16_ , pp. 507–523. Springer, 2020.
  * Zhan et al. (2022) Yu Zhan, Fenghai Li, Renliang Weng, and Wongun Choi.  Ray3d: ray-based 3d human pose estimation for monocular absolute 3d localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13116–13125, 2022.
  * Zhang (2016) Haolun Zhang.  Health diagnosis based on analysis of data captured by wearable technology devices.  _International Journal of Advanced Science and Technology_ , 95:89–96, 2016.
  * Zhang et al. (2020) Harry Zhang, Jeffrey Ichnowski, Yahav Avigal, Joseph Gonzales, Ion Stoica, and Ken Goldberg.  Dex-net ar: Distributed deep grasp planning using a commodity cellphone and augmented reality app.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 552–558. IEEE, 2020.
  * Zhang et al. (2021) Harry Zhang, Jeffrey Ichnowski, Daniel Seita, Jonathan Wang, Huang Huang, and Ken Goldberg.  Robots of the lost arc: Self-supervised learning to dynamically manipulate fixed-endpoint cables.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 4560–4567. IEEE, 2021.
  * Zhang et al. (2023) Harry Zhang, Ben Eisner, and David Held.  Flowbot++: Learning generalized articulated objects manipulation via articulation projection.  _arXiv preprint arXiv:2306.12893_ , 2023.
  * Zhang et al. (2022a) Jinlu Zhang, Yujin Chen, and Zhigang Tu.  Uncertainty-aware 3d human pose estimation from monocular video.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pp. 5102–5113, 2022a.
  * Zhang et al. (2022b) Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, and Junsong Yuan.  Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 13232–13242, 2022b.
  * Zhao et al. (2019) Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas.  Semantic graph convolutional networks for 3d human pose regression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 3425–3435, 2019.
  * Zheng et al. (2021) Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding.  3d human pose estimation with spatial and temporal transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 11656–11665, 2021.
  * Zhou et al. (2023) Jieming Zhou, Tong Zhang, Zeeshan Hayder, Lars Petersson, and Mehrtash Harandi.  Diff3dhpe: A diffusion model for 3d human pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2092–2102, 2023.
  * Zhou et al. (2017) Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei.  Towards 3d human pose estimation in the wild: a weakly-supervised approach.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 398–407, 2017.

##  Appendix A Coverage Guarantee and Empirical Coverage

Please refer to the standard proof shown in (Angelopoulos & Bates, 2021;
Shafer & Vovk, 2008) for the coverage guarantees of conformal prediction.

To evaluate the empirical coverage of CHAMP, we calculate the mean coverage
percentage across test data. Specifically, we use eq. (11) to check if the
ground truth belongs in the confidence set formed by the conformity function
on the test data:

$\displaystyle\bar{\mathcal{C}}$
$\displaystyle=\frac{1}{|{I}_{\text{test}}|}\sum_{\bm{y}_{\text{GT}}\sim{I}_{\text{test}}}\mathbbm{1}\left(\bm{y}_{\text{GT}}\in
C_{\theta}(\bm{y},\tau)\right)$
$\displaystyle=\frac{1}{|{I}_{\text{test}}|}\sum_{\bm{y}_{\text{GT}}\sim{I}_{\text{test}}}\mathbbm{1}\left(\phi_{\theta}(\tilde{\bm{y}},\bm{y}_{\text{GT}})\geq\tau\right)$

We compare the three choices of conformity scores for the empirical coverage
calculation. We use the test set from Human 3.6M and calculate the empirical
coverage values across all activities.

Figure 7: Empirical coverage comparison across three different conformity
functions. We keep the best $\lambda=0.05$ for comparison.

During training and testing, we keep $\alpha=0.1$ for CP. From Fig. 7, we see
the empirical coverage is around $90\%\pm 5\%$ for all activities, which
remains close to $1-\alpha$, and in some cases, it exceeds this value. This is
encouraging given the exchangeability assumption is not fully satisfied in our
dataset.

##  Appendix B Denoiser in Training vs Inference

We first revisit DDIM (Song & Ermon, 2020). We start with the reverse process
derivation and rewrite $q(\bm{y}_{t-1}|\bm{y}_{t},\bm{y}_{0})$ to be
parameterized by a desired standard deviation $\sigma_{t}$:

$\displaystyle\bm{y}_{t-1}$
$\displaystyle=\sqrt{\alpha_{t-1}}\bm{y}_{0}+\sqrt{1-\alpha_{t-1}}\bm{\epsilon}_{t-1}$
$\displaystyle=\sqrt{\alpha_{t-1}}\bm{y}_{0}+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\bm{\epsilon}_{t-1}+\sigma_{t}\bm{\epsilon}$
$\displaystyle=\sqrt{\alpha_{t-1}}\left(\bm{y}_{t}-\sqrt{1-\alpha_{t}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right)+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})+\sigma_{t}\bm{\epsilon}$
$\displaystyle=\sqrt{\alpha_{t-1}}\left(\bm{y}_{t}-\sqrt{1-\alpha_{t}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right)+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})+\sigma_{t}\bm{\epsilon}$

where the model $\bm{\epsilon}^{(t)}_{\theta}(\cdot)$ predicts the
$\bm{\epsilon}_{t}$ from $\bm{y}_{t}$. Since
$q(\bm{y}_{t-1}|\bm{y}_{t},\bm{y}_{0})=\mathcal{N}(\bm{y}_{t-1};\mu(\bm{y}_{t},\bm{y}_{0}),\beta_{t}\mathbf{I})$,
therefore we have:

$\tilde{\beta}_{t}=\sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$

Let $\sigma_{t}^{2}=\eta\cdot\beta_{t}$ where $\eta$ is a hyperparameter that
controls the sampling stochasticity. During generation, we don’t have to
follow the whole chain $t=1,\ldots,T$, but only a subset of steps. Denote
$t^{\prime}<t$ as two steps in this accelerated trajectory. The DDIM update
step is defined as follows:

$q_{t^{\prime},\theta}(\bm{y}_{t^{\prime}}|\bm{y}_{t},\bm{y}_{0})=\mathcal{N}\left(\bm{x}_{t^{\prime}};\sqrt{\alpha_{t^{\prime}}}\left(\bm{y}_{t}-\sqrt{1-\alpha_{t}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right),\sigma_{t}^{2}\mathbf{I}\right)$

Following D3DP (Shan et al., 2023), the denoiser model $D_{\theta}$ uses DDIM
to sample denoised poses from the corrupted ones. During training, we run DDIM
for only 1 step for the sake of efficiency:

$\bar{\bm{y}}^{h}=D_{\theta}({\bm{y}}^{h}_{T},\bm{x},T),\;{\bm{y}}^{h}_{T}\sim\mathcal{N}(0,\mathbf{I})\quad\forall
h=\\{1,\cdots,H\\}$

During inference, we run DDIM for $K=10$ times, and each step is defined as:

$\displaystyle t=T\cdot(1-k/K),\;\;t^{\prime}$
$\displaystyle=T\cdot(1-(k+1)/K)),\;\;k=\\{0,\cdots,K-1\\}$
$\displaystyle\bar{\bm{y}}^{h}$
$\displaystyle=D_{\theta}(\bar{\bm{y}}^{h}_{t},\bm{x},t),\quad\forall
h=\\{1,\cdots,H\\}$ $\displaystyle\bar{\bm{y}}^{h}_{t^{\prime}}$
$\displaystyle=\sqrt{\overline{\alpha}_{t^{\prime}}}\cdot\bar{\bm{y}}^{h}+\sqrt{1-\overline{\alpha}_{t^{\prime}}-\sigma_{t}^{2}}\cdot\bm{\epsilon}_{t}+\sigma_{t}\bm{\epsilon}$
$\displaystyle t$ $\displaystyle\leftarrow t^{\prime}$

##  Appendix C Training-Calibration-Testing Data Split

We discuss the split of training, calibration, and testing data. We first
split the testing data the same way as (Zhang et al., 2022b; Shan et al.,
2023; Gong et al., 2023) to ensure the fairness of results. We then further
split the training set into the actual training dataset and a calibration
dataset before inference. Specifically, we split the training dataset by
uniformly sampling a 2% subset as the calibration dataset, and CHAMP is only
trained on the remaining 98%. Note here that this 2% calibration dataset is
not seen during training by any means. During training, we split each mini-
batch evenly into $B_{\text{pred}}$ and $B_{\text{cal}}$.

##  Appendix D Architecture Details

We provide a more detailed version of Fig. 2. We illustrate the detailed
architecture of the denoiser model $D_{\theta}$ as well as the conformity
scoring model $s_{\theta}$ in Fig. 8. MixSTE (Zhang et al., 2022b) is used as
the backbone of the denoiser. MixSTE combines 16 alternative spatial and
temporal attention blocks. In the implementation, the channel size is set to
512. Similar to (Shan et al., 2023; Gong et al., 2023), we concatenate 2D
keypoints and noisy 3D poses as inputs and add a diffusion timestep embedding
after the input embedding using a sinusoidal function. For the conformity
scoring model, we concatenate 2D keypoints and predicted hypotheses 3D poses
as inputs, reuse the input embedding layer, and use an extra MLP layer to
project the input embeddings into a lower dimension of 3. In training, CP is
made differentiable by using soft ranking and soft assignment, resulting in a
differentiable inefficiency loss $\mathcal{L}_{\text{ineff}}$. This loss is
combined with the pose loss $\mathcal{L}_{\text{pose}}$ when backpropagating
into the network. During inference (light cyan shaded area), we refine the
hypotheses confidence set with regular CP on a held-out calibration set,
resulting in $H^{\prime}<H$ hypotheses. We then aggregate the refined set,
resulting in a single final prediction.

Figure 8: Detailed architecture of $D_{\theta}$ and $s_{\theta}$, the denoiser
and conformity score models during training and inference. During training,
the construction of refined hypotheses set after differentiable CP is done via
soft assignment. During inference, regular CP is used with the trained
conformity score function.

##  Appendix E Procrustes-MPJPE Performance

The Procrustes MPJPE (P-MPJPE) metric, which is often referred to as Protocol
#2, computes MPJPE after the estimated poses align to the ground truth using a
rigid transformation. It is another standard metric for measuring a 3D pose
estimator’s performance (Pan et al., 2023). We report the performance of the
four variants of CHAMP against other baselines on the Human3.6M dataset in
Table 3. Quantitative results suggest our method achieves SOTA performance on
this metric as well.

Procrustes Per-Joint Position Error (Protocol # 2) - mm
---
| AVG. | Dir | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT.
Single Hypothesis
STE Li et al. (2022a) | 35.2 | 32.7 | 35.5 | 32.5 | 35.4 | 35.9 | 41.6 | 33.0 | 31.9 | 45.1 | 50.1 | 36.4 | 33.5 | 35.1 | 23.9 | 25.0
P-STMO Shan et al. (2022) | 34.4 | 31.3 | 35.2 | 32.9 | 33.9 | 35.4 | 39.3 | 32.5 | 31.5 | 44.6 | 48.2 | 36.3 | 32.9 | 34.4 | 23.8 | 23.9
MixSTE Zhang et al. (2022b) | 32.6 | 30.8 | 33.1 | 30.3 | 31.8 | 33.1 | 39.1 | 31.1 | 30.5 | 42.5 | 44.5 | 34.0 | 30.8 | 32.7 | 22.1 | 22.9
DUE Zhang et al. (2022a) | 32.5 | 30.3 | 34.6 | 29.6 | 31.7 | 31.6 | 38.9 | 31.8 | 31.9 | 39.2 | 42.8 | 32.1 | 32.6 | 31.4 | 25.1 | 23.8
D3DP Shan et al. (2023) | 31.7 | 30.6 | 32.5 | 29.1 | 31.0 | 31.9 | 37.6 | 30.3 | 29.4 | 40.6 | 43.6 | 33.3 | 30.5 | 31.4 | 21.5 | 22.4
CHAMP-Naive | 31.2 | 30.9 | 32.1 | 29.4 | 30.9 | 31.2 | 35.9 | 29.4 | 28.9 | 40.1 | 41.9 | 33.1 | 30.0 | 30.8 | 20.9 | 22.4
Multiple Hypotheses
MHFormer Li et al. (2022b) | 34.4 | 34.9 | 32.8 | 33.6 | 35.3 | 39.6 | 32.0 | 32.2 | 43.5 | 48.7 | 36.4 | 32.6 | 34.3 | 23.9 | 25.1 | 34.4
GraphMDN Oikarinen et al. (2021) | 46.9 | 39.7 | 43.4 | 44.0 | 46.2 | 48.8 | 54.5 | 39.4 | 41.1 | 55.0 | 69.0 | 49.0 | 43.7 | 49.6 | 38.4 | 42.4
DiffuPose Choi et al. (2023) | 39.9 | 35.9 | 40.3 | 36.7 | 41.4 | 39.8 | 43.4 | 37.1 | 35.5 | 46.2 | 59.7 | 39.9 | 38.0 | 41.9 | 32.9 | 34.2
DiffPose Gong et al. (2023) | 28.7 | 26.3 | 29.0 | 26.1 | 27.8 | 28.4 | 34.6 | 26.9 | 26.5 | 36.8 | 39.2 | 29.4 | 26.8 | 28.4 | 18.6 | 19.2
DiffPose Feng et al. (2023) | 32.0 | 28.1 | 31.5 | 28.0 | 30.8 | 33.6 | 35.3 | 28.5 | 27.6 | 40.8 | 44.6 | 31.8 | 32.1 | 32.6 | 28.1 | 26.8
D3DP-Agg Shan et al. (2023) | 31.6 | 30.6 | 32.4 | 29.2 | 30.9 | 31.9 | 37.4 | 30.2 | 29.3 | 40.4 | 43.2 | 33.2 | 30.4 | 31.3 | 21.5 | 22.3
CHAMP-Naive | 32.0 | 31.9 | 33.1 | 30.8 | 31.1 | 32.2 | 38.1 | 29.2 | 28.3 | 41.2 | 43.1 | 34.5 | 31.2 | 30.4 | 22.6 | 23.9
CHAMP | 31.1 | 30.8 | 32.4 | 29.9 | 30.9 | 31.4 | 37.2 | 27.8 | 26.9 | 40.2 | 42.7 | 32.9 | 30.6 | 29.8 | 21.4 | 22.8
CHAMP-Agg | 30.2 | 30.1 | 31.7 | 28.6 | 30.1 | 30.2 | 36.1 | 26.9 | 25.4 | 39.1 | 41.4 | 31.9 | 29.8 | 28.9 | 20.9 | 21.9
D3DP-Best∗ Shan et al. (2023) | 28.7 | 27.5 | 29.4 | 26.6 | 27.7 | 29.2 | 34.3 | 27.5 | 26.2 | 37.3 | 39.0 | 30.3 | 27.7 | 28.2 | 19.6 | 20.3
CHAMP-Best∗ | 28.5 | 28.9 | 30.1 | 27.2 | 27.3 | 28.2 | 34.4 | 24.7 | 24.1 | 37.2 | 38.9 | 29.9 | 28.1 | 27.1 | 20.1 | 19.9

Table 3: P-MPJPE ($\downarrow$) results on Human-3.6M dataset. ∗Upper-bound
performance, needs ground truth.

##  Appendix F More Qualitative Results on In-the-Wild Videos

Figure 9: Results of running CHAMP on in-the-wild videos collected from TikTok
and YouTube.

We showcase the performance of CHAMP on more in-the-wild videos in this
section. We collect various YouTube and TikTok videos and detect 2D keypoints
with Cascaded Pyramid Network. Note that the official weights provided by CPN
are trained in COCO format, so we used a fine-tuned version of the weights for
Human3.6M universal skeleton format. With such 2D keypoints, we are able to
directly use the model trained on Human3.6M dataset and apply it to real-world
videos without any fine-tuning. In Fig. 9, for each row, we show two examples.
For every three images, the leftmost image is the RGB observation, and the
middle image is all the hypotheses proposed by CHAMP’s pose estimation
backbone, and the rightmost image is the hypotheses set refined by the learned
conformal predictor.

##  Appendix G Learned CP in CHAMP

We provide more demonstrations on the learned conformal predictor in CHAMP.
Specifically, in Fig. 10, we provide more examples of predicted hypotheses
before and after the conformal predictor powered by the learned score
function.

Figure 10: Examples of CP in CHAMP filtering out bad hypotheses during
inference. Qualitatively, the learned score function filters out outlier
predictions.