# Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric
Videos

Junyi Ma1 Jingyi Xu2 Xieyuanli Chen3 Hesheng Wang1
1IRMV Lab, Shanghai Jiao Tong University 2NAV, Shanghai Jiao Tong University
3College of Intelligence Science and Technology, National University of
Defense Technology Corresponding author<EMAIL_ADDRESS>
###### Abstract

Understanding how humans would behave during hand-object interaction is vital
for applications in service robot manipulation and extended reality. To
achieve this, some recent works have been proposed to simultaneously forecast
hand trajectories and object affordances on human egocentric videos. The joint
prediction serves as a comprehensive representation of future hand-object
interactions in 2D space, indicating potential human motion and motivation.
However, the existing approaches mostly adopt the autoregressive paradigm for
unidirectional prediction, which lacks mutual constraints within the holistic
future sequence, and accumulates errors along the time axis. Meanwhile, these
works basically overlook the effect of camera egomotion on first-person view
predictions. To address these limitations, we propose a novel diffusion-based
interaction prediction method, namely Diff-IP2D, to forecast future hand
trajectories and object affordances concurrently in an iterative non-
autoregressive manner. We transform the sequential 2D images into latent
feature space and design a denoising diffusion model to predict future latent
interaction features conditioned on past ones. Motion features are further
integrated into the conditional denoising process to enable Diff-IP2D aware of
the camera wearer’s dynamics for more accurate interaction prediction.
Extensive experiments demonstrate that our method significantly outperforms
the state-of-the-art baselines on both the off-the-shelf metrics and our newly
proposed evaluation protocol. This highlights the efficacy of leveraging a
generative paradigm for 2D hand-object interaction prediction. The code of
Diff-IP2D will be released at https://github.com/IRMVLab/Diff-IP2D.

##  1 Introduction

Accurately anticipating human intentions and future actions is important for
artificial intelligence systems in robotics and extended reality [1, 2, 3].
Recent works have tried to tackle the problem from various perspectives,
including action recognition and anticipation [4, 5, 6, 7], gaze prediction
[8, 9, 10, 11], hand trajectory prediction [12, 13, 14, 15], and object
affordance extraction [12, 16, 14, 17]. Among them, jointly predicting hand
motion and object affordances can effectively facilitate more reasonable robot
manipulation as the prior contextual information, which has been demonstrated
on some robot platforms [1, 18, 19]. We believe that deploying such models
pretrained by internet-scale human videos on robots is a promising path
towards embodied agents. Therefore, our work aims to jointly predict hand
trajectories and object affordances on egocentric videos as a concrete hand-
object interaction (HOI) expression, following the problem modeling of
previous works [12, 14].

Currently, the state-of-the-art approaches [12, 13] predicting hand
trajectories and object affordances on egocentric videos tend to exploit the
autoregressive (AR) model. They reason about the next HOI state only according
to the previous steps (Fig. 1(a)). However, expected “post-contact states”
also affect “pre-contact states” according to human intentions that persist
across the holistic HOI process as an oracle. Inspired by this, we argue that
predicting future HOI states in parallel considering the bidirectional
constraints within the holistic sequence outperforms generating the next state
autoregressively (Fig. 1(c)). With diffusion models emerging across multiple
domains [20, 21, 22, 23, 24, 25, 26, 27], their strong forecasting capability
has been widely validated. Therefore, we propose a diffusion-based method to
predict future hand-object interaction in parallel, considering bidirectional
constraints in the latent space compared to the traditional autoregressive
generation (Fig. 1(b)). In the forward process, the past and future video
images are first encoded to sequential latent features. Noises are gradually
added to the part of the future sequence while the past features remain
anchored. Subsequently, a Transformer-based network is devised for learning to
reverse the diffusion and reconstruct the input latent features. Finally, the
proposed predictors are exploited to recover future hand trajectories and
object affordances from the denoised latents. A new regularization strategy is
also proposed to link the two latent spaces adjacent to the denoising
diffusion process.

Moreover, we also identify two inherent gaps (Fig. 1(d)) affecting HOI
prediction in the existing paradigm: 1) Directly predicting the projection of
3D future hand trajectories and object affordances on 2D egocentric image
plane is an ill-posed problem involving spatial ambiguities. There is
generally a gap between 2D pixel movements and 3D real actions, which can be
bridged by spatial transformation across multiple views changing with
egomotion. 2) The past egocentric videos are absorbed to predict future
interaction states on the last observed image, which is actually a “canvas”
from a different view w.r.t all the other frames. Therefore, there is also a
gap between the last observation (egocentric view) and the other observations
(analogous to exocentric view) caused by egomotion. To fill the two gaps
together, we further propose to integrate the camera wearer’s egomotion into
our diffusion-based paradigm. It enables the denoising model aware of the
camera wearer’s dynamics and the spatial relationship between consecutive
frames.

The main contributions of this paper are as follows: 1) We propose a
diffusion-based hand-object interaction prediction method, dubbed Diff-IP2D.
To our best knowledge, this is the first work to jointly forecast future hand
trajectories and object affordances by the devised denoising diffusion
probabilistic model with only 2D egocentric videos as input. It provides a
foundation generative paradigm in the field of HOI prediction. 2) The
homography egomotion features are integrated to fill the motion-related gaps
inherent in HOI prediction on egocentric videos. 3) We extend the existing
metrics and propose the first protocol for jointly evaluating the performance
of hand trajectory prediction and object affordance prediction. 4)
Comprehensive experiments are conducted to demonstrate that our Diff-IP2D can
predict better hand trajectories and object affordances compared to the state-
of-the-art baselines, showing its potential for deployment on artificial
intelligence systems.

Figure 1: Diff-IP2D vs. Existing Paradigm. The existing HOI prediction
paradigm (a) tends to accumulate prediction errors under unidirectional
constraints. In contrast, our proposed Diff-IP2D (b) directly forecasts all
the future interaction states in parallel with denoising diffusion, mitigating
error accumulation with bidirectional constraints (c). Moreover, we integrate
egomotion information into our proposed paradigm to narrow the inherent gaps
(d) in HOI prediction.

##  2 Related work

#### Understanding hand-object interaction.

Human HOI comprehension can guide the downstream tasks in artificial
intelligence systems. As a pioneer work, Calway _et al._ [28] connect the
specific human tasks to relevant objects, revealing the importance of object-
centric understanding in different HOI modes. In contrast, Liu _et al._ [29]
focus on capturing the changeable attributes of objects, which underlines the
relationship between object-centric interaction and goal-oriented human
activities. After that, more and more works contribute to HOI understanding by
pixel-wise semantic segmentation [30, 31, 32, 33], bounding-box-wise detection
[34, 35, 36, 15], fine-grained hand/object pose estimation [38, 39, 40, 41,
42, 43]. Ego4D [44] further provides a standard benchmark that divides HOI
understanding into several predefined subtasks.

#### Predicting hand-object interaction.

Analyzing only past human behavior may be insufficient for service robot
manipulation or extended reality. Forecasting possible future object-centric
HOI states based on historical observations is also valuable, which attracts
increasing attention due to the general knowledge that can be transferred to
robot applications [1, 18, 19, 45]. For example, Dessalene _et al._ [46]
propose to generate contact anticipation maps and next active object
segmentations as future HOI predictions. Liu _et al._ [14] first achieve hand
trajectory and object affordance prediction simultaneously, revealing that
predicting hand motion benefits the extraction of interaction hotspots.
Following this work, Liu _et al._ [12] further develop an object-centric
Transformer to jointly forecast future trajectories and affordances
autoregressively, and annotate publicly available datasets to support future
works. More recently, Bao _et al._ [13] lift the problem to 3D spaces where
hand trajectories are predicted by an uncertainty-aware state space
Transformer in an autoregressive manner. However, this method needs additional
3D perception inputs from the RGB-D camera. In this work, we still achieve
joint hand trajectory and object affordance prediction on 2D human videos
rather than in 3D space. We focus on capturing more general knowledge from
only egocentric camera observations in an iterative non-autoregressive (iter-
NAR) manner, rather than the autoregressive way of the state-of-the-art works
[12, 13].

#### Diffusion-based egocentric video analysis.

Diffusion models have been successfully utilized in exocentric and egocentric
video prediction [47, 48, 49, 50, 2] due to their strong generation ability.
With only egocentric videos as inputs, diffusion-based techniques can also
achieve human mesh recovery [51, 52], 3D HOI reconstruction [53, 54], and 3D
HOI synthesizing [16, 55]. However, none of these works concentrate on the
combination of fine-grained hand trajectories and object affordances as future
HOI representations for potential utilization in artificial intelligence
systems. Our proposed Diff-IP2D first achieves this based on the denoising
diffusion probabilistic model [20], which dominates the existing paradigm [12,
13] in prediction performance on egocentric videos.

##  3 Proposed Method

###  3.1 Preliminaries

#### Task definition.

Given the video clip of past egocentric observations
$\mathcal{I}=\\{I_{t}\\}_{t=-N_{\text{p}}+1}^{0}$, we aim to predict future
hand trajectories
$\mathcal{H}=\\{H_{t}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{L}}}\\}_{t=1}^{N_{\text{f}}}(H_{t}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{L}}}\in\mathbb{R}^{2})$
and potential object contact points
$\mathcal{O}=\\{O_{n}\\}_{n=1}^{N_{\text{o}}}(O_{n}\in\mathbb{R}^{2})$, where
$N_{\text{p}}$ and $N_{\text{f}}$ are the numbers of frames in the past and
future time horizons respectively, and $N_{\text{o}}$ denotes the number of
predicted contact points used to calculate interaction hotspots as object
affordances. Following the previous works [12, 14], we predict the future
positions of the right hand, the left hand, and the affordance of the next
active object on the last observed image of the input videos.

#### Diffusion models.

In this work, we propose a diffusion-based approach to gradually corrupt the
input to noisy features and then train a denoising model to reverse this
process. We first map the input images into a latent space $\mathbf{z}_{0}\sim
q(\mathbf{z}_{0})$, which is then corrupted to a standard Gaussian noise
$\mathbf{z}_{S}\sim\mathcal{N}(0,\textbf{\text{I}})$. In the forward process,
the perturbation operation can be represented as
$q(\mathbf{z}_{s}|\mathbf{z}_{s-1})=\mathcal{N}(\mathbf{z}_{s};\sqrt{1-\beta_{s}}\mathbf{z}_{s-1},\beta_{s}\textbf{\text{I}})$,
where $\beta$ is the predefined variance scales. In the reverse process, we
set a denoising diffusion model to gradually reconstruct the latent
$\mathbf{z}_{0}$ from the noisy $\mathbf{z}_{S}$. The denoised features can be
used to recover the final future hand trajectories and object affordances.

###  3.2 Architecture

#### System overview.

Accurately reconstructing the future part of the input sequence is critical in
the diffusion-based prediction task. We empirically found that ground-truth
hand waypoints
$\mathcal{H}^{\text{gt}}=\\{H_{t}^{{\scriptstyle{\text{R}}},\text{gt}},H_{t}^{{\scriptstyle{\text{L}}},\text{gt}}\\}_{t=1}^{N_{\text{f}}}(H_{t}^{{\scriptstyle{\text{R}}},\text{gt}},H_{t}^{{\scriptstyle{\text{L}}},\text{gt}}\in\mathbb{R}^{2})$
and contact points
$\mathcal{O}^{\text{gt}}=\\{O^{\text{gt}}_{n}\\}_{n=1}^{N_{\text{o}}}(O^{\text{gt}}_{n}\in\mathbb{R}^{2})$
provide discrete and sparse supervision signals for reconstruction, which is
not enough for capturing possible high-level semantics such as human
intentions in the denoising process. Therefore, as Fig. 2 shows, we first use
Multi-Feature Extractor and Side-Oriented Fusion Module to transform the input
images into latent HOI features, and then implement diffusion-related
operation in the latent continuous space. The HOI features denoised by Motion-
Aware Denoising Transformer are further absorbed by Hand Trajectory Head and
Object Affordance Head to generate future hand trajectories and object
hotspots.

Figure 2: System Overview of Diff-IP2D. Our proposed paradigm takes in
sequential past egocentric images and jointly predicts hand trajectories and
object affordances as future HOI states. The observations are mapped to the
latent feature space for the diffusion process.

#### Multi-Feature Extractor (MFE).

Following the previous work [12], we use MFE that consists of a pretrained
Temporal Segment Network (TSN) provided by Furnari _et al._ [34], RoIAlign
[56] with average pooling, and Multilayer Perceptron (MLP) to extract hand,
object, and global intermediate features for each sequence image
$I_{t}\in\mathcal{I}$. The positions of hand-object bounding boxes detected by
the off-the-shelf approach [15] are also encoded to feature vectors fused with
hand and object intermediate features.

#### Side-Oriented Fusion Module (SOFM).

Our proposed SOFM is a learnable linear transformation to fuse the above-
mentioned three types of feature vectors into the final latent form for two
sides respectively. Specifically, the global features and right-side features
(right-hand/object features) are concatenated to the right-side HOI features
$\mathcal{F}^{\scriptstyle{\text{R}}}=\\{F_{t}^{\scriptstyle{\text{R}}}\\}_{t=-N_{\text{p}}+1}^{X}(F_{t}^{\scriptstyle{\text{R}}}\in\mathbb{R}^{a},X=N_{\text{f}}\,\text{for
training and}\,X=0\,\,\text{for inference})$. The operation and feature sizes
are the same as the left-side counterparts, leading to
$\mathcal{F}^{\scriptstyle{\text{L}}}=\\{F_{t}^{\scriptstyle{\text{L}}}\\}_{t=-N_{\text{p}}+1}^{X}$.
We further concatenate the side-oriented features along the time axis
respectively to generate the input latents
$F^{\scriptstyle{\text{R}}}_{\text{seq}}$,
$F^{\scriptstyle{\text{L}}}_{\text{seq}}\in\mathbb{R}^{(N_{\text{p}}+X)\times
a}$ for the following diffusion model.

#### Motion-Aware Denoising Transformer (MADT).

Our proposed MADT takes in the noisy latent HOI features and reconstructs
future HOI features for the following predictors conditioned on past HOI
counterparts. MADT consists of several stacked Transformer layers as shown in
Fig. 3. Inspired by the text generation technique [26], we anchor the past HOI
features for both forward and reverse processes. We only impose noises and
denoise at the positions of the future feature sequence. The features of the
two sides are denoised using the same model, leading to
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}$ and
$\hat{F}^{\scriptstyle{\text{L}}}_{\text{seq}}$. In addition, egomotion
guidance is proposed here to fill the gaps mentioned in Sec. 1. Specifically,
we first extract the Scale-Invariant Feature Transform (SIFT) descriptors to
find the pixel correspondence between two adjacent images of past observations
$\mathcal{I}$. Then we calculate the homography matrix with RANSAC that finds
a transformation to maximize the number of inliers in the keypoint pairs. We
accumulate the consecutive homography matrices and obtain
$M_{\text{seq}}\in\mathbb{R}^{N_{\text{p}}\times 3\times 3}$ representing the
camera wearer’s motion between $I_{t}$ ($t\leq 0$) and $I_{0}$. They are
further linearly embedded into an egomotion feature
$E_{\text{seq}}\in\mathbb{R}^{N_{\text{p}}\times b}$ by Motion Encoder. The
multi-head cross-attention module (MHCA) in the devised Transformer layer then
absorbs the egomotion feature to guide the denoising process. More analysis on
the use of egomotion guidance can be found in Appendix, Sec. B.

Figure 3: Architecture of our proposed MADT. MADT receives corrupted latent
HOI features with the position embedding (PE) and time embedding (TE), and
outputs denoised future HOI features under the egomotion guidance.

#### Predictors.

Our proposed predictors consist of Hand Trajectory Head (HTH) and Object
Affordance Head (OAH). HTH contains an MLP that receives the future parts of
the denoised features,
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}]$
and
$\hat{F}^{\scriptstyle{\text{L}}}_{\text{seq}}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}]$
to generate future waypoints $\mathcal{H}$ of two hands. As to OAH, we
empirically exploit Conditional Variational Autoencoder (C-VAE) [57] to
generate possible contact points $\mathcal{O}$ in the near future. Take the
right hand as an example, the condition is selected as the time-averaged
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}$ and predicted waypoints
$H_{t}^{\scriptstyle{\text{R}}}$. Note that we additionally consider denoised
future HOI features
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}[N_{\text{p}}\\!+\\!1:N_{\text{p}}\\!+\\!N_{\text{f}}]$
($t\\!>\\!0$) besides the features from the past observation ($t\\!\leq\\!0$)
for object affordance prediction. This aligns with the intuitive relationship
between the contact points and the overall interaction process. Therefore, we
integrate richer conditional features from trajectory prediction into the
object affordance prediction compared to the previous work [12] only
conditioned on historical features.

###  3.3 Training

#### Forward process.

We implement partial noising [26] in the forward process during training.
Taking the right side as an example, the output of SOFM is first extended by a
Markov transition
$q(\mathbf{z}_{0}|F^{\scriptstyle{\text{R}}}_{\text{seq}})=\mathcal{N}(F^{\scriptstyle{\text{R}}}_{\text{seq}},\beta_{0}\textbf{\text{I}})$,
where
$F^{\scriptstyle{\text{R}}}_{\text{seq}}\in\mathbb{R}^{(N_{\text{p}}+N_{\text{f}})\times
a}$. We discard the embedding process from Gong _et al._ [26] since the HOI
feature $F^{\scriptstyle{\text{R}}}_{\text{seq}}$ is already in the continuous
latent space. In each following forward step of the diffusion model, we
implement $q(\mathbf{z}_{s}|\mathbf{z}_{s-1})$ by adding noise to the future
part of $\mathbf{z}_{s-1}$, i.e.,
$\mathbf{z}_{s-1}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}]$
for both sides.

#### Reverse process.

After corrupting the initial $\mathbf{z}_{0}$ to $\mathbf{z}_{S}$ by the
forward process, our proposed MADT is adopted to denoise $\mathbf{z}_{S}$ to
$\mathbf{z}_{0}$ in a classifier-free manner. Considering the guidance of
egomotion features, the reverse process can be modeled as
$p_{\scriptscriptstyle{\text{MADT}}}(\mathbf{z}_{0:S}):=p(\mathbf{z}_{s})\prod_{s=1}^{S}p_{\scriptscriptstyle{\text{MADT}}}(\mathbf{z}_{s-1}|\mathbf{z}_{s},M_{\text{seq}})$.
Specifically, the MADT model
$f_{\scriptscriptstyle\text{MADT}}(\mathbf{z}_{s},s,M_{\text{seq}})$ predicts
the injected noise for each forward step with
$p_{\scriptscriptstyle{\text{MADT}}}(\mathbf{z}_{s-1}|\mathbf{z}_{s},M_{\text{seq}})=\mathcal{N}(\mathbf{z}_{s-1};\mu_{\scriptscriptstyle{\text{MADT}}}(\mathbf{z}_{s},s,M_{\text{seq}}),\sigma_{\scriptscriptstyle{\text{MADT}}}(\mathbf{z}_{s},s,M_{\text{seq}}))$.
The same denoising operation and motion-aware guidance are applied to HOI
features of both sides.

#### Training objective.

The loss function to train the networks in Diff-IP2D contains four parts,
including diffusion-related losses, trajectory loss, affordance loss, and an
additional regularization term (see Fig. 2). Take the right side as an
example, we use the variational lower bound
$\mathcal{L}_{\text{VLB}}^{\scriptstyle{\text{R}}}$ as the diffusion-related
losses:

$\displaystyle\mathcal{L}_{\text{VLB}}^{\scriptstyle{\text{R}}}=\sum_{s=2}^{S}||\mathbf{z}_{0}^{\scriptstyle{\text{R}}}-f_{\scriptscriptstyle\text{MADT}}(\mathbf{z}_{s}^{\scriptstyle{\text{R}}},s,M_{\text{seq}})||^{2}+||F_{\text{seq}}^{\scriptstyle{\text{R}}}-\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}}||^{2},$
(1)

where
$\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}}=f_{\scriptscriptstyle\text{MADT}}(\mathbf{z}_{1}^{\scriptstyle{\text{R}}},1,M_{\text{seq}})$.
To reconstruct hand trajectories beyond the latent feature space, we further
set trajectory loss $\mathcal{L}_{\text{traj}}^{\scriptstyle{\text{R}}}$ with
the distance between the ground-truth waypoints and the ones predicted by HTH:

$\displaystyle\mathcal{L}_{\text{traj}}^{\scriptstyle{\text{R}}}=\sum_{t=1}^{N_{\text{f}}}||H_{t}^{\scriptstyle{\text{R}}}-H_{t}^{{\scriptstyle{\text{R}}},\text{gt}}||^{2},$
(2)

where
$H_{t}^{\scriptstyle{\text{R}}}=f_{\scriptscriptstyle\text{HTH}}(\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}])$.
We only focus on the future part out of the holistic sequence for computing
$\mathcal{L}_{\text{traj}}^{\scriptstyle{\text{R}}}$ since we let HTH be more
sensitive to predictions rather than bias it to past observations. As to the
object affordance prediction, we also compute the affordance loss
$\mathcal{L}_{\text{aff}}$ after multiple stochastic sampling considering the
next active object recognized following Liu _et al._ [12] (assuming in the
right side here for brevity):

$\displaystyle\mathcal{L}_{\text{aff}}=\sum_{n=1}^{N_{\text{o}}}||O_{n}-O_{n}^{\text{gt}}||^{2}+c\mathcal{L}_{\text{KL}},$
(3)

where
$O_{n}\\!=\\!f_{\scriptscriptstyle\text{OAH}}(\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{R}}})$,
and
$\mathcal{L}_{\text{KL}}=\frac{1}{2}(-\log\sigma^{2}_{\scriptscriptstyle\text{OAH}}(\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{R}}})+\mu^{2}_{\scriptscriptstyle\text{OAH}}(\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{R}}})+\sigma^{2}_{\scriptscriptstyle\text{OAH}}(\hat{F}_{\text{seq}}^{\scriptstyle{\text{R}}},H_{t}^{\scriptstyle{\text{R}}})-1)$
is the KL-Divergence regularization for C-VAE, which is scaled by
$c=1\text{e}$-$3$. The latent features and predicted hand waypoints are fused
by MLP suggested by the previous work [12]. We consider both reconstructed
future HOI features
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}]$
and anchored past counterparts
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}[0\\!:\\!N_{\text{p}}]$ compared
to [12] as mentioned before. We also notice that the latent feature spaces
before and after the denoising diffusion process represent the same “profile”
of the input HOI sequence. Therefore, we propose an additional regularization
term implicitly linking $F^{\scriptstyle{\text{R}}}_{\text{seq}}$ and
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}$ by hand trajectory prediction:

$\displaystyle\mathcal{L}_{\text{reg}}^{\scriptstyle{\text{R}}}=\sum_{t=1}^{N_{\text{f}}}||\tilde{H}_{t}^{\scriptstyle{\text{R}}}-H_{t}^{{\scriptstyle{\text{R}}},\text{gt}}||^{2},$
(4)

where
$\tilde{H}_{t}^{\scriptstyle{\text{R}}}=f_{\scriptscriptstyle\text{HTH}}(F_{\text{seq}}^{\scriptstyle{\text{R}}}[N_{\text{p}}\\!+\\!1\\!:\\!N_{\text{p}}\\!+\\!N_{\text{f}}])$.
Although Eq. (4) does not explicitly contain the term
$\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}$, the training direction is the
same with Eq. (2), thus maintaining training stability. The regularization
helps the convergence of Diff-IP2D by consistently constraining the two latent
spaces alongside the diffusion process. Here we do not use object affordance
prediction for regularization because we empirically found that incorporating
OAH mitigates training efficiency while the positive effect is not obvious.
Finally, we get the total loss to train our proposed Diff-IP2D:

$\displaystyle\mathcal{L}_{\text{total}}=\lambda_{\text{VLB}}(\mathcal{L}_{\text{VLB}}^{\scriptstyle{\text{R}}}+\mathcal{L}_{\text{VLB}}^{\scriptstyle{\text{L}}})+\lambda_{\text{traj}}(\mathcal{L}_{\text{traj}}^{\scriptstyle{\text{R}}}+\mathcal{L}_{\text{traj}}^{\scriptstyle{\text{L}}})+\lambda_{\text{aff}}\mathcal{L}_{\text{aff}}+\lambda_{\text{reg}}(\mathcal{L}_{\text{reg}}^{\scriptstyle{\text{R}}}+\mathcal{L}_{\text{reg}}^{\scriptstyle{\text{L}}}),$
(5)

where $\lambda_{\text{VLB}}$, $\lambda_{\text{traj}}$, $\lambda_{\text{aff}}$,
and $\lambda_{\text{reg}}$ are the weights to balance different losses.
Besides, we leverage the importance sampling technique proposed in improved
DDPM [58], which promotes the training process focusing more on the steps with
relatively large $\mathcal{L}_{\text{total}}$.

###  3.4 Inference

In the inference stage, we first sample
$F^{\scriptstyle{\text{R}}}_{\text{noise}}$,
$F^{\scriptstyle{\text{L}}}_{\text{noise}}\in\mathbb{R}^{N_{\text{f}}\times
a}$ from a standard Gaussian distribution, which is then concatenated with
$F^{\scriptstyle{\text{R}}}_{\text{seq}}$,
$F^{\scriptstyle{\text{L}}}_{\text{seq}}\in\mathbb{R}^{N_{\text{p}}\times a}$
along the time axis to generate $\mathbf{z}_{S}^{\scriptstyle{\text{R}}}$ and
$\mathbf{z}_{S}^{\scriptstyle{\text{L}}}$. Then we use MADT to predict
$\mathbf{z}_{0}^{\scriptstyle{\text{R}}}$ and
$\mathbf{z}_{0}^{\scriptstyle{\text{L}}}$ based on DDIM sampling [59]. Note
that we anchor the past part of reparameterized $\mathbf{z}_{s}$ as the fixed
condition in every step of the inference process following Gong _et al._ [26].
Finally, the generated $\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}}$ and
$\hat{F}^{\scriptstyle{\text{L}}}_{\text{seq}}$ are used to predict future
hand waypoints and contact points by $f_{\scriptscriptstyle\text{HTH}}(\cdot)$
and $f_{\scriptscriptstyle\text{OAH}}(\cdot)$ as mentioned before. It can be
seen from the inference stage that Diff-IP2D can be regarded as an iter-NAR
model in the latent feature space. Compared to the state-of-the-art baselines
in an autoregressive manner, our approach shifts the iteration from

Figure 4: Comparison of AR and our iter-NAR prediction.

the time axis to the denoising direction, which is shown in Fig. 4. This
alleviates the accumulated artifacts caused by the limited iteration in the
time dimension, and maintains bidirectional constraints among the sequential
features to generate future HOI states in parallel, providing a deeper
understanding of human intention. We further present the mathematical
relationship between the two iter-NAR models, Diff-IP2D for HOI prediction and
DiffuSeq [26] for text generation in Appendix, Sec. A.

##  4 Experiments

###  4.1 Experimental setups

#### Datasets.

Following the previous work [12], we utilize three publicly available datasets
including Epic-Kitchens-55 (EK55) [60], Epic-Kitchens-100 (EK100) [61], and
EGTEA Gaze+ (EG) [11]. For the EK55 and EK100 datasets, we sample past
$N_{\text{p}}=10$ frames (2.5 s) to forecast HOI states in future
$N_{\text{f}}=4$ frames (1.0 s), both at 4 FPS. As to the EG dataset,
$N_{\text{p}}=9$ frames (1.5 s) are used for $N_{\text{f}}=3$ HOI predictions
(0.5 s) at 6 FPS. See the Appendix, Sec. C.2 for more details.

#### Diff-IP2D configuration.

MFE extracts the hand, object, and global feature vectors all with the size of
$512$ for each input image. For the EK55 and EK100 datasets, the outputs of
SOFM $F^{\scriptstyle{\text{R}}}_{\text{seq}}$,
$F^{\scriptstyle{\text{L}}}_{\text{seq}}$ have the size of ${14\times 512}$
for training and ${10\times 512}$ for inference. For the EG dataset,
$F^{\scriptstyle{\text{R}}}_{\text{seq}}$,
$F^{\scriptstyle{\text{L}}}_{\text{seq}}$ are ${9\times 512}$ for training and
${12\times 512}$ for inference. As to the diffusion process, the total number
of steps $S$ is set to $1000$. We also provide an ablation study on multiple
steps for training and inference in Appendix, Sec. D.3. The square-root noise
schedule in Diffusion-LM [62] is adopted here for the forward diffusion
process. MADT has $6$ Transformer layers (Fig. 3) for denoising, where the
embedding dimension is $512$, the number of heads is set to $4$, and the
intermediate dimension of the feed-forward layer is set to $2048$. Motion
Encoder linearly projects each homography matrix to an egomotion feature
vector of $512$. We use an MLP with hidden dimensions $256$ and $64$ to
predict the hand waypoints as HTH, and a C-VAE containing an MLP with a hidden
dimension $512$ to predict contact points as OAH. The training configurations
can be found in Appendix, Sec. C.2. In the reference stage, we generate the
$10$ candidate samples for each prediction.

#### Baseline configuration.

We choose Constant Velocity Hand (CVH), Seq2Seq [63], FHOI [14], OCT [12], and
USST [13] as the baselines for hand trajectory prediction. CVH is the most
straightforward one which assumes two hands remain in uniform motion over the
future time horizon with the average velocity during past observations.
Besides, we adjust the input and architecture of USST to the 2D prediction
task since it was originally designed for 3D hand trajectory prediction. We
choose Center Object [14], Hotspots [64], FHOI [14], OCT [12], and Final Hand
of USST [13] (USST-FH) as the baselines for object affordance prediction.
USST-FH puts a mixture of Gaussians at the last hand waypoint predicted by
USST since its vanilla version can only predict waypoints.

#### Evaluation metrics.

Following the previous work [14, 12, 13], we use Final Displacement Error
(FDE) to evaluate prediction performance on hand trajectories. Considering the
general knowledge of “post-contact trajectories” extracted from human videos
is potentially beneficial to robot manipulation [1, 18], we additionally
extend the metric Average Displacement Error to Weighted Displacement Error
(WDE):

$\displaystyle\text{WDE}=\frac{1}{2N_{\text{f}}}\sum_{{\scriptstyle{\text{R}}},{\scriptstyle{\text{L}}}}\sum_{t=1}^{N_{\text{f}}}\frac{t}{N_{\text{f}}}D(H_{t},H_{t}^{\text{gt}}),$
(6)

where $D(\cdot)$ denotes the L2 distance function and the later waypoints
contribute to larger errors. We select the mean error among the 10 samples for
each hand trajectory prediction. As to the object affordance prediction, we
use Similarity Metric (SIM) [65], AUC-Judd (AUC-J) [66], and Normalized
Scanpath Saliency (NSS) [67] as evaluation metrics. We use all 10 contact
point candidates to compute the metric values for each affordance prediction.

Moreover, we propose a novel object-centric protocol to jointly evaluate the
two prediction tasks. We first calculate the averaged hand waypoints
$\bar{H}^{\scriptstyle{\text{R}}}_{t}$ and
$\bar{H}^{\scriptstyle{\text{L}}}_{t}$ for each future timestamp from multiple
samples. Then we select the waypoint closest to each predicted contact
prediction $O_{n}$ as an additional “interaction point”, which can be
formulated by:

$\displaystyle\bar{H}_{n}^{\text{ip}}=\textrm{min}_{{\scriptstyle{\text{R}}},{\scriptstyle{\text{L}}},t}D(\bar{H}_{t},O_{n}).$
(7)

Ultimately, the joint hotspot is predicted using
$\\{\bar{H}_{n}^{\text{ip}}\cup O_{n}\\}_{n=1}^{N_{\text{o}}}$. This protocol
comprehensively considers object-centric attention since HOI changes the
object states and hand waypoints must have a strong correlation with object
positions. Note that we also use the quantitative metrics same as the ones for
object affordance prediction, which are denoted as SIM∗, AUC-J∗, and NSS∗.
More clarifications about our proposed new protocol can be found in Appendix,
Sec. C.1.

Table 1: Comparison of performance on hand trajectory and object affordance prediction approach | EK55 | EK100 | EG
---|---|---|---
WDE $\downarrow$ |  |  FDE $\downarrow$ |  WDE $\downarrow$ |  |  FDE $\downarrow$ |  WDE $\downarrow$ |  |  FDE $\downarrow$
CVH | 0.636 |  | 0.315 | 0.658 |  | 0.329 | 0.689 |  | 0.343
Seq2Seq [63] | 0.505 |  | 0.212 | 0.556 |  | 0.219 | 0.649 |  | 0.263
FHOI [14] | 0.589 |  | 0.307 | 0.550 |  | 0.274 | 0.557 |  | 0.268
OCT [12] | 0.446 |  | 0.208 | 0.467 |  | 0.206 | 0.514 |  | 0.249
USST [13] | 0.458 |  | 0.210 | 0.475 |  | 0.206 | 0.552 |  | 0.256
Diff-IP2D (ours) | 0.411 |  | 0.181 | 0.407 |  | 0.187 | 0.478 |  | 0.211
|  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$ |  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$ |  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$
Center Object [14] | 0.083 | 0.553 | 0.448 | 0.081 | 0.558 | 0.401 | 0.094 | 0.562 | 0.518
Hotspots [64] | 0.156 | 0.670 | 0.606 | 0.147 | 0.635 | 0.533 | 0.150 | 0.662 | 0.574
FHOI [14] | 0.159 | 0.655 | 0.517 | 0.120 | 0.548 | 0.418 | 0.122 | 0.506 | 0.401
OCT [12] | 0.213 | 0.710 | 0.791 | 0.187 | 0.677 | 0.695 | 0.227 | 0.704 | 0.912
USST-FH [13] | 0.208 | 0.682 | 0.757 | 0.179 | 0.658 | 0.754 | 0.190 | 0.675 | 0.729
Diff-IP2D (ours) | 0.226 | 0.725 | 0.980 | 0.211 | 0.736 | 0.917 | 0.242 | 0.722 | 0.956
|  SIM${}^{*}\\!\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$ |  SIM${}^{*\\!}\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$ |  SIM${}^{*\\!}\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$
FHOI [14] | 0.130 | 0.602 | 0.487 | 0.113 | 0.545 | 0.409 | 0.118 | 0.501 | 0.379
OCT [12] | 0.219 | 0.720 | 0.848 | 0.182 | 0.684 | 0.662 | 0.194 | 0.672 | 0.752
Diff-IP2D (ours) | 0.222 | 0.730 | 0.888 | 0.204 | 0.727 | 0.844 | 0.226 | 0.701 | 0.825
Figure 5: Visualization of hand trajectory prediction on Epic-Kitchens. The
waypoints from ground-truth labels, Diff-IP2D, and the second-best baseline
[12] are connected by red, white, and blue dashed lines respectively.

###  4.2 Separate evaluation on hand trajectory and object affordance
prediction

We first present the evaluation results on hand trajectory prediction. As Tab.
1 depicts, our proposed Diff-IP2D outperforms all the baselines on the EK55
and EK100 datasets on WDE and FED. This is mainly achieved by the devised
iter-NAR paradigm of Diff-IP2D alleviating degeneration in AR baselines, as
well as the egomotion guidance. The visualization of the related hand
prediction results is shown in Fig. 5. It can be seen that our proposed method
can better capture the camera wearer’s intention (such as putting the food in
the bowl) and generate more reasonable future trajectories even if there is a
lack of past observations for hands (such as reaching out towards the table).
Besides, our method can predict a good final hand position although there is a
large shift in the early stage (the subfigure in the bottom right corner of
Fig. 5), which benefits from our diffusion-based parallel generation. When
directly transferring the models trained on Epic-Kitchens to the unseen EG
dataset, our method still outperforms the other baselines, which improves by
7.0% and 15.3% against the second-best method on WDE and FDE respectively.
This reveals the solid generalization capability of our diffusion-based
approach across different environments.

The comparison results of object affordance prediction are also shown in Tab.
1. Our proposed Diff-IP2D predicts the hotspots with larger SIM, AUC-J, and
NSS compared to all the baselines on both Epic-Kitchens data and unseen EG
data. Fig. 6 illustrates the predicted contact points with minimum distances
to the ground-truth ones. Our proposed method focuses more on objects of
interest considering the features of the holistic interaction and potential
hand trajectories, and therefore grounds the contact points closer to the
ground-truth labels than the counterparts of the baseline. See the Appendix,
Sec. D.6 for more visualization results.

Figure 6: Visualization of object affordance prediction on Epic-Kitchens. The
contact points from ground-truth, Diff-IP2D, and the state-of-the-art baseline
OCT [12] are represented by red, white, and blue dots respectively. For a
clearer illustration, we additionally put a fixed Gaussian with each contact
point as the center.

###  4.3 Joint evaluation on hand trajectory and object affordance prediction

We further compare Diff-IP2D with the other two joint prediction baselines,
FHOI [14] and OCT [12], using our proposed object-centric protocol. The video
clips containing both ground-truth hand waypoints and contact points are used
for evaluation in this experiment. The results are also shown in Tab. 1, which
indicates that our proposed Diff-IP2D can generate the best object-centric HOI
predictions considering the two tasks concurrently on both Epic-Kitchens and
unseen EG data. The results also suggest that Diff-IP2D outperforms the
baselines on object-centric HOI prediction by focusing more attention on the
target objects and predicting reasonable hand trajectories around them.

###  4.4 Ablation study on egomotion guidance

We provide an ablation study of the egomotion features used to guide MADT
denoising on the EK55 and EK100 datasets. Here we replace the MHCA in MADT
with a multi-head self-attention module (MHSA) to remove the egomotion
guidance while keeping the same parameter number. The experimental results in
Tab. 2 show that the guidance of motion features improves our proposed
diffusion-based paradigm noticeably on both hand trajectory prediction and
object affordance prediction. This is achieved by narrowing the two gaps
caused by 2D-3D ill-posed problem and view difference mentioned in Sec. 1.
Note that the egomotion guidance is more significant on the EK100 dataset than
on the EK55 dataset. The reason could be that EK100 has a larger volume of
training data incorporating more diverse egomotion patterns than EK55, leading
to a model that can capture human dynamics better. More results of the related
joint evaluation are presented in Appendix, Sec. D.1.

Table 2: Ablation study on egomotion guidance approach | EK55 | EK100
---|---|---
WDE $\downarrow$ |  FDE $\downarrow$ |  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$ |  WDE $\downarrow$ |  FDE $\downarrow$ |  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$
Diff-IP2D* | 0.427 | 0.186 | 0.218 | 0.717 | 0.929 | 0.439 | 0.198 | 0.201 | 0.710 | 0.846
Diff-IP2D | 0.411 | 0.181 | 0.226 | 0.725 | 0.980 | 0.407 | 0.187 | 0.211 | 0.736 | 0.917
improvement | 3.7% | 2.7% | 3.7% | 1.1% | 5.5% | 7.3% | 5.6% | 5.0% | 3.7% | 8.4%
Diff-IP2D*: Diff-IP2D w/o egomotion guidance |  |  |  |

##  5 Conclusion and insights

In this paper, we propose a novel hand-object interaction prediction method
Diff-IP2D. Specifically, we implement the devised denoising diffusion in the
latent feature space under our proposed egomotion guidance, and jointly
predict future hand trajectories and object affordances with the recovered
latents on 2D egocentric videos. According to the experimental results, Diff-
IP2D dominates the existing baselines on both off-the-shelf metrics and our
new evaluation protocol, suggesting promising applications in artificial
intelligence systems. It innovatively learns to recover latent HOI features
and forecast future HOI states in parallel, which can serve as a foundation
generative paradigm for future works on predictive tasks.

## References

  * [1] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak.  Affordances from human videos as a versatile representation for robotics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778–13790, 2023.
  * [2] Matthew Chang, Aditya Prakash, and Saurabh Gupta.  Look ma, no hands! agent-environment factorization of egocentric videos.  arXiv preprint arXiv:2305.16301, 2023.
  * [3] Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, et al.  Umetrack: Unified multi-view end-to-end hand tracking for vr.  In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022.
  * [4] Xinyu Xu, Yong-Lu Li, and Cewu Lu.  Dynamic context removal: A general training strategy for robust models on video action predictive tasks.  International Journal of Computer Vision, 131(12):3272–3288, 2023.
  * [5] Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang.  Dynamic sampling networks for efficient action recognition in videos.  IEEE transactions on image processing, 29:7970–7983, 2020.
  * [6] Ce Zhang, Changcheng Fu, Shijie Wang, Nakul Agarwal, Kwonjoon Lee, Chiho Choi, and Chen Sun.  Object-centric video representation for long-term action anticipation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6751–6761, 2024.
  * [7] Yin-Dong Zheng, Guo Chen, Minglei Yuan, and Tong Lu.  Mrsn: Multi-relation support network for video action detection.  In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 1026–1031, 2023.
  * [8] Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Jiashi Feng.  Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4372–4381, 2017.
  * [9] Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg.  In the eye of transformer: Global–local correlation for egocentric gaze estimation and beyond.  International Journal of Computer Vision, 132(3):854–871, 2024.
  * [10] Yin Li, Alireza Fathi, and James M Rehg.  Learning to predict gaze in egocentric video.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3216–3223, 2013.
  * [11] Yin Li, Miao Liu, and James M Rehg.  In the eye of beholder: Joint learning of gaze and actions in first person video.  In Proceedings of the European conference on computer vision (ECCV), pages 619–635, 2018.
  * [12] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang.  Joint hand motion and interaction hotspots prediction from egocentric videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282–3292, 2022.
  * [13] Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong.  Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13702–13711, 2023.
  * [14] Miao Liu, Siyu Tang, Yin Li, and James M Rehg.  Forecasting human-object interaction: joint prediction of motor attention and actions in first person video.  In Proceedings of the European conference on computer vision (ECCV), pages 704–721, 2020.
  * [15] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey.  Understanding human hands in contact at internet scale.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869–9878, 2020.
  * [16] Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu.  Affordance diffusion: Synthesizing hand-object interactions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22479–22489, 2023.
  * [17] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui.  Interdiff: Generating 3d human-object interactions with physics-informed diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14928–14940, 2023.
  * [18] Russell Mendonca, Shikhar Bahl, and Deepak Pathak.  Structured world models from human videos.  arXiv preprint arXiv:2308.10901, 2023.
  * [19] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak.  Human-to-robot imitation in the wild.  arXiv preprint arXiv:2207.09450, 2022.
  * [20] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [21] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [22] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  * [23] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo.  Ddp: Diffusion model for dense visual prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21741–21752, 2023.
  * [24] Jiuming Liu, Guangming Wang, Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong Du, and Hesheng Wang.  Difflow3d: Toward robust uncertainty-aware scene flow estimation with diffusion model.  arXiv preprint arXiv:2311.17456, 2023.
  * [25] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [26] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong.  Diffuseq: Sequence to sequence text generation with diffusion models.  In International Conference on Learning Representations, 2023.
  * [27] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong.  Diffuseq-v2: Bridging discrete and continuous text spaces for accelerated seq2seq diffusion models.  arXiv preprint arXiv:2310.05793, 2023.
  * [28] A Calway, W Mayol-Cuevas, D Damen, O Haines, and T Leelasawassuk.  Discovering task relevant objects and their modes of interaction from multi-user egocentric video.  In British Machine Vision Conference, 2015.
  * [29] Yang Liu, Ping Wei, and Song-Chun Zhu.  Jointly recognizing object fluents and tasks in egocentric videos.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [30] Matthias Schroder and Helge Ritter.  Hand-object interaction detection with fully convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 18–25, 2017.
  * [31] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen.  Epic-kitchens visor benchmark: Video segmentations and object relations.  Advances in Neural Information Processing Systems, 35:13745–13758, 2022.
  * [32] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi.  Fine-grained egocentric hand-object segmentation: Dataset, model, and applications.  In European Conference on Computer Vision, pages 127–145. Springer, 2022.
  * [33] Richard E. L. Higgins and David F. Fouhey.  Moves: Manipulated objects in video enable segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6334–6343, June 2023.
  * [34] Antonino Furnari and Giovanni Maria Farinella.  Rolling-unrolling lstms for action anticipation from first-person video.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4021–4036, 2020.
  * [35] Hehe Fan, Tao Zhuo, Xin Yu, Yi Yang, and Mohan Kankanhalli.  Understanding atomic hand-object interaction with human intention.  IEEE Transactions on Circuits and Systems for Video Technology, 32(1):275–285, 2021.
  * [36] Tsukasa Shiota, Motohiro Takagi, Kaori Kumagai, Hitoshi Seshimo, and Yushi Aono.  Egocentric action recognition by capturing hand-object contact and object state.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6541–6551, January 2024.
  * [37] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey.  Understanding human hands in contact at internet scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [38] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  arXiv preprint arXiv:2201.02610, 2022.
  * [39] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5346–5355, 2020.
  * [40] Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang.  Harmonious feature learning for interactive hand-object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12989–12998, June 2023.
  * [41] Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu.  Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2750–2760, 2022.
  * [42] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang.  Semi-supervised 3d hand-object poses estimation with interactions in time.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14687–14697, 2021.
  * [43] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2992–2999, 2013.
  * [44] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  * [45] Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, and Shubham Tulsiani.  Towards generalizable zero-shot manipulation via translating human interaction plans.  arXiv preprint arXiv:2312.00775, 2023.
  * [46] Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos.  Forecasting action through contact representations from first person video.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6703–6714, 2023.
  * [47] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi.  Diffusion models for video prediction and infilling.  arXiv preprint arXiv:2206.07696, 2022.
  * [48] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  Advances in Neural Information Processing Systems, 35:8633–8646, 2022.
  * [49] Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal.  Mcvd-masked conditional video diffusion for prediction, generation, and interpolation.  Advances in neural information processing systems, 35:23371–23385, 2022.
  * [50] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman.  Put myself in your shoes: Lifting the egocentric perspective from exocentric videos.  arXiv preprint arXiv:2403.06351, 2024.
  * [51] Siwei Zhang, Qianli Ma, Yan Zhang, Sadegh Aliakbarian, Darren Cosker, and Siyu Tang.  Probabilistic human mesh recovery in 3d scenes from egocentric views.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7989–8000, 2023.
  * [52] Yuxuan Liu, Jianxin Yang, Xiao Gu, Yao Guo, and Guang-Zhong Yang.  Egohmr: Egocentric human mesh recovery via hierarchical latent diffusion model.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9807–9813, 2023.
  * [53] Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani.  Diffusion-guided reconstruction of everyday hand-object interaction clips.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19717–19728, October 2023.
  * [54] Zhifan Zhu and Dima Damen.  Get a grip: Reconstructing hand-object stable grasps in egocentric videos.  arXiv preprint arXiv:2312.15719, 2023.
  * [55] Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang.  Hoidiffusion: Generating realistic 3d hand-object interaction data.  arXiv preprint arXiv:2403.12011, 2024.
  * [56] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2961–2969, 2017.
  * [57] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  Advances in neural information processing systems, 28, 2015.
  * [58] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International conference on machine learning, pages 8162–8171, 2021.
  * [59] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [60] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018.
  * [61] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  International Journal of Computer Vision, pages 1–23, 2022.
  * [62] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  Advances in Neural Information Processing Systems, 35:4328–4343, 2022.
  * [63] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  Advances in neural information processing systems, 27, 2014.
  * [64] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman.  Grounded human-object interaction hotspots from video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688–8697, 2019.
  * [65] Michael J Swain and Dana H Ballard.  Color indexing.  International journal of computer vision, 7(1):11–32, 1991.
  * [66] Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba.  Learning to predict where humans look.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2106–2113, 2009.
  * [67] Robert J Peters, Asha Iyer, Laurent Itti, and Christof Koch.  Components of bottom-up gaze allocation in natural images.  Vision research, 45(18):2397–2416, 2005.
  * [68] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.

## Appendix

##  Appendix A Diff-IP2D vs. DiffuSeq

Following the derivation of DiffuSeq [26] which is used for text generation,
we prove that Diff-IP2D predicting future hand states can be regarded as an
iter-NAR process, leading to less error accumulation. We first introduce a
series of intermediate HOI states $\\{\mathbf{F}_{s}^{y}\\}_{s=0}^{S}$ decoded
from $\\{\mathbf{y}_{s}\\}_{s=0}^{S}$, where $\mathbf{y}_{s}$ denotes the
future part of $\mathbf{z}_{s}$ and
$\mathbf{y}_{S}\sim\mathcal{N}(0,\textbf{\text{I}})$. $\mathbf{F}^{x}$
represents the past latent HOI features
$F_{\text{seq}}^{\scriptstyle{\text{R}}}$ or
$F_{\text{seq}}^{\scriptstyle{\text{L}}}$ from SOFM. $\mathbf{M}$ denotes the
egomotion guidance $M_{\text{seq}}$ here and will be extended by other
perception information in our future work. Therefore, the inference process of
the diffusion-based approach can be formulated as:

$\displaystyle~{}p_{\scriptscriptstyle\text{Diff-
IP2D}}(\mathbf{F}^{y}|\mathbf{F}^{x})$ $\displaystyle=$
$\displaystyle{\sum_{\mathbf{F}^{y}_{S},\ldots,\mathbf{F}^{y}_{1}}\int_{\mathbf{y}_{S},\ldots,\mathbf{y}_{0}}}{p(\mathbf{F}^{y}|\mathbf{y}_{0},\mathbf{F}^{x})}\prod_{s=S,\ldots,1}{p(\mathbf{y}_{s-1}|\mathbf{F}^{y}_{s})p(\mathbf{F}^{y}_{s}|\mathbf{y}_{s},\mathbf{F}^{x},\mathbf{M})}$
$\displaystyle=$
$\displaystyle{\sum_{\mathbf{F}^{y}_{S},\ldots,\mathbf{F}^{y}_{1}}\int_{\mathbf{y}_{S},\ldots,\mathbf{y}_{0}}}p(\mathbf{F}^{y}_{S}|\mathbf{y}_{S},\mathbf{F}^{x})\prod_{s=S-1,\ldots,0}{p(\mathbf{F}^{y}_{s}|\mathbf{y}_{s},\mathbf{F}^{x})p(\mathbf{y}_{s}|\mathbf{F}^{y}_{s+1},\mathbf{M})}$
$\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}\mathbf{F}^{y}_{S},\ldots,\mathbf{F}^{y}_{1}\end{subarray}}p(\mathbf{F}^{y}_{S}|\mathbf{y}_{S},\mathbf{F}^{x})\prod_{t=S-1,\ldots,0}{{\int}_{\mathbf{y}_{s}}p(\mathbf{F}^{y}_{s}|\mathbf{y}_{s},\mathbf{F}^{x})p(\mathbf{y}_{s}|\mathbf{F}^{y}_{s+1},\mathbf{M})}.$
(8)

Then we marginalize over $\mathbf{y}$ and obtain the iter-NAR form of this
approach:

$\displaystyle~{}p_{\scriptscriptstyle\text{Diff-
IP2D}}(\mathbf{F}^{y}|\mathbf{F}^{x})$ $\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}\mathbf{F}^{y}_{S},\ldots,\mathbf{F}^{y}_{1}\end{subarray}}p(\mathbf{F}^{y}_{S}|\mathbf{y}_{S},\mathbf{F}^{x})\prod_{t=S-1,\ldots,0}{p(\mathbf{F}^{y}_{s}|\mathbf{F}^{y}_{s+1},\mathbf{F}^{x},\mathbf{M})}$
$\displaystyle\equiv$
$\displaystyle\sum_{\mathbf{F}^{y}_{1},\ldots,\mathbf{F}^{y}_{K-1}}{p(\mathbf{F}^{y}_{1}|\mathbf{F}^{x})\vphantom{\prod_{k=1,\ldots,K-1}}\prod_{k=1,\ldots,K-1}{p(\mathbf{F}^{y}_{k+1}|\mathbf{F}^{y}_{k},\mathbf{F}^{x},\mathbf{M})}}.$
(9)

It is nontrivial to get the explicit form of HOI states
$\\{\mathbf{F}_{s}^{y}\\}_{s=0}^{S}$ by feeding
$\\{\mathbf{y}_{s}\\}_{s=0}^{S}$ to our proposed HTH and OAH. Then we can
round them to the integer pixel coordinates in the image plane. However, it is
not easy to transform the pixel coordinates back to the latent HOI feature
space using only MFE and SOFM since we cannot recover global/hand/object
features additionally. This suggests that Eq. (9) cannot be calculated as the
previous work [26] explicitly. In this work, we solve this problem from the
perspective of the latent space, which means that what we pursue using the
diffusion model is to recover implicit features of future HOI states instead
of the explicit hand waypoints or contact points. Therefore, we can regard the
above-mentioned iterative process (latents$\rightarrow$ explicit
HOI$\rightarrow$ latents) as an equivariant mapping (latents$\rightarrow$
latents). Eq. (9) can be further replaced by Eq. (A) directly in this work,
presenting an iter-NAR form of our proposed Diff-IP2D:

$\displaystyle~{}p_{\scriptscriptstyle\text{Diff-
IP2D}}(\mathbf{y}|\mathbf{F}^{x})$ $\displaystyle=$
$\displaystyle\sum_{\mathbf{y}_{1},\ldots,\mathbf{y}_{K-1}}{p(\mathbf{y}_{1}|\mathbf{F}^{x})\vphantom{\prod_{k=1,\ldots,K-1}}\prod_{k=1,\ldots,K-1}{p(\mathbf{y}_{k+1}|\mathbf{y}_{k},\mathbf{F}^{x},\mathbf{M})}}$
$\displaystyle=$
$\displaystyle\sum_{\mathbf{y}_{1},\ldots,\mathbf{y}_{K-1}}{\prod_{i=1,\ldots,N_{\text{f}}}p(\mathbf{y}_{1,i}|\mathbf{F}^{x})\vphantom{\prod_{k=1\ldots
K-1}}\prod_{k=1,\ldots,K-1}\prod_{i=1,\ldots,N_{\text{f}}}{p(\mathbf{y}_{k+1,i}|\mathbf{y}_{k,1:N_{\text{f}}},\mathbf{F}^{x},\mathbf{M})}}.$
(10)

##  Appendix B Motion-related gaps and egomotion homography

In this section, we provide a detailed analysis for filling the motion-related
gaps mentioned in Sec. 1 with the egomotion homography. To narrow the view gap
between the last observation and the other observations, homography works as a
bridge to connect the pixel positions $\mathbf{p}_{0}$,
$\mathbf{p}_{t}\in\mathbb{R}^{2}$ of one 3D hand waypoint/contact point on
$I_{t}$ ($t\leq 0$) and $I_{0}$, which can be represented by
$\mathbf{p}_{0}=M_{t}\mathbf{p}_{t}$. We let the denoising network be aware of
the egomotion features $E_{t}$ encoded from $M_{t}$ and enable it to capture
the above-mentioned transformation when predicting future hand trajectories
and contact points on the last observed image as a canvas.

For the 2D-3D gaps, we first discover the relationship between 2D pixel
movements and 3D hand movements. For a 3D point that moves from
$\mathbf{P}_{t}\in\mathbb{R}^{3}$ in the camera coordinate system at timestamp
$t$ ($t\leq 0$) to $\mathbf{P}_{0}\in\mathbb{R}^{3}$ in the camera coordinate
system at timestamp $t=0$, we first project them to the image plane by
$\mathbf{p}_{t}=K\mathbf{P}_{t}$ and $\mathbf{p}_{0}=K\mathbf{P}_{0}$, where
$K$ is the intrinsic parameters. Then we transform $\mathbf{p}_{t}$ to the
last canvas image by $\mathbf{p}^{\prime}_{t}=M_{t}\mathbf{p}_{t}$. The 2D
pixel movement on the last image can be formulated as:

$\displaystyle\mathbf{p}_{0}-\mathbf{p}^{\prime}_{t}=K\mathbf{P}_{0}-M_{t}\mathbf{p}_{t}=K\mathbf{P}_{0}-M_{t}K\mathbf{P}_{t}.$
(11)

Therefore, the 3D action ($\mathbf{P}_{t}\rightarrow\mathbf{P}_{0}$) uniquely
corresponds to the 2D pixel movement
($\mathbf{p}_{t}\rightarrow\mathbf{p}_{0}$) once $K$ and $M_{t}$ are both
determined. Since $K$ is a constant for each video clip, only $M_{t}$ changing
along the time axis determines the spatial relationship between observations.
Therefore, we enable our proposed model aware of egomotion by encoding $M_{t}$
to a feature vector absorbed by MHCA of MADT as mentioned in Sec. 3.2,
narrowing the gap between 2D pixel movement and 3D actions. Note that we do
not utilize SE(3) here due to the scale-agnostic estimation with only 2D
images as input.

##  Appendix C Additional implementation details

###  C.1 More details about the proposed protocol for joint evaluation

In this section, we clarify the necessity of providing the novel evaluation
protocol mentioned in Sec. 4.1. Hand-object interaction is an object-centric
process [12] where the moving hands tend to change the state of the active
objects. Therefore, there must be a strong correlation between predicted hand
trajectories and object hotspots, which should be considered by the evaluation
protocol. If only separately testing hand trajectory prediction and object
affordance prediction, the performance difference in some cases as Fig. 7
illustrates cannot be identified accurately. In Fig. 7(a), the blue trajectory
and grey trajectory have the same displacement error, indicating the same
prediction performance on the off-the-shelf metrics. However, the blue one
associates weakly with the affordance map actually showing worse prediction
performance compared to the grey one. In Fig. 7(b), the two affordance
predictions have the same similarity with the ground-truth one but the red one
should be regarded as the better prediction because it leads to more coherent
intersections with the hand trajectory. Therefore, our proposed new evaluation
protocol comprehensively considers both tasks in one single metric. It
assesses object-centric interaction by the combination of potential
interaction points $\\{\bar{H}_{n}^{\text{ip}}\\}_{n=1}^{N_{\text{o}}}$ (hand
waypoints closest to predicted contact points) from the hand trajectory and
contact points $\\{O_{n}\\}_{n=1}^{N_{\text{o}}}$ from the object affordance.
This also provides a reasonable metric for future works in the literature of
hand-object interaction.

Figure 7: Two cases in which the off-the-shelf metrics cannot reasonably
distinguish the quality of different HOI predictions. We advocate that better
prediction should provide a stronger correlation with the active object.

###  C.2 More details about datasets and Diff-IP2D training configurations

The training sets of EK55 and EK100 contain 8523 and 24148 video clips
respectively. Their test sets consist of 1894 and 3513 samples for hand
trajectory evaluation, and 241 and 401 samples for object affordance
evaluation. In contrast to Epic-Kitchens, the EG dataset offers a smaller data
volume, including 1880 training samples, 442 evaluation hand trajectories, and
69 evaluation interaction hotspots. All the training sets are automatically
generated following Liu _et al._ [12]. Note that we exclusively use the test
part of the EG dataset to assess generalization ability in the experiments of
Sec. 4.2 and Sec. 4.3 since it contains insufficient training samples for
reasonable convergence.

For training Diff-IP2D, we use AdamW optimizer [68] with a learning rate
$2$e-$4$. The loss weights in Eq. (5) are set as $\lambda_{\text{VLB}}=1$,
$\lambda_{\text{traj}}=1$, $\lambda_{\text{aff}}=0.1$, and
$\lambda_{\text{reg}}=0.2$. All the networks in Diff-IP2D are trained for $30$
epochs with a batch size of $8$ on 2 A100 GPUs.

##  Appendix D Additional experimental results

###  D.1 Joint evaluation on the effect of egomotion guidance

We present the supplementary evaluation results in the ablation study on
egomotion guidance. Our proposed joint evaluation protocol is applied here to
show the positive effect of egomotion guidance for denoising diffusion. As can
be seen in Tab. 3, the use of the egomotion features enhances the joint
prediction performance of Diff-IP2D on both EK55 and EK100. EK100 has a larger
data volume which contains much more human motion patterns than EK55, leading
to larger improvement on SIM∗, AUC-J∗, and NSS∗ by $0.2\%$, $0.4\%$, and
$3.0\%$ respectively.

Table 3: Joint evaluation results in the ablation study on egomotion guidance approach | EK55 | EK100
---|---|---
SIM${}^{*\\!}\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$ |  SIM${}^{*\\!}\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$
Diff-IP2D* | 0.216 | 0.718 | 0.842 | 0.198 | 0.712 | 0.778
Diff-IP2D | 0.222 | 0.730 | 0.888 | 0.204 | 0.727 | 0.844
improvement | 2.8% | 1.7% | 5.5% | 3.0% | 2.1% | 8.5%
Diff-IP2D*: Diff-IP2D w/o egomotion guidance

###  D.2 Ablation study on observation time

We use the EK55 dataset to demonstrate the effect of observation time on HOI
prediction performance. We present the change of hand trajectory prediction
errors with different input sequence lengths $\\{2,4,6,8,10\\}$, corresponding
to the observation time
$\\{0.5\,\text{s},1.0\,\text{s},1.5\,\text{s},2.0\,\text{s},2.5\,\text{s}\\}$.
We first use Diff-IP2D trained with $10$ observation frames to implement zero-
shot prediction with different sequence lengths. Fig. 8(a) illustrates that
the prediction performance drops significantly when the number of observation
frames decreases. In contrast, once our proposed model is trained from scratch
with the predefined observation time, it generates plausible prediction
results as Fig. 8(b) shows. Especially when the number of observation frames
decreases to $4$, our method still outperforms the baseline which is trained
from scratch with $10$ observation frames. This demonstrates the strong
generation ability of our diffusion-based approach with limited conditions.

(a) Zero-shot prediction

(b) Prediction by models trained from scratch

Figure 8: Ablation study on observation time. The reference line represents
the performance of the second-best baseline trained from scratch using $10$
observation frames.

###  D.3 Ablation study on the number of diffusion steps

In this experiment, we present the effect of different diffusion steps of our
proposed Diff-IP2D. We choose four different total steps
$\\{500,1000,1500,2000\\}$ to train the denoising network and further reveal
the influence of multiple sampling intervals for inference sampling. Fig. 9
depicts the overall trend that the displacement error rises with the increase
in the sampling interval on EK55 and EK100. Diff-IP2D outperforms the second-
best baseline in Tab. 1 using sampling intervals $\leq 4$ and $\leq 6$ for all
four diffusion step numbers on EK55 and EK100 respectively. Moreover, we
surprisingly found that our model with the steps of 1000 and 2000 still has
better prediction performance with a sampling interval of 9 compared to the
reference on the two datasets.

(a) WDE vs. sampling interval on EK55

(b) WDE vs. sampling interval on EK100

Figure 9: Comparison of different numbers of diffusion steps with multiple
sampling intervals.

###  D.4 Ablation study on regularization

We provide an additional ablation study on regularization which links
$\\{F^{\scriptstyle{\text{R}}}_{\text{seq}},F^{\scriptstyle{\text{L}}}_{\text{seq}}\\}$
and
$\\{\hat{F}^{\scriptstyle{\text{R}}}_{\text{seq}},\hat{F}^{\scriptstyle{\text{L}}}_{\text{seq}}\\}$
to improve prediction performance. The experimental results on the EK100
dataset are shown in Tab. 4, which suggests that the proposed regularization
term remarkably enhances the prediction performance on both hand trajectories
and object affordances even if it is only used to link the latent space with
hand trajectory prediction.

Table 4: Ablation study on the regularization term $\mathcal{L}_{\text{reg}}$ approach | hand trajectory | object affordance | joint evaluation
---|---|---|---
WDE $\downarrow$ |  FDE $\downarrow$ |  SIM $\uparrow$ |  AUC-J $\uparrow$ |  NSS $\uparrow$ |  SIM${}^{*\\!}\uparrow$ |  AUC-J${}^{*\\!}\uparrow$ |  NSS${}^{*\\!}\uparrow$
Diff-IP2D⋆ | 0.430 | 0.195 | 0.205 | 0.718 | 0.821 | 0.180 | 0.692 | 0.722
Diff-IP2D | 0.407 | 0.187 | 0.211 | 0.736 | 0.917 | 0.204 | 0.727 | 0.844
improvement | 5.3% | 4.1% | 2.9% | 2.5% | 11.7% | 13.3% | 5.1% | 16.9%
Diff-IP2D⋆: Diff-IP2D w/o regularization |  |

###  D.5 Inference time

Figure 10: Inference time and WDE for different sampling intervals of Diff-
IP2D†.

Different applications have distinct demands to balance the real-time
performance and prediction capability in HOI prediction. For pretraining in
the field of robot manipulation, running efficiency is sometimes not crucial
due to the offline policy or slow robot movement. However, in some other cases
such as service robot control and extended reality, predictions must be
completed before the arrival of the first incoming frame. Therefore, we
propose a light version of Diff-IP2D, dubbed Diff-IP2D†, and report its
runtime of the diffusion-based inference on the EK100 dataset here. Diff-IP2D†
has the same weight as Diff-IP2D but only uses reconfigured $100$ steps
compared to the total denoising diffusion process ($S=1000$), which has also
been demonstrated to perform good prediction in the experiment. Note that we
collect the features of two sides to one batch for accelerated denoising since
the same MADT is utilized to potentially capture the interaction of two hands.
In Fig. 10, WDE reference represents the prediction performance of the second-
best baseline in Tab. 1. Time reference corresponds to the arrival time of the
next future frame. Diff-IP2D† achieves the state-of-the-art prediction
performance and simultaneously has reasonable real-time performance when the
sampling interval is set to $\\{3,4,5,7,9\\}$. Therefore, Diff-IP2D† can be
exploited in some applications with high real-time requirements.

###  D.6 Additional visualization of object affordance prediction on Epic-
Kitchens

We additionally illustrate the predicted contact points with average distances
to the ground-truth points on frames of Epic-Kitchens. As Fig. 11 shows, our
proposed method still outperforms the second-best baseline considering the
center of 10 predicted candidates.

Figure 11: Visualization of object affordance prediction grounded on frames of
Epic-Kitchens. The ground-truth contact points are represented by red dots.
The contact points predicted by our Diff-IP2D with average distances to the
ground-truth points are represented by white dots. The counterparts predicted
by OCT [12] are represented by blue dots. Figure 12: Two additional
explanatory cases.

We also provide two cases in which our Diff-IP2D predicts object affordances
away from ground truth but more reasonable than the counterparts of the
baseline. As Fig. 12 shows, our proposed Diff-IP2D focuses more on
“meaningful” parts of objects such as handles even though its prediction has a
similar distance away from ground-truth contact points compared to the
baseline.

##  Appendix E Limitations and broader impacts

While Diff-IP2D can achieve good prediction results, the relationship between
different parts of the denoising process and HOI prediction performance has
not been explored in this work. In the future, we plan to propose a more
flexible way to adaptively choose denoising steps. Additionally, the 3D pose-
aware interaction between the specific fingers of hands and predicted
affordance maps remains to be analyzed. We will combine techniques
synthesizing 3D hand-object contact with Diff-IP2D for better predictions.

Our proposed HOI prediction method would advance the development of service
robots and other artificial intelligence systems related to human life.
However, in line with the arguments of other vision-based assistive
techniques, our predictive approach heavily depends on daily video collection,
potentially exposing and compromising privacy about personal information such
as living environments and habits. To mitigate this, participants must be
safeguarded from unauthorized monitoring and data collection.