# MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild

Zeren Jiang∗1 Chen Guo∗1 Manuel Kaufmann1 Tianjian Jiang1
Julien Valentin2 Otmar Hilliges1 Jie Song1
1ETH Zürich 2Microsoft

###### Abstract

We present MultiPly, a novel framework to reconstruct multiple people in 3D
from monocular in-the-wild videos. Reconstructing multiple individuals moving
and interacting naturally from monocular in-the-wild videos poses a
challenging task. Addressing it necessitates precise pixel-level
disentanglement of individuals without any prior knowledge about the subjects.
Moreover, it requires recovering intricate and complete 3D human shapes from
short video sequences, intensifying the level of difficulty. To tackle these
challenges, we first define a layered neural representation for the entire
scene, composited by individual human and background models. We learn the
layered neural representation from videos via our layer-wise differentiable
volume rendering. This learning process is further enhanced by our hybrid
instance segmentation approach which combines the self-supervised 3D
segmentation and the promptable 2D segmentation module, yielding reliable
instance segmentation supervision even under close human interaction. A
confidence-guided optimization formulation is introduced to optimize the human
poses and shape/appearance alternately. We incorporate effective objectives to
refine human poses via photometric information and impose physically plausible
constraints on human dynamics, leading to temporally consistent 3D
reconstructions with high fidelity. The evaluation of our method shows the
superiority over prior art on publicly available datasets and in-the-wild
videos.

Figure 1: We propose MultiPly, a novel framework to reconstruct multiple
people in 3D from in-the-wild monocular videos. Our method can recover the
complete 3D human with high-fidelity shape and appearance, even in scenarios
involving occlusions and close interactions.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Despite rapid progress in estimating the 3D shape from monocular videos of a
single performer [11, 35, 42, 18], the analysis and reconstruction of several
closely interacting people is still limited. This imbalance is unsatisfying,
as group activities make up a significant portion of our lives. Although
systems for multi-person reconstruction have been previously investigated,
most require multi-view setups that constrain the capture area to a fixed
volume and demand specialized equipment and expertise to operate [36, 48, 47].
Being able to reconstruct multiple people in detailed 3D geometry and
appearance from monocular videos – which is also amenable to novice users –
would facilitate many downstream tasks in AR/VR, such as the telepresence of
groups of people, or the “replay” of social activities in 4D. Accomplishing
this is fundamentally a challenging task since it requires accurate pixel-
level disentanglement of individuals without a priori known geometries of the
subjects. To make matters worse, the task is further complicated by depth
ambiguities, complex human dynamics, and severe human-human occlusions – all
of which have to be resolved from a single, short video clip.

In this paper, we introduce a novel method, called MultiPly, that provides a
solution to this task: It takes a single video as input and outputs complete,
high-quality, and separated 3D human geometry and appearance for individuals
appearing in the scene (see Fig. 1). By embracing the promising paradigm of
neural implicit functions for 3D representations, remarkable progress has
recently been achieved in modelling detailed human geometry and appearance.
While some methods require full supervision on 3D human scans that can be
prohibitively expensive to acquire [35, 43], others only rely on readily
available monocular videos [11, 5, 17, 40, 20] to fit articulated neural
implicit fields with temporally consistent results. However, all of these
methods are designed for only a single actor and thus neglect the complexities
of the reconstruction task that arise from severe human-human occlusions. When
applied to footage where multiple people engage in natural interactions, the
aforementioned methods often result in corrupted or incomplete human
reconstructions.

Our method builds on the promise of neural implicit fields, and presents a
solution that overcomes the limitations of prior work. In doing so, several
challenges must be addressed. First, 2D and 3D points of each subject must be
precisely associated. Second, complete human models must be extracted and
maintained from only a single video. Third, and most critically, the problem
of person association and avatar creation are both significantly exacerbated
by strong occlusions and imperfect human pose estimates.

To tackle these challenges, our approach is grounded in the following core
concepts:  i) We design a unified, temporally consistent representation of
human shape and texture in canonical space that is applicable to all
individuals. This design facilitates the integration of partial observations
from the video sequence into a temporally coherent space, inherently
maintaining a complete human body.  ii) We establish a layered neural
representation for the entire scene, wherein we parameterize the humans and
the background as individual neural fields. The composition of these fields
leads to a layered, interwoven representation that covers the entire space and
which can be learned from a monocular RGB video via a proposed layer-wise
differentiable volume rendering.  iii) We introduce a hybrid instance
segmentation approach that leverages the advantages of self-supervised scene
decomposition in 3D and a learning-based promptable 2D segmentation module.
This combination results in a robust and accurate separation of individuals.
iv) To deal with imperfect pose estimates that might corrupt updates to the
avatar model, we present a confidence-guided optimization formulation that
alternately optimizes human poses and shape/appearance based on per-frame
confidence measures. This way we incorporate effective objectives that refine
the human poses through photometric information and impose physically
plausible constraints on human dynamics.

In our experiments, we demonstrate that our framework leads to robust human
instance segmentation and plausible pose estimates, achieving high-quality 3D
reconstructions of multiple people even under extremely challenging visual
complexities like severe occlusions (_cf_. Fig. 2). We meticulously ablate our
method, which uncovers its essential components. Furthermore, we conduct
comparisons with existing approaches in human reconstruction, novel view
synthesis, human instance segmentation, and pose estimation tasks, showing
that our method outperforms prior art across various settings. In summary, our
contributions are:

  * •

a novel framework, MultiPly, to reconstruct multiple detailed 3D human models
solely from in-the-wild monocular videos; and

  * •

a robust instance segmentation approach that achieves a clean separation
between people even under close interaction; and

  * •

a confidence-guided optimization formulation that leads to temporally and
spatially coherent 3D reconstructions of people with high fidelity.

##  2 Related Work

#### Monocular Single-Person Reconstruction

Reconstructing an individual from monocular observations has emerged as a
widely explored research challenge. Template-based approaches involve the
tracking of a human template using 2D observations [12]. The assumption of a
personalized template is unsuitable for more practical use cases. Follow-up
works endeavor to remove this dependency by adding the vertex offsets on top
of SMPL [1, 10]. Nevertheless, the explicit mesh representation is constrained
by a fixed resolution and topology, incapable of representing fine-grained
details. Learning-based methods that learn to regress 3D human shape from
images have shown compelling results [34, 15, 35, 52, 42, 2, 43]. A major
limitation of these methods is the necessity of high-quality 3D data for
supervision and they fail to produce space-time coherent reconstructions over
frames. Recent works employ neural rendering to train neural fields based on
videos to obtain articulated human model [37, 33, 17, 40, 20, 5, 11, 18, 6].
Among these, Vid2Avatar [11] achieves compelling 3D reconstruction for a
single subject but is not directly applicable to scenes with crowded people.
Actually, none of the aforementioned methods can be directly deployed in more
complicated multi-person scenarios. In contrast, we propose a novel framework
that can faithfully reconstruct multiple people in the scene from a monocular
video.

#### Monocular Multi-Person Reconstruction

In contrast to the notable advancements in reconstructing the clothed human
for an individual, limited emphasis has been placed on multi-person scenarios,
which are evidently more applicable to our daily experiences. Most existing
monocular works can only estimate the coarse body shapes of multiple people
from monocular observations [19, 7, 24, 38, 39, 21, 4, 14, 25, 46]. Mustafa et
al. [31] extend prior implicit methods to multiple people and recover
spatially coherent 3D human shapes from an RGB image but mainly deal with
cases where people are well-spaced and do not interact naturally in close
range. Recently, more researchers have shifted the focus to multi-person
scenarios [16, 51, 36, 47]. Even though these works achieve compelling
instance-level human reconstructions, they require expensive multi-view
imaging systems. Concurrently, Cha et al. [3] propose to reconstruct multiple
people from a single image. Such image-based methods usually fail to produce
space-time coherent reconstructions over frames. Overall, monocular multi-
person reconstruction is still an extremely under-explored problem. We propose
MultiPly to take a significant stride towards addressing this formidable task.

#### Human Instance Segmentation

Most works solve human or general object segmentation at the image level (i.e.
2D) [13, 22, 26, 41]. They are trained on images with human annotations to
directly regress the segmentation masks during inference. More recently, a
promptable segmentation model SAM has been developed to support flexible
prompting along with input images [23]. However, SAM is a semantic
segmentation method, which does not inherently support instance-level
segmentation for humans. Therefore, meticulous prompts need to be designed for
human instance segmentation tasks. Besides, these approaches are not able to
produce sharp boundaries between individuals, especially for closely
interacting people. More importantly, they do not always predict temporally
coherent segmentation masks, as they focus on image-level segmentation only
and incorporate no 3D knowledge. In this work, we optimize the instance
segmentation masks on the fly by leveraging the promptability of SAM [23] and
the self-supervised decomposition in 3D [36, 11].

##  3 Method

Figure 2: Method overview. Given an image and SMPL estimation, we sample human
points along the camera ray based on the bounding boxes of SMPL bodies and the
background points based on NeRF++. We warp sampled human points into canonical
space via inverse warping and evaluate the person-specific implicit network to
obtain the SDF and radiance values (Sec. 3.1). The layer-wise volume rendering
is then applied to learn the implicit networks from images (Sec. 3.2). We
build a closed-loop refinement for instance segmentation by dynamically
updating prompts for SAM using evolving human models (Sec. 3.3). Finally, we
formulate a confidence-guided optimization that only optimizes pose parameters
for unreliable frames and jointly optimizes pose _and_ implicit networks for
reliable frames (Sec. 3.4).

We present MultiPly, a novel framework for detailed geometry and appearance
reconstruction of multiple people from in-the-wild monocular videos. The
overview of our method is schematically illustrated in Fig. 2. Reconstructing
multiple people in 3D from a short video without a priori known geometries is
a challenging task due to complex human articulation, and strong occlusions.
To tackle these challenges, we first define a unified, temporally consistent
representation for humans and a layered neural representation for the entire
scene (Sec. 3.1). The layered neural representation is then learned from
images by performing our layer-wise differentiable volume rendering (Sec.
3.2). Given the self-supervised instance segmentation via occlusion-aware
volume rendering, we further enhance the instance segmentation supervision by
leveraging our evolving human surfaces in deformed space as progressively
updated prompts for SAM which builds closed-loop refinement of instance
segmentation in both 2D and 3D (Sec. 3.3). Finally, we formulate a confidence-
guided optimization to alternately optimize human pose and shape/appearance
(Sec. 3.4). We incorporate photometric information, robust instance
segmentation supervision, and the inter-person objectives for pose refinement
to achieve temporally and spatially coherent 3D reconstructions of people in
high quality (Sec. 3.5).

###  3.1 Layered Neural Representation

#### Neural Avatars.

For each human in the scene, we represent the 3D shape as an implicit signed-
distance field (SDF) and the appearance as a texture field in canonical space,
covering the entire space. When multiple people are present in the scene, it
leads to a layered representation where the contributing SDFs are potentially
interwoven. More specifically, we model the geometry and appearance of each
person $p$ in canonical space by a neural network $f^{p}$, which predicts the
signed distance value $s^{p}$ and the radiance value $\bm{c}^{p}$ at the query
point $\bm{x}_{c}^{p}$ in this space:

$\displaystyle\bm{c}^{p},s^{p}$
$\displaystyle=f^{p}(\bm{x}_{c}^{p},\bm{\theta}^{p}),$ (1)

where $\bm{\theta}^{p}$ denotes the person’s pose parameters, which we
concatenate to $\bm{x}_{c}^{p}$ to model pose-dependent surface deformations.
For simplicity, we use $f_{c}^{p}(\cdot)$ and $f_{s}^{p}(\cdot)$ to query
$\bm{c}^{p}$ and $s^{p}$ from the network outputs.

#### Deformation Module.

We follow a standard skeletal deformation based on SMPL [28] to find
correspondences in canonical and deformed space. A canonical point
$\bm{x}_{c}^{p}$ is mapped to the deformed point $\bm{x}_{d}^{p}$ via linear-
blend skinning (LBS) based on SMPL transformation:
$\bm{x}_{d}^{p}=T_{\text{smpl}}(\bm{x}_{c}^{p},\bm{\theta}^{p})$. Here,
$T_{\text{smpl}}(\cdot)$ denotes the SMPL-based transformation derived from
the body pose $\bm{\theta}^{p}$, which corresponds to LBS and is described in
more detail in the Supp. Mat. Inversely, the canonical correspondence
$\bm{x}_{c}^{p}$ for point $\bm{x}_{d}^{p}$ in deformed space is defined as
$\bm{x}_{c}^{p}=T_{\text{smpl}}^{-1}(\bm{x}_{d}^{p},\bm{\theta}^{p})$. To
invert LBS we use the SMPL skinning weight of the vertex closest to
$\bm{x}_{d}^{p}$.

###  3.2 Layer-Wise Volume Rendering

We seek to reconstruct all human subjects in the scene. This requires
different treatment compared to vanilla differentiable volume rendering that
only works on a single static scene [30]. In contrast, on the basis of our
layered neural avatar representation (Sec. 3.1), we introduce layer-wise
volume rendering to handle dynamic scenes with multiple subjects and inter-
occlusions. This is achieved by combining surface-based volume rendering [45]
with the re-assembly of multiple human neural layers [16]. It is essential to
note that the layer-wise volume rendering is inherently occlusion-aware.

#### Volume Rendering for Human Layers.

For each sampled camera ray $\bm{r}$, we sample the points in the observation
space along the ray based on the intersection between the oriented bounding
box of the deformed SMPL model and the camera ray. Specifically, we sample $N$
points $\\{\bm{x}_{d,1}^{p},...,\bm{x}_{d,N}^{p}\\}$ inside the $p$-th
intersected bounding box based on the two-stage sampling strategy proposed in
[45]. Then, the occupancy $o^{p}_{i}$ for the $p$-th person and the $i$-th
sampled point is calculated as follows:

$\displaystyle o^{p}_{i}$
$\displaystyle=\left(1-\exp\left(-\sigma_{i}^{p}\Delta\bm{x}_{i}\right)\right),$
$\displaystyle\sigma_{i}^{p}$
$\displaystyle=\sigma\left(f_{s}^{p}\left(T_{\text{smpl}}^{-1}\left(\bm{x}_{d,i}^{p},\bm{\theta}^{p}\right),\bm{\theta}^{p}\right)\right),$
(2)

where $\Delta\bm{x}_{i}$ is the distance between two adjacent sample points,
and $\sigma(\cdot)$ is the scaled Laplace distribution’s Cumulative
Distribution Function (CDF) defined in [45] to convert the signed distance
$s_{i}^{p}$ to volume density $\sigma_{i}^{p}$. Then we accumulate the
radiance by performing numerical quadrature among the layered density field
for multiple persons to obtain the color value:

$\hat{C}^{H}=\sum_{i=1}^{N}\sum_{p=1}^{P}\left[o_{i}^{p}\bm{c}_{i}^{p}\prod_{q=1}^{P}\prod_{j\in\mathcal{Z}_{i}^{q,p}}\left(1-o_{j}^{q}\right)\right],$
(3)

where $P$ is the total number of subjects, $\hat{C}^{H}$ is the rendered color
of humans, and $\mathcal{Z}_{i}^{q,p}$ contains all indices of points
(belonging to person $q$) whose depth value is lower than the depth value of
the $i$-th point of person $p$, _i.e_. $\mathcal{Z}_{i}^{q,p}=\\{j\in[1,N]\mid
z(\bm{x}_{d,j}^{q})<z(\bm{x}_{d,i}^{p})\\}$, where $z(\cdot)$ denotes the
distance between the sampled point and the camera origin along the z-axis.

#### Scene Composition.

We model the background in the same formulation as NeRF++ [49], denoted as
$f^{b}$. We thus obtain a color value $\hat{C}^{B}$ representing the
background’s color, which is composited with $\hat{C}^{H}$ via self-supervised
decomposition following [11] to compute the final pixel color value $\hat{C}$.
More details are shown in the Supp. Mat.

###  3.3 Progressive Prompt for SAM

Learning to disentangle and reconstruct multiple subjects by simply relying on
the automatic separation through layer-wise volume rendering is still a
severely ill-posed problem. This is due to dynamically changing lighting
effects (e.g., shadows) and potentially severe human-human occlusions and
close contact. To this end, we propose to leverage the promptable segmentation
model SAM [23] and design a progressive prompting strategy based on the
evolving human models to provide robust instance segmentation supervision. In
this section we describe how we design the prompt to get an updated SAM mask
which we later use in the optimization (Sec. 3.4 and Sec. 3.5).

We define the $p$-th human shape to be the zero-level set of the signed
distance function $f_{s}^{p}$ and apply the Multiresolution IsoSurface
Extraction (MISE) [29] to extract the mesh in canonical space, denoted as
$\mathcal{S}_{c}^{p}=\left\langle\mathcal{V}_{c}^{p},\mathcal{F}^{p}\right\rangle=\text{MISE}(f_{s}^{p},\bm{\theta}^{p})$.
Here, $\mathcal{V}_{c}^{p}$ represents the extracted vertex set in canonical
space, and $\mathcal{F}^{p}$ denotes the corresponding faces. Then, the
deformed vertex set in the observation space is:

$\mathcal{V}_{d}^{p}=\left\\{T_{\text{smpl}}(\bm{v}_{c}^{p},\bm{\theta}^{p})\mid\bm{v}_{c}^{p}\in\mathcal{V}_{c}^{p}\right\\}.$
(4)

Similarly, the deformed mesh for the $p$-th person in the observation space is
defined as
$\mathcal{S}_{d}^{p}=\left\langle\mathcal{V}_{d}^{p},\mathcal{F}^{p}\right\rangle$.
Thus, we can obtain an instance mask $\mathcal{M}_{\text{mesh}}^{p}$ by
differentiably rendering the deformed mesh. To improve efficiency, we opt for
a differentiable rasterizer $R$ rather than volume rendering:

$\mathcal{M}_{\text{mesh}}^{p}=R(\mathcal{S}_{d}^{p}).$ (5)

For the sake of clarity, we define $\mathcal{M}=1$ to represent the inside and
$\mathcal{M}=0$ to indicate either the outside or occlusion by other meshes.
The instance mask of deformed meshes $\mathcal{M}_{\text{mesh}}^{p}$ serves as
one of the prompts for SAM refinement. We further provide points as input
prompts. We begin by projecting the 3D keypoint candidates onto the image to
obtain 2D keypoints
$\mathcal{K}_{2d}^{p}=\left\\{\Pi\left(\mathcal{J}(\bm{\theta}^{p},\bm{\beta}^{p})\right)\right\\}$,
where $\mathcal{J}(\bm{\theta}^{p},\bm{\beta}^{p})$ is the 3D SMPL keypoints
given the pose $\bm{\theta}^{p}$ and body shape $\bm{\beta}^{p}$ parameters
for subject $p$, and $\Pi$ is the camera projection function. The point
prompts for the $p$-th subject are then defined by:

$\displaystyle\mathcal{P}_{+}^{p}=\left\\{\bm{k}\in\mathcal{K}_{2d}^{p}\mid\mathcal{M}_{\text{mesh}}^{p}(\bm{k})=1\right\\},$
(6) $\displaystyle\mathcal{P}_{-}^{p}=\\{\bm{k}\in{\textstyle\bigcup_{q\neq
p}}\mathcal{K}_{2d}^{q}\mid\mathcal{M}_{\text{mesh}}^{p}(\bm{k})=0\\}.$

In other words, the positive point prompts $\mathcal{P}_{+}^{p}$ include the
2D keypoints that are inside of the instance mask obtained from the deformed
mesh and are outside of the instance masks of all other meshes. The negative
point prompts are the union of all 2D keypoints of all other subjects that are
outside of the projected mesh mask $\mathcal{M}_{\text{mesh}}^{p}$. The SAM
instance mask $\mathcal{M}_{\text{sam}}^{p}$ is finally updated based on the
combination of the mask and point prompts:

$\mathcal{M}_{\text{sam}}^{p}=\text{SAM}(\mathcal{M}_{\text{mesh}}^{p},\mathcal{P}_{+}^{p},\mathcal{P}_{-}^{p}),$
(7)

Note that $\mathcal{M}_{\text{sam}}^{p}$ are progressively updated during
training.

###  3.4 Confidence-Guided Alternating Optimization

Human-human occlusions often lead to inaccurate pose and wrong depth order
estimation. A naïve joint optimization for both the pose and shape parameters
may end up with a suboptimal solution. To mitigate this, we introduce a
confidence-guided optimization strategy to alternately optimize the human
poses and shapes.

To avoid damaging shape updates that are due to wrong poses, we only optimize
pose parameters for unreliable frames and jointly optimize pose _and_ shape
parameters for reliable frames. We treat the IoU between the projected mesh
mask $\mathcal{M}_{\text{mesh}}^{p,i}$ and the refined SAM mask
$\mathcal{M}_{\text{sam}}^{p,i}$ as our confidence measure for the $p$-th
subject in frame $i$. We define reliable frames $\mathcal{I}_{r}$ to be those
frames with reliable poses based on the average IoU over all subjects:

$\mathcal{I}_{r}=\left\\{\bm{I}_{i}\in\mathcal{I}\mid\frac{1}{P}\sum_{p=1}^{P}\text{IoU}(\mathcal{M}_{\text{mesh}}^{p,i},\mathcal{M}_{\text{sam}}^{p,i})\geq\alpha\right\\},$
(8)

where $\mathcal{I}$ are all frames, and $\mathcal{I}\setminus\mathcal{I}_{r}$
are unreliable frames. $\alpha$ is a confidence threshold which is set to be
the median of all IoU values over the entire sequence. It’s important to note
that the confidence threshold $\alpha$ is dynamically updated during training
and eventually inaccurate pose estimates are corrected and all frames will be
used for joint optimization.

Metrics | Pose Estimation | Human Reconstruction
---|---|---
MPJPE $\downarrow$ |  MVE $\downarrow$ |  CD $\downarrow$ |  PCDR $\uparrow$ |  V-IoU $\uparrow$ |  $\mathbf{C}-\ell_{2}$ $\downarrow$ |  P2S $\downarrow$ |  NC $\uparrow$
Initial pose | 75.3 | 90.8 | 235.6 | 0.566 | - | - | - | -
Layer-wise volume rendering | 71.5 | 86.2 | 245.6 | 0.608 | 0.773 | 3.14 | 2.87 | 0.752
\+ Progressive SAM | 71.2 | 85.9 | 246.2 | 0.609 | 0.786 | 2.68 | 2.42 | 0.784
\+ Confidence-guided OPT | 69.4 | 83.6 | 218.4 | 0.709 | 0.816 | 2.53 | 2.34 | 0.789
Table 1: Quantitative ablation studies on Hi4D. We demonstrate the importance
of the proposed progressive prompt for SAM and confidence-guided alternating
optimization. Both key components effectively contribute to the final
reconstruction quality (_cf_. Fig. 3).

###  3.5 Objectives

Reconstruction Loss. We calculate the $L_{1}$-distance between the rendered
color $\hat{C}(\mathbf{r})$ and the image pixel’s RGB value $C(\mathbf{r})$
over all sampled rays $\mathcal{R}$:

$L_{\text{rgb}}=\frac{1}{|\mathcal{R}|}\sum_{\bm{r}\in\mathcal{R}}|C(\bm{r})-\hat{C}(\bm{r})|.$
(9)

Instance Mask Loss. We first modify Eq. 3 to differentiably render the opacity
$\hat{O}^{p}(\bm{r})$ per person per pixel:

$\hat{O}^{p}(\bm{r})=\sum_{i=1}^{N}\left[o_{i}^{p}\prod_{q=1}^{P}\prod_{j\in\mathcal{Z}_{i}^{q,p}}\left(1-o_{j}^{q}\right)\right].$
(10)

Then the instance mask loss is calculated between the refined instance mask
and the rendered pixel-wise opacity:

$L_{\text{mask}}=\frac{1}{|\mathcal{R}|}\sum_{\bm{r}\in\mathcal{R}}\sum_{p=1}^{P}|\mathcal{M}_{\text{sam}}^{p}(\bm{r})-\hat{O}^{p}(\bm{r})|.$
(11)

Eikonal Loss. Following [9], we sample points in the canonical space for each
subject and enforce the Eikonal constraint to ensure $f_{s}^{p}$ is a valid
SDF:

$L_{e}=\sum_{p=1}^{P}\mathbb{E}_{\bm{x}_{c}}\left(\left\|\nabla
f_{s}^{p}\left(\bm{x}_{c}^{p},\bm{\theta}^{p}\right)\right\|-1\right)^{2}.$
(12)

We further introduce two inter-person objectives for pose refinement to ensure
spatially coherent and physically plausible reconstructions. We apply these
constraints explicitly on the deformed mesh to refine human poses while the
deformed meshes $\mathcal{S}_{d}^{p}$ are updated on the fly during training.
To be specific, the following two additional losses are used periodically
during training to optimize pose only:

Depth Order Loss. The effect of wrong depth order on the reconstruction
quality can be severe, resulting in reversed geometry and texture. Similar to
[19], we apply a depth order loss as follows:

$L_{\text{depth}}=\sum_{(\bm{u},p,q)\in\mathcal{D}}\log(1+\exp(D_{p}(\bm{u})-D_{q}(\bm{u}))),$
(13)

where $\mathcal{D}=\left\\{(\bm{u},p,q)\mid p\neq
q,\mathcal{M}_{\text{sam}}^{p}(\bm{u})=\mathcal{M}_{\text{mesh}}^{q}(\bm{u})=1\right\\}$
represents the set of pixels $\bm{u}$ where we have depth ordering mistakes
between the $p$-th and $q$-th persons. $D_{p}(\bm{k})$ denotes the depth of
the $p$-th mesh for pixel $\bm{u}$.

Interpenetration Loss. We shoot a ray for the sampled vertex in
$\mathcal{V}_{d}^{p}$ in Eq. 4 to check the number of intersection with other
meshes. Then, we use the parity of the number of intersections to determine
whether that point is inside other meshes. Following [8], the interpenetration
loss is calculated as follows:

$L_{\text{inter}}=\sum_{p=1}^{P}\sum_{q=1,q\neq
p}^{P}\left\|\mathcal{V}_{\text{in}}^{p,q}-NN(\mathcal{V}_{\text{in}}^{p,q},S_{d}^{q})\right\|_{2},$
(14)

where $\mathcal{V}_{\text{in}}^{p,q}$ denotes the $p$-th person’s vertex which
is inside the $q$-th person’s mesh, and $NN(\mathcal{V},\mathcal{S})$ finds
the nearest vertex in $\mathcal{S}$ for each point in $\mathcal{V}$. Different
from [8, 19] where they deploy the depth order and interpenetration loss on
the naked parametric human model, we apply those two losses on our learned
pixel-wise aligned clothed human meshes, leading to a more fine-grained
optimization.

See Supp. Mat for more details about the final loss.

Input

Volume rendering

\+ Prog. SAM

\+ Conf.-guided OPT

Figure 3: Qualitative ablation studies. Our progressive prompting strategy
provides robust instance segmentation supervision and eliminates the noises
caused by the environmental dynamic effects. The confidence-guided
optimization further improves the reconstruction results and maintains
complete human bodies.

##  4 Experiments

We first introduce the datasets and metrics used for evaluation. Next,
ablation studies are conducted to demonstrate the effectiveness of our design
choices. We then compare our proposed method with state-of-the-art approaches
in four tasks, including human reconstruction, novel view synthesis, human
instance segmentation, and pose estimation.

###  4.1 Datasets and Metrics

Hi4D [47]. This dataset contains challenging human interactions between pairs
of people with ground truth meshes, human poses, and instance segmentation
masks. We use Hi4D to evaluate our approach to all tasks.

Monocular Multi-huMan (MMM). Since the Hi4D dataset only contains two-person
interactions with the static camera in the stage. In order to evaluate the
generalization of our method, we collect a dataset called Monocular Multi-
huMan (MMM) by using a hand-held smartphone, which contains six sequences with
two to four persons in each sequence. Half of the sequences are captured in
the stage with ground truth annotations for quantitative evaluation and the
others are captured in the wild for qualitative evaluation.

Metrics. We consider the following metrics for human mesh reconstruction
evaluation: volumetric IoU (V-IoU), Chamfer distance ($\mathbf{C}-\ell_{2}$)
[cm], point-to-surface distance (P2S) [cm], and normal consistency (NC).
Rendering quality is measured via three metrics: PSNR, SSIM, and LPIPS. We
assess human pose estimation using four metrics: MPJPE [mm], MVE [mm], Contact
Distance (CD) [mm], and Percentage of Correct Depth Relations (PCDR) with a
threshold of 0.15m. Lastly, we report IoU, Recall, and F1 score for human
instance segmentation.

###  4.2 Ablation study

As depicted in Tab. 1, ablation studies are conducted to demonstrate the
effectiveness of the proposed progressive SAM prompt and confidence-guided
optimization strategy. Both pose estimation and human reconstruction tasks are
evaluated here. The initial human poses are obtained from TRACE [39] and
ViTPose [44], more details are provided in the Supp. Mat. We initiate ablation
studies based on our layer-wise volume rendering by naively optimizing the
human pose and shape/appearance jointly without using SAM.

#### Human Reconstruction.

Applying instance mask supervision based on progressively refined SAM outputs
significantly improves the output quality and drastically reduces the
reconstruction error (Chamfer distance) as quantitatively shown in Tab. 1.
This is also confirmed by the qualitative results within the orange bounding
boxes highlighted in Fig. 3, where layer-wise volume rendering purely relying
on self-supervised segmentation fails to separate the dynamic shadows from the
human, leading to noisy reconstructions. The reconstruction quality is further
improved with the proposed confidence-guided optimization, as quantitatively
indicated by the last row of Tab. 1. The red bounding boxes in Fig. 3 serve as
visual evidence for the importance of our confidence-guided optimization. The
presence of incomplete human bodies, such as the broken leg, disappeared back,
and shrunken neck, is attributed to the incorrect depth order and pose
estimation error. By temporarily freezing the implicit network for frames with
unreliable poses, we circumvent such detrimental shape updates, leading to a
complete human reconstruction.

#### Pose Estimation.

We observe that this performance aligns with geometric reconstruction.
Compared with the initial pose, we achieve not only more accurate individual
poses (MPJPE and MVE) but also a better spatial arrangement between people,
reflected on CD and PCDR.

###  4.3 Reconstruction Comparisons

To the best of our knowledge, there are few video-based reconstruction methods
designed for clothed multiple people. Hence, we adapt two state-of-the-art
reconstruction approaches to our setting for comparison. ECON [43] is an
image-based 3D human reconstruction method capable of handling multi-person
scenarios. While evaluating ECON, we discard frames with incorrect bounding-
box detections for a fair comparison. Vid2Avatar (V2A) [11] is a video-based
human reconstruction method designed for a single person. We extend V2A to
multi-person scenarios by training a distinct model for each subject
individually. Our method outperforms [43, 11] by a substantial margin on both
datasets and all metrics (_cf_. Tab. 2). This disparity becomes more visible
in qualitative comparisons shown in Fig. 4. When people closely interact, both
ECON and V2A fail to recover complete human bodies but only output corrupted
reconstructions (_e.g_., missing legs/heads). Furthermore, they struggle with
the initial depth order/pose error and produce spatially incorrect
reconstructions in 3D. V2A tends to model environmental dynamic effects
(_e.g_., shadows) as the human body, resulting in noisy reconstructions. These
artifacts are highlighted within the colored bounding boxes of Fig. 4. In
contrast, our method generates complete human shapes with sharp boundaries and
spatially coherent 3D reconstructions. We attribute this superiority to our
proposed representation design and learning schemes.

Figure 4: Qualitative reconstruction comparison. We show both the overlaid and
separated reconstruction results for each method. Red bounding boxes: the
incomplete reconstruction of the occluded part. Orange bounding boxes:
incorrect instance segmentation results caused by the surrounding visual
complexities. Black bounding boxes: inaccurate spatial arrangement due to pose
error.

Dataset | Method |  V-IoU $\uparrow$ |  $\mathbf{C}-\ell_{2}$ $\downarrow$ |  P2S $\downarrow$ |  NC $\uparrow$
---|---|---|---|---|---
Hi4D |  ECON [43] | 0.787 | 3.72 | 3.59 | 0.746
V2A [11] | 0.783 | 3.02 | 2.46 | 0.775
Ours | 0.816 | 2.53 | 2.34 | 0.789
MMM |  ECON [43] | 0.760 | 4.17 | 3.71 | 0.705
V2A [11] | 0.812 | 3.34 | 2.68 | 0.735
Ours | 0.826 | 2.89 | 2.40 | 0.757

Table 2: Quantitative reconstruction evaluation. Our method consistently
outperforms all baselines on both datasets and all evaluation metrics (_cf_.
Fig. 4).

###  4.4 Novel View Synthesis Comparisons

To the best of our knowledge, there are few novel view synthesis approaches
particularly designed for clothed multiple people from monocular video. Hence,
we adapt Shuai et al. [36], which is a state-of-the-art multi-person novel
view synthesis approach from multi-view videos, to the monocular setting for a
fair comparison. We share the same human pose initialization for training.
Then, we use the ground truth human poses and camera parameters to render the
novel view. As shown in Tab. 3 our method outperforms [36] on all metrics.
Fig. 5 shows that the rendered image from [36] is more blurry and has noisy
artifacts compared to ours. The reasons are twofold: 1) it lacks a reliable
pose correction mechanism, leading to large inconsistency between human pose
and image information during training, and 2) weekly-supervised decomposition
cannot ensure robust instance segmentation under close human interaction. Our
framework generates more plausible renderings with clearly sharp boundaries.

Method |  SSIM $\uparrow$ |  PSNR $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Shuai et al. [36] | 0.898 | 19.6 | 0.1099
Ours | 0.915 | 20.7 | 0.0798
Table 3: Quantitative rendering evaluation on Hi4D. Our method achieves better
rendering quality (_cf_. Fig. 5).

Input

Reference

Shuai et al.

Ours

Figure 5: Qualitative rendering comparison. Our method achieves more plausible
renderings with sharp boundaries.

###  4.5 Instance Segmentation Comparisons

We compare our instance segmentation result from SAM after convergence with
pretrained human instance segmentation network SCHP [26]. Tab. 4 reveals that
our initial SAM outputs achieve comparable results with SCHP. However, the
initial SAM masks are unsatisfactory due to the noisy prompt from the
inaccurate SMPL estimation, leading to an incomplete and implausible
reconstruction result (_e.g_., the missing body part and self-interpenetration
in the red bounding boxes) as shown in Fig. 6. Finally, our progressive
prompting strategy based on our evolving human models helps to achieve
temporally consistent and complete segmentation masks and high-quality
reconstructions, surpassing the baseline methods.

Method |  IoU $\uparrow$ |  Recall $\uparrow$ |  F1 $\uparrow$
---|---|---|---
SCHP [26] | 0.937 | 0.983 | 0.982
Ours (Init.) | 0.943 | 0.975 | 0.984
Ours (Progressive) | 0.963 | 0.985 | 0.990
Table 4: Quantitative instance segmentation evaluation on Hi4D. Our method
achieves the best segmentation accuracy.

Init. SAM

Init. Recon

Prog. SAM

Prog. Recon.

Figure 6: Qualitative instance segmentation comparison. Progressive prompting
strategy provides more robust and fine-grained instance segmentation
supervision compared to the initial SAM outputs, leading to higher quality of
reconstructions.

###  4.6 Pose Estimation Comparisons

We conduct a comparison of our method with state-of-the-art bottom-up (TRACE
[39]) and top-down (CLIFF [27]) multi-person pose estimation approaches. To
adapt CLIFF for close human interaction, we employ ByteTrack [50] and linear
interpolation to estimate missing persons caused by detector errors. Our
approach consistently outperforms other baseline methods on all metrics as
shown in Tab. 5. Specifically, our method shows its superiority in pose
estimation accuracy of individuals (MPJPE and MVE) and more reasonable spatial
arrangement between pairs of people (CD and PCDR). This is also confirmed by
qualitative results. Please refer to the Supp. Mat.

Method |  MPJPE $\downarrow$ |  MVE $\downarrow$ |  CD $\downarrow$ |  PCDR $\uparrow$
---|---|---|---|---
CLIFF [27] | 85.7 | 102.1 | 351.7 | 0.606
TRACE [39] | 95.6 | 115.7 | 249.4 | 0.603
Ours | 69.4 | 83.6 | 218.4 | 0.709
Table 5: Quantitative pose estimation evaluation on Hi4D. Our method
outperforms state-of-the-art multi-person pose estimation methods on all
evaluation metrics.

##  5 Conclusion

In this paper, we present MultiPly, which for the first time produces
temporally and spatially coherent 3D reconstructions of multiple people with
high fidelity from monocular in-the-wild videos. We utilize carefully designed
layered neural representation and dynamically refined instance segmentation
supervision. We further introduce a confidence-guided optimization to learn
human neural layers via layer-wise volume rendering. Our method is able to
reconstruct multiple high-quality 3D human models in challenging scenarios
involving close human interactions and strong inter-person occlusions.

Limitations: The complexity of our model increases linearly with the number of
involved persons, making it inefficient for crowds. Our method does not
explicitly model hands and we see the integration of an expressive human model
[32] as a future direction. We discuss more limitations and potential negative
societal impact in Supp. Mat.

Acknowledgement: This work was partially supported by the Swiss SERI
Consolidation Grant “AI-PERCEIVE”.

## References

  * Alldieck et al. [2018] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll.  Video based reconstruction of 3d people models.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8387–8397, 2018.
  * Alldieck et al. [2022] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu.  Photorealistic monocular 3d reconstruction of humans wearing clothing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Cha et al. [2024] Junuk Cha, Hansol Lee, Jaewon Kim, Nhat Nguyen Bao Truong, Jae Shin Yoon, and Seungryul Baek.  3d reconstruction of interacting multi-person in clothing from a single image, 2024.
  * Cheng et al. [2023] Yu Cheng, Bo Wang, and Robby T. Tan.  Dual networks based 3d multi-person pose estimation from monocular video.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):1636–1651, 2023.
  * Feng et al. [2022] Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J. Black, and Timo Bolkart.  Capturing and animation of body and clothing from monocular video.  In _SIGGRAPH Asia 2022 Conference Papers_ , 2022.
  * Feng et al. [2023] Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, and Michael J. Black.  Learning disentangled avatars with hybrid 3d representations.  _arXiv_ , 2023.
  * Fieraru et al. [2020] Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu.  Three-dimensional reconstruction of human interactions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7214–7223, 2020.
  * Fieraru et al. [2021] Mihai Fieraru, Mihai Zanfir, Teodor Szente, Eduard Bazavan, Vlad Olaru, and Cristian Sminchisescu.  Remips: Physically consistent 3d reconstruction of multiple interacting people under weak supervision.  In _Advances in Neural Information Processing Systems_ , pages 19385–19397. Curran Associates, Inc., 2021.
  * Gropp et al. [2020] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  _arXiv preprint arXiv:2002.10099_ , 2020.
  * Guo et al. [2021] Chen Guo, Xu Chen, Jie Song, and Otmar Hilliges.  Human performance capture from monocular video in the wild.  In _2021 International Conference on 3D Vision (3DV)_ , pages 889–898. IEEE, 2021.
  * Guo et al. [2023] Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges.  Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Habermann et al. [2020] Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt.  Deepcap: Monocular human performance capture using weak supervision.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2020.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Huang et al. [2023] Buzhen Huang, Jingyi Ju, Zhihao Li, and Yangang Wang.  Reconstructing groups of people with hypergraph relational reasoning, 2023.
  * Huang et al. [2020] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.  Arch: Animatable reconstruction of clothed humans.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Jiakai et al. [2021] Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fuqiang, Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi.  Editable free-viewpoint video using a layered neural representation.  In _ACM SIGGRAPH_ , 2021.
  * Jiang et al. [2022a] Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang.  Selfrecon: Self reconstruction your digital avatar from monocular video.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022a.
  * Jiang et al. [2023] Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges.  Instantavatar: Learning avatars from monocular video in 60 seconds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Jiang et al. [2020] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5579–5588, 2020.
  * Jiang et al. [2022b] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan.  Neuman: Neural human radiance field from a single video.  In _Proceedings of the European conference on computer vision (ECCV)_ , 2022b.
  * Khirodkar et al. [2022] Rawal Khirodkar, Shashank Tripathi, and Kris Kitani.  Occluded human mesh recovery, 2022.
  * Kirillov et al. [2020] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9799–9808, 2020.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick.  Segment anything.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4015–4026, 2023.
  * Kocabas et al. [2021] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black.  Pare: Part attention regressor for 3d human body estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Li et al. [2023] Jiahao Li, Zongxin Yang, Xiaohan Wang, Jianxin Ma, Chang Zhou, and Yi Yang.  Jotr: 3d joint contrastive learning with transformers for occluded human mesh recovery, 2023.
  * Li et al. [2020] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang.  Self-correction for human parsing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Li et al. [2022] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan.  Cliff: Carrying location information in full frames into human pose and shape estimation.  In _ECCV_ , 2022.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  _ACM transactions on graphics (TOG)_ , 34(6):1–16, 2015.
  * Mescheder et al. [2019] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4460–4470, 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European conference on computer vision_ , pages 405–421. Springer, 2020.
  * Mustafa et al. [2021] Armin Mustafa, Akin Caliskan, Lourdes Agapito, and Adrian Hilton.  Multi-person implicit reconstruction from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14474–14483, 2021.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 10975–10985, 2019.
  * Peng et al. [2021] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9054–9063, 2021.
  * Saito et al. [2019] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2304–2314, 2019.
  * Saito et al. [2020] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 84–93, 2020.
  * Shuai et al. [2022] Qing Shuai, Chen Geng, Qi Fang, Sida Peng, Wenhao Shen, Xiaowei Zhou, and Hujun Bao.  Novel view synthesis of human interactions from sparse multi-view videos.  In _SIGGRAPH Conference Proceedings_ , 2022.
  * Su et al. [2021] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin.  A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Sun et al. [2022] Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J Black.  Putting people in their place: Monocular regression of 3d people in depth.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Sun et al. [2023] Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J. Black.  TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments.  In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Weng et al. [2022] Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman.  HumanNeRF: Free-viewpoint rendering of moving people from monocular video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16210–16220, 2022.
  * Wu et al. [2019] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Xiu et al. [2022] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black.  ICON: Implicit Clothed humans Obtained from Normals.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 13296–13306, 2022.
  * Xiu et al. [2023] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black.  ECON: Explicit Clothed humans Optimized via Normal integration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Xu et al. [2022] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  ViTPose: Simple vision transformer baselines for human pose estimation.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.
  * Ye et al. [2023] Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa.  Decoupling human and camera motion from videos in the wild.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Yin et al. [2023] Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Zarate, Jie Song, and Otmar Hilliges.  Hi4d: 4d instance segmentation of close human interaction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Zhang et al. [2021] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu.  Editable free-viewpoint video using a layered neural representation.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–18, 2021.
  * Zhang et al. [2020] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv:2010.07492_ , 2020.
  * Zhang et al. [2022] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box.  2022\.
  * Zheng et al. [2021a] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu.  Deepmulticap: Performance capture of multiple characters using sparse multiview cameras.  In _IEEE Conference on Computer Vision (ICCV 2021)_ , 2021a.
  * Zheng et al. [2021b] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai.  Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021b.


* [53] J. Yan, P. Li, and Y. Huang, “A short-term wind power scenario generation method based on conditional diffusion model,” in _2023 IEEE Sustainable Power and Energy Conference (iSPEC)_. IEEE, 2023, pp. 1–6.
  * [54] M. Seki, Y.-Z. Zhang, and S. Imoto, “Imputing time-series microbiome abundance profiles with diffusion model,” in _2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2023, pp. 914–919.
  * [55] D. Scassola, S. Saccani, G. Carbone, and L. Bortolussi, “Conditioning score-based generative models by neuro-symbolic constraints,” _arXiv preprint arXiv:2308.16534_ , 2023.
  * [56] Y. Tashiro, J. Song, Y. Song, and S. Ermon, “Csdi: Conditional score-based diffusion models for probabilistic time series imputation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 24 804–24 816, 2021.
  * [57] H. Li, G. Ditzler, J. Roveda, and A. Li, “Descod-ecg: Deep score-based diffusion model for ecg baseline wander and noise removal,” _IEEE Journal of Biomedical and Health Informatics_ , 2023.
  * [58] W. Zhao, Z. Chen, P. Xie, J. Liu, S. Hou, L. Xu, Y. Qiu, D. Wu, J. Xiao, and K. He, “Multi-task oriented diffusion model for mortality prediction in shock patients with incomplete data,” _Information Fusion_ , vol. 105, p. 102207, 2024.
  * [59] G. Liang, P. Tiwari, S. Nowaczyk, S. Byttner, and F. Alonso-Fernandez, “Dynamic causal explanation based diffusion-variational graph neural network for spatio-temporal forecasting,” _arXiv preprint arXiv:2305.09703_ , 2023.
  * [60] C. Chu, H. Zhang, P. Wang, and F. Lu, “Simulating human mobility with a trajectory generation framework based on diffusion model,” _International Journal of Geographical Information Science_ , pp. 1–32, 2024.
  * [61] M. Guimarães, C. Soares, and C. Manfletti, “Predicting the position uncertainty at the time of closest approach with diffusion models,” _arXiv preprint arXiv:2311.05417_ , 2023.
  * [62] J. Jiang, C. Han, W. X. Zhao, and J. Wang, “Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 37, no. 4, 2023, pp. 4365–4373.
  * [63] Y. Y. Choi, M. Lee, S. W. Park, S. Lee, and J. Ko, “A gated mlp architecture for learning topological dependencies in spatio-temporal graphs,” _arXiv preprint arXiv:2401.15894_ , 2024.
  * [64] J. Wiederer, A. Bouazizi, M. Troina, U. Kressel, and V. Belagiannis, “Anomaly detection in multi-agent trajectories for automated driving,” in _Conference on Robot Learning_. PMLR, 2022, pp. 1223–1233.
  * [65] Y. Lin, H. Wan, J. Hu, S. Guo, B. Yang, Y. Lin, and C. S. Jensen, “Origin-destination travel time oracle for map-based services,” _Proceedings of the ACM on Management of Data_ , vol. 1, no. 3, pp. 1–27, 2023.
  * [66] Y. Yao, Y. Liu, X. Dai, S. Chen, and Y. Lv, “A graph-based scene encoder for vehicle trajectory prediction using the diffusion model,” in _2023 International Annual Conference on Complex Systems and Intelligent Science (CSIS-IAC)_ , 2023, pp. 981–986.
  * [67] I. Kotseruba and J. K. Tsotsos, “Data limitations for modeling top-down effects on drivers’ attention,” _arXiv preprint arXiv:2404.08749_ , 2024.
  * [68] K. Ito and L. Johnson, “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [69] T. Yun, H. Jung, and J. Son, “Imputation as inpainting: Diffusion models for spatiotemporal data imputation,” 2023.
  * [70] Y. Yuan, J. Ding, C. Shao, D. Jin, and Y. Li, “Spatio-temporal diffusion point processes,” _arXiv preprint arXiv:2305.12403_ , 2023.
  * [71] T. Yan, H. Zhang, T. Zhou, Y. Zhan, and Y. Xia, “Scoregrad: Multivariate probabilistic time series forecasting with continuous energy-based generative models,” _arXiv preprint arXiv:2106.10121_ , 2021.
  * [72] S. Zhang, S. Wang, X. Tan, R. Liu, J. Zhang, and J. Wang, “sasdim: self-adaptive noise scaling diffusion model for spatial time series imputation,” _arXiv preprint arXiv:2309.01988_ , 2023.
  * [73] Y. Cheng, K. Yamashita, J. Follum, and N. Yu, “Adversarial purification for data-driven power system event classifiers with diffusion models,” _arXiv preprint arXiv:2311.07110_ , 2023.
  * [74] N. Neifar, A. Ben-Hamadou, A. Mdhaffar, and M. Jmaiel, “Diffecg: A generalized probabilistic diffusion model for ecg signals synthesis,” _arXiv preprint arXiv:2306.01875_ , 2023.
  * [75] F. Bao, C. Li, J. Sun, and J. Zhu, “Why are conditional generative models better than unconditional ones?” _arXiv preprint arXiv:2212.00362_ , 2022.
  * [76] M. Jin, Q. Wen, Y. Liang, C. Zhang, S. Xue, X. Wang, J. Zhang, Y. Wang, H. Chen, X. Li _et al._ , “Large models for time series and spatio-temporal data: A survey and outlook,” _arXiv preprint arXiv:2310.10196_ , 2023.
  * [77] M. Jin, H. Y. Koh, Q. Wen, D. Zambon, C. Alippi, G. I. Webb, I. King, and S. Pan, “A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection,” _arXiv preprint arXiv:2307.03759_ , 2023.
  * [78] K. Chen, G. Li, H. Li, Y. Wang, W. Wang, Q. Liu, and H. Wang, “Quantifying uncertainty: Air quality forecasting based on dynamic spatial-temporal denoising diffusion probabilistic model,” _Environmental Research_ , p. 118438, 2024.
  * [79] Y. Tang, H. He, Y. Wang, and Y. Wu, “Utilizing a diffusion model for pedestrian trajectory prediction in semi-open autonomous driving environments,” _IEEE Sensors Journal_ , pp. 1–1, 2024.
  * [80] T. Westny, B. Olofsson, and E. Frisk, “Diffusion-based environment-aware trajectory prediction,” _arXiv preprint arXiv:2403.11643_ , 2024.
  * [81] Y. Zhu, Y. Ye, S. Zhang, X. Zhao, and J. Yu, “Difftraj: Generating gps trajectory with diffusion probabilistic model,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [82] S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan, “Attention based spatial-temporal graph convolutional networks for traffic flow forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 922–929.
  * [83] C. Li, G. Feng, Y. Li, R. Liu, Q. Miao, and L. Chang, “Difftad: Denoising diffusion probabilistic models for vehicle trajectory anomaly detection,” _Knowledge-Based Systems_ , vol. 286, p. 111387, 2024.
  * [84] L. Li, R. Carver, I. Lopez-Gomez, F. Sha, and J. Anderson, “Seeds: Emulation of weather forecast ensembles with diffusion models,” _arXiv preprint arXiv:2306.14066_ , 2023.
  * [85] X. Ling, C. Li, F. Qin, P. Yang, and Y. Huang, “Srndiff: Short-term rainfall nowcasting with condition diffusion model,” _arXiv preprint arXiv:2402.13737_ , 2024.
  * [86] Y. Li, X. Lu, Y. Wang, and D. Dou, “Generative time series forecasting with diffusion, denoise, and disentanglement,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 009–23 022, 2022.
  * [87] OpenAI Sora, “Video generation models as world simulators,” https://openai.com/research/video-generation-models-as-world-simulators, accessed: 2024-04-29.
  * [88] L. Lin, D. Shi, A. Han, and J. Gao, “Specstg: A fast spectral diffusion framework for probabilistic spatio-temporal traffic forecasting,” _arXiv preprint arXiv:2401.08119_ , 2024.
  * [89] S. Kim, Y.-E. Lee, S.-H. Lee, and S.-W. Lee, “Diff-e: Diffusion-based learning for decoding imagined speech eeg,” _arXiv preprint arXiv:2307.14389_ , 2023.
  * [90] H. Wang, “Diffuseroll: Multi-track multi-category music generation based on diffusion model,” _arXiv preprint arXiv:2303.07794_ , 2023.
  * [91] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” _arXiv preprint arXiv:2009.09761_ , 2020.
  * [92] J. Zhang, G. Fazekas, and C. Saitis, “Fast diffusion gan model for symbolic music generation controlled by emotions,” _arXiv preprint arXiv:2310.14040_ , 2023.
  * [93] J. Leinonen, U. Hamann, D. Nerini, U. Germann, and G. Franch, “Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification,” _arXiv preprint arXiv:2304.12891_ , 2023.
  * [94] R. Kurinchi-Vendhan, “Wirediff: a wind resolution-enhancing diffusion model.”
  * [95] Z. Li, A. Sun, and C. Li, “Diffurec: A diffusion model for sequential recommendation,” _arXiv preprint arXiv:2304.00686_ , 2023.
  * [96] S. Li, H. Xiong, and Y. Chen, “Diffcharge: Generating ev charging scenarios via a denoising diffusion model,” _IEEE Transactions on Smart Grid_ , 2024.
  * [97] C. Li, G. Feng, Y. Li, R. Liu, Q. Miao, and L. Chang, “Difftad: Denoising diffusion probabilistic models for vehicle trajectory anomaly detection,” _Knowledge-Based Systems_ , vol. 286, p. 111387, 2024.
  * [98] X. Han, H. Zheng, and M. Zhou, “Card: Classification and regression diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 18 100–18 115, 2022.
  * [99] C. Yang, T. Wang, and X. Yan, “Ddmt: Denoising diffusion mask transformer models for multivariate time series anomaly detection,” _arXiv preprint arXiv:2310.08800_ , 2023.
  * [100] Y. Duan, J. Zhou, Z. Wang, Y.-C. Chang, Y.-K. Wang, and C.-T. Lin, “Domain-specific denoising diffusion probabilistic models for brain dynamics,” _arXiv preprint arXiv:2305.04200_ , 2023.
  * [101] D. Solis-Martin, J. Galan-Paez, and J. Borrego-Diaz, “D3a-ts: Denoising-driven data augmentation in time series,” _arXiv preprint arXiv:2312.05550_ , 2023.
  * [102] Y. Dai, C. Yang, K. Liu, A. Liu, and Y. Liu, “Timeddpm: Time series augmentation strategy for industrial soft sensing,” _IEEE Sensors Journal_ , 2023.
  * [103] H. Yi, L. Hou, Y. Jin, and N. A. Saeed, “Time series diffusion method: A denoising diffusion probabilistic model for vibration signal generation,” _arXiv preprint arXiv:2312.07981_ , 2023.
  * [104] M. Biloš, K. Rasul, A. Schneider, Y. Nevmyvaka, and S. Günnemann, “Modeling temporal data as continuous functions with process diffusion,” _arXiv preprint arXiv:2211.02590_ , 2022.
  * [105] J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, “Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [106] J. Crabbé, N. Huynh, J. Stanczuk, and M. van der Schaar, “Time series diffusion in the frequency domain,” _arXiv preprint arXiv:2402.05933_ , 2024.
  * [107] X. Dong, Z. Mao, Y. Sun, and X. Xu, “Short-term wind power scenario generation based on conditional latent diffusion models,” _IEEE Transactions on Sustainable Energy_ , 2023.
  * [108] Z. Zhang, H. Yang, J. Chen, and Z. Yin, “Multi-scale conditional diffusion model for deposited droplet volume measurement in inkjet printing manufacturing,” _Journal of Manufacturing Systems_ , vol. 71, pp. 595–608, 2023.
  * [109] S. Kim, S.-H. Lee, Y.-E. Lee, J.-W. Lee, J.-H. Park, and S.-W. Lee, “Brain-driven representation learning based on diffusion model,” _arXiv preprint arXiv:2311.07925_ , 2023.
  * [110] G. Sharma, A. Dhall, and R. Subramanian, “Medic: Mitigating eeg data scarcity via class-conditioned diffusion model,” in _Deep Generative Models for Health Workshop NeurIPS 2023_ , 2023.
  * [111] M. Liu, H. Huang, H. Feng, L. Sun, B. Du, and Y. Fu, “Pristi: A conditional diffusion framework for spatiotemporal imputation,” _arXiv preprint arXiv:2302.09746_ , 2023.
  * [112] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [113] Y. Ouyang, L. Xie, C. Li, and G. Cheng, “Missdiff: Training diffusion models on tabular data with missing values,” in _ICML 2023 Workshop on Structured Probabilistic Inference $\\{$$\backslash$&$\\}$ Generative Modeling_, 2023.
  * [114] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [115] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” _ACM Computing Surveys_ , vol. 56, no. 4, pp. 1–39, 2023.
  * [116] H. Lim, M. Kim, S. Park, and N. Park, “Regular time-series generation using sgm,” _arXiv preprint arXiv:2301.08518_ , 2023.
  * [117] A. Coletta, S. Gopalakrishnan, D. Borrajo, and S. Vyetrenko, “On the constrained time-series generation problem,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [118] W. Liu, J.-L. Qiu, W.-L. Zheng, and B.-L. Lu, “Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 14, no. 2, pp. 715–729, 2021.
  * [119] G. Tosato, C. M. Dalbagno, and F. Fumagalli, “Eeg synthetic data generation using probabilistic diffusion models,” _arXiv preprint arXiv:2303.06068_ , 2023.
  * [120] P. Laguna, R. G. Mark, A. Goldberg, and G. B. Moody, “A database for evaluation of algorithms for measurement of qt and other waveform intervals in the ecg,” in _Computers in cardiology 1997_. IEEE, 1997, pp. 673–676.
  * [121] G. B. Moody and R. G. Mark, “The mit-bih arrhythmia database on cd-rom and software for use with it,” in _[1990] Proceedings Computers in Cardiology_. IEEE, 1990, pp. 185–188.
  * [122] R. Bousseljot and D. Kreiseler, “Waveform recognition with 10,000 ecgs,” in _Computers in Cardiology 2000. Vol. 27 (Cat. 00CH37163)_. IEEE, 2000, pp. 331–334.
  * [123] J. M. L. Alcaraz and N. Strodthoff, “Diffusion-based conditional ecg generation with structured state space models,” _Computers in Biology and Medicine_ , p. 107115, 2023.
  * [124] I. Sanz _et al._ , “Applying generative models and transfer learning to physiological data classification,” in _Artificial Intelligence Research and Development: Proceedings of the 25th International Conference of the Catalan Association for Artificial Intelligence_ , vol. 375. IOS Press, 2023, p. 28.
  * [125] G. B. Moody and R. G. Mark, “The impact of the mit-bih arrhythmia database,” _IEEE engineering in medicine and biology magazine_ , vol. 20, no. 3, pp. 45–50, 2001.
  * [126] A. Miltiadous, K. D. Tzimourta, T. Afrantou, P. Ioannidis, N. Grigoriadis, D. G. Tsalikakis, P. Angelidis, M. G. Tsipouras, E. Glavas, N. Giannakeas _et al._ , “A dataset of scalp eeg recordings of alzheimer’s disease, frontotemporal dementia and healthy subjects from routine eeg,” _Data_ , vol. 8, no. 6, p. 95, 2023.
  * [127] B. Kemp, A. H. Zwinderman, B. Tuk, H. A. Kamphuisen, and J. J. Oberye, “Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg,” _IEEE Transactions on Biomedical Engineering_ , vol. 47, no. 9, pp. 1185–1194, 2000.
  * [128] B. Aristimunha, R. Y. de Camargo, S. Chevallier, O. Lucena, A. G. Thomas, M. J. Cardoso, W. H. L. Pinaya, and J. Dafflon, “Synthetic sleep eeg signal generation using latent diffusion models,” in _Deep Generative Models for Health Workshop NeurIPS 2023_ , 2023.
  * [129] G.-Q. Zhang, L. Cui, R. Mueller, S. Tao, M. Kim, M. Rueschman, S. Mariani, D. Mobley, and S. Redline, “The national sleep research resource: towards a sleep data commons,” _Journal of the American Medical Informatics Association_ , vol. 25, no. 10, pp. 1351–1358, 2018.
  * [130] G. Bénédict, O. Jeunen, S. Papa, S. Bhargav, D. Odijk, and M. de Rijke, “Recfusion: A binomial diffusion process for 1d data for recommendation,” _arXiv preprint arXiv:2306.08947_ , 2023.
  * [131] X. Lin, X. Chen, C. Wang, H. Shu, L. Song, B. Li _et al._ , “Discrete conditional diffusion for reranking in recommendation,” _arXiv preprint arXiv:2308.06982_ , 2023.
  * [132] W. Wang, Y. Xu, F. Feng, X. Lin, X. He, and T.-S. Chua, “Diffusion recommender model,” _arXiv preprint arXiv:2304.04971_ , 2023.
  * [133] F. M. Harper and J. A. Konstan, “The movielens datasets: History and context,” _Acm transactions on interactive intelligent systems (tiis)_ , vol. 5, no. 4, pp. 1–19, 2015.
  * [134] H. Du, H. Yuan, Z. Huang, P. Zhao, and X. Zhou, “Sequential recommendation with diffusion models,” _arXiv preprint arXiv:2304.04541_ , 2023.
  * [135] H. Ma, R. Xie, L. Meng, X. Chen, X. Zhang, L. Lin, and Z. Kang, “Plug-in diffusion model for sequential recommendation,” _arXiv preprint arXiv:2401.02913_ , 2024.
  * [136] H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers _et al._ , “The era5 global reanalysis,” _Quarterly Journal of the Royal Meteorological Society_ , vol. 146, no. 730, pp. 1999–2049, 2020.
  * [137] L. Chuyao, X. Li, and Y. Ye, “Pfst-lstm: A spatiotemporal lstm model with pseudoflow prediction for precipitation nowcasting,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. PP, no. 99, pp. 1–1, 2020.
  * [138] P. Sepulchre, A. Caubel, J.-B. Ladant, L. Bopp, O. Boucher, P. Braconnot, P. Brockmann, A. Cozic, Y. Donnadieu, J.-L. Dufresne _et al._ , “Ipsl-cm5a2–an earth system model designed for multi-millennial climate simulations,” _Geoscientific Model Development_ , vol. 13, no. 7, pp. 3011–3053, 2020.
  * [139] T. Hong, P. Pinson, S. Fan, H. Zareipour, A. Troccoli, and R. J. Hyndman, “Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond,” pp. 896–913, 2016.
  * [140] Z. J. Lee, T. Li, and S. H. Low, “Acn-data: Analysis and applications of an open ev charging dataset,” in _Proceedings of the tenth ACM international conference on future energy systems_ , 2019, pp. 139–149.
  * [141] J. R. Schofield, R. Carmichael, S. Tindemans, M. Bilton, M. Woolf, G. Strbac _et al._ , “Low carbon london project: Data from the dynamic time-of-use electricity pricing trial, 2013,” _uK Data Service, SN_ , vol. 7857, no. 2015, pp. 7857–7851, 2015.
  * [142] Z. Wang and H. Zhang, “Customized load profiles synthesis for electricity customers based on conditional diffusion models,” _arXiv preprint arXiv:2304.12076_ , 2023.
  * [143] S. Li, H. Xiong, and Y. Chen, “Diffplf: A conditional diffusion model for probabilistic forecasting of ev charging load,” _arXiv preprint arXiv:2402.13548_ , 2024.
  * [144] A. Jenkins, Z. Chen, F. S. Ng, and D. Mandic, “Improving diffusion models for ecg imputation with an augmented template prior,” _arXiv preprint arXiv:2310.15742_ , 2023.
  * [145] L. Liu, Y. Ren, Z. Lin, and Z. Zhao, “Pseudo numerical methods for diffusion models on manifolds,” _arXiv preprint arXiv:2202.09778_ , 2022.
  * [146] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 5775–5787, 2022.
  * [147] X. Mao, “The truncated euler–maruyama method for stochastic differential equations,” _Journal of Computational and Applied Mathematics_ , vol. 290, pp. 370–384, 2015.
  * [148] P. Vincent, “A connection between score matching and denoising autoencoders,” _Neural computation_ , vol. 23, no. 7, pp. 1661–1674, 2011.
  * [149] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar _et al._ , “Voicebox: Text-guided multilingual universal speech generation at scale,” _Advances in neural information processing systems_ , vol. 36, 2024.
  * [150] D. Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski, “Diffusion self-guidance for controllable image generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [151] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [152] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [153] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” _arXiv preprint arXiv:2112.10741_ , 2021.
  * [154] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14 297–14 306.
  * [155] M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. N. Suganthan, “Global context with discrete diffusion in vector quantised modelling for image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 502–11 511.
  * [156] J. Wolleb, F. Bieder, R. Sandkühler, and P. C. Cattin, “Diffusion models for medical anomaly detection,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2022, pp. 35–45.
  * [157] S. Chen, P. Sun, Y. Song, and P. Luo, “Diffusiondet: Diffusion model for object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 19 830–19 843.
  * [158] D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko, “Label-efficient semantic segmentation with diffusion models,” _arXiv preprint arXiv:2112.03126_ , 2021.
  * [159] C.-H. Chao, W.-F. Sun, B.-W. Cheng, and C.-Y. Lee, “Quasi-conservative score-based generative models,” _arXiv preprint arXiv:2209.12753_ , 2022.
  * [160] H. Chung, B. Sim, and J. C. Ye, “Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 413–12 422.
  * [161] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, “Ilvr: Conditioning method for denoising diffusion probabilistic models,” _arXiv preprint arXiv:2108.02938_ , 2021.
  * [162] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein, “Universal guidance for diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 843–852.
  * [163] Y. Yuan, J. Song, U. Iqbal, A. Vahdat, and J. Kautz, “Physdiff: Physics-guided human motion diffusion model,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 16 010–16 021.
  * [164] M. Kollovieh, A. F. Ansari, M. Bohlke-Schneider, J. Zschiegner, H. Wang, and Y. B. Wang, “Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [165] S. An, H. Lee, J. Jo, S. Lee, and S. J. Hwang, “Diffusionnag: Task-guided neural architecture generation with diffusion models,” _arXiv preprint arXiv:2305.16943_ , 2023.
  * [166] N. Torenvliet and J. Zelek, “Evaluating diffusion models for the automation of ultrasonic nondestructive evaluation data analysis,” _Algorithms_ , vol. 17, no. 4, p. 167, 2024.
  * [167] Z. Yang, J. Wu, Z. Wang, X. Wang, Y. Yuan, and X. He, “Generate what you prefer: Reshaping sequential recommendation via guided diffusion,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [168] P. Shao, J. Feng, J. Lu, P. Zhang, and C. Zou, “Data-driven and knowledge-guided denoising diffusion model for flood forecasting,” _Expert Systems with Applications_ , vol. 244, p. 122908, 2024.
  * [169] S. Zuo, V. F. Rey, S. Suh, S. Sigg, and P. Lukowicz, “Unsupervised statistical feature-guided diffusion model for sensor-based human activity recognition,” _arXiv preprint arXiv:2306.05285_ , 2023.
  * [170] J. Hu, X. Liu, Z. Fan, Y. Liang, and R. Zimmermann, “Towards unifying diffusion models for probabilistic spatio-temporal graph learning,” _arXiv preprint arXiv:2310.17360_ , 2023.
  * [171] L. Yang, Z. Zhang, W. Zhang, and S. Hong, “Score-based graph generative modeling with self-guided latent diffusion,” 2022.
  * [172] S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong, “Diffuseq: Sequence to sequence text generation with diffusion models,” _arXiv preprint arXiv:2210.08933_ , 2022.
  * [173] A. Schneuing, Y. Du, C. Harris, A. Jamasb, I. Igashov, W. Du, T. Blundell, P. Lió, C. Gomes, M. Welling _et al._ , “Structure-based drug design with equivariant diffusion models,” _arXiv preprint arXiv:2210.13695_ , 2022.
  * [174] B. D. Anderson, “Reverse-time diffusion equation models,” _Stochastic Processes and their Applications_ , vol. 12, no. 3, pp. 313–326, 1982.
  * [175] Y.-J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao, “Conditional diffusion probabilistic model for speech enhancement,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7402–7406.
  * [176] W. Tai, F. Zhou, G. Trajcevski, and T. Zhong, “Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 11, 2023, pp. 13 627–13 635.
  * [177] W. Tai, Y. Lei, F. Zhou, G. Trajcevski, and T. Zhong, “Dose: Diffusion dropout with adaptive prior for speech enhancement,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [178] B. Lay, S. Welker, J. Richter, and T. Gerkmann, “Reducing the prior mismatch of stochastic differential equations for diffusion-based speech enhancement,” _arXiv preprint arXiv:2302.14748_ , 2023.
  * [179] Z. Qiu, Y. Guo, M. Fu, H. Huang, Y. Hu, L. He, and F. Sun, “Cra-diffuse: Improved cross-domain speech enhancement based on diffusion model with tf domain pre-denoising,” in _2023 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2023, pp. 1709–1714.
  * [180] J. Serrà, S. Pascual, J. Pons, R. O. Araz, and D. Scaini, “Universal speech enhancement with score-based diffusion,” _arXiv preprint arXiv:2206.03065_ , 2022.
  * [181] H. Chen, X. Dong, Y. Kong, Z. Chen, S. Zheng, X. Hu, and X. Zhao, “Online prediction of mechanical and electrical quality in ultrasonic metal welding using time series generation and deep learning,” _Engineering Failure Analysis_ , p. 108162, 2024.
  * [182] S. S. Narasimhan, S. Agarwal, O. Akcin, S. Sanghavi, and S. Chinchali, “Time weaver: A conditional time series generation model,” _arXiv preprint arXiv:2403.02682_ , 2024.
  * [183] H. Shirzad, R. Deng, H. Zhao, and F. Tung, “Conditional diffusion models as self-supervised learning backbone for irregular time series,” in _ICLR 2024 Workshop on Learning from Time Series For Health_ , 2024.
  * [184] C. Liu, S. He, H. Liu, and J. Chen, “Intention-aware denoising diffusion model for trajectory prediction,” _arXiv preprint arXiv:2403.09190_ , 2024.
  * [185] Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Long-form music generation with latent diffusion,” _arXiv preprint arXiv:2404.10301_ , 2024.
  * [186] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [187] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [188] D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in _International conference on machine learning_. PMLR, 2015, pp. 1530–1538.
  * [189] Z. Chang, G. A. Koulieris, and H. P. Shum, “On the design fundamentals of diffusion models: A survey,” _arXiv preprint arXiv:2306.04542_ , 2023.
  * [190] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 8633–8646, 2022.
  * [191] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH 2022 conference proceedings_ , 2022, pp. 1–10.
  * [192] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 4, pp. 4713–4726, 2022.
  * [193] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg, “Structured denoising diffusion models in discrete state-spaces,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 981–17 993, 2021.
  * [194] J. Lovelace, V. Kishore, C. Wan, E. Shekhtman, and K. Q. Weinberger, “Latent diffusion for language generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [195] X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto, “Diffusion-lm improves controllable text generation,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 4328–4343, 2022.
  * [196] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , vol. 1, no. 2, p. 3, 2022.
  * [197] D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y. Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound generation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [198] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu, “Motiondiffuse: Text-driven human motion generation with diffusion model,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [199] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3836–3847.
  * [200] T. Amit, T. Shaharbany, E. Nachmani, and L. Wolf, “Segdiff: Image segmentation with diffusion probabilistic models,” _arXiv preprint arXiv:2112.00390_ , 2021.
  * [201] J. Wolleb, R. Sandkühler, F. Bieder, P. Valmaggia, and P. C. Cattin, “Diffusion models for implicit image segmentation ensembles,” in _International Conference on Medical Imaging with Deep Learning_. PMLR, 2022, pp. 1336–1348.
  * [202] G. Nam, M. Khlifi, A. Rodriguez, A. Tono, L. Zhou, and P. Guerrero, “3d-ldm: Neural implicit 3d shape generation with latent diffusion models,” _arXiv preprint arXiv:2212.00842_ , 2022.
  * [203] A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, K. Kreis _et al._ , “Lion: Latent point diffusion models for 3d shape generation,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 10 021–10 039, 2022.
  * [204] M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang, “Geodiff: A geometric diffusion model for molecular conformation generation,” _arXiv preprint arXiv:2203.02923_ , 2022.
  * [205] Z. Guo, J. Liu, Y. Wang, M. Chen, D. Wang, D. Xu, and J. Cheng, “Diffusion models in bioinformatics and computational biology,” _Nature Reviews Bioengineering_ , pp. 1–19, 2023.
  * [206] R. Huang, M. W. Lam, J. Wang, D. Su, D. Yu, Y. Ren, and Z. Zhao, “Fastdiff: A fast conditional diffusion model for high-quality speech synthesis,” _arXiv preprint arXiv:2204.09934_ , 2022.
  * [207] H. Harutyunyan, H. Khachatrian, D. C. Kale, G. Ver Steeg, and A. Galstyan, “Multitask learning and benchmarking with clinical time series data,” _Scientific data_ , vol. 6, no. 1, p. 96, 2019.
  * [208] G. Jin, Y. Liang, Y. Fang, Z. Shao, J. Huang, J. Zhang, and Y. Zheng, “Spatio-temporal graph neural networks for predictive learning in urban computing: A survey,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.
  * [209] M. Jin, Y. Zhang, W. Chen, K. Zhang, Y. Liang, B. Yang, J. Wang, S. Pan, and Q. Wen, “Position paper: What can large language models tell us about time series analysis,” _arXiv preprint arXiv:2402.02713_ , 2024.
  * [210] S. Khanna, P. Liu, L. Zhou, C. Meng, R. Rombach, M. Burke, D. Lobell, and S. Ermon, “Diffusionsat: A generative foundation model for satellite imagery,” _arXiv preprint arXiv:2312.03606_ , 2023.
  * [211] Y. Xu, Z. Liu, M. Tegmark, and T. Jaakkola, “Poisson flow generative models,” _ArXiv_ , vol. abs/2209.11178, 2022.
  * [212] Y. Xu, Z. Liu, Y. Tian, S. Tong, M. Tegmark, and T. Jaakkola, “Pfgm++: Unlocking the potential of physics-inspired generative models,” _ArXiv_ , vol. abs/2302.04265, 2023.
  * [213] Z. Liu, D. Luo, Y. Xu, T. Jaakkola, and M. Tegmark, “Genphys: From physical processes to generative models,” _ArXiv_ , vol. abs/2304.02637, 2023.
  * [214] A. Bansal, E. Borgnia, H.-M. Chu, J. S. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein, “Cold diffusion: Inverting arbitrary image transforms without noise,” _arXiv:2208.09392_ , 2022.
  * [215] T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” _arXiv:2206.00364_ , 2022.
  * [216] Q. Zhang, M. Tao, and Y. Chen, “gddim: Generalized denoising diffusion implicit models,” _arXiv preprint arXiv:2206.05564_ , 2022.
  * [217] Q. Zhang and Y. Chen, “Fast sampling of diffusion models with exponential integrator,” _arXiv preprint arXiv:2204.13902_ , 2022.
  * [218] A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas, “Gotta go fast when generating data with score-based models,” _arXiv preprint arXiv:2105.14080_ , 2021.
  * [219] Y. Xu, M. Deng, X. Cheng, Y. Tian, Z. Liu, and T. Jaakkola, “Restart sampling for improving generative processes,” _Advances in Neural Information Processing Systems_ , vol. 36, pp. 76 806–76 838, 2023.
  * [220] D. Berthelot, A. Autef, J. Lin, D. A. Yap, S. Zhai, S. Hu, D. Zheng, W. Talbott, and E. Gu, “Tract: Denoising diffusion models with transitive closure time-distillation,” _arXiv preprint arXiv:2303.04248_ , 2023.
  * [221] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever, “Consistency models,” _arXiv preprint arXiv:2303.01469_ , 2023.
  * [222] H. Zheng, P. He, W. Chen, and M. Zhou, “Truncated diffusion probabilistic models,” _arXiv preprint arXiv:2202.09671_ , vol. 1, no. 3.1, p. 2, 2022.
  * [223] D. Watson, J. Ho, M. Norouzi, and W. Chan, “Learning to efficiently sample from diffusion probabilistic models,” _arXiv preprint arXiv:2106.03802_ , 2021.
  * [224] Z. Lyu, X. Xu, C. Yang, D. Lin, and B. Dai, “Accelerating diffusion models via early stop of the diffusion process,” _arXiv preprint arXiv:2205.12524_ , 2022.
  * [225] K. Pandey, A. Mukherjee, P. Rai, and A. Kumar, “Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents,” _arXiv preprint arXiv:2201.00308_ , 2022.
  * [226] X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” _arXiv preprint arXiv:2209.03003_ , 2022.
  * [227] A. Vahdat, K. Kreis, and J. Kautz, “Score-based generative modeling in latent space,” _Advances in neural information processing systems_ , vol. 34, pp. 11 287–11 302, 2021.
  * [228] H. Zhang, R. Feng, Z. Yang, L. Huang, Y. Liu, Y. Zhang, Y. Shen, D. Zhao, J. Zhou, and F. Cheng, “Dimensionality-varying diffusion process,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14 307–14 316.
  * [229] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,” _arXiv preprint arXiv:2211.01095_ , 2022.
  * [230] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao, “Latent consistency models: Synthesizing high-resolution images with few-step inference,” _arXiv preprint arXiv:2310.04378_ , 2023.
  * [231] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 11 461–11 471.
  * [232] J. Zheng, M. Hu, Z. Fan, C. Wang, C. Ding, D. Tao, and T.-J. Cham, “Trajectory consistency distillation,” _arXiv preprint arXiv:2402.19159_ , 2024.
  * [233] W. Zhao, L. Bai, Y. Rao, J. Zhou, and J. Lu, “Unipc: A unified predictor-corrector framework for fast sampling of diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [234] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 696–10 706.
  * [235] D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” _Advances in neural information processing systems_ , vol. 34, pp. 21 696–21 707, 2021.
  * [236] B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising diffusion restoration models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 593–23 606, 2022.
  * [237] S. Rissanen, M. Heinonen, and A. Solin, “Generative modelling with inverse heat dissipation,” _arXiv preprint arXiv:2206.13397_ , 2022.
  * [238] R. Yang, P. Srivastava, and S. Mandt, “Diffusion probabilistic modeling for video generation,” _Entropy_ , vol. 25, no. 10, p. 1469, 2023.
  * [239] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [240] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong _et al._ , “Swin transformer v2: Scaling up capacity and resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 12 009–12 019.
  * [241] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [242] W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 4195–4205.
  * [243] N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie, “Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers,” _arXiv preprint arXiv:2401.08740_ , 2024.
  * [244] E. Hoogeboom and T. Salimans, “Blurring diffusion models,” _arXiv preprint arXiv:2209.05557_ , 2022.
  * [245] G. Daras, M. Delbracio, H. Talebi, A. G. Dimakis, and P. Milanfar, “Soft diffusion: Score matching for general corruptions,” _arXiv preprint arXiv:2209.05442_ , 2022.
  * [246] Z. Kong and W. Ping, “On fast sampling of diffusion probabilistic models,” _arXiv preprint arXiv:2106.00132_ , 2021.
  * [247] Y. Song, C. Durkan, I. Murray, and S. Ermon, “Maximum likelihood training of score-based diffusion models,” _Advances in neural information processing systems_ , vol. 34, pp. 1415–1428, 2021.
  * [248] C.-W. Huang, J. H. Lim, and A. C. Courville, “A variational perspective on diffusion-based generative models and score matching,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 22 863–22 876, 2021.
  * [249] E. Heitz, L. Belcour, and T. Chambon, “Iterative $\alpha$-(de) blending: A minimalist deterministic diffusion model,” in _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–8.
  * [250] M. S. Albergo and E. Vanden-Eijnden, “Building normalizing flows with stochastic interpolants,” _arXiv preprint arXiv:2209.15571_ , 2022.
  * [251] X. Su, J. Song, C. Meng, and S. Ermon, “Dual diffusion implicit bridges for image-to-image translation,” _arXiv preprint arXiv:2203.08382_ , 2022.
  * [252] G.-H. Liu, A. Vahdat, D.-A. Huang, E. A. Theodorou, W. Nie, and A. Anandkumar, “I2sb: Image-to-image schr$\backslash$”odinger bridge,” _arXiv preprint arXiv:2302.05872_ , 2023.
  * [253] S. Feng, C. Miao, Z. Zhang, and P. Zhao, “Latent diffusion transformer for probabilistic time series forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 11, 2024, pp. 11 979–11 987.
  * [254] Y. Liang, H. Wen, Y. Nie, Y. Jiang, M. Jin, D. Song, S. Pan, and Q. Wen, “Foundation models for time series analysis: A tutorial and survey,” _arXiv preprint arXiv:2403.14735_ , 2024.
  * [255] Y. Wang, H. Wu, J. Dong, Y. Liu, Y. Qiu, H. Zhang, J. Wang, and M. Long, “Timexer: Empowering transformers for time series forecasting with exogenous variables,” _arXiv preprint arXiv:2402.19072_ , 2024.
  * [256] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “Next-gpt: Any-to-any multimodal llm,” _arXiv preprint arXiv:2309.05519_ , 2023.
  * [257] Y. Chen, C. Zhang, M. Ma, Y. Liu, R. Ding, B. Li, S. He, S. Rajmohan, Q. Lin, and D. Zhang, “Imdiffusion: Imputed diffusion models for multivariate time series anomaly detection,” _arXiv preprint arXiv:2307.00754_ , 2023.
  * [258] V. Livernoche, V. Jain, Y. Hezaveh, and S. Ravanbakhsh, “On diffusion modeling for anomaly detection,” _arXiv preprint arXiv:2305.18593_ , 2023.
  * [259] I. Pintilie, A. Manolache, and F. Brad, “Time series anomaly detection using diffusion-based models,” in _2023 IEEE International Conference on Data Mining Workshops (ICDMW)_. IEEE, 2023, pp. 570–578.
  * [260] C. Xiao, Z. Gou, W. Tai, K. Zhang, and F. Zhou, “Imputation-based time-series anomaly detection with conditional weight-incremental diffusion models,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023, pp. 2742–2751.
  * [261] C. Wang, Z. Zhuang, Q. Qi, J. Wang, X. Wang, H. Sun, and J. Liao, “Drift doesn’t matter: Dynamic decomposition with diffusion reconstruction for unstable multivariate time series anomaly detection,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [262] X. Wang, H. Zhang, P. Wang, Y. Zhang, B. Wang, Z. Zhou, and Y. Wang, “An observed value consistent diffusion model for imputing missing values in multivariate time series,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023, pp. 2409–2418.
  * [263] X. Liu, J. Chen, J. Xie, and Y. Chang, “Generating hsr bogie vibration signals via pulse voltage-guided conditional diffusion model,” _arXiv preprint arXiv:2311.00496_ , 2023.
  * [264] L. Wang, “Multi-modality conditional diffusion model for time series forecasting of live sales volume,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 2675–2679.
  * [265] K. Lv, L. Yuan, and X. Ni, “Learning autoencoder diffusion models of pedestrian group relationships for multimodal trajectory prediction,” _IEEE Transactions on Instrumentation and Measurement_ , 2024.
  * [266] C. Fu, H. Kazmi, M. Quintana, and C. Miller, “Creating synthetic energy meter data using conditional diffusion and building metadata,” _arXiv preprint arXiv:2404.00525_ , 2024.
  * [267] T. Shimizu, “Diffusion model in causal inference with unmeasured confounders,” in _2023 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2023, pp. 683–688.
  * [268] A. Asperti, F. Merizzi, A. Paparella, G. Pedrazzi, M. Angelinelli, and S. Colamonaco, “Precipitation nowcasting with generative diffusion models,” _arXiv preprint arXiv:2308.06733_ , 2023.
  * [269] J. Pei, J. Wang, D. Shi, and P. Wang, “Improved efficient two-stage denoising diffusion power system measurement recovery against false data injection attacks and data losses,” _arXiv preprint arXiv:2312.04346_ , 2023.
  * [270] P. N. Mueller, “Attention-enhanced conditional-diffusion-based data synthesis for data augmentation in machine fault diagnosis,” _Engineering Applications of Artificial Intelligence_ , vol. 131, p. 107696, 2024.
  * [271] P. Zhao, X. Wang, Y. Zhang, Y. Li, H. Wang, and Y. Yang, “Diffusion-uda: Diffusion-based unsupervised domain adaptation for submersible fault diagnosis,” _Electronics Letters_ , vol. 60, no. 3, p. e13122, 2024.
  * [272] T. Y. Wong, M. H. Lim, W. K. Ngui, and M. S. Leong, “Denoising diffusion implicit model for bearing fault diagnosis under different working loads,” in _ITM Web of Conferences_ , vol. 63. EDP Sciences, 2024, p. 01025.
  * [273] K. Goel, A. Gu, C. Donahue, and C. Ré, “It’s raw! audio generation with state-space models,” _arXiv preprint arXiv:2202.09729_ , 2022.
  * [274] C. Donahue, J. McAuley, and M. Puckette, “Adversarial audio synthesis,” in _International Conference on Learning Representations_ , 2019.
  * [275] J. M. L. Alcaraz and N. Strodthoff, “Diffusion-based time series imputation and forecasting with structured state space models,” _arXiv preprint arXiv:2208.09399_ , 2022.
  * [276] J. Xu, F. Lyu, and P. C. Yuen, “Density-aware temporal attentive step-wise diffusion model for medical time series imputation,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023, pp. 2836–2845.
  * [277] Z. Liu, W. Pei, D. Lan, and Q. Ma, “Diffusion language-shapelets for semi-supervised time-series classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 13, 2024, pp. 14 079–14 087.
  * [278] Y. Li, “Ts-diffusion: Generating highly complex time series with diffusion models,” _arXiv preprint arXiv:2311.03303_ , 2023.
  * [279] Y. Yao, Y. Liu, X. Dai, S. Chen, and Y. Lv, “A graph-based scene encoder for vehicle trajectory prediction using the diffusion model,” in _2023 International Annual Conference on Complex Systems and Intelligent Science (CSIS-IAC)_. IEEE, 2023, pp. 981–986.
  * [280] Y. Li, M. R. Luyten, and M. van der Schaar, “Risk-sensitive diffusion: Learning the underlying distribution from noisy samples,” _arXiv preprint arXiv:2402.02081_ , 2024.
  * [281] Y. Liang, K. Ouyang, H. Yan, Y. Wang, Z. Tong, and R. Zimmermann, “Modeling trajectories with neural ordinary differential equations.” in _IJCAI_ , 2021, pp. 1498–1504.
  * [282] R. Hu, X. Yuan, Y. Qiao, B. Zhang, and P. Zhao, “Unsupervised anomaly detection for multivariate time series using diffusion model,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 9606–9610.
  * [283] X. Liu, Y. Xia, Y. Liang, J. Hu, Y. Wang, L. Bai, C. Huang, Z. Liu, B. Hooi, and R. Zimmermann, “Largest: A benchmark dataset for large-scale traffic forecasting,” in _Advances in Neural Information Processing Systems_ , 2023.
  * [284] S. Welker, J. Richter, and T. Gerkmann, “Speech enhancement with score-based generative models in the complex stft domain,” _arXiv preprint arXiv:2203.17004_ , 2022.
  * [285] S. Rühling Cachay, B. Zhao, H. Joren, and R. Yu, “Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [286] J. Sun, J. Zhang, Q. Li, X. Yi, Y. Liang, and Y. Zheng, “Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 34, no. 5, pp. 2348–2359, 2020.
  * [287] Y. Liang, K. Ouyang, J. Sun, Y. Wang, J. Zhang, Y. Zheng, D. Rosenblum, and R. Zimmermann, “Fine-grained urban flow prediction,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 1833–1845.
  * [288] W. Chen, Y. Liang, Y. Zhu, Y. Chang, K. Luo, H. Wen, L. Li, Y. Yu, Q. Wen, C. Chen _et al._ , “Deep learning for trajectory data management and mining: A survey and beyond,” _arXiv preprint arXiv:2403.14151_ , 2024.
  * [289] P. Chang, H. Li, S. F. Quan, S. Lu, S.-F. Wung, J. Roveda, and A. Li, “Tdstf: Transformer-based diffusion probabilistic model for sparse time series forecasting,” _arXiv preprint arXiv:2301.06625_ , 2023.
  * [290] Y. Zhu, J. J. Yu, X. Zhao, Q. Liu, Y. Ye, W. Chen, Z. Zhang, X. Wei, and Y. Liang, “Controltraj: Controllable trajectory generation with topology-constrained diffusion model,” _arXiv preprint arXiv:2404.15380_ , 2024.