17–720

# Geometric Control for Autonomous Landing on Asteroid Itokawa using visual
localization

Shankar Kulumani  Kuya Takami  and Taeyoung Lee Mechanical and Aerospace
Engineering, George Washington University, 800 22nd St NW, Washington, DC
20052, Tel: 202–994–8710, Email<EMAIL_ADDRESS>
###### Abstract

This paper considers the coupled orbit and attitude dynamics of a dumbbell
spacecraft around an asteroid. Geometric methods are used to derive the
coupled equations of motion, which are defined on the configuration space of
the special Euclidean group, and then a nonlinear controller is designed to
enable trajectory tracking of desired landing trajectories. Rather than
relying on sliding mode control or optimization based methods, the proposed
approach avoids the increased control utilization and computational complexity
inherent in other techniques. The nonlinear controller is used to track a
desired landing trajectory to the asteroid surface. A monocular imaging sensor
is used to provide position and attitude estimates using visual odometry to
enable relative state estimates. We demonstrate this control scheme with a
landing simulation about asteroid Itokawa.

##  1 Introduction

Small solar system bodies, such as asteroids and comets, are of significant
interest to the scientific community. These small bodies offer great insight
into the early formation of the solar system. This insight offers additional
detail into the formation of the Earth and also the probable formation of
other extrasolar planetary bodies. Of particular interest are those near-Earth
asteroids (NEA) which inhabit heliocentric orbits in the vicinity of the
Earth. These easily accessible bodies provide attractive targets to support
space industrialization, mining operations, and scientific missions. NEAs
potentially contain many materials such as those useful for propulsion,
construction, or for use in semiconductors. Also, many bodies contain highly
profitable materials, such as precious or strategic metals that can support a
new space focused market [1].

Furthermore, these asteroids are of keen interest for more practical purposes.
The recent meteor explosions in 2002 over Tagish Lake, Canada or over
Chelyabinsk, Russia in 2013 are clear evidence of the risk of asteroid impacts
on the Earth. These asteroids, which released an energy equivalent to
$5\text{\,}\mathrm{kt}$ of TNT, are estimated to strike the Earth on average
every year [2]. Larger bodies, such as the $60\text{\,}\mathrm{m}$ object that
exploded over Tunguska, Russia in 1908, release the energy equivalent to
$10\text{\,}\mathrm{Mt}$ of TNT and will occur on average every $1000$ years.
Asteroids and comets are the greatest threat to future civilizations and as a
result there is a focused effort to mitigate these risks [3]. A wide variety
of strategies, including nuclear standoff detonation, mass drivers, kinetic-
energy projectiles, and low-thrust deflection via electric propulsion or solar
sails, have been proposed to deal with the technically challenging asteroid
mitigation problem [4]. In spite of the significant interest in asteroid
deflection, and the extensive research by the community, the operation of
spacecraft in their vicinity remains a challenging problem.

While there has been significant study of interplanetary transfer
trajectories, relatively less analysis has been conducted on operations in the
vicinity of asteroids. The dynamic environment around asteroids is strongly
perturbed and challenging for analysis and mission operations [5, 6]. Due to
their low mass, which results in a low gravitational attraction, asteroids may
have irregular shapes and potentially chaotic spin states. As a result,
typical approaches of assuming a inverse square gravitational model are at
best inaccurate and at worst do not capture the true dynamic environment. In
addition, the vast majority of asteroid are difficult to track or measure
using current ground-based optical sensors. Due to their small size,
frequently less than $1\text{\,}\mathrm{km}$, and low albedo the reflected
energy of these asteroids is insufficient for reliable detection or tracking.
Therefore, the dynamic model of the asteroid is relatively coarse prior to
arrival of a dedicated spacecraft in the vicinity. As a result, any spacecraft
mission to an asteroid is dependent on a robust dynamic simulation and must
incorporate the ability to deal with uncertain forces and environments.

Furthermore, since the magnitude of the gravitational attraction is relatively
small, non-gravitational effects, such as solar radiation pressure or third-
body effects, become much more significant. As a result, the orbital
environment is generally quite complex and it is difficult to generate
analytical insights. One key consideration is the coupling between rotational
and translational states around the asteroid. The coupling is induced due to
the different gravitational forces experienced on various parts of the
spacecraft. The effect of the gravitational coupling is related to the
parameter $\epsilon=\frac{r}{R_{c}}$, where $r$ is the characteristic
spacecraft length and $R_{c}$ is the orbital radius [7]. For Earth based
missions, the orbital radius is several orders of magnitude larger than the
spacecraft length and $\epsilon$ is small. As a result, the corresponding
gravitational moment is weak and can be neglected. Therefore, the
translational and rotational equations of motion become decoupled and can be
considered separately, significantly simplifying the analysis. However, for
operations around an asteroid the orbital radius is much smaller, which leads
to much larger values of $\epsilon$ and much larger influence of the
rotational and translational coupling. References 8 and 9 investigated the
coupling of an elastic dumbbell spacecraft in orbit about a central body, but
only considered the case of a spherically symmetric central body. Furthermore,
the spacecraft model is assumed to remain in a planar orbit. As result, these
developments are not directly applicable to motion about an asteroid, which
experience highly non-keplerian dynamics.

An additional layer of complexity is the design of landing trajectories on
asteroids. Beginning with the first landing of NEAR Shoemaker on asteroid 433
Eros, there has been a concerted effort to develop techniques and
methodologies for asteroid landing [10, 11]. There is already considerable
knowledge on the planetary landing problem [12, 13, 14]. While conceptually
similar, the landing of spacecraft on small bodies requires additional
consideration. The surface of an asteroid is highly irregular and, as
discussed previously, there is a large coupling between the translational and
rotational dynamics of the vehicle, which is further exaggerated when close to
the surface. References 15, 16, 17 consider the soft landing problem on an
asteroid. These approaches were primarily based on nonlinear control
techniques which allowed for the development of closed loop controllers which
enable landing. However, only the translational dynamics of the body was
considered and no notion of the attitude dynamics or it’s coupling to the
position is considered. Furthermore, relatively simple gravitational models
are used which make the results unsuitable for operations near irregular
bodies.

In this paper, we develop a landing scheme for spacecraft on an asteroid. The
main objective is to construct the coupled equations of motion of a rigid
spacecraft about an asteroid. This accurate dynamic model is then used to
derive a nonlinear controller for the tracking of a landing trajectory. In
contrast to much of the previous work, we explicitly consider the
gravitational coupling between the orbit and attitude dynamics. In addition,
we utilize a polyhedron potential model to represent the shape of the
asteroid, which results in an exact closed form expression of the
gravitational potential field [18, 19]. This type of potential model is exact
given the accuracy of the shape model and valid at all point outside of the
body. As a result, the polyhedron model is ideal for all phases of spacecraft
operations, from arrival to landing.

Determination of the state of the spacecraft is typically dependent on ground
based observations with sporadic state updates. An additional capability is
possible by using on-board sensors to estimate the state of the spacecraft
relative to the asteroid [20, 21]. We seek to incorporate localization using
monocular images into the geometric nonlinear controller. This will offer the
ability to increase the accuracy and frequency of state estimates and allow
for closed loop control for landing operations. The use of imagery also
enables the use of computer vision algorithms and tools to estimate the shape
and motion of the asteroid. The combination of visual localization and
geometric control is a new avenue for the operation of spacecraft near
asteroids.

In short, this paper presents a nonlinear controller for the coupled motion of
a spacecraft around an asteroid. The dynamics are developed on the nonlinear
manifold of rigid body motions, namely the special euclidean group. This
intrinsic geometric formulation accurately captures the coupling between the
orbit and attitude dynamics. Due to the relative size of the spacecraft as
compared to the orbital radius, there is a significant gravitational moment on
the spacecraft. Through the use of the polyhedron gravitational model we
ensure an accurate representation of the gravitational moment on the
spacecraft throughout all phases of flight. Furthermore, we present a
nonlinear controller developed on the special euclidean group which
asymptotically tracks a desired landing trajectory.

##  2 Mathematical Formulation

In this paper, we consider the landing of a dumbbell model of a spacecraft
onto an asteroid. The dumbbell spacecraft consists of two masses connected by
a massless rod and is a well-known representation of a multi body spacecraft.
Furthermore, the dumbbell model captures the important interactions of the
coupling between orbital and attitude dynamics. As a result, this simple model
is useful to capture the main characteristics of a wide variety of spacecraft
configurations. Typically, spacecraft have mass concentrated in a central
structure, referred to as the bus, which houses the command and control
system, actuators, fuel, sensors etc. In addition, comparatively light-weight
solar panels extend from the bus to provide electrical energy from solar
radiation. As a result, the distributed mass of the spacecraft is captured
with the dumbbell representation. In this section, we briefly review the
polyhedron potential model and then present the derivation of the coupled
dynamics of a dumbbell spacecraft about an asteroid.

###  2.1 Polyhedron Potential Model

An accurate gravitational potential model is necessary for the operation of
spacecraft about asteroids. Additionally, a detailed shape model of the
asteroid is needed for trajectories passing close to the body. The classic
approach is to expand the gravitational potential into a harmonic series and
compute the series coefficients. However, the harmonic expansion is always an
approximation as a result of the infinite order series used in the
representation. Additionally, the harmonic model used outside of the
circumscribing sphere is not guaranteed to converge inside the sphere, which
makes it unsuitable for trajectories near the surface.

We represent the gravitational potential of the asteroid using a polyhedron
gravitation model. This model is composed of a polyhedron, which is a three-
dimensional solid body, that is defined by a series of vectors in the body-
fixed frame. The vectors define vertices in the body-fixed frame as well as
planar faces which compose the surface of the asteroid. We assume that each
face is a triangle composed of three vertices and three edges. As a result,
only two faces meet at each edge while three faces meet at each vertex. Only
the body-fixed vectors, and their associated topology, is required to define
the exterior gravitational model. References 18 and 19 give a detailed
derivation of the polyhedron model. Here, we summarize the key developments
and equations required for implementation.

Consider three vectors $\bm{v}_{1},\bm{v}_{2},\bm{v}_{3}\in\mathbb{R}^{3\times
1}$, assumed to be ordered in a counterclockwise direction about an outward
facing normal vector, which define a face. It is easy to define the three
edges of each face as

$\displaystyle\bm{e}_{i+1,i}=\bm{v}_{i+1}-\bm{v}_{i}\in\mathbb{R}^{3\times
1},$ (1)

where the index $i\in\left(1,2,3\right)$ is used to permute all edges of each
face. Since each edge is a member of two faces, there exist two edges which
are defined in opposite directions between the same vertices. We can also
define the outward normal vector to face $f$ as

$\displaystyle\hat{\bm{n}}_{f}$
$\displaystyle=\left(\bm{v}_{2}-\bm{v}_{1}\right)\times\left(\bm{v}_{3}-\bm{v}_{2}\right)\in\mathbb{R}^{3\times
1},$ (2)

and the outward facing normal vector to each edge as

$\displaystyle\hat{\bm{n}}_{i+1,i}^{f}$
$\displaystyle=\left(\bm{v}_{i+1}-\bm{v}_{i}\right)\times\hat{\bm{n}}_{f}\in\mathbb{R}^{3\times
1}.$ (3)

For each face we define the face dyad $\bm{F}_{f}$ as

$\displaystyle\bm{F}_{f}$
$\displaystyle=\hat{\bm{n}}_{f}\hat{\bm{n}}_{f}\in\mathbb{R}^{3\times 3}.$ (4)

Each edge is a member of two faces and has an outward pointing edge normal
vector, given in Equation 3, perpendicular to both the edge and the face
normal. For the edge connecting the vectors $\bm{v}_{1}$ and $\bm{v}_{2}$,
which are shared between the faces $A$ and $B$, the per edge dyad is given by

$\displaystyle\bm{E}_{12}=\hat{\bm{n}}_{A}\hat{\bm{n}}_{12}^{A}+\hat{\bm{n}}_{B}\hat{\bm{n}}_{21}^{B}\in\mathbb{R}^{3\times
3}.$ (5)

The edge dyad $\bm{E}_{e}$, is defined for each edge and is a function of the
two adjacent faces meeting at that edge. The face dyad $\bm{F}_{f}$, is
defined for each face and is a function of the face normal vectors.

Let $\bm{r}_{i}\in\mathbb{R}^{3\times 1}$ be the vector from the spacecraft to
the vertex $\bm{v}_{i}$ and it’s length is given by
$r_{i}=\left\|\bm{r}_{i}\right\|\in\mathbb{R}^{1}$. The per-edge factor
$L_{e}\in\mathbb{R}^{1}$, for the edge connecting vertices $\bm{v}_{i}$ and
$\bm{v}_{j}$, with a constant length
$e_{ij}=\left\|\bm{e}_{ij}\right\|\in\mathbb{R}^{1}$ is

$\displaystyle L_{e}$
$\displaystyle=\ln\frac{r_{i}+r_{j}+e_{ij}}{r_{i}+r_{j}-e_{ij}}.$ (6)

For the face defined by the vertices $\bm{v}_{i},\bm{v}_{j},\bm{v}_{k}$ the
per-face factor $\omega_{f}\in\mathbb{R}^{1}$ is

$\displaystyle\omega_{f}$
$\displaystyle=2\arctan\frac{\bm{r}_{i}\cdot\bm{r}_{j}\times\bm{r}_{k}}{r_{i}r_{j}r_{k}+r_{i}\left(\bm{r}_{j}\cdot\bm{r}_{k}\right)+r_{j}\left(\bm{r}_{k}\cdot\bm{r}_{i}\right)+r_{k}\left(\bm{r}_{i}\cdot\bm{r}_{j}\right)}.$
(7)

The gravitational potential due to a constant density polyhedron is given as

$\displaystyle U(\bm{r})$
$\displaystyle=\frac{1}{2}G\sigma\sum_{e\in\text{edges}}\bm{r}_{e}\cdot\bm{E}_{e}\cdot\bm{r}_{e}\cdot
L_{e}-\frac{1}{2}G\sigma\sum_{f\in\text{faces}}\bm{r}_{f}\cdot\bm{F}_{f}\cdot\bm{r}_{f}\cdot\omega_{f}\in\mathbb{R}^{1},$
(8)

where $\bm{r}_{e}$ and $\bm{r}_{f}$ are the vectors from the spacecraft to any
point on the respective edge or face, $G$ is the universal gravitational
constant, and $\sigma$ is the constant density of the asteroid. Furthermore we
can use these definitions to define the attraction, gravity gradient matrix,
and Laplacian as

$\displaystyle\nabla U(\bm{r})$
$\displaystyle=-G\sigma\sum_{e\in\text{edges}}\bm{E}_{e}\cdot\bm{r}_{e}\cdot
L_{e}+G\sigma\sum_{f\in\text{faces}}\bm{F}_{f}\cdot\bm{r}_{f}\cdot\omega_{f}\in\mathbb{R}^{3\times
1},$ (9) $\displaystyle\nabla\nabla U(\bm{r})$
$\displaystyle=G\sigma\sum_{e\in\text{edges}}\bm{E}_{e}\cdot
L_{e}-G\sigma\sum_{f\in\text{faces}}\bm{F}_{f}\cdot\omega_{f}\in\mathbb{R}^{3\times
3},$ (10) $\displaystyle\nabla^{2}U$
$\displaystyle=-G\sigma\sum_{f\in\text{faces}}\omega_{f}\in\mathbb{R}^{1}.$
(11)

One interesting thing to note is that both Equations 4 and 5 can be
precomputed without knowledge of the position of the satellite. They are both
solely functions of the vertices and edges of the polyhedral shape model and
are computed once and stored. Once a position vector $\bm{r}$ is defined, the
scalars given in Equations 6 and 7 can be computed for each face and edge.
Finally, Equation 8 is used to compute the gravitational potential on the
spacecraft. The Laplacian, defined in Equation 11, gives a simple method to
determine if the spacecraft has collided with the body [19].

###  2.2 Dumbbell Spacecraft Equations of Motion

The configuration space for rigid body motion is the semi-direct product,
$\mathsf{SE(3)}=\mathbb{R}^{3}\times\mathsf{SO(3)}$, namely the special
euclidean group. The variations should be carefully constructed such that they
respect the geometry of the configuration space. By expressing the motion of
the dumbbell directly on the special euclidean group, we avoid the issues
inherent in using other kinematic representations which fail to preserve the
geometric properties of the configuration space. The kinematics of the
dumbbell and asteroid are described in the inertial frame by

  * •

$\bm{x}\in\mathbb{R}^{3}$: the position of the center of mass of the dumbbell
spacecraft represented in the inertial frame $\bm{e}_{i}$

  * •

$R\in\mathsf{SO(3)}$: the rotation matrix which transforms vectors defined in
the spacecraft fixed frame, $\bm{b}_{i}$, to the inertial frame, $\bm{e}_{i}$

  * •

$\bm{\Omega}\in\mathbb{R}^{3}$: the angular velocity of the spacecraft body
fixed frame relative to the inertial frame and represented in the dumbbell
body fixed frame $\bm{b}_{i}$

  * •

$R_{A}\in\mathsf{SO(3)}$: the rotation matrix which transforms vectors defined
in the asteroid fixed frame, $\bm{f}_{i}$, to the inertial frame, $\bm{e}_{i}$

In this work, we assume that the asteroid is much more massive than the
spacecraft and its motion is not affected by that of the spacecraft. This
assumption allows us to treat the motion of the vehicle independently from
that of the asteroid, instead of treating the more complicated full-body
problem.

Using our kinematic variables we can define the kinetic and potential energy
of the dumbbell as

$\displaystyle T$
$\displaystyle=\frac{1}{2}m\left\|\dot{\bm{x}}\right\|^{2}+\frac{1}{2}\mathrm{tr}\negthickspace\left[S(\bm{\Omega})J_{d}S\left(\bm{\Omega}\right)^{T}\right],$
(12) $\displaystyle V(\bm{x},R)$
$\displaystyle=-m_{1}U\left(R_{A}^{T}\left(\bm{x}+R\bm{\rho}_{1}\right)\right)-m_{2}U\left(R_{A}^{T}\left(\bm{x}+R\bm{\rho}_{2}\right)\right),$
(13)

where the polyhedron potential is defined in Equation 8. The position of each
mass $m_{i}$ of the dumbbell is defined in the dumbbell fixed frame by the
vector $\bm{\rho}_{i}$. The next step is to define the variations of the
kinetic and potential energy to derive the equations of motion, which are
given as

$\displaystyle\delta V$
$\displaystyle=-\sum_{i=1}^{2}m_{i}\left(R_{A}\frac{\partial
U}{\partial\bm{z}_{i}}\right)^{T}\delta\bm{x}+m_{i}\hat{\bm{\eta}}\cdot\hat{\bm{\rho}_{1}}R^{T}R_{A}\frac{\partial
U}{\partial\bm{z}_{i}},$ (14) $\displaystyle\delta T$
$\displaystyle=\left(m_{1}+m_{2}\right)\dot{\bm{x}}^{T}\delta\dot{\bm{x}}+\frac{1}{2}\mathrm{tr}\negthickspace\left[-\dot{\hat{\bm{\eta}}}S(J\bm{\Omega})+\hat{\bm{\eta}}S(\hat{\bm{\Omega}}J\bm{\Omega})\right].$
(15)

Using the variations of the kinetic and potential energy we can derive the
equations of motion of the dumbbell spacecraft about an asteroid using
Hamilton’s principle. Hamilton’s principle then states that the variation of
the action integral

$\displaystyle\mathsf{G}=\int_{t_{0}}^{t_{f}}T(\dot{q})-V(q)dt,$ (16)

is stationary with fixed endpoints. Applying the calculus of variations and
integration by parts results in the familiar Euler-Lagrange equations of
motion. Applying the Legendre transformation allows for the same dynamics to
be expressed in an equivalent form as Hamilton’s equations [22]. The equations
of motion of a dumbbell spacecraft influenced by a polyhedron potential model
are given as

$\displaystyle\dot{\bm{x}}$ $\displaystyle=\bm{v},$ (17)
$\displaystyle\left(m_{1}+m_{2}\right)\dot{\bm{v}}$
$\displaystyle=m_{1}R_{A}\frac{\partial
U}{\partial\bm{z}_{1}}+m_{2}R_{A}\frac{\partial
U}{\partial\bm{z}_{2}}+\bm{u}_{f},$ (18) $\displaystyle\dot{R}$
$\displaystyle=RS(\bm{\Omega}),$ (19) $\displaystyle
J\dot{\bm{\Omega}}+\bm{\Omega}\times J\bm{\Omega}$
$\displaystyle=\bm{M}_{1}+\bm{M}_{2}+\bm{u}_{m}.$ (20)

The vectors $\bm{z}_{1}$ and $\bm{z}_{2}$ define the position of the dumbbell
masses as represented in the asteroid fixed frame and are defined as

$\displaystyle\bm{z}_{1}$
$\displaystyle=R_{A}^{T}\left(\bm{x}+R\bm{\rho}_{1}\right),$ (21)
$\displaystyle\bm{z}_{2}$
$\displaystyle=R_{A}^{T}\left(\bm{x}+R\bm{\rho}_{2}\right),$ (22)

where $\bm{\rho}_{i}$ defines the position of each mass in the spacecraft
fixed body frame. The gravitational moment on the dumbbell $\bm{M}_{i}$ is
defined as

$\displaystyle\bm{M}_{i}=m_{i}\left(S(R_{A}^{T}\bm{\rho}_{i})R^{T}\frac{\partial
U}{\partial\bm{z}_{i}}\right).$ (23)

The control inputs to the spacecraft are defined by $\bm{u}_{f},\bm{u}_{m}$
which define the control force represented in the inertial frame and the
control moment represented in the spacecraft frame, respectively.

###  2.3 Itokawa Shape Model and Simulated Imagery

In this work, we consider trajectories about asteroid 25413 Itokawa. Itokawa
was the target of the Hayabusa mission and detailed shape and surface maps
have been generated [23, 24, 25]. We use the estimated rotation period of
$12.1\text{\,}\mathrm{h}$ with a nominal density of
$1.9\text{\,}\mathrm{g}\text{\,}{\mathrm{cm}}^{-3}$ in the polyhedron
potential model. The shape model is composed of $786\,432$ triangular faces
and a rendering of the asteroid is provided in Figure 1. A highly detailed
model is used for the shape of the asteroid to provide a more detailed and
feature rich imaging target. However, the polyhedron potential model uses a
much coarser shape composed of $64$ faces. This greatly reduces the complexity
of the potential model without a significant difference in the qualitative
nature of the dynamic environment.

Figure 1: Blender Rendering of Asteroid 25413 Itokawa

Images of asteroid Itokawa are simulated using Blender, a free and open-source
computer graphics software program [26]. Blender is primarily used by the
animation and computer modeling fields to create computer generated images,
videos and animations for films and video games. Blender offers the capability
of accurately modeling the various effects of camera, lighting, and surface
properties of a scene using the cycles path-tracing rendering engine. A unique
feature is the ability to compile the Blender rendering software as a Python
module. This allows one to use the capabilities of Blender through a Python
API in personal scripts and functions programmatically, rather than through a
desktop based graphical interface. A camera is simulated within Blender with
the parameters shown in Table 1, which are chosen to emulate the primary
sensor of the NEAR spacecraft [27].

Parameter | Description | Value
---|---|---
FOV | horizontal field of view | $2.25\text{\,}\mathrm{\SIUnitSymbolDegree}$
| vertical field of view | $2.90\text{\,}\mathrm{\SIUnitSymbolDegree}$
Image Size | horizontal | $537\text{\,}\mathrm{px}$
| vertical | $244\text{\,}\mathrm{px}$
$f$ | focal length | $167.35\text{\,}\mathrm{mm}$
Table 1: Camera parameters used in simulation

##  3 Nonlinear Landing Controller on $\mathsf{SE(3)}$

A wide variety of control schemes have been proposed for asteroid landing
missions [16, 28]. In addition, there are a variety of controllers developed
for systems evolving on $\mathsf{SE(3)}$ [29, 30]. In this paper, we extend
their use from quadrotor aerial vehicles into the space domain. This approach
addresses many of the issues associated with the related work on asteroid
landings. The geometric control methods used to develop these nonlinear
controllers allow for the development of control systems for dynamic systems
which evolve on nonlinear manifolds. By developing the control system directly
on the nonlinear manifold, geometric control techniques provide unique
advantages as compared to those developed using local coordinate
representations. Furthermore, the geometric controller avoids the chattering
issues inherent in the previous sliding mode control approaches to asteroid
landing. In addition, rather than offering only a bounded stability guarantee,
the proposed nonlinear geometric controller guarantees almost global tracking
of the attitude and translational states. This stability guarantee is critical
for mission operations passing close to the surface over highly irregular
terrain. Furthermore, the coupled geometric controller explicitly considers
the attitude coupling of the body in contrast to many of the previous
approaches. We briefly summarize the key developments of the $\mathsf{SE(3)}$
control scheme and leave the detailed derivations to the source manuscripts
[29, 30].

In order to determine the attitude control input, we first define a desired
attitude tracking command. An arbitrary smooth attitude tracking command
$R_{d}(t)\in\mathsf{SO(3)}$ is given as a function of time. The corresponding
angular velocity command is obtained using the attitude kinematics equation,
$\hat{\Omega}_{d}=R_{d}^{T}\dot{R}_{d}$. With the desired attitude command, we
then define the errors associated with the attitude and angular velocity. The
attitude and angular velocity tracking errors must be careful chosen to remain
on the tangent bundle of $\mathsf{SO(3)}$. First, an attitude error function
is defined on $\mathsf{SO(3)}\times\mathsf{SO(3)}$ as

$\displaystyle\Psi(R,R_{d})=\frac{1}{2}\mathrm{tr}\negthickspace\left[I-R_{d}^{T}R\right].$
(24)

This positive definite function parameterizes the error between the current
attitude, $R$, and the desired attitude command $R_{d}$. Using the variations
of $\Psi$ gives the attitude tracking error vector $e_{R}\in\mathbb{R}^{3}$ as

$\displaystyle e_{R}=\frac{1}{2}\left(R_{d}^{T}R-R^{T}R_{d}^{\vee}\right).$
(25)

After further manipulation and using the attitude kinematics equation from
Equation 19, it is possible to define the angular velocity tracking error
$e_{\Omega}\in\mathbb{R}^{3}$ as

$\displaystyle e_{\Omega}=\Omega-R^{T}R_{d}\Omega_{d}.$ (26)

With the properly defined attitude error vectors the rotational control input
is defined as

$\displaystyle\bm{u}_{m}=-k_{R}e_{R}-k_{\Omega}e_{\Omega}+\Omega\times
J\Omega-J\left(\hat{\Omega}R^{T}R_{d}\Omega_{d}-R^{T}R_{d}\dot{\Omega}_{d}\right)-\bm{M}_{1}-\bm{M_{2}}$

where $k_{R},k_{\Omega}$ are positive controller constants.

The translational control input is defined in a similar manner. First we
define a smooth tracking command $x_{d}(t)\in\mathbb{R}^{3}$, which defines
the desired position of the spacecraft in the inertial frame. The tracking
error vectors are easier to define as they evolve on a Euclidean space rather
than a nonlinear manifold and are given by

$\displaystyle e_{x}=x-x_{d},$ (27) $\displaystyle e_{v}=v-\dot{x}_{d}.$ (28)

With the error variables, the translational control input is then given by

$\displaystyle\bm{u}_{f}=-k_{x}e_{x}-k_{v}e_{v}+(m_{1}+m_{2})\ddot{x}_{d}-\bm{F}_{1}-\bm{F}_{2},$
(29)

where $k_{x},k_{v}$ are positive constants. The control gains are chosen based
on the desired closed-loop system response. A variety of techniques are
available to choose these gains, but a simple linear analysis offers a
straightforward and systematic approach to choosing suitable values. We use
the control inputs defined in Equations 29 and 3 and substitute them into the
dynamic equations of motion in Equations 18 and 20. This results in the
dynamics of the error variables and the gains are chosen to ensure the error
behavior meets desired performance criteria, such as percent overshoot or
settling time [31].

###  3.1 Estimating motion from Monocular Imagery

Typically, spacecraft missions require extensive interaction from ground based
human operators. This interaction ranges from system health checks to
navigation and hardware commands. In addition, there is frequently a large
group of analysts in support of any given mission. A wide variety of factors
make human in the loop control of spacecraft especially difficult. First, the
vast distances cause significant time delays which render it impossible to
react immediately to events experienced by the spacecraft. Furthermore, deep
space missions are designed for continuous mission operations for many years
or even decades. It is becoming increasingly difficult to maintain trained and
knowledgeable staff for several decades in order to support a single mission.
In addition, these operators become increasingly scarce as the contemporary
hardware and software tools surpass those of these decades old spacecraft. As
a result, there is a large focus on completely autonomous spacecraft systems.

We seek to utilize well-proven methods in the computer vision and robotics
community to first localize the position of the spacecraft from visual
imagery. A secondary step, which is left to a subsequent publication, is then
to use this imagery to autonomously update the position of the spacecraft
while simultaneously mapping the surface. We utilize the state of the art ORB-
SLAM implementation to estimate the state of the spacecraft using monocular
images [32]. This method provides a feature based monocular SLAM system that
can operate in real-time for a wide variety of environments. ORB-SLAM builds
on proven methods in the robotics community to create a custom SLAM system
that provides for autonomous tracking and mapping in an unknown environment.
The first step in ORB-SLAM is to determine accurate features within an image
that are most easily recognized in subsequent images. This “feature-
extraction” stage uses the FAST feature detector to determine suitable
features, and then the ORB feature descriptor to a store a compact, scale-
invariant description of the feature properties. Next, these features are
matched to subsequent images in a “feature-matching” stage to determine
correspondance between subsequent images. For example, Figure 2 shows a
demonstration of features computed in two view of asteroid Itokawa. In
addition, the correspondance is accurately computed between matching features
as shown by the horizontal lines across both images. From these
correspondences, the pose of the camera system is estimated and predicted
forward in time using a constant velocity model. In a simultaneous thread, the
ORB-SLAM system builds a local map and stores feature data in order to perform
loop closure in the event of returning to a previously imaged location. We
utilize ORB-SLAM to provide an estimate of the position of the rigid
spacecraft and will use this estimate to compute the control input.

Figure 2: Demonstration of feature detection and matching between simulated
images of asteroid 25413 Itokawa

##  4 Numerical Simulation

We present a numerical simulation of a rigid dumbbell about asteroid Itokawa.
The dumbbell spacecraft is composed of two equal masses,
$m_{1},m_{2}=$500\text{\,}\mathrm{kg}$$, separated by
$l=$3\text{\,}\mathrm{m}$$. The dumbbell body frame is defined with the first
body fixed axis, $\bm{b}_{1}$, originating at the center of mass of the
spacecraft and directed along the vector from $m_{1}$ towards $m_{2}$. The
other two axes of the spacecraft fixed frame are chosen orthogonal to the
$\bm{b}_{1}$ and lie in the plane orthogonal to the dumbbell axis of symmetry.
A camera, using the parameters from Table 1, is aligned with the $\bm{b}_{1}$
axis and used to feed image data to the ORB-SLAM system. A numerical
simulation is used to demonstrate the geometric control of the coupled motion
of the spacecraft, and the ability to estimate the motion of the spacecraft
from monocular imagery.

The initial condition of the spacecraft is defined as

$\displaystyle\bm{x}_{0}$
$\displaystyle=\begin{bmatrix}0&-2.550&0\end{bmatrix}$\mathrm{km}$,$ (30)
$\displaystyle R_{0}$ $\displaystyle=\exp{\frac{\pi}{2}\bm{e}_{3}}.$ (31)

The spacecraft begins on the inertial $\bm{e}_{2}$ axis and initially pointing
at the asteroid. A tracking command is designed to transition the spacecraft
towards the asteroid fixed $\bm{f}_{1}$ axis followed by a vertical descent
along towards the asteroid surface. The translational command is divided into
two stages, a traverse step where the spacecraft follows a trajectory to align
itself with the $\bm{f}_{1}$ axis and a landing step where the spacecraft
follows a constant velocity descent towards the surface. The desired position
command is defined as

$\displaystyle\bm{x}_{d}=\begin{cases}2.550\begin{bmatrix}\sin{\omega
t}&-\cos{\omega t}&0\end{bmatrix},&t\leq t_{d}\\\
R_{A}\begin{bmatrix}\frac{2}{t_{d}}(t-t_{d})+2.550&0&0\end{bmatrix},&t>t_{d},\end{cases}$
(32)

where $\omega=\frac{\pi}{2t_{d}}$, $t_{d}$ is the time from the simulation
start when the constant velocity descent should begin, and $t$ is the
simulation time step.

The desired attitude command is chosen such that the spacecraft camera axis,
$\bm{b}_{1}$, is directed along the nadir towards the asteroid. It is
sufficient to define two orthogonal vectors to uniquely determine the attitude
of the spacecraft. The $\bm{b}_{3d}$ vector is chosen to lie in the plane
spanned by $\bm{b}_{1d}$ and $\bm{e}_{3}=\bm{f}_{3}$. The desired attitude
command is defined as

$\displaystyle\bm{b}_{1d}$
$\displaystyle=-\frac{\bm{x}}{\left\|\bm{x}\right\|},$ (33)
$\displaystyle\bm{b}_{3d}$
$\displaystyle=\frac{\bm{f}_{3}-\left(\bm{f}_{3}\cdot\bm{b}_{1d}\right)\bm{b}_{1d}}{\left\|\bm{f}_{3}-\left(\bm{f}_{3}\cdot\bm{b}_{1d}\right)\bm{b}_{1d}\right\|},$
(34) $\displaystyle\bm{b}_{2d}$ $\displaystyle=\bm{b}_{3d}\times\bm{b}_{1d},$
(35) $\displaystyle R_{d}$
$\displaystyle=\begin{bmatrix}\bm{b}_{1d}&\bm{b}_{2d}&\bm{b}_{3d}\end{bmatrix}.$
(36)

The camera axis is aligned with the spacecraft $\bm{b}_{1}$ axis, which is
directed towards the asteroid, throughout the landing trajectory as the
spacecraft moves in the equatorial plane of the asteroid.

The simulation is carried out over $7200\text{\,}\mathrm{s}$ with the
spacecraft following a circular trajectory for the first $3600\text{\,}\sec$
before vertically descending in the body fixed frame for the last
$3600\text{\,}\mathrm{s}$. Figure 3(a) shows a view from the positive
$\bm{f}_{3}$ pole of the asteroid of the simulated trajectory. The position of
the center of mass of the dumbbell is shown in blue, while the pointing
direction of the camera axis is defined in red. The attitude of the dumbbell
is displayed at several points along the trajectory demonstrating the pointing
of the camera. Furthermore, asteroid Itokawa is shown in its final orientation
at the completion of the landing simulation. Figure 3(b) shows that the
nonlinear controller is able to accurately track the desired translational
trajectory for the duration of the simulation.

(a) Planar view of landing trajectory (b) Position of spacecraft in the
inertial frame

Figure 3: Landing trajectory to asteroid Itokawa

Concurrent with the simulation, we generate images of Itokawa using Blender,
as described earlier. The images are generated at a rate of
$1\text{\,}\mathrm{Hz}$ and stored for post processing and one example view is
shown in Figure 4. From these images we estimate the position of the camera
relative to the asteroid using monocular localization. In this work, we
demonstrate the ability to generate realistic imagery and the processing of
these images to determine a state estimate of the camera pose. In future work,
we seek to use this state estimate in a closed-loop controller to enable a
fully-autonomous control system for the spacecraft.

Figure 4: Example image of Itokawa at $t=$837\text{\,}\mathrm{s}$$

From each image, a series of features are detected and tracked between
subsequent frames. A visualization of this process is shown in Figure 5(a)
which shows the tracked features of the image in green. From these tracked
features it is possible to estimate the relative motion of the camera [33].
Figure 5(b) shows a visualization of the localization process from the
imagery. Through monocular images it is not possible to compute the position
of the camera relative to the inertial frame. Rather, it is only possible to
determine a camera pose accurate to an unknown scale factor. This limitation
is intrinsic to the transformation of the three-dimensional scene onto the
two-dimensional image plane [33]. However, it is possible to compute the
relative motion between sequential images and we use this to estimate the
motion of the camera.

(a) Feature tracking visualization (b) Localization from monocular images

Figure 5: ORB-SLAM2 feature detection and localization from imagery

An additional complexity is due to the fact that the dynamic equations of
motion are defined in the inertial reference frame. As a result, the asteroid
is rotating relative to this frame while the dumbbell is moving relative to
the inertial frame. The majority of SLAM methodologies use the implicit
assumption of a stationary scene or terrain and a moving camera. As a result,
the motion estimate from the imagery is defined relative to the asteroid fixed
frame, rather than the inertial frame. However, we assume that the motion of
the asteroid is known and constant. Over the relatively short timespan of the
simulation this is an appropriate assumption and does not significantly affect
the solution.

Figure 6: Comparison of image estimate and true trajectory in asteroid fixed
frame

Figure 6 shows a comparison of the estimated position in blue and the true
position in the asteroid fixed frame. The estimate is quite accurate in the
$x$ axis. This is primarily due to the fact that during the vertical descent
that asteroid remains fixed relative to the spacecraft. As a result, a large
number of image features are able to be accurately tracked during the motion
towards the surface. This is also evident by the denser keyframes in the later
portion of the simulation as compared to the beginning. The estimate tends to
diverge in the other two axes. However, the general trend is captured in the
$y$ axis and the error in the $z$ axis is on the order of
$100\text{\,}\mathrm{m}$. This simulation demonstrates the ability to track a
desired trajectory for the coupled motion of a rigid spacecraft on
$\mathsf{SE(3)}$. Furthermore, we show the ability to estimate the motion
using a single monocular camera during a short duration landing trajectory. In
spite of the complex and difficult lighting conditions an estimate is able to
be generated.

##  5 Conclusions

There have been a variety of approaches for the analysis and design of orbital
trajectories around asteroids. Relatively less work has been directed towards
the design of landing trajectories. Furthermore, much of the previous work has
only treated the orbital or translational dynamics. The approximation of a
spacecraft as a point mass rather than an extended rigid body severely limits
the applicability and ignores a major component of the dynamic environment.
This work directly derives the equations of motion of a dumbbell spacecraft
around an asteroid described using a polyhedron potential model. We explicitly
consider the impact of the gravitational moment on both the orbit and attitude
dynamics. With this accurate equations of motion we develop geometric
nonlinear controllers which allow for the vehicle to track a desired landing
trajectory. The desired landing trajectory is accurately followed by the
nonlinear controller on $\mathsf{SE(3)}$. Furthermore, we use current image
processing techniques to estimate the motion of the spacecraft given only two-
dimensional images. In spite of the challenging environment, with limited
lighting and a reduced simulation span, the method is able to accurately track
the motion of the vehicle. Future work will focus on utilizing this estimate
in a closed-loop control and guidance scheme to enable autonomous landing and
obstacle avoidance.

## References

  * [1] S. D. Ross, “Near-Earth Asteroid Mining,” Space, 2001.
  * [2] P. Brown, R. E. Spalding, D. O. ReVelle, E. Tagliaferri, and S. P. Worden, “The Flux of Small Near-earth Objects Colliding with the Earth,” Nature, Vol. 420, Nov. 2002, pp. 294–296.
  * [3] B. Wie, “Dynamics and Control of Gravity Tractor Spacecraft for Asteroid Deflection,” Journal of Guidance, Control, and Dynamics, Vol. 31, Oct. 2008, pp. 1413–1423, 10.2514/1.32735.
  * [4] R. Adams, R. Alexander, J. Bonometti, J. Chapman, S. Fincher, R. Hopkins, M. Kalkstein, T. Polsgrove, G. Statham, and S. White, “Survey of Technologies Relevant to Defense from Near-earth Objects,” NASA Marshall Space Flight Center, NASA-TP-2004-213089, 2004.
  * [5] D. J. Scheeres, “Dynamics about Uniformly Rotating Triaxial Ellipsoids: Applications to Asteroids,” Icarus, Vol. 110, No. 2, 1994, pp. 225–238.
  * [6] D. Scheeres, B. Williams, and J. Miller, “Evaluation of the Dynamic Environment of an Asteroid: Applications to 433 Eros,” Journal of Guidance, Control, and Dynamics, Vol. 23, No. 3, 2000, pp. 466–475.
  * [7] P. Hughes, Spacecraft Attitude Dynamics.  Dover Publications, 2004.
  * [8] H. M. Elmasri and N. H. McClamroch, “Dynamics and Control Properties for an Asymmetric Dumbbell Spacecraft,” Proceedings of 2005 IEEE Conference on Control Applications, 2005., Aug. 2005, pp. 364–369, 10.1109/CCA.2005.1507152.
  * [9] A. K. Sanyal, Dynamics and Control of Multibody Systems in Central Gravity. PhD thesis, University of Michigan, 2004.
  * [10] D. W. Dunham, R. W. Farquhar, J. V. McAdams, M. Holdridge, R. Nelson, K. Whittenburg, P. Antreasian, S. Chesley, C. Helfrich, W. M. Owen, et al., “Implementation of the First Asteroid Landing,” Icarus, Vol. 159, No. 2, 2002, pp. 433–438.
  * [11] T. Kubota, T. Hashimoto, J. Kawaguchi, M. Uo, and K. I. Shirakawa, “Guidance and Navigation of Hayabusa Spacecraft for Asteroid Exploration and Sample Return Mission,” SICE-ICASE, 2006. International Joint Conference, IEEE, 2006, pp. 2793–2796.
  * [12] B. Acikmese and S. R. Ploen, “Convex Programming Approach to Powered Descent Guidance for Mars Landing,” Journal of Guidance, Control, and Dynamics, Vol. 30, 2015/09/29 2007, pp. 1353–1366, 10.2514/1.27553.
  * [13] J. S. Meditch, “On the Problem of Optimal Thrust Programming for a Lunar Soft Landing,” Automatic Control, IEEE Transactions on, Vol. 9, No. 4, 1964, pp. 477–484.
  * [14] R. N. Ingoldby, “Guidance and Control System Design of the Viking Planetary Lander,” Journal of Guidance, Control, and Dynamics, Vol. 1, 2016/09/21 1978, pp. 189–196, 10.2514/3.55763.
  * [15] M. Guelman and D. Harel, “Power Limited Soft Landing on an Asteroid,” Journal of Guidance, Control, and Dynamics, Vol. 17, No. 1, 1994, pp. 15–20.
  * [16] R. Furfaro, D. Cersosimo, and D. R. Wibben, “Asteroid Precision Landing Via Multiple Sliding Surfaces Guidance Techniques,” Journal of Guidance, Control, and Dynamics, Vol. 36, 2016/09/21 2013, pp. 1075–1092, 10.2514/1.58246.
  * [17] Z. Zexu, W. Weidong, L. Litao, H. Xiangyu, C. Hutao, L. Shuang, and C. Pingyuan, “Robust Sliding Mode Guidance and Control for Soft Landing on Small Bodies,” Journal of the Franklin Institute, Vol. 349, No. 2, 2012, pp. 493–509.  Advances in Guidance and Control of Aerospace Vehicles using Sliding Mode Control and Observation Techniques, http://dx.doi.org/10.1016/j.jfranklin.2011.07.007.
  * [18] R. A. Werner, “The Gravitational Potential of a Homogeneous Polyhedron or Don’t Cut Corners,” Celestial Mechanics and Dynamical Astronomy, Vol. 59, No. 3, 1994, pp. 253–278.
  * [19] R. A. Werner and D. J. Scheeres, “Exterior Gravitation of a Polyhedron Derived and Compared with Harmonic and Mascon Gravitation Representations of Asteroid 4769 Castalia,” Celestial Mechanics and Dynamical Astronomy, Vol. 65, No. 3, 1996, pp. 313–344.
  * [20] T. Kubota, T. Hashimoto, S. Sawai, J. Kawaguchi, K. Ninomiya, M. Uo, and K. Baba, “An Autonomous Navigation and Guidance System for MUSES-C Asteroid Landing,” Acta Astronautica, Vol. 52, No. 2, 2003, pp. 125–131.
  * [21] T. Miso, T. Hashimoto, and K. Ninomiya, “Optical Guidance for Autonomous Landing of Spacecraft,” IEEE Transactions on Aerospace and Electronic Systems, Vol. 35, Apr. 1999, pp. 459–473, 10.1109/7.766929.
  * [22] C. Lanczos, The Variational Principles of Mechanics, Vol. 4.  Courier Corporation, 1970.
  * [23] J. Kawaguchi, “Hayabusa, Summary of Guidance, Navigation and Control Achievement in Its Proximity Phase,” Astrodynamics Specialist Conference, Keystone, Colorado, 2006.
  * [24] A. Tanimoto, N. Takeishi, T. Yairi, Y. Tsuda, F. Terui, N. Ogawa, and Y. Mimasu, “Fast estimation of asteroid shape and motion for spacecraft navigation,” 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dec. 2013, pp. 1550–1555, 10.1109/ROBIO.2013.6739687.
  * [25] A. Fujiwara, J. Kawaguchi, D. Yeomans, M. Abe, T. Mukai, T. Okada, J. Saito, H. Yano, M. Yoshikawa, D. Scheeres, et al., “The Rubble-pile Asteroid Itokawa As Observed by Hayabusa,” Science, Vol. 312, No. 5778, 2006, pp. 1330–1334.
  * [26] Blender Foundation, “Blender,” Feb. 2017, https://www.blender.org/.
  * [27] E. S. Hawkins, E. H. Darlington, S. L. Murchie, K. Peacock, T. J. Harris, C. B. Hersman, M. J. Elko, D. T. Prendergast, B. W. Ballard, R. E. Gold, J. Veverka, and M. S. Robinson, Multi-Spectral Imager On the Near Earth Asteroid Rendezvous Mission, pp. 31–100.  Dordrecht: Springer Netherlands, 1997, 10.1007/978-94-011-5200-6-2.
  * [28] M. Li, W. Jing, M. Macdonald, and C. R. McInnes, “Adaptive Backstepping Control for Optimal Descent with Embedded Autonomy,” Aerospace Science and Technology, Vol. 15, No. 7, 2011, pp. 589–594, http://dx.doi.org/10.1016/j.ast.2010.12.002.
  * [29] T. Lee, M. Leoky, and N. McClamroch, “Geometric tracking control of a quadrotor UAV on SE (3),” 49th IEEE Conference on Decision and Control (CDC), No. 3, 2010, pp. 5420–5425.
  * [30] T. Lee, M. Leok, and N. H. McClamroch, “Nonlinear Robust Tracking Control of a Quadrotor UAV on Se (3),” Asian Journal of Control, Vol. 15, No. 2, 2013, pp. 391–408.
  * [31] N. S. Nise, Control Systems Engineering.  John Wiley & Sons, 4 ed., 2004.
  * [32] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM: A Versatile and Accurate Monocular SLAM System,” IEEE Transactions on Robotics, Vol. 31, Oct. 2015, pp. 1147–1163, 10.1109/TRO.2015.2463671.
  * [33] R. Szeliski, Computer Vision: Algorithms and Applications.  Springer, 2010.


# FSVVD: A Dataset of Full Scene Volumetric Video

Kaiyuan Hu1,2†, Yili Jin1,2†, Haowen Yang1,5, Junhua Liu1, Fangxin Wang2,1,3,4
1The Future Network of Intelligence Institute, The Chinese University of Hong
Kong, Shenzhen
2School of Science and Engineering, The Chinese University of Hong Kong,
Shenzhen
3The Guangdong Provincial Key Laboratory of Future Networks of Intelligence
4Peng Cheng Laboratory, 5Versee Inc.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2018; 2023)

###### Abstract.

Recent years have witnessed a rapid development of immersive multimedia which
bridges the gap between the real world and virtual space. Volumetric videos,
as an emerging representative 3D video paradigm that empowers extended
reality, stand out to provide unprecedented immersive and interactive video
watching experience. Despite the tremendous potential, the research towards 3D
volumetric video is still in its infancy, relying on sufficient and complete
datasets for further exploration. However, existing related volumetric video
datasets mostly only include a single object, lacking details about the scene
and the interaction between them. In this paper, we focus on the current most
widely used data format, point cloud, and for the first time release a full-
scene volumetric video dataset that includes multiple people and their daily
activities interacting with the external environments. Comprehensive dataset
description and analysis are conducted, with potential usage of this dataset.
The dataset and additional tools can be accessed via the following website:
https://cuhksz-inml.github.io/full_scene_volumetric_video_dataset/.

Volumetric video, datasets, XR

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 07–10, 2023; Vancouver,
Canada††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††journalyear:
2023††copyright: acmlicensed††conference: Proceedings of the 14th ACM
Multimedia Systems Conference; June 7–10, 2023; Vancouver, BC,
Canada††booktitle: Proceedings of the 14th ACM Multimedia Systems Conference
(MMSys ’23), June 7–10, 2023, Vancouver, BC, Canada††price: 15.00††doi:
10.1145/3587819.3592551††isbn: 979-8-4007-0148-1/23/06††ccs: Information
systems Multimedia databases††ccs: Computing methodologies Volumetric models
Figure 1. Sample frames of the released dataset sequence

##  1\. Introduction

Recent years have witnessed a rapid development of immersive multimedia which
bridges the gap between the real world and virtual space. Volumetric videos,
as an emerging representative 3D video paradigm that empowers extended
reality, stand out to provide unprecedented fully immersive, and 6 degree-of-
freedom (DoF) interactive video watching experience. According to industry
research provider ”The Insight Partners”, the volumetric video market is
expected to grow from US$1,898.7 million in 2021 to US$9,685.7 million by
2028.

Table 1. Comparisons with related datasets

| Point Clouds | Multi-person | Multi-view Depth | 3D-Mesh | Interactive Objects | Full Scenes | #Videos | #Categories
---|---|---|---|---|---|---|---|---
CWIPC-SXR (Reimat et al., 2021) | ✓ | ✓ | ✓ | - | ✓ | - | 21 | 4
HUMAN4D (Chatzitofis et al., 2020) | ✓ | ✓ | ✓ | ✓ | ✓ | - | 19 | 3
HUMBI (Yoon et al., 2023) | - | - | - | ✓ | - | - | 700+ | N/A
Volograms & V-SENSE (Pagés et al., 2021) | ✓ | - | - | - | - | - | 3 | 3
Ours | ✓ | ✓ | ✓ | - | ✓ | ✓ | 26 | 4

The key difference between volumetric video compared with traditional 2D flat
video (Jin et al., 2022, 2023) lies in the 3D representation, where the
commonly used formats are point cloud, mesh, voxel, and the recent implicit
neural representation. Among all these representations, point cloud is
currently the most popular due to its simplicity and easy deployment (Liu et
al., 2023). Such video transition from 2D to 3D volumetric video actually
brings quite a few new challenges, such as the vast data size, fast dynamic 6
DoF interaction, ultra-low latency requirement, client-side lightweight and
smooth playback, etc. Since the research towards 3D volumetric video is still
in its infancy, a sufficient and complete dataset is in urgent need in this
community for further exploration.

There are a few existing datasets on volumetric video, yet they either only
focus on single-person capture and reconstruction or lack the interaction
between the target person and the surrounding environment. Thus, we are
motivated to collect a comprehensive and complete dataset to facilitate
further research.

In this paper, we release a publicly available point-cloud-based Full Scene
Volumetric Video Dataset (FSVVD), which contains more than 30 different daily
life scenarios. To our best knowledge, FSVVD is the first full-scene
volumetric video dataset containing both multiple target people and the
interacting environment. We also introduce the detailed capture procedure and
release our designed processing tools. A comprehensive analysis of the impact
of factors on the volumetric dataset quality is further conducted, including
the factors of object movement, scene complexity, and illumination condition.
We find that object movement has more impact on volumetric video quality
causing ghost images. We finally point out some potential usage based on our
dataset.

The rest of the paper is organized as follows. Section 2 introduces previous
works of related datasets. Section 3 describes the whole pipeline and
principle of capturing volumetric video data. In Section 4, the content of
FSVVD is introduced, and in Section 5, we make a comprehensive analysis of the
impact of factors on the volumetric dataset quality. Finally, we point out
some potential usage of our dataset and make conclusions in Section 6 and
Section 7.

##  2\. Existing Datasets

Over the past few years, both academia and industry have shown great interest
in capturing volumetric video data using a variety of RGB-based or depth-based
methods. Such volumetric video data promote the research over immersive
multimedia technologies like inter-frame interpolation (Zhang et al., 2020),
compression & decompression, and streaming. We make a brief overview of the
related datasets, and summarize the features and modalities by making
comparisons, which is illustrated in Table 1.

Some previous works only use conventional RGB cameras for capture. HUMBI (Yoon
et al., 2023) is a publicly available multi-view image dataset, which provides
human body expressions including gaze, face, hand, body, and garment. The
capture exploited 107 synchronized HD cameras and more than 700 subjects
across gender, ethnicity, age, and style. The dataset provides 3D mesh models
along with multi-view image streams. And it aims to exert an influence on
learning and reconstructing a complete human model and complementing the
existing datasets of human body expressions with limited views and subjects.
However, depth cameras are not involved in the capture process, thus multi-
view depth data is absent.

Another available volumetric sequence dataset is Volograms & V-Sense dataset
(Pagés et al., 2021). The dataset includes three sequences featuring three
different characters, each is captured using 12 HD cameras with a different
purpose and application in mind and has different characteristics in terms of
texture and movement. Unfortunately, this dataset contains very few movements
and scenarios.

Conversely, some works exploit depth cameras in the capture process in order
to obtain volumetric data composed of point clouds. CWIPC-SXR (Reimat et al.,
2021) consisting of 21 sequences, provides a dataset for social XR scenarios
captured by commodity depth sensors. In such a release, human interaction in
social XR scenarios is depicted in the aspects of four use cases: “Education
and Training”, “Healthcare”, “Communication and Social interactions”, and
“Performance and Sports”, which only focus on the movement of individuals in
practical scenarios without the full scene.

Meanwhile, Human4D (Chatzitofis et al., 2020) dataset contains simultaneously
captured human activities using both motion capture devices and depth sensors,
such a motion capture method utilized 24 motion capture cameras to achieve
higher accuracy. And the method requires actors to wear a pure black uniform
with color dots to locate key skeleton points to obtain reference animation of
the actors, which is not practical in real-life scenarios. The dataset
provides a reproduction of human physical, daily and social activities. Though
the captured data are of high accuracy, such capture setups are difficult to
deploy in practical use and they lack interaction in the full scene as well.

Regrettably, all of the above datasets lack human interaction with objects in
a full scene, and such a feature provides opportunities in tackling
challenging tasks of multi-object occlusion (Han et al., 2020)and inter-frame
interpolation in immersive multimedia. However, FSVVD fills the gap by
providing volumetric video data that not only depicts human interaction with
objects but also the full scene involved, which aims to energize the research
on immersive multimedia topics including volumetric video streaming and real-
time telepresence.

Figure 2. The pipeline of data spatial alignment: calibration and alignment

##  3\. capture procedure

In the data acquisition process, we design a capture tool that enables
multiple devices to capture the data simultaneously. Plus, a data processing
tool to convert the raw data in binary into the readable file format of .jpg,
.ply, and .pgm. For the capture setup, we build the capture environment with 6
commodity Microsoft Azure Kinect devices in several daily scenes including an
office, a warehouse, a meeting room, and a kitchen. Each Azure Kinect device
is connected to a client PC for data pre-processing and transmission purposes,
and all of the client PCs are connected to a high-performance host PC which is
responsible for data processing. The capture environment is illuminated by 2
lamps according to needs, each having a power of 28W.

###  3.1. Software Setup

For the software setup, we build a volumetric capture tool based on the
Microsoft Azure Kinect APIs, and a data processing tool to convert the
captured raw data in binary into .ply format for further processing, which is
fitting the color data to depth data. The data in the interested region can
also be extracted by filtering using the data processing tool.

###  3.2. Hardware Setup

For the hardware setup, we only use commodity devices to build the capture
environment. We use 6 Microsoft Azure Kinect devices to capture color and
depth data, each device is connected to a client PC for data pre-processing
and transmission to the host. The client PCs are commodity laptops, which are
connected to the sensors using USB 3.0 cables. While the host is a high-
performance PC running Windows 10 Pro on an Intel i7-10700 GPU with an NVIDIA
GeForce RTX 3060 GPU and 16GB RAM that is responsible for converting raw data
in binary into colored point clouds data in the format of .ply. Each of the
Azure Kinect devices is uniformly placed facing inwards perimetrically around
the center of the capturing region with a radius of 1.75 meters. In order to
capture objects at a closer distance by exploiting the sensor’s wide
horizontal field of view (Wei et al., 2022), all of the Azure Kinect devices
are placed vertically. In addition, to attain better adjustability, we attach
each device on a tripod and set the height and angle of the sensors according
to different capture environments.

###  3.3. Synchronization

We exploit hardware synchronization in order to synchronize each sensor in
data capturing. The 6 sensors are interconnected in a special topology of
daisy-chain by 3.5mm audio cables, with 1 ‘master’ device and 5 ‘slave’
devices. During capturing, the master device will trigger other slave devices
successively with an interval of 30 microseconds for each frame, enabling the
camera array to take the shot simultaneously. Since the data flow bandwidth of
a single device is as much as 80 Mbps, to support real-time data transmission
between the host and client PC, we use a network switch of 1000 Mbps bandwidth
and multiple Ethernet wires to connect the host and the client PCs.

###  3.4. Calibration

Multi-sensor spatial alignment is essential in exploiting multiple depth
sensors. Typically, such a procedure is achieved via marker-based methods,
where all views align with respect to some markers and are then registered
among themselves. Traditional methods involve moving a checkboard pattern
within the capturing region or in the case of retro-reflective markers, a
wand, while in our implementation of spatial alignment of Azure Kinects, we
use a structure-based method without markers but instead a physical structure
assembled by 4 commonly found boxes. During the calibration process, we
assemble the alignment structure in the center of the capturing region, such
that the structure is fully exposed to all of the sensors. For that, we rely
on data-driven correspondence establishment for the initial matching (Mo et
al., 2019; Sterzentsenko et al., 2020), and global optimization for estimating
a solution with respect to the coordinate system of the structure. The
illustration of the calibration process is shown in the blue region of Figure
2.

###  3.5. Alignment

The captured data is originally in binary format, we build a data processing
tool in order to convert the data into the displayable format of .ply. The
process of alignment is illustrated in the green region of Figure 2. By
exploiting the spatial alignment parameters from the calibration process, the
captured RGB images are aligned with the depth data to obtain the colored
point clouds.

Table 2. Description of the dataset: Each sequence is recorded separately according to different screenplays, and the frame numbers are provided. The information in detail of each screenplay is provided in the screenplay.txt in each sequence package.  Category | Screenplay | Actors | Frame Number | Scene
---|---|---|---|---
| Presenting | 1 Male | 191 | meeting room
| Answering questions | 1 Male | 339 | warehouse
Education | Writing on a iPad | 1 Male | 330 | warehouse
| Talking in front of whiteboard | 1 Male | 154 | meeting room
| Clean whiteboard | 1 Male | 252 | meeting room
| Lift box | 1 Male | 256 | office
Exercise | Pulling trolley | 1 Male | 222 | warehouse
| Rope skipping | 1 Male | 252 | warehouse
| Sit down | 1 Male | 220 | office
| Stand up | 1 Male | 132 | office
| Tie shoes | 1 Male | 194 | office
| Drinking | 1 Male | 316 | office
| Sweep the floor | 1 Male | 538 | warehouse
| Chatting between 2 person | 2 Male | 191 | warehouse
Daily life | Use iPad | 1 Male | 135 | warehouse
| Use laptop | 1 Male | 399 | office
| Interview | 2 Male | 207 | office
| Discussing | 2 Male | 640 | meeting room
| Use Quest Pro | 1 Male | 180 | kitchen
| News interview | 1 Male + 1 Female | 2192 | meeting room
| Walk around | 1 Male | 127 | warehouse
| Vive interact | 1 Male | 383 | warehouse
| Use Hololens | 1 Male | 226 | office
Entertainment | Playing violin | 2 Female | 1863 | empty room
| Play VR action games | 1 Male | 434 | kitchen
| Play guitar | 1 Male | 212 | office

##  4\. Dataset

In order to reproduce real-life scenarios as realistically as possible, we
design 4 categories of volumetric data sequences with respect to 4 real-life
scenarios: ‘Education’, ‘Exercise’, ‘Daily life’, and ‘Entertainment’. Each of
the screenplays is thoroughly designed in order to fully depict daily
interactive scenarios with detail. The duration of the sequence range from 5
to 88 seconds and the total number of frame range from 127 to 2192. To assist
further research on volumetric video, we design some challenging props in
certain screenplays, for example, small objects like a VR controller are
involved in the screenplay ‘Vive interact’ and ‘Play VR action games’. The
description of the cases is shown in Table 2 in the aspects of screenplay
name, actor information, frame number, and scene.

In our recordings, we capture 26 sequences of volumetric videos in total, 21
include single actors and 5 include multiple actors. According to the design
of the screenplays, the actors are dressed corresponding to the scene. To
preserve the RGB information to the maximum extent, we compensate for the
ambient light using two LED photoflood lamps.

Figure 3. Illustration of dataset structure

The structure of the dataset is described in Figure 3. For every screenplay
folder, there are 3 sub-folders, 2 of which contain the point cloud data, one
is the data after filtering, which only focus on the contents in the center of
the capture region and the other is the original data that contains the
texture of surrounding environments. All of the volumetric sequences are
stored in .ply format, the naming convention is: screenplay-name_frame-
number_Raw/Filtered.ply. (For example, ’Use_Hololens_204_Filtered.ply’
represents the 204th frame of the screenplay of ‘using a Hololens device’,
with the data filtered by the processing tool.) The other folder contains
supplementary files, including a screenplay.txt file describing the detailed
setups of the screenplay, and a device_repository.json describing the device
configurations and the capture setups.

##  5\. Data analysis

Processing and rendering volumetric data is a huge computation drain (Gül et
al., 2020; Liu et al., 2022). We conduct several analyses on how certain
factors will impact the data quality and computational performance of our
proposed dataset in the aspects of capture device setups, object movement,
scene complexity, and illumination condition. The affected factors include
volumetric video data size, point cloud density, and image detail loss.

Figure 4. Different device setups

###  5.1. Device Setups

Due to the limitation of the sensor’s FOV (Wei et al., 2022), there will be a
loss in detail to a certain extent depending on the number of depth cameras
and occlusion conditions, especially in scenarios with multiple objects. In
order to assess the impact, we use 3, 4, and 6 sensors to capture the same
scenario for the same duration. For the capture process, we keep the capture
setups (screenplay, illumination condition, camera configurations) all the
same. To get a better observation of the impact, we capture a static scenario
with only one actor in this experiment. In each capture, the same actor stays
at the same position and makes no movement, and the duration is all set as 50
frames. The experiment result shows: For the same scenario, the average points
number per frame is 40210 for 3 devices, 57695 for 4 devices, and 81630 for 6
devices. Based on the results, we summarize the conclusion as follows: The
level of detail of the captured data is higher when employing more devices.
And a loss of detail will arise in some areas where the components are thin or
small (e.g., hand gestures). Accordingly, with the loss of detail, the number
of points per frame is lower when fewer cameras are deployed, with more
capture devices, this problem could be relieved. To illustrate our findings,
several example frames are provided in Figure 4, from which we can see the
facial details and surface integrity are enhanced with the increased number of
cameras, while the cost is the texture impairments and outliers due to the
multi-camera overlapping scans. However, the cost could be reduced by
improving illumination conditions, which will be covered in detail in the
latter discussion.

Figure 5. Comparison of different amplitudes of motion
Left: Static, Middle: Fidgets, Right: Walking

###  5.2. Movement

Fast movement has an impact on the quality of 2D videos causing ghost images,
which is also for volumetric videos. To find the impact of object movement, we
conduct a contrast experiment that includes different amplitudes of motion.
For the experiment setup, we make the same male actor standing in the center
of the capture region and act different amplitudes of motion, with the capture
duration and capture setups all the same as in the previous experiment. In
experiment 1, the actor stands still in the center with no movement, in
experiment 2, the actor stands still but makes fidgets occasionally, while in
experiment 3, the actor walks around while waving his arms. The experiment
results are depicted in Figure 5, and the details in specific regions are
zoomed in the box of different colors. From the results, we find that the
point clouds are sparser in regions where large movements are performed, which
causes the loss of detail and ghost images in specific regions.

###  5.3. Scene Complexity

The complexity of the scene has an impact on the size of the volumetric video
data, thus we capture data in the same scene with different numbers of objects
for evaluation. In such an experiment, the duration is set as 50 frames, and
the capture setups are the same 6 Azure Kinects with the same illumination
conditions. The scene layouts are illustrated in Figure 6: For scene 1, the
capture region is set to empty as a baseline. For scene 2, we add a chair in
the center of the scene. And for scene 3, we add an actor sitting on a chair.
Then we perform segmentation to the point clouds object by object to each of
the 50 frames, as a result, we find: On average, the chair consists of 7371
points and the human consists of 16351 points. The results are listed in Table
3. According to the test result, we find that the volumetric data size
increase with the complexity of the scene which contributes to the rendering
costs.

Figure 6. Comparison of different scene complexity Table 3. Composition of different objects Object | #Point Clouds | Size
---|---|---
Empty Scene | 58329 | 2607KB
Chair | 7371 | 288KB
Human | 16351 | 680KB
Figure 7. Comparison of different illumination conditions

###  5.4. Illumination Condition

The Microsoft Azure Kinect device integrates an RGB camera and a depth camera
based on IR. The final displayable point cloud is the result of the alignment
of RGB data and depth data as described in Section 3.4 and Section 3.5.
However, distortion of color information arises when the illumination
condition of the capture region is limited. To evaluate the impact of
illumination, we make a comparative trial under different illumination
conditions. In the original capture setup, we use 2 LED lamps for illumination
compensation, both are set to maximum brightness, while in this trial, we test
4 levels of illumination conditions from no external illumination to maximum
brightness of the light lamps. The result is shown in Figure 7. The left
figure shows the result of no external illumination, we can see a huge loss in
RGB detail, in addition, the point cloud density is affected as well. The
other figures show the result of 30%, 60%, and 100% illumination from the
light lamps. From the comparison of the results, we find loss in RGB detail
and point cloud density will arise under insufficient illumination.

##  6\. Uses of the dataset

This dataset is designed to promote the research of volumetric videos and each
of the screenplays is designed with careful thought in order to provide more
opportunity for algorithm optimization, parameters adjustment, and performance
evaluation in aspects including capture, rendering, and transmission.

###  6.1. Virtual & Augmented Reality

Virtual and augmented reality are two areas where volumetric video could have
a significant impact. The ability to capture and render 3D scenes with high
levels of detail and realism (Zhang et al., 2020) could enable the creation of
more immersive and engaging VR and AR experiences. For example, FSVVD could be
used to create training simulations for a variety of industries, such as
healthcare, military, and emergency response. These simulations could provide
users with a realistic and interactive experience, allowing them to practice
and learn in a safe and controllable environment.

###  6.2. Social Interaction & Communication

In addition to training simulations, FSVVD could be used to create virtual
environments for social interaction and communication. For example, the
dataset could be used to create virtual worlds where users can interact with
each other and with virtual objects and environments. These virtual worlds
could provide a rich and engaging experience for users, allowing them to
socialize, collaborate, and engage in a variety of activities.

###  6.3. Healthcare

The healthcare industry is another area where FSVVD could have a significant
impact. The ability to capture and render 3D scenes with high levels of detail
and realism could enable the development of machine learning models that are
able to recognize and classify different medical conditions and treatments.
These models could be used to assist doctors and nurses in making more
accurate and timely diagnoses and treatment decisions. In addition, the
dataset could be used to train models that are able to generate natural
language descriptions of medical procedures and diagnoses, which could be used
to improve communication and collaboration between healthcare providers and
patients.

###  6.4. AI Systems

Finally, FSVVD could be used to improve the accuracy and functionality of
virtual assistants and other AI-powered systems. The dataset could be used to
train models that are able to understand and interact with daily life
scenarios, such as recognizing and responding to different types of objects
and activities in a scene. This could enable virtual assistants and other AI
systems to provide more accurate and helpful assistance to users in their
daily lives.

##  7\. Conclusions

In conclusion, we propose FSVVD (Full Scene Volumetric Video Dataset), a
publicly available volumetric video dataset that depicts human interactions
with objects and full related scenes. This dataset provides more than 30
different daily scenarios and aims to provide a universal dataset for
evaluation and research in the application of volumetric representation in
real-life scenarios. To the best of our knowledge, FSVVD is the first
volumetric dataset that provides full scenes of daily scenarios. Additionally,
we perform and report a detailed analysis of the impact factors of volumetric
video quality, including capture setups, object movements, scene complexity,
and illumination conditions. Our future work mainly focuses on adding
sequences to cover more areas and improving capture accuracy. Overall, FSVVD
contributes to energizing the research on multimedia topics of volumetric
video and real-time telepresence.

###### Acknowledgements.

The work is supported in part by the Basic Research Project No. HZQB-
KCZYZ-2021067 of Hetao Shenzhen-HK S&T Cooperation Zone, by National Natural
Science Foundation of China (Grant No. 62102342), by Guangdong Basic and
Applied Basic Research Foundation (Grant No. 2023A1515012668), by Shenzhen
Science and Technology Program (Grant No. RCBS20221008093120047), by Shenzhen
Outstanding Talents Training Fund 202002, by Guangdong Research Projects No.
2017ZT07X152 and No. 2019CX01X104, by the Guangdong Provincial Key Laboratory
of Future Networks of Intelligence (Grant No. 2022B1212010001), by Young Elite
Scientists Sponsorship Program by CAST (Grant No. 2022QNRC001) and by The
Major Key Project of PCL Department of Broadband Communication.

## References

  * (1)
  * Chatzitofis et al. (2020) Anargyros Chatzitofis, Leonidas Saroglou, Prodromos Boutis, Petros Drakoulis, Nikolaos Zioulis, Shishir Subramanyam, Bart Kevelham, Caecilia Charbonnier, Pablo César, Dimitrios Zarpalas, Stefanos D. Kollias, and Petros Daras. 2020.  HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media.  _IEEE Access_ 8 (2020), 176241–176262.
  * Gül et al. (2020) Serhan Gül, Dimitri Podborski, Thomas Buchholz, Thomas Schierl, and Cornelius Hellge. 2020\.  Low-latency cloud-based volumetric video streaming using head motion prediction. In _Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), 2020_. ACM.
  * Han et al. (2020) Bo Han, Yu Liu, and Feng Qian. 2020.  ViVo: visibility-aware mobile volumetric video streaming. In _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (MobiCom), 2020_. ACM.
  * Jin et al. (2022) Yili Jin, Junhua Liu, Fangxin Wang, and Shuguang Cui. 2022\.  Where Are You Looking?: A Large-Scale Dataset of Head and Gaze Behavior for 360-Degree Videos and a Pilot Study. In _Proceedings of the 30th ACM International Conference on Multimedia (MM), 2022_. ACM.
  * Jin et al. (2023) Yili Jin, Junhua Liu, Fangxin Wang, and Shuguang Cui. 2023\.  Ebublio: Edge Assisted Multi-User 360-Degree Video Streaming.  _IEEE Internet Things J._ (2023).
  * Liu et al. (2023) Junhua Liu, Boxiang Zhu, Fangxin Wang, Yili Jin, Wenyi Zhang, Zihan Xu, and Shuguang Cui. 2023.  CaV3: Cache-Assisted Viewport Adaptive Volumetric Video Streaming. In _Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2023_. IEEE.
  * Liu et al. (2022) Yu Liu, Bo Han, Feng Qian, Arvind Narayanan, and Zhi-Li Zhang. 2022.  Vues: Practical Mobile Volumetric Video Streaming through Multiview Transcoding. In _Proceedings of the 28th Annual International Conference on Mobile Computing And Networking (MobiCom), 2022_. ACM.
  * Mo et al. (2019) Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J. Mitra, and Leonidas J. Guibas. 2019.  StructureNet: hierarchical graph networks for 3D shape generation.  _ACM Trans. Graph._ 38, 6 (2019), 242:1–242:19.
  * Pagés et al. (2021) Rafael Pagés, Emin Zerman, Konstantinos Amplianitis, Jan Ondřej, and Aljosa Smolic. 2021\.  Volograms & V-SENSE Volumetric Video Dataset.  _ISO/IEC JTC1/SC29/WG07 MPEG2021/m56767_ (2021).
  * Reimat et al. (2021) Ignacio Reimat, Evangelos Alexiou, Jack Jansen, Irene Viola, Shishir Subramanyam, and Pablo Cesar. 2021\.  CWIPC-SXR: Point Cloud Dynamic Human Dataset for Social XR. In _Proceedings of the 12th ACM Multimedia Systems Conference (MMSys), 2021_. ACM.
  * Sterzentsenko et al. (2020) Vladimiros Sterzentsenko, Alexandros Doumanoglou, Spyridon Thermos, Nikolaos Zioulis, Dimitrios Zarpalas, and Petros Daras. 2020.  Deep Soft Procrustes for Markerless Volumetric Sensor Alignment. In _Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2020_.
  * Wei et al. (2022) Fan Wei, Guanghua Xu, Qingqiang Wu, Jiachen Kuang, Peiyuan Tian, Penglin Qin, and Zejiang Li. 2022.  Azure Kinect Calibration and Parameter Recommendation in Different Scenarios.  _IEEE Sensors Journal_ 22, 10 (2022), 9733–9742.
  * Yoon et al. (2023) Jae Shin Yoon, Zhixuan Yu, Jaesik Park, and Hyun Soo Park. 2023\.  HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge.  _IEEE Trans. Pattern Anal. Mach. Intell._ 45, 1 (2023), 623–640.
  * Zhang et al. (2020) Anlan Zhang, Chendong Wang, Xing Liu, Bo Han, and Feng Qian. 2020.  Mobile Volumetric Video Streaming Enhanced by Super Resolution. In _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services (MobiSys), 2020_. ACM.