# Real-Time Planning Under Uncertainty for AUVs Using Virtual Maps

Ivana Collado-Gonzalez, John McConnell, Jinkun Wang, Paul Szenher, and Brendan
Englot I. Collado-Gonzalez, P. Szenher, and B. Englot are with Stevens
Institute of Technology, Hoboken, NJ, USA, {icollado, pszenher,
<EMAIL_ADDRESS>J. McConnell was previously with Stevens and is now with
the United States Naval Academy, Annapolis, MD, USA<EMAIL_ADDRESS>J.
Wang was previously with Stevens and is now with Anyware Robotics, Fremont, CA
USA. This research was supported by NSF Grant IIS-1652064 and USDA-NIFA Grant
2021-67022-35977.

###### Abstract

Reliable localization is an essential capability for marine robots navigating
in GPS-denied environments. SLAM, commonly used to mitigate dead reckoning
errors, still fails in feature-sparse environments or with limited-range
sensors. Pose estimation can be improved by incorporating the uncertainty
prediction of future poses into the planning process and choosing actions that
reduce uncertainty. However, performing belief propagation is computationally
costly, especially when operating in large-scale environments. This work
proposes a computationally efficient planning under uncertainty framework
suitable for large-scale, feature-sparse environments. Our strategy leverages
SLAM graph and occupancy map data obtained from a prior exploration phase to
create a virtual map, describing the uncertainty of each map cell using a
multivariate Gaussian. The virtual map is then used as a cost map in the
planning phase, and performing belief propagation at each step is avoided. A
receding horizon planning strategy is implemented, managing a goal-reaching
and uncertainty-reduction tradeoff. Simulation experiments in a realistic
underwater environment validate this approach. Experimental comparisons
against a full belief propagation approach and a standard shortest-distance
approach are conducted.

##  I Introduction

Autonomous Underwater Vehicles (AUVs) are being increasingly used to perform
tasks such as inspection, mapping, and exploration. Since GPS is unavailable
underwater, AUVs must rely on onboard sensors to estimate their position,
making them susceptible to drift. Simultaneous Localization and Mapping (SLAM)
has been successfully applied to correct dead reckoning drift. However, SLAM
requires environmental features to lie within the AUV’s field of view.
Therefore, robots operating in feature-sparse environments or with short-range
sensors are especially prone to accumulating pose estimation errors. Thus,
executing trajectories that do not account for robot sensing capabilities
leads to high localization uncertainty. Planning under uncertainty, however,
commonly relies on belief propagation, which is computationally expensive and
can become infeasible to compute when operating in large-scale environments.

Figure 1: Planning under uncertainty example. Top: Gazebo environment and
planning example. Blue star is start location, red cross shows goal location,
pink arrows show candidate place-revisiting actions, and green diamond
represents goal-reaching candidate action. Bottom: Virtual map and
representative resulting paths using different uncertainty handling
strategies. Shortest Distance is shown in red, Expectation Maximization in
blue, and our proposed strategy in green. The uncertainty growth from
executing each path is illustrated along the path using 95% confidence
ellipses.

Recent work has focused on reducing map and robot pose uncertainty during the
exploration of unknown environments, without considering navigation tasks that
follow the mapping mission [1]. Furthermore, strategies that do consider goal-
reaching tasks commonly assume no prior knowledge [2] or perfect prior
knowledge of the environment. However, exploration is an imperfect process,
and the resulting maps are corrupted by noise, uncertainty, and incomplete
information, and should be treated accordingly when used for subsequent
planning queries. Planning to a specified goal can similarly be corrupted by
noise and uncertainty. Choosing an uninformative trajectory to reach a goal
can lead to substantial uncertainty growth, causing robots to crash or become
”lost”. Instead of discarding SLAM graph information after exploration, this
information can be used to assess the uncertainty of the resulting map,
informing any subsequent path planning queries. Moreover, by performing SLAM
during navigation, we can adapt to small changes in the environment, manage
errors from noise, and deal with incomplete information.

In this paper, we leverage mapping and SLAM graph information from a prior
exploration run and use it to create a virtual map that will in turn be used
to perform efficient planning under uncertainty, without the need for
computationally demanding online belief propagation. Specifically, we present
a planning under uncertainty scheme for underwater robots operating at a fixed
depth in large-scale feature-sparse environments. Our main contributions are
as follows:

  * •

To our knowledge, the first algorithmic pipeline for underwater robots that
uses exploration as a pre-processing step for planning under uncertainty to
specified goals.

  * •

We leverage the resulting SLAM graph and occupancy map information to produce
a useful costmap for computationally efficient planning under uncertainty.

  * •

Realistic simulation experiments, in which receding horizon planning is used
to compare our proposed method with shortest-distance and full belief
propagation planning strategies.

The rest of this work is structured as follows. First, Sec. II describes
related works. Next, the problem formulation is explained in Sec. III,
followed by the proposed method in Sec. IV. Subsequently, results and
experiments are presented in Sec. V. Finally, in Sec. VI conclusions are
drawn.

##  II Related Works

Incorporating localization uncertainty into the planning process is important
for navigation tasks including exploration and goal-reaching missions.
Actively maintaining high localization accuracy while navigating was first
addressed in [3], by controlling robot motion and sensing actions. Many mobile
robots, including AUVs, rely heavily on perceptual features to localize in
their environment, thus perception has a substantial impact on pose
uncertainty and has been studied in several works [4, 5, 6]. Moreover, in the
context of SLAM, revisiting previously seen features is necessary to trigger
loop closures and lower uncertainty. Image saliency was used in underwater
active SLAM [7] to identify uncertainty reducing actions. Some methods [8, 9]
attempt to reduce uncertainty by performing rotational actions to observe
visual features while following a path. However, only rotational movements are
not sufficient to localize when features are out of sensing range.

Therefore, balancing the trade-off between reducing uncertainty and completing
an exploration or goal-reaching task is an important part of planning under
uncertainty strategies. To limit deviations from the main path, [7] and [9]
only compute and perform uncertainty reducing actions when the uncertainty of
the robot pose is above a certain threshold. Most active SLAM motion planning
approaches employ three stages: goal identification, utility computation, and
action selection [10]. Furthermore, the receding horizon path planning
strategy, in which robots execute only a small portion of each plan prior to
replanning, has been used to solve the active SLAM problem in numerous works
[6, 7], where the continuous feedback of mapping the environment helps
attenuate errors in tracking and perception.

Furthermore, planning under uncertainty has been tackled using Partially
Observable Markov Decision Process (POMDP)s [11]. However, POMDP approaches
grow exponentially in complexity as the number of actions and observations
increases. To reduce computational burden various strategies have been
proposed. The Belief Roadmap (BRM) [12] made belief space planning more
efficient by factoring the covariance matrix and combining multiple EKF update
steps. Still, it results in signiﬁcant computation time for running on a
complex graph in practical problems. Rapidly-exploring Random Belief Trees
(RRBTs) [13] extend the fast computing Rapidly-exploring Random Tree (RRT)*
[14] framework to take pose uncertainty into account.

In the above methods, performing perception-related belief propagation is
crucial, but often the computational bottleneck. Hence, another body of work
has focused on developing dedicated representations of the environment and the
uncertainty for efficient computation. A coarse-resolution map for rapidly
evaluating the information gain of planned actions was used in [15].
Similarly, working with a reduced local map instead of a full map was done in
[16]. To avoid online belief propagation, [17] pre-computes and saves the
information content of the environment in a 2D grid which can be used during
the robot’s mission. [18] proposes a localizability matrix over a probability
grid map which includes localizability index and direction. Yet, this approach
only considers active localization and does not perform path planning. In
[19], the most feature-rich path for a rover is calculated by analyzing the
soil map of Mars, assuming a constant covariance accumulation rate in feature-
rich and feature-sparse areas. A technique that turns the information from
publicly available maps into a localizability map is presented in [20]. This
approach uses graph SLAM, but does not incorporate path planning.

Expectation-Maximization (EM) exploration [21, 22, 23] presents the concept of
a virtual map, in which every map cell’s uncertainty is represented by a
multivariate Gaussian, and the map uncertainty resulting from future actions
can be predicted. In our present work, we do not focus on exploration, but on
the subsequent planning to specified goal states after exploration has been
performed. We adapt the virtual map presented in [21], and use it as an
uncertainty cost map for computationally efficient planning, avoiding the need
for online belief propagation. Additionally, we use a receding horizon path
planning strategy that manages a goal-reaching and place-revisiting trade-off.
In the sections below, we will describe our proposed planning under
uncertainty framework and its application to underwater navigation with sonar,
demonstrating that it achieves far lower localization uncertainty and error
than a standard shortest-distance approach, and that the errors obtained are
comparable to that of full-fledged EM belief propagation.

##  III  Background and Problem Formulation

###  III-A Simultaneous Localization And Mapping

We use an incremental graph-based pose SLAM approach. Thus, the entire robot
trajectory is repeatedly estimated, allowing the robot pose history and map to
be corrected throughout the mission. The motion model is defined by

$\mathbf{x}_{i}=f_{i}(\mathbf{x}_{i-1},\mathbf{u}_{i})+\mathbf{w}_{i},\quad\mathbf{w}_{i}\sim\mathcal{N}(0,\mathbf{Q}_{i}),$
(1)

where $\mathbf{x}_{i}=[x_{i},y_{i},\theta_{i}]$ is the state vector
describing position and orientation in $\mathbf{SE(2)}$, $\mathbf{u}_{i}$ is
the control input vector, and $\mathbf{w}_{i}$ is additive process noise with
covariance matrix $\mathbf{Q}_{i}$. The sensor model is given by

$\mathbf{z}_{k}=h_{k}(\mathbf{x}_{ik})+\mathbf{q}_{k},\quad\mathbf{q}_{k}\sim\mathcal{N}(0,\mathbf{R}_{k}),$
(2)

where we obtain measurements $\mathbf{z}_{k}$ at pose $\mathbf{x}_{i}$,
corrupted by additive sensor noise $\mathbf{q}_{k}$ with covariance matrix
$\mathbf{R}_{k}$.

Let $\mathcal{X}=\\{\mathbf{x}^{T}_{i=0}\\}$ be the set of poses from time 0
to time $T$. $\mathcal{C}$ contains all constraints between robot poses. For
each pair of poses defining a constraint $\langle i,j\rangle\in\mathcal{C}$,
the error $\mathbf{e}_{ij}$ between observed transformation $\mathbf{z}_{ij}$
and predicted transformation $\hat{\mathbf{z}}_{ij}$ is defined as:

$\mathbf{e}_{ij}(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbf{z}_{ij}-\hat{\mathbf{z}}_{ij}(\mathbf{x}_{i},\mathbf{x}_{j}),$
(3)
$\hat{\mathbf{z}}_{ij}(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbf{x}_{i}^{\top}\mathbf{x}_{j}.$
(4)

The SLAM problem can be represented as a nonlinear least squares problem [24]:

$\mathcal{X}^{*}={\underset{{\mathbf{X}}}{\arg\min}}\sum_{\langle
i,j\rangle\in\mathcal{C}}\mathbf{F}_{ij},\qquad\mathbf{F}_{ij}=\mathbf{e}_{ij}^{\top}\mathbf{\Omega}_{ij}\mathbf{e}_{ij},$
(5)

where $\mathbf{F}_{ij}$ denotes the negative log-likelihood function of one
constraint between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$, and
$\mathbf{\Omega}_{ij}$ is the information matrix. The SLAM system aims to find
a set of robot poses which minimize the total observation error.

###  III-B Virtual Map

Unlike occupancy grid maps, the virtual map [23] encodes map cell uncertainty
(represented by a multivariate Gaussian) instead of occupancy probability. Let
the virtual map $\mathcal{V}=\\{\mathbf{v}_{i}\\}$ be the set of possible
observations located at each map cell $\mathbf{v}_{i}\in\mathbb{R}^{2}$,
distributed as $\mathcal{N}(\mathbf{v}_{i},\Sigma_{\mathbf{v}_{i}})$, where
$\mathbf{v}_{i}=[x_{i},y_{i}]$ is the cell location and
$\Sigma_{\mathbf{v}_{i}}$ is the corresponding covariance matrix. When
initializing a virtual map, a prior with conservatively high initial
uncertainty is imposed on all map cells.

The map cell covariances are updated based on the current estimate of the
trajectory $\mathcal{X}_{\text{old}}$ and the history of measurements
$\mathcal{Z}$.

$q(\mathcal{V})=p(\mathcal{V|X}_{\text{old}},\mathcal{Z})$ (6)

The maximum posterior probability estimate for the virtual map can thus be
computed by

$\mathcal{V}^{*}=\underset{\mathcal{V}}{\arg\max}\
p(\mathcal{V|X}_{\text{old}},\mathcal{Z}).$ (7)

It is assumed that measurements are assigned to maximize the likelihood

$\vspace{-2mm}\mathcal{Z}=\underset{\mathcal{Z}}{\arg\max}\ h(\mathcal{X,V}).$
(8)

By measuring environmental features, we can derive the transformations between
poses with overlapping observations using Iterative Closest Point (ICP). We
essentially aim to minimize the closeness of measurements using ICP-based pose
SLAM. For more details regarding virtual map pose covariance computation, the
reader should consult [23].

###  III-C Problem Statement

In this paper, the problem of autonomously navigating to a goal location under
pose uncertainty is addressed. We adapt the virtual map, previously used for
exploration, to accommodate planning under uncertainty to specified goals. We
leverage SLAM information from a prior exploration run to compute uncertainty
values offline and store them in the form of a virtual map $\mathcal{V}$. We
then focus on finding a path to a goal that curbs the growth of uncertainty
using pose SLAM and the covariance information from our virtual map.
Specifically, the virtual map is used as a cost map during the planning stage
to estimate future uncertainty, and the SLAM graph is maintained and updated
to help mitigate dead reckoning errors and to update our occupancy map used
for planning, as the prior map is imperfect.

##  IV Methodology

Figure 2: System overview: SLAM information from a prior exploration run is
used to initialize the current SLAM instance and build a virtual map offline.
The motion planner takes the accompanying occupancy map and searches for
candidate paths. The utility of each candidate path is assessed using the
virtual map. The chosen path and the current robot pose are then passed to the
guidance module, which provides the desired velocity vector to follow the
desired path. The low-level PID controller takes the desired velocity values
and defines the required thruster forces.

In this section, we explain our proposed framework, the system overview can be
observed in Fig. 2. First, SLAM information from a prior exploration run is
taken and used to initialize our current SLAM instance and build our virtual
map $\mathcal{V}$. The motion planner takes the occupancy map $\mathcal{M}$
from SLAM and searches for candidate paths
$\mathcal{P}=\\{\mathcal{X}_{c}\\}$. Then, the utility of each candidate path
is assessed using the virtual map. The chosen path $\mathcal{X}_{d}$ and the
current robot pose $\mathbf{x}$ are then passed to the pure pursuit guidance
module, which in turn provides the appropriate desired velocity vector
$\mathbf{u}$ to follow the desired path. The low-level PID controller takes
the desired velocity values and defines thruster forces $\mathbf{\tau}$.

###  IV-A Sonar SLAM

To estimate the location of the robot, we use the same graph-based pose SLAM
framework as in [23]. The sensory inputs include an Inertial Measurement Unit
(IMU), a Doppler Velocity Log (DVL), and a sonar image. The sonar image is
first processed using Constant False Alarm Rate (CFAR) detection [25] to
identify returns from the surrounding environment. All CFAR-extracted features
in the image are converted to Cartesian coordinates.The SLAM factor graph is
denoted as

$\displaystyle\vspace{-1mm}\mathbf{f}(\boldsymbol{\Theta})=\mathbf{f}^{\text{0}}(\boldsymbol{\Theta}_{0})$
$\displaystyle\prod_{i}\mathbf{f}^{\text{O}}_{i}(\boldsymbol{\Theta}_{i})\prod_{j}\mathbf{f}^{\text{LC}}_{j}(\boldsymbol{\Theta}_{j}),\vspace{-1mm}$

where variables $\boldsymbol{\Theta}$ contain 3-DOF robot poses and
$\mathbf{f}^{\text{0}}$ denotes the initial factor graph. The odometry factor
$\mathbf{f}^{\text{O}}$ defines the relative motion between two consecutive
poses from persistent odometry measurements obtained by DVL/IMU dead
reckoning. Loop closure factors $\mathbf{f}^{\text{LC}}$ are obtained by
identifying non-sequential scan matches, applying Iterative Closest Point
(ICP) [26] between the current sonar keyframe and frames inside a given search
radius. ICP is initialized using a DVL/IMU dead reckoning pose estimate and
further optimized using consensus set maximization [27], helping ICP to avoid
local minima. Loop closure outliers are rejected by evaluating point cloud
overlap and applying Pairwise Consistent Measurement Set Maximization (PCM)
[28]. At the back end, we employ the GTSAM [29] implementation of iSAM2 [30],
which is merged with dead reckoning to provide a high-frequency state
estimate. Occupancy map $\mathcal{M}$ is updated and used for path planning.

Algorithm 1 Place-revisiting actions

Occupied cells $\mathcal{O}$, number of place-revisiting actions
$N_{\text{pr}}$, revisiting radius $r_{\text{pr}}$, minimum separation
distance $d_{\text{pr}}$, shortest path to goal $\mathcal{X}_{\text{sd}}$,
number of clusters $k$

$\mathcal{A}_{\text{pr}},\mathcal{R}\leftarrow\emptyset$

0$\mathcal{M}=\\{\mathbf{m}_{1,...,k}\\}\leftarrow\text{ComputeKMeans}(\mathcal{O},k)$

-1$i\leftarrow 1$ to $k$

-2$\mathcal{C}\leftarrow\\{(x_{i}+r_{\text{pr}}\cos(\theta),y_{i}+r_{\text{pr}}\sin(\theta),\color[rgb]{0,0,0}\theta\color[rgb]{0,0,0})\ |\ \theta\in[0,2\pi)\\}$

-3$\mathcal{R}\leftarrow\mathcal{R}\cup\\{\mathcal{C}\\}$

-4

-5$|\mathcal{A}_{\text{pr}}|<N_{\text{pr}}$

-6$\mathbf{a}*\leftarrow\arg\min_{\mathbf{r}_{i}\in\mathcal{R}}||\mathbf{r}_{i}-\mathcal{X}_{\text{sd}}||^{2}$

-7$d^{*}\leftarrow\min_{\mathbf{g}_{i}\in\mathcal{A}_{\text{pr}}}|\mathbf{a}^{*}-\mathbf{g}_{i}|$

-8$d^{*}\leq d_{\text{pr}}$

-9$\mathcal{A}_{\text{pr}}\leftarrow\mathcal{A}_{\text{pr}}\cup\mathbf{a}^{*}$

-10

-11$\mathcal{R}\leftarrow\mathcal{R}\setminus\\{\mathbf{a}^{*}\\}$

-12

-13return Candidate place-revisiting actions $\mathcal{A}_{\text{pr}}$

###  IV-B Motion Planning

Global trajectory optimization in a large-scale planning problem is
computationally expensive, and may require replanning as the trajectory is
executed. Therefore, we apply a receding horizon path planner, which also
performs trajectory tracking and incorporates perceptual feedback. Inspired by
many active SLAM motion planning approaches [10], we employ three stages: goal
identification, utility computation, and action selection. Action selection
and path planning are treated as tightly coupled problems. We consider two
different action types, one type for lowering uncertainty (place-revisiting),
and another for reaching the mission goal (shortest-distance). Place-
revisiting actions look for poses that can lower uncertainty by revisiting
locations the robot has previously observed, triggering SLAM loop closures.
Shortest-distance actions represent locations that lay on the shortest
feasible path to the goal. Fig. 1a depicts place-revisiting actions as pink
arrows and the shortest-distance action as a green diamond.

First, a finely discretized roadmap is generated at the beginning of the
mission. The boundaries of the roadmap are defined, and edges are pruned out
if obstacles are discovered. The A* algorithm [31] is used to find the
shortest path from the start position to the final mission goal without
colliding with known obstacles. Afterward, the shortest-distance action set
$\mathcal{A}_{\text{sd}}$ is defined by choosing targets at user-defined
distances $d_{\text{sd}}$ from the current position along the identified
shortest distance path. For safety, the distance to the waypoints should be
less or equal to the robot sensing range to avoid collision with possible new
obstacles and refrain from unnecessary replanning. Likewise, traversing large
featureless distances should be prevented to avoid large uncertainty growth.

The place-revisiting action set $\mathcal{A}_{\text{pr}}$ is defined following
Algorithm 1. The target locations are chosen by identifying action candidates
that are close to the shortest path while observing occupied cells. These
poses have the potential of triggering SLAM loop closures while avoiding large
deviations from the desired path. First, occupied cells in the occupancy map
are identified and clustered using the k-means algorithm. Afterward, candidate
goals are sampled along the boundary of a circle centered at each cluster
origin with a radius $r_{\text{pr}}$ that is user-specified. We then choose
the candidates that are closest to the shortest path. Finally, the A* search
algorithm is used to find paths from the current position to the final goal,
each path passing through one place-revisiting waypoint.

From the candidate paths $\mathcal{P}$ identified, one is chosen using the
utility function defined in Sec. IV-C. Once the desired path
$\mathcal{X}_{\text{d}}$ is identified, and the vehicle arrives at the place-
revisiting or shortest-distance target location, the planning strategy is
repeated until the robot arrives at its final goal.

###  IV-C Utility evaluation

We define the utility function used to choose the desired path
$\mathcal{X}_{\text{d}}=\arg\max_{X\in\mathcal{P}}U(X_{T:T+N})$ as follows:

$\displaystyle U(X_{T:T+N})=$
$\displaystyle-\sum^{n}_{i=1}\log\det(\Sigma_{\mathbf{v}_{i}})-\alpha
d(X_{T:T+N}).$ (9)

The uncertainty metric used here is the sum of the log-determinant of the
covariance matrices of the virtual map cells $\Sigma_{\mathbf{v}_{i}}$ that
will be encountered by the chosen path. The entire path from the current
position to the target position is considered. In addition to the uncertainty
term, a distance cost weighted by $\alpha$ is added to encourage a trade-off
between traveling and uncertainty reduction. A path will be discarded and
recomputed if it is found to be blocked by an obstacle.

###  IV-D Virtual Map Adaptation

In contrast to the original implementation of virtual maps, we compute virtual
map values only once at the start of a planning run. First, the SLAM graph and
occupancy map information from a prior SLAM exploration run are loaded. Then,
the past trajectory information $\mathcal{X}_{\text{old}}$ is used to estimate
virtual map covariances $\Sigma_{\mathbf{v}_{i}}$. Even though we can use the
same map resolution for path planning and uncertainty estimation, it was shown
in [21] that a lower resolution virtual map provides similar performance but
permits faster belief propagation. In the interest of computational efficiency
and fair experimental comparisons, we discretize the virtual map to be as
coarse as possible, while still capturing the essential geometric features of
the robot’s workspace.

##  V Experiments and Results

###  V-A Experiment Setup

To validate the proposed planning under uncertainty framework, we present
experimental results from a high-fidelity AUV simulation operating at a fixed
depth in different environments. The simulation setup ran on a Lambda
Workstation with an AMD 3970X 32-core processing unit and 256GB RAM, running
Ubuntu 18.04, ROS Melodic, and Gazebo (our planning implementation is
serialized and every trial used only a single core). The vehicle used in the
simulations is the RexROV2 from UUV Simulator [32]. The simulated sonar
operates at 5 Hz, has 30 meter range, and has a horizontal aperture of
$\theta=[-65^{\circ},65^{\circ}]$. Zero-mean Gaussian noise is added to range
and bearing sonar measurements: $\sigma_{r}=0.2$ m, $\sigma_{\theta}=0.02$
rad. Linear velocity values are obtained from a simulated Doppler Velocity Log
(DVL) and angular measurements are obtained from a simulated Inertial
Measurement Unit (IMU) at 50 Hz and 7 Hz respectively. Likewise, zero-mean
Gaussian noise is added with $\sigma_{x}=\sigma_{y}=0.1$ m/s and
$\sigma_{\theta}=0.015$ rad.

(a) Marina environment.

(b) Fish farm environment.

(c) Bridge-tunnel environment.

Figure 3: Virtual maps for all three simulation environments: including
planning start (blue star) and all goal positions (numbered red x’s).

Figure 4: Marina results: goals 1, 2, 3 shown from top to bottom.

Three environments were designed to simulate realistic large-scale scenarios
where an AUV might operate. The first environment, a marina, was inspired by
Penn’s Landing in Philadelphia, the second scenario is an offshore fish farm
containing six cylindrical fish pens, and the third scenario is a bridge-
tunnel system, inspired by the Chesapeake Bay bridge-tunnel. As a
preprocessing step, each environment was fully explored while performing sonar
SLAM as presented in Sec. IV-A [23]. Our AUV was teleoperated inside its
simulated environment to efficiently complete the exploration task, during
which a globally deformable occupancy grid map was constructed based on local
submaps [33]. This representation anchors a local submap to each SLAM pose
keyframe, allowing for efficient map recomputation when a trajectory segment
is changed. The trajectories used for each exploration can be observed in the
video attachment. The SLAM graph and occupancy map information were saved and
later reloaded to build the virtual map and use our planner. The occupancy
grid map was downsampled from 0.2m to 2m in cell size to produce the virtual
map.

Each scenario has one start location and three different goal locations. Each
run consisted of the robot navigating from a start to a goal location, and was
repeated for 100 trials. The results shown are the average values of all the
trials executed. Additionally, we compare our planner framework with two other
strategies, Shortest Distance (SD) and Expectation-Maximization (EM). The SD
planner just follows the shortest viable A* path to the goal. The EM planner
uses the same receding horizon style from Sec. IV-B, however, it employs full
Expectation-Maximization belief propagation as in [23] to choose each
intermediate goal.

For our experiments, we chose to have one shortest-distance action
$N_{\text{sd}}=1$ at $d_{\text{sd}}=25$ m distance, and five place revisiting
actions $N_{\text{pr}}=5$ each iteration. The place revisiting radius was
defined at $r_{\text{pr}}=15$ m. The distance weight was experimentally chosen
and set to $\alpha=3$. To evaluate our planning framework, we recorded the
pose uncertainty of the current robot pose, computed as
$\text{det}(\Sigma_{\mathbf{x}_{i}})^{\frac{1}{3}}$, and the pose error,
computed via RMSE, across all trials as functions of distance traveled.

###  V-B Results Comparison

In our results, SD refers to the Shortest Distance planner, EM refers to the
Expectation-Maximization planner, and VM refers to our proposed planner, which
uses a virtual map as a costmap to approximate belief propagation.

####  V-B1 Marina Environment

The virtual map for the marina environment can be observed in Fig. 3(a), where
the blue star represents the start location, and the red crosses depict goal
locations. Pose error and uncertainty results for each goal location can be
observed in Fig. 4. VM and EM planners achieve a similarly low uncertainty
performance, while SD uncertainty grows unbounded. Goal 3 results in Fig. 4
(bottom) show VM and EM manage to reduce the initial robot pose error, while
SD does not. Surprisingly, goal 1 results in Fig. 4 (top) reveal VM achieves a
more stable low pose error performance than EM, likely due to EM having an
erroneous loop closure. Goal 2 results in Fig. 4 (middle) convey that EM has
the best pose error result, while VM has intermediate performance, still
better than SD. Here, VM likely missed an opportunity to lower pose error by
not predicting and achieving a SLAM loop closure.

####  V-B2 Fish Farm Environment

The virtual map for the fish farm environment with start (blue star) and goal
locations (red cross) can be seen in Fig. 3(b), and corresponding results can
be observed in Fig. 5. Resembling the previous experiment, VM and EM achieve a
similarly low uncertainty performance while SD uncertainty has a higher and
less stable uncertainty result. Fig. 5 (top) shows that for goal 1 the SD
planner manages to reduce its pose error, before pose error spikes up again.
Likewise, Fig. 5 (bottom) shows that for goal 3 the SD planner manages to
lower the pose error after a spike, potentially due to a coincidental loop
closure. However, EM and VM still achieve lower pose error than SD in all
three scenarios, with EM being slightly superior.

Figure 5: Fish farm results: goals 1, 2, 3 shown from top to bottom.

####  V-B3 Bridge-tunnel Environment

The virtual map and relevant locations for the bridge-tunnel environment are
shown in Fig. 3(c), and results are shown in Fig. 6. For all three targets, VM
performance is similar to EM, while SD uncertainty grows unbounded and
exhibits higher pose estimation errors.

Figure 6: Bridge-tunnel results: goals 1, 2, 3 shown from top to bottom.

####  V-B4 Result Summary

Fig. 1 shows example paths achieved by each planner, where the ellipses
represent the evolution of uncertainty. It is clear that the VM and EM paths
are similar in error and uncertainty; they both keep close to features in the
environment and achieve low uncertainty by triggering SLAM loop closures. VM
and EM perform drastically better lowering uncertainty than the SD planner.
Lastly, Table I shows the amount of time it takes to evaluate each candidate
action and choose the appropriate desired path. In the SD planner, only one
path is assessed and chosen. In our proposed VM planner and in the EM planner,
6 candidate actions are assessed each iteration. The results show the average
time of every utility evaluation performed in all 100 trials. As expected, SD
planner has the fastest computation time while EM has the slowest. It can be
seen that EM takes 3 orders of magnitude more time than VM, while VM and SD
maintain the same order of magnitude.

Computation Time (sec)
---
| EM | VM | SD
| ($\mu$) | ($\sigma$) | ($\mu$) | ($\sigma$) | ($\mu$) | ($\sigma$)
Marina 1 | 3.3553 | 0.1446 | 0.0041 | 0.0001 | 0.0015 | 0.0002
Marina 2 | 2.9324 | 0.0598 | 0.0039 | 0.0002 | 0.0014 | 0.0002
Marina 3 | 3.1507 | 0.1396 | 0.0040 | 0.0001 | 0.0015 | 0.0004
Fish Farm 1 | 5.4149 | 0.2094 | 0.0057 | 0.0001 | 0.0023 | 0.0001
Fish Farm 2 | 4.9547 | 0.1768 | 0.0054 | 0.0001 | 0.0021 | 0.0001
Fish Farm 3 | 5.4370 | 0.2138 | 0.0057 | 0.0001 | 0.0023 | 0.0001
Bridge-Tunnel 1 | 6.2699 | 0.0744 | 0.0069 | 0.0001 | 0.0027 | 0.0001
Bridge-Tunnel 2 | 6.4998 | 0.1647 | 0.0070 | 0.0001 | 0.0027 | 0.0001
Bridge-Tunnel 3 | 6.1624 | 0.1358 | 0.0067 | 0.0001 | 0.0026 | 0.0002
TABLE I: Computation time comparison: we give the mean and standard deviation
of the utility evaluation time required in each experiment.

##  VI Conclusion

This paper presents a computationally efficient planning under uncertainty
framework for underwater robots operating in large-scale feature-sparse
environments. We adapt the concept of virtual maps, a product of a robot’s
initial exploration of a new environment, using them as costmaps to avoid
expensive online belief propagation while planning. A receding horizon motion
planning strategy is implemented for planning to specified goals in a manner
that is compatible with real AUV operations. The advantages of the proposed
framework are demonstrated in a realistic underwater simulation. Results show
a decrease in uncertainty and pose error compared to a standard shortest-
distance approach. Furthermore, our approach is much faster than full belief
propagation, while still maintaining low uncertainty and pose error. We aim
for future work to extend this strategy to 3D planning scenarios, and to
perform virtual map updates to accommodate important changes in the
environment.

## References

  * [1] N. Palomeras, N. Hurtós, E. Vidal, and M. Carreras, “Autonomous exploration of complex underwater environments using a probabilistic next-best-view planner,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1619–1625, 2019.
  * [2] E. Pairet, J. D. Hernández, M. Carreras, Y. Petillot, and M. Lahijanian, “Online mapping and motion planning under uncertainty for safe navigation in unknown environments,” _IEEE Transactions on Automation Science and Engineering_ , vol. 19, no. 4, pp. 3356–3378, 2022.
  * [3] W. Burgard, D. Fox, and S. Thrun, “Active mobile robot localization,” _International Joint Conference on Artificial Intelligence_ , 1997.
  * [4] C. Papachristos, S. Khattak, and K. Alexis, “Uncertainty-aware receding horizon exploration and mapping using aerial robots,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 4568–4575, 2017\.
  * [5] G. Costante, C. Forster, J. Delmerico, P. Valigi, and D. Scaramuzza, “Perception-aware path planning,” _arXiv preprint arXiv:1605.04151_ , 2016\.
  * [6] Z. Zhang and D. Scaramuzza, “Perception-aware receding horizon navigation for MAVs,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2534–2541, 2018.
  * [7] S. Suresh, P. Sodhi, J. G. Mangelson, D. Wettergreen, and M. Kaess, “Active SLAM using 3D submap saliency for underwater volumetric exploration,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3132–3138, 2020.
  * [8] I. Maurović, M. Seder, K. Lenac, and I. Petrović, “Path planning for active SLAM based on the D* algorithm with negative edge weights,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 48, no. 8, pp. 1321–1331, 2018.
  * [9] F. Andrade, M. Llofriu, M. Marzoa Tanco, G. Trinidad Barnech, and G. Tejera, “Active localization strategy for hypotheses pruning in challenging environments,” _Journal of Intelligent & Robotic Systems_, 2022.
  * [10] J. A. Placed, J. Strader, H. Carrillo, N. Atanasov, V. Indelman, L. Carlone, and J. A. Castellanos, “A survey on active simultaneous localization and mapping: State of the art and new frontiers,” _IEEE Transactions on Robotics_ , vol. 39, no. 3, pp. 1686–1705, 2023.
  * [11] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,” _Artificial Intelligence_ , vol. 101, no. 1, pp. 99–134, 1998.
  * [12] S. Prentice and N. Roy, “The belief roadmap: Efficient planning in belief space by factoring the covariance,” _The International Journal of Robotics Research_ , vol. 28, pp. 1448 – 1465, 2009.
  * [13] A. Bry and N. Roy, “Rapidly-exploring random belief trees for motion planning under uncertainty,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 723–730, 2011.
  * [14] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” _The International Journal of Robotics Research_ , vol. 30, no. 7, pp. 846–894, 2011.
  * [15] M. Nealson, Erik Corah and N. Michael, “Environment model adaptation for mobile robot exploration,” _Autonomous Robots_ , vol. 42, 2018.
  * [16] F. Andrade, M. Llofriu, M. M. Tanco, G. T. Barnech, and G. Tejera, “Active localization for mobile service robots in symmetrical and open environments,” _Latin American Robotics Symposium (LARS)_ , pp. 270–275, 2021.
  * [17] N. Roy, W. Burgard, D. Fox, and S. Thrun, “Coastal navigation-mobile robot navigation with uncertainty in dynamic environments,” _IEEE International Conference on Robotics and Automation (ICRA)_ , vol. 1, pp. 35–40 vol.1, 1999.
  * [18] Z. Liu, W. Chen, Y. Wang, and J. Wang, “Localizability estimation for mobile robots based on probabilistic grid map and its applications to localization,” _IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)_ , pp. 46–51, 2012.
  * [19] H. Inoue, M. Ono, S. Tamaki, and S. Adachi, “Active localization for planetary rovers,” _IEEE Aerospace Conference_ , pp. 1–7, 2016.
  * [20] V. Olga and S. Cyrill, “Improving SLAM by exploiting building information from publicly available maps and localization priors,” _Journal of Photogrammetry, Remote Sensing and Geoinformation Science_ , vol. 85, 2017.
  * [21] J. Wang and B. Englot, “Autonomous exploration with expectation-maximization,” _International Symposium on Robotics Research (ISRR)_ , pp. 759–774, 2017.
  * [22] J. Wang, T. Shan, and B. Englot, “Virtual maps for autonomous exploration with pose SLAM,” _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 4899–4906, 2019.
  * [23] J. Wang, F. Chen, Y. Huang, J. McConnell, T. Shan, and B. Englot, “Virtual maps for autonomous exploration of cluttered underwater environments,” _IEEE Journal of Oceanic Engineering_ , vol. 47, no. 4, pp. 916–935, 2022\.
  * [24] G. Grisetti, R. Kümmerle, C. Stachniss, and W. Burgard, “A tutorial on graph-based SLAM,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 2, no. 4, pp. 31–43, 2010.
  * [25] M. Richards, “Fundamentals of radar signal processing,” _McGraw-Hill Education_ , 2005.
  * [26] P. Besl and N. D. McKay, “A method for registration of 3-D shapes,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 14, no. 2, pp. 239–256, 1992.
  * [27] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” _Readings in Computer Vision_ , pp. 726–740, 1987.
  * [28] J. G. Mangelson, D. Dominic, R. M. Eustice, and R. Vasudevan, “Pairwise consistent measurement set maximization for robust multi-robot map merging,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2916–2923, 2018.
  * [29] F. Dellaert, “Factor graphs and GTSAM: A hands-on introduction,” _Georgia Institute of Technology_ , Tech. Rep. GT-RIM-CP&R-2012-002, 2012\.
  * [30] M. Kaess, H. Johannsson, R. Roberts, V. Ila, J. J. Leonard, and F. Dellaert, “iSAM2: Incremental smoothing and mapping using the bayes tree,” _The International Journal of Robotics Research_ , vol. 31, no. 2, pp. 216–235, 2012.
  * [31] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” _IEEE Transactions on Systems Science and Cybernetics_ , vol. 4, no. 2, pp. 100–107, 1968.
  * [32] M. M. M. Manhães, S. A. Scherer, M. Voss, L. R. Douat, and T. Rauschenbach, “UUV simulator: A gazebo-based package for underwater intervention and multi-robot simulation,” _OCEANS MTS/IEEE Monterey_ , pp. 1–8, 2016.
  * [33] B.-J. Ho, P. Sodhi, P. Teixeira, M. Hsiao, T. Kusnur, and M. Kaess, “Virtual occupancy grid map for submap-based pose graph SLAM and planning in 3D environments,” _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 2175–2182, 2018.


# Diffusion Models Are Innate One-Step Generators

Bowen Zheng
Institute of Neuroscience
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology
Center for Excellence in Brain Science and Intelligence Technology
Chinese Academy of Sciences
Shanghai, China
<EMAIL_ADDRESS>
&Tianming Yang
Institute of Neuroscience
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology
Center for Excellence in Brain Science and Intelligence Technology
Chinese Academy of Sciences
Shanghai, China
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Diffusion Models (DMs) have achieved great success in image generation and
other fields. By fine sampling through the trajectory defined by the SDE/ODE
solver based on a well-trained score model, DMs can generate remarkable high-
quality results. However, this precise sampling often requires multiple steps
and is computationally demanding. To address this problem, instance-based
distillation methods have been proposed to distill a one-step generator from a
DM by having a simpler student model mimic a more complex teacher model. Yet,
our research reveals an inherent limitations in these methods: the teacher
model, with more steps and more parameters, occupies different local minima
compared to the student model, leading to suboptimal performance when the
student model attempts to replicate the teacher. To avoid this problem, we
introduce a novel distributional distillation method, which uses an exclusive
distributional loss. This method exceeds state-of-the-art (SOTA) results while
requiring significantly fewer training images. Additionally, we show that DMs’
layers are activated differently at different time steps, leading to an
inherent capability to generate images in a single step. Freezing most of the
convolutional layers in a DM during distributional distillation leads to
further performance improvements. Our method achieves the SOTA results on
CIFAR-10 (FID 1.54), AFHQv2 64x64 (FID 1.23), FFHQ 64x64 (FID 0.85) and
ImageNet 64x64 (FID 1.16) with great efficiency. Most of those results are
obtained with only 5 million training images within 6 hours on 8 A100 GPUs.
This breakthrough not only enhances the understanding of efficient image
generation models but also offers a scalable framework for advancing the state
of the art in various applications.

##  1 Introduction

Figure 1: Proposed method and comparison of other methods on ImageNet 64x64.
A. Comparison on ImageNet 64x64. Our methods achieve SOTA results with fewer
training data. B. Proposed method. We initialize generator with a pre-trained
diffusion model and freeze most of the convolutional layers, then adopt
exclusively a distributional level loss.

Diffusion Models (DMs, Song & Ermon (2020a),Song & Ermon (2020b),Ho et al.
(2020)) have led to an explosion of generative vision models. By sampling
through the trajectory defined by the Probability Flow Ordinary Differential
Equation (PF ODE) or Stochastic Differential Equation (SDE) solvers (Song et
al. (2021)), DMs can generate high-quality images and videos, surpassing many
traditional methods, including Generative Adversarial Network (GAN, Goodfellow
et al. (2014)). However, to produce high-quality results, DMs require multiple
sample steps to generate an accurate trajectory, which is computationally
expensive. To alleviate this problem, a series of distillation methods
(Salimans & Ho (2022), Song et al. (2023), Kim et al. (2024)) have been
proposed to distill a one-step generative model. However, these methods often
require substantial computational resources, yet produce poorer performance
than original models.

In this work, we first demonstrate that the underlying limitation of these
instance-based distillation methods is due to the fact that the local minima
for the teacher model and the student model can be very different. We then
introduce a distributional distillation method based on GAN loss. Our method
is highly efficient, reaching state-of-the-art (SOTA) results with only 5
million training images. In comparison, instance-based distillation methods
often require hundreds of millions of training images.

Furthermore, we explore the mechanism underlying the efficient distillation of
our method. Our analyses indicate that the layers in DMs are differentially
activated at different time steps, suggesting that DMs inherently possess the
capability to generate images in a single step. Freezing most of the layers in
a diffusion model leads to even better distillation results.

In summary, our approach achieves SOTA results by a significant margin with
fewer training images (Figure 1). Our findings not only introduce a novel
approach that achieves SOTA results with drastically reduced computational
cost, but also offer valuable insights for further investigations of diffusion
distillation.

##  2 Limitation of Instance-based Distillation

###  2.1 Instance-based Distillation

Given a pre-trained score model $\mathbf{g}_{\phi}(x_{t_{i}},t_{i})$, where
$x_{t_{i}}\sim\mathcal{N}(x_{0},t_{i}^{2}I)$ is a perturbed data point,
$x_{0}$ is the clean image, $t_{i}$ is the standard deviation of the perturbed
distribution with a Gaussian kernel and sampled from a time-step scheduler
function $T(i,N)$, $N$ is total discrete steps, and $i$ is index of current
sample step, we can define an ODE solver $\mathbf{Solver_{\phi}}$:

$\displaystyle\mathbf{Solver}_{\phi}(x_{t_{i}},t_{i},t_{i-1})=\frac{t_{i}-t_{i-1}}{t_{i}}(\mathbf{g}_{\phi}(x_{t_{i}},t_{i})-x_{t_{i}})+x_{t_{i}}$
(1)

There are many different types of $\mathbf{Solver}_{\phi}$, e.g., Heun solver
introduced in EDM (Karras et al. (2022)). We use Euler solver as an example
here. Then we can sample iteratively with this solver to get the final
results.

An instance-based distillation typically has a series of teacher models
$\mathbf{H}$ and student models $\mathbf{F}_{\theta}$. These teacher models
often have more steps than student models. For example, in Progressive
Distillation (PD, Salimans & Ho (2022)), teacher models are defined as

$\mathbf{H}=\begin{cases}\mathbf{Sovler}_{\phi}(\mathbf{Solver}_{\phi}((x_{t_{i}},t_{i},t_{j}),t_{j},t_{k}),&i-j=1,j-k=1\\\
\mathbf{F}^{sg}_{\theta}(\mathbf{F}^{sg}_{\theta}((x_{t_{i}},t_{i},t_{j}),t_{j},t_{k}),&i-j\neq
1,j-k\neq 1\end{cases}$ (2)

where $sg$ is the stop gradient to prevent gradient leak to the teacher
models. More detailed introduction to other instance-based distillation
methods can be found in Appendix B.1.

In these instance-based distillation methods, a teacher model $\mathbf{H}$
contains more parameters than the student and has to forward through the
neural network multiple times, while the student model $\mathbf{F}_{\theta}$
has less parameters and passes through the neural network in a single
iteration.

###  2.2 Inconsistent local minimum

Figure 2: rel-FIDs show strong correlation with the steps and sigmas. A. rel-
FIDs (blue) and abs-FIDs (grey) for teacher models with different steps. B.
The abs-FIDs (grey) and rel-FIDs (blue) for two-step teacher models with
different sigma of intermediate point. FIDs are computed from 50,000 images
generated by each model with fixed seeds from 0 to 49,999, and an inception
model is used to generate features required for calculating FID.

The difference between the teacher and the student leads to a significant
inductive bias: the teacher model can transform between pixel space and latent
space several times while the student model can only perform the
transformation once. We speculate this may produce different optimization
landscapes and different local minima between the teacher and student model.
Forcing the student model to approximate the teacher model’s results may,
therefore, yield sub-optimal results.

To verify our hypothesis, we use the Fréchet Inception Distance (FID, Heusel
et al. (2018)) to compare between two image sets. We term the FID computed
between the teacher and the student model as relative FID (rel-FID), and the
FID computed against the training-set images as absolute FID (abs-FID).

We define a series of teacher models with 2, 4, 8, 10 steps with PD manner.
For example, two-step teacher model is parameterized as
$H=F_{\theta}(F_{\theta}(x_{t_{2}},t_{2},t_{1}),t_{1},t_{0}))$ where
$t_{i}\in\\{T(2,2),T(1,2),T(0,2)\\}$ , and so on. We train the student model
and those teacher models with GAN loss alone. Several training techniques are
used (See Appendix B.3 for more details). Finally, we compute the FID (rel-
FID) between the student model and each teacher model following the evaluation
protocol in EDM (Karras et al. (2022)). Figure 2 A. shows the abs-FIDs and
rel-FIDs for the teacher models with different numbers of steps. While abs-FID
of those teacher models are similar, the teacher models with more steps have
larger rel-FID, suggesting a larger difference compared to the student model.
When we fix the teacher models’ steps to two and adjust the sigma of the
intermediate time step, the rel-FID decreases (Figure 2 B.). Teacher models
with higher sigmas produce images more similar to those generated by the one-
step student model.

These results reveal that the teacher and student models may achieve similar
performance, measured by abs-FID, but in different ways. Teachers with fewer
steps are more similar to the one-step student, but there are still a
significant difference between them (rel-FID=1.78 for a two-step teacher). The
student model fails to mimic the teacher model at the instance level which
might be the reason that limits the performance of instance-based distillation
methods.

##  3 Distilling DMs with Distributional loss

To circumvent the limitations in the instance-based distillation methods, we
can supervise the student model with a distribution-level loss instead of
asking the student model to mimic the teacher at the instance level.

GAN loss is one of the most commonly used distribution-level loss. Distilling
DMs with GAN loss have been introduced in previous work for different
purposes. GAN loss is applied as an auxiliary loss after the instance-based
distillation to improve its performance in Consistency Trajectory Model (CTM,
Kim et al. (2024)) as originally introduced in Vector-Quantized GAN (VQ-GAN,
Esser et al. (2021)). SDXL-Lightning (Lin et al. (2024)) uses GAN loss to
replace the deterministic instance-level loss (L2 or LPIPS) with a probability
form, but the discriminator still receives instance-level information about
the time step and the beginning point of the trajectory to ensure that the
student follows the original trajectory. Distribution Matching Distillation
(DMD, Yin et al. (2023)) adopts another type of distributional distillation
method, which uses a distribution matching gradient instead of GAN loss. But
again, an instance-level pair-matching loss is added alongside the GAN loss.
Although these works do not use distributional loss alone, they demonstrate
the potential of distributional-loss based distillation.

###  3.1 GAN Distillation at Distribution level

For distilling a one-step generator from DMs, we introduce a new method called
GDD (GAN Distillation at Distribution Level) , which uses only a distribution-
level loss WITHOUT any instance-level supervision. Importantly, unlike in
SDXL-Lightning or DMD, the discriminator in our method receives real data
instead of the data generated by the teacher model as the ground truth. We use
non-saturating GAN objective defined as:

$\displaystyle\max_{\mathbf{D}}~{}\mathbb{E}_{\mathbf{x}}[\log(\mathbf{D}(\mathbf{x}))]+\mathbb{E}_{\mathbf{z}}[1-\log(\mathbf{D}(\mathbf{G}_{\theta}(\mathbf{z})))]$
(3)
$\displaystyle\min_{\mathbf{G_{\theta}}}\mathbb{E}_{\mathbf{z}}[-\log(\mathbf{D}(\mathbf{G}_{\theta}(\mathbf{z})))]$

With a pre-trained diffusion model, the score model $\mathbf{g_{\phi}}$ is
used as the generator $\mathbf{G}_{\theta}$ and a pre-trained VGG16 (Simonyan
& Zisserman (2015)) is used as a discriminator by default.

###  3.2 Experiments

Discriminator |  FID ($\downarrow$)
---|---
StyleGAN2 (Scratch) |  collapse
StyleGAN2 (Pre-trained) |  collapse
VGG16 |  4.04
LPIPS |  3.49
PG |  2.21
(a)

Augmentation |  $\gamma_{r1}$ |  FID ($\downarrow$)
---|---|---
✓ |  — |  2.21
✗ |  — |  1.98
✗ |  1e-3 |  1.77
✗ |  1e-4 |  1.66
✗ |  1e-5 |  1.66

(b)

Table 1: GDD with different discriminators, data augmentations, and
regularizations on CIFAR-10. EDM with NCSN++ is used as the generator. Results
are based on 5 million training images.

#### Baseline.

We perform our experiments on CIFAR-10 (Krizhevsky (2009)) as the baseline.
EDM with NCSN++ (Song et al. (2021)) architecture is used as the generator. We
choose VGG16 (Simonyan & Zisserman (2015)) as the baseline discriminator.
Adaptive augmentation (ADA, Karras et al. (2020a)) is adopted with r1
regularization (Mescheder et al. (2018)) where $\gamma_{r1}=0.01$ as suggested
in StyleGAN2-ADA (Karras et al. (2020a)). Our baseline setting shows good
performance on CIFAR-10 dataset with only 5M training images and already
achieves performance (FID=4.04) close to that of the Consistency Distillation
(FID=3.55, Song et al. (2023)).

#### Discriminator.

An appropriate discriminator is crucial since it represents the overall target
distribution and determines the performance of the generator. We compare the
StyleGAN2 (Both scratch and pre-trained with GAN objective, Karras et al.
(2020b)), VGG, LPIPS (Zhang et al. (2018)) and Projected GAN (PG, Sauer et al.
(2022)) discriminators. Adaptive augmentation (ADA) is adopted on StyleGAN2,
VGG and LPIPS discriminators with r1 regularization where $\gamma_{r1}=0.01$.

For the PG Discriminator, we use a non-saturating version of PG objective:

$\displaystyle\max_{\\{\mathbf{D}_{l}\\}}\sum_{l\in\mathcal{L}}\Big{(}\mathbb{E}_{\mathbf{x}}[\log\mathbf{D}_{l}(\mathbf{P}_{l}(\mathbf{x}))]+\mathbb{E}_{\mathbf{z}}[1-\log(\mathbf{D}_{l}(\mathbf{P}_{l}(\mathbf{G}_{\theta}(\mathbf{z}))))]\Big{)}$
(4)
$\displaystyle\min_{\mathbf{G_{\theta}}}\sum_{l\in\mathcal{L}}\Big{(}\mathbb{E}_{\mathbf{z}}[-\log(\mathbf{D}_{l}(\mathbf{P}_{l}(\mathbf{G}_{\theta}(\mathbf{z}))))\Big{)}$

We use VGG16 with batch normalization (VGG16-BN, Simonyan & Zisserman (2015),
Ioffe & Szegedy (2015)) and EfficientNet-lite0 (Tan & Le (2020)) as feature
networks $\mathbf{P}$. We adopt differentiable augmentation (diffAUG, Zhao et
al. (2020)) without gradient penalty by default, following original work.

As illustrated in Table LABEL:tab:discriminator, discriminators with multiple
scales are better than the vanilla VGG discriminator. Overall, the PG
discriminator with a fusion feature based on VGG16-BN and EfficientNet-lite0
achieves the best results. Therefore, we will use the PG discriminator and
objective in the following experiments.

#### Augmentation and Regularization.

Differentiable augmentation (diffAUG) was introduced in Zhao et al. (2020) to
prevent the overfitting of discriminators, but we find that it leads to poor
results in our method (Table LABEL:tab:aug_reg). This may be because EDMs are
pre-trained with data augmentation. We disable all augmentation in all of our
further experiments. In the original PG, regularization for the discriminator
was not used. However, our experiments show that it is important to apply r1
regularization on the discriminator to prevent overfitting and it improves
performance significantly (Table LABEL:tab:aug_reg). The value of
$\gamma_{r1}$ should be kept small since the discriminator of the Projected
GAN generates multiple output logits, and a large $\gamma_{r1}$ may lead to a
numerical explosion. With optimal settings, GDD reaches the SOTA results
(FID=1.66) with only 5 million training images.

##  4 The Innate Ability of DMs for One-Step Generation

We have demonstrated that GDD is both efficient and produces high-quality
results, but we wonder why it can learn to generate images in one step so
rapidly. Training DMs with instance-based distillation often requires hundreds
of millions of images, while a few million images are enough for GDD.

We hypothesize that DM layers may handle the tasks in the diffusion objective
relatively independently at different time steps. Finding a way to activate
them together may let a DM achieve what normally requires many steps in one
step. If so, we should observe that layers in diffusion models are
differentially activated across time steps. Moreover, we can freeze most of
the layers in DMs but train the model to learn how to use their innate
capabilities to produce good results.

###  4.1 Differential Activations across time steps

Figure 3: The relative value of the last convolutional layer in each UNet
block at different depth. Values are normalized with min-max normalization
across time steps for each layer. Red-to-blue color gradient indicates shallow
to deeper layers. A. Encoder. B. Decoder.

To analyze the activation of the inner layers, we record the outputs of each
UNet block’s last convolutional layer during sampling. To obtain more accurate
and comprehensive data, we set the total number of time steps to 256
($N=256$), where $i=256$ is the beginning step with $T(256,N)=\sigma_{max}$
and $i=0$ is the end step with $T(0,N)=\sigma_{min}$. Outputs are grouped by
time steps and depths first, then min-max normalization are applied across
time steps as
$\hat{O}_{t}^{l}=\frac{O_{t}^{l}-\min(O^{l})}{\max(O^{l})-\min(O^{l})}$. We
find that the convolutional layers are differentially activated at different
time steps except for the two periods in the beginning time steps at around
$\sigma=40$ (time step=225) and $\sigma=60$ (time step=243), where almost all
layers are activated. Other than these two periods, the shallow layers are
maximally activated at the end time steps, while the deep layers are more
sensitive to the beginning time steps. The middle layers are in between (Fig
3). These trends can be seen in both the encoder and the decoder, except that
several shallow layers are sensitive to the beginning time steps.

###  4.2 Achieving One-Step Generation via Freezing

The DM layers are differentially activated at different time steps. If we can
find a way to activate them simultaneously for one-step generations, we can
utilize what has already learned in these layers, their innate capabilities,
and distill a DM quickly. As the group normalization layers (both the adaptive
and the vanilla group normalizations) serve as the triggers of the
convolutional layers, we may only need to tune them during the distillation.
To test this idea, we examine the model’s performance on CIFAR-10 with most of
the convolutional layers frozen, while allowing the group normalization
layers, the input layer and the output layer (including the extra residual
down sample layer when using NCSN++) to be tuned.

Norm Layers (7.9%) |  Conv Layers (85.8%) |  QKV Projections (2.1%) |  Skip Layers (4.0%) |  FID ($\downarrow$)
---|---|---|---|---
✓ |  ✓ |  ✓ |  ✓ |  1.66
✓ |  ✗ |  ✓ |  ✓ |  1.54
✓ |  ✗ |  ✗ |  ✓ |  1.60
✓ |  ✗ |  ✗ |  ✗ |  2.51
Table 2: GDD-I: ablation experiments. ✓ indicates the corresponding layer can
be tuned and ✗ indicates frozen layers. The percentage in the brackets is the
proportion of the total parameters in these layers.

The results confirm our idea (Table 2). Freezing most of the convolutional
layers in the UNet blocks not just works. It actually produces better
performance. Further freezing the QKV projections (Vaswani et al. (2023))
causes a slight decrease in the performance, and freezing the skip layers on
the residual connections leads to worse results, suggesting that tuning of
these two types of layers is necessary for distillation. These results
strongly support our theory that the ability for one-step generation is
inherent in DMs: with most of the original parameters locked (85.8% of the
parameters contained in the convolutional layers) in the pre-trained score
model, our model achieves results better than the previous SOTA results. We
term the method that freezes most of the convolutional layers as GDD-I (GAN
Distillation at the Distribution level using Innate ability).

###  4.3 Extra Instance Level loss

In addition, we test the model’s performance when extra instance-level loss is
used. We use CD loss as an example since it is easy to implement and performs
well. The comparisons are conducted on CIFAR-10 dataset. We notice that adding
an extra CD loss significantly slows down the converging speed in the
beginning. It reaches slightly better results when compared with GDD. This
advantage disappears when compared against GDD-I (Figure 4 B.). Note that
adding an extra CD loss almost doubles the training resource and time.

We further compare our methods with the original CD and CD with frozen
convolutional layers (CD-I) (Figure 4 A.). CD-I results in poorer performance,
suggesting that forcing the student model to estimate the teacher model
requires adjustments in the convolutional layers. This further proves our
findings in Section 2.2. It is still possible to combine our method with other
instance-based distillation methods (e.g. PD, CTM, SiD (Zhou et al. (2024))).
We leave this investigation for future endeavors.

Figure 4: Comparison and effect of CD loss. A. Comparison of CD and CD-I. B.
Effect of adding an extra CD loss.

###  4.4 Comprehensive Performance

#### Basic Setting.

Finally, we perform a comprehensive test of both GDD and GDD-I on CIFAR-10,
AFHQv2 64x64 (Choi et al. (2020)), FFHQ 64x64 (Karras et al. (2019)), and
ImageNet 64x64 (Deng et al. (2009)). The results for class conditioning
generation are reported on CIFAR-10 and ImageNet 64x64. The pre-trained
diffusion models are from EDM. We use the ADM (Dhariwal & Nichol (2021))
architecture for ImageNet and the NCSN++ architecture for the other datasets.
We use lr=1e-4 for CIFAR-10, lr=2e-5 for AFHQv2 64x64 and FFHQ 64x64, and
lr=8e-6 for ImageNet. Batch size is set to 512 for ImageNet and 256 for all
other datasets. More details can be found in Appendix B.4. We report FID for
all datasets, Inception Score (IS, Salimans et al. (2016)) for CIFAR-10,
precision and recall (Kynkäänniemi et al. (2019)) for ImageNet 64x64. For the
discriminator, we use VGG16 with batch normalization and EfficientNet-lite0 as
the feature network for CIFAR-10, DeiT (Touvron et al. (2021)) and
EfficientNet-lite0 for the other datasets. The optimal settings in Section 3.2
is used.

Method |  NFE ($\downarrow$) |  | Unconditional |  | Conditional
---|---|---|---|---|---
|  |  |  FID ($\downarrow$) |  IS ($\uparrow$) |  |  FID ($\downarrow$)
Direct Generation |  |  |  |  |  |
DDPM (Ho et al. (2020)) |  1000 |  |  3.17 |  |  |
DDIM (Song et al. (2022) |  100 |  |  4.16 |  |  |
Score SDE (Song et al. (2021)) |  2000 |  |  |  |  |  2.20
DPM-Solve-3 (Lu et al. (2022)) |  48 |  |  |  |  |  2.65
EDM (Karras et al. (2022)) |  35 |  |  1.98 |  |  |  1.79
BigGAN (Brock et al. (2019)) |  1 |  |  |  |  |  14.73
StyleGAN2-ADA (Karras et al. (2020a)) |  1 |  |  2.92 |  9.82 |  |  2.42
Distillation |  |  |  |  |  |
PD (Salimans & Ho (2022)) |  1 |  |  9.12 |  |  |
DFNO (Zheng et al. (2023)) |  1 |  |  3.78 |  |  |
CD (Song et al. (2023)) |  1 |  |  3.55 |  |  |
CD (Song et al. (2023)) |  2 |  |  2.93 |  |  |
iCT (Song & Dhariwal (2023)) |  1 |  |  2.83 |  9.54 |  |
iCT (Song & Dhariwal (2023)) |  2 |  |  2.46 |  9.80 |  |
iCT-deep (Song & Dhariwal (2023)) |  1 |  |  2.51 |  9.76 |  |
iCT-deep (Song & Dhariwal (2023)) |  2 |  |  2.24 |  9.89 |  |
CTM (Kim et al. (2024)) |  1 |  |  1.98 |  |  |  1.73
CTM (Kim et al. (2024)) |  2 |  |  1.87 |  |  |  1.63
DMD (Yin et al. (2023)) |  1 |  |  2.62 |  |  |
SiD ($\alpha=1$, Zhou et al. (2024)) |  1 |  |  2.02 |  10.01 |  |  1.93
SiD ($\alpha=1.2$, Zhou et al. (2024)) |  1 |  |  1.92 |  9.98 |  |  1.71
GDD (ours) |  1 |  |  1.66 |  10.11 |  |  1.58
GDD-I (ours) |  1 |  |  1.54 |  10.10 |  |  1.44
(a)

Method |  NFE ($\downarrow$) |  FID ($\downarrow$)
---|---|---
Direct Generation |  |
EDM (Karras et al. (2022)) |  79 |  1.96
Distillation |  |
SiD ($\alpha=1.0$, Zhou et al. (2024)) |  1 |  1.62
SiD ($\alpha=1.2$, Zhou et al. (2024)) |  1 |  1.71
GDD (ours) |  1 |  1.23
GDD-I (ours) |  1 |  1.31

(b) AFHQv2 64x64, unconditional

Method |  NFE($\downarrow$) |  FID($\downarrow$)
---|---|---
Direct Generation |  |
EDM (Karras et al. (2022)) |  79 |  2.39
Distillation |  |
BOOT (Gu et al. (2023)) |  1 |  9.0
SiD ($\alpha=1.0$, Zhou et al. (2024)) |  1 |  1.71
SiD ($\alpha=1.2$, Zhou et al. (2024)) |  1 |  1.55
GDD(ours) |  1 |  1.08
GDD-I(ours) |  1 |  0.85
(c) FFHQ 64x64, unconditional

Table 3: Comprehensive comparison between GDD, GDD-I, and previoius work. Tests are done on CIFAR-10, AFHQv2 64x64 and FFHQ 64x64. Method |  NFE ($\downarrow$) |  FID ($\downarrow$) |  Prec. ($\uparrow$) |  Rec. ($\uparrow$)
---|---|---|---|---
Direct Generation |  |  |  |
RIN (Jabri et al. (2023)) |  1000 |  1.23 |  |
DDPM (Ho et al. (2020)) |  250 |  11.00 |  0.67 |  0.58
ADM (Dhariwal & Nichol (2021)) |  250 |  2.07 |  0.74 |  0.63
EDM (Karras et al. (2022)) |  79 |  2.64 |  |
iCT (Song & Dhariwal (2023)) |  1 |  4.02 |  0.70 |  0.63
iCT (Song & Dhariwal (2023)) |  2 |  3.20 |  0.73 |  0.63
iCT-deep (Song & Dhariwal (2023)) |  1 |  3.25 |  0.72 |  0.63
iCT-deep (Song & Dhariwal (2023)) |  2 |  2.77 |  0.74 |  0.62
BigGAN-deep (Brock et al. (2019)) |  1 |  4.06 |  0.79 |  0.48
StyleGAN2-XL (Sauer et al. (2022)) |  1 |  1.51 |  |
Distillation |  |  |  |
PD (Salimans & Ho (2022)) |  1 |  15.39 |  |
BOOT (Gu et al. (2023)) |  1 |  16.3 |  0.68 |  0.36
DFNO (Zheng et al. (2023)) |  1 |  7.83 |  |  0.61
CD (Song et al. (2023)) |  1 |  6.20 |  0.68 |  0.63
CD (Song et al. (2023)) |  2 |  4.70 |  0.69 |  0.64
CTM (Kim et al. (2024)) |  1 |  1.92 |  70.38 |  0.57
CTM (Kim et al. (2024)) |  2 |  1.73 |  64.29 |  0.57
DMD (Yin et al. (2023)) |  1 |  2.62 |  |
SiD ($\alpha$=1, Zhou et al. (2024)) |  1 |  2.02 |  0.73 |  0.63
SiD ($\alpha$=1.2, Zhou et al. (2024)) |  1 |  1.52 |  0.74 |  0.63
GDD(ours) |  1 |  1.42 |  0.77 |  0.59
GDD-I(ours) |  1 |  1.16 |  0.75 |  0.60
Table 4: Comprehensive comparison between GDD, GDD-I, and previous work. Tests
are done on ImageNet 64x64, with class conditioning.

#### Results.

The results can be found in Table 3 and 4. Both GDD and GDD-I achieve the SOTA
results on all datasets that we have tested. Our methods are not only suitable
for unconditional distillation, but also show great performance on class
conditioning generation tasks. Moreover, GDD-I outperforms GDD by a
significant margin in most of the datasets, except AFHQv2 64x64. On CIFAR-10,
our methods achieve the best FID of 1.54 and the best IS of 10.11. Similarly,
superior performance is found on AFHQv2 64x64 (FID=1.23) and FFHQ 64x64
(FID=0.85). In ImageNet 64x64, the model also achieves SOTA performance
(FID=1.16, precision=0.77). Together, these results demonstrate the robustness
and effectiveness of our methods. They support our hypothesis that DMs are
capable of generating results in one-step innately.

##  5 Limitations

While our approach shows great performance and speed, our experiments are
mainly done on datasets with relatively low resolution. Performance on high
resolution needs to be further verified. In Section 4, our experiments show
that DMs are differentially activated at different time steps in UNet
(Ronneberger et al. (2015)) style architectures. However, the specific roles
of these layers in multi-step generation and one-step generation tasks remain
to be explored. Additionally, further experiments are needed to determine
whether this occurs in other architectures as well (e.g. DiT (Peebles & Xie
(2023)). Finally, while our method achieves high sample quality for both
unconditional and class-conditional generation, functions like image-editing
and inpainting are yet to be implemented. A method that integrates the quality
and efficiency of our approach and the diverse capabilities of instance-based
methods remains to be explored.

##  6 Conclusion

In this work, we have identified an inherent limitation in instance-based
distillation methods, stemming from the disparities between the teacher and
the student models. To circumvent this limitation, we first propose a novel
method, GDD (GAN Distillation in Distribution level), which leverages an
exclusive distributional loss to efficiently distill a one-step generator from
diffusion models. We further investigate the activation patterns of diffusion
model layers, and develop GDD-I (GDD using Innate ability), which enhances
performance by freezing most of the convolutional layers in the GDD model
during the distillation process and outperforms the original GDD method. Our
work not only establishes new SOTA results with minimal computational
resources but also elucidates a critical principle in the field of diffusion
distillation, paving the way for more efficient and effective model training
methodologies in image generation and beyond.

## References

  * Brock et al. (2019) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis, 2019.
  * Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.  Maskgit: Masked generative image transformer, 2022.
  * Choi et al. (2020) Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains, 2020.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis, 2021.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * Gu et al. (2023) Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind.  Boot: Data-free distillation of denoising diffusion models with bootstrapping, 2023.
  * Heusel et al. (2018) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
  * Jabri et al. (2023) Allan Jabri, David Fleet, and Ting Chen.  Scalable adaptive computation for iterative generation, 2023.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks, 2019.
  * Karras et al. (2020a) Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data, 2020a.
  * Karras et al. (2020b) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan, 2020b.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models, 2022.
  * Kim et al. (2024) Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon.  Consistency trajectory models: Learning probability flow ode trajectory of diffusion, 2024.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * Kynkäänniemi et al. (2019) Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models, 2019.
  * Li et al. (2023) Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan.  Mage: Masked generative encoder to unify representation learning and image synthesis, 2023.
  * Lin et al. (2024) Shanchuan Lin, Anran Wang, and Xiao Yang.  Sdxl-lightning: Progressive adversarial diffusion distillation, 2024.
  * Lu et al. (2022) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022.
  * Mescheder et al. (2018) Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?, 2018.
  * Nichol et al. (2022) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
  * Peebles & Xie (2023) William Peebles and Saining Xie.  Scalable diffusion models with transformers, 2023.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Salimans & Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models, 2022.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans, 2016.
  * Sauer et al. (2022) Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets, 2022.
  * Simonyan & Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
  * Song et al. (2022) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models, 2022.
  * Song & Dhariwal (2023) Yang Song and Prafulla Dhariwal.  Improved techniques for training consistency models.  _ArXiv_ , abs/2310.14189, 2023.  URL https://api.semanticscholar.org/CorpusID:264426451.
  * Song & Ermon (2020a) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution, 2020a.
  * Song & Ermon (2020b) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models, 2020b.
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations, 2021.
  * Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models, 2023.
  * Tan & Le (2020) Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention, 2021.
  * Vaswani et al. (2023) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * Yin et al. (2023) Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation, 2023.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric, 2018.
  * Zhao et al. (2020) Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training, 2020.
  * Zheng et al. (2023) Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar.  Fast sampling of diffusion models via operator learning, 2023.
  * Zhou et al. (2024) Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang.  Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation, 2024.

##  Appendix A Related Works

#### Image Generation.

Image generation is an important task that has been explored over the years.
GAN(Goodfellow et al. (2014),Brock et al. (2019),Karras et al. (2020b)) models
have dominated the field for a long time. Quantization-Based Generative
Model(Esser et al. (2021), Chang et al. (2022),Li et al. (2023)) first encodes
images to discrete tokens, then use a transformer to model the probability
distribution between those tokens. Diffusion Models(Sohl-Dickstein et al.
(2015), Song & Ermon (2020a), Ho et al. (2020), Song & Ermon (2020b), Song et
al. (2021)) or score-based generative models, tries to learn an accuracy
estimation of scores (the gradient of the log probability density), from a
perturbed distribution with a Gaussian kernel to the image distribution.
Diffusion Models achieve great success especially in image generation(Dhariwal
& Nichol (2021),Nichol et al. (2022), Ramesh et al. (2022),Saharia et al.
(2022)) and are widely used in different fields.

#### Diffusion Distillation.

One of the challenges of diffusion models in practice is the high
computational cost incurred during fine multi-step generation. A series of
distillation methods have been proposed to distill diffusion models to one-
step generators. Some distillation methods (Salimans & Ho (2022),Song et al.
(2023) keep the original features of the diffusion models, such as image
inpainting and image editing. Others (Zhou et al. (2024)) focus on improving
the one-step generation results. Distillation models with GAN were introduced
recently. GAN loss has been used as an auxiliary loss of instance-level
distillation loss (Kim et al. (2024)), or a replacement of instance-level loss
(L2 or LPIPS) as in Lin et al. (2024). Both of these works achieve great
performance, showing the potential of GAN-Based distillation.

##  Appendix B Appendix

###  B.1 Definition of Instance-Based Distillation

Method |  $\mathbf{t}$ |  $\mathbf{u}$ |  $\mathbf{s}$
---|---|---|---
PD |  $T(i,N),2\leq i\leq N$ |  $T(\frac{1}{2}(i+j),N)$ |  $T(j,N),0\leq j<(i-1)$
CD |  $T(i,N),1\leq i\leq N$ |  $T(i-1,N)$ |  $T(0,N)$
CTM |  $T(i,N),2\leq i\leq N$ |  $T(j,N),1\leq j\leq i$ |  $T(k,N),0\leq k\leq j$
(a)

Method |  Student Model ($\mathbf{F}_{\theta}$) |  Teacher Model ($\mathbf{H}$)
---|---|---
PD |  $\displaystyle\frac{t-s}{t}(\mathbf{f}_{\theta}(x_{t},t)-x_{t})+x_{t}$ |  $\Big{\\{}\begin{array}[]{lr}\mathbf{Solver}_{\phi}(\mathbf{Solver}_{\phi}(x_{t},t,u),u,s),&N_{i}=N_{max}\\\ \mathbf{F}^{sg}_{\theta}(\mathbf{F}^{sg}_{\theta}(x_{t},t,u),u,s),&N_{i}\neq N_{max}\end{array}$
CD |  $\displaystyle\frac{t-s}{t}(\mathbf{f}_{\theta}(x_{t},t)-x_{t})+x_{t}$ |  $\mathbf{F}^{sg}_{\theta}(\mathbf{Solver}_{\phi}(x_{t},t,u),u,s)$
CTM |  $\displaystyle\mathbf{f}_{\theta}(x_{t},t,s)$ |  $\mathbf{F}^{sg}_{\theta}(\mathbf{Solver}_{\phi}(x_{t},t,u),u,s)$111CTM use an iterative solver here
(b)

Method |  Loss Function ($\mathcal{L}$)
---|---
PD |  $\mathbf{D}(\mathbf{f}_{\theta}(x_{t},t),\frac{t}{t-s}(\mathbf{H}(x_{t},t,u,s)-x_{t})+x_{t})$
CD |  $\mathbf{D}(\mathbf{F}_{\theta}(x_{t},t,s),\mathbf{H}(x_{t},t,u,s))$
CTM |  $\mathbf{D}(\mathbf{F}^{sg}_{\theta}(\mathbf{F}_{\theta}(x_{t},t,s),s,\sigma_{min}),\mathbf{F}^{sg}_{\theta}(\mathbf{H}(x_{t},t,u,s),s,\sigma_{min}))$
(c)

Table 5: Definition of commonly used instance-level distillation methods

Progressive Distillation (PD, Salimans & Ho (2022)) uses a progressive
distillation strategy. Using $\mathbf{G}_{\phi}$ with a sufficient number of
steps, e.g., $N_{max}=1024$ as a teacher, PD first distills a student model
$\mathbf{F}_{\theta}$ with fewer steps, typically
$N_{1}=\frac{1}{2}N_{max}=512$. Then with the fixed student model
$\mathbf{F}^{sg}_{\theta}$ as teacher where sg means stop gradient, PD further
distills the student with $N_{2}=\frac{1}{4}N_{max}=256$. This progress is
repeated until a one-step generator is produced.

Consistency Distillation (CD, Song et al. (2023)) and Consistency Trajectory
Model (CTM, Kim et al. (2024)) are similar. Both of these two methods adopt a
joint training strategy in which student model $\mathbf{F}_{\theta}$ is used
to be part of the teacher model
$\mathbf{H}(x_{t},t,u,s)=\mathbf{F}^{sg}_{\theta}(\mathbf{G}_{\phi}(x_{t},t,u),u,s)$
at the same time. CTM further projects $v_{s}$ and $\bar{v}_{s}$ to the pixel
space by $\mathbf{F}^{sg}_{\theta}(v_{s},s,\sigma_{min})$ and
$\mathbf{F}^{sg}_{\theta}(\bar{v}_{s},s,\sigma_{min})$ and then calculates
$\mathbf{D}(v_{\sigma_{min}},\bar{v}_{\sigma_{min}})$ as loss.

The three models are summarized in Table 5. We modify the definition of PD
with a EDM manner here to be consistent. $t$, $u$, $s$ are the start,
intermediate and end time steps of the trajectory, respectively. PD sets the
$u$ to the ”middle” of $t$ and $s$. CD sets the $u$ as the successor of $t$
and fixes the $s$ to zeros. PD uses a separate training strategy since the
teacher model can be different with different $N$s, and PD supervises
$f_{\theta}$ with the intersection of $x=0$ and
$\text{Slope}(x_{t},\bar{v}_{s})$, where $\text{Slope}(x_{t},\bar{v}_{s})$ is
the slope between $x_{t}$ and $\bar{v}_{s}$. CTM first translates $v_{s}$ and
$\bar{v}_{s}$ to $v_{\sigma_{min}}$ and $\bar{v}_{\sigma_{min}}$, then
calculates the loss.

###  B.2 Time Step Scheduler

When training with CD loss, we use Heun solver and set the total number of
time steps to 18 ($N=18$) following CD. The time step scheduler is slightly
different from the scheduler defined in EDM (Karras et al. (2022)). We reverse
the index and define it as a function of current step and total steps:

$T(i,N)=\left({\sigma_{max}}^{\frac{1}{\rho}}+\Big{(}1-\frac{i}{N}\Big{)}\left({\sigma_{min}}^{\frac{1}{\rho}}-{\sigma_{max}}^{\frac{1}{\rho}}\right)\right)^{\rho}$
(5)

Following Consistency Model, we set $\rho=7$, $\sigma_{min}=0.002$ and
$\sigma_{max}=80$.

###  B.3 Details for Calculating rel-FID

Both the teacher and the student model are initialized from a well-trained
score model in EDM. To ensure that each model performs well, we adopt the same
optimal setting in Section 3.2 during training. All teacher models reach
FID$\leq$2.2 in a few training steps. The rel-FIDs are then calculated for
these model as described in Section 2.2.

###  B.4 Implementation Details

Table 6: Hyperparameters used for training GDD and GDD-I Hyperparameter | CIFAR-10 | AFHQv2 $64\times 64$ | FFHQ $64\times 64$ | ImageNet $64\times 64$
---|---|---|---|---
G Architecture | NSCN++ | NSCN++ | NSCN++ | ADM
G LR | 1e-4 | 2e-5 | 2e-5 | 8e-6
D LR | 1e-4 | 4e-5 | 4e-5 | 4e-5
Optimizer | Adam | Adam | Adam | Adam
$\beta_{1}$ of Optimizer | 0 | 0 | 0 | 0
$\beta_{2}$ of Optimizer | 0.99 | 0.99 | 0.99 | 0.99
Weight decay | No | No | No | No
Batch size | 256 | 256 | 256 | 512
$\gamma_{r1}$ (Regularization) | 1e-4 | 1e-4 | 1e-4 | 4e-4
EMA half-life (Mimg) | 0.5 | 0.5 | 0.5 | 50
EMA warmup ratio | 0.05 | 0.05 | 0.05 | 0.05
Training Images (Mimg) | 5 (GDD) | 5 (GDD) | 5 (GDD) | 5 (GDD)
5 (GDD-I) | 10 (GDD-I) | 10 (GDD-I) | 5 (GDD-I)
Mixed-precision (BF16) | Yes | Yes | Yes | Yes
Dropout probability | 0.0 | 0.0 | 0.0 | 0.0
Augmentations | No | No | No | No
Number of GPUs | 8 | 8 | 8 | 8

#### Parameterization.

We follow the parameterization in Consistency Model (Song et al. (2023)) for
all of our models:

$\displaystyle
c_{\text{skip}}(t)=\frac{\sigma_{\text{data}}^{2}}{(t-\epsilon)^{2}+\sigma_{\text{data}}^{2}},\quad
c_{\text{out}}(t)=\frac{\sigma_{\text{data}}(t-\epsilon)}{\sqrt{\sigma_{\text{data}}^{2}+t^{2}}}$

We set $t=\sigma_{max}$ when applying distributional distillation with
$c_{\text{skip}}\approx 0$ and $c_{\text{out}}\approx 1$.

#### Training Details.

We distill EDMs for all datasets and methods for fair comparison. EMA decay is
applied to the weights of generator for sampling following previous work. EMA
decay rate is calculated by EMA Halflife and EMA warmup is used as in EDM. EMA
warmup ratio is set to 0.05 for all datasets, and this leads to a gradually
grow EMA decay rate. We use Adam optimizer with $\beta_{1}=0,\beta_{2}=0.99$
without weight decay for both the generator and the discriminator. No gradient
clip is applied. Mixed precision training is adopted for all experiments with
BFloat16 data type, and results with Float16 and Float32 are similar. We apply
no learning rate scheduler. Images are resized to 224 first before they are
fed to the PG discriminator or CD-LPIPS loss. All models are trained on a
cluster of Nvidia A100 GPUs. Hyperparameters used for GDD and GDD-I training
can be found in Table 6. GDD and GDD-I share the same hyperparameters except
for the number of training images, since GDD-I will not overfit when GDD gets
worse FID.

##  Appendix C Samples

We include EDM’s results for comparison. The initial noises are same as
different models. It is clear that GDD and GDD-I generate images similar to
the original model (EDM), but they are not exactly the same since our methods
allow the student model to perform differently with the teacher model.

Figure 6: CIFAR-10, EDM, NFE=18, FID=1.96 Figure 7: CIFAR-10, GDD, NFE=1,
FID=1.66 Figure 8: CIFAR-10, GDD-I, NFE=1, FID=1.54 Figure 9: CIFAR-10
(conditional), EDM (VE), NFE=18, FID=1.82 Figure 10: CIFAR-10 (conditional),
GDD, NFE=1, FID=1.58 Figure 11: CIFAR-10 (conditional), GDD-I, NFE=1, FID=1.44
Figure 12: AFHQv2 64x64, GDD, NFE=79, FID=2.17 Figure 13: AFHQv2 64x64, GDD,
NFE=1, FID=1.23 Figure 14: AFHQv2 64x64, GDD-I, NFE=1, FID=1.31 Figure 15:
FFHQ 64x64, EDM (VE), NFE=79, FID=2.60 Figure 16: FFHQ 64x64, GDD, NFE=1,
FID=1.08 Figure 17: FFHQ 64x64, GDD-I, NFE=1, FID=0.85 Figure 18: ImageNet
64x64 (conditional), EDM (VE), NFE=79, FID=2.36 Figure 19: ImageNet 64x64
(conditional), GDD, NFE=1, FID=1.42 Figure 20: ImageNet 64x64 (conditional),
GDD-I, NFE=1, FID=1.16