# Multi-Sample Long Range Path Planning under Sensing Uncertainty for Off-Road
Autonomous Driving

Matt Schmittle, Rohan Baijal, Brian Hou, Siddhartha Srinivasa, Byron Boots
Distribution Statement A (Approved for Public Release, Distribution
Unlimited). All authors are with the Paul G. Allen School of Computer Science
& Engineering, University of Washington {schmttle, rbaijal, bhou, siddh,
<EMAIL_ADDRESS>
###### Abstract

We focus on the problem of long-range dynamic replanning for off-road
autonomous vehicles, where a robot plans paths through a previously unobserved
environment while continuously receiving noisy local observations. An
effective approach for planning under sensing uncertainty is determinization,
where one converts a stochastic world into a deterministic one and plans under
this simplification. This makes the planning problem tractable, but the cost
of following the planned path in the real world may be different than in the
determinized world. This causes collisions if the determinized world
optimistically ignores obstacles, or causes unnecessarily long routes if the
determinized world pessimistically imagines more obstacles.

We aim to be robust to uncertainty over potential worlds while still achieving
the efficiency benefits of determinization. We evaluate algorithms for dynamic
replanning on a large real-world dataset of challenging long-range planning
problems from the DARPA RACER program. Our method, Dynamic Replanning via
Evaluating and Aggregating Multiple Samples (DREAMS), outperforms other
determinization-based approaches in terms of combined traversal time and
collision cost. https://sites.google.com/cs.washington.edu/dreams/

##  I Introduction

Inspired by the DARPA RACER program [1], we focus on the problem of motion
planning under sensing uncertainty for autonomous off-road vehicles travelling
over tens of kilometers. RACER challenges teams to program an autonomous off-
road vehicle equipped only with onboard sensing and compute to navigate
complex terrain (deserts, forests, hills) over long distances. Unlike on-road
driving (which contends with lanes, signs, and rules of the road), off-road
driving has much less structure. The robot can go wherever it can effectively
traverse, posing a unique robotics challenge.

This flexibility relies on the autonomy system’s onboard sensors and
perception system to discern what terrain is and is not traversable. For
example, terrain that is far from the robot may be difficult to classify
precisely due to e.g.,natural occlusions (hills, trees), few sensor readings,
or lack of training examples in the given environment. This noisy perception
creates both false obstacles and false freespace; a downstream planning
algorithm that is unaware of this uncertainty may produce dangerous collision-
bound or roundabout paths.

Thus, uncertainty is the core challenge of long-range planning: an algorithm
must appropriately consider this sensing uncertainty from perception when
planning paths (Fig. 1).

In this setting, a robot plans paths through a previously unobserved and
potentially hazardous environment, while continuously receiving and reacting
to noisy local observations.

This can be cast as a Partially Observable Markov Decision Process (POMDP).
Solving a POMDP is PSPACE-Complete [2], so we frame this as a Bayesian Dynamic
Motion Planning Problem (BDMP) to impose structure and make the problem
tractable. The BDMP problem differs from a POMDP in that uncertainty
originates only from the robot’s ignorance about the environment and the agent
maintains a posterior over possible environments given its observations. Given
the environment, the transition, the reward function and robot’s internal
state are fully observable [3].

Figure 1: An autonomous off-road vehicle’s long-range planner needs to decide
the best way up a hill, given blind spots and imperfect sensing. Figure 2:
Overview of DREAMS. _Sample & Plan_: Sample many worlds from the posterior
distribution, and plan the optimal path on a subsample of worlds ($\phi_{1}$,
$\phi_{10}$, $\phi_{50}$ above). _Evaluate_ : Evaluate the cost of each
resulting plan against the full set of sampled worlds. _Aggregate_ : Aggregate
the resulting cost distribution with a summary statistic (e.g., mean or CVaR)
_Select_ : Select the plan with minimal aggregated cost.

Previous works have exploited this structure via the framework of
determinization in the face of uncertainty—repeatedly solving and executing
relatively-inexpensive determinized planning problems—with strong theoretical
and practical results [3, 4, 5, 6]. Although determinization is
computationally efficient, it does not consider what happens when the
determinized problem that it solved diverges from reality. This can manifest
as optimism (causing collisions) or pessimism (causing roundabout paths, or no
path at all).

Our key insight is that this deficiency stems from determinization’s limited
ability to _reason about the distribution of costs over plausible worlds_.
Thus, we leverage multi-sample posterior sampling to reap some of the
computational benefits of determinization while preserving the planner’s
ability to reason across multiple plausible environments. Our resulting multi-
sample determinization algorithm, Dynamic Replanning via Evaluating and
Aggregating Multiple Samples (DREAMS), considers multiple plausible optimal
paths and multiple plausible worlds. With this framework, DREAMS enables
reasoning not just over the distribution of worlds, but also over additional
parameters such as traversal speed. We show that with the correct
instantiation, DREAMS outperforms prior determinization strategies on
realistic long-range off-road robot navigation tasks.

We make the following contributions:

  * •

We introduce DREAMS (Fig. 2), an algorithm for planning under uncertainty that
maintains the ease of planning from determinization while also being able to
consider uncertainty over worlds in decision making.

  * •

On a large dataset of challenging long-range planning problems, we demonstrate
that DREAMS plans effectively under uncertainty to achieve lower total cost
compared to other determinization methods.

##  II Related Work

There has been a fair amount of prior work on dynamic replanning under
uncertainty. D* and D* Lite are well-known dynamic replanning search
algorithms that have been demonstrated to quickly replan in real-world
settings [7, 8, 9]. Neither are designed to reason about uncertainty and are
optimistic about unknown parts of the environment. Re-planning is triggered
when the planned path is deemed in collision. In an environment with no
sensing uncertainty, this is effective because collisions can be easily
determined. In an uncertain environment, D* must either collide with an
obstacle to detect a collision or blindly trust its noisy sensors. That being
said, D*’s efficient re-use of the search tree could be incorporated into
methods that consider environment uncertainty directly.

The BDMP problem can be framed as a POMDP with unknown state, but known
transitions and rewards. POMDPs have a plethora of approaches but a notable
few rely on a fixed or sampled set of MDPs to make the problem more tractable
[10, 11, 4, 12, 13]. Most similar to our approach is DESPOT [4], which samples
a set of K scenarios and builds a tree to alleviate the curse of
dimensionality. While promising, solving a POMDP is PSPACE-complete [2] and
this work instead focuses on a tractable MDP relaxation with discrete set of
states and known transitions represented as a graph (instead of a tree) with
unknown reward.

The Canadian Traveler’s Problem [14] and specifically its stochastic variants
[15, 16, 5, 17] are most similar to the BDMP setting. In both cases, edge
collisions are discovered when the agent reaches an incident vertex. This
setting follows our observations from real mobile robotic systems: sensing is
more accurate near the robot. The stochastic variants additionally use this
information to update unobserved edge probabilities. Our setting is similar
except we discover the true cost of an edge when we traverse it, and we
additionally incorporate limited range noisy sensing.

Risk-aware planning can be seen as another form of planning under uncertainty
[18, 19, 20, 21, 22, 23]. Barbosa et al. [19] converts the popular Conditional
Value at Risk (CVaR) [24] metric into a cost function for planning and accepts
new plans only when risk increases along the current trajectory. While these
methods have shown great promise, none have investigated planning under the
setting of onboard sensing where uncertainty increases further from the robot.
Therefore, in this work we compare with these approaches as a baseline
(Section • ‣ IV-C).

Determinization, making a deterministic approximation of a stochastic problem,
has been effective for planning under uncertainty [6]. In particular, a
variety of works [3, 25, 26, 27, 18] have used posterior sampling [28] to make
the planning problem tractable and only require sample access to the
posterior. Dynamic Replanning with Posterior Sampling (DRPS) [3] samples one
problem and solves it optimally, letting sampling naturally balance
exploration and exploitation. A key observation from DRPS is that gaining
information from the world as a mobile robot is relatively easy without
explicit exploration. As the robot moves, it easily gains more sensor
information to clarify its future actions. Our work leverages this insight and
we apply posterior sampling as a determinization strategy. Sampled A* [18]
further utilizes multiple samples per replan and accepts the most likely path,
showing promising results. DREAMS also uses multiple samples, but differs in
how it selects the plan to follow by considering a distrbution of costs. We
compare DRPS and Sampled A* with DREAMS in a realistic setting where sensing
is noisy and there is a limited observation range.

##  III Bayesian Dynamic Motion Planning with Collisions

Given a start state $x_{s}$ and goal state $x_{g}$ in configuration space
$\mathcal{X}$ and a set of environments $\Phi$, we seek to minimize the
expected total time of traversing from start to goal under the distribution of
environments $P(\phi)$. To help solve this problem, we are given a measure of
uncertainty over environments modeled as the posterior distribution
$P(\phi|\psi_{t})$ where $\psi_{t}$ is the history of observations from time
$[0,t]$. This problem extends the Bayesian Dynamic Motion Planning Problem
(BDMP) [3] to consider the added cost of potential collisions. Off-road
collisions can be dangerous—injuring the rider or damaging the robot—and
require additional time to recover from.

Similar to DRPS, we focus on planning over roadmaps. Specifically, we are
given a graph $G$ with vertices $V$ and edges $E$. Each edge has a traversal
time $w:E\rightarrow{}^{+}$ and a collision status $\phi(e)$ where $\phi(e)=1$
means $e$ is a collision-free edge in world $\phi$. A path
$\xi_{t}=(e_{1},e_{2},\dots,e_{t})$ is defined as a sequence of edges. Since
we are in a dynamic setting, traversing edges adds observations to our history
$\psi_{t}$.

In the motivating RACER scenario, the goal is to get from start to goal as
fast as possible. Thus, we consider the following two metrics: Traversal Time
and Collision Cost. We additionally consider collision cost because optimizing
for traversal time alone can lead to impractical algorithms that collide
frequently with obstacles.

  * •

Traversal Time. $T(\xi)=\sum_{e\in\xi}w(e)$ is the time it takes the robot to
traverse to the goal. Edges in collision are still counted toward traversal
time.

  * •

Collision Cost. $C(\xi;\rho)=\sum_{e\in\xi}\mathds{1}(\phi(e)=0)c(e)\rho(e)$.
$c(e)$ is the collision cost and $\rho(e)$ adjusts the relative cost of a
collision compared to traversal time.

The total cost to reach the goal in one planning episode is:

$J(\xi;\rho)=T(\xi)+C(\xi;\rho)$ (1)

where $\rho(e)=\alpha$ is a constant.

##  IV Proposer-Acceptor Approach

To summarize various approaches from the literature, we decompose algorithms
into a _proposer_ and an _acceptor_. The proposer proposes a set of paths
$\Xi=\\{\xi^{0},\xi^{1},\dots,\xi^{n}\\}$. The acceptor considers the proposed
paths and accepts one. The robot then follows the accepted path for a step,
receives observations, and updates the posterior distribution. It then replans
and repeats until the goal is reached.

Figure 3:  The sensor noise levels used in testing:
$\eta_{\mathrm{low}}=10^{-4}$, $\eta_{\mathrm{med}}=10^{-3}$,
$\eta_{\mathrm{high}}=10^{-2}$. Our simplified noise model defines the
probability of receiving a correct observation for a query point distance $d$
away as $\max(\exp(-\eta d^{2}),p_{\mathrm{min}})$. The minimum probability
threshold $p_{\mathrm{min}}$ is set to 0.6 to provide some signal at the edge
of the robot’s observation range; note that the minimum possible value of
$p_{\mathrm{min}}$ for binary occupancy is 0.5 (pure noise). With these
parameters, the robot receives approximately 96%, 72%, and 61% correctly
observed pixels per observation.

###  IV-A DREAMS Proposer

The DREAMS proposer is based on posterior sampling, where at each step we are
sampling from the distribution of optimal plans
$P(\xi^{*}|\psi_{t})=P(\xi|\phi)P(\phi|\psi_{t})$. This is achieved via
sampling from the posterior over worlds $P(\phi|\psi_{t})$ and then planning
the optimal path on each sampled world (Fig. 2, _Sample & Plan_). As the robot
learns more about the world, this process naturally exploits the knowledge we
gain by reducing the spread of the distribution of worlds/plans. Similar to
Sampled A*, DREAMS samples multiple plans to approximate the distribution
$P(\xi^{*}|\psi_{t})$.

Figure 4: Traversed paths of each algorithm (blue edges) on two example worlds
from each of the Forest and Desert datasets, with the same world in each set
of four. Each algorithm receives observations with high noise, and is
penalized with a collision factor of $\alpha=10$. With high noise, DRPS
frequently backtracks and changes direction while Sampled A* incurs many
collisions (red edges). Both DREAMS variants follow more direct paths without
collisions.

###  IV-B DREAMS Acceptor

Unlike prior determinization approaches, we evaluate the cost of each sampled
plan against a distribution of sampled worlds (Fig. 2, _Evaluate_). The
sampled worlds do not need to be the same as the ones used for planning.
Empirically, we have found that planning is typically the bottleneck rather
than evaluation. Therefore, we opt to sample many more worlds for evaluation
than planning.

For each plan, we compute a summary statistic for this resulting distribution
of costs (Fig. 2, _Aggregate_) and select the plan that minimizes aggregate
cost. Depending on the application, the summary statistic can vary. For
example, selecting the minimum cost for a given plan is an optimistic strategy
that looks at a plan’s cost under the best-case scenario. CVaR summary
statistics balance the risk of high cost paths (in this case, collision-prone
paths) more carefully.

This approach is extremely flexible. In Section V, we additionally use this
acceptor to explore different velocity profiles to further reduce expected
cost. We now describe example evaluation and aggregation functions, although
these design choices will vary based on the application.

####  IV-B1 DREAMS Evaluation Function

We make a small modification to Eq. 1 to use as the DREAMS evaluation
function.

$\displaystyle\widehat{J}(\xi)$ $\displaystyle=T(\xi)+C(\xi;\tau)$ (2)
$\displaystyle\tau(e)$
$\displaystyle=\mathds{1}(e=e_{0})\alpha+\mathds{1}(e\neq
e_{0})\vspace{-1\baselineskip}$ (3)

Because DREAMS plans in a receding-horizon fashion, $\widehat{J}(\xi)$
considers the collision factor $\alpha$ only on the immediate edge and assigns
future potential collisions a relative cost of 1. Equally weighting all
collisions can create overly conservative behavior due to noisy observations
farther from the robot. Reducing future collision cost helps avoid these
scenarios.

####  IV-B2 DREAMS Aggregation Function

We optimistically take the mean of the best 75% of the distribution of costs,
calling it the Inverse CVaR. This reduces the effect of unlikely high-cost
outliers. Like CVaR it also considers the width of the distribution: as
increased uncertainty causes the cost distribution to spread out, the
increased aggregate cost promotes caution.

###  IV-C Benchmark Overview

We briefly summarize each benchmark algorithm with this framework. Each
algorithm proposes plan(s), accepts a plan, follows one edge, and replans.

  * •

DRPS [3]. Proposer: Sample one plan from the posterior. Acceptor: choose only
plan.

  * •

Sampled A* [29]. Proposer: Sample multiple plans from the posterior. Acceptor:
choose the most likely plan. Determined by computing edge centrality across
plans and accepting the plan with maximum mean edge centrality.

  * •

Direct. Proposer: Compute the plan that minimizes risk-aware evaluation cost
in expectation. Acceptor: choose only plan.

$\mathop{\mathds{E}}[\widehat{J}(\xi)]=\sum_{e\in\xi}w(e)+P(\phi(e)=0)c(e)\tau(e)$
(4)

##  V Experiments

The following simulation experiments are designed to replicate an off-road
autonomous driving scenario where the planner faces limited sensor range and
noisy perception. The robot must navigate as efficiently as possible while
avoiding dangerous collisions, re-planning at at each step.

We compare DREAMS to DRPS and Sampled A*, as both incorporate posterior
sampling and present strong results on similar problems. DRPS highlights the
difference between single and multi-sample posterior sampling, while Sampled
A* compares the evaluation/aggregation acceptor strategy with an approximate
MAP estimate. These baseline algorithms only consider the geometric path at an
arbitrary fixed speed. We include results for DREAMS-Fixed to compare most
directly with these algorithms, and provide additional results for DREAMS-
Adaptive to demonstrate our ability to evaluate paths under different
parameters in this case speed. Finally, we evaluate a benchmark that optimizes
Equation 2 directly (Direct) to demonstrate the benefits of posterior
sampling.

We consider the following hypotheses:

  1. H1.

Both DREAMS variants will incur lower total cost compared to DRPS and Sampled
A*. More sampled plans will help DREAMS reduce cost variance relative to DRPS.
Reasoning about a distribution of costs rather than accepting approximate MAP
estimate will help DREAMS incur less collision cost than Sampled A*.

  2. H2.

DREAMS-Adaptive will reduce collision cost and total cost compared to DREAMS-
Fixed. Reasoning about distribution of speeds allows the vehicle to slow down
when the likelihood of a collision increases and speed up when the path is
likely free.

  3. H3.

Increasing the number of sampled plans will reduce the total cost (with
diminishing returns). More plans will provide more options to evaluate, until
all likely options are enumerated.

  4. H4.

Increasing the number of sampled worlds considered during evaluation will
reduce the total cost (with diminishing returns). More world samples from the
distribution will better estimate the distribution.

  5. H5.

DREAMS will incur lower total cost compared to Direct. To plan directly with
the cost function, Direct replaces the indicator in Equation 2 with a
probability. This can result in extremely unlikely plans under the posterior
$P(\xi^{*}|\psi_{t})$, which DREAMS probabilistically avoids by construction.

###  V-A Experimental Setup

Figure 5: Qualitative comparison of each approach, given the exact same
sampled worlds and paths. Robot (blue), proposed paths (light orange),
accepted path (bright orange). _Top:_ All plans except DRPS find a path
through the gap. DRPS happened to sample a world that did not fit through the
gap, producing a longer route. _Bottom:_ All plans except Sampled A* reverse
from a likely obstacle in front of the robot. Sampled A* does not explicitly
consider the cost of collision and accepts a path going through the obstacle.
_Right:_ Looking at the heatmap, areas where more plans overlap are hotter. As
there is little overlap besides the start position, Sampled A* has less signal
to choose the most likely plan; its decision is almost a uniform random
sample.

####  V-A1 Real World Occupancy Grids and Speeds

We evaluate performance with two real-world datasets of long-range planning
problems through open desert environments ($N=83$) and more challenging
crowded forest environments ($N=51$). These datasets were collected through
the RACER program.

Worlds are $100\times 100$ meters at a resolution of 0.4 m/px. The robot moves
on a graph covering the space at speeds ranging from 1–10 m/s. As the robot
traverses, it receives observations at 1 Hz. Therefore, speed affects both
traversal time and the number of observations received. The robot can move in
reverse at a fixed speed of 1 m/s. For all approaches, we discourage reversing
by prompting each planning call to find a solution without reversing. If
unsuccessful, it retries with reversing allowed. We evaluate on ten random
seeds for each world $\phi$, noise level $\eta$, and collision $\alpha$.

####  V-A2 Limited Range Noisy Observations

It is a popular choice among autonomous vehicles to process raw sensor input
into semantic classification of the environment using a deep neural network
[30, 31]. Noisy sensors and limited training introduce uncertainty into the
resulting semantic segmentations. Generally, the predictions become noisier
farther away from the robot where there is less (and noisier) sensor
information. Predicted semantics eventually become too noisy and are thus
limited to a reliable range. To simulate this, we add a limited range
observation module that simulates classification of obstacles. The robot can
only observe a patch of $50\times 50$ meters centered around itself. Fig. 3
describes the sensor model within this limited range observation.

Outside of the limited range observation, we optimistically assume that the
space is free to allow posterior sampling to discover many plausible paths.
This is similar to how real-world systems work where unknown space is a fixed
cost. The posterior is updated using Bayes’ Rule.

####  V-A3 Posterior Sampling

To sample from the posterior distribution over optimal plans
$P(\xi^{*}|\psi_{t})$, we sample from the posterior distribution over worlds
$P(\phi|\psi_{t})$ and plan on each world. We sample a world by sampling over
the posterior distribution of roadmap edges, created by taking the maximum
posterior collision probability across all pixels marked by the robot’s swept
volume ($3.5\times 1.5$ meters) along an edge.

####  V-A4 Planning and Execution Parameters

For DREAMS and Sampled A*, we choose to plan with 100 posterior samples using
A*. DREAMS evaluates each plan according to (Equation 2) against $10^{4}$
sampled worlds. DREAMS-Adaptive additionally considers multiple speed
profiles. For each sampled plan, it creates five timed trajectories each with
a separate speed profile. Profiles are all 5 m/s within the observed area and
optimistically 10 m/s outside the observed area, but differ in the first edge
traversal speed $\\{1,3,5,7,10\\}$ m/s. DREAMS-Adaptive executes the chosen
speed for the chosen plan for one time step before re-planning. DREAMS-Fixed
and the other algorithms consider only one speed profile: 5m/s in observed
area, 10m/s outside observed area.

The collision cost is proportional to the robot’s pre-collision speed
(Equation 1), as higher speed collisions are more dangerous. We vary $\alpha$
across $\\{1,10,20\\}$ to characterize performance with different relative
collision costs.

####  V-A5 Metrics

We compare each algorithm’s incurred cost to the cost incurred by an oracle
$\xi^{opt}$, which has full information about the world and traverses at 10
m/s without collisions.

$\mathrm{Suboptimality}=J(\xi)/J(\xi^{opt})=J(\xi)/T(\xi^{opt})$ (5)

(a) Forest

(b) Desert

(c) Paths

(d) World Samples

Figure 6:  _Left:_ Suboptimality plots for (a) Forest and (b) Desert datasets.
We perform a Welch’s t-test for difference of means, with a Bonferroni
correction of 90 for all pairwise comparisons involving DREAMS.
$\text{*}:p<0.01,\text{**}:p<0.001,\text{***}:p<0.0001,\text{****}:p<0.00001$.
_Right:_ Ablation study for varying (c) number of sampled plans and (d) number
of world samples in evaluation. (Error bars in both figures denote 95%
confidence intervals.)  Figure 7: Comparison of DREAMS to Direct on the more
challenging Forest dataset. Direct incurs many more collisions because it does
not reason about the likelihood of the plans directly. Note: The suboptimality
axis is much higher than Fig. 6(a).

###  V-B Results

Fig. 6(a) and 6(b) show suboptimality results for the Forest and Desert
datasets. An ablation study with the more challenging Forest dataset is
visualized in Fig. 6(c) and 6(d).

Qualitative results of traversed paths on the final posterior are shown in
Fig. 4. Fig. 5 compares each algorithm under the exact same scenarios. Table I
gives planning time results.

H1. In both datasets (Fig. 6(a) and 6(b)), DREAMS-Fixed is competitive with
DRPS and Sampled A* in Low noise. It is either competitive or outperforms them
in Medium noise. In High noise, it dominates for all $\alpha$ values tested.
H1 is supported. It is not surprising that all algorithms achieve similar
results in Low noise, as the sampled plans will likely be near-optimal (i.e.,
most sampled worlds are close to the true world). In Medium noise, we observe
the benefit of multiple samples as both DREAMS variants and Sampled A* perform
better than DRPS. But in High noise, Sampled A* incurs a high collision cost
as it does not explicitly reason about collisions; all plans seem equally
likely because the distribution of plans is spread out, showing less benefit
to multiple samples without explicit collision reasoning. Fig. 5, bottom shows
an example scenario of this behavior.

H2. In Fig. 6(a) and 6(b), we see a mixed result between DREAMS variants:
DREAMS-Adaptive has statistically-significant lower cost in some cases but not
all. With Low noise, DREAMS-Adaptive can move faster reducing its traversal
time without incurring too much collision cost. In Medium noise, DREAMS-
Adaptive and DREAMS-Fixed achieve similar performance; this suggests that
these higher speeds do not properly balance speed and safety. In High noise,
DREAMS-Adaptive seems to trade-off collisions and traversal better. Because
the results are mixed, H2 is not strongly supported.

Algorithm | Proposer Time | Acceptor Time | Total Time
---|---|---|---
DREAMS-Adaptive | $1.55\pm 0.09$ | $1.59\pm 0.01$ | $3.14\pm 0.09$
DREAMS-Fixed | $1.54\pm 0.09$ | $0.33\pm 0.00$ | $1.88\pm 0.09$
DRPS | $0.02\pm 0.00$ | $0.01\pm 0.00$ | $0.03\pm 0.00$
Sampled A* | $1.47\pm 0.08$ | $1.12\pm 0.02$ | $2.58\pm 0.08$
TABLE I: Planning time evaluations for each algorithm, on the same set of 300
evaluation runs. _Proposer:_ DREAMS and Sampled A* plan 100 paths per
iteration, while DRPS plans a single path. Because there is no dependency
between planning each independent posterior sample, this can be easily
accelerated by parallelization (results sequential). _Acceptor:_ Since DRPS
only proposes one path, the acceptor time is negligible. DREAMS-A evaluates
five speeds taking more time while DREAMS-F evaluates just one. Aggregating
edge centrality across plans is also relatively expensive for Sampled A*.

H3. Fig. 6(c) shows that more sampled plans reduces the total cost across
multiple noise values for DREAMS. The leveling off at 20 plans shows that the
sampled set is fairly representative of our posterior distribution. H3 is
supported. Surprisingly, Sampled A* performed worse with increased samples at
High noise. We attribute this to High noise causing a spread out distribution
of plans where no plan has a large mean centrality, resulting in near random
choice (See Fig. 5, bottom). Increasing the number of plans introduces more
options for Sampled A* to choose from; if these are likely to be in collision,
this will generally increase the cost incurred.

H4. Fig. 6(d) shows more world samples in evaluation improves performance for
DREAMS-Adaptive at Medium and High noise. For DREAMS-Fixed, we only see change
in High noise. In this setting, it suggests a few samples captures the true
cost well in Low and Medium noise, but more samples are needed in High noise.
H4 is supported for High noise.

H5. Fig. 7 shows the suboptimality comparison between DREAMS-Adaptive and
Direct. Direct incurs a very high collision cost and total suboptimality
because it finds paths with a high likelihood of collisions. We attributed
this to the probability of collision being a multiplicative factor instead of
an actual probability, meaning Direct may choose plans that are highly
unlikely but have a low total cost. H5 is supported.

##  VI Discussion

There remain multiple limitations and avenues for future work. DREAMS
currently relies on a hand-tuned cost function for its evaluation step. While
we show it works in our setting with traversal time and collisions, if an
application considers more costs it could quickly become difficult to design a
good cost function. A learned cost function using ground truth information or
demonstrations may be a more scalable alternative. Second, we perform sampling
on a graph posterior that is obtained through taking the maximum probability
of collision along an edge. While this heuristic can give a nice distribution
of paths, it is worth exploring other sampling methods like directly sampling
costmaps from the perception model or sampling paths from a neural planner.

##  VII Disclaimer

The views, opinions and/or findings expressed are those of the author and
should not be interpreted as representing the official views or policies of
the Department of Defense or the U.S. Government.

## References

  * [1] “Robotic autonomy in complex environments with resiliency (racer),” https://www.darpa.mil/program/robotic-autonomy-in-complex-environments-with-resiliency.
  * [2] C. H. Papadimitriou and J. N. Tsitsiklis, “The complexity of markov decision processes,” _Math. Oper. Res._ , 1987.
  * [3] B. Hou and S. Srinivasa, “Dynamic replanning with posterior sampling,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2023\.
  * [4] A. Somani, N. Ye, D. Hsu, and W. S. Lee, “Despot: Online pomdp planning with regularization,” in _Advances in Neural Information Processing Systems_ , 2013.
  * [5] Z. W. Lim, D. Hsu, and W. S. Lee, “Shortest path under uncertainty : Exploration versus exploitation,” in _Conference on Uncertainty in Artificial Intelligence_ , 2017.
  * [6] S. Yoon, A. Fern, and R. Givan, “Ff-replan: A baseline for probabilistic planning,” in _International Conference on Automated Planning and Scheduling_ , 2007.
  * [7] A. Stentz, “The d* algorithm for real-time planning of optimal traverses,” Carnegie Mellon University, Pittsburgh, PA, Tech. Rep., 1994.
  * [8] S. Koenig and M. Likhachev, “D*lite,” in _AAAI Conference on Artificial Intelligence_ , 2002.
  * [9] D. Ferguson and A. Stentz, “Field d*: An interpolation-based path planner and replanner,” in _International Symposium on Robotics Research_ , 2005.
  * [10] S. Ross, J. Pineau, S. Paquet, and B. Chaib-draa, “Online planning algorithms for pomdps,” _J. Artif. Int. Res._ , 2008.
  * [11] D. Silver and J. Veness, “Monte-carlo planning in large pomdps,” in _Advances in Neural Information Processing Systems_ , 2010.
  * [12] Z. Sunberg and M. J. Kochenderfer, “Online algorithms for pomdps with continuous state, action, and observation spaces,” in _International Conference on Automated Planning and Scheduling_ , 2017.
  * [13] M. Chen, E. Frazzoli, and D. Hsu, “Pomdp-lite for robust robot planning under uncertainty,” in _IEEE International Conference on Robotics and Automation_ , 2016.
  * [14] C. H. Papadimitriou and M. Yannakakis, “Shortest paths without a map,” in _Automata, Languages and Programming_ , 1989.
  * [15] P. Eyerich, T. Keller, and M. Helmert, “High-quality policies for the canadian traveler’s problem,” _AAAI Conference on Artificial Intelligence_ , 2010\.
  * [16] D. Dey, A. Kolobov, R. Caruana, E. Kamar, E. Horvitz, and A. Kapoor, “Gauss meets canadian traveler: Shortest-path problems with correlated natural dynamics,” in _International Conference on Autonomous Agents and Multi-agent Systems_ , 2014.
  * [17] S. Yoon, A. Fern, R. Givan, and S. Kambhampati, “Probabilistic planning via determinization in hindsight,” in _AAAI Conference on Artificial Intelligence_ , 2008.
  * [18] J. J. Chung, A. J. Smith, R. Skeele, and G. A. Hollinger, “Risk-aware graph search with dynamic edge cost discovery,” _International Journal of Robotics Research_ , 2019.
  * [19] F. S. Barbosa, B. Lacerda, P. Duckworth, J. Tumova, and N. Hawes, “Risk-aware motion planning in partially known environments,” _Conference on Decision and Control_ , 2021.
  * [20] A. Suresh and S. Martínez, “Planning under risk and uncertainty based on prospect-theoretic models,” _arXiv preprint arXiv:1904.02851_ , 2019.
  * [21] S. Feyzabadi and S. Carpin, “Risk-aware path planning using hirerachical constrained markov decision processes,” in _IEEE International Conference on Automation Science and Engineering_ , 2014.
  * [22] L. Murphy and P. Newman, “Risky planning on probabilistic costmaps for path planning in outdoor environments,” _IEEE Transactions on Robotics_ , 2013\.
  * [23] X. Cai, M. Everett, J. Fink, and J. P. How, “Risk-aware off-road navigation via a learned speed distribution map,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2022.
  * [24] S. Uryasev and R. T. Rockafellar, _Conditional Value-at-Risk: Optimization Approach_. Springer US, 2001.
  * [25] J. Asmuth, L. Li, M. L. Littman, A. Nouri, and D. Wingate, “A bayesian sampling approach to exploration in reinforcement learning,” _Conference on Uncertainty in Artificial Intelligence_ , 2012.
  * [26] M. J. A. Strens, “A bayesian framework for reinforcement learning,” in _International Conference on Machine Learning_ , 2000.
  * [27] A. Wilson, A. Fern, S. Ray, and P. Tadepalli, “Multi-task reinforcement learning: A hierarchical bayesian approach,” in _International Conference on Machine Learning_ , 2007.
  * [28] W. R. Thompson, “On the likelihood that one unknown probability exceeds another in view of the evidence of two samples,” _Biometrika_ , 1933.
  * [29] L. C. Freeman, “A set of measures of centrality based on betweenness,” _Sociometry_ , 1977.
  * [30] A. Shaban, X. Meng, J. Lee, B. Boots, and D. Fox, “Semantic terrain classification for off-road autonomous driving,” in _IEEE International Conference on Robotics and Automation_ , 2022.
  * [31] R. Schmid, D. Atha, F. Schöller, S. Dey, S. Fakoorian, K. Otsu, B. Ridge, M. Bjelonic, L. Wellhausen, M. Hutter, and A. Agha-mohammadi, “Self-supervised traversability prediction by learning to reconstruct safe terrain,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2022.


11institutetext: University of Chinese Academy of Sciences, Beijing, China
22institutetext: Paul C. Lauterbur Research Center for Biomedical Imaging,
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences,
Shenzhen, Guangdong, China 33institutetext: Pengcheng Laboratory, Shenzhen,
Guangdong, China 44institutetext: Shenzhen University, Shenzhen, Guangdong,
China 55institutetext: Nanchang University, Nanchang, Jiangxi, China
55email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Rethinking the optimization process for self-supervised model-driven MRI
reconstruction

Weijian Huang 112233 Cheng Li 22 Wenxin Fan 1122 Yongjin Zhou 44 Qiegen Liu 55
Hairong Zheng 2233 Shanshan Wang 2233

###### Abstract

Recovering high-quality images from undersampled measurements is critical for
accelerated MRI reconstruction. Recently, various supervised deep learning-
based MRI reconstruction methods have been developed. Despite the achieved
promising performances, these methods require fully sampled reference data,
the acquisition of which is resource-intensive and time-consuming. Self-
supervised learning has emerged as a promising solution to alleviate the
reliance on fully sampled datasets. However, existing self-supervised methods
suffer from reconstruction errors due to the insufficient constraint enforced
on the non-sampled data points and the error accumulation happened alongside
the iterative image reconstruction process for model-driven deep learning
reconstrutions. To address these challenges, we propose K2Calibrate, a K-space
adaptation strategy for self-supervised model-driven MR reconstruction
optimization. By iteratively calibrating the learned measurements, K2Calibrate
can reduce the network’s reconstruction deterioration caused by statistically
dependent noise. Extensive experiments have been conducted on the open-source
dataset FastMRI, and K2Calibrate achieves better results than five state-of-
the-art methods. The proposed K2Calibrate is plug-and-play and can be easily
integrated with different model-driven deep learning reconstruction methods.

###### Keywords:

Self-supervised learning Image reconstruction

##  1 Introduction

Figure 1: Utilization of the learned K-space in iterative steps. The shown
mask will be multiplied with the corresponding K-space measurements in
practice. (a) Existing model-driven based reconstruction methods directly use
the learned k-space with high noise ratio for the next iteration (Mask is
represented as a blank matrix); and (b) the proposed K2Calibrate uses only
part of the reconstructed signal.

Magnetic resonance imaging (MRI) is an important tool in the clinic for the
detection of various diseases. One bottleneck issue of MRI is its time-
consuming data acquisition process. Much effort has been devoted to
alleviating this problem, among which compressed sensing (CS) based MR
reconstruction has been acknowledged as a very effective approach. It exploits
incoherent undersampling and image sparsity [1, 2, 3] to accelerate MR
imaging. Unfortunately, despite the promising performances reached with CS-
MRI, it has several limitations such as the time-consuming iterative
reconstruction process and the difficulty for the determination of its
sparsity weight hyper-parameters[4, 5].

To address these issues, deep learning (DL) has been introduced to accelerate
MR imaging. One major group of methods are based on the data-driven approach.
These methods directly learn a mapping between undersampled k-space
data/aliased-image and fully sampled k-space data/artifact-free image in an
end-to-end manner [6, 7, 8, 9, 10]. Data-driven methods are straightforward
and easy-to-implement, but they fail to exploit the physics of MRI. Model-
driven methods, on the other hand, incorporate MRI physics into DL-based MRI
reconstruction[11, 12, 13, 14, 15]. These methods unroll the iterative
reconstruction steps of solving physical models into the different layers of a
to-be-trained neural network [16, 11, 12, 17]. In this way, model-driven
methods can integrate the strength of both DL and physical models. They have
gradually become the prevalent solution for DL-based MRI reconstruction tasks.

Both data-driven and model-driven DL MRI reconstruction methods have made
great progress in accelerating MR imaging. However, their performances rely on
fully sampled and high quality reference datasets, the acquisition of which is
not always feasible. One major reason is that fully sampled data acquisition
is very time-consuming. For moving objects (such as the heart), high quality
fully sampled dataset is hard to obtain without image artifacts. For the same
reason, fully sampled data are difficult to obtain when serious signal
attenuation happens in some applications. For example, when conducting
diffusion MRI scanning with echo planar (EPI), the T2 signal decays quickly.
Thus, it is hard to adopt fully sampled data acquisition protocols. Another
reason is acquiring a large number of fully sampled data to train a DL model
is both resource- and labor-intensive.

Recently, Yaman et al.[18, 19, 20] proposed a series of MRI reconstruction
methods based on model-driven self-supervised learning (self-supervised
learning via data undersampling, SSDU), which does not require fully sampled
reference data for model training. The core idea of SSDU is to split the
obtained undersampled k-space data into two disjoint sets. One set is used as
the input to the unrolled network, and the other set is used to supervise the
training of the network. Later, Hu et al.[21] proposed an improved method
based on SSDU to constrain the reconstruction results on non-sampled data
points via a parallel network framework. Cole et al.[22] employed Generative
Adversarial Networks (GAN) to guide the optimization of the reconstruction
network via discriminating simulated and observed measurements. The above
methods have achieved promising results using self-supervised MRI
reconstruction methods. However, since the reconstruction problem is ill-posed
due to sub-Nyquist sampling, imposing precise constraints on the non-sampled
K-space data points is difficult. In addition, error propagation might happen
alongside the iterative reconstruction process, especially at the initial
iterations.

Here, we propose K2Calibrate, a convolutional neural network (CNN) based
K-space adaptation method for self-supervised unrolled networks. By sampling
k-space measurements from a Bernoulli distribution in each iteration step, the
model-driven reconstruction process is optimized, so as to suppress the noise
ratio caused by unsatisfied constraints of self-supervised models. The
proposed method has high flexibility and can be formulated in a ”plug-and-
play” way , which can be easily applied to existing self-supervised model-
driven methods without any complex operations.

##  2 Theory

Let $x$ denote the recovered image and $y$ represent the acquired k-space
measurements. MRI image recovery using model-driven methods can be formed as
the following optimization scheme:

$\mathop{\arg\min}\limits_{x}\parallel Ax-y\parallel^{2}+\lambda R(x)$ (1)

where $A=SFC$ is the encoding matrix including coil sensitivities $C$, 2-D
discrete Fourier transform $F$, and sampling matrix $S$. The first term
represents data consistency (DC) with acquired measurements. $R(\cdot)$
represents a regularization term. $\lambda$ denotes the weighting parameter.

###  2.1 Model based deep learning network

Model based deep learning networks [12, 14, 13] have received extensive
attention due to the enhanced physical interpretability. One typical example
is MoDL [12]. Let $D_{w}$ represent a learned CNN estimator of noise depending
on the parameters $w$. $x-D_{w}(x)$ can be treated as a denoiser to remove
alias artifacts and noise. Then, Eq.1 can be rewritten as:

$\mathop{\arg\min}\limits_{x}\parallel
Ax-y\parallel^{2}+\lambda\parallel(x-D_{w}(x))\parallel^{2}$ (2)

Since the non-linear mapping $D_{w}(x_{k}+\triangle x)$ can be approximated
using Taylor series [12] in the unrolled networks, the reconstructed
optimization problem can be approximated as:

$\begin{array}[]{c}x^{k+1}=\mathop{\arg\min}\limits_{x}\parallel
Ax-y\parallel^{2}+\lambda\parallel(x-z^{k}(x))\parallel^{2}\\\
z^{k}=D_{w}(x^{k})\end{array}$ (3)

Here, index $n\in[0,...,K]$ denotes the iteration number. Once $K$ is fixed,
the optimization process can be intuitively viewed as an unrolled linear CNN.
In MoDL[12], the authors used the same denoising operator $D_{w}$ and
trainable regularization parameters at each iteration to reduce the network
parameters. Since MoDL is a highly effective method, we adopt it as the
baseline for our proposed method.

###  2.2 Self-supervised MRI reconstruction

Figure 2: The self-supervised MRI reconstruction workflow. The acquired
k-space is split into sets $\Theta$ and $\Lambda$, while one is used for input
and the other is used for supervision.

Due to the difficulty of acquiring fully sampled reference data, Yaman et
al.[18] proposed a self-supervised approach called SSDU achieving promising
reconstruction performance.

Let $\Omega$ denote the undersampled K-space measurements of fully sampled
measurements $\Gamma$. For each data, $\Omega$ is divided into two random
subsets, $\Theta$ and $\Lambda$, and the relationship between them is
$\Omega=\Theta\cup\Lambda$.

To train the self-supervised model-driven network, the set of $\Theta$ is used
as the input to the network as well as the DC units, while the set $\Lambda$
is used to define the similarity self-supervised loss function:

$\mathop{\arg\min}\limits_{\omega}\emph{{L}}(\textbf{E}_{\Lambda}(y_{\Omega}),\textbf{E}_{\Lambda}(\emph{{f}}(\textbf{E}_{\Theta}(y);\omega)))$
(4)

where $\emph{{L}}(\cdot)$ is an arbitrary similarity function such as 1-norm
or 2-norm and $\emph{{f}}(;\omega)$ denotes the output of the unrolled network
with parameters $\omega$. $\textbf{E}_{\Lambda}(\cdot)$ represents a K-space
transform operator that sampling the measurements in $\Lambda$ region, and so
as to $\textbf{E}_{\Theta}(\cdot)$. For clarification, let $y$ mentioned in
equations (1,2,3) be $y_{\Omega}$ in this section. In this way, a self-
supervised strategy is implemented. This method updates the iterative network
in an efficient end-to-end training manner.

However, since the data are simulated by sub-Nyquist sampling, it inherently
cannot constrain the reconstruction on the non-sampled data points
$\Gamma\backslash\Omega$ precisely. This means once the reconstruction errors
occur, they will accumulate alongside the unrolled iterative reconstruction
process, as shown in Fig.1(a). Unfortunately, according to Eq.4, such
reconstruction errors are easy to occur since there are inherently no precise
supervision on $y_{\Gamma\backslash\Omega}$.

###  2.3 Derivation of K-space calibration

Figure 3: The proposed K2Calibrate method. In each iteration, the sampled
K-space measurements of $\theta$ are kept while the generated measurements
from the network are sampled from Bernoulli distribution.

To analyze the possible impact of reconstruction errors, we now consider the
output of a certain step of the unrolled networks
$\tilde{x}^{k}=n^{k}+x^{*k}$. For simplicity, we omit the superscript $k$ and
draw the joint distribution as follows:

$p(x^{*},n)=p(x^{*})p(n\left|x^{*}\right)$ (5)

Here, $n$ denotes the noise randomly generated from the unrolled network while
$x^{*}$ is the goal of the network in each specific iteration step. Due to the
imaging physics of MRI, we assume that $x^{*}$ is statistically dependent that
$p(x^{*}_{i}\left|x^{*}_{j}\right)\neq p(x^{*}_{i})$.

Then, we turn to consider the characteristics of the noise $n$. Since the task
is based on self-supervised learning as mentioned in 2.2, $n$ becomes more
diverse and should be considered more carefully. [23, 24] assume that $n$ of
natural image is conditional independent such that
$p(n\left|x^{*}\right)=\prod p(n_{i}\left|x_{i}^{*}\right)$. By further
assuming $\mathbb{E}[n_{i}]=0$, $n$ can be removed by using strategy of
$\mathbb{E}[\tilde{x}_{i}]=x^{*}$. However, we believe $n$ is still dependent
on $x^{*}$ due to the iterative process in MRI reconstruction that is more
difficult to remove.

Here, we propose a K-space calibration strategy (Fig.3), called ’K2Calibrate’,
to try to mitigate the effect of noise in the unrolled networks. Different
from designing a complexity prior of $n$ manually, we speculate that the
reconstruction problem can be transferred to balance $n$ and $x^{*}$ during
the iteration process. Let’s divide $n$ into two parts, $n_{s}$ and $n_{g}$,
denoting noise at location of sampled/non-sampled K-space measurements,
respectively. Since DC layers are included in the unrolled networks,
$\mathbb{E}[n_{s}]$ is closed to 0 and is much smaller than
$\mathbb{E}[n_{g}]$. Therefore, if we choose a tiny increment $n_{\delta}\in
n_{g}$, we have:

$\mathbb{E}[\tilde{x}_{s+\delta}]=\mathbb{E}[x^{*}_{s+\delta}+n_{s}+n_{\delta}]\approx
x^{*}_{s+\delta}$ (6)

which significantly reduces the scale of $n$ in $\tilde{x}$ comparing to
$\mathbb{E}[\tilde{x}_{s+g}]=x^{*}_{s+\delta}+n_{\delta}$. Based on this
observation, we can prevent the accumulation of noise in each iteration by
reducing the size of $n$ generated from the previous iteration. We modified
Eq.3 to achieve this and make sure that noise can be removed iteratively
rather than all at once in each iteration:

$\begin{array}[]{c}x^{k+1}=\mathop{\arg\min}\limits_{x}\parallel
Ax-y\parallel^{2}+\lambda\parallel(x-z^{k}(x))\parallel^{2}\\\
z^{k}=\textbf{E}^{k}_{s+\delta}(D_{w}(x^{k}))\end{array}$ (7)

In experiments, we use Bernoulli-sampling to simulate the incremental
$\delta\in g$. By adapting this calibration strategy in K-space measurements,
$D_{w}$ can be optimized to learn more robust representations since the
calibration operation reduces the propagation of $n_{g}$ in the unrolled
networks, as shown in Fig.1(b), especially in the initial iteration steps.

##  3 Experiments and Results

###  3.1 Dataset

We use an open-source MR multi-coil knee dataset, FastMRI, to verify the
effectiveness of the proposed method. FastMRI is a large-scale data collection
of clinical MR measurements released by Facebook AI Research[25]. 1,594 scans
were collected utilizing the following scanning parameters: Echo train length
4, 15-channels coils, matrix size of 320$\times$320, slice thickness 3mm, in-
plane resolution 0.5mm×0.5mm, and no gap between slices. The dataset was split
into training, validation, and testing sets. During training, we train the
models with 3$\%$ (Set-A) and 1$\%$ (Set-B) data of the training set to
evaluate the performance of different methods when different numbers of
training data are utilized.

###  3.2 Implementation details

Multiple comparison methods are implemented. Among them, we use Sigpy[26]
package to implement L1-Wavelet, SENSE, and TotalVariation (TV). DL-based
self-supervised methods include SSDU[18] and Hu et al.[21] and supervised
MoDL[12] are also experimented with. For fair comparison, all the CNN-based
methods adopt the same network architecture proposed in [12]. $D_{w}$ block
has 9 convolution layers (kernel no of 64). Each convolution layer is followed
by a non-linear activation LeakyReLu except for the final layer. The output of
each $D_{w}$ block is fed to a data consistency (DC) layer as described in
[12]. By sharing the learning weight of $D_{w}$, we unroll the network into 5
and 10 iterations according to the experiments settings, and adopt the
proposed K2Calibrate strategy at different steps in the iterations. It is
worth noting K2Calibrate is only enabled during model training and it is
removed during model testing. Adam optimizer is used with a learning rate of
$5\times 10^{-4}$ and a batch size of 2 for model training. All experiments
are performed using Pytorch in Python.

###  3.3 Results

This section presents various quantitative and qualitative evaluations to
verify the effectiveness of the proposed method.

####  3.3.1 Quantitative analysis

Two evaluation metrics, peak signal to noise ratio (PSNR) and structural
similarity index measure (SSIM), are calculated to quantitatively evaluate the
image reconstruction results. Table 1 lists the results of conventional
methods and DL-based methods. Both methods do not require ground truth fully
sampled reference data for model training. Compared with the conventional
methods, DL-based methods show better performance due to the complex denoiser
provided by its large number of parameters. When the proposed K2Calibrate in
utilized, the metrics of the two self-supervised DL-based methods improve
obviously. SSIM is increased from 0.7674 to 0.7803 for SSDU(8x) and from
0.7680 to 0.7754 for Hu(8x), respectively. Similar performance improvement is
observed for other settings, which validates K2Calibrate that can improve the
performance of different reconstruction methods.

Table 1: Quantitative results on Set-A with acceleration rates of 4 and 8 of different self-supervised MRI reconstruction methods. Method | 4x | 8x
---|---|---
PSNR | SSIM | PSNR | SSIM
L1Wavelet | 32.172 | 0.7674 | 28.704 | 0.6749
SENSE | 35.323 | 0.9121 | 29.461 | 0.7636
TV | 31.856 | 0.7725 | 29.361 | 0.7175
SSDU | 37.795 | 0.9332 | 29.584 | 0.7674
SSDU-K2C | 38.496 | 0.9402 | 30.303 | 0.7803
Hu | 37.670 | 0.9341 | 29.604 | 0.7680
Hu-K2C | 37.947 | 0.9361 | 29.841 | 0.7754

We implemented our method on the smaller training set, Set-B, and results are
listed in Table 2. Since the conventional methods are not affected by the
number of training data, the performance of them is the same as that presented
in Table 1. Compared with Table 1, the overall scores in Table 2 are slightly
decreased due to that fewer training data are utilized. However, on this
smaller-scale dataset, K2Calibrate can still improve the model performance,
indicating that the effectiveness of K2Calibrate is not dependent on the
amount of training data.

Table 2: Quantitative results on Set-B with acceleration rates of 4 and 8. K2Calibrate can still improve the reconstruction performance of the two self-supervised methods. Method | 4x | 8x
---|---|---
PSNR | SSIM | PSNR | SSIM
SSDU | 36.936 | 0.9242 | 29.447 | 0.7634
SSDU-K2C | 37.693 | 0.9313 | 29.783 | 0.7735
Hu | 36.926 | 0.9256 | 29.600 | 0.7681
Hu-K2C | 37.356 | 0.9301 | 29.642 | 0.7692

K2Calibrate is a plug-and-play module. To have a clear understanding of the
impact of K2Calibrate on the reconstruction performance, we have conducted
experiments by enabling K2Calibrate in different network training iterations.
SSDU was adopted as the base architecture in this set of experiments.
Relatively higher acceleration rates (8x and 10x) were employed to better
visualize the improvements made by K2Calibrate. Results are plotted in Fig. 4.
In the first 8 iterations, the reconstruction performance is enhanced when
more K2Calibrate modules are enabled. Compared to the method without
K2Calibrate, utilizing K2Calibrate in the first iteration increases the PSNR
value significantly from 28.54 db to 29.08 db under the acceleration rate of
10, which validates the effectiveness of K2Calibrate on noise suppression.
When more than 8 K2Calibrate modules are enabled (the number of K2calibrate is
9 or 10 in Fig.4), the performance becomes worse. This observation is in
accordance with our expectations. We have explained that the noise ratio is
relatively higher in the initial iterations, and thus utilizing K2Calibrate in
these iterations is essential for noise removal. After these iterations,
utilizing more K2Calibrate modules becomes ineffective.

Figure 4: Reconstruction results when different numbers of K2Calibrate are
enabled. Acceleration rates of 8 and 10 are utilized. The number of
K2calibrate indicate how many iterations utilized K2Calibrate. For example, if
the number of K2Calibrate is 5, it means K2Calibrate is enabled in only the
first 5 training iterations. Figure 5: Qualitative results with the
acceleration rate of 4. Blue graph represents the differences between the
fully-sampled image (Ground Truth) and the reconstructed image.

The above experiments show that self-supervised MRI reconstruction methods can
outperform the conventional methods and our method can effectively improve the
performance of the self-supervised model-driven methods regardless of the
acceleration rates utilized. By experimenting with Set A and Set B, we prove
that the performance improvements of the proposed method is not dependent on
the size of the training set. Extensive experiments have been conducted by
utilizing different numbers of K2Calibrate modules during model optimization
with higher acceleration rates of 8x and 10x. Results confirm that K2Calibrate
is effective for noise removal in initial iterations. All these experimental
results validate the performance enhancement of the proposed K2Calibrate for
self-supervised DL-based MRI reconstruction.

####  3.3.2 Qualitative Analysis

Fig.5 depicts the reconstructed images under the acceleration rate of 4 of
conventional and DL-based methods. Based on the residual maps, the
conventional methods suffer from extensive loss of structure details while
both DL-based methods can recover these details more accurately, showing that
DL-based methods are more effective for the recovery of high-frequency
measurements. Building on two state-of-the-art self-supervised deep learning
methods, K2Calibrate can further improve the reconstruction performance with
fewer errors observed in the residual maps. These results demonstrate that the
proposed method can successfully remove the residual artifacts, while
achieving higher qualitative and quantitative performances compared to the
baseline methods.

##  4 Conclusion

In this study, we provide a fresh perspective to review the optimization
process of iterative DL-based reconstruction. We propose K2Calibrate, which is
a plug-and-play module that help the unrolled network gradually reconstruct
K-space measurements. Compared to the existing self-supervised learning
methods that reconstruct all the K-space measurements at once with high
uncertainties and propagate them to the following network training iterations,
K2Calibrate can effectively reduce the error accumulation. Experiments on the
FastMRI dataset confirm that the proposed method can achieve better results
than state-of-the-art self-supervised methods even the same network
architecture is used. Overall, K2Calibrate can improve the performance of
self-supervised reconstruction methods effectively and can be easily plugged
into different model-driven DL-based methods without any additional
operations.

## References

  * [1] Michael Lustig, David Donoho, and John M Pauly.  Sparse mri: The application of compressed sensing for rapid mr imaging.  Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6):1182–1195, 2007\.
  * [2] Justin P Haldar, Diego Hernando, and Zhi-Pei Liang.  Compressed-sensing mri with random encoding.  IEEE transactions on Medical Imaging, 30(4):893–903, 2010.
  * [3] Joshua Trzasko and Armando Manduca.  Highly undersampled magnetic resonance image reconstruction via homotopic $l_{0}$-minimization.  IEEE Transactions on Medical imaging, 28(1):106–121, 2008.
  * [4] Mohammad Shahdloo, Efe Ilicak, Mohammad Tofighi, Emine U Saritas, A Enis Çetin, and Tolga Çukur.  Projection onto epigraph sets for rapid self-tuning compressed sensing mri.  IEEE transactions on medical imaging, 38(7):1677–1689, 2018.
  * [5] Sathish Ramani, Zhihao Liu, Jeffrey Rosen, Jon-Fredrik Nielsen, and Jeffrey A Fessler.  Regularization parameter selection for nonlinear iterative image restoration and mri reconstruction using gcv and sure-based methods.  IEEE Transactions on Image Processing, 21(8):3659–3672, 2012.
  * [6] Dongwook Lee, Jaejun Yoo, Sungho Tak, and Jong Chul Ye.  Deep residual learning for accelerated mri using magnitude and phase networks.  IEEE Transactions on Biomedical Engineering, 65(9):1985–1995, 2018\.
  * [7] Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang.  Accelerating magnetic resonance imaging via deep learning.  In 2016 IEEE 13th international symposium on biomedical imaging (ISBI), pages 514–517. IEEE, 2016.
  * [8] Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen.  Image reconstruction by domain-transform manifold learning.  Nature, 555(7697):487–492, 2018.
  * [9] Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas S Vasanawala, Greg Zaharchuk, Lei Xing, and John M Pauly.  Deep generative adversarial neural networks for compressive sensing mri.  IEEE transactions on medical imaging, 38(1):167–179, 2018.
  * [10] Yoseo Han, Leonard Sunwoo, and Jong Chul Ye.  $k$-space deep learning for accelerated mri.  IEEE transactions on medical imaging, 39(2):377–386, 2019.
  * [11] Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll.  Learning a variational network for reconstruction of accelerated mri data.  Magnetic resonance in medicine, 79(6):3055–3071, 2018.
  * [12] Hemant K Aggarwal, Merry P Mani, and Mathews Jacob.  Modl: Model-based deep learning architecture for inverse problems.  IEEE transactions on medical imaging, 38(2):394–405, 2018.
  * [13] Jian Zhang and Bernard Ghanem.  Ista-net: Interpretable optimization-inspired deep network for image compressive sensing.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1828–1837, 2018.
  * [14] Jian Sun, Huibin Li, Zongben Xu, et al.  Deep admm-net for compressive sensing mri.  Advances in neural information processing systems, 29, 2016.
  * [15] Chen Qin, Jo Schlemper, Jose Caballero, Anthony N Price, Joseph V Hajnal, and Daniel Rueckert.  Convolutional recurrent neural networks for dynamic mr image reconstruction.  IEEE transactions on medical imaging, 38(1):280–290, 2018.
  * [16] Dong Liang, Jing Cheng, Ziwen Ke, and Leslie Ying.  Deep magnetic resonance image reconstruction: Inverse problems meet neural networks.  IEEE Signal Processing Magazine, 37(1):141–151, 2020.
  * [17] Jo Schlemper, Jose Caballero, Joseph V Hajnal, Anthony N Price, and Daniel Rueckert.  A deep cascade of convolutional neural networks for dynamic mr image reconstruction.  IEEE transactions on Medical Imaging, 37(2):491–503, 2017.
  * [18] Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kâmil Uğurbil, and Mehmet Akçakaya.  Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data.  Magnetic resonance in medicine, 84(6):3172–3191, 2020.
  * [19] Burhaneddin Yaman, Seyed Amir Hossein Hosseini, and Mehmet Akcakaya.  Zero-shot physics-guided deep learning for subject-specific mri reconstruction.  In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021\.
  * [20] Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kâmil Uğurbil, and Mehmet Akçakaya.  Ground-truth free multi-mask self-supervised physics-guided deep learning in highly accelerated mri.  In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1850–1854. IEEE, 2021.
  * [21] Chen Hu, Cheng Li, Haifeng Wang, Qiegen Liu, Hairong Zheng, and Shanshan Wang.  Self-supervised learning for mri reconstruction with a parallel network training framework.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 382–391. Springer, 2021.
  * [22] EK Cole, Frank Ong, JM Pauly, and SS Vasanawala.  Unsupervised image reconstruction using deep generative adversarial networks.  ISMRM Work Data Sampling & Image Reconstruction, 2020.
  * [23] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.  Noise2void-learning denoising from single noisy images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2129–2137, 2019.
  * [24] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  In International Conference on Machine Learning, pages 2965–2974. PMLR, 2018.
  * [25] Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al.  fastmri: An open dataset and benchmarks for accelerated mri.  arXiv preprint arXiv:1811.08839, 2018.
  * [26] Frank Ong and Michael Lustig.  Sigpy: a python package for high performance iterative reconstruction.  In Proceedings of the ISMRM 27th Annual Meeting, Montreal, Quebec, Canada, volume 4819, 2019.