# PiRD: Physics-informed Residual Diffusion
for Flow Field Reconstruction

Siming Shana, Pengkai Wanga, Song Chenb, Jiaxu Liub, Chao Xua, Shengze Caia

a Institute of Cyber-Systems & Control, College of Control Science &
Engineering,
Zhejiang University, Hangzhou, 310027, China
b School of Mathematical Sciences, Zhejiang University, Hangzhou, 310027,
China
Corresponding author<EMAIL_ADDRESS>
###### Abstract

The use of machine learning in fluid dynamics is becoming more common to
expedite the computation when solving forward and inverse problems of partial
differential equations. Yet, a notable challenge with existing convolutional
neural network (CNN)-based methods for data fidelity enhancement is their
reliance on specific low-fidelity data patterns and distributions during the
training phase. In addition, the CNN-based method essentially treats the flow
reconstruction task as a computer vision task that prioritizes the element-
wise precision which lacks a physical and mathematical explanation. This
dependence can dramatically affect the models’ effectiveness in real-world
scenarios, especially when the low-fidelity input deviates from the training
data or contains noise not accounted for during training. The introduction of
diffusion models in this context shows promise for improving performance and
generalizability. Unlike direct mapping from a specific low-fidelity to a
high-fidelity distribution, diffusion models learn to transition from any low-
fidelity distribution towards a high-fidelity one. Our proposed model -
Physics-informed Residual Diffusion, demonstrates the capability to elevate
the quality of data from both standard low-fidelity inputs, to low-fidelity
inputs with injected Gaussian noise, and randomly collected samples. By
integrating physics-based insights into the objective function, it further
refines the accuracy and the fidelity of the inferred high-quality data.
Experimental results have shown that our approach can effectively reconstruct
high-quality outcomes for two-dimensional turbulent flows from a range of low-
fidelity input conditions without requiring retraining.

##  1 Introduction

As a typical inverse problem, reconstructing spatial fields from sparse sensor
data is a highly challenging task, yet its applications are widespread,
encompassing numerous fields such as medicine, industrial devices, remote
sensing, fluid dynamics, and more [1, 2, 3, 4]. Efforts to improve traditional
methodologies for effectively handling sparse input data have to face a
balance between computational efficiency and effectiveness [5]. Meanwhile, due
to the potential movement and loss of sparse sensors, the fidelity of
experimental setups utilizing these collected data is inevitably affected,
which leads to structural instability [6]. Moreover, the noise introduced by
the sensor measurement process cannot be ignored. All these aspects lead to a
complex challenge for reconstruction, which is difficult to meet by
traditional methods.

In fluid dynamics, Direct Numerical Simulation (DNS) is a classic method that
simulates fluid motion directly by solving the complete Navier-Stokes
equations in all scales. To obtain high-fidelity flow field information, DNS
requires significant computational resources, especially when solving ill-
posed problems or inverse problems related to turbulent flows. Indeed, owing
to computational efficiency challenges, several methods employing simplified
turbulence models for approximate calculations have been devised, such as
Large Eddy Simulation (LES) [7], Reynolds-Averaged Navier-Stokes (RANS) models
[8] and hybrid RANS-LES approaches [9]. Nonetheless, these approaches require
a trade-off between accuracy and computational expedience. In addition, data
assimilation algorithms are proposed to integrate the measurements with the
aforementioned numerical models. For example, Wang et al.[10, 11] introduced
an adjoint-variational data-assimilation algorithm to reconstruct the flow
fields from measurements and enhance traditional methods in the sparse
condition. However, these methods also suffer from computational inefficiency,
requiring iterative solutions to the Navier-Stokes equations. To mitigate the
conflicts between computational cost and simulation accuracy, we aim to draw
insights from end-to-end models designed for rapid inference to bridge the gap
between simulations and real-world flow field environments, which have
demonstrated success in diverse applications such as image processing.

In the field of image processing, the use of machine learning for super-
resolution enhancement and data recovery techniques from low-resolution or
sparse pixel points is developing rapidly, including algorithms based on
convolutional neural networks (CNN) [12, 13], transformer architectures [14,
15, 16], and generative adversarial networks (GAN) [17, 18, 19]. While super-
resolution is generally regarded as an image-based data recovery technique,
these neural network frameworks have been widely applied to fluid mechanics
[20, 21, 22, 23, 24], providing promising alternatives to traditional methods.

Despite the considerable achievements gained by CNN-based methodologies in
enhancing the resolution of specific low-fidelity image distributions, they
share a common limitation: these models are intricately designed and fine-
tuned for a distinct class of under-resolved flow field data, heavily reliant
on their training data characteristics. In contrast, real-world flow field
scenarios present a far more complex and unpredictable landscape. Sensor
placement is often sparse and varied, and the data acquired is frequently
contaminated with noise. Such conditions inevitably result in discrepancies
between training datasets and real-world testing scenarios. This limitation
presents challenges in the generalization and robustness of CNN-based models,
restricting their applicability in real-world environments.

Revisiting techniques from the field of image reconstruction, if we regard
sparse low-resolution fluid flow data as noisy experimental measurements, then
super-resolution flow field reconstruction can be considered as a natural
denoising problem. Therefore, the diffusion model in the generation problem
[25, 26, 27] also has the potential to reconstruct the sparse data flow field.
In this context, Shu et al. [28] introduced a model based on the Diffusion
Probabilistic Model (DDPM) for super-resolution tasks, demonstrating its
versatility across various scenarios to overcome the limitations in the CNN-
based model used in fluid dynamics.

However, due to the denoising nature of DDPM-based models, rather than
incorporating physics-informed constraints directly into the objective
function, a physics-conditioning embedding is applied during the training
phase. This approach allows the model to correct physics deviations in the
denoised flow field iteratively during the sampling phase. Although effective,
the reliance on the number of sampling steps to achieve minimal partial
differential equation loss may inadvertently compromise the efficiency of the
algorithm. This introduces the need for fine-tuning certain hyperparameters
for specific datasets to achieve accurate results, potentially straying from
the original intent of developing such models. Moreover, despite the
integration of physical condition embedding, the flow field reconstructed
using DDPM occasionally fails to comply with the physical constraints on a
localized scale.

To tackle these challenges, we propose a residual-based diffusion model for
flow field super-resolution in this work, inspired by Yue et al.[29] where a
diffusion model with residual shifting was presented. Unlike traditional DDPM-
based methods, the residual-based diffusion model requires fewer sampling
steps and predicts the original high-fidelity flow field directly. This
approach enables the practical application of Physics-Informed Neural Networks
(PINNs) by integrating the residuals of the governing equations within the
objective function, leading to a brand new model called physics-informed
residual diffusion (PiRD) model in this paper. Our model not only demonstrates
robust performance in reconstructing low-fidelity data across a variety of
conditions in the turbulence experiment, but also enhances both training and
sampling efficiency—requiring approximately 20 epochs for training and 20
steps for sampling.

Contributions: We make the following key contributions: (i) We present a
novel, effective and motivated approach that embeds physical information into
the diffusion model to solve the problem of flow field reconstruction under
sparse and noisy measurements. (ii) Due to the nature of the ResShift
diffusion model, our proposed method can complete the training or inference in
about 20 steps, which greatly improves the calculation speed compared with
[28], and it is beneficial for future application to actual real-time flow
field scenarios. (iii) Through some classical turbulence experiments, PiRD is
proved to be more accurate and robust than the CNN-based model for flow field
reconstruction on sparse data. Moreover, PiRD can handle various sensor
placements and noisy measurements which are not considered in the training
data.

##  2 Methods

###  2.1 Problem Setup

The primary objective of this research is to address the limitations of
current CNN-based methods in reconstructing high-fidelity (HF) flow fields,
denoted as $\omega\sim U_{\text{HF}}$, from low-fidelity (LF) observations,
denoted as $\tilde{\omega}\sim U_{\text{LF}}$. Conventional CNN-based models
like UNet, represented as $f_{\theta}$, have shown proficiency in dealing with
in-distribution low-fidelity data. This process can be described as:

$w=f_{\theta}(\tilde{\omega}),\quad\tilde{\omega}=\mathcal{D}(\omega)$

where $\mathcal{D}$ represents certain the down-sampling operation, or a
certain method to degrade the high-fidelity flow field, the neural network is
essentially trying to learn this operation to reserve this process, and in-
distribution data means the high-fidelity data that was corrupted by this
down-sampling operation $\mathcal{D}$, or in which operations that are highly
similar to it. However, the learned model can only be applicable for the
scenarios that are degraded by this specific learned $\mathcal{D}$, if the
degraded flow field is not corrupted by it, the reconstruction process still
trying to reverse $\mathcal{D}$ which ultimately make the prediction
inaccurate. Therefore, to address this problem, we build up a Markov chain
between high-fidelity flow fields $U_{\text{HF}}$ and low-fidelity flow fields
$U_{\text{LF}}$ that are independent of $\mathcal{D}$, such that we can
reconstruct low-fidelity flow fields given any possible $\mathcal{D}$. In
addition, physics-informed loss function can also be integrated into such a
process to act as not only a physical constraint to make sure that the
prediction follows the underlying physical law, but also a guide for the model
to infer the information lost in the low-fidelity flow field and ensure robust
performance across diverse and challenging scenarios.

###  2.2 Physics-informed Residual Diffusion

This section introduces our proposed model used to map the low-fidelity
distribution to the high-fidelity distribution through a learned Markov chain
and to ensure that the entire process follows the underlying physical law such
that we can eventually produce a high-fidelity prediction that also obeys the
physical law. In the following, we present the architecture of Physics-
informed Residual Diffusion (PiRD) which is composed of two essential modules:
(i) Residual Shifting Diffusion Model that could build a Markov chain between
LF distributions and HF distributions; (ii) Physics-informed neural
networks(PINNs) for fluid mechanics that ensure the predictions obey the
underlying physical law.

####  2.2.1 Residual Shifting Diffusion Model for Vorticity Field

Residual Shifting Diffusion Model was first proposed by Yue et al. [29]
developed for the image super-resolution task. This method diverges from
traditional approaches that transition high-fidelity images to Gaussian noise.
Instead, it constructs a Markov chain between high-resolution images and low-
resolution images, facilitated by adjusting the residual between them. We
apply this method as the backbone of our task, expand this ideal and further
develop this idea into the application of low-fidelity flow field super-
resolution and sparse flow field reconstruction. Let $\omega$ denoted as the
high-fidelity flow field, ad $\tilde{\omega}$ as its corresponding low-
fidelity flow field, and let $e=\tilde{\omega}-\omega$ denoted the residual
between them, this residual does not only contains the spatially related
difference between them but also contains the physical related information
lost between them. The forward diffusion Markov chain can be described as:

$\displaystyle q(\omega_{t}|w_{t-1},\tilde{\omega})$
$\displaystyle=\mathcal{N}(\omega_{t};\omega_{t-1}+\alpha_{t}e,\kappa^{2}\alpha_{t}I),\quad
t=1,2,\ldots,T,$ (1) $\displaystyle q(\omega_{t}|\omega,\tilde{\omega})$
$\displaystyle=\mathcal{N}(\omega_{t};\omega+\eta_{t}e,\kappa^{2}\eta_{t}\\{I),\quad
t=1,2,\ldots,T,$ (2)

where $\\{\eta_{t}\\}_{t=1}^{T}$ is a shifting schedule that increases with
the time step $t$ such that a smooth transition of $\omega$ to
$\tilde{\omega}$ over a sequence of $T$ steps can be built.
$\alpha_{t}=\eta_{t}-\eta_{t-1}$ for $t>1$ and $\alpha_{1}=\eta_{1}$, $\kappa$
is the key parameter that controls the progression of noise variance.
Therefore the reverse process estimates the posterior distribution
$p(w|\tilde{\omega})$ using:

$p(\omega|\tilde{\omega})=\int
p(\omega_{T}|\tilde{\omega})\prod_{t=1}^{T}p_{\theta}(\omega_{t-1}|\omega_{t},\tilde{\omega})\,dw_{1:T},$
(3)

Since in the forward diffusion process we have defined
$\omega_{T}=\omega+\eta_{T}e+\kappa\sqrt{\eta_{T}}\epsilon$, and $\eta_{T}$
$\rightarrow$ 1 we can approximate
$\omega_{T}=(\omega+e)+\kappa\epsilon=\tilde{\omega}+\kappa\epsilon$ where
$\epsilon\sim\mathcal{N}(0,I)$, consequently, we can also approximate
$p(\omega_{T}|\tilde{\omega}$) by
$\mathcal{N}(\omega_{T}|\tilde{\omega},\kappa^{2}I)$.
$p_{\theta}(\omega_{t-1}|\omega_{t},\tilde{\omega})$ is the reverse transition
kernel. By minimizing the negative evidence lower bound to make
$q(x_{t-1}|x_{t},x_{0},y_{0})$ tractable, we can get below representations:

$\min_{\theta}\sum_{t}D_{\text{KL}}[q(\omega_{t-1}|\omega_{t},\omega,\tilde{\omega})\parallel
p_{\theta}(\omega_{t-1}|\omega_{t},\tilde{\omega})],$ (4)
$q(\omega_{t-1}|\omega_{t},\omega,\tilde{\omega})=\mathcal{N}\left(\omega_{t-1};\frac{\eta_{t-1}}{\eta_{t}}\omega_{t}+\frac{\alpha_{t}}{\eta_{t}}\omega,\kappa^{2}\frac{\eta_{t-1}}{\eta_{t}}\alpha_{t}\mathbf{I}\right).$
(5)

The above reverse transition kernel shows that for each iterative backward
process, we need the original high-fidelity flow field $\omega$ to infer from
$\omega_{t}$ to $\omega_{t-1}$, therefore the objective function can be
designed as follows:

$\min_{\theta}\sum_{t}\lVert f_{\theta}(\omega_{t},t)-\omega\rVert^{2}_{2}.$
(6)

####  2.2.2 Physics-informed neural networks (PINNs) for fluid mechanics

Within the realm of fluid mechanics, the application of Physics-Informed
Neural Networks (PINNs) to the vorticity transport equations exemplifies a
sophisticated methodology for solving parametrized partial differential
equations (PDEs) that describe complex fluid behaviours. The vorticity
transport equation, pivotal for understanding fluid dynamics, models the
evolution of vorticity $\omega$, a vector field that characterizes the local
spinning motion of the fluid. This equation is succinctly represented as

$\frac{\partial\omega}{\partial
t}+\nabla\cdot(\mathbf{u}\omega)=\nu\nabla^{2}\omega,$ (7)

where $\mathbf{u}$ denotes the velocity field, $\nu$ the kinematic viscosity.

This equation embodies the conservation of vorticity in the absence of
external influences and lays the foundation for the application of PINNs in
fluid mechanics.

The PINNs framework for solving the vorticity transport equation integrates
the physics of the problem directly into the learning process of the neural
network. By approximating the solution $\hat{\omega}(\mathbf{x},t)$, the
network is trained to comply with the governing PDE across the domain $\Omega$
and throughout the time interval $[0,T]$. This training process is facilitated
by the formulation of a composite loss function $L$, which encapsulates the
adherence of the network’s output to the underlying physical laws and
empirical data. The loss function is articulated as

$L=\beta_{1}L_{\text{PDE}}+\beta_{2}L_{\text{data}}+\beta_{3}L_{\text{IC}}+\beta_{4}L_{\text{BC}},$
(8)

where each term represents a different aspect of the learning criteria:
$L_{\text{PDE}}$ penalizes deviations from the vorticity transport PDE,
$L_{\text{data}}$ quantifies the discrepancies between the predictions and
actual flow measurements, $L_{\text{IC}}$ and $L_{\text{BC}}$ ensure the
model’s output matches the initial and boundary conditions, respectively,
$\beta_{i}$ denotes the weighting coefficient for each term. To integrate the
initial and boundary conditions crucial for accurately modeling fluid
behaviour, PINNs incorporate these conditions directly into the loss function.
Initial conditions set the stage for the fluid’s state at the beginning of the
observation period, whereas boundary conditions dictate the fluid’s behaviour
at the domain’s edges, influencing the evolution of the flow over time. The
precise specification of these conditions is vital for capturing the physical
scenario under investigation, such as the generation and dynamics of vorticity
in flows past obstacles.

An adaptive optimization algorithm, typically ADAM, is employed to minimize
the loss function. This optimization approach adjusts the learning rate
dynamically, enabling efficient convergence towards a parameter set that best
represents the solution to the vorticity transport equation while respecting
both the fluid dynamics and the available empirical data.

Figure 1: The architecture of PiRD, during the forward process, a time-step
$t$ was randomly drawn from $(0,T]$, then a UNet was used for predicting
$\omega$ from $\omega_{t}$ with the objective function that both penalize the
element-wise difference but also the physics loss. During the sampling
process, the LF field $\tilde{\omega}$ was first mapped to $\omega_{T}$ then
followed by the reverse Markov chain and finally predicted HF field $\omega$.

####  2.2.3 Physics-informed Residual Diffusion

The traditional Residual Diffusion Model (RDM) operates by constructing a
Markov chain, characterizing a residual, $e$, that bridges the gap between the
high-fidelity vorticity field, $\omega$, and its low-fidelity counterpart,
$\tilde{\omega}$. This residual is dissected into two principal components:
$e_{\textit{fidelity}}$, which represents the element-wise discrepancies, and
$e_{\textit{physics}}$, which captures deviations from established physical
laws. Consequently, the transformation process from $\tilde{\omega}$ to
$\omega$ is encapsulated by the equation:

$\omega=\tilde{\omega}+e_{\textit{fidelity}}+e_{\textit{physics}}.$ (9)

While $e_{\textit{fidelity}}$ plays a pivotal role in enhancing resolution,
effectively addressing $e_{\textit{physics}}$ poses a significant challenge
due to the complex nature of fluid dynamics, and flow field reconstruction
cannot be simply seen as a mere computer vision task, which prioritizes the
pixel-wise similarity. In many scenarios, it is possible that for a pair of
predictions with the same pixel-wise mean relative error compared to the
ground truth, their respective prediction has large differences in terms of
physics property. Therefore, if the Markov chain could facilitate the
inference of a high-fidelity (HF) vorticity field from a low-fidelity (LF) one
while rigorously adhering to physical laws, the flow field reconstructed could
make sure that it not only revert $e_{fidelity}$ but also $e_{physics}$.
Nonetheless, realizing this ideal scenario requires two often unattainable
conditions in practical applications: (i) the presence of ample training data
to thoroughly represent Markov chain transitions. (ii) LF distribution
representations at the termination of the chain ($X_{T}$) that accurately
reflect all conceivable LF situations, crucial for recovering
$e_{\textit{physics}}$.

To address these challenges, we introduce Physics-informed Residual Diffusion
(PiRD). PiRD enriches the RDM framework by embedding physics-informed
constraints within the model’s loss function, ensuring adherence to physical
laws at every step of the Markov chain. This approach serves a dual purpose:
it acts as a form of data augmentation, introducing crucial information for
model learning, and it can also guarantee that each LF vorticity field is
transitioned towards the HF distribution in compliance with physical
principles without the concern that the physics-wise error will accumulate
during the sampling process. Through this integration, PiRD can effectively
map complex LF vorticity fields to HF fields, ensuring that subsequent
transitions are physically coherent. As a result, the final predicted
$\hat{\omega}$ not only exhibits high fidelity but also aligns with the
fundamental physical principles, surmounting the inherent limitations of
conventional RDM in vorticity super-resolution and reconstruction tasks.

Figure 1 delineates the core architecture of the proposed PiRD model. It
utilizes a Markov chain methodology to facilitate the reconstruction of high-
fidelity flow fields from their low-fidelity counterparts, ensuring adherence
to physical laws throughout the reconstruction process.

The loss function employed in PiRD comprises two components: the element-wise
loss and the PDE loss. The element-wise is the L2 norm between the prediction
$\hat{\omega}$ and the true flow field $\omega$, normalized by the L2 norm of
$\omega$, emphasizing the accuracy of individual elements. The PDE loss
ensures that the predicted field complies with the governing physical
equations, maintaining physical plausibility. Mathematically, the loss
function is expressed as:

$\text{Loss}=\alpha\sum_{t}\|f_{\theta}(\omega_{t},\tilde{\omega})-\omega\|_{2}^{2}+\beta\sum_{t}\|G(f_{\theta}(\omega_{t},\tilde{\omega}))\|_{2}^{2},$
(10)

where $\alpha$ and $\beta$ are the weighting parameters that balance the
significance of data fidelity and physical consistency, respectively. The
differential operator $G$, employed in the loss function, succinctly
represents the dynamics captured by the non-dimensional two-dimensional (2D)
vorticity transport equation for incompressible fluid flow. This crucial
equation is delineated as:

$\frac{\partial\omega}{\partial
t}+(u\cdot\nabla)\omega=\frac{1}{Re}\nabla^{2}\omega,$ (11)

where $\omega$ is the vorticity field, $u$ denotes the velocity field, and
$Re$ is the Reynolds number, characterizing the flow’s inertial forces
relative to its viscous forces. The term $\frac{1}{Re}\nabla^{2}\omega$
accounts for the diffusion of vorticity, with its magnitude scaled inversely
by $Re$.

In 2D incompressible flows, the velocity field $u$ can be derived from the
vorticity field $\omega$ using the stream function $\psi$, where
$\omega=-\nabla^{2}\psi$, and $u$ is obtained from the gradients of $\psi$.
This relationship allows the velocity field to be implicitly determined from
the vorticity field, facilitating the calculation of $G$ as:

$G(\hat{\omega})=\frac{\partial\hat{\omega}}{\partial
t}+(u\cdot\nabla)\hat{\omega}-\frac{1}{Re}\nabla^{2}\hat{\omega}.$ (12)

The ideal outcome, $G(\hat{\omega})=0$, indicates that the predicted vorticity
field adheres to the dynamics governed by the vorticity transport equation.
Hence, the PDE loss is defined as:

$\text{PDE loss}=\frac{1}{n}\sum_{i=1}^{n}\|G(\hat{\omega}_{i})\|_{2}^{2},$
(13)

which penalizes predictions that deviate from expected fluid behavior, thereby
enforcing compliance with fluid dynamics principles.

To effectively compute the time derivative of vorticity, the model utilizes
three channels of input corresponding to vorticity fields at consecutive time
frames, $t-1$, $t$, and $t+1$, notice that the $t$ here is different with the
$t$ in the diffusion model, instead of representing certain time step during
the forward or backward process, the $t$ here means $t$ time frame within the
whole time span of the dataset. This tri-channel approach enables the
approximation of $\frac{\partial\omega}{\partial t}$ via finite difference
methods.

The forward process models the diffusion of the flow field from a high-
fidelity state $\omega$ to a noised low-fidelity state over discrete time
steps $t=1,2,\ldots,T$. This diffusion is simulated as a Gaussian process,
given by:

$\omega_{t}=\omega+\eta_{t}e+\kappa\sqrt{\eta_{t}}\epsilon=\omega+\eta_{t}(\omega-\tilde{\omega})+\kappa\sqrt{\eta_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I)$
(14)

In the sampling process, the reverse Markov chain recovers the high-fidelity
flow field starting from the low-fidelity observations $\tilde{\omega}$. This
process inverts the forward diffusion, iteratively reconstructing the previous
time step $\omega_{t-1}$ from the noised version $\omega_{t}$, until the
original flow field $\omega_{0}$ is retrieved:

$\omega_{t-1}=\frac{\eta_{t-1}}{\eta_{t}}\omega_{t}+\frac{\alpha_{t}}{\eta_{t}}\omega+\kappa\sqrt{\frac{\eta_{t-1}}{\eta_{t}}\alpha_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I)$
(15)

We choose UNet as the underlying neural network architecture because of its
exceptional performance in capturing complex spatial hierarchies inherent in
image-like data, which is analogous to flow fields. With its encoding and
decoding pathways, the architecture’s design is well-suited to preserving
high-resolution details crucial for reconstructing the intricate structures
found in vorticity fields, rendering it an optimal choice for this
superresolution task.

##  3 Experimental Setup

Experiments were designed to evaluate the effectiveness and robustness of the
proposed method, referred to as Physics-informed Residual Diffusion(PiRD), in
reconstructing high-fidelity flow fields across various complex and practical
scenarios using a two-dimensional Kolmogorov flow dataset. The investigation
encompassed multiple tasks under different low-fidelity conditions to simulate
potential challenges. The initial set of conditions involved simple low-
fidelity scenarios where the high-fidelity flow field, originally at
$256\times 256$ pixels, was uniformly down-sampled by a scale of 4 and 8. This
represented $4\times$ and $8\times$ up-sampling tasks, respectively. The
second scenario focused on sparse and randomly-distributed data, where only a
certain percentage of the data was randomly selected for reconstruction tasks.
The sparsity levels considered were $5\%$, and $1.5625\%$ of the original
data, aiming to test the capability of reconstructing the high-fidelity flow
field from significantly reduced data points. The third scenario combined low-
fidelity conditions with varying levels of injected Gaussian noise. For this
set of experiments, the low-fidelity flow field was set down-sampled by a
scale of 4, and a sparse flow field with $1.5625\%$ of the original data.
Gaussian noise was then introduced to these low-fidelity flow fields during
the testing stage at densities ranging from $20\%$ to $100\%$. This scenario
was designed to assess the resilience of the PiRD method in handling noisy,
low-fidelity inputs.

###  3.1 Evaluation Metrices

For a quantitative assessment of the reconstruction, we adopt the Mean
Relative Error (MRE) which is the element-wise l2 norm between the prediction
and ground truth data, and divided by a normalization factor which is the l2
norm of the ground truth. We also calculate the PDE loss to evaluate how much
our prediction deviates from the underlying physical law. These two aforesaid
evaluation matrices can be described as follows:

$MRE(\hat{\omega},\omega)=\frac{\|\hat{\omega}-\omega\|_{2}}{\|\omega\|_{2}},$
(16)
$Loss_{\text{PDE}}(\hat{\omega},\omega)=\frac{1}{n}\sum_{i=1}^{n}|G(\hat{\omega}_{i})|^{2},$
(17)

where $\tilde{\omega}$ represents the predicted flow field, $\omega$
represents the original high-fidelity flow field, $G$ represents a
differential operator that is applied to the predicted flow field.

###  3.2 Dataset

The dataset we consider is a 2-dimensional Kolmogorov flow same with that in
Shu et al.[28], which is governed by the Navier-Stokes equations for
incompressible flow. The vorticity equation that characterizes this flow is
given by:

$\displaystyle\frac{\partial\omega(x,t)}{\partial
t}+u(x,t)\cdot\nabla\omega(x,t)$
$\displaystyle=\frac{1}{Re}\nabla^{2}\omega(x,t)+f(x),\quad
x\in(0,2\pi)^{2},\,t\in(0,T],$ (18) $\displaystyle\nabla\cdot u(x,t)$
$\displaystyle=0,\quad x\in(0,2\pi)^{2},\,t\in(0,T],$ (19)
$\displaystyle\omega(x,0)$ $\displaystyle=\omega_{0}(x),\quad
x\in(0,2\pi)^{2}.$ (20)

where $\omega$ represents vorticity, $u$ is the velocity field, $Re$ is the
Reynolds number (set to 1000 for this dataset), and $f(x)$ is a forcing term
defined as $f(x)=-4\cos(4x^{2})-0.1\omega(x,t)$, to mitigate energy
accumulation at larger scales. The boundary condition of this dataset is
periodic, which is also considered in the loss function in our model training.
The dataset contains 40 simulations, each with a temporal span of 10 seconds
partitioned into 320 frames, and it was down-sampled spatially to a $256\times
256$ grid from a $2048\times 2048$ grid for a more efficient training and less
requirement for computational resources.

###  3.3 Model Training

The dataset was partitioned into training ($80\%$), validation ($10\%$), and
testing ($10\%$) sets. Optimization strategies are based on a cosine annealing
learning rate, the maximum number of epochs was set to 200, the maximum and
minimum learning rate was set to $1e^{-3}$ and $1e^{-5}$ respectively, $T$ was
set to 20, and $\kappa$ was set to 2 to ensure the generalizability of
$\omega_{T}$. We set the weights for the data term and PDE term as 0.7 and
$2e^{-4}$, respectively. The entire training process was conducted on a single
RTX3090 GPU over approximately 3 hours.

##  4 Results

###  4.1 Comparison with current super-resolution methods

In this section, we conduct a quantitative analysis comparing our Physics-
informed Residual Diffusion (PiRD) model against two state-of-the-art super-
resolution methods: a modified UNet with ResNet block and attention block
which is similar to the model proposed by Pant et al.[30], and a physics-
informed Denoising Diffusion Probabilistic Model (DDPM) as described by Shu et
al. [28]. More specifically, two versions of UNet are trained, one is trained
for a direct mapping for a super-resolution task scaling from $64\times 64$ to
$256\times 256$, while for the second version of UNet, Gaussian noises are
first added to the $64\times 64$ input, then is trained to map to the
$256\times 256$ output in order to mimic the process that a denoising
diffusion model performs, we therefore denote this UNet as Noisy UNet, both
UNet models are trained with the loss function including physical laws. To
ensure a fair comparison, our PiRD is also trained for a super-resolution task
scaling from $64\times 64$ to $256\times 256$. Notice that, unlike the DDPM-
based method—which requires only the high-fidelity vorticity field for
training—our PiRD model necessitates prior knowledge of the residual between
high-fidelity (HR) and low-fidelity (LR) fields.

Subsequent tests evaluate the Mean Relative Error (MRE) and Partial
Differential Equation (PDE) loss across four different tasks: $4\times$ up-
sampling, $8\times$ up-sampling, $5\%$, and $1.5625\%$ reconstruction to
$256\times 256$ resolution. For our training and testing, we first use bicubic
interpolation on the up-sampling task and Voronoi tessellation on the
reconstruction task to make the input matches the desired dimension of our
output ($256\times 256$), we then use nearest interpolation on the up-sampling
tasks for testing the result, such that all the testing distributions and
patterns are UNSEEN for PiRD, UNet, and Noisy Unet for fair comparison. We
also iterate each batch 10 times and take the average of the flow fields as
our final prediction. This approach is designed to reduce the influence of any
randomness in the predictive outcomes. For training PiRD, $T=20$ steps for
sampling in each iteration and $\kappa$ was set to 2. In comparison, results
for the physics-informed DDPM are generated using the hyper-parameters
recommended by the original authors, with Discrete Denoising Implicit Models
(DDIM) sampling steps set at 30 and 60 respectively.

Figure 2: Flow field reconstruction from four different low-fidelity inputs (i.e., $4\times$ downsampling, $8\times$ downsampling, $5\%$ random sampling, and $1.5625\%$ random sampling). A zoomed-in comparison among PiRD, DDPM-based, and UNet methods for three testing cases. Table 1: Quantitative Comparison: MRE means the Mean Relative Error, PDE means Partial Differential Equation residual of the predicted vorticity field. The best and second best results are highlighted in bold and underline respectively. Note that the models are trained with 4x up-sampling data only.  Methods | 4x | 8x | 5% | 1.5625%
---|---|---|---|---
MRE |  PDE |  MRE |  PDE |  MRE |  PDE |  MRE |  PDE
PiRD(ours) |  0.1178 |  2.0663 |  0.3580 |  2.1594 |  0.2782 |  1.6939 |  0.3857 |  2.0917
DDPM-based - 30 |  0.2832 |  8.0110 |  0.5251 |  7.4526 |  0.2811 |  3.7741 |  0.3912 |  14.7641
DDPM-based - 60 |  0.2844 |  5.1598 |  0.5258 |  4.6567 |  0.2881 |  2.1297 |  0.4378 |  3.8925
Noisy UNet |  0.2190 |  10.0515 |  0.4015 |  8.7777 |  0.3315 |  6.6934 |  0.4073 |  6.7641
UNet |  0.1207 |  259.0664 |  0.3995 |  699.3475 |  0.3478 |  668.0065 |  0.4981 |  998.8791

As illustrated in Table 1, UNet model struggles across all the tested
scenarios particularly in terms of the PDE loss, reflecting its inability to
generalize to unseen low-fidelity datasets encountered during training.
Although it achieves the second lowest MRE on the 4x up-sampling task, it
performs worst on recovering the underlying physical law. The extremely high
PDE loss validates our opinion that the flow field reconstruction cannot be
merely seen as a computer vision task, which may result in a successful
recovery of element-wise information, but unacceptable physical-wise
information. This can be further validated by visualizing the sampled flow
fields in Figure 2. As shown, UNet model fails to generate reasonable or
smooth flow fields, especially when the low-fidelity input contains randomly
distributed measurements (e.g., the $5\%$ and $1.5625\%$ reconstruction
tasks).

Conversely, the PiRD, DDPM-based and Noisy UNet methods maintained low and
reasonable PDE losses, indicating successful recovery of the underlying
physical laws during the reconstruction process while containing a low mean
relative error. Notably, PiRD performs best on all scenarios on both MRE and
PDE loss, which validates our assumption that all different low-fidelity
scenarios can first be mapped into a collective noisy LF distribution and then
one can utilize the learned reverse Markov chain to sample the LF flow fields
back to the HF distribution.

It is noteworthy that the DDPM-based method’s performance on PDE loss is
dependent on the number of sampling steps employed; increasing the steps from
30 to 60 tends to reduce PDE loss but at the expense of higher MRE. This
trade-off arises because the DDPM-based method involves subtracting the PDE
deviation relative to the vorticity field at each sampling step — a process
that is challenging to manage without potentially compromising MRE
performance. In contrast, our PiRD model excels in predicting the high-
fidelity field with both a remarkably low MRE and an exceptionally low PDE
loss, showcasing its effectiveness and robustness in the flow field
reconstruction task. For a comprehensive and varied examination of PiRD’s
performance, we invite readers to consult the Appendix section, where the
reconstructed flow fields with different inputs are presented.

###  4.2 Evaluation on kinetic energy spectrum and vorticity distribution

The kinetic energy spectrum is a crucial metric in turbulence research and is
used to understand the distribution of energy among different scales within
the flow field. It characterizes how kinetic energy varies with the
wavenumber, revealing the flow’s energy cascade from larger to smaller scales.
It is important to highlight that despite UNet’s mean relative error (MRE)
being comparable to that of PiRD for the task of 4x up-sampling, as indicated
in Table 1, its performance on the kinetic energy spectrum is relatively poor,
as shown in Figure 3(a). Although it performs well in recovering large-scale
vortices, it fails to retrieve the vorticity at higher wavenumber. This
discrepancy further validates our view that the accurate recovery of element-
wise data does not necessarily translate to successfully retrieving physically
meaningful information. On the contrary, our method demonstrates a robust
capability to recover the energy distribution of the turbulent flow as shown
in Figure 3(a). The spectrum obtained by our technique aligns closely with the
ground truth across a range of wavenumbers, particularly in the inertial
subrange, indicating that our method successfully captures the scale-dependent
distribution of kinetic energy. The slight deviation at higher wavenumbers can
be attributed to the inherent challenges in resolving the smallest scales of
turbulence.

Moreover, the vorticity distribution provides insight into the probability of
finding certain vorticity magnitudes within the flow. A method that accurately
captures the vorticity distribution from coarse data suggests it can retain
critical features of the turbulence structure, such as vortex stretching and
tilting mechanisms. The vorticity distribution shown in Figure 3(b) further
confirms the efficacy of our proposed approach. The recovered distribution
exhibits a strong agreement with the ground truth, especially in capturing the
peak and the tails of the distribution, which are indicative of the extreme
vorticity events in the flow.

(a) Kinetic energy spectrum

(b) Vorticity distribution

Figure 3: Comparison of PiRD’s performance on (a) kinetic energy spectrum and
(b) vorticity distribution.

###  4.3 Evaluation on noisy low-fidelity vorticity field

In this section, we investigate the robustness of PiRD against a progressive
density of Gaussian noise injected into the low-fidelity input, the noise
injection and prediction process can be described below

$\hat{\omega}=f_{\theta}(\tilde{\omega}+\alpha\epsilon),\quad\epsilon\sim\mathcal{N}(0,I)$
(21)

where $\alpha$ ranges from 0.2 to 1.0 by a step size of 0.2, the low-fidelity
field is down-sampled from the high-fidelity one by a scale of 4, and randomly
selected 1.5625% of the high-fidelity data respectively, and then normalized
to Gaussian distribution before noise is injected. Table 2 illustrates the
performance of MRE and PDE loss of PiRD and UNet under different noise levels
for a 4x and a 1.5625% reconstruction task respectively. As shown in Figure 4,
as the injected noise density increases, UNet’s performance degrades
significantly. On the contrary, PiRD has stably and accurately predicted the
HF field for both scenarios. In addition, as illustrated in Figure 5, the
kinetic energy spectrum and vorticity distribution appear identical across all
noise densities compared to the case where no noise is injected. This
observation suggests that PiRD is highly robust to noise interference.
Although there is a slight degradation in MRE as the noise density increases,
the performance on PDE loss, vorticity distribution, and kinetic energy
spectrum remains consistent.

Table 2: Comparison of UNet and PiRD performance for 4x and $1.5625\%$ reconstruction tasks at various noise levels. The models are trained with clean 4x up-sampling data without any noise.  Noise Level | MRE | Equation Loss
---|---|---
4x | $1.5625\%$ | 4x | $1.5625\%$
UNet |  PiRD |  UNet |  PiRD |  UNet |  PiRD |  UNet |  PiRD
0.0 |  0.1207 |  0.1178 |  0.4981 |  0.3857 |  259.0664 |  2.0663 |  998.8791 |  2.0917
0.2 |  0.1560 |  0.1181 |  0.5045 |  0.3881 |  409.8747 |  2.0563 |  1232.7024 |  2.0917
0.4 |  0.2247 |  0.1178 |  0.5101 |  0.3893 |  801.6868 |  2.1201 |  1537.9834 |  2.0837
0.6 |  0.2990 |  0.1206 |  0.5132 |  0.3901 |  1350.3967 |  2.1059 |  1823.7584 |  2.0946
0.8 |  0.3723 |  0.1245 |  0.5160 |  0.3903 |  2006.1515 |  2.3082 |  2093.8465 |  2.3822
1.0 |  0.4435 |  0.1318 |  0.5190 |  0.3912 |  2733.3027 |  2.7030 |  2560.6975 |  2.5433
Figure 4: MRE and PDE loss against different levels of injected noise for PiRD
and UNet. The left plot is tested based on the 4x up-sampling task, and the
right plot is tested on the 1.5625% reconstruction task. Figure 5: Comparison
of PiRD’s performance on kinetic energy spectrum and vorticity distribution
for different densities of noise. The models are trained with clean 4x up-
sampling data without any noise.

##  5 Conclusion

This study highlights the inherent limitations of conventional CNN-based
methods in reconstructing flow fields. Specifically, these methods tend to
perform optimally only on data that mirrors the distribution or down-sampling
processes they were originally trained on. A mere adaptation of computer
vision techniques to flow field reconstruction tasks often leads to results
that, while visually plausible, may not adhere strictly to the mathematical
and physical principles governing the flow dynamics.

Our proposed model, PiRD, is expected to address these challenges by
establishing a Markov chain that links low-fidelity and high-fidelity flow
fields. This approach not only ensures the retention of physical laws
throughout the training and sampling processes but also exhibits remarkable
resilience to various down-sampling operations. By prioritizing both element-
wise accuracy and adherence to physical laws, PiRD may set a new benchmark in
performance across experimental datasets. PiRD is demonstrated to be promising
for flow field reconstruction tasks, merging the realms of deep learning and
fluid dynamics with unprecedented efficacy.

As the first attempt to apply the ResShift diffusion model for the flow field
reconstruction problem, the proposed method still has some limitations. While
PiRD demonstrates enhanced accuracy and robustness over CNN-based models when
dealing with unseen sparse and noisy measurements, its performance can be
further optimized when the training and testing sets are under the same
distribution. In the near future, incorporating more controllable physical
flow information into diffusion model training, as an alternative to the
conventional stacking of Gaussian noises, may be a promising direction for
developing a more accurate and generalizable flow generation model.

## References

  * [1] Kevin Course and Prasanth B Nair.  State estimation of a physical system with unknown governing equations.  Nature, 622(7982):261–267, 2023.
  * [2] N Benjamin Erichson, Lionel Mathelin, Zhewei Yao, Steven L Brunton, Michael W Mahoney, and J Nathan Kutz.  Shallow neural networks for fluid flow reconstruction with limited sensors.  Proceedings of the Royal Society A, 476(2238):20200097, 2020.
  * [3] Huanfeng Shen, Xinghua Li, Qing Cheng, Chao Zeng, Gang Yang, Huifang Li, and Liangpei Zhang.  Missing information reconstruction of remote sensing data: A technical review.  IEEE Geoscience and Remote Sensing Magazine, 3(3):61–85, 2015.
  * [4] Saiprasad Ravishankar, Jong Chul Ye, and Jeffrey A Fessler.  Image reconstruction: From sparsity to data-adaptive methods and machine learning.  Proceedings of the IEEE, 108(1):86–109, 2019.
  * [5] Steven L Brunton, Bernd R Noack, and Petros Koumoutsakos.  Machine learning for fluid mechanics.  Annual review of fluid mechanics, 52:477–508, 2020.
  * [6] Keyi Wu, Peng Chen, and Omar Ghattas.  A fast and scalable computational framework for large-scale high-dimensional bayesian optimal experimental design.  SIAM/ASA Journal on Uncertainty Quantification, 11(1):235–261, 2023.
  * [7] Ugo Piomelli.  Large-eddy simulation: achievements and challenges.  Progress in Aerospace Sciences, 35(4):335–362, 1999.
  * [8] Giancarlo Alfonsi.  Reynolds-averaged navier–stokes equations for turbulence modeling.  2009\.
  * [9] Stefan Heinz.  A review of hybrid RANS-LES methods for turbulent flows: Concepts and applications.  Progress in Aerospace Sciences, 114:100597, 2020.
  * [10] Mengze Wang and Tamer A Zaki.  State estimation in turbulent channel flow from limited observations.  Journal of Fluid Mechanics, 917:A9, 2021.
  * [11] Mengze Wang, Qi Wang, and Tamer A Zaki.  Discrete adjoint of fractional-step incompressible Navier-Stokes solver in curvilinear coordinates and application to data assimilation.  Journal of Computational Physics, 396:427–450, 2019.
  * [12] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  * [13] Chunwei Tian, Yong Xu, Wangmeng Zuo, Bob Zhang, Lunke Fei, and Chia-Wen Lin.  Coarse-to-fine cnn for image super-resolution.  IEEE Transactions on Multimedia, 23:1489–1502, 2020.
  * [14] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.  Activating more pixels in image super-resolution transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22367–22377, 2023.
  * [15] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5791–5800, 2020.
  * [16] Ziwei Luo, Youwei Li, Shen Cheng, Lei Yu, Qi Wu, Zhihong Wen, Haoqiang Fan, Jian Sun, and Shuaicheng Liu.  Bsrt: Improving burst super-resolution with swin transformer and flow-guided deformable alignment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 998–1008, 2022.
  * [17] Chenyu You, Guang Li, Yi Zhang, Xiaoliu Zhang, Hongming Shan, Mengzhou Li, Shenghong Ju, Zhen Zhao, Zhuiyang Zhang, Wenxiang Cong, et al.  Ct super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE).  IEEE Transactions on Medical Imaging, 39(1):188–203, 2019.
  * [18] Sefi Bell-Kligler, Assaf Shocher, and Michal Irani.  Blind super-resolution kernel estimation using an internal-GAN.  Advances in Neural Information Processing Systems, 32, 2019.
  * [19] Kui Jiang, Zhongyuan Wang, Peng Yi, Guangcheng Wang, Tao Lu, and Junjun Jiang.  Edge-enhanced gan for remote sensing image superresolution.  IEEE Transactions on Geoscience and Remote Sensing, 57(8):5799–5812, 2019.
  * [20] Romit Maulik, Kai Fukami, Nesar Ramachandra, Koji Fukagata, and Kunihiko Taira.  Probabilistic neural networks for fluid flow surrogate modeling and data recovery.  Physical Review Fluids, 5(10):104401, 2020.
  * [21] Han Gao, Luning Sun, and Jian-Xun Wang.  Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels.  Physics of Fluids, 33(7), 2021.
  * [22] Luning Sun and Jian-Xun Wang.  Physics-constrained bayesian neural network for fluid flow reconstruction with sparse and noisy data.  Theoretical and Applied Mechanics Letters, 10(3):161–169, 2020.
  * [23] Amirhossein Arzani, Jian-Xun Wang, and Roshan M D’Souza.  Uncovering near-wall blood flow from sparse data with physics-informed neural networks.  Physics of Fluids, 33(7), 2021.
  * [24] Tomoki Asaka, Katsunori Yoshimatsu, and Kai Schneider.  Machine learning-based vorticity evolution and super-resolution of homogeneous isotropic turbulence using wavelet projection.  Physics of Fluids, 36(2), 2024.
  * [25] Roger Ratcliff, Philip L Smith, Scott D Brown, and Gail McKoon.  Diffusion decision model: Current issues and history.  Trends in cognitive sciences, 20(4):260–281, 2016.
  * [26] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  Advances in Neural Information Processing Systems, 34:21696–21707, 2021.
  * [27] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [28] Dule Shu, Zijie Li, and Amir Barati Farimani.  A physics-informed diffusion model for high-fidelity flow field reconstruction.  Journal of Computational Physics, 478:111972, 2023.
  * [29] Zongsheng Yue, Jianyi Wang, and Chen Change Loy.  Resshift: Efficient diffusion model for image super-resolution by residual shifting.  Advances in Neural Information Processing Systems, 36, 2024.
  * [30] Pranshu Pant and Amir Barati Farimani.  Deep learning for efficient reconstruction of high-resolution turbulent dns data.  arXiv preprint arXiv:2010.11348, 2020.

## Appendix

Figure 6: PiRD’s performance on 4x upsampling task compares with other
benchmarks.

Figure 7: PiRD’s performance on 8x upsampling task compares with other
benchmarks.

Figure 8: PiRD’s performance on 5% reconstruction task compares with other
benchmarks.

Figure 9: PiRD’s performance on 1.5625% reconstruction task compares with
other benchmarks.


# Real-Time Reinforcement Learning for Vision-Based Robotics
Utilizing Local and Remote Computers

Yan Wang∗†, Gautham Vasan∗†, A. Rupam
Mahmood${}^{{\dagger}\stackon[0.5ex]{\rule{0.4pt}{4.52083pt}}{\rule{2.2604pt}{0.4pt}}}$
*Equal Contribution.
†Department of Computing Science, University of Alberta, Edmonton AB., Canada,
T6G 2E8
${}^{\stackon[0.5ex]{\rule{0.4pt}{4.52083pt}}{\rule{2.2604pt}{0.4pt}}}$ CIFAR
AI Chair, Alberta Machine Intelligence Institute (Amii)
Email: $\\{$ yan28, vasan<EMAIL_ADDRESS>
Video: https://youtu.be/7iZKryi1xSY

###### Abstract

Real-time learning is crucial for robotic agents adapting to ever-changing,
non-stationary environments. A common setup for a robotic agent is to have two
different computers simultaneously: a resource-limited local computer tethered
to the robot and a powerful remote computer connected wirelessly. Given such a
setup, it is unclear to what extent the performance of a learning system can
be affected by resource limitations and how to efficiently use the wirelessly
connected powerful computer to compensate for any performance loss. In this
paper, we implement a real-time learning system called the _Re_ mote-_Lo_ cal
_D_ istributed (ReLoD) system to distribute computations of two deep
reinforcement learning (RL) algorithms, Soft Actor-Critic (SAC) and Proximal
Policy Optimization (PPO), between a local and a remote computer. The
performance of the system is evaluated on two vision-based control tasks
developed using a robotic arm and a mobile robot. Our results show that SAC’s
performance degrades heavily on a resource-limited local computer. Strikingly,
when all computations of the learning system are deployed on a remote
workstation, SAC fails to compensate for the performance loss, indicating
that, without careful consideration, using a powerful remote computer may not
result in performance improvement. However, a carefully chosen distribution of
computations of SAC consistently and substantially improves its performance on
both tasks. On the other hand, the performance of PPO remains largely
unaffected by the distribution of computations. In addition, when all
computations happen solely on a powerful tethered computer, the performance of
our system remains on par with an existing system that is well-tuned for using
a single machine. ReLoD is the only publicly available system for real-time RL
that applies to multiple robots for vision-based tasks. The source code can be
found at https://github.com/rlai-lab/relod

##  I INTRODUCTION

Building robotic agents capable of adapting to their environments based on
environmental interactions is one of the long-standing goals of embodied
artificial intelligence. Such a capability entails learning on the fly as the
agent interacts with the physical world, also known as real-time learning.
When learning in real-time, the real world does not pause while the agent
computes actions or makes learning updates (Mahmood et al. 2018a, Ramstedt &
Pal 2019). Moreover, the agent obtains sensorimotor information from various
onboard devices and executes action commands at a specific frequency. Given
these constraints, a real-time learning agent must compute an action within a
chosen action cycle time and perform learning updates without disrupting the
periodic execution of actions (Yuan & Mahmood 2022).

Figure 1: Our proposed system ReLoD distributes computations of a learning
system between a local and a remote computer.

Reinforcement Learning (RL) is a natural way of formulating real-time control
learning tasks. Although many deep RL methods have been developed to solve
complex motor control problems (Schulman et al. 2017, Abdolmaleki et al. 2018,
Haarnoja et al. 2018) they do not easily extend to the real-time learning
setting that operates under time and resource constraints, for example, in
quadrotors and mobile robot bases. While approaches including learning from
demonstrations (Gupta et al. 2016, Vasan & Pilarski 2017), sim-to-real (Peng
et al. 2018, Bousmalis et al. 2018), and offline RL (Levine et al. 2020) have
been used to develop pre-trained agents, there has been relatively little
interest in studying real-time learning in the real world.

State-of-the-art RL algorithms such as SAC are computationally intensive, and
hence for real-time robotic control, they go together with a computationally
powerful computer tethered to the robot (Yuan & Mahmood 2022). On the other
hand, a robot agent deployed in the real world typically uses a tethered
resource-limited computer and a wirelessly connected workstation (Haarnoja et
al. 2019, Bloesch et al. 2021). In this paper, we use _local_ to refer to the
computer tethered to the robot and _remote_ to refer to the wirelessly
connected computer. Computations of a learning system using these two
computers can be distributed in different ways (e.g., see Fig. 1). However, it
is unclear how much the performance of a learning system is impacted due to
the wireless connection or resource limitations. Moreover, prior works do not
systematically study distributions of computations between local and remote
computers nor suggest how to achieve an effective distribution.

In this paper, we develop two vision-based tasks using a robotic arm and a
mobile robot, and propose a real-time RL system called the _Re_ mote-_Lo_ cal
_D_ istributed (ReLoD) system. Similarly to Yuan and Mahmood’s (2022) work,
ReLoD parallelizes computations of RL algorithms to maintain small action-
cycle times and reduce the computational overhead of real-time learning. But
unlike the prior work, it is designed to utilize both a local and a remote
computer. ReLoD supports three different modes of distribution: _Remote-Only_
that allocates all computations on the remote computer, _Local-Only_ that
allocates all computations on the local computer, and _Remote-Local_ that
carefully distributes the computations between the two computers in a specific
way.

Our results show that the performance of SAC on a tethered resource-limited
computer drops substantially compared to its performance on a tethered
powerful workstation. Surprisingly, when all computations of SAC are deployed
on a wirelessly connected powerful workstation, the performance does not
improve notably, which contradicts our intuition since this mode fully
utilizes the workstation. On the other hand, SAC’s Remote-Local mode
consistently improves its performance by a large margin on both tasks, which
indicates that a careful distribution of computations is essential to utilize
a powerful remote workstation. However, the Remote-Local mode only benefits
computationally expensive and sample efficient methods like SAC since the
relatively simpler learning algorithm PPO learns similar policies in all three
modes. We also notice that the highest average return attained by PPO is about
one-third of the highest average return attained by SAC, which indicates that
SAC is more effective in complex robotic control tasks.

Our system in the Local-Only mode can achieve a performance that is on par
with a system well-tuned for a single computer (Yuan & Mahmood 2022), though
the latter overall learns slightly faster. This property makes our system
suitable for conventional RL studies as well.

##  II RELATED WORK

A system comparable to ours is SenseAct, which provides a computational
framework for robotic learning experiments to be reproducible in different
locations and under diverse conditions (Mahmood et al. 2018b). Although
SenseAct enables the systematic design of robotic tasks for RL, it does not
address how to distribute computations of a real-time learning agent between
two computers, and the original work does not contain vision-based tasks. We
use the guiding principles of SenseAct to design the vision tasks and
systematically study the effectiveness of different distributions of
computations of a learning agent.

Krishnan et al. (2019) introduced an open-source simulator and a gym
environment for quadrotors. Given that these aerial robots need to accomplish
their tasks with limited onboard energy, it is prohibitive to run current
computationally intensive RL methods on the onboard hardware. Since onboard
computing is scarce and updating RL policies with existing methods is
computationally intensive, they carefully designed policies considering the
power and computational resources available onboard. However, they focused on
sim-to-real techniques for learning, making their approach unsuited for real-
time learning.

Nair et al. (2015) proposed a distributed learning architecture called the
GORILA framework that mainly focuses on using multiple actors and learners to
collect data in parallel and accelerate training in simulation using clusters
of CPUs and GPUs. GORILA is conceptually akin to the DistBelief (Dean et al.
2012) architecture. In contrast to the GORILA framework, our system focuses
primarily on how best to distribute the computations of a learning system
between a resource-limited local computer and a powerful remote computer to
enable effective real-time learning. In addition, the GORILA framework is
customized to Deep Q-Networks (DQN), while our system supports two different
policy gradient algorithms using a common agent interface.

Lambert et al. (2019) used a model-based reinforcement learning approach for
high-frequency control of a small quadcopter. Their proposed system is similar
to our Remote-Only mode. A recent paper by Smith et al. (2022) demonstrated
real-time learning of walking gait from scratch on a Unitree A1 quadrupedal
robot on various terrains. Their real-time synchronous training of SAC on a
laptop is similar to our Local-Only mode. The effectiveness of both these
approaches on vision-based tasks is untested.

Bloesch et al. (2021) used a distributed version of Maximum aposteriori Policy
Optimization (MPO) (Abdolmaleki et al. 2018) to learn a vision-based control
policy that can walk with Robotis OP3 bipedal robots. The robot’s onboard
computer samples actions and periodically synchronizes the policy’s neural
network weights with a remote learning process at the start of each episode.
Haarnoja et al. (2019) also proposed a similar asynchronous learning system
tailored to learn a stable gait using SAC and the minitaur robot (Kenneally et
al. 2016). These tasks do not use images. Although their proposed systems are
similar to our Remote-Local mode, these two papers aim at solving tasks
instead of systematically comparing different distributions of computations of
a learning agent between a resource-limited computer and a powerful computer.
In addition, their systems are tailored to specific tasks and algorithms and
are not publicly available, while our system is open-source, task-agnostic,
and compatible with multiple algorithms.

##  III Background

Reinforcement learning is a setting where an agent learns to control through
trial-and-error interactions with its environment. The agent-environment
interaction is modeled with a Markov Decision Process (MDP), where an agent
interacts with its environment at discrete timesteps. At the current timestep
${t}$, the agent is at state $S_{t}\in\mathcal{S}$, where it takes an action
$A_{t}\in\mathcal{A}$ using a probability distribution $\pi$ called a _policy_
: $A_{t}\sim\pi(\cdot|S_{t})$. At the subsequent timestep ${t+1}$, the agent
receives the next state $S_{t+1}$ and reward $R_{t+1}$ according to a
transition probability density function $S_{t+1},R_{t+1}\sim
p(\cdot,\cdot|S_{t},A_{t})$. An episode ends when the agent arrives at a
terminal state. The state space $\mathcal{S}$ and the action space
$\mathcal{A}$ of the tasks used in this paper are continuous.

###  III-A Soft Actor-Critic

Soft Actor-Critic (SAC) is an off-policy RL algorithm where the policy is
trained to maximize a trade-off between the expected return and the policy’s
entropy (Haarnoja et al. 2018). SAC learns a policy $\pi_{\phi}(A_{t}|S_{t})$
by maximizing the following objective:

$\displaystyle J_{\pi}(\phi)$
$\displaystyle=\mathbb{E}_{S_{t}\sim\mathcal{D}}\left[\mathbb{E}_{A_{t}\sim\pi_{\phi}}[\alpha\log{\pi_{\phi}(A_{t}|S_{t})}-Q_{\theta}(S_{t},A_{t})]\right],$

which is based on an entropy-augmented version of the standard RL objective
(Ziebart 2010). Here $\phi$ denotes the policy parameters, and
$Q_{\theta}(S_{t},A_{t})$, parameterized by $\theta$, denotes an estimate of
the Soft-Q function $q_{\pi}$ defined as
$q_{\pi}(s,a)=E_{\pi}\left[\sum_{t=0}^{T-1}R_{t+1}-\alpha\sum_{t=1}^{T-1}\log\pi(A_{t}|S_{t})\big{|}S_{0}=s,A_{0}=a\right]$.

The parameters of the soft Q-function are trained to minimize the following
objective:

$\displaystyle J_{Q}(\theta)$
$\displaystyle=\mathbb{E}_{D}\left[\frac{1}{2}\left(R_{t+1}+\gamma
V_{\tilde{\theta}}(S_{t+1})-Q_{\theta}(S_{t},A_{t})\right)^{2}\right],$

where $V_{\tilde{\theta}}$ is defined as

$\displaystyle V_{\tilde{\theta}}(S_{t})$
$\displaystyle=\mathbb{E}_{\pi}\left[Q_{\theta}(S_{t},A_{t})-\alpha\log{\pi_{\phi}(A_{t}|S_{t})}\right],$

with $\tilde{\theta}$ being the parameters of the target network. In this
paper, we use the undiscounted episodic formulation. In other words, $\gamma$
is set to 1.

###  III-B Proximal Policy Optimization (PPO)

PPO (Schulman et al. 2017) is an on-policy policy gradient algorithm that
alternates between collecting data via interactions with the environment and
optimizing the following clipped surrogate objective:

$\displaystyle L_{\theta}^{\text{CLIP}}=$
$\displaystyle\mathbb{E}_{S_{t},A_{t}\sim\pi_{\theta_{old}}}[\min(\rho_{\theta}(A_{t}|S_{t}){H}_{\theta_{\text{old}}}(S_{t},A_{t}),$
$\displaystyle\quad\text{clip}(\rho_{\theta}(A_{t}|S_{t}),1-\epsilon,1+\epsilon){H}_{\theta_{\text{old}}}(S_{t},A_{t}))],$

where
$\rho_{\theta}(A_{t}|S_{t})=\frac{\pi_{\theta}(A_{t}|S_{t})}{\pi_{\theta_{\text{old}}}(A_{t}|S_{t})}$
is the action probability ratio between the new policy and the old policy, and
$H_{\theta_{\text{old}}}(S_{t},A_{t})$ is an estimate of the advantage
function. The function
$\text{clip}(\rho_{\theta}(A_{t}|S_{t}),1-\epsilon,1+\epsilon)$ clips the
probability ratio to be no more than $1+\epsilon$ if the advantage is positive
and no less than $1-\epsilon$ if the advantage is negative. The policy is
optimized by running several iterations of stochastic gradient ascent at each
update.

##  IV TASKS

For this paper, we develop two visuomotor robotic tasks, named
_UR5-VisualReacher_ and _Create-Reacher_ , that are not easy to solve using
simple RL approaches since they require tens of thousands of images to learn
good state representations and control policies. The robots we used in this
paper are the UR5 industrial robot arm and iRobot Create2 mobile robot. All
environments are designed using the guidelines proposed by Mahmood et al.
(2018a) to reduce the computational overhead and take advantage of the
multiprocessing capability of modern computers for handling different
transmission rates.

###  IV-A UR5-VisualReacher

This environment is the same task proposed by Yuan and Mahmood (2022). It is
an image-based task that aims to move a UR5 arm’s fingertip to a random target
designated by a red (the target color) blob on a monitor. The movement of the
UR5 arm is restricted within a bounding box to prevent collisions and
accidental damage. The action space is the desired angular velocities for the
five joints between $[-0.7,0.7]$rad/s. The observation vector includes joint
angles, joint angular velocities, three consecutive images of dimension
$160\\!\times\\!90\\!\times\\!3$ taken by a camera, and the previous action.
Every 40ms, the SenseAct environment process actuates the five joints and
receives the next observation by sending low-level control commands. The
SenseAct communicator process transmits joint data every 8ms and image data
every $33.\bar{3}$ms to the environment process. Since this environment is
inherently non-Markovian, we stack three images to provide the partial history
to the learning agent. Each episode lasts 4 seconds (i.e., 100 timesteps). The
reward function is designed to encourage the following behaviors: 1) moving
the fingertip closer to the target while keeping the target centralized, and
2) avoiding abrupt twisting of the joints. Specifically, the reward function
is defined as follows (Yuan & Mahmood (2022)):

$r_{t}=\alpha\frac{1}{hw}M\circ
W-\beta\left(\bigg{|}\pi-\sum_{n=1}^{3}\omega_{n}\bigg{|}+\bigg{|}\sum_{n=4}^{5}\omega_{n}\bigg{|}\right),$

where $h$ is the height and $w$ is the width of the image observation, $M$ is
a mask matrix whose element $M_{i,j}$ is 1 if the target color is detected at
location $(i,j)$ and 0 otherwise, $W$ is a constant weight matrix whose
elements range from 0 to 1 to encourage centralization of the target, and
$\omega$ is the vector of the five joints’ angles. The coefficient $\alpha$
and $\beta$ weigh the importance of the two behaviors. In this paper, we set
$\alpha=800$ and $\beta=1$. During a reset, a new random target will be
displayed on the monitor for the next episode, and the UR5 arm will be set to
a predefined posture. The environment runs on a Lenovo Ideapad Y700 laptop.

(a)

(b)

Figure 2: The SAC-Agent in the Remote-Local Mode. The lengths of the
transition queues on the local computer and the remote computer are set to the
maximum episode length to avoid losing data.

###  IV-B Create-Reacher

This image-based task aims to move Create2 as soon as possible to a target
designated by a green piece of paper attached to a wall. The target is thought
to be reached if it occupies more than 12% pixels of the current image. The
size of the image is $160\\!\times\\!120\\!\times\\!3$. The movement of
Create2 is limited by an arena of size $110\text{cm}\times 70\text{cm}$ built
by wood and cardboard. The action is the desired speeds for the two wheels,
limited to $[-150,150]$mm/s to prevent damage to Create2. The observation
vector includes the values of the six wall sensors, three consecutive images
taken by a camera, and the previous action. Every 45ms, the SenseAct
environment process actuates the two wheels and receives the next observation
by sending low-level control commands. The SenseAct communicator process
transmits the sensory data every 15ms and the image data every $33.\bar{3}$ms
to the environment process. Similar to UR5-VisualReacher, we stack three
images to give the partial history to the learning agent. Each episode has a
maximum length of 30 seconds. The current episode will terminate earlier if
Create2 reaches the target. The reward is $-1$ per step to encourage shorter
episodes. The reset routine will bring Create2 to a random location in the
arena. During a reset, Create2 first moves backward and reorients itself in a
random direction. Our reset routine also handles recharging when the battery
life falls below a specified threshold. This environment runs on a Jetson Nano
computer.

##  V  The proposed ReLoD system

| Remote-Local | Remote-Only | Local-Only | Period of Iteration
---|---|---|---|---
Agent Interface Process (AIP) | Local | Local | Local | action cycle time
Action computation | Included in AIP | Included in LIP | Included in AIP | action cycle time
Local-Send Process | Local | N/A | N/A | action cycle time
Local-Receive Process | Local | N/A | N/A | Every $k$ iterations of LIP’s loop
Learner Interface Process (LIP) | Remote | Remote | Included in AIP | action cycle time
Replay Buffer Process | Remote | Remote | Local | hardware dependent
Update Process | Remote | Remote | Local | hardware dependent
TABLE I: A list of all processes and their distributions between the local and
the remote computers in various operational modes of the ReLoD system.

In this section, we describe our proposed ReLoD system, which distributes the
computational components of a real-time RL agent between a local computer and
a remote computer. Most deep RL systems have three computationally expensive
components: 1) Action Computation, 2) Policy Updating, and 3) Minibatch
Sampling. ReLoD spawns parallel processes for each of the three components and
runs them in an asynchronous, distributed manner to better utilize the
computational resources of the local and remote computers. It also creates
additional inter-process communication sockets to enable this distributed and
parallel architecture. ReLoD supports three different modes of operation:
Remote-Only, Remote-Local, and Local-Only. The mode of operation determines
how the various processes are distributed between the local computer and the
remote computer. ReLoD is designed to work with various RL algorithms. Users
can easily add more RL algorithms to the system by implementing the agent
interface. In this paper, we implement two state-of-the-art RL algorithms: SAC
and PPO. We call the resulting agents _SAC-Agent_ and _PPO-Agent_ ,
respectively.

Fig. 2 outlines the structure of the SAC-Agent in the Remote-Local mode. There
are three processes, called Agent-Environment Interface, Local-Send, and
Local-Receive, that run in parallel on the local computer. The Agent-
Environment Interface process computes the actions to take in the environment
(except in Remote-Only mode given in Table I). Three more processes, called
Learner Interface, Replay Buffer, and Update, run in parallel on the remote
computer. The Replay Buffer process samples mini-batches, and the Update
process updates policies. By distributing the three most computationally
expensive components of SAC on two different computers, ReLoD takes advantage
of both computers. On the other hand, in the Remote-Only mode, all agent
computations happen on the remote computer, including action computation, and
the local computer relays observations and actions between the remote computer
and the robot. More details about the process distribution for all three modes
of the SAC-Agent are given in Table I.

Note that the learning algorithm determines what processes to use in each
mode. For example, the Replay Buffer and the Update processes are specific to
our asynchronous SAC implementation adapted from Yuan and Mahmood (2022). The
PPO-Agent does not have them due to the nature of the algorithm. It updates
policies in the Learner Interface process instead.

##  VI EXPERIMENTS & RESULTS

(a)

(b)

(c)

Figure 3: Comparison of the learning performance across three operational
modes of ReLoD on UR5-VisualReacher. The wide lines for each mode are averaged
over five independent runs. The real experience time in our plots does not
include any additional components required to run an experiment, such as
resetting the environment or recharging the robot. Fig. 3(a) shows the
comparison of the learning performance of the SAC-Agent. Fig. 3(b) shows the
comparison of the learning performance of the PPO-Agent. Fig. 3(c) shows the
comparison of the Local-Only mode of the SAC-Agent with an off-the-shelf
asynchronous SAC implementation proposed by Yuan and Mahmood (2022) on a
powerful workstation.

In this paper, we first compared the performance of Local-Only SAC between a
resource-limited local computer and a powerful local workstation, referred to
as Local-WStation-Only, to evaluate the impact on performance due to resource
limitations. The default hyper-parameters provided by Yuan and Mahmood (2022)
were used in this paper. We used the IdeaPad Y700 laptop as the resource-
limited computer. It has an Intel i7-6700HQ CPU, an NVidia GTX 960M GPU, and
16GB memory. We reduced SAC’s mini-batch size to 64 on the laptop to fit the
hardware constraints. The maximum allowable buffer size was also reduced
compared to the workstation, within which size of 16000 performed well.

Considering that heavily resource-limited computers like Jetson Nano are
commonly used as the local controlling computers for small robots, we also
aimed to investigate how SAC would have performed on Jetson Nano. However,
when ReLoD was configured to Local-Only SAC, Jetson Nano could not run the
system fully onboard as its hardware and software architectures were not
optimized for updating network weights. Jetson Nano took about half a second
to complete one network update on average and became unresponsive after a
couple of updates. To overcome the architectural limitations and get an
empirical upper bound of performance on such a heavily resource-limited
computer, we emulated the capability of Jetson Nano with the laptop by
restricting its update rate and available memory. Thus, two Local-Only
variants of SAC were tested on the laptop:

  * •

_Local-Laptop-Only_ : This variant fully utilizes the computational resources
of the laptop. The policy is updated at the fastest allowable speed, that is,
back-to-back.

  * •

_Local-JNanoEm-Only_ : This variant emulates the capability of Jetson Nano.
Since the average time required for one SAC update on Jetson Nano is $500ms$
and the action cycle time of UR5-VisualReacher is $40ms$, the policy is
updated once per 12 steps on the laptop to best match the capability of Jetson
Nano. The buffer size is set to 16000, a maximum that fits Nano’s 4GB RAM.

We used UR5-VisualReacher for this comparison as mounting the laptop to
Create2 was impractical. The workstation for UR5-VisualReacher has an AMD
Ryzen Threadripper 2950 processor, an NVidia 2080Ti GPU, and 128G memory. The
results are shown in Fig 3(a).

Fig 3(a) shows that the performance of SAC on the resource-limited laptop
dropped by about 28% on average compared to that of SAC on the workstation.
Moreover, the Local-JNanoEm-Only variant struggled to learn a comparable
policy within the same time frame. Note that SAC’s Local-WStation-Only
configuration attained the highest return among all our results.

Since tethering a workstation to a mobile robot is inconvenient in most cases,
we investigated how to utilize the resource of a remote workstation to
compensate for the performance loss due to the resource-limited local
computer. We used the most common setting in practice in which the robot is
tethered to a small resource-limited computer and wirelessly connected to a
powerful workstation. To get better coverage of RL algorithms, two typical
algorithms, SAC and PPO, were tested. PPO was adapted from Achiam (2018), and
we extended it to support learning with images. The default hyper-parameters
provided by Achiam (2018) were used for PPO.

SAC is resource-demanding since it updates the policy every step and requires
a large replay buffer to store all its past experiences. PPO is relatively
simpler since it only updates the policy every few thousand steps, and its
buffer needs to hold only a few episodes generated on-policy. The SAC-Agent
was tested on UR5-VisualReacher and Create-Reacher, while the PPO-Agent was
only tested on UR5-VisualReacher. As the local computer, we used a Jetson Nano
4GB for Create-Reacher and the laptop for UR5-VisualReacher. The workstation
for Create-Reacher has an AMD Ryzen Threadripper 3970 CPU, an NVidia 3090 GPU,
and 128G memory.

The learning curves of the three modes of the SAC-Agent on UR5-VisualReacher
and Create-Reacher are shown in Fig. 3(a) and Fig. 4, respectively. Note that
for the same reason mentioned above, SAC-Agent’s Local-Only mode was not run
for Create-Reacher. Fig 3(a) reveals a counterintuitive result that SAC’s
Remote-Only mode, referred to as _Remote-WStation-Only_ in the Figures, barely
improved its performance over the Local-Only mode, which used a resource-
limited laptop. Meanwhile, the Remote-Only mode also exhibited higher variance
in the overall learning performance on both tasks. This is surprising since
this mode fully utilizes all resources of the remote workstation. We attribute
the relatively poor performance of the Remote-Only mode to the variable
latency in communicating actions over WiFi, which will not occur in the other
two modes, as the Remote-Only mode computes actions on a remote computer.
Although the Remote-Local mode also encounters latency due to WiFi
communication, the effect is much less severe as it only involves the transfer
of policies and buffer samples, while action computation still happens
locally. Our results are in line with Mahmood et al.’s (2018a) results showing
that delays occurring closer to robot actuations can be substantially more
detrimental to performance compared to delays occurring further from
actuations. A real-time learning system can get affected by delays at least
through two pathways: inference for action computation and learning update.
When a delay occurs within the system, it may spare the more important
inference pathway, as in our case. Whereas when a delay occurs closer to the
robot actuation, that is, at the periphery of the learning system, it affects
both pathways.

Fig. 3(a) and Fig. 4 suggest that SAC-Agent’s Remote-Local mode, referred to
as _Remote-WStation-Local-Laptop_ and _Remote-WStation-Local-JNano_ in the
Figures, consistently compensated for the performance loss on both tasks. As
Fig. 3(a) indicates, the highest average return attained by SAC-Agent’s
Remote-Local mode is more than 90% of the highest average return attained by
SAC in the Local-WStation-Only configuration, whose performance is the best.
However, the Remote-Local mode only benefits computationally intensive
algorithms like SAC, as Fig. 3(b) shows that PPO’s performance is nearly the
same across the three modes.

By comparing Fig. 3(a) and Fig. 3(b), we notice that SAC significantly
outperformed PPO on complex robotic tasks. All distribution modes of the SAC-
Agent except the Local-JNanoEm-Only attained more than twice the return
attained by PPO in a shorter time frame. Nevertheless, when all computations
of SAC happen on a computer incapable of performing frequent policy updates,
indicated by Local-JNanoEm-Only, SAC’s performance degrades substantially to a
degree comparable to that of PPO on the same computer.

Figure 4: Comparison of learning performance across two modes of SAC-Agent on
Create-Reacher. The wide lines for each mode are averaged over five
independent runs.

Finally, on a powerful tethered workstation, we compare the performance of
Local-Only SAC to that of the system proposed by Yuan and Mahmood (2022),
which is well-tuned for a single computer. This test is to determine if the
performance of the Local-Only mode of our system is comparable to the off-the-
shelf RL algorithm implementation given adequate resources. The environment
was UR5-VisualReacher, and five independent runs were performed for each
system. The comparison results are shown in Fig. 3(c). Both methods achieved
similar overall learning performance. Thus, our proposed system can leverage
the power of deep RL algorithms effectively, and as a result, it is also
suitable for conventional RL research.

##  VII Conclusion

In this paper, we introduced the ReLoD system for learning to control robots
by a real-time RL agent that distributes its computations between a local and
a remote computer. In our experiments with ReLoD, we demonstrated that when
SAC ran on a resource-limited computer, its performance could be dramatically
reduced. Moreover, solely deploying all computations of SAC on a wirelessly
connected remote workstation may not improve performance due to latency. On
the other hand, our results suggest that performing local action computation
and remote policy updating compensates substantially for SAC’s performance
loss. However, this distribution may not benefit all RL algorithms since PPO’s
performance was nearly unaffected by the distribution mode.

ReLoD provides a useful tool and guidance for real-time RL in vision-based
robotic control. Computationally intensive algorithms like SAC can best
utilize ReLoD’s Remote-Local mode. Moreover, ReLoD’s Local-Only mode is
suitable for conventional RL research as its performance was shown to be on
par with a well-tuned single-computer system. However, due to the latency in
communicating actions, ReLoD’s Remote-Only mode should only be used when the
local computer cannot compute actions within the action cycle time.

##  VIII Acknowledgements

We gratefully acknowledge funding from the CCAI Chairs program, the RLAI
laboratory, Amii, and NSERC of Canada. We also thankfully acknowledge the
donation of the UR5 arm from the Ocado Group.

##  IX References

Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., &
Riedmiller, M. (2018). Maximum a posteriori policy optimisation. In
International Conference on Learning Representations.

Achiam, J. (2018). Spinning Up in Deep Reinforcement Learning.

Bloesch, M., Humplik, J., Patraucean, V., Hafner, R., Haarnoja, T., Byravan,
A., . . . others (2021). Towards real robot learning in the wild: A case study
in bipedal locomotion. In 5th Annual Conference on Robot Learning.

Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M.,
. . . others (2018). Using simulation and domain adaptation to improve
efficiency of deep robotic grasping. In 2018 IEEE international conference on
robotics and automation (ICRA) (pp. 4243–4250).

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., … & Ng, A.
(2012). Large scale distributed deep networks. Advances in neural information
processing systems, 25.

Gupta, A., Eppner, C., Levine, S., & Abbeel, P. (2016). Learning dexterous
manipulation for a soft robotic hand from human demonstrations. In 2016
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
(pp. 3786–3793).

Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2019).
Learning to walk via deep reinforcement learning. In Robotics: Science and
Systems.

Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic:
Off-policy maximum entropy deep reinforcement learning with a stochastic
actor. In International conference on machine learning (pp. 1861–1870).

Kenneally, G., De, A., & Koditschek, D. E. (2016). Design principles for a
family of direct-drive legged robots. IEEE Robotics and Automation Letters,
1(2), 900– 907.

Krishnan, S., Boroujerdian, B., Fu, W., Faust, A., & Reddi, V. J. (2019). Air
learning: An ai research platform for algorithm-hardware benchmarking of
autonomous aerial robots. arXiv preprint arXiv:1906.00421.

Lambert, N. O., Drew, D. S., Yaconelli, J., Levine, S., Calandra, R., &
Pister, K. S. (2019). Low-level control of a quadrotor with deep model-based
reinforcement learning. IEEE Robotics and Automation Letters, 4(4), 4224–4230.

Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020).
Reinforcement learning with augmented data. Advances in Neural Information
Processing Systems, 33, 19884–19895.

Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020). Offline reinforcement
learning: Tutorial, review, and perspectives on open problems. arXiv preprint
arXiv:2005.01643.

Mahmood, A. R., Korenkevych, D., Komer, B. J., & Bergstra, J. (2018a). Setting
up a reinforcement learning task with a real-world robot. In 2018 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS) (pp.
4635-4640). IEEE.

Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., & Bergstra, J. (2018b).
Benchmarking reinforcement learning algorithms on real-world robots. In
Confer- ence on robot learning (pp. 561–591).

Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., De Maria,
A., . . . others (2015). Massively parallel methods for deep reinforcement
learning. arXiv preprint arXiv:1507.04296.

Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Sim-to-real
transfer of robotic control with dynamics randomization. In 2018 IEEE
international conference on robotics and automation (ICRA) (pp. 3803–3810).

Ramstedt, S., & Pal, C. (2019). Real-time reinforcement learning. Advances in
neural information processing systems, 32.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017).
Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to
walk in 20 minutes with model-free reinforcement learning. arXiv preprint
arXiv:2208.07860.

Vasan, G., & Pilarski, P. M. (2017). Learning from demonstration: Teaching a
myoelectric prosthesis with an intact limb via reinforcement learning. In 2017
International Conference on Rehabilitation Robotics (ICORR) (pp. 1457–1464).

Yuan, Y., & Mahmood, A. R. (2022). Asynchronous reinforcement learning for
real-time control of physical robots. In 2022 IEEE international conference on
robotics and automation (ICRA).

Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the
principle of maximum causal entropy. Carnegie Mellon University.