# Hybrid analysis and modeling for next generation of digital twins

Suraj Pawar1 Shady E. Ahmed1 Omer San1 Adil Rasheed2 1School of Mechanical &
Aerospace Engineering, Oklahoma State University, Stillwater, OK 74078, USA.
2Department of Engineering Cybernetics, Norwegian University of Science and
Technology, 7465 Trondheim, Norway<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The physics-based modeling has been the workhorse for many decades in many
scientific and engineering applications ranging from wind power, weather
forecasting, and aircraft design. Recently, data-driven models are
increasingly becoming popular in many branches of science and engineering due
to their non-intrusive nature and online learning capability. Despite the
robust performance of data-driven models, they are faced with challenges of
poor generalizability and difficulty in interpretation. These challenges have
encouraged the integration of physics-based models with data-driven models,
herein denoted hybrid analysis and modeling (HAM). We propose two different
frameworks under the HAM paradigm for applications relevant to wind energy in
order to bring the physical realism within emerging digital twin technologies.
The physics-guided machine learning (PGML) framework reduces the uncertainty
of neural network predictions by embedding physics-based features from a
simplified model at intermediate layers and its performance is demonstrated
for the aerodynamic force prediction task. Our results show that the proposed
PGML framework achieves approximately 75% reduction in uncertainty for smaller
angle of attacks. The interface learning (IL) framework illustrates how
different solvers can be coupled to produce a multi-fidelity model and is
successfully applied for the Boussinesq equations that govern a broad class of
transport processes. The IL approach paves the way for seamless integration of
multi-scale, multi-physics and multi-fidelity models ($M^{3}$ models).

##  1 Introduction

Some of the major challenges in realizing the potential of wind energy to meet
the global electricity demand are the need for a deeper understanding of the
physics of the atmospheric flow, science, and engineering of these large
dynamic rotating machines and synergistic optimization and control of fleets
of wind farms within the electricity grid [1]. The decades of research and
development in fluid dynamics, systems engineering, manufacturing processes,
material discovery can now be complemented with unprecedented amounts of data
generated from in-situ measurements, lab experiments, and numerical
simulations to tackle these challenges. The combination of physics-based and
data-driven models is increasing in every branch of science leading to the
hybrid analysis and modeling (HAM) approach for many scientific applications.

The HAM paradigm can be applied to a variety of tasks related to wind
research, such as digital twinning for the optimization and real-time control
of wind farms. A digital twin is defined as a virtual replica of a physical
system enabled through data collected from sensors and simulations in real-
time to solve problems such as control, optimization, monitoring, and improved
decision-making [2]. The advancement in multiphysics solver, computational
infrastructure, big data, and artificial intelligence has allowed us to make a
digital replica of large physical systems such as wind farms [3]. The digital
twin lets us examine _what if_ scenarios, evaluate the system’s response, and
select the corresponding mitigation strategies. Figure 1 shows the typical
digital twin framework for a wind farm. The digital twin of wind farms can be
useful for different purposes such as optimal control of wind turbines to
achieve the maximum performance, routine maintenance of the equipment,
accurate forecast of the power production, wind farm optimization [4], and
improved decision-making.

Figure 1: Digital twin of the wind farm can allow us to evaluate different
scenarios using hybrid models. The big data collected from IoT sensors can be
continuously assimilated to correct hybrid models and improve the state
estimation.

The success of the digital twins depends upon the type of approach that we
employ for modeling the system. The ability of the HAM to combine the
generalizability of physics-based approaches and the automatic pattern-
identification feature of data-driven approaches makes it an attractive choice
to model virtual replica of physical systems in digital twins. For example,
there are several turbine wake models that capture the flow in the wake of a
single turbine and these models predict sufficiently accurate aerodynamics of
a single turbine [5, 6]. However, these models are not enough to capture the
flows in wind farm due to wake superposition, complex terrains, deep array
effects, and neglected physics [7, 8, 9]. The observational data collected
from a variety of sensors can be assumed to comprise of these complex flow
interactions and the manifestation of all physical processes. Therefore, the
hybrid model will aid in the robustness of the digital twin rather than a pure
physics-based or pure data-driven approach.

In this work, we introduce two different frameworks under the umbrella of HAM.
The first framework is called physics-guided machine learning (PGML) where
information from simplified physics-based models is incorporated within neural
network architectures to improve the generalizability of data-driven models
[10]. The second framework is our interface learning approach that deals with
coupling different solvers and mathematical descriptions using statistical
inference tools [11, 12, 13]. One of the potential applications of this
framework can be to specify the physically accurate inlet boundary condition
for the wind farm simulation. For example, it is very common to use a coarse-
grid solver to resolve the atmospheric boundary layer flow and employ the
fine-grid solver to resolve the flow field around wind turbines in the wind
farm. For such problems, it is very important to exchange the information
between two solvers that leads to physically consistent boundary conditions,
and statistical tools like deep learning can be exploited for this.

Concisely, bringing physical realism in digital twins will need

  * •

new modelling approaches that are accurate and certain, generalizable,
computationally efficient, and trustworthy,

  * •

seamless integration of multi-scale, multi-physics and multi-fidelity models
($M^{3}$ models).

To this end, we propose two HAM approaches that address the above needs. While
the first approach PGML provides a mechanism to guide the learning of a
machine learning algorithm using simplified physics, the second approach IL
enables coupling of $M^{3}$ models.

##  2 Physics-guided machine learning

A wide variety of problems in wind energy such as aerodynamic performance
prediction [14, 15] for optimal control, design optimization, uncertainty
quantification requires the prediction of the quantity of interest in real-
time. The prediction of flow around an airfoil is a high-dimensional, and
nonlinear problem that can be solved using high-fidelity methods like
computational fluid dynamics (CFD). However, these methods are computationally
infeasible for real-time prediction. On one hand, in certain flow regimes, the
simplified methods like panel codes come with a non-negligible difference
between the actual dynamics and approximate models for real-world problems. On
the other hand, the full-order CFD simulations are computationally demanding,
thus limiting their use in many inverse modeling methodologies that require a
model run to be performed in each iteration. To overcome these challenges,
combining CFD models with machine learning to build a non-intrusive surrogate
model is gaining widespread popularity [16, 17]. One of the main challenges
with these non-intrusive models is their predictive capabilities for unseen
data and their interpretation. Even though there are methods to predict the
uncertainty estimate in the prediction of machine learning models [18, 19],
the generalizability of non-intrusive models is not on par with physics-based
models. Meanwhile, the simplified models like the Blasius boundary layer model
in fluid mechanics are highly generalizable across different conditions.
Therefore, it is important to incorporate the domain knowledge into learning,
and to this end, we exploit the relevant physics-based features from panel
methods through the PGML framework to enhance the generalizability of data-
driven surrogate model.

Figure 2: Physics guided machine learning (PGML) framework to train a learning
engine between processes $A$ and $B$: (a) a conceptual PGML framework which
shows a predictor-corrector approach to incorporate physics into machine
learning models through embedding simplified theories directly into neural
network models, and (b) the representative neural network architecture of the
PGML framework used in this study for aerodynamic forces prediction task.

We now introduce different components of the PGML framework depicted in Figure
2(a). In supervised machine learning, the input vector
$\mathbf{x}\in\mathbb{R}^{m}$ is fed to the machine learning model (for
example, the neural network in our case), and the mapping from the input
vector to the output vector $\mathbf{y}\in\mathbb{R}^{n}$ is learned through
training. The neural network is trained to learn the function $F_{\theta}$,
parameterized by $\theta$, that includes the weights and biases of each
neuron. The parameters of the neural network are optimized using the
backpropagation algorithm to minimize the cost function. Usually, for the
regression problems, the cost function is the mean squared error between true
and predicted output, i.e.,
$C(\mathbf{x},\theta)=||\mathbf{y}-F_{\theta}(\mathbf{x})||_{2}$. In the PGML
framework, the neural network is augmented with the output of the simplified
physics-based model. The features extracted from simplified physics-based
models are embedded into hidden layers along with latent variable. This is in
contrast to admitting physics-based features at the input layer in
conventional neural network architectures, which might lead to underestimation
of the effect of such physical information, especially for high dimensional
systems. For example, imagine stacking a few parameters to an input vector of
a dimension of $O(10^{3}-10^{6})$. It is highly probable that the learning
algorithm overlooks the effect of such prior knowledge in the minimization
algorithm, and a modification of the cost function becomes necessary. In the
PGML, a proper latent space is first identified and the information from the
physics-based model aids the neural network in constraining its output to a
manifold of physically realizable models.

The training data for the neural network is generated using a series of
numerical simulations performed in XFOIL [20]. We highlight here that the the
neural network can also be trained using the data gathered from CFD
simulations or wind-tunnel experiments. The lift coefficient data were
obtained for different Reynolds numbers $Re$ between $1\times 10^{6}$ and
$4\times 10^{6}$ and several angles of attacks $\alpha$ in the range of $-20$
to $+20$. A total of 168 sets of two-dimensional airfoil geometry were
generated from NACA 4-digit, NACA210, NACA220, and NACA250 series for training
the neural network. Each airfoil is represented by 201 points. The maximum
thickness of all airfoils in the training dataset was between 6% to 18% of the
chord length. We use the NACA23012 and NACA23024 airfoil geometry as the test
dataset to evaluate the predictive capability of the trained neural network.
The simplified model used to generate the physics-based feature corresponds to
the Hess-Smith panel method [21] based on potential flow theory. We note here
that our testing airfoils are selected not only from a different NACA230
series (i.e., not used in the training dataset), but also the maximum
thickness of 24% is well beyond the thickness ratio limit included in the
training dataset.

The adopted neural network architecture has four hidden layers with 20 neurons
in each hidden layer. The physical parameters, i.e., the Reynolds number and
the angle of attack are concatenated at the third hidden layer along with the
latent variables at that layer. In the PGML model, we augment the latent
variables at the third layer with the lift coefficient and the pressure drag
coefficient predicted by the panel method along with physical parameters of
the flow (i.e., the Reynolds number and angle of attack). Therefore, the third
layer of the neural network in the PGML framework has 24 latent variables. The
representative neural network for the PGML framework to predict the
aerodynamic forces on an airfoil is displayed in Figure 2(b). We utilize an
ensemble of neural networks trained using different initialization to predict
the epistemic uncertainty [22, 23]. The weights and biases of each model are
initialized using the Glorot uniform initializer and different random seed
numbers are used to ensure that different values of weights and biases are
assigned for each model. The ensemble of all these models indicates the model
uncertainty estimate of the predicted lift coefficient. Figure 3 shows the
actual and predicted lift coefficient for the NACA23012 and NACA23024 airfoil
geometry. The reference _True_ performance is obtained by XFOIL. The ML
corresponds to a simple feed-forward neural network that uses the airfoil $x$
and $y$ coordinates as the input features, and the physical parameters of the
flow are concatenated at the third hidden layer along with the latent
variables at that layer.

Figure 3: Actual versus predicted lift coefficient ($C_{L}$) for NACA23012 and
NACA23024 airfoils at $Re=3\times 10^{6}$ using ML and PGML framework. The
dashed blue curve represent the average of the predicted lift coefficient by
all data-driven models (i.e., testing runs with different initialization
seeds).

As shown in Figure 3, we can see that the uncertainty in the prediction of the
lift coefficient by the PGML model is considerably less than the ML model for
both NACA23012 and NACA23024 airfoils. The proposed PGML framework provides
significantly more accurate predictions with uncertainty reduced approximately
by 75% especially for the angle of attacks between -10 and +12 degrees. This
further illustrates the viability of the proposed PGML framework, since the
physics embedding considered here employs constant source panels and a single
vortex to approximate the potential flow around the airfoil. We can also
notice that the uncertainty is higher for the angle of attacks outside the
range of -10 to +12 degrees. This finding is not surprising as the Hess-Smith
panel method is a proven method for analysis of inviscid flow over airfoil for
the smaller angle of attacks regime. The maximum thickness of an airfoil
included in the training dataset is 18% of the chord length. Therefore, the
uncertainty in the prediction of the lift coefficient by the ML model is
higher for the NACA23024 airfoil compared to the NACA23012 airfoil. These
results clearly show the potential of the PGML framework for building
trustworthy models that can enable the digital twin of physical systems.

##  3 Interface learning

The second framework we are introducing under the umbrella of HAM is the
interface learning (IL). Multi-scale, multi-physics and multi-fidelity models
($M^{3}$ models) are the main beneficiaries from the IL methodology. Many
complex systems relevant to scientific and engineering applications include
multiple spatiotemporal scales and comprise a multifidelity problem sharing an
interface between various formulations or heterogeneous computational
entities. We refer the readers to our previous discussion about the potential
of IL approaches, with demonstrations on truncated domains [12], and mico-
macro scale solvers coupling [11]. In the present study, we are interested in
situations where part of the domain, physics, or scales are characterized by
repeating coherent structures, and can thus be represented by a reduced order
model (ROM) for computational speed-up. In the meantime, a high-fidelity full
order model (FOM) is dedicated for the rest of domain/dynamics for accuracy
requirements. However, both solvers are coupled and information should be
communicated and matched at their _interface_. To this end, we present a
robust HAM approach combining a physics-based FOM and a data-driven ROM to
form the building blocks of an integrated approach among mixed fidelity
descriptions toward predictive digital twin technologies, as depicted in
Figure 4(a).

Figure 4: Hybrid analysis and modeling (HAM) as a key enabler for ROM-FOM
coupling problems toward predictive digital twins: (a) an overview, and (b)
proposed ROM-FOM coupling frameworks.

###  3.1 ROM-FOM coupling framework

In order to demonstrate the ROM-FOM coupling framework, we consider a coupled
system as follows,

$\displaystyle\dfrac{\partial u}{\partial
t}=f_{1}(u;\mu_{1})+g_{1}(u,v;\mu_{1},\mu_{2}),$ (1)
$\displaystyle\dfrac{\partial v}{\partial
t}=f_{2}(v;\mu_{2})+g_{2}(u,v;\mu_{1},\mu_{2}),$ (2)

where $u$ and $v$ are the coupled variables and $g_{1}$ and $g_{2}$ define
this coupling, while $\mu_{1}$ and $\mu_{2}$ denote the set of system’s
parameters. We highlight that the coupled variables might represent the state
variables at different regions of the domain (e.g., multi-component systems),
different physics (e.g., fluid-structure interactions) and/or different scales
within the same domain (e.g., multiscale systems). We suppose that the
dynamics of $u$ can be approximated by a ROM while a FOM resolves $v$ and both
solvers need to communicate information to satisfy the coupling.

####  3.1.1 Reduced order model

Introducing a spatial discretization to Eq. (1), it can be rewritten in a
semi-discrete continuous-time as follows,

$\dfrac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}=\mathcal{F}(\mathbf{u},\mathbf{v};\boldsymbol{\mu})=\mathcal{L}_{1}\mathbf{u}+\mathcal{L}_{2}\mathbf{v}+\mathcal{N}(\mathbf{u},\mathbf{v}),$
(3)

where the boldfaced symbols represent the arrangement of discretized variables
in a column vector, $\boldsymbol{\mu}$ defines the system’s parameters, and
$\mathcal{F}$ is a deterministic operator with linear and nonlinear components
$\mathcal{L}$, and $\mathcal{N}$, respectively. We exploit the advances and
developments of ROM techniques to build surrogate models to economically
resolve portions of domain and/or physics. The ROM solution can thus be used
to infer the flow conditions at the interface so that a FOM solver can be
efficiently employed for the regions of interest. The standard Galerkin ansatz
is applied for the dynamics of $\mathbf{u}$ as
$\mathbf{u}(t)\approx\Phi\boldsymbol{\alpha}(t)$, where the columns of matrix
$\Phi=[\phi_{1},\phi_{2},\dots,\phi_{r}]$ form the orthonormal bases of a
reduced subspace, and $\boldsymbol{\alpha}$ defines their amplitudes. Proper
orthogonal decomposition (POD) is one popular technique to systematically
construct $\Phi$ such that the solution manifold preserves as much variance as
possible when projected onto the subspace spanned by $\Phi$ [24]. By
substituting this approximation into Eq. (3), performing the inner product
with $\Phi$, and making use of the quadratic nonlinearity in most fluid flow
systems, we get the following,

$\dfrac{\mathrm{d}\boldsymbol{\alpha}}{\mathrm{d}t}=\mathbf{\mathfrak{L}}\boldsymbol{\alpha}+\boldsymbol{\alpha}^{T}\mathbf{\mathfrak{N}}\boldsymbol{\alpha}+\mathbf{\mathfrak{C}},$
(4)

where $\mathbf{\mathfrak{L}}$ and $\mathbf{\mathfrak{N}}$ signify the model
coefficients while $\mathbf{\mathfrak{C}}$ defines the contribution of
$\mathbf{v}$ into the ROM of $\mathbf{u}$. At the interface, we introduce a
long short-term memory network to bridge the low-fidelity descriptions to
high-fidelity models in various forms of interfacial error correction or
prolongation. An array of interface modeling paradigms are sketched in Figure
4(b) and summarized as follows (see [13] for more details),

  1. 1.

_DPI: Direct Prolongation Interface._ The DPI approach provides an estimate of
the flow variables at the interface from the ROM solution. Indeed, this
prolongation map naturally results from the Galerkin ansatz, without any
interference from the ML side. However, it is known that truncated Galerkin
ROMs might yield erroneous and even unstable predictions for complex systems
[25]. Therefore, the solution from the DPI approach is potentially inaccurate,
and a correction needs to be introduced.

  2. 2.

_CPI: Correction followed by Prolongation Interface._ The CPI methodology
works by introducing the correction in the latent subspace and addresses the
deviation in modal coefficients predicted from solving the Galerkin ROM, known
as closure error. Specifically, the LSTM for CPI takes the values of modal
coefficients acquired from integrating Eq. (4) and predicts the discrepancy
between these values and their optimal values. We highlight here that the size
of the input and output vectors is $O(r)$, independent of the FOM resolution,
which offers a potential flexibility dealing with 2D and 3D problems.

  3. 3.

_UPI: Uplifted Prolongation Interface._ Although the CPI methodology cures the
closure error and provides a stabilized solution, it does not address the
projection error. Unless a large number of modes are resolved, the projection
error can be significant. An uplifting ROM has been proposed [25], where both
closure and projection errors are taken care of. In addition to the closure
modeling, the ROM subspace is expanded to recover some of the smaller scales
missing in the initial subspace as follows,

$\mathbf{u}\approx\Phi\boldsymbol{\alpha}+\Psi\boldsymbol{\beta},$ (5)

where $\Psi$ forms orthonormal basis for a $q$-dimensional subspace
complementing that spanned by $\Phi$ and with $\boldsymbol{\beta}$ being the
corresponding amplitudes. We highlight that the Galerkin ROM equations only
solve for $\boldsymbol{\alpha}$ to keep the computational cost as low as
possible. Therefore, a complementary model for $\boldsymbol{\beta}$ has to be
constructed so that the uplifting approach can be employed. A mapping from the
first $r$ modal coefficients to the next $q$ modes is assumed to exist and we
exploit the LSTM learning capabilities to infer this map from data. In
particular, the UPI architecture is trained to read the Galerkin ROM
prediction for the first $r$ modal coefficients as input, and return the true
coefficients of the first $r+q$ modes. Thus, it provides a _closure_
correction for the first $r$ modes and a _superresolution_ effect for the next
$q$ modes, simultaneously in a single network.

###  3.2 Demonstration using Boussinesq equations

The dimensionless form of the 2D incompressible Boussinesq equations can be
represented by the following two coupled transport equations in vorticity-
streamfunction formulation,

$\displaystyle\dfrac{\partial\omega}{\partial t}+J(\omega,\psi)$
$\displaystyle=\dfrac{1}{\text{Re}}\nabla^{2}\omega+\text{Ri}\dfrac{\partial\theta}{\partial
x},$ (6) $\displaystyle\dfrac{\partial\theta}{\partial t}+J(\theta,\psi)$
$\displaystyle=\dfrac{1}{\text{Re}\text{Pr}}\nabla^{2}\theta,$ (7)

where $\omega$, $\psi$ and $\theta$ denote the vorticity, streamfunction and
temperature fields, respectively. We utilize the 2D Boussinesq equation to
illustrate the ROM-FOM coupling in multi-fidelity environments. In particular,
we suppose that we are more interested in the temperature field predictions.
Thus, we dedicate a FOM solver for Eq. (7). However, the solution of this
equation requires evaluating the streamfunction field at each time step. The
kinematic relationship between vorticity and streamfunction is given by the
Poisson equation (i.e, $\nabla^{2}\psi=-\omega$), the solution of which
consumes significant amount of time and computational resources and is
considered the bottleneck for most incompressible flow solvers. Therefore, we
consider a ROM solver for the voriticity dynamics and FOM solver for the
temperature field.

For demonstration, we explore the lock-exchange problem, defined by two fluids
of different temperatures, in a rectangular domain $(x,y)\in[0,8]\times[0,1]$
with a vertical barrier dividing the domain at $x=4$, keeping the temperature
of the left half at $1.5$ and temperature of the right half at $1$. Initially,
the flow is at rest, with uniform temperatures at the right and left regions,
and the barrier is removed at $t=0$. Reynolds number of $\text{Re}=10^{4}$,
Richardson number of $\text{Ri}=4$, and Prandtl number of $\text{Pr}=1$ are
set and a Cartesian grid of $4096\times 512$ with a timestep of $\Delta
t=5\times 10^{-4}$ are used for the FOM simulations.

A two-layer LSTM with 20 hidden units in each LSTM cell constitutes our ML
architecture. We store $800$ time snapshots for POD basis construction and we
retain $r=8$ modes for the Galerkin ROM solver. We also utilize the dataset of
the stored $800$ snapshots for LSTM training and validation. During the
testing phase, the trained neural networks are deployed at each and every
timestep. This corresponds to the deployment of the presented approached
$16000$ times. Figure 5 shows the predictions of the temperature field at
final time (i.e., $t=8$) computed from DPI, CPI, and UPI approaches compared
to the FOM field. We emphasize that the ROM-FOM coupling results correspond to
the solution of the vorticity equation with a ROM solver, which feeds the FOM
solver with streamfunction to solve the temperature equation only as opposed
to the FOM results which comes from the solution of both equations using a
fully FOM simulation. Although the CPI results are better than those of DPI,
we can observe that the fine details of the flow field are not accurately
captured. On the other hand, the implementation of the UPI approach with $r=8$
and $q=8$ recovers an increased amount of the fine flow structures that are
not well-represented by the first $r=8$ modes.

Figure 5: Final temperature fields as obtained from different ROM-FOM coupling
approaches, compared to the FOM solution.

##  4 Concluding remarks

The data-driven methods are increasingly being applied in many branches of
science and engineering due to their success in automatic pattern-
identification using the data collected from sensors and numerical
simulations. Even though they offer an alternative to physics-based modeling
derived from phenomenological arguments, they are usually black box in nature
and lack generalizability. The hybrid analysis and modeling (HAM) is a newly
emerging paradigm that combines physics-based and data-driven modeling to
deliver robust and generalizable models that can enable the digital twins of
large scale physical systems.

The major contributions of this work towards HAM paradigm are

  * •

A novel deep neural network architecture that makes it possible to inject
physics during the training process. This resulted in a significant reduction
of uncertainty.

  * •

An interface learning technique that makes seamless coupling of multi fidelity
models possible.

The physics-guided machine learning (PGML) framework enhances the
generalizability of the neural network based surrogate model and its
performance is illustrated for real time aerodynamic performance prediction
task. The interface learning (IL) framework allows integration of two
different models seamlessly to build multi-fidelity models that are orders of
magnitude faster than full order physics-based models. Although PGML approach
was demonstrated to work with deep neural network, it can easily be extended
to other machine learning algorithms. Likewise the interface learning
technique that was demonstrated to couple multi-fidelity models can be used
for coupling multiscale and multiphysics models.

## Acknowledgement

The authors acknowledge the financial support from the Department of
Engineering Norwegian Research Council, the industrial partners of OPWIND:
Operational Control for Wind Power Plants (Grant No.: 268044/E20), the U.S.
DOE Early Career Research Program (Award Number DE-SC0019290), and the
National Science Foundation under Award Number DMS-2012255.

## References

## References

  * [1] Paul Veers, Katherine Dykes, Eric Lantz, Stephan Barth, Carlo L Bottasso, Ola Carlson, Andrew Clifton, Johney Green, Peter Green, Hannele Holttinen, et al.  Grand challenges in the science of wind energy.  Science, 366(6464):eaau2027, 2019.
  * [2] Adil Rasheed, Omer San, and Trond Kvamsdal.  Digital twin: Values, challenges and enablers from a modeling perspective.  IEEE Access, 8:21980–22012, 2020.
  * [3] Digital Wind Farm.  https://www.ge.com/renewableenergy/wind-energy/onshore-wind/digital-wind-farm.  Accessed: 2021-01-01.
  * [4] Garrett E Barter, Amy Robertson, and Walter Musial.  A systems engineering vision for floating offshore wind cost optimization.  Renewable Energy Focus, 34:1–16, 2020.
  * [5] Majid Bastankhah and Fernando Porté-Agel.  A new analytical model for wind-turbine wakes.  Renewable Energy, 70:116–123, 2014.
  * [6] Carl R Shapiro, Dennice F Gayme, and Charles Meneveau.  Modelling yawed wind turbine wakes: a lifting line approach.  Journal of Fluid Mechanics, 841, 2018.
  * [7] Charles Meneveau.  Big wind power: Seven questions for turbulence research.  Journal of Turbulence, 20(1):2–20, 2019.
  * [8] Evangelos S Politis, John Prospathopoulos, D Cabezon, Kurt S Hansen, PK Chaviaropoulos, and Rebecca J Barthelmie.  Modeling wake effects in large wind farms in complex terrain: the problem, the methods and the issues.  Wind Energy, 15(1):161–182, 2012.
  * [9] Wolfgang Schlez and Anja Neubert.  New developments in large wind farm modelling.  In European Wind Energy Conference, page 8, 2009.
  * [10] Suraj Pawar, Omer San, Burak Aksoylu, Adil Rasheed, and Trond Kvamsdal.  Physics guided machine learning using simplified theories.  arXiv preprint arXiv:2012.13343, 2020.
  * [11] Suraj Pawar, Shady E Ahmed, and Omer San.  Interface learning in fluid dynamics: Statistical inference of closures within micro–macro-coupling models.  Physics of Fluids, 32(9):091704, 2020.
  * [12] Shady E Ahmed, Omer San, Kursat Kara, Rami Younis, and Adil Rasheed.  Interface learning of multiphysics and multiscale systems.  Physical Review E, 102(5):053304, 2020.
  * [13] Shady E Ahmed, Omer San, Kursat Kara, Rami Younis, and Adil Rasheed.  Interface learning in zonal multifidelity simulations of fluid flows with a hybrid analysis and modeling framework.  arXiv preprint arXiv:2007.06179, 2020.
  * [14] Ge-Cheng Zha, Bruce F Carroll, Craig D Paxton, Clark A Conley, and Adam Wells.  High-performance airfoil using coflow jet flow control.  AIAA Journal, 45(8):2087–2090, 2007.
  * [15] Patrick LeGresley and Juan Alonso.  Airfoil design optimization using reduced order models based on proper orthogonal decomposition.  In Fluids 2000 conference and exhibit, page 2545, 2000.
  * [16] Yao Zhang, Woong Je Sung, and Dimitri N Mavris.  Application of convolutional neural network to predict airfoil lift coefficient.  In 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, page 1903, 2018.
  * [17] Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik.  Prediction of aerodynamic flow fields using convolutional neural networks.  Computational Mechanics, 64(2):525–545, 2019.
  * [18] Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris.  Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data.  Journal of Computational Physics, 394:56–81, 2019.
  * [19] Romit Maulik, Kai Fukami, Nesar Ramachandra, Koji Fukagata, and Kunihiko Taira.  Probabilistic neural networks for fluid flow surrogate modeling and data recovery.  Physical Review Fluids, 5(10):104401, 2020.
  * [20] Mark Drela.  XFOIL: An analysis and design system for low Reynolds number airfoils.  In Low Reynolds number aerodynamics, pages 1–12. Springer, 1989\.
  * [21] John L Hess.  Panel methods in computational fluid dynamics.  Annual Review of Fluid Mechanics, 22(1):255–274, 1990.
  * [22] Robert Tibshirani.  A comparison of some error estimates for neural network models.  Neural Computation, 8(1):152–163, 1996.
  * [23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In Advances in neural information processing systems, pages 6402–6413, 2017.
  * [24] Philip Holmes, John L Lumley, Gahl Berkooz, and Clarence W Rowley.  Turbulence, coherent structures, dynamical systems and symmetry.  Cambridge University Press, New York, 2012.
  * [25] Shady E Ahmed, Omer San, Adil Rasheed, and Traian Iliescu.  A long short-term memory embedding for hybrid uplifted reduced order models.  Physica D: Nonlinear Phenomena, page 132471, 2020.


scaletikzpicturetowidth[1]@tikzpicture

# Self-supervised 6D Object Pose Estimation for Robot Manipulation

Xinke Deng1,2, Yu Xiang1, Arsalan Mousavian1, Clemens Eppner1, Timothy Bretl2,
Dieter Fox1,3 * This work is done while the first author was an intern at
NVIDIA. 1NVIDIA, 2University of Illinois at Urbana-Champaign, 3University of
Washington

###### Abstract

To teach robots skills, it is crucial to obtain data with supervision. Since
annotating real world data is time-consuming and expensive, enabling robots to
learn in a self-supervised way is important. In this work, we introduce a
robot system for self-supervised 6D object pose estimation. Starting from
modules trained in simulation, our system is able to label real world images
with accurate 6D object poses for self-supervised learning. In addition, the
robot interacts with objects in the environment to change the object
configuration by grasping or pushing objects. In this way, our system is able
to continuously collect data and improve its pose estimation modules. We show
that the self-supervised learning improves object segmentation and 6D pose
estimation performance, and consequently enables the system to grasp objects
more reliably. A video showing the experiments can be found at
https://youtu.be/W1Y0Mmh1Gd8.

##  I INTRODUCTION

For robots to gain skills such as manipulation, learning-based methods have
received more attention recently due to their capability in handling the
complexity in robot perception. For instance, a number of 6D object pose
estimation methods using deep neural networks are introduced for model-based
grasping [1, 2]. Learning techniques also demonstrate impressive performance
in grasping unknown objects [3, 4, 5]. It is well known that training deep
neural networks requires a significant amount of data. How to obtain training
data becomes the bottleneck in applying learning techniques to different
robotic problems these days.

It is very appealing to conduct training in simulation, since synthetic data
generated from simulators are free, come with ground truth annotations, and
are easy to scale up. However, due to the domain gap, models trained
exclusively with synthetic data cannot be guaranteed to work well in the real
world. In contrast, training can be performed using data directly collected in
the real world. However, annotating real world data is time consuming and
labor intensive.

Recently, the concept of self-supervised learning is becoming attractive,
where robots autonomously annotate real world data for training. Self-
supervised learning has great potential to improve the robustness of robotic
systems and achieve life-long learning for robots. The main challenge in self-
supervision is how to annotate the real world data automatically, and
meanwhile, guarantee the accuracy of the annotations. Previous works have
explored self-supervised learning for model-free grasping [4], pixel-wise
object segmentation [6], and object detection with bounding boxes [7].
However, to the best of our knowledge, no work has been proposed for directly
annotating 6D object pose in a self-supervised fashion. This is mainly because
accurately estimating 6D object pose and automatically evaluating the
estimation is challenging.

Figure 1: Our self-supervised learning process for robot manipulation.

In this work, by leveraging the recent advances in 6D object pose estimation
with deep neural networks [1, 8, 9], we present a novel self-supervised 6D
object pose estimation system for manipulation (Fig. 1). An active vision
system is set up by mounting an RGB-D camera onto the hand of a robot
manipulator. Given a set of objects in the workspace, we control the robot arm
to collect images from different views. Meanwhile, the 6D poses of these
objects are accurately initialized and then tracked based on the PoseRBPF
framework [9], where we use the forward kinematics of the robot arm as a
motion prior in the particle filtering. Consequently, our system is able to
collect images of multiple objects from different views with accurate 6D pose
annotations. We can use this data to perform self-supervised training of the
deep neural networks in the system for object segmentation and pose
estimation. After each iteration of the multi-view pose tracking, the robot
manipulator grasps or pushes one object in the scene to generate a new scene,
and then repeats the pose tracking process. In this way, our system is able to
continuously generate new data for self-supervised learning, and improve its
pose estimation and grasping performance in a life-long learning fashion.

Figure 2: Overview of our self-supervised 6D object pose estimation system.

We conduct experiments to demonstrate how the self-supervised learning
improves the system. We first show that by using the self-annotated images to
fine-tune the semantic segmentation network, we can significantly improve the
segmentation accuracy compared to the original network trained with synthetic
data only. Semantic segmentation suffers from the domain gap severely for
objects with complex material properties, such as shininess. Fine-tuning the
network can bridge the domain gap. Second, we show that fine-tuning the auto-
encoders used in PoseRBPF [9] can improve their reconstruction quality, and
consequently improve the pose estimation accuracy. Finally, we conduct robot
grasping experiments to illustrate that our fine-tuned system can achieve a
high success rate in model-based grasping based on 6D object pose estimation.

This paper is organized as follows: After discussing related work, we
introduce our system for self-supervised 6D object pose estimation, followed
by experiments and a conclusion.

##  II RELATED WORK

6D Object Pose Estimation. Given the 3D models of objects, the goal of 6D pose
estimation is to estimate the 3D translation and the 3D rotation of these
objects from input images. Traditionally, the problem is tackled by local
feature matching [10, 11, 12] or template matching [13, 14, 15]. Recently,
learning-based methods receive more attention due to their ability to
generalize to different backgrounds, lighting, and occlusions [16, 17, 18].
Deep neural networks further boost the performance by learning more
discriminative features and classifiers [1, 19, 8, 9, 2]. However, deep
learning methods also require more data for training.

A few recent methods show that it is possible to train exclusively with
synthetic data and apply the trained network to the real world [20, 8]. But
for objects with mismatch between simulation and real world, these methods
tend to fail, such as mismatch in texture materials or lighting. Meanwhile,
some efforts are devoted to collecting large-scale real-world dataset for 6D
pose estimation [1, 21]. These methods use a moving camera around a static
scene so that the pose labels for a video sequence can be generated with
labeling one frame and tracking the camera ego-motion [1] or labeling the 3D
reconstructed scene [21]. Although tracking camera motion simplifies the
labeling process, these methods require human to arrange the scene, collect
video sequences of objects, and initialize the systems, which is difficult to
scale to a large number of scenes. In this work, we employ robots to
automatically arrange scenes and generate accurate 6D pose annotations without
pose initialization from human.

Self-supervised Learning. Due to its ability to learn from unlabeled data,
self-supervised learning has been studied in different sub-fields in AI, such
as in robotics [4, 22], computer vision [23, 24], machine learning [25] and
natural language processing [26]. The majority of these methods focus on
learning from fixed datasets. In robotics, since the robot can interact with
the real world, we can plan and control how data is collected [27, 28]. This
degree of freedom in data collection significantly boosts the self-supervised
learning efficiency.

Previous works on self-supervised learning in robot perception mainly focus on
learning object segmentation [6], object detection [7], and dense pixel-wise
correspondences [29, 30], since it is relatively easy to obtain annotations in
2D image space. In this work, we explore self-supervised learning for 6D
object pose estimation. Our system benefits from the recent advances in 6D
object pose estimation using deep neural networks [1, 8, 9]. By incorporating
these techniques into the interactive robot perception framework [31], we can
obtain accurate 6D poses of objects to improve the system.

##  III SELF-SUPERVISED 6D OBJECT POSE ESTIMATION SYSTEM

Figure 3: Pose initialization process. This module takes an RGB-D image as
input and estimates the 6D poses of all the objects in the image. The process
iterates until all object poses are accurate according to the pose evaluation
component.

###  III-A System Overview

In our system, a RGB-D camera is mounted to the hand of a robot manipulator,
where images from different views can be collected by moving the robot arm
(Fig. 1). The system consists of a pose initialization module, a pose tracking
module, a robot interaction module and a self-supervised training module (Fig.
2). 1) Given a set of objects in the robot’s workspace, the pose
initialization module estimates the 6D poses of all objects from a single
RGB-D image. To ensure accurate pose initialization, we evaluate the estimated
poses and re-initialize the unsatisfied objects until all the poses are
accurate. 2) Then the pose tracking module tracks the 6D poses of these
objects when the robot arm moves to different predefined way points. This step
also collects images with 6D object poses from different views, and saves the
data in an online fashion. Both the pose initialization and pose tracking are
based on the PoseRBPF framework using particle filtering [9]. 3) After the
data is saved, the robot interaction module controls the manipulator to either
grasp or push one object in the scene, thereby generating a new scene. In this
way, we can continuously collect data with different arrangements of objects.
4) The self-supervised training module fine-tunes the deep neural networks in
the system with the saved data to boost its performance.

###  III-B The Pose Initialization Module

Given a set of objects in a scene, the task of the pose initialization module
is to accurately estimate the 6D poses of all objects from a single RGB-D
image. Based on this estimate we can start pose tracking (Fig. 3). Pose
estimation from a single image is a challenging problem. With a new scene, the
camera is moved automatically to a viewpoint with high elevation angle, so the
pose estimation is less affected by the occlusions between objects.

####  III-B1 Object Segmentation and Detection

The first step is to detect the objects in the input image. In general, any
state-of-the-art object detector can be used, such as YOLO [32], SSD [33] or
Faster R-CNN [34]. Instead of using these top-down methods that are built on
bounding boxes, we resort to PoseCNN [1]. PoseCNN first labels every pixel
into an object class and then applies Hough voting to find the centers and
bounding boxes of the objects. PoseCNN generates segmentation masks of the
objects which can be used to segment the point clouds of the objects in pose
refinement [35]. In addition, the object centers from PoseCNN are more
accurate than bounding box centers from the top-down methods thanks to the
Hough voting strategy.

####  III-B2 3D Rotation and 3D Translation Estimation

To estimate the 3D rotation $\mathbf{R}$ and 3D translation $\mathbf{t}$ of
objects in the scene, we utilize the PoseRBPF framework [9]. PoseRBPF is a
Rao-Blackwellized particle filter combined with a learned auto-encoder network
[8] for 6D object pose estimation. In PoseRBPF, each particle consists of a 3D
translation and a full distribution over $\mathbf{SO}(3)$. A 3D translation
can be represented by $(u,v,z)$, where $(u,v)$ is the object center in the
image and $z$ is the distance of object center from the camera in 3D.

For each detected object, the 3D translations of the particles in PoseRBPF are
sampled according to the detected object center $(\hat{u},\hat{v})$ and the
observed depth image $\mathbf{D}$:
$(u_{i},v_{i})\sim\mathcal{N}\big{(}(\hat{u},\hat{v}),\sigma_{u},\sigma_{v}\big{)}$,
$z_{i}\sim\mathcal{U}\big{(}\mathbf{D}(u_{i},v_{i})-\frac{d}{2},\mathbf{D}(u_{i},v_{i})+\frac{d}{2})$,
where $(u_{i},v_{i},z_{i})$ denotes the translation of the $i$th particle,
$\sigma_{u},\sigma_{v}$ denote the standard deviations of the Gaussian
distributions for object center, and $d$ denotes the range of a uniform
distribution where $z_{i}$ is sampled from. $\mathbf{D}(u_{i},v_{i})$ is the
depth at pixel location $(u_{i},v_{i})$. Each particle determines a Region of
Interest (RoI) of the target object (Fig. 3). The RoI is fed into the auto-
encoder to compute a code which is used to measure the likelihood of the
particle in PoseRBPF. After the initial sampling of the particles, the system
performs $K$ filtering steps with the same RGB-D image to ensure the
convergence of particles, from which we can extract the expectation of 3D
rotation and 3D translation.

####  III-B3 6D Pose Refinement

Since the number of particles in PoseRBPF is limited due to time budget and
discretization of 3D rotation in each particle, we perform a continuous
optimization using 3D points from the depth image to refine the poses. Finding
correct pixels corresponding to the object is crucial in the refinement
process. We estimate the segmentation mask of the object $\mathbf{\Omega}$ by
considering both the predicted segmentation mask $\mathbf{\hat{\Omega}}$ from
PoseCNN and the predicted 6D pose $(\bar{\mathbf{t}},\bar{\mathbf{R}})$ from
PoseRBPF. We first render the object according to the estimated pose. Denoting
the rendered depth image and segmentation mask of the object as
$\bar{\mathbf{D}}$ and $\bar{\mathbf{\Omega}}$. The final segmentation mask
$\mathbf{\Omega}_{\text{obj}}$ is estimated with
$\mathbf{\Omega}_{\text{obj}}=\hat{\mathbf{\Omega}}\cap\bar{\mathbf{\Omega}}\cap\mathbf{\Omega}_{\mathbf{D}}$.
Here, $\mathbf{\Omega}_{\mathbf{D}}$ represents the pixels where the rendered
depth matches with the measured depth within a margin $m$:
$\mathbf{\Omega}_{\mathbf{D}}=\left\\{\forall(u,v),\left|\mathbf{D}(u,v)-\mathbf{\bar{D}}(u,v)\right|<m\right\\}$.
Then the point cloud of the object $\mathbf{P}_{\text{obj}}$ can be computed
by back-projecting the pixels in $\mathbf{\Omega}_{\mathbf{D}}$:

$\mathbf{P}_{\text{obj}}=\left\\{\mathbf{D}(u,v)\mathbf{K}^{-1}(u,v,1)^{T},(u,v)\in\mathbf{\Omega}_{obj}\right\\},$
(1)

where $\mathbf{K}$ represents the intrinsic matrix of the camera.

After computing the 3D points on the object, we optimize the pose by matching
these points against the Signed Distance Function (SDF) of the object model as
in [36]. The optimization problem we solve is

$(\hat{\mathbf{t}},\hat{\mathbf{R}})=\underset{\mathbf{t},\mathbf{R}}{\arg\min}\sum_{\mathbf{p}_{i}\in\mathbf{P}_{\text{obj}}}\left|\mathbf{SDF}_{\text{obj}}(\mathbf{p}_{i},\mathbf{t},\mathbf{R})\right|+\lambda\frac{1}{2}\|\mathbf{t}-\bar{\mathbf{t}}\|^{2},$
(2)

where $\mathbf{p}_{i}$ is a 3D point in the point cloud
$\mathbf{P}_{\text{obj}}$,
$\mathbf{SDF}_{\text{obj}}(\mathbf{p}_{i},\mathbf{t},\mathbf{R})$ denotes the
signed distance value by transforming the point $\mathbf{p}_{i}$ from the
camera coordinate into the object model coordinate using pose
$(\mathbf{t},\mathbf{R})$, and $\lambda$ is a weight to balance the
regularization term on the translation. Fig. 3 illustrates the matching
between the point cloud against the SDF before and after the optimization.

####  III-B4 Pose Evaluation

Due to noise in both RGB and depth measurements, failure in pose estimation is
inevitable. To detect failures and label object poses accurately, we propose a
pose evaluation process (Fig. 4) so that objects with erroneous pose
estimation will be reinitialized. We first render an object according to its
estimated pose to generate a synthetic RGB-D image. Then we compute two
metrics by comparing the synthetic image with the real image. For RGB, we
compute the code of the color rendering $\hat{\mathbf{c}}$ and the code of the
color measurement ${\mathbf{c}}$ by passing the two corresponding RoIs through
the auto-encoder in PoseRBPF. The cosine distance
$s=(\hat{\mathbf{c}}\cdot\mathbf{c})/(\left\|\hat{\mathbf{c}}\right\|\cdot\left\|\mathbf{c}\right\|)$
between the two codes is then computed to indicate the pose confidence in the
color space. For depth, we compare the rendered depth image $\hat{\mathbf{D}}$
and the measured depth image ${\mathbf{D}}$ using the mask
$\mathbf{\Omega}_{\text{obj}}$ from Eq. (1). The mean depth error is computed
to measure the pose accuracy in the depth space:
$e=\sum_{(u,v)\in\mathbf{\Omega}_{\text{obj}}}|\hat{\mathbf{D}}(u,v)-\mathbf{D}(u,v)|$.
Finally, we declare pose estimation failure if $s<s^{*}$ or $e>e^{*}$, where
$s^{*}$ and $e^{*}$ are predefined thresholds. In this case the system re-
initializes the pose of the failed object.

Figure 4: The pose evaluation component compares the rendered RGB-D image with
the real RGB-D image.

###  III-C The Pose Tracking Module

Once object poses in the scene are initialized, the robot moves the camera to
different views around the objects for data collection. Meanwhile, a
continuous sequence of video frames is collected in real time, and the poses
of the objects in the sequence are tracked with the PoseRBPF framework. The
pose tracking module generates accurate pose annotations online for self-
supervised training and enables the robot to interact with the objects
continuously.

In order to track all objects accurately, we first propagate all particles
according to the robot’s forward kinematics. Denoting the camera pose
transformation from the forward kinematics as
$(\Delta\mathbf{t},\Delta\mathbf{R})\in\mathbf{SE}(3)$, the particles are
propagated according to the following motion prior:

$\displaystyle
P(\mathbf{t}_{k}|\mathbf{t}_{k-1},\Delta\mathbf{t},\Delta\mathbf{R})$
$\displaystyle=\mathcal{N}(\Delta\mathbf{R}\mathbf{t}_{k-1}+\Delta\mathbf{t},\Sigma_{\mathbf{t}}),$
(3) $\displaystyle
P(\mathbf{R}_{k}|\mathbf{R}_{k-1},\Delta\mathbf{t},\Delta\mathbf{R})$
$\displaystyle=\mathcal{N}(\Delta\mathbf{R}\mathbf{R}_{k-1},\Sigma_{\mathbf{R}}),$
(4)

where $\Sigma_{\mathbf{t}}$ and $\Sigma_{\mathbf{R}}$ denote the covariance
matrix of the Gaussian distributions of the translation and the rotation,
respectively. Since the rotation in PoseRBPF is represented as a _full
discrete distribution_ over $\mathbf{SO}(3)$, the distributions need to be
shifted entirely. We first represent $\Delta\mathbf{R}$ as
$(\Delta\phi,\Delta\theta,\Delta\psi)$ according to the discretization of the
rotation distribution, where $\Delta\phi$, $\Delta\theta$, $\Delta\psi$
represent the changes in pitch, yaw and roll, respectively. Then the rotation
distribution is adjusted by performing bilinear interpolation of the shifted
grids $\left\\{\phi+\Delta\phi,\theta+\Delta\theta,\psi+\Delta\psi\right\\}$.

Due to the noises from the joint encoders of the robot or the bias in the
hand-eye calibration, simply applying forward kinematics might often lead to
inaccurate object poses. Similar to the pose initialization process, we
perform $1\text{\,}\mathrm{s}$tep filtering with the propagated particles
using the current RGB-D image for all the objects. To ensure the quality of
the generated data for self-supervised learning, we only save data when the
two pose evaluation metrics are above certain thresholds (Sec. III-B4). Once
the system decides to save the data, the object poses are further refined with
the SDF-based pose refinement algorithm as described in Sec. III-B3.

Figure 5: Examples of 100 precomputed grasps for 10 YCB objects.

###  III-D The Robot Interaction Module

After a scene is captured, the robot physically interacts with objects in the
scene to change their configuration. This way the system can continuously
capture new data with different poses of the objects. Two types of
interactions are used. The robot either pushes an object in a random direction
or grasps an object and places it at a different location (Fig. 1).

Pushing is defined by choosing a direction and random radius with respect to
the object center and executing the motion. To ensure that objects stay in the
workspace of the robot, objects are pushed either toward the center of mass of
other objects in the clutter or the center of the robot workspace. For
grasping, a set of diverse grasps for each object is computed offline, where
we use 100 parallel-jaw grasps sampled from the robust grasps of [37] (see
Fig. 5 for examples). During online execution, using the estimated pose of the
object, the pre-computed grasps are transformed from object to camera
coordinates. Then the robot selects any grasp that is kinematically feasible
and collision-free with other objects using the RRT∗ planning algorithm in
MoveIt! [38]. After grasping the object, the robot first rotates its gripper
to change the orientation of the object. Then a random placement location is
chosen within the robot workspace and the robot places the object.

###  III-E The Self-Supervised Training Module

With the collected images and estimated object poses, we can fine-tune the
neural networks in our system to improve the pose estimation performance.
Initially, both the network for segmentation and detection in PoseCNN and the
auto-encoders used in PoseRBPF (see Fig. 3) are trained only with synthetic
data. As a result, the system cannot even segment some textureless objects in
our experiments such as a red foam brick. Therefore, we employ the curriculum
learning idea [39] to bootstrap the system sequentially. We start with simple
scenes that contain only one object. If the network cannot segment the object,
we perform a global initialization in PoseRBPF, where particles are uniformly
sampled in the image. This process collects training data with single objects
in images. We can then fine-tune the segmentation network to make it segment
all objects. Fine-tuning with single objects also enables the network to
segment objects in cluttered scenes. In the next stage, we move to cluttered
scenes with multiple objects, and collect training data with different object
poses and occlusions to fine-tune the segmentation network and the auto-
encoders. During fine-tuning, we mix synthetic and real data to prevent the
networks from overfitting. We can iterate between data collection and fine-
tuning. Because when the networks become better, the system is more efficient
in collecting new data.

##  IV EXPERIMENTS

TABLE I: F1 score of semantic segmentation for 20 YCB objects on the test set. The best F1 score for each object are bold, and numbers below 60% are highlighted red. Model | Synth. |  | +20%
---
Real
| +40%
---
Real
| +60%
---
Real
| +80%
---
Real
| +100%
---
Real
master_chef_can | 69.3 | 88.7 | 92.8 | 91.9 | 91.6 | 93.9
cracker_box | 84.7 | 92.8 | 93.0 | 93.4 | 93.0 | 93.4
sugar_box | 83.0 | 92.0 | 92.0 | 92.4 | 92.5 | 92.6
tomato_soup_can | 83.6 | 90.2 | 90.5 | 90.8 | 91.2 | 91.4
mustard_bottle | 83.9 | 92.5 | 93.3 | 93.3 | 93.9 | 94.2
tuna_fish_can | 42.3 | 90.1 | 90.2 | 91.7 | 91.7 | 91.6
pudding_box | 61.6 | 85.7 | 84.6 | 85.9 | 87.1 | 87.0
gelatin_box | 66.6 | 83.6 | 83.2 | 83.7 | 82.0 | 84.6
potted_meat_can | 62.9 | 84.1 | 85.4 | 86.4 | 86.6 | 88.6
banana | 79.8 | 87.3 | 88.2 | 89.0 | 89.0 | 89.3
pitcher_base | 51.5 | 86.3 | 84.3 | 88.0 | 89.7 | 89.6
bleach_cleanser | 57.9 | 89.3 | 92.1 | 93.3 | 90.2 | 93.4
bowl | 69.8 | 90.4 | 92.5 | 93.2 | 94.5 | 95.4
mug | 69.2 | 90.3 | 90.9 | 91.4 | 92.0 | 91.0
power_drill | 66.1 | 84.4 | 87.3 | 88.0 | 87.5 | 88.5
wood_block | 64.2 | 82.6 | 80.2 | 85.1 | 86.0 | 86.1
scissors | 36.3 | 71.9 | 75.8 | 77.4 | 77.5 | 78.9
large_marker | 55.5 | 73.8 | 75.6 | 76.2 | 75.4 | 77.1
extra_large_clamp | 15.5 | 76.3 | 76.0 | 79.1 | 77.1 | 79.1
foam_brick | 12.2 | 86.5 | 86.7 | 87.6 | 86.6 | 88.9
MEAN | 60.8 | 85.9 | 86.7 | 87.9 | 87.8 | 88.7

###  IV-A Data Collection

We conduct experiments with 20 objects in the YCB Object and Model Set [40].
During around 12 robot hours, our system collected 497 scenes, 6,541 RGB-D
images and 22,851 object instances in these images with accurate 6D poses. In
this process, human intervention is required in two cases: 1) whenever a new
subset of the 20 YCB objects needs to be presented to the robot, and 2) to
rearrange objects if the system cannot initialize the object poses for a
while. Compared to human collected videos for pose annotation, our system
saves a significant amount of human labor, and has the ability to scale up to
a large number of scenes. For comparison, the YCB-Video dataset [1] and the
LabelFusion dataset [21] have 92 scenes and 138 scenes, respectively. We split
the collected data into a training set (265 scenes, 3,590 images) and a test
set (232 scenes, 2,951 images).

Figure 6: Evaluation of 6D object pose estimation on the test set.

###  IV-B Evaluation Metrics

We evaluate how the system improves by fine-tuning the neural networks with
the collected data. For semantic segmentation, we compute the F1 score by
comparing the predicted labels with the ground truth labels pixel-wisely. For
evaluating 6D poses, we use ADD and ADD-S metric as defined in [14, 1]:
$\textsc{ADD}=\frac{1}{m}\sum_{\mathbf{x}\in\mathcal{M}}\|(\mathbf{R}\mathbf{x}+\mathbf{t})-(\mathbf{\tilde{R}}\mathbf{x}+\mathbf{\tilde{t}})\|$,
$\textsc{ADD-S}=\frac{1}{m}\sum_{\mathbf{x}_{1}\in\mathcal{M}}\min_{\mathbf{x}_{2}\in\mathcal{M}}\|(\mathbf{R}\mathbf{x}_{1}+\mathbf{t})-(\mathbf{\tilde{R}}\mathbf{x}_{2}+\mathbf{\tilde{t}})\|$,
where $\mathcal{M}$ denotes the set of 3D model points and $m$ is the number
of points. $(\mathbf{R},\mathbf{t})$ and
$(\mathbf{\tilde{R}},\mathbf{\tilde{t}})$ are the ground truth pose and
estimated pose, respectively. To evaluate the grasping performance, we compute
the success rate, and the speed for initializing pose estimations as well as
grasping.

###  IV-C Implementation Details

We implement our system on a 7-DoF Franka Panda manipulator with an Intel
RealSense D415 camera mounted on its parallel-jaw gripper. The deep neural
networks run on two NVIDIA Titan Xp GPUs. In the pose initialization module,
the noise for sampling particles $\sigma_{u}$, $\sigma_{v}$, and $\sigma_{z}$
are set to
$20\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$,
$20\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$, and
$0.1\text{\,}\mathrm{m}$ respectively. In pose refinement, the margin for
computing the visibility mask is set to $0.02\text{\,}\mathrm{m}$, and the
weight $\lambda$ is set to $0.001\text{\,}$ In pose evaluation, thresholds for
deciding estimation failures are $0.5\text{\,}\mathrm{a}$nd
$0.03\text{\,}\mathrm{m}$ for $s^{*}$ and $e^{*}$. In pose tracking module,
process noise $\Sigma_{\mathbf{t}}$ and $\Sigma_{\mathbf{R}}$ are set to
$0.015\text{\,}\mathrm{m}$ and $0.05\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$.

###  IV-D Semantic Segmentation

By fine-tuning the segmentation network with self-collected images, we can
bridge the domain gap and significantly improve the segmentation accuracy.
Table I shows the F1 scores of 20 YCB objects on the test set. The performance
of the networks fine-tuned with different percentages of real training data is
presented. By using $20\text{\,}\mathrm{\char 37\relax}$ training data, the
overall F1 score is increased by $25\text{\,}\mathrm{\char 37\relax}$. Fine-
tuning with more data consistently improves the performance. In Table I, we
highlight objects on which synthetic training performs poorly. Most of them
are textureless (pitcher_base, wood_block, extra_large_clamp and foam_brick).
Objects with metal parts generate reflections which are not well modeled in
simulation (tuna_fish_can, potted_meat_can and scissors). For pudding_box and
power_drill, the textures of the real objects are different from the 3D models
due to product updates. large_marker is difficult since it is small in images.
All these issues make synthetic training difficult to transfer to the real
world, and our self-supervised learning system successfully adapts to the new
environment.

###  IV-E 6D Object Pose Estimation

In this experiment, we evaluate 6D object pose estimation using single RGB-D
images as input. If the system can estimate object poses quickly and
accurately from just a single image, it would significantly speed up
manipulation tasks. We evaluate our pose initialization method (Fig. 3) and
the original PoseCNN [1]. Both methods use the same network for segmentation
and detection. The main difference is in rotation and translation estimation.
PoseCNN [1] is trained to directly regress to object distance and 3D rotation,
while our method uses the auto-encoders and particle filter to estimate
translation and rotation.

Fig. 6 presents the 6D pose estimation accuracy in terms of ADD and ADD-S
using different percentages of real images for fine-tuning or number of
particles. We can see that 1) both methods improve consistently with more real
training data, and the gap between synthetic models and fine-tuned ones is
obvious. 2) Using depth is important to achieve high accuracy, and our SDF-
based pose refinement algorithm significantly boosts the accuracy. 3) The
performance of our method improves consistently with more particles in
PoseRBPF. However the improvement becomes less obvious when performing
refinement. 4) It is interesting that our RGB based particle filter with SDF
refinement achieves the best performance, which is even better than the RGB-D
particle filter with refinement. This is because the auto-encoders are trained
only with RGB images. Using RGB images in the particle filter obtains more
accurate rotation, and then SDF refinement can fix errors in translation, and
adjust rotation locally. Fig 7 shows an example on how the fine-tuned
segmentation network and autoencoders improve segmentation and pose
estimation.

Figure 7: An example where synthetic networks fail in segmentation and pose
estimation, but the fine-tuned networks succeed. Figure 8: Grasping performed
in real cluttered environments. Here we show 5 successful and 1 failed trials
on grasping YCB objects.

###  IV-F Real Robot Grasping Experiments

We evaluate the performance of our system on a pick-and-place task. The goal
is to pick up all objects in the scene and place them into a bin. The distance
between camera and clutter is around $0.5\text{\,}\mathrm{m}$ and the initial
elevation angle of the camera is $55^{\circ}$. The system first initializes
all objects in the scene. Every time before grasping the target object, all
objects are evaluated and those with a detected failure are re-initialized
based on the target object. The estimated poses are used for grasping the
objects. After picking up an object, the robot follows a predefined trajectory
to place it.

We created six different cluttered environments. Every environment consists of
five different YCB objects. In total, 14 graspable YCB objects are selected,
and 30 grasping trials are executed. Grasping examples are shown in Fig. 8. We
evaluate three metrics: 1) grasp success rate, 2) average time elapsed for
accurately initializing each object, and 3) average time elapsed for
initializing and grasping each object. The results are shown in Table II. It
is possible that the pose of the objects cannot be successfully initialized.
We allow $45\text{\,}\mathrm{s}$ for initializing each object before calling
timeout and labelling the trial as a failure. As we see, by fine-tuning with
additional self-annotated data, the system’s performance on grasping can be
significantly improved: the success rate is improved by
$85.7\text{\,}\mathrm{\char 37\relax}$, the duration for initialization is
reduced by $55.3\text{\,}\mathrm{\char 37\relax}$, the overall duration for
grasping an object is reduced by $27.3\text{\,}\mathrm{\char 37\relax}$. There
were four cases when grasps failed. Three of them were due to errors in pose
estimations; in the other one MoveIt! failed to generate a feasible
trajectory.

TABLE II: Statistics for 30 grasping trials in clutter. Grasp Trials |  | Success
---
rate [%]
| Avg. initialization
---
time per object [s]
| Avg. grasp
---
duration [s]
synthetic data only | 46.70 | 10.04 (std: 15.98) | 21.27
| synthetic + real data
---
86.70 |  4.48 (std: 5.71) | 15.47

##  V CONCLUSION

We introduce a novel self-supervised 6D object pose estimation system for
robot manipulation. Our system is able to automatically collect real world
images with accurate 6D object poses to fine-tune its neural networks for pose
estimation. In the learning stage, the robot interacts with objects in a scene
by grasping or pushing them to create new scenes, so new data can be collected
continuously. We show that, starting from neural networks trained with
synthetic images only, our system consistently improves the pose estimation
accuracy through self-supervised learning. The grasping success rate and
grasping time are also significantly improved. For future work, we plan to
investigate how to improve grasp selection and grasp execution with self-
supervised learning in our system.

## References

  * [1] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes,” in _Robotics: Science and Systems (RSS)_ , 2018.
  * [2] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6D object pose estimation by iterative dense fusion,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3343–3352.
  * [3] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” in _Robotics: Science and Systems (RSS)_ , 2017.
  * [4] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _The International Journal of Robotics Research_ , vol. 37, no. 4-5, pp. 421–436, 2018.
  * [5] A. Mousavian, C. Eppner, and D. Fox, “6-DOF GraspNet: Variational grasp generation for object manipulation,” in _International Conference on Computer Vision (ICCV)_ , 2019.
  * [6] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao, “Multi-view self-supervised deep learning for 6D pose estimation in the amazon picking challenge,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1386–1383.
  * [7] C. Mitash, K. E. Bekris, and A. Boularias, “A self-supervised learning system for object detection using physics simulation and multi-view pose estimation,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 545–551.
  * [8] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3D orientation learning for 6d object detection from RGB images,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 699–715.
  * [9] X. Deng, A. Mousavian, Y. Xiang, F. Xia, T. Bretl, and D. Fox, “PoseRBPF: A rao-blackwellized particle filter for 6D object pose tracking,” in _Robotics: Science and Systems (RSS)_ , 2019.
  * [10] D. Lowe, “Object recognition from local scale-invariant features,” in _International Conference on Computer Vision (ICCV)_ , 1999.
  * [11] F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce, “3D object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints,” _International Journal of Computer Vision (IJCV)_ , vol. 66, no. 3, pp. 231–259, 2006.
  * [12] A. Collet, M. Martinez, and S. S. Srinivasa, “The MOPED framework: Object recognition and pose estimation for manipulation,” _The International Journal of Robotics Research (IJRR)_ , vol. 30, no. 10, pp. 1284–1306, 2011.
  * [13] S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, P. Fua, and V. Lepetit, “Gradient response maps for real-time detection of textureless objects,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , vol. 34, no. 5, pp. 876–888, 2011.
  * [14] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes,” in _Asian Conference on Computer Vision (ACCV)_. Springer, 2012, pp. 548–562.
  * [15] Z. Cao, Y. Sheikh, and N. K. Banerjee, “Real-time scalable 6DOF pose estimation for textureless objects,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 2441–2448.
  * [16] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, “Learning 6D object pose estimation using 3D object coordinates,” in _European Conference on Computer Vision (ECCV)_. Springer, 2014, pp. 536–551.
  * [17] L. Bo, X. Ren, and D. Fox, “Learning hierarchical sparse features for RGB-(D) object recognition,” _The International Journal of Robotics Research (IJRR)_ , vol. 33, no. 4, pp. 581–599, 2014.
  * [18] A. Krull, E. Brachmann, F. Michel, M. Ying Yang, S. Gumhold, and C. Rother, “Learning analysis-by-synthesis for 6D pose estimation in RGB-D images,” in _International Conference on Computer Vision (ICCV)_ , 2015, pp. 954–962.
  * [19] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again,” in _International Conference on Computer Vision (ICCV)_ , 2017, pp. 1521–1529.
  * [20] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” in _Conference on Robot Learning (CoRL)_ , 2018.
  * [21] P. Marion, P. R. Florence, L. Manuelli, and R. Tedrake, “Label fusion: A pipeline for generating ground truth labels for real RGBD data of cluttered scenes,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1–8.
  * [22] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine, “Learning to poke by poking: Experiential learning of intuitive physics,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016, pp. 5074–5082.
  * [23] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _International Conference on Computer Vision (ICCV)_ , 2015, pp. 1422–1430.
  * [24] G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in _European Conference on Computer Vision (ECCV)_. Springer, 2016, pp. 577–593.
  * [25] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, “Self-taught learning: transfer learning from unlabeled data,” in _International Conference on Machine Learning (ICML)_ , 2007, pp. 759–766.
  * [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [27] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2786–2793.
  * [28] D. Pathak, Y. Shentu, D. Chen, P. Agrawal, T. Darrell, S. Levine, and J. Malik, “Learning instance segmentation by interaction,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 2042–2045.
  * [29] T. Schmidt, R. Newcombe, and D. Fox, “Self-supervised visual descriptor learning for dense correspondence,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 2, pp. 420–427, 2016.
  * [30] P. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learning dense visual object descriptors by and for robotic manipulation,” in _Conference on Robot Learning (CoRL)_ , 2018, pp. 373–385.
  * [31] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme, “Interactive perception: Leveraging action in perception and perception in action,” _IEEE Transactions on Robotics_ , vol. 33, no. 6, pp. 1273–1291, 2017.
  * [32] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 779–788.
  * [33] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in _European Conference on Computer Vision (ECCV)_. Springer, 2016, pp. 21–37.
  * [34] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015, pp. 91–99.
  * [35] J. M. Wong, V. Kee, T. Le, S. Wagner, G.-L. Mariottini, A. Schneider, L. Hamilton, R. Chipalkatty, M. Hebert, and D. M. Johnson, “SegICP: Integrated deep semantic segmentation and pose estimation,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 5784–5789.
  * [36] T. Schmidt, R. A. Newcombe, and D. Fox, “DART: Dense articulated real-time tracking.” in _Robotics: Science and Systems (RSS)_ , vol. 2, no. 1. Berkeley, CA, 2014.
  * [37] C. Eppner, A. Mousavian, and D. Fox, “A billion ways to grasps - an evaluation of grasp sampling schemes on a dense, physics-based grasp data set,” in _Proceedings of the International Symposium on Robotics Research (ISRR)_ , Hanoi, Vietnam, 2019.
  * [38] S. Chitta, I. Sucan, and S. Cousins, “MoveIt!” _IEEE Robotics & Automation Magazine_, vol. 19, no. 1, pp. 18–19, 2012.
  * [39] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _International Conference on Machine Learning (ICML)_. ACM, 2009, pp. 41–48.
  * [40] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The YCB object and model set: Towards common benchmarks for manipulation research,” in _International Conference on Advanced Robotics (ICAR)_. IEEE, 2015, pp. 510–517.