††thanks: These authors contributed equally††thanks: These authors contributed
equally††thanks: These authors contributed equally

# Spatially correlated rotational dynamics reveals strain dependence in
amorphous particle packings

Dong Wang  Department of Physics & Center for Non-linear and Complex Systems,
Duke University, Durham, North Carolina 27708, USA  Department of Mechanical
Engineering & Materials Science, Yale University, New Haven, Connecticut
06520, USA  Nima Nejadsadeghi  Mechanical Engineering, University of Kansas,
1530 W. 15th St. Lawrence, KS 66045-7609, USA  Yan Li  Computer Science &
Engineering, University of Minnesota, Minneapolis, MS, USA  Shashi Shekhar
<EMAIL_ADDRESS>Computer Science & Engineering, University of Minnesota,
Minneapolis, MS, USA  Anil Misra<EMAIL_ADDRESS>Civil, Environmental &
Architectural Engineering, University of Kansas, 1530 W. 15th St. Lawrence, KS
66045-7609, USA  Joshua A. Dijksman<EMAIL_ADDRESS>Physical Chemistry
and Soft Matter, Wageningen University & Research, Stippeneng 4, 6708 WE
Wageningen, The Netherlands

###### Abstract

Microstructural dynamics in amorphous particle packings is commonly probed by
quantifying particle displacements. While rigidity in particle packings
emerges when displacement of particles are hindered, it is not obvious how the
typically disordered displacement metrics connect to mechanical response.
Particle rotations, in contrast, are much less sensitive to confinement
effects, while still sensitive to the mechanics of the packing. So far, little
attention has been paid to connect microscopic rotational motion to mechanics
of athermal amorphous packings. We demonstrate through experimental
measurements that particle packing mechanics can be directly linked to the
rotational motion of even round particles in a sheared packing. Our results
show that the diffusive nature of rotational dynamics is highly strain
sensitive. Additionally, there is substantial spatial correlation in rotation
dynamics that is a function of the particle friction and packing density.
Analysis of our measurements reveals that particle rotation dynamics plays an
essential role in amorphous material mechanics.

††preprint: APS/123-QED

##  I Introduction

Amorphous packings of particles occur in many contexts, ranging from glassy
polymers to colloidal gels and geological sediments. These materials are well
known to have complex deformation behavior. For example, their mechanical
response is often strain history dependent [1, 2, 3, 4]. The amorphous nature
of the microstructure of these systems makes it notoriously challenging to
understand the origin of such strain history dependence and mechanical
behavior in general [5]. Notably, these “granular” material systems are
characterized by a length-scale proportional to particle size, that makes
their theoretical description using classical continuum physics concepts
particularly challenging. For accurate descriptions of amorphous packing
mechanics, the traditional views of continuum mechanics desperately needs
updating from a microscopic point of view. One route towards a more general
continuum description considers material point rotations and has an origin in
the work from the Cosserat brothers [6]. Indeed, it has long been recognized
that the rotational motion of particles in thermally driven amorphous packings
can be linked to slowdown effects and glassy dynamics [7, 8, 9]. Nevertheless,
much progress is needed to include (particle) rotational degrees-of-freedom in
continuum mechanics approaches [10, 11]. Here we show experimentally that even
in an completely athermal amorphous packing, rotational degrees-of-freedom are
directly coupled to its mechanical response, both at the particle level and
via mesoscopic spatial anticorrelations in the rotation field. Our data
suggests that particle rotations are an essential yet overlooked kinematical
quantity in the study of dense amorphous packing. In addition, the spatial
autocorrelation analysis of rotations can reveal essential features in
materials science of a large variety of materials with intrinsic length
scales.

To study the role of rotational degrees of freedom, decoupling rotation from
translation is challenging. In many circumstances the involved molecules,
colloids or grains are not spherically symmetric, hence their rotation
requires also _spatial displacement_ of their neighbors, particularly for high
density, jammed granular materials. To probe the role of _only_ the rotational
degrees of freedom in the strain dependence of amorphous packings, athermal
round particle systems are an optimal prototypical choice. Such particles can
be designed to experience contact friction, which directly couples rotational
degrees of freedom to displacements. In an athermal packing of frictional
disks, shear for even circular particles is thus directly coupled to rotations
without necessarily requiring particle displacements. While rotational
dynamics of spherical particles in athermal packings has been probed via wave-
propagation measurements [12], particle-level experimental evidence that links
rotational degrees of freedom directly to mechanics in amorphous packings has
so far not been obtained. The unique set of experimentally measured data
analyzed in this work shows that the particles’ micro-rotation dynamics are
linked to both the packing density and the particle surface characteristics
(or “friction”) which directly mediates in tangential motion of contacting
circular grain-pairs. We will see that particle rotations display strain-
induced diffusive behavior even at very small strain amplitudes. The diffusive
behavior changes with particle packing density and friction coefficient in
manners consistent with previously observed packing properties [13, 14, 15].
Additionally, particle rotations display non-local correlations as revealed by
spatial auto-correlation measures [16], indicating that the non-local
mechanical effects well known to exist in sheared glassy granular media and
amorphous materials in general [17, 18, 19] can be mediated by rotational
dynamics. Our results show that rotational degrees of freedom are a crucial
element to be considered in the quest to understand the flow behavior of
amorphous materials.

Granular material systems exhibit many non-standard physical phenomena, such
as that of negative group velocity [20, 21], frequency band gaps [22, 23, 24,
25], chirality [26] and load path dependency. The latter is the key physical
phenomenon that underlies our results: collections of discrete athermal
particles have the remarkable property that they can become _rigid_ when
assembled into certain arrangements. Referred to as granular media, these
loose particle packings can resist shear or compression when a sufficient
number of them is present per unit volume, a feature commonly called
_jamming_. Even when not spatially confined, packing of particles can enhance
its rigidity when the assembly is subject to shear strain. This shear
deformation induced rigidification is known as _shear jamming_. The peculiar
strain dependence of granular media can be indicated by the correspondence
between rotational particle motion and collective packing mechanics. Indeed,
some work had already hinted at the relevance of rotational degrees of freedom
of particles in loose particle assemblies for both slow [27, 10, 28, 29] and
fast granular flows [30, 31, 32] and recently also for the statistical
mechanics of sheared packings [33, 34].

##  II Rotation Kinematics

Figure 1: (a) Examples of the UV image taken to track particle orientation.
Blue bars show actual UV marks and white lines indicate tracked orientation.
Red circles mark edges of disks. (b - d) Examples of photoelastic particles
used for different inter-particle friction coefficients $\mu_{l,m,h}$
respectively. (e) The microrotation of grains induced by the first shear step;
particle locations shown in their initial configuration. Results obtained for
$\gamma=0.27$, $\mu_{m}$ with packing fraction $\phi$=0.816. (f) evolution of
the mean $\langle\theta_{i}^{m}\rangle$ and standard deviation $\sigma_{m}$ of
the microrotation, and the rigid body rotation $\theta^{R}$, for five shear
tests at a given density. $\theta^{R}$ grows linearly at a rate of 0.0013 rad
per frame as expected from the imposed strain.

Packing of disk-shaped particles at different packing fractions $\phi$, were
subjected to quasi-static stepwise shear to observe the diffusive rotational
dynamics of particles. The packing were imaged after each strain step. For
details of the experiments, see the Methods section. In our experiments we
have used quasi-two dimensional packing, as three dimensional shear
experiments have the propensity for formation of finite shear localization
bands into which the particle rotations typically concentrate [35, 36, 37,
29]. In contrast, our two dimensional geometry with articulated base allows us
to suppress shear localization entirely [13, 14]. A fluorescent bar placed on
the particles allows us to track the absolute orientation of every particle in
the packings; see Fig. 1a. We use three different particle friction
coefficients and refer to these as $\mu_{l}$, $\mu_{m}$ and $\mu_{h}$;
examples of these particle types are shown in Fig. 1b-d. We probe the dynamics
of orientations obtained from image analysis of each frame. An example
rotation field from a single shear step is shown in Fig. 1e for $\mu_{m}$
particles at packing fraction $\phi=0.816$. While the grain displacements tend
to follow the imposed macro-scale deformation field, there exists fluctuations
from the imposed linear macro-scale deformation field in the grain centroid
displacements that follow a complex process dictated by not only the nearest
or contacting grains, but also by their neighbors and by extension an
increasingly larger neighborhood for every grain [38, 39, 15].

Figure 2:  Standard deviation of rotations as a function of imposed shear
strain in three different packing fractions for the (a) $\mu_{l}$, (b)
$\mu_{m}$, and (c) $\mu_{h}$ particles. All experiments are repeated five
times. (d) and (e) show, respectively, the variation of the parameters $D$ and
$n$ as a function of packing fraction of the sets in $\mu_{l,m,h}$. The
highlighted data in yellow are the data used in panels a-c and Fig. 3.

Similarly, the grain rotation observed in Fig. 1e can be decomposed into two
parts. One part of the rotation of each grain is a result of the imposed
affine macro-scale deformation field, which contributes an overall rigid body
rotation. The second rotation contribution is due to the micro-scale phenomena
such as individual grain spin that we call _microrotation_. Denoting the
rotation of grain $i$ by $\theta_{i}$ and the macro-scale rigid body rotation
by $\theta^{R}$, the microrotation of grain $i$, $\theta_{i}$, is obtained as
$\theta_{i}^{m}=\theta_{i}-\theta^{R}$. The mean of the microrotations
$\langle\theta_{i}^{m}\rangle$ and the standard deviation of microrotations
$\langle{\theta_{i}^{m}}^{2}\rangle=\sigma_{m}$ change as a function of strain
as observed in Fig. 1f for all five repeats done for $\mu_{m}$ at
$\phi=0.816$. The initial frame is taken as the reference configuration to
obtain the evolution of the grain-spin measures as a function of imposed
strain. The rigid body rotation $\theta^{R}$ grows linearly with strain as
expected from the linearly increasing strain field imposed on the packing.
Notably, the mean microrotation $\langle\theta_{i}^{m}\rangle$ is zero for the
packing: that is, there is no preferred direction in which grain rotation
_fluctuations_ occur. This null-result is highly reproducible between
different repeats of the experiments and consistent with earlier numerical
simulations [40, 28, 41, 27, 37]. It is also noteworthy that the
microrotations follow a nearly Gaussian distribution for all cases (see
results in SI). However, the amplitude of grain rotation fluctuations
increases strongly with strain. Note that some of such shear induced
rotational fluctuations have been observed in the experimental work of
Matsushima in [10] on nonspherical particles.

Focusing further on the growth of $\sigma_{m}(\gamma)$ we see that its strong
growth with $\gamma$ and the reproducibility among different initial
configurations is also observed for different $\phi$ over the entire range of
relevant densities and $\mu$ as shown in Fig. 2a-c. Up to a strain of 0.15,
$\sigma_{m}$ can be well described by the empirical relation
$\sigma_{m}=D\gamma^{n}+\sigma_{0}$ as shown by the good quality of the fits.
It is tempting to interpret prefactor $D$ as a diffusion constant as done
previously for rotations induced by thermal fluctuations [42, 43]. Power law
index $n$ indicates the (weakly) nonlinear strain dependence and $\sigma_{0}$
is a possible offset that is negligible for all experiments. We see that the
strength of the fluctuations captured by $D(\phi)$ and $n(\phi)$ is very
sensitive to friction $\mu$ as shown in Fig. 2d,e. The friction dependence
does however capture the mechanical performance of the packing as well: at
large $\mu$, particle interactions associated with rotation are stronger even
at smaller $\phi$, and this trend is observed in both $D(\phi)$ and $n(\phi)$.
When considering $D$ as a diffusion constant, its thermal analogue would be
given by the ratio of thermal fluctuations and viscous damping. Such
competition can also be seen in the rotational diffusion: both $D$ and $n$
indicate that there are two mechanisms that play a role in the rotational
diffusion, which is especially visible for $\mu_{m}$. Initially, $D,n(\phi)$
grows with $\phi$, indicating the enhanced particle interactions that give
more fluctuations in rotations. However, above a certain $\phi_{c}(\mu)$, $D$
decreases, and above the packing fraction, $\phi\approx 0.80$, parameters
$D,n$ tend towards a plateau, indicating that competing mechanism emerge in
high packing fractions suppressing the growth of further rotational
fluctuations. Steric hindrance does play a role for $\mu_{h}$, the gear-shaped
particles, but less so for the much smoother $\mu_{l,m}$ particles. The
exponent for the $\mu_{h}$ can be as high as 1.4, indicating superdiffusive
behavior if we consider $\gamma$ as a time variable and
$\langle{\theta_{i}^{m}}^{2}\rangle=\sigma_{m}$ as a displacement fluctuation
metric. Interestingly, the values for $n(\phi)$ become _independent_ of $\mu$
above a volume fraction of about 0.81, tending towards a linear behavior at
very high packing fraction. These trends are even more visible if we consider
less strain, see Supplementary Information.

Figure 3: (a) The average neighborhood variance $\sigma_{m}(\gamma)$;
different colors indicate different $\mu$; different hues represent different
$\phi$; color scheme applies to all panels. (b) $I$ as a function of $\gamma$.
(c) $I_{i}$ standard deviation $\sigma_{I}$ as as function of $\gamma$. (d)
$\mu_{l}$, $\phi=0.828$ local Moran’s I. (e) $\mu_{h}$, $\phi=0.807$ local
Moran’s I.

##  III Correlations in micro-rotations

The observed Gaussian nature of the particle rotation fluctuations belies the
underlying correlations in particle motion as expected to exist in dense
amorphous packings where many grains are in contact. There are long range
correlations in particle rotations as visible in our two step approach to
quantifying the spatial autocorrelation of rotations, leading to a
mathematically well defined quantitative system average signal widely used for
geographical data called Moran’s $I$ [16]. We first compute the particle
average neighborhood rotational variance $S_{n}$. For details, see the Methods
sections. A positive/negative value of $S_{n}$ means that in general a
particle rotates in the same/opposite direction as its neighborhood. Fig. 3(a)
shows the average neighborhood variance of materials with different $\mu$ and
packing fraction $\phi$. The average neighborhood variance is clearly
monotonically dependent on both $\mu$ and $\phi$ and grows with $\gamma$: the
higher friction or density, the lower the average neighborhood variance, which
means greater difference between a particle’s micro-rotation and its
neighborhood’s. This anticorrelation makes mechanical sense: gear-like motion
forces rotation of opposite direction in interlocking particles.

A large absolute value of $S_{n}$ may however be caused by either
(dis)similarity between neighborhood particle micro-rotations, or a large
variance in the micro-rotation. To focus only on the comparison of the
dissimilarity among neighborhood particles micro-rotation across the packing,
we have to normalize $S_{n}$ by the variance of the particle micro-rotation,
$\sigma_{m}^{2}$. We showed the dynamics of $\sigma_{m}^{2}$ and its non-
monotonic dependence on friction and packing fraction in Fig. 2. By computing
$S_{n}/\sigma_{m}^{2}$, we arrive immediately at the system wide spatial
autocorrelation metric called Moran’s $I$. Generally, the micro-rotations of
the grains in all the materials in these analyses are negatively
autocorrelated: the grains rotate like a chain of gears to some extent. Figure
3(b) shows the trends of $I$ as the shear strain increases. The differences in
behavior for $\mu_{l,m,h}$ is evident: low friction particles have a weak
spatial autocorrelation, whereas particles with higher friction coefficient
develop stronger autocorrelations, with $I$ decreasing to -0.3. The difference
between the packings with a different $\phi$ is small but not insignificant.
In general, anti-autocorrelations increase with larger packing fractions.
Strikingly, also the rotational correlations are very strain sensitive, with
3% strain being enough to indicate significant difference between packings of
the different $\phi$ and $\mu$.

We go one step further and use the normalized neighborhood variance to gain
insight into the local mechanics of sheared amorphous packings. Spatial
autocorrelations as captured by $I$ are not the same everywhere; in fact there
are clusters of (anti)correlated rotations in $I_{i}$. We can quantify the
local variability of these correlations by computing the standard deviation
$\sigma_{I}$ of $I_{i}$. This metric captures the rotational “floppyness” in
the packing: at large $\phi$ in a highly overconstrained system, interlocking
grains must all have the same rotational behavior so the variability of $I$ in
the packing should be small. At smaller $\phi$, there are more ways to reach
mechanical equilibrium, hence the variability among correlations should be
higher. Similarly, $\sigma_{I}$ should express the phenomena of shear jamming:
at small strain, the shear jamming mechanisms has not been activated yet, so
$\sigma_{I}$ is small. As strain increases, the packing moves from partially
to completely constrained and should thus achieve a small $\sigma_{I}$.
Finally, the role of $\mu$ should also be non-linear: at small and large
$\mu$, the rotational variability should be high as per previous arguments, so
$\sigma_{I}(\mu)$ should have an optimum. We observe indeed all these
mechanically reasonable trends in $\sigma_{I}(\phi,\gamma,\mu)$. $I_{i}$
standard deviation is strain dependent, exhibiting a distinct peak floppiness
at about 3% strain. Note that at these strain levels, system level pressure is
undetectable, highlighting again the sensitive nature of rotations. The
connection to the rotational diffusion is also still visible in the
fluctuations of the anticorrelated micro-rotation: observe how for
$\phi>0.80$, $\sigma_{I}$ is small for all $\mu$, precisely where also the
diffusivity of rotation becomes independent of $\mu$. Finally, we show two
examples of the spatial distribution $I_{i}(x,y)$ for two situations
$\mu_{l}$, $\phi=0.828$ and $\mu_{h}$, $\phi=0.807$ in Fig. 3d,e. These
examples clearly show clusters of isotropic and and anisotropic shapes
emerging along boundaries and in the bulk of the packing. Spatial fluctuations
can span up to ten particle diameters and can be string-like or globular,
highlighting again the spatial anisotropy that can build up in the amorphous
system (see Supplementary Information videos). While the complete spatial
dynamics is neighborhood rotation similarity is challenging to interpret due
to the dual and non-monotonic role of both friction and density, we can
clearly evidence particle rotation becoming an essential parameter necessary
to include in continuum modeling theories with non-local mechanical couplings
inside sheared amorphous packings.

##  IV Conclusions

We have shown that simple shear induces spatially correlated fluctuations in
the a rotational dynamics of round, frictional particles. Individual particle
motion is diffusive, and diffusive motion is $\mu$ and $\phi$ dependent as one
would expect based on the mechanical characteristics of the packing. The local
neighborhood of particles shows on-average anticorrelated motion that reveals
that two distinct mechanisms affect the mechanics of individual grains.
Rotational motion fluctuations indicate the state of the system early in the
deformation regime after a few percent shear strain, even though the _average_
particle micro-rotation is zero. Our results indicate that rotational motion
is a highly relevant field in the study of amorphous particulate materials,
ranging from sands to frictional emulsions, colloids and even molecular
glasses. Beyond materials analyses, the results have a broader relevance to
spatial data science, particularly in reference to the “first law of
Geography” [44] stating that nearby things are similar. The value of the
widely used geographical spatial autocorrelation measure Moran’s $I$ is
negative for granular materials systems with a clear physical interpretation
related to particle friction. This finding is in contrast with a large
majority of spatial datasets coming from human-scale natural systems which
have positive spatial autocorrelation. Intriguingly, the role of absolute
interparticle orientations has long been recognized for system mechanics: the
role of the bond angle is recognized as essential in constraint counting
approaches for glassy polymeric systems [45] and is also relevant for protein
folding dynamics [46]. Not surprisingly rotational dynamics has been measured
indirectly on a system scale via dielectric spectroscopy [47], for example to
probe glassy dynamics in rotational degrees of freedom in nonspherically
symmetric glassforming molecules. Note that not only friction can make the
rotational degree of freedom relevant for the packing dynamics. Rotations also
play a role for particle packings that are composed of aspherical, adhesive or
deformable particles, which covers many types of particulate materials,
ranging from granular materials to colloids, proteins [48], emulsions and even
metamaterials in which the node hinges are not ideal [26]. In particular, it
is of interest to explore how energy is stored in sheared granular packings
and how rotations and friction in contacts play a role in this. Our work thus
suggest that rotational dynamics are a potentially unifying characteristic
through which the often suggested similarity among amorphous materials can be
understood.

###### Acknowledgements.

We thank the organizers and participants of the Lorentz Center workshop
“Granular Matter Across Scales” for fostering an environment where the seeds
for this work were planted. We are grateful to the late Robert Behringer for
always reminding us of the importance of grain rotations and friction. AM and
NN are supported in part by the United States National Science Foundation
grant CMMI-1727433 and EEC-1840432 (which also involves SS). YL is supported
by the University of Minnesota Doctoral Dissertation Fellowship.

##  V Methods

###  V.1 Experimental Setup

In our experiments, we analyze a series of experiments that allow for the
tracking of rotation of every disk-shaped particle in a $\sim$1000-particles
large shear environment in which shear bands and other large scale
inhomogeneities have been completely eliminated, as reported elsewhere [13,
14]. Shear is applied quasi-statically from an isotropic stress free state and
tracked during the initial shear transient up to a strain of 0.5. Previous
experiments have described dilatancy and displacement dynamics in these
packings [13, 14]. Within the scope of the experimentation in the current
research, we study the effect of inter-particle friction using granular
assemblies with controlled variations of friction coefficient, as well as the
effect of different initial packing fractions upon the response of the
granular assembly. One set of particles was cut from photoelastic sheets as in
previous experiments[13], having an inter-particle friction coefficient
$\mu_{m}$ of approximately 0.7. After conducting experiments with this set, we
wrapped these particles with teflon tape. Dry teflon-teflon contacts have a
friction coefficient of $\mu_{l}\sim$0.15 [15]. A third set of data was
obtained with photoelastic disks cut with fine teeth on their circumference so
that particles will interlock when they come into contact. Such a particle
shape mimics an extremely large friction coefficient; we refer to these
particles as $\mu_{h}$. The diameter ratio of big to small disks is 1.25:1,
and the number ratio is roughly 1:3.3 (big to small) for each packing.
Particles were first randomly placed in the shear cell and manually relaxed
until no inter-particle contact force very visible by eye. Then starting from
either a parallelogram or a rectangle, the shear cell was deformed by strain
steps of 0.0027. The system was then relaxed for 10 seconds followed by taking
three kinds of pictures: one with white light, one with polarized light, and
one with UV light. These three pictures reveal particle positions, particle
contact forces/pressure, and particle orientation, respectively. Such a
process of shearing, relaxing and picture taking was repeated until a certain
total shear strain was achieved. For each packing fraction and friction
coefficient, we repeated the experiment five times with the exception of the
lowest density $\mu_{l}$ runs. Note that the analysis of the images acquired
during the experiment reveals that not all the grains were detected in all
frames, where some grains move out of or inside the boundaries of the images
from one frame to another. As a result, for the analysis performed in the
current paper, only grains common between all the frames were considered, and
the grains present at one frame and not detected in another frame are
excluded. Moreover, the grains on the boundary were removed from the analysis.

###  V.2 Rotations

The rigid body rotation between any two frames can be measured as half of the
difference in the slope of straight lines fitted to the coordinates of grains
centroids in the two frames. We note that, in general, the relation between
the measured change in slope and the rigid rotation is nonlinear especially in
finite deformation. However, a linear relation in the current analysis for the
considered shear strain range is a good approximation.

###  V.3 Neighborhood Variance

The neighborhood variance of each particle refers to the product of its micro-
rotation deviation from mean micro-rotation and the mean micro-rotation
deviation of its Voronoi neighborhood from mean micro-rotation. We compute the
“average neighborhood variance” by

$S_{n}=\frac{Z^{T}WZ}{N-1},\text{ and
}Z=\Theta^{m}-\langle\theta_{i}^{m}\rangle,$ (1)

where $\Theta^{m}$ is a vector of particles’ micro-rotation whose $i$th
element is $\theta_{i}^{m}$, $\langle\theta_{i}^{m}\rangle$ is the mean of all
particles’ micro-rotation, and $N$ is the number of particles. $W$ is the row-
wise normalized spatial weight matrix. A commonly used spatial weight matrix
is the adjacency matrix whose element at the $i$th row and $j$th column
indicates whether the $i$th particle is adjacent to the $j$th particle. If the
particles are adjacent, the element is 1. Otherwise, the element is 0. A row-
wise normalized spatial weight matrix is gotten by dividing each row of a
spatial weight matrix by the row sum of the matrix. We conducted analyses of
the materials with different surfaces and density: $\mu_{l}$;
$\phi=0.783,0.810,0.828$; $\mu_{m}$; $\phi=0.692,0.758,0.816$; $\mu_{h}$;
$\phi=0.713,0.744,0.807$. The observations of each grain were the micro-
rotation. We constructed Delaunay triangles to link grains with their neighbor
grains, and removed the link whose length was greater than the sum of the
radius of the two grains connected by the link.

###  V.4 Global Moran’s $I$

Spatial autocorrelation is a measure of the correlation between spatially
proximate observations. Positive spatial autocorrelation is the tendency for
spatially proximate observations to be similar, while negative spatial
autocorrelation means spatially proximate observations tend to be different.
Global Moran’s $I$ is defined as follows:

$I=\frac{Z^{T}WZ}{Z^{T}Z}=\sigma_{n}/\sigma_{m},\text{ and
}Z=\Theta^{m}-\langle\theta_{i}^{m}\rangle,$ (2)

where $\Theta^{m}$ is a vector of particles’ micro-rotation whose $i$th
element is $\theta_{i}^{m}$, $\langle\theta_{i}^{m}\rangle$ is the mean of all
particles’ micro-rotation, which is negligible. $W$ is the the row-wise
normalized spatial weight matrix. This metric measures the average spatial
autocorrelation of the entire dataset. The expected value of global Moran’s
$I$ under the null hypothesis of no spatial autocorrelation is
$E(I)=-\frac{1}{N-1}$, where $N$ is the number of observations. In other
words, the more observations there are, the closer the expectation to 0.
Values of $I$ usually range from -1 to +1. Values significantly below $E(I)$
indicate negative spatial autocorrelation and values significantly above
$E(I)$ indicate positive spatial autocorrelation.

###  V.5 Local Moran’s $I$

There are cases where there is no global trend of spatial autocorrelation, but
there are local communities where spatial autocorrelation is strong. Local
Moran’s $I$ is used to represents the spatial autocorrelation within the local
neighborhood of each observation, which is defined as follows:

$I_{i}=\frac{z_{i}W_{i:}Z}{Z^{T}Z/(N-1)},\text{ and
}z_{i}=\theta_{i}^{m}-\langle\theta_{i}^{m}\rangle,$ (3)

where $\theta_{i}^{m}$ is the ith particle’s micro-rotation. A positive value
of $I_{i}$ means within the $i$th observation’s neighborhood the observations
are similar, while a negative value means the observations are different. In
order to analyze whether the local communities in a dataset are homogeneous
regarding to spatial autocorrelation, we compute the standard deviation of
local Moran’s $I$ defined as $\sigma_{I}$. The greater this standard
deviation, the greater the differences between local communities.

## References

  * Lade [1977] P. V. Lade, Elasto-plastic stress-strain theory for cohesionless soil with curved yield surfaces, International Journal of Solids and Structures 13, 1019 (1977).
  * Viasnoff and Lequeux [2002] V. Viasnoff and F. Lequeux, Rejuvenation and overaging in a colloidal glass under shear, Phys. Rev. Lett. 89, 065701 (2002).
  * Lee  _et al._ [2009] H.-N. Lee, K. Paeng, S. F. Swallen, and M. D. Ediger, Direct measurement of molecular mobility in actively deformed polymer glasses, Science 323, 231 (2009), https://science.sciencemag.org/content/323/5911/231.full.pdf .
  * Tighe [2014] B. P. Tighe, Shear dilatancy in marginal solids, Granular Matter 16, 203 (2014).
  * Berthier  _et al._ [2011] L. Berthier, G. Biroli, J.-P. Bouchaud, L. Cipelletti, and W. van Saarloos,  _Dynamical heterogeneities in glasses, colloids, and granular media_ , Vol. 150 (OUP Oxford, 2011).
  * Cosserat and Cosserat [1909] E. Cosserat and F. Cosserat, Theory of deformable bodies (translated by dh delphenich), Paris: Sorbonne: Herman and Sons (1909).
  * Schwartz  _et al._ [1984] L. M. Schwartz, D. L. Johnson, and S. Feng, Vibrational modes in granular materials, Phys. Rev. Lett. 52, 831 (1984).
  * Stillinger and Hodgdon [1994] F. H. Stillinger and J. A. Hodgdon, Translation-rotation paradox for diffusion in fragile glass-forming liquids, Phys. Rev. E 50, 2064 (1994).
  * Edmond  _et al._ [2012] K. V. Edmond, M. T. Elsesser, G. L. Hunter, D. J. Pine, and E. R. Weeks, Decoupling of rotational and translational diffusion in supercooled colloidal fluids, Proceedings of the National Academy of Sciences 109, 17891 (2012).
  * Matsushima  _et al._ [2003] T. Matsushima, H. Saomoto, Y. Tsubokawa, and Y. Yamada, Grain rotation versus continuum rotation during shear deformation of granular assembly, Soils and Foundations 43, 95 (2003).
  * Poorsolhjouy and Misra [2019] P. Poorsolhjouy and A. Misra, Granular micromechanics based continuum model for grain rotations and grain rotation waves, J. Mech. Phys. Solids 129, 244 (2019).
  * Merkel  _et al._ [2011] A. Merkel, V. Tournat, and V. Gusev, Experimental evidence of rotational elastic waves in granular phononic crystals, Phys. Rev. Lett. 107, 225502 (2011).
  * Ren  _et al._ [2013] J. Ren, J. A. Dijksman, and R. P. Behringer, Reynolds pressure and relaxation in a sheared granular system, Phys. Rev. Lett. 110, 018302 (2013).
  * Wang  _et al._ [2018] D. Wang, J. Ren, J. A. Dijksman, H. Zheng, and R. P. Behringer, Microscopic origins of shear jamming for 2d frictional grains, Phys. Rev. Lett. 120, 208004 (2018).
  * Wang  _et al._ [2020] D. Wang, J. A. Dijksman, J. Barés, J. Ren, and H. Zheng, Sheared amorphous packings display two separate particle transport mechanisms, Phys. Rev. Lett. 125, 138001 (2020).
  * Moran [1950] P. A. Moran, Notes on continuous stochastic phenomena, Biometrika 37, 17 (1950).
  * Hébraud and Lequeux [1998] P. Hébraud and F. Lequeux, Mode-coupling theory for the pasty rheology of soft glassy materials, Phys. Rev. Lett. 81, 2934 (1998).
  * Bocquet  _et al._ [2009] L. Bocquet, A. Colin, and A. Ajdari, Kinetic theory of plastic flow in soft glassy materials, Phys. Rev. Lett. 103, 036001 (2009).
  * Henann and Kamrin [2013] D. L. Henann and K. Kamrin, A predictive, size-dependent continuum model for dense granular flows, Proceedings of the National Academy of Sciences 110, 6730 (2013).
  * Nejadsadeghi and Misra [2020a] N. Nejadsadeghi and A. Misra, Role of higher-order inertia in modulating elastic wave dispersion in materials with granular microstructure, International Journal of Mechanical Sciences 185, 105867 (2020a).
  * Wei  _et al._ [2020] L.-S. Wei, Y.-Z. Wang, and Y.-S. Wang, Nonreciprocal transmission of nonlinear elastic wave metamaterials by incremental harmonic balance method, International Journal of Mechanical Sciences 173, 105433 (2020).
  * Mouraille and Luding [2008] O. Mouraille and S. Luding, Sound wave propagation in weakly polydisperse granular materials, Ultrasonics 48, 498 (2008), selected Papers from ICU 2007.
  * Boechler  _et al._ [2011] N. Boechler, J. Yang, G. Theocharis, P. G. Kevrekidis, and C. Daraio, Tunable vibrational band gaps in one-dimensional diatomic granular crystals with three-particle unit cells, Journal of Applied Physics 109, 074906 (2011).
  * Göncü  _et al._ [2012] F. Göncü, S. Luding, and K. Bertoldi, Exploiting pattern transformation to tune phononic band gaps in a two-dimensional granular crystal, The Journal of the Acoustical Society of America 131, EL475 (2012), https://doi.org/10.1121/1.4718384 .
  * Misra and Poorsolhjouy [2016] A. Misra and P. Poorsolhjouy, Granular micromechanics based micromorphic model predicts frequency band gaps, Continuum Mech. Thermodyn. 28, 215 (2016).
  * Misra  _et al._ [2020] A. Misra, N. Nejadsadeghi, M. De Angelo, and L. Placidi, Chiral metamaterial predicted by granular micromechanics: verified with 1d example synthesized using additive manufacturing, Continuum Mechanics and Thermodynamics 32, 1497–1513 (2020).
  * Misra and Jiang [1997] A. Misra and H. Jiang, Measured kinematic fields in the biaxial shear of granular materials, Computers and Geotechnics 20, 267 (1997).
  * Kuhn and Bagi [2004] M. R. Kuhn and K. Bagi, Contact rolling and deformation in granular media, International journal of solids and structures 41, 5793 (2004).
  * Andò  _et al._ [2012] E. Andò, S. A. Hall, G. Viggiani, J. Desrues, and P. Bésuelle, Grain-scale experimental investigation of localised deformation in sand: a discrete particle tracking approach, Acta Geotechnica 7, 1 (2012).
  * Seto  _et al._ [2013] R. Seto, R. Mari, J. F. Morris, and M. M. Denn, Discontinuous shear thickening of frictional hard-sphere suspensions, Phys. Rev. Lett. 111, 218301 (2013).
  * Lin  _et al._ [2015] N. Y. Lin, B. M. Guy, M. Hermes, C. Ness, J. Sun, W. C. Poon, and I. Cohen, Hydrodynamic and contact contributions to continuous shear thickening in colloidal suspensions, Physical review letters 115, 228304 (2015).
  * Singh  _et al._ [2020] A. Singh, C. Ness, R. Seto, J. J. de Pablo, and H. M. Jaeger, Shear thickening and jamming of dense suspensions: The “roll” of friction, Phys. Rev. Lett. 124, 248005 (2020).
  * Ciamarra  _et al._ [2012] M. P. Ciamarra, P. Richard, M. Schröter, and B. P. Tighe, Statistical mechanics for static granular media: open questions, Soft Matter 8, 9731 (2012).
  * Sun  _et al._ [2020] X. Sun, W. Kob, R. Blumenfeld, H. Tong, Y. Wang, and J. Zhang, Friction-controlled entropy-stability competition in granular systems (2020), arXiv:2007.14145 [cond-mat.soft] .
  * Cheng and Wang [2019] Z. Cheng and J. Wang, Quantification of the strain field of sands based on x-ray micro-tomography: A comparison between a grid-based method and a mesh-based method, Powder Technology 344, 314 (2019).
  * Hall  _et al._ [2010] S. A. Hall, M. Bornert, J. Desrues, Y. Pannier, N. Lenoir, G. Viggiani, and P. Bésuelle, Discrete and continuum analysis of localised deformation in sand using x-ray $\mu$ct and volumetric digital image correlation, Géotechnique 60, 315 (2010).
  * Alshibli and Alramahi [2006] K. A. Alshibli and B. A. Alramahi, Microscopic evaluation of strain distribution in granular materials during shear, Journal of geotechnical and geoenvironmental engineering 132, 80 (2006).
  * Misra and Poorsolhjouy [2017] A. Misra and P. Poorsolhjouy, Grain- and macro-scale kinematics for granular micromechanics based small deformation micromorphic continuum model, Mechanics Research Communications 81, 1 (2017).
  * Nejadsadeghi and Misra [2020b] N. Nejadsadeghi and A. Misra, Extended granular micromechanics approach: a micromorphic theory of degree n, Mathematics and Mechanics of Solids 25, 407 (2020b).
  * Aharonov and Sparks [2002] E. Aharonov and D. Sparks, Shear profiles and localization in simulations of granular materials, Physical Review E 65, 051302 (2002).
  * Calvetti  _et al._ [1997] F. Calvetti, G. Combe, and J. Lanier, Experimental micromechanical analysis of a 2d granular material: relation between structure evolution and loading path, Mechanics of Cohesive-frictional Materials: An International Journal on Experiments, Modelling and Computation of Materials and Structures 2, 121 (1997).
  * Kim  _et al._ [2011] M. Kim, S. M. Anthony, S. C. Bae, and S. Granick, Colloidal rotation near the colloidal glass transition, The Journal of Chemical Physics 135, 054905 (2011).
  * Vivek and Weeks [2017] S. Vivek and E. R. Weeks, Decoupling of translational and rotational diffusion in quasi-2d colloidal fluids, The Journal of Chemical Physics 147, 134502 (2017).
  * Tobler [1970] W. R. Tobler, A computer movie simulating urban growth in the detroit region, Economic geography 46, 234 (1970).
  * Thorpe [1995] M. Thorpe, Bulk and surface floppy modes, Journal of Non-Crystalline Solids 182, 135 (1995).
  * Jacobs  _et al._ [2002] D. J. Jacobs, L. A. Kuhn, and M. F. Thorpe, Flexible and rigid regions in proteins, in  _Rigidity Theory and Applications_, edited by M. F. Thorpe and P. M. Duxbury (Springer US, Boston, MA, 2002) pp. 357–384.
  * Madden and Kivelson [1984] P. Madden and D. Kivelson, A consistent molecular treatment of dielectric phenomena, Adv. Chem. Phys 56, 467 (1984).
  * Haridasan  _et al._ [2019] N. Haridasan, S. K. Kannam, S. Mogurampelly, and S. P. Sathian, Rotational diffusion of proteins in nanochannels, The Journal of Physical Chemistry B 123, 4825 (2019), pMID: 31117604.


[figure]subcapbesideposition=center

# Synthetic Image Data for Deep Learning

Jason W. Anderson  BMW IT Research Center
Greenville, SC
<EMAIL_ADDRESS>Marcin Ziolkowski  BMW IT Research Center
Greenville, SC
<EMAIL_ADDRESS>Ken Kennedy  BMW IT Research Center
Greenville, SC
<EMAIL_ADDRESS>Amy W. Apon  School of Computing
Clemson University
Clemson, USA
<EMAIL_ADDRESS>
###### Abstract

Realistic synthetic image data rendered from 3D models can be used to augment
image sets and train image classification semantic segmentation models. In
this work, we explore how high quality physically-based rendering and domain
randomization can efficiently create a large synthetic dataset based on
production 3D CAD models of a real vehicle. We use this dataset to quantify
the effectiveness of synthetic augmentation using U-net and Double-U-net
models. We found that, for this domain, synthetic images were an effective
technique for augmenting limited sets of real training data. We observed that
models trained on purely synthetic images had a very low mean prediction IoU
on real validation images. We also observed that adding even very small
amounts of real images to a synthetic dataset greatly improved accuracy, and
that models trained on datasets augmented with synthetic images were more
accurate than those trained on real images alone.

Finally, we found that in use cases that benefit from incremental training or
model specialization, pretraining a base model on synthetic images provided a
sizeable reduction in the training cost of transfer learning, allowing up to
90% of the model training to be front-loaded.

##  I Introduction

In the field of image classification and segmentation with deep learning
systems, access to sets of labelled training images with sufficient quantity
and quality can be a formidable barrier to training an accurate model.
Collecting, segmenting, and labelling high quality images can be prohibitively
expensive both in time and monetary cost. In some cases, the barrier can be
lowered by pretraining a model with a generic dataset such as ImageNet[1] and
then fine-tuned on a smaller set of images more directly related to the
project goals. However, depending on the specificity requirements for the
final model, a generalized dataset may not be useful.

A common alternative to vast quantities of readily available general images
and costly task-specific images is synthetic image generation, where a 3D
computer model of a scene relevant to the deep learning model is rendered to
an image, segmented and/or classified, and then used to augment the training
data available to the model. Synthetic data has been used successfully in a
growing body of research, in many cases reducing the overall cost of training
a model.

Advantages to using synthetic images are not limited to overcoming the time
and safety constraints of capturing and annotating real images. 3D modeling
systems are very flexible – scenes and assets can be changed and re-rendered
with a cost likely far less than the real world equivalent. For example, in
the use cases presented in this work, the cost of changing the vehicle CAD
model to a brand new vehicle or a new model year and then generating a new
training set is far less than that of acquiring new real world examples,
especially when the goal is to have a working detection system before the
model enters production. The costs of developing a synthetic image generation
pipeline specific to a model’s goals can be further recuperated in cases where
similar images can be used to train other models, potentially requiring only
minor alterations to the generator.

While the body of work around using synthetic images in deep learning models
has become broadened in recent years, we have found little exploration of
using synthetic images to pretrain a multistage segmentation model such as the
recently proposed DoubleU-net which has been shown to be highly accurate in
some applications. Our motivations in this work are to explore the performance
effects of training such a model in various combinations of synthetic and real
images.

We present our research on synthetic image training in the context of a real
world anomaly detection system, including the results of testing on a large
set of annotated proprietary production images. We believe the methodology
presented here can be readily applied to other systems, and make the case that
synthetic images can replace the real images and still achieve a potentially
useful level of performance.

The remainder of this paper is organized as follows. Section II describes
concepts related to deep learning systems, synthetic data generation, and the
specific models used in this paper. Section III describes the technologies and
processes used to generate synthetic images. Section IV shows how synthetic
data can augment real images to improve model accuracy. Section V compares
those results to techniques using pretrained models and transfer learning.
Finally, Section VI summarizes our key results and addresses further questions
remaining for exploration.

##  II Background and Related Work

Synthetic data can be used to train deep learning models in a number of ways.

First, in one extreme the model may be trained with only synthetic images,
which can be useful in models where acquiring examples of desired detection
conditions can be time consuming or unsafe. For example, sufficient examples
of rare flaws in products on an assembly line could be time consuming to
capture for a quality control model, and examples of unsafe conditions may be
challenging to acquire for a video surveillance system. There has been some
success with using purely synthetic data to train models [2, 3, 4], and may be
a good option depending on the use case. Real images, if they exist, can be
used as all or part of the test set to prove the model’s accuracy.

Next, synthetic images may be mixed with real images in some combination,
augmenting the size and/or variation of the training set presented to the
model. In published research, this method has been used to successfully
decrease model training cost or improve model accuracy, and in some cases
both[5, 6, 7].

Finally, synthetic images can also be used to pretrain a model in a two-stage
process, either by fitting a model to the synthetic set and then increasing
the model bias toward real world examples by iterating over the real image
set, or in a multi-model system such as DoubleU-net[8], which is the primary
focus of this paper. This method is similar to using generalized image sets to
pretrain a system (such as robotic vision) on patterns common to the real
world, and then secondary training to adapt the model to a specific
environment. [sort out citations for examples here]

Jhang et al.[9] demonstrated training a Faster R-CNN[10] object detection
model using synthetic images annotated with Unity Perception and generated at
scale with Unity Simulation. They found that while a model trained purely on a
large (400,000) set of synthetic images performed poorly at detecting objects
in situations with occlusions and low lighting, augmenting the synthetic
images with a small number of real images significantly improved the detection
accuracy over a model trained purely on a small (760) set of real images.
Their work was inspired by and complements findings from Hinterstoisser et
al.[4], who described a method for domain randomization by composing a
backdrop of random objects in front of which the objects of interest are
rendered and labeled. Our process is distinguished by using a randomly
oriented ”skybox” surrounding the subject of interest, which achieves domain
randomization with lowered scene complexity and randomized reflections.

Another method of domain adaptation to insert simulated objects of interest
into real images, such as in [11].

Rendered images of 3D scenes have been used to train object detection models
for a long time, as exemplified by [12] and [13]. More recently, advances in
3D rendering techniques have made photorealistic image generation practical.
[14] [15] [16]

Other researchers have applied full domain randomization[17] to synthetic
image generation with varying degrees of success. [4, 18, 19]. Our approach is
a hybrid between full domain randomization and photorealistic rendering,
varying the lighting and subject/background orientation and random sampling
from a set of realistic textures. The approaches described in [20], [21] and
[22] are most similar to our own in this regard.

Successful specialization of U-net models has been achieved [23, 24] using VGG
encoders[25] pretrained on the ImageNet[1] dataset.

Recent research has shown that models developed using synthetic data can be
used as a basis for more specific models. This transfer learning can be used
in many tasks, such as enhancing detection of object position in [26, 11] and
separating target objects from visual distractors in [27].

##  III Synthetic Image Generation

In this section we describe the tools and workflow developed for creating
synthetic images, followed by our experiment designs for validating the output
images and using the generated data to train a deep learning model. Software
used includes Unity 2020.1, Unity High Definition Rendering Pipeline (HDRP)
7.4.1, and PiXYZ Plugin 2019.2.1.14.

###  III-A 3D Modeling

The image generator was built as a set of scene descriptions, models, and
scripts in the Unity 3D game development platform.

For our use case, a vehicle model was translated from its native CATIAv5 CAD
format [check name] into a Unity asset with the PiXYZ plugin. Importing the
CAD object was relatively labor intensive due to a technical difficulty in
mapping part materials to Unity textures, which is an area of current work.
The work-around for our purposes was to manually assign textures to the
approximately 10,000 visible surfaces in the imported Unity asset.

###  III-B Realistic Rendering

In general, synthetic images for model training need to exemplify the
characteristics of real images that the model relies on for accurate
classification. While these qualities could be vastly different depending on
the model, for our use case we needed images that embody the broad range of
shadows and reflections seen in the production environment. Rather than
attempting to identify and optimize for the most important image features, our
approach was to create images as accurately as possible with a goal of being
indistinguishable from real images by a human observer.

Images were rendered using the Unity High Definition Rendering Pipeline. We
relied on a number of Unity features designed for high rendering accuracy, and
avoided many approximation features designed to improve rendering performance
in a game setting requiring high framerate with limited hardware resources.
Our settings for image accuracy were largely influenced by guidelines from
Unity[28]. The Unity ”camera” object was configured to mimic the properties of
the physical camera used to capture real images. A full disclosure and
justification of the rendering settings we used would be lengthy and beyond
the scope of this paper, and will be made available on publication.

We found that several external resources were very helpful in creating
realistic image rendering, especially in our use case with automotive models.
In particular, Unity’s automotive industry-focused Measured Materials
library[29] helped us simulate the paint, glass, rubber, and plastic textures
of a real vehicle. Skyboxes were sampled from the Unity HDRI pack, captured
using techniques described by Lagarde et al.[30].

###  III-C Domain Randomization

We chose a hybrid approach to domain randomization, rendering the image
subject as accurately as possible with ambient lighting similar to the
production environment. Randomized attributes included subject position
relative to the camera within plausible constraints, vehicle exterior paint
colors from a set of possible values, and a single light source (the sun) with
varying position.

To separate the subject from the background, we used a background skybox with
a very ”busy” texture, and then randomized its orientation on all 3 axis for
every scene. This served a secondary purpose in creating randomized reflection
patterns on all surfaces of the vehicle.

Randomization of objects in the scene was accomplished with a set of scripts
written in C-sharp, used natively in Unity for game logic.

###  III-D Segment Labeling

We labeled image segments by capturing multiple images from each randomized
scene – one fully rendered image, and then one false color image for each
segment. This could have been achieved in many ways, but the approach we found
to be most performant in Unity was to maintain a second ”mask” copy of the
subject model completely colored with an ”unlit” black texture, locked to the
same position as the color model. Two identical cameras in the same position
were used, one able to see the color model, background, and lighting and the
other camera only able to see the mask model.

After the normal image was captured with the color camera, the segment capture
phase would iterate through groups of components comprising each segment,
recolor the group with an unlit white texture, capture an image with the mask
camera, and then recolor the group to the unlit black texture. Figure 1 shows
the resulting image segments. This approach had the performance advantage of
minimizing the retexturing of materials on the model. This also allowed us to
capture occlusions by components not part of the segment of interest, such as
the door handles in the example images.

Figure 1: A 3D generated image (top left) in addition to a series of one-hot
encoded masks segmenting each object class.

##  IV Model Training

To validate the effectiveness of the synthetic image generator, we conducted
experiments comparing models trained with varying amounts of real labelled
images augmented with synthetic data. Our available data consisted of 14,125
labelled images of real vehicles in a production line, each of which contained
one or more examples of eight distinct feature classes. From this dataset, a
10% holdout set was randomly selected for validating models, leaving 12,712
images in the real dataset $R$ for training. The frequency of each feature’s
appearance is described in Table I, where the subset of the real image set $R$
with one or more pixels belonging to a feature class $f$ is given as
$R_{f}=\\{e|e\in R$ and $f\in e\\}$, and an example frequency of
$|R_{f}|/|R|$.

Using the synthetic image generator described in Section III, we rendered a
set of 40,406 synthetic images and labels $S$ with the same feature classes as
$R$. Due to a slightly smaller horizontal range of camera freedom, some
classes were represented more or less heavily in the synthetic set, as
detailed in Table I. However, as we weight each class equally in our metrics
and present aggregate statistics over the entire dataset, we deemed that the
example frequency weights would not affect the conclusions.

TABLE I: Feature Example Frequency in Image Sets | real images $R$ | synthetic images $S$
---|---|---
feature | examples | frequency | examples | frequency
back door | 5,994 | 47.09% | 40,231 | 99.57%
back window | 5,854 | 45.99% | 40,263 | 99.65%
rear window | 4,844 | 38.05% | 24,080 | 59.60%
front door | 6,599 | 51.84% | 22,308 | 55.21%
front window | 5,985 | 47.02% | 26,171 | 64.77%
door handle | 4,670 | 36.69% | 40,084 | 99.20%
mirror | 3,897 | 30.62% | 6,932 | 17.16%
tail light | 4,511 | 35.44% | 8,501 | 21.04%

###  IV-A Training Methodology

Images and labels were used to train U-net[31] convolutional neural network
models implemented in TensorFlow[32] 2.0.0 and Keras[33] 2.2.4-tf. Models were
trained using an NVIDIA DGX-2 with Tesla V100 GPUs running Ubuntu 18.04.4 LTS.

In this section, all U-net model structure and parameters are identical with
the exception of input datasets. The U-net implementation was derived from
code provided by Debesh et al. in their Double-U-net supplement, to be
consistent with the further work in Section V. From the original U-net
description, the only significant difference is the use of batch
normalization[34] after the convolutional layers along the contracting path,
which resulted in more consistent training and better generalization in our
use case.

A hyperparameter search using real and synthetic datasets revealed optimal
parameters that were similar enough to avoid differentiation between the
domains. As the purpose of this work is to explore the tradeoffs of synthetic
vs real data, we chose parameters that resulted in consistent and stable
training sessions rather than strictly optimizing for the highest possible
accuracy. For our datasets, a dropout probability of 0.30, a batch size of 64,
and a learning rate of 0.0020 resulted in models that converged quickly and
consistently within a reasonable limit on training time and generalized well
to the validation data.

As synthetic data can be seen as a form of data augmentation, we chose to
forego any traditional augmentation techniques (randomized cropping, gamma
shifts, etc.) to present clear results, with the single exception of randomly
flipping all training images horizontally to match the real dataset’s imaging
of both sides of the vehicle. During training, models were evaluated each
epoch against the disjoint validation set. To prevent overfitting, we used an
early stopping mechanism to halt training and revert to the best weights if no
improvement in validation set prediction loss was made over 30 epochs.

####  IV-A1 Metrics

While the image generation and training techniques share applicability with
object detection and instance segmentation models with more actionable
metrics, we quantify the performance of a standard multiclass U-net
segmentation model simply with per-pixel mean intersection-over-union (mean
IoU) with uniform class weighting and a prediction threshold of 50%.

###  IV-B Real Dataset Supplementation

[]

[]

[]

Figure 2:  Aggregated mean prediction IoU LABEL:sub@fig:real_synth_matrix_iou
of U-net models trained on random samples from real and synthetic datasets.
Models augmented with synthetic data showed up to 24.9% higher prediction
accuracy LABEL:sub@fig:real_synth_matrix_increase than the baseline,
particularly with limited amounts of real training images. The $p$-values
LABEL:sub@fig:real_synth_matrix_pvalue of one-sided T-tests,
$H_{a}:\overline{IoU}(M_{r,s})>\overline{IoU}(M_{r,0})$, show significant
accuracy increases ($p\leq 0.05$, highlighted) in most models trained with 256
or fewer real images.

To determine how supplementing a dataset of real images with synthetic images
would affect model training and accuracy, we trained instances of multiple
model classes with different mixtures of images from both sets. Subsets of the
real image set $R$ of sizes $N=\\{0,16,32,...,8192\\}$ were paired with
subsets of the synthetic image set $S$ from the same size range, forming the
axes of the 11x11 matrices shown in Figure 2 with model classes at each
intersection. For each model class, random samples from $R$ and $S$ were used
to train individual U-net segmentation models with parameters reported above.
The number of models trained in each class was sufficient that the confidence
interval ($\alpha=0.95$) width of the mean truth/prediction IoU measurements
on the real image validation set was less than 5% of the mean value, requiring
between 7 and 30 model instances for each image set size pair. We refer to the
resulting set of segmentation models as $M$, where $m_{r,s,i}\in M:r\in N,s\in
N,i\in[0..|M_{r,s}|)$ is one instance of a class of U-net models trained on
$(r,s)$ random images from datasets $R$ and $S$, and we report aggregate
statistics over the model class $M_{r,s}$ at each cell in the matrices of
Figure 2.

Figure 2 aggregates the mean IoU predictions of each trained model class on
the unseen validation set from the real image domain. We observe a general
trend of increasing accuracy with larger samples of real images, with
diminishing returns as the training images grow to sufficiently represent the
domain features. Along the horizontal axis, we see that augmentation with
synthetic data tended to increase accuracy, with greater yields in models
trained on smaller real datasets. We also observe that models trained on
purely synthetic data tend to poorly predict the real domain, even with
thousands of examples.

Figure 3: Mean IoU of model predictions on a validation set of 1176 real
images. Each IQR plot describes between 7 and 30 individual U-net models
trained on random subsets of real and synthetic images. In general, augmenting
smaller ($\leq 256$) sets of real images resulted in higher accuracy and less
variation in the trained models, with diminishing returns as the real data
became sufficiently representative of the domain.

To discuss the results of synthetic data augmentation, we first look at the
effects of augmentation on model reliability. Figure 3 shows the summary
statistics of mean validation set predictions for model classes trained on
purely real images and those augmented with 2048 synthetic images, which
details columns 0 and 2048 from Figure 2. Models trained with smaller random
samples of real images tended to show more variation in their resulting
prediction accuracy. We observe that augmentation tended to increase mean
accuracy and decrease variance in models trained with less than 256-512 real
images.

Augmenting the real training sample with varying amounts of synthetic data
yields better results, depending on how accurate the model is to begin with.
Figure 2 reshapes the data in Figure 2 as a percentage increase in mean
prediction IoU relative to that of the pure real set (column 0). We can see
that augmenting models trained with 512 or more real images only results in a
marginal increase, at best 0.6%. However, in models trained with 256 or fewer
real images, the accuracy increase is substantial, up to 25.0% when only 16
real images are available. We can also see that the addition of any amount of
real images results in models that are more accurate than those trained on
synthetic data alone. This is supported by the $p$-values of one-sided
T-tests,
$H_{a}:\overline{IoU}(M_{r,s})>\overline{IoU}(M_{r,0})~{}\forall~{}r,s\in N$,
shown in Figure 2 with $p<0.05$ highlighted.

Figure 4: Mean prediction IoU of U-net models on real images, viewed by the
ratio of real to synthetic data in the training datasets. Each trend exhibits
an inflection point where accuracy decreased, presumably due to limited
capacity of the model to encompass both the real and synthetic domain.

Figure 2 also shows that in some cases, particularly in those with 512 or more
real images, the addition of large amounts of synthetic data correlate with a
slight decrease in prediction accuracy, presumably due to dilution of the
samples from the real domain and a limited capacity of the model to encompass
both the real and synthetic domains. We can observe this trend more clearly
when viewing the relationship between real and synthetic image set sizes as a
ratio, shown in Figure 4. Each real image set size exhibits an inflection
point where accuracy declines, which we suspect is dependent on the capacity
of the model and similarity between real and synthetic data in a particular
use case.

To visualize the differences in prediction accuracy, Figure 5 presents the
segmentation maps predicted by 10 different models, trained on 16-256 real
images and augmented with either 0 or 2048 synthetic images. In contrast to
the randomly selected images used to train the models Figure 2, each real
dataset larger than 16 images is a superset of the smaller datasets, and the
same real datasets and 2048-image synthetic dataset are reused in each of the
augmented models. For this example image, the quality of the predictions are
fairly low in the pure real models, limiting usefulness depending on the use
case. The addition of synthetic images results in clearly defined door/window
boundaries with even the smallest real training set, and better identification
of smaller features such as the door handles at 64 real images compared to
requiring 128 without augmentation.

Figure 5: Segmentation map predictions of U-net models trained with pure real
images (top) vs. the same training sets augmented with 2048 synthetic images
(bottom). The input image and ground truth are shown on the left for
reference.

##  V Transfer Learning

Another potential use case for synthetic data is in pretraining models for
later improvement with real data, either as a base for multiple specialized
models or as a starting point for incremental training as real data becomes
available. Our results from the previous section indicate that U-net models
trained with 256 or fewer images from our real image dataset suffer from low
applicability to new images, so in this section we will focus on pretrained
model refinement with small numbers of real images.

The goals and requirements for transfer learning can vary widely, but in our
exploration we will focus on use cases stemming from unavailability of real
labelled training images and from the need to specialize a general model for a
particular task. As such, we will quantify results in terms of accuracy (in
this case, mean prediction IoU on real data) and training time of the model
specialization training.

###  V-A U-Net

There are many strategies for transfer learning using the U-Net model [cite],
most involving freezing, reinitializing, adding, or removing layers. It is
beyond the scope of this work to explore the many factors involved in choosing
the optimal strategy for a particular use case. We will instead focus on a
relatively simple technique that compares well to our work with a more
advanced model in the next subsection, which to train a U-Net with purely
synthetic data, and then continuing training with real images while optionally
freezing or replacing part of the model. Our base synthetic-trained U-Net
model uses parameters as described in the previous section, trained with a
larger dataset of 36,480 synthetic images, which achieved $0.954$ mean
prediction IoU on the holdout set from the same synthetic domain. Accuracy on
segmentation of real images was similar to the experiments with large pure
synthetic datasets in the previous section, only achieving a mean prediction
IoU of $0.618$ on that domain.

Starting with an identical U-Net base model initialized with random weights,
experiments were configured as follows:

  * •

synth-random \- only the contracting path (encoder) was initialized with
weights from the pretrained base, allowing the untrained expanding path
(decoder) to train completely on real data;

  * •

synth-synth \- both the encoder and decoder were initialized with pretrained
base weights;

  * •

VGG19-random \- the encoder part of the model was replaced with VGG19,
detailed below, and the decoder left with random weights;

  * •

VGG19-synth \- the encoder was replaced with VGG19, and the decoder
initialized with pretrained base weights;

  * •

control \- the base model was used without freezing or replacing layers, and
the initial random weights were unchanged. Note that this is the same
configuration as models in the previous section, and the resulting model is
trained on purely real data.

Finally, we doubled the above configurations with another parameter, choosing
to either freeze the layers of the encoder portion of the model or allow the
secondary training with real data to propagate and update the encoder weights.
Our expectations were that freezing the encoder section of the model would
reduce training time as there were less parameters to update with each back-
propagation, but could reduce the model’s ability to adapt to the new data.
Table II details the number of trainable parameters and mean training time per
image-epoch for the four resulting model architectures, which indeed shows
decreased time per image with less parameters to update.

For some experiments, the encoder layers of the model were replaced with a
VGG19[25] model pretrained with weights from ImageNet[1], following the same
procedure as the work done in [8] for comparability. With the models
initialized with pretrained weights, we continued training using randomly
selected subsets of real images until convergence, using the stopping criteria
described in the previous section. All model variant and real image sample
size permutations were repeated 30 times.

TABLE II: Model Size and Training Time |  | parameters (millions) | training time per
---|---|---|---
model | variant | total | trainable | epoch-image (s)
U-net |  | 7.77 | 7.77 | 0.0130
U-net | frozen encoder | 7.77 | 3.05 | 0.0120
U-net | VGG19 encoder | 23.86 | 23.86 | 0.0176
U-net | frozen VGG19 encoder | 23.86 | 3.83 | 0.0153
W-net | frozen 1st U-net | 10.11 | 2.34 | 0.0146
W-net | frozen VGG19 encoder | 26.59 | 6.56 | 0.0195

[]
[]

Figure 6:  Comparisons of mean prediction IoU
LABEL:sub@fig:transfer_unet_freeze_iou and training time
LABEL:sub@fig:transfer_unet_freeze_time of secondary training of pretrained
U-Net models, with the weights of the contracting path (encoder) either
trainable or frozen.

We first compare on the frozen/trainable encoder variable, visualized in
Figure 6. In models using VGG19 as the encoder, we observed greater prediction
accuracy and lower training time, while models using our encoder pretrained on
synthetic data tended to perform better when the encoder was not frozen during
secondary training. This is perhaps due to the large difference in the number
of encoder neurons, as propagating the training feedback from each example
through the larger VGG19 encoder is more costly and less impactful. We
speculate that limiting the neurons being updated each epoch lead to faster
model convergence while the models with more trainable weights slowed in
training progress enough to trigger early stopping. The training logs support
this conjecture, showing extremely slow improvement before training was
terminated. It is possible that, given enough time, the accuracy differences
between trainable and frozen versions of the same model would minimize.
However, since all models use the same early stopping criteria, we present the
results as comparable in a practical sense. In the remainder of this work,
comparisons with these models will use the better-performing frozen encoders
in the case of VGG19, and trainable encoders for the synthetic data-trained
models.

[]
[]

Figure 7:  Limiting to encoder type and decoder initial weights (synthetic
pretrained vs. random) model permutations, we observed a sizeable tradeoff
between mean prediction IoU LABEL:sub@fig:transfer_unet_iou and training time
LABEL:sub@fig:transfer_unet_time when compared to models initialized from
randomness.

Next, we compare the mean prediction accuracy of the retrained models with
frozen encoders to the control models trained from randomly initialized
weights. We observed that in cases with 64 or fewer real images, we saw an
increase in accuracy over a control model trained on purely real data.
However, in larger real image classes and with all control models trained on a
mix of real and synthetic data, we saw significantly lower accuracy in the
specialized models. We again speculate that the model training may have slowed
enough to trigger our early termination criteria, and that a combination of
refined learning rate, early termination parameters, and lengthened training
time may result in improved accuracy. Our goals in this work are in
comparability between experiments, though, so we present these results as a
baseline to be improved upon.

In comparing the prediction accuracy of U-net models with different decoder
weights, we saw mixed results; the pretrained synthetic data weights appeared
to result in lower performance in models with synthetic weighted encoders
trained on 16 or 32 real images, while having the opposite effect in models
with VGG19 encoders. In models trained on 64 or more real images, the results
were less clear; and a two-sided T-test showed insufficient difference to
conclude that the results are drawn from different distributions at $p=0.05$.

Comparing encoder paths of the different model classes was more consistent, in
that the U-net default layers trained with synthetic data resulted in higher
mean prediction accuracy than models using the VGG19 encoder trained on
ImageNet, across all real data sample sizes. We conclude from these findings
that a relatively small encoder (4.72m parameters) trained on a few thousand
images drawn from a similar synthetic domain to the target can outperform the
already impressive feature extraction of a large (23.03m parameters) encoder
trained on over a million generic real images.

Figure 7 compares the training times of retrained models to those of the
control model for each real sample size class, with results between 10.0% and
20.8% of the time required for the control. The time can be accounted for in
both the number of trainable parameters in the retrained models with frozen
encoders, and the number of epochs required to converge. As the mean training
time for a purely synthetic U-net (r=0, s=2048) is 11,648 seconds, the
training time for a retrained U-net is comparable to that of the control.

###  V-B Double-U-Net

Since the introduction of U-net in 2015, a number of derivative models have
been proposed that improve its applicability to certain use cases. One of
these, the Double-U-net[8], improves upon the localization of segment
instances by dividing the task between, as the name suggests, two U-net models
linked together. The first U-net, using a VGG19 encoder trained on ImageNet,
outputs feature maps from each level of the encoding process as well as an
intermediate segmentation map from the decoder. The segmentation map is paired
with the original image as input to the second U-net, while feature map
outputs of the first U-net are linked to corresponding layers of the second
U-net decoder. The authors’ results showed impressive accuracy gains over a
standard U-net on a variety of medical segmentation datasets.

As an exercise in applying transfer learning to a more complex model, we chose
the Double-U-net (abbreviated W-net for the remainder of this work) because of
its intuitive design as a logical extension to the standard U-net, as well as
having experience and success using the model in some production use cases.
Our experiments in this section will expand on the previous section for ease
of comparison, with the caveat that we made some implementation choices toward
this goal while potentially sacrificing some peak performance. For example,
the authors of W-net used squeeze-excite blocks[35] at the end of each
convolutional block, which is not part of the original U-net specification.
Additionally, in our image set, vehicle features were largely scale-invariant,
as the images were captured from a fixed viewpoint with a low variation in the
vehicle’s distance from the camera. This warranted omission of the Atrous
Spatial Pyramid Pooling (ASPP) block between the encoder and decoder in each
U-net, which was used in [8] to handle feature scaling. We conducted a limited
exploration and found these features to contribute little to no performance
gains on our particular use case, so we believe that the simplified model is a
better comparison to transfer learning results on a simple U-net in the
previous section.

Our W-net implementation is simply two U-net models, identical to the
implementation described in the previous section, with the following two
additions. First, as in the [8], the U-nets are connected with a pixel-wise
multiplication layer, such that the second U-net receives the original image
augmented with the segmentation map output of the first U-net. Second, the
encoder layer-wise feature maps from the first U-net are concatenated to the
inputs of the second U-net decoder, in the same manner as the feature maps
from the second U-net encoder.

Following the work in the previous section and as an analog to [8], we chose
to construct Double-U-nets with two model variations. In the first model, we
use a U-net trained on synthetic data as described above, with the entire
first U-net frozen. The second model, analogous to [8], uses a frozen VGG19
encoder and a trainable uninitialized decoder. In both models, the second
U-net is initialized with random weights and is fully trainable. Our
hyperparameter search revealed optimal parameters very close to those used to
train the individual U-nets, so we opted to keep the original parameters for
comparability.

[]
[]

Figure 8:  Results of transfer learning on U-net and W-net models with (first)
encoders trained on synthetic data or VGG19/ImageNet, compared to training of
control models initialized with random weights. The mean prediction IoU
LABEL:sub@fig:transfer_wnet_iou and training time
LABEL:sub@fig:transfer_wnet_time suggest improved accuracy of synthetic-
trained encoders, but in some cases with a time cost.

Our results, shown in Figure 8, show accuracy improvements using the W-net
model with the VGG19 encoder over all training image size classes, and similar
or better results with the synthetic-trained first U-net. The accuracy
improvements correlate with a training cost increase, however, especially with
the VGG19-based models with more layers to train. The conclusion we draw from
these results is that secondary training with a multipart model like W-net can
be a viable accuracy enhancement if the time cost can be justified.

##  VI Conclusions

We found that, for this image segmentation problem, synthetic images were an
effective technique for augmenting limited sets of real training data. We
observed that models trained on purely synthetic images had a very low mean
prediction IoU on real validation images. We also observed that adding even
very small amounts of real images to a synthetic dataset greatly improved
accuracy, and that models trained on datasets augmented with synthetic images
were more accurate than those trained on real images alone. We noted that for
this domain, 256 to 512 images seemed to be enough to train a reasonably
accurate model, with rapidly diminishing returns on adding synthetic images to
the mix, eventually resulting in lower accuracy as the real:synthetic ratio
dropped.

In use cases that benefit from incremental training or model specialization,
we found that pretraining on synthetic images provided a usable base model for
transfer learning. While we observed that models trained in a single session
outperformed those pretrained on synthetic images and retrained on real data,
we also saw that up to 90% of the total training time could be completed in
the pretraining phase.

We conclude that synthetic image generation can be beneficial to segmentation
model training when insufficient images are available to train a satisfactory
model. However, testing must be done to find the break point where adding more
synthetic images does not result in higher mean accuracy.

##  VII Future Work

A natural progression from this work is to study the characteristics of
synthetic data and identify features that contribute to model accuracy and can
be adapted to more closely resemble the real domain, while separating less
important features that should be randomized. Recent work in the field of
Generative Adversarial Networks (GANs) could be used to automate the feature
identification process and help design more robust synthetic image rendering
processes. Another interesting topic would be exploring how synthetic images
can be used in conjunction with other effective data augmentation techniques,
which unfortunately was beyond the scope of this work.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255, 00000.
  * [2] S. R. Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in _European conference on computer vision_. Springer, 2016, pp. 102–118, 00781\.
  * [3] H. Su, C. R. Qi, Y. Li, and L. J. Guibas, “Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2686–2694, 00567.
  * [4] S. Hinterstoisser, O. Pauly, H. Heibel, M. Martina, and M. Bokeloh, “An annotation saved is an annotation earned: using fully synthetic training for object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , Oct. 2019, 00004.
  * [5] X. Peng, B. Sun, K. Ali, and K. Saenko, “Learning deep object detectors from 3d models,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1278–1286, 00274.
  * [6] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1301–1310, 00164\.
  * [7] B. Sun and K. Saenko, “From Virtual to Reality: Fast Adaptation of Virtual Object Detectors to Real Domains.” in _BMVC_ , vol. 1, 2014, p. 3, 00150 Issue: 2.
  * [8] D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen, “DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation,” _arXiv preprint arXiv:2006.04868_ , 2020, zSCC: 0000003.
  * [9] Y.-C. Jhang, A. Palmar, B. Li, S. Dhakad, S. K. Vishwakarma, J. Hogins, A. Crespi, C. Kerr, S. Chockalingam, C. Romero, A. Thaman, and S. Ganguly, “Training a performant object detection ML model on synthetic data using Unity Perception tools,” Sep. 2020, 00000. [Online]. Available: https://blogs.unity3d.com/2020/09/17/training-a-performant-object-detection-ml-model-on-synthetic-data-using-unity-perception-tools/
  * [10] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016, 23133 Publisher: IEEE.
  * [11] M. Yan, I. Frosio, S. Tyree, and J. Kautz, “Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control,” _arXiv preprint arXiv:1712.03303_ , 2017.
  * [12] R. Nevatia and T. O. Binford, “Description and recognition of curved objects,” _Artificial intelligence_ , vol. 8, no. 1, pp. 77–98, 1977, 00606 Publisher: Elsevier.
  * [13] D. G. Lowe, “Three-dimensional object recognition from single two-dimensional images,” _Artificial intelligence_ , vol. 31, no. 3, pp. 355–395, 1987, 01904\.
  * [14] T. Hodaň, V. Vineet, R. Gal, E. Shalev, J. Hanzelka, T. Connell, P. Urbina, S. N. Sinha, and B. Guenter, “Photorealistic image synthesis for object instance detection,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 66–70, 00000.
  * [15] Y. Zhang, S. Song, E. Yumer, M. Savva, J.-Y. Lee, H. Jin, and T. Funkhouser, “Physically-based rendering for indoor scene understanding using convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5287–5295, 00000.
  * [16] Z. Li and N. Snavely, “Cgintrinsics: Better intrinsic image decomposition through physically-based rendering,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 371–387, 00000.
  * [17] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 23–30, 00851.
  * [18] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” _arXiv preprint arXiv:1809.10790_ , 2018, 00168.
  * [19] J. Borrego, A. Dehban, R. Figueiredo, P. Moreno, A. Bernardino, and J. Santos-Victor, “Applying domain randomization to synthetic data for object category detection,” _arXiv preprint arXiv:1807.09834_ , 2018, 00013\.
  * [20] C. Mitash, K. E. Bekris, and A. Boularias, “A self-supervised learning system for object detection using physics simulation and multi-view pose estimation,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 545–551, 00063.
  * [21] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 7249–7255, 00036.
  * [22] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 969–977, 00256.
  * [23] V. Iglovikov and A. Shvets, “Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation,” _arXiv preprint arXiv:1801.05746_ , 2018\.
  * [24] M. Frid-Adar, A. Ben-Cohen, R. Amer, and H. Greenspan, “Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pre-trained encoder,” in _Image Analysis for Moving Organ, Breast, and Thoracic Images_. Springer, 2018, pp. 159–168.
  * [25] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014, 00000\.
  * [26] T. Inoue, S. Choudhury, G. De Magistris, and S. Dasgupta, “Transfer learning from synthetic to real images using variational autoencoders for precise position detection,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 2725–2729, 00018.
  * [27] F. Zhang, J. Leitner, M. Milford, and P. Corke, “Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks,” _world_ , vol. 7, no. 8, 2017.
  * [28] P. Y. Donzallaz, “How to set up unity’s high definition render pipeline for high-end visualizations,” Jan 2020. [Online]. Available: https://blogs.unity3d.com/2020/01/09/how-to-set-up-unitys-unitys-high-definition-render-pipeline-for-high-end-visualizations/
  * [29] E. Martin and L. Vo Van, “We have you covered with the measured materials library,” Feb 2019. [Online]. Available: https://blogs.unity3d.com/2019/02/08/we-have-you-covered-with-the-measured-materials-library/
  * [30] S. Lagarde, S. Lachambre, and C. Jover, “An artist-friendly workflow for panoramic hdri,” in _ACM SIGGRAPH 2016 Courses_ , ser. SIGGRAPH ’16. New York, NY, USA: Association for Computing Machinery, 2016. [Online]. Available: https://doi.org/10.1145/2897826.2927353
  * [31] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241, 00000.
  * [32] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [33] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [34] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [35] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.