# Critical Field Strength in an Electroclinic Liquid Crystal Elastomer

Christopher M. Spillmann<EMAIL_ADDRESS>Center for
Bio/Molecular Science and Engineering, Code 6900, Naval Research Laboratory,
4555 Overlook Ave. SW, Washington, D.C., 22375 USA  Amit V. Kapur  Center for
Bio/Molecular Science and Engineering, Code 6900, Naval Research Laboratory,
4555 Overlook Ave. SW, Washington, D.C., 22375 USA  Frank W. Bentrem  Center
for Bio/Molecular Science and Engineering, Code 6900, Naval Research
Laboratory, 4555 Overlook Ave. SW, Washington, D.C., 22375 USA  Marine
Geosciences Division, Naval Research Laboratory, Stennis Space Center,
Mississippi, 39529 USA  Jawad Naciri  Center for Bio/Molecular Science and
Engineering, Code 6900, Naval Research Laboratory, 4555 Overlook Ave. SW,
Washington, D.C., 22375 USA  Banahalli R. Ratna  Center for Bio/Molecular
Science and Engineering, Code 6900, Naval Research Laboratory, 4555 Overlook
Ave. SW, Washington, D.C., 22375 USA

###### Abstract

We elucidate the polymer dynamics of a liquid crystal elastomer based on the
time-dependent response of the pendent liquid crystal mesogens. The molecular
tilt and switching time of mesogens are analyzed as a function of temperature
and cross-linking density upon application of an electric field. We observe an
unexpected maximum in the switching time of the liquid crystal mesogens at
intermediate field strength. Analysis of the molecular tilt over multiple time
regimes correlates the maximum response time with a transition to entangled
polymer dynamics at a critical field strength.

###### pacs:

61.30.-v, 61.41.+e, 73.61.Ph

††preprint: preprint

In the 1970s, Garoff and Meyer were the first to describe a distinctive class
of liquid crystal (LC) that exhibits a chiral smectic-$A$ phase (Sm-$A^{*}$)
consisting of chiral molecules with a permanent dipole close to the chiral
center Garoff and Meyer (1977). Application of an electric field couples the
molecular dipole to the field and results in a molecular tilt ($\theta$) in a
plane orthogonal to the transverse component of the dipole. This response was
termed the electroclinic effect or soft mode, analogous to the softening of
the vibration mode in a ferroelectric material as it is cooled to the Curie
temperature Garoff and Meyer (1977), and has been well documented with several
LC moieties Abdulhalim and Moddel (1991); Andersson et al. (1988); Garoff and
Meyer (1979); Nishiyama et al. (1987).

The reorientation of the molecular dipoles and the concomitant tilt in
response to an applied electric field is resisted by the intrinsic viscosity
of the material. If the LC molecules are now tethered to a polymer backbone
(Fig. 1), the electroclinic response to an electric field is significantly
constrained. The molecules maintain much of the rotational freedom necessary
for the permanent electric dipoles to align with the applied field, but the
tilt response is significantly retarded by the entangled polymer backbone. The
significant influence of the polymer network was first noted by Gebhard and
Zentel when they studied an elastomeric ferroelectric LC system Gebhard and
Zentel (1999, 2000). They noted that the polymer network “counterforce” could
be represented as a spring resisting the molecular tilt in the Sm-$A^{*}$
phase Gebhard and Zentel (1999). However, the dynamic nature of the interplay
between the LC mesogens and the underlying polymer network remains largely
unexplored.

Figure 1:  Schematic representation of smectic LC mesogens in an elastomeric
network.

In this Letter, we investigate the polymer dynamics of an electroclinic LC
elastomer as a function of cross-linking density, temperature, and field
strength. Since the LC molecules are coupled to the polymer network, the
molecular tilt response is used as an indirect probe to elucidate the polymer
dynamics in the presence of an applied field. We observe multiple time domains
in the tilt response and also a critical field strength corresponding to the
maximum switching time of the tethered LC mesogens. At this critical field
strength, the response time shows power-law dependence and supports the
presence of entanglement dynamics in the elastomer system at higher field
strengths. Thus, we demonstrate the ability to elucidate the polymer dynamics
of the system based on the reorientation of the pendent LC elements.

Based on Landau theory Clark and Lagerwall (1991), the electroclinic response
of a monomeric system to an applied field, $E$, at temperatures sufficiently
above the Sm-$A^{*}-$Sm-$C^{*}$ (chiral smectic-$C$) phase transition results
in a linear response of $\theta$ to $E$ and a characteristic switching time
$\tau$ independent of the applied field. As the system is cooled and
approaches the Sm-$A^{*}-$Sm-$C^{*}$ transition, both $\theta$ and $\tau$
increase at a given field strength. The switching time also shows a dependence
on $E$ such that it monotonically decreases with increasing field strength. It
has been previously demonstrated that the presence of a Sm-$A^{*}-$Sm-$C^{*}$
transition is not a necessary condition for the electroclinic effect Crawford
et al. (1995); Shashidhar et al. (2000). This holds true for the $\theta$ and
$\tau$ response of the monomeric mixture used in the current study (see
supplementary material, sup ).

Engineered coupling of electroclinic LCs to a polymer network allows for an
applied field to change the orientation of the pendent LC molecules and
subsequently alter the elastomer system in a reversible manner Hiraoka et al.
(2009); Kohler et al. (2005); Spillmann et al. (2007); Stannarius et al.
(2002). We have developed a free-standing electroclinic elastomer and examined
the molecular tilt, macroscopic actuation, and molecular packing of the system
Spillmann et al. (2007, 2008). We now examine the time-dependent molecular
response and for the first time use this information to shed light on the
polymer dynamics.

Synthesis of the polymerizable LC components, the diacrylate cross-linker, and
preparation of the elastomer has been previously reported Spillmann et al.
(2007); Artal et al. (2001). The structure of the molecular components and
details of the sample preparation are provided as Supplementary Material.
Three samples were prepared with 0, 2.5, and 5 mole percent (mol%) of the
cross-linker in EHC cells (E.H.C. Co., Ltd., Tokyo, Japan) to provide known
sample thickness and transparent indium tin oxide (ITO) electrodes for
electric field application. Electro-optic tilt angle and response time
measurements of the electroclinic samples follow the example of Lee and Patel
Lee and Patel (1989a, b) and are captured under temperature-controlled
conditions with a polarized light microscope. This approach is sensitive to
changes in the orientation of the rigid core (phenyl rings) of the
electroclinic molecules with respect to the crossed polarizers. The tilt angle
$\theta$ of the LC mesogens measured upon application of an electric field is
given by

$\theta=\frac{1}{4}\sin^{-1}\left(\frac{\Delta
I}{I_{\text{max}}-I_{\text{min}}}\right),$ (1)

where $I_{\text{min}}$ and $I_{\text{max}}$ are the minimum and maximum
transmitted light intensity with the LC molecular director positioned 0 and
$\pi/4$ radians with respect to the polarizer, respectively. With the sample
positioned $\pi/8$ radians with respect to the polarizer, a bipolar square
wave ($+V$ to $-V$) was applied at incrementing field strengths and the change
in transmitted light intensity through the sample, $\Delta I$, was measured.
The response time of the molecular tilt is defined as the time required for
the molecular tilt to change from 10% to 90% of the full tilt angle upon
reversal of the electric-field polarity.

The tilt angle of the electroclinic polymer and elastomer samples showed
characteristic curves that decreases with increasing temperature
(Supplementary Material, Fig. S4). However, the response time showed an
unexpected initial increase with increasing field strength, as shown in Fig.
2. The response time of elastomer samples reached a maximum at intermediate
field strengths of 15–25 V/$\mu$m and then began to decrease as $E$ was
further increased. The average field strength at which the maximum $\tau$ was
observed in the elastomer did not change significantly as a function of
temperature. In addition, no significant difference in $\tau$ or the field
strength at the maximum response time was observed when the cross-linking was
either 2.5 or 5 mol%. The polymer follows a notably different trend. At
$35^{\circ}$C, the peak response time was observed at 8 V/$\mu$m [Fig. 2(a),
black circles] and increasing the temperature to $40^{\circ}$C altered the
response to closely resemble the shape expected for a monomeric sample [Fig.
2(b), black circles]. At lower temperatures approaching the glass transition
of $\sim$28∘C, the effective polymer entanglement is large, increasing the
resistance to molecular tilting with in a pronounced increase in $\tau$. These
molecular observations provide critical insight of the elastomer response of
an electroclinic system to applied fields and expose the presence of two
opposing forces in the material: the force generated by the reorientation of
the LC mesogens in response to an applied field and the resistive force of the
polymer network.

Figure 2: Response time of electroclinic polymer (black circles) and elastomer
with 2.5 (red circles) or 5 mol% (green circles) cross-linking at (a)
$35^{\circ}$C and (b) $40^{\circ}$C. Figure 3:  (a) Semi-log plot of $\theta$
versus time of elastomer with 5 mol% cross-linking at $35^{\circ}$C. Domains
of the response are identified as regions I, II, and III. Field strengths are
4, 8, 12, 16, 20, 26, and 40 V$/\mu$m. (b) $\beta$ plotted as a function of
field strength. Guide lines are overlaid for $10\leq E\leq 20$ and $20\leq
E\leq 40$.

The origin of the light intensity changes used to monitor $\theta$ and $\tau$
is the reorientation of the LC mesogens in response to an electric field.
Since these molecules are coupled to the polymer network, we explored the
possibility of using this information to probe the polymer dynamics of the
system by examining the time dependent response over a range of field
strengths. The use of the time-dependent tilt angle as an analysis tool for
the polymer dynamics of the system is analogous to a previously reported
method that utilized light scattering of nanocolloidal probes adsorbed to a
polymer matrix Sprakel et al. (2007). In our system, it is the coupling of the
LC mesogens to the network that serves as a sensitive measurement of the
polymer dynamics. The dynamics of a polymer system is defined by the
relationship of the mean square displacement of individual segments of a
polymer chain, $\langle r(t)^{2}\rangle$, to time $t$, as

$\langle r(t)^{2}\rangle=at^{\beta},$ (2)

where $a$ and $\beta$ are the creep coefficient and exponent, respectively de
Gennes (1979); McLeish (2002). Since the mesogens are coupled to the polymer
backbone and $\theta$ is a measure of the mesogen displacement, i.e., tilt, in
response to an applied field, we assume $\theta(t)$ is an indicator, and,
therefore, approximately equivalent to $r(t)$ in the current system. This
indirect measure of the polymer dynamics is most relevant at higher field
strengths, where the coupling of the polymer backbone to the mesogenic
response is realized.

Logarithmic analysis of the relationship between $\theta$ and time is used to
examine the polymer dynamics of the electroclinic elastomer as a function of
$E$. Figure 3(a) shows a semilog plot of $\theta$ developing as a function of
time at several electric field strengths upon switching the field polarity for
an elastomer sample. The tilt angle advances through three time regimes
(regions I, II, and III) and is described in terms of the interplay between
the polymer backbone and pendent LC mesogens. Region I is the initial
uncoupled response where the LC mesogens rapidly tilt with little resistance
from the polymer backbone. Region II is where the tilt response of the coupled
mesogens becomes significantly influenced by the underlying polymer network
prior to saturation of the tilt angle in region III.

The molecular tilt in region I follows a logarithmic relationship during the
initial response to the e-field. Region II was defined as the intermediate
regime between the initial logarithmic response and the saturated tilt angle
estimated from linear fits to region I and III in Fig. 3(a). A log−log
analysis of region II shows evidence of power-law dependence. Linear
regression [Fig. 3(a), red lines] provides detail about the creep exponent
$\beta$, which is expected to depend on field strength, temperature, and
cross-linking. The values of $\beta$ as a function of electric field strength
are shown in Fig. 3(b). We note that this analysis excluded the lowest field
strengths ($<10$ V/$\mu$m), where the coupling between the mesogenic tilt and
polymer backbone accommodates small tilt angles without a significant
influence from the polymer backbone. At field strengths ranging from
$\sim$10–20 V/$\mu$m, there is a linear dependence on the applied field. At
field strengths greater than 20 V/$\mu$m, $\beta$ saturates to a constant
value of $\sim$0.22. Various models of polymer systems predict power-law
displacement, including Rouse, reptation, and entanglement dynamics de Gennes
(1979); McLeish (2002). The saturating value of $\beta$ strongly suggests a
$t^{1/4}$ dependence, which indicates the polymer backbone may be constrained
by the dynamics of polymer entanglement in this time regime de Gennes (1979);
McLeish (2002). It is important to note that previous comparative study on
entangled and covalently cross-linked polymers had shown that the dynamics are
indentical for the two cases Nicolai et al. (1999). The transition to
entanglement dynamics at $\sim$20 V/$\mu$m occurs at the same field strength
that, on average, has the longest response time in the elastomer samples (see
Fig. 2). Therefore, the polymer dynamic analysis supports the notion of a
critical field strength at which the underlying polymer network begins to
significantly reorient in response to the external stimulus.

The presence of a maximum $\tau$ at an intermediate $E$ quantifies the ability
of the electrically-induced mesogenic tilt to affect the reorientation of the
polymer backbone at various field strengths. At the lowest applied fields, the
mesogenic tilt is only a few degrees and the stress on the polymer backbone is
negligible and $\tau$ relatively fast. At slightly higher field strengths, the
molecular tilt begins to impose a significant tug on the polymer backbone. The
restraint or creep of the polymer backbone is observed as an increase in
$\tau$. The force applied by the LC reorientation has increased sufficiently
to induce slow realignment of the polymer backbone from its equilibrium state.
This continues as the field strength and molecular tilt increase to the point
where $\tau$ reaches a maximum and the force of the mesogenic tilt and the
polymer resistive force appear to reach a balance. At the highest field
strengths, the force produced by the LC mesogens overcomes the resistance of
the polymer backbone and the maximum tilt angle is reached at progressively
faster response times. It should be pointed out that when a cross-linking
agent is present in the system, there is additional restoring force introduced
above and beyond the presence of the polymer backbone that accounts for the
persistent critical field strength observed in the elastomer samples. A
schematic representation of the polymer realignment and mechanical shear
induced by the LC mesogenic tilt at increasing field strengths is provided in
the Supplementary Information, Fig. S5.

The molecular tilt angle and response time of an electroclinic liquid crystal
have been examined as a monomeric system, a polymer, and a cross-linked
network. Following polymerization, the magnitude of the molecular tilt angle
remains the same with significant increases in both the response time and the
field strength required to elicit a response. The presence of a critical
intermediary field strength is identified where the characteristic response
time reaches a maximum. The mesogenic response as a function of applied field
is related to the interplay between the force arising from the molecular tilt
of the liquid crystal mesogens and the resistive force of the entangled
polymer backbone. Analysis of the time-dependent response over the range of
field strengths provides a novel approach to examine the polymer dynamics of
the system and suggests an entanglement regime above the critical field
strength.

This work was supported by the U. S. Office of Naval Research. F.W.B. also
received support from the Naval Research Laboratory Advanced Graduate Research
Program.

## References

  * Garoff and Meyer (1977) S. Garoff and R. B. Meyer, Phys. Rev. Lett. 38, 848 (1977).
  * Abdulhalim and Moddel (1991) I. Abdulhalim and G. Moddel, Liquid Crystals 9, 493 (1991).
  * Andersson et al. (1988) G. Andersson, I. Dahl, W. Kuczynski, S. T. Lagerwall, K. Skarp, and B. Stebler, Ferroelectrics 84, 285 (1988).
  * Garoff and Meyer (1979) S. Garoff and R. B. Meyer, Phys. Rev. A 19, 338 (1979).
  * Nishiyama et al. (1987) S. Nishiyama, Y. Ouchi, H. Takezoe, and A. Fukuda, Jpn. J. Appl. Phys. 26, L1787 (1987).
  * Gebhard and Zentel (2000) E. Gebhard and R. Zentel, Macromolecular Chem. Phys. 201, 902 (2000).
  * Gebhard and Zentel (1999) E. Gebhard and R. Zentel, Liquid Crystals 26, 299 (1999).
  * Clark and Lagerwall (1991) N. A. Clark and S. T. Lagerwall, _Ferroelectric Liquid Crystals: Principles, Properties, and Applications_ (Gordon and Breach Science Publishers, Philadelphia, 1991), vol. 7, chap. 1.
  * Crawford et al. (1995) G. P. Crawford, J. Naciri, R. Shashidhar, P. Keller, and B. R. Ratna, Mol. Cryst. Liq. Cryst. 263, 223 (1995).
  * Shashidhar et al. (2000) R. Shashidhar, J. Naciri, and B. R. Ratna, Adv. Chem. Phys. 113, 51 (2000), URL http://dx.doi.org/10.1002/9780470141724.ch2.
  * (11) See supplementary material at http://link.aps.org/supplemental/10.1103/PhysRevLett.000.000000 for molecular structures, LC textures and alignment, monomeric response to an applied field, temperature-dependent response of the elastomer, and a schematic of the polymer realignment as an external field is applied.
  * Hiraoka et al. (2009) K. Hiraoka, M. Kobayasi, R. Kazama, and H. Finkelmann, Macromolecules 42, 5600 (2009).
  * Kohler et al. (2005) R. Kohler, R. Stannarius, C. Tolksdorf, and R. Zentel, Appl. Phys. A: Mater. Sci. Proc. 80, 381 (2005).
  * Stannarius et al. (2002) R. Stannarius, R. Köhler, U. Dietrich, M. Lösche, C. Tolksdorf, and R. Zentel, Phys. Rev. E 65, 41707 (2002).
  * Spillmann et al. (2007) C. M. Spillmann, B. R. Ratna, and J. Naciri, Appl. Phys. Lett. 90, 021911 (2007).
  * Spillmann et al. (2008) C. M. Spillmann, J. H. Konnert, J. R. Deschamps, J. Naciri, and B. R. Ratna, Chem. Mater. 20, 6130 (2008).
  * Artal et al. (2001) C. Artal, M. B. Ros, J. L. Serrano, N. Pereda, J. Etxebarria, C. L. Folcia, and J. Ortega, Macromolecules 34, 4244 (2001).
  * Lee and Patel (1989a) S. D. Lee and J. S. Patel, Appl. Phys. Lett. 54, 1653 (1989a).
  * Lee and Patel (1989b) S. D. Lee and J. S. Patel, Appl. Phys. Lett. 55, 122 (1989b).
  * Sprakel et al. (2007) J. Sprakel, N. A. M. Besseling, F. A. M. Leermakers, and M. A. C. Stuart, Phys. Rev. Lett. 99, 104504 (pages 4) (2007), URL http://link.aps.org/abstract/PRL/v99/e104504.
  * de Gennes (1979) P.-G. de Gennes, _Scaling Concepts in Polymer Physics_ (Cornell University Press, Ithica, 1979), 1st ed., ISBN 080141203X, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20%&path=ASIN/080141203X.
  * McLeish (2002) T. McLeish, Adv. Phys. 51, 1379 (2002).
  * Nicolai et al. (1999) T. Nicolai, F. Prochazka, and D. Durand, Phys. Rev. Lett. 82, 863 (1999).


# 4DSR-GCN: 4D Video Point Cloud Upsampling using
Graph Convolutional Networks

Lorenzo Berlincioni<EMAIL_ADDRESS>MICC, Università degli Studi
di FirenzeItaly ,  Stefano Berretti<EMAIL_ADDRESS>MICC,
Università degli Studi di FirenzeItaly ,  Marco Bertini
<EMAIL_ADDRESS>MICC, Università degli Studi di FirenzeItaly and
Alberto Del Bimbo<EMAIL_ADDRESS>MICC, Università degli Studi di
FirenzeItaly

(2023; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Time varying sequences of 3D point clouds, or 4D point clouds, are now being
acquired at an increasing pace in several applications (e.g., LiDAR in
autonomous or assisted driving). In many cases, such volume of data is
transmitted, thus requiring that proper compression tools are applied to
either reduce the resolution or the bandwidth. In this paper, we propose a new
solution for upscaling and restoration of time-varying 3D video point clouds
after they have been heavily compressed. Our model consists of a specifically
designed Graph Convolutional Network (GCN) that combines Dynamic Edge
Convolution and Graph Attention Networks for feature aggregation in a
Generative Adversarial setting. We present a different way to sample dense
point clouds with the intent to make these modules work in synergy to provide
each node enough features about its neighbourhood in order to later on
generate new vertices. Compared to other solutions in the literature that
address the same task, our proposed model is capable of obtaining comparable
results in terms of quality of the reconstruction, while using a substantially
lower number of parameters ($\simeq$ 300KB), making our solution deployable in
edge computing devices such as LiDAR.

Time varying 3D point clouds, 3D upscaling, Graph Attention Network,
Generative Adversarial setting, Super Resolution

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: ACM Multimedia; October 29–November 03, 2023;
Ottawa, Canada††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing
methodologies Point-based models††ccs: Computing methodologies Neural networks
Figure 1. Top: Schematic architecture of the Generator GCN used for our
upsampling model. A set of Edge Convolution with Attention layers are cascaded
to generate an embedding from the input point clouds; the Parallel Double
Sampling (PDS) layer is then used for upsampling, and the output is summed at
the end with the input following a residual-like schema. Bottom: A short set
of point cloud frames used as input are shown on the left; then, the output of
the intermediate Edge Convolution with Attention layers are shown in the
middle; the upsampled output point cloud is given on the right.

##  1\. Introduction

In light of emerging applications such as Augmented and Virtual Reality
(AR/VR), there is a rising interest in capturing the real world in 3D at high-
resolution. For real time applications in dynamic settings, such as 3D sensing
for robotics, telepresence, automated driving applications using LiDAR, this
technology might need high-resolution point clouds with up to millions of
points per frame. After taking into consideration the average point-cloud
video, under some constraints such as keeping the identity of a human subject
recognizable, we observe that the size of a single instance, which is a single
frame, can be approximated as $\sim$10 Mbytes, which translates to a bitrate
of $\sim$300 Mbytes per second without compression for a 30-fps dynamic point
cloud. The high data rate is one of the main problems faced by dynamic point
clouds, and efficient compression technologies to allow for the distribution
of such content are still widely sought. One result in this direction, is
represented by the Point Cloud Compression standard specifications that
include video-based PCC (V-PCC) and geometry-based PCC (G-PCC) (Graziosi et
al., 2020) as released in 2020 by the The Moving Picture Expert Group (MPEG).

Given these premises, our task is to perform upscaling and artifact removal of
sparsely populated 3D point cloud videos. The terms upscaling and artifact
removal are usually found in image/video super resolution literature and so
they might not have an immediate translation in the 3D context. We will use
the term upscale to indicate the operation by which the total number of
vertices of an input point cloud is increased; by using artifact removal,
instead, we will imply the correct reconstruction process after some sort of
compression or subsampling has been performed on an input point cloud. As
shown in Figure 2, a high compression rate can achieve acceptable bandwidth
requirements with a huge decrease in fidelity. For some applications, for
example where the user experience is important, the identity of the subject
must be maintained or, for autonomous driving, such a low-resolution is not
acceptable.

Recent approaches that tackled this task, such as (Vu et al., 2022), employed
a strategy that uses long sequences of input frames and a large encoder-
decoder model. As we will detail below, we followed a different approach.

In this paper, we pose the upscaling problem in a Generative Adversarial
setting using two architectural modules from the literature: the
EdgeConvolution (Wang et al., 2019) and the Graph Attention Network (GAT)
(Veličković et al., 2017). In particular, the input point clouds are modeled
as graphs and processed by a Graph Convolutional Network (GCN). The
convolution operation has been performed using a EdgeConv module: this module
incorporates local neighborhood information, can be stacked to learn global
shape properties, and affinity in feature space captures semantic
characteristics over potentially long distances in the original embedding.
While this module was used for CNN-based high-level tasks on point clouds,
including classification and segmentation, the GAT has been used for feature
aggregation performing an attentioned learned mean of the neighbourhood
features instead of simply averaging it out. Experiments have been performed
on the FAUST 4D dataset (Bogo et al., 2017) also in comparison with state-of-
the-art solutions. Overall, our method shown upscaling reconstructions that
are comparable with those reported in the literature, while using a lower
number of input frames and an architecture with a much lower number of
parameters. This opens the way to the deployment of our architecture in edge
computing devices.

In summary, the main contributions of our work can be summarized as follows:

  * •

We propose a new architecture for time varying point cloud upscaling that
combines together a PointNet (Qi et al., 2017b), used as a Discriminator
Network in a GAN, and a Generator that makes use of Edge Convolution on the
input graphs derived from the point clouds and a graph attention mechanism for
aggregating the features of the local neighbourhood. The resulting GAN
architecture represents a setting that, to our knowledge, has not been tried
before for this task;

  * •

The proposed solution demonstrates a clear advantage over the existing methods
in the capability of producing upscaled 3D point clouds with comparable
accuracy but using a way lower number of parameters in the architecture.
Finally, the inference time is compatible with an online application of the
method handling a stream of input frames.

Figure 2. Left: Sample of an input low-resolution point cloud with $\sim$3K
vertices. Center: our model reconstruction with $\sim$12K vertices. Right:
Ground truth point cloud with $\sim$12K vertices.

##  2\. Related work

Numerous studies have been conducted with the goal of reconstructing a 3D
model given inputs in various possible forms: a mesh, a 3D point cloud, a
collection of voxels or an implicit function. Some of these works focused on
the use of a 3D point cloud as an input (Fan et al., 2017; Ruizhongtai Qi et
al., 2017). Others, instead, used a discretized version based on voxels, such
as (Girdhar et al., 2016; Wang et al., 2017), or directly tried to reconstruct
a mesh (Liao et al., 2018; Wang et al., 2018b).

Point cloud upsampling was first approached using optimization based
solutions, while deep learning based methods were applied only more recently.
Methods from both these categories are summarized below.

Optimization-based methods. One of the first work addressing point sets
upsampling was proposed by Alexa et al. (Alexa et al., 2003). In their
approach, points at vertices of a Voronoi diagram were interpolated in the
local tangent space. Lipman et al. (Lipman et al., 2007), presented a Locally
Optimal Projection (LOP) operator performing points resampling and surface
reconstruction using L1-median. The LOP operator showed satisfactory results
even in the case the input point set was affected by noise and outliers. An
improved version of the LOP approach aiming to address the density problem of
the upscaled point set was then proposed by Huang et al. (Huang et al., 2009).
Overall, good results were demonstrated by these works though their
applicability scope was limited by the smoothness assumption of the underlying
surface, which is rarely matched by data acquired with real scanners. To
overcome such limitation, in (Huang et al., 2013) Huang et al. proposed an
edge-aware point set resampling solution that first resamples away from edges,
then progressively approaches edges and corners. One limitation of this method
is the dependence of the quality of the results from the normals accuracy at
the points, and the need for a careful tuning of the parameters. A point
representation method based on volumetric voxelization was introduced by Wu et
al. (Wu et al., 2015). As a preliminary operation, they proposed to fuse
consolidation and completion in one coherent step. However, the goal of this
operation was on filling large holes, so that global smoothness is not
enforced, making the method sensitive to large noise. All these methods are
not driven by the data, rather they strongly rely on some priors.

Deep-learning based methods. Only recently, methods have adopted deep
architectures to directly learn from point sets. This was mainly due to the
inherent difficulty of such data, where points are unordered and do not follow
any regular-grid structure in their spatial arrangement. To circumvent such
difficulty, some methods converted point clouds to other 3D representations,
based on graphs (Bruna et al., 2014; Masci et al., 2015) or volumetric grids
(Dai et al., 2017; Maturana and Scherer, 2015; Riegler et al., 2017; Wu et
al., 2015). The PointNet (Qi et al., 2017a) and Point Net++ (Qi et al., 2017b)
were the first successful attempts to directly process point clouds for
classification and segmentation purposes using a hierarchical feature learning
architecture that captures both local and global geometry contexts. Other
networks that were proposed for high-level analysis of point clouds focusing
on global or mid-level attributes of point clouds include (Hua et al., 2018;
Klokov and Lempitsky, 2017; Li et al., 2018; Qi et al., 2018; Wang et al.,
2018a). Local shape properties, like normal and curvature in point clouds,
were estimated by the network proposed in (Guerrero et al., 2018). Interesting
network architectures were also proposed for 3D reconstruction from 2D images
(Fan et al., 2017; Groueix et al., 2018; Lin et al., 2018). For example, Fan
et al. (Fan et al., 2017) addressed the problem of 3D reconstruction from a
single image, generating a straight-forward form of output–point cloud
coordinates. The 4D extension of the resulting Point Set Generation Network
(PSGN-4D) was used in several studies as a baseline for comparison.

One of the first work aiming to perform point cloud upsampling was proposed by
Yu et al. (Yu et al., 2018b). They introduced the PU-Net that learns per point
features at multiple scales, and expands the set of points using a Multi-layer
Perceptron (MLP) with multiple branches. However, to learn multi-layer
features the input point sets are downsampled, thus potentially causing a loss
of resolution. In (Yu et al., 2018a), the same authors proposed an edge-aware
network for point set consolidation (EC-Net) that uses a specific loss to
encourage learning to consolidate points for edges. On the negative side, a
very expensive edge-notation is needed for training the EC-Net. In the work of
Yifan et al. (Yifan et al., 2019), a progressive network (3PU) was proposed
that duplicates the input point patches over multiple steps. The progressive
architecture of 3PU makes its training computationally expensive. More data
are also required to supervise the middle stage outputs of the network. A
Generative Adversarial Network designed to learn upsampled point distributions
(PU-GAN) was proposed by Li et al. (Li et al., 2019), with the main
performance improvement obtained by the discriminator. Qian et al. (Qian et
al., 2020) proposed to upsample points by learning the first and second
fundamental forms of the local geometry. However, their PUGeo-Net needs
additional supervision in the form of normals. The PU-GCN proposed by Qian et
al. (Qian et al., 2021) performed upsampling by leveraging on an Inception
based module to extract multi-scale information, and using a GCN-based
upsampling module to capture local point information. This has the main
advantage of not needing for additional annotations, like edges, normals,
point clouds at intermediate resolutions, etc., while also avoiding the use of
a sophisticated discriminator.

Recently more and more works have been shifting the attention towards 4D
reconstruction, where a sequence of 3D objects is reconstructed from time-
varying point clouds given as inputs (Leroy et al., 2017; Mustafa et al.,
2016).

In the Occupancy Network (ONet) proposed by Mescheder et al. (Mescheder et
al., 2019), a 3D object was described using a continuous indicator function
that indicates which sub-sets of the 3D space the object occupies, and an iso-
surface retrieved by employing the Marching Cube algorithm. Tang et al. (Tang
et al., 2021) learned a temporal evolution of the 3D human shape through
spatially continuous transformation functions among cross-frame occupancy
fields. To this end, they established, in parallel, the dense correspondence
between predicted occupancy fields at different time steps via explicitly
learning continuous displacement vector fields from spatio-temporal shape
representations. Niemeyer et al. (Niemeyer et al., 2019) introduced a
learning-based framework for object reconstruction directly from 4D data
without predefined templates. The proposed OFlow method calculates the
integral of a motion field of 3D points in a 3D point cloud specified in space
and time to implicitly represent trajectories of all the points in dense
correspondences between occupancy fields. Vu et al. (Vu et al., 2022) proposed
a network architecture, called RFNet-4D, that jointly reconstructs objects and
their motion flows from 4D point clouds. It is shown that jointly learning
spatial and temporal features from a sequence of point clouds can leverage
individual tasks, leading to improved overall performance. To this end, a
temporal vector field learning module using unsupervised learning approach for
flow estimation was designed that, in turn, leveraged by supervised learning
of spatial structures for object reconstruction. Jiang et al. (Jiang et al.,
2021) introduced a compositional representation that disentangles shape,
initial state, and motion for a 3D object that deforms over a temporal
interval. Each component is represented by a latent code via a trained
encoder. A neural Ordinary Differential Equation (ODE) is used to model the
motion: it is trained to update the initial state conditioned on the learned
motion code, while a decoder takes the shape code and the updated state code
to reconstruct the 3D model at each time stamp. An Identity Exchange Training
(IET) strategy is also proposed to encourage the network to learn decoupling
each component.

With respect to the above solutions, our approach is characterized by a
specific design that combines two Graph Convolutional Networks (GCNs) to work
in an adversarial setting (GAN). The resulting architecture proved to be
flexible in the number of frames used as inputs and conjugated effective
reconstructions with inference times that are compatible with online
execution.

##  3\. Problem Statement

We consider a sequence of point clouds in the 3D space. Each point cloud can
be regarded as a frame of a 4D video at time $t$. In the following, we
consider $n$ point cloud frames fused together forming a time varying point
cloud as an unordered lists of ${x,y,z,t}$ points. Our task is to upscale, a
term borrowed from the 2D image super-resolution domain, each of the point
cloud (frame) of the input sequence $F_{t}$ and get a more detailed one by
leveraging the information of the previous $n-1$ low-resolution point cloud
frames (i.e., $F_{t-1},\dots,F_{t-n+1}$).

More in detail, given a buffer composed of $n$ previous frames, the input
point cloud $P_{i}$ is defined as:

(1) $P_{i}=\\{p_{-n+1},p_{-n+2},...,p_{0}\\};P_{i}\in\mathbb{R}^{\\{x,y,z,t\
\times L\times n\\}},$

where each low-resolution point cloud $p_{i}$ is composed of $L$ points:

(2) $p_{i}\in\mathbb{R}^{\\{x,y,z,t\ \times L\\}}.$

We are interested in learning a map $f(P_{i},\theta)$ from $P_{i}$
$\rightarrow$ $P_{T}$, where $P_{T}$ is the target point cloud and it
represents the zeroth frame upscaled to have $H=S\times L\times n$ points,
with $S$ being the scale factor.

(3) $P_{T}\in\mathbb{R}^{\\{x,y,z\ \times H\\}}.$

##  4\. Proposed Method

Our proposed method makes use of message passing Graph Networks, different
neighbourhood sampling techniques and Generative Adversarial training.

We employed a Graph Neural Network for this task. More in detail, our
architecture has been developed starting from (Qi et al., 2017b). The employed
architecture works on unordered lists of ${x,y,z,t}$ points, representing the
last $n$ frames fused together, using two Graph Convolutional Neural Networks
(GCNs from here on) in an adversarial setting. The discriminator is based on
(Qi et al., 2017b), while the generator improves on the architecture proposed
in (Qi et al., 2017b). In particular, we used different neighbors sampling
techniques that were developed with the intent of collecting, for each point,
features contemporaneously of its immediate neighborhood and also from
furthest vertices of the whole point cloud without making the computation too
expensive.

The fully convolutional nature of our generator network allows us to
potentially train and test at different input and output resolutions.

###  4.1. Edge Convolution and GAT

The basic module composing our generator network is made of the combination of
Edge Convolution (Wang et al., 2019) and Graph Attention Networks (GAT)
(Veličković et al., 2017). The Edge Convolution allows us to perform message
passing over a dynamic graph in which the edges are updated as the point cloud
changes. The GAT side is used to perform an attentional aggregation over the
features collected from the dynamic local neighbourhood, this in contrast with
much more common choices for aggregation such as $max$ or $average$. We refer
to this combination module as Edge Convolution with Attention.

###  4.2. Parallel Double Sampling (PDS) module

The core of the generator side of the architecture is the Parallel Double
Sampling (PDS) module that performs two different graph convolutions using two
different sets of sampled points. A simplified illustration of this module is
presented in Figure 3. For each point, two sets of operations are performed in
a parallel fashion. The first set, is a pipeline composed of:

  * •

Radius filtering: For each vertex, a filtering step leaves as neighbors, with
the capability of passing messages, only those vertices that belong to a
sphere of radius $r$, centered on the vertex.

  * •

Furthest Point Subsampling: We use the Furthest Point Subsampling (FPS)
algorithm in (Qi et al., 2017b) in order to sample temporarily, a fraction $s$
of the original points that are the farthest away, inside the radius, from a
starting point.

  * •

Convolution: Graph convolution is applied over the remaining vertices,
independently of their number, and their features are aggregated.

The second set of operations, performed in parallel to the first one, is
composed of:

  * •

K-NN: A fixed number of $k$ closest vertices is selected as neighbors.

  * •

Convolution: Graph convolution is applied over the vertices, and their
aggregated features.

Figure 3. Schematic representation of the proposed Parallel Double Sampling
(PDS) module.

Finally, the two sets of features are concatenated and fed to a linear layer
that maps $2\times Channels_{in}\rightarrow Channels_{out}$.

###  4.3. Our Architecture

The developed architecture is composed of two Graph Convolutional Networks
(GCNs) working in an adversarial setting (GAN) (Goodfellow et al., 2014). It
is illustrated in the bottom of Figure 4. Basically, the point cloud given as
input is processed as a graph using message passing based convolution.

###  4.4. Discriminator

The discriminator is inspired by the PointNet++ architecture (Qi et al.,
2017b), since it also targets a classification task. We use the same structure
that progressively reduces the number of points using max-pooling operations
and finally a sequence of linear layers before the output as shown in the
bottom part of Figure 4.

###  4.5. Generator

The generator side of the model is instead built as an initial sequence of
Edge Convolution with Attention modules followed by our Parallel Double
Sampling (PDS) module. It is also inspired by the PointNet++ architecture (Qi
et al., 2017b) but undergoing major changes as detailed in Section 4.2. In the
top part of Figure 4 a simplified visualization of the PDS generator is
presented. The generator is composed of multiple Graph Convolutions with
Attention followed by a single PDS. The intuition behind this choice is to
collect various features for each node, using different neighborhood sampling
techniques. Once the original node has been enriched with the local features,
the PDS will use them to generate multiple new vertices according to the scale
factor. Finally this new vertices position is summed with the closest one that
originated it, in a sort of residual fashion (see Figure 1).

The generator loss $L_{g}$ is composed of an adversarial component $L_{adv}$
coming from the Discriminator,a full reference reconstruction loss $L_{rec}$
computed as the Chamfer Distance between the restored point cloud and the
original one and an additional Density Loss $L_{D}$. We used the LSGAN from
(Mao et al., 2016) loss for our training, which assumes the form:

(4) $L_{adv}=\min_{G}L\left(G\right)=\frac{1}{2}\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}\left(\mathbf{z}\right)}\left[\left(D\left(G\left(\mathbf{z}\right)\right)-c\right)^{2}\right],$

for the generator, and:

(5)
$\displaystyle\min_{D}L\left(D\right)=\frac{1}{2}\mathbb{E}_{\mathbf{x}\sim
p_{data}\left(\mathbf{x}\right)}\left[\left(D\left(\mathbf{x}\right)-b\right)^{2}\right]+$
(6) $\displaystyle+\frac{1}{2}\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}\left(\mathbf{z}\right)}\left[\left(D\left(G\left(\mathbf{z}\right)\right)-a\right)^{2}\right],$

for the discriminator.

Figure 4. Schematic representation of the proposed GCN architecture. Top:
Generator architecture; Bottom: Discriminator architecture.

###  4.6. Loss functions

The model is trained end-to-end using multiple losses. Beside the adversarial
component $L_{adv}$, we also compute the point-to-set distance (Chamfer
distance) $L_{rec}$ between the reconstructed point cloud and the target one
and, similarly to (Wu et al., 2021), we take into account the neighbourhood of
each point. That is, for each reconstructed point $p_{r}\in P_{r}$, we find
the closest point $p_{t}\in P_{t}$ in the target point cloud, and compute both
the distance between them and the difference in terms of local neighbors:

(7) $L_{CD}(P_{r},P_{t})=\sum_{r\in P_{r}}\operatorname*{min}_{t\in
P_{t}}||r-t||^{2}_{2}+\sum_{t\in P_{t}}\operatorname*{min}_{r\in
P_{r}}||r-t||^{2}_{2}.$

We define a vertex $p$ neighbourhood density $D(p)$ as the normalized sum of
its neighbours in a given radius:

(8) $D({p\in P})=\frac{1}{N_{max}}\sum_{n\in Ball_{p}}1,$

(9) $L_{D}(P_{r},P_{t})=\sum_{r\in P_{r}}\operatorname*{min}_{t\in
P_{t}}||D(r)-D(t)||^{2}_{2}+\sum_{t\in P_{t}}\operatorname*{min}_{r\in
P_{r}}||D(r)-D(t)||^{2}_{2}.$

The generator final loss is therefore given by:

(10) $L_{rec}=\lambda_{1}L_{CD}+\lambda_{2}L_{D}+\lambda_{3}L_{Adv},$

where values for $\lambda_{i}$ have been empirically determined
($\lambda_{1}=1.0,\lambda_{2}=0.5,\lambda_{3}=0.1$).

##  5\. Experiments

The proposed solution for point clouds upscaling has been evaluated in a
comprehensive set of experiments. both qualitative (Section 5.3) and
quantitative (Section 5.4). An ablation study aiming to evidence the relevance
of different components of our architecture is also reported in Section 5.5.

###  5.1. Implementation details

Our model is implemented in PyTorch, using the PyTorch Geometric (PyG) library
(Fey and Lenssen, 2019). This library was built upon PyTorch and is
specifically designed for Graph Neural Networks (GNNs). The two networks are
implemented as two Message Passing Networks put in an adversarial setting.
Both the Discriminator and the Generator are optimized with Adam, using the
standard learning rate $lr=1e^{-4}$ and betas $\beta_{1}=0.9,\beta_{2}=0.999$,
using a linear decaying scheduler that drops the learning rate to 1/10th every
10 epochs. Other hyperparameters, such as the radii for the Ball Query for the
FPS sampling ($r_{small}=0.06$, $r_{large}=0.1$) and the number of neighbours
for the KNN sampling ($n_{neighbours}=9$) were empirically determined trough
grid search.

####  5.1.1. Augmentation

The training data is augmented using different operations. Each sequence of
input point clouds and its relative ground truth point cloud is randomly
flipped along any of its axes, per point random noise is added, and finally a
random scale along any axes is applied in a range between $[0.9,1.1]$. As a
form of augmentation, we also exploit the fully convolutional nature of the
generator architecture; similar to the case of 2D image super-resolution,
where patches of the target high resolution image are used in the training, we
randomly feed a 3D slice of the video instead of the full body. A final form
of augmentation used during training is a simple time inversion inside the
sequence.

|
---|---

|
---|---

Figure 5. The top and bottom row show: (left) the point clouds of a three-
frame input sequence with movements. Colors indicate the movement of a point
with respect to the previous frame; (right) different features obtained from
subsequent edge graph convolutional layers of the proposed architecture as a
response to the three-frame sequence shown on the left. It can be noted the
response of layers seems to pass from spatial to temporal details.

###  5.2. Dataset

To evaluate our proposed solution, we used the Dynamic FAUST (D-FAUST) dataset
(Bogo et al., 2017). It contains animated meshes for 129 sequences of 10 human
subjects (5 females and 5 males) with various motions such as “shake hips”,
“punching”, running on “spot”, or “one leg jump”.

In order to compare with other methods, we used the train/test split proposed
in (Niemeyer et al., 2019). For each sequence, at training time, we randomly
pick an index and then subsample the following frames according to the model’s
frame rate. We trained multiple models at different frame rates.

We also followed the evaluation setup used in (Niemeyer et al., 2019).
Specifically, for each evaluation, we carried out two case studies: seen
individuals but unseen motions (i.e., test subjects were included in the
training data but their motions were not given in the training set); and
unseen individuals but seen motions (i.e., test subjects were found only in
the test data but their motions were seen in the training set).

###  5.3. Qualitative results

Some qualitative results of the proposed upscaling are given in Figures 6 and
7. In Figure 6, the input low-resolution frame, our reconstruction point cloud
and the ground truth are given from left ro right. A second example is shown
in Figure 7, where the input frame, our reconstruction and the ground truth
are compared both in terms of point clouds (top) and in terms of mesh
reconstruction using the Poisson algorithm (bottom). Additional qualitative
results are given as videos in the supplementary material.

To give some insights on the behaviour of the network layers, we inspected the
response of the various convolutional layers given an input point cloud, and
visualized them. As an example, on the left of Figure 5, three frames of an
input point cloud are shown (the frames are taken at three consecutive times,
$t_{0}$, $t_{0}+1$ and $t_{0}+2$). Points in the clouds are colored to
highlight their movement with respect to the previous frame. On the right of
Figure 5, instead, the response features of different layers are visualized
(the depth of the layers increases from left to right). It is interesting to
note as, similarly to CNNs, depth correlates to complexity: The first
convolutional layers seem to have strong response for large physical parts of
the human subject, while the latter stages take into consideration time and
movement.

Figure 6. Left: Sample of a single frame from an input low resolution point
cloud with $\sim$512 vertices, Center: reconstruction obtained with our
proposed solution; Right: Ground truth point cloud. Figure 7. Top: Point cloud
visualization. Bottom: Mesh reconstruction using Poisson surface
reconstruction from (Kazhdan et al., 2006). Left: Sample obtained from a
single frame of an input low-resolution point cloud with 1024 vertices;
Center: Model reconstruction using our proposed approach; Right: Ground truth
point cloud.

###  5.4. Quantitative results

####  5.4.1. Evaluation Metrics

To measure the reconstruction quality we applied the standard Chamfer Distance
(CD), a point-to-set metric since following the same protocol as reported in
(Vu et al., 2022) that uses the CD as reconstruction metric for measuring the
dissimilarity between a point and a point set.

####  5.4.2. Compared Methods

We compared our approach with respect to six state-of-the-art solutions in the
literature for 4D reconstruction from point cloud sequences, namely, PSGN 4D,
ONet 4D, OFlow, LPDC, 4DCR, and RFNet-4D. The PSGN 4D extends the PSGN
approach (Fan et al., 2017) to predict a 4D point cloud, i.e., the point cloud
trajectory instead of a single point set. The ONet 4D network is an extension
of ONet (Mescheder et al., 2019) to define the occupancy field in the spatio-
temporal domain by predicting occupancy values for points sample in space and
time. The OFlow network (Niemeyer et al., 2019) assigns each 4D point an
occupancy value and a motion velocity vector and relies on the differential
equation to calculate the trajectory. The LPDC (Tang et al., 2021) learned a
temporal evolution of the 3D human shape through spatially continuous
transformation functions among cross-frame occupancy fields. The 4DCR solution
(Jiang et al., 2021) used a compositional representation that disentangles
shape, initial state, and motion for a 3D object that deforms over a temporal
interval. Finally, RFNet-4D (Vu et al., 2022) jointly reconstructs objects and
their motion flows from 4D point clouds.

####  5.4.3. Results

Tables 1 and 2 report results for our solution and for the other methods as
given in (Vu et al., 2022). For our method (last line in the tables) we used 3
frames for upscaling at 60fps with a scale factor of $\times 4$ starting from
low-resolution point clouds composed of 1024 vertices. For the unseen
individual and seen motion protocol in Table 1, our approach achieves the
second best score. From Table 2, it can be observed that our method reached a
reconstruction error of similar magnitude with respect to the two best
performing methods, i.e., RFNet-4D and LPDC. It is worth noting that RFNet-4D
obtained the reported error using a larger number of input frames (i.e., 17
against 3 to 8 as used in our tests). It was not possible to test the RFNet-4D
with our setting because the code was not publicly available.

Method | Chamfer Distance x $10^{-3}\downarrow$
---|---
PSGN-4D (Fan et al., 2017) | 0.6877
ONet-4D (Mescheder et al., 2019) | 0.7007
OFlow (Niemeyer et al., 2019) | 0.2741
4DCR (Jiang et al., 2021) | 0.2220
LPDC (Tang et al., 2021) | 0.2188
RFNet-4D (Vu et al., 2022) | 0.1594
Ours | 0.1758
Table 1. Reconstruction accuracy for the unseen individuals and seen motions protocol. We report the Chamfer distance (lower is better). Results for the best and second best performing methods are given in bold and underlined, respectively. Our approach scored the second best accuracy.  Method | Chamfer Distance x $10^{-3}\downarrow$
---|---
PSGN-4D (Fan et al., 2017) | 0.6189
ONet-4D (Mescheder et al., 2019) | 0.5921
OFlow (Niemeyer et al., 2019) | 0.1773
4DCR (Jiang et al., 2021) | 0.1667
LPDC (Tang et al., 2021) | 0.1526
RFNet-4D (Vu et al., 2022) | 0.1504
Ours | 0.1638
Table 2. Reconstruction accuracy for the seen individuals and unseen motions
protocol. We report the Chamfer distance (lower is better). Results for the
best and second best performing methods are given in bold and underlined,
respectively. Our approach results in the third best performance.

In Table 3, we report the inference time, in seconds, for various different
configurations of our model. All the measurements correspond to experiments
executed on an Nvidia 2080Ti GPU. The values reported in the table evidence
that our approach can open the way to real-time upscaling. As reported in (Vu
et al., 2022), their method used 17 inout frames to reconstruct an output
frame, while our range of frames is between 3 (for models using larger input
point clouds) and 8 (for smaller inputs) due to memory constraint at training
time.

Method | Input size | Upscale $\times$ | Inference time (s) $\downarrow$
---|---|---|---
Ours | 1024 | 3 | 0.103
Ours | 1024 | 2 | 0.089
Ours | 512 | 4 | 0.046
Ours | 512 | 2 | 0.039
Ours | 256 | 8 | 0.034
Ours | 256 | 4 | 0.030
Oflow(Mescheder et al., 2019) | - | - | 0.95
LDPC(Tang et al., 2021) | - | - | 0.44
RfNet4d(Vu et al., 2022) | - | - | 0.24
Table 3. Inference time for different configurations of our model using a
three-frames buffer. Every test was performed on an Nvidia2080Ti. For the
other models it must be noted that they used a 17 frame input sequence to
output a frame.

###  5.5. Ablation Studies

In this section, we present ablation studies to verify different aspects in
the design of our model. In particular, we verify each of the introduced
components in our architecture for 4D point clouds reconstruction by comparing
the percentage decrease of the model when some particular features are
removed.

We performed a first set of experiments by using a stream of input point
clouds at 60fps and with 256 points per frame; on this stream, we performed
upscaling from subsets of consecutive 3 frames, using an upscale factor of
$\times$2\. From Table 4, we can notice that by removing individual components
of our architecture, the performance of the model significantly and
consistently decreases. In particular, we removed the attention aggregation
module and we substituted it with a more common mean aggregation. We also
ablated the impact of the Density Loss and the adversarial component.

Variant | Chamfer x $10^{-3}\downarrow$ | % wrt F. Featured
---|---|---
No Attention | 1.193 | +3.11%
No Density Loss | 1.226 | +5.96%
No Adversarial Loss | 1.213 | +4.84%
Ours Fully Featured | 1.157 | -
Table 4. Ablation study for our model using 256 input points, 3 frames, 60fps,
and upscale factor $\times$2\.

In Table 5, we repeated the above ablation experiments using a different
setup. In this case, the frame rate is changed to 30fps, the input resolution
to 512 points per frame, and we performed upscaling using a factor of
$\times$4.

Variant | Chamfer x $10^{-3}\downarrow$ | % wrt F. Featured
---|---|---
No Attention | 0.5856 | +2.82%
No Density Loss | 0.6433 | +12.95%
No Adversarial Loss | 0.5930 | +4.13%
Ours Fully Featured | 0.5695 | -
Table 5. Ablation study for our model using 512 input points, 3 frames, 30fps,
and upscale factor $\times$4.

Also in this case, ablating the density loss term results into the most
significant decrease in the accuracy of the upscaled model. It is also
interesting to observe that, while the percentage increment in the Chamfer
distance when removing the attention layer and the adversarial loss shows
small differences between the two tables, this is not the case for the density
loss: removing this term has a much larger impact on the results in Table 5
($\sim+13\%$) than in Table 4 ($\sim+6\%$).

####  5.5.1. Importance of the temporal information

A question that arises with the proposed solution is the actual impact of
having the time buffer compared to using just the last point cloud as an
input. To compare these two solutions, we feed our model with the same frame
repeated $n$ times. In this way, we keep the comparison fair by not changing
the input size and the amount of starting points but only the information
contained within it. We refer to this setup as Static Sequence, whilst we use
the term Dynamic to refer the proposed procedure that uses $n$ different
frames. In Table 6, we report some comparative results between the two ways of
using the frames in a sequence. It can be observed that there is useful
information in the time and movement of the cloud. Just like in a 2D video,
the same frame repeated $n$ times does not contain the same amount of useful
data for reconstruction as $n$ different subsequent frames.

Sequence | Input size | Frms | $\times$ | Chamfer x $10^{-3}\downarrow$
---|---|---|---|---
Static | 256 | 3 | 4 | 2.876
Dynamic | 256 | 3 | 4 | 1.109
Static | 256 | 4 | 3 | 2.825
Dynamic | 256 | 4 | 3 | 0.745
Static | 512 | 3 | 2 | 1.851
Dynamic | 512 | 3 | 2 | 0.677
Table 6. Ablation study for our model using the aforementioned sequences at
different resolutions. It shows how the dynamic approach performs consistently
better than the static one.

##  6\. Conclusions

In this paper, we presented a fully convolutional graph-based approach for
time-varying point clouds upscaling using a novel and different approach with
respect to most of the state-of-the-art models. Our proposed method is
comparable with state-of-the-art solutions in terms of upsampling performance
but it has a much lighter architecture that allows the implementation on edge
devices with limited computational capabilities.

As a future development this type of application could be implemented as an
update for older LiDAR devices or to allow faster 3D point cloud streaming by
only transmitting/sampling a subset of the original points.

While out method tackles the problem in a different way bringing some
advantages it still has some limitations and drawbacks:

  * •

Training time and memory footprint. Not relying on an encoder-decoder model
implies having the whole point cloud at every stage of the network in memory,
this slows down training and poses some limitations in the number of input
frames;

  * •

Results for the reconstruction accuracy are comparable with those reported in
the state-of-the-art, though a bit lower.

##  7\. Acknowledgments

This work was supported by the European Commission under European Horizon 2020
Programme, grant number 951911—AI4Media.

## References

  * (1)
  * Alexa et al. (2003) M. Alexa, J. Behr, D. Cohen-Or, S. Fleishman, D. Levin, and C.T. Silva. 2003\.  Computing and rendering point set surfaces.  _IEEE Trans. on Visualization and Computer Graphics_ 9, 1 (2003), 3–15.  https://doi.org/10.1109/TVCG.2003.1175093
  * Bogo et al. (2017) Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2017.  Dynamic FAUST: Registering Human Bodies in Motion. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Bruna et al. (2014) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. 2014.  Spectral networks and locally connected networks on graphs. In _Int. Conf. on Learning Representations (ICLR)_.
  * Dai et al. (2017) Angela Dai, Charles Ruizhongtai Qi, and Matthias Niessner. 2017\.  Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Fan et al. (2017) Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017.  A Point Set Generation Network for 3D Object Reconstruction From a Single Image. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Fey and Lenssen (2019) Matthias Fey and Jan Eric Lenssen. 2019.  Fast graph representation learning with PyTorch Geometric.  _arXiv preprint arXiv:1903.02428_ (2019).
  * Girdhar et al. (2016) R. Girdhar, D.F. Fouhey, M. Rodriguez, and A. Gupta. 2016\.  Learning a Predictable and Generative Vector Representation for Objects. In _European Conf. on Computer Vision (ECCV)_.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _Advances in Neural Information Processing Systems_ , Vol. 27.  https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
  * Graziosi et al. (2020) D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai. 2020\.  An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC).  _APSIPA Transactions on Signal and Information Processing_ 9 (2020), e13.  https://doi.org/10.1017/ATSIP.2020.12
  * Groueix et al. (2018) Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 2018\.  A Papier-Mâché Approach to Learning 3D Surface Generation. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Guerrero et al. (2018) Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J. Mitra. 2018.  PCPNet Learning Local Shape Properties from Raw Point Clouds.  _Computer Graphics Forum_ 37, 2 (2018), 75–85.  https://doi.org/10.1111/cgf.13343
  * Hua et al. (2018) Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. 2018\.  Pointwise Convolutional Neural Networks. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Huang et al. (2009) Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or. 2009.  Consolidation of Unorganized Point Clouds for Surface Reconstruction.  _ACM Trans. on Graphics_ 28, 5 (dec 2009), 1–7.  https://doi.org/10.1145/1618452.1618522
  * Huang et al. (2013) Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao (Richard) Zhang. 2013\.  Edge-Aware Point Set Resampling.  _ACM Trans. on Graphics_ 32, 1, Article 9 (feb 2013), 12 pages.  https://doi.org/10.1145/2421636.2421645
  * Jiang et al. (2021) Boyan Jiang, Yinda Zhang, Xingkui Wei, Xiangyang Xue, and Yanwei Fu. 2021.  Learning Compositional Representation for 4D Captures With Neural ODE. In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_. 5340–5350.
  * Kazhdan et al. (2006) Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006\.  Poisson Surface Reconstruction. In _Eurographics Symposium on Geometry Processing_ _(SGP ’06)_. Eurographics Association, Goslar, DEU, 61–70.
  * Klokov and Lempitsky (2017) Roman Klokov and Victor Lempitsky. 2017.  Escape From Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models. In _IEEE Int. Conf. on Computer Vision (ICCV)_.
  * Leroy et al. (2017) Vincent Leroy, Jean-Sebastien Franco, and Edmond Boyer. 2017.  Multi-View Dynamic Shape Refinement Using Local Temporal Integration. In _IEEE Int. Conf. on Computer Vision (ICCV)_.
  * Li et al. (2019) Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2019.  PU-GAN: A Point Cloud Upsampling Adversarial Network. In _IEEE/CVF Int. Conf. on Computer Vision (ICCV)_.
  * Li et al. (2018) Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018\.  PointCNN: Convolution On X-Transformed Points. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 31.
  * Liao et al. (2018) Yiyi Liao, Simon Donné, and Andreas Geiger. 2018.  Deep Marching Cubes: Learning Explicit Surface Representations. In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_. 2916–2925.  https://doi.org/10.1109/CVPR.2018.00308
  * Lin et al. (2018) Chen-Hsuan Lin, Chen Kong, and Simon Lucey. 2018.  Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction. In _AAAI Conf. on Artificial Intelligence_ , Vol. 32.
  * Lipman et al. (2007) Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer. 2007.  Parameterization-Free Projection for Geometry Reconstruction.  _ACM Trans. on Graphics_ 26, 3 (jul 2007), 22–es.  https://doi.org/10.1145/1276377.1276405
  * Mao et al. (2016) Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. 2016\.  Least Squares Generative Adversarial Networks.  _2017 IEEE International Conference on Computer Vision (ICCV)_ (2016), 2813–2821.
  * Masci et al. (2015) Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. 2015.  Geodesic Convolutional Neural Networks on Riemannian Manifolds. In _IEEE Int. Conf. on Computer Vision (ICCV) Workshops_.
  * Maturana and Scherer (2015) Daniel Maturana and Sebastian Scherer. 2015.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition. In _IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_. 922–928.  https://doi.org/10.1109/IROS.2015.7353481
  * Mescheder et al. (2019) Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019\.  Occupancy networks: Learning 3D reconstruction in function space. In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_. 4460–4470.
  * Mustafa et al. (2016) Armin Mustafa, Hansung Kim, Jean-Yves Guillemaut, and Adrian Hilton. 2016.  Temporally Coherent 4D Reconstruction of Complex Dynamic Scenes. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Niemeyer et al. (2019) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2019.  Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. In _IEEE/CVF Int. Conf. on Computer Vision (ICCV)_.
  * Qi et al. (2018) Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. 2018.  Frustum PointNets for 3D Object Detection From RGB-D Data. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Qi et al. (2017a) Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017a.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Qi et al. (2017b) Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 30.
  * Qian et al. (2021) Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem. 2021.  PU-GCN: Point Cloud Upsampling Using Graph Convolutional Networks. In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_. 11683–11692.
  * Qian et al. (2020) Yue Qian, Junhui Hou, Sam Kwong, and Ying He. 2020\.  PUGeo-Net: A Geometry-Centric Network for 3D Point Cloud Upsampling. In _European Conf. on Computer Vision (ECCV)_. 752–769.
  * Riegler et al. (2017) Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017\.  OctNet: Learning Deep 3D Representations at High Resolutions. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Ruizhongtai Qi et al. (2017) Charles Ruizhongtai Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. 2017\.  Frustum PointNets for 3D Object Detection from RGB-D Data.  (11 2017).
  * Tang et al. (2021) Jiapeng Tang, Dan Xu, Kui Jia, and Lei Zhang. 2021\.  Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction.  _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ (2021), 6018–6027.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Vu et al. (2022) Tuan-Anh Vu, Duc Thanh Nguyen, Binh-Son Hua, Quang-Hieu Pham, and Sai-Kit Yeung. 2022.  RFNet-4D: Joint Object Reconstruction and Flow Estimation from 4D Point Clouds. In _European Conf. on Computer Vision (ECCV)_. 36–52.
  * Wang et al. (2018b) Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018b.  Pixel2mesh: Generating 3d mesh models from single rgb images. In _European Conf. on Computer Vision (ECCV)_. 52–67.
  * Wang et al. (2017) Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017.  O-CNN: Octree-Based Convolutional Neural Networks for 3D Shape Analysis.  _ACM Trans. on Graphics_ 36, 4, Article 72 (jul 2017), 11 pages.  https://doi.org/10.1145/3072959.3073608
  * Wang et al. (2018a) Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. 2018a.  SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2019) Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2019.  Dynamic graph cnn for learning on point clouds.  _Acm Transactions On Graphics (tog)_ 38, 5 (2019), 1–12.
  * Wu et al. (2021) Tong Wu, Liang Pan, Junzhe Zhang, Tai Wang, Ziwei Liu, and Dahua Lin. 2021\.  Density-aware chamfer distance as a comprehensive metric for point cloud completion.  _arXiv preprint arXiv:2111.12702_ (2021).
  * Wu et al. (2015) Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015.  3D ShapeNets: A Deep Representation for Volumetric Shapes. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Yifan et al. (2019) Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung. 2019.  Patch-Based Progressive 3D Point Set Upsampling. In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Yu et al. (2018a) Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2018a.  EC-Net: an Edge-aware Point set Consolidation Network. In _European Conf. on Computer Vision (ECCV)_.
  * Yu et al. (2018b) Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2018b.  PU-Net: Point Cloud Upsampling Network. In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.


# Generators and the (Accelerated) Future

J Isaacson  Particle Theory Department, Fermi National Accelerator Laboratory,
Batavia, IL 60510, USA<EMAIL_ADDRESS>
###### Abstract

With the High Luminosity LHC coming online in the near future, event
generators will need to provide very large event samples to match the
experimental precision. Currently, the estimated cost to generate these events
exceeds the computing budget of the LHC experiments. To address these issues,
the computing efficiency of event generators need to be improved. Many
different approaches are being taken to achieve this goal. I will cover the
ongoing work on implementing event generators on the GPUs, machine learning
the matrix element, machine learning the phase space, and minimizing the
number of negative weight events.

FERMILAB-CONF-22-071-T

##  1 Introduction

Collider experiments are developed by the high energy physics community to
understand the universe at the smallest distances. Event generators are tools
that have been developed to connect the theory calculations to the
experimental measurements. This connection is made through simulating the
collisions event-by-event through the use of Monte-Carlo methods. The event
generators can be broken down into separate components describing the physics
at different scales involved in the collision [1, 2]. Starting from the
highest scale and working to the lowest scale, these consist of the hard
matrix element, parton showers, and hadronization. Our fundamental theoretical
understanding is encoded in the hard matrix element through matrix element
generators. Matrix element generators are able to compute processes at tree-
level [3, 4, 5, 6, 7, 8, 9, 10], one-loop level [11, 12, 13, 14, 15, 16, 17,
18], and can handle a broad range of Beyond the Standard Model scenarios in an
automated fashion [19, 20, 21].

As typically is the case, increasing the flexibility of these programs comes
at a cost of computational efficiency. The alternative to these automated
tools is to obtain analytic calculations for each process. While this is
feasible for low multiplicity, there are no analytic results for high
multiplicity processes. Therefore, the trade-off in efficiency is acceptable
in order to simulate the needed higher multiplicity events. High multiplicity
events will become even more important with the high-luminosity phase of the
Large Hadron Collider (LHC) starting up in the near future. The datasets
collected in this phase of the LHC will have an unprecedented number of
events. Ensuring that the event generator statistics are not the dominant
uncertainty in these analyses requires a similar sized dataset. However with
current techniques, this would far exceed the computational budget for the
experiments (see the left panel of Fig. 1). The cost of these events are
dominated by the hard matrix element calculation, which scales exponentially
as a function of the multiplicity (see the right panel of Fig. 1).

Figure 1: Left: Estimated computing requirements for current (open circles),
conservative speedups (closed vertical triangles), and aggressive speedups
(closed inverted triangles) as a function of time. Additionally, estimates of
the expected computing budget are given as solid black lines. The figure is
reproduced from Ref. [22]. Right: CPU hours required to generate one million
events as a function of number of final state jets. The calculation of the
hard matrix element is shown in ref, and the calculation of the parton shower
is shown in green and blue. This figure is reproduced from Ref. [23].

To address these computing limitations, many different solutions are being
investigated. Here we will focus on four major categories of advancements
being studied. The first is developing matrix element generators that can take
advantage of hardware accelerators such as graphics processing units (GPUs),
and will be discussed in Sec. 2. The second is using machine learning to
develop a quick but accurate estimate of the matrix element to reduce the need
to perform the full calculation. Details of this approach can be found in Sec.
3. Thirdly, machine learning can also be used to more efficiently sample the
phase space integrals involved in simulating the events for the colliders,
which is explained further in Sec. 4. Finally, when going beyond tree-level
descriptions, negative weights are unavoidable. Negative weights drastically
decrease the effective sample size. Methods to reduce the negative weight
fraction are reviewed in Sec. 5.

##  2 GPU Matrix Element Generators

There are three major efforts investigating the use of GPUs for matrix element
calculations. Each group takes a different approach in handling the conversion
from CPU codes to GPU codes. These approaches are the ones developed in
BlockGen [24], MadFlow [25], and MadGraph-GPU [26].

###  2.1 BlockGen

The BlockGen framework [24] implements the Berends-Giele recursion relation
[27, 28] directly onto GPUs. They performed a comprehensive study comparing
color-ordered (BlockGen-COΣ) and color-dressed (BlockGen-CD${}_{\text{MC}}$)
implementations. The authors found that both algorithms are memory bound and
not compute bound on the GPUs. The memory required by each algorithm can be
found in Fig. 2. The large memory footprint prevents all the needed memory
fitting in the local memory, resulting in memory fetching dominating the time
per event. However, significant improvement over traditional CPU
implementations was found (see Fig 3).

Figure 2: Memory scaling for typical concurrent partonic calculations as a
function of number of particles. The top right panel shows memory independent
of the number of threads, while the left and bottom right panels show the
additional memory per thread. Figure is reproduced from Ref. [24]. Figure 3:
The comparison of different matrix element generator implementations. The
crosses denoted GPU codes, while the circles denote CPU codes. The solid black
(blue) line denotes the best performance amongst all implementations on GPUs
(CPUs). Figure is reproduced from Ref. [24].

###  2.2 MadFlow

The MadFlow framework [25] develops an interface with the MadGraph5_aMC@NLO
program [17] to generate Tensorflow [29] code. Since Tensorflow implements GPU
calculations, the generated code from MadFlow can be directly run on either
CPUs or GPUs. The MadFlow authors compared the runtime of their implementation
on a variety of GPUs and CPUs to determine the overall performance
improvement. While there is drastic improvements in computation speed at low
multiplicities, the gains at high multiplicity are less significant (see Fig.
4).

Figure 4: Performance of the MadFlow algorithm for low multiplicity (left) and
high multiplicity (right). The performance of various GPUs are shown in blue,
and the performance of various CPUs is shown in red. Figure is reproduced from
Ref. [25].

###  2.3 MadGraph-GPU

The MadGraph-GPU framework builds on the MadGraph5_aMC@NLO program to auto-
generate CUDA code [26]. This was handled through a recursive process of
optimizing the output of the C++ generator line by line, instead of
redesigning the code from the ground up for GPUs. While only preliminary
results are currently available for the process
$e^{+}e^{-}\rightarrow\mu^{+}\mu^{-}$, the performance is very promising (see
Tab. 1).

Table 1: Preliminary performance results for the MadGraph-GPU code for the process $e^{+}e^{-}\rightarrow\mu^{+}\mu^{-}$. Values in the table are reproduced from Ref. [26]. Implementation | MEs/sec (double precision) | Ratio to C++ code
---|---|---
MadEvent Fortran | $1.50\times 10^{6}$ | 1.15
Standalone C++ | $1.31\times 10^{6}$ | 1.00
Standalone CUDA: NVidia V100 | $1.37\times 10^{9}$ | 1050
Standalone CUDA: NVidia T4 | $4.01\times 10^{7}$ | 31
|  |

##  3 Machine Learning: Matrix Elements

In addition to trying to more efficiently use the available resources on high-
performance computers, other groups are investigating methods of estimating
the matrix element efficiently. These estimates can then be used to reduce the
number of exact matrix evaluations required through a multiple step
unweighting procedure. This new unweighting technique is referred to as Neural
Rejection Sampling [30]. The proposed algorithm first uses an approximation
for the matrix element, and then does a first accept-reject step. If the event
fails, then there is no need to evaluate the expensive exact matrix element.
However, if the event is accepted, the exact matrix element is evaluated and
another accept-reject step is performed to obtain an unweighted event. The
efficiency of this method can be found in Tab. 2.

Table 2: Performance measures for unweighting $W+4j$ events at the LHC. $\epsilon_{\text{full}}$ is the overall efficiency, $\epsilon_{\text{(1st/2nd),surr}}$ is the efficiency for the surrogate and full matrix element respectively, and $\langle t_{\text{full}}\rangle/\langle t_{\text{surr}}\rangle$ is the ratio of time for the full matrix element to the surrogate. Further details of each parameter and data can be found in Ref. [30]. Process | $\epsilon_{\text{full}}$ | $\langle t_{\text{full}}\rangle/\langle t_{\text{surr}}\rangle$ | $\epsilon_{\text{1st,surr}}$ | $\epsilon^{med}_{\text{2nd,surr}}$
---|---|---|---|---
$dg\rightarrow e^{-}\bar{\nu}_{e}gggu$ | $1.4\times 10^{-3}$ | 667 | $7.1\times 10^{-4}$ | $5.3\times 10^{-2}$
$dd\rightarrow e^{-}\bar{\nu}_{e}ggdu$ | $3.1\times 10^{-4}$ | 162 | $1.1\times 10^{-4}$ | $8.5\times 10^{-2}$
$ud\rightarrow e^{-}\bar{\nu}_{e}duu\bar{d}$ | $3.6\times 10^{-4}$ | 25 | $1.3\times 10^{-4}$ | $7.3\times 10^{-2}$
|  |  |  |

To obtain optimal performance of the neural rejection algortihm requires an
extremely accurate matrix element estimate. This can be obtained with the help
of a factorization aware Matrix element emulator. Such an emulator has been
proposed in Ref. [31]. It uses information about the physics to develop an
extremely accurate neural network estimate. The estimate takes advantage of
the known factorization properties of the the matrix element in the soft and
collinear limits [32]. Using this information, an ansatz can be fit through
the use of a neural network. The accuracy of this method compared to previous
results can be found in Fig. 5.

Figure 5: Comparison of a variety of neural network estimations of the matrix
element compared to the true matrix element for 3 jet (left) and 5 jet (right)
production. The factorization aware result can be found in blue and orange,
with a previous approach shown in red and green. Figure is reproduced from
Ref. [31].

##  4 Machine Learning: Event Generation

While improving the computational speed of the unweighting procedure can help
to drastically reduce the computational cost, another avenue is in using
machine learning to generate events. There are two approaches in this
direction: generating events from a set of pre-generated events [33, 34, 35,
36, 37, 38, 39, 40, 41, 42], and generating the phase space efficiently [33,
43, 44, 45, 46, 47, 48].

###  4.1 From Events

The generation of events from a set of pre-generated events are able to
quickly generate additional events that are similar to the existing ones.
However, these approaches have no guarantees on maintaining the total cross
section and struggle with peaked structures in the integrand. The work in this
direction mainly focuses on the use of GANs to generate the events, and is
very successful in reproducing smooth distributions. The current state of the
art for these methods can be seen in Fig. 6.

Figure 6: Left and Center: Event generation through the use of a trained GAN
network. The left plot shows the accuracy of GANs for smooth distributions,
while the center plot shows the accuracy of these techniques for peaked
distributions. The different curves in the center plot are modifications added
to the loss to improve the accuracy. Right: The weight distribution for even
generation. The unweighting efficiency is directly related to the narrowness
of the distribution. The neural network (green) shows a drastic improvement
over the previous optimal method VEGAS (orange). Left and center figures are
reproduced from Ref. [37], and the right figure is reproduced from Ref. [44].

###  4.2 Phase Space

The other approach attempts to use machine learning to greatly increase the
unweighting efficiency. The advantage of this approach is that it does not
require a pre-generated sample and through the unweighting procedure
guarantees to reproduce the correct cross section. The downside of this
approach is that the networks are more computationally intensive, and still
requires an unweighting step. These two issues lead to a drastically slower
method of generating events than the above approach. The main architecture
investigated are invertible networks, and some promising results have been
found in Refs. [44, 45, 46] (see the right panel of Fig. 6 and Tab. 4.2).

Table 3: Unweighting efficiencies for the default Sherpa and the improvements
from neural network phase space generation for a variety of processes of
interest. Data for the table is taken from Ref. [46].

unweighting efficiency | LO QCD
---|---
|  | $n=0$ | $n=1$ | $n=2$ | $n=3$ | $n=4$
$W^{+}+n$ jets | Sherpa | $2.8\times 10^{-1}$ | $3.8\times 10^{-2}$ | $7.5\times 10^{-3}$ | $1.5\times 10^{-3}$ | $8.3\times 10^{-4}$
| NN+NF | $6.1\times 10^{-1}$ | $1.2\times 10^{-1}$ | $1.0\times 10^{-2}$ | $1.8\times 10^{-3}$ | $8.9\times 10^{-4}$
| Gain | 2.2 | 3.3 | 1.4 | 1.2 | 1.1
$W^{-}+n$ jets | Sherpa | $2.9\times 10^{-1}$ | $4.0\times 10^{-2}$ | $7.7\times 10^{-3}$ | $2.0\times 10^{-3}$ | $9.7\times 10^{-4}$
| NN+NF | $7.0\times 10^{-1}$ | $1.5\times 10^{-1}$ | $1.1\times 10^{-2}$ | $2.2\times 10^{-3}$ | $7.9\times 10^{-4}$
| Gain | 2.4 | 3.3 | 1.4 | 1.1 | 0.82
$Z^{-}+n$ jets | Sherpa | $3.1\times 10^{-1}$ | $3.6\times 10^{-2}$ | $1.5\times 10^{-2}$ | $4.7\times 10^{-3}$ |
| NN+NF | $3.8\times 10^{-1}$ | $1.0\times 10^{-1}$ | $1.4\times 10^{-2}$ | $2.4\times 10^{-3}$ |
| Gain | 1.2 | 2.9 | 0.91 | 0.51 |
|  |  |  |  |  |

##  5 Reducing Negative Weights

In higher order calculations, negative event weights are unavoidable. Negative
weights arise in the color dipole terms, overestimation of real-emission
matrix elements, and in matching to parton showers. The impact of negative
weight events is a reduction in effective sample size, and thus statistical
accuracy. Given a set of events with weights either positive one or negative
one, with a fraction of $\epsilon$ events with negative weight, the additional
events required for the same accuracy is given as:

$f(\epsilon)=\frac{1}{(1-2\epsilon)^{2}}.$ (1)

This quickly increases as $\epsilon$ approaches 50%. Therefore, to improve the
efficiency of event generation, the fraction of negative weight events needs
to be as small as possible. There are two methods of reducing the negative
weight fraction.

The first is through the use of resampling the events to produce only positive
weight events. This can be accomplished through the use of a positiver
resampler [49], a neural network resampler [50], or cell resampling [51].

In the positiver resampler [49], events are rescaled by
$\frac{\sum_{i}w_{i}}{\sum_{i}|w_{i}|}$, partially unweighted, and then
rescaled to ensure that the original histogram bin height is restored. The
reduction in events required to reach a given precision can be seen in Tab. 4.

Table 4: Number of weighted events generated and number of events that pass analysis cuts from merged $W+0j,1j,2j$ at next-to-leading order calculation. Data is taken from Ref. [49]. Sample | Total number of events | Events included in analysis
---|---|---
weighted | 5.3M | 195k
positive only | 3.2M | 121k
unweighted | 1.5M | 52k
Positive Resampler($t$) | 659k | 25k
Positive Resampler($p_{T}^{W}$) | 33k | 33k
|  |

Figure 7: Reduction in required number of events generated through a neural
resampling procedure. Figure is reproduced from Ref. [50].

In the neural resampler [50], a weight and variance are obtained through a
neural network to estimate how close the result is to uniformly weighted
sample. The weight and variance are then used to calculate a scaling factor.
This scaling factor is used to perform an accept-reject step and then the
event weight is increased by this scaling factor. One advantage to this
technique is that the variance of the original sample is maintained. The
validation and reduction in events generated for $pp\rightarrow t\bar{t}$ at
NLO is shown in Fig. 7.

In the cell resampler [51], a phase space measure is defined to calculate the
distance between two points. The negative weight events are then chosen as a
seed of a cell. The cell size is increased until the sum of the weights in the
cell is at least twice the negative weight seed. Afterwards, all the events in
the cell are scaled
$\frac{\sum_{i\in\mathcal{C}}w_{i}}{\sum_{i\in\mathcal{C}}|w_{i}|}$. If there
is an insufficient number of events in a local region to make the weights
positive, the negative weight is kept. This drastically decreases the number
of events required and the fraction of negative weights (see Fig. 8).

Figure 8: Left: Validation that the cell resampler reproduces the expected
result. Right: The fraction of positive weight and negative weight events
before and after the cell resampler. The figure is reproduced from Ref. [51].

Finally, in addition to resampling techniques, the algorithms can be improved
to reduce the negative weight fraction. In Ref. [52], three improvements were
proposed. These improvements were matching to leading color showers, shower
vetos on hard emission events, and local K-factors from the core process. The
combination of these three improvements halved the negative weight fraction
within the study. The breakdown of the effect of each improvement can be found
in Tab. 5.

Table 5: Improvements made in the Sherpa event generator to reduce the fraction of negative weight events. Data is taken from Ref. [52]. | Negative Weight Fraction
---|---
Default | 18.1%
Leading Color Mode | 14.0%
\+ shower veto on $\mathbb{H}$-events | 9.6%
\+ local K-factor from core | 9.1%
|

##  6 Conclusions

In order to match the statistical precision of the high-luminosity LHC, a
large number of events need to be generated with the help of MC programs. The
currently available event generators require a computing budget that exceeds
the provided budget to reach the desired goal. Therefore, there have been many
steps taken to improve event generators to reduce the cost per event.

Speed improvements through the use of dedicated GPU implementations for the
hard matrix element show promise in reducing the cost, but may not be
sufficient on their own. Using machine learning techniques to provide faster
unweighting methods and improved phase space generation also help to increase
the overall speed of event generation. Finally, studying negative weight
reduction techniques works on minimizing number of simulated events required.

While each of these steps help to work towards reducing the computing costs to
the desired level, there is still more that needs to be done in order to meet
the experimental requirements.

I would like to thank Stefan Höche for useful discussions and advice on the
preparation of these proceedings. This manuscript has been authored by Fermi
Research Alliance, LLC under Contract No. DE-AC02-07CH11359 with the U.S.
Department of Energy, Office of Science, Office of High Energy Physics.

## References

  * [1] Webber B 1986 Ann. Rev. Nucl. Part. Sci. 36 253–286 URL http://inspirebeta.net/record/18125
  * [2] Buckley A et al. 2011 Phys. Rept. 504 145–233 (Preprint 1101.2599) URL http://inspirebeta.net/record/884202
  * [3] Kanaki A and Papadopoulos C G 2000 Comput. Phys. Commun. 132 306–315 (Preprint hep-ph/0002082) URL http://inspirehep.net/search?p=hep-ph/0002082
  * [4] Papadopoulos C G 2001 Comput. Phys. Commun. 137 247–254 (Preprint hep-ph/0007335) URL http://inspirehep.net/search?p=hep-ph/0007335
  * [5] Krauss F, Kuhn R and Soff G 2002 JHEP 02 044 (Preprint hep-ph/0109036) URL http://inspirehep.net/search?p=hep-ph/0109036
  * [6] Gleisberg T and Höche S 2008 JHEP 12 039 (Preprint 0808.3674) URL http://inspirehep.net/record/793879
  * [7] Maltoni F and Stelzer T 2003 JHEP 02 027 (Preprint hep-ph/0208156) URL http://inspirehep.net/search?p=hep-ph/0208156
  * [8] Alwall J et al. 2007 JHEP 09 028 (Preprint 0706.2334) URL http://inspirehep.net/search?p=arXiv:0706.2334
  * [9] Alwall J, Herquet M, Maltoni F, Mattelaer O and Stelzer T 2011 JHEP 06 128 (Preprint 1106.0522) URL http://inspirebeta.net/record/912611
  * [10] Mangano M L, Moretti M, Piccinini F, Pittau R and Polosa A D 2003 JHEP 07 001 (Preprint hep-ph/0206293) URL http://inspirehep.net/search?p=hep-ph/0206293
  * [11] Ossola G, Papadopoulos C G and Pittau R 2008 JHEP 0803 042 (Preprint 0711.3596)
  * [12] Gleisberg T and Krauss F 2008 Eur. Phys. J. C53 501–523 (Preprint 0709.2881) URL http://inspirehep.net/search?p=arXiv:0709.2881
  * [13] Berger C F, Bern Z, Dixon L J, Febres-Cordero F, Forde D, Ita H, Kosower D A and Maître D 2008 Phys. Rev. D78 036003 (Preprint 0803.4180) URL http://inspirebeta.net/record/782271
  * [14] Bevilacqua G, Czakon M, Garzelli M, van Hameren A, Kardos A et al. 2013 Comput.Phys.Commun. 184 986–997 (Preprint 1110.1499)
  * [15] Cascioli F, Maierhöfer P and Pozzorini S 2012 Phys.Rev.Lett. 108 111601 (Preprint 1111.5206) URL http://inspirehep.net/record/946998
  * [16] Cullen G, van Deurzen H, Greiner N, Heinrich G, Luisoni G et al. 2014 Eur.Phys.J. C74 3001 (Preprint 1404.7096) URL http://inspirehep.net/record/1292822
  * [17] Alwall J, Frederix R, Frixione S, Hirschi V, Maltoni F, Mattelaer O, Shao H S, Stelzer T, Torrielli P and Zaro M 2014 JHEP 07 079 (Preprint 1405.0301)
  * [18] Actis S, Denner A, Hofer L, Lang J N, Scharf A and Uccirati S 2017 Comput. Phys. Commun. 214 140–173 (Preprint 1605.01090)
  * [19] Christensen N D, de Aquino P, Degrande C, Duhr C, Fuks B, Herquet M, Maltoni F and Schumann S 2011 Eur. Phys. J. C71 1541 (Preprint 0906.2474) URL http://inspirebeta.net/record/823106
  * [20] Degrande C, Duhr C, Fuks B, Grellscheid D, Mattelaer O and Reiter T 2012 Comput.Phys.Commun. 183 1201–1214 (Preprint 1108.2040)
  * [21] Staub F 2014 Comput.Phys.Commun. 185 1773–1790 (Preprint 1309.7223)
  * [22] Calafiura P, Catmore J, Costanzo D and Di Girolamo A 2020 ATLAS HL-LHC Computing Conceptual Design Report Tech. Rep. CERN-LHCC-2020-015. LHCC-G-178 CERN Geneva URL https://cds.cern.ch/record/2729668
  * [23] Höche S, Prestel S and Schulz H 2019 Phys. Rev. D 100 014024 (Preprint 1905.05120)
  * [24] Bothmann E, Giele W, Hoeche S, Isaacson J and Knobbe M 2021 (Preprint 2106.06507)
  * [25] Carrazza S, Cruz-Martinez J, Rossi M and Zaro M 2021 Eur. Phys. J. C 81 656 (Preprint 2106.10279)
  * [26] Valassi A, Roiser S, Mattelaer O and Hageboeck S 2021 EPJ Web Conf. 251 03045 (Preprint 2106.12631)
  * [27] Berends F A and Giele W T 1988 Nucl. Phys. B306 759 URL http://inspirehep.net/search?j=NUPHA,B306,759
  * [28] Badger S, Biedermann B, Hackl L, Plefka J, Schuster T and Uwer P 2013 Phys. Rev. D87 034011 (Preprint 1206.2381)
  * [29] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G S, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y and Zheng X 2015 TensorFlow: Large-scale machine learning on heterogeneous systems software available from tensorflow.org URL https://www.tensorflow.org/
  * [30] Danziger K, Janßen T, Schumann S and Siegert F 2021 (Preprint 2109.11964)
  * [31] Maître D and Truong H 2021 JHEP 11 066 (Preprint 2107.06625)
  * [32] Catani S and Seymour M H 1997 Nucl. Phys. B485 291–419 (Preprint hep-ph/9605323) URL http://inspirehep.net/search?p=hep-ph/9605323
  * [33] Bendavid J 2017 (Preprint 1707.00028)
  * [34] Otten S, Caron S, de Swart W, van Beekveld M, Hendriks L, van Leeuwen C, Podareanu D, Ruiz de Austri R and Verheyen R 2021 Nature Commun. 12 2985 (Preprint 1901.00875)
  * [35] Hashemi B, Amin N, Datta K, Olivito D and Pierini M 2019 (Preprint 1901.05282)
  * [36] Di Sipio R, Faucci Giannelli M, Ketabchi Haghighat S and Palazzo S 2019 JHEP 08 110 (Preprint 1903.02433)
  * [37] Butter A, Plehn T and Winterhalder R 2019 SciPost Phys. 7 075 (Preprint 1907.03764)
  * [38] Carrazza S and Dreyer F A 2019 Eur. Phys. J. C 79 979 (Preprint 1909.01359)
  * [39] Ahdida C et al. (SHiP) 2019 JINST 14 P11028 (Preprint 1909.04451)
  * [40] Butter A, Plehn T and Winterhalder R 2019 (Preprint 1912.08824)
  * [41] Butter A, Diefenbacher S, Kasieczka G, Nachman B and Plehn T 2021 SciPost Phys. 10 139 (Preprint 2008.06545)
  * [42] Butter A and Plehn T 2020 (Preprint 2008.08558)
  * [43] Klimek M D and Perelstein M 2020 SciPost Phys. 9 053 (Preprint 1810.11509)
  * [44] Bothmann E, Janßen T, Knobbe M, Schmale T and Schumann S 2020 SciPost Phys. 8 069 (Preprint 2001.05478)
  * [45] Gao C, Isaacson J and Krause C 2020 Mach. Learn. Sci. Tech. 1 045023 (Preprint 2001.05486)
  * [46] Gao C, Höche S, Isaacson J, Krause C and Schulz H 2020 Phys. Rev. D 101 076002 (Preprint 2001.10028)
  * [47] Chen I K, Klimek M D and Perelstein M 2021 SciPost Phys. 10 023 (Preprint 2009.07819)
  * [48] Stienen B and Verheyen R 2021 SciPost Phys. 10 038 (Preprint 2011.13445)
  * [49] Andersen J R, Gütschow C, Maier A and Prestel S 2020 Eur. Phys. J. C 80 1007 (Preprint 2005.09375)
  * [50] Nachman B and Thaler J 2020 Phys. Rev. D 102 076004 (Preprint 2007.11586)
  * [51] Andersen J R and Maier A 2021 (Preprint 2109.07851)
  * [52] Danziger K, Höche S and Siegert F 2021 (Preprint 2110.15211)