# PUFA-GAN: A Frequency-Aware Generative Adversarial Network for 3D Point
Cloud Upsampling

Hao Liu, Hui Yuan, Junhui Hou, Raouf Hamzaoui, Wei Gao  This work was
supported in part by the National Natural Science Foundation of China under
Grants 62172259 and 61871342, the Taishan Scholar Project of Shandong Province
(tsqn202103001), the Central Guidance Fund for Local Science and Technology
Development of Shandong Province, under Grant YDZX2021002, the Hong Kong RGC
under Grants CityU11202320, and the OPPO Research Fund. Hao Liu and Hui Yuan
are with the School of Control Science and Engineering, Shandong University,
Jinan, 250061, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Junhui Hou is with the
Department of Computer Science, City University of Hong Kong, Kowloon, Hong
Kong, China.
E-mail<EMAIL_ADDRESS>Raouf Hamzaoui is with the School of Engineering
and Sustainable Development, De Montfort University, Leicester, UK.
E-mail<EMAIL_ADDRESS>Wei Gao is with the School of Electronic and
Computer Engineering, Peking University, and also with Peng Cheng Laboratory,
China.
E-mail<EMAIL_ADDRESS>Hao Liu and Hui Yuan contributed equally to this
work; Hui Yuan and Junhui Hou are corresponding authors.

###### Abstract

We propose a generative adversarial network for point cloud upsampling, which
can not only make the upsampled points evenly distributed on the underlying
surface but also efficiently generate clean high frequency regions. The
generator of our network includes a dynamic graph hierarchical residual
aggregation unit and a hierarchical residual aggregation unit for point
feature extraction and upsampling, respectively. The former extracts
multiscale point-wise descriptive features, while the latter captures rich
feature details with hierarchical residuals. To generate neat edges, our
discriminator uses a graph filter to extract and retain high frequency points.
The generated high resolution point cloud and corresponding high frequency
points help the discriminator learn the global and high frequency properties
of the point cloud. We also propose an identity distribution loss function to
make sure that the upsampled points remain on the underlying surface of the
input low resolution point cloud. To assess the regularity of the upsampled
points in high frequency regions, we introduce two evaluation metrics.
Objective and subjective results demonstrate that the visual quality of the
upsampled point clouds generated by our method is better than that of the
state-of-the-art methods.

###### Index Terms:

Point cloud upsampling, graph filter, deep learning.

##  I Introduction

With the progress of three-dimensional (3D) capture technology, 3D point
clouds [1, 2, 3] have become widely used in many fields, such as immersive
communication [4], autonomous navigation [5], etc. Point clouds captured by
existing devices (e.g., LiDAR or depth cameras) are usually non-uniform and
sparse, which negatively affects their subsequent processing. To address these
issues, we propose a deep learning-based point cloud super-resolution method.
Our method aims at solving the following ill-posed problem: given a non-
uniform, sparse, noisy, low-resolution (LR) point cloud, find a uniform,
dense, noise-free high-resolution (HR) point cloud which is consistent with
the geometry distribution of the corresponding LR point cloud.

The mainstream point cloud upsampling methods can be roughly divided into two
categories: optimization-based and deep learning-based approaches. Early
optimization-based point cloud upsampling methods [6, 7, 8, 9, 10, 11, 12, 13]
rely more on the prior geometry knowledge of the input point clouds.
Therefore, they can efficiently upsample flat regions. But for sharp regions
(corner and edge) with a complex geometry distribution, their performance is
severely limited. The existing deep learning-based methods [14, 15, 16, 17,
18, 19, 20, 21] exploit convolutional neural networks. They mainly transfer
the idea of image super-resolution to point clouds. However, all those methods
mainly focus on how to generate a uniform HR point cloud and ignore the
fundamental problem that the HR point cloud contains some noisy points. This
severely affects the reconstruction quality, especially at the corner of the
object (Fig. 1).

In this paper, we present a novel generative adversarial network (GAN) [22]
for point cloud upsampling, called PUFA-GAN. In contrast to existing methods,
our network addresses the important problem that the upsampled point cloud
usually contains apparent noise. This noise affects the regularity of the
generated shape and damages the local details of the generated HR point cloud
(Fig. 1). To this end, we explicitly extract high frequency (HF) points (i.e.,
points extracted by a high-pass graph filter [see Section 3.3.1], such as
points on a sharp corner or edge and noisy points) of the upsampled point
cloud and send them to the discriminator directly so that the network can
learn to recognize the implicit HF features and output a uniform upsampled
point cloud with clean HF regions (i.e., the regions where the HF points are
located).

Figure 1: Upsampling ($\bm{\times}$4) the non-uniform and sparse point cloud
Elephant with (a) EAR [9] (b) PU-Net [14], (c) MPU [15], (d) PU-Geo [17], (e)
PU-GAN [16], (f) PU-GCN [19], (g) Dis-PU [20] and (h) PUFA-GAN.

GAN is hard to train and reach the Nash equilibrium with the generative
learning. To balance the generation ability of the generator and the judgment
accuracy of the discriminator, we design an efficient GAN to fully explore the
global and local potential geometry characteristics of the point cloud.
Specifically, we propose a dynamic graph hierarchical residual aggregation
(DGHRA) unit and a hierarchical residual aggregation (HRA) unit for feature
extraction and upsampling in the generator. DGHRA captures multiscale point-
wise features by dynamically constructing a neighborhood graph, while HRA uses
hierarchical residuals to capture the ample detail representation of the
upsampled features, which generates natural geometric textures. Exploiting
graph signal processing [23], we leverage a graph filter (GF) [24] to extract
HF geometry information (i.e., HF points) accurately. Next, we feed the HF
points and the whole upsampled point cloud into the proposed frequency-aware
discriminator (i.e., a global discriminator head and an HF discriminator
head), respectively. In this way, the discriminator can learn both the global
and HF features implicitly to better approximate the real target distribution.
Finally, we devise an efficient identity distribution loss function to
regularize the statistical distribution of the upsampled HR point cloud to be
consistent with that of the input LR point cloud.

In summary, the key contributions of this paper are as follows.

  * •

We propose DGHRA and HRA units for point cloud feature extraction and
expansion in the generator, respectively, to extract rich details by capturing
hierarchical residual features.

  * •

We extract HF points of the upsampled point cloud with a GF and propose a
frequency-aware discriminator to better distinguish real HF points from fake
ones produced by the generator. To the best of our knowledge, we are the first
to apply a GF to a point cloud upsampling task.

  * •

We introduce a simple and effective identity distribution loss to constrain
the geometry distribution of the upsampled HR point cloud to be equal to that
of the input LR point cloud.

  * •

We propose two evaluation metrics (HF_CD and HF_HD) to assess the regularity
of the upsampled point clouds in HF regions.

The remainder of this paper is organized as follows. Section II briefly
reviews related work. Section III describes the proposed method in detail.
Experimental results and conclusions are given in Section IV and V,
respectively.

##  II RELATED WORK

Optimization-based upsampling methods. In their pioneering work, Alexa et al.
[6] generated a Voronoi diagram on the moving least squares surface and
interpolated points at the vertices of the corresponding diagram. A novel
locally optimal projection (LOP) operator was introduced by Lipman et al. [7]
to robustly resample points and reconstruct surfaces. Preiner et al. [8]
proposed a continuous LOP for fast surface reconstruction. Huang et al. [9]
presented a progressive edge-aware (EAR) point cloud upsampling method, which
first collects the points far away from the edge and then gradually upsamples
the points to approach the edge singularity. By jointly optimizing the surface
and inner points, Wu et al. [10] proposed a deep points consolidation method
to fill local holes caused by missing data. Recently, Dinesh et al. [11]
proposed a model-based point could super-resolution method via graph total
variation on surface normal. In [12], the same authors also proposed a fast
graph total variation method for color point cloud super-resolution. Borges et
al. [13] proposed an efficient point cloud super-resolution via self-
similarity of the point cloud.

Figure 2: Architecture of PUFA-GAN. The proposed network consists of two
parts: a generator and a discriminator. For an arbitrary LR input point cloud
patch with $N$ points, the generator aims to output the corresponding HR point
cloud patches with $rN$ points ($r$ is the upsampling ratio), and the
discriminator determines where the HR point cloud patches come from (i.e., the
output of the generator or the ground truth). Specifically, the feature
extraction module first learns point-wise descriptive features. Then the
upsampled features with rich details can be obtained by the feature expansion
module. Finally, the geometry generation module maps the upsampled features
back to the geometry domain and outputs HR point cloud patches. The proposed
frequency-aware discriminator guides the generator to produce more realistic
upsampling point clouds by global and HF contexts.

Deep Learning-based upsampling methods. PointNet [25], a pioneering point-
based deep learning model, uses the permutation invariance of a point cloud to
extract global features by using multilayer perceptrons (MLPs). However, MLPs
cannot capture local features well. PointNet++ [26] further uses a set
abstraction layer to efficiently collect local presentation by the ball query
method. Compared with optimization-based upsampling methods, data-driven
methods have shown impressive performance. EC-Net [27] is a deep learning-
based upsampling method that aims at generating points near the edges and on
the underlying surface, while preserving the sharp features. Yu et al. [14]
introduced an upsampling network called PU-Net which extracts multiscale
features by PointNet++ and expands them with shared MLPs. Later, Wang et al.
[15] proposed a multiscale upsampling method (MPU), which progressively
achieves efficient upsampling operations by controlling the upsampling factor
($\times$2) for hierarchical detail reconstruction. But the progressive
feature increased the complexity of the network and had potential error
accumulation. Qian et al. [17] used a local surface parameterization to
represent a 3D point cloud in a 2D parametric domain and then used the normal
vectors to generate the upsampled 3D point cloud adaptively from the 2D
parametric surface. However, the method requires the normal vectors which are
usually not available from the raw point clouds directly. In [18], the same
authors proposed an adaptive point cloud interpolation method based on linear
estimation theory. Qian et al. [19] proposed a graph convolutional network
(GCN)-based point cloud upsampling method (PU-GCN) which includes an inception
dense GCN feature extraction and a node shuffle-based feature expansion
module. Li et al. [20] proposed two cascaded sub-networks (Dis-PU) for point
cloud upsampling. In this method, the dense generator first produces a coarse
upsampled point cloud. Then, the spatial refiner restores a fine upsampled
point cloud from the coarse one by predicting the point-wise spatial offsets.
Liu et al. [21] proposed a progressive geometry upsampling method with
adversarial learning. Metzer et al. [28] proposed a point cloud consolidation
method (Self-Net) with self-supervision learning[29, 30], which uses the
curvature and density of the point cloud to generate sharp edge points or
points in sparse regions. Although both our method and Self-Net use HF points,
the optimization objectives are different. Self-Net pays more attention to
geometry sharpness by forcing more points to be close to the HF regions. This
may affect the uniformity of the upsampled point cloud, especially for a
sparse point cloud. In contrast, PUFA-GAN only limits the noise in HF regions
by distinguishing HF points, and does not push points to the HF regions.
Hence, PUFA-GAN not only maintains the global uniformity, but also generates
clean edges in HF regions.

Li et al. [16] proposed an adversarial learning network (PU-GAN) for point
cloud upsampling. They introduced an efficient global discriminator to
accurately distinguish the upsampled point clouds of the generator from the
ground truth. They also promoted the generator to produce upsampled point
clouds that are consistent with the distribution of the ground truth through
adversarial learning. However, their simple global discriminator tends to
ignore the subtle description features of the upsampled point cloud, leading
to local noise and detail distortions. Compared to PU-GAN, i) we propose a
novel DGHRA-based densely connected feature extraction and a cascaded HRA-
based feature expansion module, which makes full use of multi-scale residual
information to obtain more expressive features; ii) we propose an efficient
frequency-aware discriminator to suppress noise in HF regions; iii) in
addition to using the general loss functions in PU-GAN, we propose an identity
distribution loss function which reduces the solution space of the network and
accelerates its convergence.

Figure 3: Proposed DGHRA and HRA units.

##  III METHOD

###  III-A Overview

In general, a point cloud is a set of points given by their 3D Cartesian
coordinates (geometry information), together with a number of attributes such
as color and reflectance. In this paper, we only consider the geometry
information. Thus, a point cloud consisting of $N$ points can be represented
by a matrix ${P}\in$ ${\mathbb{R}^{{N}\times 3}}$. Given a non-uniform, sparse
point cloud ${P_{ori}}\in$ ${\mathbb{R}^{{N}\times 3}}$, the aim of the
proposed method is to output a uniform, dense upsampled point cloud
${P_{up}}\in$ ${\mathbb{R}^{{rN}\times 3}}$, where $r$ denotes the upsampling
ratio. Simultaneously, ${P_{up}}$ should satisfy the following conditions: 1)
${P_{up}}$ can generate the same underlying surface as ${P_{ori}}$. 2) the
points in ${P_{up}}$ are evenly distributed on the underlying surface. 3)
${P_{up}}$ should contain rich geometry details, especially in the HF regions.

Recently, GANs have been very successful in 2D image processing [31, 32] and
3D object generation [33, 34]. For upsampling, we propose PUFA-GAN as shown in
Fig. 2, where the generator produces an upsampled point cloud ${P_{up}}$, and
the frequency-aware discriminator is used to identify a fake (the output of
the generator) ${P_{up}}$ efficiently. Note that PUFA-GAN takes patches
instead of the whole point cloud as the processing unit. In the following,
unless otherwise specified, the mentioned point clouds (e.g., ${P_{ori}}$,
${P_{up}}$) denote patches.

The remainder of this section is organized as follows. The detailed framework
of the proposed generator is described in sub-section 3.2. The novel
frequency-aware discriminator is proposed in sub-section 3.3. Finally, a joint
loss function for end-to-end training is introduced in sub-section 3.4.

###  III-B Generator

The proposed generator includes three modules: feature extraction, feature
expansion, and geometry generation, as shown in Fig. 2 (top).

####  III-B1 Feature Extraction

The feature extraction module extracts point-wise representative features
$F\in$ ${\mathbb{R}^{{N}\times C}}$ ($C$ is the number of features) of the
input point cloud ${P_{ori}}\in$ ${\mathbb{R}^{{N}\times 3}}$. DenseNet [35]
has strong feature extraction capabilities due to its dense connections. For
each network layer, the features of all preceding layers are used as inputs,
and its own features are also used as inputs to all subsequent layers.
Inspired by this idea, we propose a densely connected feature extraction
module to connect all previous multiscale features (Fig. 2 top left).
Specifically, each feature extraction block (FEB) consists of the
concatenation of one DGHRA and several HRAs. The details of DGHRA and HRA are
as follows.

DGHRA and HRA Units. How to extract expressive point-wise features is a
critical task for the network. DGCNN [36] can effectively process point clouds
through a dynamic neighborhood feature graph for “local” feature collection.
However, the original DGCNN cannot discover fine differences between features.
The residual features [37] have been shown to be very effective for super-
resolution tasks [38, 39]. Classical stacked residual blocks can efficiently
extract subtle features dissimilarity. But the residual features are fused
with the identity features before propagation to the next residual block. As a
result, the subsequent residual blocks cannot explore pure residual features
individually, which deteriorates the network performance. RFANet [40]
effectively alleviates the above problem by explicitly extracting pure
residual features.

Inspired by DGCNN and RFANet, we present a DGHRA unit for feature extraction.
The unit can be regarded as a combination of dynamic graph construction (DGC)
and HRA. It not only constructs a multiscale feature graph based on
neighborhood information dynamically but also captures hierarchical pure
residuals from each residual block for fine grained representation.

As shown in Fig. 3, DGC first forms each point feature ${\hat{f}_{i}}$ from a
local neighborhood that is computed dynamically via k-nearest neighbor (KNN)
search based on feature similarity, i.e.,

${\hat{f}_{i}^{j}}=[{f_{i}}\odot({f_{i}}-{f_{j}})],$ (1)

where ${\hat{f}_{i}^{j}}$ is the feature gradient of the $i$-th current point
and its $j$-th feature neighbor, ${f_{i}}$ is the $i$-th current point
feature, ${f_{j}}$ is the $j$-th feature neighbor of ${f_{i}}$, and $\odot$ is
the concatenation operation.

Then the proposed HRA module is applied to obtain more elaborate features for
each point. Specifically, HRA includes four residual blocks (RBs) (Fig. 4 (a))
and one gate unit (Fig. 4 (b)). It independently extracts the pure residual
features of each RB, which are concatenated together with the output of the
last RB. Next, a gate unit is used to recalibrate the fused hierarchical
residuals before element-wise addition with the identity feature. In this way,
the role of the residuals is brought into full play, which allows the network
to fully explore the nuances among features. Finally, a max pooling operation
is applied to achieve point-wise feature sublimation by finding the maximum
point-wise feature response of all nearest neighbors. It is worth noting that
we also use cascaded HRA units in the feature expansion module of the
generator.

The RB can help preserve more details and accelerate network convergence. As
shown in Fig. 4 (a), it contains three linear layers with a rectified linear
unit (ReLU). To build a lightweight network, we use bottleneck layers to
reduce the number of trainable convolution kernels.

The gate unit recalibrates the fused features. The proposed gate unit is
composed of SE-Net [41] and a linear layer. Because there are many
redundancies among fused hierarchical residuals, we use the efficient channel
attention mechanism (SE-Net, see Fig. 4 (b)) to learn global information by
global average pooling to selectively stress significant features and suppress
invalid ones. Then, we use a linear layer to refine and align the channels of
fused residuals for subsequent element-wise addition.

Figure 4: Overview of the proposed residual block (RB) (a) and gate unit (b),
where Linear (C1, C2) indicates a linear layer, and C1 and C2 are the
dimension of the input and output feature vector.

####  III-B2 Feature Expansion

Feature expansion aims to expand the feature vector $F\in$
${\mathbb{R}^{{N}\times C}}$ to the upsampled feature vector $\hat{F}\in$
${\mathbb{R}^{{rN}\times C^{\prime}}}$, where $C^{\prime}$ is the number of
features in the upsampled points. It uses the local smoothing property of the
point cloud to generate rich upsampled features. Specifically, we first
duplicate feature vector $F\in$ ${\mathbb{R}^{{N}\times C}}$ $r$ times to get
the upsampled feature vector $\tilde{F}\in$ ${\mathbb{R}^{{rN}\times C}}$. To
increase the diversity of $\tilde{F}$, we use the 2D grid representation
strategy proposed in FoldingNet [42], i.e., we attach a 2D vector to
$\tilde{F}$ to make each upsampled point of $\tilde{F}$ unique, as shown in
Fig. 2 (top). Next, we send the upsampled feature vector with dimension
$rN\times(C+2)$ to the proposed cascaded HRA units (Fig. 2 (top)) to obtain
the upsampled fine feature vector $\hat{F}\in$ ${\mathbb{R}^{{rN}\times
C^{\prime}}}$.

####  III-B3 Geometry Generation

The geometry reconstruction module uses several MLPs to map $\hat{F}$ from the
implicit feature domain back to the explicit geometry domain (i.e., upsampled
3D geometry coordinates ${P_{up}}\in$ ${\mathbb{R}^{{rN}\times 3}}$ ). To
ensure the uniformity of the upsampled point cloud, we follow the strategy in
[16]. First, the feature expansion module generates $(r+2)N\times 3$ point
features. Then, farthest point sampling (FPS) is used to obtain the final
upsampled point cloud with size $rN\times 3$.

###  III-C Frequency-Aware Discriminator

The role of the discriminator is to identify where ${P_{up}}$ comes from,
i.e., the output of the generator (Fake) or the ground truth (True)? The
discriminator can force the generator to produce ${P_{up}}$ which approximates
the latent distribution of the target point cloud. To identify ${P_{up}}$
efficiently, the work in [16] adopts a basic encoder network [43] as the
backbone of the discriminator. However, when the local density of the input
point cloud is large (i.e., the local feature similarity is very high), a
single global discriminator cannot recognize the local details well,
especially for the HF regions which usually contain noise. In other words, it
allows point clouds with local noise to easily fool the discriminator and then
distracts the generator when generating the HF points. Therefore, the
generator will tend to produce irregular HF regions with noise, which severely
affects the upsampling performance (Fig. 1(e)).

Figure 5: High frequency points extraction by graph filter. The top row
corresponds to original point clouds, and the bottom row shows the high
frequency points after graph filtering.

To obtain a uniform point cloud with faithful HF regions, we propose a novel
frequency-aware discriminator, as shown in Fig. 2 (bottom). Specifically, for
each input ${P_{up}}\in$ ${\mathbb{R}^{{rN}\times 3}}$, a graph filter is
first used to extract the HF points ${P_{b}}\in$ ${\mathbb{R}^{{M}\times 3}}$
(see Fig. 5) explicitly, where $M(M<rN)$ denotes the number of HF points.
Then, ${P_{up}}$ and ${P_{b}}$ are fed into the proposed global and HF
discriminator, respectively, for more accurate discriminative results. The
former considers the global context for better skeleton recognition, while the
latter suppresses noise in HF regions by mining the differences of local HF
features.

####  III-C1 HF Points Extraction Using Graph Filter

Graph construction. For an arbitrary point cloud $\bm{P}\in$
${\mathbb{R}^{{K}\times 3}}$ (with $K$ points), a graph $\bm{G}$ can be
constructed as follows

$\bm{G}=\\{\bm{X}=\\{{x_{1}},{x_{2}},...,{x_{K}}\\},\bm{A}\\},$ (2)

where $\bm{X}$ is the set of nodes, node ${x_{k}}$ ($k=1,...,K$) refers to the
$k$-th point in the point cloud, and $\bm{A}\in$ ${\mathbb{R}^{{K}\times K}}$
is the adjacency matrix indicating the edge dependencies between the points.
We define $\bm{A}$ as follows:

${A_{i,j}}=\left\\{{\begin{array}[]{*{20}{c}}{f({{\left\|{{x_{i}}-{x_{j}}}\right\|}_{2}}),{\rm{}}if{{\left\|{{x_{i}}-{x_{j}}}\right\|}_{2}}<\varepsilon}\\\
{\qquad 0,\qquad\qquad otherwise,}\end{array}}\right.$ (3)

where $f(\cdot)$ is a Gaussian function, and $\varepsilon$ is a threshold.
Note that $\bm{A}$ is a symmetric matrix.

Graph filter theory. Let the adjacency matrix $\bm{A}$ be a graph shift
operator, i.e., it converts a signal $\bm{s}\in$ ${\mathbb{R}^{{K}\times 1}}$
(an attribute or feature of point clouds) into a shifted signal
$\bm{y}=\bm{A}\bm{s}$. The linear, shift-invariant all-pass graph filter
$\bm{H(A)}$ is defined as a polynomial of the graph shift operator [24], i.e.,

$\bm{H(A)}=\sum\nolimits_{l=0}^{L-1}{{h_{l}}{\bm{A}^{l}}={h_{0}}I+{h_{1}}\bm{A}+\cdot\cdot\cdot+{h_{L-1}}{\bm{A}^{L-1}}},$
(4)

where $L$ is the number of filter taps, and ${h_{l}}$ is the $l$-th filter
coefficient. Hence, the output of the all-pass graph filter can be further
written as $\bm{\tilde{y}}=\bm{H(A)}\bm{s}$. To explore the frequency
representation of graph filters, the following eigen decomposition [23] is
considered:

$\bm{H(A)}=\bm{V}\bm{H(\Lambda)}\bm{V^{-1}},$ (5)

where $\bm{V}\in$ ${\mathbb{R}^{{K}\times K}}$ is the eigenvector matrix of
$\bm{H(A)}$, $\bm{H(\Lambda)}\in$ ${\mathbb{R}^{{K}\times K}}$ is the diagonal
eigenvalue matrix with ordered eigenvalues
${\lambda_{1}}\geq{\lambda_{2}}\ldots\geq{\lambda_{K}}$ (there is a one-to-one
correspondence between the ordered eigenvalues and the graph frequencies,
i.e., the larger the eigenvalue, the lower the frequency, and vice versa
[23]), and $\bm{V^{-1}}\in$ ${\mathbb{R}^{{K}\times K}}$ is the corresponding
graph Fourier transform (GFT) basis. Therefore, the all-pass graph filter can
be described as:

$\begin{array}[]{l}\bm{\tilde{y}}=\bm{H(A)s}\\\ \ \ \
=\bm{VH(\Lambda)}\underbrace{{\bm{V^{-1}}}\bm{s}}_{GFT}\\\ \ \ \
=\bm{V}\underbrace{diag[H({\lambda_{1}}),H({\lambda_{2}}),...,H({\lambda_{K}})]\bm{\tilde{s}}}_{Filtering\;in\;graph\;Fourier\;domain}\\\
\ \ \
=\underbrace{\bm{V}diag{{[H({\lambda_{1}}){{\tilde{s}}_{1}},H({\lambda_{2}}){{\tilde{s}}_{2}},...,H({\lambda_{K}}){{\tilde{s}}_{K}}]}^{T}}}_{Inverse\;GFT},\end{array}$
(6)

i.e., the input signal $\bm{s}$ is first transformed into the graph Fourier
domain signal $\bm{\tilde{s}}$ (i.e., $\bm{V^{-1}s}$) by GFT. Then,
$\bm{\tilde{s}}$ is filtered by the frequency response
${diag[H({\lambda_{1}}),H({\lambda_{2}}),...,H({\lambda_{K}})]}$ of
$\bm{H(A)}$, where
$H({\lambda_{k}})=\sum\nolimits_{l}^{L-1}{{h_{l}}\lambda_{k}^{l}}$ [23] is the
frequency response of ${\lambda_{k}}$. Finally, an inverse GFT transforms the
filtered signal back into the graph node domain.

Extracting HF points by graph filter. Low frequency components of the point
cloud usually correspond to smooth regions (flat areas), while high frequency
components correspond to HF regions (edge, corner). In this paper, we aim to
find a filter on the graph $\bm{{G}=\\{X,{A}\\}}$ to precisely extract HF
points for an arbitrary point cloud. Therefore, we adopt a simple yet
effective Haar-like high-pass graph filter [44]. The polynomial of the graph
filter can be expressed as

$\begin{array}[]{l}\bm{H({A})}=\bm{I-{A}}\\\ \qquad\ \
=\bm{VH({\Lambda}){V^{-1}}}\\\ \qquad\ \
=\bm{V}\left[{\begin{array}[]{*{20}{c}}{1-{{\lambda}_{1}}}&0&\cdots&0\\\
0&{1-{{\lambda}_{2}}}&{\cdot\cdot\cdot}&0\\\ \vdots&\vdots&\ddots&\vdots\\\
0&0&{\cdot\cdot\cdot}&{1-{{\lambda}_{K}}}\end{array}}\right]\bm{{V^{-1}}},\end{array}$
(7)

where the frequency response of the graph filter is
$H({\lambda_{i}})=1-{\lambda_{i}}$. Because the eigenvalues of $\bm{A}$ are
arranged in descending order (i.e.,
${\lambda_{1}}\geq{\lambda_{2}}\ldots\geq{\lambda_{K}}$), $1-{\lambda_{i}}\leq
1-{\lambda_{i+1}}$. Therefore, by setting the smaller eigenvalues of
$\bm{H({A})}$ to zero, the low frequency parts are suppressed and high
frequency parts are amplified after applying the high-pass graph filter.

Given $\bm{P}\in$ ${\mathbb{R}^{{K}\times 3}}$, we can apply $\bm{H({A})}$ to
get a filtered point cloud $\bm{H({A})P}$. Specifically, because
$\bm{H({A})=I-{A}}$, the response of the $i$-th point $x_{i}$ of the point
cloud $\bm{P}$ can be formulated as:

$\bm{(H({A})P)}_{i}={x_{i}}-{\sum\nolimits_{j=1}^{K}{A}_{i,j}}{x_{j}},$ (8)

In fact, $\bm{(H({A})P)}_{i}$ is equivalent to the feature discrepancy between
the current point and the linear combination of its neighbors, which reflects
the variation between the current point and its neighbors. For example, when
the variation becomes large, the Euclidean distance between the current point
and its neighbors increases, meaning that the current point has drastically
mutated and can be regarded as an HF point.

Eventually, we can obtain the variation amplitude of each point $x_{i}$
($i=1,...,K$) by calculating the $L2$-norm of $\bm{(H({A})P)}_{i}$ in Eq.(8).
A point with a larger $L2$-norm has more HF local variations and is an HF
point candidate. By reordering the rows of the point cloud $\bm{P}\in$
${\mathbb{R}^{{K}\times 3}}$ in descending order according to the $L2$-norm
from Eq.(8), we select the top $M$ (empirically set to 256) points as HF point
cloud $\bm{P_{b}}\in$ ${\mathbb{R}^{{M}\times 3}}$.

####  III-C2 Discriminator structure

As shown in Fig. 2 (bottom), the proposed frequency-aware discriminator
includes two heads (a global and an HF head), where each head has the same
backbone [16]. In particular, the input goes through two MLP layers (32 and 64
output feature dimensions) with max pooling to get the global (resp. HF)
response vector, and the duplicated global (resp. HF) response vector is
concatenated with the output of the previous MLPs for feature enhancement.
Then, a self-attention module [45] is applied to strengthen the feature
interaction among points. Again, two MLP layers (128 and 256 output feature
dimensions) with max pooling are used to acquire the corresponding global
(resp. HF) feature response vector. Finally, the feature response of each head
is concatenated to form the final discriminant vector, and the confidence
value of the discriminator is generated by three fully connected (FC) layers.
Note that we assume the confidence value of the ground truth (True) is close
to 1 and the confidence value of the output (Fake) of the generator is close
to 0.

###  III-D Loss Functions

We propose a joint loss to train PUFA-GAN in an end-to-end fashion.

Adversarial loss. We adopt the least-squared adversarial loss [46] to train
the generator $G$ and discriminator $D$,

${L_{gen}}({P_{up}})=\frac{1}{2}{(D({P_{up}})-1)^{2}},$ (9)
${L_{dis}}({P_{up}},{P_{T}})=\frac{1}{2}(D{({P_{up}})^{2}}+{(D({P_{T}})-1)^{2}}),$
(10)

where ${P_{up}}$ is the upsampled output of generator $G$, ${P_{T}}$ is the
ground truth point cloud which has the same number of points as ${P_{up}}$.
$D({P_{up}})$ and $D({P_{T}})$ are the predictive confidence value of the
discriminator $D$. Eq.(9) can be interpreted as $G$ produces ${P_{up}}$ to
confuse $D$ as much as possible by minimizing ${L_{gen}}({P_{up}})$, while
Eq.(10) indicates that $D$ needs to learn to distinguish ${P_{up}}$ from
${P_{T}}$ accurately by minimizing ${L_{dis}}({P_{up}},{P_{T}})$.

Reconstruction loss. To make ${P_{up}}$ fit the underlying surface of
${P_{T}}$ better, we take the Earth Mover’s distance (EMD) [14] as
reconstruction loss ${L_{rec}}$ :

${L_{rec}}=\mathop{\min}\limits_{\phi:{P_{up}}\to{P_{T}}}{\sum\nolimits_{{p_{i}}\in{P_{up}}}{\left\|{{p_{i}}-\phi({p_{i}})}\right\|}_{2}},$
(11)

where ${\phi}$ is a bijection, which encourages point cloud ${P_{up}}$ located
close to the underlying surface of ${P_{T}}$. The optimal bijection
$\phi:{P_{up}}\to{P_{T}}$ is unique and invariant under infinitesimal movement
of the points [47], i.e., for each ${p_{i}}$ in ${P_{up}}$, the optimal
bijection finds the unique point $\phi({p_{i}})$ in ${P_{T}}$, which minimizes
the sum of the distances from all ${p_{i}}$ to $\phi({p_{i}})$. As the
computation of the optimal bijection is computationally expensive, we use the
approximation strategy in [47] to find a sub-optimal solution in a fixed given
time.

Uniform loss [16]. $L_{uni}$ aims to make the points uniformly distributed on
the surface of ${P_{up}}$. It contains two parts: global uniformity
${U_{imblance}}({S_{i}})$ and local uniformity ${U_{cluster}}({S_{i}})$.

${L_{uni}}=\sum\nolimits_{i=1}^{T}{{U_{imblance}}({S_{i}})\cdot{U_{cluster}}({S_{i}})},$
(12)

where $S_{i}$, $i=1,2,..,T$, is a ball queried point subset with radius
$r_{q}$, and $T$ is the number of seed points collected from $P_{up}$ by FPS.
The global uniformity ${U_{imblance}}({S_{i}})$ can be represented as

${U_{imblance}}({S_{i}})=\frac{{{{(\left|{{S_{i}}}\right|-\hat{n})}^{2}}}}{{\hat{n}}},$
(13)

where ${\hat{n}}=rN\times r_{q}^{2}$ [16] ($r$ is the upsampling ratio)
indicates the expected number of points in $S_{i}$. The global uniform
regulation ${U_{imblance}}({S_{i}})$ constrains the number of points in each
$S_{i}$ to be consistent. The local uniformity ${U_{cluster}}({S_{i}})$ can be
written as

${U_{cluster}}({S_{i}})=\sum\nolimits_{j}^{\left|{{S_{i}}}\right|}{\frac{{{{({d_{i,j}}-\hat{d})}^{2}}}}{{\hat{d}}}},$
(14)

where
$\hat{d}=\sqrt{\frac{{2\pi{r_{q}}^{2}}}{{\sqrt{3}\left|{{S_{i}}}\right|}}}$
[16] refers to the expected uniform distribution distance of points, and
${d_{i,j}}$ is the point-to-nearest-neighbor distance of the $j$-th point in
$S_{i}$. ${U_{cluster}}({S_{i}})$ forces each point in $S_{i}$ to have a
similar distance from its nearest neighbor, thus ensuring local uniformity.

Identity distribution loss. Point cloud upsampling is essentially an ill-posed
problem; therefore, the huge solution space makes it difficult for the network
to converge to a global optimum.

Inspired by [48], we propose an identity distribution loss $L_{id}$, which
reduces the potential solution space by mapping the HR output ${P_{up}}\in$
${\mathbb{R}^{{rN}\times 3}}$ back to the corresponding LR input
${P_{ori}}\in$ ${\mathbb{R}^{{N}\times 3}}$,

${L_{id}}=\mathop{\min}\limits_{\phi:{{\hat{P}}_{up}}\to{P_{ori}}}{\sum\nolimits_{{{\hat{p}}_{k}}\in{{\hat{P}}_{up}}}{\left\|{{{\hat{p}}_{k}}-\phi({{\hat{p}}_{k}})}\right\|}_{2}},$
(15)

where ${{\hat{P}}_{up}}\in$ ${\mathbb{R}^{{N}\times 3}}$ is an LR point cloud
obtained from ${P_{up}}$ by FPS, and ${\phi}$ denotes a bijection between the
subsets of equal size ${{\hat{P}}_{up}}$ and ${P_{ori}}$. $L_{id}$ not only
makes $P_{up}$ closer to ${P_{ori}}$ but also effectively reduces the possible
solution space of the network to find an optimum quickly.

Joint loss. In summary, we use the generator loss $L_{G}$ and discriminator
loss $L_{D}$ to train PUFA-GAN, where

${L_{G}}={w_{gen}}{L_{gen}}({P_{up}})+{w_{rec}}{L_{rec}}+{w_{uni}}{L_{uni}}+{w_{id}}{L_{id}},$
(16) ${L_{D}}={w_{dis}}{L_{dis}}({P_{up}},{P_{T}}),$ (17)

and ${w_{gen}}$, ${w_{rec}}$, ${w_{uni}}$, ${w_{id}}$ and ${w_{dis}}$ are
weights.

##  IV EXPERIMENTS

###  IV-A Experimental setup

Datasets. We used the PU147 dataset in PU-GAN, which includes 147 3D models
from the released datasets of PU-Net, MPU, and the Visionair repository [49],
containing objects of various shapes. Following the protocol in [16], 120
point clouds were randomly selected for training, while the rest were used for
testing. During the training, 200 patches were randomly cropped from each
training point cloud. In total, we collected 24000 patches for training.

Network details. For densely connected feature extraction, the network
included four FEBs, where each FEB consists of one DGHRA and two HRA units;
the number k of nearest neighbors was 16, and the dimension $C$ was 128. We
then used three HRA units connected in cascade in the feature expansion
module, and the dimension $C^{\prime}$ of upsampled feature of each HRA was
256. Moreover, we set $N$=256, and $\varepsilon$=0.5. Finally, we empirically
set the weight values ${w_{gen}}$, ${w_{rec}}$, ${w_{uni}}$, ${w_{id}}$ and
${w_{dis}}$ in the proposed joint loss to 1, 100, 10, 1, 1, respectively.

TABLE I: Average upsampling accuracy for the test dataset (27 point clouds
from [16]).

Method | CD | HD | P2F | Uniformity | HF_CD | HF_HD
---|---|---|---|---|---|---
| $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$
EAR [9] | 0.868 | 10.331 | 7.785 | 28.621 | 6.621 | 44.647
PU-Net [14] | 0.525 | 5.954 | 13.227 | 974.190 | 3.775 | 25.796
MPU [15] | 0.438 | 5.322 | 3.066 | 13.155 | 2.998 | 24.308
PU-Geo [17] | 0.405 | 5.338 | 3.432 | 9.259 | 3.111 | 24.778
PU-GAN [16] | 0.280 | 4.493 | 2.514 | 4.552 | 2.342 | 22.337
PU-GCN [19] | 0.323 | 4.172 | 2.926 | 9.560 | 2.978 | 25.152
Dis-PU [20] | 0.289 | 3.734 | 2.289 | 5.017 | 2.634 | 22.859
PUFA-GAN | 0.258 | 3.571 | 2.392 | 4.889 | 2.081 | 19.744

  * •

*Note that the lower the evaluation value, the better the performance. The bold red and blue values represent best and second best results, respectively.

Figure 6: Upsampling ($\bm{\times}$4) and surface reconstruction results
produced with all methods (c)-(j) from 2048 input points (a).

Training. To enhance the robustness of the network, we used the following data
enhancement strategies for input patches: point perturbation with Gaussian
noise and random rotation and scaling. We trained the proposed PUFA-GAN for
130 epochs with a batch size of 28. The Adam algorithm [50] with a two time-
scale update rule (TTUR) [51] was used, and the learning rate of the generator
and discriminator were 0.0009 and 0.0006, respectively. We implemented the
proposed method in the Tensorflow platform. A computer with an Intel Core i7
7820X processor, an NVIDIA Tesla V100 GPU, and 64GB memory was used to conduct
the experiments.

Testing. The patch-fusion strategy in [15] is used to generate the ultimate
upsampled point cloud. Specifically, a series of seed points are first picked
by FPS for each test sample. Then, for each seed point, a local patch with 256
points is generated. Next, all patches are fed into the generator and all the
upsampled patches are fused together. Finally, the ultimate upsampled point
cloud with the specified upsampling ratio is obtained by FPS from the fused
patches. However, subjective experiments showed (see Fig.1) that there may be
noise in HF regions due to the overlap between upsampled patches before patch
fusion. To alleviate this problem, we propose to use another efficient graph
filter to explicitly remove any potential high-frequency noise from each
upsampled patch before patch fusion, which is unique to our method. In this
way, we can effectively eliminate noise and retain more detailed textures in
HF regions.

###  IV-B Evaluation metrics

To assess the upsampling performance quantitatively, we used four common
evaluation metrics: Chamfer distance (CD) [47], Hausdorff distance (HD) [52],
Point-to-surface (P2F) distance [53] and Uniformity [16]. CD and HD measure
the shape difference between $P_{up}$ and $P_{T}$ with the Euclidean distance;
P2F assesses the deviation of $P_{up}$ from the underlying surface of $P_{T}$;
the Uniformity metric evaluates the uniformity of $P_{up}$, and we set
${r_{q}}^{2}=0.012$ (see Uniform loss in Section 3.4). Because existing
metrics cannot effectively appraise the difference in HF regions (which are
usually more prone to noise) between $P_{up}$ and $P_{T}$, we also propose two
metrics to evaluate the regularity in HF regions. Specifically, the HF points
of the upsampled point cloud $P_{up}$ and ground truth $P_{T}$ are first
extracted by GF. Then, the first metric, which we call HF_CD, is obtained by
calculating the CD of the corresponding HF points, i.e.,

$\displaystyle
HF\\_CD=\frac{1}{M}(\sum\nolimits_{p\in{P_{up\\_c}}}{\mathop{\min}\limits_{q\in{P_{T\\_c}}}\left\|{p-q}\right\|_{2}^{2}+}$
(18)
$\displaystyle\sum\nolimits_{q\in{P_{T\\_c}}}{\mathop{\min}\limits_{p\in{P_{up\\_c}}}\left\|{q-p}\right\|_{2}^{2})},$

and the second metric, which we call HF_HD, is obtained by calculating the HD
of the corresponding HF points, i.e.,

$\displaystyle
HF\\_HD=\max(\mathop{\max}\limits_{p\in{P_{up\\_c}}}\mathop{\min}\limits_{q\in{P_{T\\_c}}}{\left\|{p-q}\right\|_{2}},$
(19)
$\displaystyle\mathop{\max}\limits_{q\in{P_{T\\_c}}}\mathop{\min}\limits_{p\in{P_{up\\_c}}}{\left\|{q-p}\right\|_{2}}),$

Here, ${P_{up\\_c}}$ and ${P_{T\\_c}}$ are the HF points of $P_{up}$ and
$P_{T}$, respectively, $p$ and $q$ represent two points in ${P_{up\\_c}}$ and
${P_{T\\_c}}$, respectively, and $M$ is the number of HF points.

TABLE II: Train and inference time. Time | PU-Net | MPU | PU-Geo | PU-GAN | PU-GCN | Dis-PU | PUFA-GAN
---|---|---|---|---|---|---|---
Train (h) | 5 | 6 | 5 | 18 | 2 | 23 | 28
Inference (s) | 0.13 | 1.06 | 0.7 | 0.46 | 0.26 | 0.58 | 0.94
TABLE III: Upsampling ($\bm{\times}$4) results on ModelNet40. Method | CD | HD | P2F | Uniformity | HF_CD | HF_HD
---|---|---|---|---|---|---
| $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-2}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$
EAR [9] | 1.332 | 12.284 | 6.899 | 65.006 | 10.923 | 96.297
PU-Net [14] | 1.189 | 11.923 | 7.103 | 33.581 | 5.596 | 82.237
MPU [15] | 0.910 | 8.604 | 2.799 | 36.425 | 5.370 | 86.518
PU-Geo [17] | 0.576 | 7.088 | 2.828 | 32.931 | 3.246 | 75.551
PU-GAN [16] | 0.401 | 6.662 | 2.392 | 22.863 | 2.586 | 66.254
PU-GCN [19] | 0.586 | 6.539 | 2.586 | 34.937 | 3.803 | 76.215
Dis-PU [20] | 0.537 | 6.196 | 2.315 | 25.468 | 2.722 | 60.590
PUFA-GAN | 0.379 | 5.891 | 2.170 | 17.806 | 2.544 | 59.823
Figure 7: Upsampling ($\bm{\times}$4) results of high frequency points of m60
from the PU147 dataset.

In the computation of CD, HD, P2F, and Uniformity, the ground truth point
cloud $P_{T}$ was obtained by sampling 8192 points from the test point cloud
using Poisson disk sampling. The LR point cloud was obtained by selecting 2048
points from the test point cloud using Monte Carlo random sampling. In the
computation of HF_CD and HF_HD, both ${P_{up\\_c}}$ and ${P_{T\\_c}}$
consisted of 2048 points obtained from $P_{up}$ and $P_{T}$ with GF.

###  IV-C Comparison with the state-of-the-art

We compared PUFA-GAN with seven methods: optimization based EAR [9], and six
state-of-the-art learning-based methods: PU-Net [14], MPU [15], PU-GAN [16],
PU-Geo [17], PU-GCN [19] and Dis-PU [20]. For a fair comparison, we re-trained
PU-Net, MPU, PU-GAN, PU-GCN, and Dis-PU on our training dataset. This was not
possible for PU-Geo because its training requires the normal vector
information, which is not included in our training dataset. As a compromise,
we used the trained model which was provided to us by the corresponding author
[17]. The upsampling ratio $r$ was set to 4.

Quantitative results. Table 1 shows the quantitative upsampling performance of
all the methods. We can see that PUFA-GAN performs best in terms of CD
(0.258), HD (3.571), HF_CD (2.081), and HF_HD (19.744). For the P2F and
Uniformity metrics, PUFA-GAN obtained the second best results.

Qualitative results. In addition to the objective comparison, we also show in
Fig. 6 the subjective comparisons of upsamping and surface reconstruction
[54]. We can see that the outputs of EAR, PU-Net and MPU are significantly
unevenly distributed. Some local holes may also appear. For PU-Geo, PU-GAN,
PU-GCN and Dis-PU, although the generated point clouds are uniform, there are
noisy points in HF regions. Compared with these methods, PUFA-GAN not only
generates smoother point clouds attached to the underlying surface but also
forms a clear edge with fine-grained details. Specifically, the mouth of
$\bm{duck}$ (top rows) shows that our method generates a uniform and neat
edge, while other methods fail. Moreover, the zoomed-in views show that our
method preserves more detailed texture, e.g., $\bm{cow}$’s hoof (middle rows)
and $\bm{statue}$’s leg (bottom rows). Besides, we also visualize the HF
points of an example fish named $\bm{m60}$ in the PU147 dataset for all
methods (Fig. 7). The same high-pass graph filter was used to extract the HF
points. Compared with the other methods, PUFA-GAN generated a cleaner HF
region, where the points are closer to the HF points of the ground truth. More
visual comparisons can be found in the supplementary materials.

Training and inference time comparison. The time for train and inference are
compared in Table 2. Note that EAR is an optimization-based method, and for
each sparse input point cloud, we need to constantly adjust various hyper-
parameters to obtain the final dense point cloud. Therefore, we do not report
its CPU time. As shown in Table 2, the training time of PU-GCN was the
shortest (2 h), while the inference time of PU-Net was the shortest (0.13 s).
With our method, the training time and the inference time were 28 h and 0.94
s, respectively. The higher time complexity compared to the previous methods
is mainly due to the use of the GF for extracting the high frequency points.

Figure 8: Upsampling ($\bm{\times}$4) the real scanned LiDAR point clouds with
zoomed-in views (Car and Wall) from the KITTI dataset with PUFA-GAN. Figure 9:
Visual upsampling ($\bm{\times}$4) results of the unseen Airplane (a) and test
point cloud Bird (b) with input sizes 256, 1024 and 4096. Figure 10: Using
PUFA-GAN to upsample ($\bm{\times}$4) noisy point clouds (input point clouds
with Gaussian noise levels: 0, 0.5%, and 1% from left to right). Note that the
unseen Guitar (a) is selected from ModelNet40, and Sculpture belongs to our
test dataset.

###  IV-D Robustness Analysis

We further evaluated the robustness of PUFA-GAN from the following aspects:
unseen (test directly on a new dataset using a trained model) point clouds,
real scanned large-scale LiDAR point clouds, point clouds with various sizes
and noise.

Upsampling unseen point clouds. To demonstrate the generalization ability of
PUFA-GAN, we tested it on the unseen ModelNet40 [55]. We randomly selected 20
point clouds from different categories in ModelNet40. The ground truth of each
test sample containing 8192 points was obtained by the Poisson disk sampling
algorithm. At the same time, Monte-Carlo random sampling was used to acquire
the test input with 2048 points. As shown in Table 3, compared with the other
methods, PUFA-GAN achieved the best performance for all evaluation metrics.

Upsampling real scanned LiDAR point clouds. In general, real scanned LiDAR
point clouds are sparse and include a lot of noise and occlusions; thus it is
challenging to upsample them precisely. We verified the effectiveness of PUFA-
GAN on the real scanned LiDAR point clouds from the KITTI [56] dataset (Fig.
8). We can see that PUFA-GAN can generate dense and uniform scenes with rich
geometric details.

Upsampling point clouds with varying sizes. We evaluated PUFA-GAN on two
different datasets (ModelNet40 and our test datasets) for input point clouds
of different scale. The Monte-Carlo random sampling algorithm was adopted to
sample the test point clouds to input sizes 256, 1024 and 4096. Fig. 9 shows
that even for only 256 points, our method can recover the approximate shapes
of the sparse input point clouds.

TABLE IV: Ablation study on the PU147 dataset. Method | CD | HD | P2F | Uniformity | HF_CD | HF_HD
---|---|---|---|---|---|---
| $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$ | $\bm{({10^{-3}})}$
PUFA-GAN w/o DGHRA | 0.279 | 4.211 | 2.715 | 6.705 | 2.579 | 21.178
PUFA-GAN w/o HRA | 0.482 | 14.601 | 4.789 | 15.220 | 4.582 | 33.253
PUFA-GAN w/o GF | 0.258 | 3.862 | 2.408 | 4.898 | 2.044 | 21.024
PUFA-GAN w/o HFD | 0.269 | 4.059 | 2.428 | 5.374 | 2.207 | 20.301
PUFA-GAN w/o IDL | 0.266 | 4.029 | 2.403 | 5.781 | 2.187 | 20.788
PUFA-GAN | 0.258 | 3.571 | 2.392 | 4.889 | 2.081 | 19.744
Figure 11: Ablation study visual results ($\bm{\times}$4).

Upsampling noisy point clouds. To demonstrate the anti-noise ability of PUFA-
GAN, we upsampled the unseen (from ModelNet40) and test point clouds with
Gaussian noise at different noise levels. Fig. 10 shows that for the gradually
enhanced noise, our method can also preserve the natural output shape and
avoids noise interference.

###  IV-E Ablation Study

We conducted an ablation study to quantitatively and qualitatively evaluate
the performance of each proposed component, i.e., 1) the densely connected
feature extraction module, 2) the hierarchical residual-based feature expasion
module, 3) the GF unit in the patch-fusion strategy, 4) the frequency-aware
discriminator (FAD), and 5) the identity distribution loss (IDL). In the
study, the following five networks were considered.

1) PUFA-GAN w/o DGHRA: the proposed feature extraction module is replaced with
DGCNN [36].

2) PUFA-GAN w/o HRA: HRA is replaced with the same number of MLPs in the
feature expansion module.

3) PUFA-GAN w/o GF: the GF is not used in the patch-fusion strategy.

4) PUFA-GAN w/o HFD: the HF discriminator (HFD) is removed from FAD (i.e.,
only a global discriminator is kept)

5) PUFA-GAN w/o IDL: IDL is removed from the joint loss.

The results in Table 4 and Fig. 11 show that PUFA-GAN with all the proposed
components performs best. When any component was removed, the performance of
the network decreased. Particularly, we can see that the most dramatic
performance degradation occured when the HRA unit in the feature expansion
module was removed, which is in line with our expectations. This is because
the feature expansion module directly determines the quality of the upsampling
features with the help of the hierarchical residual features, while the naive
MLPs do not have the ability to explore the feature details. On the other
hand, the GF in the patch-fusion strategy had the weakest influence, since it
can be regarded as a simple post-processing step and does not affect the
entire performance of PUFA-GAN.

##  V Conclusion

We presented a novel generative adversarial network for point cloud
upsampling, namely PUFA-GAN. The generator includes three modules: feature
extraction, feature expansion, and geometry generation. Specifically, a
densely connected DGHRA-based feature extraction module is proposed to obtain
point-wise descriptive features. Then an efficient cascaded HRA-based feature
expansion module generates and enriches the feature details. Next, the
upsampled features are mapped back to the geometry domain in the geometry
generation module. To improve the quality of the upsampled point clouds, we
proposed a frequency-aware discriminator which consists of a global
discriminator and an HF discriminator. The former considers the global context
for better skeleton recognition, while the latter suppresses noise in HF
regions. To further produce clean local details, we removed the potential
noise explicitly by GF in the testing phase. In addition, we proposed a novel
identity distribution loss function to force LR input point clouds and HR
output point clouds to have identical geometry distributions. Finally, we
introduced two evaluation metrics (HF_CD and HF_HD) to evaluate the regularity
of upsampled point clouds in HF regions quantitatively. Extensive experiments
demonstrated that PUFA-GAN outperforms state-of-the-art methods.

In the future, we will design a network with an arbitrary upsampling ratio for
flexible scene switching in an end-to-end fashion. We will also explore other
research directions, such as i) upsampling point clouds with attribute
information (e.g., color, reflectance, normal vector) instead of only geometry
information; ii) taking the point cloud upsampling as a point cloud completion
problem; iii) adopting unsupervised or semi-supervised learning to alleviate
the dependence on paired and scarce training samples.

## References

  * [1] Y. Li G. Baciu, “HSGAN: hierarchical graph learning for point cloud generation,” IEEE Transactions on Image Processing, vol. 30, pp. 4540–4554, Apr. 2021.
  * [2] C. Lv, W. Lin, B. Zhao, “Approximate intrinsic voxel structure for point cloud simplification,” IEEE Transactions on Image Processing, vol. 30, pp. 7241–7255, Aug. 2021.
  * [3] W. Hu, Q. Hu, Z. Wang, X. Gao, “Dynamic point cloud denoising via manifold-to-manifold distance,” IEEE Transactions on Image Processing, vol. 30, pp. 6168–6183, Jul. 2021.
  * [4] W. Wu and C. Zhang, “Immersive 3d communication,” in Proceedings of the 22nd ACM international conference on Multimedia, Nov. 2014, pp. 1229–1230.
  * [5] X. Chen, H. Ma, J. Wan, B. Li and T. Xia, “Multi-view 3d object detection network for autonomous driving,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 6526–6534.
  * [6] M. Alexa, J. Behr, D. Cohen-Or, S. Fleishman, D. Levin, and C. T. Silva, “Computing and rendering point set surfaces,” IEEE Transactions on Visualization and Computer Graphics, vol. 9, no. 1, pp. 3–15, Jan. 2003.
  * [7] Y. Lipman, D. Cohen–Or, D. Levin, H. Tal–Ezer, “Parameterization-free projection for geometry reconstruction,” ACM SIGGRAPH, Jul. 2007.
  * [8] R. Preiner, O. Mattausch, M. Arikan, R. Pajarola, M. Wimmer, “Continuous projection for fast l1 reconstruction,” ACM Transactions on Graphics, vol. 33, no. 4, Jul. 2014.
  * [9] H. Huang, S. Wu, M. Gong, D. Cohen-Or, U. Ascher, and H. Zhang, “Edge-aware point set resampling,” ACM Transactions on Graphics, vol. 32, no. 1, pp. 1–12, Jan. 2013.
  * [10] S. Wu, H. Huang, M. Gong, M. Zwicker, and D. Cohen-Or, “Deep points consolidation,” ACM Transactions on Graphics, vol. 34, no. 6, pp. 1–13, Oct. 2015.
  * [11] C. Dinesh, G. Cheung, I. V. Bajić, “3D point cloud super-resolution via graph total variation on surface normals,” IEEE International Conference on Image Processing (ICIP), Sept. 2019, pp. 4390–4394.
  * [12] C. Dinesh, G. Cheung, I. V. Bajić, “Super-resolution of 3d color point clouds via fast graph total variation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May. 2020, pp. 1983–1987.
  * [13] T. M. Borges, D. C. Garcia and R. L. de Queiroz, “Fractional super-resolution of voxelized point clouds,” IEEE Transactions on Image Processing, vol. 31. pp. 1380–1390, Jan. 2022.
  * [14] L. Yu, X. Li, C. Fu, D. Cohen-Or, and P. Heng, “PU-Net: point cloud upsampling network,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Dec. 2018, pp. 2790–2799.
  * [15] Y. Wang, S. Wu, H. Huang, D. Cohen-Or, and O. Sorkine-Hornung, “Patch-based progressive 3d point set upsampling,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp. 5958–5967.
  * [16] R. Li, X. Li, C. Fu, D. Cohen-Or and P. Heng, “PU-GAN: a point cloud upsampling adversarial network,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Nov. 2019, pp. 7202–7211.
  * [17] Y. Qian, J. Hou, K. Kwong, Y. He, “PUGeo-Net: a geometry-centric network for 3d point cloud upsampling,” in European Conf. on Computer Vision, Nov. 2020, pp. 752–769.
  * [18] Y. Qian, J. Hou, K. Kwong, Y. He, “Deep magnification-flexible upsampling over 3d point clouds,” IEEE Transactions on Image Processing, vol.30, pp. 8354–8367, Sept. 2021.
  * [19] G. Qian, A. Abualshour, G. Li, A. Thabet, B. Ghanem, “PU-GCN: point cloud upsampling using graph convolutional networks,”in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2021.
  * [20] R. Li, X. Li, P. Heng, C. Fu, “Point cloud upsampling via disentangled refinement,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2021.
  * [21] H. Liu, H. Yuan, R. Hamzaoui, W. Gao and S. Li, “PU-Refiner: a geometry refiner with adversarial learning for point cloud upsampling,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [22] I. J. Goodfellow, J. Pouget–Abadie, M. Mirza, B. Xu, D. Warde–Farley, S. Ozair, A. Courville, Y. Bengio. “Generative adversarial nets,” in Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.
  * [23] A. Ortega, P. Frossard, J. Kovačević, J. M. F. Moura, P. Vandergheynst, “Graph signal processing: overview, challenges, and applications,” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, May. 2018.
  * [24] A. Sandryhaila and J. M. F. Moura, “Discrete signal processing on graphs: frequency analysis,” IEEE Transactions on Signal Processing, vol. 62, no. 12, pp. 3042–3054, Jun. 2014.
  * [25] R. Q. Charles, H. Su, M. Kaichun and L. J. Guibas, “PointNet: deep learning on point sets for 3d classification and segmentation,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 77–85.
  * [26] R. Q. Charles, L. Yi, H. Su, L. J. Guibas, “PointNet++: deep hierarchical feature learning on point sets in a metric space,” in Advances in Neural Information Processing Systems, 2017. pp. 5099–5108.
  * [27] L. Yu, X. Li, C. Fu, D. Cohen-Or, and P. Heng, “EC-Net: an edge-aware point set consolidation network,” in European Conf. on Computer Vision, Sept. 2018, pp. 386–402.
  * [28] G. Metzer, R. Hanocka, R. Giryes, D. Cohen-Or, “Self-sampling for neural point cloud consolidation,” ACM Transactions on Graphics, vol. 40, no. 5, pp. 1–14, Sept. 2021.
  * [29] C. Doersch, A. Gupta and A. A. Efros, “Unsupervised visual representation learning by context prediction,” IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1422–1430.
  * [30] S. Gidaris, P. Singh, N. Komodakis, “Unsupervised representation learning by predicting image rotations,” International Conference on Learning Representations(ICLR), 2018.
  * [31] P. Isola, J. Zhu, T. Zhou and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 5967–5976.
  * [32] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D. Metaxas, “StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 5908–5916.
  * [33] E. Smith, D. Meger, “Improved adversarial systems for 3D object generation and reconstruction,” in 1st Annual Conference on Robot Learning, 2017, pp. 87–96.
  * [34] B. Yang, H. Wen, S. Wang, R. Clark, A. Markham and N. Trigoni, “3D object reconstruction from a single depth view with adversarial learning,” in IEEE International Conference on Computer Vision Workshops, Oct. 2017, pp. 679–688.
  * [35] G. Huang, Z. Liu, L. V. D. Maaten, K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 2261–2269.
  * [36] Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. M. Bronstein, J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” ACM Transactions on Graphics, vol. 38. no. 5. Nov. 2018.
  * [37] K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [38] Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, Y. Xu, M. Tan, “Closed-loop matters: dual regression networks for single image super-resolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2020. pp. 5406–5415.
  * [39] T. Dai, J. Cai, Y. Zhang, S. Xia and L. Zhang, “Second-order attention network for single image super-resolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp. 11057–11066.
  * [40] J. Liu, W. Zhang, Y. Tang, J. Tang and G. Wu, “Residual feature aggregation network for image super-resolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp. 2356–2365.
  * [41] J. Hu, L. Shen, S. Albanie, G. Sun and E. Wu, “Squeeze-and-Excitation networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 8, pp. 2011–2023, Aug. 2020.
  * [42] Y. Yang, C. Feng, Y. Shen and D. Tian, “FoldingNet: point cloud auto-encoder via deep grid deformation,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 206–215.
  * [43] W. Yuan, T. Khot, D. Held, C. Mertz, M. Hebert, “PCN: point completion network,” in International Conference on 3D Vision, 2018, pp. 728–737.
  * [44] S. Chen, D. Tian, C. Feng, A. Vetro and J. Kovačević, “Fast resampling of three-dimensional point clouds via graphs,” IEEE Transactions on Signal Processing, vol. 66, no. 3, pp. 666–681, Feb. 2018.
  * [45] H. Zhang, I. J. Goodfellow, D. N. Metaxas, and A. Odena. “Self-attention generative adversarial networks,” in Int. Conf. on Machine Learning, 2019, pp. 7354–7363.
  * [46] X. Mao, Q. Li, H. Xie, R. Y.K. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” in IEEE Int. Conf. on Computer Vision, 2017, pp. 2794–2802.
  * [47] H. Fan, H. Su, and L. J. Guibas, “A point set generation network for 3D object reconstruction from a single image,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 605–613.
  * [48] J. Zhu, T. Park, P. Isola and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” IEEE International Conference on Computer Vision, Oct. 2017, pp. 2242–2251.
  * [49] Visionair. http://www.infra-visionair.eu/. Accessed: 2021-1-20.
  * [50] D. P. Kingma, J. L. Ba, “Adam: a method for stochastic optimization, ” in International Conference on Learning Representations, 2015.
  * [51] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems, 2017, pp. 6626–6637.
  * [52] M. Berger, J. A. Levine, L. G. Nonato, G. Taubin, and C. T. Silva, “A benchmark for surface reconstruction,” ACM Transactions on Graphics, vol. 32, no. 2, pp. 1–17, Apr. 2013.
  * [53] K. Low, “Linear least-squares optimization for point-to-plane icp surface registration,”, 2004.
  * [54] M. Kazhdan and H. Hoppe, “Screened poisson surface reconstruction,” ACM Transactions on Graphics, vol. 32, no. 2, pp. 1–13, Jul. 2013.
  * [55] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3D shapenets: a deep representation for volumetric shapes,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp. 1912–1920.
  * [56] A. Geiger, P. Lenz, C. Stiller, R. Urtasun, “Vision meets robotics: the kitti dataset,” The International Journal of Robotics Research, vol. 32, no. 111, 2013.


# Corrugated single layer templates for molecules: From $h$-BN Nanomesh to
Graphene based Quantum dot arrays

Haifeng Ma  Physik-Institut, Universität Zürich, Winterthurerstrasse 190,
CH-8057 Zürich, Switzerland  Mario Thomann  Physik-Institut, Universität
Zürich, Winterthurerstrasse 190, CH-8057 Zürich, Switzerland  Jeanette
Schmidlin  Physik-Institut, Universität Zürich, Winterthurerstrasse 190,
CH-8057 Zürich, Switzerland  Silvan Roth  Physik-Institut, Universität Zürich,
Winterthurerstrasse 190, CH-8057 Zürich, Switzerland  Martin Morscher  Physik-
Institut, Universität Zürich, Winterthurerstrasse 190, CH-8057 Zürich,
Switzerland  Thomas Greber<EMAIL_ADDRESS>Physik-Institut, Universität
Zürich, Winterthurerstrasse 190, CH-8057 Zürich, Switzerland

###### Abstract

Functional nano-templates enable self-assembly of otherwise impossible
arrangements of molecules. A particular class of such templates is that of
$sp^{2}$ hybridized single layers of hexagonal boron nitride or carbon
(graphene) on metal supports. If the substrate and the single layer have a
lattice mismatch, superstructures are formed. On substrates like rhodium or
ruthenium these superstructures have unit cells with $\sim$3 nm lattice
constant. They are corrugated and contain sub-units, which behave like traps
for molecules or quantum dots, which are small enough to become operational at
room temperature. For graphene on Rh(111) we emphasize a new structural
element of small extra hills within the corrugation landscape. For the case of
molecules like water it is shown that new phases assemble on such templates,
and that they can be used as ”nano-laboratories” where many individual
processes are studied in parallel. Furthermore, it is shown that the
$h$-BN/Rh(111) nanomesh displays a strong scanning tunneling microscopy
induced luminescence contrast within the 3 nm unit cell which is a way to
address trapped molecules and/or quantum dots.

hexagonal boron nitride, graphene, nano-template, quantum dot, nano-ice,
nanomesh, electroluminescence

##  I Introduction

Graphene resounds throughout the land Geim and Novoselov (2007). It is a
single layer of $sp^{2}$ hybridized carbon, has remarkable chemical stability
and physical properties like that of a conductor with high charge carrier
mobility. Her polar sister, hexagonal boron nitride, has similar chemical
stability, though is an insulator.

Single layer hexagonal boron nitride ($h$-BN) and graphene ($g$) also have
great potential as templates for molecular self-assembly. The layers are grown
and supported on transition metal surfaces Oshima and Nagashima (1997); Greber
(2010a). Here we focus on _corrugated_ single layers. The corrugation is a
vertical deformation of the surface that can be described as a static
distortion wave. The physical origin of these distortions are the mismatch and
the concomitant epitaxial stress between the overlayer and the substrate,
where the anisotropic bonding or lock in energy imposes this kind of
dislocations. The much softer out of plane modulus of the $sp^{2}$ layers
causes a large vertical distortion compared to the in-plane straining. The
wavelengths, or superlattice constants, of these static distortion waves can
be calculated from the lattice mismatch between the overlayer and the
substrate. For rhodium and ruthenium it is in the order of 3 nm and the
corrugation, or peak to peak amplitude (between 0.05 and 0.15 nm) are the
essential features and determine the template function. It has been shown that
the corrugation imposes _lateral_ electric fields, which can guide charged or
polarizable media Dil et al. (2008); Brugger et al. (2009). This property
leads to 3 bond hierarchy levels. The $\sigma$-bonds, in the order of 10 eV
provide chemical stability and robustness, the $\pi$-bonds, in the order of 1
eV, the adsorption energies, and the $\alpha$-bonds(named after the label
$\alpha$ for the polarizability $\alpha$), in the order of 100 meV, are
responsible for the lateral trapping of molecules Greber (2010b).

This article covers the basic ingredients of the geometric structure of
lattice mismatched $sp^{2}$ layers on transition metals, their potential as
”nano-laboratories”, where an example of the behavior of nano-ice clusters as
a function of temperature is given. Finally, the potential of such
superstructures as quantum dot arrays is outlined. It is shown that such
quantum dots can be addressed by electroluminescence, where the yield varies
one order of magnitude within the 3 nm unit cell of $h$-BN/Rh(111).

##  II Geometric structure

When the lattice mismatch $M$ of an overlayer with the substrate exceeds a
critical value, superstructures with large lattice constants are formed. For
parallel epitaxy we write:

$M=\frac{a_{ovl}-a_{sub}}{a_{sub}}$ (1)

where $a_{ovl}$ and $a_{sub}$ are the overlayer and the substrate lattice
constants, respectively. In this notation positive (negative) $M$ indicate
compressive (tensile) stress in the overlayer, and vice versa. For most
transition metal substrates with $h$-BN or $g$ overlayers the mismatch is
negative, i.e. tensile stress in the overlayer prevails.

If the lattice of the overlayer and the substrate are rigid and parallel, the
superstructure lattice constant gets $a_{ovl}/|M|$, where $a_{ovl}$ is the
1$\times$1 lattice constant of $h$-BN or graphene ($\sim$ 0.25 nm). Besides
the mismatch, the balance between the lock in energy and the strain energy is
decisive for the resulting morphology of the systems. Lock in energy has to be
paid when the over-layer atoms are moved parallel to the surface, away from
the lowest energy sites. For systems with small lock in energy compared to the
strain energy, we expect flat floating layers, reminiscent to incommensurate
moiré patterns without a lock in to a high symmetry direction of the
substrate. Such examples of moiré’s are e.g. $h$-BN/Pd(111)Morscher et al.
(2006); Greber et al. (2009) or $g$/Ir(111) N’Diaye et al. (2006). However, if
the unit cells of the superstructure contain regions with distinct electronic
structure, as it is e.g. the case for $h$-BN/Rh(111) Corso et al. (2004);
Berner et al. (2007) or $g$/Ru(0001) Brugger et al. (2009); Zhang et al.
(2009), it is appropriate to use a term distinct from moiré, like it is
’nanomesh’. Preobrajenski et al. were the first who also used the term
nanomesh for $g$-systems where two distinct carbon core levels have been found
Preobrajenski et al. (2008). The energy difference in the core level binding
energies was assigned to the corrugation, i.e. different ’elevations’ or
distances of the $sp^{2}$ layers from the substrate. For a superstructure of a
honeycomb lattice of $sp^{2}$ hybridized layers on a hexagonal closed packed
substrate as Rh(111) or Ru(0001), it is convenient to describe different
locations along the notation used by Auwärter and Grad et al. Auwärter et al.
(1999); Grad et al. (2003). The honeycomb lattice is made of a base with two
atoms (B,N) or (CA,CB). These atoms may sit on $top$, on $fcc$ or on $hcp$
sites within the substrate unit cell (see Figure 1). The $top$ site is
occupied if an atom of the honeycomb sits on top of a substrate atom, below
the $fcc$ hollow site no atom is found in the second substrate layer, while
there is one for the $hcp$ hollow site.

Figure 1: Scheme for the classification of the adsorption sites on a hexagonal
closed packed surface: $top$, $fcc$, and $hcp$. The dashed line shows the
(1$\times$1) unit cell. After Ref. Auwärter et al. (1999)

We note that for single domain $h$-BN structures only three of the six
combinations of the honeycomb base like ($top$,$fcc$), ($hcp$,$top$),
($fcc$,$hcp$) or ($fcc$,$top$), ($top$,$hcp$), ($hcp$,$fcc$) occur. For
graphene, where the two carbon atoms are distinct by their coordination to the
substrate only, no such domains are expected. When the honeycomb is
mismatched, i.e. does not have the lattice constant of the substrate, the
assignments (B,N)=($fcc$,$top$),($hcp$,$fcc$) etc. are only approximately
valid and we write e.g. (CA,CB)$\sim$($top$, $hcp$). With this scheme in mind
we may understand the complementarity of mismatched $h$-BN and $g$
’nanomeshes’, where it is found that about one third of the super cell
(B,N)$\sim$($hcp$,$fcc$) and (CA,CB)$\sim$($hcp$,$fcc$) do not bind to the
substrate and consequently belong to the elevated regions. For $h$-BN there is
also no bonding for boron on $top$ sites i.e. (B,N)$\sim$($top$,$hcp$), and
consequently two thirds of the mismatched $h$-BN layers are elevated and form
the connected ’wire’ network. Graphene is complementary i.e. the
(CA,CB)$\sim$($top$,$hcp$) and the (CA,CB)$\sim$($fcc$,$top$) sites bind to
the substrate and form a connected hexagonal ’valley’ network, with graphene
in close contact to the substrate Brugger et al. (2009). It has to be
emphasized that the substrate breaks the symmetry between the sublattice made
of CA and CB atoms, respectively, and disables the formation of Dirac cones,
which are the attribute of freestanding graphene. For $h$-BN this symmetry
breaking is intrinsic, since B and N are different and induce polarity with
electron transfer to nitrogen.

Figure 2 shows room temperature scanning tunneling microscopy (STM) pictures
of $g$/Rh(111) and $h$-BN/Rh(111). The relief views in a) and c) are extracted
by the WSxM Scanning Probe Microscopy software Horcas et al. (2007) from the
scanning tunneling microscopy data in b) $h$-BN/Rh(111) and d) $g$/Rh(111),
respectively. Clearly, the inverted topographies of the two layer systems can
be seen. The $h$-BN/Rh(111) nanomesh has a 12$\times$12 periodicity where 13
BN units sit on 12 Rh substrate units, which corresponds to a 3.2 nm
superlattice constant Corso et al. (2004). The labels for the two topographic
elements are ’holes’, ’pores’, ’cavities’ or ’cells’ for the (B,
N)=($fcc$,$top$) regions with close binding and ’wires’ for the (B,
N)=($hcp$,$fcc$) & ($top$,$hcp$) regions, which are elevated by about 0.1 nm.
The $g$/Rh(111) ’nanomesh’ has a periodicity of about 11$\times$11, where 12
$g$ units sit on 11 Rh substrate units, which corresponds to a 2.9 nm
superlattice constant Müller et al. (2009). The slightly smaller unit cell is
due to the smaller lattice constant of graphene compared to that of hexagonal
boron nitride. The labels for the two topographic elements are ’mounds’,
’hills’ or ’ripples’ for the ($C_{A}$, $C_{B}$)=($hcp$,$fcc$) protrusions with
loose binding and ’valleys’ for the ($C_{A}$, $C_{B}$)=($fcc$,$top$) &
($top$,$hcp$) regions, which are about 0.1 nm closer to the substrate. For the
case of $g$/Rh(111) we would like to mention a difference, compared to the
related $g$/Ru(0001) system. It can be seen that the strongest bonding does
not coincide with the ($fcc$,$top$) or ($top$,$hcp$) sites but 3 small extra
dips in the valleys where carbon atoms are closer to bridge sites are binding
strongest to the substrate Iannuzzi . This, compared to $g$/Ru(0001), new
structural element might impose extra effects in the template function and
should be further explored.

Figure 2: Scanning tunneling microscopy data of $sp^{2}$ single layers on
Rh(111). a) Relief view of $g$/Rh(111). Note the hills and the valleys, and
the small extra hills at the ($hcp$,$fcc$) sites. b) corresponding STM picture
($I_{t}=0.8$ nA, $U_{t}=-0.8$ V , 11$\times$14 nm2). c) Relief view of
$h$-BN/Rh(111) nanomesh. The elevated regions form the so called wires of the
nanomesh. d) corresponding STM picture ($I_{t}=1$ nA, $U_{t}=1$ V,
11$\times$14 nm2).

As it was inferred by photoemission from adsorbed xenon for $h$-BN/Rh(111) Dil
et al. (2008) and $g$/Ru(0001) Brugger et al. (2009), in both systems ’high’
or ’elevated’ regions have a high local work function while low regions have a
lower local work function Brugger et al. (2009). These physically and
electronically corrugated landscapes form templates for the self-assembly of
molecular arrays as it was shown for $h$-BN nanomesh Corso et al. (2004);
Berner et al. (2007); Ma et al. (2010), $g$/Rh(111) Pollard et al. (2010) or
$g$/Ru(0001) Mao et al. (2009). Also it was demonstrated that these substrates
may be used for the growth of metal nano-clusters Zhang et al. (2008); Pan et
al. (2009).

##  III Nano-laboratories: Water on the $h$-BN nanomesh

Here we want to highlight the opportunity to use a template like the $h$-BN
nanomesh as a nano-laboratory, where processes may be studied in a parallel
fashion, i.e. in an ensemble, at the same time, under the same temperature and
pressure conditions. For scanning probes this also comprises the added value
that the data are recorded with the same tip. These features increase the data
flux from the experiment by orders of magnitude. In particular we expect that
e.g. the diversification in growth processes may be studied. This nano-
laboratory assay allows, to study equilibrium as well as non equilibrium
processes. If we want to lend a picture from biology the $h$-BN nanomesh can
be considered as a ”cell culture”, though at a 3 orders of magnitude smaller
length scale. In order to illustrate the ”nano-laboratory” we show data on the
temperature evolution of water clusters in the $h$-BN nanomesh. Recently it
was shown that water self-assembles in the ’holes’ of $h$-BN nanomesh in
forming ice nanoclusters containing about 40 water molecules. The clusters
consist in a bilayer of ice, reminiscent to the basal plane of hexagonal ice,
and display proton disorder that was accessed with tunneling barrier height
measurements Ma et al. (2010). In Figure 3 the development of the ice clusters
as a function of temperature is shown, for 5 different temperatures, About 7
nanomesh ”cells” with a diameter of 2 nm contain each one ice cluster. It has
to be mentioned that the results for different temperatures do not show the
same cells, because the thermal drift in the present set up during the warm up
from 34 to 151 K does not yet allow to track individual cells. The
superstructure periodicity does, however, allow an almost perfect drift
correction at a given temperature. It can be clearly seen that the rims of the
ice clusters have a distinct behavior with respect to the bulk. Figure 3 a)
shows the ordered ice clusters made by about 40 water molecules at 34 K. Every
second water molecule shows up as a protrusion inside the $h$-BN nanomesh. At
101 K the edge of the clusters start to display disorder (Figure 3 b). Further
increase of the temperature induces an increase of the cluster height (Figure
3 c) and d)) and at 151 K the clusters sublimated (Figure 3 e) and bare
nanomesh is observed.

Figure 3:  Behavior of nano-ice clusters in the temperature range between 34 K
and 151 K as recorded by variable temperature scanning tunneling microscopy
(VT-STM). The 5 hexagonal frames with a side length of 4.2 nm show the
constant current feed-back signal, after drift correction. (a) Nano-ice
clusters at 34 K. $V_{t}=-0.05$ V, $I_{t}=100$ pA. (b) At 101 K water
molecules at the rims of the nano-ice cluster become mobile, the core remains
frozen. $V_{t}=-0.05$ V, and $I_{t}=100$ pA. (c) Low coordinated water induced
protrusions at the cluster boundary are observed at 139 K, the core features
weaken. $V_{t}=-0.01$ V, and $I_{t}=50$ pA. (d) as (c), the rims show higher
contrast than the core molecules which still show crystallinity at 143 K.
$V_{t}=-0.01$ V, and $I_{t}=50$ pA. (e) Empty nanomesh after water desorbed
from the surface. $V_{t}=-0.05$ V, and $I_{t}=40$ pA.

##  IV Addressing Room temperature Quantum dots

A quantum dot is a small physical object that is confined in 3 dimensions,
where nuclei and atoms are the most prominent examples. The localization to a
’point’ implies the absence of dispersion of the electronic states. The size
of a quantum dot determines the energy level spacing. If this spacing $\Delta
E$ is compared to $k_{B}T$ we get a measure for the temperature below which we
expect the ’dots’ to behave like quantum objects, or above which the occupancy
of different levels fluctuates. The Rydberg energy (13.6 eV), which is the
scale for the electronic level spacing in a Coulomb potential of a proton is
proportional to ${a_{o}}^{-1}$, where $a_{0}$=0.05 nm is the Bohr radius. If
we are interested in quantum dots that are operational at room temperature
($k_{B}T$=25 meV), this limits the size of quantum dots to below 500 $a_{o}$.
Though, for practical purposes the level spacing should be at least one order
of magnitude larger than $k_{B}T$ and thus room temperature quantum dots
should be smaller than 5 nm.

The $sp^{2}$ templates discussed in this paper do match this condition. Indeed
quantum dot behavior was found for graphene on ruthenium Zhang et al. (2009).
For $g$/Ru(0001) photoemission showed one set of of $sp^{2}$ valence bands,
while $h$-BN/Ru(0001) and $h$-BN/Rh(111) do show two $sp^{2}$ valence band
structures split by about 1 eV Goriachko et al. (2007). The two band structure
systems were assigned to the two regions within the super cells, where the
corrugation of the $h$-BN imposed, a mainly electrostatically driven splitting
Laskowski et al. (2007); Berner et al. (2007). As expected this splitting is
also observed with high resolution B 1s, C 1s and N 1s core level
spectroscopies Preobrajenski et al. (2008), and photoemissions from adsorbed
Xe Dil et al. (2008); Brugger et al. (2009). Interestingly the valence band
splitting was not observed for $g$/Ru(0001) Brugger et al. (2009). The obvious
difference between graphene and $h$-BN is that graphene on ruthenium has a
Fermi surface, while $h$-BN on Ru(0001) Brugger et al. (2009) or on Rh(111)
Greber et al. (2009) has not, may not explain this with a screening argument,
since the C 1s core level is still split on $g$/Ru(0001) Preobrajenski et al.
(2008). The seeming paradox can be resolved, when we assign to the hills in
the $g$/Ru(0001) superstructure an isolated, molecule like behavior, without
dispersion, which qualifies them as quantum dots arranged on a hexagonal array
with 3 nm spacing. For the case of the $h$-BN, the holes might also be
identified as quantum dots, however, angular resolved photoemission shows
dispersion of the $h$-BN valence bands, also for the bands assigned to the
holes that are separated by the superlattice constant of 3 nm Brugger et al.
(2009). A difference between the hills of $g$/Ru(0001) and the holes of
$h$-BN/Ru(0001) is the fact that $h$-BN holes are in close contact to the
substrate, while graphene hills are decoupled. This imposes a lateral vacuum
tunneling barrier for electrons on the hills, while this barrier is much lower
for the case of the $h$-BN holes that are in close proximity to the metal
substrate.

If the $sp^{2}$ templates are decorated with molecules (or clusters), the
quantum dots change and the coupling between them will be affected. It will be
interesting to further explore this coupling and to try to control it.
Electroluminescence could serve as a tool to access such information. With
scanning tunneling microscopy induced electroluminescence, light emission can
be probed as a function of the tunneling site with sub-wavelength resolution.
As we show here it is one possible road to access single unit cells and is
considered to be a realization of a nano-device, where it comes to the
transport of information localized at the nanometer scale to the macroscopic
millimeter scale.

Electroluminescence in scanning tunneling microscopy was pioneered by
Gimzewski et al. Gimzewski et al. (1988), and is a way to record inelastic
scattering in tunneling junctions, where e.g. molecular vibrations may be
resolved, if the wavelength of the emitted photons is measured Wu et al.
(2006). Figure 4 shows the correlation between topography and light emission
from $h$-BN nanomesh. Light was collected by a lens system connecting the
tunneling junction with a cooled red sensitive Hamamatsu R5929 photomultiplier
tube operating in the wavelength window between 300 and 850 nm. The tunneling
voltage was set to -2.5 V (tunneling electrons from the substrate to the tip)
and tungsten tips with a 80 nm gold coating were used. For isotropic emission
into the $2\pi$ half space above a surface, the detection probability of a 2
eV photon is 0.4 %, and the dark count rate was about 25 counts/s. The photon
map in Figure 4 b) and the corresponding cut in Figure 4 c) show strong
electroluminescence from the nanomesh ’holes’, which is more than one order of
magnitude larger than that from the wires. Though, the average quantum
efficiency (detected photons per scan line) varies in the shown data by one
order of magnitude. This variation must be ascribed to changes in the gold tip
where plasmon excitations/deexcitations cause photon emission. The arrows on
the right of Figure 4a) and b) indicate three distinct tip changes $A,B,C$. It
can be seen that the topography image before change $A$ is inverted after
change $C$. The enhancement of the quantum efficiency at change $B$ does
neither coincide with $A$ nor $C$. However, the results suggest that the
tunneling junction with the tip on top of a hole of the $h$-BN nanomesh
imposes more inelastic scattering events on the tip. The high electron
affinity and the concave form of the holes lets them act like a resonator
cavity, where the electrons are focused on the tip, and where the probability
for a plasmon excitation increases. It has to be mentioned that the inverse
situation is observed for graphene on Ru, where the poor electron affinity and
the convexity of the hills defocus electrons in a tunneling junction with the
tip on top of the hill Zhang et al. (2009).

The data shown in Figure 4 indicate that scanning tunneling microscopy induced
luminescence can be used for the identification of sites within the
12$\times$12 super cell of $h$-BN/Rh(111) with sub-nanometer resolution. Also
the experiments indicate that the control of the tip parameters is crucial for
a successful application of this effect.

Figure 4: Room temperature scanning tunneling microscopy and photon emission
scanning tunneling microscopy from $h$-BN nanomesh with a gold coated tungsten
tip. (30$\times$15 nm, $I_{t}=2.6$ nA, $V_{t}=-2.5$ V, scan time 110 s with
128 horizontal scanning lines from bottom to top). The labels $A,B,C$ indicate
tip changes (for details see text). a) Topography, note the 3 nm periodicity
of $h$-BN nanomesh, and the tip changes $A$, $B$. b) Light map, i.e.
simultaneously to a) recorded photoncurrent. For a certain line series the
luminescence is particularly high, and the periodicity of the $h$-BN nanomesh
lights up. c) Cut across the light map, along the red line in b). The
polychromatic light current is given in photons/s.

##  V Summary

In summary it is recalled that lattice mismatched $sp^{2}$ hybridized single
layers of $h$-BN and graphene may be used as templates for the self-assembly
of molecular structures. For $g$/Rh(111) a new structural element, extra
”hills” in the valleys, are emphasized. In a second section it is shown that
$h$-BN/Rh(111) can be used as a ”nano-laboratory”, where molecular processes
in individual nanomesh cells may be studied. Finally it is outlined that these
superstructures have the features of quantum dots, which are small enough to
become operational at room temperature. As an example on how such objects can
be addressed the luminescence as induced by a scanning tunneling microscope is
demonstrated to have a resolution better than one nanometer.

##  VI Acknowledgements

Financial support by the Swiss National Science Foundation is gratefully
acknowledged.

## References

  * Geim and Novoselov (2007) A. K. Geim and K. S. Novoselov, Nat. Mater. 6, 183 (2007).
  * Oshima and Nagashima (1997) C. Oshima and A. Nagashima, J. Phys.: Condens. Matter 9, 1 (1997).
  * Greber (2010a) T. Greber, Handbook of Nanophysics: Functional Nanomaterials (Taylor and Francis Books, London, 2010a).
  * Dil et al. (2008) H. Dil, J. Lobo-Checa, R. Laskowski, P. Blaha, S. Berner, J. Osterwalder, and T. Greber, Science 319, 1824 (2008).
  * Brugger et al. (2009) T. Brugger, S. Günther, B. Wang, J. H. Dil, M. L. Bocquet, J. Osterwalder, J. Wintterlin, and T. Greber, Phys. Rev. B 79, 045407 (2009).
  * Greber (2010b) T. Greber, e-J. Surf. Sci. Nanotech. 8, 62 (2010b).
  * Morscher et al. (2006) M. Morscher, M. Corso, T. Greber, and J. Osterwalder, Sur. Sci. 600, 3280 (2006).
  * Greber et al. (2009) T. Greber, M. Corso, and J. Osterwalder, Sur. Sci. 603, 1373 (2009).
  * N’Diaye et al. (2006) A. T. N’Diaye, S. Bleikamp, P. J. Feibelman, and T. Michely, Phys. Rev. L 97, 215501 (2006).
  * Corso et al. (2004) M. Corso, W. Auwärter, M. Muntwiler, A. Tamai, T. Greber, and J. Osterwalder, Science 303, 217 (2004).
  * Berner et al. (2007) S. Berner, M. Corso, R. Widmer, O. Groening, R. Laskowski, P. Blaha, K. Schwarz, A. Goriachko, H. Over, S. Gsell, et al., Angew. Chem. Int. Ed. 46, 5115 (2007).
  * Zhang et al. (2009) H. G. Zhang, H. Hu, Y. Pan, J. H. Mao, M. Gao, H. M. Guo, S. X. Du, T. Greber, and H.-J. Gao, arXiv:0911.4024. (2009).
  * Preobrajenski et al. (2008) A. B. Preobrajenski, M. L. Ng, A. S. Vinogradov, and N. Mårtensson, Phys. Rev. B 78, 073401 (2008).
  * Auwärter et al. (1999) W. Auwärter, T. J. Kreutz, T. Greber, and J. Osterwalder, Sur. Sci. 429, 229 (1999).
  * Grad et al. (2003) G. B. Grad, P. Blaha, K. Schwarz, W. Aüwarter, and T. Greber, Phys. Rev. B 68, 085404 (2003).
  * Horcas et al. (2007) I. Horcas, R. Fernández, J. M. Gómez-Rodríguez, J. Colchero, J. Gómez-Herrero, and A. M. Baro, Rev. Sci. Instrum. 78, 013705 (2007).
  * Müller et al. (2009) F. Müller, H. Sachdev, S. Hüfner, A. J. Pollard, E. W. Perkins, J. C. Russell, P. H. Beton, S. Gsell, M. Fischer, M. Schreck, et al., Small 5, 2291 (2009).
  * (18) M. Iannuzzi, Private Communication.
  * Ma et al. (2010) H. F. Ma, T. Brugger, S. Berner, Y. Ding, M. Iannuzzi, J. Hutter, J. Osterwalder, and T. Greber, ChemPhysChem 11, 399 (2010).
  * Pollard et al. (2010) A. J. Pollard, E. W. Perkins, N. A. Smith, A. Saywell, G. Goretzki, A. G. Phillips, S. P. Argent, H. Sachdev, F. Müller, S. Hüfner, et al., Angew. Chem. Int. Edit. 49, 1794 (2010).
  * Mao et al. (2009) J. H. Mao, H. G. Zhang, Y. H. Jiang, Y. Pan, M. Gao, W. D. Xiao, and H. J. Gao, J. Am. Chem. Soc. 131, 14136 (2009).
  * Zhang et al. (2008) J. Zhang, V. Sessi, C. H. Michaelis, I. Brihuega, J. Honolka, K. Kern, R. Skomski, X. Chen, G. Rojas, and A. Enders, Phys. Rev. B 78, 165430 (2008).
  * Pan et al. (2009) Y. Pan, M. Gao, L. Huang, F. Liu, and H. J. Gao, Appl. Phys. Lett. 95, 093106 (2009).
  * Goriachko et al. (2007) A. Goriachko, Y. B. He, M. Knapp, H. Over, M. Corso, T. Brugger, S. Berner, J. Osterwalder, and T. Greber, Langmuir 23, 2928 (2007).
  * Laskowski et al. (2007) R. Laskowski, P. Blaha, T. Gallauner, and K. Schwarz, Phys. Rev. L 98, 106802 (2007).
  * Gimzewski et al. (1988) J. K. Gimzewski, B. Reihl, J. H. Coombs, and R. R. Schlittler, Z. Phys. B-Condens. Mat. 72, 497 (1988).
  * Wu et al. (2006) S. W. Wu, N. Ogawa, and W. Ho, Science 312, 1362 (2006).