# Electronic fractal patterns in building Sierpinski-triangles molecular
systems

L.L.Lage  Instituto de Física, Universidade Federal Fluminense, Niterói, Av.
Litorânea sn 24210-340, RJ-Brazil  A. Latgé  Instituto de Física, Universidade
Federal Fluminense, Niterói, Av. Litorânea sn 24210-340, RJ-Brazil

###### Abstract

The Sierpinski Triangle (ST) is a fractal mathematical structure that has been
used to explore the emergence of flat bands in lattices of different
geometries and dimensions in condensed matter. Here we look into fractal
features in the electronic properties of ST flakes and molecular chains
simulating experimental synthesized fractal nanostructures. We use a single-
orbital tight binding model to study fractal properties of the electronic
states and the Landauer formalism to explore transport responses of the quasi
1D molecular chains. The self-similarity of the energy states are found
comparing different ST orders and also amplifying the energy ranges
investigated, for both flakes and quasi-1D systems. In particular, the results
for the local density of states of the theoretical molecular chains proposed
here exhibit quite similar spatial charge distribution of experimental STM
reports. The analysis of transport response of such all-carbon fractal
molecular chains can be used as a guide to propose a variety of architecture
in the synthesis of real new molecular chains.

##  I Introduction

The physical properties of confined electronic structures have been widely
investigated, mainly thanks to the possibility of tuning electronic states and
its transport responses. Nanostructured systems emerge in different shapes and
dimensions and further than being only dreamed theoretical idealization are
being synthesized making use of multiple bottom-up and up-bottom growth
processes Chandel  _et al._ (2020); Li  _et al._ (2022). In particular,
systems exhibiting dispersionless states in the electronic structure, named as
flat bands have been pursued with the help of different strategies, such as
controlling the angle of twisted bilayers Lisi  _et al._ (2021); Cao  _et al._
(2018), switching on electric and magnetic fields Rhim  _et al._ (2020); Wolf
_et al._ (2019), and also engineering the growth of buckled graphene
superlattices J.  _et al._ (2020).

Recently, the fractal dimension opened a new door in the study of the
electronic properties in confined systems Kempkes  _et al._ (2019). The
Sierpinski Triangle (ST) is a fractal mathematical structure that was used
physically as a quantum corral to confine electrons. Transport responses were
also explored in ST lattice models indicating a new scenario for correlated
topological systems in fractal dimensions Manna  _et al._ (2020). Despite the
fact ST is represented in a bidimensional form, it has a fractal dimension,
called Hausdorff dimension (D) given by $D\approx 1.585$.

Figure 1: Variations of G(3) Sierpinski Triangle flakes: (a) Conventional ST,
(b) Zigzag-Kagomé-Honeycomb ST (Z-KHST), (c) Armchair-Graphene ST (A-GST), (d)
Hexagonal ST (HST), (e) Armchair-Kagomé-Honeycomb ST (A-KHST), and (f) Zigzag-
Graphene ST (Z-GST).

Graphene lattices and other structures, as illustrated in Fig. 1, have
inspired theoretical studies on fractal Sierpinski triangle samples with
different purposes such as exploring spin polarization features Pedersen
(2020) and the emergence of flat bands Pal and Saha (2018); Leykam  _et al._
(2018); Nandy (2029). In particular, extended eigenstates are verified in the
spectrum of hexagonal ST [Fig.1 (d)], revealed as continuous bands as the
lattice is threaded by magnetic fluxes Pal and Chakrabartia (2012). The role
of fractality was already studied for diffusion, and random walks in photonic
lattices Xu  _et al._ (2021), in the synthesis of fractal supramolecular
nanostructures Wang  _et al._ (2018), all with interesting applications.
Recently, The study paves the way for future investigations of correlated
topological systems in fractal dimensions.”

One-dimensional molecular chains using Sierpinski triangles as building blocks
have been reported Zhang  _et al._ (2018) using low-temperature Scanning
Tunnelling Microscopy (STM). The systems were grown on Au(111) and the success
of the chain formation depends on the molecular coverage and matching between
molecular size and surface lattices. Experimental methods using self-
assemblyJiang  _et al._ (2020); Rastgoo-Lahrood  _et al._ (2016); Jian  _et
al._ (2015); Zhang  _et al._ (2015) and templating Li  _et al._ (2017a, 2015,
b) are being used to synthesize ST flakes and nanoribbons with transition
metals and metal-organic composites. Also, 2D crystals consisting of ST units
are reported by exploiting benzene-like molecules and Fe atoms on Au(111) by
combining molecular design and epitaxy control Zhang  _et al._ (2020).

Motivated by the rich experimental scenario based on different molecular
constructions, we propose theoretical models to describe such synthesized
lattices. Analysis of the physical dimension is performed through the imaging
process of the local density of states together with a box-counting method,
revealing fractal dimensions for the studied flake systems, as expected.
Results of local electronic density of states of simplified molecular
structures are compared with spatial charge distribution reported on distinct
STM images. Here we explore electronic and transport properties of such
molecular fractal chains, indicating the viability of tuning electronic states
by construction, providing possible smart devices.

##  II Theoretical Model and Results

A single-orbital tight-binding (TB) Hamiltonian is used to describe the ST
systems studied, given by

$\displaystyle
H=\sum_{i}\varepsilon_{i}c_{i}^{\dagger}c_{i}+\sum_{\left\langle
ij\right\rangle}t_{ij}c_{i}^{\dagger}c_{j}+\sum_{\left\langle\langle
ij\right\rangle\rangle}t^{\prime}_{ij}c_{i}^{\dagger}c_{j}+h.c.$ (1)

with $\varepsilon_{i}$ being the on-site energy for each atom located at site
$i$, $c^{\dagger}_{i}$ ($c_{i}$) is the creation (annihilation) operator of an
electron on site $i$, and $t_{i,j}$ and $t^{\prime}_{i,j}$ are the hopping
energies for nearest and second-nearest neighboring atoms, respectively.
Following the Green function formalism we obtain local density of states and
we also investigate transport properties using the Landauer approach Datta
(1995) in which the system is decoupled into three parts: central conductor
and right and left leadsSantos  _et al._ (2020); Chico  _et al._ (2015a). We
have considered semi-infinite ST chains as leads, matching perfectly with the
central region. The central advanced (a) and retarded (r) Green functions are
given as

$G^{a,r}_{c}(E)=\Big{[}\omega-{H_{c}}-\Sigma^{a,r}_{L}(E)-\Sigma^{a,r}_{R}(E)\Big{]}^{-1}\,\,,$
(2)

with $\omega=E\pm i\eta$, $\eta$ being an infinitesimal number. ${H}_{c}$ is
the Hamiltonian of the central part, and $\Sigma^{a,r}_{L,R}(E)$ correspond to
left and right self-energies, given by the related surface Green functions,
from which the coupling matrices are obtained:

$\Gamma^{L,R}(E)=i\Big{(}\Sigma_{L,R}^{r}(E)-\Sigma_{L,R}^{a}(E)\Big{)}\,\,.$
(3)

Finally, to derive the electronic conductance in ST chains,
G(E)=$2e^{2}\mathcal{T}(E)/h$, we calculate the energy-dependent transmission
given by

${\mathcal{T}(E)}=\operatorname{Tr}\left[\boldsymbol{\Gamma}^{L}\boldsymbol{G}^{r}_{c}\boldsymbol{\Gamma}^{R}\boldsymbol{G}^{a}_{c}\right]\,\,.$
(4)

###  II.1 Conventional Flake model

Inspired on the possibility of engineering real structures based on the
geometries of the Sierpinski triangle we developed a recursive process to
describe ST of different generations, based on the Green’s function formalism.
Our first emphasis is to investigate fractal features on the electronic
properties of ST flakes. Here we show the results of the density of states
(DOS) for conventional ST flakes in terms of the order generation $l$.

Figure 2: Illustration of the decimation method in which a G(2) ST is reduced
to a G(1) ST.

The electronic properties are obtained by using the single first neighbor
tight binding Hamiltonian, described in Eq.(1) and putting $t^{\prime}=0$. The
hopping energy matrices follow the corresponding connectivity of the $l$-order
ST [G(l)], illustrated in Fig.2. The DOS is given by

$\rho_{0}(E)=-\frac{1}{\pi}ImG_{00}(E).$ (5)

The Dyson equations relating the real-space Green functions, among the
15-atoms of the second ST generation, shown in Fig.2, can be written as ,

$\displaystyle G_{00}=g_{0}+g_{0}t[G_{10}+G_{20}+G_{30}+G_{40}]$ (6)
$\displaystyle G_{10}=g_{1}t[G_{00}+G_{20}+G_{60}+G_{50}]$ $\displaystyle
G_{20}=g_{2}t[G_{00}+G_{10}+G_{60}+G_{90}]$ $\displaystyle
G_{30}=g_{3}t[G_{00}+G_{40}+G_{70}+G_{110}]$ $\displaystyle
G_{40}=g_{4}t[G_{00}+G_{30}+G_{70}+G_{80}]$ $\displaystyle
G_{60}=g_{6}t[G_{10}+G_{20}+G_{50}+G_{90}]$ $\displaystyle
G_{70}=g_{7}t[G_{30}+G_{40}+G_{80}+G_{110}]\,\,,$

where the propagator $g_{i}(E)=1/(w-E_{i})$ and $t=t_{i,j}$ corresponds to the
energy hopping between two first neighboring atoms i and j in the lattice,
which were considered identical. After some algebraic manipulations involved
in a decimation procedureWang (1995), and considering $g_{i}=g_{0}$, for all
sites i, we obtain a renormalized Dyson equation for the $G_{00}$ locator,

$G_{00}=\tilde{g_{0}}+\tilde{g_{0}}\tilde{t}[G_{50}+G_{80}+G_{90}+G_{110}]\,\,,$
(7)

corresponding to a reduced ST generation, with the dressed propagator and
hopping energy given by,

$\tilde{g_{0}}=\frac{g_{0}}{1-\frac{4g_{0}^{2}t^{2}(1+g_{0}t)}{1-3g_{0}^{2}t^{2}-2g_{0}^{3}t^{3}}}\,\,,$
(8)

and

$\tilde{t}=\frac{g_{0}t^{2}(1+2g_{0}t)(1+g_{0}t)}{1-3g_{0}^{2}t^{2}-2g_{0}^{3}t^{3}}\,\,.$
(9)

By realizing $l$ iterative process it is possible to obtain the DOS of the
$(l+1)-th$ ST generation. The results for different numbers of iterative
processes (1-4) are shown in Fig.3.

Figure 3: DOS as a function of the Fermi energy, calculated from -2.5t to 4.5t
for $l$-ST varying from 1 up to 4 iterations. Zoom: DOS results within a
narrower energy range(marked in the figure), for $l$=5, 7, 9 and 20
iterations, illustrating the DOS self-similarity.

As being finite systems, the DOS are expected to exhibit a sequence of delta
functions that increase as the number of atoms is increased, as depicted in
Fig.3. A self-similarity of the DOS, emerging from higher $l$-order STs is
evident in the zoom (bottom panel) presenting the short energy range (-1.0t to
-0.75t). Particular pinned localized states are found at the same energies,
independently of the ST generation-order. More interesting to note is the fact
that all states presented in a given iterative process also appear in the LDOS
results of all subsequent iterative steps of higher order, in a cumulative
process. We should also mention that constant energy gaps at different energy
ranges are achieved as the ST flake increases, as shown in the zoom part of
Fig. 3.

###  II.2 Kagomé-Honeycomb ST Flake model

Motivated by fractal properties in more realistic systems we propose the
Kagomé-Honeycomb ST flake (KHST). In this case we solve the tight-binding
system up to second neighbors, as illustrated in Fig.4-(a) for armchair-KHST
(A-KHST) and in Fig.4-(b) for zigzag-KHST (Z-KHST). The number of atoms for
the armchair ($N_{A}$) and zigzag ($N_{Z}$) configurations are, respectively,

$\displaystyle N_{A}^{l}=12*3^{l}+\sum_{i=1}^{l}3^{i}\,\,,$ $\displaystyle
N_{Z}^{l}=15*3^{l}+4-\sum_{i=0}^{l-1}3^{i},\ {\rm for}\ l>1\ {\rm and}\
N_{Z}^{1}=49\,\,.$ (10)

All the calculations are made using on-site energies equal to $\pm\epsilon=\pm
0.25t_{1}$, with $t_{1}$ being the nearest-neighbor hopping, for the blue
($+\epsilon$) and orange ($-\epsilon$) sites, following reports on two-
dimensional covalent organic honeycomb frameworks Silveira  _et al._ (2016),
that are typical example of graphene-kagomé lattices. The second nearest
neighbor hopping energy is chosen as Kempkes  _et al._ (2019)
$t_{2}$=0.08$t_{1}$. The LDOS is calculated as a function of energy by

$LDOS(x,y,E)=\sum_{n}\lvert\Phi_{n}(x,y)\rvert^{2}\delta(E-E_{n})\,\,,$ (11)

where $x$ and $y$ are the lattice position coordinates and $\Phi(x,y)$ is the
corresponding electronic wave function of the $n^{th}$ state. To compute the
delocalized electronic contribution around the sites, given by
$\lvert\Phi_{n}(x,y)\rvert^{2}$ we transform the delta functions into
Lorentzians with $\Gamma=0.08t_{1}$.

Figure 4: Schematic view of the second generation [G(2)] of an (a) A-KHST and
a (b) Z-KHST flake at left and the eigenvalues for the 5th generation as a
function of the eigenstate indexes at right. Results considering only first
and up to second neighbor-hopping energies are depicted in black and green
dots, respectively. Insets: gap sizes versus the ST generation.

For both KHST edge geometries, the presence of a band gap is verified, for the
cases of first and up to second-neighbors, as shown in Fig.4 (right panels),
revealing a semiconducting feature to such Kagomé-like ST. Otherwise, the
graphene ST flake results, reveal a semiconducting and metallic characteristic
for armchair and zigzag GSTs, respectively Pedersen (2020) in the energy-state
maps. As noticeable from Figs.4(a) and (b), the gap increases by $\approx$
0.16$t_{1}$ when second nearest neighbors are taken into account in the TB
model, for armchair and zigzag-edged KHSTs. An interesting feature found in
the electronic properties of the KHST gaskets is the reminiscent flat band
observed in 2-dimensional Kagomé lattice, highlighted here by the high
degeneracy order of the E=-0.25$t_{1}$ and -0.41$t_{1}$ states, for first and
second-neighbor models, respectively, at the gap threshold. The number of such
localized degenerate states is expected to increase as the generation order of
the gasket is increased, although the size of the gaps do not dependent on the
generation order, as depicted in the insets of Fig.4, for armchair and zigzag
configurations. Compared with the gap dependence for GST flakes Pedersen
(2020), for which the gap size saturates for $l$ between 3 and 4, we must
comment that the number of atoms in the KHST flakes are considerable superior
at the same generation order, what may justify the constant gap size achieved
already in the first generation.

Due to the difference in the geometry and site numbers in the basic units
forming the KHSTs in comparison to the conventional ST, another fractal
dimension definition is used Foroutan-pour  _et al._ (1999). Here we adopt the
Minkowski–Bouligand dimension used for monofractals, given by

$D=\lim_{r\to 0}\frac{Log(N(r))}{Log(r^{-1})}\,\,,$ (12)

with $N(r)$ and $r$ being the number of squares covering the full system, and
the square size, respectively. These parameters are used in the box-counting
method Kaurov (2012) to perform the dimension calculation. The dimension of
each system is determined by the line slope coefficient of $Log(N(r))$ vs
$Log(r^{-1})$ graph presented in Fig.5(a).

Starting from the atomic spatial localization in the A-KHST flake (see
Fig.1(e)), the sizes (in pixels) of each square found to better adjust the
slope are 10 up to 90 pixels. In both cases we take the LDOS image (360 x 360
pixels) and binarize it in a threshold of $30\%$, to represent the most
accurate electronic distribution. The details of such numerical process is
described in Refs. Kempkes  _et al._ (2019); Foroutan-pour  _et al._ (1999).
Perfect Kagomé nanoribbons and Kagomé 2D-lattices, exhibit symmetric band
spectra, except for the presence of a flat band located exactly at $\epsilon$
or -$\epsilon$, depending on the hopping energy signal. For this reason, in
this work we choose then the values of $E=\pm\epsilon$ to calculate the
maximum contribution in LDOS indicated by its brightest spots, which gives two
different patterns, as can be seen in the binarized image shown in
Fig.5-(b,c).

Figure 5: (a) Box-counting results of the binarized images (b) and (c) at
$E=\pm 0.25t_{1}$, represented by red and black colors, respectively. LDOS of
a G(3)-A-KHST for (d) first neighbors at $E=\pm 0.25t_{1}$ and (e) second
neighbors at $E=0.25t_{1}$ and $E=-0.41t_{1}$ and corresponding DOS as
function of the energy.

In this system we have two energy values that the maximum contribution in LDOS
is spread through the flake, given origin to the patterns (b) and (c) in
Fig.5. Taking them both in the analysis, the box counting method yields a
fractal dimension close to the conventional Sierpinski triangle gasket,
$\approx 1.585$, as can be verified by the slope values shown in Fig.5(a), for
both situations ($1^{st}$ and up to $2^{nd}$ neighbor hopping). This occurs
because the brightest spots in the LDOS maps are near from the geometrical
points of a A-GST flake, that are also fractal. Otherwise, for energies where
the bright spots (high LDOS values) do not reveal the real geometry of the
flake, the fractal dimension varies between D=1.30-1.80, as verified by
Kempkes et al.Kempkes  _et al._ (2019) in hexagonal flakes, varying between
low and high contributions of each site for the LDOS, respectively. The self-
similarity observed in others ST flakes Kempkes  _et al._ (2019); Pedersen
(2020) is also evidenced here for both configurations of the KHSTs (armchair
and zigzag), independently on the order of the neighboring hopping taken into
account.

A further interesting feature to explore in zigzag and armchair KHST flakes,
described by the first and second neighbors approximation, is the asymmetric
distribution of the electronic probability distribution at
$E_{1}$=-0.25$t_{1}$ and $E_{2}$=0.25$t_{1}$ states. Considering the first
generation G(1), we found that while the $E_{1}$ state is formed by four-fold
degenerate eigenstates, the $E_{2}$ state is non-degenerated. These states are
spread differently among the orange and blue sites of the flake [see Fig. 4] ,
giving raise to the LDOS exhibited in Fig.5(d) and (e), respectively. The same
behavior is evidenced for higher orders of the KHSTs, considering both first
and up to second-neighbor hopping models. We would like to emphasize that the
inclusion of second neighbor hopping in our tight binding description did not
promote significant changes on the main physical responses explored for the
Kagomé-like ST flakes.

###  II.3 ST Mirrored Chains

Following the experimental realization of 1-D molecular chains Zhang  _et al._
(2018), with ST as building blocks grown on Au(111), we explore electronic
properties of a similar quasi-1D chain as depicted in Fig 6(a). The system was
idealized based on the coupling of two hexagonal STs spatially inverted (up
and down), with mirror symmetry. Green and black lines connect nearest and
second-nearest neighboring atoms, respectively, through the hopping terms $t$
and $t^{\prime}$ in the tight binding Hamiltonian given in Eq.(1). Differently
from the long-range ordered grown structures where Co atoms are present in
intercalated benzene latticesZhang  _et al._ (2018), our simple model involves
exclusively carbon atoms, denoted by blue dots. The electronic properties are
calculated using on site energy $\epsilon=0$ and following the relation
t’=0.08t for the hopping parameters as recently for HSTs Kempkes  _et al._
(2019).

Figure 6: (a) Effective G(1)-ST Chain. The green and black lines are related
to hopping t and t’, respectively. The system extends infinitely along the
x-direction. (b) LDOS in function of the energy at sites 1,2,3,5 and 7
identified and colored circled inside the red dashed unit-cell of the ST-
Chain, defined in part (a). The total DOS is also shown with pink shadow
regions. (c) Band structure of the ST chain for $\epsilon$=0.

The LDOS shown in Fig.6-(b) is calculated via Eq.5, where $G_{00}$ is obtained
using similar real-space decimation methods, properly constructed to infinite
periodic systems Rosales  _et al._ (2008); Chico  _et al._ (2015b); Torres
_et al._ (2021). At energies close to the Fermi level, the electronic group
velocity is near zero, giving origin to almost two dispersionless bands as can
be seen in the corresponding electronic structure shown in Fig.6-(c). These
flat bands appear as highly peaked density of states, and are highlighted in
the inset in Fig.6-(b). The different curves correspond to the assigned sites
marked with the same color in the unit cell displayed in part (a). As seen in
Fig.6-(b), at E=-0.16$t$, the main contribution becomes from sites 1, 2 and 3,
and from the symmetric upper sites 11, 10 and 9.

A better visualization of the spatial electronic distribution through the
system is displayed in Fig.7 for distinct energies. While part (a) refers to a
STM image adapted from ref. Zhang  _et al._ (2018), Figs.7 (b), (c), and (d),
are the LDOS theoretical results at $E=-0.16t$ (flat band), $0.96t$ and
$1.65t$. In the STM image a high charge distribution appears as bright spots
at the relative atomic sites. Correspondingly, in our results the color maps
refer to the LDOS intensity. The normalized LDOS results shown in Fig.7 (b)
reveal that at one of the flat bands [E=-0.16t] not all the atomic sites of
the lattice are populated, in according with the previously discussion. This
last state appears again in the Dephased ST Chain, and in special, it will not
contribute to the transport properties of these systems, as we will see in the
next session.

Figure 7: LDOS contour plots of a G(1)-ST Chain. a) STM image adapted from the
experimental data reported in Ref.Zhang  _et al._ (2018). Our theoretical LDOS
results for (b) E=-0.16$t$, (c) 0.96$t$, and (d) 1.65$t$.

Moreover, while at $E=0.96t$, only sites 2 and 5, and their equivalents in the
unit cell, contribute to the state [see Fig.6-(c)], revealing a restricted
charge distribution, for the state $E=1.65t$, Fig.6-(d), almost all the
lattice is visited generating bright spots in the geometrical net. The later
LDOS pattern strongly resembles the cited STM image [Fig.7-(a)] indicating
that our simple model can be an important tool of modelling such organic
molecular chain.

Figure 8: (a) STM images of the MSTC-2 [Adapted from Li  _et al._ (2017c)].
(b) LDOS of the MSTC for a Fermi energy equal to 2.4t. The nanoribbon unit
cell is highlighted with red dots. (c) DOS and conductance results of the MSTC
and MSTC-2 in blue and gray, respectively.

Another experimental alternative of generating 1D mirrored chains made of
Sierpiński triangles was proposed in Ref.Li  _et al._ (2017c). We call it as
mirrored chain structure-type 2 (MSTC-2). In our model up and down triangles
are now fully preserved resembling the STM images shown in Fig. 8(a). The LDOS
at a particular energy ($E=2.4t$) is depicted in part (b) where the region
enclosing the nanoribbon unit cell is marked with red circles. The results for
DOS and conductance are shown in Fig.8 (c), highlighted with shadowed blue
curves and compared with the results found for the previous molecular chain
[shaded gray curves]. It is noticeable the emergence of two narrowed energy
states near the Fermi Level for the MSTC-2, this is a characteristic property
of double ST Chains that will be explored in next section. Also, the flat-like
states in both chain configurations are preserved in the central gap in DOS,
although they are suppressed in the electronic conductance response of each
molecular designs due to the high localization features of the flat band.

###  II.4 Dephased ST Chain

Following the experimental molecular structures presented in Ref. Zhang  _et
al._ (2018), we address now another proposal for ST chain, as illustrated in
Fig. 9-(a). Differently from the previous discussed molecular chains, the new
structure are composed by dephased pairs of HSTs [G(2)], connected at
particular lateral lattice sites, simulating the packing mode of STs produced
by a combination of a Co atom and three BPyB moleculesZhang  _et al._ (2018).
The unit cells of the dephased molecular chain is marked with dashed red
lines. It is interesting to note that our theoretical result for the LDOS at
$E=1.4t$, as shown in Fig.9-(d), reproduces quite well the STM image presented
in Fig.9-(c) for the grown molecular nanostructured, reported in Ref.Zhang
_et al._ (2018).

The electronic properties of the G(2)-dephased ST is shown in Fig. 9-(b) via
the density of states and conductance results. Due to the high electronic
localization, the DOS peak at E=-0.16t does not contribute to the electronic
conductance in G(2), what happens also for dephased chains of higher ST orders
(not shown here).

Figure 9: (a) Effective G(2)-dephased ST chain, (b) Conductance and DOS, (c)
STM image [adapted from Ref. 15], and (d) LDOS at E=1.4t.

To investigate fractal properties on such nanostructured systems we
concentrate now in the electronic band structures. We present in Fig. 10 (a-c)
unit cells of three generations of the proposed molecular chain discussed in
Fig.9, G(2), G(3) and G(4), and the corresponding band structure of the 1D
chains. The dashed regions marked in the band structures presented in
Fig.10(a), related to the G(2) lattice, was enlarged in the subsequent panels
at right, revealing a single flat band and other two bands presenting a small
energy dispersion, at the lower energy range, highlighted by the red shaded
area. These later are also noted in the DOS and conductance results (not
shown). To analyze the role played by the generation order on the used HST
blocks forming the chains, we show in Fig.10 (b) and (c), for comparison, the
energy bands of the three G(3) and G(4) dephased ST chains. The auto
similarity of these systems can be evidenced looking firstly, at the same
energy region of the three generation proposed, defined by the dashed energy
range. Comparing the results, we notice in the last right panels that the
common pattern of a single flat band (happening at 0.96t in the three cases),
followed by two dispersive bands [ending at 1.1t (a), 1.0t (b) and 0.97t (c),
respectively] is slightly preserved going from G(2) to G(4). As expected, for
increasing ST generations, the repeated band structure pattern are found at
smaller energy ranges.

Figure 10: (a-c) Unit cells of G(2-4) Dephased ST Chains and corresponding
band structures, followed by its zooms, from left to right.

Moreover, the results for the energy bands reveal an increasing number of
1D-like van Hove singularities passing from G(2) to G(4) dephased chains.
Similarly as reported before Pal and Saha (2018), the number of flat bands
increases significantly with the ST order used to build the chains. Our
findings suggest that properly tuning the tight binding parameters and playing
with different connections between and inside the unit cells, better modeling
of such nanostructured systems may be achieved. In particular, considering
different on-site energies for Co atoms and BPyB molecules in our model (for
instance taking $E_{C}=-E_{Co}$), simplified here as carbon atoms, only
promotes a shift in the double flat states near the fermi level. We would like
also to draw attention to the fact that temperature and different types of
lattice imperfections, such as edge roughness, may limit the experimental
determination of the gap hierarchy gaps with decreasing size upon increasing
the order generation of the structure, mainly due to a loss of quantum
coherence and the consequent inhomogeneous broadening of electronic levels. As
previously addressed in Ref. Maciá and Adame (1996) these factors can
introduce a critical size of the system, above which finer details of the
fractal electronic spectrum may be difficult to the revealed.

##  III Conclusion

We have explored fractal features in the electronic properties of ST flakes
and molecular chains of different geometry details. The results found for the
fractal dimension of the Kagomé-like ST flakes follow recent reports on
graphene-like STs. The result is corroborated by the Kagomé ST LDOS at
particular energy states that exhibit typical graphene ST spatial
configurations. The self similarity of the energy states are found comparing
different ST generation orders and also amplifying the energy ranges
investigated, for both flakes and quasi 1D systems. In particular, the results
for the local density of states of the theoretical molecular chains proposed
here exhibit quite similar spatial charge distribution as experimental STM
reports. The analysis of transport response of such quasi 1D molecular chains
reveals localized states that do not contribute to the electronic transport.
The study can be used as guide to propose a variety of architecture in the
synthesis of real molecular chains.

Further aspects of the theoretical framework adopted were explored such as
changing the inter and intra unit cell connections of the proposed molecular
chains. Some of the schemes induce degeneracy break of particular electronic
states, favoring the disrupt of flat bands. Although a primary analysis on
more realistic on-site energies, simulating metallic atoms in the ST chains
that appear in the experiments, has not indicated great changes, more
sophisticated theoretical framework must be used for a fully study of tuning
the energy channel positions in the transport features of the chains.
Probably, the inclusion of electron-electron interaction in the model will
bring important light into the electronic occupation in conductive states, in
special, in the flat states presented.

## Acknowledgments

This work was financially supported by the Brazilian Agencies CAPES, CNPq and
by FAPERJ under the grant E-26/202.567/2019, and the INCT de Nanomateriais de
Carbono.

## References

  * Chandel  _et al._ (2020) V. S. Chandel, G. Wang, and M. Talha, Nanotechnology Reviews 9, 230 (2020).
  * Li  _et al._ (2022) X. Li, D. Han, T. Qin, J. Xiong, J. Huang, T. Wang, H. Ding, J. Hu, Q. Xu, and J. Zhu, Nanoscale 14, 6239 (2022).
  * Lisi  _et al._ (2021) S. Lisi, X. Lu, T. Benschop, T. A. de Jong, P. Stepanov, J. R. Duran, F. Margot, I. Cucchi, E. Cappelli, A. Hunter, A. Tamai, V. Kandyba, A. Giampietri, A. Barinov, J. Jobst, V. Stalman, M. Leeuwenhoek, K. Watanabe, T. Taniguchi, L. Rademaker, S. J. van der Molen, M. P. Allan, D. K. Efetov, and F. Baumberger, Nature Physics 17, 189 (2021).
  * Cao  _et al._ (2018) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Nature 556, 80 (2018).
  * Rhim  _et al._ (2020) J.-W. Rhim, K. Kim, and B.-J. Yang, Nature 584, 59 (2020).
  * Wolf  _et al._ (2019) T. M. R. Wolf, J. L. Lado, G. Blatter, and O. Zilberberg, Phys. Rev. Lett. 123, 096802 (2019).
  * J.  _et al._ (2020) M. J., M. S.P., and A. M. et al., Nature 584, 215 (2020).
  * Kempkes  _et al._ (2019) S. N. Kempkes, M. R. Slot, S. E. Freeney, S. J. M. Zevenhuizen, D. Vanmaekelbergh, I. Swart, and C. M. Smith, Nat. Phys. 15, 127–131 (2019).
  * Manna  _et al._ (2020) S. Manna, B. Pal, W. Wang, and A. E. B. Nielsen, Physical Review Research 2, 023401 (2020).
  * Pedersen (2020) T. Pedersen, Phys. Rev. B 101, 1–7 (2020).
  * Pal and Saha (2018) B. Pal and K. Saha, Phys. Rev. B 97, 195101 (2018).
  * Leykam  _et al._ (2018) D. Leykam, A. Andreanov, and S. Flach, Adv. in Phys. 3, 677 (2018).
  * Nandy (2029) A. Nandy, Physica Scripta 86, 045802 (2029).
  * Pal and Chakrabartia (2012) B. Pal and A. Chakrabartia, Eur. Phys. Journ. B 85, 307 (2012).
  * Xu  _et al._ (2021) X.-Y. Xu, X.-W. Wang, D.-Y. Chen, C. M. Smith, and X.-M. Jin, Nature Photonics 15, 1749 (2021).
  * Wang  _et al._ (2018) L. Wang, R. Liu, J. Gu, B. Song, H. Wang, X. Jiang, K. Zhang, X. Han, X.-Q. Hao, S. Bai, M. Wang, X. Li, B. Xu, and X. Li, Journal of the American Chemical Society 140, 14087 (2018).
  * Zhang  _et al._ (2018) X. Zhang, G. Gu, N. Li, H. Wang, H. Tang, Y. Zhang, S. Hou, and Y. Wang, RSC Adv. 8, 1852 (2018).
  * Jiang  _et al._ (2020) Z. Jiang, D. Liu, M. Chen, J. Wang, H. Zhao, Y. Li, Z. Zhang, T. Xie, F. Wang, X. Li, G. R. Newkome, and P. Wang, Science 23, 101064 (2020).
  * Rastgoo-Lahrood  _et al._ (2016) A. Rastgoo-Lahrood, N. Martsinovich, M. Lischka, J. Eichhorn, P. Szabelski, D. Nieckarz, T. Strunskus, K. Das, M. Schmittel, W. M. Heckl, and M. Lackinger, ACS Nano 10, 12 (2016).
  * Jian  _et al._ (2015) S. Jian, W. Yongfeng, C. Min, D. Jingxin, Z. Xiong, K. Julian, H. Gerhard, S. Xiang, G. J. Michael, and W. Kai, Nature Chemistry 7, 389 (2015).
  * Zhang  _et al._ (2015) X. Zhang, N. Li, G.-C. Gu, H. Wang, D. Nieckarz, P. Szabelski, Y. He, Y. Wang, C. Xie, Z.-Y. Shen, J.-T. Lü, H. Tang, L.-M. Peng, S.-M. Hou, K. Wu, and Y.-F. Wang, ACS Nano 9, 11909 (2015).
  * Li  _et al._ (2017a) N. Li, G. Gu, X. Zhang, D. Song, Y. Zhang, B. K. Teo, L.-M. Peng, S. Hou, and Y. Wang, Chem. Commun. 53, 3469 (2017a).
  * Li  _et al._ (2015) N. Li, X. Zhang, G.-C. Gu, H. Wang, D. Nieckarz, P. Szabelski, Y. He, Y. Wang, J.-T. Lü, H. Tang, L.-M. Peng, S.-M. Hou, K. Wu, and Y.-F. Wang, Chinese Chemical Letters 26, 1198 (2015).
  * Li  _et al._ (2017b) N. Li, G. Gu, X. Zhang, D. Song, Y. Zhang, B. K. Teo, L.-M. Peng, S. Hou, and Y. Wang, Chem. Commun. 53, 3469 (2017b).
  * Zhang  _et al._ (2020) Y. Zhang, X. Zhang, Y. Li, S. Zhao, S. Hou, K. Wu, and Y. Wang, J. Am. Chem. Soc. 142, 17928 (2020).
  * Datta (1995) S. Datta,  _In Electronic Transport in Mesoscopic Systems_ (Cambridge, 1995).
  * Santos  _et al._ (2020) H. Santos, A. Latgé, L. Brey, and L. Chico, Carbon 168, 1 (2020).
  * Chico  _et al._ (2015a) L. Chico, A. Latgé, and L. Brey, Phys. Chem. Chem. Phys. 17, 16469 (2015a).
  * Wang (1995) X. R. Wang, Phys. Rev. B 51, 9310 (1995).
  * Silveira  _et al._ (2016) O. J. Silveira, S. S. Alexandre, and H. Chacham, J. Phys. Chem. C 120, 19796 (2016).
  * Foroutan-pour  _et al._ (1999) K. Foroutan-pour, P. Dutilleul, and D. Smith, Applied Mathematics and Computation 105, 195 (1999).
  * Kaurov (2012) V. Kaurov, Measuring Fractal Dimension of Natural Object from Digital Images, Mathematica Stack Exchange (2012).
  * Rosales  _et al._ (2008) L. Rosales, M. Pacheco, Z. Barticevic, A. Latgé, and P. A. Orellana, Nanotechnology 19, 065402 (2008).
  * Chico  _et al._ (2015b) L. Chico, A. Latgé, and L. Brey, Phys. Chem. Chem. Phys. 17, 16469 (2015b).
  * Torres  _et al._ (2021) V. Torres, D. Faria, and A. Latgé, Phys. Rev. B 103, 115437 (2021).
  * Li  _et al._ (2017c) N. Li, G. Gu, X. Zhang, D. Song, Y. Zhang, B. K. Teo, L.-m. Peng, S. Hou, and Y. Wang, Chem. Commun. 53, 3469 (2017c).
  * Maciá and Adame (1996) E. Maciá and F. D. Adame, Semicond. Sci. Technol. 11, 1041 (1996).


# Prompt-guided Scene Generation for
3D Zero-Shot Learning

Majid Nasiri1, Ali Cheraghian2,5, Townim Faisal Chowdhury3, Sahar Ahmadi1,
Morteza Saberi4, Shafin Rahman3 1Business school, The University of New South
Wales, Australia,
2School of Engineering, Australian National University, Australia,
3Dept. of Electrical and Computer Engineering, North South University,
Bangladesh,
4 School of Computer Science and DSI, University of Technology Sydney,
Australia
5 Data61, Commonwealth Scientific and Industrial Research Organisation,
Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>{townim.faisal,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Zero-shot learning on 3D point cloud data is a related underexplored problem
compared to its 2D image counterpart. 3D data brings new challenges for ZSL
due to the unavailability of robust pre-trained feature extraction models. To
address this problem, we propose a prompt-guided 3D scene generation and
supervision method that augments 3D data to learn the network better,
exploring the complex interplay of seen and unseen objects. First, we merge
point clouds of two 3D models in certain ways described by a prompt. The
prompt acts like the annotation describing each 3D scene. Later, we perform
contrastive learning to train our proposed architecture in an end-to-end
manner. We argue that 3D scenes can relate objects more efficiently than
single objects because popular language models (like BERT) can achieve high
performance when objects appear in a context. Our proposed prompt-guided scene
generation method encapsulates data augmentation and prompt-based
annotation/captioning to improve 3D ZSL performance. We have achieved state-
of-the-art ZSL and generalized ZSL performance on synthetic (ModelNet40,
ModelNet10) and real-scanned (ScanOjbectNN) 3D object datasets.

###### Index Terms:

Zero-shot learning, 3D point cloud, Scene generation

##  I Introduction

Modern object recognition systems based on deep learning models require
extensive labelled datasets to perform better. However, in some fields, such
as healthcare, or scenarios, such as COVID-19, collecting significant amounts
of annotated data is tricky. To address this issue, several methods [1, 2, 3,
4, 5] develop Zero-Shot Learning (ZSL) methods on 2D image data by
transferring the knowledge of seen data to unseen classes with the help of
semantic information obtained from pre-trained object attributes or a
substantial corpus of texts. Recent achievements in ZSL on image domain [6]
and advancements in camera technology have motivated researchers to address
the ZSL problem in the 3D domain [7, 8, 9] too. This task is more challenging
than 2D image data because of the complex and unordered properties of 3D point
cloud data [9]. Moreover, collecting 3D data, scarce objects, is complex and
costly. In this paper, we aim to propose a novel ZSL method dedicated to only
3D object scenarios.

Unlike 2D image models (VGG [10], ResNet [11]), pre-trained 3D point cloud
models (Pointnet++ [12], PointNet [13]) cannot provide rich quality features
for classification. Earlier efforts of 3D ZSL [7, 8, 14] point out this
problem mentioning that 3D datasets contain a limited number of classes,
whereas 2D datasets include thousands of categories. For this, the 3D ZSL
problem inevitably exhibits poor visual-semantic alignment, hubness, and bias.
All existing works attempt to minimize those problems by proposing triplet,
hubness and biasing losses in inductive and transductive settings [7, 8, 9].
Similarly, replacing 2D feature extraction backbones with compatible 3D
models, [14] reported 2D ZSL methods’ performance on 3D data. However, none of
the methods mentioned above utilizes the 3D nature of the input data. In this
paper, we learn different object categories based on dynamically generated 3D
scenes from pre-determined language prompts [15] treated as scene captions.

Figure 1: Instead of the traditional use of the single point cloud model vs.
single word-based semantics, we train our model based on dynamically generated
3D scenes and prompt-based captions. Here, using ObjectA (guitar), ObjectB
(piano), and prompt templates (italicized sentences), we generated multiple
scenes and their captions on the right.

In recent years, contrastive learning is showing promising results in
combining vising and language tasks [16]. Here, language models (like BERT)
can better describe a concept when a context comes into play. 2D image domain
could easily take this advantage because of the availability of large-scale
datasets. In contrast, since 3D point cloud datasets are not as rich as their
2D counterparts, the 3D domain cannot efficiently utilize context benefits.
Moreover, obtaining 3D scene captions is a big challenge. To address this, we
propose a simple but effective dynamic 3D scene generation and annotation
method, particularly useful for the 3D ZSL problem. At each training
iteration, following some pre-defined prompt templates (having two
placeholders for two random classes), we dynamically construct a set of new 3D
scenes (see Fig. 1). A pre-trained BERT model extracts a word vector
description from the prompt text, and a 3D point cloud backbone (PointNet
trained from scratch) provides point cloud features. Then, a contrastive
learning strategy has been applied to the entire 3D feature and word vector
pairs. Note, unlike 2D cases, we do not use a large corpus of text annotations
for its difficulty in obtaining annotated captions for 3D data. Instead, we
use pre-defined prompt-based captions to annotate the scene, which may not be
the perfect caption. For example, ‘a bed is under the radio’ can be a caption
that may be unrealistic in natural language but included in our training.
Nevertheless, dynamically generated scenes could provide somewhat robust
context information to the network exploring detailed inter-relation of seen
categories. Notably, this approach can explore more semantic relations than
the traditional use of employing one 3D model and its word vector pair at a
time. In this way, our method relaxes the data scarcity problem by generating
many two object scenes and their prompt-based captions. It eventually improves
point cloud features vs. semantic alignment to generalize the training,
especially for generalized ZSL problems. We experiment with two ZSL setups
using 3D point cloud datasets, ModelNet40 [17], ModelNet10[17] and
ScanObjectNN [18], and report state-of-the-art results. The contributions of
this paper are:

  * •

dynamic 3D scene generation and prompt-based annotation methods for 3D ZSL
problem,

  * •

contrastive learning strategy using prompt-based 3D scene annotations,

  * •

extensive experiments on both synthetic (ModelNet40, ModelNet10) and real
(ScanObjectNN) datasets. Moreover, results are compared with both 2D image and
3D point cloud methods.

##  II Related work

Zero-shot learning: In general, two main approaches are used to solve the ZSL
problem: Embedding-based methods [19, 20, 21, 22] and Generative model-based
methods [23, 24, 25, 26]. In the first category, image features and semantic
attributes are mapped to a shared embedding space by learning a projection
function using deep networks. Frome et al. [20] proposed a deep visual-
semantic embedding model (DeViSE) to identify visual objects using both
labeled image data as well as semantic information gleaned from the
unannotated text. This model uses textual data to learn semantic relationships
between labels and explicitly maps images into a rich semantic embedding
space. Learning a linear compatibility function is not suitable for the fine-
grained classification problem. Accordingly, some methods are suggested for
learning nonlinear embedding. Whereas [22] learns a collection of linear
models while allowing each image-class pair to choose from them. Generative
model-based methods try to overcome the problem of bias and domain shift [27]
in the first method by generating image features for unseen classes using
semantic attributes. Mishra et al. [25] proposed to model the statistical
image generation process using a conditional variation auto-encoder (CVAE)
[28] and generate samples for unseen classes. In [26], by obtaining the
semantic relationship between the seen and unseen classes by introducing
Semantic Regularized Loss (SR-Loss), LsrGAN generates visual features that
maintain the same semantic relationship between both classes.

Zero-shot learning on 3D point cloud data: Despite the significant success of
the ZSL problem in 2D images, this remains a challenging issue in 3D objects.
[7]is a pioneering work in solving ZSL problem in 3D point cloud. In this
work, the PointNet [13] architecture is used to extract a feature space, and a
bilinear compatibility function is applied to associate the point cloud
feature vector with the corresponding semantic information. Given that in [7],
feature space is used as embedding space, the model becomes biased in
predicting a few specific labels for most of the test samples. This phenomenon
is called hubness and occurs more frequently in high-dimensional data[29]. In
[8], in order to solve this problem, a new loss function consisting of a
regression term and a skewness term is introduced. In [14], for the first
time, a new triple loss function was introduced to solve the transductive ZSL
and generalized zero-learning (GZSL) in 3d point cloud classification.This
loss function uses unlabeled data in an unsupervised manner and has the
ability to expand into 2D images. In [30], unlike the methods mentioned so
far, a generative ZSL method is introduced that is used for both
classification and semantic segmentation in 3D objects.

Prompt based learning: Prompt based learning is common unsupervised approach
to train a language model for NLP tasks. GPT [31, 32] models are used prompt
based learning to achieve zero shot and few shot performances. In
classification based tasks [33, 34], prompt templates are reasonably easy to
construct and helpful when training samples are few. Prompt-based learning is
still not prevalent in computer vision tasks. Tsimpoukelli et al. [15] used
prompt augmentation technique with a fixed pretrained language model for
training visual encoder model. However, in our method, we apply prompt-based
learning to train a visual model to attain zero shot performance on 3D point
cloud data.

##  III Method

Figure 2: Training and inference pipeline of proposed method. In the training
stage, a batch of generated scenes $\mathcal{G}_{i}\in{\rm I\\!R^{3}}$ are fed
into the point cloud encoder $\mathcal{H}$ to extract the feature embedding
$\textbf{h}_{i}\in{\rm I\\!R^{m}}$, then the extract feature representation
$\textbf{h}_{i}$ is forwarded into the projection module $\mathcal{P}_{p}$ in
order to map into a common space, where the feature representation is
$\textbf{z}_{i}\in{\rm I\\!R^{u}}$. Similarly, for each generated scenes,
there is a prompt description $\mathcal{T}_{i}$. Next, these prompt
description are forwarded into a text encoder, i.e. BERT, to obtain a feature
embedding $\textbf{e}_{i}\in{\rm I\\!R^{d}}$. After this step, the feature
embedding $\textbf{e}_{i}$ is mapped into $\textbf{v}_{i}\in{\rm I\\!R^{u}}$.
After calculating $\textbf{z}_{i}$ and $\textbf{v}_{i}$ for each sample in the
batch, a smility matrix is calculated for the all samples in the batch. Then,
this simsilty matrix is optimized based on a loss function. In the infernce
stage, a sample $\mathcal{X}^{u}$ is forwarded into the point cloud pipeline
to extract $\textbf{z}_{i}$. From the language side, all prompts of seen and
unseen classes are forwarded into the text pipeline, which result into the
feature representation $v_{j}$. Finally, a similarity function applied between
point cloud feature and prompt description of all classes to find the correct
class.

Motivation: ZSL methods designed for the image domain typically benefit from
strong pre-trained models, like ResNet, trained on ImageNet, which consists of
millions of labeled images with thousands of categories. Thus, the extracted
2D features are nicely clustered. Nevertheless, there is no counterpart pre-
trained model in the 3D domain. Labeled 3D datasets are usually small and
contain only limited sets of classes. For example, the Modelnet40 dataset
consists of 40 classes with only a few thousand labeled samples. As a result,
the pre-trained point cloud models such as PointNet trained on Modelnet40
generate poor-quality 3D features for novel classes with clusters that are not
as well-separated as their visual counterparts [9]. To address this issue, we
generate more 3D point cloud samples for training the pre-trained model in
this paper. Specifically, we generate synthetic point cloud scenes with their
associated scenes prompt from the semantic domain. More specifically, in our
approach, the context and relationship between objects are also considered,
which helps our model for better generalization on unseen point cloud classes.
Note, in conventional ZSL approaches [19, 20, 21, 24, 25, 26], a single object
and its associated semantic class embedding are used to train a ZSL model.

###  III-A Revisiting traditional ZSL

Suppose a 3D point cloud object instance is defined as
$\mathcal{X}=\\{\mathbf{x}_{i}\\}_{i=1}^{n}$,
$\textbf{x}_{i}\in{\mathbb{R}^{3}}$.
$\mathcal{Y}^{s}=\\{y_{1}^{s},...,y_{S}^{s}\\}$ and
$\mathcal{Y}^{u}=\\{y_{1}^{u},...,y_{S}^{u}\\}$ are seen and unseen class
label sets with sizes $S$ and $U$ respectively. Here, seen and unseen labels
are disjoint, i.e., $\mathcal{Y}^{s}\cap\mathcal{Y}^{u}=0$. Additionally,
$\mathcal{E}^{s}=\\{\phi(y_{1}^{s}),...,\phi(y_{S}^{s})\\}$ and
$\mathcal{E}^{u}=\\{\phi(y_{1}^{u}),...,\phi(y_{U}^{u})\\}$ are the sets of
semantic feature embedding for the embedding function $\phi(\cdot)$, where
$\phi(y)\in\mathbb{R}^{d}$. To this end, we define the set of $n_{s}$ seen
samples as
$\mathcal{D}^{s}=\\{(\mathcal{X}_{i}^{s},y_{i}^{s},\textbf{e}_{i}^{s})\\}_{i=1}^{n_{s}}$,
where $\mathcal{X}_{i}^{s}$ is the $i$th instance of the seen set with ground
truth $y_{i}^{s}\in\mathcal{Y}^{s}$ and semantic vector
$\mathbf{e}_{i}^{s}=\phi(y_{i}^{s})\in\mathcal{E}^{s}$. Similarly, the set of
$n_{u}$ unseen samples is defined as
$\mathcal{D}^{u}=\\{(\mathcal{X}_{i}^{u},y_{i}^{u},\textbf{e}_{i}^{u})\\}_{i=1}^{n_{u}}$,
where $\mathcal{X}_{i}^{u}$ is the $i$th sample of the unseen set with ground
truth $y_{i}^{u}\in\mathcal{Y}^{u}$ and semantic vector
$\mathbf{e}_{i}^{u}=\phi(y_{i}^{u})\in\mathcal{E}^{u}$. The aim of ZSL in the
traditional settings is to learn a prediction function $f$ as below from the
seen set $\mathcal{D}^{s}$,

$f(\mathcal{X};\textbf{W})=\arg\max_{{y}\in\mathcal{Y}^{s}\cup\mathcal{Y}^{u}}F(\mathcal{X},{y};\textbf{W}),$
(1)

where $F$ is a score function that ranks the correct target higher than the
incorrect ones, and W is the trainable parameters of $F$. The goal is to
maximizing the performance of $F$ on test samples of seen and unseen sets. $F$
is usually takes the following form [35, 36, 37],

$F(\textbf{x},{y};\textbf{W})=\theta(\mathcal{X})^{\top}W\phi({y}),$ (2)

where $\theta(\mathcal{X})$ and $\phi({y})$ are the visual and the semantic
embedding, respectively. $F$ is usually learnt by minimizing the following
loss function,

$L=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\mathcal{L}\left({y}_{i},f\left(\mathcal{X}_{i};\textbf{W}\right)\right)+\gamma\Omega(\textbf{W}),$
(3)

where $\mathcal{L}$ is usually a cross-entropy loss function to learn the
association linking the visual and the semantic domains, and $\Omega$ is the
regularization term employed to restrain the complexity of the model.

Prompt
---
This is a {Object}. | {ObjectA} is close to {ObjectB}.
A big {Object}. | A big {ObjectA} is close to {ObjectB}.
A small {Object}. | A small {ObjectA} is close to {ObjectB}.
Two {Objects}. | {ObjectA} is on {ObjectB}.
Two close {Objects}. | {ObjectA} is under {ObjectB}.
TABLE I: The set of prompts used for generating 3D point cloud scenes.

###  III-B Prompt-guided 3D scene generation for ZSL

In this paper, to address the ZSL problem, instead of using directly the seen
set $\mathcal{D}^{s}$, we generate a new set of samples, called synthetic
scenes, to train our proposed framework. To be more accurate, the synthetic
scenes are generated from the combinations of the objects in the seen set
$\mathcal{D}^{s}$. In order to generate semantically meaningful scenes, we use
a set of predefined prompts $\mathcal{P}=\\{\mathbf{p}_{i}\\}_{i=1}^{M}$ with
the size $M$, which are created manually in a prepossessing step. The
generated scene samples consist of one or two samples from seen set. To be
more specific, to generate new scenes $\mathcal{G}_{i}^{s}$, we linearly
combine point cloud samples $\mathcal{X}_{i}^{s}$ of the seen set
$\mathcal{D}^{s}$,

$\mathcal{G}_{i}^{s}=[\alpha_{j}*{T}(\mathcal{X}_{j}^{s})+\beta_{j}]+[\alpha_{k}*{T}(\mathcal{X}_{k}^{s})+\beta_{k}]$
(4)

where ${T}$ is a random augmentation function, $\alpha$ is the scaling factor,
and $\beta$ is the translation factor. It is essential to mention that
$\alpha$ and $\beta$ are defined based on the prompt semantic meaning. If a
prompt consists of two objects, we down-sample the generated scene to make its
number equal to the number of point of a single object. We define a set of
${\alpha_{small},\alpha_{big}}$ which are chosen based on the prompt to change
the size of the objects. To be more specific, the $\alpha_{small}$ is smaller
than one, and $\alpha_{big}$ is greater than one. At the end of this stage, we
create a new training set,
$\mathcal{S}=\\{\mathcal{G}_{i}^{s},\textbf{p}_{i}^{s}\\}_{i=1}^{n_{d}}$,
which consists of synthetic scenes $\mathcal{G}_{i}^{s}$ and their associated
prompt description $\textbf{p}_{i}^{s}$. In the training stage, the scene set
$\mathcal{S}$ is only used. The prompts used in this paper is shown in Table
I.

###  III-C Language inspired contrasting learning

The proposed architecture is shown in Figure 2. In the point cloud pipeline,
the generated scene sample $\mathcal{G}^{s}$ is forwarded into a point cloud
encoder $\mathcal{H}$ to extract a feature embedding $\textbf{h}_{i}\in{\rm
I\\!R^{m}}$. Then, a projection function, $\mathcal{P}$, which consist of a
few fully connected layers, is employed to map point cloud embedding features
$\textbf{h}_{i}$ into a common space $\textbf{z}_{i}\in{\rm I\\!R^{u}}$.
Similarly, in the text pipeline, a text encoder, $\mathcal{F}$, is used to
project the text prompt $\mathcal{T}^{s}_{i}$ to an embedding space,
$\textbf{p}_{i}\in{\rm I\\!R^{d}}$, where
$\textbf{p}_{i}=\phi({\mathcal{T}})$. After that, a projection function,
$\mathcal{P}_{t}$ is applied to forward the feature embedding $\textbf{p}_{i}$
into a common space $\textbf{v}_{i}\in{\rm I\\!R^{u}}$. In order to train the
proposed architecture, a minibatch of $N$ examples are randomly selected from
synthetic scenes, where the contrastive prediction task on pairs of point
cloud and prompt examples are derived from the minibatch, resulting in $2N$
data samples. The negative sample is not chosen explicitly. Instead, the
$2(N-1)$ sample, which contains of point cloud data and prompt, in the batch
are considred as the negative instances. The loss function for a positive pair
$\textbf{z}_{i}$ and $\textbf{v}_{i}$ is defined as,

$\ell_{i,j}=-\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{v}_{j}\right)/\tau\right)}{\sum_{k=1}^{2N}\mathbbm{1}_{[k\neq
i]}\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{v}_{k}\right)/\tau\right)}$
(5)

,

where
$\operatorname{sim}(\boldsymbol{z},\boldsymbol{v})=\boldsymbol{z}^{\top}\boldsymbol{v}/\|\boldsymbol{z}\|\|\boldsymbol{v}\|$
is denoted as the dot product between $l_{2}$ normalized z and v (i.e. cosine
similarity, $\mathbbm{1}_{[k\neq i]}\in\\{0,1\\}$ is the indicator function,
and $\tau$ denotes a temperature parameter. The total loss is calculated for
all positive pairs, both $(i,j)$ and $(j,i)$, in a batch. The overall training
stage is explained in Algorithm 1.

Algorithm 1 The proposed method

1:Inputs: $\mathcal{D}^{s}$

2:Output: A trained model to find $\hat{y}$ for all $\mathcal{X}^{u}$ in ZSL
mode and $\mathcal{X}^{s}\cup\mathcal{X}^{u}$ in GZSL mode.

3:repeat

4: for $\forall{\boldsymbol{I}}$ in $\mathcal{D}^{s}$ do

5: for sampled minibatch $\\{\mathcal{X}_{i}\\}_{i=1}^{N}$ in
$\mathcal{D}^{s}$ do

6: $\mathcal{G}_{i}\leftarrow$ generate new scene using Eq 4

7: $\mathcal{T}_{j}\leftarrow$ generate new prompt

8: $\textbf{h}_{i}$ $\leftarrow$ Forward sample $\mathcal{G}^{s}_{i}$ to
Encoder $\mathcal{H}$

9: $\textbf{p}_{j}$ $\leftarrow$ Forward sample $\mathcal{T}_{j}$ to Text
Encoder $\mathcal{F}$

10: $\textbf{z}_{i}$ $\leftarrow$ Forward $\textbf{h}_{i}$ to
$\mathcal{P}_{p}$

11: $\textbf{v}_{j}$ $\leftarrow$ Forward $\textbf{p}_{j}$ to
$\mathcal{P}_{t}$

12: end for

13: for $\forall i\in\\{1,...,2N\\}$ and $j\in\\{1,...,2N\\}$ do

14: calculate loss using Eq 5 for all pairs

15: end for

16: Backpropagate and update $\mathcal{H},\mathcal{P}_{p}$ and
$\mathcal{P}_{t}$

17: end for

18:until convergence

19:ZSL evaluation stage

20:$\mathcal{T}_{zsl}\leftarrow$ generate ”This is a {Object}.” prompt for
unseen classes.

21:for $\forall{\boldsymbol{I}}$ in $(\mathcal{X}^{u},\mathcal{Y}^{u})$ from
test part do

22: ${\boldsymbol{z}}_{i}$ $\leftarrow$
$\mathcal{P}_{p}(\mathcal{H}(\mathcal{X}_{i}))$

23: ${\boldsymbol{v}}_{zsl}$ $\leftarrow$
$\mathcal{P}_{t}(\mathcal{F}(\mathcal{T}_{zsl}))$

24: Return $\hat{{\boldsymbol{y}}}$ based on most similar
${\boldsymbol{v}}_{zsl}$ to ${\boldsymbol{z}}_{i}$

25:end for

26:GZSL evaluation stage

27:$\mathcal{T}_{gzsl}\leftarrow$ generate ”This is a {Object}.” prompt for
seen and unseen classes.

28:for $\forall{\boldsymbol{I}}$ in
$(\mathcal{X}^{s},\mathcal{Y}^{s},\mathcal{X}^{u},\mathcal{Y}^{u})$ from test
part do

29: ${\boldsymbol{z}}_{i}$ $\leftarrow$
$\mathcal{P}_{p}(\mathcal{H}(\mathcal{X}_{i}))$

30: ${\boldsymbol{v}}_{gzsl}$ $\leftarrow$
$\mathcal{P}_{t}(\mathcal{F}(\mathcal{T}_{gzsl}))$

31: Return $\hat{{\boldsymbol{y}}}$ based on most similar
${\boldsymbol{v}}_{gzsl}$ to ${\boldsymbol{z}}_{i}$

32:end for

| Dataset |  Total |  Seen/ | Train/
---|---|---|---|---
|  classes |  Unseen | Valid/Test
|  ModelNet40 [17] |  40 |  30/– | 5852/1560/–
ModelNet10 [17] |  10 |  –/10 | –/–/908
ScanObjectNN [18] |  15 |  –/11 | –/–/495
TABLE II: Statistics of the datasets. The total number of classes (3D models or images) in the datasets is listed, beside the exact splits employed in this paper separating the classes into seen or unseen and the elements into those applied for training or testing. The splits are from [9]. | Method (PointNet) | ModelNet10 | ScanObjectNN
---|---|---|---
ZSL | GZSL |  ZSL | GZSL
Acc |  $\operatorname{Acc}_{s}$ |  $\operatorname{Acc}_{u}$ |  HM |  Acc |  $\operatorname{Acc}_{s}$ |  $\operatorname{Acc}_{u}$ |  HM
|  DEM [4] |  19.2 |  76.0 |  7.2 |  13.1 |  14.8 |  78.9 |  3.2 |  6.2
|  LATEM [38] |  10.9 |  - |  - |  - |  9.5 |  - |  - |  -
|  GFZSL [39] |  13.1 |  80.8 |  4.1 |  7.8 |  17.9 |  83.6 |  3.6 |  6.9
|  SYNC [2] |  15.5 |  - |  - |  - |  13.3 |  - |  - |  -
|  GDAN [40] |  - |  82.1 |  0.6 |  1.2 |  - |  82.8 |  0.2 |  0.4
|  f-CLSWGAN [41] |  30.0 |  20.4 |  14.6 |  17.0 |  18.6 |  21.5 |  18.7 |  20.0
|  CADA-VAE [42] |  23.0 |  79.3 |  2.6 |  5.1 |  15.1 |  80.5 |  1.0 |  2.0
|  GXE [43] |  19.7 |  58.6 |  17.0 |  26.4 |  13.8 |  31.0 |  11.3 |  16.5
|  ZSL-3D [9] |  21.3 |  79.4 |  3.7 |  7.2 |  18.9 |  75.1 |  3.6 |  6.8
| Ours |  40.9 |  67.1 |  17.1 |  28.0 |  24.8 |  70.6 |  14.1 |  23.5
TABLE III: ZSL and GZSL results on ModelNet10 [17] and ScanObjectNN [18]
datasets

##  IV Experiment

###  IV-A Experimental Setup

We evaluate our proposed approach on three 3D datasets, ModelNet40 [17],
ModelNet10, and ScanObjectNN [17]. Here, ModelNet10 is a part of the 3D
synthetic ModelNet40 dataset. Unlike synthetic data of ModelNet40,
ScanObjectNN consists of 3D real-world point cloud data with background noise.
The statistics of the datasets are shown in Table II. We employ two different
experimental setups, including both synthetic and real-world data. Both
experimental setups are previously introduced by Cheraghian et al. [9]. The
first experimental setup is generated using only synthetic data. The seen
classes are the 30 classes of ModelNet40 that do not appear in ModelNet10, and
the unseen classes are the remaining 10 ModelNet10 classes. The second
experimental setup is more challenging and realistic than the first. This
configuration utilizes 26 ModelNet40 classes as seen and 11 ScanObjectNN
classes as unseen classes. This is a more practical setup because we can
collect many synthetic examples of seen objects during training. Nonetheless,
the model may confront many real-world 3D data instances of both seen and
unknown classes at test time.

Semantic features: For the semantic features, we employ the 768-dimensional
BERT[44] vectors for all datasets. In the non-prompt setting, we used class
name or joint class names “$objectA\ objectB$” (when generating scene
including two objects) to extract feature vectors. However, in the prompt
setting, we use the generated scene description to extract feature vectors.

Evaluation metric: We calculate the method’s performance using the top-$1$
accuracydetails. In ZSL, we predict the label of an unseen class using only
the unseen class’s label set. In generalized ZSL (GZSL), however, we predict
the class label based on both seen and unseen class labels. We additionally
report the Harmonic Mean (HM) [5] of the accuracy of the seen and unseen
classes in GZSL to identify the methods that are less biased towards the seen
classes without compromising the performance of both seen and unseen classes.

$\displaystyle\textrm{HM}=\frac{2\times acc_{s}\times
acc_{u}}{acc_{s}+acc_{u}}$ (6)

where $acc_{s}$ and $acc_{u}$ are seen and unseen class top-$1$ accuracies
respectively.

ModelNet10 | bathtub | bed | chair | desk | dresser | monitor | night stand | sofa | table | toilet | avg
---|---|---|---|---|---|---|---|---|---|---|---
baseline (A) | 0.0 | 6.0 | 66.1 | 8.1 | 73.3 | 72.0 | 39.5 | 22.0 | 0.0 | 45.0 | 33.2
Ours (D) | 2.4 | 60.5 | 92.9 | 28.3 | 59.1 | 35.1 | 40.8 | 7.2 | 0.0 | 83.3 | 40.9
TABLE IV: Per-class score evaluation on ModelNet10 [17] dataset. ScanObjectNN | cabinet | chair | desk | display | door | shelf | table | bed | sink | sofa | toilet | avg
---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline | 6.7 | 49.6 | 20.0 | 0.0 | 37.6 | 42.9 | 14.8 | 28.1 | 0.0 | 14.3 | 18.0 | 21.1
Ours | 0.0 | 0.0 | 49.4 | 0.0 | 93.3 | 57.7 | 2.5 | 6.1 | 0.0 | 25.6 | 38.1 | 24.8
TABLE V: Per-class score evaluation on ScanObjectNN [18] dataset.

Validation strategy: We establish a validation strategy to find the scaling
factors of $\alpha_{small}$ and $\alpha_{big}$ for the scene generation stage.
We randomly split the seen classes of an experimental setup into 80% for seen
validation classes and the rest of the 20% classes for unseen validation. The
grid search method is then employed to find the right hyperparameters. In the
scene generation process, we find three set of $\alpha$ scaling factors to
create “small” or “big” objects
$\\{(\alpha_{small}=0.2,\alpha_{big}=5),(\alpha_{small}=0.3,\alpha_{big}=3),(\alpha_{small}=0.5,\alpha_{big}=2),(\alpha_{small}=0.7,\alpha_{big}=1.5)\\}$
to assign “small” or “big” attribute to objects in scene.

Implementation details 111Codes and models are available at:
https://tinyurl.com/yfh63ny6: We trained our models on a single P100 GPU. Each
training epoch took about one hour for our ZSL model using the hyper-
parameters described in the paper. We trained models for a total of 100
epochs. We used the Adam [45] optimizer with $\beta_{1}$ = 0.9, $\beta_{2}$ =
0.999 and $\epsilon={10}^{-8}$. We varied the learning rate over the course of
training, started with $lr={10}^{-3}$ and decreased it by $0.5$ factor every
20 epochs. Throughout this paper, we used PointNet [13] as point cloud encoder
$\mathcal{H}$ to extract 1024 dimensional $\textbf{h}_{i}$ features. In
addition, we added two fully connected layers on top of the encoder to project
features to 512, then 128-dimensional representations $\textbf{z}_{i}$. Also,
we added a ReLU activation function after the first dense layer. For the text
pipeline, we used BERT [44] as the text encoder $\mathcal{F}$ to encode
generated prompts, following two fully connected layers with 1024 and 512
output dimensions, then a ReLU and another dense layer to provide
128-dimensional representations $\textbf{v}_{i}$. The text encoder is kept
frozen during training, similar to [15]. We use the PyTorch framework to
perform our experiments.

Figure 3: The impact of different hyper-parameters in our proposed
architecture. a) The effect of different batch size. b) the impact of scaling
factor in the scene generation process. c) The effect of various semantic
class embeddings.

###  IV-B Main results

The result of our proposed method on ModelNet10[17] and ScanObjectNN [18]
datasets are shown in Table III. We compared our proposed approach with
several 2D ZSL methods (DEM [4], SYNC [2], LATEM [38], GDAN [40], f-CLSWGAN
[41], CADA-VAE [42], GXE [43]) as well as one 3D ZSL method [9]. For a fair
comparison, we re-implemented these state-of-the-art 2D ZSL methods with
modification of the 3D point cloud. Our proposed approach significantly beats
the other methods on the ModelNet10 and ScanObjectNN datasets.

ModelNet10: For the ModelNet10 dataset, as can be seen in Table III, our
method significantly outperform other well-known methods proposed for 2D and
3D ZSL. For ZSL, we achieve nearly more than 10% improvement in comparison to
the accuracy of the second-best method in both 2D and 3D methods. Similarly,
for GZSL, we obtain the best harmonic mean 28.0% compared to the generative 2D
methods that generate samples for unseen classes and are well-known for having
fewer bias issues than traditional methods. This observation shows that ZSL on
3D data is more challenging than 2D images and the current approach proposed
for 2D images maybe are not best fit for this problem. In addition, we observe
that the 2D methods are more biased toward the seen classes, which is why the
harmonic mean is relatively low for many 2D methods. We outperform the current
3D ZSL method regarding the accuracy of unseen classes in the ZSL and GZSL
setups. As a result, we design a ZSL method specific to 3D point cloud data,
which can take advantage of this kind of data.

ScanObjectNN: In this dataset, we also outperform other state-of-the-art
compared methods. Though, it is essential to notice that methods typically
operate better on the 3D synthetic dataset (ModelNet10) than real data
(ScanObjectNN). The reason can be the domain shift from synthetic to real data
and noise in real data. As we can see in Table III, in ZSL, we obtain 24.8%
accuracy for predicting unseen classes, which is by a large margin better than
other reported methods. We can see a similar pattern for GZSL, which shows
that our proposed method can address the bias problem better than other
methods.

Per-class results: Table IV and Table V represent the performance of each
individual class from ModelNet10 and ScanObjectNN respectively. The results of
both datasets are compared to a baseline method that does not include scene
generation and prompt in its training pipeline. For ModelNet10, or ModelNet10,
we see that our solution achieves greater than 30% accuracy in 6 of the ten
classes (bed, chair, dresser, monitor, nightstand, toilet). However, the
baseline and our method do not classify instances from the table. While the
baseline method mostly predicts dresser or monitor classes due to the hubness
problem [46, 4]. On the other hand, our method can classify a few instances of
the bathtub while the baseline cannot. The hubness issue in the ScanObjectNN
is more severe since the real-world 3D data is noisier and more unordered than
synthetic data, and there is an additional domain shift between synthetic and
real data. However, our proposed method performs relatively better than the
baseline method.

###  IV-C Ablation study

Impact of batch size: Our proposed method has a remarkable influence on the
batch size. In Figure 3(a), we report ZSL ($acc_{u}$) and GZSL (HM)
performance on ModelNet10 using different batch sizes. We can observe that
increasing batch size improves the performance. When the batch size is 64, the
accuracy is at its peak for both ZSL and GZSL, and increasing the batch size
to 100 has no effect on performance.

Impact of scene generation parameters: In this part, we evaluate the impact of
scaling factor $\alpha$ in the scene generation module. As can be seen in
Figure 3(b), we achieve the best accuracy in both ZSL and GZSL when we use
$\alpha_{small}=0.5$ and $\alpha_{big}=2$.

Impact of different semantic embedding: We have shown in Figure 3(c) the
performance of our proposed method on different semantic embedding, BERT [44],
w2v [47], and GloVe [48]. Based on this experiment, we observe that BERT
achieves the best performance compared to other semantic embeddings. The
reason is that the BERT model can take advantage of the context information in
a prompt, while w2v and GloVe do not consider context knowledge. It is needed
to mention that the w2v and GloVe feature representations of prompts are
obtained by averaging among all single words in a prompt.

Method | SG | P | DC | ZSL | GZSL
---|---|---|---|---|---
$Acc_{s}$ | $Acc_{u}$ | HM
A |  |  |  | 33.2 | 73.2 | 13.9 | 23.3
B | ✓ |  | ✓ | 34.6 | 59.3 | 16.6 | 26.0
C | ✓ | ✓ |  | 38.7 | 72.1 | 15.5 | 25.5
D | ✓ | ✓ | ✓ | 40.9 | 67.1 | 17.1 | 28.0
TABLE VI: The impact of different component in our method: SG: scene
generation, P: using prompt, in non-prompt scenario we just use categorical
class labels, DC: generate scenes using instances of different classes vs
single class in each scene

Impact of scene generation and prompt: Our proposed scene generation algorithm
consists of three main components: 1) Scene Generation (SG), 2) Prompt
generation (P) as the class semantic information, and 3) Between class scene
generation (DC). The impact of these components is reported in Table VI.
Method A represents a vanilla model without using the three mentioned
components. It is seen that the baseline method gets the lowest performance
among all methods with a significant biased towards seen classes. Generating
scenes using instances of different or same classes (Method B) can alleviate
this bias problem of seen classes. As can be seen, our proposed method (Method
D) achieves the best performance while employing all three components
together.

##  V Conclusion

Collecting 3D point cloud data of objects at a massive scale has become more
accessible than ever, thanks to better 3D capture systems. Nevertheless, 3D
point cloud recognition systems are not able to manage this large-scale
scenario. To this end, in this paper, we propose a novel zero-shot learning
framework specific for 3D point cloud data to classify previously unseen data.
Our proposed approach introduces a dynamic 3D scene generation and supervision
method that generates 3D scene data to learn the network better, exploring the
complicated interplay of seen and unseen objects. Also, in our approach, to
describe the generated scenes, we employ a prompt-based annotation. After
that, to train the proposed architecture, we perform contrastive learning. We
show that 3D scenes can link objects more efficiently than single objects
thanks to transformer language models (BERT). Our proposed dynamic scene
generation approach encapsulates data augmentation and prompt-based
annotation/captioning to enhance 3D ZSL performance. We have obtained state-
of-the-art ZSL and generalized ZSL performance on synthetic (ModelNet40,
ModelNet10) and real-scanned (ScanOjbectNN) 3D object datasets.

## References

  * [1] M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, “Zero-shot learning with semantic output codes,” _NIPS_ , vol. 22, 2009.
  * [2] S. Changpinyo, W.-L. Chao, B. Gong, and F. Sha, “Synthesized classifiers for zero-shot learning,” in _CVPR_ , 2016.
  * [3] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for image classification,” _IEEE TPAMI_ , vol. 38.
  * [4] L. Zhang, T. Xiang, and S. Gong, “Learning a deep embedding model for zero-shot learning,” in _CVPR_ , 2017.
  * [5] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” _IEEE TPAMI_ , vol. 41.
  * [6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021.
  * [7] A. Cheraghian, S. Rahman, and L. Petersson, “Zero-shot learning of 3d point cloud objects,” in _MVA_ , 2019.
  * [8] A. Cheraghian, S. Rahman, D. Campbell, and L. Petersson, “Mitigating the hubness problem for zero-shot learning of 3d objects,” in _BMVC_ , 2019.
  * [9] A. Cheraghian, S. Rahman, T. F. Chowdhury, D. Campbell, and L. Petersson, “Zero-shot learning on 3d point cloud objects and beyond,” _International Journal of Computer Vision (IJCV)_ , 2022.
  * [10] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _ICLR_ , 2014.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [12] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _NIPS_ , vol. 30, 2017\.
  * [13] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _CVPR_ , 2017.
  * [14] A. Cheraghian, S. Rahman, D. Campbell, and L. Petersson, “Transductive zero-shot learning for 3d point cloud classification,” in _WACV_ , 2020.
  * [15] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” _NIPS_ , vol. 34, 2021.
  * [16] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021.
  * [17] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in _CVPR_ , 2015\.
  * [18] M. A. Uy, Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung, “Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data,” in _ICCV_ , 2019.
  * [19] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, “Zero-shot learning through cross-modal transfer,” _NIPS_ , vol. 26, 2013.
  * [20] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” _NIPS_ , vol. 26, 2013.
  * [21] Y. Li, D. Wang, H. Hu, Y. Lin, and Y. Zhuang, “Zero-shot recognition using dual visual-semantic mapping paths,” in _CVPR_ , 2017.
  * [22] Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele, “Latent embeddings for zero-shot classification,” in _CVPR_ , 2016.
  * [23] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata, “Feature generating networks for zero-shot learning,” in _CVPR_ , 2018.
  * [24] V. Khare, D. Mahajan, H. Bharadhwaj, V. K. Verma, and P. Rai, “A generative framework for zero shot learning with adversarial domain adaptation,” in _WACV_ , 2020.
  * [25] A. Mishra, S. Krishna Reddy, A. Mittal, and H. A. Murthy, “A generative model for zero shot learning using conditional variational autoencoders,” in _CVPR_ , 2018.
  * [26] M. R. Vyas, H. Venkateswara, and S. Panchanathan, “Leveraging seen and unseen semantic relationships for generative zero-shot learning,” in _ECCV_ , 2020\.
  * [27] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Transductive multi-view zero-shot learning,” _IEEE TPAMI_ , vol. 37, 2015.
  * [28] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” _NIPS_ , vol. 28, 2015.
  * [29] L. Zhang, T. Xiang, and S. Gong, “Learning a deep embedding model for zero-shot learning,” in _CVPR_ , 2017.
  * [30] B. Michele, A. Boulch, G. Puy, and R. Marlet, “Generative zero-shot learning for semantic segmentation of 3d point cloud,” _arXiv preprint arXiv:2108.06230_ , 2021.
  * [31] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _NIPS_ , vol. 33, 2020.
  * [32] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [33] T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners,” _arXiv preprint arXiv:2012.15723_ , 2020.
  * [34] T. Schick and H. Schütze, “Exploiting cloze questions for few-shot text classification and natural language inference,” in _EACL_ , 2020.
  * [35] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for attribute-based classification,” in _CVPR_ , 2013.
  * [36] Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, “Evaluation of output embeddings for fine-grained image classification,” in _CVPR_ , 2015.
  * [37] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” _NIPS_ , vol. 26, 2013.
  * [38] Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele, “Latent embeddings for zero-shot classification,” in _CVPR_ , 2016.
  * [39] V. K. Verma and P. Rai, “A simple exponential family framework for zero-shot learning,” in _ECML-PKDD_ , 2017.
  * [40] H. Huang, C. Wang, P. S. Yu, and C.-D. Wang, “Generative dual adversarial network for generalized zero-shot learning,” in _CVPR_ , 2019.
  * [41] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata, “Feature generating networks for zero-shot learning,” in _CVPR_ , 2018.
  * [42] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata, “Generalized zero and few-shot learning via aligned variational autoencoders,” in _CVPR_ , 2019.
  * [43] K. Li, M. R. Min, and Y. Fu, “Rethinking zero-shot learning: A conditional visual classification perspective,” in _ICCV_ , 2019.
  * [44] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [46] Y. Shigeto, I. Suzuki, K. Hara, M. Shimbo, and Y. Matsumoto, “Ridge regression, hubness, and zero-shot learning,” in _ECML-PKDD_ , 2015.
  * [47] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” _NIPS_ , vol. 26, 2013.
  * [48] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _EMNLP_ , 2014.