\lesssim
|a(2^{-\frac 12}z,2^{-\frac 12}z)|\eabs {(x,-\xi )}^{-\rho |\alpha +\beta |},
\end{multline*}

which implies that (<ref>) holds.

This shows that
$a$ is hypoelliptic of order $\rho _0$ in
$\wideparen \maclA _{\Sh ,\rho}^{(\omega )}(\cc {2d})$
when $\fka$ is hypoelliptic of order $\rho _0$ in
$\Sh _\rho ^{(\omega )}(\rr {2d})$.

Suppose instead that $a$ is hypoelliptic of order $\rho _0$ in
$\wideparen \maclA _{\Sh ,\rho}^{(\omega )}(\cc {2d})$. By using
Proposition <ref>,
(<ref>) and (<ref>)
instead of Proposition
<ref>, similar
computations as in the first part of the proof shows that
(<ref>) and (<ref>)
hold for some $R>0$. This shows that
$\fka$ is hypoelliptic of order $\rho _0$ in
$\Sh _\rho ^{(\omega )}(\rr {2d})$ when
$a$ is hypoelliptic of order $\rho _0$ in
$\wideparen \maclA _{\Sh ,\rho}^{(\omega )}(\cc {2d})$, and the result follows.

§.§ Ellipticity in the case of polynomial symbols

Next we discuss ellipticity for polynomial symbols, i. e.

(x,ξ)    = ∑_|α+β| ≤Nc(α,β)
x,ξ   ∈d,

a(z,w)    = ∑_|α+β| ≤Nc(α,β)z^αw^β,

z,w    ∈d.

The corresponding principal symbols are
_p (x,ξ)    = ∑_|α+β| = Nc(α,β)x^αξ^β,
x,ξ   ∈d,

a_p (z,w )    = ∑_|α+β| = Nc(α,β)
z^αw ^β,
z,w    ∈d,

First we relate polynomials on $\rr {2d}$ to Shubin classes.

Let $\fka$ and $\fka _p$ be as in (<ref>)
and (<ref>)
for some $c(\alpha ,\beta )\in \mathbf C$, $\alpha ,\beta \in \nn d$
and $N\ge 0$, and let $\omega _{N}(x,\xi )=\eabs {(x,\xi )}^{N}$,
$x,\xi \in \rr d$. Then the following is true:

* $\fka \in \Sh _1^{(\omega _{N})}(\rr {2d})$;

* $\fka$ is elliptic with respect to $\omega _{N}$, if and only if
$\fka _p(x,\xi )\neq 0$ when $(x,\xi )\neq 0$.

The result can be considered folklore. In order to be self-contained we present
the arguments.

First we prove (1). Let $t=\max (|x_1|,\dots ,|x_d|,|\xi _1|,\dots ,|\xi _d|)$ when
$x=(x_1,\dots ,x_d)\in \rr d$ and $\xi =(\xi _1,\dots ,\xi _d)\in \rr d$. Then

|\fka (x,\xi )| \le  \sum _{|\alpha +\beta | \le N_0}|c(\alpha ,\beta )| t^{|\alpha +\beta |}
\lesssim  1+t^{N}
\le \eabs {(x,\xi )}^{N},

which gives the desired bound for $|\fka (x,\xi)|$. Since the degree of a polynomial
is lowered by at least one for every differentiation we get

|\partial^\alpha \fka (x,\xi )| \lesssim \eabs {(x,\xi )}^{N-|\alpha |}

for every $\alpha \in \nn {2d}$, which gives (1).

In order to prove (2) we let $\fka _p$ be as in (<ref>).
First suppose that $\fka _p(x,\xi )\neq 0$ when $(x,\xi )\neq (0,0)$, and let $g$ be the continuous
function on $\rr {2d}\setminus 0$ given by

g(x,\xi ) =\frac {|\fka _p(x,\xi )|}{|(x,\xi )|^{N}},\qquad (x,\xi )\neq (0,0).

Since $g$ is continuous and positive, and the sphere

\mathbf S^{2d-1}=\sets {(x,\xi )\in \rr {2d}}{|x|^2+|\xi |^2 = 1}

is compact, it follows that there are constants $c_1,c_2>0$ such that

c_1\le g(x,\xi )\le c_2,\qquad (x,\xi )\in \mathbf S^{2d-1}.

By homogeneity it now follows

c_1|(x,\xi )|^{N} \le |\fka _p(x,\xi )| \le c_2|(x,\xi )|^{N},\qquad x,\xi \in \rr d.

Hence, if

\fkb (x,\xi )= \fka (x,\xi )-\fka _p(x,\xi )
= \sum _{|\alpha +\beta | \le N-1}c(\alpha ,\beta )x^\alpha \xi ^\beta ,

then the first part of the proof implies that for some constants $C>0$ and $R>0$
we have

\begin{equation*}
|\fka (x,\xi )| \ge |\fka _p(x,\xi )| -|\fkb (x,\xi )| \ge c_1|(x,\xi )|^{N_0} - C\eabs {(x,\xi )}^{N-1}
\gtrsim \eabs {(x,\xi )}^{N}
\end{equation*}

when $|(x,\xi )|\ge R$. Hence $\fka$ is elliptic with respect to $\omega _{N_0}$.

Suppose instead $\fka _p(x_0,\xi _0)=0$ for some $(x_0,\xi _0)\neq (0,0)$. For any
$(x,\xi )=(tx_0,t\xi _0)$ we have

\begin{multline*}
|\fka (x,\xi )| \le |\fka _p(x,\xi )| +|\fkb (x,\xi )|
|t^N\fka _p(x_0,\xi _0)| +|\fkb (x,\xi )|
\\[1ex]
|\fkb (x,\xi )| \lesssim \eabs {(x,\xi )}^{N-1},
\end{multline*}

giving that $|\fka (x,\xi )|\gtrsim \eabs {(x,\xi )}^{N}$, $|(x,\xi )|\ge R$,
cannot hold for any $R>0$.

By Theorems <ref>, <ref>
and Proposition <ref> we get the following.
The details are left for the reader.

Let $a$ and $a_p$ be as in (<ref>)
and (<ref>)
for some $c(\alpha ,\beta )\in \mathbf C$, $\alpha ,\beta \in \nn d$
and $N\ge 0$, and let $\omega _{N}(x,\xi )=\eabs {(x,\xi )}^{N}$,
$x,\xi \in \rr d$.
Then the following is true:

* $a\in \maclA _{\Sh ,1}^{(\omega _{N})}(\cc {2d})$;

* $a$ is elliptic in $\maclA _{\Sh ,1}^{(\omega _{N})}(\cc {2d})$ if and only if $a_p(z,z)\neq 0$
when $z\neq 0$.

Let $\fka$, $\fka _p$, $a$ and $a_p$ be as in
Then it follows from Propositions <ref> and
Proposition <ref> that $\fka$ is elliptic, if
and only if $\fka _p$ is elliptic, and that $a$ is elliptic, if and only if $a_p$
is elliptic.

We have now the following.

Let $\fka \in \Sh _1^{(\omega _{N})}(\rr {2d})$ and $\fka _p$ be as in (<ref>)
and (<ref>)
for some $c(\alpha ,\beta )\in \mathbf C$, $\alpha ,\beta \in \nn d$
and $N\ge 0$.
Then the following is true:

* the principal symbol $a_p(z,w)$ of $\mathsf S_{\mathfrak V}\fka$
is given by

\begin{equation}
\label{Eq:ComplexPrincipalSymb}
a_p(z,w) = 2^{-\frac N2}
\sum _{|\alpha +\beta |= N}
c(\alpha ,\beta )i^{|\beta |}
(z+\overline w)^\alpha (z-\overline w)^\beta
\text ;
\end{equation}

* $\fka$ is elliptic in $\Sh _1^{(\omega _{N})}(\rr {2d})$ if and only if
$a_p$ is elliptic in $\maclA _{\Sh ,1}^{(\omega _{N})}(\cc {2d})$;

* $\fka _p(x,\xi )> 0$ for every $(x,\xi )\neq (0,0)$,
if and only if
$a_p(z,z)> 0$ for every $z\neq 0$.

Let $z=x+i\xi$, $x,\xi \in \rr d$, i.  e. $x=\frac 12(z+\overline z)$ and
$\xi =\frac 1{2i}(z-\overline z)$. By Theorem <ref> we

\begin{equation}\label{Eq:PrincipalSymbolShubinWickIdent}
a_p(2^{-\frac 12}z,2^{-\frac 12}z) = \fka _p(x,-\xi ).
\end{equation}

This implies

\begin{multline*}
a_p(z,z) = 2^\frac N2\sum _{|\alpha +\beta | = N}c(\alpha ,\beta )x^\alpha (-\xi )^\beta
\\[1ex]
= 2^\frac N2\sum _{|\alpha +\beta | = N}c(\alpha ,\beta )2^{-|\alpha |}
(z+\overline z)^\alpha (2i)^{-|\beta |}(-(z-\overline z) )^\beta ,
\end{multline*}

which gives

\begin{equation}
\tag*{(\ref{Eq:ComplexPrincipalSymb})$'$}
a_p(z,z) = 2^{-\frac N2}
\sum _{|\alpha +\beta |= N}
c(\alpha ,\beta )i^{|\beta |}
(z+\overline z)^\alpha (z-\overline z)^\beta .
\end{equation}

The formula (<ref>) now follows from
(<ref>)$'$ and analytic continuation, using the
fact that $a_p(z,w)$ is analytic in $z$ and conjugate analytic in $w$.

The assertion (2) follows by a combination of Corollary <ref>,
Propositions <ref> and <ref>,
and the assertion (3) is a direct consequence of

§ A NECESSARY CONDITION FOR POLYNOMIALLY BOUNDED
WICK SYMBOLS

In <cit.> Folland shows that polynomial symbols for
pseudo-differential operators correspond
to polynomial Wick and anti-Wick symbols.
Thus partial differential operators with polynomial coefficients
corresponds to polynomial Wick symbols.

Here we show that a Wick symbol that is polynomially bounded
must be a polynomial. This gives a characterization of Wick
symbols corresponding to polynomial symbols for pseudo-differential

Cauchy's integral formula implies that an entire function
which is polynomially bounded must
be a polynomial:

Let $F\in A(\cc d)$ have Maclaurin series

F(z) = \sum _{\alpha \in \nn d} c(\alpha )e_\alpha (z),\quad z\in \cc d.

Suppose that for some $j\in \{ 1,\dots ,d\}$, $C>0$, $N \ge 0$,  and an open neighbourhood
$I\subseteq \mathbf C$ of the origin we have

|F(z)|\le C \eabs {z_j}^N,\quad z_j\in \mathbf C,

provided $z_k\in I$, $k\in \{ 1,\dots ,d\} \setminus \{ j\}$. Then $c(\alpha ) = 0$
when $\alpha _j > N$.

By interchanging the variables, we may assume that $j=d$. Let
$R\ge 1$ and $\ep >0$ be chosen such that

D_\ep \equiv \sets {z_0\in \mathbf C}{|z_0|\le \ep} \subseteq I.

Take $\alpha \in \nn d$ such that $\alpha _d>N$, let
$\beta =(\alpha _1+1,\dots ,\alpha _d+1)\in \nn d$ and
$\gamma _\ep \subseteq \mathbf C$ be the
boundary circle of $D_\ep$. Then Cauchy's integral formula gives

\begin{multline*}
\frac {|c(\alpha )|}{\alpha !^{\frac 12}}
\left |
\frac { \partial^\alpha F (0) }{\alpha !}
\right |
= (2\pi )^{-d}
\left |
\idotsint _{\gamma_\ep ^{d-1}}
\left (
\int _{|z_d|=R}
\frac { F(z)}{z^\beta}\, dz_d
\right )
\, dz_1\cdots dz_{d-1}
\right |
\\[1ex]
\le
(2\pi )^{-d}
\idotsint _{\gamma_\ep ^{d-1}}
\left (
\int _{|z_d|=R}
\frac {| F(z)|}{|z^\beta |}\, |dz_d |
\right )
\, |dz_1|\cdots |dz_{d-1}|
\\[1ex]
\lesssim
R^{-\alpha _d} \eabs R^N\ep ^{-(\alpha _1+\dots +\alpha _{d-1})}
\to 0
\end{multline*}

as $R\to \infty$.

Let $a\in \wideparen A(\cc {2d})$ and suppose

\begin{equation}\label{Eq:GlobPolEst}
|a(z,w)|\lesssim \eabs{(z,w)} ^N
\end{equation}

for some $N\ge 0$. Then $a$ is a polynomial in $z\in \cc d$ and $\overline w\in \cc d$
of degree at most $N$.

By Proposition <ref> it follows that $a$ is a
polynomial of degree at most $2dN$. We need to prove that the degree
is at most $N$. In order to do this we may assume that $a$ has degree
at least one.

For some integer $M\ge 1$ we have

a(z,w) = a_M(z,w) +a_{M-1}(z,w),

\begin{align*}
\sum _{|\alpha +\beta |=M}c(\alpha ,\beta )z^\alpha \overline w^\beta
\intertext{is non-trivial and}
\sum _{|\alpha +\beta |\le M-1}c(\alpha ,\beta )z^\alpha \overline w^\beta .
\end{align*}

Since $a_M$ is non-trivial, there are $z_0,w_0\in \cc d$ such that
$|z_0|^2+|w_0|^2=1$ and $|a_M(z_0,w_0)|=c_0\neq 0$. By
homogeneity we get

|a_M(tz_0,tw_0)| = c_0|t|^M,\qquad t \in \ro.

In the same way we get

|a_{M-1}(tz_0,tw_0)| \le C(1+|t|)^{M-1},\qquad t\in \ro

for some constant $C$ which is independent of $t$.

Suppose contrary to the assumption that $M>N$. For $t\in \ro$ with
$|t|\ge 1$ we have

\begin{multline*}
\left |
\frac {a(tz_0,tw_0)}{\eabs {(tz_0,tw_0)}^N}
\right |
\gtrsim
|t|^{-N}\left (|a_M(tz_0,tw_0)| - |a_{M-1}(tz_0,tw_0)|    \right )
\\[1ex]
\ge
|t|^{-N}\left (c_0|t|^M - C(1+|t|)^{M-1} \right ) \to \infty
\quad \text{as}\quad
|t|\to \infty .
\end{multline*}

This contradicts (<ref>), and the hence our assumption
that $M>N$ must be false.

[1]
A. Abdeljawad, M. Cappiello, J. Toft,
Pseudo-differential calculus in anisotropic Gelfand-Shilov setting,
Integr. Equ. Oper. Theory 91 (2019), 91–26.

[2]
A. Abdeljawad, S. Coriasco, N. Teofanov,
Bilinear Pseudo-differential Operators with Gevrey–Hörmander Symbols,
Mediterr. J. Math. 17 120 (2020).

[3]
A. Abdeljawad, C. Fernandez, A. Galbis, J. Toft, R Üster,
Characterizations of Pilipović spaces
by powers of harmonic oscillator, arXiv:1905.10732 (2020).

[4]
V. Bargmann,
On a Hilbert space of analytic
functions and an associated integral transform, Comm. Pure
Appl. Math. 14 (1961), 187–214.

[5]
V. Bargmann,
On a Hilbert space of analytic functions and an associated
integral transform. Part II. A family of related
function spaces. Application to distribution theory, Comm. Pure
Appl. Math. 20 (1967), 1–101.

[6]
F. A. Berezin,
Wick and anti-Wick symbols of operators,
Mat. Sb. (N.S.) 86 (1971), 578–610.

[7]
F. A. Berezin,
Covariant and contravariant symbols of operators,
Izv. Akad. Nauk SSSR 36 (1972), 1134–1167.

[8] J. M. Bony, J. Y. Chemin Espaces Functionnels
Associés au Calcul de Weyl-Hörmander, Bull. Soc. math. France
122 (1994), 77–118.

[9]
M. Cappiello, R. Schulz, P. Wahlberg,
Conormal distributions in the Shubin calculus of pseudodifferential operators,
J. Math. Phys. 59 (2018) 021502.

[10]
M. Cappiello, J. Toft,
Pseudo-differential operators
in a Gelfand–Shilov setting, Math. Nachr. 290 (5-6) (2017), 738–755.

[11] E. Cordero, S. Pilipović, L. Rodino, N. Teofanov,
Quasianalytic Gelfand-Shilov spaces with applications
to localization operators, Rocky Mountain. J. Math. 40 (2010),

[12] P. Boggiatto, E. Buzano, L. Rodino
Global Hypoellipticity and Spectral Theory, Akademie Verlag, Berlin, 1996.

[13]  G. B. Folland, Harmonic analysis in phase space,
Princeton University Press, Princeton, 1989.

[14] I. M. Gelfand, G. E. Shilov, Generalized functions, II-III,
Academic Press, NewYork London, 1968.

[15] K. Gröchenig, Foundations of
time-frequency analysis,
Birkhäuser, Boston, 2001.

[16] K. Gröchenig, Weight functions in time-frequency analysis,
in: L. Rodino, M. W. Wong (Eds.) Pseudodifferential
Operators: Partial Differential Equations and Time-Frequency Analysis,
Fields Institute Comm., 52 (2007), pp. 343–366.

[17] K. Gröchenig, G. Zimmermann,
Spaces of test functions via the STFT J. Funct. Spaces Appl. 2
(2004), 25–53.

[18] L. Hörmander, The analysis of linear
partial differential operators, vol I–III,
Springer-Verlag, Berlin Heidelberg NewYork Tokyo, 1983, 1985.

[19] E. H. Lieb,
Integral bounds for radar ambiguity functions and Wigner distributions,
J. Math. Phys. 31 (1990), 594–599.

[20] E. H. Lieb,
The stability of matter from atoms to stars,
Bull. Amer. Math. Soc. 22 (1990), 1–49.

[21]  E. H. Lieb, J. P. Solovej,
Quantum coherent operators: a generalization of coherent states,
Lett. Math. Phys. 22 (1991), 145–154.

[22] Z. Lozanov Crvenković, D. Perišić,
Hermite expansions of elements of Gelfand Shilov spaces
in quasianalytic and non quasianalytic case,
Novi Sad J. Math. 37 (2007), 129–147.

[23]
B. Prangoski,
Pseudodifferential operators of infinite order in spaces
of tempered ultradistributions,
J. Pseudo-Differ. Oper. Appl. 4  (2013), 495–549.

[24] M. Reed and B. Simon,  Methods of modern
mathematical physics, Academic Press, London New York, 1979.

[25]
M. A. Shubin, Pseudodifferential operators and spectral theory,
Springer, 2001.

[26]
N. Teofanov,
Ultradistributions and time-frequency analysis. In
“Pseudo-differential operators and related topics”, 173–192, Oper.
Theory Adv. Appl., 164, Birkhäuser, Basel, 2006.

[27]
N. Teofanov,
Gelfand-Shilov spaces and localization operators,
Funct. Anal. Approx. Comput. 7 (2015), 135–158.

[28]
N. Teofanov, J. Toft,
Pseudo-differential calculus in a Bargmann setting,
Ann. Acad. Sci. Fenn. Math. 45 (2020), 227–257.

[29] J. Toft, The Bargmann transform on modulation and
Gelfand-Shilov spaces, with applications to Toeplitz and pseudo-differential
operators,  J. Pseudo-Differ. Oper. Appl.  3  (2012), 145–227.

[30] J. Toft,
Images of function and distribution spaces under the
Bargmann transform, J. Pseudo-Differ. Oper. Appl. 8 (2017), 83–139.

[31] G. C. Wick,
The evaluation of the collision matrix, Phys. Rev. 80 (1950),


# Unveiling Multilinguality in Transformer Models: Exploring Language
Specificity in Feed-Forward Networks

Sunit Bhattacharya and Ondřej Bojar
Institute of Formal and Applied Linguistics
Faculty of Mathematics and Physics
Charles University
<EMAIL_ADDRESS>

###### Abstract

Recent research suggests that the feed-forward module within Transformers can
be viewed as a collection of key-value memories, where the keys learn to
capture specific patterns from the input based on the training examples. The
values then combine the output from the ‘memories’ of the keys to generate
predictions about the next token. This leads to an incremental process of
prediction that gradually converges towards the final token choice near the
output layers.

This interesting perspective raises questions about how multilingual models
might leverage this mechanism. Specifically, for autoregressive models trained
on two or more languages, do all neurons (across layers) respond equally to
all languages? No! Our hypothesis centers around the notion that during
pretraining, certain model parameters learn strong language-specific features,
while others learn more language-agnostic (shared across languages) features.
To validate this, we conduct experiments utilizing parallel corpora of two
languages that the model was initially pretrained on. Our findings reveal that
the layers closest to the network’s input or output tend to exhibit more
language-specific behaviour compared to the layers in the middle.

##  1 Introduction

One of the least studied aspects of the Transformer Vaswani et al. (2017)
models in general and Large Language Models (LLMs) in particular is the feed-
forward layers (FFNs). Although they contain almost two-thirds of the
parameters, it is only recently111Although the work by Wang and Tu (2020) is
relevant in this regard, their analysis was done for all the components of the
Transformer and not just the FFNs. that their role in the working of the
models is being seriously studied.

Figure 1: Transformer block and the structure of FFN

Geva et al. (2021, 2022) have earlier demonstrated that FFNs could be seen as
“key-value memories” where each neuron (key)222While Geva et al. (2021) use
the word _‘keys’_ , some other authors use the word _neuron_ in this context.
in the lower sub-layer of the FFN gets triggered by specific patterns in the
input data and the higher sub-layer (values) produces a distribution over the
output vocabulary. This leads us to a perspective (Figure 1) where the FFN
first captures certain patterns or concepts333Shallow processing would require
them to be good at capturing certain syntax patterns while semantic processing
would require them to be good at capturing more thematic/conceptual patterns.
in the input (conceptualization), selects the important aspects (using the
activation function i.e. selection) and then combines them to emit an output
which can be interpreted as a prediction of the possible next-word token for
that layer, i.e. synthesis. To highlight this view throughout the rest of the
paper, we will use the term _‘detectors’_ instead of the rather generic ‘keys’
to refer to the neurons in the earlier layer and _‘combinators’_ instead of
‘values’ to refer to the later layer. Repeating this across layers leads to a
process of incremental prediction of the next token, with the prediction from
previous layers being refined in the next layers Belrose et al. (2023). This
perspective however raises an important question. For models trained with a
causal-language modeling objective in multilingual settings, what sort of
patterns do the detectors encode across layers? More precisely, are some
detectors triggered by input only from specific languages?

In this paper, we investigate this phenomenon of language specificity of the
detectors in a multilingual model, pretrained on 30 languages from 16 language
families. Earlier work has shown that Transformer models encode more shallow
features in the earlier layers444close to the input while encoding more
semantic features in the later layers555near the output Tenney et al. (2019).
We hypothesise that the shallow processing would require more language-
specific detectors than the semantic aspects of the input. And hence, we posit
that during pretraining of the multilingual models, two kinds of neurons would
emerge: language-specific and language-agnostic.

Thorough investigations into the role of the FFN layers in Transformer is an
interesting research direction, and to our best knowledge, this is the first
work that tries to look at the FFN666in a decoder-only Transformer model from
the perspective of multilinguality. The rest of the paper is structured as
follows: a brief discussion of the related works (Section 2) is followed by
the description of the models and data (Section 3) and models (Section 4).
This is followed by the presentation (Section 5) and simultaneous discussion
of the results (Sections 6 and 7).

##  2 Related Work

Exploring the role and capabilities of the FFN sub-layer in Transformer models
is a still nascent field of research with only a few papers exploring their
working. As mentioned earlier, Geva et al. (2021, 2022) have proposed an
interesting perspective of looking at how the FFN layer of the Transformer
contributes during language generation. Recent work Meng et al. (2022); Yao et
al. (2022) exploring the capabilities of the FFN has also looked into how the
activations of FFNs could be used for understanding how autoregressive models
deal with facts. Other works (Li et al., 2022; Zhang et al., 2022) have
analysed activation patterns in FFNs to study sparsity in Transformers. In
other words, they show that only a few neurons in the FFNs are activated
corresponding to inputs to Transformers.

On the front of studying multilingual models, Libovický et al. (2019)
demonstrated that representations in encoder-only models can be split into
language specific and language-neutral components. But to our best knowledge,
no equivalent study has been done for autoregressive language models.
Additionally, Deshpande et al. (2022); Blevins et al. (2022); Lauscher et al.
(2020); Choudhury and Deshpande (2021); Kudugunta et al. (2019) have studied
the pretraining behaviour and capabilities of various encoder-only
multilingual models. More recently, Pfeiffer et al. (2022) demonstrated how
separating parameters into language-specific modules during training can help
improve the performance across languages.

From the perspective of studying multilinguality in the human brain,
neuroimaging studies Crinion et al. (2006); Videsott et al. (2010); Miozzo et
al. (2010) have shown that although neural circuits for different languages
are highly overlapping, there are distinct brain areas for language-specific
processing and areas that are language-agnostic.

##  3 Model and testing data

We use a pretrained XGLM model Lin et al. (2021) with 1.7 billion parameters,
available on the Hugging Face Wolf et al. (2019)
repository777https://huggingface.co/facebook/xglm-1.7B for our experiments.

We use sentences from the training data of the CzEng 2.0
corpus888https://ufal.mff.cuni.cz/czeng Kocmi et al. (2020) for our
experiments. The model description of the XGLM model states that the model was
trained on CommonCrawl data of various languages. CzEng heavily relies on
various freely accessible web sources and a part of the data included in CzEng
is also drawn from CommonCrawl among other sources. Thus, we expect that the
sentences used for the experiments are of the same domain/style as the model
was originally trained on, and they can even overlap. We do not consider such
a possible overlap a serious problem for our analysis, because we are not
measuring any processing performance or generalization capability.

##  4 Experiment

We first extract a sample of sentences from the CzEng corpus, giving us a set
of Czech and English parallel sentences. We only select sentences with lengths
between 20 and 50. We then feed the model with all ‘prefixes’ of the sampled
sentences from both languages. In other words, for each sentence, we
incrementally feed the model one subword at a time and record our
observations. For instance, for a Czech sentence like “Tenhle úkol je obtížný”
(This task is difficult), the prefixes fed to the model would be “Tenhle”,
“Tenhle úkol”, “Tenhle úkol je” and “Tenhle úkol je obtížný”. The parallel
sentences ensure that the semantic contents of the sentences for the two
languages are similar. We go on to collect the data about the model state
corresponding to each prefix.

Figure 2: FFN in close detail

From the collected data999from all sentences across Czech and English, we
extract the “selection coefficients” corresponding to each prefix for all
detectors across the layers of the model. Specifically, for detector $d_{i}$
in layer $L_{j}$, we define the selection coefficient for a prefix $p_{k}$ as:

$C^{\left(L_{j},d_{i}\right)}_{p_{k}}=GeLU\\{d_{i}(p_{k})\\}$ (1)

Thus, for each prefix we obtain layer-wise selection coefficients for the
detectors (an example can be visualised in Table 1). We then sort the
detectors based on the values of their corresponding selection coefficients.
We posit that for a layer, certain detectors are triggered by specific prefix
templates or languages. The selection coefficient is the indicator of the
extent to which a particular detector is triggered by a prefix. Thus,
observing the selection coefficients of the detectors across prefixes of
different languages should indicate which (and how many) detectors are
relevant bilingually and which (and how many) are relevant only for one of the
two examined languages. We do this by analysing the top-k detectors after
sorting the detectors by decreasing selection coefficients.

Table 1: Selection coefficients of $m$ detectors in layer $L$ for a total of $n$ prefixes Lang1, sent1, prefix_1 | $C_{11}C_{12}C_{13}\ldots C_{1m}$
---|---
Lang1, sent1, prefix_2 | $C_{21}C_{22}C_{23}\ldots C_{2m}$
$\vdots$ | $\vdots$
Lang2, sentN, prefix_xx | $C_{k1}C_{k2}C_{k3}\ldots C_{km}$
Lang2, sentN, prefix_xy | $C_{n1}C_{n2}C_{n3}\ldots C_{nm}$

##  5 Observations

As an example, Table 2 shows the top-1 detector (detector with maximum
selection coefficient) for the prefixes of an English and Czech sentence.

Prefix | Detector
---|---
Europol | 2149
Europol zpracovává | 2149
Europol zpracovává a | 3942
Europol zpracovává a předává | 200
Europol zpracovává a předává údaje | 200
Europol | 2149
Europol shall | 2149
Europol shall process | 2149
Europol shall process and | 3424
Europol shall process and transfer | 2149
Table 2: Prefixes from an example Czech-English sentence pair, listing the
most active detector ID (according to selection coefficients) from layer 1.

In the following sections, we present the results from our observations of the
selection coefficients of detectors across the layers of the model.

###  5.1 Distribution of active detectors across layers

We collect the indices of the top-10 and top-100101010The top-10 list implies
that we extract the list of the 10 detectors that had the maximum selection
coefficients for a prefix. Similarly, for the top-100 list, we extract 100
detectors with the maximum selection coefficients. detectors for each prefix.
For a prefix $P_{i}$ of all the considered prefixes $P_{0},P_{1},...,P_{n}$,
we denote the set of the top detectors $D_{i}$ where $|D_{i}|=t$ (i.e. the set
cardinality of $|D_{i}|$ is $t$). This way, we collect the list of the top $t$
detectors for all prefixes in a layer. For each layer $L_{k}$, we obtain
$L_{k}=D_{0}\cup D_{1}\cup...\cup D_{n}$ and we plot the $|L_{k}|$ across the
layers (e.g. Figure 3). In other words, we are checking how many unique
detectors across prefixes belong to the list of 10 or 100 most active
detectors for that layer. The fewer detectors in this set, the more “compact"
the representation of these sentences are. The more detectors is in this set,
the more “network capacity" is used when processing the given sentences. We
make the plots for each of the two languages. Hence, using the example in
Table 2: for layer $1$, we have $L^{en}_{1}=\left(2149,3424\right)$ and
$L^{cs}_{1}=\left(2149,3942,200\right)$ and so $|L^{en}_{1}|=2$ and
$|L^{cs}_{1}|=3$.

Figure 3: Number of top detectors ($|L_{i}|$) used across layers when
processing Czech (top plot) and English (bottom plot) sentences.

Figure 3 shows that the top-100 list does not seem to show any pattern, unlike
the top-10 list. We observe that for each prefix, only certain detectors
exhibit high values of selection coefficient. Selecting the top-100 leads to
the inclusion of many detectors that repeatedly appear across many prefixes
with tiny values of selection coefficient. We reason that, this leads to the
pattern seen with the top-10 list. We also posit that this is a callback to
the previous research that has indicated that FFNs exhibit patterns of sparse
activation.

The top-10 list shows that the number of detectors for both languages
increases between layers 1 to 4 (near the input) and then decrease between
layers 19 to 24 (near the output). Since this observation also includes
detectors that get triggered for both languages111111for example, detector
2149 in the example shown in Table 2, we analyse the number of detectors that
are intersecting between the two languages (Czech and English). That is, for
each layer $L_{k}$, we identify the intersecting detectors
$I_{k}=L^{cs}_{i}\cap L^{en}_{i}$. In other words, we examine how the number
of keys getting triggered by both English and Czech prefixes (multilingual
detectors) vary across the layers.

Figure 4: Distribution of multilingual detectors (intersecting detectors)

As Figure 4 shows, the number of intersecting detectors also follows the same
pattern as observed in Figure 3. The number starts increasing in the layers
near the input and decrease near the output. It may be argued that the spike
in the number of unique detectors (for individual languages) in the middle
layers might imply that the number of intersecting detectors would also
increase in the middle layers. However, we argue that it might not always be
the case. We validate our argument in the following sections.

To look at the language specific responses of the detectors across the layers,
we look at the set difference of the detectors seen in, Figure 3 i.e. the
language-specific detectors. So, for some layer $k$, we analyse
$en_{k}=L^{en}_{k}\setminus L^{cs}_{i}$ and121212From the example in Table 2,
$en_{k}=3424$ and $cs_{k}=3942,200$ $cs_{k}=L^{cs}_{i}\setminus L^{en}_{i}$.
From the results in Figure 5, we see that there is a steady drop in the number
of Czech-specific detectors in the middle layers. No such effect is seen for
English. Also, across all the results presented here, we note that the
observed number of detectors getting triggered by English prefixes is
considerably higher than that of Czech prefixes.

Figure 5: Distribution of language specific detectors

Next, we determine to what extent the actual language can be identified from
the detector activity.

###  5.2 Layers close to the input and output are language specific

To confirm the existence of language-specific detectors, we train a linear
classifier over all the detectors for each layer. The task of the classifier
is to use the selection coefficients to determine if the given prefix was in
English or Czech. The results from the experiment are shown in Figure 6. In
the plot, we show the number of detectors across different performance
brackets. Each series shows the number of detectors classifying with an
accuracy of $>=k\%$.

Figure 6: Classification percentages across layers. The colour indicates the
reached accuracy level of the prediction.

We see that for performance brackets $<80\%$, the layer closer to the input
shows the highest accuracy in predicting the language. Again for slabs,
$>70\%$ we see that the accuracy increases in the last few layers. Thus, we
conclude that layers closer to the input and output are more language-specific
than the others.

##  6 Discussion

We started with the hypothesis that language-specific detectors would be more
common in the layers closer to the input and output. We analysed the detectors
across the layers using sentences from a Czech-English parallel corpus. We
note that in the underlying XGLM model, English (with 803,527 million training
tokens) was much more dominant than Czech (with 8,616 million training tokens)
Lin et al. (2021). We thus consider the model to be a primarily English model
that saw some Czech sentences during pretraining. From the results, we observe
that the layers closer to the input and output indeed perform more language
specific processing than others. We also see that considerably lower number of
detectors are triggered by the Czech prefixes than English prefixes, probably
reflecting the data imbalance during training. While looking at the behaviour
of Czech-specific detectors, we find that their numbers drop near the middle
layers (8-15). We know that the model is primarily English centric. And since
it is well known that higher-layers of Transformers are involved in more
semantic processing, it is likely that the model uses more language-agnostic
detectors and only a few Czech-specific detectors for processing semantic
aspects of the input. Studies with humans have previously shown that semantic
processing in humans is often language-agnostic. We thus see a possible way to
connect these observations in the future.

From a different perspective, the analysis of the selection coefficients also
agrees with the recent theories and observations about the sparse nature of
FFN modules. We hypothesise that the sparsity (lesser numbers of unique
detectors) might be an indicator of shallow processing and density might be an
indicator of semantic processing. The sparsity argument might also be extended
to claim that only a subset of detectors are required for language specific
processing while greater numbers of detectors are required for more language-
agnostic (i.e. semantic) However, such claims warrant extensive
experimentation that we wish to conduct as a followup to this work.

##  7 Conclusion

In this study, we focused on the analysis of the Feed Forward Layers (FFNs) of
a pretrained multilingual Transformer model. We look at the FFNs as a system
that first identifies patterns in the input representations (detector),
selects the relevant information (selector), and then combines it to make a
guess of the next token (combiner). We assess the degree of language
specificity of the detectors in this multilingual model with two experiments.
We observe that there are greater number of language specific detectors near
the input and output of the model. Additionally, we observe how data imbalance
during training is reflected in the behaviour of the multilingual detectors.
We also try to link our observations with recent studies on the sparse
activations in FFNs. Overall, our findings shed light on the language
specificity of FFNs in multilingual models.

## Limitations

While our analysis provides valuable insights into the behaviour of
“detectors” in a multilingual Transformer model’s Feed Forward Layers (FFNs),
there is an important limitation to consider. Our analysis is limited to only
the XGLM model. This work does not consider the multilingual dynamics of other
models. Also, our study is centred on the Czech-English language pair.
Different languages exhibit diverse linguistic characteristics and
complexities, and the behaviour of detectors could vary significantly across
various language pairs. Extrapolating our findings to multilingual behaviour
involving other languages requires caution and further investigation. Further,
while we categorize detectors as language-specific or multilingual based on
their activation patterns, the specific linguistic cues that trigger their
activation remain complex and challenging to interpret. Our study focuses on
the quantitative aspects of detector behaviour, and a deeper qualitative
analysis of the linguistic information captured by these detectors could
provide additional insights.

## Ethics Statement

As the work is dedicated to evaluating existing models on publicly available
datasets, we are not aware of any potential ethical issues or negative
impacts.

## Future Work

We wish to extend this work and test the generalizability of our hypothesis
across more language pairs and other multilingual autoregressive language
models.

##  8 Acknowledgements

This work has been funded from the 19-26934X (NEUREM3) grant of the Czech
Science Foundation and the grant 205-09/260698 (SVV) of Charles University.
The work has also been supported by the Ministry of Education, Youth and
Sports of the Czech Republic, Project No. LM2023062 (LINDAT/CLARIAH-CZ).

## References

  * Belrose et al. (2023) Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023.  Eliciting latent predictions from transformers with the tuned lens.  _arXiv preprint arXiv:2303.08112_.
  * Blevins et al. (2022) Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022.  Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models.  _arXiv preprint arXiv:2205.11758_.
  * Choudhury and Deshpande (2021) Monojit Choudhury and Amit Deshpande. 2021.  How linguistically fair are multilingual pre-trained language models?  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 12710–12718.
  * Crinion et al. (2006) Jenny Crinion, Robert Turner, Alice Grogan, Takashi Hanakawa, Uta Noppeney, Joseph T Devlin, Toshihiko Aso, Shinichi Urayama, Hidenao Fukuyama, Katharine Stockton, et al. 2006.  Language control in the bilingual brain.  _Science_ , 312(5779):1537–1540.
  * Deshpande et al. (2022) Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022.  When is bert multilingual? isolating crucial ingredients for cross-lingual transfer.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3610–3623.
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  _arXiv preprint arXiv:2203.14680_.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495.
  * Kocmi et al. (2020) Tom Kocmi, Martin Popel, and Ondrej Bojar. 2020.  Announcing czeng 2.0 parallel corpus with over 2 gigawords.  _arXiv preprint arXiv:2007.03006_.
  * Kudugunta et al. (2019) Sneha Reddy Kudugunta, Ankur Bapna, Isaac Caswell, Naveen Arivazhagan, and Orhan Firat. 2019.  Investigating multilingual nmt representations at scale.  _arXiv preprint arXiv:1909.02197_.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499.
  * Li et al. (2022) Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. 2022.  Large models are parsimonious learners: Activation sparsity in trained transformers.  _arXiv preprint arXiv:2210.06313_.
  * Libovický et al. (2019) Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2019.  How language-neutral is multilingual bert?  _arXiv preprint arXiv:1911.03310_.
  * Lin et al. (2021) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021.  Few-shot learning with multilingual language models.  _arXiv preprint arXiv:2112.10668_.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Miozzo et al. (2010) Michele Miozzo, Albert Costa, Mireia Hernandez, and Brenda Rapp. 2010.  Lexical processing in the bilingual brain: Evidence from grammatical/morphological deficits.  _Aphasiology_ , 24(2):262–287.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022.  Lifting the curse of multilinguality by pre-training modular transformers.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3479–3495.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  Bert rediscovers the classical nlp pipeline.  _arXiv preprint arXiv:1905.05950_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Videsott et al. (2010) Gerda Videsott, Bärbel Herrnberger, Klaus Hoenig, Edgar Schilly, Jo Grothe, Werner Wiater, Manfred Spitzer, and Markus Kiefer. 2010.  Speaking in multiple languages: Neural correlates of language proficiency in multilingual word production.  _Brain and language_ , 113(3):103–112.
  * Wang and Tu (2020) Wenxuan Wang and Zhaopeng Tu. 2020.  Rethinking the value of transformer components.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6019–6029.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Yao et al. (2022) Yunzhi Yao, Shaohan Huang, Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. 2022\.  Kformer: Knowledge injection in transformer feed-forward layers.  In _Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24–25, 2022, Proceedings, Part I_ , pages 131–143. Springer.
  * Zhang et al. (2022) Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022\.  Moefication: Transformer feed-forward layers are mixtures of experts.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 877–890.


# Lidar Upsampling with Sliced Wasserstein Distance

Artem Savkin1,2
TUM, BMW  Yida Wang1
TUM  Sebastian Wirkert2
BMW  Nassir Navab1,3
TUM, JHU  Federico Tombari1,4
TUM, Google

###### Abstract

Lidar became an important component of the perception systems in autonomous
driving. But challenges of training data acquisition and annotation made
emphasized the role of the sensor to sensor domain adaptation. In this work,
we address the problem of lidar upsampling. Learning on lidar point clouds is
rather a challenging task due to their irregular and sparse structure. Here we
propose a method for lidar point cloud upsampling which can reconstruct fine-
grained lidar scan patterns. The key idea is to utilize edge-aware dense
convolutions for both feature extraction and feature expansion. Additionally
applying a more accurate Sliced Wasserstein Distance facilitates learning of
the fine lidar sweep structures. This in turn enables our method to employ a
one-stage upsampling paradigm without the need for coarse and fine
reconstruction. We conduct several experiments to evaluate our method and
demonstrate that it provides better upsampling.

††1TUM, 85748 Munich, Germany<EMAIL_ADDRESS><EMAIL_ADDRESS>††2BMW
AG, 80809 Munich (Germany)††3JHU, Baltimore, MD 21218, United States††4Google,
8002 Zurich, Switzerland

##  I Introduction

Lidar can significantly improve the quality and reliability of the perception
systems of autonomous vehicles. As an active sensor, lidar takes accurate
measurements of the surrounding environment and provides spatial information
about the traffic scene in form of point clouds. It is robust to challenging
light conditions and can significantly contribute to the functional safety of
fully autonomous systems. Also, the development and manufacturing of lidar
have been democratized considerably in recent years. Democratization enabled
the equipment of the serial vehicles with this sensor. As a result, lidar
became a crucial component of state-of-the-art autonomous driving.

At the same time, machine perception on point clouds is still challenging
since this kind of data is unstructured and irregular. However, in computer
vision, neural networks have already proved superior to traditional
algorithms.

Figure 1: Results of $2\times$ upsampling for $8129$ points patches from KITTI
dataset.

Their advantage is especially evident in the tasks of segmentation or
detection where understanding of the semantics of the underlying scene is
involved. A wide variety of methods dedicated to 3D object detection,
tracking, and segmentation successfully adapted deep learning techniques to
point clouds to tackle the challenges mentioned above [26, 34, 29, 43].

In practical applications such as autonomous driving, data acquisition and
annotation required for training such perception methods is a expensive and
inflexible process. Such rigidness is characterized by a model’s performance
degradation attributed to data modality change e.g., sensor viewpoint,
resolution, etc. [7]. For example, [3] demonstrates that lidar segmentation
methods reveal significant performance degradation when evaluated on the
samples from a lidar in which mounting position, number of scan lines, or
intensity calibration differs from the training data. Table II shows similar
behavior, demonstrated by the object detection networks operating on lidar
samples with different resolutions. The phenomenon which causes degradation is
commonly referred to as domain shift [31] and addressed through transfer
learning. They aim to address the domain shift problem by learning the domain-
invariant features or by learning the direct mapping between source and target
domains. Still, models are supposed to be compatible with novel sensor
generations and, thus, with data modalities or domains. In this work, we
address the problem of such sensor-to-sensor domain adaptation as a mechanism
to avoid actual data re-acquiring or re-annotating the data. In particular, we
focus here on the task with low- and high-resolution lidars. The latter is
distinguished by a higher number of scan lines in the vertical field of view.
Thus, the overarching goal is to sample the data points from the target
distribution or, in other words, such point clouds which resemble
characteristics of data samples from the target domain. Our task is to
generate a point cloud sample with more scan lines given a low-resolution
lidar scan sample. We consider both modalities separate domains and employ a
point cloud transfer technique to solve them.

Conv1dEdgeConvSampleLinearMaxpoolRepl.Interp.Reshape$N\times d$$N\times
C$$N/r\times C$$C$$N/r\times C$$N\times C$$N\times 2C$$2N\times C$$2N\times
d$$Ext$$Exp$$Rec$ Figure 2: Overview of the network with feature extraction
(Ext), feature expansion (Exp) and coordinate reconstruction (Rec).

Existing methods for lidar generation mainly rely on distance measures such as
Chamfer Distance (CD) and Earth Mover’s Distance (EMD) [44, 47]. Although
several works [11, 1] establish the superiority of EMD over CD as more
discriminative for point cloud learning, the CD simply nearest neighbor based
is still more efficient. Yet, from our observations, both measures do not
suffice to generate fine-structure lidar point clouds. CD fails to equally
distribute additional points across the lidar scan, thus erroneously favoring
the known or source point locations. Standard EMD approximation based on
auction algorithm [11] fails to reconstruct the fine scan pattern of lidar
sweeps accurately. This phenomenon is also confirmed by experiments in [40].
Multiple works have addressed the computational complexity of EMD and the
whole family of Wasserstein distances (WD) [17, 15, 39, 22] in various tasks.

The contribution of this work is twofold. Firstly, it provides an extensive
analysis of applicability of established distance metrics such as Chamfer
distance, approximated EMD, and SWD for lidar scan generation and upsampling.
Secondly, based on this analysis, it proposes a new lidar upsampling method,
which is dedicated to the problem of cross-sensor domain adaptation between
low-resolution and high-resolution lidars. We argue that, unlike existing
point cloud upsampling methods, our edge-aware one-stage (no coarse
reconstruction) approach based on Sliced-Wasserstein distance can reconstruct
fine details of lidar scans and surpass existing upsampling techniques.

##  II Related work

(a) Source

(b) CD

(c) EMD

(d) Ours

Figure 3: Point cloud reconstruction using PointNetFCAE with conventional loss
functions.

The lidar super-resolution or lidar upsampling task has been addressed by the
recent learning-based methods, which can be categorized into grid-based
methods [28] and point-based [18] methods.

###  II-A Grid-based lidar generation

Given an input condition, e.g., vector, the generation task is to produce a
$N\times d$ point cloud of size $N$ and dimensionality $d$. Grid-based methods
utilize the range image representation of point clouds projected into the
cylindrical coordinate system. [4] adapts generative adversarial models
applied on such range images to produce large-scale lidar scans. Unlike the
lidar GAN approach, which uses real lidar scans for the discriminator
training, [28] uses training exclusively from simulated data to produce scans
comparable with real high-resolution lidar. [16] improves range image
upscaling by utilizing a widely acclaimed attention mechanism. Finally, [14]
approaches integrating an uncertainty estimation mechanism into the upsampling
process to meet real-time constraints. In this work, we aim to develop a
rather generic domain adaptation technique for lidar upsampling, which falls
into the latter category of point-set-based methods.

###  II-B Point-based lidar generation

Typical point-based generation methods such as AtlasNet [13], and FoldingNet
[41] operate directly on the point sets where they approach the problem by
creating 2D patches and aligning them to the underlying point cloud shape.
With the emergence of generative neural nets, several works like PCGAN [1],
GCN-GAN [35], and TreeGAN [30] applied adversarial training frameworks to the
point clouds. Luo et al. [21] employed a concept similar to image diffusion
for the point cloud generation. More recent generative methods utilize
normalizing flows - PointFlow [40], gradient fields - ShapeGF [5] and
autoregression - PointGrow [32].

Compared to generation, the upsampling aims to compute a $rN\times d$ output
given a $N\times d$ point cloud, with upsampling ratio $r$ so that it
resembles the spatial characteristics of the underlying 3d shape. Methods
introduced by Yu et al. [45] and Zhang et al. [47] were pioneering deep
learning-based approaches for that task. PU-Net [45] relied on Point-Net++
encoder [26] for feature extraction from patches then applied two sets of
linear layers onto features replicated from $N\times C$ to $N\times rC$ which
are then merely reshaped into $rN\times C$. PU-Net is followed by EC-net [44],
which introduced a loss function that aimed to minimize the point-to-edge
distance to make the network more edge-aware. Later 3PU [42] explored multi-
scale progressive patch-based upsampling by breaking down a $r$-factor
upsampling network into several $2\times$ upsampling networks. It introduces
dense connections or blocks for the feature extraction and defines local
neighborhoods later in a feature space via kNN based on feature similarity.
PU-GAN [18] adopts an adversarial framework for point cloud upsampling by
formulating a discriminator network that learns to evaluate generated high-
resolution point clouds. It introduces a up-down-up expansion unit, which
enables a so-called self-correction of expanded features. More recent methods
include PU-GCN [27] and Dis-PU[19].

Another task worth mentioning is point cloud completion, whose goal is to
generate complete point clouds and create dense scans from incomplete sparse
ones. Here pioneering learning method was PCN [46]. PCN first learns the
global feature of the underlying shape from partial observation and then
reconstructs a complete but coarse point cloud of lower resolution, which is
up-sampled with folding operations in the second phase. TopNet [33] method
utilizes a tree structure for generating complete dense point clouds.
Following the two-stage coarse-fine point cloud generation MSN [20] uses the
morphing principle and joint CD and EMD loss. Two-staged ECG [24] improves on
the generation of the edges by exploiting the edge-preserved pooling. Finally,
CDN [37] extends a traditional GAN framework to guarantee that local areas
follow the same pattern as ground truth.

Compared to previous point cloud generation works, our method is dedicated to
precisely reconstructing fine-grained lidar point clouds. Discussed point-
based methods reveal a relatively restricted applicability area that embraces
simplistic, ShapeNet-alike [6] data. Point clouds generated by a lidar sensor
demonstrate significantly more complex underlying geometry, scan pattern, and
data variance. Typically methods trained on PCN dataset [46] derived from
synthetic ShapeNet CAD models embrace eight categories of objects (e.g.,
plane, car, chair) whereas lidar scans of the KITTI dataset [12] in turn were
obtained during drives in the real world and contain multiple classes of
objects in a single scan. Thus, the task of point cloud generation in the
lidar domain remains under-explored. To our best knowledge, the proposed
method is the only point-based method able to reconstruct complex lidar point
clouds in the upsampling task accurately.

##  III Approach

Our method relies on the edge-aware feature extraction, and extension
mechanism [42] and projection-based 1-Wasserstein distance. This training
objective reveals significantly improved sensitivity to fine-grained lidar
scan patterns as opposed to conventional loss functions.

###  III-A Model

(a) Source

(b) CD

(c) EMD

(d) SWD

(e) Ground Truth

Figure 4: Lidar upsampling using proposed network with conventional loss
functions.

The proposed model is built upon established upsampling carcass with three
steps: feature extraction, feature expansion, and coordinate reconstruction.
The feature extractor is implemented as a plain encoder-decoder architecture
and similarly to 3PU [42] employs dense per point feature learning using
Conv1d and EdgeConv [38] layers. An EdgeConv layer applies the kNN algorithm
to find clusters of points and preserves characteristic attributes like edges.
Following the edge-preserving principle, the feature expansion module also
takes advantage of EdgeConv layer for generating the $N\times 2C$ upsampled
feature vector. In the final step, a set of linear layers reconstruct the
resulting $N\times 3$ coordinates.

Figure 5: Distance metrics on jitter (normalized).

Figure 6: Distance metrics on rotation (normalized).

Feature Extraction. As mentioned before, the encoder first takes the $N\times
d$ point cloud as input and translates it into $N/r\times C^{\prime}$ feature
vector by applying several $Conv1d$ plus $EdgeConv$ blocks. Here, $N$ is the
original number of points, $r$ is downsampling ratio, $d$ \- point cloud
dimensionality (in our experiments $d=3$) and $C^{\prime}$ is the resulting
size of the feature vector. The common farthest point sampling technique
achieves downsampling from $N$ to $N/r$. This feature vector is transformed to
$C^{\prime}$-sized global feature, which will be upscaled into $N\times C$ by
replication and interpolation. For that, replicated $C^{\prime}$ feature
vectors are just stacked together and interpolated using inverse distance
weighted average from [26].

Feature Expansion. Inspired by [24], we design the feature expansion module
along the lines of edge awareness by utilizing the EdgeConv layer. This layer
employs the kNN algorithm and enables the module to find clusters based on
similarities in the feature space. It takes a $N\times C$ learned feature
vector and transforms it into $N\times 2C$ upsampled feature vector and then
reshaped to $2N\times C$ tensor.

Coordinates Reconstruction. the resulting tensor will be then processed by a
set of linear layers to reconstruct the point dimensions $2N\times d$. Fig. 2
visualizes the overall architecture.

###  III-B Loss Function

We argue that conventional loss functions based on Chamfer Distance (CD) and
approximations of Earth Mover’s Distance (EMD) [20] suffer from the lack of
sensitivity to the fine-grained scan patterns of lidar sensors. As mentioned
before, CD unequally distributes generated points across the lidar scan in
point cloud reconstruction and favors the known or source point locations. A
possible reason for that could be the nature of a generative task where
produced point clouds at the initial state represent unstructured point blobs
so that CD fails to find correspondences. On the other hand, the reason for
EMD to fail inaccurate reconstruction of the fine scan patterns could lie in
commonly used approximations which lead to biased or noisy gradients [40] and
herewith affect the ability of a model to distinguish the fine details.

(a) Source

(b) PU-Net

(c) 3PU

(d) PU-GAN

(e) Ours

(f) Ground truth

Figure 7: Qualitative comparison of the $2\times$ lidar upsampling results on
KITTI test.

Following the Optimal Transport (OT) perspective, we consider the distance
between point clouds as the distance between two distributions with a natural
way to measure it by a metric function defined between probability
distributions - Wasserstein distance. Yet accurate, exact computation of this
metric is computationally intractable. The assumption is that more accurate
distance calculation is a key to precise lidar scan reconstruction and
upsampling. We argue that higher accuracy of point cloud reconstruction, which
is sensitive to fine scan patterns, can be achieved by applying the Sliced
Wasserstein (SW) distance as it possesses equivalent statistical properties to
WD [15, 23]. The intuition behind SW distance resides in the idea of the
existence of a closed-form OT solution in a uni-dimensional space. For two
point sets $X$ and $Y$ of equal size such as
$X,Y\subset\mathbb{R}^{3}:|X|=|Y|$, EMD between $X$ and $Y$ can be calculated
as:

$EMD(X,Y)=\min_{\phi:X\rightarrow Y}\sum_{x\in X}\|x-\phi(x)\|_{2}$ (1)

where $\phi:X\rightarrow Y$ is a bijection [11]. The more general form of EMD
in the case of $|X|=|Y|$:

$EMD(X,Y)=\inf_{P\in\Pi(X,Y)}\mathbb{E}_{(x,y)\sim P}[d(x,y)]$ (2)

is equivalent to Wasserstein distance with $p=1$:

$W_{p}(X,Y)=\inf_{P\in\Pi(X,Y)}\mathbb{E}_{(x,y)\sim P}[d^{p}(x,y)]^{1/p}$ (3)

where $d$ is a metric and $\Pi(X,Y)$ is a set of all joint distributions of
$P(x,y)$ of random variables $x$ and $y$ with marginal distributions $X$ and
$Y$:

$\Pi(X,Y)=\\{P_{X,Y}(x,y)\\}$

. In practice, computation of 1 or 3 is intractable, so an approximation
scheme such as an auction algorithm must be applied [20] yet it cannot
guarantee the required bijection assignment. As opposed to such approximation,
the Sliced Wasserstein approach exploits the fact that $W_{p}$ has a closed-
form solution for univariate probability distributions [15]. Thus several
works utilize this fact to calculate $W_{p}$ [39, 9]:

$SW_{p}(X,Y)=\int_{\mathbb{S}^{d-1}}W_{p}(X^{s},Y^{s})ds$ (4)

where $X^{s}$ and $Y^{s}$ are projections of $X$ and $Y$ onto the direction
$s$ of all possible directions of a unit sphere $\mathbb{S}^{d-1}$. In
practice, $SW_{p}$ is approximated by only taking a finite number of random
directions from $\mathbb{S}^{d-1}$. Moreover, in a uni-dimensional case, $d$
can be obtained by just sorting [9]. Hence our final objective can be defined
as follows:

$\tilde{SW}_{p}(X,Y)=\frac{1}{|\tilde{\mathbb{S}}|}\sum_{s\subset\mathbb{\tilde{S}}}\frac{1}{|X|}\sum_{i}\|X^{s}_{i}-Y^{s}_{i}\|_{2}$
(5)

Thus, the final optimization problem can be formulated as follows:

$\underset{\theta}{\mathrm{argmin}}\>\mathbb{E}_{X\sim
D_{X}}(\tilde{SW}_{p}(X,f_{\theta}(X)))$ (6)

##  IV Experiments

###  IV-A Datasets

Our experiments are based on the popular public datasets SemanticKITTI [2] and
Waymo Open Dataset [10]. SemanticKITTI is a large-scale computer vision
dataset that provides 22 sequences of lidar scans with overall around $43000$
scans alongside point cloud segmentation labels. The dataset provides
measurements in German inner cities, residential areas, and highway traffic
scenes during various daylight and weather conditions. In our experiments we
operate with original KITTI [12] dataset split of the sequences divided into
train and test sets of the sizes $7481$ and $7518$ scans respectively.
Semantic labels of the dataset contain information about $28$ classes such as
road, sidewalk, person, car. The Waymo dataset, in turn, provides $1150$
sequences of $20$ seconds each. In total, it offers around $230K$ annotated
samples with the ground truth for object detection and tracking tasks. For our
purposes, every $10$th lidar scan has been sampled from all data sequences,
resulting in $15947$ samples training dataset and $4038$ validation dataset.
Also, from the five lidars, we only use the scans provided by the top-mounted
one.

Figure 8: Lidar upsampling results (middle) on KITTI val patches with input
samples (top) and ground truth (bottom).

###  IV-B Loss

We propose to train lidar generation models using loss function based on
Sliced-Wasserstein distance as opposed to commonly utilized CD, EMD-based
losses. Our benchmark experiments suggest that these metrics are sufficient
for a model to learn the shapes of point clouds but not the fine structures.
Described behavior can be observed in Fig. 3, which shows the results of the
reconstruction of the point cloud patches provided by a vanilla auto-encoder
network PointNetFCAE trained respectively with CD and EMD loss. This auto-
encoder model consists of a Pointnet [25] based encoder and a fully connected
decoder. Even though the sizes of the visualized patches are equal ($2048$
points), CD looks underpopulated in certain regions. [1] refers to this
phenomenon as CD blindness, which manifests in point placement where the
average mass locates and fails to distinguish such poor allocation from the
true one. EMD, in turn, appears to reconstruct rough point distribution but is
considerably distorted. That finding is aligned with observations in [11] and
mirrors its mean-shape experiments.

Additionally, we compare the proposed loss function with existing error
metrics concerning their sensitivity. The results of this analysis are shown
in Fig. 5 and Fig. 6. The comparison is conducted between Sinkhorn loss [8],
Sliced Gromov-Wasserstein [36], Sliced Wasserstein Distance [15], Chamfer
loss, and EMD loss. Here we execute two experiments by gradually applying a
corresponding transformation on the point cloud and measuring the distances
between transformed and original samples. In the first experiment, Gaussian
noise is introduced into a sample, whereas during the second experiment, the
sample is rotated along the vertical axis. In both cases, analyzed metrics
behaved consistently by changing to non-zero values as the transformation
became more significant in the first experiment and turned back to zero after
full rotation in the second experiment. Despite that, Sliced Wasserstein
Distance could identify the slightest point cloud changes by indicating higher
discrepancy values in the early transformation iterations. The influence of
different loss functions CD, EMD, and SWD on the lidar upsampling performed by
our model will be discussed in the ablation study in the section IV-C

| CD | HD | EMD | SWD
---|---|---|---|---
PU-Net [44] | 0.5272 | 1.2627 | 0.3851 | 15.4002
3PU [42] | 0.1172 | 0.4151 | 0.2740 | 5.2078
PU-GAN [19] | 0.0426 | 0.1892 | 0.2504 | 1.3553
Ours | 0.0435 | 0.1694 | 0.0775 | 1.1742

TABLE I: Quantitative comparison for $2\times$ upsampling of $8192$ points on
KITTI val dataset.

Figure 9: Lidar upsampling results (middle) on Waymo val patches with input
samples (top) and ground truth (bottom)

###  IV-C Upsampling

The training occurs supervised where input patches of size $2048$ were
obtained by decimating the ground truth samples by a factor of two. Decimation
is performed based on rasterized input point clouds. For the Waymo dataset,
every second scan line of the original cylindrical projection of size $W\times
H\times 3$ of the lidar scans was dropped/rejected before projecting the
points into the lidar coordinate system. Similarly, we processed the
cylindrical projections of the KITTI dataset. As the KITTI dataset provides
only raw point clouds, they are required to be rasterized before decimation.
Therefore, KITTI scans were arranged into $64\times 2048$ grid based on sensor
vertical resolution and azimuth angle for rasterization. This preprocessing is
shown in Fig. 1. With that data, the network is trained for $2000$ consecutive
epochs without pre-trained weights initialization and using Adam optimizer.
The result after the training memory footprint is $22$ Mb. Training is
accomplished on NVIDIA Quadro RTX 6000 GPU($24$GB) with a single forward pass
that takes, on average $0.2$ sec. To evaluate the generated results, we
conducted several experiments and assessed our results both qualitatively and
quantitatively. The results of the lidar scan upsampling on the KITTI dataset
are shown in Fig. 7. To visualize the effects introduced by our approach, we
display both sources and upsampled scans together with point clouds generated
by the state-of-the-art upsampling methods 3PU, PU-Net, and PU-GAN. We employ
original training protocols from respective reports to train those models on
lidar point clouds. As reported in the original publication, PU-GAN was
trained with adversarial and EMD-based reconstruction loss, 3PU was trained
with modified counter-outlier CD loss, and PU-Net used plain EMD loss with
repulsion loss. First, we want to draw attention to the fact that the proposed
method recovers the scan lines missing in the input samples and, in
consequence, effectively reconstructs the high-resolution target scans. In
comparison with the results of the existing methods, samples provided by our
approach reveal no point scattering in the local neighborhood but rather
provide distinguished scan lines. Another characteristic of the generated
scans worth considering is the actual shape of the scans. In the upsampled
lidar sweeps, recovered scan lines indeed follow the spatial characteristics
of the underlying geometry.

Additionally, we evaluate our method of upsampling the point cloud patches on
a larger scale. The results of $2\times$ upsampling for patches of size $8192$
for both datasets KITTI and Waymo are depicted in Fig. 8 and 9. Here, we also
want to emphasize the reconstruction quality of the upsampled scans.

Validation |  Car easy |  Car mod |  Car hard |  Ped. easy |  Ped. mod |  Ped. hard |  Cyclist easy |  Cyclist mod |  Cyclist hard
---|---|---|---|---|---|---|---|---|---
Low res | 78.8 | 65.0 | 59.7 | 7.5 | 4.6 | 4.2 | 22.9 | 16.2 | 14.9
High res | 87.7 | 88.0 | 86.6 | 57.3 | 51.4 | 46.8 | 81.5 | 62.9 | 59.0
Upsampled | 76.2 | 57.1 | 52.4 | 27.2 | 21.2 | 18.7 | 21.9 | 13.0 | 11.7

TABLE II: Results of the PointPillar for KITTI dataset trained on high
resolution lidar data and validated on decimated, original and upsampled by
our method val sets.

To perform a quantitative evaluation of the results, we assess the generated
data using traditional metrics such as CD, Hausdorf distance (HD), EMD, and
introduced SWD. For the best-performing network snapshots, the metrics
mentioned above are reported in table I. Our generated data reveals
substantial performance improvement concerning HD and SWD.

Furthermore, to verify our hypothesis regarding the SWD loss, we study the
effects of different losses CD, EMD, and SWD on the upsampling performance of
our model. Here we train our network on the KITTI dataset with the patches of
size $1024$ and upsampling ratio $2$ with a similar setup as described before
but with varying loss functions. The results of this study are demonstrated in
Fig. 4. They confirm our initial assumption and are aligned with the
observations made before. Similarly to the reconstruction experiment here, CD
leads to sparseness areas, and EMD leads to distortions. That observation also
confirms the findings of previous works, which state that CD is a weaker
metric than 1-Wasserstein and SWD approximation is more accurate than
approximated EMD.

##  V Conclusion

In this work, we provide a network for lidar point cloud upsampling, which
aims to reconstruct the fine-grained lidar scan patterns accurately. Our
method reveals the required sensitivity to lidar scan lines and shows
convincing results on lidar upsampling on a smaller scale of scan patches. Our
deep network utilizes the edge-aware dense convolutions for feature extraction
and expansion together with Sliced Wasserstein Distance. Both key features
enable our method to perform lidar upsampling in a direct, lightweight one-
stage fashion, excluding the coarse and fine reconstruction. The proposed
method improves on both qualitatively and quantitatively measured baseline
methods with HD, EMD, and SWD metrics. One of the limitations of our approach,
though, is the scale of processed samples. The extension of our method to
large-scale lidar scans ($\geq 100K$ points) remains for future work.

##  VI Acknowledgment

The research leading to these results is funded by the German Federal Ministry
for Economic Affairs and Energy within the project KI Delta Learning. The
authors would like to thank the consortium for the successful cooperation.

## References

  * [1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. J. Guibas.  Learning representations and generative models for 3d point clouds.  In ICML, 2018.
  * [2] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall.  SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences.  In ICCV, 2019.
  * [3] B. Besic, N. Gosala, D. Cattaneo, and A. Valada.  Unsupervised domain adaptation for lidar panoptic segmentation.  In RAL, 2022.
  * [4] L. Caccia, H. van Hoof, A. Courville, and J. Pineau.  Deep generative modeling of lidar data.  In IROS, 2019.
  * [5] R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan.  Learning gradient fields for shape generation.  In ECCV, 2020.
  * [6] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu.  ShapeNet: An Information-Rich 3D Model Repository.  In arxiv, 2015.
  * [7] B. Coors, A. P. Condurache, and A. Geiger.  Nova: Learning to see in novel viewpoints and domains.  In 3DV, 2019.
  * [8] M. Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transportation distances.  In NIPS, 2013.
  * [9] I. Deshpande, Y.-T. Hu, R. Sun, A. Pyrros, N. Siddiqui, S. Koyejo, Z. Zhao, D. Forsyth, and A. Schwing.  Max-sliced wasserstein distance and its use for gans.  In CVPR, 2019.
  * [10] S. et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [11] H. Fan, H. Su, and L. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In CVPR, 2017.
  * [12] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [13] T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry.  Atlasnet: A papier-mâché approach to learning 3d surface generation.  In CVPR, 2018.
  * [14] Y. Jung, S.-W. Seo, and S.-W. Kim.  Fast point clouds upsampling with uncertainty quantification for autonomous vehicles.  In ICRA, 2022.
  * [15] S. Kolouri, P. E. Pope, C. E. Martin, and G. K. Rohde.  Sliced-wasserstein auto-encoders.  In ICLR, 2017.
  * [16] Y. Kwon, Youngsun adnd Sung and S.-E. Yoon.  Implicit lidar network:lidar super-resolution via interpolation weight prediction.  In ICRA, 2022.
  * [17] C.-L. Li, M. Zaheer, Y. Zhang, B. Póczos, and R. Salakhutdinov.  Point cloud gan.  In ICLR, 2018.
  * [18] R. Li, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng.  Pu-gan: a point cloud upsampling adversarial network.  In ICCV, 2019.
  * [19] R. Li, X. Li, P.-A. Heng, and C.-W. Fu.  Point cloud upsampling via disentangled refinement.  In CVPR, 2021.
  * [20] M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu.  Morphing and sampling network for dense point cloud completion.  In AAAI, 2020.
  * [21] S. Luo and W. Hu.  Diffusion probabilistic models for 3d point cloud generation.  In CVPR, 2021.
  * [22] K. Nguyen, N. Ho, T. Pham, and H. Bui.  Distributional sliced-wasserstein and applications to generative modeling.  In ICLR, 2021.
  * [23] T. Nguyen, Q.-H. Pham, T. Le, T. Pham, N. Ho, and B.-S. Hua.  Point-set distances for learning representations of 3d point clouds.  In ICCV, 2021.
  * [24] L. Pan.  Ecg: Edge-aware point cloud completion with graph convolution.  In IEEE IROS, 2020.
  * [25] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [26] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In NIPS, 2017.
  * [27] G. Qian, A. Abualshour, G. Li, A. Thabet, and B. Ghanem.  Pu-gcn: Point cloud upsampling using graph convolutional networks.  In CVPR, 2021.
  * [28] T. Shan, J. Wang, F. Chen, P. Szenher, and B. Englot.  Simulation-based lidar super-resolution for ground vehicles.  In RAS, 2020.
  * [29] S. Shi, X. Wang, and H. Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In CVPR, 2019.
  * [30] D. W. Shu, S. W. Park, and J. Kwon.  3d point cloud generative adversarial network based on tree structured graph convolutions.  In ICCV, 2019.
  * [31] M. Sugiyama and M. Kawanabe.  Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation.  The MIT Press, 2012.
  * [32] Y. Sun, Y. Wang, Z. Liu, J. E. Siegel, and S. E. Sarma.  Pointgrow: Autoregressively learned point cloud generation with self-attention.  In WACV, 2020.
  * [33] L. P. Tchapmi, V. Kosaraju, S. H. Rezatofighi, I. Reid, and S. Savarese.  Topnet: Structural point cloud deco.  In CVPR, 2019.
  * [34] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In ICCV, 2019.
  * [35] D. Valsesia, G. Fracastoro, and E. Magli.  Learning localized generative models for 3d point clouds via graph convolution.  In ICLR, 2019.
  * [36] T. Vayer, R. Flamary, R. Tavenard, L. Chapel, and N. Courty.  Sliced gromov-wasserstein.  In NIPS, 2020.
  * [37] X. Wang, M. H. Ang Jr, and G. H. Lee.  Cascaded refinement network for point cloud completion with self-supervision.  In CVPR, 2020.
  * [38] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon.  Dynamic graph cnn for learning on point clouds.  In Transactions on Graphics, 2019.
  * [39] J. Wu, Z. Huang, D. Acharya, W. Li, J. Thoma, D. P. Paudel, and L. Van Gool.  Sliced wasserstein generative models.  In CVPR, 2019.
  * [40] G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In ICCV, 2019.
  * [41] Y. Yang, C. Feng, Y. Shen, and D. Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In CVPR, 2018.
  * [42] W. Yifan, S. Wu, H. Huang, D. Cohen-Or, and O. Sorkine-Hornung.  Patch-based progressive 3d point set upsampling.  In CVPR, 2019.
  * [43] T. Yin, X. Zhou, and P. Krähenbühl.  Center-based 3d object detection and tracking.  In CVPR, 2021.
  * [44] L. Yu, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng.  Ec-net: an edge-aware point set consolidation network.  In ECCV, 2018.
  * [45] L. Yu, X. Li, C.-W. Fu, and H. P.-A. Cohen-Or, Daniel.  Pu-net: Point cloud upsampling network.  In CVPR, 2018.
  * [46] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert.  Pcn: Point completion network.  In International Conference on 3D Vision, 2018.
  * [47] W. Zhang, H. Jiang, Z. Yang, S. Yamakawa, K. Shimada, and L. B. Kara.  Data-driven upsampling of point clouds.  In CAD, 2018.