# Legendre Transformation under Micro Canonical Ensemble

Jingxu Wu1, Chenjia Li2, Zhenzhou Lei1∗, Tuerdi Wumaier1∗, Congyu Li3, Yan
Wang4, & Zekun Wang5
1Key Laboratory of New Energy Materials Research, Xinjiang Institute of
Engineering,Urumqi, China
2Department of Physics, Moscow Lomonosov State University, Moscow, Russia
3Cuiying Honors College, Lanzhou University, Lanzhou, China
4School of Physics Science and Technology, Lanzhou University, Lanzhou, China
5College of Physics, Jilin University, Changchun, China
Corresponding author email<EMAIL_ADDRESS>author email
<EMAIL_ADDRESS>
###### Abstract

The Legendre transformation is a crucial tool in theoretical physics, known
for its symmetry, especially when applied to multivariate functions. In
statistical mechanics, ensembles represent the central focus. Leveraging the
dimensionless aspect of Legendre transformation, this paper explores the
transformation process from the entropy characteristic function of
microcanonical ensembles to the analogous definition of partition function
transformation. Additionally, it derives characteristic functions, partition
functions, and establishes their interrelations, along with deriving
corresponding thermodynamic formulas for various ensembles. This streamlined
approach sheds light on the fundamental principles of statistical mechanics
and underscores the symmetry inherent in Legendre transformation.

##  I Introduction

The Lagrange transformation, attributed to the pioneering work of Joseph-Louis
Lagrange in the late 18th century[1,2,3], stands as a fundamental concept in
classical mechanics, offering a powerful mathematical framework for the
analysis of dynamic systems. This transformation serves as a cornerstone in
the field, providing a means to elegantly describe and solve complex physical
problems by converting the system’s representation from Cartesian coordinates
to generalized coordinates and momenta.

Introduced as part of Lagrange’s efforts to reformulate classical mechanics,
the Lagrange transformation offers a more streamlined approach to solving
problems involving the motion of particles and rigid bodies. By expressing the
system’s dynamics in terms of generalized coordinates and momenta, rather than
traditional position and momentum variables, the transformation enables a more
comprehensive and intuitive understanding of the underlying physics[4,5,6].

Through the lens of the Lagrange transformation, Hamilton’s principle becomes
a central guiding principle, allowing for the derivation of Hamilton’s
equations of motion. These equations, in turn, provide a concise and powerful
framework for describing the evolution of a system over time, offering
insights into the underlying dynamics and enabling the prediction of future
states.

In this paper, we explore the historical context and theoretical underpinnings
of the Lagrange transformation, its mathematical formulation, and its
practical applications in classical mechanics. By elucidating the principles
and techniques involved, we aim to provide a comprehensive understanding of
this fundamental concept and its significance in the broader context of
theoretical physics[7,8,9,10]. The Lagrange transformation originates from the
following partial differential equation:

$R\frac{\partial^{2}z}{\partial x^{2}}+S\frac{\partial^{2}z}{\partial
x\partial y}+T\frac{\partial^{2}z}{\partial y}=0\quad\text{(1.1)}$

Where, if we let $p=\frac{\partial z}{\partial x}$ and $q=\frac{\partial
z}{\partial y}$, and further assume that $R$, $S$, and $T$ are only functions
of $p$ and $q$, then for the tangent plane of the surface $z=f(x,y)$ given by:

$px+qy+z-v=0\quad\text{(1.2)}$

There should exist:

$R\frac{\partial^{2}v}{\partial q^{2}}-S\frac{\partial^{2}v}{\partial
p\partial q}+T\frac{\partial^{2}v}{\partial q^{2}}=0\quad\text{(1.3)}$

Eq. (1.2) provides a transformation between the variables $x$, $y$, and their
conjugate variables $p$, $q$. Specifically, the transformation can be
expressed as the partial derivatives:

$\frac{\partial z}{\partial x}=p,\quad\frac{\partial z}{\partial
y}=q\quad,\frac{\partial v}{\partial p}=x,\quad\frac{\partial v}{\partial
q}=y\quad\text{(1.4)}$

Considering the Jacobian matrices of the transformation should be mutually
inverse, i.e.,

$\begin{pmatrix}\frac{\partial p}{\partial x}&\frac{\partial p}{\partial y}\\\
\frac{\partial q}{\partial x}&\frac{\partial q}{\partial
y}\end{pmatrix}=\begin{pmatrix}\frac{\partial z^{2}}{\partial
x^{2}}&\frac{\partial z^{2}}{\partial x\partial y}\\\ \frac{\partial
z^{2}}{\partial x\partial y}&\frac{\partial z^{2}}{\partial
y^{2}}\end{pmatrix},\begin{pmatrix}\frac{\partial x}{\partial
p}&\frac{\partial y}{\partial p}\\\ \frac{\partial x}{\partial
q}&\frac{\partial y}{\partial q}\end{pmatrix}=\begin{pmatrix}\frac{\partial
v^{2}}{\partial p^{2}}&\frac{\partial v^{2}}{\partial p\partial q}\\\
\frac{\partial v^{2}}{\partial p\partial q}&\frac{\partial v^{2}}{\partial
q^{2}}\end{pmatrix}$

We obtain:

$\displaystyle\frac{\partial^{2}z}{\partial x^{2}}$
$\displaystyle=\frac{1}{\Delta}\frac{\partial^{2}v}{\partial q^{2}}$
$\displaystyle\frac{\partial^{2}z}{\partial x\partial y}$
$\displaystyle=\frac{1}{\Delta}\frac{\partial^{2}v}{\partial p\partial q}$
$\displaystyle\frac{\partial^{2}z}{\partial y^{2}}$
$\displaystyle=\frac{1}{\Delta}\frac{\partial^{2}v}{\partial p^{2}}$

and

$\Delta=\left\|\begin{pmatrix}\frac{\partial^{2}v}{\partial
p^{2}}&\frac{\partial^{2}v}{\partial p\partial q}\\\
\frac{\partial^{2}v}{\partial p\partial q}&\frac{\partial^{2}v}{\partial
q^{2}}\end{pmatrix}\right\|$

This transformation converts a quasi-linear Eq. (1.1) into a linear Eq.(1.3).

##  II Lagrange transformation

We have

$\displaystyle y$ $\displaystyle=y(x)\quad$ (2.1) $\displaystyle y^{\prime}$
$\displaystyle=t=\frac{{dy}}{{dx}}\quad$ (2.2) $\displaystyle u$
$\displaystyle=u(t)\quad$ (2.3) $\displaystyle u^{\prime}$
$\displaystyle=\frac{{du}}{{dt}}\quad$ (2.4)

From (2.2), solve for x as a function of t and substitute into the following

$\displaystyle u$ $\displaystyle=xt-y\quad$ (2.5)

Differentiating the expression $u=xt-y$ with respect to $t$, we get:

$\displaystyle du$ $\displaystyle=xdt+tdx-\frac{dy}{dx}dx$
$\displaystyle=xdt+(t-\frac{dy}{dx})dx$ $\displaystyle=xdt$

From this, it’s evident that $u$ is a function of $t$, and its derivative with
respect to $t$ is $x$. Eq. (2.5) determines a transformation between the
variables $u$, $y$, $x$, and $t$. It maps $y=y(x)$ to

$t=t(x),\quad u=u(t)\quad(2.6)$

Similarly,$y=xt-u$ from Eq.(2.5), it follows that it defines another
transformation that is the inverse of the one above. Hence, these two
transformations are mutually inverse, and their relation is symmetric.

The Legendre transformation has a property: if two curves are tangent on the
$x$-$y$ plane, then after transformation to the $u$-$t$ plane, they are also
tangent, and vice versa. Transformations with this property are called contact
transformations. The Legendre transformation is a special case of contact
transformations.

Extending the above ideas to the case of multiple variables, consider a
function $U$ of $n$ variables $q_{1},q_{2},\ldots,q_{n}$, which has continuous
partial derivatives up to the second order. Let $U$ be transformed into a new
set of variables $Q_{1},Q_{2},\ldots,Q_{n}$ according to the equation:

$Q_{i}=\frac{\partial U}{\partial q_{i}},(i=1,2,\ldots,n)\quad(2.7)$

They constitute a set of transformations with respect to the original
variables $q_{1},q_{2},\ldots,q_{n}$. The Jacobian determinant of these
transformations with respect to the original variables
$q_{1},q_{2},\ldots,q_{n}$ is given by:

$\left\|\frac{\partial Q_{i}}{\partial
q_{j}}\right\|=\left\|\frac{\partial^{2}U}{\partial q_{i}\partial
q_{j}}\right\|$

From Eq.(2.7), the original variables can be solved out as

$q_{i}=q_{i}(Q_{1},Q_{2},\ldots,Q_{n}),\quad(i=1,2,\ldots,n)\quad(2.8)$

Considering the new function:

$U^{C}=\sum_{i=1}^{n}Q_{i}\cdot q_{i}-U\quad(2.9)$

Differentiating the above equation, we get:

$dU^{C}=\sum_{i=1}^{n}\frac{\partial U^{C}}{\partial
q_{i}}\,dq_{i}=\sum_{i=1}^{n}{\partial q_{i}}\,dQ_{i}$

We have thus demonstrated:

$q_{i}=\frac{\partial U^{C}}{\partial Q_{i}}\quad(i=1,2,\ldots,n)\quad(2.10)$

The relationship between two functions $U$ and $U^{C}$ is given by Eq.(2.9).
The corresponding relationships between variables and functions are
respectively given by Eqs.(2.7) and (2.10). They encapsulate many duality
relationships in mechanics and physics.

##  III The Symmetry of Legendre Transform

Multivariable function: $\Phi=\Phi(x_{1},x_{2},\ldots,x_{n})$, where $x_{i}\
(i=1,2,\ldots,n)$ are independent variables. For convenience, let’s assume
$\Phi$ is smooth and differentiable in $N$-dimensional space, such that at
each point, $s_{i}=\frac{\partial\Phi}{\partial
x_{i}}=\frac{\partial\Phi}{\partial x_{i}}$, then

$d\Phi=\sum_{i=1}^{n}\frac{\partial\Phi}{\partial
x_{i}}dx_{i}=\sum_{i=1}^{n}s_{i}dx_{i}\quad(3.1)$

[11,12]

If we take $m\ (m\leq n)$ of the variables from the function
$\Phi=\Phi(x_{1},x_{2},\ldots,x_{n})$, the independent variables $x_{j}\
(j=1,2,\ldots,m)$ are replaced by their corresponding variables $s_{j}$,
forming a new function $\Psi$, defined by
$L_{j}:\Phi(x_{1},x_{2},\ldots,x_{m},x_{m+1},\ldots,x_{n})\rightarrow\Psi(s_{1},s_{2},\ldots,s_{m},x_{m+1},\ldots,x_{n})$,
where

$\Psi=\sum_{j=1}^{m}s_{j}x_{j}-\Phi\quad(3.2).$

Considering Eq.(3.1), we have
$d\Psi=\sum_{j=1}^{m}(s_{j}dx_{j}+x_{j}ds_{j})-\sum_{i=1}^{n}s_{i}dx_{i}$,
where $j=1,2,\ldots,m$; $i=1,2,\ldots,n$; and $m\leq n$. Simplifying, we get

$d\Psi=\sum_{j=1}^{m}x_{j}ds_{j}-\sum_{i=m+1}^{n}s_{i}dx_{i}\quad(3.3).$

Since any given set of variables uniquely determines a function, different
functions are essentially distinguished by their sets of independent
variables. Therefore, the differential form of any function can be expressed
as

$d\Psi=\sum_{j=1}^{m}\frac{\partial\Psi}{\partial
s_{j}}ds_{j}+\sum_{i=m+1}^{n}\frac{\partial\Psi}{\partial
x_{i}}dx_{i}\quad(3.4).$

Comparing the first half of Eq. (3.4) with Eq. (3.3), we find
$x_{j}=\frac{\partial\Psi}{\partial s_{j}}$, which corresponds to
$s_{j}=\frac{\partial\Phi}{\partial x_{j}}$. This shows that the two variables
before and after the transformation are interrelated. Additionally, Eq. (3.2)
can be rewritten as

$\Phi+\Psi=\sum_{j=1}^{m}s_{j}x_{j}\quad(3.5)$

The functions before and after the transformation are equivalent,
demonstrating the clear symmetry of the Legendre transformation.

##  IV Legendre transformation enters statistical mechanics

The ensemble method is a core concept in the teaching of statistical
mechanics, generally studying three ensembles: microcanonical ensemble,
canonical ensemble, and grand canonical ensemble. For a system with $n$
independent variables, the number $N$ of equivalent characteristic functions
is given by

$N=\sum_{i=0}^{n}C_{i}^{n}=2^{n}\quad(4.1)$

For a uniform system with three independent variables, there are 8 equivalent
characteristic functions, implying a total of 8 ensembles. Starting from the
fundamental relationship of the microcanonical ensemble, this paper utilizes
the Legendre transformation of entropy and analogously defines the
transformation of the partition function, while obtaining the characteristic
functions and partition functions of various ensembles.

The characteristic function of the microcanonical ensemble is entropy, with
the corresponding thermodynamic formula:

$S=k\ln\Omega(E,V,N)\quad(4.2)$

It is often challenging to derive other functions from entropy using Legendre
transformation. Due to historical reasons, the variables involved in Legendre
transformation do not always appear in pairs. For example, the energy $E$ of a
system is paired with the reciprocal of temperature ($\beta=1/kT$, where $k$
is the Boltzmann constant). However, temperature is commonly used in many
relationships, such as the familiar equation $F=E-TS$, which relates free
energy and entropy but obscures the symmetry between $\beta$ and $E$. If we
define dimensionless quantities: $S^{\prime}=S/k$, $F^{\prime}=\beta F$, their
duality can be perfectly represented as $F^{\prime}(\beta)+S^{\prime}(E)=\beta
E$. Therefore, we rewrite Eq. (4.2) as

$S/k=\ln\Omega(E,V,N)\quad(4.3)$

In differential form:

$d\ln\Omega(E,V,N)=dS(k)=\beta dE+\gamma dV+\alpha dN\quad(4.4)$

Starting from the dimensionless quantity $S^{\prime}$, using Legendre
transformation to obtain other dimensionless quantities, we have 3 pairs of
corresponding variables: $\beta,E$; $\alpha,N$; $\gamma,V$. To express the
dimensionless quantities as characteristic functions in statistical mechanics,
we also need to know the common expressions of $\beta,\alpha,\gamma$. From the
thermodynamic formula $dS=(1/T)dE+(p/T)dV-(\mu/T)dN$, we obtain

$dS(k)=(1/kT)dE+(p/kT)dV-(\mu/kT)dN\quad(4.5)$

Comparing Eqs. (4.4) and (4.5), we have

$\beta=1/kT,\quad\gamma=\beta p,\quad\alpha=-\beta\mu\quad(4.6)$

Then Eq. (4.4) becomes

$d\ln\Omega=dS(k)=\beta dE+\beta pdV-\beta\mu dN$

From this equation, we derive:

$\beta=\frac{\partial\ln\Omega}{\partial E}\bigg{|}_{V,N},\quad
p=\frac{1}{\beta}\frac{\partial\ln\Omega}{\partial
V}\bigg{|}_{E,N},\quad\mu=-\frac{1}{\beta}\frac{\partial\ln\Omega}{\partial
N}\bigg{|}_{E,V}$

##  V The Application of Legendre Transform in Statistical Mechanics

Starting from $S_{k}(E,V,N)$ and performing Legendre transformations to other
functions, we can analogously define the transformation of partition
functions.

5.1 Transformation of characteristic functions

There are a total of 7 characteristic functions equivalent to $S_{k}(E,V,N)$,
denoted by $x_{1}=E$, $x_{2}=V$, $x_{3}=N$, $s_{1}=\beta$, $s_{2}=\gamma$,
$s_{3}=\alpha$.

5.1.1 Transformation of microcanonical ensemble characteristic functions

The characteristic function of the microcanonical ensemble is entropy, and
there are 3 possible Legendre transformations when changing one variable of
$S_{k}(E,V,N)$, corresponding to 3 characteristic functions and 3 ensembles.

If the variable transformation is $E\rightarrow\beta$, meaning only energy
exchange between the system and the surroundings, we have from Eq. (3.5):

$\Psi_{1}(\beta,V,N)+S_{k}(E,V,N)=\beta E\quad(5.1)$

Rewriting this as $\Psi_{1}(\beta,V,N)=\beta E-S_{k}(E,V,N)$ and substituting
$\beta=1/kT$, we get:

$\Psi_{1}(\beta,V,N)=\beta E-\beta TS=\beta(E-TS)\quad(5.2)$

Letting $\Psi_{1}=\beta F$, we obtain $F+TS=E$, which is the familiar
thermodynamic equation. Here, $F(T,V,N)$ is the Helmholtz free energy, the
characteristic function of the canonical ensemble. From equation (3), we get
the differential equation for the function: $d\Psi_{1}=Ed\beta-\gamma
dV-\alpha dN$, and considering $\gamma=\beta p$, $\alpha=-\beta\mu$, we have:

$d\Psi_{1}=Ed\beta-\beta pdV+\beta\mu dN\quad(5.3)$

If we substitute $\Psi_{1}=\beta F$ into this equation, we get $dF=-SdT-
pdV+\mu dN$, which is the familiar differential equation for the Helmholtz
free energy.

If the variable transformation is $V\rightarrow\gamma$, meaning only force
interactions between the system and the surroundings, we obtain a new ensemble
characterized by:

$\Psi_{2}(E,\gamma,N)+S_{k}(E,V,N)=\gamma V$

If the variable transformation is $N\rightarrow\alpha$, meaning only particle
number exchange between the system and the surroundings, we obtain another new
ensemble characterized by:

$\Psi_{3}(E,V,\alpha)+S_{k}(E,V,N)=\alpha N$

5.1.2 Transformation of characteristic functions by changing two variables

If we change two variables of $S_{k}(E,V,N)$, there are 3 possible Legendre
transformations, corresponding to 3 characteristic functions and 3 ensembles.

If the variable transformation is $E\rightarrow\beta$, $V\rightarrow\gamma$,
meaning only energy exchange and force interactions between the system and the
surroundings, we have from Eq. (3.5):

$\Psi_{4}(\beta,\gamma,N)+S_{k}(E,V,N)=\beta E+\gamma V$

Letting $\Psi_{4}=\beta G$ and substituting $\beta=1/kT$, $\gamma=\beta p$, we
obtain $G+TS=E+pV$, where $G$ is the Gibbs function, a familiar thermodynamic
quantity.

If the variable transformation is $E\rightarrow\beta$, $N\rightarrow\alpha$,
meaning only energy exchange and particle number exchange between the system
and the surroundings, we obtain another ensemble characterized by:

$\Psi_{5}(\beta,V,\alpha)+S_{k}(E,V,N)=\beta E+\alpha N$

Letting $\Psi_{5}=\beta J$ and substituting $\beta=1/kT$, $\alpha=-\beta\mu$,
we obtain $J+TS=E-\mu N$, where $J(T,V,\mu)$ is the grand canonical potential,
precisely the characteristic function of the grand canonical ensemble.

If the variable transformation is $V\rightarrow\gamma$, $N\rightarrow\alpha$,
meaning only force interactions and particle number exchange between the
system and the surroundings, we obtain another ensemble characterized by:

$\Psi_{6}(E,\gamma,\alpha)+S_{k}(E,V,N)=\gamma V+\alpha N$

5.1.3 Transformation of characteristic functions by changing three variables

If we change three variables of $S_{k}(E,V,N)$, there is 1 possible Legendre
transformation, corresponding to 1 characteristic function and 1 ensemble.

If the variable transformation is $E\rightarrow\beta$, $V\rightarrow\gamma$,
$N\rightarrow\alpha$, meaning energy exchange, force interactions, and
particle number exchange between the system and the surroundings, we obtain
another ensemble characterized by:

$\Psi_{7}(\beta,\gamma,\alpha)+S_{k}(E,V,N)=\beta E+\gamma V+\alpha N$

This last Legendre transformation changes three variables, resulting in the
zero ensemble. Therefore, there are effectively 7 ensemble distributions.

5.2 Transformation of partition functions

Considering $\beta=1/kT$, the partition function of the microcanonical
ensemble is $\Omega(E,V,N)=e^{S_{k}}=e^{\beta TS}$.

If the variable transformation is $E\rightarrow\beta$, the characteristic
function is transformed from $S_{k}(E,V,N)$ to $\Psi_{1}(\beta,V,N)$ as
follows:

$\Psi_{1}(\beta,V,N)=\beta E-\beta TS=\beta F\quad(5.4)$

Correspondingly, the partition function $\Omega(E,V,N)$ is transformed to:

$Z(\beta,V,N)=\sum_{E}\Omega(E,V,N)e^{-\beta E}=\sum_{E}e^{\beta TS}e^{-\beta
E}\quad(5.5)$

which is the partition function of the canonical ensemble.

Similarly, by analogy with Legendre transformations of characteristic
functions, we can derive the partition functions of other ensembles.

If the variable transformation is $V\rightarrow\gamma$, the partition function
is transformed to:

$A(E,\gamma,N)=\sum_{E}\Omega(E,V,N)e^{-\gamma V}$

If the variable transformation is $N\rightarrow\alpha$, the partition function
is transformed to:

$B(E,V,\alpha)=\sum_{E}\Omega(E,V,N)e^{-\alpha N}$

If the variable transformation is $E\rightarrow\beta$, $V\rightarrow\gamma$,
the partition function is transformed to:

$C(\beta,\gamma,N)=\sum_{E}\Omega(E,V,N)e^{-\beta E}e^{-\gamma V}$

If the variable transformation is $E\rightarrow\beta$, $N\rightarrow\alpha$,
the partition function is transformed to:

$\Xi(\beta,V,\alpha)=\sum_{E}\Omega(E,V,N)e^{-\beta E}e^{-\alpha N}$

which is the partition function of the grand canonical ensemble.

If the variable transformation is $V\rightarrow\gamma$, $N\rightarrow\alpha$,
the partition function is transformed to:

$D(E,\gamma,\alpha)=\sum_{E}\Omega(E,V,N)e^{-\gamma V}e^{-\alpha N}$

If the variable transformation is $E\rightarrow\beta$, $V\rightarrow\gamma$,
$N\rightarrow\alpha$, the partition function is transformed to:

$M(\beta,\gamma,\alpha)=\sum_{E}\Omega(E,V,N)e^{-\beta E}e^{-\gamma
V}e^{-\alpha N}$

5.3 Relationship between partition functions and characteristic functions

For the partition function of the canonical ensemble given in Eq. (5.5),
taking the logarithm of both sides:

$\ln Z(\beta,V,N)=\beta TS-\beta E=-\beta(E-TS)$

Considering Eq. (5.3), we have the relationship between the partition function
and the characteristic function of the canonical ensemble:

$\ln Z(\beta,V,N)=-\Psi_{1}(\beta,V,N)=-\beta F\quad(5.6)$

Therefore, the characteristic function $F=-kT\ln Z(\beta,V,N)$, which is an
important expression for thermodynamic quantities in the canonical ensemble.
Considering Eq.(5.2), we have the thermodynamic formulas for the canonical
ensemble:

$E=-\left(\frac{\partial\ln Z}{\partial\beta}\right)_{V,N},\quad
p=\frac{1}{\beta}\left(\frac{\partial\ln Z}{\partial V}\right)_{\beta},$
$\quad\mu=-\frac{1}{\beta}\left(\frac{\partial\ln Z}{\partial
N}\right)_{\beta}\quad(5.7)$

which are consistent with what we learned in statistical mechanics.

Similarly, by analogy with Legendre transformations of characteristic
functions, we can derive the relationships between partition functions and
characteristic functions for other ensembles along with their corresponding
thermodynamic formulas.

##  VI Conclusion

By deducing the Legendre transform and its application in multivariate
functions in statistical mechanics, we can further understand the symmetry and
universality of the Legendre transformation. From the above introduction, the
transformation of characteristic functions unifies with the transformation of
partition functions as introduced in statistical mechanics. From a
mathematical perspective, transitioning from one ensemble to another in
statistical mechanics is essentially a Legendre transformation involving
variable transformations. Moreover, Legendre transformation does not alter the
properties of the system. Therefore, from this perspective, these ensembles
are equivalent.

## References

  * [1] riffiths, D. J. (2005). Introduction to Quantum Mechanics. Prentice Hall.
  * [2] rnold, V. I. (1978). Mathematical Methods of Classical Mechanics. Springer-verlage.
  * [3] acKay, D. J. C. (2003). Legendre Transformations and Maximum Entropy. arXiv preprint arXiv:physics/0301031.
  * [4] hen, S.-H., et al. (2019). Legendre transformation and a Legendre symmetry. arXiv preprint arXiv:1904.12527.
  * [5] rishnan, C., et al. (2021). Quantum Legendre Transform. arXiv preprint arXiv:2103.02864.
  * [6] urakami, Y., et al. (2021). Stochastic thermodynamics of quantum systems: a quantum Legendre transform approach. arXiv preprint arXiv:2107.05054.
  * [7] iong, W., et al. (2019). Legendre Transformation in Finite Quantum Systems. arXiv preprint arXiv:1907.04494.
  * [8] inchenko, Y. (2019). On the Legendre Transform of a Convex Function. arXiv preprint arXiv:1901.06363.
  * [9] iebl, S., et al. (2019). The Legendre Transformation and its Applications. arXiv preprint arXiv:1906.05424.
  * [10] ataki, G. (2001). The Convex Hull of the Legendre Transform. arXiv preprint arXiv:math/0102007.
  * [11] oemisch, W., et al. (1993). The Generalized Legendre Transform in Convex Optimization. arXiv preprint arXiv:math/9301212.
  * [12] ramér, H. (1946). Differential entropy and the Legendre transform. arXiv preprint arXiv:math/0108119.


# Low-Resource White-Box Semantic Segmentation of Supporting Towers on 3D
Point Clouds via Signature Shape Identification

Diogo Lavado, Cláudia Soares
NOVA School of Science and Technology
Lisbon
<EMAIL_ADDRESS>&Alessandra Micheletti, Giovanni Bocchi
University of Milan,
Milan
<EMAIL_ADDRESS>
&Alex Coronati, Manuel Silva
EDP NEW,
Lisbon,
<EMAIL_ADDRESS>
Patrizio Frosini,
University of Bologna,
Bologna,
<EMAIL_ADDRESS>
###### Abstract

Research in 3D semantic segmentation has been increasing performance metrics,
like the IoU, by scaling model complexity and computational resources, leaving
behind researchers and practitioners that (1) cannot access the necessary
resources and (2) do need transparency on the model decision mechanisms. In
this paper, we propose SCENE-Net, a low-resource white-box model for 3D point
cloud semantic segmentation. SCENE-Net identifies signature shapes on the
point cloud via group equivariant non-expansive operators (GENEOs), providing
intrinsic geometric interpretability. Our training time on a laptop is 85 min,
and our inference time is 20 ms. SCENE-Net has 11 trainable geometrical
parameters, and requires fewer data than black-box models. SCENE–Net offers
robustness to noisy labeling and data imbalance and has comparable IoU to
state-of-the-art methods. With this paper, we release a 40 000 Km labeled
dataset of rural terrain point clouds and our code implementation.

##  1 Introduction

(a) TS40K Sample

(b) SCENE-Net

(c) Baseline CNN

Figure 1: Signature shapes for power line supporting tower detection. For our
TS40K sample shown in (a), SCENE-Net accurately detects the body of the tower
(b), while a comparable CNN has a large false positive area in the vegetation
(c). Our model is interpretable with 11 trainable geometric parameters whereas
the CNN has a total of 2190 parameters. The ground and power lines are
mislabeled in the ground truth.

Powerful Machine Learning (ML) algorithms applied to critical applications,
such as autonomous driving or environmental protection, highlight the
importance of (1) ease of implementation for non-tech organizations entailing
data efficiency and general-purpose hardware, and (2) transparent models
regarding their decision-making process, thus ensuring a responsible
deployment [1, 2, 3]. Most methods in Explainable AI (XAI) provide post hoc
explanations to black-box models (i.e., algorithms unintelligible to humans).
However, these are often limited in terms of their model fidelity [1, 4], that
is, they provide explanations for the predictions of the underlying model
(e.g., heatmaps [5, 6] and input masks [7, 8]), instead of providing a
mechanistic understanding of its inner-workings. Conversely, intrinsic
interpretability methods (i.e., white-box models) provide an understanding of
their decisions through their architecture and parameters [4]. Transparency is
achieved by enforcing constraints that reflect domain knowledge and simple
designs [1, 9], which can result in a loss in performance when compared to
complex black-boxes.

We propose a novel white-box model, SCENE-Net, that provides intrinsic
geometric interpretability by leveraging on group equivariant non-expansive
operators (GENEOs) [10, 11]. Unlike traditional interpretable models, GENEOs
are complex observers parameterized with meaningful geometric features. In our
case, task dependency comes as a collaboration of Machine Learning and
electrical utility teams to transparently segment power line supporting towers
on 3D point clouds to inspect extensive power grids automatically. Electrical
grid operators have the critical job of assessing the risk of contact between
the power grid and its environment to prevent failures and forest fires. These
grids spread over countries and even continents, thus making careful
inspection an important and challenging problem. Often, this task is based on
LiDAR large-scale point clouds with high-point density, no sparsity, and no
object occlusion. However, the captured point clouds are quite extensive and
mostly composed of rural areas. These data are different from large urban
datasets for autonomous driving [12, 13] due to the point of view, point
density, occlusion, and extension. To bootstrap this work, we created a
labeled dataset of 40 000 Km of rural and forest terrain, and the Transmission
System, named TS40K. These point clouds show noisy labels and class imbalance
(see Appendix B for details), and our SCENE-Net is robust to labeling noise as
it encodes the geometric properties we need to detect.

Moreover, practitioners in high-risk tasks, such as autonomous driving and
power grid inspection, are often limited in terms of resources, namely
computational power and available data, to train and deploy state-of-the-art
models [14, 15]. This clashes with the current trend in DL to scale up models
in both complexity and needed resources in order to maximize performance, for
example, state-of-the-art 3D semantic segmentation models [16, 17, 18, 19, 20]
follow this trend. Our model, SCENE-Net, maintains a simple design
conventional to white-boxes (it is composed of 11 trainable parameters) that
allows for resource-efficient training while taking advantage of powerful Deep
Learning (DL) strategies, such as convolutions. By assessing our model on the
SemanticKITTI benchmark [13], we show that SCENE-Net achieves performance on-
par with state-of-the-art methods in pole segmentation.

Our main contributions are:

  * •

SCENE-Net is the first white-box model for 3D semantic segmentation on large-
scale landscapes, including non-urban environments (Section 4);

  * •

The architecture of SCENE-Net has fewer trainable parameters than traditional
methods and is resource-efficient in both data and computational requirements
(Section 5);

  * •

Empirically, SCENE-Net is intrinsically and posthoc interpretable and robust
under noisy labels, with au par IoU (Section 5);

  * •

We present TS40K, a new 3D point cloud dataset covering 40 000 Km of non-urban
terrain, with more than 9000 million 3D points (details in Appendix B);

##  2 Related Work

#### Point Cloud Semantic Segmentation.

Processing point clouds is a challenging task due to their unstructured nature
and invariance to permutations. Voxel-based strategies endow point clouds with
structure in order to apply 3D convolutions [21, 22, 23]. However, memory
footprint is too large for high-resolution voxel grids, while low resolution
entails information loss. Subsequent methods try to answer these issues by
employing sparse convolutions [24, 25] and octree-based CNNs [26, 27]. Point-
based models take point clouds directly as input. The work of PointNet [28]
and PointNet++ [29] inspired the use of point sub-sampling strategies with
feature aggregation techniques to learn local features on each sub-point [30,
31]. Convolution-based methods [32, 33, 34, 16, 35] demonstrate good
performance on 3D semantic segmentation benchmarks, such as SemanticKITTI [13]
and SensatUrban [36]. Following this strategy, recent methods exploit multi-
representation fusion, i.e, they combine different mediums (voxel grids, raw
point clouds, and projection images) to boost feature retrieval [37, 17, 18,
20] and achieve top performance on the above benchmarks. While voxel-based
methods are computationally expensive due to 3D convolutions on high-
resolution voxel grids, point-based strategies have to use costly neighbor
searching to extract local information. We propose a voxel-based architecture
that is time-efficient with high-resolution voxel grids, with shapes of
$64^{3}$ and $128^{3}$. Moreover, learning from imbalanced and noisy data is
still a challenging task in point cloud segmentation [38], SCENE-Net is
interpretable and robust to these conditions.

#### Explainable Machine Learning.

Explainability is a crucial aspect of ML methods in high-stakes tasks such as
autonomous driving [1, 2, 3]. Two main approaches have been proposed in the
literature: post hoc explainability, and intrinsic interpretability. Post hoc
methods, such as LIME [7], meaningful perturbations [8], anchors [39], and
ontologies [40], are applied to trained black-box models and provide instance-
based explanations that correlate model predictions to the given input. These
methods are model-agnostic, and thus more flexible, but they often lack
mechanistic cause-effect relations and have a limited understanding of feature
importance [4]. For instance, a dog image and random noise may generate
similar importance heatmaps for the same class with the LIME method [4].
Moreover, they introduce computational overhead, which may limit their
application in real-world scenarios with complex black-box models, such as in
the 3D semantic segmentation task.

In contrast, intrinsic interpretability methods provide an understanding of
their decisions through their architecture and parameters [4]. Decision trees
and linear regression are examples of white-box models. However, transparency
is usually achieved by imposing domain constraints and simple designs, which
implies limited performance compared to deep neural networks. Recent advances
in interpretable techniques, such as concept whitening [9] and interpretable
CNNs [41], have shown that interpretability does not have to imply performance
loss. However, these methods provide evidential interpretability, that is,
they offer intrinsic explanations to model predictions that are still linked
to human interpretations and may imply an evidential correlation, but not
causation.

We propose a white-box model, SCENE-Net, with intrinsic geometric
interpretability that is not subject to human interpretation. SCENE-Net
analyzes the input 3D space according to prior knowledge of the geometry of
objects of interest, which is encoded in functional observers and whose
parameters are fine-tuned during training. These observers encode high-level
geometrical concepts. Thus, our predictions exhibit direct mechanistic cause-
effect w.r.t. the learned observers. SCENE-Net maintains a simple model design
in high-level mathematical operations while taking advantage of DL complex
convolutional kernels.

##  3 Group Equivariant Non-Expansive Operators (GENEOs).

GENEOs are the building blocks of a mathematical framework [10] that formally
describes machine learning agents as a set of operators acting on the input
data. These operators provide a measure of the world, just as CNN kernels
learn essential features to, for instance, recognize objects. Such agents can
be thought of as observers that analyze data. They transform it into higher-
level representations while respecting a set of properties (i.e., a group of
transformations). An appropriate observer transforms data in such a way that
respects the right group of transformations, that is, it commutes with these
transformations. Formally, we say that the observer is equivariant with
respect to a group of transformations. The framework takes advantage of
topological data analysis (TDA) to describe data as topological spaces.
Specifically, a set of data $X$ is represented by a topological space $\Phi$
with admissible functions $\varphi\colon X\to\mathbb{R}^{3}$. $\Phi$ can be
thought of as a set of admissible measurements that we can perform on the
measurement space $X$. For example, images can be seen as functions assigning
RGB values to pixels. This not only provides uniformity to the framework but
also allows us to shift our attention from raw data to the space of
measurements that characterizes it. Now that the input data is well
represented, let us introduce how the framework defines prior knowledge. Data
properties are defined through maps from $X$ to $X$ that are $\Phi$-preserving
homeomorphisms. That is, the composition of functions in $\Phi$ with such
homeomorphisms produces functions that still belong to $\Phi$. Therefore, we
can define a group $G$ of $\Phi$-preserving homeomorphisms, representing a
group of transformations on the input data for which we require equivariance
to be respected. In other words, $G$ is the group of properties that we chose
to enforce equivariance w.r.t. the geometry in the original data. It is
through $G$ that we embed prior knowledge into a GENEO model. Following the
previous example, planar translations can define a subgroup of $G$.

Let us consider the notion of a perception pair $(\Phi,G)$: it is composed of
all admissible measurements $\Phi$ and a subgroup of $\Phi$-preserving
homeomorphisms $G$.

######  Definition 3.1 (Group Equivariant Non-Expansive Operator (GENEO)).

Consider two perception pairs $(\Phi,G)$ and $(\Psi,H)$ and a homomorphism
$T\colon G\to H$. A map $F\colon\Phi\to\Psi$ is a group equivariant non-
expansive operator if it exhibits equivariance:

$\forall\varphi\in\Phi,\forall g\in G,F(\varphi\circ g)=F(\varphi)\circ T(g)$
(1)

and is non-expansive:

$\displaystyle\begin{split}\forall\varphi_{1},\varphi_{2}\in\Phi,\|F(\varphi_{1})-F(\varphi_{2})\|_{\infty}&\leq\|\varphi_{1}-\varphi_{2}\|_{\infty}\end{split}$
(2)

Non-expansivity and convexity are essential for the applicability of GENEOs in
a machine-learning context. When the spaces $\Phi$ and $\Psi$ are compact,
non-expansivity guarantees that the space of all GENEOs $\mathcal{F}$ is
compact as well. Compactness ensures that any operator can be approximated by
a finite set of operators sampled in the same space. Moreover, by assuming
that $\Psi$ is convex, [10] proves that $\mathcal{F}$ is also convex.
Convexity guarantees that the convex combination of GENEOs is also a GENEO.
Therefore, these results prove that any GENEO can be efficiently approximated
by a certain number of other GENEOs in the same space.

In addition to drastically reducing the number of parameters in the modeling
of the considered problems and making their solution more transparent, we
underline that the use of GENEOs makes available various theoretical results
that allow us to take advantage of a new mathematical theory of knowledge
engineering. We stress that, besides the cited compactness and convexity
theorems, algebraic methods concerning the construction of GENEOs are already
available [42, 43, 44]

##  4 SCENE-Net: Signature geometriC Equivariant Non-Expansive operator
Network

In this section, we introduce the overall architecture of SCENE-Net. Next, we
define the geometrical properties that describe power line supporting towers.
Lastly, we detail the loss function used to train the observer.

Figure 2: Pipeline of SCENE-Net: an input point cloud $\mathcal{P}$ is
measured according to function $\varphi$ and voxelized. This representation
then is fed to a GENEO-layer, where each operator $\Gamma_{i}^{\vartheta_{i}}$
separately convolves the input. A GENEO observer $\mathcal{H}$ is then
achieved by a convex combination of the operators in the GENEO layer.
$\mathcal{M}$ transforms the analysis of the observer into a probability of
belonging to a tower. Lastly, a threshold operation is applied to classify the
voxels. Note that this final step occurs after training is completed.

#### Overview

3D Point clouds are generally denoted as
$\mathcal{P}\in\mathbb{R}^{N\times(3+d)}$, where $N$ is the number of points
and $3+d$ is the cardinality of spatial coordinates plus any point-wise
features, such as colors or normal vectors. The input point cloud is first
transformed in accordance with a measurement function
$\varphi\colon\mathbb{R}^{3}\to\\{0,1\\}$ that signals the presence of 3D
points in a voxel discretization. Next, the transformed input is fed to a
layer of multiple GENEOs (GENEO-layer), each chosen randomly from a parametric
family of operators, and defined by a set of trainable shape parameters
$\vartheta_{i}$ (Fig. 2). Such GENEOs are in the form of convolutional
operators with carefully designed kernels as described later. Not only is
convolution a well-studied operation, but it also offers equivariance w.r.t.
translations by definition. During training, it is not the kernels themselves
that are fine-tuned with back-propagation, since this would not preserve
equivariance at each optimization step. Instead, the error is propagated to
the shape parameters $\vartheta_{i}$ of each operator. Following the GENEO-
layer, its set of operators
$\Gamma=\\{\Gamma_{i}^{\vartheta_{i}}\\}_{i=1}^{K}$, with shape parameters
$\vartheta=\vartheta_{1},\dots,\vartheta_{k}$, are combined through convex
combination with weights $\lambda=(\lambda_{1},\dots,\lambda_{k})^{T}$ by
$\underset{\lambda,\vartheta}{\mathcal{H}}\colon\mathcal{P}\to\mathcal{P}$
such that

$\displaystyle\begin{split}&\underset{\lambda,\vartheta}{\mathcal{H}}(x)=\sum_{i=1}^{K}\lambda_{i}\Gamma_{i}^{\vartheta_{i}}(\varphi)(x)\end{split}$
(3)

Since the convex combination of GENEOs is also a GENEO [10], $\mathcal{H}$
preserves the equivariance of each operator $\Gamma^{\vartheta}\in\Gamma$. In
fact, $\mathcal{H}$ defines a GENEO observer that analyzes the 3D input scenes
looking for the geometrical properties encoded in $\Gamma$. The convex
coefficients $\lambda$ represent the overall contribution of each operator
$\Gamma_{i}^{\vartheta}$ to $\mathcal{H}$ to the analysis. The parameters
grant our model its intrinsic interpretability. They are learned during
training and represent geometric properties and the importance of each
$\Gamma^{\vartheta}$ in modeling the ground truth.

Next, we transform the observer’s analysis into a probability of each 3D voxel
belonging to a supporting tower as a model
$\underset{\lambda,\vartheta}{\mathcal{M}}\colon\mathcal{P}\to[0,1]^{N}$

$\displaystyle\begin{split}&\underset{\lambda,\vartheta}{\mathcal{M}}(x)=\bigg{(}\tanh\Big{(}\underset{\lambda,\vartheta}{\mathcal{H}}(x)\Big{)}\bigg{)}_{+}\end{split},$

where $(t)_{+}=\max\\{0,t\\}$ is the rectified linear unit (ReLU). Negative
signals in $\mathcal{H}(x)$ represent patterns that do not exhibit the sought-
out geometrical properties. Conversely, positive values quantify their
presence. Therefore, $\tanh$ compresses the observer’s value distribution into
[-1, 1], and the ReLU is then applied to enforce a zero probability to
negative signals. Lastly, a probability threshold $\tau\in[0,1]$ is defined
through hyperparameter fine-tuning and applied to $\mathcal{M}$ resulting in a
map
$\widetilde{\underset{\lambda,\vartheta}{\mathcal{M}}}\colon\mathcal{P}\times\mathbb{R}\to\\{0,1\\}^{N}$

$\displaystyle\begin{split}&\widetilde{\underset{\lambda,\vartheta}{\mathcal{M}}}(x,\tau)=\Big{\\{}\underset{\lambda,\vartheta}{\mathcal{M}}(x)\Big{\\}}\geq\tau\end{split},$

where $\widetilde{\mathcal{M}}$ denotes the SCENE-Net model.

#### Knowledge Engineering via GENEOs.

In this section, we formally define the knowledge embedded in the observer
$\mathcal{H}$. The following GENEOs describe power line supporting towers in
order to fully discriminate them from their environment.

#### Cylinder GENEO.

The most striking characteristic of supporting towers against the rural
environment is their long, vertical and narrow structure. As such, their
identification is equivariant w.r.t. rotations along the z-axis and
translations in the xy plane, which we encode by the means of a cylinder.

######  Definition 4.1.

In order to promote smooth patterns, a cylinder is defined by
$g_{Cy}\colon\mathbb{R}^{3}\to[0,1]$:

$\displaystyle\begin{split}&g_{Cy}(x)=e^{-\frac{1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-r^{2})^{2}}\end{split}$

where $\pi_{-3}(x)=(\pi_{1}(x),\pi_{2}(x),0)$ and $\pi_{i}$ defines a
projection function of the $i$th element of the input vector.

Definition 4.1 is a smoothed characterization of the Cylinder defined in
Appendix A.1. The function $g_{Cy}$ defines a smoothed cylinder centered in
$c$ by means of a Gaussian function, with the distance between $x$ and the
cylinder’s radius ($r$) as its mean. The shape parameters are the Gaussian’s
standard deviation and $r$, defined as $\vartheta_{Cy}=[r,\sigma]$.

GENEOs act on functions, transforming them to remain equivariant to a specific
group of transformations. Our GENEOs act upon $\Phi$, the topological space
representing $\mathcal{P}$ with admissible functions
$\varphi\colon\mathbb{R}^{3}\to\\{0,1\\}$. Specifically, we work with
appropriate $\varphi\in\Phi$ functions that represent point clouds and
preserve their geometry. For instance, $\varphi$ can be a function that
signals the presence of 3D points in a voxel grid. Therefore, the cylinder
GENEO $\Gamma_{Cy}^{\vartheta}$ transforms $\varphi$ into a new function that
detects sections in the input point cloud that demonstrate the properties of
$g_{Cy}$ and, simultaneously, preserves the geometry of the 3D scene

$\displaystyle\Gamma_{Cy}^{\vartheta}\colon\Phi\to\Psi,\qquad\psi_{Cy}=\Gamma_{Cy}^{\vartheta}(\varphi)$
$\displaystyle\psi_{Cy}(x)=\int_{\mathbb{R}^{3}}\tilde{g}_{Cy}(y)\varphi(x-y)dy$

where $\Psi$ is a new topological space that represents $\mathcal{P}$ with
functions $\psi\colon\mathbb{R}^{3}\to[0,1]$ and $\tilde{g}_{Cy}$ defines a
normalized Cylinder. The kernel $g_{Cy}$ is normalized to have a zero-sum to
promote the stability of the observer. This way, we encourage the geometrical
properties that exhibit the sought-out group of transformations and punish
those which do not. Thus, $\psi_{Cy}(x)$ assumes positive values for 3D points
near the radius, whereas negative values discourage shapes that do not fall
under the $g_{Cy}$ definition. This leads to a more precise detection of the
encoded group of transformations. The cylinder kernel discretized in a voxel
grid can be seen in Fig. 3(a).

#### Arrow GENEO.

Towers are not the only element in rural environments characterized by a
vertical and narrow structure. The identification of trees also shows
equivariance w.r.t. rotations along the z-axis. Therefore, it is not enough to
detect the body of towers, we also require the power lines that they support.
To this end, we define a cylinder following the rationale behind the cylinder
GENEO with a cone on top of it. This arrow defines equivariance w.r.t. the
different angles at which power lines may find their supporting tower.

######  Definition 4.2.

The function describing the Arrow is defined as
$g_{Ar}\colon\mathbb{R}^{3}\to[0,1]$:

$\displaystyle\begin{split}&g_{Ar}(x)=\left\\{\begin{array}[]{ll}e^{\frac{-1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-r^{2})^{2}}&\mbox{if
}\pi_{3}(x)<h\\\
e^{\frac{-1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-(r_{c}\tan(\beta\pi))^{2})^{2}}&\mbox{if
}\pi_{3}(x)\geq h\end{array}\right.\end{split}$

with $\beta\in[0,0.5)$ defining the inclination of the cone.

Definition 4.2 is a smoothed characterization of the Arrow defined in Appendix
A.2. The radii of the cylinder and cone are defined by $r$ and $r_{c}$,
respectively, with $c$ as their center. Lastly, $h$ defines the height at
which the cone is placed on top of the cylinder. Thus, the shape parameters of
the Arrow are defined by the vector $\vartheta_{Ar}=[r,\sigma,h,r_{c},\beta].$
Lastly, we are also interested that this kernel sums to zero, so we define

$\displaystyle\Gamma_{Ar}^{\vartheta}\colon\Phi\to\Psi,\qquad\psi_{Ar}=\Gamma_{Ar}^{\vartheta}(\varphi)$
$\displaystyle\psi_{Ar}(x)=\int_{\mathbb{R}^{3}}\tilde{g}_{Ar}(y)\varphi(x-y)dy,$

where $\tilde{g}_{Ar}(y)$ represents a normalized Arrow kernel. Its
discretization is depicted in Fig. 3(b).

#### Negative Sphere GENEO.

Detecting power lines does not exclude the remaining objects in the scene
whose identification also demonstrates equivariance w.r.t. rotations along the
z-axis. Tree elements, such as bushes, are especially frequent in the TS40K
dataset. Thus, we designed a negative sphere to diminish their detection and
simultaneously punish the geometry of trees.

######  Definition 4.3.

The Negative Sphere $g_{NS}\colon\mathbb{R}^{3}\to[-\omega,1[$ is defined as

$\displaystyle\begin{split}&g_{NS}(x)=-\omega
e^{\frac{-1}{2\sigma^{2}}(\|x-c\|^{2}-r^{2})^{2}}.\end{split}$

with $\omega\in]0,1]$ defining a small negative weight that punishes the
spherical shape.

Definition 4.3 is a smoothed characterization of the Negative Sphere in
Appendix A.3. The shape parameters of this operator are
$\vartheta_{NS}=[r,\sigma,\omega]$. Since we wish to discourage spherical
patterns following the definition of $g_{NS}$, so we do not enforce that its
space sums to zero, obtaining

$\displaystyle\Gamma_{NS}^{\vartheta}\colon\Phi\to\Psi_{NS},\qquad\psi_{NS}=\Gamma_{NS}^{\vartheta}(\varphi)$
$\displaystyle\psi_{NS}(x)=\int_{\mathbb{R}^{3}}g_{NS}(y)\varphi(x-y)dy.$

where $\Psi_{NS}$ is a topological space containing functions
$\psi\colon\mathbb{R}^{3}\to[-\omega,1[$. Fig. 3(c) depicts the computation of
this kernel in a voxel grid.

(a) Cylinder

(b) Arrow

(c) Negative Sphere

Figure 3: GENEO kernels discretized in a voxel grid and colored according to
weight distribution.

#### GENEO Loss.

The use of GENEOs in knowledge embedding forces our model to uphold the
convexity of the observer during training. Thus, our problem statement is
represented by the following optimization problem

$\displaystyle\begin{split}\operatorname*{\textrm{minimize}}_{\lambda,\vartheta}\quad&\mathop{\mathbb{E}}_{X,y,\alpha,\epsilon}\bigg{\\{}\mathcal{L}_{seg}(\lambda,\vartheta)\bigg{\\}}\\\
\textrm{s.t.}\quad&\vartheta\geq 0\\\ \quad&\lambda^{T}\textbf{1}=1\\\
\quad&\lambda\geq 0,\end{split}$

where the segmentation loss $\mathcal{L}_{seg}$ is defined as

$\displaystyle\mathcal{L}_{seg}(\lambda,\vartheta)=f_{w}(\alpha,\epsilon,y)\Big{(}\underset{\lambda,\vartheta}{\mathcal{M}}(X)-y\Big{)}^{2}.$

The loss uses a weighted squared error following the weighting scheme $f_{w}$
proposed in [45] to mitigate data imbalance. The hyperparameter $\alpha$
emphasizes the weighting scheme, whereas $\epsilon$ is a small positive number
that ensures positive weights. Thus, $\mathop{\mathbb{E}}\\{\cdot\\}$
represents the expectation of the segmentation loss over the data
distribution. The above constraints ensure that our model $\mathcal{M}$
maintains convexity throughout training, with 1 denoting a vector composed of
entries one. The reparametrization of the hyperparameters $\lambda$ to obtain
an equivalent optimization problem, considering
$\lambda_{k}=1-\sum_{i=1}^{K-1}\lambda_{i}$, thus obtaining Problem (4),

$\displaystyle\begin{split}\operatorname*{\textrm{minimize}}_{\lambda,\vartheta}\quad&\mathop{\mathbb{E}}_{X,y,\alpha,\epsilon}\bigg{\\{}{\mathcal{L}_{seg}}(\lambda,\vartheta)\bigg{\\}}\\\
\textrm{s.t.}\quad&\vartheta\geq 0\\\ \quad&\lambda\geq 0\end{split},$ (4)

allows for dropping one of the constraints. Then, we ensure non-negativity of
$\mathcal{M}$’s trainable parameters $\lambda,\vartheta$ by relaxing Problem
(4) and introducing a penalty in the optimization cost definition as

$\displaystyle\begin{aligned}
\operatorname*{\textrm{minimize}}_{\lambda,\vartheta}&\quad\mathop{\mathbb{E}}_{X,y,\alpha,\epsilon}\bigg{\\{}{\mathcal{L}_{seg}}(\lambda,\vartheta)\bigg{\\}}\;+\rho_{l}\Big{(}\sum_{i}^{K}h(\lambda_{i})\Big{)}+\rho_{t}\Big{(}\sum_{i}^{K}\sum_{j}^{T_{i}}h(\vartheta_{ij})\Big{)},\\\
\end{aligned}$ (5)

where $h(x)=\big{(}-x\big{)}_{+}$, $\rho_{l}$ and $\rho_{t}$ are scaling
factors of the negativity penalty $h$ and $T_{i}$ is the number of shape
parameters in $\vartheta_{i}$. GENEO final loss optimization is formalized in
Problem (5). It consists of a data fidelity component (i.e.,
$\mathcal{L}_{seg}$) and two penalties on negative parameters.

##  5 Experiments

In this Section, we assess properties of our model SCENE-Net that help
electrical companies in the inspection of power lines: (1) interpretability of
the model, (2) accuracy, (3) robustness to noisy labels, (4) training and
inference time, and (5) performance on the SemanticKITTI benchmark. Further
details about the TS40K dataset, the training protocol, inference performance
with high-resolution voxel grids, and ablation studies can be found in the
Supplementary Material.

#### Interpretability of the trained SCENE-Net: The meaning of the 11 learned
parameters.

To understand if the model parameters are interpretable, we inspect SCENE-
Net’s 11 trainable parameters $\vartheta$ and $\lambda$ after training. Each
$\vartheta_{i}\in\vartheta$ holds the learned shape parameters of a
geometrical operator $\Gamma_{i}$, such as their height or radius. The convex
coefficients $\lambda$ weigh each operator $\Gamma_{i}$ in our model’s
analysis. For example, we can conclude that the instance $\vartheta_{NS}$ of
the Negative Sphere GENEO ($\Gamma_{NS}$) holds a weight of 76.34% on SCENE-
Net’s output (Fig. 4). The geometric nature of the observer and combination
parameters endow intrinsic interpretability to SCENE-Net.

Figure 4: The trainable parameters of SCENE-Net, $\vartheta$ and $\lambda$.
Parameter $h^{*}$ is not trainable, and $\lambda_{NS}^{*}$ is defined as a
function of the other mixing weights
$\lambda_{NS}^{*}=1-\lambda_{Ar}-\lambda_{Cy}$.

#### Post-hoc interpretation for specific predictions.

We can correlate the detection of scene elements, such as vegetation, to the
contributions of each GENEO. This provides an extra layer of transparency to
our model. The Arrow kernel is responsible for the detection of towers, the
Cylinder aids this process and diminishes the detection of vegetation, and the
Negative Sphere stabilizes the model by balancing contributions of the
previous kernels (Fig 5).

(a) TS40K scene

(b) Cylinder

(c) Arrow

(d) Negative Sphere

Figure 5: Post hoc analysis of SCENE-Net. We can examine the activation of
each geometric operator and correlate it to the detection of certain elements
in the scene. We see that the Arrow is responsible for the most activation,
while the Negative Sphere has a smaller absolute value.

#### Qualitative accuracy and quantitative metrics: SCENE-Net is more precise
in detecting towers than a baseline CNN.

To evaluate if SCENE-Net can correctly identify towers in landscapes of the
noisy TS40K dataset, we chose the task of 3D semantic segmentation of power
line towers. We trained SCENE-Net and a baseline CNN according to the protocol
described in the Supplementary Material. We use a CNN with similar
architecture and the same base operator (i.e., convolution) for feature
retrieval. The main difference is their kernel initialization: SCENE-Net
kernels are randomly initialized, but belong to a precise family of operators,
while CNN kernels are completely random. Running models for 3D point cloud
semantic segmentation [16, 37, 18, 20] was not done due to their computational
requirements. The application penalizes the false positives more, thus we will
emphasize Precision. Due to the imbalanced nature of the labels, we measured
overall Precision, Recall, and Intersection over Union (IoU). Quantitatively,
we observe a lift in Precision of 38%, and of 5% in IoU, and a drop of 13% in
Recall (Table 1).

Table 1: 3D semantic segmentation metrics on TS40K. Method | Precision | Recall | IoU
---|---|---|---
CNN | 0.44 ($\pm$ 0.07) |  0.26 ($\pm$ 0.02) | 0.53
SCENE-Net |  0.82 ($\pm$ 0.08) | 0.13 ($\pm$ 0.05) | 0.58
Figure 6: Precision-Recall curve for SCENE-Net and the CNN benchmark, with
changing detection threshold. Although our model SCENE-Net has two orders of
magnitude fewer parameters than the CNN, it attains a comparable area under
the P-R curve.

The lower Recall of SCENE-Net is due to mislabeled points (Figs. 1 and 7), and
our choice to privilege Precision over Recall, in view of the fact that the
Precision – Recall curve is slightly better for SCENE-Net (Fig 6).

#### SCENE-Net is robust to noisy labels.

It is important to assess the resilience to noisy labels in the Ground-Truth
(GT) since 3D point clouds show more than 50% of mislabeled points. These
examples are abundant in the dataset and SCENE-Net is able to recover the body
of the tower without detecting ground and power line patches that are
mislabeled as tower (Fig. 7). Most noisy labels on this kind of dataset are
due to annotation excess around the object of interest and are not randomly
distributed. These consistently incorrect labels entail low Recall values
(Tab. 1).

(a) TS40K scene

(b) SCENE-Net prediction against the GT

Figure 7: SCENE-Net is robust against mislabeled data. Fig. 7(b) compares the
prediction of SCENE-Net against the ground truth in Fig. 7(a). SCENE-Net
detects the body of the tower, ignoring the patch of ground mislabeled as a
tower.

#### SCENE-Net has low data requirements and has modest training time in
common hardware.

The design of SCENE-Net embedded with GENEO observers culminates in a model
with 11 meaningful trainable parameters. This enables the use of common
hardware (see Appendix D.2 for hardware specs) and a low data regime to train
our model. The results reported in table 2 were achieved with 5% of the
SemanticKITTI training set. Training SCENE-Net with 50% and 100% of the
available training data leads to a variation of 0.5% in pole IoU performance.
Moreover, the number of parameters of SCENE-Net remains unchanged regardless
of kernel size, whereas in traditional models, such as the baseline CNN with
2190 parameters, the number of parameters grows exponentially with larger
kernel sizes.

#### SCENE-Net on the SemanticKITTI: an efficient model for low-resource
contexts.

In Table 2, we present a comprehensive comparison of the performance of SCENE-
Net against state-of-the-art models for the task of 3D semantic segmentation
on the SemanticKITTI benchmark, specifically in terms of pole IoU, number of
parameters, and the ratio of pole IoU to number of parameters. For this
problem, we add to the GENEO loss in (5) the Tversky loss [46] to boost IoU
performance of SCENE-Net:

$\displaystyle\mathcal{L}_{Tversky}(y,\hat{y})=1-\frac{y\hat{y}+\delta}{y\hat{y}+\alpha(\textbf{1}-y)\hat{y}+\beta
y(\textbf{1}-\hat{y})+\delta}$

where $y$, $\hat{y}$ are the ground truth and model prediction,
$\alpha,\beta>0$ are the penalty factors for false positives and false
negatives respectively, and $\delta>0$ is a smoothing term.

The comparison results demonstrate that SCENE-Net is a highly efficient model
in terms of its parameter contribution. Our model has the lowest number of
parameters, with at least a 5-order magnitude difference from the other
models. SCENE-Net also has the highest ratio of pole IoU to a number of
parameters, indicating that it can achieve a high level of performance with a
minimal number of parameters. Although SCENE-Net does not achieve the highest
pole IoU performance, it is somewhat on par with state-of-the-art models.
Additionally, SCENE-Net provides intrinsic geometric interpretability and
resource efficiency, which makes it a valuable model in high-risk tasks that
require trustworthy predictions and good performance but have limited data and
computing power.

Table 2: Semantic segmentation on SemanticKITTI. Large models are included for comparison but cannot be used in low-resource contexts. Parameter efficiency is $\frac{\textrm{Pole IoU}}{\textrm{log \\#Parameters}}$. Method |  | Pole
---
IoU
| #Parameters
---
(M)
| Parameter
---
Efficiency
PointNet++ [29] | 16.9 | 1.48 | 1.19
TangentConv [47] | 35.8 | 0.4 | 2.77
KPConv [16] | 56.4 | 14.9 | 3.41
RandLA-Net [30] | 51.0 | 1.24 | 3.63
RPVNet [18] | 64.8 | 24.8 | 3.80
SparseConv [24] | 57.9 | 2.7 | 3.91
JS3C-Net [19] | 60.7 | 2.7 | 4.09
SPVNAS [17] | 64.3 | 12.5 | 4.62
SCENE-Net (Ours) | 57.5 | 1.1e-5 | 23.98
Figure 8: Semantic Segmentation results on the SemanticKITTI benchmark. Log
scale is used for more intelligible comparison.

(a) SemanticKITTI scene

(b) SCENE-Net prediction against the GT

Figure 9: Qualitative results of SCENE-Net on SemanticKITTI for pole
detection. Fig. 9(b) compares the prediction of SCENE-Net against the ground
truth in Fig. 9(a). SCENE-Net detects the body of the pole while disregarding
the rest of the 3D scene.

##  6 Discussion

Traditional companies, like utilities, need a resource-efficient, responsible
application of ML models for the segmentation of real-world point clouds,
e.g., for inspecting thousands of kilometers of a power grid. In this paper,
we present SCENE-Net, a low-resource white-box model for 3D semantic
segmentation. Our approach offers a unique combination of intrinsic geometric
interpretability, resource efficiency, and on-par state-of-the-art
performance.

#### Limitations

Our model prioritizes transparency and performance over broader applicability
while allowing a flexible extension. In this paper, we segmented pole-like
structures. State-of-the-art methods often show similar trade-offs, for
example, 3D semantic segmentation models are tailored for autonomous driving
[19, 18]. SCENE-Net requires a knowledge engineering phase that is not
necessary for black-box models. Despite these limitations, we believe that the
transparency and efficiency of SCENE-Net make it a valuable tool for high-
stakes applications. To detect other shapes, other geometrical observers have
to be created. As the convex combination of GENEOs is a GENEO, this problem is
mitigated by creating a library of primary shapes to be combined to form more
complex geometrical structures. This is relevant follow-up work, but out of
the scope of this paper. Multiclass and multilabel segmentation can be
achieved by combining different binary class segmentation models.

#### Impact

From our experience deploying SCENE-Net within a utility company, low-resource
transparent systems can critically help human decision-making—here, by
facilitating fast and careful inspection of power lines with interpretable
signals of observed geometrical properties. With only three observers and 11
meaningful trainable parameters, SCENE-Net can help reduce the risk of power
outages and forest fires by learning from data.

## References

  * [1] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3):31–57, 2018.
  * [2] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  ACM computing surveys (CSUR), 51(5):1–42, 2018.
  * [3] Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [4] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215, 2019.
  * [5] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782–791, June 2021.
  * [6] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  arXiv preprint arXiv:1905.09418, 2019.
  * [7] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [8] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In Proceedings of the IEEE international conference on computer vision, pages 3429–3437, 2017.
  * [9] Zhi Chen, Yijie Bei, and Cynthia Rudin.  Concept whitening for interpretable image recognition.  Nature Machine Intelligence, 2(12):772–782, 2020.
  * [10] Mattia G Bergomi, Patrizio Frosini, Daniela Giorgi, and Nicola Quercioli.  Towards a topological–geometrical theory of group equivariant non-expansive operators for data analysis and machine learning.  Nature Machine Intelligence, 1(9):423–433, 2019.
  * [11] Pasquale Cascarano, Patrizio Frosini, Nicola Quercioli, and Amir Saki.  On the geometric and riemannian structure of the spaces of group equivariant non-expansive operators.  arXiv preprint arXiv:2103.02543, 2021.
  * [12] Timo Hackel, Nikolay Savinov, Lubor Ladicky, Jan D Wegner, Konrad Schindler, and Marc Pollefeys.  Semantic3d. net: A new large-scale point cloud classification benchmark.  arXiv preprint arXiv:1704.03847, 2017.
  * [13] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall.  Semantickitti: A dataset for semantic scene understanding of lidar sequences.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297–9307, 2019.
  * [14] Khan Muhammad, Amin Ullah, Jaime Lloret, Javier Del Ser, and Victor Hugo C de Albuquerque.  Deep learning for safe autonomous driving: Current challenges and future directions.  IEEE Transactions on Intelligent Transportation Systems, 22(7):4316–4336, 2020.
  * [15] Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, José Santamaría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan.  Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions.  Journal of big Data, 8(1):1–74, 2021.
  * [16] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019.
  * [17] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han.  Searching efficient 3d architectures with sparse point-voxel convolution.  In European conference on computer vision, pages 685–702. Springer, 2020.
  * [18] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu.  Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16024–16033, 2021.
  * [19] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui.  Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3101–3109, 2021.
  * [20] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shenghui Cui, and Zhen Li.  2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds.  arXiv e-prints, page arXiv:2207.04397, July 2022.
  * [21] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3431–3440, 2015.
  * [22] Dario Rethage, Johanna Wald, Jurgen Sturm, Nassir Navab, and Federico Tombari.  Fully-convolutional point networks for large-scale point clouds.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 596–611, 2018.
  * [23] Lyne Tchapmi, Christopher Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese.  Segcloud: Semantic segmentation of 3d point clouds.  In 2017 International Conference on 3D vision (3DV), pages 537–547. IEEE, 2017.
  * [24] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 9224–9232, 2018.
  * [25] Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz.  Splatnet: Sparse lattice networks for point cloud processing.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2530–2539, 2018.
  * [26] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong.  O-CNN: Octree-based convolutional neural networks for 3d shape analysis.  ACM Transactions On Graphics (TOG), 36(4):1–11, 2017.
  * [27] Truc Le and Ye Duan.  Pointgrid: A deep network for 3d shape understanding.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 9204–9214, 2018.
  * [28] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
  * [29] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [30] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11108–11117, 2020.
  * [31] Mingye Xu, Zhipeng Zhou, and Yu Qiao.  Geometry sharing network for 3d point cloud classification and segmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12500–12507, 2020.
  * [32] Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung.  Pointwise convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 984–993, 2018.
  * [33] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  PointCNN: Convolution on x-transformed points.  Advances in Neural Information Processing Systems, 31, 2018.
  * [34] Wenxuan Wu, Zhongang Qi, and Li Fuxin.  Pointconv: Deep convolutional networks on 3d point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2019.
  * [35] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin.  Cylindrical and asymmetrical 3d convolution networks for lidar segmentation.  In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 9939–9948, 2021.
  * [36] Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham.  Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4977–4987, 2021.
  * [37] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu.  2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network.  In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 12547–12556, 2021.
  * [38] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun.  Deep learning for 3d point clouds: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(12):4338–4364, 2020.
  * [39] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Anchors: High-precision model-agnostic explanations.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [40] Manuel Ribeiro and Joao Leite.  Aligning artificial neural networks and ontologies towards explainable ai.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4932–4940, 2021.
  * [41] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu.  Interpretable convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [42] Giovanni Bocchi, Patrizio Frosini, Alessandra Micheletti, Alessandro Pedretti, Carmen Gratteri, Filippo Lunghini, Andrea Rosario Beccari, and Carmine Talarico.  Geneonet: A new machine learning paradigm based on group equivariant non-expansive operators. an application to protein pocket detection.  arXiv preprint arXiv:2202.00451, 2022.
  * [43] Francesco Conti, Patrizio Frosini, and Nicola Quercioli.  On the construction of group equivariant non-expansive operators via permutants and symmetric functions.  Frontiers in Artificial Intelligence, 5:16, 2022.
  * [44] Stefano Botteghi, Martina Brasini, Patrizio Frosini, and Nicola Quercioli.  On the finite representation of group equivariant operators via permutant measures.  arXiv preprint arXiv:2008.06340, 2020.
  * [45] Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho.  Density-based weighting for imbalanced regression.  Machine Learning, 110(8):2187–2211, 2021.
  * [46] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour.  Tversky loss function for image segmentation using 3d fully convolutional deep networks.  In Machine Learning in Medical Imaging: 8th International Workshop, MLMI 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 10, 2017, Proceedings 8, pages 379–387. Springer, 2017.
  * [47] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou.  Tangent convolutions for dense prediction in 3d.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3887–3896, 2018.
  * [48] Leiqing Ding, Jianjun Wang, and Yikai Wu.  Electric power line patrol operation based on vision and laser slam fusion perception.  In 2021 IEEE 4th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), pages 125–129. IEEE, 2021.
  * [49] Tao Guo, Lianggang Xu, Yuran Chen, Yong Liu, Jian Zhao, Xiao Luo, and Shaohua Wu.  Research on point cloud power line segmentation and fitting algorithm.  In 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), volume 1, pages 2404–2409. IEEE, 2019\.
  * [50] Guo Tao, Xu Lianggang, Yang Heng, Yang Liugui, Zhao Jian, Wu Shaohua, and Wang Di.  Study on segmentation algorithm with missing point cloud in power line.  In 2019 IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pages 1895–1899. IEEE, 2019.

##  Appendix A Signature Shape Definitions

Although functions $g$ are straightforward to understand and deduce, for
completeness we present the exact signature shape definitions from which
functions $g$ were derived.

###  A.1 Cylinder Definition

######  Definition A.1.

The most striking characteristic of supporting towers against the rural
environment is their long, vertical and narrow structure. As such, their
detection is equivariant w.r.t. both rotations along the z-axis and
translations in the xy plane, which we encode by means of a cylinder:

$\displaystyle f_{Cy}\colon\mathbb{R}^{3}\to\\{0,1\\}$ $\displaystyle
f_{Cy}(x)=\left\\{\begin{array}[]{ll}1&\mbox{if
}\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}=r^{2}\\\ 0&\mbox{if
}\text{otherwise}\end{array}\right.,$

where $\pi_{i}$ defines a projection function of the $i$th element of the
input vector. However, data patterns show smoothing, so we relax this
condition by defining $g_{Cy}\colon\mathbb{R}^{3}\to[0,1]$:

$\displaystyle\begin{split}&g_{Cy}(x)=e^{-\frac{1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-r^{2})^{2}}\end{split}.$
(6)

The function $g_{Cy}$ defines a cylinder centered in $c$ by means of a
Gaussian function, with the distance between $x$ and the cylinder’s radius
($r$) as its mean. The shape parameters of the Cylinder are the standard
deviation of the Gaussian and radius $r$ and are defined by
$\vartheta_{Cy}=[r,\sigma]$.

###  A.2 Arrow Definition

######  Definition A.2.

Towers are not the only element in rural environments characterized by a
vertical and narrow structure. For example, the detection of trees is also
equivariant w.r.t. rotations along the z-axis. Therefore, it is not enough to
detect the body of towers, we also require the power lines that they support.
To this end, we define a cylinder following the rationale behind the Cylinder
GENEO with a cone on top of it. This arrow defines equivariance w.r.t. the
different angles that power lines may have with a supporting tower. Formally,
this function can be defined as $f_{Ar}\colon\mathbb{R}^{3}\to\\{0,1\\}$

$\displaystyle\begin{split}&f_{Ar}(x)=\left\\{\begin{array}[]{lll}1&\mbox{if
}\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}=r^{2}\\\ &\wedge\pi_{3}(x)<h\\\ 1&\mbox{if
}\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}=(r_{c}\tan(\beta\pi))^{2}\\\
&\wedge\pi_{3}(x)\geq h\\\ 0&\mbox{if
}\text{otherwise}\end{array}\right.,\end{split}$

with $\beta\in[0,0.5)$ defining the cone’s inclination. The cylinder and cone
radii are defined by $r$ and $r_{c}$, respectively, with $c$ as their center.
Lastly, $h$ defines the height at which the arrow is placed on top of the
cylinder. This definition is too strict to yield any feasible results in real-
world scenarios, so we smooth the conditions as follows

$\displaystyle\begin{split}&g_{Ar}\colon\mathbb{R}^{3}\to[0,1]\\\
&g_{Ar}(x)=\left\\{\begin{array}[]{ll}e^{\frac{-1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-r^{2})^{2}}&\mbox{if
}\pi_{3}(x)<h\\\
e^{\frac{-1}{2\sigma^{2}}(\|\pi_{-3}(x)-\pi_{-3}(c)\|^{2}-(r_{c}\tan(\beta\pi))^{2})^{2}}&\mbox{if
}\pi_{3}(x)\geq h\end{array}\right.\end{split}.$ (7)

Thus, the shape parameters of the Arrow are defined by the vector
$\vartheta_{Ar}=[r,\sigma,h,r_{c},\beta].$

###  A.3 Negative Sphere Definition

######  Definition A.3.

Detecting power lines does not exclude the remaining objects in the scene
whose detection also demonstrates equivariance w.r.t. rotations along the
z-axis. Arboreal elements are especially frequent in the TS40K dataset. Thus,
we designed a negative sphere to diminish their detection and, simultaneously,
punish the geometry of trees $f_{NS}\colon\mathbb{R}^{3}\to\\{-\omega,1\\}$ as

$\displaystyle\begin{split}&f_{NS}(x)=\left\\{\begin{array}[]{ll}1&\mbox{if
}\|x-c\|^{2}=r^{2}\\\ -\omega&\mbox{if
}\text{otherwise}\end{array}\right.,\end{split}$

with $\omega\in]0,1]$ defining a small negative weight that punishes the
spherical shape. Next, we proceed with the relaxation of the sphere definition
with function $g_{NS}$ as

$\displaystyle\begin{split}&g_{NS}\colon\mathbb{R}^{3}\to[-\omega,1[\\\
&g_{NS}(x)=e^{\frac{-1}{2\sigma^{2}}(\|x-c\|^{2}-r^{2})^{2}}-\omega.\end{split}$
(8)

The shape parameters of this operator are defined as
$\vartheta_{NS}=[r,\sigma,\omega]$.

##  Appendix B TS40K Dataset

Electrical companies are responsible for the maintenance and inspection of the
transmission system. They deploy low-flying helicopters to scan rural
environments, from a BEV perspective, where the electrical grid is located.
The produced point clouds exhibit different data properties when compared to
3D scenes captured from other viewpoints, such as from a vehicle. Namely, they
show high point density and no object occlusion, scene elements present
homogeneous density and no sparsity. Then, the acquired 3D data is processed
by maintenance personnel. Specifically, as the raw point clouds are dense and
mainly encompass rural areas, data is sectioned into strips of land focused on
the transmission system (Figure 10).

Figure 10: Visualization of TS40K raw point cloud with colored labels.

The raw data comprises several LiDAR files containing roughly 40 000
kilometers of the above land strips. 3D points therein are labeled with one
out of 22 possible classes, such as power lines and their supporting towers,
low and medium vegetation, rivers, railroads, human-made structures that do
not belong to the transmission network, the ground, and optic cables, among
others. Table 3 depicts these classes and their density in the dataset. Rail
lines and road surfaces constitute the majority of the dataset (63%), whereas
classes of interest, such as power line supporting towers, make up less than
5% of the overall data.

Table 3: Available classes in the TS40K dataset and their distribution. Rail
lines and road surfaces constitute the majority of the dataset (63%). Our
class of interest, power line towers, only makes up 0.52%. Moreover, around
40% of the tower points are mislabelled.

Label | Class | Density(%) | Label | Class | Density(%)
---|---|---|---|---|---
0 | Created | 0 | 11 | Road surface | 18.758
1 | Unclassified | 0.571 | 12 | Overlap points | 23.403
2 | Ground | 0.529 | 13 | Medium Reliability | 0
3 | Low vegetation | 0.681 | 14 | Low Reliability | 0
4 | Medium vegetation | 0.241 | 15 |  Power line support tower | 0.519
5 | Natural obstacle | 1.069 | 16 | Main power line | 0.907
6 | Human structures | 0 | 17 | Other power line | 0.002
7 | Low point | 0.362 | 18 | Fiber optic cable | 0
8 | Model key points | 0 | 19 | Not rated object to be consider | 8.205
9 | Water | 0 | 20 | Not rated object to be ignored | 0
10 | Rail | 44.752 | 21 | Incidents | 0

##  Appendix C Related Work

###  C.1 Power line segmentation from 3D point clouds.

Power line inspection is generally performed by on-site maintenance personnel
and manned helicopters that examine the power grid with portable devices or
the naked eye. These methods are costly, inefficient, and demanding for staff.
Thus, process automation is crucial for operators. To this end, unmanned
aerial vehicles (UAVs) carrying LiDAR sensors are deployed to scan the power
grid and capture a 3D point cloud representation of the environment. In the
work [48], the authors combine SLAM algorithms with multi-sensor data to
patrol the electrical grid with UAVs. This method employs a multi-view-based
approach to point cloud segmentation, so the 3D reconstructions from 2D raster
maps usually introduce information loss and decrease in accuracy. Alternative
methods project point clouds onto the $xy$-plane in order to cluster them [49]
to segment power lines. This approach does not consider ground and irregular
terrain and focuses on segmenting incomplete power lines. Other methods take
advantage of fine-grained elevation statistics of the original point cloud and
$xy$-plane projections [50]. The proposals above focus on the segmentation of
high-voltage power lines and disregard their supporting towers. By
incorporating prior knowledge, our proposal segments supporting towers of any
voltage. Not only are these structures also subject to inspections, but they
serve as a point of reference for the location of power lines. In addition, by
taking into account raw 3D scenes, other scene elements, such as vegetation,
can be segmented to assess the risk of contact between the power grid and the
environment.

##  Appendix D Experiments

###  D.1 Dataset.

We evaluate the effectiveness of SCENE-Net on the TS40K dataset. To mitigate
the severe data imbalance in our class of interest, we created an ancillary
dataset focused on power line-supporting towers with 2823 samples. For each
tower in the 3D data, we crop the ground around it with a radius equal to its
height. This process introduces bias on classical machine learning agents,
such as CNNs. Contrastingly, SCENE-Net is optimized, after incorporating the
appropriate prior knowledge, to detect the chosen features that describe the
GT. A biased training dataset results in an agent tailored to detect
supporting towers’ geometry. Elements in the 3D scene that do not align with
these properties are not signaled by SCENE-Net. Furthermore, we discretize the
cropped point clouds with a volume of $64^{3}$ voxels. If a voxel contains any
3D tower points, it is labeled as a tower voxel, otherwise, it is labeled as a
non-tower voxel. This emphasizes the geometry of supporting towers so that
they can be better described by SCENE-Net. To represent the 3D input, all non-
empty voxels are given a value of 1. This measurement function preserves the
structure of raw point clouds and mitigates the difference in point density
between supporting towers and other classes.

###  D.2 Training Protocol.

During the end-to-end training process of SCENE-Net, we adopt the following
settings: batch size is 8 for a total of 50 epochs. We employ the RMSProp
optimizer with a learning rate of 0.001. The weighting scheme parameters
$\alpha$ and $\epsilon$ are set to 5 and 0.1, respectively. While both scaling
factors of the non-positive penalty $\rho_{l}$ and $\rho_{t}$ are set to 5.
The kernel size used to discretize the GENEO operators is $9^{3}$. The GENEOs
parameters $\vartheta$ are randomly initialized under suitable and positive
ranges. While the convex coefficients $\lambda_{0},\dots,\lambda_{n-1}$ are
randomly initialized in the range $[0,\frac{2}{N-1}]$ to promote a valid
convex space for $\mathcal{H}$. To demonstrate that SCENE-Net achieves good
results even with fewer data, we use 20% of the dataset for training, 10% for
validation, and 70% for testing. All experiments were conducted on an NVIDIA
GeForce RTX 3070 GPU.

###  D.3 Ablation Studies.

Table 4: Ablation Study of SCENE-Net on TS40K validation set. Model | Cylinder | Arrow | Neg. Sphere | Precision | Recall | IoU
---|---|---|---|---|---|---
A | 1 | 0 | 0 | 0 | 0 | 0
B | 0 | 1 | 0 | 0 | 0 | 0
C | 1 | 0 | 1 | 0.34 | 0.01 | 0.12
D | 0 | 1 | 1 | 0.13 | 0.01 | 0.08
E (Ours) | 1 | 1 | 1 | 0.82 | 0.13 | 0.58
F | 2 | 2 | 2 | 0.56 | 0.16 | 0.53
G | 3 | 3 | 3 | 0.37 | 0.22 | 0.56

In this section, we conduct ablation studies on SCENE-Net’s architecture,
specifically on the number of instances of each GENEO. All ablated models were
tested on the TS40K validation set. Table 4 shows the following results:
Models A and B are each equipped with a single GENEO, demonstrating an overall
poor performance. The Negative Sphere (NS) GENEO proved essential for our
observer to disregard arboreal elements in the scene. Models C and D study if
employing the Cylinder or Arrow combined with NS is enough to analyze the
TS40K scenes. However, SCENE-Net’s architecture (model E) yields better
results. Lastly, models F and G test the use of multiple instances of each
GENEO, however, this proved to decrease performance when compared to model E.

###  D.4 Post-hoc interpretation of SCENE-Net analysis.

The geometric operators that embed SCENE-Net with prior knowledge not only
provide intrinsic interpretability through its parameters $\vartheta$ and
$\lambda$ but also enable a post-hoc interpretation of SCENE-Net’s
predictions. Specifically, we can analyze the observations of each GENEO
operator separately before their convex combination. This way, we can
correlate the detection or lack thereof of certain scene elements, such as
bushes and towers, to the contributions of each GENEO operator. This provides
an extra layer of transparency to our model and facilitates the knowledge
engineering phase of GENEO-powered models.

###  D.5 SCENE-Net inference in high resolution, when trained with low-
resolution kernel sizes.

One of the issues of voxel-based models is the computational cost of 3D
convolutions with large kernels and high-resolution voxel grids. Here, a CNN
architecture leads to an exponential increase in training time. SCENE-Net has
a continuous functional observer of the raw input providing an analysis of its
components. Unlike traditional models, this definition is independent from the
input size as well as its own discretization (kernel size). In this
experiment, we trained SCENE-Net with voxel grids of $64^{3}$ and then applied
to higher resolutions, such as $128^{3}$, with good qualitative results (Fig.
11). The kernel size used to discretize the operators can be fine-tuned to
enhance performance (Tab. 5).

(a) Sample discretized in a $128^{3}$ grid

(b) SCENE-Net prediction against the GT

Figure 11: SCENE-Net is independent of the input and kernel size. Our model
was trained with voxel grids of shape $64^{3}$ and kernel size $9^{3}$. Fig.
11(b) shows SCENE-Net’s prediction against the ground truth of the $128^{3}$
input grid in Fig. 11(a) using a kernel-size of $12\times 5\times 5$.  Table
5: Performance of SCENE-Net with different kernel sizes on TS40K. SCENE-Net is
trained with a kernel size of $9^{3}$, which is later fine-tuned to find the
sweet spot between Precision and Recall.
Kernel-size | Precision | Recall | IoU
---|---|---|---
$9\times 9\times 9$ | 0.37 ($\pm$ 0.02) |  0.22 ($\pm$ 0.01) | 0.58
$9\times 5\times 5$ |  0.68 ($\pm$ 0.08) | 0.16 ($\pm$ 0.05) | 0.58

###  D.6 Template Matching Comparison.

Template matching offers a direct approach to pattern detection by measuring
the relation between the input data and a specific pattern of interest. Since
we define the properties that constitute supporting towers, a comparison
between SCENE-Net and this method is reasonable.

Our model combines GENEOs through convex combination and all shape parameters
and convex coefficients are found through backpropagation, creating an
aggregated GENEO operator - a data observer for composite shape signatures
(e.g., cylinder+arrow+negative sphere). In geometrical methods, such as
template matching, this composition is not trivial. Employing template
matching on 3D data is slow, and the parameter initialization (e.g., a
Cylinder radius) is a crucial step. In addition, template matching is not
directly applicable to some patterns, e.g., the Neg. Sphere, as it tries to
diminish the activation of specific elements. We tested template matching of
the TS40K with a Cylinder (the same definition used in SCENE-Net) with a
radius equal to the average tower radius in the training set. It runs for 12
hours on the validation set of TS40K with an average precision (AP) of 0.001,
while SCENE-Net’s AP = 0.2.

###  D.7 Qualitative Results.

In Figure 12, we can analyze the qualitative results on the TS40K dataset (a)
of SCENE-Net (c) against a CNN is similar architecture as baseline (b). Even
though the CNN achieves a higher Recall on the majority of the samples, it
classifies most vertical scene elements as towers, which ultimately leads to a
poor segmentation of supporting towers. In contrast, SCENE-Net segments the
body of towers and rejects other vertical objects that do not exhibit the
prior knowledge encoded in the model.

() TS40K Sample

() SCENE-Net

() CNN Baseline

Figure 12: Qualitative results of SCENE-Net on the testing set of TS40K,
against a CNN with similar architecture. Note that, in the last example,
SCENE-Net clearly identifies a second unlabeled tower, whereas CNN identifies
both the second tower and vegetation as towers.