# LMSeg: A deep graph message-passing network for efficient and accurate
semantic segmentation of large-scale 3D landscape meshes

Zexian Huang<EMAIL_ADDRESS>Kourosh Khoshelham
<EMAIL_ADDRESS>Gunditj Mirring Traditional Owners Corporation
<EMAIL_ADDRESS>Martin Tomko<EMAIL_ADDRESS>
###### Abstract

Semantic segmentation of large-scale 3D landscape meshes is pivotal for
various geospatial applications, including spatial analysis, automatic mapping
and localization of target objects, and urban planning and development. This
requires an efficient and accurate 3D perception system to understand and
analyze real-world environments. However, traditional mesh segmentation
methods face challenges in accurately segmenting small objects and maintaining
computational efficiency due to the complexity and large size of 3D landscape
mesh datasets. This paper presents an end-to-end deep graph message-passing
network, LMSeg, designed to efficiently and accurately perform semantic
segmentation on large-scale 3D landscape meshes. The proposed approach takes
the barycentric dual graph of meshes as inputs and applies deep message-
passing neural networks to hierarchically capture the geometric and spatial
features from the barycentric graph structures and learn intricate semantic
information from textured meshes. The hierarchical and local pooling of the
barycentric graph, along with the effective geometry aggregation modules of
LMSeg, enable fast inference and accurate segmentation of small-sized and
irregular mesh objects in various complex landscapes. Extensive experiments on
two benchmark datasets (natural and urban landscapes) demonstrate that LMSeg
significantly outperforms existing learning-based segmentation methods in
terms of object segmentation accuracy and computational efficiency.
Furthermore, our method exhibits strong generalization capabilities across
diverse landscapes and demonstrates robust resilience against varying mesh
densities and landscape topologies.

###### keywords:

Semantic segmentation , Message-passing neural network , 3D landscape meshes ,
Barycentric dual graph , End-to-end architecture

[inst1]organization=The University of Melbourne,city=Parkville, postcode=3010,
state=Victoria, country=Australia

[inst2]organization=Gunditj Mirring Traditional Owners
Corporation,addressline=248 Condah Estate Road, city=Breakaway Creek,
postcode=3303, state=Victoria, country=Australia

##  1 Introduction

Recent advancements in data acquisition and terrain reconstruction methods
enabled efficient, automated modelling of large-scale landscape terrains.
Modelling of landscapes relies on the representation of the terrain and
potential spatial objects on the terrain as 2.5D rasters (regularly-spaced
cells with elevation), or as textured 3D meshes of irregular triangles.
Compared to 3D point cloud or 2.5D raster landscape representations, 3D mesh
modelling of landscapes _explicitly_ capture geometric information and
semantic attributes (e.g., texture/landcover, aspect, gradient) associated
with triangle surfaces encoding explicit face adjacency. Current 3D mesh
_segmentation_ methods are, however, still underdeveloped and underused
compared to raster segmentation, and are facing significant challenges with:
(1) the accuracy of segmentation, especially on small objects; (2) the
tractability and efficiency of computation (i.e, efficient model training and
inference); and with (3) evaluation on diverse, large-scale landscape mesh
datasets, as discussed in a recent survey (Adam et al., 2023). These
challenges explicitly point out the ongoing problems in existing landscape
segmentation models, suggesting limited accuracy in segmenting complex terrain
structures and high computational complexity that limits the model
applications to large-scale landscape mesh datasets (Kölle et al., 2021; Gao
et al., 2021). With the segmentation accuracy and efficiency issues, we pose
the following question: _How to improve the accuracy and efficiency of mesh
segmentation by leveraging the geometric (i.e., face normals and Euclidean
coordinates) and semantic (i.e., RGB or HSV color) information of large-scale
landscape meshes?_ We hypothesize that the barycentric dual graph
representation of landscape meshes can tightly capture the rich semantic and
informative topography of mesh surfaces, which enable end-to-end model
trainability with graph-based learning architectures (Gilmer et al., 2017) and
facilitate expressive feature learning for accurate and efficient landscape
segmentation.

We propose an accurate and efficient deep network architecture, Landscape Mesh
Segmentation Network (LMSeg), for semantic segmentation of large-scale high-
resolution 3D landscape mesh data. LMSeg takes as input the barycentric dual
of triangle meshes (i.e., the barycentric dual graph) constructed from
irregular 3D point clouds as Triangular Irregular Networks (TINs). We append
geometric features of the TIN (i.e., face normals and Euclidean point
coordinates) and/or overlaid semantic features (e.g., HSV or RGB colour) of
the triangle mesh as node features of the barycentric dual graph for model
learning, without requiring additional hand-crafted mesh features as in
previous methods (Rouhani et al., 2017; Gao et al., 2021; Weixiao et al.,
2023). The barycentric dual graph captures the topological information of the
primal mesh (i.e., explicit face adjacency) in the barycentric graph _edges_ ,
and carries the geometric and semantic information of mesh faces as attributes
of the barycentric graph _nodes_ (Milano et al., 2020; Yang et al., 2023).
This is to enable the encoding of information attached to areas (triangle
faces) to a single node, facilitating the integration with current standard
graph-based learning architectures and efficient end-to-end trainability
(learning) and inference on the resulting barycentric graph.

LMSeg’s segmentation accuracy is grounded in its ability to hierarchically and
adaptively learn local latent geometric embedding from the input barycentric
dual graph. We propose an effective _geometry aggregation_ \+ (GA+) module
that supports hierarchical learning of expressive local geometric latent
embedding via positional embedding and adaptive feature aggregation. Efficient
computation is achieved through random node sub-sampling and heuristic edge
similarity pooling to hierarchically sub-sample (reduce) barycentric dual
graphs of the landscape mesh.

We evaluate the proposed LMSeg segmentation model on a large-scale public
urban mesh dataset SUM (Gao et al., 2021). LMSeg model achieves a higher
accuracy, 73.0% mIoU and 79.8% mAcc. at fast inference time in comparison with
existing learning-based segmentation methods. We further test the segmentation
performance of LMSeg on a natural landscape mesh dataset generated from high-
resolution georeferenced lidar point clouds data covering the Budj Bim
National Park in Victoria, Australia (Budj Bim Wall (BBW) dataset). On the BBW
dataset, LMSeg exhibits strong performance with 59.5 % mIoU and 74.6% F1 score
for the segmentation of irregular stone wall structures under real-world,
natural landscape conditions (with variable vegetation occlusion). Further
ablation study of LMSeg suggests that the geometric (i.e., face normals) and
semantic features (e.g., HSV or RGB colour) of triangle meshes are crucial
input features, which variably impact the segmentation performance of learning
models at both datasets.

##  2 Background

###  2.1 Surface Representations and Segmentation

The segmentation of landscape surfaces (with or without 3D objects) has been
the subject of significant interest in the spatial sciences. Extensive studies
have conducted on semantic segmentation of real-world large-scale landscapes,
increasingly at fine resolutions (Blaschke, 2010). Three dominant methods
exist for the representation of such landscapes: (1) directly measured
irregularly (cloud) or regular (grid) sets of georeferenced points capturing
3D coordinates of point locations on a surface; (2) regular cell-based
georeferenced raster representations of aggregate (typically, mean) elevation
values in the area covered by a cell, or (3) mesh representations of
Triangular Irregular Networks (TIN) of terrains Peucker et al. (1976).
Typically, lidar-based unstructured point clouds are transformed, via
interpolation to raster representations, or by meshing to TINs. Meshed TINs
therefore typically include all (or a subset of) measured point cloud data,
but construct a digital model of a surface where the top/bottom orientation of
a face (normal) can be captured (see Section 3.1.

In raster-based landscape segmentation, deep learning methods usually leverage
convolutional neural networks (CNN) (He et al., 2016; Girshick et al., 2014)
to learn deep semantic features for input images and classify image pixels to
semantic labels. CNNs are commonly applied to raster-based surface
segmentation, due to the problem’s equivalence to image analysis, for which
CNNs were developed. A recent method from Wang et al. (2022) applied the
Vision Transformer (ViT) architecture (Dosovitskiy et al., 2020) for semantic
segmentation of remote sensing imagery and achieved state-of-art segmentation
accuracy. However, the 2D imagery segmentation methods can not effectively
solve the object occlusion problem in 2D imagery domain.

When applied to 3D point cloud segmentation of landscapes, point-based (Qi et
al., 2017a, b; Tailor et al., 2021; Zhao et al., 2021; Hu et al., 2020; Thomas
et al., 2019) and graph-based methods (Wang et al., 2019; Li et al., 2019;
Landrieu and Simonovsky, 2018) have demonstrated the importance of learning
local geometric point features from locally grouped neighbours (to construct
the graphs) for point clouds analysis (e.g., classification and segmentation
tasks) and graph-structure learning (e.g., node and graph classifications).
These 3D point cloud segmentation methods (Qi et al., 2017a, b; Landrieu and
Simonovsky, 2018; Hu et al., 2020; Thomas et al., 2019) rarely utilise
additional geometric and semantic point features (i.e., point normals and
texture information) for landscape segmentation due to the lack of diverse
point features provided in landscape datasets.

###  2.2 Mesh-based Surface Segmentation

In contrast to the landscape surface segmentation on 2D rasters and 3D point
clouds, triangular mesh segmentation models assign a semantic label from a set
of semantic classes directly to input 3D TIN mesh faces (possibly attributed
with auxiliary input features) (Gao et al., 2021; Adam et al., 2023). Most of
these models are most commonly applied 3D models of objects, rather than to
landscape surfaces.

The mesh-based model by Rouhani et al. (2017) partitions textured meshes into
superfacets by a region growing approach based on geometric similarity
measures of triangle facets, then computes a set of geometric and photometric
features for each superfacet. A random decision tree classifier is then
trained on hand-crafted features, and a Markov Random Field predicts
corresponding semantic labels for each superfacet. Following this work, SUM-RF
from Gao et al. (2021) proposed a semi-automatic annotation method for
labelling a large-scale urban landscape mesh dataset, which includes an
initial segmentation stage that over-segments input meshes into superfacets
and trains a random forest classifier based on hand-crafted features of
superfacets for automatic mesh labelling. While these approaches (Rouhani et
al., 2017; Gao et al., 2021) achieve stable segmentation performance on large-
scale landscape mesh datasets, they require additional hand-crafting mesh
features (i.e., geometric and radiometric features) based on over-segmented
mesh superfacets for subsequent learning tasks, which inhibits end-to-end
model trainability.

Hanocka et al. (2019) propose MeshCNN, an intrinsic mesh model that applies
deep convolution and pooling directly on mesh edges. Extensions of MeshCNN,
MeshNet (Feng et al., 2019) and MeshNet++ (Singh et al., 2021) take mesh faces
as inputs to spatial and structural descriptors and generate a set of initial
spatial and structural mesh features, on which a mesh convolution network is
then trained for classiﬁcation and retrieval tasks. A different approach by
Tang and Qiu (2021) converts 3D meshes into graph data and aggregates the
spatial and structural mesh features from the local neighborhoods of graph
nodes with a densely connected graph convolution network (Kipf and Welling,
2017) for object part segmentation. Guan et al. (2021) transforms 3D meshes
into barycentric point sets and construct dual local graphs, and apply a
local-global combined point-based network for object part segmentation. While
the methods discussed above attain end-to-end model trainability, they are
mostly tested on small-scale synthetic 3D mesh data (i.e., clean and hand-
crafted manifold meshes: cubes, spheres and torus, or models of toys)
(Bronstein et al., 2011; Wu et al., 2015; Hanocka et al., 2019) and are rarely
tested on large-scale non-manifold georeferenced landscape meshes (i.e., 2.5D
surfaces), possibly suffering from incompletness (gaps), and containing
defects and noise.

###  2.3 Learning Models for Urban Meshes

To evaluate mesh segmentation and scene understanding methods on real-world
landscape mesh data, a learning-based approach by Tutzauer et al. (2019)
computes the barycenter of landscape surface mesh triangles as point clouds. A
multi-branch 1D CNN network is trained on the geometric and radiometric
features of point clouds (i.e., the barycenter of landscape meshes) to predict
per-face labels. Later, Laupheimer et al. (2020) associated 3D lidar point
clouds with textured meshes and apply a PointNet++ (Qi et al., 2017b) for mesh
segmentation. The method by Tang et al. (2022) abstracts the triangular meshes
into the barycentric point clouds of meshes and applies a point-based
Transformer architecture (Zhao et al., 2021) for downstream mesh segmentation.
However, the point-based transformer learning architecture (Zhao et al., 2021)
in Tang et al. (2022) is comparatively computationally expensive and requires
over 40 hours of training time on a proprietary urban dataset for mesh
segmentation, which significantly limits the model applicability for larger
mesh datasets. The approach proposed by Yang et al. (2023) represents the
triangle meshes of landscape as surface graphs (i.e., barycentric dual graphs)
and then applies a hierarchical deep network for mesh segmentation. This
approach resembles a point-based segmentation architecture (Qi et al., 2017b)
and hierarchically constructs surface connective neighbourhoods (SC
neighborhood) of barycentric dual graphs for hierarchical feature learning on
input meshes. Additional TextureConv blocks based on 2D convolution networks
are required to learn facet texture features on the textures of triangle
meshes for semantic segmentation but limits the end-to-end trainability of
proposed model. PSSNet by Weixiao et al. (2023) proposed a two-stage deep
learning-based architecture for urban mesh segmentation, where the geometric
node and edge features are extracted by deep neural networks for superfacet
classification. The two-stage PSSNet learning architecture requires a super-
point grouping and edge dependency extraction process in the first stage for
the subsequent model learning in the second stage. This model design fails to
jointly optimise the two-stage model architecture and therefore is prohibitive
to end-to-end model trainability and causing inefficient model inference.

##  3 Methodology

In this section, we introduce the building blocks of LMSeg, a deep message-
passing network proposed for efficient and accurate large-scale landscape mesh
segmentation tasks. First, in Section 3.1 we provide the formal definition of
the barycentric dual graph input used by LMSeg, encoding the landscape surface
mesh representation for learning and inference. The overall network
architecture of LMSeg is then described in Section 3.2. Section 3.3 presents
the Geometry Aggregation+ (GA+) module providing an effective geometry
aggregation to LMSeg, which adaptively learns geometric latent embeddings on
the barycentric dual graph $\mathcal{G}(\mathcal{M})$. Section 3.4 then
discusses the efficient hierarchical node sub-sampling and local edge pooling
method of LMSeg.

Figure 1: Overall architecture of LMSeg. (a). Input mesh is converted into
barycentric dual graph with mesh texture and face normal features. (b). LMSeg
encoder consists of random node sub-sampling, HGA+, edge pooling and LGA+
modules for hierarchical and local feature learning. A residual MLP takes
concatenated LGA+ and HGA+ features as inputs and updates graph node features.
(c). LMSeg decoder consists of feature propagation layers, which progressively
up-sample the size of deep encoder features back to the original input size. N
denotes the number of input nodes of barycentric dual graph, and C refers to
the input node feature dimensions.

###  3.1 Triangular Mesh and its Barycentric Dual Graph

Consider a landscape surface $\mathcal{S}$ represented as a triangular mesh
$\mathcal{M}=(\mathcal{V},\mathcal{E},\mathcal{F})$, where
$\mathcal{V}\in\mathbb{R}^{N\times 3}$ is a set of unordered points captured
by their spatial coordinates on $\mathcal{S}$ (therefore
$\mathcal{V}\in\mathcal{S}$), $\mathcal{E}\in\mathbb{R}^{N\times N}$ is a
matrix of edge indices $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$,
which denotes a set of undirected edges that connect nodes in $\mathcal{V}$,
$\mathcal{F}=\\{(i,j,k)\in\mathcal{V}\\}$ is a set of ordered triplets
constituting the triangle faces of the mesh. Together, nodes $\mathcal{V}$,
edges $\mathcal{E}$ and faces $\mathcal{F}$ form a 3D mesh model capturing the
geometry of the surface ( terrain and potential 3D objects) in real-world
environments (Peucker et al., 1976; Adam et al., 2023).

The triangle mesh $\mathcal{M}$ is converted into its topological dual, the
barycentric graph $\mathcal{G}$. Concretely, an undirected barycentric graph
$\mathcal{G}(\mathcal{M})$ represents a triangular surface mesh as a node
$\mathcal{V}$ matrix of the spatial coordinates of barycenters of the
triangular mesh faces $\mathcal{F}\in\mathcal{M}$ and an edge $\mathcal{E}$
matrix capturing the face adjacency
$\mathcal{E}\subseteq\mathcal{F}_{\mathcal{M}}\times\mathcal{F}_{\mathcal{M}}$.
An additional node feature matrix $\mathcal{X}\in\mathbb{R}^{N\times C}$
encodes the corresponding face features (i.e., attributes) from $\mathcal{M}$
(e.g., face normals and texture information).

###  3.2 Network Architecture

We present the overall network architecture of LMSeg in Fig. 1, which shares a
encoder-decoder network architecture as Qi et al. (2017b).

#### Inputs:

the node features $\mathcal{X}$ (i.e., mesh textures and face normals) and
coordinates $\mathcal{V}$ of the barycentric graph $\mathcal{G}(\mathcal{M})$
are passed to a multilayer perceptron (MLP) layer to generate the latent node
embedding of input features. The latent node embedding along with the initial
edge matrix $\mathcal{E}$ of $\mathcal{G}(\mathcal{M})$ is passed to a GA+
module (Sec. 3.3) to learn local geometric features capturing the high-
frequency details of the 3D geometry of the surface.

#### Encoder:

the latent node embedding of GA+ module from inputs enter the encoder blocks,
as displayed in Fig.1, following the sequence: random node sub-sampling
$\rightarrow$ hierarchical geometry aggregation+ (HGA+) $\rightarrow$ edge
similarity pooling $\rightarrow$ local geometry aggregation+ (LGA+)
$\rightarrow$ residual MLP (ResMLP). The latent node embedding of HGA+ and
LGA+ modules in each encoder block are concatenated as dense features and
passed to the ResMLP layer to learn deep latent features with residual
connections (He et al., 2016; Ma et al., 2022). Overall, each encoder block
downsamples the vertex number of $\mathcal{G}(\mathcal{M})$ by 1/3
hierarchically with the random node sub-sampling and doubles the feature
dimensions of latent node embedding of $\mathcal{G}(\mathcal{M})$ with the
ResMLP layer.

#### Decoder:

a symmetric decoder design is implemented, following the pattern established
by point-based learning networks (Qi et al., 2017b; Qian et al., 2022). The
symmetric decoder progressively up-samples the size of deep encoder features
back to the original input size. In each decoder block, the feature
propagation layer interpolates the abstract features of the input encoder
feature at layer $l$ based on inverse distance weighted average and
$k$-nearest neighbors ($k$=3). The abstract features are added to the
homologous encoder features from layer $l-1$ and passed to an MLP layer.

###  3.3 Geometry Aggregation Module GA+

GA+ modules are the main feature learning components in LMSeg encoder, as
displayed in Fig.2, which is a graph message-passing network consists of
geometric feature learning of local graph neighbourhoods, incorporation of
positional embedding to local geometric features and the learnable feature
aggregation.

Figure 2: Different graph message-passing networks adopted in the geometry
aggregation+ (GA+) module and typical point-based / graph-based learning
approaches (Qi et al., 2017b; Wang et al., 2019).

#### Graph message-passing convolution:

The aim of the GA+ module learns a local geometric latent embedding via graph
message-passing convolution. Following the generic recipes of (Qi et al.,
2017b; Tailor et al., 2021; Zhao et al., 2021), the local geometric latent
embedding of $\mathcal{G}(\mathcal{M})$ is learned as:

$\displaystyle\mathbf{h}_{j}=\psi(\mathbf{v}_{i},\mathbf{x}_{i},\mathbf{x}_{rel}),$
(1)

where $\psi$ denotes a learnable function (i.e., shared MLPs),
$\mathbf{x}_{i}\in\mathcal{X}$ are the input mesh features (i.e., face normals
and texture information) of the target node $\mathbf{v}_{i}\in\mathcal{V}$,
and
$\mathbf{x}_{rel}=\frac{(\mathbf{x}_{j}-\mathbf{x}_{i})}{\sigma+\epsilon},\mathbf{x}_{j}\in\mathcal{X}_{\mathcal{K}_{i}}$
is the normalised relative node feature between the source and target node
features in an undirected local graph neighbourhood grouped by edges
$\mathcal{E}$ or a $k$-nearest neighbors ($k$-NN) function. In the normalised
relative node features, the $\sigma$ represents the standard deviation of
relative node features and $\epsilon$ is a small value (i.e., $1e^{-5}$) for
numerical stability (Ma et al., 2022).

#### Positional Embedding:

As pointed out by Tancik et al. (2020); Zhang et al. (2023), the positional
embedding using trigonometric functions in Transformer-based (Vaswani et al.,
2017) models for 3D point cloud learning is commonly applied for describing
the position of an entity in a sequence and enables MLPs capturing the high-
frequency latent embedding and fine-grained details of 3D structures in low
dimensional domains. As the barycentric graph $\mathcal{G}(\mathcal{M})$ of
triangle mesh encodes both the topology (by edges $\mathcal{E}$) and geometric
information (by nodes $\mathcal{V}$), the high-frequency spatial structures of
$\mathcal{G}(\mathcal{M})$ can be effectively captured by transforming low-
dimensional Euclidean inputs (i.e., $\mathbf{v}\in\mathcal{V}$) to high-
dimensional positional embeddings:

$\displaystyle PE(\mathbf{v}_{rel},2c)$
$\displaystyle=\mathbf{sin}(\alpha*\mathbf{v}_{j}/\beta^{(2c/d)}),$ (2)
$\displaystyle PE(\mathbf{v}_{rel},2c+1)$
$\displaystyle=\mathbf{cos}(\alpha*\mathbf{v}_{j}/\beta^{(2c/d)}),$

where $\mathbf{v}_{rel}=(\mathbf{v}_{j}-\mathbf{v}_{i})$ is the relative
spatial position of the source and target nodes, $\alpha$ (=100) controls the
granularity of the positions (i.e., the level of fine-grained changes in
vertex positions), $\beta$ (=1000) is the base frequency of sinusoidal
functions, $c$ refers to the input channels and $d$ is the output dimensions
of positional embedding vector $PE(\mathbf{v}_{j})$.

The local geometric latent embedding $\mathbf{h}_{j}$ via Eq.1 is then updated
using $PE(\mathbf{v}_{j})$ from Eq.2, giving:

$\displaystyle\mathbf{h}^{PE}_{j}=\phi(\mathbf{h}_{j}*PE(\mathbf{v}_{rel})+PE(\mathbf{v}_{rel})),$
(3)

where $\phi$ represents a learnable function (i.e., shared ResMLPs).

#### Learnable feature aggregation:

aggregation of latent embeddings (i.e., via sum, max and mean functions) are
commonly used to update latent node embeddings during graph message-passing in
the graph neighbourhood $\mathcal{X}_{\mathcal{K}_{i}}$:

$\displaystyle\mathbf{h}_{i}=aggr(\mathbf{h}^{PE}_{j}),\mathbf{h}_{i}\in\mathcal{H}^{l},j\in\mathcal{K}_{i},$
(4)

where $\mathcal{H}^{l}\in\mathbb{R}^{N\times D}$ refers to an aggregated node
feature matrix of $\mathcal{G}(\mathcal{M})$) at layer $l$.

As per Xu et al. (2018), sum aggregation is the most expressive in learning
graph structures, whereas mean aggregation performs well when the graph’s
statistical distribution needs to be preserved for downstream tasks.
Aggregation by max supports learning distinct node features that best
represent the neighbourhood, which is of importance in point clouds. Yet,
while simple aggregation functions exhibit desirable geometric invariant
properties (i.e., permutation invariance and equivariance), they can not
effectively aggregate geometric graph features in a generalized feature space,
as discussed in Li et al. (2023), causing suboptimal performances in
downstream tasks.

Following the definition in Li et al. (2023), we therefore propose a
differentiable generalised aggregation function for local geometric feature
aggregation. Specifically, we consider a learnable softmax function:

$\displaystyle aggr_{\mathbf{softmax}}(\mathbf{h}^{PE}_{j};\mathbf{t})$
$\displaystyle=\sum_{j\in\mathcal{K}_{i}}\mathbf{softmax}(\mathbf{h}^{PE}_{j};\mathbf{t})*\mathbf{h}^{PE}_{j},$
(5) $\displaystyle\mathbf{softmax}(\mathbf{h}^{PE}_{j};\mathbf{t})$
$\displaystyle=\dfrac{exp(\mathbf{t}*\mathbf{h}^{PE}_{j})}{\sum_{n\in\mathcal{K}_{i}}exp(\mathbf{t}*\mathbf{h}^{PE}_{n})},$

where $\mathbf{t}$ refers to a differentiable inverse temperature parameter
for scaling the randomness of features for softmax input. For
$\mathbf{t}\in(-\infty,+\infty)$, the function
$aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}\to+\infty)$ approximates
$aggr_{\mathbf{max}}$; the function
$aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}\to-\infty)$ approximates
$aggr_{\mathbf{min}}$; and finally
$aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}=0)$ approximates
$aggr_{\mathbf{mean}}$. In the differentiable setting, learning the inverse
temperature parameter $\mathbf{t}$ of the softmax function allows for
aggregating generalised features in a latent space between the aggregated max,
min and mean features.

Thus, the learnable generalised aggregation function enables adaptive learning
of generalised geometric features (i.e., the representative topology and
geometry features specific to the barycentric dual graph
$\mathcal{G}(\mathcal{M})$). Here, we consider two types of aggregation: a
fixed, non-differentiable simple max and mean aggregations to capture the
distinct node features, and the aggregated statistical properties and
distribution of node features; and a generalised, differentiable
$\mathbf{softmax}(\mathbf{h}^{PE}_{j};\mathbf{t}=1)$ to aggregate the weighted
sum of node features. We then add the two types of aggregation:

$\displaystyle\mathbf{h}^{aggr}_{i}=aggr_{\mathbf{max}}(\mathbf{h}^{PE}_{j})$
(6) $\displaystyle+\ aggr_{\mathbf{mean}}(\mathbf{h}^{PE}_{j})$
$\displaystyle+\ aggr_{\mathbf{softmax}}(\mathbf{h}^{PE}_{j};\mathbf{t}=1.0).$

###  3.4 Hierarchical & Local Pooling

Pooling operations are fundamental building blocks in hierarchical and local
information learning of 3D geometry (Zhang et al., 2023). Hierarchical pooling
of 3D data is typically implemented by the heuristic farthest Point Sampling
(FPS) (Qi et al., 2017b), which commonly applied in 3D point cloud learning
with the computational complexity $\mathcal{O}(N^{2})$ (Hu et al., 2021), or
quadric error metric (QEM) simplification (Garland and Heckbert, 1997), which
downsamples the 3D geometry of point clouds or mesh vertices while preserving
the geometric information. The computational bottleneck of 3D segmentation
networks indeed stems from these hierarchical pooling approaches, as confirmed
by (Hu et al., 2020).

To improve segmentation efficiency on large-scale landscape surface data, here
we apply the random node sub-sampling for hierarchical pooling of the
barycentric graph $\mathcal{G}(\mathcal{M})$. However, the random sub-sampling
of barycentric graph could randomly drop important nodes that preserve
meaningful information of graph (i.e., geometric features). Therefore, we
propose to utilise an edge similarity pooling approach to reconstruct and
maintain the local graph neighbourhoods of the hierarchically pooled graph of
$\mathcal{G}(\mathcal{M})$.

#### Random node sub-sampling:

In comparison to FPS, random sub-sampling uniformly subsets $\mathcal{S}$
points from $\mathcal{N}$ input points with a computational complexity
$\mathcal{O}(1)$. Here we randomly sub-sample $\mathcal{S}$ number of node
features from $\mathcal{G}(\mathcal{M})$ at the current layer, giving
$\mathcal{H}_{\mathcal{S}}^{l}=\\{h_{1},h_{2},...,h_{\mathcal{S}}\\}\in_{\mathcal{R}}\mathcal{H}_{\mathcal{N}}^{l-1}$.
We collapse the edges connecting removed nodes and retain only edges
connecting the sub-sampled nodes,
$\mathcal{E}^{l}_{sparse}\subseteq\mathcal{H}_{\mathcal{S}}^{l}\times\mathcal{H}_{\mathcal{S}}^{l}$.
Together the random node sub-sampling operation produces a sparse barycentric
graph
$\mathcal{G}^{l}(\mathcal{M})=(\mathcal{V}_{\mathcal{S}}^{l},\mathcal{E}^{l}_{sparse})$
with node features $\mathcal{H}_{\mathcal{S}}^{l}$.

#### Edge similarity pooling:

a major drawback of random hierarchical pooling is the loss of geodesic
relations (i.e., the face adjacency of original mesh $\mathcal{M}$) in the
sparse barycentric graph $\mathcal{G}^{l}(\mathcal{M})$. Yet, Schult et al.
(2020) have empirically demonstrated the benefits of preserving geodesic
features in mesh-based learning. They suggested the QEM mesh simplification
for the generation of a hierarchy of coarse meshes retaining geodesic
information after pooling operations. However, QEM is not directly applicable
to barycentric graph settings, and the QEM mesh simplification is not well
supported by GPU computations, and therefore inadequate for end-to-end
training.

Instead, we propose a heuristic edge similarity pooling method that locally
appends new edges to the downsampled barycentric graph containing sparse edge
connections. Since the original barycentric $\mathcal{G}^{0}(\mathcal{M})$ has
edges $\mathcal{E}^{0}$ modelling the face adjacency (i.e., 1-ring neighbours)
of input mesh $\mathcal{M}$, we safely assume for each node in the pooled,
sparse barycentric $\mathcal{G}^{l}(\mathcal{M})$ to have $k$ geodesic
neighbours, where $\\{k\in\mathbb{Z}:1\leq k\leq 3\\}$. We then compute $k=3$
nearest neighbours for each node based on the vertex coordinates
$v_{i}\in\mathcal{V}_{\mathcal{S}}^{l}$, giving a new edge matrix
$\mathcal{E}^{l}_{k=3}\subseteq\mathcal{H}_{\mathcal{S}}^{l}\times\mathcal{H}_{\mathcal{S}}^{l}$.
We then update the sparse edges with the newly computed dense edges:
$\mathcal{E}^{l}_{dense}=\mathcal{E}^{l}_{sparse}+\mathcal{E}^{l}_{k=3}$.
These dense edges might, however, encode redundant Euclidean information from
the $k$ nearest neighbour clustering. To better reflect the geodesic relations
of the original barycentric graph, we drop redundant edges from the dense
edges $\mathcal{E}^{l}_{dense}$ based on the cosine similarity of node
features connected by an edge $(h_{i},h_{j})\in\mathcal{E}^{l}_{dense}$:

$\displaystyle\mathcal{S}_{\mathcal{C}}(h_{i},h_{j})=\dfrac{h_{i}\cdot
h_{j}}{max(\lVert h_{i}\rVert_{2}\cdot\lVert h_{j}\rVert_{2})}.$ (7)

This results in a matrix capturing the similarity of node features in latent
space connected by edges in $\mathcal{E}^{l}_{dense}$. The redundant edges
connecting spatially nearby nodes with low similarity (i.e., a pre-defined
threshold value) are dropped, producing a locally geodesic edge matrix
$\mathcal{E}^{l}_{local}$.

Compared to the dynamic edge pooling approach in Wang et al. (2019), which
dynamically recomputes the graph neighbours of 3D point clouds using the
$k$-nearest neighbour clustering in the latent feature space, the proposed
heuristic edge similarity pooling computes local $k$-nearest neighbours in
Euclidean space, a more efficient computation with low memory consumption.

#### Hierarchical & local latent embedding:

We finally compute the $k$-nearest neighbours ($k$=20) for each
$v_{i}\in\mathcal{V}_{\mathcal{S}}^{l}$ from the original
$\mathcal{V}_{\mathcal{N}}^{l-1}$, which constitute a hierarchical Euclidean
neighbourhood encoded by edge $\mathcal{E}^{l}_{hier}$.

With $\mathcal{E}^{l}_{hier}$ and $\mathcal{E}^{l}_{local}$ as inputs to the
GA+ layer (Sec. 3.3), this results in the Hierarchical Geometry Aggregation
(HGA+) and Local Geometry Aggregation (HGA+) layers, respectively capturing
hierarchical Euclidean and locally geodesic features of the barycentric graph
$\mathcal{G}^{l}(\mathcal{M})$.

##  4 Experiments

We conduct experiments on two large-scale landscape surface mesh datasets
(Sec. 4.1): SUM (Gao et al., 2021) and the Budj Bim Wall (BBW) dataset, to
evaluate the performance of LMSeg on object semantic segmentation task. We
compare model performances of LMSeg with existing learning-based segmentation
models and report the quantitative and qualitative segmentation results in
Sec. 4.4.

###  4.1 Dataset

SUM (Gao et al., 2021) is a public benchmark dataset of semantic urban meshes
(Fig. 3), covering approx. 8$km^{2}$ total captured surface area (4$km^{2}$
map area, plus objects) in the central region of Helsinki, Finland, classified
into six semantic object classes, plus an unclassified complement class. SUM
includes 64 tiles: 40 annotated training tiles, 12 validation and 12 test
tiles, randomly selected. Each tile covers a 250$m^{2}$ map area with mesh
face density of $\sim$6.5 faces/m2. The textured mesh tiles in SUM are
generated from oblique aerial imagery by triangulation, dense image matching,
and mesh surface reconstruction.

Figure 3: Non-uniformly textured mesh, 3D point cloud and barycentric dual
graph of the SUM dataset, respectively. 3D point clouds are densely sampled
from the textured mesh in point density of 30 points per mesh surface area
($m^{2}$), as recorded in Gao et al. (2021).

BBW dataset is a lidar-scanned point-cloud dataset of the UNESCO World
Heritage cultural landscape (Smith et al., 2019; Bell et al., 2019), covered
by the Budj Bim National Park in southwest Victoria, Australia (Fig.4). This
is one of the areas with the highest density of European historic dry-stone
walls in Australia. The dataset was collected in 2020 by the Department of
Environment, Land, Water and Planning in Victoria, Australia for the Gunditj
Mirring Traditional Owners Corporation.

The BBW dataset is a subset of the full dataset, capturing the northern part
of the data. It is spatially divided into six equal, rectangular areas, as
shown in Fig.5, where areas 1, 3, 5, 6 are part of the training set (2633
tiles), area 4 is the validation set (716 tiles) and area 2 is the test set
(647 tiles). Each tile in BBW dataset is a textured landscape mesh of
400$m^{2}$ map area (with face density of $\sim$45 faces/m2) semi-manually
annotated into binary semantic labels (wall vs. other terrain). The textured
mesh tiles of BBW dataset are constructed from filtered ground surface points
resulting from ground/non-ground classification by the data producer. The mesh
2.5D surface is constructed by Delaunay triangulation of the ground surface 3D
points. The texture (color information) of the colorized lidar point clouds is
projected onto the mesh faces.

Figure 4: Near-uniform textured triangular mesh, 3D lidar point cloud (ALS)
and barycentric dual graph of BBW dataset, respectively. Figure 5: Data
splitting of Budj Bim Wall dataset. Train set: Area 1, 3, 5, 6. Validation
set: Area 4. Test set: Area 2. Colour annotations denote the spatial locations
of European historic dry-stone walls located near Tae Rak (Lake Condah),
Victoria, Australia.

###  4.2 Data Pre-processing

During pre-processing of the landscape mesh data, the primal meshes
$\mathcal{M}$ are converted to their barycentric dual graph
$\mathcal{G}(\mathcal{M})$. Data (geometric) augmentations are applied to the
training data: normalise scale normalise graph node positions to interval
$\left[-1,1\right]$; random rotation of each tile around $z$-axis by random
large angles $\in\left[-180^{\circ},180^{\circ}\right]$, around $x$\- and
$y$-axis by random small angles $\in\left[-1^{\circ},1^{\circ}\right]$; random
jitter translation of the graph node positions individually within a small
interval $\in\left[-0.001,0.001\right]$, post-normalisation. The colour
texture information of mesh faces (as HSV values) and face normals are used as
barycentric graph node features.

###  4.3 Implementation

We train LMSeg with an AdamW optimizer (Loshchilov and Hutter, 2017) of weight
decay 0.0001 with an initial learning rate of 0.005 for SUM dataset with batch
size of 1 and 0.01 for BBW with batch size of 4, respectively. Learning rates
were scheduled to decay by the cosine annealing scheduler. We apply a cross
entropy, resp. binary loss (for BBW) for supervised multi/binary-class
segmentation with the label smoothing technique to effectively prevent over-
confident predictions of deep networks. To mitigate the unbalanced semantic-
class distribution in datasets, we apply a normalised class weight vector
computed from the class distribution of datasets to cross, resp. binary
entropy loss. The mean Intersection-over-Union (mIoU) and mean accuracy
(mAcc.) metrics are used for model evaluation of multi-class segmentation on
SUM dataset. For model evaluation of binary-class segmentation on BBW dataset,
we apply the mIoU and F1 score (F1) metrics. All experiments and baselines are
implemented in PyTorch Geometric (Fey and Lenssen, 2019) on a Linux server
with a NVIDIA A40-24Q GPU.

Table 1: 3D semantic segmentation results on SUM dataset. Proportions of surface area (per semantic object class) of test set are displayed next to each category in (%). Model results are adopted from the public benchmark SUM (Gao et al., 2021) and approach TexturalSG (Yang et al., 2023). We compare the performance of LMSeg (average performances over five runs) with existing learning-based segmentation methods. Results reported are per-class IoU (%), mean IoU (%), overall accuracy (%) and mean accuracy (%). Model efficiency is evaluated by inference time (minute) on test dataset. Best results marked in bold. Modality | Method |  Terra. (23.6%) |  H-veg. (14.3%) |  Build. (50.7%) |  Water (4.8%) |  Vehic. (1.5%) |  Boat (2.1%) | mIoU | O.A. | mAcc. | Sampling |  Params. (M) |  $t$ inf. (Min)
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Point Clouds |  PointNet (Qi et al., 2017a) | 56.3 | 14.9 | 66.7 | 83.8 | 0.0 | 0.0 |  36.9 $\pm$ 2.3 |  71.4 $\pm$ 2.1 |  46.1 $\pm$ 2.6 | - | 3.6 | 1
RandLANet (Hu et al., 2020) | 38.9 | 59.6 | 81.5 | 27.7 | 22.0 | 2.1 |  38.6 $\pm$ 4.6 |  74.9 $\pm$ 3.2 |  53.3 $\pm$ 5.1 | Random | 1.3 | 52
SPG (Landrieu and Simonovsky, 2018) | 56.4 | 61.8 | 87.4 | 36.5 | 34.4 | 6.2 |  47.1 $\pm$ 2.4 |  79.0 $\pm$ 2.8 |  64.8 $\pm$ 1.2 | - | - | 26
PointNet++ (Qi et al., 2017b) | 68.0 | 73.1 | 84.2 | 69.9 | 0.5 | 1.7 |  49.5 $\pm$ 2.1 |  85.5 $\pm$ 0.9 |  57.8 $\pm$ 1.8 | FPS | 1.0 | 3
KPConv (Thomas et al., 2019) | 86.5 | 88.4 | 92.7 | 77.7 | 54.3 | 13.3 |  68.8 $\pm$ 5.7 |  93.3 $\pm$ 1.5 |  73.7 $\pm$ 5.4 | FPS | 15.0 | 42
Mesh |  RF-MRF (Rouhani et al., 2017) | 77.4 | 87.5 | 91.3 | 83.7 | 23.8 | 1.7 |  60.9 $\pm$ 0.0 |  91.2 $\pm$ 0.0 |  65.9 $\pm$ 0.0 | - | - | 15
SUM-RF (Gao et al., 2021) | 83.3 | 90.5 | 92.5 | 86.0 | 37.3 | 7.4 |  66.2 $\pm$ 0.0 |  93.0 $\pm$ 0.0 |  70.6 $\pm$ 0.0 | - | - | 18
TexturalSG (Yang et al., 2023) | 88.2 | 91.0 | 92.9 | 90.1 | 47.6 | 18.9 |  71.5 $\pm$ 1.9 |  94.1 $\pm$ 1.2 | - | Random | - | -
| LMSeg (Ours-avg) | 81.4 | 93.4 | 93.3 | 68.2 | 71.9 | 30.0 | 73.0 $\pm.03$ | 94.0 $\pm.07$ | 79.8 $\pm.06$ | Random | 1.7 | 2
Table 2: 3D semantic segmentation results on BBW dataset. Proportions of surface area (per semantic object class) of test set: Wall (11.4%) vs. Terra. (88.5%). Performance reported is mean IoU (%) and F1 score (%). Model efficiency is evaluated by inference time (second) on test and validation sets. We present the model performance on validation (Area 4 - Top) and test (Area 2 - Bottom) set. Best results marked in bold. Method | F1 | mIoU | Sampling |  Params. (M) |  $t$ inf. (Sec.)
---|---|---|---|---|---
RandLANet (Hu et al., 2020) | 66.4 | 50.4 | Random | 1.3 | 24.2
PointNet++ (Qi et al., 2017b) | 60.1 | 42.9 | FPS | 1.0 | 42.7
PointTransformer (Zhao et al., 2021) | 59.7 | 42.6 | FPS | 5.0 | 92.2
DeeperGCNs (Li et al., 2023) | 48.3 | 32.8 | - | 1.9 | 47.8
Graph U-Net (Gao and Ji, 2019) | 39.7 | 25.1 |  Top$k$ | 0.85 | 44.2
LMSeg (Ours) | 74.0 | 58.5 | Random | 1.7 | 27.3
RandLANet (Hu et al., 2020) | 69.8 | 53.7 | Random | 1.3 | 22.8
PointNet++ (Qi et al., 2017b) | 60.3 | 43.3 | FPS | 1.0 | 42.3
PointTransformer (Zhao et al., 2021) | 64.5 | 47.5 | FPS | 5.0 | 90.1
DeeperGCNs (Li et al., 2023) | 51.6 | 35.5 | - | 1.9 | 46.8
Graph U-Net (Gao and Ji, 2019) | 43.0 | 28.9 |  Top$k$ | 0.85 | 44.7
LMSeg (Ours) | 74.6 | 59.5 | Random | 1.7 | 27.7
Figure 6: Qualitative performance of LMSeg on SUM dataset. Figure 7:
Qualitative performance of LMSeg on BBW dataset.

(a) Feature maps visualised for SUM dataset, where different colors of
surfaces mapping to over-segmented surfaces.

(b) Feature maps visualised for BBW dataset, where lighter colors indicate
stronger feature responses.

Figure 8: Feature maps of final feature propagation layer of LMSeg (Fig. 1) on
SUM (top) and BBW (bottom) datasets.

###  4.4 Results

#### Performance on SUM:

Table 1 presents the performance of LMSeg evaluated on SUM. LMSeg achieves an
overall performance of 73.0% mIou and 79.8% mAcc. on SUM and significantly
outperforms all point-based learning methods (Qi et al., 2017a, b; Hu et al.,
2020; Thomas et al., 2019), trained on the original 3D point clouds densely
sampled from the faces of textured meshes (See Fig.3).

LMSeg also outperforms existing mesh-based learning approaches, RF-MRF
(Rouhani et al., 2017) and SUM-RF (Gao et al., 2021), trained on over-
segmented surperfacets (i.e., clusters of adjacent mesh faces that share
similar characteristics) based on hand-crafted geometric and radiometric mesh
features. Comparing to a more recent mesh-based learning approach, the
TexturalSG (Yang et al., 2023), which leverages the surface graph of textured
meshes as inputs, LMSeg achieves an overall comparable segmentation
performance and substantially outperforms TexturalSG on small-object classes
(i.e., Vehic. and Boat).

LMseg achieves a highly efficient inference performance on SUM dataset,
requiring only $\sim$2 minutes (the second-best inference performance after
PointNet++) for mesh segmentation on test dataset. We attribute the efficient
inference performance of LMSeg to the random node sub-sampling and heuristic
edge similarity pooling method proposed in Section 3.4. The point-based
method, KPConv, has comparable overall performance (68.8% mIou and 73.7%
mAcc.) as LMSeg, but requires nearly 10$\times$ more training parameters and a
14$\times$ longer inference time for mesh segmentation.

LMSeg achieves superior performance on four out of six semantic object classes
(i.e., H-veg: 93.4%, Build.: 93.3%, Vehic.: 71.9% and Boat: 30.0%) comparing
to existing learning-based segmentation methods. Notably, LMSeg outperforms
other segmentation methods on small-object class (i.e., Vehic. and Boat) by a
large margin, suggesting the geometrical advantages of the graph message-
passing GA+ layers and adaptive feature aggregation proposed in Section 3.3.
The feature maps of SUM in Fig. 8(a) further show that LMSeg can effectively
learn strong latent embeddings for large (i.e., Build. and H-veg.) and small
non-planar objects (i.e., Vehic. and Boat) in large-scale urban landscapes.

The qualitative results of LMSeg tested on SUM are presented through error
maps in Fig. 6. Prediction errors occur mostly at the boundary of two
connected objects in ambiguous regions (i.e., ground surfaces adjacent to
boats and water surface, as shown in row 5, Fig. 6). These ambiguous regions
likely cover areas with common labelling errors in triangle mesh annotations,
as discussed extensively in (Gao et al., 2021). Other errors happen on
segmentation between ground and water surfaces (i.e., Water and Terra.
classes), sharing geometrically similar barycentric graph structures but cover
un-even portions of the surface area of SUM classes (i.e., Terra. class
(23.6%) vs. Water class (4.8%)).

#### Performance on BBW:

We present LMSeg validation results (on Area 4) and test results (on Area 2)
in Table 2 and the qualitative performance in Fig. 7. For comparison, we
include different point-based learning methods (i.e., RandLANet (Hu et al.,
2020), PointNet++ (Qi et al., 2017b) and Point Transformer (Zhao et al.,
2021)) and graph-based learning methods (i.e., DeeperGCNs (Li et al., 2023)
and Graph U-Net (Gao and Ji, 2019)), ensuring comparable model sizes and
training parameters. In practice, the graph-based methods take the barycentric
graph of the processed landscape mesh as inputs and the point-based methods
take the vertex points of the barycentric graph as inputs. Overall, LMSeg
achieves the best mIoU of 59.5% and 58.5%, F1 score of 74.6% and 74.0% among
all methods on test (Area 2) and validation (Area 4) sets.

Natural landscape meshes in BBW, unlike urban landscape meshes in SUM, have
dense, uniformly distributed triangles capturing fine-grained geometric
objects (i.e., walls and ditches) of the terrain surface. Point-based learning
methods (i.e., RandLANet (Hu et al., 2020): 50.4% and 53.7% mIoU, Point
Transformer (Zhao et al., 2021): 42.6% and 47.5% mIoU, PointNet++ (Qi et al.,
2017b): 42.9% and 43.3% mIoU) perform generally better than graph-based
learning methods with deep residual graph convolution networks (i.e.,
DeeperGCNs (Li et al., 2023): 32.8% and 35.5% mIoU and Graph U-Net (Gao and
Ji, 2019): 25.1% and 28.9 mIoU) operating on the barycentric graph of meshes.

Comparing to point-based methods (Qi et al., 2017b; Hu et al., 2020; Zhao et
al., 2021) that commonly leverage relative point positions to learn local
geometric features of 3D point clouds in local neighbourhoods (Eq. 1) and
apply simple aggregations (i.e., max or mean function) for local geometric
aggregation, LMSeg’s GA+ layers more effectively capture strong and salient
geometric features (i.e., slopes uprising from planar surfaces and terrains)
and adaptively aggregate latent geometric features from local neighbours via
the proposed learnable aggregation layers (Eq. LABEL:eq:aggr_softmax_add). The
feature maps of BBW in Fig. 8(b) visualise the strong feature response (light
color) on the locations of irregular stone wall structures compared to the
weak feature response (dark color) on terrains in natural landscapes.

Comparing to graph-based methods (i.e., DeeperGCNs with very deep 28-ResGCN
layers, and Graph U-Net with hierarchical graph structure learning via graph
convolution layers and Top$k$ node sub-sampling), LMSeg’s HGA+ and LGA+ layers
have the benefits of learning hierarchical and local geometric features of
barycentric graphs, leading to a more efficient and accurate model
performance.

Error maps of LMSeg on the BBW dataset (Fig. 4) show prediction errors mainly
at ambiguous boundaries between irregular stone walls and the terrain, likely
due to misclassified labels from human annotators.

Table 3: List of ablated models tested in the Ablation study. Ablated Model | Description
---|---
SUM-A |  Without HGA+ components.
SUM-B |  Without LGA+ components.
SUM-C |  Without learnable feature aggregation.
SUM-D and E |  With $8$\- and $12$-nearest neighbours.
SUM-F |  Without SUM HSV colour.
SUM-G |  Without SUM face normals.
BBW-A and B |  Tested on 75%-res. meshes.
BBW-C and D |  Tested on 50%-res. meshes.
BBW-E and F |  Tested on 30%-res. meshes.
BBW-G and H |  Without BBW HSV colour.
BBW-I and J |  Without BBW face normals.
Table 4: Ablation study of LMSeg on SUM dataset. $\Delta$ refers to the performance (mIoU) difference of ablated models compared to LMSeg. Model |  $k$-nbrs. | Encoder | Terra. | H-veg. | Build. | Water | Vehic. | Boat | mIoU | $\Delta$
---|---|---|---|---|---|---|---|---|---|---
SUM-A |  3∗ | LGA+ | 73.7 | 90.4 | 91.2 | 33.2 | 50.6 | 6.8 | 57.7 | -15.3
SUM-B | 16 | HGA+ | 78.9 | 92.4 | 92.5 | 63.0 | 68.0 | 27.3 | 70.4 | -2.6
SUM-C |  16 (3∗) |  HGA & LGA | 77.7 | 92.4 | 92.6 | 58.7 | 61.5 | 30.7 | 68.9 | -4.1
SUM-D |  8 (3∗) |  HGA+ & LGA+ | 77.9 | 92.3 | 92.2 | 58.4 | 61.7 | 26.1 | 68.1 | -4.9
SUM-E |  12 (3∗) |  HGA+ & LGA+ | 81.7 | 93.6 | 93.3 | 69.2 | 70.9 | 29.6 | 73.1 | +0.1
SUM-F |  16 (3∗) |  HGA+ & LGA+ | 71.0 | 65.8 | 82.6 | 40.1 | 58.0 | 25.9 | 57.2 | -15.8
SUM-G |  16 (3∗) |  HGA+ & LGA+ | 79.0 | 93.1 | 92.5 | 68.4 | 68.7 | 29.3 | 71.9 | -1.1
LMSeg (Ours-avg) | 16 (3∗) |  HGA+ & LGA+ | 81.4 | 93.4 | 93.3 | 68.2 | 71.9 | 30.0 | 73.0 | -
Table 5: Ablation study of LMSeg on BBW dataset. $\Delta$ refers to the performance (mIoU) difference of ablated models compared to LMSeg performance on evaluation (Area 4 - Top) and test (Area 2 - Bottom) set. Mesh resolution is the ratio of face numbers of simplified mesh to original mesh. Model |  Mesh Res. ($\%$) | F1 | mIoU | $\Delta$
---|---|---|---|---
BBW-A | 75.0 | 74.3 | 59.1 | +0.6
BBW-C | 50.0 | 69.8 | 53.4 | -5.1
BBW-E | 30.0 | 54.0 | 37.4 | -21.1
BBW-G | 100.0 | 70.7 | 54.7 | -3.8
BBW-I | 100.0 | 73.6 | 58.3 | -0.2
LMSeg (Ours) | 100.0 | 74.0 | 58.5 | -
BBW-B | 75.0 | 75.1 | 60.6 | +1.1
BBW-D | 50.0 | 71.8 | 56.3 | -3.2
BBW-F | 30.0 | 59.1 | 42.2 | -17.3
BBW-H | 100.0 | 72.1 | 56.4 | -3.1
BBW-J | 100.0 | 74.5 | 59.4 | -0.1
LMSeg (Ours) | 100.0 | 74.6 | 59.5 | -

(a) SUM dataset.

(b) BBW dataset.

Figure 9: The weight vector distributions of inverse temperature $t$ on SUM
and BBW dataset. Weights are initialised to ones to instantiate softmax
aggregation.

##  5 Discussion

We conduct a series of ablation studies of LMseg on the SUM and BBW datasets
(Table 3) and present the results of the ablated models in Table 4 and 5,
respectively, to gain an in-depth understanding and identify the effective
components of the model.

#### Effect of HGA+ and LGA+:

Removing either HGA+ or LGA+ components from the LMSeg architecture triggers
performance deterioration (LMSeg mIoU drop: 73.0% $\rightarrow$ SUM-A: 57.7%
and SUM-B: 70.4%, Table 4). Removing the HGA+ components (SUM-A) from LMSeg
significantly lowers the model’s segmentation performance on all object
classes. The decrease in performance caused by removing LGA+ (SUM-B) from
LMSeg impacts the Water and small-object classes (i.e., Vehic.: 71.9%
$\rightarrow$ 68.0%, Boat: 30.0% $\rightarrow$ 27.3% and Water: 68.2%
$\rightarrow$ 63.0%). The ablation study of HGA+ and LGA+ components of LMSeg
confirms the importance of learning hierarchical and local geometric features
for landscape mesh segmentation, in particular for the segmentation of small-
object classes.

#### Effect of learnable feature aggregation:

SUM-C (Table 4) tests the effect of learnable feature aggregation in GA+
layers of LMSeg on landscape mesh segmentation. SUM-C removes the proposed
adaptive aggregation in HGA+ and LGA+ modules (Eq. LABEL:eq:aggr_softmax_add)
and applies only the simple aggregations (i.e., max and mean function), as
commonly applied in point-based learning methods (Qi et al., 2017b; Hu et al.,
2020; Zhao et al., 2021). We observe that SUM-C with vanilla-GA layers (i.e.,
HGA and LGA) leads to reduced performance (LMSeg mIoU: 73.0% $\rightarrow$
SUM-C: 68.9%) across object classes (except for Boat) with the most severe
mIoU deterioration on Water and Vehic. classes (Water: 68.2% $\rightarrow$
58.7%, Vehic.: 71.9% $\rightarrow$ 61.5%).

We further delve into the effect of learnable feature aggregation by
visualising the weight vector distributions of inverse temperature $t$ (all-
one vectors to instantiate softmax aggregation) of learnable aggregation
layers (Fig. 9). Fig. 9(a)-(b) show that the learnable aggregation layers
trained on SUM and BBW datasets enable adaptive aggregation of latent
geometric features. This is the effect of approximation of generalised
aggregate functions by the learning of inverse temperature $t$ (Li et al.,
2023): $aggr_{\mathbf{mean}}=aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}=0)$,
$aggr_{\mathbf{min}}=aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}\to-\infty)$ and
$aggr_{\mathbf{max}}=aggr_{\mathbf{softmax}}(\cdot;\mathbf{t}\to+\infty)$. The
quantitative results of ablated SUM-C and the visualisation of weight vector
distributions of inverse temperature $t$ collectively demonstrate the
effectiveness and advantage of learnable aggregation layers for landscape mesh
segmentation over simple aggregations (Qi et al., 2017b; Hu et al., 2020; Zhao
et al., 2021).

#### Effect of local nearest neighbours:

SUM-D and E examine the effect of $k$-nearest neighbors in HGA+ layers on
model performance. Results show that reducing $k$ leads to fluctuating mIoU:
73.0% (LMSeg, $k=16$) $\rightarrow$ 73.1% (SUM-D, $k=12$) and 68.1% (SUM-E,
$k=8$). Therefore, $k$ is a tunable hyper-parameter for the target mesh
dataset. Increasing $k$ can potentially capture richer geometric information
on hierarchical sub-sampled barycentric graph
$\mathcal{G}^{l-1}(\mathcal{M})\rightarrow\mathcal{G}^{l}(\mathcal{M})$ for
better performance but at a higher computational cost.

#### Effect of mesh resolution:

We investigated LMSeg’s performance on natural landscape meshes at 75%, 50%,
and 30% resolutions of the BBW dataset (Table 5). The meshes were simplified
via QEM simplification to coarser resolutions and converted to barycentric
graphs for input. At 75% resolution, models BBW-A (58.5% $\rightarrow$ 59.1%
on Area 4) and BBW-E (59.5% $\rightarrow$ 60.6% on Area 2) showed slight mIoU
improvements. At 50% resolution, models BBW-B (58.5% $\rightarrow$ 53.4% on
Area 4) and BBW-F (59.5% $\rightarrow$ 56.3% on Area 2) suffered minor mIoU
degradation. At 30% resolution, models BBW-C (58.5% $\rightarrow$ 37.4% on
Area 4) and BBW-G (59.5% $\rightarrow$ 42.2% on Area 2) experienced
significant mIoU degradation. These results demonstrate LMSeg’s robustness
across varying landscape resolutions.

#### Effect of mesh features:

LMSeg has been tested with with face normals and HSV colour features
$\mathcal{X}$ appended to nodes $\mathcal{V}$ of the barycentric graph
$\mathcal{G}(\mathcal{M})$.

Models SUM-F and SUM-G (Table 3) test the impact of HSV colour and face
normals, respectively, on SUM dataset. The result of ablated SUM-G on Table 4
indicates that the face normals have a mild impact on model performance,
decreasing mIoU by 1.1$\%$ (73.0% $\rightarrow$ 71.9%). In contrast, the
removal of HSV color features (i.e., ablated SUM-F) significantly degrades
mIoU by 15.8% (73.0% $\rightarrow$ 57.2%) across object classes, indicating
that HSV color is crucial for distinguishing objects (i.e., H-veg., Water and
Vehic.) in urban landscapes.

Models BBW-G, H, and BBW-I, J (Table 3) assessed the impact of HSV color and
face normals on the BBW validation and test sets. Removing HSV color features
(i.e., BBW-G and H) resulted in relatively small performance drops (58.5%
$\rightarrow$ 54.7% on Area 4 and 59.5% $\rightarrow$ 56.4% on Area 2). The
removal of face normals ((i.e., BBW-I and J) showed negligible changes (58.5%
$\rightarrow$ 58.3% on Area 4 and 59.5% $\rightarrow$ 59.4% on Area 2). This
is likely because the geometric features (face normals) of terrain and wall
classes are similar, and the semantic information of stone walls is affected
by bush/tree objects, as shown in Fig. 7, making these features less effective
for segmentation.

##  6 Conclusions

In this study, we introduce LMSeg, an efficient deep graph message-passing
network for accurate semantic segmentation of large-scale 3D landscape meshes.
LMSeg utilizes barycentric dual graphs of landscape meshes, enabling end-to-
end model training with deep graph message-passing networks. It leverages
geometric and texture features of triangular surface meshes (face normals and
color information) for accurate segmentation, learning hierarchical and local
geometric features through Geometry Aggregation Plus (GA+) layers.

Our results on large-scale urban and natural landscape datasets show that
LMSeg outperforms existing mesh segmentation methods in accuracy and inference
efficiency, with notable improvements in segmenting small mesh objects (i.e.,
Vehic. and Boat in SUM) and irregular stone wall structures.

However, LMSeg shows lower performance on mesh objects with geometrically
similar barycentric graph structures (e.g., Water and Terra. in SUM) due to
limited network receptive fields. Future research will focus on developing a
network capable of learning finer semantic mesh features and capturing
comprehensive global spatial information with a larger receptive field.

## CRediT authorship contribution statement

Zexian Huang: Conceptualization, Writing – original draft, Writing – review &
editing, Software, Methodology, Investigation. Kourosh Khoshelham: Writing –
review & editing, Supervision. Gunditj Mirring Traditional Owners Corporation:
Supervision. Martin Tomko: Writing – review & editing, Supervision.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data availability

The source code and data support the findings of this study are available in
figshare.com with the anonymous link:
https://figshare.com/s/046d7cf4576c75ed0571.

## Acknowledgements

We thank Gunditj Mirring Traditional Owners Corporation (GMTOC) for providing
the research data and overseeing its usage in this study, as well as Mashnoon
Islam for his dedicated data annotation work; their invaluable cooperation,
support, and meticulous efforts have significantly contributed to the quality
of our research.

## References

  * Adam et al. (2023) Adam, J.M., Liu, W., Zang, Y., Afzal, M.K., Bello, S.A., Muhammad, A.U., Wang, C., Li, J., 2023.  Deep learning-based semantic segmentation of urban-scale 3d meshes in remote sensing: A survey.  International Journal of Applied Earth Observation and Geoinformation 121, 103365.
  * Bell et al. (2019) Bell, D., Molloy, L., Prpic, J., Tomko, M., 2019.  A new world heritage site for aboriginal engineering.  Nature 572, 32–33.
  * Blaschke (2010) Blaschke, T., 2010.  Object based image analysis for remote sensing.  ISPRS journal of photogrammetry and remote sensing 65, 2–16.
  * Bronstein et al. (2011) Bronstein, A.M., Bronstein, M.M., Guibas, L.J., Ovsjanikov, M., 2011.  Shape google: Geometric words and expressions for invariant shape retrieval.  ACM Transactions on Graphics (TOG) 30, 1–20.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2020.  An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations.
  * Feng et al. (2019) Feng, Y., Feng, Y., You, H., Zhao, X., Gao, Y., 2019.  Meshnet: Mesh neural network for 3d shape representation, in: Proceedings of the AAAI conference on artificial intelligence, pp. 8279–8286.
  * Fey and Lenssen (2019) Fey, M., Lenssen, J.E., 2019.  Fast graph representation learning with pytorch geometric.  arXiv preprint arXiv:1903.02428 .
  * Gao and Ji (2019) Gao, H., Ji, S., 2019.  Graph u-nets, in: international conference on machine learning, PMLR. pp. 2083–2092.
  * Gao et al. (2021) Gao, W., Nan, L., Boom, B., Ledoux, H., 2021.  Sum: A benchmark dataset of semantic urban meshes.  ISPRS Journal of Photogrammetry and Remote Sensing 179, 108–120.
  * Garland and Heckbert (1997) Garland, M., Heckbert, P.S., 1997.  Surface simplification using quadric error metrics, in: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pp. 209–216.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E., 2017.  Neural message passing for quantum chemistry, in: International conference on machine learning, PMLR. pp. 1263–1272.
  * Girshick et al. (2014) Girshick, R., Donahue, J., Darrell, T., Malik, J., 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587.
  * Guan et al. (2021) Guan, B., Li, H., Zhou, F., Lin, S., Wang, R., 2021.  Lgcpnet: local-global combined point-based network for shape segmentation.  Computers & Graphics 97, 208–216.
  * Hanocka et al. (2019) Hanocka, R., Hertz, A., Fish, N., Giryes, R., Fleishman, S., Cohen-Or, D., 2019.  Meshcnn: a network with an edge.  ACM Transactions on Graphics (ToG) 38, 1–12.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hu et al. (2020) Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., Markham, A., 2020.  Randla-net: Efficient semantic segmentation of large-scale point clouds, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11108–11117.
  * Hu et al. (2021) Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., Markham, A., 2021.  Learning semantic segmentation of large-scale point clouds with random sampling.  IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 8338–8354.
  * Kipf and Welling (2017) Kipf, T.N., Welling, M., 2017.  Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net.  URL: https://openreview.net/forum?id=SJU4ayYgl.
  * Kölle et al. (2021) Kölle, M., Laupheimer, D., Schmohl, S., Haala, N., Rottensteiner, F., Wegner, J.D., Ledoux, H., 2021.  The hessigheim 3d (h3d) benchmark on semantic segmentation of high-resolution 3d point clouds and textured meshes from uav lidar and multi-view-stereo.  ISPRS Open Journal of Photogrammetry and Remote Sensing 1, 100001.
  * Landrieu and Simonovsky (2018) Landrieu, L., Simonovsky, M., 2018.  Large-scale point cloud semantic segmentation with superpoint graphs, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4558–4567.
  * Laupheimer et al. (2020) Laupheimer, D., Shams Eddin, M., Haala, N., 2020.  On the association of lidar point clouds and textured meshes for multi-modal semantic segmentation.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2, 509–516.
  * Li et al. (2019) Li, G., Muller, M., Thabet, A., Ghanem, B., 2019.  Deepgcns: Can gcns go as deep as cnns?, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 9267–9276.
  * Li et al. (2023) Li, G., Xiong, C., Qian, G., Thabet, A., Ghanem, B., 2023.  Deepergcn: training deeper gcns with generalized aggregation functions.  IEEE Transactions on Pattern Analysis and Machine Intelligence .
  * Loshchilov and Hutter (2017) Loshchilov, I., Hutter, F., 2017.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101 .
  * Ma et al. (2022) Ma, X., Qin, C., You, H., Ran, H., Fu, Y., 2022.  Rethinking network design and local geometry in point cloud: A simple residual mlp framework.  arXiv preprint arXiv:2202.07123 .
  * Milano et al. (2020) Milano, F., Loquercio, A., Rosinol, A., Scaramuzza, D., Carlone, L., 2020.  Primal-dual mesh convolutional neural networks.  Advances in Neural Information Processing Systems 33, 952–963.
  * Peucker et al. (1976) Peucker, T.K., Fowler, R.J., Little, J.J., Mark, D.M., COLUMBIA)*, S.T.D.L.V.B., 1976.  Digital representation of threedimensional surfaces by Triangulated Irregular Networks (TIN)(Revised).  Simon Fraser University, Burnaby, British Columbia, Canada and Office of Naval Research, Geography Programs, Arlington, Virginia, USA, Technical Report No 10.
  * Qi et al. (2017a) Qi, C.R., Su, H., Mo, K., Guibas, L.J., 2017a.  Pointnet: Deep learning on point sets for 3d classification and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652–660.
  * Qi et al. (2017b) Qi, C.R., Yi, L., Su, H., Guibas, L.J., 2017b.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems 30.
  * Qian et al. (2022) Qian, G., Li, Y., Peng, H., Mai, J., Hammoud, H., Elhoseiny, M., Ghanem, B., 2022.  Pointnext: Revisiting pointnet++ with improved training and scaling strategies.  Advances in Neural Information Processing Systems 35, 23192–23204.
  * Rouhani et al. (2017) Rouhani, M., Lafarge, F., Alliez, P., 2017.  Semantic segmentation of 3d textured meshes for urban scene analysis.  ISPRS Journal of Photogrammetry and Remote Sensing 123, 124–139.
  * Schult et al. (2020) Schult, J., Engelmann, F., Kontogianni, T., Leibe, B., 2020.  Dualconvmesh-net: Joint geodesic and euclidean convolutions on 3d meshes, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612–8622.
  * Singh et al. (2021) Singh, V.V., Sheshappanavar, S.V., Kambhamettu, C., 2021.  Meshnet++: A network with a face., in: ACM Multimedia, pp. 4883–4891.
  * Smith et al. (2019) Smith, A., McNiven, I.J., Rose, D., Brown, S., Johnston, C., Crocker, S., 2019.  Indigenous knowledge and resource management as world heritage values: Budj bim cultural landscape, australia.  Archaeologies 15, 285–313.
  * Tailor et al. (2021) Tailor, S.A., de Jong, R., Azevedo, T., Mattina, M., Maji, P., 2021.  Towards efficient point cloud graph neural networks through architectural simplification, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2095–2104.
  * Tancik et al. (2020) Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R., 2020.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems 33, 7537–7547.
  * Tang et al. (2022) Tang, R., Xia, M., Yang, Y., Zhang, C., 2022.  A deep-learning model for semantic segmentation of meshes from uav oblique images.  International Journal of Remote Sensing 43, 4774–4792.
  * Tang and Qiu (2021) Tang, W., Qiu, G., 2021.  Dense graph convolutional neural networks on 3d meshes for 3d object segmentation and classification.  Image and Vision Computing 114, 104265.
  * Thomas et al. (2019) Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J., 2019.  Kpconv: Flexible and deformable convolution for point clouds, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 6411–6420.
  * Tutzauer et al. (2019) Tutzauer, P., Laupheimer, D., Haala, N., 2019.  Semantic urban mesh enhancement utilizing a hybrid model.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 4, 175–182.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017.  Attention is all you need.  Advances in neural information processing systems 30.
  * Wang et al. (2022) Wang, L., Li, R., Zhang, C., Fang, S., Duan, C., Meng, X., Atkinson, P.M., 2022.  Unetformer: A unet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery.  ISPRS Journal of Photogrammetry and Remote Sensing 190, 196–214.
  * Wang et al. (2019) Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M., 2019.  Dynamic graph cnn for learning on point clouds.  ACM Transactions on Graphics (tog) 38, 1–12.
  * Weixiao et al. (2023) Weixiao, G., Nan, L., Boom, B., Ledoux, H., 2023.  Pssnet: Planarity-sensible semantic segmentation of large-scale urban meshes.  ISPRS Journal of Photogrammetry and Remote Sensing 196, 32–44.
  * Wu et al. (2015) Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J., 2015.  3d shapenets: A deep representation for volumetric shapes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912–1920.
  * Xu et al. (2018) Xu, K., Hu, W., Leskovec, J., Jegelka, S., 2018.  How powerful are graph neural networks?  arXiv preprint arXiv:1810.00826 .
  * Yang et al. (2023) Yang, Y., Tang, R., Xia, M., Zhang, C., 2023.  A surface graph based deep learning framework for large-scale urban mesh semantic segmentation.  International Journal of Applied Earth Observation and Geoinformation 119, 103322.
  * Zhang et al. (2023) Zhang, R., Wang, L., Wang, Y., Gao, P., Li, H., Shi, J., 2023.  Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis.  arXiv preprint arXiv:2303.08134 .
  * Zhao et al. (2021) Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V., 2021.  Point transformer, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259–16268.