# LSK3DNet: Towards Effective and Efficient 3D Perception with Large
Sparse Kernels

Tuo Feng1, Wenguan Wang2, Fan Ma2, Yi Yang2
1 ReLER, AAII, University of Technology Sydney  2 ReLER, CCAI, Zhejiang
University
https://github.com/FengZicai/LSK3DNet

###### Abstract

Autonomous systems need to process large-scale, sparse, and irregular point
clouds with limited compute resources. Consequently, it is essential to
develop LiDAR perception methods that are both efficient and
effective.${}_{\\!}$ Although naive- ly enlarging 3D kernel size can enhance
performance, it will also lead to a cubically-increasing overhead. Therefore,
it is crucial to develop streamlined 3D large kernel designs that eliminate
redundant weights and work effectively with larger kernels. In this paper, we
propose an efficient and effective Large Sparse Kernel 3D Neural Network
(LSK3DNet) that leverages dynamic pruning to amplify the 3D kernel size. Our
method comprises two core components: Spatial-wise Dynamic Sparsity (SDS) and
Channel-wise Weight Selection (CWS). SDS dynamically prunes and regrows
volumetric weights from the beginning to learn a large sparse 3D kernel. It
not only boosts performance but also significantly reduces model size and
computational cost.${}_{\\!}$ Moreover, CWS selects the most important
channels for 3D convolution during training and subsequently prunes the
redundant channels to accelerate inference for 3D vision tasks. We demonstrate
the effectiveness of LSK3DNet on three benchmark datasets and five tracks
compared with classical models and large kernel designs. Notably, LSK3DNet
achieves the state-of-the-art performance on SemanticKITTI (_i.e_., 75.6% on
single-scan and 63.4% on multi-scan), with roughly 40% model size reduction
and 60% computing operations reduction compared to the naive large 3D kernel
model.

##  1 Introduction

Autonomous systems, such as intelligent robots and self-driving cars, are
equipped with 3D LiDAR sensors and data processing platforms. The LiDAR sensor
generates point clouds, which serve as the input for the processing platform.
The platform performs perception tasks such as semantic segmentation [1, 2, 3]
and object detection [4, 5, 6], providing a foundation for motion planning and
decision-making [7] in autonomous systems. On the one hand, the processing
platform is required to handle large-scale, sparse, and irregular point
clouds. On the other hand, the platform’s computing resources are limited.
Therefore, it is crucial to develop effective and efficient LiDAR perception
methods. Point-based methods [1, 8, 9] rely on computationally expensive,
memory inefﬁcient, and time-consuming point sampling strategy [10]. In
contrast, sparse convolution [11, 12, 13, 14] has been widely adopted for
processing large-scale point clouds up to $160\\!\times\\!160\\!\times\\!20$
meters [12, 11, 15, 16, 5, 17].

[18]

Figure 1: Illustrations on Spatial-wise Group Convolution [18], SDS, and CWS.
The spatial dimensions $K_{s}$ (_i.e_., $K_{1}$, $K_{2}$, $K_{3}$) and channel
dimensions ($D_{in}$/$D_{out}$) are shown. Spatial-wise Group Convolution
shares the weights within each spatial group during training, leading to
redundant model weights. In contrast, SDS removes non-salient weights and
redundancies that are not sensitive to the input in each group, while CWS
eliminates redundant weights in a channel-wise manner (Sec. 1).

Nevertheless, the application of such a technique in autonomous systems has
also been hindered by limited computational resources [16]. PV-KD [19]
attempts to reduce the model size with knowledge distillation, but its
performance is bounded by its teacher model [15]. In opposition, large kernels
can enhance performance through the advantages of large receptive fields [20,
21, 18]. However, naive 3D large kernels will significantly increase
computational burdens. LargeKernel3D [18] explores large 3D kernels with
Spatial-wise Group Convolution to circumvent optimization and efficiency
problems caused by naive 3D large kernels. However, LargeKernel3D has a lot of
redundant model weights during training as it shares the weights in each
spatial group (See Fig. 1). Moreover, the performance of LargeKernel3D [18]
drops when scaling up the kernel size over $7\\!\times\\!7\\!\times\\!7$.

JS3C-Net[22]

SPVNAS[16]

MinkUNet[12]

Cylinder3D[15]

RPVNet[23]

SphereFormer[24]

LSK3DNet

PV-KD[19]

2DPASS[3]

($\uparrow$)

Figure 2: ${}_{\\!}$Performance${}_{\\!}$ (mIoU)${}_{\\!}$ vs.${}_{\\!}$
Inference${}_{\\!}$ Speed${}_{\\!}$ (FPS)${}_{\\!}$ on${}_{\\!}$ SemanticKITTI
[25] single-scan challenge (Sec. 1).

In this study, we propose a Large Sparse Kernel 3D Neural Network (LSK3DNet),
which is an efficient and effective 3D perception model. LSK3DNet incorporates
two key components, namely Spatial-wise Dynamic Sparsity (SDS) and Channel-
wise Weight Selection (CWS). These components enhance performance on
perception tasks and overcome the challenges of high computational costs. SDS
enlarges 3D kernels and reduces model parameters by learning sparse 3D kernels
from scratch. CWS increases the model width and learns channel-wise importance
during training, resulting in accelerated inference by pruning redundant
channels. As shown in Fig. 1, SDS can prune 3D kernels to remove redundant
weights, while CWS can eliminate redundant channels in an online manner. Our
SDS and CWS complement each other following a guideline of “using spatial
sparse groups, expanding width without more parameters”, in contrast to “using
sparse groups, expanding more” [21].

Specifically, SDS implements a remove-and-regrow process within each spatial
group and preserves the sparsity in each group during dynamic training. This
stands in stark contrast to the 2D approach [21], as it achieves dynamic
sparsity in depth-wise groups. SDS allows to scale up the receptive fields
with large kernel sizes, easily reaching or surpassing
$9\\!\times\\!9\\!\times\\!9$, thereby achieving higher performance. In
addition, CWS selectively identifies salient channels during training. It
speeds up inference on the 3D vision tasks by pruning non-salient channel
parameters. In this way, LSK3DNet can benefit from expanded width to achieve
enhanced performance, but maintain the original model size during inference.
Reducing the models’ complexity (_e.g_., model size) is a big benefit and a
straightforward way to make them deployable on resource-constrained devices
[26]. We conduct experiments on three benchmark datasets and five
tracks.${}_{\\!}$ Our method achieves better performance compared to state-of-
the-art methods [24, 23, 15] on SemanticKITTI [25] but with faster inference
speed (Fig. 2). Moreover, LSK3DNet outperforms the prior 3D large kernel
method of LargeKernel3D [18] on ScanNet v2 [27] and KITTI [28].

In a nutshell, our contributions are as follows:

  * •

We propose LSK3DNet to enhance performance in perception tasks and mitigate
high computational costs, surpassing state-of-the-art models while reducing
model size by 40% and computing operations by 60%.

  * •

We propose SDS to scale up 3D kernels. It learns large sparse kernels by
dynamically pruning and regrowing weights from scratch, thereby reducing model
size and computational operations.

  * •

We develop CWS to improve performance by expanding the width. CWS assesses the
importance of channels during training and speeds up inference by pruning
redundant channel-wise parameters.

##  2 Related Work

Large-Kernel 3D Models. In 3D vision, research on 3D large kernel is very
limited. LargeKernel3D [18] has demonstrated that sizeable kernels can be
successfully employed and bring positive results for 3D networks.${}_{\\!}$
Spatial-wise Group Convolution [18] enables to achieve a kernel size of
$7\\!\times\\!7\\!\times\\!7$. However, it shares the weights within each
spatial group during training, leading to redundant model weights (See Fig.
1). Moreover, the performance of LargeKernel3D drops when scaling up the
kernel size over $7\\!\times\\!7\\!\times\\!7$.

Large-Kernel${}_{\\!}$ 2D${}_{\\!}$ CNN${}_{\\!}$ Models. In the 2010s,
various large-kernel settings were investigated. LR-Net [29], Inceptions [30,
31], and GCNs [32] explored 2D large kernels of $7\times 7$, $11\times 11$,
and $15\times 15$ respectively. Due to the widespread adoption [33] of VGG
[34], research into large kernels was largely overlooked in favour of multiple
smaller kernels ($1\\!\times\\!1$ or $3\\!\times\\!3$) to obtain a larger
receptive field [35, 36, 37, 38] during the past decade. Recently, certain
studies have reintroduced large kernels in CNNs.${}_{\\!}$ RepLKNet [20]
examines the effects of large kernels in CNNs, and for the first time it is
able to increase kernel size to $31\\!\times\\!31$. It achieves comparable
results to those of Swin Transformer [39].${}_{\\!}$ Following “use sparse
groups, expand more”, SLaK [21] has achieved an impressive kernel size of
$51\\!\times\\!51$.${}_{\\!}$ SLaK has to strike a trade-off between sparsity
and model width since increased model width leads to an increase in model
size, a phenomenon that becomes even more pronounced in 3D vision.${}_{\\!}$
However, CWS effectively decouples the two objectives of improved performance
and reduced model size, enabling expanded width without increasing the model
size.

3D Backbones. Generally speaking, there are three mainstreams of 3D backbones:
i) Point-based methods [40, 41, 42, 43, 44, 45, 46, 47, 48, 1, 49, 50, 51, 8,
9, 52, 53, 54, 55] approximate a permutation-invariant set function, and
directly learn features from raw point clouds. Researchers have observed an
unsatisfactory outcome for large-scale environments [56]. ii) Projection-based
methods transform unstructured points into regular 2D images and thus employ
CNNs [57, 58, 59, 60, 61, 62]. However, the projection could leave out
critical geometric details and cause unavoidable information loss.${}_{\\!}$
iii) Voxel-based methods have used sparse convolution [12, 11] to design more
powerful networks for various 3D tasks [15, 63].${}_{\\!}$ Sparse convolution
facilitates convolution operations only on non-void voxels.${}_{\\!}$
Recently, fusion-based methods [16, 23, 3] have become upsurge. The point-
based high-resolution branch mitigates the performance degradation, which is
caused by aggressive downsampling [16, 64] (_i.e_., regular sparse
convolution) of the voxel branch.

Dynamic${}_{\\!}$ Sparse${}_{\\!}$ Training (DST). DST can train${}_{\\!}$
sparse${}_{\\!}$ neural${}_{\\!}$ networks${}_{\\!}$ from${}_{\\!}$
scratch,${}_{\\!}$ resulting${}_{\\!}$ in${}_{\\!}$ both${}_{\\!}$ a${}_{\\!}$
speedy${}_{\\!}$ training${}_{\\!}$ and${}_{\\!}$ prediction${}_{\\!}$
procedure.${}_{\\!}$ During${}_{\\!}$ training,${}_{\\!}$ DST [65, 66, 67, 68,
69, 70, 71, 72, 73]${}_{\\!}$ alters${}_{\\!}$ the${}_{\\!}$
location${}_{\\!}$ of${}_{\\!}$ non-zero${}_{\\!}$ weights${}_{\\!}$
with${}_{\\!}$ per pre-defined${}_{\\!}$ rules,${}_{\\!}$ thereby${}_{\\!}$
creating${}_{\\!}$ a${}_{\\!}$ sparse${}_{\\!}$ representation${}_{\\!}$
and${}_{\\!}$ cutting${}_{\\!}$ down${}_{\\!}$ the${}_{\\!}$ number${}_{\\!}$
of${}_{\\!}$ calculations.${}_{\\!}$ This${}_{\\!}$ paradigm${}_{\\!}$
commences${}_{\\!}$ without${}_{\\!}$ prior${}_{\\!}$ knowledge${}_{\\!}$
and${}_{\\!}$ simultaneously${}_{\\!}$ refines${}_{\\!}$ the${}_{\\!}$ non-
zero${}_{\\!}$ locations${}_{\\!}$ and${}_{\\!}$ weights.${}_{\\!}$
The${}_{\\!}$ attractive${}_{\\!}$ aspect${}_{\\!}$ of${}_{\\!}$ DST${}_{\\!}$
is${}_{\\!}$ that${}_{\\!}$ it${}_{\\!}$ is${}_{\\!}$ sparse${}_{\\!}$
from${}_{\\!}$ the${}_{\\!}$ beginning,${}_{\\!}$ resulting${}_{\\!}$
in${}_{\\!}$ lower${}_{\\!}$ FLOPs${}_{\\!}$ and${}_{\\!}$ memory${}_{\\!}$
consumption${}_{\\!}$ for${}_{\\!}$ training${}_{\\!}$ and${}_{\\!}$
inference${}_{\\!}$ compared${}_{\\!}$ to${}_{\\!}$ a${}_{\\!}$
dense${}_{\\!}$ model.${}_{\\!}$ Generally,${}_{\\!}$ the${}_{\\!}$
pruning${}_{\\!}$ process [74]${}_{\\!}$ can${}_{\\!}$ be${}_{\\!}$
completed${}_{\\!}$ either${}_{\\!}$ by${}_{\\!}$ threshold-based${}_{\\!}$
pruning [65, 75, 66, 70]${}_{\\!}$ or${}_{\\!}$ by${}_{\\!}$ magnitude-
based${}_{\\!}$ pruning [67, 76].${}_{\\!}$ In${}_{\\!}$ addition,${}_{\\!}$
new${}_{\\!}$ weights${}_{\\!}$ are${}_{\\!}$ regrown${}_{\\!}$ with${}_{\\!}$
randomness${}_{\\!}$ growing [75, 66, 70],${}_{\\!}$ momentum${}_{\\!}$
growing [67],${}_{\\!}$ and${}_{\\!}$ gradient-based${}_{\\!}$ growing [69,
76, 77, 68].${}_{\\!}$ Here, we efficiently integrate dynamic sparsity with 3D
kernels in large kernel 3D networks.${}_{\\!}$ This design satisfies the
anticipation for a larger parameter space, as a wide-ranging exploration of
the parameter space is of significance to DST [71, 78, 73].

##  3 Methodology

In this section, we first formulate the problem in Sec. 3.1, then move on to
a concise overview of submanifold sparse convolution in Sec. 3.2. Afterwards,
the details of Spatial-wise Dynamic Sparsity (SDS) and Channel-wise Weight
Selection (CWS) are described in Sec. 3.3 and 3.4. Lastly, network
architecture is outlined in Sec. 3.5.

###  3.1 Problem Formulation

For 3D perception tasks, suppose that the input is a set of unordered point
$\mathcal{X}\\!=\\!\left\\{x_{i}\in\mathbb{R}^{D_{in}}:i=1,2,\ldots,N\right\\}$,
and $D_{in}$ is the dimensionality of the input features.${}_{\\!}$
$\mathcal{L}$ is a set of object/point labels, which varies according to
different datasets. For segmentation task, LSK3DNet assigns each point $x_{i}$
with a predicted label $y_{i}$, resulting in a point-wise prediction set
$\mathcal{Y}\\!=\left\\{y_{i}:i=1,2,\ldots,N\right\\}$.${}_{\\!}$ In voxel-
based methods, point clouds are converted to voxel representations.${}_{\\!}$
We denote the input sparse tensor with $X_{s}^{in}=[C,F]$.${}_{\\!}$
Coordinate matrix $C\\!\in\\!\mathbb{N}^{N\times 3\\!}_{+}$ includes
3-dimensional discrete coordinates, and feature matrix
$F\\!\in\\!\mathbb{R}^{N\times D_{in}}$ has $D_{in}$-dimensional features.

$\bm{W}_{\\!D}$

$\bm{W}_{\\!S}$

Figure 3: Spatial-wise Dynamic Sparsity. The utilization of SDS enables us to
create and train sparse kernel 3D neural networks from the beginning. The
sparse weights in each spatial group are firstly initialized by Sparse Kernel
Initialization (Eq.${}_{\\!}$ (2)), and then regularly altered by discarding
the least significant connections and introducing new ones (Eq.${}_{\\!}$
(3)). The sparse kernels are steadily improved by this dynamic process, which
leads to a more thorough collection of local features. Note that different
spatial sparse group has different sparse distribution. Here we take a
sparsity of 22% for example (Sec. 3.3 and Alg. 1).

###  3.2 Review of Submanifold Sparse Convolution

Regular sparse convolution has widespread use in U-Net type 3D networks [11,
12].${}_{\\!}$ However, it significantly reduces the sparsity level of point
data [64], obfuscates feature distinctions [64], and leads to low-resolution
and indistinguishable small objects [16].${}_{\\!}$ In our LSK3DNet, we
discard regular sparse convolution and exclusively employ submanifold sparse
convolution for feature extraction.

In a $3$-dimensional space $C\\!\in\\!\mathbb{N}^{N\times 3\\!}_{+}$, the
$D_{in}$-dimensional input feature at $c\\!\in\\!C$ can be denoted as
$x^{in}_{c}\\!\in\\!\mathbb{R}^{D_{in}}$, and the 3D kernel is represented by
$\bm{W}\\!\in\\!\mathbb{R}^{K^{3}\\!\times\\!D_{\text{out}}\\!\times\\!D_{\text{in}}}$.${}_{\\!}$
We divide the kernel into $K^{3}$ spatial weights, each with a size of
$D_{out}\\!\times\\!D_{in}$, and denote the divided weights as $\bm{W}_{i}$.
The submanifold sparse convolution is formulated as follows:

$\small x^{{out}}_{c}=\sum_{i\in\mathcal{V}^{D}(c,C)}\bm{W}_{i}\thinspace
x^{in}_{c+i},\quad c\in C,\vspace{-2pt}$ (1)

where $x_{c}^{out}$ represents the current voxel on which the submanifold
sparse convolution is applied; $\bm{W}_{i}$ corresponds to $x_{c+i}^{in}$,
which represents the $i$-th adjacent input voxel ($[C_{c+i},F_{c+i}]$);
$\mathcal{V}^{D}$ is a set of offsets that define the shape of a kernel, and
$\mathcal{V}^{D}(c,C)\\!=\\!\\{i|c+i\\!\in\\!C,i\\!\in\\!\mathcal{V}^{D}\\!\\}$
is a set of offsets from the current center $c$ that exist in $C$.${}_{\\!}$
It should be noted that the input coordinates and output coordinates are
equivalent; in other words, they share the same $C$.${}_{\\!}$ Due to this
restriction, insufficient information flow exists while differentiating the
distinct spatial characteristics [64]. Large receptive fields could be a
potential solution to this problem.

###  3.3 Spatial-wise Dynamic Sparsity (SDS)

SDS incorporates two essential components: Sparse Kernel Initialization and
Sparse Weight Pruning and Growth.${}_{\\!}$ SDS is applied to the 3D large
kernel layers, while other layers are kept dense.${}_{\\!}$ When initialized
with Sparse Kernel Initialization, all spatial groups of sparse tensors have
the same level of sparsity $s$, where sparsity refers to the fraction of zeros
in sparse kernels.${}_{\\!}$ We adjust the sparse position within spatial
groups of 3D large kernel layers with an adaptation frequency
$f_{a}$.${}_{\\!}$ During training, at regular intervals determined by
$f_{a}$, the adjustable weights in the kernels are modified through a two-
phase procedure, which includes sparse weight pruning and sparse weight
growth.${}_{\\!}$ We maintain a constant number of non-zero parameters
(_i.e_., sparsity $s$) throughout the training process.

Sparse Kernel Initialization. Prior research [79] has explored the
distribution of sparsity parameters across multiple layers.${}_{\\!}$ The
proportion and adjustment of sparse weights are regulated by the layer-wise
sparsity ratio, which determines how the weights are modified to seek more
efficient sparse structures during training. However, this method is not
suitable for Spatial-wise Group Convolution, since 3D large kernels are
divided into multiple spatial groups. To ensure that each group has non-zero
weights, we execute Erdős-Rényi (ER) based sparsity ratios [66, 69, 80] in
each spatial group. Dense weights $\bm{W}_{\\!D}$ are firstly initialized in a
standard way (_i.e_., kaiming uniform initialization [81]). The binary Mask
$\bm{M}$ is then applied to these dense weights; $\bm{M}$ is generated using
an ER-based method with a sparsity of $s$. The zero-weight number in $\bm{M}$
is scaled by
$1\\!-\\!\frac{D_{in}\\!+\\!D_{out}\\!+\\!K_{1}^{g}\\!+\\!K_{2}^{g}\\!+\\!K_{3}^{g}}{D_{in}\\!\times\\!D_{out}\\!\times\\!K_{1}^{g}\\!\times\\!K_{2}^{g}\\!\times\\!K_{3}^{g}}$,
where $K_{1}^{g}$, $K_{2}^{g}$, and $K_{3}^{g}$ denote the spatial group
sizes. Therefore, the sparse weights $\bm{W}_{\\!S}$ are initialized as
follows:

$\small\begin{gathered}\small\bm{W}_{\\!S}\leftarrow\bm{W}_{\\!D}\odot\bm{M},\end{gathered}\vspace{-2pt}$
(2)

where $\odot$ represents the Hadamard product.

To design 3D networks with sparse groups, following the recipe of “using
spatial sparse groups, expanding width without more parameters”, we have
chosen Spatial-wise Group Convolution rather than Depth-wise Group
Convolution. Our experiments have also shown that Depth-wise Group Convolution
is unsuitable for large 3D kernels. This aligns with the findings in
LargeKernel3D [18]. This experiment is provided in the supplementary
materials.

Sparse${}_{\\!}$ Weight${}_{\\!}$ Pruning${}_{\\!}$ and${}_{\\!}$ Growth.
Previous research [21] in 2D convolution has illustrated that dynamic sparsity
can effectively enlarge kernel sizes and enhance scalability. But Depth-wise
Group Convolution is unsuitable for large 3D kernels [18]. Therefore, we adapt
sparse weights in each spatial group dynamically to accommodate the data.
Specifically, a certain rate of connections (_i.e_., prune rate $p$) with the
lowest magnitude are eliminated, and then we generate an equal number of new
random connections. The regrow weights are randomly placed at zero positions
within each spatial group. This is achieved by modifying the mask $\bm{M}$
(See Alg. 1). The positions of the eliminated connections and the newly
generated connections are denoted as $\bm{E}$ and $\bm{G}$, respectively. The
formula for pruning and regrowth can be expressed as follows:

$\small\begin{gathered}\small\bm{W}_{\\!S}\leftarrow\bm{W}_{\\!S}\odot(\bm{M}-\bm{E}),\
\bm{W}_{\\!S}\leftarrow\bm{W}_{\\!S}\odot(\bm{M}+\bm{G}).\end{gathered}\vspace{-2pt}$
(3)

By doing so, the sparsity of each spatial group is kept steady. Moreover, the
weights can be adjusted dynamically, allowing for improved local
characteristics. Please refer to Fig. 3 for details.${}_{\\!}$ Once training
is complete, mask $\bm{M}$ in spatial groups is also recorded. Unlike previous
methods [21], we employ the Spatial-wise Group Convolution and partition
dynamic sparse processes into separate spatial groups. Pruning and growth are
conducted independently within each group without interfering with each other.

We can assume that weights are changing factors over time. Then, removing the
least important weights is akin to the selection phase in natural evolution.
Alternatively, the random addition of new weights is analogous to the
alteration stage of evolutionary selection. This phenomenon is also analogous
to a biological process in the brain during sleep, known as synaptic
shrinking. Researchers found that the weakest neural link in the brain weakens
during slumber, while the vital neural connections remain how they are before.
This shows that one of the main roles of sleeping is to reset the overall
synaptic strength [82, 83].

Figure 4: LSK3DNet and LSK Block. (1)${}_{\\!}$ LSK3DNet: Point clouds are fed
into voxelizer for voxel-wise features. Then we extract features with LSK
Block and Point Branch. The final prediction is a point-wise output. Here MLPs
is the standard Multi-Layer Perceptrons.${}_{\\!}$ (2)${}_{\\!}$ Large Sparse
Kernel Block (LSK Block) (Sec. 3.5).

###  3.4 Channel-wise Weight Selection (CWS)

Despite SLaK [21] employs DST to enlarge the 2D kernel size, it mitigates the
performance degradation caused by sparsity in the way of expanding model
width.${}_{\\!}$ It strives to strike a balance between sparsity and
width.${}_{\\!}$ However, expanding width leads to the issue of increased
computational burden.${}_{\\!}$ Therefore, SLaK [21] faces the dilemma that
larger sparsity and smaller model size cannot be achieved simultaneously.
Instead of naively increasing network width, we propose CWS to decouple
sparsity and width. It selectively chooses the most salient channels and
obtains improved performance while keeping the original model size during
inference.

CWS operates in an online mode with a sorting frequency of $f_{s}$. A single
weight sorting cycle involves multiple iterations of Sparse Weight Pruning and
Growth, _i.e_., each cycle of weight sorting is a multiple of the adaptation
rate $f_{a}$.${}_{\\!}$ The width factor $w$ determines the extent to which
the number of channels is augmented, _i.e_., extending $D$-dimension to
$w\\!\times\\!D$-dimension.${}_{\\!}$ Once the expanded model has been
created, we consider the $D$-dimensional small model to be embedded within the
$w\\!\times\\!D$-dimensional augmented model [84]. When the number of training
iterations reaches the sorting condition, the channels are sorted based on
their L1 value (_i.e_., SortChannels), prompting the model to focus on the
most relevant channels. During validation, we select the most important
$D$-dimensional channels from $w\\!\times\\!D$-dimensional channels (_i.e_.,
SelectChannels). The pseudo-code for the entire algorithm is provided in Alg.
1. This operation effectively achieves higher performance while maintaining
the parameters within the expected size. This is especially important in
deployments where memory usage and computational efficiency are critical
factors.

Table 1: Results of LSK3DNet on SemanticKITTI [25] single-scan test (Sec.
4.2). Regarding input data format, P denotes points, V represents
voxelizations, R signifies range images, and 2DPASS incorporates additional 2D
data. “ms/s” means “milliseconds per scene”.

Methods |  Input |  mIoU |  road |  sidewalk |  parking |  other-ground |  building |  car |  truck |  bicycle |  motorcycle |  other-vehicle |  vegetation |  trunk |  terrain |  person |  bicyclist |  motorcyclist |  fence |  pole |  traffic sign |  speed (ms/s)
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointNet++ [41] | P | 20.1 | 72.0 | 41.8 | 18.7 | 5.6 | 62.3 | 53.7 | 0.9 | 1.9 | 0.2 | 0.2 | 46.5 | 13.8 | 30.0 | 0.9 | 1.0 | 0.0 | 16.9 | 6.0 | 8.9 | 5900
TangentConv [57] | R | 40.9 | 83.9 | 63.9 | 33.4 | 15.4 | 83.4 | 90.8 | 15.2 | 2.7 | 16.5 | 12.1 | 79.5 | 49.3 | 58.1 | 23.0 | 28.4 | 8.1 | 49.0 | 35.8 | 28.5 | 3000
PolarNet [62] | R | 54.3 | 90.8 | 74.4 | 61.7 | 21.7 | 90.0 | 93.8 | 22.9 | 40.3 | 30.1 | 28.5 | 84.0 | 65.5 | 67.8 | 43.2 | 40.2 | 5.6 | 61.3 | 51.8 | 57.5 | 62
RandLA-Net [10] | P | 55.9 | 90.5 | 74.0 | 61.8 | 24.5 | 89.7 | 94.2 | 43.9 | 29.8 | 32.2 | 39.1 | 83.8 | 63.6 | 68.6 | 48.4 | 47.4 | 9.4 | 60.4 | 51.0 | 50.7 | 880
SqueezeSegV3 [61] | R | 55.9 | 91.7 | 74.8 | 63.4 | 26.4 | 89.0 | 92.5 | 29.6 | 38.7 | 36.5 | 33.0 | 82.0 | 58.7 | 65.4 | 45.6 | 46.2 | 20.1 | 59.4 | 49.6 | 58.9 | 238
KPConv [42] | P | 58.8 | 90.3 | 72.7 | 61.3 | 31.5 | 90.5 | 95.0 | 33.4 | 30.2 | 42.5 | 44.3 | 84.8 | 69.2 | 69.1 | 61.5 | 61.6 | 11.8 | 64.2 | 56.4 | 47.4 | -
JS3C-Net [22] | V | 66.0 | 88.9 | 72.1 | 61.9 | 31.9 | 92.5 | 95.8 | 54.3 | 59.3 | 52.9 | 46.0 | 84.5 | 69.8 | 67.9 | 69.5 | 65.4 | 39.9 | 70.8 | 60.7 | 68.7 | 471
SPVNAS [16] | PV | 67.0 | 90.2 | 75.4 | 67.6 | 21.8 | 91.6 | 97.2 | 56.6 | 50.6 | 50.4 | 58.0 | 86.1 | 73.4 | 71.0 | 67.4 | 67.1 | 50.3 | 66.9 | 64.3 | 67.3 | 259
Cylinder3D [15] | V | 68.9 | 92.2 | 77.0 | 65.0 | 32.3 | 90.7 | 97.1 | 50.8 | 67.6 | 63.8 | 58.5 | 85.6 | 72.5 | 69.8 | 73.7 | 69.2 | 48.0 | 66.5 | 62.4 | 66.2 | 131
RPVNet [23] | RPV | 70.3 | 93.4 | 80.7 | 70.3 | 33.3 | 93.5 | 97.6 | 44.2 | 68.4 | 68.7 | 61.1 | 86.5 | 75.1 | 71.7 | 75.9 | 74.4 | 43.4 | 72.1 | 64.8 | 61.4 | 168
(AF)2-S3Net [85] | V | 70.8 | 92.0 | 76.2 | 66.8 | 45.8 | 92.5 | 94.3 | 40.2 | 63.0 | 81.4 | 40.0 | 78.6 | 68.0 | 63.1 | 76.4 | 81.7 | 77.7 | 69.6 | 64.0 | 73.3 | -
PV-KD [19] | V | 71.2 | 91.8 | 77.5 | 70.9 | 41.0 | 92.4 | 97.0 | 53.5 | 67.9 | 69.3 | 60.2 | 86.5 | 73.8 | 71.9 | 75.1 | 73.5 | 50.5 | 69.4 | 64.9 | 65.8 | 76
2DPASS [3] | PV | 72.9 | 89.7 | 74.7 | 67.4 | 40.0 | 93.5 | 97.0 | 61.1 | 63.6 | 63.4 | 61.5 | 86.2 | 73.9 | 71.0 | 77.9 | 81.3 | 74.1 | 72.9 | 65.0 | 70.4 | 62
SphereFormer [24] | V | 74.8 | 91.8 | 78.2 | 69.7 | 41.3 | 93.8 | 97.5 | 59.6 | 70.1 | 70.5 | 67.7 | 86.7 | 75.1 | 72.4 | 79.0 | 80.4 | 75.3 | 72.8 | 66.8 | 72.9 | 123
LSK3DNet | PV | 75.6 | 92.2 | 78.9 | 70.2 | 41.8 | 92.7 | 97.3 | 61.0 | 71.4 | 75.6 | 64.2 | 86.4 | 72.7 | 71.9 | 81.2 | 80.6 | 85.2 | 72.0 | 67.0 | 74.6 | 89

###  3.5 Network Architecture

Segmentation Baseline. U-Net type 3D networks (such as SparseConvNet [11] and
MinkUNet [12]) employ aggressive downsampling (_i.e_., regular sparse
convolution) to increase the receptive field but at the cost of reduced
resolution. However, with a low resolution, several points or tiny objects
could be combined into one single grid and become indistinguishable [16]. So
SPVCNN [16] is equipped with a high-resolution point-based branch.
Subsequently, 2DPASS [3] further modifies multi-representation branches by
omitting the regular sparse convolution. This is because it dilates all sparse
features and blurs valuable information, increasing the burden for following
layers [64]. Only submanifold sparse convolution is employed for feature
extraction in Modified SPVCNN [3]. By limiting the output feature positions to
the input positions, submanifold sparse convolution is able to circumvent the
computation burden. We use Modified SPVCNN as a baseline for our segmentation
network.${}_{\\!}$ This 3D network is compact yet powerful and can generate
high-resolution representations from sparse point clouds in large-scale
scenes.${}_{\\!}$ However, one challenge we face is that the restricted area
of submanifold sparse convolution limits the information flow and makes it
difficult to distinguish different spatial characteristics of the scene. To
address this challenge, we introduce LSK Block, which stands for Large Sparse
Kernel Block. This block increases the kernel sizes of submanifold sparse
convolution and expands the receptive field to facilitate information flow.
LSK Block has a standard residual structure that adds the output of identity
mapping to that of two stacked large kernel convolutions (Fig. 4 (2)). Our
network does not need the parallel convolutional branch with dilatation
convolution [18]. The details of our segmentation network architecture are
listed below:

Algorithm 1 Pseudo-code of SDS and CWS

1: set adaptation frequency $f_{a}$, sparsity $s$;

2: set sorting frequency $f_{s}$ and width factor $w$;

3: expand model width form $D$ dimension to $w\\!\times\\!D$ dimension;

4: initialize dense model $\bm{W}_{\\!D}^{w\\!\times\\!D}$;

5: initialize sparse layers $\bm{W}_{\\!S}^{w\\!\times\\!D}$;

6: for each training iteration $i$ do

7:
$\bm{W}_{\\!S}^{w\\!\times\\!D}\leftarrow$NormalTraining$(\bm{W}_{\\!S}^{w\\!\times\\!D})$;

8: if ($i\%{f_{a}}$) equals to 0 then

9: $\bm{W}_{\\!S}^{w\\!\times\\!D}\leftarrow$
$\bm{W}_{\\!S}^{w\\!\times\\!D}\odot(\bm{M}-\bm{E})$;

10: $\bm{W}_{\\!S}^{w\\!\times\\!D}\leftarrow$
$\bm{W}_{\\!S}^{w\\!\times\\!D}\odot(\bm{M}+\bm{G})$;

11: end if

12: if ($i\%{f_{s}}$) equals to 0 then

13:
$\bm{W}_{\\!S}^{w\\!\times\\!D}\leftarrow$SortChannels$(\bm{W}_{\\!S}^{w\\!\times\\!D})$;

14: end if

15: if validation then

16:
$\bm{W}_{\\!S}^{w\\!\times\\!D}\leftarrow$SortChannels$(\bm{W}_{\\!S}^{w\\!\times\\!D})$;

17:
$\bm{W}_{\\!S}^{D}\leftarrow$SelectChannels$(\bm{W}_{\\!S}^{w\\!\times\\!D})$;

18: NormalValidation$(\bm{W}_{\\!S}^{D})$;

19: end if

20: end for

  * •

Segmentation Network on SemanticKITTI [25]. LSK- 3DNet (See Fig. 4 (1))
divides the entire scene into voxels, each with a size of 5 cm.${}_{\\!}$ It
has four scales of {2, 4, 8, 16}.${}_{\\!}$ The hiden size $D$ of the entire
network is 64. We deploy LSK Blocks in
SparseBasicBlock111https://github.com/yanx27/2DPASS. Following [86, 15, 3], we
employ the weighted cross-entropy loss to optimize point accuracy and utilize
the lovasz-softmax [87] loss to maximize the intersection-over-union.

  * •

Segmentation Network on ScanNet v2 [27]. This segmentation network has the
same architecture as that on SemanticKITTI [25]. The entire scene is split
with a voxel size of 2 cm, with scales of {2, 4, 8, 16, 16}. The hidden size
$D$ is 128. Following [88], we take the weighted cross-entropy loss as the
objective function.

Detection Network on KITTI. We take Voxel R-CNN [5] as a baseline.
Specifically, we retain the backbone of Voxel R-CNN and substitute plain
sparse convolutional block with LSK Block in the first three stages. Other
settings remain the same as [18].${}_{\\!}$ The input spatial shape is {1600,
1408, 41}, and the channels of stages are {16, 16, 32, 64, 64}.

##  4 Experiment

###  4.1 Setups and Implementations

Dataset. To evaluate our method, we perform experiments

on three benchmark datasets and five tracks:

  * •

SemanticKITTI [25] has $43,\\!551$ traffic point cloud scenes with fine
annotations, split into $19,\\!130/4,\\!071/20,\\!350$ scenes for
train/val/test. The dataset has $28$ semantic classes, but only $19$ classes
are used for single-scan track and $25$ classes for multi-scan track.

  * •

ScanNet v2 [27] contains $1,\\!201/312/100$ indoor scenes for train, val, and
test splits, respectively. There are $20$ semantic categories for ScanNet20
track and $200$ categories for ScanNet200 track.

  * •

KITTI [28] has $7,\\!481/7,\\!518$ samples for train and test.${}_{\\!}$ We
follow the frequently used train/val split [89] to divide the training samples
into train split ($3,\\!712$ samples) and val split ($3,\\!769$ samples).

Table 2: Results of LSK3DNet on SemanticKITTI [25] multi-scan test set. The
arrow below classes indicate moving classes (Sec. 4.2).

Method |  Input |  mIoU |  Acc |  $\underrightarrow{\text{car}}$ |  $\underrightarrow{\text{truck}}$ |  $\underrightarrow{\text{other-vehicle }}$ |  $\underrightarrow{\text{person}}$ |  $\underrightarrow{\text{bicyclist}}$ |  $\underrightarrow{\text{motorcyclist}}$
---|---|---|---|---|---|---|---|---|---
LatticeNet [90] | P | 45.2 | 89.3 | 54.8 | 3.5 | 0.6 | 49.9 | 44.6 | 64.3
TLSeg [91] | R | 47.0 | 89.6 | 68.2 | 2.1 | 12.4 | 40.4 | 42.8 | 12.9
KPConv [42] | P | 51.2 | 89.3 | 69.4 | 5.8 | 4.7 | 67.5 | 67.4 | 47.2
Cylinder3D [15] | V | 52.5 | 91.0 | 74.9 | 0.0 | 0.1 | 65.7 | 68.3 | 11.9
(AF)2-S3Net [85] | V | 56.9 | 88.1 | 65.3 | 5.6 | 3.9 | 67.6 | 66.4 | 59.6
2DPASS [3] | PV | 62.4 | 91.4 | 82.1 | 16.1 | 3.8 | 80.3 | 71.2 | 73.1
LSK3DNet | PV | 63.4 | 92.2 | 84.4 | 7.2 | 40.9 | 77.4 | 69.9 | 72.1

Training and Testing Details. We utilize AdamW optimizer with OneCycleLR
scheduler starting with a learning rate of 5e-3. Data augmentation is also
employed, such as random flipping, scaling, rotation around the gravity axis,
spatial translation. We apply instance CutMix [23] and Test Time Augmentation
(TTA) [3] to the SemanticKITTI test benchmark, and enhance the model with
extra training epochs.

Metrics. We employ mean class intersection over union (mIoU) and overall
accuracy (Acc) metrics as the evaluation criterion for 3D semantic
segmentation tasks, as outlined in [25]. What is more, we calculate Average
Precision (AP) by recalling 11 positions for 3D object detection.

Table 3: Results of LSK3DNet and other tate-of-the-art methods on ScanNet v2
[27] (Sec. 4.2). S-Net stands for ScanNet.

Method |  Input | S-Net20 | S-Net20 | S-Net200
---|---|---|---|---
Val | Test | Val
PointNet++ [41] | P | 53.5 | 55.7 | -
3DMV [92] | P | - | 48.4 | -
PanopticFusion [93] | P | - | 52.9 | -
PointCNN [94] | P | - | 45.8 | -
PointConv [95] | P | 61.0 | 66.6 | -
JointPointBased [96] | P | 69.2 | 63.4 | -
PointASNL [97] | P | 63.5 | 66.6 | -
SegGCN [98] | P | - | 58.9 | -
RandLA-Net [86] | P | - | 64.5 | -
KPConv [42] | P | 69.2 | 68.6 | -
JSENet [99] | P | - | 69.9 | -
FusionNet [100] | P | - | 68.8 | -
Point Transformer [8] | P | 70.6 | - | -
Fast Point Transformer [101] | P | 72.1 | - | -
Stratified Transformer [9] | P | 74.3 | 73.7 | -
Point Transformer v2 [88] | P | 75.4 | 75.2 | 31.9
SparseConvNet [11, 88] | V | 69.3 | 72.5 | 28.8
MinkUNet [12] | V | 72.2 | 73.6 | -
LargeKernel3D [18] | V | 73.5 | 73.9 | -
LSK3DNet | PV | 75.7 | 75.5 | 33.1

Table 4: Results of 3D object detection methods on the car class of KITTI val
set [28] (Sec. 4.3).

Method |  Input | 3D AP (IoU=0.7)
---|---|---
Easy | Moderate | Hard
VoxelNet [102] | V | 81.97 | 65.46 | 62.85
PointPillars [103] | R | 86.62 | 76.06 | 68.91
SECOND [63] | V | 88.61 | 78.62 | 77.22
Point R-CNN [104] | P | 88.88 | 78.63 | 77.38
Part-$A^{2}$ [105] | V | 89.47 | 79.47 | 78.54
PV-RCNN [4] | PV | 89.35 | 83.69 | 78.70
Focals Conv [64] | V | 89.52 | 84.93 | 79.18
Voxel R-CNN [5] | V | 89.41 | 84.52 | 78.93
LargeKernel3D [18] | V | 89.52 | 85.07 | 79.32
LSK3DNet | V | 90.16 | 85.61 | 79.53

###  4.2 3D Semantic Segmentation

Results on SemanticKITTI. We test LSK3DNet on both single-scan and multi-scan
tracks [25].${}_{\\!}$ Tab. 1 presents the quantitative results of
SemanticKITTI single-scan track. LSK3DNet outperforms 2DPASS [3] in terms of
both mIoU and IoU scores for most categories.${}_{\\!}$ Moreover, SDS reduces
the model size of naive 3D kernels and CWS keeps the model size within the
desired size. So, LSK3DNet has a faster running speed than most prior methods,
but it’s performance significantly benefits from the large kernels.${}_{\\!}$
Tab. 2 shows the results of SemanticKITTI multi-scan track. The mIoU and Acc
are computed over all 25 classes.${}_{\\!}$ Due to page limitations, we only
report the per-class IoUs for moving categories. Under this challenging
setting, LSK3DNet surpasses 2DPASS [3] in both mIoU and Acc.${}_{\\!}$
Notably, LSK3DNet achieves state-of-the-art performance on both single-scan
and multi-scan tracks in SemanticKITTI.${}_{\\!}$ Surpassing over the SOTA
2D-3D method 2DPASS is more valuable, which utilizes both 2D and 3D data while
our LSK3DNet takes as input only the 3D data.${}_{\\!}$ Visualization results
are in the supplementary materials.

Results on ScanNet V2. We compare LSK3DNet with previous state-of-the-art
methods on ScanNet v2 [27], a large-scale dataset for 3D indoor scene
segmentation. ScanNet v2 has two tracks: ScanNet20 and ScanNet200, which use
20 and 200 semantic classes respectively. Tab. 3 presents the quantitative
results of our model and other methods on both tracks. LSK3DNet achieves
higher performance than previous methods in both tracks.${}_{\\!}$ Our
performance is superior to transformer-based methods [8, 101, 9, 88],
including Point Transformer v2. Moreover, LSK3DNet has a clear advantage over
LargeKernel3D [18], improving the mIoU by 2.2% and 1.6% on the ScanNet v2 val
and test set, respectively.

###  4.3 3D Object Detection

We have also evaluated the detection performance of LSK- 3DNet on the val
split for car category, as shown in Tab. 4.${}_{\\!}$ We report the average
precision metric for a 3D bounding box with 11 recall positions (Recall 11).
Based on Voxel R-CNN [5], we showcase the effectiveness of the LSK Block by
comparing it with LargeKernel3D [18], a previous 3D large-kernel method.
LSK3DNet achieves better results in three difficulty levels, compared with [5,
18]. Visualization comparison can be found in the supplementary material.

###  4.4 Ablation Studies

We first explore the effect of kernel size on performance, and then the
overall architecture design. Next, we explain the hyperparameter choices of
SDS and CWS.${}_{\\!}$ All ablation experiments are performed on the val set
of SemancKITTI.

Kernel Size. We use Modified SPVCNN as the baseline and then explore 3D kernel
sizes under two settings: naive dense large kernel and our LSK3DNet. In the
former setting, the dense 3D kernel is straightforwardly expanded. In the
latter one, LSK3DNet is trained simultaneously with SDS and CWS, meaning that
our LSK3DNet can learn a large sparse kernel model from the beginning.

The baseline does not use aggressive downsampling which is common in most
U-type 3D networks [11, 12]. However, submanifold sparse convolution with
small kernels may lose important information flow, especially for the
spatially disconnected features. We solve this problem by enlarging the size
of the 3D convolution kernel to obtain a large receptive field. This agrees
with the performance improvement in Tab. 5, when the kernel size increases.
In contrast to this observation, LargeKernel3D [18] shows the opposite trends;
we empirically attribute this to the fact that LargeKernel3D is based on
U-type 3D networks (Sec. 3.5). For a better understanding of Effective
Receptive Fields (ERFs) size, Fig. 5 illustrates the comparison between the
Baseline and LSK3DNet. We follow the definition of ERFs [106], and measure the
gradient of input point data to the central point of the features generated in
the last layer. Compared to the baseline, the high-contribution points of
LSK3DNet are distributed over a larger input range, indicating a larger ERF.
Additionally, to enhance comprehension of the learned kernels, we offer kernel
visualization in Fig. 5, providing insights into sparse training.

Baseline

LSK3DNet

Kernel visualization

Figure 5: Effective Receptive Fields (ERFs) of Baseline and LSK- 3DNet.
LSK3DNet has a larger ERF size. Additionally, we provide visualization of
learned sparse kernels, where all weight values have taken the absolute value
and normalization operations. The black areas indicate positions with weight
values of zero.

Compared to the “naive dense large kernel” of each kernel size, LSK3DNet
exhibits superior performance in its corresponding size. LSK3DNet not only
leverages SDS to expand its receptive field but also starts from scratch to
learn 3D sparse kernels. Concurrently, CWS widens the network during training
while pruning redundant channels to accelerate inference. Furthermore, SDS and
CWS address the issues of overparameterization and overfitting that arise from
simply increasing the kernel’s size and expanding the model’s width. The best
performance of LSK3DNet is achieved at the $9\\!\times\\!9\\!\times\\!9$ size,
and the following experiments are conducted with $9\\!\times\\!9\\!\times\\!9$
size. Moreover, due to the extra dimension in 3D kernels compared to 2D
kernels, naively enlarging the 3D kernel size will lead to a cubically-
increasing overhead. Our LSK3DNet can effectively reduce 19.1M parameters and
approximately 60% of computing operations when compared to the naive large
kernel network. This aspect significantly enhances the value of SDS and CWS
for 3D kernels.

Table 5: Segmentation performance of Modified SPVCNN with various large kernel
settings. “Dense” refers to directly enlarging the kernel size. “Ours” refers
to training with SDS and CWS (Sec. 4.4). The unit for “Param” is in million
(M), “FLOPs” is in billion (G), and “Speed” is measured in milliseconds per
scene.

| Kernel Size | mIoU | Param | FLOPs |  Speed $\downarrow$
---|---|---|---|---|---
Dense ($D=64$) | $3,3,3$ | 65.2 | 1.9 | 78.4 | 36
$5,5,5$ | 65.6 | 8.4 | 127.8 | 57
$7,7,7$ | 66.2 | 22.6 | 381.2 | 65
$9,9,9$ | 67.5 | 47.9 | 1916.3 | 93
$11,11,11$ | 66.6 | 87.4 | 3192.0 | 134
Dense ($1.8\\!\times\\!D$) | $5,5,5$ | 66.2 | 27.0 | 444.0 | 84
$7,7,7$ | 66.5 | 73.1 | 1551.2 | 123
$9,9,9$ | 67.2 | 154.7 | 3030.0 | 177
$11,11,11$ | 66.3 | 282.1 | 7139.2 | 250
Ours ($1.8\\!\times\\!D$) | $5,5,5$ | 66.8 | 5.1 | 203.8 | 46
$7,7,7$ | 68.1 | 13.6 | 249.4 | 63
$\bf{9,9,9}$ | 70.2 | 28.8 | 763.6 | 89
$11,11,11$ | 70.1 | 52.5 | 1847.4 | 116

Table 6: Overall architecture design with $9\\!\times\\!9\\!\times\\!9$ size.
Here, we report Param, FLOPs, and Speed for the inference model (Sec. 4.4).

Method | mIoU | Param | FLOPs |  Speed $\downarrow$
---|---|---|---|---
Dense ($1.0\\!\times\\!D$) | 67.5 | 47.9 | 1916.3 | 93
Dense ($1.8\\!\times\\!D$) | 67.2 | 154.7 | 3030.0 | 177
SDS ($1.8\\!\times\\!D$) | 69.3 | 93.0 | 2201.4 | 173
CWS ($1.8\\!\times\\!D$) | 69.1 | 47.9 | 1916.3 | 93
LSK3DNet ($\bf{1.8\\!\times\\!D}$) | 70.2 | 28.8 | 763.6 | 89

Overall Architecture Design. In Tab. 6, Dense ($1.8\\!\times\\!D$) does not
yield improvement due to overparameterization and overfitting. Although SDS
maintains a model width of $1.8\\!\times\\!D$ during inference, it achieves
improved performance due to sparse training and the large receptive field. CWS
effectively addresses overfitting by selecting salient channels, thereby
enhancing performance. By combining SDS and CWS, LSK3DNet can achieve the best
performance while significantly reducing the model size and computational
cost.

Spatial-wise Dynamic Sparsity. To control the degree of sparse weight
adaptation in LSK3DNet, we have adjusted three hyperparameters: the adaptation
frequency $f_{a}$, sparsity rate $s$, and prune rate $p$ (Sec. 3.3).
Adaptation frequency $f_{a}$ indicates how often we update the sparse weights
during training. Sparsity rate $s$ specifies how sparse the 3D large kernels
are.${}_{\\!}$ Prune rate $p$ shows the fraction of updated weights in one
adaptation. We have conducted experiments with different values of these
hyperparameters and report the results in Tab. 7 and Tab. 8. Based on our
empirical findings, we have chosen $f_{a}\\!=\\!2000$, $s\\!=\\!0.4$, and
$p\\!=\\!0.3$ as the optimal settings for LSK3DNet.

Table 7: Training and inference speed analysis, speed is measured in
milliseconds per scene. The kernels have $9\\!\times\\!9\\!\times\\!9$ size,
“T” and “I” represent training and inference speed, respectively (Sec. 4.4).

Sparsity |  Dense ($1.0\\!\times\\!D$) |  SDS ($1.8\\!\times\\!D$) |  LSK3DNet ($1.8\\!\times\\!D$)
---|---|---|---
s | mIoU |  T $\downarrow$ |  I $\downarrow$ | mIoU |  T $\downarrow$ |  I $\downarrow$ | mIoU |  T $\downarrow$ |  I $\downarrow$
0 | 67.5 | 451 | 93 | - | - | - | - | - | -
0.1 | - | - | - | 68.5 | 489 | 176 | 69.1 | 491 | 92
0.2 | - | - | - | 68.9 | 478 | 176 | 69.7 | 482 | 91
0.4 | - | - | - | 69.3 | 463 | 173 | 70.2 | 467 | 89
0.6 | - | - | - | 69.0 | 450 | 171 | 68.8 | 454 | 87
0.8 | - | - | - | 68.7 | 439 | 171 | 67.2 | 443 | 87

Table 8: ${}_{\\!}$Ablation${}_{\\!}$ studies${}_{\\!}$ of${}_{\\!}$ Spatial-
wise Dynamic Sparsity on adaptation frequency $f_{a}$ and prune rate $p$
(Sec. 4.4).

Adaptation $f_{a}$ | mIoU
---|---
100 | 68.8
1000 | 69.6
2000 | 70.2
3000 | 70.0
4000 | 69.8

Pruning $p$ | mIoU
---|---
0.10 | 69.9
0.20 | 70.1
0.30 | 70.2
0.50 | 70.0
0.70 | 69.7

Channel-wise Weight Selection. Another aspect of our method that we
investigated is the CWS. This technique involves two hyperparameters: the
sorting frequency $f_{s}$ and the width factor $w$ (Sec. 3.4). The former
determines how frequently we sort and select channels for large kernels during
training.${}_{\\!}$ It is expressed as a multiple of $f_{a}$, the adaptation
frequency, meaning that we perform channel sorting after a certain number of
weight adaptation cycles. When $f_{s}\\!=\\!6\\!\times\\!f_{a}$, the optimal
performance is achieved. The latter controls the times of network width
compared to the target model. We find that increasing the width factor up to
$1.8\times$ improves the model’s performance, but beyond that point, there is
no significant gain. Therefore, we opt for a width of $1.8\times$ to avoid
more extra compute resources that a wider width would require during training.

Table 9: ${}_{\\!}$Ablation${}_{\\!}$ studies${}_{\\!}$ of${}_{\\!}$ Channel-
wise Weight Selection on sorting frequency $f_{s}$ and width factor $w$ (Sec.
4.4).

Sorting $f_{s}$ | mIoU
---|---
$1\times f_{a}$ | 68.2
$2\times f_{a}$ | 69.1
$4\times f_{a}$ | 69.8
$\bf{6\times f_{a}}$ | 70.2
$10\times f_{a}$ | 69.9

Width $w$ | mIoU
---|---
$1.1\times$ | 68.4
$1.5\times$ | 69.6
$\bf{1.8\times}$ | 70.2
$2.1\times$ | 69.9
$2.5\times$ | 70.1

##  5 Conclusion

We propose Spatial-wise Dynamic Sparsity to scale up 3D kernels beyond
$9\\!\times\\!9\\!\times\\!9$, which prunes the volumetric weight and reduces
the parameter size of large kernel layers. Our LSK3DNet can benefit from a
large receptive field without increasing the computational cost compared to a
naive 3D large kernel. Channel-wise Weight Selection expands the model width
during training, and then sorts and selects important channels during
validation to get a model of the expected size. In this way, we achieve “using
spatial sparse groups, expanding width without more parameters”. We evaluate
our method on the SemanticKITTI and achieve state-of-the-art performance. Our
LSK3DNet also surpasses previous 3D large kernel methods on ScanNet v2 and
KITTI.

## References

  * Zhou et al. [2021] Haoran Zhou, Yidan Feng, Mingsheng Fang, Mingqiang Wei, Jing Qin, and Tong Lu.  Adaptive graph convolution for point cloud analysis.  In _ICCV_ , 2021.
  * Feng et al. [2023] Tuo Feng, Wenguan Wang, Xiaohan Wang, Yi Yang, and Qinghua Zheng.  Clustering based point cloud representation learning for 3d analysis.  In _ICCV_ , 2023.
  * Yan et al. [2022] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li.  2dpass: 2d priors assisted semantic segmentation on lidar point clouds.  In _ECCV_ , 2022.
  * Shi et al. [2020] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In _CVPR_ , 2020.
  * Deng et al. [2021] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li.  Voxel r-cnn: Towards high performance voxel-based 3d object detection.  In _AAAI_ , 2021.
  * Yin et al. [2024] Junbo Yin, Jianbing Shen, Runnan Chen, Wei Li, Ruigang Yang, Pascal Frossard, and Wenguan Wang.  Is-fusion: Instance-scene collaborative fusion for multimodal 3d object detection.  In _CVPR_ , 2024.
  * Feng et al. [2024] Tuo Feng, Ruijie Quan, Xiaohan Wang, Wenguan Wang, and Yi Yang.  Interpretable3d: An ad-hoc interpretable classifier for 3d point clouds.  In _AAAI_ , 2024.
  * Zhao et al. [2021] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun.  Point transformer.  In _ICCV_ , 2021.
  * Lai et al. [2022] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia.  Stratified transformer for 3d point cloud segmentation.  In _CVPR_ , 2022.
  * Hu et al. [2020] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In _CVPR_ , 2020.
  * Graham et al. [2018] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In _CVPR_ , 2018.
  * Choy et al. [2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In _CVPR_ , 2019.
  * Yin et al. [2022a] Junbo Yin, Jin Fang, Dingfu Zhou, Liangjun Zhang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang.  Semi-supervised 3d object detection with proficient teachers.  In _ECCV_ , 2022a.
  * Yin et al. [2022b] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang.  Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection.  In _ECCV_ , 2022b.
  * Zhu et al. [2021] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin.  Cylindrical and asymmetrical 3d convolution networks for lidar segmentation.  In _CVPR_ , 2021.
  * Tang et al. [2020] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han.  Searching efficient 3d architectures with sparse point-voxel convolution.  In _ECCV_ , 2020.
  * Yin et al. [2021] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In _CVPR_ , 2021.
  * Chen et al. [2023] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia.  Scaling up kernels in 3d cnns.  _CVPR_ , 2023.
  * Hou et al. [2022] Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li.  Point-to-voxel knowledge distillation for lidar semantic segmentation.  In _CVPR_ , 2022.
  * Ding et al. [2022] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding.  Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.  In _CVPR_ , 2022.
  * Liu et al. [2022] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang.  More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity.  _arXiv preprint arXiv:2207.03620_ , 2022.
  * Yan et al. [2021] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui.  Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion.  In _AAAI_ , 2021.
  * Xu et al. [2021] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu.  Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation.  In _ICCV_ , 2021.
  * Lai et al. [2023] Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, and Jiaya Jia.  Spherical transformer for lidar-based 3d recognition.  In _CVPR_ , 2023.
  * Behley et al. [2019] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall.  Semantickitti: A dataset for semantic scene understanding of lidar sequences.  In _ICCV_ , 2019.
  * Kamath and Renuka [2023] Vidya Kamath and A Renuka.  Deep learning based object detection for resource constrained devices-systematic review, future trends and challenges ahead.  _Neurocomputing_ , 2023.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , 2017.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Hu et al. [2019] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In _ICCV_ , 2019.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , 2016.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _AAAI_ , 2017.
  * Peng et al. [2017] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun.  Large kernel matters–improve semantic segmentation by global convolutional network.  In _CVPR_ , 2017.
  * Wang et al. [2022] Wenguan Wang, Guolei Sun, and Luc Van Gool.  Looking beyond single images for weakly supervised semantic segmentation learning.  _IEEE TPAMI_ , 2022.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _CVPR_ , 2017.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _CVPR_ , 2017.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Qi et al. [2017a] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  Pointnet: Deep learning on point sets for 3D classification and segmentation.  In _CVPR_ , 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In _NeurIPS_ , 2017b.
  * Thomas et al. [2019] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In _ICCV_ , 2019.
  * Xu et al. [2021] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi.  Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds.  In _CVPR_ , 2021.
  * Fan et al. [2021] Siqi Fan, Qiulei Dong, Fenghua Zhu, Yisheng Lv, Peijun Ye, and Fei-Yue Wang.  Scf-net: Learning spatial contextual features for large-scale point cloud segmentation.  In _CVPR_ , 2021.
  * Ran et al. [2021] Haoxi Ran, Wei Zhuo, Jun Liu, and Li Lu.  Learning inner-group relations on point clouds.  In _ICCV_ , 2021.
  * Xiang et al. [2021] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai.  Walk in the cloud: Learning curves for point clouds shape analysis.  In _ICCV_ , 2021.
  * Liu et al. [2019] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan.  Relation-shape convolutional neural network for point cloud analysis.  In _CVPR_ , 2019.
  * Xu et al. [2020] Qiangeng Xu, Xudong Sun, Cho-Ying Wu, Panqu Wang, and Ulrich Neumann.  Grid-gcn for fast and scalable point cloud learning.  In _CVPR_ , 2020.
  * Wang et al. [2019] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan.  Graph attention convolution for point cloud semantic segmentation.  In _CVPR_ , 2019.
  * Lian et al. [2019] Yanchao Lian, Tuo Feng, and Jinliu Zhou.  A dense pointnet++ architecture for 3d point cloud semantic segmentation.  In _IGARSS_ , 2019.
  * Qian et al. [2021] Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem.  Pu-gcn: Point cloud upsampling using graph convolutional networks.  In _CVPR_ , 2021.
  * Fan et al. [2021] Hehe Fan, Xin Yu, Yi Yang, and Mohan Kankanhalli.  Deep hierarchical representation of point cloud videos via spatio-temporal decomposition.  _IEEE TPAMI_ , 44(12):9918–9930, 2021.
  * Fan et al. [2022] Hehe Fan, Yi Yang, and Mohan Kankanhalli.  Point spatio-temporal transformer networks for point cloud video modeling.  _IEEE TPAMI_ , 45(2):2181–2192, 2022.
  * Fan et al. [2020] Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang, and Mohan Kankanhalli.  Pstnet: Point spatio-temporal convolution on point cloud sequences.  In _ICLR_ , 2020.
  * Fan et al. [2021] Hehe Fan, Yi Yang, and Mohan Kankanhalli.  Point 4d transformer networks for spatio-temporal modeling in point cloud videos.  In _CVPR_ , 2021.
  * Hu et al. [2021] Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham.  Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges.  In _CVPR_ , 2021.
  * Tatarchenko et al. [2018] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou.  Tangent convolutions for dense prediction in 3d.  In _CVPR_ , 2018.
  * Wu et al. [2018] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer.  Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud.  In _ICRA_ , 2018.
  * Wu et al. [2019] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer.  Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud.  In _ICRA_ , 2019.
  * Cortinhal et al. [2020] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy.  Salsanext: fast, uncertainty-aware semantic segmentation of lidar point clouds for autonomous driving.  _arXiv preprint arXiv:2003.03653_ , 2020.
  * Xu et al. [2020] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.  Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation.  In _ECCV_ , 2020.
  * Zhang et al. [2020] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh.  Polarnet: An improved grid representation for online lidar point clouds semantic segmentation.  In _CVPR_ , 2020.
  * Yan et al. [2018] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  _Sensors_ , 18(10):3337, 2018.
  * Chen et al. [2022] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia.  Focal sparse convolutional networks for 3d object detection.  In _CVPR_ , 2022.
  * Bellec et al. [2017] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein.  Deep rewiring: Training very sparse deep networks.  _arXiv preprint arXiv:1711.05136_ , 2017.
  * Mocanu et al. [2018] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta.  Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.  _Nature communications_ , 9(1):2383, 2018.
  * Dettmers and Zettlemoyer [2019] Tim Dettmers and Luke Zettlemoyer.  Sparse networks from scratch: Faster training without losing performance.  _arXiv preprint arXiv:1907.04840_ , 2019.
  * Liu et al. [2021] Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy.  Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware.  _Neural Computing and Applications_ , 33:2589–2604, 2021\.
  * Evci et al. [2020] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.  Rigging the lottery: Making all tickets winners.  In _ICML_ , 2020.
  * Mostafa and Wang [2019] Hesham Mostafa and Xin Wang.  Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization.  In _ICML_ , 2019.
  * Jayakumar et al. [2020] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen.  Top-kast: Top-k always sparse training.  In _NeurIPS_ , 2020.
  * Chen et al. [2021] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang.  Chasing sparsity in vision transformers: An end-to-end exploration.  In _NeurIPS_ , 2021.
  * Liu et al. [2021] Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy.  Selfish sparse rnn training.  In _ICML_ , 2021.
  * Han et al. [2016] Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, et al.  Dsd: Dense-sparse-dense training for deep neural networks.  In _ICLR_ , 2016.
  * Mocanu et al. [2017] Decebal Constantin Mocanu et al.  _Network computations in artificial intelligence_.  Technische Universiteit Eindhoven, 2017.
  * Dai et al. [2019a] Xiaoliang Dai, Hongxu Yin, and Niraj K Jha.  Nest: A neural network synthesis tool based on a grow-and-prune paradigm.  _IEEE Transactions on Computers_ , 68(10):1487–1497, 2019a.
  * Dai et al. [2019b] Xiaoliang Dai, Hongxu Yin, and Niraj K Jha.  Grow and prune compact, fast, and accurate lstms.  _IEEE Transactions on Computers_ , 69(3):441–452, 2019b.
  * Raihan and Aamodt [2020] Md Aamir Raihan and Tor Aamodt.  Sparse weight activation training.  In _NeurIPS_ , 2020.
  * Lee et al. [2018] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr.  Snip: Single-shot network pruning based on connection sensitivity.  _arXiv preprint arXiv:1810.02340_ , 2018.
  * Liu et al. [2022] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy.  The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.  In _ICLR_ , 2022.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _ICCV_ , 2015.
  * Diering et al. [2017] Graham H Diering, Raja S Nirujogi, Richard H Roth, Paul F Worley, Akhilesh Pandey, and Richard L Huganir.  Homer1a drives homeostatic scaling-down of excitatory synapses during sleep.  _Science_ , 355(6324):511–515, 2017.
  * De Vivo et al. [2017] Luisa De Vivo, Michele Bellesi, William Marshall, Eric A Bushong, Mark H Ellisman, Giulio Tononi, and Chiara Cirelli.  Ultrastructural evidence for synaptic scaling across the wake/sleep cycle.  _Science_ , 355(6324):507–510, 2017.
  * Cai et al. [2022] Han Cai, Chuang Gan, Ji Lin, and Song Han.  Network augmentation for tiny deep learning.  In _ICLR_ , 2022.
  * Cheng et al. [2021] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu.  (af)2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network.  In _CVPR_ , 2021.
  * Hu et al. [2020] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In _CVPR_ , 2020.
  * Berman et al. [2018] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko.  The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.  In _CVPR_ , 2018.
  * Wu et al. [2022] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao.  Point transformer v2: Grouped vector attention and partition-based pooling.  In _NeurIPS_ , 2022.
  * Meng et al. [2020] Qinghao Meng, Wenguan Wang, Tianfei Zhou, Jianbing Shen, Luc Van Gool, and Dengxin Dai.  Weakly supervised 3d object detection from lidar point cloud.  In _ECCV_ , 2020.
  * Rosu et al. [2019] Radu Alexandru Rosu, Peer Schütt, Jan Quenzel, and Sven Behnke.  Latticenet: Fast point cloud segmentation using permutohedral lattices.  _arXiv preprint arXiv:1912.05905_ , 2019.
  * Duerr et al. [2020] Fabian Duerr, Mario Pfaller, Hendrik Weigel, and Jürgen Beyerer.  Lidar-based recurrent 3d semantic segmentation with temporal memory alignment.  In _3DV_ , 2020.
  * Dai and Nießner [2018] Angela Dai and Matthias Nießner.  3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation.  In _ECCV_ , 2018.
  * Narita et al. [2019] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji.  Panopticfusion: Online volumetric semantic mapping at the level of stuff and things.  In _IROS_ , 2019.
  * Li et al. [2018] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  Pointcnn: Convolution on x-transformed points.  In _NeurIPS_ , 2018.
  * Wu et al. [2019] Wenxuan Wu, Zhongang Qi, and Li Fuxin.  Pointconv: Deep convolutional networks on 3d point clouds.  In _CVPR_ , 2019.
  * Chiang et al. [2019] Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, and Winston H Hsu.  A unified point-based framework for 3d segmentation.  In _3DV_ , 2019.
  * Yan et al. [2020] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui.  Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling.  In _CVPR_ , 2020.
  * Lei et al. [2020] Huan Lei, Naveed Akhtar, and Ajmal Mian.  Seggcn: Efficient 3d point cloud segmentation with fuzzy spherical kernel.  In _CVPR_ , 2020.
  * Hu et al. [2020] Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai.  Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds.  In _ECCV_ , 2020.
  * Zhang et al. [2020] Feihu Zhang, Jin Fang, Benjamin Wah, and Philip Torr.  Deep fusionnet for point cloud semantic segmentation.  In _ECCV_ , 2020.
  * Park et al. [2022] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park.  Fast point transformer.  In _CVPR_ , 2022.
  * Qi et al. [2019] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In _ICCV_ , 2019.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _CVPR_ , 2019.
  * Shi et al. [2019] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In _CVPR_ , 2019.
  * Shi et al. [2020] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network.  _IEEE TPAMI_ , 43(8):2647–2664, 2020.
  * Luo et al. [2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In _NeurIPS_ , 2016.
  * Lu et al. [2023] Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang.  Link: Linear kernel for lidar-based 3d perception.  In _CVPR_ , 2023.
  * Ma et al. [2022] Tianyu Ma, Alan Q Wang, Adrian V Dalca, and Mert R Sabuncu.  Hyper-convolutions via implicit kernels for medical imaging.  _arXiv preprint arXiv:2202.02701_ , 2022.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _CVPR_ , 2020.
  * Genova et al. [2021] Kyle Genova, Xiaoqi Yin, Abhijit Kundu, Caroline Pantofaru, Forrester Cole, Avneesh Sud, Brian Brewington, Brian Shucker, and Thomas Funkhouser.  Learning 3d semantic segmentation with only 2d image supervision.  In _3DV_ , 2021.
  * Lebedev and Lempitsky [2016] Vadim Lebedev and Victor Lempitsky.  Fast convnets using group-wise brain damage.  In _CVPR_ , 2016.
  * Changpinyo et al. [2017] Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov.  The power of sparsity in convolutional neural networks.(2017).  _arXiv preprint cs.CV/1702.06257_ , 2017.
  * Gale et al. [2020] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen.  Sparse gpu kernels for deep learning.  In _SC20_ , 2020.
  * Choquette et al. [2021] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky.  Nvidia a100 tensor core gpu: Performance and innovation.  _IEEE Micro_ , 41(2):29–35, 2021.
  * Milioto et al. [2019] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss.  Rangenet++: Fast and accurate lidar semantic segmentation.  In _IROS_ , 2019.
  * Liong et al. [2020] Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong.  Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation.  _arXiv preprint arXiv:2012.04934_ , 2020.
  * Shi et al. [2020] Hanyu Shi, Guosheng Lin, Hao Wang, Tzu-Yi Hung, and Zhenhua Wang.  Spsequencenet: Semantic segmentation network on 4d point clouds.  In _CVPR_ , 2020.

Supplementary Material

To enhance the clarity of the main paper, we include the following sections in
this supplementary material. We first introduce the analysis and comparison
with previous methods in Sec. A. Ablation studies on spatial group size,
Effective Receptive Fields (ERFs), inference speed and FLOPs, and
Convolutional Schemes are elaborated in Sec. B. More qualitative and
quantitative results are further presented and analyzed in Sec. C. Finally,
limitations and societal impact are discussed in Sec. D.

##  Appendix A Comparison and Discussion

Comparison with LinK [107] in $s$ and $r$ Settings. Block Based Aggregation in
LinK [107] divides the entire input space into several non-overlapping blocks.
Then, it utilizes block-wise proxy aggregation to extract the corresponding
features of proxy nodes.${}_{\\!}$ Furthermore, the corresponding features of
the center are obtained by aggregating the features of the neighboring blocks
closest to the center. In LinK, the variable $s$ represents the scale of the
block, while $r$ denotes the scale of the block query. Actually, this paradigm
is not novel; for example, a similar paradigm has already been employed in
2DPASS [3]. It first divides the entire input space into non-overlapping
voxels (blocks) at four scales: 2, 4, 8, and 16. The features of these voxels
are obtained by averaging the features within each voxel. Then, 3D sparse
convolution is applied to extract features from adjacent voxels.${}_{\\!}$
Here, the kernel size of sparse convolution plays the same role as
$r$.${}_{\\!}$ In addition, we also use $s$ to represent the above four
scales.${}_{\\!}$ In Tab. S1, we have fairly compared the receptive field
sizes of each module for three methods, based on the default hayaparameters of
models on the SemanticKITTI validation dataset [25].${}_{\\!}$ Specifically,
three modules are ELKBlock222https://github.com/MCG-NJU/LinK in Link,
SPVBlock333https://github.com/yanx27/2DPASS in 2DPASS, and LSK Block in our
LSK3DNet. LinK achieves an equivalent receptive field size of $21\times
21\times 21$ for the four ELKBlocks in LinK. However, the second, third, and
fourth LSK Blocks in LSK3DNet clearly exhibit larger equivalent receptive
fields.

Table S1: Receptive field size. RF represents the receptive field size, with
subscripts denoting the index of the scale.

Size |  $s_{1}$/$r_{1}$ |  $RF_{1}$
---|---|---
LinK [107] |  3/7 |  $21\times 21\times 21$
2DPASS [3] |  2/3 |  $6\times 6\times 6$
LSK3DNet |  2/9 |  $18\times 18\times 18$
Size |  $s_{2}$/$r_{2}$ |  $RF_{2}$
LinK [107] |  3/7 |  $21\times 21\times 21$
2DPASS [3] |  4/3 |  $12\times 12\times 12$
LSK3DNet |  4/9 |  $36\times 36\times 36$
Size |  $s_{3}$/$r_{3}$ |  $RF_{3}$
LinK [107] |  3/7 |  $21\times 21\times 21$
2DPASS [3] |  8/3 |  $24\times 24\times 24$
LSK3DNet |  8/9 |  $72\times 72\times 72$
Size |  $s_{4}$/$r_{4}$ |  $RF_{4}$
LinK [107] |  3/7 |  $21\times 21\times 21$
2DPASS [3] |  16/3 |  $48\times 48\times 48$
LSK3DNet |  16/9 |  $144\times 144\times 144$

The scales mentioned in section 3.5 and the convolution kernel size have
similar spatial meanings to the $s$ and $r$ settings in LinK. Please note that
we only adopted kernel size (i.e., $r$ in LinK [107]) as a metric for large
kernel design in the main paper, following LargeKernel3D [18]. Please do not
confuse these two settings.

Performance of LinK. Upon comparing all the results reported in Link [107] and
Model Zoo2, we notice that the highest performance of 67.7% is achieved with a
$(2\times 3)^{3}$ receptive field size, surpassing other sizes such as
$(3\times 7)^{3}$ and $(3\times 5)^{3}$. Interestingly, the $(2\times 3)^{3}$
receptive field size appears noticeably smaller than $(3\times 7)^{3}$. It
turns out that Link [107] achieves its stronger performance with a smaller
receptive field.

Kernel Size of LargeKernel3D on ScanNet v2 [27]. The claim that “the
performance of LargeKernel3D [18] drops when scaling up the kernel size over
$7\\!\times\\!7\\!\times\\!7$” is concluded from Table S -
9444https://arxiv.org/pdf/2206.10555v1.pdf.

##  Appendix B Ablation Studies

Spatial Group Size. We report the results of the ablation study on the kernel
size selection in Tab. S2.${}_{\\!}$ kernel size means the absolute kernel
size. group means the number of groups to split kernels.${}_{\\!}$ We conduct
the experiment on LSK3DNet on the SemanticKITTI dataset, with the same
training hyper-parameters in the main paper. For $3\\!\times\\!3\\!\times\\!3$
group and $9\\!\times\\!9\\!\times\\!9$ kernel size, every group has {3,3,3}
divisions in each dimension. Moreover, $9\\!\times\\!9\\!\times\\!9$ is split
into $5\\!\times\\!5\\!\times\\!5$ with {2,2,1,2,2} divisions.${}_{\\!}$ For
$13\\!\times\\!13\\!\times\\!3$ kernel size, $3\\!\times\\!3\\!\times\\!3$
group has {4,5,4} divisions in 13-dimensional axis, and similarly, {2,3,3,3,2}
divisions for $5\\!\times\\!5\\!\times\\!3$ group, {2,2,2,1,2,2,2} divisions
for $7\\!\times\\!7\\!\times\\!3$ group. We finally chose the
$9\\!\times\\!9\\!\times\\!9$ kernel size and $3\\!\times\\!3\\!\times\\!3$
group based on our empirical results. We speculate that this is because each
group has a $3\\!\times\\!3\\!\times\\!3$ size, showing more parameter space
to explore than other group settings. Previous studies have shown that
exploring a large parameter space is important for dynamic sparse training
[71, 78, 73]. In addition, the kernel size of $13\\!\times\\!13\\!\times\\!3$
does not achieve better performance, being affected by overparameterization
and overfitting issues.

Table S2: Ablation studies on spatial group size.

kernel size |  Group |  mIoU (%)
---|---|---
$9\\!\times\\!9\\!\times\\!9$ |  $3\\!\times\\!3\\!\times\\!3$ |  70.2
$5\\!\times\\!5\\!\times\\!5$ |  69.1
$13\times 13\times 3$ |  $3\\!\times\\!3\\!\times\\!3$ |  69.9
$5\\!\times\\!5\\!\times\\!3$ |  69.5
$7\\!\times\\!7\\!\times\\!3$ |  68.9

Table S3: Ablation studies on Depth-wise Group Convolution and Spatial-wise
Group Convolution on ScanNet v2 [27] and SemanticKITTI [25].

Methods | Kernel | mIoU (%)
---|---|---
ScanNet v2
MinkowskiNet-14 [18] \+ depth-wise conv \+ spatial-wise conv \+ spatial-wise conv | $7\\!\times\\!7\\!\times\\!7$ | 68.6
$7\\!\times\\!7\\!\times\\!7$ | 56.4
$7\\!\times\\!7\\!\times\\!7\to 3\\!\times\\!3\\!\times\\!3$ | 69.7
$9\\!\times\\!9\\!\times\\!9\to 7\\!\times\\!7\\!\times\\!7$ | 69.6
MinkowskiNet-34 [18] \+ depth-wise conv \+ spatial-wise conv | $7\\!\times\\!7\\!\times\\!7$ | 68.6
$7\\!\times\\!7\\!\times\\!7$ | 68.7
$7\\!\times\\!7\\!\times\\!7\to 3\\!\times\\!3\\!\times\\!3$ | 73.2
Modified SPVCNN \+ depth-wise conv (SDS and CWS) \+ spatial-wise conv (SDS and CWS) | $9\\!\times\\!9\\!\times\\!9$ | 72.4
$9\\!\times\\!9\\!\times\\!9$ | 67.0
$9\\!\times\\!9\\!\times\\!9$ | 75.7
SemanticKITTI
Modified SPVCNN \+ depth-wise conv (SDS and CWS) \+ spatial-wise conv (SDS and CWS) | $9\\!\times\\!9\\!\times\\!9$ | 67.5
$9\\!\times\\!9\\!\times\\!9$ | 65.4
$9\\!\times\\!9\\!\times\\!9$ | 70.2

Inference Speed and FLOPs. As described in §3.3 and Sec. D, SDS involves
unstructured sparsity, whose GPU acceleration is still a challenging issue.
This is why SDS could in theory save computation, but in practice that doesn’t
happen (see previous studies of Dynamic Sparse Training [65, 66, 67, 68, 69,
70, 71, 72, 73] for related discussion). Thus, only small speed change (4ms)
is observed with SDS in Tab. 6. Our speed up mainly comes from CWS. CWS can
significantly reduce the model size at the channel level, leading to a primary
acceleration of 84 ms (177ms$\rightarrow$93ms). Moreover, for a similar
reason, we present the theoretical inference FLOPs of the models under
different settings.

CWS. Our CWS is of great importance for building 3D large kernels.${}_{\\!}$
Previous 2D large sparse kernels like [21] follows a regime of “use sparse
groups, expand more”; they attain better performance with enlarged network
width, inevitably causing increased model size. This becomes more pronounced
in 3D. For a naive dense kernel, lifting ($64$-$d$,
$9\\!\times\\!9\\!\times\\!9$) to ($128$-$d$, $9\\!\times\\!9\\!\times\\!9$)
will cause a significant (somewhat unacceptable) increase of parameter number:
47.94M $\rightarrow$ 191.69M. As shown in Tab. 7 of the main paper, LSK3DNet
(with CWS) successfully achieves better performance while using fewer
parameters.

Kernel Size. Due to the increased dimensionality, enlarging 3D kernel size
brings much heavier computational burden, compared with 2D kernel.${}_{\\!}$
As shown in Tab. 5 of the main paper, comparing Dense
($7\\!\times\\!7\\!\times\\!7$, $1\\!\times\\!D$) and Dense
($9\\!\times\\!9\\!\times\\!9$, $1\\!\times\\!D$), computational
burden${}_{\\!}$ has${}_{\\!}$ 5$\times$${}_{\\!}$ rise:${}_{\\!}$ 381.2G
$\\!\rightarrow\\!$ 1916.3G.${}_{\\!}$ Though${}_{\\!}$ large${}_{\\!}$ kernel
gains larger receptive fields, it brings risk of overparameterization and
overfitting, and is hard to train. [108, 12, 18, 21] encounter a similar
issue, _e.g_., increasing $3\\!\times\\!3\\!\times\\!3$ to
$7\\!\times\\!7\\!\times\\!7$ leads to inferior results [18],
$31\\!\times\\!31$ performs worse than $7\\!\times\\!7$ [21].

Convolutional Schemes. The results of MinkowskiNet-14 and MinkowskiNet-34 are
directly taken from LargeKernel3D [18].${}_{\\!}$ MinkowskiNet-14 shows
stagnation when attempting to expand the convolution kernel to
$9\\!\times\\!9\\!\times\\!9$. As shown in Tab. S3, we use Modified SPVCNN as
the baseline and conduct experiments on ScanNet v2 [27] and SemanticKITTI
[25]. Our findings indicate that Spatial-wise Group Convolution is more
suitable for 3D large kernel designs, aligning with the results observed in
LargeKernel3D [18]. Therefore, we carry out dynamic sparse training within the
spatial-wise groups.

##  Appendix C More Qualitative and Quantitative Results

Quantitative Results on NuScenes. In Tab. S4, we present the results of
semantic segmentation on the nuScenes validation set [109].${}_{\\!}$ Our
approach consistently surpasses others by a significant margin, establishing
itself as the state-of-the-art (SOTA) performer on this benchmark.${}_{\\!}$
What’s particularly intriguing is that our method relies solely on LiDAR data,
yet it outperforms multi-modal techniques [110, 3], which incorporate
supplementary 2D information.

Concrete Results on SemanticKITTI Multi-scan Test. The full results of our
LSK3DNet on the SemanticKITTI [25] multi-scan test challenge are shown in Tab.
S5 and S6.${}_{\\!}$ Our method achieves a state-of-the-art performance on
both mIoU and Acc metrics, surpassing other methods in 12 out of the 25
categories. Unlike the previous method 2DPASS, which relies on 2D images as an
auxiliary input, our algorithm is purely based on point cloud data, which is a
valuable advantage.

Voxel R-CNN

LSK3DNet

Figure S1: Qualitative detection results of LSK3DNet and Voxel R-CNN on the
KITTI val split (Sec. 4.3). Results in the red box are false positives.

Qualitative Detection Results for KITTI. We compare the visualization results
between LSK3DNet and Voxel R-CNN for the car category on the KITTI val split
[28], as illustrated in Fig. S1.${}_{\\!}$ The blue-bordered boxes depict
ground truth bounding boxes, while the green-bordered boxes represent
predicted bounding boxes. In comparison to Voxel R-CNN, LSK3DNet demonstrates
more accurate prediction results.

Qualitative Results for SemanticKITTI. We provide visual examples of our
algorithm on two challenges for 3D semantic segmentation: SemanticKITTI [25]
single-scan val challenge and SemanticKITTI [25] multi-scan test
challenge.${}_{\\!}$ The corresponding figures are Fig. S2 and Fig. S3,
respectively. Moreover, we display the predicted labels of three successive
frames in one single frame in Fig. S3. It can be seen that our LSK3DNet
produces more accurate predictions than the baseline (Modified SPVCNN [3]). By
a larger kernel size, our LSK3DNet expands the receptive field of submanifold
convolution and enhances the flow of discrete spatial information. This
results in better capturing the object boundaries and distinguishing between
different semantic classes. As shown, our LSK3DNet can segment the ground
classes and natural objects more effectively.

##  Appendix D Limitation and Societal Impact

Limitation. Dense matrix multiplications on graphics processing units (GPUs)
are the foundation of the current state-of-the-art deep learning
methods.${}_{\\!}$ However, sparse matrix multiplications,${}_{\\!}$ which are
crucial for Dynamic Sparse Training operations, cannot be efficiently
performed [111, 112, 66].${}_{\\!}$ Hence, the FLOPs for inference that we
present in the main paper are based on theoretical calculations. This creates
a need for optimized hardware that can handle such operations.${}_{\\!}$
However, there is a good news that sparsity support is becoming a more common
feature in hardware design for many companies and researchers [68, 113,
114].${}_{\\!}$ Our work aims to inspire more progress in this direction,
especially for 3D tasks.

Societal Impact. Self-driving cars need to efficiently and accurately
understand 3D scenes to ensure safe driving. Since passenger safety is the top
priority for autonomous vehicles, 3D perception models are required to achieve
both high accuracy and low latency. Autonomous vehicles have limited hardware
resources due to the physical limitations of size and heat dissipation.
Therefore, it is important to design 3D neural networks that are both
efficient and effective for constrained computing resources. We use the method
of Spatial Sparse Group to further expand the 3D convolution kernel size,
which overcomes the performance drop of previous 3D large kernel methods.
Moreover, the model is very efficient due to dynamic sparse and Channel-wise
Weight Selection.

License. We study 3D semantic segmentation and object detection on four famous
datasets.${}_{\\!}$ We use the SemanticKITTI dataset under the permission of
its creators and authors by registering at
https://codalab.lisn.upsaclay.fr/competitions/6280.${}_{\\!}$ Scannet v2 is
released under the ScanNet Terms of Use
(https://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf).${}_{\\!}$ KITTI is
published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0
License.${}_{\\!}$ nuScenes is available for non-commercial use subject to the
Terms of Use (See https://www.nuscenes.org/terms-of-use).

Computing Infrastructure. The hardware configuration consists of an Intel(R)
Xeon(R) Gold 6132 CPU @ 2.60GHz and 18GB of memory.${}_{\\!}$ The operating
system is Ubuntu 18.04. All the experiments use Tesla V100 GPUs with 32GB of
VRAM. Moreover, our method is executed on top of the PyTorch framework.

Table S4: Semantic segmentation results on NuScenes validation set [109].
Regarding input data format, P denotes points, V represents voxelizations, R
signifies range images, 2D3DNet and 2DPASS incorporate additional 2D data.
mIoU (%) and IoUs (%) are reported.

Method |  Input |  mIoU |  barrier |  bicycle |  bus |  car |  construction |  motorcycle |  pedestrian |  traffic cone |  trailer |  truck |  driveable |  other flat |  sidewalk |  terrain |  manmade |  vegetation
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(AF)2-S3Net [85] | V | 62.2 | 60.3 | 12.6 | 82.3 | 80.0 | 20.1 | 62.0 | 59.0 | 49.0 | 42.2 | 67.4 | 94.2 | 68.0 | 64.1 | 68.6 | 82.9 | 82.4
RangeNet++ [115] | R | 65.5 | 66.0 | 21.3 | 77.2 | 80.9 | 30.2 | 66.8 | 69.6 | 52.1 | 54.2 | 72.3 | 94.1 | 66.6 | 63.5 | 70.1 | 83.1 | 79.8
PolarNet [62] | R | 71.0 | 74.7 | 28.2 | 85.3 | 90.9 | 35.1 | 77.5 | 71.3 | 58.8 | 57.4 | 76.1 | 96.5 | 71.1 | 74.7 | 74.0 | 87.3 | 85.7
Salsanext [60] | R | 72.2 | 74.8 | 34.1 | 85.9 | 88.4 | 42.2 | 72.4 | 72.2 | 63.1 | 61.3 | 76.5 | 96.0 | 70.8 | 71.2 | 71.5 | 86.7 | 84.4
AMVNet [116] | P | 76.1 | 79.8 | 32.4 | 82.2 | 86.4 | 62.5 | 81.9 | 75.3 | 72.3 | 83.5 | 65.1 | 97.4 | 67.0 | 78.8 | 74.6 | 90.8 | 87.9
Cylinder3D [15] | V | 76.1 | 76.4 | 40.3 | 91.2 | 93.8 | 51.3 | 78.0 | 78.9 | 64.9 | 62.1 | 84.4 | 96.8 | 71.6 | 76.4 | 75.4 | 90.5 | 87.4
RPVNet [23] | RPV | 77.6 | 78.2 | 43.4 | 92.7 | 93.2 | 49.0 | 85.7 | 80.5 | 66.0 | 66.9 | 84.0 | 96.9 | 73.5 | 75.9 | 76.0 | 90.6 | 88.9
PVKD [19] | V | 76.0 | 76.2 | 40.0 | 90.2 | 94.0 | 50.9 | 77.4 | 78.8 | 64.7 | 62.0 | 84.1 | 96.6 | 71.4 | 76.4 | 76.3 | 90.3 | 86.9
2D3DNet [110] | V | 79.0 | 78.3 | 55.1 | 95.4 | 87.7 | 59.4 | 79.3 | 80.7 | 70.2 | 68.2 | 86.6 | 96.1 | 74.9 | 75.7 | 75.1 | 91.4 | 89.9
2DPASS [3] | PV | 79.4 | 78.8 | 49.6 | 95.6 | 93.6 | 60.0 | 84.1 | 82.2 | 67.5 | 72.6 | 88.1 | 96.8 | 72.8 | 76.2 | 76.5 | 89.4 | 87.2
SphereFormer [24] | V | 79.5 | 78.7 | 46.7 | 95.2 | 93.7 | 54.0 | 88.9 | 81.1 | 68.0 | 74.2 | 86.2 | 97.2 | 74.3 | 76.3 | 75.8 | 91.4 | 89.7
LSK3DNet | PV | 80.1 | 80.0 | 52.5 | 94.5 | 91.8 | 58.8 | 85.8 | 84.4 | 71.2 | 73.8 | 88.3 | 96.9 | 74.8 | 75.9 | 75.9 | 89.3 | 87.5

Table S5: ${}_{\\!}$Quantitative${}_{\\!}$ results${}_{\\!}$ on${}_{\\!}$
SemanticKITTI${}_{\\!}$ [25]${}_{\\!}$ multi-scan${}_{\\!}$
challenge${}_{\\!}$ test${}_{\\!}$ (Sec. 4.2) - Part I. mIoU (%) and IoUs (%)
are reported.

Method | mIoU | Acc |  road |  sidewalk |  parking |  other-ground |  building |  car |  moving car |  truck |  moving truck |  bicycle |  motorcycle |  other-vehicle |  moving other-vehicle
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
TangentConv [57] | 34.1 | - | 83.9 | 64.0 | 38.3 | 15.3 | 85.8 | 84.9 | 40.3 | 21.1 | 1.1 | 2.0 | 18.2 | 18.5 | 6.4
DarkNet53 [25] | 41.6 | - | 91.6 | 75.3 | 64.9 | 27.5 | 85.2 | 84.1 | 61.5 | 20.0 | 14.1 | 30.4 | 32.9 | 20.7 | 15.2
TemporalLidarSeg [91] | 47.0 | 89.6 | 91.8 | 75.8 | 59.6 | 23.2 | 89.8 | 92.1 | 68.2 | 39.2 | 2.1 | 47.7 | 40.9 | 35.0 | 12.4
SpSeqnet [117] | 43.1 | - | 90.1 | 73.9 | 57.6 | 27.1 | 91.2 | 88.5 | 53.2 | 29.2 | 41.2 | 24.0 | 26.2 | 22.7 | 26.2
KPConv [42] | 51.2 | 89.3 | 86.5 | 70.5 | 58.4 | 26.7 | 90.8 | 93.7 | 69.4 | 42.5 | 5.8 | 44.9 | 47.2 | 38.6 | 4.7
Cylinder3D [15] | 52.5 | 91.0 | 90.7 | 74.5 | 65.0 | 32.3 | 92.6 | 94.6 | 74.9 | 41.3 | 0.0 | 67.6 | 63.8 | 38.8 | 0.1
(AF)2-S3Net [85] | 56.9 | 88.1 | 91.3 | 72.5 | 68.8 | 53.5 | 87.9 | 91.8 | 65.3 | 15.7 | 5.6 | 65.4 | 86.8 | 27.5 | 3.9
PV-KD [19] | 58.2 | 91.9 | 92.4 | 77.4 | 69.9 | 31.5 | 92.7 | 96.2 | 84.3 | 50.0 | 20.9 | 64.9 | 64.8 | 46.4 | 19.0
2DPASS [3] | 62.4 | 91.4 | 89.7 | 74.7 | 67.4 | 40.0 | 93.6 | 96.2 | 82.1 | 48.2 | 16.1 | 63.6 | 63.7 | 52.7 | 3.8
LSK3DNet | 63.4 | 92.2 | 92.1 | 79.0 | 67.4 | 41.0 | 92.9 | 96.4 | 84.4 | 58.1 | 7.2 | 71.2 | 73.9 | 61.8 | 40.9

Table S6: ${}_{\\!}$Quantitative${}_{\\!}$ results${}_{\\!}$ on${}_{\\!}$
SemanticKITTI${}_{\\!}$ [25]${}_{\\!}$ multi-scan${}_{\\!}$
challenge${}_{\\!}$ test${}_{\\!}$ (Sec. 4.2) - Part II. mIoU (%) and IoUs
(%) are reported.

Method | mIoU | Acc |  vegetation |  trunk |  terrain |  person |  moving person |  bicyclist |  moving bicyclist |  motorcyclist |  moving motorcyclist |  fence |  pole |  traffic-sign
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
TangentConv [57] | 34.1 | - | 79.5 | 43.2 | 56.7 | 1.6 | 1.9 | 0.0 | 30.1 | 0.0 | 42.2 | 49.1 | 36.4 | 31.2
DarkNet53 [25] | 41.6 | - | 78.4 | 50.7 | 64.8 | 7.5 | 0.2 | 0.0 | 28.9 | 0.0 | 37.8 | 56.5 | 38.1 | 53.3
TemporalLidarSeg [91] | 47.0 | 89.6 | 82.3 | 62.5 | 64.7 | 14.4 | 40.4 | 0.0 | 42.8 | 0.0 | 12.9 | 63.8 | 52.6 | 60.4
SpSeqnet [117] | 43.1 | - | 84.0 | 66.0 | 65.7 | 6.3 | 36.2 | 0.0 | 2.3 | 0.0 | 0.1 | 66.8 | 50.8 | 48.7
KPConv [42] | 51.2 | 89.3 | 84.6 | 70.3 | 66.0 | 21.6 | 67.5 | 0.0 | 67.4 | 0.0 | 47.2 | 64.5 | 57.0 | 53.9
Cylinder3D [15] | 52.5 | 91.0 | 85.8 | 72.0 | 68.9 | 12.5 | 65.7 | 1.7 | 68.3 | 0.2 | 11.9 | 66.0 | 63.1 | 61.4
(AF)2-S3Net [85] | 56.9 | 88.1 | 75.1 | 64.6 | 57.4 | 16.4 | 67.6 | 15.1 | 66.4 | 67.1 | 59.6 | 63.2 | 62.6 | 71.0
PV-KD [19] | 58.2 | 91.9 | 86.4 | 74.1 | 70.2 | 16.6 | 68.5 | 0.0 | 69.2 | 2.0 | 50.5 | 70.3 | 66.9 | 70.6
2DPASS [3] | 62.4 | 91.4 | 86.2 | 73.9 | 71.0 | 35.4 | 80.3 | 7.9 | 71.2 | 62.0 | 73.1 | 72.9 | 65.0 | 70.5
LSK3DNet | 63.4 | 92.2 | 86.9 | 74.2 | 72.6 | 29.3 | 77.4 | 0.1 | 69.9 | 22.8 | 72.1 | 72.3 | 66.7 | 74.3

Error by Baseline

Error by LSK3DNet

Ground Truth

Figure S2: Error maps of Baseline and Ours on SemanticKITTI [25] single-scan
val${}_{\\!}$ (Sec. 4.2).

Error by Baseline

Error by LSK3DNet

Ground Truth

Figure S3: Error maps of Baseline and Ours on SemanticKITTI [25] multi-scan
val${}_{\\!}$ (Sec. 4.2).