# Knowledge Distillation from 3D to Bird’s-Eye-View for LiDAR Semantic
Segmentation ††thanks: This work is supported by Shanghai Municipal Science
and Technology Major Project (No. 2018SHZDZX01), ZJLab, Shanghai Center for
Brain Science and Brain-Inspired Technology.

Feng Jiang
Haiqiang Zhang  ISTBI
Fudan University
Shanghai, China
<EMAIL_ADDRESS>Mogo Auto
Beijing, China
<EMAIL_ADDRESS>Heng Gao
Ru Wan  ISTBI
Fudan University
Shanghai, China
<EMAIL_ADDRESS>Mogo Auto
Beijing, China
<EMAIL_ADDRESS>Shoumeng Qiu
Jian Pu∗ School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>ISTBI, Fudan University
Shanghai, China
<EMAIL_ADDRESS>
###### Abstract

LiDAR point cloud segmentation is one of the most fundamental tasks for
autonomous driving scene understanding. However, it is difficult for existing
models to achieve both high inference speed and accuracy simultaneously. For
example, voxel-based methods perform well in accuracy, while Bird’s-Eye-View
(BEV)-based methods can achieve real-time inference. To overcome this issue,
we develop an effective 3D-to-BEV knowledge distillation method that transfers
rich knowledge from 3D voxel-based models to BEV-based models. Our framework
mainly consists of two modules: the voxel-to-pillar distillation module and
the label-weight distillation module. Voxel-to-pillar distillation distills
sparse 3D features to BEV features for middle layers to make the BEV-based
model aware of more structural and geometric information. Label-weight
distillation helps the model pay more attention to regions with more height
information. Finally, we conduct experiments on the SemanticKITTI dataset and
Paris-Lille-3D. The results on SemanticKITTI show more than 5% improvement on
the test set, especially for classes such as motorcycle and person, with more
than 15% improvement. The code can be accessed at
https://github.com/fengjiang5/Knowledge-Distillation-from-Cylinder3D-to-
PolarNet.

###### Index Terms:

Scene understanding, point clouds, knowledge distillation, semantic
segmentation

##  I Introduction

The 3D point clouds can effectively capture the real-world scene while
preserving geometric and structural information to the greatest extent. Point
cloud semantic segmentation plays a very important role in the perception of
the surrounding environment, especially in autonomous driving, which aims to
predict a label for each point in the current scan [1]. Therefore, balancing
speed and accuracy is especially important.

Currently, deep learning-based LiDAR segmentation methods can be generally
categorized as point-based [2, 3, 4], projection-based [5, 6, 7, 8] and voxel-
based [9, 10]. For instance, PointNet [2] is a point-based method that only
uses a stack of MLPs to learn the representations of raw point clouds
directly. Projection-based methods can be further divided into two categories:
range-based methods that use spherical projection [6, 8] and Bird’s-Eye-View
(BEV)-based methods [5]. After point cloud projection, these methods can take
full advantage of traditional 2D convolutional neural networks. The inference
speed of these methods is very fast and can meet the requirements of real-time
[11], but the precision of these models is not satisfactory. In contrast,
voxel-based methods [9, 10] can achieve high accuracy but are difficult to
apply in practice due to the use of time-consuming and computationally
expensive sparse operators [12]. Geoffrey Hinton proposed knowledge
distillation [13], which is usually used for model compression. It trains the
simple student model by using the outputs of the teacher model as a
supervisory signal. Currently, many methods used for point cloud segmentation
apply knowledge distillation for model compression [10] or better feature
extraction [14]. However, few methods focus on knowledge distillation between
two different point cloud segmentation methods.

Therefore, the trade-off between high accuracy and high inference speed is a
pivotal problem for practical applications, such as autonomous driving. To
address this issue, we propose a novel framework to improve the accuracy of
BEV-based models while maintaining their real-time inference speed. We develop
voxel-to-pillar distillation and label-weight distillation modules for
knowledge distillation. Specifically, voxel-to-pillar distillation is an
attention-based method for middle feature distillation that utilizes cross-
attention mechanisms and MSE loss to help BEV-based models learn more
structural and spatial geometric information from voxel-based models. Label-
weight distillation is used for the last layer before classification, which
selects regions with relatively large values in the height count map for
distillation since these regions are the main reason for the performance gap.

We evaluate our model on the SemanticKITTI dataset [1] and Paris-Lille-3D
[15], and it outperforms the original PolarNet [5] by 5% mIoU on the test set
of SemanticKITTI. We also conduct ablation studies to demonstrate the effect
of the distillation modules. The results show that our framework can largely
improve the segmentation performance of PolarNet, especially the performance
for classes that have more height information, such as motorcycles and persons
whose heights along the z-axis are compressed under the Bird’s-Eye-View.

The main contributions of our work are threefold:

  * •

We propose an effective knowledge distillation framework that transfers
knowledge from 3D voxel-based to BEV-based models for LiDAR point cloud
segmentation.

  * •

We develop a novel voxel-to-pillar distillation module that helps BEV-based
models learn more deep structural and spatial geometric information. Moreover,
to reduce the height information loss, we design a label-weight distillation
module focusing more on key regions with rich height information.

  * •

We conduct experiments on the SemanticKITTI dataset. The results show that our
framework can outperform the original by 5% mIoU on the test set, especially
in classes such as motorcycle and person, with more than 15% improvement. We
also conduct experiments on Paris-Lille-3D, and the results demonstrate our
method’s effectiveness and generalization power.

##  II Related Work

###  II-A LiDAR Segmentation

Currently, LiDAR point cloud segmentation methods [5, 6, 7, 9, 16, 17], which
are crucial for autonomous driving scene understanding, can be mainly
separated into three categories: point-based, projection-based and voxel-
based. Point-based methods, such as PointNet [2] and KPConv [3], consume raw
point clouds as input without any voxelization or other intermediate
representations, which is straightforward. However, these methods are
computationally intensive due to a large number of laser points.

To overcome this problem, methods based on range view (RV) and bird’s-eye-view
(BEV) first project 3D points into 2D space and then feed the projected maps
into a 2D convolution network for feature extraction. For instance, PolarNet
[5] quantizes the points into polar BEV grids and uses ring CNNs for
downstream segmentation tasks, which improves accuracy while maintaining real-
time inference speed. However, projection-based methods will inevitably lose
considerable geometric information due to dimension compression. Cylinder3D
[9], a classic voxel-based method, proposes to use a cylindrical partition and
an asymmetrical 3D sparse module to retain 3D representation and tackle the
issues induced by the point cloud’s sparsity and unevenness of distribution at
the same time. Although voxel-based methods have the ability to achieve high
accuracy, they demand a huge amount of memory usage and computing resources.
Therefore, they are difficult to use in real-world applications such as real
vehicle deployment. To reach high precision and decrease inference time, we
propose a novel knowledge distillation framework to help BEV-based models
learn more geometric and structural information.

###  II-B Knowledge Distillation

Knowledge distillation, first proposed by Geoffrey Hinton in [13], is a
generic model compression architecture that transfers the outstanding learning
representation capability of cumbersome and large models to small but easily
deployable models. In addition, it can also be used for auxiliary supervision
when training the model [14, 18].

Wu et al. [18] proposed ADD, an attention-based depth knowledge distillation
framework with a 3D-aware positional encoding that uses an extra lidar branch
to capture depth information. They used cross-attention to improve training
from the lidar branch to the main branch, and the results showed that this is
better than the simple use of MSE loss. Hou [10] claimed that their Point-to-
Voxel Knowledge Distillation (PVKD) is the first work that applied knowledge
distillation for LiDAR semantic segmentation. In their work, they proposed a
supervoxel partition method to divide the point clouds into several
supervoxels and designed a difficulty-aware sampling strategy to more
frequently sample supervoxels containing less-frequent classes and faraway
objects. Yan [14] proposed a 2D priors assisted semantic segmentation method
to boost the representation learning on point clouds. They have achieved good
results but are still limited by speed because of sparse operators.

The previous methods rarely involved knowledge distillation from voxel-based
methods to BEV-based methods. Our proposed framework can solve this problem
well, especially for classes where much information is lost during projection,
such as motorcycles and persons.

##  III Method

Figure 1: The overview of our framework consists of three main parts: the
teacher model, the student model and the knowledge distillation model. During
training, we use pretrained weights for the teacher model and update the
parameters of the student model and knowledge distillation model. Voxel-to-
pillar distillation (VPD) is used for general middle layers, and label-weight
distillation (LWD) is specifically used for the layer before classification.

Given a certain scan of point clouds $\mathbf{P}\in\mathbb{R}^{N\times 4}$,
the objective of semantic segmentation is to predict a label for each point,
where $N$ is the number of points. Each point contains ($x,y,z,i$), where
$(x,y,z)$ are the Cartesian coordinates relative to the LiDAR scanner and $i$
is the reflection intensity.

We propose a novel framework that can transfer knowledge from accurate voxel-
based methods to efficient BEV-based methods through knowledge distillation.
Specifically, we propose two general modules for knowledge distillation from
3D sparse features to BEV features to reduce the gap between these two
methods. First, the point clouds independently pass through the voxel-based
methods and the BEV-based methods. Then voxel-to-pillar distillation and
label-weight distillation are used for certain layers to help the BEV-based
methods learn more geometric and structural information. The details of the
proposed framework are as follows.

###  III-A Framework Overview

The architecture of our framework is shown in Fig. 1. Our framework has three
main parts, similar to other knowledge distillation methods [10, 14, 18]. The
first is the teacher model, and here, we choose voxel-based methods as our
teacher model, which have high accuracy but suffer from a computational
burden. Voxel-based methods usually encode each voxel and have more geometric
and structural features. We choose BEV-based methods as the student model
because they are usually efficient for practical application. The last part is
the knowledge distillation model, which is the main part of our framework.
Benefiting from our well-designed voxel-to-pillar distillation module and
label-weight distillation module, the student model can learn more valuable
information during training and improve the inference performance without
extra computation. In addition, we also used logit distillation similar to
others [10, 13].

Figure 2: Illustration of the voxel-to-pillar distillation (VPD) module. We
flatten the 3D sparse features of the teacher model and use MLPs to transfer
the domain to match the corresponding features of the student model. Then, we
use a cross-attention module between the 3D sparse features and the BEV
features to reinforce the learning capabilities of the student model.

###  III-B Voxel-to-Pillar Distillation

Voxel-to-pillar distillation is one of our framework’s main modules of
knowledge distillation, as illustrated in Fig. 2. Generally, voxel-based
methods use sparse operators [12] because the point clouds are sparse and
unevenly distributed. To maintain the sparsity of the point clouds and reduce
the computational cost, most sparse operators make the features of
intermediate layers still sparse, making voxel-based methods slow and hard to
align directly with BEV-based methods. BEV-based methods take pillars as
pseudo images, so they can directly use efficient CNNs designed for image
processing, but the information is lost at the data preprocessing stage. Our
proposed module helps BEV-based methods overcome this problem to some extent.

The feature of voxel-based methods is denoted as $F_{V}^{i}$, and the
corresponding BEV-based feature is denoted as $F_{B}^{i}$, where $i$ means the
$i$-th layer and $F_{B}^{i}\in\mathbb{R}^{N\times
C_{V}},F_{V}^{i}\in\mathbb{R}^{N\times C_{B}}$. For simplicity, here, we
assume that $F_{V}^{i}$ and $F_{B}^{i}$ have the same size in the $x$-$y$
plane. First, we use sparse convolution to compress the features of voxel-
based methods $F_{V}^{i}$ in the z-axis and denote it as $F_{VC}^{i}$, where
$F_{VC}^{i}=f(F_{V}^{i})$ and $f:\mathbb{R}^{N\times
C_{V}}\rightarrow\mathbb{R}^{N\times C_{B}}$. Since $F_{VC}^{i}$ is sparse, we
need to obtain the coordinates of non-empty voxels, which are used to match
the features of BEV-based methods exactly. Second, we flatten features from
voxel-based methods and BEV-based methods, which are denoted as $f_{V}^{i}$
and $f_{B}^{i}$. $f_{V}^{i}$ and $f_{B}^{i}$ are from different methods, so
they are in different domain spaces and usually have different distributions.
To solve this problem, we use the domain transfer method learned from BYOL
[19], specifically two MLPs with normalization. Specifically,

$\displaystyle f_{V}^{i}=$ $\displaystyle\text{MLP}(F(C(F_{V}^{i})))$ (1)
$\displaystyle f_{B}^{i}=$ $\displaystyle F(F_{B}^{i}),$

where MLP means domain transfer, $F$ means flattening of the features and $C$
means compression of the shape of the features. Then, we produce the Key from
$f_{B}^{i}$ and obtain the Query and Value from $f_{V}^{i}$ by MLPs. We
integrate these in a cross-attention module by

$Attention(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (2)

such as [18] to generate student features $f_{B^{\prime}}^{i}$ and use MSE
loss between $f_{B^{\prime}}^{i}$ and $f_{V}^{i}$ to make the student methods
learn the lost information during data preprocessing. We normalize the
features $f_{B^{\prime}}^{i}$ and $f_{V}^{i}$ learned from BYOL [19], which
means that we focus more on directional differences than numerical
differences. Therefore, the point-to-voxel distillation loss is given as
follows:

$\mathcal{L}_{VPD}\left(f_{B^{\prime}},f_{V}\right)=\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{1}{N_{i}}\left\|\frac{f_{B^{\prime}}^{i}}{\left\|f_{B^{\prime}}^{i}\right\|_{2}}-\frac{f_{V}^{i}}{\left\|f_{V}^{i}\right\|_{2}}\right\|_{2}^{2}$
(3)

where $N_{i}$ is the number of nonempty voxels and $\mathcal{I}$ is the subset
of layers used for distillation.

###  III-C Label-Weight Distillation

Label-weight distillation is used for the last layer before classification.
One of the main reasons for the performance degradation of the BEV-based
methods is that they lose considerable height information, which plays an
important role, especially for hard labels and hard scenes. Fig. 4 in Sec.
IV-E demonstrates this point of view. When we project the points onto a
$x$-$y$ plane, there may only be $4\sim 6$ pillars belonging to the class
person, and the class road will mislead the model because the proportion of
points on the road surface is large, although these pillars belong to the
class person. We cannot use voxel-to-pillar distillation directly because we
need global information in the middle layers, so we propose label-weight
distillation, which focuses on the key regions. The last layer is more
sensitive to height loss than the middle layers, so we use height embedding to
record the height information better.

Fig. 3 shows the details of label-weight distillation. Because we want the
model to focus on some key regions and the last layer always has a large size,
selecting some regions for distillation is necessary. We divide the whole
scene into $K$ regions. According to the above analysis, we design a method to
select regions with more non-empty voxels along the z-axis. Specifically, we
count the height map $H$ for sparse voxel features, and the value is the
number of nonempty voxels in the current position. Then, calculate weights for
each region according to their ground truth label. The weight and the
probability of selecting the $i$-th region are defined as
$W_{i}=H_{i}/\text{sum}(H)$, $P_{i}=W_{i}/\text{sum}(W)$, where
$i\in\\{1\cdots K\\}$. We note that we do not select the region with no height
information. Our proposed method is more suitable for distillation from voxel-
based methods to BEV-based methods, especially for the height loss situation.

Figure 3: Illustration of the label-weight distillation (LWD) module. We
combine height embeddings with the original teacher features and choose
features of the corresponding location using proposed selective sampling.
Finally, we apply MSE loss with label-generated weights between selected
features from the teacher and the student models.

Specifically, we first add the height embedding $F_{h}$ with features from
voxel-based methods $f_{V}$, which enables the model to encode deeper high-
level information. We use two sparse convolution layers on $f_{V}$ after
height encoding to compress the 3D features and transfer the domain to match
the features of the BEV-based methods simultaneously. Then, we select $M$
regions for features from BEV-based methods $f_{B}$ and features
$f_{V^{\prime}}$ after compression. Finally, we flatten the features and use
the MSE loss between $f_{B}$ and $f_{V^{{}^{\prime}}}$ with weights from
labels. The label-weight distillation loss is given as follows:

$\mathcal{L}_{LWD}\left(f_{B},f_{V^{{}^{\prime}}}\right)=\frac{1}{\sum_{j=1}^{M}{N_{j}}}\left\|\frac{f_{B}^{j}}{\left\|f_{B}^{j}\right\|_{2}}-\frac{f_{V^{{}^{\prime}}}^{j}}{\left\|f_{V^{{}^{\prime}}}^{j}\right\|_{2}}\right\|_{2}^{2}$
(4)

where $j$ represents the $j$-th selected region and $N_{j}$ represents the
number of nonempty voxels in the $j$-th region.

###  III-D Loss Function

Our loss function has two parts: the original segmentation loss of BEV-based
methods $\mathcal{L}_{\text{wce }}$ and $\mathcal{L}_{\text{lovasz }}$, the
loss of distillation $\mathcal{L}_{\text{{VPD}}}$, the loss of label-weight
distillation $\mathcal{L}_{\text{LWD}}$ and the loss of logit distillation
$\mathcal{L}_{\text{Logit}}$.

$\mathcal{L}_{\text{wce }}$ means weighted cross-entropy loss. We determine
the weight according to the reciprocal proportion of different categories of
points in the whole training set. We assume that there are $C$ classes, $x$ is
the input, $y$ is the target, $w=[w_{0},,w_{1},\cdots,w_{C-1}]$ is the class
weight, and $N$ is the point number of current batch.

$\mathcal{L}_{\text{wce
}}=-\sum_{n=1}^{N}\frac{1}{\sum_{n=1}^{N}w_{y_{n}}}w_{y_{n}}\log\frac{\exp\left(x_{n,y_{n}}\right)}{\sum_{c=1}^{C}\exp\left(x_{n,c}\right)}.$
(5)

$\mathcal{L}_{\text{lovasz }}$ is proposed by [20], which is directly used on
the intersection-over-union (IoU) score. Given the labels $y$, the predicts
$x$, the IoU of class $c$ is defined as

$J_{c}\left(y,x\right)=\frac{\left|\left\\{y=c\right\\}\cap\\{x=c\\}\right|}{\left|\left\\{y=c\right\\}\cup\\{x=c\\}\right|}.$
(6) $\mathcal{L}_{\text{lovasz }}=1-J_{c}\left(y,x\right).$ (7)

$\mathcal{L}_{\text{Logit}}$ is used in the probability of prediction like
[13]. Thus, we can obtain the final loss by

$\displaystyle\mathcal{L}=$ $\displaystyle\mathcal{L}_{\text{wce
}}+\mathcal{L}_{\text{lovasz
}}+\beta_{1}\mathcal{L}_{\text{{VPD}}}\left(f_{B^{\prime}}^{i},f_{V^{\prime}}^{i}\right)$
(8)
$\displaystyle+\beta_{2}\mathcal{L}_{\text{LWD}}\left(f_{B},f_{V}\right)+\beta_{3}\mathcal{L}_{\text{Logit}},$

where $\beta_{1},\beta_{2},\beta_{3}$ are the balance coefficients.

##  IV Experiment

###  IV-A Datasets and Metrics

SemanticKITTI [1] is a large dataset for LiDAR point cloud semantic
segmentation, which provides annotations for all points. It has 22 point cloud
sequences, each collected from a different scene. As recommended, we use 00 to
10 for training, 08 for validation during training, and 11 to 21 for testing.

We use the evaluation metric of the mean intersection-over-union (mIoU) over
all classes. The IoU is defined as
$mIoU=\frac{1}{n}\sum_{c=1}^{n}\frac{TP_{c}}{TP_{c}+FP_{c}+FN_{c}}$, where
$TP_{c}$ denotes the number of true positive points for class $c$, $FP_{c}$
denotes the number of false positives, and $FN_{c}$ is the number of false
negatives. The mIoU is the average of all classes.

###  IV-B Implementation Details

We use Cylinder3D[9] as an example of voxel-based methods and PolarNet[5] for
BEV-based methods. Cylinder3D uses cylindrical partition and sparse
convolution network, while PolarNet first projects the point clouds to BEVs
and uses ring convolution designed for BEV-based methods. They use a similar
pipeline for data preprocessing, point feature extraction with PointNet[2],
and encoder and decoder modules.

TABLE I: Quantitative comparison of PolarNet[5] trained with our framework. The results are reported in terms of the mIoU on the SemanticKITTI test set[1], and $*$ indicates that the result is from the original paper. We obtain 56.1 mIoU with the officially released code. The first two groups in the table are point-based and projection-based (including BEV-based and range-based) models. The last group includes our teacher model Cylinder3D[9], the original student model PolarNet [5] and the student model trained by our framework. Categories | Methods |  car |  bicycle |  motorcycle |  truck |  other-vehicle |  person |  bicyclist |  motorcyclist |  road |  parking |  sidewalk |  other-ground |  building |  fence |  vegetation |  trunk |  terrain |  pole |  traffic-sign | mIoU | FPS
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Point |  PointNet [2] | 46.3 | 1.3 | 0.3 | 0.1 | 0.8 | 0.2 | 0.2 | 0.0 | 61.6 | 15.8 | 35.7 | 1.4 | 41.4 | 12.9 | 31.0 | 4.6 | 17.6 | 2.4 | 3.7 | 14.6 | 2
LatticeNet [21] | 92.9 | 16.6 | 22.2 | 26.6 | 21.4 | 35.6 | 43.0 | 46.0 | 90.0 | 59.4 | 74.1 | 22.0 | 88.2 | 58.8 | 81.7 | 63.6 | 63.1 | 51.9 | 48.4 | 52.9 | 7
RandLa-Net [4] | 94.2 | 26.0 | 25.8 | 40.1 | 38.9 | 49.2 | 48.2 | 7.2 | 90.7 | 60.3 | 73.7 | 20.4 | 86.9 | 56.3 | 81.4 | 61.3 | 66.8 | 49.2 | 47.7 | 53.9 | 22
KPConv [3] | 96.0 | 30.2 | 42.5 | 33.4 | 44.3 | 61.5 | 61.6 | 11.8 | 88.8 | 61.3 | 72.7 | 31.6 | 90.5 | 64.2 | 84.8 | 69.2 | 69.1 | 56.4 | 47.4 | 58.8 | 38
Projection |  SqueezeSegV2 [7] | 82.7 | 21.0 | 22.6 | 14.5 | 15.9 | 20.2 | 24.3 | 2.9 | 88.5 | 42.4 | 65.5 | 18.7 | 73.8 | 41.0 | 68.5 | 36.9 | 58.9 | 12.9 | 41.0 | 39.6 | 83
RangeNet53 [6] | 91.4 | 25.7 | 34.4 | 25.7 | 23.0 | 38.3 | 38.8 | 4.8 | 91.8 | 65.0 | 75.2 | 27.8 | 87.4 | 58.6 | 80.5 | 55.1 | 64.6 | 47.9 | 55.9 | 52.2 | 12
SqueezeSegV3 [17] | 92.5 | 38.7 | 36.5 | 29.6 | 33.0 | 45.6 | 46.2 | 20.1 | 91.7 | 63.4 | 74.8 | 26.4 | 89.0 | 59.4 | 82.0 | 58.7 | 65.4 | 49.6 | 58.9 | 55.9 | 6
3D-MiniNet [11] | 90.5 | 42.3 | 42.1 | 28.5 | 29.4 | 47.8 | 44.1 | 14.5 | 91.6 | 64.2 | 74.5 | 25.4 | 89.4 | 60.8 | 82.8 | 60.8 | 66.7 | 48.0 | 56.6 | 55.8 | 28
SalsaNext [8] | 91.9 | 48.3 | 38.6 | 38.9 | 31.9 | 60.2 | 59.0 | 19.4 | 91.7 | 63.7 | 75.8 | 29.1 | 90.2 | 64.2 | 81.8 | 63.6 | 66.5 | 54.3 | 62.1 | 59.5 | 24
Voxel |  Cylinder3D [9] | 97.1 | 67.6 | 64.0 | 59.0 | 58.6 | 73.9 | 67.9 | 36.0 | 91.4 | 65.1 | 75.5 | 32.3 | 91.0 | 66.5 | 85.4 | 71.8 | 68.5 | 62.6 | 65.6 | 67.8 | 6
Projection |  PolarNet [5] * | 93.8 | 40.3 | 30.1 | 22.9 | 28.5 | 43.2 | 40.2 | 5.6 | 90.8 | 61.7 | 74.4 | 21.7 | 90.0 | 61.3 | 84.0 | 65.5 | 67.8 | 51.8 | 57.5 | 54.3 | 16
Projection | PolarNet+Ours | 95.2 | 53.5 | 45.9 | 41.8 | 42.0 | 60.8 | 54.1 | 29.8 | 91.8 | 63.6 | 75.8 | 24.8 | 90.5 | 61.6 | 82.1 | 65.2 | 66.3 | 54.4 | 61.3 | 61.1 | 16

We use the original model structure and parameters of Cylinder3D[9] and
PolarNet[5]. Here, we list our main settings and parameters used in knowledge
distillation. We select the 2nd and 3rd layers in the encoder and decoder for
voxel-to-pillar distillation and the last layer for label-weight distillation.
We divide the whole scene into 24 regions and set M to 2 for selective
sampling for label-weight distillation. For other parameters, we set the
batchsize to 2 and use Adam[22] with a learning rate of 0.001 for
optimization. We train our model for 40 epochs. The coefficients of loss
balance are $2,2,1$.

###  IV-C Results

The results on the SemanticKITTI dataset [1] are shown in Tab. I. We compare
different methods on the test set, including point-based methods such as
KPConv [3] and RandLa-Net [4] and projection-based methods such as RangeNet
[6], SqueezeSegV2 [7] and SalsaNext [8]. The last groups are our teacher model
Cylinder3D [9], original PolarNet [5] and PolarNet trained with our framework.
It can be seen that our model surpasses the baseline by 5% mIoU. In Sec. III,
we mention that height information loss is the main reason for BEV-based
methods performing worse than voxel-based methods. The experiment also proves
that some classes, such as motorcycle, truck, and person, have a huge gap
between Cylinder3D and PolarNet. Our proposed method mainly focuses on height
information lost through voxel-to-pillar distillation and label-weight
distillation. From the experiment, we can see that classes with richer height
information perform better than the baseline.

We also conduct experiments on Like Paris-Lille-3D [15]. The results shown in
Tab. II demonstrate our method’s effectiveness and generalization power.

TABLE II: Results on Paris-Lille-3D [15]. Model |  Categories |  mIoU |  FPS
---|---|---|---
Cylinder3D | Voxel | 73.03 | 6
PolarNet | Projection | 64.04 | 13
PolarNet+ours | Projection | 65.71 | 13

###  IV-D Ablation Studies

TABLE III: Ablation studies for network components on the SemanticKITTI validation set.  No. | PolarNet | Logit KD | VPD | LWD | mIoU
---|---|---|---|---|---
1 | $\sqrt{}$ |  |  |  | 57.08
2 | $\sqrt{}$ | $\sqrt{}$ |  |  | 57.31
3 | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | 59.46
4 | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ | 59.32
5 | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | 60.38
TABLE IV: Ablation studies for different strategies for the voxel-to-pillar distillation module on the SemanticKITTI validation set.  No. |  Compression mode |  Domain Transfer |  Cross Attention | mIoU
---|---|---|---|---
1 | scatter-max |  |  | 58.05
2 | sparse conv |  |  | 58.21
3 | sparse conv | $\sqrt{}$ |  | 59.08
4 | scatter-max | $\sqrt{}$ | $\sqrt{}$ | 59.32
5 | sparse conv | $\sqrt{}$ | $\sqrt{}$ | 59.46

In this part, we first perform ablation studies to show the effect of
different modules in our framework. The results on the SemanticKITTI
validation set are shown in Tab. III. The results are improved after applying
logit knowledge distillation between the classification logit of Cylinder3D
[9] and PolarNet [5]. Furthermore, voxel-to-pillar distillation and label-
weight distillation, as the main components of our proposed framework, can
improve the results by 2.38% and 2.24%, respectively. Finally, all these
modules can improve the results on the validation set by approximately 3.3%
mIoU.

Then, we further explore the results of adopting different strategies in the
voxel-to-pillar distillation module. We make some different variants of our
model to show the effect of different settings. The result is shown in Tab.
IV. We test different kinds of sparse operators, such as scatter-max and
sparse convolution, used for changing the resolution of features, and we can
see that sparse convolution is more appropriate to compress the 3D sparse
features to the BEV plane. Domain transfer and cross-attention further improve
the performance of our model.

###  IV-E Visualization

Fig. 4 shows the visual comparison between Cylinder3D, PolarNet, and our
method. We can see that the error numbers of different methods are different
at locations with obvious height changes or rich semantic information. Fig.
4(a) and Fig. 4(b) are the point clouds and ground truth, and Fig. 4(c) is the
number of non-empty voxels at different locations, where the darker the color
is, the greater the height change. Fig. 4(d), Fig. 4(e) and Fig. 4(f) are the
error number counts at different locations of Cylinder3D [9], PolarNet [5] and
our method. As shown in the red circles, our method can predict objects such
as buildings, trucks, etc., as correctly as Cylinder3D, but PolarNet often
makes incorrect predictions in this case.

(a) Point clouds

(b) GT

(c) Height Map

(d) Cylinder3D

(e) PolarNet

(f) Ours

Figure 4: Comparisons of false prediction maps of Cylinder3D, PolarNet and our
method on the SemanticKITTI dataset [1].

##  V Conclusions

In this paper, we propose a general knowledge distillation framework from
voxel-based models to BEV-based models for point cloud semantic segmentation.
Voxel-to-pillar distillation distills sparse 3D features to 2D BEV features
for middle layers and makes the BEV-based model learn more structural and
geometric knowledge. Label-weight distillation is used for the last layer
before classification, which helps the model pay more attention to regions
with more height information. Experiments on the SemanticKITTI dataset and
Paris-Lille-3D demonstrate that our method can outperform the baseline by 5%
mIoU on the test set, especially for classes such as motorcycle and person,
with more than 15% improvement.

## References

  * [1] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall,  “SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,”  in ICCV, 2019.
  * [2] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas,  “Pointnet: Deep learning on point sets for 3d classification and segmentation,”  in CVPR, 2017, pp. 652–660.
  * [3] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas,  “Kpconv: Flexible and deformable convolution for point clouds,”  in ICCV, 2019, pp. 6411–6420.
  * [4] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham,  “Randla-net: Efficient semantic segmentation of large-scale point clouds,”  in CVPR, 2020, pp. 11108–11117.
  * [5] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh,  “Polarnet: An improved grid representation for online lidar point clouds semantic segmentation,”  in CVPR, 2020, pp. 9601–9610.
  * [6] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss,  “Rangenet++: Fast and accurate lidar semantic segmentation,”  in IROS, 2019, pp. 4213–4220.
  * [7] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer,  “Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud,”  in ICRA, 2019, pp. 4376–4382.
  * [8] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy,  “Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds,”  in ISVS, 2020, pp. 207–222.
  * [9] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin,  “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,”  in CVPR, 2021, pp. 9939–9948.
  * [10] Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li,  “Point-to-voxel knowledge distillation for lidar semantic segmentation,”  in CVPR, 2022, pp. 8479–8488.
  * [11] Inigo Alonso, Luis Riazuelo, Luis Montesano, and Ana C Murillo,  “3d-mininet: Learning a 2d representation from point clouds for fast and efficient 3d lidar semantic segmentation,”  IEEE Robotics Autom. Lett, vol. 5, no. 4, pp. 5432–5439, 2020.
  * [12] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten,  “3d semantic segmentation with submanifold sparse convolutional networks,”  in CVPR, 2018, pp. 9224–9232.
  * [13] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.,  “Distilling the knowledge in a neural network,”  Statistics, vol. 1050, no. 9, 2015.
  * [14] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li,  “2dpass: 2d priors assisted semantic segmentation on lidar point clouds,”  in ECCV, 2022, pp. 677–695.
  * [15] Xavier Roynard, Jean-Emmanuel Deschaud, and François Goulette,  “Paris-lille-3d: A large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification,”  The International Journal of Robotics Research, vol. 37, no. 6, pp. 545–557, 2018.
  * [16] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer,  “Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud,”  in ICRA, 2018, pp. 1887–1893.
  * [17] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka,  “Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation,”  in ECCV, 2020, pp. 1–19.
  * [18] Zizhang Wu, Yunzhe Wu, Jian Pu, Xianzhi Li, and Xiaoquan Wang,  “Attention-based depth distillation with 3d-aware positional encoding for monocular 3d object detection,”  arXiv:2211.16779, 2022.
  * [19] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.,  “Bootstrap your own latent-a new approach to self-supervised learning,”  pp. 21271–21284, 2020.
  * [20] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko,  “The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,”  in CVPR, 2018, pp. 4413–4421.
  * [21] Radu Alexandru Rosu, Peer Schütt, Jan Quenzel, and Sven Behnke,  “Latticenet: Fast point cloud segmentation using permutohedral lattices,”  arXiv preprint arXiv:1912.05905, 2019.
  * [22] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in ICLR, 2015.


From Binary Error Correcting Codes
to a Relation Between Maximal D=4 and D=3 Supergravities

Thomas Fischbacher1 and Krzysztof Pilch2

1 Google Research,

Brandschenkestrasse 110, 8002 Zürich, Switzerland

2 Department of Physics and Astronomy,

University of Southern California,

Los Angeles, CA 90089-0484, USA

tfish @ google.com, pilch @ usc.edu

###### Abstract

This short note provides (TensorFlow-based) numerical evidence for the
embedability (in the limit of a scalar parameter going to infinity) of the
scalar potential of dyonic $\mathcal{N}=8,\;D=4\;SO(8)$ supergravity into the
scalar potential of $\mathcal{N}=16,\;D=3\;SO(8)\times SO(8)$ supergravity.
One finds that the dyonic $\omega$-rotation gets identified with the compact
$U(1)$ part of the $SL(2)$ factor of the $SL(2)\times E_{7(+7)}$ subgroup of
$E_{8(+8)}$.

##  1 Claims and Insights

This short note is accompanied by a Google Colab notebook111Available at
https://github.com/google-research/google-
research/tree/master/m_theory/colab/hamming78.ipynb, and also alongside the
arXiv source code of this article. The reader can launch this via web browser
by navigating to https://colab.research.google.com/, selecting ‘GitHub’ as
source for a new notebook, and pasting the above url. (based on TensorFlow
[1]) that numerically demonstrates the validity of each of these claims:

  1. 1.

(From [2], Eq. (7.5)): One can embed $\mathcal{M}_{14}:=(SU(1,1)/U(1))^{\times
7}$ in such a way into $E_{7(+7)}$ that the holomorphic superpotential is in
$1{:}1$ correspondence with the code words of the 1-bit error correcting
(7,4,3) Hamming code [3]:

$\begin{aligned} \mathcal{W}_{7}:=\\\ \\\ \\\ \\\
\end{aligned}\quad\begin{gathered}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{5}\zeta_{6}\zeta_{7}\\\
+\zeta_{3}\zeta_{5}\zeta_{6}\zeta_{7}+\zeta_{2}\zeta_{4}\zeta_{5}\zeta_{7}+\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{6}+\zeta_{1}\zeta_{3}\zeta_{4}\zeta_{5}+\zeta_{1}\zeta_{4}\zeta_{6}\zeta_{7}+\zeta_{1}\zeta_{2}\zeta_{5}\zeta_{6}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{7}\\\
+\zeta_{1}\zeta_{2}\zeta_{4}+\zeta_{1}\zeta_{3}\zeta_{6}+\zeta_{1}\zeta_{5}\zeta_{7}+\zeta_{2}\zeta_{6}\zeta_{7}+\zeta_{2}\zeta_{3}\zeta_{5}+\zeta_{3}\zeta_{4}\zeta_{7}+\zeta_{4}\zeta_{5}\zeta_{6}\\\
+1\,.\end{gathered}$ (1)

  2. 2.

Expanding the $(7,4,3)$ Hamming code with a parity bit to the self-dual
$(8,4,4)$ Hamming code, we can define a corresponding hypothesized holomorphic
superpotential as follows, by adding a factor $\zeta_{8}$ to those summands
that have an odd number of $\zeta$-factors:

$\begin{aligned} \mathcal{W}_{8}:=\\\ \\\ \\\ \\\
\end{aligned}\quad\begin{gathered}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{5}\zeta_{6}\zeta_{7}\zeta_{8}\\\
+\zeta_{3}\zeta_{5}\zeta_{6}\zeta_{7}+\zeta_{2}\zeta_{4}\zeta_{5}\zeta_{7}+\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{6}+\zeta_{1}\zeta_{3}\zeta_{4}\zeta_{5}+\zeta_{1}\zeta_{4}\zeta_{6}\zeta_{7}+\zeta_{1}\zeta_{2}\zeta_{5}\zeta_{6}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{7}\\\
+\zeta_{1}\zeta_{2}\zeta_{4}\zeta_{8}+\zeta_{1}\zeta_{3}\zeta_{6}\zeta_{8}+\zeta_{1}\zeta_{5}\zeta_{7}\zeta_{8}+\zeta_{2}\zeta_{6}\zeta_{7}\zeta_{8}+\zeta_{2}\zeta_{3}\zeta_{5}\zeta_{8}+\zeta_{3}\zeta_{4}\zeta_{7}\zeta_{8}+\zeta_{4}\zeta_{5}\zeta_{6}\zeta_{8}\\\
+1\,.\end{gathered}$ (2)

Observing that the scalar potential corresponding to such a holomorphic
superpotential on $\left(SU(1,1)/U(1)\right)^{\times 8}$ indeed does have many
equilibria that align nicely (after rescaling the cosmological constant) with
equilibiria reported in [4] for $\mathcal{N}=16,\;D=3\;SO(8)\times SO(8)$
supergravity, one may conjecture that one can indeed obtain this “$(8,4,4)$
Hamming code holomorphic superpotential” from the $A_{1}$-tensor of maximal
$D=3$ supergravity. This indeed holds – the details can be found in appendix
B.

  3. 3.

Starting from the commonly used roots for the $\mathfrak{e}_{8(+8)}$ algebra,
where the $120-8=112$ roots of the compact $\mathfrak{spin}(16)$ subalgebra
are given by $(\pm 1;\pm 1;0;0;0;0;0;0)+\\{\text{permutations}\\}$, and the
$128$ “$\mathfrak{spin}(16)$-spinor” roots corresponding to the generators
used to define the scalar manifold of $SO(8)\times SO(8)$ supergravity [5, 6],
$(\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2};\pm\frac{1}{2})$
(where the total number of $(-)$ signs is _even_), it is possible to choose
eight positive roots from the $128$ such that when adding the corresponding
eight negative roots to the set, no pair taken from these 16 roots have the
same sign in exactly two positions222This would be a requirement for the
commutator of the associated ladder operators to belong to
$\mathfrak{spin}(16)$, but not the $\mathfrak{u}(1)^{8}$ generated by the
commutators of the ladder operators for each positive root and its associated
negative root.. For any such choice, adding the corresponding negative roots,
and encoding a $(+)$-sign as 1 and a $(-)$-sign as 0 (or vice versa) gives us
sixteen eight-bit code words that correspond to a self-dual $(8,4,4)$ Hamming
code.333A related well-known observation is that scaling the self-dual $E_{8}$
lattice to integer coordinates and then taking coordinates modulo 2 yields the
$(8,4,4)$ self-dual Hamming code. Doing the same for the $E_{7}$ root lattice
yields the $(7,3,4)$ ‘little Hamming code’, while doing this for the dual
$E_{7}$ weight lattice ($E_{7}^{*}$) yields the $(7,4,3)$ Hamming code, see
e.g. [7, 8]. These sixteen roots then correspond to a
$\mathfrak{sl}(2)^{\times 8}$ subalgebra of $\mathfrak{e}_{8}$.

  4. 4.

Performing $\omega$-deformation [9, 10, 11, 12] of
$\mathcal{N}=8,\,D=4\,SO(8)$ supergravity [13], the superpotential in Eq. (1)
acquires phase factors $\phi:=\exp(-i\omega)$ on summands with an _odd_ number
of $\zeta$-factors and $\bar{\phi}=\exp(+i\omega)$ on summands with an _even_
number of $\zeta$-factors:

$\begin{aligned} \mathcal{W}_{7c}:=\\\ \\\ \\\ \\\
\end{aligned}\quad\begin{gathered}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{5}\zeta_{6}\zeta_{7}\phi\\\
+\zeta_{3}\zeta_{5}\zeta_{6}\zeta_{7}\bar{\phi}+\zeta_{2}\zeta_{4}\zeta_{5}\zeta_{7}\bar{\phi}+\zeta_{2}\zeta_{3}\zeta_{4}\zeta_{6}\bar{\phi}+\zeta_{1}\zeta_{3}\zeta_{4}\zeta_{5}\bar{\phi}+\zeta_{1}\zeta_{4}\zeta_{6}\zeta_{7}\bar{\phi}+\zeta_{1}\zeta_{2}\zeta_{5}\zeta_{6}\bar{\phi}+\zeta_{1}\zeta_{2}\zeta_{3}\zeta_{7}\bar{\phi}\\\
+\zeta_{1}\zeta_{2}\zeta_{4}\phi+\zeta_{1}\zeta_{3}\zeta_{6}\phi+\zeta_{1}\zeta_{5}\zeta_{7}\phi+\zeta_{2}\zeta_{6}\zeta_{7}\phi+\zeta_{2}\zeta_{3}\zeta_{5}\phi+\zeta_{3}\zeta_{4}\zeta_{7}\phi+\zeta_{4}\zeta_{5}\zeta_{6}\phi\\\
+\bar{\phi}\,.\end{gathered}$ (3)

Observing that the scalar potential does not change if the superpotential gets
multiplied by a complex number of magnitude 1, and multiplying the above
expression with $\phi$ shows
$\bar{\phi}\mathcal{W}_{7c}=\mathcal{W}_{8|\zeta_{8}=\phi^{2}}$. Indeed, one
finds that for $\omega=\pi/8$, the corresponding scalar potential on
$(SU(1,1)/U(1))^{\times 7}$ has equilibria for which the cosmological
constants closely correspond to known solutions of the ‘dyonic SO(8)’ gauging
with $\omega=\pi/8$ [14]. The relation between the scalar potentials and
superpotentials is given in appendix B.

  5. 5.

The above properties suggest that, at least on $(SL(2)/U(1))^{\times
7}\sim(SU(1,1)(2)/U(1))^{\times 7}$, we might be able to retrieve the scalar
potential of $D=4\,SO(8)$ supergravity from that of $D=3\,SO(8)\times SO(8)$
supergravity by taking some suitable $\zeta_{8}\to 1$ limit444Given that the
$\zeta$ parameters are coordinates in the Poincare disc model of the
hyperbolic plane, this is at infinite distance from the origin. – and
correspondingly, get the scalar potential of dyonic $D=4\,SO(8)_{c}$
supergravity by taking some $\zeta_{8}\to\exp(i\omega)$ limit. Hence, it seems
natural to expect that a corresponding limit may exist for the full scalar
potential: Using the $E_{7(+7)}\times SL(2)\subset E_{8(+8)}$ embedding for
which we have ${\bf 248}\mapsto({\bf 133},{\bf 1})+({\bf 56},{\bf 2})+({\bf
1},{\bf 3})$, the $SL(2)$ becomes the eighth $SL(2)$ in $E_{8}$ that commutes
with the seven $SL(2)$s whose noncompact directions yield $\mathcal{M}_{14}$.
Considering the triality-symmetric constructions of
$\mathfrak{e}_{7}=\mathfrak{spin}(8)+{\bf 35}_{v}+{\bf 35}_{s}+{\bf 35}_{c}$
and $\mathfrak{e}_{8}=\mathfrak{spin(8)}^{L}+\mathfrak{spin(8)}^{R}+({\bf
8}^{L}_{v},{\bf 8}^{R}_{v})+({\bf 8}^{L}_{s},{\bf 8}^{R}_{s})+({\bf
8}^{L}_{c},{\bf 8}^{R}_{c})$, it is clear how
$\mathfrak{e}_{7}+\mathfrak{sl}(2)$ is obtained from the ‘symmetric’ pieces of
the decomposition of $\mathfrak{e}_{8}$ with respect to the555Given that we
can apply a triality relabeling on one of the $\mathfrak{spin}(8)$ algebras,
there is more than one way to take a diagonal. The relevant diagonal here does
not involve a triality rotation. diagonal $\mathfrak{spin}(8)$ subalgebra of
$\mathfrak{spin}(8)^{L}+\mathfrak{spin}(8)^{R}$.

Using the corresponding embedding of the
$\mathfrak{e}_{7(7)}+\mathfrak{sl}(2)$ $D=4$ scalar manifold coset generators
${\bf 35}_{s}+{\bf 35}_{c}+{\bf 1}_{s}+{\bf 1}_{c}$ (‘symmetric traceless
$8\times 8$ matrices over the spinors and co-spinors from $\mathfrak{e}_{7}$
plus multiples-of-the-identity trace-parts from $\mathfrak{sl}(2)$’) into the
space of $D=4$ scalar manifold coset generators $({\bf 8}^{L}_{s},{\bf
8}^{R}_{s})+({\bf 8}^{L}_{c},{\bf 8}^{R}_{c})$ via a linear function
$E(v_{70},s,c):\mathbb{R}^{70+2}\to\mathbb{R}^{128}$, one finds for the $D=3$
scalar potential of $SO(8)\times SO(8)$ supergravity:
$g_{D=3}^{-2}V_{D=3}(E(\vec{0},s,0))<0$, and for $s>0$: $|\nabla V|>0$. These
are non-equilibrium points with negative cosmological constant. If we now
introduce an auxiliary (helper) function666 The factor $-6$ is for alignment
with the usual normalization of the $D=4$ scalar potential.
$H:\mathbb{R}^{70+1}\to\mathbb{R}$ as:

$H(\vec{v},s):=(-6)\,\cdot\,\frac{V_{D=3}(E(\vec{v},s,0))}{V_{D=3}(E(\vec{0},s,0))},$
(4)

then we may conjecture that $H$ is related to the $D=4$ scalar potential of
$SO(8)$ supergravity $g_{D=4}^{-2}V_{D=4}:\mathbb{R}^{70}\to\mathbb{R}$ via:

$V_{D=4}(\vec{v})=\lim_{s\to\infty}H(\vec{v},s).$ (5)

Numerical evidence strongly supports that this hypothesis holds _on the full
70-dimensional scalar manifold of $\mathcal{N}=8,\,D=4\,SO(8)$ Supergravity_!

  6. 6.

The generalization to dyonic-$SO(8)$ also holds. Specifically, with

$H_{c}(\vec{v},s,\omega):=(-6)\,\cdot\,\frac{V_{D=3}(E(\vec{v},s\cos(2\,\omega),s\sin(2\,\omega)))}{V_{D=3}(E(\vec{0},s\cos(2\,\omega),s\sin(2\,\omega)))},$
(6)

we find:

$V_{D=4}(\vec{v},\omega)=\lim_{s\to\infty}H_{c}(\vec{v},s,\omega).$ (7)

(As one would expect from the $\omega$-invariance of the $SO(8)$-symmetric
vacuum of $SO(8)$ supergravity, one actually finds
$V_{D=3}(E(\vec{0},s\cos(2\,\omega),s\sin(2\,\omega)))=V_{D=3}(E(\vec{0},s,0))$,
so the above expression, presented ‘in symmetric form’, can be simplified.)
Appendix A shows the numerical evidence, verifiable by running the
accompanying Google Colab notebook.

##  2 Discussion

The maximal (32 supercharges) gauged $D=2+1$ supergravity of Nicolai and
Samtleben [5, 6] so far has been mostly regarded as an exotic curiosity, as to
this date there is no known way to embed it into M theory. Correspondingly, it
has perhaps not yet received as much attention as this note suggests it should
have – given that we observe that it indeed seems to be closely related to the
$S^{7}$-compactification of 11-dimensional supergravity, i.e. the de Wit-
Nicolai model – as well as the dyonic deformations of that model [10], for
which there currently is no known way to embed these into M theory, either
[15, 16]. Naturally, this then means that taking the limit in a different way
will also allow us to retrieve scalar potentials of other gaugings with
already-known M theory embeddings, such as that of ‘dyonic ISO(7)
supergravity’ [17, 18].

###  2.1 Early Clues

As it is often useful to understand the intuition that underlies an idea, it
may be appropriate to list some major clues that contributed to generating the
idea of exploring the final claim in the list presented above. In
chronological order, these clues were:

  * •

The (stable and also unstable) equilibria of maximal supergravities often have
remarkable similarities across different dimensions. Notably, this also holds
in particular for $D=4$ and $D=3$. For example, whereas maximal $D=5$
supergravity has a $SU(2)\times U(1)\;\mathcal{N}=2$ vacuum, maximal $D=4$
supergravity has a $SU(3)\times U(1)\;\mathcal{N}=2$ vacuum; in $D=4$, we see
a $G_{2}\;\mathcal{N}=1$ vacuum, whereas in $D=3$, we find $G_{2}\times
G_{2}\;\mathcal{N}=(1,1)$, etc. – see [19, 20, 4].

  * •

As the problem of finding equilibria can be expressed entirely in terms of
geometric invariants, the relevant properties of the equilibria can be
expressed in terms of algebraic numbers. There is a general tendency for the
$D=3$ expressions to often have remarkably low algebraic complexity (see e.g.
[21]), just as if $D=4$ had to rebalance terms to make up for some loss of a
more fundamental symmetry.

  * •

John Baez’s article about triality and the exceptional groups [22] clearly was
inspirational for structuring the code that does calculations in $E_{7}$ in
such a way that it emphasizes the role of triality, despite virtually all of
the other literature only using (anti)self-dual four-form language for
$E_{7}$.

  * •

Closely studying the long list of equilibria of $SO(8)$ supergravity [23]
reveals some remarkable coincidences, such as the existence of a triplet of
equilibria with residual symmetry $SO(4)$ where embeddings of $SO(4)$ into
$SO(8)$ are related by triality. Likewise, there are closely-related-via-
triality pairs of solutions, such as S0668704–S0698771, S0869596–S0983994,
S1068971–S1301601, etc., that are related by triality (see also [24], as well
as [25]).

  * •

There have been various earlier indications that the 7-bit Hamming code is
useful to understand some nontrivial aspects of M theory [26, 27].

###  2.2 Outlook

It certainly is bemusing to observe how intuition related to binary error
correcting codes did provide a relevant clue here towards uncovering a
relation between $D=4$ and $D=3$ supergravities – especially with a view on
Wheeler’s “it from bit” essay [28] which proposes an agenda that includes “
_[Translating] the quantum versions of string theory and of Einstein’s
geometrodynamics from the language of continuum to the language of bits_ ”.
One may wonder whether there are more interesting insights that could be
obtained by focusing on the relation between remarkable lattices and binary
codes – noting however that the (even unimodular Lorentzian) $E_{10}$ root
lattice [29] does not directly correspond to an error correcting binary code –
likely due to the implicit notion of ‘Euclidean distance’ in the definition of
error-correcting codes. This might, however, be fixable, and suggests that a
study of the relation between $\mathfrak{e}_{10}$ and generalized binary codes
might bear fruit.

While our focus here was exclusively on the scalar potential, this is of
course closely linked to the entire structure of the model supersymmetry.
Nominally, we are here observing a correspondence between $D=3$ and $D=4$
supergravity in some “AdS radius goes to zero” (i.e. $g^{-2}V\to-\infty$)
limit. To do this, we had to ad-hoc fix one scalar parameter and move it
towards infinity without Supergravity offering a mechanism to stabilize this
configuration. We may, at this point, only speculate whether M theory also in
this setting “fights against being squeezed” by growing new spatial dimensions
via some tower of massive excitations (which would mean: degrees of freedom
not present in the supergravity truncation) collapsing to zero mass. Given our
current understanding of M theory, this speculation is however too outlandish
to be taken seriously.

More tangibly, the observation that there is a $SO(8)$ subgroup of $E_{8(8)}$
that rotates the eight commuting $SL(2)$s may provide useful to extract
additional information about the structure of the $D=4$ potential, given that
this $SO(8)$ cannot be a subgroup of $E_{7}$ (since it mixes the seven $U(1)$s
sitting inside $E_{7}$ with the one outside). This might lead to an
explanation for some observations about the equilibria of the $D=4$ scalar
potential that are currently hard to explain, such as the high degeneracies in
the mass spectra of the equilibrium S1800000 _despite complete breaking of
$SO(8)$_ with zero residual symmetry – neither Lie nor discrete. Signs of a
hidden $E_{8(+8)}$ symmetry in maximal $D=4$ supergravity are, of course, not
new (e.g. [30]), and so the hope is that the rather concrete new puzzle piece
explained in this work will lead to new angles of attack to resolve the
question about the underlying symmetries of M theory.

Acknowledgments

Thomas Fischbacher would like to thank Moritz Firsching for independently
confirming claim 3, and also Jyrki Alakuijala, George Toderici, Ashok Popat,
Rahul Sukthankar, Jay Yagnik, and Jeff Dean for on-going support and
encouragement of research that comprises a unusual but scientifically
successful applications of TensorFlow. We also would like to thank Gianluca
Inverso, David Berman, Nikolay Bobev, Fridrik Freyr Gautason, and Hermann
Nicolai for useful discussions. Krzysztof Pilch is supported in part by DOE
grant DE-SC0011687.

Appendix

##  Appendix A Notebook transcript from checking claim 6

It certainly is gratifying to look at the numbers that substantiate the claim
in Eq. (6). Below, we see what happens when one randomly picks ten (generic,
non-equilibrium) points on the $E_{7(+7)}/(SU(8)/\mathbb{Z}_{2})$ manifold
and, in $E_{8(8)}$, rotates outwards using $SL(2)$, both for $\omega=0$ and
some generic $\omega$.

⬇

>>> check_so8c_limit(omega=0, r=3, num_spot_checks=10)

V_so8c = -10.4047666737, V_so8xso8 = -10.4056327510, rel_delta = 0.000083

V_so8c = -9.4520901976, V_so8xso8 = -9.4528652474, rel_delta = 0.000082

V_so8c = -5.0839620894, V_so8xso8 = -5.0848796316, rel_delta = 0.000180

V_so8c = +10.8347526006, V_so8xso8 = +10.8334767696, rel_delta = 0.000118

V_so8c = -10.6441227948, V_so8xso8 = -10.6448297935, rel_delta = 0.000066

V_so8c = -7.1489366176, V_so8xso8 = -7.1496814995, rel_delta = 0.000104

V_so8c = +1.3503469257, V_so8xso8 = +1.3489888355, rel_delta = 0.001006

V_so8c = +26.0683069298, V_so8xso8 = +26.0658295953, rel_delta = 0.000095

V_so8c = +9.4490606160, V_so8xso8 = +9.4480316002, rel_delta = 0.000109

V_so8c = +9.7993167162, V_so8xso8 = +9.7976288261, rel_delta = 0.000172

>>> check_so8c_limit(omega=0, r=3.5, num_spot_checks=10)

V_so8c = -10.4047666737, V_so8xso8 = -10.4048838868, rel_delta = 0.000011

V_so8c = -9.4520901976, V_so8xso8 = -9.4521950911, rel_delta = 0.000011

V_so8c = -5.0839620894, V_so8xso8 = -5.0840862675, rel_delta = 0.000024

V_so8c = +10.8347526006, V_so8xso8 = +10.8345799326, rel_delta = 0.000016

V_so8c = -10.6441227948, V_so8xso8 = -10.6442184784, rel_delta = 0.000009

V_so8c = -7.1489366176, V_so8xso8 = -7.1490374282, rel_delta = 0.000014

V_so8c = +1.3503469257, V_so8xso8 = +1.3501631249, rel_delta = 0.000136

V_so8c = +26.0683069298, V_so8xso8 = +26.0679716529, rel_delta = 0.000013

V_so8c = +9.4490606160, V_so8xso8 = +9.4489213515, rel_delta = 0.000015

V_so8c = +9.7993167162, V_so8xso8 = +9.7990882811, rel_delta = 0.000023

>>> check_so8c_limit(omega=-4.567, r=3.0, num_spot_checks=10, scale=0.21)

V_so8c = +62.8771050816, V_so8xso8 = +62.8744930759, rel_delta = 0.000042

V_so8c = +4.4054319480, V_so8xso8 = +4.4041637306, rel_delta = 0.000288

V_so8c = +11.2968023928, V_so8xso8 = +11.2953240524, rel_delta = 0.000131

V_so8c = +15.7005232999, V_so8xso8 = +15.6989137243, rel_delta = 0.000103

V_so8c = +3.2290683270, V_so8xso8 = +3.2278830222, rel_delta = 0.000367

V_so8c = -3.3373855762, V_so8xso8 = -3.3383922744, rel_delta = 0.000302

V_so8c = +28.9523837775, V_so8xso8 = +28.9501294956, rel_delta = 0.000078

V_so8c = +90.8875944078, V_so8xso8 = +90.8832244306, rel_delta = 0.000048

V_so8c = -5.0578665526, V_so8xso8 = -5.0587664349, rel_delta = 0.000178

V_so8c = +5.9928537431, V_so8xso8 = +5.9908625218, rel_delta = 0.000332

>>> check_so8c_limit(omega=-4.567, r=3.5, num_spot_checks=10, scale=0.21)

V_so8c = +62.8771050816, V_so8xso8 = +62.8767515791, rel_delta = 0.000006

V_so8c = +4.4054319480, V_so8xso8 = +4.4052603105, rel_delta = 0.000039

V_so8c = +11.2968023928, V_so8xso8 = +11.2966023177, rel_delta = 0.000018

V_so8c = +15.7005232999, V_so8xso8 = +15.7003054637, rel_delta = 0.000014

V_so8c = +3.2290683270, V_so8xso8 = +3.2289079107, rel_delta = 0.000050

V_so8c = -3.3373855762, V_so8xso8 = -3.3375218204, rel_delta = 0.000041

V_so8c = +28.9523837775, V_so8xso8 = +28.9520786883, rel_delta = 0.000011

V_so8c = +90.8875944078, V_so8xso8 = +90.8870029854, rel_delta = 0.000007

V_so8c = -5.0578665526, V_so8xso8 = -5.0579883407, rel_delta = 0.000024

V_so8c = +5.9928537431, V_so8xso8 = +5.9925842557, rel_delta = 0.000045

# ’Farther out’:

>>> check_so8c_limit(omega=-4.567, r=4.0, num_spot_checks=10, scale=0.21)

V_so8c = +62.8771050816, V_so8xso8 = +62.8770572401, rel_delta = 0.000001

V_so8c = +4.4054319480, V_so8xso8 = +4.4054087194, rel_delta = 0.000005

V_so8c = +11.2968023928, V_so8xso8 = +11.2967753155, rel_delta = 0.000002

V_so8c = +15.7005232999, V_so8xso8 = +15.7004938190, rel_delta = 0.000002

V_so8c = +3.2290683270, V_so8xso8 = +3.2290466170, rel_delta = 0.000007

V_so8c = -3.3373855762, V_so8xso8 = -3.3374040149, rel_delta = 0.000006

V_so8c = +28.9523837775, V_so8xso8 = +28.9523424881, rel_delta = 0.000001

V_so8c = +90.8875944078, V_so8xso8 = +90.8875143673, rel_delta = 0.000001

V_so8c = -5.0578665526, V_so8xso8 = -5.0578830348, rel_delta = 0.000003

V_so8c = +5.9928537431, V_so8xso8 = +5.9928172719, rel_delta = 0.000006

##  Appendix B Scalar Potentials from Superpotentials

While numerics currently often appears to be the most powerful tool to study
the scalar potentials of maximal $D=4,5,6$ supergravities on the _full_ coset
manifolds $E_{d(+d)}/\mathcal{K}(E_{d(+d)})$, consistent truncation to maximal
sets of commuting $SU(1,1)\simeq SL(2)$ subgroups yields analytically rather
manageable expressions on these low-dimensional subspaces777This likely may be
a useful starting point for explorations of larger subspaces, observing that
the Fano plane also shows in the decomposition of $E_{7}$, respectively
$E_{8}$, into irreducible representations of $SL(2)^{\times 7,8}$..

Following the conventions of [2], we start from the Kähler potentials for the
product manifold of seven, respectively eight, Poincare discs:

$\mathcal{K}^{(7,8)}=-\sum_{j=1}^{7,\;\text{resp.}\;8}\log(1-\zeta_{j}\bar{\zeta}_{j}).$
(B.8)

From this, we obtain the Kähler metric and its inverse:

$\mathcal{K}_{a\bar{b}}:=\partial_{a}\partial_{\bar{b}}\mathcal{K},\quad\mathcal{K}^{a\bar{b}}=\left(\mathcal{K}_{a\bar{b}}\right)^{-1}.$
(B.9)

With the covariant derivative being given by

$\nabla_{a}(\cdot)=\partial_{a}(\cdot)+(\cdot)\partial_{a}\mathcal{K},$ (B.10)

the scalar potential of $\mathcal{N}=8,\;D=4\;SO(8)$ on
$\mathcal{M}_{14}=(SU(1,1)/U(1))^{\times 7}$ is given by

$V_{D=4|\mathcal{M}_{14}}=2\exp(\mathcal{K})\,\left(\mathcal{K}^{a\bar{b}}\nabla_{a}\mathcal{W}_{7}\nabla_{\bar{b}}\overline{\mathcal{W}_{7}}-3\,\mathcal{W}_{7}\overline{\mathcal{W}_{7}}\right),$
(B.11)

while the scalar potential of $\mathcal{N}=16,\;D=3\;SO(8)\times SO(8)$
supergravity on $\mathcal{M}_{16}:=(SU(1,1)/U(1))^{\times 8}$ is found to
match

$V_{D=3|\mathcal{M}_{16}}=2\exp(\mathcal{K})\,\left(\mathcal{K}^{a\bar{b}}\nabla_{a}\mathcal{W}_{8}\nabla_{\bar{b}}\overline{\mathcal{W}_{8}}-4\,\mathcal{W}_{8}\overline{\mathcal{W}_{8}}\right).$
(B.12)

In both cases, the superpotential can be read off from the $A_{1}$-tensor of
the model: For $D=4$, there is a 8-vector $X^{i}$ such that
$A^{1}_{ij}X^{i}X^{j}\cdot\prod_{k}\left(1-\zeta_{k}\bar{\zeta}_{k}\right)^{1/2}=\mathcal{W}_{7}$,
and in $D=3$, there is a 16-vector $Y^{I}$ such that
$A^{1}_{IJ}Y^{I}Y^{J}\cdot\prod_{k}\left(1-\zeta_{k}\bar{\zeta}_{k}\right)^{1/2}=\mathcal{W}_{8}$.

## References

  * [1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean et al., _TensorFlow: A system for large-scale machine learning_ , in _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_ , pp. 265–283, 2016, https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf.
  * [2] N. Bobev, T. Fischbacher and K. Pilch, _Properties of the new $\mathcal{N}$ = 1 AdS4 vacuum of maximal supergravity_, _JHEP_ 01 (2020) 099 [1909.10969].
  * [3] R.W. Hamming, _Error detecting and error correcting codes_ , _The Bell System Technical Journal_ 29 (1950) 147.
  * [4] T. Fischbacher, _The many vacua of gauged extended supergravities_ , _General Relativity and Gravitation_ 41 (2009) 315.
  * [5] H. Nicolai and H. Samtleben, _Maximal gauged supergravity in three-dimensions_ , _Phys. Rev. Lett._ 86 (2001) 1686 [hep-th/0010076].
  * [6] H. Nicolai and H. Samtleben, _Compact and noncompact gauged maximal supergravities in three-dimensions_ , _JHEP_ 04 (2001) 022 [hep-th/0103032].
  * [7] J.H. Conway and N.J.A. Sloane, _Sphere packings, lattices and groups_ , vol. 290, Springer Science & Business Media (2013).
  * [8] P. Belitz, _Applications on multi-dimensional sphere packings: derivative-free optimization_ , University of California, San Diego (2011).
  * [9] G. Dall’Agata and G. Inverso, _On the vacua of $\mathcal{N}$ = 8 gauged supergravity in 4 dimensions_, _Nuclear Physics B_ 859 (2012) 70.
  * [10] G. Dall’Agata, G. Inverso and M. Trigiante, _Evidence for a Family of SO(8) Gauged Supergravity Theories_ , _Physical Review Letters_ 109 (2012) 201301.
  * [11] B. de Wit and H. Nicolai, _Deformations of gauged SO(8) supergravity and supergravity in eleven dimensions_ , _JHEP_ 05 (2013) 077 [1302.6219].
  * [12] G. Dall’Agata, G. Inverso and A. Marrani, _Symplectic deformations of gauged maximal supergravity_ , _Journal of High Energy Physics_ 2014 (2014) .
  * [13] B. de Wit and H. Nicolai, _N=8 Supergravity_ , _Nucl. Phys. B_ 208 (1982) 323.
  * [14] D. Bermann, T. Fischbacher, G. Inverso and B. Scellier, _Vacua of dyonic SO(8) supergravity (in preparation)_ , .
  * [15] K. Lee, C. Strickland-Constable and D. Waldram, _New gaugings and non-geometry_ , 2015.
  * [16] G. Inverso, _Generalised scherk-schwarz reductions from gauged supergravity_ , _Journal of High Energy Physics_ 2017 (2017) .
  * [17] A. Guarino, D.L. Jafferis and O. Varela, _String theory origin of dyonic n=8 supergravity and its chern-simons duals_ , _Physical Review Letters_ 115 (2015) .
  * [18] A. Guarino and O. Varela, _Dyonic iso(7) supergravity and the duality hierarchy_ , _Journal of High Energy Physics_ 2016 (2016) .
  * [19] A. Khavaev, K. Pilch and N.P. Warner, _New vacua of gauged N=8 supergravity in five-dimensions_ , _Phys. Lett. B_ 487 (2000) 14 [hep-th/9812035].
  * [20] N. Warner, _Some new extrema of the scalar potential of gauged $\mathcal{N}$ = 8 supergravity_, _Physics Letters B_ 128 (1983) 169.
  * [21] T. Fischbacher, H. Nicolai and H. Samtleben, _Vacua of maximal gauged D = 3 supergravities_ , _Class. Quant. Grav._ 19 (2002) 5297 [hep-th/0207206].
  * [22] J. Baez, “This week’s finds in mathematical physics (week 90).”
  * [23] I.M. Comsa, M. Firsching and T. Fischbacher, _SO(8) Supergravity and the Magic of Machine Learning_ , _JHEP_ 08 (2019) 057 [1906.00207].
  * [24] A. Borghese, A. Guarino and D. Roest, _Triality, Periodicity and Stability of SO(8) Gauged Supergravity_ , _JHEP_ 05 (2013) 107 [1302.6057].
  * [25] T. Fischbacher, K. Pilch and N.P. Warner, _New supersymmetric and stable, non-supersymmetric phases in supergravity and holographic field theory_ , 2010\.
  * [26] L. Borsten, M.J. Duff and P. Lévay, _The black-hole/qubit correspondence: an up-to-date review_ , _Classical and Quantum Gravity_ 29 (2012) 224008.
  * [27] M. Gunaydin, R. Kallosh, A. Linde and Y. Yamada, _M-theory Cosmology, Octonions, Error Correcting Codes_ , _JHEP_ 01 (2021) 160 [2008.01494].
  * [28] J.A. Wheeler, _Information, physics, quantum: The search for links_ , in _3rd International Symposium on Foundations of Quantum Mechanics in Light_ , 1989.
  * [29] R. Gebert and H. Nicolai, _E10 for beginners_ , in _Strings and Symmetries_ , G. Aktaş, C. Saçlioğlu and M. Serdaroğlu, eds., (Berlin, Heidelberg), pp. 197–210, Springer Berlin Heidelberg, 1995.
  * [30] S. Ananth, L. Brink and S. Majumdar, _E 8 in $\mathcal{N}=8$ supergravity in four dimensions_, _JHEP_ 01 (2018) 024 [1711.09110].