mathx"17

# DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognition

Sijie Wang1, Rui She1, Qiyu Kang1111Corresponding author: Qiyu Kang., Xingchao
Jian1, Kai Zhao1, Yang Song2, Wee Peng Tay1

###### Abstract

The utilization of multi-modal sensor data in visual place recognition (VPR)
has demonstrated enhanced performance compared to single-modal counterparts.
Nonetheless, integrating additional sensors comes with elevated costs and may
not be feasible for systems that demand lightweight operation, thereby
impacting the practical deployment of VPR. To address this issue, we resort to
knowledge distillation, which empowers single-modal students to learn from
cross-modal teachers without introducing additional sensors during inference.
Despite the notable advancements achieved by current distillation approaches,
the exploration of feature relationships remains an under-explored area. In
order to tackle the challenge of cross-modal distillation in VPR, we present
DistilVPR, a novel distillation pipeline for VPR. We propose leveraging
feature relationships from multiple agents, including self-agents and cross-
agents for teacher and student neural networks. Furthermore, we integrate
various manifolds, characterized by different space curvatures for exploring
feature relationships. This approach enhances the diversity of feature
relationships, including Euclidean, spherical, and hyperbolic relationship
modules, thereby enhancing the overall representational capacity. The
experiments demonstrate that our proposed pipeline achieves state-of-the-art
performance compared to other distillation baselines. We also conduct
necessary ablation studies to show design effectiveness. The code is released
at: https://github.com/sijieaaa/DistilVPR

##  1 Introduction

Figure 1: The pipeline of cross-modal KD to transfer knowledge from the cross-
modal teacher to single-modal students.

Visual place recognition (VPR) serves as a foundational task in localization,
aiming at identifying locations by comparing visual sensor data, such as
camera images and LiDAR point clouds, to stored references in a database. This
task finds application in diverse domains, including autonomous driving (Chen
et al. 2023), precise positioning (Sarlin et al. 2019), and augmented reality
(Sarlin et al. 2022).

Traditional VPR solutions rely on handcrafted features like Vector of Locally
Aggregated Descriptor (VLAD) (Jégou et al. 2011) and Bag of Words (BoW)
(Gálvez-López and Tardos 2012). These methods often fall short in challenging
conditions including changing lighting, view distortions, and environmental
perturbations due to their dependence on manual design.

The rise of deep learning has inspired data-driven VPR approaches that can
tackle these challenges. NetVLAD (Arandjelovic et al. 2016) skillfully
combines deep convolutional networks (CNNs) and traditional VLAD to enhance
the robustness and efficacy of image scene feature extraction. This innovation
paved the way for subsequent learning-based strategies. Addressing camera
sensitivity to illumination, PointNetVLAD (Uy and Lee 2018) suggests utilizing
point clouds from LiDARs which, unlike cameras, actively project laser beams
to perceive surroundings, thus rendering them more resistant to lighting
variations. Moreover, integrating data from multiple sensors can yield a more
resilient and high-performing VPR model. In this vein, MinkLoc++ (Komorowski,
Wysoczańska, and Trzcinski 2021) leverages both images and point clouds as
inputs to achieve efficient multi-modal feature extraction, showcasing
superiority over single-modal alternatives.

While integrating various sensors can elevate model performance, it also
incurs additional expenses. Moreover, lightweight mobile systems might not
support heavy sensors, such as LiDARs, making multi-modal sensors impractical.

Although using multiple sensors during inference is not favored, we can
harness this cross-modal knowledge during student model training. This is
where cross-modal knowledge distillation (KD) enters the picture.
Specifically, in the student training phase, as depicted in Fig. 1, distinct
modalities can be fed into the pre-trained teacher model. The extracted
teacher features can then guide single-modal student models in learning
superior features through additional supervision. During inference, student
models can still rely on single-modal data, eliminating the need to
accommodate multiple sensors.

Given the inconsistency in feature embedding across different modalities,
directly compelling students to learn teacher features would be intricate. In
contrast, the relational KD paradigm (Park et al. 2019), which delves into
feature relationships, offers a more suitable approach to address this
inconsistency. However, the vanilla relational KD solution only considers
feature relationships in limited embedding spaces, and they restrict
relationship computing within the same knowledge agents (i.e. either teacher-
teacher relationships or student-student relationships). These limitations
hinder the efficient transfer of knowledge from teachers to students.

To mitigate these issues, we propose DistilVPR, a novel cross-modal
distillation pipeline for VPR. Our contributions can be summarized as follows:

  * •

We present DistilVPR, a cross-modal KD solution uniquely tailored for VPR.
This framework extends the scope of feature relationships, encompassing both
self-agents and cross-agents to facilitate a more comprehensive exploration of
knowledge. In addition, our approach performs feature embedding in multiple
manifolds with diverse feature geodesic measurements, enhancing the
construction of effective feature relationships

  * •

Through extensive experiments, we showcase the remarkable performance of
DistilVPR when compared to previous KD baselines. Our approach achieves state-
of-the-art (SOTA) performance in the task of cross-modal distillation for VPR.
Furthermore, we rigorously investigate our design through vital ablation
studies, providing empirical evidence of the efficacy of our proposed
methodology.

##  2 Related Work and Preliminary

In this section, we introduce related works and necessary manifold
preliminaries.

###  2.1 Visual Place Recognition

NetVLAD (Arandjelovic et al. 2016) pioneers the combination of the traditional
VLAD descriptor and the CNN to construct a learnable aggregation layer. Its
success has paved the road for many VPR models. PointNetVLAD (Uy and Lee 2018)
leverages point clouds instead of images to conduct place recognition. The
point cloud features are extracted by PointNet (Qi et al. 2017) and then fed
into a NetVLAD layer to produce the final global descriptor of the scene.
MinkLoc3D (Komorowski 2021) is built based on a sparse 3D CNN for point cloud
feature expression.

The aforementioned works use either images or point clouds for VPR. We now
review approaches that take the multi-modal fusion strategy. Cues-Net (Oertel,
Cieslewski, and Scaramuzza 2020) generates pseudo 3D point clouds from image
sequences using Direct Sparse Odometry (DSO)(Engel, Koltun, and Cremers 2017).
PIC-Net (Lu et al. 2020) transforms night images into the daytime style to
reduce the impact of illumination perturbations on images. CORAL (Pan et al.
2021) projects the 3D point cloud using bird’s-eye-view (BEV) mapping, such
that a 2D image backbone can be applied on both the point cloud branch and the
image branch. MinkLoc++ (Komorowski, Wysoczańska, and Trzcinski 2021) follows
the style of the MinkLoc3D series to achieve sparse 3D feature representation.
The final global descriptor is concatenated with the 2D image descriptor and
3D point cloud descriptor. AdaFusion (Lai, Yin, and Scherer 2022) leverages a
multi-scale attention module that hierarchically aggregates multi-modal
features.

###  2.2 Knowledge Distillation

KD has emerged as a pivotal technique in model compression and multi-modal
learning, enabling the transfer of knowledge from complex teacher models to
compact or cross-modal student models. Vanilla KD (Hinton, Vinyals, and Dean
2015) first introduces the concept of KD to compress the knowledge from larger
teacher models to smaller student models. RKD (Park et al. 2019) emphasizes
the self-relationships present in the data samples of both teacher and student
outputs. This approach serves as an implicit distillation solution,
facilitating the transfer of the teacher’s knowledge to the student model. AFD
(Ji, Heo, and Park 2021) employs an attention-based meta-network to acquire
relative similarities among features and then employs these identified
similarities to regulate the intensity of distillation for all feasible pairs.
MKD (Jin, Wang, and Lin 2023) conducts prediction alignment at the instance of
three different levels simultaneously, which include the instance, batch, and
class levels.

There are also some other distillation works focusing on various tasks. 2DPass
(Yan et al. 2022) employs an innovative approach to enhance semantic
information extraction from multi-modal data with the integration of two key
components: auxiliary modal fusion and multi-scale fusion-to-single
distillation. LSD-Net (Peng et al. 2022) leverages dual distillation to
transfer teacher patterns into students for lightweight VPR. EPC-Net (Hui et
al. 2022) proposes ProxyConv, which is a lightweight module for local
geometric feature aggregation. It uses a grouped VLAD network to form the
global descriptors. To train its more lightweight version, the final feature
is distilled from the larger teacher network to the smaller student network.
CSD (Wu et al. 2022) represents a flexible framework for asymmetric similarity
distillation to enhance the small query model for image retrieval. UniDistill
(Zhou et al. 2023) digs into BEV object detection and leverages KD from
features, relationships, and responses. LiDAR2Map (Wang et al. 2023b) presents
an online camera-to-LiDAR distillation scheme to facilitate semantic
information from images to point clouds for semantic map segmentation.

###  2.3 Manifold Preliminary

The concept of a manifold (Zhao et al. 2023; Wang et al. 2023a; She et al.
2023; Shi et al. 2023) serves as a generalization of surfaces in higher
dimensions, extending the notion of well-behaved geometrical structures. A
manifold $\mathcal{M}$ is a topological space that locally resembles the
Euclidean space near each point $p\in\mathcal{M}$. For each point $p$, it is
possible to establish a homeomorphism between a neighborhood of $p$ and the
Euclidean space.

The tangent space $T_{p}\mathcal{M}$ at a point $p$ on $\mathcal{M}$ can be
visualized as a hyperplane that provides the best approximation of
$\mathcal{M}$ in the vicinity of $p$. Alternatively, $T_{p}\mathcal{M}$ is the
space that encompasses all the possible directions of curves on $\mathcal{M}$
passing through $p$. The elements residing within $T_{p}\mathcal{M}$ are
referred to as tangent vectors. Essentially, the tangent space
$T_{p}\mathcal{M}$ characterizes the local linear approximation of
$\mathcal{M}$ near the point $p$. It captures the intrinsic geometry of
$\mathcal{M}$.

A metric tensor $g_{p}$ is an additional structure associated with each point
$p$ on a manifold $\mathcal{M}$. By smoothly varying across $\mathcal{M}$, the
metric tensor provides a consistent way to measure distances throughout the
manifold. Given two points $p,q\in\mathcal{M}$, the geodesic distance $d(p,q)$
is obtained as the shortest length of curves that connect point $p$ and $q$.

##  3 Proposed Pipeline

In this section, we first provide the problem formulation. Then, we introduce
the DistilVPR architecture in detail.

###  3.1 Problem Formulation

In this study, we address the challenge of cross-modal KD for VPR. We focus on
a scenario where a pre-trained teacher model is provided, capable of
processing images and/or point clouds as inputs for multi-modal VPR. The
single-modal student models accept either image inputs or point cloud inputs.
Our objective is to distill the teacher’s knowledge to the students,
empowering them to acquire enhanced understanding during training. This, in
turn, improves student performance during inference without the requirement
for cross-modal sensors.

Specifically, we denote a batch of teacher outputs as222We denote
$[N]=\\{1,\ldots,N\\}$ for simplification.
$\mathbf{T}=\left\\{\mathbf{t}_{i}\in\mathbb{R}^{C}\nonscript\,:\allowbreak\nonscript\,\mathopen{}i\in[N]\right\\}$
and student outputs as
$\mathbf{S}=\left\\{\mathbf{s}_{i}\in\mathbb{R}^{C}\nonscript\,:\allowbreak\nonscript\,\mathopen{}i\in[N]\right\\}$,
with the batch size $N$ and the same output channel size333We assume the
teacher and the student have the same output channel size. $C$ .

###  3.2 Relational Distillation

There are typically two ways to conduct KD, including direct KD and relational
KD. Direct KD is a straightforward way that directly applies sample-wise
supervision by minimizing the loss

$\displaystyle\mathcal{L}_{\rm{direct}}=\sum_{i\in[N]}\ell\left(\mathbf{t}_{i},\mathbf{s}_{i}\right),$
(1)

where $\ell\left(\cdot\right)$ denotes the loss function. This approach pulls
student embeddings towards teacher embeddings, which can be regarded as
sample-wise supervision.

By contrast, relational KD does not apply explicit sample-wise supervision.
Instead, it measures inter-sample relationships, which can be regarded as
implicit knowledge. Relational KD is formed by minimizing

$\displaystyle\mathcal{L}_{\rm{relationship}}=\sum_{i,j\in[N]}\
\ell\left(r\left(\mathbf{t}_{i},\mathbf{t}_{j}\right),\
r\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)\ \right),$ (2)

where $r(\cdot,\cdot)$ is the relational function to compute embedding
distances.

Through our experiments, we have observed that compared with direct KD,
relational KD is inherently a better choice for cross-modal KD in VPR for the
following reasons. On one hand, in VPR, places are recognized by computing
query-database similarity, where the training goal is to minimize the query-
positive distance and maximize the query-negative distance. The relative
feature relationships are more critical than the absolute feature embeddings,
for which the relational KD scheme that explores relative embedding distance
would be a more suitable solution for VPR. On the other hand, cross-modal
features may have inherently different embedding patterns. Thus it would be
intractable to force single-modal features to be embedded in the same space as
multi-modal features using direct KD schemes. Based on these insights, we
follow the relational KD scheme in 2 to design a more efficient cross-modal
distillation solution.

###  3.3 Multi-agent Relationship

We generically call a teacher output $\mathbf{t}_{i}$ or student output
$\mathbf{s}_{i}$ an _agent_. One limitation of the basic relational KD is that
it confines the computation of relationships within the same type of agent,
i.e., teacher-teacher $r(\mathbf{t}_{i},\mathbf{t}_{j})$ and student-student
$r(\mathbf{s}_{i},\mathbf{s}_{j})$. Despite relational KD being able to
achieve considerably better performance than direct KD counterparts, it lacks
a more generalized consideration of the combination of different agents.

Figure 2: Three generalized relational KD schemes.

To generalize the combination of different agents, there are three scenarios
for relationship computation as shown in Fig. 2. We expand 2 to further
explore not only the self-agent relationships,
$r(\mathbf{t}_{i},\mathbf{t}_{j})$ and $r(\mathbf{s}_{i},\mathbf{s}_{j})$, but
also the cross-agent relationship, $r(\mathbf{t}_{i},\mathbf{s}_{j})$.
Specifically, the three generalized relational KD losses are formulated as:

$\displaystyle\mathcal{L}_{\rm{tt-ss}}=\sum_{i,j\in[N]}\
\ell\left(r\left(\mathbf{t}_{i},\mathbf{t}_{j}\right),\
r\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)\ \right),$ (3)
$\displaystyle\mathcal{L}_{\rm{ts-ss}}=\sum_{i,j\in[N]}\
\ell\left(r\left(\mathbf{t}_{i},\mathbf{s}_{j}\right),\
r\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)\ \right),$ (4)
$\displaystyle\mathcal{L}_{\rm{tt-ts}}=\sum_{i,j\in[N]}\
\ell\left(r\left(\mathbf{t}_{i},\mathbf{t}_{j}\right),\
r\left(\mathbf{t}_{i},\mathbf{s}_{j}\right)\ \right).$ (5)

From above, 3 is the vanilla relational KD scheme in 2. By contrast, 4 and 5
are two additional schemes with 5 used in CSD (Wu et al. 2022). In the two
schemes, cross-agent relationship
$r\left(\mathbf{t}_{i},\mathbf{s}_{j}\right)$ bridges the gap between teacher
features and student features. The additional information would contribute to
a more effective KD process.

Comparing 4 and 5, we have empirically found that 4 generally outperforms 5,
as seen in Table 4. This may be attributed to the fact that within the four
variables in 5, only one pertains to the learnable student features, while the
remaining three variables are associated with the fixed teacher features.
Consequently, the optimal solution domain for minimizing 5 becomes
constrained. For example, considering $r$ as the Euclidean distance function,
the optimal solution is confined to a sphere. Similarly, this constraint could
potentially elucidate why direct KD in 1 yields inferior results compared to
relational KD, given that direct KD also involves only one variable for
student features, resulting in the optimal solution domain that is a single
point.

Based on these insights, we thus use 4 to compute cross-agent relationships,
along with 3 for self-agent relationships. These distinct relationship
patterns constitute the core components of our approach.

###  3.4 Multi-manifold Relationship

Since different modal features may not be embedded similarly, it becomes
essential to adopt a more comprehensive metric for measuring agent
relationships. Consequently, we introduce combined manifold spaces to augment
the effectiveness of relational KD.

Different feature manifolds can be categorized based on their curvature. The
Euclidean space represents the most prevalent manifold with zero curvature,
while the spherical manifold exhibits positive curvature, and the hyperbolic
manifold has negative curvature. By amalgamating multiple manifolds, we can
facilitate features to possess more comprehensive embedding relationships by
leveraging distinct geodesic distances.

#### Euclidean Relationship.

The Euclidean space serves as a prominent example of a flat manifold,
exhibiting zero curvature across all points. Within Euclidean space, the
calculation of the geodesic distance between any two points is given by the
conventional Euclidean distance formula. The distance $d_{\rm{euc}}$ is the
straight-line distance between two points $\mathbf{x},\mathbf{y}$ in a
Cartesian coordinate system given by

$\displaystyle
d_{\rm{euc}}\left(\mathbf{x},\mathbf{y}\right)=\lVert\mathbf{x}-\mathbf{y}\rVert,$
(6)

where $\lVert\cdot\rVert$ denotes the $\mathcal{L}_{2}$ norm.

In our work, the Euclidean distance yields the Euclidean-based losses as:

$\displaystyle\mathcal{L}^{\rm{euc}}_{\rm{tt-ss}}=\sum_{i,j\in[N]}\ell\left(\
d_{\rm{euc}}\left(\mathbf{t}_{i},\mathbf{t}_{j}\right),\
d_{\rm{euc}}\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)\ \right),$ (7)
$\displaystyle\mathcal{L}^{\rm{euc}}_{\rm{ts-ss}}=\sum_{i,j\in[N]}\ell\left(\
d_{\rm{euc}}\left(\mathbf{t}_{i},\mathbf{s}_{j}\right),\
d_{\rm{euc}}\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)\ \right).$ (8)

#### Spherical Relationship.

The second relationship we consider is the spherical relationship. In contrast
to Euclidean space, the spherical manifold displays a distinct characteristic
by possessing a constant positive curvature. The geodesic distance between two
points is calculated based on the angular separation between the points and
the radius of the sphere. Following previous works (Zhou et al. 2023; Hou et
al. 2022), we adopt the cosine distance to explore the spherical-based
relationship, which is given by

$\displaystyle
d_{\rm{cos}}(\mathbf{x},\mathbf{y})=\frac{\langle\mathbf{x},\mathbf{y}\rangle}{\lVert\mathbf{x}\rVert\lVert\mathbf{y}\rVert},$
(9)

where $\langle\cdot,\cdot\rangle$ denotes the inner product.

Then we incorporate the cosine distance as the second consideration to explore
positive-curvature relationships, and the losses are formulated as:

$\displaystyle\mathcal{L}^{\rm{cos}}_{\rm{tt-
ss}}=\sum_{i,j\in[N]}\ell\left(d_{\rm{cos}}(\mathbf{t}_{i},\mathbf{t}_{j}),d_{\rm{cos}}(\mathbf{s}_{i},\mathbf{s}_{j})\right),$
(10) $\displaystyle\mathcal{L}^{\rm{cos}}_{\rm{ts-
ss}}=\sum_{i,j\in[N]}\ell\left(d_{\rm{cos}}(\mathbf{t}_{i},\mathbf{s}_{j}),d_{\rm{cos}}(\mathbf{s}_{i},\mathbf{s}_{j})\right).$
(11)

#### Hyperbolic Relationship.

A comprehensive relationship evaluation would benefit more from various
feature pattern exploration of KD agents, and it can thus contribute to more
effective KD from the teacher to the cross-modal student. However, the above
two measurements explore feature relationships in either zero-curvature or
positive-curvature manifolds as in RKD (Park et al. 2019). There is a lack of
consideration of relationships in negative curvature manifolds, which would
result in insufficient KD. To this end, we introduce the third relationship
based on the negative curvature manifold.

In Riemannian geometry, the hyperbolic space is defined as the Riemannian
manifold with constant negative curvature. The Poincaré ball is the most
common conformal model of hyperbolic geometry. It has been used to embed
features in various tasks (Tifrea, Bécigneul, and Ganea 2018; Liu, Nickel, and
Kiela 2019; Wang et al. 2023a). The $n$-dimensional Poincaré ball is defined
on
$\mathbb{D}_{c}^{n}=\left\\{\mathbf{p}\in\mathbb{R}^{n}\nonscript\,:\allowbreak\nonscript\,\mathopen{}c\lVert\mathbf{p}\rVert<1\right\\}$
with curvature $-c^{2}$. The Poincaré ball is equipped with a constant metric
tensor $\mathbf{g}=\lambda^{2}_{c}\mathbf{I}^{n}$, where
$\lambda_{c}=\frac{2}{1-c\lVert\mathbf{p}\rVert^{2}}$ is the conformal factor.

Given a pair $\mathbf{p},\mathbf{q}\in\mathbb{D}_{c}^{n}$, the mobius addition
$\oplus_{c}$ is defined as:

$\displaystyle\mathbf{p}\oplus_{c}\mathbf{q}=\frac{\left(1+2c\langle\mathbf{p},\mathbf{q}\rangle+c\lVert\mathbf{q}\rVert^{2}\right)\mathbf{p}+\left(1-c\lVert\mathbf{p}\rVert^{2}\right)\mathbf{q}}{1+2c\langle\mathbf{p},\mathbf{q}\rangle+c^{2}\lVert\mathbf{p}\rVert^{2}\lVert\mathbf{q}\rVert^{2}}.$
(12)

For a fixed base point $\mathbf{z}\in\mathbb{D}_{c}^{n}$, the exponential
mapping function
$\exp_{\mathbf{z}}^{c}:\mathbb{R}^{n}\rightarrow\mathbb{D}_{c}^{n}$ maps
points from the tangent Euclidean space to the hyperbolic space:

$\displaystyle\exp_{\mathbf{z}}^{c}(\mathbf{v})=\mathbf{z}\oplus_{c}\left(\tanh\left(\sqrt{c}\frac{\lambda_{c}\lVert\mathbf{v}\rVert}{2}\right)\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}\right).$
(13)

By setting the origin as the fixed base point, the exponential map can be
simplified as

$\displaystyle\exp^{c}_{0}(\mathbf{v})=\tanh\left(\sqrt{c}\lVert\mathbf{v}\rVert\right)\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}.$
(14)

After exponential mapping, the geodesic distance in the hyperbolic manifold
(hyperbolic distance) can be obtained as

$\displaystyle
d_{\rm{hyp}}(\mathbf{p},\mathbf{q})=\frac{2}{\sqrt{c}}\operatorname{arctanh}\left(\sqrt{c}\lVert-\mathbf{p}\oplus_{c}\mathbf{q}\rVert\right).$
(15)

In our work, we embed both teacher outputs and student outputs in the Poincaré
ball, and the hyperbolic losses are computed as:

$\displaystyle\mathcal{L}_{\rm{tt-ss}}^{\rm{hyp}}=\sum_{i,j\in[N]}\ell\left(\
d_{\rm{hyp}}\left(\mathbf{t}_{i}^{\rm{hyp}},\mathbf{t}_{j}^{\rm{hyp}}\right),d_{\rm{hyp}}\left(\mathbf{s}_{i}^{\rm{hyp}},\mathbf{s}_{j}^{\rm{hyp}}\right)\
\right),$ (16) $\displaystyle\mathcal{L}_{\rm{ts-
ss}}^{\rm{hyp}}=\sum_{i,j\in[N]}\ell\left(\
d_{\rm{hyp}}\left(\mathbf{t}_{i}^{\rm{hyp}},\mathbf{s}_{j}^{\rm{hyp}}\right),d_{\rm{hyp}}\left(\mathbf{s}_{i}^{\rm{hyp}},\mathbf{s}_{j}^{\rm{hyp}}\right)\
\right),$ (17)

where $\mathbf{t}_{i}^{\rm{hyp}}=\rm{exp}^{c}_{0}(\mathbf{t}_{i})$ and
$\mathbf{s}_{i}^{\rm{hyp}}=\rm{exp}^{c}_{0}(\mathbf{s}_{i})$ are hyperbolic
teacher and student embeddings, respectively.

###  3.5 Overall Loss Function

Finally, we combine the insights from multiple agents and multiple manifolds
to construct our distillation pipeline. Specifically, we first formulate two
distillation losses, including the self-agent distillation loss
$\mathcal{L}_{\rm{KD-S}}$ and the cross-agent distillation loss
$\mathcal{L}_{\rm{KD-C}}$ respectively as:

$\displaystyle\mathcal{L}_{\rm{KD-S}}=\mathcal{L}_{\rm{tt-
ss}}^{\rm{euc}}+\mathcal{L}_{\rm{tt-ss}}^{\rm{cos}}+\mathcal{L}_{\rm{tt-
ss}}^{\rm{hyp}},$ (18)
$\displaystyle\mathcal{L}_{\rm{KD-C}}=\mathcal{L}_{\rm{ts-
ss}}^{\rm{euc}}+\mathcal{L}_{\rm{ts-ss}}^{\rm{cos}}+\mathcal{L}_{\rm{ts-
ss}}^{\rm{hyp}}.$ (19)

Subsequently, with weight hyperparameters $\lambda_{\rm{S}},\lambda_{\rm{C}}$
and the triplet loss as the VPR task loss $\mathcal{L}_{\rm{task}}$, we
propose three different overall losses. They are denoted as DistilVPR-S,
DistilVPR-C, and DistilVPR-SC, respectively:

$\displaystyle\mathcal{L}_{\rm{DistilVPR-S}}=\mathcal{L}_{\rm{task}}+\lambda_{\rm{S}}\mathcal{L}_{\rm{KD-S}},$
(20)
$\displaystyle\mathcal{L}_{\rm{DistilVPR-C}}=\mathcal{L}_{\rm{task}}+\lambda_{\rm{C}}\mathcal{L}_{\rm{KD-C}},$
(21) $\displaystyle\mathcal{L}_{\rm{DistilVPR-
SC}}=\mathcal{L}_{\rm{task}}+\lambda_{\rm{S}}\mathcal{L}_{\rm{KD-S}}+\lambda_{\rm{C}}\mathcal{L}_{\rm{KD-C}}.$
(22)

##  4 Experiments

In this section, we conduct experiments to compare DistilVPR defined in 20, 21
and 22 with other KD baselines. We also provide necessary ablation studies to
verify the design efficacy.

###  4.1 Datasets and Implementation Details

#### Oxford RobotCar.

The Oxford RobotCar dataset (Maddern et al. 2017) is a large-scale autonomous
driving dataset, which provides a rich collection of sensor data, including
images and point clouds. It also encompasses various driving scenarios with
different weather conditions, traffic patterns, and pedestrian interactions.
We use the processed point clouds provided by PointNetVLAD(Uy and Lee 2018)
which is the standard benchmark data for point cloud and multi-modal (image +
point cloud) place recognition. Since it is equipped with both images and
point clouds, the Oxford RobotCar dataset would be a suitable platform to test
the performance of multi-modal teachers and single-modal students.

#### Boreas.

The Boreas dataset (Burnett et al. 2022) is gathered by conducting multiple
drives along a consistent route over one year, thereby capturing notable
seasonal fluctuations. It comprises an extensive collection of over 350 km of
driving data, featuring numerous sequences recorded under challenging weather
conditions, including rain, heavy snow, and night. It also provides multi-
modal sensor data such as images and point clouds, and thus can also serve as
a benchmark for both multi-modal and single-modal models.

#### Implementation Details.

We choose two SOTA multi-modal place recognition models as teachers, including
MinkLoc++ (Komorowski, Wysoczańska, and Trzcinski 2021) and AdaFusion (Lai,
Yin, and Scherer 2022). We use their single-modal branches as students to test
the effectiveness of cross-modal KD. We use the Adam optimizer to train both
teachers and students. The learning rate is set as $1e-4$ and $1e-3$ for the
image branch and the point cloud branch respectively. Both teacher models and
student models are trained for $60$ epochs with $128$ batch size. All
experiments are conducted on an A100 GPU. We follow previous works to use the
same evaluation protocol, including Average Recall@1 (AR@1) and Average
Recall@1% (AR@1%). More details are provided in the supplement.

###  4.2 Main Results

#### Fusion-to-single Distillation.

As shown in Table 1 and Table 2, our proposed three KD schemes can achieve
considerably better performance compared with other counterparts in the Oxford
and the Boreas datasets. In addition, our schemes can handle various fusion-
to-single KD tasks, including fusion-to-2D and fusion-to-3D, which further
underscores the efficacy and generalization ability. We have also noticed that
relational KD schemes generally outperform the direct KD counterparts, which
shows that the key to effective distillation for VPR lies in the exploration
of feature relationships rather than mere feature alignment.

Moreover, we have found that the 3D point cloud inputs can always contribute
better VPR performance compared with the 2D image inputs. This trend holds
across both datasets, with the gap being particularly pronounced in the more
challenging Boreas dataset. This observation reinforces the assertion that
utilizing point cloud data is pivotal in achieving effective VPR results.

Distillation Method | T: MinkLoc++ | T: MinkLoc++ | T: AdaFusion | T: AdaFusion
---|---|---|---|---
S: MinkLoc++2D | S: MinkLoc++3D | S: AdaFusion-2D | S: AdaFusion-3D
AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1
Teacher | 99.4 | 97.2 | 99.4 | 97.2 | 99.0 | 96.6 | 99.0 | 96.6
Student w/o distil. | 94.7 | 85.7 | 98.1 | 94.4 | 94.2 | 84.2 | 98.0 | 93.8
*KD (Hinton, Vinyals, and Dean 2015) | 95.2 | 84.6 | 98.1 | 94.3 | 95.2 | 84.6 | 97.8 | 93.7
AFD (Ji, Heo, and Park 2021) | 95.2 | 84.7 | 98.1 | 94.2 | 94.5 | 82.9 | 97.8 | 93.2
EPC-Net (Hui et al. 2022) | 95.4 | 85.6 | 97.9 | 93.8 | 95.3 | 85.1 | 98.1 | 93.7
RKD (Park et al. 2019) | 96.5 | 88.5 | 98.3 | 94.5 | 96.2 | 87.3 | 98.2 | 94.4
CSD (Wu et al. 2022) | 95.4 | 86.0 | 98.1 | 94.4 | 95.3 | 85.4 | 98.0 | 93.8
LSD-Net (Peng et al. 2022) | 95.8 | 86.1 | 98.2 | 94.1 | 95.9 | 86.2 | 97.9 | 93.9
MKD (Jin, Wang, and Lin 2023) | 95.0 | 85.4 | 98.1 | 93.9 | 95.1 | 84.5 | 97.8 | 93.7
(ours) DistilVPR-S | 96.7 | 88.7 | 98.3 | 95.2 | 96.2 | 87.4 | 98.0 | 94.2
(ours) DistilVPR-C | 97.3 | 91.1 | 98.1 | 94.4 | 96.6 | 88.8 | 98.0 | 93.7
(ours) DistilVPR-SC | 97.0 | 90.0 | 98.3 | 94.6 | 96.7 | 89.0 | 98.3 | 94.7
Table 1: Fusion-to-single distillation comparison on the Oxford RobotCar dataset. "T:" and "S:" stand for the teacher model and the student model respectively. Direct distillation solutions are marked with "*", while relational solutions are without any mark. The best results are bold and underlined, while the second-best results are underlined only.  Distillation Method | T: MinkLoc++ | T: MinkLoc++ | T: AdaFusion | T: AdaFusion
---|---|---|---|---
S: MinkLoc++2D | S: MinkLoc++3D | S: AdaFusion-2D | S: AdaFusion-3D
AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1
Teacher | 98.9 | 93.1 | 98.9 | 93.1 | 98.9 | 93.2 | 98.9 | 93.2
Student w/o distil. | 75.2 | 60.0 | 98.5 | 91.0 | 74.5 | 59.6 | 98.9 | 91.5
*KD (Hinton, Vinyals, and Dean 2015) | 75.8 | 61.4 | 98.1 | 90.4 | 76.9 | 60.3 | 98.5 | 91.7
AFD (Ji, Heo, and Park 2021) | 75.5 | 60.4 | 97.4 | 88.4 | 75.9 | 58.5 | 98.7 | 92.7
EPC-Net (Hui et al. 2022) | 75.3 | 60.8 | 98.0 | 89.8 | 75.4 | 60.5 | 98.9 | 92.4
RKD (Park et al. 2019) | 78.0 | 62.9 | 99.1 | 91.6 | 78.8 | 62.5 | 98.9 | 93.9
CSD (Wu et al. 2022) | 76.3 | 61.4 | 98.2 | 90.9 | 77.0 | 61.2 | 99.1 | 92.8
LSD-Net (Peng et al. 2022) | 74.5 | 59.3 | 98.7 | 92.0 | 76.7 | 60.4 | 98.5 | 92.2
MKD (Jin, Wang, and Lin 2023) | 77.6 | 61.3 | 96.9 | 88.2 | 76.5 | 61.0 | 98.9 | 92.2
(ours) DistilVPR-S | 77.3 | 63.4 | 98.5 | 92.1 | 80.1 | 64.1 | 99.3 | 94.0
(ours) DistilVPR-C | 77.0 | 65.1 | 97.8 | 90.5 | 79.7 | 64.7 | 98.8 | 92.3
(ours) DistilVPR-SC | 79.3 | 67.2 | 98.3 | 91.3 | 78.0 | 65.5 | 99.3 | 93.6
Table 2: Fusion-to-single distillation comparison on the Boreas dataset.

#### 3D-to-2D and Big-to-small Distillation.

We evaluate the cross-modal distillation performance by training teachers with
pure 3D point cloud inputs and students with pure 2D images. As illustrated in
Table 3, the distinct advantages of DistilVPR become more evident in this
context. Notably, in the 3D-to-2D scenarios, DistilVPR-SC exhibits notably
superior performance compared to other baselines. This result underscores the
pronounced effectiveness of our methodology in addressing the intricate
challenge of distillation across disparate modalities. We also assess the
basic scenario of distillation from a larger model to a smaller one, as
presented in Table 3. In this setting, our proposed approach continues to
demonstrate effective distillation performance.

Distillation Method | T: MinkLoc++2D-Big | T: MinkLoc++3D | T: AdaFusion-2D-Big | T: AdaFusion-3D
---|---|---|---|---
S: MinkLoc++2D | S: MinkLoc++2D | S: AdaFusion-2D | S: AdaFusion-2D
AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1 | AR@1% | AR@1
Teacher | 80.3 | 66.4 | 98.5 | 91.0 | 87.8 | 64.7 | 98.9 | 91.5
Student w/o distil. | 75.2 | 60.0 | 75.2 | 60.0 | 74.5 | 59.6 | 74.5 | 59.6
*KD (Hinton, Vinyals, and Dean 2015) | 75.1 | 60.1 | 74.9 | 58.6 | 76.8 | 60.6 | 75.8 | 59.0
AFD (Ji, Heo, and Park 2021) | 77.3 | 62.5 | 75.3 | 56.5 | 76.2 | 58.5 | 76.5 | 59.4
EPC-Net (Hui et al. 2022) | 74.8 | 59.2 | 73.5 | 58.6 | 77.3 | 60.4 | 75.2 | 60.5
RKD (Park et al. 2019) | 76.4 | 61.7 | 76.2 | 60.4 | 75.8 | 61.5 | 77.4 | 61.4
CSD (Wu et al. 2022) | 77.3 | 60.3 | 76.2 | 60.3 | 76.1 | 60.2 | 76.8 | 61.0
LSD-Net (Peng et al. 2022) | 77.6 | 61.9 | 75.1 | 57.0 | 74.8 | 60.3 | 74.7 | 59.1
MKD (Jin, Wang, and Lin 2023) | 77.2 | 60.1 | 75.5 | 59.0 | 74.1 | 60.0 | 75.5 | 60.1
(ours) DistilVPR-S | 77.9 | 62.0 | 76.4 | 61.3 | 76.8 | 62.2 | 77.1 | 62.6
(ours) DistilVPR-C | 77.0 | 64.2 | 78.0 | 66.4 | 77.3 | 64.2 | 78.4 | 65.9
(ours) DistilVPR-SC | 77.1 | 65.4 | 81.1 | 68.2 | 76.8 | 64.7 | 79.0 | 66.5
Table 3: Big-to-small and 3D-to-2D distillation comparison on the Boreas
dataset.

###  4.3 Ablation Study

#### Agent Relationships.

We compare the performance of different relationships as in Table 4. The
combination of using both self-agent and cross-agent relationships achieves
optimal performance, which verifies the effectiveness of our multi-agent
relationships.

Method | AR@1% | AR@1
---|---|---
w/o distil. | 75.2 | 59.3
$\mathcal{L}_{\rm{tt-ss}}$ in 3 | 76.4 | 61.3
$\mathcal{L}_{\rm{ts-ss}}$ in 4 | 78.0 | 66.4
$\mathcal{L}_{\rm{tt-ts}}$ in 5 | 76.1 | 60.6
$\mathcal{L}_{\rm{tt-ss}}$ \+ $\mathcal{L}_{\rm{ts-ss}}$ | 81.1 | 68.2
Table 4: Ablation study on the self-agent and cross-agent relationship
computation.

#### Manifold Relationships.

We proceed to examine the utilization of different relationship distances, as
detailed in Table 5. Notably, the three fundamental distances yield comparable
individual performances. Further using only two manifold distances with
insufficient curvature exploration could not always bring improvements
compared with using a single manifold. By contrast, through the fusion of
sufficient relationship distances across multiple manifolds with consideration
of all types of curvatures, a remarkable enhancement in distillation
performance is observed. This substantiates the effectiveness of our approach
in exploiting feature relationships within diverse curvature manifolds.

$d_{\rm{euc}}$ | $d_{\rm{cos}}$ | $d_{\rm{hyp}}$ | Ours-S | Ours-C | Ours-SC
---|---|---|---|---|---
✓ |  |  | 59.9 | 65.2 | 66.8
| ✓ |  | 60.2 | 64.9 | 66.5
|  | ✓ | 60.0 | 65.6 | 67.0
✓ | ✓ |  | 60.5 | 65.8 | 67.4
✓ |  | ✓ | 60.2 | 66.0 | 66.8
| ✓ | ✓ | 60.1 | 66.1 | 66.9
✓ | ✓ | ✓ | 61.3 | 66.4 | 68.2
Table 5: AR@1 comparison on different distance functions and relationship
agent combinations.

#### Different Teacher Modalities.

In Table 6, we present a comparison of the distillation performance achieved
with different teachers. Intriguingly, it is observed that the 3D-based model
MinkLoc++3D can even outperform the fusion model MinkLoc++ in terms of
distillation efficiency. This finding underscores the notion that a good task
performer might not necessarily translate into a good teacher for
distillation.

Teacher | T: AR@1 | S: AR@1 |
---|---|---|---
MinkLoc++ | 93.1 | 67.2 |
MinkLoc++3D | 91.3 | 68.2 |
MinkLoc++2D-big | 66.4 | 65.4 |
Table 6: Distillation from different teachers. The student is MinkLoc++2D with
DistilVPR-SC.

#### Visualization.

A more detailed example is illustrated in Fig. 3 with visualized salience
maps. Distillation facilitates the student in emphasizing scene-specific
objects such as buildings, which showcases the effectiveness of teacher
knowledge.

Figure 3: Visualization of the salience maps. With distillation from the
teacher, the student is guided to focus on scene-specific objects such as
buildings.

##  5 Conclusion and Limitations

This paper presents DistilVPR, a novel cross-modal distillation pipeline
designed for enhancing visual place recognition. We harness multi-agent and
multi-manifold relationships to facilitate knowledge exploration, leading to
superior performance compared to other distillation baselines.

A limitation of our approach lies in its assumption of identical feature
dimensions between teachers and students, potentially restricting its
applicability. Nevertheless, this limitation could be addressed by employing a
feature adaptor to align the feature dimensions of teachers and students.

##  6 Acknowledgement

This research is supported by the Singapore Ministry of Education Academic
Research Fund Tier 2 grant MOE-T2EP20220-0002, and the National Research
Foundation, Singapore and Infocomm Media Development Authority under its
Future Communications Research and Development Programme. The computational
work for this article was partially performed on resources of the National
Supercomputing Centre, Singapore (https://www.nscc.sg).

## References

  * Arandjelovic et al. (2016) Arandjelovic, R.; Gronat, P.; Torii, A.; Pajdla, T.; and Sivic, J. 2016.  NetVLAD: CNN architecture for weakly supervised place recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5297–5307.
  * Burnett et al. (2022) Burnett, K.; Yoon, D. J.; Wu, Y.; Li, A. Z.; Zhang, H.; Lu, S.; Qian, J.; Tseng, W.-K.; Lambert, A.; Leung, K. Y.; et al. 2022.  Boreas: A multi-season autonomous driving dataset.  _arXiv preprint arXiv:2203.10168_.
  * Chen et al. (2023) Chen, C.; Liu, X.; Li, Y.; Ding, L.; and Feng, C. 2023.  DeepMapping2: Self-Supervised Large-Scale LiDAR Map Optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9306–9316.
  * Engel, Koltun, and Cremers (2017) Engel, J.; Koltun, V.; and Cremers, D. 2017.  Direct sparse odometry.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(3): 611–625.
  * Gálvez-López and Tardos (2012) Gálvez-López, D.; and Tardos, J. D. 2012.  Bags of binary words for fast place recognition in image sequences.  _IEEE Transactions on Robotics_ , 28(5): 1188–1197.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hou et al. (2022) Hou, Y.; Zhu, X.; Ma, Y.; Loy, C. C.; and Li, Y. 2022.  Point-to-voxel knowledge distillation for lidar semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8479–8488.
  * Hui et al. (2022) Hui, L.; Cheng, M.; Xie, J.; Yang, J.; and Cheng, M.-M. 2022.  Efficient 3D point cloud feature learning for large-scale place recognition.  _IEEE Transactions on Image Processing_ , 31: 1258–1270.
  * Jégou et al. (2011) Jégou, H.; Perronnin, F.; Douze, M.; Sánchez, J.; Pérez, P.; and Schmid, C. 2011.  Aggregating local image descriptors into compact codes.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(9): 1704–1716.
  * Ji, Heo, and Park (2021) Ji, M.; Heo, B.; and Park, S. 2021.  Show, attend and distill: Knowledge distillation via attention-based feature matching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 7945–7952.
  * Jin, Wang, and Lin (2023) Jin, Y.; Wang, J.; and Lin, D. 2023.  Multi-Level Logit Distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 24276–24285.
  * Komorowski (2021) Komorowski, J. 2021.  MinkLoc3d: Point cloud based large-scale place recognition.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 1790–1799.
  * Komorowski, Wysoczańska, and Trzcinski (2021) Komorowski, J.; Wysoczańska, M.; and Trzcinski, T. 2021.  MinkLoc++: Lidar and monocular image fusion for place recognition.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , 1–8. IEEE.
  * Lai, Yin, and Scherer (2022) Lai, H.; Yin, P.; and Scherer, S. 2022.  AdaFusion: Visual-lidar fusion with adaptive weights for place recognition.  _IEEE Robotics and Automation Letters_ , 7(4): 12038–12045.
  * Liu, Nickel, and Kiela (2019) Liu, Q.; Nickel, M.; and Kiela, D. 2019.  Hyperbolic graph neural networks.  _Advances in neural information processing systems_ , 32.
  * Lu et al. (2020) Lu, Y.; Yang, F.; Chen, F.; and Xie, D. 2020.  PIC-Net: Point cloud and image collaboration network for large-scale place recognition.  _arXiv preprint arXiv:2008.00658_.
  * Maddern et al. (2017) Maddern, W.; Pascoe, G.; Linegar, C.; and Newman, P. 2017.  1 year, 1000 km: The Oxford RobotCar dataset.  _The International Journal of Robotics Research_ , 36(1): 3–15.
  * Oertel, Cieslewski, and Scaramuzza (2020) Oertel, A.; Cieslewski, T.; and Scaramuzza, D. 2020.  Augmenting visual place recognition with structural cues.  _IEEE Robotics and Automation Letters_ , 5(4): 5534–5541.
  * Pan et al. (2021) Pan, Y.; Xu, X.; Li, W.; Cui, Y.; Wang, Y.; and Xiong, R. 2021.  CORAL: Colored structural representation for bi-modal place recognition.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2084–2091. IEEE.
  * Park et al. (2019) Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019.  Relational knowledge distillation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 3967–3976.
  * Peng et al. (2022) Peng, G.; Huang, Y.; Li, H.; Wu, Z.; and Wang, D. 2022.  LSDNet: A Lightweight Self-Attentional Distillation Network for Visual Place Recognition.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 6608–6613. IEEE.
  * Qi et al. (2017) Qi, C. R.; Su, H.; Mo, K.; and Guibas, L. J. 2017.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 652–660.
  * Sarlin et al. (2019) Sarlin, P.-E.; Cadena, C.; Siegwart, R.; and Dymczyk, M. 2019.  From coarse to fine: Robust hierarchical localization at large scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12716–12725.
  * Sarlin et al. (2022) Sarlin, P.-E.; Dusmanu, M.; Schönberger, J. L.; Speciale, P.; Gruber, L.; Larsson, V.; Miksik, O.; and Pollefeys, M. 2022.  LaMAR: Benchmarking localization and mapping for augmented reality.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII_ , 686–704. Springer.
  * She et al. (2023) She, R.; Kang, Q.; Wang, S.; Yáng, Y.-R.; Zhao, K.; Song, Y.; and Tay, W. P. 2023.  Robustmat: Neural diffusion for street landmark patch matching under challenging environments.  _IEEE Transactions on Image Processing_.
  * Shi et al. (2023) Shi, J.; Chen, G.; Zhao, Y.; and Tao, R. 2023.  Synchrosqueezed Fractional Wavelet Transform: A New High-Resolution Time-Frequency Representation.  _IEEE Transactions on Signal Processing_ , 71: 264–278.
  * Tifrea, Bécigneul, and Ganea (2018) Tifrea, A.; Bécigneul, G.; and Ganea, O.-E. 2018.  Poincar$\backslash$’e glove: Hyperbolic word embeddings.  _arXiv preprint arXiv:1810.06546_.
  * Uy and Lee (2018) Uy, M. A.; and Lee, G. H. 2018.  PointNetVLAD: Deep point cloud based retrieval for large-scale place recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4470–4479.
  * Wang et al. (2023a) Wang, S.; Kang, Q.; She, R.; Wang, W.; Zhao, K.; Song, Y.; and Tay, W. P. 2023a.  HypLiLoc: Towards effective liDAR pose regression with hyperbolic fusion.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_.
  * Wang et al. (2023b) Wang, S.; Li, W.; Liu, W.; Liu, X.; and Zhu, J. 2023b.  LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5186–5195.
  * Wu et al. (2022) Wu, H.; Wang, M.; Zhou, W.; Li, H.; and Tian, Q. 2022.  Contextual similarity distillation for asymmetric image retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9489–9498.
  * Yan et al. (2022) Yan, X.; Gao, J.; Zheng, C.; Zheng, C.; Zhang, R.; Cui, S.; and Li, Z. 2022.  2dpass: 2d priors assisted semantic segmentation on lidar point clouds.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII_ , 677–695. Springer Nature Switzerland Cham.
  * Zhao et al. (2023) Zhao, K.; Kang, Q.; Song, Y.; She, R.; Wang, S.; and Tay, W. P. 2023.  Adversarial robustness in graph neural networks: A Hamiltonian approach.  _arXiv preprint arXiv:2310.06396_.
  * Zhou et al. (2023) Zhou, S.; Liu, W.; Hu, C.; Zhou, S.; and Ma, C. 2023.  UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird’s-Eye View.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5116–5125.

Supplement
---

##  7 Implementation Details

We choose two SOTA multi-modal place recognition models as teachers, including
MinkLoc++ (Komorowski, Wysoczańska, and Trzcinski 2021) and AdaFusion (Lai,
Yin, and Scherer 2022). We use their single-modal branches as students to test
the effectiveness of cross-modal KD. The scene descriptor dim of the MinkLoc++
series is $384$, while the scene descriptor dim of the AdaFusion series is
$256$. We use the Adam optimizer to train both teachers and students. The
learning rate is set as $1e-4$ and $1e-3$ for the image branch and the point
cloud branch respectively. The learning rate decays by $\times 0.1$ at epoch
$40$. Both teacher models and student models are trained for $60$ epochs with
$128$ batch size. All experiments are conducted on an A100 GPU using PyTorch.
We follow previous works to use the same evaluation protocol, including
Average Recall@1 (AR@1) and Average Recall@1% (AR@1%).

##  8 More Results

###  8.1 Cross Model Distillation

Although we assume the same output channel sizes for both teacher and student
models in the main paper, we have also conducted evaluations in the cross-
model scenario, where teacher and student models possess different channel
sizes. To mitigate the resultant feature inconsistency, we integrate a fully-
connected (fc) layer as an adaptor to harmonize feature channels.

More specifically, we examine two distinct scenarios: one with the adaptor
integrated into the teacher model (as shown in Table 7) and the other with the
adaptor in the student model (as shown in Table 8). Remarkably, our proposed
DistilVPR-SC achieves SOTA performance in 7 out of 8 metrics in the two
tables, thereby further validating the efficacy and broad applicability of our
approach. Furthermore, when comparing the two methods of implementing the
adaptor, it is generally observed that integrating the adaptor within the
teacher models yields better results compared to the alternative of placing it
within the student models.

Distillation Method | T: MinkLoc++3D | T: AdaFusion-3D
---|---|---
S: AdaFusion-2D | S: MinkLoc++2D
AR@1% | AR@1 | AR@1% | AR@1
Teacher | 98.5 | 91.0 | 98.9 | 91.5
Student w/o distil. | 74.5 | 59.6 | 75.2 | 60.0
*KD (Hinton, Vinyals, and Dean 2015) | 76.4 | 58.9 | 74.7 | 59.9
AFD (Ji, Heo, and Park 2021) | 73.8 | 56.9 | 74.2 | 56.9
EPC-Net (Hui et al. 2022) | 75.6 | 56.9 | 75.9 | 58.3
RKD (Park et al. 2019) | 76.3 | 60.1 | 79.0 | 62.8
CSD (Wu et al. 2022) | 74.7 | 59.9 | 76.2 | 61.2
LSD-Net (Peng et al. 2022) | 74.5 | 58.2 | 74.7 | 59.4
MKD (Jin, Wang, and Lin 2023) | 76.9 | 58.8 | 76.7 | 60.8
(ours) DistilVPR-S | 76.5 | 60.3 | 78.1 | 63.2
(ours) DistilVPR-C | 75.8 | 61.6 | 77.5 | 62.1
(ours) DistilVPR-SC | 77.6 | 63.1 | 80.7 | 66.3
Table 7: Cross model 3D-to-2D distillation comparison on the Boreas dataset. The adaptor is equipped on the teacher model. Distillation Method | T: MinkLoc++3D | T: AdaFusion-3D
---|---|---
S: AdaFusion-2D | S: MinkLoc++2D
AR@1% | AR@1 | AR@1% | AR@1
Teacher | 98.5 | 91.0 | 98.9 | 91.5
Student w/o distil. | 74.5 | 59.6 | 75.2 | 60.0
*KD (Hinton, Vinyals, and Dean 2015) | 74.2 | 58.6 | 77.1 | 61.1
AFD (Ji, Heo, and Park 2021) | 76.1 | 59.6 | 75.7 | 58.8
EPC-Net (Hui et al. 2022) | 75.4 | 60.5 | 75.4 | 60.6
RKD (Park et al. 2019) | 75.9 | 62.0 | 75.1 | 60.4
CSD (Wu et al. 2022) | 74.1 | 60.2 | 76.2 | 60.9
LSD-Net (Peng et al. 2022) | 71.6 | 58.5 | 74.9 | 59.2
MKD (Jin, Wang, and Lin 2023) | 73.4 | 58.7 | 76.0 | 61.2
(ours) DistilVPR-S | 76.6 | 61.6 | 74.7 | 61.7
(ours) DistilVPR-C | 76.6 | 61.7 | 75.8 | 60.6
(ours) DistilVPR-SC | 77.4 | 62.4 | 76.1 | 63.4
Table 8: Cross model 3D-to-2D distillation comparison on the Boreas dataset.
The adaptor is equipped on the student model.

###  8.2 Visualization

We present visualizations of several place recognition examples in Fig. 4.
These examples showcase the effectiveness of the distillation approaches in
enhancing performance. Notably, in comparison to the results obtained using
RKD (Park et al. 2019), our proposed DistilVPR-SC yields better outcomes,
particularly evident in the accurate recognition of challenging rain scenes.

Figure 4: Visualizations of place recognition examples.

##  9 Compared Baselines

The baselines we used in our experiments can be classified into direct KD
solutions and relational KD solutions.

  * •

Direct KD baselines include: vanilla KD (Hinton, Vinyals, and Dean 2015), AFD
(Ji, Heo, and Park 2021), EPC-Net (Hui et al. 2022).

  * •

Relational KD baselines include: RKD (Park et al. 2019), CSD (Wu et al. 2022),
LSD-Net (Peng et al. 2022), MKD (Jin, Wang, and Lin 2023).

##  10 Open Sources

The two used datasets are both open-source datasets and can be available
online.

  * •

The Oxford RobotCar dataset.
https://robotcar-dataset.robots.ox.ac.uk/

  * •

The Boreas dataset.
https://www.boreas.utias.utoronto.ca/#/

Our code is mainly built based on the following repositories.

  * •

MinkLoc++
https://github.com/jac99/MinkLocMultimodal

  * •

AdaFusion
https://github.com/MetaSLAM/AdaFusion


[1,2]Kimberley Lennon

1]Materials and Engineering Research Institute, Sheffield Hallam University,
Howard Street, Sheffield, S1 1WB, UK

2]Applied Radiation Technology Group, UK Atomic Energy Authority, Culham
Campus, Abingdon, OX14 3DB, UK

# Machine Learning Based Compton Suppression for Nuclear Fusion Plasma
Diagnostics

<EMAIL_ADDRESS>Chantal Shand<EMAIL_ADDRESS>Robin Smith
<EMAIL_ADDRESS>[  [

###### Abstract

Diagnostics are critical on the path to commercial fusion reactors, since
measurements and characterisation of the plasma is important for sustaining
fusion reactions. Gamma spectroscopy is commonly used to provide information
about the neutron energy spectrum from activation analysis, which can be used
to calculate the neutron flux and fusion power. The detection limits for
measuring nuclear dosimetry reactions used in such diagnostics are
fundamentally related to Compton scattering events making up a background
continuum in measured spectra. This background lies in the same energy region
as peaks from low-energy gamma rays, leading to detection and characterisation
limitations. This paper presents a digital machine learning Compton
suppression algorithm (MLCSA), that uses state-of-the-art machine learning
techniques to perform pulse shape discrimination (PSD) for high purity
germanium (HPGe) detectors. The MLCSA identifies key features of individual
pulses to differentiate between those that are generated from photopeaks and
Compton scatter events. Compton events are then rejected, reducing the low
energy background. This novel suppression algorithm improves gamma
spectroscopy results by lowering minimum detectable activity (MDA) limits and
thus reducing the measurement time required to reach the desired detection
limit. In this paper, the performance of the MLCSA is demonstrated using an
HPGe detector, with a gamma spectrum containing americium-241 (Am-241) and
cobalt-60 (Co-60). The MDA of Am-241 improved by 51% and the signal to noise
ratio (SNR) improved by 49%, while the Co-60 peaks were partially preserved
(reduced by 78%). The MLCSA requires no modelling of the specific detector and
so has the potential to be detector agnostic, meaning the technique could be
applied to a variety of detector types and applications.

###### keywords:

Machine Learning, Gamma Spectroscopy, Fusion, Diagnostics

##  1 Introduction

Gamma spectroscopy is a common method used in the nuclear industry to identify
and quantify the presence of radiation. High purity germanium (HPGe) detectors
are often selected to measure low intensity or complex gamma-ray signatures
due to their excellent $\sim$ keV resolution and will be the focus of this
paper. In the nuclear fusion field, gamma spectroscopy is used in waste
characterisation, materials research for future fusion machines, neutron flux
quantification via activation foils [1], and many other areas. The nuclear
structure of radionuclides common to fusion research mean it is often the case
that low activity nuclides emitting low energy $\gamma$ rays need to be
identified in the presence of higher energy $\gamma$ emitters. For example,
the neutron activation of composite materials, such as stainless steel,
readily produces complex activation networks, resulting in the emission of a
vast spectrum of $\gamma$ energies [2]. This poses a problem, as when higher
energy photons interact within the detector they can Compton scatter [3, 4]
out of the crystal and only deposit a fraction of their energy. This
contributes to a continuum of detected energies, which extends to the lower
energy part of the spectrum (figure 1). For example, in irradiated steel, the
presence of Co-57 is indicated by a low-energy 122 keV gamma ray, which will
be masked by the Compton scattering contribution of high energy Co-60 $\gamma$
rays of 1173 and 1332 keV. This elevated background increases the low energy
minimum detectable activities (MDA), which negatively impacts the
characterisation work.

Figure 1: a) Gamma-ray interactions in a HPGe crystal: one is fully
photoelectrically absorbed and produces a full energy photopeak (blue/dark)
and the other Compton scatters and contributes to the Compton continuum
(green/light). b) Example pulses from each interaction type. c) A typical
energy spectrum showing Compton and photoelectric events

There are some existing solutions to reduce the Compton scatter influence in
gamma-ray spectra. One example is a Compton veto ring, where a ring of gamma
spectroscopy detectors, typically sodium iodide (NaI), surround a HPGe
crystal. An example of such a system is shown in figure 2, located at the
radiological assay and detection lab (RADLab), at the United Kingdom atomic
energy authority (UKAEA) in Oxfordshire. If a photon is detected in the HPGe
crystal and NaI crystal within a set time window, the signal is rejected from
the spectrum as a Compton scatter event. This method is effective at reducing
the Compton continuum, but it often falsely reduces the photopeaks of interest
[5]. A physical Compton suppression system is also bulky, expensive, and is
not easily re-configurable for multiple detector types.

In this work, the physical Compton suppression system used at UKAEA has been
demonstrated to reduce the Compton background on an Am-241 (59 keV photopeak)
and Co-60 spectrum by $\sim$ 88%, however it reduced the Co-60 photopeaks by
$\sim$ 77% and reduced the Am-241 photopeak by 18% (based on equation 5 in
section 3.2.1, peak height reduction as a percentage for energy regions of
interest, discussed in sections 3.2.1 to 3.2.2).

Figure 2: The Mirion broad energy germanium (BEGe) detector used in this work,
with the physical Compton suppression system surrounding the main HPGe crystal

An alternative technique is to use digital algorithms that do not rely on
additional hardware like a Compton veto ring. Past attempts [6] have utilised
a pulse rise time cut off to preserve low energy pulses corresponding to
photoelectric absorption, while removing pulses contributing to the Compton
continuum. Low energy photons are more likely to undergo photoelectric
absorption close to the surface of the detector, resulting in a long charge
collection time in the HPGe as the electrons drift further through the crystal
to the collection electrodes. In contrast, high energy photons more readily
pass through the detector, and are more likely to Compton scatter and deposit
their energy further inside the crystal. This results in a shorter rise time
of the generated electrical pulses. Previous solutions have rejected all
pulses below a rise time threshold [6] (therefore removing pulses from high
energy photons) and required extensive modelling of the specific detector to
determine the optimal rise time cutoff. While effective, it relies on
classification based on a single parameter, is not practical for preserving a
broad range of energies, and is difficult to deploy to multiple types of
detectors as the models would require modification.

Another digital technique is the use of machine learning algorithms for pulse
shape discrimination (PSD). Examples of this include gamma particle tracking
[7] and $\alpha$-$\gamma$ PSD [8]. However $\gamma$-$\gamma$ PSD for Compton
suppression on HPGe detectors has not been undertaken. This work presents a
novel digital solution to Compton suppression, the machine learning Compton
suppression algorithm (MLCSA), which performs Compton suppression (via
$\gamma$-$\gamma$ PSD) on pre-amplified pulses to reduce the Compton
continuum, while preserving higher energy photopeaks and not relying on any
detector modelling or additional hardware. The MLCSA comprises a convolutional
neural network (CNN) as a supervised, classification machine learning model.
Incoming pulses are processed and the CNN classifies each pulse as either a
pulse from a photoelectric absorption (photopeak) or from a Compton scatter
event (background). The scattered pulses are rejected and the output is a
spectrum containing only pulses classified as photopeaks.

##  2 Methods

The MLCSA is an algorithm that performs Compton suppression on pulses from a
HPGe detector and is part of a process that comprises four components, as
shown in figure 3: detector (data acquisition), digitiser (signal/pulse
processing), MLCSA (classifying), and Compton-suppressed spectrum (results).
The methodology to these components are described in the following sections.

Figure 3: Digital Compton suppression flow diagram showing the four
components, including the MLCSA

###  2.1 Detector and data collection

The HPGe detector used to gather training data (which are later split into
training and testing data) and live data (live data refers to an unseen,
‘realistic’ data set, one which might be seen in a standard lab measurement of
a mixed $\gamma$ source) was the Mirion broad energy germanium (BEGe) detector
[9] (figure 2, model: BE3825, serial number: b13135). Five calibration sources
were measured using the BEGe detector in this work to obtain training data:
americium-241 (Am-241), cobalt-60 (Co-60), barium-133 (Ba-133), caesium-137
(Cs-137), and manganese-54 (Mn-54). All sources except for the Mn-54 source
were positioned 10 cm from the end cap for measuring to reduce coincidence
summing effects [3]. The Mn-54 source was placed at 0 cm due to its lower
activity. For collection of the training data, each source was placed on the
detector individually. Based on the energies of the detected pulses, these
training data were split into _Compton_ and _photopeak_ categories. For the
live data scenario, both the Am-241 and Co-60 sources were placed near the
detector simultaneously (10 cm from the end cap) to create a realistic multi-
nuclide spectrum.

###  2.2 Digital pulse processing

The pre-amplified pulses, referred to as raw pulses, were collected and
processed by a Red Pitaya digitiser (the STEMlab 125-14 [10]) with the trigger
settings set as, Voltage range: (0, 20) V; Voltage trigger: (-0.35, -0.4) V;
trigger edge: positive; length of pulse: 200 ns.

The measured raw pulses were subsequently modified so that the CNN could be
trained purely on the shape of the pulses. Raw pulses could not be used
directly for training the CNN as the algorithm was found to use features such
as pulse height and noise level to characterise pulses in a way that lead to
overfitting [11]. Therefore, a pulse processing pipeline, shown in figure 4,
was created to process the pulses for the training algorithm. This process
included filtering, labelling, transformations, and regularisation techniques
[11] to reduce overfitting.

Figure 4: Pulse processing pipeline which is described in detail in the main
text. The greyed out steps are not used when applying this process to the test
and live data set

The pipeline starts by importing raw data, and removing pulses that exceed a
maximum voltage and those below a pre-determined noise level. Then the data
for each individual source are split into photopeak and scatter categories and
labelled accordingly (based on their pulse heights). Pulses are then
normalised to remove pulse height information and are randomly translated
along the time axis. Gaussian noise is then added so that every pulse has the
same noise level, irrespective of its energy. Finally the number of events in
the photopeak and scatter categories are standardised. Example pulses for
Am-241 and Co-60 photopeaks, and Co-60 Compton scatter, are shown in figure 5.

Figure 5: Examples of the final training pulses, with a Co-60 photopeak pulse
in blue, a Co-60 Compton scatter pulse in green, and an Am-241 pulse in red

Once the complete training data set (consisting of five nuclides) had been
collected and processed through the pipeline, it was split into two sub sets,
as is common in machine learning training: a training set (80%) and a testing
set (20%).

The processing pipeline for live data was similar to the training pipeline,
except for the removal of the greyed out steps in figure 4. The split & label
step was removed because that information cannot be known for a realistic
mixed $\gamma$ source. The time translation step was removed because it is
unnecessary due to the way the CNN was trained. The trim data step was also
not possible because the data originate from a mixed source hence the number
of each type of event present is unknown.

###  2.3 MLCSA

####  2.3.1 Convolutional Neural Networks

The MLCSA is a supervised classification machine learning model, and
specifically uses a convolutional neural network (CNN). The CNNs are a class
of artificial neural networks and are tailored for handling structured grid-
like data. They are typically used in fields such as object recognition, image
classification, and image segmentation. They comprise multiple layers,
including convolutional and pooling layers, and excel at feature extraction
and pattern recognition within images [12]. The convolutional layers use
learnable filters to scan and identify local features, while pooling layers
reduce spatial dimensions, enhancing computational efficiency and maintaining
feature robustness [11].

The CNN architecture used in this work is a basic one-dimensional (1D) CNN
[13], specifically a sequential model with convolutional, max pooling,
flatten, and dense layers (convolutional and pooling layers for feature
extraction and dense layers for classification). This architecture is suitable
for processing 1D sequences, such as time series data or any other 1D data
[11]. The architecture was implemented through standard _sklearn_ and _keras_
Python libraries.

####  2.3.2 Training and Testing process

The parameters (number of hidden layers, batches, and epochs) for the CNN in
the MLCSA were selected and tuned using a process of random search hyper-
parameter optimisation to find the best values for each [11]. Each set of
parameters were evaluated by checking the effect on the model performance,
using common methods such as a confusion matrix, accuracy, precision, recall,
and cross-validation (CV) score [11, 14]. These evaluation methods all compare
the known labels to the model predicted labels and all except the confusion
matrix are recorded as a percentage. This process was repeated until there
were no further improvements to the evaluation results (results didn’t
increase in value), and at that point the CNN was concluded to have reached
peak performance. The final parameters that provided the best results (shown
in section 3.1), and therefore the ones used in the final CNN model were: 4
layers (3 hidden), 128 batches, and 30 epochs. The confusion matrix for the
final model is shown in figure 6, where a perfect model would have none-zero
values only on its main diagonal (top left to bottom right) [11].

Figure 6: Confusion matrix for the pulses, corresponding to the optimal model
parameters during training, where S is scatter and P is photopeak. The rows
represent the true label and the columns represent the predicted labels

Once the CNN parameters were tuned, the model was trained using the 80% split
part of the full training data set, which had been processed as described in
the steps in figure 4. The final labelled training pulses were passed to the
CNN model, which mapped and learnt features in the pulses based on their
label. Then the performance of the CNN was evaluated (accuracy, precision,
recall, and CV score), where the trained model was passed only the pulses and
not the labels, so that the predictions could be made by the CNN and then
compared to the actual classifications. Then, the model was passed the pulses
without labels from the 20% test set, and its predictions were evaluated in
the same way. The model had never seen the pulses of the test set, and so if
the performance evaluations in classifying the two sets (training and testing)
are similar, then the model can be seen as generalisable to new data and not
likely to overfit.

####  2.3.3 Live data process

Once the CNN model was confirmed to be performing well, it was incorporated
into the digital Compton suppression flow process (figure 3) as the MLCSA
step. The processed pulses for a new, live data set, containing pulses from
Am-241 and Co-60, were passed to the MLCSA, where the output was pulses with
predicted labels of photopeaks (1) or scatter (0).

###  2.4 Spectrum

The final step in the digital Compton suppression procedure, shown in figure
3, is the production of a spectrum using the MLCSA’s predictions, where pulses
predicted and labelled as photopeaks (1) were extracted and plotted as a final
‘peak only’ spectrum and the scatter pulses (labelled 0) were discarded.

##  3 Results

###  3.1 MLCSA performance on the test data set

The CNN model, which was trained on the 80% data set, was evaluated with the
20% subset of data. The evaluation results for the test data are shown in
table 1, which shows that the CNN performed strongly, with all evaluation
metrics greater than 70%. The high CV score indicated that the model is
generalisable and not likely to over-fit on new data.

Table 1: Evaluation results of the trained CNN (trained with data from 5 nuclides) on the 20% test data set, all values given as a percentage. Accuracy | Precision | Recall | CV score
---|---|---|---
78.7 | 72.5 | 92.4 | 84.5
|  |  |

###  3.2 MLCSA performance on the live data set

Once the CNN model was trained and evaluated on the small 20% data set, it was
used in the MLCSA with the live data set, comprising an Am-241 and Co-60
source simultaneously positioned 10 cm away from the detector. The data were
collected and processed through the digital Compton suppression flow from
figure 3, which included the second pulse processing pipeline without the
greyed out processes from figure 4. The result of the MLCSA is shown in figure
7, which shows the before and after spectra.

Figure 7: Mixed Am-241 and Co-60 spectrum before (line) and after (filled) the
MLCSA. Top shows the full spectrum and bottom is zoomed along the vertical
axis

####  3.2.1 Spectrum evaluation metrics

Three metrics were used to quantify and evaluate the Compton suppression
performance of the MLCSA on the final spectrum.

The first was MDA improvement of the Am-241 photopeak. The MDA is a
calculation of the lowest measurable activity for a specific set up, and was
calculated using the Currie method [15, 16] as

$\textrm{MDA (Bq)}=\left(\frac{2.71+4.65\sqrt{B}}{br\times\varepsilon\times
t}\right),$ (1)

where $B$ is the background sum including a region of 10 keV either side of
the photopeak, $br$ is the branching ratio, $\varepsilon$ is efficiency, and
$t$ is count time.

The second metric was signal to noise ratio (SNR). The signal is defined as
the net area of the photopeak above the background level and the noise is
defined as the area below the peak [17]:

$\textrm{SNR}=\left(\frac{an}{bg}\right)$ (2)

where

$an=\left(af-bg\right)$ (3)

and

$bg=\left(\frac{C}{2}\right)\times\left(B1+B2\right)$ (4)

where $an$ is net peak area, $bg$ is the background area underneath the
photopeak, $af$ is the full area, C is the number of channels in the
photopeak, $B1$ and $B2$ are the numbers of counts at the start and end of the
photopeak, respectively [17].

The final metric was the percentage difference between before and after the
MLCSA was applied, including: decrease in MDA of the Am-241 photopeak,
reduction in photopeak counts, reduction in max Compton counts on the spectrum
for a range of energies, and SNR improvement of Am-241. These were calculated
as

$\%\textrm{Diff}=\left(\frac{a-b}{b}\right)\times 100,$ (5)

where $b$ and $a$ refer to the relevant value before and after the MLCSA,
respectively.

####  3.2.2 Spectrum evaluation performance

The MLCSA was able to significantly reduce the Compton continuum throughout
the whole spectrum, with the largest reductions at the lower energy region
(the 200 keV region was reduced by $\sim$ 90%), while almost fully preserving
the Am-241 photopeak (the 59 keV photopeak was only reduced by 15%). The Co-60
photopeaks were preserved, but were significantly reduced in counts ($\sim$
78%). However, it should be noted that the lower energy (1172 keV) Co-60
photopeak was not provided in the training data set, and so the presence of
the photopeak after the MLCSA suggests the CNN model is able to generalise to
unseen photopeak pulses.

The percentage reduction in photopeaks in the before and after spectrum was
calculated from the data shown in figure 7. The reductions for the 59 keV
Am-241 photopeak, 1173 keV / 1332 keV Co-60 photopeaks, and the overall counts
reduction in two scatter areas, defined as a low scatter region at 200 keV and
a high scatter region at 400 keV, are shown in table 2. The percentage
difference was significant on the scatter regions after the MLCSA, while
remaining desirably low for the low energy Am-241 photopeak. The Co-60 peaks
were reduced significantly, however this is similar to the physical Compton
veto system as discussed in Section 1 [5] and is an improvement on the output
of other digital methods [6] which completely remove the higher energy
photopeaks.

Table 2: Evaluation of the photopeak counts reduction, and Compton counts reduction (at 200 and 400 keV), as a percentage difference between before and after the application of the MLCSA. Energy, keV | Peak reduction, %
---|---
59 | 15
1173 | 79
1332 | 77
200 | 99
400 | 95
|

The SNR was calculated on the Am-241 photopeak using Equations 2 \- 4 for the
before and after spectrum, producing values of 1.23 and 1.83 respectively.
This produced a significant improvement in the SNR of the Am-24 photopeak,
with a percent increase of 49% (calculated using equation 5), which indicates
a reduction in the Compton background at the 59 keV photopeak.

The MDA of the Am-241 59 keV photopeak was calculated using Equation 1 for the
before and after MLCSA spectra, and the improvement was calculated using
Equation 5. The MDA before was 156 Bq and after was 76 Bq, this produced a
reduction of 51%. Compared to other digital methods that achieved an MDA
reduction of 37% [6], this is a significant improvement.

##  4 Discussion

The MLCSA reduced the Compton continuum and improved the SNR ratio of the low
energy Am-241 photopeak in the presence of higher energy $\gamma$ emitters.
The resulting spectrum from the MLCSA on the Am-241 and Co-60 spectrum show
that the Compton continuum from Co-60 scatter can be reduced by $\sim$ 90%
using machine learning (specifically a basic 1D sequential CNN), while
retaining the Am-241 and Co-60 peaks. The SNR ratio of the Am-241 peak was
improved by 49%, and the MDA improved by 51% - this will lead to better
detection and more accurate activity quantification, especially in the case
where the lower energy photopeaks are present in lower quantities. This shows
the potential for improving Compton suppression systems as this method is
similar in performance to the physical Compton veto ring, but with a digital
advantage that it could be readily adapted to other detectors.

This work did not involve complex modelling of a detector, but instead relied
on measuring a set of reference sources. Therefore, this could be applied to
other detectors as the CNN model would only need re-training with pulses from
that detector. This means the MLCSA is potentially detector agnostic, which
could be demonstrated with further work and development. Other future
developments of this algorithm could include: making the MLCSA run in live
time, including more nuclides to the training and testing sets to make it more
widely applicable, and improving the machine learning model performance.

With the significant improvement to low energy nuclide detection in the form
of Compton suppression, the MLCSA has the potential to improve fusion
diagnostics by improving the detection of key low energy nuclides. As an
example, in foil activation experiments, information about the plasma can be
calculated from key lines in the gamma spectrum following irradiation of
activation foils in a fusion environment [1, 5, 18, 19, 20, 21, 22]. The lines
of interest are often low energy, can be low abundance, and are often obscured
by the Compton scattering of higher energy activation products. The MLCSA
could be applied to such spectra, which would reduce the Compton continuum and
improve the analysis and therefore could reduce the errors on the fusion power
calculations.

The MLCSA developed in this work has the potential to have a broader impact on
the nuclear industry. Gamma spectroscopy is a method used in radioactive waste
management and disposal (fission and fusion [23]), where disposal costs are
calculated per unit of activity [24]. Reduced Compton backgrounds from the
MLCSA could make gamma spectroscopy systems more accurate, which could have
financial benefits relating to disposal costs.

## Declarations

### Funding

This work has been part-funded by the EPSRC Energy Programme (grant number
EP/W006839/1) and the UK STFC, (grant number ST/V001086/1). To obtain further
information on the data and models underlying this paper please contact
<EMAIL_ADDRESS>For the purpose of open access, the author has
applied a Creative Commons Attribution (CC BY) licence to any Author Accepted
Manuscript version arising from this submission.

### Conflict of interest

The authors have no competing interests to declare that are relevant to the
content of this article.

### Data Availability

All data generated or analysed in this work are available on request.

## References

  *   * [1] Packer, L. W. _et al._ Technological exploitation of the JET neutron environment: Progress in ITER materials irradiation and nuclear analysis.  _Nuclear Fusion_ 61 (2021).
  * [2] Goodell, J. J. _et al._ Determining the Activation Network of Stainless Steel in Different Neutron Energy Regimes and Decay Scenarios Using Foil Activation Experiments and FISPACT-II Calculations.  _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors, and Associated Equipment_ (2018).
  * [3] Krane, K. S., Wlley, J., York, N., Brisbane, C. & Singapore, T.  _Introductory Nuclear Physics_ (1988).
  * [4] Gilmore, G. R.  _Practical Gamma‐Ray Spectrometry_ 2 edn (Wiley, 2008).
  * [5] Grove, C. L. _et al._ Initial Gamma Spectroscopy of ITER Material Irradiated in the JET D-T Neutron Environment .
  * [6] Tree, K. A.  _Enhanced Nuclear Waste Assay_.  Ph.D. thesis (2019).
  * [7] Fraser Holloway, Harkness-Brennan, L. & Kurlin, V.  _The Development of Novel Pulse Shape Analysis Algorithms for AGATA_.  Ph.D. thesis (2022).
  * [8] Yoshino, M. _et al._ Comparative pulse shape discrimination study for Ca(Br, I) scintillators using machine learning and conventional methods (2021).  URL http://arxiv.org/abs/2110.01992.
  * [9] Mirion Technologies.  Broad Energy Germanium Detectors (2021).  URL https://mirionprodstorage.blob.core.windows.net/prod-20220822/cms4_mirion/files/pdf/spec-sheets/spc-134-en-a-bege.pdf.
  * [10] Red Pitaya.  Red Pitaya STEMlab 125-14.  URL https://redpitaya.com/stemlab-125-14/.
  * [11] Géron, A.  _Hands-on machine learning with Scikit-Learn, Keras & TensorFlow_ 2 edn (OReilly, 2019).
  * [12] Krizhevsky, A., Sutskever, I. & Hinton, G. E.  ImageNet Classification with Deep Convolutional Neural Networks.  Tech. Rep.  URL http://code.google.com/p/cuda-convnet/.
  * [13] Kiranyaz, S. _et al._ 1D convolutional neural networks and applications: A survey.  _Mechanical Systems and Signal Processing_ 151 (2021).
  * [14] Jordan, M., Kleinberg, J. & Schölkopf, B.  _Pattern Recognition and Machine Learning_ (2006).
  * [15] Currie. L. A.  Limits for Qualitative Detection and Quantitative Determination. Application to radiochemistry.  _Analytical Chemistry_ 40.3, 586–593 (1968).
  * [16] Nuclear Regulatory Commission.  Minimum Detectable Activity (MDA) calculations.  URL https://www.nrc.gov/docs/ML0706/ML070600299.pdf.
  * [17] Mirion Technologies.  Nuclear measurement fundamental principle spectrum analysis.  URL https://www.mirion.com/discover/knowledge-hub?resourceType=Articles&resourceType=Education.
  * [18] Coventry, M. D. & Krites, A. M.  Measurement of D-7Li Neutron Production in Neutron Generators Using the Threshold Activation Foil Technique.  _Physics Procedia_ 90, 85–91 (2017).
  * [19] Wong, O., Smith, R., Nobs, C. R. & Bruce, A. M.  Optimising Foil Selection for Neutron Activation Systems.  _Journal of Fusion Energy_ 41 (2022).
  * [20] Chae, S., Lee, J. Y. & Kim, Y. S.  Neutron diagnostics using nickel foil activation analysis in the KSTAR.  _Nuclear Engineering and Technology_ 53, 3012–3017 (2021).
  * [21] Chiesa, D. _et al._ Measurement of the neutron flux at spallation sources using multi-foil activation (2018).  URL http://arxiv.org/abs/1803.00605http://dx.doi.org/10.1016/j.nima.2018.06.016.
  * [22] Stainer, T. _et al._ 14 MeV neutron irradiation experiments - Gamma spectroscopy analysis and validation automation.  _International Conference on Physics of Reactors: Transition to a Scalable Nuclear Future, PHYSOR 2020_ 2020-March, 1786–1795 (2020).
  * [23] Gonzalez De Vicente, S. M. _et al._ Overview on the management of radioactive waste from fusion facilities: ITER, demonstration machines and power plants.  _Nuclear Fusion_ 62 (2022).
  * [24] The environmental permitting regulations 2016, chapter 23 (2016).  URL https://www.legislation.gov.uk/ukdsi/2016/9780111150184/schedule/23/data.htm?wrap=true.