# Batch Group Normalization

Xiao-Yun Zhou1, Jiacheng Sun2, Nanyang Ye3, Xu Lan4, Qijun Luo5, Bo-Lin Lai 3,
Pedro Esperanca2, Guang-Zhong Yang 3, Zhenguo Li2

###### Abstract

Deep Convolutional Neural Networks (DCNNs) are hard and time-consuming to
train. Normalization is one of the effective solutions. Among previous
normalization methods, Batch Normalization (BN) performs well at medium and
large batch sizes and is with good generalizability to multiple vision tasks,
while its performance degrades significantly at small batch sizes. In this
paper, we find that BN saturates at extreme large batch sizes, i.e., 128
images per worker111i.e., GPU, as well and propose that the
degradation/saturation of BN at small/extreme large batch sizes is caused by
noisy/confused statistic calculation. Hence without adding new trainable
parameters, using multiple-layer or multi-iteration information, or
introducing extra computation, Batch Group Normalization (BGN) is proposed to
solve the noisy/confused statistic calculation of BN at small/extreme large
batch sizes with introducing the channel, height and width dimension to
compensate. The group technique in Group Normalization (GN) is used and a
hyper-parameter $\rm G$ is used to control the number of feature instances
used for statistic calulation, hence to offer neither noisy nor confused
statistic for different batch sizes. We empirically demonstrate that BGN
consistently outperforms BN, Instance Normalization (IN), Layer Normalization
(LN), GN, and Positional Normalization (PN), across a wide spectrum of vision
tasks, including image classification, Neural Architecture Search (NAS),
adversarial learning, Few Shot Learning (FSL) and Unsupervised Domain
Adaptation (UDA), indicating its good performance, robust stability to batch
size and wide generalizability. For example, for training ResNet-50 on
ImageNet with a batch size of 2, BN achieves Top1 accuracy of $66.512\%$ while
BGN achieves $76.096\%$ with notable improvement.

## Introduction

Since AlexNet was proposed in (Krizhevsky, Sutskever, and Hinton 2012), Deep
Convolutional Neural Network (DCNN) has been a popular method for vision tasks
including image classification (Deng et al. 2009), object detection (Lin et
al. 2014) and semantic segmentation (Everingham et al. ). DCNNs are usually
composed of convolutional layers, normalization layers, activation layers,
etc. Normalization layers are important in improving performance and speeding
up training.

Batch Normalization (BN) was one of the early proposed normalization methods
(Ioffe and Szegedy 2015) and is widely used. It normalizes the feature map
with the mean and variance calculated along with the batch, height, and width
dimension of a feature map and then re-scales and re-shifts the normalized
feature map to ensure DCNN representation ability. Based on BN, many
normalization methods for other tasks have been proposed. For example, Layer
Normalization (LN) was proposed for calculating the statistics along the
channel, height and width dimension for Recurrent Neural Network (RNN) (Ba,
Kiros, and Hinton 2016). Weight Normalization (WN) was proposed to
parameterize the weight vector for supervised image recognition, generative
modelling, and deep reinforcement learning (Salimans and Kingma 2016).
Divisive Normalization which includes BN and LN as special cases was proposed
for image classification, language modeling and super-resolution (Ren et al.
2016). Instance Normalization (IN) where the statistics were calculated from
the height and width dimension was proposed for fast stylization (Ulyanov,
Vedaldi, and Lempitsky 2016). Instead of calculating the statistics from data,
Normalization Propagation estimated them data-independently from the
distribution in layers (Arpit et al. 2016). Group Normalization divided the
channels into groups and calculated the statistics for each grouped channel,
height and width dimension, showing stability to batch sizes (Wu and He 2018).
Positional Normalization (PN) was proposed to calculate the statistics along
the channel dimension for generative networks (Li et al. 2019a).

Among these normalization methods, BN can usually achieve good performance at
medium and large batch sizes. However, its performance degrades at small batch
sizes, as shown in pre works (Wu and He 2018; Ioffe 2017). Furthermore, as
shown in our experiments, BN’s performance saturates at extreme large batch
sizes, i.e., 128 images per worker. GN enjoys a greater degree of stability at
different batch sizes, while slightly under-performs BN at medium and large
batch sizes. Other normalization methods, including IN, LN and PN perform well
in specific tasks, but are usually less generalizable to and under-perform in
other vision tasks. As reviewed in Related Work, many works have been
conducted on proposing new normalization methods with good performance,
stability and generalizability.

In this paper, unlike those reviewed works where additional trainable
parameters, extra computation or additional information are used, Batch Group
Normalization (BGN) is parameter- and computation-efficient. We know the fact
that mini-batch training usually can perform better than single-batch (use a
single image example as the DCNN input per iteration) and all-batch training
(use all image examples as the DCNN input per iteration), as single-batch
training can indicate noisy gradient while all-batch training may not indicate
representative gradient (each image example indicates gradient with different
directions, thus, adding them all indicates confused gradient). Inspired by
this fact, we think the number of feature instances in statistic calculation
in normalization should also be moderate, i.e., the degraded/saturated
performance of BN on small/extreme large batch sizes is due to noisy/confused
statistic calculation.

Figure 1: top). The difference between the statistic calculation in BN, IN,
LN, GN, PN and BGN. Each subplot shows a feature map tensor, with N, C and (H,
W) as the batch, channel and spatial axes. The pixels in blue are used to
compute the statistics. This figure is inspired by (Wu and He 2018). bottom).
The Top1 accuracy of training ResNet-50 on ImageNet, with different batch
sizes, and with BN, IN, LN, GN, PN and BGN as the normalization.

Hence, BGN is proposed to facilitate the performance degradation/saturation of
BN at small/extreme large batch sizes with the group technique from GN. It
merges the channel, height and width dimensions into a new dimension, divides
the new dimension into feature groups, and calculates the statistics across
the whole mini-batch and feature group. A hyper-parameter $\rm G$ is used to
control the level of feature division and to supply proper statistics for
different batch sizes. The difference between BN, IN, LN, GN, PN and BGN, with
respect to the dimensions along which the statistics are computed, are
illustrated in Fig. 1(top). The Top1 accuracy of training ResNet-50 on
ImageNet by using BN, IN, LN, GN, PN and BGN as the normalization layer are
shown in Fig. 1(bottom). Without adding trainable parameters, using extra
information or requiring extra computation, BGN achieves both good performance
and stability at different batch sizes. We test on Neural Architecture Search
(NAS), adversarial learning, Few Show Learning (FSL) and Unsupervised Domain
Adaptation (UDA), BGN outperforms BN, IN, LN, GN and PN, showing its good
generalizability.

## Related Work

Why normalization works? The effectiveness of BN has been attributed to
internal covariate shift where the distribution of each layer’s input changes
and hence lower learning rate and careful parameter initialization are
essential to guarantee good training in DCNNs without normalization (Ioffe and
Szegedy 2015). Other works have investigated the reasons for the success of
normalization. For example, Santurkar et al. (Santurkar et al. 2018) proposed
that the effectiveness of BN has little to do with internal covariate shift,
but to make the optimization landscape smoother and hence to introduce stable
gradients and faster training. Bjorck et al. (Bjorck et al. 2018) proposed
that allowing larger learning rates is the main reason for BN to achieve
faster convergence and better generalization. A theoretical support was
supplied for the effectiveness of BN in tuning learning rates less (Arora,
Lyu, and Li 2019). Luo et al. demonstrated that BN and regularization share
the same traits (Luo et al. 2019b). A quantitative analysis was provided to
compare the gradient descent with and without BN on Ordinary Least Squares
(OLS) (Cai, Li, and Shen 2019). In (Fan 2017), it was shown with fuzzy neural
networks that BN estimates the correct bias induced by the generalized hamming
distance. In contrast, BN was proved to be the cause of the gradient explosion
for DCNNs without residual learning (Yang et al. 2019). In (Li et al. 2019b),
from the theoretical and statistical aspects, the disharmony between dropout
and BN was explained. It was claimed that BN is not unique for stable
training, higher learning rates, accelerated convergence and improved
generalization, and can be replaced by better initialization (Zhang, Dauphin,
and Ma 2019; De and Smith 2020). Normalization layers were shown to introduce
a stable gradient magnitude when training the Long-Short-Term Memory (LSTM)
(Hou et al. 2019). Even though many interesting theories have been proposed to
explain the effectiveness of BN, there is still a lack of consensus.

Improvements Previous works can be further improved. Centered WN proposed to
add a learnable parameter to adjust the weight norm in WN (Huang et al. 2017).
Recurrent BN was proposed to not only apply BN in the input-to-hidden, but
also to the hidden-to-hidden transformation of RNNs (Cooijmans et al. 2016).
Batch Renormalization was proposed to decrease the dependence of BN on the
size of mini-batches (Ioffe 2017). Riemannian approach was combined to BN on
the space of weight vectors, which improves BN’s performance on various DCNN
architectures and datasets (Cho and Lee 2017). EvalNorm was proposed to
estimate the corrected normalization statistics during evaluation to address
the performance degradation of BN (Singh and Shrivastava 2019). Moving Average
BN proposed using the batch statistics in the backward propagation of
traditional BN (Yan et al. 2019). Based on LN, Adaptive Normalization proposed
to modify bias and gain by using a new transformation function (Xu et al.
2019a). Root Mean Square LN proposed to abandon the re-centering and keep the
re-scaling in LN (Zhang and Sennrich 2019).

Researchers are also looking into non-linear normalizing techniques with extra
computation. For example, ZCA was used to replace the centering and scaling
calculations in BN, resulting in Decorrelated BN (Huang et al. 2018).
Iterative Normalization proposed using Newton iterations to avoid the eigen-
decomposition in Decorrelated BN, indicating much more efficient whitening
(Huang et al. 2019). Instead of normalizing in the spatial space, Spectral
Normalization was proposed to normalize the spectral norm of weights and was
used in Generative Adversarial Networks (GANs) (Miyato et al. 2018) and
adversarial training (Farnia, Zhang, and Tse 2019).

Learning-to-normalize is also explored. Different normalization methods
introduced before can be combined to achieve better performance and
generalization. Batch-Instance Normalization (BIN) was proposed for adaptively
style-invariant neural networks, with using a trainable parameter to combine
the feature maps calculated from BN and IN (Nam and Kim 2018). Switchable
Normalization (SN) was proposed by using trainable parameters on the mean and
variance calculated in BN, IN and LN to learn new statistics (Luo et al.
2019a, c). Sparse SN used SparseMax to make the trainable parameters in SN
sparse (Shao et al. 2019). In Instance-Level Meta Normalization (Jia, Chen,
and Chen 2019), feature feed-forward and gradient back-propagation were used
to learn the normalization parameters.

Others Normalization can be used as a method to achieve a task directly. For
example, a real-time and arbitrary style transfer was achieved by aligning the
mean and variance of the content features to that of the style features (Huang
and Belongie 2017). Domain adaptation was achieved by changing the statistics
from the source domain to the target domain (Li et al. 2018b). New tasks can
be solved by using BN to combine learned tasks (Data et al. 2018). Specific
normalization methods have also been specifically proposed for cross-domain
tasks (Li and Vasconcelos 2019; Wang et al. 2019), global covariance pooling
networks (Li et al. 2018a), multitask learning (Chen et al. 2018; Deecke,
Murray, and Bilen 2019), UDA (Chang et al. 2019), semantic image synthesis
(Park et al. 2019), medical area (Zhou and Yang 2019; Zhou et al. 2019; Zhou
2019), and scene text detection (Xu et al. 2019b). Kalman Normalization (KN)
was proposed to combine internal representations across multiple DCNN layers
(Wang et al. 2018). And, instead of using $L^{2}$ norm, $L^{1}$ and
$L^{\infty}$ norm were proposed for numerical stability in low-precision
calculation (Hoffer et al. 2018).

## Methodology

In a DCNN with $\rm L$ layers, for an input feature map
$\mathcal{F}^{l},l\in[1,\rm L]$, which is usually with four dimensions $(\rm
N,C,H,W)$, where $\rm N,C,H,W$ is the batch, channel, height and width
dimension respectively. For simplification,
$n\in\rm[1,N],c\in\rm[1,C],h\in\rm[1,H],w\in\rm[1,W]$ are the corresponding
batch, channel, height and width indices and will not be repeatly defined in
the following usage. The feature map at $(l+1)$th layer is calculated as:

$\mathcal{F}^{l+1}=\psi(\theta(\phi(\mathcal{F}^{l},w^{l},b^{l}),\gamma^{l},\beta^{l}))$
(1)

where $w^{l}$ and $b^{l}$ are the trainable weight and bias parameters in
convolutional layers, $\gamma^{l}$ and $\beta^{l}$ are the trainable re-scale
and re-shift parameters in normalization layers, $\psi(\cdot)$ is the
activation function. $\theta(\cdot)$ is the normalization function.
$\phi(\cdot)$ is the convolusion function.

A typical normalization layer includes four steps: (1) divide the feature map
into feature groups; (2) calculate the $mean$ and $variance$ statistics for
each feature group; (3) normalize each feature group with the calculated
statistics; (4) re-scale and re-shift the normalized feature map to maintain
the DCNN representation ability. For example, in BN, the feature map is
divided along the channel dimension, the $mean$ $\mu_{c}$ and $variance$
$\delta_{c}^{2}$ are calculated along the batch, height and width dimension
as:

$\mu_{c}=\frac{1}{\rm N\times\rm H\times\rm W}\sum\limits_{n=1}^{\rm
N}\sum\limits_{h=1}^{\rm H}\sum\limits_{w=1}^{\rm W}f_{n,c,h,w}$ (2)

$\delta_{c}^{2}=\frac{1}{\rm N\times\rm H\times\rm W}\sum\limits_{n=1}^{\rm
N}\sum\limits_{h=1}^{\rm H}\sum\limits_{w=1}^{\rm W}(f_{n,c,h,w}-\mu_{c})^{2}$
(3)

Then the feature map is normalized as:

$\displaystyle\hat{f}_{n,c,h,w}=\frac{f_{n,c,h,w}-\mu_{c}}{\sqrt{\delta_{c}^{2}+\epsilon}}$
(4)

$\epsilon$ is a small number added for division stability. In order to
maintain the DCNN representation ability, extra trainable parameters are added
for each feature channel:

$f^{\prime}_{n,c,h,w}=\gamma_{c}\hat{f}_{n,c,h,w}+\beta_{c}$ (5)

Table 1: The Top1 validation accuracy of BGN on ImageNet with ResNet-50. G is set to be from 512 to 1. Batch size | Group number
---|---
512 | 256 | 128 | 64 | 32 | 16 | 8 | 4 | 2 | 1
128 | 77.008 | 75.936 | 76.256 | 76.096 | 75.984 | 75.504 | 76.016 | 75.632 | 75.520 | 75.360
2 | 63.968 | 66.000 | 68.160 | 70.064 | 71.856 | 72.832 | 74.928 | 75.120 | 75.968 | 76.096

By including multiple examples into statistic calculation, BN enjoys good
performance at medium and large batch sizes and good generalizability to
multiple vision tasks, i.e. NAS. However, its performance degrades
dramatically, i.e., $>10\%$ in our ImageNet experiment, at small batch sizes.
In order to improve this shortage, GN includes grouped channel dimension into
statistic calculation:

$\mu_{n,g}=\frac{1}{\rm M\times\rm H\times\rm W}\sum\limits_{m=(g-1)\cdot\rm
M+1}^{\rm g\cdot M}\sum\limits_{h=1}^{\rm H}\sum\limits_{w=1}^{\rm
W}f_{n,m,h,w}$ (6)

$\delta_{n,g}^{2}=\frac{1}{\rm M\times\rm H\times\rm
W}\sum\limits_{m=(g-1)\cdot\rm M+1}^{\rm g\cdot M}\sum\limits_{h=1}^{\rm
H}\sum\limits_{w=1}^{\rm W}(f_{n,m,h,w}-\mu_{n,g})^{2}$ (7)

where $g\in\rm[1,G]$, $\rm G$ is a hyper parameter - group number, $\rm
M=C//G$, $//$ is floor division. GN enjoys good stability to different batch
sizes, however, its performance is slightly lower than BN at medium and large
batch sizes and its generalizability to other vision tasks is weaker than BN.
Except these known phenomena, our experiments show that BN saturates at
extreme large batch sizes.

We think the degradation/saturation of BN at small/extreme large batch sizes
are caused by noisy/confused statistic calculation. Similar indication also
exists in mini-batch training, where single-batch/all-batch training are
usually worse than mini-batch training, as noisy/confused gradients are
calculated. To facilitate this, we propose BGN where the number of feature
instances used for statistic calculation is controlled to be proper by using
the group technique in GN. To be in more details, we first merge the channel,
height and width dimensions into a new dimension and achieve
$\rm\mathcal{F}^{l}_{N\times D}$, where $\rm D=C\times H\times W$. The mean
$\mu_{g}$ and variance $\delta_{g}^{2}$ are calculated along the batch and new
dimension as:

$\mu_{g}=\frac{1}{\rm N\times\rm S}\sum\limits_{n=1}^{\rm
N}\sum\limits_{d=(g-1)\cdot\rm S+1}^{\rm g\cdot S}f_{n,d}$ (8)

$\delta_{g}^{2}=\frac{1}{\rm N\times\rm S}\sum\limits_{n=1}^{\rm
N}\sum\limits_{d=(g-1)\cdot\rm S+1}^{\rm g\cdot S}(f_{n,d}-\mu_{g})^{2}$ (9)

where $\rm G$ is the number of groups that the new dimension is divided and is
a hyper-parameter, $\rm S=M/G$ is the number of instances inside each divided
feature group. When the batch size is small, a small G is used to combine the
whole new dimension into statistic calculation to avoid noisy statistics,
while when the batch size is large, a large G is used to split the new
dimension into small pieces for calculating the statistics to avoid confused
statistics. $\gamma_{c}$ and $\beta_{c}$ are used in the same way as BN. In
BN, $\mu_{c}$ and $\delta_{c}^{2}$ used in the test are the moving average of
that in the training stage. The proposed BGN uses this policy as well.

Relation to General Batch Group Normalization (GBGN) GBGN (Summers and Dinneen
2019) could be confused as a similar work to BGN, we clarify our contributions
as below: 1\. We first propose the saturation of BN at extreme large batch
sizes. 2\. We first propose that the degradation/saturation of BN at
small/extreme large batch size are caused by noisy/confused statistic
calculation. 3\. We propose to use group in the channel, height and width
dimension to compensate (in GBGN, the group is along the batch and channel
dimension). 4\. We offer extensive experiments on image classification, NAS,
adversarial learning, FSL and UDA to validate our thoughts.

## Experiments

Among the reviewed normalization methods, BN, IN, LN, GN, and PN222Re-
injection path was used in the original PN (Li et al. 2019a), in this paper,
for a fair comparison, it is not included. are suitable for being baselines of
BGN, as other normalization methods usually use additional trainable
parameters, non-linear normalization or information from multiple layers,
which are orthogonal to and can be combined to BGN to improve its performance
further. BGN is validated on applications, including image classification,
NAS, adversarial training, FSL and UDA.

### Image Classification on ImageNet with ResNet-50

Image classification is one of the applications used to validate BGN. We focus
on ImageNet (Krizhevsky, Sutskever, and Hinton 2012) which contains $\rm
1.28M$ training images and $50000$ validation images. The model used is
ResNet-50 (He et al. 2016).

Implementation details: 8 GPUs are used in all ImageNet experiments. The
gradients used for backpropagation are averaged across 8 GPUs, while the mean
and variance used in BN and BGN are calculated within each GPU. $\gamma_{c}$
and $\beta_{c}$ are initialized as 1 and 0 respectively, while all other
trainable parameters are initialized as in (He et al. 2016). 120 epochs are
trained with the learning rate decayed by $10\times$ at the 30th, 60th, and
90th epoch. The initial learning rates for the experiments with batch sizes of
128, 64, 32, 16, 8, 4 and 2 are 0.4, 0.2, 0.1, 0.05, 0.025, 0.0125 and 0.00625
respectively, following (Goyal et al. 2017). Stochastic Gradient Descent (SGD)
is used as the optimizer. A weight decay of $10^{-4}$ is applied to all
trainable parameters. For the validation, each image is cropped into
$224\times 224$ patches from the center, and the Top1 accuracy is reported.
Following (Wu and He 2018), the median accuracy in the last five epochs is
reported to reduce the random variance. All experiments are trained under the
same programming implementation, with replacing the normalization layer into
BN, IN, LN, GN, PN, and BGN respectively.

Hyper-parameter - $\rm G$: to explore the hyper-parameter $\rm G$, BGN with a
group number of 512, 256, 128, 64, 32, 16, 8, 4, 2 and 1 respectively are used
as the normalization layer in ResNet-50 for ImageNet classification. The
largest (according to GPU memory) and smallest batch size in our experiments -
128 and 2 are tested. The Top1 accuracy of the validation dataset is shown in
Tab. 1. We can see that a large G - 512 is suitable for a large batch size -
128, while a small G - 1 is suitable for a small batch size - 2. It can
support our claims that a proper number of feature instances is important for
the statistic calculation in normalization. When the batch size is
large/small, a large/small $\rm G$ is used to split/combine the new dimension
to avoid confused/noisy statistic calculation. The accuracy variance for batch
size 128 is smaller than that for batch size of 2, indicating that either
saturation is less serious than degradation in normalization or a large batch
size of 128 has not reached the saturation edge yet.

Comparison with baselines: BN, IN, LN, GN, PN, GBGN and BGN are used as the
normalization layer in ResNet-50, with a batch size of 128, 64, 32, 16, 8, 4
and 2 respectively. The group number in GN is set as 32, which was claimed as
the optimal configuration for GN (Wu and He 2018). The group number for
channel is set as 32 while that for batch is set as the batch size for GBGN
based on our experience. With this setting, GBGN equals to BGN at moderate
batch sizes, hence GBGN is only included in ImageNet experiments and is
ignored in the following few experiments where moderate batch sizes are used.
The group number in BGN is set to be 512, 256, 128, 64, 16, 2 and 1 for batch
sizes of 128, 64, 32, 16, 8, 4 and 2 respectively. We choose $\rm G$ for the
largest and smallest batch size according to Tab. 1 while choose $\rm G$ for
other batch sizes with interpolation. The Top1 accuracy is shown in Tab. 2. We
can see that the proposed BGN out-performs all previous methods, including BN,
IN, LN, GN, PN and GBGN at all different batch sizes. To be specific, BN
approaches BGN’s performance at large batch sizes, however, its performance
degrades quickly at small batch sizes. GBGN is proposed for small batch sizes,
however it under-performs BGN with $4.24\%$ at the batch size of 2, indicating
the importance of introducing the whole channel, height and width dimension to
compensate the noisy statistic calculation. IN overall performs not well on
ImageNet classification. LN, GN and PN achieve average Top1 accuracy of
$75.191\%$, $76.073\%$, $74.167\%$ respectively, while the proposed BGN
achieves higher average Top1 accuracy of $76.594\%$.

Table 2: The Top1 validation accuracy of BN, IN, LN, GN, PN, GBGN and BGN on ImageNet with ResNet-50 and with different batch sizes from 128 to 2. method | Batch size |
---|---|---
128 | 64 | 32 | 16 | 8 | 4 | 2
BN | 76.832 | 76.832 | 76.656 | 75.744 | 75.632 | 72.752 | 66.512
IN | 72.096 | 71.136 | 69.984 | 71.136 | 70.432 | 70.656 | 70.512
LN | 73.600 | 75.760 | 75.808 | 75.264 | 75.168 | 75.296 | 75.440
GN | 75.776 | 76.544 | 75.952 | 76.496 | 75.632 | 76.240 | 75.872
PN | 71.952 | 74.448 | 74.464 | 74.816 | 74.48 | 74.416 | 74.592
GBGN | 75.984 | 76.624 | 76.368 | 76.592 | 75.840 | 75.024 | 71.856
BGN | 77.008 | 77.104 | 76.784 | 76.624 | 76.208 | 76.336 | 76.096

### Image Classification on CIFAR-10 with NAS

Except manually designed and regular DCNN, BGN is applicable to automatically-
designed and less-regular ones as well. We experiment with cell-based
architectures designed automatically with NAS, specifically DARTS (Liu,
Simonyan, and Yang 2019) and Multi-agent Neural Architecture Search (MANAS
(Carlucci et al. 2019)). For DARTS, we experiment with normalization methods
for both the searching and training. For MANAS, we experiment with
normalization methods for the training only.

DARTS and MANAS share the same search space, the family of architectures
searched (the search space; see Fig. 2) is composed of a sequence of cells,
where each cell is a directed acyclic graph with nodes representing feature
maps and edges representing network operations, e.g. convolutions or pooling
layers ((Carlucci et al. 2019) and references therein). Given a set of
possible operations, DARTS encodes the architecture search space with
continuous parameters to form a one-shot model and performs searching by
training the one-shot model with bi-level optimization, where the model
weights and architecture parameters are optimized with training and validation
data alternatively. MANAS uses a multi-agent learning approach (the search
strategy) to find the combination of operations leading to the best-performing
architecture according the validation accuracy.

Figure 2: Cell-based architectures. Example of a normal cell (top) and a
reduction cell (bottom) in the DARTS and MANAS search space. Each cell has $2$
input (green), $4$ internal (blue) and $1$ output node (yellow). Multiple
cells are connected in a feedforward fashion to create a DCNN.

DARTS training configuration: we follow the same experiment setting as in
(Liu, Simonyan, and Yang 2019). We replace the BN layers in DARTS with IN, LN,
GN, PN and BGN in both search and evaluation stage. We search for 8 cells in
$50$ epochs with batch size $64$ and initial number of channels as 16. We use
SGD to optimize the model weights with initial learning rate $0.025$, momentum
$0.9$ and weight decay $3\times 10^{-4}$. Adam (Kingma and Ba 2014) is used to
optimize architecture parameters with initial learning rate $3\times 10^{-4}$,
momentum $(0.5,0.999)$ and weight decay $10^{-3}$. We use network of 20 cells
and 36 initial channels for evaluation to ensure a comparable model size as
other baseline models. We use the whole training set to train the model for
600 epochs with batch size 96 to ensure convergence. For GN, we use $\rm G=32$
in (Wu and He 2018) while for BGN, we use $\rm G=256$ following Tab. 2. Other
hyper-parameters are set the same as the ones in the search stage.

The best 20-cell architecture searched on CIFAR-10 by DARTS is trained from
scratch with corresponding normalization methods used during the search phase.
The validation accuracy of each method is reported in Tab. 3. We can see that
IN and LN fails to converge while BGN out-performs GN and PN significantly and
outperforms BN slightly. The accuracy of BN is re-implication of (Liu,
Simonyan, and Yang 2019).

Table 3: The validation accuracy on CIFAR-10 with using BN, IN, LN, GN, PN and BGN in DARTS for the search and evaluation phase. Normalization layer | BN | IN | LN | GN | PN | BGN
---|---|---|---|---|---|---
accuracy | $97.33$ | - | - | $94.78$ | 94.41 | 97.40

MANAS training configuration: a single GPU is used to train the searched
neural architectures (by BN) with replacing the normalization layers into BN,
IN, LN, GN, PN and BGN. For GN, we use the best configuration $\rm G=32$ in
(Wu and He 2018) while for BGN, we use $\rm G=64$. The network training
protocol is the same as in (Carlucci et al. 2019), with the following
hyperparameters: batch size $64$, epochs $600$, cutout length $16$, drop path
probability $0.2$, gradient clip $5.0$, initial channels $36$, Cross-Entropy
loss, SGD optimizer, learning rate decayed from $0.025$ to $0.0$, momentum
$0.9$, weight decay $0.0003$.

The best 20-cell architecture searched on CIFAR-10 by MANAS is retrained from
scratch with different normalization methods in place of the original BN used
during the search phase. The validation accuracy of each method is reported in
Tab. 4. We can see that IN, LN and PN fails to converge while BGN out-performs
GN significantly and under-performs BN only slightly. It is worth noting that
BN is used as the normalization layer in the neural architecture search phase,
hence BN is at an advantage in this comparison.

Table 4: The validation accuracy on CIFAR-10 with replacing the normalization layer in the nerual architecture searched by MANAS to BN, IN, LN, GN, PN and BGN. Normalization layer | BN | IN | LN | GN | PN | BGN
---|---|---|---|---|---|---
accuracy | 97.18 | - | - | $95.52$ | - | $97.15$

DARTS experiment shows that BGN is generalizable to NAS for both search and
evaluation. MANAS experiment shows that BGN is generalizable to less-regular
neural architectures searched from NAS method.

### Adversarial Training on CIFAR-10

DCNNs have been known to be vulnerable to malicious perturbed examples, known
as adversarial attacks. Adversarial training was proposed to counter this
problem. In this experiment, we apply BGN to adversarial training and compare
its performance to BN, IN, LN, GN, and PN.

Implementation details: the WideResNet (Zagoruyko and Komodakis 2016) with the
depth set as 10 and the wide factor set as 2 is used for image classification
tasks on the CIFAR-10. The neural network is trained and evaluated against a
four-step Projected Gradient Descent (PGD) attack. For the PGD attack, we set
the step size as $2/255$, and the maximum perturbation norm as 0.0157. 200
epochs are trained until convergence. Due to the specialty of adversarial
training, $\rm G=128$ is used in GN and BGN. It will divide images into
patches, which can help to improve the robustness by breaking the correlation
of adversarial attacks in different image blocks and constraining the
adversarial attacks on the features within a limited range. This effect holds
some similarity to the spectral normalization in (Farnia, Zhang, and Tse
2019). In the experiment, we use the Adam optimizer with a learning rate of
0.01.

The robust and clean accuracy of training WideResNet with BN, IN, LN, GN, PN
and BGN as the normalization layer are shown in Tab. 5. The robust accuracy is
more important than the clean accuracy in judging an adversarial network. PN
experiences convergence difficulty and fails to converge. BGN out-performs BN
and IN with a certain margin and out-performs LN and GN significantly.

Table 5: The robust and clean validation accuracy of adversarial training with BN, IN, LN, GN, PN and BGN as the normalization layer in WideResNet. The clean/robust accuracy is evaluated on the clean/PGD attacked data. Accuracy | BN | IN | LN | GN | PN | BGN
---|---|---|---|---|---|---
robust accuracy | 48.79 | 48.45 | 44.15 | 44.38 | $-$ | 49.64
clean accuracy | 72.09 | 72.35 | 64.1 | 68.96 | $-$ | 72.94

### Few Shot Learning

We evaluate BGN on FSL task. FSL aims to train models capable of recognizing
new, previously unseen categories using only limited training samples.
Basically, a training dataset with sufficient annotated samples comprise base
categories. The test dataset contains $C$ novel classes, each of which is
associated with only a few $K$ labelled samples (e.g. $\leq 5$ samples)
compose the support set, while the remaining unlabelled samples consist the
query set are used for evaluation (See Fig. 3). This is also referred to as a
$C$-way $K$-shot FSL classification problem.

Figure 3: The illustration of FSL classification problem. The base category
contains sufficient labelled samples, while the model aims at generalizing
well on the novel category in the query set with limited labelled samples
available in support set.

Implementation details: we experiment with imprinted weights (Qi, Brown, and
Lowe 2018) model, which is one of the state-of-art metric-based FSL approaches
and is widely used as the baseline in the current FSL community (Lifchitz et
al. 2019; Su, Maji, and Hariharan 2019; Gidaris and Komodakis 2018). At
training time, a cosine classifier is learned on top of feature extraction
layers and each column of classifier parameter weights can be regarded as a
prototype for the respective class. At test time, a new class prototype (new
column of classifier weight parameters) is defined by averaging the feature
representation of support images, and the unlabelled images are classified via
a nearest neighbor strategy. We test different settings, including 5-way
1-shot and 5-way 5-shot for the ResNet-12 backbone (Oreshkin, López, and
Lacoste 2018) on _mini_ ImageNet (Vinyals et al. 2016). We use the training
protocol described in (Gidaris and Komodakis 2018): our model is optimized
using SGD with Nesterov momentum set to $0.9$, weight decay to 0.0005, mini-
batch size to 256, and 60 epochs. All input images were resized to $84\times
84$. The learning rate was initialized to 0.1, and changed to 0.006, 0.0012,
and 0.00024 at the 20th, 40th and 50th, respectively.

The mean accuracy of replacing the normalization layers in Imprinted Weights
to BN, IN, LN, GN, PN and BGN, of training on _mini_ ImageNet, and of the
5-way 1-shot and 5-shot tasks are shown Tab. 6. We can see that BGN out-
performs BN slightly while out-performs IN, LN, GN and PN significantly,
indicating the generalizability of BGN when the very limited labeled data is
available.

Table 6:  The mean accuracy of the 5-way 1-shot and 5-shot tasks on _mini_ ImageNet of Imprinted Weights with using ResNet-12 as a backbone. The normalization layer is replaced to BN, IN, LN, GN, PN and BGN. The mean accuracy of 600 randomly generated test episodes with 95% confidence intervals is reported. Model | BN | IN | LN | GN | PN | BGN
---|---|---|---|---|---|---
1-shot | 59.30 | 52.17 | 57.82 | 56.55 | 56.59 | 59.50
5-shot | 76.22 | 70.49 | 74.87 | 73.20 | 73.89 | 76.32

### Unsupervised Domain Adaptation on Office-31

UDA aims to learn models on a target domain while annotations are only
accessible in a related source domain. Normalization layers have effect of
aligning feature distributions and reducing the domain gap (Li et al. 2016;
Cariucci et al. 2017). We evaluate BGN and other normalization layers on a
widely adopted UDA benchmark Office-31 (Saenko et al. 2010), which consists of
4110 images belonging to 31 classes, with three different domains: Amazon,
Webcam and Digital SLR camera (DSLR). CAN (Kang et al. 2019) is adopted as our
model with replacing original BN with different normalization layers.

Implementation details: we follow the official released code’s implementation
of CAN and use ImageNet-pretrained ResNet-50 as the model’s backbone. For
tasks da (from domain DSLR to Amazon), wa and wd, the hyper-parameter $\rm G$
is set to 512. For ad and aw, we reduce $\rm G$ to 1 and 8 relatively as the
source domain Amazon’s backgrounds are totally white and may result in noisy
statistics when the group size is small. For dw, $\rm G$ is set to 32. We use
Adam optimizer to optimize our model (Kingma and Ba 2014). The learning rate
is set to 0.001 and exponential learning rate decay is applied with decay rate
0.1 and decay step 25. The mini-batch size is set to 30. The training stops
when the distance between source and target features’ center is smaller than
0.001.

The results of BN, IN, LN, GN, PN and BGN are summarized in Table 7. We can
see that BGN outperforms other normalization layers in most adaptation tasks,
especially in wa with an $1.5\%$ accuracy improvement.

Table 7: The adaptation accuracy on Office-31 of CAN model with BN, IN, LN, GN, PN and BGN as the normalization layer. The result of each entry is averaged by three runs. model | ad | da | wa | aw | dw | wd | mean
---|---|---|---|---|---|---|---
BN | 94.8 | 77.2 | 76.1 | 94.2 | 98.4 | 99.7 | 90.1
BGN | 95.2 | 78.5 | 78.5 | 94.2 | 99.1 | 99.9 | 90.9
GN | 90.0 | 77.2 | 77.2 | 91.1 | 96.9 | 99.1 | 88.6
IN | 88.0 | 76.1 | 75.4 | 90.0 | 97.2 | 98.5 | 87.5
LN | 92.8 | 76.8 | 76.0 | 91.1 | 97.8 | 98.7 | 88.9
PN | 90.9 | 76.5 | 77.1 | 90.6 | 97.8 | 99.5 | 88.7

## Conclusion

BGN is proposed with good performance, stability and generalizability and
without using additional trainable parameters, information across multiple
layers or iterations, or extra computation. BGN facilitates the noisy/confused
statistic calculation in BN with adaptively introducing feature instances from
the grouped (channel, height and width) dimensions and uses a hyper-parameter
$\rm G$ to control the size of divided feature groups. It is intuitive to
implement, is orthogonal to and can be used in addition to many methods
reviewed in Related Work to further improve performance.

## References

  * Arora, Lyu, and Li (2019) Arora, S.; Lyu, K.; and Li, Z. 2019.  Theoretical analysis of auto rate-tuning by batch normalization.  In _ICLR_.
  * Arpit et al. (2016) Arpit, D.; Zhou, Y.; Kota, B.; and Govindaraju, V. 2016.  Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks.  In _ICML_ , 1168–1176.
  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _NeurIPS Deep Learning Symposium_ .
  * Bjorck et al. (2018) Bjorck, N.; Gomes, C. P.; Selman, B.; and Weinberger, K. Q. 2018.  Understanding batch normalization.  In _NeurIPS_ , 7694–7705.
  * Cai, Li, and Shen (2019) Cai, Y.; Li, Q.; and Shen, Z. 2019.  A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent.  In _ICML_ , 882–890.
  * Cariucci et al. (2017) Cariucci, F. M.; Porzi, L.; Caputo, B.; Ricci, E.; and Bulo, S. R. 2017.  Autodial: Automatic domain alignment layers.  In _ICCV_ , 5077–5085. IEEE.
  * Carlucci et al. (2019) Carlucci, F.; Esperança, P. M.; Singh, M.; Yang, A.; Gabillon, V.; Xu, H.; Chen, Z.; and Wang, J. 2019.  MANAS: Multi-agent neural architecture search.  _arxiv:1909.01051_ .
  * Chang et al. (2019) Chang, W.-G.; You, T.; Seo, S.; Kwak, S.; and Han, B. 2019.  Domain-Specific Batch Normalization for Unsupervised Domain Adaptation.  In _CVPR_ , 7354–7362.
  * Chen et al. (2018) Chen, Z.; Badrinarayanan, V.; Lee, C.-Y.; and Rabinovich, A. 2018.  GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks.  In _ICML_ , 794–803.
  * Cho and Lee (2017) Cho, M.; and Lee, J. 2017.  Riemannian approach to batch normalization.  In _NeurIPS_ , 5225–5235.
  * Cooijmans et al. (2016) Cooijmans, T.; Ballas, N.; Laurent, C.; Gülçehre, Ç.; and Courville, A. 2016.  Recurrent batch normalization.  _ICLR_ .
  * Data et al. (2018) Data, G. W. P.; Ngu, K.; Murray, D. W.; and Prisacariu, V. A. 2018.  Interpolating Convolutional Neural Networks Using Batch Normalization.  In _ECCV_ , 591–606. Springer.
  * De and Smith (2020) De, S.; and Smith, S. L. 2020.  Batch normalization biases deep residual networks towards shallow paths.  _arXiv preprint arXiv:2002.10444_ .
  * Deecke, Murray, and Bilen (2019) Deecke, L.; Murray, I.; and Bilen, H. 2019.  Mode Normalization.  In _ICLR_.  URL https://openreview.net/forum?id=HyN-M2Rctm.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 248–255. Ieee.
  * (16) Everingham, M.; Van Gool, L.; Williams, C. K. I.; Winn, J.; and Zisserman, A. ????  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * Fan (2017) Fan, L. 2017.  Revisit fuzzy neural network: Demystifying batch normalization and ReLU with generalized hamming network.  In _NeurIPS_ , 1923–1932.
  * Farnia, Zhang, and Tse (2019) Farnia, F.; Zhang, J.; and Tse, D. 2019.  Generalizable Adversarial Training via Spectral Normalization.  In _ICLR_.  URL https://openreview.net/forum?id=Hyx4knR9Ym.
  * Gidaris and Komodakis (2018) Gidaris, S.; and Komodakis, N. 2018.  Dynamic few-shot visual learning without forgetting.  In _CVPR_.
  * Goyal et al. (2017) Goyal, P.; Dollár, P.; Girshick, R.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv preprint arXiv:1706.02677_ .
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _CVPR_ , 770–778.
  * Hoffer et al. (2018) Hoffer, E.; Banner, R.; Golan, I.; and Soudry, D. 2018.  Norm matters: efficient and accurate normalization schemes in deep networks.  In _NeurIPS_ , 2160–2170.
  * Hou et al. (2019) Hou, L.; Zhu, J.; Kwok, J.; Gao, F.; Qin, T.; and Liu, T.-y. 2019.  Normalization Helps Training of Quantized LSTM.  In _NeurIPS_ , 7344–7354.
  * Huang et al. (2017) Huang, L.; Liu, X.; Liu, Y.; Lang, B.; and Tao, D. 2017.  Centered weight normalization in accelerating training of deep neural networks.  In _ICCV_ , 2803–2811.
  * Huang et al. (2018) Huang, L.; Yang, D.; Lang, B.; and Deng, J. 2018.  Decorrelated batch normalization.  In _CVPR_ , 791–800.
  * Huang et al. (2019) Huang, L.; Zhou, Y.; Zhu, F.; Liu, L.; and Shao, L. 2019.  Iterative normalization: Beyond standardization towards efficient whitening.  In _CVPR_ , 4874–4883.
  * Huang and Belongie (2017) Huang, X.; and Belongie, S. 2017.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _ICCV_ , 1501–1510.
  * Ioffe (2017) Ioffe, S. 2017.  Batch renormalization: Towards reducing minibatch dependence in batch-normalized models.  In _NeurIPS_ , 1945–1953.
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In _ICML_ , 448–456.
  * Jia, Chen, and Chen (2019) Jia, S.; Chen, D.-J.; and Chen, H.-T. 2019.  Instance-Level Meta Normalization.  In _CVPR_ , 4865–4873.
  * Kang et al. (2019) Kang, G.; Jiang, L.; Yang, Y.; and Hauptmann, A. G. 2019.  Contrastive adaptation network for unsupervised domain adaptation.  In _CVPR_ , 4893–4902.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ .
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  Imagenet classification with deep convolutional neural networks.  In _NeurIPS_ , 1097–1105.
  * Li et al. (2019a) Li, B.; Wu, F.; Weinberger, K. Q.; and Belongie, S. 2019a.  Positional Normalization.  In _NeurIPS_ , 1620–1632.
  * Li et al. (2018a) Li, P.; Xie, J.; Wang, Q.; and Gao, Z. 2018a.  Towards faster training of global covariance pooling networks by iterative matrix square root normalization.  In _CVPR_ , 947–955.
  * Li et al. (2019b) Li, X.; Chen, S.; Hu, X.; and Yang, J. 2019b.  Understanding the disharmony between dropout and batch normalization by variance shift.  In _CVPR_ , 2682–2690.
  * Li and Vasconcelos (2019) Li, Y.; and Vasconcelos, N. 2019.  Efficient Multi-Domain Learning by Covariance Normalization.  In _CVPR_ , 5424–5433.
  * Li et al. (2018b) Li, Y.; Wang, N.; Shi, J.; Hou, X.; and Liu, J. 2018b.  Adaptive batch normalization for practical domain adaptation.  _PR_ 80: 109–117.
  * Li et al. (2016) Li, Y.; Wang, N.; Shi, J.; Liu, J.; and Hou, X. 2016.  Revisiting batch normalization for practical domain adaptation.  _arXiv preprint arXiv:1603.04779_ .
  * Lifchitz et al. (2019) Lifchitz, Y.; Avrithis, Y.; Picard, S.; and Bursuc, A. 2019.  Dense Classification and Implanting for Few-Shot Learning.  _CVPR_ .
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _ECCV_ , 740–755. Springer.
  * Liu, Simonyan, and Yang (2019) Liu, H.; Simonyan, K.; and Yang, Y. 2019.  DARTS: Differentiable Architecture Search.  In _International Conference on Learning Representations_.  URL https://arxiv.org/abs/1806.09055.
  * Luo et al. (2019a) Luo, P.; Ren, J.; Peng, Z.; Zhang, R.; and Li, J. 2019a.  Differentiable Learning-to-Normalize via Switchable Normalization.  In _ICLR_.  URL https://openreview.net/forum?id=ryggIs0cYQ.
  * Luo et al. (2019b) Luo, P.; Wang, X.; Shao, W.; and Peng, Z. 2019b.  Towards Understanding Regularization in Batch Normalization.  In _ICLR_.  URL https://openreview.net/forum?id=HJlLKjR9FQ.
  * Luo et al. (2019c) Luo, P.; Zhang, R.; Ren, J.; Peng, Z.; and Li, J. 2019c.  Switchable normalization for learning-to-normalize deep representation.  _TPAMI_ .
  * Miyato et al. (2018) Miyato, T.; Kataoka, T.; Koyama, M.; and Yoshida, Y. 2018.  Spectral normalization for generative adversarial networks.  _arXiv preprint arXiv:1802.05957_ .
  * Nam and Kim (2018) Nam, H.; and Kim, H.-E. 2018.  Batch-instance normalization for adaptively style-invariant neural networks.  In _NeurIPS_ , 2558–2567.
  * Oreshkin, López, and Lacoste (2018) Oreshkin, B.; López, P. R.; and Lacoste, A. 2018.  Tadam: Task dependent adaptive metric for improved few-shot learning.  In _NeurIPS_.
  * Park et al. (2019) Park, T.; Liu, M.-Y.; Wang, T.-C.; and Zhu, J.-Y. 2019.  Semantic image synthesis with spatially-adaptive normalization.  In _CVPR_ , 2337–2346.
  * Qi, Brown, and Lowe (2018) Qi, H.; Brown, M.; and Lowe, D. G. 2018.  Low-shot learning with imprinted weights.  In _CVPR_ , 5822–5830.
  * Ren et al. (2016) Ren, M.; Liao, R.; Urtasun, R.; Sinz, F. H.; and Zemel, R. S. 2016.  Normalizing the normalizers: Comparing and extending network normalization schemes.  _ICLR_ .
  * Saenko et al. (2010) Saenko, K.; Kulis, B.; Fritz, M.; and Darrell, T. 2010.  Adapting visual category models to new domains.  In _ECCV_ , 213–226. Springer.
  * Salimans and Kingma (2016) Salimans, T.; and Kingma, D. P. 2016.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  In _NeurIPS_ , 901–909.
  * Santurkar et al. (2018) Santurkar, S.; Tsipras, D.; Ilyas, A.; and Madry, A. 2018.  How does batch normalization help optimization?  In _NeurIPS_ , 2483–2493.
  * Shao et al. (2019) Shao, W.; Meng, T.; Li, J.; Zhang, R.; Li, Y.; Wang, X.; and Luo, P. 2019.  Ssn: Learning sparse switchable normalization via sparsestmax.  In _CVPR_ , 443–451.
  * Singh and Shrivastava (2019) Singh, S.; and Shrivastava, A. 2019.  EvalNorm: Estimating Batch Normalization Statistics for Evaluation.  In _ICCV_ , 3633–3641.
  * Su, Maji, and Hariharan (2019) Su, J.-C.; Maji, S.; and Hariharan, B. 2019.  Boosting Supervision with Self-Supervision for Few-shot Learning.  _arXiv_ .
  * Summers and Dinneen (2019) Summers, C.; and Dinneen, M. J. 2019.  Four Things Everyone Should Know to Improve Batch Normalization.  _arXiv preprint arXiv:1906.03548_ .
  * Ulyanov, Vedaldi, and Lempitsky (2016) Ulyanov, D.; Vedaldi, A.; and Lempitsky, V. 2016.  Instance normalization: The missing ingredient for fast stylization.  _arXiv preprint arXiv:1607.08022_ .
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016.  Matching networks for one shot learning.  In _NeurIPS_.
  * Wang et al. (2018) Wang, G.; Luo, P.; Wang, X.; Lin, L.; et al. 2018.  Kalman normalization: Normalizing internal representations across network layers.  In _NeurIPS_ , 21–31.
  * Wang et al. (2019) Wang, X.; Jin, Y.; Long, M.; Wang, J.; and Jordan, M. I. 2019.  Transferable Normalization: Towards Improving Transferability of Deep Neural Networks.  In _NeurIPS_ , 1951–1961.
  * Wu and He (2018) Wu, Y.; and He, K. 2018.  Group normalization.  In _ECCV_ , 3–19.
  * Xu et al. (2019a) Xu, J.; Sun, X.; Zhang, Z.; Zhao, G.; and Lin, J. 2019a.  Understanding and Improving Layer Normalization.  In _NeurIPS_ , 4383–4393.
  * Xu et al. (2019b) Xu, Y.; Duan, J.; Kuang, Z.; Yue, X.; Sun, H.; Guan, Y.; and Zhang, W. 2019b.  Geometry Normalization Networks for Accurate Scene Text Detection.  In _ICCV_ , 9137–9146.
  * Yan et al. (2019) Yan, J.; Wan, R.; Zhang, X.; Zhang, W.; Wei, Y.; and Sun, J. 2019.  Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization.  In _ICLR_.
  * Yang et al. (2019) Yang, G.; Pennington, J.; Rao, V.; Sohl-Dickstein, J.; and Schoenholz, S. S. 2019\.  A Mean Field Theory of Batch Normalization.  In _ICLR_.  URL https://openreview.net/forum?id=SyMDXnCcF7.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide Residual Networks.  In Richard C. Wilson, E. R. H.; and Smith, W. A. P., eds., _BMVC_ , 87.1–87.12. BMVA Press.  ISBN 1-901725-59-6.  doi:10.5244/C.30.87.  URL https://dx.doi.org/10.5244/C.30.87.
  * Zhang and Sennrich (2019) Zhang, B.; and Sennrich, R. 2019.  Root Mean Square Layer Normalization.  In _NeurIPS_ , 12360–12371.
  * Zhang, Dauphin, and Ma (2019) Zhang, H.; Dauphin, Y. N.; and Ma, T. 2019.  Residual Learning Without Normalization via Better Initialization.  In _ICLR_.  URL https://openreview.net/forum?id=H1gsz30cKX.
  * Zhou (2019) Zhou, X. 2019.  3D shape instantiation for intra-operative navigation from a single 2D projection .
  * Zhou et al. (2019) Zhou, X.-Y.; Li, P.; Wang, Z.-Y.; and Yang, G.-Z. 2019.  U-net training with instance-layer normalization.  In _International Workshop on Multiscale Multimodal Medical Imaging_ , 101–108. Springer.
  * Zhou and Yang (2019) Zhou, X.-Y.; and Yang, G.-Z. 2019.  Normalization in training U-Net for 2-D biomedical semantic segmentation.  _IEEE Robotics and Automation Letters_ 4(2): 1792–1799.


# RaSpectLoc: RAman SPECTroscopy-dependent robot LOCalisation

Christopher Thirgood1a, Oscar Mendez1a, Erin Chao Ling1b, Jon Storey2, Simon
Hadfield1a 1aCVSSP, Computer Science and Electronic Engineering, University of
Surrey, Guildford, Surrey, United Kingdom {c.thirgood, s.hadfield,
<EMAIL_ADDRESS>Institute for People-Centered Artificial
Intelligence, University of Surrey, Guildford Surrey, United Kingdom
<EMAIL_ADDRESS>3D Robotics, Tonbridge, Kent, UK
<EMAIL_ADDRESS>
###### Abstract

This paper presents a new information source for supporting robot
localisation: material composition. The proposed method complements the
existing visual, structural, and semantic cues utilized in the literature.
However, it has a distinct advantage in its ability to differentiate
structurally [23], visually [25] or categorically [1] similar objects such as
different doors, by using Raman spectrometers. Such devices can identify the
material of objects it probes through the bonds between the material’s
molecules. Unlike similar sensors, such as mass spectroscopy, it does so
without damaging the material or environment. In addition to introducing the
first material-based localisation algorithm, this paper supports the future
growth of the field by presenting a gazebo plugin for Raman spectrometers,
material sensing demonstrations, as well as the first-ever localisation data-
set with benchmarks for material-based localisation. This benchmarking shows
that the proposed technique results in a significant improvement over current
state-of-the-art localisation techniques, achieving 16% more accurate
localisation than the leading baseline.
The code and dataset will be released at:
https://github.com/ThirgoodC/RaSpectLoc

##  I Introduction

Mobile robots have historically relied on depth sensors for localisation
tasks, more recently, visual (RGB) sensors have become ubiquitous. However,
these traditional RGB approaches face challenges in urban environments where
large planar regions of uniform colour often dominate and distinct visual
landmarks are lacking. This is especially problematic in self-similar spaces
such as hotels with identical rooms and rotational symmetric floorplans.

A novel solution is to sense the material composition of the environment,
which can help distinguish visually similar objects, such as different doors.
Additionally, small deviations in material composition, such as impurities in
concrete or uneven levels of corrosion, can serve as local landmarks. These
observations can be made and the material composition identified using sensors
such as mass spectrometers and Raman spectrometers.

In this paper, we propose a new localisation approach that leverages the
capabilities of Raman spectroscopy. Raman probes are commonly used in physics
as active sensors to analyze at a range, without damaging the subject matter.
They work by analyzing molecular interactions and bonds through light
scattering. This results in a Raman spectrum featuring peaks corresponding to
specific molecular bond vibrations.

We present an approach that uses spectral responses produced by a Raman
spectrometer as its central sensing unit. It can work with or without range
information and demonstrates superior performance to vision and depth-based
methods. Furthermore, it effectively handles inaccuracies introduced in raw
responses from the sensor.

Fig. 1 demonstrates a visualisation of RaSpectLoc, where spectra are compared
against each other within a material map.

Figure 1: Visualisation of the RaSpectLoc system in action. Spectral readings
are compared between the environment and Map along a number of bearing
vectors.

In summary, the contributions of this paper are:

  1. 1.

A novel approach in mobile robotics that utilizes material-based spectroscopic
data for localisation.

  2. 2.

A plug-in for the popular simulator, Gazebo, to simulate Raman spectrometers.

  3. 3.

A new dataset containing maps and recordings of material composition from a
test environment.

  4. 4.

A benchmarked performance evaluation of our approach which provides a far
greater average trajectory error compared to current state-of-the-art
localisation approaches that utilize RGBD sensors.

##  II Literature Review

Mobile robot applications have traditionally used scan-matching approaches to
localise within a known map. These approaches have historically been dominated
by particle filter-based methods, such as AMCL [23] and GMapping [26] which
have been popular solutions due to the integration with the ROS navigation
stack [4]. The technique is range-based (RMCL) and is highly adaptable,
adjusting the number of hypotheses generated by the particle filter as
confidence in the result improves. However, AMCL may fail if the map and the
sourced environment differ. Many ‘Simultaneous Localisation and Mapping’
(SLAM) systems intend to solve this by generating a map at run-time and
identifying ‘loop closures’ to solve for drift [2, 3] for static environments.
For typical dynamic scenes such as industrial work-spaces [5, 19] these
approaches have attempted to track the pose using an apriori map and secondary
short-term map at two different timescales. The problem with such a method is
that it is too computationally complex for mobile robotics to reliably
localize for long periods with accumulated errors. Our approach aims to
resolve this problem via the use of efficient spectral comparison functions
which can provide continuous similarity scores. These are robust to
misalignments.

Many RMCL approaches seek to enhance MCL localisation for mobile robots by
using 3D point clouds generated from 3D LiDAR systems. Maken et al. [20]
utilized Iterative Closest Point (ICP) with LiDAR sensors for pose estimation
and localisation with MCL. Although improvements were evident in the results,
the accuracy of the covariance estimation of the ICP output poses is
dramatically affected by positioning accuracy. In recent deployments, machine
learning models have become common in localisation systems that utilize point
clouds to improve accuracy and reduce covariance estimation. For example,
SegMap proposed by Dube et al. [7] uses a Convolutional Neural Network (CNN)
to identify and segment geometric primitives in point clouds. However, the use
of local and global descriptors leads to slow performance due to the expensive
descriptor similarity networks. As a solution, SegSemMap [6] proposed by
Cramariuc et al. enriches point clouds with CNN segmentation overlays from RGB
cameras and adds them to the map. These approaches still require a compact and
computationally efficient representation of the objects for segmentation,
providing a high representation strength. Although SegMap and SemSegMap do not
use an MCL approach, many segmentation-based MCL approaches have been used in
combination with geometry and range, like Hendrikx et al. [16]. Another
example is 3D geometry in BIM models which can be used to estimate the camera
pose. The models are then segmented to gain local information about the area
using range readings from a 2D range-based sensor. Mendez et al. proposed
SeDAR [1] which aims to simplify these models with a more robust approach to
RMCL by removing the depth readings altogether. With a semantic understanding
of the environment through a simple CNN, the weights of the particles are
adjusted based on the likelihood of particle observations in the segmented
map. In contrast to the discrete semantic label comparisons used in SeDAR [1],
our proposed material comparison offers a much finer level of matching
granularity. Furthermore, computing likelihoods directly from the Raman
spectra in the environment without visual classification networks
significantly reduces computational costs.

There have been attempts to extend the applicability of AMCL to non-visual
sensing modalities. One such example is seen with haptic sensors, which can
provide touch-based sensing of the environment [32]. Buchanan et al. [8] have
proposed to use data segments from the surroundings with a neural network to
update MCL from a map of different terrains. However, this approach requires
extensive data on the environment’s terrain and prolonged pre-training, which
can be challenging. Another unconventional sensing approach was proposed by
Serrano et al. [9], who used the Wi-Fi signal strength of mobile devices for
MCL. Although this approach is dependent on the Wi-Fi signal data strength, it
can be unreliable in complex indoor environments where signals are absorbed
through many walls before being sensed by the receiver. Visible Light
Positioning (VLP) sensors have also been used, with W. Guan et al. [10] being
able to identify objects in a room using LED patterns from lights and other
devices. Despite these unconventional sensing methods, they still face
challenges with even minor changes in the environment, leading to significant
inaccuracies in the MCL algorithm. In contrast, Raman Spectrometers have the
advantage of treating every part of the environment as an identifying landmark
due to its material fingerprint sensing.

Raman spectrometers are remote scanning instruments often used to identify
materials or substances. Noise in the sensor comes in three forms, shot, dark
and read noise which couples to the Poisson distribution of the data,
background noise and electronics of the device, respectively [21]. Such noise
can be overwhelmed by gaussian noise from the device in the process of
scanning with modern probes. To identify or recognise similar Raman Spectra,
most approaches use a large database with neural networks producing long
training and inference times [14, 30]. In contrast, our approach simplifies
and extends this by using the likelihood between stored spectra in the map and
the latest spectra read from the environment. De-noising methods such as those
proposed by Lussier et al. and Horgan et al. [11, 14] show the benefits of
using deep-learning models to remove noise, but can also be done via other
methods suggested by Zhao et al. [17]. Analytical likelihood similarity
algorithms have been proposed to identify the type of material from the peaks
of the spectra as proposed by Foster et al. [12]. We propose alternative Raman
spectral similarity functions in RaSpectLoc by providing new algorithms while
still delivering a system with low computational complexity. The most
effective similarity functions for Raman spectra necessitate peak
classification through algorithmic convolutions [15]. This is infeasible for
MCL approaches as they tend to take too long to process each hypothesis.

The main focus of Raman spectrometry in the literature is remote sensing of
chemicals using mobile robots [13], whereas we go further and apply this
research for a robot localisation approach. This paper proposes a novel
approach, utilizing innovative localisation techniques and algorithmic
similarity functions for mobile robots equipped with underutilized Raman
Spectral sensors.

##  III Problem definition

Monte-Carlo Localisation by Dellaert et al. [23] provides a framework which
performs matching to generate pose hypotheses in a map. The process is as
follows:

  1. 1.

Particles are sampled, either uniformly around a space or in a Gaussian around
a pose hypothesis

  2. 2.

Particles are propagated through a motion model, typically the robot’s
odometry with added Gaussian noise.

  3. 3.

Each particle is given a weight based on the accuracy of its observations
against the map, typically matching a range and bearing scan line to an
existing map

  4. 4.

A re-sampling step is performed, proportional to the particle’s likelihood,
before the process repeats.

The current pose $x_{t}\in SE(2)$, of the robot, can be estimated as a set of
possible samples: $\mathbb{S}_{t}=\\{s_{t}^{i};i=1\dots N\\}$, given the wheel
odometry measurements: $\mathbb{U}_{t}=\\{u_{j};j=1\dots t\\}$ and depth
sensor measurements: $\mathbb{Z}_{t}=\\{z_{j};j=1\dots t\\}$ and a 2D map
$\mathbb{V}$. If all previous odometry and depth measurements are equally
weighted, the posterior probability
$P(s_{t}^{i}|\mathbb{U}_{t},\mathbb{Z}_{t},\mathbb{V})$ can be decomposed into
an online sequential process:

$P(s^{i}_{t}|\mathbb{U}_{t},\mathbb{Z}_{t},\mathbb{V})=P(s^{i}_{t}|u_{t},s^{i}_{t-1})P(\mathbb{Z}_{t-1}|\mathbb{U}_{t-1},s^{i}_{t-1},\mathbb{V})$
(1)

The motion model in the localisation process is determined by the odometry
measurements received from the robot, represented by $u_{t}$. This information
is used to “shift” the particles, assigning a likelihood based on the
probability of the final position given the measured odometry. The particles
are propagated based on $u_{t}$, with Gaussian noise added to model sensor
noises, as follows:

$P\left(s_{t}^{i^{\prime}}|u_{t},s_{t-1}^{i}\right)\sim\mathcal{N}\left(u_{t}+s_{t-1}^{i},\Upsilon_{t}\right)$
(2)

where $\Upsilon_{t}$ is the covariance of the odometry and $\mathcal{N}$ is a
normal distribution across the dimensions of SE(2).

The sensor model $P(\mathbb{Z}_{t-1}|\mathbb{U}_{t-1},s^{i}_{t-1},\mathbb{V})$
measures how well the range scan fits the pose hypothesis. The probability of
each range-scan $\left(r_{t}\right)$ is estimated under the assumption all
scans are independent. Two sensor models which are commonly used with AMCL are
the “Beam” and “Likelihood-field” models. The Beam model is a raycasting
operation where a ray is cast starting from hypothesis $s^{i}_{t}$ along the
bearing $\theta^{k}_{t}$ and terminates when an occupied cell is reached. The
likelihood is estimated as:

$P_{R}\left(\mathbb{Z}_{t}^{k}|s_{t}^{i^{\prime}}\right)=\exp\left({\frac{-\left(r_{t}^{k}-r_{t}^{k\ast}\right)^{2}}{2\sigma_{o}^{2}}}\right)$
(3)

where $\sigma$ is the variance of the sensor measurement noise and
$r_{t}=r_{t}^{k}\forall k=0...N$. The likelihood field model uses a field
similar to a chamfer map to quickly estimate the distance to the nearest
geometry in the floorplan, eliminating the need for costly raycasting
operations. The chamfer map is defined as:

$C_{i,j}=\min_{k,l}|[i-k,j-l]|,\mathbb{V}_{k,l}\neq 0$ (4)

where $C_{i,j}$ is the cost value of the chamfer map at position $(i,j)$,
while $\min$ is a minimum value function and points $(k,l)$ which iterates
over the neighbors of $(i,j)$. In the likelihood-field model, assuming a
Gaussian error distribution, the weight of each particle, $s^{\prime}$, can be
estimated as:

$P_{r}(\mathbb{Z}_{t}^{k}|s_{t}^{i^{\prime}},\mathbb{V})=\exp\left({-\frac{\delta_{o}^{2}}{2\sigma_{o}^{2}}}\right)$
(5)

where $\delta_{o}$ is the value obtained from the distance map and
$\sigma_{o}$ is dictated by the noise characteristics of the sensor. During
runtime, the endpoint of each bearing-range tuple is computed for each pose
hypothesis, and its probability is related to the distance stored in the
chamfer map. The likelihood field model is faster and produces more accurate
results compared to the beam model as it is more robust to orientation errors.

RaSpectLoc is similar to AMCL but replaces many of the operations listed above
for material-based sensing. Our method includes adapting a floorplan for Raman
spectra, a novel spectral-based raycasting and a particle weight calculation
method for each hypothesis.

Figure 2: Material Map used in experimentation

##  IV METHODOLOGY

Figure 3: RaSpectLoc system diagram

Fig. 3 illustrates an overview of the RaSpectLoc system. The system requires
input odometry of the robot and a materials map with the Raman spectra
embedded. After initialisation, section IV-B details how Raman spectra are
accumulated around the robot. Section IV-C describes how the weight is
calculated based on the spectral input from the mounted probe of the mobile
robot. The particles propagate according to the motion model of Eq. 2. The
system then repeats this process measuring the materials in the local
environment around the robot with the Raman Probe and comparing them against
the map.

###  IV-A Material Floorplans

The RaSpectLoc system offers a more innovative solution, utilizing spectral
data obtained from a Raman probe. The system compares this data against
previously recorded Raman spectra that are incorporated into the map. In this
work, each cell on the floorplan is associated with a unique Raman spectrum
measurement. This map is used in the ray casting operation to determine the
likelihood between the data and ends of the rays. The materials represented in
our floorplan for this work include various spectra for painted walls,
laminate wood, painted metals, wood and plastic. These materials are broadly
grouped based on similarity for visualization purposes, depicted in Fig. 2.

The map is converted to an occupancy grid when initially uploaded to a map
server. If $\mathbb{M}$ is a set of possible 2D positions, the map can then be
defined as $\mathbb{V}=\\{v_{m};m\in\mathbb{M}\\}$. Each embedded spectrum is
formally described by $\mathbb{I}_{i}=\\{i_{0},\dots,i_{n}\\}$ where $i$
reflects the intensity of a particular wavelength.

###  IV-B Raman Sensor

Figure 4: RP1000 Raman Probe [31]

At the time $t$, the sensor on the robot generates a message consisting of a
tuple of ranges, bearings, and Raman spectra, $\mathbb{Z}_{t}$. Modern Raman
spectra allow a non-invasive technique with background noise scattering
removal for high-accuracy readings with little post-processing for smoothed
spectral results. RaSpectLoc can also operate without an explicit range sensor
(i.e. only using bearing and spectra tuples) as explained in section IV-D. In
this case, raycasting is used as with the “beam” sensor model, and the range
component of the likelihood is omitted. The measurements are configured as
$\mathbb{Z}_{t}=[\langle
r_{t}^{k},\Theta_{t}^{k},\mathbb{I}_{t}^{k}\rangle;k=1\dots k]$, which
represents the range from the depth camera, the bearing along the scan line,
and the Raman spectrum respectively. The scan line is assumed to be parallel
to the ground plane and aligned with the horizontal axis of the mobile robot’s
sensor mounting point. Raman measurements are taken by rotating the joint
around the robot to complete a single scan message with $k$ measurements. In
the gazebo simulator, our gazebo plug-in can be mounted anywhere on a modelled
robot and produce a rotating scan line.

Our approach in RaSpectLoc aims to tackle the challenges faced in localisation
for modular robots, which are currently dominated by geometry primitive
recognition methods that require higher-end hardware. We leverage the
similarity between spectra that are read from the environment after
normalisation and baseline correction is performed on them. The information we
gather is converted into chamfer maps, enabling the RaSpectLoc system to
calculate the similarity for each particle. Despite the advanced technology,
current Raman probes still have a level of noise that must be accounted for
when computing the similarity between observation and map. The shot, readout,
thermal background, and baseline noise are Poisson distributed and assumed to
be proportional to the square root of the number of photons detected. Most
Raman systems require proximity to the target in question. The IS-Instruments
RP1000 probe, coupled to a HES2000, provides the capability of making Raman
measurements at ranges over 1m. Such a probe is ideal for robot deployment and
an ideal candidate for this application.

The noise in low response areas of spectra substantially affects the results
using many comparison functions. When there is a significant number of
(photonic) events, Poisson noise is indistinguishable from Gaussian noise as
described by Larkin [27] and also by Lewis et al. [28]. Therefore, we use a
Squared exponential mapping function to convert spectrum distance to a
likelihood, $P_{s}$, mapped between $0~{}\overrightarrow{}~{}1$:

$P_{s}\left(\mathbb{Z}_{t}|s_{t},\mathbb{V}\right)=\mathrm{exp}\left(\dfrac{-f\left(\mathbb{I}^{k}_{t},\mathbb{I}^{m}\right)^{2}}{K}\right)$
(6)

where $f\left(\mathbb{I}^{k}_{t},\mathbb{I}^{m}\right)$ is the distance
function of the spectra $\mathbb{I}^{k}_{t}$ and $\mathbb{I}^{m}$ which is the
spectra at endpoint of a raycasted beam $m$. While $K$ is a constant used to
scale the different distance metrics into the same range. Our approach
supports previous state-of-the-art mathematical similarity functions for
spectra [15], in addition to originally proposed methods.

###  IV-C Spectral Similarity Functions

In RaSpectLoc, various similarity functions can be used to compare Raman
spectra and calculate the particle weights. We will describe these functions
in detail and their advantages and disadvantages. The similarity functions
described in this section are the Spectral Linear Kernel, Modified Euclidean
distance [15], Wasserstein distance [18], Kullback-Leibler distance and
Spectral Angle Mapping (SAM).

####  IV-C1 Spectral Linear Kernel

The spectral linear kernel (SLK), proposed by Conroy et al. [18], was designed
for comparing Raman spectra. It can be expressed by:

$f_{slk}(\mathbb{I}^{k}_{t},\mathbb{I}^{m})=\\!\sum_{i_{n}^{k}\in\mathbb{I}^{k}_{t}}\left(i^{k}_{n}\cdot
i^{m}_{n}+\\!\sum_{j=n-W}^{j=n+W}(i^{k}_{n}-i^{k}_{j})(i^{m}_{n}-i^{m}_{j})\\!\right)$
(7)

where $\mathbb{I}^{k}$ and $\mathbb{I}^{m}$ are the input spectra. The kernel
considers the original intensity values at each wave number and includes the
difference between the intensity of its neighbouring points on the spectrum.
The windowed difference product helps compare the relative shapes of the two
spectra. However, relative to other functions mentioned in this paper it is
slow and results can be poor when calculating the similarity over long windows
of non-similar regions in the two spectra.

####  IV-C2 Modified Euclidian Spectral Similarity metric (Mod. L2)

The approach introduced by Khan et al. [15] is an alternative to the Euclidian
similarity calculated between each intensity. This approach gives equal
importance to the peaks of each spectrum but also rewards spectra that have
shorter distances between the peaks. The squared distance between intensities
of the spectra can be calculated via
$D^{i}_{\mathbb{I}}=(\mathbb{I}^{k}_{i}-\mathbb{I}^{m}_{i})^{2}$. This can
then be used under the conditions in equation (7) to calculate the Mod. L2
distance:

$\\!f_{\\!me}(\mathbb{I}^{k}_{t},\mathbb{I}^{m})^{2}=\begin{cases}\\!\displaystyle\sum_{i_{n}^{k}\in\mathbb{I}^{k}_{t}}\\!\frac{1}{w}\ast
D^{i}_{\mathbb{I}},&\\!\text{if }x_{i}\neq 0\text{ AND }z_{i}>x_{i}\\\
\\!\displaystyle\sum_{i_{n}^{k}\in\mathbb{I}^{k}_{t}}\\!w\ast
D^{i}_{\mathbb{I}},&\text{if }x_{i}=0\text{ AND }z_{i}>x_{i}\\\
\\!\displaystyle\sum_{i_{n}^{k}\in\mathbb{I}^{k}_{t}}\\!D^{i}_{\mathbb{I}},\\!&\text{otherwise}\end{cases}$
(8)

The function above calculates a distance between the spectra using a weight,
$w$, found by:

$w=\frac{max(\mathbb{I}^{k}_{t})}{1-max(\mathbb{I}^{k}_{t})}$ (9)

The weight $w$ is utilized as a penalty/reward scheme for relative changes in
individual intensities at each wavelength. The drawback of this function is
that the weight does not work when the $maxQ$ value is less than 0.5. To
resolve this issue, the weight is inverted to correct the error from the
original equation.

####  IV-C3 Wassertein Distance

The Wasserstein distance, also known as the Earth Mover’s Distance, is a
metric that quantifies the effort required to transform one distribution into
another. It was suggested in previous literature to be an effective method of
evaluating the similarity of spectra by Gao et al. [28]. Despite its accuracy,
the Wasserstein distance is complex and slow compared to the other functions
mentioned in this section. This is explored in Section V.

$f_{W}(\mathbb{I}^{k}_{t},\mathbb{I}^{m})=min\\{E(t)|t:\mathbb{I}^{k}_{t}\rightarrow\mathbb{I}^{m}\\}$
(10)

Where $t$ is a transportation plan that maps the elements of
$\mathbb{I}^{k}_{t}$ to the elements of $\mathbb{I}^{m}$, and $E(t)$ is the
cost of the transportation plan $t$, which is typically represented as the sum
of the distances between the corresponding vector elements in
$\mathbb{I}^{k}_{t}$ and $\mathbb{I}^{m}$.

####  IV-C4 Kullback-Leibler Divergence

The KL-Divergence is a highly efficient yet effective distance to compare two
equally binned distributions. Formally it is a generalisation on the L2 Norm
and is defined as the

$f_{kl}(\mathbb{I}^{k}_{t}||\mathbb{I}^{m})=\sum_{i_{n}^{k}\in\mathbb{I}^{k}_{t}}i^{k}_{n}\log\frac{i^{k}_{n}}{i^{m}_{n}}$
(11)

The KL-divergence quantifies the information lost when approximating
$\mathbb{I}^{k}_{t}$ with $\mathbb{I}^{m}$. It is a non-symmetric metric,
meaning that $f_{KL}(\mathbb{I}^{k}_{t}||\mathbb{I}^{m})\neq
f_{KL}(\mathbb{I}^{k}_{t}||\mathbb{I}^{m})$. The KL divergence is sensitive to
tiny differences between distributions, which is desirable in the context of
Raman spectra, where little variations in intensity values can indicate the
presence of different materials.

####  IV-C5 Spectral Angle Mapping

The Spectral Angle Mapping (SAM) function is a similarity metric commonly used
to calculate the angle between two spectra in a high-dimensional space. Each
dimension represents a different Raman shift. The angle between the two
spectra shows their similarity, with a smaller angle indicating higher
similarity. Formally the function can be expressed by:

$f_{sam}(\mathbb{I}^{k}_{t}||\mathbb{I}^{m})=\cos^{-1}\left(\frac{\mathbb{I}^{k}_{t}\cdot\mathbb{I}^{m}}{\left\lVert\mathbb{I}^{k}_{t}\right\rVert\left\lVert\mathbb{I}^{m}\right\rVert}\right)$
(12)

SAM is founded on the idea that similar spectra should have similar shapes and
relative intensities across different Raman shifts benefiting from the
similarity angle returned around the shape of each peak. Furthermore, the
function is also scale-invariant, meaning it is not affected by differences in
the absolute intensity of the spectra. However, SAM is quite sensitive to
noise and variations of the baseline of the Raman spectra since they affect
the shape of the peaks. SAM also fails to consider the intensity of the
spectra and assumes spectra are linearly mixed. This assumption may not be the
case for some materials in the environment.

###  IV-D Sensor Models

We adapt both the “beam” and “likelihood-field” sensor models for use with
Raman Spectra.

The Beam model depends on ranges as part of the weight for each particle. This
is because the beam model depends on the depth of the scan from the
environment and the end of the beam projected into the map, $\mathbb{V}$.
While the probability of an observation given the map and pose for the beam
model can be expressed by a weighted sum of the two probabilities:

$\begin{split}P_{r}(\mathbb{Z}_{t}^{k}|s_{t}^{i},\mathbb{V})=&\
\epsilon_{R}P_{R}(\mathbb{Z}_{t}^{k}|s_{t}^{i},\mathbb{V})+\epsilon_{m}P_{m}(\mathbb{Z}_{t}^{k}|s_{t}^{i},\mathbb{V})\end{split}$
(13)

For the Beam-model of RaSpectLoc, the likelihood of a particle is defined by
two user-defined weights, $\epsilon_{R}$ and $\epsilon_{m}$. When
$\epsilon_{m}$ is zero, the likelihood is equivalent to the standard RMCL. The
optimal weight configuration is evaluated in Section V. Unlike range scanners,
the standard deviation $\sigma^{{}^{\prime}}$ of equation 2 cannot be linked
to the sensor’s physical properties. Instead, it is estimated from the prior
of each material spectra on the map, avoiding the need for tuning. However,
the Beam Model has drawbacks to its approach as it lacks smoothness since it
depends on the resolution of the map. This is worse for mobile robots as it
becomes more memory dependent to increase the number of beams and resolution
of the map.

The likelihood-field model provides a smoother approach providing gradients
between each cell. Furthermore, it can use ranges, bearing and material
likelihood or only bearing and material likelihoods. The likelihood field
model in our approach generates a chamfer map to the nearest occupied cell for
each material based on equation 2:

$C_{i,j}\left(\mathbb{I}\right)=\min_{k,l}\left(C_{k,l}\left(\mathbb{I}\right)+f\left(\mathbb{I}_{i,j},\mathbb{I}_{k,l}\right)\right)$
(14)

Where $f$ is the chosen spectral likelihood function mentioned in Section
III-B. For every
$\mathbb{Z}_{t}=[\langle\Theta_{t}^{k},\mathbb{I}_{t}^{k}\rangle;k=1\dots k]$,
the raycasting is performed. Instead of comparing the range with the map, the
spectral likelihood can be used to calculate the cost:

$P_{r}(\mathbb{Z}_{t}^{k}|s_{t}^{i},\mathbb{V})=P_{m}(\mathbb{Z}_{t}^{k}|s_{t}^{i},\mathbb{V})$
(15)

This method is a combination of the beam and likelihood model. Hence, in Eq.
13, when $\epsilon_{R}$ is zero, the approach solely relies on the Raman
spectra information in the floorplan and material similarity likelihood.

The nearest occupied cell is identified through similar AMCL range-based
raycasting. The distances obtained are utilized to improve the smoothness of
Eq. 13, which suggests that the likelihood of the spectra similarity is
proportional to the distribution of material similarity in terms of angle. As
a result, this approach is scale-invariant, provided that the aspect ratio of
the map is maintained.

##  V Experiments and Results

Our research focuses on MCL-based localisation using Raman Spectra from a
single probe. This section presents the overall performance improvements of
our system over other state-of-the-art localisation approaches. This dataset
will be released to the community to support future research. RaSpectLoc can
choose between the beam or the likelihood field model which are described in
Section IV-D. The ablation and parameter exploration section, V-B, illustrates
the difference in the performance of RaSpectLoc due to changes in likelihood
functions and configurations of $\epsilon$. Whereas the Quantitative Results
section V-C will compare RaSpectLoc with various state-of-the-art baselines.
These include, AMCL [23], SeDAR [1], visual pose recognition (PoseNet [25]),
range-based SLAM (GMapping [26]) and Monocular SLAM (ORBSLAM 3 [24]).

Spectral Similarity Function Analysis (m)
---
Function | RMSE | Mean | Median | SD | Min | Max
SLK | 0.18 | 0.17 | 0.16 | 0.07 | 0.03 | 0.38
Mod.L2 | 0.14 | 0.13 | 0.12 | 0.06 | 0.03 | 0.41
Wasserstein | 0.17 | 0.14 | 0.13 | 0.09 | 0.02 | 0.44
KL.D | 0.15 | 0.13 | 0.11 | 0.08 | 0.02 | 0.34
SAM | 0.18 | 0.16 | 0.14 | 0.09 | 0.03 | 0.42

TABLE I: Spectral Similarity Function Experiments

Spectral Similarity Function Analysis (RPE)
---
Function |  | Translational
---
RMSE(m)
| Rotational
---
RMSE(deg)
| Translational
---
Mean(m)
| Rotational
---
Mean(deg)
| Translational
---
Median(m)
| Rotational
---
Median(deg)
| Translational
---
SD(m)
| Rotational
---
SD(deg)
| Translational
---
Max(m)
| Rotational
---
Max(deg)
Wasserstein | 0.06 | 2.02 | 0.05 | 1.39 | 0.04 | 0.91 | 0.03 | 1.46 | 0.22 | 9.67
SLK | 0.07 | 2.32 | 0.06 | 1.58 | 0.05 | 0.97 | 0.04 | 1.70 | 0.20 | 11.9
KL.D | 0.05 | 2.03 | 0.04 | 1.34 | 0.04 | 0.81 | 0.02 | 1.52 | 0.16 | 10.86
Mod.L2 | 0.06 | 2.23 | 0.05 | 1.5 | 0.04 | 0.95 | 0.03 | 1.64 | 0.17 | 10.17
SAM | 0.07 | 2.34 | 0.06 | 1.57 | 0.05 | 0.96 | 0.04 | 1.73 | 0.31 | 9.94

TABLE II: Spectral Similarity Function Analysis (RPE)

###  V-A Experimental Simulation Setup

The following sections evaluate the performance of RaSpectLoc using a dataset
consisting of 5 mobile robot trajectories around a University. The dataset
includes Raman Spectra, RGBD images and odometry information. Additionally, a
material-embedded floorplan with Raman spectra at each coordinate is provided.

To quantitatively evaluate the performance of RaSpectLoc and state-of-the-art
methods, the Absolute Trajectory Error (ATE) and the Relative Pose Error (RPE)
metrics presented by Strum et al. [22] are used. The ATE calculates the rigid
transformation estimation between the two paths. The RPE evaluates the
relative rotation and relative translation for all points in the path,
ignoring the effects of drift over time in the trajectory. The RMSE, mean,
median, standard deviation, minimum distance and maximum distance values of
the residual position error are provided for the resulting ATE and RPE of the
experiments. For evaluation in Table I, AMCL, SeDAR [1] and RaSpectLoc is
given a coarse initialisation with standard deviations of 2.0m in (x, y) and
2.0 radians in $\theta$. The system was run with a maximum of 1000 particles
placed around the covariance ellipse. The error was recorded for each new set
of accumulated Raman spectra into the published tuple $\mathbb{Z}_{t}$.

###  V-B Ablation and parameter exploration

Figure 5: ATE:RMSE error against time taken to complete the trajectory

This section focuses on the performance of the RaSpectLoc system using various
spectral similarity functions and optimal weights, $\epsilon$, to accurately
compare against modern localisation approaches. We evaluate the performance
using the ATE and RPE of the produced trajectories with the likelihood-field
model. Additionally, we consider the accuracy of determining the correct
spectrum after adding Raman-probe shot noise. This approach should provide a
more realistic evaluation of the similarity functions compared to post-
processed spectra that do not contain these intensity artefacts or noise. For
Eq. 12 the value of $\beta$ is set to 1. The results in Tables I and II
indicate that the Mod. L2 function produces the best accuracy and performance
for RaSpectLoc. The results of the Mod. L2 function is because it provides a
descriptive way of examining the similarity between peaks with a reward and
punishment scheme. In testing, the Wasserstein distance takes significantly
longer to process than other functions, often leading to performance issues
during periods of high message frequency. This is highlighted in Fig. 5 by the
logarithmic axis for the time taken to complete a 5min 55sec rosbag recording.
One obvious limitation of RaSpectLoc from this graph is the increased run-time
compared to AMCL, although this comes with a significant reduction in error.

Weight configuration tests (ATE)
---
Weights | RMSE | Mean | Median | SD | Min | Max
$\epsilon_{R}$ | $\epsilon_{m}$
0.2 | 0.8 | 0.19 | 0.17 | 0.15 | 0.09 | 0.01 | 0.43
0.4 | 0.6 | 0.19 | 0.17 | 0.16 | 0.09 | 0.04 | 0.38
0.5 | 0.5 | 0.18 | 0.15 | 0.13 | 0.10 | 0.03 | 0.50
0.6 | 0.4 | 0.22 | 0.20 | 0.20 | 0.10 | 0.08 | 0.47
0.8 | 0.2 | 0.21 | 0.19 | 0.19 | 0.11 | 0.08 | 0.45

TABLE III: Weight experiments ablation tests

As shown in Table III, the performance of RaSpectLoc varies with the weight
distribution. The best-performing weights are 0.5 for the material and ranges.

Compared to Table I the results still show that a ‘spectra-only’ approach
outweighs the combined range and spectral data approach seen in Table III.
This result is likely due to geometric cues in a hallway environment. These
are relatively weak compared to the fine-grained spectral data that provide a
greater understanding of the current pose for the system. Findings in these
experiments are highlighted in Section V-C, where we will compare RaSpectLoc’s
performance against other state-of-the-art systems.

###  V-C Quantitative Results

Tables IV and V showcase the results of our tests against various state-of-
the-art localisation approaches in terms of Absolute Trajectory Error (ATE)
and Relative Pose Error (RPE) metrics. Experiments are compared with
$\epsilon_{R}$ and $\epsilon_{m}$ set to 0.5 and 0.5, respectively, as per the
optimal configuration found in Section V-B for our combined localisation model
with the Mod. L2 likelihood function (Section IV-C2). These tests are
performed using the same RGB images and depth scan messages along a single
trajectory. This separation allows us to compare our results against three
non-MCL approaches: ORBSLAM, GMapping and PoseNet.

Average Trajectory Error (m)
---
Approach | RMSE | Mean | Median | SD | Min | Max
AMCL | 0.24 | 0.21 | 0.2 | 0.11 | 0.04 | 0.95
SeDAR | 0.19 | 0.16 | 0.14 | 0.10 | 0.02 | 0.55
ORBSLAM3 (MONO) | 7.17 | 6.54 | 6.81 | 2.93 | 0.55 | 11.57
GMapping | 0.71 | 0.63 | 0.57 | 0.31 | 0.32 | 1.43
Posenet | 4.64 | 2.58 | 1.42 | 3.85 | 0.02 | 25.66
RaSpectLoc (combined) | 0.18 | 0.15 | 0.13 | 0.10 | 0.03 | 0.50
RaSpectLoc (Materials only) | 0.14 | 0.13 | 0.12 | 0.06 | 0.03 | 0.41

TABLE IV: ATE: Baseline localisation comparisons

Relative Pose Error (m/deg)
---
Approach | RMSE | Mean | Median | SD | Max
AMCL | 0.07/2.68 | 0.05/1.68 | 0.04/0.68 | 0.05/2.09 | 0.37/11.89
SeDAR | 0.06/2.21 | 0.04/1.5 | 0.04/0.9 | 0.03/1.72 | 0.2/12.47
ORBSLAM3 (MONO) | 0.42/14.7 | 0.41/5.78 | 0.41/0.86 | 0.10/13.58 | 0.77/61.60
GMapping | 0.40/26.65 | 0.18/13.05 | 0.31/8.5 | 0.35/23.23 | 1.49/92.65
Posenet | 4.72/47.70 | 2.23/28.40 | 0.89/9.8 | 4.16/38.33 | 28.96/179.6
RaSpectLoc (combined) | 0.06/2.31 | 0.06/1.52 | 0.05/0.94 | 0.03/1.74 | 0.21/9.77
RaSpectLoc (Materials) | 0.05/2.03 | 0.04/1.34 | 0.04/0.81 | 0.02/1.52 | 0.16/10.86

TABLE V: RPE: Baseline localisation comparisons

Our experiments revealed that RaSpectLoc consistently outperforms the other
approaches in accuracy, robustness, ATE and RPE metrics. The results showed
that our method was able to accurately estimate the 3-DoF pose of the robot
even in challenging environments with noise and dynamic obstacles.
Furthermore, our experiments highlighted the ability of RaSpectLoc to scale to
larger environments and handle noise-riddled Raman Spectra, making it a highly
competitive solution for real-world applications.

##  VI Conclusion

In conclusion, this paper has presented a novel approach for mobile robot
localisation that harnesses the power of Raman spectra. With its ability to
recognise subtle material-composition landmarks, RaSpectLoc can outperform
traditional RGBD systems. This recognition has significant implications for
localisation in challenging environments, evident from our practical
experimental section (V-C). The future of this field is even more promising,
as we envision a future where robots can exploit spectral data to navigate and
map hazardous environments, such as nuclear decommissioning sites. Such a SLAM
system would provide a more detailed understanding of the environment while
reducing the risk posed to human workers. Raman spectra could also assist in
detecting anomalies in these high-risk locations with the potential to
revolutionize the field and greatly benefit society.

##  VII Acknowledgements

This work was partially supported by Industrial 3D Robotics (I3D), IS-
Instruments and partially funded by the EPSRC under grant agreement
EP/S035761/1.

## References

  * [1] Mendez, O., Hadfield, S., Pugeault, N. and Bowden, R. (2020) “SeDAR: reading floorplans like a human—using deep learning to enable human-inspired localisation”. International Journal of Computer Vision, 128(5), pp. 1286-1310. (doi: 10.1007/s11263-019-01239-4)
  * [2] D. Talwar and S. Jung, “Particle Filter-based Localization of a Mobile Robot by Using a Single Lidar Sensor under SLAM in ROS Environment,” 2019 19th International Conference on Control, Automation and Systems (ICCAS), 2019, pp. 1112-1115, doi: 10.23919/ICCAS47443.2019.8971555.
  * [3] Thale, Sumegh and Prabhu, Mihir and Thakur, Pranjali and Kadam, Pratik. (2020). “ROS-based SLAM implementation for Autonomous navigation using Turtlebot”. ITM Web of Conferences. 32. 01011. 10.1051/itmconf/20203201011.
  * [4] Stanford Artificial Intelligence Laboratory et al., “Robotic Operating System”, Available: https://www.ros.org. [Accessed: Jul. 4, 2023].
  * [5] S. Z. Ahmed, V. B. Saputra, S. Verma, K. Zhang and A. H. Adiwahono, “Sparse-3D Lidar Outdoor Map-Based Autonomous Vehicle Localization,” 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 1614-1619, doi: 10.1109/IROS40897.2019.8967596.
  * [6] A. Cramariuc et al., “SemSegMap – 3D Segment-based Semantic Localization,” 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 1183-1190, doi: 10.1109/IROS51168.2021.9636156.
  * [7] Dube, Renaud & Cramariuc, Andrei & Dugas, Daniel & Nieto, Juan & Siegwart, Roland & Cadena, Cesar. (2018). “SegMap: 3D Segment Mapping using Data-Driven Descriptors.” 10.15607/RSS.2018.XIV.003.
  * [8] Buchanan, R., Bednarek, J., Camurri, M. et al. Navigating by touch: haptic Monte Carlo localization via geometric sensing and terrain classification. Auton Robot 45, 843–857 (2021). https://doi.org/10.1007/s10514-021-10013-w
  * [9] Serrano, Oscar & Cañas Plaza, José & Matellán, Vicente & Rodero, Luis. (2014). Robot localization using WiFi signal without intensity map.
  * [10] W. Guan, S. Chen, S. Wen, Z. Tan, H. Song and W. Hou, “High-Accuracy Robot Indoor Localization Scheme Based on Robot Operating System Using Visible Light Positioning” in IEEE Photonics Journal, vol. 12, no. 2, pp. 1-16, April 2020, doi: 10.1109/JPHOT.2020.2981485.
  * [11] Conor C. Horgan, Magnus Jensen, Anika Nagelkerke, Jean-PhillipeSt-Pierre, Tom Vercauteren, Molly M. Stevens, and Mads S. Bergholt, “High-throughput molecular imaging via deep learning enabled Raman spectroscopy”, Analytical Chemistry2021, 93, 48, 15850-15860.
  * [12] Gerasimov, V. A.,Gurovich, A. M., Kostrin, D. K., Selivanov, L. M., Simon, V. A., Stuchenkov, A. B.,Paltcev, A. V., and Uhov, A. A. (2016). Raman spectroscopy for identification of woodspecies. Journal of Physics: Conference Series, 741, 012131. https://doi.org/10.1088/1742-6596/741/1/012131.
  * [13] Foster, M, Wharton, M, Brooks, W, Goundry, M, Warren, C, Storey, J. Remote sensing of chemical agents within nuclear facilities using Raman spectroscopy. J Raman Spectrosc. 2020; 51: 2543– 2551. https://doi.org/10.1002/jrs.6016
  * [14] Félix Lussier, Vincent Thibault, Benjamin Charron, Gregory Q. Wallace, Jean-Francois Masson, Deep learning and artificial intelligence methods for Raman and surface-enhanced Raman scattering, TrAC Trends in Analytical Chemistry, Volume 124, 2020,115796, ISSN 0165-9936, https://doi.org/10.1016/j.trac.2019.115796.
  * [15] Shehroz S. Khan, Michael G. Madden, New similarity metrics for Raman spectroscopy, Chemometrics and Intelligent Laboratory Systems, https://doi.org/10.1016/j.chemolab.2012.03.007.
  * [16] R. W. M. Hendrikx, P. Pauwels, E. Torta, H. P. J. Bruyninckx and M. J. G. van de Molengraft, “Connecting Semantic Building Information Models and Robotics: An application to 2D LiDAR-based localization,” 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 11654-11660, doi: 10.1109/ICRA48506.2021.9561129.
  * [17] Zhao, X.Y. & Liu, G.Y. & Sui, Y.T. & Xu, M. & Tong, L.. (2020). Denoising method for Raman spectra with low signal-to-noise ratio based on feature extraction. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy. 250. 119374. 10.1016/j.saa.2020.119374.
  * [18] J. Conroy, A.G. Ryder, M.N. Leger, K. Hennessy, M.G. Madden, “Qualitative and quantitative analysis of chlorinated solvents using Raman spectroscopy and machine learning”, Proc. of SPIE, the International Society for Optical Engineering, 2005, pp. 131–142.
  * [19] R. Valencia, J. Saarinen, H. Andreasson, J. Vallvé, J. Andrade-Cetto and A. J. Lilienthal, “Localization in highly dynamic environments using dual-timescale NDT-MCL”, Proc. ICRA, pp. 3956-3962, May 2014.
  * [20] Maken FA, Ramos F, Ott L. Bayesian iterative closest point for mobile robot localization. The International Journal of Robotics Research. 2022;41(9-10):851-874. doi:10.1177/02783649221101417.
  * [21] Jahn, I.J.; Grjasnow, A.; John, H.; Weber, K.; Popp, J.; Hauswald, W. Noise Sources and Requirements for Confocal Raman Spectrometers in Biosensor Applications. Sensors 2021, 21, 5067. https://doi.org/10.3390/s21155067
  * [22] Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D. (2012). A benchmark for the evaluation of RGB-D SLAM systems. In International conference on intelligent robots and systems (IROS) (pp. 573–580). IEEE/RSJ. https://doi.org/10.1109/IROS. 2012.6385773.
  * [23] Dellaert, F., Fox, D., Burgard, W., & Thrun, S. (1999). Monte Carlo localization for mobile robots.In International conference on robotics and automation (ICRA) (pp. 1322–1328). IEEE, Detroit.
  * [24] Campos, C., Elvira, R., Rodríguez, J. J., Montiel, J. M., & Tardós, J. D. (2020). ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM. arXiv. https://doi.org/10.1109/TRO.2021.3075644
  * [25] Kendall, A., Grimes, M. & Cipolla, R. (2015). PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization.. ICCV (p./pp. 2938-2946), : IEEE Computer Society. ISBN: 978-1-4673-8391-2
  * [26] Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard: Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters, IEEE Transactions on Robotics, Volume 23, pages 34-46, 2007
  * [27] P. J. Larkin, “Infrared and Raman Spectroscopy: Principles and Spectral Interpretation,” Oxford: Elsevier, 2011.
  * [28] I. R. Lewis and H. G. M. Edwards (eds.), “Handbook of Raman Spectroscopy,” Bristol: Taylor & Francis, 2001.
  * [29] J. Gao, Y. Hu, and Y. Zhao, “A novel approach for raman spectra comparison: Wasserstein distance with local constraints,” Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, vol. 239, 2020, 118448.
  * [30] Liu, J., Osadchy, M., Ashton, L., Foster, M., Solomon, C.J. and Gibson, S.J. (2017). Deep convolutional neural networks for Raman spectrum recognition: a unified solution. The Analyst, 142(21), pp.4067–4074. doi:https://doi.org/10.1039/c7an01371j.
  * [31] Marsh-Collings, S. (n.d.). Raman Probes. [online] IS-Instruments. Available at: https://is-instruments.com/raman-probe/ [Accessed Jul. 4, 2023].
  * [32] M. Łysakowski, M. R. Nowicki, R. Buchanan, M. Camurri, M. Fallon and K. Walas, “Unsupervised Learning of Terrain Representations for Haptic Monte Carlo Localization,” 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA, 2022, pp. 4642-4648, doi: 10.1109/ICRA46639.2022.9812296.