#  HotCold Block: Fooling Thermal Infrared Detectors with
a Novel Wearable Design

Hui Wei1,2, Zhixiang Wang3,4,5, Xuemei Jia1,2,
Yinqiang Zheng3, Hao Tang6, Shin’ichi Satoh5,3, Zheng Wang1,2 Corresponding
Author

###### Abstract

Adversarial attacks on thermal infrared imaging expose the risk of related
applications. Estimating the security of these systems is essential for safely
deploying them in the real world. In many cases, realizing the attacks in the
physical space requires elaborate special perturbations. These solutions are
often _impractical_ and _attention-grabbing_. To address the need for a
physically practical and stealthy adversarial attack, we introduce HotCold
Block, a novel physical attack for infrared detectors that hide persons
utilizing the wearable Warming Paste and Cooling Paste. By attaching these
readily available temperature-controlled materials to the body, HotCold Block
evades human eyes efficiently. Moreover, unlike existing methods that build
adversarial patches with complex texture and structure features, HotCold Block
utilizes an SSP-oriented adversarial optimization algorithm that enables
attacks with pure color blocks and explores the influence of size, shape, and
position on attack performance. Extensive experimental results in both digital
and physical environments demonstrate the performance of our proposed HotCold
Block. _Code is available: https://github.com/weihui1308/HOTCOLDBlock_.

## Introduction

Deep Neural Networks (DNNs) have achieved great success in various fields.
They work not only well in visible light but also in thermal infrared imaging,
_e.g._ , thermal infrared detection systems that are widely used in autonomous
driving, night surveillance, temperature measurement, _etc_. However,
adversarial attacks in both _digital_ and _physical_ worlds expose the
vulnerability of DNNs, raising concerns about the security of related
applications. The security of DNNs has attracted significant attention in
_visible_ light (Xu et al. 2021; Duan et al. 2021; Cai et al. 2022; Wang et
al. 2022; Hu et al. 2022; Zhong et al. 2022) but has not been fully explored
in thermal _infrared_ imaging.

Figure 1:  Different infrared attack methods. Our HotCold Block is effective
and stealthy. It achieves competitive attack performance on YOLOv5 (Jocher
2020) while evades human eyes better.  Figure 2:  Distinguishing the proposed
method with existing methods. Unlike prior methods, the proposed HotCold Block
optimizes the size, shape, and position of patches that are pervasively
overlooked.

This paper focuses on the _physical_ adversarial attack on _infrared_
detectors (for brevity, we refer to “thermal infrared” as “infrared”
throughout the paper), which _hides_ persons from smart infrared cameras.
Different from adversarial attack in the _digital_ world that directly injects
adversarial perturbations to captured images, adversarial attack in the
_physical_ world requires physically realizable objects—defined as adversarial
medium—to compose adversarial perturbations. Recently, Zhu et al. (2021, 2022)
successively propose to perform attacks on infrared detectors with patches
that consists of small light bulbs and invisible clothing made of aerogel.
While they achieve reasonable attack effectiveness, their adversarial mediums
are attention-grabbing and look unnatural to the human, making the attack
suspicious.

To address the aforementioned problems, we introduce new adversarial
mediums—Warming Paste and Cooling Paste—based on our findings. First, such
temperature-controlled materials can affect infrared imaging. They appear as
pure color blocks under the infrared camera and interfere with detectors
consequently. Second, they are wearable, making the attacks quite stealthy.
Figure 1 gives the comparison samples. Third, they are physically practical
for performing the attacks, only need to paste them on the body. And finally,
the Warming Paste and Cooling Paste are readily available without complicated
hand-crafting. The cost to execute an attack with them is less than $1.
Consequently, they are ideal for attacking infrared detectors, and we pose a
new research question: _how to design effective patterns with these new
adversarial mediums_?

This research problem is challenging since images of these materials are
simple under infrared cameras and can hardly construct complex patterns.
Existing physical adversarial attacks (Thys, Van Ranst, and Goedemé 2019; Liu
et al. 2020; Tan et al. 2021; Hu et al. 2021) mainly use patch-based methods,
which replace a localized region of the targeted image with an elaborate
patch. In this regard, researchers delve into the structure and texture
features of the patch. They aim to generate particular patterns that can fool
the DNNs. Intuitively, the attack effectiveness would suffer from the simple
structure and texture of the patch.

Considering this limitation and the imaging characteristics of the Warming
Paste and Cooling Paste, we propose HotCold Block, which exploits pure color
blocks to achieve physically practical and stealthy adversarial attacks under
infrared cameras (Figure 2). To improve the performances in attack, we analyze
the 5 attributes of patches: shape, size, position, structure, and texture,
and design SSP-oriented adversarial optimization, which optimize the patch’s
size, shape, and position on the target human body simultaneously. To the best
of our knowledge, we are the first to reveal how the lower-level
attributes—size, shape, and position—compared to the high-level
counterparts—texture and structure—affect the attack performance. Extensive
experimental results on the digital and physical world demonstrate that
HotCold Block can effectively attack infrared detectors while ensuring quite
stealthy.

Our main contributions are summarized below:

  * •

We propose a new stealthy adversarial attack via the wearable temperature-
sensitive materials Warming Paste and Cooling Paste, called HotCold Block,
which is physically practical.

  * •

We develop an SSP-oriented adversarial optimization that considers three
lower-level features of patches simultaneously, _i.e._ , size, shape, and
position, instead of setting them manually like most prior works.

  * •

We evaluate our method on mainstream detectors. Extensive experiments in
digital and physical space show that our HotCold Block achieves competitive
performance on effectiveness, stealthiness, and robustness.

## Related Work

#### Patch-based adversarial attacks.

The patch-based adversarial attack is defined as an attack that is able to
fool DNNs with elaborate patches and has been frequently applied to physical
attacks (Liu et al. 2019a, 2020; Zolfi et al. 2021). Generally, this type of
method replaces a localized region of the threat image with a patch,
regardless of perturbation constraint (Brown et al. 2017; Liu et al. 2019b;
Zhu et al. 2021). In recent years, researchers aimed to trade off the
effectiveness and stealthiness of the patch. For instance, Thys, Van Ranst,
and Goedemé (2019) managed to generate smoother textures with the total
variation loss. After that, Wu et al. (2020) and Xu et al. (2020) printed
patches on the clothing to evade human eyes, _e.g._ , adversarial T-shirt and
invisibility cloak. Some recent papers generated cartoon-like patches that
look more natural (Tan et al. 2021; Hu et al. 2021). To sum up, previous
methods mainly focus on designing special structures and textures for
adversarial patches while overlooking the more general attributes: size,
shape, and position. In this paper, we explore the influence of those three
attributes on attack effectiveness.

#### Attacks to thermal infrared imaging.

Unlike the extensive research work on adversarial attacks in visible light
images (Wei et al. 2022), to the best of our knowledge, only two publications
focus on the safety of thermal infrared imaging. Zhu et al. (2021) proposed a
patch-based adversarial attack, which uses small glowing light bulbs to
manufacture special infrared patterns. The following year, Zhu et al. (2022)
designed infrared invisible clothing based on a new material aerogel that
successfully evades person detectors. Obviously, whether small glowing light
bulbs or clothes made of aerogel material, they have a common shortcoming:
they are attention-grabbing when performing attacks. This is contrary to the
mission of adversarial attacks. Unlike those works, we propose a physically
practical and stealthy adversarial attack called HotCold Block. The
adversarial mediums we use are the Warming Paste and Cooling Paste, wearable
and readily available temperature-controlled materials.

## Method

This section presents our method, _i.e._ , HotCold Block. We first introduce
the modeling of the adversarial medium and then describe the SSP-oriented
adversarial optimization.

### Problem Definition

Given an input image $I$ and the distribution of all original images
$\mathcal{D}$, each $I{\in}\mathcal{D}$ contains one or multiple person
instances. The pre-trained person detector $f:I{\to}\mathcal{Y}$ can predict
labels $\hat{\mathcal{Y}}$ matching the true labels $\mathcal{Y}$ that
includes position of the bounding boxes ${{\mathcal{V}}_{pos}}$, the object
probability ${{\mathcal{V}}_{obj}}$ and the class score
${{\mathcal{V}}_{cls}}$:

$\hat{\mathcal{Y}}:=[{{\mathcal{V}}_{pos}},{{\mathcal{V}}_{obj}},{{\mathcal{V}}_{cls}}]=f(I).$
(1)

Our goal is to fool the person detector so that it cannot identify the person,
_i.e._ , ${{\mathcal{V}}_{obj}}{=}0$. In this paper, we use a patch-based
attack method, which replaces localized regions of the original image with
patches. We denote the threat image as ${I}_{adv}$. The goal can be described
as follows:

$\arg\min{{\mathcal{V}}_{obj}}=\arg\underset{i}{\mathop{\min}}\,{{f}}({{I}_{adv}}),$
(2)

where $i$ is the index of the $i$-th image in $\mathcal{D}$.

###  HotCold Block Modeling

HotCold Block aims to fool the infrared detector using the Warming Paste and
Cooling Paste, essentially a patch-based adversarial attack. A patch generally
has five attributes: size, shape, position, structure, and texture. In this
paper, we delve into the size, shape, and position (“SSP” for short) of
patches, with the two primary considerations: _(i)_ in the previous work, the
lower-level features SSP of patches are pervasively set manually. Their
influence on attacks has not been fully explored compared to the high level of
features—texture, and structure. _(ii)_ since the Warming Paste and Cooling
Paste are imaged as pure color blocks under the infrared camera with simple
structure and texture, studying the SSP of color blocks would be more
meaningful.

Figure 3: Example of nine-square-grid states for shape modeling. We use the
optimization algorithm to find the optimal shape that minimizes the object
probability ${{\mathcal{V}}_{obj}}$. As shown, the available options are
diverse and flexible.

In practice, allowing the optimization algorithm to fit an arbitrary shape is
unreasonable. One is because some shapes are not physically achievable with
our adversarial medium, and the other is due to the high complexity of area
calculation for some shapes. These problems are especially prominent for
irregular concave polygons. Thus, we use a nine-square-grid to model our
adversarial medium in the digital space, as shown in Figure 3. Consequently,
not only are the aforementioned problems tackled, but we can simulate the SSP
of patches reasonably and conveniently.

Figure 4:  The hardware used for physical attack. (a) our image acquisition
device. (b) the Warming Paste with their images in RGB-Infrared space. (c) the
Cooling Paste with their images in RGB-Infrared space.

#### Size.

Here, we define the size as the area of patches. Since HotCold Block uses more
than one patch to attack the targeted infrared detector, the size depends on
the number $m$ of patches, the number $n$ of the occupied grids, and the side
length $l$ of the nine-square-grid. Note that the small size facilitates
stealthiness. To minimize patch size, we design a mechanism to trade off the
size and attack effectiveness. Concretely, we take the growth part of the size
as a penalty term, represented as

${\mathcal{L}_{obj}}={{\mathcal{V}}_{obj}}+\underbrace{\lambda{{\Delta}_{\uparrow}}{{(\frac{n(m\times{{l}^{2}})}{9})}}}_{\text{
penalty term}}\,,$ (3)

where $\lambda$ is a hyperparameter to prevent ${{\mathcal{V}}_{obj}}$ from
overwhelming the penalty term. The sign ${{\Delta}_{\uparrow}}$ represents
positive growth, and we set the penalty term to $0$ when it decreases.

#### Shape.

For the shape of patches, we determine it with a $3{\times}3$ matrix
${\mathcal{M}}$. The 0-1 value of the matrix controls the state of each grid
in the nine-square-grid, _e.g._ , ${\mathcal{M}}{=}[[0,1,0],[1,1,1],[0,1,0]]$
represents the state shown in Figure 3 ($n{=}5$). As shown, we can exploit
flexible and complex combinations to obtain a plethora of patch shapes. The
optimization algorithm in our method is dedicated to finding the optimal shape
to achieve a higher attack success rate.

#### Position.

To improve the attack effectiveness, HotCold Block employs a multi-patch joint
attack strategy. Our adversarial mediums, the Warming Paste and Cooling Paste,
are suitable for this, and it is realizable and natural to achieve in the
physical space. Here, we find an appropriate set of coordinates
$\mathcal{P}{=}\\{({{x}_{1}},{{y}_{1}}),({{x}_{2}},{{y}_{2}}),...,({{x}_{m}},{{y}_{m}})\\}$
to determine the position of the top-left vertex of each patch, where $m$ is
the number of patches. The corresponding vertices coordinate $\mathcal{P}$ is
served as the parameter for optimization.

Algorithm 1 SSP-oriented Adversarial Optimization

Input: Dataset ${\mathcal{D}}$, Detector $f$.
Parameter: A vector of parameter set
${\mathcal{O}}=\\{{\mathcal{M}},{\mathcal{P}}\\}$
Output: ${\mathcal{M}}$, ${\mathcal{P}}$

1: Let $t=0$.

2: Initialization: Randomly set ${\mathcal{M}}$, ${\mathcal{P}}$

3: InitializeSwarm(particles)

4: for $i\leftarrow 0$ to $epoch$ do

5: while $I=iterator({\mathcal{D}})$ is not Null do

6: ${{I}_{adv}}\leftarrow apply(I,{\mathcal{M}},{\mathcal{P}})$

7: ${\mathcal{L}_{obj}}=f({{I}_{adv}})$

8: ${{\mathcal{M}}^{{}^{\prime}}},{{\mathcal{P}}^{{}^{\prime}}}\leftarrow$
particles.$move({{\vec{x}}_{i}},{{\vec{v}}_{i}})$

9: $I_{adv}^{{}^{\prime}}\leftarrow
apply(I,{{\mathcal{M}}^{{}^{\prime}}},{{\mathcal{P}}^{{}^{\prime}}})$

10: ${\mathcal{L}_{obj}^{{}^{\prime}}}=f({I_{adv}^{{}^{\prime}}})$

11: if ${\mathcal{L}_{obj}^{{}^{\prime}}}<{\mathcal{L}_{obj}}$ then

12: ${\mathcal{M}}\leftarrow{{\mathcal{M}}^{{}^{\prime}}}$,
${\mathcal{P}}\leftarrow{{\mathcal{P}}^{{}^{\prime}}}$

13: particles.$update$(PersonalBest: ${{\vec{pb}}_{i}}$)

14: swarm.$update$(GlobalBest: ${{\vec{gb}}_{i}}$)

15: else

16: pass

17: end if

18: end while

19: end for

20: return GlobalBest: $\\{{\mathcal{M}},{\mathcal{P}}\\}$

### SSP-oriented Adversarial Optimization

In Figure 4, we display the adversarial mediums and their imaging under the
infrared camera. Based on the aforementioned modeling, we develop an SSP-
oriented adversarial optimization algorithm for performing successful attacks,
with the core objectives of 1) minimizing patch Size, 2) finding the optimal
Shape, 3) learning suitable Positions.

Before describing the optimization algorithm, we analyze the optimization
objective and optimization parameters. The focus of the optimization objective
is to minimize ${\mathcal{L}_{obj}}$, aiming to result in person invisibility
by detector $f$. The optimized features are the patch’s size, shape, and
position. Formally, we list the optimization parameter set
${\mathcal{O}}{=}\\{{\mathcal{M}},{\mathcal{P}}\\}$.

Since calculating a backward gradient on all operations of HotCold Block is
challenging and the parameter values in ${\mathcal{O}}$ are discrete, solving
this problem with the popular gradient descent optimization algorithm (Ruder
2016) is not appropriate. Inspired by Zhong et al. (2022), we exploit the
particle swarm optimization (PSO) strategy (Poli, Kennedy, and Blackwell
2007), which is a bio-inspired algorithm and does not use the gradient of the
problem being optimized.

Based on the PSO, we design the SSP-oriented adversarial optimization.
Specifically, a number of simple entities, the particles, are placed in the
search space of our problem. Each individual in the particle swarm is composed
of ${\mathcal{O}}{=}\\{{\mathcal{M}},{\mathcal{P}}\\}$. On each iteration, all
the particles adjust their velocities ${{\vec{v}}_{i}}$ and positions
${{\vec{x}}_{i}}$. If one position is better than any that has been found so
far, then the value is stored as the globe best position ${{\vec{gb}}_{i}}$ of
the swarm. Meanwhile, the individual particle has its own personal best
position ${{\vec{pb}}_{i}}$. The pseudocode of SSP-oriented Adversarial
Optimization is shown in Algorithm 1. We start with a number of random points.
All the particles move in the direction of decreasing ${\mathcal{L}_{obj}}$.
Each movement of particles is influenced by the ${{\vec{pb}}_{i}}$ but is also
guided toward the ${{\vec{gb}}_{i}}$, which is found by the entire swarm of
particles. In our optimization, the number of parameters is few, reduced by
4000 times, compared to updating the patch’s structure and texture (a
$300\times 300$ patch that has $9\times{10}^{4}$ update pixels).

## Experiments

In this section, we carefully evaluate the performance of our HotCold Block on
the three criteria: _effectiveness_ , _stealthiness_ , and _robustness_.

Num of Patches $m$ | Method | Side length $l$ (%)
---|---|---
6 | 8 | 10 | 12 | 14 | 16 | Average
AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$
1 | R | 94.1 |  1.3 | 94.4 |  0.7 | 93.4 |  0.0 | 93.3 |  2.2 | 93.0 |  2.9 | 91.1 |  7.0 | 93.2 |  2.4
MR | 94.7 |  0.0 | 94.2 |  4.0 | 94.3 |  1.8 | 93.7 |  3.5 | 94.3 |  2.2 | 93.7 |  2.6 | 94.2 |  2.4
HCB | 93.4 |  1.8 | 93.2 |  2.9 | 90.7 |  3.7 | 88.0 |  8.4 | 85.6 |  9.0 | 80.9 | 14.7 | 88.6 |  6.8
2 | R | 93.4 |  3.7 | 92.9 |  3.7 | 92.6 |  1.7 | 91.0 |  5.0 | 87.4 | 12.7 | 81.8 | 18.5 | 89.9 |  7.6
MR | 93.9 |  2.6 | 93.2 |  2.8 | 92.5 |  2.2 | 92.2 |  5.3 | 90.5 |  6.4 | 75.0 | 24.9 | 89.6 |  7.4
HCB | 92.0 |  5.9 | 87.4 |  7.5 | 81.0 | 21.5 | 70.3 | 26.8 | 65.3 | 29.2 | 54.8 | 40.9 | 75.1 | 22.0
3 | R | 93.3 |  0.0 | 92.2 |  1.3 | 91.2 |  5.5 | 86.5 | 14.9 | 82.5 | 15.4 | 76.6 | 21.3 | 87.1 |  9.7
MR | 94.0 |  3.1 | 93.3 |  2.2 | 86.4 | 13.0 | 79.0 | 19.4 | 70.0 | 33.0 | 58.8 | 35.8 | 80.3 | 17.8
HCB | 89.4 |  6.6 | 86.0 | 10.8 | 70.4 | 28.6 | 58.7 | 33.8 | 52.0 | 37.8 | 38.3 | 49.2 | 65.8 | 27.8
4 | R | 91.5 |  3.1 | 89.7 |  4.6 | 87.6 | 11.6 | 83.7 | 17.8 | 73.8 | 25.9 | 66.1 | 35.9 | 82.1 | 16.5
MR | 93.5 |  3.5 | 92.3 |  4.8 | 81.8 | 19.3 | 74.7 | 23.9 | 65.8 | 37.2 | 49.0 | 38.7 | 76.2 | 21.2
HCB | 81.6 | 13.2 | 69.5 | 25.0 | 66.6 | 31.9 | 43.0 | 40.4 | 26.6 | 59.3 | 29.3 | 59.8 | 52.8 | 38.3
5 | R | 92.1 |  1.1 | 87.6 | 10.1 | 81.7 | 19.3 | 72.5 | 26.1 | 63.7 | 36.1 | 47.2 | 46.8 | 74.1 | 23.3
MR | 91.8 |  2.9 | 86.8 |  9.9 | 64.9 | 31.4 | 48.3 | 47.9 | 28.2 | 69.4 | 14.1 | 77.8 | 55.7 | 40.0
HCB | 83.5 | 13.6 | 58.4 | 38.3 | 49.8 | 47.2 | 34.6 | 58.7 | 22.9 | 68.8 | 11.6 | 77.8 | 43.5 | 50.7
6 | R | 90.7 |  7.5 | 87.6 | 10.1 | 78.4 | 19.1 | 64.0 | 33.0 | 62.0 | 39.3 | 31.0 | 59.9 | 69.0 | 28.2
MR | 90.9 |  7.9 | 84.9 | 13.4 | 76.8 | 19.8 | 44.8 | 50.1 | 25.9 | 67.3 |  8.9 | 82.6 | 55.4 | 40.2
HCB | 75.3 | 22.9 | 59.7 | 39.3 | 40.1 | 57.4 | 35.7 | 60.9 | 11.0 | 82.2 | 10.2 | 83.5 | 38.7 | 57.7
7 | R | 88.2 |  8.8 | 81.9 | 15.8 | 68.4 | 29.7 | 59.4 | 37.9 | 48.4 | 45.1 | 25.2 | 64.6 | 61.9 | 33.7
MR | 88.6 | 13.6 | 83.0 | 14.3 | 69.4 | 25.7 | 28.5 | 64.2 | 14.5 | 73.0 |  5.7 | 88.4 | 48.3 | 46.5
HCB | 71.9 | 26.2 | 57.4 | 34.5 | 36.1 | 56.9 | 24.0 | 66.8 | 24.8 | 71.0 |  5.9 | 89.9 | 36.7 | 57.6

Table 1: Quantitative results on the FLIR ADAS test set at varying setups. We
report AP (%), ASR (%) for our adversarial attack method HotCold Block (HCB)
_vs._ the random block attack (R) and the manual-random block attack (MR),
under varying numbers $m$ of patches and side lengths $l$ (% of the person’s
height).

### Experimental Settings

#### Datasets.

1) Digital adversarial attack. Following Zhu et al. (2021), we evaluate the
performance of our method on the Teledyne FLIR ADAS Thermal dataset (FLIR
2022b)111Note that since the Teledyne FLIR company released an updated version
in January 2022, we use v2.0 instead of v1.0. The updated dataset not only
expands labels to 15 categories _vs._ 5 original categories, but also the
scale of the annotated image with an +83% increase compared to the v1.0
release.. Infrared images were acquired with a Teledyne FLIR Tau2 (13 mm f/1.0
with a 45-degree HFOV and 37-degree VFOV). The thermal camera operated in
T-linear mode. We filter the original dataset for better fitting to the patch-
based adversarial attack, with two conditions of _(i)_ the images contain
“person” category, _(ii)_ the bodies of persons in the images have a height of
more than 120 pixels. Finally, 1,255 images are available, of which 878 are
the training set with 1,366 eligible “person” labels and 377 are the testing
set with 598 eligible “person” labels. 2) Physical adversarial attack. In the
physical space, we capture infrared images with a FLIR ONE Pro camera (FLIR
2022a), which has a thermal resolution of $160{\times}120$. During capture,
the camera is connected to a Xiaomi phone for real-time image display (see
Figure 4(a)). We build lab setups that allow for shooting distances from 0 to
4 meters, record 8 videos in different scenes, and then extract one frame per
second. We capture a total of 112 images and use LabelImg222LabelImg:
https://github.com/heartexlabs/labelImg to annotate them with 224 labels.

#### Evaluation metrics.

We aim to hide the person from detectors. To this end, we adopt the Average
Precision (AP) metric to evaluate the performance of detectors on the threat
dataset. Note that lower AP indicates stronger attack effect. In addition, the
Attack Success Rate (ASR) is used to evaluate the effectiveness of our attack
methods, which is defined as the percentage of positive and total samples as
follows:

$\text{ASR}(X)=1-\frac{1}{N}\sum\limits_{i=0}^{N}{\text{sign}(labe{{l}_{i}})},$
(4)

$\displaystyle\text{sign}(labe{{l}_{i}})=\left\\{\begin{matrix}1,&labe{{l}_{i}}\in{{L}_{pre}}\\\
0,&\text{otherwise.}\\\ \end{matrix}\right.$ (5)

where $N$ is the number of all true positive labels detected in the dataset
$X$ when there is no attack, ${{L}_{pre}}$ is the set of all labels detected
under attacking. The higher the ASR, the more effective the adversarial attack
method is.

#### Competing methods.

We compare our method with the only 2 methods in the field of infrared attack:

  * •

Bulb Attack (Zhu et al. 2021): a physical attack method that fools infrared
pedestrian detectors using small bulbs.

  * •

QR Attack (Zhu et al. 2022): a multi-angle physical attack method that designs
the adversarial “QR code” pattern for attacking infrared detectors.

#### Implementation details.

We use YOLOv5 (Jocher 2020) as our target model since it is a fast, effective,
and widely-used detector. For infrared detection, we use the pre-trained
weights on the MSCOCO Dataset (Lin et al. 2014) as initialization and fine-
tune the model on the FLIR ADAS Dataset. The AP score of the fine-tuned target
model achieves $94.8\%$ on the test set. We set the population size of the
particle swarm to 100 and the block’s pixel value to 0.2. We conduct all
experiments on a device with an NVIDIA GeForce RTX 3090 GPU, and all our codes
are implemented in PyTorch. Our used Warming Paste can keep on heating for 6
hours with an average temperature of 53 $\tccentigrade$, and Cooling Paste can
cool down to 24 $\tccentigrade$ for 4 hours.

Figure 5: Example results of digital attacks. The bounding boxes indicate the
infrared detector successfully detects the person.  Figure 6: Quantitative
results. (a) Precision-Recall curve in the digital space. (b) The AP(%) and
ASR(%) of HotCold Block at different distances in the physical space.

### Evaluation of Effectiveness

#### Digital adversarial attack.

In the digital space, we attack every image in the test set of the FLIR ADAS
dataset with HotCold Block. We run our attack with controlled numbers of
patches and side lengths. Table Experiments reports the effectiveness
evaluation results for our method (HCB) _vs._ the random block attack (R) and
the manual-random block attack (MR). MR refers to random positions, but manual
intervention to avoid overlap between patches for fair comparison. Through the
analysis of all experimental setups and results in Table Experiments, we can
reach the following three conclusions: 1) HotCold Block comprehensively
outperforms random and manual-random, demonstrating that our method is
feasible and effective; 2) The attack effectiveness is strengthening as the
number $m$ of patches and the side length $l$ increase, in keeping with our
expectations; 3) HotCold Block lowers the AP to 43.0% and achieves 40.4% ASR
under $m{=}4$ and $l{=}12$, showing that our method is physically practical
due to the legitimate configuration. Note that our following experiments are
in this configuration unless otherwise specified.

In Figure 5, we show qualitative examples comparing our results with those of
the baseline methods. Moreover, we draw the P-R curve for quantitative
evaluation in Figure 6(a). It is clear that HotCold Block achieves competitive
performance. For example, it causes the AP of YOLOv5 to drop by 51.8%,
significantly outperforming 9.5% and 24.1% achieved by the only two baseline
methods. Note that, although we applied four patches for attacking (as shown
in Figure 5(e)), the total area is similar to the area of 1 patch applied in
the comparison methods.

#### Physical adversarial attack.

Figure 6(b) and Figures 8 depict the quantitative and qualitative results,
respectively. Our attack is effective and achieves an over 90% ASR at 1m and
2m shooting distances. Although only attacking one person in the frame, the AP
drops to 73.6% on average. Observe that with the increasing shooting distance,
the ASR becomes relatively low. By comparison, the shape of the patch suffers
from obvious deformation due to the insufficient infrared camera resolution.
This deformation may degrade the attack effectiveness. Nevertheless, the ASR
is still around 34%. The results show that HotCold Block is physically
practical and evades infrared detectors in the real world. See _Supplementary
Material_ for the video demo.

### Evaluation of Stealthiness

As aforementioned, considering the stealthiness of attacks, we choose the
Warming Paste and Cooling Paste as our adversarial medium. In Figure 7(a), it
is clear that the adversarial mediums are in harmony with society and do not
draw attention to themselves. Moreover, even if the status quo is still
unsatisfactory, our method allows hiding the adversarial medium by wearing
another garment on the outer surface. As shown in Figure 7(b), the human eyes
cannot recognize adversarial mediums on the human body in RGB space. Note that
the change from the “exposed” to the “covered” hardly affects the infrared
camera’s imaging. Compared with the baseline methods, HotCold Block
successfully achieves fairly imperceptible attacks.

Figure 7: Examples of RGB-Infrared image pairs with exposed and covered
adversarial mediums. Figure 8: Example results of physical attacks for HotCold
Block at different distances.

### Evaluation of Robustness

We evaluate the attack robustness of our approach across various detectors
under the black box setting, including YOLOv3 (Redmon and Farhadi 2018), DETR
(Carion et al. 2020), RetinaNet (Lin et al. 2017), Faster RCNN (Ren et al.
2016), and Mask RCNN (He et al. 2017). The detectors are pre-trained on the
MSCOCO Dataset (Lin et al. 2014) and fine-tuned on the FLIR ADAS Dataset.
Table 2 reports ASR and the changes in AP. We can see that the detectors have
a significant degradation in performance when facing HotCold Block, which
makes AP drop by 44.9% and achieves 32.2% in ASR on average. By comparing the
results, we also notice that DETR is minimally affected. One possible reason
is that the transformer-based network used in DETR is beneficial in defending
against adversarial attacks. Broadly, our attack is shown to be robust under
the majority of models.

Detector |  _w/o_ Attack |  _w/_ Attack
---|---|---
AP$\downarrow$ | ASR$\uparrow$ | AP$\downarrow$ | ASR$\uparrow$
YOLOv3 | 95.4 | – | 52.5 | 40.1
YOLOv5 | 94.8 | – | 43.0 | 40.4
DETR | 94.2 | – | 65.1 |  4.5
RetinaNet | 93.6 | – | 57.5 | 22.6
Faster RCNN | 94.5 | – | 31.9 | 49.4
Mask RCNN | 95.7 | – | 48.8 | 36.3
Average | 94.7 | – | 49.8 | 32.2

Table 2: Evaluation across various detectors.

### Parameter and Ablation Studies

#### Effect of $\lambda$.

Larger $\lambda$ have more effect on stealthiness by reducing the size but are
less effective. We evaluate the AP and ASR of our attack with different
$\lambda$. As shown in Figure 9(a), the attack effectiveness of HotCold Block
remains stable when $\lambda$ equals 0 to 3, and drops dramatically when
$\lambda{=}4$. Moreover, We analyze the state of the nine-square-grid and
observe that as $\lambda$ increases, fewer grids are activated, _i.e._ , the
actual area of the patch is smaller. To sum up, by modifying $\lambda$, our
proposed method can trade off the visual stealthiness and effectiveness, and
we set $\lambda{=}3$ to obtain the optimal balance.

#### Effect of the block’s pixel value.

We then show how the block’s pixel value impacts the adversarial effectiveness
of HotCold Block. Figure 9(b) shows the results. Observe that as the pixel
value increases, the attack capability exhibits a concave curve, which means
that taking values close to 0 or 1 favors the attack. This change is because
when the patch and the body are fused, _i.e._ , their pixel values are close,
and the patch no longer has attack capabilities. Note that the values of the
Warming Paste and Cooling Paste under the infrared camera are nearly 0.9 and
0.2, respectively. Therefore, the adversarial mediums we chose are appropriate
for efficient physical attacks.

### Defense Discussion

Attack and defense develop parallel, like an arms race, to improve the model’s
capabilities. Here, we discuss the method for defending against HotCold Block.
Based on our understanding of HotCold Block, we apply the adversarial training
(Bai et al. 2021) that aims to enhance the robustness of models intrinsically.
Concretely, we augment training data with adversarial examples generated by
HotCold Block in each training loop. Then we perform attacks on the retrained
model to verify its effectiveness in the digital space. By comparing the
results, where the AP achieves 94.9% with no attack and 91.6% with attacks,
the retrained model becomes more robust with no performance loss. Therefore,
we can use our image augmentation method to improve the detectors’ performance
further. In this regard, our work has important practical significance for
applying DNNs-based models in the real world.

Figure 9: Parameter and ablation studies of hyperparameter $\lambda$ (a) and
the block’s pixel value (b).

## Limitation

Although HotCold Block shows excellent performance in attacking infrared
detectors, it is hard to implement multi-angle attacks due to the segment-
missing problem (Hu et al. 2022). Concretely, since the blocks produced by the
Warming Paste and Cooling Paste would be obscured when the viewing angle
changes, causing them to disappear from the infrared camera, HotCold Block
would drop in the attack success rate. We believe simulating the 3D virtual
human body and sticking adversarial blocks on the surface can solve this
problem.

## Conclusion

In this paper, we propose a novel physical adversarial attack called HotCold
Block that applies the Warming Paste and Cooling Paste to hide persons from
being detected by infrared detectors. Our wearable adversarial mediums are
physically practical and stealthy due to their intrinsic properties. Moreover,
we design an SSP-oriented adversarial optimization, which delves into the
feature space of size, shape, and position rather than texture and structure.
Extensive experiments in both digital and physical spaces show that our
HotCold Block evade both human eyes and detection models more effectively than
existing methods.

##### Ethics statement.

Our work successfully achieves physical attacks and illustrates the
vulnerability of deep learning models. As the medium of adversarial attack is
readily available and operational, most existing deep learning applications
have potential security threats. Our work is dedicated to providing motivation
and insights for better defense against malicious attacks.

## References

  * Bai et al. (2021) Bai, T.; Luo, J.; Zhao, J.; Wen, B.; and Wang, Q. 2021.  Recent Advances in Adversarial Training for Adversarial Robustness.  In _International Joint Conference on Artificial Intelligence_.  Survey Track.
  * Brown et al. (2017) Brown, T. B.; Mané, D.; Roy, A.; Abadi, M.; and Gilmer, J. 2017.  Adversarial patch.  _arXiv preprint arXiv:1712.09665_.
  * Cai et al. (2022) Cai, Z.; Xie, X.; Li, S.; Yin, M.; Song, C.; Krishnamurthy, S. V.; Roy-Chowdhury, A. K.; and Asif, M. S. 2022.  Context-aware transfer attacks for object detection.  In _AAAI Conference on Artificial Intelligence_.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-End Object Detection with Transformers.  In _European conference on computer vision_.
  * Duan et al. (2021) Duan, R.; Mao, X.; Qin, A. K.; Chen, Y.; Ye, S.; He, Y.; and Yang, Y. 2021.  Adversarial laser beam: Effective physical-world attack to dnns in a blink.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * FLIR (2022a) FLIR. 2022a.  FLIR ONE Pro Camera.  [EB/OL].  https://www.flir.com/products/flir-one-pro/.
  * FLIR (2022b) FLIR. 2022b.  Teledyne FLIR Free ADAS Thermal Datasets v2.  [EB/OL].  https://adas-dataset-v2.flirconservator.com/.
  * He et al. (2017) He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. 2017.  Mask r-cnn.  In _IEEE/CVF International Conference on Computer Vision_.
  * Hu et al. (2021) Hu, Y.-C.-T.; Kung, B.-H.; Tan, D. S.; Chen, J.-C.; Hua, K.-L.; and Cheng, W.-H. 2021.  Naturalistic Physical Adversarial Patch for Object Detectors.  In _IEEE/CVF International Conference on Computer Vision_.
  * Hu et al. (2022) Hu, Z.; Huang, S.; Zhu, X.; Sun, F.; Zhang, B.; and Hu, X. 2022.  Adversarial Texture for Fooling Person Detectors in the Physical World.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Jocher (2020) Jocher, G. 2020.  yolov5.  https://github.com/ultralytics/yolov5.
  * Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017.  Focal loss for dense object detection.  In _IEEE/CVF International Conference on Computer Vision_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_.
  * Liu et al. (2019a) Liu, A.; Liu, X.; Fan, J.; Ma, Y.; Zhang, A.; Xie, H.; and Tao, D. 2019a.  Perceptual-sensitive gan for generating adversarial patches.  In _AAAI conference on artificial intelligence_.
  * Liu et al. (2020) Liu, A.; Wang, J.; Liu, X.; Cao, B.; Zhang, C.; and Yu, H. 2020.  Bias-based universal adversarial patch attack for automatic check-out.  In _European Conference on Computer Vision_.
  * Liu et al. (2019b) Liu, X.; Yang, H.; Liu, Z.; Song, L.; Chen, Y.; and Li, H. 2019b.  DPATCH: An Adversarial Patch Attack on Object Detectors.  In _AAAI Conference on Artificial Intelligence Workshop on Artificial Intelligence Safety_.
  * Poli, Kennedy, and Blackwell (2007) Poli, R.; Kennedy, J.; and Blackwell, T. 2007.  Particle swarm optimization.  _Swarm intelligence_ , 1(1): 33–57.
  * Redmon and Farhadi (2018) Redmon, J.; and Farhadi, A. 2018.  YOLOv3: An Incremental Improvement.  arXiv:1804.02767.
  * Ren et al. (2016) Ren, S.; He, K.; Girshick, R.; and Sun, J. 2016.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(6): 1137–1149.
  * Ruder (2016) Ruder, S. 2016.  An overview of gradient descent optimization algorithms.  _arXiv preprint arXiv:1609.04747_.
  * Tan et al. (2021) Tan, J.; Ji, N.; Xie, H.; and Xiang, X. 2021.  Legitimate Adversarial Patches: Evading Human Eyes and Detection Models in the Physical World.  In _ACM International Conference on Multimedia_.
  * Thys, Van Ranst, and Goedemé (2019) Thys, S.; Van Ranst, W.; and Goedemé, T. 2019.  Fooling automated surveillance cameras: adversarial patches to attack person detection.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_.
  * Wang et al. (2022) Wang, D.; Jiang, T.; Sun, J.; Zhou, W.; Gong, Z.; Zhang, X.; Yao, W.; and Chen, X. 2022.  Fca: Learning a 3d full-coverage vehicle camouflage for multi-view physical adversarial attack.  In _AAAI Conference on Artificial Intelligence_.
  * Wei et al. (2022) Wei, H.; Tang, H.; Jia, X.; Yu, H.; Li, Z.; Wang, Z.; Satoh, S.; and Wang, Z. 2022\.  Physical Adversarial Attack meets Computer Vision: A Decade Survey.  _arXiv preprint arXiv:2209.15179_.
  * Wu et al. (2020) Wu, Z.; Lim, S.-N.; Davis, L. S.; and Goldstein, T. 2020.  Making an invisibility cloak: Real world adversarial attacks on object detectors.  In _European Conference on Computer Vision_.
  * Xu et al. (2020) Xu, K.; Zhang, G.; Liu, S.; Fan, Q.; Sun, M.; Chen, H.; Chen, P.-Y.; Wang, Y.; and Lin, X. 2020.  Adversarial t-shirt! evading person detectors in a physical world.  In _European conference on computer vision_.
  * Xu et al. (2021) Xu, Q.; Tao, G.; Cheng, S.; and Zhang, X. 2021.  Towards feature space adversarial attack by style perturbation.  In _AAAI Conference on Artificial Intelligence_.
  * Zhong et al. (2022) Zhong, Y.; Liu, X.; Zhai, D.; Jiang, J.; and Ji, X. 2022.  Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Zhu et al. (2022) Zhu, X.; Hu, Z.; Huang, S.; Li, J.; and Hu, X. 2022.  Infrared Invisible Clothing: Hiding from Infrared Detectors at Multiple Angles in Real World.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Zhu et al. (2021) Zhu, X.; Li, X.; Li, J.; Wang, Z.; and Hu, X. 2021.  Fooling thermal infrared pedestrian detectors in real world using small bulbs.  In _AAAI Conference on Artificial Intelligence_.
  * Zolfi et al. (2021) Zolfi, A.; Kravchik, M.; Elovici, Y.; and Shabtai, A. 2021.  The Translucent Patch: A Physical and Universal Attack on Object Detectors.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_.


# Advancing EEG/MEG Source Imaging with Geometric-Informed Basis Functions
††thanks: This work was supported by the National Natural Science Foundation
of China (62001205), National Key R&D Program of China (2021YFF1200804),
Shenzhen Science and Technology Innovation Committee (2022410129,
KCXFZ2020122117340001).

Song Wang$1$, Chen Wei$1$, Kexin Lou$1$, Dongfeng Gu$1,2$, and Quanying
Liu$1$∗ $1$Department of Biomedical Engineering, Southern University of
Science and Technology, Shenzhen, China
$2$School of Public Health and Emergency Management, Southern University of
Science and Technology, Shenzhen, China  ∗ Corresponding Author: Quanying Liu
Email<EMAIL_ADDRESS>
###### Abstract

Electroencephalography (EEG) and Magnetoencephalography (MEG) are pivotal in
understanding brain activity but are limited by their poor spatial resolution.
EEG/MEG source imaging (ESI) infers the high-resolution electric field
distribution in the brain based on the low-resolution scalp EEG/MEG
observations. However, the ESI problem is ill-posed, and how to bring
neuroscience priors into ESI method is the key. Here, we present a novel
method which utilizes the Brain Geometric-informed Basis Functions (GBFs) as
priors to enhance EEG/MEG source imaging. Through comprehensive experiments on
both synthetic data and real task EEG data, we demonstrate the superiority of
GBFs over traditional spatial basis functions (e.g., Harmonic and MSP), as
well as existing ESI methods (e.g., dSPM, MNE, sLORETA, eLORETA). GBFs provide
robust ESI results under different noise levels, and result in biologically
interpretable EEG sources. We believe the high-resolution EEG source imaging
from GBFs will greatly advance neuroscience research.

###### Index Terms:

EEG, EEG/MEG source imaging (ESI), Geometry-Informed Basis, Inverse Problem

##  I Introduction

EEG and MEG are non-invasive recordings of electrophysiological signals on the
scalp. Despite their excellent temporal resolution, they suffer from poor
spatial resolution of neural activity [1, 2]. To this end, EEG/MEG source
imaging (ESI) has been developed for inferring the neural sources according to
the scalp recordings, which offers a more detailed understanding of neural
dynamics and brain function [3].

Given the vast number of potential neural sources in the brain and few sensor-
level observations, ESI is an ill-posed inverse problem. Additional
constraints on brain sources are necessary. Traditional ESI methods (e.g.,
MNE, sLORETA/eLORETA and dSPM) add regularizations based on neuroscience prior
knowledge (e.g., spatial/temporal smoothness or sparsity of brain sources).
Rather than using these simple priors, other complex priors have adopted from
the brain spatial basis functions (SBFs), including Multivariate Source Pre-
localization combined with Data-Driven Parcellization (MSP+DDP)[4, 5],
Gaussian kernels[6, 2], and Spherical Harmonics[7]. However, these SBFs-based
ESI methods have their own limitations. Specifically, the MSP+DDP method is
data-driven and highly sensitive to noise. The sparse Gaussian kernels cannot
cover the whole brain and lack biological interpretability. Harmonic method
misses biological information about the brain structure and function.

Figure 1: Geometric-informed basis function for EEG source imaging. a, Head
model (middle) is constructed using individual MRI and BEM method, the scalp
EEG (left) can be recovered with forward projecting of EEG source using lead
field matrix, and the brain surface geometry mesh (right) is obtained from
individual MRI. A series of GBFs (bottom) are calculated by eigendecomposition
of the brain geometric. b, The eigenvalues of GBFs (left) and its prior
distribution (right).

Here, we develop the geometric-informed basis functions (GBFs)-based ESI
method. GBFs, derived from brain geometry using MRI, are reported to be
promising basis functions for fitting fMRI data[8]. These GBFs are task-
context independent and are reliable at low signal-to-noise ratios (SNRs).
Such properties of GBFs bring unique benefits to ESI. Our study has three main
contributions:
1) We develop a GBFs-based ESI method that leverages the brain geometric as
prior (Fig. 1).
2) We present a framework to generate the EEG source-scalp data from fMRI meta
sources (Fig. 2a).
3) We demonstrate the superior performance of GBFs using the synthetic data
(Fig. 2b & 3) and the real task EEG data (Fig. 4).

##  II Methods

###  II-A EEG/MEG Source Imaging

ESI is constituted by a forward problem (i.e., head model construction) and an
inverse problem (i.e., source localization).

ESI Forward Problem: Mathematically, the forward process from neural sources
$x$ to sensor measurements $y$ can be formulated as follows:

$y=Kx+\varepsilon,$ (1)

where $x\in\mathbb{R}^{N}$ represents the neural activity in $N$ sources, and
$y\in\mathbb{R}^{M}$ denotes the measurements in $M$ sensors, and
$\varepsilon$ is the measurement noise, $K\in\mathbb{R}^{M\times N}$ is the
lead-field matrix, which embodies the head volume conductivity model,
describing a linear mapping relationship between sensor signals and source
activities [9].

ESI Inverse Problem: In Bayesianism, ESI inverse problem is formulated to
maximize the posterior probability $P(x|y,K)$, given a prior distribution of
brain sources $P(x)$ and the forward model in Eq. (1), and the scalp
measurements $y$.

###  II-B GBF Algorithm to extract individual GBFs

Our ESI method employs GBFs [8], derived from a cortical surface mesh
generated by $Freesurfer$ [10]. We present a GBF algorithm to extract
individual GBFs. It takes into consideration the individual cortical
geometries to enhance applicability and accuracy. Moreover, it can extract
GBFs directly from the source space rather than from T1-weighted (T1W) images.
These advantages fulfill critical needs for ESI, allowing EEG researchers to
easily employ these extracted GBFs for ESI as they inherently match the source
space.

The computational efficiency of GBF algorithm is improved by down-sampling the
mesh (10242 vertices per hemisphere), while keeping adaptability to
accommodate various source space sizes. We use an adapted Laplace–Beltrami
Operator (LBO) for the cortex mesh within 3D Euclidean space[11]:

$\Delta:=\frac{1}{W}\sum_{i,j}\frac{\partial}{\partial
x_{i}}\left(g^{ij}W\frac{\partial}{\partial x_{j}}\right)$ (2)

where $x_{i},x_{j}$ are coordinates, $g^{ij}$ is the inverse of the metric
tensor $g$, and $W=\sqrt{\text{det}(G)}$ with $G$ being the matrix
representation of $g$. The metric tensor $g$ represents the local geometric
properties of the cortical surface (e.g., the curvature and shape), describing
how distances and angles are measured on the surface at each point. The
inverse $g^{ij}$ is used for transforming vectors between coordinate systems
and calculating inner products, ensuring accurate representation of cortical
curvature and shape variations.

This LBO operator $\Delta$ is further used to decompose the cortical mesh into
geometric eigenmodes (300 in our study), defined as:
$\Delta\psi=-\lambda\psi$, where $\psi\in\mathbb{R}^{S\times N}$ denotes the
geometric eigenmodes(lower panel in Fig.1.a), where S is the number of the
eigenmodes, and $\lambda\in\mathbb{R}^{S}$ indicates their eigenvalues. The
eigenvalues $\lambda$ are ordered by their spatial frequency, with lower
frequencies corresponding to broader spatial patterns, and then transformed
into its prior variance for source imaging (See Fig. 1b).

Figure 2: Generating synthetic dataset and comparisons of ESI results. a, The
EEG source-scalp paired data is generated by fMRI meta-analysis (ICA) from
NeuroVault images, transformed from volume space (MNI) to surface space
(fsaverage), and finally projected to EEG scalp. b, We exhibit the ground
truth (gray), the ESI results from our method (yellow), compared with other
basis functions-based methods (red) and traditional ESI methods (blue).

###  II-C Solving ESI inverse problem with spatial basis functions

We further employ these spatial basis functions for source imaging, and
formulate as $x=A\theta$, where $A$ consists of spatial basis functions and
$\theta$ denotes the corresponding coefficients. For EEG/MEG data $y$, this
relationship is modeled as:

$y=KA\theta+\varepsilon.$ (3)

We simplify this to $KA=L$ for further analysis.

Assuming that $\theta$ follows a Gaussian distribution $P(\theta)\sim
N(0,\Sigma)$, the covariance matrix $\Sigma$ is defined as a diagonal matrix,
with its elements being the reciprocal of eigenvalues of GBFs. As Fig. 1b, to
ensure numerical stability, a regularization term is introduced:

$diag(\Sigma)=\frac{1}{\lambda+0.1\bar{\lambda}},$ (4)

where $\bar{\lambda}=\frac{1}{L}\sum\lambda$. The MAP estimation of $\theta$
is then determined by:

$\hat{\theta}=(L^{T}L+\beta\Sigma^{-1})^{-1}L^{T}y,$ (5)

with $\beta$ representing the regularization coefficient.

Our approach offers a detailed yet efficient solution for EEG/MEG imaging,
enhancing the ESI accuracy by incorporating spatial basis functions. It aligns
well with established Minimum Norm methods[12] and can be integrated into
advanced Bayesian and deep learning frameworks[2].

##  III Experiments

###  III-A Synthetic source-scalp data

####  III-A1 Generating source spatial maps based on NeuroVault images

We present a data generation framework using the spatial maps from fMRI meta-
analysis for creating a realistic synthetic dataset (Fig.2a). Specifically, we
randomly selected 2100 statistical images from NeuroVault [13], which were
transformed into MNI-152 volume space and annotated with 1207 Neurosynth terms
[14]. We construct an imaging-term matrix and run Independent Component
Analysis (ICA) to extract 50 spatial maps using $Nilearn$. Then, we manually
select 40 spatial maps according to their associated terms, and transform them
to fsaverage surface space using $Neuromaps$ [15, 16]. These maps served as
source spatial maps for our synthetic dataset and the corresponding scalp data
is obtained by forward projection.

####  III-A2 Noise settings

Figure 3: Comparison of ESI Methods under Various Noise Conditions and SNRs.
a, Comparison of the SE metrics across different SNRs for Gaussian noise
(left) and realistic noise (right). Our GBFs-based ESI method consistently
outperforms other methods under varying SNR conditions. b, Performance
comparison of different ESI methods at SNR=5, in terms of four evaluation
metrics. GBFs show superior performance across all metrics compared to other
competing methods.

(1) Gaussian Noise: We assume the sensor noise $n\in\mathbb{R}^{M}$ is
channel-independent, and i.i.d. to the normal distribution,
$n\sim\mathcal{N}(0,I)$. (2) Realistic Noise: We generate the realistic noise
by using the covariance matrix from the real task EEG data of the VEPCON
dataset [17]. The procedure is encapsulated in the following equation,
$\mathbf{N}_{\text{R}}=\sqrt{\text{diag}(\mathbf{V})}\cdot\mathbf{E}\cdot\mathbf{R}$,
where $\mathbf{V}$ and $\mathbf{E}$ are the eigenvalues and eigenvectors of
the normalized covariance matrix, and R is a matrix sampled from standard
normal distribution.

Generating noisy data at different SNRs: The generated sensor noise is further
scaled to different SNRs, according to
$\text{SNR}_{\text{dB}}=10\log_{10}\left(\frac{P_{\text{signal}}}{P_{\text{noise}}}\right)$.
Finally, we mix the clean signal and the noise with SNR ranging from -20 to
20.

###  III-B Competing methods and evaluation metrics

####  III-B1 ESI methods using other basis functions

We implemented the ESI methods using other two types of SBFs (i.e., Harmonic
and MSP) as competing methods. The Harmonic method conceptualizes each
hemisphere as a sphere, projects each source onto the spherical surface, and
then employs Legendre polynomials to solve the spherical harmonic problem,
deriving a set of SBFs. In our implementation, we selected polynomials up to
the 6th degree, yielding 49 harmonic SBFs. MSP method uses Singular Value
Decomposition (SVD) of the whitened lead-field matrix $\mathbf{K}$, and then
take the right singular vectors as the MSP spatial basis functions.

####  III-B2 Evaluation metrics

For the synthetic data, we have the ground-truth sources. We use Shape Error
(SE), Mean Correlation Coefficient (MCC), Localization Error (LE) and Source
Divergence (SD) as evaluation metrics, which have been widely used in previous
ESI studies[6, 2].

###  III-C The real task EEG data

We assess the efficacy of our GBFs-based ESI method using two real EEG
datasets. (1) The VEPCON dataset includes individual T1 MRI and 128-channel
EEG recordings with 2048 sampling rating during a visual task, discriminating
faces from scrambled images [17]. The raw EEG signals were preprocessed with
the standard procedure (e.g., downsampled to 250 Hz, detrending, filtering and
ICA-based artifact removal). (2) The OSE dataset
(https://vbmeg.atr.jp/nictitaku209/) includes the individual T1 MRI and
64-channel EEG recordings during a somatosensory task (i.e., the right median
nerve electrical stimulation with eye closed).

##  IV Results

###  IV-A ESI results for the synthetic dataset

Fig. 2b & Fig. 3 show a comprehensive evaluation of our GBFs-based ESI,
demonstrating its superiority over all the competing methods, and across
various testing conditions. Specifically, Fig. 2b exhibit the estimated
sources from 6 ESI methods using synthetic auditory EEG with Gaussian noise at
5 SNR. Our GBFs method uncovers the auditory region highly resemble to the
ground truth sources. In Fig. 3a, we test the robustness of GBFs across a
spectrum of SNR levels, ranging from -20 to 20, under Gaussian noise and
realistic noise conditions, respectively. The results indicate GBFs reliably
outperform all competing ESI methods, in terms of the SE metric. Moreover,
Fig. 3b shows the efficacy of GBFs in four evaluation metrics (SE, MCC, SD,
and LE) at SNR=5, indicating the superiority of GBF-based ESI in all metrics.

###  IV-B ESI results for two real EEG datasets

In Fig.4, we illustrate the estimated sources from our GBFs-based ESI method,
compared with other two SBFs-based ESI (Harmonic and MSP) and four traditional
ESI methods (MNE, dSPM, sLORETA and eLORTEA). Note that real EEG datasets do
not have the ground-truth source images. We expected that the sources of EEG
for the visual task and for the somatosensory task would be located in visual
areas and somatosensory areas, respectively. The results in Fig.4 show that
the source maps obtained by our GBFs-based ESI method are in good agreement
with expectations and consistent with neuroscientific knowledge.

Figure 4: ESI results for visual task EEG (a) and somatosensory task EEG (b).
We exhibit the ESI results from our method (yellow), compared with other basis
functions-based method (red), and the traditional EEG source localization
methods (blue).

##  V Conclusion

In summary, we developed the GBFs-based ESI method, using brain geometric
priors to advance EEG/MEG source imaging. To validate our method, we presented
a framework to synthesize EEG source-scalp data using brain sources from fMRI
meta-analysis. Using the synthetic data, we demonstrated the superior
performance of GBFs-based ESI over traditional ESI methods and over other
spatial basis functions. GBFs can generalize well to the real task EEG data.
GBFs-based ESI deals with the spatial resolution limit of EEG/MEG, offering a
reliable, biologically interpretable tool for EEG source localization.

## References

  * [1] S. Baillet, “Magnetoencephalography for brain electrophysiology and imaging,” _Nature neuroscience_ , vol. 20, no. 3, pp. 327–339, 2017.
  * [2] C. Wei, K. Lou, Z. Wang, M. Zhao, D. Mantini, and Q. Liu, “Edge sparse basis network: a deep learning framework for eeg source localization,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [3] Q. Liu, M. Ganzetti, N. Wenderoth, and D. Mantini, “Detecting large-scale brain networks using eeg: impact of electrode density, head modeling and source localization,” _Frontiers in neuroinformatics_ , vol. 12, p. 4, 2018.
  * [4] J. Mattout, M. Pélégrini-Issac, L. Garnero, and H. Benali, “Multivariate source prelocalization (msp): use of functionally informed basis functions for better conditioning the meg inverse problem,” _NeuroImage_ , vol. 26, no. 2, pp. 356–373, 2005.
  * [5] R. A. Chowdhury, J. M. Lina, E. Kobayashi, and C. Grova, “Meg source localization of spatially extended generators of epileptic activity: comparing entropic and hierarchical bayesian approaches,” _PloS one_ , vol. 8, no. 2, p. e55969, 2013.
  * [6] S. Haufe, R. Tomioka, T. Dickhaus, C. Sannelli, B. Blankertz, G. Nolte, and K.-R. Müller, “Large-scale eeg/meg source localization with spatial flexibility,” _NeuroImage_ , vol. 54, no. 2, pp. 851–859, 2011.
  * [7] Y. Petrov, “Harmony: Eeg/meg linear inverse source reconstruction in the anatomical basis of spherical harmonics,” 2012.
  * [8] J. C. Pang, K. M. Aquino, M. Oldehinkel, P. A. Robinson, B. D. Fulcher, M. Breakspear, and A. Fornito, “Geometric constraints on human brain function,” _Nature_ , pp. 1–9, 2023.
  * [9] J. Vorwerk, R. Oostenveld, M. C. Piastra, L. Magyari, and C. H. Wolters, “The fieldtrip-simbio pipeline for eeg forward solutions,” _Biomedical engineering online_ , vol. 17, no. 1, pp. 1–17, 2018.
  * [10] B. Fischl, “Freesurfer,” _Neuroimage_ , vol. 62, no. 2, pp. 774–781, 2012.
  * [11] S. Seo and M. K. Chung, “Laplace-beltrami eigenfunction expansion of cortical manifolds,” in _2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro_. IEEE, 2011, pp. 372–375.
  * [12] B. He, A. Sohrabpour, E. Brown, and Z. Liu, “Electrophysiological source imaging: a noninvasive window to brain dynamics,” _Annual review of biomedical engineering_ , vol. 20, pp. 171–196, 2018.
  * [13] K. J. Gorgolewski, G. Varoquaux, G. Rivera, Y. Schwarz, S. S. Ghosh, C. Maumet, V. V. Sochat, T. E. Nichols, R. A. Poldrack, J.-B. Poline _et al._ , “Neurovault. org: a web-based repository for collecting and sharing unthresholded statistical maps of the human brain,” _Frontiers in neuroinformatics_ , vol. 9, p. 8, 2015.
  * [14] T. Yarkoni, R. A. Poldrack, T. E. Nichols, D. C. Van Essen, and T. D. Wager, “Large-scale automated synthesis of human functional neuroimaging data,” _Nature methods_ , vol. 8, no. 8, pp. 665–670, 2011.
  * [15] R. D. Markello, J. Y. Hansen, Z.-Q. Liu, V. Bazinet, G. Shafiei, L. E. Suárez, N. Blostein, J. Seidlitz, S. Baillet, T. D. Satterthwaite _et al._ , “Neuromaps: structural and functional interpretation of brain maps,” _Nature Methods_ , vol. 19, no. 11, pp. 1472–1479, 2022.
  * [16] J. Wu, G. H. Ngo, D. Greve, J. Li, T. He, B. Fischl, S. B. Eickhoff, and B. T. Yeo, “Accurate nonlinear mapping between mni volumetric and freesurfer surface coordinate systems,” _Human brain mapping_ , vol. 39, no. 9, pp. 3793–3808, 2018.
  * [17] D. Pascucci, S. Tourbier, J. Rué-Queralt, M. Carboni, P. Hagmann, and G. Plomp, “Source imaging of high-density visual evoked potentials with multi-scale brain parcellations and connectomes,” _Scientific Data_ , vol. 9, no. 1, p. 9, 2022.