# Background Debiased SAR Target Recognition via Causal Interventional
Regularizer

Hongwei Dong  Fangzhou Han  Lingyu Si  Wenwen Qiang
Lamei Zhang  Science & Technology on Integrated Information System Laboratory,
Institute of Software, Chinese Academy of Sciences, Beijing, China  University
of Chinese Academy of Sciences, Beijing, China  Department of Information
Engineering, Harbin Institute of Technology, Harbin, China

###### Abstract

Recent studies have utilized deep learning (DL) techniques to automatically
extract features from synthetic aperture radar (SAR) images, which shows great
promise for enhancing the performance of SAR automatic target recognition
(ATR). However, our research reveals a previously overlooked issue: SAR images
to be recognized include not only the foreground (i.e., the target), but also
a certain size of the background area. When a DL-model is trained exclusively
on foreground data, its recognition performance is significantly superior to a
model trained on original data that includes both foreground and background.
This suggests that the presence of background impedes the ability of the DL-
model to learn additional semantic information about the target. To address
this issue, we construct a structural causal model (SCM) that incorporates the
background as a confounder. Based on the constructed SCM, we propose a causal
intervention based regularization method to eliminate the negative impact of
background on feature semantic learning and achieve background debiased SAR-
ATR. The proposed causal interventional regularizer can be integrated into any
existing DL-based SAR-ATR models to mitigate the impact of background
interference on the feature extraction and recognition accuracy. Experimental
results on the Moving and Stationary Target Acquisition and Recognition
(MSTAR) dataset indicate that the proposed method can enhance the efficiency
of existing DL-based methods in a plug-and-play manner.

###### keywords:

Deep learning, Causal inference, Synthetic aperture radar, Automatic target
recognition, Background debias

††journal: ISPRS Journal of Photogrammetry and Remote Sensing

##  1 Introduction

Synthetic aperture radar (SAR) is an active remote sensor that enables all-
weather and day-and-night detection, making it a crucial component of Earth
observation systems (Moreira et al., 2013). One of the primary objectives of
SAR image data processing is automatic target recognition (ATR) (El-Darymli et
al., 2016). Achieving this goal requires automated and intelligent methods
that can accurately recognize targets by extracting discriminative features
from SAR images, which remains a challenging task (Kechagias-Stamatis and
Aouf, 2021).

Deep learning (DL) overcomes the limitations of traditional manual feature
engineering and delivers superior performance in a data-driven manner, making
it a powerful solution to address the aforementioned challenge (LeCun et al.,
2015). Researchers have made initial forays into DL-based SAR-ATR methods in
recent years (Chen et al., 2016), but further refinement is necessary before
they can be effectively applied to real-world scenarios due to their
relatively short development time (Zhu et al., 2021).

The general structure of SAR-ATR systems consists of three phases: detection,
discrimination, and recognition. In related studies, the first two phases,
commonly known as the focus-of-attention (FOA), are employed to locate the
region of interest (ROI) that may contain targets, without necessitating any
human intervention. Both traditional (El-Darymli et al., 2013) and DL-based
approaches (Zou et al., 2022) have demonstrated satisfactory FOA performance.

Using the results of FOA as input, the SAR-ATR method is designed to infer the
classes of targets that may be included in the ROI. In this phase, the
discriminability of extracted features (i.e., the feature with larger inter-
class distance and smaller intra-class distance is more discriminative) is
crucial for recognition performance. Various traditional methods have been
proposed to improve the feature discriminability, such as hand-crafted feature
extractors (Amrani et al., 2018), attributed scattering centers (Li and Du,
2019), low-rank matrix factorization (Zhang et al., 2018). However, DL-based
methods offer substantial advantages in terms of performance compared to
traditional approaches. Chen et al. (2016) and Ding et al. (2016) conduct
pioneering work that utilize DL-models for SAR image feature extraction and
target recognition, setting the foundation for further advancements.
Subsequently, a significant portion of research has focused on refining the
model architecture to enhance the performance of SAR-ATR, including multi-
stream (Pei et al., 2017), attention mechanism (Zhang et al., 2020), capsule
structure (Ren et al., 2021), vision transformer (Wang et al., 2022), etc.
Given that SAR image processing often encounters the issue of speckle noise,
some studies have proposed methods for extracting the speckle-noise-invariant
features (Kwak et al., 2019; Lei et al., 2021). Another aspect of the research
focuses on adversarial learning, which involves using generative models for
data augmentation (Sun et al., 2019) or performing adversarial attacks on
existing models (Peng et al., 2022; Du and Zhang, 2021) to enhance the
robustness of their feature extraction. In addition, research on DL-based
methods for environments with a limited number of training samples (Fu et al.,
2022; Wen et al., 2021) and edge devices (Wang et al., 2021) is gradually
emerging.

Figure 1: Typical targets in SAR images. (a) Ship target (Huang et al., 2018).
(b) Vehicle target (AFRL and DARPA, 2020). (c) Aircraft target (Sun et al.,
2022).

The aforementioned studies have advanced the capability of SAR-ATR from a
methodological perspective, but have overlooked the issue of background
interference. As shown in Fig. 1, the ROI includes not only the foreground
(i.e., target), but also a certain size of the background. Therefore, the
following question arises:

  * -

When the foreground and background of two SAR images are different, do the DL-
model extracted features belong to the foreground, the background, or a
mixture of both?

A satisfactory answer to this question is that the DL-model extracts semantic
information solely from the foreground, as the background is irrelevant for
target recognition. In this way, the discriminability of extracted features
can be maximized. However, in reality, the situation may differ. Firstly, it
should be noted that eliminating the background in ROI can be challenging due
to the irregular shape and non-standardized size of non-cooperative targets
(Huang et al., 2018; AFRL and DARPA, 2020; Sun et al., 2022). Therefore, the
input of SAR-ATR typically includes both foreground and background. Then, we
conduct a motivating experiment to demonstrate that the features extracted by
DL-models are a mixture of foreground and background elements.

Specifically, we employ two DL-based models, i.e., VGG16 (Simonyan and
Zisserman, 2014) and ResNet18 (He et al., 2016), and evaluate their
performance on the Moving and Stationary Target Acquisition and Recognition
(MSTAR) dataset (AFRL and DARPA, 2020). This dataset is constructed for
scientific research purposes and contains only cooperative vehicle targets
that possess similar sizes, roughly measuring 40$\times$40\. Based on this
observation, we evaluate the performance of two DL-models under four distinct
crop conditions: preserving only the central 40$\times$40, 64$\times$64,
88$\times$88, and 128$\times$128 image crops, while masking the remaining
areas. The accuracy and feature visualizations of different experimental
setups are shown in Fig. 2.

Figure 2: Results of the motivating experiment for four image crop methods.
(a) Recognition accuracy of different center-crops. (b)-(c) Feature
visualization of $64\times 64$, $88\times 88$ and $128\times 128$ center-crops
using t-SNE (Van der Maaten and Hinton, 2008).

As shown in Fig. 2(a), the accuracy of SAR-ATR progressively decreases as the
image background intensifies. Moreover, Fig. 2(b)-(d) demonstrate that the
inclusion of background reduces the distance between different classes in the
latent space, thereby diminishing the discriminability of extracted features.
This phenomenon is reasonable from a causal perspective since there is no
semantic correlation between the foreground and background. Therefore, the
elements belonging to the background in the extracted features cannot have a
positive impact on target recognition. Both the experimental and theoretical
analyses indicate that the background is likely to interfere with the feature
extraction process, which should focus on the foreground, leading to biased
feature learning objectives and the deterioration of feature quality.

To address this issue, we utilize a structural causal model (SCM) (Pearl,
2009) to depict the causal relationships among the variables in the SAR-ATR
process, including the input, DL-model, foreground, background, and
prediction. In the proposed SCM, the background is set as the confounder,
which is consistent with the motivating experimental results. By analyzing
this SCM, we obtain a theoretical solution for treating background
interference. This guides us to represent the theoretical background
elimination process as a causal interventional regularizer. As a result, we
can achieve background debiased feature learning, support better feature
discriminability, and subsequently obtain satisfactory SAR-ATR performance. In
summary, the main contributions of this paper are three-fold:

  * (1)

We highlight a critical but overlooked issue that the background exerts an
adverse influence on the performance of SAR-ATR, and offer a remedy based on
the perspective of causal inference.

  * (2)

We construct a SCM based on the process of SAR image feature learning to
depict the causal relationships among the input, DL-model, foreground,
background, and prediction. This provides a theoretical explanation and
solution for background interference.

  * (3)

Guided by the constructed SCM, we propose a causal intervention based
regularization method. This method can be utilized to effectively mitigate the
adverse effects of background on feature learning of the DL-model in a plug-
and-play manner, leading to significant enhancement in the performance of SAR-
ATR.

The remainder of this paper is organized as follows. Section 2 briefly reviews
related works. Section 3 describes the proposed method. Experiment results and
analyses are presented in Section 4. Finally, Section 5 concludes this paper.

##  2 Related Works

This section provides a comprehensive review of the research literature
related to this paper, including DL-based SAR-ATR methods, background related
methods, and causality theory.

###  2.1 General Paradigm of DL-based SAR-ATR

Convolutional neural network (CNN) is one of the most significant DL
techniques and has achieved state-of-the-art results in various image
processing applications, particularly image recognition. The fundamental
prototype of CNN, proposed by LeCun et al. (1989), is characterized by
desirable properties such as weight sharing and local connection. Krizhevsky
et al. (2012) have made significant contributions to the development of
current CNN models by using ReLU activation, local response normalization, and
GPU training to enhance model capacity and efficiency. CNN is typically
regarded as an end-to-end methodology for feature extraction and
classification, providing automated feature engineering that is more powerful
than traditional hand-crafted features and kernel methods.

The seminal studies of DL-based SAR-ATR method are conducted by Chen et al.
(2016) and Ding et al. (2016). These studies utilize stacked CNN models
composed of convolution, activation, pooling, and fully-connected layers to
extract abstract but discriminative deep features from SAR images to achieve
promising performance in target recognition. Despite considerable research in
this area and some progress being made, the background debiased DL-based SAR-
ATR approach has not been explored so far, which is the objective of this
paper.

###  2.2 Background Debias Related Studies in SAR-ATR

Several studies have analyzed the effect of background on SAR-ATR (Papson and
Narayanan, 2012; Cui et al., 2005; Lombardo et al., 2001). Empirical evidence
provides by Zhou et al. (2019) demonstrates the significant impact of
background replacement on the performance of SAR-ATR. Moreover, Belloni et al.
(2020) discuss the role of target, shadow, and clutter in recognition to gain
insight into the decision-making process of DL-models.

While there is currently no research exploring background bias from a
methodological perspective, the impact of this issue has been observed in the
implementation of many SAR-ATR methods. In related studies, the negative
effects caused by the background are mitigated by employing two main
techniques, i.e., image pre-processing and data augmentation. The image pre-
processing method (Feng et al., 2022) involves a multi-stage algorithm that
begins by detecting the edges of the target, allowing for the segmentation of
the target region from the background. Then, chip recognition is performed to
improve performance. On the other hand, the data augmentation method (Ding et
al., 2016; Chen et al., 2016) is based on sampling a large number of chips
from the original image with the same foreground but slightly different
backgrounds. These operations prompt the DL-model to learn background-
irrelevant features.

However, both of the techniques mentioned above have obvious drawbacks:
Firstly, the image pre-processing method not only risks losing target details
but also has a high level of complexity. Secondly, the data augmentation
method relies on a strong prior, i.e., all targets to be recognized have a
fixed and known size. This approach can be challenging when dealing with the
recognition problem of non-cooperative targets. In this paper, we address this
issue from a causal inference perspective and propose an end-to-end method
with a plug-and-play capability.

###  2.3 Causality Theory

Causality has become as a significant subfield of statistics, with its own
conceptual framework, descriptive language, and methodological system. A
comprehensive review of the existing literature in (Schölkopf, 2022)
highlights how causality theory has been instrumental in the development of
machine learning by addressing the deficiencies of current approaches.

SCM (Pearl, 2009) is a crucial tool in causality theory as it can effectively
depict the interdependencies between factors involved in a process using a
graph structure. By scrutinizing the SCM, every conceivable alteration that
may take place within a process and the pluralistic consequences arising from
these alterations may be identified. With the help of SCM, the distinction
between correlation and causality can be explored, leading to the current
causal inference framework (Glymour et al., 2016). This framework decomposes
the problem into three levels: correlation, intervention, and counterfactual,
corresponding to observation, action, and imagination, respectively.
Intervention, which involves altering part of the variable generation
mechanism while preserving the remainder, is a critical operation for
determining and quantifying causality. Scientific research commonly employs
intervention, such as through randomized controlled experiments, to establish
the presence and direction of causality.

##  3 Methodology

This section presents a detailed introduction to the proposed method for
causal intervention based regularization. First, we introduce the construction
and analyses of the SAR-ATR oriented SCM. Next, we provide a comprehensive
overview of the proposed method, including its model architecture and loss
function.

In contrast to many variants in this field that prioritize model refinement,
the proposed method focuses on a background deibiased feature learning
process. This approach eliminates interference from the background in DL-based
SAR-ATR models, guided by causal intervention. This process significantly
enhances the discriminability of the extracted features and resulting in
superior recognition performance.

###  3.1 Causal Analysis

To illustrate the impact of background in the process of SAR-ATR, we use the
SCM as a basis for our analysis. The SCM is a directed acyclic graph in which
nodes represent variables and directed edges indicate causality between them.

Figure 3: Illustration of the constructed SCM. It depict the causal
relationships among the variables in the SAR-ATR process, including the input,
DL-model, foreground, background, and prediction. The top part uses examples
to demonstrate the conceptual causality. The bottom part is the abstracted SCM
for DL-based SAR-ATR.

As shown in Fig. 3, we refer to the input SAR images as $\bm{X}$, the feature
extractor of the DL-based SAR-ATR model as $\bm{X_{s}}$, and the features
extracted by $\bm{X_{s}}$ as $(\bm{X_{f}};\bm{X_{b}})$, where $\bm{X_{f}}$
corresponds to foreground features and $\bm{X_{b}}$ corresponds to background
features. The recognition of the label $\bm{Y}$ is supported by both
$\bm{X_{f}}$ and $\bm{X_{b}}$. In the following, we provide a detailed
description of the proposed SCM and the underlying principles behind its
construction.

$\bm{X}\rightarrow\bm{X_{s}}$ indicates that the DL-model $\bm{X_{s}}$ is used
to extract features from the input SAR images $\bm{X}$.
$\bm{X_{s}}\rightarrow\bm{X_{b}}$ indicates that background features
$\bm{X_{b}}$ is extracted by the model $\bm{X_{s}}$.
$\bm{X_{s}}\rightarrow\bm{X_{f}}$ indicates that $\bm{X_{f}}$ is extracted by
$\bm{X_{s}}$. $\bm{X_{b}}\rightarrow\bm{Y}$: This link indicates that
recognition of input SAR images relies on the information in the background
features $\bm{X_{b}}$, since all DL-model extracted features are used for the
prediction of $\bm{Y}$. $\bm{X_{f}}\rightarrow\bm{Y}$: This link represents
that the label of an input sample is determined by its foreground features,
which is the true causality.

As previously analyzed, the ideal DL-model should capture the true causality
between $\bm{X_{f}}$ and $\bm{Y}$. Therefore, we expect the prediction of the
input SAR image to be driven primarily by the foreground features, rather than
the background features. However, as shown in the constructed SCM, the
conventional DL-model fails to capture such causality because the prediction
$\bm{Y}$ is not only determined by $\bm{X_{f}}$ via the link
$\bm{X_{f}}\rightarrow\bm{Y}$, but also by the spurious correlation via the
link $\bm{X_{f}}\leftarrow\bm{X_{s}}\rightarrow\bm{X_{b}}\rightarrow\bm{Y}$.
It means that the DL-based feature extractor $\bm{X_{s}}$ generates the
background features $\bm{X_{b}}$ from the background of $\bm{X}$. These
features provide part of the semantics that are utilized when recognizing
targets.

The above causalities can be reflected in the motivating experiment
corresponding to Fig. 2: The performance of DL-based SAR-ATR models trained on
input SAR images that include both foreground and background is significantly
lower compared to models trained on background-removed data. To capture the
true causality while suppressing the spurious correlation, we perform backdoor
adjustment by intervening on $\bm{X_{f}}$ using the $\textit{do}(\cdot)$
operation. This allows us to express the causality between $\bm{X_{f}}$ and
$\bm{Y}$ as $P(\bm{Y}|\textit{do}(\bm{X_{f}}))$, rather than relying on
$P(\bm{Y}|\bm{X_{f}})$.

It is widely recognized that the channels of the extracted features from any
DL-based feature extractor are closely related to the underlying feature
semantics (Zhou et al., 2016). Then we can simply assume that the output of
the last network layer of $\bm{X_{s}}$, i.e., $(\bm{X_{f}};\bm{X_{b}})$, can
be mapped into a series of feature semantics, denoted as
$(\bm{X_{f}};\bm{X_{b}})\mapsto\\{F_{i}\\}_{i=1}^{i=n}$, where
$F_{i}\in\mathbb{R}^{n_{c}}$ represents a stratification of feature semantics.
Considering that the confounder $\bm{X_{b}}$ is contained in
$(\bm{X_{f}};\bm{X_{b}})$ and $F$ is the feature semantics mapped from
$(\bm{X_{f}};\bm{X_{b}})$, it follows that a portion of each $F_{i}$
(belonging to the foreground) is informative for target recognition, while
another portion (belonging to the background) occupies the position that
should belong to the target information. Therefore, the goal of backdoor
adjustment should be to emphasize the foreground-related portions, and to
suppress the background-related portions of $F$ that are irrelevant and
harmful to target recognition. In this way, the true causality between
$\bm{X_{f}}$ and $\bm{Y}$ can be excavated. Formally, the backdoor adjustment
based $\textit{do}(\cdot)$ operation can be expressed as follows:

$P(\bm{Y}|\textit{do}(\bm{X_{f}}))=\sum_{i=1}^{n}P(\bm{Y}|\bm{X_{f}},F_{i})P(F_{i}).$
(1)

This equation presents a theoretical solution to eliminate background
interference. In the following subsection, we will delve into the specifics of
implementing the terms described in (1), and we will also detail our proposed
method guided by this theoretical solution.

###  3.2 Causal Interventional Regularizer

In this subsection, we embody the theoretical background interference
elimination process as a causal interventional regularizer.

Before introducing the proposed method, we note that in conventional DL-based
models, the functional implementation of $P(\bm{Y}|\bm{X_{f}})$ is provided by
a supervised learning process with the cross-entropy loss function, as
follows:

$P(\bm{Y}|\bm{X_{f}})=-\sum_{k=1}^{K}\mathbb{I}(\bm{Y}==k)\log\frac{\exp
c_{k}((\bm{X_{f}};\bm{X_{b}}))}{\sum_{j=1}^{K}\exp
c_{j}((\bm{X_{f}};\bm{X_{b}}))}$ (2)

where $k$ represents the $k$th class, $c$ is the mapping:
$\mathbb{R}^{n_{c}}\rightarrow\mathbb{R}^{K}$ from the last network layer to
the softmax classification layer, and $j$ is the $j$th dimension of the
prediction probability.

Figure 4: Illustration of the proposed causal intervention based
regularization method. It is comprised of two modules, i.e., the feature
extraction module and the semantic activation module. The former is a
conventional DL-model for extracting the foreground and background features
from the input SAR images. The latter can be seen as a causal interventional
regularizer that is used to intervene on the obtained feature semantics, thus
achieving the background debiased feature learning. In this figure, the
instances and labels of training samples, network architectures, and
intermediate outputs are marked by red, blue, and green boxes, respectively.

To embody the backdoor adjustment process described in (1), the first step is
to provide detailed functional implementations for
$P(\bm{Y}|\bm{X_{f}},F_{i})$ and $P(F_{i})$. Considering that $F_{i}$ contains
portions belonging to both foreground and background, generating large weights
for the foreground-related channels and small weights for the background-
related channels of $F_{i}$ is necessary. By re-weighting $F_{i}$ based on
these weights, we can effectively address the need for background interference
elimination.

Based on the above analyses, when a weight vector
$\alpha_{i}=\left[\alpha_{i,1},\alpha_{i,2},\cdots,\alpha_{i,n_{c}}\right]^{\text{T}}$
is given, the functional implementation for the first term of (1) can be
expressed as:

$P(\bm{Y}|\bm{X_{f}},F_{i})=-\sum_{k=1}^{K}\mathbb{I}(\bm{Y}==k)\log\frac{\exp
c_{k}(F_{i}\odot\alpha_{i})}{\sum_{j=1}^{K}\exp c_{j}(F_{i}\odot\alpha_{i})}$
(3)

where
$F_{i}\odot\alpha_{i}=\left[F_{i,1}\cdot\alpha_{i,1},F_{i,2}\cdot\alpha_{i,2},\cdots,F_{i,n_{c}}\cdot\alpha_{i,n_{c}}\right]$.
Therefore, given a weight matrix
$A=\left[\alpha_{1},\alpha_{2},\cdots,\alpha_{n}\right]$, and let
$P(F_{i})=1/n$, the overall implementation of the backdoor adjustment based
$\textit{do}(\cdot)$ operation can be expressed as:

$P(\bm{Y}|\textit{do}(\bm{X_{f}}))=-\sum_{i=1}^{n}(\sum_{k=1}^{K}\mathbb{I}(\bm{Y}==k)\log\frac{\exp
c_{k}(F_{i}\odot\alpha_{i})\times\frac{1}{n}}{\sum_{j=1}^{K}\exp
c_{j}(F_{i}\odot\alpha_{i})}).$ (4)

The aforementioned approach shifts our focus from providing functional
implementations for (1) to the task of generating the weight matrix $A$
required in (4). To accomplish this, we use a learnable network to generate
the required weight matrix. This idea inspires us to propose a novel
regularization method with plug-and-play capability based on causal
intervention. The DL-model that incorporates the proposed causal
interventional regularizer is comprised of two modules, i.e., the feature
extraction module and the semantic activation module. The unified framework
for this approach is depicted in Fig. 4. In the following, we give detailed
descriptions of the two modules and the functional learning objective.

####  3.2.1 Feature Extraction Module

When integrating the proposed method with DL-based SAR-ATR models, a
conventional DL-based feature extractor is utilized as the backbone to extract
task-specific features from the SAR images containing both foreground and
background. Therefore, the backbone of this module corresponds to $\bm{X_{s}}$
in Fig. 3, while its output corresponds to the feature semantics of
$(\bm{X_{f}};\bm{X_{b}})$.

We denote the training set as
$D_{\text{train}}=\\{X^{i},Y^{i}\\}_{i=1}^{i=N}$, where
$X^{i}\in\mathbb{R}^{w\times h\times c}$ represents the instance with a size
of $w\times h\times c$, and $Y^{i}\in\mathbb{R}$ represents the label in this
set. The forward propagation of the feature extraction module can be expressed
as $f_{\text{sem}}(X^{i})$, where $f_{\text{sem}}(\cdot)$ is the mapping of
the backbone of this module, parameterized by $W_{\text{sem}}$. It is used to
map the input to the semantic space, i.e., $f_{\text{sem}}:\mathbb{R}^{w\times
h\times c}\mapsto\mathbb{R}^{n\times n_{c}}$.

The proposed method has a plug-and-play capability, as we do not impose any
restrictions on the feature extractor. This is evident from the definition of
$f_{\text{sem}}(\cdot)$, allowing any DL-model to be used to build the feature
extraction module.

####  3.2.2 Semantic Activation Module

After obtaining the preliminary feature semantics $F$ using the feature
extraction module, another backbone is utilized by this module to extract the
semantic-activation-specific features and generate the weight matrix in (4).
Subsequently, the process of re-weighting from the preliminary to the
background debiased feature semantics, i.e., $F_{i}\odot\alpha_{i}$, can be
performed by intervening the feature semantics.

The forward propagation of the semantic activation module can be expressed as
$f_{\text{sem}}(X^{i})\odot f_{\text{sam}}(X^{i})$, where
$f_{\text{sam}}(\cdot)$ is the mapping of the backbone of this module,
parameterized by $W_{\text{sam}}$. It is used to map the input to the semantic
activation space, i.e., $f_{\text{sam}}:\mathbb{R}^{w\times h\times
c}\mapsto\mathbb{R}^{n\times n_{c}}$, which activates the feature semantics
belonging to the foreground while suppressing those belonging to the
background, thereby excavating the true causality.

####  3.2.3 Overall Learning Objective

The classification layer, parameterized by $W_{\text{cls}}$, is used to map
the preliminary and background debiased feature semantics to prediction
probabilities. Then the forward prediction and loss back-propagation of our
proposed framework can be supported. Following the definitions in (2) to (4),
for the training set $D_{\text{train}}$, the learning objective for the
feature extraction module can be expressed as the following cross-entropy
loss:

$\begin{split}&L_{\text{ce}}(X,Y,W_{\text{sem}},W_{\text{cls}})=\\\
&-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K}\mathbb{I}(Y^{i}==k)\log\frac{\exp
c_{k}(f_{\text{sem}}(X^{i}))}{\sum_{j=1}^{K}\exp
c_{j}(f_{\text{sem}}(X^{i}))}.\end{split}$ (5)

Besides, the $t$th learning objective for the semantic activation module can
be expressed as the following causal interventional regularizer loss:

$\begin{split}&L_{\text{cr}}^{t}(X,Y,W_{\text{sem}},W_{\text{sam}},W_{\text{cls}})=\\\
&-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K}\mathbb{I}(Y^{i}==k)\log\frac{\exp
c_{k}(f_{\text{sem}}^{t}(X^{i})\odot
f_{\text{sam}}^{t}(X^{i}))\times\frac{1}{n}}{\sum_{j=1}^{K}\exp
c_{j}(f_{\text{sem}}^{t}(X^{i})\odot f_{\text{sam}}^{t}(X^{i}))},\end{split}$
(6)

and the overall learning objective for this module is the sum of $n$ times
calculation for (6). Based on (5) and (6), we can obtain the overall learning
objective of the unified framework shown in Fig. 4:

$L_{\text{total}}=L_{\text{ce}}+\lambda\sum_{t=1}^{n}L_{\text{cr}}^{t}$ (7)

where $\lambda$ is the hyperparameter to control the trade-off. The two terms
of (7) correspond to the $P(\bm{Y}|\bm{X_{f}})$ and
$P(\bm{Y}|\textit{do}(\bm{X_{f}}))$ in the theoretical causal analysis, which
reflects the integration of the conventional DL-model with our proposed causal
interventional regularizer.

It is important to note that in a conventional DL-based SAR-ATR model, the
loss function only includes the $L_{\text{ce}}$ term. Therefore, the proposed
method can be seen as a way to regularize the conventional model using the
causal inference induced $L_{\text{cr}}$ term. That is why we refer to the
proposed method as causal interventional regularizer in this paper.

The training process of the proposed method is outlined as Algorithm 1.

Algorithm 1 Training the Background Debiased SAR-ATR Model

0: Model parameter initialization $W_{\text{sem}}$, $W_{\text{sam}}$, and
$W_{\text{cls}}$; Epoch $I$;Mini-batch size $B$;Learning rate
$\eta$;Hyperparameter $\lambda$;Training and validation sets
$D_{\text{train}}$ and $D_{\text{val}}.$

1: for epoch in $I$: do

2: for batch in $B$: do

3: Forward propagation by $f_{\text{sem}}$, $f_{\text{sam}}$, and $c$ with
trainable parameters of $W_{\text{sem}}$, $W_{\text{sam}}$, and
$W_{\text{cls}}$

4: # update the classification layer

5: $W_{\text{cls}}\leftarrow
W_{\text{cls}}-\eta\nabla_{W_{\text{cls}}}L_{\text{total}}$

6: # update the semantic activation module

7: $W_{\text{sam}}\leftarrow
W_{\text{sam}}-\eta\nabla_{W_{\text{sam}}}L_{\text{total}}$

8: # update the feature extraction module

9: $W_{\text{sem}}\leftarrow
W_{\text{sem}}-\eta\nabla_{W_{\text{sem}}}L_{\text{total}}$

10: end for

11: end for

11: The DL-model with the highest validation accuracy.

##  4 Experiments

In this section, we first introduce the dataset used in the experiments. Then
we proceed to demonstrate the performance of both the proposed method and the
compared methods. Finally, we analyze the effect of the hyperparameter in the
proposed method on its target recognition performance.

###  4.1 MSTAR Dataset

The experimental data utilized in this paper is obtained through the
employment of the Sandia National Laboratory SAR sensor platform. This venture
received funding from the Defense Advanced Research Projects Agency and the
Air Force Research Laboratory, as an integral component of the MSTAR program
(AFRL and DARPA, 2020). The dataset released to the public comprise ten
distinct ground target classes, namely armored personnel carrier types BMP-2,
BRDM-2, BTR-60, and BTR-70; tanks T-62 and T-72; rocket launcher 2S1; air
defense unit ZSU-234; truck ZIL-131; and bulldozer D7. The data is collected
using an X-band SAR sensor, in a spotlight mode with a resolution of 1-ft and
full aspect coverage (spanning $0^{\circ}$ to $360^{\circ}$).

(a) 2S1

(b) BMP-2

(c) BRDM-2

(d) BTR-60

(e) BTR-70

(f) D7

(g) T-62

(h) T-72

(i) ZIL-131

(j) ZSU-234

Figure 5: Ten types of military targets in the MSTAR dataset in the optical
and SAR formats. (a) 2S1. (b) BMP-2. (c) BRDM-2. (d) BTR-60. (e) BTR-70. (f)
D7. (g) T-62. (h) T-72. (i) ZIL-131. (j) ZUS-234.

The MSTAR benchmark dataset is widely employed for evaluating the performance
of SAR-ATR methods. Fig. 5 illustrates the SAR images of ten target classes
contained in the MSTAR dataset, captured at similar aspect angles. In
addition, it displays their corresponding optical images. In our experiments,
involved SAR-ATR methods are thoroughly evaluated under standard operating
conditions (SOC) as well as extended operating conditions (EOC) to ensure
their appropriate evaluation. SOC primarily includes utilizing serial numbers
and target configurations in the testing set that align with those in the
training set, however, with varying aspects and depression angles. On the
other hand, EOC involves substantial differences between the training and
testing sets, highlighting notable variations in depression angle, target
articulation, and version variants.

###  4.2 Experimental Settings

The effectiveness of the proposed method is demonstrated through comparisons
with notable alternatives, including VGG16 (Simonyan and Zisserman, 2014),
ResNet18 (He et al., 2016), and A-ConvNet (Chen et al., 2016). The former two
are the predominant DL-models utilized in image recognition, whereas A-ConvNet
holds extensive application in SAR-ATR domain.

In the experiments, we use a VGG16 model as the backbone of the semantic
activation module. The patch size of input images is limited to
128$\times$128\. The size of obtained feature map from backbone is
4$\times$4$\times$512\. When integrating the proposed method with the
comparison methods, they will be used as the feature extractor in the feature
extraction module in the proposed framework, and a fully-connected layer is
attached behind them to generate the feature semantics.

Sophisticated data pre-processing and augmentation techniques are frequently
employed in related studies for better recognition performance. Nevertheless,
to illustrate the inherent target recognition proficiency of the DL-models, we
refrain from carrying out any pre-processing activities, such as de-speckle,
on the SAR images to be recognized in the experimental setup. Furthermore, to
accentuate the background debias capability of the proposed method, we do not
perform any data augmentation for VGG16 and ResNet18 models, but retain the
original data augmentation operation of A-ConvNet, which involves randomly
sampling a substantial quantity of $88\times 88$ patches from the original
$128\times 128$ SAR images. This data augmentation method aims to mitigate the
negative impact of background on the feature learning of targets. The
aforementioned settings facilitate a straightforward comparison between the
effectiveness of the proposed and the data augmentation methods for the issue
of background debias.

The mini-batch stochastic gradient descent based method is used for model
optimization, with a batch size of $64$, and a training epoch of $100$. In
order to accelerate the optimization process and obtain more precise
approximate solutions, the Adam optimizer is employed with the learning rate
of $0.01$. Upon completion of the training process, the model with the highest
validation accuracy is preserved.

The experiments of this study employ a NVIDIA RTX A6000 GPU, and the program
is implemented using the PyTorch DL framework.

###  4.3 Experiments under SOC

In the experimental setup of SOC, which is listed in Table 1, the involved
methods are evaluated on the ten-class recognition problem.

Table 1: Number of Training and Testing samples for the SOC Experimental Setup

Class | Training Set $(17^{\circ})$ | Testing Set $(15^{\circ})$
---|---|---
2S1 | 299 | 274
BMP-2 | 233 | 196
BRDM-2 | 298 | 274
BTR-60 | 256 | 195
BTR-70 | 233 | 196
D7 | 299 | 274
T-62 | 299 | 273
T-72 | 232 | 196
ZIL-131 | 299 | 274
ZIL-234 | 299 | 274

Table 2: The comparison of Recognition Accuracy (%) under SOC

| CNN | VGG16 | ResNet18 | A-ConvNet
---|---|---|---|---
Accuracy(%) | 91.483 | 92.783 | 95.685 | 98.376
CNN+Ours | VGG16+Ours | ResNet18+Ours | A-ConvNet+Ours
95.637 | 98.557 | 98.215 | 99.228

Figure 6: The comparison of confusion matrices under SOC. (a) Results of
VGG16. (b) Results of background debiased VGG16.

It must be noted that the target serial number is identical in both the
training and testing sets, although they vary in their azimuth and depression
angles. The training images are acquired at a depression angle of
$17^{\circ}$, whereas the testing images are captured with a depression angle
of $15^{\circ}$.

Table 2 shows the comparison of the testing accuracy of CNN, VGG16, ResNet18,
A-Convnet before and after adding the proposed causal interventional
regularizer. In addition, the comparison of confusion matrices before and
after background debias for the VGG16 model is shown in Fig. 6.

As can be seen from the experimental results, for the three DL-models without
data augmentation, they also show significant improvements in the testing
accuracy after adding the proposed causal interventional regularizer. Among
them, background debiased VGG16 model even outperforms the A-ConvNet model,
which expands the original training set 1681 times by a random clipping based
data augmentation method. Specifically, after the causal-driven regularizer is
inserted, the involved four DL-models have 4.154%, 5.774%, 2.530%, and 0.852%
improvement in the recognition accuracy, respectively. The greatly improved
SAR-ATR performance shows that even if there are many background areas in the
ROI that interfere with target semantic learning, the conventional DL-based
models can still effectively focus on the extraction of foreground information
after adding the proposed causal interventional regularizer. This demonstrates
the satisfactory background debiased capability of the proposed method. In
addition, since the proposal is a plug-and-play method, there is no need to
make any modifications to the original model, which greatly improves the
generalizability of various DL-based recognition models in the field of SAR-
ATR.

###  4.4 Experiments under EOC

It is widely acknowledged that SAR images are highly susceptible to variations
in depression angles. Therefore, the robustness of SAR-ATR methods to the
changes in depression angle is important. To this end, we assess the proposed
method in terms of large depression angle changes (indicated by EOC-1).

Table 3: Details of EOC-1 Experimental Setup

Target Type | Training Set | Testing Set
---|---|---
Class | Serial No. | Depression | No. Samples | Depression | No. Samples
2S1 | B01 | $17^{\circ}$ | 299 | $30^{\circ}$ | 288
BRDM-2 | E71 | $17^{\circ}$ | 298 | $30^{\circ}$ | 287
T-72 | A64 | $17^{\circ}$ | 299 | $30^{\circ}$ | 288
ZSU-234 | D08 | $17^{\circ}$ | 299 | $30^{\circ}$ | 288

Table 3 enumerates that among the targets in the MSTAR dataset, only four
classes (2S1, BRDM-2, T-72, and ZSU-234) possess targets obtained at a
depression angle of $30^{\circ}$. Consequently, we evaluate the involved DL-
models on these samples, wherein the corresponding training set is the data of
these four classes of targets in SOC, i.e., the same targets obtained at a
depression angle of $17^{\circ}$. The large disparity in depression angle may
result in a dissimilar depiction of same targets in identical postures,
thereby augmenting the difficulty of recognition.

The EOC-1 dataset is classified using two models: the conventional VGG16 model
and the VGG16 model incorporating the proposed causal interventional
regularizer. The confusion matrix for large depression angle variations is
presented in Fig. 7. The experimental results indicate that the VGG16 model
achieves the accuracy of 96.351% while our proposed model has a higher
accuracy of 98.436%. This suggests that the proposed method is capable of
improving the robustness of DL-based SAR-ATR models even under challenging
conditions such as large depression angles.

Figure 7: The comparison of confusion matrices under EOC-1. (a) Results of
VGG16. (b) Results of background debiased VGG16.

In another EOC testing scenario, i.e., EOC-2, the assessment of SAR-ATR
methods revolves around variations in target configuration. The objective of
this validation is to evaluate the performance of the model to recognize
different variants of the same target. Specifically, the training set
encompasses BMP-2, BRDM-2, BTR-70 and T-72 at depression angles of
$15^{\circ}$ and $17^{\circ}$, whereas the testing set comprises of
diversified versions of T-72. The variance in target representation between
the two sets makes it arduous for the testing samples to be recognized as T-72
targets, thus intensifying the task challenge. Details of EOC-2 experimental
setup are shown in Table 4.

Table 4: Details of EOC-2 Experimental Setup

Partition | Class | Serial No. | Depression | No. Samples
---|---|---|---|---
Training Set | BMP-2 | C21 | $17^{\circ}$ | 233
BRDM-2 | E71 | $17^{\circ}$ | 298
BTR-70 | C71 | $17^{\circ}$ | 233
T-72 | 132 | $17^{\circ}$ | 232
Testing Set | T-72 | S7 |  $15^{\circ}$/$17^{\circ}$ | 419
A32 |  $15^{\circ}$/$17^{\circ}$ | 572
A62 |  $15^{\circ}$/$17^{\circ}$ | 573
A63 |  $15^{\circ}$/$17^{\circ}$ | 573
A64 |  $15^{\circ}$/$17^{\circ}$ | 573

Experimental results under EOC-2 can be seen from Table 5. Similar to the
previous experiments, we choose VGG16 as a representative of the conventional
DL-models to test its change in recognition accuracy before and after the
addition of the proposed causal interventional regularizer.

Table 5: The comparison of Recognition Accuracy (%) under EOC-2

Serial No. | No. Samples | VGG16 | VGG16+Ours
---|---|---|---
A32 | 572 | 88.462 | 95.979
A62 | 573 | 95.637 | 97.906
A63 | 573 | 94.939 | 96.684
A64 | 573 | 92.845 | 95.637
S7 | 419 | 79.714 | 80.191
Total | 2710 | 90.923 | 94.022

It is relatively obvious from the experimental results that the incorporation
of the proposed method improves the accuracy of the conventional DL-model for
the recognition of all variant classes of T-72. In particular, the recognition
accuracy of the serial number A32 is improved by 7.517%, which is a
considerable improvement. The above results effectively verify the
effectiveness of the proposed method for background debias when recognizing
variants of the same target.

###  4.5 Hyperparameter Analysis

To comprehensively investigate the impact of hyperparameter on the SAR-ATR
accuracy of the proposed method, we employ the VGG16 model with the proposed
regularization method and conduct experiments under SOC. Our primary focus is
on assessing the significance of the hyperparameter $\lambda$ in (7) on
recognition accuracy while moderating the proportion of the causal
interventional regularizer induced loss $L_{\text{cr}}$ in the overall loss
$L_{\text{total}}$, which denotes the intervention degree of the proposed
method in the feature extraction of conventional DL-model. We achieve this
goal by setting the value of $\lambda$ to $\\{0.001,0.01,0.1,0.5,1.0\\}$ and
conduct comparative experiments. The results are presented in Fig. 8.

Figure 8: The effect of hyperparameter $\lambda$ on the recognition accuracy.

As demonstrated in Fig. 8, the selection of hyperparameter $\lambda$ is
critical in promoting a desirable background debias effect. Notably, the
highest accuracy is attainable when $\lambda$ is set to $0.1$; deviating from
this value, whether by setting it too high or low, can result in a significant
decrease in recognition accuracy.

##  5 Conclusion

In this paper, we present the first attempt to investigate the potential of
background debias on DL-based SAR-ATR models from a methodological
perspective. Specifically, we propose a novel causal intervention based
regularization method to eliminate the interference of background on the
extraction of target semantic information. The proposed method is developed
based on the analysis of the SAR-ATR oriented SCM, which depicts the causal
relationships among the input, DL-model, foreground, background, and
prediction.

We model the background in the SAR image to be recognized as a confounder for
the feature extraction of DL-models, and eliminate it through backdoor
adjustment based causal intervention. This theoretical solution for
eliminating background interference is then transformed into a regularization
term for conventional DL-models, resulting in background debiased feature
learning. This method extracts more target semantics and improves the
discriminability of the extracted features, while also being easy to
implement, low complexity, widely adaptable, and having plug-and-play
capability.

Experiments on the MSTAR dataset demonstrate that our proposed method offers
advantages over conventional approaches, as evidenced by improved recognition
performance when combined with any conventional DL-model. Moving forward, we
plan to explore the combination of causal inference and diverse tasks such as
SAR target detection and change detection. We will also explore other learning
pipelines, including self-supervised learning, few-shot learning, and model
compression.

## CRediT authorship contribution statement

H.D. and F.H. were responsible for the conceptualization, design and
development of the methods, writing programmes, running experiments, analyzing
the results, and writing the paper. L.S., W.Q. and L.Z. were responsible for
investigation, funding requests and supervision. All authors reviewed the
manuscript.

## Conflict of interest

The authors declare that they have no conflict of interest.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China (62271172), in part by the China Postdoctoral Science Foundation
(2023M733615).

References

## References

  * AFRL and DARPA (2020) AFRL, DARPA, 2020.  Sensor data management system website, MSTAR database.  https://www.sdms.afrl.af.mil/index.php?collection=mstar.
  * Amrani et al. (2018) Amrani, M., Jiang, F., Xu, Y., Liu, S., Zhang, S., 2018\.  SAR-oriented visual saliency model and directed acyclic graph support vector metric based target classification.  IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 11, 3794–3810.
  * Belloni et al. (2020) Belloni, C., Balleri, A., Aouf, N., Le Caillec, J.M., Merlet, T., 2020.  Explainability of deep SAR ATR through feature analysis.  IEEE Trans. Aerosp. Electron. Syst. 57, 659–673.
  * Chen et al. (2016) Chen, S., Wang, H., Xu, F., Jin, Y.Q., 2016.  Target classification using the deep convolutional networks for SAR images.  IEEE Trans. Geosci. Remote Sens. 54, 4806–4817.
  * Cui et al. (2005) Cui, J., Gudnason, J., Brookes, M., 2005.  Automatic recognition of MSTAR targets using radar shadow and superresolution features, in: Proc.IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), pp. 589–589.
  * Ding et al. (2016) Ding, J., Chen, B., Liu, H., Huang, M., 2016.  Convolutional neural network with data augmentation for SAR target recognition.  IEEE Geosci. Remote Sens. Lett. 13, 364–368.
  * Du and Zhang (2021) Du, C., Zhang, L., 2021.  Adversarial attack for SAR target recognition based on UNet-generative adversarial network.  Remote Sens. 13.
  * El-Darymli et al. (2016) El-Darymli, K., Gill, E.W., Mcguire, P., Power, D., Moloney, C., 2016.  Automatic target recognition in synthetic aperture radar imagery: A state-of-the-art review.  IEEE Access 4, 6014–6058.
  * El-Darymli et al. (2013) El-Darymli, K., McGuire, P., Power, D., Moloney, C., 2013\.  Target detection in synthetic aperture radar imagery: A state-of-the-art survey.  J. Appl. Remote. Sens. 7, 071598\.
  * Feng et al. (2022) Feng, S., Ji, K., Ma, X., Zhang, L., Kuang, G., 2022\.  Target region segmentation in SAR vehicle chip image with ACM net.  IEEE Geosci. Remote Sens. Lett. 19, 4014605.
  * Fu et al. (2022) Fu, K., Zhang, T., Zhang, Y., Wang, Z., Sun, X., 2022\.  Few-shot SAR target classification via metalearning.  IEEE Trans. Geosci. Remote Sens. 60, 2000314.
  * Glymour et al. (2016) Glymour, M., Pearl, J., Jewell, N.P., 2016.  Causal inference in statistics: A primer.  John Wiley & Sons, Inc., Hoboken, NJ, USA.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 770–778.
  * Huang et al. (2018) Huang, L., Liu, B., Li, B., Guo, W., Yu, W., Zhang, Z., Yu, W., 2018.  OpenSARShip: A dataset dedicated to Sentinel-1 ship interpretation.  IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 11, 195–208.
  * Kechagias-Stamatis and Aouf (2021) Kechagias-Stamatis, O., Aouf, N., 2021\.  Automatic target recognition on synthetic aperture radar imagery: A survey.  IEEE Aerosp. Electron. Syst. Mag. 36, 56–81.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks, in: Proc. Adv. Neural Inf. Process. Syst. (NIPS), pp. 1097–1105.
  * Kwak et al. (2019) Kwak, Y., Song, W.J., Kim, S.E., 2019.  Speckle-noise-invariant convolutional neural network for SAR target recognition.  IEEE Geosci. Remote Sens. Lett. 16, 549–553.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., Hinton, G.E., 2015.  Deep learning.  Nature 521, 436–444.
  * LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., Jackel, L., 1989.  Backpropagation applied to handwritten zip code recognition.  Neural Comput. 1, 541–551.
  * Lei et al. (2021) Lei, P., Zheng, T., Wang, J., Bai, X., 2021.  A joint convolutional neural network for simultaneous despeckling and classification of SAR targets.  IEEE Geosci. Remote Sens. Lett. 18, 1610–1614.
  * Li and Du (2019) Li, T., Du, L., 2019.  SAR automatic target recognition based on attribute scattering center model and discriminative dictionary learning.  IEEE Sens. J. 19, 4598–4611.
  * Lombardo et al. (2001) Lombardo, P., Sciotti, M., Kaplan, L.M., 2001.  SAR prescreening using both target and shadow information, in: Proc. IEEE Radar Conf. (RadarConf), pp. 147–152.
  * Van der Maaten and Hinton (2008) Van der Maaten, L., Hinton, G.E., 2008\.  Visualizing data using t-SNE.  J. Mach. Learn. Res. 9, 2579–2605.
  * Moreira et al. (2013) Moreira, A., Prats-Iraola, P., Younis, M., Krieger, G., Hajnsek, I., Papathanassiou, K.P., 2013\.  A tutorial on synthetic aperture radar.  IEEE Geosci. Remote Sens. Mag. 1, 6–43.
  * Papson and Narayanan (2012) Papson, S., Narayanan, R.M., 2012\.  Classification via the shadow region in SAR imagery.  IEEE Trans. Aerosp. Electron. Syst. 48, 969–980.
  * Pearl (2009) Pearl, J., 2009.  Causality.  Cambridge University Press, Cambridge, UK.
  * Pei et al. (2017) Pei, J., Huang, Y., Huo, W., Zhang, Y., Yang, J., Yeo, T.S., 2017.  SAR automatic target recognition based on multiview deep learning framework.  IEEE Trans. Geosci. Remote Sens. 56, 2196–2210.
  * Peng et al. (2022) Peng, B., Peng, B., Zhou, J., Xie, J., Liu, L., 2022\.  Scattering model guided adversarial examples for SAR target recognition: Attack and defense.  IEEE Trans. Geosci. Remote Sens. 60, 5236217.
  * Ren et al. (2021) Ren, H., Yu, X., Zou, L., Zhou, Y., Wang, X., Bruzzone, L., 2021.  Extended convolutional capsule network with application on SAR automatic target recognition.  Signal Process. 183, 108021\.
  * Schölkopf (2022) Schölkopf, B., 2022.  Causality for Machine Learning.  Association for Computing Machinery, New York, NY, USA.
  * Simonyan and Zisserman (2014) Simonyan, K., Zisserman, A., 2014\.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 .
  * Sun et al. (2022) Sun, X., Lv, Y., Wang, Z., Fu, K., 2022.  SCAN: Scattering characteristics analysis network for few-shot aircraft classification in high-resolution SAR images.  IEEE Trans. Geosci. Remote Sens. 60, 5226517.
  * Sun et al. (2019) Sun, Y., Wang, Y., Liu, H., Wang, N., Wang, J., 2019\.  SAR target recognition with limited training data based on angular rotation generative network.  IEEE Geosci. Remote Sens. Lett. 17, 1928–1932.
  * Wang et al. (2022) Wang, C., Huang, Y., Liu, X., Pei, J., Zhang, Y., Yang, J., 2022.  Global in local: A convolutional transformer for SAR ATR FSL.  IEEE Geosci. Remote Sens. Lett. 19, 4509605.
  * Wang et al. (2021) Wang, Z., Du, L., Li, Y., 2021\.  Boosting lightweight CNNs through network pruning and knowledge distillation for SAR target recognition.  IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 14, 8386–8397.
  * Wen et al. (2021) Wen, Z., Liu, Z., Zhang, S., Pan, Q., 2021.  Rotation awareness based self-supervised learning for SAR target recognition with limited training samples.  IEEE Trans. Image Process. 30, 7266–7279.
  * Zhang et al. (2020) Zhang, M., An, J., Yang, L.D., Wu, L., Lu, X.Q., et al., 2020\.  Convolutional neural network with attention mechanism for SAR automatic target recognition.  IEEE Geosci. Remote Sens. Lett. 19, 4004205.
  * Zhang et al. (2018) Zhang, X., Wang, Y., Li, D., Tan, Z., Liu, S., 2018\.  Fusion of multifeature low-rank representation for synthetic aperture radar target configuration recognition.  IEEE Geosci. Remote Sens. Lett. 15, 1402–1406.
  * Zhou et al. (2016) Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A., 2016.  Learning deep features for discriminative localization, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 2921–2929.
  * Zhou et al. (2019) Zhou, Y., Chen, T., Tian, J., Zhou, Z., Wang, C., Yang, X., Shi, J., 2019.  Complex background SAR target recognition based on convolution neural network, in: Proc. IEEE 6th Asia-Pacific Conf. Synthetic Aperture Radar (APSAR), pp. 1–4.
  * Zhu et al. (2021) Zhu, X.X., Montazeri, S., Ali, M., Hua, Y., Wang, Y., Mou, L., Shi, Y., Xu, F., Bamler, R., 2021.  Deep learning meets SAR: Concepts, models, pitfalls, and perspectives.  IEEE Geosci. Remote Sens. Mag. 9, 143–172.
  * Zou et al. (2022) Zou, B., Qin, J., Zhang, L., 2022.  Vehicle detection based on semantic-context enhancement for high-resolution SAR images in complex background.  IEEE Geosci. Remote Sens. Lett. 19, 4503905.


# Weak Measurement of Berry’s Phase

Z. Gedik<EMAIL_ADDRESS>Faculty of Engineering and Natural Sciences,
Sabanci University, Tuzla, Istanbul 34956, Turkey

###### Abstract

Quantum measurements can be generalized to include complex quantities. It is
possible to relate the quantum weak values of projection operators to the
third order Bargmann invariants. The argument of the weak value becomes, up to
a sign, equal to the Berry’s phase associated with the three state vectors. In
case of symmetric informationally complete, positive operator valued measures
(SIC-POVMs), this relation takes a particularly simple form. Alternating
strong and weak measurements can be used to determine Berry’s phase directly,
which demonstrates that not only their real and imaginary parts but also
moduli and arguments of weak values have a physical significance. For an
arbitrary projection operator, weak value is real when the projector, pre- and
post-selected states lie on a so-called null phase curve which includes the
geodesic containing the three states as a special case.

weak measurement, weak value, Bargmann invariant, geometric phase, Berry
phase, SIC-POVM

###### pacs:

03.65.Ta, 03.65.Vf, 03.67.-a, 42.50.Dv

Quantum weak values have been first introduced by Aharonov, Albert, and
Vaidman in the context of weak measurements Aharonov . It has been argued that
the process of obtaining a weak value might not be called as a quantum
measurement Leggett ; Peres ; Reply . Independent of discussions on the
relation between weak measurement and notion of standard measuring procedure,
weak values are well defined quantities which can be beyond the range of
eigenvalues (greater than the maximum or smaller than the minimum)and they can
be complex Vaidman .

Unitary evolution implied by the Schrödinger equation allows us to evaluate
not only the future quantum state vectors but also the past ones. In other
words, forward and backward evolutions are on equal footing. On the other
hand, because of the collapse phenomenon, measurement is an irreversible
process where there is no time symmetry. The collapse destroys the information
about the initial state. Aharonov, Bergmann, and Lebowitz developed a theory
to formulate quantum process of measurement in a time symmetric way Aharonov2
. For this purpose, they introduced two state vectors instead of a single one.
Having an initial state (or so-called pre-selected state), we can obtain an
ensemble subset by choosing a particular final (so-called post-selected)
state. Additional information coming from the post-selected state leads to
time symmetry similar to the unitary evolution case. Physics of the system
during the time interval in-between can be described by a forward evolving
pre-selected and a backward evolving post-selected states. That is why time-
symmetric formulation is also know as the two-state vector formalism of
quantum mechanics.

In this work, we present an interpretation of weak values where instead of
real and imaginary parts, their moduli and arguments are considered. It turns
out that, the argument of the weak value can be related to Berry’s phase Berry
. More precisely, for measurement of projection operators, the argument
becomes nothing but the phase of a third order Bargmann invariant Bargmann .
Symmetric informationally complete, positive operator valued measures (SIC-
POVMs) can used to demonstrate experimentally the suggested measurement method
Jauch ; Renes ; Zauner . In case of projection operators, we show that weak
value is real if the projector, pre- and post-selected states are on a so-
called null phase curve where the geodesic connecting the two states is a
special case Rabei .

The concept of weak values has been invented to obtain information about the
system between pre- and post-selected states. It is possible to use the von
Neumann model of measurement in the weak interaction limit vonNeumann . The
system and the measurement apparatus become in contact to obtain some
information about the system and yet the post-selected state is preserved
Aharonov . The basic idea of weak measurement is to induce a weak coupling
between a system observable and a quantum pointer so that the time evolution
of the state vector is minimally modified. The coupling can be made weak
enough to have the same outcome probabilities of the final state relative to
the no coupling situation and yet an information about the system between
initial $\mid\Psi\rangle$ and final $\mid\Phi\rangle$ states can be obtained
characterized by a quantity called weak value Aharonov

$A_{w}=\frac{\langle\Phi\mid A\mid\Psi\rangle}{\langle\Phi\mid\Psi\rangle}.$
(1)

If the overlap $\langle\Phi\mid\Psi\rangle$ between the pre- and post-selected
states is small enough, the weak value can be arbitrarily large, beyond the
maximum eigenvalue of $A$. Furthermore, $A_{w}$ can be a complex number.
Derivation of weak value expression also gives hint on possible interpretation
of complex values. It has been argued that the imaginary part of the weak
value does not provide information about the observable but instead it
describes how the initial state would be unitarily disturbed Dressel .

According to the measurement axiom, measurement of a dynamical variable $A$
results in collapse of the state vector $\mid\Psi\rangle$ on one of the
eigenvectors $\mid a\rangle$ of $A$ with eigenvalue $a$. Since, Born rule
assigns the probability $\mid\langle a\mid\Psi\rangle\mid^{2}$ for this event,
the mean or the expectation value is $\langle A\rangle=\sum_{a}a\mid\langle
a\mid\Psi\rangle\mid^{2}$. Interestingly, by noting that $a=\langle a\mid
A\mid\Psi\rangle/\langle a\mid\Psi\rangle$, the same expression can be written
in terms of an arbitrary basis rather than $\\{\mid a\rangle\\}$. For example,
for a hermitian operator $B$ with eigenvectors $\mid b\rangle$,

$\langle A\rangle=\sum_{b}\frac{\langle b\mid A\mid\Psi\rangle}{\langle
b\mid\Psi\rangle}\mid\langle b\mid\Psi\rangle\mid^{2}$ (2)

suggesting that $\langle b\mid A\mid\Psi\rangle/\langle b\mid\Psi\rangle$,
which is nothing but the weak value for pre-selected $\mid\Psi\rangle$ and
post-selected $\mid b\rangle$ states, is analogues to eigenvalue $a$ Aharonov3
; Tollaksen . If we relax the rule that measurement of $A$ results in a state
vector in the set $\\{\mid a\rangle\\}$, we can generalize the measurement
axiom to include complex outcomes or the weak values. For example, let us
consider a spin-1/2 particle in spin up state along $z$ direction. If we
measure its spin along $x$ direction ($A=\sigma_{x}$), we find $+1$ or $-1$
with probabilities 1/2 which gives zero mean value. If we consider the same
measurement with $B=\sigma_{y}$, corresponding values are, with equal
probabilities, $-i$ and $i$ which again results in vanishing expectation
value. Equation (2) suggests that $A_{w}$ can be replaced by its real part. A
more careful treatment of the problem by considering a system weakly
interacting with a measurement device shows that the pointer of the latter is
displaced by an amount given by the real part of the weak value Aharonov . It
is also possible to interpret a weak measurement as a positive operator valued
measure where the final state is not necessarily an eigenstate of the measured
dynamical value Jauch .

Interpretation of complex values has been an important problem since the early
days of quantum mechanics. Obtaining probabilities by evaluating modulus
squares of complex probability amplitudes is the most well known example. In
this work, we present an alternative interpretation of complex weak values by
using their polar forms.

When $A$ is a projection operator so that it is of the form
$A=\mid\Omega\rangle\langle\Omega\mid$, after multiplying the numerator and
the denominator by $\langle\Psi\mid\Phi\rangle$, the weak value becomes

$A_{w}=\frac{\langle\Phi\mid\Omega\rangle\langle\Omega\mid\Psi\rangle\langle\Psi\mid\Phi\rangle}{\mid\langle\Phi\mid\Psi\rangle\mid^{2}}=\frac{\Delta_{3}(\Phi,\Omega,\Psi)}{\mid\langle\Phi\mid\Psi\rangle\mid^{2}}.$
(3)

Multiplication of the numerator and the denominator of $A_{w}$ by
$\langle\Psi\mid\Phi\rangle$ is a trivial mathematical operation but the
resulting expression has an important physical interpretation. The cyclic
multiplication of inner products of state vectors
$\Delta_{3}(\Phi,\Omega,\Psi)$ in the numerator of $A_{w}$ is a typical
example for the multi-vertex invariants introduced by Bargmann in his
alternative proof of the Wigner theorem and they are known as Bargamann
invariants Bargmann . It is clear that Bargmann invariants do not change under
unitary operations while they are mapped to their complex conjugates by anti-
unitary transformations. After Berry’s discovery, it has been understood that
the argument of Bargamann invariant has a physical significance which is now
commonly known as geometric phase. We note that the same definition of phase
relation between nonorthogonal states has been introduced by Pancharatnam in
optics Pancharatnam . Possible relation between Bargmann invariant and
Pancharatnam phases for the special case of two state systems has been studied
Tamate . Pancharatnam’s geometric phase has been associated with the phase of
complex weak values Sjoqvist . Geometric phase interpretation of arguments of
complex values in a quantum eraser interference experiment has also been
discussed Cormann .

It has been shown that the argument of the Bargmann invariant, like
$\Delta_{3}(\Phi,\Omega,\Psi)$ in the numerator of $A_{w}$, is negative of the
geometric phase corresponding to the three geodesics connecting the three
state vectors Mukunda ; Hangan . Therefore,

$\arg A_{w}=\arg\Delta_{3}(\Phi,\Omega,\Psi)=-\phi_{g}(\Phi,\Omega,\Psi)$ (4)

where $\phi_{g}(\Phi,\Omega,\Psi)$ denotes the geometric phase corresponding
to the geodesic triangle with vertices $\mid\Phi\rangle$, $\mid\Omega\rangle$,
and $\mid\Psi\rangle$. More generally, the phase of an $n-$vertex Bargmann
invariant is identical to the geometric phase for a closed ray-space curve
connecting the $n$ state vectors with geodesics. Since, n-vertex structures
can be triangularized, three-vertex invariants can be thought of as the
primitive building blocks of the geometric phase. It is also possible to
interpret this observation as another manifestation of the fact that weak
values respect sums, but not products Hall . We note that exchange of the pre-
and the post-selected states results in complex conjugation of the weak value.
Hence, time-reversal operation leads to simply a change in sign of Berry’s
phase.

When an operator $A$ acts on a state vector $\mid\Psi\rangle$, the resulting
state can be decomposed into two pieces

$A\mid\Psi\rangle=\langle A\rangle\mid\Psi\rangle+\Delta
A\mid\Psi^{\bot}\rangle$ (5)

where $\Delta A^{2}=\langle A^{2}\rangle-\langle A\rangle^{2}$ and
$\mid\Psi^{\bot}\rangle$ is a state vector perpendicular to $\mid\Psi\rangle$
Aharonov4 . Therefore, the weak value can be written as

$A_{w}=\langle A\rangle+\Delta
A\frac{\langle\Phi\mid\Psi^{\bot}\rangle}{\langle\Phi\mid\Psi\rangle}$ (6)

and hence imaginary part of $A_{w}$ is dis proportional to the ratio
${\langle\Phi\mid\Psi^{\bot}\rangle}/{\langle\Phi\mid\Psi\rangle}$. We note
the the modulus of this ratio is nothing but $(1-p)/p$ where
$p=\mid\langle\Phi\mid\Psi\rangle\mid^{2}$ is the probability of obtaining
post-selected state $\mid\Phi\rangle$ for pre-selected state
$\mid\Psi\rangle$.

The idea of positive operator valued measures is quite old Jauch . However,
the symmetric and informationally complete case has been a subject of great
interest because of the recent developments in quantum information theory
Renes ; Zauner . Density matrix of a $d-$dimensional system has $d^{2}-1$ free
parameters. One of the main motivations of SIC-POVMs is to determine these
parameters in an optimal way. This task is achieved by entangling the
$d-$dimensional system with a $d^{2}-$dimensional one so that projective
measurements performed on the latter leads to collapse of the former on
equiangular states, _i.e._ state vectors having the same inner product moduli.
In other words, by using SIC-POVMs, arbitrary states of $d-$dimensional system
are collapsed on one of the $d^{2}$ equiangular state vectors in contrast to
standard projective measurements where only $d$ orthogonal final states are
possible. The completeness relation imply that equiangular vectors
$\mid\psi_{i}\rangle$ must satisfy
$\mid\langle\psi_{i}\mid\psi_{j}\rangle\mid=1/\sqrt{d+1}$ for $i\neq j$. In
1999, Zauner conjectured that it is always possible to find $d^{2}$ such
vectors in $d$ dimension and this is still an open problem Zauner .
Introducing the projection operators
$\Pi_{i}=\mid\psi_{i}\rangle\langle\psi_{i}\mid$, we can write the overlap
condition as $\mathrm{Tr}\Pi_{i}\Pi_{j}=1/(d+1)$.

Now, let us consider a weak measurement where pre- and post-selected states
are $\mid\Psi\rangle=\mid\psi_{i}\rangle$ and
$\mid\Phi\rangle=\mid\psi_{k}\rangle$, respectively. Let us further assume
that
$A=\mid\Omega\rangle\langle\Omega\mid=\mid\psi_{j}\rangle\langle\psi_{j}\mid$,
Since, all of the inner products have the same modulus, the weak value takes
the form

$A_{w}=\frac{e^{i\theta_{ijk}}}{\sqrt{d+1}}$ (7)

where $\theta_{ijk}\in[0,2\pi)$. We note that while the initial
$\mid\Psi\rangle=\mid\psi_{i}\rangle$ and the final
$\mid\Phi\rangle=\mid\psi_{k}\rangle$ states are fixed by projective (strong)
measurements, $\mid\psi_{j}\rangle$ is determined by a weak measurement. In
practice, SIC-POVMs can be applied on the system with strong-weak-strong
measurement pattern and weak values corresponding to triples with different
$i$, $j$ and $k$ values can be used to evaluate $\theta_{ijk}$.

The simplest case to apply the suggested measurement procedure is two level
systems where $\mathrm{Tr}\Pi_{i}\Pi_{j}=1/3$ for any pair of 4 SIC-POVM
vectors forming the vertices of a regular tetrahedron in the Bloch sphere.
Since there are 4 possible triplets, one might expect 4 different
$\theta_{ijk}$ values. However, symmetry of the tetrahedron results in, apart
from sign changes coming from permutations, a single value which is
$\theta_{ijk}=\pi/2$. This special case is interesting in that all weak values
are purely imaginary. Therefore, measuring device reads a null value. In three
dimensions, it can be shown that 84 triple combinations (28 choices) results
in 5 different values Zhu .

For SIC-POVMs, Bargmann invariants
$\mathrm{Tr}\Pi_{i}\Pi_{j}\Pi_{k}=e^{i\theta_{ijk}}/(d+1)^{3/2}$ become
completely symmetric 3-index tensors and they play a central role in
representation of density matrices. A density operator
$\rho=\sum_{i}\lambda_{i}\Pi_{i}$ describes a pure state if and only if
$\mathrm{Tr}\rho^{2}=\mathrm{Tr}\rho^{3}=1$ Fuchs . Therefore along with
$\Sigma_{i}\lambda_{i}=\Sigma_{i}\lambda_{i}^{2}=1$,the equation

$\Sigma_{i}\lambda_{i}^{3}+\frac{6}{d\sqrt{d+1}}\Sigma_{\\{i,j,k\\}}\cos\theta_{\\{i,j,k\\}}\lambda_{i}\lambda_{j}\lambda_{k}=1$
(8)

determines all the valid expansion coefficient sets $\\{\lambda_{i}\\}$
corresponding to a pure state. Here, the sum is over triple combinations
$\\{i,j,k\\}$ where all indices are different from each other. Since, we are
summing over combinations, $\\{i,j,k\\}$ and $\\{j,i,k\\}$ denote the same
term. Six possible permutations contribute only two different values
(depending on wheter the permutation is even or odd) and therefore we can
combine them in a single cosine term. Clearly, the angles
$\theta_{\\{i,j,k\\}}$ are not independent. The relations among
$\theta_{ijk}$’s restrict $\theta_{\\{i,j,k\\}}$’s, too. For example,
$\theta_{123}\theta_{134}\theta_{124}=\theta_{234}$ implies a similar relation
for $\theta_{\\{i,j,k\\}}$’s. Obtaining the corresponding probabilities
$p_{i}$ from the expansion coefficients $\lambda_{i}$ is straightforward from
the relation $\lambda_{i}=(d+1)p_{i}-1/d$ Fuchs .

For an arbitrary projection operator $A=\mid\Omega\rangle\langle\Omega\mid$,
we can ask if the weak value $A_{w}$ is real. This is equivalent to looking
for vanishing Bargmann invariant phase or geometric phase. Null phase curves
have been introduced in search of a generalized connection between Bargmann
invariants and geometric phases Rabei . Geometric phase vanishes when three
vertices lie on a null phase curve. This is also the case when the curve is a
geodesic. Null phase condition is necessary for geodesics but not sufficient.
It is possible to generalize this definition to arbitrary phase values. In
this case, each constant phase curve connecting the pre- and the post-selected
states will describe the projection operators giving that particular weak
value argument.

In conclusion, arguments of quantum weak values can be related to geometric
phases in a very simple way. Therefore, not only their real and imaginary
parts but also moduli and arguments of weak values have a physical
significance, the latter being a direct measure of Berry’s phase. In
particular, alternating projective and weak SIC-POVMs can be used to determine
the angular relations among projection operators. For a projection operator,
the question of when a weak value is a real number turns out to be equivalent
to having real Bargmann invariants.

The author acknowledges helpful discussions with G. B. Bağcı, O. Pusuluk, and
V. Vedral.

## References

  * (1) Y. Aharonov, D. Z. Albert, and L. Vaidman, Phys. Rev. Lett. 60, 1351 (1988).
  * (2) A. J. Leggett, Phys. Rev. Lett. 62, 2325 (1989).
  * (3) A. Peres, Phys. Rev. Lett. 62, 2326 (1989).
  * (4) Y. Aharonov and L. Vaidman, Aharonov and Vaidman Reply, Phys. Rev. Lett. 62, 2327 (1989).
  * (5) L. Vaidman, A. Ben-Israel, J. Dziewior, L. Knips, M. Weißl, J. Meinecke, C. Schwemmer, R. Ber, and H. Weinfurter, Phys. Rev. A 96, 032114 (2017).
  * (6) Y. Aharonov, P. G. Bergmann, and J. L. Lebowitz, Phys. Rev. 134, B1410 (1964).
  * (7) M. V. Berry, Proc. R. Soc. Lond. A 392, 45 (1984).
  * (8) J. M. Jauch and C. Piron, Helv. Phys. Acta 45, 559 (1967).
  * (9) V. Bargmann, J. Math. Phys. 5, 862 (1964).
  * (10) J. M. Renes, R. Blume-Kohout, A. J. Scott, and C. M. Caves, J. Math. Phys. 45, 2171, (2004).
  * (11) G. Zauner, Int. J. Quantum Inf. 9, 445 (2011).
  * (12) E. M. Rabei, Arvind, N. Mukunda, and R. Simon Phys. Rev. A 60, 3397 (1999).
  * (13) J. von Neumann, _Mathematical Foundations of Quantum Mechanics_ , Princeton Univ. Press, Princeton, N.J., 1955.
  * (14) J. Dressel and A. N. Jordan, Phys. Rev. A 85, 012107 (2012).
  * (15) Y. Aharonov and A. Botero, Phys. Rev. A 92, 052111 (2005).
  * (16) J. Tollaksen, Y. Aharonov, A. Casher, T. Kaufherr, and S. Nussinov, New J. Phys. 12, 013023 (2010).
  * (17) S. Pancharatnam, Proc. Ind. Acad. Sci. A 44, 247 (1956).
  * (18) S. Tamate, H. Kobayashi, T. Nakanishi, K. Sugiyama, and M. Kitano, New J. Phys. 11, 093025 (2009).
  * (19) E. Sjöqvist, Phys. Lett. A 359, 187 (2006).
  * (20) M. Cormann, M. Remy, B. Kolaric, and Y. Caudano, Phys. Rev. A 93, 042124 (2016).
  * (21) N. Mukunda and R. Simon, Ann. Phys. 228, 205(1993).
  * (22) TH. Hangan and G. Masala, Geometriea Dedicata 49, 129 (1994).
  * (23) M. J. W. Hall, A. K. Pati, and J. Wu, Phys. Rev. A 93, 052118 (2016).
  * (24) Y. Aharonov and L. Vaidman, Phys. Rev. A 41, 11 (1990).
  * (25) H. Zhu, J. Phys. A: Math. Theor. 43, 305305 (2010).
  * (26) C. A. Fuchs and R. Shack, Rev. Mod. Phys. 85, 1693 (2013).