# Visualization of Supervised and Self-Supervised Neural Networks
via Attribution Guided Factorization

Shir Gur, Ameen Ali, Lior Wolf

###### Abstract

Neural network visualization techniques mark image locations by their
relevancy to the network’s classification. Existing methods are effective in
highlighting the regions that affect the resulting classification the most.
However, as we show, these methods are limited in their ability to identify
the support for alternative classifications, an effect we name the saliency
bias hypothesis. In this work, we integrate two lines of research: gradient-
based methods and attribution-based methods, and develop an algorithm that
provides per-class explainability. The algorithm back-projects the per pixel
local influence, in a manner that is guided by the local attributions, while
correcting for salient features that would otherwise bias the explanation. In
an extensive battery of experiments, we demonstrate the ability of our methods
to class-specific visualization, and not just the predicted label. Remarkably,
the method obtains state of the art results in benchmarks that are commonly
applied to gradient-based methods as well as in those that are employed mostly
for evaluating attribution methods. Using a new unsupervised procedure, our
method is also successful in demonstrating that self-supervised methods learn
semantic information. Our code is available at:
https://github.com/shirgur/AGFVisualization.

##  1 Introduction

|  |  |  |  |
---|---|---|---|---|---
(a) | (b) | (c) | (d) | (e) | (f)
Figure 1: Visualizations by our method for a pre-trained VGG-19. (a,d) input
images. (b,e) the heatmap generated for the top label. (c,f) same for the 2nd
highest prediction.

The most common class of explainability methods for image classifiers
visualize the reason behind the classification of the network as a heatmap.
These methods can make the rationale of the decision accessible to humans,
leading to higher confidence in the ability of the classifier to focus on the
relevant parts of the image, and not on spurious associations, and help debug
the model. In addition to the human user, the “computer user” can also benefit
from such methods, which can seed image segmentation techniques (Ahn, Cho, and
Kwak 2019; Huang et al. 2018; Wang et al. 2019; Hoyer et al. 2019), or help
focus generative image models, among other tasks.

The prominent methods in the field can be divided into two families: (i)
gradient-based maps, which consider the gradient signal as it is computed by
the conventional backpropagation approach (Sundararajan, Taly, and Yan 2017;
Smilkov et al. 2017; Srinivas and Fleuret 2019; Selvaraju et al. 2017) and
(ii) relevance propagation methods (Bach et al. 2015; Nam et al. 2019; Gu,
Yang, and Tresp 2018; Iwana, Kuroki, and Uchida 2019), which project high-
level activations back to the input domain, mostly based on the deep Taylor
decomposition by (Montavon et al. 2017). The two families are used for
different purposes, and are evaluated by different sets of experiments and
performance metrics. As we show in Sec. 5, both types of methods have
complementary sets of advantages and disadvantages: gradient based methods,
such as Grad-CAM, are able to provide a class specific visualization for deep
layers, but fail to do so for the input image, and also provide a unilateral
result. In contrast, attribution based methods, excel in visualizing at the
input image level, and have bipartite results, but lack in visualizing class
specific explanations.

We present a novel method for class specific visualization of deep image
recognition models. The method is able to overcome the limitations of the
previous work, by combining ideas from both families of methods, and
accumulating across the layers both gradient information, and relevance
attribution. The method corrects for what we term the saliency bias. This bias
draws the attention of the network towards the salient activations, and can
prevent visualizing other image objects. This has led to the claims that
visualization methods mimic the behavior of edge detectors, and that the
generated heatmaps are largely independent of the network weights (Adebayo et
al. 2018).

There are two different questions that explainability method often tackle: (i)
which pixels affect classification the most, and (ii) which pixels are
identified as belonging to the predicted class. Our method answers the second
one and outperforms the relevant literature methods in multiple ways. First,
the locations we identify are much more important to the classification
outcome than those of the baselines, when looking inside the region of a
target class. Second, we are able to identify regions of multiple classes in
each image, and not just the prominent class, see Fig. 1. Our method greatly
outperforms recent multi-class work (Gu, Yang, and Tresp 2018; Iwana, Kuroki,
and Uchida 2019).

The main contributions of our work are: (1) a novel explainability method,
that combines both gradient and attribution techniques, as well as a new
attribution guided factorization technique for extracting informative class-
specific attributions from the input feature map and its gradients. (2) we
point to and correct, for the first time, for the saliency bias. This bias
hindered all previous attempts to explain more than the top decision, and is
the underlying reason for the failures of explainability methods demonstrated
in the literature. (3) state-of-the-art performance in both negative
perturbation and in segmentation-based evaluation. The former is often used to
evaluate attribution methods, and the latter is often used to evaluate
gradient-based methods. (4) using a novel procedure, we show for the first
time as far as we can ascertain, that self-supervised networks implicitly
learn semantic segmentation information.

##  2 Related work

Many explainability methods belong to one of two classes: attribution and
gradient methods. Both aim to explain and visualize neural networks, but
differ in their underlying concepts. A summary of these methods with their
individual properties is presented in Tab. 1.

Methods outside these two classes include those that generate salient feature
maps (Dabkowski and Gal 2017; Simonyan, Vedaldi, and Zisserman 2013; Mahendran
and Vedaldi 2016; Zhou et al. 2016; Zeiler and Fergus 2014; Zhou et al. 2018),
Activation Maximization (Erhan et al. 2009) and Excitation Backprop (Zhang et
al. 2018). Extremal Perturbation methods (Fong, Patrick, and Vedaldi 2019;
Fong and Vedaldi 2017) are applicable to black box models, but suffer from
high computational complexity. Shapley-value based methods (Lundberg and Lee
2017), despite their theoretical appeal, are known to perform poorly in
practice. Therefore, while we compare empirically with several Shaply and
perturbation methods, we focus on the the newer gradient and attribution
methods.

Attribution propagation methods follow the Deep Taylor Decomposition (DTD)
method, of Montavon et al. (2017), which decompose the network classification
decision into the contributions of its input elements. Following this line of
work, methods, such as Layer-wise Relevance Propagation (LRP)by Bach et al.
(2015), use DTD to propagate relevance from the predicated class, backward, to
the input image, in neural networks with a rectified linear unit (ReLU) non-
linearity. The PatterNet and PaternAttribution (Kindermans et al. 2017)
methods yield similar results to LRP. A disadvantage of LRP, is that it is
class agnostic, meaning that propagating from different classes yields the
same visualization. Contrastive-LRP (CLRP) by Gu, Yang, and Tresp (2018) and
Softmax-Gradient-LRP (SGLRP) by Iwana, Kuroki, and Uchida (2019) use LRP to
propagate results of the target class and, in contrast to all other classes,
in order to produce a class specific visualization. Nam et al. (2019)
presented RAP, a DTD approach that partitions attributions to positive and
negative influences, following a mean-shift approach. This approach is class
agnostic, as we demonstrate in Sec. 5. Deep Learning Important FeaTures
(DeepLIFT) (Shrikumar, Greenside, and Kundaje 2017) decomposes the output
prediction, by assigning the differences of contribution scores between the
activation of each neuron to its reference activation.

Gradient based methods use backpropagation to compute the gradients with
respect to the layer’s input feature map, using the chain rule. The
Gradient*Input method by Shrikumar et al. (2016) computes the (signed) partial
derivatives of the output with respect to the input, multiplying it by the
input itself. Integrated Gradients (Sundararajan, Taly, and Yan 2017), similar
to Shrikumar et al. (2016), computes the multiplication of the inputs with its
derivatives, only they compute the average gradient while performing a linear
interpolation of the input, according to some baseline that is defined by the
user. SmoothGrad by Smilkov et al. (2017), visualize the mean gradients of the
input, while adding to the input image a random Gaussian noise at each
iteration. The FullGrad method by Srinivas and Fleuret (2019) suggests
computing the gradients of each layer’s input, or bias, followed by a post-
processing operator, usually consisting of the absolute value, reshaping to
the size of the input, and summing over the entire network. Where for the
input layer, they also multiply by the input image before post-processing. As
we show in Sec. 5, FullGrad produces class agnostic visualizations. On the
other hand, Grad-CAM by Selvaraju et al. (2017) is a class specific approach,
combining both the input features, and the gradients of a network’s layer.
This approach is commonly used in many applications due to this property, but
its disadvantage rests in the fact that it can only produce results for very
deep layers, resulting in coarse visualization due to the low spatial
dimension of such deep layers.

| Int. | Smooth | LRP | LRPαβ | Full | Grad | RAP | CLRP | SGLRP | Ours
---|---|---|---|---|---|---|---|---|---|---
| Grad | Grad | Grad | CAM
Gradients | ✓ | ✓ |  |  | ✓ | ✓ |  |  |  | ✓
Attribution |  |  | ✓ | ✓ |  |  | ✓ | ✓ | ✓ | ✓
Class Specific |  |  |  |  |  | ✓ |  | ✓ | ✓ | ✓
Input Domain | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓
Table 1: Properties of various visualization methods, which can be largely
divided into gradient based and attribution based. Most methods are not class
specific, and except for Grad-CAM, all methods project all the way back to the
input image.

##  3 Propagation methods

We define the building blocks of attribution propagation and gradient
propagation that are used in our method.

Attribution Propagation: Let $x^{(n)}$, $\theta^{(n)}$ be the input feature
map and weights of layer $L^{(n)}$, respectively, where $n\in[1\dots N]$ is
the layer index in a network, consisting of $N$ layers. In this field’s
terminology, layer $n-1$ is downstream of layer $n$, layer $N$ processes the
input, while layer $1$ produces the final output.

Let $x^{(n-1)}=L^{(n)}(x^{(n)},\theta^{(n)})$ to be the result of applying
layer $L^{(n)}$ on $x^{(n)}$. The relevancy of layer $L^{(n)}$ is given by
$R^{(n)}$ and is also known as the attribution.

###### Definition 1

(Montavon et al. 2017) The generic attribution propagation rule is defined,
for two tensors, $\mathbf{X}$ and $\Theta$, as:

$\displaystyle R^{(n)}_{j}$
$\displaystyle=\mathcal{G}_{j}^{(n)}(\mathbf{X},\Theta,R^{(n-1)})$ (1)
$\displaystyle=\sum_{i}\mathbf{X}_{j}\frac{\partial
L^{(n)}_{i}(\mathbf{X},\Theta)}{\partial\mathbf{X}_{j}}\frac{R^{(n-1)}_{i}}{\sum_{i^{\prime}}L^{(n)}_{i^{\prime}}(\mathbf{X},\Theta)}$

Typically, $\mathbf{X}$ is related to the layer’s input $x^{(n)}$, and
$\Theta$ to its weights $\theta^{(n)}$. LRP (Binder et al. 2016) can be
written in this notation by setting $\mathbf{X}={x}^{(n)+}$ and
$\Theta={\theta}^{(n)+}$, where $\tau^{+}=\max(0,\tau)$ for a tensor $\tau$.
Note that Def. 1 satisfies the conservation rule:

$\displaystyle\sum_{j}R^{(n)}_{j}=\sum_{i}R^{(n-1)}_{i}$ (2)

Let $y$ be the output vector of a classification network with $\mathcal{C}$
classes, and let $y^{t}$ represent the specific value of class
$t\in\mathcal{C}$. LRP defines $R^{(0)}\in\mathbb{R}^{|\mathcal{C}|}$ to be
the a zeros vector, except for index $t$, where $R_{t}^{(0)}=y^{t}$.
Similarly, the CLRP (Gu, Yang, and Tresp 2018) and SGLRP (Iwana, Kuroki, and
Uchida 2019) methods calculate the difference between two LRP results,
initialized with two opposing $R^{(0)}$ for “target” and “rest”, propagating
relevance from the “target” class, and the “rest” of the classes, e.g.CLRP is
defined as:

$\displaystyle CLRP$
$\displaystyle=R^{(N)}_{tgt}-\mathcal{N}\big{(}R^{(N)}_{rst},R^{(N)}_{tgt}\big{)}$
(3)

where $R^{(0)}_{tgt}=R^{(0)}$, $R^{(0)}_{rst}=(y-R^{(0)})/(|\mathcal{C}|-1)$,
and $\mathcal{N}$ is a normalization term $\mathcal{N}(a,b)=a\frac{\sum
b}{\sum a}$.

$\Delta$-Shift: Def. 1 presented a generic propagation rule that satisfies the
conservation rule in Eq. 2. However, in many cases, we would like to add a
residual signal denoting another type of attribution. The $\Delta$-Shift
corrects for the deviation from the conservation rule.

###### Definition 2

Given a generic propagation result $\mathcal{G}^{(n)}$, following Eq. 2, and a
residual tensor $\mathbf{r}^{(n)}$, the $\Delta$-Shift is defined as follows:

$\displaystyle\Delta^{(n)}_{\text{\em
shift}}(\mathcal{G}^{(n)},\mathbf{r}^{(n)})=\mathcal{G}^{(n)}+\mathbf{r}^{(n)}-\frac{\sum\mathbf{r}^{(n)}}{\sum\mathbb{1}_{\mathcal{G}^{(n)}\neq
0}}$ (4)

Note that we divide the sum of the residual signal by the number of non-zero
neurons. While not formulated this way, the RAP method (Nam et al. 2019)
employs this type of correction defined in Def. 2.

Gradient Propagation: The propagation in a neural network is defined by the
chain rule.

###### Definition 3

Let $\mathcal{L}$ be the loss of a neural network. The input feature
gradients, $x^{(n)}$ of layer $L^{(n)}$, with respect to $\mathcal{L}$ are
defined by the chain rule as follows:

$\displaystyle\nabla x_{j}^{(n)}:=\frac{\partial\mathcal{L}}{\partial
x_{j}^{(n)}}$ $\displaystyle=\sum_{i}\frac{\partial\mathcal{L}}{\partial
x_{i}^{(n-1)}}\frac{\partial x_{i}^{(n-1)}}{\partial x_{j}^{(n)}}$ (5)

Methods such as FullGrad (Srinivas and Fleuret 2019) and SmoothGrad (Smilkov
et al. 2017) use the raw gradients, as defined in Eq.5, for visualization.
Grad-CAM (Selvaraju et al. 2017), on the other hand, performs a weighted
combination of the input feature gradients, in order to obtain a class
specific visualization, defined as follows:

$\displaystyle\text{Grad-CAM}\big{(}x^{(n)},\nabla x^{(n)}\big{)}$
$\displaystyle=\bigg{(}\frac{1}{|C|}\sum_{c\in[C]}x_{c}^{(n)}\sum\limits_{\begin{subarray}{c}h\in[H]\\\
w\in[W]\end{subarray}}\nabla x_{c,h,w}^{(n)}\bigg{)}^{+}$ (6)

where $\nabla x_{c,h,w}^{(n)}$ is the specific value of the gradient
$C$-channel tensor $x^{(n)}$ at channel $c$ and pixel $(h,w)$, and
$x_{c}^{(n)}$ is the entire channel, which is a matrix of size $H\times W$.

Guided Factorization: The explanation should create a clear separation between
the positively contributing regions, or the foreground, and the negatively
contributing ones, referred to as the background. This is true for both the
activations and the gradients during propagation. Ideally, the relevant data
would be partitioned into two clusters — one for positive contributions and
one for the negative contributions. We follow the partition problem (Yuan,
Wang, and Cheriyadat 2015; Gao et al. 2016), in which the data is divided
spatially between positive and negative locations, in accordance with the sign
of a partition map $\phi\in\mathbb{R}^{H\times W}$.

Given a tensor $\mathbf{Y}\in\mathbb{R}^{C\times H\times W}$, we re-write it
as a matrix in the form of $\mathbf{Y}\in\mathbb{R}^{C\times HW}$. We compute
the Heaviside function of $\mathbf{Y}$ using a $\operatorname{sigmoid}$
function: $\mathbf{H}=\operatorname{sigmoid}(\mathbf{Y})$. The matrix
$H\in[0,1]^{C\times HW}$ is a positive-matrix, and we consider the following
two-class non-negative matrix factorization $\mathbf{H}=\mathbf{R}\mathbf{W}$,
where $\mathbf{W}\in{\mathbb{R}^{+}}^{2\times HW}$ contains the spatial mixing
weights, and the representative matrix
$\mathbf{R}=[\mathbf{R}_{b}\,\mathbf{R}_{f}]$, defined by the mean of each
class in the data tensor $\mathbf{H}$ based on the assignment of $\phi$:

$\displaystyle{\mathbf{R}^{f}_{c}}$
$\displaystyle=\frac{\sum\limits_{i}^{HW}\mathbf{H}(\mathbf{Y})_{c,i}\odot\mathbb{1}_{\phi_{i}>0}}{\sum\limits_{i}^{HW}\mathbb{1}_{\phi_{i}>0}}\,,{\mathbf{R}^{b}_{c}}$
$\displaystyle=\frac{\sum\limits_{i}^{HW}\mathbf{H}(\mathbf{Y})_{c,i}\odot\mathbb{1}_{\phi_{i}\leqslant
0}}{\sum\limits_{i}^{HW}\mathbb{1}_{\phi_{i}\leqslant 0}}$

where $\mathbf{R}^{f},\mathbf{R}^{b}\in{\mathbb{R}^{+}}^{C}$, $c\in C$ is the
channel dimension and $\odot$ denotes the Hadamard product.

We estimate the matrix $\mathbf{W}$ of positive weights by least squares
$\mathbf{W}=[\mathbf{W}_{b}\,\mathbf{W}_{f}]=((\mathbf{R}^{\top}\mathbf{R})^{-1}\mathbf{R}^{\top}\mathbf{H})^{+}$,
where $\mathbf{W}_{f},\mathbf{W}_{b}\in{\mathbb{R}^{+}}^{HW}$. Combining the
foreground weights $W_{f}$ with the background weights $W_{b}$ into the same
axis is done by using both negative and positive values, leading to the
following operator:
$\mathcal{F}(\mathbf{Y},\phi)=\mathbf{W}_{f}-\mathbf{W}_{b}$, where
$\mathcal{F}$ is a function that takes $\mathbf{Y}$ and $\phi$, as inputs. We
further normalize $\mathbf{Y}$ using
$\mathcal{N}_{\max}(a)=\frac{a}{\max(a)}$, to allow multiple streams to be
integrated together:

$\displaystyle\bar{\mathcal{F}}(\mathbf{Y},\phi)=\mathcal{F}(\mathcal{N}_{\max}(\mathbf{Y}),\phi)$
(7)

##  4 The Integrated Method

Let $M$ be a multiclass CNN classifier ($\mathcal{C}$ labels), and $I=x^{(N)}$
be the input image. The network $M$ outputs a score vector
$y\in\mathbb{R}^{|\mathcal{C}|}$, obtained before applying the
$\operatorname{softmax}$ operator. Given any target class $t$, our goal is to
explain where (spatially) in $I$ lies the support for class $t$. The method is
composed of two streams, gradients and attribution propagation. Each step, we
use the previous values of the two, and compute the current layer’s input
gradient and attribution.

There are three major components to our method, (i) propagating attribution
results using Def. 1, (ii) factorizing the activations and the gradients in a
manner that is guided by the attribution, and (iii) performing attribution
aggregation and shifting the values, such that the conservation rule is
preserved. The shift splits the neurons into those with a positive and
negative attribution. The complete algorithm is listed as part of the
supplementary material.

###  4.1 Initial Attribution Propagation

As shown in (Gu, Yang, and Tresp 2018; Iwana, Kuroki, and Uchida 2019), the
use of different initial relevance can result in improved results. Let
$\Phi^{(n)}$ and $\Phi^{(n-1)}$ be the output and input class attribution maps
of layer $L^{(n)}$, respectively. We employ the following initial attribution
for explaining decision $t$. Let $y:=x^{(0)}$ be the output vector of the
classification network (logits), we compute the initial attribution
$\Phi^{(1)}$ as:

$\hat{y}=\operatorname{softmax}\big{(}y^{t}\exp{\big{(}-\frac{1}{2}\big{(}\frac{y-y^{t}}{\max\|y-y^{t}\|_{1}}\big{)}^{2}\big{)}}\big{)}\\\
\frac{\partial\hat{y}^{t}}{\partial
x^{(1)}_{j}}=\sum_{i}\frac{\partial\hat{y}^{t}}{\partial y_{i}}\frac{\partial
y_{i}}{\partial
x_{j}^{(1)}}\,,\Phi^{(1)}=x^{(1)}\odot\frac{\partial\hat{y}^{t}}{\partial
x^{(1)}}$ (8)

In this formulation, we replace the psuedo-probabilities of vector $y$ with
another vector, in which class $t$ that we wish to provide an explanation for
is highlighted, and the rest of the classes are scored by the closeness of
their assigned probability to that of $t$. This way, the explanation is no
longer dominated by the predicted class.

| Input | LRP | FullGrad | Grad | RAP | CLRP | SGLRP | Ours
---|---|---|---|---|---|---|---|---
| Image | CAM
(a) |  |  |  |  |  |  |  |
(b) |  |  |  |  |  |  |  |
(c) |  Dog $\rightarrow$ Cat $\rightarrow$ |  |  |  |  |  |  |
|  |  |  |  |  |
Figure 2: (a) A comparison between methods for VGG-19. (b) Same for ResNet-50. (c) Visualization of two different classes for VGG-19. Many more results can be found in the supplementary material. |
---|---
(a) | (b)
Figure 3: Negative perturbation results on the ImageNet validation set of (a)
the predicted and (b) the target class. Show is the change in accuracy, when
removing a fraction of the image according to the attribution value, starting
from lowest to highest.

| Int. | Smooth | LRP | LRPαβ | Full | Grad | RAP | CLRP | SGLRP | Ours
---|---|---|---|---|---|---|---|---|---|---
| Grad | Grad | Grad | CAM
Predicted | 10.4 | 12.0 | 26.8 | 18.8 | 32.1 | 37.8 | 38.7 | 29.8 | 29.9 | 38.9
Target | 10.5 | 12.1 | 26.8 | 18.8 | 32.4 | 39.4 | 38.7 | 31.6 | 32.4 | 40.0

Table 2: Area Under the Curve (AUC) for the two negative perturbation tests,
showing results for predicted and target class. The class-agnostic methods
either perform worse or experience insignificant change on the target class
test.

|  | Gradient | Meaningful | Int. | Smooth | LRP | LRPαβ | Full | Grad | RAP | CLRP | SGLRP | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Dataset | Metric | SHAP | Perturbation | Grad | Grad | Grad | CAM
ImageNet | Pixel Acc. | 49.9 | 29.6 | 72.7 | 70.1 | 74.9 | 29.6 | 75.8 | 66.5 | 78.8 | 53.3 | 54.5 | 79.9
mAP | 50.0 | 45.4 | 67.4 | 65.1 | 69.9 | 47.5 | 70.6 | 62.4 | 73.7 | 54.6 | 55.1 | 76.7
VOC’12 | Pixel Acc. | 9.3 | 10.2 | 70.1 | 69.9 | 66.6 | 70.8 | 26.9 | 52.6 | 72.8 | 72.8 | 73.1 | 77.5
mAP | 35.2 | 22.7 | 34.9 | 34.3 | 40.8 | 37.0 | 19.7 | 41.7 | 39.6 | 37.9 | 32.6 | 45.9

Table 3: Quantitative segmentation results on (a) ImageNet and (b) PASCAL-VOC
2012.

###  4.2 Class Attribution Propagation

As mentioned above, there are two streams in our method. The first propagates
the gradients, which are used for the factorization process we discuss in the
next section, and the other is responsible for propagating the resulting class
attribution maps $\Phi^{(n-1)}$, using DTD-type propagations. This second
stream is more involved, and the computation of $\Phi^{(n)}$ has multiple
steps.

The first step is to propagate $\Phi^{(n-1)}$ through $L^{(n)}$ following Eq.
1, using two variants. Both variants employ $\Phi^{(n-1)}$ as the tensor to be
propagated, but depending on the setting of $\mathbf{X}$ and $\Theta$, result
in different outcomes. The first variant considers the absolute influence
$\mathbf{C}^{(n)}$, defined by:

$\displaystyle\mathbf{C}^{(n)}=\mathcal{G}^{(n)}(|x^{(n)}|,|\theta^{(n)}|,\Phi^{(n-1)})$
(9)

The second variant computes the input-agnostic influence $\mathbf{A}^{(n)}$,
following Eq. 1:

$\displaystyle\mathbf{A}^{(n)}=\mathcal{G}^{(n)}(\mathbb{1},|\theta^{(n)}|,\Phi^{(n-1)})$
(10)

where $\mathbb{1}$ is an all-ones tensor of the shape of $x^{(n)}$. We choose
the input-agnostic propagation because features in shallow layers, such as
edges, are more local and less semantic. It, therefore, reduces the
sensitivity to texture.

###  4.3 Residual Update

As part of the method, we compute in addition to $\mathbf{C}^{(n)}$, the
factorization of both the input feature map of layer $L^{(n)}$ and its
gradients. This branch is defined by the chain rule in Eq. 5, where we now
consider $\mathcal{L}=\hat{y}^{t}$. The factorization results in foreground
and background partitions, using guidance from $\mathbf{C}^{(n)}$. This
partition follows the idea of our attribution properties, where positive
values are part of class $t$, and negatives otherwise. We, therefore, employ
the following attribution guided factorization (Eq. 7 with respect to
$x^{(n)}$ and $\nabla x^{(n)}$) as follows:

$\mathbf{F}^{(n)}_{x}=\bar{\mathcal{F}}\big{(}x^{(n)},\mathbf{C}^{(n)}\big{)}^{+}\,,\quad\mathbf{F}^{(n)}_{\nabla
x}=\bar{\mathcal{F}}\big{(}\nabla x^{(n)},\mathbf{C}^{(n)}\big{)}^{+}$ (11)

note that we only consider the positive values of the factorization update,
and that the two results are normalized by their maximal value. Similarly to
(Sundararajan, Taly, and Yan 2017; Selvaraju et al. 2017), we define the
input-gradient interaction:

$\displaystyle\mathbf{M}^{(n)}_{x\nabla
x}=\mathcal{N}_{\max}\bigg{(}\bigg{(}\frac{1}{|C|}\sum_{c\in[C]}\big{(}x_{c}^{(n)}\odot\nabla
x_{c}^{(n)}\big{)}\bigg{)}^{+}\bigg{)}$ (12)

The residual attribution is then defined by all attributions other than
$\mathbf{C}^{(n)}$:

$\displaystyle\mathbf{r}^{(n)}=\mathbf{A}^{(n)}+\mathbf{F}^{(n)}_{\nabla
x}+\frac{\mathbf{F}^{(n)}_{x}+\mathbf{M}^{(n)}_{x\nabla
x}}{1+\exp(-\mathbf{C}^{(n)})}$ (13)

We observe that both $\mathbf{F}^{(n)}_{x}$ and $\mathbf{M}^{(n)}_{x\nabla x}$
are affected by the input feature map, resulting in the saliency bias effect
(see Sec. 1). As a result, we penalize their sum according to
$\mathbf{C}^{(n)}$, in a manner that emphasises positive attribution regions.

We note that
$\sum_{j}(\mathbf{C}_{j}^{(n)}+\mathbf{r}_{j}^{(n)})\neq\sum_{i}\Phi_{i}^{(n-1)}$,
and the residual needs to be compensated for, in order to preserve the
conservation rule. Therefore, we perform a $\Delta$-shift as defined in Def.
2, resulting in the final attribution:

$\displaystyle\Phi^{(n)}=\Delta^{(n)}_{\text{\em
shift}}(\mathbf{C}^{(n)},\mathbf{r}^{(n)})$ (14)

###  4.4 Explaining Self-Supervised Learning (SSL)

SSL is proving to be increasingly powerful and greatly reduces the need for
labeled samples. However, no explainability method was applied to verify that
these models, which are often based on image augmentations, do not ignore
localized image features.

Since no label information is used, we rely on the classifier of the self-
supervised task itself, which has nothing to do with the classes of the
datasets. We consider for each image, the image that is closest to it in the
penultimate layer’s activations. We then subtract the logits of the self
supervised task of the image to be visualized and its nearest neighbor, to
emphasize what is unique to the current image, and then use explainability
methods on the predicted class of the self-supervised task. See supplementary
for more details of this novel procedure.

##  5 Experiments

Qualitative Evaluation: Fig. 2(a,b) present sample visualization on a
representative set of images for networks trained on ImageNet, using VGG19 and
ResNet-50, respectively. In these figures, we visualize the top-predicted
class. More results can be found in the supplementary, including the output
for the rest of the methods, which are similar to other methods and are
removed for brevity.

The preferable visualization quality provided by our method is strikingly
evident. One can observe that (i) LRP, FullGrad and Grad-CAM output only
positive results, wherein LRP edges are most significant, and in all three,
the threshold between the object and background is ambiguous. (ii) CLRP and
SGLRP, which apply LRP twice, have volatile outputs. (iii) RAP is the most
consistent, other than ours, but falls behind in object coverage and
boundaries. (iv) Our method produces relatively complete regions with clear
boundaries between positive and negative regions.

In order to test whether each method is class-agnostic or not, we feed the
classifier images containing two clearly seen objects, and propagate each
object class separately. In Fig. 2(c) we present results for a sample image.
As can be seen, LRP, FullGrad and RAP output similar visualizations for both
classes. Grad-CAM, on the other hand, clearly shows a coarse region of the
target class, but lacks the spatial resolution. CLRP and SGLRP both achieve
class separation, and yet, they are highly biased toward image edges, and do
not present a clear separation between the object and its background. Our
method provides the clearest visualization, which is both highly correlated
with the target class, and is less sensitive toward edges. More samples can be
found in the supplementary.

Quantitative Experiments: We employ two experiment settings that are used in
the literature, negative perturbation and segmentation tests. We evaluate our
method using three common datasets: (i) the validation set of ImageNet
(Russakovsky et al. 2015) (ILSVRC) 2012, consisting of 50K images from 1000
classes, (ii) an annotated subset of ImageNet called ImageNet-Segmentation
(Guillaumin, Küttel, and Ferrari 2014), containing 4,276 images from 445
categories, and (iii) the PASCAL-VOC 2012 dataset, depicting 20 foreground
object classes and one background class, and containing 10,582 images for
training, 1449 images for validation and 1,456 images for testing.

Negative Perturbation Experiments: The negative perturbation test is composed
of two stages, first, a pre-trained network is used to generate the
visualizations of the ImageNet validation set. In our experiments, we use the
VGG-19 architecture, trained on the full ImageNet training set. Second, we
mask out an increasing portion of the image, starting from lowest to highest
values, determined by the explainability method. At each step, we compute the
mean accuracy of the pre-trained network. We repeat this test twice: once for
the explanation of the top-1 predicted class, and once for the ground truth
class. The results are presented in Fig. 3 and Tab. 3. As can be seen, our
method achieves the best performance across both tests, where the margin is
highest when removing $40\%-80\%$ of the pixels.

|
---|---
(a) | (b)
Figure 4:  Quantitative results for self-supervised methods in the
segmentation task. (a) Comparison of different explainability methods for
SCAN, SeLa and RotNet. (b) Per-layer performance of RotNet using linear-
probes.

| Ours | RAP
---|---|---
Method | ($\Delta$) | ($\textstyle\sum$) | w/o | ($\Delta$) | ($\textstyle\sum$) | w/o
Supervised | 42.4 | 42.1 | 41.2 | 41.6 | 41.2 | 40.0
SeLa | 37.8 | 37.4 | 36.9 | 37.2 | 36.3 | 34.5
SCAN | 38.1 | 37.9 | 37.0 | 37.4 | 36.8 | 34.8
Table 4:  AUC for negative perturbation tests for self-supervised methods -
SeLa and SCAN. ($\Delta$) is our method; ($\textstyle\sum$) is an alternative
that adds instead of subtracts, w/o does not consider the neighbor at all.
RAP, which is the best baseline in Tab. 3, is used as a baseline.

| Ours | Only | without | $\mathbf{r}^{(n)}=$ Grad-CAM
---|---|---|---|---
| $\mathbf{C}^{(n)}$ | $\mathbf{A}^{(n)}$ | $\mathbf{F}^{(n)}_{x}$ | $\mathbf{F}^{(n)}_{\nabla x}$ | $\mathbf{M}^{(n)}$ | $({1+\exp(-\mathbf{C}^{(n)})})^{-1}$
AUC | 38.9 | 38.2 | 34.6 | 37.5 | 37.1 | 37.2 | 37.5 | 37.1

Table 5: AUC results in the negative perturbation test for variations of our
method.

Semantic Segmentation Metrics: To evaluate the segmentation quality obtained
by each explainability method, we compare each to the ground truth
segmentation maps of the ImageNet-Segmentation dataset, and the PASCAL-VOC
2012, evaluating by pixel-accuracy and mean average-precision. We follow the
literature benchmarks for explainability that are based on labeled segments
(Nam et al. 2019). Note that these are not meant to provide a full weakly-
supervised segmentation solution, which is often obtained in an iterative
manner. Rather, its goal is to demonstrate the ability of each method without
follow-up training. For the first dataset, we employed the pre-trained VGG19
classifier trained on ImageNet training set, and compute the explanation for
the top-predicted class (we have no access to the ground truth class) and
compare it to the ground truth mask provided in the dataset. For the second,
we trained a multi-label classifier on the PASCAL-VOC 2012 training set, and
consider labels with a probability larger than 0.5 to extract the
explainability maps. For methods that provide both positive and negative
values (Gradient SHAP, LRP$\alpha-\beta$, RAP, CLRP, SGLRP, and ours), we
consider the positive part as the segmentation map of that object. For methods
that provide only positive values (Integrated Grad, Smooth Grad, Full Grad,
GradCAM, LRP, Meaningful Perturbation), we threshold the obtained maps at the
mean value to obtain the segmentation map. Results are reported in Tab. 3,
demonstrating a clear advantage of our method over all nine baseline methods,
for all datasets and metrics. Other methods seem to work well only in one of
the datasets or present a trade-off between the two metrics.

Explainability for Self-Supervised Models: We use three state-of-the-art SSL
models: ResNet-50 trained with either SCAN (Van Gansbeke et al. 2020) or SeLa
(Asano, Rupprecht, and Vedaldi 2019b), and an Alexnet by Asano, Rupprecht, and
Vedaldi (2019a), which we denote as RotNet.

Fig. 5(a) shows the segmentation performance for the ImageNet ground truth
class of the completely unsupervised SSL methods (color) using different
explainability methods (shape). For all models our explainability method
outperforms the baselines in both mAP and pixel accuracy, except for RotNet
where the mAP is considerably better and the pixel accuracy is slightly lower.
Fig. 5(b) shows the increase of segmentation performance for RotNet with our
method, as we visualize deeper layers of SSL Alexnet, using a supervised
linear post-training as proposed by Asano, Rupprecht, and Vedaldi (2019a).
This finding is aligned with the classification results by a single linear
layer of Asano, Rupprecht, and Vedaldi (2019a), which improve with the layer
index.

Tab. 5 compares RAP (which seems to be the strongest baseline) and our method
in the predicted SSL class (not ImageNet) negative perturbation setting.
Evidently, our method has superior performance. SSL results also seem
correlated with the fully supervised ones (note that the architecture and the
processing of the fully connected layer is different from the one used in Tab.
3). Two alternatives to our novel SSL procedure are also presented: in one,
the difference from the neighbor is replaced with a sum. In the other, no
comparison to the neighbor takes place. Our procedure for SSL explainability
is superior for both our method and RAP.

Ablation Study: By repeatedly employing normalization, our method is kept
parameter-free. In Tab. 5, we present negative perturbation results for
methods that are obtained by removing one components out of our complete
method. We also present the results of a similar method in which the guided
attribution based residual-term $\mathbf{r}$ is replaced by a GradCam term. As
can be seen, each of these modifications damages the performance to some
degree. Without any residual term, the method is slightly worse than RAP,
while a partial residual term further hurts performance. In the supplementary
material, we show visual results of the different components and variations of
our method and present observations on the contribution of each part to the
final outcome.

##  6 Conclusions

Explainability plays a major rule in debugging neural networks, creating trust
in the predictive capabilities of networks beyond the specific test dataset,
and in seeding downstream methods that analyze the images spatially. Previous
visualization methods are either class-agnostic, low-resolution, or neglect
much of the object’s region, focusing on edges. The separation between
relevant and irrelevant image parts provided by the previous methods is also
often blurry. In this work, we present a novel explainability method that
outputs class-dependent explanations that are clearer and more exact than
those presented by the many existing methods tested. The new method is based
on combining concepts from the two major branches of the current literature:
attribution methods and gradient methods. This combination is done, on equal
grounds, through the usage of a non-negative matrix factorization technique
that partitions the image into foreground and background regions. Finally, we
propose a novel procedure for evaluating the explainability of SSL methods.

## Acknowledgment

This project has received funding from the European Research Council (ERC)
under the European Unions Horizon 2020 research and innovation programme
(grant ERC CoG 725974). The contribution of the first author is part of a
Ph.D. thesis research conducted at Tel Aviv University.

## References

  * Adebayo et al. (2018) Adebayo, J.; Gilmer, J.; Muelly, M.; Goodfellow, I.; Hardt, M.; and Kim, B. 2018\.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , 9505–9515.
  * Ahn, Cho, and Kwak (2019) Ahn, J.; Cho, S.; and Kwak, S. 2019.  Weakly supervised learning of instance segmentation with inter-pixel relations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2209–2218.
  * Asano, Rupprecht, and Vedaldi (2019a) Asano, Y. M.; Rupprecht, C.; and Vedaldi, A. 2019a.  A critical analysis of self-supervision, or what we can learn from a single image.  _arXiv preprint arXiv:1904.13132_ .
  * Asano, Rupprecht, and Vedaldi (2019b) Asano, Y. M.; Rupprecht, C.; and Vedaldi, A. 2019b.  Self-labelling via simultaneous clustering and representation learning.  _arXiv preprint arXiv:1911.05371_ .
  * Bach et al. (2015) Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; and Samek, W. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ 10(7): e0130140.
  * Binder et al. (2016) Binder, A.; Montavon, G.; Lapuschkin, S.; Müller, K.-R.; and Samek, W. 2016\.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In _International Conference on Artificial Neural Networks_ , 63–71. Springer.
  * Dabkowski and Gal (2017) Dabkowski, P.; and Gal, Y. 2017.  Real time image saliency for black box classifiers.  In _Advances in Neural Information Processing Systems_ , 6970–6979.
  * Erhan et al. (2009) Erhan, D.; Bengio, Y.; Courville, A.; and Vincent, P. 2009.  Visualizing higher-layer features of a deep network.  _University of Montreal_ 1341(3): 1.
  * Fong, Patrick, and Vedaldi (2019) Fong, R.; Patrick, M.; and Vedaldi, A. 2019.  Understanding deep networks via extremal perturbations and smooth masks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2950–2958.
  * Fong and Vedaldi (2017) Fong, R. C.; and Vedaldi, A. 2017.  Interpretable explanations of black boxes by meaningful perturbation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 3429–3437.
  * Gao et al. (2016) Gao, M.; Chen, H.; Zheng, S.; and Fang, B. 2016.  A factorization based active contour model for texture segmentation.  In _2016 IEEE International Conference on Image Processing (ICIP)_ , 4309–4313. IEEE.
  * Gu, Yang, and Tresp (2018) Gu, J.; Yang, Y.; and Tresp, V. 2018.  Understanding individual decisions of cnns via contrastive backpropagation.  In _Asian Conference on Computer Vision_ , 119–134. Springer.
  * Guillaumin, Küttel, and Ferrari (2014) Guillaumin, M.; Küttel, D.; and Ferrari, V. 2014.  Imagenet auto-annotation with segmentation propagation.  _International Journal of Computer Vision_ 110(3): 328–348.
  * Hoyer et al. (2019) Hoyer, L.; Munoz, M.; Katiyar, P.; Khoreva, A.; and Fischer, V. 2019.  Grid saliency for context explanations of semantic segmentation.  In _Advances in Neural Information Processing Systems_ , 6462–6473.
  * Huang et al. (2018) Huang, Z.; Wang, X.; Wang, J.; Liu, W.; and Wang, J. 2018.  Weakly-supervised semantic segmentation network with deep seeded region growing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 7014–7023.
  * Iwana, Kuroki, and Uchida (2019) Iwana, B. K.; Kuroki, R.; and Uchida, S. 2019.  Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation.  _arXiv preprint arXiv:1908.04351_ .
  * Kindermans et al. (2017) Kindermans, P.-J.; Schütt, K. T.; Alber, M.; Müller, K.-R.; Erhan, D.; Kim, B.; and Dähne, S. 2017.  Learning how to explain neural networks: Patternnet and patternattribution.  _arXiv preprint arXiv:1705.05598_ .
  * Lundberg and Lee (2017) Lundberg, S. M.; and Lee, S.-I. 2017.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , 4765–4774.
  * Mahendran and Vedaldi (2016) Mahendran, A.; and Vedaldi, A. 2016.  Visualizing deep convolutional neural networks using natural pre-images.  _International Journal of Computer Vision_ 120(3): 233–255.
  * Montavon et al. (2017) Montavon, G.; Lapuschkin, S.; Binder, A.; Samek, W.; and Müller, K.-R. 2017\.  Explaining nonlinear classification decisions with deep taylor decomposition.  _Pattern Recognition_ 65: 211–222.
  * Nam et al. (2019) Nam, W.-J.; Gur, S.; Choi, J.; Wolf, L.; and Lee, S.-W. 2019.  Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks.  _arXiv preprint arXiv:1904.00605_ .
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115(3): 211–252.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Shrikumar, Greenside, and Kundaje (2017) Shrikumar, A.; Greenside, P.; and Kundaje, A. 2017.  Learning important features through propagating activation differences.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , 3145–3153. JMLR. org.
  * Shrikumar et al. (2016) Shrikumar, A.; Greenside, P.; Shcherbina, A.; and Kundaje, A. 2016.  Not just a black box: Learning important features through propagating activation differences.  _arXiv preprint arXiv:1605.01713_ .
  * Simonyan, Vedaldi, and Zisserman (2013) Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2013.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ .
  * Smilkov et al. (2017) Smilkov, D.; Thorat, N.; Kim, B.; Viégas, F.; and Wattenberg, M. 2017.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ .
  * Srinivas and Fleuret (2019) Srinivas, S.; and Fleuret, F. 2019.  Full-gradient representation for neural network visualization.  In _Advances in Neural Information Processing Systems_ , 4126–4135.
  * Sundararajan, Taly, and Yan (2017) Sundararajan, M.; Taly, A.; and Yan, Q. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , 3319–3328. JMLR. org.
  * Van Gansbeke et al. (2020) Van Gansbeke, W.; Vandenhende, S.; Georgoulis, S.; Proesmans, M.; and Van Gool, L. 2020.  SCAN: Learning to Classify Images without Labels.  In _European Conference on Computer Vision (ECCV)_.
  * Wang et al. (2019) Wang, Y.; Zhang, J.; Kan, M.; Shan, S.; and Chen, X. 2019.  Self-supervised Scale Equivariant Network for Weakly Supervised Semantic Segmentation.  _arXiv preprint arXiv:1909.03714_ .
  * Yuan, Wang, and Cheriyadat (2015) Yuan, J.; Wang, D.; and Cheriyadat, A. M. 2015.  Factorization-based texture segmentation.  _IEEE Transactions on Image Processing_ 24(11): 3488–3497.
  * Zeiler and Fergus (2014) Zeiler, M. D.; and Fergus, R. 2014.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , 818–833. Springer.
  * Zhang et al. (2018) Zhang, J.; Bargal, S. A.; Lin, Z.; Brandt, J.; Shen, X.; and Sclaroff, S. 2018.  Top-down neural attention by excitation backprop.  _International Journal of Computer Vision_ 126(10): 1084–1102.
  * Zhou et al. (2018) Zhou, B.; Bau, D.; Oliva, A.; and Torralba, A. 2018.  Interpreting deep visual representations via network dissection.  _IEEE transactions on pattern analysis and machine intelligence_ .
  * Zhou et al. (2016) Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2921–2929.

Supplementary: Differentiable Gaussian PSF Layer

##  Appendix A Code

The code is written in PyTorch, and is provided as part of the supplementary
material, including a Jupyter notebook with examples. Fig. 2,2 show
screenshots from the notebook.

Figure 1: Visualization of two different classes for VGG-19.

Figure 2: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

##  Appendix B Algorithm

Our method is listed in Alg. 1.
Note that for the linear layers of the network, we compute the residual using
only $\mathbf{F}_{x\nabla x}$ in the following way:

$\displaystyle\mathbf{r}^{(n)}$
$\displaystyle=\begin{cases}\mathbf{M}^{(n)}_{x\nabla x},&\text{First linear
layer}\\\ 0,&\text{Otherwise}\end{cases}$ (1)

We note that $x$ and $\nabla x$ are reshaped into their previous (2D) form.

Algorithm 1 Class Attribution Propagation with Guided Factorization.

1:$M:$ Neural network model, $I\in\mathbb{R}^{C\times H\times W}:$ Input
images, $t\in\mathcal{C}:$ Target class.

2:$M(I)$ $\triangleright$ Forward-pass, save intermediate feature maps

3:$\sigma\leftarrow\max\|y-y^{t}\|_{1}$

4:$\hat{y}\leftarrow\operatorname{softmax}\big{(}y^{t}\exp{\big{(}-\frac{1}{2}\big{(}\frac{y-y^{t}}{\sigma}\big{)}^{2}\big{)}}\big{)}$

5:$\Phi^{(1)}\leftarrow x^{(1)}\odot\frac{\partial\hat{y}^{t}}{\partial
x^{(1)}}$ $\triangleright$ Initial Attribution - First Linear layer

6:for linear layers, $n>1$ do

7:
$\mathbf{C}_{j}^{(n)}\leftarrow\mathcal{G}^{(n)}(|x^{(n)}|,|\theta^{(n)}|,\Phi^{(n-1)})$
$\triangleright$ Absolute influence

8: if next layer is 2D then

9: Reshape $x$ and $\nabla x$ to the previous 2D form

10:
$\mathbf{M}^{(n)}\leftarrow\mathcal{N}_{\max}\bigg{(}\bigg{(}\frac{1}{|C|}\sum_{c\in[C]}\big{(}x_{c}^{(n)}\odot\nabla
x_{c}^{(n)}\big{)}\bigg{)}^{+}\bigg{)}$

11: $\mathbf{r}^{(n)}\leftarrow\mathbf{M}^{(n)}$ $\triangleright$ Residual

12: else

13: $\mathbf{r}^{(n)}\leftarrow 0$

14: $\Phi^{(n)}\leftarrow\Delta^{(n)}_{\text{\em
shift}}(\mathbf{C}^{(n)},\mathbf{r}^{(n)})$ $\triangleright$ Shifting by the
residual

15:for convolution layers do

16: Compute $\mathbf{C}^{(n)}$, $\mathbf{M}^{(n)}$

17:
$\mathbf{A}^{(n)}_{j}\leftarrow\mathcal{G}^{(n)}(\mathbb{1},|\theta^{(n)}|,\Phi^{(n-1)})$
$\triangleright$ Input agnostic attribution

18:
$\mathbf{F}^{(n)}_{x}\leftarrow\bar{\mathcal{F}}(x^{(n)},\mathbf{C}^{(n)})^{+}$
$\triangleright$ Factorization of input feature map

19: $\mathbf{F}^{(n)}_{\nabla x}\leftarrow\bar{\mathcal{F}}(\nabla
x^{(n)},\mathbf{C}^{(n)})^{+}$ $\triangleright$ Factorization of input feature
map gradients

20: $\mathbf{r}^{(n)}\leftarrow\mathbf{A}^{(n)}+\mathbf{F}^{(n)}_{\nabla
x}+\frac{\mathbf{F}^{(n)}_{x}+\mathbf{M}^{(n)}_{x\nabla
x}}{1+\exp(-\mathbf{C}^{(n)})}$ $\triangleright$ Residual

21: Compute $\Phi^{(n)}$

##  Appendix C Ablation Study

By repeatedly employing normalization, our method is kept parameter-free. The
different components of it, presented in Sec.4 of the paper, are visualized in
Fig 3. One can observe several behaviors of the different component, (i)
across all components, the semantic information is best visible in deeper
layers of the network, where residual information is becoming more texture-
oriented at shallower layers. (ii) The difference between $F_{x}$ and
$F_{\nabla x}$ is mostly visible in the out-of-class regions: $F_{x}$ is
derived from the data directly and is biased toward input-specific
activations, resulting in highlights from the class that is not being
visualized from layer 5 onward. (iii) The input-agnostic data term
$\mathbf{A}$ is more blurry than $\mathbf{C}$, as a result of using the
$\mathbb{1}$ tensor as input. It can, therefore, serve as a regularization
term that is less susceptible to image edges.

Figure 3: Method decomposition on the image of Fig.1(a) of the paper.
Visualization of the different elements of the method for a pre-trained VGG-19
network. Predicted top-class is “dog”, propagated class is “cat”. (1, 5, 11) -
Convolution layer number.

##  Appendix D Additional Results

In Tab. 6 and 7 we show additional results for two different Grad-CAM
variants, considering visualisation from shallower layers as denoted by Grad-
CAM* = 4-layers shallower and Grad-CAM** = 8-layers shallower. An example of
the different outputs can be seen in Fig. 3 of the previous section, where the
semantic information of the Grad=CAM is less visible as we go shallower, where
as in our method, the sematic information is becoming more fine grained.
Additionally, we show the inferior results of Gradient SHAP and DeepLIFT SHAP
for negative perturbation.

Table 6: Area Under the Curve (AUC) values for the two negative perturbation tests. Grad-CAM variants : * = 4-layers shallower, ** = 8-layer shallower. | Gradient | DeepLIFT | Grad-CAM | Grad-CAM* | Grad-CAM** | Ours
---|---|---|---|---|---|---
| SHAP | SHAP
Predicted | 7.5 | 7.8 | 37.8 | 29.9 | 16.0 | 38.9
Target | 7.7 | 8.1 | 39.4 | 30.9 | 16.5 | 40.0
Table 7: Quantitative segmentation results on (a) ImageNet and (b) PASCAL-VOC 2012. |  | Grad-CAM | Grad-CAM* | Grad-CAM** | Ours
---|---|---|---|---|---
ImageNet | Pixel Acc. | 66.5 | 65.5 | 50.5 | 79.9
mAP | 62.4 | 61.6 | 51.1 | 76.7
VOC’12 | Pixel Acc. | 52.6 | 18.4 | 17.2 | 77.5
mAP | 41.7 | 44.5 | 43.4 | 45.9

##  Appendix E Self-Supervised Learning

Our submission is the first contribution, as far as we can ascertain, that
employs explainability methods on Self Supervised Learning methods. We
evaluate our method on three recent state-of-the-art models. First, we show
our method performance on AlexNet model, trained by RotNet in a completely
self-supervised fashion which depends only on data augmentations specially
predicting the 2d image rotations that is applied to the image that it gets as
input. In order to evaluate the quality of the features learned we follow
(only for the RotNet experiments) a similar approach used in the literature by
using a supervised linear post-training in order to visualize deeper layers.

Fig. 4 shows the visualization obtained for the image on the left when
training a supervised linear classifier after each layer and applying our
method. As can be seen, early layers learns edges-like patterns and the deeper
layer learns more complicated patterns, more visualizations are shown in the
end of the supplementary.

Figure 4: The visualization after each layer. (a) original image. (b)
visualization after first layer. (c) visualization after second layer. (d)
visualization after third layer. (e) visualization after the fourth layer. (e)
visualization after last layer.

In order to evaluate SSL methods without any additional supervision, we employ
the classifier the SSL is trained with. This is presented in Sec. 4.4 of the
paper and is given below as pseudocode in Alg 2.

Algorithm 2 self-supervised explainability by adopting nearest neighbors

1:$I$ input image, $\mathcal{S}$ set of images, $M=\phi_{\tau}\circ\phi_{F}$
the SSL network, where $\phi_{F}$ extracts the features and $\phi_{\tau}$ is
the linear SSL classifier.

2:$L_{I}=\phi_{F}(I)$ $\triangleright$ $L_{I}$ is the latent vector of image
$I$

3:$\mathcal{L}\leftarrow\\{\phi_{F}(J)|J\in\mathcal{S}\\}$ $\triangleright$
$\mathcal{L}$ is the set of all latent vectors for all images in $\mathcal{S}$

4:$L_{N}=\arg\min_{L\in\mathcal{L}}\|L_{I}-L\|$ $\triangleright$ $L_{N}$ is
the nearest neighbor of $L$

5:${S}_{I}=L_{I}-L_{N}$ $\triangleright$ Subtracting $L_{N}$ from $L_{I}$ to
emphasize the unique elements of $L_{I}$

6:$v=\phi_{\tau}({S}_{I})$ $\triangleright$ Forward pass with the new latent
vector

7:$t=\arg\max v$ $\triangleright$ Choose the class with the highest
probability

8:Apply Alg. 1, with the input tuple $[\phi_{\tau}(S_{I}),\phi_{F}(I)],I,t$

##  Appendix F Additional Results - Multi Label

Figure 5: Visualization of two different classes for VGG-19. Figure 6:
Visualization of two different classes for VGG-19.

##  Appendix G Additional Results - Top Class

Figure 7: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 8: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 9: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 10: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 11: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 12: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 13: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 14: Visualization of the top predicted class of a VGG-19 ImageNet
trained network.

Figure 15: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 16: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 17: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 18: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 19: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 20: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

Figure 21: Visualization of the top predicted class of a ResNet-50 ImageNet
trained network.

##  Appendix H Additional Results - AlexNet Probes

Input | Layer 1. | Layer 2. | Layer 3. | Layer 4. | Layer 5.
---|---|---|---|---|---
Image
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Figure 22: The visualization after each layer using linear probes on the
RotNet AlexNet model.

##  Appendix I Additional Results - Self-labeling (SeLa) Method

For each explainability method, we present results obtained by simply
projecting the self labeling label with the highest probability (a very
simplified version of our procedure that does not involve the nearest neighbor
computation) as well as the one for Alg. 2.

Input | LRPαβ | LRPαβ | LRP | LRP | RAP | RAP | Ours | Ours
---|---|---|---|---|---|---|---|---
Image | with Alg.2 | with Alg.2 | with Alg.2 | with Alg.2
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
Figure 23: The visualization of different explanability methods on Resnet-50
trained in self-supervised regime using self-labeling method.

##  Appendix J Additional Results - SCAN Method

For each explainability method, we present results obtained by simply
projecting the self labeling label with the highest probability (a very
simplified version of our procedure that does not involve the nearest neighbor
computation) as well as the one for Alg. 2.

Input | LRPαβ | LRPαβ | LRP | LRP | RAP | RAP | Ours | Ours
---|---|---|---|---|---|---|---|---
Image | with Alg.2 | with Alg.2 | with Alg.2 | with Alg.2
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
Figure 24: The visualization of different explanability methods on Resnet-50
trained in self-supervised regime using SCAN method.


# Leggett-Garg test of macrorealism using indefinite causal order of
measurements

A. K. Pan<EMAIL_ADDRESS>Department of Physics, Indian Institute of
Technology Hyderabad, Kandi, Telengana-502284, India

###### Abstract

Macrorealism is a belief that constitutes the core of our perception of
reality in the everyday world. The Leggett-Garg (LG) test is a conceptually
elegant approach for probing the compatibility between the notion of
macrorealism and quantum theory. However, a conclusive LG test hinges on how
one fixes the operational invasiveness loophole, i.e., how the statistical
form of non-invasive measurability assumption is guaranteed in an LG test.
Despite many attempts to close this loophole, no consensus has been achieved
yet. In this work, we propose a simple and elegant scheme based on indefinite
causal order in quantum switch experiment, which enables us to close this
loophole, and eventually, the LG test becomes a conclusive test of
macrorealism.

The notion of macrorealism - the macroscopic objects in our everyday world has
well-defined properties irrespective of being actually observed - is a well-
accepted belief. It is the same tenet that constitutes the core of our
classical worldview about the nature of reality. The moon exists objectively
and has a definite position in the sky even when no one looks. However, since
the inception of quantum theory, the extent to which this view is compatible
with the standard framework has triggered intense debates among physicists and
philosophers and remains the subject of active research.

The pertinent question is how such a macrorealistic worldview fits into the
quantum theory unless the division between micro and macro-world is imposed.
Even so, where to put the ‘cut’ (_’a la_ Heisenberg [1]) and how the quantum
theory fixes it remains a ticklish question. This issue first gets its firm
ground through the Schr$\ddot{o}$ dinger’s famous thought experiment [2]
involving an unfortunate cat. From his viewpoint, there is no logical reason
to describe a cat (macro-object) and a neutron (micro-object) in different
levels of reality within quantum formalism, in contrast to Bohr’s view of the
‘quantum system’ and ‘classical apparatus’.

There have been many attempts to pose the appropriate questions relevant to
this issue. We may try here to put it in a somewhat different manner by first
accepting the fact that we are undoubtedly _not_ interested in the question
that in which limit quantum theory reduces to the classical one. Rather, we
seek to answer whether a macro-object follows our everyday worldview of
realism if the universality of quantum theory is accepted. It is desired that
being an universal theory the quantum theory must be able to recover the
nature of reality involving macro-objects. Many proposals have been put
forwarded to probe this issue, viz. the decoherence program [3], coarse-
graining measurements [4], unified micro-macro description [5, 6], macro-
objectification of quantum coherence of large objects [7, 8]. However, they do
not directly address the (in)compatibility between macrorealism and quantum
theory.

The Leggett and Garg (LG) test [9, 10] is a conceptually elegant, direct and
empirically testable approach. It consists of two main assumptions. i)
_Macrorealism_ _p_ er se: Macroscopic system remains in a definite state
(maybe unknown) at any instant of time irrespective of observation. ii) _Non-
invasive measurability_ : State of the macro-system can be determined (at
least in principle) without affecting the system itself and its subsequent
dynamics. The first assumption seems quite obvious and fits with our
perception of the nature around us while dealing with macro-object. The second
one is also derivable from our everyday experience. LG test claims to
establish the untenability of macrorealism in quantum theory, hence violating
one or both assumptions.

Let us briefly recapitulate the essence of LG framework. Consider the
measurement of a dichotomic observable $\hat{M}_{1}$ at three different times
$t_{1}$, $t_{2}$ and $t_{3}$ $(t_{3}\geq t_{2}\geq t_{1})$. In between the
measurements at two different times, the system unitarily evolves under a
suitable Hamiltonian $\hat{H}$, so that
$M_{j}=U^{\dagger}_{t_{j}-t_{i}}M_{i}U_{t_{j}-t_{i}}$ where $i,j=1,2,3$ and
$j>i$. Here $U_{t_{j}-t_{i}}=e^{-i\hat{H}(t_{j}-t_{i})/\hbar}$. The three-time
Leggett-Garg inequality (LGI) can be written as

$K_{3}=1+\sum\limits_{m_{i},m_{j}=\pm
1}m_{i}m_{j}\langle\hat{M_{i}}\hat{M_{j}}\rangle\geq{0}$ (1)

which is derived based on the above two assumptions. The correlation
$\langle{M_{i}}{M_{j}}\rangle=\sum\limits_{m_{i},m_{j}=\pm
1}m_{i}m_{j}p(m_{i},m_{j})$ and $p(m_{i},m_{j})$ is the pair-wise sequential
joint probability of outcomes $m_{i}$ and $m_{j}$. The quantum violation of
LGI in Eq.(1) for any two-level system is extensively studied in the
literature both theoretically [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25] and experimentally [26, 27, 28, 29, 30, 31, 32]. For an extensive
review, we refer [33].

We note that the standard three-time LGIs in Eq. (1) do not provide the
necessary and sufficient conditions for macrorealism. The quantum violation of
LGIs implies the violation of macrorealism, but the converse does not hold.
The crucial question we address in this work is whether the violation of an
LGI unequivocally warrants that the system is not behaving macrorealistically.

To pose this question more precisely, a comparison between LG and Bell tests
would be helpful. LGIs are often considered as the temporal analog of Bell’s
inequalities primarily due to the structural resemblance between Bell-CHSH
inequalities and four-time LGIs. However, LG test shows serious pathology [11,
13, 14, 16, 17, 18, 19, 20, 22, 23, 24, 25] in comparison to a Bell test of
local realism. In the latter case, the statistical version of locality
condition, the no-signaling in space is satisfied in any physical theory.
Hence, the Bell test unequivocally establishes the violation of both locality
and realism. On the other hand, the operational non-invasiveness (commonly
known as no-signaling in time) - the statistical version of the non-invasive
measurability assumption is not generally satisfied in quantum theory. A
skeptic (stubborn macrorealist) can then rightfully argue that she can salvage
macrorealism in quantum theory by simply abandoning non-invasive
measurability. Thus, unless the operational non-invasiveness loophole (also
known as clumsiness loophole [34]) is fixed in the experiment, the LG test
cannot be considered as a conclusive test of macrorealism.

Many attempts have already been made to close this loophole by proposing
various forms of measurement schemes. Leggett himself was well aware of it,
and he proposed the ideal negative-result measurement. Many experimental tests
[30, 27, 29, 26, 32] of LGI have been performed by using it. However, since in
negative-result measurement the collapse of state occurs, i.e., system and its
subsequent dynamics [12] is disturbed, then up to what extent this approach
satisfies the operational non-invasiveness remains debatable. Another approach
is by using the technique of weak measurement [35] where the strength of the
measurement is possible to adjust and, in principle, the back action (the
invasiveness) on the system can be reduced to an arbitrarily small amount [28,
31]. Again, how this idea of assuming weak coupling (still cause small back-
action) in quantum measurement will convince a macrorealist remains
unsatisfactory [12, 16, 15, 36].

LG test involves the measurement of non-commuting observables. There is no
unique prescription to fix the joint probability for such observables in the
standard framework of quantum measurement theory. However, it is common in the
literature to use quasiprobability in such a scenario. In the LG scenario, one
may also define suitable quasiprobability as pointed out by Halliwell [16]. We
demonstrate how suitably defined quasiprobability enables a conclusive LG test
by providing the explicit measurement scheme based on the quantum switch
experiment that uses indefinite causal order of measurements. We note here
that the sequential correlation of two non-commuting observables in Eq. (1)
can also be written in a form

$\displaystyle\langle{M_{i}}{M_{j}}\rangle_{seq}=\frac{1}{2}Tr\left[\\{M_{i},M_{j}\\}\rho\right]$
(2)

where $\rho$ is the density matrix. The proof of Eq. (2) is given in Appendix
A. This indicates a superposition of measurements of $M_{i}$ and $M_{j}$ are
needed, which can be achieved through the indefinite order of measurements. We
also argue here how our approach fixes the operational non-invasiveness
loophole.

A couple of remarks on the non-invasive measurability condition could be
useful here. This condition can be interpreted in two different ways,
contingent upon how the sequential measurements are performed in an
experiment. Commonly three sequential correlations given in Eq. (1) are
performed involving the measurements at two times. This is analogous to the
Bell test, and hence for a conclusive LG test, the operational non-
invasiveness must be satisfied for the pair-wise case instead of the triple-
wise case. This is a weaker reading of non-invasive measurability condition as
pointed out in [24]. The stronger version of the non-invasive measurability
condition assumes the satisfaction of operational non-invasiveness for the
sequential measurements at three times. We show that for both weaker and
stronger readings of the non-invasive measurability condition, our scheme
works.

However, for better understanding of our idea, let us first present the two-
time LG scenario involving two-time sequential correlation only. LGIs for two-
time measurement scenario can be formulated as

$\displaystyle
G_{2}=1+m_{i}\langle\hat{M_{i}}\rangle+m_{j}\langle\hat{M_{j}}\rangle+m_{i}m_{j}\langle\hat{M_{i}}\hat{M_{j}}\rangle\geq
0$ (3)

with $i,j=1,2,3$ and $j>i$. For different values of $m_{i}=\pm 1$ and
$m_{j}=\pm 1$ we have four two-time LGIs. Halliwell [16, 18] argued that the
above set of LGIs provides necessary and sufficient conditions for
macrorealism in the two-time LG scenario.

We shortly see that $G_{2}$ are actually quasiprobabilities. As usual, a
quasiprobability follows all of Kolmogorov’s axioms for probability except the
property of positivity. Negative and complex quasiprobabilities have immense
implications in quantum information processing [37], quantum tomography [38]
and quantum metrology [39]. It has also been demonstrated that the negativity
and contextuality are equivalent notions of nonclassicality [40]. Dressel [41]
discussed that there is a close connection between quasiprobability and weak
value[35]. The widely known quasiprobaility is the Wigner distribution [42].
For discrete system, one of the well-known quasiprobabilities is Kirkwood
distribution [43] which is in general complex and the real part of of it is
known as Merganau-Hill distribution [44, 45]. Given a density matrix $\rho$,
Merganau-Hill distribution can be written as

$\displaystyle q_{Q}(m_{i},m_{j})=Re[\langle m_{i}|m_{j}\rangle\langle
m_{j}|\rho|m_{i}\rangle]$ (4)

satisfying $\sum_{m_{i},m_{j}}q_{Q}(m_{i},m_{j})=1$. Here
$\\{|m_{i}\rangle\\}$ and $\\{|m_{j}\rangle\\}$ are two arbitrary bases on the
same Hilbert space in which $\rho$ belongs. Using the moment expansion
corresponding to dichotomic observables here we can write the quasiprobability
in a macrorealistic model as

$\displaystyle q(m_{i},m_{j})=\frac{1}{4}(1+m_{i}\langle
M_{i}\rangle+m_{j}\langle M_{j}\rangle+m_{i}m_{j}\langle M_{i}M_{j}\rangle)$
(5)

which is nothing but the two-time LGIs given by Eq.(3), with a multiplicative
factor 4. Thus, the negativity of a quasiprobability, i.e.,
$q_{Q}(m_{i},m_{j})<0$ implies the quantum violation of two-time LGIs.

Let us discuss the interesting properties of $q_{Q}(m_{i},m_{j})$. First, it
correctly reproduces the sequential correlation in quantum theory, so that

$\displaystyle\langle M_{i}M_{j}\rangle=$
$\displaystyle\sum\limits_{m_{i},m_{j}}m_{i}m_{j}\ q_{Q}(m_{i},m_{j})$ (6)
$\displaystyle=\sum\limits_{m_{i},m_{j}}m_{i}m_{j}\ p(m_{i},m_{j})$

Second, the most important property for the present work, $q_{Q}(m_{i},m_{j})$
satisfies the operational non-invasivness in quantum theory, i.e.,

$\displaystyle
p(m_{j})=\sum\limits_{m_{i}}q_{Q}(m_{i},m_{j})=Tr[|m_{j}\rangle\langle
m_{j}|\rho]$ (7a) $\displaystyle
p(m_{i})=\sum\limits_{m_{j}}q_{Q}(m_{i},m_{j})=Tr[|m_{i}\rangle\langle
m_{i}|\rho]$ (7b)

Explicitly, the Eqs. (7a) implies that $p(m_{j})$ remains independent of the
fact if a prior measurement of $M_{i}$ is performed. Hence the operational
non-invasiveness condition is satisfied by the quasiprobabilities
$q_{Q}(m_{2},m_{3})$. But $p(m_{j})\neq\sum\limits_{m_{i}}p(m_{i},m_{j})$ in
general. Since the correlations $\langle M_{i}M_{j}\rangle$ remains same one
may then use $q_{Q}(m_{2},m_{3})$ instead of $p(m_{2},m_{3})$ to obtain the
sequential correlations in three-time LG scenario.

Importantly, since operational non-disturbance remains satisfied,
$q_{Q}(m_{i},m_{j})<0$ constitutes a conclusive test macrorealism provided it
can be directly tested in the experiment. We show that the quantum switch
experiment [49, 50, 51, 52] realizing indefinite causal order of measurements
can provide a direct test of $q_{Q}(m_{i},m_{j})$ and constitute a loophole-
free test of macrorealism.

It is worthy of mentioning here that in an earlier work [36] we have shown
that a Mach-Zehender interference experiment itself enables a direct test of
macrorealism. In that, no apparatus was involved in a standard path-only
interference experiment, and the system itself serves as apparatus so that the
measurement effectively causes no disturbance. It was shown [36] that the
destructive interference, anomalous weak value, and the violation of two-time
LGIs are well connected. The work in [36] was further extended for double-slit
interference experiment [47] and several other interesting effects are
discussed. It is then obvious but interesting here to note that there is a
close connection between $q_{Q}(m_{i},m_{j})$ and weak value. However, this
weak value has no bearing on the weak coupling requirement in traditional weak
measurement [35]. For pure state, $\rho=|\psi_{i}\rangle\langle\psi_{i}|$, we
can write

$\displaystyle q_{Q}(m_{i},m_{j})=(|m_{i}\rangle\langle
m_{i}|)_{w}|\langle\psi_{i}|m_{j}\rangle|^{2}$ (8)

where $(|m_{i}\rangle\langle m_{i}|)_{w}$ is the weak value of the projector
defined as $(|m_{i}\rangle\langle m_{i}|)_{w}=\langle
m_{j}|m_{i}\rangle\langle m_{i}|\psi_{i}\rangle/\langle
m_{j}|\psi_{i}\rangle$. From Eq. (8) we can then say that when
$q_{Q}(m_{i},m_{j})<0$, the weak value $(|m_{i}\rangle\langle m_{i}|)_{w}$ of
the projector is anomalous and vice-versa. There exists many choices of
$|m_{i}\rangle$, $|m_{j}\rangle$ and $|\psi_{i}\rangle$ are available for
which $q_{Q}(m_{i},m_{j})<0$ can be found.

The experimental determination of $q_{Q}(m_{i},m_{j})$ has been attempted by
assembling the separately measured single and pair-wise correlations in Eq.
(3)through the noninvasive continuous-in-time velocity measurements [15, 53].
However, while measuring joint correlation of the non-commuting observable
$M_{i}$ and $M_{j}$, the disturbance will play a role again. We provide a
scheme for a direct test of $q_{Q}(m_{i},m_{j})$ in quantum switch experiment
[49, 50, 51]. which involves an interesting interferometric setup involving
indefinite causal order of two non-commuting observables. For this, let us
note that for a pure state $\rho_{s}=|\psi_{i}\rangle\langle\psi_{i}|$, the
quasipropabilities $q(m_{i},m_{j})$ can be written as

$\displaystyle
q_{Q}(m_{i},m_{j})=\frac{1}{2}\langle\psi_{i}|\left\\{\pi_{m_{i}},\pi_{m_{j}}\right\\}|\psi_{i}\rangle$
(9)

where $\pi(m_{i(j)})$ are the projectors
$\pi(m_{i(j)})=|m_{i(j)}\rangle\langle
m_{i(j)}|=(\mathbb{I}+m_{i(j)}M_{i(j)})/2$.

We consider an extended Mach-Zehender (MZ) setup (Fig.1) which realizes the
indefinite causal order of observables called quantum switch [49] which has
recently been realized in experiment [50, 51]. The setup consists of a
polarizing beam-splitter (PBS), a $50:50$ beam splitter $BS$ , eight mirrors
(not marked), a phase-shifter $(PS)$ and two detectors ($D_{1}$ and $D_{2}$).
The photons having initial state $|\psi\rangle\otimes|+\rangle$ where
$|\psi\rangle$ and $|+\rangle=\left(|H\rangle+|V\rangle\right)/\sqrt{2}$ are
path and system states respectively. After the PBS the state becomes

$|\Psi\rangle=\frac{|\psi_{H}\rangle|H\rangle+|\psi_{V}\rangle|V\rangle}{\sqrt{2}}$
(10)

Figure 1: The quantum SWITCH experiment realizing indefinite causal order of
measurements. See text for details.

After the PBS, the photon passes through the unitary channels performing the
measurements of $M_{i}$ and $M_{j}$ but indefinitely in causal order. This
action is the key to obtain the anti-commutation in Eq. (9). For our purpose
here, we consider a particular outcomes $m_{i(j)}$ of $M_{i(j)}$, i.e., we
realize the statistics corresponding to projectors $\pi_{m_{i(j)}}$.

After the action of phase-shifter (PS) along the channel $|\psi_{H}\rangle$
and recombination at BS followed by polarization measurement in $\\{+,-\\}$
basis, the state of the photon becomes

$\displaystyle|\Psi_{F}\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{2\sqrt{2}}\Big{[}\Big{(}\\{\pi_{m_{i}},\pi_{m_{j}}\\}|+\rangle+[\pi_{m_{i}},\pi_{m_{j}}]|-\rangle\Big{)}|\psi_{3}\rangle$
(11) $\displaystyle+$ $\displaystyle
i\Big{(}[\pi_{m_{j}},\pi_{m_{i}}]|+\rangle-\\{\pi_{m_{i}},\pi_{m_{j}}\\}|-\rangle\Big{)}|\psi_{4}\rangle\Big{]}$

where we have written the polarization in $\\{|+\rangle,|-\rangle\\}$ basis.
Further, the post-selection in the state $|+\rangle$ along the path
$|\psi_{3}\rangle$ or in the state $|-\rangle$ along the path
$|\psi_{4}\rangle$ provide the quasiprobability in Eq. (9), if
$|\psi_{i}\rangle$ is identified as $|+\rangle$. The detailed calculation of
Eq. (11) is given in Appendix B. Hence, the two-time LGI can be experimentally
tested while the operational non-invasiveness is satisfied.

Using the $q_{Q}(m_{i},m_{j})$ the pair-wise correlations $\langle
M_{i}M_{j}\rangle$ can also be obtained. The three-time LGI $K_{3}$ in Eq. (1)
can thus be tested when only pair-wise correlations are considered. Hence, the
quasiprobility $q_{Q}(m_{i},m_{j})$ enables the loophole-free test of two-time
LGI.

We now consider the case of three-time LGI involving triple-wise correlations.
For this, we introduce doubly Kirkwood quasiprobability distribution as

$q_{Q}(m_{1},m_{2},m_{3})=\frac{1}{2}Re\left[\langle m_{1}|m_{2}\rangle\langle
m_{2}|\rho|m_{3}\rangle\langle m_{3}|m_{1}\rangle\right]$ (12)

where $\\{m_{1}\\}$, $\\{m_{2}\\}$, $\\{m_{3}\\}$ are three different bases on
the same Hilbert space in which $\rho$ belongs. We can identify the bases
corresponding to the measurements of $M_{1}$, $M_{2}$ and $M_{3}$ in LG
scenario. For the pure state $\rho=|\psi\rangle\langle\psi|$, the
quasiprobability $q_{Q}(m_{1},m_{2},m_{3})$ can be written as

$\displaystyle
q_{Q}(m_{1},m_{2},m_{3})=\frac{1}{2}\frac{Re\left[q_{Q}(m_{1},m_{3})q^{\ast}_{Q}(m_{2},m_{3})\right]}{|\langle\psi|m_{3}\rangle|^{2}}$
(13)

It is straightforward to check that the sequential correlation remains same as

$\displaystyle\langle M_{j}M_{k}\rangle=$
$\displaystyle\sum\limits_{m_{i},m_{j},m_{k}}m_{j}m_{k}\
q_{Q}(m_{i},m_{j},m_{k})$ (14)
$\displaystyle=\sum\limits_{m_{j},m_{k}}m_{j}m_{k}\ p(m_{j},m_{k})$

where $j,k=1,2,3$ with $j<k$. Another important point property
$q_{Q}(m_{i},m_{j},m_{k})$ is that it satisfies all the operational non-
invasiveness conditions so that

$\displaystyle
p(m_{3})=\sum\limits_{m_{1},m_{2}}q_{Q}(m_{1},m_{2},m_{3})=Tr[|m_{3}\rangle\langle
m_{3}|\rho]$ (15) $\displaystyle
p(m_{2})=\sum\limits_{m_{1},m_{3}}q_{Q}(m_{1},m_{2},m_{3})=Tr[|m_{2}\rangle\langle
m_{2}|\rho]$ (16)

and

$\displaystyle p(m_{1},m_{2})$ $\displaystyle=$
$\displaystyle\sum\limits_{m_{2}}q_{Q}(m_{1},m_{2},m_{3})$ $\displaystyle=$
$\displaystyle Tr[|m_{1}\rangle\langle m_{1}|\rho|m_{1}\rangle\langle
m_{1}|m_{3}\rangle\langle m_{3}|]$ $\displaystyle p(m_{2},m_{3})$
$\displaystyle=$ $\displaystyle\sum\limits_{m_{1}}q_{Q}(m_{1},m_{2},m_{3})$
$\displaystyle=$ $\displaystyle Tr[|m_{2}\rangle\langle
m_{2}|\rho|m_{2}\rangle\langle m_{2}|m_{3}\rangle\langle m_{3}|]$

Note that $q_{Q}(m_{i},m_{j},m_{k})>0$ does not imply three-time LGI in Eq.
(1) as was the case for two-time LGIs. In recent work, Majdy et al. [46]
proposed a three-time LGI similar to the two-time LGI as

$\displaystyle G_{3}$ $\displaystyle=$
$\displaystyle\frac{1}{8}\big{(}1+3\sum\limits_{i=1}^{3}m_{i}\langle
M_{i}\rangle$ $\displaystyle+$ $\displaystyle\sum\limits_{i\neq
j=1}^{3}m_{i}m_{j}\langle M_{i}M_{j}\rangle+m_{1}m_{2}m_{3}\langle
M_{1}M_{2}M_{3}\rangle\big{)}$

This inequality is violated in quantum theory even when two-time and three-
time LGIs in Eqs. (1-3) are not violated. Importantly,
$q_{Q}(m_{i},m_{j},m_{k})>0$ implies the inequality Eq. (Leggett-Garg test of
macrorealism using indefinite causal order of measurements) with a
multiplicative factor $1/8$. Hence, a direct test of
$q_{Q}(m_{i},m_{j},m_{k})$ provides a conclusive test of three-time LGIs in
Eq. (Leggett-Garg test of macrorealism using indefinite causal order of
measurements). A simple modification of the setup presented in Fig. 1 will
serve this purpose. Note here that both weaker and stronger versions of
operational non-invasiveness are satisfied here.

One can also test the standard three-time LGI given by Eq. (1) by considering
a suitable combination of two three-time quasiprobabilities as

$q_{Q}(m_{i},-m_{j},m_{k})+q_{Q}(-m_{i},m_{j},-m_{k})\geq 0$ (20)

This means that the negativity of both of them provides the quantum violation
of three-time LGI, but the negativity of one of them does not guarantee it.
For testing the LGIs in Eq. (1), quasiprobabilities
$q_{Q}(m_{i},-m_{j},m_{k})$ and $q_{Q}(-m_{i},m_{j},-m_{k})$ has to be
separately measured. This can also be done by incorporating simple
modifications in the setup in Fig.1. We have not provided a detailed
description of such modification, but it is quite obvious.

In conclusion, we have provided a protocol to demonstrate the loophole-free LG
test of macrorealism through the quantum switch experiment that realizes the
indefinite causal order of measurements. Two-time LGIs in Eq. (3) three-time
LGIs provide the necessary and sufficient condition for macrorealism [16], but
the same is not valid for three-time LGIs [14]. Note that except for the CHSH
inequality, the Bell test does not provide the necessary and sufficient
condition for local realism. However, the violation of a Bell’s inequality
unequivocally warrants the violation of local realism. However, the same
argument does not hold for the LG test of macrorealism, as the violation of
LGIs can be attributed to the violation of operational non-invasiveness alone.
A skeptic can then always salvage the macrorealism _per se_. Hence, in
contrast to a Bell violation (where no-signaling in spatially separated
systems is always satisfied), the LG violation for macrorealism is
inconclusive unless the operational non-invasiveness loophole is fixed in the
experiment.

We stress again that in this work, we are not interested in whether LGIs
provide necessary and sufficient conditions. Instead, we are concerned whether
the LG test is conclusive enough to refute the LG notion of macrorealism in
quantum theory whenever we have the quantum violation of a given LGI. We
demonstrated that suitably defined quasiprobabilities - the ones that
reproduce the sequential correlations correctly and satisfy the operational
invasiveness- can provide a conclusive LG test of macrorealism while negative.
We demonstrated that such quasiprobabilities could be directly tested in a
quantum switch experiment that realizes the indefinite causal order of
measurements. Our measurement scheme thus closes the operational non-
invasiveness loophole so that the LG test becomes a conclusive test of
macrorealism when LGI is violated by quantum theory. This would then convince
a skeptic that macrorealism cannot be salvaged by abandoning the non-invasive
measurability condition and, indeed, incompatible with quantum theory. Our
scheme thus provides a decisive and conclusive test of macrorealism that the
LG test alluded to.

We may remark here that an alternate formulation for testing macrorealism -
the no-signaling in time conditions that provides necessary and sufficient
conditions for macrorealism for a three-time LG scenario. We agree with
Halliwell [16] that NSIT is more of a test of non-invasive measurability than
macrorealism _per se_. On the other hand, LGIs test both the assumptions
provided the operational non-invasiveness is guaranteed in the experiment.
Also, there remains debate regarding the relationship between the two
assumptions involved in the LG framework. As per Leggett’s viewpoint, the non-
invasive measurability implies macrorealism _per se_ and this view is
supported in [13]. However, Maroney and Timpson [12] argued that non-invasive
measurability condition is independent of macrorealism _per se_. Such kind of
independence is actually desired to test LGIs in a theory-independent
framework.from the viewpoint of the ontological model framework advocated by
Spekkens [48] However, this issue is not yet settled. Studies along this
direction could be an exciting avenue for future works.

##  Appendix A Derivation of Eq. (2)

The sequential correlation in Eq. (2) can be calculated as follows. Given a
density matrix $\rho$ the correlation function for the sequential measurement
of two observables $\hat{M}_{i}$ and $\hat{M}_{j}$ can be written as

$\displaystyle\langle\hat{M}_{i}\hat{M}_{j}\rangle_{seq}=\sum_{m_{i},m_{j}=\pm
1}{m_{i}m_{j}}P({m_{i}},{m_{j}})$ (21)

with $j>i$ and the joint probability of the sequential quantum measurement is
given by

$\displaystyle
P({m_{i}},{m_{j}})=Tr\left[\left(\sqrt{E_{M_{i}}^{m_{i}}}^{\dagger}\rho\sqrt{E_{M_{i}}^{m_{i}}}\right)\Pi_{M_{j}}^{m_{j}}\right]$
(22)

where $E_{M_{i}}^{m_{i}}$ are the positive-operator-valued-measures (POVMs)
corresponding to the measurement of a dichotomic $\hat{M}_{i}$ with $m_{i}=\pm
1$. In a sequential measurement, we may consider the measurement of
$\hat{M}_{j}$ is a projective measurement. If $\\{\Pi_{M_{j}}^{m_{j}}\\}$ are
the projectors of dichotomic observable $\hat{M}_{j}$ then
$\hat{M}_{j}=\Pi_{M_{j}}^{+}-\Pi_{M_{j}}^{-}$ with $m_{j}=\pm 1$.

By using Eq. (22) we can re-write Eq.(21) as,

$\displaystyle\langle\hat{M}_{i}\hat{M}_{j}\rangle_{seq}=\sum_{m_{i}=\pm
1}{m_{i}}Tr\left[\left(\sqrt{E_{M_{i}}^{m_{i}}}^{\dagger}\rho\sqrt{E_{M_{i}}^{m_{i}}}\right)\Pi_{M_{j}}^{+}\right]$
(23) $\displaystyle-$ $\displaystyle\sum_{m_{i}=\pm
1}{m_{i}}Tr\left[\left(\sqrt{E_{M_{i}}^{m_{i}}}^{\dagger}\rho\sqrt{E_{M_{i}}^{m_{i}}}\right)\Pi_{M_{j}}^{-}\right]$

Let us consider the POVMs corresponding to the measurement of $M_{i}$ as

$\displaystyle
E_{M_{i}}^{\pm}=\frac{1\pm\lambda}{2}\Pi_{M_{i}}^{\pm}+\frac{1\mp\lambda}{2}\Pi_{M_{i}}^{\mp}$
(24)

where $\lambda$ is the unsharpness parameter and $\Pi_{M_{i}}^{\pm}$ are
projectors of $\hat{M}_{i}$. Plugging Eq. (24) in Eq. (23), we get

$\displaystyle\langle\hat{M}_{i}\hat{M}_{j}\rangle_{seq}=\lambda\Big{(}Tr[\rho\left(\Pi_{M_{i}}^{+}{M_{j}}\
\Pi_{M_{i}}^{+}-\Pi_{M_{i}}^{-}{M_{j}}\ \Pi_{M_{i}}^{-}\right)]\Big{)}$

Putting $\Pi_{M_{i}}^{\pm}=(\mathbb{I}\pm\hat{M}_{i})/2$ for dichotomic
observable, Eq.(A) can be simplified as

$\displaystyle\langle\hat{M}_{i}\hat{M}_{j}\rangle_{seq}=\frac{\lambda}{2}Tr\left[\rho\left\\{\hat{M}_{i},\hat{M}_{j}\right\\}\right]$
(26)

which is the Eq. (2) in the main text. If the measurement of $M_{i}$ is sharp
we need to take $\lambda=1$.

##  Appendix B Explicit derivation of Eq. (11)

As considered in the main text let us consider the system state
$|+\rangle=\frac{1}{\sqrt{2}}\left(|H\rangle+|V\rangle\right)$ entering the MZ
setup and $|\psi_{H}\rangle$ and $|\psi_{V}\rangle$ are the control states so
that the application of indefinite causal order of unitaries can be written as

$\displaystyle\mathcal{U}_{ij}=U_{i}U_{j}\otimes|\psi_{H}\rangle\langle\psi_{H}|+U_{j}U_{i}\otimes|\psi_{V}\rangle\langle\psi_{V}|$
(27)

where $U_{i}$ and $U_{j}$ are the unitary operators corresponding to the
measurements of $M_{i}$ and $M_{j}$ respectively. The state
$\rho=|\Psi\rangle\langle\Psi|$ after PBS is given by Eq. (10) in the
manuscript. The state $\rho$ passes through $\mathcal{U}_{ij}$. We consider
the initial apparatus states are $|\xi_{i}\rangle$ and $|\xi_{i}\rangle$
corresponding to the measurements of $M_{i}$ and $M_{j}$ respectively. Hence
the total system-control-apparatus state can be written as
$\rho_{SA}=\rho\otimes|\xi_{i}\rangle\langle\xi_{i}|\otimes|\xi_{j}\rangle\langle\xi_{j}|$.
After the unitary operation as in Eq. (27) the state $\rho_{SA}$ becomes

$\displaystyle\rho^{\prime}_{SA}=\mathcal{U}_{ij}\ \rho_{SA}\
\mathcal{U}_{ij}^{\dagger}$ (28)

which can be explicitly written by using Eq. (27) as

$\displaystyle\rho^{\prime}_{SA}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\Big{[}U_{i}U_{j}|\xi_{i}\rangle\langle\xi_{i}|\otimes|\xi_{j}\rangle\langle\xi_{j}|\otimes|H\rangle\langle
H|U_{i}^{\dagger}U_{j}^{\dagger}\otimes|\psi_{H}\rangle\langle\psi_{H}|$ (29)
$\displaystyle+$
$\displaystyle\left(U_{i}U_{j}|\xi_{i}\rangle\langle\xi_{i}|\otimes|\xi_{j}\rangle\langle\xi_{j}|\otimes|H\rangle\langle
V|U_{j}^{\dagger}U_{i}^{\dagger}\otimes|\psi_{H}\rangle\langle\psi_{V}|\right)$
$\displaystyle+$
$\displaystyle\left(U_{j}U_{i}|\xi_{i}\rangle\langle\xi_{i}|\otimes|\xi_{j}\rangle\langle\xi_{j}|\otimes|V\rangle\langle
H|U_{i}^{\dagger}U_{j}^{\dagger}\otimes|\psi_{V}\rangle\langle\psi_{H}|\right)$
$\displaystyle+$ $\displaystyle
U_{j}U_{i}|\xi_{i}\rangle\langle\xi_{i}|\otimes|\xi_{j}\rangle\langle\xi_{j}|\otimes|V\rangle\langle
V|U_{j}^{\dagger}U_{i}^{\dagger}\otimes|\psi_{V}\rangle\langle\psi_{V}|\Big{]}$

Now, by considering the post-measurement apparatus states are
$|\xi(m_{i})\rangle$ and $|\xi(m_{j})\rangle$ for the measurements of $M_{i}$
and $M_{j}$ respectively, and by following the standard notion of generalized
quantum measurement theory we can define the measurement operators as
$N_{m_{i}}=\langle\xi(m_{i})|U_{i}|\xi_{i}\rangle$ and
$N_{{m_{j}}}=\langle\xi(m_{j})|U_{j}|\xi_{j}\rangle$. The POVMs can be written
as $E_{M_{i}}^{m_{i}}=N_{m_{i}}^{\dagger}N_{m_{i}}$ and
$E_{M_{j}}^{m_{j}}=N_{m_{j}}^{\dagger}N_{m_{j}}$. For particular outcomes of
$m_{i}$ and $m_{j}$, we can write the reduced system-control state as

$\displaystyle\rho^{\prime}_{S}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\Big{[}N_{i}N_{j}|H\rangle\langle
H|N_{i}^{\dagger}N_{j}^{\dagger}\otimes|\psi_{H}\rangle\langle\psi_{H}|$
$\displaystyle+$ $\displaystyle N_{i}N_{j}|H\rangle\langle
V|N_{j}^{\dagger}N_{i}^{\dagger}\otimes|\psi_{H}\rangle\langle\psi_{V}|$
$\displaystyle+$ $\displaystyle N_{j}N_{i}|V\rangle\langle
H|N_{i}^{\dagger}N_{j}^{\dagger}\otimes|\psi_{V}\rangle\langle\psi_{H}|$
$\displaystyle+$ $\displaystyle N_{j}N_{i}|V\rangle\langle
V|N_{j}^{\dagger}N_{i}^{\dagger}\otimes|\psi_{V}\rangle\langle\psi_{V}|\Big{]}$

which can also be written as
$\rho^{\prime}_{S}=|\Psi^{\prime}\rangle\langle\Psi^{\prime}|$ where

$\displaystyle|\Psi^{\prime}\rangle=\frac{1}{\sqrt{2}}\left((N_{i}N_{j}|\psi_{H}\rangle|H\rangle+(N_{j}N_{i}|\psi_{V}\rangle|V\rangle\right)$
(31)

For projective measurements we can take $N_{m_{i}}$ and $N_{m_{j}}$ as
$\pi_{m_{i}}$ and $\pi_{m_{j}}$ respectively. Now, after the application of a
phase-shifter (introducing $\pi$ phase shift) along $|\psi_{H}\rangle$ and the
recombination the state at second BS followed by a polarization measurement,
we get

$\displaystyle|\Psi_{F}\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{2\sqrt{2}}\Big{[}\Big{(}\\{\pi_{m_{i}},\pi_{m_{j}}\\}|+\rangle+[\pi_{m_{i}},\pi_{m_{j}}]|-\rangle\Big{)}|\psi_{3}\rangle$
(32) $\displaystyle+$ $\displaystyle
i\Big{(}[\pi_{m_{j}},\pi_{m_{i}}]|+\rangle-\\{\pi_{m_{i}},\pi_{m_{j}}\\}|-\rangle\Big{)}|\psi_{4}\rangle\Big{]}$

which is the Eq. (11) in the main text. To get Eq. (32), we take
$|\psi_{H}\rangle=\frac{1}{\sqrt{2}}\left(|\psi_{3}\rangle+i|\psi_{4}\rangle\right)$
and
$|\psi_{V}\rangle=\frac{1}{\sqrt{2}}\left(i|\psi_{3}\rangle+|\psi_{4}\rangle\right)$
for the BS along with
$|H\rangle=\frac{1}{\sqrt{2}}\left(|+\rangle+|-\rangle\right)$ and
$|V\rangle=\frac{1}{\sqrt{2}}\left(|+\rangle-|-\rangle\right)$ for
polarization measurement, as shown in Fig. 1.

Finally, if the post-selection is performed in the state
$|\psi_{3}\rangle|+\rangle$, we get the quasiprobability
$q_{Q}(m_{i},m_{j})=\frac{1}{2}\langle+|\\{\pi_{m_{i}},\pi_{m_{j}}\\}|+\rangle$
with a multiplicative factor $1/\sqrt{2}$.

###### Acknowledgements.

The author is indebted to Prof. J. J. Halliwell for many stimulating email
exchanges on LGIs over the years and acknowledges the support from the
research grant DST/ICPS/QuEST/2019/4.

## References

  * [1] W. Heisenberg, Zeitschrift f$\ddot{u}$r Physik, 33, 879 (1925).
  * [2] E. Schroedinger, Naturwissenschaften, 23, 807 (1935).
  * [3] H. D. Zeh, Found. Phys. 1, 69 (1970), W. H. Zurek, Phys. Rev. D 26, 1862 (1982).
  * [4] J. Kofler and C. Brukner, Phys. Rev. Lett. 99, 180403 (2007).
  * [5] G. C. Ghirardi, A. Rimini, and T. Weber, Phys. Rev. D 34, 470 (1986).
  * [6] A. Bassi and G. Ghirardi, Phys. Rep. 379, 257 (2003).
  * [7] M. Arndt et al., Nature, 401, 680 (1999).
  * [8] S. Gerlich et al., Nature Comm. 2, 263 (2011).
  * [9] A. J. Leggett and A. Garg, Phys. Rev. Lett. 54, 857 (1985).
  * [10] A. J. Leggett, J. Phys.: Condens. Matter 14, R415 (2002).
  * [11] N. S. Williams and A. N. Jordan, Phys. Rev. Lett. 100, 026804 (2008).
  * [12] O. Maroney and C. Timpson, arXiv: 1412.6139.
  * [13] L. Clemente and J. Kofler, Phys. Rev. A, 91, 062103 (2015).
  * [14] L. Clemente and J. Kofler, Phys. Rev. Lett. 116, 150401 (2016).
  * [15] J. J. Halliwell, Phys. Rev. A 94, 052114 (2016).
  * [16] J. J. Halliwell, Phys. Rev. A, 93, 022123 (2016).
  * [17] S. Kumari and A. K. Pan , EPL, 0002, 118 (2017).
  * [18] J. J. Halliwell, Phys. Rev. A , 96, 012121 (2017).
  * [19] S. Kumari and A. K. Pan, Phys. Rev. A 96, 042107(2017).
  * [20] A. Kumari, Md. Qutubuddin and A. K. Pan,Phys. Rev. A 98, 042135 (2018).
  * [21] S. Bose, D. Home and S. Mal, Phys. Rev. Lett., 120, 210402 (2018).
  * [22] A. K. Pan, Md. Qutubuddin and S. Kumari, Phys. Rev. A, 98, 062115(2018).
  * [23] R. Uola, G. Vitagliano and C. Budroni, Phys. Rev. A 100, 042117 (2019).
  * [24] J. J. Halliwell,Phys. Rev. A 99, 022119 (2019).
  * [25] J. J. Halliwell and C. Mawby, Phys. Rev. A 100, 042103 (2019).
  * [26] C. Robens, W. Alt, D. Meschede, C. Emary, and A. Alberti Phys. Rev. X 5, 011003 (2015).
  * [27] A. Palacios-Laloy et al., Nat. Phys. 6, 442 (2010).
  * [28] J. Dressel, C. J. Broadbent, J. C. Howell, and A. N. Jordan Phys. Rev. Lett. 106, 040402, (2011).
  * [29] M. E. Goggin et al., Proc. Natl. Acad. Sci. USA 108,1256 (2011).
  * [30] G. C. Knee _et al.,_ , Nature Commun. 7, 13253 (2016).
  * [31] A. Avella _et al.,_ Phys. Rev. A, 96, 052123 (2017).
  * [32] K. Joarder, D. Saha, D. Home and U. Sinha, arXiv: 2105.11881.
  * [33] C. Emary, N. Lambert, and F. Nori, Rep. Prog. Phys. 77, 016001 (2014).
  * [34] M. Wild and A. Mizel, Found. Phys. 42, 256 (2012).
  * [35] Y. Aharonov, D. Albert, L. Vaidmann, Phys Rev. Lett. 60, 1351 (1998).
  * [36] A.K. Pan, Phys. Rev. A 102, 032206 (2020).
  * [37] C. Ferrie, Rep. Prog. Phys. 74, 116001 (2011).
  * [38] J. S. Lundeen and C. Bamber, Phys. Rev. Lett. 108, 070402 (2012).
  * [39] D. R. Arvidsson-Shukur, N. Y. Halpern, H. V. Lepage, A. A. Lasek, C. H. Barnes, and S. Lloyd, Nat. Commun. 11, 1 (2020).
  * [40] R. Spekkens, Phys. Rev. Lett. 101, 020401 (2008).
  * [41] J. Dressel, Phys. Rev. A 91, 032116 (2015).
  * [42] E. P. Wigner, Phys. Rev. 40, 749 (1932).
  * [43] J. G. Kirkwood, Phys, Rev. 44, 31 (1933).
  * [44] H. Margenau and R. N. Hill, Prog. Theor. Phys. 26, 722 (1961).
  * [45] l. M. Johansen, Phys. Rev. A 76, 012119 (2007).
  * [46] S-S. Majidy, J. J. Halliwell and R. Laflamme, Phys. Rev. A 103, 062212 (2021).
  * [47] J. J. Halliwell, A. Bhatnagar, E. Ireland, H. Nadeem, and V. Wimalaweera, Phys. Rev. A 103, 032218 (2021).
  * [48] R. W. Spekkens, Phys. Rev. A71, 052108 (2005).
  * [49] G. Chiribella, G. M. D’Ariano, P. Perinotti, and B. Valiron, Phys. Rev. A 88, 022318 (2013).
  * [50] K. Goswami, C. Giarmatzi, M. Kewming, F. Costa, C. Branciard, J. Romero, and A. G. White, Phys. Rev. Lett. 121 090503 (2018).
  * [51] Y. Guo _et al._ , Phys. Rev. Lett. 124 030502 (2020).
  * [52] M. Ban, Phys. Lett. A, 403, 127383 (2021).
  * [53] S-S. Majidy, H. Katiyar, G. Anikeeva, J. Halliwell, and R. Laflamme, Phys. Rev. A 100, 042325 (2019).