[1]

[cor1]Corresponding author: Hongbing Ji

# Cluster-CAM: Cluster-Weighted Visual Interpretation of CNNs’ Decision in
Image Classification

Zhenpeng Feng School of Electronic Engineering, Xidian University, Xi’an,
China  Hongbing Ji<EMAIL_ADDRESS>Miloš Daković Xiyang Cui Mingzhe Zhu
Ljubiša Stanković Faculty of Electrical Engineering, University of Montenegro,
Podgorica, Montenegro

###### Abstract

Despite the tremendous success of convolutional neural networks (CNNs) in
computer vision, the mechanism of CNNs still lacks clear interpretation.
Currently, class activation mapping (CAM), a famous visualization technique to
interpret CNN’s decision, has drawn increasing attention. Gradient-based CAMs
are efficient while the performance is heavily affected by gradient vanishing
and exploding. In contrast, gradient-free CAMs can avoid computing gradients
to produce more understandable results. However, existing gradient-free CAMs
are quite time-consuming because hundreds of forward interference per image
are required. In this paper, we proposed Cluster-CAM, an effective and
efficient gradient-free CNN interpretation algorithm. Cluster-CAM can
significantly reduce the times of forward propagation by splitting the feature
maps into clusters in an unsupervised manner. Furthermore, we propose an
artful strategy to forge a cognition-base map and cognition-scissors from
clustered feature maps. The final salience heatmap will be computed by merging
the above cognition maps. Qualitative results conspicuously show that Cluster-
CAM can produce heatmaps where the highlighted regions match the human’s
cognition more precisely than existing CAMs. The quantitative evaluation
further demonstrates the superiority of Cluster-CAM in both effectiveness and
efficiency.

###### keywords:

keywords-1 explainable artificial intelligencekeywords-2 class activation
mapping keywords-3 clustering algorithm keywords-4 image classification

##  1 Introduction

Convolutional neural networks (CNNs) have provided a basis for numerous
remarkable achievements in various computer vision tasks like, for example,
image classification [9, 8, 27, 13], object detection [19, 43, 1, 12], and
semantic segmentation [14, 11, 3]. Despite CNNs’ extraordinary performance,
they still lack a clear interpretation of the inner mechanism [10, 42, 21].
This lack of transparency can indeed be a disqualifying factor in some
peculiar scenarios where mistakes in interpretation can jeopardize human life
and health, like in medical image processing or autonomous vehicles [20, 38,
34, 32]. Therefore, it is highly desirable to find a way to understand and
explain what exactly CNNs have learned during the training process [26, 31,
16].

Recently, Class Activation Mapping (CAM), a visual interpretation technique,
has drawn increasing attention [30, 33]. CAM aims at highlighting salience
regions of an input image for CNN’s decision using a linearly weighted
combination of feature maps. Vanilla CAM directly utilizes the weight of each
feature map after global average pooling (GAP) corresponding to the target
class, so it is only available for CNNs with GAP [40]. To further extend CAM
to more complex CNN structures, numerous modified CAMs are proposed and they
can be broadly categorized as: 1) gradient-based CAMs, and 2) gradient-free
CAMs. Gradient-based CAMs (e.g. Grad-CAM [23], Grad-CAM++ [2], SmoothCAM++
[17], XGrad-CAM [7] etc.) define the weights of each feature map using the
average of partial gradient of the predicted score with respect to the feature
maps. Gradient-based CAMs are usually computed efficiently. However, their
weights lack reasonable explanation and are easily impacted by gradient
exploding or vanishing. To address this limitation, some gradient-free CAMs
are proposed. They define an intuitive impact of each feature map on the
predicted score instead of using the gradient. Examples of gradient-free CAMs
are the Ablation CAM [18] and the Score-CAM [35]. Gradient-free CAMs can
provide a more explainable weight definition than gradient-based CAMs in most
cases. Fig. 1 shows the salience heatmaps produced by several aforementioned
CAMs.

Although gradient-free CAMs define the weights more reasonably, they are
usually very time-consuming since hundreds of forward propagations per image
are required. To improve the efficiency of gradient-free CAMs, Q. Zhang et al.
proposed Group-CAM where feature maps are split into several groups [37]. In
this case, only several forward propagations are needed in computing the
weights. Nonetheless, the feature maps are split without any regulation in the
Group-CAM. Actually, various feature maps have learned different semantic
concepts relevant/irrelevant to the object. Therefore, the feature maps should
be split into groups/clusters. Those with similar semantics should be assigned
to the same group. Z. Feng et al. proposed SC-SM CAM using spectral clustering
to accomplish this goal, particularly for synthetic aperture radar (SAR)
images [5]. However, no further analysis or modification of weights is
mentioned in the SC-SM CAM [6, 5].

In this paper, we propose a Cluster-CAM, an effective and efficient gradient-
free CAM, based on unsupervised clustering. In Cluster-CAM, an unsupervised
clustering technique, K-means/spectral clustering is adopted to split feature
maps into several clusters. Subsequently, we provide an artful strategy to
merge those feature maps into a cognition-base map and a cognition-scissors
map which will be combined as the final salience heatmap. The highlights of
this paper are as follows:

  * •

We propose a Cluster-CAM, as the first attempt to provide a cluster-weighted
CAM framework via unsupervised clustering based on optical images.

  * •

We provide a novel and artful weight-forming strategy to merge the cognition-
base map and cognition-scissors map. These two maps greatly match the human’s
cognition and intuition, thus the weights are completely reasonable and
understandable.

  * •

Cluster-CAM is effective and efficient, which outperforms existing gradient-
free CAMs in performance in most cases with significantly lower computing
costs.

The rest of this paper is organized as follows. Section 2 introduces the basic
knowledge of various CAMs. Section 3 elaborates on how to generate salience
heatmaps by Cluster-CAM. In Section 4, various experiments are implemented to
demonstrate the validity of Cluster-CAM and further analyze the experimental
results from various aspects. Section 5 concludes this paper.

##  2 Related Work

As discussed in Section 1, the key issue in interpreting CNN’s decision is to
explain what the neural network learned to finish a reasonable inference [42,
10]. To visualize what CNN focuses on the input image, numerous interpretation
algorithms are proposed [36, 24, 40, 39], among which Class Activation Mapping
draws the most increasing attention due to its simplicity and good
performance.

Vanilla CAM: B. Zhou et al. firstly proposed the vanilla CAM to produce a
salience heatmap by a linearly weighted combination of feature maps,
$\mathbf{F}_{n}$, with elements $F_{n}(i,j),\ n=1,2,...,N$, at the target
convolutional layer before classification [40]

$\displaystyle{M}_{c}(i,j)=\sum_{n}\alpha^{c}_{n}F_{n}(i,j)$ (1)
$\displaystyle
S_{c}=\sum_{n}\omega^{c}_{n}\sum_{i,j}F_{n}(i,j)=\sum_{n}\alpha^{c}_{n}\sum_{i,j}F_{n}(i,j),$
(2)

where ${M}_{c}(i,j)$ is a heat-map and $S_{c}$ denotes the predicted score for
the target class $c$. Thus, $\alpha^{c}$ are defined by the weights,
$\omega^{c}$, of each feature map corresponding to $c$-th unit in the
classification layer. Therefore, CAM is only available with CNNs with global-
average pooling following the last convolutional layer. To extend CAM to all
CNNs, many modified CAMs are further proposed by manipulating the definition
of weights, which are generally categorized as: 1) gradient-based CAMs; 2)
gradient-free CAMs.

###  2.1 Gradient-based Class Activation Mapping

Gradient-based CAM: Selvaraju et al. proposed Grad-CAM to visualize any
classification CNN architectures by weighting the feature maps in a certain
convolutional layer with the gradients of the predicted score,
$\mathbf{s}_{c}$ with respect to the elements of $\mathbf{F}_{n}$ [23], as

$\displaystyle\alpha^{c,Grad}_{n}=\sum_{i,j}\frac{\partial\mathbf{s}_{c}}{\partial{F}_{n}(i,j)},$
(3)

where different from (2), $\mathbf{s}_{c}$ is a sparse vector whose elements
are zeros except the $c$-th element, which is equal to $S_{c}$. However, the
highlighted regions generated by Grad-CAM are usually much smaller than the
object. To provide a precise highlighted region, some further modified CAMs
are proposed, like Grad-CAM++[2]. A. Chattopadhyay et al proposed Grad-CAM++
which can produce more precise highlighted locality. Grad-CAM++ assumes
different elements in the gradient matrix should have different contributions
to features maps, thus an extra factor is introduced to realize this
assumption using higher order partial gradient, as:

$\boldsymbol{\alpha}^{c,Grad++}_{n}=\frac{\frac{\partial^{2}\mathbf{s}_{c}}{\partial({F}_{n}(i,j))^{2}}}{2\frac{\partial^{2}\mathbf{s}_{c}}{\partial({F}_{n}(i,j))^{2}}+\sum_{a,b}{F}_{n}(a,b)\frac{\partial^{3}\mathbf{s}_{c}}{({F}_{n}(i,j))^{3}}}\sum_{i,j}\frac{\partial\mathbf{s}_{c}}{\partial{F}_{n}(i,j)}.$
(4)

However, the gradient,
$\frac{\partial\mathbf{s}_{c}}{\partial\mathbf{F}_{n}}$, is usually heavily
noised or somtimes even all-zero. It is probably because 1) CNN is trained to
learn a generalized capability to classify a general concept rather than a
specific object. 2) some unreasonable phenomena emerged in CNN’s training,
like gradient vanishing and gradient exploding. D. Omeiza et al. proposed
Smooth Grad-CAM++ to further suppress the noise. The weights of Smooth Grad-
CAM++ are defined using the average of the gradients as:

$\displaystyle\alpha^{c,SmoothGrad++}_{n}=$ (5)
$\displaystyle\frac{\frac{1}{m}\sum_{m=1}^{M}D_{1}^{n}}{2\frac{1}{m}\sum_{m=1}^{M}D_{2}^{n}+\sum_{a,b}F_{n}(a,b)\frac{1}{m}\sum_{m=1}^{M}D_{3}^{n}}(\frac{1}{m}\sum_{m=1}^{M}D_{1}^{n})$

where $D_{1}^{n}=\sum_{i,j}\frac{\partial\mathbf{s}_{c}}{\partial
F_{n}(i,j)}$,
$D_{2}^{n}=\sum_{i,j}\frac{\partial^{2}\mathbf{s}_{c}}{\partial(F_{n}(i,j))^{2}}$,
and
$D_{3}^{n}=\sum_{i,j}\frac{\partial^{3}\mathbf{s}_{c}}{\partial(F_{n}(i,j))^{3}}$
when the input is added with random noise for $M$ times ($M$ is a constant
integer). This smoothing strategy is intuitive but still rough for some
complex CNN structures. To further enhance the interpretability of weights, R.
Fu et al. proposed XGrad-CAM by introducing two completely explainable axioms
to form the weight:

$\displaystyle\alpha^{c,XGrad}_{n}=\sum_{i,j}\frac{{F}_{n}(i,j)}{\sum_{i,j}{F}_{n}(i,j)}\frac{\partial\mathbf{s}_{c}}{\partial{F}_{n}(i,j)}.$
(6)

Note neither Smooth Grad-CAM++ nor XGrad CAM can guarantee completely avoiding
the above unreasonable phenomena in gradient computing.

Figure 1: The heatmaps are produced by different CAMs. The first column is
input images (indigo finch, eagle, rooster, and ostrich from top to bottom).
The second to fifth columns are heatmaps produced by Grad-CAM, Grad-CAM++,
Ablation-CAM, Score-CAM, and Cluster-CAM.

###  2.2 Gradient-free Class Activation Mapping

Gradient-free CAM:To completely solve the problems resulting from gradient
computing, some gradient-free CAMs are proposed, i.e., Ablation-CAM [18] and
Score-CAM [35], to form the weights using the impact of each feature map on
the predicted score instead of using gradient. In Ablation-CAM, the weights
are defined as:

$\alpha^{c,Ablation}_{n}=\frac{S_{c}-S_{c,n}}{S_{c}},$ (7)

where $S_{c,n}$ denotes the predicted score for class $c$ when $n$-th feature
map is set to zero. In this case, a large weight will be assigned to the
current feature map if the removal of it can lead to a dramatic drop in the
predicted score ($S_{c}-S_{c,n}$ is a large value) and vice versa. The authors
argue that Ablation-CAM is immune to both saturation which marks a filter as
important although it is not important, and explosion which marks a filter
that has a very small influence as having high importance. Different from
Ablation-CAM, Score-CAM considers measuring the impact of the feature map by
introducing the input image, $\mathbf{X}$, as

$\displaystyle\alpha^{c,Score}_{n}$
$\displaystyle=S_{c}(\mathbf{X}~{}\circ~{}\mathbf{H}_{n})-S_{c}(\mathbf{X}_{b})$
(8) $\displaystyle\mathbf{H}_{n}$
$\displaystyle=s(\mathrm{U_{p}}(\mathbf{F}_{n})),$ (9)

where $\circ$ denotes the element-wise multiplication, $\mathbf{X}_{b}$ is a
baseline image which can be set the input image itself,
$\mathrm{U_{p}}(\cdot)$ denotes the operation that upsamples $\mathbf{F}_{n}$
into the input size and $s(\cdot)$ is a normalization function that maps each
element in the input matrix into [$0$, $1$].
$\mathbf{X}~{}\circ~{}\mathbf{H}_{n}$ can be deemed as filtering which only
passes elements in $\mathbf{X}$ masked by $\mathbf{H}_{n}$, thus a large
weight will be assigned if most target-discriminative are preserved by the
current feature map, i.e. $f(\mathbf{X}~{}\circ~{}\mathbf{H}_{n})$ is higher
than $S_{c}(\mathbf{X}_{b})$ and vice versa. Currently, gradient-free CAMs
have drawn more attention than gradient-based CAMs due to their superior
performance and explainable definition of weights. However, gradient-free CAMs
are much more time-consuming than gradient-based CAMs because hundreds or even
thousands of forward interference are required while those gradient-based CAMs
only require one forward interference.

##  3 Methodology

In this section, we will first introduce some basic concepts on graph-based
spectral clustering and K-means. Then we present the detailed procedures of
Cluster-CAM.

Figure 2: The flowchart of Cluster-CAM.

###  3.1 Spectral Clustering and K-means

Spectral clustering is a widely-used unsupervised clustering algorithm based
on graph signal processing [28, 29, 15, 22]. Specifically, the processed data
(feature maps, $\mathbf{F}_{n}$) are regarded as vertices in a graph topology.
Then the elements, ${S}(i,j)$, of the similarity matrix, $\mathbf{S}$, can be
defined:

$\displaystyle{S}(i,j)=\mathrm{similarity}(\mathbf{F}_{i},\mathbf{F}_{j}),$
(10)

where $\mathrm{similarity}(\cdot)$ refers to a function that measures the
similarity between two vertices (feature maps). If we use the structural
similarity index (SSIM), then it ranges from $0$ (no similarity) to $1$
(identical feature maps). The elements of the weighted adjacency matrix,
$\mathbf{A}$, can be defined as:

$\left\\{\begin{array}[]{lr}{A}(i,j)=\exp(-(1-S(i,j))/\sigma),~{}~{}\text{if}~{}~{}{S}(i,j)>\theta,&\\\
{A}(i,j)=0,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{else,}&\end{array}\right.$
(11)

where $\theta$ is a threshold to keep the direct edge in the corresponding
graph for two neighboring vertices and $\sigma$ is a parameter. Note that, by
definition of similarity, this adjacency matrix is a symmetric matrix
resulting in an undirected graph, that is, ${A}(i,j)={A}(j,i)$.

The similarity can be defined using the difference between two vertices
(feature maps), $d(i,j)=||\mathbf{F}_{i}-\mathbf{F}_{i}||$. Then the weighted
adjacency matrix is defined by

$\left\\{\begin{array}[]{lr}{A}(i,j)=\exp(-d^{2}(i,j)/\sigma^{2}),~{}~{}\text{if}~{}~{}{S}(i,j)>\theta,&\\\
{A}(i,j)=0,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{else,}&\end{array}\right.$
(12)

where $\theta$ and $\sigma$ have the same role as in (11).

In order to produce the vectors for spectral clustering, now we continue and
compute the graph Laplacian matrix, $\mathbf{L}$, as

$\displaystyle\mathbf{L}=\mathbf{D}-\mathbf{A}$ (13)

where ${D}(i,i)=\sum_{j}{A}(i,j)$ are the elements of the degree matrix
$\mathbf{D}$ which is diagonal.

In practice, the graph Laplacian matrix usually can be normalized, as

$\displaystyle\mathbf{L}_{N}=\mathbf{D}^{-\frac{1}{2}}\mathbf{L}\mathbf{D}^{-\frac{1}{2}}=\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}.$
(14)

The clustering results obtained using these two matrices are very similar.

The eigendecomposition of the graph Laplacian

$\displaystyle\mathbf{L}=\mathbf{U}^{T}\boldsymbol{\Lambda}\mathbf{U}.$ (15)

Results in eigenvectors $\mathbf{u}_{1}$, $\mathbf{u}_{2}$, $\cdots$,
$\mathbf{u}_{N}$ that are the columns of matrix $\mathbf{U}$. The smoothness
index of these vectors is equal to the corresponding eigenvalue $\lambda_{i}$.
In clustering the data into two clusters only the eigenvector $\mathbf{u}_{2}$
is used (Fiedler vector) since the vector $\mathbf{u}_{1}$ is omitted as its
elements are constant.

If we want to get a few clusters ($Q$ clusters) then we can use $K$ the
smoothest eigenvectors, $\mathbf{u}_{2}$, $\mathbf{u}_{3}$ $\cdots$,
$\mathbf{u}_{K+1}$, written in the matrix form as

$\mathbf{B}=\begin{bmatrix}\mathbf{u}_{2}&\mathbf{u}_{3}&\cdots&\mathbf{u}_{K+1}\end{bmatrix}=\begin{bmatrix}u_{12}&u_{13}&\cdots&u_{1(K+1)}\\\
u_{22}&u_{23}&\cdots&u_{2(K+1)}\\\ \vdots&\vdots&\ddots&\vdots\\\
u_{N2}&u_{N3}&\cdots&u_{N(K+1)}\end{bmatrix},$ (16)

where $N$ features with $K$ dimension are considered.

The clusters are determined based on the $K$-dimensional spectral similarity
vectors, $\mathbf{q}_{1}=[u_{12},\ u_{13},\ldots,u_{1(K+1)}]$,
$\mathbf{q}_{2}=[u_{22},\ u_{23},\ldots,u_{2(K+1)}]$, $\ldots$,
$\mathbf{q}_{N}=[u_{N2},\ u_{N3},\ldots,u_{N(K+1)}]$, defined for vertices
(features $\mathbf{F}_{1}$, $\mathbf{F}_{2}$, $\ldots$ , $\mathbf{F}_{N}$).

In this way, the dimension of the measuring distance is significantly reduced
from the original $N$ dimensional space in
$d(i,j)=||\mathbf{F}_{i}-\mathbf{F}_{i}||$ to a very low $K$-dimensional
spaces of spectral vectors $\mathbf{q}_{n}$.

Finally, the clustering result (the data grouped into $Q$ clusters) can be
refined using K-means and the Euclidean distance
$d(i,j)=||\mathbf{F}_{i}-\mathbf{F}_{i}||$.

Note that the traditional K-means algorithm can be used with an initial random
clustering of feature maps into $Q$ clusters, with a slower convergence due to
random initialization. In this case, all the feature maps are grouped into $Q$
initial clusters, $\mathbb{Q}_{q}$, $q=1,2,\dots,Q$. Means of the feature maps
are calculated for each cluster,
$M_{q}=\mathrm{mean}(\mathbf{F}_{n},n\in\mathbb{Q}_{q})$. The distance of each
feature map is checked with respect to each of the mean $M_{q}$. The feature
map is reassigned to the cluster whose mean is the closest to the considered
feature map. After all feature maps are considered, the means are recalculated
for the new clusters. The procedure is repeated until no feature map changes
its cluster.

###  3.2 Cluster-CAM

Now we are ready to introduce spectral clustering and K-means in Cluster-CAM.
Here the feature maps, $\mathbf{F}$, represent the vertices in (10). Take
Euclidean distance as similarity measurement, (10) can be expressed as:

$\displaystyle{S}(i,j)=\exp{\\{-||\mathbf{F}_{i}-\mathbf{F}_{j}||}\\},$ (17)

where a shorter distance means a higher similarity. By substituting (17) into
(11), (13), (14), and (16), we can split $N$ feature maps into $Q$ clusters,
$\mathbb{Q}_{q}$, $q=1,2,\dots,Q$, $Q\ll N$. Then we can obtain the $Q$
representative feature maps,
$\tilde{\mathbf{F}}=[\mathbf{\tilde{F}}_{1},\mathbf{\tilde{F}}_{2},\ldots,\mathbf{\tilde{F}}_{Q}]$,
by calculating the mean of feature maps in each cluster, as

$\displaystyle\mathbf{\tilde{F}}_{q}=\mathrm{mean}\\{\mathbf{F}_{n},~{}n\in\mathbb{Q}_{q}\\},\
\ ~{}q=1,\cdots,Q.$ (18)

Next we obtain the Hadamard product of $\mathbf{\tilde{F}}$ and $\mathbf{X}$
($\mathbf{\tilde{F}}$ will be upsampled to the same size of $\mathbf{X}$).
This processing can be deemed as filtering that mainly passes those elements
corresponding to large values in $\mathbf{\tilde{F}}$. The predicted score of
each masked image is computed as:

$\displaystyle\mathbf{y}=$ $\displaystyle[y_{1},y_{2},\ldots,y_{Q}]^{T}$
$\displaystyle=$
$\displaystyle[S_{c,1}(\mathbf{\tilde{F}}_{1}~{}\circ~{}\mathbf{X}),\ldots,S_{c,Q}(\mathbf{\tilde{F}}_{Q}~{}\circ~{}\mathbf{X})].$
(19)

In this case, we can obtain the cognition-base map and cognition scissors as

$\displaystyle\mathbf{\tilde{F}}_{\text{base}}$
$\displaystyle=\mathbf{F}_{q_{\max}},\quad
q_{\max}=\mathop{\arg}\limits_{q}\max(\mathbf{y})$ (20)
$\displaystyle\mathbf{\tilde{F}}_{\text{scissors}}$
$\displaystyle=\mathbf{F}_{q_{\min}},\quad
q_{\min}=\mathop{\arg}\limits_{q}\min(\mathbf{y}).$ (21)

Next, we can semantically couple the cognition-base map and cognition-scissors
to form the salience heatmap, as:

$\displaystyle\mathbf{H}^{Cluster}=\beta\mathbf{\tilde{F}}_{\text{base}}-(1-\beta)\mathbf{\tilde{F}}_{\text{scissors}},$
(22)

where $\beta\in[0,1]$ is a balance factor to adjust the importance of
cognition-base map and cognition-scissors.

##  4 Experiments

In this section, we will present and analyze the performance of Cluster-CAM
from various perspectives. Firstly we will briefly describe the dataset used
in our experiments. Then we verify the superiority of Cluster-CAM to other
existing CAMs.

###  4.1 Experimental Setup

Dataset: In the following experiments, CNNs are trained on a prevalent
benchmark, i.e., ILSVRC [4]. In ILSVRC, there are around 1.2 million images
with $1000$ categories for training, and $50$ thousand images with $1000$
categories for validation.

Network Structure: In this paper, several classic CNNs, AlexNet [9] and VGG-16
[25], are used as classification models. Alex-Net is proposed by A. Krizhevsky
et al., which consists of $5$ convolutional units (a stack of convolutional
layers, ReLU, and max pooling) and $3$ fully-connected layers. VGG-16 is a
very deep CNN with $13$ convolutional layers and $3$ fully-connected layers.
VGG-16 has approximately 134M trainable parameters regardless of the output
layer.

Figure 3: The analysis of feature maps for images of multiple objects. The
heatmaps produced by different CAMs for the target label elkhound (first row
in the top-left subfigure). The clustered feature maps ( the second row in the
top-left subfigure) and corresponding masked images (the third row in the top-
left subfigure). Note that the cognition-base map and cognition-scissors are
marked by green and red squares, respectively. The predicted score for the
current class is provided above each masked image. The results with the target
label spaniel are organized in the same structure (top-right subfigure). The
indigo finch and goldfinch are shown in the bottom-left and bottom-right
subfigures, respectively.

###  4.2 Performance of Discriminative Localization

Fig. 1 shows the salience heatmaps of different input images (indigo finch,
eagle, rooster, and ostrich) by Grad-CAM, Grad-CAM++, Ablation-CAM, Score-CAM,
and Cluster CAM. Visually, in comparison to existing CAMs, the highlighted
region produced by Cluster-CAM mostly matches human’s intuitive understanding
of the discriminative part of the specific object. Take the indigo finch as an
example, Grad-CAM only highlights the head of the bird, whereas Grad-CAM++ and
Ablation-CAM highlight the complete finch but the branch (object-irrelevant
information) is also included. Score-CAM and Cluster-CAM highlight the finch
body without the branch. But obviously, the region produced by Cluster-CAM
matches the profile of the finch more precisely than Score-CAM.

###  4.3 CNN’s Cognitive Explanation

Cognition Analysis of Multi-objects Images: Images of multiple objects are
optimal samples to verify the rationality of the cognition-base map and
semantic-scissors in (20) and (21). As we discussed in Section 3, a reasonable
cognition-base map should incorporate the object-relevant information as much
as possible, while the corresponding cognition-scissors should include such
information as less as better. Therefore, it is necessary to check whether the
cognition-base map and cognition-scissors can interchange a multi-objects
image if the target class is changed to another object. Fig. 3 shows the
cognition-base map and cognition-scissors of two muti-objects images as well
as the corresponding masked images. There are two types of dogs in the first
image, i.e. elkhound (the big gray dog on the left) and spaniel (the tiny
brown dog on the right). If the target class is elkhound, the third and the
fourth clustered feature map are cognition-base map and cognition-scissors,
respectively (marked by green and red squares). It matches human’s cognition
because cognition-base map incorporates both objects and cognition-scissors
only selects the spaniel, thus the highlighted region will only be
concentrated on the elkhound, as shown in the first row in the top-left
subfigure in Fig. 3. When the target class is changed to the spaniel, the
cognition-base map and cognition-scissors are also interchanged, as shown in
the third row in the top-right subfigure in Fig. 3. The same phenomenon also
emerges in indigo finch and goldfinch, as shown in the bottom subfigures in
Fig. 3. In this case, Cluster-CAM provides solid evidence that CNN’s
recognition mechanism is similar to human cognition in multiple objects
classification.

Figure 4: The cognition-base map (green square) and cognition-scissors (red
square) in merged feature maps (top). The images are masked by corresponding
feature masks as well as the predicted score (middle). Nine images of guenon
and nine images of langur (bottom). Note that the discriminative
characteristics of guenon (golden hair and white mouth) and langur (black
face) are labeled with green and red circles, respectively.

Cognition Analysis of Fine-grained Images: To further understand how CNN
utilizes the learned information to make decisions, we can use CAM to
interpret CNNs in fine-grained image classification. Fine-grained
classification aims to distinguish subordinate categories within entry-level
categories. Examples include recognizing species of birds such as northern
cardinal or indigo bunting; monkeys such as guenon or langur. Fine-grained
classification often requires much more detailed information compared with
generic object recognition, like the texture of the skin, the thickness of the
fur, etc, so CAMs on fine-grained images can tell whether the information is
reasonably learned by CNN for classification. Fig. 4 shows the heatmaps
generated by several mentioned CAMs given the input image of a guenon in the
first row. Interestingly, they focus on completely different parts of the
guenon. Grad-CAM and Grad-CAM++ highlight the guenon’s eyes and cheek,
respectively. Ablation-CAM and Score-CAM both highlight the guenon’s face,
whereas Cluster-CAM only highlights the guenon’s forehead. Intuitively,
Ablation-CAM and Score-CAM seem the most reasonable but the cognition-base map
and cognition-scissors clearly show that the forehead is the most
discriminative part but the face is negative for guenon’s classification. It
will be understood if we further study the difference in species between
guenon and langur. Guenon (widely distributed in Africa) is characterized by
blond hair on the forehead and a busty white lip, whereas, langur (distributed
in Asia) is characterized by a completely black face. We mark their
characteristics by green and red circles in the third row in Fig. 4. It is the
reason why the third feature map (face) is deemed as cognition-scissors, i.e.,
the black face is an interference factor for guenon’s categorizing. This
example perfectly demonstrates the rationality of Cluster-CAM, particularly
the cognition-scissors.

###  4.4 Ablation Study

Figure 5: The heatmaps produced by Cluster-CAM with different numbers of
clusters for VGG-16. $L$ refers to the indices of layers in VGG-16.

Figure 6: The heatmaps produced by Cluster-CAM (K-means) with different
numbers of clusters for AlexNet (top) and VGG-16(bottom).

Analysis of Different Layers: Most CNNs are constructed by a cascade of
convolutional blocks (a block consists of convolutional layers, nonlinear
activation, pooling operation, etc.). Fig. 5 shows the salience heatmaps of
different convolutional blocks in VGG-16. The results basically match human’s
intuition that the shallow layers mainly capture some detailed information
(e.g., texture and edge), whereas deep layers concentrate on those parts with
clearer semantics.

Number of Clusters and CNN Structures: The number of clusters usually plays a
critical role in clustering algorithms. Here we vary the cluster number from
$2$ to $8$ and present the corresponding salience heatmaps for AlexNet and
VGG-16 in Fig. 6. It shows the salience heatmaps are sensitive to the number
of clusters and are different with CNN models. For AlexNet, the highlighted
region in heatmaps could be semantic chaos if the feature maps are split into
too many or too few clusters. It is probably because a large number of
clusters may introduce too many detailed patterns of the object and a small
number of clusters may directly include background information. Therefore, the
number of clusters should be selected as a median value. Note it is only an
empirical conclusion and exclusion exists that the optimal value is $2$ for
the fourth row (beer) in Fig.6. It is probably because the object is simple
and in regular shape, thus only two clusters are enough to represent all
necessary information. For VGG-16, the highlighted region is more concentrated
on a specific part of the object than AlexNet. It is probably because more
detailed discriminative information could be captured in VGG-16 which has much
deeper layers than AlexNet.

Clustering Method:

Figure 7: The heatmaps produced by Cluster-CAM with different numbers of
clusters by spectral clustering (top) and heatmaps produced by Cluster-CAM
with different numbers of eigenvectors (bottom).

In Section 3, we introduced two clustering algorithms, i.e., K-means and
spectral clustering. Here we take each feature map as a vertex in the graph
and use distance to construct the similarity matrix, adjacent matrix, degree
matrix, and Laplacian matrix using (17), (10), (11), and (14). Fig. 7 shows
the salience heatmaps produced by spectral clustering with different clusters
and different eigenvectors. It can be observed that the heatmaps are highly
related to the number of eigenvectors rather than the clusters. Note that the
optimal number of eigenvectors is highly related to the image itself, thus
careful manipulation of this parameter is required for different objects to
obtain the best heatmap. Therefore we will only use K-means to compute the
qualitative evaluation metrics in the next section.

###  4.5 Qualitative Evaluation

####  4.5.1 Performance Evaluation

To further evaluate the interpreting performance of Cluster-CAM
quantitatively, two widely-used evaluation metrics are adopted in this paper,
i.e., confidence drop and increase number [36, 41]. First of all, let’s think
about what kind of heatmap can be regarded as a good interpretation of CNN. A
natural and intuitive idea is to measure how much the confidence (predicted
score) of the target class will drop when the original image is partly
occluded according to the heatmap. Specifically, for each image, a
corresponding explanation map $\mathbf{L}_{c}$ is generated by element-wise
multiplication of the heatmaps and the current image as in (9) and (8).

Confidence drop: This metric compares the average drop of the model’s
confidence for a particular class in an image after occlusion as:

$\mathrm{confidence\\_drop}=\frac{S_{c}(\mathbf{X})-S_{c}(\mathbf{X}~{}\circ~{}\mathbf{H}_{n})}{S_{c}(\mathbf{X})},$
(23)

For example, assume that CNN predicts an object indigo finch in an image
$\mathbf{X}$ with confidence $0.8$. When we input the explanation map,
$\mathbf{X}~{}\circ~{}\mathbf{H}$, of this image, the CNN’s confidence in the
class indigo finch falls to $0.6$. Then the confidence_drop would be $25\%$.
It means that the most discriminative part ($75\%$) is included in the
highlighted region. Confidence drop is expected to be lower for a better CAM
and is usually averaged over many images.

Increase number measures how many times the CNN’s prediction score for $c$
increased when the masked image is input. Specifically, it happens sometimes
that the object is completely included and interference parts are occluded
(e.g., the object-irrelevant parts and background) in the highlighted region.
In this case, there will be an increase in the CNN’s predicted score for the
class (i.e., confidence drop $<0$). This value is computed as a percentage
through the whole dataset.

Table 1 shows two evaluation metrics of the entire validation set in ILSVRC
dataset ($\downarrow$ means the lower value is better and $\uparrow$ means the
higher value is better). These two metrics clearly demonstrate the superiority
of Cluster-CAM to other existing CAMs. The metrics are computed in Pytorch
1.8.0+cudnn11.1, NVIDIA RTX-3070.

Table 1: Performance Evaluation Metrics.  Method | Confidence drop $\downarrow$ | Increase number $\uparrow$
---|---|---
Grad-CAM | $17.94$ | $19.15$
Grad-CAM++ | $18.44$ | $19.75$
Ablation-CAM | $12.38$ | $24.67$
Score-CAM | $12.21$ | $25.48$
Cluster-CAM | $\mathbf{11.60}$ | $\mathbf{26.10}$

####  4.5.2 Efficiency Evaluation

Here we present two efficiency metrics, i.e., the average computing time and
the number of forward propagation (FP) per image in Table 2. It is clear that
Cluster-CAM greatly reduces the number of FP compared with Ablation-CAM and
Score-CAM. Naturally, a significant improvement in efficiency emerges from
Cluster-CAM, i.e., Cluster-CAM is $5.7$ times faster than Ablation-CAM and
$12.1$ times faster than Score-CAM. Therefore, Cluster-CAM can obtain better
visualization and interpretation performance than gradient-based and gradient-
free CAMs with efficiency closer to gradient-based CAMs.

Table 2: Efficiency Evaluation Metrics. Here $Q=6$ in Cluster-CAM.  Method | Computing time $\downarrow$ | number of FP $\downarrow$
---|---|---
Grad-CAM | $0.078$ | $1$
Grad-CAM++ | $0.141$ | $1$
Ablation-CAM | $2.206$ | $256$
Score-CAM | $4.647$ | $256$
Cluster-CAM | $0.382$ | $6$

##  5 Conclusion

In this paper, we proposed Cluster-CAM, an effective and efficient CNN
interpretation technique based on unsupervised clustering algorithms. Cluster-
CAM is the first attempt to comprehensively analyze how to split feature maps
into different groups and provide an artful strategy to remove the object-
irrelevant elements by defining cognition-scissors. In Cluster-CAM, only
several times of forward propagation is required per image while it is usually
more than hundreds for other gradient-free CAMs. Qualitative and quantitative
experimental results verified Cluster-CAM can obtain even better performance
than gradient-free CAMs with much lower computing time.

## Data Availability Statements

ILSVRC dataset can be downloaded from the website
https://www.image-net.org/challenges/LSVRC/.

## Acknowledgments

This work is funded by the National Natural Science Foundation of China (Grant
No. 62276204, 61871301, 62071349), Project 2021ZDZX-GY-0001, science and
technology project of Xianyang city.

## References

  * Cao et al. [2019] Cao, J., Pang, Y., Han, J., Li, X., 2019.  Hierarchical shot detector, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 9705–9714.
  * Chattopadhay et al. [2018] Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N., 2018\.  Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks, in: In Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 839–847.
  * Chen et al. [2022] Chen, H., Jin, Y., Jin, G., Zhu, C., Chen, E., 2022\.  Semisupervised semantic segmentation by improving prediction confidence.  IEEE Transactions on Neural Networks and Learning Systems 33, 4991–5003.  doi:10.1109/TNNLS.2021.3066850.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009\.  Imagenet: A large-scale hierarchical image database, in: In proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255.  doi:10.1109/CVPR.2009.5206848.
  * Feng et al. [2021a] Feng, Z., Ji, H., Stanković, L., Fan, J., Zhu, M., 2021a.  SC-SM CAM: An efficient visual interpretation of CNN for SAR images target recognition.  Remote Sensing 13, 4139\.
  * Feng et al. [2021b] Feng, Z., Zhu, M., Stanković, L., Ji, H., 2021b.  Self-matching CAM: A novel accurate visual explanation of CNNs for SAR image interpretation.  Remote Sensing 13, 1772\.
  * Fu et al. [2020] Fu, R., Hu, Q., Dong, X., Guo, Y., Gao, Y., Li, B., 2020.  Axiom-based Grad-CAM: Towards accurate visualization and explanation of CNNs, in: In Proceedings of the 2020 British Machine Vision Conference (BMVC 2020).
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: In Proceedings of 2016 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778.  doi:10.1109/CVPR.2016.90.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks, in: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.
  * Lapuschkin et al. [2019] Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R., 2019\.  Unmasking clever hans predictors and assessing what machines really learn.  Nature communications 10, 1–8.
  * Liang et al. [2018] Liang, X., Hu, Z., Zhang, H., Lin, L., Xing, E.P., 2018\.  Symbolic graph reasoning meets convolutions.  Advances in Neural Information Processing Systems 31\.
  * Liu et al. [2023] Liu, J., Zhang, F., Zhou, Z., Wang, J., 2023.  Bfmnet: Bilateral feature fusion network with multi-scale context aggregation for real-time semantic segmentation.  Neurocomputing 521, 27–40.  doi:https://doi.org/10.1016/j.neucom.2022.11.084.
  * Liu et al. [2022a] Liu, K., Meng, R., Li, L., Mao, J., Chen, H., 2022a.  Sisl-net: Saliency-guided self-supervised learning network for image classification.  Neurocomputing 510, 193–202.  doi:https://doi.org/10.1016/j.neucom.2022.09.029.
  * Liu et al. [2022b] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022b.  A convnet for the 2020s, in: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976–11986.
  * Ma et al. [2021] Ma, X., Zhang, S., Pena-Pena, K., Arce, G.R., 2021\.  Fast spectral clustering method based on graph similarity matrix completion.  Signal Processing 189, 108301\.  doi:https://doi.org/10.1016/j.sigpro.2021.108301.
  * Macpherson et al. [2021] Macpherson, T., Churchland, A., Sejnowski, T., DiCarlo, J., Kamitani, Y., Takahashi, H., Hikida, T., 2021.  Natural and artificial intelligence: A brief introduction to the interplay between ai and neuroscience research.  Neural Networks 144, 603–613.  doi:https://doi.org/10.1016/j.neunet.2021.09.018.
  * Omeiza et al. [2019] Omeiza, D., Speakman, S., Cintas, C., Weldermariam, K., 2019\.  Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models.  arXiv preprint arXiv:1908.01224 .
  * Ramaswamy et al. [2020] Ramaswamy, H.G., et al., 2020.  Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization, in: In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 983–991.
  * Redmon et al. [2016] Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016\.  You only look once: Unified, real-time object detection, in: In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788.  doi:10.1109/CVPR.2016.91.
  * Ren et al. [2021] Ren, J., Li, M., Liu, Z., Zhang, Q., 2021.  Interpreting and disentangling feature components of various complexity from DNNs, in: In proceedings of International Conference on Machine Learning, PMLR. pp. 8971–8981.
  * Saleem et al. [2022] Saleem, R., Yuan, B., Kurugollu, F., Anjum, A., Liu, L., 2022.  Explaining deep neural networks: A survey on the global interpretation methods.  Neurocomputing 513, 165–180.  doi:https://doi.org/10.1016/j.neucom.2022.09.129.
  * Scalzo et al. [2023] Scalzo, B., Stanković, L., Daković, M., Constantinides, A.G., Mandic, D.P., 2023.  A class of doubly stochastic shift operators for random graph signals and their boundedness.  Neural Networks 158, 83–88.  doi:https://doi.org/10.1016/j.neunet.2022.10.035.
  * Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017\.  Grad-CAM: Visual explanations from deep networks via gradient-based localization, in: In Proceedings of the 2017 IEEE international conference on computer vision, pp. 618–626.
  * Simonyan et al. [2013] Simonyan, K., Vedaldi, A., Zisserman, A., 2013.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034 .
  * Simonyan and Zisserman. [2015] Simonyan, K., Zisserman., A., 2015\.  Very deep convolutional networks for large-scale image recognition, in: 3rd International Conference on Learning Representations (ICLR 2015), pp. 1–14.
  * Spinelli et al. [2022] Spinelli, I., Scardapane, S., Uncini, A., 2022.  A meta-learning approach for training explainable graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems , 1–9doi:10.1109/TNNLS.2022.3171398.
  * Srinivas et al. [2021] Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., Vaswani, A., 2021\.  Bottleneck transformers for visual recognition, in: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition (CVPR), pp. 16519–16529.
  * Stankovic et al. [2017] Stankovic, L., Dakovic, M., Sejdic, E., 2017.  Vertex-frequency analysis: A way to localize graph spectral components [lecture notes].  IEEE Signal Processing Magazine 34, 176–182.  doi:10.1109/MSP.2017.2696572.
  * Stankovic et al. [2019] Stankovic, L., Mandic, D.P., Dakovic, M., Kisil, I., Sejdic, E., Constantinides, A.G., 2019\.  Understanding the basis of graph signal processing via an intuitive example-driven approach.  IEEE Signal Processing Magazine 36, 133–145.  doi:10.1109/MSP.2019.2929832.
  * Sun et al. [2022] Sun, S., Song, B., Cai, X., Du, X., Guizani, M., 2022\.  CAMA: Class activation mapping disruptive attack for deep neural networks.  Neurocomputing 500, 989–1002.  doi:https://doi.org/10.1016/j.neucom.2022.05.065.
  * Tan et al. [2022] Tan, R., Gao, L., Khan, N., Guan, L., 2022.  Interpretable artificial intelligence through locality guided neural networks.  Neural Networks 155, 58–73.  doi:https://doi.org/10.1016/j.neunet.2022.08.009.
  * Townsend et al. [2020] Townsend, J., Chaton, T., Monteiro, J.M., 2020.  Extracting relational explanations from deep neural networks: A survey from a neural-symbolic perspective.  IEEE Transactions on Neural Networks and Learning Systems 31, 3456–3470.  doi:10.1109/TNNLS.2019.2944672.
  * Tu et al. [2021] Tu, Z., Zhou, A., Gan, C., Jiang, B., Hussain, A., Luo, B., 2021.  A novel domain activation mapping-guided network (DA-GNT) for visual tracking.  Neurocomputing 449, 443–454.  doi:https://doi.org/10.1016/j.neucom.2021.03.056.
  * Vlahek and Mongus [2021] Vlahek, D., Mongus, D., 2021\.  An efficient iterative approach to explainable feature learning.  IEEE Transactions on Neural Networks and Learning Systems , 1–13doi:10.1109/TNNLS.2021.3107049.
  * Wang et al. [2020] Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., Hu, X., 2020.  Score-CAM: Score-weighted visual explanations for convolutional neural networks, in: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) workshops, pp. 24–25.
  * Zeiler and Fergus [2014] Zeiler, M.D., Fergus, R., 2014\.  Visualizing and understanding convolutional networks, in: European conference on computer vision, Springer. pp. 818–833.
  * Zhang et al. [2021] Zhang, Q., Rao, L., Yang, Y., 2021.  Group-CAM: Group score-weighted visual explanations for deep convolutional networks.  arXiv preprint arXiv:2103.13859 .
  * Zhao et al. [2019] Zhao, Z., Xie, X., Wang, C., Liu, W., Shi, G., Du, J., 2019.  Visualizing and understanding of learned compressive sensing with residual network.  Neurocomputing 359, 185–198.  doi:https://doi.org/10.1016/j.neucom.2019.05.043.
  * Zhou et al. [2019] Zhou, B., Bau, D., Oliva, A., Torralba, A., 2019.  Interpreting deep visual representations via network dissection.  IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 2131–2145.  doi:10.1109/TPAMI.2018.2858759.
  * Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A., 2016.  Learning deep features for discriminative localization, in: In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929.
  * Zhou and Kainz [2018] Zhou, K., Kainz, B., 2018.  Efficient image evidence analysis of cnn classification results.  arXiv preprint arXiv:1801.01693 .
  * Zhu et al. [2022] Zhu, M., Feng, Z., Stanković, L., Ding, L., Fan, J., Zhou, X., 2022.  A probe-feature for specific emitter identification using axiom-based grad-cam.  Signal Processing , 108685.
  * Zhu et al. [2017] Zhu, Y., Zhao, C., Wang, J., Zhao, X., Wu, Y., Lu, H., 2017.  Couplenet: Coupling global structure with local parts for object detection, in: Proceedings of the IEEE international conference on computer vision, pp. 4126–4134.

zhenpeng.jpg Zhenpeng Feng was born in Xianyang, Shaanxi, China in 1996. He
received a B.E. degree in School of Electronic Engineering, Xidian University
in 2019. He is currently a Ph.D. student in explainable artificial
intelligence at School of Electronic Engineering, Xidian University. He is
also a visiting student in the University of Montenegro, working with Prof.
Ljubiša Stanković’s research team. His research interests include interpreting
deep neural networks and signal processing.

hongbingji.png Hongbing Ji received a B.S. degree in radar engineering, an
M.S. degree in circuit, signals, and systems, and the Ph.D. degree in signal
and information processing from Xidian University, Xi’an, China, in 1983,
1989, and 1999, respectively. He is currently a full professor at Xidian
University and a senior member of IEEE. His research interests include pattern
recognition, radar signal processing, and multi-sensor information fusion.

milos.png Miloš Daković was born in 1970 in Nikšić, Montenegro. He received a
B.S. in 1996, an M.S. in 2001, and a Ph.D. in 2005, all in electrical
engineering from the University of Montenegro. He is a full professor at the
University of Montenegro. His research interests are in signal processing,
time-frequency signal analysis, compressive sensing, radar signal processing,
and graph signal processing.

xiyangcui.jpg Xiyang Cui was born in Handan, Hebei, China in 1997. He received
the B.E. degree and M.E. degree in Electronic Information Engineering and
Electrical Circuit System from School of Electronic Engineering, Xidian
University in 2019 and 2021, respectively. He is currently an investigator of
an electronic company and collaborates with Zhenpeng Feng and Prof. Ljubiša
Stanković in scientific research. His research interests include electrical
circuit design and image processing.

mingzhezhu.jpg Mingzhe Zhu was born in China in 1982. He received a B.S.
degree in signal and information processing, a Ph.D. degree in pattern
recognition and intelligent system from Xidian University in 2004 and 2010,
respectively. He is currently an associate professor at School of Electronic
Engineering, Xidian University. His research interests include non-stationary
signal processing, time-frequency analysis, and target recognition.

ljubisa.jpg Ljubiša Stanković was born in Montenegro, 1960. He was at the Ruhr
University Bochum, 1997-1999, supported by the AvH Foundation. Stankovic was
the Rector of the University of Montenegro 2003-2008, the Ambassador of
Montenegro to the UK, 2011-2015, and a visiting academic to the Imperial
College London, 2012-2013. He published almost 200 journal papers. He is a
member of the National Academy of Science and Arts (CANU) and the Academia
Europaea. Stanković won the Best paper award from the EURASIP in 2017 and the
IEEE SPM Best Column Award for 2020. Stanković is a professor at the
University of Montenegro and a Fellow of the IEEE.