#  MexGen: An Effective and Efficient Information Gain Approximation for
Information Gathering Path Planning

Joshua Chesser1, Thuraiappah Sathyan2 and Damith C. Ranasinghe1 *This work was
supported by grant funding from Lockeed Martin, Australia, Australian Research
Council (ARC) grant LP200301507 and the Australian Government’s Research
Training Program Scholarship (RTPS).1Joshua Chesser and Damith C. Ranasinghe
are with the School of Computer & Mathematical Sciences, The University of
Adelaide, SA 5005, Australia<EMAIL_ADDRESS><EMAIL_ADDRESS>Thuraiappah is with STELaRLab,
Lockheed Martin Australia<EMAIL_ADDRESS>
###### Abstract

Autonomous robots for gathering information on objects of interest has
numerous real-world applications because of they improve efficiency,
performance and safety. Realizing autonomy demands online planning algorithms
to solve sequential decision making problems under uncertainty; because,
objects of interest are often dynamic, object state, such as location is not
directly observable and are obtained from noisy measurements. Such planning
problems are notoriously difficult due to the combinatorial nature of
predicting the future to make optimal decisions. For information theoretic
planning algorithms, we develop a computationally efficient and effective
approximation for the difficult problem of predicting the likely sensor
measurements from uncertain belief states. The approach more accurately
predicts information gain from information gathering actions. Our theoretical
analysis proves the proposed formulation achieves a lower prediction error
than the current efficient-method. We demonstrate improved performance gains
in radio-source tracking and localization problems using extensive simulated
and field experiments with a multirotor aerial robot.

##  I Introduction

Autonomous sensing can robotize dull, dangerous, labour-intensive information
gathering tasks, such as search and rescue [1], exploration [2] and wildlife
tracking for conservation [3, 4, 5, 6, 7, 8]. Typical systems are built with
agents capable of performing various information gathering actions, such as
changing sensor positions to optimize the sensor-to-target geometry to gain
information, e.g. position and velocity, on objects of interest. Building
autonomous agents for the task requires solving a planning problem—a
sequential decision making process where each agent repeatedly selects the
next best information gathering action. Critically, to perform a task
effectively, planning methods must deal with uncertainty—future actions need
to: i) rely on imperfect state information, such as positions of wildlife,
gathered by noisy measurements; and ii) account for non-deterministic system
dynamics, such as movements and behaviors of tracked wildlife.

Partially observable Markov decision process (POMDP) formulations are widely
used for online planning problems under uncertainty [5, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18]. Our work focuses on information theoretic POMDP planning
problems for object tracking. The goal is to select the optimal actions to
obtain the most informative measurements to accurately estimate the state of
dynamic objects of interest.

Optimal planning requires tackling the computationally challenging problem of
estimating the information gain of future actions with respect to the large
space of possible future measurements. An agent must consider the result of
every possible measurement that could be obtained after every possible action
at each decision step over a look-ahead horizon. The result is an
exponentially increasing number of possible action-measurement sequences as
the horizon increases; notably, solving POMDPs optimally is PSPACE-complete
[19]. Hence, a significant challenge is formulating suitable approximations
for the problem to achieve computationally efficient, yet effective planning
decisions for information gathering tasks.

(a)  Figure 1: Trajectory heat maps from field experiments. We compared the
current efficient method (PIM) and a computationally expensive method (MC-8)
for computing information gain from possible future measurements with ours for
a radio source localization task. Trajectory planning decisions, i.e.
information gathering actions, are more effective with ours—resulting in a
$40$% reduction in mission completion times in the best case (vs. PIM, the
next best method) and visibly shorter paths leading to consistently localizing
the closest object and navigating to reduce the uncertainty of the next object
(see $\S$ IV-D). The demonstration video is at: https://youtu.be/XrsCC6MkaB4

Existing work approximate the information gain with: i) a subset of possible
measurements sampled for each action from the future belief state with Monte
Carlo (MC) methods [20, 10, 12, 9, 13, 14]. The method’s effectiveness depends
on the number of samples but increasing samples increases computational costs
exponentially; or ii) a single noiseless measurement generated from a likely
future state approximation—the predicted ideal measurement (PIM) [21, 5, 8].
The methods assume the true future state is a mode of the predicted belief
state distribution and approximates it with a point estimate.

Figure 2: Consider an agent with an onboard sensor for acquiring noisy range
measurements tasked with tracking the position an object of interest (e.g.
radio source). At time $t=0$, the agent evaluates Action 2’s reward out of
three possible actions over a look ahead horizon of $t=1$. The red contours
depict the belief state probability density. In True, we update the belief
state with a measurement from the ground truth object position at $t=1$ for
comparison. PIM and MexGen tiles show the Updated Belief State given the
predicted range measurement generated at $t=1$. Now, PIM, the existing
computationally efficient approximation, incorrectly predicts an unrealistic
measurement from an estimated future estate of an object. The result is an
over confident future object belief state that is significantly different from
the Truth. In contrast, MexGen uses the expected measurement and updates the
belief state to predict the future state; this result compares well with the
True state at $t=1$ in contrast to PIM and leads to a better approximation of
information gain from Action 2.

This study formulates a computationally efficient, yet more effective
approximation for information gain. We propose computing information gain of
each information gathering action at each decision step using the expected
measurement generated from the predicted belief state to yield the updated
belief state—or future belief state. Effectively, we exploit the asymmetric
computational cost of the measurement function and the belief state update to
achieve a good enough information gain approximation (typically the computing
cost of the measurement function is significantly lower).

Our approach—measurement expectation generation (MexGen)—as shown in Figure 1,
leads to more effective planning decisions. PIM leads to poor planning
decisions as we explain later in Figure 2 while MC, even with 8 samples and
taking $7.6\times$ longer for a planning iteration compared to MexGen, a seen
in Figure 6, is not an adequate approximation. Through a theoretical analysis
and a series of extensive experiments, we show our method achieves overall
better performance—effectiveness of planning decisions and computational
efficiency—compared to existing methods. In summary, our key contributions
are:

  * •

We formulate a new, more effective, yet computationally efficient method for
generating measurements from uncertain future states for planning problems
($\S$ III).

  * •

We prove the proposed method achieves a lower measurement prediction error,
and consequently a better information gain approximation, than the previous
computationally efficient method ($\S$ III-A).

  * •

We demonstrate the effectiveness and efficiency of our proposed method in
planning for information gathering scenarios using extensive simulation and
field experiments with an aerial robot platform ($\S$ IV).

In general, our work contributes to realizing autonomous sensing applications
under unavoidable real-world limits on hardware cost, weight and size of
unmanned aerial vehicles.

##  II Background and Related Work

In this section we formally define the problem and discuss related work.

###  II-A Problem Definition

We consider an agent performing mobile object tracking using noisy sensor
measurements. At time $t$, the agent moves based on the control input
$\mathbf{u}_{t}\in\mathbb{U}_{t}$, where $\mathbb{U}_{t}$ is the set of
admissible actions. It then receives $m$ noisy measurements
$\mathbf{Z}_{t}=\\{\mathbf{z}_{t}^{(1)},\dots,\mathbf{z}_{t}^{(m)}\\}$,
composed of possible detections from $n$ objects with states
$\mathbf{X}_{t}=\\{\mathbf{x}^{(1)}_{t},\dots,\mathbf{x}^{(n)}_{t}\\}$. Each
object produces at most one measurement, and the sensor can produce spurious
clutter measurements. The object measurements are sensor dependent and modeled
by the measurement function, $\mathbf{z}=h(\mathbf{x})$ while the object
dynamics are modeled by the uncertain motion model,
$p(\mathbf{X}_{t+1}|\mathbf{u}_{t+1},\mathbf{X}_{t})$.

Bayesian object tracking algorithms [22] use prediction and update procedures
to maintain a multi-object probability density
$p(\mathbf{X}_{t}|\mathbf{u}_{1:t},\mathbf{Z}_{1:t})$. The motion model is
used to predict the density based on the control action, $\mathbf{u}_{t+1}$,
to obtain $p(\mathbf{X}_{t+1}|\mathbf{u}_{1:t+1},\mathbf{Z}_{1:t})$. The
measurement model is used to update the density with new measurements,
$\mathbf{Z}_{t+1}$, to obtain
$p(\mathbf{X}_{t+1}|\mathbf{u}_{1:t+1},\mathbf{Z}_{1:t+1})$. The agent’s
objective is to choose the next action $\mathbf{u}^{*}_{t+1}$ that maximises
the expected reward
$\mathbb{E}\left[R(p(\mathbf{X}_{t}|\mathbf{u}_{1:t},\mathbf{Z}_{1:t}),\mathbf{u}_{t+1:t+H},\mathbf{Z}_{t+1:t+H})\right]$
over the finite horizon $H$. Next we describe the information theoretic POMDP
problem formulation for achieving the objective.

Online Information Theoretic Planning with POMDP.  A POMDP for information
theoretic planning comprises the current belief state $B_{t}$, the set of
admissible actions $\mathbf{u}_{t}\in\mathbb{U}_{t}$, and a reward determined
by the action $r(B_{t-1},\mathbf{u}_{t})$ reward and information gain
$\Psi(B_{t},B_{t-1})$. For object tracking problems, the belief state is the
object probability density,
$B_{t}=p(\mathbf{X}_{t}|\mathbf{u}_{1:t},\mathbf{Z}_{1:t})$. Given a sequence
of actions $\mathbf{u}_{t+1:t+H}$ and measurements resulting from those
actions $\mathbf{Z}_{t+1:t+H}$, the reward over the horizon starting from
belief state $B_{t}$ is computed as the discounted sum of immediate rewards.

$\begin{split}R(&B_{t},\mathbf{u}_{t+1:t+H},\mathbf{Z}_{t+1:t+H})=\\\
&\sum_{k=1}^{H}\lambda^{k-1}\left[r(B_{t+k-1},\mathbf{u}_{t+k})+\Psi(B_{t+k},B_{t+k-1})\right]\end{split}$
(1)

where $0<\lambda\ \leq 1$ is the discount factor. The future belief states are
a result of the future action and measurement:

$B_{t+1}=\text{update}(B_{t},\mathbf{u}_{t+1},\mathbf{Z}_{t+1})$ (2)

The next action is then selected as:

$\mathbf{u}^{*}_{t+1}=\operatorname*{arg\,max}_{\mathbf{u}_{t+1}\in\mathbb{U}_{t+1}}\mathbb{E}\left[R(B_{t},\mathbf{u}_{t+1:t+H},\mathbf{Z}_{t+1:t+H})\right]$
(3)

where the expectation, $\mathbb{E}$, is taken with respect to the future
actions $\mathbf{u}_{t+1:t+H}$ and measurements $\mathbf{Z}_{t+1:t+H}$.

Information theoretic planning uses measures of information gain to drive the
planner to choose the most informative actions. Information quantities, such
as Shannon Entropy, Kullback-Leibler divergence, and Renyi Divergence, are
commonly used in the reward function [21, 9, 5, 10, 20].

Computing the expectation in (3) requires evaluating all possible sequences of
actions and measurements. Thus, if there are $A$ possible actions and $Z$
possible measurements at each future horizon time, the complexity of computing
$\mathbf{u}^{*}_{t+1}$ is $O(A^{H}Z^{H})$. Monte Carlo Tree Search methods are
often used to compute the expectation in (3) by focusing computational effort
on promising action-measurement sequences [12, 13, 15]. If the belief state
can be modeled with a Gaussian distribution, iterative Linear Quadratic
Regulator [23] methods can be employed. A recent method uses simplified belief
states to speedup the computation of information gain [14]. However, it is
still necessary to consider the many possible future measurements throughout
the tree search for solutions. In general, for non-Gaussian/non-linear
systems, the challenge of computing the expectation in (3) with respect to
future measurements remains, which is the focus of our work.

Without loss of generality, for the challenge of computing the expectation in
(3), we consider the problem of future measurement generation for single-
object belief state, i.e.,
$B_{t}=p(\mathbf{x}|\mathbf{u}_{1:t},\mathbf{Z}_{1:t})$. Because, the multi-
object case is simply an extension of the single object problem (measurements
can be generated for each object in the multi-object density).

###  II-B Related Work

Two common approaches are to use Monte-Carlo sampling [20, 10, 12, 9, 13, 14]
or the Predicted Ideal Measurement [21, 5, 8, 6]. The methods are tasked with
generating the next likely measurement, $\mathbf{Z}_{k}$, from a belief state
$B_{k}$ for the challenging problem of computing the expectation in (3).

MC Sampling.  Here, a random state is sampled from the object density at each
time step, $\mathbf{x}\sim B_{t}$. Next, the noisy measurement function
$h(\mathbf{x})$ is used to generate a measurement from the sampled state,
$\mathbf{z}=h(\mathbf{x})$.

The expectation is approximated by taking the average over $M$ repetitions of
this process. The number of repetitions directly affects the planning
complexity. Effectively, the number of measurements at each horizon time
becomes $1$, however, as this is repeated $M$ times, the planning complexity
is $O(A^{H}M)$. Under computational constraints it may be impossible to
perform enough repetitions to obtain a good approximation, making it
infeasible to use this method.

Predicted Ideal Measurement (PIM). The general approach aims to create a
deterministic generator such that the number of possible measurements $Z=1$,
reducing the planning complexity to $O(A^{H})$. To generate a measurement with
this approach, first the estimated state is computed from the predicted belief
state, $\bar{\mathbf{x}}=\mathbb{E}\left[B_{t}\right]$. Then, the noiseless
measurement function $\bar{h}(\mathbf{x})$ is applied to the estimated state
to obtain the generated measurement, $\mathbf{z}=\bar{h}(\bar{\mathbf{x}})$.

While the method can significantly reduce planning complexity, it relies
solely on the estimated state to adequately represent the belief state. But,
if the transition or measurement function is non-linear, as is the case in
many real-world systems, it is possible for the belief state to become multi-
modal. Now, the estimated state may poorly represent the belief state and
result in unrealistic predicted measurements, leading to sub-optimal
decisions. This problem is illustrated in Figure 2, where a PIM method
predicted measurement leads to a confident, yet extremely unlikely belief
state. We propose an alternative, described in $\S$ III.

##  III Proposed Measurement Prediction Method

We focus on the class of problems where the measurement model is approximately
unimodal with no assumptions of the modality of the belief state. Notably,
although we assume an approximate unimodal measurement model, our theoretical
analysis in Section III-A demonstrates our approximation always yields a lower
prediction error compared to the computationally efficient alternative, PIM.

The key idea for the proposed method, MexGen, is to compute information gain
using the expected measurement instead of a predicted future state as in PIM.
Figure 2 illustrates our approach and the formulation’s ability to better
approximate a measurement from a predicted future belief state compared with
PIM when the belief state is multi-modal. More specifically, the expected
measurement is computed from a set of randomly sampled measurements.

First, $M$ states are randomly sampled from the belief state,
$\mathbf{x}^{(i)}\sim B_{t}$, where $i\in[1,\dots,M]$. Next, the noisy
measurement function is applied to each of the sampled states to obtain $M$
sampled measurements, $\mathbf{z}^{(i)}=h(\mathbf{x}^{(i)})$. The predicted
future measurement, $\mathbf{z}$, is generated from the expectation of these
sampled measurements:

$\mathbf{z}=\dfrac{1}{M}\sum_{i=1}^{M}\mathbf{z}^{(i)}$ (4)

The number of samples affects the variance of the generated measurement. If a
sufficient number of samples are used, the variance will be sufficiently small
but $Z=1$, hence, only a single measurement simulation per action is needed.
Even though the method employs measurement sampling, the computational cost of
the measurement function is typically significantly lower than the cost of the
belief update step. Thus, even if many samples are used, we still expect the
method to provide significant improvements to run-time performance over the MC
method. This is demonstrated in $\S$ IV-C. Further, we show analytically in
$\S$ III-A that our method can achieve a lower measurement prediction error
compared with PIM for arbitrary measurement models.

###  III-A Theoretical Analysis

The theoretical basis behind the MexGen method is to minimise the error of the
predicted measurement with respect to the possible future measurements. To
compare PIM and MexGen, we use the Mean Squared Error (MSE) between the
generated measurement $\hat{z}$ and the posterior measurement distribution.
The MSE is defined as:

$\operatorname{MSE}(\hat{z}|B_{k+1|k})=\sum_{z\in
Z}p(z|B_{k+1|k})(\hat{z}-z)^{2}$ (5)

where $p(z|B_{k+1|k})$ is the predicted measurement density from the belief
state $B_{k|k}$.

###### Lemma 1

The MSE of a PIM measurement is lower bounded by the expected MSE of an MexGen
measurement, within some bound $\varepsilon$ that depends on the number of
samples used in MexGen:

$\operatorname{MSE}_{\text{PIM}}+\varepsilon\geq\mathbb{E}\left[\operatorname{MSE}_{\text{{MexGen}}}\right],\leavevmode\nobreak\
\text{where}\leavevmode\nobreak\ \varepsilon=\dfrac{1}{M}\sigma^{2}$ (6)

and $\sigma^{2}$ is the variance of $p(z|B_{k+1|k})$.

Proof: First, the measurement that minimises (5) is the expected value of the
predicted measurement density:

$\bar{z}=\mathbb{E}[z|B_{k+1|k}]=\sum_{z\in Z}p(z|B_{k+1|k})z$ (7)

which has the following MSE (with the dependence on $B_{k+1|k}$ omitted for
notational simplicity):

$\begin{split}\operatorname{MSE}(\bar{z})&=\sum_{z\in Z}p(z)(\bar{z}-z)^{2}\\\
&=\sum_{z\in Z}p(z)\bar{z}^{2}-2\bar{z}\sum_{z\in Z}p(z)z+\sum_{z\in
Z}p(z)z^{2}\\\ &=\bar{z}^{2}-2\bar{z}^{2}+\mathbb{E}\left[z^{2}\right]\\\
&=-\bar{z}^{2}+\operatorname{Var}\left[z\right]+\bar{z}^{2}=\sigma^{2}\\\
\end{split}$ (8)

Note that variance is defined as:

$\operatorname{Var}\left[z\right]=\mathbb{E}\left[z^{2}\right]-\mathbb{E}\left[z\right]^{2}$
(9)

Let $\hat{z}$ be the MexGen measurement $\mathbf{z}$ from (4). We can see that
each sample is selected proportional to $p(z|B_{k+1|k})$ as
$\mathbf{x}^{(i)}\sim B_{k+1|k}$ and $h(\mathbf{x}^{(i)})$ generates a
measurement proportional to $p(\mathbf{z}|\mathbf{x}^{(i)})$. Thus, as $M$ in
(4) increases, $\hat{z}$ approaches $\bar{z}$. However, with only a finite
number of samples, $\hat{z}$ is randomly distributed. According to the Central
Limit theorem, the distribution of $\hat{z}$ is approximately normally
distributed around the true mean $\bar{z}$ with variance $\varepsilon$ when
$M$ is large enough, $\hat{z}\sim\mathcal{N}(\bar{z},\varepsilon)$.

Using this approximation, for a given $\hat{z}$, the MSE can be computed from
$\bar{z}$, using (5)

$\operatorname{MSE}(\hat{z})=\operatorname{MSE}(\bar{z}+v)=\sum_{z\in
Z}p(z)(\bar{z}+v-z)^{2}$ (10)

where $v\sim\mathcal{N}(0,\varepsilon)$ is the approximately normally
distributed random noise of the MexGen measurement. Re-writing (10) by
expanding the square gives us:

$\begin{split}\operatorname{MSE}(\hat{z})=&\sum_{z\in
Z}\left[p(z)(\bar{z}-z)^{2}\right]\\\ &+2v\sum_{z\in
Z}\left[p(z)(\bar{z}-z)\right]+v^{2}\end{split}$ (11)

Here, the first term on the right hand side is $\operatorname{MSE}(\bar{z})$.
The second term is the mean error (ME) of $\bar{z}$ multiplied by a constant,
which is always zero:

$\displaystyle\begin{split}\operatorname{ME}(\bar{z})&=\sum_{z\in
Z}p(z)(\bar{z}-z)\\\ &=\sum_{z\in Z}p(z)\bar{z}-\sum_{z\in
Z}p(z)z=\bar{z}-\bar{z}=0\end{split}$ (12)

Therefore, the MSE of an MexGen measurement is:

$\operatorname{MSE}(\hat{z})=\operatorname{MSE}(\bar{z})+v^{2}$ (13)

and the expected MSE is:

$\begin{split}\mathbb{E}\left[\operatorname{MSE}(\hat{z})\right]&=\operatorname{MSE}(\bar{z})+\mathbb{E}\left[v^{2}\right]\\\
&=\operatorname{MSE}(\bar{z})+\varepsilon\end{split}$ (14)

where $\mathbb{E}[v^{2}]=\varepsilon$ when $v\sim\mathcal{N}(0,\varepsilon)$
using (9).

Conversely, generating a measurement from the expected object state, as done
with the PIM method, produces the following MSE:

$\displaystyle\begin{split}\operatorname{MSE}(\bar{h}(\bar{x}))&=\sum_{z\in
Z}p(z)(\bar{h}(\bar{x})-z)^{2}\\\
&=\bar{h}(\bar{x})^{2}-2\bar{z}\bar{h}(\bar{x})+\mathbb{E}[z^{2}]\\\
&=\bar{h}(\bar{x})^{2}-2\bar{z}\bar{h}(\bar{x})+(\bar{z})^{2}+\text{Var}[z]\\\
&=(\bar{h}(\bar{x})-\bar{z})^{2}+\operatorname{MSE}(\bar{z})\end{split}$ (15)

Here, the first term is always positive or zero, thus, the MSE of the PIM
method must be lower bounded by $\operatorname{MSE}(\bar{z})$:

$\operatorname{MSE}_{\text{PIM}}\geq\operatorname{MSE}(\bar{z}).$ (16)

Adding $\varepsilon$ to both sides thus satisfies (6), then
$\operatorname{MSE}_{\text{PIMS}}+\varepsilon\geq\operatorname{MSE}(\bar{z})+\varepsilon$.
Hence:

$\begin{split}\operatorname{MSE}_{\text{PIM}}+\varepsilon&\geq\mathbb{E}\left[\operatorname{MSE}_{\text{{MexGen}}}\right]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \blacksquare\end{split}$ (17)

Remark 1. From (14), our MexGen method’s error depends on the predicted
measurement variance and can be reduced by increasing samples. But, from (15),
the error of the PIM method depends on $\bar{h}(\cdot)$ and $\bar{x}$ and
cannot be reduced.

Remark 2. If the system is linear that $\bar{h}(\bar{x})=\bar{z}$ and
$\operatorname{MSE}_{\text{PIM}}=\operatorname{MSE}(\bar{z})$, otherwise, it
is possible that $(\bar{h}(\bar{x})-\bar{z})^{2}>0$. In these cases, there
will always be a finite number of samples $M$ for MexGen, such that
$\mathbb{E}[\operatorname{MSE}_{\text{{MexGen}}}]<\operatorname{MSE}_{\text{PIM}}$:

$\begin{split}\operatorname{MSE}(\bar{z})+(\bar{h}(\bar{x})-\bar{z})^{2}&>\operatorname{MSE}(\bar{z})+\dfrac{1}{M}\sigma^{2}\\\
M&>\dfrac{\sigma^{2}}{(\bar{h}(\bar{x})-\bar{z})^{2}}\end{split}$ (18)

Importantly, this indicates, for any non-linear system, MexGen can perform
equally or better than PIM with respect to measurement prediction error.

##  IV Simulation and Field Experiments

We perform a series of extensive simulations and a real-world experiment to
evaluate the proposed measurement generation method. In particular:

  * •

Object following scenario ($\S$ IV-A).

  * •

Multi-object tracking & localisation scenario ($\S$ IV-B).

  * •

Run-time analysis of methods ($\S$ IV-C).

  * •

Field trials with an aerial robot ($\S$ IV-D).

In the these experiments, the agent is a quad-copter flying $50$ m above
ground, with a $15$ m/s top speed, and $4.0$ m/s2 acceleration. For object
tracking, we do not solve the object-to-measurement association problem and,
as in typical radio source localization problems, assume this can be
determined from the measurement [5, 9]. Hence, for each object, we employ a
recursive Bayesian filter, in particular a particle-based Bernoulli Filter
[24], together with the following measurement and motion (dynamic) models for
objects.

Measurement Model. Our experiments consider radio source signal strength
measurements periodically emitted from objects (such as rescue beacons). The
model follows propagation path loss from the Friis transmission equation,
where the received power in dB at a receiver $u$ is modeled by:

$P(x,u)=P^{d_{0}}_{r}-10n\log_{10}\left(\|x-u\|/d_{0}\right)+g_{dir}(x,u)$
(19)

where $x$ is the radio source location, $\|x-u\|$ is the distance from the
source to the receiver; $P^{d_{0}}_{r}$ is the received power at some
reference distance $d_{0}$; $n$ is the path loss exponent that models the
environmental factors; $g_{dir}(x,u)$ is the antenna gain based on the
relative source location and receiver heading. The measurement function is
thus:

$h(x,u)=P(x,u)+\zeta$ (20)

where $\zeta\sim\mathcal{N}(0,R)$ is the zero-mean, Gaussian measurement noise
with variance $R$. For these experiments, we set $R=2.5^{2}$,
$P^{d_{0}}_{r}=-7.0$, $d_{0}=1.0$, $n=1.5$ from field data, and $g_{dir}(x,u)$
to use a monopole antenna gain pattern. We use this measurement model to
demonstrate how a non-linear measurement model can affect the planning
decisions.

Motion Models.  In the simulated experiments, we use four different object
motion models: i) Random Walk (RW), ii) Constant Velocity (CV), iii) CV with
infrequent turning (CV-IFT), and iv) CV-IFT with mismatched filter. To model a
maneuvering object, the CV-IFT model uses an interacting multiple model (IMM)
where the object state switches between three modes; 1) CV; 2) constant turn
rate with acceleration (CTRA) [25]; 3) negative CTRA to allow the object to
move straight, turn left, and turn right. For this case, we use the IMM
particle filter [26] for state estimation. All of the models are known to the
filter, except the CV-IFT with mismatched filter scenarios; here, the filter
assumes a CV model for objects but with increased noise to compensate for
possible manoeuvres. This hard setting simulates a dynamic model mismatch
experienced in practical settings.

Path Planning.  The action set for the agent consists of the eight cardinal
directions, where the agent will simply head in the selected direction until a
new direction is chosen by the path planning algorithm. Online path planning
is implemented by approximating the expectations in (3) with simulations. For
each action $\mathbf{u}_{t+1}$, action sequences are simulated starting with
$\mathbf{u}_{t+1}$,
$\hat{\mathbf{u}}_{t+1:t+H}=(\mathbf{u}_{t+1},\dots,\mathbf{u}_{t+H})$. The
simulation is then played out by repeating the following four-step process:

  1. 1.

Use the next action $\mathbf{u}_{k+1}$ to compute the predicted belief state
$B_{k+1|k}=p(\mathbf{X}_{k+1}|\mathbf{u}_{1:k+1},\mathbf{Z}_{1:k})$.

  2. 2.

Generate the measurements $\mathbf{Z}_{k+1}$ from the predicted belief state
using one of the measurement prediction methods:
$\mathbf{Z}_{k+1}=g(B_{k+1|k})$

  3. 3.

Update the belief state using the generated measurement to obtain the
posterior belief state:
$B_{k+1}=p(\mathbf{X}_{k+1}|\mathbf{u}_{1:k+1},\mathbf{Z}_{1:k+1})$

  4. 4.

Compute the reward of the action:
$r_{k}=r(B_{k},\mathbf{u}_{k+1})+\Psi(B_{k+1},B_{k})$.

Here, $r_{k}$ is used to compute the sum of discounted rewards (1), and the
action with the highest average sum is selected. We choose to use Renyi
Divergence to formulate our reward as it is readily computed from a particle
approximation of a belief density and is successfully used in variety of
scenarios [9, 21, 20]. Renyi Divergence between the two distributions $p$ and
$q$ is computed as:

$\Psi(p,q)=\dfrac{1}{\alpha-1}\log\int p(x)^{\alpha}q(x)^{1-\alpha}\,dx$ (21)

where $\alpha>0$ controls the effect of the tails of the two distributions. We
set $\alpha=0.1$ for all experiments as this has demonstrated the best
planning performance, although the choice does not greatly impact performance.

###  IV-A Object Following

Here, we evaluate the impact of the different measurement prediction methods
on planning to follow and track an object. In each scenario, a controllable
agent aims to minimize the uncertainty of an object. The planner re-plans
every $5$ seconds, with a $10$ second look-ahead time. Planning is performed
using a belief state of $1000$ particles sampled from the filter. The search
space is $4$ km by $4$ km square, with the object being at a height of $0$ m,
and the agent starting at the centre. For the MC method, each action sequence
is simulated $8$ times (MC-8), chosen so the simulations complete in a
reasonable time frame. For the MexGen method, we use $512$ measurement samples
as negligible improvement is seen with more.

Trajectory Generation. For each scenario, we perform extensive Monte-Carlo
(MC) simulations. First, we generate $50$ unique starting states for each
scenario. Then, we run $30$ MC simulations for each starting state, giving the
planner $600$ simulated seconds to find and follow the object. Overall, $1500$
simulations are performed for each scenario, totalling to $250$ hours of
simulated flight time. The object states for each unique trajectory are
intialized in the following way (examples of these trajectories are
illustrated in Figure 3):

  * •

Position (all models): The initial object position for trajectories are
uniformly sampled from the search space.

  * •

Velocity (CV and CV-IFT): The velocities are initialized by uniformly sampling
a direction and speed from the intervals $[0^{\circ},360^{\circ}]$ and
$[5\leavevmode\nobreak\ m/s,6\leavevmode\nobreak\ m/s]$ respectively.

  * •

Turn Rate (CV-IFT): The turn rate is uniformly sampled from the interval
$[10^{\circ}/s,15^{\circ}/s]$.

  * •

IMM Transition Matrix (CV-IFT): The state transitions are set to occur every
$1$ second with a transition probability matrix with the rows,
$c_{1}=[0.975,0.0125,0.0125]$, $c_{2}=[0.225,0.775,0.0]$,
$c_{3}=[0.225,0.0,0.775]$.

(a)

(b)

(c)

Figure 3: Agent and object path extracts in the object following experiments
($\square$: represent the starting positions; $\triangle$: represent the
ending positions).

Performance Measures. After every filter update, we record two metrics; i)
Error: the absolute error between the estimated object position and the ground
truth position; and ii) Tr(Cov): the trace of the object positional
covariance.

(a)

(b)

(c)

(d)

Figure 4: Object following results showing estimation error and uncertainty of
estimates (position components of the covariance matrix) as Tr(Cov) from 50
unique initial states $\times$ 30 MC simulations per method. The median values
are plotted at each time step for: LABEL:sub@fig:follow_static Random Walk,
LABEL:sub@fig:follow_cv Constant Velocity, LABEL:sub@fig:follow_cv_ift CV-IFT,
and LABEL:sub@fig:follow_cv_ift_mm CV-IFT with mismatched filter (the hard
estimation problem setting). More effective planning decisions with our MexGen
method yield improved tracking accuracy and lower uncertainty over time.

Results. Importantly, the results in Figure 4 show that MexGen consistently
leads to improved error and lower covariance compared to other methods.
Overall, the MC method yields increased variance in planning decisions, often
causing the agent to repeatedly change direction. Consequently, making it more
difficult for the agent to maintain proximity with the object and generally
resulting in higher covariance and error compared to MexGen. Interestingly, in
Figure 4b, we see the MC and PIM methods display high uncertainty over time
compared to MexGen. This illustrates the impact of poor information gathering
actions resulting from the methods. Objects in the CV model can travel away
from the agent and poor action choices can increase the object-to-agent
distance; consequentially, leading to less informative measurements. Although
in Figure 4c, the impact is less pronounced, as CV-IFT model keeps the object
closer to its initial position because the object frequently changes direction
(see Figure 3).

We can observe a more prominent impact of poor information gathering action
decisions with PIM under the CV-IFT with mismatched filter settings. Figure 4d
shows that the agent using PIM is unable to find the object with the error and
covariance increasing over time. The mismatch between the object dynamics and
the filter’s motion model, reflective of settings in practice, becomes
especially problematic for PIM-based measurement predictions. Although
increased process noise could account for the mismatch by allowing the belief
state to become more uncertain, the point estimate fails to capture the mode
of the belief state distribution. Consequently, the PIM method predicts future
measurements poorly and leads to selecting ineffective information gathering
actions.

###  IV-B Multi-Object Tracking and Localisation

In this section, we evaluate the impact of the measurement prediction methods
on planning time to localize $4$ objects. This considers an information
gathering scenario to find and inspect a number of objects, e.g., to monitor
the health of endangered animals [5, 9]. Here, planning is performed by opting
to reduce the position uncertainty of the closest object—the action that
maximizes information gain—as determined by the estimated states. When the
trace of position components of the covariance for an object indicates
uncertainty is less than approximately $50$ m with 95% confidence, it is
considered localized and the agent will not select it again for planning.

In these experiments, we reduce the initial search space to a 1 km by 1 km
area to ensure the MC simulations complete within a reasonable time-frame.
But, we increase the look-ahead time to 20 s to better plan for events where
two objects are greatly separated as they move in opposite directions.

(a)

(b)

(c)

Figure 5: Agent and object path extracts in the multi-object tracking and
localisation experiments ($\square$: the starting positions; $\triangle$:
ending positions).

Object Trajectory Generation. In these experiments, we used the trajectory
generation method in $\S$ IV-A but reduce the speed of the objects of the CV
and CV-IFT case to be sampled from the interval $[2.5$ m/s, $3$ m/s$]$ and
select starting positions such that each object remains within the search area
for at least $650$ seconds. Example trajectories for the experiments are given
in Figure 5. Again, we generate results from 50 unique initial states $\times$
30 Monte Carlo simulations (1500 MC simulations) conducted per method per
motion model.

Performance Measures.  i) Error: we record the absolute estimation error of
objects when they are considered localised; ii) Time: once all objects are
localised, we record the time for task completion. If the time taken $>1000$ s
we consider the task a fail and discard the result; and iii) Success: report
the ratio of successful trials to total runs as success rate.

Results. The mean and standard deviation results presented in Table I show
that MexGen is able to localise the objects on average faster than the other
two methods. Similar to the results in $\S$ IV-A, we see a greater difference
in the results as the complexity of the object motion increases. Further, we
see a drastic decrease in the success rate of MC and PIM in the CV-IFT and CV-
IFT filter mismatch cases, while MexGen remains close to $1.0$. In the
Mismatch filter case, the hard setting, the success rate of PIM and MC are
reduced further, while MexGen remains high. Consequently, we can observe
MexGen to provide better measurement approximations under high model and state
uncertainty leading to improved selections of information gathering actions.

Table I: Multi-Object tracking & localisation results from 50 unique initial states $\times$ 30 MC simulations per method per model.  Model | Method | Success | Error $\pm\sigma$ (m) | Time $\pm\sigma$ (s)
---|---|---|---|---
| PIM | 1.0 | $12.4\pm 9.2$ | $297.7\pm 93.7$
RW | MC-8 | 1.0 | $12.4\pm 8.6$ | $255.6\pm 64.4$
| MexGen | 1.0 | $12.1\pm 8.4$ | $\boldsymbol{231.0\pm 55.6}$
| PIM | 0.98 | $11.7\pm 9.2$ | $444.7\pm 115.1$
CV | MC-8 | 1.0 | $11.7\pm 8.9$ | $405.2\pm 92.0$
| MexGen | 1.0 | $12.1\pm 9.7$ | $\boldsymbol{360.1\pm 74.8}$
| PIM | 0.53 | $12.8\pm 9.7$ | $645.5\pm 194.7$
CV-IFT | MC-8 | 0.90 | $12.5\pm 9.4$ | $558.6\pm 183.6$
| MexGen | 0.99 | $12.6\pm 9.6$ | $\boldsymbol{465.5\pm 159.9}$
CV-IFT Mismatch (hard setting) | PIM | 0.22 | $12.0\pm 7.4$ | $726.4\pm 181.4$
MC-8 | 0.66 | $11.5\pm 7.1$ | $651.0\pm 185.2$
MexGen | 0.90 | $11.5\pm 7.1$ | $\boldsymbol{567.4\pm 185.9}$

###  IV-C Run-Time Analysis

Figure 6: Run-time analysis results for planners with the different
measurement prediction methods running on the processor used on our aerial
robot (total time is divided into the planning stages in $\S$ IV). Our MexGen
method achieves $7.6\times$ speed-up compared to MC-8 and significantly better
task performance than PIM (see Table I and II). Notably, the relatively small
measure time of MC-8 is because PIM and MexGen measure times include the cost
of the expectation. The results demonstrate the asymmetric computational cost
of the measurement function (Measure time) and the belief state update (Update
time) we leverage in our approximation.

In this section we analyse the run-time of the planner with the different
measurement generation methods. All experiments are run on a DJI Manifold 2-C
processor used on our aerial robot. To provide greater insights into how the
measurement generation method affects planning time, we record the time spent
in each step of the planning algorithm ($\S$ II-A) separately. The medians of
these results are presented in Figure 6, with the sum of medians approximating
the expected total run-time.

From Figure 6, we see that the belief update step dominates the run-time for
all methods. Importantly, generating the measurement samples in MexGen only
results in a small increase compared to PIM, while using MC with $8$ sequence
repetitions results in a significant increase to run-time. Notably,
computational costs are exacerbated when the state space (information to
gather) is large, where predicting and updating the belief state can become
computationally taxing as the probability of being in each of the many states
must be maintained. Consequently, the computational cost of an already complex
problem becomes prohibitive when a large horizon time is considered without an
effective approximation, such as our MexGen.

(a)  Figure 7: The aerial robot with the on-board sensor systems for detecting
the Very High Frequency (VHF) radio sources used in our field trials and
satellite image of the 41 hectare site with a hilly terrain used for the
trials. Figure 8: Example paths followed using PIM vs. MexGen (ours). PIM
leads to selecting less effective actions as it can overestimate the
information gain for actions. For example, at $t=170$, PIM guided robot heads
towards the center of the multi-modal belief state, away from the true object
location and conducts multiple maneuvers near the center, seen at $t=350$. In
contrast, our approach leads to selecting more effective actions as it yields
better estimates of information gain for actions; the robot maneuvers closer
to the objects of interest to reduce range measurement errors and estimation
uncertainty.

###  IV-D Field Experiments

In this section we evaluate the effectiveness of our measurement generation
method in a real-world object localisation scenario using a lightweight quad-
copter platform. In this scenario—modeled after applications in [27, 10, 9, 5,
7], using the measurement model in (19)—four radio tags (Objects 1-4) pulsing
every second on unique frequencies are placed within a hilly environment. The
quad-copter and the hilly environment are depicted in Figure 7. The system can
run fully autonomously. Planning proceeds as in $\S$ IV-B. Importantly, due to
the challenging terrain and to enable us to compare across methods, the tags
are stationary throughout the experiments. The starting location is the
entrance to the field site, where our equipment was located and with a safe
take-off and landing zone for the drone.

Results. The localisation time and error results are in Table II with two
example missions shown in Figure 8 with heat-maps of the paths for all
missions presented in Figure 1. MexGen was able to consistently localise the
objects in shorter times than PIM and MC-8 while all methods achieve similar
localisation accuracy. Figure 1 indicates that PIM tends to head towards the
centre of the environment where the initial radio source point estimates are,
while MexGen selects the more effective, direct paths, towards the objects.
Figure 8 illustrates an instance where PIM generates erroneous measurements
that over estimate the information gain and result in less effective actions.
Even with a $7.6\times$ increase in computational time (see Figure 6), MC-8
demonstrated poor action selection and longer localisation times compared to
MexGen. These results confirm our findings with simulation-based experiments.

Table II: Field experiment results with our aerial robot platform. Method | Missions | Error $\pm\sigma$ (m) | Time $\pm\sigma$ (s)
---|---|---|---
MexGen (Ours) | 10 | $68\pm 16$ | $\mathbf{290\pm 46}$
MC-8 | 10 | $75\pm 25$ | $326\pm 45$
PIM | 10 | $69\pm 12$ | $333\pm 40$

##  V Conclusions

We consider the problem of estimating the information gain of future actions
with respect to the large space of possible future measurements. We propose a
new, computationally efficient and effective method for predicting
measurements to approximate information gain for online planning algorithms to
select the best information gathering actions. We demonstrated our
method—MexGen—provides improved planning performance compared to baseline
methods, with a low computational complexity in a variety of simulated and
real-world scenarios. In general, the belief state and measurement function
can be multi-modal, requiring a computationally expensive method considering
multiple measurements, such as MC, to plan effectively. We have not considered
such a setting. However, even in these settings, MexGen will always yield a
lower prediction error compared to PIM (see $\S$ III-A).

## References

  * [1] D. C. Schedl, I. Kurmi, and O. Bimber, “An autonomous drone for search and rescue in forests using airborne optical sectioning,” _Sci. Robot._ , vol. 6, no. 55, p. eabg1188, 2021.
  * [2] Y. Tao, Y. Wu, B. Li, F. Cladera, A. Zhou, D. Thakur, and V. Kumar, “SEER: Safe Efficient Exploration for Aerial Robots using Learning to Predict Information Gain,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2023, pp. 1235–1241.
  * [3] J. Vander Hook, P. Tokekar, and V. Isler, “Cautious greedy strategy for bearing-only active localization: Analysis and field experiments,” _Journal of Field Robotics_ , vol. 31, no. 2, pp. 296–318, 2014.
  * [4] H. Bayram, J. V. Hook, and V. Isler, “Gathering bearing data for target localization,” _IEEE Robotics and Automation Letters_ , vol. 1, no. 1, pp. 369–374, 2016.
  * [5] O. M. Cliff, D. L. Saunders, and R. Fitch, “Robotic ecology: Tracking small dynamic animals with an autonomous aerial vehicle,” _Sci. Robot._ , vol. 3, no. 23, p. eaat8409, 2018.
  * [6] F. Chen, H. V. Nguyen, D. A. Taggart, K. Falkner, S. H. Rezatofighi, and D. C. Ranasinghe, “ConservationBots: Autonomous aerial robot for fast robust wildlife tracking in complex terrains,” _J. Field Robot._ , 2023.
  * [7] H. Bayram, J. Vander Hook, and V. Isler, “Gathering bearing data for target localization,” _Robot. and Autom. Letters_ , vol. 1, no. 1, pp. 369–374, 2016.
  * [8] M. K. Yılmaz and H. Bayram, “Particle filter-based aerial tracking for moving targets,” _J. Field Robot._ , vol. 40, no. 2, pp. 368–392, 2023.
  * [9] H. V. Nguyen, M. Chesser, L. P. Koh, S. H. Rezatofighi, and D. C. Ranasinghe, “TrackerBots: Autonomous unmanned aerial vehicle for real-time localization and tracking of multiple radio-tagged animals,” _J. Field Robot._ , vol. 36, no. 3, pp. 617–635, 2019.
  * [10] L. Dressel and M. J. Kochenderfer, “Hunting Drones with Other Drones: Tracking a Moving Radio Target,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2019, pp. 1905–1912.
  * [11] M. Araya, O. Buffet, V. Thomas, and F. Charpillet, “A POMDP Extension with Belief-dependent Rewards,” in _Adv. Neural Inf. Process. Syst. (NIPS)_ , 2010.
  * [12] Z. Sunberg and M. Kochenderfer, “Online Algorithms for POMDPs with Continuous State, Action, and Observation Spaces,” _Int. Conf. on Autom. Planning and Scheduling_ , vol. 28, no. 1, pp. 259–263, 2018.
  * [13] J. Fischer and O. S. Tas, “Information Particle Filter Tree: An Online Algorithm for POMDPs with Belief-Based Rewards on Continuous Domains,” in _Int. Conf. Mach. Learning (ICML)_ , 2020, pp. 3177–3187.
  * [14] O. Sztyglic and V. Indelman, “Speeding up POMDP Planning via Simplification,” in _IEEE Int. Conf. Intell. Robots Syst. (IROS)_ , 2022, pp. 7174–7181.
  * [15] J. Ott, E. Balaban, and M. J. Kochenderfer, “Sequential Bayesian Optimization for Adaptive Informative Path Planning with Multimodal Sensing,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2023, pp. 7894–7901.
  * [16] L. Burks, I. Loefgren, and N. R. Ahmed, “Optimal continuous state pomdp planning with semantic observations: A variational approach,” _IEEE Trans. Robot._ , vol. 35, no. 6, pp. 1488–1507, 2019.
  * [17] P. Cai, Y. Luo, D. Hsu, and W. S. Lee, “HyP-DESPOT: A hybrid parallel algorithm for online planning under uncertainty,” _Int. J. Robot. Res._ , vol. 40, no. 2-3, pp. 558–573, 2021.
  * [18] H. V. Nguyen, H. Rezatofighi, B.-N. Vo, and D. C. Ranasinghe, “Multi-objective multi-agent planning for jointly discovering and tracking mobile objects,” _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 05, pp. 7227–7235, 2020.
  * [19] C. H. Papadimitriou and J. N. Tsitsiklis, “The complexity of Markov decision processes,” _Mathematics of operations research_ , vol. 12, no. 3, pp. 441–450, 1987.
  * [20] M. Beard, B.-T. Vo, B.-N. Vo, and S. Arulampalam, “Void probabilities and Cauchy–Schwarz divergence for generalized labeled multi-Bernoulli models,” _IEEE Trans. Signal Process._ , vol. 65, no. 19, pp. 5047–5061, 2017.
  * [21] B. Ristic, B.-N. Vo, and D. Clark, “A Note on the Reward Function for PHD Filters with Sensor Control,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 47, no. 2, pp. 1521–1529, 2011.
  * [22] R. P. S. Mahler, _Statistical Multisource-Multitarget Information Fusion_. USA: Artech House, Inc., 2007.
  * [23] K. Sun and V. Kumar, “Belief Space Planning for Mobile Robots With Range Sensors Using iLQG,” _Robot. and Autom. Letters_ , vol. 6, no. 2, pp. 1902–1909, Apr. 2021.
  * [24] B. Ristic, B.-T. Vo, B.-N. Vo, and A. Farina, “A tutorial on Bernoulli filters: theory, implementation and applications,” _IEEE Trans. Signal Process._ , vol. 61, no. 13, pp. 3406–3430, 2013.
  * [25] D. Svensson, “Derivation of the discrete-time constant turn rate and acceleration motion model,” in _Sensor Data Fusion: Trends, Solutions, Applications (SDF)_ , 2019, pp. 1–5.
  * [26] S. McGinnity and G. Irwin, “Multiple model bootstrap filter for maneuvering target tracking,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 36, no. 3, pp. 1006–1012, 2000.
  * [27] L. Dressel and M. J. Kochenderfer, “Pseudo-bearing Measurements for Improved Localization of Radio Sources with Multirotor UAVs,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2018, pp. 6560–6565.


# Hybrid CNN -Interpreter: Interprete local and global contexts for CNN-based
Models

Wenli Yang
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
&Guan Huang
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
Renjie Li
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
&Jiahao Yu
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
&Yanyu Chen
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
&Quan Bai
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>
&Byeong Kang
School of Information and Communication Technology
University of Tasmania
Australia
<EMAIL_ADDRESS>

###### Abstract

Convolutional neural network (CNN) models have seen advanced improvements in
performance in various domains, but lack of interpretability is a major
barrier to assurance and regulation during operation for acceptance and
deployment of AI-assisted applications. There have been many works on input
interpretability focusing on analyzing the input-output relations, but the
internal logic of models has not been clarified in the current mainstream
interpretability methods. In this study, we propose a novel hybrid CNN-
interpreter through: (1) An original forward propagation mechanism to examine
the layer-specific prediction results for local interpretability. (2) A new
global interpretability that indicates the feature correlation and filter
importance effects. By combining the local and global interpretabilities,
hybrid CNN-interpreter enables us to have a solid understanding and monitoring
of model context during the whole learning process with detailed and
consistent representations. Finally, the proposed interpretabilities have been
demonstrated to adapt to various CNN-based model structures.

_K_ eywords Hybrid CNN-interpreter, local interpretability, global
interpretability, correlation, filter importance

Figure 1: Hybrid CNN-interpreter: understand how different convolutional
neural networks learn through layers and filters: 1) Local interpretability to
represent each layer’s output and learning ability; 2) Global interpretability
to explore the representations ability of feature maps(learned by different
filters) by using regression models; 3) feature importance analytic to
indicate the contributions of convolution filters when making predictions.
Overall, it enables for both local and global interpretability for different
CNN structures.

##  1 Introduction

Although the performance of convolutional neural network (CNN) models has
significantly increased over the past decade, yet without reliable
interpretabilities that effectively represent the learning processes, humans
still consider untrust¬worthy when continue to view CNN models to be used as
unreliable when utilized for real-world decision making [1]. The lack of trust
undermines the deployment of CNN-based technologies into many domains, such as
medical diagnosis [2], healthcare [3], autonomous driving [4], etc. This
motivates the need for building trust in these CNN-assisted decision making.
Building trust by adding interpretabilities has significant theoretical and
practical value for both improving model performance and enhancing
transparency. Interpretabilities can be represented by using a variety of
expressions, such as visual interpretability and semantic interpretability
[5]. Among different expressions, the visual interpretability of CNN-based
models is the most fundamental and direct way to explain the network
representations.

In terms of visual interpretability methods, the interpretability of CNN-based
models emphasizes either visualization of training data rules [6] or the
visualization inside the model [5]. Presently, most visual interpretability
methods focused more on the understanding of input features on the model
performance ensured rather than on the context of CNN-based models, which can
reveal the learning process through each layer and the generic features
learned from any black box models. Additionally, the internal logic of models
has not been clarified in the current mainstream visual interpretability
methods. Most existing internal logical discussion about models is used for
tree-based models [7]. For CNN-based models, to the best of our knowledge,
correlation interpretabilities have not been discussed so far.

We present a novel hybrid CNN interpreter, which builds stacking ensemble
models of each layer for local interpretabilities and extends local
interpretabilities to compute global correlation to assess the importance of
convolution filters. The results are illustrated by binary classification of
brain tumor data in the different model structures. It makes three innovative
improvements:

  1. 1.

Build an original stacking forward propagation algorithm by computing the
contributions of each layer to the final prediction. In stacking, each time
takes the outputs of the specific layer as input and connects to the final
probability mapping layer directly, which can use the set of predictions as a
local context and examine feature maps learned by different layers’ ability
for the final prediction contributions.

  2. 2.

Extend local interpretabilities to a global interpretation by layer-based
regression models, which are constructed by using all the local
interpretabilities as dependent variables and each feature map representation
as an independent variable across the entire dataset. This enables the
examination of the representation ability of feature maps (learned by
different filters) for interpreting a model’s global behavior.

  3. 3.

Extract filter importance by assigning scores to each filter in each layer of
a model that indicates the correlation of convolution filters when making a
prediction, which can help people understand the relationship between the
filters and the target variable and can conditionally improve the performance
of the model, potentially making the model lighter and speeding up the model’s
working by removing the unimportant filters.

  4. 4.

Applicable for a broad range of CNN-based models, which can be utilized to
interpret both sequence and nonsequence-focused models. For the sequence
models, the interpretabilities can be through each layer to provide the
learning context, whereas for the nonsequence models, the interpretabilities
can be customized for different stages, blocks, or specific layers, etc. This
reveals how different learning is in different models and in different
training processes.

##  2 Related Work

###  2.1 Convolutional neural network structures

CNNs with strong representation ability of deep structures have ever-
increasing popularity in many applications. Figure 1 shows the historical
evolution of various CNN models. AlexNet [8] is a leading structure of
convolutional neural networks and has huge applications for classification
tasks. The evolution after the AlexNet can be mainly summarized in two ways:
sequence-focused models by increasing the depth of networks and nonsequence-
focused models by adding units or modules.

Figure 2: Summarization of CNN structures

Sequence focused models: by leveraging the sequence information through layers
to improve the model performance, such as VGG16 and VGG19 [9], MSRANet [10].
All of these models are focused on using multiple convolutional layers and
activation layers to increase the depth of network, which can extract deeper
and better features than the simple structure.

Non-sequence-focused models: by using modular structures to add functional
units or components to extend the width of network. Typical examples such as
GoogLeNet [11], Inception [12]. GoogLeNet utilizes multiple branches, which
allows the network to choose between multiple convolutional filter sizes in
each branch. An inception network comprises repeating components referred to
as inception modules to increase the representational power of a neural
network.

By combining them together, ResNet [13] uses four stages made up of residual
blocks, each of which uses several residual blocks with convolutional blocks
and identity blocks. Finally, residual blocks are stacked together on top of
each other to form the whole network, which can gain better performance from
considerably increased depth.

In this paper, we will pick up AlexNet and ResNet as two representative models
to interpret the learning process of different model structures.

###  2.2 Visual interpretabilities of CNN-based models

According to different implementation methods, the visual interpretability
methods can be divided into three different categories: input visualization
method, model visualization method, and mixed visualization method. Table 1
states the overall summary of possible explainable methods.

Table 1: Different types of visual interpretability methods | Types of Interpretable
---
Method
Description
Input Visualization |  | Provides an accessible way to view and understand the impact
---
of initial input data on the final model performance.
Model Visualization | Provides analytics of layer-based outputs inside models
Mixed Visualization |  | Provides both input visualization and model visualization,
---
which focused on building connections or correlations.

The input visualization explains processes simultaneously from the initial
input stage to the final output result. For example, Jeyaraj and Nadar [14]
stated a regression-based partitioned method for oral cancer diagnosis. The
network was demonstrated with two partitioned layers for labelling and
categorized a multidimensional hyperspectral picture by tagging the region of
interest. It handled feature maps with little variation and complex vector
feature maps. Another example was the importance estimation network produced
by Gu et al. [15], which was diagnosed by the classification network by
investigating the irrelevant information. The model aims to detect the most
significant sections of the original input images and provided an accurate
diagnosis after being trained with the proper regularization settings. The
input visualization merely provides a user-friendly interface for seeing and
understanding input data, but no information on how the relevant features
contribute to the prediction is supplied.

Generally, model visualization is based on the level of layers and finds out
the prediction results of the model between different layers. For example,
Graziani et al.[16] employed “The concept activation vector (TCAV)” to
transform medical pictures into quantitative characteristics by using
radiomics, which focused on explaining the predicted output globally according
to high-level visual features. Additionally, Villain et al. stated a novel
GradCAM method on brain MRI image datasets [17]. To identify the regions of
interest by the visual convolutional neural network models, the authors
claimed that the output of every convolution layer was collected by the global
average pool layer and merged to obtain a single activation map to visualize.
The model visualization could find hidden problems inside the model, but the
currently selected features used for model interpretability may not be
repeatable in each input.

Hybrid visualization model refers to a combination of both input visualization
and model visualization. This model tries to focus on not only explaining the
local prediction but also exploring global knowledge inside the model. Hybrid
visualization model concentrates on finding the connections or correlations
between the input aspect and model layer aspect. Unfortunately, to my
knowledge, there are no obvious research on mixed visualization method.

In this paper, we discuss how to interpret CNN-based models using hybrid
visualization to make the interpretability appropriate for different model
structures. We describe the local interpretability by building a set of
stacked ensemble models to provide various input visualization. Then, we
extend local interpretability to model visualization by using regression-based
analytics. Finally, we identify the importance of each filter in each layer to
represent the global context of the models.

##  3 Method

Hybrid CNN-interpreter aims to explain the deep learning model’s different
layers and the filter’s feature representation ability for the final
prediction contribution. It provides global insight into not only the model’s
layer level, but also the filter level along that layer. The output of the
hybrid CNN-interpreter is a layer and filter-based importance distribution
matrix, representing the importance level of feature maps learned from
different layers and filters by the model. The hybrid CNN-interpreter consists
of the original CNN-model forward propagation module, linear regression module
and filter importance analysis module.

Figure 3: Hybrid CNN-interpreter framework: including Original CNN-model
Forward Propagation, Linear Regression Module and Filter Importance Analysis.

###  3.1 Stacking Forward Propagation

In the first original forward propagation module, each image is sent to the
original model. We skip and connect each layer ($l_{i}(X_{ij})$) directly to
the final probability mapping layer ($f(\cdot)$), and the output probability
($Y_{i}$) is recorded. This is to examine feature maps learned by different
layers’ ability for the final classification contribution (Equation 1).

$\displaystyle f[l_{i}(X_{ij})]=Y_{i}$ (1)

where $X_{ij}$ represents the $j^{th}$ feature map at the $i^{th}$ layer.

###  3.2 Linear Regression Module

In the second step, a linear regression model is constructed by using the
output probability as a dependent variable and each feature map’s mean value
as independent variables. This is to examine feature maps’ (learned by
different filters) representation ability for the classification (Equation 2).

$\displaystyle Y_{i}=b_{i0}+\sum{b_{ij}\overline{X}_{ij}}$ (2)

where $\overline{X}_{ij}$ represents the mean value of feature map $X_{ij}$.
We applied L2 regularization and the objective function is represented in
Equation 3.

$\displaystyle L=\sum{(\hat{Y_{i}}-Y_{i})^{2}+\lambda\sum{b_{ij}^{2}}}$ (3)

where $\hat{Y_{i}}$ is the predicted value and $Y_{i}$ is the true value.

Algorithm 1 indicates the process of how $\overline{X}_{ij}$ are extracted
from the model outputs.

Algorithm 1 Mean values of feature maps extraction

1:procedure export-features($img,lnames$)$\triangleright$ The input of this
algorithm are images and the layer names of the model

2: $model.predict(img)\leftarrow features$ $\triangleright$ The feature maps
are obtained by visualisation the model prediction.

3: for $lname,features$ do

4: $mean$-$list=$ [ ]

5: for $i$ in range($features$-$number$) do

6: $x$ = $features$[0, :, :, i]

7: $x$ = $x$ \- $x$.mean() $\triangleright$ $x$.mean() the mathematical mean
value of $x$

8: $x$ = $x$ / $x$.std() $\triangleright$ $x$.std() the mathematical standard
deviation value of $x$

9: $x$ = $x$ * 32

10: $x$ = $x$ \+ 64

11: $x$ = numpy.clip($x$, 0, 255) $\triangleright$ Use numpy.clip() function
to limit values outside the interval are clipped to the interval edges

12: $mean$-$list$.append($x$) $\triangleright$ Store the mean value of the
feature maps into $mean$-$list$

13: end for

14: end for

15: return $mean$-$list$$\triangleright$ The output of the algorithm is a list
with the mean value of the feature maps

16:end procedure

In our experiment, we use Ridge regression model [18] to calculate the
variance of our feature maps. The Ridge model solves regression problems with
an $L2$ regularization loss function, the equation of the loss is shown in
Equation 3. In the Ridge model, the parameter $\alpha$ ($\alpha$ = 1 in our
experiment) is used to control the regularization strength. The input of our
regression model is the mathematical mean value of our feature maps and the
output of the CNN models after prediction (confidence score in our
experiment). Algorithm 2 shows the details of

Algorithm 2 Regression Algorithm

procedure regression($layer$-$names$,$f$-$mean$,CNN-
predictions)$\triangleright$ The input of this algorithm are mean values of
the feature maps and the layer names of the model

2: x = $f$-$mean$

y = CNN-predictions

4: model = Ridge(alpha=1.0) $\triangleright$ The model is loaded from sklearn
library

predictions = model.predict(x)

6: coefficients, standard-errors, t-values, p-values = sklearn-math-
function(y, predictions,features) $\triangleright$ we use a combination of
mathematical tools from sklearn library to calculate coefficients, standard-
errors, t-values and p-values

return coefficients, standard-errors, t-values, p-values

8:end procedure

Then, the regression model will return four parameters to represent the
importance of our feature map, i.e. the coefficient of determination
(${R}^{2}$), standard-errors (SEs), t-values and p-values. The equation of
${R}^{2}$ is demonstrated in 4. The meaning of ${R}^{2}$ is the proportion of
the variance of the variable that is explained by this predicted model. In our
experiment, we use ${R}^{2}$ to evaluate the reliability of the prediction, if
the value of ${R}^{2}$ close to 1, which means the model predictions are
reliable and effective. Conversely, if the value of ${R}^{2}$ is close to 0,
the predictions of the model are not reliable and ineffective.

$\displaystyle
R^{2}=\dfrac{\sum_{i}e_{i}^{2}}{\sum_{i}\left(y_{i}-\overline{y}\right)^{2}}$
(4)

The equation of SE is shown in Equation 5. The SE is another statistical tool
for us to evaluate the degree of deviation of the sample mean from the overall
true mean. Normally, a higher SE value indicates that the mean value of the
feature maps is more distinct, whereas a smaller SE value means the features
extracted are similar.

$\displaystyle SE=\dfrac{\sigma}{\sqrt{n}}$ (5)

where $SE$ is the standard error of the sample, $\sigma$ is the sample
standard deviation and $n$ is the number of samples.

###  3.3 Filter Importance Analysis Module

In the third filter importance analysis module, a matrix of different filters’
representation scores has been calculated, the matrix helps people understand
the contribution of the feature map learned by different filters in each layer
of the network to the final classification result.

In our settings, the t-value (Equation 6) is used to indicate the level of
difference between features. The t-value is calculated as the difference
expressed in standard error units. It means that the standard deviation
estimated is far away from 0. A large t-value indicates the evidence against
the null hypothesis, in other words, we could declare a relationship between
the CNN-predictions and the feature maps.

$\displaystyle t=\dfrac{\overline{x}-\mu_{0}}{\dfrac{s}{\sqrt{n}}}$ (6)

where $\overline{x}$ is the sample mean, $\mu_{0}$ is the population mean, $s$
is the sample standard deviation and $n$ is sample size.

We also used p-value (Equation 7) to evaluate two variables, namely, they are
the importance of the features and the CNN model prediction accuracy. We want
to examine the relationship between the predictor variables and the response
variable to find out if higher accuracy in model prediction will result in
higher importance values in the feature maps’ mean values.

$\displaystyle
p=\dfrac{\widehat{p}-p0}{\dfrac{\sqrt{p_{0}\left(1-p_{0}\right)}}{n}}$ (7)

where $\widehat{p}$ is sample proportion, $p_{0}$ is assumed population
proportion in the null hypothesis and $n$ is the sample size.

##  4 Experiment and Discussion

###  4.1 Data preparation

A total of 253 public brain MRI images dataset (155 tumorous images and 98
nontumorous images) are used to build the classification models. Since CNNs
can be independent of translation, view position, size, and lighting [19], the
data augmentation was applied by manually flipping and rotating the image
sets. Moreover, considering the dimension of the image may be changed after
rotation [20], the image will be only rotated arbitrarily between 0 and 10
degrees in our experiments.

Finally, after the above data augmentation, the dataset has 1,085 positive and
980 negative examples. We use 70% of the images for training and 15% of the
images for validating to generate the AlexNet and ResNet classifier models.
The remaining of 15% of the images are for model interpretability in our
proposed hybrid CNN-interpreter.

###  4.2 Model selection

In this study, two alternative CNNs, namely, AlexNet and ResNet, are selected
to demonstrate the detailed and consistent interpretabilities of our proposed
methods. The model structures of AlexNet and ResNet are represented in Table 1
and 3 respectively.

Table 2: The network structure of AlexNet model Layer type | Size of output feature map | Number of filters
---|---|---
Conv_2d | 59*59 | 64
Bn | 59*59 | 64
Max_pooling_2d | 29*29 | 64
Conv_2d_1 | 29*29 | 64
Bn_1 | 29*29 | 64
Max_pooling_2d_1 | 14*14 | 64
Conv_2d_2 | 14*14 | 64
Bn_2 | 14*14 | 64
Conv_2d_3 | 14*14 | 64
Bn_3 | 14*14 | 64
Conv_2d_4 | 14*14 | 64
Bn_4 | 14*14 | 64
Max_pooling_2d_2 | 7*7 | 64
Table 3: The network structure of ResNet model Stage | Operations | Size of output feature map | Number of filters
---|---|---|---
Stage 1 | Conv_1 | 120*120 | 64
Bn_1 | 120*120 | 64
Activation | 120*120 | 64
Max_pooling_2d | 59*59 | 64
Stage 2 | Stage2_input | 59*59 | 64
Convolutional block | 59*59 | 256
Identity block 1 | 59*59 | 256
Identity block 2 | 59*59 | 256
Stage 3 | Stage3_input | 30*30 | 128
Convolutional block | 30*30 | 512
Identity block 1 | 30*30 | 512
Identity block 2 | 30*30 | 512
Identity block 3 | 30*30 | 512
Stage 4 | Stage4_input | 15*15 | 256
Convolutional block | 15*15 | 1024
Identity block 1 | 15*15 | 1024
Identity block 2 | 15*15 | 1024
Identity block 3 | 15*15 | 1024
Identity block 4 | 15*15 | 1024
Identity block 5 | 15*15 | 1024
Stage 5 | Stage5_input | 8*8 | 512
Convolutional block | 8*8 | 2048
Identity block 1 | 8*8 | 2048
Identity block 2 | 8*8 | 2048

###  4.3 Local interpretability for CNN-based Models

The hybrid CNN-interpreter enables local interpretability with individual
repeatable capacity by building a set of stacked ensemble submodels. Each
submodel is developed by connecting the output of each layer in the original
model and the final probability mapping layer to output prediction results,
representing the model’s learning ability. For the AlexNet model, the local
interpretability will focus on each layer. For the ResNet model, the local
interpretability will concentrate on each stage and component in each stage.
The hybrid CNN interpreter will collect this set of prediction results for
individual predictions.

In our experiments, four different samples are input into the AlexNet model to
explore the local interpretability, where we pick up both tumour and nontumour
images as well as both the final correct prediction and wrong prediction as
shown in the Table 4.3.

Figures | Tumor image | Nontumor image | Result
---|---|---|---
Figure 4 (a) |  |  |
Figure 4 (b) |  |  |
Figure 4 (c) |  |  |
Figure 4 (d) |  |  |
Figure 4: Local interpretability results of AlextNet model. a,Wrong prediction
results of brain tumour.b,Correct prediction results of brain tumour.c,Wrong
prediction results of non-brain tumour.d,Correct prediction results of non-
brain tumour

Figure 4 represents the prediction results of each layer in the AlexNet model,
which is guaranteed consistent for repeatable running. From the local
interpretability, we can see that each layer of different samples makes
different individual predictions. Moreover, it is not an actual rule "more
layers = better performance". For example, the $Conv2d\\_1$ layer demonstrates
the best performance in Figure (a) and the $Batch\\_normalization\\_2$ layer
achieved the better performance in Figure 4(c).

Additionally, we interpret the ResNet model by computing a set of prediction
results for each stage and each block as well. As shown in Figure 5, the
ResNet model shows that stages two and three have better performance compared
with other stages in this test sample. For each internal stage, our proposed
local interpretability can also provide a shortcut how learning by jumping
over different residual blocks. For example, stage 5 is composed of one
convolutional block and two identity blocks. The best performance in this
stage was achieved by connecting the input layer and the first identity block.

Figure 5: Local interpretability results of ResNet Model: tumor image with
final correct prediction results

###  4.4 Global interpretability for CNN-based Models

By combining the local interpretability of each layer across the entire
training data, we present the regression-based methods to provide global
insight into each filter in each layer, which will capture the global pattern
of filters and the relationship between feature representations and prediction
results. The experiments from the hybrid CNN-interpreter for global
interpretability cover: 1) summary plots of the linear coefficient to
represent the strength and direction of the linear relationship between each
filter and prediction results in each layer, stage or internal block; 2)
correlation analysis through layers of the AlexNet model and stages/blocks of
the ResNet model respectively; 3) Filter importance analysis to reveal the
details of positive and negative feature representations.

We summarize the linear coefficients of all layers in the AlexNet model as
shown in Figure 6, and the distribution of each filter’s coefficient in each
layer can also be displayed by highlighting the positive as blue and the
negative as red. Based on the results, we can see that filters 20 and 31 show
the majority of positive effects among all the layers, whereas filters 28 and
35 show negative effects in most layers.

Figure 6: Summary plot of linear coefficients in AlextNet Model

We also get the linear coefficients of all stages and every internal block in
the ResNet model.The overall distribution of coefficients among all stages can
be shown in Figure 7 (a), and by using stage 4 as a reference, the positive
and negative coefficients can be displayed in Figure 7 (b). Furthermore, to
explore richer information, we can set a small range of filters such as
filters 256-288 in stage 4, some consistent patterns among all the internal
blocks can be revealed as shown in Figure 7 (c), filter 263 is positive in
most blocks, and filters 259, 270, 284, and 285 usually are negative in
majority internal blocks. This type of information can provide evidence for
deriving lightweight models by eliminating layers, stages, or filters.

Figure 7: Summary plot of linear coefficients in ResNet Model. a,Linear
coefficients of all stages. b,Positive and negative coefficient distribution
in stage 4. c, Coefficient patterns of internal blocks in stage 4 by setting a
specific filter range.

Based on the linear coefficient results above, we analyze the correlation
between layers in the AlexNet model and the correlation between stages and
internal blocks in each stage. Figure 8 (a), we shows that the pairwise layers
such as $Conv2d$ and $Batch\\_normalization(Bn)$, $Conv2d\\_1$ and
$Batch\\_normalization\\_1(Bn\\_1)$, etc. always have strong correlation with
each other, which can reflect the local interpretability that these pairwise
layers get the similar prediction results as well. Moreover, we can clearly
see the $Con2d\\_4$ and $Bn\\_4$ layers have global negative effects for the
final prediction results, compared with the local interpretability in Figure4,
the prediction results in the $Max\\_pooling\\_2d\\_2$ layer almost get
noticeably different results compared with $Conv2d\\_4$ layer and $Bn\\_4$
layers, which can approve the consistency between local and global
interpretability.

For the ResNet model, Figure 8 (b) shows that $stage4$ has negative effects
from $stage1$ to $stage3$. This reflects the consistency with the local
interpretability in Figure 5 that the performance increased from $stage1$ to
$stage3$ but dropped starting from the $stage4$. Moreover, the correlations
among the internal blocks in $stage4$ indicate a generally positive
correlation with each residual block. However, the correlations get less and
less along with the blocks getting deeper, such as the correlation between
$Conv_{b}lock$ and following identity blocks, is getting lower as shown in
Figure 8 (c).

Figure 8: Correlation analysis among layers, stages or blocks for global
interpretability. a,Correlation coefficients between layers in AlexNet Model.
b,Correlation coefficients between stages in ResNet Model.c,Correlation
coefficients between internal blocks in ResNet Model (by using stage 4 as
reference).

Finally, we extract the most essential filters in each layer to reveal the
details of positive and negative feature representations. Figure 9 shows each
layer’s most significant five positives and five negatives using the same
sample image in Figure 4 (a). From Figure 9, we can see the different filters
positively and negatively impact on different layers. For example, filter 20
positively impacts a total of nine layers that always focus on lower and
middle layers, whereas filter 37 shows a positive impact on deeper layers.
Filters 35 and 28 have a negative impact throughout the whole learning
process. Additionally, the pairwise learning layers (pair of convolutional
layer and batch normalization layer) show similar feature representations,
which is consistent with our local interpretability. Figure 10 summarises the
filter importance analysis for the ResNet model. We pick up the input layer
and output layer for each stage as well as the output layer for each internal
block to show the feature map representations and their linear coefficient.
Since the ResNet model is much more deeper than the AlexNet model, we can see
that the deeper the layer, the more ambiguous the feature maps and the more
difficult it is for the humans to understand their semantic information.

By combining local interpretability and global interpretability, the
experiments show that when the forward probability of the submodel is high,
the features learned by the selected top five positive filters are consistent
with human cognition with more significant semantic information, such as both
the five filters (20,26,62,45,and 31) of $Conv\\_2d\\_1$ layer in AlexNet
model and the filters (151,24,118,190,and 48) of $identity\\_block\\_2$ output
layer in ResNet model showing the part of important features in the tumor
area. Moreover, when the forward probability of the submodel is low, filters
marked as negative effects can learn valuable semantic information. For
example, the feature representation of filter 15 of the $Mp\\_2d\\_2$ layer in
the AlexNet model actually covers part of the tumor area, and the filter 45 of
the input layer in $stage3$ in the ResNet model can represent parts of
features in the tumor area as well. In contrast, filters considered positive
by the model cannot learn the critical parts of the image (e.g., 42 and 59
filters in $Bn\\_3$ layer in the AlexNet model as well as 63 and 107 filters
of the input layer in $stage3$) can be understood that wrong cognition of the
model leads to a low forward probability result. These could also verify the
validity and consistency of our hybrid CNN-interpreter between the local and
global interpretabilities.

Figure 9: Summary of filter importance analysis for AlexNet model Figure 10:
Summary of filter importance analysis for ResNet model

##  5 Conclusion and Future Work

The hybrid CNN interpreter helps users to have a deep conceptual understanding
of CNN-based models by providing both local and global interpretabilites. The
local interpretability by using original forward propagation can reveal how
image data progresses through the layers of the CNN-based models, whereas the
global filter importance based on the linear regression module can indicate
how much each filter contributes to the model prediction.

The hybrid CNN interpreter can be widely used in different computer vision
tasks, such as classification, object detection, and segmentation. The
proposed interpreter can be flexibly adapted to various CNNs (layer-based,
stage-based, or internal-block-based). The correlations between layers,
stages, or blocks can also be generalized to understand the context of complex
CNN structures. By demonstrating how to apply the hybrid CNN interpreter to
explain different types of CNN-based models, we take brain tumor
classification tasks to provide an interpretation of the AlexNet and ResNet
models. The experiment results showed the efficiency and consistency between
local and global interpretabilities, as well as among different models.

The proposed interpreter can be used to improve and debug models and to make
CNN-based models more accurate and reliable. The local interpreter can reveal
prediction results for specific samples of different learning processes, and
based on these interpretabilities, we can set the gating and memory mechanisms
to debug models that can help us identify where to access that memory and
where to ignore it during the whole learning process. For example, the model
can skip some layers with bad performances and tell where to access and make
connections again. The feature correlations and filter importance identified
by the global interpreter enable developers to determine which filter can be
eliminated to get better performance.

## References

  * [1] Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil K Jain, and Jiliang Tang.  Trustworthy ai: A computational perspective.  arXiv preprint arXiv:2107.06641, 2021.
  * [2] Yiming Zhang, Ying Weng, and Jonathan Lund.  Applications of explainable artificial intelligence in diagnosis and surgery.  Diagnostics, 12(2):237, 2022.
  * [3] Urja Pawar, Donna O’Shea, Susan Rea, and Ruairi O’Reilly.  Explainable ai in healthcare.  In 2020 International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA), pages 1–2. IEEE, 2020.
  * [4] Shahin Atakishiyev, Mohammad Salameh, Hengshuai Yao, and Randy Goebel.  Explainable artificial intelligence for autonomous driving: A comprehensive overview and field guide for future research directions.  arXiv preprint arXiv:2112.11561, 2021.
  * [5] Quan-shi Zhang and Song-Chun Zhu.  Visual interpretability for deep learning: a survey.  Frontiers of Information Technology & Electronic Engineering, 19(1):27–39, 2018.
  * [6] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  arXiv preprint arXiv:1801.01401, 2018.
  * [7] Kerelous Waghen and Mohamed-Salah Ouali.  Multi-level interpretable logic tree analysis: A data-driven approach for hierarchical causality analysis.  Expert Systems with Applications, 178:115035, 2021.
  * [8] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [9] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [10] Qiuyu Zheng, Zengzhao Chen, Hai Liu, Yuanyuan Lu, and Jiawen Li.  Msranet: Learning discriminative embeddings for speaker verification via channel and spatial attention mechanism in alterable scenarios.  Available at SSRN 4178119.
  * [11] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Pandia Rajan Jeyaraj and Edward Rajan Samuel Nadar.  Computer-assisted medical image classification for early diagnosis of oral cancer employing deep learning algorithm.  Journal of cancer research and clinical oncology, 145(4):829–837, 2019.
  * [15] Donghao Gu, Yaowei Li, Feng Jiang, Zhaojing Wen, Shaohui Liu, Wuzhen Shi, Guangming Lu, and Changsheng Zhou.  Vinet: A visually interpretable image diagnosis network.  IEEE Transactions on Multimedia, 22(7):1720–1729, 2020.
  * [16] Mara Graziani, Vincent Andrearczyk, Stéphane Marchand-Maillet, and Henning Müller.  Concept attribution: Explaining cnn decisions to physicians.  Computers in biology and medicine, 123:103865, 2020.
  * [17] Edouard Villain, Giulia Maria Mattia, Federico Nemmi, Patrice Péran, Xavier Franceries, and Marie Véronique le Lann.  Visual interpretation of cnn decision-making process using simulated brain mri.  In 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), pages 515–520. IEEE, 2021.
  * [18] Arthur E Hoerl and Robert W Kennard.  Ridge regression: Biased estimation for nonorthogonal problems.  Technometrics, 12(1):55–67, 1970.
  * [19] Jiachen Yang, Chenguang Wang, Bin Jiang, Houbing Song, and Qinggang Meng.  Visual perception enabled industry intelligence: state of the art, challenges and prospects.  IEEE Transactions on Industrial Informatics, 17(3):2204–2219, 2020\.
  * [20] Chuanfei Hu and Yongxiong Wang.  An efficient convolutional neural network model based on object-level attention mechanism for casting defect detection on radiography images.  IEEE Transactions on Industrial Electronics, 67(12):10922–10930, 2020.