11institutetext: Department of Computer Science, University of Münster,
Germany 11email<EMAIL_ADDRESS>
# DeepCSHAP: Utilizing Shapley Values to Explain Deep Complex-Valued Neural
Networks

Florian Eilers 11 0000-0003-0726-3287 Xiaoyi Jiang 11 0000-0001-7678-9528

###### Abstract

Deep Neural Networks are widely used in academy as well as corporate and
public applications, including safety critical applications such as health
care and autonomous driving. The ability to explain their output is critical
for safety reasons as well as acceptance among applicants. A multitude of
methods have been proposed to explain real-valued neural networks. Recently,
complex-valued neural networks have emerged as a new class of neural networks
dealing with complex-valued input data without the necessity of projecting
them onto $\mathbb{R}^{2}$. This brings up the need to develop explanation
algorithms for this kind of neural networks. In this paper we provide these
developments. While we focus on adapting the widely used DeepSHAP algorithm to
the complex domain, we also present versions of four gradient based
explanation methods suitable for use in complex-valued neural networks. We
evaluate the explanation quality of all presented algorithms and provide all
of them as an open source library adaptable to most recent complex-valued
neural network architectures.

###### Keywords:

Explainable AI Complex-valued Neural Networks

##  1 Introduction

Many applications profit from the success of deep neural networks. Most
applications deal with real-valued numbers, such as natural images, encodings
for natural language or real-valued signals. The standard real-valued deep
neural networks are suitable to solve problems with these kinds of data. There
is a class of applications, however, that naturally deal with complex numbers
in the data, such as MRI [8], PolSAR satellite images [4] or Fourier
transforms of real-valued signals for frequency analysis [29]. These kinds of
data motivate the use of complex-valued neural networks (CVNNs), where all
building blocks of deep neural networks are translated to the complex domain
to directly process these data without use of an isomorphism from $\mathbb{C}$
to $\mathbb{R}^{2}$. It has been shown that in many applications (but not all
[32]) complex-valued neural networks show superior performance, especially
when dealing with complex-valued input data but sometimes even when dealing
with real-valued input data [15].

On major drawback of real- and complex-valued deep learning architectures is
their lack of interpretation. Their (often) millions of parameters make them
hard to understand and interpret, leading to the often referred to black box.
To (partially) solve this issue, many methods in the regime of explainable
artificial intelligence (XAI) have been proposed for real-valued neural
networks. Early works in this regime work with explanations based on gradients
[24, 25, 27] while more recent works make use of more sophisticated methods,
such as local linearization of the model [22, 23] or Shapley values - a
measure for cooperative games from game theory [26].

To the best of our knowledge, no explanation methods have been proposed to
explain complex-valued neural networks. Directly applying real-valued
explanation algorithms to complex-valued architectures is often impossible and
the adaptions are non trivial. In this paper, we aim to fill this gap by
developing a complex-valued variant of the widely used DeepSHAP [20]
algorithm. We provide proofs for the complex-valued chain rule used within the
algorithm as well as for some desirable properties that reduce computational
complexity. Additionally, we adapt four gradient based explanation methods to
the complex domain. We then evaluate explanation quality of all of these
methods on two datasets - one real-valued and one complex-valued dataset.
Additionally, we validate our theoretical findings on these two datasets.

The key contributions of our work are:

  1. 1.

Development of a complex-valued DeepSHAP algorithm, including the proof of a
complex-valued chain rule.

  2. 2.

Adaptation of four gradient based explanation methods to the complex domain.

  3. 3.

Evaluation of explanation quality of these five methods on two complex-valued
neural networks applied to a real-valued and a complex-valued dataset.

  4. 4.

Validation of theoretical results on these two datasets.

The source code with a pytorch library containing all the explanation methods
in section 4 and section 5, as well as all experiments from section 6 is
publicly available111In the supplementary material for reviewers, online after
publication..

##  2 Related work

###  2.1 Explanation methods

Early explanation methods are mostly based on gradients. Simonyan et al. [24]
use gradients as an explanation method. This idea is then extended to Guided
Backpropagation [25], where negative gradients are zeroed out throughout the
backward pass. Sundararajan et al. [27] propose the integrated gradients
methods, where instead of using the plain gradients, they integrate the
gradients over an interpolation from a starting value (e.g. an all zeros
image) to the actual image. Lastly, Bach et al. propose Layerwise Relevance
Propagation [3], which is not a gradient based method per se, but is
equivalent to computing the gradient times the input under certain assumptions
[13].

LIME by Riberiro et al. [22] aims to locally explain a model’s output by
fitting a linear model in a neighborhood of a certain input. This local linear
model is easily explainable and is used to locally explain the more complex
model.

DeepLIFT by Shrikumar et al. [23] introduces an explanation method that only
needs one backward pass to explain a model output for a specific input. They
define heuristic rules how to explain single layers and then define a chain
rule on how to explain a full network as a concatenation of these layers.

SHAP [20] is a framework that defines a class of explanation methods -
additive feature attribution methods - by an additive property, which is
already satisfied by LIME and DeepLIFT. They formulate three axioms based on
Shapley values from game theory and show that their defined SHAP values is the
only explanation method to fulfill all of those axioms. They then derive
kernelSHAP, a combination of Linear LIME and Shapley values as well as
DeepSHAP, which makes use of the chain rule from DeepLIFT but replaces the
heuristic rules with SHAP values for every layer. A complex-valued adaptation
of DeepSHAP is the main focus of this work, so DeepSHAP will be explained in
detail in subsection 3.2. Many variants of these methods have been proposed
and are widely used in the literature [1].

###  2.2 Complex-valued neural networks

After early works by Hirose et al. [11], the main building blocks for complex-
valued convolutional neural networks have been explored by Trabelsi et al.
[29]. More recently there have also been developments of a complex-valued
transformer architecture [31, 9] and many more architectures [17]. In the last
two years, besides numerous applications, there have been multiple theoretical
works on complex-valued neural networks, such as analysis of training behavior
[28] and the ability to generalize [6] as well as hardware optimizations for
complex-valued calculations [33]. A multitude of complex-valued architectures
have been used for applications such as MRI reconstruction [8, 30], PolSAR
image processing [18, 4], speech enhancement [7, 12]. A broad overview over
the field can be found in [5, 17].

To the best of our knowledge, there exist no works on explainability complex-
valued neural networks. Mayer et al. [21] mention that they are not able to
explain their model, since SHAP in its current form is not suitable for
complex-valued neural networks. There are works that use explainability
methods for real-valued neural networks to process complex-valued data [19],
which is a different approach that does not benefit from the success of
complex-valued neural networks and is thus out of scope of this paper.

##  3 Real-valued explaination methods

###  3.1 Gradient based methods

A straightforward method for model explanation is to use the gradient of the
output w.r.t to the input pixels as an explanation as proposed in [24]. This
can be modified to integrated gradients, where the input image is interpolated
from some starting value (e.g. a black/all zeros image) and the gradients of
all interpolation steps are integrated [27]. Additionally, by multiplying the
gradients with the input image an approximation of Layerwise Relevance
Propagation [3] can be computed, as shown by Kindermans et al. [13].

Lastly Guided Backpropagation [25] uses gradients as an explanation but zeros
out negative gradients (i.e. applies ReLU on the gradients) throughout the
backward pass.

###  3.2 DeepSHAP

Lundberg et al. [20] define a class of explanation methods - additive feature
attribution methods - as explanation models that fulfill Equation 1. For some
fixed input $x$, let $f_{x}$ be a (local) explanation model, which is a
simplified version of the original model that acts on a simplified binary
model input defined by a local mapping $h_{x}(z^{\prime})=z$ while assuring
$f_{x}(z^{\prime})\approx f(h_{x}(z^{\prime}))$ if $z^{\prime}\approx
x^{\prime}$. An additive feature attribution method is then defined as a model
fulfilling the equation:

$f_{x}(z^{\prime})=\phi_{0}+\sum_{j=1}^{n}\phi_{j}z^{\prime}_{j}$ (1)

Here $z^{\prime}$ is a binary variable describing the presence or absence of
features of the simplified input. They show that multiple methods from the
literature [22, 3, 23, 26] already fulfill this definition. They then define a
set of desirable axioms for additive feature attribution methods inspired by
Shapley values:

Local Accuracy describes the ability of the explanation model to be accurate
when evaluated at the target data point. For a model $f$ and its explanation
model $f_{x}$ it should hold that: $f(x)=f_{x}(x^{\prime})$.

Missingness describes that the contribution of a feature that is already
missing in the simplified input $x^{\prime}$, should be zero:
$x_{j}=0\Rightarrow\phi_{j}=0$.

Consistency means that if a feature has less impact (or stronger negative
impact) on a second model, then on the original model, the contribution
assigned to that feature for the second model should be smaller then that of
the original model: For a simplified input $z^{\prime}$ let $z^{\prime}/j$ be
$z^{\prime}$ modified s.t. $z^{\prime}_{j}=0$. For two models $f$ and
$f^{\prime}$ and their explanation models $f_{x}$ and $f^{\prime}_{x}$ it
should hold that:

$\displaystyle f^{\prime}_{x}(z^{\prime})-f^{\prime}_{x}(z^{\prime}/j)$
$\displaystyle\leq f_{x}(z^{\prime})-f_{x}(z/j)\text{ for all
}z^{\prime}\in\\{0,1\\}^{n}$ $\displaystyle\Rightarrow\phi_{j}(f^{\prime},x)$
$\displaystyle\leq\phi_{j}(f,x)$ (2)

Lundberg et al. [20] show that a unique function delivers the solution to
these axioms:

$\displaystyle\phi_{j}(f,x)$ $\displaystyle:=\sum_{z^{\prime}\subset
x^{\prime}\setminus j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime})\right)$ (3) $\displaystyle\text{with }G_{Mz^{\prime}}$
$\displaystyle:=\frac{|z^{\prime}|!(M-|z^{\prime}|-1)!}{M!}$

The solution to this equation yields an additive feature attribution model
that the authors call SHapley Additive exPlanations (SHAP). They then define
different model specific and agnostic approaches to compute the SHAP values.

We focus now on an approach designed for deep neural networks: DeepSHAP. Since
the contributions $\phi_{j}(g,x)$ are easy to calculate for simple building
blocks of neural networks, such as linear layers (e.g. convolutions, fully
connected layers or normalization layers) and pointwise activation functions
(e.g. ReLU, sigmoid), we just need to calculate the contribution of a full
model, which is the concatenation of those layers, from the contributions of
the individual layers. For this sake the authors use an idea from DeepLIFT
[23]: Propagating Contributions through layers by a custom defined chain rule.
For that they define multipliers as contributions relative to the input
difference:

$m_{xf}^{j}:=\frac{\phi_{j}(f,x)}{x_{j}-\mathbb{E}(x_{j})}$ (4)

They then define a chain rule for these multipliers to propagate the
contributions through the layers, similarly of how gradients are
backpropagated from individual layer-gradients to calculate the gradient of
the full model. Let $h=f\circ g$ with $g(y)=x\in\mathbb{R}^{n}$, then the
chain rule states:

$\displaystyle m_{yh}^{k}=\sum_{j=1}^{n}m_{yg_{j}}^{k}m_{xf}^{j}$ (5)

Applying this chain rule iteratively through all the layers makes it possible
to compute the multipliers and then by inverting Equation 4 also the
contribution of the full neural network.

##  4 Complex-valued Gradient based methods

In this section, we describe how to adapt commonly used gradient based methods
to the complex domain. Since the adaption is rather straight forward, we do
not consider this a key contribution, but want to summarize the adaptations,
since - to the best of our knowledge - this has not been done before.
Additionally, these methods will provide a baseline for the experiments in
section 6 to compare Deep$\mathbb{C}$SHAP which is introduced in section 5.
The adaption of all methods from subsection 3.1 can be accomplished with the
use of Wirtinger calculus. The Wirtinger derivative for a function
$f:\mathbb{C}\to A$ with $A=\mathbb{C}\text{ or }\mathbb{R}$ are defined as:

$\displaystyle\frac{\delta f}{\delta z}$
$\displaystyle:=\frac{1}{2}\left(\frac{\delta f}{\delta(z)}-i\frac{\delta
f}{\delta\mathcal{I}(z)}\right)$ (6) $\displaystyle\frac{\delta
f}{\delta\overline{z}}$ $\displaystyle:=\frac{1}{2}\left(\frac{\delta
f}{\delta(z)}+i\frac{\delta f}{\delta\mathcal{I}(z)}\right)$ (7)

The Wirtinger derivative w.r.t. the complex conjugate $\frac{\delta
f}{\delta\overline{z}}$ has the property of pointing in the direction of the
strongest ascend and the magnitude of the Wirtinger gradient is equal to the
amount of change in that direction [11], just as the gradient does for real-
valued functions. Thus, replacing the real-valued gradient in the methods from
subsection 3.1 with $\frac{\delta f}{\delta\overline{z}}$ yields explanation
methods with a similar interpretation as the methods in subsection 3.1.
However, the Wirtinger gradient is complex-valued for complex-valued inputs
and is thus not feasible to directly generate a saliency map. The two straight
forward options are to take either the magnitude of the complex gradient, or
add up its real and imaginary part. The first variant has the drawback of
neglecting the sign of the gradient, the second one may show a contribution of
zero for non-zero gradients, if the real and imaginary part cancel out. We
test both variants in section 6.

To adapt the Guided Backpropagation [25] it is necessary to define an
equivalent formulation of "zeroing out negative gradients" in the backward
pass for the Wirtinger gradients. There are two different interpretations of
the mapping: $ReLU(x)=max(0,x)$ in the real domain, which lead to different
adaptions to the complex domain. The first interpretation is to define a
region of unsuitable values (in the real case all negative numbers) and
interpret ReLU as the function setting all unsuitable (i.e. negative) values
to zero. Secondly, ReLU can be viewed as a projection: $\mathbb{R}$ to
$\mathbb{R}^{+}_{0}$. Thus, every value in $\mathbb{R}$ is mapped to the
closest value in $\mathbb{R}^{+}_{0}$, projecting negative values to zero and
positive values to themselves. These interpretations are visualized in Figure
1 a) and c), respectively.

Following the first interpretation we propose to map every input to zero that
is not in the first quadrant of $\mathbb{C}$ (i.e. real and imaginary parts
are positive), thus defining the quadrants 2-4 as the region of unsuitable
values (i.e. the equivalent to negative values in the real case). Following
the second of the two interpretations, we propose to project $\mathbb{C}$ to
the first quadrant of $\mathbb{C}$, which yields the function
$f(x)=max(0,(x))+i\ max(0,\mathcal{I}(x))$. Both variants are visualized in
Figure 1 b) and d), respectively. The resulting formulations for complex-
valued ReLU variants are in line with the ReLU variants for deep complex
networks as proposed by Trabelsi et al. [29], thus we call these variants
Guided $z$-Backpropagation and Guided $\mathbb{C}$-Backpropagation,
respectively.

[width=]./sec/figures/real_kill

(a) ReLU: Set unsuitable region ($x<0$) to zeros.

[width=]./sec/figures/complex_kill

(b) zReLU: Set unsuitable region ($x\in\mathbb{C}_{2,3,4}$) to zeros.

[width=]./sec/figures/real_proj

(c) ReLU: Projection $\mathbb{R}\to\mathbb{R}^{+}_{0}$.

[width=]./sec/figures/complex_proj

(d) $\mathbb{C}$ReLU: Projection $\mathbb{C}\to\mathbb{C}_{1}$

Figure 1: Two different interpretations of ReLU and their corresponding
complex-valued adaptions. $\mathbb{C}_{n}$ denotes the $n$-th quadrant.

##  5 Deep$\mathbb{C}$SHAP

This section marks the main contribution of this work. We aim to make DeepSHAP
applicable to complex-valued neural networks since it is not applicable to
complex-valued neural networks, due to its chain rule based on real-valued
multipliers, as explained in subsection 5.2.

###  5.1 $\mathbb{C}$SHAP values

The first question that arises is how to define SHAP values in the complex-
valued context, which is possible by simply adopting the definition from
Equation 3:

$\displaystyle\phi_{j}(f,x)$ $\displaystyle:=\sum_{z^{\prime}\subset
x^{\prime}\setminus j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime})\right)$ (8) $\displaystyle\text{with }G_{Mz^{\prime}}$
$\displaystyle:=\frac{|z^{\prime}|!(M-|z^{\prime}|-1)!}{M!}$

Additionally, we define $\phi_{0}(f,x):=f_{x}(\emptyset)$ as the output of the
model, if no feature is present. Notably, we have that if $f:\mathbb{C}^{n}\to
A$, with $A=\mathbb{R}\text{ or }\mathbb{C}$, then $\phi\in A$. Additionally,
if $f:\mathbb{C}^{n}\to\mathbb{C}$ we can split the output into its real and
imaginary parts and explain them separately, since it holds that:

$\displaystyle\phi_{j}((f),x)$ $\displaystyle=(\phi_{j}(f,x))$
$\displaystyle\phi_{j}(\mathcal{I}(f),x)$
$\displaystyle=\mathcal{I}(\phi_{j}(f,x))$ (9)

###  5.2 Chain rule for Deep$\mathbb{C}$SHAP

As pointed out before, DeepSHAP is not suitable to be used for complex-valued
neural networks [21]. The reason is that the multiplier defined in Equation 4
is not suitable in complex-valued scenarios since complex-valued neural
networks are usually not holomorphic (i.e. not complex differentiable) due to
real output neurons or non-holomorphic activation functions. The quotient as
defined in Equation 4 does not converge for $(x_{j}-\mathbb{E}(x_{j}))\to 0$
and is thus not suitable for small perturbations.

A similar problem arises in the optimization of complex-valued neural
networks, which is solved by using the Wirtinger derivatives for optimization
instead of the complex derivative. The Wirtinger calculus motivates an
alternative formulation of multipliers and a complex-valued chain rule that
can be applied to complex-valued neural networks.

To define this complex-valued chain rule, we have to first define partial
contributions of the real and imaginary part. This definition is motivated by
the following equation:

$\displaystyle\phi_{j}(f,x)$ $\displaystyle=\sum_{z^{\prime}\subset
x^{\prime}\setminus j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime})\right)$ (10) $\displaystyle=\sum_{z^{\prime}\subset
x^{\prime}\setminus j}G_{Mz^{\prime}}(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime}\cup(j))$
$\displaystyle+f_{x}(z^{\prime}\cup(j))-f_{x}(z^{\prime}))$ (11)
$\displaystyle=\sum_{z^{\prime}\subset x^{\prime}\setminus
j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime}\cup(j))\right)$ $\displaystyle+\sum_{z^{\prime}\subset
x^{\prime}\setminus
j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup(j))-f_{x}(z^{\prime})\right)$ (12)

where $(j)$ means to have only the real part of the $j$-th feature present,
while the imaginary part stays absent. We now define the partial contributions
as the two summands of Equation 12:

$\displaystyle\phi_{(j)}(f,x)$ $\displaystyle=\sum_{z^{\prime}\subset
x^{\prime}\setminus
j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup(j))-f_{x}(z^{\prime})\right)$ (13)
$\displaystyle\phi_{\mathcal{I}(j)}(f,x)$
$\displaystyle=\sum_{z^{\prime}\subset x^{\prime}\setminus
j}G_{Mz^{\prime}}\left(f_{x}(z^{\prime}\cup
j)-f_{x}(z^{\prime}\cup(j))\right)$ (14)

Intuitively, the real partial contributions of a feature can be seen as the
contributions of the real part of $x$, if the imaginary part of that feature
is missing and the imaginary partial contribution as the contribution of the
imaginary part, when the real part is never missing. These quantities
themselves do not hold meaningful information but are important for the
formulation of the chain rule later on since we can use them to define the
partial multipliers:

$\displaystyle
m^{(j)}_{xf}=\frac{\phi_{(j)}(f,x)}{(x_{j}-\mathbb{E}(x_{j}))};\hskip
5.69054ptm^{\mathcal{I}(j)}_{xf}=\frac{\phi_{\mathcal{I}(j)}(f,x)}{\mathcal{I}(x_{j}-\mathbb{E}(x_{j}))}$
(15)

Inspired by Wirtinger calculus, we can now define the actual multipliers used
for the chain rule:

$\displaystyle m^{j}_{xf}=\frac{1}{2}\left(m^{(j)}_{xf}-i\
m^{\mathcal{I}(j)}_{xf}\right)$ (16) $\displaystyle
m^{j}_{\overline{x}f}=\frac{1}{2}\left(m^{(j)}_{xf}+i\
m^{\mathcal{I}(j)}_{xf}\right)$ (17)

Before proceeding to the chain rule, we want to note two properties of the
multipliers that will come in handy later on to reduce the computational load:

$\displaystyle f:\mathbb{C}\to\mathbb{R}$ $\displaystyle\Rightarrow
m^{j}_{\overline{x}f}=\overline{m^{j}_{xf}}$ (18) $\displaystyle f\text{
linear }$ $\displaystyle\Rightarrow m^{j}_{xf}=\frac{\delta f}{\delta
x_{j}}\text{ and }m^{j}_{\overline{x}f}=\frac{\delta
f}{\delta\overline{x_{j}}}$ (19)

where the derivative operators in the second property are meant as Wirtinger
operators. Proofs for these properties as well as for the now following chain
rule can be found in the supplementary material. The complex-valued chain rule
for a concatenation $h=f\circ g$ with $g(y)=x\in\mathbb{C}^{n}$ is:

$\displaystyle m^{k}_{yh}$
$\displaystyle=\sum_{j=1}^{n}(m^{k}_{yg_{j}}m^{j}_{xf}+\overline{m^{k}_{yg_{j}}}m^{j}_{\overline{x}f})$
(20) $\displaystyle m^{k}_{\overline{y}h}$
$\displaystyle=\sum_{j=1}^{n}(m^{k}_{\overline{y}g_{j}}m^{j}_{xf}+\overline{m^{k}_{\overline{y}g_{j}}}m^{j}_{\overline{x}f})$
(21)

Since we are interested in the contributions rather than the multipliers, we
have to reconstruct them after the use of the chain rule:

$\displaystyle\phi_{(k)}(h,x)$
$\displaystyle=(m^{k}_{yh}+m^{k}_{\overline{y}h})(x_{k}-\mathbb{E}(x_{k}))$
(22) $\displaystyle\phi_{\mathcal{I}(k)}(h,x)$
$\displaystyle=(m^{k}_{yh}-m^{k}_{\overline{y}h})i\mathcal{I}(x_{k}-\mathbb{E}(x_{k}))$
(23) $\displaystyle\phi_{k}(h,x)$
$\displaystyle=\phi_{(k)}(h,x)+\phi_{\mathcal{I}(k)}(h,x)$ (24)

We have summarized how to apply the chain rule to calculate the contributions
of a concatenation of functions in Algorithm 1.

Note that by Equation 9 we can always assume the outer function $f$ to be
real-valued since if it is not, we can just calculate the contributions to the
real and imaginary part separately by replacing $f$ with $(f)$ and
$\mathcal{I}(f)$. Thus, we can make use of Equation 18 to simplify the
calculation of the chain rule.

For numerical stability, we adopt an adaptation from DeepSHAP: If the
denominator in the fraction in the multipliers in Equation 15 is close to zero
(in our case smaller than $10^{-6}$), we replace it with the respective
Wirtinger derivative.

Since the for-loop in Algorithm 1 can be vectorized and all operations can be
defined as matrix multiplications, the algorithm can be efficiently computed
on a GPU using deep learning frameworks (in our case: Pytorch). The
contributions can then be computed in a single backward pass within
milliseconds, see appendix for a runtime analysis.

###  5.3 Max$\mathbb{C}$SHAP

To apply the chain rule from subsection 5.2 we need to pre calculate the
(partial) contributions $\phi$ and $\phi_{\mathcal{I}}$ for every layer. [20]
derives SHAP values for real-valued max pooling layers, however their
algorithm is not suitable for complex-valued max pooling. Since the complex
numbers are not ordered, max pooling is commonly done by ordering by the
magnitude of the patch, thus for an image patch
$\mathcal{X}=\\{x_{1},\ldots,x_{n}\\}$, we can define complex max pooling as

$\displaystyle\mathbb{C}\text{maxpool}=\operatorname{argmax}_{x\in\mathcal{X}}|x|$
(25)

In Algorithm 2 we define an algorithm to efficiently compute the contributions
of a max pooling layer. For a presorted input (which can be accomplished in
$O(n\log_{2}n)$) the complexity of the algorithm is $O(n^{2})$.

Algorithm 1

Complex-valued chain rule for contributions $\phi_{k}(h,x)$ of $h=f\circ g$:

$\text{Analytic. calc }\phi_{(k)}(g,y),\phi_{\mathcal{I}(k)}(g,y)$
$\triangleright$ $*$

$m^{(k)}_{yg}\leftarrow\frac{\phi_{(k)}(g,y)}{(y_{k}-\mathbb{E}(y_{k}))}$

$m^{\mathcal{I}(k)}_{yg}\leftarrow\frac{\phi_{\mathcal{I}(k)}(g,y)}{\mathcal{I}(y_{k}-\mathbb{E}(y_{k}))}$

$m^{k}_{yg}\leftarrow\frac{1}{2}\left(m^{(k)}_{yg}-i\
m^{\mathcal{I}(k)}_{yg}\right)$

$m^{k}_{\overline{y}g}\leftarrow\frac{1}{2}\left(m^{(k)}_{yg}+i\
m^{\mathcal{I}(k)}_{yg}\right)$

for $j=0$ to $n$ do

$\text{Analytic. calc }\phi_{(j)}(f,x),\phi_{\mathcal{I}(j)}(f,x)$
$\triangleright$ $*$

$m^{(j)}_{xf}\leftarrow\frac{\phi_{(j)}(f,x)}{(x_{j}-\mathbb{E}(x_{j}))}$

$m^{\mathcal{I}(j)}_{xf}\leftarrow\frac{\phi_{\mathcal{I}(j)}(f,x)}{\mathcal{I}(x_{j}-\mathbb{E}(x_{j}))}$

$m^{j}_{xf}\leftarrow\frac{1}{2}\left(m^{(j)}_{xf}-i\
m^{\mathcal{I}(j)}_{xf}\right)$

$m^{j}_{\overline{x}f}\leftarrow\frac{1}{2}\left(m^{(j)}_{xf}+i\
m^{\mathcal{I}(j)}_{xf}\right)$

end for

$m^{k}_{yh}\leftarrow\sum_{j=1}^{n}(m^{k}_{yg_{j}}m^{j}_{xf}+\overline{m^{k}_{yg_{j}}}m^{j}_{\overline{x}f})$

$m^{k}_{\overline{y}h}\leftarrow\sum_{j=1}^{n}(m^{k}_{\overline{y}g_{j}}m^{j}_{xf}+\overline{m^{k}_{\overline{y}g_{j}}}m^{j}_{\overline{x}f})$

$\phi_{(k)}(h,x)\leftarrow(m^{k}_{yh}+m^{k}_{\overline{y}h})(x_{k}-\mathbb{E}(x_{k}))$

$\phi_{\mathcal{I}(k)}(h,x)\leftarrow(m^{k}_{yh}-m^{k}_{\overline{y}h})\mathcal{I}(x_{k}-\mathbb{E}(x_{k}))$

$\phi_{k}(h,x)\leftarrow\phi_{(k)}(h,x)+\phi_{\mathcal{I}(k)}(h,x)$

$*$ easy and fast for building blocks of CVNNs (e.g. linear layers, 1-to-1
activation etc.)

Algorithm 2

Contributions $\phi_{k}$ of $\mathbb{C}$max pooling:

Precomputed vector $M$ $\triangleright$ *

Input $X=(X_{1},\ldots,X_{n})$

Reference $Y=(Y_{1},\ldots,Y_{n})$

$m\leftarrow\operatorname{argmax}_{p}(|X|,|Y|)$ $\triangleright$ **

$s\leftarrow\operatorname{argmax}(\operatorname{argmin}_{p}(|X|,|Y|))$

for $k=0$ to $n-1$ do

$m_{x}[k]\leftarrow\operatorname{argmin}(X[k],s)$

$m_{y}[k]\leftarrow\operatorname{argmin}(Y[k],s)$

$z_{x}\leftarrow\operatorname{sort}(m_{x},\text{key}=|\cdot|)$

$z_{y}\leftarrow\operatorname{sort}(m_{y},\text{key}=|\cdot|)$

$\phi_{k}\leftarrow\langle z_{x},M\rangle-\langle z_{y},M\rangle$

end for

$*$ constant vector $M$ only depends on $n$ and can be pre computed for every
max pooling layer. It incorporates the weighting factors $G_{Mz^{\prime}}$
from Equation 8.
$**$ $\operatorname{argmax}_{p}$ denotes pointwise argmax

##  6 Evaluation

We evaluate our method Deep$\mathbb{C}$SHAP in two experiments: An experiment
on MNIST [16] as designed by Shrikumar et al. [23] and already presented for
the evaluation of the real-valued DeepSHAP [20]. Additionally, we evaluate
with a self designed experiment on a complex-valued PolSAR dataset [2].
Details on the training procedure for both experiments can be found in the
supplementary material. Since no other explanation methods for complex-valued
neural networks have been proposed, there is no competitive baseline, we could
compare to. Comparing to an explanation algorithm for real-valued neural
networks is not suitable either, since when comparing explanation methods
between different models (e.g. a real-valued and a complex-valued to be able
to apply the respective explanation method) we would not know, if a difference
in our experiments has to be attributed to the difference in models, or the
difference in explanation methods. Thus we treat the adaptations of gradient
based methods in section 4 as baseline and compare our proposed
Deep$\mathbb{C}$SHAP against those.

Additional to evaluating explanation quality, we also test our theoretical
results from section 5 by verifying core properties of Deep$\mathbb{C}$SHAP
computationally.

Figure 3: Qualitative results for MNIST experiment. Explanation scores for
output 8 and 3 are shown and the image after masking 20% of the most important
pixels for both predictions. All images are normalized in itself,
normalization over all methods is not suitable due to different orders of
magnitude.

###  6.1 MNIST experiment

We use a small complex-valued CNN with three blocks of $\mathbb{C}$
convolutions, $\mathbb{C}$Relu activations and $\mathbb{C}$ max pooling
followed by a 2 layer MLP as a classifier on the MNIST dataset. The model
obtains a 88% accuracy on the digit classification task on the test set of
MNIST. We adopt the experimental setup as performed by [23, 20]: For a preset
source class we randomly select an image from the test set and use the
explanation methods to explain the source class output and the output for a
fixed target class. We then identify up to 20% of the most important pixels
for classifying the class correctly into the source class rather then into the
target class by choosing the pixels with the highest explanation scores and
remove these pixels by setting them to zero. We then input the modified image
into the model and evaluate the change in log-odds compared to the initial
input. A high score thus indicates the ability of the explanation algorithm to
identify the most important features for a classification into a specific
class. Qualitative results are visualized in Figure 3. All explanations are
very blurry except Deep$\mathbb{C}$SHAP and $|grad*input|$, which are the only
methods whose explanations focus on the actual content of the image. This
indicates that these methods assign contribution to the most important pixels.
This is also visible in the quantitative results in Figure 4. There it shows
that Deep$\mathbb{C}$SHAP performs best on this task. This result is in line
with the results in [20], where a similar result was shown for a real-valued
neural network. While none of the gradient based methods is on par with
Deep$\mathbb{C}$SHAP, gradient times input and integrated gradients in the
variant with the sum of real and imaginary parts show similarly good results.
It is evident from these results, that the absolute value of the gradient
leads to a worse performing explanation method, than the sum of real and
imaginary part. The former performes very poorly for all methods except
Guided$\mathbb{C}$Backpropagation. As for the two variants for Guided
Backpropagation, the $\mathbb{C}$-ReLU version performs meaningfully better
then the $z$-ReLU variant.

Figure 4: Quantitative results on MNIST experiment. Change in log-odds after
masking 20% of the most relevant features for classes 8 and 3 according to
different explanation algorithms. The median is shown in red.

###  6.2 PolSAR experiment

For this experiments we trained a complex-valued version of the ResNet18 [10]
with an Adam optimizer [14] on the S1SLC$\\_$CVDL [2] datasets. It consists of
276571 complex-valued PolSAR image patches of size $100\\!\times\\!100$ from
three north American city areas. They are labeled with seven classes:
Agriculture, Forest, High-Density Urban, High-Rise Urban, Low-Density Urban,
Industrial and Water. Per image patch two complex-valued channels, HH and HV
polarization, are available. As can be seen in Figure 5, these images are very
noisy and thus a visual explanation is not feasible.

To still be able to evaluate explanation quality on this dataset, we designed
an experiment on this task: We take two patches and input the four channels
into the complex-valued neural network. As output we expect the classes of
both patches, making this a multiclass classification problem. The trained
model obtains an accuracy of 93% on this task (detailed results in the supp.
material).

Figure 5: Example image from the PolSAR S1SLC$\\_$CVDL dataset. Magnitude and
Phase information of the complex-valued two channel (HH and HV) images are
shown.

If we explain the output log-odds for the one of the two correct labels, we
know that it should be explained by the two corresponding input channels and
the other two input channels should have no or a negative influence on the
prediction. We then measure to which extent this is fulfilled for a specific
image by this measure: Let $X^{1},X^{2},Y^{1},Y^{2}$ be the four input
channels from two patches from different classes. Let $\phi_{k}(f,Z)$ be the
explanation by an explanation method of pixel $k\in\\{1,...,n\\}$ in channel
$Z$. We then calculate:

$\frac{\displaystyle\sum_{l=1}^{2}\sum_{k=0}^{n}\phi_{k}(f,X^{l})}{\displaystyle\sum_{l=1}^{2}\sum_{k=0}^{n}\phi_{k}(f,X^{l})+\sum_{l=1}^{2}\sum_{k=0}^{n}\phi_{k}(f,Y^{l})}$
(26)

For strictly positive explanations, this measure is bounded at 1 and would
signal that all important features are in the correct channels. For possibly
negative values of signed explanation method (i.e. Deep$\mathbb{C}$SHAP and
the signed gradient variants) a value greater than 1 indicates that the
incorrect channels contributed negatively to the final decision, which is
desirable (since these channels show an image of a different class). Hence a
score $\geq 1$ is desirable.

Figure 6 shows that Deep$\mathbb{C}$SHAP performs best on this task. All the
gradient based methods perform relatively poorly. The results from the MNIST
experiment are mostly confirmed, as the $\mathcal{R}+\mathcal{I}$ variants of
gradient times input and integrated gradients perform best. Overall, the
absolute value with median scores around 0.5, indicating that they marked the
features of the incorrect channels to be equally important as the features of
the correct channels. All variants of the Guided Backpropagation perform
poorly, while the simple gradient explanation, gradient multiplied by input
and integrated gradient show median scores around 1 when the real and
imaginary parts are added.

###  6.3 Testing SHAP axioms

Figure 6: Quantitative results for the PolSAR experiment. Fraction of
calculated contributions from the correct channels versus all channels are
visualized for all explanation methods introduced in this paper. For gradient
based methods $|\cdot|$ denotes the absolute value of the respecting method,
R+I denotes adding up real and imaginary part.

To verify our theoretical results from section 5 we test Missingness and Local
Accuracy as two of the axioms of the SHAP framework.

To test Missingness, we report the percentage of missing inputs that are
assigned a value other than $0$ by Deep$\mathbb{C}$SHAP. To test Local
Accuracy, we report the average of the following equation over all outputs and
the whole test set. Let $X$ be an input image and $f$ the prediction model:

$\left|\left(\sum_{j=1}^{n}\phi_{j}(f,X)+\phi_{0}\right)-f(X)\right|$ (27)

As Table 1 shows, Missingness is always fulfilled. The (absolute) Local
Accuracy Error of $9.8\times 10^{-6}$ and $3.5\times 10^{-3}$ (resulting in a
relative error of $2.3\times 10^{-7}$ and $2.7\times 10^{-5}$) on MNIST and
S1SLC$\\_$CVDL respectively can be explained by the adaptations made for
numeric stability.

|  MNIST |  S1SLC$\\_$CVDL
---|---|---
Local Accuracy Error | $9.8\times 10^{-6}$ | $3.5\times 10^{-3}$
Average $|$output$|$ | 39.21 | 129.27
Missingness Error | 0.0% | 0.0%

Table 1: Validation of Local Accuracy and Missingness in Deep$\mathbb{C}$SHAP.
Local Accuracy error denotes the value from Equation 27. Average absolute
value of the model output is reported to make the results for Local Accuracy
comparable. Missingness error denotes the fraction of missing features that
are assigned a non-zero contribution.

.

##  7 Conclusion

In this paper we have introduced multiple explanation algorithms for complex-
valued neural networks by adapting four gradient based explanation methods to
the complex domain and by developing a complex-valued version of the well
known DeepSHAP [20] explanation method, which we call Deep$\mathbb{C}$SHAP.
The major theoretical result to obtain the latter is the complex-valued chain
rule for contributions as introduced in section 5. By providing these methods,
we fill the gap of missing explanation methods for complex-valued neural
networks, which are rising in popularity for applications with real- and
complex-valued data types.

We analyze explanation quality on the real-valued MNIST dataset and the real
world complex-valued PolSAR S1SLC$\\_$CVDL dataset. We show that our method
Deep$\mathbb{C}$SHAP outperforms the gradient based methods on both tasks.

Additionally, we have validated our theoretical results on these datasets. We
show that the SHAP properties Local Accuracy and Missingness are fulfilled.
This and the theoretical guarantees given by the theoretical foundation of
Shapley values make this method suited to explain complex-valued neural
networks.

As future work, we aim to adopt more explanation methods to the complex
domain, such as LIME and kernelSHAP. Additionally, applying
Deep$\mathbb{C}$SHAP on different modalities will be interesting, especially
modalities that are easily visually interpretable, such as MRI images to make
use of the generated head maps, rather than quantitative results on the hard-
to-understand PolSAR images.

## References

  * [1] Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018)
  * [2] Asiyabi, R.M., Datcu, M., Anghel, A., Nies, H.: Complex-valued end-to-end deep network with coherency preservation for complex-valued SAR data reconstruction and classification. IEEE Transactions on Geoscience and Remote Sensing (2023)
  * [3] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015)
  * [4] Barrachina, J.A., Ren, C., Morisseau, C., Vieillard, G., Ovarlez, J.P.: Impact of PolSAR pre-processing and balancing methods on complex-valued neural networks segmentation tasks. IEEE Open Journal of Signal Processing 4, 157–166 (2023)
  * [5] Bassey, J., Qian, L., Li, X.: A survey of complex-valued neural networks. arXiv:2101.12249 (2021)
  * [6] Chen, H., He, F., Lei, S., Tao, D.: Spectral complexity-scaled generalisation bound of complex-valued neural networks. Artificial Intelligence 322, 103951 (2023)
  * [7] Choi, H.S., Kim, J.H., Huh, J., Kim, A., Ha, J.W., Lee, K.: Phase-aware speech enhancement with deep complex U-Net. In: ICLR (2018)
  * [8] Cole, E., Cheng, J., Pauly, J., Vasanawala, S.: Analysis of deep complex-valued convolutional neural networks for MRI reconstruction and phase-focused applications. Magnetic Resonance in Medicine 86(2), 1093–1109 (2021)
  * [9] Eilers, F., Jiang, X.: Building blocks for a complex-valued transformer architecture. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2023)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [11] Hirose, A.: Complex-Valued Neural Networks: Theories and Applications. World Scientific (2003)
  * [12] Hu, Y., Liu, Y., Lv, S., Xing, M., Zhang, S., Fu, Y., Wu, J., Zhang, B., Xie, L.: DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement. In: INTERSPEECH (2020)
  * [13] Kindermans, P.J., Schütt, K., Müller, K.R., Dähne, S.: Investigating the influence of noise and distractors on the interpretation of neural networks. arXiv:1611.07270 (2016)
  * [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations ICLR (2015)
  * [15] Ko, M., Panchal, U.K., Andrade-Loarca, H., Mendez-Vazquez, A.: Coshnet: A hybird complex valued neural network using shearlets. arXiv:2208.06882 (2022)
  * [16] LeCun, Y.: The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998)
  * [17] Lee, C., Hasegawa, H., Gao, S.: Complex-valued neural networks: A comprehensive survey. IEEE/CAA Journal of Automatica Sinica 9(8), 1406–1426 (2022)
  * [18] Li, X., Sun, Q., Li, L., Liu, X., Liu, H., Jiao, L., Liu, F.: SSCV-GANs: Semi-supervised complex-valued GANs for PolSAR image classification. IEEE Access 8, 146560–146576 (2020)
  * [19] Lin, Q.H., Niu, Y.W., Sui, J., Zhao, W.D., Zhuo, C., Calhoun, V.D.: SSPNet: An interpretable 3D-CNN for classification of schizophrenia using phase maps of resting-state complex-valued fMRI data. Medical Image Analysis 79, 102430 (2022)
  * [20] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems 30 (2017)
  * [21] Mayer, K.S., Müller, C., Soares, J.A., de Castro, F.C.C., Arantes, D.S.: Deep phase-transmittance RBF neural network for beamforming with multiple users. IEEE Wireless Communications Letters 11(7), 1498–1502 (2022)
  * [22] Ribeiro, M.T., Singh, S., Guestrin, C.: "Why should I trust you?" explaining the predictions of any classifier. In: 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1135–1144 (2016)
  * [23] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning. pp. 3145–3153. PMLR (2017)
  * [24] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. In: 2nd International Conference on Learning Representations (ICLR) (2014)
  * [25] Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.A.: Striving for simplicity: The all convolutional net. In: 3rd International Conference on Learning Representations (ICLR) (2015)
  * [26] Štrumbelj, E., Kononenko, I.: Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems 41, 647–665 (2014)
  * [27] Sundararajan, M., Taly, A., Yan, Q.: Gradients of counterfactuals. arXiv:1611.02639 (2016)
  * [28] Tan, Z.H., Xie, Y., Jiang, Y., Zhou, Z.H.: Real-valued backpropagation is unsuitable for complex-valued neural networks. Advances in Neural Information Processing Systems 35, 34052–34063 (2022)
  * [29] Trabelsi, C., Bilaniuk, O., Zhang, Y., Serdyuk, D., Subramanian, S., Santos, J.F., Mehri, S., Rostamzadeh, N., Bengio, Y., Pal, C.J.: Deep complex networks. In: 6th International Conference on Learning Representations (ICLR) (2018)
  * [30] Vasudeva, B., Deora, P., Bhattacharya, S., Pradhan, P.M.: Compressed sensing MRI reconstruction with Co-VeGAN: Complex-valued generative adversarial network. In: WACV. pp. 1779–1788 (2022)
  * [31] Yang, M., Ma, M.Q., Li, D., Tsai, Y.H.H., Salakhutdinov, R.: Complex transformer: A framework for modeling complex-valued sequence. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4232–4236 (2020)
  * [32] Yin, D., Luo, C., Xiong, Z., Zeng, W.: Phasen: A phase-and-harmonics-aware speech enhancement network. In: AAAI Conference on Artificial Intelligence. vol. 34, pp. 9458–9465 (2020)
  * [33] Zhang, H., Gu, M., Jiang, X., Thompson, J., Cai, H., Paesani, S., Santagati, R., Laing, A., Zhang, Y., Yung, M., et al.: An optical neural chip for implementing complex-valued neural network. Nature Communications 12(1), 457 (2021)

Supplementary Material

##  8 Proof of properties of complex-valued multipliers

We want to prove the following properties for complex-valued multipliers as
mentioned in the main paper in Equations 18 \- 19:

$\displaystyle f:\mathbb{C}\to\mathbb{R}$ $\displaystyle\Rightarrow
m^{j}_{\overline{x}f}=\overline{m^{j}_{xf}}$ (28) $\displaystyle f\text{
linear }$ $\displaystyle\Rightarrow m^{j}_{xf}=\frac{\delta f}{\delta
x_{j}}\text{ and }m^{j}_{\overline{x}f}=\frac{\delta
f}{\delta\overline{x_{j}}}$ (29)

Equation 28 follows from the fact that if $f:\mathbb{C}\to\mathbb{R}$, then
$\phi_{j}((f),x)$ and $\phi_{j}(\mathcal{I}(f),x)$ are real-valued and then
also $m^{(j)}_{xf}$ and $m^{\mathcal{I}(j)}_{xf}$. Thus it holds that:

$\displaystyle m^{j}_{\overline{x}f}$
$\displaystyle=\frac{1}{2}\left(m^{(j)}_{xf}+i\
m^{\mathcal{I}(j)}_{xf}\right)$ (30)
$\displaystyle=\overline{\frac{1}{2}\left(m^{(j)}_{xf}-i\
m^{\mathcal{I}(j)}_{xf}\right)}$ (31) $\displaystyle=\overline{m^{j}_{xf}}$
(32)

To prove Equation 29 let $f(x):=Ax$ with $A\in\mathbb{C}^{1\times n}$ (for
multidimensional output, we analyze each output separately and thus every row
of $A$ separately), then with Equation 13 / 14 and $\Delta
x:=x-\mathbb{E}(x)$, we get:

$\displaystyle\phi_{(j)}(f,x)$ $\displaystyle=A_{j}(\Delta x_{j})\text{ and }$
(33) $\displaystyle\phi_{\mathcal{I}(j)}(f,x)$
$\displaystyle=iA_{j}\mathcal{I}(\Delta x_{j})$ (34)

and thus:

$\displaystyle m^{(j)}_{xf}$ $\displaystyle=\frac{A_{j}(\Delta x_{j})}{(\Delta
x_{j})}=A_{j}\text{ and }$ (35) $\displaystyle m^{\mathcal{I}(j)}_{xf}$
$\displaystyle=\frac{iA_{j}\mathcal{I}(\Delta x_{j})}{\mathcal{I}(\Delta
x_{j})}=iA_{j}$ (36)

It then follows:

$\displaystyle m^{j}_{xf}$
$\displaystyle=\frac{1}{2}(A_{j}-i^{2}A_{j})=A_{j}=\frac{\delta f}{\delta
x_{j}}$ (37) $\displaystyle m^{j}_{\overline{x}f}$
$\displaystyle=\frac{1}{2}(A_{j}+i^{2}A_{j})=0=\frac{\delta
f}{\delta\overline{x_{j}}}$ (38)

$\Box$

##  9 Proof of chain rule

To prove the chain rule for complex-valued multipliers (Equations 20 and 21 in
the main paper), we have to show:

$\displaystyle\sum_{k\leq m}\phi_{k}(h,y)=h(y)-\phi_{0}$ (39)

when applying the chain rule for complex-valued multipliers. For an easier
notation let $\Delta z=z-\mathbb{E}(z)$. Then it holds:

$\displaystyle\sum_{k\leq m}\phi_{k}(h,y)$ (40) $\displaystyle=$
$\displaystyle\sum_{k\leq m}(\phi_{(k)}(h,y)+\phi_{\mathcal{I}(k)}(h,y))$ (41)
$\displaystyle=$ $\displaystyle\sum_{k\leq m}m^{(k)}_{yh}(\Delta
y_{k})+m^{\mathcal{I}(k)}_{yh}\mathcal{I}(\Delta y_{k})$ (42) $\displaystyle=$
$\displaystyle\sum_{k\leq m}m^{(k)}_{yh}(\Delta
y_{k})+m^{\mathcal{I}(k)}_{yh}\mathcal{I}(\Delta y_{k})$ (43) $\displaystyle=$
$\displaystyle\sum_{k\leq m}(m^{k}_{\overline{y}h}+m^{k}_{yh})(\Delta
y_{k})+(m^{k}_{yh}-m^{k}_{\overline{y}h})i\ \mathcal{I}(\Delta y_{k})$ (44)
$\displaystyle=$ $\displaystyle\sum_{k\leq m}\sum_{j\leq
n}((m^{k}_{\overline{y}g_{j}}m^{j}_{xf}+\overline{m^{k}_{\overline{y}g_{j}}}m^{j}_{\overline{x}f})$
(45)
$\displaystyle+(m^{k}_{yg_{j}}m^{j}_{xf}+\overline{m^{k}_{yg_{j}}}m^{j}_{\overline{x}f}))(\Delta
y_{k})$ (46)
$\displaystyle+((m^{k}_{yg_{j}}m^{j}_{xf}+\overline{m^{k}_{yg_{j}}}m^{j}_{\overline{x}f})$
(47)
$\displaystyle-(m^{k}_{\overline{y}g_{j}}m^{j}_{xf}+\overline{m^{k}_{\overline{y}g_{j}}}m^{j}_{\overline{x}f}))i\
\mathcal{I}(\Delta y_{k})$ (48) $\displaystyle=$ $\displaystyle\sum_{k\leq
m}\sum_{j\leq
n}((m^{(k)}_{yg_{j}}m^{j}_{xf}+\overline{m}^{(k)}_{yg_{j}}m^{j}_{\overline{x}f})(\Delta
y_{k})$ (49)
$\displaystyle+(m^{\mathcal{I}(k)}_{yg_{j}}m^{j}_{xf}+\overline{m}^{\mathcal{I}(k)}_{yg_{j}}m^{j}_{\overline{x}f})(-i)i\
\mathcal{I}(\Delta y_{k}))$ (50) $\displaystyle=$ $\displaystyle\sum_{j\leq
n}(m^{j}_{xf}(\Delta x_{j})+m^{j}_{\overline{x}f}\overline{(\Delta x_{j})})$
(51) $\displaystyle=$ $\displaystyle\sum_{j\leq
n}((m^{j}_{xf}+m^{j}_{\overline{x}f})(\Delta
x_{j})+(m^{j}_{xf}-m^{j}_{\overline{x}f})\mathcal{I}(\Delta x_{j}))$ (52)
$\displaystyle=$ $\displaystyle\sum_{j\leq
n}\phi_{(j)}(f,x)+\phi_{\mathcal{I}(j)}(f,x)$ (53) $\displaystyle=$
$\displaystyle\sum_{j\leq n}\phi_{j}(f,x)$ (54) $\displaystyle=$
$\displaystyle f(x)-\phi_{0}$ (55) $\displaystyle=$ $\displaystyle
h(y)-\phi_{0}$ (56)

$\Box$

| Agriculture | Forest | High Density | Buildings | Low Density | Industrial | Water
---|---|---|---|---|---|---|---
|  |  | Urban |  | Urban |  |
Precision | 0.999 | 0.999 | 0.999 | 0.997 | 0.999 | 0.999 | 0.999
Recall | 0.941 | 0.946 | 0.680 | 0.722 | 0.650 | 0.863 | 0.999

Table 2: Per class precision and recall of the complex-valued ResNet18 trained
on the S1SLC$\\_$CVDL dataset.

.

Figure 7: $\mathbb{C}$Convolutional model designed for the MNIST dataset task.

##  10 Details on training and results – PolSAR

The experimental setup is explained in detail subsection 6.2. We used a
complex-valued version of the ResNet18 [10] with four input channels and 7
outputs classes, where the ground truth label has one or two classes, making
this a multiclass classification problem. We trained the model with an Adam
[14] optimizer in pytorch. The model adaptation to the complex domain was
made, as suggested by Trabelsi et al. [29]. The training was performed with
the following hyperparameters:

  * •

Learning rate: 0.001

  * •

Epochs: 20

  * •

Loss: binary cross entropy

  * •

Adam betas: 0.9, 0.999

The test accuracy of the trained model is 0.93, precision and recall for the
individual classes can be seen in Table 2.

##  11 Details on training and results – MNIST

We use a small complex-valued CNN with three blocks of $\mathbb{C}$
convolutions, $\mathbb{C}$Relu activations and $\mathbb{C}$ max pooling
followed by a 2 layer MLP as a classifier on the MNIST dataset as can be seen
in Figure 7. We train with an Adam [14] optimizer in pytorch. The model was
build from building blocks as suggested by Trabelsi et al. [29] with max
pooling layers as described in subsection 5.3. The training was performed with
the following hyperparameters:

  * •

Learning rate: 0.001

  * •

Epochs: 20

  * •

Loss: cross entropy

  * •

Adam betas: 0.9, 0.999

The test accuracy of the trained model is 0.88.

##  12 Runtime analysis of explanation methods

|  Deep$\mathbb{C}$SHAP | Grad | Grad*Input |  $\mathbb{C}$-Backprop | z-Backprop | int. Grad
---|---|---|---|---|---|---
MNIST | 132 | 5.0/4.7 | 4.7/4.7 | 5.3/5.3 | 5.2/5.4 | 22.6/22.6
S1SLC$\\_$CVDL | 380 | 57.2/57.8 | 59.4/59.7 | 64.1/63.8 | 63.3/63.3 | 302/302
Table 3: Runtime of different explanation methods on both datasets in
milliseconds. For gradient based methods (all but Deep$\mathbb{C}$SHAP) the
first number shows the result for the version when using the absolute value,
the second number when using $+\mathcal{I}$.

For both experiments from section 6 we performed a runtime analysis for all
explaination methods presented in section 5 and section 4. Results can be seen
in Table 3. As expected, all the gradient based methods are very fast. Just
the integrated gradient method is significantly slower than the other methods,
which is expected, since for the the integration the gradient has to be
calculated for multiple inputs. We use 5 integrands which corresponds with the
slow down factor of $\sim 5$. As can be observed, Deep$\mathbb{C}$SHAP is the
slowest of all the explanation methods. It is however still relatively fast,
being able to explain $\sim 8$ images per second on MNIST and $\sim 3$ images
per second on the S1SLC$\\_$CVDL PolSAR dataset.


157 2021 ACML 2021

# Pyramid Correlation based Deep Hough Voting
for Visual Object Tracking

Ying Wang<EMAIL_ADDRESS>
Tingfa Xu∗<EMAIL_ADDRESS>
Jianan Li∗<EMAIL_ADDRESS>
Shenwang Jiang<EMAIL_ADDRESS>
Junjie Chen<EMAIL_ADDRESS>
Beijing Institute of Technology  Beijing  China ∗Tingfa Xu and Jianan Li are
the corresponding authors.

###### Abstract

Most of the existing Siamese-based trackers treat tracking problem as a
parallel task of classification and regression. However, some studies show
that the sibling head structure could lead to suboptimal solutions during the
network training. Through experiments we find that, without regression, the
performance could be equally promising as long as we delicately design the
network to suit the training objective. We introduce a novel voting-based
classification-only tracking algorithm named Pyramid Correlation based Deep
Hough Voting (short for PCDHV), to jointly locate the top-left and bottom-
right corners of the target. Specifically we innovatively construct a Pyramid
Correlation module to equip the embedded feature with fine-grained local
structures and global spatial contexts; The elaborately designed Deep Hough
Voting module further take over, integrating long-range dependencies of pixels
to perceive corners; In addition, the prevalent discretization gap is simply
yet effectively alleviated by increasing the spatial resolution of the feature
maps while exploiting channel-space relationships. The algorithm is general,
robust and simple. We demonstrate the effectiveness of the module through a
series of ablation experiments. Without bells and whistles, our tracker
achieves better or comparable performance to the SOTA algorithms on three
challenging benchmarks (TrackingNet, GOT-10k and LaSOT) while running at a
real-time speed of 80 FPS. Codes and models will be released.

###### keywords:

Visual object tracking $\cdot$ Siamese-based tracking algorithm $\cdot$
Anchor-free $\cdot$ Key point based $\cdot$ Pixel-wise correlation $\cdot$
Hough voting

††editors: Vineeth N Balasubramanian and Ivor Tsang

##  1 Introduction

Visual object tracking is a fundamental task in computer vision, whose purpose
is to predict the position of a target in subsequent frames given its precise
state in the initial frame. It has been widely used in applications such as
surveillance, robotics, autonomous driving [Gupta et al.(2017)Gupta, Kumar,
Behera, and Subramanian, Li et al.(2013)Li, Hu, Shen, Zhang, Dick, and Hengel,
Kiani Galoogahi et al.(2017)Kiani Galoogahi, Fagg, and Lucey]. Despite
remarkable progress made in recent decades, challenges such as large
occlusion, severe deformation and similar object interference still stand to
be overcome [Wu et al.(2015)Wu, Lim, and Yang, Zhang et al.(2014)Zhang, Zhang,
and Yang].

Figure 1:  Structure comparison of (a) SiamRPN++, (b) SiamFC++ and (c) our
approach. Symbols in red, orange and green represent the classification
branch, the regression branch and the final predicted target box,
respectively. Contrary to previous algorithms, we discard the sibling heads
model and turn to locate the target sorely by the classification branch,
separating corner points from background through a voting mechanism.

Siamese-based algorithms contribute significantly to the field of visual
tracking. It treats the tracking task as a target matching problem by learning
the general similarity map between the target template and the search region.
Recent years have witnessed a greater tendency for many Siamese-based
algorithms [Li et al.(2018)Li, Yan, Wu, Zhu, and Hu, Li et al.(2019)Li, Wu,
Wang, Zhang, Xing, and Yan, Danelljan et al.(2019)Danelljan, Bhat, Khan, and
Felsberg, Voigtlaender et al.(2020)Voigtlaender, Luiten, Torr, and Leibe,
Zhang et al.(2020)Zhang, Peng, Fu, Li, and Hu, Yan et al.(2021)Yan, Peng, Fu,
Wang, and Lu] to treat tracking problem as a combination of a parallel
classification and regression. To be specific, anchor-based algorithms SiamRPN
[Li et al.(2018)Li, Yan, Wu, Zhu, and Hu], SiamRPN++ [Li et al.(2019)Li, Wu,
Wang, Zhang, Xing, and Yan], DaSiam [Zhu et al.(2018)Zhu, Wang, Li, Wu, Yan,
and Hu] introduce the region proposal extraction subnetwork (RPN) [Ren et
al.(2017)Ren, He, Girshick, and Sun] and use a classification branch for
proposal selection, further regressing the four offsets between the anchor and
the corresponding groundtruth (as illustrate in Fig.1 (a)). While anchor-free
tracking algorithms SiamFC++ [Xu et al.(2020)Xu, Wang, Li, Yuan, and Yu],
SiamCAR [Guo et al.(2020)Guo, Wang, Cui, Wang, and Chen] classifies all
positions within the object bounding box as positive with the help of
centerness branch and regresses the four distances between the center-point
and the the object boundaries (as illustrate in Fig.1 (b)). Although these
approaches obtain balanced accuracy and speed, it has been proposed that the
tracking performance is prone to fall into suboptimal due to the essential
misalignment of the two branches and that substantially the classification
branches contribute more [Song et al.(2020)Song, Liu, and Wang, Cheng et
al.(2018)Cheng, Wei, Shi, Feris, Xiong, and Huang, Yan et al.(2021)Yan, Peng,
Fu, Wang, and Lu]. Therefore, in our work, we design a voting-based
classification-only tracking algorithm (short for PCDHV), solely using a
classification branch to generate corner voting map, jointly locating the top-
left and bottom-right corners of the target (as illustrate in Fig.1 (c)). The
whole structure consists of three modules: Siamese-based feature extraction,
Pyramid Correlation and Deep Hough Voting.

Concretely, the traditional Siamese structure is adopted first for feature
extraction of target template and the search region, whose outputs are then
embedded through cross-correlation to learn the similarity. Most of the
existing Siamese-based algorithms employ naïve-correlation [Bertinetto et
al.(2016)Bertinetto, Valmadre, Henriques, Vedaldi, and Torr, Li et
al.(2018)Li, Yan, Wu, Zhu, and Hu] and depth-wise correlation [Li et
al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan, Xu et al.(2020)Xu, Wang, Li,
Yuan, and Yu], taking the entire template feature as a convolution kernel.
However, in the process of tracking, the target may suffer large appearance
variance, thus matching a fixed kernel feature as a whole with the vastly
changing search features may drastically degrade the quality of correlation
map. Considering the corner locating objective which requires the correlation
feature rich in fine-grained local structure, we argue that the pixel-wise
correlation is more suitable. Innovatively, we propose Pyramid Correlation to
extract corner-favoriable fusion feature through a series of blocks: spatial
feature selection, pyramid feature pooling and group pixel-level correlation.
These blocks simultaneously equip the correlation feature with global spatial
contexts, making the tracker robust to deformation.

Besides, corner locating also requires long-range dependencies of pixels. To
this end, we elaborately design a Deep Hough Voting module to further take
over, which consists of three blocks of vote generation, vote refinement and
vote aggregation, precisely produce feature maps representing the location
probabilities of the top-left and bottom-right corners. The vote generation
block first applies several convolutional layers to adjust feature into
appropriate shape, expanding the receptive field at the same time. Then vote
refinement block, namely a modified position-aware non-local block, mines the
dense contextual information and pairing two corner feature more closely. Grid
channel and Pixel Shuffle mechanism [Shi et al.(2016)Shi, Caballero, Huszár,
Totz, Aitken, Bishop, Rueckert, and Wang] are deployed to obtain location
information and expand feature size respectively, further exploiting the
channel-spatial relationships. Finally, the voting module proposed in HoughNet
[Samet et al.(2020)Samet, Hicsonmez, and Akbas] takes responsibility of the
voting aggregation block to capture long-range dependencies. The three blocks
work together to produce high-quality corner heatmaps, with the peak position
of each map representing the location of the predicted corner point.

Inevitably, the discretization gap caused by the stride of the network has a
dramatic negative impact on the tracking accuracy. While many superior
algorithms resort to introducing regression branch to bridge this gap, we
instead choose to compensate by inserting up-sampling operations in the
structure. As a result, our PCDHV can localize the bounding box in a precise
way, as can be seen intuitively from the experimental results below.

We evaluate our PCDHV on three challenging large-scale benchmarks, including
GOT-10k [Huang et al.(2019)Huang, Zhao, and Huang], TrackingNet [Mueller et
al.(2018)Mueller, Bibi, Giancola, Alsubaihi, and Ghanem] and LaSOT [Fan et
al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling]. Without
bells and whistles, our tracker can achieve better or comparable performance
to the state-of-the-art trackers. The network is general, robust yet simple,
requires no tedious parameter adjustment and heuristic knowledge. Ablation
studies are conducted to verify the effectiveness of each component. Our main
contributions are summarized as follows.

  * •

We formulate object tracking problem as a classification-only problem, aiming
to distinguish the top-left and bottom-right corners of the target from the
searching area. The algorithm is general, robust yet simple, achieving better
or comparable results to the SOTA algorithms on several mainstream benchmarks.
The tracking speed is also impressive with 80 FPS.

  * •

We design the Pyramid Correlation to equip the correlation feature with fine-
grained local structures and global spatial contexts, providing rich
information for subsequent processes.

  * •

We preform Deep Hough Voting on the correlation feature, further capturing the
channel-spatial relationships and long range dependencies, enabling the peak
position of each feature map to accurately represent the target position.

##  2 Related work

#### Siamese-based algorithm

Recently, Siamese-based trackers have attracted great attention from the
visual tracking community due to their satisfactory balance between
performance and efficiency. SiamFC [Bertinetto et al.(2016)Bertinetto,
Valmadre, Henriques, Vedaldi, and Torr] first learns the similarity map
between the target template and the search region through a cross-correlation
operation. SiamRPN [Li et al.(2018)Li, Yan, Wu, Zhu, and Hu], SiamRPN++ [Li et
al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan], DaSiam [Zhu et al.(2018)Zhu,
Wang, Li, Wu, Yan, and Hu]introduce the region proposal extraction subnetwork
(RPN) [Ren et al.(2017)Ren, He, Girshick, and Sun] into the Siamese structure,
using a classification branch for foreground-background estimation and a
regression branch for anchor adjustment. Although these anchor-based
algorithms achieve state-of-the-art results on many challenging benchmarks,
the pre-defined anchor settings introduce many hyperparameters and
computational complexity. Anchor-free tracking algorithms are further raised
for better performance. SiamFC++ [Xu et al.(2020)Xu, Wang, Li, Yuan, and Yu],
SiamCAR [Guo et al.(2020)Guo, Wang, Cui, Wang, and Chen] predict the
probability of a point being the target center by classification firstly, and
then regress the distances between the center-point and the the object
boundaries. The above mentioned algorithms all embed target feature and search
feature through cross-correlation, while SiamGAT [Guo et al.(2021)Guo, Shao,
Cui, Wang, Zhang, and Shen] choose to establish part-to-part correspondence
between the two Siamese branches feature through graph attention mechanism.

#### Misalignment of the sibling head

As is obvious, most of the existing siamese-based tracking algorithms
incorporate regression branch, but we hold a different view. It turns out
through experiments that features in salient areas may have rich
classification information, while features around the boundary are more
suitable for bounding box regression [Song et al.(2020)Song, Liu, and Wang].
When the shared features extracted from the Siamese network are applied to
both classification and localization branches, the performance is prone to
fall into suboptimal due to the essential misalignment of the two branches
[Cheng et al.(2018)Cheng, Wei, Shi, Feris, Xiong, and Huang]. To solve this
problem, [Song et al.(2020)Song, Liu, and Wang] proposes a simple TSD operator
to deal with the tangled tasks conflict through a task-aware proposal
estimation and a detection head. [Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu]
trains the network by a two-stage approach through decoupling the
classification branch from the regression branch. We seek an alternative way
to explore a higher performance classifier, using purely classification branch
to separate corner points from all pixels. The performance could be equally
promising attributed to the sophisticated design of the network.

##  3 Deep hough voting for visual tracking

###  3.1 Overall Architecture

Fig. 2 shows the overall structure of our algorithm solely composed of a
classification branch. We adopt Siamese structure with parameter-shared
backbone network for feature extraction to generate template feature and
search feature. An extra CNN-Upsample layer with unshared parameters is used
for feature adjustment. Then, the Pyramid Correlation module, with blocks of
spatial feature selection, pyramid feature pooling and group pixel-level
correlation, is applied to obtain corner-favorable correlation feature.
Finally, the Deep Hough Voting module, with blocks of vote generation, vote
refinement and vote aggregation, is applied for accurate corner estimation.
Two heatmaps respectively corresponding to the top-left corner and bottom-
right corner are output for evaluation. Without extra tricks, the peak
position of each map is assumed to be the location of the predicted corner
point.

Figure 2: Architecture of our PCDHV tracking framework with three fundamental
components: Feature Extraction, Pyramid Correlation and Deep Hough Voting. The
peak positions of the two output heatmaps respectively represent the locations
of the predicted top-left and bottom-right corners.

###  3.2 Pyramid correlation

Fig. 3 depicts the flowchart of our pyramid correlation, which composes of
three steps of spatial feature selection, pyramid feature pooling, and group
pixel-level correlation. To facilitate notation, we denote
$\bm{F}_{T}\in\mathbb{R}^{h\times w\times C}$ as the template feature map and
$\bm{F}_{S}\in\mathbb{R}^{H\times W\times C}$ as the search feature.

#### Spatial feature selection.

For precise corner localization, we first highlight the parts of the template
that are beneficial to locating corners while suppressing the less useful
parts through a spatial attention mechanism. Concretely, we apply two $1\times
1$ convolution, sequentially followed by activation of ReLU and Sigmoid, upon
$\bm{F}_{T}$ to produce a channel-wise spatial attention map, which is used to
adjust the importance of the template feature both in a pixel-wise and
channel-wise manner. As such, the resulting attentive template feature,
denoted as $\bm{F}_{T}^{A}$, is more prominent and contributes to generating
informative correlation maps favoring the identification of target corner
points.

#### Pyramid feature pooling.

Given $\bm{F}_{T}^{A}\in\mathbb{R}^{h\times w\times C}$, this step aims to
construct a set of pixel-level template features comprised of both fine-
grained local structures and global spatial contexts. To this end, we first
perform pyramid pooling on $\bm{F}_{T}^{A}$, implemented by a group of max-
pooling with various odd kernel sizes, leading to a set of pyramid features
with decreasing spatial resolutions but enlarging respective fields. We then
decompose these pyramid feature maps in spatial dimension, resulting in a pool
of totally $M$ pixel-level feature vectors $\bm{K_{i}}\in\mathbb{R}^{1\times
1\times C},i\in(0,M-1)$. In our implementation, we perform max-pooling with
odd kernel sizes of $3\times 3$, $5\times 5$, and global max-pooling with
kernel size of $6\times 6$ on $\bm{F}_{T}^{A}\in\mathbb{R}^{6\times 6\times
256}$, the outputs of which, together with $\bm{F}_{T}^{A}$, are then
decompose into $57=6^{2}+4^{2}+2^{2}+1$ pixel-level feature vectors for
further use.

Figure 3: The flowchart of Pyramid Correlation. The template feature is first
processed by spatial feature selection, pyramid feature pooling and then
correlate with the search feature in a manner of group pixel-level
correlation.

#### Group pixel-level correlation.

This step is designed to perform correlation between the set of pixel-level
template features $\bm{K_{i}}\in\mathbb{R}^{1\times 1\times C},i\in(0,M-1)$
and the search feature $\bm{F}_{S}\in\mathbb{R}^{H\times W\times C}$.
Nevertheless, naive correlation operation produces only one-channel response
maps, which causes severe compression or even lost of information. To
alleviate this, we propose a new group pixel-level correlation by first
splitting both the template and search features into $N$ channel groups (we
set $N=8$ here), forming $\bm{K_{i}^{N}}\in\mathbb{R}^{1\times 1\times C/N}$
and $\bm{F}_{S}^{N}\in\mathbb{R}^{H\times W\times C/N}$. Naive correlation is
then performed corresponding in a group-wise manner, producing a $N$-channel
correlation map $\bm{G_{i}}\in\mathbb{R}^{H\times W\times N}$. Finally, we
concatenate the correlation maps produced by all of the pixel-level template
features in the channel dimension, achieving the final correlation results of
$MN$ channels, which can be denotes as $\bm{G}\in\mathbb{R}^{H\times W\times
MN}$. Feature scale changes are clearly shown in Fig. 3.

###  3.3 Deep Hough Voting

The top-left and bottom-right corners can be far away from the target body
thus hard to regress directly in a local manner. We propose to locate target
corners accurately by integrating near and long-range evidences through Hough
voting. Specifically, the occurrence probability of a target’s corner at a
given point is determined by the sum of votes received from surroundings. As
with the vote-field with $R$ regions designed in HoughNet [Samet et
al.(2020)Samet, Hicsonmez, and Akbas], votes from both near and far distances
can be collected simultaneously. Fig. 2 shows the diagram of our deep hough
voting module comprised of three blocks that progressively implement vote
generation, vote refinement and vote aggregation, as detailed below.

#### Vote generation.

This step aims to generate $H_{S}\times W_{S}\times 2R$ voting map
$\bm{F}_{V}$ from the $H\times W\times MN$ correlation map, where $H_{S}$ and
$W_{S}$ are spatial dimensions, $R$ is the number of regions in the vote
filed, and $2$ is the number of corners to be voted for. We use two
convolutional layers with $1\times 1$ filters for channel reduction, and
another three unpadded convolutional layers with $3\times 3$ filters to
enlarge respective field. Therefore, the spatial resolution of the feature is
reduced from $H\times W$ to $H_{S}\times W_{S}$.

Figure 4: Vote refinement: a modified position-aware non-local block. The grid
channel and Pixel Shuffle mechanism are deployed respectively to obtain
location information and expand feature size.

#### Vote refinement.

The layers for vote generation are only local operations and may overlook the
dependency among votes generated by positions across long distance. Motivated
by this, this step refines the votes at all positions in a collaborative
manner by capturing their long-range dependencies. Considering the relations
of votes of two positions highly depend on their relative locations, two extra
channels are added to $\bm{F}_{V}$: the normalized $x$ and $y$ coordinates of
each spatial position, leading to a $H_{S}\times W_{S}\times(2R+2)$ sized
voting map $\bm{F}_{V}^{{}^{\prime}}$. We compute interactions between any two
positions, and refine the votes at one position as a weighted sum of the votes
at all positions, thus obtaining a refined voting map $\bm{F}_{R}$:

$\bm{F}_{R}=ReLU\\{\mathcal{P}[z(\theta({\bm{F}_{V}^{{}^{\prime}})^{T}}\phi(\bm{F}_{V}^{{}^{\prime}})g(\bm{F}_{V}^{{}^{\prime}}))]+\mathcal{U}[\bm{F}_{V}]\\},$
(1)

where $\theta$, $\phi$, $g$ and $z$ represent linear embedding implemented by
$1\times 1$ convolution. We set the number of output channels of $z$ to be
four times the number of channels in $\bm{F}_{V}$, and exploit inherent
channel-spatial relationships to achieve two-fold spatial expansion of voting
map by converting the augmented channel information into space via
PixelShuffle operation [Shi et al.(2016)Shi, Caballero, Huszár, Totz, Aitken,
Bishop, Rueckert, and Wang], as denoted by
$\mathcal{P}\left\\{\cdot\right\\}$. Accordingly, the residual connection is
up-sampled by bi-linear interpolation $\mathcal{U}\left\\{\cdot\right\\}$. The
summation of the two up-sampled features, activated by ReLU, is taken as
output refined voting map, denoted as $\bm{F}_{R}\in\mathbb{R}^{2H_{S}\times
2W_{S}\times 2R}$.

Figure 5: Vote aggregation: the vote module proposed by HoughNet. The figure
shows an example of the voting process of position $(i,j)$ in top-left corner
_presence maps_ , where the vote module collects vote from surrounding pixels
through a deconvolution operation.

#### Vote aggregation.

This step converts the refined voting map to $2H_{S}\times 2W_{S}\times 1$
sized _presence maps_ for the top-left and bottom-right corner, respectively.
Peaks in these maps will indicate the position of target corners. We take the
vote module proposed by HoughNet [Samet et al.(2020)Samet, Hicsonmez, and
Akbas] as our vote aggregation block, where the set of votes from surrounding
pixels can be efficiently aggregated through a deconvolution operation with a
fixed-weight (unlearnable) log-polar _vote field_ as kernel. Fig. 5 detailedly
illustrates the voting process of position $(i,j)$ in top-left corner
_presence maps_ , taking $\bm{F}_{R0}\in\mathbb{R}^{2H_{S}\times 2W_{S}\times
R}$ (the first R channels of $\bm{F}_{R}$) as input. To be specific, we
slightly adjust the _vote field’s_ number of ring radii and angle bins in
HoughNet [Samet et al.(2020)Samet, Hicsonmez, and Akbas] to form
$\bm{K}_{D}\in\mathbb{R}^{H_{0}\times W_{0}\times R}$ and $R=9$ here. The
texture regions in different channels (in _vote field_ , where the channel
distribution is not explicitly presented) represent the weights of the same
shaped color regions in corresponding channels (in _vote map_), from which the
target point $(i,j)$ get votes. And the sum of all weighted votes corresponds
to the final vote value at point $(i,j)$ on the top-left corner _presence
map_. $\bm{F}_{R1}\in\mathbb{R}^{2H_{S}\times 2W_{S}\times R}$ ((the rest R
channels of $\bm{F}_{R}$)) will be derived by the same operation to get
bottom-right corner _presence maps_.

###  3.4 Spatial resolution enhancement

We use GoogLeNet [Szegedy et al.(2014)Szegedy, Liu, Jia, Sermanet, and
Rabinovich] pretrained on ImageNet [Krizhevsky et al.(2012)Krizhevsky,
Sutskever, and Hinton] as backbone whose stride equals 8. The large stride,
which results in quantification error when mutually mapping the feature maps
to original sequence, may severely hurt subsequent localization accuracy of
corners. Instead of introducing regression branch to bridge this
discretization gap, we use a simple yet effective compensation solution by
inserting two upsampling operations in the subsequent modules to improve
feature resolution, one of which is added right after the backbone feature
extraction, and the other is fused into the deep hough voting module as
mentioned above. Finally the total stride is reduced to 2, achieving a balance
between accuracy and efficient training.

###  3.5 Optimization

The groundtruth map $Y\in\mathbb{R}^{2H_{S}\times 2W_{S}\times 2}$ is
constructed under the instruction of CornerNet [Law and Deng(2018)]. We first
map the corner point coordinates $c(x,y)$ from the search map to the feature
map through $\lfloor\frac{c(x,y)-\Delta_{o}}{s}\rfloor$ , where $s$ and
$\Delta_{o}$ respectively denote total stride and total offset of the entire
un-padded network, and $\lfloor\cdot\rfloor$ denotes the floor function. We
set the positive area of groundtruth map be a 2D Gaussian kernel of R radius
placed at the center of the corner points for efficient training, where R is
determined by the at least d IOU between a pair of corners and the target
groundtruth (we set $d=0.5$ here). While the rest of the area is considered to
be negative. As with many keypoint-based algorithms, we adopt Focal Loss [Lin
et al.(2017)Lin, Goyal, Girshick, He, and Dollar] as the training objective.

##  4 Experiments

To extensively evaluate the proposed method, we compare our PCDHV with several
state-of-the-art trackers on three large-scale datasets of TrackingNet,
GOT-10k and LaSOT.

###  4.1 Implementation Details

#### Training.

The network is trained on three currently prevalent large scale video datasets
of GOT-10k [Huang et al.(2019)Huang, Zhao, and Huang], LaSOT [Fan et
al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling] and
TrackingNet [Mueller et al.(2018)Mueller, Bibi, Giancola, Alsubaihi, and
Ghanem] in form of template-search image pair, the deviation between which is
restrict within an interval of less than 100 frames. The input size of
template image is $127\times 127$ pixels, while the search image is $303\times
303$ pixels to capture more corner-related information. We adopt GoogLeNet
[Szegedy et al.(2014)Szegedy, Liu, Jia, Sermanet, and Rabinovich] pre-trained
on ImageNet [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] as
backbone with a total stride of 8. Due to the presence of two up sampling
operations in the structure, the total stride is reduced to 2, enabling the
output feature resolution reach to 88 pixels, which alleviates the impact of
discrete sampling while smooths the training process. As for the training
process, convolutional layers in addition to backbone are initialized using a
zero-centered Gaussian distribution with a standard deviation of 0.01. The
network is set to trained 40 epochs in total, each of which contains 150k
image pairs. The first 5 epochs act as warm up with learning rate linearly
increased from $10^{-6}$ to $8\times 10^{-3}$, while the rest using a cosine
annealing learning rate schedule. Backbone parameters are unfrozen part by
part and eventually all parameters can be updated jointly. We apply stochastic
gradient descent (SGD) with a momentum of 0.9 and a weight decay of $10^{-4}$
for optimization. The whole training process is performed on four NVIDIA RTX
2080Ti GPUs.

#### Testing.

In the inference phase, the network generates two score maps corresponding to
the existence probability of the top-left corner and the bottom-right corner,
respectively. Without bells and whistles, the exact position with the highest
score of each is chosen as the position of corner, independent of any form of
penalty strategy. As for the update, a linear interpolation operation is
applied to target size to smooth out the variation of the bounding box. Our
tracker achieves a competitive speed of over 80 FPS.

###  4.2 Evaluation on TrackingNet, GOT-10k and LaSOT Datasets

#### Results on TrackingNet.

TrackingNet [Mueller et al.(2018)Mueller, Bibi, Giancola, Alsubaihi, and
Ghanem] contains 30,000 sequences with 14 million dense annotations and a test
set of 511 sequences. It covers different object classes and scenes and
requires the tracker to have both discriminative and generative capabilities.
Precision, normalized precision, and AUC scores are used to evaluate trackers.
The results on TrackingNet are shown in Table 1. It can be seen that we
outperform our competitor CGACD [Du et al.(2020)Du, Liu, Zhao, and Tang], a
corner-based tracking algorithm as well, by a large margin of 4.5%. Moreover,
compared with the state-of-the-art Siamese based tracking algorithms of
SiamAttn [Yu et al.(2020)Yu, Xiong, Huang, and Scott] and SiamGAT[Guo et
al.(2021)Guo, Shao, Cui, Wang, Zhang, and Shen], we performs the best in both
$AUC$ and $P$, but regrettably, less well in $P_{norm}$. We argue the reason
is that $P_{norm}$ appreciates models friendly to targets with large scale
variations. Since our work pursues solutions with high simplicity and
generality, we directly apply vanilla backbones for feature extraction without
any additional tailored components, despite not ranking first in every metric,
but achieving the ideal trade-off between accuracy and efficiency. The leading
results on such a large dataset also illustrate the well generalization
ability of our tracking algorithm.

Table 1: Performance comparisons on TrackingNet test set. The top three
results are highlighted in red, green and blue, respectively.

SiamFC SiamRPN++ ATOM DiMP SiamFC++ D3S CGACD KYS SiamAttn* SiamGAT* PCDHV AUC
57.1 73.3 70.3 74.0 75.4 72.8 71.1 74.0 74.3 75.3 76.0 Pnorm 66.3 80.0 77.1
80.1 80.0 76.8 80.0 80.0 81.0 80.7 80.4 P 53.3 69.4 64.8 68.7 70.5 66.4 69.3
68.8 70.5 69.8 72.1

  * •

* The performances of SiamAttn and SiamGAT are the results of our reproduction on the TrackingNet test set using the GOT-10k Python toolkit, same as PCDHV.

#### Results on GOT-10k.

GOT-10k [Huang et al.(2019)Huang, Zhao, and Huang] is a challenging large-
scale dataset which contains 10,000 videos in train subset and 180 videos in
both val and test subset, all of which are moving objects in real-world. Since
there is no class intersection between its train and test subsets, which
contributes to its difficulty, the tracking result can reflect the
generalization ability of the algorithm to unseen object classes. Average
Overlap ($AO$) and Success Rate ($SR$) are adopted as performance metrics,
with higher values resulting in better performance. We follow the protocol of
GOT-10k and train our model with only its training subset. The results on
GOT-10k are shown in Table 2. It can be seen that our PCDHV, though slightly
inferior in $AO$ and $SR50$, can perform the best in $SR75$. We argue that
trackers with individual mechanism of distinguish foreground from background
may enjoy outstanding performance on recalls ($AO$ and $SR50$). While our
straightforward PCDHV prefers to well respect the edge (corner) information
and thus enjoys powerful localization capability, leading to higher $SR75$. We
further test the speed of several preeminent tracking methods using GOT-10k
test set on a single RTX-2080Ti GPU. The last line in Table 2 shows our PCDHV
clearly stands out regarding accuracy and speed trade-off.

Table 2: Performance comparisons on GOT-10k test set. The top three results
are highlighted in red, green and blue, respectively.

SiamFC ECO SiamRPN++ ATOM SiamCAR Ocean-offline SiamFC++ D3S SiamAttn* SiamGAT
PCDHV AO 34.8 31.6 51.7 55.6 56.9 59.2 59.5 59.7 59.3 62.7 60.9 $SR_{50}$ 35.3
30.9 61.6 63.4 67.0 69.5 69.5 67.6 70.0 74.3 71.3 $SR_{75}$ 9.8 11.1 32.5 40.2
41.5 47.3 47.9 46.2 45.6 48.8 50.2 $FPS$ - - 33 25 - - 65 - 19 49 80

  * •

* The performance of SiamAttn is the result of our reproduction on the GOT-10k test set using the GOT-10k Python toolkit, same as PCDHV.

Figure 6: Precision and success plots on LaSOT.

#### Results on LaSOT.

LaSOT [Fan et al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling]
a large-scale, high-quality, and densely annotated dataset for long-term
tracking, with 1,400 sequences under Protocol I while 280 under Protocol II.
Each of these sequences is long, with an average of 2,500 frames, facilitating
the detection of the tracker’s long-term performance. Success and Precision
are used for evaluation. For fair comparison, we follow Protocol II under
which trackers are trained on LaSOT train subset and evaluated on LaSOT test
subset. The results on LaSOT are shown in Fig.6. Compared with SOTA anchor-
based tracking algorithm SiamRPN++[Li et al.(2019)Li, Wu, Wang, Zhang, Xing,
and Yan], our tracker improves the AUC score by 4.4 points, which is
comparable to the results of the SOTA anchor-free tracking algorithm SiamFC++
[Xu et al.(2020)Xu, Wang, Li, Yuan, and Yu].

#### Analysis and Discussion.

It is worth noting that the performance of our algorithm at higher IoU
thresholds is outstanding, which can be seen from metrics such as $SR_{75}$ in
GOT-10k (Table 2) and the latter half of the success curves in LaSOT (Fig.6).
We gather more results of $SR$ on GOT-10k val set, see Fig.8. The figure shows
our PCDHV achieves significant higher rates at large IoU thresholds, which
clearly proves its superiority in precisely locating targets. We argue that
this impressive result mainly brought by the robust and informative features
and its increased resolution, as can be illustrated by the ablation
experiments. We also perform attribute analysis of our PCDHV and several
competing counterparts on LaSOT test set (with the $AUC$ of PCDHV reported
following every attribute name), as shown in Fig.8. Our PCDHV performs better
than other prior arts on most attributes. While in the case of fast motion,
full occlusion and background clustering, all trackers behave relatively
poorly. We assume them as common challenges faced by offline tracking
algorithms without global search or re-detection mechanism.

Figure 7: SR across IoU thresholds
on GOT-10k val set.

Figure 8: Attribute analysis on LaSOT test set.

###  4.3 Ablation Studies

We perform a component-wise analysis on the GOT-10k benchmark [Huang et
al.(2019)Huang, Zhao, and Huang] which can only be evaluated using an online
server, enhancing the fairness and credibility of the test results.
Experiments are conducted through gradually adding each component to the
baseline model to independently verify the effectiveness of our Pyramid
Correlation and Deep Hough Voting.

#### Pyramid correlation.

The baseline model (line 1 in Table 3) here is obtained by degenerating the
pyramid correlation into a depth-wise correlation [Li et al.(2019)Li, Wu,
Wang, Zhang, Xing, and Yan] while preserving other structures. As listed in
Table 3, the tracker gained 1.3%, 1.2%, 0.6%, and 1.4% improvement in $AO$
with the addition operations of pixel-wise correlation, pyramid pooling,
attention mechanism and grouping operation, respectively. Each block is thus
proved to have contribution on the final result through increases the fine-
grained local details and the global spatial context on features.

Table 3: Effectiveness of each component of Pyramid correlation.

Pixel-Wise Pyramid Attention Group AO $\rm SR_{75}$ Baseline 56.7 46.7 ✓ 58.0
48.1 ✓ ✓ 59.2 48.7 ✓ ✓ ✓ 59.5 50.0 ✓ ✓ ✓ ✓ 60.9 50.2

#### Deep Hough Voting.

As the vote aggregation block essentially has a quantitative requirement of
$2R$ on the channel of the input feature, we degrade the vote generation block
into a rough $1\times 1$ convolutional layer merely responsible for channel
reduction, combine which with the vote aggregation block as the baseline
model. It can be seen from line 2 in Table 4 that, despite the feature size
shrinkage brought by the un-padded $3\times 3$ convolutional layers, our vote
generation block can still work well with vote aggregation module and achieves
a performance lift of 0.7% on $AO$, which indicates that the gain in
perceptual field increment is greater than the gain in feature size increment.
An notable improvement achieved through the incorporation of position-aware
non-local block in vote refinement block, as shown in line 4, which can be
contributed by both operations of grid fusing (3.2% gain on $AO$, as shown in
line 3) and up sampling (2.6% gain on $AO$, comparing line 3 with line 4).

Table 4: Effectiveness of each component of Deep Hough Voting.

Generation Refinemen Aggregation AO $\rm SR_{75}$ Baseline depth-wise ✓ 54.4
43.1 ✓ ✓ 55.1 45.5 ✓ ✓(-up)* ✓ 58.3 48.1 ✓ ✓ ✓ 60.9 50.2 * _✓(-up) denotes
remove upsample from refinement block_

###  4.4 Qualitative Results

Figure 9: Qualitative results compare with Ocean and SiamFC++ on three
challenging sequences in GOT-10k val set.

Fig. 9 qualitatively shows our tracking results compared with state-of-the-art
trackers Ocean [Zhang et al.(2020)Zhang, Peng, Fu, Li, and Hu] and SiamFC++
[Xu et al.(2020)Xu, Wang, Li, Yuan, and Yu] on three challenging sequences in
GOT-10k val set [Huang et al.(2019)Huang, Zhao, and Huang]. It is challenging
for offline trackers to accurately assess the exact position of the target
when it undergoes large deformation (first row), occlusion (second row) and
out-of-plane (third row) changes. Our tracker performs well than the trendy
and advanced anchor-free algorithm, while maintaining a very accurate target
position estimation and impressive tracking speed.

##  5 Conclusion

In this paper, we introduce a novel vote-based classification-only tracking
algorithm PCDHV, locating the target by accurately estimating their top-left
and bottom-right corner. We innovatively proposal a Pyramid Correlation to
equip the correlation feature with fine-grained local structure and global
spatial context. Then, the Deep Hough Voting takes over and further captures
the channel-spatial relationships and long ranger dependencies, realizing that
the maximum value of the output feature can accurately correspond to the
target location. The discretization gap between feature maps and original
tracking sequence is mitigated by two well-designed up sampling mechanisms.
Extensive experiments show that PCDHV achieves better or comparable results
than SOTA algorithm on several mainstream datasets. The tracking speed is also
impressive with 80 FPS. We will try to further explore the adaptability of
deeper feature extraction networks to our algorithm with an expectation of
better performance improvement.

This work was supported by the Key Laboratory Foundation under Grant
TCGZ2020C004, 202020429036, 3040012222101 and 3040036722103.

## References

  * [Bertinetto et al.(2016)Bertinetto, Valmadre, Henriques, Vedaldi, and Torr] Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, and Philip H S Torr.  Fully-convolutional siamese networks for object tracking.  In _ECCV_ , 2016.
  * [Cheng et al.(2018)Cheng, Wei, Shi, Feris, Xiong, and Huang] Bowen Cheng, Yunchao Wei, Honghui Shi, Rogerio Feris, Jinjun Xiong, and Thomas Huang.  Revisiting rcnn: On awakening the classification power of faster rcnn.  In _ECCV_ , 2018.
  * [Danelljan et al.(2019)Danelljan, Bhat, Khan, and Felsberg] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  Atom: Accurate tracking by overlap maximization.  In _CVPR_ , 2019.
  * [Du et al.(2020)Du, Liu, Zhao, and Tang] Fei Du, Peng Liu, Wei Zhao, and Xianglong Tang.  Correlation-guided attention for corner detection based visual tracking.  In _CVPR_ , 2020.
  * [Fan et al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In _CVPR_ , 2019.
  * [Guo et al.(2020)Guo, Wang, Cui, Wang, and Chen] Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen.  Siamcar: Siamese fully convolutional classification and regression for visual tracking.  In _CVPR_ , 2020.
  * [Guo et al.(2021)Guo, Shao, Cui, Wang, Zhang, and Shen] Dongyan Guo, Yanyan Shao, Ying Cui, Zhenhua Wang, Liyan Zhang, and Chunhua Shen.  Graph attention tracking.  In _CVPR_ , 2021.
  * [Gupta et al.(2017)Gupta, Kumar, Behera, and Subramanian] Meenakshi Gupta, Swagat Kumar, Laxmidhar Behera, and Venkatesh K. Subramanian.  A novel vision-based tracking algorithm for a human-following mobile robot.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 47(7):1415–1427, 2017.
  * [Huang et al.(2019)Huang, Zhao, and Huang] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  Got-10k: A large high-diversity benchmark for generic object tracking in the wild.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [Kiani Galoogahi et al.(2017)Kiani Galoogahi, Fagg, and Lucey] Hamed Kiani Galoogahi, Ashton Fagg, and Simon Lucey.  Learning background-aware correlation filters for visual tracking.  In _ICCV_ , 2017.
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25:1097–1105, 2012.
  * [Law and Deng(2018)] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In _ECCV_ , 2018.
  * [Li et al.(2018)Li, Yan, Wu, Zhu, and Hu] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In _CVPR_ , 2018.
  * [Li et al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In _CVPR_ , 2019.
  * [Li et al.(2013)Li, Hu, Shen, Zhang, Dick, and Hengel] Xi Li, Weiming Hu, Chunhua Shen, Zhongfei Zhang, Anthony Dick, and Anton Van Den Hengel.  A survey of appearance models in visual object tracking.  _ACM transactions on Intelligent Systems and Technology (TIST)_ , 4(4):1–48, 2013.
  * [Lin et al.(2017)Lin, Goyal, Girshick, He, and Dollar] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar.  Focal loss for dense object detection.  In _ICCV_ , 2017.
  * [Mueller et al.(2018)Mueller, Bibi, Giancola, Alsubaihi, and Ghanem] Matthias Mueller, A. Bibi, S. Giancola, S. Alsubaihi, and B. Ghanem.  Trackingnet: A large-scale dataset and benchmark for object tracking in the wild.  In _ECCV_ , 2018.
  * [Ren et al.(2017)Ren, He, Girshick, and Sun] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(6):1137–1149, 2017.
  * [Samet et al.(2020)Samet, Hicsonmez, and Akbas] Nermin Samet, Samet Hicsonmez, and Emre Akbas.  Houghnet: Integrating near and long-range evidence for bottom-up object detection.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _ECCV_ , 2020.
  * [Shi et al.(2016)Shi, Caballero, Huszár, Totz, Aitken, Bishop, Rueckert, and Wang] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _CVPR_ , 2016.
  * [Song et al.(2020)Song, Liu, and Wang] Guanglu Song, Yu Liu, and Xiaogang Wang.  Revisiting the sibling head in object detector.  In _CVPR_ , 2020.
  * [Szegedy et al.(2014)Szegedy, Liu, Jia, Sermanet, and Rabinovich] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, and Andrew Rabinovich.  Going deeper with convolutions.  _IEEE Computer Society_ , 2014.
  * [Voigtlaender et al.(2020)Voigtlaender, Luiten, Torr, and Leibe] Paul Voigtlaender, Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe.  Siam r-cnn: Visual tracking by re-detection.  In _CVPR_ , 2020.
  * [Wu et al.(2015)Wu, Lim, and Yang] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang.  Object tracking benchmark.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 37(9):1834–1848, 2015.
  * [Xu et al.(2020)Xu, Wang, Li, Yuan, and Yu] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu.  Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines.  In _AAAI_ , 2020.
  * [Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking, 2021.
  * [Yu et al.(2020)Yu, Xiong, Huang, and Scott] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R. Scott.  Deformable siamese attention networks for visual object tracking.  In _CVPR_ , 2020.
  * [Zhang et al.(2014)Zhang, Zhang, and Yang] Kaihua Zhang, Lei Zhang, and Ming-Hsuan Yang.  Fast compressive tracking.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(10):2002–2015, 2014.
  * [Zhang et al.(2020)Zhang, Peng, Fu, Li, and Hu] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu.  Ocean: Object-aware anchor-free tracking.  In _ECCV_ , 2020.
  * [Zhu et al.(2018)Zhu, Wang, Li, Wu, Yan, and Hu] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu.  Distractor-aware siamese networks for visual object tracking.  In _ECCV_ , 2018.