Top quark properties at ATLAS

Dilip Jana (_on behalf of the ATLAS Collaboration_)

_University of Oklahoma, Norman, OK. 73019, USA_

###### Abstract

The ATLAS potential for the study of the top quark properties and physics
beyond the Standard Model in the top quark sector is described. The
measurements of the top quark charge, the spin and spin correlations, the
Standard Model decay ($t\rightarrow bW$), rare top quark decays associated to
flavour changing neutral currents ($t\rightarrow qX$ with $X$ = gluon, $Z$,
photon) and $t\bar{t}$ resonances are discussed. The sensitivity of the ATLAS
experiment is estimated for an expected luminosity of $1$ fb-1 at the LHC,
using full simulation Monte Carlo samples. For the Standard Model measurements
the expected precision is presented. For the tests of physics beyond the
Standard Model, the 5$\sigma$ discovery potential (in the presence of a
signal) and the 95% Confidence Level (C.L.) limit (in the absence of a signal)
are given.

##  I Introduction

The top quark was discovered in 1995 at Fermilab in pair production mode
($t\bar{t}$ events) through strong interactions[1]. Several properties of the
top quark such as the mass, charge, lifetime, production cross-section and
rare decay through flavour changing neutral currents (FCNC) have been explored
in Fermilab, but most of these studies are limited by low statistics. Due to
the high event rates at LHC (one $t\bar{t}$ event per second at a luminosity
of $10^{33}$ cm-2s-1) at ATLAS, these top quark properties can be studied
extensively at ATLAS giving the possibility to discover physics beyond the
Standard Model. Due to the very short life time, the top quark decays before
it has time to hadronise. But its spin properties are not washed out by
hadronization, rather the top quark spin information propagates to its decay
products. This unique feature allows direct top quark spin studies. The top
quark spin can be reconstructed by measuring the angular distributions of its
decay products in the top quark rest frame. Measurements of $W$-boson
polarization complement top quark spin studies which can disentangle the
origin of new physics. In the Standard Model, flavour changing neutral
currents (FCNC) are strongly suppressed at the tree level due to the Glashow-
Iliopoulos-Maiani mechanism. At the one loop level, small FCNC contributions
are expected due to the CKM mixing matrix. The existence of $q\bar{q}$ bound
states (mesons) of all other quarks encourages us to look for $t\bar{t}$ bound
states. New resonances and gauge bosons strongly coupled to the top quark are
expected in several theoretical models which can decay into $t\bar{t}$ pairs,
leading to deviations from Standard Model $t\bar{t}$ production cross-section
and top quark kinematics[2]. These new particles can reveal themselves in the
$t\bar{t}$ invariant mass distribution.

##  II Basic Event Selection

We have used semileptonic ($t\bar{t}\rightarrow WWb\bar{b}\rightarrow l\nu
j_{1}j_{2}b\bar{b}$ with $l=e,\mu$) and dileptonic ($t\bar{t}\rightarrow
WWb\bar{b}\rightarrow l\nu l^{{}^{\prime}}\nu^{{}^{\prime}}b\bar{b}$ with
$l,l^{{}^{\prime}}=e,\mu$) decays of $t\bar{t}$ events for the top quark
charge reconstruction. We have used only the semileptonic decay channel for
$Wtb$ anomalous coupling, $t\bar{t}$ spin and spin correlation and $t\bar{t}$
resonance. For the semileptonic topology, we require exactly one isolated
electron (muon) with $|\eta|<2.5$ and $p_{\mathrm{T}}>25{\mathrm{\
Ge\kern-1.00006ptV}}$ ($p_{\mathrm{T}}>20{\mathrm{\ Ge\kern-1.00006ptV}}$), at
least 4 jets with $|\eta|<2.5$ and $p_{\mathrm{T}}>30{\mathrm{\
Ge\kern-1.00006ptV}}$, at least 2 jets tagged as $b$-jets and missing
transverse energy above 20 GeV[3]. For the dileptonic topology, we require
exactly two isolated electrons (muons) with $|\eta|<2.5$ and
$p_{\mathrm{T}}>25{\mathrm{\ Ge\kern-1.00006ptV}}$
($p_{\mathrm{T}}>20{\mathrm{\ Ge\kern-1.00006ptV}}$), at least 2 jets with
$|\eta|<2.5$ and $p_{\mathrm{T}}>30{\mathrm{\ Ge\kern-1.00006ptV}}$, at least
2 jets tagged as $b$-jets and missing transverse energy above 20 GeV[3]. Since
the final state topology for the rare top quark decays via FCNC are different
from semileptonic and dileptonic topologies, we have used different selection
criteria which will be described in section 2.3.

###  II.1 Top quark charge measurement

We have presented the measurement of the top quark charge based on the
reconstruction of the charge of the top quark decay products. The $W$ boson
charge can be directly measured easily using its leptonic decay modes. Due to
quark confinement inside hadrons, we cannot measure the $b$ quark charge
directly. We have used $b$-jet charge weighting (weighted sum of all the
tracks in the jet) and semileptonic $b$-decay approaches to measure $b$ quark
charge. By using the weighting technique, it is possible to distinguish
between the $b$-jet charges associated with leptons of opposite charges with a
$5\sigma$ significance with only $0.1$ fb-1 of data ( $1$ fb-1 for
semileptonic $b$-decay approach) which allows the Standard Model
(${t\rightarrow W^{+}b}$) and exotic (${t^{\prime}\rightarrow W^{-}b}$)
scenarios to be distinguished. Reconstruction of the magnitude of the top
quark charge seems to be possible with $\simeq 1$ fb-1 using the weighting
technique, but it is necessary to check the performance of the method with
real data. The reconstructed $b$ quark and top quark charge are shown in
Figure 1 with $1$ fb-1 of simulated data. The resulting top quark charge is
$Q_{t}=0.67\pm 0.06\;(stat)\pm 0.08\;(syst)$.

Figure 1: Left: the $b$-jet charge ($Q_{\mathrm{{b}}}$) distribution; right:
the reconstructed top quark charge ($Q\mathrm{{}_{t}}$).

###  II.2 Top quark spin and spin correlations and _Wtb_ anomalous couplings

In the Standard Model, the top quarks are produced unpolarised in $t\bar{t}$
events, but their spins are correlated[4]. The production asymmetry ($A$ and
$A_{\mathrm{D}}$) can be obtained from the angular distribution of the top
quark decay products. In addition to the $t\bar{t}$ spin correlation, we can
measure the $W$ polarization. The $W$-boson can be produced with
right($F_{\mathrm{R}}$), left($F_{\mathrm{L}}$) or longitudinal
polarizations($F_{\mathrm{0}}$) with
$F_{\mathrm{0}}+F_{\mathrm{L}}+F_{\mathrm{R}}=1$. The expected measurement
results, using 1 fb-1 of simulated data, are shown in Table 1. It is also
possible to parameterise new physics in the $Wtb$ vertex using anomalous
coupling parameters $V_{\mathrm{L}}$, $V_{\mathrm{R}}$, $g_{\mathrm{L}}$ and
$g_{\mathrm{R}}$. Figure 2 shows the expected 68% C.L. allowed regions on the
$Wtb$ anomalous couplings for 1 fb-1.

Table 1: $W$-boson polarization and top quark spin correlation parameters with statistical and systematic errors.  $W$-boson polarization | $F_{\mathrm{L}}$ | $F_{\mathrm{0}}$ | $F_{\mathrm{R}}$
---|---|---|---
| 0.29 $\pm$0.02 $\pm$0.03 | 0.70 $\pm$0.04 $\pm$0.02 | 0.01 $\pm$0.02 $\pm$0.02
$t\bar{t}$ spin correlation | $A$ | $A_{\mathrm{D}}$ |
| 0.67 $\pm$0.17$\pm$0.18 | -0.40 $\pm$0.11 $\pm$0.09 |

Figure 2: The expected 68% C.L. allowed regions on the $Wtb$ anomalous
couplings for $L=1$ fb-1

Figure 3: 95% C.L. expected limits on the $BR(t\to q\gamma)$ vs $BR(t\to qZ)$

Figure 4: 5$\sigma$ discovery potential of a generic narrow $t\bar{t}$
resonance as a function of the integrated luminosity.

###  II.3 ATLAS sensitivity to FCNC top quark decays

We have studied the rare top quark decays via FCNC ($t\rightarrow qX$,
$X=\gamma,Z,g$) using $t\bar{t}$ events in $1$ fb-1 of simulated LHC data. One
of the top quarks is assumed to decay through its dominant decay mode ($t\to
bW$), while the other top quark decays via one of the FCNC modes ($t\to qZ$,
$t\to q\gamma$, $t\to qg$). Due to the large QCD background, it is very
difficult to search for FCNC signal using modes where $W$ or $Z$ decay
hadronically. Due to this reason, only leptonic decays of both $W$ and $Z$
were taken into account. For signal events, we have used $t\bar{t}\to b\ell\nu
qX$, where $X=\gamma,Z\to\ell\ell,g$ and $\ell=\mathrm{e},\mu$ and taken into
account the expected Standard Model backgrounds.

For $t\bar{t}\to bWq\gamma$, we require exactly one lepton with
$p_{\mathrm{T}}>25$ GeV, at least two jets with $p_{\mathrm{T}}>20$ GeV, one
$\gamma$ with $p_{\mathrm{T}}>25$ GeV and $\not\\!p_{\mathrm{T}}>20$ GeV. For
$t\bar{t}\to bWqg$, we require exactly one lepton with $p_{\mathrm{T}}>25$
GeV, exactly three jets with $p_{\mathrm{T}}>40,20,20$ GeV and
$\not\\!p_{\mathrm{T}}>20$ GeV. For $t\bar{t}\to bWqZ$, we require exactly
three leptons with $p_{\mathrm{T}}>25,15,15$ GeV, at least two jets with
$p_{\mathrm{T}}>30,20$ GeV and $\not\\!p_{\mathrm{T}}>20$ GeV. The neutrino
four momentum was estimated using a kinematic fit[3]. The expected 95% C.L.
upper limits on the branching ratios for $t\to qZ,t\to q\gamma,t\to qg$ are
10-3, 10-3, 10-2 respectively using $1$ fb-1 simulated data. Figure 3 shows
the expected 95% C.L. for the first $1$ fb-1 in the absence of signal for the
$t\to q\gamma$ and $t\to qZ$ channels.

###  II.4 $t\bar{t}$ resonances

The discovery potential for generic $t\bar{t}$ resonances with the ATLAS
detector has been explored as a function of the resonance mass for the
semileptonic $t\bar{t}$ channels[5]. $t\bar{t}$ resonances were produced with
Pythia for $Z^{\prime}\to t\bar{t}$ channel. The common selection criteria
have been applied for event reconstruction. The main source of background for
$t\bar{t}$ resonances is the Standard Model $t\bar{t}$ events (other
backgrounds like $W$+jets are negligible). It is possible to discover a 700
GeV $Z^{\prime}$ resonance produced with a $\sigma\times Br(Z^{\prime}\to
t\bar{t})$ of 11 pb with a 5$\sigma$ significance with 1 fb-1 of data [Figure
4]. Using a model-independent approach, ATLAS can exclude Kaluza-Klein gluon
resonances upto 1.5 TeV with only 1 fb-1 data[3].

##  III References

[1] F. Abe $et$ $al.$ (CDF Collaboration), Phys. Rev. Lett. 74, 2626 (1995);
S. Abachi $et$ $al.$ (D0 Collaboration, $ibid.$ 74, 2632 (1995).

[2] Lillie, B and Randall, L. and Wang, L.-T., hep-ph/0701166v1 (2007)

[3] ATLAS Collaboration, CERN-OPEN–2008-020, Geneva, 2008 to appear.

[4] W. Bernreuther, Nucl. Phys. B 690 81 (2004).

[5] E. Cogneras and D. Pallin, ATL-PHYS-PUB-2006-033 (2006).

##  IV Acknowledgements

The author would like to thank the organizers of ICHEP08 conference for
creating fruitful collaborative environment. My sincere thanks to Antonio
Onofre, Patrick Skubic and Martine Bosman for valuable suggestions.


# Spectral Introspection Identifies Group Training Dynamics in Deep Neural
Networks for Neuroimaging

Bradley T. Baker∗,†, Vince D. Calhoun∗,†,⋄, Sergey M. Plis∗,†

($*$The Georgia State University, Georgia Institute of Technology, Emory
University Center for Translational Research in Neuroimaging and Data Science
(TReNDS)
$\dagger$Georgia State University
$\star$Georgia Institute of Technology
$\diamond$Emory University)

###### Abstract

Neural networks, whice have had a profound effect on how researchers study
complex phenomena, do so through a complex, nonlinear mathematical structure
which can be difficult for human researchers to interpret. This obstacle can
be especially salient when researchers want to better understand the emergence
of particular model behaviors such as bias, overfitting, overparametrization,
and more. In Neuroimaging, the understanding of how such phenomena emerge is
fundamental to preventing and informing users of the potential risks involved
in practice. In this work, we present a novel introspection framework for Deep
Learning on Neuroimaging data, which exploits the natural structure of
gradient computations via the singular value decomposition of gradient
components during reverse-mode auto-differentiation. Unlike post-hoc
introspection techniques, which require fully-trained models for evaluation,
our method allows for the study of training dynamics on the fly, and even more
interestingly, allow for the decomposition of gradients based on which samples
belong to particular groups of interest. We demonstrate how the gradient
spectra for several common deep learning models differ between schizophrenia
and control participants from the COBRE study, and illustrate how these
trajectories may reveal specific training dynamics helpful for further
analysis.

##  1 Introduction

In recent years, a broad library of “introspection” methods have emerged to
mitigate the difficulty of straightforward interpretation [1, 2, 3, 4, 5, 6].
In particular, most state of the art research on DNN interpretation has
utilized post-hoc analysis of model gradients. For example, the popular
integrated gradients measure [1], saliency maps [6], and other gradient-based
methods [3, 4] have proved popular in applications to convolutional neural
networks as they provide simple spatial visualizations which make for easy
building of intuition. While these methods are extremely popular, the
resulting visualizations tend to be noisy and inconsistent [7, 8, 9, 10], and
recent research in the field has attempted to mitigate these limitations, for
example by using refining maps based on region attribution density [11, 12],
providing an adaptive interpolation path [13], or imposing geometric
constraints on produced maps [14, 15].

While post-hoc gradient analysis techniques can be useful for interpretation
on pre-trained models, these methods are not useful for evaluating training
dynamics; they can explain where a model currently is, but now how it arrived
there. While theoretical analyses of dynamics [16, 17] provide some guidance
to describing model dynamics, new empirical metrics and visualizations are
also important for building intuition about model behavior while also
accumulating evidence for further theoretical interpretation.

One existing method which empirically describes model dynamics is the
information-plane method [18, 19, 20], in which the mutual information between
layer weights and the input and output spaces is computed and visualized
during training. While this method provides fascinating visualizations of
dynamic behavior, the general interpretation of the dynamics adhering to the
information bottleneck principle [21] is not immediately clear [22] and can be
sensitive to architectural choices or the empirical method used to estimate
mutual information.

We present a novel empirical method for analyzing model learning dynamics,
which builds off of our theoretical work studying gradient rank. We call our
method AutoSpec, as it utilizes unique opportunities within auto-
differnetiation to analyze model learning dynamics. While we previously showed
that model architecture can impose theoretical limits on gradient rank, there
is more that can be observed at work within auto-differentiation. Namely, we
show that the singular values of the gradient and of the component matrices
which are used to compute it can be studied they dynamically evolve during
model training. Furthermore, because we can analyze gradients on-the-fly
within the auto-differentiation mechanism, we have the unique opportunity to
analyze these dynamics as they adhere to individual samples from the training
data set. As long as these samples have some kind of common group labelling,
we can thus do statistical comparisons of gradient trajectories between groups
without breaking normal training behavior. This further allows our method to
stand out from post-hoc methods which not only occur outside of training, but
can only be evaluated for between different classes in disjoint contexts.

###  1.1 Auto Differentiation

##  2 Methods

In this section, we provide an overview of the methods at work in AutoSpec.
First, we provide a brier review of how the gradient of the weights is
computed within auto-differentiation, recall how the spectrum is bounded by
particular architectural decisions, and show how the spectrum of the gradient
relates to the spectrum of the input activations and adjoint variables
accumulated within auto-differentiation. We then describe how auto-
differentiation uniquely allows us to analyze dynamics between particular
groups of samples.

###  2.1 Gradient Spectra via Auto-Differentiation

Recall that during reverse-mode auto-differentiation, the gradient of the
weights at a given layer $i$ is computed as a product of the input activations
$\mathbb{A}$ and the adjoint variable $\mathbb{\Delta}$ which is the partial
derivative computed on the output neurons during back-propagation. Formally,
if we have weights $\mathbb{W}_{i}\in\mathbb{R}^{h_{i-1}\times h_{i}}$ where
$h_{i-1}$ and $h_{i}$ are the number of input and output neurons respectively.
Formally, we write this as:

$\displaystyle\nabla_{\mathbf{W}_{i}}$
$\displaystyle=\mathbf{A}^{\top}_{i-1}\mathbf{\Delta}_{i}$ (1)

where $\mathbb{A}\in\mathbb{R}^{N\times h_{i-1}}$ and
$\mathbb{\Delta}\in\mathbb{R}^{N\times h_{i}}$ are the input activations and
adjoint variables with batch size $N$. The Singular Value Decompositions of
$\nabla_{\mathbf{W}_{i}}$, $\mathbf{A}_{i-1}$ and $\mathbf{\Delta}_{i}$ can be
written as:

$\displaystyle\nabla_{\mathbf{W}_{i}}$
$\displaystyle=\mathbf{U}_{\nabla_{i}}\mathbf{\Sigma}_{\nabla_{i}}\mathbf{V}_{\nabla_{i}}^{\top},\,\,\mathbf{A}_{i-1}=\mathbf{U}_{A_{i-1}}\mathbf{\Sigma}_{A_{i-1}}\mathbf{V}_{A_{i-1}}^{\top},\,\,\mathbf{\Delta}_{i}=\mathbf{U}_{\Delta_{i}}\mathbf{\Sigma}_{\Delta_{i}}\mathbf{V}_{\Delta_{i}}^{\top}$
(2)

We can then write $\mathbf{\nabla_{\mathbf{W}_{i}}}$ as a product of the SVDs
of $\mathbf{A}_{i-1}$ and $\mathbf{\Delta}_{i}$, and use the fact that the
$\mathbf{U}$ matrices are orthogonal to get:

$\displaystyle\nabla_{\mathbf{W}_{i}}$
$\displaystyle=\mathbf{V}_{A_{i-1}}\mathbf{\Sigma}_{A_{i-1}}\mathbf{U}_{A_{i-1}}\mathbf{U}_{\Delta_{i}}^{\top}\mathbf{\Sigma}_{\Delta_{i}}\mathbf{V}_{\Delta_{i}}^{\top}$
(3)
$\displaystyle=\mathbf{V}_{A_{i-1}}\mathbf{\Sigma}_{A_{i-1}}\mathbf{\Sigma}_{\Delta_{i}}\mathbf{V}_{\Delta_{i}}^{\top}$
(4)

Thus, we can see that the singular values of $\nabla_{\mathbf{W}_{i}}$ are
just the singular values of the first $\min(h_{i-1},h)$ singular values from
the input activations and adjoint variable.

For the sake of analysis, we can compute the SVD of all three statistics-of-
interest just by computing the SVD and the input activations and adjoint
matrices; however, because the batch size $N$ might be large, if the gradient
is the only statistic of interest, it would often be more efficient to compute
the SVD of $\nabla_{\mathbf{W}_{i}}$ directly.

For networks which utilize parameter tying, such as Convolutional or Recurrent
Neural Networks, the gradients are often accumulated over time and space. If
desired, our unique perspective from within Auto-Differentiation allows us to
peek further into these dimensions, characterizing the spectra not only of the
gradient of the weights, but of the gradient of the weights over the dimension
of tying.

###  2.2 Identifying Group Differences

One of the advantages of computing our introspection statistics within auto-
differentiation is that we have access to the individual gradients for each
input sample to the model. Thus, we can evaluate how particular groups of
samples individually contribute to the aggregated gradients prior to the
aggregated gradient. Formally, if we have $C$ distinct groups of samples in
our training set, we can compute the set of $C$ gradients and their SVD as

$\displaystyle\\{\nabla_{c}$
$\displaystyle=\mathbf{U}_{c}\mathbf{\Sigma}_{c}\mathbf{V}_{c}^{\top}\\}_{c=1}^{C}$
(5)

we can then perform statistical testing by accumulating these group-specific
statistics during training, and evaluating the differences between groups. For
example, if we perform a two-tailed T-test, we can obtain a measure of which
training steps were significant between groups if we treat the number of
singular values as features. Additionally, we can obtain a measure of per-
singular-value significance by taking the T-tests between the transpose.

###  2.3 Data sets and Experimental Design

To demonstrate the kinds of dynamics which AutoSpec can reveal, we have
organized a battery of experiments across different data modalities and
architecture types.

First, we use two numerical data sets to show how AutoSpec allows for dynanic
introspection on Multi-Layer Perceptrons, Elman Cell RNNs, and 2-D CNNs. Our
choice of numerical data sets are the MNIST and Sinusoid data sets.

We then move from numerical data to an application in Neuroimaging analysis.
We first apply a Multi-Layer perceptron on FreeSurfer volumes, and then move
to analyzing functional MRI from the COBRE data set [23], which is a well-
studied data set especially for applications of deep learning [24, 25, 26,
27]. For our demonstration, we perform Spatially Constrained Independent
Component Analysis using the NeuroMark template [28], which provides us with
53 neurologically relevant spatially independent maps and associated time-
series. Using the time-series data, we demonstrate how AutoSpec can reveal
group-specific gradient dynamics in 1D and 2D CNNs, LSTMs and the BERT
transformer [29]. We then utiliez the spatial maps (aggregated over the number
of components by taking the maximum over the voxel dimension) to demonstrate
group-specific gradient dynamics in 3D-CNNs.

For all architectures and all data sets, we evaluate a few different scenarios
demonstrating the diversity of dynamics available in gradient spectra. For all
models and datasets, we perform two tasks: classification and auto-encoding;
however, we only demonstrate one model instance and group differences for the
auto-encoding task for clarity. Additionally, we evaluate “wide/shallow” (1
layer with 128 neurons) and “deep/thin” (3 layers with 8 neurons) variants of
each model. We finally compare how a different choice of activation function
can affect dynamics by evaluating each model with Sigmoid and Tanh activations
in constrast to ReLU activations which we use elsewhere. For all analyses, we
perform two-tailed T-Tests between each pair of classes in a given data set to
demonstrate where significant group differences emerge within a particular
scenario. Group comparisons for all architectural variants are included in
supplementary material.

A detailed outline of our experimental is included in 1 and 2. When not
otherwise specified, we use ReLU activations in all models, a learning rate of
$1\times 10^{-3}$, and 1000 epochs of training. Where possible, we perform
full-batch training rather than SGD, as the batch size can artificially
restrict the rank of the gradient. For the sake of this demonstration, we set
all models to use the same seed, and we only evaluate the models in a one-shot
training scenario.

##  3 Results

In this section, we present the empirical results demonstrating how AutoSpec
can be used to reveal group gradient dynamics in deep neural networks. All of
our figures follow the same format as follows: panels A and B compare the
dynamics between a model trained for sample reconstruction (panel A) and for
classification (panel B); panels C and D compare dynamics between tanh (panel
C) and relu (panel D) activations, panels E and F compare dynamics between
“thin” (panel E) and “wide” (panel F) variants of the base network with 8 and
64 neurons respectively. For a review of how each experiment is organized into
panels see table 1.

The experiments on the MNIST data set are included in 4 and 11 for the MLP and
2DCNN architectures respectively. The experiments for the sinusoid data set
evaluated with an RNN can be found in 18. The MLP applied to FSL data can be
found in 25. The experiments on COBRE ICA time-series can be found in figures
6.5, 6.6 and 6.7 for the LSTM, BERT, and 1D-CNN architectures respectively.
Finally, the experiments on COBRE ICA spatial maps can be found in figure 6.8.
See table 2 for a review of which data set and architecture combinations can
be found in particular figures.

Table 1:  Panel | Task | Model Dims | Activation
---|---|---|---
A | Auto-Encoding | [32] | ReLU
B | Classification | [32] | ReLU
C | Auto-Encoding | [32] | Tanh
D | Auto-Encoding | [32] | Sigmoid
E | Auto-Encoding | [8] | ReLU
F | Auto-Encoding | [64] | ReLU
G | Auto-Encoding (Group Differences) | [32] | ReLU
H | Classification (Group Differences) | [32] | ReLU
Table 2:  Dataset | Modality | Model | Figure
---|---|---|---
MNIST | Image | MLP | 4
MNIST | Image | 2DCNN | 11
SINUSOID | Time-Series | RNN | 18
FreeSurfer | Tabular | MLP | 25
COBRE | ICA Time-Series | LSTM | 32
COBRE | ICA Time-Series | BERT | 39
COBRE | ICA Time-Series | 1D-CNN | 46
COBRE | ICA Spatial Mapps | 3D-CNN | 53

##  4 Discussion

In this work, we have introduced a new method for model introspection called
AutoSpec, in which we utilize the singular value decomposition to study the
dynamic evolution of the spectrum of the gradient and its component matrices.
Our method reveals fascinating dynamics at work in a number of model
architectures, and also allows us to identify unique dynamics belonging to
particular groups within a data set. We demonstrated our model on numerical
datasets for sequence and image reconstruction and classification, and also
demonstrated the identification of group differences on a real neuroimaging
data set.

We will provide a brief discussion of some of the observed differences in
dynamics; however, we would like to stipulate that any general conclusions
regarding these dynamics will require further experimentation testing specific
architecture choices systematically over many repetitions, seeds, and data
sets. First of all we notice that for all data sets and architectures, the
dynamics we find with AutoSpec show very different trajectories between and
AutoEncoding and Classification task (see 1(a) and 1(b) for the MLP applied to
MNIST for example). Particularly the input layers across all cases are
affected by the change in the output structure, and corresponding singular
values in output layers are also different between the two tasks. The choice
of activation function can affect the gradient spectrum as well - we notice
for example when we compare the Sigmoid and Tanh activated LSTM (32) and 1D
CNN (46), we can see slight differences in the trajectory toward the start of
training, but the overall evolution stays the same. In the 2D (11) and 3D (53)
CNNs, however, we notice that Sigmoid and Tanh activations affect the spectrum
quite differently, with the effects in the 3D CNN particularly noticeable
early in the training period. The dynamics in the layers pulled from the BERT
architecture (39) are difficult to interpret as singular values tend to jump
drastically between individual epochs, perhaps indicated; however, even in
this noisy scenario, the BERT architecture reflects a general trend of
shrinking singular values which occurs in other architures during training.

In each of our analyses, we also compute group differences between the
observed dynamics. In general, smaller observed singular values tend to change
more drastically; however, because the values are so small and near the
threshold for machine epsilon of floating point numbers, it is unclear if any
observed differences are merely the result of noise. We do see larger singular
values show significant differences during training across a few different
architectures however, and interestingly, these differences are often
contained with a few layers. For example, the MLP autoencoder trained on FSL
data shows significant differences between SZ and HC groups in the middle
singular values of the output layer (see 25(a)), while the corresponding
classifier shows more group differences in the input layer (see 25(b)). The
LSTM, BERT and 1D CNN models all show significant differences between male SZ
and HC groups, with the LSTM showing differences mostly the Hidden-to-Hidden
gradients (see 32(a) and 32(b)), the 1D CNN showing significant differences at
the output layer (see 46(a) and 46(b)), and BERT showing differences across
the entire model for the autoencoder task (see 39(a)) with almost no
significant differences in the classifier ((see 39(b)). While more work is
needed to investigate what these differences mean for how the model treats
different sample groups differently, our finding of significant effects across
multiple tasks and architectures demonstrates that these kinds of dynamics may
be useful for further, targeted investigation.

The major limitation of our AutoSpec framework is the computational overhead
required for computing the singular values on-the-fly during training. In
general for a matrix $A\in\mathbb{R}^{m\times n}$, the complexity required for
computing the SVD is $O(\min(mn^{2},m^{2}n))$. If we want to analyze $L$
different layers at $T$ different training periods, the complexity of our
method further increases to $O(LT\min(mn^{2},m^{2}n))$. Even more overhead is
accumulated if we perform group-specific analyses. As such, AutoSpec in
practical usage will require long runtimes during training to perform a
comprehensive analysis; however, limiting the analysis to specific singular
values, layers, periods during training, or groups would reduce this overhead.
One potential direction of future work could derive an analytic update to the
gradient spectrum based on the initial SVD of the weights and input data, and
thus avoid recomputing the SVD entirely for each update.

The computational overhead has limited our experiments in this work to smaller
architectures, or variants of large architectures with smaller dimension
sizes. In future work, we would like to expand the analysis to one or two
larger scale architectures to provide principled insights into the dynamics of
these models; however, for the sake of surveying many model types in this work
we have kept the scope smaller for scalability.

Figure 1: Differences in Auto-Differentiation Spectra Dynamics for the first 4
classes in the MNIST data set, trained with various architectures and tasks
with a Multi-Layer Perceptron.

(a) MLP Autoencoder with hidden dim 32 and ReLU activations

(b) MLP Classifier with hidden dim 32 and ReLU activations

(a) MLP Autoencoder with hidden dim 32 and Sigmoid activations

(b) MLP Autoencoder with hidden dim 32 and Tanh activations

(a) MLP Autoencoder with hidden dim 8 and ReLU activations

(b) MLP Autoencoder with hidden dim 64 and ReLU activations

(a) Significant spectral differences between groups 0 and 3: MLP Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between groups 0 and 3: MLP Classifier
with hidden dim 32 and ReLU activations

Figure 4: Differences in Auto-Differentiation Spectra Dynamics for the first 4
classes in the MNIST data set, trained with various architectures and tasks
with a Multi-Layer Perceptron. Figure 5: Differences in Auto-Differentiation
Spectra Dynamics for the first 4 classes in the MNIST data set, trained with
various architectures and tasks with a 2D CNN. (a) CNN2D Autoencoder with
hidden dim 8 and ReLU activations (a) CNN2D Classifier with hidden dim 8 and
ReLU activations (a) CNN2D Autoencoder with hidden dim 8 and Sigmoid
activations (a) CNN2D Autoencoder with hidden dim 8 and Tanh activations (a)
CNN2D Classifier with hidden dim 8 and Sigmoid activations (a) CNN2D
Classifier with hidden dim 8 and Tanh activations

(a) Significant spectral differences between groups 0 and 3: CNN2D Autoencoder
with hidden dim 8 and ReLU activations

(b) Significant spectral differences between groups 0 and 3: CNN2D Classifier
with hidden dim 8 and ReLU activations

Figure 11: Differences in Auto-Differentiation Spectra Dynamics for the first
4 classes in the MNIST data set, trained with various architectures and tasks
with a 2D CNN. Figure 12: Differences in Auto-Differentiation Spectra Dynamics
for the first 4 classes in the Sinusoid data set, trained with various
architectures and tasks with an RNN (a) RNN Autoencoder with hidden dim 32 and
ReLU activations (a) RNN Classifier with hidden dim 32 and ReLU activations
(a) RNN Autoencoder with hidden dim 32 and Sigmoid activations (a) RNN
Autoencoder with hidden dim 32 and Tanh activations (a) RNN Autoencoder with
hidden dim 8 and ReLU activations (a) RNN Autoencoder with hidden dim 64 and
ReLU activations

(a) Significant spectral differences between groups 0 and 3: RNN Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between groups 0 and 3: RNN Classifier
with hidden dim 32 and ReLU activations

Figure 18: Differences in Auto-Differentiation Spectra Dynamics for the first
4 classes in the Sinusoid data set, trained with various architectures and
tasks with an RNN Figure 19: Differences in Auto-Differentiation Spectra
Dynamics on the FSL data set, trained with various architectures and tasks
with a Multi-Layer Perceptron. (a) MLP Autoencoder with hidden dim 32 and ReLU
activations (a) MLP Classifier with hidden dim 32 and ReLU activations (a) MLP
Autoencoder with hidden dim 32 and Sigmoid activations (a) MLP Autoencoder
with hidden dim 32 and Tanh activations (a) MLP Autoencoder with hidden dim 8
and ReLU activations (a) MLP Autoencoder with hidden dim 64 and ReLU
activations

(a) Significant spectral differences between groups SZ and HC: MLP Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between groups SZ and HC: MLP Classifier
with hidden dim 32 and ReLU activations

Figure 25: Differences in Auto-Differentiation Spectra Dynamics on the FSL
data set, trained with various architectures and tasks with a Multi-Layer
Perceptron. Figure 26: Differences in Auto-Differentiation Spectra Dynamics on
the COBRE data set, trained with various architectures and tasks with an LSTM.
(a) LSTM Autoencoder with hidden dim 32 and ReLU activations (a) LSTM
Classifier with hidden dim 32 and ReLU activations (a) LSTM Autoencoder with
hidden dim 32 and Sigmoid activations (a) LSTM Autoencoder with hidden dim 32
and Tanh activations (a) LSTM Autoencoder with hidden dim 8 and ReLU
activations (a) LSTM Autoencoder with hidden dim 64 and ReLU activations

(a) Significant spectral differences between HC, SZ and Sex: LSTM Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between HC, SZ and Sex: LSTM Classifier
with hidden dim 32 and ReLU activations

Figure 32: Differences in Auto-Differentiation Spectra Dynamics on the COBRE
data set, trained with various architectures and tasks with an LSTM. Figure
33: Differences in Auto-Differentiation Spectra Dynamics on the COBRE data
set, trained with various architectures and tasks with a BERT Transformer. (a)
BERT Autoencoder with hidden dim 32 and ReLU activations (a) BERT Classifier
with hidden dim 32 and ReLU activations (a) BERT Autoencoder with hidden dim
32 and Sigmoid activations (a) BERT Autoencoder with hidden dim 32 and Tanh
activations (a) BERT Autoencoder with hidden dim 8 and ReLU activations (a)
BERT Autoencoder with hidden dim 64 and ReLU activations

(a) Significant spectral differences between HC, SZ and Sex: BERT Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between HC, SZ and Sex: BERT Classifier
with hidden dim 32 and ReLU activations

Figure 39: Differences in Auto-Differentiation Spectra Dynamics on the COBRE
data set, trained with various architectures and tasks with a BERT
Transformer. Figure 40: Differences in Auto-Differentiation Spectra Dynamics
on the COBRE data set, trained with various architectures and tasks with a 1D
CNN. (a) CNN1D Autoencoder with hidden dim 32 and ReLU activations (a) CNN1D
Classifier with hidden dim 32 and ReLU activations (a) CNN1D Autoencoder with
hidden dim 32 and Sigmoid activations (a) CNN1D Autoencoder with hidden dim 32
and Tanh activations (a) CNN1D Autoencoder with hidden dim 8 and ReLU
activations (a) CNN1D Autoencoder with hidden dim 64 and ReLU activations

(a) Significant spectral differences between HC, SZ and Sex: CNN1D Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between HC, SZ and Sex: CNN1D Classifier
with hidden dim 32 and ReLU activations

Figure 46: Differences in Auto-Differentiation Spectra Dynamics on the COBRE
data set, trained with various architectures and tasks with a 1D CNN.

Figure 47: Differences in Auto-Differentiation Spectra Dynamics on the COBRE
data set, trained with various architectures and tasks with a 3D CNN. (a)
CNN3D Autoencoder with hidden dim 32 and ReLU activations (a) CNN3D Classifier
with hidden dim 32 and ReLU activations (a) CNN3D Autoencoder with hidden dim
32 and Sigmoid activations (a) CNN3D Autoencoder with hidden dim 32 and Tanh
activations (a) CNN3D Autoencoder with hidden dim 8 and ReLU activations (a)
CNN3D Autoencoder with hidden dim 64 and ReLU activations

(a) Significant spectral differences between HC, SZ and Sex: CNN3D Autoencoder
with hidden dim 32 and ReLU activations

(b) Significant spectral differences between HC, SZ and Sex: CNN3D Classifier
with hidden dim 32 and ReLU activations

Figure 53: Differences in Auto-Differentiation Spectra Dynamics on the COBRE
data set, trained with various architectures and tasks with a 3D CNN.

## References

  * [1] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in International conference on machine learning, pp. 3319–3328, PMLR, 2017.
  * [2] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision, pp. 618–626, 2017.
  * [3] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Müller, “How to explain individual classification decisions,” The Journal of Machine Learning Research, vol. 11, pp. 1803–1831, 2010.
  * [4] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034, 2013.
  * [5] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PloS one, vol. 10, no. 7, p. e0130140, 2015.
  * [6] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in International conference on machine learning, pp. 3145–3153, PMLR, 2017.
  * [7] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” arXiv preprint arXiv:1706.03825, 2017.
  * [8] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R. Müller, “Explaining nonlinear classification decisions with deep taylor decomposition,” Pattern recognition, vol. 65, pp. 211–222, 2017.
  * [9] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K.-R. Müller, “Evaluating the visualization of what a deep neural network has learned,” IEEE transactions on neural networks and learning systems, vol. 28, no. 11, pp. 2660–2673, 2016.
  * [10] P. Sturmfels, S. Lundberg, and S.-I. Lee, “Visualizing the impact of feature attribution baselines,” Distill, vol. 5, no. 1, p. e22, 2020.
  * [11] A. Kapishnikov, T. Bolukbasi, F. Viégas, and M. Terry, “Xrai: Better attributions through regions,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4948–4957, 2019.
  * [12] S. Xu, S. Venugopalan, and M. Sundararajan, “Attribution in scale and space,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9680–9689, 2020.
  * [13] A. Kapishnikov, S. Venugopalan, B. Avci, B. Wedin, M. Terry, and T. Bolukbasi, “Guided integrated gradients: An adaptive path method for removing noise,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5050–5058, 2021.
  * [14] M. M. Rahman, N. Lewis, and S. Plis, “Geometrically guided saliency maps,” in ICLR 2022 Workshop on PAIR $\\{$$\backslash$textasciicircum$\\}$ 2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2022.
  * [15] M. M. Rahman, N. Lewis, and S. Plis, “Geometrically guided integrated gradients,” arXiv preprint arXiv:2206.05903, 2022.
  * [16] A. M. Saxe, J. L. McClelland, and S. Ganguli, “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,” arXiv preprint arXiv:1312.6120, 2013.
  * [17] A. M. Saxe, J. L. McClelland, and S. Ganguli, “A mathematical theory of semantic development in deep neural networks,” Proceedings of the National Academy of Sciences, vol. 116, no. 23, pp. 11537–11546, 2019.
  * [18] N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in 2015 ieee information theory workshop (itw), pp. 1–5, IEEE, 2015.
  * [19] R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810, 2017.
  * [20] B. T. Baker, N. Lewis, D. Saha, M. A. Rahaman, S. Plis, and V. Calhoun, “Information bottleneck for multi-task lstms,” in NeurIPS 2022 Workshop on Information-Theoretic Principles in Cognitive Systems, 2022.
  * [21] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” arXiv preprint physics/0004057, 2000.
  * [22] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox, “On the information bottleneck theory of deep learning,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 12, p. 124020, 2019.
  * [23] A. R. Mayer, D. Ruhl, F. Merideth, J. Ling, F. M. Hanlon, J. Bustillo, and J. Canive, “Functional imaging of the hemodynamic sensory gating response in schizophrenia,” Human brain mapping, vol. 34, no. 9, pp. 2302–2312, 2013.
  * [24] P. Patel, P. Aggarwal, and A. Gupta, “Classification of schizophrenia versus normal subjects using deep learning,” in Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–6, 2016.
  * [25] J. Oh, B.-L. Oh, K.-U. Lee, J.-H. Chae, and K. Yun, “Identifying schizophrenia using structural mri with a deep learning algorithm,” Frontiers in psychiatry, vol. 11, p. 16, 2020.
  * [26] Y. Zhu, S. Fu, S. Yang, P. Liang, and Y. Tan, “Weighted deep forest for schizophrenia data classification,” IEEE Access, vol. 8, pp. 62698–62705, 2020.
  * [27] U. Mahmood, M. M. Rahman, A. Fedorov, N. Lewis, Z. Fu, V. D. Calhoun, and S. M. Plis, “Whole milc: generalizing learned dynamics across tasks, datasets, and populations,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VII 23, pp. 407–417, Springer, 2020.
  * [28] Y. Du, Z. Fu, J. Sui, S. Gao, Y. Xing, D. Lin, M. Salman, A. Abrol, M. A. Rahaman, J. Chen, et al., “Neuromark: An automated and adaptive ica based pipeline to identify reproducible fmri markers of brain disorders,” NeuroImage: Clinical, vol. 28, p. 102375, 2020.
  * [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017.