# Searching for internal symbols underlying deep learning

Jung H. Lee
Pacific Northwest National Laboratory
Seattle, WA
<EMAIL_ADDRESS>
&Sujith Vijayan
School of Neuroscience
Virginia Tech
Blacksburg, VA
<EMAIL_ADDRESS>

###### Abstract

Deep learning (DL) enables deep neural networks (DNNs) to automatically learn
complex tasks or rules from given examples without instructions or guiding
principles. As we do not engineer DNNs’ functions, it is extremely difficult
to diagnose their decisions, and multiple lines of studies proposed to explain
principles of DNNs/DL operations. Notably, one line of studies suggests that
DNNs may learn concepts, the high level features recognizable to humans. Thus,
we hypothesized that DNNs develop abstract codes, not necessarily recognizable
to humans, which can be used to augment DNNs’ decision-making. To address this
hypothesis, we combined foundation segmentation models and unsupervised
learning to extract internal codes and identify potential use of abstract
codes to make DL’s decision-making more reliable and safer.

##  1 Introduction

Deep learning (DL) can automatically wire deep neural networks (DNNs) to
perform complex tasks, and DNNs’ learning capability allows them to outperform
humans in some tasks [1, 2, 3]. Notably, instead of expert instructions or
feature engineering, DL requires only training examples to solve highly
complex problems. However, as DNNs’ structures are automatically determined
without guiding principles, we do not fully understand their decision-making
process. That is, DNNs may be useful black box decision-makers, but without
safety measures, deploying them in high-stakes domains (e.g., autonomous cars)
comes with unpredictable risks [4, 5]. For the moment, the current generation
of DL does not warrant essential safety to be deployed in safety-critical or
high-stakes domains. To address this shortcoming, earlier studies sought
potential methods that could decode and explain DNNs’ decisions. There are
multiple lines of research in explainable AI (e.g., see Refs. [6, 7, 8, 9,
10]), but the most relevant to this study is the one probing correlations
between hidden layer representations and inputs or between hidden layer
representations and outputs (i.e., decisions). Feature visualization [11] and
network dissection [12] sought direct links between inputs and hidden layers,
while linear probes investigated correlations between outputs and hidden
layers [13, 14].

It is notable that the mapping between hidden layers and visual features in
input layers, in principle, establishes links between high level patterns
recognizable to humans (often referred to as ‘concepts’) and hidden neurons.
The Test with Concept Activation Vectors [14] explicitly extended this idea to
detect concepts encoded in neural networks. The concepts (i.e., perceptible
patterns) coded in neural networks could deepen our understanding of DNNs’
operating principles and eventually provide explanations (readable to humans)
of DNNs’ decisions. More importantly, it raises the possibility that DNNs
could build some abstract codes in hidden layers, some of which could directly
be linked to concepts. In this study, we extended this line of studies by
seeking abstract codes, which are not necessarily recognizable to humans, in
hidden layers functionally linked to DNNs’ decisions. We also explored
potential methods of utilizing hidden layers’ abstract codes to make DNNs’
operations/decision-making more reliable and safer.

To make a clear distinction between human recognizable concepts and DNNs’ own
abstract codes, we will refer to DNNs’ abstract codes as ‘symbols’ hereafter.
Our analysis of hidden layer representations is based on four assumptions.
First, symbols ($S_{i}$, abstract codes of DNNs) are encoded across feature
maps in each layer. Second, responses of individual hidden layers $H_{i}$ can
be described by the sum of symbols $S_{i}$ and residuals $\epsilon$;
$H_{i}=S_{i}+\epsilon$. Conversely, we can map individual representations
$H_{i}$ into nearest $S_{i}$ by minimizing $\epsilon$. Third, there is a
finite number of symbols in DNNs, which is far less than the possible number
of hidden layer representations. Fourth, symbols are spatially local and thus
need to be extracted from proper receptive fields or regions of interest
(ROIs). With these assumptions in mind, we performed unsupervised clustering
analysis with hidden layer responses ($H_{i}$) to determine the cluster
centers and identify ‘symbols’. Our analysis suggests that these symbols can
mediate semantic meanings and allow us to 1) monitor the models’ decision-
making process, 2) detect abnormal operations related to adversarial
perturbations and out-of-distribution (OOD) examples and 3) temporarily learn
about OOD examples. Based on the results, we argue that hidden layers’
responses can be converted to discrete symbols that can help us build more
reliable and safer DNNs.

##  2 Extracting symbols underlying DNNs’ operations

To test if DNNs learn to use symbols to make decisions, we analyzed ImageNet
models’ hidden layer responses to a subset (Mixed_13) of ImageNet [15],
previously curated by a python machine learning library [16]. Mixed_13
contains 78 classes out of 1,000 classes of ImageNet, which belong to one of
13 super-classes (Supplementary Table1). Mixed_13 consists of two disjointed
‘training’ and ‘test’ sets. To extract the symbols, we assumed that the
symbols associated with DNN’s decision-making would be observed frequently.
For instance, if a symbol is associated with a tiger, a picture of a tiger
would elicit it frequently. With this assumption in mind, we presented the
inputs from Mixed_13 and searched for repeatedly occurring hidden layer
responses by using unsupervised clustering analysis.

We note that analyzing hidden layer responses can be challenging for two
reasons. First, hidden layer responses are extremely noisy high-dimensional
data. A massive number of hidden neurons exist in modern DNNs, and their
responses are determined with diverse inputs, making dimension reduction
essential (see [17, 18, 13] , for instance). Second, visual scenes often
contain multiple objects rather than a single one. For instance, a visual
scene may include a tiger or tigers walking along the river, playing with
other tigers and/or climbing a tree. Thus, if we want to extract codes related
to tigers, it is necessary to isolate regions of interest (ROIs) containing
tigers and exclude confounding objects (e.g., tree or river). However,
ImageNet training examples are labeled with single class names and do not
provide sufficient information to identify ROIs.

To address these challenges, we first used segmentation models to extract ROIs
and recorded hidden layer responses spatially corresponding to ROIs. Then, we
used ROI-pooling operation, proposed for object detectors [19], to extract
smooth 3-by-3 activation vectors per single feature map. Once the activation
vectors were extracted, we conducted clustering algorithms to determine the
codes repeatedly occurring with Mixed_13 (a subset of ImageNets). In section
2.1, we discuss how ROIs are extracted from Mixed_13. In section 2.2, we
present the details of extracting hidden layer responses. In section 2.3, we
explain our clustering analysis in detail.

###  2.1 ROIs Identification

We note that ‘Second Thought Certification’ (STCert) proposed in an earlier
study [20] can be used to identify ROIs related to visual objects with high
accuracy. STCert is a two-step process (Fig. 1). First, it uses foundational
segmentation models [21, 22] to detect ROIs related to DNNs’ predictions. For
instance, if a DNN predicts a tiger, STCert asks the segmentation model to
return the bounding box (i.e., ROI) of the tiger in the picture. Second, it
crops ROIs and uses them to confirm or reject DNNs’ original predictions.
STCert provides a more reliable ROI estimates than a single step segmentation,
since its segmentation is crosschecked with a classifier. As such, we used
STCert to identify ROIs in this study to extract ROIs from randomly chosen 200
training examples for each class in Mixed_13; see Supplementary text A.1 for
more detail. Since Mixed_13 contains 78 classes (Supplementary Table1), the
total number of training examples is 15,600. Among them, we first selected
examples, on which DNNs’ predictions were correct to minimize the bias induced
by the incorrect predictions/decisions of DNNs (i.e., incorrect mapping
between ROI and the inputs’ labels). For some of the images, STCert did not
find ROIs or found multiple ROIs, which means the actual number of ROIs are
determined by a few factors, not just the number of training examples.

Figure 1: Schematics of STCert [20].

###  2.2 Generating hidden layer activity vectors

Once ROIs were identified, we sampled activations ($H_{i}$) of hidden neurons,
whose receptive fields correspond to ROIs. To reduce noise and filter out non-
essential information, we used ROI-pooling [19] to convert hidden layer
activations into 3-by-3 activation maps. That is, the size of hidden layer
activation maps can be arbitrary depending on the size of objects, but ROI-
pooling always turns them into 3-by-3 activation maps. It should be noted that
individual activations in 3-by-3 activation maps reflect pixels at different
spatial locations. As we assumed that symbols are collectively encoded neurons
that share the same receptive field, we built vector codes (referred to as
activity vectors below) by collecting activations at the same location in the
maps across channels. For instance, there are 512 channels in ResNet18’s layer
4, and we have 9 location-specific activation maps in each channel. In this
case, we aggregated activations from the same location in the activation map
from all 512 channels and then created 9 activity vectors, each of which
contains 512 components, from the layer per image.

In this study, we analyzed 4 hidden layers of popular DNNs, ResNet18,
ResNet50, VGG19 and DenseNet121 [23, 24, 25]. ResNet and DenseNet consist of 4
composite blocks, and their outputs were analyzed. VGG19 contains 5 max
pooling layers, whose outputs roughly correspond to ResNet’s composite block’s
outputs. We selected the last 4 max-pooling layers and analyzed them, as they
match the layers of ResNet and DenseNet. We used Pytorch [26] and Timm [27]
python deep learning libraries to implement these models. A workstation with
Intel Core9 CPU and RTX 4090 was used for all our experiments and analyses.

###  2.3 Clustering analysis of hidden layer activity vectors

We turned to unsupervised clustering analysis to identify representative
activity vectors equivalent to symbols in this study. We also note that many
activity vectors contain low values. With these low-value activity vectors,
clustering algorithms consider the zero-valued vector as the most prominent
cluster center, which is not informative. Thus, we removed the vectors
consisting of values lower than the mean activity value estimated on each
layer. Next, we used UMAP analysis [28, 29] to further reduce them to
3-dimensional vectors; see Table 1 for the parameters used in the analysis.
These three dimensional vectors were analyzed to detect symbols via
clustering. It should be noted that the exact number of clusters existing in
the activity vector space are unknown, and the result of clustering analysis
strongly depends on the number of predefined clusters. To address this issue,
we used X-means clustering [30] to automatically discover an optimal number of
clusters (i.e., the number of symbols). We note that X-means is an extension
of Kmeans clustering, which uses Bayesian Information Criterion (BIC) to
determine the optimal number of cluster, and a python package ‘pyclustering’
[31] was used to implement X-means in this study. In our analysis, we restrict
the maximum number of classes to 1,000. Table 2 shows the identified number of
clusters in 4 layers of DNNs tested. As shown in the table, the optimal
numbers of clusters in early layers are smaller than 1,000, but those in the
late layers hit the maximum value, suggesting that homogeneity of activity
vectors increased, as layers go deeper.

Table 1: List of Parameters used in UMAP analysis. Parameter | Value
---|---
n_neighbors | 50
min_dist | 0.1
n_component | 3
metric | Euclidean

##  3 Symbols Analysis

###  3.1 Links between symbols and semantic meanings of inputs

Table 2: Number of symbols identified from 4 models. For VGG19, we list 4 last max-pooling layers, and for all other layers, we list 4 outputs of 4 functional blocks. Layer | ResNet18 | ResNet50 | VGG19 | DenseNet121
---|---|---|---|---
1 | 960 | 967 | 837 | 895
2 | 940 | 992 | 1000 | 1000
3 | 1000 | 1000 | 1000 | 1000
4 | 1000 | 1000 | 1000 | 1000

Then, we asked if the identified symbols (i.e., cluster centers) can be linked
to semantic meanings of inputs (i.e., labels) by correlating them with inputs’
labels. Specifically, for each image ($img_{k}$), we obtained ROIs ($ROI_{m}$)
and 9 symbols ($S_{mn}$, where $n=1,...,9$). These 9 symbols $S_{mn}$ were
correlated with the labels ($l_{m}$) of $ROI_{m}$, which were determined by
the label of $img_{k}$ containing $ROI_{m}$. That is, Correlation Map
($CM(i,j)$) evolves over all identified ROIs obtained from the training
examples of Mixed_13 according to Eq. 1.

$CM(i,j)=\begin{cases}CM(i,j)+1&\text{if $i=S_{mn}$, $j=l_{m}$},\text{where
}n=1,...,9\\\ CM(i,j)&\text{otherwise}\par\end{cases}$ (1)

, where $CM(i,j)$ is the component of Correlation map at $i^{th}$ row and
$j^{th}$ column; $S_{mn}$ denotes the $n^{th}$ (out of 9) symbol related to
$m^{th}$ ROI; where $l_{m}$ is the label of $m^{th}$ ROI (i.e., $img_{k}$).

If a symbol is associated with a specific class, it would appear frequently
when the class object is presented, but it would not appear when other class
objects are presented. Fig. 2 shows the correlation $CM(i,j)$ between 100
randomly chosen symbols from 4 layers of ResNet18 and 78 classes. $Y$-axis
denotes 100 randomly chosen symbols, and $x$-axis denotes the label (i.e.,
class). We found two types of symbols. First, some symbols were associated
with a broad range of examples. Second, some symbols were selectively linked
to specific classes. Symbols specific to classes were rare in early layers but
became more common in late layers. We found equivalent results in ResNet50,
VGG19 and DenseNet121 (Supplementary Fig. 1). In addition, we note that many
symbols can be merged together (Supplementary Fig. 2) and thus, did not
increase the maximum number of clusters above 1000. These results suggest that
some symbols could be directly linked to inputs’ labels, and thus, one could
infer the labels from their associated symbols.

Figure 2: The correlations between the symbols of ResNet18 and classes. We
show the correlation maps $CM(i,j)$ between 100 symbols randomly chosen and
all 78 classes of Mixed_13. $y$-axis denotes the indices of symbols, $x$-axis
denotes the class. (A)-(D), the correlations observed in layers 1-4,
respectively.

To test this possibility, we obtained the symbols associated with test
examples of Mixed_13 and asked if they can be used to predict the labels of
inputs (i.e., ROIs). In this experiment, we collected ROIs from all test
images whether or not DNNs make correct predictions. From these ROIs, we
collected activity vectors and obtained the test symbols (i.e., the nearest
cluster centers). These ‘test symbols’ were used to predict the likelihood of
each class by using $CM(i,j)$ (Eq. 1). As we aimed to compute the probability
of the class, the correlation map $CM(i,j)$ was normalized along with a row
(i.e., over classes) using SoftMax (2). By using the normalized Correlation
Map $P(i,j)$, we calculated the probability $P(i,j)^{m}$ that $m^{th}$ ROI
belonged to the class $j$ when the symbol $S_{i}$ was observed (Eq. 2). That
is, symbol indices were used as hash keys for a look-up table.

$P(i=S_{i},j)^{m}=\frac{e^{C(i=S_{i},j)}}{\sum_{n=1}^{78}e^{C(i=S_{i},n)}}$
(2)

As we obtained 9 symbols from each ROI, we took average probability over 9
(Eq. 3) and then determine the most probable class as the prediction $Pr_{m}$
(Eq. 4).

$P(j)^{m}=\frac{1}{9}\Sigma_{i=1}^{9}P(i,j)^{m}$ (3)

$Pr_{m}=\operatorname*{argmax}_{j}P(j)^{m}$ (4)

Fig. 3 shows the accuracy of the symbol-based predictions depending on the
layer. We note that the prediction accuracy is around 80% in layer 4 and much
lower in earlier layers. These accuracies cannot be directly compared to the
models’ accuracy because the choice is limited to one out of 78 classes, not
1000 classes. However, the accuracy in layer 4 and earlier layers were
significantly higher than chance (1/78), supporting that the identified
symbols were associated with semantic meanings.

If symbols are associated with semantic meanings, they could also be
associated with DNNs’ predictions. We tested this possibility by correlating
the symbols and DNNs’ predictions on Mixed_13 training examples (once again,
200 examples per class) instead of inputs’ labels. As stated above, we used
the symbols to infer their predictions on test examples and found that the
symbols are predictive of DNNs’ answers on test examples (Fig. 3B), raising
the possibility that DNNs could utilize symbols to make decisions. The
observed link between symbols and DNNs’ decisions leads us to hypothesize that
symbols can be used to augment DNNs’ decision-making process. Below we propose
potential ways to make DNNs’ operations more reliable using symbols.

Figure 3: Symbol-based predictions. (A), Symbol-based prediction on the labels
of test examples of Mixed_13. (B), Symbol-based prediction on the DNNs’
answers.

###  3.2 How confident are you, DNNs?

In the analysis above, we used $CM(i,j)$ to infer the most probable class by
using symbol as a hash key and $CM(i,j)$ as a look-up table. However, this
look-up table can be read in opposite direction. If we know the label $j$ of
inputs, we can infer how likely individual symbols $S_{i}$ appear in DNNs.
With this idea in mind, we defined ‘Expected Symbol Score (ESS)’ per layer
with respective label/class $j$ (Eq. 5).

$ESS_{j}=\frac{1}{9}\Sigma_{S_{1},...,S_{9}}P(S_{i},j)^{m}$ (5)

, where $S_{i}$ denotes the indices of 9 symbols obtained from each ROI. That
is, $ESS_{j}$ with respective class $j$ is the mean value of 9 probability
from $P(i,j)^{m}$; see Eq. (5). We estimated $ESS_{j}$ from all 4 layers.
First, we asked if ESSs obtained from layer 4 can be correlated with the
accuracy of DNNs’ answers. Specifically, we split ESSs into two distributions
depending on whether or not DNNs made the correct decisions.

We found that ESSs were significantly higher when predictions were correct
(see data marked by * in Fig. 4A) than when they were incorrect (see data
without * in Fig. 4A). To quantify the separation between correct and
incorrect predictions, we estimated AUROC (the orange line in Fig. 4B),
suggesting that ESSs can indicate DNNs’ likelihood of making correct
predictions. We also examined ESSs from layers 1, 2 and 3 and AUROC between
correct and incorrect answers (Fig. 4B). Further, we examined the norm of ESSs
$||(ESS_{1},ESS_{2},ESS_{3},ESS_{4})||$ from all layers, and AUROC estimated
from this norm were similar to those obtained from layer 4 (see blue line Fig.
4B).

Figure 4: Symbol-based confidence level. (A), Expected symbol score (ESS) in
ResNet18(Res18), ResNet50(Res50), VGG19 and DenseNet121(DENSE). The asterisks
denote ESSs when DNNs make correct predictions, which are higher than those
when DNNs make incorrect predictions. (B), AUROC between ESS when DNNs are
correct and ESS when DNNs are wrong. We tested ESS from layer 4(orange), layer
3(green), layer2(red) and layer1(violet). Also we tested the norm from all
layers, and AUROC between the norms is shown in blue.

###  3.3 Detecting out-of-distribution (OOD) examples

Detecting OOD examples has been a central challenge in computer vision models.
During training, models are forced to choose one of the predefined choices.
After training, they must choose the most probable answer on any given input,
even when the inputs are not related to its training. By nature, predictions
on OOD examples are random and should be ignored. To address this shortcoming,
OOD detection algorithms have been proposed to inform DL models (and its
users) when to ignore inputs; see [32] for a review.

We assumed that hidden layer responses (i.e., symbols), highly optimized for
in-distribution examples during training, are in synchronization across layers
during inference, when inputs are drawn from in-distribution, but they can be
out of synchronization with each other, when inputs are drawn from out-of-
distribution. Further, we speculated that ESSs can be used to measure the
synchrony between layers and thus detect OOD examples. To test this line of
hypotheses, we extracted the symbols from NINCO dataset [33], which contain
OOD examples carefully curated against ImageNet.

In our analysis, we first used $CM(i,j)$ of layer 4 to predict the most
probable class, which were incorrect, as the inputs were drawn from OOD, and
tested ESSs (once again, expected symbol scores) in layers 1,2 and 3. If the
symbols are truly out of synchronization due to OOD examples, we anticipated
low expected symbol scores in all three layers. We chose 18 classes from NINCO
dataset, and ROIs of NINCO images were determined by GroundingDINO (an open-
set segmentation model[21]) using their real class labels. We evaluated the
expected symbol scores and compared them with the ESSs of normal images from
the test set of Mixed_13.

Fig. 5A visualizes ESSs ($ESS_{1},ESS_{2},ESS_{3}$) from layers 1, 2, 3 in a
3D space. Blue circles and orange triangles denote in-distribution and OOD
examples respectively. As shown in the figure, ESSs were significantly
different between the in-distribution and OOD examples, and we quantified this
observation by estimating AUROC. The estimated AUROC from ESSs of all 3 layers
and their norms $||(ESS_{1},ESS_{2},ESS_{3})||$ suggest that ESSs are well
separated between normal and OOD examples and can be used to detect OOD
examples.

Figure 5: (A), Comparison of ESSs between in-distribution and OOD examples.
Orange triangles denote ESSs from layers 1,2 and 3 when inputs are drawn from
OOD. Blue circles denote ESSs when inputs are drawn from in-distribution
examples. (B), AUROC between ESSs for all 4 DNNs. AUROC were calculated using
ESSs from layers 3, 2 and 1. Additionally, ESSs were also calculated using the
norm of ESS from all 3 layers.

###  3.4 Detecting adversarial examples

Since the discovery of DL models’ vulnerability to adversarial examples [34,
35], adversarial perturbation has emerged as one of the greatest threats to
DL’s safety. Given that all inputs are progressively processed in modern DNNs,
it seems natural to assume that adversarial perturbation may change internal
representation on layer-by-layer basis. An earlier study [36] found that the
difference in representations in early layers was not noticeable, but in late
layers it grew significantly. We asked if ESSs can be used to detect the
influence on adversarial perturbation. Specifically, we crafted adversarial
examples from validation images of Mixed_13 using AutoAttack [37], which is
known to be one of the most effective attacks. In the experiment, we used a
‘standard’ mode with $eps=0.03$ and $norm=Linf$. For individual inputs, we
used labels of clean images to detect ROIs and estimated the corresponding
symbols; that is, ROIs are identified with original labels, not manipulated
predictions.

For ROIs, we estimated the norm of ESSs from all 4 layers
$||(ESS_{1},ESS_{2},ESS_{3},ESS_{4})||$ and calculated AUROC by comparing the
norms of ESSs between normal and adversarial inputs. We note that the AUROC
estimated from the norms were close to 0.5 (see the blue line in Fig. 6),
suggesting that the symbols in DNNs were not significantly different between
normal and adversarial examples. Consistent with this finding, the symbols of
adversarial inputs in layer 4 predict the original classes as accurately as
the symbols of the normal inputs do (the thick green line in Fig. 6). These
results suggest that the symbols are much more resilient than the models’
final predictions. That is, the detection of adversarial inputs may require
methods other than comparing internal symbols.

So far, ESSs have been computed using the most probable class predicted by the
layer 4 correlation map $CM(i,j)$, but they can also be computed using DNNs’
predictions. If ESSs are computed with DNNs’ predictions, we can effectively
evaluate the consistency between models’ decisions and internal symbols, which
we assume could be useful in detecting adversarial perturbation. However, we
analyzed unmodified ImageNet models, whose output nodes correspond to 1,000
classes, not 78 of Mixed_13, Consequently, the predictions were not
necessarily confined to 78 classes of Mixed_13. Thus, to build ESSs using
their predictions, we selected adversarial and normal symbols when predictions
are one of 78 classes (see Table 3 for the number of ROIs included in this
analysis). This Decision-based ESSs are compared in Fig. 6 (see the orange
line). As shown in the figure, Decision-based ESSs are significantly different
for all 4 models, suggesting that ESSs with respect to DNNs’ (manipulated)
predictions can be used to detect adversarial perturbation.

Figure 6: Symbol-based detection of adversarial perturbations. The orange line denotes AUROC calculated from decision-based ESS, whereas the blue dashed line denotes AUROC calculated ESS. The green line denotes the accuracy of symbol-based predictions on the adversarial inputs. Table 3: Number of normal and adversarial ROIs. For both normal and adversarial inputs, we first selected inputs (i.e., images), on which DNNs’ predictions were one of 78 classes of Mixed_13. Then, we selected all ROIs from these examples (both normal and adversarial). We list the number of ROIs selected below. It should be noted that images are not chosen using the correctness of DNNs’ predictions. Model | Normal | Adversarial
---|---|---
ResNet18 | 4002 | 1161
ResNet50 | 4320 | 1432
VGG19 | 3989 | 1015
DenseNet121 | 4133 | 765

###  3.5 Temporary learning

One of DL’s shortcomings is the lack of continual learning ability, whereas
the brain can learn continuously. The brain can use previous knowledge to
incorporate new knowledge, but DL models’ catastrophic forgetting, which
deletes previously obtained information to learn the new, prevents continual
learning. Since symbols can serve as previous knowledge, we speculate that the
temporary mapping between symbols of in-distribution examples and OOD examples
may allow a short-term (temporary) learning of OOD examples, which can serve
as a stop-gap measure, until a more formal retraining becomes available. To
address this possibility, we randomly split NINCO symbols (obtained from 18
classes) into 2 distinct distributions, which are training and test sets. The
size of the training and test sets are half of the full size of NINCO symbols.
Then, the training symbols were used to construct $CM(i,j)$, and we asked if
the test symbols can predict the inputs’ classes. As selecting training/test
symbols is done randomly, we resampled them 100 times and reported the results
from all 100 experiments (Fig. 7). We observed notable differences between the
models but do not know the reasons behind them. More importantly, the accuracy
is around 40-50% (Fig. 7) for all 4 models, supporting that internal symbols
can be used for short-term learning.

Figure 7: The accuracy of predictions on the OOD examples for ResNet18(Res18),
ResNet50(Res50), VGG19 and DenseNet121(DENSE). We resampled trainig and test
examples from 18 classes of NINCO dataset [33] for 100 times, and collected
their accuracies.

##  4 Discussion

We probed the existence of internal symbols closely linked to DNNs’ decision
making. More specifically, we combined STCert and traditional machine learning
algorithms to detect internal symbols. Our analysis of these internal symbols
suggest that they can be used to 1) predict levels of accuracies of DNNs’
answers, 2) detect OOD and adversarial inputs and 3) enable temporary learning
of OOD examples. Based on our analysis, we propose that decoding of internal
symbols could play a crucial role in making DNNs’ operations more reliable and
safer.

###  4.1 Limitations

We should underline that the purpose of our study is not to provide a
comprehensive survey of methods that can extract internal symbols and
potential utilities of internal symbols. Instead, we focused on searching for
positive evidence of the existence of internal symbols associated with DNNs’
decisions and exploring their applications. Below, we list the limitations of
our analysis as potential guides for future studies on internal symbols.

  * •

We selected Mixed_13 and ImageNet models for two reasons. First, if we extract
symbols from poorly trained models, the analysis can be biased. Second, as the
training set of ImageNet is huge, analyzing them requires substantial
computing resources. As Mixed_13 contains images from well-conceived 13 visual
categories and balanced number of classes across all categories, it is a good
subset to analyze the internal symbols related to ImageNet. However, this
raises the question of scalability of our analysis, and we will address this
in the future.

  * •

We extracted symbols on layer-by-layer basis and assumed that they are encoded
collectively by neurons with the same receptive fields across channels.
However, the feature visualization shows that some single neurons can encode
complex features recognizable to humans. We speculate that some symbols are
encoded by a set of neurons, and others, by single neurons, but this needs
further study.

  * •

Our analysis searched for symbols related to labels of ImageNet images, but
visual objects rarely exist alone in visual scenes. Other objects or
background can serve as context to an object of interest. That is, the symbols
can interact with one another. Such functional links between symbols are
ignored in our analysis.

  * •

We examined convolutional networks (CNNs) pretrained on ImageNets. Although
CNNs are one of the popular architectures, transformers have been widely used,
especially in language models. As transformers are based on fully connected
layers and self-attention modules, it remains unclear how to probe internal
symbols relevant to their decision-making.

###  4.2 Broader Impacts

We propose a potential algorithm to analyze DL models’ decision. As this
algorithm aims to reduce DL models’ mistakes, we expect it could be helpful in
making DL more reliable and are not aware of any potential negative impacts.

## References

  * [1] Yann Lecun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [2] Ajay Shrestha and Ausif Mahmood.  Review of deep learning algorithms and architectures.  IEEE Access, 7:53040–53065, 2019.
  * [3] Saptarshi Sengupta, Sanchita Basak, Pallabi Saikia, Sayak Paul, Vasilios Tsalavoutis, Frederick Atiah, Vadlamani Ravi, and Alan Peters.  A review of deep learning with special emphasis on architectures, applications and recent trends.  Knowledge-Based Systems, 194:105596, 2020.
  * [4] Zachary C. Lipton.  The Mythos of Model Interpretability.  In ICML WHI, 2016.
  * [5] Cynthia Rudin.  Please Stop Explaining Black Box Models for High Stakes Decisions.  In NIPS Workshop, 2018.
  * [6] Christoph Molnar.  Interpretable Machine Learning.  2 edition, 2022.
  * [7] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.  Explainable ai: A review of machine learning interpretability methods.  Entropy, 23(1), 2021.
  * [8] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  International Journal of Computer Vision, 128(2):336–359, oct 2019.
  * [9] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017.
  * [10] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross.  Towards better understanding of gradient-based attribution methods for deep neural networks.  In International Conference on Learning Representations, 2018.
  * [11] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2017.  https://distill.pub/2017/feature-visualization.
  * [12] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3319–3327. IEEE Computer Society, 2017.
  * [13] Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes, 2018.
  * [14] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV).  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2668–2677. PMLR, 10–15 Jul 2018.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [16] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras.  Robustness (python library), 2019.
  * [17] Archit Rathore, Nithin Chalapathi, Sourabh Palande, and Bei Wang.  Topoact: Visually exploring the shape of activations in deep learning, 2021.
  * [18] Emilie Purvine, Davis Brown, Brett Jefferson, Cliff Joslyn, Brenda Praggastis, Archit Rathore, Madelyn Shapiro, Bei Wang, and Youjia Zhou.  Experimental observations of the topology of convolutional neural network activations, 2022.
  * [19] Ross Girshick.  Fast r-cnn.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015.
  * [20] Jung H. Lee and Sujith Vijayan.  Having second thoughts? let’s hear it, 2023.
  * [21] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
  * [22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  arXiv:2304.02643, 2023.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [24] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [25] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2018.
  * [26] Adam Paszke, Sam Gross, Soumith Chintala, Edward Chanan, Gregory Yang, Zachary DeVito, Alban Lin, Zeming Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In NIPS Autodiff Workshop, 2017.
  * [27] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [28] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction, 2018.
  * [29] Tim Sainburg, Leland McInnes, and Timothy Q. Gentner.  Parametric umap: learning embeddings with deep neural networks for representation and semi-supervised learning.  ArXiv e-prints, 2020.
  * [30] Dan Pelleg and Andrew W. Moore.  X-means: Extending k-means with efficient estimation of the number of clusters.  In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, page 727–734, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.
  * [31] Andrei Novikov.  Pyclustering: Data mining library.  Journal of Open Source Software, 4(36):1230, apr 2019.
  * [32] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey, 2024.
  * [33] Julian Bitterwolf, Maximilian Mueller, and Matthias Hein.  In or out? fixing imagenet out-of-distribution detection evaluation.  In ICML, 2023.
  * [34] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay.  Adversarial attacks and defences: A survey, 2018.
  * [35] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li.  Adversarial examples: Attacks and defenses for deep learning.  IEEE Transactions on Neural Networks and Learning Systems, 30(9):2805–2824, 2019.
  * [36] Gabriel D. Cantareira, Rodrigo F. Mello, and Fernando V. Paulovich.  Explainable adversarial attacks in deep neural networks using activation profiles, 2021.
  * [37] Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2206–2216. PMLR, 2020.

##  Appendix A Appendix and Supplementary Material

###  A.1 STCert implementation

The two versions of STCert were discussed in the original study [20]. The
naive STCert uses the bounding boxes returned by the segmentation models as
ROIs. The context-aware STCert adds some backgrounds to the bounding boxes to
create ROIs. As the context-aware STCert is more robust, we adopted the
context-aware STCert in this study. The workflow of STCert can be summarized
as follows:

  1. 1.

Detect bounding boxes.

  2. 2.

Create 5 candidates of bounding boxes by enlarging ROIs. The first candidate
is ROI itself.

  3. 3.

Crop pixels within the candidates and forward them into DNNs. That is, we have
the list of 5 second thought predictions.

  4. 4.

Examine if the original prediction is included in the list of second
predictions.

  5. 5.

Return the bounding box as ROI, if the original prediction is in the list. If
not, ignore the bounding box and return Null.

###  A.2 Potential links to the brain structure

Our analysis raises the possibility that internal symbols can be used to
augment DNNs’ decision-making process. It may even be possible for a second
network to use symbols to correct responses in each layer locally during
inference. When immediate decisions are necessary, DNNs’ decisions can be
used, as they are, but for more reliable answers, a second network may use
symbols to modify hidden layers’ responses. This may remind some readers of
thalamo-cortical loops of the brain. Our analysis has been inspired by
thalamo-cortical loops and indicates that thalamo-cortical loop-like error
correction could be constructed using internal symbols to build more reliable
DNNs.

Supplementary Table 1: List of classes in the Mixed_13. Category | Class
---|---
Fish |  | ‘tench’, ‘goldfish’, ‘great_white_shark’,
---
‘tiger_shark’, ‘hammerhead’, ‘electric_ray’
Bird |  | ‘cock’, ‘hen’, ‘ostrich’,
---
‘brambling’, ‘goldfinch’, ‘house_finch’
Dog |  | ‘Chihuahua’, ‘Japanese_spaniel’, ‘Maltese_dog’,
---
‘Pekinese’, ‘Shih-Tzu’, ‘Blenheim_spaniel’
Carnivore |  | ‘cougar’, ‘lynx’, ‘leopard’,
---
‘snow_leopard’, ‘jaguar’, ‘lion’
Insect |  | ‘tiger_beetle’, ‘ladybug’,‘ground_beetle’,
---
‘long-horned_beetle’, ‘leaf_beetle’, ’dung_beetle’
Primate |  | ‘guenon’, ‘patas’, ‘baboon’,
---
‘macaque’, ‘langur’, ‘colobus’
Car |  | ‘ambulance’, ‘cab’, ‘convertible’,
---
‘jeep’, ‘limousine’, ‘station_wagon’
Furniture |  | ‘barber chair’, ‘bassinet’, ‘bookcase’,
---
‘chiffonier’, ‘china_cabinet’, ‘cradle’
Computer |  | ‘desktop_computer’, ‘hand-held_computer’,
---
‘laptop’, ‘notebook’, ‘slide_rule’, ‘web_site’
Fruit |  | ‘Granny_Smith’, ‘rapeseed’, ‘corn’,
---
‘acorn’, ‘hip’, ‘buckeye’
Fungus |  | ‘coral_fungus’, ‘agaric’,‘gyromitra’,
---
‘stinkhorn’, ‘earthstar’, ‘hen-of-the-woods’
Truck |  | ‘fire_engine’, ‘garbage_truck’, ‘moving_van’,
---
‘pickup’, ‘police_van’, ‘tow_truck’
Supplementary Figure 1: The correlations between the symbols and classes. Top,
middle and bottom rows show the correlations observed from ResNet50, VGG19 and
DenseNet121. $y$-axis denotes the indices of symbols, $x$-axis denotes the
class. (A)-(D), the correlations observed in layers 1-4, respectively.
Supplementary Figure 2: The correlations between symbols and classes in
ResNet18. These plots show the correlation maps between 200 consecutive
symbols and the label of inputs observed in layer 1 (A), layer 2 (B), layer 3
(C) and layer 4 (D). We note that some consecutive symbols are correlated with
the same classes and thus can be merged together, especially in layer 4.