# Predicting Students’ Exam Scores
Using Physiological Signals

Willie Kang∗1,2, Sean Kim∗1,3, Eliot Yoo∗1,4, and Samuel Kim1

1 IF Research Lab, La Palma, CA, USA
2 El Toro High School, Lake Forest, CA, USA
3 Oxford Academy, Cypress, CA, USA
4 Cypress High School, Cypress, CA, USA
{wildmanwillie25, seankim.hahjean<EMAIL_ADDRESS>
<EMAIL_ADDRESS>∗ These authors contributed equally. The names are
listed in an alphabetical order.

###### Abstract

While acute stress has been shown to have both positive and negative effects
on performance, not much is known about the impacts of stress on students’
grades during examinations. To answer this question, we examined whether a
correlation could be found between physiological stress signals and exam
performance. We conducted this study using multiple physiological signals of
ten undergraduate students over three different exams. The study focused on
three signals, i.e., skin temperature, heart rate, and electrodermal activity.
We extracted statistics as features and fed them into a variety of binary
classifiers to predict relatively higher or lower grades. Experimental results
showed up to 0.81 ROC-AUC with $k$-nearest neighbor algorithm among various
machine learning algorithms.

##  I INTRODUCTION

College students are prone to stress due to the highly transitional and
demanding nature of their lives, which may be because of rigorous academic
requirements, an unfamiliar environment, and separation from home. Academic
stress is a regular part of the lives of students, and may result from
pressures to perform, perceptions of workloads and exams, and time restraints
[1]. Failure to cope with such high stress can lead to various negative
effects. Severe academic stress decreases academic performance and hinders the
ability to study effectively [2, 3]. Overall, stress has been shown to
negatively impact sleep quality, well being, and affectivity, which in turn
negatively impacts general health [4].

Additionally, students may experience more severe issues during examination
season. This period is often marked by high stress and anticipation, with
numerous important projects, papers, and exams all colliding. During this
time, sleep quality has been shown to decrease and caffeine consumption has
been shown to increase [5, 6].

Students are also adversely impacted by test anxiety. Higher levels of
cognitive test anxiety have been associated with significantly lower test
scores [7]. A study of nursing students has also shown that test anxiety
causes physical, emotional, and cognitive detriments, which hinders academic
success [8]. There also exists an inverse relationship between test anxiety
and grade point average in both graduate and undergraduate students [9].

Exam stress and anxiety is a significant problem that affects all students.
Working on this issue can lead to not only academic improvements, but physical
and mental health benefits. Being able to predict exam performance through
common physiological signals that correlate with stress can serve as a useful
tool to help address the issue of test anxiety. Therefore, this study aims to
look at the viability of predicting exam scores with physiological signals
using machine learning algorithms.

(a) Skin temperature - Midterm 1

(b) Skin temperature - Midterm 2

(c) Skin temperature - Final

(d) Heart Rate - Midterm 1

(e) Heart Rate - Midterm 2

(f) Heart Rate - Final

(g) Electrodermal activity - Midterm 1

(h) Electrodermal activity - Midterm 2

(i) Electrodermal activity - Final

Figure 1: Physiological signals of the individual students during exams.

##  II Procedure

###  II-A Data Source

The data we used was collected from a study conducted at the University of
Houston on eleven undergraduate students (nine males, two females) who were
tracked across three major exams: two midterms and a final exam [10]. The
students wore E4 wristbands that measured skin conductance, electrodermal
activity (EDA), heart rate, blood volume pulse, skin surface temperature,
inter-beat interval, and accelerometer data.

Of the eleven participants, one student was provided additional accommodations
due to the University of Houston disability accommodation guidelines. Data
from this participant was discarded as it involved a factor not consistent
with the other participants. See [11] for more details.

For our research, we chose to incorporate skin temperature, heart rate, and
EDA measurements. Figure 1 shows the selected physiological signals of
individual students collected during different examinations.

###  II-B Pre-Processing

Firstly, we synchronized all the measurements aligned at the same timestamp.
Since the data was collected in an asynchronized manner, we dropped out any
measurements that are outside of common time periods.

Secondly, we found some outliers and missing values in measurements.
Therefore, we applied a filtering method, moving average low-pass filter to be
specific, to remove possible noise and outliers.

Lastly, the physiological signals can be influenced by personal biases and
environmental factors. For example, individual skin temperatures can be
influenced by the room temperature and some students can have innately higher
heart rates then the others. To mitigate these biases, we normalized the data
before inputting the data into the machine learning algorithms. The
normalization was done both on a student-level and a test-level. We used the
z-normalization so that individual instruments have zero means and unit
standard deviations, i.e.

$x(t)=\frac{x(t)-\mu}{\sigma}$ (1)

where $x(t)$ represents measured value of the instrument at time $t$, and
$\mu$ and $\sigma$ are the notation for the average and the standard deviation
of the measurement over time, respectively.

###  II-C Feature Extraction

As described earlier, we used skin temperature, heart rate, and EDA of the
students. After the pre-processing, we extracted the statistics of a
physiological signal as feature vectors to the instrument during an exam. The
statistics consist of mean, standard deviation, minimum, maximum, and median
(the feature dimension is 5). Then, we concatenate all the features to create
one super-vector to represent overall physiological behaviors during the exam
(the dimension of the super-vector is 15).

Since one student takes three different exams, i.e. two midterms and one
final, each student will have three different physiological behavior features
and corresponding test scores.

Figure 2: Basic diagram of each validation step for one-student-leave-out
experimental setup.

##  III Experiments

###  III-A Experimental Setup

We used all the features regardless of exam types so that each student has
three different scores and corresponding physiological features. The train and
test sets were split in a one-student-leave-out way which means nine students
would be used to train the classifier and the other student would be used to
test it. This creates 10-fold cross-validation, and each validation task
consists of 27 training samples and 3 test samples. Figure 2 illustrates this
scenario as a simple diagram.

We designed the experiments as binary classification tasks. In this regard, we
built models to classify whether students received a score higher than 80

We repeated the experiments 10 times so that we can get the average
performance of individual machine learning algorithms.

###  III-B Classifiers

Multiple machine learning models were used. Using a diverse amount of
classifiers allows for the various algorithms to search for a correlation
between the stress signal values and the performance of the student. These
machine learning models were the Random Forest (RF, with a gridsearch
technique for best parameters in each validation task), Stochastic Gradient
Descent (SGD, with log-loss), Support Vector Machine (SVM, with RBF kernel and
$C=1$), and $k$-nearest neighbor (KNN, with $k=5$) classifiers.

###  III-C Results

Figure 3 and Table I show the results of the binary classification tasks in
terms of ROC-AUC using various machine learning algorithms. Overall, the KNN
gave the best results with a 0.81 ROC-AUC on average in the relationship
between stress levels and high scoring on exams. This classifier shows that
there exists a correlation between stress and test scores that could be
further investigated to find a stronger relation on how stress levels can
affect the performance of a student. The SVM Classifier produced the second
best results with a 0.80 ROC-AUC in the relationship between stress and exam
scores. This further shows that there is a considerable correlation between
stress and scores.

On the other hand, RF and SGD did not yield sufficient ROC-AUC scores, which
indicate that those machine learning algorithms are not performing well enough
to model the relationship between physiological behaviors and test scores.

###  III-D Limitations

One limitation of our study is the small number of statistics extracted from
the chosen physiological signals during feature extraction. We only utilized
basic statistics as features. Using more comprehensive features may serve to
better map the physiological signals to the exam scores. Furthermore,
analyzing a larger dataset may help improve the accuracy of results.

Figure 3: ROC curves with various machine learning algorithms. | RF | SGD | SVM | KNN
---|---|---|---|---
ROC-AUC | 0.54 | 0.56 | 0.80 | 0.81
(0.09) | (0.06) | (0.06) | (0.00)
TABLE I: Average ROC-AUC scores (standard deviation) with various machine
learning algorithms.

##  IV Conclusion

The present research examined how stress affects academic performance through
physiological signals. The results of this study support the initial
hypothesis, suggesting a correlation between stress and exam results. These
preliminary results have multiple implications for future research and further
developments in the field. By looking at stress measurements, we can formulate
strategies to maximize academic performance by looking at optimal levels of
stress. Additionally, we can identify what factors have the greatest impact on
stress and academic performance. Certain physiological signals may have
detrimental effects on performance as they increase, while others may function
in the inverse. Measurements that have the greatest impact on academic
performance can be further investigated through various research and testing.
The hypothesis attempted to form a general correlation based on the limited
data and information available, but the opportunities for further improvement
and different program creation are abundant. We expect future works to use
this research as a foundation for more elaboration and growth within the
fields of academics and stress.

## References

  * [1] Dalia Bedewy and Adel Gabriel,  “Examining perceptions of academic stress and its sources among university students: The perception of academic stress scale,”  Health Psychology Open, vol. 2, no. 2, pp. 205510291559671, 2015\.
  * [2] Mussarat Khan,  “Effect of perceived academic stress on students’ performance,”  FWU Journal of Social Sciences, 08 2018.
  * [3] Nudrat Sohail,  “Stress and academic performance among medical students,”  Journal of the College of Physicians and Surgeons–Pakistan : JCPSP, vol. 23, pp. 67–71, 01 2013.
  * [4] Kathrin Wunsch, Nadine Kasten, and Reinhard Fuchs,  “The effect of physical activity on sleep quality, well-being, and affect in academic stress periods,”  Nature and Science of Sleep, vol. Volume 9, pp. 117–126, 2017\.
  * [5] Matthias Zunhammer, Peter Eichhammer, and Volker Busch,  “Sleep quality during exam stress: The role of alcohol, caffeine and nicotine,”  PLoS ONE, vol. 9, no. 10, 2014.
  * [6] Rachel Campbell, Bart Soenens, Wim Beyers, and Maarten Vansteenkiste,  “University students’ sleep during an exam period: The role of basic psychological needs and stress,”  Motivation and Emotion, vol. 42, no. 5, pp. 671–681, 2018.
  * [7] Jerrell C. Cassady and Ronald E. Johnson,  “Cognitive test anxiety and academic performance,”  Contemporary Educational Psychology, vol. 27, no. 2, pp. 270–295, 2002.
  * [8] April L. Shapiro,  “Test anxiety among nursing students: A systematic review,”  Teaching and Learning in Nursing, vol. 9, no. 4, pp. 193–202, 2014\.
  * [9] Mark S. Chapell, Z. Benjamin Blanding, Michael E. Silverstein, Masami Takahashi, Brian Newman, Aaron Gubi, and Nicole McCann,  “Test anxiety and academic performance in undergraduate and graduate students.,”  Journal of Educational Psychology, vol. 97, no. 2, pp. 268–274, 2005.
  * [10] M. R. Amin, D. Wickramasuriya, and R. T. Faghih,  “A wearable exam stress dataset for predicting grades using physiological signals,”  2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), 2022.
  * [11] M. R. Amin, D. Wickramasuriya, and R. T. Faghih,  “A wearable exam stress dataset for predicting cognitive performance in real-world settings,”  PhysioNet, 2022.


# Effective Layer Pruning Through Similarity Metric Perspective

Ian Pons, Bruno Yamamoto, Anna H. Reali Costa and Artur Jordao
Escola Politécnica, Universidade de São Paulo

###### Abstract

Deep neural networks have been the predominant paradigm in machine learning
for solving cognitive tasks. Such models, however, are restricted by a high
computational overhead, limiting their applicability and hindering
advancements in the field. Extensive research demonstrated that pruning
structures from these models is a straightforward approach to reducing network
complexity. In this direction, most efforts focus on removing weights or
filters. Studies have also been devoted to layer pruning as it promotes
superior computational gains. However, layer pruning often hurts the network
predictive ability (i.e., accuracy) at high compression rates. This work
introduces an effective layer-pruning strategy that meets all underlying
properties pursued by pruning methods. Our method estimates the relative
importance of a layer using the Centered Kernel Alignment (CKA) metric,
employed to measure the similarity between the representations of the unpruned
model and a candidate layer for pruning. We confirm the effectiveness of our
method on standard architectures and benchmarks, in which it outperforms
existing layer-pruning strategies and other state-of-the-art pruning
techniques. Particularly, we remove more than $75\%$ of computation while
improving predictive ability. At higher compression regimes, our method
exhibits negligible accuracy drop, while other methods notably deteriorate
model accuracy. Apart from these benefits, our pruned models exhibit
robustness to adversarial and out-of-distribution samples.

##  1 Introduction

It is well known that deep neural networks are capable of obtaining remarkable
results in various cognitive fields, often outperforming humans from image
recognition to complex games such as chess and go [40, 14]. However, this
performance comes with high computational cost and storage demand. Advances in
the foundation model paradigm – large deep learning models trained on a broad
range of data with the capacity to transfer its knowledge to unseen
(downstream) tasks – have further intensified the resource-intensive nature of
the field, as large models play a crucial role in transferring knowledge to
downstream tasks [4, 2].

Figure 1: Comparison with state-of-the-art on the popular ResNet56 + CIFAR-10
setting. (Here, for illustration purposes, we abuse notation and bound the
ideal point close to two percentage points (pp); however, it may be higher).
Overall, our method obtains the best compromises between accuracy and
computational reduction (estimated by Floating Point Operations – FLOPs).
Specifically, our method dominates existing layer pruning methods (indicated
by symbol *) by a remarkable margin. Compared to state-of-the-art pruning
techniques, our method removes more than $75\%$ of FLOPs without hurting
accuracy (sometimes improving it). Other methods, however, degrade accuracy
when operating at these high FLOP reduction regimes. Since our method is
orthogonal to modern structured filter pruning, we can combine them to achieve
even higher computation gains (e.g., Ours + $\ell_{1}$-norm). The behavior
shown in this figure is consistent across other benchmarks and architectures.

To mitigate the above issues, compression techniques are becoming more popular
due to their positive results in improving the resource demand of deep models
[18, 47, 44]. Among the most promising techniques, pruning emerges as a
straightforward approach capable of enhancing different performance metrics
such as Floating Point Operations (FLOPs), memory consumption and number of
parameters. At the heart of a pruning technique lies the task of accurately
estimating the importance of structures that compose a model and subsequently
removing the least important ones.

Studies classify pruning into unstructured and structured categories [18, 47,
44]. The former removes individual connections (weights), while the latter
focuses on eliminating entire structures such as filters and layers. Despite
achieving high compression rates, unstructured approaches promote theoretical
inference speed-up only, requiring specific hardware to handle sparse matrix
computations to obtain practical speed-up [44, 47]. On the other hand,
structured pruning facilitates practical acceleration without any
hardware/software constraints. Furthermore, its advantages extend beyond
computational gains: it acts as a regularization mechanism that improves
generalization and robustness [52, 3].

Most efforts on structured pruning strategies focus on eliminating small
structures and are often optimized for standard metrics such as FLOP and
parameter reduction [18, 47]. Unfortunately, recent studies suggest that these
metrics may correlate weakly with inference time [39, 7, 42]. Nevertheless,
layer pruning reduces network depth, which directly addresses model latency
while also providing all the benefits of filter pruning, such as FLOP and
parameter reduction, without specialized software or hardware [25, 49, 54].
The idea behind pruning layers is not novel and dates back to 2016 [43, 21].
Efforts in this direction, however, either apply simple filter criteria and
combine (e.g., average) the scores to compose the importance of a layer [25,
49] or solve a (computationally expensive) multi objective optimization [54].
Therefore, one of the challenges in layer removal is developing a criterion
capable of accurately ranking the importance of all layers. It turns out that
existing criteria operate well on small structures but may be inadequate when
applied to large ones, primarily because of varying magnitudes (i.e.,
$\ell_{1}$-norm and its variations) exhibited by layers [49, 24].
Additionally, recent studies highlight that different layers play a distinct
role in the expressive power and training dynamics of deep models [49, 34, 6].
Such factors suggest that simple criteria are unable to characterize all these
underlying properties exhibited by layers. Lastly, a layer-pruning method must
inherit a fundamental requirement of pruning techniques: remove structures
without significantly compromising predictive ability.

To meet the aforementioned requirements and achieve computational-friendly
models, we propose a novel layer pruning method. Our method relies on the
hypothesis that similar representations between a dense (unpruned) network and
its optimal sparse (pruned) candidate indicate lower relative importance.
Existing evidence support this idea, revealing that layers share similar
representations [49, 34]. By eliminating unimportant layers, we can preserve
predictive capability and reduce computational demand.

For this purpose, we employ Centered Kernel Alignment (CKA) due to its
effectiveness and flexibility in measuring similarity between two networks
[26, 35, 36, 34]. Leveraging CKA, the overall of our method is the following.
Given a dense network, we first extract its representation from some input
examples. Here, representation refers to the feature maps of the layer just
before the classification layer. Then, we create a temporary pruned model by
removing a candidate layer. Building upon previous works [43, 49], at this
step, we avoid any fine-tuning or parameter adjustment, since modern
architectures are robust to single layer removal and perturbations. Afterward,
for each temporary model ($n$ layers, $n$ candidates), we compare their
representations with the original network using CKA. Finally, we select the
temporary network that yields the highest similarity, thus removing the
unimportant layer.

Contributions. We highlight the following key contributions. First, we propose
a novel pruning criterion that leverages an effective similarity
representation metric: CKA. To the best of our knowledge, we are the first to
explore CKA as a pruning criterion, as previous works widely employ it for
comparing network representations. Powered by this criterion, we develop a
layer-pruning method that removes entire layers from neural networks without
compromising predictive ability. Such a result is possible since our criterion
identifies unimportant layers – layers that, when removed, preserve similarity
regarding the original model. In particular, we aggressively prune
ResNet56/110 and ResNet50, encouraging them to converge towards ResNet20 in
terms of depth and computational cost but preserving their original accuracy.
Second, unlike most existing layer-pruning criteria that fail to capture
underlying properties of layers, our method effectively assigns layer
importance and thus prevents model collapse. Besides, it is efficient and
scales linearly as a function of the network depth. Third, we outperform
state-of-the-art pruning methods by a notable margin (see Figure 1). We
believe our results open new opportunities to prune through the lens of
emerging similarities metrics [46, 9] and encourage further efforts on layer
pruning.

Through extensive experiments on standard architectures and benchmarks, we
demonstrate that our method outperforms state-of-the-art pruning approaches.
Specifically, it surpasses existing layer-pruning strategies by a large
margin. In particular, as we increase the levels of FLOP reduction, most
layer-pruning methods fail to preserve accuracy, even when equipped with
additional techniques such as knowledge distillation [5, 54]. Our method, on
the other hand, successfully maintains accuracy while eliminating more than
$75\%$ of FLOPs. At a FLOP reduction exceeding $80\%$, our method exhibits
negligible accuracy drop, whereas other state-of-the-art techniques are unable
to achieve similar performance without compromising accuracy roughly $2\times$
more. We also demonstrate that our method preserves generalization in out-of-
distribution and adversarial robustness scenarios, which is crucial for
deploying pruned models in security-critical applications such as autonomous
driving. In terms of Green AI [27, 41, 11], our method reduces the carbon
emissions required in the training/fine-tuning phase by up to $80.85\%$,
representing an important step towards sustainable AI. Code available at:
github.com/IanPons/CKA-Layer-Pruning

##  2 Related Work

The main (and most challenging) task of pruning is to estimate the relative
importance of a given structure to differentiate between those essential for
predictive ability and the less important ones. A popular criterion focuses on
the magnitude of weights, namely $\ell_{p}$-norm. Researchers extensively
explore these criteria in the context of the lottery ticket hypothesis and
pruning at initialization [44]. Despite their simplicity, previous works
pointed out pitfalls in these criteria [49, 22, 17]. For example, Huang et al.
[22] argued that constraining the analysis to surrounding structures, as
$\ell_{1}$-norm does, incurs a low variance of importance scores, hindering
unimportant structure search. Furthermore, comparing norms across layers
becomes impractical, as different layers exhibit distinct magnitudes, posing a
challenge for global pruning (i.e., ranking all structures at once) [49, 24].
These issues have motivated efforts towards more elaborate criteria [39].
Taking the work by Lin et al. [31] as an example, the authors proposed
estimating filter importance based on the rank of its feature maps. Pruning
strategies that leverage information from feature maps (thus involving data
forwarding through the network) are named _data-driven_ techniques. Since we
measure similarity from feature maps, our method belongs to this category of
pruning.

Shen et al. [39] measure filter importance based on the Taylor expansion of
the loss change. Importantly, they highlighted the relevance of focusing on
latency instead of standard metrics such as FLOPs. To tackle this challenge,
the authors transformed the objective of maximizing accuracy within a given
latency budget into a resource allocation optimization problem, then solved it
using the Knapsack paradigm. In an alternative line of research, studies have
demonstrated that standard performance metrics may correlate weakly with
inference time [7, 42]. Aligned with these efforts, we demonstrate that our
method achieves notable latency improvements and other computational benefits.
Differently from Shen et al. [39], we address the accuracy/latency trade-off
without solving any optimization problem. This is possible because layer
pruning reduces network depth, directly translating into latency improvement,
and our criterion accurately identifies unimportant layers that preserve
accuracy, enabling us to achieve higher FLOP reductions while maintaining
accuracy simultaneously.

According to existing works [47, 18], most efforts have been devoted to filter
pruning techniques. In contrast to this family of methods that may exhibit
bias toward specific metrics like FLOPs or parameters [7, 42], layer pruning
achieves performance gains across all computational metrics [5, 25, 54]. In
this direction, Chen et al. [5] proposed learning classifiers using features
from prunable layers to assign their importance. Following this modeling,
layer importance relies on the performance of classifiers. Similar to ours,
the criterion by Chen et al. [5] is layer-specific; however, our criterion
focuses on similarity representations through CKA, which we reveal to be more
effective. More recently, the work by Zhang and Liu [50] disconnects residual
mapping and estimates its effect using Taylor expansion. Zhou et al. [54]
proposed an evolutionary-based approach, using the weights distribution as one
of the inputs for creating the initial population of candidate pruned
networks. It is worth mentioning that the methods by Zhou et al. [54] and Chen
et al. [5] require knowledge distillation to recover accuracy from the pruned
models, while our method relies on straightforward fine-tuning rounds. Such
observations suggest that our CKA criterion is more precise than the
previously mentioned strategies for selecting layers.

Apart from pruning, efforts have also been devoted to understanding the role
of layers in the expressive power and training dynamics of the models [49, 34,
6]. For example, Masarczyk et al. [34] suggested that the layers of deep
networks split into two distinct groups. The initial layers have linearly
separable representations, and the subsequent layers, or the tunnel, have less
impact on the performance, compressing the already learned representations.
This behavior, named _Tunnel Effect_ , emerges at the early stages of the
training process and corroborates with the notion of redundancy in
overparameterized models. Additionally, their work argued that the tunnel is
responsible for the performance degradation in out-of-distribution (OOD)
samples. We show that our layer pruning method preserves OOD generalization,
indicating that its degradation is not restricted to tunnel layers. In
summary, we believe our work contributes to these efforts by demonstrating
that unimportant layers can be effectively identified and removed without
compromising the expressive power of the model and its training dynamics.

##  3 Preliminaries and Proposed Method

Problem Statement. According to previous works [43, 21, 8], residual-based
architectures enable the information flow (i.e., the representation) to take
different paths through the network. Thereby, layers may not always strongly
depend on each other, reinforcing the idea of redundancy in this type of
structure, which suggests the possibility of removing layers without
compromising the network representation. It follows that a subset of layers
plays a crucial role in the network performance [49, 34, 6]. Upon this
evidence, our problem becomes identifying and removing unimportant layers,
preserving the representation capacity of the model, and avoiding network
collapse. Formally, given a network $\mathcal{N}$ composed of a layer set $L$,
our goal is to remove certain layers to produce a shallower network
$\mathcal{N^{\prime}}$ composed by $L^{\prime}$, where $|L^{\prime}|\ll|L|$
and the accuracy of $\mathcal{N^{\prime}}$ is as close as possible (ideally
better) than its unpruned version $\mathcal{N}$.

Naively, one could estimate optimal layers to prune by iterating over all
possible candidates, removing one at a time, fine-tuning the model, and
selecting the candidate that exhibits the lowest performance degradation.
However, this approach becomes computationally expensive as the network depth
increases, hence it is unfeasible for most modern architectures and large
scale datasets.

Following previous layer-pruning works [5, 54, 50], we indeed remove building
blocks (set of layers) instead of just individual layers. Throughout the text,
however, we opt to use the term _layer pruning_ rather than _block pruning_ to
maintain simplicity and clarity. We provide technical details involving layer
pruning in Appendix 6.1.

Definitions. Consider $X$ and $Y$ a set of training samples (e.g., images) and
their respective class labels. Let $\mathcal{N}$ be a dense (unpruned) network
trained using $X$ and $Y$ (i.e., the traditional supervised paradigm).
Consider $M(\cdot,X)$ as a function that extracts the representation of a
network from the samples $X$. Following Xu et al. [10], $M$ extracts the
feature maps from the layer immediately preceding the classification layer of
the network. It is worth mentioning that $M$ does not take into account the
labels $Y$. Let $l_{i}\in L$ be the candidate layers (i.e., layers the pruning
can eliminate) and, finally, define $\mathcal{N}_{l_{i}}$ as a pruned
candidate network yielded by removing the layer $l_{i}$ from $\mathcal{N}$.

Proposed Criterion. For each $l_{i}\in L$, we obtain $\mathcal{N}_{l_{i}}$
w.r.t the previous definition, and apply $M(\mathcal{N}_{l_{i}},X)$ to extract
its representation, denoted by $R_{l_{i}}$. Define $s(\cdot,\cdot)$ as our CKA
criterion which takes $R$ and $R_{l_{i}}$, where $R\leftarrow
M(\mathcal{N},X)$ (i.e., the original representation), and outputs the score
(importance) of $l_{i}$. Following Kornblith et al. [26], we compute CKA in
terms of

$\displaystyle
CKA(R,R_{l_{i}})=\frac{HSIC(R,R_{l_{i}})}{\sqrt{HSIC(R,R)HSIC(R_{l_{i}},R_{l_{i}})}},$
(1)

where HSIC is the Hilbert-Schmidt Independence Criterion [12]. Due to space
constraints, we refer interested readers to the works by Kornblith et al. [26]
and Nguyen et al. [35, 36] for additional information.

It follows from Equation 1 that $CKA(R,R_{l_{i}})$ $\in$ $[0,1]$, where a
value of $1$ indicates identical feature maps (i.e., the highest similarity
preservation). However, an intuitive practice is to remove the lowest-scoring
candidate layer. Therefore, we adjust the score in terms of
$s(R,R_{l_{i}})=1-CKA(R,R_{l_{i}})$, ensuring that lower scores are assigned
to layers yielding more similar representations.

Algorithm 1 Layer Pruning using our CKA criterion

0: Input: Trained Neural Network $\mathcal{N}$, Candidate Layers $l_{i}\in L$
Training samples $X$

0: Output: Pruned Version of $\mathcal{N}$

1: $R\leftarrow M(\mathcal{N},X)$

2: for $i\leftarrow 1$ to $|L|$ do

3: $\mathcal{N}_{l_{i}}\leftarrow\mathcal{N}\setminus l_{i}$ $\triangleright$
Removes layer $l_{i}$ from $\mathcal{N}$

4: $R_{i}\leftarrow M(\mathcal{N}_{l_{i}},X)$ $\triangleright$ Representation
extraction of $\mathcal{N}_{l_{i}}$

5: $S\leftarrow S\cup s(R,R_{l_{i}})$

6: end for

7: $j\leftarrow argmin(S)$ $\triangleright$ Index of lowest score in $S$

8: $\mathcal{N}\leftarrow\mathcal{N}_{l_{j}}$ $\triangleright$ $\mathcal{N}$
becomes its pruned version

9: Update $\mathcal{N}$ via standard fine-tuning on $X$

Algorithm 1 summarizes the process above. From it, we highlight the following
points. First, after estimating the importance of all candidate layers, we
indeed remove the lowest scoring one. Second, representation extractions
employ the same set $X$. Finally, the construction of $N_{l_{i}}$ does not
involve any fine-tuning.

A commonly explored approach in prior studies involves iteratively repeating
the pruning and fine-tuning process. We follow such practice by iterating
Algorithm 1, where the input to the next iteration is the fine-tuned
$\mathcal{N}$ (see line 8 in Algorithm 1). We report the iteration number
(hence, the number of removed layers) as a subscript; for example, CKAi means
we perform $i$ iterations to obtain the corresponding pruned model. Note that,
for a single iteration, our method scales linearly w.r.t the number of layers,
implying an $O(|L|)$ complexity.

##  4 Experiments

Experimental Setup. We conduct experiments on CIFAR-10, CIFAR-100 and ImageNet
using different versions of the ResNet architecture [16]. Such settings are a
common choice for general compression/acceleration studies [5, 25, 49, 54,
18]. Throughout both training and fine-tuning phases, we apply random crop and
horizontal random flip as data augmentation [16]. We choose this simple setup
to highlight the genuine advantages of our method. In Appendices 6.3 and 6.4,
we also conduct experiments on MobileNetV2 and Transformer architectures.

To compare the predictive ability of the unpruned models with their pruned
counterparts, we follow common practices and report the difference between
accuracies [18]. In this metric, negative and positive values indicate a
decrease and an improvement in accuracy (in percentage points – pp),
respectively.

The Effect of Layer Pruning on Efficiency. Our point of start is illustrating
the advantages of layer over filter pruning, as the latter is the most popular
family of methods that yield gains without requiring specific hardware [39,
39, 18].

According to recent studies [7, 42], standard metrics such as FLOPs and
parameters, when singly employed, may overlook model efficiency. Therefore, we
begin our discussion by considering latency – the time for forwarding a sample
(or a set) through the network. To do so, we follow the same process as Jordao
et al. [24], which creates two pruned networks: one obtained through layer
removal and the other from filters, aiming for both models to have a similar
number of neurons (filters). This procedure makes possible a fair comparison
in terms of latency performance.

Iteratively repeating this process yields models with varying numbers of
filters removed, from which we measure their average latency across $30$ runs
by forwarding $10K$ samples and report the speed-up obtained from the pruning
process with respect to the original (unpruned) model.

Figure 2: Relationship between the number of filters removed (x-axis) and
latency speed-up (y-axis) for models obtained from filter and layer pruning.
Importantly, such a comparison is possible because when pruning removes
layers, it eliminates all filters from that layer. Left and right plots stand
for ResNet56 and ResNet110, respectively. Overall, layer pruning notably
promotes higher speed-up than filter pruning.

Figure 2 shows the results. It follows that layer pruning yields a higher
speed-up than filter removal. For example, in ResNet110, with both methods
eliminating around a thousand filters, layer pruning achieves an $11$ pp
speedup over filter pruning. This advantage persists even when removing
approximately $1,600$ filters, underscoring the effectiveness of removing
layers for network acceleration. Such gains have motivated previous efforts on
layer removal [25, 50, 54].

Effectiveness of the Proposed CKA Criterion. Informed by the previous
findings, we now turn our attention to evaluating the effectiveness of the
proposed CKA criterion in assigning layer importance. For this purpose, we
take into account representative layer pruning techniques [25, 49, 54, 5]. It
is worth mentioning that we exclude works on dynamic inference since they
belong to a different category of compression and acceleration techniques
[15].

Table 1 summarizes the results. According to this table, our method
outperforms existing techniques by a large margin. On ResNet56, compared to
the best strategy in terms of delta in accuracy, LPSR [50], our method
outperforms it by more than $0.76$ pp while exhibiting better FLOP gains.
Regarding FLOP reduction, the best method underperforms ours by $3.4$ pp. This
behavior is prevalent in ResNet110 and ResNet50 (on ImageNet). Notably, we
reduce around $2\times$ more FLOPs than other criteria while obtaining an
improvement in accuracy.

Table 1: Comparison with state-of-the-art layer-pruning methods. The symbols (+) and (- ) denote increase and decrease in accuracy regarding the original (unpruned) network, respectively. We highlight the best results in bold. | Method |  $\Delta$ Acc. | FLOPs (%)
---|---|---|---
ResNet56 on CIFAR10 |  PLS [25] | (–) 0.98 | 30.00
FRPP [5] | (+) 0.26 | 34.80
ESNB [54] | (–) 0.62 | 52.60
LPSR [50] | (+) 0.19 | 52.75
CKA15 (ours) | (+) 0.95 | 56.29
PLS [25] | (–) 0.91 | 62.69
LPSR [50] | (–) 0.87 | 71.65
CKA19 (ours) | (+) 0.16 | 71.30
CKA20 (ours) | (+) 0.08 | 75.05
ResNet110 on CIFAR10 |  ESNB [54] | (+) 1.15 | 29.89
PLS [25] | (+) 0.06 | 37.73
CKA27 (Ours) | (+) 1.16 | 50.33
CKA36 (Ours) | (+) 0.80 | 67.10
ResNet50 on ImageNet |  LPSR [50] | (-) 1.38 | 37.38
CKA6 (Ours) | (-) 0.18 | 39.62
PLS [25] | (-) 0.67 | 45.28
CKA7 (Ours) | (-) 0.90 | 45.28

The reason for these remarkable results is that our method carefully selects
which layers to eliminate. For example, Jordao et al. [25] and Zang et al.
[50] compute scores for layers by aggregating the sum of scores from the
individual filters that compose a layer. Table 1 suggests that this
aggregating scheme may be inappropriate. This finding concurs with the
observations made by Masarczyk et al. [34], where the authors argued that
aggregating all features of a layer to compose its final representation is
suboptimal, particularly for transfer learning.

In terms of computational cost, compared to Zhou et al. [54], our method is
more cost-friendly. It turns out that this approach solves the score
assignment problem through an evolutionary algorithm. Therefore, their method
scales expensively as the depth (i.e., $|L|$) increases. On the other hand, to
prune a model with $|L|$ layers our approach requires $|L|$ forwards and CKA
comparisons, scaling linearly (see Algorithm 1). The method by Jordao et al.
[25] is also linear w.r.t the number of layers, however, it is unable to prune
a layer from any region of the network. Specifically, to eliminate a layer
$i$, their method requires the removal of all subsequent layers $j$ where
$i<j<|L|$.

The previous evidence corroborates the suitability of our criterion for
selecting unimportant layers compared to existing state-of-the-art layer
pruning methods. Importantly, the discussion above confirms our hypothesis
that similar representations between a dense (unpruned) network and its
optimal pruning candidate indicate lower relative importance.

Comparison with the State of the Art. The previous experiments shed light on
the benefits of layer pruning and the effectiveness of our criterion for
selecting layers to remove. We now compare our method with general state-of-
the-art pruning techniques. For this purpose, we evaluate our method against
the most recent and top-performing techniques mainly based on the survey by He
et al. [18]. More specifically, we consider methods capable of achieving
notable FLOP reduction with negligible accuracy drop. For a fair comparison,
we report the results of each method according to the original paper.

Table 2: Comparison with state-of-the-art pruning methods on CIFAR-10. For each level of FLOP reduction ($\%$), we highlight the best results in bold. | Method |  $\Delta$ Acc. | FLOPs (%)
---|---|---|---
ResNet56 |  DECORE [1] (CVPR, 2022) | +0.08 | 26.30
HALP [39] (NeurIPS, 2022) | +0.03 | 33.72
SOKS [32] (TNNLS, 2023) | +0.16 | 35.91
CKA10 (ours) | +1.25 | 37.52
GKP-TMI [53] (ICLR, 2022) | +0.22 | 43.23
GCNP [23] (IJCAI, 2022) | +0.13 | 48.31
CKA13 (ours) | +0.86 | 48.78
GNN-RL [48] (ICML, 2022) | +0.10 | 54.00
RL-MCTS [45] (WACV, 2022) | +0.36 | 55.00
WhiteBox [51] (TNNLS, 2023) | +0.28 | 55.60
CLR-RNF [29] (TNNLS, 2023) | \+ 0.01 | 57.30
CKA16 (ours) | +0.78 | 60.04
DAIS [13] (TNNLS, 2023) | -1.00 | 70.90
HRank [31] (CVPR, 2020) | -2.54 | 74.09
GCNP [23] (IJCAI, 2022) | -0.97 | 77.22
CKA20 (ours) | +0.08 | 75.05
CKA21 (ours) | -0.66 | 78.80
DECORE [1] (CVPR, 2022) | -2.41 | 81.50
CKA21 (ours) + $\ell_{1}$ | -0.94 | 84.70
ResNet110 |  DECORE [1] (CVPR, 2022) | +0.38 | 35.43
HRank [31] (CVPR, 2020) | +0.73 | 41.20
GKP-TMI [53] (ICLR, 2022) | +0.64 | 43.31
CKA24 (ours) | +1.37 | 44.73
DAIS [13] (TNNLS, 2023) | -0.60 | 60.00
DECORE [1] (CVPR, 2022) | 0.00 | 61.78
CKA35 (ours) | +0.89 | 65.23
EPruner [30] (TNNLS, 2022) | +0.12 | 65.91
CRL-RNF [29] (TNNLS, 2023) | +0.14 | 66.00
WhiteBox [51] (TNNLS, 2023) | +0.62 | 66.00
CCEP [38] (IJCAI, 2022) | -0.22 | 67.09
CKA36 (ours) | +0.80 | 67.10
HRank [31] (CVPR, 2020) | -0.85 | 68.64
CKA38 (ours) | +0.59 | 70.83
DECORE [1] (CVPR, 2022) | -0.79 | 76.92
CKA41 (ours) | +0.23 | 76.42
CKA47 (ours) | -0.41 | 87.61

Table 2 shows the results on CIFAR-10 for ResNet56/110. On these
architectures, our method outperforms state-of-the-art techniques by removing
more FLOPs and achieving the best delta in accuracy. For example, in Table 2
(left), within comparable FLOP reduction regimes, we outperform state-of-the-
art methods by a margin starting at approximately 0.4 pp (compared to RL-MCTS
[45]) and reaching up to more than 2.5 pp (compared to HRank [31]).

Table 2 poses an interesting behavior: at high FLOP reduction levels (i.e.,
above $70\%$), all methods fail to preserve accuracy. In contrast, our method
removes more than $75\%$ of FLOPs with no accuracy drop. Most cases, our
method promotes predictive ability improvements. This benefit is expected, as
layer pruning (and its variations) acts as a form of regularization [21, 15].
Table 2 highlights this behavior in other pruning techniques, but unlike ours,
exhibited only in low compression regimes.

As we mentioned before, our method is orthogonal to other pruning categories
(i.e., the ones in Table 2); therefore, we can combine it with these
techniques. Built upon previous ideas [25], we take one of our pruned models
and further prune it using the popular $\ell_{1}$-norm filter pruning. In this
scenario, we achieve even better results, surpassing our best performance
gains (using layer pruning only) in terms of FLOP reduction by 5.9 pp.
Specifically, our method achieves a FLOP reduction above $80\%$ while
maintaining the accuracy drop below one pp. The single method paired with this
level of reduction, DECORE [1], exhibits an accuracy degradation of $2.41$ pp
compared to the original model.

Table 3: Comparison with state-of-the-art pruning methods on ImageNet using ResNet50 and CIFAR-100 using ResNet56. For each level of FLOP reduction ($\%$), we highlight the best results in bold. $\Delta$ Acc. on ImageNet considers Top1 accuracy. | Method |  $\Delta$ Acc. | FLOPs (%)
---|---|---|---
ResNet50 on ImageNet |  DECORE [1] (CVPR, 2022) | (+) 0.16 | 13.45
SOSP [37] (ICLR, 2022) | (+) 0.41 | 21.00
GKP-TMI [53] (ICLR, 2022) | (-) 0.19 | 22.50
CKA3 (Ours) | (+) 1.11 | 22.64
SOSP [37] (ICLR, 2022) | (+) 0.45 | 28.00
CKA4 (Ours) | (+) 0.74 | 28.30
GKP-TMI [53] (ICLR, 2022) | (-) 0.62 | 33.74
LPSR [50] (SPL, 2022) | (-) 0.57 | 37.38
CKA6 (Ours) | (-) 0.18 | 39.62
CLR-RNF [29] (TNNLS, 2023) | (-) 1.16 | 40.39
DECORE [1] (CVPR, 2022) | (-) 1.57 | 42.30
HRank [31] (CVPR, 2020) | (-) 1.17 | 43.77
SOSP [37] (ICLR, 2022) | (-) 0.94 | 45.00
WhiteBox [51] (TNNLS, 2023) | (-) 0.83 | 45.60
CKA7 (Ours) | (-) 0.90 | 45.28
DECORE [1] (CVPR, 2022) | (-) 4.09 | 60.88
CKA9 (Ours) + $\ell_{1}$-norm | (-) 5.15 | 62.00
ResNet56 on CIFAR-100 |  DLRFC [19] (ECCV, 2022) | (+) 0.27 | 25.50
FRPP [5] (TPAMI, 2019) | (-) 0.23 | 38.30
GCNP [23] (IJCAI, 2022) | 0.00 | 48.77
EKG [28] (ECCV, 2022) | (+) 0.31 | 50.00
GCNP [23] (IJCAI, 2022) | (-) 0.64 | 52.22
LPSR [50] (SPL, 2022) | (-) 1.22 | 52.68
DAIS [13] (TNNLS, 2023) | (+) 0.81 | 53.60
CKA17 (ours) | (+) 0.71 | 63.79
CKA19 (ours) | (-) 0.59 | 71.29
CKA20 (ours) | (-) 1.96 | 75.05

We also evaluate our method on ImageNet and CIFAR-100 in Table 3. On these
datasets, we observe a similar trend with the CIFAR-10 discussion when
comparing our method against state-of-the-art pruning techniques.
Particularly, on ImageNet, the layer-pruning approach by Zhang et al. [50],
LPSR, notably hurts the accuracy, whereas our method is capable of improving
it while removing more FLOPs. It is important to mention that, due to
technical details (see Appendix 6.1), layer-pruning methods are restricted to
a set of candidate layers. In our experiments, we reach the limit of layer
removal, reported in Table 3 (left). Therefore, we combine our method with the
$\ell_{1}$-norm criterion to further prune our model in terms of filters,
achieving a higher computational reduction of $62.00\%$. It is important to
note that more elaborate combinations could minimize the accuracy drop, but we
leave this exploration for future work.

Effectiveness in Shallow Architectures. Among the keys to the success of
pruning is the overparameterized regime of neural networks, particularly
evident in deep models. Although our method is suitable in such cases, this
experiment verifies its applicability to shallow models (i.e., ResNet32/44).
Due to space constraints, we refer interested readers to Appendix 6.3 for
detailed results.

On these architectures, our results align with previous experiments on deeper
models (e.g., ResNet56/110), obtaining a satisfactory performance.
Specifically, we remove $54.60\%$ and $62.95\%$ of FLOPs on ResNet32/44,
respectively, without compromising accuracy. Beyond these levels of FLOP
reduction, we observe a slight drop in accuracy, although it remains
negligible (below $0.3$ pp). Importantly, this behavior only occurs with deep
networks when the FLOP reduction is above $70\%$. We also compare the
performance of our layer-pruning method against state-of-the-art methods in
these shallow architectures. Our method achieves competitive results,
outperforming existing methods in most cases. We also evaluate our method on
the MobileNetV2 and Transformer (see Appendices 6.3 and 6.4) architectures. On
these architectures, we observe the same trend: our method removes notable
FLOPs without hurting predictive ability.

Overall, the previous discussion confirms that our method is effective for
both deep and shallow networks, as well as for other modern architectures.

Robustness to Adversarial Samples. Previous research have demonstrated the
potential of pruning as a successful defense mechanism against adversarial
attacks and out-of-distribution examples [3]. In particular, evaluating pruned
models in adversarial scenarios plays a critical role, as we need to guarantee
the trustworthiness of these models before deploying them in real-world
applications such as autonomous driving. To assess the adversarial robustness
of the pruned models, we employ CIFAR-C [20], ImageNet-C [20] and CIFAR-10.2
[33].

On these datasets, our pruned models obtained superior robustness compared to
the unpruned model. We notice a similar trend when evaluating the pruned
models against the FGSM attack (see Figure 4 in supplementary material).

The previous discussion indicates that our method operated as a defense
mechanism against adversarial attacks. Such a finding is unsurprising, as
previous works have demonstrated the potential of pruning as a defense
mechanism against adversarial attacks [24]. Conversely, Bair et al. [3] argued
that pruning networks hurt generalization on OOD. We observe that the pruned
networks yielded by our method preserve OOD generalization. In this direction,
Masarczyk et al. [34] observed that some layers hinder OOD generalization.
According to this experiment, our method mitigates this problem, even though
it was not specifically designed for this purpose. Further analysis is
required in this context, such as considering more attacks, but we leave it
for future research.

GreenAI and Transfer Learning. The current consensus is that deeper models
yield better predictive ability. A consequence of this paradigm is the
computational overhead seen in modern architectures, contributing to an
increase in energy demands, both in the training and deployment phases.
According to previous works [27, 41, 11], these demands result in high carbon
emissions (CO2). Fortunately, the benefits in FLOP reduction and latency
promoted by our method directly translate into a reduction of CO2. For
example, our best-pruned version of ResNet56 implies a reduction of CO2 by
$67.88\%$ during the fine-tuning. On ResNet110, our pruned model at the
highest FLOP reduction regime leads to $80.85\%$ of CO2 reduction. We can
further evidence this practical reduction in transfer learning scenarios,
where fine-tuning the models is necessary for downstream tasks. To do so, we
employ the pruned versions of ResNet56 on CIFAR-100 and transfer their
knowledge to CIFAR-10. Interestingly, we observe that our pruned model with
the highest FLOP reduction achieved a CO2 reduction of $68.23\%$ while
maintaining accuracy within $1$ pp compared to the unpruned model. In
addition, pruned models with lower FLOP reductions achieve better transfer
learning results, corroborating the findings by Xu et al. [47]. Such behavior
suggests a challenge for the current evaluation of pruning techniques: the
quality of pruning should consider its performance in transfer learning tasks.

We believe the results above pose an important step towards Green AI.
Particularly on the learning paradigm involving foundation models, as the
success of this emerging field relies on transfer-learning (and self-
supervised), hence, requiring fine-tuning [4, 10, 2].

##  5 Conclusions

Layer pruning emerges as an exciting compression and acceleration technique
due to more pronounced benefits in FLOP reduction and latency speed-up than
other forms of pruning. Despite achieving promising results, existing criteria
for selecting layers fail to fully characterize the underlying properties of
these structures. To mitigate this, we proposed a novel criterion for
identifying unimportant layers. Our method leverages the Centered Kernel
Alignment (CKA) similarity metric to select such layers from a set of
candidates. Powered by CKA, we showed that similar representations between a
dense (unpruned) network and its optimal pruning candidate indicate lower
relative importance, thus capturing underlying properties exhibited by layers
and preventing model collapse. Extensive experiments on standard benchmarks
and architectures confirm the effectiveness of our method. Specifically, our
method outperforms existing layer-pruning techniques in terms of both accuracy
and FLOP reduction by a large margin. Compared to other state-of-the-art
pruning methods, we obtain the best compromise between accuracy and FLOP
reduction. Particularly, at high FLOP reduction levels all methods fail to
preserve accuracy, whereas our method exhibits either an improvement or
negligible drop. In addition, our pruned models exhibit robustness to
adversarial samples and positive out-of-distribution generation. Finally, our
work also poses an important step towards Green AI by reducing up to $80.85\%$
of carbon emissions required for training and fine-tuning modern
architectures.

## Acknowledgments

The authors would like to thank grant #2023/11163-0, São Paulo Research
Foundation (FAPESP), and grant #402734/2023-8, National Council for Scientific
and Technological Development (CNPq). Anna H. Reali Costa would like to thank
grant #312360/2023-1 CNPq.

## References

  * [1] Manoj Alwani, Yang Wang, and Vashisht Madhavan.  DECORE: deep compression with reinforcement learning.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [2] Xavier Amatriain, Ananth Sankar, Ananth Sankar, Praveen Kumar Bodigutla, Timothy J. Hazen, and Michaeel Kazi.  Transformer models: an introduction and catalog.  arXiv, 2023.
  * [3] Anna Bair, Hongxu Yin, Maying Shen, Pavlo Molchanov, and Jose M. Alvarez.  Adaptive sharpness-aware pruning for robust sparse networks.  In International Conference on Learning Representations (ICLR), 2024\.
  * [4] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, and et al.  On the opportunities and risks of foundation models.  ArXiv, 2021.
  * [5] Shi Chen and Qi Zhao.  Shallowing deep networks: Layer-wise pruning based on feature representations.  Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [6] Yixiong Chen, Alan L. Yuille, and Zongwei Zhou.  Which layer is learning faster? A systematic exploration of layer-wise convergence rate for deep neural networks.  In International Conference on Learning Representations (ICLR), 2023\.
  * [7] Mostafa Dehghani, Yi Tay, Anurag Arnab, Lucas Beyer, and Ashish Vaswani.  The efficiency misnomer.  In International Conference on Learning Representations (ICLR), 2022\.
  * [8] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas.  Attention is not all you need: pure attention loses rank doubly exponentially with depth.  In International Conference on Machine Learning (ICML), 2021.
  * [9] Lyndon R. Duong, Jingyang Zhou, Josue Nassar, Jules Berman, Jeroen Olieslagers, and Alex H. Williams.  Representational dissimilarity metric spaces for stochastic neural networks.  In International Conference on Learning Representations (ICLR), 2023\.
  * [10] Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C. Mozer.  Head2toe: Utilizing intermediate representations for better transfer learning.  In International Conference on Machine Learning (ICML), 2022.
  * [11] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang.  LLMCarbon: Modeling the end-to-end carbon footprint of large language models.  In International Conference on Learning Representations (ICLR), 2024\.
  * [12] Arthur Gretton, Olivier Bousquet, Alexander J. Smola, and Bernhard Schölkopf.  Measuring statistical dependence with hilbert-schmidt norms.  In Algorithmic Learning Theory (ALT), 2005.
  * [13] Yushuo Guan, Ning Liu, Pengyu Zhao, Zhengping Che, Kaigui Bian, Yanzhi Wang, and Jian Tang.  DAIS: automatic channel pruning via differentiable annealing indicator search.  Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [14] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, Z. Yang, Y. Zhang, and D. Tao.  A survey on vision transformer.  Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [15] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang.  Dynamic neural networks: A survey.  Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Computer Vision and Pattern Recognition (CVPR), 2016.
  * [17] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang.  Filter pruning via geometric median for deep convolutional neural networks acceleration.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [18] Yang He and Lingao Xiao.  Structured pruning for deep convolutional neural networks: A survey.  Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [19] Zhiqiang He, Yaguan Qian, Yuqi Wang, Bin Wang, Xiaohui Guan, Zhaoquan Gu, Xiang Ling, Shaoning Zeng, Haijiang Wang, and Wujie Zhou.  Filter pruning via feature discrimination in deep neural networks.  In European Conference on Computer Vision (ECCV), 2022.
  * [20] Dan Hendrycks and Thomas G. Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In International Conference on Learning Representations (ICLR), 2019\.
  * [21] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  In European Conference on Computer Vision (ECCV), 2016.
  * [22] Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo.  Rethinking the pruning criteria for convolutional neural network.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [23] Di Jiang, Yuan Cao, and Qiang Yang.  On the channel pruning using graph convolution network for convolutional neural network acceleration.  In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
  * [24] Artur Jordao, George Corrêa de Araújo, Helena de Almeida Maia, and Hélio Pedrini.  When layers play the lottery, all tickets win at initialization.  In International Conference on Computer Vision Workshops (ICCVW), 2023.
  * [25] Artur Jordao, Maiko Lie, and William Robson Schwartz.  Discriminative layer pruning for convolutional neural networks.  Journal of Selected Topics in Signal Processing, 2020.
  * [26] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton.  Similarity of neural network representations revisited.  In International Conference on Machine Learning (ICML), 2019.
  * [27] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres.  Quantifying the carbon emissions of machine learning.  In Neural Information Processing Systems (NeurIPS). 2019.
  * [28] Seunghyun Lee and Byung Cheol Song.  Ensemble knowledge guided sub-network search and fine-tuning for filter pruning.  In European Conference on Computer Vision (ECCV), 2022.
  * [29] Mingbao Lin, Liujuan Cao, Yuxin Zhang, Ling Shao, Chia-Wen Lin, and Rongrong Ji.  Pruning networks with cross-layer ranking & k-reciprocal nearest filters.  Transactions on Neural Networks and Learning Systems, 2023.
  * [30] Mingbao Lin, Rongrong Ji, Shaojie Li, Yan Wang, Yongjian Wu, Feiyue Huang, and Qixiang Ye.  Network pruning using adaptive exemplar filters.  Transactions on Neural Networks and Learning Systems, 2022.
  * [31] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao.  Hrank: Filter pruning using high-rank feature map.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [32] Guangzhe Liu, Ke Zhang, and Meibo Lv.  SOKS: automatic searching of the optimal kernel shapes for stripe-wise network pruning.  Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [33] Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt.  Harder or different? a closer look at distribution shift in dataset reproduction.  In International Conference on Machine Learning (ICML), 2020.
  * [34] Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miłoś, and Tomasz Trzcinski.  The tunnel effect: Building data representations in deep neural networks.  In Neural Information Processing Systems (NeurIPS), 2023.
  * [35] Thao Nguyen, Maithra Raghu, and Simon Kornblith.  Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth.  In International Conference on Learning Representations (ICLR), 2021\.
  * [36] Thao Nguyen, Maithra Raghu, and Simon Kornblith.  On the origins of the block structure phenomenon in neural network representations.  Transactions on Machine Learning Research, 2022.
  * [37] Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb.  SOSP: efficiently capturing global correlations by second-order structured pruning.  In International Conference on Learning Representations (ICRL), 2022\.
  * [38] Haopu Shang, Jia-Liang Wu, Wenjing Hong, and Chao Qian.  Neural network pruning by cooperative coevolution.  In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
  * [39] Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and José M. Álvarez.  Structural pruning via latency-saliency knapsack.  In Neural Information Processing Systems (NeurIPS), 2022.
  * [40] David Silver, Aja Huang, Chris J. Maddison, and et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 2016.
  * [41] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  In Conference of the Association for Computational Linguistics (ACL), 2019.
  * [42] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan.  Mobileone: An improved one millisecond mobile backbone.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [43] Andreas Veit, Michael J. Wilber, and Serge J. Belongie.  Residual networks behave like ensembles of relatively shallow networks.  In Neural Information Processing Systems (NeurIPS), 2016.
  * [44] Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu.  Recent advances on neural network pruning at initialization.  In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
  * [45] Zi Wang and Chengcheng Li.  Channel pruning via lookahead search guided reinforcement learning.  In Winter Conference on Applications of Computer Vision (WACV), 2022\.
  * [46] Alex H. Williams, Erin Kunz, Simon Kornblith, and Scott W. derman.  Generalized shape metrics on neural representations.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [47] Canwen Xu and Julian J. McAuley.  A survey on model compression and acceleration for pretrained language models.  In Conference on Artificial Intelligence (AAAI), 2023.
  * [48] Sixing Yu, Arya Mazaheri, and Ali Jannesari.  Topology-aware network pruning using multi-stage graph embedding and reinforcement learning.  In International Conference on Machine Learning (ICML), 2022.
  * [49] Chiyuan Zhang, Samy Bengio, and Yoram Singer.  Are all layers created equal?  Journal of Machine Learning Research, 2022.
  * [50] Ke Zhang and Guangzhe Liu.  Layer pruning for obtaining shallower resnets.  Signal Processing Letters, 2022.
  * [51] Yuxin Zhang, Mingbao Lin, Chia-Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, and Rongrong Ji.  Carrying out CNN channel pruning in a white box.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [52] Qi Zhao and Christian Wressnegger.  Holistic adversarially robust pruning.  In International Conference on Learning Representations (ICLR), 2023\.
  * [53] Shaochen Zhong, Guanqun Zhang, Ningjia Huang, and Shuai Xu.  Revisit kernel pruning with lottery regulated grouped convolutions.  In International Conference on Learning Representations (ICLR), 2022\.
  * [54] Yao Zhou, Gary G. Yen, and Zhang Yi.  Evolutionary shallowing deep neural networks at block levels.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.

##  6 Appendix

###  6.1 Technical Details Involving Layer Pruning

Let $\mathcal{N}$ be a network composed of $L$ layers. We can express the
output of $\mathcal{N}$ as a set of $|L|$ transformations
$f_{i}(.),i\in\\{1,...,|L|\\}$. For the sake of simplicity, each $f_{i}$
consists of a series of convolution, batch normalization, and activation
operations. In this definition, we obtain the network output ($y$) by
forwarding the input data, denoted by $X$, through the sequential layers $f$,
where the input of layer $i$ is the output of the previous layer $i-1$;
therefore, $y=f_{|L|}(...f_{2}(f_{1}(X)))$. This composes the idea behind
plain networks (i.e., VGG and AlexNet).

In residual-like networks, we can express the output $y_{i}$ of layer $i$ in
terms of the transformation $f_{i}$ and the output $y_{i-1}$ from the previous
layer (see Figure 3 top). Formally, we write:

$y_{i}=f_{i}(y_{i-1})+y_{i-1}.$ (2)

Equation 2 composes a residual module, where the rightmost part is named
_identity-mapping shortcut_ (or identity for short). From a theoretical
perspective, pruning the $i$-th layer corresponds to letting $y_{i}=y_{i-1}$.
From a technical perspective, pruning the $i$-th layer involves connecting the
output of layer $i-1$ to the input of layer $i+1$. Due to the residual nature
(skip connection), we could accomplish this by just zeroing out the weights of
$f_{i}(y_{i-1})$; thus, eliminating its contribution in Equation 2. However,
this process does not ensure practical speed-up without specialized hardware
for sparse computing. Instead, after selecting a victim layer (let’s say layer
$i$), we obtain a pruned model according to the following process. First, we
create a novel architecture without layer $i$, resulting in a model comprising
only the surviving layers (the pruned model). Then, from the old architecture
(unpruned model), we transfer the weights of the survival layers (we can apply
this idea to a set of layers at once) to this novel architecture. Figure 3
(bottom) illustrates this process.

From the process above, we highlight that the layer-pruning process removes
building blocks (i.e., a set of transformations, $f_{i}(.)$) a.k.a modules.
Therefore, the pruning also eliminates the corresponding activations and
normalization layers from the block. This is a common process in layer-pruning
techniques [50, 25, 54].

As a final note, the pruning process cannot remove all layers composing a
model due to incompatible dimensions between the input and output tensor
between stages (layers operating on representations in the same resolution).
In particular, the pruning is unable to remove layers between stages. Thus,
given an architecture of $k$ layers within a stage, the pruning can remove at
most $k-2$ layers within this stage. Importantly, this is the reason why we
are unable to remove all layers of a model. This analysis corresponds to
ResNet architecture and can vary depending on the architecture design of the
model.

Figure 3: Architecture of a residual-like network. Top. The rationale behind
this architecture is that the output of a layer takes into account the
transformation performed by it ($f$) plus ($\oplus$) the input ($y$) it
receives. Due to this essence, when we disable layer $i$ (its transformation –
dashed lines), the output (representation) of layer $i-1$ is propagated to
layer $i+1$, which means that the output $y_{i}$ belongs $y_{i-1}$. For the
sake of simplicity, we omit the batch normalization and activation layers,
which are also transferred in the process of layer removal. Bottom. Process to
eliminate a layer from a technical perspective and, thus, obtain practical
speed-up gains. After selecting the victim layer (i.e., Layer $i$), we create
a novel architecture without it and, then, transfer the weights (red dashed
arrows) of the corresponding survival layers.

###  6.2 Adversarial Attacks

Figure 4: Results of pruned models for different adversarial attacks. Green
and blue points correspond to an accuracy improvement and degradation,
respectively. Dotted lines separate the plots into improvement and degradation
groups. Top-Right: Results on out-of-distribution using CIFAR-10.2 [33]. Top-
Left: Results on adversarial robustness using CIFAR-C [20]. Bottom-Left:
Results on FGSM adversarial attack. Bottom-Right: Results on ImageNet-C using
pruned models from ResNet50

Efforts toward a deeper understanding of the roles played by layers in network
generalization and the effect of pruning in adversarial attack scenarios
reinforce the idea that improving OOD and adversarial robustness while
reducing computational demand is accomplishable for pruning methods [34, 3].
Such compromises are crucial for deploying these models in safety-critical
applications e.g., autonomous driving and robotics. In this experiment, we
demonstrate that our pruned models improve OOD generalization and adversarial
robustness. For this purpose, we compare the performance on different
adversarial tasks of ResNet56 and its pruned models using our method. Figure 4
shows the results. According to this figure, our pruned models obtain positive
robustness, highlighting their effectiveness even on severe compression rates.
Specifically, on CIFAR-10.2 [33] and CIFAR-C [20], we reduce more than 70% of
computational cost while exhibiting improvements. Notably, only three pruned
models obtained lower performance compared to the unpruned model, yet less
than one pp. In the FGSM attack, regardless of the FLOP reduction levels, our
pruned models dominate its unpruned version in terms of adversarial
robustness.

It is worth mentioning that during the pruning process, we avoid any defense
mechanism. Therefore, the preceding discussion confirms that the benefits of
our pruned models extend beyond computational gains.

###  6.3 Results on Shallow Architectures

In this experiment, we further explore the effectiveness of our method in
shallow architectures: ResNet32 and ResNet44. We also consider the lightweight
architecture MobileNetV2. Table 4 summarizes the results.

According to Table 4, our method removes up to $54.61\%$ and $62.95\%$ of
FLOPs from residual architectures without compromising model accuracy. On
higher FLOP reduction regimes, the performance drop is negligible (i.e., less
than one pp.). This behavior is consistent with our analysis considering
deeper models and supports the effectiveness of our method on shallow
architectures.

Finally, our pruned models outperform state-of-the-art pruning methods; the
only exception is the method by Nonnenmacher et al. [37] on ResNet32. It turns
out that our method reached the limit of layer removal (i.e., there are no
more available layers to remove, see Section 6.1). Therefore, to achieve
higher computational gains we should combine it with filter-pruning
strategies. As a final note, the reason few methods appear in the table is
that there is a lack of pruning studies reporting results on these
architectures [18].

Table 4: Comparison of state-of-the-art pruning methods on CIFAR-10 using ResNet32, ResNet44 and MobileNetV2. The symbols (+) and (-) denote increase and decrease in accuracy regarding the original (unpruned) network, respectively. For each level of FLOP reduction, we highlight the best results in bold. | Method |  $\Delta$ Acc. | FLOPs (%)
---|---|---|---
ResNet32 |  GKP-TMI [53] (ICLR, 2022) | (+)0.22 | 43.10
SOKS [32] (TNNLS, 2023) | (-) 0.38 | 46.85
CKA (ours) | (+) 0.68 | 47.78
DAIS [13] (TNNLS, 2023) | (+) 0.57 | 53.90
SOKS [32] (TNNLS, 2023) | (-) 0.80 | 54.58
CKA (ours) | (+) 0.05 | 54.61
SOSP [37] (ICLR, 2022) | (-) 0.24 | 67.36
CKA (ours) | (-) 0.18 | 61.44
ResNet44 | DCP-CAC (TNNLS, 2022) | (-) 0.03 | 50.04
AGMC (ICCV, 2021) | (-) 0.82 | 50.00
CKA (ours) | (+) 0.47 | 53.27
CKA (ours) | (+) 0.22 | 62.95
CKA (ours) | (-) 0.29 | 72.64
MobileNetV2 | CKA (ours) | (+) 0.17 | 26.60
CKA (ours) | (-) 0.37 | 31.28
CKA (ours) + $\ell_{1}$ | (-) 2.82 | 82.89
CKA (ours) + $\ell_{1}$ | (-) 3.44 | 85.06

###  6.4 Results on the Transformer Architecture

Recent advancements in foundation models and general-purpose tasks often
leverage Transformer architectures and their variants.

In the context of layer-pruning, Dong et al. [8] noted that Transformer-like
architectures exhibit analogous behavior to ResNets – they experience either
no degradation or only negligible drops in accuracy when removing certain
layers. Such a claim provides a guarantee to perform layer-pruning on these
architectures.

In this experiment, we evaluate the effectiveness of our layer-pruning method
on the widely used Transformer architecture. Unfortunately, our limited
computational budget prevents us from considering Visual Transformers, which
typically require thousands of samples (e.g., JFT-300M) to achieve competitive
results compared to convolutional networks. Thereby, we assess our layer-
pruning technique in Transformers for human activity recognition based on
wearable sensors, a popular application involving tabular data. Details about
these datasets are available at this link:
_https://doi.org/10.1016/j.neucom.2020.04.151_.

Figure 5: Performance of our layer-pruning method on Transformer architecture
for human activity recognition based on wearable sensors (tabular data). Each
point denotes a pruned model and the black-dashed line indicates the point
where the drop in accuracy is zero; thus, points above this line (green) stand
for pruned models with an improved accuracy compared to the original,
unpruned, model.

Our Transformer architecture (unpruned) comprises $10$ layers, each with $128$
heads and projection dimensions of $64$. For each dataset, we train this
Transformer architecture for $200$ epochs and subsequently prune it similarly
to the approach described in the main body of the paper. We emphasize that our
objective here is not to advance the state-of-the-art; rather, we aim to
demonstrate that the effectiveness of our layer-pruning extends beyond ResNet
architectures.

Figure 5 shows the results. In this figure, the black dashed line shows the
point where the drop in accuracy is zero; hence, pruned models (green points)
above this line exhibit an accuracy improvement. From Figure 5, we observe
that most pruned models exhibited no accuracy drop, even on high compression
regimes. Therefore, we conjecture that on these datasets the layer-pruning
technique operated as a strong regularization mechanism. We observe that this
regularization mechanism performs well as a function of the dataset size. For
example, the datasets on the upper side of Figure 5 are at least 3 times
smaller (in terms of training size) compared to the ones on the lower side. We
intend to further explore this behavior in future research, particularly in
low-data regimes, which is a well-known deficit in Transformer-like
architectures. In summary, the results of this experiment confirm the
effectiveness of our layer-pruning method on the Transformer architecture for
tabular data.