# What Causes Exceptions in Machine Learning Applications? Mining Machine
Learning-Related Stack Traces on Stack Overflow

Amin Ghadesi, and Maxime Lamothe, and Heng Li  Amin Ghadesi, Maxime Lamothe
and Heng Li are with the Department of Computer Engineering and Software
Engineering, Polytechnique Montreal, Canada.
E-mail: {amin.ghadesi, maxime.lamothe<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML), including deep learning, has recently gained tremendous
popularity in a wide range of applications. However, like traditional
software, ML applications are not immune to the bugs that result from
programming errors. Explicit programming errors usually manifest through error
messages and stack traces. These stack traces describe the chain of function
calls that lead to an anomalous situation, or exception. Indeed, these
exceptions may cross the entire software stack (including applications and
libraries). Thus, studying the patterns in stack traces can help practitioners
and researchers understand the causes of exceptions in ML applications and the
challenges faced by ML developers. To that end, we mine Stack Overflow (SO)
and study 11,449 stack traces related to seven popular Python ML libraries.
First, we observe that ML questions that contain stack traces gain more
popularity than questions without stack traces; however, they are less likely
to get accepted answers. Second, we observe that recurrent patterns exists in
ML stack traces, even across different ML libraries, with a small portion of
patterns covering many stack traces. Third, we derive five high-level
categories and 25 low-level types from the stack trace patterns: most patterns
are related to python basic syntax, model training, parallelization, data
transformation, and subprocess invocation. Furthermore, the patterns related
to subprocess invocation, external module execution, and remote API call are
among the least likely to get accepted answers on SO. Our findings provide
insights for researchers, ML library providers, and ML application developers
to improve the quality of ML libraries and their applications.

###### Index Terms:

Machine learning applications, stack traces, Stack Overflow, empirical
software engineering.

##  1 Introduction

The popularity of machine learning (ML) (including deep learning) applications
has grown rapidly in recent years [stt6, ref8]. Indeed, a $2020$ Deloitte
survey of $2,737$ IT companies from nine countries found that modern machine
learning technologies were being used by more than $50\%$ of these companies,
and that $95\%$ of the respondents were planning to use them within the
following year [deloitte]. Furthermore, in $2022$, The NewVantage Partners’
Data and AI Leadership Executive Survey demonstrated that $91.7\%$ of
organizations were expanding their investment in data and AI activities, at
the core of which is ML [ref8].

The growing trend in ML adoption delivers new challenges for ML developers
(i.e., software developers and data scientists) [islam2019comprehensive,
zhang2019empirical, alshangiti2019developing, hamidi2021towards]. When ML
developers face issues in their development process, they can turn to
technical question and answer (Q&A) forums such as Stack Overflow (SO) to
answer their questions [alshangiti2019developing, bangash2019developers,
islam2019developers, hamidi2021towards, zhang2019empirical]. In this work, we
analyze such questions to understand the stack traces that developers face
when they develop ML programs. We target ML-related questions on SO because,
with over $22$ million questions, $33$ million answers, and roughly $18$
million users, SO is the largest technical Q&A forum [st2020new]. A $2021$
survey on SO [stSr2021] has shown that $80$ percent of respondents visit the
forum weekly, and $50$ percent visit it daily. We, therefore, leverage SO to
obtain data on exceptions that developers face when they develop ML programs
because it is both popular and ingrained in developer work habits.

Prior works have studied the issues and challenges of developing machine
learning applications [islam2019comprehensive, zhang2019empirical,
alshangiti2019developing, hamidi2021towards, sun2017empirical,
bangash2019developers]. Indeed, it has been shown that the root causes of bugs
and ML library challenges can be uncovered through a combination of bug
reports and ML posts from GitHub and SO. However, the stack traces of ML
applications remain unstudied.

A stack trace reports the chain of function calls at play when an anomalous
situation, or exception, occurs. At the time of the exception, these function
calls are either under execution or waiting for other function calls to be
completed. The stack traces provide clues for understanding the errors that
lead to exceptions (i.e., what chain of function calls lead to the exception).
Therefore, in this work, we study the stack traces of ML applications to
understand what the causes of exceptions in ML applications. Our work can
provide a complementary perspective to existing works that study the issues
and challenges of developing ML applications.

In this work, we study ~$11$K ML-related stack traces posted in SO questions,
aiming to understand common patterns that cause exceptions in ML applications.
To identify ML-related SO questions, we consider questions related to the use
of seven popular Python ML libraries, including TensorFlow [tf], Keras [ks],
Scikit-learn [sk], PyTorch [pt], NLTK [nl], HuggingFace [hugh], and Spark ML
[sp]. We focus on the Python language as it is the dominant programming
language for ML application development [pyref, pyref8]. We perform
quantitative and qualitative analyses to understand the characteristics of SO
questions with stack traces and to analyze the patterns of stack traces
therein. In particular, we structure our study and findings along the
following three research questions (RQs):

  * •

RQ1. What are the characteristics of ML-related posts with stack traces?
$12.6\%$ of posts related to our studied libraries include a stack trace.
Although these posts receive more attention, they are less likely to have
accepted answers, and take longer to be answered.

  * •

RQ2: What are the characteristics of the stack trace patterns in ML-related
questions? Most patterns are short, ($80\%$ of the patterns contain fewer than
five calls), and $19.53\%$ are shared among at least two ML libraries. We also
find that a small percentage of patterns ($20\%$) span a large number of
questions (up to $85\%$).

  * •

RQ3: What are the categories of the stack trace patterns, and which categories
are most challenging for developers? Through a qualitative study, we identify
five high-level categories and $25$ low-level types of stack trace patterns
associated with ML development. We find that exceptions are often caused by
misunderstandings or uncertainties related to ML API usage, data formats, or
language constructs. Furthermore, exceptions related to external dependencies
or manipulations of artifacts are less likely to receive timely community
support.

The main contributions of our work include:

  * •

This study is the first step towards understanding the stack traces of machine
learning applications and their related issues posted on SO.

  * •

We observe that questions with stack traces are less likely to get timely
accepted answers. Forum moderators and researchers should pay particular
attention to these questions to help developers.

  * •

We observe patterns in stack traces of ML applications, and find that a small
subset of them cover a wide range of questions, even across different
libraries. Focusing on these prevalent patterns could produce impactful
improvements.

  * •

We provide a hierarchical taxonomy of the stack trace patterns of ML
applications and recommend actions to improve ML applications and their usage.

  * •

Our observations identify the most challenging pattern types and provide
critical insight for future research.

The data and scripts used to produce this work are shared in a replication
package [replication:Mining:2022].

##  2 Experiment setup

Figure 1 gives an overview of our approach. We first extract ML-related posts
(which contain questions and answers) from SO. Then, we extract stack traces
from the posts (through regular expressions) and use pattern mining techniques
to discover patterns in the stack traces. The extracted stack traces and their
patterns are used to answer our research questions. In the rest of this
section, we elaborate on each of these steps to precisely capture each step of
our experiment setup. The detailed analysis of each research question is
presented in Section 3.

Figure 1: An overview of our study design

###  2.1 Subject Data

We use SO posts as our primary data source to study the stack traces of ML
applications. Specifically, we use SOTorrent [stt9], an open dataset, to
access the set of SO posts which contains questions, answers, and their
metadata (e.g., question tags, view counts, scores, question and answer
creation time, and whether an answer is accepted or not, etc). We use the last
released version of SOTorrent, released on $2020$-$12$-$31$, and use Google
BigQuery to access the dataset. The dataset contains the entire version
history of all SO posts in the official SO data dump from its first post on
July $31$, $2008$, until $2020$-$12$-$31$.

###  2.2 Data Collection

### [DC1] Identify ML library-related tags

We use SO question tags to identify ML-related posts. Using these tags, we
focus on the questions related to the Python programming language and our
selected ML libraries (e.g., TensorFlow). We choose the Python programming
language because it is the most popular programming language for data
scientists [pypop2, stSr2021] and involves libraries that are popular in the
ML field [comp, ref22, tax]. To identify the tags related to Python ML
libraries, we first define our studied libraries.

Defining studied ML libraries. We consider SO questions related to seven
popular ML libraries: TensorFlow [tf], PyTorch [pt], Scikit-learn [sk], Keras
[ks], NLTK [nl], HuggingFace [hugh], and Spark ML [sp]. These popular and
state-of-the-art libraries provide ready-made algorithms that allow ML
developers to efficiently apply ML-based solutions in their applications. We
select these seven ML libraries based on (1) their popularity metrics on
GitHub [gitpop1, gitpop2] (e.g., the number of forks, stars, and releases), as
well as (2) their usage in previous studies [comp, ref22, tax, stSr2021].

Identifying a tag-set. We leverage
StackExchange111https://data.stackexchange.com/stackoverflow/queries and SO
tags222https://stackoverflow.com/tags to find and extract tag names related to
ML libraries. The StackExchange query space provides an environment to find
all of the existing tags that match a specified pattern through the LIKE
operator. For example, to find the tags related to the TensorFlow library, we
use the query TagName LIKE ’%tensorflow%’ THEN ’tensorflow’. This finds all of
the versions and subcategories related to TensorFlow like tensorflow-lite and
tensorflow2.x. We use a combination of queries to obtain a relevant tag-set
for each of our studied ML libraries. Our replication package
[replication:Mining:2022] contains our queries and our complete tag-set.

### [DC2] Collect ML library-related posts

To extract posts based on our tag-set, we use the Google Cloud Platform
service (BigQuery). We export all post answers and ML-related questions using
two queries. We use the output of these queries to extract the specific part
of the metadata that we need to answer our research questions.

### [DC3] Extract post components

Posts on SO can be composed of text, images, code snippets, and more. For
simplicity, as shown in Figure 2, we summarize each post’s body into text and
code blocks. Text Blocks contain the questioner’s textual descriptions, and
Code Blocks are composed of code snippets or reports such as stack traces. In
this study, we focus on posts that contain Code Blocks with stack traces.
Although SO often uses the (<pre><code>) tag to distinguish the Code Blocks,
there are cases where Code Blocks obey other tags. Thus, we apply a
lightweight heuristic approach that uses multiple regular expressions to
identify Code Blocks in the SO posts and automatically extract all Code Blocks
and Text Blocks. Our heuristics are provided as scripts in our replication
package.

Figure 2: A truncated example of a SO question. (ID=52253378)

###  2.3 Data Analysis

### [DE 1] Filter posts with stack traces

Our research focuses stack traces. Therefore we wish to distinguish between
questions with Code Blocks with stack traces and other types of Code Blocks
such as code snippets. Regardless of operating system (OS) type, all stack
traces contain OS paths. We use this knowledge to extract stack traces
automatically through regular expressions.

### [DE 2] Mine patterns in stack traces

Stack trace transformation. The regular expressions used in DE1 provide us
with two advantages: (1) we can differentiate between types of Code Blocks,
and (2) we can capture the specific parts of each stack trace. Figure 3
indicates a stack trace recognized using our regular expressions. Moreover,
this figure shows how we convert each stack trace to a list of pairs. Each
pair contains the file’s and function’s names, which come from each line in
the stack trace. The combination of file and function names, the yellow and
red colored words in Figure 3, gives us unique pairs that we use to feed our
pattern mining procedure.

Stack trace mining. In this paper we seek to uncover stack trace patterns and
minimize the information lost due to the aggregation of different stack
traces. For this purpose, we implement CC-Span [ccspan], to mine closed
contiguous sequential patterns. The algorithm considers two aspects: the
items’ adjacency and the patterns’ closure. We use the unique pairs obtained
from the previous step as input.

Figure 3: A truncated example of stack trace transformation. (ID=70716638)

### [DE 3] Perform qualitative studies of stack trace patterns

We performed qualitative studies to identify categories of stack trace
patterns, to better understand the types of stack trace-related issues faced
by ML developers and their challenges. Our qualitative studies are detailed in
Section LABEL:sec:RQ3.

##  3 Research questions and results

(RQ1) What are the characteristics of ML-related posts with stack traces?
---

### Motivation

When ML developers face challenges, they can post questions that contain stack
traces on SO. These stack traces can indicate errors in their code, and
communicate information about the challenges they face in their development
process. Studying the characteristics of SO questions with stack traces can
help ML developers and forum providers understand the profile of these
questions and their level of community support. Therefore, this RQ proposes a
quantitative analysis of the characteristics of ML-related SO questions with
stack traces, and compares them statistically to SO questions without stack
traces. This RQ also provides context for our next two RQs, where we aim to
study the patterns in the stack traces of ML-related questions.

### Approach

In this RQ, we exclude post answers to concentrate on the characteristics of
SO questions with stack traces and compare them to those without stack traces.
In particular, we study the questions’ lengths, the questions’ popularity
(e.g., view count), and how likely these questions are to receive answers that
are accepted by the questioners. Studied characteristics of SO posts. Below we
describe the characteristic variables that we study to determine the status of
ML-related stack traces on SO.

  * Question Length: measures the number of words used in all Text Blocks of a question.

  * Code Length: measures the size of the Code Blocks of a question. We count the number of lines in each Code Block of a question. Our calculation ignores empty lines.

  * Question Score: a measure of popularity. The question score shows the difference between the number of upvotes and the number of downvotes for a question [score_ref].

  * Comment: a measure of popularity. A count of comments for a given question.

  * View: a measure of popularity. SO counts the number of views for every page load333https://meta.stackexchange.com/questions/36728/how-are-the-number-of-views-in-a-question-calculated. The total number of views for a given question.

  * Answer Count: measures the count of proposed answers for a given question.

  * First Answer: measures the hours between a question’s publication and when it receives its first answer.

  * Accepted Answer: measures the hours between a question’s publication and when it receives an accepted answer. Question posters can choose one answer that they believe is the best answer to their questions.

We apply two statistical tests (i.e., Mann-Whitney U Test and Two-sample
proportion Z Test) to compare these characteristics between questions with and
without stack traces.

Mann–Whitney U Test. We apply this test [utest] to assess whether the data for
ML-related stack traces can be considered statistically independent from
questions w/o stack traces. We chose a nonparametric test because we do not
know the distribution of our data a priory. Moreover, we execute the test at
the $5\%$ significance level (i.e., $\alpha=0.05$).

Two-sample proportion Z Test [ztest1, ztest2]. We apply this test in Table IV,
to compare the ratio of each ML library that gets accepted answers, for groups
with and without stack traces. We use this statistical test because there is
insufficient information to utilize the chi-squared distribution. We execute
the test using a 5% significance level.

### Results

TABLE I: The Number of questions on SO for each studied ML library, including
all questions, those that contain Code Blocks and those that contain stack
traces

ML Libraries # Ques. # Ques. # Ques. OS1 with code block with stack trace
Unix-based Windows TensorFlow 39,690 32,368 (81.6%) 5,436 (13.6%) 69.1% 30.9%
Keras 21,568 18,506 (85.8%) 2,715 (12.5%) 70.8% 29.2% Scikit-learn 18,133
14,795 (81.6%) 2,009 (11.0%) 62.0% 38.0% PyTorch 5,540 4,620 (83.4%) 713
(12.8%) 76.2% 23.8% NLTK 5,592 4,381 (78.3%) 564 (10.0%) 58.7% 41.3% Hugging
Face 236 204 (86.4%) 40 (16.9%) 87.5% 12.5% Spark ML 141 126 (89.4%) 16
(11.3%) 93.8% 6.20% 1 OS stands for Operating System.

An average of 12.6% of ML-related questions provide stack traces. While the
number of questions fluctuates based on the ML library, the percentage of
questions with stack traces is relatively stable. As seen in Table I,
TensorFlow ($13.6\%$) has the most questions with stack traces, and NLTK has
the least ($10.0\%$). This shows that a number of developers see an inherent
value in adding stack traces to their SO questions. Furthermore, based on the
stack trace paths, we find that most SO questions that contain stack traces
come from Unix-based systems. This can be interpreted in two ways. Either most
ML developers work and execute their code on Unix-based systems, or Unix-based
developers are more likely to add a stack trace to their SO questions. Future
work is needed to determine which is the case. However, in all cases, we find
that stack traces are used in ML-related questions irrespective of the ML
library in question.

TABLE II: The median question and code lengths for the SO posts related to our
studied ML libraries

Ques. Length (Words) Code Length (LOC) ML Libraries with stack trace w/o1
stack trace Sig.2 with stack trace w/o stack trace Sig. TensorFlow 69.0 84.0
*** 25.0 19.0 *** Keras 73.0 86.0 *** 24.0 21.0 *** Scikit-learn 67.0 79.0 ***
18.0 14.0 *** PyTorch 68.0 77.0 *** 19.0 17.0 * NLTK 61.0 72.0 *** 15.0 11.0
*** Hugging Face 67.5 90.0 * 18.0 14.0 o Spark ML 83.5 80.0 o 43.0 16.0 *** 1
w/o = without 2 Statistical significance of explanatory power according to
Mann–Whitney U Test: o $p\geq 0.05$; * $p<0.05$; ** $p<0.01$; *** $p<0.001$

Questions with stack traces have shorter natural language descriptions and
longer Code Blocks compared to questions without stack traces. Table II
displays the median text length and code length of questions with and without
stack traces. Irrespective of library, questions with stack traces are less
wordy (median $11$ fewer words on average) but use more lines of code (median
$5$ more LOC on average) compared to questions without stack traces.
Therefore, it appears that stack traces allow question posters to express
their problem with fewer words by relying on the lines of code provided by the
stack traces themselves.

TABLE III: The median community support variables for SO posts of our studied
ML libraries

Ques. Score Comment View Answer Count ML Libraries with stack trace w/o1 stack
trace Sig.2 with stack trace w/o stack trace Sig. with stack trace w/o stack
trace Sig. with stack trace w/o stack trace Sig. TensorFlow 0 1 *** 1 0 ***
324.0 202.0 *** 1 1 ** Keras 1 0 o 1 1 ** 282.0 162.0 *** 1 1 o Scikit-learn 1
1 o 2 1 *** 562.0 360.5 *** 1 1 *** PyTorch 0 1 ** 0 0 * 345.0 168.0 *** 1 1 o
NLTK 1 1 * 2 1 *** 651.5 394.0 *** 1 1 o Hugging Face 0 0 o 2 0 ** 134.0 131.5
o 1 1 o Spark ML 1.5 1 o 0 0 o 597.5 651.0 o 1 1 o Baseline 1 - 1 - 294 - 1 1
w/o = without 2 Statistical significance of explanatory power according to
Mann–Whitney U Test: o $p\geq 0.05$; * $p<0.05$; ** $p<0.01$; *** $p<0.001$

Questions with stack traces gain more popularity (in terms of view counts and
comment numbers) than questions without stack traces. Table III compares the
status of community support, including score, comment, view, and answer count
among ML-related questions with and without stack traces against a baseline
comprised of all Stack Overflow questions. While questions with stack traces
garner lower scores than questions without stack traces, the median difference
is small (~$1$). In terms of comment count and view count, questions with
stack traces get more, or an equivalent number of comments and views compared
to questions without stack traces and the SO baseline. Finally, we find that
the median answer count amongst different ML libraries is the same in all
groups, with a median answer count of $1$. Therefore, it appears that using
stack traces in ML-library questions encourages comments and views.

TABLE IV: The median time to answer for SO posts of our studied ML libraries

First Answer (Hour) Accepted Answer (Hour) ML Libraries with stack trace w/o1
stack trace Sig.2 with stack trace w/o stack trace Sig.2 ratio (with stack
trace) ratio (w/o stack trace) Sig.3 TensorFlow 5.82 1.88 *** 3.96 2.40 ***
0.33 0.39 *** Keras 3.27 1.86 *** 2.24 1.84 ** 0.36 0.39 *** Scikit-learn 2.97
1.53 *** 2.50 1.56 *** 0.41 0.48 *** PyTorch 4.10 2.53 *** 3.08 2.53 o 0.38
0.42 *** NLTK 2.39 1.14 *** 2.91 1.16 *** 0.40 0.50 *** Hugging Face 21.24
18.48 o 0.98 14.10 o 0.27 0.42 o Spark ML 2.40 2.43 o 1.97 1.79 o 0.50 0.49 o
1 w/o = without $-$ Statistical significance of explanatory power according to
Mann–Whitney U Test2 and Two-sample proportion Z Test3: o $p\geq 0.05$; *
$p<0.05$; ** $p<0.01$; *** $p<0.001$

Questions with stack traces are less likely to get answers and take longer to
get them when they do. We use the Two-sample proportion Z Test to compare the
ratio of questions that receive an accepted answer with and without stack
traces. Using Table IV, we can see that questions with stack traces are less
likely to receive an accepted answer. Indeed, for all statistically
significant results, questions with stack traces take longer to obtain a first
answer, and an accepted answer. While some libraries obtain answers faster
than others, the trend holds across libraries, despite the overall answer
speed. Indeed, Table IV shows that questions with stack traces present unique
characteristics that are distinct from questions without stack traces, even
from questions that may contain other Code Blocks, such as code snippets.

Questions with stack traces rarely contain more than one stack trace. Table
LABEL:tab:thesame presents the distribution of the stack traces on SO for each
ML library. In all cases, questions rarely (at most $6\%$) contain more than
one stack trace.

TABLE V: The distribution of stack traces on SO for our studied ML libraries


August 2022

Entropy of the Universe and Hierarchical Dark Matter
Paul H<EMAIL_ADDRESS>
Dipartimento di Matematica e Fisica ”Ennio De Giorgi”,
Università del Salento and INFN-Lecce,
Via Arnesano, 73100 Lecce, Italy.

We discuss the relationship between dark matter and the entropy of the
universe with the premise that dark matter exists in the form of primordial
black holes (PBHs) in a hierarchy of mass tiers. The lightest tier are all
PBHs with masses below one hundred solar masses. The second lightest tier are
intermediate-mass PIMBHs within galaxies including the Milky Way. Supermassive
black holes at galactic centres are in the third tier. We are led to speculate
that there exists a fourth tier of extremely massive PBHs, more massive than
entire galaxies. We discuss future observations by the Rubin Observatory and
the James Webb Space Telescope.

##  1 Introduction

In particle theory, the concept of entropy is generally not fundamental
because for one elementary particle entropy is neither defined nor useful.

In general relativity and cosmology, the situation is different. For black
holes, entropy is a central and useful concept. For cosmology, the entropy of
the universe has often been considered, although not emphasised enough. We
shall argue that the origin and nature of cosmological dark matter can be best
understood by consideration of the entropy of the universe. We have made such
an argument four years ago[1] but that discussion was too diluted by
considering simultaneously dark matter being made from elementary particles
such as WIMPs and axions, as were favoured three decades ago[2].

In this paper, we dispose of microscopic candidates in one paragraph. The
standard model of particle theory (SM) has two examples of lack of
naturalness, the Higgs boson and the strong CP problem. Our position is that
to understand these we still need to understand better the SM itself.
Regarding the strong CP problem, it is too ad hoc to posit a spontaneously
broken global symmetry and consequences which include an axion. Concerning the
WIMP, the idea that dark matter experiences weak interactions arose from
assuming TeV-scale supersymmetry which is now disfavoured by LHC data. To
identify the dark matter, we instead look up.

Assuming dark matter is astrophysical, and that the reason for its existence
lies in the Second Law of Thermodynamics, we shall be led uniquely to the dark
matter constituent as the Primordial Black Hole (PBH). We must admit that
there is no observational evidence for any PBH, but according to our
discussion PBHs must exist. In the ensuing discussion, we shall speculate that
they exist in abundance in four tiers of mass up to and including at several
galactic masses.

Because PBH entropy goes like mass squared, we are mainly interested in masses
satisfying $M_{PBH}>100M_{\odot}$. From here on, we shall adopt the unadorned
acronym PBH to denote only those with masses which satisfy $M<100M_{\odot}$.
In the earliest discussions of PBHs, they were tacitly assumed to be this
light, usually even much lighter than the Sun. This lightest tier will
contribute a negligible fraction of the total dark matter entropy but can
contribute a few percent of the total dark matter mass. Within the Milky Way,
we use the acronym PIMBH for intermediate mass PBHs in the mass range
$10^{2}M_{\odot}<M_{PIMBH}<10^{5}M_{\odot}$. Outside the Milky Way we
entertain all masses $10^{2}M_{\odot}<M_{PBH}<10^{17}M_{\odot}$. Of these, we
use PSMBH for supermassive PBHs in the mass range
$10^{5}M_{\odot}<M_{PSMBH}<10^{11}M_{\odot}$ and PEMBH for extra massive PBHs
with $10^{11}M_{\odot}<M_{PEMBH}<10^{17}M_{\odot}$.

Although the visible universe (VU) is not a black hole, its Schwartzschild
radius is about 68% of its physical radius, 30 Gly versus 44 Gly, so it is
close. This curious fact seems to have no bearing on the nature of dark
matter. A few more acronyms will be useful: CMB, CIB and CXB. CMB is the
familiar cosmic microwave background while I and X refer to Infra-red and
X-ray respectively.

There exist a number of constraints on PBHs derived from astronomical
observations[3, 4]. We would advise caution in interpreting constraints
derived from CMB distortion caused by additional microwaves resulting from
X-ray emission by accreted matter. The accretion model often used is of a
spherically symmetric Bondi-type which can overestimate accretion by as much
as four orders of magnitude and hence lead to constraints which are far too
stringent. This is not to say that all such constraints are wrong, only that
they do not follow from the arguments given.

The arrangement of the paper is as follows. We discuss entropy and the second
law in sections 2 and 3, then in section 4 primordial black holes. In sections
5 and 6 we discuss two methods of PBH detection, microlensing and cosmic
infrared background respectively. Finally, in section 7 we discuss our
results.

##  2 Entropy

We begin with the premise that the early universe be regarded in an
approximate sense as a thermodynamically-isolated system for the purposes of
our discussion. It certainly contains a number of particles, $\sim 10^{80}$,
vastly larger than the numbers normally appearing in statistical mechanics,
such as Avogadro’s number, $\sim 6\times 10^{23}$ molecules per mole.

No heat ever enters or leaves and it can be considered as though its surface
were covered by a perfect thermal insulator. It is impracticable to solve all
the Boltzmann transport equations so it is mandatory to use thermodynamic
arguments, provided that we may argue that the system is proximate to thermal
equilibrium.

Making the then-unsupported assumption in 1872 [5] of atoms and molecules,
Boltzmann discovered the quantity $S(t)$ in terms of the molecular momentum
distribution function $f(\textbf{p},t)$

$S(t)=-\int d\textbf{p}f(\textbf{p},t)\log f(\textbf{p}.t)$ (1)

which satisfies

$\left(\frac{dS(t)}{dt}\right)\geq 0$ (2)

and can be identified with the thermodynamic entropy. The crucial inequality,
Eq(2), the Second Law, was derived in [5] for non-equilibrium systems assuming
only the Boltzmann transport equations and the ergodic hypothesis.

Ascertaining the nature of the dark matter can be regarded as a detective’s
mission and there are useful clues in the visible universe. In [1], we made an
inventory of the entropies of the known objects in the visible universe, using
a venerable source, the book [6]. Let us model the visible universe as
containing $10^{11}$ galaxies each of mass $10^{12}M_{\odot}$ and each
containing one central SMBH with mass $10^{7}M_{\odot}$. We recall the
dimensionless entropy of a black hole $S/k(M_{BH}=\eta M_{\odot})\sim
10^{78}\eta^{2}$. Then the inventory is

  * •

SMBHs $\sim 10^{103}$

  * •

Photons $\sim 10^{88}$

  * •

Neutrinos $\sim 10^{88}$

  * •

Baryons $\sim 10^{80}$

We regard this entropy inventory as a first clue. From the point of view of
entropy the Universe would be only infinitesimally changed if everything
except the SMBHs were removed. This suggests that more generally black holes
totally dominate the entropy, as we shall find in the sequel.

A second remarkable fact about the visible universe is the near-perfect black-
body spectrum of the CMB which originated some 300,000 years after the
beginning of the present expansion era, or after the Big Bang in a more
familiar language. We are not tied to a Big Bang which could well be replaced
by a bounce in a cyclic cosmology.

The precise CMB spectrum is a second clue about dark matter. It suggests that
the plasma of electrons and protons prior to recombination is in excellent
thermal equilibrium, and hence the matter sector was in thermal equilibrium
for the first 300,000 years. This, combined with the thermal isolation
mentioned already, underwrites the use of entropy, and the second law, during
this period.

A third clue and final one about dark matter lies with the holographic
principle [7] which provides, as upper limit on the entropy of the visible
universe, the area of its surface in units of the Planck length. Given its
present co-moving radius $44$ Gly this requires $S/k\leq 10^{123}$. The
entropy of the contents which is so bounded might nevertheless tend to
approach[8] a limit which is many orders of magnitude higher than the total
entropy in the limited inventory listed above.

##  3 Second Law

For primordial black holes (PBHs) formed at cosmic time $t$, their mass may be
taken to be governed by the horizon size, giving

$M_{PBH}=10^{5}M_{\odot}\left(\frac{t}{1\sec}\right)$ (3)

so that PBHs with masses $10^{2}M_{\odot}<M_{PBH}<10^{17}M_{\odot}$ are
produced for $10^{-3}s<t<30ky$. The top few orders of magnitude are unlikely,
but possible.

A tendency to increase the entropy of the universe towards $S_{U}/k\sim
10^{123}$ can be most readily achieved by the formation of PBHs, the more
massive the better, because $S_{BH}/k\sim 10^{78}\eta^{2}$ for mass
$M_{BH}=\eta M_{\odot}.$ For example, in the case that a PEMBH existed with
$M_{PEMBH}\sim 10^{17}M_{\odot}$ it would have $S/k\sim 10^{112}$ which is a
billion times the entropy of the items listed in our previous inventory.

The PBH mass function is unknown so we must make reasonable conjectures which
may approximate Nature. For a preliminary discussion we may take monochromatic
distributions separately for PIMBHs, PSMBHs and PEMBHs. The real mass function
is expected to be smoother but the general features in our discussion of
entropy should remain valid.

In a toy model for the visible universe we include $10^{11}$ galaxies each
with mass $10^{12}M_{\odot}$. As a hierarchical dark matter we shall take as
illustration all PIMBHs with $100M_{\odot}$; all PSMBHs with
$10^{7}M_{\odot}$; all PEMBHs at $10^{14}M_{\odot}$. Let the number of each
type be $n_{I}$, $n_{S}$ and $n_{E}$, respectively. The total dark matter mass
is then

$M=\left(10^{2}n_{I}+10^{7}n_{S}+10^{14}n_{E}\right)M_{\odot}$ (4)

while the total entropy contributed by all PBHs is

$S/k=\left(10^{82}n_{I}+10^{92}n_{S}+10^{106}n_{E}\right)$ (5)

Let us begin with the middle one of the three hierarchical tiers, the
supermassive black holes known to reside in galactic centres. In our toy
model, $n_{S}$ is equal to the number of galaxies $n_{S}=10^{11}$ so their
total mass and entropy are, from Eq.(4),

$M(PSMBHs)=10^{18}M_{\odot}$ (6)

and, from Eq.(5),

$S(PSMBHs)/k=10^{103}$ (7)

Before considering Eqs.(4) and (5) further, let us step back and ask which of
the three terms in each equation is most likely to be dominant? The answer is
different for Eqs.(4) and (5) because entropy $S/k$ and mass $M$ have the
relationship $S/k\propto M^{2}$.

The total mass in Eq.(4) is comparable to the total mass of the visible
universe which is $\sim 10^{123}M_{\odot}$. Comparison with $M(PSMBHs)$ in
Eq.(6) then show that the second term in the R.H.S. of Eq.(4) is sub-dominant,
being several orders of magnitude less than the L.H.S.

Now let us discuss the first term on the R.H.S. In our toy model every galaxy
has mass $10^{12}M_{\odot}$ which is dominated by the dark matter halo made up
of $100M_{\odot}$ PIMBHs and therefore, since there are $10^{11}$ galaxies, we
take $n_{I}=(10^{11})\times(10^{10})=10^{21}$ whereupon the total mass and
entropy of the PIMBHs are, from Eq.(4),

$M(PIMBHs)=10^{23}M_{\odot}$ (8)

and, from Eq.(5),

$S(PIMBHs)/k=10^{103}$ (9)

From Eq.(8) we deduce that the first term on the R.H.S. of Eq.(4) is a
dominant term. We already know that the second term on the R.H.S. is
relatively small. What about the third and last term? At this stage, we can
say little except that observation is consistent with it vanishing. Perhaps
surprisingly, to jump ahead, after discussion of the entropy equation, Eq.(5),
we shall suggest the third term on the R.H.S. of Eq.(4) is comparable to the
first term on the R.H.S. of Eq.(4), thus providing a rather novel viewpoint of
dark matter.

Substituting our choices $n_{I}=10^{21}$ and $n_{S}=10^{11}$ into Eq.(5) we
find for the total entropy

$S/k=\left(2\times 10^{103}+10^{106}n_{E}\right)$ (10)

to be compared to the total mass

$M=\left(10^{23}+10^{18}+10^{14}n_{E}\right)M_{\odot}$ (11)

In Eq.(11), for consistency we must bound the parameter $n_{E}$ from above by
$n_{E}\leq 10^{9}$ to avoid overclosing the universe. It is interesting to
study the upper limit of $n_{E}$ in the entropy equation, Eq.(10). This gives
$\sim 10^{115}$ to be compared with the holographic bound on the entropy[7, 8]
which is $\sim 10^{123}$.

In the absence of any observational evidence about either dark matter or
primordial black holes, we need to look at the visible universe from the two
theoretical viewpoints of mass and entropy. This suggests the most likely
scenario which is $n_{E}\sim 10^{9}$. This predicts that our toy universe
contains of order one billion extra-massive black hole with masses
$O(10^{14}M_{\odot})$ or perhaps a smaller number of even more massive PBHs.
Because of their extraordinarily high masses, these PEMBHs are not expected to
be associated with a specific galaxy or cluster of galaxies.

##  4 Primordial Black Holes

If black holes make up all the dark matter, they cannot be all gravity-
collapse black holes because of baryon mumber conservation. The amount of dark
matter is more than five times that of baryons. Therefore, most or all dark-
matter black holes must instead be primordial.

PBHs are black holes formed in the early universe when there is s high density
and sufficiently large fluctuations and inhomogeneities. Their existence was
first conjectured in the 1960s in the Soviet Union[10] and independently in
the 1970s, in the West[11]. Initially it was realised that only PBHs with mass
greater than $10^{-18}M_{\odot}$ could survive until the present time because
of Hawking evaporation. Nevertheless, it was generally assumed that PBHs were
all very much lighter than the Sun and hence even more lighter than all the
PBHs considered in the bulk of this paper.

During this early era of extremely light PBHs, the seminal idea that PBHs
could form all the dark matter was proposed in 1975 by Chapline[12]. In 2009
[13] and in 2010 [14] the relevance of entropy in cosmological evolution
emerged.

Beginning in 2010[15], the upper limit on PBH mass was removed by showing that
in a specific model of hybrid inflation, with two stages of inflation, a
parametric resonance could mathematically yield fluctuations and
inhomogeneities of arbitrarily large size. We regard this as merely an
existence theorem and that such formation might take place without inflation.

The possibility of PBHs with many solar masses led to the 2015 dark matter
proposal in [9] that PIMBHs provide an excellent astrophysical candidate for
dark matter in the Milky Way halo, especially given the absence of a
compelling elementary particle candidate either within the standard model or
in any plausible extension thereof. This was further underscored in [16]. Both
of these papers emphasised microlensing by PIMBHs of starlight from the
Magellanic Clouds [17] as a promising method for detection of PIMBHs in the
Milky Way.

These PIMBHs are now to be regarded as the second of four mass tiers, the
third being the supermassive PSMBHs at galactic centres and the fourth being
extremely massive PEMBHs, more massive than galaxies. The first tier contains
all PBHs with masses below $100M_{\odot}$.

Returning to our thermodynamic arguments about entropy, we use the entropy
inventory of the known entities to observe the idea that very massive black
holes already dominate entropy through the PSMBHs which we assume are
primordial because there seems to be insufficient cosmic time for stellar mass
black holes adequately to grow by accretion and mergers.

For the entropy of the universe to be nearer to its holographic upper limit,
we are led to introduce $10^{9}$ PEMBHs of $10^{14}M_{\odot}$ to reach
$S/k\sim 10^{115}$. To achieve the maximum $S/k\sim 10^{123}$ is possible with
just ten PEMBHs of $10^{22}M_{\odot}$ which, if true, would be revolutionary.

We expect PEMBHs not to be associated, in general, with specific luminous
galaxies or clusters of galaxies, so we do not discuss here the interesting
topic of co-evolution[18]. Nevertheless, it is interesting to learn that for
masses $>10^{12}M_{\odot}$ accretion should proceed [19] in a non-luminous
manner so that such a PEMBH can never appear in a quasar.

A Kerr black hole is characterised by only three parameters $M,S$ and $Q$ and
in astrophysics it had been common to assume that the electric charge
vanishes. Recent papers [20, 21, 22, 23] have seriously queried this
assumption for PSMBHs. For example, in [20] an upper limit on a non-zero
electric charge of the Milky Way’s PSMBH, $SgrA^{*}$, has been given as
$3\times 10^{8}$C. The exciting possibilIty of non-vanishing electric charges
for PEMBHs also merits sedulous study.

PEMBHs have also been discussed by Carr et al. in [24]. We already mentioned
one of the first proposals of PBHs involved Carr in 1974[11]. Nobody has
contributed more papers on the study of PBHs than Carr as exemplified by
papers in 1975[25], 2010[26] and 2016[27].

##  5 Microlensing

Gravitational lensing of a distant star by a nearer massive object or lens,
moving across the field of view, gives rise to an enhancement of the star and
to a temporal light curve whose duration is proportional to the square root of
the mass of the lens, as displayed in Eq.(12).

Aa already mentioned, a direct way to discover PIMBHs in the Milky Way would
be to use microlensing[9, 16] of light from the stars in the Magellanic
clouds. Assuming a transit velocity $200$km/s an estimate of the duration
$\hat{t}$ of the light curve at half maximum is

$\hat{t}\sim 0.2y\left(\frac{M_{lens}}{M_{\odot}}\right)^{\frac{1}{2}}$ (12)

which means that for $10^{2}M_{\odot}<M_{PIMBH}<10^{5}M_{\odot}$ the duration
of the light curve is in the range $2y<\hat{t}<60y$. Masses below
$2,500M_{\odot}$ with $\hat{t}<10y$ are clearly the most practicable to
measure.

A successful precursor was an experiment by the MACHO Collaboration[17] in the
1990s. In the 2020s, microlensing searches at the Vera Rubin Observatory [28]
could repeat this success for the much higher mass ranges of the MACHOs
expected for the dark matter inside the Milky Way.

The MACHO collaboration, 1992-99, used the observatory at Mount Stromlo near
Canberra, Australia. it was a 1.27 m telescope with two 16-Magapixel cameras.
They showed that the technique could be achieved successfully to discover
MACHOs, as well as confirming this prediction by Einstein’s general
relativity. The highest duration of their more than a dozen light-curves was
230 days corresponding to a mass close to $10M_{\odot}$.

An attempt was made to use the Blanco 4m telescope at Cerro Tololo, Chile with
the DECam having 570 Megapixels in order to find light-curves with durations
of two years or more, and hence, by Eq.(12), lenses with $M>100M_{\odot}$. The
longer durations led, however, to crowding in the field of view such that it
was impracticable to track a specific target star.

A more powerful telescope under construction at Cerro Panchon, also in Chile,
is the Vera Rubin Observatory[28] expected to start taking data in 2023. Its
telescope is 8.4 metres and its camera has 3.2 Gigapixels, both significantly
larger, and we can reasonably hope that it can microlens multi-year-duration
light curves and possibly confirm the existence of PIMBHs in the Milky Way.

##  6 Cosmic Infrared Background

At large red-shifts $Z>15$, a population of PBHs would be expected to accrete
matter and emit in X-ray and UV radiation which will be redshifted into the
CIB to be probed for the first time by the James Webb Space Telescope [29]
which could therefore provide support for PBH formation.

Analysis of a specific PBH formation model [30] supports this idea that the
JWST observations in the infrared could provide relevant information about
whether PBHs really are formed in the early universe.

This is important because although we have plenty of evidence for the
existence of black holes, whether any of them is primordial is not known. The
gravitational wave detectors[31] LIGO, VIRGO and KAGRA have discovered mergers
in black hole binaries with initial black holes in the mass range
$3-85M_{\odot}$. We suspect that all or most of these are not primordial but
that is only conjecture.

The supermassive black holes at galactic centres, including Sgr A* at the
centre of the Milky Way, are well established and are primordial in our toy
model. Whether that is the case in Nature is unknown.

Because of the no-hair theorem that black holes are completely characterised
by their mass, spin and electric charge (usually taken to be zero), there is
no way to tell directly whether a given black hole is primordial or the result
of gravitational collapse of a star.

The distinction between a primordial and a non-primordial black hole can be
made only from knowledge of its history. For example, if it existed before
star formation, it must be primordial. The infra-red data from JWST might be
able to provide useful insight into this central question.

##  7 Discussion

It is familiar to study a mass-energy pie-chart of the universe with
approximately 5% baryonic normal matter, 25% dark matter and 70% dark energy.
The entropy pie-chart is very different if the toy model considered in this
papers resembles Nature. The slices corresponding to normal matter and dark
energy are extremely thin and the pie is essentially all dark matter.

In this article we have attempted to justify better the discussion of our
previous 2018 paper[1] which argued that entropy and the second law applied to
the early universe provide a raison d’être for the dark matter. In [9] and
[16] we proposed that the dark matter constituents in the Milky Way are
PIMBHs, a second tier of PBH beyond the light ones with less than one hundred
solar masses.

Here we have included the supermassive PSMBHs at the galactic centres as a
third tier of dark matter with a similar primordial origin to replace the
conventional wisdom that SMBHs arise from accretion and merging of black holes
which arise from gravity collapse of stars.

We have gone one step further and discussed a fourth tier of the extremely
massive PEMBHs, more massive than clusters, whose entropy far exceeds that of
the PIMBHs and PSMBHs. If this is correct then although normal matter
contributes as much as 5% of the mass-energy pie-chart of the universe, its
contribution to an entropy pie-chart is truly infinitesimal.

Since it has never been observed except by its gravity, it does seem most
likely that dark matter has no direct or even indirect connection to the
standard model of strong and electroweak interactions in particle theory,
including extensions thereof aimed to ameliorate problems with naturalness
existing therein with respect to the Higgs boson and the strong CP problem.

The three clues we have mentioned in the Introduction, the dominance of black
holes in the entropy inventory, the CMB spectrum and the holographic entropy
maximum all hint toward PBHs as the dark matter constituent.

One ambiguity is whether the maximum entropy limit suggested by holography
should be saturated in which case the mass function for the PEMBHs must be
extended to high values.

## Acknowledgements

We thank the Department of Physics at the University of Salento for an
affiliation. We thank G.F. Chapline and G. ’t Hooft for useful discussions.

## References

  * [1] P.H. Frampton,
Int. J. Mod. Phys. A33, 1830030 (2018).
arXiv1804.03516[physics.gen-ph].

  * [2] E.W. Kolb and M.S. Turner,
The Early Universe, Frontiers in Physics. (1990).

  * [3] M. Sasaki, T. Suyama, T. Tanaka and S. Yokohama,
Class. Quant. Grav. 35, 063001 (2018).
arXiv:1801.05235[astro-ph.CO].

  * [4] B.J. Carr, K.Kohri, Y. Sendouda and J. Yokoyama,
Rept. Prog. Phys. 84, 116902 (2021).
arXiv:2002.12778[astro-ph.CO].

  * [5] L. Boltzmann,
Sitzungber.Akad.Wiss. 66, 275 (1872).

  * [6] S. Weinberg,
Gravitation and Cosmology.
John Wiley (1972).

  * [7] G.’t Hooft,
Conf.Proc.C 930308 (1993);
also in Salamfestschrift (1993).

  * [8] C. Coriano, P.H. Frampton and J.E. Kim,
Mod.Phys.Lett. A36, 2150241\. (2021).
arXiv:2102.11826[hep-ph]

  * [9] P.H. Frampton,
Mod.Phys.Lett. A31, 1650093 (2016).
arXiv:0510.00400[hep-ph].

  * [10] I. D. Novikov and Ya. B. Zeldovich,
Annu. Rev. Astron. Astrophys. 5, 627 (1967).

  * [11] B. J. Carr and S. W. Hawking,
MNRAS 168, 399 (1974).

  * [12] G.F. Chapline,
Nature 253, 251 (1975).

  * [13] P.H. Frampton, S.D.H. Hsu, T.W. Kephart and D. Reeb,
Class.Quant.Grav. 26, 145005 (2009).
arXiv:0801.1847[hep-th].

  * [14] G.F. Chapline,
OALib J. 1, 1101174 (2010).

  * [15] P.H. Frampton, M. Kawasaki, F. Takahashi and T. Yanagida.
JCAP 04:023 (2010).
arXiv:1001.2308[hep-ph].

  * [16] G.F. Chapline and P.H. Frampton,
JCAP 1611:042 (2016).
1608.04297[gr-qc]

  * [17] C. Alcock, et al. (MACHO Collaboration),
Astrophys.J. 542, 281 (2000).
arXiv:astro-ph/0001272.

  * [18] J. Kormandy and L.C. Ho,
Ann. Rev.Astron.Astrophys. 51, 511 (2013).
arXiv:1304.7762[astro-ph.CO].

  * [19] A.R. King,
MNRAS 456L, 109 (2016).
arXiv:1511.08502[astro-ph.GA].

  * [20] M. Zajacek, A.Tursunev, A. Eckart and S. Britzen,
MNRAS. 480, 4408 (2018).
arXiv:1808.07327[astro-ph.GA].

  * [21] A.R. King and J.E. Pringle,
Astrophys. J. Lett. 918, L22 (2021).
arXiv:2107.12384[astro-ph.HE].

  * [22] S.S. Komissarov,
MNRAS 512, 2798 (2022).
arXiv:2108.08161[astro-ph.HE].

  * [23] I.J. Araya, N.D. Padilla, M.E. Rubio, J. Sureda,
J. Mangana and L. Osorio,
arXiv:2207.05829[astro-ph.CO].

  * [24] B.J. Carr, F. Kuhnel and L. Visinelli,
MNRAS 501, 2029 (2021).
arXiv:2008.08077[astro-ph.CO].

  * [25] B.J. Carr,
Astrophys. J. 201, 1 (1975).

  * [26] B.J. Carr, K. Kohri, Y. Sendouda and J. Yokoyama,
Phys. Rev. D81, 104019 (2010).
arXiv:0912.5297[astro-ph.CO].

  * [27] B.J. Carr, F. Kuhnel and M. Sandstad,
Phys. Rev. D44, 083504 (2016).
arXiv:1607.06077[astro-ph.CO].

  * [28] For the Rubin Observatory, see: https://www.lsst.org
  * [29] For the James Webb Space Telescope, see: https://www.jwst.nasa.gov
  * [30] N. Cappelluti, G. Hasinger and P. Natarajan.
Astrophys. J. 926, 205 (2022).
arXiv:2109.08701[astro-ph.CO].

  * [31] R. Abbott, et al.,
(LIGO Scientific and VIRGO and KAGRA Collaborations),
arXiv:2111.03606[gr-qc].


# Block Pruning for Enhanced Efficiency in Convolutional Neural Networks
††thanks:

Cheng-En Wu  University of Wisconsin-Madison
<EMAIL_ADDRESS>Azadeh Davoodi  University of Wisconsin-Madison
<EMAIL_ADDRESS>Yu Hen Hu  University of Wisconsin-Madison
<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel approach to network pruning, targeting block
pruning in deep neural networks for edge computing environments. Our method
diverges from traditional techniques that utilize proxy metrics, instead
employing a direct block removal strategy to assess the impact on
classification accuracy. This hands-on approach allows for an accurate
evaluation of each block’s importance. We conducted extensive experiments on
CIFAR-10, CIFAR-100, and ImageNet datasets using ResNet architectures. Our
results demonstrate the efficacy of our method, particularly on large-scale
datasets like ImageNet with ResNet50, where it excelled in reducing model size
while retaining high accuracy, even when pruning a significant portion of the
network. The findings underscore our method’s capability in maintaining an
optimal balance between model size and performance, especially in resource-
constrained edge computing scenarios.

###### Index Terms:

model pruning, block pruning

##  I Introduction

In the rapidly evolving landscape of edge computing, the deployment of machine
learning models on edge devices is constrained by limited memory and
computational power. To address these challenges, we present an innovative
approach that leverages pre-existing pre-trained models as an initiation
point. Our primary objective is to derive pruned models that demand less
computational power, thus yielding quicker inference speeds and reduced memory
consumption.

We commence with block pruning, and depending on the necessity, progress to
channel pruning followed by weight pruning. After each pruning phase, we
harness the power of transfer learning to fine-tune the modified units.
Recognizing the inefficiencies of exhaustive search techniques, we introduce
heuristic methods to identify potential candidates for pruned models. Our
proposed heuristic methods not only economize on time but also closely
approximate optimal solutions.

Block pruning for convolutional models includes the accurate estimation of a
model’s performance after selective block removal. Brute force search methods
have relied on exhaustive search, which is computationally expensive and falls
within the NP-hard problem space—a terrain not favorable to computational
efficiency. To this end, the question arises: can we employ heuristic
polynomial-time solutions to tackle this subset selection problem efficiently?

Existing algorithms[1, 2, 3, 4], offer varying approaches to this issue,
generally adopting proxy methods for performance estimation. These methods
only provide an approximate inference of the model’s performance based on the
perturbation of selected subsets, often relying solely on validation data for
evaluation. Such an approach inevitably results in less accurate performance
metrics and leaves unaddressed challenges like dimensionality mismatch
following block removal.

In contrast, our proposed method adopts a more direct approach by executing
block pruning first and then applying transfer learning to fine-tune the
pruned architecture. This allows for an exact evaluation of the model’s
performance, as opposed to relying on proxy methods. Our approach also
accounts for potential issues like dimensionality mismatch by incorporating
additional transfer learning steps. The result is a more accurate, and
potentially more efficient, model pruning technique.

##  II Related Work

Block pruning in deep neural networks (DNNs) is essential for optimizing
computational efficiency and reducing storage requirements, particularly in
resource-constrained environments. Various methods have been developed, each
addressing this challenge uniquely. Sparse Structure Selection (SSS)[5]
integrates pruning and training through an L1 regularization approach,
focusing on scaling factors to align the pruning process with ongoing
training, thereby ensuring minimal impact on network performance while
enhancing efficiency. Proxy Classifier-based Methods, such as those proposed
by Chen et al.[1] and Elkerdawy et al.[2], utilize proxy classifiers for block
selection based on intermediate feature representations, pruning blocks with
minimal impact on overall performance. Discrimination Based Block-Level
Pruning (DBP), proposed by Wang et al.[6], accelerates deep model performance
by targeting blocks that contribute less to the model’s discrimination
capacity. SRinit[3] employs stochastic re-initialization of layer parameters
to identify layers that are less sensitive to such changes. Layers with low
accuracy drop upon re-initialization are deemed less crucial and suitable for
pruning, thereby maintaining the model’s performance integrity.

##  III Proposed Method

###  III-A Types of Convolutional blocks

Convolutional blocks are integral components of many modern deep learning
architectures. They encompass a variety of structural designs, each with
specific functions and characteristics. In the following, we describe common
types of convolutional blocks, highlighting their key features and roles in
enhancing network performance and efficiency.

Fully Connected (FC) block: This block is composed of a fully connected (FC)
layer coupled with a ReLu (Rectified Linear Unit) layer. The FC layer offers
global context by connecting every neuron in the layer to every neuron in the
preceding layer. The ReLu layer is a non-linear operation that helps the model
learn complex patterns within the data.

Basic Block: The basic block is made up of a fully Convolutional (Conv) layer
and a ReLu layer. The Conv layer performs a dot product between its weights
and a small region it is connected to in the input volume, thereby preserving
the spatial relationship between pixels.

Pooling Block: This block is constructed with a fully Convolutional (Conv)
layer, followed by a Pooling and ReLu layer. The pooling operation reduces the
spatial size of the Conv layer, thereby reducing the computational complexity
and the number of parameters, and in turn, the risk of overfitting.

Residual Block[7]: This block contains a series of Conv layers and ReLu
activations followed by a shortcut or skip connection. The Residual block
enables the training of deep networks by allowing the gradient to be directly
backpropagated to earlier layers.

Residual Block w/ 1x1 Conv: Similar to the standard Residual block, but with
the addition of a 1x1 Conv layer. The 1x1 Conv layer, also known as pointwise
convolution, changes the dimensionality of the input feature maps either by
increasing or reducing the number of channels, adding an extra level of
abstraction.

Dense Block[8]: Originating from the DenseNet architecture, the Dense block is
assembled using multiple convolution blocks, all of which maintain a
consistent number of output channels. This structure follows a feed-forward
design wherein each layer is directly interconnected with all subsequent
layers, thereby promoting feature reuse throughout the network.

MobileNet Block: This block is based on the MobileNet[9] architecture which
uses depthwise separable convolutions instead of the standard convolutions.
This technique reduces the number of parameters, thus decreasing the
computational cost and making the network lighter and faster, which is ideal
for mobile and embedded vision applications.

MobileNet V2 Stride=1 Block: This block is a variation of the MobileNet block
that maintains the resolution of the input. It uses inverted residuals and
linear bottlenecks to improve the network’s efficiency, as introduced in
MobileNetV2[10].

MobileNet V2 Stride=2 Block: Similar to the MobileNet V2 Stride=1 block but
with a stride of 2, which means that the block reduces the spatial dimensions
of the output by half, further increasing the computational efficiency.

Inception Block: This block introduced in the Inception network[11] (or
GoogLeNet), consists of parallel paths with convolutions of different sizes
(1x1, 3x3, and 5x5), all of which are processed simultaneously. A unique
characteristic of the Inception block is that it also includes a parallel max
pooling path.

Figure 1: The diagram shows a deep neural network composed of several stacked
Convolutional (Conv) blocks, each of which can be classified into 10 distinct
types. We have enforced a limitation that restricts pruning on the truncated
Conv block (which is the initial Conv in the network), each of the first
blocks within the Conv Blocks, as well as the final fully connected (FC)
layer.

###  III-B Baseline

We use two baseline methods for block pruning in our approach. The first
method is sequential block pruning, which prunes blocks from the end of the
network to the beginning. This method is based on the principle that later
blocks in deep neural networks might be less crucial for maintaining overall
network performance, especially in models with a significant depth. The blocks
are pruned sequentially, which allows for a systematic reduction in network
complexity while monitoring the impact on performance. The second method is
SRinit, as described in [3]. This method adopts a novel approach by adding
noise to the blocks to evaluate the importance of each block. Specifically, it
involves the stochastic re-initialization of layer parameters, where layers
with a low accuracy drop after re-initialization are considered to be more
redundant. These layers are less likely to compromise the prediction ability
of the network if removed.

###  III-C Our Method

Unlike baseline methods that rely on proxy metrics to evaluate the importance
of each block, our approach directly removes individual valid blocks to
ascertain their impact on classification accuracy. This hands-on strategy
offers a genuine reflection of the importance of each pruned block. We
identify valid blocks, which do not serve as bridging elements between blocks
with different channel numbers, for pruning. As illustrated in Fig1, certain
blocks, like the stem convolutional (Conv) layer and the first block in each
stacked layer, are exempted from pruning. Let $N$ represent our neural network
architecture, consisting of a sequence of blocks $B_{1},B_{2},\ldots,B_{n}$,
and $V$ be the set of indices of valid blocks. Our pruning process $P(N,i)$
involves removing block $B_{i}$ from $N$, where $i\in V$. The impact of
pruning on network performance is measured by the classification accuracy
$A(N^{\prime})$ of the pruned network. This method not only simplifies the
pruning process by addressing the dimension mismatch issue without needing
compensatory blocks but also maintains the integrity and performance of the
network architecture.

##  IV Experiments

Figure 2: ResNet20 on Cifar10 Figure 3: ResNet20 on Cifar100 Figure 4:
ResNet56 on Cifar10 Figure 5: ResNet56 Cifar100 Figure 6: ResNet50 ImageNet

###  IV-A Experimental Setup

Datasets. In this study, we primarily focus on three prevalent datasets to
validate the efficiency of our method: CIFAR-10, CIFAR-100, and ImageNet.
CIFAR-10 and CIFAR-100 are known to consist of 60K 32 × 32 RGB images, with
CIFAR-10 segregating them into 10 classes and CIFAR-100 into 100 fine-grained
classes. ImageNet, on the other hand, comprises larger 224 × 224 RGB images
with 1.2M for training and 50K for testing across 1000 distinct object
classes.

Implementation Details. Our implementation involves using ResNet56 for
CIFAR-10 and CIFAR-100 datasets, and ResNet50 for the ImageNet dataset. We
employ Stochastic Gradient Descent (SGD) as the optimization algorithm across
all experiments. To test inference speed, we input a single image to the model
and execute the process 1000 times, calculating the average time to evaluate
the model’s performance post-pruning. Following the block pruning process, we
fine-tune the networks for further optimization. For CIFAR-10 and CIFAR-100,
this fine-tuning phase extends for 50 epochs, starting with an initial
learning rate of 0.001. For the ImageNet dataset, we fine-tune for 10 epochs
with an initial learning rate set at 0.0001. In all cases, we maintain a
consistent weight decay of 0.005 and a batch size of 128. Additionally, a
decay factor of 0.5 is applied to the learning rate at specific epochs: at 20,
30, and 40 for CIFAR-10 and CIFAR-100, and at 5 and 8 for ImageNet.

###  IV-B Main Results

In our study, we present an analysis of accuracy versus inference speed for
all experiments conducted. The data points commence with the original model,
represented by the rightmost point, which includes all blocks. Subsequently,
we incrementally remove one block at a time, progressing towards the left in
our Figs. This process continues until all valid blocks are pruned.

Comparison with Optimum In our analysis, we focus on ResNet20 due to its
manageable size, featuring only six valid blocks eligible for pruning
according to our criteria. This allows for a comprehensive brute-force search
to identify the best combination of blocks for pruning. We compare our
method’s performance against two baseline methods. The results, illustrated in
Fig2, show that our approach outperforms the others on the CIFAR-10 dataset.
On CIFAR-100, our method significantly surpasses the baseline methods as shown
in Fig3, indicating its robustness across different datasets. These findings
suggest that our pruning strategy yields solutions close to the optimal
configuration, demonstrating the efficacy of our approach in efficiently
pruning networks while maintaining high performance.

Deeper Network Exploring the applicability of our pruning method to more
complex architectures, we extended our experiments to deeper networks. The
results from these tests are shown in Fig4 and Fig5. Despite the increased
complexity and depth of these networks, our pruning method consistently
outperformed the baseline methods. This was evident across various metrics,
most notably in maintaining high accuracy while significantly improving
inference speed. The resilience of our approach in handling deeper network
structures underscores its robustness and adaptability. These results
reinforce the effectiveness of our pruning strategy, even as network
complexity scales up, offering a promising solution for optimizing larger,
more intricate models in real-world applications.

Large Scale Dataset To evaluate the scalability of our approach with large-
scale datasets, we applied our pruning method to ResNet50 trained on the
ImageNet dataset. The results of this experiment, as depicted in Fig6, clearly
demonstrate the effectiveness of our method in a more demanding context. Our
approach achieved superior results compared to the baseline methods,
particularly when the number of pruned blocks was less than half of the total
blocks in the network. This indicates not only the efficiency of our pruning
technique in reducing the model size but also its ability to retain high
accuracy, even with significant reduction in network complexity. These
findings highlight the potential of our method in handling large-scale
datasets and complex network architectures, providing an optimal balance
between model size and performance.

##  V Conclusion

In this paper, we introduced a novel block pruning method for deep neural
networks, particularly tailored for edge computing environments. Our approach,
distinct from traditional pruning methods, directly removes blocks to assess
their impact on classification accuracy. This strategy offers a more precise
understanding of each block’s contribution to the overall network performance.
Our extensive experiments with ResNet architectures on CIFAR-10, CIFAR-100,
and ImageNet datasets have demonstrated the effectiveness of our method.
Notably, on large-scale datasets like ImageNet, our approach achieved
significant model size reduction while maintaining high accuracy, even with
substantial network pruning. This underscores our method’s efficacy in
balancing model size and performance, a crucial factor in resource-constrained
scenarios.

## References

  * [1] S. Chen and Q. Zhao, “Shallowing deep networks: Layer-wise pruning based on feature representations,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 12, pp. 3048–3056, 2018.
  * [2] S. Elkerdawy, M. Elhoushi, A. Singh, H. Zhang, and N. Ray, “To filter prune, or to layer prune, that is the question,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , November 2020.
  * [3] H. Tang, Y. Lu, and Q. Xuan, “Sr-init: An interpretable layer pruning method,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [4] K. Zhang and G. Liu, “Layer pruning for obtaining shallower resnets,” _IEEE Signal Processing Letters_ , vol. 29, pp. 1172–1176, 2022.
  * [5] Z. Huang and N. Wang, “Data-driven sparse structure selection for deep neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 304–320.
  * [6] W. Wang, S. Zhao, M. Chen, J. Hu, D. Cai, and H. Liu, “Dbp: Discrimination based block-level pruning for deep model acceleration,” _arXiv preprint arXiv:1912.10178_ , 2019.
  * [7] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [8] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [9] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [10] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [11] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 31, no. 1, 2017.