# Receptive Field Refinement for Convolutional Neural Networks Reliably
Improves Predictive Performance

Mats L. Richter
Mila - Quebec AI Institute
Université de Montréal
Montreal
<EMAIL_ADDRESS>
&Christopher Pal
Mila - Quebec AI Institute
Polytechnique Montreal
Canada CIFAR AI Chair
<EMAIL_ADDRESS>

###### Abstract

Minimal changes to neural architectures (e.g. changing a single hyperparameter
in a key layer), can lead to significant gains in predictive performance in
Convolutional Neural Networks (CNNs). In this work, we present a new approach
to receptive field analysis that can yield these types of theoretical and
empirical performance gains across twenty well-known CNN architectures
examined in our experiments. By further developing and formalizing the
analysis of receptive field expansion in convolutional neural networks, we can
predict unproductive layers in an automated manner before ever training a
model. This allows us to optimize the parameter-efficiency of a given
architecture at low cost. Our method is computationally simple and can be done
in an automated manner or even manually with minimal effort for most common
architectures. We demonstrate the effectiveness of this approach by increasing
parameter efficiency across past and current top-performing CNN-architectures.
Specifically, our approach is able to improve ImageNet1K performance across a
wide range of well-known, state-of-the-art (SOTA) model classes, including:
VGG Nets, MobileNetV1, MobileNetV3, NASNet A (mobile), MnasNet, EfficientNet,
and ConvNeXt – leading to a new SOTA result for each model class.

ptinput imagewith
paddinginputlayerutilizedlayerutilizedlayerunutilizedlayerAll layers are now
fully utilized(better predictive performance)Only fully utilized layers
remain(better predictive performance)adjust downsamplingto slow receptivefield
growthshorter and widerarchitecture

Figure 1: Receptive-field-based refining of a CNN: Layers with oversized
receptive fields are underutilized and therefore inefficient. Slowing the
receptive field (RF) growth reactivates these layers, increasing predictive
performance. Alternatively, such layers can be removed, and the network width
can be adjusted to account for the cut parameters.

##  1 Introduction

Deep learning in computer vision has shown a long-term trend towards bigger
and computationally more expensive architectures to achieve higher predictive
performance [1, 2]. Considering that not every developer has access to the
compute resources necessary for training SOTA-models, the current state of
trial-and-error-heavy development becomes increasingly unsustainable.
Unfortunately, modern Neural Architecture Search (NAS) algorithms suffer from
similar problems, since they require an extensive number of costly model-
training cycles to yield competitive results (e.g. on standard ImageNet1k
evaluations) [3, 4, 1].

In contrast to this, we develop a methodology for improving the predictive
performance of convolutional neural network (CNN) architectures reliably and
cheaply. This is achieved by analyzing the receptive field of the individual
layers and using this information to adapt the model to the input resolution,
removing or reactivating underutilized layers in the process.

To demonstrate this, we apply minor changes derived from our analysis to a
variety of popular architectures (like MobileNetV3, EfficientNet, and ConvNeXt
[5, 6, 7]). Our experiments show that model changes derived through our
approach consistently yield significant performance improvements. Importantly,
our approach is based only on neural architecture properties and the
resolution of input imagery. Therefore, a model does not need to be trained or
even initialized to diagnose network inefficiencies, reducing trial-and-error
greatly. Our proposed procedure is also different from zero-cost NAS, [8],
which can be summarized as forms of low-cost proxies for the performance of
models relative to each other. Instead, we identify and localize
inefficiencies inside a specific architecture and demonstrate how to resolve
these, gaining reliable increases in predictive performance. In short, our
core contributions are as follows:

  * •

We propose a simple analysis method, the “minimal feasible input resolution
($I_{min}$)”, which can be used to quickly and simply uncover design
inefficiencies in CNN architectures (section 3).

  * •

Using $I_{min}$, we uncover design inefficiencies across twenty popular CNN
models, even in architectures specifically designed to be efficient such as
MobileNetV3 [5] and EfficientNet [6] (section 5).

  * •

Based on our method, we propose general strategies for refining these
inefficient architectures to increase predictive performance (section 5). We
also provide an algorithm that automates the proposed analysis in our
supplementary material.

  * •

We resolve the discovered inefficiencies with minimal changes to the CNNs,
reliably improving predictive performance in all cases (section 5.1, section
5.2).

##  2 Related Work

Figure 2: An example for a the superstructure of a CNN, in which the building
blocks are embedded.

The overall topologies of modern CNN architectures mainly orient themselves on
building-block-style models [6, 9, 2] highly driven by convention. This is, to
some degree, necessary, since the search space of neural architectures would
otherwise be excessively large [4]. However, the current convention also
introduces biases into the chosen design directions, favoring for example the
optimization of building blocks over the layout of the overall superstructure
of the model [6, 9, 10, 5, 4, 3, 1, 11, 12, 13]. The superstructure is the
overall layout of the neural architecture, in which the building blocks are
embedded (see example in figure 2). The main feature extractor is subdivided
into multiple stages of building blocks with increasing filter sizes and
separated by downsampling layers, followed by the output head of the model.
Our proposed analysis method for refining neural architectures is primarily
operating on the level of the superstructure by suggesting changes in the
number of layers or the number and positioning of downsampling layers.
Therefore, our approach should be seen not as contrary but as complementary to
the current design processes in CNN-architectures. This allows users to
reliably tap into currently underexplored design-pathways for optimizing the
predictive performance of a CNN architecture.

The receptive field is a widely known property of convolutional neural
networks and has been studied from various angles. The works of [14] and [15]
have explored receptive fields from an analytical perspective. The work of
[14] in particular provides the basic formal definitions and computational
methods that are used throughout this work. An empirical study on the
effective receptive field was conducted by [16], demonstrating that the actual
receptive field is substantially smaller than the formal definition suggests.
All the prior works above on receptive fields have suggested that the
receptive field could be used to optimize neural architectures, but they did
not provide concrete approaches to reliably adjust architectures to improve
performance. At the same time, since the receptive field is such a central
property of CNNs, it could be argued that it has been leveraged implicitly to
improve model performance in many independent cases. For example, invariance
to object size and problem-specific improvements in tasks like cancer
mammography classification [17, 18, 19] and object detection [20, 21, 22] are
commonly achieved by recombining features from layers with heterogeneous
receptive field sizes.
In contrast to more informal, observational, and implicit procedures, we
analyze receptive fields explicitly to improve the predictive quality of CNN
models. Other concrete attempts to explicitly utilize receptive field analysis
to improve existing architectures include [23], who proposed a pruning
strategy to improve predictive performance on low-resolution datasets. Other
work such as DynOPool [24], Shape Adaptor [25] and N-Jet [26] adapt the
downsampling rate (and therefore the receptive field size) that is trained
alongside the network. Our approach deviates from all of these in multiple
significant ways. First, DynOPool, Shape Adaptor, and N-Jet learn the
downsampling procedures during the training process, whereas we use an offline
analysis to guide design decisions made before training. The aforementioned
techniques also alter the model footprint during training, resulting in
potential instabilities, making resource consumption harder to control. In
contrast, our approach is a substantially less complex procedure. It does not
require training a model, since it is purely based on an analysis of the
architecture, and it can be utilized to deliver consistent improvements on a
wide range of models on visually complex datasets like ImageNet1k. Finally,
despite the higher complexity, prior work has not succeeded in demonstrating
competitive increases in CNN predictive performance with contemporary and
current SOTA-models on standard datasets like ImageNet1K, which we do achieve
with our work.

##  3 Our Method

In this section, we elaborate on the formal and theoretical foundation of our
approach. We first introduce the general concept of the receptive field and
the minimum receptive field size. We then briefly discuss unproductive layers
in CNN-architectures, a predictable and localizable inefficiency in a CNN
caused by a mismatch between input resolutions and CNN-architecture. Finally,
we characterize underutilized CNN-architectures based on these predictable
inefficiencies and propose two basic approaches on how to modify the
superstructure of an arbitrary feed-forward CNN architecture to resolve them.

input imageinputlayerparallellayerscombinedlayer

Figure 3: The value for $r_{\ell,\text{max}}$ is computed from the path with
the largest receptive field (blue). $r_{\ell,\text{min}}$ is computed from the
path of smallest receptive field (green).

###  3.1 The Receptive Field

For a sequential, fully convolutional network $L$ with layers
$\ell{}=1,2,3,...,L$, the receptive field of a layer can be described as an
area of shape $(r_{\ell{}}\times r_{\ell{}})$ on the input image. Only
information present in this area can influence the output of a specific
feature, locally restricting the feature extraction. For the first layer, the
receptive field is equivalent to the shape of the kernel $r_{0}=k_{0}$. We are
assuming square kernels in all layers, to simplify the computation.
Consecutive layers with a kernel size $(k_{\ell{}}\times k_{\ell{}})>(1\times
1)$ expand the receptive field by covering multiple input-feature-map
positions. Essentially, $r_{\ell{}}$ grows by $k_{\ell{}}-1$ from the previous
layer. Downsampling accelerates this process further, by reducing the
resolution of the input feature map of the following layers. The lowering of
the resolution increases the effective size of the kernel by the same
cumulative factor: $\prod^{\ell-1}_{i=0}g_{i}$, where $g_{i}$ represents the
stride size of a layer $i$. This allows us to compute the receptive field size
of a layer $\ell{}$ using the following recursive formula:

$r_{\ell}=r_{\ell-1}+(k_{\ell{}}-1)\prod^{\ell-1}_{i=0}g_{i}$ (1)

We substitute $\Delta r_{\ell{}}=(k_{\ell{}}-1)\prod^{\ell-1}_{i=0}g_{i}$.
Since $r_{0}=k_{0}$, we can rewrite the equation non-recursively:

$r_{\ell}=k_{0}+\sum^{\ell}_{i=1}\Delta
r_{\ell{}}=k_{0}+\sum^{\ell}_{i=1}\Bigg{[}k_{i}\prod^{i-1}_{j=0}g_{j}\Bigg{]}$
(2)

While this method allows us to determine the receptive field for sequences of
layers like the VGG-network family, more recent architectures cannot be
described adequately as a sequence of layers. For example, most architectures
feature skip-connections, which effectively represent an alternative pathway
through the network’s structure. In such scenarios, the receptive field is
commonly computed from the path with the largest receptive field expansion
[14, 15, 27, 28, 29, 20], ignoring the rest of the network’s topology leading
to the target layer. We refer to this definition of the receptive field as
$r_{\ell{},\text{max}}$. This value can be interpreted as the upper bound for
the size of features extractable by the layer $\ell$. However, with modern
architectures being very deep, this value commonly exceeds the input
resolution, making it not very useful for determining realistic limits of
feature extraction within the architecture. For example, the
$r_{\ell{},\text{max}}$ of ResNet18’s final layer is 435, which is almost
double the $224\times 224$ pixel input resolution. EfficientNetB7’s final
convolutional layer has a $r_{\ell{},\text{max}}=3079$ while processing an
$600\times 600$ input. We find that the minimum receptive field size
$r_{\ell,\text{min}}$, a lower bound for the most local information, is more
useful for diagnosing and resolving design inefficiencies in neural
architectures [23]. Since we assume the architecture to be non-recurrent,
there is a finite set of sequences $S_{\ell{}}$ leading from the input to the
target layer $\ell$. Computing all receptive field sizes results in $r_{\ell}$
being a set of values. Since this set is always finite when the architecture
is non-recurrent, we can trivially obtain the minimum receptive field size
$r_{\ell,\text{min}}$:

$r_{\ell,\text{min}}=\min(\\{r_{\ell,s}\mid s\in S_{\ell{}}\\})$ (3)
$=\min\Bigg{(}\Big{\\{}k_{0}+\sum^{s}_{i=1}\Bigg{[}k_{i}\prod^{i-1}_{j=0}g_{j}\Bigg{]}\mid
s\in S_{\ell{}}\Big{\\}}\Bigg{)}$ (4)

This process is illustrated on a simple example in Fig. 3, alongside the
analog computation of the $r_{\ell{},\text{max}}$.

###  3.2 Unproductive Layers and the Relationship of Input Resolution and
Neural Architecture

The minimum receptive field size $r_{\ell{},\text{min}}$ represents the lower
bound for the locality of the features extractable by a layer $\ell$. This
implies that a layer should be unable to extract new features when it’s input
already has a receptive field size $r_{\ell{}-1,\text{min}}$ as large or
larger than the image:

$r_{\ell{}-1,\text{min}}\geq I_{res}$ (5)

where $(I_{res}\times I_{res})$ is the resolution of the input image. While
this analysis method is an oversimplification of the inference process, it was
empirically demonstrated to have predictive power: It has been shown that the
linear separability of the latent representations does not improve in a CNN
classifier when passed through a layer with $r_{\ell{}-1,\text{min}}\geq
I_{res}$ [29, 23, 30]. We refer to such layers as unproductive layers, since
they require computational resources at training and inference time, but do
not produce measurable benefits in predictive quality, making them a
localizable inefficiency within the architecture. Removing unproductive layers
has been observed to result in no reduction in predictive performance and an
increase in computational and parameter efficiency [23, 29, 30].

###  3.3 Formalizing Receptive Field Analysis

Previous works [23, 29, 31] have explored unproductive layers primarily on
low-resolution datasets like Cifar10. Since the tested CNN classifiers are
usually optimized for ImageNet1k and other higher-resolution datasets,
unproductive layers occur regularly and in great number, when trained on the
datasets’ low native resolution. However, the resulting inefficiencies cannot
be considered a flaw in the design of the investigated architectures, since
the architectures are transplanted into a low-resolution setup they were not
designed to handle. In this work, we are interested in looking for
unproductive layers that occur in the setup the architectures were originally
designed for. To do this efficiently we propose a simple procedure, to
diagnose inefficient architectures: We define a fully utilized CNN
architecture $A$ as an architecture which can fully utilize the receptive
field expansion of every layer. This can be expressed using the following
condition:

$\forall\ell\in A:r_{\ell,\text{min}}<I_{\text{res}}$ (6)

Since this definition is conditioned on the size of the input image
$I_{\text{res}}$, which can be chosen freely, we can deduct a minimum input
resolution for which this condition holds true:

$I_{min}=\max(\\{r_{\ell,\text{min}}\mid\ell{}\in A\\})$ (7)

Essentially, a CNN-architecture is guaranteed to not have unproductive layers,
when it is trained with an input size of $I_{min}$ or higher, since every
layer can enrich its input with additional context. We choose
$r_{\ell,\text{min}}$ instead of $r_{\ell-1,\text{min}}$, to also exclude
architectures with layers that partially exceed the boundaries of the image
when expanding the receptive field. While these layers have been empirically
shown to advance the quality of the prediction, their contributions seem to
limited by not being able to fully utilize the receptive field expansion due
to the minimum receptive field going beyond the images’ borders [29, 23].
Theoretically, this definition could also be expanded to an interval
($I_{min},I_{max})$ instead of a lower bound, where the largest feasible input
resolution $I_{max}$ describes the largest possible feature that can be
extracted based on the maximum receptive field sizes
$I_{max}=\max_{l}(r_{\ell,max})$. However, we neglect this factor in this
work, since modern architectures usually feature large values for $I_{max}$
far beyond 2000 pixels, which exceeds commonly used input resolutions of
classifiers by orders of magnitude. Thus, this work focuses exclusively on
unproductive layers and on the input size lower bound $I_{min}$.

##  4 Comparison with Shape Adaptor

##  5 Uncovering Design Inefficiencies in Popular Architectures

Table 1: Our list of underutilized architectures according to our analysis of the receptive field. Since $I_{min}\geq I_{res}$ for all networks, lowering $I_{min}$ through architectural changes will result in improving the predictive performance. Architecture | $I_{res}$ | $I_{min}$
---|---|---
MobileNet [32] | $224\times 224$ | $315\times 315$
MobileNetV3 large [5] | $224\times 224$ | $263\times 263$
MobileNetV3 small [5] | $224\times 224$ | $303\times 303$
EfficientNetB0 [6] | $224\times 224$ | $299\times 299$
EfficientNetB1 [6] | $240\times 240$ | $299\times 299$
EfficientNetB2 [6] | $260\times 260$ | $299\times 299$
NASNet A (mobile) [4] | $224\times 224$ | $327\times 327$
VGG19 [33] | $224\times 224$ | $268\times 268$
MnasNet (0.5) [3] | $224\times 224$ | $283\times 283$
MnasNet (0.75) [3] | $224\times 224$ | $283\times 283$
MnasNet (1.0) [3] | $224\times 224$ | $283\times 283$
MnasNet (1.3) [3] | $224\times 224$ | $283\times 283$
ConvNeXt (T) [7] | $224\times 224$ | $224\times 224$
ConvNeXt (S) [7] | $224\times 224$ | $224\times 224$
ConvNeXt (M) [7] | $224\times 224$ | $224\times 224$
ConvNeXt (L) [7] | $224\times 224$ | $224\times 224$
ConvNeXt (XL) [7] | $224\times 224$ | $224\times 224$
Table 2: Our EfficientNet-variants trained on ImageNet1k compared to the original models. Depressing $I_{min}$ has shown to be an effective strategy, both variants of lowering $I_{min}$ are outperforming the original models. Architecture | Variant | Acc@1 | Acc@5 | FLOPs | Params | $I_{res}$ | $I_{min}$
---|---|---|---|---|---|---|---
EfficientNet B0 | original [6] | 77.1% | 93.3% | 0.42B | 5.3M | 224 | 299
| shorter & wider (ours) | 78.3% | 93.9% | 0.72B | 5.5M | 224 | 171
| less downsampling (ours) | 78.5% | 94.4% | 1.78B | 5.3M | 224 | 203
EfficientNet B1 | original [6] | 79.1% | 94.4% | 0.74B | 7.8M | 240 | 299
| shorter & wider (ours) | 79.6% | 94.8% | 1.43B | 7.7M | 240 | 235
| less downsampling (ours) | 80.6% | 95.2% | 1.67B | 7.8M | 240 | 203
EfficientNet B2 | original [6] | 80.1% | 94.9% | 1.06B | 9.2M | 260 | 299
| shorter & wider (ours) | 80.7% | 95.2% | 2.27B | 9.6M | 260 | 235
| less downsampling (ours) | 81.2% | 95.6% | 2.46B | 9.2M | 260 | 203
EfficientNet B3 | original [6] | 81.6% | 95.7% | 1.96B | 12M | 300 | 299
| shorter & wider (ours) | 81.8% | 95.9% | 3.72B | 12M | 300 | 235
| less downsampling (ours) | 82.8% | 96.0% | 4.41B | 12M | 300 | 203
EfficientNet B4 | original [6] | 82.9% | 96.4% | 4.66B | 19M | 380 | 299
| shorter & wider (ours) | 83.4% | 96.7% | 9.27B | 19M | 380 | 235
| less downsampling (ours) | 83.3% | 96.7% | 11.2B | 19M | 380 | 203
EfficientNet B5 | original [6] | 83.6% | 96.7% | 10.8B | 30M | 456 | 299
| shorter & wider (ours) | 83.8% | 96.7% | 23.2B | 32M | 456 | 235
| less downsampling (ours) | 83.8% | 96.8% | 25.8B | 30M | 456 | 203

We proceed to investigate popular architectures by computing their respective
values for $I_{min}$. In table 1 we provide selected results for architectures
with $I_{min}\geq I_{res}$. A full table of all tested architectures can be
found in table 7 in the supplementary material. According to our previous
definition, we consider the parameters in the architectures in table 1
underutilized. These results suggest that gains in predictive performance
should be possible for these architectures without increasing the number of
parameters.

We hypothesize that this can be achieved by lowering $I_{min}$ such that
$I_{min}<I_{res}$, without significantly changing the number of parameters in
the architecture. We propose two distinct procedures, illustrated in figure 1:
First, reactivating underutilized layers by lowering $r_{\ell{},min}$, such
that $r_{\ell{},min}<I_{min}$ for all previously underutilized layers $\ell$.
This is easily achieved by reducing the downsampling in the architecture,
decreasing the growth of the receptive field. Effectively, we change the
stage-layout of the architecture’s superstructure to slow the receptive field
growth, resulting in full utilization of the architecture’s parameters.
Alternatively, we remove the underutilized layers with $r_{\ell,min}\geq
I_{res}$ and distribute the parameters to preceding layers to keep the
capacity of the model constant. This can also be seen as a change in the
network’s superstructure, reducing the depth and increase the width of some
stages without necessarily changing the design of the individual building
blocks.

In both cases, the changes are very minimal and generally require the
alteration of only a few lines of code. Such changes are also more economical
regarding the computational and memory footprint than simply increasing
$I_{res}$ to achieve the same difference between $I_{min}$ and $I_{res}$,
since not the entire architecture’s footprint is scaled quadratically. To
further make exploring the effects of the changes to $I_{min}$ as easy as
possible, we provide code that automates the extraction of receptive field
information directly from PyTorch-models and provide it in our supplementary
material.

###  5.1 Optimizing the EfficientNet Architectures

Table 3: When training EfficientNet-models on ImageNet1k without scaling $I_{res}$, our variants outperform the underutilized original consistently, demonstrating that scaling $I_{res}$ partially cannibalized the effect of depressing $I_{min}$ in table 2. | Var. | Acc@1 | FLOPs | $I_{res}$
---|---|---|---|---
B0 | original [6] | 77.1% | 0.42B | 224
| shorter & wider (ours) | 78.2% | 0.84B | 224
| less downsampling (ours) | 78.5% | 0.94B | 224
B1 | original [6] | 78.8% | 0.62B | 224
| shorter & wider (ours) | 79.4% | 1.24B | 224
| less downsampling (ours) | 80.3% | 1.45B | 224
B2 | original [6] | 79.6% | 0.71B | 224
| shorter & wider (ours) | 80.2% | 1.56B | 224
| less downsampling (ours) | 80.9% | 1.71B | 224
B3 | original [6] | 80.5% | 1.03B | 224
| shorter & wider (ours) | 81.2% | 2.03B | 224
| less downsampling (ours) | 81.8% | 2.40B | 224
B4 | original [6] | 81.2% | 1.59B | 224
| shorter & wider (ours) | 81.9% | 3.17B | 224
| less downsampling (ours) | 82.5% | 3.83B | 224
B5 | original [6] | 81.7% | 2.48B | 224
| shorter & wider (ours) | 82.5% | 5.42B | 224
| less downsampling (ours) | 82.8% | 6.08B | 224
B6 | original [6] | 80.7% | 3.52B | 224
| shorter & wider (ours) | 82.3% | 7.59B | 224
| less downsampling (ours) | 82.9% | 8.68B | 224
B7 | original [6] | 81.8% | 5.39B | 224
| shorter & wider (ours) | 82.2% | 11.4B | 224
| less downsampling (ours) | 83.0% | 13.4B | 224

From table 1, we observe that EfficientNet B0, B1 and B2 are not fully
utilized, since $I_{min}>I_{res}$. EfficientNet models consist of 7 stages
composed of a number of MBConvBlocks with the same number of filters and
kernel-sizes. The first block within stages 2, 3, 4 and 6 are also
downsampling the feature maps. To investigate the effectiveness of lowering
$I_{min}$, we test both strategies proposed in the previous section on
EfficientNet.

Table 4: Classification accuracies of improved architectures on ImageNet1K. In general, shrinking $I_{min}$ below the input resolution $I_{res}$ improves the predictive performance in all tested scenarios. This is true when comparing the performance to the original accuracy achieved by the respective authors and our reproduction using the same training setup as our improved variants. Architecture | Variant | Acc@1 | Acc@5 | FLOPs | Params | $I_{res}$ | $I_{min}$
---|---|---|---|---|---|---|---
VGG19 | original [33] | 74.5% | 92.0% | 19.71B | 143.67M | 224 | 268
| reproduction | 76.1% | 92.9% | 19.71B | 143.67M | 224 | 268
| reduced $I_{min}$ (ours) | 76.3% | 93.0% | 51.71B | 143.67M | 224 | 220
MobileNetV1 | original [32] | 70.6% | not provided | 0.59B | 4.23M | 224 | 315
| reproduction | 73.8% | 91.5% | 0.59B | 4.23M | 224 | 315
| reduced $I_{min}$ (ours) | 74.6% | 92.0% | 1.39B | 4.23M | 224 | 219
MobileNetV3 (small) | original [5] | 67.4% | not provided | 0.06B | 2.54M | 224 | 303
| reproduction | 67.2% | 87.5% | 0.06B | 2.54M | 224 | 303
| reduced $I_{min}$ (ours) | 70.6% | 89.4% | 0.18B | 2.54M | 224 | 175
MobileNetV3 (large) | original [5] | 75.2% | not provided | 0.24B | 5.48M | 224 | 263
| pretrained (21K) [34] | 78.0% | not provided | 0.24B | 5.48M | 224 | 263
| reproduction | 77.1% | 93.5% | 0.24B | 5.48M | 224 | 263
| reduced $I_{min}$ (ours) | 78.5% | 94.2% | 0.38B | 5.48M | 224 | 199
MNASNet(0.5) | original [3] | 68.9% | not provided | 0.12B | 2.21M | 224 | 283
| reproduction | 68.2% | 87.9% | 0.12B | 2.21M | 224 | 283
| reduced $I_{min}$ (ours) | 69.8% | 88.8% | 0.24B | 2.21M | 224 | 219
MNASNet(0.75) | original [3] | 73.3% | not provided | 0.24B | 3.17M | 224 | 283
| reproduction | 73.9% | 91.1% | 0.24B | 3.17M | 224 | 283
| reduced $I_{min}$ (ours) | 74.4% | 91.8% | 0.47B | 3.17M | 224 | 219
MNASNet(1.0) | original [3] | 75.2% | not provided | 0.34B | 4.38M | 224 | 283
| reproduction | 73.5% | 91.1% | 0.34B | 4.38M | 224 | 283
| reduced $I_{min}$ (ours) | 75.7% | 92.6% | 0.73B | 4.38M | 224 | 219
MNASNet(1.3) | original [3] | 77.2% | not provided | 0.56B | 6.28M | 224 | 283
| reproduction | 76.5% | 93.3% | 0.56B | 6.28M | 224 | 283
| reduced $I_{min}$ (ours) | 77.4% | 93.6% | 1.17B | 6.28M | 224 | 219
NASNet A (mobile) | original [4] | 74.0% | 91.6% | 0.59B | 5.29M | 224 | 327
| reproduction | 75.1% | 92.5% | 0.59B | 5.29M | 224 | 327
| reduced $I_{min}$ (ours) | 78.0% | 93.9% | 2.35B | 5.29M | 224 | 165

We first test removing underutilized layers and making the architecture
partially wider to account for the cut parameters: We remove all building
blocks featuring at least one layer with $r_{\ell{},\text{min}}\geq I_{res}$,
which is the entire stage 7 and stage 6 except for the first building block.
To maintain the number of parameters of the original EfficientNetB0
architecture the number of filters in stage 5 are increased from 112 to 165
and in stage 6 from 192 to 320. We refer to this variant as “shorter & wider”.
The second variant is reactivating underutilized layers by reducing
downsampling, as described in the previous section. In this case, the
downsampling is removed from the fourth stage of the model. This change is
sufficient to lower the receptive field growth enough to make the architecture
fully utilized. We refer to this variant as “less downsampling”. The described
variants were made with three objectives in mind. First, conduct minimal
changes to the original architecture’s structure. Second, depress the
receptive field sizes enough such that $I_{min}<I_{res}=224$ and finally for
$I_{min}$ to be as close to $I_{res}$ as possible. The latter constraint is
used to avoid some limitations of this analysis method. The authors of [7]
have demonstrated that having very low receptive fields by training completely
downsampling-free models has a negative effect on performance. We therefore
suspect that there is a soft lower-limit to beneficial effects of depressing
the receptive field. For this reason, we make changes that result in $I_{min}$
being as close to $I_{res}$ as possible while still being smaller. To obtain
modified variants of EfficientNetB1 and higher, we use the same scaling
strategy as the original EfficientNet and apply it on our variants. Please
note that the differently sized variants mostly share a value for $I_{min}$,
which is due to skip connections in the building blocks providing pathways
that do not expand the receptive field, thereby keeping
$r_{\ell{},\text{min}}$ low.

From the results in table 2, we observe that both modifications improved the
top1 and top5 accuracy of the respective model. In some cases, like
EfficientNetB1 the reduced striding resulted in performance higher than the
original B2-version of the model. Interestingly, the computationally more
expensive change yields higher performance gains. Quintessentially, an
increase in predicting performance and parameter-efficiency is achieved by
increasing the computational footprint of the model.

EfficientNet models from B3 to B5 were fully utilized due to their higher
$I_{res}$. Our variants also outperform these models, but to a lesser degree,
since the increasing $I_{res}$ is cannibalizing the effects of lowering
$I_{min}$ This can be demonstrated further, by training all EfficientNet
models by not scaling $I_{res}$ with the architecture. In this setup, all the
original EfficientNet architectures are not fully utilized and are performing
consistently worse than both improved versions (see table 3). In both cases,
the more computationally expensive changes that resolved underutilization
yields higher accuracies, which is consistent with prior observations.
Finally, the depicted results outperform variants of EfficientNetB0 that use
DynOPool [24], a dynamic pooling operator automatically adjusting the model’s
receptive field, which only achieved 72.8% top1-accuracy for EfficientNetB0.
We provide further comparisons with similar approaches like Shape Adaptor [25]
in the supplementary material.

###  5.2 Optimizing other Popular Architectures

To further demonstrate the effectiveness of our approach we test different
modification procedures on the other architectures shown to be not fully
utilized based on table 1. In this section, we primarily focus on changing the
downsampling within an architecture, since these changes were more minimally
invasive and yield higher predictive performance. In an effort to further
showcase the robustness of this procedure, we try in all cases to keep the
alterations as minimal as possible to depress $I_{min}$, while also varying
the approach from architecture to architecture. For all tested models, we only
ever started and completed training on a single modified variant and no
hyperparameter optimization was conducted, making this refinement process
completely trial-and-error free. This is possible, since $I_{min}$ and
$I_{res}$ are known before training, allowing to alter the architecture in the
intended way without the need for training. The changes made to each
architecture are as follows:

Table 5: Classification accuracies of improved ConvNeXt on ImageNet1k. Similar to other architectures, ConvNeXt improves substantially by lowering $I_{min}$. Architecture | Variant | Acc@1 | Acc@5 | FLOPs | Params | $I_{res}$ | $I_{min}$
---|---|---|---|---|---|---|---
ConvNeXt T | original [7] | 82.1% | not provided | 4.46B | 28.58M | 224 | 224
| reduced $I_{min}$ (ours) | 83.1% | 96.4% | 17.81B | 28.58M | 224 | 112
ConvNeXt S | original [7] | 83.1% | not provided | 8.70B | 50.22M | 224 | 224
| reduced $I_{min}$ (ours) | 84.0% | 96.6% | 34.76B | 50.22M | 224 | 112
ConvNeXt B | original [7] | 83.8% | not provided | 15.38B | 88.59M | 224 | 224
| reduced $I_{min}$ (ours) | 84.3% | 96.9% | 61.45B | 88.56M | 224 | 112
| original (finetuned) [7] | 85.1% | not provided | 45.19B | 88.59M | 384 | 224
| reduced $I_{min}$ (ours) (finetuned) | 85.5% | 97.5% | 180.59B | 88.56M | 384 | 112
| original (21K finetuned) [7] | 86.8% | not provided | 45.19B | 88.59M | 384 | 224
| reduced $I_{min}$ (21K finetuned) (ours) | 87.1% | 98.2% | 180.59B | 88.56M | 384 | 112
ConvNeXt L | original [7] | 84.3% | not provided | 34.39B | 197.77M | 224 | 224
| reduced $I_{min}$ (ours) | 84.7% | 97.1% | 137.49B | 197.76M | 224 | 112
| original (finetuned) [7] | 85.5% | not provided | 101.08B | 197.77M | 384 | 224
| reduced $I_{min}$ (finetuned) (ours) | 85.7% | 97.6% | 404.06B | 197.76M | 384 | 112

  * •

VGG19 $\Delta I_{min}:=269\rightarrow 220$
The first MaxPooling layer is placed after the fourth convolutional layer.
Afterwards, pooling is applied after very three convolutional layers.

  * •

MobileNetV1 $\Delta I_{min}:=315\rightarrow 219$
Downsampling occurs now in the layers 1, 3, 5 and 11 instead of 1, 3, 5, 7,
13.

  * •

MobileNetV3 (Small) $\Delta I_{min}:=303\rightarrow 175$
Changed the third downsampling layer’s stride size from 2 to 1.

  * •

MobileNetV3 (Large) $\Delta I_{min}:=263\rightarrow 199$
Changed the last downsampling layer’s stride size from 2 to 1.

  * •

MNASNet (all variants) $\Delta I_{min}:=283\rightarrow 219$
Changed the last downsampling layer’s stride size from 2 to 1.

  * •

NASNet A (mobile) $\Delta I_{min}:=327\rightarrow 165$
Changed stride size of the first stem layer from 2 to 1.

  * •

ConvNeXt (all variants) $\Delta I_{min}:=224\rightarrow 112$
Changed the patching layer from $4\times 4$ patches to $2\times 2$ patches.

We train all models on ImageNet1k using the standard configuration described
in table 6 of the supplementary material. Additionally, to account for the
influence on hyperparameters, we train the original architectures using the
same setup and report the performance results alongside the results of the
original authors and our improved models in table 4: For all tested models,
our improved variant reliably outperforms the original model and our
reproduction of the original model using our standard training setup.
Therefore, we see this as further confirmation that depressing $I_{min}$ below
the input resolution to fully utilize the network is a robust analysis method
for reliably optimizing architectures. Finally, we would like to highlight the
results of ConvNeXt in table 5. Not only were we able to outperform ConvNeXt
on all size variants using the standard ImageNet1K training, but also when
fine-tuning ConvNeXt B and L on a higher resolution. We were also able to
outperform ConvNeXt B and L when pretraining on ImageNet21k (see supplementary
material). To our knowledge this is achieving a new state-of-the-art for CNN-
models that did not use complex pretraining procedures [35], distilling [36]
or massive pretraining datasets like JFT300M [37, 2].

##  6 Summary and Discussion

In conclusion, we have shown that the inductive bias of CNNs can be
effectively used to guide the practitioner’s design decisions regarding the
network’s superstructure. We derive a simple but effective analysis method to
guide the performance oriented refinement of CNN-superstructures. Using this
analysis method, we uncover design inefficiencies in various popular
architectures, demonstrating the utility of finding and fixing issues with
underutilized layers. Perhaps surprisingly, this analysis is even effective at
identifying underutilized layers across a wide variety of the highest
performing modern architectures. By applying different strategies to resolve
those inefficiencies we can improve the predictive performance of all tested
models on the first attempt, achieving new state-of-the-art results in the
process, without increasing the number of parameters in the model. While we
universally increased the parameter-efficiency of our models, this came at
increased computational cost, indicating that an inherent trade-off between
parameter and computational efficiency exists specific for each model class.
This is interesting from the perspective of scaling laws [38], since we
effectively demonstrate that parameters can be substituted for additional
compute and do not necessarily need to be scaled together to be effective.

When practically applying these ideas to potential changes to a CNN
architecture, decisions are influenced by current model inefficiencies, the
superstructure of the architecture, the compute budget for training the model
and the desired improvement in predictive performance. In this context, the
advantage of our approach is that choosing to spend additional computational
resources for increased predictive performance can be made intentionally and
reliably, not requiring expensive trial-and-error based comparative evaluation
between model variants.

To make our receptive field analysis approach more useful and accessible we
have created an implementation compatible with TensorFlow and PyTorch, which
will be provided in our supplementary material and via PyPi.

##  7 Acknowledgement

This work was supported by a fellowship within the IFI program of the German
Academic Exchange Service (DAAD). We would also like to thank Google’s TPU
Research Cloud (TRC) for Cloud TPUs used in this research.

## References

  * [1] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le.  Regularized evolution for image classifier architecture search.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4780–4789, Jul. 2019.
  * [2] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan.  High-performance large-scale image recognition without normalization, 2021\.
  * [3] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le.  Mnasnet: Platform-aware neural architecture search for mobile.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2815–2823, 2019.
  * [4] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018.
  * [5] Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu.  Searching for mobilenetv3.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1314–1324. IEEE, 2019.
  * [6] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 2019.
  * [7] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [8] Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas Donald Lane.  Zero-cost proxies for lightweight {nas}.  In International Conference on Learning Representations, 2021.
  * [9] Mingxing Tan and Quoc V. Le.  Efficientnetv2: Smaller models and faster training.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 2021.
  * [10] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4510–4520, 2018.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, pages 630–645. Springer, 2016.
  * [13] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1–87.12. BMVA Press, September 2016.
  * [14] André Araujo, Wade Norris, and Jack Sim.  Computing receptive fields of convolutional neural networks.  Distill, 2019.  https://distill.pub/2019/computing-receptive-fields.
  * [15] Hung Le and Ali Borji.  What are the receptive, effective receptive, and projective fields of neurons in convolutional neural networks?, 2017.
  * [16] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 4905–4913, Red Hook, NY, USA, 2016. Curran Associates Inc.
  * [17] Alexander Rakhlin, Alexey Shvets, Vladimir Iglovikov, and Alexandr A Kalinin.  Deep convolutional neural networks for breast cancer histology image analysis.  In international conference image analysis and recognition, pages 737–744. Springer, 2018.
  * [18] Nan Wu, Jason Phang, Jungkyu Park, Yiqiu Shen, Zhe Huang, Masha Zorin, Stanislaw Jastrzebski, Thibault Févry, Joe Katsnelson, Eric Kim, Stacey Wolfson, Ujas Parikh, Sushma Gaddam, Leng Leng Young Lin, Kara Ho, Joshua D. Weinstein, Beatriu Reig, Yiming Gao, Hildegard Toth, Kristine Pysarenko, Alana Lewin, Jiyon Lee, Krystal Airola, Eralda Mema, Stephanie Chung, Esther Hwang, Naziya Samreen, S. Gene Kim, Laura Heacock, Linda Moy, Kyunghyun Cho, and Krzysztof J. Geras.  Deep neural networks improve radiologists’ performance in breast cancer screening.  IEEE Trans. Medical Imaging, 39(4):1184–1194, 2020.
  * [19] Yiqiu Shen, Nan Wu, Jason Phang, Jungkyu Park, Kangning Liu, Sudarshini Tyagi, Laura Heacock, S. Gene Kim, Linda Moy, Kyunghyun Cho, and Krzysztof J. Geras.  An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization.  Medical Image Anal., 68:101908, 2021.
  * [20] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 936–944, 2017.
  * [21] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement, 2018.
  * [22] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection, 2020.
  * [23] Mats L. Richter, Julius Schöning, and Ulf Krumnack.  Should You Go Deeper? Optimizing Convolutional Neural Network Architectures without Training by Receptive Field Analysis (in print).  In 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021, Miami, FL, USA, December 13-16, 2021. IEEE, 2021\.
  * [24] Dong-Hwan Jang, Sanghyeok Chu, Joonhyuk Kim, and Bohyung Han.  Pooling revisited: Your receptive field is suboptimal.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 549–558, June 2022.
  * [25] Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi, and Edward Johns.  Shape adaptor: A learnable resizing module.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV (12), volume 12357 of Lecture Notes in Computer Science, pages 661–677. Springer, 2020.
  * [26] Silvia L. Pintea, Nergis Tömen, Stanley F. Goes, Marco Loog, and Jan C. van Gemert.  Resolution learning in deep convolutional networks using scale-space theory.  Trans. Img. Proc., 30:8342–8353, jan 2021.
  * [27] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016.
  * [28] Khaled Koutini, Hamid Eghbal-zadeh, and Gerhard Widmer.  Receptive field regularization techniques for audio classification and tagging with deep convolutional neural networks.  IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:1987–2000, may 2021.
  * [29] Mats L. Richter, Wolf Byttner, Ulf Krumnack, Anna Wiedenroth, Ludwig Schallner, and Justin Shenk.  (Input) Size Matters for CNN Classifiers.  In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter, editors, Artificial Neural Networks and Machine Learning – ICANN 2021, pages 133–144, Cham, 2021. Springer International Publishing.
  * [30] Mats L. Richter, Justin Shenk, Wolf Byttner, Anders Arpteg, and Mikael Huss.  Feature Space Saturation during Training (in print).  In "32st British Machine Vision Conference 2021, BMVC 2021, Virtual Event, UK, November 22-25, 2021". BMVA Press, 2021.
  * [31] Mats L. Richter, Leila Malihi, Anne-Kathrin Patricia Windler, and Ulf Krumnack.  Exploring the Properties and Evolution of Neural Network Eigenspaces during Training.  In "The 12th Iranian and the second International Conference on Machine Vision and Image Processing MVIP (in print)". IEEE explore, 2022\.
  * [32] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  CoRR, abs/1704.04861, 2017.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [34] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * [35] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models, 2022.
  * [36] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021.
  * [37] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [38] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  CoRR, abs/2001.08361, 2020.
  * [39] Ross Wightman, Hugo Touvron, and Herve Jegou.  Resnet strikes back: An improved training procedure in timm.  In NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
  * [40] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18613–18624. Curran Associates, Inc., 2020.
  * [41] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [42] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [43] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):13001–13008, Apr. 2020.
  * [44] Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le.  Autoaugment: Learning augmentation strategies from data.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 113–123. Computer Vision Foundation / IEEE, 2019.
  * [45] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.  Deep networks with stochastic depth, 2016.
  * [46] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017.
  * [47] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1800–1807, 2017.
  * [48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015.
  * [49] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size, 2016.  cite arxiv:1602.07360Comment: In ICLR Format.
  * [50] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 4278–4284. AAAI Press, 2017.
  * [51] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016.
  * [52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [53] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.

##  Appendix A Codebases for training models

For training ConvNeXt we use the official repository published by the original
authors:
https://github.com/facebookresearch/ConvNeXt.
The remaining models are trained using the training-script of the timm-
library:
https://github.com/rwightman/pytorch-image-models.git.

##  Appendix B Training Setups

In this section, we briefly discuss the training setup used for all trainings
throughout this work. The first configuration in table 6 is strongly inspired
by the A2 hyperparameter-configuration by [39]. We use this configuration,
since it reflects a state-of-the-art setup for CNN-classifiers in computer
vision [2, 7, 6]. It is also known to generalize well over a wide set of
models, achieving very competitive performance on old and new architectures
[39]. For this reason, we see this setup as a good common testing ground to
evaluate the effects of our architectural changes.

Since ConvNeXt is the only model performing substantially worse on this
configuration, we use its original setup and codebase [7] for training, pre-
training and fine-tuning. This also allows us to very directly compare our
improved version with the original models.

However, when pretraining on ImageNet21K, there is one key-derivation: The
original authors use the Fall11 release of the unprocessed dataset, which is
no longer available. We use the Winter21-version of ImageNet21K-P, which was
the only version of this dataset available to us. Based on the ablative
studies of [34] on the different releases of ImageNet21K this change likely
reduced the performance of our pretrained models slightly (see supplementary
material of [34]). However, since our models still outperform the original
architectures we argue that the results still support the claims made in this
work.

Table 6: Hyperparameter settings used throughout this paper. The standard setup did not produce a competitive result for ConvNeXt-models, which is the reason why it is trained on its original setup and codebase published by the authors [7]. training config | Standard Setup | ConvNeXt | ConvNeXt (finetuning 1K) | ConvNeXt (finetuning 21K)
---|---|---|---|---
weight init | $\mathcal{U}(-\sqrt{k},\sqrt{k})$ | trunc. normal (0.2) | finetuning | finetuning
optimizer | Lamb | AdamW | AdamW | AdamW
learning rate | 5e-3 | 4e-3 | 5e-5 | 5e-5
lr. decay | cosine | cosine | cosine | cosine
weight decay | 0.02 | 0.02 | 1e-8 | 1e-8
opt. momentum | - | $\beta_{1},\beta_{2}=0.9,0.999$ | $\beta_{1},\beta_{2}=0.9,0.999$ | $\beta_{1},\beta_{2}=0.9,0.999$
batch size | 2048 | 4096 | 512 | 512
epochs | 300 | 300 | 30 | 30
warmup epochs | 5 | 20 | 0 | 0
warmup sched. | linear | linear | N/A | N/A
randaug [40] | None | (9, 0.5) | (9, 0.5) | (9, 0.5)
mixup [41] | 0.1 | 0.8 | None | None
cutmix [42] | 1.0 | 1.0 | None | None
rand. erasing [43] | 0.0 | 0.25 | 0.25 | 0.25
color jitter | 0.4 | None | None | None
crop pct. | 0.95 | 0.95 | 0.95 | 0.95
auto aug. [44] | False | True | True | True
smoothing [27] | 0.0 | 0.1 | 0.1 | 0.1
stoc. depth [45] | 0.2 | 0.1/0.4/0.5/0.5 | 0.8/0.95 | 0.2
Loss | CE | BCE | BCE | BCE

##  Appendix C Computing $I_{min}$ and $I_{res}$ for various popular
architectures.

In table 1, in the main part of this work, we present the respective $I_{min}$
for various popular architectures with $I_{min}\geq I_{res}$. In table 7 we
present the results for all architectures tested during this work.

Table 7: Minimum feasible input resolution $I_{min}$ and the design resolution
$I_{res}$ of popular CNN architectures. Architectures with $I_{min}\geq
I_{res}$ are not fully utilizing all layers and are marked red in this table.

architecture |  $I_{res}$ |  $I_{min}$ |  |  architecture |  $I_{res}$ |  $I_{min}$
---|---|---|---|---|---|---
DenseNet121 [46] |  $224\times 224$ |  $103\times 103$ |  |  MobileNet [32] |  $224\times 224$ |  $315\times 315$
DenseNet161 [46] |  $224\times 224$ |  $103\times 103$ |  |  MobileNetV2 [10] |  $224\times 224$ |  $163\times 163$
DenseNet169 [46] |  $224\times 224$ |  $103\times 103$ |  |  MobileNetV3 large [5] |  $224\times 224$ |  $263\times 263$
DenseNet201 [46] |  $224\times 224$ |  $103\times 103$ |  |  MobileNetV3 small [5] |  $224\times 224$ |  $303\times 303$
EfficientNetB0 [6] |  $224\times 224$ |  $299\times 299$ |  |  Xception [47] |  $299\times 299$ |  $135\times 135$
EfficientNetB1 [6] |  $240\times 240$ |  $299\times 299$ |  |  NASNet A (large) [4] |  $331\times 331$ |  $327\times 327$
EfficientNetB2 [6] |  $260\times 260$ |  $299\times 299$ |  |  NASNet A (mobile) [4] |  $224\times 224$ |  $327\times 327$
EfficientNetB3 [6] |  $300\times 300$ |  $299\times 299$ |  |  ResNet18 [11] |  $224\times 224$ |  $139\times 139$
EfficientNetB4 [6] |  $380\times 380$ |  $299\times 299$ |  |  ResNet34 [11] |  $224\times 224$ |  $139\times 139$
EfficientNetB5 [6] |  $456\times 456$ |  $299\times 299$ |  |  ResNet50$\dagger$ [11] |  $224\times 224$ |  $96\times 96$
EfficientNetB6 [6] |  $528\times 528$ |  $299\times 299$ |  |  ResNet101$\dagger$ [11] |  $224\times 224$ |  $96\times 96$
EfficientNetB7 [6] |  $600\times 600$ |  $299\times 299$ |  |  ResNet152$\dagger$ [11] |  $224\times 224$ |  $96\times 96$
VGG11 [33] |  $224\times 224$ |  $150\times 150$ |  |  GoogLeNet [48] |  $224\times 224$ |  $123\times 123$
VGG13 [33] |  $224\times 224$ |  $156\times 156$ |  |  SqueezeNet 1.0 [49] |  $224\times 224$ |  $67\times 67$
VGG16 [33] |  $224\times 224$ |  $212\times 212$ |  |  SqueezeNet 1.1 [49] |  $227\times 227$ |  $63\times 63$
VGG19 [33] |  $224\times 224$ |  $268\times 268$ |  |  InceptionResNetV2 [50] |  $299\times 299$ |  $143\times 143$
MnasNet (0.5) [3] |  $224\times 224$ |  $283\times 283$ |  |  ConvNeXt (T) [7] |  $224\times 224$ |  $224\times 224$
MnasNet (0.75) [3] |  $224\times 224$ |  $283\times 283$ |  |  ConvNeXt (S) [7] |  $224\times 224$ |  $224\times 224$
MnasNet (1.0) [3] |  $224\times 224$ |  $283\times 283$ |  |  ConvNeXt (M) [7] |  $224\times 224$ |  $224\times 224$
MnasNet (1.3) [3] |  $224\times 224$ |  $283\times 283$ |  |  ConvNeXt (L) [7] |  $224\times 224$ |  $224\times 224$
InceptionV3 [51] |  $299\times 299$ |  $239\times 239$ |  |  ConvNeXt (XL) [7] |  $224\times 224$ |  $224\times 224$

##  Appendix D Exploring Finetuning Procedures Using our Improved ConvNeXt B

In this section, we briefly explore the effects of different finetuning
procedures on our improved version of ConvNeXt, to demonstrate the robustness
of our approach. The depicted examples of pretraining and finetuning are using
the same configurations and procedures described by the original authors [7].
We compare our model with the results of the original authors in table 8. From
the results we observe that our model is able to outperform ConvNeXt B in all
tested scenarios, which involves training from scratch as well as pretraining
from ImageNet1K and 21K. Please note that we do not test ConvNeXt L, which we
briefly mentioned in the main text of this work, this is an unfortunate typo.

Table 8: Classification accuracies of improved ConvNeXt B on ImageNet1K. Improvements from lowering $I_{min}$ can be observed using different pretraining and finetuning procedures. Architecture | Variant | Acc@1 | Acc@5 | FLOPs | Params | $I_{res}$ | $I_{min}$
---|---|---|---|---|---|---|---
ConvNeXt B | original [7] | 83.8% | not provided | 15.38B | 88.59M | 224 | 224
(training from scratch) | reduced $I_{min}$ (ours) | 84.3% | 96.9% | 61.45B | 88.56M | 224 | 112
ConvNeXt B | original [7] | 85.1% | not provided | 45.19B | 88.59M | 384 | 224
(pretraining on ImageNet1K) | reduced $I_{min}$ (ours) | 85.5% | 97.5% | 180.59B | 88.56M | 384 | 112
ConvNeXt B | original [7] | 85.8% | not provided | 15.38B | 88.59M | 224 | 224
(pretraining on ImageNet21K-P) | reduced $I_{min}$ (ours) | 86.0% | 97.9% | 61.45B | 88.59M | 224 | 112
ConvNeXt B | original [7] | 86.8% | not provided | 45.19B | 88.59M | 384 | 224
(pretraining on ImageNet21K-P) | reduced $I_{min}$ (ours) | 87.1% | 98.2% | 180.59B | 88.56M | 384 | 112

##  Appendix E Comparison to DynOPool and Shape Adaptor

Table 9: Comparison of DynOPool and Shape Adaptor with our approach. Model | Method | Cifar100 | Cifar10
---|---|---|---
ResNet50 | DynOPool [24] | 80.6% | not provided
| Shape Adaptor [25] | 80.3% | 95.5%
| Human [25] | 78.5% | 95.5%
| reduced $I_{min}$ (ours) | 78.8% | 95.9%
VGG16 | DynOPool [24] | 79.8% | not provided
| Shape Adaptor | 79.2% | 95.4%
| Human [25] | 75.4% | 94.1%
| reduced $I_{min}$ (ours) | 77.9% |  95.7%
MobileNetV2 | DynOPool [24] | 76.2% | not provided
| Shape Adaptor [25] | 75.7% | 93.9%
| Human [25] | 73.8% | 93.7%
| reduced $I_{min}$ (ours) | 78.7% | 95.3%

DynOPool [24] as well as Shape Adaptor [25] directly adapt the receptive field
by altering the downsampling procedures during training without changing the
number of parameters. In this section, we briefly compare the effectiveness of
our guided manual approach to these automated procedures.

For this comparison, we use Cifar10 and Cifar100, training at the datasets’
native resolution of $32\times 32$ pixels. There are two reasons for this:
First DynOPool as well as Shape Adaptor focus on low-resolution datasets in
their experiments. Second, all three architectures used by the authors of both
works (ResNet50, MobileNetV2 and VGG16) have $I_{min}<I_{res}$ for their
respective ImageNet1K setups (see table 7), which makes them fully utilized
according to our definition. However, $I_{res}=32\leq I_{min}$ for Cifar10 and
Cifar100, making all three architectures very inefficient in this setup,
allowing us to optimize these architectures. For training, we reuse the same
setup as for our other experiments, reducing the batch size to 128 images.
Like in our previous experiments, we only implement and train a single model
variant for each architecture. We train improved variants with
$I_{min}<I_{res}$ for MobileNetV2, ResNet50 and VGG16 and compare the results
to DynOPool and Shape Adaptor variants in table 9.

From the results we observe that our approach generally provides an
improvement over the human baseline and also improves upon DynOPool and Shape
Adaptor. However, in two cases, Resnet50 and VGG16 on Cifar100, it was
impossible to provide similar predictive performance, even when trying to
match $I_{min}$ of our variant with $I_{min}$ of the Shape Adaptor variant of
the same model, this performance could not be achieved. We attribute this to
the fact that the automated approaches use interpolation instead of
downsampling, allowing the model to more gradually increase the receptive
field growth. This could be interesting for future work, since interpolation
as a downsampling layer is not commonly used outside the investigated
approaches. Interestingly, all models produced by Shape Adaptor that we
investigated in greater detail provided architectures with $I_{min}<I_{res}$,
which we see as a confirmation of our approach. The exact modification for
refining the architecture to be fully utilized in Cifar10 and Cifar100 can be
found below:

  * •

ResNet50 $\Delta I_{min}:=96\rightarrow 6$
The stem is replaced with a $3\times 3$ convolution, mimicking the original
Cifar-ResNet by [11]. The only downsampling block left is the first bottleneck
block with 1024 output filters / 256 filters in the $3\times 3$ convolution.

  * •

VGG16 $\Delta I_{min}:=212\rightarrow 30$
Removed all pooling layers. Placed a single MaxPooling-Layer before the 3
final Conv-Layers of the feature extractor.

  * •

MobileNetV2 $\Delta I_{min}:=163\rightarrow 29$
Removed downsampling from all layers except the one closest to the output.

##  Appendix F Receptive Field Analysis Toolbox

We provide RFA-Toolbox
https://anonymous.4open.science/r/receptive_field_analysis_toolbox-1CDD/README.md,
a simple package to conduct receptive field analysis. Since the implementation
has been a substantial amount of work and required (among other things) the
reverse engineering of undocumented functions of the PyTorch-JIT-compiler, we
provide some more detailed overview over the package:

###  F.1 Overview

Figure 4: Software Architecture of RFA-Toolbox.

In this section, we provide an overview on the inner workings of our tooling
for extracting receptive field information from neural architectures. Before
we discuss the individual functionalities in greater detail, we provide a more
general overview over RFA-Toolbox to put the components into context. The
structure of the Python-package is depicted in figure 4. The key component of
RFA-Toolbox is a proprietary model-graph, which contains information critical
for computing the receptive field. All other functionalities of RFA-Toolbox
operate directly on this graph structure instead of the original model. Any
model can be directly implemented in a declarative style using the proprietary
graph of RFA-Toolbox. However, we primarily see this as a fallback-option,
since RFA-Toolbox also allows ingesting architectures from TensorFlow and
PyTorch directly from the API, creating a proprietary graph in the process.
Obtaining a graph from a model is handled by a set of encoders, which are
specialized to handle one specific framework each. This makes RFA-Toolbox
independent of the framework of the model. It also allows us to add support
for additional frameworks in the future without needing to implement
functionalities of the API for multiple frameworks separately. Besides the
aforementioned processing of models into the proprietary graphs, all other
exposed functions of the API are operating on a provided graph-instance to
extract a specific piece of information like obtaining all nodes of
unproductive and underutilized layers. In summary, the API of the RFA-Toolbox
currently provides four basic functionalities:

  1. 1.

Ingesting a model from a known framework like PyTorch and TensorFlow [52, 53]
and extract relevant information into a proprietary graph representation (see
section F.2)

  2. 2.

Finding unproductive and underutilized layers in a given architecture based on
our analysis method (see section F.4)

  3. 3.

Compute $I_{min}$ (see section F.4)

  4. 4.

Visualizing the architecture and highlighting unproductive and underutilized
layers (see section F.4)

###  F.2 Ingesting Models into RFA-Toolbox

One of the key reasons why RFA-Toolbox is easy to use is that models from
common frameworks like PyTorch and TensorFlow can directly be processed. We
refer to this process of extracting information about the architecture into
our proprietary graph-structure as ingesting a model. We will first describe
the proprietary graph-structure in greater detail, which is used for all
internal processing. This is followed by a description of the different
encoders’ implementations, which handle the process of ingestion for
TensorFlow and PyTorch respectively.

###  F.3 Proprietary Graph Structure

The proprietary graph structure is not a fully functioning framework for
implementing neural networks. Only some critical peaces of information are
extracted from the original model. This involves the overall topology of the
model as well as information that is either relevant for the receptive field
computation like kernel and stride sizes or commonly depicted in
visualizations like the name of the layer, the number of units etc. However,
the amount of information may be extended in the future to accommodate
decoding a (potentially optimized) architecture back to a functioning
framework. One other important functionality of the graph is the receptive
field computation. All necessary computations to obtain all receptive field
sizes from all sequences leading to a node are done automatically during the
instantiation of each node, based on the receptive-field information held by
the node’s direct predecessors. The graph is constructed bottom-up from the
input to one or more outputs. Models with more than a single input are
currently not supported. Edges are implicitly defined by lists containing
pointers to direct predecessor and successor nodes held by each individual
node. The nodes also contain functionality that allows them to decide whether
they are unproductive and underutilized layers.

####  F.3.1 Keras / Tensorflow

The encoder for TensorFlow is compatible exclusively with keras.Model, which
is currently the default way of implementing models in TensorFlow. RFA-Toolbox
obtains the structure and meta-information from the keras.Model by decoding
its json-String representation. This decoding process is based on a set of
handlers that are called for each node in a specific sequence.

Figure 5: The information extraction process in the Keras and PyTorch-
Encoders.

Each handler can process a defined set of nodes in the compute-graph, and
every node will be processed by exactly one handler. The handlers will attempt
to process the node in the order of their specificity. More specific handlers
can only process a narrow subset of layers, for example only convolutional
layers, but can extract more metadata from the respective layers. This process
is illustrated in figure 5. The final handler is capable of processing any
node, but will not extract any information from that node except for the name.
Layers that are processed like this are treated like convolutional layers with
a kernel and stride size of 1 on all axes. This is done to make these layers
neutral for the computation of the receptive field and the future growth of
the receptive field. By studying various implementations of models in
TensorFlow, we find that this process can handle all pre-implemented
classifier networks in the TensorFlow library and compute their receptive
field sizes correctly. The aforementioned list of handlers is mutable and can
be expanded and modified manually by the user if required. This could be
necessary if novel custom layers are not processed correctly or require
certain additional information to be extracted.

####  F.3.2 PyTorch

Different from TensorFlow, PyTorch operates on a dynamically generated compute
graph, which cannot be trivially extracted from the model itself. Furthermore,
torch.nn.Module-instances, the basic building blocks of neural networks in
PyTorch, may contain additional modules and complex processing logic. Thus, to
obtain the structure of the compute graph, we need to partially reverse
engineer the JIT-compiler of PyTorch. The JIT-compiler of PyTorch can trace
the path of a tensor of data through the neural network, thereby constructing
a graph of all layers and function calls that are involved in the processing
of this tracing-tensor. By using the string-representation of this JIT-
compiled graph, we can reconstruct the order of function-calls and modules
that processed the data. In essence, the graph allows us to reconstruct the
neural architecture’s topology. From the naming of the nodes in the compute
graph, we can map the nodes of the compute graph to the torch.nn.Module-
instances in the network. It is also possible to obtain function-calls and
their parameters from the graph. This is crucial when functional-versions of
layers that can affect the receptive field analysis, like pooling or
convolutional layers, are used instead of their torch.nn.Module-based
counterparts. However, one minor draw-back of this method is that many nodes
in the compute-graph are present for purely technical reasons, making the
resulting graph very hard to read and more complex than necessary. To enhance
the readability of the graph, multiple post-processing steps remove such nodes
from the graph. These post-processing steps are not removing any computation
that would affect the result of the receptive field analysis, so the results
are consistent with the same model implemented in TensorFlow. The extraction
of meta-information, while technically more complex, is utilizing the same
handler-pattern described for the processing of TensorFlow-models.
Additionally, due to the nesting of modules, RFA-toolbox provides a simple
interface as well to allow the easy modification of handlers for PyTorch
models. This allows the user to view PyTorch-models in different levels of
abstraction, which may be useful when visualizing very complex architectures.

However, the reliance of this process on the JIT-compiler also puts
limitations on the functionality of the encoder. Since the graph-structure is
dependent on the trace of a data-tensor being propagated through the network,
structural changes that occur based on the state of the model are only
reflecting the state of the model at the time the tracing took place. This
means that only the computations and modules in the control-flow are
recognized that are executed by the traced data. Loops are entirely
incompatible with this kind of extraction process. However, while this seems
like a severe limitation, only a single model-family (DenseNet by [46]) from
the torchvision-library is incompatible with RFA-Toolbox due to these
limitations.

###  F.4 Predicting and Visualizing Unproductive and Underutilized Layers

Based on the graph extracted from the model, unproductive layers can be
predicted using receptive field analysis. Most importantly $I_{min}$ can be
computed for any non-recurrent graph. From the API, it is also possible to
obtain lists of nodes in the graph with $I_{res}<=r_{\ell-1,min}$. Based on
the layer’s names, these nodes can be traced back to the layers, modules and
function-calls in the original model. This allows the user to embed RFA-
Toolbox in automated systems like Neural Architecture Search to filter or
optimize architectures with predicted inefficiencies. Alternatively, RFA-
Toolbox also supports a manual approach by providing functionality to
visualize architectures and highlight unproductive layers (see figure 6).
Additional to unproductive layers, we also provide an additional layer-state
in this work for visualization purposes. Underutilized layers are layers with
$I_{res}\leq r_{\ell,min}$. This means that they expand the receptive field
past the boundaries of the image. Depending on the exact values $r_{\ell-1}$
and $I_{res}$ and $r_{\ell}$ this can mean that this layer is not utilizing
its full potential. For example, when $r_{l-1}$ is very close to the input
resolution $I_{res}$, the layer cannot integrate much novel context by
expanding the receptive field.

Figure 6: Visualization of a simple example architecture implemented in
PyTorch. Unproductive layers for an input resolution of $32\times 32$ pixels
are colored red, underutilized layers are colored orange.