# Search Space Adaptation for Differentiable Neural Architecture Search in
Image Classification

Youngkee Kim  Korea University
Seoul, Korea
<EMAIL_ADDRESS>Soyi Jung  Hallym University
Chuncheon, Korea
<EMAIL_ADDRESS>Minseok Choi  Kyung Hee University
Yong-in, Korea
<EMAIL_ADDRESS>Joongheon Kim  Korea University
Seoul, Korea
<EMAIL_ADDRESS>
###### Abstract

As deep neural networks achieve unprecedented performance in various tasks,
neural architecture search (NAS), a research field for designing neural
network architectures with automated processes, is actively underway. More
recently, differentiable NAS has a great impact by reducing the search cost to
the level of training a single network. Besides, the search space that defines
candidate architectures to be searched directly affects the performance of the
final architecture. In this paper, we propose an adaptation scheme of the
search space by introducing a search scope. The effectiveness of proposed
method is demonstrated with ProxylessNAS for the image classification task.
Furthermore, we visualize the trajectory of architecture parameter updates and
provide insights to improve the architecture search.

##  I Introduction

In recent decades, deep neural networks (DNNs) in machine learning (ML) have
excelled in computer vision tasks with the ability of convolution neural
networks (CNNs) to extract spatial information [1], [2]. Designing an
appropriate architecture of DNN for the target task is essential to obtain
good performances; however, it is quite heuristic and unexplainable[3]. For
this reason, neural architecture search (NAS) has emerged to optimize the
neural architecture design.

One of early NAS studies introduces a reinforcement learning (RL)-based
controller that constructs candidate architectures by iteratively selecting
architectural hyperparameters [4]. A subsequent study demonstrates that RL-
based NAS can discover new architectures outperforming handcrafted
architectures by human experts [5]. In addition, another NAS approach achieves
cutting edge with an evolutionary algorithm that continues until the best
architecture remains [6].

Despite the accomplishment of conventional NAS, the iterative training for
candidate architectures incurs enormous computational costs that hinder
practical applications [7]. To cope with this problem, an one-shot NAS
approach has emerged, which aims to train only one neural network instead of
training each candidate independently [8]. In particular, a study proposes a
differentiable NAS (DNAS) approach that applies a continuous relaxation to
candidate operations so that the search space can be represented as a
differentiable overparameterized neural network [9]. Accordingly, many follow-
up studies have been conducted based on DNAS due to the significant reduction
in the search cost. However, DNAS approaches are GPU memory-intensive because
the entire overparameterized network has to be loaded onto GPUs during search.
For this reason, a technique for binarizing architecture parameters of the
overparameterized network at run-time has been proposed to address the GPU
memory problem [10].

A common fact of the above NAS approaches is that the search space is closely
related to the performance of the final architecture. In other words, the
search space has to be defined carefully, considering the trade-off between
the search cost and the quality of the final architecture.

##  II ProxylessNAS

ProxylessNAS is one of the DNAS-based approaches that aims to directly search
architectures for large-scale target tasks rather than proxy tasks. [10]. We
modify and utilize the strategy of ProxylessNAS to search our proposed search
space. Similar to typical DNAS approaches, ProxylessNAS alternately optimizes
model weights and architecture parameters via gradient-based methods. However,
ProxylessNAS introduces binary gates that binarize architecture parameters of
an overparameterized network to load only one active path at run-time. This
technique alleviates the high GPU memory consumption issue of DNAS caused by
loading the entire overparameterized network to update model weights.
Therefore, ProxylessNAS can search a large search space and dataset by
reducing the memory footprint of the network to load onto GPUs.

Also, ProxylessNAS considers inference latency as the objective of its
architecture search. Most hardware-aware NAS approaches calculate FLOPs of
candidate architectures to reflect the inference latency. On the other hand,
ProxylessNAS computes the expected latency of candidate architectures based on
the pre-calculated latency of each operation. The expected latency is
formulated in a differentiable way and incorporated into the loss function.
Note that we exclude the latency related term from our loss function to focus
on the effect related to the accuracy.

##  III Search Space Adaptation

From the DNAS perspective, the search space is a set of candidate
architectures and can be represented as an overparameterized network. Note
that we refer to the overparameterized network as the search network in this
section. Our proposed adaptive search space is defined by a search unit block
and a search scope. The search unit block is determined by candidate
operations, and the search scope indicates the coverage of the base
architecture to be replaced by the search unit block.

(a) Bottleneck block. (b) Search unit block.

Figure 1: Unit blocks of ResNet-50 and search network.

###  III-A Search unit block

The proposed search space refers to ResNet-50 as the base architecture [11].
ResNet-50 consists of 16 bottleneck blocks, with a total of 50 layers. The
bottleneck block and the search unit block are depicted in the Fig. 1. We
define the search unit block by placing our candidate operations in the
bottleneck block. There are two factors that define our candidate operations.
One is the kernel size of convolution filter (i.e., 3, 5), and the other is
the type of activation function (i.e., ReLU, LeakyReLU, Mish) following
convolution operations. The variation of kernel size offers the possibility to
improve the performance of CNNs by changing the size of receptive fields [12].
We also include a couple of recently used activation functions based on the
previous research that the type of activation affects the accuracy of the same
model [13].

---
Figure 2: Transition to search network by selected search scope.

###  III-B Search scope

The adaptive search space is determined by selecting a search scope level. The
search network is constructed by replacing bottleneck blocks of ResNet-50
within the search scope with predefined search unit blocks. We first group the
architecture of ResNet-50 into four parts according to the number of feature
channels. Then we determine four levels of search scope starting from the
deepest part and gradually expanding. The idea of deep-to-shallow expansion
comes from the previous research that features eventually transit from general
to specific along CNNs [14]. Fig. 2 depicts the transition to the search
network by the selected medium search scope. As a result, a wider search scope
generates a larger search network, which means that more candidate
architectures are considered during the search.

##  IV Experiments and Results

###  IV-A Implementation details

Dataset. We demonstrate the proposed search space with ProxylessNAS on
CIFAR-10 for the image classification task. In order to focus on the
effectiveness of the search space, we apply common data preprocessing. The
input image is cropped into random parts and resized to $224\times 224$. The
resized image is flipped horizontally with a given probability of 0.5 before
normalization. The batch size for both training and validation data is set to
64.

Framework. We leverage the Neural Network Intelligence (NNI) with Pytorch to
implement our experiments. NNI is an open-source AutoML toolkit led by
Microsoft that covers the entire ML pipeline, including feature engineering,
neural architecture search, hyperparameter tuning and model compression [15].
We demonstrate the proposed search space with customized ProxylessNAS
implemented on NNI.

Training details. Our DNAS with ProxylessNAS trains the search network via
gradient-based methods. Therefore, there are two kinds of training, one is
training of the search network and the other is retraining of the discovered
architecture. In the search network training, model weights of the search
network are updated on the training set using the SGD optimizer with Nesterov
momentum. We set the initial learning rate to 0.05, the momentum value to 0.9
and the weight decay to 4e-5. Conversely, the Adam optimizer with an initial
learning rate of 0.001 and no weight decay is used to update architecture
parameters on the validation set. The path with the highest architecture
parameters constitutes the final architecture after the search training.
Finally, we retrain the determined architecture from scratch on the training
set, under the same settings for updating model weights in the search
training.

###  IV-B Results

As we explain in Sec. III, each level of search scope is applied to obtain the
search space. We independently perform architecture searches for each search
space and evaluate discovered architectures under the identical setting.
Furthermore, we visualize the trajectory of architecture parameter updates and
identify insights from trends.

Model | Params | Top-1 Accuracy (%)
---|---|---
ResNet-50 (baseline) | 23.53M | 91.10
PResNet-50-s (ours) | 23.53M | 92.80
PResNet-50-m (ours) | 24.58M | 92.35
PResNet-50-l (ours) | 23.79M | 92.07
PResNet-50-f (ours) | 25.23M | 93.22
Table I: Performance comparison on CIFAR-10.
---
Figure 3: Architecture parameter update trajectories with the large search
scope.

Performance comparison of searched architectures. We explore each search space
based on aforementioned training details and evaluate the performance of
searched architectures by retraining from scratch. Top-1 accuracy results of
searched architectures on CIFAR-10 are summarized in Table I. We name the
family of searched architectures as PResNet-50 with a letter indicating the
level of search scope. Note that searched architectures outperform the
baseline by up to 2.12%. Some models have more parameters than the baseline
due to the fact that we consider larger kernel size of convolution filter,
while the type of activation function does not affect the size of parameters.

Although PResNet-50-f is superior to other searched models, it is the most
expensive model in terms of search time due to the huge search space. The
search training for PResNet-50-f consumes about $\times 2.07$ longer than
PResNet-50-s on a GeForce RTX 2080 Ti. Therefore, a smaller search scope can
be a good choice depending on a given budget.

Visualization of architecture parameter updates. We track changes in
architecture parameters to capture properties of the proposed search space
during the search. Fig. 3 shows trajectories of architecture parameter updates
of the deepest and third deepest search blocks with the large search scope.
Block numbers are assigned sequentially from the deepest search block. Note
that represented values on the y-axis are calculated by applying a softmax to
real-valued architecture parameters for better understanding. Firstly,
architecture parameters with a smaller search scope converge relatively
quickly compared to a larger scope. In Fig. 3, the kernel size of both search
blocks is determined to be 3 at approximately 300 search epochs, whereas it
takes less than 100 epochs for the small search scope. The type of activation
function seems meaningless on these blocks, but has a different effect on
other blocks. Secondly, we find that architecture parameters of some blocks
tend to become unstable again when the search lasts excessively. In Fig. 3,
architecture parameters are updated in a certain direction until 400 epochs,
while they show erratic movements after that. Thus, a carefully designed
criterion for determining when to stop the search is important to avoid
performance degradation.

##  V Conclusion

We introduce an adaptation of search space with a search unit block and
selectable search scopes. The effectiveness of proposed method is verified
with customized ProxylessNAS implemented on NNI framework. We also provide
several insights to improve DNAS by tracking changes in architecture
parameters during the search. Future work could be a study of a tailored
search scope for a given circumstance and appropriate criteria for the search
exit point.

## Acknowledgment

This work was supported by Institute of Information & Communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No. 2021-0-00766, Development of Integrated Development
Framework that supports Automatic Neural Network Generation and Deployment
optimized for Runtime Environment). Joongheon Kim is a corresponding author of
this paper.

## References

  * [1] H. Baek, W. J. Yun, Y. Kwak, S. Jung, M. Ji, M. Bennis, J. Park, and J. Kim, “Joint superposition coding and training for federated learning over multi-width neural networks,” _CoRR_ , vol. abs/2112.02543, December 2021\.
  * [2] J. Park, S. Samarakoon, A. Elgabli, J. Kim, M. Bennis, S.-L. Kim, and M. Debbah, “Communication-efficient and distributed learning over wireless networks: Principles and applications,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 796–819, May 2021.
  * [3] M. Shin, J. Kim, and M. Levorato, “Auction-based charging scheduling with deep learning framework for multi-drone networks,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 5, pp. 4235–4248, May 2019.
  * [4] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in _proc. of the International Conference on Learning Representations (ICLR)_ , Toulon, France, April 2017.
  * [5] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in _Proc. of the CVPR_ , UT, USA, June 2018, pp. 8697–8710.
  * [6] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in _Proc. of the AAAI_ , Hawaii, USA, January 2019, pp. 4780–4789.
  * [7] Y. Kim, W. J. Yun, Y. K. Lee, and J. Kim, “Two-stage architectural fine-tuning with neural architecture search using early-stopping in image classification,” _arXiv preprint arXiv:2202.08604_ , 2022.
  * [8] Y. Kim, W. J. Yun, Y. K. Lee, S. Jung, and J. Kim, “Trends in neural architecture search: Towards the acceleration of search,” in _Proc. of the International Conference on Information and Communication Technology Convergence (ICTC)_ , Jeju Island, Korea, Republic of, October 2021, pp. 421–424.
  * [9] H. Liu, K. Simonyan, and Y. Yang, “DARTS: differentiable architecture search,” in _Proc. of the International Conference on Learning Representations (ICLR)_ , LA, USA, May 2019.
  * [10] H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” in _Proc. of the International Conference on Learning Representations (ICLR)_ , LA, USA, May 2019.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. of the CVPR_ , NV, USA, June 2016, pp. 770–778.
  * [12] W. Luo, Y. Li, R. Urtasun, and R. S. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” in _Proc. of the NIPS_ , Barcelona, Spain, December 2016, pp. 4898–4906.
  * [13] D. Misra, “Mish: A self regularized non-monotonic activation function,” in _Proc. of the British Machine Vision Conference (BMVC)_ , Virtual, UK, September 2020.
  * [14] M. Long, Y. Cao, J. Wang, and M. I. Jordan, “Learning transferable features with deep adaptation networks,” in _Proc. of the ICML_ , Lille, France, July 2015, pp. 97–105.
  * [15] Microsoft. (2022, January) Neural Network Intelligence. [Online]. Available: https://github.com/microsoft/nni