11institutetext: Yokohama National University, Kanagawa, Japan
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>shirakawa-shinichi-
<EMAIL_ADDRESS>22institutetext: SkillUp AI Co., Ltd., Tokyo, Japan
22email<EMAIL_ADDRESS>
# Efficient Search of Multiple Neural Architectures with Different
Complexities via Importance Sampling

Yuhei Noda 11 0000-0001-9646-2506 Shota Saito 1122 0000-0002-9863-6765
Shinichi Shirakawa 11 0000-0002-4659-6108

###### Abstract

Neural architecture search (NAS) aims to automate architecture design
processes and improve the performance of deep neural networks. Platform-aware
NAS methods consider both performance and complexity and can find well-
performing architectures with low computational resources. Although ordinary
NAS methods result in tremendous computational costs owing to the repetition
of model training, one-shot NAS, which trains the weights of a supernetwork
containing all candidate architectures only once during the search process,
has been reported to result in a lower search cost. This study focuses on the
architecture complexity-aware one-shot NAS that optimizes the objective
function composed of the weighted sum of two metrics, such as the predictive
performance and number of parameters. In existing methods, the architecture
search process must be run multiple times with different coefficients of the
weighted sum to obtain multiple architectures with different complexities.
This study aims at reducing the search cost associated with finding multiple
architectures. The proposed method uses multiple distributions to generate
architectures with different complexities and updates each distribution using
the samples obtained from multiple distributions based on importance sampling.
The proposed method allows us to obtain multiple architectures with different
complexities in a single architecture search, resulting in reducing the search
cost. The proposed method is applied to the architecture search of
convolutional neural networks on the CIAFR-10 and ImageNet datasets.
Consequently, compared with baseline methods, the proposed method finds
multiple architectures with varying complexities while requiring less
computational effort.

###### Keywords:

Neural Architecture Search Convolutional Neural Network Importance Sampling
Natural Gradient

##  1 Introduction

Architecture design is a key factor in accelerating the performance of deep
neural networks (DNNs); however, the associated process is arduous for
practitioners. Neural architecture search (NAS), aimed at automating the
design of DNN architectures, has been actively studied in recent years [7].
Popular methods often optimize architectures using evolutionary algorithms
[16, 21] or reinforcement learning [27]. These early NAS methods optimize the
architecture in hyperparameter optimization frameworks, which requires a
significant amount of time for architecture search due to the repetition of
model training. One-shot NAS, e.g., [12, 15, 20], is a promising approach for
reducing the computational cost of NAS. One-Shot NAS simultaneously optimizes
the weight and architecture parameters considering an extensive network
(supernetwork) that includes many candidate architectures as its subnetworks.
Because the weight parameters are shared between subnetworks, one-shot NAS
optimizes the weights in the supernetwork only once during the search process,
thus significantly reducing the search cost.

DNNs are often implemented in devices with limited computational resources,
such as embedded and mobile devices. In such cases, NAS methods are required
to find an architecture with good prediction performance and low computation
and memory usage. Therefore, NAS methods have been developed for optimizing
the prediction performance and architecture complexity, such as FLOPs,
latency, and the number of weight parameters. The method proposed in [24]
includes a term related to the latency in the objective function and
successfully finds a highly accurate architecture while suppressing latency.
This study focuses on the one-shot NAS method proposed in [17], which
introduces a regularization of the architecture complexity. This method uses
binary variables to represent the architecture and a multivariate Bernoulli
distribution as the law of binary variables. The architecture search is
performed by updating the parameters of the Bernoulli distribution to minimize
the weighted sum of the predictive loss and the regularization term related to
the complexity of the architecture. Although this approach reduces the number
of connections in a densely connected convolutional neural network (CNN), the
architecture search space is limited because it must represent the
architecture using binary variables. This implies that directly employing the
state-of-the-art architecture search space represented by categorical
variables [15, 28] is not straightforward. Additionally, obtaining multiple
architectures with different complexities requires repeated architecture
searches with different regularization coefficients in the objective function,
which, in turn, increases computational cost.

This study extends the method proposed in [17] to overcome the limitation
described above. We adopt categorical distributions as the architecture
distribution and propose a regularization term for the architecture
complexity. We also derive the analytical natural gradient of the proposed
regularization term. Thereafter, we propose an efficient search method to
simultaneously obtain multiple architectures with different complexities in a
single architecture search using importance sampling. The proposed method is
then evaluated by applying it to the architecture search of CNNs on the
CIFAR-10 [11] and ImageNet [6] datasets. The experimental results indicate
that the proposed method can obtain multiple architectures with different
complexities in a single search, and its prediction performance is comparable
with that demonstrated by baseline methods.

##  2 Probabilistic Model-Based One-Shot NAS with Complexity Regularization

This section details the one-shot NAS framework using the architecture
complexity regularization proposed in [17]. We denote the DNN parameterized by
architecture parameters $M$ and weights $W$ as $\phi(M,W)$ and assume that
$\phi$ is differentiable with respect to (w.r.t.) $W$ but non-differentiable
w.r.t. $M$. The architecture parameters $M$ determine the types of connections
and operations in DNN. The architecture defined by $M$ corresponds to a
subnetwork in the supernetwork and shares weights in the operations between
different architectures.

Let us consider the optimization of $W$ and $M$ to minimize both the loss, for
instance, the cross-entropy loss, and the regularization metric w.r.t. the
complexity of the architecture. We denote the loss for the dataset
$\mathcal{D}$ and the regularization term as $\mathcal{L}(M,W,\mathcal{D})$
and $\mathcal{R}(M)$, respectively. In [17], the weighted sum of the two
terms, $\mathcal{F}(M,W)=\mathcal{L}(M,W,\mathcal{D})+\epsilon\mathcal{R}(M)$,
has been adopted as the objective function, where $\epsilon$ represents the
regularization coefficient that balances the two terms. However, because the
architecture parameters $M$ are non-differentiable and often discrete, we
cannot optimize $M$ by a gradient method. To relax the problem, we introduce
the parametric probability distribution of $M$ and denote it as
$P_{\theta}(M)$, where $\theta$ denotes the distribution parameters. Instead
of directly optimizing $\mathcal{F}(M,W)$, we optimize $\theta$ by minimizing
the expected loss of $\mathcal{F}(M,W)$ under $P_{\theta}(M)$ as
$\mathcal{G}(\theta,W)=\mathbb{E}_{{P}_{\theta}}\left[\mathcal{L}(M,W,\mathcal{D})\right]+\epsilon\mathbb{E}_{{P}_{\theta}}\left[\mathcal{R}(M)\right]$.

As the objective function $\mathcal{G}(\theta,W)$ is differentiable w.r.t.
both $W$ and $\theta$, we can optimize it by a gradient method. We update the
distribution parameters $\theta$ to the natural gradient direction [2], which
is the steepest direction when considering the Kullback–Leibler divergence as
the pseudo-distance in the distribution parameter space, and it is given by
the product of the inverse of the Fisher information matrix (FIM) and
Euclidean gradient. We use the vanilla gradient to optimize $W$ as usual DNN
training. The gradients w.r.t. $W$ and $\theta$ are given by

$\displaystyle\nabla_{W}\mathcal{G}(\theta,W)=\mathbb{E}_{{P}_{\theta}}\left[\nabla_{W}\mathcal{L}(M,W,\mathcal{D})\right]$
(1)
$\displaystyle{\tilde{\nabla}}_{\theta}\mathcal{G}(\theta,W)=\mathbb{E}_{{P}_{\theta}}\left[\mathcal{L}(M,W,\mathcal{D}){\tilde{\nabla}}_{\theta}\ln
P_{\theta}(M)\right]+\epsilon{\tilde{\nabla}}_{\theta}\mathbb{E}_{{P}_{\theta}}\left[\mathcal{R}(M)\right]\enspace,$
(2)

where ${\tilde{\nabla}}_{\theta}=F(\theta)^{-1}\nabla_{\theta}$ represents the
natural gradient operator. Here, $F(\theta)$ indicates the FIM of
$P_{\theta}(M)$. Optimizing $\theta$ using (2) with $\epsilon=0$ operates in a
manner similar to information geometric optimization [14], which is a unified
framework for probabilistic model-based evolutionary algorithms. In most
cases, it is difficult to compute the gradients (1) and (2). Therefore, the
gradients (1) and (2) are approximated using Monte Carlo methods with
$\lambda$ architecture parameters $M_{1},M_{2},\dots,M_{\lambda}$ sampled from
$P_{\theta}(M)$ as follows:

$\displaystyle\nabla_{W}\mathcal{G}(\theta,W)\approx$
$\displaystyle\frac{1}{\lambda}\sum_{i=1}^{\lambda}\nabla_{W}\mathcal{L}(M_{i},W,\mathcal{D})$
(3) $\displaystyle{\tilde{\nabla}}_{\theta}\mathcal{G}(\theta,W)\approx$
$\displaystyle\frac{1}{\lambda}\sum_{i=1}^{\lambda}\mathcal{L}(M_{i},W,\mathcal{D}){\tilde{\nabla}}_{\theta}\ln
p_{\theta}(M_{i})+\epsilon{\tilde{\nabla}}_{\theta}\mathbb{E}_{{P}_{\theta}}\left[\mathcal{R}(M)\right]\enspace.$
(4)

Because the scale of the loss affects the magnitude of the natural gradient,
we transform $\mathcal{L}(M_{i},W,\mathcal{D})$ into the quantile-based
utility value under $P_{\theta}(M)$, as done in [14]. The probability of
sampling a solution with a loss value less than or equal to
$\mathcal{L}(M_{i},W,\mathcal{D})$ is estimated as
$\bar{q}_{\theta}^{\leqslant}(M_{i})=\lambda^{-1}\sum_{k=1}^{\lambda}\mathbb{I}\\{\mathcal{L}(M_{k},W,\mathcal{D})\leqslant\mathcal{L}(M_{i},W,\mathcal{D})\\}$,
where $\mathbb{I}\\{\cdot\\}$ denotes the indicator function. We use the
utility function of $\hat{s}_{i}=w(\bar{q}_{\theta}^{\leqslant}(M_{i}))$,
instead of $\mathcal{L}(M_{i},W,\mathcal{D})$, to update the distribution
parameters $\theta$.111This utility definition does not assume the possibility
of sampling architectures with the same loss value. Although it could happen
in our case, we use this utility definition for simplicity. A rigorous
definition can be found in [14, 19]. Specifically, we use the following
function for $w$.

$\displaystyle w(x)=\begin{cases}-2&(x\leqslant 0.25)\\\ 0&(0.25<x\leqslant
0.75)\\\ 2&(0.75<x)\end{cases}$

Consequently, the update rule for $\theta$ at the $t$-th iteration is given by

$\theta^{(t+1)}=\theta^{(t)}-\eta\left(\frac{1}{\lambda}\sum_{i=1}^{\lambda}\hat{s}_{i}{\tilde{\nabla}}_{\theta}\ln
p_{\theta}(M_{i})+\epsilon{\tilde{\nabla}}_{\theta}\mathbb{E}_{{P}_{\theta}}\left[\mathcal{R}(M)\right]\right)\enspace,$
(5)

where $\eta$ represents the learning rate for $\theta$. We note that the
weights $W$ can be updated using any stochastic gradient descent (SGD) method
with (3).

##  3 Proposed Method

In [17], the binary vector has been adopted as the architecture parameter.
However, state-of-the-art architecture search spaces, such as [12, 15], are
defined using categorical variables. In addition, repeating the architecture
search is required to obtain multiple architectures with different
complexities. We first introduce the categorical distribution as
$P_{\theta}(M)$ in the framework considered in [17]. Subsequently, we propose
simultaneously optimizing multiple categorical distributions, each
corresponding to a different regularization coefficient, to obtain multiple
architectures with varying complexities in a single search. Each categorical
distribution is updated by exploiting samples from other distributions to
realize an efficient search process.

###  3.1 Introducing Categorical Distributions

The DNN architecture is represented by the following $D$ dimensional
categorical variables: $h=(h_{1},\dots,h_{D})$. The $d$-th categorical
variable $h_{d}$ possesses $K_{d}$ candidate categories and determines
operations or connections in the DNN. For instance, one can determine the
kernel size of a convolution layer. We denote categorical variables by one-hot
vectors as $M=(m_{1},\dots,m_{D})$, where
$m_{d}=(m_{d,1},\dots,m_{d,K_{d}})^{\mathrm{T}}\in\\{0,1\\}^{K_{d}}$. When
$h_{d}$ is the $k$-th category, $m_{d,k}=1$, and other elements of $m_{d}$ are
zero. We consider the categorical distribution as the distribution of the
architecture parameters, which is described as
$P_{\theta}(M)=\prod_{d=1}^{D}\prod_{k=1}^{K_{d}}\left(\theta_{d,k}\right)^{m_{d,k}}$,
where $\theta_{d,k}\in[0,1]$ is the probability of being $m_{d,k}=1$.

We choose the number of weight parameters as the complexity metric for the
regularization term $\mathcal{R}(M)$ to penalize the complicated architecture.
Let us denote the number of weight parameters in the operation corresponding
to $m_{d,k}$ as $c_{d,k}$; then, we define the regularization term as
$\mathcal{R}(M)=\sum_{d=1}^{D}\sum_{k=1}^{K_{d}}c_{d,k}m_{d,k}$. The expected
value of $\mathcal{R}(M)$ under $P_{\theta}(M)$ is described as

$\mathbb{E}_{P_{\theta}}\left[\mathcal{R}(M)\right]=\sum_{d=1}^{D}\sum_{k=1}^{K_{d}}c_{d,k}\theta_{d,k}\enspace.$
(6)

It should be noted that the distribution parameter of the last category can be
given by $\theta_{d,K_{d}}=1-\sum_{k=1}^{K_{d}-1}\theta_{d,k}$ owing to
$\sum_{k=1}^{K_{d}}\theta_{d,k}=1$; consequently, we can introduce the
notation of the distribution parameter vector without the last category’s
parameter as
$\bar{\theta}_{d}=(\theta_{d,1},\theta_{d,2},\dots,\theta_{d,K_{d}-1})^{\mathrm{T}}$.

Next, we derive the natural gradient of
$\mathbb{E}_{P_{\theta}}\left[\mathcal{R}(M)\right]$. The vanilla gradient of
$\mathbb{E}_{P_{\theta}}\left[\mathcal{R}(M)\right]$ w.r.t the $d$-th
distribution parameters $\bar{\theta}_{d}$ is given by
$\nabla_{\bar{\theta}_{d}}\mathbb{E}_{P_{\theta}}\left[\mathcal{R}(M)\right]=\bar{c}_{d}-c_{d,K_{d}}\mathbf{1}$,
where $\bar{c}_{d}=(c_{d,1},c_{d,2},\dots,c_{d,K_{d}-1})^{\mathrm{T}}$, and
$\mathbf{1}$ represents the all-ones vector. The FIM is a block diagonal
matrix because our categorical variables are independent. The inverse of the
$d$-th block in the FIM is given by
$F(\bar{\theta}_{d})^{-1}=\operatornamewithlimits{diag}(\bar{\theta}_{d})-\bar{\theta}_{d}\bar{\theta}_{d}^{\mathrm{T}}$.
Then, we can obtain the natural gradient of (6) as
${\tilde{\nabla}}_{\bar{\theta}_{d}}\mathbb{E}_{P_{\theta}}\left[\mathcal{R}(M)\right]=\bar{c}_{d}\odot\bar{\theta}_{d}-\left(\bar{c}_{d}^{\mathrm{T}}\bar{\theta}_{d}+c_{d,K_{d}}(1-\bar{\theta}_{d}^{\mathrm{T}}{\bf
1})\right)\bar{\theta}_{d}=\left(\bar{c}_{d}-\mathcal{Q}_{d}{\bf
1}\right)\odot\bar{\theta}_{d}$, where $\odot$ indicates the element-wise
product and $\mathcal{Q}_{d}=\sum_{k=1}^{K_{d}}c_{d,k}\theta_{d,k}$. According
to [1], the natural gradient of the log-likelihood is given by
${\tilde{\nabla}}_{\bar{\theta}_{d}}\ln
P_{\theta}(M)=\bar{m}_{d}-\bar{\theta}_{d}$, where
$\bar{m}_{d}=(m_{d,1},\dots,m_{d,K_{d}-1})^{\mathrm{T}}$. We then obtain the
update rule of $\bar{\theta}_{d}$ as

$\bar{\theta}_{d}^{(t+1)}=\bar{\theta}_{d}^{(t)}-\eta\left(\frac{1}{\lambda}\sum_{i=1}^{\lambda}\hat{s}_{i}(\bar{m}_{d}^{(i)}-\bar{\theta}_{d}^{(t)})+\epsilon\left(\bar{c}_{d}-\mathcal{Q}_{d}{\bf
1}\right)\odot\bar{\theta}_{d}^{(t)}\right)\enspace,$ (7)

where $\bar{m}_{d}^{(i)}$ indicates the $d$-th one-hot vector without the last
element of the $i$-th sample. Additionally,
$\theta_{d,K_{d}}^{(t+1)}=1-\sum_{k=1}^{K_{d}-1}\theta_{d,k}^{(t+1)}$ is given
by

$\displaystyle\theta_{d,K_{d}}^{(t+1)}=\theta_{d,K_{d}}^{(t)}-\eta\left(\frac{1}{\lambda}\sum_{i=1}^{\lambda}\hat{s}_{i}\left(m_{d,K_{d}}^{(i)}-\theta_{d,K_{d}}^{(t)}\right)+\epsilon\left(c_{d,K_{d}}-\mathcal{Q}_{d}\right)\theta_{d,K_{d}}^{(t)}\right)\enspace.$
(8)

According to (7) and (8), we can replace $\bar{\theta}_{d}$ in (7) with
$\theta_{d}=(\theta_{d,1},\theta_{d,2},\dots,\theta_{d,K_{d}})^{\mathrm{T}}$
and update the distribution parameter $\theta_{d}$ using the replaced update
rule.

###  3.2 Searching Multiple Architectures via Importance Sampling

In existing methods [3, 17], a search for the architecture must be performed
multiple times by altering the regularization coefficient to obtain multiple
architectures with different complexities. Herein, we propose a method for
finding multiple architectures within a single search, thereby reducing the
search cost. The idea is to jointly update the multiple distributions
corresponding to different complexities by exploiting the samples drawn from
other distributions via importance sampling. Let us consider $N$ distribution
parameters, $\boldsymbol{\theta}=(\theta^{(1)},\dots,\theta^{(N)})$,
corresponding to different regularization coefficients
$\epsilon_{1},\dots,\epsilon_{N}$. The objective function of each distribution
is defined by
$\mathcal{G}(\theta^{(n)},W)=\mathbb{E}_{{P}_{\theta^{(n)}}}\left[\mathcal{L}(M,W,\mathcal{D})\right]+\epsilon_{n}\mathbb{E}_{{P}_{\theta^{(n)}}}\left[\mathcal{R}(M)\right]$.
We sample $\lambda$ architecture parameters from the mixture distribution
$P_{\boldsymbol{\theta}}(M)=N^{-1}\sum_{n=1}^{N}P_{\theta^{(n)}}(M)$ at each
iteration and update each distribution using the samples obtained from the
mixture. Based on the importance sampling technique used in [18, 19], the
probability $\bar{q}_{\theta^{(n)}}^{\leqslant}(M_{i})$ is estimated by

$\displaystyle\bar{q}_{\theta^{(n)}}^{\leqslant}(M_{i})=\frac{1}{\lambda}\sum_{k=1}^{\lambda}r_{k}^{(n)}\mathbb{I}\left\\{\mathcal{L}\left(M_{k},W,\mathcal{D}\right)\leqslant\mathcal{L}\left(M_{i},W,\mathcal{D}\right)\right\\}\enspace,$
(9)

where
$r_{k}^{(n)}=\frac{P_{\theta^{(n)}}(M_{k})}{P_{\boldsymbol{\theta}}(M_{k})}$
indicates the likelihood ratio. Then, the utility of
$\hat{s}_{i}^{(n)}=w(\bar{q}_{\theta^{(n)}}^{\leqslant}(M_{i}))$ is used to
update $\theta^{(n)}$. Similarly, the natural gradient can be approximated via
importance sampling, and we obtain the update rule of $\theta_{d}^{(n)}$ as

$\displaystyle\theta_{d}^{(n)}\leftarrow\theta_{d}^{(n)}-\eta\left(\frac{1}{\lambda}\sum_{i=1}^{\lambda}\hat{s}_{i}^{(n)}r_{i}^{(n)}\left(m_{d}^{(i)}-\theta_{d}^{(n)}\right)+\epsilon_{n}\left(c_{d}-\mathcal{Q}_{d}{\bf
1}\right)\odot\theta_{d}^{(n)}\right)\enspace.$ (10)

Here, we ignore the notation of the time step $t$ for simplicity.

Algorithm 1 Architecture Search Procedure of the Proposed Method

0: Dataset $\mathcal{D}=\\{\mathcal{D}_{W},\mathcal{D}_{\theta}\\}$

1: Initialize $W$ and $\theta^{(1)}$, $\theta^{(2)},\dots,\theta^{(N)}$

2: for $t=1,\dots,T_{W}$ do

3: Sample mini-batch $\tilde{\mathcal{D}}_{W}$ from $\mathcal{D}_{W}$

4: Sample $\lambda$ architectures from uniform distribution and update $W$
using (3)

5: end for

6: for $t=1,\dots,T_{\theta}$ do

7: Sample mini-batch $\tilde{\mathcal{D}}_{\theta}$ from
$\mathcal{D}_{\theta}$

8: Sample $\lambda$ architectures from $P_{\boldsymbol{\theta}}(M)$ and update
$\theta^{(n)}$ for $n=1,\dots,N$ by (10)

9: end for

###  3.3 Overall Algorithm

The architecture search procedure followed by the proposed method is presented
in Algorithm 1. The dataset $\mathcal{D}$ is divided into $\mathcal{D}_{W}$
and $\mathcal{D}_{\theta}$, and the resulting datasets are used to update the
weights and distribution parameters, respectively. Although the method in [17]
jointly optimizes the weights $W$ and distribution parameters $\theta$, the
proposed method separates the optimization of $W$ and $\theta$. That is, we
first optimize $W$ under a uniform distribution and then optimize $\theta$
using the trained weights $W$. A separate (two-stage) optimization of the
weight and architecture parameters has been conducted in recent NAS-related
studies [4, 8, 10], and the approach has demonstrated promising performance.

In the optimization phase of $W$, $\lambda$ architecture parameters
$M_{1},\dots,M_{\lambda}$ are sampled from a discrete uniform distribution,
and the weight parameters of $W$ are updated using (3). This update of $W$ is
repeated $T_{W}$ times. Then, in the optimization phase of $\theta$, $\lambda$
architecture parameters $M_{1},\dots,M_{\lambda}$ are sampled from the mixture
distribution $P_{\boldsymbol{\theta}}(M)$, and the distribution parameters
$\boldsymbol{\theta}=(\theta^{(1)},\dots,\theta^{(N)})$ for different
regularization coefficients $\epsilon_{1},\dots,\epsilon_{N}$ are updated
using (10). Following the architecture search, we determine the final
architectures by
$M^{*}_{(n)}=\operatornamewithlimits{argmax}_{M}P_{\theta^{(n)}}(M)$ for
$n=1,\dots,N$ and obtain multiple architectures with different complexities.
Then, we retrain the weights of the final architecture $M^{*}_{(n)}$ from
scratch using the dataset $\mathcal{D}$.

##  4 Experiment and Results

This section evaluates the proposed method on image classification tasks. Our
algorithms were run using NVIDIA Tesla V100 GPUs (32 GB memory).

###  4.1 CIFAR-10

####  4.1.1 Experimental Settings

The CIFAR-10 [11] dataset contains 50,000 training and 10,000 test images, and
each image is labeled using one class out of 10. We adopt the cell-based CNN
architecture search space used in [1, 15] and follow the experimental setting
in [1]. In the architecture search phase, we stack six normal and two
reduction cells and set the number of channels in the first cell to 16. The
architectures of the normal and reduction cells are searched by NAS
algorithms. The training data $\mathcal{D}$ are divided into $\mathcal{D}_{W}$
and $\mathcal{D}_{\theta}$, which are then used to update the weights and
distribution parameters, respectively. Both mini-batch sizes
$|\tilde{\mathcal{D}}_{W}|$ and $|\tilde{\mathcal{D}}_{\theta}|$ are set to
64. We set the sample size of the architecture $\lambda$ to 2. The weights and
distribution parameters are both updated for 200 epochs, respectively, i.e.,
$T_{W}=T_{\theta}=200$. For updating the weights $W$, we use SGD with a
momentum of 0.9 and set the weight decay to $3\times 10^{-4}$. According to
the cosine schedule [13], the learning rate gradually decreases from 0.025 to
0. For updating the distribution parameters $\theta$, we set the learning rate
to $\eta_{\theta}=(\sum_{d=1}^{D}K_{d})^{-1}=1/180$ and the regularization
coefficient $\epsilon_{n}$ to $\\{0.0,0.1,0.3,0.5\\}$. In the retraining
phase, we set the number of normal cells to 10 and the number of channels in
the first cell to 50. The other retraining settings are the same as [1].

We compare the proposed method with two baseline one-shot NAS methods. The
first method, presented in Algorithm 2 and termed Method 1 (Simultaneous), is
a straightforward extension of the method considered in [17]. This algorithm
simultaneously updates the weights and distribution parameters and performs
the architecture search several times with different regularization
coefficients $\epsilon$ to obtain multiple architectures. The second method,
presented in Algorithm 3 and termed Method 2 (Separate), separates the weight
optimization and architecture search, similar to the proposed method, but
performs the architecture search several times with different $\epsilon$. The
second method is advantageous compared to Method 1 (Simultaneous) because it
performs the weight optimization only once; however, it is still inefficient
compared to the proposed method because it requires multiple runs during the
architecture search phase. The experiment uses the same number of epochs to
optimize the weights and distribution parameters in a single search as in the
proposed method, i.e., $T=T_{W}=T_{\theta}=200$ in Algorithms 2 and 3.
Moreover, we perform the random search as the simplest baseline, which
randomly samples architectures from the search space and retrains them. We
sample architectures with weight parameters of 2.5M (million), 3.0M, 4.0M, and
5.0M. We reported the median values among three independent trials for all
algorithms.

Algorithm 2 Method 1 (Simultaneous)

0: Dataset $\mathcal{D}=\\{\mathcal{D}_{W},\mathcal{D}_{\theta}\\}$

1: Initialize $\theta^{(1)}$, $\theta^{(2)},\dots,\theta^{(N)}$

2: for $n=1,\dots,N$ do

3: Initialize $W$

4: for $t=1,\dots,T$ do

5: Sample mini-batch $\tilde{\mathcal{D}}_{W}$ from $\mathcal{D}_{W}$

6: Sample $\lambda$ architectures from $P_{\theta^{(n)}}(M)$ and update $W$
using (3)

7: Sample mini-batch $\tilde{\mathcal{D}}_{\theta}$ from
$\mathcal{D}_{\theta}$

8: Sample $\lambda$ architectures from $P_{\theta^{(n)}}(M)$ and update
$\theta^{(n)}$ by (7)

9: end for

10: end for

Algorithm 3 Method 2 (Separate)

0: Dataset $\mathcal{D}=\\{\mathcal{D}_{W},\mathcal{D}_{\theta}\\}$

1: Initialize $W$ and $\theta^{(1)}$, $\theta^{(2)},\dots,\theta^{(N)}$

2: for $t=1,\dots,T_{W}$ do

3: Sample mini-batch $\tilde{\mathcal{D}}_{W}$ from $\mathcal{D}_{W}$

4: Sample $\lambda$ architectures from uniform distribution and update $W$
using (3)

5: end for

6: for $n=1,\dots,N$ do

7: for $t=1,\dots,T_{\theta}$ do

8: Sample mini-batch $\tilde{\mathcal{D}}_{\theta}$ from
$\mathcal{D}_{\theta}$

9: Sample $\lambda$ architectures from $P_{\theta^{(n)}}(M)$ and update
$\theta^{(n)}$ by (7)

10: end for

11: end for

####  4.1.2 Results and Discussions

Figure 1: Relationship between the number of parameters and test error in
CIFAR-10.

Table 1: Search cost to obtain 4 architectures in CIFAR-10. The search cost
excludes the architecture retraining cost.
Method | Search Cost
---|---
$[\mathrm{GPU\>hours}]$
Method 1 | 16.4
(Simultaneous)
Method 2 | 5.2
(Separate)
Proposed | 3.4
Method

Figure 1 and Table 1 show the test error of the obtained architectures and
search cost, respectively. The proposed method achieves better accuracies than
those obtained by Method 1 (Simultaneous). In Method 1, the weight and
distribution parameters are updated simultaneously. As the convergence speed
of the operations in the cells differs, the distribution parameters converge
to select the architecture that minimizes the loss early, resulting in a
search failure. This difficulty associated with one-shot NAS during
simultaneous optimization of weights and architectures has been pointed out in
[26, 5]. In comparison with Method 1, the proposed method selects well-
performed architectures because all operations are equally selected and
trained during the weight training stage. Table 1 shows that the proposed
method obtains four architectures in approximately one-fifth of the search
time required by Method 1. While Method 1 requires as many searches as the
number of architectures to be obtained, the proposed method obtains multiple
architectures in a single search.

Comparing the proposed method with Method 2 (Separate), the proposed method
obtains multiple architectures with different parameters without causing a
degradation of the prediction accuracy. Both methods require a single
optimization of the weights, but the proposed method updates multiple
distribution parameters using the architecture samples from the mixture
distribution. Therefore, the proposed method does not need to repeat the
architecture search. Consequently, the proposed method reduces the search cost
compared to Method 2. Finally, the architectures obtained by the proposed
method exhibit better prediction accuracies than those obtained via a random
search, suggesting that the architecture search is effective.

###  4.2 ImageNet

####  4.2.1 Experimental Settings

ImageNet [6] is a large-scale image classification dataset consisting of 1,000
classes containing approximately 1.28 million training images and 50,000
validation images. We use the CNN architecture search space proposed in
ProxylessNAS [3] and evaluate the performance of the obtained architectures
using the validation data. For the training data, we follow the pre-processing
and data augmentation methods in [10].

During the search phase, we update the distribution parameters $\theta$ with
50,000 randomly selected images from the training data $\mathcal{D}$ and
update the weights $W$ with the remaining training data. We update the weights
and distribution parameters for 60 epochs. We set the mini-batch sizes
$|\tilde{\mathcal{D}}_{W}|$ and $|\tilde{\mathcal{D}}_{\theta}|$ to 350 and
the number of samples $\lambda$ to 8. For updating the weights $W$, we use SGD
with a momentum of 0.9 and set the weight decay to $5\times 10^{-5}$.
According to the cosine schedule [13], the initial learning rate decreases
from 0.068 to 0. For updating the distribution parameters, we set the learning
rate and regularization coefficient to
$\eta_{\theta}=(\sum_{d}^{D}K_{d})^{-1}=1/141$ and
$\epsilon_{n}\in\\{0.0,0.5,1.0\\}$, respectively.

In the retraining phase, we update the weights for 350 epochs with a mini-
batch size of 768. We use RMSProp and set the weight decay to $1\times
10^{-5}$. In the first five epochs, the learning rate increases linearly from
0 to 0.192. Thereafter, the learning rate gradually decreases by multiplying
0.963 every three epochs. We use the label smoothing technique [22] and
introduce the squeeze and excitation module [9] into the MBConv operations.
During inference, the model exponential moving average (EMA) is applied to
calculate the prediction accuracy of the test data. These retraining settings
are based on [10].

Table 2: Result of ImageNet. $N$ represents the number of architectures to be searched. The values of the existing methods are referred from the literature. Method | Params $[\mathrm{M}]$ | Top-1 Accuracy $[\%]$ | Search Cost $[\mathrm{GPU\>hours}]$
---|---|---|---
MnasNet-A2 [23] | 4.8 | 75.6 | 40,000$N$
ProxylessNAS [3] | 4.4 | 75.3 | 200$N$
GreedyNAS-C [25] | 4.7 | 76.2 | 168+24$N$
SGNAS-C [10] | 4.7 | 76.2 | 285
Proposed method ($\epsilon=1.0$) | 4.3 | 75.8 | 164
MnasNet-A3 [23] | 5.2 | 76.7 | 40,000$N$
GreedyNAS-B [25] | 5.2 | 76.8 | 168+24$N$
SGNAS-B [10] | 5.5 | 76.8 | 285
Proposed method ($\epsilon=0.5$) | 5.4 | 76.8 | 164
SCARLET-A [4] | 6.7 | 76.9 | 240+48$N$
GreedyNAS-A [25] | 6.5 | 77.1 | 168+24$N$
SGNAS-A [10] | 6.0 | 77.1 | 285
Proposed method ($\epsilon=0.0$) | 6.5 | 77.2 | 164

####  4.2.2 Results and Discussions

Table 2 shows the results of the proposed method and the existing NAS methods.
The search cost indicates the cost to obtain $N$ optimized architectures. The
proposed method ($\epsilon$ = 1.0) demonstrates a prediction accuracy of
75.8$\%$ with 4.4M parameters. This accuracy is worse than that of the
existing methods; however, the number of parameters is lower than that in the
existing methods. The prediction accuracies of the proposed method ($\epsilon$
= 0.5 and 0.0) are 76.8$\%$ and 77.2$\%$, respectively, indicating that these
prediction accuracies are equal to or superior to those of existing methods.
The search cost of the proposed method is lower than that of existing NAS
methods. MnasNet and ProxylessNAS require $N$ architecture searches to obtain
$N$ architectures with varying complexities, similar to Method 1 described in
Section 4.1. GreedyNAS and SCARLET-NAS perform the architecture search
multiple times after optimizing the supernet weights, similar to Method 2
described in Section 4.1. Similar to the proposed method, SGNAS can obtain
multiple structures in a single architecture search. However, SGNAS needs to
train a DNN as the structure generator, which is more expensive than the
proposed method. Our method results in a lower search cost compared with that
in existing methods because it simultaneously updates multiple distributions
by sharing the samples via importance sampling and realizes an efficient
architecture search.

##  5 Conclusion

This paper has proposed a method for one-shot NAS that can efficiently find
multiple architectures with different architecture complexities. We extended
the method proposed in [17] to be able to use categorical variables and have
derived the natural gradient of the regularization term. Subsequently, we have
proposed an efficient method to search multiple architectures via importance
sampling. The experimental results produced using CIFAR-10 and ImageNet show
that the proposed method obtains multiple well-performed architectures with
different complexities by incurring less computational cost than the baseline
methods. Most NAS methods use fixed training hyperparameters, despite their
impact on the performance. A possible future work could be developing a method
for the joint optimization of both the architecture and training parameters,
further improving the NAS performance.

####  5.0.1 Acknowledgments

This work was partially supported by NEDO (JPNP18002), JSPS KAKENHI Grant
Number JP20H04240, and JST PRESTO Grant Number JPMJPR2133.

## References

  * [1] Akimoto, Y., Shirakawa, S., Yoshinari, N., Uchida, K., Saito, S., Nishida, K.: Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search. In: International Conference on Machine Learning (ICML) (2019)
  * [2] Amari, S.: Natural Gradient Works Efficiently in Learning. Neural Computation 10(2), 251–276 (1998)
  * [3] Cai, H., Zhu, L., Han, S.: ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In: International Conference on Learning Representations (ICLR) (2019)
  * [4] Chu, X., Zhang, B., Li, Q., Xu, R., Li, X.: SCARLET-NAS: Bridging the Gap Between Scalability and Fairness in Neural Architecture Search. In: ICCV Workshops (2021), https://arxiv.org/abs/1908.06022
  * [5] Chu, X., Zhou, T., Zhang, B., Li, J.: Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search. In: 16th Europoean Conference on Computer Vision (ECCV) (2020)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
  * [7] Elsken, T., Metzen, J.H., Hutter, F.: Neural Architecture Search: A Survey. Journal of Machine Learning Research 20(55), 1–21 (2019)
  * [8] Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single Path One-Shot Neural Architecture Search with Uniform Sampling. In: 16th Europoean Conference on Computer Vision (ECCV) (2020)
  * [9] Hu, J., Shen, L., Sun, G.: Squeeze-and-Excitation Networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [10] Huang, S., Chu, W.: Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [11] Krizhevsky, A.: Learning Multiple Layers of Features From Tiny Images. Tech. rep., Department of Computer Science, University of Toronto (2009)
  * [12] Liu, H., Simonyan, K., Yang, Y.: DARTS: Differentiable Architecture Search. In: International Conference on Learning Representations (ICLR) (2019)
  * [13] Loshchilov, I., Hutter, F.: SGDR: Stochastic Gradient Descent with Warm Restarts. In: International Conference on Learning Representations (ICLR) (2017)
  * [14] Ollivier, Y., Arnold, L., Auger, A., Hansen, N.: Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles. Journal of Machine Learning Research 18(18), 1–65 (2017)
  * [15] Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient Neural Architecture Search via Parameter Sharing. In: International Conference on Machine Learning (ICML) (2018)
  * [16] Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., Kurakin, A.: Large-Scale Evolution of Image Classifiers. In: International Conference on Machine Learning (ICML) (2017)
  * [17] Saito, S., Shirakawa, S.: Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures. In: 28th International Conference on Artificial Neural Networks (ICANN) (2019)
  * [18] Shirakawa, S., Akimoto, Y., Ouchi, K., Ohara, K.: Sample Reuse in the Covariance Matrix Adaptation Evolution Strategy Based on Importance Sampling. In: Genetic and Evolutionary Computation Conference (GECCO) (2015)
  * [19] Shirakawa, S., Akimoto, Y., Ouchi, K., Ohara, K.: Sample Reuse via Importance Sampling in Information Geometric Optimization. arXiv:1805.12388 (2018), https://arxiv.org/abs/1805.12388
  * [20] Shirakawa, S., Iwata, Y., Akimoto, Y.: Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling. In: 32nd AAAI Conference on Artificial Intelligence (AAAI) (2018)
  * [21] Suganuma, M., Shirakawa, S., Nagao, T.: A Genetic Programming Approach to Designing Convolutional Neural Network Architectures. In: Genetic and Evolutionary Computation Conference (GECCO) (2017)
  * [22] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the Inception Architecture for Computer Vision. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [23] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: MnasNet: Platform-Aware Neural Architecture Search for Mobile. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [24] Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [25] You, S., Huang, T., Yang, M., Wang, F., Qian, C., Zhang, C.: GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [26] Zhou, P., Xiong, C., Socher, R., Hoi, S.C.H.: Theory-Inspired Path-Regularized Differential Network Architecture Search. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 33, pp. 8296–8307 (2020)
  * [27] Zoph, B., Le, Q.V.: Neural Architecture Search with Reinforcement Learning. In: International Conference on Learning Representations (ICLR) (2017)
  * [28] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning Transferable Architectures for Scalable Image Recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)


# On Clausius’ approach to entropy and analogies in non-equilibrium

Giovanni Jona-Lasinio
Dipartimento di Fisica, Università di Roma “La Sapienza”,
and Istituto Nazionale di Fisica Nucleare,
Piazzale A. Moro 2, Roma 00185, Italy
E-mail<EMAIL_ADDRESS>
###### Abstract

In his ninth memoir Clausius summarizes the two principles of thermodynamics
as follows

> The whole mechanical theory of heat rests on two fundamental theorems, -
> that of equivalence of heat and work, and that of equivalence of
> transformations.

This paper contains an introduction to Clausius’ approach to entropy as
illustrated in his original articles and describes an analogy in the
macroscopic fluctuation theory of non–equilibrium diffusive systems.

##  1 Introduction

I often asked myself: “How did Clausius get the idea of entropy in a pure
thermodynamic approach?” . Entropy is a notion which carries some mystery with
it, as was emphasized by a qualified opinion. There is a quantity called
entropy in information theory which measures the content of information of a
message. A story tells that such a terminology was suggested to Shannon, the
inventor of the theory, by the famous mathematician John Von Neumann with the
comment

> “You should call it entropy, for two reasons. In the first place your
> uncertainty function has been used in statistical mechanics under that name,
> so it already has a name. In the second place, and more important, no one
> really knows what entropy is, so in a debate you will always have the
> advantage.”

The fourth, sixth and ninth memoirs of Clausius by the titles “On a modified
form of the second fundamental theorem of the mechanical theory of heat” [1];
“On the application of the theorem of the equivalence of transformations to
interior work” [2]; “On several convenient forms of the fundamental equations
of the mechanical theory of heat” [3] respectively, are very illuminating,
however for some reason his arguments did not attract so much attention and
are not reported in the treatises on thermodynamics I know.

Clausius idea, as developed in his fourth memoir, is that there must exist a
quantity called the the equivalence value of a transformation expressing the
strict connection of a transformation of heat into work with transfer of heat
from a higher temperature to a lower one as in a Carnot cycle. The equivalence
value is what he will call later the entropy. All the reversible
transformations connecting two given states have the same equivalence value,
so that entropy is an invariant under variation of the protocols connecting
those states.

The existence of entropy in classical equilibrium thermodynamics implies that
the integral $\int{\frac{\delta Q}{T}}$ does not depend on the particular
reversible quasi-static transformation joining two fixed initial and final
states. For example in the simple case of an ideal gas we have the exact
1-form

$dS=C_{p}{\frac{dT}{T}}-nR{\frac{dp}{p}}$ (1)

In this note, after illustrating Clausius’ reasoning 111I became aware only
recently of the paper [8] where Clausius approach is also reviewed., we wish
to show that a formula obtained in [4, 5] represents a generalization to
stationary non-equilibrium states in the context of an important class of
irreversible processes. These are the diffusive processes considered in the
macroscopic fluctuation theory (MFT).

The macroscopic fluctuation theory deals mainly with diffusive systems
boundary driven and in presence of external fields. For these systems a finite
time approach to macroscopic transformations has been developed [4, 5]. Quasi
static transformations are obtained as limits of scaled hydrodynamic equations
with suitable time dependent boundary conditions.

A quasi-static transformation is a continuous succession of stationary states
modified by $1/\tau$ corrections, where $\tau$ is a large time. Then one takes
the limit $\tau\rightarrow\infty$. Furthermore new exact expressions
connecting the equilibrium free energy or its derivatives to deviations from
stationarity follow from a systematic development in powers of $1/\tau$.

The fourth memoir of Clausius [1] clarifies how the idea of entropy emerged
thermodynamically. He concentrates on transformations rather than states and
wants to characterize the relationship between transforming heat into work and
the transition of heat between two temperatures in a generalized Carnot’s
cycle. The leading idea is that there exists a quantity called the equivalence
value of a transformation, characterising these different changes of state. He
then gives a general argument to show that such a quantity must have the form
$\int dQ/T$.

Let us quote from his fourth memoir

> “Carnot’s theorem,….,expresses a relation between two kinds of
> transformations, the transformation of heat into work, and the passage of
> heat from a warmer to a colder body, which may be regarded as the
> transformation of heat at the higher, into heat at a lower temperature…..
>
> For example, let the quantity of heat $Q$, produced in any manner whatever
> from work, be received by the body $K$; then by the foregoing cyclical
> process [he considers a generalized Carnot’s cycle] it can be withdrawn from
> $K$ and transformed back into work, but at the same time the quantity of
> heat $Q_{1}$ will pass from $K_{1}$ to $K_{2}$; or if the quantity of heat
> $Q_{1}$ had previously been transferred from $K_{1}$ to $K_{2}$, this can be
> again restored to $K_{1}$ by the reversed cyclical process wherby the
> transformation of work into the quantity of heat $Q$ of the temperature of
> the body $K$ will take place.
>
> We see, therefore, that these two transformations may be regarded as
> phenomena of the same nature, and we may call two transformations which can
> thus mutually replace one another _e_ quivalent. We have now to find the law
> according to which the transformations must be expressed as mathematical
> magnitudes, in order that the equivalence of two tranformations may be
> evident from the equality of their values. The mathematical value of a
> transformation thus determided may be called its _e_ quivalence-value
> (Aequivalenzwerth).”

Here is Clausius’ argument to determine such equivalence-value

> “With respect to the magnitude of the equivalence-value, it is first of all
> clear that the value of a transformation from work into heat must be
> proportional to the quantity of heat produced; and besides this it can only
> depend upon the temperature. Hence the equivalence-value of the
> transformation of work into the quantity of heat $Q$, of the temperature
> $t$, may be represented generally by
>
> $Q\,f(t)$ (2)
>
> wherein $f(t)$ is a function of the temperature which is the same for all
> cases….In a similar manner the value of the passage of the quantity of heat
> $Q$, from the temperature $t_{1}$ to the temperature $t_{2}$, must be
> proportional to the quantity $Q$, and besides this, can only depend on the
> two temperatures. In general therefore, it may be expressed by
>
> $Q\,F(t_{1},t_{2})$ (3)
>
> wherein $F(t_{1},t_{2})$ is a function of both temperatures, which is the
> same for all cases, and of which we at present only know that, without
> changing its numerical value, it must change its sign when the two
> temperatures are interchanged; so that
>
> $F(t_{1},t_{2})=-F(t_{2},t_{1})$ (4)
>
> ”

Clausius concludes that the relationship between $f(t)$ and $F(t_{1},t_{2})$
is

$F(t_{1},t_{2})=f(t_{2})-f(t_{1})$ (5)

He writes $f(t)=\frac{1}{T}$ where $T$ is provisionally a function of
temperature that will be identified later with the absolute temperature. In
his ninth memoir [3] he will summarize the two principles of thermodynamics as
follows

> “The whole mechanical theory of heat rests on two fundamental theorems, -
> that of equivalence of heat and work, and that of equivalence of
> transformations.”

Some comments are here in order. The equivalence of transformations in the
above reasoning is restricted to heat going into work and heat passing from
higher temperature to lower temperature. This is equivalent to the second
principle. However once the existence of entropy is established it follows
that to all reversible transformations between given initial and final states
of a system can be attributed the same equivalence value i.e. the difference
of entropies between initial and final state.

We now show that an analogy exists in the dissipative context of
non–equilibrium diffusive systems. To make the paper selfcontained we collect
here some facts and formalism of the macroscopic fluctuation theory, a kind of
ritual, which can be found in greater detail in [4, 5] and references therein.

##  2 Macroscopic Dynamics

We denote by $\Lambda\subset{\mathbf{R}}^{d}$ the bounded region occupied by
the system, by $\partial\Lambda$ the boundary of $\Lambda$, by $x$ the
macroscopic space coordinates and by $t$ the macroscopic time. The system is
in contact with boundary reservoirs, characterized by their chemical potential
$\lambda(t,x)$, $x\in\partial\Lambda$ and under the action of an external
field $E(t,x)$.

At the macroscopic level the assumption is that the system is completely
described by the local density $\rho(t,x)$ and the local density current
$j(t,x)$ and their evolution is given by the continuity equation together with
the constitutive equation which expresses the current as a function of the
density. Namely,

$\begin{cases}\partial_{t}\rho(t)+\nabla\cdot j(t)=0,\\\
j(t)=J(t,\rho(t)),\end{cases}$ (6)

where we omit the explicit dependence on the space variable $x\in\Lambda$. For
driven diffusive systems the constitutive equation takes the form

$J(t,\rho)=-D(\rho)\nabla\rho+\chi(\rho)\,E(t),$ (7)

where the _diffusion coefficient_ $D(\rho)$ and the _mobility_ $\chi(\rho)$
are assumed to be $d\times d$ symmetric and positive definite matrices. This
holds in the context of stochastic lattice gases [4]. Equation (7) relies on
the local equilibrium hypothesis, small local gradients and linear response to
the external field. The evolution of the density is thus given by the driven
diffusive equation

$\partial_{t}\rho(t)+\nabla\cdot\big{(}\chi(\rho)E(t)\big{)}=\nabla\cdot\big{(}D(\rho)\nabla\rho\big{)}.$
(8)

The transport coefficients $D$ and $\chi$ satisfy the local Einstein relation

$D(\rho)=\chi(\rho)\,f^{\prime\prime}(\rho),$ (9)

where $f$ is the equilibrium free energy per unit volume which we assume to
depend on the local value $\rho(x)$.

Equations (6)–(7) have to be supplemented by the appropriate boundary
conditions on $\partial\Lambda$ due to the interaction with the external
reservoirs. If $\lambda(t,x)$, $x\in\partial\Lambda$, is the chemical
potential of the external reservoirs, the boundary condition reads

$f^{\prime}\big{(}\rho(t,x)\big{)}=\lambda(t,x),\qquad x\in\partial\Lambda.$
(10)

If the chemical potential and external field do not depend on time, we denote
by $\bar{\rho}=\bar{\rho}_{\lambda,E}$ the stationary solution of (8),(10),

$\begin{cases}\nabla\cdot
J(\bar{\rho})=\nabla\cdot\big{(}-D(\bar{\rho})\nabla\bar{\rho}+\chi(\bar{\rho})\,E\big{)}=0,\\\
f^{\prime}(\bar{\rho}(x))=\lambda(x),\qquad x\in\partial\Lambda.\end{cases}$
(11)

We will assume that this stationary solution is unique. The stationary density
profile $\bar{\rho}$ is characterized by the vanishing of the divergence of
the associated current, $\nabla\cdot J(\bar{\rho})=0$. A special situation is
when the current itself vanishes, $J(\bar{\rho})=0$; if this is the case,
which means that the boundary conditions and the external field balance each
other, we say that the system is in a macroscopic equilibrium state; this can
be viewed as a macroscopic counterpart to detailed balance [6].

Homogeneous equilibrium states correspond to the case in which the external
field vanishes and the chemical potential is constant in space. Inhomogeneous
equilibrium states correspond to the case in which the external field is
gradient, $E=-\nabla U$, and it is possible to choose the arbitrary constant
in the definition of $U$ such that $U(x)=-\lambda(x)$, $x\in\partial\Lambda$.
The stationary equation $\nabla\cdot J=0$ with the associated boundary
conditions play a role akin to the equation of state in equilibrium
thermodynamics.

##  3 Transformations and Energy Balance

Consider a system in a time dependent environment, that is, $E$ and $\lambda$
depend on time. The work done by the environment on the system in the time
interval $[0,T]$ is

$\begin{split}W_{[0,T]}=&\int_{0}^{T}\\!dt\,\Big{\\{}\int_{\Lambda}\\!dx\,j(t)\cdot
E(t)-\int_{\partial\Lambda}\\!d\sigma\,\lambda(t)\>j(t)\cdot\hat{n}\Big{\\}},\end{split}$
(12)

where $\hat{n}$ is the outer normal to $\partial\Lambda$ and $d\sigma$ is the
surface measure on $\partial\Lambda$. The first term on the right hand side is
the energy provided by the external field while the second is the energy
provided by the reservoirs.

Fix time dependent paths $\lambda(t)$ of the chemical potential and $E(t)$ of
the field. Given a density profile $\rho_{0}$, let $\rho(t)$, $j(t)$, $t\geq
0$, be the solution of (6)–(10) with initial condition $\rho_{0}$. By using
the Einstein relation (9) and the boundary condition
$f^{\prime}(\rho(t))=\lambda(t)$, an application of the divergence theorem
yields

$W_{[0,T]}\,=\,F(\rho(T))-F(\rho(0))+\,\int_{0}^{T}\\!dt\int_{\Lambda}\\!dx\;j(t)\cdot\chi(\rho(t))^{-1}j(t),$
(13)

where $F$ is the equilibrium free energy functional,

$F(\rho)=\int_{\Lambda}\\!dx\>f(\rho(x)).$ (14)

The step from (12) to (13) uses the constitutive equation (7) and the Einstein
relationship (9). It tells us that for the class of systems considered the
change of the free energy $\Delta F$ is, as expected, the difference between
the total work and the total dissipation. Notice that if the system is out of
equilibrium for an infinite time both these quantities are infinite. The
currents $j(t)$ must be evaluated on the solutions of the hydrodynamic
equations.

From (13) follows the Clausius inequality

$W_{[0,T]}\,\geq\,F(\rho(T))-F(\rho(0))$ (15)

For quasi-static transformations between nonequilibrium stationary states, the
Clausius inequality does not carry any significant information. In fact, the
energy dissipated along such transformations will necessarily include the
contribution needed to maintain the nonequilibrium stationary states, which is
infinite in an unbounded time window. It is however possible to formulate a
meaningful version of the inequality for nonequilibrium states by introducing
a _renormalized work_ $W^{\mathrm{ren}}$ that is defined by subtracting the
energy needed to maintain the nonequilibrium state [4]. This energy interprets
in the present context what is called housekeping heat in [7]

To analyze transformations over a long interval $[0,\tau]$ driven by slowly
changing boundary conditions and external fields, it is convenient to rescale
time and introduce the dimensionless variable $s={t}/{\tau}$. The _protocol_
of a transformation is defined therefore by a choice of the external drivings
$E(s,x)$, $x\in\Lambda$, and $\lambda(s,x)$, $x\in\partial\Lambda$,
$s\in[0,1]$.

Introduce the expansion

$\begin{array}[]{l}\displaystyle{\rho^{\tau}(\tau
s)=\bar{\rho}(s)+\tfrac{1}{\tau}\,r(s)+o\big{(}\tfrac{1}{\tau}\big{)}\,,}\\\
\displaystyle{j^{\tau}(\tau
s)=J(s,\bar{\rho}(s))+\tfrac{1}{\tau}\,g(s)+o\big{(}\tfrac{1}{\tau}\big{)}\,.}\end{array}$
(16)

It is easy to see that $r,g$ obey the following equations

$\left\\{\begin{array}[]{l}\partial_{s}\bar{\rho}(s)+\nabla\cdot g(s)=0\\\
g(s)=-\nabla\cdot\big{(}D(\bar{\rho}(s))r(s)\big{)}+r(s)\chi^{\prime}(\bar{\rho}(s))E(s)\\\
r(s,x)=0,\;x\in\partial\Lambda\end{array}\right.$ (17)

which implies that $r,g$ depend linearly on $\partial_{s}\bar{\rho}$.

The formula we want to discuss is

$\begin{split}F\big{(}\bar{\rho}(1)\big{)}-F\big{(}\bar{\rho}(0)\big{)}&\;=\int_{0}^{1}\\!ds\int_{\Lambda}\\!dx\,E(s)\cdot
g(s)-\int_{0}^{1}\\!ds\int_{\partial\Lambda}\\!d\sigma\,\lambda(s)g(s)\cdot\hat{n}\\\
&\;\;+\int_{0}^{1}\\!ds\int_{\Lambda}\\!dx\,r(s)J(s,\bar{\rho}(s))\cdot(\chi^{-1})^{\prime}\big{(}\bar{\rho}(s)\big{)}J(s,\bar{\rho}(s))\,.\end{split}$
(18)

For a derivation we refer to [4, 5].

Taking the pair $E(x,s),\lambda(x,s)$ as our state variables, a transformation
corresponds to a path $\gamma$ in an infinite dimensional space. Then the
right hand side of (18) can be represented as a line integral of an exact 1–
form

$\int_{\gamma}A_{1}\,dE+A_{2}\,d\lambda\,.$ (19)

We now take into account that $\bar{\rho}$ depends on time only through
$\lambda,E$

$\partial_{s}\bar{\rho}={\frac{\delta\bar{\rho}}{\delta\lambda}}\partial_{s}\lambda+{\frac{\delta\bar{\rho}}{\delta
E}}\partial_{s}E\,.$ (20)

Inserting this expression into the formulas for $r,g$ in (18), separating
$\partial_{s}\lambda$ and $\partial_{s}E$ we can obtain explicitly
$A_{1},A_{2}$.

Equation (18) establishes a mathematical equivalence of quasi–static
transformations for a class of dissipative systems. These transformations can
be energetically very different as the energy necessary to keep the system out
of equilibrium per unit of time can differ considerably. Concrete examples
will be discussed elsewhere.

##  4 Concluding remarks

For diffusive systems different equivalent transformations in the sense of
(18) can be very different as far as energy consumption is concerned. To be
concrete let us consider a conductor in a potential difference $\Delta V$ that
we want to increase to a value $\Delta V^{\prime}$. A quasi–static
transformation increasing slowly $\Delta V$ would produce a very large
quantity of heat, infinite in the limit, that we can possibly reduce through
the following procedure that we formulate in a slightly more general form.

Consider a quasi-static transformation between the stationary states
corresponding to the values $E_{i},\lambda_{i}$ and $E_{f},\lambda_{f}$. You
may ideally split the transformation in the following way.

Change abruptly the field from $E_{i}$ to $E_{i}^{{}^{\prime}}$ where
$E_{i}^{{}^{\prime}}$ is the value corresponding to equilibrium, i.e. $J=0$,
with chemical potential $\lambda_{i}$. The system will relax to equilibrium.
Then perform a quasi-static transformation via equilibrium states until you
reach the final value $\lambda_{f}$. No housekeeping heat. Finally bring
abruptly the field to its final value $E_{f}$. The system will relax to its
final stationary state.

Of course in every concrete case one must estimate the energy cost of the
splitting and decide whether over a long but finite interval of time of a real
transformation it is convenient. In [5] we discussed a criterion of optimality
based on the minimisation of the renormalized work which led to some
unexpected outcome also for a system of free particles.

In conclusion, it is somewhat surprising that an equivalence class of
transformations could be defined for irreversible dissipative systems like
purely diffusive systems. It is interesting to investigate the possibility of
generalising this analysis to a more general setting like reaction–diffusion
macroscopic dynamics where presumably we can extend the macroscopic
fluctuation theory.

### Acknowledgements

I wish to acknowledge my long standing collaboration with L. Bertini, A. De
Sole, D. Gabrielli, C. Landim which ispired this paper.

## References

  * [1] Clausius R., _On a modified form of the second fundamental theorem of the mechanical theory of heat_ , Pogg. Annalen, xciii 481 (1854); translated in Phil. Mag. xii 81 (1856).
  * [2] Clausius R. , _“On the application of the theorem of the equivalence of transformations to interior work”_ , Pogg. Annalen, cxvi 73 (1862); translated in Phil. Mag. S.4. xxiv 81 (1862).
  * [3] Clausius R.; _On several convenient forms of the fundamental equations of the mechanical theory of heat_ , Pogg. Annalen cxxv 353 (1865); translated in Journ. de Liouville 10 361 (1865).
  * [4] Bertini L., De Sole A., Gabrielli D., Jona-Lasinio G., Landim C.; _Macroscopic fluctuation theory_ , Rev. Mod. Phys. 87 593 (2015), ArXiv:1404.6466.
  * [5] Bertini L., De Sole A., Gabrielli D., Jona-Lasinio G., Landim C.; _Quantitative analysis of Clausius inequality_ , J. Stat. Mech. Theory Exp. P10018 (2015).
  * [6] Bertini L., De Sole A., Gabrielli D., Jona-Lasinio G., Landim C.; _Towards a non–equilibrium thermodynamics: a self–contained macroscopic description of driven diffusive systems_ , J. Stat. Phys. 135, 857 (2009).
  * [7] Oono Y., Paniconi M., _Steady state thermodynamics_ , Progr. Theor. Phys. Suppl. 130, 29 (1998).
  * [8] Ti-Wei Xue, Zeng-Yuan Guo, _What is the Real Clausius Statement of the Second Law of Thermodynamics?_ , Entropy 2019, 21, 926.