# Three Dimensional Activity Volcano Plot under External Electric Field

Changming Ke  Key Laboratory for Quantum Materials of Zhejiang Province,
School of Science, Westlake University, Hangzhou, Zhejiang 310024, China
Institute of Natural Sciences, Westlake Institute for Advanced Study,
Hangzhou, Zhejiang 310024, China  Zijing Lin  Hefei National Laboratory for
Physical Sciences at Microscales, CAS Key Laboratory of Strongly-Coupled
Quantum Matter Physics, Department of Physics, University of Science and
Technology of China, Hefei 230026, China  Shi Liu<EMAIL_ADDRESS>Key
Laboratory for Quantum Materials of Zhejiang Province, School of Science,
Westlake University, Hangzhou, Zhejiang 310024, China  Institute of Natural
Sciences, Westlake Institute for Advanced Study, Hangzhou, Zhejiang 310024,
China

###### Abstract

An external electric field (EEF) can impact a broad range of catalytic
processes beyond redox systems. Computational design of catalysts under EEFs
targeting specific operation conditions essentially requires accurate
predictions of the response of a complex physicochemical system to collective
parameters such as EEF strength/direction and temperature. Here, we develop a
first-principles-based multiscale approach that enables efficient EEF-
dependent kinetic modeling of heterogeneous catalysis. Taking steam reforming
of methanol as an example, we find that the methanol conversion rate exhibits
strong nonlinear response to temperature and EEF, and the optimal field line
and constant carbon concentration line defined in the temperature–EEF
parameter space serve as powerful metrics for catalyst design. Assisted with a
deep neural network, we establish a three-dimensional activity volcano plot
under EEFs for thousands of metallic alloys.

Much like Earth’s climate and spin glass, heterogeneous catalysis is an
archetypal complex system and plays a pivotal role in our society. Electric
field (EF) can manipulate the chemical reactivity and selectivity by tuning
the relative stability of polar or ionic reactants and transition states in
chemical reactions [1, 2, 3, 4, 5, 6], opening up exciting opportunities to
control catalytic reactions in an on-demand manner using EFs, hereinafter
referred to as electrostatic catalysis. To take full advantage of
electrostatic catalysis, a fundamental challenge is to predict the response of
heterogeneous catalysis, a complex physicochemical system often involving a
network of chemical reactions and physical processes, to collective reaction
parameters, specifically field strength and temperature.

The presence of an EF under realistic reaction conditions can have both
intrinsic and extrinsic origins [7, 2, 4, 6]. It is more convenient in
practice to directly apply an external EF (EEF) to a reactor as the strength
and direction of the field is readily tunable. To apply EEFs in large-scale
reactions, several reactor systems have been developed [8, 9, 10], i.e.,
probe-bed-probe reactors that have a conductive catalyst bed placed in the gap
between two external probes and continuous-circuit reactors in which the
conductive catalyst bed is integrated as an electrical circuit. The catalyst
bed often consists of catalytically active metal nanoparticles that can
generate very high surface EFs in the presence of EEFs due to the curvature-
induced field enhancement effect [11]. Additionally, surface EFs are always
perpendicular to the metal surface, potentially allowing for a fine control of
the relative orientation between the EEF and adsorbed molecules. Therefore,
understanding the effects of EEFs on heterogeneous catalysis by metals is a
major focus of electrostatic catalysis [12, 13, 11].

Methanol, as an important hydrogen-storage liquid fuel [14], can be used for
on-board hydrogen production through steam reforming of methanol (SRM) [15],
in which methanol reacts with steam and produces hydrogen, CO, and CO2 at
about 300 ∘C. However, the extensively used catalysts in industry often suffer
from catalyst deactivation due to coke deposition and sintering [16]. Since
the elementary reactions of SRM involve adsorbates possessing large electric
dipole moments, we expect that an EEF will bring appreciable impact on the
kinetics and thermodynamics of the SRM process as the energetics of molecules
of large dipole moments are susceptible to EEFs [17, 5]. Considering the
critical role of SRM played in the green methanol economy [14, 18], it is
imperative to design next-generation catalysts superior to current ones,
particularly for low-temperature SRM. In this work, we propose to use EEF as a
“smart agent” to improve both the catalytic activity and coke resistance.

First-principles-based computational catalyst design is emerging as a
promising approach to obtain high-quality catalysts [19, 20, 21]. Pertinent to
electrostatic catalysis, finite-field density functional theory (DFT)
calculations have been used to quantify the activation energies ($\Delta
E_{a}$) of elementary reactions on metal surfaces in the presence of EEFs of
varying magnitudes, revealing important atomistic insights that helped the
understanding of EEF effects [3, 13, 22]. However, most previous studies only
focused on a few representative electric fields (e. g., negative versus
positive) and their impact on the catalytic activity of a specific metal (e.
g., Ni, Ag) [13, 22], partly due to the expensive computational cost of
finite-field DFT calculations. For a complex process like SRM comprising a
network of elementary reactions, the EEF dependence of the overall catalytic
performance is likely nonlinear such that the search of an optimal EEF should
be performed in order to maximize the efficiency but has never been done.
Moreover, the design of high-quality electrostatic catalysts targeting
specific operation conditions requires detailed understandings of the
collective effects of EEFs and various reaction parameters such as
temperature, gas composition, and partial pressure. All these difficulties
essentially forbid a DFT-based high-throughput computational catalyst
screening in the presence of EEFs.

Here, using SRM as an example, we develop a deep-learning-assisted first-
principles-based multiscale method that leads to the first three-dimensional
(3D) activity volcano plot under EEFs, quantifying the field-dependent SRM
activity for thousands of metallic alloys. The multsicale framework consists
of three steps, as shown in Fig. 1a. First, we derive a general analytical
relationship between the electric field ($\mathbf{F}$) and $\Delta E_{a}$ of a
surface reaction within the harmonic approximation. This enables a rapid and
accurate predication of $\Delta E_{a}[\mathbf{F}]$ at any given field
strength. The analytical relationship is then incorporated into a microkinetic
mode of SRM on Ni(111) surface, resulting in a continuous stirred-tank reactor
model to simulate SRM under EEFs. The microkinetic modeling reveals a highly
nonlinear temperature-dependent EEF effect: a positive EEF can increase the
conversion of CH3OH at high temperatures ($>$ 350∘C) but suppress the
conversion at low temperatures ($<$ 250∘C), highlighting the necessity of
multiscale modeling for catalyst design under EEFs. Finally, by combining the
linear scaling relationship [23, 24], a DFT-based deep neural network (DNN)
that rapidly predicts C* and O* adsorption energies [19], and a simplified
kinetic model derived from the microkinetic model, we are able to construct
the 3D activity volcano plot to identify high-quality catalysts for low-
temperature SRM under EEFs.

Figure 1: (a) Workflow of multiscale catalyst design under EEFs. (b)
Comparison of EEF-induced activation energy change ($\Delta\Delta
E_{a}[\mathbf{F}]$) and adsorption energy change ($\Delta E_{\rm
ad}[\mathbf{F}]$) estimated with the analytical relationships and direct DFT
calculations for tightly bonded adsorbates and transition states. (c)
Comparison of analytical and DFT values of $\Delta E_{\rm ad}[\mathbf{F}]$ for
three representative species. The inset shows H2O will desorbe from the
surface due to a strong negative field.

MethodDFT calculations are performed using Vienna ab initio Simulation Package
(VASP) [25, 26]. The interaction between core ion and electrons is described
by the projector augmented wave (PAW) method [27]. The Perdew-Wang-91 (PW91)
functional is chosen as the exchange-correlation functional [28]. The plane-
wave kinetic energy cutoff is 400 eV and a 5$\times$5$\times$1 Monkhorst-Pack
grid is used to sample the Brillouin zone. A three-layer 3$\times$3 Ni slab
with a vacuum region of more than 10 Å thick is constructed to simulate the
Ni(111) surface. The method proposed by Neugebauer and Scheffler [29] is
employed to apply an EEF normal to the metal surface. Climbing image nudged
elastic band (CL-NEB) method [30] and dimer method [31] are used to identify
transition states of surface reactions, and the electronic energy and atomic
force are converged respectively to 10-7 eV and 0.01 eV/Å. More computational
details can be found in Supplementary Materials.

Activation and adsorption energy under EEFs within the harmonic
approximationFor an elementary surface reaction step, the EEF-induced changes
in adsorption and activation energy have been attributed to various factors
such as modified charge transfer between adsorbates and metal surfaces [32],
bond elongation/contraction of adsorbates as well as the shift of metal work
function [33]. These important yet intimately coupled atomistic mechanisms
seem to suggest a quantitative determination of the EEF-induced energy change
can only be achieved via genuine ab initio modeling that captures all the
factors. We now show that $\Delta E_{a}[\mathbf{F}]$ at a given field
($\mathbf{F}$) can be readily obtained using only zero-field parameters within
the harmonic approximation.

Without loss of generality, a simple elementary reaction,
R*$\rightarrow$TS*$\rightarrow$P* is considered, where R*, TS*, and P* are
reactant, transition state, and product adsorbed on the metal surface,
respectively, and the EEF is applied along the surface normal. For an isolated
species $i$ under an EEF, its free energy in the form of Taylor series up to
the second order is

$E_{i}[\mathbf{F}]=E^{0}-\mathbf{d}_{i}\cdot\mathbf{F}-\frac{1}{2}\alpha_{i}\mathbf{F}^{2},$
(1)

where $E^{0}$ is the zero-field energy, $\mathbf{d}$ is the dipole moment, and
$\alpha$ is the polarizability. Equation 1 applies to both isolated molecules
as well as the slab model with adsorbates (denoted as $s$+$i$). It is
straightforward to derive

$\Delta E_{a}[\mathbf{F}]=\Delta E_{a}^{0}-(\mathbf{d}_{s+\rm
TS^{*}}-\mathbf{d}_{s+\rm R^{*}})\cdot\mathbf{F}-\frac{1}{2}(\alpha_{s+\rm
TS^{*}}-\alpha_{s+\rm R^{*}})\mathbf{F}^{2}$ (2)

where $\Delta E^{0}_{a}=E^{0}_{s+\rm TS^{*}}-E^{0}_{s+\rm R^{*}}$ is the zero-
field activation energy. We note that $\mathbf{d}_{s+\rm TS^{*}}$ and
$\alpha_{s+\rm TS^{*}}$ ($\mathbf{d}_{s+\rm R^{*}}$ and $\alpha_{s+\rm
R^{*}}$) are the electric dipole moment and polarizability, respectively, of
the whole bounded system comprised of the adsorbed TS* (R*) and the slab used
to model the surface. It is evident that all reaction-specific parameters
($\Delta E_{a}^{0}$, $\mathbf{d}$, $\alpha$ ) in Eq. 2 can be computed from
DFT and perturbation theory at zero field. A generalization of Eq. 2 gives the
adsorption energy under EEFs,

$E_{\rm ad}[\mathbf{F}]=E_{\rm
ad}^{0}-(\mathbf{d}_{s+i}-\mathbf{d}_{s})\cdot\mathbf{F}-\frac{1}{2}(\alpha_{s+i}-\alpha_{s})\mathbf{F}^{2}$
(3)

where $E_{\rm ad}^{0}=E^{0}_{s+i}-E^{0}_{s}-E^{0}_{i}$. We note that our
definition of adsorption energy captures the effect of EEF-driven gas
diffusion and is subtlety different from the conventional definition (see
details in Supplementary Materials, Sect. II).

Expressions similar to Eqs. 2–3 have been derived previously[3, 22] but,
surprisingly, have never been used to calculate $\Delta E_{a}[\mathbf{F}]$ or
$E_{\rm ad}[\mathbf{F}]$, likely because of the general assumption that EEF-
induced structural and electronic changes of metal surface-adsorbate complex
would be highly anharmonic thus beyond the applicability of harmonic
approximation. We compare the EEF-induced energy change, $\Delta\Delta
E_{a}[\mathbf{F}]=\Delta E_{a}[\mathbf{F}]-\Delta E_{a}^{0}$ and $\Delta
E_{\rm ad}[\mathbf{F}]=E_{\rm ad}[\mathbf{F}]-E_{\rm ad}^{0}$, obtained with
Eqs. 2–3 using only zero-field parameters (see values in Supplementary
Materials, Sect. III) and those from direct DFT calculations in Fig.1b. With a
mean absolute error of only 1 meV and 2 meV for $\Delta\Delta
E_{a}[\mathbf{F}]$ and $\Delta E_{\rm ad}[\mathbf{F}]$, respectively, the
simple analytical relationships described by Eqs. 2–3 have remarkable accuracy
over a wide range of field strengths. We further compare the analytical and
DFT values of $\Delta E_{\rm ad}[\mathbf{F}]$ for three representative
molecules, H2O, CH3OH, and CH3O in Fig.1c, and find a satisfying agreement for
$-0.4<\mathbf{F}<0.7$ eV/Å. Consistent with previous DFT investigations [3,
34], a negative field will induce desorption ($\Delta E_{\rm
ad}[\mathbf{F}]>0$) while a positive field may facilitate the adsorption
($\Delta E_{\rm ad}[\mathbf{F}]<0$) for a molecule with negative electric
dipole moment on the metal surface. It is only when the molecules become
desorbed due to a strong negative field (e. g., $-0.5$ V/Å for H2O), the
analytical value of $\Delta E_{\rm ad}[\mathbf{F}]$ starts deviating from the
DFT result.

The demonstrated quantitative nature of a rather simple analytical theory
linking field strength to EEF-induced energy change seems counterintuitive.
From Eqs. 2–3, it is clear that only the whole bounded system (the slab with
an adsorbed molecule) is relevant to EEF-dependent terms. We argue that for
the the whole bounded system, the impact of an EEF on the total energy is a
small perturbation that could be well described within the harmonic
approximation, despite highly anharmonic local changes between the adsorbate
and the metal surface (see numerical verification in Fig. S3).

Figure 2: Nonlinear temperature-dependent EEF effect in SRM. (a) Reaction
network of SRM considered in this work. The green arrows highlight two RDSs.
(b) Methanol conversion rate (left) and CO* coverage (right) as a function of
electric field strength at different temperatures for S:M = 3:1. (c)
Sensitivity analysis of SRM. (d) Impacts of EEFs on the two RDSs at 350∘C,
300∘C, and 250∘C. The width of the arrow scales with the rate. (e) Countour
plot of methanol conversion rate as a function of temperature and field
strength for S:M = 1:1.

Microkinetics of SRM under EEFsThe reaction network of SRM, illustrated in
Fig.2a, includes CH3OH and H2O dehydrogenation and CO* oxidation. Based on
Eqs. 2–3, we construct a microkinetic model that simulates the Ni-catalyzed
SRM process in the presence of EEFs under realistic conditions that take into
account the impacts of gas composition, temperature, and inlet velocity. The
validity of the microkinetic model is confirmed by comparing theoretical
predictions with available experimental results (see details in Supplementary
Materials, Sect. IV). For a 3:1 steam to methanol ratio (S:M = 3:1), the
temperature- and field-dependence of the conversion of CH3OH is presented in
Fig. 2b where we define an “optimal field line” (OFL) that tracks the electric
field resulting in the highest conversion. The OFL reveals several interesting
characteristics of SRM under EEFs. At high temperatures $\geq 310^{\circ}$C, a
positive field can promote the overall reaction rate compared to the zero-
field case, and the degree of enhancement scales with the field strength.
Interestingly, at an intermediate temperature such as $300^{\circ}$C, a large
positive field that is to the right of the OFL will suppress the conversion.
When the OFL crosses the zero field and enters into the negative field region,
it means a negative field now can facilitate the conversion; this happens at
$\approx$250 ∘C, but a too strong negative field also becomes unfavorable.

We perform sensitivity analysis by computing the partial derivatives of
methanol conversion rate with respect to the rate constant of the elementary
reaction [35]. As shown in Fig. 2c, there are two RDSs in SRM, the
dehydrogenation of CH3O*, CH3O*+* $\rightarrow$ CH2O*+H*, and the consumption
of CO*, CO*+O*$\rightarrow$CO2. The nontrivial evolution of OFL is due to a
delicate balance between the two RDSs with the dehydrogenation of CH3O* being
more EEF sensitive while the consumption of CO* being more temperature
sensitive. Specifically, as shown in Fig. 2d, at a high temperature of
$350^{\circ}$C, the overall reaction rate of SRM is dictated by the
dehydrogenation of CH3O* as the consumption of CO* is fast enough at such high
temperature. Therefore, a stronger positive EEF that reduces the activation
energy of CH3O* dehydration leads to increased methanol conversion. When the
temperature is reduced to $300^{\circ}$C, the consumption of CO* remains fast
such that applying a positive EEF below $\mathbf{F}_{\rm OFL}$ can still
facilitate the conversion by increasing the rate of CH3O* dehydrogenation.
However, after passing $\mathbf{F}_{\rm OFL}$, the CO* generation will
outspeed the consumption, causing CO poisoning of Ni. At a low temperature of
$250^{\circ}$C, the CO* consumption becomes rather slow thus requiring a
negative field to suppress CH3O* dehydration to inhibit CO poisoning; a too
strong negative field that severely slows down CH3O* dehydration will
unsurprisingly cause a low methanol conversion rate. Simply put, the OFL
essentially marks the boundary between CH3OH-dehydrogenation-controlled region
and CO-poisoning-controlled region.

It is well known that coking is the most common deactivation mechanism of
industrial SRM catalysts [36, 37], and increasing the steam pressure is the
general approach to suppress coking. Though a high S:M ratio (e. g., 3:1) is
beneficial for enhancing coke resistance, it will raise energy consumption for
steam heating, reduce the methanol conversion rate, and decrease the H2
content of final products. It would be helpful to have an efficient tool to
identify the optimal conditions for balanced SRM rate and coke resistance. For
S:M = 1:1, we construct a contour plot of methanol conversion rate as a
function of temperature and EEF strength (Fig. 2e), showing both OFL and
constant carbon concentration lines (3CLs). For a target level of coke
resistance (a selected 3CL, denoted as 3CLt), the lower part of the merged OFL
and 3CLt gives the best operating EEF for a wide range of temperatures.

3D activity volcano plot under EEFsA detailed study of the microkinetic model
of SRM shows that the methanol conversion rate ($r_{\rm M}[\mathbf{F}]$) can
be quantified with a simplified pathway [38],

${\rm CH_{3}OH}\leftrightarrows{\rm CH_{3}OH*}\leftrightarrows{\rm
CH_{3}O*}\stackrel{{\scriptstyle k^{\rm RDS}}}{{\longrightarrow}}{\rm
CH_{2}O*}$

in which CH3O*+* $\rightarrow$ CH2O*+H* is the RDS. The corresponding rate
equation reads

${r_{\rm M}[\mathbf{F}]=A\cdot\exp(-\frac{\Delta E_{a}^{\rm
RDS}[\mathbf{F}]}{RT})\frac{P_{\rm CH_{3}OH}[\mathbf{F}]}{\sqrt{P_{\rm
H_{2}}[\mathbf{F}]}}\cdot\theta[\mathbf{F}]^{2}.}$ (4)

Here, $P_{\rm CH_{3}OH}[\mathbf{F}]$, $P_{\rm H_{2}}[\mathbf{F}]$,
$\theta[\mathbf{F}]$ are the field-dependent partial pressure of CH3OH, H2,
and the coverage of active sites that depend on the adsorption energies of
CO*, O*, and H* (see Section VI in Supplementary Materials); $A$ is a
temperature dependent parameter deduced from DFT calculations. The simplified
kinetic model based on Eq. 4 is capable of reproducing the results of the
microkinetic model based on the complete reaction network but with a fraction
of the computational cost.

To construct the activity volcano plot under EEFs for a large number of
metallic catalysts, accurate and rapid determinations of $E_{\rm
ad}[\mathbf{F}]$ and $\Delta E_{a}[\mathbf{F}]$ are prerequisite. We find that
the EEF-induced energy change turns out to be rather insensitive to the metal
type, which greatly simplifies the problem (see Supplementary Material Sect.
VII). That is, for an adsorbate $i$ on a given metal surface ($m$), the field-
dependent adsorption energy is reduced to $E_{{\rm ad}}[\mathbf{F},m]=E_{{\rm
ad}}^{0}[m]-(\mathbf{d}_{s_{0}+i}-\mathbf{d}_{s_{0}})\cdot\mathbf{F}-\frac{1}{2}(\alpha_{s_{0}+i}-\alpha_{s_{0}})\mathbf{F}^{2}$,
where $E_{{\rm ad}}^{0}[m]$ is the zero-field adsorption energy on metal $m$
that can be estimated using C* and O* adsorption energies on metal $m$ with
the well-known linear scaling relationship [19] while the last two terms only
depend on $\mathbf{F}$ and are computed using a reference metal surface
$s_{0}$. Equipped with a DNN that can quickly predict the zero-field
adsorption energies of C* and O* [19] for various metallic alloys and the
simplified kinetic model (Eq. 4), we finally construct a 3D activity volcano
plot that quantitatively predicts the menthol conversion in the presence of
EEFs for 1711 metallic alloys (Fig. 3a) at a low temperature of 200 ∘C. This
3D volcano plot contains rich information and is worthy of detailed
investigations. Here we only highlight the catalytic performance of NiZn.
Despite having a low SRM activity at zero field, NiZn can promote the methanol
conversion in the presence of an EEF of 0.5 V/Å at 200 ∘C. Combined with its
proved coke resistance [19], EEF-assisted NiZn is recommended as an efficient
and environment-friendly catalyst for SRM at 200 ∘C and S:M = 1:1.

Figure 3:  (a) 3D activity volcano plot for SRM at 200 ∘C and S:M = 1:1 under
EEFs with a slice shown in (b).

Conclusion.In summary, we demonstrate that a simple analytical theory within
the harmonic approximation can accurately predict finite-field energetics
using only zero-field parameters, eliminating the needs of expensive finite-
field DFT calculations for field-dependent microkinetic modeling. Focusing on
an important process in green methanol economy, steam reforming of methanol,
we reveal nontrivial collective effects of EEF and temperature: a positive EEF
enhances the methanol conversion at higher temperatures while suppressing the
overall reaction at lower temperatures. The introduction of OFL and 3CLt in
the temperature–EEF parameter space allows for facile determinations of best
operating EEF for a wide range of temperatures. Finally, using a chemistry-
based simplified kinetic model and a first-principles-based DNN, we construct
a 3D activity volcano plot under EEFs for 1711 metallic alloys and identify
EEF-assisted NiZn for low-temperature steam reforming of methanol. We expect
the multiscale approach developed in this work can be readily applied to other
heterogeneous catalysis by metal under EEFs.

###### Acknowledgements.

C.K., S.L. acknowledge the supports from Westlake Education Foundation. C. K.
acknowledge the help from Yudi Yang during the preparation of the manuscript.
The computational resource is provided by Westlake HPC Center.

## References

  * Shaik  _et al._ [2016] S. Shaik, D. Mandal, and R. Ramanan, Oriented electric fields as future smart reagents in chemistry, Nat. Chem. 8, 1091 (2016).
  * Shaik  _et al._ [2018] S. Shaik, R. Ramanan, D. Danovich, and D. Mandal, Structure and reactivity/selectivity control by oriented-external electric fields, Chem. Soc. Rev. 47, 5125 (2018).
  * Che  _et al._ [2018] F. Che, J. T. Gray, S. Ha, N. Kruse, S. L. Scott, and J.-S. McEwen, Elucidating the roles of electric fields in catalysis A perspective, ACS Catal. 8, 5153 (2018).
  * Ciampi  _et al._ [2018] S. Ciampi, N. Darwish, H. M. Aitken, I. Díez-Pérez, and M. L. Coote, Harnessing electrostatic catalysis in single molecule, electrochemical and chemical systems: a rapidly growing experimental tool box, Chem. Soc. Rev. 47, 5146 (2018).
  * Shaik  _et al._ [2020] S. Shaik, D. Danovich, J. Joy, Z. Wang, and T. Stuyver, Electric-field mediated chemistry: Uncovering and exploiting the potential of (oriented) electric fields to exert chemical catalysis and reaction control, J. Am. Chem. Soc. 142, 12551 (2020).
  * Stuyver  _et al._ [2020] T. Stuyver, D. Danovich, J. Joy, and S. Shaik, External electric field effects on chemical structure and reactivity, WIREs Comput. Mol. Sci. 10, e1438 (2020).
  * Aragonès  _et al._ [2016] A. C. Aragonès, N. L. Haworth, N. Darwish, S. Ciampi, N. J. Bloomfield, G. G. Wallace, I. Diez-Perez, and M. L. Coote, Electrostatic catalysis of a diels–alder reaction, Nature 531, 88 (2016).
  * Oshima  _et al._ [2013] K. Oshima, T. Shinagawa, M. Haraguchi, and Y. Sekine, Low temperature hydrogen production by catalytic steam reforming of methane in an electric field, Int. J. Hydrogen Energ. 38, 3003 (2013).
  * Oshima  _et al._ [2014] K. Oshima, T. Shinagawa, Y. Nogami, R. Manabe, S. Ogo, and Y. Sekine, Low temperature catalytic reverse water gas shift reaction assisted by an electric field, Catal. Today 232, 27 (2014).
  * Sekine  _et al._ [2009] Y. Sekine, M. Tomioka, M. Matsukata, and E. Kikuchi, Catalytic degradation of ethanol in an electric field, Catal. Today 146, 183 (2009).
  * Gray  _et al._ [2020] J. T. Gray, K. Agarwal, J.-H. Cho, J.-I. Yang, and S. Ha, Estimating surface electric fields using reactive formic acid probes and SEM image brightness analysis, Chem. Eng. J. 402, 125640 (2020).
  * Che  _et al._ [2017a] F. Che, J. T. Gray, S. Ha, and J.-S. McEwen, Improving ni catalysts using electric fields: A DFT and experimental study of the methane steam reforming reaction, ACS Catal. 7, 551 (2017a).
  * Che  _et al._ [2017b] F. Che, J. T. Gray, S. Ha, and J.-S. McEwen, Reducing reaction temperature, steam requirements, and coke formation during methane steam reforming using electric fields: A microkinetic modeling and experimental study, ACS Catal. 7, 6957 (2017b).
  * Shih  _et al._ [2018] C. F. Shih, T. Zhang, J. Li, and C. Bai, Powering the future with liquid sunshine, Joule 2, 1925 (2018).
  * Trimm and Önsan [2001] D. L. Trimm and Z. I. Önsan, Onboard fuel conversion for hydrogen-fuel-cell-driven vehicles, Catal. Rev. 43, 31 (2001).
  * Sá  _et al._ [2010] S. Sá, H. Silva, L. Brandão, J. M. Sousa, and A. Mendes, Catalysts for methanol steam reforming—a review, Applied Catalysis B: Environmental 99, 43 (2010).
  * Wang  _et al._ [2018] Z. Wang, D. Danovich, R. Ramanan, and S. Shaik, Oriented-external electric fields create absolute enantioselectivity in diels–alder reactions Importance of the molecular dipole moment, J. Am. Chem. Soc. 140, 13350 (2018).
  * Olah [2005] G. A. Olah, Beyond oil and gas: The methanol economy, Angew. Chem. Int. Ed. 44, 2636 (2005).
  * Ke  _et al._ [2021] C. Ke, W. He, S. Liu, X. Ru, S. Liu, and Z. Lin, Multiscale catalyst design for steam methane reforming assisted by deep learning, J. Phys. Chem. C 125, 10860 (2021).
  * Han  _et al._ [2021] Z.-K. Han, D. Sarker, R. Ouyang, A. Mazheika, Y. Gao, and S. V. Levchenko, Single-atom alloy catalysts designed by first-principles calculations and artificial intelligence, Nat. Commun. 12, 1833 (2021).
  * Fu  _et al._ [2020] Z. Fu, B. Yang, and R. Wu, Understanding the activity of single-atom catalysis from frontier orbitals, Phys. Rev. Lett. 125, 156001 (2020).
  * Chen  _et al._ [2016] L. D. Chen, M. Urushihara, K. Chan, and J. K. Nørskov, Electric field effects in electrochemical co2 reduction, ACS Catal. 6, 7133 (2016).
  * Michaelides  _et al._ [2003] A. Michaelides, Z.-P. Liu, C. J. Zhang, A. Alavi, D. A. King, and P. Hu, Identification of general linear relationships between activation energies and enthalpy changes for dissociation reactions at surfaces, J. Am. Chem. Soc. 125, 3704 (2003).
  * Abild-Pedersen  _et al._ [2007] F. Abild-Pedersen, J. Greeley, F. Studt, J. Rossmeisl, T. R. Munter, P. G. Moses, E. Skúlason, T. Bligaard, and J. K. Nørskov, Scaling properties of adsorption energies for hydrogen-containing molecules on transition-metal surfaces, Phys. Rev. Lett. 99, 016105 (2007).
  * Kresse and J [1996a] G. Kresse and F. J, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Phys. Rev. B 54, 11169 (1996a).
  * Kresse and J [1996b] G. Kresse and F. J, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Comput. Mater. Sci. 6, 15 (1996b).
  * Blochl [1994] P. E. Blochl, Projector augmented-wave method, Phys. Rev. B 50, 17953 (1994).
  * Perdew and Wang [1992] J. P. Perdew and Y. Wang, Accurate and simple analytic representation of the electron-gas correlation energy, Phys. Rev. B 45, 13244 (1992).
  * Neugebauer and Scheffler [1992] J. Neugebauer and M. Scheffler, Adsorbate-substrate and adsorbate-adsorbate interactions of Na and K adlayers on Al(111), Phys. Rev. B 46, 16067 (1992).
  * Henkelman  _et al._ [2000] G. Henkelman, B. P. Uberuaga, and H. Jónsson, A climbing image nudged elastic band method for finding saddle points and minimum energy paths, J. Chem. Phys. 113, 9901 (2000).
  * Henkelman and Jónsson [1999] G. Henkelman and H. Jónsson, A dimer method for finding saddle points on high dimensional potential surfaces using only first derivatives, J. Chem. Phys. 111, 7010 (1999).
  * Yeh  _et al._ [2019] C.-H. Yeh, T. M. L. Pham, S. Nachimuthu, and J.-C. Jiang, Effect of external electric field on methane conversion on iro2 (110) surface: A density functional theory study, ACS Catal. 9, 8230 (2019).
  * Pacchioni  _et al._ [1997] G. Pacchioni, J. R. Lomas, and F. Illas, Electric field effects in heterogeneous catalysis, J. Mol. Catal. A 119, 263 (1997).
  * Che  _et al._ [2016] F. Che, S. Ha, and J.-S. McEwen, Elucidating the field influence on the energetics of the methane steam reforming reaction: A density functional theory study, Appl. Catal. B 195, 77 (2016).
  * Campbell and Mao [2021] C. T. Campbell and Z. Mao, Analysis and prediction of reaction kinetics using the degree of rate control, J. Catal. 404, 647 (2021).
  * Valdés-Solís  _et al._ [2006] T. Valdés-Solís, G. Marbán, and A. Fuertes, Nanosized catalysts for the production of hydrogen by methanol steam reforming, Catal. Today 116, 354 (2006).
  * Twigg and Spencer [2003] M. V. Twigg and M. S. Spencer, Deactivation of copper metal catalysts for methanol decomposition, methanol steam reforming and methanol synthesis, Top. Catal. 22, 191 (2003).
  * Ke and Lin [2020] C. Ke and Z. Lin, Density functional theory based micro- and macro-kinetic studies of ni-catalyzed methanol steam reforming, Catalysts 10, 349 (2020).


# Efficient Transformer-based Hyper-parameter Optimization for Resource-
constrained IoT Environments

Ibrahim Shaer Soodeh Nikan, and Abdallah Shami Ibrahim Shaer, Soodeh Nikan and
Abdallah Shami are with the Department of Electrical and Computer Engineering,
University of Western Ontario, London, ON N6A 3K7, Canada (emails:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>0000-0002-2723-4662
0000-0003-2887-0350

###### Abstract

The hyper-parameter optimization (HPO) process is imperative for finding the
best-performing Convolutional Neural Networks (CNNs). The automation process
of HPO is characterized by its sizable computational footprint and its lack of
transparency; both important factors in a resource-constrained Internet of
Things (IoT) environment. In this paper, we address these problems by
proposing a novel approach that combines transformer architecture and actor-
critic Reinforcement Learning (RL) model, TRL-HPO, equipped with multi-headed
attention that enables parallelization and progressive generation of layers.
These assumptions are founded empirically by evaluating TRL-HPO on the MNIST
dataset and comparing it with state-of-the-art approaches that build CNN
models from scratch. The results show that TRL-HPO outperforms the
classification results of these approaches by 6.8% within the same time frame,
demonstrating the efficiency of TRL-HPO for the HPO process. The analysis of
the results identifies the main culprit for performance degradation attributed
to stacking fully connected layers. This paper identifies new avenues for
improving RL-based HPO processes in resource-constrained environments.

###### Index Terms:

Hyper-parameter Optimization, Reinforcement Learning, Transformers, Image
Classification, MNIST dataset, Resource constraints, IoT Environment

##  I Introduction

Convolutional Neural Networks (CNNs) are the staple implementation of neural
networks (NNs) for image classification and object detection. The progress in
this field is attributed to the increased complexity of CNN architectures in
terms of the type of connections among different layers and their depth, which
increased the computational demands. As a result, the accuracy of these models
depends on the number and the type of layers, the connections between these
layers, and the parameters assigned to each layer [1]. In addition to
accuracy, other important factors that dictate the design of CNNs include
training time, inference time, and energy consumption, which are emphasized in
the resource-constrained Internet of Things (IoT) environment. The number of
possible choices for designing CNNs is extremely large, which is prohibitive
for manual search. These circumstances have promoted the automated CNN
architecture search, achieved through the field of Neural Architecture Search
(NAS).

The NAS field has recently seen great progress, due to the incorporation of
Reinforcement Learning (RL) agents to search for the best CNN configurations.
The RL’s appeal stems from its generalizability of hyper-parameter (HP)
combinations via function approximation and the trial-and-error approach,
which has the potential to reduce the computational demand of NAS [2]. This
field is dominated by the literature that either builds CNNs from scratch
using basic NN layers or a pre-defined collection of these layers, or
optimizes the HPs of an already-existing CNN.

Autonomous vehicles (AV) are part of the envisioned IoT applications that
utilize edge servers of limited computational resources. The realization of
AVs heavily depends on image classification models deployed on these servers.
As such, the disparate computational capabilities of edge servers are
prohibitive for the prolonged execution of large models. Model partitioning is
key to resolving these limitations toward fulfilling the promise of AVs [3].
This requires gathering insights into the contribution of each CNN model’s
layers to the classification results. In current implementations, this vision
is stalled by multiple oversights. The sequential nature of model generation
results from the dependence between layer combinations. This dependence can be
reflected by the recurrent structures of Recurrent Neural Networks (RNNs).
Therefore, RNN-based controllers constructing CNN models dominated the RL-
based NAS field [4, 5]. The one-at-a-time processing of inputs inhibits their
parallelization, resulting in their insurmountable computational footprint
that is ill-fitted for the resource-scarce vehicular environment. On a
different note, these controllers are built to receive a reward when a
terminal condition is encountered, masking the contribution of each layer to
prediction results. Within the constraints of computational resources and
model partitioning requirements, the state-of-the-art (SOTA) approaches are
limited by their impracticability and lack of transparency, hindering their
deployment in real-world environments.

This paper presents Transformer-based Reinforcement Learning Hyper-parameter
Optimization (TRL-HPO) that alleviates the shortcomings of the current
methods. The TRL-HPO addresses the transparency and long-computational times
of RL-based solutions with competitive performances. The Transformer
architecture overshadows the RNN-based methods by integrating the multi-headed
self-attention mechanism [6]. The TRL-HPO controller equipped with the
attention mechanism allows us to gain insights from the layer-generation
process, enhancing its interpretability. Moreover, the transformers’
architecture supports parallelization, which addresses the long computation
times of its RNN-based counterparts [6]. The reward is produced with every
generated layer to showcase improvement instead of waiting for termination
conditions to unfold, a condition matching CNN partitioning requirements, as
depicted in Figure 1. The contributions of this paper are as follows:

  * •

Propose a novel Hyper-parameter Optimization (HPO) process named TRL-HPO that
is the first to combine transformer architecture and actor-critic RL;

  * •

Enhance the CNN’s model performance generated within a shorter period compared
to SOTA approaches using the TRL-HPO process;

  * •

Improve the transparency of the CNN model generation process by examining the
attention-reward affinities and their layer combinations;

  * •

Create open challenges related to RL-based HPO processes, including TRL-HPO,
for researchers to address.

The rest of the paper is organized as follows. Section II presents the related
work and its limitations. Section III details the proposed framework. Section
IV explains the implementation details and evaluation criteria. Section V
analyzes the obtained results. Section VI concludes the paper.

##  II Related Work

The field of hyper-parameter optimization (HPO) is an important research topic
in Machine Learning (ML) with practical implementations in real-world
environments that enhance the performance of Deep Neural Networks (DNNs).

Bayesian Optimization (BO), Evolutionary Search (ES) algorithms, and RL agents
are the three main tools for HPO implementation. BO and ES methods are limited
by their assumptions [1] and lack of generality [7], which favors the
application of RL techniques. The works of Baker et al. [5] and Zoph et al.
[4] are the first to propose incorporating RL methods into generating CNN
models. The former work utilized RNN-based controllers using an off-policy RL
algorithm to sample CNN architectures while the latter utilized a value-based
Q-learning approach. The results of these seminal works highlighted the trade-
off between the running time of HPO methods and the accuracy of the obtained
models.

The computational footprint of RL implementations for HPO promoted the
utilization of multi-agent RL that shrinks the state space of each agent. The
work of Neary [8] uses multiple agents to optimize HPs of the CNNs built from
scratch, whereby a master agent orchestrates the synchronization between the
other agents outputting HPs. On the other hand, the work in [9] assigns each
agent to optimize the HPs of an already-existing CNN layer, such that the
dependence in HP space is mapped using a shared Q-table between consecutive
layers. While all of these studies focus on a single objective, the works
presented by Hsu et al. [10] and Tan et al. [11] incorporate multiple
objectives in the reward function formulation. MONAS [10] considers the energy
and accuracy constraints, whereas MNASNet [11] integrates the inference
latency of developed models.

The shortcomings exhibited by the integration of RL into the HPO process in
these implementations hinder their deployment in IoT environments. First, the
adopted models lack the transparency that shows the interdependence between
different layers. Second, some of these approaches take considerable time to
converge, which is a privilege that is not always available in resource-
constrained environments. Third, some works focus on optimizing the HPs of a
specific layer, reducing the state space at the expense of the layer’s
diversity. Lastly, many of the works include prior knowledge instilled into
the layer-generation process, such as the addition of dropout layers [4, 5].
These limitations are addressed in this paper by proposing a transformer-based
RL controller and the reward function formulation. The multi-headed attention
mechanism facilitates the training process and adds transparency to the model
generation process by analyzing the attention values. On the other hand, the
reward function reflects the contribution of each layer to enhancing the
classification results, a condition favorable to CNN model partitioning.

[scale=0.45]architecture_v3.PNG

Figure 1: TRL-HPO framework

##  III Methodology: TRL-HPO

This section explains the building blocks of TRL-HPO, including the
transformer and RL’s actor-critic architectures, which are shown in Figure 1,
and the motivation for this combination toward a more efficient and
transparent HPO process.

###  III-A Transformer

The vanilla transformer is a sequence-to-sequence model, which avoids the
recurrence structures of RNNs by integrating the innovative self-attention
mechanism [6]. This architecture enables parallelization accelerating the
training of transformer models. The building blocks of transformers include
attention mechanism, multi-headed attention, position-wise feed-forward
network, and positional encoding (PE). The transformers follow the encoder-
decoder structure thereby each of these structures is formed by stacking
combinations of these identical layers. The ordering of these layers with
respect to the encoder and decoder is depicted in Figure 1.

The Positional Encoding computes the sequence’s order and is added to the
input embedding of the encoder and decoder stacks so that the order is
incorporated into the input and output data. To obtain a unique order, a
sinusoidal function takes as inputs the position and the embedding dimensions.
This structure extrapolates to sequences that are longer than the ones
encountered during training [6]. The self-attention module calculates the
weights assigned by each input sequence to other sequences. This way the
recurrence structure is avoided because self-attention reflects the
dependencies between input and specific output and improves the modelling of
long-range dependencies [6]. As such, the attention mechanism resembles a
fully connected layer (FCL), such that the weights reflect the pairwise
relationship from previous inputs. To fully exploit this method, the
transformer includes multi-headed attention that replicates the attention
calculation. This process generates different representations of the input
data at different positions. The calculation of these attention matrices is
independent and facilitates their parallelization. Lastly, a feed-forward
neural network is used as the subsequent layers to the multi-head attention
mechanism.

###  III-B Actor-Critic Reinforcement Learning

RL teaches an agent to perform a task by accumulating experiences from
interacting with its environment. Its actions are refined based on a reward
function that signals the utility of these actions [12]. In connection to the
HPO process, the RL’s experimentation with the absence of any prior knowledge
mirrors the inability to hypothesize the best combination of basic CNN layers.
The only knowledge for the HPO process relates to the order of the stacked
layer that starts with grid-like inputs and ends with a fully connected layer
(FCL). As such, the combination of unknown orientation and trial-and-error
experimentation matches the requirements of the HPO process.

The RL methods can be split into value-based and policy-based methods. The
value-based method estimates the quality of a state-action pair using a value
function and optimizes this value iteratively using actions that maximize it.
The high variability of value estimations and their under-performance in
scenarios with continuous action spaces promotes policy-based methods [12].
The policy-based methods focus on optimizing the policy to maximize the
expected cumulative rewards [12]. However, the latter methods suffer from
sample inefficiency, which is an asset of value-based methods. To complement
these methods’ advantages, an actor-critic approach combines both policy-based
(actor) and value-based (critic) methods [12]. Projected to the HPO problem,
the actor outputs an action that maps to a layer and its HPs. The critic
evaluates this action by outputting a value showing the quality of the taken
action [12].

###  III-C Transformers and Actor-critic RL

As the name suggests, the proposed framework, Transformer-based Reinforcement
Learning HPO (TRL-HPO) is the convergence of transformers and actor-critic RL.
The actor and the critic are implemented using the transformer architecture.
The steps involved in the actions’ generation, CNN model construction, and
their evaluation are depicted in Figure 1. The combination of TRL-HPO enables
the RL agent to harness the strengths of transformers to the benefit of the
HPO process. This new framework is the first trial to integrate transformers
into the HPO process, opening a new frontier toward the exploitation of this
novel architecture to address a lingering problem in the field of ML. This
experiment does not plan to only exhibit the viability of transformers’
implementation to HPO, but it rather analyzes the suitability of this
architecture to the HPO process.

We analyze TRL-HPO based on its inherent advantages and its benefits versus
the SOTA approaches. In terms of the framework, two main advantages are
reaped. The first is that TRL-HPO builds models from scratch, rendering it a
general-purpose method. The proof of concept is constrained to a single use
case; however, the framework can be expanded to any problem that utilizes DNN
models. The second concerns the definition of the reward function that is
progressively updated with each stacked layer. While this definition incurs an
extra computational footprint by training progressively deeper models, it
facilitates understanding the contribution of each layer. This feature fits
the requirements of environments that seek to balance resource constraints and
accuracy objectives. The transformer architecture results in two inherent
advantages to the HPO process. The first is connected to the parallelization
enabled by the multi-headed attention mechanism, which reduces the running
time of the HPO process. The second benefit stems from the attention mechanism
that showcases the relationships between generated layers. A by-product of
this feature is the inference of the combination of layers that produce better
results, which enhances the transparency of the HPO process.

##  IV Implementation Details

The evaluation of TRL-HPO is conducted on the MNIST dataset [13]. This dataset
is a large database of handwritten digits, from 0 to 9, containing 60,000
training images and 10,000 testing images. Each image is a greyscale image of
28 $\times$ 28 dimensions. This dataset serves as a good proof-of-concept for
the proposed framework, due to its availability and limited complexity. The
former factor enables a fair comparison with other SOTA approaches, such as
[5, 8]. The latter factor facilitates training CNN models in a limited time;
an advantage when working with hardware constraints. The comparison with SOTA
approaches is based on the convergence time to the best solution (hrs), the
classification accuracy (%), and the best CNN classification model (%)
obtained by each SOTA approach upon the completion of TRL-HPO process
(AccTime). We compare TRL-HPO with Baker et al. [5] and Neary [8], which are
the only RL-based solutions that build CNN models from scratch.

Layer | HP | Values
---|---|---
CNN | filters | {8, 16, …, 128}
kernel | {3, 5, 7}
stride | {1, 2, 3}
FCL | neurons | {16, 24, …, 512}
bias | {T, F}
activations |  | {None, relu, leakyrelu,
---
tanh, sigmoid, elu, gelu}
MaxPool | kernel | {2, 3, …,, 8}
stride | {1, 2, 3}
padding | {0, 1, 2, 3}
TABLE I: Set of hyper-parameters

The generated CNN models are constrained to 6 layers, formed as a combination
of the three basic layers: CNN layer, FCL layer, and Maximum Pooling (MaxPool)
layer. The set of HPs used in the experimental procedure is summarized in
table I. Stacking a combination of these layers adapts to the input structure.
In connection to the RL agent, the design of the state and action space and
the reward function is imperative to drive the RL agents’ action generation,
which requires defining the transformers’ input sequence.

The reward function represents a layer’s contribution towards improving the
classification results, which means that the DNN model’s first layer produces
the highest rewards compared to any subsequent layers. Each layer is
represented as a combination of the representation of the layer itself and its
HP, and the performance of the obtained model on the MNIST’s validation set.
The generated layers’ representation is obtained from the output layer of the
RL’s actor, summarized in four values. These values represent the action space
that is mapped first to a layer and then to that layer’s HPs. The model’s
performance is represented with 32 values, each calculating the model’s
accuracy results on each validation set’s batch of size 16. The heterogeneity
in the dimensions of the action space and the performance requires a mapping
to a uniform representation. This goal is achieved using a static NN that
takes these inputs and produces a uniform output of 64 values, referred to as
Intermediate Model Representation (IMR). When a layer and its performance are
obtained, the state space, reflecting the current state of the environment,
should also change. Since the state space represents a sequence of layers, the
index corresponding to the generated layer is updated with the IMR. With more
layers, each index is updated with a new representation. This way the state
space reflects the two important pieces of information for each layer, its
HPs, and its performance. This process was touched upon in Figure 1.

The stopping criteria for the RL agent involve either generating more than 6
layers, minimal improvement in the performance with the addition of layers
(0.001), or accuracy below 60%. These criteria are defined to avoid any
unnecessary generation of models. A Deep Deterministic Policy Gradient (DDPG)
[14] handles the continuous action spaces in this environment. The critic and
the actor consist of target and online subnets to avoid radical changes in
critic and actor updates [14]. The training of RL agents involves exploration
and exploitation of the gained knowledge. In DDPG, the agent explores by
adding random noise to the sampled actions, so that the agent can experiment
with different combinations of layers and obtain their performance results. To
remove data correlations for the RL agent inherent in the sequential structure
of the data, experience replay (ER) [14] buffer is utilized to store data
during the exploration stage.

The training process is realized over several episodes. In each episode, 10
full models (i.e., until a stopping criterion is met) are generated via the
utilization of CPU parallelization. Once the ER buffer is full, five RL
optimization rounds are implemented in each episode. The number of episodes
and the size of the ER buffer are important parameters for the HPO process.
Their optimization is imperative to obtain models with good performances in a
short period. During the HPO process, the training set consists of 20,000
images while the validation set includes 10,000 images out of the original
60,000 images. The experimental procedure is conducted on SHARCNET’s Graham
cluster on a node with one V100 GPU, 64 Gbs of RAM, and 12 cores, such that
each experiment does not exceed 7 days to avoid long queuing times. The
implementation will be available on a GitHub repository upon publication.

##  V Results and Discussion

This section reports the results of the experimental procedure and analyzes
them based on the defined performance metrics and layer-generation
interpretability. TRL-HPO results are obtained in the exploitation phase of
the offline RL.

###  V-A TRL-HPO vs. SOTA

Methods |  | Best Accuracy
---
(%)
| Running Time
---
(hrs)
| AccTime
---
(%)
TRL-HPO | 99.1 | 99.3 | 94.6 % $\pm$ 3.8%
MetaQNN [5] | 99.5 | 192-240 | 88.5 % $\pm$ 4.3%
Neary [8] | 95.8 | 1.78 | 95.8%
TABLE II: TRL-HPO versus SOTA in terms of accuracy and running time

Table II shows the results of TRL-HPO versus SOTA methods that built CNN
models from scratch. Based on evaluation results, MetaQNN produces the best
model when classifying MNIST data reported in the exploration phase. This
result is attributed to the freedom endowed to the controller to generate
models up to 12 layers and the integration of priors via the addition of
dropout layers every 2 layers [5]. The accumulation of more layers improves
classification results; however, at the expense of wider search space that
increases the running time. This is evident with the computational footprint
of MetaQNN that exceeds TRL-HPO, despite running on 8$\times$ the amount of
GPU resources. On the other hand, the approach in [8] is the first to converge
based on the reported results, given it is a Multi-agent RL algorithm.
However, this method experimented with four configurations, highlighting the
limited exploration of this strategy and the pre-mature convergence of the
master agent. This limitation is reflected by its poor best models compared to
the other two approaches, excluding it from further analysis. Since the larger
search space of MetaQNN would not provide a fair comparison with TRL-HPO, it
was imperative to compare these two approaches using AccTime. Within TRL-HPO
convergence time, its generated models exhibit the best performance compared
to the SOTA models. Despite TRL-HPO’s need to generate more models given the
requirement of progressive reward, it outperforms other models in the AccTime
metric. Compared to other SOTA methods, TRL-HPO can progressively generate CNN
models with satisfactory performance in a shorter period. These two conditions
are important for resource-constrained IoT environments. In terms of AccTime,
on average, TRL-HPO outperforms MetaQNN by 6.8 %.

###  V-B Layer Analysis

Investigating the effect of the addition of layers is central to understanding
the contribution of each layer towards improving accuracy results and the
combination of layers that yields the best performances. This analysis unveils
the complex relationships between layers facilitating the transparency of the
HPO process. Both goals were considered in the design of TRL-HPO, representing
a key differentiating factor versus SOTA methods. To gain these insights, two
questions need to be answered. (1) What are the layer combinations degrading
the performance of CNN models? and (2) How are layer affinities reflected in
the attention mechanism?

[scale=0.4]neg_reward_prev_layer.png

Figure 2: Layer affinity for models with negative rewards

Figure 2 summarizes the distribution of layer combinations with negative
rewards. Two combinations stand out, Class 1 represented by Conv2D and Conv2D
combinations and Class 2 represented by two consecutive FCLs. A layer
combination refers to the layer that produced a negative reward and its
previous layer, shedding light on the combinations that degrade model
performance. Two main insights can be gathered from Figure 2. The first is
that the negative reward is overwhelmingly attributed to the accumulation of
FCLs. This means that the stacking of multiple of these layers is superfluous
on the MNIST dataset, a conclusion that aligns with the best models obtained
in [5]. In a similar vein, the second insight highlights that additional
Conv2D layers can be detrimental to model performance, especially with MNIST
data of limited visual complexity. As such, these observations are beneficial
to garner knowledge about superfluous stacking of layers, suitable for
environments with resource constraints.

[scale=0.4]diff_attn.png

Figure 3: Differences in attention between layers of best reward and layers
with negative reward

Figure 3 depicts the distribution of the differences in attention values
between layers with the best performances and layers with negative rewards.
The distribution demonstrates the close fidelity of the attention mechanism
and the utility of a specific layer in its classification performance. As
such, the attention mechanism captures the close affinity between layers
assigning higher weights to promising layers while shunning layers with poor
performances. The difference in attention values averages 0.22, a significant
value considering that some attention is assigned to the layers that are yet
to be generated. In these cases, the attention values of the layer degrading
model performances are higher than these to-be-generated layers. Therefore, we
conclude that the attention mechanism reflects the relationship between layers
within the context of generating best-performing models.

##  VI Open Challenges and Future Recommedations

The implementation of the TRL-HPO process opens new avenues for exploring the
improvement of this process, which spans many research and practical questions
about RL-based HPO implementations.

###  VI-A Computational Time

Despite improvements in the efficiency of the TRL-HPO process, it has yet to
converge in a short time. The bottleneck in the training process originates
from two main sources: (1) the training of individual models, and (2)
optimization rounds of the actor-critic approach. The first source is a
necessary evil because of the exploration requirements and the need to
evaluate the actor’s actions. Determining the training data size, the number
of epochs, and learning rates is required so that close to optimal accuracy is
achieved. During the exploration and action assessment phases, repetitive
models can be generated. Storing these models is a method to avoid retraining
them, which can overwhelm the available GPU or RAM resources, especially with
a large ER buffer. An alternative method is storing already trained models in
a database or a hash table, whereby the model is represented via its HP or its
hash value. A salient issue relates to querying algorithms that should search
for HP representations available in the database. The querying time will
increase with the expansion of the database, requiring a more intricate search
procedure. These methods should be investigated to budget the computational
time towards more fruitful procedures that benefit RL-based solutions.

###  VI-B Exploration

RL implementations depend on the trial-and-error procedure taking place in the
explorations phase. However, a shortcoming of this phase, either through
DDPG’s random noise or $\epsilon$-greedy algorithms [5], relates to its lack
of direction. This means that the agent continues exploring unpromising areas
of the search space with relatively poor performances. Therefore, to reduce
this unnecessary exploration, it is imperative to include priors or rectify
the exploration stage. As such, aspects of the BO process should be integrated
into the RL-based strategies to streamline the exploration process. The
randomness can be incorporated using a stochastic policy, which is part of the
soft actor-critic policy gradient approach [15]. In the exploration stage, the
stochastic policy outputs the mean and standard deviation of a normal
distribution. The standard deviation is progressively reduced to achieve a
deterministic policy based on the reward function. As such, this suggestion
opens new frontiers toward examining methods that can associate the
uncertainty of the stochastic policy with steering the RL’s agent exploration.

###  VI-C Reward Function

The progressive reward function devised in TRL-HPO formulation is foundational
to gaining insights into the contributions of each layer to performance
enhancement. However, this layer accumulation process exposes two limitations.
The first limitation pertains to the training of many models, even with a
single layer, to infer this performance. This pre-requisite restricts
expanding the model generation to a few layers, which applies to our
experimental procedure. These constraints can be sidestepped via block
generation at the expense of sacrificing transparency and layer diversity. A
block is defined as a set of similar basic layers. This small modification is
reflected by altering the actor’s output to produce the number of these
similar blocks. The second limitation of this process relates to the training
procedure. The HPO procedure is conducted on a subset of the training set to
facilitate the training and validation procedures. This means that the
performance of the generated CNN models can significantly vary, depending on
the train and test splits. In light of the stopping criteria, this factor is
crucial in deciding whether to halt or continue the layer generation process.

The reward function that drives the model generation process can be adjusted
to achieve better results. Instead of performance differences, the reward can
be the inverse distance to a perfect metric (100% accuracy in our example). In
this way, the rewards are less sparse but require the tuning of a discount
rate. Additionally, other metrics, such as f1-score and top-3 accuracy can
replace accumulated accuracy. The adoption of any of these metrics depends on
the practitioners’ priorities.

###  VI-D Transferability

With each new contribution toward the HPO procedure, questions of
generalizability and transferability loom to undermine their usefulness, which
also applies to RL-based solutions. In TRL-HPO, the actor and critic are
constructed to generate and evaluate models using basic layers of FCL, CNN,
and MaxPool layers, necessary to build DNN models. The transferability comes
into question when TRL-HPO trained on one dataset needs to be applied to other
datasets. In that regard, the main concern relates to the size of the input
sequence, trained on the initial dataset, which constrains the size of the
produced models. This raises the question of how to expand the input sequence
to involve more layers?.

The answer to this question requires re-defining the concept of the input
sequence (state) supplied to the actor and critic. In its current form, each
index in the state sequence combines the layer definition and its performance.
To enable transferability and the expansion of TRL-HPO to more profound use
cases, the actor should output an additional value representing the number of
similar layers. The remaining alteration is to determine the upper bound of
the permitted number of these layers so that the model’s capacity would match
the dataset’s complexity.

##  VII Conclusion

The HPO process is a fundamental step in the ML pipeline that contributes to
enhancing model performance. However, its computational footprint is
prohibitive for widespread adoption and current methods overlook the
transparency in the layer generation process. These factors are imperative to
realize IoT applications’ function, AVs in particular, requiring model
partitioning to fulfill the resource constraints of edge environments. To
address these limitations, this paper proposes TRL-HPO framework that combines
transformers with an RL actor-critic approach. The attention mechanism,
parallelization, and the progressive generation of layers are all novel
properties of this framework within the transparency and time requirements.
These advantageous factors were empirically established compared to SOTA
approaches using the MNIST dataset. Moreover, a list of research questions
scrutinizing RL-based solutions, including TRL-HPO, is presented along with a
set of recommendations. Future work will target the presented questions to
design a general-purpose HPO process.

## References

  * [1] T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” _The Journal of Machine Learning Research_ , vol. 20, no. 1, pp. 1997–2017, 2019.
  * [2] D. Baymurzina, E. Golikov, and M. Burtsev, “A review of neural architecture search,” _Neurocomputing_ , vol. 474, pp. 82–93, 2022.
  * [3] X. Xu, S. Tang, L. Qi, X. Zhou, F. Dai, and W. Dou, “Cnn partitioning and offloading for vehicular edge networks in web3,” _IEEE Communications Magazine_ , 2023.
  * [4] B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in _International Conference on Learning Representations_ , 2016.
  * [5] B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” in _International Conference on Learning Representations_ , 2016.
  * [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [7] I. Shaer and A. Shami, “Corrfl: correlation-based neural network architecture for unavailability concerns in a heterogeneous iot environment,” _IEEE Transactions on Network and Service Management_ , 2023.
  * [8] P. Neary, “Automatic hyperparameter tuning in deep convolutional neural networks using asynchronous reinforcement learning,” in _2018 IEEE International Conference on Cognitive Computing (ICCC)_. IEEE Computer Society, 2018, pp. 73–77.
  * [9] A. Iranfar, M. Zapater, and D. Atienza, “Multiagent reinforcement learning for hyperparameter optimization of convolutional neural networks,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 41, no. 4, pp. 1034–1047, 2021.
  * [10] C.-H. Hsu, S.-H. Chang, J.-H. Liang, H.-P. Chou, C.-H. Liu, S.-C. Chang, J.-Y. Pan, Y.-T. Chen, W. Wei, and D.-C. Juan, “Monas: Multi-objective neural architecture search using reinforcement learning,” _arXiv preprint arXiv:1806.10332_ , 2018.
  * [11] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, “Mnasnet: Platform-aware neural architecture search for mobile,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 2820–2828.
  * [12] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [13] Y. LeCun, “The mnist database of handwritten digits,” _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * [14] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [15] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.