††thanks: Denotes equal contribution

# Chemical reactions in imperfect cavities: enhancement, suppression, and
resonance

John P. Philbin¶<EMAIL_ADDRESS>Harvard John A. Paulson School of
Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA
Yu Wang¶<EMAIL_ADDRESS>Department of Chemistry, School of Science,
Westlake University, Hangzhou 310024 Zhejiang, China  Institute of Natural
Sciences, Westlake Institute for Advanced Study, Hangzhou 310024 Zhejiang,
China  Prineha Narang<EMAIL_ADDRESS>Harvard John A. Paulson School
of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138,
USA  Wenjie Dou<EMAIL_ADDRESS>Department of Chemistry, School of
Science, Westlake University, Hangzhou 310024 Zhejiang, China  Department of
Physics, School of Science, Westlake University, Hangzhou 310024 Zhejiang,
China  Institute of Natural Sciences, Westlake Institute for Advanced Study,
Hangzhou 310024 Zhejiang, China

###### Abstract

The use of optical cavities to control chemical reactions has been of great
interest recently, following demonstrations of enhancement, suppression, and
negligible effects on chemical reaction rates depending on the specific
reaction and cavity frequency. In this work, we study the reaction rate inside
imperfect cavities, where we introduce a broadening parameter in the spectral
density to mimic Fabry-Pérot cavities. We investigate cavity modifications to
reaction rates using non-Markovian Langevin dynamics with frictional and
random forces to account for the presence of imperfect optical cavities. We
demonstrate that in the regime of weak solvent and cavity friction, the cavity
can enhance chemical reaction rates. On the other hand, in the high friction
regime, cavities can suppress chemical reactions. Furthermore, we find that
the broadening of the cavity spectral density gives rise to blue shifts of the
resonance conditions and, surprisingly, increases the sharpness of the
resonance effect.

††preprint: APS/123-QED

##  I Introduction

The concept of selectively controlling the rate of a chemical reaction by
shining light in resonance with a specific molecular bond has been explored
for many decades. Two drawbacks of this scheme are that it requires light
energy as an input and, arguably more importantly, molecular bonds often
quickly transfer the vibrational energy in the mode of interest to other
vibrations and degrees of freedom. A possible way to circumvent these two
obstacles is to tune the frequencies of an optical cavity to selectively alter
the ground state chemical reactivity of molecules, which was demonstrated
recently experimentally.Thomas  _et al._ (2016, 2019); Nagarajan  _et al._
(2021) This research avenue has attracted many experimental,Lather  _et al._
(2019); Wiesehan and Xiong (2021); Imperatore  _et al._ (2021); Mandal  _et
al._ (2022) computational,Ruggenthaler  _et al._ (2014); Flick  _et al._
(2017); Flick and Narang (2018); Fregoni  _et al._ (2018); Flick and Narang
(2020); Haugland  _et al._ (2020); Deprince (2021); Schäfer  _et al._ (2021);
Pavosevic  _et al._ (2022) and theoreticalCampos-Gonzalez-Angulo  _et al._
(2019); Li  _et al._ (2020, 2021); Yang and Cao (2021) investigations in
recent years as which reactions can be manipulated by optical cavities and the
underlying mechanism of how cavities can modify chemical reactivity are still
an open questions.Kéna-Cohen and Yuen-Zhou (2019); Hertzog  _et al._ (2019);
Simpkins  _et al._ (2021)

In this work, we study the reaction rate inside an optical cavity by
developing a non-Markovian dynamical model. The effect of the cavity mode on
the reaction mode dynamics is incorporated into friction and random forces.Dou
_et al._ (2017); Dou and Subotnik (2018) This model allows us to continuously
model Fabry-Pérot cavities from their ideal perfect single mode cavity limit
to more realistic lossy, imperfect cavities.Wang  _et al._ (2021) In Kramer’s
theory dealing with Ohmic friction, the rate of escape of a particle over a
potential barrier increases linearly with increasing friction in the
underdamped limit and decreases inversely with the friction strength in the
strong damping limit.Kramers (1940) Grote and Hynes proceeded to solve the
Kramer’s rate problem in the presence of memory friction, in the regime of
moderate to strong friction.Grote and Hynes (1980) The continuum limit version
of Kramer’s theory that cover the whole range of friction is known as Pollak,
Grabert, and Hänggi (PGH) theory.Pollak  _et al._ (1989) To cover the whole
range of friction, our numerical and analytical analysis show that optical
cavities can both enhance or suppress chemical reaction rates, depending on
the magnitude of the solvent friction as well as the cavity-molecule coupling
strength. Interestingly, increasing the cavity-molecule coupling strength
gives rise to a blue shift of the cavity frequency that has the greatest
impact on the reaction rate (i.e. blue shift of the resonance condition)
whereas making the cavity more imperfect by increasing the broadening
parameter of the cavity spectral density results in a blue shift of the
resonance condition.

The manuscript is organized as follows. In Sec. II, we introduce our model and
derive the Langevin dynamics for the molecules in the presence of a Markovian
phonon bath and non-Markovian cavity mode. In Sec. III, we perform both
numerical and analytical analysis to obtain cavity-modified chemical reaction
rates within our model. Lastly, we conclude in Sec. IV.

Figure 1: Schematic representation of light-matter vibrational strong coupling
in an imperfect cavity and the spectra density of the cavity.

##  II Theory

Here, we consider the reaction mode to be a one-dimension double well
potential energy surface (PES). The reaction mode couples to a set of harmonic
oscillators corresponding to the solvent environment as well as a set of
harmonic oscillators corresponding to the optical cavity. The total
Hamiltonian that we utilize is given by the Pauli-Fierz nonrelativistic
quantum electrodynamics (QED) Hamiltonian in the dipole gauge and in the long-
wavelength limit:Rokaj  _et al._ (2018); Schäfer  _et al._ (2020)

$\displaystyle H=$
$\displaystyle~{}\frac{p^{2}}{2m}+V(x)+\frac{1}{2}\sum_{j}m_{j}[\dot{q}_{j}^{2}+\omega_{j}^{2}(q_{j}+c_{j}x)^{2}]$
$\displaystyle+\frac{1}{2}\sum_{k}[P_{k}^{2}+\tilde{\omega}_{k}^{2}(Q_{j}+\mu_{k}x)^{2}].$
(1)

Here, $x$ and $p$ are the coordinates for the reaction mode, $q_{j}$ and
$\dot{q}_{j}$ are the coordinates for the other vibrational modes that will
compose the Markovian bath, $Q_{k}$ and $P_{k}$ are the coordinates for the
cavity modes that will compose the non-Markovian bath, $c_{j}$ and $\mu_{k}$
are coupling strengths of the reaction coordinates to the vibrational bath
modes and cavity modes. The double well PES for the reaction mode is given by
Dou  _et al._ (2015):

$V(x)=\frac{1}{2}m\omega_{0}^{2}x^{2}-a\cdot\log(1+e^{-(\sqrt{2}gx+E_{d})/a}),$
(2)

where $\omega_{0}=0.003$, $a=0.02$, $g=0.02$, $E_{d}=\frac{40}{3}$ are the
parameters used in our calculations below and we set $\hbar=1$ throughout this
work. In the semi-classical limit, we can write down the equations of motion
for the reaction mode as:

$\displaystyle m\ddot{x}=$
$\displaystyle-\frac{dV}{dx}-\sum_{j}m_{j}\omega_{j}^{2}(c_{j}q_{j}+c_{j}^{2}x)$
(3) $\displaystyle-\sum_{k}\tilde{\omega}_{k}^{2}(\mu_{k}Q_{k}+\mu_{k}^{2}x).$

Similarly, we can also write down the equation of motions for the vibrational
bath modes and cavity modes

$\displaystyle m_{j}\ddot{q}_{j}$
$\displaystyle=-m_{j}\omega_{j}^{2}q_{j}-m_{j}\omega_{j}^{2}c_{j}x$ (4)
$\displaystyle\ddot{Q}_{k}$
$\displaystyle=-\tilde{\omega}_{k}^{2}Q_{k}-\tilde{\omega}_{k}^{2}\mu_{k}x.$
(5)

Because the above equations are linear, they can be solved explicitly: Nitzan
(2006)

$\displaystyle q_{j}(t)=$
$\displaystyle~{}q_{j0}\cos(\omega_{j}t)+\frac{\dot{q}_{j0}}{\omega_{j}}\sin(\omega_{j}t)-c_{j}x(t)$
(6)
$\displaystyle+c_{j}\int_{0}^{t}d\tau\cos(\omega_{j}(t-\tau))\dot{x}(\tau),$

where $q_{j0}$ and $\dot{q}_{j0}$ are the initial position and momenta for the
solvent environment. We can solve for $P_{k},Q_{k}$ in a similar manner,

$\displaystyle Q_{k}(t)=$
$\displaystyle~{}Q_{k0}\cos(\tilde{\omega}_{k}t)+\frac{\dot{Q}_{k0}}{\tilde{\omega}_{k}}\sin(\tilde{\omega}_{k}t)-\mu_{k}x(t)$
(7)
$\displaystyle+\mu_{k}\int_{0}^{t}d\tau\cos(\tilde{\omega}_{k}(t-\tau))\dot{x}(\tau).$

If we plug in Eq. (6) and Eq. (7) into the equation of motion for $x$ given by
Eq. (3), we find a generalized Langevin equation:

$\displaystyle m\ddot{x}=$
$\displaystyle-\frac{dV}{dx}-\int_{0}^{t}d\tau(Z_{p}(t-\tau)+Z_{c}(t-\tau))\dot{x}(\tau)$
$\displaystyle+R_{p}(t)+R_{c}(t)$ (8)

where

$\displaystyle Z_{p}(t)$
$\displaystyle=\sum_{j}m_{j}\omega_{j}^{2}c_{j}^{2}\cos(\omega_{j}t)$ (9)
$\displaystyle Z_{c}(t)$
$\displaystyle=\sum_{k}\tilde{\omega}_{k}^{2}\mu_{k}^{2}\cos(\tilde{\omega}_{k}t)$
(10) $\displaystyle R_{p}(t)$
$\displaystyle=-\sum_{j}m_{j}\omega_{j}^{2}c_{j}(q_{j0}\cos(\omega_{j}t)+\frac{\dot{q}_{j0}}{\omega_{j}}\sin(\omega_{j}t))$
(11) $\displaystyle R_{c}(t)$
$\displaystyle=-\sum_{k}\tilde{\omega}_{k}^{2}\mu_{k}(Q_{k0}\cos(\tilde{\omega}_{k}t)+\frac{\dot{Q}_{k0}}{\tilde{\omega}_{k}}\sin(\tilde{\omega}_{k}t)).$
(12)

Here $R_{p}(t)$ and $R_{c}(t)$ are random forces from the phonon bath and
cavity respectively, and $Z_{c}(t)$ and $Z_{p}(t)$ are the corresponding
friction kernels. Below we consider the case that friction and random forces
from the phonon environments are Markovian, such that the Langevin equation
can be simplified as

$\displaystyle m\ddot{x}=$ $\displaystyle-\frac{dV}{dx}-\int_{0}^{t}d\tau
Z_{c}(t-\tau)\dot{x}(\tau)+R_{c}(t)$ (13)
$\displaystyle+\gamma_{p}\dot{x}(t)+R_{p}(t)$

We now turn our attention to the frictional force from the cavity. We consider
the spectral density with broadening parameters in an imperfect cavity.Wang
_et al._ (2021) In particular, the spectral density is taken to the form of a
Cauchy distribution to mimic a Fabry-Pérot cavity:

$\displaystyle
Z_{c}(\omega)=2\gamma_{c}\omega_{c}\Gamma\frac{\omega^{3}}{(\omega^{2}-\omega_{c}^{2})^{2}+(\omega\Gamma)^{2}}.$
(14)

In the above equation (Eq. 14), $\gamma_{c}$ quantifies the coupling strength
of the reaction coordinate to the cavity, $\omega_{c}$ corresponding to the
the center frequency where the cavity density of states is largest (and is the
frequency of the one cavity mode in the perfect cavity scenario), and $\Gamma$
is the broadening parameter that determines the extent to which the cavity is
imperfect (i.e. lossy). As $\Gamma\rightarrow 0$, $Z_{c}(\omega)$ approaches a
$\delta$-function, reflecting a perfect cavity.

The memory kernel is then given by

$\displaystyle
Z_{c}(t)=\frac{2}{\pi}\int_{0}^{\infty}\frac{Z_{c}(\omega)}{\omega}\cos{(\omega
t)}d\omega$ (15)

The solution cavity memory kernel in the time domain has two solutions,
depending on the cavity center frequency and broadening. When
$\omega_{c}>\Gamma/2$, the cavity memory kernel is

$\displaystyle Z_{c}(t)=2\gamma_{c}\omega_{c}e^{-\frac{\Gamma
t}{2}}\left(\frac{\cos(\omega_{1}t)}{2}-\frac{\Gamma\sin(\omega_{1}t)}{4\omega_{1}}\right)$
(16)

where we defined $\omega_{1}=\sqrt{\omega_{c}^{2}-\Gamma^{2}/4}$. And when
$\omega_{c}<\Gamma/2$, the cavity memory kernel is

$\displaystyle Z_{c}(t)=2\gamma_{c}\omega_{c}e^{-\frac{\Gamma
t}{2}}\left(\frac{\cosh(\omega_{1}t)}{2}-\frac{\Gamma\sinh(\omega_{1}t)}{4\omega_{1}}\right).$
(17)

with $\omega_{1}=\sqrt{\Gamma^{2}/4-\omega_{c}^{2}}$. As shown in Fig. 2, the
cavity memory kernel in the time domain converges toward a $\delta$-function
as $\Gamma$ increases. Whereas, when $\Gamma$ approaches to 0, the memory
kernel will oscillate as a cosine function. The cavity random force $R_{c}(t)$
is related to the memory kernel $Z_{c}(t)$ via the fluctuation–dissipation
theorem:

$\displaystyle\langle R_{c}(0)R_{c}(t)\rangle=mk_{B}TZ_{c}(t).$ (18)

Note that the time correlation function of the random force is non-Markovian.
To generate such non-Markovian random force, we propagate the following
equations of motion:

$\displaystyle\dot{Y}=\sqrt{\frac{\omega_{c}}{\Gamma}}\omega_{c}R_{c}$ (19)
$\displaystyle\dot{R}_{c}=\sqrt{\frac{\omega_{c}}{\Gamma}}(-\omega_{c}Y-\Gamma
R_{c}+\Gamma\xi(t)).$ (20)

Here, $\xi(t)$ is a Markovian random variable from a Gaussian distribution
with a standard deviation of $\sigma=\sqrt{2m\gamma_{c}k_{B}T/dt}$, where $dt$
is the time step interval. $Y$ is an auxiliary variable. We can show that, in
the long time limit, the above equations generate $R_{c}$ that satisfies the
correlation function in Eq. (18).

As for the solvent (i.e. vibrational) bath, the random force $R_{p}(t)$ is
Markovian ($\langle R_{p}(0)R_{p}(t)\rangle=2mk_{B}T\gamma_{p}\delta(t)$),
which is set to be a Gaussian random variable with a standard deviation of
$\sigma_{p}=\sqrt{2m\gamma_{p}k_{B}T/dt)}$. We use $4^{\text{th}}$-order
Runge-Kutta to integrate Eqs. (13), (19) and (20). Lastly, unless stated
otherwise, we perform thermal averages over $10,000$ trajectories in our non-
Markovian Langevin dynamics simulations.

Figure 2: The memory kernel $Z_{c}(t)$ as a function of time with a small
$\Gamma$ (black line) and a large $\Gamma$ (red line). The other model
parameters used here are: $\gamma_{c}=0.01$, $\omega_{c}=0.001$.

##  III Results and Discussion

We first investigate cavity modified reaction rate in the weak solvent
friction regime, namely when $\gamma_{p}$ is small. We calculate the reaction
rate by initializing all trajectories in the reactant well with a Boltzmann
distribution and monitoring the number of trajectories that end up in the
product well as a function of time by numerically solving the non-Markovian
Langevin dynamics, as illustrated in Fig. 3. We find that, with a fixed photon
frequency $\omega_{c}=0.005$, a turnover occurs in the chemical reaction rate
as a function of the cavity friction $\gamma_{c}$. Specifically, we find that
initially increasing the cavity friction results increases the reaction rate
but, eventually, increasing the cavity friction leads to a decrease in the
reaction rate. This turnover is the well-known Kramer’s turnover which occurs
in the overdamped regime.Hänggi  _et al._ (1990) Therefore, with small solvent
friction, the Kramer turnover effect can be observed here as a function of the
cavity friction.

Figure 3: Product populations as a function of time with different cavity
friction $\gamma_{c}$, calculated by non-Markovian Langevin dynamics. The
other model parameters used here are: $\omega_{0}=0.003$, $\omega_{b}=0.007$,
barrier height $E_{b}=0.01947$, $\gamma_{p}=0.00001$, $\omega_{c}=0.005$,
$k_{\text{B}}T=0.005$, $\Gamma=0.01$.

We then proceed to investigate the reaction rate as a function of the cavity
frequency, where the rate is extracted by exponentially fitting the product
population as a function of time. In the limit of weak solvent friction, we
see that the cavity can enhance reaction rates ($\omega_{c}\approx\omega_{0}$)
(Fig. 4). The maximum cavity-modified reaction rate enhancement occurs at a
cavity frequency around the vibrational frequency of the reactant well
($\omega_{0}=0.003$, Fig. 4). Upon increasing the cavity friction $\gamma_{c}$
from $0.001$ to $0.005$, the cavity enhances the reaction even more as the
total friction remains in the underdamped regime. Further increasing the
cavity friction will eventually results in suppressing chemical reactions as
shown in Fig. 3 (overdamped regime).

Figure 4: Cavity-modified reaction rate as a function of the cavity photon
frequency $\omega_{c}$ at two different cavity friction values,
$\gamma_{c}=0.001,0.005$, calculated by non-Markovian Langevin dynamics. The
other model parameters used here are: $\omega_{0}=0.003$, $\omega_{b}=0.007$,
barrier height $E_{b}=0.01947$, $\gamma_{p}=0.00001$, $k_{\text{B}}T=0.005$,
$\Gamma=0.01$.

We now turn our focus to the strong solvent friction regime, namely when
$\gamma_{p}$ is large. In this regime, the reaction rates are monotonically
suppressed upon increasing the cavity friction, which is in agreement with
Grote-Hynes (GH) theory.Grote and Hynes (1980); Li  _et al._ (2021); Mandal
_et al._ (2022) In this limit, the reaction rate is given by

$\displaystyle k=\kappa_{\text{GH}}k_{\text{TST}}$ (21)

Here $k_{\text{TST}}$ is the transition state theory rate.
$\kappa_{\text{GH}}$ is the Grote-Hynes transmission coefficient. We can
employ an analytical approach for calculating $\kappa_{\text{GH}}$ to
understand how cavities can modify chemical reactions in this regime as a
function of the cavity frequency, $\omega_{c}$. Particularly, the GH
coefficient is given by

$\displaystyle\lambda^{2}=\omega_{b}^{2}-\lambda\cdot(\tilde{\xi}_{c}(\lambda)+\tilde{\xi}_{p}(\lambda)),$
(22)

where $\tilde{\xi}_{c}(\lambda)$ and $\tilde{\xi}_{p}(\lambda)$ are the
Laplace transformed cavity photon friction and solvent vibrational bath
friction, respectively. Using the spectral density $Z_{c}(\omega)$, one can
write $\tilde{\xi}_{c}(\lambda)$ as:

$\displaystyle\tilde{\xi}_{c}(\lambda)=\frac{2}{\pi}\int_{0}^{\infty}\frac{Z_{c}(\omega)}{\omega}\frac{\lambda}{\lambda^{2}+\omega^{2}}d\omega.$
(23)

For solvent bath friction, we have $\tilde{\xi}_{p}=\gamma_{p}$. GH
coefficient is then obtained from $\kappa_{\text{GH}}=\lambda/\omega_{b}$.

In Fig. 5, we plot $\kappa_{\text{GH}}$ as a function of $\omega_{c}$
analytically. The two green lines show reaction rates for two perfect cavities
with different cavity frictions ($\gamma_{c}$) in comparison to two red lines
which show reaction rates for imperfect cavities (i.e. cavities with broadened
spectral densities) with all other parameters held constant. In agreement with
recent reports,Li  _et al._ (2021); Mandal  _et al._ (2022) when the cavity
friction ($\gamma_{c}$) is small, the minimum of $\kappa_{\text{GH}}$ appears
at a cavity frequency close to the barrier frequency, $\omega_{b}=0.007$. Upon
increasing $\gamma_{c}$, the cavity frequency corresponding to the minimum of
$\kappa_{\text{GH}}$ is shifted to lower energies (i.e. the resonance
condition is red shifted). A noteworthy finding reported here for the first
time is that effect of imperfect cavities on the resonance conditions. Fig. 5
shows that the cavity frequency with greatest impact on the reaction rates is
shifted to higher energies (i.e. blue shifted). This is shown clearly by the
maximum rate suppression occurring at higher cavity frequencies for the
imperfect cavities (red lines) compared to the corresponding perfect cavities
(green lines). Additionally, the resonance condition surprisingly sharpens in
the case of a broaden cavity, especially on the low frequency side of the
maximum rate suppression.

Figure 5: Cavity-modified transmission coefficient $\kappa_{\text{GH}}$ for
the reaction rate as a function of the cavity photon frequency $\omega_{c}$ at
different cavity frictions $\gamma_{c}=0.001,0.005$. The other model
parameters used here are: $\omega_{0}=0.003$, $\omega_{b}=0.007$, barrier
height $E_{b}=0.01947$, $\gamma_{p}=0.01$, $k_{\text{B}}T=0.005$, for the
green lines, $\Gamma=0.0001$, for the red lines, $\Gamma=0.01$.

To understand the red and blue shifts, we examine the GH theory again. To
simplify the analysis here, we eliminate the friction from phonon bath. In the
limit of $\Gamma=0$ (perfect cavity), the GH equation becomes

$\displaystyle\lambda^{2}=\omega_{b}^{2}-2\gamma_{c}\omega_{c}\frac{\lambda^{2}}{\lambda^{2}+\omega_{c}^{2}}.$
(24)

We can solve for $\omega_{c}$ where we reach minimum $\kappa_{\text{GH}}$ by
finding the solution $\partial\lambda/\partial\omega_{c}=0$ (see the
Appendix):

$\displaystyle\omega_{c}=-\frac{\gamma_{c}}{2}+\sqrt{\frac{\gamma_{c}^{2}}{4}+\omega_{b}^{2}}\approx\omega_{b}-\frac{\gamma_{c}}{2}$
(25)

Here, we have expanded the above result to the first order in $\gamma_{c}$.
Clearly, the coupling strength $\gamma_{c}$ will introduce a red shift for the
resonance condition $\omega_{c}=\omega_{b}$.

For the imperfect cavity with non-zero $\Gamma$, we can introduce a factor $Q$
to approximate the following integral

$\displaystyle\frac{2}{\pi}\int_{0}^{\infty}\frac{\Gamma\omega^{2}}{(\omega^{2}-\omega_{c}^{2})^{2}+(\omega\Gamma)^{2}}\frac{\lambda}{\lambda^{2}+\omega^{2}}d\omega=\frac{\lambda}{\lambda^{2}+(Q\omega_{c})^{2}}.$
(26)

In the limit $\Gamma=0$, we have $Q=1$. Such that we recover the results for
the perfect cavity. The non-zero $\Gamma$ will lead to $Q<1$ for an imperfect
cavity. The minimum $\kappa_{\text{GH}}$ is then located at

$\displaystyle\omega_{c}=\frac{1}{Q^{2}}\left(-\frac{\gamma_{c}}{2}+\sqrt{\frac{\gamma_{c}^{2}}{4}+\omega_{b}^{2}}\right)$
(27)

Since $Q<1$, the broadening will give rise to blue shift.

In Fig. 6, we further verify the effects of broadening on the transmission
coefficient $\kappa_{\text{GH}}$ as a function of the cavity frequency
$\omega_{c}$ using numerical and analytic methods. Specifically, the green
lines and red lines utilize the analytic expressions for a perfect and
imperfect cavity, respectively and the black lines are the numerical results
obtained from the full non-Markovian Langevin dynamics with the same $\Gamma$
as the red lines. Fig. 6a shows that when $\Gamma$ is small, there is no
broadening effect in the spectral density of the cavity mode such that these
three methods predict the same $\kappa_{\text{GH}}$ for all values of
$\omega_{c}$. In particular, $\kappa_{\text{GH}}$ exhibits a minimum when the
cavity frequency is close to the barrier frequency ($\omega_{b}=0.007$). On
the other hand, for the relatively larger $\Gamma$ value shown in Fig. 6b, the
broadening effect of the cavity spectral density leads to a blue shift in the
frequency at which the cavity has the largest impact on the reaction rate, and
the width of the resonance narrows. This observation is in agreement with our
analytical analysis above. Furthermore, the broadening effects tend to reduce
the efficiency of the cavity mode on the reaction rate. With a very large
$\Gamma$, we expect that the cavity will not affect the chemical reaction at
all.

Figure 6: Cavity-modified transmission coefficient $\kappa_{\text{GH}}$ for
the reaction rate as a function of photon frequency $\omega_{c}$ for (a) a
perfect cavity ($\Gamma=0.0001$) and (b) an imperfect cavity ($\Gamma=0.01$).
The other model parameters used here are: $\omega_{0}=0.003$,
$\omega_{b}=0.007$, barrier height $E_{b}=0.01947$, $k_{\text{B}}T=0.005$,
$\gamma_{p}=0.001$, $\gamma_{c}=0.001$.

##  IV Conclusions

We developed a non-Markovian friction and random force model to study how
imperfect optical cavities can modify chemical reaction rates. We found that
in the small solvent friction regime, cavities can enhance chemical reaction
rates while in the large solvent friction regime cavities suppress reactions.
We also reported that imperfect cavities have resonance conditions that are
both shifted to higher frequencies and sharper relative to otherwise identical
perfect cavities.

After completing this work, we noticed that two similar results were reported
in the literature.Sun and Vendrell (2022); Lindoy  _et al._ (2022) Our work is
in agreement with these findings and differentiates itself from those works as
we investigated the effect of imperfect cavities, both numerically and
analytically, on chemical reaction rates while the other reports focused on
perfect cavities.

###### Acknowledgements.

Y.W and W.D. acknowledge support from Westlake University. J.P.P. acknowledges
support from the Harvard University Center for the Environment. P.N.
acknowledges support as a Moore Inventor Fellow through Grant No. GBMF8048 and
gratefully acknowledges support from the Gordon and Betty Moore Foundation as
well as support from a NSF CAREER Award under Grant No. NSF-ECCS-1944085.

*

## Appendix A

As stated in the main text, to simplify the analysis in the large friction
regime, we eliminate the friction from phonon bath. In the limit of $\Gamma=0$
(perfect cavity), we have

$\displaystyle\lambda^{2}=\omega_{b}^{2}-2\gamma_{c}\omega_{c}\frac{\lambda^{2}}{\lambda^{2}+\omega_{c}^{2}}.$
(28)

When $\frac{\partial\lambda}{\partial\omega_{c}}=0$ we have a relation between
$\lambda_{min}$ and $\omega_{c}$ as

$\displaystyle\lambda_{min}^{2}=\frac{\omega_{b}^{2}}{1+\frac{\gamma_{c}}{\omega_{c}}}$
(29)

Substituting Eq. 29 into Eq. 28 we could get a relationship as follows

$\displaystyle\lambda_{min}=\omega_{c}$ (30)

Substituting Eq. 30 into Eq. 29 obtains a function relating to the
$\omega_{c}$ that corresponding to the $\lambda_{min}$ (i.e. the minimum of
$\kappa_{\text{GH}}$)

$\displaystyle\omega_{c}^{2}+\gamma_{c}\omega_{c}-\omega_{b}^{2}=0$ (31)

When solving the above equation we get

$\displaystyle\omega_{c}=-\frac{\gamma_{c}}{2}+\sqrt{\frac{\gamma_{c}^{2}}{4}+\omega_{b}^{2}}$
(32)

It is evidently that when increasing the $\gamma_{c}$, the $\omega_{c}$
corresponding to the minimum $\kappa_{\text{GH}}$ will red shift.

For the imperfect cavity, when introducing a $Q$ factor, the GH equation
becomes

$\displaystyle\lambda^{2}=\omega_{b}^{2}-2\gamma_{c}\omega_{c}\frac{\lambda^{2}}{\lambda^{2}+(Q\omega_{c})^{2}}.$
(33)

When $\frac{\partial\lambda}{\partial\omega_{c}}=0$ we have a relation between
$\lambda_{min}$ and $\omega_{c}$ as

$\displaystyle\lambda_{min}^{2}=\frac{\omega_{b}^{2}}{1+\frac{\gamma_{c}}{Q^{2}\omega_{c}}}$
(34)

Substituting Eq. 34 into Eq. 33 we could obtain

$\displaystyle\lambda_{min}=Q\omega_{c}$ (35)

Substituting Eq. 35 into Eq. 34 gets a function relating to the $\omega_{c}$
that corresponding to the $\lambda_{min}$ in the imperfect cavity

$\displaystyle Q^{2}\omega_{c}^{2}+{\gamma_{c}}\omega_{c}-\omega_{b}^{2}=0$
(36)

When solving the above equation we get

$\displaystyle\omega_{c}=\frac{1}{Q^{2}}\left(-\frac{\gamma_{c}}{2}+\sqrt{\frac{\gamma_{c}^{2}}{4}+\omega_{b}^{2}}\right)$
(37)

Note that $Q<1$. The resonance $\omega_{c}$ will blue shift in this imperfect
cavity.

## References

  * Thomas  _et al._ (2016) A. Thomas, J. George, A. Shalabney, M. Dryzhakov, S. J. Varma, J. Moran, T. Chervy, X. Zhong, E. Devaux, C. Genet, and Others, Ang. Chem. Int. Ed. 128, 11634 (2016).
  * Thomas  _et al._ (2019) A. Thomas, L. Lethuillier-Karl, K. Nagarajan, R. M. A. Vergauwe, J. George, T. Chervy, A. Shalabney, E. Devaux, C. Genet, J. Moran, and T. W. Ebbesen, Science 363, 615 (2019).
  * Nagarajan  _et al._ (2021) K. Nagarajan, A. Thomas, and T. W. Ebbesen, J. Am. Chem. Soc. 143, 16877 (2021).
  * Lather  _et al._ (2019) J. Lather, P. Bhatt, A. Thomas, T. W. Ebbesen, and J. George, Ang. Chem. Int. Ed. 58, 10635 (2019).
  * Wiesehan and Xiong (2021) G. D. Wiesehan and W. Xiong, J. Chem. Phys. 155, 241103 (2021).
  * Imperatore  _et al._ (2021) M. V. Imperatore, J. B. Asbury, and N. C. Giebink, J. Chem. Phys. 154, 191103 (2021).
  * Mandal  _et al._ (2022) A. Mandal, X. Li, and P. Huo, J. Chem. Phys. 156, 014101 (2022).
  * Ruggenthaler  _et al._ (2014) M. Ruggenthaler, J. Flick, C. Pellegrini, H. Appel, I. V. Tokatly, and A. Rubio, Phys. Rev. A 90, 012508 (2014).
  * Flick  _et al._ (2017) J. Flick, M. Ruggenthaler, H. Appel, and A. Rubio, Proc. Natl. Acad. Sci. U. S. A. 114, 3026 (2017).
  * Flick and Narang (2018) J. Flick and P. Narang, Phys. Rev. Lett. 121, 113002 (2018).
  * Fregoni  _et al._ (2018) J. Fregoni, G. Granucci, E. Coccia, M. Persico, and S. Corni, Nat. Commun. 9, 4688 (2018).
  * Flick and Narang (2020) J. Flick and P. Narang, J. Chem. Phys. 153, 094116 (2020).
  * Haugland  _et al._ (2020) T. S. Haugland, E. Ronca, E. F. Kjønstad, A. Rubio, and H. Koch, Phys. Rev. X 10, 041043 (2020).
  * Deprince (2021) A. E. Deprince, J. Chem. Phys. 154, 094112 (2021).
  * Schäfer  _et al._ (2021) C. Schäfer, J. Flick, E. Ronca, P. Narang, and A. Rubio, arXiv , 2104.12429 (2021).
  * Pavosevic  _et al._ (2022) F. Pavosevic, S. Hammes-Schiffer, A. Rubio, and J. Flick, J. Am. Chem. Soc. 144, 4995 (2022).
  * Campos-Gonzalez-Angulo  _et al._ (2019) J. A. Campos-Gonzalez-Angulo, R. F. Ribeiro, and J. Yuen-Zhou, Nat. Commun. 10, 4685 (2019).
  * Li  _et al._ (2020) T. E. Li, A. Nitzan, and J. E. Subotnik, J. Chem. Phys. 152, 234107 (2020).
  * Li  _et al._ (2021) X. Li, A. Mandal, and P. Huo, Nat. Commun. 12, 1315 (2021).
  * Yang and Cao (2021) P. Y. Yang and J. Cao, J. Phys. Chem. Lett. 12, 9531 (2021).
  * Kéna-Cohen and Yuen-Zhou (2019) S. Kéna-Cohen and J. Yuen-Zhou, ACS Cent. Sci. 5, 386 (2019).
  * Hertzog  _et al._ (2019) M. Hertzog, M. Wang, J. Mony, and K. Börjesson, Chem. Soc. Rev. 48, 937 (2019).
  * Simpkins  _et al._ (2021) B. S. Simpkins, A. D. Dunkelberger, and J. C. Owrutsky, J. Phys. Chem. C 125, 19081 (2021).
  * Dou  _et al._ (2017) W. Dou, G. Miao, and J. E. Subotnik, Phys. Rev. Lett. 119, 046001 (2017).
  * Dou and Subotnik (2018) W. Dou and J. E. Subotnik, J. Chem. Phys. 148, 230901 (2018).
  * Wang  _et al._ (2021) D. S. Wang, T. Neuman, J. Flick, and P. Narang, J. Chem. Phys. 154, 104109 (2021).
  * Kramers (1940) H. A. Kramers, Physica 7, 284 (1940).
  * Grote and Hynes (1980) R. F. Grote and J. T. Hynes, J. Chem. Phys. 73, 2715 (1980).
  * Pollak  _et al._ (1989) E. Pollak, H. Grabert, and P. Hänggi, J. Chem. Phys. 91, 4073 (1989).
  * Rokaj  _et al._ (2018) V. Rokaj, D. M. Welakuh, M. Ruggenthaler, and A. Rubio, J. Phys. B Atom., Mol. Opt. Phys. 51, 34005 (2018).
  * Schäfer  _et al._ (2020) C. Schäfer, M. Ruggenthaler, V. Rokaj, and A. Rubio, ACS Photonics 7, 975 (2020).
  * Dou  _et al._ (2015) W. Dou, A. Nitzan, and J. E. Subotnik, J. Chem. Phys. 143, 054103 (2015).
  * Nitzan (2006) A. Nitzan,  _Chemical Dynamics in Condensed Phases: Relaxation, Transfer, and Reactions in Condensed Molecular Systems_ (Oxford University Press, 2006).
  * Hänggi  _et al._ (1990) P. Hänggi, P. Talkner, and M. Borkovec, Rev. Mod. Phys. 62, 251 (1990).
  * Sun and Vendrell (2022) J. Sun and O. Vendrell, J. Phys. Chem. Lett. 13, 4441 (2022).
  * Lindoy  _et al._ (2022) L. P. Lindoy, A. Mandal, and D. R. Reichman, arXiv , 2205.05142 (2022).


# Handwritten Digit Recognition using Machine and Deep Learning Algorithms

Ritik Dixit  Computer Science and Engineering
Acropolis Institute of Technology & Research
Indore, India
<EMAIL_ADDRESS>Rishika Kushwah  Computer Science and Engineering
Acropolis Institute of Technology & Research
Indore, India
<EMAIL_ADDRESS>Samay Pashine  Computer Science and Engineering
Acropolis Institute of Technology & Research
Indore, India
<EMAIL_ADDRESS>and

###### Abstract

The reliance of humans over machines has never been so high such that from
object classification in photographs to adding sound to silent movies
everything can be performed with the help of deep learning and machine
learning algorithms. Likewise, Handwritten text recognition is one of the
significant areas of research and development with a streaming number of
possibilities that could be attained. Handwriting recognition (HWR), also
known as Handwritten Text Recognition (HTR), is the ability of a computer to
receive and interpret intelligible handwritten input from sources such as
paper documents, photographs, touch-screens and other devices [1]. Apparently,
in this paper, we have performed handwritten digit recognition with the help
of MNIST datasets using Support Vector Machines (SVM), Multi-Layer Perceptron
(MLP) and Convolution Neural Network (CNN) models. Our main objective is to
compare the accuracy of the models stated above along with their execution
time to get the best possible model for digit recognition.

Keywords: Deep Learning, Machine Learning, Handwritten Digit Recognition,
MNIST datasets, Support Vector Machines (SVM), Multi-Layered Perceptron (MLP),
and Convolution Neural Network (CNN).

##  I INTRODUCTION

Handwritten digit recognition is the ability of a computer to recognize the
human handwritten digits from different sources like images, papers, touch
screens, etc, and classify them into 10 predefined classes (0-9). This has
been a topic of boundless-research in the field of deep learning. Digit
recognition has many applications like number plate recognition, postal mail
sorting, bank check processing, etc [2]. In Handwritten digit recognition, we
face many challenges because of different styles of writing of different
peoples as it is not an Optical character recognition. This research provides
a comprehensive comparison between different machine learning and deep
learning algorithms for the purpose of handwritten digit recognition. For
this, we have used Support Vector Machine, Multilayer Perceptron, and
Convolutional Neural Network. The comparison between these algorithms is
carried out on the basis of their accuracy, errors, and testing-training time
corroborated by plots and charts that have been constructed using matplotlib
for visualization.

The accuracy of any model is paramount as more accurate models make better
decisions. The models with low accuracy are not suitable for real-world
applications. Ex- For an automated bank cheque processing system where the
system recognizes the amount and date on the check, high accuracy is very
critical. If the system incorrectly recognizes a digit, it can lead to major
damage which is not desirable. That’s why an algorithm with high accuracy is
required in these real-world applications. Hence, we are providing a
comparison of different algorithms based on their accuracy so that the most
accurate algorithm with the least chances of errors can be employed in various
applications of handwritten digit recognition.

This paper provides a reasonable understanding of machine learning and deep
learning algorithms like SVM, CNN, and MLP for handwritten digit recognition.
It furthermore gives you the information about which algorithm is efficient in
performing the task of digit recognition. In further sections of this paper,
we will be discussing the related work that has been done in this field
followed by the methodology and implementation of all the three algorithms for
the fairer understanding of them. Next, it presents the conclusion and result
bolstered by the work we have done in this paper. Moreover, it will also give
you some potential future enhancements that can be done in this field. The
last section of this paper contains citations and references used.

##  II RELATED WORK

With the humanization of machines, there has been a substantial amount of
research and development work that has given a surge to deep learning and
machine learning along with artificial intelligence. With time, machines are
getting more and more sophisticated, from calculating the basic sums to doing
retina recognition they have made our lives more secure and manageable.
Likewise, handwritten text recognition is an important application of deep
learning and machine learning which is helpful in detecting forgeries and a
wide range of research has already been done that encompasses a comprehensive
study and implementation of various popular algorithms like works done by S M
Shamim [3], Anuj Dutt [4], Norhidayu binti [5] and Hongkai Wang [8] to compare
the different models of CNN with the fundamental machine learning algorithms
on different grounds like performance rate, execution time, complexity and so
on to assess each algorithm explicitly. [3] concluded that the Multilayer
Perceptron classifier gave the most accurate results with minimum error rate
followed by Support Vector Machine, Random Forest Algorithm, Bayes Net, Naïve
Bayes, j48, and Random Tree respectively while [4] presented a comparison
between SVM, CNN, KNN, RFC and were able to achieve the highest accuracy of
98.72% using CNN (which took maximum execution time) and lowest accuracy using
RFC. [5] did the detailed study-comparison on SVM, KNN and MLP models to
classify the handwritten text and concluded that KNN and SVM predict all the
classes of dataset correctly with 99.26% accuracy but the thing process goes
little complicated with MLP when it was having trouble classifying number 9,
for which the authors suggested to use CNN with Keras to improve the
classification. While [8] has focused on comparing deep learning methods with
machine learning methods and comparing their characteristics to know which is
better for classifying mediastinal lymph node metastasis of non-small cell
lung cancer from 18 F-FDG PET/CT images and also to compare the discriminative
power of the recently popular PET/CT texture features with the widely used
diagnostic features. It concluded that the performance of CNN is not
significantly different from the best classical methods and human doctors for
classifying mediastinal lymph node metastasis of NSCLC from PET/CT images.
However, CNN does not make use of the import diagnostic features, which have
been proved more discriminative than the texture features for classifying
small-sized lymph nodes. Therefore, incorporating the diagnostic features into
CNN is a promising direction for future research.

All we need is lots of data and information and we will be able to train a big
neural net to do what we want, so a convolution can be understood as ”looking
at functions surrounding to make a precise prognosis of its outcome.” [6], [7]
has used a convolution neural network for handwritten digit recognition using
MNIST datasets. [6] has used 7 layered CNN model with 5 hidden layers along
with gradient descent and back prorogation model to find and compare the
accuracy on different epochs, thereby getting maximum accuracy of 99.2% while
in [7], they have briefly discussed different components of CNN, its
advancement from LeNet-5 to SENet and comparisons between different model like
AlexNet, DenseNet and ResNet. The research outputs the LeNet-5 and LeNet-5
(with distortion) achieved test error rate of 0.95% and 0.8% respectively on
MNIST data set, the architecture and accuracy rate of AlexNet is same as
LeNet-5 but much bigger with around 4096000 parameters and ”Squeeze-and-
Excitation network” (SENet) have become the winner of ILSVRC-2017 since they
have reduced the top-5 error rate to 2.25% and by far the most sophisticated
model of CNN in existence.

##  III METHODOLOGY

The comparison of the algorithms (Support vector machines, Multi-layered
perceptron and Convolutional neural network) is based on the characteristic
chart of each algorithm on common grounds like dataset, the number of epochs,
complexity of the algorithm, accuracy of each algorithm, specification of the
device (Ubuntu 20.04 LTS, i5 7th gen processor) used to execute the program
and runtime of the algorithm, under ideal condition.

A. DATASET

Handwritten character recognition is an expansive research area that already
contains detailed ways of implementation which include major learning
datasets, popular algorithms, features scaling and feature extraction methods.
MNIST dataset (Modified National Institute of Standards and Technology
database) is the subset of the NIST dataset which is a combination of two of
NIST’s databases: Special Database 1 and Special Database 3. Special Database
1 and Special Database 3 consist of digits written by high school students and
employees of the United States Census Bureau, respectively. MNIST contains a
total of 70,000 handwritten digit images (60,000 - training set and 10,000 -
test set) in 28x28 pixel bounding box and anti-aliased. All these images have
corresponding Y values which apprises what the digit is.

Figure 1. Bar graph illustrating the MNIST handwritten digit training dataset
(Label vs Total number of training samples).

Figure 2. Plotting of some random MNIST Handwritten digits.

B. SUPPORT VECTOR MACHINE

Support Vector Machine (SVM) is a supervised machine learning algorithm. In
this, we generally plot data items in n-dimensional space where n is the
number of features, a particular coordinate represents the value of a feature,
we perform the classification by finding the hyperplane that distinguishes the
two classes. It will choose the hyperplane that separates the classes
correctly. SVM chooses the extreme vectors that help in creating the
hyperplane. These extreme cases are called support vectors, and hence the
algorithm is termed as Support Vector Machine. There are mainly two types of
SVMs, linear and non-linear SVM. In this paper, we have used Linear SVM for
handwritten digit recognition [10].

Figure 3. This image describes the working mechanism of SVM Classification
with supporting vectors and hyperplanes.

C. MULTILAYERED PERCEPTRON

A multilayer perceptron (MLP) is a class of feedforward artificial neural
networks (ANN). It consists of three layers: input layer, hidden layer and
output layer. Each layer consists of several nodes that are also formally
referred to as neurons and each node is interconnected to every other node of
the next layer. In basic MLP there are 3 layers but the number of hidden
layers can increase to any number as per the problem with no restriction on
the number of nodes. The number of nodes in the input and output layer depends
on the number of attributes and apparent classes in the dataset respectively.
The particular number of hidden layers or numbers of nodes in the hidden layer
is difficult to determine due to the model erratic nature and therefore
selected experimentally. Every hidden layer of the model can have different
activation functions for processing. For learning purposes, it uses a
supervised learning technique called backpropagation. In the MLP, the
connection of the nodes consists of a weight that gets adjusted to synchronize
with each connection in the training process of the model[11].

Figure 4. This figure illustrates the basic architecture of the Multilayer
perceptron with variable specification of the network.

D. CONVOLUTIONAL NEURAL NETWORK

CNN is a deep learning algorithm that is widely used for image recognition and
classification. It is a class of deep neural networks that require minimum
pre-processing. It inputs the image in the form of small chunks rather than
inputting a single pixel at a time, so the network can detect uncertain
patterns (edges) in the image more efficiently. CNN contains 3 layers namely,
an input layer, an output layer, and multiple hidden layers which include
Convolutional layers, Pooling layers(Max and Average pooling), Fully connected
layers (FC), and normalization layers [12]. CNN uses a filter (kernel) which
is an array of weights to extract features from the input image. CNN employs
different activation functions at each layer to add some non-linearity [13].
As we move into the CNN, we observe the height and width decrease while the
number of channels increases. Finally, the generated column matrix is used to
predict the output [14].

Figure 5. This figure shows the architectural design of CNN layers in the form
of a Flow chart.

E. VISUALIZATION

In this research, we have used the MNIST dataset (i.e. handwritten digit
dataset) to compare different level algorithm of deep and machine learning
(i.e. SVM, ANN-MLP, CNN) on the basis of execution time, complexity, accuracy
rate, number of epochs and number of hidden layers (in the case of deep
learning algorithms). To visualize the information obtained by the detailed
analysis of algorithms we have used bar graphs and tabular format charts using
module matplotlib, which gives us the most precise visuals of the step by step
advances of the algorithms in recognizing the digit. The graphs are given at
each vital part of the programs to give visuals of each part to bolster the
outcome.

##  IV IMPLEMENTATION

To compare the algorithms based on working accuracy, execution time,
complexity, and the number of epochs (in deep learning algorithms) we have
used three different classifiers:

  * •

Support Vector Machine Classifier

  * •

ANN - Multilayer Perceptron Classifier

  * •

Convolutional Neural Network Classifier

We have discussed in detail about the implementation of each algorithm
explicitly below to create a flow of this analysis to create a fluent and
accurate comparison.

I. PRE-PROCESSING

Pre-processing is an initial step in the machine and deep learning which
focuses on improving the input data by reducing unwanted impurities and
redundancy. To simplify and break down the input data we reshaped all the
images present in the dataset in 2-dimensional images i.e (28,28,1). Each
pixel value of the images lies between 0 to 255 so, we Normalized these pixel
values by converting the dataset into ’float32’ and then dividing by 255.0 so
that the input features will range between 0.0 to 1.0. Next, we performed one-
hot encoding to convert the y values into zeros and ones, making each number
categorical, for example, an output value 4 will be converted into an array of
zero and one i.e [0,0,0,0,1,0,0,0,0,0].

II. SUPPORT VECTOR MACHINE

The SVM in scikit-learn [16] supports both dense (numpy.ndarray and
convertible to that by numpy.asarray) and sparse (any scipy.sparse) sample
vectors as input. In scikit-learn, SVC, NuSVC and LinearSVC are classes
capable of performing multi-class classification on a dataset. In this paper
we have used LinearSVC for classification of MNIST datasets that make use of a
Linear kernel implemented with the help of LIBLINEAR [17].

Various scikit-learn libraries like NumPy, matplotlib, pandas, Sklearn and
seaborn have been used for the implementation purpose. Firstly, we will
download the MNIST datasets, followed by loading it and reading those CSV
files using pandas.

After this, plotting of some samples as well as converting into matrix
followed by normalization and scaling of features have been done. Finally, we
have created a linear SVM model and confusion matrix that is used to measure
the accuracy of the model [9].

III. MULTILAYERED PERCEPTRON

The implementation of Handwritten digits recognition by Multilayer perceptron
[18] which is also known as feedforward artificial neural network is done with
the help of Keras module to create an MLP model of Sequential class and add
respective hidden layers with different activation function to take an image
of 28x28 pixel size as input. After creating a sequential model, we added a
Dense layer of different specifications and Drop out layers as shown in the
image below. The block diagram is given here for reference. Once you have the
training and test data, you can follow these steps to train a neural network
in Keras.

Figure 6. Sequential Block Diagram of Multi-layers perceptron model built with
the help of Keras module [19].

We used a neural network with 4 hidden layers and an output layer with 10
units (i.e. total number of labels). The number of units in the hidden layers
is kept to be 512. The input to the network is the 784-dimensional array
converted from the 28×28 image. We used the Sequential model for building the
network. In the Sequential model, we can just stack up layers by adding the
desired layer one by one. We used the Dense layer, also called a fully
connected layer since we are building a feedforward network in which all the
neurons from one layer are connected to the neurons in the previous layer.
Apart from the Dense layer, we added the ReLU activation function which is
required to introduce non-linearity to the model. This will help the network
learn non-linear decision boundaries. The last layer is a softmax layer as it
is a multiclass classification problem [19].

IV. CONVOLUTIONAL NEURAL NETWORK

The implementation of handwritten digit recognition by Convolutional Neural
Network [15] is done using Keras. It is an open-source neural network library
that is used to design and implement deep learning models. From Keras, we have
used a Sequential class which allowed us to create model layer-by-layer. The
dimension of the input image is set to 28(Height), 28(Width), 1(Number of
channels). Next, we created the model whose first layer is a Conv layer [20].
This layer uses a matrix to convolve around the input data across its height
and width and extract features from it. This matrix is called a Filter or
Kernel. The values in the filter matrix are weights. We have used 32 filters
each of the dimensions (3,3) with a stride of 1. Stride determines the number
of pixels shifts. Convolution of filter over the input data gives us
activation maps whose dimension is given by the formula: ((N + 2P - F)/S) + 1
where N= dimension of input image, P= padding, F= filter dimension and
S=stride. In this layer, Depth (number of channels) of the output image is
equal to the number of filters used. To increase the non-linearity, we have
used an activation function that is Relu [21]. Next, another convolutional
layer is used in which we have applied 64 filters of the same dimensions (3,3)
with a stride of 1 and the Relu function.

Next, to these layers, the pooling layer [22] is used which reduces the
dimensionality of the image and computation in the network. We have employed
MAX-pooling which keeps only the maximum value from a pool. The depth of the
network remains unchanged in this layer. We have kept the pool-size (2,2) with
a stride of 2, so every 4 pixels will become a single pixel. To avoid
overfitting in the model, Dropout layer [23] is used which drops some neurons
which are chosen randomly so that the model can be simplified. We have set the
probability of a node getting dropped out to 0.25 or 25%. Following it,
Flatten Layer [23] is used which involves flattening i.e. generating a column
matrix (vector) from the 2-dimensional matrix. This column vector will be fed
into the fully connected layer [24]. This layer consists of 128 neurons with a
dropout probability of 0.5 or 50%. After applying the Relu activation
function, the output is fed into the last layer of the model that is the
output layer. This layer has 10 neurons that represent classes (numbers from 0
to 9) and the SoftMax function [25] is employed to perform the classification.
This function returns probability distribution over all the 10 classes. The
class with the maximum probability is the output.

Figure 7. Detailed architecture of Convolutional Neural Network with apt
specifications of each layer

##  V RESULT

After implementing all the three algorithms that are SVM, MLP and CNN we have
compared their accuracies and execution time with the help of experimental
graphs for perspicuous understanding. We have taken into account the Training
and Testing Accuracy of all the models stated above. After executing all the
models, we found that SVM has the highest accuracy on training data while on
testing dataset CNN accomplishes the utmost accuracy. Additionally, we have
compared the execution time to gain more insight into the working of the
algorithms. Generally, the running time of an algorithm depends on the number
of operations it has performed. So, we have trained our deep learning model up
to 30 epochs and SVM models according to norms to get the apt outcome. SVM
took the minimum time for execution while CNN accounts for the maximum running
time.

TABLE I: Comparison Analysis of different models. MODEL | TRAINING RATE | TESTING RATE | EXECUTION TIME
---|---|---|---
SVM | 99.98% | 94.005% | 1:35 min
MLP | 99.92% | 98.85% | 2:32 min
CNN | 99.53% | 99.31% | 44:02 min

Figure 8. This table represents the overall performance for each model. The
table contains 5 columns, the 2nd column represents model name, 3rd and 4th
column represents the training and testing accuracy of models, and 5th column
represents execution time of models.

Figure 9. Bar graph depicting accuracy comparison (SVM (Train: 99.98%, Test:
93.77%), MLP (Train: 99.92%, Test: 98.85%), CNN (Train: 99.53%, Test:
99.31%)).

Figure 10. Bar graph showing execution time comparison of SVM, MLP and CNN
(SVM: 1.58 mins, MLP: 2.53 mins, CNN: 44.05 mins).

Furthermore, we visualized the performance measure of deep learning models and
how they ameliorated their accuracy and reduced the error rate concerning the
number of epochs. The significance of sketching the graph is to know where we
should apply early stop so that we can avoid the problem of overfitting as
after some epochs, change in accuracy becomes constant.

Figure 11. Graph illustrating the transition of training loss with increasing
number of epochs in Multilayer Perceptron (Loss rate v/s Number of epochs).

Figure 12. Graph illustrating the transition of training accuracy with
increasing number of epochs in Multilayer Perceptron (Accuracy v/s Number of
epochs).

Figure 13. Graph illustrating the transition of training loss of CNN with
increasing number of epochs (Loss rate v/s Number of epochs).

Figure 14. Graph illustrating the transition of training accuracy of CNN with
increasing number of epochs (Accuracy v/s Number of epochs).

##  VI CONCLUSION

In this research, we have implemented three models for handwritten digit
recognition using MNIST datasets, based on deep and machine learning
algorithms. We compared them based on their characteristics to appraise the
most accurate model among them. Support vector machines are one of the basic
classifiers that’s why it’s faster than most algorithms and in this case,
gives the maximum training accuracy rate but due to its simplicity, it’s not
possible to classify complex and ambiguous images as accurately as achieved
with MLP and CNN algorithms. We have found that CNN gave the most accurate
results for handwritten digit recognition. So, this makes us conclude that CNN
is best suitable for any type of prediction problem including image data as an
input. Next, by comparing execution time of the algorithms we have concluded
that increasing the number of epochs without changing the configuration of the
algorithm is useless because of the limitation of a certain model and we have
noticed that after a certain number of epochs the model starts overfitting the
dataset and give us the biased prediction.

##  VII FUTURE ENHANCEMENT

The future development of the applications based on algorithms of deep and
machine learning is practically boundless. In the future, we can work on a
denser or hybrid algorithm than the current set of algorithms with more
manifold data to achieve the solutions to many problems.

In future, the application of these algorithms lies from the public to high-
level authorities, as from the differentiation of the algorithms above and
with future development we can attain high-level functioning applications
which can be used in the classified or government agencies as well as for the
common people, we can use these algorithms in hospitals application for
detailed medical diagnosis, treatment and monitoring the patients, we can use
it in surveillances system to keep tracks of the suspicious activity under the
system, in fingerprint and retinal scanners, database filtering applications,
Equipment checking for national forces and many more problems of both major
and minor category. The advancement in this field can help us create an
environment of safety, awareness and comfort by using these algorithms in day
to day application and high-level application (i.e. Corporate level or
Government level). Application-based on artificial intelligence and deep
learning is the future of the technological world because of their absolute
accuracy and advantages over many major problems.

## References

  * [1] “Handwriting recognition”:
$https://en.wikipedia.org/wiki/Handwriting_{r}ecognition$

  * [2] “What can a digit recognizer be used for?”:
$https://www.quora.com/What-can-a-digit-recognizer-be-used-for$

  * [3] ”Handwritten Digit Recognition using Machine Learning Algorithms”, S M Shamim, Mohammad Badrul Alam Miah, Angona Sarker, Masud Rana & Abdullah Al Jobair.
  * [4] ”Handwritten Digit Recognition Using Deep Learning”, Anuj Dutt and Aashi Dutt.
  * [5] ”Handwritten recognition using SVM, KNN, and Neural networks”, Norhidayu binti Abdul Hamid, Nilam Nur Binti Amir Sharif.
  * [6] ”Recognition of Handwritten Digit using Convolutional Neural Network in Python with Tensorflow and Comparison of Performance for Various Hidden Layers”, Fathma Siddique, Shadman Sakib, Md. Abu Bakr Siddique.
  * [7] ”Advancements in Image Classification using Convolutional Neural Network” by Farhana Sultana, Abu Sufian & Paramartha Dutta.
  * [8] ”Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18 F-FDG PET/CT images” by Hongkai Wang, Zongwei Zhou, Yingci Li, Zhonghua Chen, Peiou Lu, Wenzhi Wang, Wanyu Liu, and Lijuan Yu.
  * [9] $https://github.com/rishikakushwah16/SVM\\_digit$
$\\_recognition\\_using\\_MNIST\\_dataset$

  * [10] “Support Vector Machine Algorithm”:
$https://www.javatpoint.com/machine-learning-support-vector-machine-algorithm$

  * [11] https://machinelearningmastery.com/neural-networks-crash-course/
  * [12] “An Introduction to Convolutional Neural Networks”:
$researchgate.net/publication/285164623\\_An\\_Introduction\\_to$
$\\_Convolutional\\_Neural\\_Networks$

  * [13] “Activation Functions: Comparison of Trends in Practice and Research for Deep Learning”, Chigozie Enyinna Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall:
$https://arxiv.org/pdf/1811.03378.pdf$

  * [14] “Basic Overview of Convolutional Neural Network”:
https://medium.com/dataseries/basic-overview-of-convolutional-neural-network-
cnn-4fcc7dbb4f17

  * [15] https://github.com/dixitritik17/Handwritten-Digit-Recognition .
  * [16] https://scikit-learn.org/stable/modules/svm.html
  * [17] https://en.wikipedia.org/wiki/LIBSVM .
  * [18] $https://github.com/Flame-Atlas/Handwritten\\_Digit\\_$
$Recognition\\_using\\_MLP/blob/master/ANN.py$

  * [19] $https://www.learnopencv.com/image-classification$
$-using-feedforward-neural-network-in-keras/$

  * [20] “How Do Convolutional Layers Work in Deep Learning Neural Networks”:
$https://machinelearningmastery.com/convolutional$
$-layers-for-deep-learning-neural-networks/$

  * [21] https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/ .
  * [22] https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/ .
  * [23] https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5 .
  * [24] https://medium.com/dataseries/basic-overview-of-convolutional-neural-network-cnn-4fcc7dbb4f17 .
  * [25] https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d .