††thanks: These authors contributed equally††thanks: These authors contributed
equally

# Floquet engineering of strongly-driven excitons in monolayer tungsten
disulfide

Yuki Kobayashi<EMAIL_ADDRESS>Stanford PULSE Institute, SLAC National
Accelerator Laboratory, Menlo Park, CA 94025, USA  Department of Applied
Physics, Stanford University, Stanford, CA 94305, USA  Christian Heide
Stanford PULSE Institute, SLAC National Accelerator Laboratory, Menlo Park, CA
94025, USA  Department of Applied Physics, Stanford University, Stanford, CA
94305, USA  Amalya C. Johnson  Department of Materials Science and
Engineering, Stanford University, Stanford, CA 94305, USA  Fang Liu  Stanford
PULSE Institute, SLAC National Accelerator Laboratory, Menlo Park, CA 94025,
USA  Department of Chemistry, Stanford University, Stanford, CA 94305, USA
David A. Reis  Stanford PULSE Institute, SLAC National Accelerator Laboratory,
Menlo Park, CA 94025, USA  Department of Applied Physics, Stanford University,
Stanford, CA 94305, USA  Department of Photon Science, Stanford University,
Stanford, CA 94305, USA  Tony F. Heinz  Stanford PULSE Institute, SLAC
National Accelerator Laboratory, Menlo Park, CA 94025, USA  Department of
Applied Physics, Stanford University, Stanford, CA 94305, USA  Department of
Photon Science, Stanford University, Stanford, CA 94305, USA  Shambhu Ghimire
<EMAIL_ADDRESS>Stanford PULSE Institute, SLAC National Accelerator
Laboratory, Menlo Park, CA 94025, USA

Interactions of quantum materials with strong-laser fields can induce exotic
nonequilibrium electronic states [1]. Monolayer transition-metal
dichalcogenides, a new class of direct-gap semiconductors with prominent
quantum confinement [2], offer exceptional opportunities toward Floquet
engineering of quasiparticle electron-hole states, or excitons [3]. Strong-
field driving has a potential to achieve enhanced control of electronic band
structure, thus a possibility to open a new realm of exciton light-matter
interactions. However, experimental implementation of strongly-driven excitons
has so far remained out of reach. Here, we use mid-infrared laser pulses below
the optical bandgap to excite monolayer tungsten disulfide up to a field
strength of 0.3 V/nm, and demonstrate strong-field light dressing of excitons
in the excess of a hundred millielectronvolt. Our high-sensitivity transient
absorption spectroscopy further reveals formation of a virtual absorption
feature below the $\bm{1s}$-exciton resonance, which is assigned to a light-
dressed sideband from the dark $\bm{2p}$-exciton state. Quantum-mechanical
simulations substantiate the experimental results and enable us to retrieve
real-space movies of the exciton dynamics. This study advances our
understanding of the exciton dynamics in the strong-field regime, and
showcases the possibility of harnessing ultrafast, strong-field phenomena in
device applications of two-dimensional materials.

Laser-field manipulation of electronic band structure is at the frontier of
quantum-materials research [1]. In particular, monolayer transition-metal
dichalcogenides (TMDs) exhibit remarkable excitonic properties as a result of
their reduced dimensionality [2], and recent spectroscopic studies have shown
their promising potential toward novel optoelectronic devices [4, 5, 6, 7, 8,
9, 10, 11, 12, 13]. One of the common approaches is to coherently drive the
interband transition by a near-resonant visible pulse and induce an AC-Stark
shift to the exciton resonance [4, 5]. Another possible yet largely unexplored
approach is to drive excitons with a strong-laser field, thereby directly
imparting momentum from the oscillating laser field to the electron-hole
pairs. This limit of light-matter interactions is known as the strong-field
regime [14], in which unique laser-induced phenomena occur such as high-
harmonic generation [15], subcycle dynamic Franz-Keldysh effects [16], and
predicted control of topological properties [17].

To describe the strong-field regime of exciton dynamics, it is instructive to
consider the critical-field strength ($F_{\text{c}}$) that is required to
break apart an exciton into a free electron-hole pair. In the dipole limit,
the critical-field strength can be approximated as a ratio between the binding
energy ($E_{\text{b}}$) and Bohr radius ($a_{\text{B}}$) such that
$F_{\text{c}}=\frac{E_{\text{b}}}{ea_{\text{B}}}$. Typical parameters of
$E_{\text{b}}\approx 300$ meV and $a_{\text{B}}\approx 1$ nm [18] yield the
critical-field strength of 0.3 V/nm (laser intensity of 12 GW/cm2). Previous
studies have been limited below this order of the field strength; a recent
study by Cunningham et al. reported the largest AC-Stark shift of 32 meV with
a visible excitation, but the applied field strength was $<0.03$ V/nm ($<0.14$
GW/cm2) [8].

Driving and probing strong-field processes in two-dimensional materials poses
several experimental challenges. First, sample damage may occur as a result of
real-carrier generation by irradiation of an intense laser. Second, light-
dressing effects are a coherent process that is induced only during the laser
excitation, and probing cannot rely on incoherent optical processes such as
photoluminescence. Lastly, experimental signatures of the field-driven
dynamics, particularly those from exciton states, are yet to be established.

Figure 1:  Transient-absorption spectra of strongly-driven excitons in
monolayer WS2. a, Outline of the experiments. A mid-infrared pump pulse drives
coherent exciton dynamics in a monolayer WS2, and a visible probe pulse
encodes the light-dressing effects in its transmission spectrum. b-d,
Transient-absorption spectra ($\Delta$OD) of monolayer WS2 at 77K driven by a
4-$\upmu$m mid-infrared field. The positive (negative) delays correspond to
the mid-infrared (visible) pulse arriving first. Positive spectral chirp in
the visible spectrum causes overall tilt of the transient-absorption signals,
but this does not affect our main discussion. At a field strength of 0.08 V/nm
(b), the light-dressing effect induces a blue shift of the $1s$-exciton signal
via the AC-Stark shift. As the intensity increases to 0.2 V/nm (c), the AC-
Stark shift grows, and a new absorption feature emerges below the $1s$-exciton
resonance, which corresponds to a light-dressed sideband of the $2p$-exciton
state, or the $2p^{-1}$-exciton state. At the highest intensity of 0.3 V/nm
(d), significant blue shifts are characterized both for the $1s$\- and
$2p^{-1}$-exciton states. The energy diagram on the right shows the estimated
energies of the unperturbed light-dressed exciton states. e, Driving-
wavelength dependence of the AC-Stark shift at the $1s$-exciton resonance at
zero delay time. The blue shift under the 4-$\upmu$m driving changes to a
symmetric splitting under 7-$\upmu$m driving, which yields an estimated
$1s$-$2p$ transition energy of 0.18 eV. These measurements are performed at
room temperature.

Here, we overcome these challenges and demonstrate strong-field control of
exciton states in monolayer tungsten disulfide (WS2). Our main experimental
findings include an anomalously large energy shift to the $1s$-exciton
resonance, and emergence of a light-dressed Floquet sideband from the
$2p$-exciton state. Experimentally, a 200-fs mid-infrared pulse of 4.0-um
wavelength (photon energy of 0.31 eV, frequency of 75 THz) is used to excite
monolayer WS2 (Fig. 1a). The small photon energy of the mid-infrared pulse
allows us to reach a field strength of 0.3 V/nm without damaging the sample,
while light-dressing effects are induced by coherently driving internal
resonance of the exciton states [19, 9]. Such an intense mid-infrared pulse is
produced by way of difference-frequency generation in a gallium-selenide
crystal by using an output of a high-power optical-parametric amplification
system. We employ optical transient-absorption spectroscopy as a probing
method, wherein a broadband visible pulse encodes the exciton dynamics in its
transmission spectrum. Differential absorption ($\Delta$OD) is obtained by
taking the pump-on ($I_{\text{on}}$) and pump-off ($I_{\text{off}}$) spectra
at a controlled delay time $t$, such that
$\Delta\text{OD}=-\log(I_{\text{on}}/I_{\text{off}})$. A high signal
sensitivity is achieved by means of single-shot detection with a fast-rate
spectrometer; accumulation of 30,000 laser shots leads to the detection
sensitivity of $3\times 10^{-5}$. A high-quality, millimeter-scale sample of
monolayer WS2 is prepared by the gold-tape exfoliation method, and it was kept
in a vacuum cryostat at 77 K [20, 21].

Transient-absorption spectra of monolayer WS2 under 4-$\upmu$m driving are
shown in Figs. 1b-d. At a lower mid-infrared intensity (Fig. 1b, 0.08 V/nm),
the $1s$-exciton signal at 2.07 eV shows a blue shift. This asymmetric
behavior cannot be explained by peak broadening [11], indicating that the
underlying exciton light-matter interactions are a coherent process.
Additional measurements are carried out by changing the mid-infrared
wavelength (Fig. 1e), and the transient-absorption signal is found to be
symmetric under 7-$\upmu$m driving (photon energy of 0.18 eV). The symmetric
splitting (i.e., Autler-Townes splitting) signifies that the 7-$\upmu$m field
is near-resonant to the intraexcitonic transition [9], thus yielding an
estimated $1s$-$2p$ transition energy of 0.18 eV.

At a higher intensity (Fig. 1c, 0.2 V/nm), a large ($\sim 40$ meV) blue shift
is observed at the $1s$-exciton resonance. More importantly, a new absorption
feature emerges below the $1s$-exciton resonance, which we assign to a single-
photon-dressed Floquet sideband of the $2p$-exciton state, or the
$2p^{-1}$-exciton state. The original $2p$-exciton state is dipole forbidden
from the ground state and does not directly contribute to absorption spectra
[22]. Single-photon dressing, however, can create a replica of the
$2p$-exciton state that has the opposite parity. Hybridization between the
bright $1s$-exciton state can impart optical strength under a moderate driving
intensity, thus creating a new virtual state that is present only during the
mid-infrared driving.

There are two adjunct features that are worthy of note (Fig. 1c). One is
hyperbolic sidebands that appear around the $1s$-exciton signal in the
negative delays (i.e., the visible pulse arriving first). This is due to the
perturbation of the free-induction decay at the $1s$-exciton resonance by the
strong mid-infrared field. Similar features are observed commonly in time-
resolved spectroscopy [23]. The other feature is possible implication of
carrier generation by the strong mid-infrared field. A long delay-time
measurement shows that the differential absorption at the $1s$-exciton
resonance at 3000 fs is as small as $0.8\times 10^{-3}$ (Supplementary), and
thus it is safe to assume that the effect of carrier generation is negligibly
small.

As the driving intensity increases to 0.3 V/nm (Fig. 1d), the blue shift at
the $1s$-exciton resonance starts to entail significant peak broadening, which
is indicative of exciton ionization. It is also characterized that the
$2p^{-1}$-exciton signal experiences a large ($\sim 40$ meV) blue shift by the
intense mid-infrared field. As we will explain, this is an unexpected result
from the conventional two-level picture; if the AC-Stark shift causes a blue
shift to the $1s$-exciton state, the paired $2p^{-1}$-exciton state should
experience the same amount of energy shift to the opposite direction, thus a
red shift.

Figure 2:  Light-dressing mechanisms of strongly-driven excitons. a, An
effective two-dimensional potential of excitons and associated energy levels.
An external laser field can excite internal resonance of excitons such as the
$1s$-$2p$ transition, or drive the exciton wavepacket into the quasifree
region of the potential. b,c, Schematic illustration of the light-dressing
effects. In the bound-bound coupling case (b), an external laser field can
create light-dressed sidebands (LDS) and the mixing between the light-dressed
states induces the AC-Stark shift (ACS). In the bound-continuum coupling case
(c), excitons can gain kinetic energy from the external laser field in the
quasifree region, and the renormalization of the energy levels results in the
dynamic Franz-Keldysh effect (DFKE).

We proceed to theoretical analysis of the experimental results. Our primary
interest lies in finding optical signatures of the field-driven dynamics of
excitons. The mechanisms of the mid-infrared light dressing are illustrated in
Figs. 2a-c. When the excitation is limited to the lower bound states, such as
the $1s$\- and $2p$-exciton states, the light-dressing effect can be
understood within the Floquet theory (Fig. 2b) [24, 25]. The external laser
field creates replicas of the exciton states that are evenly spaced by the
driving-photon energy, and hybridization between the neighboring states leads
to the AC-Stark shifts [9]. In the other case where a strong laser field is
applied, exciton wavepackets can be driven out of the bound potential to the
quasifree region (Fig. 2c). The excitons in the asymptote can gain significant
kinetic energy from the oscillating laser field, which leads to
renormalization of energy levels (i.e., the dynamic Franz-Keldysh effect) [26,
27]. In the case of a parabolic band (i.e., quasifree electrons), the
renormalization results in a universal blue shift of the energy structure by
the amount equal to the ponderomotive energy [28].

Figure 3:  Optical signatures of the field-driven exciton dynamics. a-c,
Transient-absorption spectra ($\Delta$OD) of monolayer WS2 driven by a
4-$\upmu$m mid-infrared field at various intensities. The experimental
transient-absorption spectra (a) are measured at -60 fs and 60 fs delay times
for above and below the $1s$-exciton resonance, respectively, to account for
positive chirp in the visible spectrum. The experimental results are compared
to the simulation results obtained by the 2D-TDSE method (b) and the $1s$-$2p$
model (c). d, Comparison of the transient-absorption spectra at low (0.05
V/nm) and high (0.25 V/nm) mid-infrared field intensities. The arrows indicate
the peak energies of the $1s$\- and $2p^{-1}$-exciton states. e, Summary of
the experimental and simulated peak energies for the $1s$\- and
$2p^{-1}$-exciton states.

We simulate transient-absorption spectra of the strongly-driven excitons with
two models. In the first model, the exciton light-matter interactions are
fully considered, including the bound-bound coupling, bound-continuum
coupling, and ionization. This is achieved by numerically solving the time-
dependent Schrödinger equation (TDSE) for an exciton wavepacket in the
nonhydrogenic two-dimensional Keldysh potential [29, 18],

$\displaystyle V(r)=-\frac{\pi
e^{2}}{2r_{0}}\left[H_{0}\left(\frac{r}{r_{0}}\right)-Y_{0}\left(\frac{r}{r_{0}}\right)\right],$
(1)

where $H_{0}$ and $Y_{0}$ are Struve and Bessel functions, and $r_{0}$ is the
screening length of Coulomb attraction in monolayer WS2 (Fig. 2a). The reduced
mass of the exciton is fixed at $\mu=0.16m_{0}$ [30], and the free parameter
of the screening length is chosen to be $r_{0}=50$ Å so that the experimental
$1s$-$2p$ transition energy of 0.18 eV is reproduced. The other model uses a
minimum discrete system that consists only of the ground, $1s$-, and
$2p$-exciton states. In this case, the mid-infrared field can drive only one
bound-bound coupling, i.e., the $1s$-$2p$ transition, and the exciton dynamics
in the quasifree region are excluded. In both models, the absorption cross
section of the exciton wavepacket is calculated as
$\sigma(\omega)\propto\omega\operatorname{Im}\left[\frac{\widetilde{d}(\omega)}{\widetilde{E}(\omega)}\right],$
where $\widetilde{d}(\omega)$ and $\widetilde{E}(\omega)$ are the Fourier-
transformed optical dipole moment and laser electric field, respectively (see
Methods for more details) [31].

Figure 3 shows the comparison of the experimental (Fig. 3a) and simulated
(Figs. 3b,c) transient-absorption spectra at various mid-infrared intensities.
To account for the visible spectral chirp in the experiments, the transient-
absorption signals below and above the $1s$-exciton resonance are measured at
$60$ fs and $-60$ fs, respectively In the low-intensity case at 0.05 V/nm
(Fig. 3d), the AC-Stark shift is the main feature of the transient-absorption
signals, and the two models reproduce the experimental result. In the high-
intensity case at 0.25 V/nm (Fig. 3d), the 2D-TDSE method captures the
anomalously large blue shift in the 1$s$-exciton signal as well as the
emergence of the light-dressed 2$p^{-1}$-exciton state. In contrast, the
$1s$-$2p$ model exhibits a poor matching; the energy of the $1s$\- and
$2p^{-1}$-exciton signals are underestimated by $\sim 30$ meV and $\sim 80$
meV, respectively. Overall, the 2D-TDSE method successfully reproduces the
$1s$\- and $2p^{-1}$-exciton signals throughout the intensity regime measured
in the experiments, whereas the $1s$-$2p$ model works only at the lower field
strengths (Fig. 3e). The contrast between the two models originates from the
fact that the 2D-TDSE method fully considers the bound-bound and bound-
continuum couplings that are expressed within the grid space ($r\leq 300$ Å),
whereas the $1s$-$2p$ model is strictly restricted to the single bound-bound
coupling.

Figure 4:  Snapshots of the exciton wavepacket. a Schematic of the exciton
dynamics in a low-intensity case (0.05 V/nm). (b) Snapshots of the exciton
wavepacket (squared probability distribution) obtained in the 2D-TDSE
simulations. The wavepacket remains localized at the potential center, and it
exhibits only one node after two cycles ($t=2T$) of the mid-infrared
excitation. This shows that the excitation is limited to the $2p$-exciton
state. (c) Schematic of the exciton dynamics in a high-intensity case (0.25
V/nm). d, Here, the strong mid-infrared field drives the exciton wavepacket
out of the bound potential into the quasifree region within a cycle. The
wavepacket is no longer localized at the potential center, and multiple nodes
appearing in the snapshots represent the interference between the excited-
state quasicontinuum.

The good agreement between the experiments and the 2D-TDSE simulation further
allows us to obtain real-space pictures of the strongly-driven excitons (Fig.
4). Here, the initial wavepacket is assumed to be the $1s$-exciton state
launched by the visible pulse at $t=0$, and the mid-infrared driving field is
a sine pulse (i.e., the electric field is zero at $t=0$). In the low intensity
case at 0.05 V/nm (Figs. 4a,b), the exciton wavepacket remains localized
around the potential center. After two cycles ($t=2T$), the exciton wavepacket
exhibits only one node structure, which shows that the excitation is limited
to the $2p$-exciton state. In the high intensity case at 0.25 V/nm (Figs.
4c,d), the center-of-mass of the wavepacket is moved by $\sim 30$ Å only after
a half cycle ($t=T/2$). Multiple nodal features emerge at later times, which
originate from interference between the excited-state quasicontinuum. Note
that a certain portion of the population remains at the potential center,
which shows that we are in the regime where both the bound-bound coupling
(i.e., AC-Stark shift) and the bound-continuum coupling (i.e., dynamic Franz-
Keldysh effect) are in effect. These observations, in combination with the
results in Fig. 3, establish the role of the field-driven dynamics of excitons
under the strong mid-infrared driving.

In conclusion, we investigated the strong-field regime of the exciton dynamics
in monolayer tungusten disulfide by using an intense mid-infrared pump in a
transient-absorption configuration. We characterized an anomalously large
($\sim 140$ meV) blue shift in the $1s$-exciton resonance and a completely new
light-dressed sideband originating from the optically-dark $2p$-exciton state.
We reproduced our experimental results by solving the time-dependent
Schrödinger equation in a two-dimensional Keldysh potential, which
subsequently allowed us to reconstruct the subcycle evolution of the exciton
wavepackets. At moderate intensities, the wavepacket oscillates with the field
but remains localized at the potential center, whereas at the highest
intensities, the wavepacket spreads significantly such that multiple nodal
structures arise from interference between the bound wavepacket and the
continuum. Previously, subcycle control of electron motion was studied for
atomic systems [32, 33]. Now that the applicability of the same concept is
established for two-dimensional materials, it might be possible to trigger and
control active motion of excitons at the fundamental time scale of the
driving-laser fields. This will open the path toward atomically-thin
electroabsorption modulators with operation frequencies in the petahertz
regime [34]. Our results also suggest that strongly-driven excitons could
affect the high-harmonic generation process in monolayer TMDs, where enhanced
per-layer efficiency has been reported compared to the bulk [35].

## References

  * McIver  _et al._ [2020] J. W. McIver, B. Schulte, F.-U. Stein, T. Matsuyama, G. Jotzu, G. Meier, and A. Cavalleri, Light-induced anomalous hall effect in graphene, Nature Physics 16, 38 (2020).
  * Mak  _et al._ [2010] K. F. Mak, C. Lee, J. Hone, J. Shan, and T. F. Heinz, Atomically thin mos2: A new direct-gap semiconductor, Phys. Rev. Lett. 105, 136805 (2010).
  * Wang  _et al._ [2018] G. Wang, A. Chernikov, M. M. Glazov, T. F. Heinz, X. Marie, T. Amand, and B. Urbaszek, Colloquium: Excitons in atomically thin transition metal dichalcogenides, Rev. Mod. Phys. 90, 021001 (2018).
  * Kim  _et al._ [2014] J. Kim, X. Hong, C. Jin, S.-F. Shi, C.-Y. S. Chang, M.-H. Chiu, L.-J. Li, and F. Wang, Ultrafast generation of pseudo-magnetic field for valley excitons in wse¡sub¿2¡/sub¿ monolayers, Science 346, 1205 (2014).
  * Sie  _et al._ [2015] E. J. Sie, J. W. McIver, Y.-H. Lee, L. Fu, J. Kong, and N. Gedik, Valley-selective optical stark effect in monolayer ws2, Nature Materials 14, 290 (2015).
  * Ye  _et al._ [2017] Z. Ye, D. Sun, and T. F. Heinz, Optical manipulation of valley pseudospin, Nature Physics 13, 26 (2017).
  * Yong  _et al._ [2018] C.-K. Yong, J. Horng, Y. Shen, H. Cai, A. Wang, C.-S. Yang, C.-K. Lin, S. Zhao, K. Watanabe, T. Taniguchi, S. Tongay, and F. Wang, Biexcitonic optical stark effects in monolayer molybdenum diselenide, Nature Physics 14, 1092 (2018).
  * Cunningham  _et al._ [2019] P. D. Cunningham, A. T. Hanbicki, T. L. Reinecke, K. M. McCreary, and B. T. Jonker, Resonant optical stark effect in monolayer ws2, Nature Communications 10, 5539 (2019).
  * Yong  _et al._ [2019] C.-K. Yong, M. I. B. Utama, C. S. Ong, T. Cao, E. C. Regan, J. Horng, Y. Shen, H. Cai, K. Watanabe, T. Taniguchi, S. Tongay, H. Deng, A. Zettl, S. G. Louie, and F. Wang, Valley-dependent exciton fine structure and autler–townes doublets from berry phases in monolayer mose2, Nature Materials 18, 1065 (2019).
  * Morrow  _et al._ [2020] D. J. Morrow, D. D. Kohler, Y. Zhao, J. M. Scheeler, S. Jin, and J. C. Wright, Quantum interference between the optical stark effect and resonant harmonic generation in ${\mathrm{ws}}_{2}$, Phys. Rev. B 102, 161401 (2020).
  * Shi  _et al._ [2020] J. Shi, E. Baldini, S. Latini, S. A. Sato, Y. Zhang, B. C. Pein, P.-C. Shen, J. Kong, A. Rubio, N. Gedik, and K. A. Nelson, Room temperature terahertz electroabsorption modulation by excitons in monolayer transition metal dichalcogenides, Nano Letters 20, 5214 (2020).
  * Earl  _et al._ [2021] S. K. Earl, M. A. Conway, J. B. Muir, M. Wurdack, E. A. Ostrovskaya, J. O. Tollerud, and J. A. Davis, Coherent dynamics of floquet-bloch states in monolayer $\mathrm{W}{\mathrm{s}}_{2}$ reveals fast adiabatic switching, Phys. Rev. B 104, L060303 (2021).
  * LaMountain  _et al._ [2021] T. LaMountain, J. Nelson, E. J. Lenferink, S. H. Amsterdam, A. A. Murthy, H. Zeng, T. J. Marks, V. P. Dravid, M. C. Hersam, and N. P. Stern, Valley-selective optical stark effect of exciton-polaritons in a monolayer semiconductor, Nature Communications 12, 4530 (2021).
  * Brabec and Krausz [2000] T. Brabec and F. Krausz, Intense few-cycle laser fields: Frontiers of nonlinear optics, Rev. Mod. Phys. 72, 545 (2000).
  * Ghimire  _et al._ [2011] S. Ghimire, A. D. DiChiara, E. Sistrunk, P. Agostini, L. F. DiMauro, and D. A. Reis, Observation of high-order harmonic generation in a bulk crystal, Nature Physics 7, 138 (2011).
  * Lucchini  _et al._ [2016] M. Lucchini, S. A. Sato, A. Ludwig, J. Herrmann, M. Volkov, L. Kasmi, Y. Shinohara, K. Yabana, L. Gallmann, and U. Keller, Attosecond dynamical franz-keldysh effect in polycrystalline diamond, Science 353, 916 (2016).
  * Silva  _et al._ [2019] R. E. F. Silva, Á. Jiménez-Galán, B. Amorim, O. Smirnova, and M. Ivanov, Topological strong-field physics on sub-laser-cycle timescale, Nature Photonics 13, 849 (2019).
  * Chernikov  _et al._ [2014] A. Chernikov, T. C. Berkelbach, H. M. Hill, A. Rigosi, Y. Li, O. B. Aslan, D. R. Reichman, M. S. Hybertsen, and T. F. Heinz, Exciton binding energy and nonhydrogenic rydberg series in monolayer ${\mathrm{ws}}_{2}$, Phys. Rev. Lett. 113, 076802 (2014).
  * Poellmann  _et al._ [2015] C. Poellmann, P. Steinleitner, U. Leierseder, P. Nagler, G. Plechinger, M. Porer, R. Bratschitsch, C. Schüller, T. Korn, and R. Huber, Resonant internal quantum transitions and femtosecond radiative decay of excitons in monolayer wse2, Nature Materials 14, 889 (2015).
  * Ajayi  _et al._ [2017] O. A. Ajayi, J. V. Ardelean, G. D. Shepard, J. Wang, A. Antony, T. Taniguchi, K. Watanabe, T. F. Heinz, S. Strauf, X.-Y. Zhu, and J. C. Hone, Approaching the intrinsic photoluminescence linewidth in transition metal dichalcogenide monolayers, 2D Materials 4, 031011 (2017).
  * Liu  _et al._ [2020] F. Liu, W. Wu, Y. Bai, S. H. Chae, Q. Li, J. Wang, J. Hone, and X.-Y. Zhu, Disassembling 2d van der waals crystals into macroscopic monolayers and reassembling into artificial lattices, Science 367, 903 (2020).
  * Ye  _et al._ [2014] Z. Ye, T. Cao, K. O’Brien, H. Zhu, X. Yin, Y. Wang, S. G. Louie, and X. Zhang, Probing excitonic dark states in single-layer tungsten disulphide, Nature 513, 214 (2014).
  * Lindberg and Koch [1988] M. Lindberg and S. W. Koch, Transient oscillations and dynamic stark effect in semiconductors, Phys. Rev. B 38, 7607 (1988).
  * Shirley [1965] J. H. Shirley, Solution of the schrödinger equation with a hamiltonian periodic in time, Phys. Rev. 138, B979 (1965).
  * Sambe [1973] H. Sambe, Steady states and quasienergies of a quantum-mechanical system in an oscillating field, Phys. Rev. A 7, 2203 (1973).
  * Jauho and Johnsen [1996] A. P. Jauho and K. Johnsen, Dynamical franz-keldysh effect, Phys. Rev. Lett. 76, 4576 (1996).
  * Nordstrom  _et al._ [1998] K. B. Nordstrom, K. Johnsen, S. J. Allen, A.-P. Jauho, B. Birnir, J. Kono, T. Noda, H. Akiyama, and H. Sakaki, Excitonic dynamical franz-keldysh effect, Phys. Rev. Lett. 81, 457 (1998).
  * Tsuji  _et al._ [2008] N. Tsuji, T. Oka, and H. Aoki, Correlated electron systems periodically driven out of equilibrium: $\text{Floquet}+\text{DMFT}$ formalism, Phys. Rev. B 78, 235124 (2008).
  * Keldysh [1979] L. V. Keldysh, Coulomb interaction in thin semiconductor and semimetal films, JETP Lett. 29, 716 (1979).
  * Berkelbach  _et al._ [2013] T. C. Berkelbach, M. S. Hybertsen, and D. R. Reichman, Theory of neutral and charged excitons in monolayer transition metal dichalcogenides, Phys. Rev. B 88, 045318 (2013).
  * Wu  _et al._ [2016] M. Wu, S. Chen, S. Camp, K. J. Schafer, and M. B. Gaarde, Theory of strong-field attosecond transient absorption, J. Phys. B 49, 062003 (2016).
  * Chini  _et al._ [2013] M. Chini, X. Wang, Y. Cheng, Y. Wu, D. Zhao, D. A. Telnov, S.-I. Chu, and Z. Chang, Sub-cycle oscillations in virtual states brought to light., Sci. Rep. 3, 1105 (2013).
  * Ott  _et al._ [2014] C. Ott, A. Kaldun, L. Argenti, P. Raith, K. Meyer, M. Laux, Y. Zhang, A. Blättermann, S. Hagstotz, T. Ding, R. Heck, J. Madroñero, F. Martín, and T. Pfeifer, Reconstruction and control of a time-dependent two-electron wave packet, Nature 516, 374 (2014).
  * Kruchinin  _et al._ [2018] S. Y. Kruchinin, F. Krausz, and V. S. Yakovlev, Colloquium: Strong-field phenomena in periodic systems, Rev. Mod. Phys. 90, 021002 (2018).
  * Liu  _et al._ [2017] H. Liu, Y. Li, Y. S. You, S. Ghimire, T. F. Heinz, and D. A. Reis, High-harmonic generation from an atomically thin semiconductor, Nature Physics 13, 262 (2017).
  * Colbert and Miller [1992] D. T. Colbert and W. H. Miller, A novel discrete variable representation for quantum mechanical reactive scattering via the s‐matrix kohn method, The Journal of Chemical Physics 96, 1982 (1992).
  * Glutsch  _et al._ [1996] S. Glutsch, D. S. Chemla, and F. Bechstedt, Numerical calculation of the optical absorption in semiconductor quantum structures, Phys. Rev. B 54, 11592 (1996).
  * Zhang and Zhao [2006] T. Y. Zhang and W. Zhao, Franz-keldysh effect and dynamical franz-keldysh effect of cylindrical quantum wires, Phys. Rev. B 73, 245337 (2006).

##  I Acknowledgments

We thank Jiaojian Shi, Ignacio Franco, and Vishal Tiwari for discussion. This
work was primarily supported by the US Department of Energy, Office of
Science, Basic Energy Sciences, Chemical Sciences, Geosciences, and
Biosciences Division through the AMOS program. F.L. was supported by the
Terman Fellowship and startup funds from the Department of Chemistry at
Stanford University. Y.K. acknowledges support from the Urbanek-Chorodow
Fellowship from Stanford University. C.H acknowledges support from the W. M.
Keck Foundation and the Humboldt Research Fellowship.

##  II Author contributions

Y.K. and C.H. performed the experiments. A.C.J. and F.L. fabricated the
sample. Y.K. performed the simulations and analyzed the results. F.L., D.A.R.,
T.F.H., and S.G. supervised the project. All authors contributed to the
preparation of the manuscript.

##  III Competing interests

The authors declare no competing interests.

##  IV Data availability

The experimental and simulation data presented in this study are available
from the corresponding author upon request.

##  V Methods

Experimental details of optical transient-absorption spectroscopy. The output
from the Ti:sapphire amplifier (Evolution, Coherent Inc., 6 mJ, 45 fs, 790 nm,
1 kHz) is used to pump an optical-parametric amplification (OPA) system
(TOPAS-HE, Light Conversion Inc.). The signal ($\sim 1300$ nm) and idler
($\sim 1900$ nm) from the OPA are mixed in a GaSe crystal (Eksma Optics Inc.,
z-cut, 0.5 mm thick) for difference-frequency generation, and the generated
mid-infrared pulse is spectrally filtered by a bandpass filter that is
centered at 4.0 $\upmu$m (Thorlabs Inc., FB4000-500). For the 7-$\upmu$m
measurement, a different bandpass filter at 7.0 $\upmu$m (Thorlabs Inc.,
FB7000-500) is used and the incident angle to the GaSe crystal is adjusted for
phase matching at 7 $\upmu$m. A combination of wire-grid polarizers are used
to clean the polarization of the mid-infrared pulse while enabling continuous
tuning of the laser intensity. The broadband visible pulse is produced by
focusing the signal pulse to a 3-mm sapphire plate. The mid-infrared pump and
the visible probe are focused onto a sample by a 150-mm ZnSe lens and by a
100-mm off-axis parabola, respectively, in a noncollinear geometry with a
crossing angle of $\sim 25^{\circ}$. The monolayer WS2 is placed in an optical
cryostat (ST-100, Lake Shore Cryotronics, Inc.) at a temperature of 77 K. The
transmitted visible spectra are collected by a CaF2 lens and directed into a
fast-rate spectrometer (Ocean Insight Inc., Ocean-FX). The pump-on and pump-
off spectra ($I_{\text{on}}$ and $I_{\text{off}}$) are recorded on a single-
shot basis of 1 kHz, and the differential absorption
$\Delta\text{OD}=-\log(I_{\text{on}}/I_{\text{off}})$ is obtained at
controlled delay times. The detection sensitivity of $3\times 10^{-5}$ is
achieved for accumulation of 30,000 spectra (i.e., 15,000 pump-on and pump-off
spectra each). A monolayer WS2 sample is exfoliated from bulk single crystal
(HQ graphene) on a passivated fused silica substrate (100 um thick, double-
side polished, MTI corporation) by the gold-tape exfoliation method [20, 21].

2D-TDSE simulation. In the two-dimensional time-dependent Schrödinger equation
(2D-TDSE) method, the Hamiltonian of the exciton wavepacket is given by

$\displaystyle H(x,y,t)=-\frac{1}{2\mu}\frac{\partial^{2}}{\partial
x^{2}}-\frac{1}{2\mu}\frac{\partial^{2}}{\partial y^{2}}-V(x,y)$
$\displaystyle-\overrightarrow{E}(t)\cdot\overrightarrow{d}(x,y)-\frac{i\Gamma}{2},$
(2)

where $\mu$ is the reduced mass of an exciton, $V(x,y)$ is the effective model
potential, $\overrightarrow{E}(t)$ is the external laser field,
$\overrightarrow{d}$ is the dipole moment, and $\Gamma$ is the
phenomenological parameter to account for the finite lifetime of excitons.
Atomic units are used in this section. As described in the main text, the
reduced mass of the exciton is set to be $\mu=0.16m_{0}$ as determined
previously by density-functional-theory calculations [30]. The free parameter
of the screening length is chosen to be $r_{0}=50$ Å such that the
experimental $1s$-$2p$ transition energy of 0.18 eV is reproduced. The exciton
binding energies reported for monolayer WS2 range from 0.32 eV [18] to 0.7 eV
[22], and the value obtained with these parameters, 0.42 eV, falls within this
range. The phenomenological lifetime of the excitons is set to be $\Gamma=15$
meV, which corresponds to a $1/e$ lifetime of 44 fs, to match the experimental
linewidth of the $1s$-exciton signal. The mid-infrared laser is approximated
as a 75-fs Gaussian pulse with a center wavelength of $4\upmu$m, and the
visible laser is approximated as a 1-fs Gaussian pulse with a center
wavelength of $600$ nm. In the calculations, the laser fields are assumed to
be polarized along the $x$ direction. The exciton wavepacket is represented by
the sinc-function discrete variable representation (sinc-DVR) [36] on
$80\times 80$ two-dimensional grids that span from -300 Å to 300 Å. Complex
absorption potential is defined at the grid boundary to prevent reflection of
the wavepacket, which also models strong-field ionization of the excitons. The
time-dependent Schrödinger equation is solved numerically by using the fourth-
order Runge-Kutta method at a step size of 31 as up to 500 fs. The absorption
cross section of the exciton wavepacket is calculated as

$\displaystyle\sigma(\omega)\propto\omega\operatorname{Im}\left[\frac{\widetilde{d}(\omega)}{\widetilde{E}(\omega)}\right],$
(3)

where $\widetilde{d}(\omega)$ and $\widetilde{E}(\omega)$ are the Fourier-
transformed optical dipole moment and laser electric field, respectively [31].
The optical dipole moment $d(t)$ is considered proportional to the density of
the wavepacket at $x,y=0$, which is a common approach used to calculate
absorption spectra of semiconducting nanomaterials [37, 38]. A constant energy
shift is added to the absorption cross section so that the position of the
$1s$-exciton signal matches with the one measured in the experiments (2.07
eV). For the simulation results presented in Fig. 3, the absorption spectra
are averaged over two mid-infrared pulses, a cosine pulse and a sine pulse, to
take into account the randomized carrier-envelope phase of the experimental
mid-infrared pulses.

$\bm{1s}$-$\bm{2p}$ model simulation. In the $1s$-$2p$ model, the Hamiltonian
of the exciton states is simplified as,

$\displaystyle H(t)=\left(\begin{matrix}0&E(t)\cdot d_{g,1s}&0\\\ E(t)\cdot
d_{g,1s}&\epsilon_{1s}-i\Gamma/2&E(t)\cdot d_{1s,2p}\\\ 0&E(t)\cdot
d_{1s,2p}&\epsilon_{2p}-i\Gamma/2\end{matrix}\right),$ (4)

where $\epsilon_{i}$ and $d_{i,j}$ represent the energy and transition dipole
moment, respectively. These state parameters are obtained from the effective
model potential used in the 2D-TDSE method. The same parameters and methods as
in the 2D-TDSE method are used for the laser field, exciton lifetime, and time
propagation. The optical dipole moment is obtained as

$\displaystyle d(t)=\braket{\Psi(t)}{\bm{d}}{\Psi(t)},$ (5)

where $\Psi_{(}t)$ is a column vector for the state coefficients and $\bm{d}$
is the transition dipole matrix. The absorption cross section is calculated in
the same way by using the Eq. (3).


# Analyzing Compression Techniques for Computer Vision

Imran Khan
PhD ORIE
<EMAIL_ADDRESS>Maniratnam Mandal
PhD ECE
<EMAIL_ADDRESS>
###### Abstract

Compressing deep networks is highly desirable for practical use-cases in
computer vision applications. Several techniques have been explored in the
literature, and research has been done in finding efficient strategies for
combining them. For this project, we aimed to explore three different basic
compression techniques - knowledge distillation, pruning, and quantization for
small-scale recognition tasks. Along with the basic methods, we also test the
efficacy of combining them in a sequential manner. We analyze them using MNIST
and CIFAR-10 datasets and present the results along with few observations
inferred from them.

##  1 Introduction

Neural network compression is an important issue for improving the usability
of networks in smaller and less powerful devices. As the size of deep networks
increases, they become increasingly more demanding in their processing power
and storage requirements. Some of the largest neural networks built for vision
tasks contain parameters on the scale of billions. As the growing trend is
that neural networks are increasingly used in mobile phones, it is sometimes
infeasible for extremely large networks to be trained, or even deployed.

In addition to storage and memory concerns with over-parameterized neural
networks, there are other concerns that would warrant smaller, more usable
networks. It is important to note the speed and latency concerns of networks
are just as important. Additionally, it is estimated that the GBT-3 network,
with 175 Billion parameters, emitted a total of 552.1 tons of carbon dioxide
in training [21]. Thus, if reducing the size of networks can lead to increased
energy efficiency, there are potential environmental and ethical reasons for
wanting to reduce the size of extremely large networks.

One simple solution to combat this problem is to make smaller networks with
much fewer and manageable parameters. While this solution would decrease the
size of the networks, this comes at the cost of decreased model performance.
This would not be favorable for many recognition based applications, be it
auto-navigation, image captioning, visual question answering, or face
recognition-based security measures. Similarly, another simple solution would
be to “outsource” the model to a server. This would mean processing the model
on a server, which would require sending large amounts of data back and forth
between the mobile phone and the server. This is also not a perfect solution
to our problem, because this would require a great internet connection for the
data to be sent, and would incur unnecessary delays that slows down the
processing. For example, imagine having to wait every time you wanted to use
facial recognition to unlock your phone.

These solutions, although feasible, are not sufficient. The field of network
compression has introduced several techniques for mitigating this. The goal of
neural network compression is to obtain the same or similar generalization
performance by efficient storage, optimization, computation, and often
reduction in network size. The reduction in storage and computation leads to
critical improvements in real-time applications such as in mobile devices,
smart wearables, online learning, etc. Some of the recent major techniques in
network compression are – parameter pruning and quantization, low-rank
factorization, transferred/compact convolutional filters, and knowledge
distillation [19].

According to [24], these methods can be categorized into two common streams:
the ”transfer” stream and the ”compress” stream. Techniques in the transfer
stream is focused on training a new small network, whereas techniques in the
compress stream are focused on decreasing the size of the model during or
after training. As the techniques have been developed independently, they can
be used in conjunction and to complement each other. The goal of this project
is to analyze and combine these techniques to achieve higher levels of
compression while preserving performance. The major gap that this project is
hoping to fill is the gap between the two streams. We would like to show
empirical results of neural network compression techniques that can be done
through a combination of transferring knowledge to a smaller network and
simultaneously compressing the networks.

Our paper is organized as follows: the following section contains a review of
the literature that has already been done on the subject of neural network
compression; Section 3 discusses the methods we choose to use in our
experiments; Section 4 goes into more detail about the details of our
experimental setup; Section 5 talks about our results; and we end with a
conclusion in Section 6.

##  2 Related Work

###  2.1 Quantization

Quantization has been widely used as a compression technique across all fields
of engineering sciences. Reducing the number of bits to represent each weight
can lead to reduced memory and increased computation [23]. [4] aimed to
decrease the size of the network through storing with a smaller format. In
this paper, the authors test three benchmark data sets with three different
storing formats: floating point, fixed point, and dynamic fixed point. They
find that very low precision storage is good enough for running trained
networks, and that it is also sufficient for training them.

Binary weight neural networks have also been explored [5], although the
performance of such networks take a major hit in the case of deep CNNs. [13]
introduces Quantization Aware Training, which is a method of helping the
accuracy of a model be less affected with Post Training Quantization, by
accounting for the quantization loss in the training phase of the model. This
will be discussed in more detail in Section 3.3.

###  2.2 Pruning

Early pruning based methods reduced the number of connections among the layers
based on the loss function [19]. Pruning methods also include removing
redundant neurons, remove redundant connections, quantizing and encoding
weights, and parameter sharing. HashedNets uses a low-cost hash function to
store and group weights for parameter sharing [2]. Some compact DNNs are
trained with sparsity constraints during optimization for regularization.
Pruning with regularization can require a large time to converge and usually
involve intricate parameter tuning, thus increasing the training and inference
time. Optimal Brain Damage [6] also shows a more robust method of pruning. By
removing unimportant weights from a network, they claim they can get improved
generalization, require fewer training data points, and improve the speed of
learning. This technique takes second-derivative information to balance
between network complexity and training set error.

###  2.3 Knowledge Distillation

Knowledge Distillation (KD) was originally introduced as a training procedure
that aims to transfer the knowledge learned by a deeper network (Teacher) to
another relatively shallower one (Student), while maintaining the same level
of performance [10]. The motivation behind this technique is to reduce the
computational complexity of some operations, or compression of large networks,
such that devices with limited capacity (e.g. smartphones) could benefit from
the achievements of deep learning. In KD, the student network is trained not
with the original hard binary labels of the dataset, but instead with the soft
labels taken from the output probability of the teacher network. The student
model learns to reproduce the output of the teacher model which is used for
training. In [10], an ensemble of teacher networks was compressed into a
shallow student using the softmax output of the teachers. FitNets [22]
compress thin deep networks into wide shallow ones while preserving the
performance, and have also been used to learn full feature maps from the
teacher. There have been extensions of KD where students have been trained to
approximate Monte Carlo teacher [12], trained to represent knowledge in higher
neural layers [16], or using multiple networks of decreasing depth in between
[17] to transfer knowledge from teacher to assistant to student. Although KD
can compress very deep models into much shallower ones, it takes the largest
hit in performance among all the compression techniques due to its limited
representation capabilities.

###  2.4 Combined Methods

In [9], the authors use a combined pipeline of Pruning, Quantization, and
Huffman Coding. They first prune the model, then use weight sharing and post-
training quantization to further reduce the size of their model by 39-50x
compression. They manage to achieve this compression with no loss in the
accuracy of the model.

###  2.5 Other Methods

In [7], the authors demonstrate that there is significant redundancy in the
weights of large deep learning networks. With a few weight values for each
feature they claim it is possible to accurately predict the few unknown
weights. Additionally, they show that not only can the parameter values be
predicted, but many of them are simply not necessary. In this paper, they
train several different networks by learning only a fraction of the parameters
and predicting the remainder. In the best cases they are able to predict more
than 95% of the weights of a network without a decrease in accuracy.

In [1], the authors show that shallow fully connected models can learn the
same functions, regardless of how complex, that were learned by deeper models
and attain the same performance of the deep models. In some cases the shallow
neural networks learned these functions using the same number of total
parameters as the deeper model. They evaluate the method on the TIMIT phoneme
recognition task and successfully train shallow feed forward networks that
perform just as well as deep convolutional networks. They suggest that there
exist better algorithms for training shallow feedforward nets than those
currently available.

Other methods of network compression include using low-rank filters to
accelerate convolution [8], using transferred convolutional filters [3],
attention-based mechanisms to reduce computations [25], randomly dropping and
bypassing layers with identity function [11], etc.

##  3 Methods

In this section, we will introduce the methods we use for our experiments. We
introduce the baseline model, the compression techniques we use, and the
motivation for these selections.

###  3.1 Baseline Network Architecture

The proposed network architectures are shown in Figures 1 and 2.

Figure 1: Model Architectures for MNIST Figure 2: Model Architectures for
CIFAR10

We use two different base models for MNIST [15] and CIFAR-10 [14] datsets.
Because we wanted to test the effect of network compression on convolutional
layers and dense layers separately, both the models were created accordingly.
The base MNIST model has two convolutional layers of size 128 and 256,
followed by a hidden dense layer of size 100. Whereas, the CIFAR-10 base model
has 4 convolutional layers of sizes 128,128,256, and 256, followed by two
hidden layers of size 256. The MNIST and CFIAR-10 base models have aorund
1.55M and 1.37M trainable parameters respectively.

###  3.2 Knowledge Distillation

To test the efficacy of knowledge distillation, we tray both single-step and
multi-step distillation methods. In the former case, we train a student model
with the output generated from the teacher (baseline) model as labels, while
in the latter case, we introduce a teacher assistant model [18] as well. For
both the techniques, the student (compressed) model was kept the same. The
MNIST student has two convolutional layers of size 16 and 32, followed by
flattening and the output dense layer. There was no hidden dense layer in this
case. Same as the corresponding teacher, the CIFAR-10 student has 4
convolutional layers (size 64, 64, 128, and 128) and two dense hidden layers
of size 128. The MNIST and CIFAR-10 students have around 20.5K and 343.6K
parameters, leading to $>7500$% and $>400$% compression respectively, when
compared to their corresponding teacher models.

For testing the efficacy of multi-step distillation, we train a Teacher
Assistant model with labels generated by the teacher, and then train the
student using the teacher assistant labels. The Teacher Assistant model is
smaller than the Teacher but larger than the Student model for both cases. The
MNIST TA has around 176K parameters, and the CIFAR-10 Teacher Assistant has
around 1.11M parameters. For both single and multi-step distillation, we also
study the effect of distillation temperature on student accuracy.

###  3.3 Quantization

We examined two methods of quantization of our networks: Post-Training
Quantization and Quantization Aware Training. The first quantization technique
we explored was Post Training Quantization. Post Training Quantization is
performed on a pre-trained model, and it simply reduces the bit-precision of
the weights and activation functions within the network. For example, a
network which is normally stored with 32-bit float weights can be instead
stored with 8-bit weights, for a theoretical four times decrease in storage
size of the network [13].

The second quantization technique we used was Quantization Aware Training is a
method of training a neural network that is meant to help the model accuracy
downstream when later performing Post-Training Quantization. Quantization
Aware Training works by emulating lower precision steps in the forward pass,
while keeping the backward pass the same. This way, the optimizer can account
for the quantization error in the training phase of the network, and can
modify weights step-wise accordingly. Then, at the end of training the
network, we can store the weights with Post-Training Quantization and should
theoretically, get better validation scores than we would have without the
Quantization Aware Training [20].

###  3.4 Pruning

We compared two different pruning strategies on both of our base models -
global and local pruning. For each, we explored both constant sparsity and
polynomial decay with changing sparsity training schedules. All the pruning
strategies we used in our project was unstructured magnitude-based, which
involved iteratively pruning low magnitude, thus less important, weights
during training until the desired sparsity is reached.

Polynomial decay involves two different sparsity parameters - the initial
sparsity and the final desired sparsity. We trained the models by varying them
in three different ways - 0 to 80%, 0 to 50%, and 50 to 80% weight sparsity.
In constant decay, the sparsity remains the same as the model is fine tuned.
In this case, we compare three different pruning outcomes with 20%, 50%, and
80% sparsities.

Along with global pruning, we also experimented with partial local pruning. We
wanted to study the effects of pruning on CNN and dense parts of a network.
Hence, for each dataset baseline, we prune the convolutional layers and dense
layers separately with both polynomial decay and constant pruning strategies
as described before. For all the pruning methods, the pruned trained network
was fine tuned for two epochs.

###  3.5 Combined Techniques

We also explored the combination of different compression techniques to study
their efficacy. Primarily, we sequentially combine knowledge distillation with
pruning and quantization separately. For the first set of experiments, we
first compress the teacher using knowledge distillation and then apply pruning
and quantization separately. And for the second set of experiments, we first
prune and quantize the teacher, followed by training the student. Although the
second experiment does not compress the (final) student model and thus not
contribute to further compression, we were curious to study the effects of
teacher pruning on distillation.

##  4 Experimental Setup

In this section, we explain the details of how we carry out the experiments,
including the implementation and the experimental design.

###  4.1 Datasets

The datasets we used for experimentation on the model were the MNIST Digits
dataset and the CIFAR-10 image recognition dataset. The MNIST dataset consists
of 60000 training images of labeled handwritten digits, and 10000 testing
images. The digits are centered and size normalized, and each image is
represented by a 28x28 grid of pixels. There are a total of 10 labels, one for
each digit 0-9 [15].

The CIFAR-10 dataset consists of 60000 training images and 10000 test images.
This dataset also has ten labels: airplanes, automobiles, birds, cats, deer,
dogs, frogs, horses, ships, and trucks. The images here are larger than the
MNIST images. They are represented as color images in a 32x32x3 tensor [14].
We included this dataset as part of our experiment because we were interested
in how the additional dimension of color might affect the networks compression
ratios and accuracies.

###  4.2 Implementation

All experiments were done in Python using Tensorflow libraries and a Keras
framework. We used the tensorflow_model_optimization package to help with
quantization aware training. As there were no readily available packages for
knowledge distillation in TensorFlow, we had to create our own class for
distillation. We used Google Colab for our programming environment.

For the implementation of our algorithms, we used five epochs for our main
models and teacher models, and when we trained student models for knowledge
distillation, we used three epochs. We used an ADAM optimizer for all models
and it’s default learning rate of 0.001. We used the Sparse Categorical Cross
Entropy Loss as our loss function. For reproducability, we set a random seed
of 1234. Before training, we also normalized all pixel values which originally
given to a value between 0 and 1.

###  4.3 Evaluation Metrics

To evaluate our experiments, we looked at three main metrics. The first
metric, was the testing accuracy of each model. With each of the datasets, we
held out the test set as to not contaminate the model, and for training we
incorporated a 0.1 validation split. The testing accuracy can be measured as a
percent of the test set correctly predicted.

The second metric we looked at was the storage size of the model. To find the
storage size, we saved all models in a TFLite format and read the total number
of bytes that were saved in the saved model’s file path.

The final metric we observed was called the efficacy metric. This was simply
calculated as a ratio of the model accuracy to the size of the model. While
this last metric is not very intuitive, we wanted to include it as a way of
evaluating a compression technique with a mix of how its model’s accuracy
changes with the changes in size.

##  5 Results

In this section, we will go over the results of the different experiments we
ran.

###  5.1 Knowledge Distillation

Figure 3: Evaluating KD on MNIST:  The performance of the Teacher, TA, and
Student models on the MNIST data set [15] for different distillation
temperatures. Figure 4: Evaluating KD on CIFAR-10:  The performance of the
Teacher, TA, and Student models on the CIFAR-10 data set [14] for different
distillation temperatures.

Figures 3 shows the performance of the teacher, teacher assistant, and student
models (Sec. 3.2) on the MNIST [15] data set for distillation temperatures
$T=\\{1,2,5,10,20,30,40\\}$. We notice that distillation almost always leads
to an improvement for both the single step and multi-step cases. When compared
between the two, we observe that introducing TA improves the performance of
the student for lower distillation temperatures, whereas for high
temperatures, the performance remains almost the same. Furthermore, the
performance of TA is slightly better than the student models, though the
student model contains only about $11$% number of parameters compared to TA.
This points to the high compression capabilities of knowledge distillation.
The optimal distillation temperature seems to be around 15–30.

Similar experiments on the CIFAR-10 models (Fig. 4) shows a slightly different
trend. Here too, for most cases, the distilled models perform better than the
teacher. But, unlike the MNIST case, here the TA leads to better performance
for higher distillation temperatures, and for lower temperatures, TA does no
help. For both the plots, we notice the performance to be higher in the middle
and lower for low and high temperatures. Low T results in harder labels and
hence lower accuracy, whereas high T leads to softer labels but with larger
entropy, which may have caused the lower accuracy for high temperatures.

###  5.2 Pruning

Global Pruning:  Table 1 shows the performance of the unstructured pruning for
both the polynomial decay and constant sparsity schedules. The percentage
improvements over the original baselines have been presented. We observe that
performance increases with increase in sparsity. For these set of experiments,
the fine tuned pruned models always performed better than the original
baselines. Similar to quantization, we noticed a huge improvement in model
size reduction, although the accuracy might have slightly changed.

| MNIST | CIFAR-10
---|---|---
Model | Acc | Size(MB) | Acc | Size(MB)
PD 50%–80% | +0.48 | -68 | +4.57 | -68
PD 0%–80% | +0.77 | -68 | +3.27 | -68
PD 50%–80% | +0.62 | -37 | +1.68 | -37
CS 20% | +0.54 | -11.5 | +1.28 | -11.4
CS 50% | +0.65 | -37 | +2.83 | -37
CS 80% | +0.86 | -68 | +3.44 | -68
Table 1: Results of global unstructured pruning on the two baselines. All
numbers show the percentage improvement over the original baseline models.

Local Partial Pruning:  We experimented with pruning the dense layers and
convolutional layers separately. The results of local partial pruning on the
CIFAR-10 baseline is presented in Table 2. We notice a slightly better
performance when pruning the convolutional layers, which might be due to the
innate higher sparsity in them compared to the fully connected layers. Also,
comparing to Table 1, we observe that, when compared to global pruning, local
pruning is generally less advantageous in terms of compression efficacy
(performance/size).

| Dense Layers | Conv Layers
---|---|---
Model | Acc | Size(MB) | Acc | Size(MB)
PD 50%–80% | +3.8 | -68 | +3.9 | -68
PD 0%–80% | +3.8 | -68 | +4.2 | -68
PD 50%–80% | +2.6 | -37 | +2.9 | -37
CS 20% | +1.2 | -11.5 | +1.03 | -11.4
CS 50% | +2.7 | -37 | +1.92 | -37
CS 80% | +3.6 | -68 | +4.4 | -68
Table 2: Results of local unstructured pruning on the dense layers and
convolutional layers separately for the CIFAR-10 baseline. All numbers show
the percentage improvement over the original baseline models.

###  5.3 Quantization

| MNIST | CIFAR-10
---|---|---
Model | Accuracy | Size (MB) | Accuracy | Size (MB)
Original Model | 97.74 | 5.923 | 70.42 | 5.22
Quantization Aware Training | 98.12 | 1.494 | 69.55 | 1.332
Model + 8-bit Post Training Quantization | 97.75 | 1.494 | 70.36 | 1.326
Model + 16-bit Post Training Quantization | 97.74 | 2.964 | 70.42 | 2.614
Distilled Model (Teacher $\rightarrow$ Student) | 97.61 | 0.081 | 70.62 | 1.315
Quantization Aware Teacher $\rightarrow$ Distilled Student | 97.61 | 0.081 | 70.62 | 1.315
Teacher $\rightarrow$ Quantization Aware Student | 97.55 | 0.024 | 67.93 | 0.345
Teacher $\rightarrow$ Student + 8-bit Post Training Quantization | 97.62 | 0.023 | 70.59 | 0.341
Teacher $\rightarrow$ Student + 16-bit Post Training Quantization | 97.61 | 0.042 | 70.62 | 0.662
Table 3: Results of all Quantization Experiments

The results of the quantization experiments are shown in Table 3. What we can
look at first is the accuracies and sizes of our baseline models- in the MNIST
case, the baseline model performs well with an accuracy of 97.74, with a size
of 5.923 MB. In the CIFAR-10 case, the baseline model has an accuracy of 70.42
and a storage size of 5.22 MB.

The first observation we can take notice of is the results of the Quantization
Aware Training experiment. The accuracy in the case of MNIST actually improved
from the original model, while the accuracy of CIFAR-10 went down, but not
significantly. Overall, the accuracies here did not change much, but the size
of the model decreased significantly. When we applied the Quantization Aware
Training, we also apply a 8-bit Post Training Quantization. The size of the
model thus decreases to 1.494 MB in the case of MNIST, and it decreases to
1.332 MB in the case of CIFAR-10. These decreases represent an approximate
compression ratio of 4x.

The next thing we can notice from the table are the the changes in the
original model when just Post Training Quantization is applied. We can notice
that the accuracies of the quantized models in MNIST remain almost exactly the
same as the original model (97.75 and 97.74 in the 8-bit and the 16-bit
models, respectively, vs 97.74 in the original model), and that the accuracies
of the CIFAR-10 16-bit quantized model is exactly equal to the accuracy of the
CIFAR-10 original model. We can, however, see a very slight decrease in
accuracy of the the CIFAR-10 8-bit model, but it is not very significant. We
can also observe the storage size differences in all of these models. As
expected, the 8-bit models empirically show an approximate compression ratio
of 4x from the original 32-bit model, and the 16-bit models empirically show
an approximate compression ratio of 2x.

Another interesting observation is the comparison between the Quantization
Aware Trained network and the Post-Trained models. It is interesting to note
that the Quantization Aware Trained model has a higher accuracy in the MNIST
case, which is what is expected from having the Quantization Aware forward
passes incorporated in the optimizer. However, we don’t see that same pattern
in the CIFAR-10 case, where the Quantization Aware accuracy is actually less
than both of the Post-Trained models’ accuracies. The model sizes remain
almost exactly the same, which is expected when they both have the same total
number of parameters and the same bit-precision storage.

###  5.4 Combined Techniques

KD + Pruning:  Table 4 shows the comparison between the individual techniques
with the combined one. We notice that, although the accuracy of the prune
student decreased after pruning, it was accompanied with a huge reduction in
size. So, for simple models and data sets, combining the two methods may lead
to huge improvements.

| MNIST | CIFAR-10
---|---|---
Model | Accuracy | Size (MB) | Accuracy | Size (MB)
Original (Teacher) | 97.83 | 5.76 | 69.68 | 5.08
Teacher $\rightarrow$ Pruning | 98.29 | 1.84 | 72.87 | 1.62
Teacher $\rightarrow$ Student | 97.75 | 0.08 | 70.34 | 1.28
Teacher $\rightarrow$ Pruning $\rightarrow$ Student | 97.75 | 0.08 | 71.40 | 1.28
Teacher $\rightarrow$ Student $\rightarrow$ Pruning | 96.94 | 0.03 | 72.16 | 0.42
Table 4: Results of experiments combining knowledge distillation and global
unstructured pruning on the two baselines.

KD + Quantization:  From Table 3, we observe that the performance of the
Distilled Model (Teacher to Student) is exactly equal to the Distilled Model
(Quantization Aware Trained Teacher to Student). Essentially, this shows that
there was no difference in that distillation process when we gave it the
original teacher model or the quantization aware teacher model. What we can
see, however, is the large difference in the sizes of the student model when
compared to the teacher model, and the very small difference in the accuracy.
In the MNIST case, the storage size of the model goes from 5.923 MB to 0.081
MB, which is a compression ratio of almost 75x. This shows that a much smaller
network can be learned from a significantly larger network.

Furthermore, we can sequentially add on post training quantization to the
distillation process. This even further decreases the size of the model by
another 4x, without affecting the accuracy. In the MNIST case, the final
compression ratio after going through both distillation and 8-bit post-
training quantization is more than 250x. This is an enormous decrease in the
size of the model, with minuscule change in the accuracy of the model.

##  6 Conclusions

Performance of all of the methods have been summarized in Table 5.

Model | MNIST | CIFAR-10
---|---|---
Original (Teacher) | 16.84 | 13.72
Knowledge Distillation | 1219.13 | 54.87
KD with Teaching Assitant | 1220.13 | 54.66
Global Pruning | 53.34 | 44.98
Local Pruning (Dense) | 52.66 | 44.38
Local Pruning (Conv) | 53.21 | 44.61
Quant. Aware Training | 65.68 | 52.21
KD + Pruning | 3258.33 | 171.81
KD + 8-bit Post-train Quant. | 4244.35 | 207.01
Table 5: Performance efficacy (Accuracy/Size) of all the methods compared in
this paper when tested on both the datasets.

In general, we noticed that for almost all techniques, accuracy of the models
improved after compression. This might have been due to our models being
larger than the easy and small data sets we tested them on. Although, our
experiments points towards excellent performance efficacy for the compressed
models, the results might change when tested on much deeper networks and on
large hard data sets.

The other interesting conclusion we can draw from the knowledge distillation
experiments is the advantage of introducing teacher assistant models when the
gap between the student and the teacher is large. Although multi-step
distillation helps in these cases, it will be interesting to see the
performance for deeper models and for more than one steps in between teacher
and student. Again, our conclusion is very rudimentary and these need to be
explored further.

Finally, from our experiments, we do notice the huge benefit of combining
distillation with pruning and quantization. We can achieve highly compressed
models without much change in performance.

As part of future work, we would like to explore harder data sets with many
more classes, test on deeper models, and experiment on combining other modern
compression techniques. We tried to explore the very basic compression
techniques in a simple manner, but many more sophisticated methodologies have
been explored in the literature. Another part of potential future work could
be to employ these algorithms on commonly used pre-existing networks, such as
AlexNet, or even on much larger networks where network compression may
actually be a problem, like DenseNet. Furthermore, we would like to experiment
with even smaller student models in the future, to see just how compact a
network can get while still performing well. It would also be interesting to
work on an algorithm for recursively removing layers of a deep network in a
manner analogous to backwards regression. A comprehensive and thorough
analysis of all would be highly desirable. This project serves as a stepping
stone in that direction.

## References

  * [1] Lei Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?, 2014.
  * [2] Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen.  Compressing neural networks with the hashing trick, 2015.
  * [3] Taco S. Cohen and Max Welling.  Group equivariant convolutional networks.  Proceedings of the International Conference on Machine Learning (ICML), 2016.
  * [4] Matthieu Courbariaux, Jean-Pierre David, and Yoshua Bengio.  Low precision storage for deep learning.  2015\.
  * [5] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, 2016.
  * [6] Yann Le Cun, John S. Denker, and Sara A. Sol1a.  Optimal brain damage, 1990.
  * [7] Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas.  Predicting parameters in deep learning, 2014.
  * [8] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus.  Exploiting linear structure within convolutional networks for efficient evaluation.  NIPS’14, page 1269–1277, 2014.
  * [9] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2015.
  * [10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [11] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.  Deep networks with stochastic depth, 2016.
  * [12] Anoop Korattikara, Vivek Rathod, Kevin Murphy, and Max Welling.  Bayesian dark knowledge, 2015.
  * [13] Raghuraman Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepapep?, 2018.
  * [14] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).
  * [15] Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.
  * [16] Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang.  Face model compression by distilling knowledge from neurons.  In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 3560–3566. AAAI Press, 2016.
  * [17] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher.  CoRR, abs/1902.03393, 2019.
  * [18] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant, 2019.
  * [19] James O’ Neill.  An overview of neural network compression, 2020.
  * [20] TensorFlow Model Optimization.  Quantization aware training, 2021.
  * [21] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean.  Carbon emissions and large neural network training.  2021\.
  * [22] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets, 2014.
  * [23] Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao.  Improving the speed of neural networks on cpus.  In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011.
  * [24] Atlas Wang.  Resource efficient learning, 2021.
  * [25] Di Wu, Lionel Pigou, Pieter-Jan Kindermans, Nam Do-Hoang Le, Ling Shao, Joni Dambre, and Jean-Marc Odobez.  Deep dynamic neural networks for multimodal gesture segmentation and recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1583–1597, 2016.