# Quantum-Acoustical Drude Peak Shift

J. Keski-Rahkonen<EMAIL_ADDRESS>Department of Physics,
Harvard University, Harvard University, Cambridge, MA 02138, USA  Department
of Chemistry and Chemical Biology, Harvard University, Cambridge, MA 02138,
USA  X.-Y. Ouyang  Yuanpei College, Peking University, No.5 Yiheyuan Rd,
Beijing 100871, China  School of Physics, Peking University, No.5 Yiheyuan Rd,
Beijing 100871, China  S. Yuan  School of Physics, Peking University, No.5
Yiheyuan Rd, Beijing 100871, China  A.M. Graf  Harvard John A. Paulson School
of Engineering and Applied Sciences, Harvard, Cambridge, Massachusetts 02138,
USA  A. Aydin  Department of Physics, Harvard University, Harvard University,
Cambridge, MA 02138, USA  Faculty of Engineering and Natural Sciences, Sabanci
University, 34956 Tuzla, Istanbul, Turkey  E.J. Heller  Department of Physics,
Harvard University, Harvard University, Cambridge, MA 02138, USA  Department
of Chemistry and Chemical Biology, Harvard University, Cambridge, MA 02138,
USA

###### Abstract

Quantum acoustics – a recently developed framework parallel to quantum optics
– establishes a nonperturbative and coherent treatment of the electron-phonon
interaction in real space. The quantum-acoustical representation reveals a
displaced Drude peak hid ing in plain sight within the venerable Fröhlich
model: the optical conductivity exhibits a finite frequency maximum in the
far-infrared range and the d.c. conductivity is suppressed. Our results
elucidate the origin of the high-temperature absorption peaks in strange or
bad metals, revealing that dynamical lattice disorder steers the system
towards a non-Drude behavior.

††preprint: APS/123-QED

Stretching over four decades, an intensive theoretical pursuit has
concentrated on finding an all-embracing explanation for a plethora of
puzzling phenomena which have been colloquially labeled as “bad” or “strange”.
These kinds of “bizarre” materials seem to defy the traditional paradigms for
electron behavior [1] in metals. Mysteries abound, such as high-temperature
superconductivity beyond the grasp of the BCS theory [2, 3], the paradoxical
existence of pseudogaps [4, 5, 6, 7, 8] and charge density waves [9, 10, 11,
12], the violation of the Mott-Ioffle-Regel (MIR) limit [13, 14], not to
mention the major dilemma of linear-in-temperature resistivity over a wide
temperature range (see, e.g., Refs. [15, 16, 17, 18, 19, 20, 21]) at the
mysterious but ubiquitous Planckian bound [22]. This list of theoretical
challenges also includes the elusive emergence of displaced Drude peaks (DDP)
[23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37]: a prominent
absorption peak located typically in the infrared range, signaling a breakdown
of the conventional Drude picture. This Letter reveals the origin of this
phenomenon as the result of strong electron-phonon interaction if treated
correctly as nonperturbative and coherent.

Figure 1: Quantum acoustics. Illustration of the coherent state lattice
vibrations at a certain temperature. Electrons experience a spatially
continuous internal field formed by the thermal acoustic distortions. While
undergoing quasi-elastic scattering events, like due to impurities, electrons
can also be incipiently trapped by valleys of slowly undulating and
propagating deformation potential when their kinetic energy is comparable to
the fluctuations of the deformation potential.

A putative culprit for the observed Drude shift in the optical conductivity
peak has been suggested to be some as yet unidentified dynamical disorder that
generates evanescent localization, thus hampering but not precluding charge
carrier diffusion [38, 39, 40, 41, 42]; Consequently, the zero-frequency
conductivity does not vanish completely, but it is strongly suppressed,
favoring the DDP phenomenology. This scenario contrasts with alternative
points of view, like the common arguments resorting to collective modes [43,
44] or to strong electron-electron correlations [45, 46].

Here we show that a morphing potential landscape of hills and valleys stemming
from thermal lattice vibrations by the Fröhlich Hamiltonian (see Ref. [47]),
as illustrated in Fig. 1 is the sought-after sea of “slowly moving bosonic
impurities” [41], or the cryptic “self-induced randomness” [42, 48]. In a
broader milieu, the subtle interplay between the Anderson localization and
lattice vibrations has been encountered in a wide class of random metal alloys
and other degenerate disordered systems. In fact, the intricate game of being
localized or not was identified early on by Gogolin et al. [49, 50] and
Thouless [51], even pondered by Anderson himself [52, 53].

The random fluctuations introduced by lattice motion slowly but surely
scramble the quantum interference required for localization of the electronic
state, resulting in _transient_ localization (for capturing the essential
aspects of this phenomenon, see, e.g., Refs. [54]), which has lately been of
interest in the context of crystalline organic semiconductors [55, 56] and
halide perovskites [57].

The quantum-acoustic route to linear and universal resistivity in strange
metals [58] has opened up a new path unrelated to quantum criticality [59,
60], and not relying on (strong) electron-electron interaction [48, 61],
instead starting with the standard Frölich Hamiltonian.

Following the path paved in Refs. [47, 58], here we demonstrate the formation
of a DDP due to the electrons interacting with fluctuating lattice degrees of
freedom. We go further by showing that this mechanism gives rise to a
temperature dependence of spectral features in agreement with experimental DDP
observations in strange metals.

More specifically, we consider the following Fröhlich Hamiltonian [62, 63]
describing the lowest-order (linear) lattice-electron coupling [64]:

$\mathcal{H}_{\textrm{F}}=\sum_{\mathbf{p}}\varepsilon_{\mathbf{p}}c_{\mathbf{p}}c_{\mathbf{p}}^{\dagger}+\sum_{\mathbf{q}}\hbar\omega_{\mathbf{q}}a_{\mathbf{q}}^{\dagger}a_{\mathbf{q}}+\sum_{\mathbf{p}\mathbf{q}}g_{\mathbf{q}}c_{\mathbf{p}+\mathbf{q}}^{\dagger}c_{\mathbf{p}}\Big{(}a_{\mathbf{q}}+a_{\mathbf{-q}}^{\dagger}\Big{)}$
(1)

where $c_{\mathbf{p}}$ $(c_{\mathbf{p}}^{\dagger})$ is the creation
(annihilation) operator for electrons with momentum $\mathbf{p}$ and energy
$\varepsilon_{\mathbf{p}}$; whereas $a_{\mathbf{q}}$
$(a_{\mathbf{q}}^{\dagger})$ is the creation (annihilation) operator for
longitudinal acoustic phonons of wave vector $\mathbf{q}$ and energy
$\hbar\omega_{\mathbf{q}}$, respectively. The electron-phonon interaction is
defined by its Fourier components $g_{\mathbf{q}}$. By following the steps of
the recently established coherent state formalism in Ref. [47], the
Hamiltonian gives rise to an undulating and propagating potential landscape
(see Appendix A):

$\displaystyle V_{D}(\mathbf{r},t)$
$\displaystyle=\sum_{\begin{subarray}{c}\mathbf{q}\end{subarray}}^{|\mathbf{q}|\leq
q_{D}}g_{\mathbf{q}}\sqrt{\langle
n_{\mathbf{q}}\rangle_{\textrm{th}}}\cos(\mathbf{q}\cdot\mathbf{r}-\omega_{\mathbf{q}}t+\varphi_{\mathbf{q}})$
(2)

where $q_{D}$ is Debye wavenumber, $\mathbf{r}$ is continuous position,
$\varphi_{\mathbf{q}}=\textrm{arg}(\alpha_{\mathbf{q}})$ is the (random) phase
of a coherent state $|\alpha_{\mathbf{q}}\rangle$, and the mode population is
determined by $\langle n_{\mathbf{q}}\rangle_{\textrm{th}}$.

The coherent state picture developed here is the dual partner of the
traditional number state description of electron-lattice dynamics, so widely
successful for describing electron resistivity [65]. In addition to recovering
the results of the conventional Bloch-Grüneisen theory [66, 67], the coherent
state representation extends beyond perturbation theory (see Ref. [47] for a
more detailed discussion). The coherent state limit of quantum acoustics
reveals a real-space, time-dependent description of electron-lattice
interaction. A very similar notion was introduced in 1957 by Hanbury Brown and
Twiss for the vector potential of a blackbody field [68], with the essential
difference of missing the ultraviolet cut-off, i.e., the Debye wavenumber in
the definition of our deformation potential originating from the minimal
lattice spacing.

This follows the analogous quantum optics pioneered by Glauber [69], a
neglected wave perspective for lattice vibrations – _quantum acoustics_.
Bardeen and Shockley, in the 1950’s, regarded dynamical lattice distortions in
nonpolar semiconductors [70, 71], and it seemed they would have been happy
with a coherent state description, but the theory was subsumed by a number
state perspective.

Within the present deformation potential framework, an electron undergoes
quasielastic, coherence-preserving scattering events when roaming through the
slowly altering potential landscape of hills and valleys. In this work, we
focus on three prototypical compounds classified as strange/bad metals, namely
LSCO, Bi2212 and Sr3Ru2O7 whose material properties are given in Appendix B.
However, we want to stress that the physics we find below transcends the
material-specific constraints: In general, dynamical disorder, caused by
lattice vibrations here, temporarily confines electrons to nest in its
instantaneous potential wells (see Fig. 1), hallmarking of transient
localization dynamics, and consequently results in the buildup of a DDP.

Figure 2: Transient localized states. A selection of eigenstates (red color
scheme) near the Fermi level is shown for the considered prototype materials
at four temperatures, where the gray scale represents the corresponding frozen
deformation potential. An increasing temperature leads to more spatially
confined states that are linked to fugitive, Anderson-localized states in
transient dynamics. Figure 3: Frozen versus dynamical quantum acoustic
vibration field. The upper and lower panels display the optical conductivity
for the three materials at different temperatures, resolved within the static
and dynamical potential landscapes averaged over 100 and 10 realizations,
respectively. The back dash line marks the Debye frequency of the given
material below which the frozen potential assumption breaks down. A key
dynamical effect is the saturation of conductivity in the regime
$\omega\lesssim\omega_{D}$, instead of the suppression evident in the static
case. However, regardless of the deformation potential dynamics and the chosen
material, the optical conductivity peak shifts from the Drude-peak edict of
situating at $\omega=0$ to higher energies and broadens as the temperature
increases.

It appears that, regarding photoabsorption, electrons are insensitive to the
lattice dynamics when $\omega\gg w_{D}$. In other words, the deformation
landscape appears as if it is stationary for an electron, which is confirmed
by our results below.

Motivated by this, we initially freeze the potential and study its transitory
electronic eigenstates. Since the allowed transitions take place near the
Fermi level, we are allowed to focus on the states lying within the stack of
$\varepsilon_{F}\pm 3k_{\textrm{B}}T$ (see Appendix C.1). Fig 2 shows examples
of eigenstates for the three materials at different temperatures along with
the profile of the deformation potential. Despite the diversity among the
physical settings, all three materials share similar qualitative
characteristics: Instead of being spatially extended, as observed at lower
temperatures ($E_{F}/V_{\textrm{rms}}\gtrsim 1$), the relevant states appear
to be localized in the dips of the potential at temperatures
$E_{F}/V_{\textrm{rms}}\lesssim 1$ where the potential fluctuation
$V_{\textrm{rms}}$ is comparable to Fermi energy $E_{F}$. These eigenstates of
the frozen potential relate to the localized states associated with the
transient dynamics.

Subsequently, we compute the corresponding optical conductivity
$\sigma(\hbar\omega)$ within the conventional Kubo formalism employing the
numerically resolved eigenstates, detailed in Appendix C.1. In the upper panel
of Fig 3, we present the optical conductivity at various temperatures for the
three chosen materials, averaged over an ensemble of 100 random realizations
of the deformation potential. With increasing temperature, the optical
conductivity evolves from the Drude-peak behavior of having a sharp maximum
value located at $\omega=0$ into a displaced peak: the maximum conductivity
point steadily shifts towards higher energies $\hbar\omega$ and the
conductivity peak profile broadens.

In addition, we show the temperature dependence of the peak locations and
their width in Fig. 4. We determine the peak location
$\hbar\omega_{\textrm{p}}$ as the energy at which the optical conductivity
$\sigma(\hbar\omega)$ reaches its maximum. The peak width
$\hbar\Delta\omega_{\textrm{p}}$ is then defined in a similar manner as in
Ref. [72]: the distance between the maximum and the optical conductivity point
in the high-energy tail where the height of the maximum is dropped by 50%.
Fig. 4 further confirms and quantifies the migration and broadening of the DDP
with increasing temperature present in Fig. 3.

To further validate the frozen potential results above, we expand our DDP
analysis by computing the optical conductivity while considering the temporal
evolution of the deformation potential. Nonetheless, we can still determine
the conductivity $\sigma(\hbar\omega)$ by utilizing the Kubo formalism, as
explicated in Appendix C.2. In short, we take advantage of the already defined
frozen deformation potentials as initial conditions, and let it thereafter
unfold according to Eq. 2.

Figure 4: Position and width of the Drude peak. The panels show the mean
location (left) and width (right) of the optical conductivity peak as a
function of temperature for the studied materials, averaged over 100 (static)
and 10 (dynamic) deformation potential realizations, with error bars
representing the standard deviation within the given ensemble. Specifically,
as the temperature increases, there is a generic upward shift in the Drude
peaks towards higher energies (upper panel), accompanied by a broadening of
the peaks (lower panel). Colored dashed curves show the fittings for the DDP
location estimated with the assumption that it is mostly determined by the
strength of the deformation potential ($\hbar\omega_{p}\propto
V_{\textrm{rms}}$). Furthermore, the width of the DDPs live near to Planckian
bound ($\hbar\Delta\omega_{p}\sim k_{B}T$), which is indicated by the black
dashed line.

The lower panel of Fig. 3 shows the optical conductivity spectrum of the three
materials at different temperatures in the case of a dynamical potential
field, averaged over 10 realizations of the distorted potential landscape. As
expected, the dynamical conductivity spectrum deviates from the frozen
potential prediction when $\omega\lesssim\omega_{D}$ is indicated by the black
dash line in Fig. 3. Instead of strong suppression near to the d.c.
conductivity as in the static landscape situation, we observe a saturation of
the optical conductivity within an energy window of the order of
$0.1\,\textrm{eV}$ near the zero frequency. This can be interpreted as the
reversed adiabatic approximation where the external electric field of
frequency $\omega\ll\omega_{D}$ is a slow varying degree of freedom and thus
is roughly static compared to the fluctuations of the lattice, yielding
virtually the same conductivity as the d.c. conductivity and manifesting as a
conductivity plateau below the Debye frequency. Nevertheless, there is still a
generic trend similar to the frozen potential approximation: the higher
temperature yields a more substantial DDP, suggesting that the transiently
localized states are at play in both cases. This tendency is also evident in
Fig. 4 where the increase in temperature moves the DDP to higher absorption
frequencies while broadening the peak at the same time.

The physical picture behind the observed DDP evolution is that the increase in
temperature has a two-fold effect. First, it yields stronger spatially
localized, transient electronic states at the Fermi energy (shift to higher
frequencies); electrons either residing in local potential wells (frozen) or
nesting in instantaneous potential pockets (dynamic). These local wells or
nests become more energetically confining as the deformation potential
strengthens with the rising temperature. As a result, the location of the DDP
roughly scales like $\hbar\omega_{\textrm{p}}\sim V_{\textrm{rms}}$, which
defines the fitting illustrated by the colored dashed curves in the upper
panel of Fig. 4. In general, the peak location migrates like
$\hbar\omega_{\textrm{p}}\sim(k_{B}T)^{3/2}$ at low temperatures $T\ll T_{D}$
and $\hbar\omega_{\textrm{p}}\sim(k_{B}T)^{1/2}$ at high temperatures $T\gg
T_{D}$ . In addition, the presence of eigenstate localization caused by the
frozen lattice disorder is known to result in band tails in the density of
states [47] that has later shown to persist even under quantum-acoustical
lattice dynamics [73].

On the other hand, a higher temperature permits a wider energy window for
electronic transitions to occur (the broadening of the peak). Whereas the
transition element between the (momentarily) localized states dictates the
location of DDP, the width is instead determined by the broadening function
characterizing the energetically allowed transition. As indicated by the black
dashed line in the lower panel of Fig. 4, the widths of the DDP behave roughly
as $\hbar\Delta\omega_{\textrm{p}}\sim k_{\textrm{B}}T$ that is interestingly
more accurate in the case of the dynamical potential landscape. The quantum-
acoustical DDP is thus intimately connected to the ambiguous Planckian
timescale $\hbar/k_{\textrm{B}}T$ that underpins the linear-in-temperature
resistivity exhibited by numerous families of bad and strange metals (for
comparison, see Ref. [72]). In addition, this observed correlation between the
width of the DDP and Planckian behavior supports the prospect of the near-
universal transport by transient dynamics reported in Ref. [47, 58].

In the quantum-acoustic DDP scheme, there are no extrinsic sources, such as
defects or impurities, which could also generate or enhance a shift in optical
conductivity (see, e.g., Refs. [74, 75, 76]). In other words, the disorder at
the origin of our DDP is _self-generated_ , arising from the existence of
thermally fluctuating lattice degrees of freedom that significantly affect the
charge carrier dynamics. In particular, our DDP gives a unique temperature-
dependent fingerprint, clearly distinguishing it from an impurity-induced DDP.
Furthermore, the transient dynamics driving the birth of an acoustical DPP
resides within a dynamical regime of nonperturbative and coherent electron-
lattice motion, thus lying outside the reach of the conventional perturbative
or Boltzmann transport methods (see Ref. [47]). In fact, many bad and strange
metals are on the verge of a transient localization and/or have strong
electron-phonon coupling [46, 1].

Moreover, alongside the DDP formation and Planckian resistivity, the
deformation potential perspective offers a natural pathway for charge carriers
in strange metals to cross the MIR bound with impunity at high temperatures
[13, 14], as already asserted in Ref. [47]. We explicitly demonstrated the
violation of MIR limit with quantum acoustics in another paper [58]. Our
approach also carries the potential to enlighten the perplexing phenomenon of
pseudogaps [4, 5, 6, 7, 8] and charge density waves [9, 10, 11, 12]. In
particular, we see that when electron motion strongly couples and synchronizes
with low-energy lattice vibration modes, it creates a favorable environment
for incommensurate charge density order. Likewise, a similar resonance
promoted by the deformation potential could result in temperature-dependent
pseudogaps, i.e., a substantial suppression in the density of the low-energy
excitations, which eventually melt away leaving the pseudogap phase regime. We
aim to study these considerations in future research.

In conclusion, we have introduced the phenomenon of quantum-acoustical Drude
peak displacement, which involves the temperature-dependent shift and
broadening of the optical conductivity peak to finite frequencies,
demonstrated here for three archetypal strange metals. Overall, the coherent
state picture of lattice vibrations, which has always been at one’s disposal
but not utilized, provides a fresh perspective on the investigation of the
mysteries of bad and strange metals. The manifested shift in perspective
simply comes from the coherent state limit of quantum acoustics.

###### Acknowledgements.

We are thankful for the useful discussions with D. Kim, B. Halperin, S. Das
Sarma, J. H. Miller Jr, R. L. Greene, R. Lobo and A. P. Mackenzie.
Furthermore, J.K.-R. thanks the Emil Aaltonen Foundation, Vaisala Foundation,
and the Oskar Huttunen Foundation, and A. M. G. thanks the Harvard Quantum
Initiative for financial support.

##  Appendix A Deformation potential

Here we provide a brief overview of the coherent state formalism and the
origin of the deformation potential; for a comprehensive discussion about the
topic, see Ref. [47].

In general, a lattice deformation is treated in terms of a quantum
longitudinal displacement field $\hat{\mathbf{u}}(\mathbf{r},t)$. The
longitudinal displacement of an atom at a position $\mathbf{r}$ from its
equilibrium position is

$\displaystyle\begin{aligned}
\hat{\mathbf{u}}(\mathbf{r},t)&=-i\sum_{\mathbf{q}}\hat{\mathbf{q}}\sqrt{\frac{\hbar}{2\rho\mathcal{V}\omega_{\mathbf{q}}}}\\\
&\qquad\qquad\quad\times\left(a_{\mathbf{q}}e^{-i\omega_{\mathbf{q}}t}-a_{-\mathbf{q}}^{\dagger}e^{i\omega_{\mathbf{q}}t}\right)e^{i\mathbf{q}\cdot\mathbf{r}},\end{aligned}$

where $\mathbf{q}$ is wave vector of a normal mode, $\omega_{\mathbf{q}}$ is
phonon frequency, $\rho$ is mass density of the lattice, $\mathcal{V}$ is
volume (area in 2D), and $t$ is time. We omit writing the polarization vector
and phonon branch indices in the subscripts since we will only deal with the
longitudinal acoustic modes of the lattice vibrations, the main scattering
mechanism for the charge carriers. Subsequently, we define the deformation
potential as the first-order correction in the expansion of band energy due to
atomic displacements in the following way

$\begin{split}\hat{V}_{D}(\mathbf{r},t)=&E_{d}\nabla\cdot\hat{\mathbf{u}}(\mathbf{r},t)\\\
=&\sum_{\textbf{q}}^{|\mathbf{q}|\leq
q_{D}}E_{d}\sqrt{\frac{\hbar}{2\rho\mathcal{V}\omega_{\mathbf{q}}}}|\mathbf{q}|(a_{\mathbf{q}}+a_{\mathbf{-q}}^{\dagger})e^{i\textbf{q}\cdot\textbf{r}}\\\
=&\sum_{\textbf{q}}^{|\mathbf{q}|\leq
q_{D}}g_{\mathbf{q}}(a_{\mathbf{q}}+a_{\mathbf{-q}}^{\dagger})e^{i\mathbf{q}\cdot\mathbf{r}},\end{split}$

where the deformation potential constant $E_{d}$ characterizes the coupling
between electrons and the lattice, and the mode summation is restricted by the
Debye wavenumber $q_{D}$. For convenience, we convert the material constant
into the mode-depend co-factor of

$g_{\mathbf{q}}=E_{d}\sqrt{\frac{\hbar}{2\rho\mathcal{V}\omega_{\mathbf{q}}}}|\mathbf{q}|=E_{d}\sqrt{\frac{2\hbar|\mathbf{q}|}{\rho\mathcal{V}v_{s}}},$

where the latter form is achieved by assuming the linear dispersion
$\omega_{\mathbf{q}}=v_{s}|\mathbf{q}|$, coupling given by the speed of sound
$v_{s}$. Thus, the electron-phonon interaction term corresponds to the
Fröhlich Hamiltonian in the main text.

The next ingredient we introduce is the coherent state picture. Within this
framework, each normal mode of lattice vibration with a wave vector
$\mathbf{q}$ is associated with a coherent state
$|\alpha_{\mathbf{q}}\rangle$. By employing the independence of normal modes,
entire lattice vibrations can be expressed as the product state of the
coherent states $|\alpha_{\mathbf{q}}\rangle$ as a single multimode coherent
state of

$\displaystyle\begin{aligned}
|\bm{\alpha}\rangle=\prod_{\mathbf{q}}|\alpha_{\mathbf{q}}\rangle,\end{aligned}$

whose expectation value defines the deformation potential

$\displaystyle\begin{aligned}
V_{D}(\mathbf{r},t)&=\langle\bm{\alpha}|\hat{V}_{D}(\mathbf{r},t)|\bm{\alpha}\rangle\\\
&=\sum_{\mathbf{q}}^{|\mathbf{q}|\leq
q_{D}}g_{\mathbf{q}}(\alpha_{\mathbf{q}}e^{-i\omega_{\mathbf{q}}t}+\alpha_{-\mathbf{q}}^{*}e^{i\omega_{\mathbf{q}}t})e^{i\mathbf{q}\cdot\mathbf{r}}.\end{aligned}$

This quasi-classical field of lattice vibrations corresponds to the
deformation potential presented in the main text by considering a thermal
coherent state

$\alpha_{\mathbf{q}}=\sqrt{\langle
n_{\mathbf{q}}\rangle_{\textrm{th}}}\exp(i\varphi_{\mathbf{q}}),$

where $\sqrt{\langle
n_{\mathbf{q}}\rangle_{\textrm{th}}}=|\alpha_{\mathbf{q}}|$ is the thermal
amplitude given by the Bose-Einstein distribution and
$\varphi_{\mathbf{q}}=\arg(\alpha_{\mathbf{q}})$ is the random phase of the
coherent state determining the initial conditions.

The deformation potential in itself is a curious mathematical object. First,
it is homogeneously random in space and in time, meaning the probability
distribution of deformation potential is independent on a position
$\mathbf{r}$ or time t given that the phases $\varphi_{\mathbf{q}}$ are random
variables. Thus, each reasonably large spatiotemporal section of the potential
is statistically indistinguishable from another. Second, although the
deformation potential averages to zero, its root-mean-square identifies the
strength of lattice disorder, growing in temperature as

$V_{\textrm{rms}}^{2}=\frac{2E_{d}^{2}\hbar}{\pi\rho
v_{s}}\int_{0}^{q_{D}}\frac{q^{2}\textrm{d}q}{e^{\hbar v_{s}q/k_{B}T}-1}.$

In particular, we employ the expression above to analyze the evolution of the
peak location of a displaced Drude peak as a function of temperature.

Regarding electron dynamics, we focus on the quantum dynamics of an electron
under the following time-dependent Hamiltonian:

$\displaystyle\mathcal{H}_{0}=\frac{|\mathbf{p}|^{2}}{2m}+V_{D}(\mathbf{r},t),$

where $m$ is the effective (band) mass of the electron. This effective
Hamiltonian is the electron part of the considered Fröhlich Hamiltonian
described within the effective mass approximation.

A comparison of electrons lying in the neighborhood of the Fermi energy with
the strength of the deformation potential determines whether the scattering
can be treated perturbatively or not [58]. Furthermore, quantum coherence of
electrons is important in scattering when the wavelength of the Fermi
wavelength $\lambda_{F}$ is not much less than twice the lattice constant $a$.
It should be emphasized that the term “coherence” is employed here to describe
the spatial phase coherence of the electronic wavefunction, and should not be
conflated with the “coherent-versus-incoherent-metals” nomenclature, which
pertains to the breakdown of the quasi-particle paradigm. In summary, dynamics
is coarsely classified as

$\frac{E_{F}}{V_{\textrm{rms}}}=\begin{cases}\bar{E}\gg
1&\rightarrow\;\text{Perturbative}\\\ \bar{E}\sim 1\;\text{or}\;\bar{E}\ll
1&\rightarrow\;\text{Nonperturbative},\end{cases}$

and

$\frac{\lambda_{F}}{2a}=\begin{cases}\bar{\lambda}\ll
1&\rightarrow\;\text{Incoherent}\\\ \bar{\lambda}\sim
1\;\text{or}\;\bar{\lambda}>1&\rightarrow\;\text{Coherent}.\end{cases}$

##  Appendix B Material parameters

Table 1 contains the relevant material parameters for determining the optical
conductivity within the deformation potential scheme. The strange metals in
the Table below possess two characteristic attributes: relatively high
deformation potential constant and low Fermi energy compared to normal metals.
However, we want to point out the robustness of the results reported in the
main text against the reasonable range of material parameters, in a similar
manner as analyzed in Ref. [58].

| LSCO | Bi2212 | Sr3Ru2O7
---|---|---|---
$n$ [$10^{27}\,\textrm{m}^{-3}$] | 7.8 | 6.8 | 0.5
$m^{*}$ [$(m_{e})$] | 9.8 | 8.4 | 6.8
$v_{s}$ [m/s] | 6000 | 2460 | 5850
$E_{d}$ [eV] | 20 | 10 | 20
$\rho$ [$10^{-6}\,\text{kg/m}^{2}$] | 3.6 | 5.2 | 8.9
$E_{F}$ [eV] | 0.12 | 0.15 | 0.03
$a$ [Å] | 3.8 | 5.4 | 3.9
$T_{D}$ [K] | 427 | 123 | 406
Table 1: Material parameters that are used for three different strange metals.

##  Appendix C Optical conductivity

Here we discuss the linear response theory behind our optical conductivity
results. In general, the (complex) conductivity tensor $\sigma$ relates the
current density $j_{x}(t)$ to the applied electric field
$\textrm{Re}[E_{x}\exp(i\omega t)]$. Within the Kubo formalism, if the system
is in thermal equilibrium with a heat reservoir with temperature $T$, the
conductivity tensor for non-interacting particles is

$\sigma(\omega)=\lim_{\eta\to
0^{+}}\frac{\mathcal{V}}{\hbar\omega}\int_{0}^{\infty}\textrm{d}t\,\Big{\langle}[\hat{j}_{x}(t),\hat{j}_{x}(0)]\Big{\rangle}e^{i(\omega+i\eta)t}+\frac{ie^{2}n}{m\omega},$

$\mathcal{V}$ is the volume of the system, $m$ is the effective mass of the
electron, $n$ represents the number density of electrons and
$\langle\cdots\rangle$ refers to the averaging over equilibrium thermal
ensemble. We also include an infinitesimal increment $i\eta$ to ensure that
the integrand vanishes exponentially as $t\rightarrow\infty$, and thus the
integral is well-defined.

###  C.1 Frozen approximation (Adiabatic limit)

In the regime of $\omega\gg\omega_{D}$, the time scale of the perturbing
electric field on the system is much smaller than the timescale describing the
motion of lattice. Therefore, a good, first approximation would be to assume
that lattice is motionless. In fact, this frozen lattice approximation relates
to the adiabatic approximation which is basically omitting the dynamics of the
slow degrees of freedom when considering the fast degrees of motion.

When we freeze the deformation potential, we first solve the electronic states
$|n\rangle$ and corresponding energies $\varepsilon_{n}$ determined by the
equation

$\left(\frac{\mathbf{p}^{2}}{2m^{*}}+V_{D}(\mathbf{r})\right)|n\rangle=\varepsilon_{n}|n\rangle.$

Then, by utilizing this eigenbasis, the Kubo formula for the optical
conductivity takes the form of

$\begin{split}\mathrm{Re}[\sigma(\omega)]=-2\lim_{\eta\to
0^{+}}\frac{e^{2}\hbar}{\mathcal{V}}&\sum_{n,m}\frac{f(\varepsilon_{n})-f(\varepsilon_{m})}{\varepsilon_{n}-\varepsilon_{m}}\\\
&\times\frac{|\bra{n}\hat{v}_{x}\ket{m}|^{2}}{(\hbar\omega+\varepsilon_{n}-\varepsilon_{m})^{2}+\eta^{2}}\eta.\end{split}$

where the factor of two represents the spin degree of freedom. Theoretically,
the $\eta$ should be infinitely small, but in practice, we choose the
parameter $\eta$ to be a small enough not to impact the results. More
specifically, the value of $\eta$ is taken roughly to be the energy gap
between adjacent energy levels, thus guaranteeing that the fluctuations of the
optical conductivity will not be too large.

###  C.2 Dynamical field (Diabatic limit)

On the other hand, when $\omega\lesssim\omega_{D}$, the frozen potential
approximation is not valid. However, we can still determine the optical
conductivity by employing the Kubo formalism in the following manner,

$\begin{split}\mathrm{Re}[\sigma(\omega)]=-&2\lim_{\eta\to
0^{+}}\frac{e^{2}}{\mathcal{V}}\sum_{n,m}\frac{f(\varepsilon_{n})-f(\varepsilon_{m})}{\varepsilon_{n}-\varepsilon_{m}}\\\
&\times\int_{0}^{\infty}\mathrm{d}t\bra{n}\hat{v}_{x}(t)\ket{m}\bra{m}\hat{v}_{x}\ket{n}e^{i(\omega+i\eta)t},\end{split}$

where $\hat{v}_{x}(t)$ is the velocity operator in the Heisenberg picture, and
all the eigenstates and eigenvalues are taken to be that of the initial
condition. This dynamical formulation reduces to the Kubo formula above when
the deformation potential is treated as static, but it is also well-defined
for dynamical deformation potential.

## References

  * Varma [2020] C. M. Varma, Colloquium : Linear in temperature resistivity and associated mysteries including high temperature superconductivity, Rev. Mod. Phys. 92, 031001 (2020).
  * Pickett [2023] W. E. Pickett, Colloquium: Room temperature superconductivity: The roles of theory and materials design, Rev. Mod. Phys. 95, 021001 (2023).
  * Zhou  _et al._ [2021] X. Zhou, W.-S. Lee, M. Imada, N. Trivedi, P. Phillips, H.-Y. Kee, P. Törmä, and M. Eremets, High-temperature superconductivity, Nat. Rev. Phys. 3, 462 (2021).
  * Millis [2006] A. Millis, Gaps and our understanding, Science 314, 1888 (2006).
  * Timusk and Statt [1999] T. Timusk and B. Statt, The pseudogap in high-temperature superconductors: an experimental survey, Rep. Prog, Phys. 62, 61 (1999).
  * Damascelli  _et al._ [2003] A. Damascelli, Z. Hussain, and Z.-X. Shen, Angle-resolved photoemission studies of the cuprate superconductors, Rev. Mod. Phys. 75, 473 (2003).
  * Basov and Timusk [2005] D. N. Basov and T. Timusk, Electrodynamics of high-${T}_{c}$ superconductors, Rev. Mod. Phys. 77, 721 (2005).
  * Fischer  _et al._ [2007] O. Fischer, M. Kugler, I. Maggio-Aprile, C. Berthod, and C. Renner, Scanning tunneling spectroscopy of high-temperature superconductors, Rev. Mod. Phys. 79, 353 (2007).
  * Grüner [1988] G. Grüner, The dynamics of charge-density waves, Rev. Mod. Phys. 60, 1129 (1988).
  * Wu  _et al._ [2011] T. Wu, H. Mayaffre, S. Krämer, M. Horvatić, C. Berthier, W. Hardy, R. Liang, D. Bonn, and M.-H. Julien, Magnetic-field-induced charge-stripe order in the high-temperature superconductor yba2cu3o y, Nature 477, 191 (2011).
  * Chang  _et al._ [2012] J. Chang, E. Blackburn, A. Holmes, N. B. Christensen, J. Larsen, J. Mesot, R. Liang, D. Bonn, W. Hardy, A. Watenphul, _et al._ , Direct observation of competition between superconductivity and charge density wave order in yba2cu3o6. 67, Nat. Phys. 8, 871 (2012).
  * Ghiringhelli  _et al._ [2012] G. Ghiringhelli, M. Le Tacon, M. Minola, S. Blanco-Canosa, C. Mazzoli, N. Brookes, G. De Luca, A. Frano, D. Hawthorn, F. He, _et al._ , Long-range incommensurate charge fluctuations in (y, nd) ba2cu3o6+ x, Science 337, 821 (2012).
  * ‖  _et al._ [2004] N. E. H. ‖, K. Takenaka, and H. Takagi, Universality of the mott–ioffe–regel limit in metals, Philos. Mag. 84, 2847 (2004).
  * Gunnarsson  _et al._ [2003] O. Gunnarsson, M. Calandra, and J. E. Han, Colloquium: Saturation of electrical resistivity, Rev. Mod. Phys. 75, 1085 (2003).
  * Hussey  _et al._ [2011] N. E. Hussey, R. A. Cooper, X. Xu, Y. Wang, I. Mouzopoulou, B. Vignolle, and C. Proust, Dichotomy in the ¡i¿t¡/i¿-linear resistivity in hole-doped cuprates, Philos. Trans. R. Soc. A 369, 1626 (2011).
  * Hashimoto  _et al._ [2012] K. Hashimoto, K. Cho, T. Shibauchi, S. Kasahara, Y. Mizukami, R. Katsumata, Y. Tsuruhara, T. Terashima, H. Ikeda, M. A. Tanatar, H. Kitano, N. Salovich, R. W. Giannetta, P. Walmsley, A. Carrington, R. Prozorov, and Y. Matsuda, A sharp peak of the zero-temperature penetration depth at optimal composition in bafe2(as1-xpx)2, Science 336, 1554 (2012).
  * Bruin  _et al._ [2013] J. A. N. Bruin, H. Sakai, R. S. Perry, and A. P. Mackenzie, Similarity of scattering rates in metals showing T-linear resistivity, Science 339, 804 (2013).
  * Legros  _et al._ [2018] A. Legros, S. Benhabib, W. Tabis, F. Laliberté, M. Dion, M. Lizaire, B. Vignolle, D. Vignolles, H. Raffy, Z. Z. Li, P. Auban-Senzier, N. Doiron-Leyraud, P. Fournier, D. Colson, L. Taillefer, and C. Proust, Universal t-linear resistivity and planckian dissipation in overdoped cuprates, Nat. Phys. 15, 142 (2018).
  * Polshyn  _et al._ [2019] H. Polshyn, M. Yankowitz, S. Chen, Y. Zhang, K. Watanabe, T. Taniguchi, C. R. Dean, and A. F. Young, Large linear-in-temperature resistivity in twisted bilayer graphene, Nat. Phys. 15, 1011 (2019).
  * Grissonnanche  _et al._ [2021] G. Grissonnanche, Y. Fang, A. Legros, S. Verret, F. Laliberté, C. Collignon, J. Zhou, D. Graf, P. A. Goddard, L. Taillefer, and B. J. Ramshaw, Linear-in temperature resistivity from an isotropic planckian scattering rate, Nature 595, 667 (2021).
  * Chu  _et al._ [2022] Y. Chu, L. Liu, C. Shen, J. Tian, J. Tang, Y. Zhao, J. Liu, Y. Yuan, Y. Ji, R. Yang, K. Watanabe, T. Taniguchi, D. Shi, F. Wu, W. Yang, and G. Zhang, Temperature-linear resistivity in twisted double bilayer graphene, Phys. Rev. B 106, 035107 (2022).
  * Hartnoll and Mackenzie [2022] S. A. Hartnoll and A. P. Mackenzie, Colloquium: Planckian dissipation in metals, Rev. Mod. Phys. 94, 041002 (2022).
  * Kostic  _et al._ [1998] P. Kostic, Y. Okada, N. C. Collins, Z. Schlesinger, J. W. Reiner, L. Klein, A. Kapitulnik, T. H. Geballe, and M. R. Beasley, Non-fermi-liquid behavior of $\mathrm{SrRuO}{}_{3}$: Evidence from infrared conductivity, Phys. Rev. Lett. 81, 2498 (1998).
  * Lee  _et al._ [2002] Y. S. Lee, J. Yu, J. S. Lee, T. W. Noh, T.-H. Gimm, H.-Y. Choi, and C. B. Eom, Non-fermi liquid behavior and scaling of the low-frequency suppression in the optical conductivity spectra of ${\mathrm{caruo}}_{3}$, Phys. Rev. B 66, 041104 (2002).
  * Bernhard  _et al._ [2004] C. Bernhard, A. V. Boris, N. N. Kovaleva, G. Khaliullin, A. V. Pimenov, L. Yu, D. P. Chen, C. T. Lin, and B. Keimer, Charge ordering and magnetopolarons in ${\mathrm{n}\mathrm{a}}_{0.82}{\mathrm{c}\mathrm{o}\mathrm{o}}_{2}$, Phys. Rev. Lett. 93, 167003 (2004).
  * Tsvetkov  _et al._ [1997] A. A. Tsvetkov, J. Schützmann, J. I. Gorina, G. A. Kaljushnaia, and D. van der Marel, In-plane optical response of ${\mathrm{bi}}_{2}$${\mathrm{sr}}_{2}$${\mathrm{cuo}}_{6}$, Phys. Rev. B 55, 14152 (1997).
  * Takenaka  _et al._ [2003] K. Takenaka, J. Nohara, R. Shiozaki, and S. Sugai, Incoherent charge dynamics of ${\mathrm{la}}_{2-x}{\mathrm{sr}}_{x}{\mathrm{cuo}}_{4}:$ dynamical localization and resistivity saturation, Phys. Rev. B 68, 134501 (2003).
  * Osafune  _et al._ [1999] T. Osafune, N. Motoyama, H. Eisaki, S. Uchida, and S. Tajima, Pseudogap and collective mode in the optical conductivity spectra of hole-doped ladders in ${\mathrm{sr}}_{14-\mathit{x}}{\mathrm{ca}}_{\mathit{x}}{\mathrm{cu}}_{24}{O}_{41}$, Phys. Rev. Lett. 82, 1313 (1999).
  * Rozenberg  _et al._ [1995] M. J. Rozenberg, G. Kotliar, H. Kajueter, G. A. Thomas, D. H. Rapkine, J. M. Honig, and P. Metcalf, Optical conductivity in mott-hubbard systems, Phys. Rev. Lett. 75, 105 (1995).
  * Jönsson  _et al._ [2007] P. E. Jönsson, K. Takenaka, S. Niitaka, T. Sasagawa, S. Sugai, and H. Takagi, Correlation-driven heavy-fermion formation in ${\mathrm{liv}}_{2}{\mathrm{o}}_{4}$, Phys. Rev. Lett. 99, 167402 (2007).
  * Takenaka  _et al._ [1999] K. Takenaka, Y. Sawaki, and S. Sugai, Incoherent-to-coherent crossover of optical spectra in ${\mathrm{la}}_{0.825}{\mathrm{sr}}_{0.175}{\mathrm{mno}}_{3}:$ temperature-dependent reflectivity spectra measured on cleaved surfaces, Phys. Rev. B 60, 13011 (1999).
  * Takenaka  _et al._ [2002] K. Takenaka, R. Shiozaki, and S. Sugai, Charge dynamics of a double-exchange ferromagnet ${\mathrm{la}}_{1-x}{\mathrm{sr}}_{x}{\mathrm{mno}}_{3}$, Phys. Rev. B 65, 184436 (2002).
  * Dong  _et al._ [1999] J. Dong, J. L. Musfeldt, J. A. Schlueter, J. M. Williams, P. G. Nixon, R. W. Winter, and G. L. Gard, Optical properties of ${\beta}^{{}^{\prime\prime}}-(\mathrm{ET}{)}_{2}{\mathrm{sf}}_{5}{\mathrm{ch}}_{2}{\mathrm{cf}}_{2}{\mathrm{so}}_{3}:$ a layered molecular superconductor with large discrete counterions, Phys. Rev. B 60, 4342 (1999).
  * Takenaka  _et al._ [2005] K. Takenaka, M. Tamura, N. Tajima, H. Takagi, J. Nohara, and S. Sugai, Collapse of coherent quasiparticle states in $\theta\mathrm{\text{$-$}}(\mathrm{BEDT}\mathrm{\text{$-$}}\mathrm{TTF}{)}_{2}{\mathrm{i}}_{3}$ observed by optical spectroscopy, Phys. Rev. Lett. 95, 227801 (2005).
  * Wang  _et al._ [2004] N. L. Wang, P. Zheng, D. Wu, Y. C. Ma, T. Xiang, R. Y. Jin, and D. Mandrus, Infrared probe of the electronic structure and charge dynamics of ${\mathrm{n}\mathrm{a}}_{0.7}\mathrm{C}\mathrm{o}{\mathrm{o}}_{2}$, Phys. Rev. Lett. 93, 237007 (2004).
  * Jaramillo  _et al._ [2014] R. Jaramillo, S. D. Ha, D. Silevitch, and S. Ramanathan, Origins of bad-metal conductivity and the insulator–metal transition in the rare-earth nickelates, Nat. Phys. 10, 304 (2014).
  * Hwang  _et al._ [2007] J. Hwang, T. Timusk, and G. Gu, Doping dependent optical properties of bi2sr2cacu2o8+ $\delta$, J. Condens. Matter Phys. 19, 125208 (2007).
  * Lee and Ramakrishnan [1985] P. A. Lee and T. V. Ramakrishnan, Disordered electronic systems, Rev. Mod. Phys. 57, 287 (1985).
  * Smith [2001] N. V. Smith, Classical generalization of the drude formula for the optical conductivity, Phys. Rev. B 64, 155106 (2001).
  * Kaveh and Mott [1982] M. Kaveh and N. Mott, Universal dependences of the conductivity of metallic disordered systems on temperature, magnetic field and frequency, J. Phys. C: Solid State Phys. 15, L707 (1982).
  * Fratini and Ciuchi [2021] S. Fratini and S. Ciuchi, Displaced Drude peak and bad metal from the interaction with slow fluctuations., SciPost Phys. 11, 39 (2021).
  * Pustogow  _et al._ [2021] A. Pustogow, Y. Saito, A. Löhle, M. S. Alonso, A. Kawamoto, V. Dobrosavljević, M. Dressel, and S. Fratini, Rise and fall of landau’s quasiparticles while approaching the mott transition, Nat. Commun. 12, 1571 (2021).
  * Caprara  _et al._ [2002] S. Caprara, C. Di Castro, S. Fratini, and M. Grilli, Anomalous optical absorption in the normal state of overdoped cuprates near the charge-ordering instability, Phys. Rev. Lett. 88, 147001 (2002).
  * Delacrétaz  _et al._ [2017] L. V. Delacrétaz, B. Goutéraux, S. A. Hartnoll, and A. Karlsson, Bad Metals from Fluctuating Density Waves, SciPost Phys. 3, 025 (2017).
  * Basov  _et al._ [2011] D. N. Basov, R. D. Averitt, D. van der Marel, M. Dressel, and K. Haule, Electrodynamics of correlated electron materials, Rev. Mod. Phys. 83, 471 (2011).
  * Lanzara  _et al._ [2001] A. Lanzara, P. V. Bogdanov, X. J. Zhou, S. A. Kellar, D. L. Feng, E. D. Lu, T. Yoshida, H. Eisaki, A. Fujimori, K. Kishio, J.-I. Shimoyama, T. Noda, S. Uchida, Z. Hussain, and Z.-X. Shen, Evidence for ubiquitous strong electron–phonon coupling in high-temperature superconductors, Nature 412, 510 (2001).
  * Kim  _et al._ [2022] D. Kim, A. Aydin, A. Daza, K. N. Avanaki, J. Keski-Rahkonen, and E. J. Heller, Coherent charge carrier dynamics in the presence of thermal lattice vibrations, Phys. Rev. B 106, 054311 (2022).
  * Patel  _et al._ [2023] A. A. Patel, H. Guo, I. Esterlis, and S. Sachdev, Universal theory of strange metals from spatially random interactions, Science 381, 790 (2023).
  * Gogolin  _et al._ [1975] A. Gogolin, V. Mel’Nikov, and E. Rashba, Conductivity in a disordered one-dimensional system induced by electron-phonon interaction, Soviet Journal of Experimental and Theoretical Physics 42, 168 (1975).
  * Rashba  _et al._ [1977] E. Rashba, A. Gogolin, and V. Mel’nikov, In organic conductors and semiconductors, volume 65 of lecture notes in physics (1977).
  * Thouless [1977] D. J. Thouless, Maximum metallic resistance in thin wires, Phys. Rev. Lett. 39, 1167 (1977).
  * Abrahams [2010] E. Abrahams,  _50 years of Anderson Localization_ , Vol. 24 (world scientific, 2010).
  * Anderson [1958] P. W. Anderson, Absence of diffusion in certain random lattices, Phys. Rev. 109, 1492 (1958).
  * Di Sante  _et al._ [2017] D. Di Sante, S. Fratini, V. Dobrosavljević, and S. Ciuchi, Disorder-driven metal-insulator transitions in deformable lattices, Phys. Rev. Lett. 118, 036602 (2017).
  * Fratini  _et al._ [2016] S. Fratini, D. Mayou, and S. Ciuchi, The transient localization scenario for charge transport in crystalline organic materials, Adv. Funct. Mater. 26, 2292 (2016).
  * Troisi and Orlandi [2006] A. Troisi and G. Orlandi, Charge-transport regime of crystalline organic semiconductors: Diffusion limited by thermal off-diagonal electronic disorder, Phys. Rev. Lett. 96, 086601 (2006).
  * Lacroix  _et al._ [2020] A. Lacroix, G. T. de Laissardière, P. Quémerais, J.-P. Julien, and D. Mayou, Modeling of electronic mobilities in halide perovskites: Adiabatic quantum localization scenario, Phys. Rev. Lett. 124, 196601 (2020).
  * Aydin  _et al._ [2023] A. Aydin, D. Kim, J. Keski-Rahkonen, and E. J. Heller, Return of the phonons: Planckian scattering emerges from vibronic dynamics, arXiv preprint arXiv:2303.06077 (2023).
  * Sachdev [1999] S. Sachdev, Quantum phase transitions, Physics world 12, 33 (1999).
  * Chowdhury  _et al._ [2022] D. Chowdhury, A. Georges, O. Parcollet, and S. Sachdev, Sachdev-ye-kitaev models and beyond: Window into non-fermi liquids, Rev. Mod. Phys. 94, 035004 (2022).
  * Sachdev [2023] S. Sachdev,  _Quantum Phases of Matter_ (Cambridge University Press, 2023).
  * Fröhlich [1954] H. Fröhlich, Electrons in lattice fields, Advances in Physics 3, 325 (1954).
  * Fröhlich and Mott [1937] H. Fröhlich and N. F. Mott, Theory of electrical breakdown in ionic crystals, Proc. R. Soc. A 160, 230 (1937).
  * Mahan [2000] G. D. Mahan,  _Many-particle physics_ (Springer Science & Business Media, 2000).
  * Ashcroft and Mermin [1976] N. Ashcroft and N. Mermin,  _Solid State Physics_ , HRW international editions (Holt, Rinehart and Winston, 1976).
  * Bloch [1930] F. Bloch, Zum elektrischen widerstandsgesetz bei tiefen temperaturen, Zeitschrift für Physik 59, 208 (1930).
  * Grüneisen [1933] E. Grüneisen, Ann. Phys., Lpz. 16 (1933).
  * Brown and Twiss [1957] R. H. Brown and R. Q. Twiss, Interferometry of the intensity fluctuations in light. i. basic theory: The correlation between photons in coherent beams of radiation, Proc. R. Soc. A 242, 300 (1957).
  * Glauber [1963] R. J. Glauber, Coherent and Incoherent States of the Radiation Field, Phys. Rev 131, 2766 (1963).
  * Shockley and Bardeen [1950] W. Shockley and J. Bardeen, Energy bands and mobilities in monatomic semiconductors, Phys. Rev. 77, 407 (1950).
  * Bardeen and Shockley [1950] J. Bardeen and W. Shockley, Deformation potentials and mobilities in non-polar crystals, Phys. Rev. 80, 72 (1950).
  * Delacrétaz  _et al._ [2017] L. V. Delacrétaz, B. Goutéraux, S. A. Hartnoll, and A. Karlsson, Bad metals from fluctuating density waves, SciPost Phys 3, 025 (2017).
  * Kim and Halperin [2023] D. Kim and B. I. Halperin, Low-energy tail of the spectral density for a particle interacting with a quantum phonon bath, Phys. Rev. B 107, 224311 (2023).
  * Basov  _et al._ [1998] D. N. Basov, B. Dabrowski, and T. Timusk, Infrared probe of transition from superconductor to nonmetal in ${\mathrm{yba}}_{2}({\mathrm{cu}}_{1-\mathit{x}}{\mathrm{zn}}_{\mathit{x}}{)}_{4}{O}_{8}$, Phys. Rev. Lett. 81, 2132 (1998).
  * Wang  _et al._ [1998] N. L. Wang, S. Tajima, A. I. Rykov, and K. Tomimoto, Zn-substitution effects on the optical conductivity in ${\mathrm{yba}}_{2}{\mathrm{cu}}_{3}{\mathrm{o}}_{7\mathrm{-}\mathrm{\delta}}$ crystals: Strong pair breaking and reduction of in-plane anisotropy, Phys. Rev. B 57, R11081 (1998).
  * Basov  _et al._ [1994] D. N. Basov, A. V. Puchkov, R. A. Hughes, T. Strach, J. Preston, T. Timusk, D. A. Bonn, R. Liang, and W. N. Hardy, Disorder and superconducting-state conductivity of single crystals of ${\mathrm{yba}}_{2}$${\mathrm{cu}}_{3}$${\mathrm{o}}_{6.95}$, Phys. Rev. B 49, 12165 (1994).


11institutetext: 1DIFFER, Eindhoven, the Netherlands
2Techincal University of Eindhoven, Eindhoven, the Netherlands
a _Contact:_ of B.J.J. Kremers 11email<EMAIL_ADDRESS>
# Two step clustering for data reduction combining DBSCAN and k-means
clustering

B.J.J. Kremers1,a A. Ho2 J. Citrin1,2 K.L. van der Plassche1,2

(Received: date / Accepted: date)

###### Abstract

A novel combination of two widely-used clustering algorithms is proposed here
for the detection and reduction of high data density regions. The Density
Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used
for the detection of high data density regions and the k-means algorithm for
reduction. The proposed algorithm iterates while successively decrementing the
DBSCAN search radius, allowing for an adaptive reduction factor based on the
effective data density. The algorithm is demonstrated for a physics simulation
application, where a surrogate model for fusion reactor plasma turbulence is
generated with neural networks. A training dataset for the surrogate model is
created with a quasilinear gyrokinetics code for turbulent transport
calculations in fusion plasmas. The training set consists of model inputs
derived from a repository of experimental measurements, meaning there is a
potential risk of over-representing specific regions of this input parameter
space. By applying the proposed reduction algorithm to this dataset, this
study demonstrates that the training dataset can be reduced by a factor
$\sim$20 using the proposed algorithm, without a noticeable loss in the
surrogate model accuracy. This reduction provides a novel way of analyzing
existing high-dimensional datasets for biases and consequently reducing them,
which lowers the cost of re-populating that parameter space with higher
quality data.

###### Keywords:

Clustering Data reduction DBSCAN k-means surrogate neural network Data
optimization

##  1 Introduction

Multiscale modelling is used to simulate complex phenomena in many areas of
science and engineering Groen2014 (1), such as material science Suter2015 (2),
bio-medicine Tahir2014 (3, 4, 5), engineering Delgado-Buscalioni2003 (6) and
plasma physics. However, the evaluation of these multiscale models are
generally computationally expensive. Within the field of plasma physics,
recent developments have shown that targeted neural network (NN) surrogate
models can reduce this computational time by orders of magnitude
VanDePlassche2020 (7). Unfortunately, acquiring a sufficiently representative
dataset to train the NN in this application involves running the multiscale
simulation itself. Thus, while NNs are fast to evaluate, they remain
computationally expensive to develop Alowayyed2019 (8). A more practical
approach to building such a database with no _a priori_ information is to
collect it from previous and current phenomenological studies that have been
accumulated over time. This causes the data distribution to be determined by
factors outside the considerations of surrogate model development and may
result in the over-representation of certain regions in the parameter space,
i.e. a biased dataset.

This paper proposes an algorithm for the detection and reduction of high data
density space. In order to achieve this, a two-step procedure is proposed
using the DBSCAN Ester2009 (9) and $k$-means Stuart1982 (10) algorithms. The
DBSCAN algorithm is used to detect clusters with a user-defined minimum data
density, as well as for initial outlier filtering. These clusters are then
individually divided into $k$ evenly-dispersed groups by the $k$-means
algorithm. The centroids of these groups are then used for data reduction by
replacing each group with the actual data point closest to the identified
centroid. The general difficulty in using the $k$-means algorithm alone is
determining the value for $k$.

DBSCAN uses global parameters, defined in Chapter 2, which prevent the
algorithm to differentiate clusters, or even regions within clusters, based on
their densities. Several algorithms are proposed in literature to overcome
these issues, which fall into two major categories: a partitioning-based or a
non-partitioning-based approach Kim2019 (11). A few partitioning-based
algorithms include GMDBSCAN Xiaoyun2008 (12), ADBSCAN Khan2019 (13), PACA-
DBSCAN Jiang2011 (14), APSCAN Chen2011 (15), AA-DBSCAN Kim2019 (11) and kAA-
DBSCAN Kim2019 (11). A few non-partitioning based approaches such as OPTICS
Ankerst1999 (16) and DPC Rodriguez2014 (17). However, DBSCAN also provides an
estimate for the minimum data density of a cluster via its neighbourhood
radius parameter, which can be subsequently used to advise a reasonable value
for $k$.

To that effect, we propose a new approach for a flexible data reduction
algorithm based on the detection of clusters with varying densities. The
combination of algorithms proposed here is similar to a recently-proposed
solution to the multiple circle detection problem Scitovski2021. However, the
modifications made to the $k$-means algorithm to identify the centers of
intersecting generalized circles does not serve the purpose of replacing a
collection of data points with a reduced representative set. In addition, the
iterative DBSCAN execution differs from currently available algorithms in that
the clustering parameters are modified per iteration, such that it detects
regions of increasing data density. As points are removed sequentially over
each iteration, less internal memory is used for the detection and reduction
of these regions. Due to this, the algorithm is more scalable to larger non-
uniform datasets such as those compiled in a top-down fashion via data mining,
as opposed to those built from the bottom-up using experimental design
techniques.

Section 2 provides an overview of the DBSCAN algorithm, the k-means algorithm
and neural network surrogate modeling. In Section 3 we propose the two step
clustering algorithm for data reduction. In Section 4, the algorithm is
evaluated on two separate datasets: an artificially-generated test dataset for
ease of visualization and an experimentally-derived dataset, generously
provided by the Joint European Torus experimental fusion plasma device located
in Culham, UK, to demonstrate an application of the algorithm in neural
network surrogate modeling. While the input side of the datasets was selected,
the corresponding output side for the supervised learning application was
created using QuaLiKiz Bourdelle2007 (18), a quasilinear gyrokinetics code for
turbulence modeling in fusion plasmas. Section 5 summarizes and concludes the
paper.

##  2 Properties of existing algorithms

These descriptions are based on a data-set, $D$, in a $d$-dimensional space,
$\mathbb{R}^{d}$, consisting of $n$ objects.

###  2.1 DBSCAN

DBSCAN is an algorithm for density-based clustering originally designed to
discover clusters, $C$, of arbitrary shapes in spatial data Ester2009 (9). The
algorithm takes two input parameters:

  * •

$\epsilon$, the radius of a hypersphere drawn around each point, known as a
neighborhood, $N_{\epsilon}$;

  * •

$MinPts$, the minimum number of points in the neighbourhood needed to be
defined as a part of a cluster, including the current point.

With this spatial definition, the relations between the points, $p,q\in D$,
can be categorized as follows:

###### Definition 1

(Directly density-reachable) Object $p$ is considered directly density-
reachable from object $q$ if $p\in N_{\epsilon}(q)$.

###### Definition 2

(Density-reachable) Object $p$ is considered density-reachable from object
$q$, where $p,\,q\in D$, if a chain of objects, $\\{p_{1},\dots,p_{k+1}\\}\in
D$, exists such that $p_{i+1}$ is directly density-reachable from $p_{i}$,
where $i\in[1,k]$, $p_{1}=p$ and $p_{k+1}=q$.

###### Definition 3

(Cluster, $C$) A subset of $D$ where $|C|\geq MinPts$ and all objects $p\in C$
are density-reachable from all other objects $q\in C$.

DBSCAN then finds these clusters by labelling points using the following
definitions Ester2009 (9):

###### Definition 4

(Core object, $x_{\text{core}}$) If object $x$ is such that
$|N_{\epsilon}(x)|\geq MinPts$, then $x$ is considered a core object.

###### Definition 5

(Border object, $x_{\text{border}}$) If object $x$ is directly density-
reachable from a core object but $|N_{\epsilon}(x)|<MinPts$, then $x$ is
considered a border object.

###### Definition 6

(Noise object, $x_{\text{noise}}$) If object $x$ is neither a core nor a
border object, then $x$ is considered a noise object.

Once the entire dataset is labelled, it groups together all points.

###  2.2 $k$-means

The $k$-means algorithm is designed to partition a dataset into regions or
clusters, $R$. This algorithm accepts a single input parameter:

  * •

$k$, the number of clusters into which the data is partitioned.

Once the value is set, $k$ centroids, $\\{m_{1},\dots,m_{k}\\}$, are
determined and each object, $x$, is associated to the nearest centroid. This
determination is done by minimizing the spatial variance between the centroids
and their associated data points.

Initially, the centroids are randomly chosen and the algorithm iterates two
processes over a counter, $i$, until the centroids no longer change. In the
first process, each object is assigned to the cluster, $R^{(t)}$, with the
least squared Euclidean distance away from it. This can be mathematically
expressed as follows:

$R_{i}^{(t)}=\\{x:||x-m_{i}^{(t)}||^{2}\leq||x-m_{j}^{(t)}||^{2}\;\forall\;j\in[1,k]\\}$
(1)

where each point is assigned to exactly one cluster. In the second process,
the cluster centroids are recalculated based on the mean of the objects
assigned to it. This can be expressed as:

$m_{i+1}^{(t)}=\frac{1}{|R_{i}^{(t)}|}\sum_{R_{i}^{(t)}}x$ (2)

###  2.3 Neural networks

A neural network (NN) is a universal approximator consisting of nodes, each
having a weight $w_{j}$ and a bias $b_{j}$. By interconnecting layers of these
nodes and designing the connections purposefully, the NN can be made suitable
for both regression or classification tasks. These weights and biases have to
be learned by optimizing an objective function, $\mathcal{U}$. Typically, this
objective function minimally consists of a goodness-of-fit term to fit the
training dataset and a regularization term to avoid overfitting.

This study focuses on a previous fully-connected feed-forward NNs (FFNN), used
to reproduce the input-output mapping of the quasilinear gyrokinetic transport
model QuaLiKiz Citrin2017 (19, 20) for turbulence predictions in fusion
plasmas. A custom architecture and cost function was used for training these
NNs VanDePlassche2020 (7), which were designed to better capture the desired
physical characteristics of the problem.

##  3 Two step clustering algorithm

This section describes the proposed two-step algorithm, firstly by explaining
the concept behind combining the DBSCAN and $k$-means algorithms and then
expanding its application into an iterative procedure. Figure 1 depicts the
basic steps of the algorithm on a toy example, containing 3 densely populated
regions in a 2D space. While not explicitly shown in the figure, it is often
convenient to scale and translate the data ranges such that they have unit
variance and zero mean. Starting with a dataset, $D$, as shown in Figure 1(a),
the general two-step clustering algorithm goes through the following steps:

  1. 1.

Apply DBSCAN algorithm to a dataset, $D$, to determine regions, $C$, of high
data density, resulting in Figure 1(b).

  2. 2.

Label all points where $D\notin C$ as $C_{noise}$ and discard.

  3. 3.

Apply $k$-means algorithm to each individual cluster, $C$, splitting it into
$k$ regions, $R_{C}$, resulting in Figure 1(c).

  4. 4.

The nearest data points, $x\in D$, to the $k$-means cluster centroids are
determined. These points become the output reduced dataset, $D^{\prime}$, as
shown in Figure 1(d).

A summary of the input parameters to the algorithm can be found in Table 1.

(a) Original dataset with three dense clusters and noise.

(b) Three clusters depicted in yellow green and red are detected by the DBSCAN
algorithm. Black points are considered noise that do not fall into any
cluster.

(c) Using $k$-means on the individual DBSCAN clusters the clusters are split
into smaller partitions. Noise points are removed from the data-set.

(d) Final data-set, k-means clusters replaced by point closest to the mean of
the cluster.

Figure 1: Schematic example of two-step algorithm, with data cluster detection
using DBSCAN and cluster reduction using k-means.

While these steps form the basis for the two-step clustering algorithm, the
application also aims to distinguish between regions of varying local data
density, $d$. Thus, this two-step algorithm is iterated over $i$ steps while
decreasing the radius, $\epsilon_{i}$, in order to find increasingly dense
regions in $D_{i}$. In order to improve the robustness of the final reduced
dataset, the first iteration of this loop doubles as a noise filter. For this
reason, it was necessary to decouple the neighbourhood radius used in the
initial pass, $\epsilon_{0}$, and the neighbourhood radius used in the first
real iteration, $\epsilon_{1}$.

For the purposes of this algorithm, the local density was defined as:

$d=\frac{\bar{n}}{V\\!\left(\epsilon_{i},N\right)}$ (3)

where $\bar{n}$ is the average number of neighbours over all $x_{core}\in C$,
$V$ is the volume of an $N$-dimensional ball, and $N$ is the dimensionality of
the dataset $D$. This allows the $k$-means parameter, $k$, to be chosen
adaptively for each density bin. The adaptive selection of $k$ was implemented
using the following expression:

$\tilde{k}=an\left(1-\zeta\cdot\left[1-e^{-\tau
d}\right]\right)\,,\quad\tau\in\left(0,\infty\right)$ (4)
$k=\begin{cases}\left\lfloor\tilde{k}\right\rfloor,&\text{for }\tilde{k}>1\\\
1,&\text{otherwise}\end{cases}$ (5)

where $a$ is a fractional value representing the desired global reduction
ratio, $\zeta$ is a weighting factor between a uniform reduction scheme and a
density-dependent reduction scheme, and $\tau$ is a control parameter setting
how quickly $k\to 1$ in the density-dependent scheme.

The form of Equation (4) was chosen purely on functionality. By examining the
limits of $\zeta$, it can be easily seen that choosing $\zeta=0$ enforces
$k=an$ for all clusters. This selects a uniform reduction ratio throughout the
entire final dataset based on the user-defined value of $a$. Choosing
$\zeta=1$ causes $k$ to be determined solely by the last term in Equation (4).
The combination of Equations (4) and (5) can be dissected further by examining
the limits of $\tau$. As $\tau\to 0$, $k=an$ and a uniform reduction ratio is
again applied. However, as $\tau\to\infty$, $\tilde{k}\to 0$ and $k\to 1$
meaning that all DBSCAN clusters are reduced to a single centroid. In general,
it is recommend to have a reasonably small but finite value for $\tau$, such
that $k\to 1$ gradually as $d$ increases, meaning a higher reduction ratio for
more dense clusters.

In addition to the three parameters described in Equation (4) replacing $k$
and $\epsilon_{0}$ replacing $\epsilon$, another six input parameters are
introduced to the algorithm:

  * •

$a$, the global reduction ratio, which also defines the minimum reduction
factor;

  * •

$a_{noise}$, the reduction ratio to be applied to $\left\\{x_{noise}\right\\}$
in all iterations except the initial denoising pass;

  * •

$\epsilon_{1}$, the starting neighbourhood radius to be used after the initial
denoising pass;

  * •

$\Delta\epsilon$, the step size applied to $\epsilon_{1}$ between consecutive
iterations after the initial denoising pass;

  * •

$n_{\text{min}}$, the number of points in a cluster below which the $k$-means
reduction step is not applied to the cluster;

  * •

$n_{\text{max}}$, the number of points in a cluster above which it is
considered too large to be reduced in a given iteration.

The following definitions are used for the reduction and iterations:

###### Definition 7

(Saved cluster $C_{s}$) Clusters $C$ where $MinPts\leq|C|\leq n_{\text{min}}$
are defined as saved clusters and transferred directly into the final dataset,
$D_{f}$. It is assumed to contain sufficiently valuable information due to its
sparseness.

###### Definition 8

(Reduced cluster $C_{r}$) Clusters $C$ where $n_{\text{min}}<|C|\leq
n_{\text{max}}$ are defined as reduced clusters, to which the $k$-means
reduction step is applied. The data points in these clusters nearest to the
$k$-means centroids, $D^{\prime}$, are transferred to the final dataset,
$D_{f}$.

###### Definition 9

(Passed cluster $C_{p}$) Clusters $C$ where $n_{\text{max}}<|C|$ are defined
as passed clusters as they contain sufficiently high data density to be
subject to additional iterations with lower $\epsilon$. None of the points in
these clusters are transferred to the final dataset, $D_{f}$, in the current
iteration.

After the initial DBSCAN execution, the set of noise points,
$\left\\{x_{noise}\right\\}$ are discarded, $C_{s}$ are added to the final
database, and the k-means algorithm is applied to reduce the individual
reduction clusters, $C_{r}$. The $C_{p}$ are then passed to the next iteration
DBSCAN, where the procedure is slightly modified compared to the initial pass.
Specifically, $\left\\{x_{noise}\right\\}$ are now passed as an independent
cluster to the k-means algorithm using a $k$ determined by $a_{noise}$ and
$\zeta=0$. The resulting reduced set of noise points are added to the $C_{s}$
and are thus added to the final database $D_{f}$. Each following iteration
reduces $\epsilon$ by $\Delta\epsilon$ meaning DBSCAN selects out regions of
increasingly higher data density from the dataset. The iterations continue
until either all of the original data has passed through the $k$-means
reduction process or a user-defined maximum density is reached. This maximum
density is determined through a minimum neighbourhood radius,
$\epsilon_{\text{min}}$, and was implemented to prevent extremely dense
regions from slowing down the algorithm and biasing the final data-set,
$D_{f}$.

Algorithm 1 Two-step clustering procedure

Load and rescale $D$

$D_{i}=D$

while $D_{i}$ not empty do

DBSCAN $D_{i}$

Define $D_{n}=\\{x_{noise}\\}$

if initial pass then

Remove $D_{n}$ from $D_{i}$

else

Calculate adaptive $k$ with $a=a_{noise}$, $\zeta=0$

Apply $k$-means to $D_{n}$

Find $D^{\prime}=\\{x_{m}:\min|x_{m}-m_{i}|\;\forall\;i\in\left[1,k\right]\\}$

Move $D^{\prime}$ to $D_{f}$

Remove remaining $x\in D_{n}$ from $D_{i}$

end if

for $C_{s}$ do

Move $C_{s}$ to $D_{f}$

end for

for $C_{r}$ do

Calculate $d$ and adaptive $k$

Apply $k$-means to $C_{r}$

Find $D^{\prime}=\\{x_{m}:\min|x_{m}-m_{i}|\;\forall\;i\in\left[1,k\right]\\}$

Move $D^{\prime}$ to $D_{f}$

Remove remaining $x\in C_{r}$ from $D_{i}$

end for

Decrement $\epsilon$ by $\Delta\epsilon$

if $\epsilon\leq\epsilon_{\text{min}}$ then

Calculate $d$ and adaptive $k$

Apply $k$-means to $D_{i}$

Find $D^{\prime}=\\{x_{m}:\min|x_{m}-m_{i}|\;\forall\;i\in\left[1,k\right]\\}$

Move $D^{\prime}$ to $D_{f}$

Remove all remaining $x\in D_{i}$ from $D_{i}$

end if

end while

Save $D_{f}$

Parameter | Symbol | Range | Recommended
---|---|---|---
Initial pass neighborhood radius | $\epsilon_{0}$ | $\mathbb{R}\in\left(0,\infty\right)$ | 1.0
First iteration neighborhood radius | $\epsilon_{1}$ | $\mathbb{R}\in\left(0,\infty\right)$ | 0.75
Neighbourhood radius step size | $\delta\epsilon$ | $\mathbb{R}\in\left(0,\infty\right)$ | 0.05
Minimum neighborhood radius | $\epsilon_{\text{min}}$ | $\mathbb{R}\in\left(0,\infty\right)$ | 0.01
Minimum cluster size | $MinPts$ | $\mathbb{Z}\in\left[2,\infty\right)$ | $\leq 10$
Minimum reduced cluster size | $n_{\text{min}}$ | $\mathbb{Z}\in\left[2,\infty\right)$ | same as $MinPts$
Maximum reduced cluster size | $n_{\text{max}}$ | $\mathbb{Z}\in\left[2,\infty\right)$ | 100
Reduction scheme parameter | $\zeta$ | $\mathbb{R}\in\left[0,1\right]$ | 1.0
Density-dependent factor | $\tau$ | $\mathbb{R}\in\left(0,\infty\right)$ | 0.001
Table 1: Input parameters for the two-step reduction algorithm and their
recommended starting values.

At this point, it is important to note that the recommended value for
$n_{\text{min}}$ is the same value as $MinPts$. This effectively removes the
presence of saved clusters, $C_{s}$, within the algorithm. In practice, the
presence of these clusters tended to significantly hinder the efficacy of the
density-dependent reduction scheme, meaning the $\zeta=0$ behaviour closely
resembled that of $\zeta=1$. However, this functionality is retained in the
code in the case that another application has need for it.

##  4 Demonstrative Applications

In order to demonstrate the capability of the two-step clustering algorithm
described in Section 3, it was applied to two test cases. The first is a 3D
sample dataset created from 2 input parameters and 1 output parameter from the
QuaLiKiz turbulent transport code, described in Section 4.1. Although the full
input-output dimensionality of the code is much higher, this selection allows
for an easy visualization of the results from the clustering algorithm. The
second test case is of a 16D dataset, 15 inputs and 1 output, described in
Section 4.2. This dataset is based on the same experimental data used in the
previous NN development study using this physics code Ho2021 (21). Additional
NNs were trained using the final reduced dataset, $D_{f}$, from the two-step
algorithm and compared to the original NN.

At this point, it should be noted that the specific choice to include an
output parameter is these test cases is due to the application on NN
regression training datasets. The algorithm itself does not functionally
distinguish between the various columns in the provided data table.

###  4.1 3D test dataset

The 3D dataset consists of two input parameters, namely the logarithmic
electron temperature gradient, $R/L_{Te}$, and the logarithmic ion temperature
gradient, $R/L_{Ti}$, and one output parameter, the ion temperature gradient
(ITG) ion heat flux, $q_{i,\text{ITG}}$. In order to test the density-
dependent functionality of the proposed two-step clustering algorithm, the
input half of the dataset was sampled explicitly to contain a variable data
density. The input parameters were sampled such that
$R/L_{Te},R/L_{Ti}\in\left[0,18\right]$, using a base uniform distribution
with Gaussians added on top. The locations for these Gaussian distributions
are $(4.5,4.5)$, $(13.5,4.5)$, $(4.5,13.5)$, $(13.5,13.5)$ with an increasing
maximum probability per addition. These sampled inputs are then inserted into
QuaLiKiz to calculate $q_{i,\text{ITG}}$, with the other settings provided in
Appendix. The resulting 3D dataset is shown in Figure 2.

Figure 2: Test 3D dataset (two-input, one-output) generated via random
sampling on the inputs, $R/L_{Te}$ and $R/L_{Ti}$, and using QuaLiKiz to
compute the output, $q_{i,\text{ITG}}$, with normalized units (GB). The input
space was sampled within (0,18) using a fixed uniform density with four
Gaussian peaks with varying peak data densities added on top.

In principle, the algorithm is capable of including multiple output dimensions
but this demonstration only uses one for ease of interpretation.

####  4.1.1 3D dataset reduction

The two-step clustering algorithm is then applied to the 3D dataset described
in Section 4.1. A scan of the various input parameters of the algorithm
results in different overall reduction factors, as expected, where the overall
reduction factor, $f$, is defined in this study as follows:

$f=\frac{n_{\textrm{orig}}}{n_{\text{red}}}$ (6)

where $n_{\textrm{orig}}$ is the number of points in the original dataset and
$n_{\text{red}}$ is the number of points in the reduced dataset.

An example of a reduced 3D dataset can be seen in Figures 3 and 4, using the
same code inputs with the exception of $\zeta=0$ and $\zeta=1$, respectively.
As shown in the figures, the overall reduction factor is larger for the
density-dependent $k$ configuration ($f=10$) as opposed to the global $k$
configuration ($f=8$). For both sets, the core features and distributions of
the dataset are still visible, showing that it is possible to reduce a dataset
using the two-step clustering algorithm.

Figure 3: Reduced 3D sample dataset using $\zeta=0$, applying an overall
reduction factor, $f=5.2$. As shown, the density distributions in the original
dataset are mostly kept and all four high-density regions visually
distinguishable. Figure 4: Reduced 3D sample dataset using $\zeta=1$, applying
an overall reduction factor, $f=8.7$. As shown, the density distributions
within the high-density regions have been reduced dramatically without
significantly impacting the density in the lower-density background. The two
high-density regions at low $R/L_{T_{i}}$ are almost visually
indistinguishable from the background.

However, given that the two high-density regions near $R/L_{Ti}=0.0$ in Figure
4 are nearly imperceptible, it is likely possible to use a combination of
$\zeta$ and $\epsilon_{min}$ to achieve an output with a uniform density. For
reasons of exposition, an example of this being targeted to flatten the
density of these two regions is shown in Figure 5.

Figure 5: Reduced 3D sample dataset using $\zeta=1$ and $\epsilon_{min}=0.2$,
applying an overall reduction factor, $f=10.0$. As shown, the density
distributions within the two upper regions are more uniform due to the change
in $\epsilon_{min}$, demonstrating its ability to remove high density spikes
resulting from highly repetitive events occurring in the data.

####  4.1.2 3D dataset neural network training

The next step is the training NNs on these reduced datasets, using the same
feed-forward NN model and training pipeline used previously VanDePlassche2020
(7). For each dataset, 10 individual NNs are trained to form a committee of
NNs. This allows an estimate of the regression uncertainties due to the random
initialization of the training algorithm Ho2021 (21). The NN contains 3 hidden
layers, having 30 neurons per layer and each neuron using the $\tanh$
activation function. The Adam optimization algorithm and recommended settings
were used in the training pipeline. This standardization of the training
methodology within this study allows any noticeable trends in regression
performance to be attributed primarily to the reduction of the training
dataset.

The final step is to compare the NNs trained using the reduced dataset to the
one trained using the original dataset. This is done using the two standard NN
training metrics, namely the root-mean-squared error (RMSE) and the mean
absolute error (MAE). These values are evaluated for each individual NN and
then combined into a single mean and standard deviation per reduction factor,
$f$, varied within this exercise by altering the global reduction ratio
parameter, $a$. The remaining parameters were set to the recommended values
listed in Table 1 along with $MinPts=2$ and $n_{\text{min}}=6$. The difference
between these values and the recommended values are because the latter have
been refined since this test was performed, with the refined values shown in
the table. Additionally, the RMSE and MAE metrics are computed with respect to
the entirety of the original “non-reduced” dataset, in order to allow the
impact on regression quality to be separated from the overall statistics of
the dataset when interpreting the results.

Figures 6 and 7 display the RMSE and the MAE, respectively, as function of the
reduction factor, $f$. For both metrics, no significant increase in error is
detected at relatively low reduction factors. This is taken to imply that the
reduced dataset retains the key features needed for an equivalent NN
regression to be trained are still present. For this dataset, a steady
increase in RMSE and MAE is detected for $f\geq 15$, showing that the two-step
algorithm is reducing so much that essential dataset features for an
equivalent NN regression are no longer captured. This result shows that the
original dataset is likely over-populated for the features present in the
underlying functions for the purposes of training NN regressions.

Figure 6: Root mean squared error (RMSE) for the surrogate models trained
using the reduced 3D dataset compared against one trained using the non-
reduced dataset, evaluated across all of the input points in non-reduced
dataset. By interpreting this metric as a measure of the general
reproducability of the surrogate model using the reduced dataset, a
significant loss in model accuracy is observed at a specific overall reduction
factor, $f\simeq 15$. Figure 7: Mean absolute error (MAE) for the surrogate
models trained using the reduced 3D dataset compared against one trained using
the non-reduced 3D dataset, evaluated across all of the input points in the
non-reduced dataset. By interpreting this metric as a measure of the general
reproducability of the surrogate model using the reduced dataset, a
significant loss in model accuracy is observed at a specific overall reduction
factor, $f\simeq 15$.

###  4.2 Extension to target 16D application

The experimental data used in this test case was provided by the Joint
European Torus (JET) experimental fusion plasma device, located in Culham, UK.
This data is processed as described in Ref. Ho2019 (22). This has led to a
dataset with 15 input dimensions, which are then used to calculate particle,
heat and momentum fluxes with the use of QuaLiKiz Bourdelle2007 (18). For the
proof-of-principle extension, the ion heat flux, $q_{i,\text{ITG}}$, was again
chosen as the single output dimension, making this a 16D dataset. The two-
step-clustering algorithm was applied to this dataset to reduce it and
additional NN surrogate models were trained from those reduced datasets.

Similar to the previous 3D toy case, these “clustered” NNs were then compared
to the original model via the RMSE and MAE metrics computed with respect to
the entire non-reduced dataset. Also similarly, the reduction algorithm was
executed over a number of different reduction factors by varying the global
reduction ratio parameter, $a$, while setting the remaining parameters
according to the recommendations in Table 1 along with $MinPts=2$ and
$n_{\text{min}}=6$. As shown in Figures 8 and 9, the resulting RMSE and MAE
metrics exhibit similar features except with increase occurring for $f\geq
25$.

Figure 8: Root mean squared error (RMSE) for the surrogate models trained
using the reduced 16D dataset compared against one trained using the 16D non-
reduced dataset, evaluated across all of the input points in non-reduced
dataset. By interpreting this metric as a measure of the general
reproducability of the surrogate model using the reduced dataset, a
significant loss in model accuracy is observed at a specific overall reduction
factor, $f\simeq 25$. Figure 9: Mean absolute error (MAE) for the surrogate
models trained using the reduced 16D dataset compared against one trained
using the 16D non-reduced dataset, evaluated across all of the input points in
non-reduced dataset. By interpreting this metric as a measure of the general
reproducability of the surrogate model using the reduced dataset, a
significant loss in model accuracy is observed at a specific overall reduction
factor, $f\simeq 25$.

A more direct comparison of the two NNs at the elbow ($f\simeq 25$) can be
seen in Figure 10. This application uses an ensemble of NNs, each identical in
architecture but trained on the same dataset except with a different
initialization. The training dataset was filtered in a way to ensure that
high-density regions in the dataset are strongly correlated with low ensemble
variance Ho2021 (21). As expected, there is good agreement between the NNs
trained on the original and the reduced dataset where the predicted variance
is the lowest. However, the discrepancy between the two NN ensembles are most
noticeable in the low-density regions, where the noise filtering in the
reduction process likely removed a significant portion of data. While it can
be argued that this noise filtering has improved the match between the NN and
the original data in the high-density region, the overall appropriateness of
this noise filtering is strongly dependent on the application and its
investigation is outside the scope of this study. However, the ability to
adjust $\epsilon_{0}$ when running the algorithm allows some control over the
denoising process for such an investigation.

Figure 10: Comparison of the ion heat flux as a function of one input
parameter in the 15D dataset, $q$, between the original NN (blue line),
reduced NN (red line) and the underlying physics model (green points). The
reduction did not significantly affect the high-density regions, indicated by
the locations with small uncertainty ranges, but did reduce the fit quality in
the low-density region. A reduction of $f\simeq 25$ was applied to the data to
produce this result.

##  5 Conclusions

In this work, a two-step algorithm is proposed for the detection and reduction
of high data density regions within a larger $N$-dimensional dataset. This
algorithm is based on a combination of the widely-used clustering algorithms,
DBSCAN and $k$-means, applied sequentially and iteratively to determine an
density-dependent reduction ratio. The DBSCAN algorithm is first used to
detect clusters of high data density, and the information about those clusters
is used by $k$-means to partition them into representative regions for
reduction. This work is novel not only in that it proposes a robust data
detection and reduction scheme, but also proposes using DBSCAN iteratively to
distinguish between regions of varying density.

An application of the two-step clustering reduction algorithm was performed,
reducing the training dataset previous used for training NN surrogate models
of fusion plasma turbulent transport phenomena. The resulting analysis showed
that the dataset in this application could potentially be reduced by a factor
of $\sim$20 before significantly impacting the accuracy of the NN surrogate
model. This has implications in lowering the computation expense of generating
these datasets and/or expanding the parameter space included in such dataset.
The method is also envisioned to help with maintaining a living dataset, in
which new data can be continuously accumulated and trimmed without fear of
unintentionally biasing the dataset. Future work can be performed to improve
the generalizability and robustness of the $k$ calculation, in order to remove
extraneous parameters to the algorithm. One such example is implementing an
internal calculation of $\tau$ such that $\zeta$ moves between applying an
approximately uniform reduction factor to enforcing an approximately uniform
dataset density. Overall, the authors see many potential use-cases for this
algorithm in various machine learning applications where the dataset is either
collected on-the-fly or via a top-down approach.

##  6 Source Code

The complete code can be found at:
%\documentclass[twocolumn]{svjour3}␣␣␣␣␣␣␣␣␣␣%␣twocolumn%\smartqed␣␣%␣flush␣right␣qed␣marks,␣e.g.␣at␣end␣of␣proofhttps://www.qualikiz.com
and https://gitlab.com/BartKremers/two-step-clustering.

## References

  * (1) Derek Groen, Stefan J. Zasada and Peter V. Coveney  “Survey of multiscale and multiphysics applications and communities”  In _Computing in Science and Engineering_ 16.2 IEEE, 2014, pp. 34–43  DOI: 10.1109/MCSE.2013.47
  * (2) James L. Suter, Derek Groen and Peter V. Coveney  “Chemically specifi C multiscale modeling of clay-polymer nanocomposites reveals intercalation dynamics, tactoid self-assembly and emergent materials properties”  In _Advanced Materials_ 27.6, 2015, pp. 966–984  DOI: 10.1002/adma.201403361
  * (3) Hannan Tahir et al.  “Endothelial repair process and its relevance to longitudinal neointimal tissue patterns: Comparing histology with in silico modelling”  In _Journal of the Royal Society Interface_ 11.94, 2014  DOI: 10.1098/rsif.2014.0022
  * (4) D. Groen et al.  “Flexible composition and execution of high performance, high fidelity multiscale biomedical simulations”  In _Interface Focus_ 3.2, 2013  DOI: 10.1098/rsfs.2012.0087
  * (5) Alfons G. Hoekstra et al.  “Towards the virtual artery: A multiscale model for vascular physiology at the physics-chemistry-biology interface”  In _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 374.2080, 2016  DOI: 10.1098/rsta.2016.0146
  * (6) R. Delgado-Buscalioni and P.. Coveney  “Continuum-particle hybrid coupling for mass, momentum, and energy transfers in unsteady fluid flow”  In _Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics_ 67.4, 2003, pp. 13  DOI: 10.1103/PhysRevE.67.046704
  * (7) K.. Van De Plassche et al.  “Fast modeling of turbulent transport in fusion plasmas using neural networks”  In _Physics of Plasmas_ 27.2, 2020, pp. 1–17  DOI: 10.1063/1.5134126
  * (8) S. Alowayyed et al.  “Patterns for High Performance Multiscale Computing”  In _Future Generation Computer Systems_ 91 Elsevier B.V., 2019, pp. 335–346  DOI: 10.1016/j.future.2018.08.045
  * (9) Martin Ester and Hans Kriegel  “Density-Based Clustering Methods”  In _Comprehensive Chemometrics_ 2, 2009, pp. 635–654  DOI: 10.1016/B978-044452701-1.00067-3
  * (10) Stuart P. Lloyd  “Least Squares Quantization in Pcm”  In _IEEE Transactions on Information Theory 28 (2)_ IEEE, 1982, pp. 129–37
  * (11) Jeong Hun Kim, Jong Hyeok Choi, Kwan Hee Yoo and Aziz Nasridinov  “AA-DBSCAN: an approximate adaptive DBSCAN for finding clusters with varying densities”  In _Journal of Supercomputing_ 75.1 Springer US, 2019, pp. 142–169  DOI: 10.1007/s11227-018-2380-z
  * (12) Chen Xiaoyun, Min Yufang, Zhao Yan and Wang Ping  “GMDBSCAN: Multi-density DBSCAN cluster based on grid”  In _IEEE International Conference on e-Business Engineering, ICEBE’08 - Workshops: AiR’08, EM2I’08, SOAIC’08, SOKM’08, BIMA’08, DKEEE’08_ IEEE, 2008, pp. 780–783  DOI: 10.1109/ICEBE.2008.54
  * (13) Mohammad Mahmudur Rahman Khan, Md Abu Bakr Siddique, Rezoana Bente Arif and Mahjabin Rahman Oishe  “ADBSCAN: Adaptive density-based spatial clustering of applications with noise for identifying clusters with varying densities”  In _4th International Conference on Electrical Engineering and Information and Communication Technology, iCEEiCT 2018_ , 2019, pp. 107–111  DOI: 10.1109/CEEICT.2018.8628138
  * (14) Hua Jiang et al.  “A new hybrid method based on partitioning-based DBSCAN and ant clustering”  In _Expert Systems with Applications_ 38.8 Elsevier Ltd, 2011, pp. 9373–9381  DOI: 10.1016/j.eswa.2011.01.135
  * (15) Xiaoming Chen, Wanquan Liu, Huining Qiu and Jianhuang Lai  “APSCAN: A parameter free algorithm for clustering”  In _Pattern Recognition Letters_ 32.7 Elsevier B.V., 2011, pp. 973–986  DOI: 10.1016/j.patrec.2011.02.001
  * (16) Mihael Ankerst, Markus M. Breunig, Hans Peter Kriegel and Jörg Sander  “OPTICS: Ordering Points to Identify the Clustering Structure”  In _SIGMOD Record (ACM Special Interest Group on Management of Data)_ 28.2, 1999, pp. 49–60  DOI: 10.1145/304181.304187
  * (17) Alex Rodriguez and Alessandro Laio  “Clustering by fast search and find of density peaks”  In _Science_ 344.6191, 2014, pp. 1492–1496  DOI: 10.1126/science.1242072
  * (18) C. Bourdelle et al.  “A new gyrokinetic quasilinear transport model applied to particle transport in tokamak plasmas”  In _Physics of Plasmas_ 14.11, 2007  DOI: 10.1063/1.2800869
  * (19) J Citrin et al.  “Tractable flux-driven temperature, density, and rotation profile evolution with the quasilinear gyrokinetic transport model QuaLiKiz”  In _Plasma Physics and Controlled Fusion_ 59.12, 2017, pp. 124005  DOI: 10.1088/1361-6587/aa8aeb
  * (20) C Bourdelle et al.  “Core turbulent transport in tokamak plasmas: bridging theory and experiment with QuaLiKiz”  In _Plasma Physics and Controlled Fusion_ 58.1, 2016, pp. 014036  DOI: 10.1088/0741-3335/58/1/014036
  * (21) A. Ho et al.  “Neural network surrogate of QuaLiKiz using JET experimental data to populate training space”  In _Physics of Plasmas_ 28.3, 2021, pp. 032305  DOI: 10.1063/5.0038290
  * (22) A. Ho et al.  “Application of Gaussian process regression to plasma turbulent transport model validation via integrated modelling”  In _Nuclear Fusion_ 59.5, 2019  DOI: 10.1088/1741-4326/ab065a

Scitovski2021