# Phonon thermal transport shaped by strong spin-phonon scattering in a Kitaev
material Na2Co2TeO6

Xiaochen Hong1,2,∗ Matthias Gillig2 Weiliang Yao3 Lukas Janssen4,† Vilmos
Kocsis2 Sebastian Gass2 Yuan Li3,5 Anja U. B. Wolter2 Bernd Büchner2,6
Christian Hess1,2,‡ 1Fakultät für Mathematik und Naturwissenschaften,
Bergische Universität Wuppertal, 42097 Wuppertal, Germany
2Leibniz-Institute for Solid State and Materials Research (IFW-Dresden), 01069
Dresden, Germany
3International Center for Quantum Materials, School of Physics, Peking
University, 100871 Beijing, China
4Institut für Theoretische Physik and Würzburg-Dresden Cluster of Excellence
$ct.qmat$, Technische Universität Dresden, 01062 Dresden, Germany
5Collaborative Innovation Center of Quantum Matter, 100871 Beijing, China
6Institute of Solid State and Materials Physics and Würzburg-Dresden Cluster
of Excellence $ct.qmat$, Technische Universität Dresden, 01062 Dresden,
Germany

The recent report of a half-quantized thermal Hall effect in the Kitaev
material $\alpha$-RuCl3 has sparked a strong debate on whether it is generated
by Majorana fermion edge currents or whether other more conventional
mechanisms involving magnons or phonons are at its origin. A more direct
evidence for Majorana fermions which could be expected to arise from a
contribution to the longitudinal heat conductivity $\kappa_{xx}$ at
$T\rightarrow 0$ is elusive due to a very complex magnetic field dependence of
$\kappa_{xx}$. Here, we report very low temperature (below 1 K) thermal
conductivity ($\kappa$) of another candidate Kitaev material, Na2Co2TeO6. The
application of a magnetic field along different principal axes of the crystal
reveals a strong directional-dependent magnetic-field ($\bf B$) impact on
$\kappa$. We show that no evidence for mobile quasiparticles except phonons
can be concluded at any field from 0 T to the field polarized state. In
particular, severely scattered phonon transport is observed across the $B-T$
phase diagram, which is attributed to prominent magnetic fluctuations.
Cascades of phase transitions are uncovered for all $\bf B$ directions by
probing the strength of magnetic fluctuations via a precise record of
$\kappa$($B$). Our results thus rule out recent proposals for itinerant
magnetic excitations in Na2Co2TeO6, and emphasise the importance of
discriminating true spin liquid transport properties from scattered phonons in
candidate materials.

Figure 1: Thermal conductivity of Na2Co2TeO6 and the impact of the in-plane
magnetic fields. a, In-plane thermal conductivity of five Na2Co2TeO6 crystals
without magnetic field. The thermal gradient ($\nabla T$) was set
$\bf\parallel$ $zigzag$ for Sample#1 and Sample#4, and $\bf\parallel$
$armchair$ for the other three samples. By extrapolating the data to $T=$ 0 K,
a residual linear term ($\kappa/T|_{T\rightarrow 0}$) can be resolved for all
five $\kappa/T$($T$) curves, especially clear for Sample#1, as indicated by
the dashed blue line. b, The field dependence of the $\kappa/T$($T$) curves
for Sample#1, focusing on the small field limit which covers a small portion
of c, the full field dependence of the $\kappa/T$($T$) curves, as indicated by
the gray background. At $T=$ 1 K, the $\kappa$ of Sample#1 experience a
70-fold increase from $B=$ 0 T to the saturation field. d, The $\kappa/T$($T$)
curves display a typical phononic $\propto T^{\alpha}$ behavior with $\alpha=$
1.5 at high enough fields. The field direction does not affect the saturated
$\kappa/T$($T$) curves.

Frustrated magnets refer to a set of materials that do not order down to
temperatures well below the corresponding magnetic interaction strength. In
some cases, the system is left with a ground state called quantum spin liquid
(QSL), which is characterized by entangled spins, topological orders, and
fractionalised magnetic excitations Balents ; Savary ; Zhou . Kitaev QSL,
proposed for bond-dependent nearest-neighbour interacting spins on the two-
dimensional honeycomb lattice Kitaev , is among the rare examples of exactly
solvable QSL models YaoH ; McGreevy ; Wen ; Moessner . Recent research
interest is further fueled by possible achieving Kitaev QSL in realistic
materials. In particular, fingerprints of novel QSL-related excitations were
revealed in $\alpha$-RuCl3 Takagi ; Broholm , both in its zero-field phase
above the magnetic order Banerjee , and more intriguingly, in its field-
induced quantum disordered state Kasahara .

However, those sought-after signatures for QSLs observed in $\alpha$-RuCl3
always come along with alternative explanations. To be more specific on
transport experiments, the report of half-integer quantization of thermal Hall
conductance was once regarded as a definitive proof for the existence of
Majorana fermions in the field-induced quantum disordered state in
$\alpha$-RuCl3 Kasahara . However, following experiments noticed such
quantisation is less reproducible, and attributed the non-quantised
temperature-dependent thermal Hall effect to phonons Lefrancois , or
topological magnons CzajkaNM . On the other hand, quantum oscillations of
longitudinal thermal conductivity in the quantum disordered state of
$\alpha$-RuCl3 suggest some fractionalised magnetic excitations forming a
Fermi surface CzajkaNP . But such spectacle is also argued to be a result
simply due to scattered phonons Lefrancois , in particular taking into account
that no itinerant magnetic excitation has been found in a previous low
temperature work YuYJ . On top of all these inconsistencies are the facts that
$\alpha$-RuCl3 suffers from the notorious stacking fault problem ZhangHD ,
from the dechlorination and oxidation after heat treatment Breitner , and more
generally from sample dependence Kasa . Thus, it is indispensable to seek for
another example that can contribute to clarify to what extent these results
are intrinsic to a Kitaev material.

In this paper, we studied the ground state properties of a quantum magnet
Na2Co2TeO6 (NCTO), which was recently investigated as a candidate Kitaev QSL
LiuHM ; Lin ; Hong , through the prism of low temperature thermal
conductivity. Similar to its more famous cousin $\alpha$-RuCl3, NCTO also
features an intermediate quantum disordered phase sandwiched between the low
field magnetically ordered state and the high-field polarized state Lin ; Hong
, where a QSL state is prospected. In this work, we report an anisotropic and
astonishing large $\bf B$ impact on $\kappa$ at low temperatures ($<1$ K). The
analysis of our data demonstrates that there is $no$ evidence for mobile
gapless QSL excitations ($\kappa_{QSL}$) contributing to $\kappa$. Instead,
strongly scattered phonon transport ($\kappa_{ph}$), a quite traditional
mechanism, is responsible for all the observed phenomena. Utilizing the high
resolution $\kappa(B)$ data, we reveal a large density of magnetic
excitiations in the zero-field ground state of NCTO, further promoted
excitations by magnetic field perpendicular to the honeycomb plane, and
possible quantum criticality driven by in-plane magnetic field.

The residual linear term of thermal conductivity extrapolated to the low
temperature limit ($\kappa_{0}/T\equiv\kappa/T|_{T\rightarrow 0}$) is
persistently scrutinized for QSL candidates Zhou ; Yamashita2012 , since its
existence is practically regarded as a criterion for itinerant fractionalised
magnetic excitations Li2020 ; PanBY ; Hartstein ; Yamashita2010 ; Sato . Fig.
1a shows $\kappa/T$ of five NCTO crystals in zero magnetic field as a function
of $T$. Albeit certain sample dependence, the $\kappa/T(T)$ profiles of all
investigated crystals do not follow the simple power-law behavior expected for
common insulators. Instead, regardless of the direction along which the
thermal gradient $\nabla T$ is established (the two in-plane principal axes
for the Co-honeycomb lattice, denoted as $zigzag$ and $armchair$ directions,
indicated by the schematic insets), all $\kappa/T(T)$ profiles trend to end
with a finite $\kappa_{0}/T$.

Among them, Sample#1 is especially notable for displaying a pronounced hump
which emerges below about 250 mK. $\kappa_{0}/T$ of Sample#1, extracted from a
linear fit to its data above 250 mK, as highlighted by the dashed line in Fig.
1a, is 0.048 mW/K-2cm-1. At first glance, this finite $\kappa_{0}/T$ is
consistent with the notion that the ground state of NCTO is a QSL, since
$\kappa_{QSL}$ contributed by the putative fractionalized magnetic excitations
is expected to scale linearly with $T$ at the lowest temperatures
Yamashita2012 . We are aware of a recent work reporting similar data of NCTO,
where these results have been interpreted as evidences for $\kappa_{QSL}$
Guang . However, as clearly shown in Fig. 1a, the upturn of $\kappa/T(T)$ for
Sample#1 (same for Sample#4 and Sample#5) terminates abruptly below about 100
mK, followed by a sharp drop, which results in a vanishing residual linear
term that is incompatible with a gapless QSL ground state. Furthermore, we
argue such observation is at odds with a hypothesised gapped QSL state with a
gap size of the order $\sim$ 100 mK. When a small in-plane magnetic field is
applied, $\kappa_{QSL}/T$($T$) at temperatures below the initial gap energy
scale is expected to increase due to a proliferation of quasiparticle
excitations, while $\kappa/T$($T$) at higher temperatures should be less
sensitive to the field since $\kappa_{ph}$ is not directly affected. This is
just opposite to our observations. As depicted in Fig. 1b, the field changes
$\kappa/T$($T$) fundamentally above 100 mK, while it has basically no impact
on the data at the lowest temperature.

Figure 2: Anisotropic in-plane magnetic field effects on the thermal
conductivity. Panels a, b, and c, plot the normalized $\kappa$($B$) profiles
of different combinations of ($\nabla T$, $B$) orientations at selected
temperatures. The purple curves and the green curves depict the data collected
with field increasing and decreasing, respectively. All $\kappa$($B$) curves
experience a saturation at high fields, signaling that a polarized state was
reached in this study. Panels a and b depict the results of the same crystal
(Sample#1) with the same $\nabla T$ orientation ($\nabla T\parallel zigzag$)
but with different $\bf B$ directions. They behave fundamentally different.
Meanwhile Panel b resembles Panel c which is the data collected from another
crystal (Sample#2) with a different $\nabla T$ orientation ($\nabla T\parallel
armchair$) but the same field direction ($\bf B\parallel$ $armchair$). As
highlighted in the inset of Panel c, an oscillation-like feature can be
distinguished before an abrupt increase of the $\kappa$($B$) for the field-up
ramp. The anomalies in all $\kappa/T$($B$) curves are marked by different
symbols. d, The temperature evolution of these anomalies of $\kappa$($B$)
profiles. They are grouped into different categories as glided by the
background stripes. The field values for Sample#2 are multiplied by 96%, in
order to compromise the possible (out-of-plane) misalignment. e, A
representative case of analysing the renormalized phonon-spin scattering
strength ($r_{sp}$/$r_{p}$, see text). The peak at 7.5 T and 10.2 T (the field
values for Sample#2 are multiplied by 96% as mentioned above) are fit to
standard Gaussian function and skewed Gaussian function, respectively. The
height and span of the peaks ($h_{0}$, $w_{0}$ for the 7.5 T peak, $h_{1}$,
$w_{1}$, and $w_{2}$ for the 10.2 T peak) are extracted as shown in the
figure, and are summarized in Panel f and Panel g, respectively.

Higher magnetic fields have a drastic impact on the $\kappa(T)/T$ curves until
they finally saturate into a power-law behavior over the whole temperature
range at the highest field. A representative case of Sample#1 is shown in Fig.
1c, along with the others displayed in Supp. Note 1 SM . As can be seen in
Fig. 1d, regardless of the (in-plane) field directions, $\kappa(T)/T$ curves
for a given sample saturate at the same value.

Recent high-field magnetization experiments found the field-polarised state is
reached for in-plane magnetic field higher than 10 T in NCTO Zhang . As a
result, spins can not contribute anymore, neither directly, nor indirectly
through scattering at the highest field reached in this study. The saturated
$\kappa(T)/T$ thus represents a standard ballistic phonon heat conductivity
$\kappa_{ph}$ in the low-temperature limit with $\kappa_{ph}/T\propto
T^{\alpha}$, of which the exponent $\alpha$ = 1.5. The sample independent
$\alpha<2$ indicates that the phonon reflection at the sample surface is not
perfectly diffuse, and should be attributed to the intrinsic surface roughness
of the as-grown NCTO crystals. One can thus take the $\kappa(T)/T$ curves at
highest in-plane field as the intrinsic phonon thermal conductivity
$\kappa_{ph}/T(T)$ free from the spin-phonon scattering.

As can be concluded from Fig. 1 and Supp. Fig. 1, there is no evidence for
finite $\kappa_{0}/T$ at any field. Furthermore, $\kappa/T(T)$ profiles at
zero and low fields never exceed the saturated $\kappa_{ph}/T(T)$ over the
whole temperature range. Hence, any evidence for a direct magnetic thermal
transport channel on top of $\kappa_{ph}$, including $\kappa_{QSL}$, is
elusive. If present, it must be negligibly small since the total measured
$\kappa$ at zero field is already at a very small value and at finite fields
the saturated $\kappa_{ph}$ is never exceeded. Hence, the intricate landscape
of $\kappa(T)/T$ and its field dependence can be reasonably accounted for by
strong phonon-spin scattering, which is the main theme of the rest of the
paper.

The field dependence of $\kappa/T$($T$) is non-monotonic, see Fig. 1c. A
closer inspection can be performed by isothermal $\kappa/T$($B$) measurements.
Fig. 2a-c exhibit the representative data of two samples with three different
($\nabla T$, $\bf B$) direction combinations. A straightforward conclusion is
that $\kappa(B)$ is dictated by the $\bf B$ direction, irrelevant to the
$\nabla T$ direction. That further underlines the dominance of scattered
$\kappa_{ph}$ in NCTO over the whole field-temperature phase space. More
specifically, a magnetic field applied along the $zigzag$ and $armchair$
directions on the same sample results in sharply different $\kappa$($B$)
profiles. For $\bf B\parallel$ $zigzag$ (Fig. 2a), there is strictly $no$
hysteresis behavior. The overall feature turns out to be a broad dip centered
around 10.2 T, with many minor anomalies marked by different symbols in the
figure. For $\bf B\parallel$ $armchair$ (Fig. 2b, Fig. 2c and Supp. Fig.
2e-2k), there are two very sharp dips at 7.5 T and 10.2 T. In contrast to the
$\bf B\parallel$ $zigzag$ case, a large hysteresis between 3 T and 6.5 T is
evident. The abrupt increase of $\kappa$ in the up-ramp isotherm points to a
first-order-like transition. Note that the $\bf B$ direction dependent
occurrence of hysteresis was already mentioned by higher temperature
investigations Hong ; Yao2020 . More discussion on the hysteretic behavior can
be found in the Supp. Note 2 SM . With both $\bf B$ directions, $\kappa$
saturates at high enough fields, as indicated by the dashed lines.

Figure 3:  Further suppression of $\kappa$ by out-of-plane magnetic fields. a,
The in-plane $\kappa$ of Sample#3 under out-of-plane magnetic fields. The
geometry is sketched in the inset. $\nabla T$ is along the $armchair$
direction, and $\bf B$ is perpendicular to the honeycomb plane. A rapid field
suppression of $\kappa/T$ is recognized at finite temperatures below about 3
T. At higher fields, the $\kappa/T$($T$) curves changes nonmonotonically in a
narrow range. b, The $\kappa/T$($B$) profiles at three representative
temperatures. They clearly show a fast drop at field 2 T $<B<$ 4 T, followed
by an oscillation-like feature at higher fields. As highlighted in the inset,
the oscillation profiles change little with temperature.

All the salient features in Fig. 2a-c and Supp. Fig. 2 are summarized in Fig.
2d. Beside the saturation field, which increases roughy linearly with
temperature, the other anomalies are basically temperature independent. Such
rich phase diagram suggests the landscape of successive $\bf B$ induced phase
transitions of the Heisenberg-Kitaev model might be inherited by NCTO Janssen
. Remarkably, three different finite-temperature anomalies marked by the
yellow, red, and blue shaded areas seem to merge at around 10.2 T at 0 K,
suggesting a quantum critical point. We expect the presented result to
motivate further theoretical studies that account for specified parameters of
NCTO in order to demystify these phases.

Some theoretical exploration on the Heisenberg-Kitaev honeycomb model have
already revealed the potentially very rich phase diagram tuned by magnetic
field Janssen . Of particular interest is a multi-Q phase that can be
stabilized in finite fields Janssen . Recent experiments proposed the zero-
field ground state of NCTO could be a triple-Q order Chen ; Lee ; Kruger ;
Yao2022 ; Kikuchi , potentially stabilized by the proposed ring exchange
interactions Kruger . If confirmed, it could be a natural explanation of the
extensive scattering of $\kappa_{ph}$ at zero field, since a triple-Q ordered
state guarantees larger density of state (DOS) of excitations around zero
momentum ($\Gamma$ point) which interact strongly with phonons Kruger . These
magnetic excitations should in principle contribute to specific heat. A decent
experiment to exam this property is still pending for NCTO since heat capacity
measurement at very low temperature is technically challenging Wilhelm .
Nevertheless, in Supp. Note 3 we argue our raw data can provide some indirect
evidences for huge specific heat in the zero-field ground state SM . Our data
thus corroborate the notion of very large magnetic DOS as is expected for a
triple-Q state.

Next we comment on the oscillation-like $\kappa$($B$) in the intermediate
field range when $\bf B\parallel$ $armchair$, as highlighted in the inset of
Fig. 2c. A similar feature exists but is less obvious in another sample and at
higher temperatures, see Supp. Fig. 2g, 2h, and 2j SM . Gapless QSL states
that foster quantum oscillation of presumably charge neutral magnetic
excitations are indeed anticipated for the Kitaev model Kitaev ; Gohlke ;
Hickey , and was invoked to interpret the $\kappa$($B$) oscillations in
$\alpha$-RuCl3 CzajkaNP . Nevertheless, we believe quantum oscillations of
$\kappa_{QSL}$ should be irrelevant to NCTO. First of all, the oscillation
feature in NCTO exists in the hysteresis region, thus in a magnetically
ordered state. Besides, the fact that there is no evidence for itinerant
$\kappa_{QSL}$ at any field does not favor a quantum oscillation picture.
Additionally, it is hard to imagine a significant direct contribution of
$\kappa_{QSL}$ which is exempted from the strong fluctuations that results in
a reduction of $\kappa_{ph}$ by orders of magnitude. With this in mind, more
caution should be taken for uniqueness of possible Kitaev QSL signatures
observed in this material class, since thermal transport properties of NCTO
share impressively similar features with $\alpha$-RuCl3 Hentrich ; Hentrich2 ;
Hong ; Gillig .

In view of the scattered phononic thermal conductivity has been revealed for
NCTO, it is interesting to utilize the field evolution of $\kappa/T$ as a
natural probe for the phonon-spin scattering in this material Linarite ; YCOB
. The relatively simple profiles of the $\bf B\parallel$ $armchair$ cases
(Fig. 2b and 2c) allow a straightforward analysis. According to Matthiessen’s
rule, the total phonon scattering rate is the sum of all independent
ingredients, in this case the phonon-spin scattering ($r_{sp}$) and the
intrinsic phononic scattering ($r_{p}$). At very low temperatures, $r_{p}$ is
determined only by the crystal size and is temperature independent because the
phonon wave length diverges. It is the only scattering mechanism at play in
the field polarized phase in the temperature range covered by this study. So
the renormalized $r_{sp}$/$r_{p}$ ($\equiv\kappa_{ph}/\kappa(B)-$1) is an
index reflecting the magnetic scattering strength YCOB . Fig. 2e exhibits one
representative case for Sample#2 at about 260 mK. The $B=7.5$ T peak is
perfectly reproduced by a standard Gaussian distribution, which indicates
randomly distributed disorder. However, the $B=10.2$ T peak can only be
reasonably fit by a skewed Gaussian distribution with different width at half
maximum on both sides ($w_{1}\neq w_{2}$). The extracted parameters are
summarized in Fig. 2f and Fig. 2g. The fact that $w_{1}>w_{2}$ might be key
for identifying the actual type of phases surrounding the 10.2 T criticality,
which needs material specific calculations to unravel.

Finally, at low temperatures, when $\bf B$ is applied perpendicular to the
NCTO honeycomb plane, $\kappa$ gets further suppressed beyond a critical field
of $B\approx 3$ T, as shown in Fig. 3a. In the language of magnetic
scattering, it means the already strong magnetic fluctuations at zero field
are even further promoted, contradictory to the intuition that field alignment
of the magnetic moments should reduce them.

Indeed, there are some theoretical considerations predicting $\bf B\parallel$
$c$ can lead to abundant phase transitions in the original Kitaev model and
its variants Li ; Sorensen ; Chern ; Gohlke ; Janssen . Especially, quantum
fluctuations in some of these intermediate phases are expected to be rather
strong Chern , in line with our observations that $\kappa_{ph}$ gets further
suppressed compared to the zero-field phase. As displayed more clearly in Fig.
3b, $\kappa/T$($B$) experiences a rapid drop between 2 T and 4 T. It is non-
monotonic at higher field, featured by a broad hump around 8 T and a broad dip
around 15 T, highlighted by the inset. We note the anomaly at about $15$ T can
be related to phase transitions revealed by a recent magnetization work Zhang
, while the rapid drop between $2$ T and $4$ T do not have a similar
correspondence.

To conclude, our high-quality low-temperature thermal transport study on NCTO
single crystals established the dominance of phonon thermal transport in its
magnetic ground states, and exclude itinerant magnetic excitations of any
form. The field evolution of $\kappa$, which manifests the strength of phonon-
spin scattering, was taken as a probe to elucidate the intricate magnetic
phases in NCTO. The strongly scattered phonons in the zero-field ground state
indicates large DOS of magnetic excitations, in line with the proposal of a
triple-Q state in NCTO. The in-plane $\kappa$ depends significantly of the in-
plane field direction, underlines the bond-dependent interactions of a Kitaev
material. However, a criticality at $10.2$ T is present regardless of the in-
plane field direction. Field perpendicular to the honeycomb plane also induces
multiple magnetic phases, where phonon-spin scattering is even more prominent
than the zero-field phase. We expect similar physics should be quite common
among QSL candidate materials, since QSLs are intrinsically at the edge of
competing orders. The pervasive phase boundaries should be careful considered
when analysing existing and future experimental results of QSL candidate
materials.

Methods
Sample preparation
Single crystals of Na2Co2TeO6 were prepared with a modified flux method.
Na2CO3, Co3O4 and TeO2 powders were grounded and loaded into an alumina
crucible. Excess amount of TeO2 served as a self-flux. The crucible was heated
up to 1050∘C, kept for two days, before cooling down to 600∘C at a rate of
6.5∘C per hour. Ruby-colored hexagonal flakes of typical size
$\sim$10$\times$10$\times$0.1 mm3 can be mechanically collected out of bluish
violet residue. The harvested single crystals were further washed with a NaOH
solution. Basic characterizations of the as-grown crystals can be found in Ref
Yao2020 .
The samples were cut into a size of about $\sim$5$\times$1$\times$0.1 mm3, the
longest direction along which $\nabla T$ was generated are the $armchair$ or
$zigzag$ directions, respectively. In order to make thermal contacts to the
fresh surface, the out-most layers of the as-grown crystals were cleaved off
by a blade just before the silver paint was glued to it. As a result, the
thickness of the samples were reduced to about 60 $\mu$m.
Heat transport measurements
The thermal conductivities were measured in a dilution refrigerator, using a
standard four-wire steady-state method with two RuO2 chip thermometers,
calibrated $in$ $situ$ against a reference RuO2 thermometer. For fixed field
$\kappa(T)$ measurements, each sample was initially cooled down from room
temperature without magnetic field. The data were collected with gradually
increased magnetic field in order to avoid the complicated situation of
involving the hysteresis effect. $\kappa(T)$ data were collected in a steady-
state manner (see Supp. Fig. 3 SM ).
To measure the $\kappa(B)$ isotherms, the system was cooled from the
paramagnetic phase in zero field. Then the sample temperature was kept at a
set point, and the field was changed from 0 T to the highest field (up-ramp),
followed by decreasing to 0 T (down-ramp). The field was changed at a speed no
more than 20 mT per minute, in order to minimize its heating effect, and to
keep the system in a (quasi-)thermal equilibrium state. In some cases, the
field were ramped up and down multiple times, see Supp. Note 2 for more
details SM .
We wish to point out that according to our experience, NCTO seems to have very
strong anisotropic in-plane magnetic susceptibility at very low temperatures,
that tend to align the $armchair$ direction to the field. We are aware of a
recently posted paper which reports a direct measurement of this magnetic
anisotropy that underpins our conjecture Lin2022 . In order to avoid
experimental artifacts caused by such effect, the cooler side of all samples
were glued to the heat sink (a gilded silver bulk) directly with a strong
epoxy (Wakefield DeltaBond-152 two component adhesives). The samples and the
wires attached to them were checked carefully under a microscope after the
experiments, to confirm they had not bent during the measurements.

Thermodynamic measurements
The magnetization measurements were carried out using a SQUID magnetometer
(iHelium3, MPMS-XL, Quantum Design). The magnetostriction was measured using a
commercially available dilatometer (Standard probe, Kuechler) compatible with
the Quantum Design PPMS systems. These measurements are based on the
capacitance measurement technique (AH2700A, Andeen-Hagerling).

## References

  * (1) Balents, L., Spin liquids in frustrated magnets. _Nature_ 464, 199–208 (2010). https://doi.org/10.1038/nature08917
  * (2) Savary, L. $\&$ Balents, L., Quantum spin liquids: a review. _Rep. Prog. Phys._ 80, 106502 (2017). DOI 10.1088/0034-4885/80/1/016502
  * (3) Zhou, Y., Kanoda, K. $\&$ Ng, T.K. Quantum spin liquid states. _Rev. Mod. Phys._ 89, 025003 (2017). https://doi.org/10.1103/RevModPhys.89.025003
  * (4) Kitaev, A., Anyons in an exactly solved model and beyond. _Ann. Phys._ 321, 2-111 (2006). https://doi.org/10.1016/j.aop.2005.10.005
  * (5) Moessner, R., $\&$ Sondhi, S. L., Resonating Valence Bond Phase in the Triangular Lattice Quantum Dimer Model. _Phys. Rev. Lett._ 86, 1881 (2001). https://doi.org/10.1103/PhysRevLett.86.1881
  * (6) Wen, X.-G., Quantum Orders in an Exact Soluble Model. _Phys. Rev. Lett._ 90, 016803 (2003). https://doi.org/10.1103/PhysRevLett.90.016803
  * (7) Yao, H., Zhang, S.-C., $\&$ Kivelson, S. A., Algebraic Spin Liquid in an Exactly Solvable Spin Model. _Phys. Rev. Lett._ 102, 217202 (2009). https://doi.org/10.1103/PhysRevLett.102.217202
  * (8) Ben-Zion, D., Das, D., $\&$ McGreevy, J., Exactly solvable models of spin liquids with spinons, and of three-dimensional topological paramagnets. _Phys. Rev. B_ 93, 155147 (2016). https://doi.org/10.1103/PhysRevB.93.155147
  * (9) Takagi, H., Takayama, T., Jackeli, G., Khaliullin, G. $\&$ Nagler, S. E. Concept and realization of Kitaev quantum spin liquids. _Nat. Rev. Phys._ 1, 264–280 (2019). https://doi.org/10.1038/s42254-019-0038-2
  * (10) Broholm, C., Cava, R. J., Kivelson, S. A., Nocera, D. G. Norman, M. R. $\&$ Senthil, T., Quantum spin liquids. _Science_ 367, 263 (2020). DOI: 10.1126/science.aay0668
  * (11) Banerjee, A. _et al._ , Proximate Kitaev quantum spin liquid behavior in a honeycomb magnet. _Nat. Mat._ 15, 733–740 (2016). https://doi.org/10.1038/nmat4604
  * (12) Kasahara, Y. _et al._ , Majorana quantization and half-integer thermal quantum Hall effect in a Kitaev spin liquid. _Nature_ 559, 227–231 (2018). https://doi.org/10.1038/s41586-018-0274-0
  * (13) Lefrançois, É., Baglo, J., Barthélemy, Q., Kim, S., Kim, Y.-J. $\&$ Taillefer, L., Oscillations in the magnetothermal conductivity of $\alpha$-RuCl3: Evidence of transition anomalies. _Phys. Rev. B_ 107, 064408 (2023). https://doi.org/10.1103/PhysRevB.107.064408
  * (14) Czajka, P. _et al._ , Planar thermal Hall effect of topological bosons in the Kitaev magnet $\alpha$-RuCl3. _Nat. Mater._ 22, 36–41 (2023). https://doi.org/10.1038/s41563-022-01397-w
  * (15) Czajka, P. _et al._ , Oscillations of the thermal conductivity in the spin-liquid state of $\alpha$-RuCl3. _Nat. Phys._ 17, 915–919 (2021). https://doi.org/10.1038/s41567-021-01243-x
  * (16) Yu, Y.J. _et al._ , Ultralow-Temperature Thermal Conductivity of the Kitaev Honeycomb Magnet $\alpha$-RuCl3 across the Field-Induced Phase Transition. _Phys. Rev. Lett._ 120, 067202 (2018). https://doi.org/10.1103/PhysRevLett.120.067202
  * (17) Zhang, H. _et al._ , Stacking disorder and thermal transport properties of $\alpha$-RuCl3. _online preprint_ arXiv:2303.03682
  * (18) Breitner, F. A., Jesche, A., Tsurkan, A., $\&$ Gegenwart, P., Thermal decomposition of the Kitaev material $\alpha$-RuCl3 and its influence on low-temperature behavior. _online preprint_ arXiv:2303.11308
  * (19) Kasahara, Y. _et al._ , Quantized and unquantized thermal Hall conductance of the Kitaev spin liquid candidate $\alpha$-RuCl3. _Phys. Rev. B_ 106, L060410 (2022). https://doi.org/10.1103/PhysRevB.106.L060410
  * (20) Liu, H. and $\&$ Khaliullin, G., Pseudospin exchange interactions in $d^{7}$ cobalt compounds: Possible realization of the Kitaev model. _Phys. Rev. B_ 97, 014407 (2018). https://doi.org/10.1103/PhysRevB.97.014407
  * (21) Lin, G. _et al._ , Field-induced quantum spin disordered state in spin-1/2 honeycomb magnet Na2Co2TeO6. _Nat. Commun._ 12, 5559 (2021). https://doi.org/10.1038/s41467-021-25567-7
  * (22) Hong, X. _et al._ , Strongly scattered phonon heat transport of the candidate Kitaev material Na2Co2TeO6. _Phys. Rev. B_ 104, 144426 (2021). https://doi.org/10.1103/PhysRevB.104.144426
  * (23) Yamashita, M., Shibauchi, T. $\&$ Matsuda, Y., Thermal-transport studies on two-dimensional quantum spin liquids. _Chemphyschem_ 13, 74–78 (2012). https://doi.org/10.1002/cphc.201100556
  * (24) Yamashita, M. _et al._ , Highly Mobile Gapless Excitations in a Two-Dimensional Candidate Quantum Spin Liquid. _Science_ 328, 1246-1248 (2010). DOI: 10.1126/science.118820
  * (25) Hartstein, M. _et al._ , Fermi surface in the absence of a Fermi liquid in the Kondo insulator SmB6. _Nat. Phys._ 14, 166–172 (2018). https://doi.org/10.1038/nphys4295
  * (26) Li, N. _et al._ , Possible itinerant excitations and quantum spin state transitions in the effective spin-1/2 triangular-lattice antiferromagnet Na2BaCo(PO4)2. _Nat. Commun._ 11, 4216 (2020). https://doi.org/10.1038/s41467-020-18041-3
  * (27) Sato, Y. _et al._ , Charge-neutral fermions and magnetic field-driven instability in insulating YbIr3Si7. _Nat. Commun._ 13, 394 (2022). https://doi.org/10.1038/s41467-021-27541-9
  * (28) Pan, B. _et al._ , Unambiguous Experimental Verification of Linear-in-Temperature Spinon Thermal Conductivity in an Antiferromagnetic Heisenberg Chain. _Phys. Rev. Lett._ 129, 167201 (2022). https://doi.org/10.1103/PhysRevLett.129.167201
  * (29) Guang, S. K. _et al._ , Thermal transport of fractionalized antiferromagnetic and field-induced states in the Kitaev material Na2Co2TeO6. _Phys. Rev. B_ 107, 184423 (2023). https://doi.org/10.1103/PhysRevB.107.184423
  * (30) Supplementary Materials attached at the end of this file
  * (31) Zhang, S. _et al._ , Electronic and magnetic phase diagrams of Kitaev quantum spin liquid candidate Na2Co2TeO6. _online preprint_ arXiv:2212.03849
  * (32) Yao, W. $\&$ Li, Y., Ferrimagnetism and anisotropic phase tunability by magnetic fields in Na2Co2TeO6. _Phys. Rev. B_ 101, 085120 (2020). https://doi.org/10.1103/PhysRevB.101.085120
  * (33) Janssen, L., Andrade, E. C. $\&$ Vojta, M., Honeycomb-Lattice Heisenberg-Kitaev Model in a Magnetic Field: Spin Canting, Metamagnetism, and Vortex Crystals. _Phys. Rev. Lett._ 117, 277202 (2016). https://doi.org/10.1103/PhysRevLett.117.277202
  * (34) Chen, W. _et al._ , Spin-orbit phase behavior of Na2Co2TeO6 at low temperatures. _Phys. Rev. B_ 103, L180404 (2021). https://doi.org/10.1103/PhysRevB.103.L180404
  * (35) Lee, C. H. _et al._ , Multistage development of anisotropic magnetic correlations in the Co-based honeycomb lattice Na2Co2TeO6. _Phys. Rev. B_ 103, 214447 (2021). https://doi.org/10.1103/PhysRevB.103.214447
  * (36) Krüger, W. G. F., Chen, W., Jin, X., Li, Y. $\&$ Janssen, L., Triple-Q order in Na2Co2TeO6 from proximity to hidden-SU(2)-symmetric point. _online preprint_ arXiv: 2211.16957
  * (37) Yao, W. _et al._ , Magnetic ground state of the Kitaev Na2Co2TeO6 spin liquid candidate. _Phys. Rev. Research_ 5, L022045 (2023). https://doi.org/10.1103/PhysRevResearch.5.L022045
  * (38) Kikuchi, J., Kamoda, T., Mera, N., Takahashi, Y., Okumura, K. $\&$ Yasui, Y., Field evolution of magnetic phases and spin dynamics in the honeycomb lattice magnet Na2Co2TeO6: 23Na NMR study. _Phys. Rev. B_ 106, 224416 (2022). https://doi.org/10.1103/PhysRevB.106.224416
  * (39) Wilhelm, H., Lühmann, T., Rus, T., $\&$ Steglich, F., A compensated heat-pulse calorimeter for low temperatures. _Rev. Sci. Instrum._ 75, 2700–270 (2004). https://doi.org/10.1063/1.1771486
  * (40) Gohlke, M., Moessner, R. $\&$ Pollmann, F., Dynamical and topological properties of the Kitaev model in a [111] magnetic field. _Phys. Rev. B_ 98, 014418 (2018). https://doi.org/10.1103/PhysRevB.98.014418
  * (41) Hickey, C. $\&$ Trebst, S., Emergence of a field-driven U(1) spin liquid in the Kitaev honeycomb model. _Nat. Commun._ 10, 530 (2019). https://doi.org/10.1038/s41467-019-08459-9
  * (42) Hentrich, R. _et al._ , Unusual phonon heat transport in $\alpha$-RuCl3: Strong spin-phonon scattering and field-induced spin gap. _Phys. Rev. Lett._ 120, 117204 (2018). https://doi.org/10.1103/PhysRevLett.120.117204
  * (43) Hentrich, R. _et al._ , Large thermal Hall effect in $\alpha$-RuCl3: Evidence for heat transport by Kitaev-Heisenberg paramagnons. _Phys. Rev. B_ 99, 085136 (2019). https://doi.org/10.1103/PhysRevB.99.085136
  * (44) Gillig, M. _et al._ , Phononic-magnetic dichotomy of the thermal Hall effect in the Kitaev-Heisenberg candidate material Na2Co2TeO6. _online preprint_ arXiv:2303.03067
  * (45) Gillig, M. _et al._ , Thermal transport of the frustrated spin-chain mineral linarite: Magnetic heat transport and strong spin-phonon scattering. _Phys. Rev. B_ 104, 235129 (2021). https://doi.org/10.1103/PhysRevB.104.235129
  * (46) Hong, X. _et al._ , Heat transport of the kagome Heisenberg quantum spin liquid candidate YCu3(OH)6.5Br2.5: Localized magnetic excitations and a putative spin gap. _Phys. Rev. B_ 106, L220406 (2022). https://doi.org/10.1103/PhysRevB.106.L220406
  * (47) Sørensen, E. S., Catuneanu, A., Gordon, J. S. $\&$ Kee, H.-Y., Heart of Entanglement: Chiral, Nematic, and Incommensurate Phases in the Kitaev-Gamma Ladder in a Field. _Phys. Rev. X_ 11, 011013 (2021). https://doi.org/10.1103/PhysRevX.11.011013
  * (48) Li, H. _et al._ , Identification of magnetic interactions and high-field quantum spin liquid in $\alpha$-RuCl3. _Nat. Commun._ 12, 4007 (2021). https://doi.org/10.1038/s41467-021-24257-8
  * (49) Chern, L. E., Kaneko, R., Lee, H.-Y. $\&$ Kim, Y. B., Magnetic field induced competing phases in spin-orbital entangled Kitaev magnets. _Phys. Rev. Research_ 2, 013014 (2020). https://doi.org/10.1103/PhysRevResearch.2.013014
  * (50) Lin, G. _et al._ , Evidence for field induced quantum spin liquid behavior in a spin-1/2 honeycomb magnet. _online preprint at Research Square_ https://doi.org/10.21203/rs.3.rs-2034295/v1.

* * *

Corresponding author email:
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Acknowledgements
We thank Jeroen van den Brink, Matthias Vojta, Vladislav Kataev, Christoph
Wellm, Satoshi Nishimoto, Wenjie Chen, Yang Xu, and Wilhelm Krüger for
illuminating discussions and collaboration on related work. We would like to
thank Nicolás Pérez for generously sharing the equipment, and thank Danny
Baumann and Tino Schreiner for their technical assistance, especially during
the pandemic lab-shutdown crisis.
Funding Acknowledgement:
This work has been supported by the Deutsche Forschungsgemeinschaft (DFG)
through SFB 1143 (Project-ID No.247310070) and the Würzburg-Dresden Cluster of
Excellence $ct.qmat$ (EXC 2147, Project-ID No. 390858490).
C.H. has received funding from the European Research Council (ERC) under the
European Union’s-Horizon 2020 research and innovation programme (Grant
Agreement No. 647276-MARS-ERC-2014-CoG).
L.J. was supported by the DFG through the Emmy Noether program (JA2306/4-1,
Project No. 411750675).
V.K. was supported by the Alexander von Humboldt Foundation.
Y.L. acknowledges funding support from the National Basic Research Program of
China (Grant No. 2021YFA1401901) and the NSF of China (Grant No. 12061131004).

Author contributions
B.B., C.H., and Y.L. conceived and initiated the study. W.Y. and Y.L. grown
the crystals. X.H. and M.G. performed the transport measurements. V.K., S.G.,
and A.U.B.W. performed the thermodynamic measurements. X.H., C.H., and L.J.
analysed the data. X.H., C.H., L.J., and V.K. wrote the manuscript with input
from all authors. C.H. supervised the project.

Additional Information
Supplementary information is available at the end of the paper.

Competing financial interests
The authors declare no competing financial interests.


# Multi-agent Databases via Independent Learning
(Extended Abstract)

Chi Zhang  Brandeis University<EMAIL_ADDRESS>,  Olga Papaemmanouil
Brandeis University<EMAIL_ADDRESS>and  Josiah Hanna  University of
Wisconsin-Madison<EMAIL_ADDRESS>
###### Abstract.

Machine learning is rapidly being used in database research to improve the
effectiveness of numerous tasks included but not limited to query
optimization, workload scheduling, physical design, etc. essential database
components, such as the optimizer, scheduler, and physical designer.
Currently, the research focus has been on replacing a single database
component responsible for one task by its learning-based counterpart. However,
query performance is not simply determined by the performance of a single
component, but by the cooperation of multiple ones. As such, learned based
database components need to collaborate during both training and execution in
order to develop policies that meet end performance goals. Thus, the paper
attempts to address the question ” _Is it possible to design a database
consisting of various learned components that cooperatively work to improve
end-to-end query latency?_ ”.

To answer this question, we introduce MADB (Multi-Agent DB), a proof-of-
concept system that incorporates a learned query scheduler and a learned query
optimizer. MADB leverages a cooperative multi-agent reinforcement learning
approach that allows the two components to exchange the context of their
decisions with each other and collaboratively work towards reducing the query
latency. Preliminary results demonstrate that MADB can outperform the non-
cooperative integration of learned components.

##  1\. Introduction

Database (DB) systems drive various user-facing applications and services and
play a key role in user-perceived performance. Modern DBs have many components
(optimizer, scheduler, physical designer, etc.) and machine learning is
rapidly leveraged to replace numerous essential data management tasks, by
their learning-based counterparts (deep_card_est2, ; negi2020cost, ;
yang2019deep, ; van2017automatic, ; marcusbenchmarking, ; sun2019end, ;
yangneurocard, ; tzoumas2008reinforcement, ). These ML-driven approaches have
demonstrated great potential to improve query performance as well as
automatically adapt to the dynamic environments modern data management
applications often operate in.

Today, the bulk of work on learned databases focuses on replacing a _single_
component or task of the database with a learned-based solution and these
learned components are designed _independently_ of each other. In reality
though, production database settings require careful optimizations across all
these components as query performance is determined by the cooperation of
these components. For example, the optimizer’s data access decisions should be
in coordination with the physical designer’s re-distribution actions and the
buffer pool manager’s page caching decisions. However, with the replacement of
these components by learned ones, it is unclear if and how these per-component
solutions can be integrated into the same database and how they can
cooperatively yield end-to-end benefits .

In this paper, we argue that new database designs that optimize decisions
across learned components (aka learning _agents_) are fundamental to good end-
to-end query performance. We introduce, _Multi-Agent DB_ (MADB), a multi-agent
database system that demonstrates the benefits of such across-component
coordination. MADB includes two key reinforcement learning agents: a learned
query scheduler and a learned query optimizer. Despite the fact that these two
ML-driven systems operate in the same environment, they have different tasks
and performance goals. Given a set of queries queued for execution, the query
scheduler agent strives to to identify the query execution ordering that
improves the data cache utilization. The objective of the query optimizer
agent is to identify the query execution plan that minimizes a given query’s
execution time.

While both of these goals relate to query performance, without coordination
these components might miss opportunities for further performance improvements
or might even make conflicting decisions. For instance, the scheduler learns
to schedule queries that reuses cached data by observing both the contents of
the buffer pool and the expected execution plans of the queued queries.
However, query plans decisions (i.e., full scans vs index scans, join
orderings) are determined by the optimizer’s model which adjusts its policy
based on the observed execution time of each plan. Execution times though
depend on buffer pool hits which are determined by the data access methods
used by the query plan. In a non-cooperative multi-agent setting, such
interdependent decisions are not captured properly, leading to obsolete
policies for all involved agents.

To address this challenge, this paper attempts to address the question “ _Is
it possible to design a database consisting of various learning components
that cooperatively work to improve end-to-end query performance?_ “. We study
this question by incorporating into MADB a cooperative multi-agent
reinforcement learning (MARL) approach. MARL (tan1993multi, ; zhang2021multi,
) is a sub-field of reinforcement learning that studies how multiple
reinforcement learning models (agents) interact in a common environment in
order to accomplish a common goal. In particular, MADB leverages independent
learning (IL) (de2020independent, ; lowe2017multi, ), a MARL approach that
distributes and decentralizes the construction of a system’s agents (in our
case learned database components). Based on IL, the learned scheduler and
learned optimizer of MADB are trained on the same environment, adjust their
policies based on each others observations, and improve end-to-end query
performance, while maintaining their own performance objectives.

We present our system and formalized our learning task in Section 2. We
present preliminary experimental results from a proof-of-concept prototype
implementation in Section 3, and in Section 4 we highlight directions for
future work.

##  2\. MADB Overview

This section describes MADB, a database that combines two learned components:
a scheduler and an optimizer. We first present these two models as non-
cooperating agents. We then describe how they collaborative to improve query
latency.

###  2.1. Learned Query Scheduler

MADB’s uses the SmartQueue learned scheduler (zhang, ). It uses a deep
reinforcement learning (DRL) model that simultaneously estimates and tries to
improve a weighted average between short-term buffer hits and the long-term
impact of query scheduling choices. Next, we briefly describe the scheduler
and its inputs assuming no cooperation with the optimizer. More details can be
found (zhang, ).

Our scheduler uses a Q-learning approach (deep_rl, ) to decide which query to
execute next. It operates over a set of environmental states $S_{1}$ (the
contents in the buffer pool, query access patterns) and a set of actions
$A_{1}$ (candidate queries to execute next). The query scheduler models the
scheduling problem as a Markov Decision Process (MDP) (rl_book, ): by
selecting one query from the queue to execute, the agent transitions from its
current state to a new one. By executing a new query on the current state, the
agent receives a reward. In this case, the reward $R_{1}{}_{t}$ for an action
$A_{1}{}_{t}$ is calculated as the buffer hit ratio of the executed query.
$\frac{\mbox{{buffer hits}}}{\mbox{{total block requests}}}$. The scheduler
employs a continuous learning process. As more queries arrive and scheduled
for execution by the model, the agent collects more information on the context
of these decisions and their impact on its reward/ performance goal and adapts
its policy accordingly.

One first input to the scheduler’s model is the state of the buffer pool,
namely which blocks are currently cached in memory. This information is
captured in a 2-dimensional bitmap $B$ where rows represent base relations and
columns represent data blocks. The $(i,j)$ entry is 1 if the $j$-th block of
relation $i$ is cached and 0 otherwise. Each row is downsided by computing a
simple moving average over the number of its blocks elements. The second input
feature is the expected data block requests of each queued query. The
scheduler collects $q$’s query plan and estimates the probability of each
input table’s data block being accessed. We also include index data access
blocks when used by the query plan. The query vector is downsized in the same
as the buffer bitmap.

###  2.2. Learned Query Optimizer

MADB’s second learned component is a query optimizer. While a number of
learning-based query optimization approaches have been proposed (rejoin, ;
neo, ; wang2016database, ; krishnan2018learning, ; ortiz2018learning, ;
trummer2018skinnerdb, ; marcus2020bao, ; negi2021steering, ), our current
learned optimizer is based on a variation of ReJoin (rejoin, ), a deep
reinforcement learning optimizer that decides the best join orderings of a
query plan. Join ordering can have a significant impact on query execution and
using a deep reinforcement learning approach have already been proven
effective (rejoin, ; neo, ).

In contrast to ReJoin, our optimizer is trained by learning from
demonstrations (schaarschmidt2018lift, ; marcus2018towards, ; neo, ), which
builds a model in two steps. The first phase allows the learned agent to
understand how a traditional query optimizer optimizes a query (i.e., picks
the join orderings in our case). This method begins by training an agent to
mimic the behavior of a traditional optimizer (aka expert) and alleviate the
cold start problem of machine learning algorithms.

The model uses as a Markov Decision Process (MDP) (rl_book, ). When a query
$q$ is sent by the scheduler, the optimizer takes an action: it chooses which
base relation to join next and moves from the current state to the next (that
includes a new join ordering). The process continues until all relations to be
joined are included in the join ordering. Given this join ordering, the
optimizer generates the rest of the query plan, which includes information
about data access operators, etc and the plan is executed. The execution time
of each join is recorded and is used as the reward for the action that added
that join.

Once the agent has been trained on a representative workload to mimic the
traditional optimzer’s join ordering decisions, the models is used to decide
the ordering of queries during runtime. As more queries are executed, the
Q-learning reward function of the optimizer is fine-tuned, making it more
appropriate for long-term future execution decisions. More details for
optimizers following the learning from demonstration approach can be found
(neo, ; marcus2018towards, ).

The first input for the query optimizer is the join ordering of the _latest_
executed query. This state is represented as a matrix with rows for base
relations and columns for joins. If base relation $A$ is to be joined with
base relation $B$ and that join is in the $k$-level of the join tree, the
matrix values of both of these relations is set to k. Low-level base relations
are joined first, then high-level base relations. The second input of the
optimizer is a vector indicating the base relations to be joined in the
current query.

Figure 1. MADB System Model

###  2.3. Cooperative learning in MADB

In this section we describe how the scheduler and the optimizer agents can
effectively cooperate in the same environment. Towards this goal, we use a
multi-agent reinforcement learning (MARL) approach that requires that all
agents in the environment coordinate and collaborate in order to accomplish a
common goal (tan1993multi, ; zhang2021multi, ; de2020independent, ).

Different types of coordination and collaboration approaches have been
proposed for MARL techniques (zhang2021multi, ). In this work, we chose to
capture the interaction between the scheduler and the optimizer using an an
independent learning (IL) approach (de2020independent, ). In IL, the MARL
problem is decomposed into a problem of multiple independent agents, in which
all other agents are considered to be part of the environment and an agent’s
learning policy are determined solely by the agent’s local observation
history. More specifically, we use the popular paradigm of centralized-
training-decentralized-execution (de2020independent, ; lowe2017multi, ) in
which each agent learns a policy based on information they observe at
execution-time but uses shared information during training.

In our setting, the query scheduler and the query optimizer are trained
concurrently with the goal of lowering the overall execution time of query
workload. However, they achieve this goal by decomposing this central goal
into two decentralized problem, with the scheduler focusing on improving the
average buffer hit ratio and the optimizer on reducing the latency of a single
query. Figure 1 depicts how these agents observe the environment’s state,
collaborate and share information to ultimately accomplish the common goal of
reducing query latency of the whole query workload.

Let us assume there are N queries in the scheduler’s queue
($Q_{1}$,…$Q_{n}$.). Once the scheduler decides a query $Q_{i}$ to execute, it
sends that to the optimizer who will generate its execution plan. Each agent
will make its decisions based on its own past observations. However, in our
cooperative setting, these observations include information from the other
agent. That way, each agent is seen as part of the other agent’s environment.
Specifically, when the query scheduler receives all the current information in
the environment, it will take an action and schedule the next query $Q_{i}$ to
execute, which it will then communicate and pass to the query optimizer.

The query scheduler observes (a) the status of the buffer pool, i.e., which
data blocks are cached and (b) the expected data block requests of queries in
the queue. To facilitate the coordination between the scheduler and the
optimizer, we have also added the optimizer’s state into the input features of
the scheduler. This allows the scheduler to handle the optimizer as part of
its environment. Each time a query $Q_{i}$ is sent to the optimizer’s, the
optimizer sends to the scheduler an encoding of the last executed query plan
as well as an encoding of the join orderings decided by the optimizer for
$Q_{i}$. The encoding of the execution plan is similar to the one used by
ReJOIN (rejoin, ). Join ordering are encoded by assigning a value $k$ to all
joined base relations where $k$ denotes the level that base relation appears
in the join query plan.

The query scheduler’s state is also shared with the optimizer. This state
includes the encoding of the buffer contents as well as the encoding of the
expected data blocks to be access the query $Q_{i}$. Given these information,
the query optimizer decides the best join ordering for $Q_{i}$. The database
will execute $Q_{i}$ based on the query optimizer’s recommended query plan,
and the actual latency and average buffer hit ratio will be collected as
reward in order to update the weights of respective reward functions of the
two agents. This process of training is continuous: as queries enter the
system both agents adjust their observations as well as their action and
policies.

The scheduling policy of the query scheduler is expressed as a Q-learning
reward function $Q(S_{1}{}_{t},A_{1}{}_{t})$, that outputs a Q-value for
taking an action $A_{1}{}_{t}$ (i.e., a query to execute next) on a current
state $S_{1}{}_{t}$. The Q-value $Q(S_{1}{}_{t},A_{1}{}_{t})$ is calculated by
adding the maximum reward from future states to the reward for obtaining the
present state, effectively influencing the current scheduling decision. This
potential reward is a weighted average of all future buffer hit ratios
starting from the current state. The agent learns a new policy
$Q^{new}(S_{1}{}_{t},A_{1}{}_{t})$ after each action $A_{1}{}_{t}$ on a state
$S_{1}{}_{t}$:

(1)
$Q\left(S_{1{}_{t}},A_{1{}_{t}}\right)+\alpha\left[R_{1{}_{t}}+\gamma\max_{\alpha}\left(Q\left(S_{1{}_{(}t+1)},\alpha\right)-Q\left(S_{1{}_{t}},A_{1{}_{t}}\right)\right)\right]$

The learning policy of the query optimizer is to decide which base relation to
join next. After joining all relevant base relations, a final query execution
plan is produced. When a query is finished, the agent takes an action
$A_{2}{}_{t}$ and shows which base relation to be joined next based on current
state $S_{2}{}_{t}$. The reward $R_{2}{}_{t}$ for the action $A_{2}{}_{t}$ is
calculated based on the query latency $L$. Thus, we will train and learn the
policy that maximizes its total reward $Q_{2}$. In each step, the agent
updates $Q(S_{2}{}_{t},A_{2}{}_{t})$ by recursively discounting future
utilities and weighting them by a possible learning rate $\alpha$.

(2)
$Q\left(S_{2{}_{t}},A_{2{}_{t}}\right)+\alpha\left[R_{2{}_{t}}+\gamma\max_{\alpha}\left(Q\left(S_{2{}_{(}t+1)},\alpha\right)-Q\left(S_{2{}_{t}},A_{2{}_{t}}\right)\right)\right]$

In above 2 equations, the parameter $\gamma$ is the discount factor which
weighs the contribution of short-term vs. long-term rewards. Adjusting the
value of $\gamma$ will diminish or increase the contribution of future
rewards. The parameter $\alpha$ is the learning rate or step size. This simply
determines to what extent newly acquired information overrides old
information: a low learning rate implies that new information should be
treated skeptically, and may be appropriate when a workload is mostly stable
but contains some outliers. A high learning rate implies that new information
is more fully trusted, and may be appropriate when query workloads smoothly
change over time. Since the above are a recursive equations, they starts with
making arbitrary assumptions for all $Q$-values. However, as more experience
is collected through the execution of incoming queries, the network likely
converges to the optimal policy (dqn, ).

(a) Overall performance improvemet

(b) Cache utilization improvement

(c) Query plan improvement

Figure 2. MADB performance on unseen query templates

##  3\. Preliminary Experiments

We present preliminary results demonstrating that MADB can cooperative learned
components can the overall query performance compared with alternative
techniques.

Experimental Setup Our experimental analysis uses the TPC-DS benchmark
(nambiar2006making, ). We deployed a 49GB database on a single node server
with four cores and 32GB of RAM. Our database engine was PostgreSQL (url-
postgres, ) with a 4GB shared buffer pool111PostgreSQL was configured to
bypass the OS filesystem cache.. Our learned agents (scheduler and optimizer)
are implemented in Keras(keras, ) using a fully-connected neural network with
two hidden layers each with 128 neurons. We use an adaptive learning rate
optimization approach (slos, ) and the mean squared error as our loss
function. We generate random query instances from the 99 TPC-DS templates. Our
agents are trained using $900$ query instances generated from 79 training TPC-
DS templates. The testing set includes $100$ instances generated from the
remaining 20 TPC-DS templates.

MADB Performance We implemented a proof-of-concept prototype of _MADB_ and
studied its effectiveness. It uses an Independent Learning (IL) approach where
the scheduler and optimizer are trained in parallel as queries are scheduled
and executed and share information about their environments with each other as
described in Section 2.3.

We compare MADB to three other approaches. PL_NoSharing includes two agents
that are trained in parallel but do not share information. In ISO_NoSharing
the agents are are trained offline, in isolation from each other (but on the
same training workload) and the final trained models are used to schedule and
execute the testing queries. NoLearning uses no learned components: it
combines a first-come-first-served scheduler and PostgresSQL’s optimizer.

Figure 2(a) demonstrates the impact of parallel training and information
sharing. Here, we increase the size of our training queries from 100 to 900
and each point shows the total time to schedule and execute our 100 testing
queries. As MADB is trained on more queries, the execution time of the unseen
testing queries improves. MADB also outperforms all other approaches.
PL_NoSharing is the second most performing approach, but still follows behind
MADB by 7%, demonstrating that state sharing between agents is as important as
training concurrently on the same environment. ISO_NoSharing offers a slower
execution time (12% alower than MADB) since the agents are trained is
isolation (and do not share any information) and NoLearning has the worse
performance (24% slower than MADB).

Per Component Performance Next, we studied each learned agent’s performance
(as defined by its own reward function) and how they perform when trained on a
multi-agent environment. Figure 2(b) shows how the learned scheduler performs
as we increase the number of training queries. We measure the average buffer
hit ratio of $100$ test queries in unseen templates while increasing our
training set. We observe that the buffer hit ratio in MADB is higher than the
alternative approaches. On average, in MADB, 65% of the data may be retrieved
from the cache rather than the disk, which is 6% higher than the cache
utilization of PL_NoSharing, 15% higher than by ISO_NoSharing and 46% higher
than NoLearning.

Figure 2(c) illustrates the latency of various execution plans generated by
the different techniques. The execution plans generated by MADB consistently
consistently offer lower latency. Specifically, MADB plans are require (in
average) 9% less time than the plans of the PL_NoSharing approach, 11% less
time than the optimizer’s query plan under the ISO_NoSharing framework, and
over 32% on average less time than the non-intelligent framework NoLearning.
Our results, although preliminary, demonstrate that learned database
components trained and executed in cooperative setting as independent learners
can lead to significant performance improvement.

Impact of Information Sharing Lastly we experimented with our agents sharing
different types of information.
MARL2_DifScheduler does not receive data block information (the probability of
each data blocks being accessed by a candidate query), but its input includes
which tables participate in the query operation. MARL3_DifOptimizer no longer
observes the most recent query execution plan and instead simply records this
execution plan’s cost. In Fugre 3 shows that the information shared and
observed by the agents in MADB can maximize the system’s performance as more
queries are executed since it outperforms by $6\%$ the different scheduler and
$12\%$ the different optimizer, respectively.

Figure 3. Performance of different sharing information

##  4\. Conclusion

We have presented MADB, a multi-agent database system that incorporates a
learned query scheduler and a learned query optimizer. Both learned agents in
MADB are trained using a cooperative multi-agent reinforcement learning
approach, where the agents exchanging information with the common goal of
reducing the query execution time. MADB offers substantial improvements over
non-cooperative multi-agent learning systems as well as non-learned
approaches. This demonstrates that cooperatively integrating multiple learned
database components has the potential to identify optimization opportunities
that would have been missed by the naive integration of learned components.

## References

  * [1] PostgreSQL database, http://www.postgresql.org/.
  * [2] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath.  A Brief Survey of Deep Reinforcement Learning.  IEEE Signal Processing ’17.
  * [3] C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson.  Is independent learning all you need in the starcraft multi-agent challenge?  arXiv preprint arXiv:2011.09533, 2020.
  * [4] Keras: The Python Deep Learning library.  https://keras.io//.
  * [5] A. Kipf, T. Kipf, B. Radke, V. Leis, P. Boncz, and A. Kemper.  Learned Cardinalities: Estimating Correlated Joins with Deep Learning.  CIDR ’19.
  * [6] S. Krishnan, Z. Yang, K. Goldberg, J. Hellerstein, and I. Stoica.  Learning to optimize join queries with deep reinforcement learning.  arXiv preprint arXiv:1808.03196, 2018.
  * [7] W. Lang, S. Shankar, J. M. Patel, and A. Kalhan.  Towards Multi-Tenant Performance SLOs.  ICDE ’14.
  * [8] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  NeurIPS’17.
  * [9] R. Marcus, A. Kipf, A. van Renen, M. Stoian, S. Misra, A. Kemper, T. Neumann, and T. Kraska.  Benchmarking learned indexes.  VLDB ’21.
  * [10] R. Marcus, P. Negi, H. Mao, N. Tatbul, M. Alizadeh, and T. Kraska.  Bao: Learning to steer query optimizers.  SIGMOD’20.
  * [11] R. Marcus, P. Negi, H. Mao, C. Zhang, M. Alizadeh, T. Kraska, O. Papaemmanouil, and N. Tatbul.  Neo: A Learned Query Optimizer.  VLDB ’19.
  * [12] R. Marcus and O. Papaemmanouil.  Deep reinforcement learning for join order enumeration.  aiDm@SIGMOD’18.
  * [13] R. Marcus and O. Papaemmanouil.  Towards a hands-free query optimizer through deep learning.  CIDR ’19.
  * [14] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature ’15.
  * [15] R. O. Nambiar and M. Poess.  The Making of TPC-DS.  VLDB ’06.
  * [16] P. Negi, M. Interlandi, R. Marcus, M. Alizadeh, T. Kraska, M. Friedman, and A. Jindal.  Steering query optimizers: A practical take on big data workloads.  SIGMOD’21.
  * [17] P. Negi, R. Marcus, H. Mao, N. Tatbul, T. Kraska, and M. Alizadeh.  Cost-guided cardinality estimation: Focus where it matters.  ICDEW’20.
  * [18] J. Ortiz, M. Balazinska, J. Gehrke, and S. S. Keerthi.  Learning state representations for query optimization with deep reinforcement learning.  DEEM’18.
  * [19] M. Schaarschmidt, A. Kuhnle, B. Ellis, K. Fricke, F. Gessert, and E. Yoneki.  Lift: Reinforcement learning in computer systems by learning from demonstrations.  arXiv preprint arXiv:1808.07903, 2018.
  * [20] J. Sun and G. Li.  An end-to-end learning-based cost estimator.  VLDB’19.
  * [21] R. S. Sutton, A. G. Barto, et al.  Introduction to reinforcement learning.  1998\.
  * [22] M. Tan.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  ICML’93.
  * [23] I. Trummer, S. Moseley, D. Maram, S. Jo, and J. Antonakakis.  Skinnerdb: regret-bounded query evaluation via reinforcement learning.  VLDB’18.
  * [24] K. Tzoumas, T. Sellis, and C. S. Jensen.  A reinforcement learning approach for adaptive query processing.  History, 2008.
  * [25] D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang.  Automatic database management system tuning through large-scale machine learning.  SIGMOD’17.
  * [26] W. Wang, M. Zhang, G. Chen, H. Jagadish, B. C. Ooi, and K.-L. Tan.  Database meets deep learning: Challenges and opportunities.  SIGMOD’16.
  * [27] Z. Yang, A. Kamsetty, S. Luan, E. Liang, Y. Duan, X. Chen, and I. Stoica.  Neurocard: One cardinality estimator for all tables.  VLDB ’21.
  * [28] Z. Yang, E. Liang, A. Kamsetty, C. Wu, Y. Duan, X. Chen, P. Abbeel, J. M. Hellerstein, S. Krishnan, and I. Stoica.  Deep unsupervised cardinality estimation.  VLDB’19.
  * [29] C. Zhang, R. Marcus, A. Kleiman, and O. Papaemmanouil.  Buffer pool aware query scheduling via deep reinforcement learning.  AIDB@VLDB’20.
  * [30] K. Zhang, Z. Yang, and T. Başar.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  Handbook of Reinforcement Learning and Control, 2021.