††thanks: These authors contributed equally to this work.††thanks: These
authors contributed equally to this work.

# Observation of metallic electronic structure in a single-atomic-layer oxide

Byungmin Sohn  Center for Correlated Electron Systems, Institute for Basic
Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea  Jeong Rae Kim  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Department
of Physics and Astronomy, Seoul National University, Seoul 08826, Korea
Choong H. Kim  Center for Correlated Electron Systems, Institute for Basic
Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea  Sangmin Lee  Department of Materials
Science and Engineering and Research Institute of Advanced Materials, Seoul
National University, Seoul 08826, Korea  Sungsoo Hahn  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Department
of Physics and Astronomy, Seoul National University, Seoul 08826, Korea
Younsik Kim  Center for Correlated Electron Systems, Institute for Basic
Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea  Soonsang Huh  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Department
of Physics and Astronomy, Seoul National University, Seoul 08826, Korea
Donghan Kim  Center for Correlated Electron Systems, Institute for Basic
Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea  Youngdo Kim  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Department
of Physics and Astronomy, Seoul National University, Seoul 08826, Korea
Wonshik Kyung  Center for Correlated Electron Systems, Institute for Basic
Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea  Minsoo Kim  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Department
of Physics and Astronomy, Seoul National University, Seoul 08826, Korea
Miyoung Kim  Department of Materials Science and Engineering and Research
Institute of Advanced Materials, Seoul National University, Seoul 08826, Korea
Tae Won Noh<EMAIL_ADDRESS>Center for Correlated Electron Systems, Institute
for Basic Science, Seoul 08826, Korea  Department of Physics and Astronomy,
Seoul National University, Seoul 08826, Korea  Changyoung Kim
<EMAIL_ADDRESS>Center for Correlated Electron Systems, Institute for
Basic Science, Seoul 08826, Korea  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Korea

(August 28, 2024)

###### Abstract

Correlated electrons in transition metal oxides (TMOs) exhibit a variety of
emergent phases. When TMOs are confined to a single-atomic-layer thickness,
experiments so far have shown that they usually lose diverse properties and
become insulators. In an attempt to extend the range of electronic phases of
the single-atomic-layer oxide, we search for a metallic phase in a monolayer-
thick epitaxial SrRuO3 film. Combining atomic-scale epitaxy and angle-resolved
photoemission measurements, we show that the monolayer SrRuO3 is a strongly
correlated metal. Systematic investigation reveals that the interplay between
dimensionality and electronic correlation makes the monolayer SrRuO3 an
incoherent metal with orbital-selective correlation. Furthermore, the unique
electronic phase of the monolayer SrRuO3 is found to be highly tunable, as
charge modulation demonstrates an incoherent-to-coherent crossover of the two-
dimensional metal. Our work emphasizes the potentially rich phases of single-
atomic-layer oxides and provides a guide to the manipulation of their two-
dimensional correlated electron systems.

Introduction

With the size of electronic devices getting ever smaller, continued efforts to
use different schemes, such as single-molecule transistors, are being made to
overcome the quantum limit. The discovery of graphene in 2004 and later van
der Waals material groups suggests a new direction of atomically thin two-
dimensional (2D) electronics [1, 2, 3]. From the fundamental scientific point
of view, an ideal 2D system can provide a new physics distinct from that of
three-dimensional (3D) systems. Accordingly, the 2D van der Waals material
groups, associated devices [4], heterostructures [5], and Moiré superlattices
[6, 7] have been the most intensively studied subjects in condensed matter
physics in the past 15 years. However, it is still quite difficult to obtain
high-quality and large-size flakes of van der Waals materials for practical
applications.

On the other hand, the strongly correlated electron system of transition metal
oxides possesses numerous exotic electronic phases with novel physical
properties, $e.g.$, high-temperature superconductivity, Mott transition, and
ferromagnetism [8, 9, 10]. Most of these phases appear in three-dimensionally
connected oxide crystals such as perovskite oxides. These exotic properties in
principle can be utilized for device applications. In light of the direction
in the 2D van der Waals materials research, transition metal oxides with
single-atomic-layer thickness can offer new functionalities in the nanometer
scale by bridging correlated electron systems and the field of 2D materials.
However, only a few oxides are known to have van der Waals bonding and can be
exfoliated into atomically thin flakes [11, 12].

Epitaxial thin film growth can provide an alternative approach to constructing
artificial 2D oxides and their devices on a range of lattice-matched oxide
substrates [13] or Si wafer [14]. Despite extensive efforts to realize the 2D
electronic oxides, most of the ultrathin oxide films exhibit a metal-to-
insulator transition in approaching the single-atomic-layer limit [15, 16],
which is usually attributed to correlation-driven Mott insulating phases or
localization effects. Such monotonous behavior detains functional spectrum of
2D oxides and limits the integration of the intrinsic physical properties into
real devices. Therefore, the demonstration of a metallic single-atomic-layer
oxide is highly desired to extend the boundary of functionalities and
application possibilities for oxide films.

Here, we report the metallic electronic structure of a single-atomic-layer
oxide, a monolayer SrRuO3 (SRO) film with a single RuO2 atomic plane. The
challenge in the investigation of a single-atomic-layer oxide arises from the
film’s vulnerability to extrinsic disorders. We employed angle-resolved
photoemission spectroscopy (ARPES) measurement to obtain the intrinsic
electronic structure of a single unit-cell (uc) SRO. The electronic structure
of a monolayer SRO is found to be strongly correlated. We unveil the origin of
the correlated phase and demonstrate the tunable electronic phase by charge
modulation. This work adds new electronic states to the library of single-
atomic-layer oxides which has been limited to insulators until now.

Figure 1: Observation of a metallic single-atomic-layer oxide in charging-free
ultrathin SrRuO3 (SRO) heterostructures. a, A schematic of a monolayer SRO
grown on a (001)-oriented SrTiO3 (STO) layer. b, A schematic of a charging-
free ultrathin SRO heterostructure composed of 4 unit-cell (uc) SRO layer
(conducting layer), 10 uc STO layer (buffer layer), and n uc ultrathin SRO
layer, sequentially grown on a STO (001) substrate. The conducting layer
prevents the charging effect during the photoemission measurement. The buffer
layer decouples the electronic structure of the ultrathin SRO layer from that
of the conducting layer. c, Angle-integrated photoemission spectra from
charging-free ultrathin SRO heterostructures. Energy distribution curves
(EDCs) are integrated in the range of $-0.6\leavevmode\nobreak\ \AA^{-1}\leqq
k_{y}\leqq 0.6\leavevmode\nobreak\ \AA^{-1}$ and $k_{x}$ = $0$. n indicates
the number of SRO layers. The Fermi edge is persistently seen down to the
monolayer. The inset shows a magnified view of the monolayer spectrum near the
Fermi level. A hump marked with a red inverted triangle appears at a high-
binding energy in a monolayer SRO and will be discussed in Fig. 5d. d, Atomic-
scale imaging of the charging-free monolayer SRO heterostructure obtained by
high-angle annular dark-field scanning transmission electronic microscopy
(HAADF-STEM). e, HAADF-STEM intensity across the monolayer SRO and adjacent
interfaces of the yellow dashed box in (d). f, Atomic-scale energy-dispersive
X-ray spectroscopy (EDS) analysis on the charging-free monolayer SRO
heterostructure.  Figure 2: Fermi surfaces (FSs) of ultrathin SRO films. a,
FSs of ultrathin SRO layers with specified thickness. Angle-resolved
photoemission spectroscopy (ARPES) data were taken at 10 K with He-I$\alpha$
light (21.2 eV), and integrated within $E_{F}\pm 10$ meV. b, Energy
isosurfaces of two-dimensional (2D) SRO calculated by density functional
theory (DFT) calculation, together with schematic energy isosurfaces. DFT
results are color-coded for $d_{xy}$ (blue) and $d_{yz,zx}$ (red) orbital
contributions to the bands. The VHS represents Van Hove singularity. c,
$\Gamma$-$M$ high-symmetry cuts (Cut 1 in (a)) and momentum distribution
curves (MDCs) at $E_{F}$. The inverted triangles next to MDCs represent peaks
of the $\alpha$ band. Both $\alpha$ and $\beta$ bands are resolved in 1 uc.

Results

Fabrication of charging-free ultrathin heterostructures

In the study of ultrathin oxide films, transport measurements are routinely
performed to study the physical properties. However, transport properties can
be sensitive to extrinsic effects such as disorder [17]. Such extrinsic
effects tend to be more pronounced for films with reduced thicknesses due to
the increased scattering from interfaces and surfaces [18]. More importantly,
transport measurements require a conducting path in the film over a
macroscopic scale, which is not generally guaranteed. Particularly, the
conducting channel can be broken near the step-edges of substrates [19]. In
that regard, ARPES, which detects electrons with periodic motion and does not
require macroscopic connectivity, can be an effective tool to study the
intrinsic electronic property of ultrathin oxide films.

There have been several ARPES studies on ultrathin epitaxial transition metal
oxide films [20, 21, 22, 23, 24]. For the case of SrIrO3 and LaNiO3, the
studies have even reached the monolayer limit, for which insulating electronic
structures have been observed. Here, we note the similarity between monolayer
oxide films and their quasi-two-dimensional counterparts; both Sr2IrO4 [25]
and NdSrNiO4 [26] are antiferromagnetic insulators. Based on this analogy, we
chose a single-atomic-layer-thick SRO film (Fig. 1a) whose electronic ground
state is currently under debate [27, 28]. The monolayer SRO can be also
regarded as a two-dimensional analogue of a metallic single layer perovskite
Sr2RuO4 which has been intensively studied for its unconventional
superconductivity [29] and recently for magnetism [30].

We grew high-quality and charging-free ultrathin SRO heterostructures with
various thicknesses. Figure 1b shows a schematic of the SRO heterostructure
which consists of 4 uc SRO layer, 10 uc SrTiO3 (STO) buffer layer, and n uc
ultrathin SRO layer (n uc SRO), sequentially grown on an STO (001) substrate.
In ARPES measurements on ultrathin films, escaping photoelectrons can cause a
charging effect which distorts the measured spectra. By introducing a current
path of 4 uc-thick SRO layer (conducting layer), we successfully removed the
charging effect in the measurements of ultrathin SRO (Fig. 1c) [See
Supplementary Materials (SM) for comparison of the ARPES data with and without
the conducting layer]. The 10 uc STO buffer layer (4 nm thickness) decouples
the electronic structure of the topmost ultrathin SRO from that of the
conducting layer [See SM for details].

Figure 3: Thickness-driven electronic and magnetic transition in ultrathin SRO
layers. a, A density functional theory (DFT)-calculated energy versus momentum
($E$-$k$) dispersion of 2D SRO along $\Gamma$-$X$ high-symmetry line. Open
trangles and squares indicate replica bands due to the
$\sqrt{2}\times\sqrt{2}$ octahedral rotation. b, High-symmetry cuts
($\Gamma$-$X$) with a specified thickness of SRO layers along the $k_{x}$ = 0
line. Coherent $\beta$ and $\gamma$ band dispersions are observed in 4 and 3
uc, whereas only an incoherent $\gamma$ is observed in 2 and 1 uc. c,
Thickness-dependent EDCs near the X point, $(k_{x},k_{y})$ = $(0,\pm 0.75)$,
marked with a red dotted line in b. Inverted triangles indicate coherent peaks
from the $\gamma$ band. The coherent peaks systematically disappear as the
thickness is reduced. d, Thickness-dependent spin-resolved EDCs measured at 10
K near $\Gamma$. The spin majority and minority spectra for 4 and 3 uc films
show a difference, showing net spin polarization, whereas the difference
vanishes for 2 and 1 uc films.

Scanning transmission electron microscopy (STEM) provides atomic-scale
visualization of our charging-free ultrathin SRO heterostructures. Figure 1d
displays a cross-sectional high-angle annular dark-field STEM (HAADF-STEM)
image with STO [100] zone axis. To image the monolayer SRO, we deposited an
additional 10 uc STO capping layer to protect the monolayer SRO. The
conducting layer, buffer layer, monolayer SRO, and capping layer are well
organized in the preferred order. In the plot of HAADF-STEM intensity across
the monolayer SRO and adjacent interfaces shown in Fig. 1e, an abrupt Ru peak
out of the surrounding Ti peaks is evident. The abrupt SRO/STO interfaces are
further corroborated by energy-dispersive X-ray spectroscopy (EDS) analysis in
Fig. 1f. Taken together, we confirmed that our ultrathin SRO layers possess
lateral uniformity and atomically sharp interfaces, which will allow for a
systematic ARPES study.

Electronic structures of atomically-thin SRO

Using the charging-free heterostructures in Fig. 1b, we examine the electronic
ground state of the ultrathin SRO film. We performed in-situ ARPES on
heterostructures with $n=$ 0, 1, 2, 3, and 4 uc. Figure 1c shows angle-
integrated photoemission spectra of the heterostructures near $\Gamma$.
Definite Fermi edges survive down to the monolayer, showing the persistent
density of states (DOS) at the Fermi level. A pronounced spectral weight is
observed near the Fermi level for 4 and 3 uc samples, consistent with
previously reported thick film ARPES results [31]. As the thickness is
reduced, the spectral weight shifts toward the high-binding energy side,
resulting in weak but definite spectral weight at the Fermi level for the 1 uc
film. While the underlying mechanism for the spectral weight transfer needs
further investigations, we can clearly observe a metallic electronic ground
state for the monolayer SRO.

In order to better understand the metallic behavior of the monolayer SRO film
in view of the electronic structure, we investigate the thickness-dependent
evolution of the Fermi surfaces (FSs). FSs plotted in Fig. 2a are consistent
with the angle-integrated photoemission results in Fig. 1c, showing metallic
FSs down to the monolayer limit. Here, following the convention used for
Sr2RuO4 [32], we label the three FSs of the $t_{2g}$ bands as $\alpha$,
$\beta$, and $\gamma$ as shown in Fig. 2b. The FS maps of 4 and 3 uc films in
Fig. 2a show that $\alpha$ and $\beta$ bands are sharp while the $\gamma$ is
broad. For the thinner 2 and 1 uc films, $\alpha$ and $\beta$ bands are also
slightly broadened. Otherwise, FSs do not show a qualitative change as the
thickness varies.

Density functional theory (DFT) calculation of a monolayer SRO in Fig. 2b
reproduces the characteristic band structure of experimentally observed FSs
[See “Methods” for DFT calculation of a monolayer SRO]. The van Hove
singularity (VHS) of the $\gamma$ band is located at the $X$ point. The broad
spectral weight at the $X$ point on the $k_{x}$ = 0 line (See Fig. 2a)
indicates that the VHS lies close to the Fermi level. We would like to note
that the spectral weight near the $X$ point on the $k_{y}$ = 0 line is weak
due to the matrix element effect; our ARPES data were measured with He
I$\alpha$ light mostly polarized in the vertical direction [33].

To look at more detailed thickness-dependent evolution of the $\alpha$ and
$\beta$ bands which are composed of $d_{yz}$ and $d_{zx}$ orbitals, we plot in
Fig. 2c $\Gamma$-$M$ high-symmetry cut (‘Cut’ 1 in Fig. 2a) data and momentum
distribution curves (MDCs) at the Fermi energy. We observe only two bands for
the 2 and 1 uc films despite the three $t_{2g}$ orbitals. The two dispersive
bands are attributed to $\alpha$ and $\beta$ bands located near
$k_{\parallel}$ = 0.6 and 0.8$\leavevmode\nobreak\ \AA^{-1}$, respectively,
while the $\gamma$ band can be counted out as we will discuss later. The
$\alpha$ band, marked with an inverted triangle in the MDC plot in the upper
panel, is resolved at the Fermi level regardless of the thickness [See SM for
the analysis on the $\beta$ band]. Putting it all together, the $\alpha$ and
$\beta$ bands are dispersive at all thicknesses but their spectral functions
gradually become less coherent as the thickness is reduced.

We now turn our attention to the $\gamma$ band to understand the origin of the
thickness-dependent spectral weight transfer to high-binding energy in
ultrathin SROs. The VHS of the $\gamma$ band at the $X$ point lies close to
the Fermi level which leads to the high DOS (Fig. 3a). Since the high DOS from
a VHS can significantly affect the physical properties, it is worth examining
the electronic band structures near the VHS. Figure 3b shows the $E$-k
dispersions along the $k_{x}$ = 0 line, indicated as ‘Cut 2’ in Fig. 2a. In
addition to the $\beta$ band, a coherent heavy $\gamma$ band is clearly
observed in the 4 and 3 uc data whereas the 2 and 1 uc SROs data show only an
incoherent $\gamma$ band. As a side note, we attribute the weak $\beta$ band
to the matrix element effect as $\alpha$ and $\beta$ bands remain consistently
coherent at the Fermi level in the $\Gamma$-$M$ (‘Cut 1’ in Fig. 2c) and the
$\Gamma$-$X$ data along the $k_{y}$ = 0 line [See SM for details] [33].

The thickness-dependent evolution of the $\gamma$ band can be better seen in
energy distribution curves (EDCs) near the X point, ($k_{x}$, $k_{y}$) = (0,
0.75), shown in Fig. 3c. The coherent peak marked by the inverted triangle
appears as a kink in the EDCs in the 4 and 3 uc SRO data. On the other hand,
the coherent peak disappears in the 2 and 1 uc SRO data, in which the spectral
weight is believed to be transferred to the incoherent band. Such behavior is
reminiscent of the strongly correlated metallic states in the vicinity of a
Mott state as observed in, for example, under-doped cuprates above its
superconducting dome [34]. Considering what we learned from the EDCs, it can
be deduced that the thickness-dependent electronic transition is closely
related to the strong correlation in the $\gamma$ band. Therefore, we may
refer the electronic state of monolayer SRO to an orbital-selectively
correlated and incoherent metal. The orbital selective correlated metallic
behavior should be closely related to the orbital selective Mott phase found
in Ca2-xSrxRuO4 [35]

Finally, we discuss the magnetism in these films. While bulk SRO is
ferromagnet [36], its quasi-2D analogue, Sr2RuO4, does not show ferromagnetism
[29]. It implies a possible magnetic transition in the ultrathin SRO films. We
investigated the spin polarization of ultrathin SRO films by performing spin-
resolved ARPES (SARPES). Figure 3d shows spin-resolved EDCs measured at 10 K
near $\Gamma$. In the high-binding energy region, we observe a sizable
difference in the intensity between majority and minority spins for 4 and 3 uc
SRO films, indicating ferromagnetism [24]. However, the ferromagnetism is not
observed for the 2 and 1 uc SRO films for which no difference is seen in the
SARPES data.

Figure 4: Interplay between dimensionality, electronic correlation, and
magnetism. a,b, Schematic illustrations of (a) quantum confinement (QC) effect
and (b) VHS. The out-of-plane orbitals, $d_{yz}$ and $d_{zx}$, are mainly
involved in the QC effect. The in-plane orbital, $d_{xy}$, is responsible for
the high density of states (DOS) at the Fermi level due to the VHS. c,d,
Schematic illustration of (c) electronic structures without correlation and
(d) thickness-dependent correlation effect on $d_{xy}$ band. The red dotted
line represents the $d_{yz}$ and $d_{zx}$ orbital partial DOS for bulk SRO.
The high DOS at the Fermi level from the $d_{xy}$ VHS gives rise to a strong
electronic correlation. In thick films (more than 3 uc), the DOS at the Fermi
level can be significantly reduced via ferromagnetic spin splitting. In the
monolayer SRO case, the spin splitting is inhibited due to the QC effect. As a
result, the $d_{xy}$ DOS at the Fermi level is reduced via opening a soft gap
[35]. When VHS is far from the Fermi level, the coherent states can be
recovered.  Figure 5: Incoherent-to-coherent crossover in monolayer SRO. a, FS
maps of pristine and K-dosed monolayer SROs measured at 10 K. b, Band
dispersions of pristine and K-dosed monolayer SROs along the $k_{y}$ = - 0.2
$\AA^{-1}$ line (red dotted line in a). c,d, EDCs from pristine and K-dosed
monolayer SRO films near the X and $\Gamma$ points normalized by $E$ = - 0.6
and - 2 eV, respectively. With K dosing, a quasi-particle peak reappears near
the Fermi level, while the hump peak in the high-binding energy region
disappears, as marked by inverted triangles. ‘K-dosed$\times$2’ indicates
twice the dosing amount of ‘K-dosed’.

Dimensionality-driven electronic transition

Note that the different behaviors between in- and out-of-plane orbital bands
ascertains the quantum confinement (QC) effect of ultrathin SRO films [37];
the reduced dimensionality affects only the out-of-plane orbital $d_{yz}$ and
$d_{zx}$ bands (Fig. 4a). We also notice that there is a remarkable
coincidence between the band coherence and ferromagnetism in ultrathin SRO
films. Both coherent-to-incoherent and ferromagnetic-to-nonmagnetic
transitions start to occur at the 2 uc thickness, which suggests a strong
interplay between electronic correlation and magnetism.

In 3D cubic SRO, three $t_{2g}$ orbitals ($d_{xy}$, $d_{yz}$, and $d_{zx}$)
are degenerate and each one of them has a VHS. The octahedral rotation as well
as the epitaxial strain slightly breaks the degeneracy, resulting in a lower
energy level for the $d_{xy}$ VHS, as schematically shown in the Fig. 4c [38,
39]. The QC effect selectively reconstructs the electronic structure of
$d_{yz}$ and $d_{zx}$ orbitals. In the 2D monolayer limit, the $d_{yz}$ and
$d_{zx}$ bands have strong 1D singularities at the band edges and reduced DOS
at the Fermi level (‘2D’ in Fig. 4c). This reduction in the Fermi-level DOS
induces the thickness-driven ferromagnetic-to-nonmagnetic transition between 3
and 2 uc SRO films [37].

The high DOS from the $d_{xy}$ VHS (Fig. 4c) can enhance the effective
electron correlation, $e.g.$, Coulomb interaction and Hund’s coupling [40].
The instability from the high DOS at the Fermi level can be avoided by
splitting the $d_{xy}$ band into spin majority and minority bands as shown for
4 and 3 uc SRO (left panel, Fig. 4d). However, without the ferromagnetic spin
splitting, the $d_{xy}$ band in 1 and 2 uc films retains the VHS, and thus the
high DOS near the Fermi level. Then, the strong correlation drives monolayer
SRO system to an incoherent metallic phase (right panel, Fig. 4d). The bilayer
SRO case is presumably at the boundary between the coherent ferromagnetic
metal and incoherent correlated metal.

Control of the 2D correlated electronic phase

With the underlying mechanism for the thickness-driven electronic transition
understood, we attempt to exploit the mechanism to control the electronic
phase of the monolayer SRO. As the key to the incoherent-metallic phases is
the VHS, tuning the chemical potential is likely to break the correlated phase
(right panel, Fig. 4d). We used in-situ K dosing to dope electrons into the
incoherent metallic phase of monolayer SRO. Figure 5a shows FS maps of
monolayer SRO before and after K dosing. The $k_{F}$ of the $\beta$ band
changes from 0.52 to 0.60 $\AA^{-1}$ along the $k_{x}$ = 0 line, indicating
that electrons are doped into the system [see SM for $E$-$k$ dispersions and
MDCs]. Overall, the K-dosed FS feature is much better resolved in comparison
with the result of the pristine case.

The most distinct change occurs in the $\gamma$ band. Figure 5b shows $E$-$k$
band dispersions along the $k_{x}$ = -0.2 $\AA^{-1}$ line before and after K
dosing. The strongly correlated $\gamma$ band has never been sharply resolved
in the pristine state but it appears coherent with a clear dispersive feature
after K dosing. The $\gamma$ FS became a hole pocket, which indicates the VHS
is now located below the Fermi level (Fig. 2b). This is consistent with our
scenario proposed above; the electron doping moves the VHS away from the Fermi
level, and consequently the spectral weight is transferred from the incoherent
to coherent peaks. EDCs in Fig. 5c show spectral-weight transfer as well as
the emergence of the coherent quasi-particle peak.

We also noticed that the high-binding hump-like peak at $E$ = -1.5 eV, which
is only observed in monolayer SRO (Fig. 1c), disappears after K dosing as
shown in Fig. 5d. A similar hump-like peak at high-binding energy has been
reported in photoemission spectroscopy results of CaRuO3 and (Ca,Sr)VO3 [41,
42] and was attributed to a strong electronic correlation effect. When the
Coulomb interaction between electrons increases, a coherent peak near the
Fermi level is expected to be suppressed because its spectral weight is
transferred to the incoherent lower Hubbard band [43, 44, 42]. Thus, we
believe that the appearance of the high-binding energy hump with decreasing
thickness is also a sign of enhanced electronic correlation in thinner films.
Disappearance of the hump-like peak upon K dosing reveals that electron
correlations become weaker as the VHS moves away from the Fermi level.

It is worth mentioning a couple of findings from the electron-doping
experiments of single-atomic-layer SRO. First of all, we can exclude extrinsic
disorders as the origin of the incoherent metallicity in the monolayer SRO. If
the incoherent metallicity were due to defects or disorders, it would have
been persistent even after K dosing. The reappearance of the coherent peak and
disappearance of the high-binding energy hump peak with K dosing support the
view for intrinsic nature of the observed incoherent metallicity of monolayer
SRO. The other is that K-dosing results highlight the rich spectrum of
electronic phases in monolayer SRO and their giant tunability via charge
modulation. The considerable spectral weight at the Fermi energy and clear
dispersive bands indicate a good metallicity in the electron-doped monolayer
SRO. The incoherent-to-coherent crossover might be exploited to realize novel
atomically thin oxide field-effect-transistors which have not been realized so
far.

Outlook

2D materials and their applications are a rapidly growing field in
contemporary condensed matter physics and materials science. This trend is
becoming predominant not only for van der Waals materials but also for oxides.
In very recent years, free-standing oxide membranes have been prepared out of
the epitaxial oxide heterostructures [45], and the size has reached a wafer
scale [46]. Moreover, the high-quality membranes of dielectric oxides are
shown to maintain their crystallinity even down to the monolayer limit [47].
By demonstrating a metallic single-atomic-layer oxide, our work expands the
scope of 2D oxides that has been limited to insulators so far. The strong
electronic correlation gives rise to highly tunable correlated electronic
phases, which will be a distinct and advantageous feature of 2D oxides for
future research on device applications. We expect that other emergent phases
in oxides, such as unconventional superconductivity [48], could appear in a
single-atomic-layer and thus that our findings pave the way to the two-
dimensional correlated electronics [49, 50].

## Methods

Fabrication of heterostructures

Epitaxial SrRuO3 and SrTiO3 thin films were grown on (001)-oriented SrTiO3
single crystal substrates by the pulsed laser deposition technique. Prior to
the growth, the SrTiO3 substrate was dipped in deionized water and sonicated
for 30 minutes. The substrate was subsequently in-situ annealed in the growth
chamber, and the annealing temperature, background oxygen partial pressure,
and annealing time were $1,050\,^{\circ}{\rm C}$, 5.0 $\times$10–6 Torr, and
30 minutes, respectively. Polycrystalline SrRuO3 and SrTiO3 targets were
ablated using a KrF excimer laser. For the growth of SrRuO3 films, the
substrate temperature, background oxygen partial pressure, and laser energy
density were kept at $670\,^{\circ}{\rm C}$, 100 mTorr, and 1.9 J/cm2,
respectively. For the growth of SrTiO3 films, the substrate temperature,
background oxygen partial pressure, and laser energy density were kept at
$670\,^{\circ}{\rm C}$, 10 mTorr, and 1.2 J/cm2, respectively. After the
growth, all samples have been cooled down to the room-temperature with a rate
of $50\,^{\circ}{\rm C}/$min in an oxygen partial pressure of $100$ mTorr.

In-situ angle-resolved photoemission spectroscopy

In-situ ARPES measurements were performed at 10 K using the home lab system
equipped with a Scienta DA30 analyzer and a discharge lamp from Fermi
instrument. He-I$\alpha$ ($hv=21.2$ eV) light partially polalized with linear
vertical was used. Low-energy electron diffraction patterns are taken after
ARPES measurements. Spin polarization was measured with a spin-resolved ARPES
system in our laboratory. The system was equipped with a SPECS PHOIBOS 225
analyzer and a very low energy electron diffraction (VLEED) spin detector. For
the spin detector, an oxidized iron film deposited on W(100) was used as the
scattering target. He I$\alpha$ ($hv=21.2$ eV) light was used as the light
source. To clean the surface of SRO thin films, we post-annealed them at
$550\,^{\circ}{\rm C}$ for 10 mins [See SM for details].

First-principles calculation

We performed first-principles calculation using the DFT method without spin-
orbit coupling. The PBEsol form of the exchange-correlation functional was
used as implemented in VASP [51, 52]. To simulate our experimental situation,
we prepared a slab geometry with 20 $\AA$ vacuum in which 4 uc of SrTiO3 is
sandwiched by 1 uc of SrRuO3 which preserves the mirror symmetry with respect
to the middle SrO layer. We used a 600 eV plane wave cut-off energy and 12
$\times$ $12\times$ 1 $k$-points for all calculations and the projector
augmented wave method. During the geometry optimizations, the in-plane lattice
constant was fixed at the experimental value of SrTiO3, 3.905 $\AA$ and the
tolerance on atomic forces was set to 4 meV $\AA^{–1}$. The electronic density
of states was calculated using a fine mesh 12 $\times$ $12\times$ 1
$k$-points. For the Fermi surface calculations, we used the PyProcar package
[53] to unfold the band structure. The chemical potential in the calculated
dispersion was shifted by 70 meV so that $k_{F}$ of the $\beta$ band coinsides
with the experimental value.

Scanning transmission electron microscopy measurement

Cross-sectional scanning transmission electron microscopy (STEM) specimen was
prepared utilizing focused ion beam milling with FEI Helios 650 FIB and
further thinned by focused Ar-ion milling with Fischione NanoMill 1040. STEM
images and energy dispersive X-ray spectroscopy (EDS) were acquired using
Thermo Fisher Scientific Themis Z equipped with a corrector of spherical
aberrations, a high-brightness Schottky-field emission gun operated at a 300
kV electron acceleration voltage, and Super-X EDX system. The semi-convergence
angle of the electron probe was 25.1 mrad.

Competing Interests

The authors declare no competing interests.

Data availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

Acknowledgments

We gratefully acknowledge useful discussions with Woo Seok Choi, Kookrin Char,
Kee Hoon Kim, Hongki Min, Bohm Jung Yang, Sang Mo Yang, and Seo Hyoung Chang.
This work is supported by the Institute for Basic science in Korea (Grant No.
IBS-R009-D1, IBS-R009-G3). We acknowledge the support from the Korean
government through National Research Foundation (2017R1A2B3011629). Cs-
corrected STEM works were supported by the Research Institute of Advanced
Materials (RIAM) in Seoul National University.

## References

  * [1] Novoselov, K. S. _et al._ Electric field effect in atomically thin carbon films.  _Science_ 306, 666–669 (2004).
  * [2] Zhang, Y., Tan, Y.-W., Stormer, H. L. & Kim, P.  Experimental observation of the quantum Hall effect and Berry’s phase in graphene.  _Nature_ 438, 201–204 (2005).
  * [3] Radisavljevic, B., Radenovic, A., Brivio, J., Giacometti, V. & Kis, A.  Single-layer MoS2 transistors.  _Nat. Nanotechnol._ 6, 147–150 (2011).
  * [4] Dean, C. R. _et al._ Boron nitride substrates for high-quality graphene electronics.  _Nat. Nanotechnol._ 5, 722–726 (2010).
  * [5] Geim, A. K. & Grigorieva, I. V.  Van der Waals heterostructures.  _Nature_ 499, 419–425 (2013).
  * [6] Cao, Y. _et al._ Correlated insulator behaviour at half-filling in magic-angle graphene superlattices.  _Nature_ 556, 80–84 (2018).
  * [7] Cao, Y. _et al._ Unconventional superconductivity in magic-angle graphene superlattices.  _Nature_ 556, 43–50 (2018).
  * [8] Imada, M., Fujimori, A. & Tokura, Y.  Metal-insulator transitions.  _Rev. Mod. Phys._ 70, 1039 (1998).
  * [9] Dagotto, E.  Correlated electrons in high-temperature superconductors.  _Rev. Mod. Phys._ 66, 763 (1994).
  * [10] Tokura, Y. & Tomioka, Y.  Colossal magnetoresistive manganites.  _J. Magn. Magn. Mater._ 200, 1–23 (1999).
  * [11] Yu, Y. _et al._ High-temperature superconductivity in monolayer Bi2Sr2CaCu2O8+δ.  _Nature_ 575, 156–163 (2019).
  * [12] Hu, G. _et al._ Topological polaritons and photonic magic angles in twisted $\alpha$-MoO3 bilayers.  _Nature_ 582, 209–213 (2020).
  * [13] Schlom, D. G. _et al._ Elastic strain engineering of ferroic oxides.  _MRS Bulletin_ 39, 118–130 (2014).
  * [14] McKee, R. A., Walker, F. J. & Chisholm, M. F.  Crystalline oxides on silicon: the first five monolayers.  _Phys. Rev. Lett._ 81, 3014 (1998).
  * [15] Scherwitzl, R. _et al._ Metal-insulator transition in ultrathin LaNiO3 films.  _Phys. Rev. Lett._ 106, 246403 (2011).
  * [16] Huijben, M. _et al._ Critical thickness and orbital ordering in ultrathin La0.7Sr0.3MnO3 films.  _Phys. Rev. B_ 78, 094413 (2008).
  * [17] Anderson, P. W.  Absence of diffusion in certain random lattices.  _Phys. Rev._ 109, 1492 (1958).
  * [18] Ge, J.-F. _et al._ Superconductivity above 100 K in single-layer FeSe films on doped SrTiO3.  _Nat. Mater._ 14, 285–289 (2015).
  * [19] Lagally, M. G. & Zhang, Z.  Thin-film cliffhanger.  _Nature_ 417, 907–909 (2002).
  * [20] Schütz, P. _et al._ Dimensionality-driven metal-insulator transition in spin-orbit-coupled SrIrO3.  _Phys. Rev. Lett._ 119, 256404 (2017).
  * [21] Yoshimatsu, K. _et al._ Metallic quantum well states in artificial structures of strongly correlated oxide.  _Science_ 333, 319–322 (2011).
  * [22] King, P. _et al._ Atomic-scale control of competing electronic phases in ultrathin LaNiO3.  _Nat. Nanotechnol._ 9, 443–447 (2014).
  * [23] Yoo, H. K. _et al._ Thickness-dependent electronic structure in ultrathin LaNiO3 films under tensile strain.  _Phys. Rev. B_ 93, 035141 (2016).
  * [24] Sohn, B. _et al._ Sign-tunable anomalous hall effect induced by symmetry-protected nodal structures in ferromagnetic perovskite oxide thin films.  _arXiv:1912.04757_ (2019).
  * [25] Kim, B. _et al._ Novel $J_{eff}$ = 1/2 Mott state induced by relativistic spin-orbit coupling in Sr2IrO4.  _Phys. Rev. Lett._ 101, 076402 (2008).
  * [26] Uchida, M. _et al._ Pseudogap of metallic layered nickelate R2-xSrxNiO4 (R = Nd, Eu) crystals measured using angle-resolved photoemission spectroscopy.  _Phys. Rev. Lett._ 106, 027001 (2011).
  * [27] Jeong, S. G. _et al._ Phase instability amid dimensional crossover in artificial oxide crystal.  _Phys. Rev. Lett._ 124, 026401 (2020).
  * [28] Boschker, H. _et al._ Ferromagnetism and conductivity in atomically thin SrRuO3.  _Phys. Rev. X_ 9, 011027 (2019).
  * [29] Mackenzie, A. P. & Maeno, Y.  The superconductivity of Sr2RuO4 and the physics of spin-triplet pairing.  _Rev. Mod. Phys._ 75, 657 (2003).
  * [30] Grinenko, V. _et al._ Split superconducting and time-reversal symmetry-breaking transitions in Sr2RuO4 under stress.  _Nat. Phys._ 1–7 (2021).
  * [31] Shai, D. _et al._ Quasiparticle mass enhancement and temperature dependence of the electronic structure of ferromagnetic SrRuO3 thin films.  _Phys. Rev. Lett._ 110, 087004 (2013).
  * [32] Puchkov, A., Shen, Z.-X., Kimura, T. & Tokura, Y.  ARPES results on Sr2RuO4: Fermi surface revisited.  _Phys. Rev. B_ 58, R13322 (1998).
  * [33] Iwasawa, H. _et al._ Interplay among coulomb interaction, spin-orbit interaction, and multiple electron-boson interactions in Sr2RuO4.  _Phys. Rev. Lett._ 105, 226406 (2010).
  * [34] Chen, S.-D. _et al._ Incoherent strange metal sharply bounded by a critical doping in Bi2212.  _Science_ 366, 1099–1102 (2019).
  * [35] Kim, M. _et al._ Observation of kondo hybridization with an orbital-selective mott phase in 4d Ca2-xSrxRuO4.  _arXiv:2102.09760_ (2021).
  * [36] Koster, G. _et al._ Structure, physical properties, and applications of SrRuO3 thin films.  _Rev. Mod. Phys._ 84, 253 (2012).
  * [37] Chang, Y. J. _et al._ Fundamental thickness limit of itinerant ferromagnetic SrRuO3 thin films.  _Phys. Rev. Lett._ 103, 057201 (2009).
  * [38] Kim, B., Khmelevskyi, S., Franchini, C., Mazin, I. & Kim, K.  SrRuO3-SrTiO3 heterostructure as a possible platform for studying unconventional superconductivity in Sr2RuO4.  _Phys. Rev. B_ 101, 220502 (2020).
  * [39] Ko, E., Kim, B., Kim, C. & Choi, H. J.  Strong orbital-dependent d-band hybridization and fermi-surface reconstruction in metallic Ca2-xSrxRuO4.  _Phys. Rev. Lett._ 98, 226401 (2007).
  * [40] Lee, H. J., Kim, C. H. & Go, A.  Interplay between spin-orbit coupling and Van Hove singularity in the Hund’s metallicity of Sr2RuO4.  _Phys. Rev. B_ 102, 195115 (2020).
  * [41] Maiti, K. _et al._ Electronic structure of Ca1-xSrxVO3: A tale of two energy scales.  _Europhys. Lett._ 55, 246 (2001).
  * [42] Yang, H. F. _et al._ Comparative angle-resolved photoemission spectroscopy study of CaRuO3 and SrRuO3 thin films: pronounced spectral weight transfer and possible precursor of lower hubbard band.  _Phys. Rev. B_ 94, 115151 (2016).
  * [43] Mo, S.-K. _et al._ Prominent quasiparticle peak in the photoemission spectrum of the metallic phase of V2O3.  _Phys. Rev. Lett._ 90, 186403 (2003).
  * [44] Neupane, M. _et al._ Observation of a novel orbital selective mott transition in Ca1.8Sr0.2RuO4.  _Phys. Rev. Lett._ 103, 097001 (2009).
  * [45] Lu, D. _et al._ Synthesis of freestanding single-crystal perovskite films and heterostructures by etching of sacrificial water-soluble layers.  _Nat. Mater._ 15, 1255–1260 (2016).
  * [46] Kum, H. S. _et al._ Heterogeneous integration of single-crystalline complex-oxide membranes.  _Nature_ 578, 75–81 (2020).
  * [47] Ji, D. _et al._ Freestanding crystalline oxide perovskites down to the monolayer limit.  _Nature_ 570, 87–90 (2019).
  * [48] Kim, Y. K. _et al._ Fermi arcs in a doped pseudospin-1/2 Heisenberg antiferromagnet.  _Science_ 345, 187–190 (2014).
  * [49] Takagi, H. & Hwang, H. Y.  An emergent change of phase for electronics.  _Science_ 327, 1601–1602 (2010).
  * [50] Ahn, C., Triscone, J.-M. & Mannhart, J.  Electric field effect in correlated oxide systems.  _Nature_ 424, 1015–1018 (2003).
  * [51] Kresse, G. & Furthmüller, J.  Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.  _Phys. Rev. B_ 54, 11169 (1996).
  * [52] Kresse, G. & Joubert, D.  From ultrasoft pseudopotentials to the projector augmented-wave method.  _Phys. Rev. B_ 59, 1758 (1999).
  * [53] Herath, U. _et al._ Pyprocar: A Python library for electronic structure pre/post-processing.  _Comput. Phys. Commun._ 251, 107080 (2020).


11institutetext: Indian Institute of Technology Bombay, Mumbai, India
https://www.iitb.ac.in/

# Heterogeneous graphs model spatial relationship between biological entities
for breast cancer diagnosis

Akhila Krishna K 11 Ravi Kant Gupta 11 Nikhil Cherian Kurian 11 Pranav Jeevan
11 Amit Sethi 11

###### Abstract

The heterogeneity of breast cancer presents considerable challenges for its
early detection, prognosis, and treatment selection. Convolutional neural
networks often neglect the spatial relationships within histopathological
images, which can limit their accuracy. Graph neural networks (GNNs) offer a
promising solution by coding the spatial relationships within images. Prior
studies have investigated the modeling of histopathological images as cell and
tissue graphs, but they have not fully tapped into the potential of extracting
interrelationships between these biological entities. In this paper, we
present a novel approach using a heterogeneous GNN that captures the spatial
and hierarchical relations between cell and tissue graphs to enhance the
extraction of useful information from histopathological images. We also
compare the performance of a cross-attention-based network and a transformer
architecture for modeling the intricate relationships within tissue and cell
graphs. Our model demonstrates superior efficiency in terms of parameter count
and achieves higher accuracy compared to the transformer-based state-of-the-
art approach on three publicly available breast cancer datasets – BRIGHT,
BreakHis, and BACH.

###### Keywords:

Graph Heterogeneous Histology Transformer

##  1 Introduction

Breast cancer is the most common cancer among women globally, and it continues
to pose significant challenges for early diagnosis, prognosis, and treatment
decisions, given its diverse molecular and clinical subtypes [29]. To address
these challenges, recent advancements in machine learning techniques have
paved the way for improved accuracy and personalized treatment strategies.
However, most convolutional neural networks (CNNs) overlook the spatial
relationships within histopathological images, treating them as regular grids
of pixels [5, 14, 16, 19]. To overcome this limitation, graph neural networks
(GNNs) have emerged as a promising alternative for classifying breast cancer.

GNNs are designed to handle complex graph structures, making them well-suited
for tasks that involve analyzing relationships between entities [27, 26, 18].
By representing histopathological images as graphs, with image regions and
structures as nodes and their spatial relationships as edges, GNNs can capture
the inherent spatial context within the images [21, 24, 3, 10, 28]. This
allows them to extract valuable information and patterns that may be missed by
other machine-learning methods.

In this paper, we propose the use of heterogeneous graph convolutions between
the cell and the tissue graphs to enhance the extraction of spatial
relationships within histology images. This approach allows for the
incorporation of diverse, multi-scale, and comprehensive features and spatial
relationships by modeling both cell and tissue structures as well as their
hierarchical relationships. Specifically, we introduce three features that
have not been previously used in GNNs for histopathology images: (1)
heterogeneous graph convolutions along with a transformer which outperforms
[15] on BRIGHT [7], (2) an adaptive weighted aggregation technique with
heterogeneous convolutions that outperforms [15] and is more efficient in
terms of the number of parameters, and (3) the cross-attention modules of
CrossVit [9] along with heterogeneous graph convolutions to extract the
spatial relationship between cell and tissue graphs. We also analyzed
different methods of k-nearest neighbor (kNN) graph building for cell and
tissue graphs and found that edges based on node feature similarities
performed better than other methods such as spatial closeness [22, 23] or
dynamically learnable layers for edge creation [15]. Extensive experiments
conducted on three publicly available breast cancer histology datasets –
BRIGHT [7], BACH [4], and BreakHis [6] – demonstrate the gains of our method.

##  2 Related Work

The first work on entity-level analysis for histology was [28] in which a cell
graph was created and a graph convolutional network was used for processing
the graph. Other works, such as [22, 23, 15], have also focused on the entity-
level analysis of histology images by constructing cell graphs and tissue
graphs and by constructing more than two levels of subgraphs to capture the
spatial relationship in the image. In [22], the authors introduced an LSTM-
based feature aggregation technique to capture long-range dependencies within
the graphs. By utilizing the LSTM, they aimed to capture the temporal
dependencies between cells and tissues, enhancing the understanding of their
relationships. In contrast, [15] took a different approach by constructing
more than two levels of subgraphs and using a spatial hierarchical graph
neural network. This network aimed to capture both long-range dependencies and
the relationships between the cell and tissue graphs. To achieve this, the
authors employed a transformer-based feature aggregation technique, leveraging
the transformer’s ability to capture complex patterns and dependencies in the
data. However, it was observed that both of these approaches fell short of
fully extracting the intricate relationship between tissue and cell graphs.
Hence, there is a scope to explore alternative methods or combinations of
techniques to better understand and leverage the spatial relationships within
histology images for improved analysis and classification of breast cancer and
other diseases.

##  3 Methodology

Figure 1: Cell and tissue graph formation

###  3.1 Tissue and cell graph extraction from histology images

The interrelationship between cells and tissue is investigated through the
construction of cell and tissue graphs. The goal is to perform multi-level
feature analysis by extracting relevant features from these entities. To
construct the cell graph, a pre-trained model called HoverNet [12] is used for
nuclei detection. The feature representation for each nucleus is extracted by
processing patches around the nuclei with a ResNet34 encoder [14, 15].
Similarly, the tissue entities are identified using segmentation masks, which
are obtained by applying the Simple Linear Iterative Clustering (SLIC)
algorithm for superpixel segmentation [1, 15]. Once the tissue entities are
identified, their feature representations are extracted in the same manner as
the cell entities. We then utilize kNN to get the $k$ most similar nodes of
all nodes based on the distance of node feature representations for the
formation of edges between cell entities and tissue entities. In our
experiment, we used $k=5$ for all models and datasets. The edges between cells
and tissues are formed by using the spatial position of cells and tissues. We
treat a cell node $C_{i}$ and a tissue node $T_{i}$ as connected if $C_{i}\in
T_{i}$.

###  3.2 Heterogeneous Graph Convolution

The interaction between cell and tissue graphs has to be captured effectively
for better analysis which is done using heterogeneous graph convolutions. We
define a heterogenous graph($H$) as a union of cell-to-cell, tissue-to-tissue,
and cell-to-tissue relations (which are defined by edges) along with their
node features.

$H={\\{C,T,E_{cell->cell},E_{tissue->tissue},E_{cell->tissue}\\}}$ (1)

where $C$ and $T$ are the features of the node in the cell graph and the
tissue graph and $E_{A->B}$ is the list of edges between the elements in sets
A and B.

We utilize Graph Sage convolutions[13] to transmit messages individually from
the source node to the target node for each relation. When multiple
relationships direct to the same target node, the outcomes are combined using
a specified aggregation method. This is how heterogeneous graph convolution is
implemented. We can formulate it as follows: let $z_{jR_{i}}$ be the output
vector representations on node $j$ due to Graph Sage convolution on the nodes
defined by relation $R_{i}$, where
$R_{i}\in\\{cell->cell,cell->tissue,tissue->tissue\\}$, then the final vector
representation on node $j$, $z_{j}$ is

$z_{j}=Aggregator(\\{z_{jR_{u}},\forall u\in U\\})$ (2)

where $U$ is the set of relations directing to the node $j$. This is
implemented using PyTorch geometric library.

Figure 2: Proposed architectural variants of Heterogeneous Graph Neural
Network (HG): (a) HG with adaptive weighted aggregation of multi-level
features, (b) HG with CrossVit for hierarchical feature fusion, and (c) HG
with transformer for attentive interaction.

###  3.3 Adaptive weighted aggregation for multi-level feature fusion

Low-level and high-level features are important in classification tasks. So we
use skip connections for its extraction and adaptive weighted aggregation
layer for feature fusion. Adaptive weighted aggregation layer $f$ is
represented as follows :

$f(w_{1},w_{2},...w_{n},F_{1},F_{2}....F_{n})=w_{1}F_{1}+w_{2}F_{2}+w_{3}F_{3}+.......w_{n}F_{n},$
(3)

where $w_{i}$ is trainable weights of dimension $1\times 1$ and $F_{i}$ is
feature vector of dimension $256\times 1$ for $i\in{1,2,3,...n}$.

###  3.4 Cross-attention feature fusion and attentive interaction using
transformers

The transformer encoder of [25] which uses self-attention is used for
extracting the long-range dependencies between the nodes in the graph.
Formally, let us say the transformer encoder is $T$ and the input to $T$ is
$Concat(N_{c},N_{t})$ where $N_{c}$ and $N_{t}$ are node features of cell and
tissue graph respectively. The output from the transformer encoder goes to MLP
layers to get the final output. $N_{c}$ and $N_{t}$ are of dimension $C\times
1\times 256$ and $C\times 1\times 256$ respectively where $C$ is the number of
nodes in the cell graph (we extrapolate the tissue graph with zeros for
expanding the dimension of it to the same dimension as that of cell graph).

Cross-attention modules of [9] are used for extracting the interaction between
cell graph and tissue graph which then is then fed into multi-layer perceptron
layers to get the required output.

##  4 Experiments and Results

###  4.1 Clinical Datasets and evaluation methods

For assessing the utility of graph based deep learning methods, breast cancer
histology datasets form an ideal test bed due to the diversity of its
subtypes. We used three datasets in this work. The BRIGHT dataset is a breast
cancer subtyping dataset released as a part of the Breast tumor Image
classification on Gigapixel Histopathological images challenge [7]. The breast
cancer histopathology image datasets (BACH) [4] has four disease states –
normal, benign, ductal carcinoma in-situ (DCIS), and invasive ductal carcinoma
(IDC). The BreakHis [6] has malignant and non-malignant classes.

####  4.1.1 BRIGHT Dataset

The BRIGHT dataset contains 4025 hematoxylin & Eosin (H&E) stained breast
cancer histology images scanned at 0.25mn/pixel resolution. It is classified
into 6 classes: Pathological Benign (PB), Usual Ductal Hyperplasia (UDH), Flat
Epithelial Atypia (FEA), Atypical Ductal Hyperplasia (ADH), Ductal Carcinoma
In Situ (DCIS) and Invasive Carcinoma (IC). These are further grouped into 3:
cancerous tumors (DCIS and IC), non-cancerous tumors (PB and UDH) and pre-
cancerous tumors (FEA and ADH) [7].

####  4.1.2 BACH Dataset

The BACH dataset contains 400 hematoxylin and eosin (H&E) stained breast
cancer histology images with pixel scale $0.42\mu m\times 0.42\mu m$. It is
classified into 4 classes: normal, benign, in situ carcinoma, and invasive
carcinoma [4].

####  4.1.3 BreakHis dataset

The BreakHis dataset contains 9109 microscopic images of breast cancer at 400X
magnifying factor. It contains 2480 benign and 5429 malignant tissue images
[6].

###  4.2 Experimental Setup

All the experiments are implemented in Pytorch and using Pytorch-geometric and
histocartography library [17]. The F-score was used as the evaluation metric.
We proposed three models and compared each other and also compared with the
state-of-the-art model [15]. Hovernet was used for nuclei detection in all
models for all datasets. The feature vector of length 512 for each nucleus was
extracted from a patch size of 72, 72, and 48 around each nuclei using Resnet
34 model [14] for the BRIGHT, BACH, and BreakHis respectively. We also
compared 3 methods of edge formation of graphs: dynamic structural learning
[15], kNN algorithm on the distance between cell entities and the tissue
entities, and kNN algorithm on node feature representation of cell entities
and tissue entities. The batch size was set to 32. A learning rate of
$1e^{-4}$ and Adam optimizer with a weight decay of $5e^{-4}$ were used while
training the models.

###  4.3 Comparison with the state-of-the-art methods

The results for six-class classification and three-class classification on the
BRIGHT dataset are listed in Table 1 and 2 respectively. The four-class
classification on the BACH dataset is shown in Table 3 and the 2-class
classification on the BreakHis dataset is shown in Table 4.

Table 1: Weighted F-score (%) of HG compared with other methods on BRIGHT dataset for six-class classification. Model |  DCIS |  IC |  PB |  UDH |  ADH |  FEA |  Weighted F-score |  Number of parameters (in million)
---|---|---|---|---|---|---|---|---
SHNN |  72.9 |  87.3 |  74.7 |  51.8 |  45.0 |  73.9 | 69.6 | 2.2
HG + AWA |  72.8 |  88.0 |  72.7 |  54.1 |  48.9 |  74.2 | 70.1 | 1.5
HG + CrossVit |  77.9 |  88.0 |  71.1 |  53 |  54 |  73 | 71.3 | 2.5
HG+transformer |  84 |  92 |  79 |  54 |  49 |  78.5 | 75.3 | 2.0
Table 2: Weighted F-score (%) of HG compared with other methods on BRIGHT dataset across four test folds for three-class classification. Model | 1 | 2 | 3 | 4 | $\mu\pm\sigma$ | Cancerous | Non-cancerous | Pre-cancerous
---|---|---|---|---|---|---|---|---
SHNN | 78 | 78.11 | 78.93 | 79.21 | 78.56$\pm$0.56 | 85.21$\pm$2.07 | 78.08$\pm$0.81 | 72.11$\pm$0.82
HG + CrossVit | 78.75 | 78.28 | 77.2 | 74.29 | 77.13$\pm$2.00 | 83.93$\pm$1.12 | 76.29$\pm$3.08 | 70.38$\pm$2.35
HG + AWA | 80.24 | 77.98 | 80.82 | 80.55 | 79.89$\pm$1.30 | 87.08$\pm$1.49 | 78.09$\pm$2.64 | 73.75$\pm$1.36
HG+transformer | 83.45 | 82.59 | 81.86 | 80.36 | 82.06$\pm$1.31 | 88.78$\pm$1.96 | 80.61$\pm$1.33 | 76.08 $\pm$1.36
Table 3: Weighted F-score (%) of HG compared with other methods on BACH dataset. Model | Normal | Benign | InSitu | Invasive |  Weighted F-score
---|---|---|---|---|---
SHNN | 90 | 80 | 85.10 | 84.21 | 84.83
HG+transformer | 88.88 | 86.48 | 85.20 | 90.10 | 86.62
Table 4: Weighted F-score (%) of HG compared with other methods on BreakHis dataset. Model | Malignant | Benign |  Weighted F-score
---|---|---|---
SHNN | 89.16 | 78.17 | 85.46
HG+transformer | 95.62 | 91.70 | 94.30

###  4.4 Ablation Studies

We performed studies on different methods of graph formation and removing and
adding different layers of the model. The results are listed in Table 5. The
spatial hierarchical neural network (SHNN) of [15] is Graph Sage convolutions
in homogeneous graphs with a transformer. It can be observed that the addition
of a transformer to Graph Sage convolutions in the SHNN led to an increase in
F-score by 4% whereas in the case of a heterogeneous convolutional network the
addition of a transformer to heterogeneous convolutions (HG) gave an increase
in the F-score of nearly 7%. It can be concluded that heterogeneous
convolutions led to an increase in interactions between cell and tissue
features such that the output of these convolutions has better features for
the transformer to work on than that of normal sage convolutions. It is also
to be noted that in our model we used just two heterogeneous convolutional
layers as opposed to 6 graph sage convolutional layers in [15].

We tried three different graph formation methods for cell and tissue graph
formation. The results are listed in Table 6. Our method of graph formation
using kNN algorithm on node features was seen to perform better than all of
the other three methods.

Table 5: Weighted F-score (%) of different graph-based methods compared on BRIGHT dataset. Model |  DCIS |  IC |  PB |  UDH |  ADH |  FEA | Weighted F-score
---|---|---|---|---|---|---|---
Graph Sage Conv |  72.1 |  84.8 |  70 |  39 |  44.2 |  68.5 | 65.3
SHNN |  72.9 |  87.3 |  74.7 |  51.8 |  45 |  73.9 | 69.6
HG |  76.5 |  85.6 |  72.5 |  49.1 |  40.25 |  72 | 68.2
HG+transformer |  84 |  92 |  79 |  54 |  49 |  78.5 | 75.3
Table 6: Weighted F-score (%) of different edge-formation methods compared on BRIGHT dataset. Model |  DCIS |  IC |  PB |  UDH |  ADH |  FEA | Weighted F-score
---|---|---|---|---|---|---|---
DSL |  81.7 |  90.9 |  78.0 |  55.0 |  48.3 |  75.2 | 73.6
KNN on distance |  81.1 |  90.4 |  77.8 |  56.8 |  57.7 |  75.9 | 74.9
KNN on node features |  84.0 |  92 |  79 |  54 |  49 |  78.5 | 75.3

##  5 Conclusion

We introduced three novel architectures for histopathological image
classification based on heterogeneous graph neural networks. Our research
highlights the capability of heterogeneous graph convolutions in capturing the
spatial as well as hierarchical relationship within the images, which allows
it to surpass the performance of existing methods for histopathology image
analysis. This emphasizes the significance of considering the relationship
between cells and the surrounding tissue area for accurate cancer
classification. Additionally, we established that the self-attention-based
model outperforms the cross-attention-based model. We attribute this
observation to the ability of self-attention to extract long-range
dependencies within the graphs. Furthermore, we demonstrated the importance of
analyzing the relationship between similar parts of the histopathology image,
showcasing that constructing the graphs based on the similarity between node
features yields superior results compared to other approaches, such as those
based on spatial distance. However, we think that in the future both
similarity and spatial distance should be combined for graph edge formation.
It would also be good to explore novel techniques for enhancing graph
convolutions in order to extract long-range dependencies within the graph more
effectively. Additionally, there is a potential to develop innovative methods
for graph pooling that minimize information loss. These directions of research
would contribute to further advancements in histopathological image
classification using heterogeneous graph neural networks.

## References

  * [1] Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34(11), 2274–2282 (2012)
  * [2] Anand, D., Gadiya, S., Sethi, A.: Histographs: graphs in histopathology. In: Medical Imaging 2020: Digital Pathology. vol. 11320, pp. 150–155. SPIE (2020)
  * [3] Anklin, V., Pati, P., Jaume, G., Bozorgtabar, B., Foncubierta-Rodriguez, A., Thiran, J.P., Sibony, M., Gabrani, M., Goksel, O.: Learning whole-slide segmentation from inexact and incomplete labels using tissue graphs. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24. pp. 636–646. Springer (2021)
  * [4] Aresta, G., Araújo, T., Kwok, S., Chennamsetty, S.S., Safwan, M., Alex, V., Marami, B., Prastawa, M., Chan, M., Donovan, M., et al.: Bach: Grand challenge on breast cancer histology images. Medical image analysis 56, 122–139 (2019)
  * [5] Bai, J., Jiang, H., Li, S., Ma, X., et al.: Nhl pathological image classification based on hierarchical local information and googlenet-based representations. BioMed research international 2019 (2019)
  * [6] Benhammou, Y., Achchab, B., Herrera, F., Tabik, S.: Breakhis based breast cancer automatic diagnosis using deep learning: Taxonomy, survey and insights. Neurocomputing 375, 9–24 (2020)
  * [7] Brancati, N., Anniciello, A.M., Pati, P., Riccio, D., Scognamiglio, G., Jaume, G., De Pietro, G., Di Bonito, M., Foncubierta, A., Botti, G., et al.: Bracs: A dataset for breast carcinoma subtyping in h&e histology images. Database 2022, baac093 (2022)
  * [8] Buterez, D., Janet, J.P., Kiddle, S.J., Oglic, D., Liò, P.: Graph neural networks with adaptive readouts. Advances in Neural Information Processing Systems 35, 19746–19758 (2022)
  * [9] Chen, C.F.R., Fan, Q., Panda, R.: Crossvit: Cross-attention multi-scale vision transformer for image classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 357–366 (2021)
  * [10] Chen, R.J., Lu, M.Y., Wang, J., Williamson, D.F., Rodig, S.J., Lindeman, N.I., Mahmood, F.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging 41(4), 757–770 (2020)
  * [11] Fey, M., Lenssen, J.E.: Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 (2019)
  * [12] Graham, S., Vu, Q.D., Raza, S.E.A., Azam, A., Tsang, Y.W., Kwak, J.T., Rajpoot, N.: Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis 58, 101563 (2019)
  * [13] Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. Advances in neural information processing systems 30 (2017)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] Hou, W., Huang, H., Peng, Q., Yu, R., Yu, L., Wang, L.: Spatial-hierarchical graph neural network with dynamic structure learning for histological image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 181–191. Springer (2022)
  * [16] Hou, W., Wang, L., Cai, S., Lin, Z., Yu, R., Qin, J.: Early neoplasia identification in barrett’s esophagus via attentive hierarchical aggregation and self-distillation. Medical image analysis 72, 102092 (2021)
  * [17] Jaume, G., Pati, P., Anklin, V., Foncubierta, A., Gabrani, M.: Histocartography: A toolkit for graph analytics in digital pathology. In: MICCAI Workshop on Computational Pathology. pp. 117–128. PMLR (2021)
  * [18] Jia, Z., Lin, Y., Wang, J., Zhou, R., Ning, X., He, Y., Zhao, Y.: Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification. In: IJCAI. vol. 2021, pp. 1324–1330 (2020)
  * [19] Li, Y., Xie, X., Shen, L., Liu, S.: Reverse active learning based atrous densenet for pathological image classification. BMC bioinformatics 20(1), 1–15 (2019)
  * [20] Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015)
  * [21] Lu, W., Graham, S., Bilal, M., Rajpoot, N., Minhas, F.: Capturing cellular topology in multi-gigapixel pathology images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 260–261 (2020)
  * [22] Pati, P., Jaume, G., Fernandes, L.A., Foncubierta-Rodríguez, A., Feroce, F., Anniciello, A.M., Scognamiglio, G., Brancati, N., Riccio, D., Di Bonito, M., et al.: Hact-net: A hierarchical cell-to-tissue graph neural network for histopathological image classification. In: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis: Second International Workshop, UNSURE 2020, and Third International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings 2. pp. 208–219. Springer (2020)
  * [23] Pati, P., Jaume, G., Foncubierta-Rodriguez, A., Feroce, F., Anniciello, A.M., Scognamiglio, G., Brancati, N., Fiche, M., Dubruc, E., Riccio, D., et al.: Hierarchical graph representations in digital pathology. Medical image analysis 75, 102264 (2022)
  * [24] Raju, A., Yao, J., Haq, M.M., Jonnagaddala, J., Huang, J.: Graph attention multi-instance learning for accurate colorectal cancer staging. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23\. pp. 529–539. Springer (2020)
  * [25] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [26] Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y.: A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems 32(1), 4–24 (2020)
  * [27] Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., Sun, M.: Graph neural networks: A review of methods and applications. AI open 1, 57–81 (2020)
  * [28] Zhou, Y., Graham, S., Alemi Koohbanani, N., Shaban, M., Heng, P.A., Rajpoot, N.: Cgc-net: Cell graph convolutional network for grading of colorectal cancer histology images. In: Proceedings of the IEEE/CVF international conference on computer vision workshops. pp. 0–0 (2019)
  * [29] Zielinska, H.A., Daly, C.S., Alghamdi, A., Bahl, A., Sohail, M., White, P., Dean, S.R., Holly, J.M., Perks, C.M.: Interaction between grp78 and igfbp-3 affects tumourigenesis and prognosis in breast cancer patients. Cancers 12(12), 3821 (2020)