# Revisiting the physical origin and nature of surface states in inverted-band
semiconductors

Alexander Khaetskii  Air Force Research Laboratory, Wright-Patterson AFB, Ohio
45433, USA  Vitaly Golovach  Centro de Física de Materiales (CFM-MPC), Centro
Mixto CSIC-UPV/EHU, 20018 Donostia-San Sebastián, Spain  Departamento de
Polímeros y Materiales Avanzados: Física, Química y Tecnología, Facultad de
Química, University of the Basque Country UPV/EHU, 20080 Donostia-San
Sebastián, Spain  Donostia International Physics Center (DIPC), 20018
Donostia-San Sebastián, Spain  IKERBASQUE, Basque Foundation for Science,
48013 Bilbao, Spain  Arnold Kiefer  Air Force Research Laboratory, Wright-
Patterson AFB, Ohio 45433, USA

(August 27, 2024)

###### Abstract

We revisit the problem of surface states in semiconductors with inverted band
structures, such as $\alpha$-Sn and HgTe. We unravel the confusion that arose
over the past decade regarding the origin of the surface states, their
topological nature, and the role of strain. Within a single minimalistic
description, we reconcile different solutions found in the 1980s with the
results obtained from modern-day numerical simulations, allowing us to
unambiguously identify all branches of surface states around the
$\Gamma$-point of the Brillouin zone in different regimes. We also show that
strain is a smooth “deformation” to the surface states, following the usual
_continuity principle_ of physics, and not leading to any drastic change of
the physical properties in these materials, in contrast to what has recently
been advanced in the literature. We consider biaxial in-plane strain that is
either tensile or compressive, leading to different branches of surface states
for topological insulators and Dirac semimetals, respectively. Our model can
help in interpreting numerous experiments on topological surface states
originating from inverted-band semiconductors.

##  I Introduction

Electronic surface states of solids have long been a topic of interest because
of their importance in practical physical devices. The origin and properties
of surface states for many semiconductor solids have been explained in great
detail over several decades, beginning with Tamm and Shockley Tamm ; Shockley
. More recently, the prediction and discovery of topological insulators and
their surface states have revived interest in this topic owing to their
special properties Hasan ; Qi . Present-day researchers often generate models
of surface bands with sophisticated computational software, but they do not
usually provide insight into the physical origin of these surface bands, often
leading to a misguided treatment. Furthermore, recent works seem unaware of
the foundational analytical work conducted decades ago that provide similar
physical predictions about surface states though at that time not recognizing
them as having a specific topological character Dyak ; Volkov ; Cade ; Suris ;
Volkov1 .

Topological insulators (TIs) serve as an excellent bridge between traditional
electronic bandstructure theory and a more modern approach including
topological aspects. TIs emerge from a class of narrow-gap materials whose
strong spin-orbit coupling leads to an inverted band structure and the
formation of helical (i.e. spin-momentum locked) states on their surfaces Dyak
; Volkov ; Cade ; Suris ; Volkov1 ; Bernevig ; Kane ; Dai  that are protected
from nonmagnetic perturbations. Growing attention to topological insulators is
fueled by a fundamental interest in solid-state spin-physics and the prospect
of designing novel electronic devices Ando . Topological surface states have
been observed in a number of 3D compounds, such as Bi2Se3, Bi2Te3, etc. Zhang
; Xia  Examples of 2D TIs with 1D helical channels at the sample edges include
HgTe/CdHgTe Bernevig1 ; Konig  and InAs/GaSb Liu ; Krishtopenko  quantum wells
of certain thicknesses. Experimentally, surface states in 3D topological
insulators are usually revealed by the angle- and spin-resolved photoemission
spectroscopy Xia ; Hsieh  and magneto-transport measurements Brune ; Kozlov ;
Checkelsky ; Analytis ; Ren ; Xia1 ; Pan .

Strain is frequently present in real films and plays a role in revealing
topological states by breaking crystalographic symmetries. Several attempts to
reveal experimentally the surface states in strained HgTe films (3D
topological insulators) were made in magneto-transport experiments Brune ;
Kozlov . It is interesting that the theoretical interpretation of the
experimental data in these two works are based on very different physical
pictures of the surface states in HgTe material. In particular, the surface
states in Ref. Kozlov,  are treated as linear in k-vector Dirac states that
exist in the gap between conduction s-band and light-hole p-band when the
presence of the heavy-hole band is totally ignored. In other words, the
topological surface states and bulk heavy-hole states are considered as
completely independent. In an opposing treatment offered in Ref. Brune, ,
strong hybridization between surface states and bulk heavy-hole states makes
the physical picture totally different, where only the surface states lying
within the strain-created gap between the light-hole and heavy-hole bands have
physical meaning. Such a difference in interpretation requires further
examination. As we will explain in this work, we rather support the physical
description of the surface states used in Ref. Brune, .

Another material attracting much attention now is $\alpha$-Sn, which has a
band structure similar to HgTe. Recently published research concerning surface
states in $\alpha$-Sn (see, for example, Refs. Ohtsubo, ; Barfuss, ; Kufner, ;
Rogalev, ; Shia, ; Ding, ; Ding1, ) clearly demonstrate the confusion that
exists in the community about the role of band structure and strain. Many
authors who treat the surface states by various numerical methods often ignore
the presence of the heavy-hole band and solve the problem with only the light-
hole and electron s-type bands. As we show below, this leads to an incorrect
physical picture of the surface states, see Fig. 1. Moreover, application of
even a small strain in their solutions causes a sudden appearence of the Dirac
surface states deeply below the degeneracy point ($\Gamma_{8}$) of the light
and heavy holes, i.e. within the $\Gamma_{8}-\Gamma_{6}$ gap. This manifests
the unphysical character of the obtained solutions. (We mean here a tensile
in-plane strain applied to the inverted-band-gap semiconductor that opens a
gap at the $\Gamma$-point between the light and heavy-hole bands). The
troubles outlined above lead, in turn, to difficulties interpreting the
experimental data obtained with the surface states in the materials like HgTe
and $\alpha$-Sn.

Figure 1: Incorrect picture of surface states obtained after neglecting the
coupling to the heavy-hole band (HH). The surface states are formed in a two-
band model consisting of electrons (e) and light holes (LH) in an inverted-
band semiconductor.

The goal of this work is to clarify the roles of band structure, strain, and
spin in constructing a physical picture of surface states consistent with
conventional solid-state theory. The authors of many results obtained solely
by numerical calculation seem completely unaware of previous analytical works
that treated surface states in similar materials (like HgTe) decades ago Dyak
; Volkov ; Cade ; Suris ; Volkov1 , making them difficult to relate. We seek
to reconcile these differences and show below that those older works contain
all the essential physics and treat the band structure of the materials
correctly while still describing the surface states completely. In particular,
we describe the intimate relation of the surface states in HgTe gapless
semiconductor first predicted within the Luttinger model in Ref. Dyak,  and
surface states obtained within the more general Kane model that also takes the
s-band into account. We demonstrate that these Dyakonov-Khaetskii (DK) states
Dyak  possess all the essential features of the topological states in the
presence of strain which induces a gap between the light and heavy-hole bands
(topological insulator regime). As such, they should play the most important
role in the experiments that probe the topological properties of the sample
when the Fermi level is located within this strain-induced gap. Moreover, we
show that strain does not drastically change the properties of surface states
that form the Dirac cone within the $\Gamma_{8}-\Gamma_{6}$ gap, which is
again in strong contrast to the results advanced recently in the literature.

We take a pedagogical approach, beginning with a simple, idealized model that
captures the essential physics. As such we consider a variant of the Kane k-P
model with infinite mass of heavy holes in the case of a single abrupt
interface between an inverted material like HgTe and a direct material like
CdTe or vacuum. This model takes into account three bands: an s-type
conduction band and p-type light and heavy hole bands, see Fig. 2. The split-
off band is relatively far away in energy (strong spin-orbit coupling) and
does not play a significant role in forming the surface states. By considering
the flat heavy-hole band we deal with the case of a real gap between hole
bands ($\Gamma_{8}$) and electron s-band ($\Gamma_{6}$) when the density of
states within this gap is zero. These features of the model make the physics
of the surface states located in the $\Gamma_{8}-\Gamma_{6}$ gap very simple
and reveals their essential features. We then address the realistic case of
finite heavy-hole mass and show the surface states located within the
$\Gamma_{8}-\Gamma_{6}$ gap overlap the bulk heavy-hole states in energy and
are strongly hybridized with them, making those surface states marginal in
observable phenomena.

For a detailed description of the surface states that exist within the
projected gap between the light and heavy-hole bands (DK states) we use the
Luttinger model which is applicable for the case of arbitrary masses of light
and heavy holes. We were able to find exact analytical solutions for the
surface states within this model for both signs of the strain energy $\Delta$,
covering the TI regime ($\Delta>0$) and the Dirac semimetal regime
($\Delta<0$). In the latter case, which is achieved by applying a compressive
in-plane strain (hence tensile in the normal direction), an intersection of
the light-hole and heavy-hole bands occurs forming two Dirac points with
linear bulk dispersion nearby. This Dirac semimetal phase has been attracting
a lot of attention recently both theoretically and experimentally, see Refs.
(Ding, ; Ding1, ). We show that the dispersion of surface states which arise
around the conic point of the bulk projected states crucially depends on the
applied strain, in contrast to the TI regime.

Our simple models allow us to trace the origins of the incorrect physical
picture of the surface states obtained in many recent publications, namely,
neglecting the coupling to the heavy holes band that leads directly to the
simple picture of linear in k-vector Dirac surface states crossing the whole
gap between the conduction and light-hole bands, see Fig. 1. We stress again
that this latter picture does not reflect the true nature of the surface
states.

Figure 2: Generic band structure variation across an interface between an
inverted-band-gap semiconductor (SC) (left: $E_{g}<0$) and a SC with a regular
band order (right: $E_{g}>0$). A tensile in-plane strain applied to the
inverted-band-gap SC induces a splitting between the light-hole and heavy-hole
states and opens a gap $\Delta$ at the $\Gamma$-point. At certain energies
(not shown), the interface supports surface states with a probability density
which exponentially decays into both regions, thus being localized at the
interface (bottom panel).

##  II The $3\times 3$ Kane model, an abrupt interface of two materials -with
inverted and direct bands

We consider here the simplest possible problem: the interface between two
materials, one of them having an inverted band structure and the other -a
direct one. Our description is quite general and is applicable for many
materials, but as a concrete example one can consider the (HgTe/CdTe) case.
Additionally, we assume a biaxial tensile in-plane strain applied to the
inverted band material as is often accomplished in practice by epitaxial
growth of a thin film on a much thicker substrate, see Fig. 2. The band
structures are described by the $6\times 6$ Kane model that takes into account
three bands, electrons (e) (s-symmetry band), and two p-bands: light (lh) and
heavy holes (hh). To simplify the treatment, we can from the very beginning
use the time-reversal character of the problem and reduce the model to
$3\times 3$ by choosing the proper coordinate system Subashiev . We choose the
angular momentum quantization axis z in the plane of the interface and the
direction of the carrier motion along y; x denotes the direction normal to the
interface. Then the two groups of states (e 1/2, lh -1/2, hh 3/2) and (e -1/2,
lh 1/2, hh -3/2) do not mix. The Hamiltonian matrix for the first group of
states is

$\displaystyle\hat{H}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}\epsilon_{c1,2}&Pk_{-}/\sqrt{6}&Pk_{+}/\sqrt{2}\\\
Pk_{+}/\sqrt{6}&\epsilon_{v1,2}-\Delta/4&\sqrt{3}\Delta/4\\\
Pk_{-}/\sqrt{2}&\sqrt{3}\Delta/4&\epsilon_{v1,2}+\Delta/4\end{bmatrix}$ (1)

(The Hamiltonian matrix that refers to the second group of states with the
opposite sign of the angular momentum projection on the z-axis can be obtained
from Eq. (1) by replacing $k_{y}$ by $-k_{y}$). In Eq. (1) $P$ is the Kane
matrix element which we consider coordinate independent,
$\hat{k}_{\pm}=\hat{k}_{x}\pm ik_{y}$, $\hat{k}_{x}=-id/dx$,
$\epsilon_{c1,2},\epsilon_{v1,2}$ are conduction- and valence-band center
energies in inverted-band semiconductor I and in semiconductor II with direct
bands, respectively. We assume that HgTe layer is strained due to the lattice
mismatch between CdTe and HgTe and this tensile in-plane strain opens the gap
$\Delta>0$ between the light and heavy-hole bands. If the CdTe is oriented
along (001), then the strain tensor components are given by
$\epsilon_{yy}=\epsilon_{zz}=(a_{\textrm{CdTe}}-a_{\textrm{{HgTe}}})/a_{\textrm{HgTe}}>0$
(tensile in-plane strain), $\epsilon_{xx}<0$; the non-diagonal components are
equal to zero. Here $a_{\textrm{CdTe}}$ and $a_{\textrm{HgTe}}$ are the
lattice constants of the corresponding materials. The effects of the strain
tensor are incorporated in the Kane model through the Bir-Pikus Hamiltonian
Bir , which is easily obtained from the Kane Hamiltonian with the substitution
$k_{i}k_{j}\rightarrow\epsilon_{ij}$. The value of the strain gap in our case
is given by $\Delta=2b(\epsilon_{xx}-\epsilon_{yy})>0$, where $b$ is the
uniaxial deformation potential ($b=-1.5eV$ for HgTe).

With the proper unitary transformation one can bring the Hamiltonian Eq. (1)
to the following form:

$\displaystyle\hat{H}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}\epsilon_{c1,2}&\frac{(2\hat{k}_{x}+ik_{y})P}{\sqrt{6}}&iPk_{y}/\sqrt{2}\\\
\frac{(2\hat{k}_{x}-ik_{y})P}{\sqrt{6}}&\epsilon_{v1,2}+\Delta/2&0\\\
-iPk_{y}/\sqrt{2}&0&\epsilon_{v1,2}-\Delta/2\end{bmatrix}$ (2)

###  II.1 No coupling to the heavy-hole band

Before proceeding further we can make here the following important note which
explains the origin of the wrong physical picture of the surface states
obtained in Refs. Ohtsubo, ; Barfuss, ; Kufner, ; Rogalev, . If one neglects
altogether the interaction of the $\Gamma_{6}$ band with the heavy-hole band,
given by elements $H_{13}$ and $H_{31}$ in Eq. (2), then one obtains the
anisotropic Dirac model which describes the interaction of the two bands,
electrons and light holes, as described by the 2x2 block in the upper left
corner of the matrix in Eq. (2). Let us find the energy dispersion of the
surface states for this reduced case, considering semi-infinite sample of HgTe
located at $x<0$ and CdTe sample located at $x>0$. The wave functions decay in
the left-hand region as $\exp(\kappa_{1}x)$ and in the right-hand region as
$\exp(-\kappa_{2}x)$ (i.e. $k_{x1}=-i\kappa_{1},k_{x2}=i\kappa_{2}$, with
positive $\kappa_{1,2}$); the dependence along the y-coordinate is given by
$\exp(ik_{y}y)$. For simplicity we consider here the symmetric case
$\Delta=0,\epsilon_{v1}-\epsilon_{c1}=\epsilon_{c2}-\epsilon_{v2}=\epsilon_{g}>0$.
Then from the bulk energy dispersion
$\epsilon_{b}^{2}=\epsilon_{g}^{2}/4+(P^{2}/6)(k_{y}^{2}+4k_{x}^{2})$ we
easily find
$\kappa_{1}=\kappa_{2}=\kappa=(1/2)\sqrt{k_{y}^{2}+(6/P^{2})(\epsilon_{g}^{2}/4-\epsilon^{2})}$,
where $\epsilon$ is the energy of the surface state obtained from the boundary
conditions. As the boundary conditions we use the continuity of both
components of the two-component spinors found in regions I and II. For our
symmetric case it gives the following equation:
$-4\kappa\epsilon=k_{y}\epsilon_{g}$. Using the expression for the $\kappa$,
we finally obtain the equation for the surface states energy

$\epsilon\sqrt{k_{y}^{2}+\frac{6}{P^{2}}\big{(}\frac{\epsilon_{g}^{2}}{4}-\epsilon^{2}\big{)}}=-\frac{k_{y}}{2}\epsilon_{g}$
(3)

This equation has the exact solution $\epsilon=-Pk_{y}/\sqrt{6}$. (The second
branch of surface states is obtained by changing the sign of $k_{y}$). Note
that the surface curve $\epsilon=Pk_{y}/\sqrt{6}$ asymptotically approches the
bulk light-hole dispersion curve at large $k_{y}$ and $k_{x}=0$. Thus we see
that the Dirac solution for the surface states is obtained by total neglect of
the interaction with the heavy-hole band.

###  II.2 Full treatment

Let us now find the correct energy spectrum of the surface states for the same
problem with heavy-hole interactions included. As it follows from Hamiltonian
(2), an interaction of the $\Gamma_{6}$ band with the heavy-hole band is given
by the $ik_{y}P/\sqrt{2}$ element, which does not depend on $d/dx$. For that
reason we can exclude the heavy hole component of the wave function and still
work with the two-band model, i.e. electrons and light holes. It is easy to
work out from Eq. (2) that this procedure leads to the system of two equations
in each region: $\hat{H}_{eff}\,\,\chi=0$, $\chi_{tr}=(u,v)$ with the
“effective” Hamiltonian (see also Ref. Volkov1, ):

$\displaystyle\hat{H}_{eff}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}\epsilon_{c1,2}-\epsilon-\frac{P^{2}k_{y}^{2}}{2(\epsilon_{v1,2}-\epsilon-\Delta/2)}&\frac{(2\hat{k}_{x}+ik_{y})P}{\sqrt{6}}\\\
\frac{(2\hat{k}_{x}-ik_{y})P}{\sqrt{6}}&\epsilon_{v1,2}-\epsilon+\Delta/2\end{bmatrix}$
(4)

We assume that strain is zero in region II and $\Delta$ is positive in region
I, i.e. a gap is open between the light and heavy-hole bands (see Fig. 2).
From Eq. (4) we find the equation for the energy dispersions of the bulk
materials:

$\displaystyle[\epsilon_{v1,2}-\epsilon-\frac{\Delta}{2}]\cdot[(\epsilon_{c1,2}-\epsilon)(\epsilon_{v1,2}-\epsilon+\frac{\Delta}{2})$
$\displaystyle-\frac{2}{3}P^{2}(k_{x}^{2}+k_{y}^{2})]=P^{2}k_{y}^{2}\frac{\Delta}{2}$
(5)

We see that in the absence of strain the spectrum of heavy holes with infinite
mass is simple: $\epsilon_{h}=\epsilon_{v1,2}$. Strain mixes the light and
heavy holes, the heavy holes acquire finite mass in the y-direction (within
the plane), and the spectrum becomes complicated. In the x-direction, however,
when $k_{y}=0$, the spectra of light and heavy particles remain simple:

$\displaystyle(\epsilon_{c1,2}-\epsilon(k_{x}))(\epsilon_{v1,2}-\epsilon(k_{x})+\frac{\Delta}{2})$
$\displaystyle=$ $\displaystyle\frac{2}{3}P^{2}k_{x}^{2}$ (6)
$\displaystyle\epsilon_{h1,2}(k_{x})$ $\displaystyle=$
$\displaystyle\epsilon_{v1,2}-\frac{\Delta}{2}$ (7)

Equation (6) gives the bulk spectrum of the light particles—electrons and
light holes, with dispersions $\epsilon_{e}(k_{x})$ and $\epsilon_{l}(k_{x})$,
respectively.

We seek for the surface state that is localized at the interface ($x=0$ ) and
decays in the left-hand region as $\exp(\kappa_{1}x)$ and in the right-hand
region as $\exp(-\kappa_{2}x)$ ( $\kappa_{1,2}>0$). The corresponding
expressions for the $\kappa_{1,2}$ can be found from Eq. (5):

$\displaystyle\kappa_{1}^{2}$ $\displaystyle=$ $\displaystyle
k_{y}^{2}+\frac{3(\epsilon-\epsilon_{c1})(\epsilon_{v1}-\epsilon+\frac{\Delta}{2})}{2P^{2}}+\frac{3\Delta
k_{y}^{2}}{4(\epsilon_{v1}-\epsilon-\frac{\Delta}{2})};$ (8)
$\displaystyle\kappa_{2}^{2}$ $\displaystyle=$ $\displaystyle
k_{y}^{2}+\frac{3(\epsilon_{c2}-\epsilon)(\epsilon-\epsilon_{v2})}{2P^{2}}.$
(9)

The spinors in both regions (I) and (II) can be easily found with the help of
Eq. (4); then from the continuity conditions of both components of the wave
functions at $x=0$ we find:

$(\epsilon_{v1}-\epsilon+\frac{\Delta}{2})(2\kappa_{2}-k_{y})=(\epsilon-\epsilon_{v2})(2\kappa_{1}+k_{y}).$
(10)

The surface states spectra are determined by the solution of the system of
equations (8,9,10).

Figure 3: (a) An interface with an antisymmetric variation of the bulk band
structure. For simplicity, the strain is omitted here and the heavy-hole mass
is set to infinity. (b) The surface states arising from the band alignment in
(a), showing two types of states near the center of the Brillouin zone: linear
Dirac states and quadratic Dyakonov-Khaetskii states (DK states). The
projected bulk spectrum is shown by the shaded gray regions. All surface
states are singly-degenerate with a distinct spinor structure originating from
one of the two time-reversal-related blocks of the Hamiltonian as shown by the
spin-up and spin-down arrows. The asymptote of the Dirac states at small in-
plane momenta (dashed line $a$) differs strongly from the asymptote of the DK
states at large momenta (dot-and-dashed line $b$), refuting the picture of a
narrow-in-energy interruption of the Dirac states by the heavy-hole band.

####  II.2.1 Symmetric case, $\Delta=0$,
$|\epsilon_{g1}|=\epsilon_{g2}=\epsilon_{g}$

The discussion of the solution for the surface states within the full
treatment starts with the symmetric case (Fig. 3(a)) $\Delta=0$,
$\epsilon_{v1}=-\epsilon_{c1}=\epsilon_{c2}=-\epsilon_{v2}=\epsilon_{g}/2>0$
so we may compare it with the results obtained in section II.1. In this case
we have
$\kappa_{1}=\kappa_{2}=\kappa=\sqrt{k_{y}^{2}+(3/2P^{2})(\epsilon_{g}^{2}/4-\epsilon^{2})}$,
and from Eq. (10) one has the same equation as before,
$-4\kappa\epsilon=k_{y}\epsilon_{g}$, therefore finally we obtain

$\epsilon\sqrt{4k_{y}^{2}+\frac{6}{P^{2}}\big{(}\frac{\epsilon_{g}^{2}}{4}-\epsilon^{2}\big{)}}=-\frac{k_{y}}{2}\epsilon_{g}.$
(11)

(The time-reversed solutions can be obtained by changing the sign of $k_{y}$
in Eq. (11)). Despite the fact that Eq. (3) and Eq. (11) are nearly the same
(except for the coefficient of the first term under the radical), they have
completely different solutions, see Fig. 3(b). For $|k_{y}|\ll\epsilon_{g}/P$
Eq. (11) has the same behavior: $\epsilon=-Pk_{y}/\sqrt{6}$; however, at large
$|k_{y}|\gg\epsilon_{g}/P$ the surface branch saturates at $+\epsilon_{g}/4$
(at negative $k_{y}$) and at $-\epsilon_{g}/4$ (at positive $k_{y}$). Similar
behavior was found in Refs. Cade, ; Suris, ; Volkov1, . This behavior is due
to an interaction with the heavy-hole band which has the character of
repulsion.

Besides the surface states that lie within the fundamental gap, Eq. (11) has
the solutions which start from the heavy-hole bands
$(\epsilon=\epsilon_{g}/2,\epsilon=-\epsilon_{g}/2)$, see Fig. 3(b). For the
first of those additional solutions, for example, we obtain at small
$|k_{y}|P\ll\epsilon_{g}$

$\epsilon_{s1}=\frac{\epsilon_{g}}{2}+\frac{P^{2}k_{y}^{2}}{2\epsilon_{g}}.$
(12)

This solution follows from Eq. (11) at $k_{y}<0$. (For $k_{y}>0$ the solution
with this energy does not exist since the left-hand side of Eq. (11) is
strictly positive). The same energy dispersion for $k_{y}>0$ follows from the
time-reversed Hamiltonian and corresponds to the opposite spin direction. It
means that there is a momentum-spin locking for these solutions, and back
scattering is strictly forbidden. We will discuss this issue in more detail
below. The effective mass of this surface state, Eq. (12), is by the factor of
$4/3$ larger than the effective mass of the light hole $m_{l}$. (The latter is
found from the bulk dispersion equation for the light holes:
$\epsilon_{l}(k_{y})=\sqrt{\epsilon_{g}^{2}/4+(2/3)P^{2}k_{y}^{2}}$ ). This
new surface branch was predicted theoretically for HgTe for the first time in
Ref. Dyak, . The effective mass of this surface state depends on the ratio
$\beta$ of the bulk effective masses of light and heavy holes and approaches
the value $(4/3)m_{l}$ (see Ref. Dyak, ) in the limit $\beta\to 0$ considered
here. As it follows from Eq. (11) (and from the time-reversed one), at large
$|k_{y}|\gg\epsilon_{g}/P$ the surface branch in question asymptotically
approaches the bulk light-hole band:
$\epsilon_{s1}\to\sqrt{2/3}P|k_{y}|+(3\sqrt{3}/32\sqrt{2})(\epsilon_{g}^{2}/P|k_{y}|)$.

####  II.2.2 Evolution of surface states with the strength of interaction
between $\Gamma_{6}$ and heavy-hole band

To better understand the origin of different branches of surface states
presented in Fig. 3(b), it is instructive to follow the evolution of these
states as a function of the strength of the interaction of the $\Gamma_{6}$
band with the heavy-hole band. To this end we multiply the corresponding
matrix elements in Eq. (2) by the factor $\alpha$, where $\alpha$ can take the
values in the interval between 0 and 1, obtaining the Hamiltonian

$\displaystyle\hat{H}_{\alpha}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}\epsilon_{c1,2}&\frac{(2\hat{k}_{x}+ik_{y})P}{\sqrt{6}}&i\alpha
Pk_{y}/\sqrt{2}\\\
\frac{(2\hat{k}_{x}-ik_{y})P}{\sqrt{6}}&\epsilon_{v1,2}+\Delta/2&0\\\ -i\alpha
Pk_{y}/\sqrt{2}&0&\epsilon_{v1,2}-\Delta/2\end{bmatrix}$ (13)

Then for the symmetric case we obtain the following equation which describes
the surface energy spectrum

$\epsilon\sqrt{(1+3\alpha^{2})k_{y}^{2}+\frac{6}{P^{2}}\big{(}\frac{\epsilon_{g}^{2}}{4}-\epsilon^{2}\big{)}}=-\frac{k_{y}}{2}\epsilon_{g}.$
(14)

This equation reproduces Eq. (3) and Eq. (11) in the limits $\alpha\to 0$ and
$\alpha\to 1$, correspondingly. It is useful to take the square of this
equation to obtain

$(\frac{\epsilon_{g}^{2}}{4}-\epsilon^{2})(k_{y}^{2}-\frac{6\epsilon^{2}}{P^{2}})=3\alpha^{2}k_{y}^{2}\epsilon^{2}$
(15)

Let us investigate the solutions in the $\alpha\to 0$ limit. There are two of
them with the anticrossing point $\tilde{k}_{y}=\sqrt{3/2}(\epsilon_{g}/P)$,
see Fig. 4(b). The first solution has the dispersion
$\epsilon=\epsilon_{g}/2+(\alpha^{2}P^{2}k_{y}^{2}/2\epsilon_{g})$ at small
$k_{y}$. (Note that this dispersion coincides with the one given by Eq. (12))
at $\alpha=1$). At large $k_{y}$ this branch is close to the Dirac solution
$Pk_{y}/\sqrt{6}$. The second solution coincides with the Dirac one at small
$k_{y}$ and is close to the heavy-hole band ($\epsilon_{g}/2$) at large
$k_{y}$. These two branches repel each other with increasing $\alpha$ and
eventually approach the configuration shown in Fig. 3(b) (red curves). For
example, the asymptote of the DK state at large $k_{y}$ is
$Pk_{y}\sqrt{(1+3\alpha^{2})/6}|_{\alpha=1}=Pk_{y}\sqrt{2/3}$ (asymptote b in
Fig. 3(b)).

Thus we see that the surface states presented in Fig. 3 (see also Fig. 5
below) are intimately related to the Dirac states. In particular, DK surface
state Dyak  predicted for the first time within the Luttinger model is the
result of strong hybridization of the Dirac state with the heavy-hole states
(see also Ref. (Krishtopenko1, )). The authors of Ref. Brune,  claimed the
same, though they referred to the states only as “special” rather than
identify them as the DK states previously established in Ref. Dyak, .

Figure 4: Dirac cone influenced by a weak coupling to the heavy-hole band. (a)
The widening of the light-hole (LH) and electron (e) branches in the in-plane
momentum direction upon weakening of the coupling to the heavy-hole branch.
The coupling to the heavy-hole branch is full for $\alpha=1$ (solid lines),
corresponding to a pristine semiconductor, and absent for $\alpha=0$ (dashed
lines), when dispensing with the heavy-hole branch. The asymptotes $a$ and $b$
delimit the bands for $\alpha=0$ and $1$, respectively. (b) The surface states
obtained in the limit $\alpha\ll 1$, showing an avoided crossing of the Dirac
cone with the heavy-hole bands. Here, we consider the “symmetric case” of Fig.
3 for a small $\alpha$ introduced alike for the inverted-band-gap and regular
semiconductors.

####  II.2.3 High barrier between inverted and direct materials

The surface states spectra depend strongly on the height of the barrier
between inverted and direct materials. If the fundamental gap in the direct
material $\epsilon_{g2}$ is much larger than the modulus of the gap in the
inverted one, then Eq. (10) (and time-reversed one) read

$\frac{\epsilon_{g1}}{2}-\epsilon-\sqrt{(\frac{\epsilon_{g1}}{2})^{2}-\epsilon^{2}+\frac{2}{3}P^{2}k_{y}^{2}}=\pm\frac{Pk_{y}}{\sqrt{6}}.$
(16)

As before, we first consider here $\Delta=0$ case and symmetric energy
diagram:
$\epsilon_{v1}=-\epsilon_{c1}=\epsilon_{g1}/2>0;\epsilon_{c2}=-\epsilon_{v2}=\epsilon_{g2}/2\to\infty$.
Equation (16) can be easily solved analytically and surface curves are
presented in Fig. 5. We see that the surface branches which pass through the
center of the gap (Dirac point) do not saturate at large $|k_{y}|$ but go
downward. It happens because the heavy-hole band in the barrier region is
located deeply below in energy which strongly suppresses repulsion from it.
These two surface branches have the following exact dispersions

$\displaystyle\epsilon_{\downarrow}=\frac{\epsilon_{g1}}{4}+\frac{Pk_{y}}{2\sqrt{6}}-\sqrt{(\frac{\epsilon_{g1}}{4})^{2}-\frac{Pk_{y}\epsilon_{g1}}{4\sqrt{6}}+\frac{7}{24}P^{2}k_{y}^{2}}$
$\displaystyle\epsilon_{\uparrow}=\frac{\epsilon_{g1}}{4}-\frac{Pk_{y}}{2\sqrt{6}}-\sqrt{(\frac{\epsilon_{g1}}{4})^{2}+\frac{Pk_{y}\epsilon_{g1}}{4\sqrt{6}}+\frac{7}{24}P^{2}k_{y}^{2}}$
(17)

As for the DK surface branch, it has the quadratic dispersion for small
$k_{y}P\ll\epsilon_{g1}$ with the same effective mass as it was for the case
of equal gaps of direct and inverted materials, see Eq. (12). The behavior of
this branch at larger $k_{y}$ can be traced with the exact solution of Eq.
(16) (for positive $k_{y}$ we take the minus sign in the right-hand side of
this equation). This solution reads

$\epsilon_{s1}=\frac{\epsilon_{g1}}{4}+\frac{Pk_{y}}{2\sqrt{6}}+\sqrt{\left(\frac{\epsilon_{g1}}{4}\right)^{2}-\frac{Pk_{y}\epsilon_{g1}}{4\sqrt{6}}+\frac{7}{24}P^{2}k_{y}^{2}};\,\,k_{y}\geq
0.$ (18)

It can be easily checked from Eq. (16) that this solution is only valid at
$Pk_{y}\leq\sqrt{2/3}\epsilon_{g1}$. At this critical $k_{y}$ the DK surface
state merges with the light-hole band at the energy value which is equal to
$(5/6)\epsilon_{g1}$, see Fig. 5(b).

Figure 5: Effect of a high barrier on the surface states. (a) An interface
between an inverted-band-gap SC ($E_{g}<0$) and a regular SC with a large
fundamental gap ($E_{g}\to+\infty$). (b) Similar to Fig. 3, two types of
surface states occur in the limit of small in-plane momenta, namely the linear
Dirac states and the quadratic DK states. At large momenta, the lower cone of
the Dirac states becomes sharper than the original low-momentum asymptote $a$,
whereas the upper cone reverses slope and becomes blunter. The DK states
terminate at a critical value of momentum $k_{y}\sim\epsilon_{g}/P$ (see solid
circles), at which the localization length diverges ($\kappa_{1}\to 0$) and
the surface state _flows_ tangentially into the bulk continuum.

####  II.2.4 Finite effective mass of the heavy-hole band

So far we have been considering the case when an effective mass of the heavy-
hole (HH) band is infinite. It is important to realize that in this limit
there is a true gap (i.e. density of states is zero) between the valence
$\Gamma_{8}$ bands and conduction $\Gamma_{6}$ band, Figs. (3,5). When
effective mass of heavy holes is finite, their energy starts to depend on the
$k_{x}$-wave vector component perpendicular to the interface. It means that
the surface states located within the $\Gamma_{8}-\Gamma_{6}$ gap lie on top
of the continuum of the heavy-hole states corresponding to different $k_{x}$
and strong hybridization between them and surface states will occur. As a
result, the electrons occupying the surface states located within the
$\Gamma_{8}-\Gamma_{6}$ gap will not show in transport measurements the
properties exclusive for topological surface states not hybridized with any
bulk states. Thus, they cannot be distingushed as a separate group of
carriers. (We should mention, however, that at small $k_{y}$, at which the
overlap of these states with heavy holes is weak, they can be detected with
the ARPES experiments, see Refs. Rogalev, ; Rauch1, ). On the other hand, the
surface states lying in the projected gap between the light-hole and heavy-
hole bands found in Ref. Dyak,  have true physical meaning. (The similar ideas
about the hybridization and its effect on the surface states were developed in
Ref. Brune, ). Now we will study how these states are transformed in the
presence of the strain which opens a true gap between the light-hole and
heavy-hole bands.

###  II.3 Strain applied to the inverted region

We now analyze the same surface-states problem with strain applied to the
inverted region, thereby opening the gap $\Delta$ between light-hole and
heavy-hole bands, see Fig. 2. We consider the case $\Delta\ll\epsilon_{g1}$,
which is of practical importance, and will assume first the symmetric case,
$\epsilon_{v1}=-\epsilon_{c1}=\epsilon_{c2}=-\epsilon_{v2}=\epsilon_{g}/2>0$.
Clearly, the physical picture of the states lying within a
$\Gamma_{8}-\Gamma_{6}$ gap cannot change appreciably with application of
strain, see Figs. (3,5). Indeed, the perturbation theory with respect to the
parameter $\Delta/\epsilon_{g}\ll 1$ shows that the dispersion curves within
the $\Gamma_{8}-\Gamma_{6}$ gap shift only slightly (for example, the Dirac
point shifts from the middle of the gap by the value of the order of
$\Delta$). The only strong change of the spectrum occurs in the energy
interval of the order of $\Delta$ near the top of the valence band of HgTe.
Strain opens a gap; therefore, when the Fermi level is located within this
gap, there is no conductivity (at low temperature) in the bulk and no elastic
scattering between surface and bulk states. We stress, however, that the
topological character of this surface state, Eq. (12), exists even in the
absence of the strain (as we have already mentioned above). In particular, all
the characteristic features of the topological state, such as a spin-momentum
locking and absence of the back scattering, exist already for the solution,
Eq. (12), obtained without strain.

Let us find the energy spectrum of the DK surface state in the presence of
strain. From the system of equations (8,9,10) it is easy to find that at
$k_{y}\to 0$ the solution has the form $\epsilon=\epsilon_{v1}-\Delta/2+\delta
P^{2}k_{y}^{2}$ with
$\kappa_{1}^{2}=(3\Delta/2P^{2})(\epsilon_{g}-\Delta/2-1/(2\delta))$ and
$\kappa_{2}^{2}=(3\Delta/4P^{2})(\epsilon_{g}-\Delta/2)$. Then from Eq. (10)
we can calculate $\delta$ and, finally, the surface state spectrum at
$k_{y}\to 0$ reads

$\epsilon=\epsilon_{v1}-\frac{\Delta}{2}+\frac{P^{2}k_{y}^{2}}{2\epsilon_{g}-\Delta-\frac{2\Delta^{2}}{(2\epsilon_{g}-\Delta)}}$
(19)

We note that Eq. (19) is valid at arbitrary values of strain gap $\Delta$, not
necessarily much smaller than $\epsilon_{g}$.

For the case of a very large gap of the direct material
($\epsilon_{g2}\to\infty$), we obtain a similar equation for the surface state
dispersion at $k_{y}\to 0$

$\epsilon=\epsilon_{v1}-\frac{\Delta}{2}+\frac{P^{2}k_{y}^{2}}{2\epsilon_{g1}-3\Delta}$
(20)

At large $k_{y}$ when the energy of the surface branch becomes much larger
than $\Delta$, the surface curve coincides with the one obtained without any
strain.

Thus, the surface branch starts at the top of the heavy-hole band and at large
$k_{y}$ approaches the light-hole band, see Fig. 6(a) and similar picture in
the Supplementary Material of Ref. Brune, .

Figure 6: Effect of a tensile in-plane strain on the surface states studied in
two models. (a) Kane model with heavy-hole flat band: The strain splits and
intermixes the bulk light-hole and heavy hole bands, opening a global gap of
magnitude $\Delta/4$ and inducing a dispersion in the heavy-hole band. The DK
states depart from the heavy-hole band at zero momentum and merge with the
light-hole band at large momenta. (b) Luttinger model (finite heavy-hole mass
but no conduction bands): Qualitatively, the same result as in (a), except
that the DK states depart from the heavy-hole band at finite momenta.

The results of the k-P -model agree very well with the results obtained from
more sophisticated methods, like the ab-initio calculations, for Hg-terminated
HgTe (001) surface. For Te-terminated surface it is not necessarily the case,
see Refs. (Rauch, ; Wu, ).

##  III Surface states within Luttinger model with strain

As we have demonstrated, the surface states that lie within the gap opened by
strain between light and heavy-hole bands are of special importance. These
states can be studied in more detail within the Luttinger model which
describes the valence $\Gamma_{8}$ light and heavy-hole bands. This model
allows exact analytical results for the spectrum and spinors of the surface
states for an arbitrary ratio between the effective masses of light and heavy
holes, and represents a more accurate description compared to the model we
used above. In Ref. Dyak,  the solution for those states was found in the
absence of strain.

The Luttinger Hamiltonian for HgTe which describes the group of states
$|lh-1/2,hh+3/2>$ has the form

$\displaystyle\hat{H}_{L}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}k^{2}(\gamma-\tilde{\gamma})-\Delta/4&\,\,\,\,\,\sqrt{3}\tilde{\gamma}k_{+}^{2}+\sqrt{3}\Delta/4\\\
\sqrt{3}\tilde{\gamma}k_{-}^{2}+\sqrt{3}\Delta/4&\,\,\,\,\,k^{2}(\gamma+\tilde{\gamma})+\Delta/4\end{bmatrix}$
(21)

The energy is counted from the degeneracy point of light-hole and heavy-hole
bands in the HgTe region in the absence of strain ($\epsilon_{v1}=0$),
$\Delta>0$ is the gap value due to strain, $\hat{k}_{\pm}=\hat{k}_{x}\pm
ik_{y}$, $\hat{k}_{x}=-id/dx$, $\hat{k}^{2}=k_{y}^{2}+\hat{k}_{x}^{2}$, and
wave vector within the boundary is again directed along the y-axis,
$\gamma,\tilde{\gamma}$ are the Luttinger constants which determine the bulk
spectra of light and heavy holes in the absence of strain:
$\epsilon_{l,h}=k^{2}(\gamma\pm 2\tilde{\gamma})$. We assume that
$\gamma,\tilde{\gamma}>0$ and
$(\gamma-2\tilde{\gamma})<0,(\gamma+2\tilde{\gamma})>0$ (gapless material). As
before, the Hamiltonian describing the time-reversed solutions is obtained
from Eq. (21) by replacing $k_{y}$ by $-k_{y}$.

We consider the HgTe sample which occupies the $x>0$ space and $x=0$ is the
boundary with the vacuum or wide gap direct material (infinite barrier). We
then seek for the surface states located within the gap created by strain. The
x-components of the wave vectors corresponding to the light and heavy holes
for a given energy $\epsilon$ of the surface state are given by
$k_{l,x}=i\kappa_{l}$, $k_{h,x}=i\kappa_{h}$, $\kappa_{l,h}>0$, and the state
exponentially decays towards the bulk of HgTe region. The expressions for
$\kappa_{l,h}$ are

$k_{y}^{2}-\kappa^{2}_{l,h}=\frac{\gamma\epsilon+\tilde{\gamma}\Delta\mp\sqrt{(\gamma\Delta/2+2\tilde{\gamma}\epsilon)^{2}+3\tilde{\gamma}\Delta
k_{y}^{2}(4\tilde{\gamma}^{2}-\gamma^{2})}}{(\gamma^{2}-4\tilde{\gamma}^{2})}$
(22)

Here $-$ corresponds to $\kappa_{l}$ and $+$ to $\kappa_{h}$. The
corresponding two-component spinor wave functions look like

$\displaystyle\Psi_{l,h}(x)$ $\displaystyle\propto$ $\displaystyle
e^{ik_{y}y}e^{-\kappa_{l,h}x}\left(\begin{array}[]{cc}\sqrt{3}\Delta/4-\sqrt{3}\tilde{\gamma}(k_{y}+\kappa_{l,h})^{2}&\\\
\epsilon+\Delta/4-(\gamma-\tilde{\gamma})(k_{y}^{2}-\kappa_{l,h}^{2})\end{array}\right)$
(25) (26)

###  III.1 Boundary condition and solution

The wave function of the surface state is the superposition of the functions
Eq. (26): $\Psi_{s}(x)=a\Psi_{l}(x)+b\Psi_{h}(x)$. For an infinite barrier we
use zero boundary conditions at $x=0$: $\Psi_{s}(x=0)=0$, obtaining the
following equation for the energy $\epsilon$ of the surface state:

$\displaystyle[\epsilon+\Delta/4-(\gamma-\tilde{\gamma})(k_{y}^{2}-\kappa_{h}^{2})][\Delta/4-\tilde{\gamma}(k_{y}+\kappa_{l})^{2}]$
$\displaystyle=[\epsilon+\Delta/4-(\gamma-\tilde{\gamma})(k_{y}^{2}-\kappa_{l}^{2})][\Delta/4-\tilde{\gamma}(k_{y}+\kappa_{h})^{2}]$
(27)

Positive real values of $\kappa_{l,h}$ found from Eq. (22) should be used
while solving Eq. (27).

The exact solution for the surface state dispersion reads:

$\epsilon=-\left(\frac{\gamma}{2\tilde{\gamma}}\right)\frac{\Delta}{2}+k_{y}^{2}(\gamma+2\tilde{\gamma})\left[1-\frac{(1+\sqrt{3\beta})^{2}}{4}\right],$
(28)

where $\beta=(2\tilde{\gamma}-\gamma)/(\gamma+2\tilde{\gamma})>0$ is the ratio
of the bulk effective masses of the light and heavy holes. The solution Eq.
(28) (we call it further as DK 1 branch) coincides at $\Delta=0$ with the
solution obtained in Ref. Dyak, . It is interesting that the spectrum in the
presence of the strain differs from the one at $\Delta=0$ only by the constant
shift proportional to the strain value. Using this dispersion we can find from
Eq. (22) the values of $\kappa_{l,h}$:

$\kappa_{l,h}=\frac{(1+\beta)\sqrt{3}}{4\sqrt{\beta}}k_{y}\mp\frac{k_{y}}{\sqrt{3\beta}}\sqrt{\left[1-\frac{(1+\sqrt{3\beta})^{2}}{4}\right]^{2}+\frac{3\beta\Delta}{4\tilde{\gamma}k_{y}^{2}}}$
(29)

This solution is valid at $k_{y}>0$. Moreover, from the condition that
$\kappa_{l}>0$ we get the critical value of $k_{y}^{\star}$, thus the surface
state actually exists at $|k_{y}|>k_{y}^{\star}$, see Fig. 6(b).

$k_{y}^{\star}=\sqrt{\frac{\Delta}{\tilde{\gamma}[2+\sqrt{\frac{3}{\beta}}(1-\beta)]}}$
(30)

It is easy to see that at this critical $k_{y}$ the surface state curve merges
with the bulk heavy-hole band (corresponding to $k_{x}=0$). In accordance with
the results obtained above, in the limit $\beta=0$ (infinite mass of the heavy
holes) the critical value $k_{y}^{\star}\to 0$. Moreover, since
$k_{y}^{\star}$ must be real, we obtain the following condition: $\beta<3$. It
coincides with the one obtained in Ref. Dyak,  for the case $\Delta=0$. As we
know from Ref. Dyak, , in the case $\Delta=0$ there are two solutions for the
surface states. This is true also in the presence of the strain. For the case
of $\Delta\geq 0$ the second solution (DK 2 branch) exists only at
$\beta>1/3$. Since this case does not correspond to real HgTe or $\alpha$-Sn
materials, we do not consider it here. However, in the case of $\Delta<0$,
which corresponds to the compressive in-plane strain, the situation is
qualitatively different. This latter Dirac semimetal case is considered in
section IV.

###  III.2 Spin structure of the surface state

Let us describe the spin structure of the solution obtained in this section.
We have mentioned already that even without strain the DK surface state Dyak
has all the features of a topological surface state for a 3D topological
insulator: momentum-spin locking and absence of back scattering. Indeed, from
Eqs. (26,28) at $\Delta=0$ we find the following wave function at
$k_{z}=0,k_{y}>0$

$\displaystyle\Psi_{\downarrow}(x)=(e^{-\kappa_{l}x}-e^{-\kappa_{h}x})\cdot\frac{1}{2\sqrt{1+\beta}}\left(\begin{array}[]{cc}\sqrt{3}+\sqrt{\beta}&\\\
\sqrt{3\beta}-1\end{array}\right)$ (33) (34)

Let us calculate the average value of the different components of spin using
this function. While doing so we should remember that the upper coefficient of
the spinor Eq. (34) corresponds to $-1/2$ projection over the z-axis and the
lower coefficient to the $+3/2$ projection. Coefficients corresponding to
$+1/2,-3/2$ projections are equal to zero. Then using $4\times 4$ matrices
$\hat{J}_{x},\hat{J}_{y},\hat{J}_{z}$ corresponding to the angular momentum
3/2, we obtain $<J_{x}>=0,<J_{y}>=0$, and

$\displaystyle<J_{z}>$ $\displaystyle=$
$\displaystyle\frac{1}{4(1+\beta)}\left[\frac{3}{2}(\sqrt{3\beta}-1)^{2}-\frac{1}{2}(\sqrt{3}+\sqrt{\beta})^{2}\right]$
(35) $\displaystyle=$ $\displaystyle\frac{\beta-\sqrt{3\beta}}{(1+\beta)}$

Thus, the average spin is parallel to the boundary and directed perpendicular
to the electron momentum. For the opposite momentum direction within the plane
one obtains the average spin which is the opposite sign of the value given by
Eq. (35); therefore, one has the momentum-spin locking for this surface state.

Finally, we present here the result obtained within the Luttinger model for
the average spin of the surface electron in the presence of strain,
$\Delta>0$. At the conditions $\sqrt{\beta}\ll 1$ and
$\Delta/(\tilde{\gamma}k_{y}^{2})<\sqrt{3/\beta}$, we obtain

$<J_{z}>=-\sqrt{3\beta}\frac{1}{1-\Delta/(9\tilde{\gamma}k_{y}^{2})}$ (36)

As before, $<J_{x}>=0,<J_{y}>=0$, i.e. spin is parallel to the boundary and
perpendicular to the momentum direction. (At $\Delta=0$ and $\beta\ll 1$ the
result Eq. (36) coincides with the one given by Eq. (35)). It is interesting
that in contrast to the case without strain, the average spin now increases
with the lowering of energy, getting near the $k_{y}^{\star}$, Eq. (30), the
maximum value which is of order of unity even for $\beta\ll 1$.

We consider the case of isotropic Luttinger model; therefore, the spin-
momentum locking is an exact characteristic of the surface states, that can be
probed experimentally. By adding higher order terms one can create some
warping. (In practice, it can be done by growing film of topological material
on the substrate material with the proper symmetry). In the particular case of
the ”hexagonal warping”, Ref. Fu, , the component of the spin normal to the
plane appears; however, the component of the spin parallel to the plane still
strongly correlates with the momentum direction. Therefore, in a sense,
complete breaking of the locking does not happen.

##  IV Surface states in the Dirac semimetal regime (Luttinger model)

So far we have considered the case of tensile in-plane strain ($\Delta>0$),
which opens the gap between the light and heavy-hole bands, creating the
topological insulator in the material like HgTe. In the opposite case,
$\Delta<0$, one has the situation of the Dirac semimetal where the light-hole
band and the heavy-hole band overlap, forming two Dirac points. The bulk
energy dispersion near those points is a linear function of the in-plane wave
vector, see Fig. (7). The case of the Dirac semimetal has received much less
attention in the literature, especially its analytical treatment. We consider
here the problem of surface states in the Dirac semimetal regime within the
Luttinger model. Similar to the $\Delta>0$ case, it is possible to find an
exact analytical solution for the spectrum and wave functions of the surface
states within the Luttinger model (i.e. for energies smaller than the
$\Gamma_{8}-\Gamma_{6}$ gap). As far as we know, it is the first time when a
comprehensive analytical solution for this problem is found. Note, that we do
not study here the surface states located within the $\Gamma_{8}-\Gamma_{6}$
gap. It is clear, however, that similar to the previous cases considered in
this paper, those states lie on top of the continuum of the heavy-hole states
and are strongly hybridized with them. Therefore, there are no linear in
k-vector surface states going from s-band to the light-hole band across the
$\Gamma_{8}-\Gamma_{6}$ gap, i.e. the popular picture is again incorrect.

We start with the bulk spectrum near the Dirac point at $\Delta<0$. Again the
$x$-axis is perpendicular to the boundary, and the wave vector along the
boundary is equal to $k_{y}$. From the expression for the dispersions of the
light and heavy holes

$\epsilon_{l,h}=\gamma(k_{x}^{2}+k_{y}^{2})\pm
2\tilde{\gamma}\sqrt{\left(\frac{\Delta}{4\tilde{\gamma}}+k_{x}^{2}\right)^{2}+k_{y}^{2}\left(k_{y}^{2}+2k_{x}^{2}-\frac{\Delta}{4\tilde{\gamma}}\right)}$
(37)

one can easily see that at $k_{x}^{2}=|\Delta|/4\tilde{\gamma}$ (Dirac point)
the bulk spectrum is linear

$\epsilon_{b}(k_{y})=\frac{\gamma|\Delta|}{4\tilde{\gamma}}\pm
k_{y}\sqrt{3\tilde{\gamma}|\Delta|};\,\,\,|k_{y}|\ll\sqrt{|\Delta|/\tilde{\gamma}}.$
(38)

We consider now the surface states. We will show that the two branches of
parabolic surface states start from the bulk Dirac point. It is easy to
realize that we can use the same Eq. (27) for the dispersion of the surface
states. The only difference now is that the strain is negative: $\Delta<0$.
Therefore, the solutions Eqs. (28,29) are still valid. We see that the first
(DK 1) branch, Eq. (28), starts at the energy of the bulk Dirac point:
$\gamma|\Delta|/4\tilde{\gamma}$, see Fig. (7), and has the same mass in the
absence of strain. The expressions for the $\kappa_{l,h}$ are still given by
Eq. (29), the only difference from the case $\Delta>0$ is that now quantities
$\kappa_{l,h}$ become complex at small enough values of $k_{y}$. Consider the
positive $k_{y}$, the character of the surface wave function is different
depending on whether $k_{y}$ is smaller or bigger than $k_{y1}$, where

$k_{y1}=2\sqrt{\frac{\beta|\Delta|}{\tilde{\gamma}}}\cdot\frac{1}{|1-\sqrt{3\beta}|(\sqrt{3}+\sqrt{\beta})}.$
(39)

If $k_{y}<k_{y1}$, then $\kappa_{l}=\kappa_{h}^{\star}=a-ib$, and the surface
function has the form $\Psi_{s}(x)\propto\sin(bx)\cdot\exp(-ax)$ with $a,b$ to
be found from Eq. (29). In the case $k_{y}>k_{y1}$ both $\kappa_{l,h}$ are
real and positive, and the surface wave function has the form
$\Psi_{s}(x)\propto(e^{-\kappa_{l}x}-e^{-\kappa_{h}x})$, with $\kappa_{l,h}$
given by Eq. (29) at $\beta<3$.

The interesting feature of the problem is that there is a great difference in
the number of DK surface branches which exist at positive or negative
$\Delta$. In the case of $\Delta>0$, at $\beta<1/3$ (which is the case of HgTe
and $\alpha$-Sn materials) there is only one branch DK 1 of surface states. We
will show now that in the case of $\Delta<0$, considered here, there are
always two DK surface branches (even for $\beta<1/3$). Thus, in the case of
Dirac semimetal, one always has two DK surface branches, which for the
unstrained sample were found in the original work Ref. (Dyak, ). We should
mention that the authors of Ref. (Kibis, ) found only one DK branch in the
regime of a Dirac semimetal. Moreover, these two branches continue until the
$k_{y}=0$ point, in strong contrast to the conclusion made in Ref. (Kibis, ).

Let us discuss the second surface branch, which exists at $\Delta<0$. It is
the solution of Eq. (27) where the change $k_{y}\rightarrow-k_{y}$ is made,
i.e. for positive $k_{y}$ it is obtained from the second block of basis states
and has an opposite spin compared to the first (DK 1) solution. The exact
solution for the dispersion of the second surface branch (DK 2) reads:

$\epsilon=\left(\frac{\gamma}{2\tilde{\gamma}}\right)\frac{|\Delta|}{2}+k_{y}^{2}(\gamma+2\tilde{\gamma})\left[1-\frac{(1-\sqrt{3\beta})^{2}}{4}\right].$
(40)

The corresponding values of $\kappa_{l,h}$ are:

$\kappa_{l,h}=\frac{(1+\beta)\sqrt{3}}{4\sqrt{\beta}}k_{y}\mp\frac{k_{y}}{\sqrt{3\beta}}\sqrt{\left[1-\frac{(1-\sqrt{3\beta})^{2}}{4}\right]^{2}+\frac{3\beta\Delta}{4\tilde{\gamma}k_{y}^{2}}}$
(41)

This solution is valid at $k_{y}>0$. For small enough $k_{y}$ the quantities
$\kappa_{l,h}$ are complex again, and one can have a localized state even at
$\beta<1/3$, in strong contrast to the cases $\Delta\geq 0$. At large enough
$k_{y}$, however, when both $\kappa_{l,h}$ are real, from the condition that
$\kappa_{l}>0$ we get the critical value $k_{y}=\tilde{k}_{y}$, after which
the second surface branch ceases to exist (it merges with the bulk band), see
Fig. (7). The expression for $\tilde{k}_{y}$ reads

$\tilde{\gamma}\tilde{k}_{y}^{2}=\frac{\sqrt{\beta}|\Delta|}{(1-\sqrt{3\beta})(\sqrt{3}+\sqrt{\beta})};\,\,\,\beta<1/3.$
(42)

Figure 7: Case of compressive in-plane strain ($\Delta<0$). Two parabolic
surface-state branches (DK 1 and DK 2) arise around the conic point of the
bulk projected states located at the energy
$\epsilon_{D}=(\gamma|\Delta|/4\tilde{\gamma})$. The DK 2 branch flows into
the bulk spectrum at the value of in-plane momenta given by Eq. (42). At
$\Delta=0$, the DK 2 branch arises only for $\beta>1/3$, see Ref. Dyak, .

Thus we see that the dispersions of the surface states found by us here are
very much different from the ones usually used, see, for example, Refs. (Ding,
; Ding1, ). The authors of Refs. (Ding, ; Ding1, ) came to the conclusions
that the effects they observe are due to the Dirac semimetal surface states of
$\alpha$-Sn layer rather than due to the bulk states of this material. One of
the effects is the switching of a magnet by spin-orbit torque from a Dirac
semimetal by a charge current at room temperature, without an external
magnetic field. This shows that the topological Dirac semimetal $\alpha$-Sn is
a promising material for the spintronic applications. In that respect, the
results obtained by us are potentially important for the correct
interpretation of the current and future experiments.

##  V Conclusions

We have reconsidered the physical origin of surface states in inverted-band
semiconductors, such as HgTe and $\alpha$-Sn.

By considering analytically two simple and exactly solvable models we have
clarified several important issues concerning the nature of the states, among
them the role of band structure and the role of strain. In particular, we have
shown that neglecting the coupling to the heavy-holes band leads to the
incorrect physical picture of the surface states obtained in many recent
publications. Such a procedure results in the simple picture of linear in
k-vector Dirac surface states crossing the whole gap between the conduction
and light-hole bands, see Fig. 2(b). This latter picture does not reflect the
true nature of the surface states. We have also shown that in a topological
insulator regime an applied strain is a smooth “deformation” to the surface
states that does not lead to any drastic change of the physical properties in
these materials, in contrast to what has recently been published in the
literature. In the Dirac semimetal regime, however, the physics of surface
states which arise around the conic point of the bulk projected states
crucially depends on the applied strain.

We have reconciled different analytical solutions found in the 1980s with the
results obtained recently by many groups using numerical simulations. In
particular, we have demonstrated that the DK surface state Dyak  predicted for
HgTe for the first time within the Luttinger model is the result of strong
hybridization of the Dirac state with the heavy-hole states. We have also
shown that these states Dyak  possess all the essential features of the
topological states in the presence of strain that induces a gap between the
light and heavy-hole bands. As such, they should play the most important role
in the experiments that probe the topological properties of the sample when
the Fermi level is located within this strain-induced gap. In contrast, the
surface states located within the $\Gamma_{8}-\Gamma_{6}$ gap lie on top of
the continuum of the heavy-hole states and strong hybridization with them
makes those surface states marginal in observable phenomena. They still can be
detected with the ARPES experiments at small 2D momenta along the interface
where the overlap of these states with heavy holes is weak, see Refs.
(Rogalev, ; Rauch1, ).

The results obtained by us can be used for the correct interpretation of the
current and future experiments with materials such as $\alpha$-Sn and HgTe.

This work is supported by the Air Force Office of Scientific Research
(FA9550-AFOSR-20RYCOR039). This research was performed while A. Khaetskii held
an NRC Research Associateship award at the Sensors Directorate, Air Force
Research Laboratory. This work was performed in part at Aspen Center for
Physics, which is supported by National Science Foundation grant PHY-1607611.
A. Khaetskii acknowleges the hospitality of the ACP.

## References

  * (1) I. Tamm, Phys. Z. Sowjetunion. 1, 733 (1932).
  * (2) W. Shockley, Phys. Rev. 56, 317 (1939).
  * (3) M. Z. Hasan, C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * (4) X.-L. Qi, S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
  * (5) M.I. Dyakonov, A.V. Khaetskii, JETP Lett. 33, 110 (1981).
  * (6) B. A. Volkov, O. A. Pankratov, JETP Lett. 42, 178 (1985).
  * (7) N. A. Cade, J. Phys. C: Solid State Phys. 18 5135 (1985).
  * (8) R. A. Suris, Sov. Phys. Semicond. 20, 1258 (1986).
  * (9) O.A. Pankratov, S.V. Pakhomov and B.A. Volkov, Solid State Communications 61, 93 (1987).
  * (10) B. A. Bernevig, S.-C. Zhang, Phys. Rev. Lett. 96, 106802 (2006).
  * (11) C. L. Kane, E. J.Mele, Phys. Rev. Lett. 95, 146802 (2005).
  * (12) X. Dai, T.L. Hughes, X.-L, Qi, Z. Fang, and S.-C. Zhang, Phys. Rev. B 77 125319 (2008).
  * (13) Y. Ando, J. Phys. Soc. Japan 82, 102001 (2013).
  * (14) H. Zhang, C.-X. Liu, X.-L. Qi, X. Dai, Z. Fang, S.-C. Zhang, Nat. Phys. 5, 438 (2009).
  * (15) Y. Xia, D. Qian, D.Hsieh, L.Wray, A. Pal, H. Lin, A. Bansil, D. Grauer, Y. S. Hor, R. J. Cava, M. Z. Hasan, Nat. Phys. 5, 398 (2009).
  * (16) A. Bernevig, T. L. Hughes, S.-C. Zhang, Science 314, 1757 (2006).
  * (17) M. König, S. Wiedmann, C. Brüne, A. Roth, H. Buhmann, L. W. Molenkamp, X.-L. Qi, S.-C. Zhang, Science 318, 766 (2007).
  * (18) C. Liu, T. L. Hughes, X.-L. Qi, K. Wang, S.-C. Zhang, Phys. Rev. Lett. 100, 236601 ( 2008).
  * (19) S. S. Krishtopenko, S. Ruffenach, F. Gonzalez-Posada, G. Boissier, M. Marcinkiewicz, M. A. Fadeev, A. M. Kadykov, V. V. Rumyantsev, S. V. Morozov, V. I. Gavrilenko, C. Consejo, W. Desrat, B. Jouault, W. Knap, E. Tournie, F. Teppe, Phys. Rev. B 97, 245419 (2018).
  * (20) D. Hsieh, Y. Xia, D. Qian, L. Wray, J. H. Dil, F. Meier, J. Osterwalder, L. Patthey, J. G. Checkelsky, N. P. Ong, A. V. Fedorov, H. Lin, A. Bansil, D. Grauer, Y. S. Hor, R. J. Cava, M. Z. Hasan, Nature 460, 1101 (2009).
  * (21) C. Brüne, C. X. Liu, E. G. Novik, E. M. Hankiewicz, H. Buhmann, Y. L. Chen, X. L. Qi, Z. X. Shen, S.-C. Zhang, L. W. Molenkamp, Phys.Rev. Lett. 106, 126803 (2011).
  * (22) D. A. Kozlov, Z. D. Kvon, E. B. Olshanetsky, N. N. Mikhailov, S. A. Dvoretsky, D. Weiss, Phys. Rev. Lett. 112, 196801 (2014).
  * (23) J. G. Checkelsky, Y. S. Hor,M.-H. Liu, D.-X. Qu, R. J. Cava, N. P. Ong, Phys. Rev. Lett. 103, 246601 (2009).
  * (24) J. G. Analytis, J.-H. Chu, Y. Chen, F. Corredor, R. D. McDonald, Z. X. Shen, I. R. Fisher, Phys. Rev. B 81, 205407 (2010).
  * (25) Z. Ren, A. A. Taskin, S. Sasaki, K. Segawa, Y. Ando, Phys. Rev. B 84, 165311 (2011).
  * (26) B. Xia, P. Ren, A. Sulaev, P. Liu, S.-Q. Shen, L. Wang, Phys. Rev. B 87, 085442 ( 2013).
  * (27) Y. Pan, D. Wu, J. R. Angevaare, H. Luigjes, E. Frantzeskakis, N. de Jong, E. vanHeumen, T. V. Bay, B. Zwartsenberg, Y. K. Huang, New J. Phys. 16, 123035 (2014).
  * (28) Yo. Ohtsubo, P. Le Fevre, Francois Bertran, and A. Taleb-Ibrahmi, Phys. Rev. Lett. 111, 216401 (2013).
  * (29) A. Barfuss, L. Dudy, M. R. Scholz et al., Phys. Rev. Lett. 111, 157205 (2013).
  * (30) S. Küfner, M. Fitzner, and F. Bechstedt, Phys. Rev. B 90, 125312 (2014).
  * (31) V. A. Rogalev, T. Rauch, M. R. Scholz et al., Phys. Rev. B 95, 161117 (R) (2017).
  * (32) Z. Shia, X. Wanga, C. Xu, P. Wang, Y. Liu, T.-C.Chiang, Physics Letters A 384, 126782 (2020).
  * (33) J. Ding, C. Liu, V. Kalappattil et al., Adv. Mater., 2005909 (2021).
  * (34) J. Ding, C. Liu, Y. Zhang et al., Adv. Funct. Mater., 2008411 (2021).
  * (35) L.G. Gerchikov, and A.V. Subashiev, Phys. Stat. Sol. (b) 160, 443 (1990).
  * (36) G. Bir and G.E. Pikus, Symmetry and Strain-induced Effects in Semiconductors (John-Wiley and Sons, Chichester, 1974).
  * (37) T. Rauch, S. Achilles, J. Henk, and I. Mertig, Phys. Rev. B 96, 035124 (2017).
  * (38) T. Rauch et al., Phys. Rev. Materials 3, 064203 (2019).
  * (39) S.S. Krishtopenko, M. Antezza, and F. Teppe, J. Phys.: Condens. Matter 32, 165501 (2020).
  * (40) S.-C. Wu, B. Yan, and C. Felser, EPL 107, 57006 (2014).
  * (41) L. Fu, Phys. Rev. Lett. 103, 266801 (2009).
  * (42) O.V. Kibis, O. Kyriienko, and I. A. Shelykh, New J. Phys. 21, 043016 (2019).


# multi-modality microscopy image style transfer for nuclei segmentation

###### Abstract

Annotating microscopy images for nuclei segmentation is laborious and time-
consuming. To leverage the few existing annotations, also across multiple
modalities, we propose a novel microscopy-style augmentation technique based
on a generative adversarial network (GAN). Unlike other style transfer
methods, it can not only deal with different cell assay types and lighting
conditions, but also with different imaging modalities, such as bright-field
and fluorescence microscopy. Using disentangled representations for content
and style, we can preserve the structure of the original image while altering
its style during augmentation. We evaluate our data augmentation on the 2018
Data Science Bowl dataset consisting of various cell assays, lighting
conditions, and imaging modalities. With our style augmentation, the
segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei
segmentation algorithms in the competition increases significantly. Thus, our
augmentation technique renders the downstream task more robust to the test
data heterogeneity and helps counteract class imbalance without resampling of
minority classes.

Index Terms—  Style transfer, Data augmentation, Nuclei segmentation

##  1 Introduction

The evaluation of cell-level features is a key task in the histopathological
workflow. Features, such as nuclei shape and distribution, are used to
determine cell, tissue, and cancer types and are therefore relevant for the
clinical diagnosis of cancer, e.g., for cancer identification, grading, and
prognosis [1]. Deep learning serves as a promising tool to quantify these
features in an automated manner [2]. However, the performance of these models
depends heavily on the quality and quantity of training data since they
require accurate and time-consuming segmentation masks. To tackle this, the
Kaggle competition 2018 Data Science Bowl
(DSB)111https://www.kaggle.com/c/data-science-bowl-2018 was initiated to find
segmentation algorithms that are robust and effective in various microscopy
set-ups, cell-lines, and, most significantly, types of light microscopy. Among
3,891 submissions during the challenge, the top-performing methods were mostly
based on the most common segmentation network architectures, Mask R-CNN [3]
and U-Net [4]. The most important factors influencing the competition ranking
were the amount of training data222Some teams also used private training data
aside from the data provided by the challenge organizer, complex pre-
processing and post-processing, and ensemble learning. Yet, a common problem
that encumbers the performance is dataset bias: while segmenting fluorescent
images, which make up most of the training set, is relatively easy and
accurate, segmenting less-represented image types, such as bright-field images
of stained tissue sections, is difficult and inaccurate [5].

In this paper, we propose to facilitate network training by a GAN-based style
transfer data augmentation technique as has been shown to be very effective
for histological images [6]. By synthesizing less-represented image types from
well-represented ones, our style augmentation can increase the amount of
images of minority types in the training set. More specifically, we use
disentangled representations in the style transfer, i.e., we decompose the
input into its content and style representations and only alter the style
without changing the content. Since the structure is preserved, the style
augmented images have the same nuclei locations and shapes as the original
image. Therefore, our augmentation technique enables the segmentation network
to see a multitude of styles already during training, rendering it more robust
to the heterogeneity in the test data.

##  2 Method

Our nuclei segmentation workflow consists of data augmentation, split into
clustering and style transfer, training an instance segmentation network and,
finally, evaluating it with test time augmentation as shown in Figure 1. In
the following, we detail each of the steps.

Fig. 1: Overview of the nuclei segmentation pipeline with multi-modality style
transfer for data augmentation.

###  2.1 Dataset description

The dataset was provided by the competition organizers and is publicly
available in the Broad Bioimage Benchmark Collection with access number
BBBC038333https://bbbc.broadinstitute.org/BBBC038. It is divided into three
sets: a training set of 670 images with nuclei segmentation masks provided
(29,464 nuclei), a first-stage test set of 65 images (4,152 nuclei), and a
second-stage test that includes 3,019 images (37,333 nuclei). The evaluation
of nuclei segmentation for the annotated first-stage test set was disclosed
later by the organizers. However, the official evaluation and ranking of a
segmentation algorithm are based on the performance of the second-stage test
set, which hides the annotated nuclei masks from the participants. Both
training and test images are very diverse in terms of lighting conditions,
cell lines, nuclei densities, and microscopy imaging modalities (with a few
examples shown in Figure 1a and 2). It is worth noting that the test set is
not only substantially greater in size than the training set, but also
includes novel image modalities that have never been seen during training,
rendering the segmentation task extremely challenging.

###  2.2 Data augmentation step I: clustering training images into different
modalities

Since the dataset does not contain any metadata that can be used to determine
the image modalities, we first divide our training images into multiple
domains to perform image style transfer. Each cluster should ideally
correspond to a different modality. As a result, we used the classic K-means
algorithm to divide these images into six distinct clusters based on their hue
saturation value (HSV). Each cluster should ideally reflect a modality that
constitutes a distinct domain. We also observed that the fluorescent images
are typically dark and have low contrast. So we use the Contrast Limited
Adaptive Histogram Equalization (CLAHE)444https://scikit-
image.org/docs/dev/auto_examples/color_exposure/plot_equalize.html for
contrast enhancement. For each section of an image, CLAHE computes a contrast
histogram and adjusts the local contrast accordingly if that section is darker
or brighter than the rest of the image.

Fig. 2: (a) PCA decomposition of hue saturation values of all training images
and their class distribution. Each point represents one image, colored
according to the corresponding cluster. (b) Exemplary images for each cluster.

###  2.3 Data augmentation step II: multi-modality style transfer

As shown in Figure 1e, for two exemplary images from different domains, the
modality transfer model based on [7] first decomposes each image into a
domain-invariant content using a content encoder $E_{c}$, and a domain-
specific attribute using an attribute encoder $E_{a}$, respectively. The
content encoding and the attribute encoding, together with the domain
information, can then be used to generate a synthetic image, where the
structure is preserved. To create various synthetic images as augmentation
during training of the segmentation network, we sample attribute and domain
vector randomly, while leaving the content-encoding fixed (see Figure 1f). We
apply this augmentation technique randomly to half of the training images
additionally to standard augmentations.

The content space is assumed to be shared across the domains. Therefore,
during training, the content vector $z_{c}$ is exchanged between the
modalities. To compute the cross-cycle consistency loss, this is repeated to
reconstruct the original image since we trained without paired images. The
final objective function is given by

$\displaystyle L_{\text{total}}$ $\displaystyle=$ $\displaystyle
w_{cc}L_{cc}+w_{c}L_{c}+w_{d}L_{d}+w_{\text{recon}}L_{\text{recon}}$
$\displaystyle+w_{\text{latent}}L_{\text{latent}}+w_{KL}L_{KL},$

where $L_{cc}$ is the cycle-consistency loss, $L_{c}$ and $L_{d}$ are
adversarial losses for the content and the attribute encoder,
$L_{\text{recon}}$ is an $L_{1}$-loss for image reconstruction (synthesizing
an image using its own content and attribute), $L_{\text{latent}}$ is an
$L_{1}$-loss for latent space reconstruction, and $L_{KL}$ enforces the latent
attribute space to be distributed according to the standard normal
distribution. Please refer to [7] for a detailed explanation of each loss.

Since the content encoding is fixed during augmentation, the augmented image
has the same nuclei location and shape as the input image and thus inherits
the nuclei segmentation mask from the original image. This is a key difference
between our approach and a common cycleGAN-based image style transfer [8], as
there is no guarantee of content invariance after the cycleGAN transfer.

###  2.4 Mask R-CNN for instance segmentation

For nuclei segmentation, we used two implementations published in the
competition leaderboard555https://github.com/Lopezurrutia/DSB_2018,
https://github.com/mirzaevinom/data_science_bowl_2018 as baselines. Both teams
use Mask R-CNN [3] for segmentation. The implementations differ in the image
pre-processing and preparation of the network training. We choose these two
methods because of their publicly accessible code, sufficient documentation,
excellent performance (both methods were ranked among the top-5 in the
leaderboard), as well as relatively simple network training (only one model
needs to be trained, by contrast, the top-1 approach trained 32 U-nets, which
would consume considerably more training time and resources). Notably, the
second approach developed by the team mirzaevinom used additional data beyond
the data provided by the organizer. However, we only used the annotated
training data provided by the organizer to train our model so our model
performance differs slightly from their results reported in the leaderboard.
For each method, we train two models, one with and one without multi-modality
style transfer.

###  2.5 Test time augmentation

In the inference stage, we use test time augmentation (TTA). TTA uses simple
geometric augmentation on test images, e.g., rotating, flipping, color
jittering, and image rescaling, and aggregates the model predictions on these
augmented images. Thereby, TTA improves the robustness of Mask R-CNN without
any additional training cost, leading to an increase in segmentation accuracy
when compared to prediction on the original image [9].

##  3 Results and Discussion

###  3.1 Clustering into modalities

Figure 2b shows that the training set can be clustered into six sets that
correspond indeed to the different modalities. These clusters can also be seen
in the low-dimensional principal component analysis (PCA) embedding of the
training data visualized in Figure 2a. We observe that the training set is
highly imbalanced: over $75\%$ of all training images are fluorescent images,
whilst the remaining bright-field images are grouped into five distinct,
partly very small clusters.

###  3.2 Improved segmentation performance by style transfer augmentation

Figure 3 shows a few examples of images created by our multi-modality style
transfer GAN from one domain to the others. Visual assessment reveals that our
generated images resemble real ones, implying that they are well suited to the
augmentation task.

Fig. 3: Examples of the outcome of multi-modality style transfer. First
column: original images. Remaining columns: result of each domain’s multi-
modality style transfer with self-reconstruction on the diagonal, highlighted
in blue boxes.

We quantify the add-on value of our proposed augmentation method by training
models with and without augmentation of the multi-modality style transfer GAN,
trained with only the training data set. The final evaluation score is based
on the Intersection-over-Union (IoU) metric, determined by submitting our
segmentation results of the second-stage test dataset to the Kaggle
competition666https://www.kaggle.com/c/data-science-bowl-2018/submit. As shown
in Table 1, including style transfer augmentation in the method increased the
nuclei segmentation accuracy from $53.2\%$ to $60.9\%$ for the method I and
$59.9\%$ to $61.3\%$ for method II, respectively. Notably, a score around
$61\%$ was ranked among the top-5 and almost only achieved by using additional
datasets. In Figure 4, we can observe that our augmentation technique using
the multi-modality style transfer GAN improves the segmentation results,
especially in less-represented image modalities such as bright-field images of
stained tissue sections and fluorescence images of large nuclei (most
fluorescent images in the training set contain only small nuclei).

In addition to the above two baseline methods, we also quote the result from
the team BIOMAGic ($57.0\%$ IoU), as it was the only one to use style transfer
among the top-25 submissions during the competition. Unlike our style transfer
between multiple microscopy image modalities, BIOMAGic (and its successor
nucleAIzer [10]) perform style transfer between images and their segmentation
masks, which can be trained only by annotated images. By contrast, our method
can be trained in an unsupervised fashion using images without annotated
segmentation masks. In particular, their style transfer approach required
training 134 networks for fine-grained image clusters for the same
competition, whereas our approach only required the training of one network.

Fig. 4: Segmentation results without (first row) and with (second row) our augmentation. Predicted mask contours in common are shown in blue. Yellow contours in the second row show the improvements compared to the first row. Methods | IoU score
---|---
BIOMAGic | 0.570
Deep Retina w/o our augmentation | 0.532
Deep Retina w/ our augmentation | 0.609
Inom Mirzaev w/o our augmentation | 0.599
Inom Mirzaev w/ our augmentation | 0.613
Table 1: IoU scores of different segmentation methods.

Although our approach is effective in transferring style across the majority
of modalities, it also has limitations. When using our style transfer GAN from
image modalities that only show nuclei to another modality where both nuclei
and cytoplasm are visible, unrealistic images are generated as demonstrated in
Figure 5. This is because that these two modalities do not only differ in
styles but also in content, hence the content and style cannot be efficiently
disentangled.

##  4 Conclusion

Fig. 5: The style transfer GAN fails when images at two modalities contain
different content.

In summary, we developed an augmentation technique using a multi-modality
style transfer GAN to transfer microscopy nuclei images from one modality to
another. During training a Mask R-CNN for nuclei segmentation, this
augmentation strategy facilitates the training by increasing the diversity of
the training images, hence making it more robust to the test data
heterogeneity and resulting in better segmentation accuracy.

##  5 Compliance with Ethical Standards

This research study was conducted retrospectively using human subject data
made available in the Broad Bioimage Benchmark Colletction with accession
number BBBC038777https://bbbc.broadinstitute.org/BBBC038. Ethical approval was
not required as confirmed by the license attached with the open-access data.

## References

  * [1] Metin N Gurcan, et al.,  “Histopathological image analysis: A review,”  IEEE reviews in biomedical engineering, vol. 2, pp. 147–171, 2009\.
  * [2] Erick Moen, et al.,  “Deep learning for cellular image analysis,”  vol. 16, pp. 1233–1246. Springer Science and Business Media LLC, May 2019.
  * [3] Kaiming He, et al.,  “Mask r-CNN,”  in 2017 IEEE International Conference on Computer Vision (ICCV). Oct. 2017, IEEE.
  * [4] Olaf Ronneberger, et al.,  “U-net: Convolutional networks for biomedical image segmentation,”  in Lecture Notes in Computer Science, pp. 234–241. Springer International Publishing, 2015.
  * [5] Juan C. Caicedo, et al.,  “Nucleus segmentation across imaging experiments: the 2018 data science bowl,”  Nature Methods, vol. 16, no. 12, pp. 1247–1253, Oct. 2019.
  * [6] Sophia J. Wagner, et al.,  “Structure-preserving multi-domain stain color augmentation using style-transfer with disentangled representations,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 257–266.
  * [7] Hsin-Ying Lee, et al.,  “Diverse image-to-image translation via disentangled representations,”  in European Conference on Computer Vision, 2018.
  * [8] Jun-Yan Zhu, et al.,  “Unpaired image-to-image translation using cycle-consistent adversarial networks,”  Oct. 2017, IEEE.
  * [9] Nikita Moshkov, et al.,  “Test-time augmentation for deep learning-based cell segmentation on microscopy images,”  Scientific Reports, vol. 10, no. 1, Mar. 2020.
  * [10] Reka Hollandi, et al.,  “nucleAIzer: A parameter-free deep learning framework for nucleus segmentation using image style transfer,”  Cell Systems, vol. 10, no. 5, pp. 453–458.e6, May 2020.