# Interacting spin-3/2 fermions in a Luttinger (semi)metal:
competing phases and their selection in the global phase diagram

Andr$\acute{\mbox{a}}$s L. Szab$\acute{\mbox{o}}$ Max-Planck-Institut für
Physik komplexer Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany  Roderich
Moessner  Max-Planck-Institut für Physik komplexer Systeme, Nöthnitzer Str.
38, 01187 Dresden, Germany  Bitan Roy  Max-Planck-Institut für Physik
komplexer Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany

###### Abstract

We compute the effects of electronic interactions on gapless spin-3/2
excitations that in a noninteracting system emerge at a bi-quadratic touching
of Kramers degenerate valence and conduction bands in three dimensions, also
known as a Luttinger semimetal. This model can describe the low-energy physics
of HgTe, gray-Sn, 227 pyrochlore iridates and half-Heuslers. For the sake of
concreteness we only consider the short-range components of the Coulomb
interaction (extended Hubbardlike model). By combining mean-field analysis
with a renormalization group (RG) calculation (controlled by a “small”
parameter $\epsilon$, where $\epsilon=d-2$), we construct multiple cuts of the
global phase diagram of interacting spin-3/2 fermions at zero and finite
temperature and chemical doping. Such phase diagrams display a rich confluence
of competing orders, among which rotational symmetry breaking nematic
insulators and time reversal symmetry breaking magnetic orders (supporting
Weyl quasiparticles) are the prominent candidates for excitonic phases. We
also show that even repulsive interactions can be conducive to both mundane
$s$-wave and topological $d$-wave pairings. The reconstructed band structure
(within the mean-field approximation) inside the ordered phases allows us to
organize them according to the energy (entropy) gain in the following
(reverse) order: $s$-wave pairing, nematic phases, magnetic orders and
$d$-wave pairings, at zero chemical doping. However, the paired states are
energetically superior over the excitonic ones for finite doping. The phase
diagrams obtained from the RG analysis show that for sufficiently strong
interactions, an ordered phase with higher energy (entropy) gain is realized
at low (high) temperature. In addition, we establish a “selection rule”
between the interaction channels and the resulting ordered phases, suggesting
that repulsive short-range interactions in the magnetic (nematic) channels are
conducive to the nucleation of $d$-wave ($s$-wave) pairing among spin-3/2
fermions. We believe that the proposed methodology can shed light on the
global phase diagram of various two and three dimensional interacting multi-
band systems, such as Dirac materials, doped topological insulators and the
like.

##  I Introduction

The discovery of new phases of matter, and the study of the transitions
between them, form the core of condensed matter and materials physics chaikin
. Much experimental ingenuity is devoted to realising and controlling
different tuning parameters in the laboratory–such as temperature, pressure,
magnetic field, chemical composition–to drive a system from one phase to the
other chaikin . Most simply, the appearance of different phases can often be
appreciated from the competition between energy and entropy. For example, with
decreasing temperature transitions from water vapor to liquid to ice are
intimately tied with the reduction of entropy or gain in energy. Similarly, in
a metal, entropy of gapless excitations on the Fermi surface is exchanged for
condensation energy as a superconducting gap opens tinkham .

With increasing complexity of quantum materials attained in recent decades,
the richness of the global phase diagram of strongly correlated materials has
amplified enormously. And various prototypical representatives, such as
cuprates, pnictides, heavy fermion compounds display concurrence of competing
orders, among which spin- and charge-density-wave, superconductivity,
nematicity are the most prominent ones. Besides establishing the existence of
– and, hopefully, eventually utilizing in technological applications – these
phenomena, an obvious challenge is to discover any simplifying perspective, or
at least heuristic classification scheme, for predicting and classifying them:
do there exist any organising principles among multiple competing orders that
can shed light on the global phase diagram of strongly correlated materials?
Restricting ourselves to a specific but remarkably rich metallic system, we
here give a partially affirmative answer to this question. We study a
collection of strongly interacting spin-3/2 fermions in three dimensions that
in the normal phase display a bi-quadratic touching of Kramers degenerate
valence and conduction bands at an isolated point in the Brillouin zone, see
Fig. 1. This system is also known as Luttinger (semi)metal luttinger ;
murakami-zhang-nagaosa .

Such peculiar quasiparticle excitations can be found for example in HgTe hgte
, gray-Sn gray-sn-1 ; gray-sn-2 , 227 pyrochlore iridates (Ln2Ir2O7, where Ln
is the lanthanide element) Savrasov ; Balents1 ; Exp:Nakatsuji-1 ;
Exp:Nakatsuji-2 ; Exp:armitage  and half-Heusler compounds (such as LnPtBi,
LnPdBi) Exp:cava ; Exp:felser ; binghai . Bi-quadratic band touching has
recently been observed in the normal state of Pr2Ir2O7 Exp:Nakatsuji-1  and
Nd2Ir2O7 Exp:Nakatsuji-2  via angle resolved photo emission spectroscopy
(ARPES). While most of the iridium-based oxides support all-in all-out
arrangement of 4d Ir-electrons takagi ; krempa-1 ; krempa-2 ; nagaosa ;
Balents3 ; yamaji ; troyer ; tokura-1 ; tokura-2 , a singular member of this
family, namely Pr2Ir2O7, possibly resides at the brink of a metallic spin-ice
or 3-in 1-out ordering and supports a large anomalous Hall conductivity AHE-1
; AHE-2 ; AHE-3 ; goswami-roy-dassarma . While these materials harbor various
competing magnetic orders (due to comparable Hubbard and spin-orbit
interactions), smallness of the Fermi surface prohibits the onset of
superconductivity at the lowest achievable temperature. On the other hand,
half-Heusler compounds accommodate both anti-ferromagentism and unconventional
superconductivity Exp:Takabatake ; Exp:Paglione-1a ; Exp:Bay2012 ;
Exp:visser-1 ; Exp:Taillefer ; Exp:Zhang ; Exp:visser-2 ; Exp:Paglione-1 ;
Exp:Paglione-2 ; Exp:TbPdBi . A transition between them can be triggered by
tuning the de Gennes factor Exp:Paglione-1 . Moreover, the superconducting
YPtBi (with transition temperature $T_{c}=0.78$K) supports gapless BdG
quasiparticles at low temperatures Exp:Paglione-2 . Therefore, besides its
genuine fundamental importance, our quest focuses on a timely issue due to
growing material relevance of interacting spin-3/2 fermions, which has
triggered a recent surge of theoretical works krempa-1 ; krempa-2 ; nagaosa ;
Balents3 ; yamaji ; troyer ; goswami-roy-dassarma ; balents-kim ; Herbut-1 ;
lai-roy-goswami ; herbut-1a ; Herbut-2 ; kharitonov ; arago ; BJ-Yang ;
Fang2015 ; Yang16 ; brydon-1 ; brydon-2 ; Herbut-3 ; savary-1 ; roy-
nevidomskyy ; savary-2 ; sato ; CXLiu-1 ; CXLiu-2 ; brydon-3 ; sunbinLee .

Figure 1: A schematic phase diagram of noninteracting Luttinger fermions. The
red dot represents the Luttinger semimetal fixed point separating electron-
and hole-doped Fermi liquid of spin-3/2 fermions. The shaded sector
corresponds to the quantum critical regime associated with the $z=2$ fixed
point in $d=3$ (the red dot), controlling scaling properties in this regime
(see Sec. II.1). Above a nonuniversal energy high energy cutoff $E_{\Lambda}$
(red dashed line), spin-3/2 fermions lose their jurisdiction. The crossover
temperature $T_{\ast}$ (black dashed lines) above which critical Luttinger
fermions are operative is given by $T_{\ast}\sim|n|^{z/d}$, with $z=2$ and
$d=3$, where $n$ is the carrier density measured from the bi-quadratic band
touching point. In this work we demonstrate the role of electron-electron
interactions on the quantum critical regime (the shaded region), at finite
temperature and chemical doping (see Sec. II.2 for summary of our results).

The effects of electronic interactions on spin-3/2 fermions are addressed
within the framework of an extended Hubbardlike model, composed of only the
short-range components of repulsive Coulomb interactions. 111In this work we
neglect the long-range tail of the Coulomb interaction. When the chemical
potential is pinned at the band touching point, long-range Coulomb interaction
can give rise to an infrared stable non-Fermi liquid fixed point Abrikosov1 ;
Abrikosov2 ; balents-kim , which, may however be unstable toward the formation
of an excitonic phase kohn ; Herbut-1 ; herbut-1a . By contrast, at finite
chemical doping long-range interaction suffers conventional Thomas-Fermi
screening and becomes short-ranged in nature. A systematic incorporation of
the long-range interaction into our discussion is left for a future
investigation. Due to the vanishing density of states (DoS) in a Luttinger
system (namely, $\varrho(E)\sim\sqrt{E}$), sufficiently weak short-range
interactions are irrelevant perturbations. Therefore, any ordering takes place
at finite coupling. We here employ a renormalization group (RG) analysis to
construct various cuts of the global phase diagram at zero as well as finite
chemical doping [see Figs. 3, 4 and 5], and combine it with mean-field
analysis to gain insight into the organizing principle among distinct broken-
symmetry phases (BSPs) 222All cuts of the global phase diagram are obtained
from a RG analysis, accounting for interaction effects on Luttinger fermions.
Such an RG analysis can only predict the phase boundaries between the
Luttinger (semi)metal and various BSPs. However, our RG analysis is not
applicable inside a BSP and cannot capture order-order transitions. The color
fillings in Figs. 3, 4, 5, 6, 7 and 16 are implemented solely for the clarity
of presentation. . A gist of our findings can be summarized as follows.

1\. By computing the reconstructed band structure (within the mean-field
approximation) we organize dominant BSPs according to the condensation energy
and entropy gains. Results are summarized in Fig. 2. We note that while the
stiffness (uniform or anisotropic) of the band gap measures the condensation
energy gain, the amount of gapless quasiparticles (resulting in power-law
scaling of DoS at low energies) measures the entropy inside the ordered phase.
All cuts of the global phase diagram (obtained from a RG analysis) show that
the low (high) temperature phase yields larger condensation energy (entropy)
gain, see Fig. 3.

2\. The quasiparticle spectra inside the $s$-wave paired state and two nematic
phases (belonging to the $T_{2g}$ and $E_{g}$ representations of the cubic or
$O_{h}$ point group) are fully gapped. Hence, nucleation of these three phases
leads to the maximal gain of condensation energy, and they appear as the
dominant BSPs at zero temperature, as shown in Fig. 3.

3\. At finite temperature condensation energy gain competes with the entropy,
and phases with higher entropy are realized at higher temperatures. Onset of
any nematicity results in an anisotropic gap, in contrast to the situation
with an $s$-wave pairing. Thus former orderings are endowed with larger
(smaller) entropy (condensation energy), and are found at higher temperatures
in Figs. 3, 3. By contrast, the dominant magnetic orders (belonging to the
$A_{2u}$ and $T_{1u}$ representations) produce gapless Weyl quasiparticles and
result in $\varrho(E)\sim|E|^{2}$ scaling of the DoS at low energies. Hence,
these two magnetic orders carry larger entropy than the nematic phases or the
$s$-wave pairing, and can only be found at finite temperature, see Figs. 3, 3.
Luttinger semimetal (LSM)-$A_{2u}$ magnetic order (results from the all-in
all-out state in pyrochlore lattice Savrasov ; Balents3 ) transition at finite
temperature is consistent with the experimental observation in Nd2Ir2O7 takagi
, while the $T_{1u}$ magnetic ordering (yielding a 3-in 1-out ordering in
pyrochlore lattice and supporting anomalous Hall effect goswami-roy-dassarma )
can be germane for Pr2Ir2O7 AHE-1 ; AHE-2 ; AHE-3  333ARPES measurements in
Nd2Ir2O7 Exp:Nakatsuji-2  and Pr2Ir2O7 Exp:Nakatsuji-1  suggest that the LSM
in these compound is isotropic..

4\. Local four fermion interactions in the nematic channels are conducive for
$s$-wave pairing (at zero and finite chemical doping), see Fig. 3, 3 and 4 ,
whereas short-range magnetic interactions give birth to topological $d$-wave
pairing (only at finite chemical doping), see Fig. 5. We further elaborate
such an emergent “selection rule” in Sec. II.2. Confluence of magnetic order
and $d$-wave pairing (resulting in gapless BdG quasiparticles, found in YPtBi
Exp:Paglione-2 ) is in (qualitative) agreement with the global phase diagram
of LnPdBi Exp:Paglione-1 .

Figure 2:  Hierarchy of various dominant orders in an interacting Luttinger
semimetal according to the gain of the condensation energy ($\Delta_{F}$) and
entropy ($\Delta_{S}$) inside the ordered phases. The condensation energy
(entropy) gain increases in the direction of the $\Delta_{F}(\Delta_{S})$
arrow. If an order-parameter matrix ($M$) anti-commutes (commutes) with
$N^{M}_{\rm anti}(N^{M}_{\rm com})$ number of matrices appearing in the
Luttinger model [see Eq. (1)], then $\Delta_{F}\sim N^{M}_{\rm anti}$ and
$\Delta_{S}\sim N^{M}_{\rm com}$ (qualitatively), see Sec. IV.4 for detailed
discussion. At finite chemical doping superconducting orders are always
energetically superior over the excitonic ones. In the global phase diagram,
phases with higher gain in condensation energy (entropy) appear at low (high)
temperature, see for example Figs. 3, 4 and 5. We also display the scaling of
density of states [$\varrho(E)$] at low-energies in the presence of both point
and line nodes.

The theoretical approach outlined in this work is quite general and can be
extended to address the effects of electronic interactions in various strongly
correlated multi-band systems, among which two-dimensional Dirac and quadratic
fermions (respectively relevant for monolayer and bilayer graphene) graphene-
RMP ; Balatsky , three-dimensional doped topological, crystalline and Kondo
insulators TI-RMP-1 ; TI-RMP-2 ; TKI-Review , Weyl materials Weyl-RMP ,
twisted bilayer graphene tblg-1 ; tblg-2 ; tblg-3 , are the most prominent and
experimentally pertinent ones. In the future we will systematically study
these systems.

###  I.1 Outline

The rest of the paper is organized as follows. In the next section we present
an extended summary of our main results. The low-energy description of the
Luttinger model and its symmetry properties are discussed in Sec. III. In Sec.
IV we discuss the reconstructed band structure inside various excitonic and
superconducting phases. In Sec. V we introduce the interacting model for
spin-3/2 fermions and analyze the propensity toward the formation of various
orderings within a mean-field approximation. Sec. VI is devoted to a
renormalization group (RG) analysis of interacting Luttinger fermions at zero
and finite temperatures and chemical doping. We summarize the main results and
highlight some future outlooks in Sec. VII. Additional technical details are
relegated to appendices.

##  II Extended Summary

Our starting point is a collection of spin-3/2 fermions for which the normal
state is described by a bi-quadractic touching of Kramers degenerate valence
and conduction bands. The corresponding Hamiltonian operator is luttinger ;
murakami-zhang-nagaosa

$~{}\hat{h}_{\rm L}({\bf
k})=-\frac{k^{2}}{2m}\left[\cos\alpha\sum^{3}_{j=1}\hat{d}_{j}\Gamma_{j}+\sin\alpha\sum^{5}_{j=4}\hat{d}_{j}\Gamma_{j}\right]-\mu,$
(1)

where $\hat{d}_{j}$s are five $d$-wave harmonics in three dimensions,
$\Gamma_{j}$s are five mutually anticommuting four dimensional Hermitian
matrices, and $m/[\cos\alpha\;(\sin\alpha)]$ is the effective mass for gapless
excitations in the $T_{2g}(E_{g})$ orbitals in a cubic environment. Momentum
$k$ and chemical potential $\mu$ are measured from the band touching point.
Additional details of this model are discussed in Sec. III and Appendix C. The
mass anisotropy parameter ($\alpha$) lies within the range
$0\leq\alpha\leq\frac{\pi}{2}$ goswami-roy-dassarma . But, for the sake of
concreteness we restrict our focus on the isotropic system with
$\alpha=\frac{\pi}{4}$. For discussion on the evolution of phase diagrams with
varying $\alpha$, see Secs. V.1 and VI, and Figs. 6 and 16. The LSM is
realized as an _unstable_ fixed point at $\mu=0$, the red dot in Fig. 1. This
fixed point is characterized by the dynamic scaling exponent $z=2$,
determining the relative scaling between energy and momentum according to
$E\sim|{\bf k}|^{z}$. The chemical potential is a _relevant_ perturbation at
this fixed point, with the scaling dimension $[\mu]=2$. Hence, the correlation
length exponent at this fixed point is $\nu=1/2$. Therefore, the LSM can be
envisioned as a quantum critical point (QCP) separating electron-doped (for
$\mu>0$, the brown region) and hole-doped (for $\mu<0$, the green region)
Fermi liquid phases, as shown in Fig. 1. Our discussion is focused on the
quantum critical regime (the shaded region).

The crossover temperature ($T_{\ast}$) separating the quantum critical regime
accommodating gapless spin-3/2 excitations from the Fermi liquid phases can be
estimated in the following way

$T_{\ast}\sim\frac{\hbar^{2}}{2m}\times\frac{1}{\xi^{2}}\sim\frac{\hbar^{2}}{2m}\>|n|^{2/3},$
(2)

where $\xi$ is a characteristic length scale and $|n|\sim|\mu|/\xi^{d}$ is the
carrier density. Two critical exponents ($z$ and $\nu$) and the dimensionality
of the system ($d=3$) determine the scaling of various thermodynamic (such as
specific heat, compressibility) and transport (such as dynamic conductivity)
quantities in this regime.

Figure 3:  Various cuts of the finite temperature ($t$) phase diagram of an
interacting, but isotropic Luttinger semimetal. Panels (a) and (b)
respectively depict onset of $T_{2g}$ and $E_{g}$ nematic orders at higher
temperatures as we tune interactions in these two channels. Recall while the
$s$-wave superconductor yields fully gapped spectra, both nematic phases
produce anisotropic gaps [see Sec. IV.3]. Hence, the energy-entropy
competition [see Sec. IV.4] favors $s$-wave pairing (nematic phases) at low
(high) temperature. When we tune the strength of (c) $A_{2u}$ and (d) $T_{1u}$
magnetic interactions, $E_{g}$ and $T_{2g}$ nematic orders set in at low
temperature, respectively, and corresponding magnetic orders nucleate only at
higher temperature, since both magnetic orders produce gapless Weyl fermions
(less gain of condensation energy, but higher entropy). See Sec. VI.3 for
details of the renormalization group analysis at finite temperature. The white
regions represent Luttinger semimetal without any ordering. The interaction
couplings ($g_{{}_{i}}$) are measured in units of $\epsilon$ (also in Figs. 4,
5, 6, 7 and 16).

###  II.1 Critical scaling in noninteracting system

The free-energy density (up to an unimportant temperature ($T$) independent
constant) inside the critical regime is given by (setting $k_{B}=1$)

$~{}f=T^{5/2}\;\frac{(2m)^{\frac{3}{2}}}{4\pi^{\frac{3}{2}}}\left[{\rm
Li}_{\frac{5}{2}}\left(-e^{\mu/T}\right)+{\rm
Li}_{\frac{5}{2}}\left(-e^{-\mu/T}\right)\right],$ (3)

where ${\rm Li}$ represents the polylogarithimic function. The specific heat
of this system is given by

$~{}C_{V}=-T\frac{\partial^{2}f}{\partial T^{2}}\approx
T^{3/2}\>\frac{(2m)^{\frac{3}{2}}}{32\pi^{\frac{3}{2}}}\left[15a-b\;\frac{\mu^{2}}{T^{2}}\right],$
(4)

for $\mu/T\ll 1$ (ensuring that the system resides inside the critical
regime), where $a\approx 1.7244$ and $b\approx 0.6049$. From the above
expression of the free-energy density we can also extract the scaling of
compressibility ($\kappa$), given by

$~{}\kappa=-\frac{\partial^{2}f}{\partial\mu^{2}}\approx\sqrt{T}\>\frac{(2m)^{\frac{3}{2}}}{2\pi^{\frac{3}{2}}}\left[b+6c\;\frac{\mu^{2}}{T^{2}}\right],$
(5)

where $c\approx 0.00989$. Therefore, the presence of finite chemical doping
does not alter the leading power-law scaling of physical observables, such as
$C_{V}\sim T^{3/2}$, $\kappa\sim\sqrt{T}$, but only provides _subleading_
corrections, which are suppressed by a parametrically small quantity $\mu/T$.
444The scaling of specific heat and compressibility is determined by the
dimensionality ($d$) and dynamic scaling exponent ($z$) according to
$C_{V}\sim T^{1+d/z}$ and $\kappa\sim T^{d/z}$, respectively. Also note that

$~{}\frac{C_{V}/T}{\kappa}\approx 5.37611$ (6)

is a universal ratio, capturing the signature of a $z=2$ quantum critical
point in $d=3$. A detailed derivation of this analysis is shown in Appendix A.
Qualitatively similar sub-leading corrections are also found in the scaling of
the dynamic conductivity, which we discuss now.

Gauge invariance mandates that the conductivity ($\sigma$) must scale as
$\sigma\sim\xi^{2-d}$. Hence for a collection of $z=2$ quasiparticles (such as
the Luttinger fermions), $\sigma\sim\sqrt{T}$ or $\sqrt{\omega}$ in three
spatial dimensions. Indeed we find that the Drude (Dr) component of the
dynamic conductivity in the Luttinger system is given by [see Appendix B]

$~{}\sigma_{\rm
Dr}(\omega,T)=e^{2}\;\delta\left(\frac{\omega}{T}\right)\sqrt{m\;T}\>\>F_{\rm
Dr}\left(\frac{\mu}{T}\right),$ (7)

where $F_{Dr}(x)$ is a monotonically increasing universal function of its
argument [see Eq. (79) and Fig. 21] and $\omega$ is the frequency. On the
other hand, the inter-band (IB) component of the optical conductivity reads as

$~{}\sigma_{\rm
IB}(\omega,T)=e^{2}\sqrt{m\;\omega}\>\>\sum_{\tau=\pm}\;\tanh\left(\frac{\omega+2\tau\mu}{4T}\right).$
(8)

Hence, inter-band component of the optical conductivity vanishes as
$\sqrt{\omega}$ as $\omega\to 0$ and the LSM can be identified as a _power-law
insulator_.

Therefore, even when the chemical doping is finite there exists a wide quantum
critical regime, shown in Fig. 1, where the scaling of thermodynamic and
transport quantities are essentially governed by $z=2$ quasiparticles, and the
chemical potential provides only sub-leading corrections. Next we highlight
the imprint of finite temperature and chemical doping on the global phase
diagram of interacting spin-3/2 fermions.

###  II.2 Electron-electron interactions in a Luttinger (semi)metal

In this work we compute the effects of electron-electron interactions on
Luttinger fermions, occupying the critical regime of the noninteracting fixed
point, see Fig. 1. In this regime any short-range or local four-fermion
interaction ($\lambda$) is an _irrelevant_ perturbation, since

$[\lambda]=z-d=2-3=-1,$

due to the vanishing DoS, namely $\varrho(E)\sim\sqrt{E}$. We use a RG
analysis, tailored to address the effects of electronic interactions on
Luttinger fermions in $d=3$, constituting a $z=2$ band structure, to arrive at
various cuts of the global phase diagram of this system. If, on the other
hand, temperature and chemical doping are such that the system resides inside
a Fermi liquid phase, the notion of $z=2$ nodal quasiparticles becomes moot
and our RG analysis loses its jurisdiction. 555 Note that in the presence of a
Fermi surface the DoS is constant and the interaction coupling $\lambda$ is
dimensionless. Consequently a Fermi liquid becomes unstable toward the
formation of a superconductor (often non-$s$-wave) even in the presence of
repulsive electron-electron interactions, following the spirit of the Kohn-
Luttinger mechanism Sankar-RG-RMP ; kohn-Luttinger ; KL-Chubukov-Review ;
zanchi-schulz ; hlubina ; raghu-kivelson  and the superconducting transition
temperature ($T_{c}$) mimics the BCS-scaling law $T_{c}\sim\exp[-1/\lambda]$.
Furthermore, we augment the RG analysis with an organizing principle based on
the competition between energy and entropy. To this end we rely on the
computation of the reconstructed band structure inside the ordered phases
within the mean-field approximation. Subsequently, we also promote a
“selection rule” among neighboring phases in the global phase diagram,
originating purely from their algebraic or symmetry properties. For the sake
of simplicity we concentrate on the isotropic system ($\alpha=\frac{\pi}{4}$)
in the following three subsections. Nonetheless, our results hold (at least
qualitatively) for any arbitrary value of $\alpha$, as summarized in the last
subsection.

IIB1. Organizing principle: Emergent Topology & Energy-Entropy

Let us first promote an organizing principle among BSPs according to their
contribution to the energy and entropy gain and anticipate their presence in
the global phase diagram. In what follows we highlight the reconstructed band
structure inside the dominant ordered phases within a mean-field or Hartree-
Fock approximation, which by construction undermines the ordered parameter
fluctuations. The emergent band topology is computed by diagonalizing an
effective single-particle Hamiltonian, composed of the noninteracting
Luttinger Hamiltonian and corresponding order parameter, see Sec. IV.3 for
details.

Perhaps it is natural to anticipate that at zero temperature strong electronic
interactions favor the phases that produce the largest spectral gap, as the
onset of these ordered states offers maximal gain of condensation energy. In a
LSM there are three candidate BSPs that yield fully gapped quasiparticle
spectra: (a) an $s$-wave superconductor, producing a uniform and isotropic
gap, and (b) two nematic orders (belonging to the $T_{2g}$ and $E_{g}$
representations), producing anisotropic gaps. As shown in Fig. 3, only these
three phases can be found in an isotropic and interacting LSM at zero
temperature.

The energy-entropy competition, leading to an organizing principle among
competing phases at finite temperature, can be appreciated from the scaling of
the DoS or the stiffness (uniform or anisotropic) of the spectral gap. As
mentioned above, the $s$-wave pairing and nematic orders respectively produce
uniform and anisotropic gaps, whereas two magnetic orders, belonging to the
$A_{2u}$ and $T_{1u}$ representations, respectively produce eight Balents3
and two goswami-roy-dassarma  isolated simple Weyl nodes, around which the DoS
vanishes as $\varrho(E)\sim|E|^{2}$ for sufficienlty low energies. On the
other hand, each copy of the $d$-wave pairings accommodates two nodal loops
for which the low-energy DoS scales as $\varrho(E)\sim|E|$ [see Sec. IV.4]
brydon-1 ; Herbut-3 ; roy-nevidomskyy ; savary-2 ; sunbinLee . Since we are
interested in energy or temperature scales much smaller than the ultraviolet
cutoff or bandwidth ($|E|\ll 1$), the structure of the spectral gap (isotropic
or anisotropic) and power-law scaling of DoS carry sufficient information to
organize the ordered phases according to their contribution to condensation
energy and entropy, summarized in Fig. 2. In brief, existence of more gapless
points (resulting in higher DoS near $E=0$) yields larger entropy, while a
more uniform gap leads to higher gain in condensation energy. From various
cuts of the global phase diagram at finite temperature, see Fig. 3, we note
the following common feature: _Among competing orders, the one with maximal
gain in condensation energy appears at low temperature, while the phase with
higher entropy is realized at higher temperature_ , in accordance with the
general principle of energy-entropy competition. Since the DoS in a LSM scales
as $\varrho(E)\sim\sqrt{E}$ (maximal entropy), it can always be found at
sufficiently high (weak) enough temperature (interactions).

A similar conclusion can also be arrived at when the chemical potential is
placed away from the band touching point. At finite chemical doping all
particle-hole (two nematic and two magnetic) orders produce a Fermi surface
(according to the Luttinger theorem Luttinger_Theorem ) and hence a finite
DoS. By contrast, any superconducting order at finite doping maximally gaps
the Fermi surface. Therefore, at finite chemical doping superconducting orders
are energetically superior to the excitonic orders, and they can be realized
at sufficiently low temperature even in the presence of repulsive electronic
interactions. Concomitantly, the particle-hole orders are pushed to the higher
temperature and interaction regime, see Figs. 4 and 5.

Even though we gain valuable insights into the organization of various BSPs in
the global phase diagram of strongly interacting spin-3/2 fermions from the
competition between energy and entropy inside the ordered phases (guided by
emergent topology of reconstructed band structure), the phase diagrams shown
in Figs. 3, 4 and 5 are obtained from an unbiased RG analysis, which
systematically accounts for quantum fluctuations beyond the saddle point or
mean-field approximation. Next we highlight the key aspects of the RG
analysis.

Figure 4:  Realization of $s$-wave pairing from repulsive electronic
interactions in the (a) $T_{2g}$ and (b) $E_{g}$ nematic channels, for finite
$\mu$. In the presence of chemical doping the $s$-wave pairing occupies a
larger portion of the phase diagrams and the threshold couplings for the onset
of two nematic orders get pushed toward stronger couplings [compare with Figs.
3 and 3]. Hence, nematic interactions favor the onset of $s$-wave pairing in
an isotropic Luttinger system ($\alpha=\frac{\pi}{4}$).

IIB2. Methodology: Renormalization Group

The RG analysis we pursue in this work is controlled by a “small” parameter
$\epsilon$, measuring the deviation from the lower critical two spatial
dimensions ($d=2$) of the theory, where local four-fermion interactions are
_marginal_ , with $\epsilon=d-z=d-2$, and hence $[\lambda]=-\epsilon$. Both
temperature and chemical potential (bearing the scaling dimension of energy)
are _relevant_ perturbations at the $z=2$ fixed point, with scaling dimension
$[T]=[\mu]=z=2$. The leading order RG analysis can be summarized in terms of
the following set of coupled flow equations

$\displaystyle\frac{dt}{d\ell}=zt,\quad\frac{d\mu}{d\ell}=z\mu,$ (9)
$\displaystyle\frac{dg_{{}_{i}}}{d\ell}=-\epsilon
g_{{}_{i}}+\sum_{j,k}g_{{}_{j}}g_{{}_{k}}\>H_{jk}\left(\alpha,t,\mu\right),$

where $\ell$ is the logarithm of the RG scale, $t=2mT/\Lambda^{2}$ and
$\tilde{\mu}=2m\mu/\Lambda^{2}$ are respectively the dimensionless temperature
and chemical potential. For brevity we take $\tilde{\mu}\to\mu$ in the above
flow equations. Here $\left\\{g_{{}_{i}}\right\\}$ represents the set of
dimensionless four-fermion interactions with
$g_{{}_{i}}\sim\Lambda^{\epsilon}\lambda_{i}$ and
$H_{jk}\left(\alpha,t,\mu\right)$ are functions of the mass anisotropy
parameter $\alpha$, $t$ and $\mu$. The RG flow equations for $g_{i}$s are
obtained by systematically accounting for quantum corrections to the quadratic
order in the $g_{{}_{i}}$s. The relevant Feynman diagrams are shown in Fig.
14. A more detailed discussion of the RG analysis is presented in Sec. VI and
Appendix E. Some salient features of the RG analysis are the followings.

Figure 5:  Top row: Onset of $d$-wave pairing (belonging to the $E_{g}$
representation) at low temperature from magnetic interaction in the $A_{2u}$
channel, namely $g_{{}_{3}}$, in the presence of finite chemical doping $\mu$
(cf. Fig. 3 for $\mu=0$). The $A_{2u}$ magnetic order gets pushed toward
stronger coupling with increasing $\mu$. Bottom row: Similar phase diagrams
depicting the onset of $d$-wave pairing (belonging to the $T_{2g}$
representation) from strong repulsive magnetic interaction in the $T_{1u}$
channel, when $\mu>0$ (compare it with Fig. 3 for $\mu=0$). With increasing
chemical potential the onset of the $T_{1u}$ magnetic order takes place at
stronger coupling. Hence, magnetic interactions are conducive for the
nucleation of $d$-wave pairings in correlated Luttinger metal. Due to
nonmonotonic behavior of the phase boundaries we do not use color filling in
the lower panel.

  1. 1.

Temperature ($t$) and chemical potential ($\mu$) provide two infrared cutoffs
chakravarty ; vafek , respectively given by

$~{}\ell^{t}_{\ast}=\frac{1}{z}\;\ln\left(\frac{1}{t(0)}\right),\>\>\>\ell^{\mu}_{\ast}=\frac{1}{z}\;\ln\left(\frac{1}{\mu(0)}\right)$
(10)

for the flow of quartic couplings $\left\\{g_{{}_{i}}\right\\}$, where $t(0)$
and $\mu(0)$ represent the bare values ($<1$). Ultimately
$\ell_{\ast}=\min\left(\ell^{t}_{\ast},\ell^{\mu}_{\ast}\right)$ stops the
flow of four-fermion interactions. At zero temperature and chemical doping the
system is devoid of any such natural infrared cutoff, implying
$\ell_{\ast}\to\infty$.

  2. 2.

Any weak local four-fermion interaction is an irrelevant perturbation and all
orderings (realized when $g_{{}_{i}}(\ell_{\ast})\to\infty$) take place at
finite coupling $g_{{}_{i}}\sim\epsilon$ through quantum phase transitions
(QPTs). Such QPTs are controlled by quantum critical points (QCPs) and all
transitions are continuous in nature. The universality class of the transition
is determined by two critical exponents, given by

$~{}\nu^{-1}=\epsilon+{\mathcal{O}}(\epsilon^{2})\quad\text{and}\quad
z=2+{\mathcal{O}}(\epsilon),$ (11)

and for the physically relevant situation $\epsilon=1$.

Using the RG analysis we arrive at various cuts of the global phase diagram of
interacting spin-3/2 fermions at (1) zero chemical doping [see Fig. 3] and (2)
for finite-$\mu$ [see Figs. 4 and 5]. The universality class of the QPT leaves
its signature on the scaling of the transition temperature ($t_{c}$). Note
that $t_{c}\sim|\delta_{i}|^{\nu z}$ sondhi-RMP ; sachdev-book , where
$\delta_{i}=\left(g_{{}_{i}}-g^{\ast}_{{}_{i}}\right)/g^{\ast}_{{}_{i}}$ is
the reduced distance from the critical point, located at $g^{\ast}_{{}_{i}}$.
Hence, $t_{c}\sim|\delta|^{2}$ for $\nu=1$ and $z=2$, obtained from the
leading order $\epsilon$ expansion, after setting $\epsilon=1$, irrespective
of the choice of the coupling constant and the resulting BSP (see Fig. 17). We
discuss this issue in detail in Sec. VI.3.

Figure 6: Specific cuts of the phase diagram of an interacting, but
anisotropic Luttinger metal ($\alpha\neq\frac{\pi}{4}$), showing the
appearance of various superconducting phases at finite chemical doping
($|\mu|>0$) from repulsive electron-electron interactions, accommodating only
excitonic orders for $\mu=0$. Panels (a) and (b) show appearance of a
conventional $s$-wave pairing (blue region) from strong repulsive interaction
in the $T_{2g}$ and $E_{g}$ nematic channels for $\alpha=1.5$ and
$\alpha=0.1$, respectively [see also Figs. 16 and 16]. These two nematic
phases are respectively denoted by red and golden yellow regions. Nucleation
of various topological $d$-wave pairings, belonging to the $E_{g}$ (dark
yellow) and $T_{2g}$ (dark green) representations, from strong repulsive
magnetic interactions in the $A_{2u}$ and $T_{1u}$ channels, are respectively
shown in panel (c) and (d) for $\alpha=1.5$ and $\alpha=0.1$ [see also Figs.
16 and 16]. The $A_{2u}$ and $T_{1u}$ magnetic phases are shown in purple and
magenta. Due to a large separation of the interaction strength $g_{{}_{4}}$
required for any ordering at $\mu=0$ and $|\mu|>0$, we display the $\mu=0$ cut
of the phase diagram from panel (d) in Fig. 7. The region at weaker
interaction and higher temperature is occupied by correlated Luttinger metal,
without any long-range ordering. In panels (a) and (b) $100\;g_{{}_{i}}\to
g_{{}_{i}}$ and in (c) $10\;g_{{}_{i}}\to g_{{}_{i}}$.

IIB3. Competing Orders & Selection Rule

The correspondence between a given interaction coupling and the resulting
phases can be appreciated by formulating the whole theory in the basis of an
eight component Nambu-doubled spinor $\Psi_{\rm Nam}$, introduced in Sec.
III.3. In this basis the Luttinger Hamiltonian $\hat{h}_{\rm L}({\bf
k})\to\eta_{3}\hat{h}_{\rm L}({\bf k})$. Pauli matrices $\\{\eta_{\mu}\\}$
operate on the Nambu or particle-hole index. Any four-fermion interaction
takes the form $g_{{}_{\rm I}}\;(\Psi^{\dagger}_{\rm Nam}\;\hat{\rm
I}\;\Psi_{\rm Nam})^{2}$ and an order parameter ($\Delta_{\rm O}$) couples to
a fermion bilinear according to $\Delta_{\rm O}\;(\Psi^{\dagger}_{\rm
Nam}\;\hat{{\rm O}}\;\Psi^{\dagger}_{\rm Nam})$, where $\hat{\rm I}$ and
$\hat{\rm O}$ are eight dimensional Hermitian matrices. We argue that when
$g_{{}_{\rm I}}$ is sufficiently strong, it can support _only_ two types of
ordered phases, for which 666If $\hat{\rm O}$ and $\hat{\rm I}$ are multi-
component vectors of $M_{\rm O}$ and $M_{\rm I}$ elements, respectively, then
condition (2) is satisfied when at least $\lceil\frac{M_{\rm O}}{2}\rceil$
matrices anti-commute with $\lceil\frac{M_{\rm I}}{2}\rceil$ matrices, where
$\lceil\cdots\rceil$ is the ceiling function.

$~{}\text{either}\>\>\>(1)\>\hat{\rm O}\equiv\hat{\rm
I}\>\>\>\text{or}\>\>\>(2)\>\\{\hat{\rm O},\hat{\rm I}\\}=0.$ (12)

This outcome can be appreciated in the following way.

When an interaction coupling $g_{{}_{\rm I}}$ diverges toward $+\infty$ under
coarse graining (indicating onset of a BSP), it provides _positive_ scaling
dimension to an order parameter field $\Delta_{\rm O}$ only when one of the
above two conditions is satisfied. We substantiate this argument by
considering the relevant Feynman diagrams [see Fig. 15] in Sec. VI.5. Even
though we arrive at this “selection rule” among competing orders from a
leading order RG calculation, such a simple argument relies on internal
symmetries among competing orders (breaking different symmetries) and is
expected to hold at the non-perturbative level. We now support this claim by
focusing on a specific example.

Let us choose a particular four-fermion interaction (in the $T_{2g}$ nematic
channel)

$g_{{}_{1}}\sum^{3}_{j=1}\left(\Psi^{\dagger}_{\rm
Nam}\>\eta_{3}\;\Gamma_{j}\>\Psi_{\rm Nam}\right)^{2}.$

From the phase diagrams shown in Figs. 3 for zero and finite temperature and
4(left) for finite chemical doping, we find that when this coupling constant
is sufficiently strong, it supports two distinct BSPs.

  1. 1.

A nematic order following the $T_{2g}$ representation, for which $\hat{\rm
O}=\eta_{3}\\{\Gamma_{1},\Gamma_{2},\Gamma_{3}\\}$. In this case selection
rule (1) from Eq. (12) is satisfied, since $\hat{\rm
I}=\eta_{3}\\{\Gamma_{1},\Gamma_{2},\Gamma_{3}\\}$, and hence $\hat{\rm
O}\equiv\hat{\rm I}$.

  2. 2.

An $s$-wave superconductor following the trivial $A_{1g}$ representation, for
which $\hat{\rm O}=\\{\eta_{1},\eta_{2}\\}\Gamma_{0}$, where $\Gamma_{0}$ is a
four-dimensional identity matrix. The onset of $s$-wave pairing follows from
selection rule (2) in Eq. (12), since $\\{\hat{\rm O},\hat{\rm I}\\}=0$.

Figure 7: The phase diagram of an interacting Luttinger semimetal for
$\alpha=0.1$ and $\mu=0$, obtained by tuning the strength of the magnetic
interaction in the $T_{1u}$ channel ($g_{{}_{4}}$).

Moreover we realize that the $T_{2g}$ nematic order and the $s$-wave pairing
together constitute an $O(5)$ vector,
$\left\\{\eta_{3}\Gamma_{1},\eta_{3}\Gamma_{2},\eta_{3}\Gamma_{3},\eta_{1}\Gamma_{0},\eta_{2}\Gamma_{0}\right\\}$,
of five mutually anti-commuting matrices, reflecting the enlarged internal
symmetry between these two orders. Following the same spirit we arrive at the
following observations.

  1. 1.

Four fermion interaction in the $E_{g}$ nematic channel ($g_{{}_{2}}$)
supports (a) a nematic order transforming under the $E_{g}$ representation
[satisfying selection rule (1)] and (b) an $s$-wave superconductor [satisfying
selection rule (2)], as shown in Figs. 3 and 4(right). One can construct an
$O(4)$ vector by combining these two order-parameters [see Eq. (43)].

  2. 2.

Four fermion interaction in the $A_{2u}$ channel ($g_{{}_{3}}$) supports (a) a
magnetic order transforming under the $A_{2u}$ representation [satisfying
selection rule (1)], and (b) $E_{g}$ nematic order and $d$-wave pairings
[satisfying selection rule (2)], as shown in Figs. 3 and 5(top). Notice, we
can construct multiple copies of composite SU(2) order parameters, by
combining the $A_{2u}$ order with $E_{g}$ nematic or $d$-wave pairing, see
Figs. 18(a) and 19.

  3. 3.

Four fermion interaction in the $T_{1u}$ magnetic channel ($g_{{}_{4}}$)
supports (a) a magnetic order transforming under the $T_{1u}$ representation
[satisfying selection rule (1)], and (b) $T_{2g}$ nematic and $d$-wave
pairings [satisfying selection rule (2)], as shown in Figs. 3 and 5(bottom).
Combining the magnetic order with $T_{2g}$ nematic or $d$-wave pairing we can
construct multiple copies of composite SU(2) vector, see Figs. 18(b), 18(c)
and 20.

A more detailed discussion supporting these scenarios is presented in Sec.
VI.5. Therefore, combining the energy-entropy competition (obtained from the
reconstructed band topology within a mean-field approximation) and an unbiased
(controlled by a small parameter $\epsilon$) RG analysis with the selection
rule we gain valuable insights into the nature of broken symmetry phases,
competing orders and quantum critical phenomena in the global phase diagram of
strongly interacting spin-3/2 fermions.

IIB4. Anisotropic Luttinger (semi)metal

So far, we centered our focus on the isotropic system [realized for
$\alpha=\frac{\pi}{4}$ in Eq. (1)]. Note that for $\alpha=\frac{\pi}{4}$ the
system enjoys an enlarged (but artificial) SO(3) rotational symmetry. However,
in a cubic environment $\alpha\neq\frac{\pi}{4}$ in general. Nonetheless, all
the central results we quoted in the last three subsections hold (at least
qualitatively) for any arbitrary value of $\alpha$:
$0\leq\alpha\leq\frac{\pi}{2}$. The discussion on the role of the mass
anisotropy parameter $\alpha$ on the global phase diagram of interacting
spin-3/2 fermions is rather technical, which we address in depth in Secs. V.1,
VI.1, VI.4. We here only quote some key results, which nicely corroborate with
the rest of the discussion from this section.

  1. 1.

We identify the mass anisotropy parameter as a valuable non-thermal tuning
parameter, and for suitable choices of this parameter one can find (a)
$T_{2g}$ nematic and $A_{2u}$ magnetic order respectively for strong enough
$g_{{}_{1}}$ and $g_{{}_{3}}$ couplings (when $\alpha\to\frac{\pi}{2}$), see
Figs. 6 and 6, (b) $E_{g}$ nematic and $T_{1u}$ magnetic orders for strong
enough $g_{{}_{2}}$ and $g_{{}_{4}}$ (when $\alpha\to 0$), as shown in Figs. 6
and 7 at zero temperature and chemical doping, respectively. These outcomes
are in agreement with selection rule (1).

  2. 2.

At finite chemical doping (a) an $s$-wave pairing emerges from repulsive
electronic interaction in the $T_{2g}$ channel [see Fig. 6] as well as $E_{g}$
channel [see Fig. 6], (b) $d$-wave pairings belonging to the $E_{g}$ and
$T_{2g}$ representations respectively appear for repulsive interaction in the
$A_{2u}$ channel [see Fig. 6] and $T_{1u}$ channel [see Fig. 6]. These
outcomes are in accordance with selection rule (2), as we argued previously
for an isotropic Luttinger system.

We now proceed to a detailed discussion on each component of our work,
starting from the noninteracting Luttinger model.

##  III Luttinger model

We begin the discussion with the Luttinger model describing a bi-quadratic
touching of Kramers degenerate valence and conduction bands at an isolated
point (here chosen to be the $\Gamma=(0,0,0)$ point, for convenience) in the
Brillouin zone. In this section, we first present the low-energy Hamiltonian
and discuss its symmetry properties (Sec. III.1). Subsequently, we introduce
the corresponding imaginary time ($\tau$) or Euclidean action and the notion
of the renormalization group (RG) scaling (Sec. III.2). Finally, we define an
eight-component Nambu-doubled basis that allows us to capture all, including
both particle-hole or excitonic and particle-particle or superconducting,
orders within a unified framework (Sec. III.3).

###  III.1 Hamiltonian and Symmetries

The Hamiltonian operator describing a bi-quadratic touching of Kramers
degenerate valence and conduction bands in three dimensions is given by
luttinger ; murakami-zhang-nagaosa

$~{}\hat{h}_{\mathrm{L}}({\bf
k})=-k^{2}\left[{\sum^{3}_{j=1}}\frac{\hat{d}_{j}(\hat{{\bf
k}})}{2m_{1}}\Gamma_{j}+{\sum^{5}_{j=4}}\frac{\hat{d}_{j}(\hat{{\bf
k}})}{2m_{2}}\Gamma_{j}\right]-\mu\Gamma_{0},$ (13)

where $\Gamma_{0}$ is the four-dimensional identity matrix. Chemical potential
$\mu$ and momenta ${\bf k}$ are measured from the band touching point. Here,
$\hat{{\bf d}}(\hat{{\bf k}})$ is a five-dimensional unit vector that
transforms under the $l=2$ representation under the orbital $SO(3)$ rotations.
Hence $\hat{{\bf d}}(\hat{{\bf k}})$ is constructed from the $d$-wave form
factors or spherical harmonics $Y^{m}_{l=2}(\theta,\phi)$, as shown in
Appendix C. The corresponding four-component spinor basis is given by

$~{}\Psi^{\top}_{\bf k}=\left(c_{{\bf k},+\frac{3}{2}},c_{{\bf
k},+\frac{1}{2}},c_{{\bf k},-\frac{1}{2}},c_{{\bf k},-\frac{3}{2}}\right),$
(14)

where $c_{{\bf k},j}$ is the fermionic annihilation operator with momenta
${\bf k}$ and spin projection $j=\pm 3/2$ and $\pm 1/2$. The five mutually
anti-commuting $\Gamma$ matrices are defined as

$\displaystyle~{}\Gamma_{1}=\kappa_{3}\sigma_{2},\>\>\Gamma_{2}=\kappa_{3}\sigma_{1},\>\>\Gamma_{3}=\kappa_{3}\sigma_{0},$
$\displaystyle\Gamma_{4}=\kappa_{1}\sigma_{0},\>\>\Gamma_{5}=\kappa_{3}\sigma_{3}.$
(15)

Two sets of two dimensional Pauli matrices $\\{\kappa_{\nu}\\}$ and
$\\{\sigma_{\nu}\\}$ respectively operate on the sign (${\rm sgn}[j]$) and
magnitude ($|j|\in\\{1/2,3/2\\}$) of the spin projections, where
$\nu=0,1,2,3$. To close the Clifford algebra of all four-dimensional Hermitian
matrices we also define ten commutators according to
$\Gamma_{jk}=\left[\Gamma_{j},\Gamma_{k}\right]/(2i)$, with $j>k$ and
$j,k=1,\cdots,5$. All sixteen four-dimensional matrices can be expressed in
terms of the products of spin-3/2 matrices (${\mathbf{J}}$), as also shown in
Appendix C.

The energy spectra for Luttinger fermions are given by $\pm E_{s}({\bf
k})-\mu$, where for $s=\pm 1$

$E_{s}({\bf
k})=\frac{k^{2}}{2m}\sqrt{\cos^{2}\alpha\sum^{3}_{j=1}\hat{d}^{2}_{j}+\sin^{2}\alpha\sum^{5}_{j=4}\hat{d}^{2}_{j}},$
(16)

reflecting the quadratic band touching for $\mu=0$, which is protected by the
cubic symmetry. For brevity we drop the explicit dependence of
$\\{\hat{d}_{j}\\}$ on $\hat{\bf k}$.

Notice that the independence of $E_{s}({\bf k})$ on $s$ manifests the Kramers
degeneracy of the valence and conduction bands, ensured by (1) the time-
reversal ($\mathcal{T}$) and (2) the parity or inversion ($\mathcal{P}$)
symmetries. Specifically, under the reversal of time, ${\bf k}\to-{\bf k}$ and
$\Psi_{\bf k}\to\Gamma_{1}\Gamma_{3}\Psi_{-{\bf k}}$ and hence
${\mathcal{T}}=\Gamma_{1}\Gamma_{3}{\mathcal{K}}$, where ${\mathcal{K}}$ is
the complex conjugation, yielding ${\mathcal{T}}^{2}=-1$ (reflecting Kramers
degeneracy of bands). Under the inversion ${\mathcal{P}}:{\bf k}\to-{\bf k}$
and $\Psi_{\bf k}\to\Psi_{-{\bf k}}$.

The “average” mass $m$ and the mass anisotropy parameter $\alpha$ are
respectively given by goswami-roy-dassarma

$~{}m=\frac{m_{1}m_{2}}{m_{1}+m_{2}},\>\>\alpha=\tan^{-1}\left(\frac{m_{2}}{m_{1}}\right).$
(17)

Note that $\\{\hat{d}_{j}\\}$ for $j=1,2,3$ and $j=4,5$ belong to the $T_{2g}$
(three component) and $E_{g}$ (two component) representations of the cubic or
octahedral ($O_{h}$) point group, and $m_{1}$ and $m_{2}$ are effective masses
in these two orbitals, respectively. The mass anisotropy parameter $\alpha$
allows us to smoothly interpolate between (1) the $m_{1}\to\infty$ limit when
the dispersion of the $T_{2g}$ orbital becomes flat, yielding $\alpha\to 0$
and (2) $m_{2}\to\infty$ when the $E_{g}$ orbital becomes non-dispersive,
leading to $\alpha\to\frac{\pi}{2}$. For $\alpha=\frac{\pi}{4}$ or
$m_{1}=m_{2}$, the Luttinger model enjoys an enlarged spherical symmetry. Any
$\alpha\neq\frac{\pi}{4}$ captures a _quadrupolar distortion_ in the system
(still preserving the cubic symmetry). In what follows, we treat $\alpha$ as a
_non-thermal_ tuning parameter to explore the territory of interacting
Luttinger fermions.

The connection between the spin projections ($j=\pm 3/2$ and $\pm 1/2$) and
the bands can be appreciated most economically by taking ${\bf k}=(0,0,k)$.
For such a specific choice of momentum axis, the Luttinger Hamiltonian takes a
_diagonal_ form, given by

$\hat{h}_{\rm L}(k\hat{z})=\frac{k^{2}}{2m_{2}}{\rm
Diag.}\left[-1,1,1,-1\right]-\mu.$ (18)

From the above expression we can immediately infer that the pseudospin
projections on the valence and conduction bands are respectively $|j|=3/2$ and
$1/2$.

###  III.2 Lagrangian and Scaling

The imaginary time ($\tau$) Euclidean action corresponding to the non-
interacting Luttinger model is given by

$~{}S_{0}=\int d\tau d^{d}{\bf x}\>\Psi^{\dagger}(\tau,{\bf x})\;\hat{h}_{\rm
L}({\bf k}\to-i\nabla)\;\Psi(\tau,{\bf x}).$ (19)

The action remains invariant under the following rescaling of
space-(imaginary)time coordinates and the fermionic field

$~{}{\bf x}\to e^{\ell}\;{\bf x},\>\>\tau\to e^{z\ell}\;\tau,\>\>\Psi\to
e^{-d\ell/2}\;\Psi,$ (20)

where $z$ is the dynamic scaling exponent, measuring the relative scaling
between energy and momentum according to $E({\bf k})\sim|{\bf k}|^{z}$. For
Luttinger fermions $z=2$. The parameter $\ell$ is the logarithm of the RG
scale. In what follows in Secs. V and VI, we use the above scaling ansatz
while addressing the effects of electronic interactions in this system. Under
the above rescaling of parameters, the temperature ($T$) and chemical
potential ($\mu$) scale as

$T\to e^{-z\ell}T,\>\>\mu\to e^{-z\ell}\mu.$ (21)

Therefore, the scaling dimension of these two quantities is $[T]=[\mu]=z$
(same as that of energy). Throughout, we use the _natural unit_ , in which
$\hbar=k_{B}=1$.

###  III.3 Nambu doubling

To facilitate the forthcoming discussion we here introduce an eight-component
Nambu-doubled spinor basis (suitable to capture both exitonic and
superconducting orders within a unified framework) according to

$~{}\Psi_{\rm Nam}=\left[\begin{array}[]{c}\Psi_{\bf k}\\\
\Gamma_{1}\Gamma_{3}\left(\Psi^{\dagger}_{-{\bf
k}}\right)^{\top}\end{array}\right],$ (22)

where $\Psi_{\bf k}$ is a four-component spinor, see Eq. (14). In the lower
block of $\Psi_{\rm Nam}$ we absorb the unitary part of the time-reversal
operator ${\mathcal{T}}$, ensuring that the eight-component Nambu spinor
($\Psi_{\rm Nam}$) transforms the same way as the original four component
spinor $\Psi_{\bf k}$ under the $SU(2)$ pseudospin rotation. In this basis the
eight-dimensional Luttinger Hamiltonian takes a simple form

$~{}\hat{h}^{\rm Nam}_{\mathrm{L}}({\bf k})=\eta_{3}\;\hat{h}_{\rm L}({\bf
k}),$ (23)

and the time-reversal operator becomes ${\mathcal{T}}_{\rm
Nam}=\eta_{0}\Gamma_{1}\Gamma_{3}{\mathcal{K}}$. The newly introduced set of
Pauli matrices $\\{\eta_{\nu}\\}$ operates on the Nambu or particle-hole
indices, with $\nu=0,1,2,3$. Therefore, by construction while the excitonic
orders assume _block-diagonal_ form, all superconducting orders are _block-
off-diagonal_ in the Nambu subspace. Note that $\hat{h}^{\rm
Nam}_{\mathrm{L}}({\bf k})$ _commutes_ with the number operator
$\hat{N}=\eta_{3}\Gamma_{0}$.

##  IV Broken symmetry phases

Next we discuss possible BSPs in this system. We introduce various possible
excitonic and superconducting orders in the Nambu basis ($\Psi_{\rm Nam}$) in
two subsequent sections. Finally, we discuss the reconstructed band structure
and emergent topology inside the ordered phases.

###  IV.1 Particle-hole or excitonic orders

The effective single-particle Hamiltonian in the presence of all possible
momentum-independent or local or intra-unit cell excitonic orders is given by

$~{}H^{{\rm exc}}_{{\rm local}}=\int d^{3}{\bf r}\>\>\left(\Psi^{\dagger}_{\rm
Nam}\;\hat{h}^{{\rm exc}}_{{\rm local}}\;\Psi_{\rm Nam}\right),$ (24)

where

$\displaystyle~{}\hat{h}^{{\rm exc}}_{{\rm local}}$ $\displaystyle=$
$\displaystyle\overbrace{\Delta_{0}\eta_{3}\Gamma_{0}}^{\rm
Density}\quad+\overbrace{\eta_{3}\left[\sum^{3}_{j=1}\Delta^{j}_{1}\Gamma_{j}+\sum^{5}_{j=4}\Delta^{j}_{2}\Gamma_{j}\right]}^{\rm
Nematic}$ $\displaystyle+$
$\displaystyle\underbrace{\eta_{0}\left[\Delta_{3}\Gamma_{45}+\sum^{3}_{j=1}\Delta^{j}_{4}\Gamma_{45}\Gamma_{j}+\sum^{3}_{j=1}\sum^{5}_{k=4}\Delta^{jk}_{5}\Gamma_{jk}\right]}_{\rm
Magnetic}.$

The ordered phases can be classified according to their transformation under
the cubic ($O_{h}$) point group symmetry. Regular fermionic density
($\Delta_{0}$) does not break any symmetry (hence does not correspond to any
ordering) and transforms under the trivial $A_{1g}$ representation. A three-
component _nematic_ order-parameter, constituted by
$\vec{\Delta}_{1}=\left(\Delta^{1}_{1},\Delta^{2}_{1},\Delta^{3}_{1}\right)$,
transforms under the $T_{2g}$ representation. By contrast, a two-component
nematic order transforming under the $E_{g}$ representation is captured by
$\vec{\Delta}_{2}=\left(\Delta^{1}_{2},\Delta^{2}_{2}\right)$. Both of them
break only the cubic symmetry, but preserve time-reversal and inversion
symmetries. The ordered phase represents either a time-reversal invariant
insulator or a Dirac semimetal, about which more in a moment [see Sec. IV.3].
Since five $\Gamma$ matrices transform as components of a rank-2 tensor under
SO(3) rotations, the two nematic phases represent _quadrupolar_ orders, see
Appendix C.

All ordered phases shown in the second line of Eq. (IV.1) break time-reversal
symmetry and represent different magnetic orders. For example, $\Delta_{3}$
corresponds to an _octupolar_ order (since $\Gamma_{45}\sim J_{x}J_{y}J_{z}$),
transforming under the singlet $A_{2u}$ representation. In a pyrochlore
lattice of 227 iridates such an ordered phase represents the “ _all-in all-
out_ ” arrangement of electronic spin between two adjacent corner-shared
tetrahedra Savrasov ; Balents3 . By contrast, “ _two-in two-out_ ” or “ _spin-
ice_ ” magnetic orderings on a pyrochlore lattice are represented by a three-
component vector
$\vec{\Delta}_{4}=\left(\Delta^{1}_{4},\Delta^{2}_{4},\Delta^{3}_{4}\right)$
(accounting for six possible two-in two-out arrangements in a single
tetrahedron). Since $\Gamma_{45}\Gamma_{j}\sim 7J_{j}-4J^{3}_{j}$ such an
ordered phase contains a linear superposition of dipolar and octupolar
moments, and transforms under the $T_{1u}$ representation goswami-roy-dassarma
. Any other magnetic ordering can be represented by a six component vector
$\Delta^{jk}_{5}$ with $j=1,2,3$ and $k=4,5$. No physical realization of such
multi-component magnetic ordering in any material is currently known, and we
do not delve into the discussion on such ordering for the rest of the paper.

###  IV.2 Particle-particle or superconducting orders

The effective single particle Hamiltonian in the presence of all possible
momentum-independent or local or intra-unit cell superconducting orders reads
brydon-1 ; Herbut-3 ; savary-1 ; roy-nevidomskyy

$\displaystyle~{}H^{{\rm pair}}_{{\rm local}}=\int d^{3}{\bf
r}\>\>\left(\Psi^{\dagger}_{\rm Nam}\>\hat{h}^{{\rm pair}}_{{\rm
local}}\>\Psi_{\rm Nam}\right),$ (26)

where

$\displaystyle~{}\hat{h}^{{\rm pair}}_{\rm local}$ $\displaystyle=$
$\displaystyle\left(\eta_{1}\cos\phi+\eta_{2}\sin\phi\right)\bigg{[}\overbrace{\Delta^{\rm
p}_{A_{1g}}\Gamma_{0}}^{\rm s-wave}$ (27) $\displaystyle+$
$\displaystyle\underbrace{\sum^{3}_{j=1}\Delta^{{\rm
p},j}_{T_{2g}}\Gamma_{j}+\sum^{5}_{j=4}\Delta^{{\rm
p},j}_{E_{g}}\Gamma_{j}}_{\rm d-wave}\bigg{]},$

and $\phi$ is the global $U(1)$ superconducting phase. Any pairing
proportional to $\eta_{1}(\eta_{2})$ preserves (breaks) time-reversal symmetry
(recall that the time-reversal operator in the Nambu basis is
${\mathcal{T}}_{\rm Nam}=\eta_{0}\Gamma_{1}\Gamma_{3}{\mathcal{K}}$). Here,
$\Delta^{\rm p}_{A_{1g}}$ is the amplitude of the $s$-wave pairing,
transforming under the $A_{1g}$ representation. The $s$-wave pairing breaks
only the global $U(1)$ symmetry, but preserves the cubic symmetry. On the
other hand, $\Delta^{{\rm p},j}_{T_{2g}}$ captures the amplitude of three
$d$-wave pairings (for $j=1,2,3$) transforming under the $T_{2g}$
representation, and $\Delta^{{\rm p},j}_{E_{g}}$ for $j=4,5$ represents the
amplitude of two $d$-wave pairings belonging to the $E_{g}$ representation.
Notice $\\{\Gamma_{j},\;j=1,\cdots,5\\}$ can be expressed in terms of the
product of two spin-3/2 matrices, and all five $d$-wave pairings break the
cubic symmetry, while introducing a lattice distortion or electronic
nematicity in the system. Hence, they stand as representatives of _quadrupolar
nematic superconductors_.

###  IV.3 Reconstructed band structure and emergent topology

Figure 8:  (a) Four phase lockings (blue, red, green and black dots) of the
three component $T_{2g}$ nematic order, see Eq. (IV.3), that yield gapless
phase (topological Dirac semimetal) inside the ordered state. Corresponding
locations of two Dirac points in momentum space are shown in panel (b). For
any other generic phase locking, the ordered phase is a time-reversal symmetry
preserving insulator. Dirac points are located along the body-diagonals
($C_{3v}$ axes in a cubic system).

Next we consider the reconstructed band structure inside different BSPs which
provides valuable information regarding the emergent topology inside ordered
phases. The onset of any ordering discussed in the previous sections
destabilizes the bi-quadratic touching and gives rise to either gapped or
gapless quasiparticles (see below). Furthermore, this exercise will allow us
to appreciate the energy-entropy competition among different orderings [see
Sec. IV.4], which ultimately plays a decisive role in the organization of
various phases in the global phase diagram of interacting Luttinger fermions.

1\. $T_{2g}$ nematicity: The three component order-parameter for the $T_{2g}$
nematic phase gives birth to gapless quasiparticles for the following _four_
configurations

$\displaystyle~{}\vec{\Delta}_{1}=\frac{|\Delta_{1}|}{\sqrt{3}}\bigg{\\{}\underbrace{\left(+,+,+\right)}_{1},\underbrace{\left(-,-,+\right)}_{2},\underbrace{\left(+,-,-\right)}_{3},\underbrace{\left(-,+,-\right)}_{4}\bigg{\\}}.$

These four phase lockings are respectively shown as blue, red, green and black
points in Fig. 8. The gapless phase corresponds to a _topological_ Dirac
semimetal (since nematicity preserves the Kramers degeneracy of valence and
conduction bands), similar to the ones recently found in Cd3As2 cdas:Exp  and
Na3Bi nabi:Exp . The DoS in a Dirac semimetal vanishes as
$\varrho(E)\sim|E|^{2}$. The Dirac points are located along the body diagonals
(the $C_{3v}$ axes) of a cubic system and respectively placed at

$\displaystyle{\bf
k}=\pm\bigg{\\{}\underbrace{\left(1,1,1\right)}_{1},\underbrace{\left(1,1,-1\right)}_{2},\underbrace{\left(1,-1,1\right)}_{3},\underbrace{\left(1,-1,-1\right)}_{4}\bigg{\\}}k_{0},$

where $k_{0}=\left[2m_{1}\Delta_{1}/3\right]^{1/2}$, as shown in Fig. 8. For
any other phase locking within the $T_{2g}$ sector the system becomes an
_insulator_. The spectral gap in the insulating phase is _anisotropic_ and it
is energetically superior over the gapless Dirac semimetal phase.

2\. $E_{g}$ nematicity: The two component $E_{g}$ nematic order is most
conveniently described in terms of the following parametrization

$\vec{\Delta}_{2}=\frac{|\Delta_{2}|}{\sqrt{2}}\left(\sin\phi_{{}_{\rm
E_{g}}},\>\cos\phi_{{}_{\rm E_{g}}}\right),$ (30)

where $\phi_{{}_{\rm E_{g}}}$ is the internal angle in the order-parameter
space. Only for

$~{}\phi_{{}_{\rm
E_{g}}}=\left\\{\underbrace{0}_{1},\>\>\underbrace{2\pi/3}_{2},\>\>\underbrace{4\pi/3}_{3}\right\\}$
(31)

the quasi-particle spectra are gapless, as shown in Fig. 9, and the ordered
phase represents a topological Dirac semimetal. Specifically, for
$\phi_{{}_{\rm E_{g}}}=0,2\pi/3,4\pi/3$, the Dirac points are respectively
located on $k_{z}$, $k_{x}$ and $k_{y}$ axes (the $C_{4v}$ axes), see Fig. 9,
and the separation of two Dirac points is given by $2k_{0}$, where
$k_{0}=\left[2m_{2}\Delta_{2}/\sqrt{2}\right]^{1/2}$. For any other phase
locking within the $E_{g}$ sector, the system becomes an insulator.

Figure 9:  (a) Three possible phase lockings (shown by blue, black and red
dots) of two-component $E_{g}$ nematic order, see Eq. (31), that give rise to
topological Dirac semimetal inside the ordered phase. The Dirac points inside
the $E_{g}$ nematic phase are located on the $k_{z}$, $k_{x}$ and $k_{y}$ axes
($C_{4v}$ axes in a cubic system), as shown in panel (b), in contrast to the
situation inside the $T_{2g}$ nematic phase, see Fig. 8. For any other phase
locking the system is an insulator.

3\. $A_{2u}$ magnet: In the presence of an octupolar $A_{2u}$ ordering, the
two-fold degeneracy of the valence and conduction band gets lifted and a pair
of Kramers non-degenerate bands touch each other at the following _eight_
points in the Brillouin zone [see Fig. 10]

${\bf k}=\left(\pm 1,\pm 1,\pm 1\right)\;k_{0},$ (32)

where $k_{0}=\sqrt{2m_{1}\Delta_{3}/3}$. They represent simple Weyl points,
which act as source (4 of them) and sink (4 of them) of Abelian Berry
curvature of unit strength. However, due to an octupolar arrangement of the
Weyl nodes, the net Berry curvature through any high-symmetry plane is
precisely _zero_ and this phase does not support any anomalous Hall effect.
The DoS at low energies then scales as $\varrho(E)\sim|E|^{2}$ Savrasov ;
Balents3 ; goswami-roy-dassarma .

4\. $T_{1u}$ magnet: For each component of $T_{1u}$ magnetic order
(represented by the matrix operator $\Gamma_{45}\Gamma_{j}$ with $j=1,2,3$)
the ordered phase supports _two_ Weyl nodes along one of the $C_{4v}$ axes and
a _nodal-loop_ in the corresponding basal plane. For example, when
$\langle\Psi^{\dagger}\Gamma_{45}\Gamma_{3}\Psi\rangle\equiv\Delta^{3}_{4}\neq
0$ the left and right chiral Weyl nodes are located at $(0,0,\pm k_{0})$ where
$k_{0}=\sqrt{2m_{2}\Delta_{4}}$ and a nodal-loop is found in the $k_{x}-k_{y}$
plane, as shown in Fig. 10. Similarly, for $j=1$ and $2$ the Weyl nodes are
separated along the $k_{x}$ and $k_{y}$ axes, and the nodal-loops are
respectively found in the $k_{y}-k_{z}$ and $k_{x}-k_{z}$ planes. Due to the
presence of two Weyl nodes, each configuration of two-in two-out magnetic
order supports a finite anomalous Hall effect in the plane perpendicular to
the separation of the Weyl nodes. However, any _triplet_ magnetic order,
represented by $\vec{\Delta}_{4}=|\Delta_{4}|(\pm 1,\pm 1,\pm 1)/\sqrt{3}$,
gets rid of the nodal loop and supports only _two_ Weyl nodes along one of the
body-diagonals ($C_{3v}$ axes). Hence, triplet $T_{1u}$ magnetic orders are
energetically favored over their uniaxial counterparts goswami-roy-dassarma .
777The low energy DoS in the presence of a nodal loop and two point nodes (due
to a uniaxial $T_{1u}$ order) is dominated by the former and scales as
$\varrho(E)\sim|E|$, while in a triplet $T_{1u}$ state the DoS scales as
$\varrho(E)\sim|E|^{2}$ (due to the point nodes). Hence, formation of the
triplet ordering causes power-law suppression of the DoS and increases the
condensation energy gain.

5\. $A_{1g}$ or $s$-wave pairing: Notice that the matrix operator representing
an $s$-wave pairing fully anti-commutes with the Luttinger Hamiltonian (for
any value of $\alpha$) and thus corresponds to a _mass_ for Luttinger
fermions. The quasiparticle spectra inside the paired state is fully gapped,
but the phase is topologically _trivial_.

Figure 10: (a) Location of eight Weyl nodes in the presence of $A_{2u}$
octupolar order, which in a pyrochlore lattice corresponds to the all-in all-
out magnetic order. Here, red and gray dots respectively correspond to the
source ($+$) and sink ($-$) of Abelian Berry curvature, with monopole charge
$\pm 1$. (b) Nodal structure in the presence of a uniaxial $T_{1u}$ order
(when its moment points along $\hat{z}$), supporting two Weyl nodes separated
along $\hat{k}_{z}$ (red and gray dots) and a nodal loop (dark yellow ring) in
the $k_{x}-k_{y}$ plane. When the moment of the uniaxial $T_{1u}$ order points
along $\hat{x}$ and $\hat{y}$ direction, the Weyl nodes are respectively
separated along the $k_{x}$ and $k_{y}$ axes, and the nodal-loops are found in
the $k_{y}-k_{z}$ and $k_{x}-k_{z}$ plane. By contrast, inside a _triplet_
$T_{1u}$ phase, only two Weyl nodes are found along one of the body diagonals.

6\. $T_{2g}$ pairing: Three $d$-wave pairings, proportional to $\Gamma_{1}$,
$\Gamma_{2}$ and $\Gamma_{3}$ matrices, belong to the $T_{2g}$ representation
and respectively possess the symmetry of $d_{yz}$, $d_{xz}$ and $d_{xy}$
pairings. Each component supports two nodal loops in the ordered phase, as
shown in the first three rows of Table 1 brydon-1 ; roy-nevidomskyy . Two
nodal loops for the $\Gamma_{3}$ or $d_{xy}$ pairing are shown in Fig. 11. The
two nodal loops for $\Gamma_{1}$ or $d_{yz}$ and $\Gamma_{2}$ or $d_{xz}$
pairings can respectively be obtained by rotating the ones shown for $d_{xy}$
pairing by $\frac{\pi}{2}$, with respect to the $k_{y}$ and $k_{x}$ axes.

7\. $E_{g}$ pairing: $E_{g}$ pairings proportional to $\Gamma_{4}$ and
$\Gamma_{5}$ matrices respectively possess the symmetry of $d_{x^{2}-y^{2}}$
and $d_{3z^{2}-r^{2}}$ pairings and each of them supports two nodal loops, as
shown in the last two rows of Table 1 brydon-1 ; roy-nevidomskyy . Note that
two nodal loops for the $d_{x^{2}-y^{2}}$ pairing can be realized by rotating
the ones for the $d_{xy}$ pairing by $\frac{\pi}{4}$ about the $k_{z}$ axis.
However, two nodal loops for the $d_{3z^{2}-r^{2}}$ pairing, shown in Fig. 11,
cannot be rotated into the ones for $d_{x^{2}-y^{2}}$ pairing. Therefore,
despite the fact that the $d_{3z^{2}-r^{2}}$ and $d_{x^{2}-y^{2}}$ pairings
belong to the same $E_{g}$ representation, they are not energetically
degenerate roy-nevidomskyy ; sigrist-RMP . Since the radius of the nodal loops
for the $d_{3z^{2}-r^{2}}$ pairing is the _smallest_ , this paired state is
the energetically most favorable among five $d$-wave pairings. 888Even though
$d+id$ type, such as $d_{x^{2}-y^{2}}+id_{3z^{2}-r^{2}}$, pairing can
eliminate nodal loops from the spectra in favor of point nodes around which
$\varrho(E)\sim|E|$, strong inter-band coupling causes inflation of such nodes
and yields Fermi surface of BdG quasiparticles, leading to a constant DoS at
lowest energy, followed by $\varrho(E)\sim|E|$ brydon-2 . Presently, it is not
very clear between (a) individual $d$-wave pairing and (b) $d+id$ type
pairing, which one is energetically more advantageous.

Pairing | IREP. | Equations for nodal loops | Symmetry
---|---|---|---
$\Gamma_{1}$ | $T_{2g}$ |  $k^{2}_{x}+k^{2}_{y}=2m\Delta$, $k^{2}_{x}+k^{2}_{z}=2m\Delta$ | $d_{yz}$
$\Gamma_{2}$ | $T_{2g}$ |  $k^{2}_{x}+k^{2}_{y}=2m\Delta$, $k^{2}_{z}+k^{2}_{y}=2m\Delta$ | $d_{xz}$
$\Gamma_{3}$ | $T_{2g}$ |  $k^{2}_{y}+k^{2}_{z}=2m\Delta$, $k^{2}_{x}+k^{2}_{z}=2m\Delta$ | $d_{xy}$
$\Gamma_{4}$ | $E_{g}$ |  $k^{2}_{z}+k^{2}_{\perp}=2m\Delta$, $k_{x}=\pm k_{y}$ | $d_{x^{2}-y^{2}}$
$\Gamma_{5}$ | $E_{g}$ |  $k^{2}_{\perp}=4m\Delta/3$, $k_{z}=\pm k_{\perp}/\sqrt{2}$ | $d_{3z^{2}-r^{2}}$
Table 1:  The structure of two nodal loops in the presence of five individual
$d$-wave pairings, belonging to the $T_{2g}$ and $E_{g}$ representations,
where $k^{2}_{\perp}=k^{2}_{x}+k^{2}_{y}$. We display the symmetry of each
$d$-wave pairing in the proximity to the Fermi surface (realized on the
conduction or valence band) in the last column. Note that two nodal loops for
$d_{xy}$, $d_{xz}$, $d_{yz}$ and $d_{x^{2}-y^{2}}$ parings can be rotated into
each other, while those in the presence of $d_{3z^{2}-r^{2}}$ pairing are
disconnected from the remaining ones, see Fig. 11. For the sake of simplicity
we here assume $m_{1}=m_{2}=m$, for which the nodal loops are _circular_ in
shape. For $m_{1}\neq m_{2}$, the nodal loops become _elliptic_. Here,
$\Delta$ is the amplitude of $d$-wave pairings.

###  IV.4 Energy and Entropy Inside Ordered Phases

Figure 11: Structure of two nodal loops in the presence of an underlying (a)
$d_{xy}$ and (b) $d_{3z^{2}-r^{2}}$ pairings. These two pairings are
respectively represented by $\Gamma_{3}$ and $\Gamma_{5}$ matrices for the
Luttinger fermions. Note that the nodal loops for $d_{yz}(\Gamma_{1})$,
$d_{xz}(\Gamma_{2})$, $d_{x^{2}-y^{2}}(\Gamma_{4})$ pairings can be obtained
by rotating the ones shown here for $d_{xy}$ pairing about suitable momentum
axes. However, the two nodal loops for the $d_{3z^{2}-r^{2}}$ pairing are
disjoint from the remaining ones (note these two nodal loops do not cross each
other), see Sec. IV.3 and Table 1.

From the computation of the reconstructed band structure we can gain insight
into the condensation energy ($\Delta_{F}$) and entropy ($\Delta_{S}$) inside
the ordered phases. While the _stiffness_ of the spectral gap measures the
gain of condensation energy, the scaling of the DoS at low-energies (due to
gapless quasiparticles) measures the entropy. Recall that the $s$-wave pairing
yields fully gapped spectra (isotropic), while the nematic orders produce
either an anisotropic gap or gapless quasiparticles. Hence, the former
ordering is associated with higher (lower) gain in condensation energy
(entropy). On the other hand, the DoS vanishes as $\varrho(E)\sim|E|$ and
$|E|^{2}$ respectively in the presence of a nodal-loop and Dirac or Weyl
points. We found that the $A_{2u}$ magnetic order gives birth to eight Weyl
nodes, while only two Weyl nodes can be found inside the _triplet_ $T_{1u}$
magnetic order. By contrast, all five $d$-wave pairings are accompanied by two
nodal loops (see Table 1). Therefore, we can organize these ordered phases
according to their contribution to (a) condensation energy and (b) entropy
gain, as shown in Fig. 2. The LSM, on the other hand, accommodates the largest
amount of gapless fermionic excitations near zero energy, where the DoS
vanishes as $\varrho(E)\sim\sqrt{E}$. Hence, the LSM is endowed with largest
entropy. 999Such an organization of ordered phases according to their
contributions to the gain of condensation energy and entropy is purely based
on the power-law dependence of low-energy DoS or the stiffness (isotropic or
anisotropic) of the spectral gap. This procedure, however, cannot distinguish
two phases with similar scaling of the DoS, such as between $A_{2u}$ and
triplet $T_{1u}$ magnetic orders (producing Weyl nodes), or the stiffness of
the spectral gap, such as between $T_{2g}$ and $E_{g}$ nematic orders
(producing anisotropic gaps). A more microscopic analysis is needed to resolve
these situations.

We now present a simple prescription to estimate (at least qualitatively) the
gain of condensation energy or entropy. Let us assume that the eight-
dimensional Hermitian matrix $M$, entering the definition of an order
parameter as $\Delta_{M}\equiv\langle\Psi^{\dagger}_{\rm Nam}M\Psi_{\rm
Nam}\rangle$ respectively anticommutes and commutes with $N^{M}_{\rm anti}$
and $N^{M}_{\rm comm}$ number of matrices appearing in the Luttinger
Hamiltonian $\hat{h}^{\rm Nam}_{\rm L}({\bf k})$ (when $\mu=0$), and thus

$~{}N^{M}_{\rm anti}+N^{M}_{\rm comm}=5.$ (33)

Therefore, for the $s$-wave pairing, two nematic orders, two magnetic orders
and five $d$-wave pairings $N^{M}_{\rm anti}=5,4,2,1$, while $N^{M}_{\rm
comm}=0,1,3,4$, respectively. Then for any ordered phase

$\Delta_{F}\sim N^{M}_{\rm anti},\quad\Delta_{S}\sim N^{M}_{\rm comm}.$ (34)

This correspondence can be anchored from a simple example. Let us choose
$s$-wave pairing, for which $N^{M}_{\rm anti}=5,N^{M}_{\rm comm}=0$ and
chemical potential, for which $N^{M}_{\rm anti}=0,N^{M}_{\rm comm}=5$, as two
perturbations in a LSM. While the $s$-wave pairing yields an isotropic gap, a
finite chemical doping creates a Fermi surface (producing a constant DoS).
Consequently, the $s$-wave pairing (chemical doping) is accompanied by larger
gain of condensation energy (entropy). Therefore, the analysis of
reconstructed band structure and emergent topology inside BSPs allows us to
organize them according to the gain of condensation energy and entropy. The RG
analysis at finite temperature (in the regime where the dimensionless
temperature $t=2mT/\Lambda^{2}\ll 1$) captures such energy-entropy
competition, which we discuss in Sec. VI.3, see also Fig. 3.

Figure 12: The Feynamn diagram contributing to the leading order bare
susceptibility ($\chi_{{}_{M}}$), given in Eq. (38), for zero external
momentum and frequency. The _red_ dashed (_black_ solid) lines represent the
order parameter (fermionic) fields. The two vertices (_green_ dots) are
accompanied by the appropriate eight-dimensional Hermitian order-parameter
matrix $M$, appearing in the corresponding fermion bilinear
$\Psi^{\dagger}_{\rm Nam}M\Psi_{\rm Nam}$.

The hierarchy of the energy and entropy gains inside the ordered phases
changes when the chemical potential is placed away from the band touching
point (i.e., $\mu\neq 0$). Since any pairing operator _anti-commutes_ with the
number operator ($\hat{N}=\eta_{3}\Gamma_{0}$), superconducting orders
maximally gap (either fully by the $s$-wave pairing or partially by the
individual $d$-wave pairings) the Fermi surface. By contrast, any excitonic
order always gives birth to a Fermi surface, according to the Luttinger
theorem Luttinger_Theorem . Hence, at finite doping all superconductors are
energetically superior over the particle-hole orders, while excitonic orders
are accompanied by larger entropy (due to presence of a Fermi surface). The
energy-entropy competition at finite-$\mu$ is also captured by the RG
analysis, discussed in Sec. VI.4, leading to the phase diagrams shown in Figs.
4, 5 and 6.

##  V Electron-electron interactions

Next we proceed to demonstrate the onset of various BSPs, discussed in the
previous section, triggered by _repulsive_ (at the bare level) electron-
electron interactions. As mentioned earlier we will focus only on the local or
short-range part of Coulomb interaction and neglect its long-range tail. For
the sake of concreteness we assume that the local interactions are density-
density in nature. Any generic local density-density interaction (such as the
ones appearing in an extended Hubbard model, for example) can be captured by
_six_ quartic terms and the corresponding interacting Hamiltonian reads

$\displaystyle~{}H_{\rm int}$ $\displaystyle=$
$\displaystyle-\bigg{[}\lambda_{0}(\Psi^{\dagger}\Psi)^{2}+\lambda_{1}\sum^{3}_{j=1}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}$
$\displaystyle+$
$\displaystyle\lambda_{2}\sum^{5}_{j=4}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}+\lambda_{3}(\Psi^{\dagger}\Gamma_{45}\Psi)^{2}$
$\displaystyle+$
$\displaystyle\lambda_{4}\sum^{3}_{j=1}\left(\Psi^{\dagger}\Gamma_{j}\Gamma_{45}\Psi\right)^{2}+\lambda_{5}\sum^{3}_{j=1}\sum^{5}_{k=4}\left(\Psi^{\dagger}\Gamma_{jk}\Psi\right)^{2}\bigg{]}.$

In this notation $\lambda_{j}>0$ corresponds to repulsive interaction.
However, all four-fermion interactions are _not_ linearly independent due to
the existence of _Fierz identity_ among sixteen four-dimensional Hermitian
matrices, closing a $U(4)$ Clifford algebra [see Appendix D] HJR-Fierz ;
RoySDS-Fierz ; RGJ-Fierz . It turns out that any generic local interaction can
be expressed in terms of only _three_ quartic terms, and we conveniently
(without any loss of generality) choose them to be $\lambda_{0}$,
$\lambda_{1}$ and $\lambda_{2}$. Following the Fierz relations we can express
local quartic terms proportional to $\lambda_{3}$, $\lambda_{4}$,
$\lambda_{5}$ as linear combinations of above three, see Eq. (D). Whenever we
generate four-fermion interactions proportional to $\lambda_{3,4,5}$ during
the coarse-graining (discussed in Sec. VI), they can immediately be expressed
in terms of $\lambda_{0,1,2}$, and the interacting model defined in terms of
$\lambda_{0,1,2}$ [see Eq. (36) below] always remains closed under the RG
procedure to any order in the perturbation theory.

The imaginary time Euclidean action for the interacting system is given by

$\displaystyle~{}S_{\rm int}$ $\displaystyle=$ $\displaystyle S_{0}-\int d\tau
d^{d}{\bf
r}\bigg{[}\lambda_{0}(\Psi^{\dagger}\Psi)^{2}+\lambda_{1}\sum^{3}_{j=1}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}$
(36) $\displaystyle+$
$\displaystyle\lambda_{2}\sum^{5}_{j=4}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}\bigg{]},$

where $\Psi\equiv\Psi(\tau,{\bf r})$ and
$\Psi^{\dagger}\equiv\Psi^{\dagger}(\tau,{\bf r})$. Under the rescaling of
space-time(imaginary) coordinates and the fermionic fields, see Eq. (20), the
local four-fermion interaction scales as

$\lambda_{j}\to e^{(d-z)\ell}\lambda_{j}.$ (37)

Hence the scaling dimension of quartic couplings is $[\lambda_{j}]=z-d$. For
$z=2$ and $d=3$, $[\lambda_{j}]=-1$, and any weak local interaction is an
_irrelevant_ perturbation and leaves the Luttinger fermions unaffected.
Therefore, any ordering sets in at an intermediate strength of coupling
through QPT. In Sec. VI we demonstrate appearances of various BSPs using a RG
analysis, controlled via an $\epsilon$-expansion, where $\epsilon=d-2$, about
the _lower-critical_ two spatial dimension of this theory. Within the
framework of the $\epsilon$-expansion, such QPTs take place at a critical
interaction strength $\lambda^{\ast}_{j}\sim\epsilon$, and in three spatial
dimensions ($d=3$) $\epsilon=1$. Before proceeding to the RG analysis, we seek
to gain some insight into the propensity toward various orderings by computing
the corresponding mean-field susceptibility for a wide range of the mass
anisotropy parameter ($\alpha$). Readers interested in the RG analysis may
skip the following discussion and directly go to Sec. VI.

Figure 13:  Bare mean-field susceptibility [see Eq. (38)] for zero external
momentum and frequency [see Fig. 12 for the relevant Feynman diagram] for
various orderings at zero temperature and chemical doping, as a function of
$\alpha$ [parametrizing the anisotropy between the mass parameters in the
$T_{2g}$ and $E_{g}$ orbitals]. Here, $\chi$ is measured in units of
$m\Lambda$. For $\alpha=\frac{\pi}{4}$ two nematic orders and two $d$-wave
pairings (belonging to the $T_{2g}$ and $E_{g}$ representations) possess equal
susceptibilities, and so do two magnetic orders (within the $A_{2u}$ and
$T_{1u}$ representations). The $A_{1g}$ $s$-wave pairing always possesses the
largest susceptibility (for any $\alpha$) as it represents a _mass_ for
spin-3/2 fermions. Susceptibilities for the $s$-wave pairing, $T_{2g}$ nematic
and $A_{2u}$ magnetic orders display exact degeneracy as
$\alpha\to\frac{\pi}{2}$, when all of them become _mass_ [see Sec. V.1.2]. On
the other hand, as $\alpha\to 0$ the $s$-wave pairing and $E_{g}$ nematicity
become _mass_ and their bare susceptibilities are degenerate and largest [see
Sec. V.1.3]. For detailed discussion consult Sec. V.1.

###  V.1 Mean-field susceptibility

To gain insights into the propensity toward the formation of various
orderings, we first compute the bare mean-field susceptibility
($\chi_{{}_{M}}$) of all possible symmetry allowed fermionic bilinears
$\Psi_{\rm Nam}^{\dagger}M\Psi_{\rm Nam}$, where $M$ is an eight dimensional
Hermitian matrix (see Sec. IV). For simplicity we set $\mu=0$. For zero
external momentum and frequency this quantity is given by

$~{}\chi_{{}_{M}}=-\frac{1}{2}\int\frac{d^{3}{\bf
k}}{(2\pi)^{3}}\sum_{i\omega_{n}}{\bf Tr}\left[MG_{\bf k}(i\omega_{n})MG_{\bf
k}(i\omega_{n})\right].$ (38)

The relevant Feynman diagram is shown in Fig. 12 and the “$-$” sign arises
from the fermion bubble. Here $G_{\bf k}(i\omega_{n})$ is the fermionic
Green’s function in the Nambu doubled basis. The factor of $1/2$ takes care of
the artificial Nambu doubling. Results are displayed in Fig. 13. Next we
discuss the scaling of $\chi_{{}_{M}}$ in different channels for a few
specific values of the mass anisotropy parameter.

####  V.1.1  Isotropic Luttinger Semimetal ($\alpha=\frac{\pi}{4}$)

For $\alpha=\frac{\pi}{4}$, the effective masses for the $T_{2g}$ and $E_{g}$
orbitals are equal (i.e. $m_{1}=m_{2}$) and the system enjoys an enlarged
spherical symmetry. Since each one of the five $\Gamma$-matrices (representing
two nematic orders) anti-commutes with four matrices and commutes with one
matrix appearing in the Luttinger Hamiltonian, two nematic orders belonging to
the $T_{2g}$ (red curve) and $E_{g}$ (orange curve) representations possess
_equal_ susceptibility. On the other hand, all ten commutators (representing
various magnetic orders) anti-commute with three and commute with two matrices
appearing in this model. Hence, magnetic orders in the $A_{2u}$ (purple curve)
and $T_{1u}$ (magenta curve) channels also possess equal susceptibility. Two
copies of the $d$-wave pairing, transforming under the $T_{2g}$ (dark green
curve) and $E_{g}$ (dark yellow curve) representations, have degenerate
susceptibilities, as all five $d$-wave pairing matrices commute with four
matrices and anti-commute with only one matrix appearing in the Luttinger
model. As the $s$-wave pairing fully anti-commutes with the Luttinger
Hamiltonian, it always possesses the largest susceptibility (blue curve) for
any $\alpha$.

Susceptibilities for different BSPs (characterized by the fermion bilinear
$\Psi_{\rm Nam}^{\dagger}M\Psi_{\rm Nam}$) $\sim N^{M}_{\rm anti}$, the number
of matrices in $\hat{h}^{\rm Nam}_{\mathrm{L}}({\bf k})$ anti-commuting with
$M$. This simple correspondence is operative irrespective of the choice of
$\alpha$. Note that in an isotropic system $N_{\rm anti}^{M}=5,4,3$ and $1$
for the $s$-wave pairing, nematicity, magnetic orders and $d$-wave pairings,
respectively, hence

$\chi_{{}_{M}}\left({\alpha=\frac{\pi}{4}}\right):{s{\rm-wave}}>{\rm
nematic}>{\rm magnetic}>{d{\rm-wave}}.$

Computation of the bare susceptibility suggests a strong propensity toward the
formation of $s$-wave pairing and two nematic orders in the world of
interacting spin-3/2 fermions with isotropic dispersion. The magnetic orders
and $d$-wave pairings are expected to be suppressed near
$\alpha=\frac{\pi}{4}$, at least at zero temperature [see Figs. 3 and 16]. We
also note that the gain in free-energy [see Sec. IV.4] and the mean-field
susceptibility follow the same hierarchy $\Delta_{F},\chi_{{}_{M}}\sim
N^{M}_{\rm anti}$.

####  V.1.2 Anisotropic Luttinger Semimetal near $\alpha=\frac{\pi}{2}$

When the effective mass in the $T_{2g}$ orbital becomes sufficiently large,
the Luttinger model simplifies to

$\lim_{{\alpha\to\frac{\pi}{2}}}\hat{h}^{\rm Nam}_{\rm L}({\bf
k})=-\eta_{3}\>\frac{k^{2}}{2m_{2}}\sum^{5}_{j=4}\Gamma_{j}\hat{d}_{j}(\hat{{\bf
k}}).$ (39)

This Hamiltonian possesses an emergent $SU(2)\otimes U(1)$ chiral symmetry,
where $\\{\Gamma_{45}\Gamma_{j}\\}$ with $j=1,2,3$ are the three generators of
an $SU(2)$ rotation, whereas a $U(1)$ rotation is generated by
$\hat{N}=\eta_{3}\Gamma_{0}$, the number operator. In this limit the
Hamiltonian is similar to the one for spinless fermions in Bernal-stacked
bilayer graphene, which altogether supports _six_ masses, given by 101010When
an order parameter matrix fully anticommutes with the noninteracting
Hamiltonian, we coin it as mass order.

$~{}{\bf
M}_{\frac{\pi}{2}}=\left\\{\overbrace{\underbrace{\eta_{3}\left(\Gamma_{1},\Gamma_{2},\Gamma_{3}\right)}_{\mbox{$T_{2g}$
nematic}},\underbrace{\left(\eta_{1},\eta_{2}\right)\Gamma_{0}}_{\mbox{$s$-wave}}}^{\mbox{SO(5)
vector}},\underbrace{\eta_{0}\Gamma_{45}}_{\mbox{$A_{2u}$ magnet}}\right\\}.$
(40)

Note that three components of the $T_{2g}$ nematicity break the continuous
$SU(2)$ chiral symmetry, while the $s$-wave pairing breaks the global $U(1)$
symmetry. On the other hand, the $A_{2u}$ magnet transforms as a scalar under
the chiral rotation and breaks only time-reversal-symmetry. These three mass
orders possess the largest and equal susceptibilities as
$\alpha\to\frac{\pi}{2}$, see Fig. 13. Therefore, repulsive interactions favor
two excitonic masses for zero [see Figs. 16 and 16] and $s$-wave pairing [see
Fig. 6] for finite chemical doping.

Also note that each member of the following vector

$~{}{\bf
M}^{\prime}_{\frac{\pi}{2}}=\left\\{\overbrace{\underbrace{\eta_{3}\left(\Gamma_{4},\Gamma_{5}\right)}_{\mbox{$E_{g}$
nematic}},\underbrace{\left(\eta_{1},\eta_{2}\right)(\Gamma_{4},\Gamma_{5})}_{\mbox{$E_{g}$
d-wave}}}^{\mbox{multiplet of SO(3) vectors}}\right\\}.$ (41)

anti-commutes and commutes with one matrix appearing in
$\lim_{{\alpha\to\frac{\pi}{2}}}\hat{h}^{\rm Nam}_{\rm L}({\bf k})$. Hence,
$E_{g}$ nematicity and $d$-wave pairing have identical susceptibilities as
$\alpha\to\frac{\pi}{2}$, but

$\chi_{{}_{{\bf M}^{\prime}_{\frac{\pi}{2}}}}\><\>\chi_{{}_{{\bf
M}_{\frac{\pi}{2}}}}.$

As a result such an anisotropic system can accommodate an $E_{g}$ $d$-wave
pairing at finite chemical doping, see Fig. 6. However, $T_{1u}$ magnet and
$T_{2g}$ $d$-wave superconductor have exactly _zero_ susceptibility as they
_fully commute_ with $\lim_{{\alpha\to\frac{\pi}{2}}}\hat{h}^{\rm Nam}_{\rm
L}({\bf k})$. Hence, onset of these two orders is unlikely when
$\alpha\approx\frac{\pi}{2}$.

Figure 14:  (a) Bare four-fermion interaction vertex
$\left(\Psi^{\dagger}M\Psi\right)^{2}$, and (b) self-energy correction due to
four-fermion interaction. Note contribution from Feymann diagram (b) is finite
only when the chemical potential ($\mu$) is finite, and renormalizes $\mu$
(see Sec. VI). Feynman diagrams (c)-(f) yield corrections to the bare
interaction vertex to the leading order in the $\epsilon$-expansion, where
$\epsilon=d-2$. Here, solid lines represent fermions, and $M$ and $N$ are
four-dimensional Hermitian matrices. While the red lines in (b)-(f) correspond
to the fast modes (living within a thin Wilsonian shell
$\Lambda\;e^{-\ell}<|{\bf k}|<\Lambda$, where $\Lambda$ is the ultraviolet
momentum cut-off), the black lines are the slow modes with $|{\bf
k}|<\Lambda\;e^{-\ell}$. Recall $\ell$ is the logarithm of the renormalization
group scale.

####  V.1.3 Anisotropic Luttinger Semimetal near $\alpha=0$

Finally, we compare the susceptibility for various orders when the mass of the
$E_{g}$ orbital becomes sufficiently large. The Luttinger Hamiltonian then
takes the form 111111This Hamiltonian is quite similar to the one for three-
dimensional massless Dirac fermions, with the crucial difference that for the
Dirac Hamiltonian $\hat{d}_{j}(\hat{k})\sim k_{j}$, while for the Luttinger
Hamiltonian
$\hat{d}_{j}(\hat{k})\sim|\epsilon_{jlm}|\;\hat{k}_{l}\;\hat{k}_{m}$, where
$j,l,m=1,2,3$.

$~{}\lim_{\alpha\to 0}\hat{h}^{\rm Nam}_{\rm L}({\bf
k})=-\eta_{3}\>\frac{k^{2}}{2m_{1}}\sum^{3}_{j=1}\Gamma_{j}\hat{d}_{j}(\hat{{\bf
k}}),$ (42)

which possesses an emergent $U(1)\otimes U(1)$ symmetry, generated by
$\eta_{0}\Gamma_{45}$ and $\eta_{3}\Gamma_{0}$. The mass orders in this
limiting scenario constitute the following vector

$~{}{\bf
M}_{0}=\left\\{\overbrace{\underbrace{\eta_{3}\left(\Gamma_{4},\Gamma_{5}\right)}_{\mbox{$E_{g}$
nematic}},\underbrace{\left(\eta_{1},\eta_{2}\right)\Gamma_{0}}_{\mbox{$s$-wave}}}^{\mbox{SO(4)
vector}}\right\\}.$ (43)

and consequently the $E_{g}$ nematicity and $s$-wave pairing acquire an
identical and the largest susceptibility as $\alpha\to 0$, see Fig. 13. Hence,
a competition between these two ordered phases can be anticipated near
$\alpha=0$ [see Figs. 6 and 16]. Any order parameter from the following vector
anti-commutes with two matrices and commutes with one matrix appearing in Eq.
(42)

$~{}{\bf
M}^{\prime}_{0}=\left\\{\overbrace{\underbrace{\eta_{3}\left(\Gamma_{1},\Gamma_{2},\Gamma_{3}\right)}_{\mbox{$T_{2g}$
nematic}},\>\>\underbrace{\eta_{0}\Gamma_{45}\left(\Gamma_{1},\Gamma_{2},\Gamma_{3}\right)}_{\mbox{$T_{1u}$
magnet}}}^{\mbox{multiplet of SO(3) vectors}}\right\\},$ (44)

and they also possess degenerate susceptibilities, but

$\chi_{{}_{{\bf M}^{\prime}_{0}}}\><\>\chi_{{}_{{\bf M}_{0}}}.$

As a result, a competition between $T_{1u}$ magnet and $T_{2g}$ nematicity can
also be observed around $\alpha=0$, see Fig. 16. On the other hand, the
$T_{2g}$ $d$-wave pairing matrices anti-commute with one matrix and commute
with two matrices appearing in Eq. (42) and its susceptibility is smaller than
the orders appearing in ${\bf M}_{0}$ and ${\bf M}^{\prime}_{0}$. Nonetheless,
when assisted by finite chemical doping, the $T_{2g}$ $d$-wave pairing can be
realized even for repulsive magnetic interaction in the $T_{1u}$ channel, as
shown in Fig. 6. Finally, we note that $A_{2u}$ magnet and $E_{g}$ $d$-wave
pairing fully commute with $\lim_{\alpha\to 0}\hat{h}({\bf k})$ and possess
_zero_ susceptibility. Hence, onset of these two orders around $\alpha=0$ is
unlikely.

Figure 15:  (a) The bare vertex associated with the source term
$\Psi^{\dagger}_{\rm Nam}M\Psi_{\rm Nam}$. The leading order renormalization
of such vertices arises from Feynman diagrams (b) and (c), yielding the RG
flow of the source terms, displayed in Eq. (49). Here, wavy lines stand for
the source field, while solid lines for fermions, and the dashed lines for the
interaction vertex. The black (red) solid lines represent slow (fast) modes.

Note that various matrices appearing in ${\bf M}_{\frac{\pi}{2}}$, ${\bf
M}^{\prime}_{\frac{\pi}{2}}$, ${\bf M}_{0}$, ${\bf M}^{\prime}_{0}$ can form
composite order-parameters and the enlarged symmetries among distinct
orderings are displayed in Eqs. (40), (41), (43), (44). Such enlargement of
order-parameter vectors plays an important role in determining the confluence
of competing orders, which we discuss in Sec. VI.5.

##  VI Renomralization group analysis

After gaining insight into the propensity toward various orderings in the
Luttinger system, next we seek to investigate the onset of different BSPs and
the competition among them within the framework of an unbiased RG analysis.
This will allow us to go beyond the mean-field analysis, presented in the last
section, and systematically incorporate fluctuations. In what follows we here
restrict ourselves to the leading order in the $\epsilon$ expansion, where
$\epsilon=d-2$, and account for corrections to the bare interaction vertices
($\lambda_{j}$s) to quadratic order in the coupling constants. The relevant
Feynman diagrams are shown in Fig 14. After performing summation over
fermionic Matsubara frequencies $\omega_{n}=(2n+1)\pi T$ with $-\infty\leq
n\leq\infty$, we integrate out a thin Wilsonian shell $\Lambda e^{-\ell}<|{\bf
k}|<\Lambda$ to arrive at the following RG flow equations

$~{}\beta_{g_{{}_{i}}}=-\epsilon
g_{{}_{i}}+\sum^{2}_{j=0}g^{2}_{{}_{j}}H^{i}_{jj}(\alpha,t,\mu)+{\sum^{2}_{j,k=0}}^{\prime}g_{{}_{j}}g_{{}_{k}}H^{i}_{jk}(\alpha,t,\mu),$
(45)

for $i=0,1,2$, where $\beta_{X}\equiv dX/d\ell$, in terms of dimensionless
quantities defined as

$g_{{}_{i}}=\frac{m\lambda_{i}\Lambda^{\epsilon}}{4(2\pi)^{3}},\>\>\>t=\frac{2mT}{\Lambda^{2}},\>\>\>\tilde{\mu}=\frac{2m\mu}{\Lambda^{2}}.$
(46)

The prime symbol in the summation indicates that $j>k$. For notational clarity
we take $\tilde{\mu}\to\mu$, and the functions $H^{i}_{jk}(\alpha,t,\mu)$ for
$i,j,k=0,1,2$ are shown in Appendix E. Due to their lengthy expressions (which
are not very instructive in particular) we here only display the schematic
form of the flow equations. Both temperature and chemical potential also flow
under coarse graining [see first line of Eq. (9)] as relevant perturbations
with bare scaling dimensions $[t]=[\mu]=z$, where $z=2$ for the Luttinger
system. These two flow equations can then be solved, respectively yielding

$t(\ell)=t(0)\;e^{z\ell},\quad\mu(\ell)=\mu(0)\;e^{z\ell},$ (47)

where $t(0)$ and $\mu(0)$ are the bare values. We supply these solutions to
Eq. (45) to find the phase diagram of interacting Luttinger fermions using the
following prescription. 121212We should note that Feynman diagram (b) in Fig.
14 provides interaction driven corrections (linear in $g_{{}_{i}}$) to $\mu$.
However, to maintain the order by order correction to quartic interactions, we
neglect such corrections in Eq. (45) within the framework of the leading order
$\epsilon$-expansion.

Coup. | $\alpha=\frac{\pi}{4}$ | $\alpha=1.5$ | $\alpha=0.04$
---|---|---|---
QCP${}^{1}_{\frac{\pi}{4}}$ | QCP${}^{1}_{\frac{\pi}{2}}$ | QCP${}^{2}_{\frac{\pi}{2}}$ | BCP${}_{\frac{\pi}{2}}$ | QCP${}^{1}_{0}$ | QCP${}^{2}_{0}$ | BCP0
$g^{\ast}_{{}_{0}}\times 10^{3}$ | $2.24$ | $1.49$ | $-2.93$ | $-1.58$ | $1.95$ | $-5.98$ | $-5.60$
$g^{\ast}_{{}_{1}}\times 10^{3}$ | $2.03$ | $1.47$ | $-2.84$ | $-1.33$ | $1.71$ | $4.80$ | $4.97$
$g^{\ast}_{{}_{2}}\times 10^{3}$ | $2.03$ | $1.26$ | $3.76$ | $5.54$ | $1.94$ | $-5.87$ | $-5.48$
Table 2: Locations of quantum critical points (QCPs) possessing one unstable
direction and bi-critical points (BCPs) possessing two unstable directions for
three specific choices of the mass anisotropy parameter $\alpha$ [see Eq.
(17)]. Note for $\alpha=\frac{\pi}{4}$ the Luttinger model possesses an
emergent spherical symmetry, while for $\alpha\to 0\;(\frac{\pi}{2})$ the band
dispersion in the $E_{g}(T_{2g})$ orbitals becomes almost flat. For
$\alpha=\frac{\pi}{4}$ the coupled RG flow equations support only one QCP,
denoted by QCP${}^{1}_{\frac{\pi}{4}}$, while for $\alpha\to\frac{\pi}{2}$ and
$\alpha\to 0$ we find two QCPs, respectively denoted by
QCP${}^{j}_{\frac{\pi}{2}}$ and QCP${}^{j}_{0}$ for $j=1,2$, see also Ref.
Herbut-2 . For these two limiting cases the flow equations also support one
BCP, respectively identified as BCP${}_{\frac{\pi}{2}}$ and BCP0. Existence of
such BCP in the presence of two QCPs is necessary to ensure the continuity of
the RG flow trajectories and separate the basins of attraction of two QCPs.
All coupling constants at the fixed points are measured in units of
$\epsilon$, where $\epsilon=d-2$ measures the deviation from the lower
critical two spatial dimensions, where all local quartic interactions are
_marginal_. Note that QCP${}^{1}_{\frac{\pi}{4}}$,
QCP${}^{1}_{\frac{\pi}{2}}$, QCP${}^{1}_{0}$ represent the same QCP, only its
location shifts as we tune $\alpha$. By contrast, QCP${}^{2}_{\frac{\pi}{2}}$
and QCP${}^{2}_{0}$ are solely introduced by mass anisotropy and bear no
analog to the fixed points found around $\alpha=\pi/4$. Besides the above QCPs
and the BCPs, there always exists a trivial Gaussian fixed point, representing
the non-interacting Luttinger semimetal, at
$(g^{\ast}_{{}_{0}},g^{\ast}_{{}_{1}},g^{\ast}_{{}_{2}})=(0,0,0)$, endowed
with three stable directions. See Appendix F for details.

While the _divergence_ of at least one of the quartic couplings (i.e.,
$g_{{}_{i}}\to+\infty$) under coarse graining indicates the onset of a BSP, to
unambiguously determine the pattern of symmetry breaking we also account for
the leading order RG flow of all source terms, associated with different BSPs,
discussed in Sec. IV. The effective action in the presence of all symmetry
allowed fermionic bilinears [see Eqs. (24)-(27)] is given by

$\displaystyle~{}S_{s}=\int d\tau\;d^{3}{\bf r}\;\Psi^{\dagger}_{\rm
Nam}\;\left(\hat{h}^{\rm exc}_{\rm local}+\hat{h}^{\rm pair}_{\rm
local}\right)\;\Psi_{\rm Nam},$ (48)

with $\Psi^{\dagger}_{\rm Nam}\equiv\Psi^{\dagger}_{\rm Nam}(\tau,{\bf x})$
and $\Psi_{\rm Nam}\equiv\Psi_{\rm Nam}(\tau,{\bf x})$. Relevant Feynman
diagrams are shown in Fig. 15. The resulting RG flow equations take the
following schematic form

$~{}\frac{d\ln\Delta_{i}}{d\ell}-2=\sum^{2}_{j=0}F^{j}_{i}\left(\alpha,t,\mu\right)\;g_{j}.$
(49)

See Appendix E for explicit form of these flow equations. The quantities
appearing on the right hand side of each equation, represent the _scaling
dimension_ of the corresponding order-parameter.

Sr | $\alpha=\frac{\pi}{4}$ | $\alpha=1.5$ | $\alpha=0.04$
---|---|---|---
QCP${}^{1}_{\frac{\pi}{4}}$ | QCP${}^{1}_{\frac{\pi}{2}}$ | QCP${}^{2}_{\frac{\pi}{2}}$ | BCP${}_{\frac{\pi}{2}}$ | QCP${}^{1}_{0}$ | QCP${}^{2}_{0}$ | BCP0
$\Delta_{0}$ | 0 | 0 | 0 | 0 | 0 | 0 | 0
$\Delta_{1}$ | _0.426_ | _0.531_ | -0.238 | 0.378 | 0.353 | _0.614_ | 0.656
$\Delta_{2}$ | _0.426_ | 0.275 | _0.325_ | 0.678 | _0.539_ | -0.153 | -0.070
$\Delta_{3}$ | -0.076 | -0.204 | 1.147 | 1.006 | -0.004 | -0.024 | -0.024
$\Delta_{4}$ | -0.076 | -0.007 | -0.021 | -0.030 | -0.134 | 0.731 | 0.699
$\Delta_{5}$ | -0.076 | -0.094 | 0.182 | 0.092 | -0.062 | 0.020 | 0.011
$\Delta^{\rm p}_{A_{1g}}$ | 0.551 | 0.545 | -0.254 | 0.355 | 0.547 | -0.166 | -0.083
$\Delta^{\rm p}_{T_{2g}}$ | -0.034 | -0.004 | -0.011 | -0.016 | -0.059 | 0.016 | 0.006
$\Delta^{\rm p}_{E_{g}}$ | -0.034 | -0.088 | 0.169 | 0.073 | -0.002 | -0.012 | -0.012
Table 3:  Scaling dimensions (in units of $\epsilon$) of various source (Sr)
terms or fermion bilinears at different fixed points (reported in Table 2),
obtained by substituting fixed point values of the coupling constants
$g^{\ast}_{{}_{0}}$, $g^{\ast}_{{}_{1}}$ and $g^{\ast}_{{}_{2}}$ on the right-
hand side of the corresponding flow equation [see Eq. (49)] at $t=0$ and
$\mu=0$. At each QCP the largest scaling dimension is shown in bold, while the
second largest ones are shown in italic. At the two magnetic QCPs
(QCP${}^{2}_{\frac{\pi}{2}}$ and QCP${}^{2}_{0}$), the largest scaling
dimensions for the superconducting channel (namely $d$-wave pairings) are
shown in blue.

We simultaneously run the flow of the quartic couplings ($g_{{}_{j}}$s) and
the source terms ($\Delta_{j}$s). When at least one of the quartic couplings
_diverges_ and flows toward $\to+\infty$ (thus indicating onset of a BSP), we
identify the source term (say $\Delta_{j}$) that diverges toward $\to+\infty$
_fastest_ (assuming a possible scenario when more than one source term diverge
toward $+\infty$). The BSP is then characterized by the order-parameter
$\Delta_{j}\neq 0$. We use this strategy to determine various cuts of the
global phase diagram of spin-3/2 Luttinger fermions, displayed in Fig. 16 (for
$t=\mu=0$), Fig. 3 (for $\mu=0$, but finite-$t$) and Figs. 4, 5 and 6 (for
finite-$t$ and finite-$\mu$). Next we discuss these cases in three subsequent
sections.

###  VI.1 Quantum criticality in Luttinger semimetal

We first discuss the effects of electronic interactions on a LSM (i.e. when
the chemical potential is pinned at the band touching point) at zero
temperature. The RG flow equations for $\mu=0$ and $t=0$ can be derived by
taking the limit $\mu\to 0$ and then $t\to 0$ in Eq. (45), suggesting that
weak interactions are irrelevant perturbations and any ordering takes place at
finite coupling $g_{{}_{i}}\sim\epsilon$ through a QPT. Next we discuss the
following three cases separately (i) isotropic Luttinger system
($\alpha=\frac{\pi}{4}$), large mass for (ii) the $T_{2g}$ orbital
($\alpha\to\frac{\pi}{2}$) and (iii) the $E_{g}$ orbital ($\alpha\to 0$). Such
systematic analysis will allow us to anchor our anticipations regarding the
nature of BSPs from the mean-field susceptibility, discussed in Sec. V.1. Note
at $t=\mu=0$, the system is devoid of any natural infrared cutoff as
$\ell^{t}_{\ast},\ell^{\mu}_{\ast}\to\infty$. Hence, we run the flows of
quartic couplings up to an RG time $\ell_{\ast}\to\infty$ to determine the
stability of LSM and the flows of the source terms to pin the pattern of
symmetry breaking.

####  VI.1.1  Isotropic Luttinger semimetal ($\alpha=\frac{\pi}{4}$)

For $\alpha=\frac{\pi}{4}$ the coupled RG flow equations support only one QCP,
reported in Table 2 and identified as QCP${}^{1}_{\frac{\pi}{4}}$, besides the
trivial (and fully stable) Gaussian fixed point at
$g^{\ast}_{{}_{0}}=g^{\ast}_{{}_{1}}=g^{\ast}_{{}_{2}}=0$ (representing the
stable LSM). This QCP controls QPTs from LSM to various BSPs (depending on the
interaction channel). To gain insight into the nature of the candidate
competing BSPs, we compute the scaling dimensions for all fermion bilinears at
this QCP. The results are summarized in Table 3. Note that the $s$-wave
pairing has the largest scaling dimension at this QCP, while two nematic
orders possess degenerate but second largest (and positive) scaling
dimensions. But, the rest of the fermion bilinears possess _negative_ scaling
dimensions. Notice that the scaling dimensions for different orders at this
QCP follow the same hierarchy as the mean-field susceptibilities, discussed in
Sec. V.1.1.

The phase diagrams in an interacting LSM are displayed in Fig. 3 for various
interaction channels. Around $\alpha=\frac{\pi}{4}$ strong repulsive nematic
interactions ($g_{{}_{1}}$ and $g_{{}_{2}}$) favor $s$-wave pairing even in
the absence of a Fermi surface (since $\mu=0$), see Figs. 3 and 3. On the
other hand, strong magnetic interactions in the $A_{2u}$ ($g_{{}_{3}}$) and
$T_{1u}$ ($g_{{}_{4}}$) channels respectively support $T_{2g}$ [see Fig. 3]
and $E_{g}$ [see Fig. 3] nematicities. However, we could not find any magnetic
ordering or $d$-wave pairing in the very close vicinity to
$\alpha=\frac{\pi}{4}$, at least when $t=0$, see Fig. 16. Therefore, the
computation of mean-field susceptibilities and scaling dimensions of fermion
bilinears, in corroboration with our unbiased RG calculation, show that the
strongly interacting isotropic LSM becomes unstable toward the formation of
two nematic orders and $s$-wave pairing at the lowest temperature.

Figure 16: Specific cuts of the global phase diagram of an interacting
Luttinger semimetal at zero temperature, obtained from an RG analysis [see
Sec. VI.1]. Here $\alpha$ is the mass anisotropy parameter. For
$\alpha=\frac{\pi}{4}$ the system possesses an enlarged spherical symmetry,
whereas for $\alpha=0$ and $\frac{\pi}{2}$ respectively the $E_{g}$ and
$T_{2g}$ orbital possess _infinite_ mass [see Sec. III]. Here $g_{{}_{1}}$ and
$g_{{}_{2}}$ respectively capture the strength of repulsive interactions in
the $T_{2g}$ and $E_{g}$ nematic channels. Magnetic interactions in the
$A_{2u}$ and $T_{1u}$ channels are respectively denoted by $g_{{}_{3}}$ and
$g_{{}_{4}}$. Notice that even in the absence of a Fermi surface, pure
repulsive nematic interactions are conducive to $s$-wave pairing among
spin-3/2 fermions at zero temperature [see panels (a) and (b)].

####  VI.1.2 Anisotropic Luttinger semimetal: $\alpha\to\frac{\pi}{2}$

Next we turn our focus to the vicinity of $\alpha=\frac{\pi}{2}$. The coupled
flow equations then support _two_ QCPs [denoted by QCP${}^{1}_{\frac{\pi}{2}}$
and QCP${}^{2}_{\frac{\pi}{2}}$] and _one_ bi-critical point [denoted by
BCP${}_{\frac{\pi}{2}}$], see Table 2. The BCP possesses _two_ unstable
directions. Notice QCP${}^{1}_{\frac{\pi}{2}}$ is the same as
QCP${}^{1}_{\frac{\pi}{4}}$, only shifted toward weaker coupling, which can be
verified from the fact that the signs of the coupling constants and scaling
dimensions for all fermion bilinears are _identical_ at these two QCPs [see
Tables 2 and 3].

On the other hand, QCP${}^{2}_{\frac{\pi}{2}}$ is new and bears no resemblance
to any fixed points we found for $\alpha=\frac{\pi}{4}$. This QCP is induced
by the mass anisotropy of Luttinger fermions. At this QCP the $A_{2u}$
magnetic ($E_{g}$ nematic) order possesses the largest (second largest)
scaling dimension, see Table 3. Therefore, when repulsive interaction in the
$A_{2u}$ channel dominates among various finite range components of the
Coulomb interaction, the Luttinger semimetal can display a competition between
these two orders as we approach $\alpha=\frac{\pi}{2}$ from an isotropic
system, see Fig. 16 [consult also Sec. VI.5].

Among three possible local pairings the $E_{g}$ $d$-wave superconductor
possesses the _largest_ (and positive) scaling dimension at this QCP. Hence,
the emergence of this paired state can be anticipated when the LSM is doped
away from the charge-neutrality point around $\alpha=\frac{\pi}{2}$, see Fig.
6 [consult Sec. VI.4 for details].

####  VI.1.3 Anisotropic Luttinger semimetal: $\alpha\to 0$

Finally, we approach the opposite limit, when the mass in the $E_{g}$ orbital
becomes sufficiently large, i.e. $m_{2}\gg m_{1}$ or equivalently $\alpha\to
0$. In this regime the RG flow equations support two QCPs [denoted by
QCP${}^{1}_{0}$ and QCP${}^{2}_{0}$] and a BCP [denoted by BCP0], see Table 2.
Note that QCP${}^{1}_{0}$ is similar to QCP${}^{1}_{\frac{\pi}{4}}$, only
shifted toward weaker coupling. By contrast, QCP${}^{2}_{0}$ is induced by the
mass anisotropy. At this QCP, the $T_{1u}$ magnetic ($T_{2g}$ nematic) order
possesses the largest (second largest) scaling dimension, see Table 3.
Therefore, when repulsive interaction in the $T_{1u}$ channel dominates we
expect a strong competition between these two orderings, as one tunes toward
$\alpha\to 0$ starting from an isotropic system, see Fig. 16.

Also note that among three local pairings, the $d$-wave one transforming under
the $T_{2g}$ representation possesses the largest (and positive) scaling
dimension. Hence, around $\alpha=0$ we expect onset of this paired state when
the chemical potential is placed away from the band touching point, see Fig. 6
and discussion in Sec. VI.4.

Finally, we comment on the role of the BCPs possessing two unstable
directions, see Table 2. Note that a BCP can only be found when there exists
two QCPs in the three dimensional coupling constant space
$\left(g_{{}_{0}},g_{{}_{1}},g_{{}_{2}}\right)$. The existence of a BCP
separates the basin of attraction of two QCPs and ensures continuity of the RG
flow trajectories in coupling constant space.

###  VI.2 Universality class and critical exponents

All QCPs, listed in Table 2, are characterized by only _one_ unstable
direction or _positive_ eigenvalue of the corresponding _stability matrix_
[see Appendix F], defined as

$M_{ij}\left(g_{{}_{0}},g_{{}_{1}},g_{{}_{2}}\right)=\frac{d}{dg_{{}_{j}}}\beta_{g_{{}_{i}}}.$
(50)

To the leading order in the $\epsilon$-expansion the positive eigenvalue at
all QCPs is exactly $\epsilon$, which in turn determines the _correlation
length exponent_ ($\nu$) according to

$\nu^{-1}=\epsilon+{\mathcal{O}}\left(\epsilon^{2}\right).$ (51)

For the physically relevant situation $\epsilon=1$ and we obtain $\nu=1$. The
fact that $\nu$ is the same at all QCPs is, however, only an artifact of the
leading order $\epsilon$-expansion. Generically $\nu$ is expected to be
distinct at different QCPs, once we account for higher order corrections in
$\epsilon$.

Since to the leading order in the $\epsilon$-expansion, local interactions do
not yield any correction to fermion self-energy, the _dynamic scaling
exponent_ ($z$) at all interacting QCPs is

$z=2+{\mathcal{O}}\left(\epsilon\right).$ (52)

Together the correlation length and dynamic scaling exponents determine the
universality class of all continuous QPTs from a LSM to various BSPs. In the
next section, we will discuss the imprint of these two exponents on the
scaling of the transition temperature ($t_{c}$) associated with the finite
temperature order-disorder transitions.

###  VI.3 Interacting Luttinger Semimetal at Finite Temperature

Figure 17: Scaling of dimensionless transition temperature ($t_{c}$) with the
reduced distance
($\delta_{j}=(g_{{}_{j}}-g^{\ast}_{{}_{j}})/g^{\ast}_{{}_{j}}$) from the
quantum critical point, located at $g_{{}_{j}}=g^{\ast}_{{}_{j}}$ for
$j=1,2,3,4$, see Sec. VI.3. The mass anisotropy parameter $\alpha$ and nature
of the ordered phase for $\delta_{j}>0$ in each panel are quoted explicitly.
Note that $t_{c}\sim\delta^{2}_{j}$ for $\delta\ll 1$, in agreement with field
theoretic prediction. The QCPs controlling the Luttinger semimetal to ordered
states quantum phase transitions at $t=0$ are quoted in each panel. Red dots
represent numerically obtained transition temperature and the blue lines
correspond to least-square fit of $t_{c}\sim\delta^{2}_{j}$.

Even though our RG analysis in the previous section was performed at zero
temperature, one can still find the imprint of various interacting QCPs at
finite temperatures. The RG flow equations at finite temperature can be
derived from Eq. (45) by taking the $\mu\to 0$ limit in
$H^{l}_{jk}(\alpha,t,\mu)$. Recall that temperature introduces a natural
_infrared_ cutoff for the flow equations $\ell^{t}_{\ast}$ [see Eq. (9)].
Physically such an infrared cutoff corresponds to a scenario when the
renormalized temperature $t(\ell)$ becomes comparable to the ultraviolet
energy $E_{\Lambda}=\Lambda^{2}/(2m)$ [see Fig. 1], beyond which the notion of
quadratically dispersing fermions becomes moot and the flow equations from Eq.
(45) lose their jurisdiction.

To capture the effects of electronic interactions in a LSM at finite-$t$, we
run three quartic coupling constants up to a scale $\ell\leq\ell^{t}_{\ast}$.
Now depending on the bare strength of interactions, two situations arise

$\text{(a)}\>\>\>g\left(\ell^{t}_{\ast}\right)<1,\quad{\rm
or}\quad\text{(b)}\>\>\>g\left(\ell^{t}_{\ast}\right)>1,$

respectively representing a disordered LSM (without any long-range ordering)
or onset of a BSP at finite temperature. Hence, for a given strength of
interaction $g>g_{\ast}$, where $g_{\ast}$ is the requisite critical strength
of interaction for a BSP at $t=0$, we always find a temperature $t_{c}$ above
(below) which the BSP disappears (appears). We identify $t_{c}$ as the
_critical or transition temperature_. None of the coupling constants diverge
for $t>t_{c}$. All ordered phases can display true long-range order at finite
$t$ in three dimensions and $t_{c}$ corresponds to a genuine transition
temperature.

General scaling theory suggests that the transition temperature scales as
sondhi-RMP ; sachdev-book

$t_{c}\sim\delta^{\nu z},$ (53)

for $\delta\ll 1$, where $\delta=\left(g-g_{\ast}\right)/g_{\ast}$ is the
_reduced distance_ from a QCP, located at $g=g_{\ast}$ (say). Hence, for
interacting LSM, $t_{c}\sim\delta^{2}$ for $\epsilon=1$ (i.e., the prediction
from the leading order $\epsilon$-expansion). The scaling of critical or
transition temperature for various choices of coupling constants and resulting
BSPs, and different choices of the mass anisotropy parameter $\alpha$, are
shown in Fig. 17, indicating a fairly good agreement with the field theoretic
prediction $t_{c}\sim\delta^{2}$ around all QCPs, reported in Table 2.

####  VI.3.1 Phase diagrams at finite temperature

Besides the scaling of the transition temperature, we also investigate the
phase diagram of an interacting LSM at finite temperature, allowing us to
demonstrate the competition between condensation energy gain and entropy. For
concreteness we focus on the isotropic system ($\alpha=\frac{\pi}{4}$), where
this competition is most pronounced. As argued in Sec. IV.4, the onset of
$s$-wave pairing leads to the maximal gain in condensation energy, while the
two nematic orders produce higher entropy in comparison to the former. Two
specific cuts of the global phase diagram, Figs. 3 and 3, show that while
$s$-wave pairing is realized at low temperature, nematicities set in at higher
temperature as we increase the strength of nematic interactions ($g_{{}_{1}}$
and $g_{{}_{2}}$) in the system.

By contrast, when we tune the magnetic interactions (namely $g_{{}_{3}}$,
$g_{{}_{4}}$), an isotropic LSM becomes unstable in favor of two nematic
orders at $t=0$. Such an outcome can be substantiated from the simple picture
of condensation energy gain, as $A_{2u}$ and $T_{1u}$ magnetic orders
accommodate Weyl nodes (yielding more entropy), while nematicities produce
anisotropic spectral gap (leading to higher gain of condensation energy), see
Sec. IV.4. As we tune the strength of the $A_{2u}$ ($T_{1u}$) magnetic
interaction, $E_{g}$ ($T_{2g}$) nematic order sets in at lower and $A_{2u}$
($T_{1u}$) magnet at higher temperature, see Figs. 3 and 3.

The LSM is endowed with the largest entropy in the global phase diagram of
interacting spin-3/2 fermions, since $\varrho(E)\sim\sqrt{E}$, in comparison
to any BSP. Consequently, the requisite strength of interactions for any
ordering increases with increasing temperature, irrespective of the nature of
the BSP, see Fig. 3. Therefore, our RG analysis for an interacting LSM at
finite temperature corroborates the energy-entropy competition picture and
substantiates the following outcome: _ordered phases providing larger
condensation energy gain are found at lower temperature, while at higher
temperature, phases with larger entropy are favored_. This observation is also
consistent with the notion of reconstructed band structure and emergent
topology, discussed in Sec. IV.3. Therefore, the phase diagram of an
interacting LSM at finite temperature is guided by topological structure
(gapped or nodal) of competing BSPs.

We close this section by answering the following question: Why do we find two
magnetic orders in an isotropic LSM at finite temperature, since this system
supports only one QCP, see Table 2, where all the magnetic orders bear
negative scaling dimension (see Table 3)? Note that any QCP can only be
accessed at $t=0$, whereas finite-$t$ introduces an infrared cutoff
($\ell^{t}_{\ast}$) for the RG flow of the quartic couplings, and thus
prohibits a direct access to any QCP. Hence, at finite-$t$, when magnetic
interactions are sufficiently strong, the system can bypass the basin of
attraction of QCP${}^{1}_{\frac{\pi}{4}}$ and nucleate magnetic phases at
moderately high temperature.

###  VI.4 RG analysis in Luttinger metal

Finally we proceed to the RG analysis when the chemical potential ($\mu$) is
placed away from the band touching point. The chemical potential introduces
yet another infrared cutoff $\ell^{\mu}_{\ast}$ [see Eq. (10)], suggesting
that the RG flow equations of the three quartic coupling constants should be
stopped when the renoramlized chemical potential $\mu(\ell)$ reaches the scale
of the band-width $E_{\Lambda}=\Lambda^{2}/(2m)$. At finite temperature and
chemical doping, two infrared scales compete and the _smaller_ one
$\ell_{\ast}$ (say), given by

$\ell_{\ast}={\rm min.}\;\left(\ell^{\mu}_{\ast},\ell^{t}_{\ast}\right)$ (54)

determines the ultimate infrared cutoff for the RG flow of $g_{{}_{j}}$. Now
depending on the bare strength of the coupling constants one of the following
two situations arises

$\text{(a)}\>\>\>g_{j}(\ell_{\ast})>1\quad{\rm
or}\quad\text{(b)}\>\>\>g_{j}(\ell_{\ast})<1.$

While (a) indicates onset of a BSP, (b) represents a stable Luttinger metal.
The resulting phase diagrams for various choices of chemical potential,
temperature, coupling constants and the mass anisotropy parameter ($\alpha$)
are shown in Figs. 4, 5 and 6.

We note that in an isotropic system and for strong enough nematic interactions
an $s$-wave pairing can be realized even for zero chemical doping [see Figs. 3
and 3]. With increasing doping the $s$-wave pairing occupies larger portion of
the phase diagram, while two nematic phases get pushed toward stronger
coupling, see Fig. 4. However, the $d$-wave pairings do not set in for zero
chemical doping. Nonetheless, when the magnetic interactions in the $A_{2u}$
and $T_{1u}$ channels are strong, the presence of a Fermi surface is conducive
to the nucleation of $d$-wave pairings, belonging to the $E_{g}$ and $T_{2g}$
representations, respectively, see Fig. 5.

Now we focus on the anisotropic system. We chose the mass anisotropy parameter
$\alpha$ such that at zero chemical doping the repulsive electronic
interactions accommodate either nematic or magnetic orders, see Fig. 16.
Specifically for $\alpha=1.5$ (close to $\frac{\pi}{2}$) the system enters
into the $T_{2g}$ nematic [see Fig. 6] or $A_{2u}$ magnetic [see Fig. 6]
phase, while for $\alpha=0.1$ (close to $0$) we find $E_{g}$ nematic [see Fig.
6] or $T_{1u}$ magnetic [see Fig. 7] order. For such specific choices of
$\alpha$, the QPTs into $T_{2g}$ nematic, $E_{g}$ nematic, $A_{2u}$ magnetic
and $T_{1u}$ magnetic orders are respectively controlled by QCP${}^{1}_{0}$,
QCP${}^{1}_{\frac{\pi}{2}}$, QCP${}^{2}_{\frac{\pi}{2}}$ and QCP${}^{2}_{0}$
[see Sec. VI.1 and Table 2]. At QPT${}^{1}_{0}$ and
QPT${}^{1}_{\frac{\pi}{2}}$ the $s$-wave pairing possesses the largest scaling
dimension among all possible local pairings [see Table 3]. Hence, in the
presence of finite chemical doping, repulsive interactions in the nematic
channels become conducive to $s$-wave pairing, as shown in Figs. 6 and 6. On
the other hand, at QPT${}^{2}_{\frac{\pi}{2}}$ (QCP${}^{2}_{0}$), the $d$-wave
pairing belonging to the $E_{g}$ ($T_{2g}$) representation possesses the
largest scaling dimension. Therefore, repulsive interactions in the $A_{2u}$
($g_{{}_{3}}$) and $T_{1u}$ ($g_{{}_{4}}$) magnetic channels become conducive
to the nucleation of $E_{g}$ [for $\alpha=1.5$] and $T_{2g}$ [for
$\alpha=0.1$] $d$-wave pairings, respectively, as shown in Figs. 6 and 6. We
conclude that _nematic and magnetic interactions among spin-3/2 Luttinger
fermions are respectively conducive to the $s$-wave and $d$-wave pairings_.
Otherwise, at finite chemical doping the excitonic orderings set in only for
stronger couplings. Such a generic feature is also consistent with the energy-
entropy competition picture as the superconducting phases maximally gap the
Fermi surface (yielding optimal gain of condensation energy), while exitonic
orders are accompanied by a Fermi surface with constant DoS (producing more
entropy).

Figure 18: Schematic representations of SU(2) symmetry among (a) $E_{g}$
nematic and $A_{2u}$ magnetic orders, and (b)-(c) components of $T_{2g}$
nematic and $T_{1u}$ magnetic orders (with $i\neq j\neq k=1,2,3$). Three
vertices of each triangle are occupied by the order parameter matrices (in
red), represented in the Nambu doubled basis ($\Psi_{\rm Nam}$). Three sides
of each triangle represent SU(2) rotations. The generators of SU(2) rotations
(also in Nambu doubled basis) are shown in blue.

###  VI.5 Competing Orders and Selection Rule

So far we have presented an extensive analysis of the role of electronic
interactions among spin-3/2 fermions. We showed multiple cuts of the global
phase diagram at zero and finite temperature and chemical doping (see Figs. 3,
4 and 5). In addition, we also addressed the imprint of the quadrupolar
deformation (or the mass anisotropy parameter $\alpha$) on a such phase
diagram (see Figs. 6, 7 and 16). Altogether we unearth a rich confluence of
competing orders in this system. In this context an important question arises
quite naturally: _Is there a selection rule among short-range interactions in
different channels (such as nematic and magnetic) and various ordered states
(such as $s$\- and $d$-wave pairings, magnetic phases and nematic orders) for
interacting spin-3/2 fermions?_ In this section we attempt to provide an
affirmative (at least partially) answer to this question.

To this end it is convenient to express the quartic terms appearing in $H_{\rm
int}$ [see Eq. (V)] in the Nambu doubled basis. For concreteness, we focus on
the relevant interaction channels, namely $\lambda_{1,2,3,4}$. In the Nambu
basis these four quartic terms take the form

$\displaystyle~{}\left\\{\begin{array}[]{c}\sum^{3}_{j=1}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}\\\
\sum^{5}_{j=4}\left(\Psi^{\dagger}\Gamma_{j}\Psi\right)^{2}\\\
\left(\Psi^{\dagger}\Gamma_{45}\Psi\right)^{2}\\\
\sum^{3}_{j=1}\left(\Psi^{\dagger}\Gamma_{j}\Gamma_{45}\Psi\right)^{2}\\\
\end{array}\right\\}$ (59)
$\displaystyle\to\frac{1}{2}\left\\{\begin{array}[]{c}\sum^{3}_{j=1}\left(\Psi^{\dagger}_{\rm
Nam}\eta_{3}\Gamma_{j}\Psi_{\rm Nam}\right)^{2}\\\
\sum^{5}_{j=4}\left(\Psi^{\dagger}_{\rm Nam}\eta_{3}\Gamma_{j}\Psi_{\rm
Nam}\right)^{2}\\\ \left(\Psi^{\dagger}_{\rm Nam}\eta_{0}\Gamma_{45}\Psi_{\rm
Nam}\right)^{2}\\\ \sum^{3}_{j=1}\left(\Psi^{\dagger}_{\rm
Nam}\eta_{0}\Gamma_{j}\Gamma_{45}\Psi_{\rm Nam}\right)^{2}\\\
\end{array}\right\\}.$ (64)

The factor of $1/2$ takes care of the artificial Nambu doubling. The above
question can then be reformulated in the following way. When at least one of
the coupling constants, say $\lambda_{j}$, diverges toward $+\infty$, what is
the nature of the resulting BSP; or which one of the source terms, appearing
in Eq. (48), simultaneously diverges toward $+\infty$? This is our quest in
this section.

Figure 19: Schematic representation of SU(2) symmetry between the $A_{2u}$
magnetic order and $d$-wave pairing belonging to the $E_{g}$ representation.
Notations are the same as in Fig. 18. In panel (a) $j=1,2$, whereas in panel
(b) $j=4,5$.

When the coupling constant $\lambda$ of the quartic interaction
$\lambda\left(\Psi^{\dagger}_{\rm Nam}\;M^{\rm Int}_{{\rm Nam}}\;\Psi_{\rm
Nam}\right)^{2}$ diverges toward $+\infty$ it nucleates a BSP, characterized
by the order-parameter $\langle\Psi^{\dagger}_{\rm Nam}\;M_{\rm or}\;\Psi_{\rm
Nam}\rangle$, only if one of the following two conditions is satisfied

$~{}{\rm(1)}\>\>M_{\rm or}=M^{\rm Int}_{{\rm Nam}}\quad{\rm
or}\quad{\rm(2)}\>\>\left\\{M_{\rm or},M^{\rm Int}_{{\rm Nam}}\right\\}=0.$
(65)

If, on the other hand, $M^{\rm Int}_{{\rm Nam}}$ and $M_{\rm or}$ have more
than one component, then selection rule (2) requires a slight modification,
see footnote 6.

The above selection rule can be justified from the Feynman diagrams shown in
Fig. 15. A source term diverges toward $+\infty$ (indicating onset of a BSP)
only when the net contribution from diagrams (b) and (c) is _positive_.
Feynman diagram (b) gives non-zero and _positive_ contribution only when
condition (1) from Eq. (65) is satisfied (due to the ${\rm\bf Tr}$ arising
from the fermion bubble). Even though diagram (c) then yields a negative
contribution, all togther they still produce a _positive definite_ quantity,
since the contribution from (b) dominates over (c), as the former one involves
${\rm\bf Tr}$. Hence, when condition (1) is satisfied, interaction $\lambda$
can enhance the propensity toward the formation of a BSP with $M_{\rm
or}=M^{\rm Int}_{{\rm Nam}}$.

On the other hand, when condition (1) is not satisfied, only Feynman diagram
(c) contributes [due to ${\rm\bf Tr}$ involved in (b)]. The contribution from
this diagram is _positive_ only when condition (2) is satisfied. By contrast,
when $M_{\rm or}\neq M^{\rm Int}_{{\rm Nam},j}$ and $\left[M_{\rm or},M^{\rm
Int}_{{\rm Nam},j}\right]=0$ the net contribution from (b) and (c) is
_negative_. Interaction coupling $\lambda$ then does not support such an
ordered phase. All phase diagrams presented in this work support one of these
two selection rules (discussed below).

The above selection rule can be phrased in a slightly different fashion. Two
ordered phases respectively breaking $O(M_{1})$ and $O(M_{2})$ symmetries,
131313 In this notation a $Z_{2}$ symmetry breaking order-parameter is denoted
by an $O(1)$ vector. can reside next to each other (when we tune the strength
of a particular interaction channel) if they constitute an $O(M)$ symmetric
_composite_ order parameter, where

$~{}M_{1},M_{2}\leq M\leq M_{1}+M_{2}.$ (66)

We invite the readers to verify the equivalence between Eqs. (65) and (66).
Next, we discuss some prototypical examples to support our claims.

Figure 20: Schematic representation of SU(2) symmetry between the components
of the $T_{1u}$ magnetic order and $T_{2g}$ $d$-wave pairing, with $i\neq
j\neq k=1,2,3$ [for (a), (b), (c)] and $m=1,2$ in (a) and (c). Notations are
the same as in Fig. 18.

  1. 1.

The $T_{2g}$ nematic order (an O(3) order-parameter) and $s$-wave pairing (an
O(2) order-parameter) constitute an $O(5)$ vector [see Eq. (40)], and these
two ordered phases reside next to each other, see Figs. 3, 4 (left), 6, 16,
when we tune the strength of $g_{{}_{1}}$.

  2. 2.

The $E_{2g}$ nematicity (an O(2) order-parameter) and $s$-wave pairing
constitute an $O(4)$ composite order parameter [see Eq. (43)], and Figs. 3, 4
(right), 6, 16 display a confluence of these two ordered phases, as one tunes
the interaction $g_{{}_{2}}$.

  3. 3.

The $E_{g}$ nematicity and $A_{2u}$ magnet (described by an O(1) or $Z_{2}$
order-parameter) form an $O(3)$ vector [see Fig. 18(a)], and these two ordered
phases often (in particular, when $g_{{}_{3}}$ is tuned) reside next to each
other, see Figs. 3 and 16.

  4. 4.

One can construct multiple copies of $O(3)$ composite order parameters by
combining the components of $T_{2g}$ nematic and $T_{1u}$ (an O(3) order-
parameter) magnetic orders [see Fig. 18(b) and (c)], and these two phases can
be realized by tuning the quartic interaction $g_{{}_{4}}$, see Figs 3 and 16.

  5. 5.

$A_{2u}$ magnetic order and $E_{g}$ $d$-wave pairing (an O(2) order-parameter)
can be combined to form $O(3)$ vectors [see Fig. 19]. When the chemical
potential is finite and we tune the strength of $g_{{}_{3}}$, the system
accommodates a paired (magnetic) state at low (high) temperature, see Figs. 5
(top) and 6.

  6. 6.

Finally, note that multiple copies of an $O(3)$ vector can be formed by
combining the components of $T_{1u}$ magnetic order and $T_{2g}$ $d$-wave
pairing (an O(2) order-parameter), see Fig. 20. These two phases reside next
to each other at finite chemical doping when we tune the quartic coupling
$g_{{}_{4}}$, see Figs. 5 (bottom) and 6.

From the matrix representations of all quartic interactions [see Eq. (59)] and
order parameters [see Eqs. (IV.1) and (27)] the readers can convince
themselves that the above examples are in agreement with our proposed
selection rule from Eqs. (65) and (66). It is admitted that we arrive at the
conclusion from a leading order RG analysis. However, the alternative version
of the selection rule [see Eq. (66)] solely relies on the internal symmetry
among competing orders. We therefore believe that our proposed selection rule
is ultimately non-perturbative in nature, which can be tested in numerical
experiments, for example.

##  VII Summary and Discussions

To summarize we have presented a comprehensive analysis on the role of
electron-electron interactions in a three-dimensional Luttinger system,
describing a bi-quadratic touching of Kramers degenerate valence and
conduction bands (in the absence of chemical doping) of effective spin-3/2
fermions at an isolated point in the Brillouin zone. This model can succinctly
capture the low-energy physics of HgTe hgte , gray-Sn gray-sn-1 ; gray-sn-2 ,
227 pyrochlore iridates Savrasov ; Balents1 ; Exp:Nakatsuji-1 ;
Exp:Nakatsuji-2 ; Exp:armitage  and half-Heuslers Exp:cava ; Exp:felser ;
binghai . For concreteness, we focused only on the short-range components of
the Coulomb interaction (such as the ones appearing in an extended Hubbard
model), and neglected its long-range tail. Due to the vanishing density of
states, namely $\varrho(E)\sim\sqrt{E}$, sufficiently weak, but generic local
four-fermion interactions are irrelevant perturbations in this system.
Consequently, any ordering sets in at an intermediate coupling through quantum
phase transitions. We here address the instability of interacting Luttinger
fermions at finite coupling within the framework of a renormalization group
analysis, controlled by a “ _small_ ” parameter $\epsilon$, where
$\epsilon=d-2$. Notice that in two spatial dimensions a bi-quadratic band
touching (similar to the situation in Bernal-stacked bilayer graphene) yields
a _constant_ density of states, and local four-fermion interactions are
_marginal_ in $d=2$. Hence, our renormalization group analysis is performed
about the _lower-critical_ two spatial dimensions. In this framework all
quantum phase transitions from a disordered Luttinger (semi)metal to any
ordered phase take place at $g^{\ast}_{{}_{j}}\sim\epsilon$, where
$g_{{}_{j}}$ is the dimensionless coupling constant, and for three dimensions
$\epsilon=1$. We here restrict ourselves to repulsive (at the bare level)