# Gliner Vacuum, Self-consistent Theory of Ruppeiner Geometry for Regular
Black Holes

Chen Lan<EMAIL_ADDRESS>School of Physics, Nankai University, 94 Weijin
Road, Tianjin 300071, China  Yan-Gang Miao Corresponding author,
<EMAIL_ADDRESS>School of Physics, Nankai University, 94 Weijin Road,
Tianjin 300071, China

###### Abstract

In the view of Gliner vacuum, we remove the deformations in the first law of
mechanics for regular black holes, where one part of deformations associated
with black hole mass will be absorbed into enthalpy or internal energy and the
other part associated with parameters rather than mass will constitute a
natural $V$-$P$ term. The improved first law of mechanics redisplays its
resemblance to the first law of thermodynamic systems, which implies a
restored correspondence of the mechanic variables to the thermodynamic ones.
In particular, the linear relation between the entropy and horizon area
remains unchanged for regular black holes. Based on the modified first law of
thermodynamics, we establish a self-consistent theory of Ruppeiner geometry
and obtain a universal attractive property for the microstructure of regular
black holes. In addition, the repulsive and attractive interactions inside and
outside regular black holes are analyzed in detail.

##  1 Introduction

It is closely related [1, 2] to the _first law of thermodynamics_ (1LT) and
its corresponding entropy to construct the Ruppeiner geometry for black holes
(BHs). As is known, the 1LT is usually deduced from its resemblance to the
_first law of mechanics_ (1LM). Based on $T\propto\kappa$ calculated from
quantum theory [3], one can read from the resemblance the linear relation:
$S\propto A$. This is now known as the entropy/are law, where the
corresponding entropy is dubbed as the Bekenstein-Hawking entropy (BHE) [4,
5]. The 1LT cannot be correctly obtained when the resemblance mentioned above
breaks. Furthermore, if the entropy calculated from path-integral approach [6]
or Wald’s method [7, 8] does not coincide with that from the 1LT, the
Ruppeiner geometry based on the 1LT will be unreliable.

A regular black hole (RBH) is such a system that its 1LM is deformed [9],
which brings about the breaking of the resemblance between its 1LM and 1LT.
From this point of view, a RBH does not have a well-defined 1LT, and then its
Ruppeiner geometry is suspect. As the 1LT is considered to be the basis of
Ruppeiner geometry and many other research topics, such as the superradiance
and area spectrum, etc., the lack of a well-defined 1LT leads to an obstacle
for us to establish the Ruppeiner geometry for RBHs.

We take the noncommutative geometry inspired BH [10] as an example of RBHs to
show its present issue. Its shape function, $f(r,M,\theta)$, contains two
parameters, where one is BH mass $M$ and the other $\theta$ is related to the
minimal length of a noncommutative space. If $\theta$ is set to be constant,
the 1LT takes the form, $\mathrm{d}M=T\mathrm{d}S$, where $T$ is obtained from
the Euclidean path integral near the horizon of this BH, see e.g. Ref. [11].
The entropy calculated from such a 1LT breaks the entropy/area law, $S\neq
A/4$. On the other hand, if the entropy of this BH without backreaction obeys
[12] the entropy/area law, the 1LT does not holds, i.e., $\mathrm{d}M\neq
T\mathrm{d}S$, when the Hawking temperature is not modified and still
calculated by the path-integral approach. In a word, this is the present
situation for any RBHs111Their metrics satisfy $g_{tt}g_{rr}=-1$. that either
the entropy/area law or the 1LT breaks if the Hawking temperature is computed
in accordance with the formula, $-g_{tt}^{\prime}(r_{\rm H})/{(4\pi)}$.

In this work we persist in adopting the Hawking temperature formula and
maintain both the entropy/area law and the 1LT unbroken by applying the radial
pressure of Gliner vacuum [13, 14] as RBH’s pressure. Due to the pressure of
Gliner vacuum, we can remove the deformations [15] in the 1LM for RBHs. In
other words, the deformation related to $M$ will be absorbed into enthalpy or
internal energy, and the other deformations corresponding to the parameters
rather than $M$ constitute a natural $V$-$P$ term. For RBHs this procedure not
only offers a solution to reconstruct the 1LT, consequently a self-consistent
theory of Ruppeiner geometry, but also provides a new insight into the 1LT
related research topics, like the superradiance and area spectrum mentioned
above.

This paper is organized as follows. In Sec. 2, we analyze the RBHs with a
spherical symmetry and a single shape function in terms of Ricci
decomposition. Next, we examine in Sec. 3 the deformations of the first law of
mechanics in the RBHs. We discuss in Sec. 4 how to remove the deformations in
the first law of mechanics for some known RBHs in the view of Gliner vacuum.
In Sec. 5 we establish a self-consistent theory of Ruppeiner geometry for the
RBHs with a spherical symmetry and a single shape function in accordance with
a well-defined 1LT. The results show that all the RHBs are of attractive
interaction. It is known that the matters generating the RBHs violate the
strong energy condition (SEC) around the centre, i.e., they have repulsive
interaction. Therefore, we try in Sec. 6 to provide an explanation in terms of
the SEC on how the matters with repulsive interaction create the RBHs with
attractive interaction. Finally, we give our summary in Sec. 7.

##  2 Regular black hole

We start with a spherically symmetric metric
$g_{\mu\nu}=\operatorname{diag}\\{-f(r),f^{-1}(r),r^{2},r^{2}\sin^{2}\theta\\}$
with the shape function,

$f(r)=1-\frac{2M}{r}\sigma\left(r,M,\alpha_{i}\right),$ (1)

where $\alpha_{i}$’s are parameters rather than mass $M$. If $\sigma$
converges to one when $r$ goes to infinity, the corresponding metric is
asymptotic to the external spacetime of Schwarzschild BHs. In fact, the
asymptotic flatness of our metric only requires $\lim_{r\to\infty}\sigma/r=0$,
i.e., $\sigma$ is divergent slower than $r$ as $r\to\infty$, which is weaker
than the condition of asymptotic to the Schwarzschild spacetime. The
requirement of asymptotic to the Schwarzschild BH simply guarantees the
attractive nature (strong energy condition) and causality (dominant energy
condition) outside the horizon. In addition, for simplicity we suppose that
the BHs depicted by Eq. (1) are of two horizons at most.

It is convenient to apply the Ricci decomposition [16] for our investigations
of RBHs,

$R_{\mu\nu\rho\sigma}=W_{\mu\nu\rho\sigma}+S_{\mu\nu\rho\sigma}+E_{\mu\nu\rho\sigma},$
(2)

where $W_{\mu\nu\rho\sigma}$ is the Weyl tensor, and

$\displaystyle S_{\alpha\beta\mu\nu}$ $\displaystyle=$
$\displaystyle\frac{R}{12}g_{\alpha[\mu}g_{\nu]\beta},$ (3) $\displaystyle
E_{\alpha\beta\mu\nu}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(g_{\alpha[\mu}Z_{\nu]\beta}-g_{\beta[\mu}Z_{\nu]\alpha}\right),$
(4)

with the traceless tensor,

$Z_{\mu\nu}\equiv R_{\mu\nu}-\frac{1}{4}g_{\mu\nu}R.$ (5)

Thus, we rewrite $\sigma$ in terms of the curvature invariants,222In fact, $W$
and $E$ can be expressed by the three independent curvature invariants, $R$,
$R_{2}$, and $K$, see App. A for the details. $R$, $W$, and $E$, when $r$ is
around the centre of RBHs,

$\sigma=\frac{r^{3}}{24M}\left(R-2\sqrt{3W}+3\sqrt{2E}\right),$ (6)

where $W\equiv W_{\mu\nu\rho\sigma}W^{\mu\nu\rho\sigma}$ and $E\equiv
E_{\mu\nu\rho\sigma}E^{\mu\nu\rho\sigma}$, and both $W$ and $E$ are
nonnegative. According to the relations,

$\displaystyle R_{2}$ $\displaystyle\equiv$ $\displaystyle
R_{\mu\nu}R^{\mu\nu}=\frac{1}{4}\left(R^{2}+2E\right),$ (7) $\displaystyle K$
$\displaystyle\equiv$ $\displaystyle
R_{\mu\nu\rho\sigma}R^{\mu\nu\rho\sigma}=\frac{1}{6}R^{2}+W+E,$ (8)

the contraction of Ricci tensors $R_{2}$ and the Kretschmann scalar $K$ are
also nonnegative. Based on the Ricci decomposition Eq. (2), the shape function
then becomes

$f(r)=1-\frac{\lambda(r)}{3}r^{2},\qquad 4\lambda(r)\equiv
R-2\sqrt{3W}+3\sqrt{2E}.$ (9)

That a BH is regular means that the independent curvature invariants, $R$,
$R_{2}$, and $K$, are convergent to finite constants everywhere in this BH
spacetime. In particular, as $r\to 0^{+}$ we have

$R\sim R(0),\qquad R_{2}\sim R_{2}(0),\qquad K\sim K(0),$ (10)

with $R(0),R_{2}(0),K(0)=\text{const}.<\infty$, where the zero argument refers
to the limit to the center of BHs. We can see from Eqs. (1) and (6) that the
finiteness of the shape function requests that $\sigma$ converges to zero
faster than $r^{3}$, i.e., $\lim_{r\to 0}\sigma/r^{3}=\text{const}.<\infty$,
see App. B for the detailed analyzes of $R$ and $K$.

It is worthy to mention that the treatment to RBHs is different from that to
singular BHs (SBHs). For the latter, a complete Lagrangian is given, its
equations of motion are solved, and thus the metric of SBHs is obtained. For
the former, however, the first step is to propose such a metric that the
corresponding curvature invariants are finite and geodesics are complete at
the singularity of this metric; and the second step is to construct the
related Lagrangian that generates this RBH, i.e., to find out the reasonable
matter source.

According to the behaviors of $\sigma$, the metrics of RBHs can be classified
into two types. The first type of RBHs is constrained locally by
$0<\sigma\left(r,M,\alpha_{i}\right)\leq 1$, thus the outer horizon $r_{+}$
must be upper bounded by the Schwarzschild radius, $r_{\rm Sch}=2M$, i.e.,

$\frac{r_{+}}{r_{\rm Sch}}=\sigma\left(r_{+},M,\alpha_{i}\right)\leq 1.$ (11)

The second type of RHBs is picked out by the local condition,
$\sigma\left(r,M,\alpha_{i}\right)>1$. As the matter sources associated with
the metrics in the second type are not clear, we concentrate only on the first
type and assume that $\sigma$ is a monotonically increasing function333The
monotonicity comes from the positivity of energy density of matters, see Eq.
(69). Thus, $\sigma(r)$ is a sigmoid function in the range of
$r\in[0,\infty)$. of $r$, $\mathrm{d}\sigma/\mathrm{d}r>0$, in the remaining
of the present work.

In addition, let us make a brief discussion about the factor $\lambda(r)$ in
Eq. (9). It is not difficult to check $\lambda(r)\sim R(0)/4$ in the limit of
$r\to 0^{+}$, because $R\sim O(1)$, $E\sim O(r^{2})$, and $W\sim O(r^{2})$,
i.e. the orders of $E$ and $W$ are higher than that of $R$. The same result
can also be obtained by the method shown in Refs. [17, 18], see App. B. Then
changing the parameters (mass, charge, etc.) in $f(r_{+})=0$, such that
$r_{+}$ approaches to zero, we obtain (see Eq. (94))

$1-\frac{R(0)}{12}r_{+}^{2}+O\left(r_{+}^{3}\right)=0,$ (12)

which implies that the horizon exists only for nonnegative $R(0)$. Meanwhile,
the negative $R$ is forbidden by the dominant energy condition [17]. In other
words, a RHB is of a dS core rather than an AdS one around the center if
$R(0)$ does not vanish. Incidentally, if such a RHB is immersed in an AdS
spacetime with the cosmological constant $\tilde{\Lambda}$, the corresponding
shape function becomes

$f(r)=1-\frac{\lambda(r)}{3}r^{2}-\frac{\tilde{\Lambda}}{3}r^{2},\qquad\tilde{\Lambda}<0,$
(13)

which gives rise to the fact that $\lambda(0)$ is larger than
$-\tilde{\Lambda}$, otherwise no horizons exist.

##  3 Deformation of the first law of mechanics

SBHs, e.g., the Schwarzschild BH and Reissner-Nördstrom (RN) BH, can be
regarded as thermodynamic systems because their mechanic laws have a
resemblance to the thermodynamic ones. Nevertheless, this resemblance is
broken in RBHs. If RHBs are of the shape function of Eq. (1), we make
differentiation on the two sides of $r_{+}=2M\sigma(r_{+},M,\alpha_{i})$ and
obtain

$\mathrm{d}r_{+}=\frac{2\left(M\sum_{i}\partial_{\alpha_{i}}\sigma\mathrm{d}\alpha_{i}+M\partial_{M}\sigma\mathrm{d}M+\sigma\mathrm{d}M\right)}{1-2M\partial_{r_{+}}\sigma}.$
(14)

To construct the 1LM, by substituting Eq. (14) into
$\kappa\mathrm{d}A/(8\uppi)$, where $A=4\uppi r_{+}^{2}$ and $\kappa$ is
surface gravity at $r_{+}$, we give

$\frac{\kappa}{8\uppi}\mathrm{d}A=(1-\tau)\mathrm{d}M+\sum_{i}\beta_{i}\mathrm{d}\alpha_{i}+\Delta(\alpha_{i}),$
(15)

where $\beta_{i}$ is the conjugate of $\alpha_{i}$, $\tau$ defined by

$\tau\equiv
1-\frac{r_{+}}{2M}-M\frac{\partial\sigma(r_{+},M,\alpha_{i})}{\partial M}$
(16)

means the deformation associated with mass $M$, and $\Delta(\alpha_{i})$
stands for the deformations associated with $\alpha_{i}$’s. In order to
restore the resemblance between the mechanic and thermodynamic laws for RBHs,
we have to remove reasonably those deformations just mentioned. In the
following, we analyze the difficulties that we shall encounter and propose a
possible way to overcome them.

###  3.1 Deformations associated with $M$ and $\alpha_{i}$

If one simply applies the traditional replacement for Eq. (15),

$\kappa\to 2\uppi T,\qquad A\to 4S,\qquad M\to E,$ (17)

one will obtain the formula,

$T\mathrm{d}S=(1-\tau)\mathrm{d}E+\sum_{i}\beta_{i}\mathrm{d}\alpha_{i}+\Delta(\alpha_{i}),$
(18)

which cannot be regarded as the 1TL of an isolated system because of the
deformations. In other words, either the RBH is not an isolated system or the
correspondence between mechanic and thermodynamic variables Eq. (17) is not
appropriate.

The usual attempt is to let the term $\tau\mathrm{d}M$ in Eq. (15) be absorbed
into the entropy of RBHs in order to restore the resemblance between the
mechanic and thermodynamic laws for RBHs. To this end, we define the
conditional entropy describing the entropy of RBHs,

$\displaystyle S_{c}$ $\displaystyle\equiv$ $\displaystyle\int^{r_{+}}_{r_{\rm
ext}}\frac{\mathrm{d}M}{T}$ (19) $\displaystyle=$ $\displaystyle\int_{r_{\rm
ext}}^{r_{+}}\frac{\mathrm{d}A}{4}+\int_{r_{\rm
ext}}^{r_{+}}\frac{\tau}{T}\left(\frac{\mathrm{d}M}{\mathrm{d}\tilde{r}_{+}}\right)_{\alpha_{i}}\mathrm{d}\tilde{r}_{+},$

where $T$ is Hawking temperature and $r_{\rm ext}$ denotes the horizon radius
of extreme RBHs. In Eq. (19), the first term is just the BHE, while the second
one represents the deformation or deviation from the BHE. We can prove that
$\tau>0$ when $0<\sigma\leq 1$, which implies that the deformation in Eq. (19)
is positive. Let us make an analysis from two aspects.

  * •

If $\sigma$ does not contain $M$, our statement is obviously true due to Eq.
(11), that is, $\tau>0$ when $0<\sigma\leq 1$.

  * •

If $\sigma$ depends on $M$ explicitly, we make a dimensionless rescaling by
$M$ for all variables, such as $r_{+}$ being rescaled to $x_{+}\equiv
r_{+}/(2M)$, thus the shape function becomes

$f=1-\frac{\sigma(x,\widetilde{\alpha}_{i})}{x},$ (20)

where $\widetilde{\alpha}_{i}$ is the dimensionless counterpart of
$\alpha_{i}$ which is rescaled by a power function of $M$. Then repeating the
procedure proceeded at the beginning of this section, we obtain

$\tau=1-x_{+}\left[1-\frac{\partial\sigma(x_{+},\widetilde{\alpha}_{i})}{\partial
x_{+}}\right].$ (21)

Since the slope of tangent line of function $\sigma(x,\widetilde{\alpha}_{i})$
at the outer horizon $x_{+}$ is not greater than unit when $0<\sigma\leq 1$
and $x_{+}<1$, we verify the above statement.

If the BHE were replaced by the conditional entropy, we would remove the
deformation associated with $M$ but we would no longer maintain the linear
relation between $S_{c}$ and horizon area as a price. Furthermore, the
deformation with respect to $\alpha_{i}$ still remains. As a matter of fact,
due to lack of resemblance between RBHs and the traditional thermodynamic
systems, the introduction of $S_{c}$ does not work well in removing the
deformations associated with $M$ and $\alpha_{i}$. More importantly, $S_{c}$
is not an independent variable because its definition is subject to the 1TL.

###  3.2 Proposal for removing deformations

In fact, starting with the classical action and observing the partition
function at the zero-loop approximation, we find that the Einstein-Hilbert
action of RBHs with the shape function Eq. (1) contributes one part of entropy
and the Gibbons-Hawking-York surface term provides the other part of
contributions to the entropy of RBHs, and that the combination of the two
contributions recovers the linear relation $S\propto A$. Let us give the
derivation. The Einstein-Hilbert action takes the form,

$I_{\rm
EH}=\frac{2M-r_{+}\left[1+2M\sigma^{\prime}\left(r_{+}\right)\right]}{4T},$
(22)

where $\sigma$ converges to unit as $r\to\infty$ and the prime denotes the
derivative with respect to the radial coordinate, and the Gibbons-Hawking-York
surface term is

$I_{\rm GHY}=-\frac{M}{2T},$ (23)

where the flat spacetime is selected as background reference [19]. As a
result, the total action reads

$I_{\rm tot}=I_{\rm EH}+I_{\rm GHY}=-\uppi
r_{+}^{2}+\beta\frac{r_{+}}{2},\qquad\beta\equiv\frac{1}{T},$ (24)

with which the entropy can be computed by

$S=\beta\frac{\partial I_{\rm tot}}{\partial\beta}-I_{\rm tot}=\uppi
r_{+}^{2}.$ (25)

That is to say, the semiclassical approach prefers that the entropy of RBHs is
$A/4$ rather than $S_{c}$. The same result can also be obtained by the Wald
method [19]. The situation of RBHs is similar to the case of dielectric in an
external electric field [20] where the variation of internal energy does not
alter the entropy of dielectric system. That is to say, the deformation
associated with mass $M$ does not depend on the thermodynamic state of RBHs,
thus it should not affect the entropy of RBHs.

In conclusion, our proposal is to maintain the linear relation, $S\propto A$,
and simultaneously to remove the deformations mentioned in the above
subsection, that is, to establish the resemblance between the mechanic and
thermodynamic laws for RBHs in terms of the pressure of Gliner vacuum [13, 14]
which is dealt with as the pressure of RBHs. Thanks to the pressure of Gliner
vacuum, the deformation associated with $M$ will be absorbed into the enthalpy
or internal energy and the deformation associated with $\alpha_{i}$ will
constitute a naturally $V$-$P$ term. In this way, all deformations disappear
and the other terms are well defined in Eq. (15), which gives rise to a normal
1LT in formulation.

We shall see that our proposal works well for some known RBH models as
examples. It is necessary for us to choose models because we need the
formulations of shape functions, however, we shall see that our proposal is
valid for all RBHs with a single shape function depicted by Eq. (1).

##  4 Pressure and reconstruction of the first law of mechanics

It is widely known that the negative cosmological constant can be interpreted
[21] as thermal pressure of BHs, while the positive cosmological constant has
a problem of thermal equilibrium [22], which is caused by the existence of a
cosmological horizon. No matter the cosmological constant is negative or
positive, the essence of this idea is to regard the pressure of _vacuum_ as
the pressure of BHs. Nevertheless, a RBH has its own vacuum even though it is
not involved in an additional AdS (or dS) term in metric. This seed goes back
to Sakharov and Gliner [23, 13], who reexplained the vacuum as spacetime
filled with vacuum. Such a vacuum is currently dubbed as Gliner vacuum. It
will be our key to reconstruct the 1LM for RBHs. Next, we introduce the
pressure of Gliner vacuum in the following two models.

###  4.1 Model I

The first model is generated [24] by an anisotropic vacuum and its shape
function reads

$f(r)=1-\frac{2M}{r}\sigma(r,M,\Lambda),\qquad\sigma(r,M,\Lambda)=1-\exp\left(-\frac{\Lambda}{6M}r^{3}\right).$
(26)

Here $\Lambda$ is cosmological constant. The asymptotic flatness at infinity
requires that $\Lambda$ be positive. Moreover, $f(r)$ approaches to de Sitter
space, i.e., $f(r)\sim 1-\Lambda r^{2}/3+O(r^{5})$, as $r\to 0$. When $r$
becomes large, the term $\exp[-\Lambda r^{3}/(6M)]$ goes to zero, thus $f(r)$
is asymptotic to the Schwarzschild black hole. Since $\sigma\leq 1$ for
$r\in\mathbb{R}^{+}$, the outer horizon $r_{\rm+}$ is restricted by $r_{\rm
ext}\leq r_{\rm+}\leq 2M$ when $\Lambda>9.28/(4M^{2})$. The extreme radius is
$r_{\rm ext}\approx 0.85\times 2M$. For $\Lambda<0$, there is no real solution
for $f(r_{+})=0$. Therefore, this model has no cosmological horizons, and thus
does not suffer from the problem of thermal equilibrium.

The Smarr formula was obtained from the total mass represented by the Komar
integral which can be separated [25] into two parts. The first part is a
surface integral over the horizon, and gives $\kappa A/(4\uppi)$; while the
second one, including a deviation of the first law, is a volume integral with
one boundary at spatial infinity and the other at event horizon [9, 26].
Combining the two parts, one obtains the Smarr formula,

$M=\frac{\kappa A}{4\uppi}+\epsilon M+\frac{1}{2}\epsilon\Lambda r_{+}^{3},$
(27)

where $\epsilon\equiv\exp\left[-\Lambda r_{+}^{3}/(6M)\right]>0$. This Smarr
formula suggests an extended phase space. However, if one applied
$P=-\Lambda/(8\uppi)$ as thermal pressure [21], the first law of mechanics
would be deformed, i.e.,

$\frac{\kappa}{8\uppi}\mathrm{d}A=(1-\tau)\mathrm{d}M-\epsilon V\mathrm{d}P,$
(28)

where $\tau\equiv\epsilon\left[1-\ln\left(\epsilon\right)\right]$, and
$V\equiv 4\uppi r_{+}^{3}/3$ is the thermal volume inside the horizon.
Further, we find that the $1$-form $\kappa\mathrm{d}A/(8\uppi)+V\mathrm{d}P$
does not satisfy the integrable condition, i.e., it cannot be written as a
total derivative of any functions.

By introducing the radial pressure of Gliner vacuum at the outer horizon,

$P_{+}=\left.\frac{G^{r}_{\;r}}{8\uppi}\right|_{r=r_{+}}=-\epsilon\frac{\Lambda}{8\uppi},$
(29)

which is negative, as the pressure for Model I, where $G^{r}_{\;r}$ is $r$-$r$
component of Einstein tensor $G_{\mu\nu}$, we note that the last term of Eq.
(27) is nothing else but $-3VP_{+}$. As a result, following the way in Ref.
[15] and introducing _enthalpy_ ,

$H=(1-\tau)M,$ (30)

we can reconstruct the 1LM from Eq. (28),

$\frac{\kappa}{8\uppi}\mathrm{d}A=\mathrm{d}H-V\mathrm{d}P_{+},$ (31)

and write the corresponding 1LT,

$\mathrm{d}H=T\mathrm{d}S+V\mathrm{d}P_{+},$ (32)

with

$T=\frac{\kappa}{2\uppi},\qquad S=\frac{A}{4}.$ (33)

The deformations of Eq. (28) are completely removed in Eq. (31).

Alternatively, Eq. (31) can be rewritten as

$\frac{\kappa}{8\uppi}\mathrm{d}A=\mathrm{d}U+P_{+}\mathrm{d}V,$ (34)

with _total internal energy_

$U=(1-\epsilon)M=H-VP_{+},$ (35)

which suggests that the deviation from $M$ in Eq. (27) should be absorbed into
internal energy. The other compelling reason to think of $(1-\epsilon)M$ as
the energy is that it can be calculated by the integration of energy density
$\rho$ over the whole space inside the RBH, i.e.,

$(1-\epsilon)M=\int^{r_{+}}_{0}\int^{2\uppi}_{0}\int^{\pi}_{0}\sqrt{-g}\rho\mathrm{d}r\mathrm{d}\phi\mathrm{d}\theta=\frac{r_{+}}{2},$
(36)

where $\rho$ is defined by
$\rho=-G^{0}_{\;0}/(8\uppi)=\Lambda\exp\left[-\Lambda
r^{3}/(6M)\right]/(8\uppi)$ according to Einstein’s equation,
$G^{\mu}_{\;\nu}=8\uppi T^{\mu}_{\;\nu}$. In other words, $M$ is the total
energy filled in the whole space of the RBH, while $\epsilon M$ is the energy
outside the RBH.

Thus, the internal energy of the RBH should be the energy enclosed in the
event horizon, and the corresponding 1LT can be cast as follows:

$\mathrm{d}U=T\mathrm{d}S-P_{+}\mathrm{d}V,$ (37)

where every term is well-defined. As we expect, the resemblance between the
mechanic and thermodynamic laws for this RBH is restored, see Eqs. (31) and
(32) or Eqs. (34) and (37), and simultaneously the entropy is just the
Bekenstein-Hawking entropy obtained by path-integral and Wald’s method [19].

###  4.2 Model II

Now let us turn to the second model, Model I immersed in AdS spacetime, which
can be established if the other cosmological constant $\widetilde{\Lambda}$ is
introduced into Model I,

$f(r)=1-\frac{2M}{r}\sigma(r,M,\Lambda)-\frac{\widetilde{\Lambda}}{3}r^{2},\qquad\sigma(r,M,\Lambda)=1-\exp\left(-\frac{\Lambda}{6M}r^{3}\right).$
(38)

There is no doubt that such a metric is a solution of Einstein’s equations
with a cosmological constant term [24]. The homogeneity of universe remains
unchanged if $\Lambda$ is treated as a local character, i.e., $\Lambda$
dominates only the inside (and around certain range) of this RBH. Meanwhile,
to bypass the problem of thermal equilibrium [27], we demand
$\widetilde{\Lambda}<0$. The Smarr formula can be calculated by the same way
as that mentioned above,

$M=\frac{\kappa A}{4\uppi}+\epsilon M+\frac{1}{2}\epsilon\Lambda
r_{+}^{3}+\frac{1}{3}\widetilde{\Lambda}r_{+}^{3},$ (39)

where $\epsilon$ has the same form as that of Model I, but the energy density
now is calculated by $\rho=-(G^{0}_{\;0}+\widetilde{\Lambda})/(8\uppi)$
because of $G^{\mu}_{\;\nu}+\widetilde{\Lambda}g^{\mu}_{\;\nu}=8\uppi
T^{\mu}_{\;\nu}$. We note that the deformations still exit even if the AdS
constant $\widetilde{\Lambda}$ is regarded as pressure. If we regard the
radial pressure of Gliner vacuum as the pressure, we can obtain the same 1LM
as Eq. (31), where the _enthalpy_ is defined in Eq. (30) and the Gliner
pressure takes the form,

$P_{+}=-\frac{1}{8\uppi}\left(\epsilon\Lambda+\widetilde{\Lambda}\right).$
(40)

The competition of two cosmological constants appears in the pressure. As we
have noted in Sec. 2, $\epsilon\Lambda$ should be larger than
$-\widetilde{\Lambda}$, otherwise there will be no horizons. As a result, the
Gliner pressure in Model II is negative, and this pressure is different from
the one simply introduced from the AdS cosmological constants.

###  4.3 Other models

The models in the above two subsections give a heuristic evidence that a well-
defined 1LM can be reconstructed without any deformations and the traditional
area law, $S=A/4$, still holds when we introduce the radial pressure of Gliner
vacuum. It will be seen in this subsection that some other models, including
those with Gliner vacuum or vacuum-like mediums, also support this statement.

(i) Model in Ref. [10]. This model is also called noncommutative geometry
inspired black hole. The shape function takes the form,

$f(r)=1-\frac{4M}{\sqrt{\uppi}\,r}\gamma\left(\frac{3}{2},\frac{r^{2}}{4\theta}\right).$
(41)

The Samrr formula is

$M=\frac{A\kappa}{4\uppi}+\epsilon
M+\frac{Mr_{+}^{3}\mathrm{e}^{-\frac{r_{+}^{2}}{4\theta}}}{2\sqrt{\uppi}\theta^{3/2}},$
(42)

where in the second term on the right hand side $\epsilon$ equals

$\epsilon=\frac{2}{\sqrt{\uppi}}\Gamma\left(\frac{3}{2},\frac{r_{+}^{2}}{4\theta}\right),$
(43)

which coincides with the result given by Eq. (36); the third term corresponds
to $-3VP_{+}$, and the radial pressure of Gliner vacuum at the outer horizon
reads

$P_{+}=-\frac{Me^{-\frac{r_{+}^{2}}{4\theta}}}{8\uppi^{3/2}\theta^{3/2}}.$
(44)

We can also give the 1LM Eq. (31) with the enthalpy and internal energy,

$H=\frac{r_{+}}{2}+\frac{r_{+}^{4}}{12\theta
r_{+}-12\sqrt{\uppi}\,\theta^{3/2}\exp\left({\frac{r_{+}^{2}}{4\theta}}\right)\text{erf}\left(\frac{r_{+}}{2\sqrt{\theta}}\right)},\qquad
U=M\left[\text{erf}\left(\frac{r_{+}}{2\sqrt{\theta}}\right)-\frac{r_{+}\mathrm{e}^{-\frac{r_{+}^{2}}{4\theta}}}{\sqrt{\uppi}\sqrt{\theta}}\right]=\frac{r_{+}}{2}.$
(45)

(ii) Model in Ref. [28]. The shape function is

$f(r)=1-\frac{2Mr}{r^{2}+6M/(r\Lambda)},$ (46)

where $\Lambda$ is supposed to be positive. The Smarr formula has the form,

$M=\frac{\kappa A}{4\uppi}+\frac{6M^{2}}{\Lambda r_{+}^{3}+6M}+\frac{18\Lambda
M^{2}r_{+}^{3}}{\left(\Lambda r_{+}^{3}+6M\right)^{2}},$ (47)

where the second term on the right hand side stands for $\epsilon M$ with
$\epsilon=6M/(6M+\Lambda r_{+}^{3})$, which coincides with the result given by
Eq. (36); the third term corresponds to $-3VP_{+}$, and the radial pressure of
Gliner vacuum at the outer horizon equals

$P_{+}=-\frac{9\Lambda M^{2}}{2\uppi\left(6M+\Lambda r_{+}^{3}\right)^{2}}.$
(48)

The same procedure leads to the 1LM Eq. (31) with the enthalpy and internal
energy,

$H=\frac{r_{+}^{2}}{4M},\qquad U=(1-\epsilon)M=\frac{r_{+}}{2}.$ (49)

(iii) Model in Ref. [29]. This model is usually called Bardeen’s black hole.
The shape function is

$f(r)=1-\frac{2Mr^{2}}{\left(b^{2}+r^{2}\right)^{3/2}},$ (50)

where $b$ is magnetic charge. The Smarr formula then reads

$M=\frac{A\kappa}{4\uppi}+M\left[1-\frac{r_{+}^{3}}{\left(b^{2}+r_{+}^{2}\right)^{3/2}}\right]+\frac{3b^{2}Mr_{+}^{3}}{\left(b^{2}+r_{+}^{2}\right)^{5/2}},$
(51)

where we have used $\epsilon=1-r_{+}^{3}/\left(b^{2}+r_{+}^{2}\right)^{3/2}$.
In this model the pressure of Gilner vacuum takes the form,

$P_{+}=-\frac{3b^{2}M}{4\uppi\left(b^{2}+r_{+}^{2}\right)^{5/2}}.$ (52)

In the 1LM the enthalpy and the internal energy equal

$H=\frac{r_{+}^{3}}{2\left(b^{2}+r_{+}^{2}\right)},\qquad
U=\frac{Mr_{+}^{3}}{\left(b^{2}+r_{+}^{2}\right){}^{3/2}}=\frac{r_{+}}{2}.$
(53)

(iv) Model in Ref. [30]. This model is generated by a vacuum-like medium. The
shape function takes the form,

$f(r)=1-\frac{2M}{r}\exp\left({-\frac{q^{2}}{2Mr}}\right),$ (54)

where $q$ stands for electric charge. The Smarr formula is

$M=\frac{A\kappa}{4\uppi}+\left(1-\mathrm{e}^{-\frac{q^{2}}{2Mr_{+}}}\right)M+\frac{q^{2}}{2r_{+}}\mathrm{e}^{-\frac{q^{2}}{2Mr_{+}}},$
(55)

where $\epsilon$ and $P_{+}$ read

$\epsilon=1-\mathrm{e}^{-\frac{q^{2}}{2Mr_{+}}},\qquad
P_{+}=-\frac{q^{2}}{8\uppi r_{+}^{4}}\mathrm{e}^{-\frac{q^{2}}{2Mr_{+}}}.$
(56)

The enthalpy and internal energy are

$H=\frac{r_{+}}{2}-\frac{r_{+}}{6}\ln\left(\frac{2M}{r_{+}}\right),\qquad
U=M\mathrm{e}^{-\frac{q^{2}}{2Mr_{+}}}=\frac{r_{+}}{2}.$ (57)

For the above four RBHs, we emphasize that their internal energies calculated
by Eq. (36) equal $r_{+}/2$. This is not a coincidence. In fact, Eq. (36)
gives exactly $r_{+}/2$ for all spherically symmetric RBHs with the shape
function Eq. (1).

An interesting property noted in Ref. [15] is that the $\Phi$-$Q$ term in
Reissner-Nördstrom (RN) BHs can be absorbed into $V$-$P_{+}$ term, such that
$\kappa\mathrm{d}A/(8\uppi)=\mathrm{d}M-\Phi\mathrm{d}Q$ becomes Eq. (31).
This result implies that the Gliner vacuum provides a unified treatment for
both RBHs and SBHs in the establishment of 1LM. In this way, the enthalpy and
internal energy of RN BHs take the form,

$H=\frac{r_{+}}{2}-\frac{Q^{2}}{6r_{+}},\qquad U=\frac{r_{+}}{2}.$ (58)

When $Q\to 0$, the enthalpy of RN BHs reduces to that of Schwarzschild BHs,
$H=r_{+}/2=M$, which coincides with its internal energy. For RN-AdS BHs, one
has $H=U-Q^{2}/(6r_{+})-\Lambda r_{+}^{3}/6$ with $U=r_{+}/2$. It is worthy to
emphasize that the internal energies of SBHs, such as the Schwarzschild BHs,
RN BHs, and RN-AdS BHs we just mentioned, are calculated from the 1TL but not
from Eq. (36). The reason is that the energy density of SBHs is singular or
vanishing at $r=0$, which leads to divergence of Eq. (36) or makes the
integration trivial. In order to make Eq. (36) suitable for SBHs, we introduce
a cutoff of radial coordinate for the SBHs whose energy densities are
singular, such that the integration is regularized. Taking the RN BHs as an
example, the regularized integration of internal energy is

$U=\int^{r_{\rm+}}_{r_{0}}\int^{2\uppi}_{0}\int^{\pi}_{0}\sqrt{-g}\rho\mathrm{d}r\mathrm{d}\phi\mathrm{d}\theta,\qquad\rho=\frac{Q^{2}}{8\uppi
r^{4}},$ (59)

where $r_{0}$ is cutoff. Since $U=r_{+}/2$, we can fix $r_{0}=Q^{2}/(2M)$.
Meanwhile, we can prove $r_{0}<r_{-}$, i.e.,

$Q^{2}/(2M)<M-\sqrt{M^{2}-Q^{2}},$ (60)

which implies that the cutoff is located insider the inner horizon. As to the
Schwarzschild BHs, one can use the interior metric of Schwarzschild BHs [31]
and suppose that the radius of the matter surface equals $2M$, thus one can
derive the energy density,

$\rho(r)=\frac{3}{32\uppi
M^{2}}\left[\mathcal{H}\left(\frac{r}{2M}\right)-\mathcal{H}\left(\frac{r}{2M}-1\right)\right],$
(61)

where $\mathcal{H}(\cdot)$ is the Heaviside function. It is clear that the
energy in the volume $V=4\uppi r_{+}^{2}/3$ takes the value $U=r_{+}/2=M$.

Now we summarize the features of the new pressure we have proposed in Models I
& II and also applied to the other four models. At first, the Gliner pressure
is not universal, i.e., it has different values for different models, which is
similar to the Hawking temperature that is horizon dependent. Next, there are
no conceptional contradictions between the pressure and the vacuum or vacuum-
like matter that generates RBHs, i.e., it is widely known that the RHBs are
generated by the vacuum or vacuum-like matter with negative pressure rather
than the AdS constant with positive pressure. At last, we find the following
relation,

$|P_{+}|\propto R(0),$ (62)

where $R(0)$ can be explained as the average energy of vacuum. This gives rise
to our conclusion that the new pressure we proposed, i.e., the radial pressure
of Gliner vacuum originates from the average energy of Gliner vacuum, which is
consistent with the case for SBHs where the pressure comes simply from the
average energy of (AdS) vacuum. In summary, by introducing the Gliner pressure
as the pressure of RBHs we avoid such a contradiction that a RBH generated by
the matter (vacuum) with negative pressure has positive pressure because this
contradiction is physically unacceptable.

##  5 Ruppeiner geometry of regular black holes

Having the reconstructed 1LT, see Eq. (32), at hand, we apply the Gibbs
energy, $G=H-TS$, as the starting point and calculate the line element by
following Ref. [32],

$\Delta l^{2}=\frac{1}{k_{\rm
B}}\left(C_{P_{+}}\mathrm{d}T^{2}+\frac{1}{T}\frac{\partial V}{\partial
P_{+}}\mathrm{d}P_{+}^{2}\right),$ (63)

where $k_{\rm B}$ is Boltzmann constant and the capacity at constant pressure
is defined as usual,

$C_{P_{+}}=\frac{1}{T}\left(\frac{\partial T}{\partial S}\right)_{P_{+}}.$
(64)

Considering the metric with the shape function, Eq. (1), we find

$P_{+}=-\frac{M\sigma^{\prime}(r_{+})}{4\uppi r_{+}^{2}},\qquad
T=\frac{1}{4\uppi r_{+}}-\frac{M\sigma^{\prime}(r_{+})}{2\uppi r_{+}},$ (65)

and then obtain the equation of state by combining the above two formulas,

$P_{+}=\frac{T}{2r_{+}}-\frac{1}{8\uppi r_{+}^{2}},$ (66)

whose dependence on parameters, such as mass, charge, etc., is hidden in the
outer horizon. Based on this state of equation, we derive the thermodynamic
curvature for a general spherically symmetric RBH with Eq. (1) as its shape
function,

$\frac{\cal R}{2\uppi
T^{2}}=y\left(\sqrt{1-y}-3\right)-4\left(\sqrt{1-y}-1\right),$ (67)

where $y$ defined by $y\equiv 2P_{+}/(\uppi T^{2})$ is dimensionless. Since
$P_{+}<0$, we deduce ${\cal R}/T^{2}<0$. That is to say, the interaction of
all spherical symmetric RBHs with a single shape function is attractive from
the microscopic perspective. Nevertheless, it is known that RHBs are generated
by the matters which are of repulsive interaction around the centre, thus a
natural problem is how a repulsive matter forms an attractive black hole. Let
us give a quantitative analysis in terms of the strong energy condition (SEC)
of RBHs.

##  6 Repulsive and attractive interactions inside and outside regular black
holes

Instead of analyzing the energy-momentum tensor of matter $T^{\mu}_{\;\nu}$,
we concentrate on the diagonalized Einstein tensor,

$G^{\mu}_{\;\nu}=\operatorname{diag}\left\\{-\frac{2M\sigma^{\prime}}{r^{2}},-\frac{2M\sigma^{\prime}}{r^{2}},-\frac{M\sigma^{\prime\prime}}{r},-\frac{M\sigma^{\prime\prime}}{r}\right\\},$
(68)

because $G^{\mu}_{\;\nu}$ and $T^{\mu}_{\;\nu}$ are equivalent due to the
Einstein equation $G^{\mu}_{\;\nu}=8\uppi T^{\mu}_{\;\nu}$. The energy density
$\rho$ and pressures $\rho_{r}$ and $\rho_{\perp}$ can then be obtained,

$\rho=\frac{M\sigma^{\prime}}{4\uppi r^{2}},\qquad
p_{r}=-\frac{M\sigma^{\prime}}{4\uppi r^{2}},\qquad
p_{\perp}=-\frac{M\sigma^{\prime\prime}}{8\uppi r}.$ (69)

Moreover, in order to discuss the SEC, we introduce parameter $\gamma$,

$\gamma\equiv\rho+p_{r}+2p_{\perp}=-\frac{M\sigma^{\prime\prime}}{4\uppi r},$
(70)

whose sign indicates the attractive ($\gamma>0$) or repulsive ($\gamma<0$)
interaction. This can be understood clearly from Raychaudhuri’s equation [33].
Moreover, if the expansion, rotation, and shear terms in Raychaudhuri’s
equation are neglected when compared with the variation of expansion, one has

$\frac{\mathrm{d}\xi}{\mathrm{d}\tau}=-4\uppi\gamma,$ (71)

where $\xi$ denotes the expansion of geodesics and $\tau$ affine parameter.
Based on this equation, one can determine that the gravity is attractive
($\gamma>0$) or repulsive ($\gamma<0$).

Further, since $\sigma\sim r^{n}$ with $n\geq 3$ as $r\to 0$, see App. B for
the details, we can deduce that $\sigma^{\prime\prime}>0$ and $\gamma<0$
around $r=0$, namely, the matters generating RBHs are of repulsive
interaction, which is also known as the violation of SEC [14]. Meanwhile,
there is a special point arising from $\sigma^{\prime\prime}(r_{*})=0$, where
$r_{*}$ can be regarded as the point of phase transition. This point is
special because the matters generating RBHs are of repulsive interaction in
the range of $0<r<r_{*}$, while they are of attractive interaction in the
range of $r>r_{*}$, i.e., $r_{*}$ separates the two phases of RBHs.

Now let us estimate the position of $r_{*}$. At first, we note that Eq. (70)
can be regarded as Newton’s equation of one-dimensional particle with mass
$M$, i.e.,

$-\Phi^{\prime}=M\sigma^{\prime\prime},\qquad\Phi^{\prime}\equiv 4\uppi
r\gamma,$ (72)

where $\Phi$ is “potential” and can be solved analytically,

$\Phi=-M\sigma^{\prime}+\Phi_{0}.$ (73)

Here $\Phi_{0}$ is an integration constant and $\Phi-\Phi_{0}<0$ in
$r\in(0,\infty)$ because $\sigma$ is a monotone increasing function of $r$.
Then, considering the asymptotic behaviors of $\sigma$ at $r=0$ and
$r\to\infty$, we obtain

$\lim_{r\to 0}\sigma^{\prime}=0,\qquad\lim_{r\to\infty}\sigma^{\prime}=0,$
(74)

which implies that $\Phi$ is a potential well with one global minimum at
$r=r_{*}$. The reason is that $\sigma$ is a sigmoid function and thus its
first derivative is bell shaped. In other words, we have
$\sigma^{\prime}(r_{*})>\sigma^{\prime}(r_{\rm ext})=1/(2M)$, where
$\sigma^{\prime}(r_{\rm ext})=1/(2M)$ comes from the combination of $T(r_{\rm
ext})=0$ and $f(r_{\rm ext})=0$, but we still cannot determine whether
$r_{*}>r_{\rm ext}$ or $r_{*}<r_{\rm ext}$, where $r_{\rm ext}$ denotes the
radius of extreme RBHs. At last, we know that the temperature of a RBH is
nonnegative and vanishes at $r=r_{\rm ext}$, from which we can deduce
$T^{\prime}(r_{\rm ext})>0$, i.e.,

$\sigma^{\prime\prime}\left(r_{\rm
ext}\right)<\frac{2\sigma^{\prime}\left(r_{\rm ext}\right)}{r_{\rm
ext}}-\frac{2\sigma\left(r_{\rm ext}\right)}{r_{\rm ext}^{2}},$ (75)

then applying $\sigma^{\prime}\left(r_{\rm ext}\right)=1/(2M)$ and
$\sigma\left(r_{\rm ext}\right)=r_{\rm ext}/(2M)$ to replace
$\sigma^{\prime}\left(r_{\rm ext}\right)$ and $\sigma\left(r_{\rm
ext}\right)$, we arrive at

$\sigma^{\prime\prime}\left(r_{\rm ext}\right)<0,$ (76)

which helps us rule out $r_{*}>r_{\rm ext}$.

In summary, we have $r_{*}<r_{\rm ext}$, i.e., the point of phase transition
should be located inside the extreme horizon, which gives us an explanation on
how a repulsive matter forms an attractive black hole. The whole physical
picture should be like this: Along the radial coordinate, the matter first
shows the repulsive interaction around $r=0$; when $r$ passes through the
phase transition point $r_{*}$, the repulsive interaction becomes the
attractive one. Since the thermodynamics describes a BH as a quantum system
from outside, the interaction of RBHs should reflect the attractive nature.
Here we have provided an explanation on how a repulsive matter forms a black
hole with attractive interaction. In addition, we note from Eq. (85),

$\gamma\propto-\left(R-\sqrt{2E}\right),$ (77)

which implies that the sign of $\gamma$ depends on the competition between two
scalar curvatures $R$ and $E$ outside RBHs. When $R<\sqrt{2E}$, the
interaction is attractive, while $R>\sqrt{2E}$ means a repulsive interaction.
The balance $R=\sqrt{2E}$ corresponds to the phase transition point $r_{*}$.

##  7 Summary

Starting from the idea of Gliner vacuum, we apply the approach given in Ref.
[15] to remove the deformations in the 1LM for RHBs. In addition, we provide a
possible explanation on the deformation of mass term. The new 1LM redisplays
the resemblance between RBHs and the traditional thermodynamic systems. In
other words, all the variables in the new 1LM have their own thermodynamic
counterparts, in particular, the area law, $S\propto A$, is recovered. Based
on the reconstructed 1LM, we give a self-consistent theory of Ruppeiner
geometry, and show that all RBHs with a spherical symmetry and a single shape
function should have an attractive interaction in the range of $r\geq r_{\rm
ext}>r_{*}$ from the microscopic perspective. This shows that our analyses of
interactions inside and outside RBHs are consistent with the Ruppeiner
thermodynamic geometry we established in Sec. 5. Furthermore, the new 1LM
offers a universal treatment for both RBHs and SBHs. Finally, the explanation
is given on how a repulsive matter forms a RBH with attractive interaction.
Our result may shed light on solving the related problems in superradiance and
area spectrum for RBHs.

## Acknowledgments

C. Lan would like to acknowledge H. Geng for useful discussions and constant
supports. The authors are also grateful to H. Yang and X.-C. Cai for valuable
remarks and suggestions. This work was supported in part by the National
Natural Science Foundation of China under Grant Nos. 11675081 and 12175108.

##  Appendix A The representation of shape function via curvature invariants

We start with the curvature invariants that are expressed [30] by the shape
function, Eq. (1),

$\displaystyle R$ $\displaystyle=$
$\displaystyle\frac{2M}{r^{2}}\left(2\sigma^{\prime}+r\sigma^{\prime\prime}\right),$
(78) $\displaystyle R_{2}$ $\displaystyle=$
$\displaystyle\frac{2M^{2}}{r^{4}}\left[4(\sigma^{\prime})^{2}+r^{2}(\sigma^{\prime\prime})^{2}\right],$
(79) $\displaystyle K$ $\displaystyle=$
$\displaystyle\frac{4M^{2}}{r^{6}}\Big{\\{}4\left[3\sigma^{2}-4r\sigma\sigma^{\prime}+2r^{2}(\sigma^{\prime})^{2}\right]+4r^{2}\left(\sigma-r\sigma^{\prime}\right)\sigma^{\prime\prime}+r^{4}(\sigma^{\prime\prime})^{2}\Big{\\}}.$
(80)

Although these equation are derived from a RBH, they are valid for all
spherically symmetric BHs with a single shape function. We note that $R$,
$R_{2}$, and $K$ contain $\sigma$ and its first and second derivatives with
respect to $r$. Thus, we can solve these three algebraic equations and express
$\sigma$ and its derivatives in terms of the curvature invariants. By ignoring
redundant roots,444There are four roots originally, two of them are removed by
the weak or null energy condition, $r\sigma^{\prime\prime}\leq
2\sigma^{\prime}$. we obtain

$\displaystyle\sigma$ $\displaystyle=$
$\displaystyle\frac{r^{3}}{24M}\left(R\pm
2\sqrt{3K+R^{2}-6R_{2}}+3\sqrt{4R_{2}-R^{2}}\right),$
$\displaystyle\sigma^{\prime}$ $\displaystyle=$
$\displaystyle\frac{r^{2}}{8M}\left(R+\sqrt{4R_{2}-R^{2}}\right),$
$\displaystyle\sigma^{\prime\prime}$ $\displaystyle=$
$\displaystyle\frac{r}{4M}\left(R-\sqrt{4R_{2}-R^{2}}\right),$ (81)

where the different signs in $\sigma$ correspond to two regions separated by
the line $r^{2}\sigma^{\prime\prime}+6\sigma=4r\sigma^{\prime}$ in the
parameter space. The plus sign depicts the region
$r^{2}\sigma^{\prime\prime}+6\sigma>4r\sigma^{\prime}$, while the minus one
$r^{2}\sigma^{\prime\prime}+6\sigma<4r\sigma^{\prime}$ is more closer to the
centre. On the other side, the Riemann tensor breaks down into three parts in
terms of the Ricci decomposition [16, 34],
$R_{\mu\nu\rho\sigma}=W_{\mu\nu\rho\sigma}+S_{\mu\nu\rho\sigma}+E_{\mu\nu\rho\sigma}$,
where $W_{\mu\nu\rho\sigma}$ is traceless part called the Weyl tensor,
$S_{\mu\nu\rho\sigma}$ is scalar part, and $E_{\mu\nu\rho\sigma}$ is semi-
traceless part. Moreover, we have the following relations,

$E\equiv
E_{\mu\nu\rho\sigma}E^{\mu\nu\rho\sigma}=2R_{2}-\frac{R^{2}}{2},\qquad W\equiv
W_{\mu\nu\rho\sigma}W^{\mu\nu\rho\sigma}=K-2R_{2}+\frac{R^{2}}{3}.$ (82)

Then applying these relations to replace $R_{2}$ and $K$ in Eq. (A), we obtain

$\displaystyle\sigma$ $\displaystyle=$
$\displaystyle\frac{r^{3}}{24M}\left(R+3\sqrt{2E}\pm 2\sqrt{3W}\right),$ (83)
$\displaystyle\sigma^{\prime}$ $\displaystyle=$
$\displaystyle\frac{r^{2}}{8M}\left(R+\sqrt{2E}\right),$ (84)
$\displaystyle\sigma^{\prime\prime}$ $\displaystyle=$
$\displaystyle\frac{r}{4M}\left(R-\sqrt{2E}\right).$ (85)

We can also express the Hawking temperature and Gliner pressure in terms of
the curvature invariants,

$T=\frac{r}{24\uppi}\left(\sqrt{3W}-R\right)\Big{|}_{r=r_{+}},\qquad
P_{+}=-\frac{1}{32\uppi}\left(\sqrt{2E}+R\right)\Big{|}_{r=r_{+}},$ (86)

where the case of the minus sign before $\sqrt{3W}$ in $T$ has been ruled out
due to the positivity of temperature. The zero point of $T$ as a function of
$r_{+}$ corresponds to the solution of algebraic equation $\sqrt{3W}=R$. In
other words, $\sqrt{3W}=R$ signifies the ground state of BH configurations.

##  Appendix B The asymptotic behavior of $\sigma$ around $r=0$

Using Eq. (1), we compute the scalar curvature,

$R(r)=\frac{2M\left(2\sigma^{\prime}+r\sigma^{\prime\prime}\right)}{r^{2}}.$
(87)

If $R(r)$ is finite around $r=0$, its Taylor expansion has the form,

$R(r)=\sum_{n=0}^{\infty}\frac{r^{n}R^{(n)}(0)}{n!}.$ (88)

By solving the ordinary differential equation, Eq. (87), and using the above
Taylor expansion, we obtain a general solution for $\sigma$,

$\tilde{\sigma}=c_{1}+\frac{c_{2}}{r}+r^{3}\sum_{n=0}^{\infty}\frac{r^{n}R^{(n)}(0)}{2M(n+3)(n+4)n!},$
(89)

where $c_{1}$ and $c_{2}$ are two integration constants. Substituting Eq. (89)
into the Kretschmann scalar, we can separate the scalar into two parts, where
one is finite and the other divergent at $r=0$,

$K=K^{\rm fin}+K^{\rm div},$ (90)

with

$\displaystyle K^{\rm fin}$ $\displaystyle=$ $\displaystyle
4\sum_{n=0}^{\infty}\frac{r^{n}R^{(n)}(0)}{(n+3)(n+4)n!}\sum_{m=0}^{\infty}\frac{\left(m^{2}+7m+6\right)r^{m}R^{(m)}(0)}{(m+3)(m+4)m!}$
(91)
$\displaystyle+4\sum_{m=0}^{\infty}\frac{m(2m+3)r^{m}R^{(m)}(0)}{(m+3)(m+4)m!}\sum_{n=0}^{\infty}\frac{nr^{n}R^{(n)}(0)}{(n+3)(n+4)n!}+r^{2}\left(\sum_{n=0}^{\infty}\frac{(n-1)nr^{n}R^{(n)}(0)}{(n+3)(n+4)n!}\right)^{2},$

and

$\displaystyle K^{\rm div}$ $\displaystyle=$
$\displaystyle\frac{8c_{2}M}{r^{3}}\sum_{n=0}^{\infty}\frac{n(n+1)r^{n}R^{(n)}(0)}{(n+3)(n+4)n!}+\frac{8c_{1}M}{r^{4}}\sum_{n=0}^{\infty}\frac{n(3n+5)r^{n}R^{(n)}(0)}{(n+3)(n+4)n!}$
(92)
$\displaystyle+\frac{192c_{1}c_{2}M^{2}}{r^{7}}+\frac{48c_{2}^{2}M^{2}}{r^{6}}+\frac{224c_{1}^{2}M^{2}}{r^{8}}.$

A regular black hole implies $K^{\rm div}=0$, i.e., $c_{1}=0=c_{2}$. As a
result, we derive the asymptotic behavior of the Kretschmann scalar,

$K=K^{\rm
fin}\sim\frac{\left[R(0)\right]^{2}}{6}+\frac{R(0)R^{\prime}(0)}{3}r+O(r^{2}),$
(93)

and the asymptotic behavior of $\sigma$ around $r=0$,

$\sigma=r^{3}\sum_{n=0}^{\infty}\frac{r^{n}R^{(n)}(0)}{2M(n+3)(n+4)n!}\sim\frac{r^{3}R(0)}{24M}+\frac{r^{4}R^{\prime}(0)}{40M}+O(r^{5}).$
(94)

## References

  * [1] George Ruppeiner.  Riemannian geometry in thermodynamic fluctuation theory.  Rev. Mod. Phys., 67:605–659, 1995.  [Erratum: Rev.Mod.Phys. 68, 313–313 (1996)].  doi:10.1103/RevModPhys.67.605.
  * [2] Shao-Wen Wei, Yu-Xiao Liu, and Robert B. Mann.  Repulsive interactions and universal properties of charged Anti–de Sitter black hole microstructures.  Phys. Rev. Lett., 123(7):071103, 2019.  arXiv:1906.10840, doi:10.1103/PhysRevLett.123.071103.
  * [3] S. W. Hawking.  Particle creation by black holes.  Commun. Math. Phys., 43(3):199–220, August 1975.  [Erratum: Commun.Math.Phys. 46, 206 (1976)].  doi:10.1007/BF02345020.
  * [4] Jacob D. Bekenstein.  Black holes and entropy.  Phys. Rev. D, 7:2333–2346, 1973.  doi:10.1103/PhysRevD.7.2333.
  * [5] S. W. Hawking.  Gravitational radiation from colliding black holes.  Phys. Rev. Lett., 26:1344–1346, 1971.  doi:10.1103/PhysRevLett.26.1344.
  * [6] G. W. Gibbons and S. W. Hawking.  Action integrals and partition functions in quantum gravity.  Phys. Rev. D, 15:2752–2756, may 1977.  doi:10.1103/PhysRevD.15.2752.
  * [7] Robert M. Wald.  Black hole entropy is the noether charge.  Phys. Rev. D, 48:R3427–R3431, October 1993.  arXiv:gr-qc/9307038, doi:10.1103/PhysRevD.48.R3427.
  * [8] Ram Brustein, Dan Gorbonos, and Merav Hadad.  Wald’s entropy is equal to a quarter of the horizon area in units of the effective gravitational coupling.  Phys. Rev. D, 79:044025, February 2009.  arXiv:0712.3206, doi:10.1103/PhysRevD.79.044025.
  * [9] Yuan Zhang and Sijie Gao.  First law and Smarr formula of black hole mechanics in nonlinear gauge theories.  Class. Quant. Grav., 35(14):145007, October 2016.  arXiv:1610.01237, doi:10.1088/1361-6382/aac9d4.
  * [10] Piero Nicolini, Anais Smailagic, and Euro Spallucci.  Noncommutative geometry inspired Schwarzschild black hole.  Phys. Lett. B, 632(4):547–551, January 2006.  arXiv:gr-qc/0510112, doi:10.1016/j.physletb.2005.11.004.
  * [11] Atish Dabholkar and Suresh Nampuri.  Quantum black holes.  Lect. Notes Phys., 851:165–232, 2012.  arXiv:1208.4814, doi:10.1007/978-3-642-25947-0_5.
  * [12] Rabin Banerjee, Bibhas Ranjan Majhi, and Saurav Samanta.  Noncommutative Black Hole Thermodynamics.  Phys. Rev. D, 77:124035, 2008.  arXiv:0801.3583, doi:10.1103/PhysRevD.77.124035.
  * [13] E. B. Gliner.  Algebraic Properties of the Energy-momentum Tensor and Vacuum-like States o+ Matter.  Sov. Phys. JETP, 22:378, 1966.
  * [14] Stefano Ansoldi.  Spherical black holes with regular center: A Review of existing models including a recent realization with Gaussian sources.  In Conference on Black Holes and Naked Singularities, 2 2008.  arXiv:0802.0330.
  * [15] Mustapha Azreg-Aïnou.  Black hole thermodynamics: No inconsistency via the inclusion of the missing $P-V$ terms.  Phys. Rev. D, 91:064049, 2015.  arXiv:1411.2386, doi:10.1103/PhysRevD.91.064049.
  * [16] Steven Weinberg.  Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity.  John Wiley and Sons, New York, 1972.
  * [17] Gustavo Melgarejo, Ernesto Contreras, and Pedro Bargueño.  Regular black holes with exotic topologies.  Phys. Dark Univ., 30:100709, 2020.  doi:10.1016/j.dark.2020.100709.
  * [18] Pedro Bargueño.  Some global, analytical and topological properties of regular black holes.  Phys. Rev. D, 102(10):104028, 2020.  arXiv:2008.02680, doi:10.1103/PhysRevD.102.104028.
  * [19] Chen Lan and Yan-Gang Miao.  Entropy and topology of regular black holes.  5 2021.  arXiv:2105.00218.
  * [20] Lev Davidovich Landau and E. M. Lifshits.  Electrodynamics of continuous media, volume 8 of Course of Theoretical Physics.  Fizmatlit, Moscow, 2005.
  * [21] David Kastor, Sourya Ray, and Jennie Traschen.  Enthalpy and the mechanics of AdS black holes.  Class. Quant. Grav., 26(19):195011, April 2009.  arXiv:0904.2765, doi:10.1088/0264-9381/26/19/195011.
  * [22] Brian P. Dolan.  The cosmological constant and black-hole thermodynamic potentials.  Class. Quant. Grav., 28(12):125020, May 2011.  arXiv:1008.5023, doi:10.1088/0264-9381/28/12/125020.
  * [23] A. D. Sakharov.  The initial stage of an expanding Universe and the appearance of a nonuniform distribution of matter.  Sov. Phys. JETP, 22:241, 1966.
  * [24] I. Dymnikova.  Vacuum nonsingular black hole.  Gen. Rel. Grav., 24:235–242, 1992.  doi:10.1007/BF00760226.
  * [25] James M. Bardeen, B. Carter, and S. W. Hawking.  The four laws of black hole mechanics.  Commun. Math. Phys., 31(2):161–170, June 1973.  doi:10.1007/BF01645742.
  * [26] Luka Gulin and Ivica Smolić.  Generalizations of the Smarr formula for black holes with nonlinear electromagnetic fields.  Class. Quant. Grav., 35(2):025015, October 2017.  arXiv:1710.04660, doi:10.1088/1361-6382/aa9dfd.
  * [27] I. Dymnikova and B. Soltysek.  Spherically symmetric space-time with two cosmological constants.  Gen. Rel. Grav., 30:1775–1793, 1998.  doi:10.1023/A:1026619228583.
  * [28] Sean A. Hayward.  Formation and evaporation of nonsingular black holes.  Phys. Rev. Lett., 96:031103, January 2006.  arXiv:gr-qc/0506126, doi:10.1103/PhysRevLett.96.031103.
  * [29] James M Bardeen.  Non-singular general-relativistic gravitational collapse.  In Conference Proceedings of GR5, Tbilisi, USSR, volume 174, 1968\.
  * [30] Leonardo Balart and Elias C. Vagenas.  Regular black holes with a nonlinear electrodynamics source.  Phys. Rev. D, 90:124045, December 2014.  arXiv:1408.0306, doi:10.1103/PhysRevD.90.124045.
  * [31] Karl Schwarzschild.  On the gravitational field of a sphere of incompressible fluid according to Einstein’s theory.  Sitzungsber. Preuss. Akad. Wiss. Berlin (Math. Phys. ), 1916:424–434, 1916.  arXiv:physics/9912033.
  * [32] Shao-Wen Wei, Yu-Xiao Liu, and Robert B. Mann.  Ruppeiner geometry, phase transitions, and the microstructure of charged AdS black holes.  Phys. Rev. D, 100(12):124033, 2019.  arXiv:1909.03887, doi:10.1103/PhysRevD.100.124033.
  * [33] Sean M. Carroll.  Spacetime and Geometry.  Cambridge University Press, 7 2019.
  * [34] Robert M. Wald.  General Relativity.  University of Chicago Press, Chicago, USA, 1984.  doi:10.7208/chicago/9780226870373.001.0001.


# Label Agnostic Pre-training for Zero-shot Text Classification

Christopher Clarke Yuzhao Heng Yiping Kang Krisztian Flautner
Lingjia Tang Jason Mars
Computer Science & Engineering
University of Michigan
Ann Arbor, MI
{csclarke, stefanhg, ypkang, manowar, lingjia<EMAIL_ADDRESS>

###### Abstract

Conventional approaches to text classification typically assume the existence
of a fixed set of predefined labels to which a given text can be classified.
However, in real-world applications, there exists an infinite label space for
describing a given text. In addition, depending on the aspect (sentiment,
topic, etc.) and domain of the text (finance, legal, etc.), the interpretation
of the label can vary greatly. This makes the task of text classification,
particularly in the zero-shot scenario, extremely challenging. In this paper,
we investigate the task of zero-shot text classification with the aim of
improving the ability of pre-trained language models (PLMs) to generalize to
both seen and unseen data across varying aspects and domains. To solve this we
introduce two new simple yet effective pre-training strategies, Implicit and
Explicit pre-training. These methods inject aspect-level understanding into
the model at train time with the goal of conditioning the model to build task-
level understanding. To evaluate this, we construct and release UTCD, a new
benchmark dataset for evaluating text classification in zero-shot settings.
Experimental results on UTCD show that our approach achieves improved zero-
shot generalization on a suite of challenging datasets across an array of
zero-shot formalizations.

##  1 Introduction

Text classification is the process of categorizing text into sets of organized
groups where each set consists of similar content in a well-defined manner
Minaee et al. (2021); Joulin et al. (2016). Supervised approaches have
achieved great success in recent years due to the availability of rich
training data and the advent of large pre-trained language models such as BERT
Devlin et al. (2018). These conventional approaches typically assume the
presence of a pre-defined set of labels to which a given text can be
classified. However, in real-world applications, several challenges emerge:

1) The label space is constantly evolving. Over time, new labels are
constantly emerging and the definition of the label space is constantly being
refined. For example, intent classification systems such as those used in
chatbots and dialogue systems are constantly introducing new intents as their
range of supported features increases. Social networks such as Twitter
encounter new and emerging topics on a daily basis from massive amounts of
content that need to be classified. Figure 1 shows an example of this emerging
label space.

2) The range of applications for text classification is vast. Text
classification is pivotal to many different application areas from sentiment
analysis to topic labeling, etc, and is used in a variety of domains such as
finance, health, etc. When applied to this conglomeration of uses, it is
typically assumed that there exists a comprehensive dataset of well-defined
text-label pairs for each use case. However, in many real-world settings,
annotated data is either scarce or unavailable entirely. Additionally, the use
of dedicated models for each task is impractical due to the additional compute
overhead and maintenance, thus making it difficult to scale over time.

Figure 1: Zero-shot Text Classification Problem: In real-world applications,
the model needs to adapt to unseen labels. For a given aspect and domain, the
interpretation of a given text-label pair can vary greatly.

Zero-shot learning (ZSL) is aimed at addressing these constraints. Zero-shot
Learners are models capable of predicting unseen classes. When applied to text
classification, these models aim to associate a piece of text with a given
label without the need for having been trained on that label. However, despite
recent advancements in the capabilities of PLMs, zero-shot models still vastly
underperform their supervised counterparts Pushp and Srivastava (2017); Puri
and Catanzaro (2019); Brown et al. (2020). As such, this remains an open
research problem.

In this paper, we investigate the challenge of reducing the aforementioned
performance gap present in these zero-shot models compared to their supervised
counterparts on unseen data. We theorize that the poor generalization of these
zero-shot models is due to their lack of aspect-level understanding during
their training process. To alleviate this we introduce two new simple yet
effective pre-training strategies, Implicit and Explicit pre-training which
specifically inject aspect-level understanding into the model.

In order to evaluate these strategies, we canvas the range of zero-shot
formalizations for enabling zero-shot text classification on PLMs and apply
our techniques. Additionally, we introduce the Universal Text Classification
Dataset (UTCD), a large-scale text classification dataset for evaluating zero-
shot text classification. UTCD is a compilation of 18 classification datasets
spanning 3 main aspects of Sentiment, Intent/Dialogue, and Topic
classification. Our results on UTCD show that by employing both our implicit
and explicit pre-training strategies we can achieve improved zero-shot
performance on a suite of challenging datasets for which the model was not
trained on.

Specifically, this paper makes the following contributions:

  * •

We introduce Implicit & Explicit pre-training, two new simple yet effective
pre-training strategies for improving zero-shot performance.

  * •

We construct and release UTCD, a new benchmark dataset for evaluating text
classification systems across a suite of diverse tasks and domains. We release
our models and dataset111https://github.com/ChrisIsKing/zero-shot-text-
classification.

  * •

We conduct a thorough evaluation of various zero-shot text classification
formalizations showing the effectiveness of our training strategies on each as
well as insights gained.

Figure 2: Zero-shot Text Classification Formalizations: Part (a) illustrates
the binary classification formalization described in section 2 where
concatenated <text, label> pairs are passed as input to the model. Part (b)
illustrates dual encoding where text label pairs are encoded separately and
scored via a distance metric. Part (c) illustrates text classification where
the model generates desired label based on a natural language instruction
template.

##  2 Task Formulation

In this section, we introduce the task of zero-shot text classification and
describe a set of formalizations for facilitating the classification of text
in a zero-shot manner, i.e. being able to predict unseen labels.

#### Conventional Text Classification

Text classification approaches using PLMs assume the existence of a pre-
defined set of labels $\left\\{y_{i}\right\\}_{n}^{1}$ where for a given input
sequence $X$, the model outputs a representation of that sequence as a
sequence of hidden states $\left\\{h_{i}\right\\}_{l}^{1}$. Hidden states in
the final layer are pooled to a single vector. In the case of BERT Devlin et
al. (2018), the $[\mathrm{CLS}]$ token is taken, and a linear softmax layer is
added to predict the probability distribution of the label set:

$\vec{\mathrm{P}}\left(\left\\{y_{i}\right\\}_{n}^{1}\big{|}~{}h\right)=\mathrm{softmax}(Wh)$
(1)

For the zero-shot scenario, this approach breaks since the output class set
$\left\\{y_{i}\right\\}_{n}^{1}$ is fixed. This prevents the classification of
text to new labels unless the model is re-trained with the new label set or a
mapping of existing labels to unseen labels is built, both of which are
impractical and cumbersome for real-world scenarios.

###  2.1 Binary Zero-shot Classification

To facilitate zero-shot classification of PLMs, Halder et al. (2020); Pushp
and Srivastava (2017); Yin et al. (2019) formulate text classification as a
series of binary classification tasks:

$f(\mathrm{label}(y_{i}),x)=\mathrm{P}\left(\mathrm{True}~{}|~{}y_{i},x\right)\
$ (2)

The model is provided with a concatenation of the class label
$\mathrm{label}(y_{i})$ and input text and the output layer generates a binary
$\mathrm{True}/\mathrm{False}$ prediction with a confidence score
$\mathrm{P}$. The $\mathrm{True}$-prediction class with the highest confidence
is selected as the final prediction, that is,

$\hat{y}=\underset{i\;\in\;\\{1\ldots
n\\}}{\operatorname{arg}\operatorname{max}}\;f(\mathrm{label}(y_{i}),x)$ (3)

where $n$ is the number of classes/labels. Such cross-attention (CA) models
apply attention layers on the text and labels jointly, which intuitively
allows for rich interactions. This architecture is shown in part (a) of Figure
2.

###  2.2 Dual Encoding Zero-shot Classification

In contrast to cross-attention based architectures, Dual Encoder models
(Reimers and Gurevych, 2019; Casanueva et al., 2020a; Clarke et al., 2022)
instead focus on learning representations for a given text and label
independently. They separately embed the text and label, via an encoder $\Phi$
and compute pair-wise scores $S$ based on the encoded representations with a
distance metric $Dist$, such as dot-product or cosine similarity:

$S(x,y_{i})=Dist\left(\Phi(x),\Phi(y_{i})\right)$ (4)

Sentence-Bert (Reimers and Gurevych, 2019) takes PLMs such as BERT and RoBERTa
as the base encoder and use siamese networks to derive sentence embeddings by
comparing similarities between sentence pairs as shown in part (b) of Figure
2. For text classification, this architecture can be used to derive latent
representations for a given text and label and classify a sequence $x$
according to:

$\hat{y}=\underset{i\;\in\;\\{1\ldots
n\\}}{\operatorname{arg}\operatorname{max}}\;S(x,y_{i})$ (5)

###  2.3 Generative Classification

Lastly, the generative formulation of zero-shot text classification uses
autoregressive language models by passing in text and label sets as natural
language prompts and training the model to generate the target label token by
token. As described in Puri and Catanzaro (2019), we reformulate the text
classification problem as a multiple choice question answering problem. The
model is provided with a multiple-choice question description containing each
class label in natural language, and trained to generate the correct answer,
as shown in part (c) of Figure 2. The intuition behind this approach is to
train the model to use common sense reasoning to select the most probable
description of the text data from a provided list of rich natural language
classes. Given some input text $t$, the model is optimized with the next token
prediction language modeling loss:

$\sum_{t}\mathcal{L}(w_{t},P(\hat{w}_{t}|w_{[1,t-1]}))$ (6)

Figure 3: Zero-shot Text Classification Training Strategies. Part (a) shows
standard model training where a text and the set of label options are passed
to the model. Part (b) illustrates implicit training where the aspect is
additionally passed as input. Part (c) shows injecting aspect knowledge to the
model explicitly through gradient update, to initialize subsequent training.

##  3 Method

In this section, we outline the methodology for our Implicit & Explicit pre-
training strategies which allow us to inject aspect-specific knowledge into
PLMs to improve generalization to unseen data. We first define the term aspect
and outline the gap between the performance of the zero-shot models shown in
section 2 on seen data compared to that of unseen data. Lastly, we describe
our intuition behind why localization of aspect knowledge helps to bridge this
gap.

#### Aspect Definition

In the scope of this work, we define an aspect as the type of task to which a
given set of datasets belong too. For example, sentiment is considered an
aspect because it cleanly defines a task definition of understanding the
emotion conveyed in a given text. This definition holds true even if the
domain of the data changes. e.g sentiment detection of news data vs sentiment
of social media tweets. In addition to having a clean task definition, we
stipulate that the set of labels considered in a given aspect must convey that
aspect. e.g For intent, the label "turn off alarm" conveys that the text
describes the intention to do something.

###  3.1 Transfer Learning for Text Classification

The prevailing method for training models to perform classification tasks is
to add a linear head on top of a pre-trained language model and fine-tune the
entire network on labeled data Devlin et al. (2018). However, when scaled to
multi-task, multi-domain applications these models suffer from issues such as
catastrophic forgetting and conflicting knowledge transfer across tasks
Aribandi et al. (2021); Geva et al. (2021); Clark et al. (2019); Alonso and
Plank (2016). We observe a similar trend in the Bert Seq-CLS row of Table 3
and 2, where despite the overarching task of text classification remaining the
same when scaling the output space of the classification head to more labels
across aspects, we see heavy performance degradation compared to having
individual dataset models. For example, in table 3 training a multi-dataset
BERT sequence classifier performs worse for every benchmark dataset compared
to its single-dataset counterpart. Additionally, for the zero-shot
formalizations, we observe the lowest positive transfer on datasets with the
lowest level of token overlap between labels seen during training and out-of-
domain labels, as shown in Figure 4. We theorize that the reason for this
phenomenon is that the model is over-fitting to the specific labels seen
during training instead of generalizing to the "aspect".

Dataset |  Aspect |  Train/Test |  #labels
---|---|---|---
in-domain
GoEmotions | sentiment | 43K/5.4K | 28
TweetEval | sentiment | 45K/12K | 3
Emotion | sentiment | 16K/2K | 6
SGD | intent | 16K/4.2K | 26
Clinc-150 | intent | 15K/4.5K | 150
SLURP | intent | 12K/2.6K | 75
AG News | topic | 120K/7.6K | 4
DBpedia | topic | 560K/70K | 14
Yahoo | topic | 1.4M/60K | 10

Dataset |  Aspect |  Train/Test |  #labels
---|---|---|---
out-of-domain
Amazon Polarity | sentiment | 3.6M/400K | 2
Fin. Phrase Bank | sentiment | 1.8k/453 | 3
Yelp | sentiment | 650K/50K | 3
Banking77 | intent | 10K/3.1K | 77
SNIPS | intent | 14K/697 | 7
NLU Eval | intent | 21K/5.2K | 68
MultiEURLEX | topic | 55K/5K | 21
Patent | topic | 25K/5K | 9
Consumer Finance | topic | 630K/160K | 18

Table 1: Universal Text Classification Dataset (UTCD) consists of the
following datasets: Demszky et al. (2020); Barbieri et al. (2020); Saravia et
al. (2018); Rastogi et al. (2020); Larson et al. (2019); Bastianelli et al.
(2020); Zhang et al. (2015); Auer et al. (2007); Malo et al. (2014); Casanueva
et al. (2020b); Coucke et al. (2018); Xingkun Liu and Rieser (2019); Chalkidis
et al. (2021); Sharma et al. (2019); Bureau (2012)

###  3.2 Implicit Training

In order to introduce aspect specification into our zero-shot models, we take
inspiration from T5’s Raffel et al. (2019) text-to-text framework for multi-
task generalization. In this framework, the model is fed some text for context
and is then asked to produce some output text. As an example, to ask the model
to translate the sentence “That is good.” from English to German, the model
would be fed the sequence “translate English to German: That is good.” and
would be trained to output “Das ist gut.” Similarly, for each aspect (as
defined in section 4), we introduce a conditional aspect token to the model
input that acts as a context for that specific aspect. As such, in addition to
learning the best contextual representation for the <text, label> input pair,
the model implicitly learns a higher level understanding of the underlying
aspect. By adding this conditional representation, even as the label space
changes, the model is better able to understand the aspect at hand. This is
shown in part(b) of figure 3. In the case of implicit binary zero-shot
classification, the model is additionally provided with a concatenation of the
aspect token and the output is selected as:

$\hat{y}=\underset{i\in\\{1\ldots
n\\}}{\operatorname{arg}\operatorname{max}}\;f(\mathrm{label}(y_{i}),\mathrm{aspect}(a_{y_{i}}),x)$
(7)

###  3.3 Explicit Training

Given our hypothesis that these language models will be able to generalize to
unseen labels as a result of implicitly learning the task at hand, we explore
the idea of explicitly training this generalization in a supervised manner.
Instead of adding a conditional aspect token, we add an additional pre-
training step in which the model is trained on aspect detection. This step
acts as an initialization process whereby the model representations are tuned
at the aspect level first. Once this step is completed the model is then fine-
tuned for its respective zero-shot classification objective. This process is
shown in part (c) of figure 3. For a given text $x$ this explicit training
step is defined as:

$\vec{\mathrm{P}}\left(\left\\{a_{j}\right\\}_{m}^{1}\big{|}~{}h\right)=\mathrm{softmax}(Wh)$
(8)

##  4 UTCD: Universal Text Classification Dataset

In order to test the zero-shot generalization of these NLP models we introduce
UTCD. UTCD is a compilation of 18 classification datasets spanning 3 main
aspects of Sentiment, Intent/Dialogue, and Topic classification. A breakdown
of each dataset is provided in appendix A. UTCD focuses on the task of zero-
shot text classification where the candidate labels are descriptive of the
text being classified. To make NLP models more broadly useful, zero-shot
techniques need to be capable of label, domain & aspect transfer. As such, in
the construction of UTCD we enforce the following principles:

#### Textual labels

In UTCD, we mandate the use of textual labels. While numerical label values
are often used in classification tasks, descriptive textual labels such as
those present in the datasets across UTCD enable the development of techniques
that can leverage the class name which is instrumental in providing zero-shot
support. As such, for each of the compiled datasets, labels are standardized
such that the labels are descriptive of the text in natural language.

#### Diverse domains and Sequence lengths

In addition to broad coverage of aspects, UTCD compiles diverse data across
several domains such as Banking, Finance, Legal, etc each comprising varied
length sequences (long and short). The datasets are listed in Table 1.

As described in section 3, we define aspect as the sub-task type to which a
given set of datasets can belong too. We simulate the Zero-shot learning case
by splitting UTCD into in-domain, data a given model would be trained on, and
out-of-domain, data with novel classes unseen during training. Additionally,
to prevent data imbalance across aspects, we sub-sample the in-domain datasets
such that the total number of unique text in each aspect is the same while
maintaining class label distribution for each dataset. Class imbalance is
known to degrade performance in deep learning models Buda et al. (2018); Ochal
et al. (2021). We observe a similar trend where aspect normalization results
in performance improvement.

Model |  Training Strategy | Sentiment | Intent | Topic |  Average
---|---|---|---|---|---
Amazon Polarity |  Fin. Phrase Bank |  Yelp |  Banking 77 |  SNIPS |  NLU Eval |  Multi EURLEX |  Patent |  Consumer Finance
BERT Seq-CLS* |  individual | 96.0 | 97.2 | 84.8 | 88.6 | 99.0 | 88.9 | 94.8 | 64.1 | 82.6 | 88.4
full | 93.1 | 24.9 | 79.0 | 84.7 | 97.3 | 87.4 | 81.4 | 50.2 | 76.9 | 75.0
Binary BERT | vanilla | 80.7 | 68.9 | 58.5 | 51.4 | 82.9 | 71.6 | 28.7 | 13.6 | 22.3 | 53.2
implicit (ours) | 80.1 | 66.0 | 59.8 | 51.3 | 82.5 | 73.1 | 30.3 | 15.2 | 23.4 | 53.5
explicit (ours) | 76.1 | 66.7 | 56.0 | 49.8 | 83.8 | 69.6 | 44.5 | 19.5 | 30.2 | 55.1
Bi- Encoder | vanilla | 69.9 | 71.7 | 46.5 | 9.4 | 70.4 | 71.1 | 33.5 | 11.7 | 18.4 | 44.7
implicit (ours) | 79.6 | 64.0 | 56.8 | 21.1 | 72.5 | 61.9 | 35.4 | 9.6 | 11.3 | 45.8
explicit (ours) | 71.5 | 63.6 | 52.1 | 9.7 | 71.9 | 70.0 | 27.4 | 9.3 | 27.0 | 44.7
GPT-2$\dagger$ | vanilla | 88.3 | 71.1 | 70.9 | 22.8 | 52.2 | 61.7 | 22.3 | 23.5 | 12.6 | 47.3
implicit (ours) | 89.3 | 61.4 | 71.9 | 16.5 | 33.7 | 63.1 | 18.6 | 25.8 | 12.2 | 43.6
explicit (ours) | 89.7 | 75.9 | 71.5 | 22.4 | 54.1 | 60.7 | 23.5 | 21.6 | 13.9 | 48.2
BART$\ddagger$ |  Zero-shot | 91.0 | 40.2 | 75.2 | 42.2 | 61.4 | 40.1 | 19.8 | 8.9 | 24.6 | 44.8
GPT-3$\ddagger$ |  Zero-shot | 54.4 | 52.8 | 77.0 | 23.7 | 13.9 | 37.9 | - | - | - | 43.3
Table 2:  Aspect-Normalized out-of-domain accuracy. *Supervised upper bound,
not a zero-shot framework. $\dagger$In case none of the given labels are
generated at inference, the generated text is embedded and compared with label
embeddings. $\ddagger$Out-of-the-box zero-shot classifier.

##  5 Experimental Setup

#### Model Architectures

For binary classification, we use BERTBASE with sentence pair classification
as in Devlin et al. (2018). For dual encoding classification, we use Sentence-
BERT (Reimers and Gurevych, 2019) with BERTBASE as the base encoder, mean
pooling, and cosine similarity as the distance metric. For generative
classification, we use the 345M GPT-2 (Radford et al., 2019) as the language
model and the input representation described in Puri and Catanzaro (2019).
These models are denoted Binary BERT, Bi-Encoder, and GPT-2 respectively.

#### Training

We train all models with AdamW (Loshchilov and Hutter, 2019) and weight decay
of 0.01 on all in-domain data for 3 epochs, for both pre-training and fine-
tuning stages. For explicit pre-training, we use a learning rate of
$2\text{e-}5$, batch size of 16, and linear learning rate warmup over the
first 10% steps with a cosine schedule. For binary and dual encoding we use a
learning rate of $2\text{e-}5$, batch size of 16, with 10% warmup and a linear
schedule. For generative classification fine-tuning, we use a learning rate of
$4\text{e-}5$, batch size of 128, with 1% warmup and a cosine schedule as
reported in Puri and Catanzaro (2019). We pre-process data and train all
models with different random seeds over multiple runs.

Model |  Training Strategy | Sentiment | Intent | Topic |  Average
---|---|---|---|---|---
Go Emotions |  Tweet Eval |  Emotion |  SGD |  Clinc -150 |  SLURP |  AG News |  DBpedia |  Yahoo
BERT Seq-CLS* |  individual | 63.0 | 69.5 | 92.9 | 78.7 | 95.2 | 85.5 | 94.1 | 99.2 | 73.4 | 83.4
full | 56.7 | 55.4 | 91.1 | 80.5 | 82.9 | 77.3 | 86.7 | 98.6 | 66.6 | 77.3
Binary BERT | vanilla | 59.3 | 67.6 | 92.4 | 91.5 | 87.8 | 81.8 | 90.0 | 98.9 | 67.9 | 81.9
implicit (ours) | 59.9 | 67.2 | 91.8 | 93.5 | 87.1 | 81.8 | 89.2 | 98.9 | 68.1 | 82.0
explicit (ours) | 60.2 | 66.6 | 91.8 | 93.4 | 88.0 | 80.4 | 88.7 | 98.9 | 67.8 | 81.7
Bi- Encoder | vanilla | 59.2 | 65.7 | 92.8 | 82.2 | 84.4 | 79.9 | 89.3 | 99.0 | 67.4 | 80.0
implicit (ours) | 56.9 | 66.0 | 90.9 | 81.3 | 82 | 78.9 | 88.8 | 99.0 | 67.9 | 79.1
explicit (ours) | 58.8 | 66.8 | 91.8 | 82.7 | 83.3 | 79.9 | 89.5 | 98.9 | 67.7 | 80.0
GPT-2$\dagger$ | vanilla | 58.8 | 70.6 | 75.9 | 84.2 | 81.4 | 75.3 | 86.7 | 98.5 | 68.3 | 77.7
implicit (ours) | 59.0 | 70.3 | 71.4 | 84.7 | 81.7 | 73.1 | 87.7 | 98.4 | 68.3 | 77.2
explicit (ours) | 60.1 | 70.1 | 76.4 | 84.3 | 81.9 | 76.7 | 87.9 | 98.6 | 68.6 | 78.3
BART$\ddagger$ |  Zero-shot | 24.2 | 47.8 | 37.7 | 41.4 | 50.4 | 27.5 | 71.7 | 65 | 49.2 | 46.1
Table 3:  Aspect-Normalized in-domain accuracy.

##  6 Results & Discussion

In this section we present and analyze the results of our experiments,
detailing our insights and discussing the implications of each of our
techniques.

#### Evaluation Task

We report accuracy on the test set of all in-domain and out-of-domain
datasets. In multi-label cases where there is more than one valid label, the
prediction is considered correct if the model predicts any one of the correct
labels. For generative classification, we observe instances in which GPT-2 may
not generate one of the label options, a known problem for PLM generation
(Radford and Narasimhan, 2018; Pascual et al., 2021). In such cases, we
consider the label option most similar to the generated answer as prediction,
by mapping the generated output and the valid classes to an embedding space.
For this encoding, we use the pre-trained model MPNet (Song et al., 2020) with
mean pooling encoder from Sentence-BERT (Reimers and Gurevych, 2019) for
mapping the labels and cosine similarity as the distance metric. This ensures
the consistency of GPT-2’s output with the other zero-shot formalizations.

#### Upper-bound & Zero-shot Baselines

To gauge the ability of our models to generalize to unseen data, we establish
our upper-bound as the performance of a fully supervised model on the target
data. Specifically, we fine-tune two variations of BERTBASE for sequence
classification which we denote as "individual" and "full". For individual, we
fine-tune a dedicated classification model for each dataset in UTCD. For full,
we fine-tune a single model for all datasets. Additionally, we compare the
zero-shot performance of our models to the popular LLM GPT-3 Brown et al.
(2020), and BART MNLI Yin et al. (2019) which is the most popular and widely
downloaded zero-shot model on Huggingface
Hub222https://huggingface.co/facebook/bart-large-mnli.

###  6.1 Out-of-domain Performance

In table 2, we report results on the out-of-domain test set for UTCD. To
evaluate the ability of our zero-shot models to adapt to unseen data, we
evaluate our fine-tuned models from table 3 on the out-of-domain test set
without training on any out-of-domain data. Across the zero-shot
formalizations, we observe that our explicit Binary BERT achieves the best
performance with a 2% increase over its vanilla counterpart. Thus showing the
power of the explicit pre-training strategy for binary classification
formalization.

When compared to the "full" supervised out-of-domain model, despite having not
been trained on any data from the target dataset, across the aspects of
sentiment and intent, our models are able to generalize well. Specifically,
across all formalizations, our models are able to outperform the supervised
model on the financial phrase bank dataset. We observe that this drop is due
to conflicting domain data. UTCD’s out-of-domain set consists of similar
financial datasets in the other aspects of intent and topic. Given that
examples from the finance phrase banks dataset are general in nature, without
seeing the label, it is difficult for the sequence classifier to understand
the task at hand, thus causing it to classify to conflicting labels from
similar datasets. This showcases the need to include aspect-specific
knowledge.

Lastly, when inspecting the performance of vanilla fine-tuning compared
implicit and explicit training, we are able to outperform vanilla on
generalizing to unseen data on 6, 6, and 8 of the 9 datasets in out-of-domain
UTCD across Binary BERT, Bi-encoder, and GPT-2 models respectively. In
particular, for explicit training on Binary BERT, we achieve a massive
improvement in zero-shot generalization (as much as +%16 for the topic aspect,
+9% on average). Additionally, in comparison to the massive zero-shot
baselines of BART and GPT-3 our models are able to outperform on 7 and 8 of
the 9 datasets respectively.

###  6.2 In-domain Performance

In table 3, we report results on the in-domain test set for UTCD. For in-
domain, we conduct implicit & explicit training across each zero-shot
formalization. We observe that when compared with the "full" supervised model,
our zero-shot models are more performant while maintaining the flexibility of
facilitating zero-shot. When compared with the "individual" variation, as our
zero-shot models are trained jointly across different datasets, we achieve
better performance than the single supervised model on datasets such as SGD,
showing the power of knowledge transfer from other intent datasets such as
Clinc-150 & SLURP.

For vanilla fine-tuning without implicit or explicit training, we observe that
across zero-shot formalizations, injecting task specification through implicit
and explicit pre-training preserves performance for in-domain data. Showing
that while achieving better zero-shot transfer ability our models do not
suffer performance loss on data already seen during training.

###  6.3 Importance of Label token overlap

In addition to the need for aspect-specific knowledge, we also observe a high
correlation in zero-shot generalization results between the overlap of tokens
seen during training and those evaluated on the out-of-domain test. Figure 4
shows the pairwise overlap of label tokens across the in-domain and out-of-
domain datasets. When inspected across aspects, we see that our models are
able to achieve the best out-of-domain performance on datasets with the most
overlapping label tokens to those seen during training.

##  7 Related Work

Zero-shot text classification is the task of classifying text into novel
categories unseen during training. Early zero-shot classification studies
frame the problem as binary classification on whether a given label describes
the text (Pushp and Srivastava, 2017; Yin et al., 2019). With the advancement
of PLMs, subsequent works (Yin et al., 2019; Puri and Catanzaro, 2019) rely on
transformer architectures to learn representations from descriptive labels
passed in. In particular, Puri and Catanzaro (2019) fine-tune an
autoregressive language model to generate titles based on a prompt template
containing Tweet articles and a list of title options. Though the model is
trained on a great variety of title options, the approach limits the learning
to topic classification only, as the authors only analyze performance on topic
datasets, unlike our approach which considers a wide array of aspects, each
requiring focus on different sections of a given text.

Yin et al. (2019) similarly categorize zero-shot text classification by
aspects and implicitly introduce aspects during training with a dedicated
template for each aspect. They further propose the classification of a text,
label pair as a logic entailment problem. However, the authors analyze a less
challenging zero-shot case where a model is trained on a subset of text, label
pairs, and evaluated on the remaining text with unseen labels in the same
domain. Additionally, the authors introduce WordNet definition of the labels
as the labels are all single words. This process requires manual intervention
and is not applicable for multiple-word label sequences common in intent
classification, such as "Check Balance". Our work evaluates a more diverse set
of datasets for each aspect and a more comprehensive set of zero-shot
architectures.

Figure 4: UTCD Out-of-domain Dataset Label Pair-wise Overlap with In-domain
Dataset. 0 is no overlap, 100 is exactly the same label set. From sentiment to
intent to topic, label overlap decreases in general.

##  8 Conclusion

In this paper, we investigate the task of zero-shot text classification with
the aim of improving the ability of PLMs to generalize both seen and unseen
data across domains without the need for additional training. We introduce two
new simple yet effective pre-training strategies, Implicit training & Explicit
pre-training which specifically inject aspect-level understanding into the
model at train time. To evaluate this, we release UTCD, a new benchmark
dataset for evaluating text classification in zero-shot settings. Experimental
results on UTCD show that our approach achieves improved zero-shot
generalization on a suite of challenging datasets in UTCD and across many
zero-shot formalizations.

##  9 Limitations

While our approach is shown to be effective in improving the zero-shot
adaption ability of these PLMs, the scope of this work has only been extended
to English languages and has not been tested on other languages. In addition,
another limitation of this work is the scope of the aspect. Aspect is defined
across 3 main categories of intent, sentiment, and topic in the work. However,
given the massive space of text label interpretations, our aspect range can be
refined and expanded even further, lending to more analysis of the stability
of implicit & explicit training as the number of aspects grows. We do not
investigate this scenario in this work.

## Acknowledgements

We thank our anonymous reviewers for their feedback and suggestions. This work
is supported in part by award NSF1539011 by the National Science Foundation.

## References

  * Alonso and Plank (2016) Héctor Martínez Alonso and Barbara Plank. 2016.  When is multitask learning effective? semantic sequence prediction under varying data conditions.
  * Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021.  Ext5: Towards extreme multi-task scaling for transfer learning.
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _The Semantic Web_ , pages 722–735, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  TweetEval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650, Online. Association for Computational Linguistics.
  * Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020\.  SLURP: A spoken language understanding resource package.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7252–7262, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Buda et al. (2018) Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. 2018.  A systematic study of the class imbalance problem in convolutional neural networks.  _Neural Networks_ , 106:249–259.
  * Bureau (2012) Consumer Financial Protection Bureau. 2012.  Consumer complaint database.  https://www.consumerfinance.gov/data-research/consumer-complaints/.  Accessed: Jun. 24th, 2022.
  * Casanueva et al. (2020a) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020a.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45, Online. Association for Computational Linguistics.
  * Casanueva et al. (2020b) Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020b.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020_.  Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
  * Chalkidis et al. (2021) Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021.  Multieurlex – a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019.  BAM! born-again multi-task networks for natural language understanding.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5931–5937, Florence, Italy. Association for Computational Linguistics.
  * Clarke et al. (2022) Christopher Clarke, Joseph Peper, Karthik Krishnamurthy, Walter Talamonti, Kevin Leach, Walter Lasecki, Yiping Kang, Lingjia Tang, and Jason Mars. 2022.  One agent to rule them all: Towards multi-agent conversational AI.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3258–3267, Dublin, Ireland. Association for Computational Linguistics.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _ArXiv_ , abs/1805.10190.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan S. Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  Goemotions: A dataset of fine-grained emotions.  _ArXiv_ , abs/2005.00547.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Geva et al. (2021) Mor Geva, Uri Katz, Aviv Ben-Arie, and Jonathan Berant. 2021.  What’s in your head? emergent behaviour in multi-task transformer models.
  * Halder et al. (2020) Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020.  Task aware representation of sentences for generic text classification.  In _COLING 2020, 28th International Conference on Computational Linguistics_.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Malo et al. (2014) P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014.  Good debt or bad debt: Detecting semantic orientations in economic texts.  _Journal of the Association for Information Science and Technology_ , 65.
  * Minaee et al. (2021) Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021.  Deep learning–based text classification: A comprehensive review.  _ACM Comput. Surv._ , 54(3).
  * Ochal et al. (2021) Mateusz Ochal, Massimiliano Patacchiola, Amos Storkey, Jose Vazquez, and Sen Wang. 2021.  Few-shot learning with class imbalance.
  * Pascual et al. (2021) Damian Pascual, Béni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. 2021.  A plug-and-play method for controlled text generation.  _ArXiv_ , abs/2109.09707.
  * Puri and Catanzaro (2019) Raul Puri and Bryan Catanzaro. 2019.  Zero-shot text classification with generative language models.  _ArXiv_ , abs/1912.10165.
  * Pushp and Srivastava (2017) Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. 2017.  Train once, test anywhere: Zero-shot learning for text classification.  _CoRR_ , abs/1712.05972.
  * Radford and Narasimhan (2018) Alec Radford and Karthik Narasimhan. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8689–8696.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Saravia et al. (2018) Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018\.  CARER: Contextualized affect representations for emotion recognition.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3687–3697, Brussels, Belgium. Association for Computational Linguistics.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  BIGPATENT: A large-scale dataset for abstractive and coherent summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2204–2213, Florence, Italy. Association for Computational Linguistics.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 16857–16867. Curran Associates, Inc.
  * Xingkun Liu and Rieser (2019) Pawel Swietojanski Xingkun Liu, Arash Eshghi and Verena Rieser. 2019.  Benchmarking natural language understanding services for building conversational agents.  In _Proceedings of the Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS)_ , pages xxx–xxx, Ortigia, Siracusa (SR), Italy. Springer.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.

##  Appendix A UTCD Datasets

UTCD is a compilation of 18 classification datasets spanning 3 categories of
Sentiment, Intent/Dialogue and Topic classification. UTCD focuses on the task
of zero-shot text classification where the candidate labels are descriptive of
the text being classified. UTCD consists of 6M/800K train/test examples.

For sentiment we have the datasets Go Emotion Demszky et al. (2020), TweetEval
Barbieri et al. (2020), Emotion Saravia et al. (2018), Amazon Polarity Zhang
et al. (2015), Finance Phrasebank Malo et al. (2014) and Yelp Zhang et al.
(2015). The GoEmotions dataset contains 58k carefully curated Reddit comments
labeled for 27 emotion categories or Neutral. The TweetEval dataset consists
of seven heterogenous tasks in Twitter, all framed as multi-class tweet
classification. The tasks include - irony, hate, offensive, stance, emoji,
emotion, and sentiment. We used the sentiment portion of this dataset for
UTCD. Emotion is a dataset of English Twitter messages with six basic
emotions: anger, fear, joy, love, sadness, and surprise. The Amazon Polarity
dataset consists of reviews from Amazon. The data spans a period of 18 years,
including 35 million reviews up to March 2013. Reviews include product and
user information, ratings, and a plaintext review. The Finance Phrasebank
dataset consists of 4840 sentences from English language financial news
categorised by sentiment. The Yelp dataset consists of over 600k reviews for
the task of sentiment classification.

For the intent/dialogue aspect we have the datasets: Schema Guided Dialgoue
Rastogi et al. (2020) is an annotated multi-domain, task-oriented
conversations between a human and a virtual assistant. Clinc-150 Larson et al.
(2019) is an intent classification (text classification) dataset consisting of
150 in-domain intent classes. SLURP Bastianelli et al. (2020) is dialuge
dataset derived from SLU systems English spanning 18 domains. Banking77
Casanueva et al. (2020b) is an intent classification dataset for the banking
domain. It comprises 13,083 customer service queries labeled with 77 intents.
Snips is an NLU dataset of over 16,000 crowdsourced queries distributed among
7 user intents. NLU Evaluation Xingkun Liu and Rieser (2019) is an NLU dataset
from the conversational domain annotated with corresponding intents and
dialogue scenarios.

Lastly, for the topic aspect we have the datasets: AG News Zhang et al. (2015)
is a topic classification dataset extract from the AG News article corpus. It
consist of 4 classes from the original corpus. Each class contains 30,000
training samples and 1,900 testing samples. Yahoo Answers dataset Zhang et al.
(2015) contains 4,483,032 questions and their answers across 10 categories.
Each class contains 140,000 training samples and 5,000 testing samples.
DBpedia Auer et al. (2007) dataset is a topic classification dataset
constructed from picking 14 non-overlapping classes from DBpedia 2014. Multi
Eurlex Chalkidis et al. (2021) is a multilingual dataset for topic
classification of legal documents. The dataset comprises 65k European Union
(EU) laws, officially translated in 23 languages, annotated with multiple
labels from the EUROVOC taxonomy. Big Patent Sharma et al. (2019) is a topic
classification dataset for the legal domain consisting of 1.3 million records
of U.S. patent documents along with human written abstractive summaries.
Consumer Finance Bureau (2012) dataset is a collection of complaints about
consumer financial products and services sent to companies for response.