# Direct correspondence between Newtonian gravitation and general relativity

Thomas Buchert  Univ Lyon, Ens de Lyon, Univ Lyon1, CNRS, Centre de Recherche
Astrophysique de Lyon UMR5574, F-69007, Lyon, France
Email<EMAIL_ADDRESS>
###### Abstract

We present a strategy to obtain equations of general relativity for an
irrotational dust continuum within a flow-orthogonal foliation of spacetime
from the equations of Newtonian gravitation, and vice versa, without employing
a weak field expansion or a limiting process on the speed of light. We argue
that writing Newton’s equations in a Lagrangian frame and relaxing
integrability of vector gradients is sufficient to obtain equations that are
identical to Einstein’s equations in (3+1)-form when respecting the Lorentzian
signature of the time parametrization. We discuss implications and provide an
outlook on how to extend the obtained correspondence to more general
spacetimes.

##  I Newtonian limits and weak fields:
is this subject settled ?

The question of how to obtain the Newtonian limit of general relativity enjoys
various answers. Many of the practical implementations of a Newtonian limit
are heuristic by e.g. expanding Einstein’s equations with respect to a flat
background spacetime and keeping only linear terms as perturbations of metric
components (weak field approximation) together with sending the causality
constant $1/c$ to zero, thus “opening up” the local light cones. A systematic
approach is the frame theory by Ehlers ehlers:frametheory  that comprises both
theories in a single theory using the causality constant as a parameter, built
on earlier efforts around the Newton-Cartan theory (for details and references
see Ehlers’ work ehlers:frametheory , the editorial to Ehlers’ work bm , and
Ehlers’ investigation of examples ehlers:examples ). Ellis ellis:relativistic
has nicely compared the (1+3)-Einstein equations and their correspondences in
Newtonian gravitation. There are also proposals of a dictionary between the
theories for metric perturbations to linear order N-GRnumerics1 ;
wald:dictionary  that have been applied in the context of general-relativistic
exact solutions and numerical simulations KoksbangHannestad , N-GRnumerics2 .
All of these approaches imply that both theories are substantially different
and that one has to neglect terms in Einstein’s equations to obtain back the
Newtonian equations of gravitation. It is, however, a matter of definition of
the Newtonian limit and the way the Newtonian equations of gravitation are
written.

In this Letter, we will investigate a two-step strategy for arriving at
Einstein’s field equations from Newton’s equations, in the context of an
irrotational dust matter model, that does not need elements of an
approximation. As a first step, we will demonstrate that the Newtonian
equations and Einstein’s equations in a flow-orthogonal foliation of spacetime
are algebraically correspondent, if the former are written in the Lagrangian
rest frame of the dust. As a second step, we will show that the Newtonian
equations become identical to their relativistic counterparts, if an
integrability requirement of the resulting tensor coefficients is relaxed in
the Newtonian equations. These statements extend to variables such as
connection, Ricci and Weyl curvatures, defined geometrically within general
relativity but are also algebraically present in the Newtonian equations, if
written in the Lagrangian frame.

We organize this Letter as follows. Sec. II recalls Newton’s equations for a
self-gravitating continuum of dust. It is then argued in Sec. III that the
Lagrangian form of Newton’s equations can be subjected to a “recipe” that
follows an idea by Einstein to “throw out the Euclidean embedding space” and
that employs Cartan coframes defining nonintegrable deformations of the fluid.
We so obtain Einstein’s equations in a flow-orthogonal (3+1)-setting. Sec. IV
discusses the result and provides an outlook on how to generalize the obtained
correspondence.

NOTATIONS: Bold notation will be used for vectors and forms. The vector
product is denoted by $\times$, while the wedge product $\wedge$ denotes the
antisymmetrized tensor product $\otimes$, with components
$({\bm{a}}\otimes{\bm{b}})_{ij}=a_{i}b_{j}$. We define symmetrization and
antisymmetrization of indices by $v_{(i,j)}=\tfrac{1}{2}(v_{i,j}+v_{j,i})$,
$v_{[i,j]}=\tfrac{1}{2}(v_{i,j}-v_{j,i})$, respectively; $i,j,k\ldots=1,2,3$.
Spatial derivatives with respect to Eulerian coordinates $\bm{x}$ are denoted
by a comma, while later spatial derivatives with respect to Lagrangian
coordinates $\bm{X}$ are denoted by a vertical slash; an overdot will be used
to represent the Lagrangian time derivative; summation over repeated indices
is understood.

##  II Newtonian equations for a dust continuum

We will employ a hydrodynamic picture and consider a self-gravitating
continuum of dust, i.e. pressureless matter with rest mass density
$\varrho({\bm{x}},t)$ and velocity field ${\bm{v}}({\bm{x}},t)$ (the words
‘matter’, ‘dust’, ‘fluid’ will be used interchangeably). Both fields are
represented in terms of Eulerian (inertial, nonrotating) coordinates $\bm{x}$
and a time parameter $t$. The acceleration field is equivalent to the
gravitational field strength $\bm{g}({\bm{x}},t)$ due to the equivalence of
inertial and gravitational mass,

$\frac{{\mathrm{d}}}{{\mathrm{d}}t}\bm{v}=\bm{g}\quad;\quad\frac{{\mathrm{d}}}{{\mathrm{d}}t}:=\frac{\partial}{\partial
t}\Big{|}_{\bm{x}}+\bm{v}\cdot\bm{\nabla}=\frac{\partial}{\partial
t}\Big{|}_{\bm{X}}\ ,$ (1)

where we introduced the Lagrangian time derivative
${\mathrm{d}}/{\mathrm{d}}t$ that reduces to a partial time derivative in a
Lagrangian coordinate system $\bm{X}$. In Eulerian coordinate components, we
write Euler’s equation (1) and its Eulerian spatial derivative:

$\frac{\partial}{\partial t}v^{i}+v^{k}v^{i}_{\
,k}=g^{i}\quad,\quad\frac{\mathrm{d}}{{\mathrm{d}}t}v^{i}_{\ ,j}+v^{i}_{\
,k}v^{k}_{\ ,j}=g^{i}_{\ ,j}\ .$ (2)

The Newtonian continuum theory of gravitation is a vector theory, so that the
sources of the curl and divergence of $\bm{g}$ suffice to define the complete
set of field equations (up to a harmonic vector field):

$\bm{\nabla}\times\bm{g}=\bm{0}\quad;\quad\bm{\nabla}\cdot\bm{g}=\Lambda-4\pi
G\varrho\ ,$ (3)

where $G$ denotes Newton’s gravitational coupling constant; for completeness
we included the cosmological constant $\Lambda$ (here with dimension ${\rm
time}^{-2}$). The rest mass density $\varrho$ obeys the continuity equation,

$\frac{{\mathrm{d}}}{{\mathrm{d}}t}\varrho+\varrho\bm{\nabla}\cdot\bm{v}=0\ .$
(4)

The Euler-Newton system comprises (1), (3) and (4). This overdetermined set of
equations can be written as a set of five equations for five variables
($\varrho,v^{i},\Phi$) through introduction of the gravitational potential,
$\bm{g}=:-\bm{\nabla}\Phi$.

In this Letter we will restrict ourselves to irrotational flows,
$v_{[i,j]}=0$. For later considerations we add the Newtonian gravitoelectric
tidal tensor (inserting (2) and (3) in the second line):

$\displaystyle{\mathcal{E}}_{ij}:$
$\displaystyle=g_{(i,j)}-\frac{1}{3}\delta_{ij}g^{k}_{\ ,k}$ (5)
$\displaystyle=\frac{\mathrm{d}}{{\mathrm{d}}t}v_{(i,j)}+v^{k}_{\
,(i}v_{j),k}-\frac{1}{3}\delta_{ij}(\Lambda-4\pi G\varrho)\ .$

##  III Realizing Einstein’s vision of a transition strategy

In his Kyoto address in December 1922 ishiwara , Albert Einstein expressed an
intuition of an interesting strategy that we will freely interpret in this
Letter. He said: If all accelerated systems are equivalent, then Euclidean
geometry cannot hold in all of them. To throw out geometry [the Euclidean
embedding space] and keep [vectorial] physical laws is equivalent to
describing thoughts without words. We must search for words before we can
express thoughts […] This point remained unsoluble to me until 1912, when I
suddenly realized, that Gauß’s theory of surfaces holds the key for unlocking
this mystery. […].

We have interpreted Einstein’s words [in brackets] to infer that the writing
of Newton’s equations in terms of vectors as functions of inertial coordinates
implies an embedding into a global vector space. A modern strategy to “throw
out” this embedding space consists in (i) moving to a Lagrangian
representation of Newton’s equations, introducing (intrinsic and noninertial)
Lagrangian coordinates, and (ii) relaxing integrability of the resulting
tensor coefficients, i.e. replacing exact gradients by general one-form
fields.

To achieve (i), we introduce a one-parameter family of spatial diffeomorphisms
to Lagrangian coordinates $\bm{X}$, labeling fluid parcels along their
trajectories, identified with their Eulerian positions at some initial time
$t_{\rm i}$ ehlersbuchert :111Henceforth, we use the indices $a,b,c\ldots$ as
counters, while $i,j,k\ldots$ remain coordinate indices referring to an exact
basis. We realize that the vector components $f^{a}$ are to be written with
counters, but if there exists an embedding space endowed with coordinates
$x^{i}=f^{i\equiv a}$, then these counters are also coordinate indices.

$(\bm{X},t)\mapsto(\bm{x},t)=({\bm{f}}({\bm{X},t}),t)\ ;\
\bm{X}:=\bm{f}(\bm{X},t_{\rm i})\ .$ (6)

We call $\bm{f}$ the field of trajectories and ${\bf d}f^{a}=f^{a}_{\ |k}{\bf
d}X^{k}$ the Lagrangian deformation gradient, represented in the exact
Lagrangian basis. Lagrangian coordinates are intrinsic to the fluid continuum
and they can as well be used as coordinates in a local chart around a point in
a spatial Riemannian manifold. Let us consider the following bilinear metric
form, the first representation of which we call the Lagrangian metric,

${{}^{3}}{\bm{\delta}}:=\delta_{ab}\,f^{a}_{\;|i}f^{b}_{\;|j}{\bf
d}X^{i}\otimes{\bf d}X^{j}=\delta_{ij}\;{\bf d}x^{i}\otimes{\bf d}x^{j}\ .$
(7)

This metric is Euclidean, since we can find a one-parameter family of
diffeomorphisms, namely ${\bm{X}}={\bm{h}}({\bm{x}},t)$, with
${\bm{h}}={\bm{f}}^{-1}$, that transforms the first representation into the
second.

A Riemannian 3-metric ${{}^{3}}{\bm{s}}$ can be written in terms of three
spatial Cartan coframes ${\bm{\eta}}^{a}$ that provide a more elementary
“reference body” than the metric,

${{}^{3}}{\bm{s}}:=\delta_{ab}\;\bm{\eta}^{a}\otimes\bm{\eta}^{b}=\delta_{ab}\;\eta^{a}_{\;i}\,\eta^{b}_{\;j}\,{\bf
d}X^{i}\otimes{\bf d}X^{j}\ .$ (8)

We notice that the mapping

$\bm{\eta}^{a}\;\mapsto\;{\bf d}f^{a}\ ,$ (9)

which we henceforth call Euclidean restriction, implies that the Riemannian
3-metric reduces to the Euclidean metric in the Lagrangian representation. In
the integrable case of exact one-form fields ${\bf d}f^{a}$, the Lagrangian
metric embodies the geometry of the fluid, however, still embedded into
Euclidean space.

We henceforth use the wording integrable when we want to express that the
coefficient matrix $\eta^{a}_{\ i}$ of $\bm{\eta}^{a}=\eta^{a}_{\ i}{\bf
d}X^{i}$ can be obtained through spatial derivatives of vector components
$f^{a}$, i.e. $\bm{\eta}^{a}$ are exact, and we say generalized gradient when
we mean relaxation of integrability that realizes step (ii) of the outlined
strategy.

In what follows we will apply the outlined two-step strategy, where we first
concentrate on kinematic properties of the fluid. To this end we are going to
find the analogy to the Newtonian velocity gradient $v^{a}_{\ ,b}$, now
written both with counter indices, since there will no longer be a reference
to an exact Eulerian basis after relaxing integrability in the sense of
inverting (9). Relating this to the Lagrangian gradient of
$\bm{v}=\dot{\bm{f}}({\bm{X}},t)$ involves the inverse transformation
${\bm{h}}({\bm{x}},t)$: $v^{a}_{\ ,b}=v^{a}_{\ |k}h^{k}_{\
,b}={\dot{f}}^{a}_{\ |k}h^{k}_{\ ,b}$.

Moving to the nonintegrable form of the velocity gradient, we have to
introduce the inverse matrix to ${\eta}^{a}_{\ k}$. We define three frame
fields, ${\bm{e}}_{b}$ (the Dreibein at the worldlines of fluid parcels),
being dual to Cartan’s coframe fields. We express both in the respective local
basis systems (${\bf d}X^{k}$ for forms and $\bm{\partial}_{X^{k}}$ for
vectors):

$\bm{\eta}^{a}={\eta}^{a}_{\ k}\,{\bf d}X^{k}\,;\,{\bm{e}}_{b}=e_{b}^{\
k}\,\bm{\partial}_{X^{k}}\,;\,{\eta}^{a}_{\ k}e_{b}^{\ k}=\delta^{a}_{\
b}\,;\,{\eta}^{a}_{\ k}e_{a}^{\ \ell}=\delta_{k}^{\ \ell}\;.$ (10)

Consequently, the nonintegrable form of the Newtonian velocity gradient is
represented by

$\Theta^{a}_{\ b}:={\dot{\eta}}^{a}_{\ k}e_{b}^{\ k}\ .$ (11)

It is expressed in the nonexact basis (remember that the velocity gradient has
both Eulerian values and derivatives with respect to Eulerian coordinates). We
transform this object into our local exact (Lagrangian) basis with the help of
the transformation matrices (10) and arrive at:

$\Theta^{i}_{\ j}:=e_{a}^{\ i}\eta^{b}_{\ j}\,\Theta^{a}_{\ b}=e_{a}^{\
i}{\dot{\eta}}^{a}_{\ j}\ .$ (12a) This field can be entirely expressed in
terms of coframe fields through the algebraic identity $e_{a}^{\
i}=\frac{1}{2J}\;\epsilon_{abc}\epsilon^{ik\ell}\,\eta^{b}_{\;k}\,\eta^{c}_{\;\ell}\
,$ (12b) with the Levi-Cività symbol $\epsilon_{abc}$, and the nonintegrable
analog of the Jacobian of the spatial diffeomorphism (6), $J:=\det(\eta^{a}_{\
i})=\frac{1}{6}\epsilon_{abc}\epsilon^{ijk}\,\eta^{a}_{\ i}\eta^{b}_{\
j}\eta^{c}_{\ k}\ .$ (12c)

We notice that the variable (12a) that generalizes the Newtonian velocity
gradient has mixed indices, which holds true for the transformation of other
Newtonian fields too. The expansion tensor is then formed by lowering the
upper index using the nonintegrable form of the Lagrangian metric (8),
$\Theta_{ij}=\delta_{ab}\eta^{a}_{\ i}\eta^{b}_{\ k}\Theta^{k}_{\
j}=\delta_{ab}\,\eta^{a}_{\ i}{\dot{\eta}}^{b}_{\ j}$, with the rate of
expansion $\Theta^{k}_{\ k}=:\Theta$. The vanishing of the vorticity,
$v_{[i,j]}=0$, so translates to the symmetry condition $\Theta_{[ij]}=0$.

Turning now to dynamical properties of the dust fluid, we introduce the
nonintegrable form of the field strength gradient along the above lines,
$g^{a}_{\ ,b}\mapsto\mathcal{F}^{a}_{\ b}={\ddot{\eta}}^{a}_{\ k}e_{b}^{\ k}$,
${\mathcal{F}}^{i}_{\ j}:=e_{a}^{\ i}\eta^{b}_{\ j}\,\mathcal{F}^{a}_{\
b}=e_{a}^{\ i}{\ddot{\eta}}^{a}_{\ j}$ and
$\mathcal{F}_{ij}=\delta_{ab}\,\eta^{a}_{\ i}{\ddot{\eta}}^{b}_{\ j}$,
yielding the nonintegrable version of Euler’s equation (2):

${\mathcal{F}}^{i}_{\ j}=\dot{\Theta}^{i}_{\ j}+\Theta^{i}_{\ k}\Theta^{k}_{\
j}\ .$ (13a) The field equations (3) generalize to the set
${\mathcal{F}}^{k}_{\ k}=\Lambda-4\pi
G\varrho\quad;\quad{\mathcal{F}}_{[ij]}=0\ .$ (13b) In the integrable version
of Newton’s equations, these are enough to determine the gravitational field.
However, in the nonintegrable version the tracefree symmetric part must be
part of the gravitational field tensor. We therefore add the nonintegrable
form of the Newtonian tidal tensor (5) (omitting here the redundant
symmetrization): $\displaystyle-{E}^{i}_{\ j}:$
$\displaystyle=\mathcal{F}^{i}_{\ j}-\frac{1}{3}\mathcal{F}^{k}_{\
k}\delta^{i}_{\ j}$ (13c) $\displaystyle=\dot{\Theta}^{i}_{\ j}+\Theta^{i}_{\
k}\Theta^{k}_{\ j}-\frac{1}{3}(\Lambda-4\pi G\varrho)\delta^{i}_{\ j}\ ,$

where we introduced a sign convention that we will explain below.

We now show that Equations (13), together with the nonintegrable Lagrangian
form of (4), $\dot{\varrho}+\Theta\varrho=0$, are identical to Einstein’s
equations in a fluid-orthogonal foliation of spacetime via the definition of a
new (from the Newtonian point of view auxiliary) field:

$-\mathcal{R}^{i}_{\ j}:={\dot{\Theta}}^{i}_{\ j}+\Theta{\Theta}^{i}_{\
j}-(\Lambda+4\pi G\varrho)\delta^{i}_{\ j}\ .$ (14)

Equation (14) implies a key equation of the correspondence: with (13a) we
obtain a relation of the generalized field strength gradient to this newly
defined field:

${\mathcal{F}}^{i}_{\ j}=-\mathcal{R}^{i}_{\ j}+(\Lambda+4\pi
G\varrho)\delta^{i}_{\ j}+\Theta^{i}_{\ k}\Theta^{k}_{\ j}-\Theta\Theta^{i}_{\
j}\ .$ (15)

In the geometrical context of general relativity this field is the spatial
Ricci tensor, $\mathcal{R}_{ij}=\delta_{ab}\eta^{a}_{\ i}\eta^{b}_{\
k}\mathcal{R}^{k}_{\ j}$, the key equation (15) is known to emerge from the
Gauß embedding equation using the nonintegrable Euler equation (13a): the
components of the generalized Newtonian field strength gradient form
components of the spacetime Riemann tensor, $-{\mathcal{F}}^{i}_{\
j}={{}^{4}}R^{i}_{\ 0j0}$. Imposing the field equations (13b), the trace of
(15) becomes the energy constraint, and the antisymmetric part of (15)
vanishes due to the vanishing of the vorticity in a flow-orthogonal foliation:
${\mathcal{F}}_{[ij]}=\delta_{ab}\,\eta^{a}_{\ [i}{\ddot{\eta}}^{b}_{\
j]}=(\delta_{ab}\,\eta^{a}_{\ [i}{\dot{\eta}}^{b}_{\
j]})^{\hbox{\bigastfont.}}=0$. The nonintegrable form of the Newtonian tidal
tensor (13c) is the spatially projected gravitoelectric part of the Weyl
tensor ehlersbuchert:weyl . It reduces to (5) in the Euclidean restriction
(9), up to a sign convention.222The sign convention difference arises, since
in the geometrical context we consider $E_{ij}$ as (“passive”) curvature,
while in Newtonian theory the corresponding field is defined “actively” in
terms of gravitational acceleration. This remark also applies to the extrinsic
curvature $K_{ij}$ vs. expansion $\Theta_{ij}=-K_{ij}$.

For completeness we list the Einstein equations for an irrotational dust fluid
in a flow-orthogonal foliation of spacetime in the usual
(3+1)-representation:333The metric signature is taken to be $(-,+,+,+)$, and
the speed of light $c=1$. Greek indices run through $\mu,\nu\ldots=0,1,2,3$,
and the semicolon denotes covariant derivative with respect to the 4-metric,
while a double vertical slash denotes covariant spatial derivative with
respect to the 3-metric ${{}^{3}}\bm{s}$ with components $s_{ij}$.

$\displaystyle{\dot{\varrho}}+\Theta\varrho=0\ ;$ (16a)
$\displaystyle{\dot{s}}_{ij}=2\,s_{ik}\Theta^{k}_{\ j}\ \ ;\ \
\Theta_{[ij]}=0\ ;$ (16b) $\displaystyle{\dot{\Theta}}^{i}_{\
j}+\Theta{\Theta}^{i}_{\ j}=-\mathcal{R}^{i}_{~{}j}+(\Lambda+4\pi
G\varrho)\delta^{i}_{\ j}\ ;$ (16c) $\displaystyle\Theta^{2}-\Theta^{i}_{\
j}\,\Theta^{j}_{\ i}=-{\mathcal{R}}^{k}_{\ k}+2\Lambda+16\pi G\varrho\ ;$
(16d) $\displaystyle\Theta^{k}_{\ j||k}-\Theta_{||j}\,=\,0\ .$ (16e)

The first equation arises from $T^{\mu\nu}=\varrho u^{\mu}u^{\nu}$, with the
conservation law $T^{\mu\nu}_{\ \ ;\nu}=0$, while the second defines the
expansion tensor (or minus the extrinsic curvature); the third are its $6$
evolution equations that are identical to the nonintegrable Euler equation
(13a) by redefining $\mathcal{R}^{i}_{\ j}$ through (15), and the fourth is
one of the four constraint equations, the energy constraint, all – as shown –
arising from our strategy.

The momentum constraints (16e) seem to not directly arise from the Newtonian
system because their Euclidean restriction (9) does not imply a constraint.
This can be traced back to the fact that the spatially projected
gravitomagnetic part of the Weyl tensor $H_{ij}$ vanishes in the Euclidean
restriction (9), in the current setting:

$-H^{i}_{\ j}=\frac{1}{J}\epsilon^{ik\ell}\Theta_{jk||\ell}\ \mapsto\ 0\ ;\
J\neq 0\ ;$ (17)

$H_{[ij]}=0$ implies (16e). This result is in agreement with the Newtonian
limit in Ehlers’ frame theory ehlersbuchert:weyl , and here trivially follows
via integrability that implies the commutation of second derivatives, see
subsections III.A.3 and III.A.4 in rza1 .

We can derive (16e) by starting with the trivial Newtonian identity that
second derivatives of the velocity commute, $v^{a}_{\ ,b\,c}-v^{a}_{\
,c\,b}=0$. Calculating $v^{a}_{\ ,b}=v^{i}_{\ |j}f^{a}_{\ |i}h^{j}_{\ ,b}$,
transforming the second derivatives to Lagrangian coordinates and projecting
onto the Lagrangian basis (step (i) of our strategy) yields:444Notice that the
(noncovariant) Christoffel connection coefficients $\Gamma^{j}_{\ n\ell}$ do
not vanish in the Euclidean restriction (9): they result in a Newtonian
integrable connection (since the Lagrangian coordinate system is noninertial):
$\Gamma^{j}_{\ \ell n}=\Gamma^{j}_{\ n\ell}\mapsto{}^{\rm N}\Gamma^{j}_{\
n\ell}=h_{\ ,c}^{j}f^{c}_{\ |\ell n}=-h^{j}_{\ ,ab}f^{a}_{\ |\ell}f^{b}_{\
|n}={}^{\rm N}\Gamma^{j}_{\ \ell n}$ (18) (both forms appear in the
calculation of (III)). However, the Euclidean restriction of the (covariant)
Cartan connection ${\bf d}{\bm{\eta}}^{a}=:-{\bm{\omega}}^{a}_{\
b}\wedge{\bm{\eta}}^{b}$ vanishes, i.e. the covariant requirement of
integrability, ${\bf d}^{2}f^{a}={\bf 0}$, with ${\bf d}^{2}:={\bf d}\circ{\bf
d}$, holds.
Notice also that, starting with the transformation of the vector,
$v^{a}=f^{a}_{\ |i}v^{i}$, instead of its gradient, will result in extra terms
proportional to the Lagrangian velocity $v^{i}=0$, which consequently vanish
in a Lagrangian frame.

$\displaystyle(v^{a}_{\ ,bc}-v^{a}_{\ ,cb})(h^{k}_{\ ,a}f^{b}_{\ |n}f^{c}_{\
|\ell})=v^{k}_{\ |n\ell}-v^{k}_{\ |\ell n}$ $\displaystyle+{}^{N}\Gamma^{j}_{\
\ell n}v^{k}_{\ |j}-{}^{N}\Gamma^{j}_{\ n\ell}v^{k}_{\ |j}+{}^{N}\Gamma^{k}_{\
\ell i}v^{i}_{\ |n}-{}^{N}\Gamma^{k}_{\ ni}v^{i}_{\ |\ell}$
$\displaystyle=v^{k}_{\ |n||\ell}-v^{k}_{\
|\ell||n}=\epsilon_{in\ell}\epsilon^{ijm}v^{k}_{\ |j||m}\ .$ (19)

Relaxing integrability (step (ii) of our strategy) then results in the
Peterson-Mainardi-Codazzi identity in the flow-orthogonal foliation
[Sect.8.3]alcubierreGR , eric  (using (17) in the second equality):

$\Theta^{k}_{\ n||\ell}-\Theta^{k}_{\
\ell||n}=\epsilon_{in\ell}\epsilon^{ijm}\Theta^{k}_{\
j||m}=-J\epsilon_{in\ell}H^{ik}\ ,$ (20)

the trace of which ($k=\ell$) is (16e). Note that the trace of the
gravitomagnetic part of the Weyl tensor vanishes due to the symmetry condition
$\Theta_{[pj]}=0$. In the integrable case both sides are identically zero.

An important remark is in order here. Both steps (i) and (ii) are crucial for
our transformation strategy, but notice that step (ii) produces a
nonintegrable deformation leading to a general description of spatial
deformations in terms of Cartan coframe fields. Therefore, following from step
(ii), we can derive all the elements of geometry like a nonintegrable
connection and curvature via Cartan’s structure equations in space, ${\bf
d}{\bm{\eta}}^{a}=-\bm{\omega}^{a}_{\ b}\wedge\bm{\eta}^{b}\neq{\bf 0}$ and
$\bm{\Omega}^{a}_{\ b}:={\bf d}\bm{\omega}^{a}_{\ b}+\bm{\omega}^{a}_{\
c}\wedge\bm{\omega}^{c}_{\ b}\neq{\bf 0}$, together with the spatial Bianchi
identities ${\bf d}^{2}{\bm{\eta}}^{a}={\bm{0}}$ and ${\bf
d}^{2}{\bm{\omega}}^{a}_{\ b}={\bm{0}}$.

The final element of the correspondence arises when constructing the spacetime
metric. We notice that the Newtonian equations and the (3+1)-Einstein
equations appear to be parametrized by the coordinate $t$: our two-step
strategy produces the correct equations. However, when constructing a
4-dimensional spacetime, the introduction of the Lorentzian signature in the
4-metric is required: ${{}^{4}}{\bm{s}}:=-{\bf d}t\otimes{\bf
d}t+\delta_{ab}\;\eta^{a}_{\;i}\,\eta^{b}_{\;j}\,{\bf d}X^{i}\otimes{\bf
d}X^{j}$. The Euclidean restriction (9) is then extended to spacetime and
becomes the restriction to Minkowski spacetime. We understand that the
Lagrangian representation is a crucial cornerstone of the correspondence:
Lagrangian observers are at rest and “do not see” the local light cone, since
they do not experience a boost. The Lagrangian observers have just to be told
that their distances in time direction count negatively in a causal
4-dimensional spacetime.

##  IV Summary and discussion

We looked at the Newtonian equations for self-gravitating systems in the
Lagrangian frame together with a generalization to a nonintegrable form of the
Newtonian deformation gradient ${\bf d}f^{a}$. We restricted our investigation
to the matter model of irrotational dust. We argued that the nonintegrable
form is equivalent to Einstein’s equations in a flow-orthogonal
(3+1)-foliation of spacetime. We observe that there is no weak field
approximation and no limiting process to be performed. None of the parts in
Einstein’s equations are neglected, which paints an alternative picture to the
current understanding of Newtonian and post-Newtonian dynamics. Newton’s
theory appears to be stronger than believed by employing a modern
interpretation. It will be interesting to revisit predictions of general
relativity where Newtonian predictions in Eulerian representation appear to
fall short.

We could furthermore argue that the integrable (Lagrange-Newton) form is a
measure-zero representation of the general form that, from a pragmatic point
of view, can never be realized: any realization, e.g. a numerical
implementation of an exact gradient ${\bf d}f^{a}$, will be limited by finite
precision and generically produces a nonintegrable field. Newtonian gradients
form a measure zero set of fluid deformations and meet the strong condition
${\bf d}^{2}f^{a}={\bf 0}$ (second derivatives commute). Hence, any
realization will instantly produce the nonintegrable form, and therefore a
nonintegrable connection and curvature via Cartan’s structure equations. A
small perturbation of strict integrability can lead to strong curvature
without there being a smooth limit to Euclidean space. As a pronounced
summary: an attempt to realize Newtonian dynamics in the Lagrangian
representation will, in practice, be a realization of general relativity.

Mathematically, the requirement of integrability has nevertheless interesting
and important implications. We think of the spatial integration of an
integrable field vs. a nonintegrable field.555The difference can be best seen
by performing a Hodge decomposition of Cartan forms into exact, co-exact and
harmonic forms, for a component writing of one-forms in this context see rza4
. Integrable parts will allow for a transformation to surface integrals on the
boundary of a spatial domain, while nonintegrable parts remain in the bulk. An
example is the backreaction problem in cosmology, i.e. the impact of
inhomogeneities on global properties of world models buchert:dust ,GBC : in
flat space, the relevant terms describing a nonvanishing impact are
divergences of vector fields and as a result vanish for isolated systems and
for periodic boundary conditions corresponding to a 3-torus topological
architecture buchert:average . For nonflat spatial sections, i.e. in the
nonintegrable situation, backreaction terms in general furnish a global
contribution.

The reader may find examples where the proposed correspondence has already
been successfully employed in the transition from Lagrangian perturbation
solutions in Newtonian theory to corresponding general-relativistic
perturbation solutions, see the series of papers following rza1  and the
recent review paper Universe . We also point the reader to the construction of
exact solutions of general relativity from Newtonian solutions, see Kasai1995
, rza6  for Szekeres class II solutions with their corresponding Euclidean
class Buchert1989AA , that appear as subclasses of first-order Lagrangian
perturbation solutions at a FLRW (Friedmann-Lemaître-Robertson-Walker)
background; for Szekeres class I solutions see beyond .

It is possible to extend the proposed strategy to more general spacetimes. In
order to describe e.g. vortical flows and more general fluids, we have to
consider tilted flows within a general ADM (Arnowitt-Deser-Misner) foliation
of spacetime. For this purpose we have to diffeomorph the exact basis to
obtain the general metric form (with lapse $N$ and shift $N^{i}$), below
exemplified for a comoving description, where the coordinate velocity is set
to zero. The line element reads:

${{}^{4}}ds^{2}=-\frac{N^{2}}{\gamma^{2}}dt^{2}+2N{\rm
v}_{i}\,dt\,dX^{i}+s_{ij}\,dX^{i}dX^{j}\ ,$ (21)

with the covariant 3-velocity ${\rm v}^{i}=(N^{i}/N)$, the induced spatial
metric components $s_{ij}$, and a 4-velocity that is tilted with respect to
the hypersurface normal BMR :

$u^{\mu}=\frac{\gamma}{N}(1,0,0,0)\quad;\quad
u_{\mu}=(-\frac{N}{\gamma},\gamma{\rm v}_{i})\ .$ (22)

In the general comoving setting the appearance of the Lorentz factor $\gamma$
resurrects the causality constant, while the appearence of the lapse $N$ makes
the time deformation nonintegrable. Rendering the tilted 4-velocity
Lagrangian, $u^{\mu}=(1,0,0,0)$, as our strategy demands, a proper-time
foliation $\tau=\int(N/\gamma)dt=const.$, i.e. $N=\gamma$, investigated in BMR
, can be considered. A correspondence in proper-time foliation can thus be set
up in the spirit of what has been said in this Letter. Its realization is the
subject of work in progress. In the tilted foliation, gravitomagnetic
extensions of Newton’s theory, as proposed by Heaviside heaviside  (see also
the comments in bm ), will become relevant.

Acknowledgments: This work is part of a project that has received funding from
the European Research Council under the European Union’s Horizon 2020 research
and innovation program (grant agreement ERC advanced grant 740021-ARTHUS, PI:
TB). Thanks to Hamed Barzegar, Henk van Elst, Asta Heinesen and Pierre Mourier
for valuable remarks on the manuscript, and to an anonymous referee for
insightful and constructive suggestions.

## References

  * (1) M. Alcubierre, Introduction to 3+1 Numerical Relativity. International Series of Monographs on Physics, Oxford Academic, Oxford (2008)
  * (2) F. Al Roumi, T. Buchert and A. Wiegand, Lagrangian theory of structure formation in relativistic cosmology. IV. Lagrangian approach to gravitational waves. Phys. Rev. D 96, 123538 (2017) [arXiv:1711.01597]
  * (3) L. Brunswic and T. Buchert, Gauss–Bonnet–Chern approach to the averaged Universe. Class. Quant. Grav. 37, 215022 (2020) [arXiv:2002.08336]
  * (4) T. Buchert, A class of solutions in Newtonian cosmology and the pancake theory. Astron. Astrophys. 223, 9 (1989)
  * (5) T. Buchert, On average properties of inhomogeneous fluids in general relativity: dust cosmologies. Gen. Rel. Grav. 32, 105 (2000) [arXiv:gr-qc/9906015]
  * (6) T. Buchert, I. Delgado Gaspar and J.J. Ostrowski, On general-relativistic Lagrangian perturbation theory and its non-perturbative generalization. Universe 8, 583 (2022) [arXiv:2209.13417]
  * (7) T. Buchert and J. Ehlers, Averaging inhomogeneous Newtonian cosmologies. Astron. Astrophys. 320, 1 (1997) [arXiv:astro-ph/9510056]
  * (8) T. Buchert and T. Mädler, Editorial Note to: On the Newtonian Limit of Einstein’s Theory of Gravitation (by Jürgen Ehlers). Gen. Rel. Grav. 51, 162 (2019) [arXiv:1910.12106]
  * (9) T. Buchert, P. Mourier and X. Roy, On average properties of inhomogeneous fluids in general relativity III: general fluid cosmologies. Gen. Rel. Grav. 52, 27 (2020) [arXiv:1912.04213]
  * (10) T. Buchert and M. Ostermann, Lagrangian theory of structure formation in relativistic cosmology. I. Lagrangian framework and definition of a nonperturbative approximation. Phys. Rev. D 86, 023520 (2012) [arXiv:1203.6263]
  * (11) N.E. Chisari and M. Zaldarriaga, Connection between Newtonian simulations and general relativity. Phys. Rev. D 83, 123505 (2011) Erratum: Phys. Rev. D 84, 089901 (2011) [arXiv:1101.3555]
  * (12) I. Delgado Gaspar and T. Buchert, Lagrangian theory of structure formation in relativistic cosmology. VI. Comparison with Szekeres exact solutions. Phys. Rev. D 103, 023513 (2021) [arXiv:2009.06339]
  * (13) I. Delgado Gaspar, T. Buchert and J.J. Ostrowski, Beyond relativistic Lagrangian perturbation theory. I. An exact-solution controlled model for structure formation. Phys. Rev. D 107, 024018 (2023) [arXiv:2210.04004]
  * (14) W.E. East, R. Wojtak and T. Abel, Comparing fully general relativistic and Newtonian calculations of structure formation. Phys. Rev. D 97, 043509 (2018) [arXiv:1711.06681]
  * (15) J. Ehlers, Über den Newtonschen Grenzwert der Einsteinschen Gravitationstheorie. In: Nitsch, J., Pfarr, J., Stachow, E.-W. (eds.) Grundlagenprobleme der modernen Physik: Festschrift für Peter Mittelstaedt zum 50. Geburtstag, pp. 65-84. Bibliographisches Institut, Mannheim, Wien, Zürich (1981); Republication of: On the Newtonian limit of Einstein’s theory of gravitation. Gen. Rel. Grav. 51, 163 (2019)
  * (16) J. Ehlers, Examples of Newtonian limits of relativistic space-times. Class. Quant. Grav. 14, A119 (1997)
  * (17) J. Ehlers and T. Buchert, Newtonian cosmology in Lagrangian formulation: Foundations and perturbation theory. Gen. Rel. Grav. 29, 733 (1997) [arXiv:astro-ph/9609036]
  * (18) J. Ehlers and T. Buchert, On the Newtonian limit of the Weyl tensor, Gen. Rel. Grav. 41, 2153 (2009) [arXiv:0907.2645]
  * (19) G.F.R. Ellis, Republication of: Relativistic Cosmology, Gen. Rel. Grav. 41, 581 (2009)
  * (20) E. Gourgoulhon, 3+1 formalism in general relativity. Bases of numerical relativity. Lecture Notes in Physics vol. 846, Springer, Berlin (2012) [arXiv:gr-qc/0703035]
  * (21) S.R. Green and R.M. Wald, Newtonian and relativistic cosmologies. Phys. Rev. D 85, 063512 (2012) [arXiv:1111.2997]
  * (22) O. Heaviside, A gravitational and electromagnetic analogy. The Electrician 31, 281 (part I), 359 (part II) (1893) html online by O. D. Jefimenko
  * (23) J. Ishiwara, Einstein K$\overline{o}$en–Roku (Records of Professor Einstein’s Lectures). Ishiwara & Hitoshi (eds.), Tokyo–Tosho Corporation, Tokyo (1971)
  * (24) M. Kasai, Tetrad-based perturbative approach to inhomogeneous universes: a general relativistic version of the Zel’dovich approximation. Phys. Rev. D 52, 5605 (1995)
  * (25) S.M. Koksbang and S. Hannestad, Methods for studying the accuracy of light propagation in N-body simulations. Phys. Rev. D 91, 043508 (2015) [arXiv:1501.01413]


# Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning

Jian-Guo Zhang1, Trung Bui2, Seunghyun Yoon2, Xiang Chen2, Zhiwei Liu1
Congying Xia1, Quan Hung Tran2, Walter Chang2, Philip Yu1
1 University of Illinois at Chicago, Chicago, USA
2Adobe Research, San Jose, USA
{jzhan51, zliu213, cxia8<EMAIL_ADDRESS>
{bui, syoon, xiangche, qtran<EMAIL_ADDRESS>
Work done while the first author was an intern at Adobe Research.

###### Abstract

In this work, we focus on a more challenging few-shot intent detection
scenario where many intents are fine-grained and semantically similar. We
present a simple yet effective few-shot intent detection schema via
contrastive pre-training and fine-tuning. Specifically, we first conduct self-
supervised contrastive pre-training on collected intent datasets, which
implicitly learns to discriminate semantically similar utterances without
using any labels. We then perform few-shot intent detection together with
supervised contrastive learning, which explicitly pulls utterances from the
same intent closer and pushes utterances across different intents farther.
Experimental results show that our proposed method achieves state-of-the-art
performance on three challenging intent detection datasets under 5-shot and
10-shot settings.

##  1 Introduction

Intent detection, aiming to identify intents from user utterances, is a key
component in task-oriented dialog systems. In real systems such as Amazon
Alexa, correctly identifying user intents is crucial for downstream tasks
(Zhang et al., 2020b; Ham et al., 2020). A practical challenge is data
scarcity as it is expensive to annotate enough examples for emerging intents,
and how to accurately identify intents in few-shot learning has raised
attention.

Existing methods address the few-shot intent detection tasks mainly from two
perspectives: (1) data augmentation and (2) task-adaptive training with pre-
trained models. For the first category, Zhang et al. (2020a) and Mehri et al.
(2020b) propose a nearest neighbor classification schema with full use of the
limited training examples in both training and inference stages. Xia et al.
(2020b) and Peng et al. (2020) propose to generate utterances for emerging
intents based on variational autoencoder (Kingma and Welling, 2013) and GPT-2
(Radford et al., 2019), respectively. For the second category, Casanueva et
al. (2020) and Mehri et al. (2020a) conduct intent detection by leveraging
related conversational pre-training models based on a few hundred million
conversations. Meanwhile, they devise a task-adaptive training schema where
the model is pre-trained on all relative intent datasets or the target intent
datasets with mask language modeling.

However, previous methods such as data augmentation related models Liu et al.
(2021c) are inefficient for training and hard to scale to tasks with lots of
intents. Moreover, these models do not tackle well the following scenarios: In
real scenarios, the few-shot intent detection could be more challenging when
there exist many fine-grained intents, especially semantically similar
intents. For instance, BANKING77 (Casanueva et al., 2020) has a single domain
with 77 intents, and CLINC150 (Larson et al., 2019) has ten domains with 150
intents. Many intents in the datasets are similar. Therefore, training models
is rather challenging when there are only limited examples.

Inspired by the recent success of contrastive learning (He et al., 2020; Gunel
et al., 2020; Chen et al., 2020; Radford et al., 2021; Liu et al., 2021a; Gao
et al., 2021; Liu et al., 2021b), which aims to enhance discrimination
abilities of models, this work proposes improving few-shot intent detection
via Contrastive Pre-training and Fine-Tuning (CPFT). Intuitively, we first
learn to implicitly discriminate semantically similar utterances via
contrastive self-supervised pre-training on intent datasets without using any
intent labels. We then jointly perform few-shot intent detection and
supervised contrastive learning. The supervised contrastive learning helps the
model explicitly learn to pull utterances from the same intent close and push
utterances across different intents apart.

Our contributions are summarized as follows: 1) We design a simple yet
effective few-shot intent detection schema via contrastive pre-training and
fine-tuning. 2) Experimental results verify the state-of-the-art performance
of CPFT on three challenging datasets under 5-shot and 10-shot settings.

##  2 Related Work

Since this work is related to few-shot intent detection and contrastive
learning, we review recent work from both areas in this section.

The few-shot intent detection task typically includes three scenarios: (1)
learn a intent detection model with only $K$ examples for each intent (Zhang
et al., 2020a; Mehri et al., 2020a; Casanueva et al., 2020); (3) learn to
identify both in-domain and out-of-scope queries with only $K$ examples for
each intent (Zhang et al., 2020a, 2021; Xia et al., 2021b). (2) given a model
trained on existing intents with all examples, learn to generalize the model
to new intents with only $K$ examples for each new intent (Xia et al., 2020a,
b, 2021a).

In this work, we focus on the first scenario, and several methods have been
proposed to tackle the challenge. Specifically, Zhang et al. (2020a) proposes
a data augmentation schema, which pre-trains a model on annotated pairs from
natural language inference (NLI) datasets and designs the nearest neighbor
classification schema to adopt the transfer learning and classify user
intents. However, the training is expensive and hard to scale to tasks with
hundreds of intents Liu et al. (2020). Mehri et al. (2020b); Casanueva et al.
(2020) propose the task-adaptive training, which leverages models pre-trained
from a few hundred million dialogues to tackle few-shot intent detection. It
also includes an unsupervised mask language modeling loss on the target intent
datasets and shows promising improvements.

Contrastive learning has shown superior performance on various domains, such
as visual representation (He et al., 2020; Chen et al., 2020; Radford et al.,
2021), graph representation Qiu et al. (2020); You et al. (2020), and
recommender systems Liu et al. (2021b). Moreover, recent works also adopt
contrastive learning in natural language processing tasks (Gunel et al., 2020;
Liu et al., 2021a; Gao et al., 2021), which employs the contrastive learning
to train the encoder. Specifically, Gunel et al. (2020) designs a supervised
contrastive learning loss for fine-tuning data. Gao et al. (2021) designs a
simple contrastive learning framework through dropout and it shows state-of-
the-art performance on unsupervised and full-shot supervised semantic textual
similarity tasks. Liu et al. (2021a) designs self-supervised Mirror-BERT
framework with two types of data augmentation: randomly erase or mask parts of
the input texts; feature level augmentation through dropout.

Our work differs from them in several respects: Firstly, we specifically
tackle the few-shot intent detection task rather than the general full-shot
learning; Secondly, we design a schema and employ contrastive learning in both
self-supervised pre-training and supervised fine-tuning stages.

##  3 CPFT Methodology

We consider a few-shot intent detection task that handles $C$ user intents,
where the task is to classify a user utterance $u$ into one of the $C$
classes. We set balanced K-shot learning for each intent Zhang et al. (2020a);
Casanueva et al. (2020), i.e., each intent only includes $K$ examples in the
training data. As such, there are in total $C\cdot K$ training examples.

In the following section, we first describe the self-supervised contrastive
pre-training for utterance understanding before introducing the supervised
fine-tuning for few-shot intent detection.

###  3.1 Self-supervised Pre-training

We retrieve the feature representation $\mathbf{h}_{i}$ for the $i$-th user
utterance through an encoder model, which in this paper is BERT Devlin et al.
(2019), i.e., $\mathbf{h}_{i}=\text{BERT}(u_{i})$. We implicitly learn the
sentence-level utterance understanding and discriminate semantically similar
utterances through the self-supervised contrastive learning method (Wu et al.,
2020b; Liu et al., 2021a; Gao et al., 2021):

$\mathcal{L}_{\text{uns\\_{cl}}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\text{sim}(\mathbf{h}_{i},\bar{\mathbf{h}}_{i})/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(\mathbf{h}_{i},\bar{\mathbf{h}}_{j})/\tau)},$
(1)

where $N$ is the number of sentences in a batch. $\tau$ is a temperature
parameter that controls the penalty to negative samples.
$sim(\mathbf{h}_{i},\bar{\mathbf{h}_{i}})$ denotes the cosine similarity
between two input vectors $\mathbf{h}_{i}$ and $\bar{\mathbf{h}_{i}}$.
$\bar{\mathbf{h}_{i}}$ represents the representation of sentence
$\bar{u_{i}}$, where $\bar{u_{i}}$ is from the same sentence $u_{i}$ but few
(10%) tokens are randomly masked Devlin et al. (2019). Specifically, we
dynamically mask tokens during batch training (Wu et al., 2020a), i.e., a
sentence has different masked positions across different training epochs, and
we find it is beneficial to the utterance understanding. The sentence $u_{i}$
and $\bar{u_{i}}$ are inputted together to a single encoder during the batch
training (Gao et al., 2021).

Besides the sentence-level enhancement, we also add the mask language modeling
loss Devlin et al. (2019); Wu et al. (2020a) to enhance the token-level
utterance understanding:

$\mathcal{L}_{\text{mlm}}=-\frac{1}{M}\sum_{m=1}^{M}\log P(x_{m}),$ (2)

where $P(x_{m})$ denotes the predicted probability of a masked token $x_{m}$
over the total vocabulary, and $M$ is the number of masked tokens in each
batch.

Our total loss for each batch is
$\mathcal{L}_{\text{stage1}}=\mathcal{L}_{\text{uns\\_{cl}}}+\lambda\mathcal{L}_{\text{mlm}}$,
where $\lambda$ is a weight hyper-parameter.

###  3.2 Supervised Fine-tuning

Through self-supervised learning in the first stage, the model efficiently
utilizes many unlabeled user utterances. The model is given very limited
examples in the second stage, such as 5 and 10 examples for each intent. To
better understanding user intents, especially when intents are similar to each
other, we utilize a supervised contrastive learning method (Gunel et al.,
2020) and train it together with an intent classification loss. We treat two
utterances from the same class as a positive pair and the two utterances
across different classes as a negative pair for contrastive learning. Unlike
the previous work, the utterance and itself could also be a positive pair as
we input them together to the single encoder. Their feature representations
are different due to the dropout of BERT. The corresponding loss is shown as
the following:

$\mathcal{L}_{\text{s\\_{cl}}}=-\frac{1}{T}\sum_{i=1}^{N}\sum_{j=1}^{N}\mathds{1}_{y_{i}=y_{j}}\log\frac{\exp(\text{sim}(\mathbf{h}_{i},\mathbf{h}_{j})/\tau)}{\sum_{n=1}^{N}\exp(\text{sim}(\mathbf{h}_{i},\mathbf{h}_{n})/\tau)},$
(3)

where $T$ is the number of pairs from the same classes in the batch.

Next is the intent classification loss:

$\mathcal{L}_{\text{intent}}=-\frac{1}{N}\sum_{j=1}^{C}\sum_{i=1}^{N}\log
P(C_{j}|u_{i}),$ (4)

where $P(C_{j}|u_{i})$ is the predicted probability of the $i$-th sentence to
be the $j$-th intent class.

We jointly train the two losses together at each batch:
$\mathcal{L}_{\text{stage2}}=\mathcal{L}_{\text{s\\_{cl}}}+\lambda^{\prime}\mathcal{L}_{\text{intent}}$,
where $\lambda^{\prime}$ is a weight hyper-parameter.

Name | # Utterance | # Intent | # Domain
---|---|---|---
CLINC150 (Larson et al., 2019) | 18200 | 150 | 10
BANKING77 (Casanueva et al., 2020) | 10162 | 77 | 1
HWU64 (Liu et al., 2019) | 10030 | 64 | 21
TOP (Gupta et al., 2018) | 35741 | 25 | 2
SNIPS (Coucke et al., 2018) | 9888 | 5 | -
ATIS (Tur et al., 2010) | 4978 | 21 | -

Table 1: Data statistics for intent detection datasets.

| CLINC150 | BANKING77 | HWU64
---|---|---|---
Model | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot
RoBERTa+Classifier (Zhang et al., 2020a) | 87.99 | 91.55 | 74.04 | 84.27 | 75.56 | 82.90
USE (Casanueva et al., 2020) | 87.82 | 90.85 | 76.29 | 84.23 | 77.79 | 83.75
CONVERT (Casanueva et al., 2020) | 89.22 | 92.62 | 75.32 | 83.32 | 76.95 | 82.65
USE+CONVERT (Casanueva et al., 2020) | 90.49 | 93.26 | 77.75 | 85.19 | 80.01 | 85.83
CONVBERT (Mehri et al., 2020a) | - | 92.10 | - | 83.63 | - | 83.77
CONVBERT + MLM (Mehri et al., 2020a) | - | 92.75 | - | 83.99 | - | 84.52
CONVBERT + Combined (Mehri et al., 2020b) | - | 93.97 | - | 85.95 | - | 86.28
DNNC (Zhang et al., 2020a) | 91.02 | 93.76 | 80.40 | 86.71 | 80.46 | 84.72
CPFT | 92.34 | 94.18 | 80.86 | 87.20 | 82.03 | 87.13

Table 2: Testing accuracy ($\times 100\%$) on three datasets under 5-shot and
10-shot settings.

##  4 Experimental Settings

###  4.1 Datasets

#### Pre-training Datasets

We collected six public datasets consisting of different user intents. The
dataset statistics are shown in Table 1.111https://github.com/jianguoz/Few-
Shot-Intent-Detection For fair comparisons, we exclude their test sets during
the pre-training phase, which is different from previous work (Mehri et al.,
2020a, b), where they use the whole datasets. We also remove utterances with
less than five tokens, and there are 80,782 training utterances in total. We
conduct self-supervised pre-training on the collected utterances without using
labels.

#### Evaluation Datasets

To better study the more challenging fine-grained few-shot intent detection
problem and compare with recent state-of-the-art baselines, we pick up three
challenging intent detection datasets for evaluation, i.e., CLINC150 (Larson
et al., 2019), BANKING77 (Casanueva et al., 2020) and HWU64 (Liu et al.,
2019). CLINC150 contains 23,700 utterances across ten different domains, and
there are in total 150 intents. BANKING77 contains 13,083 utterances with a
single banking domain and 77 intents. HWU64 includes 25,716 utterances with 64
intents spanning 21 domains. We follow the setup of Mehri et al. (2020a),
where a small portion of the training set is separated as a validation set,
and the test set is unchanged. Following previous work, we repeat our few-shot
learning model training five times and report the average accuracy.

###  4.2 Model Training and Baselines

We utilize RoBERTa with base configuration, i.e., roberta-base as the BERT
encoder. We pre-train the combined intent datasets without test sets in the
contrastive pre-training stage for 15 epochs, where we set the batch size to
64, $\tau$ to 0.1, and $\lambda$ to 1.0. The pre-training phase takes around
2.5 hours on a single NVIDIA Tesla V100 GPU with 32GB memory. We fine-tune the
model under 5-shot (5 training examples per intent) and 10-shot settings (10
training examples per intent). We set the batch size to 16, and do hyper-
parameters search for $\tau\,{\in}\,\\{0.1,0.3,0.5\\}$ and
$\lambda^{\prime}\,{\in}\,\\{0.01,0.03,0.05\\}$; the fine-tuning takes five
minutes for each run with 30 epochs. We apply label smoothing to the intent
classification loss, following Zhang et al. (2020a).

#### Baselines

We compare with six strong models. 1, RoBERTa+Classifier (Zhang et al.,
2020a): it is a RoBERTa-based classification model. 2, USE Yang et al. (2020):
it is the large multilingual model pre-trained on 16 languages. 3, CONVERT
(Casanueva et al., 2020): it is an intent detection model with dual encoders,
and the dual encoder models are pre-trained on 654 million (input, response)
pairs from Reddit. 4, CONVEBERT (Mehri et al., 2020a): it fine-tunes BERT on a
large open-domain dialogue corpus with 700 million conversations. 5,
CONVEBERT+Comined (Mehri et al., 2020b): it is an intent detection model based
on CONVEBERT, with example-driven training based on similarity matching and
observers for transformer attentions. It also conducts task-adaptive self-
supervised learning with mask language modeling (MLM) on the intent detection
datasets. Combine represents the best MLM+Example+Observers setting in the
referenced paper. 6, DNNC (Zhang et al., 2020a): it is a discriminative
nearest-neighbor model which finds the best-matched example from the training
set through similarity matching. The model conducts data augmentation during
training and boosts performance by pre-training on three natural language
inference tasks.

| CLINC150 | BANKING77 | HWU64
---|---|---|---
Model | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot
CPFT | 92.34 | 94.18 | 80.86 | 87.20 | 82.03 | 87.13
w/o Contrastive pre-training | -4.15 | -2.63 | -4.11 | -2.37 | -6.01 | -4.17
w/o Supervised contrastive learning | -0.56 | -0.32 | -2.06 | -0.88 | -1.14 | -0.27
w/o Contrastive pre-training + w/o Supervised contrastive learning | -4.35 | -2.69 | -6.82 | -2.93 | -6.47 | -4.23

Table 3: Testing accuracy ($\times 100\%$) of CPFT with variants on three
datasets under 5-shot and 10-shot settings.

##  5 Experimental Results

We show the overall comparisons on three datasets in Table 2. The proposed
CPFT method achieves the best performance across all datasets under both the
5-shot and 10-shot settings. Specifically, CPFT outperforms DNNC by 1.32% and
1.57% on CLINC150 and HWU64 under the 5-shot setting, respectively. It also
improves DNNC by 2.41% on HWU64 under the 10-shot setting. Our variances are
also lower when compared with DNNC: Ours vs. DNNC: 0.39 vs. 0.57 and 0.18 vs.
0.42 on CLINC150; 0.20 vs. 0.88 and 0.48 vs. 0.21 on BANKING77; 0.51 vs. 1.00
and 0.25 vs. 0.38 on HWU64 under 5-shot and 10-shot settings, respectively.
The improvements indicate that our proposed method has a better ability to
discriminate semantically similar intents than the strong discriminate
nearest-neighbor model with data augmentation. Moreover, the DNNC training is
expensive, as when training models on a single NVIDIA Tesla V100 GPU with 32GB
memory, DNNC takes more than $3$ hours for 10-shot learning on CLINC150, and
it needs to retrain the model for every new setting. CPFT only needs $2.5$
hours for one-time pre-training, and the fine-tuning only takes five minutes
for each new setting. Compared with CONVBERT+MLM, which does a self-supervised
pre-training with MLM on the intent detection datasets, CPFT improves the
performance by 1.43%, 3.21%, and 2.61% on CLINC150, BANKING77, and HWU64 under
10-shot setting, respectively. CPFL also outperforms CONVBERT+Combined, which
further adds examples-driven training and specific transformer attention
design. We contribute the performance improvements to contrastive learning,
which help the model discriminate semantically similar intents.

##  6 Ablation Study and Analysis

#### Is the schema with both stages necessary?

We conduct ablation study to investigate the effects of self-supervised
contrastive pre-training and supervised contrastive fine-tuning. Table 3 shows
the testing results of CPFT with model variants on three datasets.
Experimental results indicate that both stages are necessary to achieve the
best performance. The self-supervised contrastive pre-training on the first
stage is essential as the performance drops significantly on all datasets. We
hypothesize that contrastive pre-training on the intent datasets without using
labels benefits the discrimination of semantically similar utterances.
Additionally, the performance also drops if without supervised contrastive
learning during the few-shot fine-tuning stage. Specifically, it drops by 2%
on BANKING77 under the 5-shot setting; the reason is that BANKING77 is a
single domain dataset with many similar intents, where supervised contrastive
learning can explicitly discriminate semantically similar intents with very
limited training examples. We also jointly train the first and second stages
together, and compared with the proposed CPFT schema, we observe minimal
improvements. The joint training is also costly as it requires retraining the
model every time for new settings.

#### Is contrastive pre-training beneficial to the target intent dataset?

Additionally, we study whether contrastive pre-training can benefit the intent
detection when excluding the target datasets. Specifically, we pre-train the
model on the datasets except for the HWU64 dataset on the first stage and do
few-shot learning on HWU64 during the second stage. Compared to the model
without contrastive pre-training on the first stage, the performances are
improved by 1.98% and 1.21% under 5-shot and 10-shot settings, respectively.
The improvements indicate that the contrastive pre-training is helpful to
transfer knowledge to new datasets. However, there are still performance drops
compared to the contrastive pre-training, including the HWU64 dataset. Which
shows that it is beneficial to include the target dataset during self-
supervised contrastive learning. We leave whether self-supervised contrastive
pre-training only on the target intent dataset benefits as a future study.

#### Is the training sensitive to hyper-parameters?

We also study the effects of hyper-parameters of contrastive learning, i.e.,
the temperature $\tau$ and weight $\lambda^{\prime}$. We set
$\tau\,{\in}\,\\{0.05,0.1,0.3,0.5\\}$ and
$\lambda^{\prime}\,{\in}\,\\{0.01,0.03,0.05,0.1\\}$. In our primary
experiments, we do not find $\tau$ has a notable influence during the self-
supervised contrastive pre-training on the first stage. Besides, we found that
a batch size larger than 32 works well in the pre-training phase. However,
during the few-shot fine-tuning stage, when setting $\tau$ to a small value
$0.05$, which heavily enforces the penalty to hard negative examples and
$\lambda^{\prime}$ to a large value $0.1$, which increases the weight of
supervised contrastive learning loss, the performance drops significantly. In
addition, the batch size influences performance on this stage. Therefore, few-
shot supervised contrastive loss is sensitive to hyper-parameters when there
are limited training examples.

##  7 Conclusion

In this paper, we improve the performance of few-shot intent detection via
contrastive pre-training and fine-tuning. It first conducts self-supervised
contrastive pre-training on collected intent detection datasets without using
any labels, where the model implicitly learns to separate fine-grained
intents. Then it performs the few-shot fine-tuning based on the joint intent
classification loss and supervised contrastive learning loss, where the
supervised contrastive loss encourages the model to distinguish intents
explicitly. Experimental results on three challenging datasets show that our
proposed method achieves state-of-the-art performance.

##  8 Acknowledgements

This work is supported in part by NSF under grants III-1763325, III-1909323,
III-2106758, and SaTC-1930941. We thank the anonymous reviewers for their
helpful and thoughtful comments.

## References

  * Casanueva et al. (2020) Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020.  Efficient intent detection with dual sentence encoders.  _ACL 2020_ , page 38.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _ICML_ , pages 1597–1607.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_ , pages 4171–4186.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _EMNLP_.
  * Gunel et al. (2020) Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020.  Supervised contrastive learning for pre-trained language model fine-tuning.  _arXiv preprint arXiv:2011.01403_.
  * Gupta et al. (2018) Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018.  Semantic parsing for task oriented dialog using hierarchical representations.  In _EMNLP_ , pages 2787–2792.
  * Ham et al. (2020) Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020.  End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2.  In _ACL_ , pages 583–592.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , pages 9729–9738.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction.  In _EMNLP_ , pages 1311–1316.
  * Liu et al. (2021a) Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021a.  Fast, effective and self-supervised: Transforming masked language models into universal lexical and sentence encoders.  _arXiv preprint arXiv:2104.08027_.
  * Liu et al. (2019) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019.  Benchmarking natural language understanding services for building conversational agents.  _arXiv preprint arXiv:1903.05566_.
  * Liu et al. (2021b) Zhiwei Liu, Yongjun Chen, Jia Li, Philip S Yu, Julian McAuley, and Caiming Xiong. 2021b.  Contrastive self-supervised sequential recommendation with robust augmentation.  _arXiv preprint arXiv:2108.06479_.
  * Liu et al. (2021c) Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S Yu. 2021c.  Augmenting sequential recommendation with pseudo-prior items via reversely pre-training transformer.  _arXiv preprint arXiv:2105.00522_.
  * Liu et al. (2020) Zhiwei Liu, Xiaohan Li, Ziwei Fan, Stephen Guo, Kannan Achan, and S Yu Philip. 2020\.  Basket recommendation with multi-intent translation graph neural network.  In _IEEE Big Data_ , pages 728–737. IEEE.
  * Mehri et al. (2020a) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020a.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _arXiv preprint arXiv:2009.13570_.
  * Mehri et al. (2020b) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020b.  Example-driven intent prediction with observers.  _arXiv preprint arXiv:2010.08684_.
  * Peng et al. (2020) Baolin Peng, Chenguang Zhu, Michael Zeng, and Jianfeng Gao. 2020.  Data augmentation for spoken language understanding via pretrained language models.  _arXiv preprint arXiv:2004.13952_.
  * Qiu et al. (2020) Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020.  Gcc: Graph contrastive coding for graph neural network pre-training.  In _KDD_ , pages 1150–1160.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  _Image_ , 2:T2.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Tur et al. (2010) Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010.  What is left to be understood in atis?  In _SLT_ , pages 19–24. IEEE.
  * Wu et al. (2020a) Chien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming Xiong. 2020a.  ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues.  _EMNLP_.
  * Wu et al. (2020b) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020b.  Clear: Contrastive learning for sentence representation.  _arXiv preprint arXiv:2012.15466_.
  * Xia et al. (2021a) Congying Xia, Caiming Xiong, and Philip Yu. 2021a.  Pseudo siamese network for few-shot intent generation.  In _ACM SIGIR_.
  * Xia et al. (2020a) Congying Xia, Caiming Xiong, Philip Yu, and Richard Socher. 2020a.  Composed variational natural language generation for few-shot intents.  In _EMNLP: Findings_ , pages 3379–3388.
  * Xia et al. (2021b) Congying Xia, Wenpeng Yin, Yihao Feng, and S Yu Philip. 2021b.  Incremental few-shot text classification with multi-round new classes: Formulation, dataset and system.  In _NAACL-HLT_ , pages 1351–1360.
  * Xia et al. (2020b) Congying Xia, Chenwei Zhang, Hoang Nguyen, Jiawei Zhang, and Philip Yu. 2020b.  Cg-bert: Conditional text generation with bert for generalized few-shot intent detection.  _arXiv preprint arXiv:2004.01881_.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, et al. 2020.  Multilingual universal sentence encoder for semantic retrieval.  In _ACL_ , pages 87–94.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020.  Graph contrastive learning with augmentations.  _NeurIPS_ , 33:5812–5823.
  * Zhang et al. (2021) Jian-Guo Zhang, Kazuma Hashimoto, Yao Wan, Ye Liu, Caiming Xiong, and Philip S Yu. 2021.  Are pretrained transformers robust in intent classification? a missing ingredient in evaluation of out-of-scope intent detection.  _arXiv preprint arXiv:2106.04564_.
  * Zhang et al. (2020a) Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, S Yu Philip, Richard Socher, and Caiming Xiong. 2020a.  Discriminative nearest neighbor few-shot intent detection by transferring natural language inference.  In _EMNLP_ , pages 5064–5082.
  * Zhang et al. (2020b) Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, S Yu Philip, Richard Socher, and Caiming Xiong. 2020b.  Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking.  In _*SEM_ , pages 154–167. Association for Computational Linguistics.