# low mach number limit of full compressible Navier-Stokes equations with
revised Maxwell law

Zhao Wang and Yuxi Hu

###### Abstract.

In this paper, we study the low Mach number limit of the full compressible
Navier-Stokes equations with revised Maxwell law. By applying the uniform
estimation of the error system, we prove that the solutions of the full
compressible Navier-Stokes equations with time relaxation converge to that of
the incompressible Navier-Stokes equations as the Mach number tends to zero.
Moreover, the convergence rates are also obtained.
Keywords: Mach limit; Navier-Stokes equations; revised Maxwell law; energy
methods

Zhao Wang , Department of Mathematics, China University of Mining and
Technology, Beijing, 100083, P.R. China
Yuxi Hu, Department of Mathematics, China University of Mining and Technology,
Beijing, 100083, P.R. China<EMAIL_ADDRESS>
##  1\. Introduction

In this paper, we study the low Mach number limit of the full Navier-Stokes
equations with revised Maxwell law. The 3-d full compressible Navier-Stokes
equations are constructed from continuity equation, momentum equation and
energy equation which can be written in the following form (Euler coordinates)

$\begin{cases}\partial_{t}\rho+\text{div}(\rho u)=0,\\\ \partial_{t}(\rho
u)+\text{div}(\rho u\otimes u)+\nabla p=\text{div}S,\\\
\partial_{t}[\rho(e+\frac{1}{2}u^{2})]+\text{div}[\rho
u(e+\frac{1}{2}u^{2})+up]+\text{div}q=\text{div}(uS).\end{cases}$ (1.1)

Here $\rho$ and $u=(u_{1},u_{2},u_{3})$ represent fluid density, velocity,
respectively. The pressure $p=p(\rho,\theta)$ and internal energy
$e=(\rho,\theta)$ are smooth functions of the density $\rho$ and the absolute
temperature $\theta$, satisfying

$\rho^{2}e_{\rho}(\rho,\theta)=p(\rho,\theta)-\theta p_{\theta}(\rho,\theta).$

For classical fluids, the stress tensor $S$ and heat flux $q$ are given by
Newtonian law:

$S=\mu\left(\nabla u+(\nabla
u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right)+\lambda\text{div}uI_{3},$ (1.2)

and Fourier’s law:

$q+\kappa\nabla\theta=0.$ (1.3)

Where $I_{3}$ denotes the identity matrix in $\mathbb{R}^{3}$. $\nabla$ is the
gradient operator with respect to the space variable $x=(x_{1},x_{2},x_{3})$
and $\text{A}^{\top}$ stands for the transpose of the matrix A. $\mu$,
$\lambda$ and $\kappa$ are positive constants.

As is well-known, the Fourier’s law (1.3) predicts that heat waves have an
infinite propagation speed. In order to overcome the contradiction, Cattaneo
introduced the following well-known Cattaneo’s heat transfer law:

$\tau\partial_{t}q+q+\kappa\nabla\theta=0.$ (1.4)

For macromolecular or polymeric fluids, Maxwell [1] (1867) proposed so called
Maxwell law :

$\tau^{\prime}\partial_{t}S+S=\mu\left(\nabla u+(\nabla
u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right)+\lambda\text{div}uI_{3},$ (1.5)

with the relaxation parameter $\tau^{\prime}>0$. This relation indicates there
exists a time lag in response of stress tensor to velocity gradient and
divergence. Actually, this time lag exists even in simple fluids, water for
example, which can be negligible because of the small effect under macroscopic
conditions. However, the time lag can not be negligible for other fluids or
under other conditions, see [23, 24, 25] for more details. As for the model
systems with equation (1.5), see [6] for incompressible Navier-Stokes
equations. Yong [3] divided $S$ into two parts $S=S_{1}+S_{2}I_{3}$ and did
the relaxation for each one as follows:

$\begin{cases}\tau_{1}\partial_{t}S_{1}+S_{1}=\mu\left(\nabla u+(\nabla
u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right),\\\
\tau_{2}\partial_{t}S_{2}+S_{2}=\lambda\text{div}u.\end{cases}$ (1.6)

Here $S_{1}$ is a $n\times n$ square matrix and symmetric and traceless if it
was symmetric and traceless initially, and $S_{2}$ is a scalar variable. From
mathematic point of view, revised Maxwell’s relation make the relaxed system
(isentropic regime) symmetric. And as physical explanation, $\tau_{1}$ and
$\tau_{2}$ are shear relaxation time and compressional relaxation time, see a
similar revised Maxwell model in [4] for compressible viscoelastic fluid
(isentropic case). Moreover, local well-posedness and singular limit of
isentropic compressible Navier-Stokes equations with revised Maxwell’s law
have been established by Yong [3]. Later, Hu and Racke [5] study the non-
isentropic regime. They proved global well-posedness for small initial data
and investigated the relaxation limit as $\tau_{1}=\tau_{2}=\tau\rightarrow
0$.

The question of the low Mach number limit in fluid dynamics has received
considerable attention and has been the subject of research for more than 30
years. Some fundamental facts on incompressible limit in fluid dynamics have
been established by Klainerman and Majda in [8] for local smooth solution in
the case of no physical boundary condition and with well-prepared initial
data. Hoff, Danchin, Lions and Masmoudi [9, 10, 11, 12] also made a great
contribution to low Mach number limit for the isentropic compressible Navier-
Stokes equations under different conditions. For the low Mach number limit of
the full compressible Navier-Stokes equations, see [13] for local classical
solution with the effects of large temperature variations and thermal
condition, [14] for local strong solution in bounded region with Dirichlet
boundary condition for velocity and Neumann boundary condition for
temperature, and [15, 18, 19] in one-dimensional and three-dimensional regime.
Also see [16, 17, 21, 22] for more results. The whole work above were aimed at
the classical compressible Navier-Stokes equations with different initial-
boundary value in bounded and (or) unbounded regions.

Motivated by [20] where the author investigated the low Mach number limit of
the full compressible Navier-Stokes equations with Cattaneo’s heat transfer
law and rigorously justified that, in the framework of classical solutions
with small density, temperature and heat flux variations, the solutions of the
Navier-Stokes-Cattaneo system converge to that of the incompressible Navier-
Stokes equations as the Mach number tends to zero. The main purpose of this
paper is to present the rigorous justification of the low Mach number limit of
the Navier-Stokes equations with revised Maxwell’s relation for small
variations of density and temperature. We shall focus our study on the
polytropic ideal gas obeying the perfect relations:

$p=\Re\rho\theta,\quad e=C_{V}\theta,$ (1.7)

where $\Re$ and $C_{V}>0$ are the generic gases constant and the specific heat
at constant volume, respectively. The ratio of specific heats is
$\gamma=1+\frac{\Re}{C_{V}}$. Then we can take $\Re=1$ without loss of
generality.

We rewrite the system $\eqref{c N-S},\eqref{Fourier},\eqref{revised Maxwell}$
in the following form

$\begin{cases}\partial_{t}\rho+u\nabla\rho+\rho\text{div}u=0,\\\
\rho\partial_{t}u+\rho u\nabla u+\nabla(\rho\theta)=\text{div}S_{1}+\nabla
S_{2},\\\ \rho e_{\theta}\partial_{t}\theta+\rho
e_{\theta}u\nabla\theta+\theta
p_{\theta}\text{div}u-\kappa\triangle\theta=(S_{1}+S_{2}I_{3})\nabla u,\\\
\tau_{1}\partial_{t}S_{1}+S_{1}=\mu\left(\nabla u+(\nabla
u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right),\\\
\tau_{2}\partial_{t}S_{2}+S_{2}=\lambda\text{div}u.\end{cases}$ (1.8)

Scale $\rho,u,\theta,S_{1},S_{2}$ in the following way:

$\displaystyle\rho(x,t)=\widetilde{\rho}(x,\epsilon t),\quad
u(x,t)=\epsilon\widetilde{u}(x,\epsilon
t),\quad\theta(x,t)=\widetilde{\theta}(x,\epsilon t),$ $\displaystyle
S_{1}(x,t)=\epsilon^{2}\widetilde{S_{1}}(x,\epsilon t),\quad
S_{2}(x,t)=\epsilon^{2}\widetilde{S_{2}}(x,\epsilon t),$

and assume that the coefficients $\mu,\lambda,\kappa,\tau_{1}$ and $\tau_{2}$
are small and scaled as:

$\mu=\epsilon\mu^{\epsilon},\quad\lambda=\epsilon\lambda^{\epsilon},\quad\kappa=\epsilon\kappa^{\epsilon},\quad\tau_{1}=\frac{\tau_{1}^{\epsilon}}{\epsilon},\quad\tau_{2}=\frac{\tau_{2}^{\epsilon}}{\epsilon},$

where $\epsilon>0\in(0,1)$ is the reference Mach number and normalized
coefficients
$\mu^{\epsilon},\lambda^{\epsilon},\kappa^{\epsilon},\tau_{1}^{\epsilon}$ and
$\tau_{2}^{\epsilon}$ satisfy

$\mu^{\epsilon}\rightarrow\bar{\mu},\quad\lambda^{\epsilon}\rightarrow\bar{\lambda},\quad\kappa^{\epsilon}\rightarrow\bar{\kappa},\quad\tau_{1}^{\epsilon}\rightarrow
0,\quad\tau_{2}^{\epsilon}\rightarrow 0,\quad\text{as}\quad\epsilon\rightarrow
0.$

We recover the non-dimensional system as follows (still denoted by
$\rho,u,\theta,S_{1},S_{2}$):

$\begin{cases}\partial_{t}\rho+u\nabla\rho+\rho\text{div}u=0,\\\
\rho\partial_{t}u+\rho u\nabla
u+\frac{\nabla(\rho\theta)}{\epsilon^{2}}=\text{div}S_{1}+\nabla S_{2},\\\
\rho e_{\theta}\partial_{t}\theta+\rho e_{\theta}u\nabla\theta+\theta
p_{\theta}\text{div}u-\kappa\triangle\theta=\epsilon^{2}(S_{1}+S_{2}I_{3})\nabla
u,\\\ \tau_{1}^{\epsilon}\partial_{t}S_{1}+S_{1}=\mu^{\epsilon}\left(\nabla
u+(\nabla u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right),\\\
\tau_{2}^{\epsilon}\partial_{t}S_{2}+S_{2}=\lambda^{\epsilon}\text{div}u.\end{cases}$
(1.9)

We further restrict ourselves to the small density and temperature variations,
so we assume that

$\rho=1+\epsilon\eta,\quad\theta=1+\epsilon\phi.$

Then we can rewrite (1.9) (for $\eta,u,\phi,S_{1},S_{2}$) as

$\begin{cases}\partial_{t}\eta+u\nabla\eta+\frac{1+\epsilon\eta}{\epsilon}\text{div}u=0,\\\
(1+\epsilon\eta)(\partial_{t}u+u\cdot\nabla
u)+\frac{1}{\epsilon}((1+\epsilon\phi)\nabla\eta+(1+\epsilon\eta)\nabla\phi)=\text{div}S_{1}+\nabla
S_{2},\\\
(1+\epsilon\eta)(\partial_{t}\phi+u\cdot\nabla\phi)+\frac{\gamma-1}{\epsilon}(1+\epsilon\phi)(1+\epsilon\eta)\text{div}u-\kappa^{\epsilon}\triangle\phi=\epsilon(S_{1}+S_{2}I_{3})\nabla
u,\\\ \tau_{1}^{\epsilon}\partial_{t}S_{1}+S_{1}=\mu^{\epsilon}\left(\nabla
u+(\nabla u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right),\\\
\tau_{2}^{\epsilon}\partial_{t}S_{2}+S_{2}=\lambda^{\epsilon}\text{div}u.\end{cases}$
(1.10)

The system (1.10) is equipped with the initial data

$(\eta,u,\phi,S_{1},S_{2})|_{t=0}=(\eta_{0}(x),u_{0}(x),\phi_{0}(x),S_{10}(x),S_{20}(x)).$
(1.11)

The formal limit as $\epsilon\longrightarrow 0$ of (1.10) is the
incompressible Navier-Stokes equations (we suppose that the limit
$u\longrightarrow w$ exists.)

$\begin{cases}\partial_{t}w+(w\cdot\nabla)w+\nabla\pi=\bar{\mu}\triangle w,\\\
\text{div}w=0,\end{cases}$ (1.12)

with the initial data

$w|_{t=0}=w_{0}(x),$ (1.13)

where $\pi$ represents the limiting pressure.

In the present paper, we shall establish the above limit rigorously in a
convex compact subset of G where G denotes the physical state spaces of
$(\eta,u,\phi,S_{1},S_{2})$. Moreover, using the theory of symmetric
hyperbolic parabolic system, the system (1.10) admits a smooth solution on the
time interval where the smooth solution of the incompressible Navier-Stokes
equations exists. The main result of this paper is stated here.

######  Theorem 1.1.

Let $s\geq 4$ be integers. Suppose that the initial data (1.11) satisfy

$\left|\left|\left(\eta_{0}(x),u_{0}(x)-w_{0}(x),\phi_{0}(x)\right)\right|\right|_{s}=O(\epsilon),$
$\left|\left|\left(S_{10}(x)-\mu^{\epsilon}(\nabla w_{0}(x)+(\nabla
w_{0}(x))^{\top}),S_{20}(x)\right)\right|\right|_{s}=O(\sqrt{\epsilon}).$

Let $(w,\pi)$ be a smooth solution to the system $\eqref{incompressible
n-s}\sim\eqref{incompressible initial data}$ on $[0,T_{*}]\times G$. We assume
$\pi\in C([0,T_{*}],H^{s+2})\cap C^{1}([0,T_{*}],H^{s+1})$ and $w\in
C([0,T_{*}],H^{s+3})\cap C^{1}([0,T_{*}],H^{s+1})$. Then then there exists a
constant $\epsilon_{0}>0$, such that, for all $\epsilon<\epsilon_{0}$, the
system (1.10) with initial data (1.11) has a unique smooth solution
$(\rho,u,\phi,S_{1},S_{2})\in C([0,T_{*}],H^{s})$. Moreover,there exists a
positive constant $K>0$,independent of $\epsilon$, such that, for all
$\epsilon<\epsilon_{0}$,

$\sup_{t\in[0,T_{*}]}\left|\left|(\eta-\frac{\epsilon\pi}{2},u-w,\phi-\frac{\epsilon\pi}{2})\right|\right|_{s}\leq
K\epsilon,$ $\sup_{t\in[0,T_{*}]}\left|\left|(S_{1}-\mu^{\epsilon}(\nabla
w+(\nabla
w)^{\top}),S_{2}-\epsilon\lambda^{\epsilon}\pi)\right|\right|_{s}\leq
K\sqrt{\epsilon}.$

######  Remark 1.1.

It is needed and reasonable to require the high regularity of $(w,\pi)$ in
order to estimate some terms in error system. See more details in Section 3.

The rest of present paper is arranged as follows. In Section 2, we show a
local existence theorem by transforming the system (1.10) into a symmetric
hyperbolic-parabolic type system. Then, we construct the approximate system to
the system (1.10). In Section 3, we first establish the uniform estimations of
the error system. Then we prove the convergence of relaxed compressible system
(1.10) to the incompressible system (1.12) and drive the convergence rates.

Finally, let’s end up this section with some notations used throughout the
current paper. We use the letter C to denote various positive constants
independent of $\epsilon$. Denote by
$\nabla^{\alpha}=\partial_{x_{1}}^{\alpha_{1}}\partial_{x_{2}}^{\alpha_{2}}\partial_{x_{3}}^{\alpha_{3}}$
the partial derivative of order $|\alpha|=\alpha_{1}+\alpha_{3}+\alpha_{3}$
with the multi-index $\alpha=(\alpha_{1},\alpha_{2},\alpha_{3})$. $L^{2}$
denotes the spaces of measurable function which are square integrable, with
the norm $||\cdot||$. $H^{s}(s\geq 0)$ denotes the Sobolev space of
$L^{2}$-functions $f$ whose derivatives $\partial_{x}^{j}f,j=1,\cdots s$ are
also $L^{2}$-functions, with the norm $||\cdot||_{s}$. Let T and B be a
positive constant and a Banach space, respectively. $C^{k}(0,T;B)(k\geq 0)$
denotes the space of B-valued k-times continuously differentiable functions on
$[0,T]$, and $L^{2}(0,T;B)$ denotes the space of B-valued $L^{2}$-functions on
$[0,T]$. The corresponding space B-valued functions on $[0,\infty]$ are
defined similarly.

##  2\. Local existence and construction of approximation solutions

We first show the local existence of the solutions of (1.10). To this end, we
rewrite (1.10) in the form:

$\begin{cases}\partial_{t}\eta+u\nabla\eta+\frac{1+\varepsilon\eta}{\epsilon}\text{div}u=0,\\\
(1+\epsilon\eta)(\partial_{t}u+u\cdot\nabla
u)+\frac{1}{\epsilon}((1+\epsilon\phi)\nabla\eta+(1+\epsilon\eta)\nabla\phi)-(\text{div}S_{1}+\nabla
S_{2})=0,\\\
(1+\epsilon\eta)(\partial_{t}\phi+u\cdot\nabla\phi)+\frac{\gamma-1}{\epsilon}(1+\epsilon\phi)(1+\epsilon\eta)\text{div}u-\kappa^{\epsilon}\triangle\phi=\epsilon[(S_{1}+S_{2}I_{3})\nabla
u],\\\ \tau_{1}^{\epsilon}\partial_{t}S_{1}+S_{1}-\mu^{\epsilon}\left(\nabla
u+(\nabla u)^{\top}-\frac{2}{3}\text{div}uI_{3}\right)=0,\\\
\tau_{2}^{\epsilon}\partial_{t}S_{2}+S_{2}-\lambda^{\epsilon}\text{div}u=0.\end{cases}$
(2.1)

Without loss of generality, we assume $S_{1}$ take the following form:

$S_{1}=\begin{bmatrix}a_{11}&a_{12}&a_{13}\\\ a_{12}&a_{22}&a_{23}\\\
a_{13}&a_{23}&-(a_{11}+a_{22})\end{bmatrix}.$

Let $U^{\epsilon}=(\eta,u,a_{11},a_{12},a_{13},a_{22},a_{23},S_{2})$. Then, we
have

$\begin{cases}A_{0}\partial_{t}U^{\varepsilon}+\sum_{j=1}^{3}A_{j}\partial_{j}U^{\varepsilon}+BU^{\varepsilon}=F,\\\
(1+\epsilon\eta)\partial_{t}\phi-\kappa^{\epsilon}\triangle\phi=H.\end{cases}$
(2.2)

Here
$A_{0}=diag\left\\{\frac{1+\epsilon\phi}{1+\epsilon\eta},(1+\epsilon\eta),(1+\epsilon\eta),(1+\epsilon\eta),\frac{3\tau_{1}^{\epsilon}}{4\mu^{\epsilon}},\frac{\tau_{1}^{\epsilon}}{\mu^{\epsilon}},\frac{\tau_{1}^{\epsilon}}{\mu^{\epsilon}},\frac{3\tau_{1}^{\epsilon}}{4\mu^{\epsilon}},\frac{\tau_{1}^{\epsilon}}{\mu^{\epsilon}},\frac{\tau_{2}^{\epsilon}}{\lambda^{\epsilon}}\right\\},$
$B=diag\left\\{0,0,0,0,\frac{3}{4\mu^{\epsilon}},\frac{1}{\mu^{\epsilon}},\frac{1}{\mu^{\epsilon}},\frac{3}{4\mu^{\epsilon}},\frac{1}{\mu^{\epsilon}},\frac{1}{\lambda^{\epsilon}}\right\\},F=\left(0,\frac{(1+\epsilon\eta)}{\epsilon}\nabla\phi,0,0,0,0,0,0\right),\\\
H=\epsilon[(S_{1}+S_{2}I_{3})\nabla
u]-\frac{\gamma-1}{\epsilon}(1+\epsilon\phi)(1+\epsilon\eta)divu-(1+\epsilon\eta)u\cdot\nabla\phi,$

$\sum_{j=1}^{3}A_{j}\xi_{j}=\begin{bmatrix}\frac{1+\epsilon\phi}{1+\epsilon\eta}u\xi&\frac{1}{\epsilon}(1+\epsilon\phi)\xi&0_{1\times
5}&0\\\ \frac{1}{\epsilon}(1+\epsilon\phi)\xi^{\top}&(1+\epsilon\eta)u\xi
I_{3}&C_{3\times 5}&-\xi^{\top}\\\ 0_{5\times 1}&D_{5\times 3}&0_{5\times
5}&0_{5\times 1}\\\ 0&-\xi&0_{1\times 5}&0\\\ \end{bmatrix},$

where

$C_{3\times 5}=\begin{bmatrix}-\xi_{1}&-\xi_{2}&-\xi_{3}&0&0\\\
0&-\xi_{1}&0&-\xi_{2}&-\xi_{3}\\\ \xi_{3}&0&-\xi_{1}&\xi_{3}&-\xi_{2}\\\
\end{bmatrix},\quad D_{5\times
3}=\begin{bmatrix}-\xi_{1}&\frac{1}{2}\xi_{2}&\frac{1}{2}\xi_{3}\\\
-\xi_{2}&-\xi_{1}&0\\\ -\xi_{3}&0&-\xi_{1}\\\
\frac{1}{2}\xi_{1}&-\xi_{2}&\frac{1}{2}\xi_{3}\\\ 0&-\xi_{3}&-\xi_{2}\\\
\end{bmatrix},$

for each $\xi\in\mathcal{S}^{3}$.

We do the transformation:
$b_{11}=\frac{a_{11}+a_{22}}{2},b_{22}=\frac{a_{11}-a_{22}}{2}$ which
particularly implies $a_{11}=b_{11}+a_{22},a_{22}=b_{11}-b_{22}$ and let
$\widetilde{U^{\epsilon}}=(\eta,u,b_{11},a_{12},a_{13},b_{22},a_{23},S_{2})$.
Then, it is easily to show that the system (2.2) is a symmetric hyperbolic
parabolic system (see [5] for more details). Therefore, we obtain the local
existence of solutions for the system (2.1) .

######  Lemma 2.1 (Local existence).

Under the conditions in Theorem 1.1, the system (2.1) alow a unique classical
solutions $(\rho,u,\phi,S_{1},S_{2})$ for each sufficiently small
$\varepsilon$ on $[0,T^{\varepsilon}]\times G$ satisfying

$(\rho,u,\phi,S_{1},S_{2})\in C([0,T^{\varepsilon}]\times H^{s})\cap
C^{1}([0,T^{\varepsilon}]\times H^{s-1}).$

Now we define

$T_{\epsilon}=sup\\{T^{\epsilon}:(\rho,u,\phi,S_{1},S_{2})\in
C([0,T^{\epsilon}]\times H^{s})\\}.$

We shall make use of the following convergence-stability lemma to show that
$\liminf_{\epsilon\rightarrow 0}T_{\varepsilon}>0.$

######  Lemma 2.2.

$($[2, 20]$)$ Let $s\geq 4$. Suppose that $U^{\epsilon}_{0}\in G_{0}$ and the
following convergence assumption (H) holds.

(H) For each $\epsilon$, there exists $T_{*}\geq 0$ and $U_{\epsilon}\in
L^{\infty}([0,T_{*}];H^{s})$ satisfying

$\bigcap_{x,t,\epsilon}U_{\epsilon}\\{(x,t)\\}\subset\subset G$

such that, for $t\in(0,\min\\{T_{*},T_{\epsilon}\\})$,

$\sup_{x,t}|U^{\epsilon}-U_{\epsilon}|=o(1),\quad\sup_{t}||U^{\epsilon}-U_{\epsilon}||_{s}=O(1).$

Then there exists an $\bar{\epsilon}>0$ such that,
$\forall\epsilon\in(0,\bar{\epsilon}]$, it holds that $T_{\epsilon}>T_{*}$.

Now, we construct the approximation
$U_{\varepsilon}=(\eta_{\epsilon},u_{\epsilon},\phi_{\epsilon},S_{1\epsilon},S_{2\epsilon})$
to the original system (2.1) with
$\eta_{\epsilon}=\frac{\epsilon\pi}{2},u_{\epsilon}=w,\phi_{\epsilon}=\frac{\epsilon\pi}{2},S_{1\epsilon}=\mu^{\epsilon}(\nabla
w+(\nabla w)^{\top}),S_{2\epsilon}=\epsilon\lambda^{\epsilon}\pi,$ where
$(w,\pi)$ is the smooth solution to the system (1.12),(1.13). It is easy to
verify that
$(\eta_{\epsilon},u_{\epsilon},\phi_{\epsilon},S_{1\epsilon},S_{2\epsilon})$
satisfy

$\begin{cases}\partial_{t}\eta_{\epsilon}+u_{\epsilon}\nabla\eta_{\epsilon}+\frac{1+\varepsilon\eta_{\epsilon}}{\epsilon}\text{div}u_{\epsilon}=\frac{\epsilon}{2}(\pi_{t}+w\nabla\pi)=:f_{1},\\\
(1+\epsilon\eta_{\epsilon})(\partial_{t}u_{\epsilon}+u_{\epsilon}\cdot\nabla
u_{\epsilon})+\frac{1}{\epsilon}((1+\epsilon\phi_{\epsilon})\nabla\eta_{\epsilon}+(1+\epsilon\eta_{\epsilon})\nabla\phi_{\epsilon})-(\text{div}S_{1\epsilon}+\nabla
S_{2\epsilon})\\\ \quad=\frac{\epsilon^{2}\pi}{2}(w_{t}+w\cdot\nabla
w+\nabla\pi)-\epsilon\lambda^{\epsilon}\nabla\pi=:f_{2},\\\
(1+\epsilon\eta_{\epsilon})(\partial_{t}\phi_{\epsilon}+u_{\epsilon}\cdot\nabla\phi_{\epsilon})+\frac{\gamma-1}{\epsilon}(1+\epsilon\phi_{\epsilon})(1+\epsilon\eta_{\epsilon})\text{div}u_{\epsilon}-\kappa^{\epsilon}\triangle\phi_{\epsilon}\\\
\quad=(\frac{\epsilon}{2}+\frac{\epsilon^{3}\pi}{4})(\pi_{t}+w\nabla\pi)-\frac{\epsilon\kappa}{2}\triangle\pi=:f_{3},\\\
\tau_{1}^{\epsilon}\partial_{t}S_{1\epsilon}+S_{1\epsilon}-\mu^{\epsilon}\left(\nabla
u_{\epsilon}+(\nabla
u_{\epsilon})^{\top}-\frac{2}{3}\text{div}u_{\epsilon}I_{3}\right)=\tau_{1}^{\epsilon}\mu^{\epsilon}(\nabla
w_{t}+(\nabla w_{t})^{\top})=:f_{4},\\\
\tau_{2}^{\epsilon}\partial_{t}S_{2\epsilon}+S_{2\epsilon}-\lambda^{\epsilon}\text{div}u_{\epsilon}=\epsilon\lambda^{\epsilon}(\tau_{2}\pi_{t}+\pi)=:f_{5}.\\\
\end{cases}$ (2.3)

Due to the regularity assumptions on $(w,\pi)$ in Theorem 1.1 and
$\tau_{1}^{\epsilon}=O(\epsilon)$ and $\tau_{2}^{\epsilon}=O(\epsilon)$, we
have the fact:

$||f_{i}||_{s}\leq C\epsilon,\quad i=1,2,3,4,5.$ (2.4)

##  3\. The uniform estimates of the error system

In this section, we first establish the uniform estimates of the error system.
Then based on this result, we verify the convergence assumption (H) and
accomplish the proof of Theorem 1.1. To this end, we now introduce

$\eta^{d}=\eta-\eta_{\epsilon},u^{d}=u-u_{\epsilon},\phi^{d}=\phi-\phi_{\epsilon},S_{1}^{d}=S_{1}-S_{1\epsilon},S_{2}^{d}=S_{2}-S_{2\epsilon}$

and utilize (2.1) and (2.3). Then, we get the error system in the following
form

$\begin{cases}\partial_{t}\eta^{d}+(u\cdot\nabla)\eta^{d}+\frac{1+\epsilon\eta}{\epsilon}\text{div}u^{d}=-u^{d}\nabla\eta_{\epsilon}-f_{1},\\\
\partial_{t}u^{d}+(u\cdot\nabla)u^{d}+\frac{1}{\epsilon}\left[\frac{(1+\epsilon\phi)\nabla\eta^{d}}{(1+\epsilon\eta)}+\nabla\phi^{d}\right]-\frac{1}{(1+\epsilon\eta)}(\text{div}S_{1}^{d}+\nabla
S_{2}^{d})\\\
\qquad\qquad=-\frac{\epsilon\eta^{d}\partial_{t}u_{\epsilon}}{(1+\epsilon\eta)}-\frac{((1+\epsilon\eta)u-(1+\epsilon\eta_{\epsilon})u_{\epsilon})}{(1+\epsilon\eta)}\cdot\nabla
u_{\epsilon}-\frac{(\eta^{d}\nabla\phi_{\epsilon}+\phi^{d}\nabla\eta_{\epsilon})}{(1+\epsilon\eta)}-\frac{1}{(1+\epsilon\eta)}f_{2},\\\
\partial_{t}\phi^{d}+(u\cdot\nabla)\phi^{d}+\frac{\gamma-1}{\epsilon}(1+\epsilon\phi)\text{div}u^{d}-\frac{\kappa^{\epsilon}}{(1+\epsilon\eta)}\triangle\phi^{d}\\\
\qquad\qquad=-\frac{\epsilon\eta^{d}\partial_{t}\phi_{\epsilon}}{(1+\epsilon\eta)}-\frac{((1+\epsilon\eta)u-(1+\epsilon\eta_{\epsilon})u_{\epsilon})}{(1+\epsilon\eta)}\cdot\nabla\phi_{\epsilon}+\frac{\epsilon[(S_{1}+S_{2}I_{3})\nabla
u]}{(1+\epsilon\eta)}-\frac{1}{(1+\epsilon\eta)}f_{3},\\\
\tau_{1}^{\epsilon}\partial_{t}S_{1}^{d}+S_{1}^{d}-\mu^{\epsilon}\left(\nabla
u^{d}+(\nabla u^{d})^{\top}-\frac{2}{3}\text{div}u^{d}I_{3}\right)=-f_{4},\\\
\tau_{2}^{\epsilon}\partial_{t}S_{2}^{d}+S_{2}^{d}-\lambda^{\epsilon}\text{div}u^{d}=-f_{5}.\\\
\end{cases}$ (3.1)

Define

$E=E(t):=\left|\left|(\rho^{d},u^{d},\phi^{d},\sqrt{\tau_{1}^{\epsilon}}S_{1}^{d},\sqrt{\tau_{2}^{\epsilon}}S_{2}^{d})\right|\right|_{s}.$

Note that

$||(\rho_{\epsilon},u_{\epsilon},\phi_{\epsilon},S_{1\epsilon},S_{2\epsilon})||_{s}\leq
C$

and

$||(\rho,u,\phi)||_{s}\leq C(1+E),\quad||S_{1}||_{s}\leq
C\left(1+\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}E\right),\quad||S_{2}||_{s}\leq
C\left(1+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}E\right).$

Our aim is to show that, for small $\epsilon$ and for all
$t\in(0,\min\\{T_{\epsilon},T_{*}\\})$, $E(t)\leq C\epsilon$, provided that
$E(0)\leq C\epsilon_{0}$ is suitably small. Now, we list the Moser-type
inequalities (see [7]) and the nonlinear Gronwall-type inequality (see [2])
which will be used in the following lemma 3.3.

######  Lemma 3.1 (Moser-type inequalities).

Let $k\in\mathbb{N}$. Then there is a constant $c=c(k,n)>0$ such that for all
f,g $\in W^{k,2}\cap L^{\infty}$,and $\alpha\in\mathbb{N}_{0}^{n},|\alpha|\leq
k$, the following inequalities hold:

$\displaystyle||\nabla^{\alpha}(fg)||\leq
c(||f||_{\infty}\cdot||\nabla^{k}g||+||g||_{\infty}\cdot||\nabla^{k}f||),$
$\displaystyle||\nabla^{\alpha}(fg)-f\nabla^{\alpha}g||\leq c(||\nabla
f||_{\infty}\cdot||\nabla^{k-1}g||+||g||_{\infty}\cdot||\nabla^{k}f||).$

######  Lemma 3.2 (Nonlinear Gronwall-type inequality).

Suppose $\sigma(t)$ is a positive $C^{1}$-function of $t\in[0,T)$ with
$T\leq\infty$, $m>1$ and $b_{1}(t),b_{2}(t)$ are integrable on $[0,T)$. If

$\sigma^{\prime}(t)\leq b_{2}\sigma^{m}(t)+b_{1}\sigma(t),$

then there exists a $\delta>0$, depending only on m, $C_{1b}$ and $C_{2b}$,
such that

$\sup_{t\in[0,T)}\sigma(t)\leq e^{C_{1b}},$

whenever $\sigma(0)\in(0,\delta]$. Here

$C_{1b}=\sup_{t\in[0,T)}\int_{0}^{t}b_{1}(s)ds\quad and\quad
C_{2b}=\sup_{t\in[0,T)}\int_{0}^{t}max\\{b_{2}(s),0\\}ds.$

The following lemma is essential to prove the Theorem 1.1.

######  Lemma 3.3.

For suitably small $\epsilon\in(0,1)$, we have

$\frac{dE^{2}}{dt}\leq C(1+E^{2})E^{2}+C\epsilon^{2}.$

###### Proof.

Applying $\nabla^{\alpha}(|\alpha|\leq s)$ to the error system (3.1),
multiplying the result by
$\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d},\\\
(1+\epsilon\eta)\nabla^{\alpha}u^{d},\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d},\frac{1}{2\mu^{\epsilon}}\nabla^{\alpha}S_{1}^{d},\frac{1}{\lambda^{\epsilon}}\nabla^{\alpha}S_{2}^{d}$
and integrating with respect to $x$, we get

$\displaystyle\frac{1}{2}\frac{d}{dt}\int$
$\displaystyle\left\\{\frac{1+\epsilon\phi}{1+\epsilon\eta}(\nabla^{\alpha}\eta^{d})^{2}+(1+\epsilon\eta)(\nabla^{\alpha}u^{d})^{2}+\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}(\nabla^{\alpha}\phi^{d})^{2}\right.$
(3.2)
$\displaystyle\left.+\frac{\tau_{1}^{\epsilon}}{2\mu^{\epsilon}}(\nabla^{\alpha}S_{1}^{d})^{2}+\frac{\tau_{2}^{\epsilon}}{\lambda^{\epsilon}}(\nabla^{\alpha}S_{2}^{d})^{2}\right\\}dx$
$\displaystyle+\int$
$\displaystyle\left\\{\frac{1}{2\mu^{\epsilon}}(\nabla^{\alpha}S_{1}^{d})^{2}+\frac{\kappa^{\epsilon}}{(\gamma-1)(1+\epsilon\phi)}(\nabla^{\alpha+1}\phi^{d})^{2}+\frac{1}{\lambda^{\epsilon}}(\nabla^{\alpha}S_{2}^{d})^{2}\right\\}dx$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{3}|T_{i}|+\sum_{i=1}^{5}|F_{i}|+\sum_{i=1}^{6}|G_{i}|+\sum_{i=1}^{7}|N_{i}|+|D|,$

where

$\displaystyle
T_{1}=\int\frac{1}{2}\left(\frac{1+\epsilon\phi}{1+\epsilon\eta}\right)_{t}(\nabla^{\alpha}\eta^{d})^{2}dx,T_{2}=\int\frac{1}{2}(1+\epsilon\eta)_{t}(\nabla^{\alpha}u^{d})^{2}dx,$
$\displaystyle
T_{3}=\int\frac{1}{2}\left(\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\right)_{t}(\nabla^{\alpha}\phi^{d})^{2}dx,F_{1}=\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\nabla^{\alpha}f_{1}dx,$
$\displaystyle
F_{2}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(\frac{f_{2}}{1+\epsilon\eta}\right)dx,F_{3}=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}\left(\frac{f_{3}}{1+\epsilon\eta}\right)dx,$
$\displaystyle
F_{4}=\int\frac{1}{2\mu^{\epsilon}}\nabla^{\alpha}S_{1}^{d}\nabla^{\alpha}f_{4}dx,F_{5}=\int\frac{1}{\lambda^{\epsilon}}\nabla^{\alpha}S_{2}^{d}\nabla^{\alpha}f_{5}dx,$
$\displaystyle
G_{1}=\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\nabla^{\alpha}(u\cdot\nabla\eta^{d})dx,G_{2}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(u\cdot\nabla
u^{d}\right)dx,$ $\displaystyle
G_{3}=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}\left(u\cdot\nabla\phi^{d}\right)dx,$
$\displaystyle
G_{4}=\frac{1}{\epsilon}\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\nabla^{\alpha}\left[(1+\epsilon\eta)\text{div}u^{d}\right]+(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left[\frac{(1+\epsilon\phi)}{(1+\epsilon\eta)}\nabla\eta^{d}\right]dx,$
$\displaystyle
G_{5}=\frac{1}{\epsilon}\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha+1}\phi^{d}+\frac{1+\epsilon\eta}{1+\epsilon\phi}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}[(1+\epsilon\phi)\text{div}u^{d}]dx,$
$\displaystyle
G_{6}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(\frac{1}{(1+\epsilon\eta)}(\text{div}S_{1}^{d}+\nabla
S_{2}^{d})\right)$
$\displaystyle\qquad\quad+\frac{1}{2}\nabla^{\alpha}S_{1}^{d}\nabla^{\alpha}\left(\nabla
u^{d}+(\nabla
u^{d})^{\top}-\frac{2}{3}\text{div}u^{d}I_{3}\right)+\nabla^{\alpha}S_{2}^{d}\nabla^{\alpha}(\text{div}u^{d})dx,$
$\displaystyle
N_{1}=\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\nabla^{\alpha}(u^{d}\nabla\eta_{\epsilon})dx,N_{2}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(\frac{\epsilon\eta^{d}\partial_{t}u_{\epsilon}}{1+\epsilon\eta}\right)dx,$
$\displaystyle
N_{3}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left[\frac{(1+\epsilon\eta)u-(1+\epsilon\eta_{\epsilon})u_{\epsilon}}{(1+\epsilon\eta)}\cdot\nabla
u_{\epsilon}\right]dx,$ $\displaystyle
N_{4}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(\frac{\eta^{d}\nabla\phi_{\epsilon}+\phi^{d}\nabla\eta_{\epsilon}}{1+\epsilon\eta}\right)dx,N_{5}=\int\frac{(1+\epsilon\eta)\nabla^{\alpha}\phi^{d}}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\left(\frac{\epsilon\eta^{d}\partial_{t}\phi_{\epsilon}}{1+\epsilon\eta}\right)dx,$
$\displaystyle
N_{6}=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}\left[\frac{(1+\epsilon\eta)u-(1+\epsilon\eta_{\epsilon})u_{\epsilon}}{(1+\epsilon\eta)}\cdot\nabla\phi_{\epsilon}\right]dx,$
$\displaystyle
N_{7}=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx.$ $\displaystyle
D=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\left[\nabla^{\alpha}\left(\frac{\kappa^{\epsilon}}{(1+\epsilon\eta)}\triangle\phi^{d}\right)-\frac{\kappa^{\epsilon}}{(1+\epsilon\eta)}\nabla^{\alpha}\triangle\phi^{d}\right]$
$\displaystyle\qquad\quad+\nabla\left[\frac{\kappa^{\epsilon}}{(\gamma-1)(1+\epsilon\phi)}\right]\nabla^{\alpha}\phi^{d}\nabla^{\alpha+1}\phi^{d}dx.$

Note that in the time interval $[0,min({T_{*},T_{\epsilon}})]$, both
$(\eta,u,\phi,S_{1},S_{2})(t)$ and
$(\eta_{\epsilon},u_{\epsilon},\phi_{\epsilon},S_{1\epsilon},S_{2\epsilon})(t)$
are regular enough and take values in a convex compact subset of G. It’s easy
to know

$\displaystyle|(F_{1},F_{2},F_{3})|\leq
C\epsilon^{2}+C||\nabla^{\alpha}(\eta^{d},u^{d},\phi^{d})||^{2}\leq
C\epsilon^{2}+CE^{2},$ $\displaystyle|F_{4}|\leq
C\epsilon^{2}+\delta_{1}||\nabla^{\alpha}S_{1}^{d}||^{2},\quad|F_{5}|\leq
C\epsilon^{2}+\delta_{2}||\nabla^{\alpha}S_{2}^{d}||^{2}.$

We now estimate $G_{1}\sim G_{6}$.

$\displaystyle|G_{1}|$
$\displaystyle\leq\left|\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\left[\nabla^{\alpha}(u\nabla\eta^{d})-u\nabla^{\alpha+1}\eta^{d}\right]+\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}u\nabla^{\alpha+1}\eta^{d}dx\right|$
$\displaystyle\leq
C(||u||_{\infty}||\nabla^{\alpha}\eta^{d}||+||\nabla\eta^{d}||_{\infty}||\nabla^{\alpha}u||)||\nabla^{\alpha}\eta^{d}||+\left|\left|\text{div}\left(\frac{(1+\epsilon\phi)u}{1+\epsilon\eta}\right)\right|\right|_{\infty}||\nabla^{\alpha}\eta^{d}||^{2}$
$\displaystyle\leq C(1+E)E^{2}.$

$G_{2}\sim G_{3}$ can be estimated in the similar way. As for the singular
terms $G_{4}\sim G_{6}$, we have

$\displaystyle G_{4}=\frac{1}{\epsilon}\int$
$\displaystyle(1+\epsilon\phi)\text{div}(\nabla^{\alpha}u^{d}\nabla^{\alpha}\eta^{d})$
$\displaystyle+\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\\{\nabla^{\alpha}[(1+\epsilon\eta)\text{div}u^{d}]-(1+\epsilon\eta)\nabla^{\alpha}\text{div}u^{d}\\}$
$\displaystyle+(1+\epsilon\eta)\nabla^{\alpha}u^{d}\left[\nabla^{\alpha}\left(\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla\eta^{d}\right)-\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}(\nabla\eta^{d})\right]dx$
$\displaystyle\leq C||($
$\displaystyle\nabla^{\alpha}u^{d},\nabla^{\alpha}\eta^{d})||\Big{(}||\nabla\eta||_{\infty}||\nabla^{\alpha}u^{d}||+||\text{div}u^{d}||_{\infty}||\nabla^{\alpha}\eta||$
$\displaystyle+||\nabla\eta||_{\infty}||\nabla\phi||_{\infty}||\nabla^{\alpha}\eta^{d}||+||\nabla\eta^{d}||_{\infty}||\nabla^{\alpha}\phi||||\nabla^{\alpha}\eta||\Big{)}$
$\displaystyle\leq CE$ ${}^{2}(1+E^{2})$

and

$\displaystyle G_{5}=\frac{1}{\epsilon}\int$
$\displaystyle(1+\epsilon\eta)\text{div}(\nabla^{\alpha}u^{d}\nabla^{\alpha}\phi^{d})$
$\displaystyle+\frac{1+\epsilon\eta}{1+\epsilon\phi}\nabla^{\alpha}u^{d}\\{\nabla^{\alpha}[(1+\epsilon\phi)\text{div}u^{d}]-(1+\epsilon\phi)\nabla^{\alpha}\text{div}u^{d}\\}dx$
$\displaystyle\leq C||($
$\displaystyle\nabla^{\alpha}u^{d},\nabla^{\alpha}\phi^{d})||\Big{(}||\nabla\eta||_{\infty}+||\nabla\phi||_{\infty}||\nabla^{\alpha}u^{d}||+||\text{div}u^{d}||_{\infty}||\nabla^{\alpha}\phi||\Big{)}$
$\displaystyle\leq CE$ ${}^{2}(1+E).$

$G_{6}$ can be estimated similarly and the result is

$G_{6}\leq\delta_{3}||(S_{1}^{d},S_{2}^{d})||_{s}^{2}+C\epsilon^{2}(1+E^{2})E^{2}.$

Note that when estimating $G_{6}$, the following two essential cancelation
relations are used

$\int\nabla^{\alpha}(\text{div}S_{1}^{d})\nabla^{\alpha}u^{d}dx=-\int\frac{1}{2}\nabla^{\alpha}\left(\nabla
u^{d}+(\nabla
u^{d})^{\top}-\frac{2}{3}\text{div}u^{d}I_{3}\right)\nabla^{\alpha}S_{1}^{d}dx$

and

$\int\nabla^{\alpha}(\nabla
S_{2}^{d})\nabla^{\alpha}u^{d}dx=-\int\nabla^{\alpha}S_{2}^{d}\nabla^{\alpha}(\text{div}u^{d})dx.$

These two relations can be easily shown to hold by doing partial integration
and using the fact that $S^{d}_{1}$ is a symmetric and traceless matrix. As
for $N_{i}(i=1,2,3,4,5,6)$, noting that $N_{4}\sim N_{6}$ are similar to
$N_{1}\sim N_{3}$, it is enough to estimate $N_{1}\sim N_{3}$.

$\displaystyle N_{1}$
$\displaystyle=\int\frac{1+\epsilon\phi}{1+\epsilon\eta}\nabla^{\alpha}\eta^{d}\nabla^{\alpha}(u^{d}\nabla\eta_{\epsilon})dx$
$\displaystyle\leq\left|\left|\frac{1+\epsilon\phi}{1+\epsilon\eta}\right|\right|_{\infty}||\nabla^{\alpha}\eta^{d}||\left(||\nabla\eta_{\epsilon}||_{\infty}||\nabla^{\alpha}u^{d}||+||u^{d}||_{\infty}||\nabla^{\alpha+1}\eta_{\epsilon}||\right)$
$\displaystyle\leq C\epsilon E^{2},$

similarly,

$\displaystyle
N_{2}=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left(\frac{\epsilon\eta^{d}\partial_{t}u_{\epsilon}}{1+\epsilon\eta}\right)dx\leq
C\epsilon E^{2}$

and

$\displaystyle N_{3}$
$\displaystyle=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left[\frac{(1+\epsilon\eta)u-(1+\epsilon\eta_{\epsilon})u_{\epsilon}}{1+\epsilon\eta}\cdot\nabla
u_{\epsilon}\right]dx$
$\displaystyle=\int(1+\epsilon\eta)\nabla^{\alpha}u^{d}\nabla^{\alpha}\left[\frac{u^{d}+\epsilon(\eta^{d}u+\eta_{\epsilon}u^{d})}{1+\epsilon\eta}\cdot\nabla
u_{\epsilon}\right]dx$ $\displaystyle\leq C(E^{2}+(1+\epsilon)E^{3}).$

We estimate $N_{7}$ for $1\leq|\alpha|\leq s$ and $|\alpha|=0$.
Case 1: for $1\leq|\alpha|\leq s$, We have

$\displaystyle N_{7}$
$\displaystyle=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\nabla^{\alpha}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx$
$\displaystyle=-\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha+1}\phi^{d}\nabla^{\alpha-1}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx$
$\displaystyle=-\int\nabla\left[\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\right]\nabla^{\alpha}\phi^{d}\nabla^{\alpha-1}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx$ $\displaystyle=:N_{7}^{1}+N_{7}^{2}$

with

$\displaystyle|N_{7}^{1}|$
$\displaystyle=\left|\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha+1}\phi^{d}\nabla^{\alpha-1}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx\right|$
$\displaystyle\leq\int\delta_{4}(\nabla^{\alpha+1}\phi^{d})^{2}dx+\epsilon^{2}(1+\epsilon\eta)^{2}\left\\{\nabla^{\alpha-1}\left[\frac{(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]\right\\}^{2}dx$
$\displaystyle\leq\delta_{4}\int(\nabla^{\alpha+1}\phi^{d})^{2}dx+C\epsilon^{2}(1+E^{4})+\epsilon^{2}E^{2}\left(\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}\right),$

and

$\displaystyle|N_{7}^{2}|$
$\displaystyle=\left|\int\nabla\left[\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\right]\nabla^{\alpha}\phi^{d}\nabla^{\alpha-1}\left[\frac{\epsilon(S_{1}+S_{2}I_{3})\nabla
u}{(1+\epsilon\eta)}\right]dx\right|$
$\displaystyle\leq\epsilon^{2}||\nabla(\eta,\phi)||_{\infty}||\nabla^{\alpha}\phi^{d}||(||(S_{1},S_{2},\eta)||_{\infty}||\nabla^{\alpha}u||+||\nabla
u||_{\infty}||\nabla^{\alpha-1}(S_{1},S_{2},\eta)||)$ $\displaystyle\leq
C\epsilon^{2}(1+E)^{2}E\left(1+E+\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}E+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}E\right)$
$\displaystyle\leq
C\epsilon^{2}(1+E^{4})+C\epsilon^{2}E^{2}(1+E^{2})\left(\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}\right).$

Case 2: for $|\alpha|=0$, It holds that

$\displaystyle N_{7}$
$\displaystyle=\int\frac{\epsilon}{(\gamma-1)(1+\epsilon\phi)}\phi^{d}(S_{1}+S_{2}I_{3})\nabla
udx$ $\displaystyle\leq\epsilon||\phi^{d}||||(S_{1}+S_{2}I_{3})\nabla u||$
$\displaystyle\leq C\epsilon
E(1+E)\left(1+\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}E+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}E\right)$
$\displaystyle\leq
CE^{2}(1+E^{2})+C\epsilon^{2}+\epsilon^{2}E^{2}\left(\frac{1}{\sqrt{\tau_{1}^{\epsilon}}}+\frac{1}{\sqrt{\tau_{2}^{\epsilon}}}\right).$

Therefore, we obtain

$N_{7}\leq\delta_{4}\int(\nabla^{\alpha+1}\phi^{d})^{2}dx+CE^{2}(1+E^{2})\left(1+\frac{\epsilon^{2}}{\sqrt{\tau_{1}^{\epsilon}}}+\frac{\epsilon^{2}}{\sqrt{\tau_{2}^{\epsilon}}}\right)+C\epsilon^{2}.$

While

$\displaystyle D$
$\displaystyle=\int\frac{1+\epsilon\eta}{(\gamma-1)(1+\epsilon\phi)}\nabla^{\alpha}\phi^{d}\left[\nabla^{\alpha}\left(\frac{\kappa^{\epsilon}}{(1+\epsilon\eta)}\triangle\phi^{d}\right)-\frac{\kappa^{\epsilon}}{(1+\epsilon\eta)}\nabla^{\alpha}\triangle\phi^{d}\right]$
$\displaystyle\qquad+\nabla\left[\frac{\kappa^{\epsilon}}{(\gamma-1)(1+\epsilon\phi)}\right]\nabla^{\alpha}\phi^{d}\nabla^{\alpha+1}\phi^{d}dx.$
$\displaystyle\leq||\nabla^{\alpha}\phi^{d}||(||\triangle\phi^{d}||_{\infty}||\nabla^{\alpha}\eta||+\epsilon||\nabla(\phi,\eta)||_{\infty}||\nabla^{\alpha+1}\phi^{d}||)$
$\displaystyle\leq CE^{2}(1+E)+C\epsilon E(1+E)||\nabla^{\alpha+1}\phi^{d}||$
$\displaystyle\leq
CE^{2}(1+E)+C\epsilon^{2}E^{2}(1+E)^{2}+\delta_{5}||\nabla^{\alpha+1}\phi^{d}||^{2}.$

We now estimate the last three terms $T_{1}\sim T_{3}$.

$\displaystyle|T_{i}|$ $\displaystyle\leq
C\epsilon||(\eta_{t},\phi_{t})||_{\infty}||\nabla^{\alpha}(\eta^{d},u^{d},\phi^{d})||^{2}\leq
C\epsilon||(\eta_{t},\phi_{t})||_{\infty}E^{2}$ $\displaystyle\leq
C\epsilon(1+||(\eta^{d}_{t},\phi^{d}_{t})||_{\infty})E^{2}\leq
C(1+E^{2})E^{2}.$

Based on the above estimates, we can take
$\delta_{1}+\delta_{3},\delta_{2}+\delta_{3}$ and $\delta_{4}+\delta_{5}$
suitably small such that
$(\delta_{1}+\delta_{3})||\nabla^{\alpha}S_{1}^{d}||^{2},(\delta_{2}+\delta_{3})||\nabla^{\alpha}S_{2}^{d}||^{2}$
and $(\delta_{4}+\delta_{5})||\nabla^{\alpha+1}\phi^{d}||^{2}$ can be absorbed
by the integrals of
$(\nabla^{\alpha}S_{1}^{d})^{2},(\nabla^{\alpha}S_{2}^{d})^{2}$ and
$(\nabla^{\alpha+1}\phi^{d})^{2}$ on the left side. Therefore, we have the
fact

$\frac{dE^{2}}{dt}\leq C(1+E^{2})E^{2}+C\epsilon^{2}.$

The proof is accomplished. ∎

Now we turn to prove Theorem 1.1. We integrate the inequality in Lemma 3.3
over (0, t) with $t\leq min\\{T_{\epsilon},T_{*}\\}$ to obtain

$E^{2}\leq
CE(0)^{2}+C\int_{0}^{t}(1+E^{2})E^{2}dt+C\int_{0}^{t}\epsilon^{2}dt.$

With the help of the condition in Theorem 1.1 and Gronwall’s lemma, we
conclude that

$E^{2}\leq
C\epsilon^{2}\text{exp}\left\\{C\int_{0}^{t}(1+E^{2})dt\right\\}\equiv\Phi(t).$

Moreover, it is easy to know

$\Phi^{\prime}(t)\leq C\Phi(t)(1+E^{2})=C\left[\Phi(t)+\Phi^{2}(t)\right].$

By employing the nonlinear Gronwall-type inequality, we conclude that there
exists a constant $K$ such that

$E\leq K\epsilon$

for all $t\in(0,\min\\{T_{\epsilon},T_{*}\\})$, provided that
$\Phi(0)=C\epsilon^{2}$ is suitably small.

## References

  * [1] Maxwell, J.C.: On the dynamical theory of gases, Phil. Trans. R. Soc. Lond. $\mathbf{157}$, 49-88 (1867)
  * [2] Yong, W.A.: Singular Perturbations of First-Order Hyperbolic Systems with Stiff Source Terms. J. Differential Equations. $\mathbf{155}$(1), 89-132 (1999)
  * [3] Yong, W.A.: Newtonian limit of Maxwell fluid flows, Arch. Ration. Mech. Anal. $\mathbf{214}$(3), 913-922 (2014)
  * [4] Chakraborty, D., Sader, J.E.: Constitutive models for linear compressible viscoelastic flows of simple liquids at nanometer length scales, Physics of Fluids $\mathbf{27}$, 052002 (2015)
  * [5] Hu, Y., Racke, R.: Compressible Navier-Stokes Equations with Revised Maxwells Law. J. Math. Fluid Mech. $\mathbf{19}$(1), 77-90 (2017)
  * [6] Racke, R., Saal, J.: Hyperbolic Navier-Stokes equations II: Global existence of small solutions. Evol. Equ. Control Theory. $\mathbf{1}$, 217-234 (2012)
  * [7] Racke, R.: Lectures on Nonlinear Evolution Equations. Vieweg Verlag, 1992. Theory. $\mathbf{1}$, 217-234 (2012)
  * [8] Klaierman, S., Majda, A.: singular perturbations of quasilinear hyperbolic systems with large parameters and the incompressible limit of compressible fluids. Comm. Pure Appl. Math. $\mathbf{34}$, 481-524 (1981)
  * [9] Hoff, D.: The Zero-Mach Limit of Compressible Flows. Comm. Math. Phys. $\mathbf{192}$(3), 543-554 (1998)
  * [10] Danchin, R.: Zero Mach Number Limit for Compressible Flows with Periodic Boundary Conditions. Amer. J. Math. $\mathbf{124}$(6), 1153-1219 (2002)
  * [11] Lions, P.L., Masmoudi, N.: Incompressible limit for a viscous compressible fluid. J. Math. Pures Appl. $\mathbf{77}$(6), 585-627 (1998)
  * [12] Desjardins, B., Grenier, E., Lions, P.L.: N. Masmoudi, Incompressible Limit for Solutions of the Isentropic Navier-Stokes Equations with Dirichlet Boundary Conditions. J. Math. Pures Appl. $\mathbf{78}$(5), 461-471 (1999)
  * [13] Alazard, T.: Low Mach number limit of the full Navier-Stokes equations. Arch. Ration. Mech. Anal. $\mathbf{180}$(1), 1-73 (2006)
  * [14] Guo, B., Zeng, L., Ni, G.: Low Mach number limit of strong solutions to 3-D full Navier-Stokes equations with Dirichlet boundary condition. Adv. Math. (China) $\mathbf{48}$(6), 667-691 (2019)
  * [15] Ou, Y.: Low Mach number limit for the non-isentropic Navier-Stokes equations. J. Differential Equations. $\mathbf{246}$(11), 4441-4465 (2009)
  * [16] Feireisl, E., Novotný, A.: The low Mach number limit for the full Navier-Stokes-Fourier system. Arch. Ration. Mech. Anal. $\mathbf{186}$(1), 77-107 (2007)
  * [17] Feireisl, E., Novotný, A.: H. Petzeltova, On the incompressible limit for the Navier-Stokes-Fourier system in domains with wavy bottoms. Math. Models Methods Appl. Sci. $\mathbf{18}$(2), 291-324 (2008)
  * [18] Jiang, S., Ou, Y.: Incompressible limit of the non-isentropic Navier-Stokes equations with well-prepared initial data in three-dimensional bounded domains. J. Math. Pures Appl. $\mathbf{96}$(1), 1-28 (2011)
  * [19] Dou, C., Jiang, S., Ou, Y.: Low Mach number limit of full Navier-Stokes equations in a 3D bounded domain. J. Differential Equations. $\mathbf{258}$(2), 379-398 (2015)
  * [20] Zhang, S.: Low Mach number limit for the full compressible Navier-Stokes equations with Cattaneo’s heat transfer law. Nonlinear Anal. $\mathbf{184}$, 83-94 (2019)
  * [21] Li, F., Mu, Y.: Low Mach number limit of the full compressible Navier-Stokes-Maxwell system. J. Math. Anal. Appl. $\mathbf{412}$(1), 334-344 (2014)
  * [22] Sha, K., Li, Y.: Low Mach number limit of the three-dimensional full compressible Navier-Stokes-Korteweg equations. Z. Angew. Math. Phys. $\mathbf{70}$, 169 (2019) https://doi.org/10.1007/s00033-019-1215-y.
  * [23] Maisano, G. et. al.: Evidence of anomalous acoustic behavior from Brillouin scattering in supercooled water. Phys. Rev. Lett. $\mathbf{52}$, 1025 (1984)
  * [24] Pelton, M. et. al.: Viscoelastic flows in simple liquids generated by vibrating nanostructures. Phys. Rev. Lett. $\mathbf{111}$, 244-502 (2013)
  * [25] Sette, F. et. al.: Collective dynamics in water by high energy resolution inelastic X-Ray scattering. Phys. Rev. Lett. $\mathbf{75}$, 850 (1995)


# RAEE: A Training-Free Retrieval-Augmented Early Exiting Framework for
Efficient Inference

Lianming Huang∗, Shangyu Wu
City University of Hong Kong
&Yufei Cui
MILA, McGill University
&Ying Xiong
MBZUAI
&Xue Liu
MILA, McGill University
&Tei-Wei Kuo
National Taiwan University
&Nan Guan
City University of Hong Kong &Chun Jason Xue
MBZUAI
Authors contributed equally to this research.

###### Abstract

Deploying large language model inference remains challenging due to their high
computational overhead. Early exiting accelerates model inference by
adaptively reducing the number of inference layers. Existing methods require
training internal classifiers to determine whether to exit at each
intermediate layer. However, such classifier-based early exiting frameworks
require significant effort to design and train the classifiers. To address
these limitations, this paper proposes RAEE, a training-free Retrieval-
Augmented Early Exiting framework for efficient inference. First, this paper
demonstrates that the early exiting problem can be modeled as a distribution
prediction problem, where the distribution is approximated using similar
data’s existing information. Next, the paper details the process of collecting
existing information to build the retrieval database. Finally, based on the
pre-built retrieval database, RAEE leverages the retrieved similar data’s
exiting information to guide the backbone model to exit at the layer, which is
predicted by the approximated distribution. Experimental results demonstrate
that the proposed RAEE can significantly accelerate inference. RAEE also
achieves state-of-the-art zero-shot performance on 8 classification tasks.

##  1 Introduction

Large language models have been widely used in various application scenarios
due to their excellent performance (Thoppilan et al., 2022; Touvron et al.,
2023; Scao et al., 2022). However, deploying large language models on
resource-constrained devices is still challenging due to the high
computational overheads of performing model inference (Dao et al., 2022; Liu
et al., 2023). Model pruning, as an advanced technique, provides a new
direction for efficient inference (Valicenti et al., 2023; Ma et al., 2023).
It selectively removes less important weights or connections from the neural
network to reduce complexity and computational requirements without
significantly degrading performance. One popular model pruning method is the
early exiting technique, which speeds up inference by adaptively reducing the
number of inference layers.

Most early exiting frameworks (Liu et al., 2020; Zhu, 2021; Fan et al., 2024)
leverage classifiers to predict the exiting layer and then stop the inference
at the predicted exiting layer. Those classifier-based early exiting
frameworks can be categorized into three types according to the training
strategies. One type is the training-based early exiting (Zhu, 2021; Zhou et
al., 2020; Zhu et al., 2023), which requires training the classifiers along
with the inference model. These methods introduce significant training and
fine-tuning overheads, particularly when applied to large language models.
Another branch of classifier-based early exiting can be concluded as semi-
training-based early exiting (Fan et al., 2024). The backbone models in this
type of early exiting framework would not be updated, and only classifiers
would be fitted to predict the exiting layer. These methods may not capture
the patterns between inputs and exiting layers well, requiring significant
human effort in feature engineering. The last one is training-free early
exiting (Sun et al., 2022), which requires no parameter updates and uses
heuristics to determine the exiting layer. These methods lack the
generalization ability in predicting the exiting layer and often fail to
achieve optimal or sub-optimal solutions.

To address the above limitations, this paper first shows that the exiting
layer predictions can be solved by predicting from an exiting distribution.
Then, this paper presents the observations that similar data’s exiting
information can be used to approximate the exiting distribution. Based on
these observations, this paper proposes the RAEE, a training-free retrieval-
augmented early exiting framework for efficient inference. RAEE collects
exiting information from training data and builds the indexing to retrieve
similar data’s exiting information. During the inference, RAEE predicts the
exiting layer based on the top-k nearest neighbors’ exiting information and
stops the model forwarding at the predicted exiting layer.

We conduct comprehensive experiments to evaluate the proposed RAEE and various
comparison methods on 8 classification tasks. Experimental results demonstrate
that RAEE can accelerate the model inference while achieving state-of-the-art
performance. Codes are available at
111https://anonymous.4open.science/r/RAEE-D724.

The main contributions of this paper are:

  * •

Models the early exiting problem as a distribution prediction problem;

  * •

Shows that the exiting distribution can be approximated by similar data’s
exiting information;

  * •

Proposes a training-free retrieval-augmented early exiting framework to
accelerate the model inference;

  * •

Experimental results demonstrate that the proposed RAEE can achieve state-of-
the-art performance.

Figure 1: The overview of retrieval-augmented early exiting framework.

##  2 Motivations

In this section, we demonstrate why using retrieval-based techniques is a
simple yet effective way to augment the early exiting framework during the
inference stage.

Problem Statement. Formally, early exiting can be defined as follows: Given a
backbone model $\mathcal{M}$ with $m$ layers and an input $x$, the goal of the
early exiting framework is to design an exiting function or classifier
$l=f(x)$ to determine whether to exit at the layer $l$. The final prediction
$y$ is then transformed from the intermediate output states $h_{l}$ of the
$l$-th layer. And the final prediction probability can be formulated as,

$P(y\;|\;x)=P(y\;|\;h_{f(x)}).$ (1)

, where the exiting function or the exiting classifier $f(x)$ is trained or
built on the downstream tasks’ train data $\mathcal{D}$.

Limitations of Existing Early Exiting. Existing early exiting frameworks can
be categorized into training-based, semi-training-based, and training-free
exiting methods. For training-based exiting, most works jointly optimize the
exiting classifiers and the backbone model (Zhu et al., 2023, 2021; Zhu,
2021). These works fine-tune the pre-trained backbone model with exiting
mechanisms over downstream tasks’ training data and update both the backbone
model and classifier parameters. Although these approaches could adapt the
model with the early exiting framework and accelerate the inference while
maintaining performance, the fine-tuning overhead is still a non-negligible
amount, especially when fine-tuning modern large language models.

Instead of training both classifiers and the backbone model, other works (Fan
et al., 2024) only fit the classifiers based on features extracted from the
backbone model, with parameters in the backbone model fixed. These works can
significantly reduce the training overhead of early exiting framework, as they
aim to address the early exiting problem using efficient traditional machine
learning approaches for classifier fitting, e.g., techniques for fitting
support vector machine. However, such methods that only train classifiers may
not effectively capture the patterns between inputs and exiting layers. The
reasons may lie in that 1) the classifiers only focus on distinguishing the
extract features at each layer, rather than recognize the patterns between
input embeddings and exiting layers; 2) the feature extraction requires human
effort to design, where classifiers only indirectly learn where to exit.

Training-free early exiting frameworks refer to computing the exiting layer
with heuristics, e.g., hashing functions (Sun et al., 2022). These techniques
basically introduce no or few training overhead and are easy to set up.
Although these works can significantly accelerate the building process as well
as the classifying process, they lack generalization in exiting layer
predictions. The predictions made by these heuristics may also be sensitive to
different inputs, leading to unstable performance of the early exiting
framework.

---

Figure 2: The probability of different exiting layers on SST-2 test data and
corresponding top-8 nearest neighbor SST-2 training data.

Motivations of Retrieval-Augmented Early Exiting. To address the above
limitations, this paper aims to leverage retrieval-based techniques to guide
the early exiting, which requires no parameters to update and has a
generalization capability. Existing retrieval databases basically use
clustering and product quantization to build the retrieval indexing over
millions of embeddings, which can perform efficient approximate nearest
neighbor searching. The building process of the indexing is not resource-
constrained, which can run on either GPUs or CPUs. Besides, since the
retrieval database stores the original data, every retrieval in the retrieval-
augmented early exiting can be regarded as a generalization over several
semantically similar data, while those original data may be used to train
classifiers or build hashing functions in other early exiting frameworks. The
retrieval-augmented techniques also exhibit strong adaptability to new data.
By simply adding new data to the retrieval database, the index can retrieve
up-to-date information.

To further demonstrate the efficacy of retrieval-augmented early exiting, this
paper conducts some analysis experiments in Figure 2 to show the probability
of different exiting layers on two SST-2 test data and corresponding top-8
nearest neighbor SST-2 training data. The probability in Figure 2 is
calculated as the normalized logits of the answer label token. Note that it is
impossible to obtain the probability of each exiting layer during the real
inference as no labels are provided. The blue line with dots in Figure 2 shows
the probability of correct predictions exiting at each layer. For example, in
Figure 2 (a), when exiting at layer 18, the probability of correct predictions
is about 0.68, while exiting at layer 9 only has a probability of 0.01. The
box plot in Figure 2 presents the probability statistics of the top-8 nearest
neighbor training data, indicating the probability of exiting at each layer.
For instance, when exiting at layer 18, the maximum exit probability is
approximately 0.81, the minimum is around 0.39, and the box plot indicates
that half of the neighbors exhibit a probability larger than 0.54.

The blue line in Figure 2 represents the target distribution for the
classifier to learn. The experimental results in Figure 2 (a) and (b) reveal
that the exiting probabilities exhibit a similar pattern to those of the
nearest neighbors. Additionally, the exiting layer varies across different
inputs. Consequently, the experimental results lead to two key conclusions:

  1. 1.

The exiting probability can be approximated by the probability of the top-k
nearest neighbors.

  2. 2.

Different inputs exhibit distinct probabilities for exiting layers.

These observations motivate us to propose a retrieval-augmented early exiting
framework.

Algorithm 1 Collect the keys and values for building the retrieval database.

0: Training data
$\mathcal{D}=\\{(x^{train}_{1},y^{train}_{1}),\ldots,(x^{train}_{|\mathcal{D}|},y^{train}_{|\mathcal{D}|})\\}$,
backbone model $\mathcal{M}$ with $m$ layers
$\\{\mathcal{L}_{1},\ldots,\mathcal{L}_{m}\\}$, encoder $\mathcal{E}$ (None
value means no encoder is provided).

0: Keys $\mathcal{K}$ and values $\mathcal{V}$.

1: $\mathcal{K}=[],\mathcal{V}=[]$

2: for $i=1,\ldots,|\mathcal{D}|$ do

3: $v_{i}=[]$;

4: $h_{0}=\mathcal{M}_{emb}(x^{train}_{i})$;

5: for j=1, …, m do

6: $h_{j}=\mathcal{L}_{j}(h_{j-1})$; /* Compute the intermediate outputs of
the layer $j$ */

7: $logits=\mathcal{M}_{lm\\_head}(h_{j})$; /* Predict from the layer $j$ */

8: $\hat{y}=\arg\max logits$;

9: $p^{j}_{i}=\max logits$;

10: if $\hat{y}$ is equal to $y^{train}_{i}$ then

11: Add $(j,p^{j}_{i})$ into $v_{i}$; /* Store the possible exiting layer */

12: end if

13: end for

14: Add $v_{i}$ into $\mathcal{V}$;

15: if $\mathcal{E}$ is None then

16: Add $h_{0}$ into $\mathcal{K}$; /* Store the embeddings of backbone model
when no encoder model */

17: end if

18: end for

19: if $\mathcal{E}$ is not None then

20: Add all $\mathcal{E}(x^{train}_{i})$ into $\mathcal{K}$; /* Store the
embeddings of encoder */

21: end if

22: return $\mathcal{K},\mathcal{V}$;

##  3 Methodology

In this section, this paper describes the proposed retrieval-augmented early
exiting framework in detail. First, this paper introduces the details of how
to build the retrieval database for the early exiting. Then, this paper
presents the retrieval-augmented early exiting baseline called RAEE.

###  3.1 Building the Retrieval Database for Early Exiting

The critical factor in constructing the retrieval database is selecting
appropriate keys and values. To avoid introducing too much retrieving
overheads, this paper only retrieves once at the beginning of the backbone
model. Therefore, as shown in the top part of Figure 1, the keys $\mathcal{K}$
are input embeddings of training data, which can be obtained from an extra
encoder model $\mathcal{E}$ , such as BERT (Devlin et al., 2019), or the
outputs of embedding layers in the backbone model $\mathcal{M}_{emb}$,

$\mathcal{K}=\\{e_{i}\\}^{|\mathcal{D}|}_{i=1}=\\{\mathcal{E}(x^{train}_{i})\\}^{|\mathcal{D}|}_{i=1}.$
(2)

For the values, this paper collects a set of possible exiting layers $l_{i}$
and corresponding probabilities $p_{i}$ for each embedding $e_{i}$, i.e.,
$v_{i}=\\{(l^{j}_{i},p^{j}_{i})\\}^{m_{i}}_{j=1}$, where $m_{i}$ indicates the
number of possible exiting layers for the embedding $e_{i}$. The layer $l$
chosen as the exiting layer is determined by whether the outputs of this layer
$h_{l}$ can be used to make the right predictions $\hat{y}$ compared to the
training labels $y^{train}$. Then, the values $\mathcal{V}$ are all sets of
possible exiting layers,

$\mathcal{V}=\\{v_{i}\\}^{|D|}_{i=1}=\left\\{\\{(l^{j}_{i},p^{j}_{i})\\}^{m_{i}}_{j=1}\right\\}^{|D|}_{i=1}.$
(3)

Algorithm 1 shows the detailed steps of preparing keys and values. The
collecting process requires no parameters to update, only model inference is
performed. After collecting keys and values for the retrieval databases, this
paper uses state-of-the-art approximate nearest neighbor search indexing ,
such as FAISS (Johnson et al., 2019), and efficient key-value stores to build
the retrieval database.

###  3.2 RAEE

Algorithm 1 Model inference with the synchronized retrieval-augmented early
exiting.

0: Input $x$, backbone model $\mathcal{M}$ with $m$ layers
$\\{\mathcal{L}_{1},\ldots,\mathcal{L}_{m}\\}$, encoder $\mathcal{E}$,
indexing $\mathcal{I}$, top-$k$, the exiting layer determination function
$f(\cdot)$.

0: Final prediction $\hat{y}$.

1: $h_{0}=\mathcal{M}_{emb}(x)$;

2: if $\mathcal{E}$ is not None then

3: $e_{query}=\mathcal{E}(x)$; /* Encode the inputs when no encoder */

4: else

5: $e_{query}=h_{0}$; /* Use the embeddings of backbone model */

6: end if

7: $\\{(v_{i},dis_{i})\\}^{k}_{i=1}=\mathcal{I}(e_{query},k)$; /* Retrieve the
possible exiting layers */

8: $l=f\left(\\{(v_{1},dis_{1}),\ldots,(v_{k},dis_{k})\\}\right)$; /* Obtain
the exiting layer */

9: for $i=1,\ldots,l$ do

10: $h_{i}=\mathcal{L}_{i}(h_{i-1})$; /* Perform model inference with early
exiting */

11: end for

12: $logits=\mathcal{M}_{lm\\_head}(h_{l})$; /* Predict based on the layer
$h_{l}$ outputs */

13: $\hat{y}=\arg\max logits$;

14: return $\hat{y}$;

In this section, this paper then presents a retrieval-augmented early exiting
framework named RAEE to accelerate the model inference. RAEE regards the
exiting layer as a random variable $z$, taking values in the set of
$\\{1,\ldots,m\\}$, where $m$ is the total number of layers in the backbone
model $\mathcal{M}$. The probability mass function $P(z=l)$ represents the
probability of the case that the backbone model exits at the layer $l$. With
the gold label, we can observe that the random variable $Z$ follows an unknown
discrete distribution $F$. Then, this paper shows how to leverage the
retrieval database to approximate the distribution $F$.

Given an input $x$, RAEE first retrieves top-$k$ nearest neighbors
$\\{v_{1},\ldots,v_{k}\\}$, where each neighbor $v_{i}$ has $m_{i}$ possible
exiting layers. Naturally, we can approximate the distribution $F$ by
estimating the probability function $P(z=l)$,

$P(z=l\;|\;x)=\sum^{k}_{i=1}P(v_{i}\;|\;x)\cdot\mathbbm{1}\left(any(l^{j}_{i}=l\;\&\&\;p^{j}_{i}\geq\tau)\right)\cdot
p^{j}_{i}$ (4)

, where $\mathbbm{1}$ is the indicator function that returns 1 if the
condition is true and 0 otherwise, $any(\cdot)$ is the function that returns
true if one condition is true and false otherwise, $\tau$ is the threshold for
filtering the layers with extremely low probability. Since different neighbors
should have different contributions to the probability function $P(z=l)$, RAEE
uses the reciprocal of the scaled distance between each neighbor and the query
to estimate the contribution,

$P(v_{i}\;|\;x)=\frac{\min\left(\\{distance(v_{j},x)\\}^{k}_{j=1}\right)}{distance(v_{i},x)}$
(5)

Then, RAEE designs a function $f(x)$ to determine the exiting layer, which
selects the layer that maximizes the probability function $P(z=l)$,

$f(x)=\arg\max_{l}P(z=l\;|\;x)$ (6)

The bottom part of Figure 1 shows the inference workflow of RAEE.
Specifically, RAEE first simultaneously feeds the inputs into both the
backbone model for the label predictions and the same encoder used in the
building process for the query embeddings. Then, the retriever in RAEE
retrieves the top-$k$ nearest neighbors in the retrieval databases based on
the query embeddings. After obtaining all possible exiting layers of $k$
nearest neighbors, RAEE computes the exiting layers based on the Equations
4-6. Finally, RAEE stops the forwarding at the calculated exiting layer, and
passes the intermediate outputs of the exiting layer to the final prediction
layer , e.g., LM Head in language models, to obtain the final predictions
(Equation 1).

Algorithm 1 shows the detailed steps of the synchronized version of RAEE. For
more efficient implementations, the inference process (Line 9-11) and the
retrieving process (Line 2-8) can be executed in parallel. Specifically, we
can use two threads to execute each process. During the inference process with
asynchronized RAEE, the calculated intermediate results would be collected for
early exiting. There are two cases of executions considering the speed of each
process, 1) if the retrieving process ends before the calculation of the
exiting layer, the inference process will continue forwarding until the
exiting layer; 2) if the inference process has computed the results of the
exiting layer when the retrieving process ends, the inference process will
stop immediately and return the stored intermediate results of the exiting
layer. Therefore, the overall running time is the maximal time between the
inference process and the retrieving process.

Table 1: Zero-shot results on 8 classification tasks. For each method, the results in the upper row are model performance, and the results in the lower row are inference latency (ms). The results in bold are the best results on the task. ‘D-BERT’ and ‘D-RoBERTa’ refer to DistilBERT and DistilRoBERTa, respectively. Methods | SST-2 | SST-5 | MR | CR | MPQA | SUBJ | TREC | CoLA | Avg
---|---|---|---|---|---|---|---|---|---
Pretrained Models |  |  |  |  |  |  |  |
LM-BFF | 83.60 | 34.98 | 80.80 | 79.55 | 67.60 | 51.45 | 32.40 | 2.03 | 54.05
51.61 | 52.60 | 52.53 | 52.22 | 52.12 | 52.22 | 52.25 | 51.82 | 52.17
ElasticBERT | 51.15 | 27.78 | 50.10 | 50.00 | 47.05 | 50.00 | 18.60 | 6.02 | 37.59
46.30 | 46.59 | 46.76 | 46.97 | 46.92 | 46.98 | 47.74 | 46.60 | 46.86
Static Models |  |  |  |  |  |  |  |
D-BERT | 71.90 | 25.61 | 67.65 | 77.10 | 64.85 | 70.35 | 28.40 | -0.70 | 50.65
9.39 | 9.57 | 9.16 | 9.17 | 8.96 | 9.29 | 9.78 | 9.27 | 9.32
D-RoBERTa | 81.77 | 28.37 | 77.95 | 84.20 | 69.50 | 54.35 | 32.00 | -1.18 | 53.37
9.50 | 9.13 | 9.21 | 9.17 | 9.07 | 9.30 | 9.91 | 9.35 | 9.33
Dynamic Models |  |  |  |  |  |  |  |
HashEE | 52.75 | 18.64 | 48.95 | 52.50 | 55.85 | 45.35 | 23.80 | -3.86 | 36.75
354.64 | 345.55 | 348.19 | 354.84 | 337.68 | 341.46 | 339.94 | 339.88 | 345.27
AdaInfer | 50.92 | 34.98 | 50.45 | 49.60 | 60.90 | 50.65 | 32.40 | -1.62 | 41.04
96.17 | 5,400.36 | 0.50 | 13.98 | 3,726.39 | 22.88 | 444.80 | 582.58 | 1,285.96
RAEE | 84.06 | 33.44 | 81.75 | 67.90 | 79.25 | 83.60 | 60.60 | 15.55 | 63.27
43.53 | 38.14 | 44.33 | 36.63 | 40.64 | 33.87 | 36.87 | 30.94 | 38.12

##  4 Experiments

In this section, this paper first introduces the dataset and the experimental
setting. Then, this paper presents the main results on 8 classification tasks.
Besides, this paper also conducts the ablation study and analysis of RAEE to
show the impact factors on model performance.

###  4.1 Dataset and Experimental Setup

Datasets We conduct comprehensive experiments across 8 classification tasks
from GLUE benchmark(Wang et al., 2019). These tasks cover sentiment analysis,
opinion polarity analysis, grammatical judgment, natural language inference,
paraphrasing, etc.

Experimental Settings The proposed RAEE was implemented using PyTorch
framework and Transformer, utilizing the computational power of one NVIDIA
Quadro RTX5000 GPU with 16GB GPU memory and an 8-core Intel CPU with 46GB main
memory. The experiments were conducted in the zero-shot setting. The
evaluation metric is accuracy, except for CoLA, which is measured by the
Matthew correlation coefficient. The backbone model of RAEE is based on
RoBERTa-large. The number of retrieved nearest neighbors is set to 12 in this
paper.

To validate the effectiveness, we compared RAEE with three types of methods
without any fine-tuning. Pretrained Models: 1) LM-BFF (Gao et al., 2021), a
prompt-based approach using RoBERTa-large; 2) ElasticBERT (Liu et al., 2022),
a pre-trained multi-exit transformer model, where the large version is used in
this paper. Static Models: 1) DistilBERT (Sanh et al., 2019), a distilled
version of BERT-base model; 2) DistilRoBERTa (Sanh et al., 2019), a distilled
version of the RoBERTa-base model. Dynamic Models: 1) HashEE (Sun et al.,
2022), a hash-based early exiting approach whose backbone model is
ElasticBERT-large; 2) AdaInfer (Fan et al., 2024), an SVM-based early exiting
method with our reproduced version on RoBERTa-large. The max sequence length
of all models is set to 512. The templates are listed in the Appendix.

Table 2: The impact of retrieval number $k$ on the distribution approximation. $k$ | SST-2 | SST-5 | MR | CR | MPQA | SUBJ | TREC | CoLA | Avg
---|---|---|---|---|---|---|---|---|---
2 | 78.21 | 32.94 | 77.35 | 68.50 | 78.25 | 80.05 | 60.20 | 14.26 | 61.22
4 | 81.08 | 33.48 | 79.40 | 67.90 | 79.45 | 82.95 | 61.20 | 14.78 | 62.53
8 | 83.60 | 34.12 | 81.00 | 68.05 | 79.75 | 83.65 | 61.20 | 12.28 | 62.96
12 | 84.06 | 33.44 | 81.75 | 67.90 | 79.25 | 83.60 | 60.60 | 15.55 | 63.27
16 | 83.72 | 32.71 | 82.05 | 69.10 | 78.70 | 83.35 | 61.20 | 15.24 | 63.26
20 | 83.14 | 33.53 | 81.85 | 69.70 | 78.45 | 83.05 | 61.00 | 11.94 | 62.83

###  4.2 Main Results

Table 1 presents the main results, comparing the performance and latency of
the RAEE model against three other model types across eight classification
tasks. For fair comparisons, this paper only compares semi-training-based
early exiting such as AdaInfer (Fan et al., 2024) and training-free early
exiting such as HashEE (Sun et al., 2022). For the model performance, the
proposed RAEE can achieve state-of-the-art zero-shot performance on average.
Compared to the backbone model such as LM-BFF (Gao et al., 2021) without early
exiting, RAEE can outperform it on most tasks. Besides, RAEE can also
significantly outperform existing early exiting frameworks on all tasks. The
above results imply that the proposed RAEE can improve the model’s robustness.

For the inference latency, the proposed RAEE can achieve a faster model
inference than that of the comparable-scale pre-trained models and existing
dynamic models. The inference latency of existing dynamic models is much
longer than others. The reasons lie in that the classifiers in these early
exiting frameworks have a poor performance in zero-shot scenarios, requiring
to more advanced optimizations, such as joint tuning with the backbone model
such as HashEE (Sun et al., 2022) or concatenating more examples for in-
context learning such as AdaInfer (Fan et al., 2024). Compared to static
models, RAEE is much slower. This is because those static models are
architecturally smaller than the backbone model used in RAEE. For example, the
number of layers for DistilRoBERTa is only 6 layers, while the backbone model
used in RAEE has 24 layers. However, obtaining those static models requires
large computational resources to fine-tune from a pre trained model, while the
proposed RAEE is a training-free technique.

###  4.3 Ablation Study of Top-$k$

Table 2 illustrates the impact of varying the number of retrievals on the
distribution approximation. As $k$ increases, the proposed RAEE improves the
overall performance from 61.22 to 63.27. This suggests that more retrieved
exiting information can help enhance the approximation performance. However,
when $k$ beyond 12, the overall performance degrades from 63.27 to 62.83. The
reasons may lie in that providing the exiting information of the retrievals
that are not quite related to the query would introduce noise, thus misleading
the final predictions. This also implies that only a few exiting information
is enough to approximate the exiting distribution, and the time cost of the
retrieving process can be saved.

###  4.4 Analysis on the Retrieval Database Size

Table 3: The impact of retrieval database size on the distribution approximation. The percentage refers to the amount of training data that is used to build the retrieval database. Database Size | SST-2 | SST-5 | MR | CR | MPQA | SUBJ | TREC | CoLA | Avg
---|---|---|---|---|---|---|---|---|---
20% | 83.14 | 32.35 | 81.40 | 67.80 | 75.55 | 77.85 | 57.20 | 11.47 | 60.85
50% | 82.45 | 32.31 | 81.30 | 65.35 | 76.65 | 82.65 | 58.20 | 16.83 | 61.97
100% | 84.06 | 33.44 | 81.75 | 67.90 | 79.25 | 83.60 | 60.60 | 15.55 | 63.27

Table 3 shows the performance of RAEE with different sizes of retrieval
databases. The size of the retrieval databases implies how similar embeddings
would be retrieved, thus impacting the confidence of the provided exiting
information. As the database size increases, the performance of RAEE increases
significantly from 60.85 to 63.27 on average. This demonstrates that
collecting more data can improve the generalization of RAEE, thus
approximating the exiting distribution more accurately.

###  4.5 Latency Breakdown of RAEE

Table 4: The latency (ms) of each process in RAEE. Process | SST-2 | SST-5 | MR | CR | MPQA | SUBJ | TREC | CoLA | Avg
---|---|---|---|---|---|---|---|---|---
Encoding | 3.30 | 3.07 | 3.21 | 2.92 | 2.86 | 3.07 | 3.42 | 3.23 | 3.14
Retrieving | 0.07 | 0.08 | 0.08 | 0.06 | 0.07 | 0.07 | 0.07 | 0.08 | 0.07
Forwarding | 40.15 | 34.99 | 41.04 | 33.64 | 37.71 | 30.73 | 33.37 | 27.63 | 34.91

Table 4 shows the latency of each process in RAEE. From the experimental
results, the time cost of encoding and retrieving accounts for only a small
part of the overall inference time. Compared to the inference latency of the
backbone model without early exiting, such as LM-BFF (Gao et al., 2021), in
Table 1, the time cost of model forwarding is reduced by about 17.26 ms on
average. Additionally, the inference latency can be further improved by
asynchronously executing the encoding and retrieving processes, thereby hiding
the overhead within the model inference process. The time cost of encoding can
also be reduced by using smaller encoders.

###  4.6 Exiting Layers

Table 5 shows the average exiting layers of the proposed RAEE and AdaInfer
across various tasks. RAEE’s average exiting layer is 15.98, notably lower
than the backbone model’s total layers 24. This result also aligns with the
expectations in the motivation example. This suggests that RAEE can accurately
approximate the gold exiting layer distribution by using the retrieval
database. Although AdaInfer exits earlier than RAEE, it exhibits quite poor
performance, as shown in Table 1. The reason may be that during the zero-shot
inference scenario, the collected features can only provide limited
information for the SVM, thus resulting in unstable prediction performance.

Table 5: The exiting layers of RAEE and AdaInfer. Models | SST-2 | SST-5 | MR | CR | MPQA | SUBJ | TREC | CoLA | Avg
---|---|---|---|---|---|---|---|---|---
AdaInfer | 1.00 | 24.00 | 0.00 | 0.47 | 18.00 | 0.08 | 24.00 | 4.00 | 8.94
RAEE | 18.61 | 15.87 | 18.69 | 15.38 | 17.32 | 14.00 | 15.25 | 12.70 | 15.98

##  5 Related Work

###  5.1 Early Exiting Framework

Model inference with early exit has been a popular pruning method to reduce
both computation and memory overhead on text classification or generation
tasks. Most existing works (Bae et al., 2023; Kong et al., 2022; Ji et al.,
2023; Wolczyk et al., 2021; Hooper et al., 2023) introduce classifiers in each
layer to determine whether the inference should continue. (Xin et al., 2020;
Liu et al., 2020; Zhou et al., 2020; He et al., 2021) train the classifier by
minimizing the differences between each layer’s outputs and final outputs,
then perform early-exit-based inference according to a threshold. (Liao et
al., 2021) incorporates the past and future information to predict the early
exit layer. Instead of training a neural network as classifiers, (Fan et al.,
2024) only fit the machine learning classifiers on the extracted features for
the early exiting. (Zhu, 2021; Zhu et al., 2021, 2023; Zhang et al., 2023a)
focus on designing novel loss functions for training a more robust classifier.
(Li et al., 2021) incorporates sentence-level features as well as token-level
features to predict the early exiting. Different from those works, our method
does not require training the classifier. (Sun et al., 2022) proposed a hash-
based early exiting method that uses the hashing functions to map tokens to
exiting layers. Our method predicts exit layers using a pre-built database,
resulting in better generalization. Other works (Jazbec et al., 2023; Li et
al., 2023) focus on designing early exiting frameworks for image
classification tasks. They are optimizations in the different domains compared
to our works.

###  5.2 Retrieval-based Augmentations

Retrieval-based augmentations (Li et al., 2022; Wang et al., 2023; Xiong et
al., 2023; Cui et al., 2023; Wu et al., 2024) have been widely used in various
natural language processing (NLP) tasks and achieved remarkable performance.
Existing works mostly leverage external knowledge databases to augment
generator models on various text-generation tasks, such as language modeling
(Khandelwal et al., 2020; Lewis et al., 2020; Borgeaud et al., 2022),
question-answering (Guu et al., 2020; Izacard & Grave, 2021), machine
translation (Khandelwal et al., 2021; Wang et al., 2022), dialogue system
(Cheng et al., 2023). Those works focus on improving the model’s generation
quality, while our work aims to use the retrieval knowledge to accelerate the
model’s inference. Additionally, other works have improved model generation
efficiency using external retrieval databases. (Zhang et al., 2023b) stores
the generation trajectories of diffusion models and retrieves similar
trajectories to skip several intermediate sampling steps. These works are out
of the scope of the research problems in this paper.

##  6 Limitations

The limitations of RAEE may lie in the following aspects. Building Overheads.
Although building the retrieval database is an offline process that can be
completed in a few hours, there is still a trade-off between the
generalization capability and the building overheads. Analysis experiments
have demonstrated that more data can significantly improve the RAEE’s
generalization capability. Retrieving Quality. Various types of indexing can
be chosen for building the retrieval databases. Different retrievers have
different retrieving qualities. The indexing used in RAEE may not be the
optimal choice across different tasks. Addressing these limitations is not the
main focus of this paper, while future optimizations on those topics can be
combined with the proposed RAEE.

##  7 Conclusion

This paper models the early exiting problem as a distribution approximation
problem and observes that similar data’s exiting information can be used to
approximate. Then, this paper proposes a retrieval-augmented early exiting
framework named RAEE. Experimental results show that RAEE can accelerate the
model inference while improving the model performance. For future work, it is
worthwhile to explore the performance of applying the retrieval-augmented
early exiting framework in fine-tuning scenarios.

## References

  * Bae et al. (2023) Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun.  Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5910–5924, 2023.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 2022 International Conference on Machine Learning (ICML)_ , pp. 2206–2240, 2022.
  * Cheng et al. (2023) Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan.  Lift yourself up: Retrieval-augmented text generation with self-memory.  In _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Cui et al. (2023) Yufei Cui, Ziquan Liu, Yixin Chen, Yuchen Lu, Xinyue Yu, Xue (Steve) Liu, Tei-Wei Kuo, Miguel Rodrigues, Chun Jason Xue, and Antoni B. Chan.  Retrieval-augmented multiple instance learning.  In _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS)_ , 2023.
  * Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), _Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pp. 4171–4186. Association for Computational Linguistics, 2019.
  * Fan et al. (2024) Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang.  Not all layers of llms are necessary during inference.  _CoRR_ , abs/2403.02181, 2024.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)_ , pp. 3816–3830. Association for Computational Linguistics, 2021.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  Retrieval augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning (ICML)_ , pp. 3929–3938, 2020.
  * He et al. (2021) Xuanli He, Iman Keivanloo, Yi Xu, Xiang He, Belinda Zeng, Santosh Rajagopalan, and Trishul Chilimbi.  Magic pyramid: Accelerating inference with early exiting and token pruning.  _CoRR_ , abs/2111.00230, 2021.
  * Hooper et al. (2023) Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Yakun Sophia Shao.  SPEED: speculative pipelined execution for efficient decoding.  _CoRR_ , abs/2310.12072, 2023.
  * Izacard & Grave (2021) Gautier Izacard and Edouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL): Main Volume_ , pp. 874–880, 2021.
  * Jazbec et al. (2023) Metod Jazbec, James Urquhart Allingham, Dan Zhang, and Eric T. Nalisnick.  Towards anytime classification in early-exit architectures by enforcing conditional monotonicity.  In _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Ji et al. (2023) Yixin Ji, Jikai Wang, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang.  Early exit with disentangled representation and equiangular tight frame.  In _Findings of the Association for Computational Linguistics (ACL)_ , pp. 14128–14142, 2023.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  In _Proceedings of the 8th International Conference on Learning Representations (ICLR)_ , 2020.
  * Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Nearest neighbor machine translation.  In _Proceedings of the 9th International Conference on Learning Representations (ICLR)_ , 2021.
  * Kong et al. (2022) Jun Kong, Jin Wang, Liang-Chih Yu, and Xuejie Zhang.  Accelerating inference for pretrained language models by unified multi-perspective early exiting.  In _Proceedings of the 29th International Conference on Computational Linguistics (COLING)_ , pp. 4677–4686, 2022.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Li et al. (2021) Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang.  Accelerating BERT inference for sequence labeling via early-exit.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)_ , pp. 189–199, 2021.
  * Li et al. (2023) Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda.  SEENN: towards temporal spiking early exit neural networks.  In _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Li et al. (2022) Zonglin Li, Ruiqi Guo, and Sanjiv Kumar.  Decoupled context processing for context augmented language modeling.  In _Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Liao et al. (2021) Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He.  A global past-future early exit method for accelerating inference of pre-trained language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT): Human Language Technologies_ , pp. 2013–2023, 2021.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju.  Fastbert: a self-distilling BERT with adaptive inference time.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pp. 6035–6044, 2020.
  * Liu et al. (2022) Xiangyang Liu, Tianxiang Sun, Junliang He, Jiawen Wu, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu.  Towards efficient NLP: A standard evaluation and A strong baseline.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies_ , pp. 3288–3303, 2022.
  * Liu et al. (2023) Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen.  Deja vu: Contextual sparsity for efficient llms at inference time.  In _Proceedings of the 2023 International Conference on Machine Learning (ICML)_ , pp. 22137–22176, 2023.
  * Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang.  Llm-pruner: On the structural pruning of large language models.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108, 2019.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100, 2022.
  * Sun et al. (2022) Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, and Xipeng Qiu.  A simple hash-based early exiting approach for language understanding and generation.  In _Findings of the Association for Computational Linguistics (ACL)_ , pp. 2409–2421, 2022.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le.  Lamda: Language models for dialog applications.  _CoRR_ , abs/2201.08239, 2022.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971, 2023.
  * Valicenti et al. (2023) Tim Valicenti, Justice Vidal, and Ritik Patnaik.  Mini-gpts: Efficient large language models through contextual pruning.  _CoRR_ , abs/2312.12682, 2023.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 7th International Conference on Learning Representations (ICLR)_ , 2019.
  * Wang et al. (2022) Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng.  Training data is more valuable than you think: A simple and effective method by retrieving from training data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pp. 3170–3179, 2022.
  * Wang et al. (2023) Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei.  Augmenting language models with long-term memory.  In _Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Wolczyk et al. (2021) Maciej Wolczyk, Bartosz Wójcik, Klaudia Balazy, Igor T. Podolak, Jacek Tabor, Marek Smieja, and Tomasz Trzcinski.  Zero time waste: Recycling predictions in early exit neural networks.  In _Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , pp. 2516–2528, 2021.
  * Wu et al. (2024) Shangyu Wu, Ying Xiong, Yufei Cui, Xue Liu, Buzhou Tang, Tei-Wei Kuo, and Chun Jason Xue.  Improving natural language understanding with computation-efficient retrieval representation fusion.  _CoRR_ , abs/2401.02993, 2024.
  * Xin et al. (2020) Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin.  Deebert: Dynamic early exiting for accelerating BERT inference.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pp. 2246–2251, 2020.
  * Xiong et al. (2023) Ying Xiong, Xin Yang, Linjing Liu, Ka-Chun Wong, Qingcai Chen, Yang Xiang, and Buzhou Tang.  EARA: improving biomedical semantic textual similarity with entity-aligned attention and retrieval augmentation.  In _Findings of the Association for Computational Linguistics (EMNLP)_ , pp. 8760–8771, 2023.
  * Zhang et al. (2023a) Jingfan Zhang, Ming Tan, Pengyu Dai, and Wei Zhu.  LECO: improving early exiting via learned exits and comparison-based exiting mechanism.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL): Student Research Workshop_ , pp. 298–309, 2023a.
  * Zhang et al. (2023b) Kexun Zhang, Xianjun Yang, William Yang Wang, and Lei Li.  Redi: Efficient learning-free diffusion inference via trajectory retrieval.  In _Proceedings of the 2023 International Conference on Machine Learning (ICML)_ , pp. 41770–41785, 2023b.
  * Zhou et al. (2020) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei.  BERT loses patience: Fast and robust inference with early exit.  In _Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Zhu (2021) Wei Zhu.  Leebert: Learned early exit for BERT with cross-level optimization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)_ , pp. 2968–2980, 2021.
  * Zhu et al. (2021) Wei Zhu, Xiaoling Wang, Yuan Ni, and Guotong Xie.  GAML-BERT: improving BERT early exiting by gradient aligned mutual learning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 3033–3044, 2021.
  * Zhu et al. (2023) Wei Zhu, Peng Wang, Yuan Ni, Guotong Xie, and Xiaoling Wang.  BADGE: speeding up BERT inference after deployment via block-wise bypasses and divergence-based early exiting.  In _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics (ACL): Industry Track_ , pp. 500–509, 2023.

##  Appendix A Templates on All Tasks

Table 6 provides an overview of the manual templates and selected label words
used for each dataset in this paper. These templates and label words were
created following LM-BFF (Gao et al., 2021).

Table 6: Templates and label words used in this paper. Task | Prompts | Label word
---|---|---
SST-2 |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“terrible”, “1”:“great”
SST-5 |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“terrible”,“1”: “bad”,
|  | “2”: “okay”,“3”: “good”,“4”: “great”
MR |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“terrible”, “1”:“great”
CR |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“terrible”, “1”:“great”
MPQA |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“terrible”, “1”:“great”
SUBJ |  [CLS] $x$ This is [MASK]. [SEP] | “0”:“subjective”, “1”:“objective”
TREC |  [CLS] [MASK] $x$ [SEP] | “0”:“Description”,“1”:“Entity”,“2”:“Expression”,
|  | “3”:“Human”,“4”:“Location”,“5”:“Number”
CoLA |  [CLS] $x$ It was [MASK]. [SEP] | “0”:“incorrect”, “1”:“correct”