# Classification of conformal solitons in pseudo-Euclidean spaces

Burcu Bektaş Demirci  Fatih Sultan Mehmet Vakıf University, Haliç Campus,
Faculty of Engineering, Department of Civil Engineering, 34445, Beyoğlu,
İstanbul, Turkey<EMAIL_ADDRESS>,  Shunya Fujii  Department of
Mathematics, Shimane University, Nishikawatsu 1060 Matsue, 690-8504, Japan.
<EMAIL_ADDRESS>and  Shun Maeta  Department of Mathematics, Faculty of
Education, Chiba University, 1-33, Yayoicho, Inage-ku, Chiba-shi, Chiba,
263-8522 Japan<EMAIL_ADDRESS>or<EMAIL_ADDRESS>
###### Abstract.

In this paper, we completely classify conformal solitons on pseudo–Riemannian
hypersurfaces in pseudo–Euclidean spaces arisen from the position vector
field. In particular, the classification of Yamabe solitons on
pseudo–Riemannian hypersurfaces in pseudo–Euclidean spaces arisen from the
position vector field can be obtained.

###### Key words and phrases:

Yamabe solitons; conformal solitons; pseudo-Euclidean spaces

###### 2010 Mathematics Subject Classification:

53C25, 53C40, 53C42

The third author is partially supported by the Grant-in-Aid for Young
Scientists, No.19K14534, Japan Society for the Promotion of Science.

##  1\. Introduction

Soliton solutions have been developed rapidly in recent years as important
object in geometric flows. In fact, they appear as possible singularity
models. Therefore, one of the most interesting problem is the classification
problem of them. For Ricci solitons, that is, the special solutions of the
Ricci flow. S. Brendle [2] brought significant progress. He showed that ”any
3-dimensional complete noncompact $\kappa$-noncollapsed gradient steady Ricci
soliton with positive curvature is the Bryant soliton” which is a famous
conjecture of G. Perelman [13].

For Yamabe solitons, that is, the special solutions of the Yamabe flow, there
are many classification results under some assumptions of the curvatures (cf.
[4], [5], [8]). Recently, the third author classified nontrivial 3-dimensional
complete gradient Yamabe solitons [11]. To understand the Yamabe soliton, many
generalizations of it have been introduced. For example, almost Yamabe
solitons [1] (see also [14]), gradient $k$-Yamabe solitons [5], $h$-almost
gradient Yamabe solitons [19] have been introduced. The most general notion of
Yamabe solitons might be conformal solitons [9] (see also [16], [15]):

######  Definition 1.1 ([9]).

A Riemannian manifold $(M,g)$ is called a conformal soliton if there exists a
complete vector field $v$ such that

(1) $\varphi g=\frac{1}{2}\mathcal{L}_{v}g,$

where $\varphi:M\rightarrow\mathbb{R}$ is a smooth function and
$\mathcal{L}_{v}g$ is the Lie derivative of the metric tensor $g$ with respect
to $v$. If $v\equiv 0$, then the conformal soliton is called trivial.

J. Cheeger and T. H. Colding studied conformal gradient solitons and gave a
characterization theorem [6]. Recently, the second and the third authors
classified conformal solitons on hypersurfaces in Euclidean spaces under some
assumptions [9].

Although the notion of conformal solitons was given and studied in a
Riemannian manifold, the conformal soliton equation (1) can be defined in a
pseudo–Riemannian manifold as follows.

######  Definition 1.2.

A pseudo–Riemannian manifold $(M,g)$ is called a conformal soliton if there
exists a complete vector field $v$ such that

(2) $\varphi g=\frac{1}{2}\mathcal{L}_{v}g,$

where $\varphi:M\rightarrow\mathbb{R}$ is a smooth function and
$\mathcal{L}_{v}g$ is the Lie derivative of the metric tensor $g$ with respect
to $v$. We denote the conformal soliton by $(M,g,v,\varphi)$. If $v\equiv 0$,
then $(M,g,v,\varphi)$ is called trivial. If $v$ is the gradient of some
smooth function $f$ on $M$, then $(M,g,f)$ is called a conformal gradient
soliton. For smooth functions $f$ and $\varphi$, we have $\varphi
g=\nabla\nabla f.$

######  Remark 1.3.

Conformal solitons include Yamabe solitons, almost Yamabe solitons, gradient
$k$-Yamabe solitons, $h$-almost Yamabe solitons and conformal gradient
solitons. Therefore, all the results in this paper can be applied to all these
solitons.

In [23], M. Brozos-Vázquez et. al. studied three dimensional Lorentzian
homogeneous Ricci soliton and the first author [24] also classified the Ricci
soliton on a pseudo–Riemannian hypersurface of a Minkowski space
$\mathbb{E}^{4}_{1}$ whose the potential vector field is the tangential part
of the position vector.

The position vector $V$ of a pseudo–Riemannian submanifold in the
pseudo–Euclidean space is the most elementary and geometric object, see [20,
21]. A. Fialkow [22] introduced the notion of concircular vector field $v$ on
a Riemannian manifold $M$ as vector field which satisfy

(3) $\nabla_{X}v=\mu X,\;\;X\in TM$

where $\nabla$ denotes the Levi–Civita connection of $M$, $TM$ is the tangent
bundle of $M$ and $\mu$ is a nontrivial function on $M$. As a particular case
for the function $\mu=1$, a concircular vector field $v$ is called a
concurrent vector field. The notion of concircular vector fields on Riemannian
manifolds can be extended naturally to concircular vector fields on
pseudo–Riemannian manifolds. There are several studies of concurrent vector
fields (see for example [3], [17] and [18]). The position vector field $V$ on
a pseudo-Euclidean space $\mathbb{E}_{s}^{m}$ satisfies $\nabla_{X}V=X$, for
any vector field $X$. Therefore, the position vector $V$ is the best known
example as concurrent vector field.

In this paper, we completely classify conformal solitons on a
pseudo–Riemannian hypersurface in a pseudo-Euclidean space
$\mathbb{E}^{n+1}_{s}$ arisen from the position vector field $V$.

######  Theorem 1.4.

Any conformal soliton $(M,g,V^{T},\varphi)$ on a pseudo–Riemannian
hypersurface $M$ in a pseudo-Euclidean space $\mathbb{E}^{n+1}_{s}$ is
contained in a hyperplane, a conic hypersurface, a pseudo hyperbolic space, or
a pseudo sphere.

As a corollary, we completely classify Yamabe solitons on a pseudo–Riemannian
hypersurface in a pseudo-Euclidean space arisen from the position vector
field.

######  Corollary 1.5.

Any Yamabe soliton $(M,g,V^{T},\varphi)$ on a pseudo–Riemannian hypersurface
$M$ in a pseudo-Euclidean space $\mathbb{E}^{n+1}_{s}$ is contained in a
hyperplane, a conic hypersurface, a pseudo hyperbolic space, or a pseudo
sphere.

##  2\. Preliminaries

Let $(N,\tilde{g})$ be an $m$-dimensional pseudo-Riemannian manifold and
$(M,g)$ be an $n$-dimensional submanifold in $(N,\tilde{g})$. All manifolds in
this paper are assumed to be smooth, orientable and connected. We denote Levi-
Civita connections on $(M,g)$ and $(N,\tilde{g})$ by $\nabla$ and
$\tilde{\nabla}$, respectively. The Lie derivative of $g$ is defined by

$\left(\mathcal{L}_{X}g\right)(Y,Z)=X(g(Y,Z))-g([X,Y],Z)-g(Y,[X,Z]),$

for any vector fields $X,Y,Z$ on $M$.

For any vector fields $X,Y$ tangent to $M$ and $\eta$ normal to $M$, the
formula of Gauss is given by

${\widetilde{\nabla}}_{X}Y={\nabla}_{X}Y+h(X,Y),$

where ${\nabla}_{X}Y$ and $h(X,Y)$ are the tangential and the normal
components of ${\tilde{\nabla}}_{X}Y$. The formula of Weingarten is given by

${\widetilde{\nabla}}_{X}\eta=-A_{\eta}(X)+D_{X}\eta,$

where $-A_{\eta}(X)$ and $D_{X}\eta$ are the tangential and the normal
components of ${\widetilde{\nabla}}_{X}\eta$. $A_{\eta}(X)$ and $h(X,Y)$ are
related by

$g(A_{\eta}(X),Y)=\tilde{g}(h(X,Y),\eta).$

The mean curvature vector $H$ of $M$ in $N$ is given by

$\displaystyle H(p)=\frac{1}{n}~{}\sum_{i=1}^{n}\epsilon_{i}h(e_{i},e_{i}),$

where $\\{e_{i}\\}_{i=1}^{n}$ is any orthonormal frame on $M$ at $p$, and
$\epsilon_{i}=g(e_{i},e_{i}).$ For any vector fields $X,Y,Z,W$ tangent to $M$,
the equation of Gauss is given by

$\tilde{g}(\widetilde{Rm}(X,Y)Z,W)=$ | $g(Rm(X,Y)Z,W)$
---|---
| $+\tilde{g}(h(X,Z),h(Y,W))$
| $-\tilde{g}(h(X,W),h(Y,Z)),$

where $Rm$ and $\widetilde{Rm}$ are Riemannian curvature tensors of $M$ and
$N$, respectively. The equation of Codazzi is given by

$(\widetilde{Rm}(X,Y)Z)^{\perp}=({\bar{\nabla}}_{X}h)(Y,Z)-({\bar{\nabla}}_{Y}h)(X,Z),$

where $(\widetilde{Rm}(X,Y)Z)^{\perp}$ is the normal component of
$\widetilde{Rm}(X,Y)Z$ and ${\bar{\nabla}}_{X}h$ is defined by

$({\bar{\nabla}}_{X}h)(Y,Z)=D_{X}h(Y,Z)-h({\nabla}_{X}Y,Z)-h(Y,{\nabla}_{X}Z).$

If $N$ is a space of constant curvature, then the equation of Codazzi reduces
to

$0=({\bar{\nabla}}_{X}h)(Y,Z)-({\bar{\nabla}}_{Y}h)(X,Z).$

Let $\mathbb{E}^{n+1}_{s}$ be an $(n+1)$-dimensional pseudo–Euclidean space
with the flat metric $\tilde{g}$ given as follows

$\tilde{g}=-dx_{1}^{2}-\cdots-dx_{s}^{2}+dx_{s+1}^{2}+\cdots+dx_{n+1}^{2},$

where $(x_{1},\cdots,x_{s},x_{s+1},\cdots,x_{n+1})$ denotes the usual
coordinates in $\mathbb{R}^{n+1}$. The pseudo–Riemannian space forms in
$\mathbb{E}^{n+1}_{s}$ are defined by

(4) $\displaystyle\mathbb{S}^{n}_{s}(c^{2})$ $\displaystyle=\\{{\bf
x}\in\mathbb{E}^{n+1}_{s}\;|\;\tilde{g}({\bf x},{\bf x})=c^{-2}\\}$ (5)
$\displaystyle\mathbb{H}^{n}_{s-1}(-c^{2})$ $\displaystyle=\\{{\bf
x}\in\mathbb{E}^{n+1}_{s}\;|\;\tilde{g}({\bf x},{\bf x})=-c^{-2}\\}.$

These spaces are complete and of constant curvature $c^{2}$ and $-c^{2}$,
respectively. $\mathbb{S}^{n}_{s}(c^{2})$ and $\mathbb{H}^{n}_{s-1}(-c^{2})$
are called a pseudo–sphere and a pseudo–hyperbolic space, respectively.

######  Remark 2.1.

The shape operator of the pseudo–Riemannian hypersurface $M$ in the
Lorentz–Minkowski space $\mathbb{E}^{n+1}_{1}$ can be expressed in one of the
following types (cf. [12] P.261, see also [10]),

$(1)~{}~{}\left(\begin{array}[]{cccc}\kappa_{1}&&&\text{\huge{0}}\\\
&\kappa_{2}&&\\\ &&\ddots&\\\
\text{\huge{0}}&&&\kappa_{n}\end{array}\right),\qquad(2)~{}~{}\left(\begin{array}[]{cc|ccc}\kappa_{1}&-a&&&\text{\huge{0}}\\\
a&\kappa_{1}&&&\\\ \hline\cr&&\kappa_{3}&&\\\ &&&\ddots&\\\
\text{\huge{0}}&&&&\kappa_{n}\end{array}\right)~{}~{}~{}a\not=0,$

for an orthonormal frame $\\{E_{1},E_{2},\cdots,E_{n+1}\\}$;

$\displaystyle\tilde{g}(E_{1},E_{1})=-1,\qquad\tilde{g}(E_{1},E_{i})=0,\quad
i=2,3,\cdots,n+1,$ $\displaystyle\tilde{g}(E_{i},E_{j})=\delta_{ij},\quad
2\leq i,j\leq n+1,$

and

$(3)~{}~{}\left(\begin{array}[]{cc|ccc}\kappa_{1}&0&&&\text{\huge{0}}\\\
1&\kappa_{1}&&&\\\ \hline\cr&&\kappa_{3}&&\\\ &&&\ddots&\\\
\text{\huge{0}}&&&&\kappa_{n}\end{array}\right),\qquad(2)~{}~{}\left(\begin{array}[]{ccc|ccc}\kappa_{1}&0&0&&&\\\
0&\kappa_{1}&1&&\text{\huge{0}}&\\\ -1&0&\kappa_{1}&&&\\\
\hline\cr&&&\kappa_{4}&&\\\ &\text{\huge{0}}&&&\ddots&\\\
&&&&&\kappa_{n}\end{array}\right),$

for a pseudo-orthonormal frame $\\{E_{1},E_{2},\cdots,E_{n+1}\\}$;

$\displaystyle\tilde{g}(E_{1},E_{2})=-1,\qquad\tilde{g}(E_{1},E_{1})=\tilde{g}(E_{2},E_{2})=0,$
$\displaystyle\tilde{g}(E_{i},E_{j})=0,\quad i=1,2,~{}j=3,\cdots,n+1,$
$\displaystyle\tilde{g}(E_{i},E_{j})=\delta_{ij},\quad 3\leq i,j\leq n+1.$

##  3\. Conformal solitons with a concurrent vector field

In this section, we consider a conformal soliton with a concurrent vector
field.

######  Proposition 3.1.

Any conformal soliton $(M,g,v,\varphi)$ which has a concurrent vector field
$v$ is a conformal gradient soliton with $\varphi=1$.

###### Proof.

Since $v$ is a concurrent vector field, we have

(6) $g(v,X)=g(v,{\nabla}_{X}v)=X(\frac{1}{2}g(v,v)),$

and

(7) $\left(\mathcal{L}_{v}g\right)(X,Y)$ | = $vg(X,Y)-g([v,X],Y)-g(X,[v,Y])$
---|---
| $=vg(X,Y)-vg(X,Y)+g({\nabla}_{X}v,Y)+g(X,{\nabla}_{Y}v)$
| $=2g(X,Y),$

for any vector fields $X$, $Y$ on $M$. By putting $f=\frac{1}{2}g(v,v)$ on the
equation $(\ref{c.0})$, we obtain $v=\nabla f$. Substituting $(\ref{c.1})$
into $(\ref{CS})$, we have

$\varphi=1.$

∎

##  4\. Conformal solitons on pseudo–Riemannian submanifolds

In this section, we assume that $(N,\tilde{g})$ is a pseudo–Riemannian
manifold endowed with a concurrent vector field $V$ and $(M,g)$ is a
pseudo–Riemannian submanifold in $(N,\tilde{g})$. $V^{T}$ and $V^{\perp}$
denote the tangential and the normal components of $V$, respectively.

Firstly, we show the following lemma which will be used later for the purpose
of classification of the conformal solitons.

######  Lemma 4.1.

Any conformal soliton $(M,g,V^{T},\varphi)$ on a pseudo–Riemannian submanifold
$M$ in $N$ satisfies

(8) $(\varphi-1)g(X,Y)=g(A_{V^{\perp}}(X),Y),$

for any vector fields $X,Y$ on $M$.

###### Proof.

From the definition of the Lie derivative, we have

(9) $(\mathcal{L}_{V^{T}}g)(X,Y)$ | = $V^{T}g(X,Y)-g(\nabla_{V^{T}}X-\nabla_{X}{V^{T}},Y)-g(X,\nabla_{V^{T}}Y-\nabla_{Y}{V^{T}})$
---|---
| $=g(\nabla_{X}{V^{T}},Y)+g(X,\nabla_{Y}{V^{T}})$
|
$=\tilde{g}(\tilde{\nabla}_{X}{V}-\tilde{\nabla}_{X}{V^{\perp}},Y)+\tilde{g}(X,\tilde{\nabla}_{Y}{V}-\tilde{\nabla}_{Y}{V^{\perp}})$
|  $=2g(X,Y)+2g(A_{V^{\perp}}(X),Y)$,

for any vector fields $X,Y$ on $M$. Combining $(\ref{s.3})$ with
$(\ref{CSn})$, we obtain $(\ref{ENSAYS})$. ∎

######  Proposition 4.2.

Any conformal soliton $(M,g,V^{T},\varphi)$ on a pseudo–Riemannian submanifold
$M$ in $N$ is a conformal gradient soliton.

###### Proof.

Set

$\displaystyle f=\frac{1}{2}~{}\tilde{g}(V,V).$

For any vector field $X$ on $M$, we obtain

$g(V^{T},X)=\tilde{g}(V,X)=\tilde{g}(V,\tilde{\nabla}_{X}V)=X(\frac{1}{2}\tilde{g}(V,V))=Xf=g(\nabla
f,X).$

∎

######  Proposition 4.3.

If a conformal soliton $(M,g,V^{T},\varphi)$ on a pseudo–Riemannian
submanifold $M$ in $N$ is minimal, then $\varphi=1$.

###### Proof.

Let $\\{e_{1},\cdots,e_{n}\\}$ be an orthonormal frame on $M$. By Lemma 4.1,
we have

$(\varphi-1)g_{ij}=g(A_{V^{\perp}}(e_{i}),e_{j})=\tilde{g}(h(e_{i},e_{j}),V^{\perp}).$

Since $M$ is minimal and taking the trace, we obtain

$n(\varphi-1)=n\tilde{g}(H,V^{\perp})=0.$

Therefore, we conclude that

$\varphi=1.$

∎

##  5\. Proof of Theorem 1.4

We hereafter denote $V$ by the position vector field of a pseudo–Riemannian
hypersurface in a pseudo-Euclidean space $\mathbb{E}^{n+1}_{s}$. In this
section, we give the proof of Theorem 1.4.

To show Theorem 1.4, we first consider the case that $V=V^{T}.$ B. Y. Chen and
Y. M. Oh [7] showed the following proposition.

######  Proposition 5.1 ([7]).

Let $M$ be a pseudo–Riemannian submanifold in a pseudo-Euclidean space
$\mathbb{E}^{m}_{s}$. If the position vector field $V$ of $M$ in
$\mathbb{E}^{m}_{s}$ is either space-like or time-like, then $V=V^{T}$ holds
identically if and only if $M$ is a conic submanifolds with the vertex at the
origin.

We can relax the assumption, and get the following.

######  Proposition 5.2.

Let $M$ be a pseudo-Riemannian submanifold in a pseudo-Euclidean space
$\mathbb{E}^{m}_{s}$. Then, $V=V^{T}$ holds identically if and only if $M$ is
a conic submanifolds with the vertex at the origin.

###### Proof.

For the position vector field
$V=\displaystyle\sum_{i=1}^{m}x_{i}\frac{\partial}{\partial x_{i}}$, let
$k=\sqrt{\displaystyle\sum_{i=1}^{m}x_{i}^{2}}$. If $V=V^{T}$ on $M$,
$e=\frac{1}{k}V$ is a vector field tangent to $M$. Then we get

$e=\tilde{\nabla}_{e}V=\tilde{\nabla}_{e}(ke)=e(k)e+k\tilde{\nabla}_{e}e,$

and

$e(k)=(\displaystyle\sum_{i=1}^{m}\frac{x_{i}}{k}\frac{\partial}{\partial
x_{i}})(k)=\displaystyle\sum_{i=1}^{m}(\frac{x_{i}}{k}\cdot\frac{1}{2}\cdot\frac{2x_{i}}{k})=1.$

So we conclude $\tilde{\nabla}_{e}e=0$. Therefore, by the same argument as in
the proof of Proposition 5.1, one can get the proposition. ∎

###### Proof of Theorem 1.4.

Let $\alpha$ be a mean curvature and $\lambda$ be a support function of $M$,
i.e., $H=\alpha N$ and $\lambda=\tilde{g}(N,V)$ with a unit normal vector
field $N$. Set $\epsilon_{N}=\tilde{g}(N,N)=\pm 1$ and $U_{0}=\\{x\in
M|\lambda=0\\}$.

Let $f$ be a smooth function on $M$ defined by

$f(x)\coloneqq\det A(x)\hskip 14.45377ptx\in M.$

Case 1. $U_{0}=\emptyset$: If $V^{\perp}\not=0$, Lemma 4.1 implies that the
pseudo–Riemannian hypersurface $M$ must have a diagonalizable shape operator
with respect to orthonormal frame $e_{i}$ for $i=1,\cdots,n$ . From Lemma 4.1,
we have

$(\varphi-1)g_{ij}=\tilde{g}(h(e_{i},e_{j}),V^{\perp})=\tilde{g}(\epsilon_{N}{\kappa}_{i}g_{ij}N,V)=\epsilon_{N}{\kappa}_{i}g_{ij}\lambda,$

where $A_{N}(e_{i})={\kappa}_{i}e_{i},~{}(i=1,\cdots,n)$. Hence we have

(10) $\varphi-1=\epsilon_{N}\lambda{\kappa}_{i}.$

By taking the summation, we obtain

(11) $\varphi-1=\lambda\alpha.$

Comparing $(\ref{e.1})$ and $(\ref{e.2})$, we have

${\kappa}_{i}=\epsilon_{N}\alpha.$

Thus $M$ is totally umbilical with $A_{N}(e_{i})=\epsilon_{N}\alpha e_{i}$ and
$h$ satisfies $h(X,Y)=\alpha g(X,Y)N$. Since $N$ is a unit normal vector
field, we have

$0={\widetilde{\nabla}}_{X}(\tilde{g}(N,N))=2\tilde{g}({\widetilde{\nabla}}_{X}N,N)=2\tilde{g}(D_{X}N,N).$

Therefore, $D_{X}N=0$. Hence we obtain

$({\bar{\nabla}}_{X}h)(Y,Z)=$ | $D_{X}h(Y,Z)-h({\nabla}_{X}Y,Z)-h(Y,{\nabla}_{X}Z)$
---|---
$=$ | $X(\alpha)g(Y,Z)N,$

for any vector fields $X,Y,Z$ on $M$. From the equation of Codazzi, we have

$X(\alpha)Y=Y(\alpha)X.$

Since we can assume that $X$ and $Y$ are linearly independent, we conclude
that $\alpha$ and $f$ are constant respectively.

If $\alpha=0$, then by ${\widetilde{\nabla}}_{X}N=0,$ $N$, restricted to $M$,
is a constant vector field in $\mathbb{E}^{n+1}_{s}$ and we have

${\widetilde{\nabla}}_{X}(\tilde{g}(V,N))=\tilde{g}({\widetilde{\nabla}}_{X}V,N)+\tilde{g}(V,{\widetilde{\nabla}}_{X}N)=\tilde{g}(X,N)=0.$

This shows that $\lambda=\tilde{g}(V,N)$ is a nonzero constant when $V$ and
$N$ are restricted to $M$. Therefore, $M$ is contained in a hyperplane normal
to $N$ which does not through the origin and $\varphi=1$.

If $\alpha\not=0$, then we have

${\widetilde{\nabla}}_{X}(V+\epsilon_{N}{\alpha}^{-1}N)=X+\epsilon_{N}{\alpha}^{-1}{\tilde{\nabla}}_{X}N=X+\epsilon_{N}{\alpha}^{-1}(-A_{N}(X))=0.$

This shows that the vector field $V+\epsilon_{N}{\alpha}^{-1}N$, restricted to
$M$, is a constant one in $\mathbb{E}^{n+1}_{s}.$ Therefore, $M$ is contained
in a pseudo hyperbolic space $\mathbb{H}^{n}_{s-1}$ or a pseudo sphere
$\mathbb{S}^{n}_{s}$ of $\mathbb{E}^{n+1}_{s}$.

Case 2. $U_{0}=M$:

We have $V=V^{T}$. By Proposition 5.2, we obtain that $M$ is contained in a
conic hypersurface.

Case 3. Others:

Take $p\in M\backslash U_{0}$, that is, $\lambda\not=0$ on some open set
$\Omega\ni p$. By the same argument as in Case 1, we have $\Omega$ is an open
portion of a hyperplane, a pseudo hyperbolic space $\mathbb{H}^{n}_{s-1}$ or a
pseudo sphere $\mathbb{S}^{n}_{s}$ of $\mathbb{E}^{n+1}_{s}$.

We consider the case that $\Omega$ is an open portion of a hyperplane. Without
loss of generality, we can take $\Omega$ as the maximum connected component
which is an open set including $p$ on $M\backslash U_{0}$. On $\Omega$,
$\lambda=\tilde{g}(V,N)(\not=0)$ is constant, say $\lambda_{\Omega}$. Since
$M$ is connected, if $\Omega$ is closed, then $\Omega=M$, which is a
contradiction. If $M$ is not closed, then we have
$\partial\Omega\not=\emptyset$ and $\partial\Omega\cap\Omega=\emptyset.$ Take
$q\in\partial\Omega$. Since $\lambda$ is continuous, we have
$\lambda(q)=\lambda_{\Omega}$. Thus we can take an open neighborhood $U_{q}$
of $q$ such that $\lambda\not=0$ on $U_{q}$. Since $\Omega$ is the maximum
connected component, we have a contradiction. Hence, we have that $\Omega$ is
an open portion of a pseudo hyperbolic space $\mathbb{H}^{n}_{s-1}$ or a
pseudo sphere $\mathbb{S}^{n}_{s}$ of $\mathbb{E}^{n+1}_{s}$.

Assume that Int$U_{0}\neq\emptyset$. On $U_{0}$, we have $V=V^{T}$. Since $V$
is a concurrent vector field,

$X=\widetilde{\nabla}_{X}V^{T}=\nabla_{X}V^{T}+h(X,V^{T}),$

on Int$U_{0}$. So we obtain $h(X,V^{T})=0.$ Therefore, we have
$A_{N}(V^{T})=0$. Thus, the shape operator has zero eigenvalue. Hence, $f=\det
A=0$ on Int$U_{0}$. From this, $X(f)=0$ on Int$U_{0}$ for any vector field $X$
on $M$. From Case 1, $X(f)=0$ on $M\backslash U_{0}$. This means $X(f)=0$ on
$M$. So we conclude that $f$ is constant on $M$. But $f\neq 0$ on $M\backslash
U_{0}$, this is a contradiction. Therefore, Int$U_{0}=\emptyset$.

By the same argument as in Case 1, $\kappa_{i}=\kappa_{j}$ and
$X(\kappa_{i})=0$ on $M\backslash U_{0}$ for any $1\leq i,j\leq n$ and any
vector field $X$. From this and Int$U_{0}=\emptyset$, $\kappa_{i}$ is constant
which doesn’t depend on $i$ on $M$.

Therefore, $M$ is contained in a pseudo hyperbolic space
$\mathbb{H}^{n}_{s-1}$ or a pseudo sphere $\mathbb{S}^{n}_{s}$ of
$\mathbb{E}^{n+1}_{s}$. ∎

## References

  * [1] E. Barbosa and E. Ribeiro, On conformal solutions of the Yamabe flow, Arch. Math. 101 (2013), 79-89.
  * [2] S. Brendle, Rotational symmetry of self-similar solutions to the Ricci flow, Invent. Math. (2013) 194, 731–764.
  * [3] F. Brickell and K. Yano, Concurrent vector fields and Minkowski structures, Kodai Math. Sem. rep. 26 (1974), 22-28.
  * [4] H.-D. Cao, X. Sun and Y. Zhang, On the structure of gradient Yamabe solitons, Math. Res. Lett. 19 (2012), 767–774.
  * [5] G. Catino, C. Mantegazza and L. Mazzieri, On the global structure of conformal gradient solitons with nonnegative Ricci tensor, Comm. Contempt. Math. 14 (2012), 12pp.
  * [6] J. Cheeger and T. H. Colding, Lower bounds on Ricci curvature and the almost rigidity of warped products, Ann. math. (2) 144 (1) (1996), 189-237.
  * [7] B. Y. Chen and Y. M. Oh Classification of rectifying space-like submanifolds in pseudo-Euclidean spaces, Int. Electron. J. Geom. 10 (2017), 86-95.
  * [8] P. Daskalopoulos, and N. Sesum, The classification of locally conformally flat Yamabe solitons, Adv. Math. 240 (2013), 346–369.
  * [9] S. Fujii and S. Maeta, Classification of generalized Yamabe solitons in Euclidean spaces, Internat. J. Math. 32 (2021), 12 pages.
  * [10] P. Lucas, H. Fabián Ramírez–Ospina Hypersurfaces in the Lorentz-Minkowski space satisfying $L_{k}\psi=A\psi+b$, Geom. dedicata 153 (2011), 151–175.
  * [11] S. Maeta, Classification of three dimensional complete gradient Yamabe solitons, arXiv:2104.12391 [math DG].
  * [12] B. O’Neill, Semi-Riemannian Geometry with Applications to Relativity, Academic Press, New York, London (1983)
  * [13] G. Perelman, The entropy formula for the Ricci flow and its geometric applications, arXiv math.DG/0211159, (2002).
  * [14] T. Seko and S. Maeta, Classification of almost Yamabe solitons in Euclidean spaces, J. Geom. Phys. 136 (2019), 97-103.
  * [15] Y. Tashiro, Complete Riemannian manifolds and some vector fields, Trans. Amer. math. Soc., 117 (1965), 251–275.
  * [16] K. Yano, Concircular geometry I, Proc Imp. Acad. Tokyo, 16 (1940), 195–200.
  * [17] K. Yano, Sur le parall$\acute{e}$lisme et la concourance dans l’espace de Riemann, Proc. Imp. Acad. Tokyo, 19 (1943), 189-197.
  * [18] K. Yano and B. Y. Chen, On the concurrent vector fields of immersed manifolds, Kodai Math. Sem. Rep. 23 (1971), 343-350.
  * [19] F. Zeng, On the h-almost Yamabe soliton J. Math. Study (to appear).
  * [20] B. Y. Chen, Topics in differential geometry associated with position vector fields on Euclidean submanifolds, Arab Journal of Mathematical Sciences, 23(1) (2017), 1-17.
  * [21] B. -Y. Chen, Euclidean submanifolds via tangential components of their position vector fields, Mathematics, 5(4) (2017), 51.
  * [22] A. Fialkow, Conformal geodesics, Transactions of the American Mathematical Society, 45(3) (1939), 443-473.
  * [23] M. Brozos–Vázquez, G. Calvaruso, E. Garcia–Rio, S. Gavino-Fernandez, Three–dimensional Lorentzian homogeneous Ricci solitons. Israel Journal of Mathematics, 188(1) (2012), 385-403.
  * [24] B. B. Demirci, Ricci solitons on pseudo–Riemannian hypersurfaces of 4–dimensional Minkowski space Journal of Geometry and Physics, 2022, 104451.


# KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language
Models via Knowledge Distillation

Marzieh S. Tahaei
Noah’s Ark Lab,
Huawei Technologies Canada
<EMAIL_ADDRESS>
&Ella Charlaix
Noah’s Ark Lab,
Huawei Technologies Canada
<EMAIL_ADDRESS>
&Vahid Partovi Nia
Noah’s Ark Lab
Huawei Technologies Canada
<EMAIL_ADDRESS>
Ali Ghodsi
Department of Statistics and
Actuarial Science, University of Waterloo
<EMAIL_ADDRESS>
&Mehdi Rezagholizadeh
Noah’s Ark Lab,
Huawei Technologies Canada
<EMAIL_ADDRESS>
###### Abstract

The development of over-parameterized pre-trained language models has made a
significant contribution toward the success of natural language processing.
While over-parameterization of these models is the key to their generalization
power, it makes them unsuitable for deployment on low-capacity devices. We
push the limits of state-of-the-art Transformer-based pre-trained language
model compression using Kronecker decomposition. We use this decomposition for
compression of the embedding layer, all linear mappings in the multi-head
attention, and the feed-forward network modules in the Transformer layer. We
perform intermediate-layer knowledge distillation using the uncompressed model
as the teacher to improve the performance of the compressed model. We present
our KroneckerBERT, a compressed version of the $\text{BERT}_{\text{BASE}}$
model obtained using this framework. We evaluate the performance of
KroneckerBERT on well-known NLP benchmarks and show that for a high
compression factor of 19 (5% of the size of the $\text{BERT}_{\text{BASE}}$
model), our KroneckerBERT outperforms state-of-the-art compression methods on
the GLUE. Our experiments indicate that the proposed model has promising out-
of-distribution robustness and is superior to the state-of-the-art compression
methods on SQuAD.

##  1 Introduction

In recent years, the emergence of Pre-trained Language Models (PLMs) has led
to a significant breakthrough in Natural Language Processing (NLP). The
introduction of Transformers and unsupervised pre-training on enormous
unlabeled data are the two main factors that contribute to this success.

Transformer-based models (Devlin et al., 2018; Radford et al., 2019; Yang et
al., 2019; Shoeybi et al., 2019) are powerful yet highly over-parameterized.
The enormous size of these models does not meet the constraints imposed by
edge devices on memory, latency, and energy consumption. Therefore there has
been a growing interest in developing new methodologies and frameworks for the
compression of these large PLMs. Similar to other deep learning models, the
main directions for the compression of these models include low-bit
quantization (Gong et al., 2014; Prato et al., 2019), network pruning (Han et
al., 2015), matrix decomposition (Yu et al., 2017; Lioutas et al., 2020) and
Knowledge distillation (Hinton et al., 2015). These methods are either used in
isolation or in combination to improve compression-performance trade-off.

Recent works (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2019; Sun et
al., 2020b; Xu et al., 2020) have been quite successful in compressing
Transformer-based PLMs to a certain degree; however, extreme compression of
these model (compression factors >10) is still quite challenging. Several
works (Mao et al., 2020; Zhao et al., 2019, 2021) that have tried to go beyond
the compression factor of 10, have done so at the expense of a significant
drop in performance. This work proposes a novel framework that uses Kronecker
decomposition for extreme compression of Transformer-based PLMs and provides a
very promising compression-performance trade-off. Similar to other
decomposition methods, Kronecker decomposition can be used to represent weight
matrices in NNs to reduce the model size and the computation overhead. Unlike
the well-known SVD decomposition, Kronecker decomposition retains the rank of
the matrix and hence provides a different expressiveness compared to SVD.

We use Kronecker decomposition for the compression of both Transformer layers
and the embedding layer. For Transformer layers, the compression is achieved
by representing every weight matrix both in the multi-head attention (MHA) and
the feed-forward neural network (FFN) as a Kronecker product of two smaller
matrices. We also propose a Kronecker decomposition for compression of the
embedding layer. Previous works have tried different techniques to reduce the
enormous memory consumption of this layer (Khrulkov et al., 2019; Li et al.,
2018). Our Kronecker decomposition method can substantially reduce the amount
of required memory while maintaining low computation.

Using Kronecker decomposition for large compression factors, which is the
focus of this paper, leads to a reduction in the model expressiveness. To
address this issue, we propose to distill knowledge from the intermediate
layers of the original uncompressed network to the Kronecker network during
training. We use this approach for compression of the
$\text{BERT}_{\text{BASE}}$ model. We show that for a large compression factor
of 19$\times$, KroneckerBERT provides significantly better performance than
state-of-the-art compressed BERT models. While our evaluations in this work
are limited to BERT, this framework can be easily used to compress other
Transformer-based NLP models. The main contributions of this paper are as
follows:

  * •

Developing a framework for compression of the embedding layer using the
Kronecker decomposition.

  * •

Deploying the Kronecker decomposition for the compression of Transformer
modules.

  * •

Proposing a KD method to improve the performance of the Kronecker network.

  * •

Evaluating the proposed framework for compression of BERT${}_{\text{BASE}}$
model on well-known NLP benchmarks

Figure 1: An example of Kronecker product of two 2 by 2 matrices

##  2 Related Work

In this section, we first go through some of the most related works for BERT
compression in the literature and then review the few works that have used
Kronecker decomposition for compression of CNNs and RNNs.

###  2.1 Pre-trained Language Model Compression

In recent years, many model compression methods have been proposed to reduce
the size of PLMs while maintaining their performance on different tasks. KD,
which was first introduced by (Buciluǎ et al., 2006) and then later
generalized by (Hinton et al., 2015), is a popular compression method where a
small student network is trained to mimic the behavior of a larger teacher
network. Recently, using KD for the compression of PLMs has gained a growing
interest in the NLP community. BERT-PKD (Sun et al., 2019), uses KD to
transfer knowledge from the teacher’s intermediate layers to the student in
the fine-tuning stage. TinyBERT (Jiao et al., 2019) uses a two-step
distillation method applied both at the pre-training and at the fine-tuning
stage. MobileBERT (Sun et al., 2020b) also uses an intermediate-layer
knowledge distillation methodology, but the teacher and the student are
designed by incorporating inverted-bottleneck. Several works combine different
compression techniques such as knowledge distillation and pruning with matrix
factorization (Mao et al., 2020) or quantization (Kim and Awadalla, 2020). In
(Mao et al., 2020), the authors present LadaBERT, a lightweight model
compression pipeline combining SVD-based matrix factorization with weight
pruning, as well as a knowledge distillation methodology as described in (Jiao
et al., 2019).

To achieve higher compression factors, authors in (Zhao et al., 2019) use a
layer-wise KD method to reduce the size of vocabulary (from the usual 30k to
5k) and the width of the layers. Similarly, Zhao et al. 2021 uses a mixed-
vocabulary training method to train models with a smaller vocabulary.

###  2.2 Kronecker Factorization

Kronecker products have previously been utilized for the compression of CNNs
and small RNNs. Zhou and Wu 2015 was the first work that utilized Kronecker
decomposition for NN compression. They used summation of multiple Kronecker
products to replace weight matrices in the fully connected and convolution
layers in simple CNN architectures like AlexNet. Thakker et al., 2020 used
Kronecker product for the compression of very small language models for
deployment on IoT devices. To reduce the amount of performance drop after
compression, they propose a hybrid approach where the weight matrix is
decomposed into an upper part and lower part. The upper part remains un-
factorized, and only the lower part is factorized using the Kronecker product.
More recently, Thakker et al. 2020 tried to extend the previous work to non-
IoT applications. Inspired by robust PCA, they add a sparse matrix to
Kronecker product factorization and propose an algorithm for learning these
two matrices together.

To the best of our knowledge, this work is the first attempt to compress
Transformer-based language models using Kronecker decomposition. Unlike prior
arts, this work uses a simple Kronecker product of two matrices for the
representation of linear layers and uses KD framework to improve the
performance of the compressed model.

Figure 2: Illustration of our proposed method for the compression of the
embedding layer. Left: conventional embedding stored in a lookup table. Right:
Our proposed compression method where the original embedding matrix is
represented as a Kronecker product of a matrix and a row vector. The matrix is
stored in a lookup table to minimize computation over head.

##  3 Methodology

In this section, we first introduce the background of Kronecker decomposition.
We then explain our compression method in detail.

###  3.1 Kronecker Product

Kronecker product is an operation that is applied on two matrices resulting in
a block matrix. Let $A$ be a matrix $\in R^{m_{1}\times n_{1}}$, and let $B$
be a matrix $\in R^{m_{2}\times n_{2}}$, then the Kronecker product of $A$ and
$B$ denoted by $\otimes$ is a block matrix, where each block $(i,j)$ is
obtained by multiplying the element $A_{i,j}$ by matrix $B$. Therefore, the of
the resulting matrix $A\otimes B$ is $\in R^{m\times n}$ where $m=m_{1}m_{2}$
and $n=n_{1}n_{2}$. Figure 1 illustrates the Kronecker product between two
small matrices. See (Graham, 2018) for more detailed information on Kronecker
product.

###  3.2 Kronecker factorization

We can use Kronecker products to represent the weight matrices in Neural
Networks (NNs). When decomposing a matrix $W\in R^{m\times n}$, as $A\otimes
B$, there are different choices for the shapes of $A$ and $B$. In fact the
dimensions of $A$ i.e $m_{1}$ and $n_{1}$ can be any factor of $m$ and $n$
respectively. The dimensions of $B$ will then be equal to $m_{2}=m/m_{1}$ and
$n_{2}=n/n_{1}$. We can achieve different compression factors by changing the
shape of these two matrices.

####  3.2.1 Memory and computation reduction

When representing $W$ as $A\otimes B$, the number of elements is reduced from
$mn$ to $m_{1}n_{1}+m_{2}n_{2}$. Moreover, using the Kronecker product to
represent linear layers can reduce the required computation. A trivial way to
multiply the Kronecker product $A\otimes B$ with an input vector $x$, is to
first reconstruct $W$ by obtaining $A\otimes B$ and then multiply the result
with $x$, which is extremely inefficient both with respect to memory and
computation. A much more efficient way is to use Eq.1 which is a well-known
property of the Kronecker product that allows obtaining $(A\otimes B)x$
without explicit reconstruction, $A\otimes B$ (Lutkepohl, 1997):

$(A\otimes B)x=\mathcal{V}(B\mathcal{R}_{n_{2}\times n_{1}}(x)A^{\top})$ (1)

where $A^{\top}$ is $A$ transpose. Here, $\mathcal{V}$ is an operation that
transforms a matrix to a vector by stacking its columns and
$\mathcal{R}_{n_{2}\times n_{1}}(x)$ is an operation that converts a vector
$x$ to a matrix of size $n_{2}\times n_{1}$ by dividing the vector to columns
of size $n_{2}$ and concatenating the resulting columns together. The
consequence of performing multiplication in this way is that it reduces the
number of FLOPs from $(2m_{1}m_{2}-1)n_{1}n_{2}$ to:

$\displaystyle\min\big{(}(2n_{2}-1)m_{2}n_{1}+(2n_{1}-1)m_{2}m_{1},$
$\displaystyle(2n_{1}-1)n_{2}m_{1}+(2n_{2}-1)m_{2}m_{1}\big{)}$ (2)

We use this method for implementation of the Kronecker layers.

Figure 3: Illustration of the proposed framework. Left: A diagram of the
teacher BERT model and the student KronckerBERT. Right: The two stage KD
methodology used to train KroneckerBERT.

###  3.3 Kronecker Embedding layer

The embedding layer in large language models is a very large lookup table
$X\in R^{v\times d}$, where $v$ is the size of the dictionary and $d$ is the
embedding dimension. In order to compress $X$ using Kronecker factorization,
the first step is to define the shape of Kronecker factors $A^{E}$ and
$B^{E}$. We define $A^{E}$ to be a matrix of size $v\times\frac{d}{n}$ and
$B^{E}$ to be a row vector of size $n$. There are two reasons for defining
$B^{E}$ as a row vector. 1) it allows disentangled embedding of each word
since every word has a unique row in $A^{E}$. 2) the embedding of each word
can be obtained efficiently in $\mathcal{O}(d)$. More precisely, the embedding
for the $i$’th word in the dictionary can be obtained by the Kronecker product
between $A^{E}_{i}$ and $B^{E}$:

$X_{i}=A^{E}_{i}\otimes B^{E}$ (3)

where$A^{E}$ is stored as a lookup table. Note that since $A^{E}_{i}$ is of
size $1\times\frac{d}{n}$ and $B^{E}$ is of size $1\times n$, the computation
complexity of this operation is $\mathcal{O}(d)$. Figure 2 shows an
illustration of the Kronecker embedding layer.

###  3.4 Kronecker Transformer

The Transformer layer is composed of two main components: MHA and FFN. We use
Kronecker decomposition to compress both. In the Transformer block, the self-
attention mechanism is done by projecting the input into the Key, Query, and
Value embeddings and obtaining the attention matrices through the following:

$\displaystyle O=\frac{QK^{T}}{\sqrt{d_{k}}}$ (4)
$\displaystyle\textrm{Attention}(Q,K,V)=\textrm{softmax}(O)V$

where $Q$, $K$, and $V$ are obtained by multiplying the input by $W^{Q}$,
$W^{K}$, $W^{V}$ respectively. In a multi-head attention (MHA) module, there
is a separate $W^{Q}_{l}$, $W^{K}_{l}$, and $W^{V}_{l}$ matrix per attention
head to allow for a richer representation of the data. In the implementation
usually, matrices from all heads are stacked together resulting in 3 matrices.

$\displaystyle W^{\prime
k}=\textrm{concat}(W^{K}_{1},..,W^{K}_{l},...,W^{K}_{L})$ (5) $\displaystyle
W^{\prime Q}=\textrm{concat}(W^{Q}_{1},..,W^{Q}_{l},...,W^{Q}_{L})$
$\displaystyle W^{\prime
V}=\textrm{concat}(W^{V}_{1},..,W^{V}_{l},...,W^{V}_{L})$

where $L$ is the number of attention heads. Instead of decomposing the
matrices of each head separately, we use Kronecker decomposition after
concatenation:

$\displaystyle W^{\prime K}=A^{k}\otimes B^{K}$ (6) $\displaystyle W^{\prime
Q}=A^{Q}\otimes B^{Q}$ $\displaystyle W^{\prime V}=A^{V}\otimes B^{V}$

By choosing $m_{2}$ to be smaller than the output dimension of each attention
head, matrix $B$ in the Kronecker decomposition is shared among all attention
heads resulting in more compression. The result of applying Eq.4 is then fed
to a linear mapping ($W^{\textrm{O}}$) to produce the MHA output. We use
Kronecker decomposition for compressing this linear mapping as well the two
weight matrices in the subsequent FFN block:

$\displaystyle W^{\textrm{O}}=A^{\textrm{O}}\otimes B^{\textrm{O}}$ (7)
$\displaystyle W_{1}=A_{1}\otimes B_{1}$ (8) $\displaystyle W_{2}=A_{2}\otimes
B_{2}$ (9)

###  3.5 Knowledge distillation

In the following section, we describe how KD is used to improve the training
of the KroneckerBERT model.

Model | CMP |  $\mathbf{W^{K},W^{Q},}$ $\mathbf{W^{V},W^{O}}$ |  $\mathbf{W_{1}},[\mathbf{W_{2}}]^{T}$ |  $\mathbf{W^{E}}$
---|---|---|---|---
|  | ${n,m}$ |  ${d}$
BERT${}_{\text{BASE}}$ | 1$\times$ |  768, 768 |  768, 3072 |  768
|  | ${n_{1},m_{1}}$ |  ${n}$
KroneckerBERT8 | 7.7$\times$ |  384, 384 |  2, 8 |  8
KroneckerBERT19 | 19.3$\times$ |  48, 384 |  2, 16 |  12

Table 1:  Configuration of the Kronecker layers for the two KroneckerBERT
models used in this paper. CMP stands for compression factor.$n$ and $m$ are
respectively the input and output dimensions of the weight matrices ($W\in
R^{m\times n}$). $m_{1},n_{1}$ indicates the shape of the first Kronecker
factor ($A\in R^{m_{1}\times n_{1}}$). For embedding layer we only need to set
the size of the row vector $B^{E}\in R^{1\times n}$.

####  3.5.1 Intermediate KD

Let $S$ be the student, and $T$ be the teacher, then for a batch of data
$(X,Y)$, we define $f_{l}^{S}(X)$ and$f_{l}^{T}(X)$ as the output of the
$l^{th}$ layer for the student network and the teacher network respectively.
The teacher here is the $\text{BERT}_{\text{BASE}}$ and the student is its
corresponding KroneckerBERT that is obtained by replacing the embedding layer
and the linear mappings in MHA and FFN modules with Kronecker factors(see
Sections 3.3 and 3.4 for details). Note that like other decomposition methods,
when we use Kronecker factorization to compress the model, the number of
layers and the dimensions of the input and output of each layer remain intact.
Therefore, when performing intermediate layer KD, we can directly obtain the
difference in the output of a specific layer in the teacher and student
networks without any projection. In the proposed framework, the intermediate
KD from the teacher to student occurs at the embedding layer output, attention
matrices and FFN outputs:

$\displaystyle\mathcal{L}_{\textrm{Embedding}}(X)=\textrm{MSE}\big{(}{E}^{S}(X),{E}^{T}(X)\big{)}$
(10)
$\displaystyle\mathcal{L}_{\textrm{Attention}}(X)=\sum_{l}\textrm{MSE}\big{(}{O}_{l}^{S}(X),{O}_{l}^{T}(X)\big{)}$
(11)
$\displaystyle\mathcal{L}_{\textrm{FFN}}(X)=\sum_{l}\textrm{MSE}\big{(}{H}_{l}^{S}(X),{H}_{l}^{T}(X)\big{)}$
(12)

where $E^{S}$ and $E^{T}$ are the output of the embedding layer from the
student and the teacher respectively. $O_{l}^{S}$ and $O_{l}^{T}$ are the
attention matrices (Eq.4), ${H}_{l}^{S}$ and ${H}_{l}^{T}$ are the outputs of
the FFN, of layer $l$ in the student and the teacher respectively.

So far we have been using MSE of the intermediate features for transferring
knowledge from the teacher to the student. Therefore, each element in the
feature vector of the student and teacher is compared independently of other
elements. Inspired by (Sun et al., 2020a), and in order to have a richer
comparison between the student and the teacher networks, we also add a
projection loss term. To obtain the projection loss we first average pool the
FFN output and attention outputs of the last layer ($H_{L}$ and $A_{L}$) and
concatenate them together to obtain a feature vector. We project the feature
vector obtained from the teacher using a learnable weight matrix $P\in
R^{2d\times 2d}$, where $d$ is the hidden dimension of the Transformer. We
then obtain the projection loss as the MSE between student’s features and
teacher’s projected features:

$\displaystyle
g^{S}(x)=\textrm{concat}\big{[}\text{pool}(A^{S}_{L}),\text{pool}(H^{S}_{L})\big{]},$
(13) $\displaystyle
g^{T}(x)=\textrm{concat}\big{[}\text{pool}(A^{T}_{L}),\text{pool}(H^{T}_{L}))\big{]},$
(14)
$\displaystyle\mathcal{L}_{Projection}(x)=\textrm{MSE}(g^{S}(x),Pg^{T}(x)).$
(15)

Our final loss is as follows:

$\displaystyle\mathcal{L}(x,y)=$
$\displaystyle\sum_{(x,y)}\mathcal{L}_{Embedding}(x)+$ (16)
$\displaystyle\mathcal{L}_{Attention}(x)+\mathcal{L}_{FFN}(x)+$
$\displaystyle\mathcal{L}_{Projection}(x)+\mathcal{L}_{Logits}(x)+$
$\displaystyle\mathcal{L}_{CE}(x,y).$

Note that, unlike some other KD methods where the motivation for this
projection is to match the dimension of the student and teacher features, here
we use it to obtain a richer representation of the knowledge.

####  3.5.2 KD at pre-training

Inspired by (Jiao et al., 2019) we use KD at the pre-training stage to capture
the general domain knowledge from the teacher. In pre-training distillation,
the teacher is the $\text{BERT}_{\text{BASE}}$ model that is pre-trained on
BookCorpus (Zhu et al., 2015) and English Wikipedia. Intermediate layer KD is
then used to train the KroneckeBERT network in the general domain. KD at pre-
training improves the initialization of the Kronecker model for the task-
specific KD stage. Similar to (Jiao et al., 2019) the loss at pre-training
stage only involves the intermediate layers
$\mathcal{L}_{Embedding}(x)+\mathcal{L}_{Attention}(x)+\mathcal{L}_{FFN}(x)$.
Unlike (Jiao et al., 2019), we perform pre-training distillation only on a
small portion of the dataset (5% of the English Wikipedia) for a few epochs (3
epochs) which makes our training far more efficient.

###  3.6 Model Settings

The first step of the proposed framework is to design the Kronecker layers by
defining the shape of Kronecker factors matrices $A$ and $B$. To do this we
need to set the shape of one of these matrices and the other one can be
obtained accordingly. Therefore we only searched among different choices for
$m_{1}$ and $n_{1}$ which are limited to the factors of the original weight
matrix (m and n respectively). We used the same configuration for all the
matrices in the MHA. Also For the FFN, we chose the configuration for one
layer, and for the other layer, the dimensions are swapped. For the embedding
layer, since $B^{E}$ is a row vector, we only need to choose $n$.

The shapes of the Kronecker factors are chosen to obtain the desired
compression factor. However, there may be multiple configurations to achieve
that, among which we chose the one that leads to minimum FLOPS according to
Eq.2. Table 1 summarises the configuration of Kronecker factorization for the
two compression factors used in this work.

###  3.7 Implementation details

For KD at the pre-training stage, the KroneckerBERT model is initialized using
the teacher (pre-trained BERT${}_{\text{BASE}}$ model). This means that for
layers that are not compressed, the values are copied from the teacher to the
student. For Kronecker decomposed layers, the $L_{2}$ norm approximation (Van
Loan, 2000) is used to approximate Kronecker factors ($A$ and $B$) from the
pre-trained BERT${}_{\text{BASE}}$ model. In the pre-training stage, 5% of the
English Wikipedia was used for 3 epochs. The batch size in pre-training was
set to 64 and the learning rate was set e-3. After pre-training, the obtained
Kronecker model is used to initialize the Kronecker layers in the student
model for task-specific fine-tuning. The Prediction layer is initialized from
the fine-tuned BERT${}_{\text{BASE}}$ teacher. For fine-tuning on each task,
we optimize the hyper-parameters based on the performance of the model on the
dev set. See appendix for more details on the results on dev set and the
selected hyperparamters.

Model | Params | MNLI-(m/mm) | SST-2 | MRPC | CoLA | QQP | QNLI | RTE | STS-B | Avg
---|---|---|---|---|---|---|---|---|---|---
BERT${}_{\text{BASE}}$ | 108.5M | 83.9/83.4 | 93.4 | 87.9 | 52.8 | 71.1 | 90.9 | 67 | 85.2 | 79.5
BERT4-PKD | 52.2M | 79.9/79.3 | 89.4 | 82.6 | 24.8 | 70.2 | 85.1 | 62.3 | 79.8 | 72.6
TinyBERT | 14.5M | 82.5/81.8 | 92.6 | 86.4 | 44.1 | 71.3 | 87.7 | 66.6 | 80.4 | 77.0
LadaBERT3 | 15M | 82.1/81.8 | 89.9 | - | - | 69.4 | 84.5 | - | - | -
KroneckerBERT8 | 14.3M |  82.9/81.7 | 91.2 | 88.5 | 31.2 | 70.8 | 88.4 | 66.9 | 83.1 | 76.1
SharedProject | 5.6M | 76.4/75.2 | 84.7 | 84.9 | - | - | - | - | - | -
LadaBERT4 | 11M | 75.8/76.1 | 84.0 | - | - | 67.4 | 75.1 | - | - | -
KroneckerBERT19 | 5.7M | 79.4/81.6 | 89.2 | 86.9 | 25.8 | 69.2 | 86.2 | 62.7 | 78.2 | 73.1
Table 2: Results on the test set of GLUE official benchmark. The results for BERT, BERT4-PKD and TinyBERT are taken from (Jiao et al., 2019). For all other baselines the results are taken from their associated papers. SharedProject and LadaBERT refer to (Zhao et al., 2019) and (Mao et al., 2020) respectively. Note that TinyBERT performs pre-training KD on the entire Wikipedia dataset and fine-tuning on Augmented data whereas we only perform pre-training KD on 5% of the Wikipedia and we do not use data augmentation for fine-tuning. Model |  MRPC $\rightarrow$ PAWS |  RTE $\rightarrow$ HANS |  SST-2 $\rightarrow$ IMDb
---|---|---|---
BERT${}_{\text{BASE}}$ |  61.3 |  50.7 |  88.0
TinyBERT |  61.3 |  51.2 |  78.5
KroneckerBERT8 |  61.4 |  52.7 |  81.0
Table 3: The results of out of distribution experiment. Fined-tuned models on
MRPC, RTE and SST-2 are evaluated on PAWS, HANS and IMDb respectively.

|  | SQuAD1.1 | SQuAD2.0
---|---|---|---
Model | Compress | EM | F1 | EM | F1
BERT${}_{\text{BASE}}$ | 1$\times$ | 80.5 | 88 | 74.5 | 77.7
BERT4-PKD | 2.1$\times$ | 70.1 | 79.5 | 60.8 | 64.6
TinyBERT | 7.5$\times$ | 72.7 | 82.1 | 68.2 | 71.8
KroneckerBERT8 | 7.6$\times$ | 75.4 | 84.2 | 69.9 | 73.5
KroneckerBERT19 | 19.3$\times$ | 66.7 | 77.8 | 63.0 | 67.0

Table 4:  Results of the baselines and KroneckerBERT on question SQuAD dev
dataset. The results of the baselines are taken from (Jiao et al., 2019).

##  4 Experiments

In this section, we compare our KroneckerBERT with the sate-of-the-art
compression methods applied to BERT on GLUE and SQuAD. We also investigate the
effect of KD through ablation experiments and investigate the robustness of
our model to out-of-distribution samples. Moreover we show the speedup results
of deploying our model on edge devices and justify our findings.

###  4.1 Baselines

As for baselines we select two categories of compression methods, those with
compression factor <10 and those with compression factor >10\. For a fair
comparison, we select models that have BERT${}_{\text{BASE}}$ as the teacher.
In the first category, we both have BERT${}_{\text{PKD}}$ (Sun et al., 2019)
with low compression factor, and models with similar compression factor as our
KroneckerBERT8: TinyBERT (Jiao et al., 2019) and LadaBERT (Mao et al., 2020).

Note that TinyBERT is not directly comparable since they do KD at pre-training
on the entire Wikipedia dataset and also they do an extensive augmentation on
GLUE for KD in the fine-tuning stage (x20). We exclude MobileBERT (Sun et al.,
2020b) since they use a redesign of BERT${}_{\text{LARGE}}$ with inverted-
bottleneck as the teacher. For the second category, we compare our results
with SharedProject (Zhao et al., 2019) and (Mao et al., 2020) with compression
factor in the rage of 10-20. We exclude (Zhao et al., 2021) since they use
BERT${}_{\text{LARGE}}$ as the teacher.

###  4.2 Results on GLUE benchmark

We evaluated the proposed framework on the General Language Understanding
Evaluation (GLUE) (Wang et al., 2018) benchmark which consists of 9 natural
language understanding tasks. We submitted the predictions of our proposed
models on the test data sets for different tasks to the official GLUE
benchmark (https://gluebenchmark.com/). Table 2 summarizes the results. The
results are divided into two categories: in the upper part of the table
KroneckerBERT8 are compared to several state-of-the-art KD methods that have a
compression factor less than 10. In the lower part of the table, we compare
the performance of KroneckerBERT19 with state-of-the-art models that have a
compression factor greater than 10. We can see that when the compression
factor is less than 10, KroneckerBERT outperforms other baselines, except
TinyBERT, on every task of GLUE. As for TinyBERT KroneckerBERT has a very
similar performance in all the tasks except for CoLA. Note TinyBERT performs
pre-training KD on the entire Wikipedia dataset and uses an extensive data
augmentation (20 times the original data) in the fine-tuning stage to obtain
these results. Moreover, the average performance of KroneckerBERT8 excluding
CoLA is 81.7 compared to 81.2 for TinyBERT. For higher compression factors,
KroneckerBERT outperforms all other baselines on all available results.

###  4.3 Out of distribution robustness

It is shown that pre-trained Transformer-based language models are robust to
out-of-distribution (OOD) samples (Hendrycks et al., 2020). In this section,
we investigate how the proposed compression method affects the OOD robustness
of BERT by evaluating the fined-tuned models on MRPC, RTE, and SST-2 on PAWS
(Zhang et al., 2019), HANS (McCoy et al., 2019), and IMDb (Maas et al., 2011)
respectively. We compare OOD robustness with the teacher,
BERT${}_{\text{BASE}}$ and TinyBERT. TinyBERT fine-tuned checkpoint are
obtained from their repsitory. Table 3 lists the results. We can see the fine-
tuned KroneckerBERT8 models on MRPC and RTE are robust to OOD since there is a
small increase in performance compared to BERT${}_{\text{BASE}}$. On IMDb,
there is a significant drop in performance after compression, but our
KroenckerBERT8 is still more robust than TinyBERT. In fact, KroencekrBERT8
outperforms TinyBERT on all the three OOD experiments.

###  4.4 Results on SQuAD

In this section, we evaluate the performance of the proposed model on SQuAD
datasets. SQuAD1.1 (Rajpurkar et al., 2016) is a large-scale reading
comprehension that contain questions that have answers in given context.
SQuAD2.0 (Kudo and Richardson, 2018) also contains unanswerable questions.
Table 4 summarises the performance on dev set. For both SQuAD1.1 and SQuAD2.0,
KroneckerBERT8 can significantly outperform other baselines. We have also
listed the performance of KroneckerBERT19. The results of baselines with
higher compression factors on SQuAD were not available.

Figure 4: T-SNE visualization of the output of the middle Transformer layer of
the fine-tuned models on SST-2 dev. Left: Fine-tuned BERT${}_{\text{BASE}}$,
middle: KroneckerBERT8 fine-tuned without KD, right: KroneckerBERT8 when
trained using KD in two stages. The colours indicate the positive and negative
classes.

Pre-training |  Fine-tuning | MNLI-m | SST-2 | MRPC
---|---|---|---|---
None |  No KD | 66.0 | 81.3 | 68.3
None |  KD | 80.7 | 86.2 | 70.3
KD |  No KD | 77.0 | 87.2 | 78.17
KD |  KD | 82.8 | 90.6 | 86.6

Table 5: Ablation study of the effect of KD in the pre-training and fine-
tuning stages on the performance of the model on GLUE dev. The teacher for KD
at the pre-training stage and at the fine-tuning stage is the pre-trained and
the fined-tuned BERT${}_{\text{BASE}}$ respectively.

###  4.5 Ablation study on the effect of KD

In this section, we investigate the effect of KD in reducing the gap between
the compressed KroneckerBERT model and the original BERT${}_{\text{BASE}}$
network. Table 5 summarises the results for KroneckerBERT8. Our proposed
method uses KD in both the pre-training and the fine-tuning stages. For this
ablation study, pre-training is either not performed or is done using KD with
the pre-trained BERT${}_{\text{BASE}}$ as the teacher. We perform experiments
on 3 tasks from the GLUE benchmark with different sizes of training data,
namely MNLI-m, SST-2, and MRPC. We can see that without KD, Kronecker
decomposition leads to a significant drop in performance. For MNLI-m we can
see that KD at fine-tuning stages is more effective than in performing pre-
training with KD. For SST-2 and MRPC, performing pre-training with KD is more
effective. For all tasks, the highest performance is obtained when the two-
stage KD is used(last row).

We also used t-SNE to visualize the output of the FFN of the middle layer
(layer 6) of the fine-tuned KroneckerBERT8 with and without KD in comparison
with the fine-tuned teacher, on SST-2 dev. Figure 4 shows the results. See how
KD helps the features of the middle layer to be more separable with respect to
the task compared to the no KD case.

Model | FLOPS | Inference Speedup
---|---|---
|  | Smartphone 1 | Smartphone 2 | CPU
BERT${}_{\text{BASE}}$ | 22B | 1 | 1 | 1
KroneckerBERT8 | 5.5B | 0.94 | 0.74 | 0.81
KroneckerBERT19 | 1.4B | 2.4 | 1.93 | 1.1

Table 6: Number of FLOPS and inference speed up on 2 smartphones and Intel CPU
(Intel Xeon Gold6140) for the proposed model with respect to
BERT${}_{\text{BASE}}$.

###  4.6 Inference speedup: Discussion and results

Decomposition methods like SVD, Tucker, etc. all try to represent a given
tensor as multiple significantly smaller tensors and thus reduce the number of
parameters. The same is also true for the number of operations involved in the
forward pass: instead of one large matrix operation, we often have multiple
small matrix operations which can lead to a reduction in the number of FLOPS.
However, this reduction in the number of FLOPS may not lead to latency
reduction since these smaller matrix operations have to be performed in
serial. Kronecker decomposition is no exception in this regard. In order to
perform Kronecker product matrix multiplication $(A\otimes B)X$ we need to
multiply 3 matrices as in Eq.1. Multiplication of 3 matrices is done through
two matrix-matrix multiplication in serial. Serial nature of this operation
limits utilization of parallel processing units on modern CPUs and GPUs.
Therefore without modification of the underlying software/hardware, the
reduction of the number of FLOPs may not lead to latency reduction on high-
performance devices.

In order to investigate how this reduction in the number of FLOPS is
translated to a reduction in the latency, for high compression factors, we
serialized the Kronecker19 using PyTorch’s TorchScript and deployed it on two
smartphones as well as on a high-performance CPU (18 core Intel©Xeon©GOLD
6140). The specification of octa-core processors for smartphone 1 and 2 is
(4x2.4GHz, 4x1.8GHz) and (2x2.6 GHz,2x1.9GHz, 4x1.8GHz) respectively. We
measured the latency over 50 runs of a single input sequence of length 128.
The results are listed in Table 6. We can see that for the high compression
factor of 19, the compressed model can have up to 2.4 speed up on edge
devices. However, on the high-performance CPU with much higher memory and
computation capacity using KroneckerBERT does not lead to speed-up which
confirms the above argument.

##  5 Conclusion

We introduced a novel framework for compressing Transformer-based language
models. The proposed model uses Kronecker decomposition for the compression of
the embedding layer and the linear mappings within the Transformer blocks. The
proposed framework was used to compress the BERT${}_{\text{BASE}}$ model. We
used an efficient two-stage KD method to train the compressed model. We show
that the proposed framework can significantly reduce the size and the number
of computations while outperforming state-of-the-art for high compression
factors of at least $10\times$. The proposed methodology can be applied to any
other Transformer based language model.

## Acknowledgements

Authors would like to thank Mahdi Zolnouri for deploying the models on
cellphones and obtaining latency results and Aref Jaffari for preparing the
OOD datasets. We also would like to thank Seyed Alireza Ghaffari and Eyyüb
Sari for informative discussions throughout this project.

## References

  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 535–541.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gong et al. (2014) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014.  Compressing deep convolutional networks using vector quantization.  _arXiv preprint arXiv:1412.6115_.
  * Graham (2018) Alexander Graham. 2018.  _Kronecker products and matrix calculus with applications_.  Courier Dover Publications.
  * Han et al. (2015) Song Han, Huizi Mao, and William J Dally. 2015.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _arXiv preprint arXiv:1510.00149_.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  _arXiv preprint arXiv:2004.06100_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019.  Tinybert: Distilling bert for natural language understanding.  _arXiv preprint arXiv:1909.10351_.
  * Khrulkov et al. (2019) Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets. 2019\.  Tensorized embedding layers for efficient model compression.  _arXiv preprint arXiv:1901.10787_.
  * Kim and Awadalla (2020) Young Jin Kim and Hany Hassan Awadalla. 2020.  Fastformers: Highly efficient transformer models for natural language understanding.  _arXiv preprint arXiv:2010.13382_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_.
  * Li et al. (2018) Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. 2018\.  Slim embedding layers for recurrent neural language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Lioutas et al. (2020) Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md Akmal Haidar, and Mehdi Rezagholizadeh. 2020.  Improving word embedding factorization for compression using distilled nonlinear neural decomposition.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 2774–2784.
  * Lutkepohl (1997) Helmut Lutkepohl. 1997.  Handbook of matrices.  _Computational statistics and Data analysis_ , 2(25):243.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mao et al. (2020) Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, and Jing Bai. 2020.  Ladabert: Lightweight adaptation of bert through hybrid model compression.  _arXiv preprint arXiv:2004.04124_.
  * McCoy et al. (2019) R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  _arXiv preprint arXiv:1902.01007_.
  * Prato et al. (2019) Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. 2019.  Fully quantized transformer for machine translation.  _arXiv preprint arXiv:1910.10485_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016\.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  _arXiv e-prints_ , page arXiv:1606.05250.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for bert model compression.  _arXiv preprint arXiv:1908.09355_.
  * Sun et al. (2020a) Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, and Jingjing Liu. 2020a.  Contrastive distillation on intermediate representations for language model compression.  _arXiv preprint arXiv:2009.14167_.
  * Sun et al. (2020b) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020b.  Mobilebert: a compact task-agnostic bert for resource-limited devices.  _arXiv preprint arXiv:2004.02984_.
  * Thakker et al. (2020) Urmish Thakker, Paul Whatamough, Matthew Mattina, and Jesse Beu. 2020.  Compressing language models using doped kronecker products.  _arXiv preprint arXiv:2001.08896_.
  * Van Loan (2000) Charles F Van Loan. 2000.  The ubiquitous kronecker product.  _Journal of computational and applied mathematics_ , 123(1-2):85–100.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Xu et al. (2020) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020.  Bert-of-theseus: Compressing bert by progressive module replacing.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_.
  * Yu et al. (2017) Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017.  On compressing deep models by low rank and sparse decomposition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7370–7379.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  Paws: Paraphrase adversaries from word scrambling.  _arXiv preprint arXiv:1904.01130_.
  * Zhao et al. (2019) Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019.  Extreme language model compression with optimal subwords and shared projections.  _arXiv preprint arXiv:1909.11687_.
  * Zhao et al. (2021) Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2021.  Extremely small bert models from mixed-vocabulary training.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2753–2759.
  * Zhou and Wu (2015) Shuchang Zhou and Jia-Nan Wu. 2015.  Compression of fully-connected layer in neural network by kronecker product.  _arXiv preprint arXiv:1507.05775_.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_ , pages 19–27.