# A further investigation on the order of the Schur multiplier of $p$-groups

Peyman Niroomand  School of Mathematics and Computer Science
Damghan University, Damghan, Iran<EMAIL_ADDRESS><EMAIL_ADDRESS>and  Farangis Johari  Department of Pure Mathematics
Ferdowsi University of Mashhad, Mashhad, Iran<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

For a $p$-group $G$ of order $p^{n}$ with the derived subgroup of order
$p^{k}$ if $d=d(G),$ the minimal number of elements required to generate $G,$
then the order of the Schur multiplier of $G$ is bounded by
$p^{\frac{1}{2}(d-1)(n-k+2)+1}.$ In the current manuscript, we find the
structure of all $p$-groups that attains the mentioned bound. Moreover, we
show that all of them are capable.

###### Key words and phrases:

Schur multiplier, capable groups, finite $p$-groups

Mathematics Subject Classification 2010. Primary 20D15; Secondary 20E34. 20F18

##  1\. Motivation and Preliminaries

The Schur multiplier, $\mathcal{M}(G),$ of a group $G$ was appeared in works
of Schur in $1904$ during works on the projective representation of groups.
The Schur multiplier was studied by several authors and proved to be an
important tools in the classification of $p$-groups. For a $p$-group of order
$p^{n},$ by a result of Green in [9], we have $|\mathcal{M}(G)|\leq
p^{\frac{1}{2}n(n-1)-t(G)}$ for $t(G)\geq 0.$ Several authors tried to
characterize the structure of $p$-groups by invoking $t(G).$ The reader can
find the structure of $p$-groups when $t(G)\in\\{0,\ldots,6\\}$ in [1, 6, 10,
16, 21, 22]. Later in [15], the first author improved the Green’s bound and
showed for any non-abelian $p$-groups of order $p^{n}$ with
$|G^{\prime}|=p^{k},$ we have

(1.1) $\displaystyle|\mathcal{M}(G)|\leq p^{\frac{1}{2}(n-k-1)(n+k-2)+1}.$

He also characterized all of $p$-groups that attain the upper bound when
$m=1.$ Recently in [17], Rai improved this bound. He showed for a $p$-group of
order $p^{n}$ with $|G^{\prime}|=p^{k}$ and $d(G)=d,$ we have

(1.2) $\displaystyle|\mathcal{M}(G)|\leq p^{\frac{1}{2}(d-1)(n+k-2)+1}.$

By the same motivation of [10], the main part of this paper is devoted to find
the structure of all $p$-groups that attain the bound (1.2), and then we show
that all of them are capable.

Recall that a group $G$ is called capable provided that $G\cong H/Z(H)$ for a
group $H.$ Beyl et al. in [3] gave a criterion for detecting capable groups.
They showed a group $G$ is capable if and only if the epicenter of
$G,Z^{*}(G),$ is trivial. Ellis in [8] showed $Z^{\wedge}(G)=Z^{*}(G),$ in
which, $Z^{\wedge}(G),$ the exterior center of $G$ is the set of all elements
$g$ of $G$ for which $g\wedge h=1_{G\wedge G}$ for all $h\in G$ and $\wedge$
denotes the operator of the non-abelian exterior square (see for instance [8]
to find more information in this topics).

The following technical result characterizes the structure of all minimal non-
abelian $p$-groups.

######  Lemma 1.1.

[2, Exercise 8a.] and [20] Let $G$ be a minimal non-abelian $p$-group. Then
$|G^{\prime}|=p$ and $G$ is one of the following groups:

  * $(a).$

$G\cong\langle
a,b|a^{p^{m}}=1,b^{p^{n}}=1,[a,b]=a^{p^{m-1}},[a,b,a]=[a,b,b]=1\rangle,m\geq
2,n\geq 1,|G|=p^{m+n}$ and $Z(G)=\langle a^{p}\rangle\times\langle
b^{p}\rangle$.

  * $(b).$

$G\cong\langle
a,b|a^{p^{m}}=b^{p^{n}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle$ is of order
$p^{m+n+1}$ and if $p=2,$ then $m+n>2.$ Moreover, $Z(G)=\langle
a^{p}\rangle\times\langle b^{p}\rangle\times\langle c\rangle.$

  * $(c).$

$G\cong Q_{8}.$

######  Proposition 1.2.

Let $G$ be a minimal non-abelian $p$-group such that $G\cong\langle
a,b|a^{p^{m}}=1,b^{p^{n}}=1,[a,b]=a^{p^{m-1}},[a,b,a]=[a,b,b]=1\rangle,m\geq
2,n\geq 1$ for $p>2$ and for $p=2,$ $m\geq 3.$ If $G/G^{\prime}$ is
homocyclic, then $n=m-1,$ $G$ is non-capable and $Z^{\wedge}(G)=G^{\prime}.$

###### Proof.

Since $G/G^{\prime}$ is homocyclic and
$G/G^{\prime}\cong\mathbb{Z}_{p^{n}}\times\mathbb{Z}_{p^{m-1}},$ we have
$n=m-1.$ We claim that $Z^{\wedge}(G)=G^{\prime}.$ [3, Corollary 7.4] implies
$G/G^{\prime}$ is capable and so $Z^{\wedge}(G)\subseteq
G^{\prime}\cong\mathbb{Z}_{p},$ by [3, Corollary 2.2]. It is sufficient to
show that $G^{\prime}=\langle a^{p^{m-1}}\rangle\subseteq Z^{\wedge}(G).$
Since $b^{{}^{p^{m-1}}}=1,$ we have

$\displaystyle 1_{G\wedge G}$ $\displaystyle=b^{{}^{p^{m-1}}}\wedge
a=\prod_{i=p^{m-1}-1}^{0}(^{b^{i}}(b\wedge
a))=\prod_{i=p^{m-1}-1}^{0}(a\wedge[b^{i},a]a)$
$\displaystyle=\prod_{i=p^{m-1}-1}^{1}(b\wedge[b,a])^{i}\prod_{i=p^{m-1}-1}^{1}(b\wedge
a)=\big{(}\prod_{i=p^{m-1}-1}^{1}(b\wedge[b,a])^{i}\big{)}(b\wedge
a)^{p^{m-1}}.$

Clearly,

$\displaystyle\prod_{i=p^{m-1}-1}^{1}(b\wedge[b,a])^{i}$
$\displaystyle=(b\wedge[b,a])^{\frac{1}{2}p^{m-1}(p^{m-1}-1)}$
$\displaystyle=(b\wedge[b,a]^{\frac{1}{2}p^{m-1}(p^{m-1}-1)})=(b\wedge[b^{\frac{1}{2}p^{m-1}(p^{m-1}-1)},a]).$

Now $[a,b]=a^{p^{m-1}},$ and so
$[b,a]^{\frac{1}{2}p^{m-1}(p^{m-1}-1)}=(a^{p^{m-1}})^{\frac{1}{2}p^{m-1}(p^{m-1}-1)}.$
Since $a^{p^{m}}=1,$ we have $[b,a]^{\frac{1}{2}p^{m-1}(p^{m-1}-1)}=1$ and so
$\displaystyle\prod_{i=p^{m-1}-1}^{1}(b\wedge[b,a])^{i}=1.$ Thus

$b^{p^{m-1}}\wedge a=(b\wedge a)^{p^{m-1}}=1_{G\wedge G}.$

We will show that $a^{p^{m-1}}\wedge b=(a\wedge b)^{p^{m-1}}.$ Since

$a^{p^{m-1}}\wedge b=\prod_{i=p^{m-1}-1}^{0}(^{a^{i}}(a\wedge
b))=\prod_{i=p^{m-1}-1}^{0}(a\wedge[a^{i},b]b)=\prod_{i=p^{m-1}-1}^{0}(a\wedge
b)=(a\wedge b)^{p^{m-1}},$

we have $a^{p^{m-1}}\wedge b=(a\wedge b)^{p^{m-1}}=(b\wedge
a)^{-p^{m-1}}=(b^{p^{m-1}}\wedge a)^{-1}=1_{G\wedge G}.$ Moreover,
$a\wedge[a,b]=a\wedge a^{p^{m-1}}=1_{G\wedge G}.$ Thus we have $1\neq
a^{p^{m-1}}\in Z^{\wedge}(G)$ and so $Z^{\wedge}(G)=G^{\prime}.$ Hence $G$ is
non-capable. ∎

Let $F/R$ be a free presentation for a group $G$ and $\pi:\tilde{x}R\in
F/R\mapsto x\in G$ be a natural homomorphism and $e(X)$ is used to denote the
exponent of a group $X.$ The next result is extracted from the work of
Blackburn and Evens in [4, Remark, Section 3].

######  Theorem 1.3.

Let $G$ be a finite non-abelian $p$-group of class two and
$e(G/G^{\prime})=p^{s}$. Then

$1\rightarrow\ker g\rightarrow G^{\prime}\otimes
G/G^{\prime}\xrightarrow{g}\mathcal{M}(G)\rightarrow\mathcal{M}(G/G^{\prime})\rightarrow
G^{\prime}\rightarrow 1$

is exact, in where

$g:x\otimes(zG^{\prime})\in G^{\prime}\otimes
G/G^{\prime}\rightarrow[\tilde{x},\tilde{z}][R,F]\in\mathcal{M}(G)=(R\cap
F^{\prime})/[R,F],$

$\pi(\tilde{x}R)=x$ and $\pi(\tilde{z}R)=z.$ Moreover, $\langle([x,y]\otimes
zG^{\prime})([z,x]\otimes yG^{\prime})([y,z]\otimes
xG^{\prime}),w^{p^{s}}\otimes wG^{\prime}|x,y,z,w\in G\rangle\subseteq\ker g.$

###### Proof.

By [13, Corollary 3.2.4], we get

$1\rightarrow\ker g\rightarrow G^{\prime}\otimes
G/G^{\prime}\xrightarrow{g}\mathcal{M}(G)\rightarrow\mathcal{M}(G/G^{\prime})\rightarrow
G^{\prime}\rightarrow 1$

is exact, in where

$g:x\otimes(zG^{\prime})\in G^{\prime}\otimes
G/G^{\prime}\rightarrow[\tilde{x},\tilde{z}][R,F]\in\mathcal{M}(G)=(R\cap
F^{\prime})/[R,F],$

$\pi(\tilde{x}R)=x$ and $\pi(\tilde{z}R)=z.$ Similar to the proof of [4,
Theorem 3.1], we have $\langle([x,y]\otimes zG^{\prime})([z,x]\otimes
yG^{\prime})([y,z]\otimes xG^{\prime}),w^{p^{s}}\otimes wG^{\prime}|x,y,z,w\in
G\rangle\subseteq\ker g,$ as required. ∎

We need the following result for the next investigation.

######  Proposition 1.4.

[7, Proposition 1] and [5, 10] Let $G$ be a finite non-abelian $p$-group of
class $c.$

  * $(i)$

The map

$\Psi_{2}:(G/Z(G))^{(ab)}\otimes(G/Z(G))^{(ab)}\otimes(G/Z(G))^{(ab)}\rightarrow\big{(}G^{\prime}/\gamma_{3}(G)\big{)}\otimes
G/G^{\prime}$

given by $xG^{\prime}Z(G)\otimes yG^{\prime}Z(G)\otimes
zG^{\prime}Z(G)\mapsto$

$([x,y]\gamma_{3}(G)\otimes zG^{\prime})([z,x]\gamma_{3}(G)\otimes
yG^{\prime})([y,z]\gamma_{3}(G)\otimes xG^{\prime})$

is a homomorphism. If any two elements of the set $\\{x,y,z\\}$ are linearly
dependent. Then $\Psi_{2}(xG^{\prime}Z(G)\otimes yG^{\prime}Z(G)\otimes
zG^{\prime}Z(G))=1_{\big{(}G^{\prime}/\gamma_{3}(G)\big{)}\otimes
G/G^{\prime}}.$

  * $(ii)$

The map

$\displaystyle\Psi_{3}:$
$\displaystyle(G/Z(G))^{(ab)}\otimes(G/Z(G))^{(ab)}\otimes(G/Z(G))^{(ab)}\otimes(G/Z(G))^{(ab)}$
$\displaystyle\rightarrow\gamma_{3}(G)/\gamma_{4}(G)\otimes(G/Z(G))^{(ab)}~{}\text{given
by}~{}$
$\displaystyle\big{(}xG^{\prime}Z(G)\big{)}\otimes\big{(}yG^{\prime}Z(G)\big{)}\otimes\big{(}zG^{\prime}Z(G)\big{)}\otimes\big{(}wG^{\prime}Z(G)\big{)}$
$\displaystyle\mapsto\big{(}[[x,y],z]\gamma_{4}(G)\otimes
wG^{\prime}Z(G)\big{)}\big{(}[w,[x,y]]\gamma_{4}(G)\otimes
zG^{\prime}Z(G)\big{)}$ $\displaystyle\big{(}[[z,w],x]\gamma_{4}(G)\otimes
yG^{\prime}Z(G)\big{)}\big{(}[y,[z,w]]\gamma_{4}(G)\otimes
xG^{\prime}Z(G)\big{)}$

is a homomorphism.

Let $d(G)$ denote the minimal number of elements required to generate $G.$

######  Theorem 1.5.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ of class $c$ with
$|G^{\prime}|=p^{k}$ and $d=d(G).$ Then

$\displaystyle|G\wedge G||\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|$
$\displaystyle\leq|G\wedge
G|\prod_{i=2}^{c}|\ker\alpha_{i}|=|\mathcal{M}(G)||G^{\prime}|\prod_{i=2}^{c}|\ker\alpha_{i}|$
$\displaystyle=|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}|\leq|\mathcal{M}(G/G^{\prime})|p^{kd},$

where $\alpha_{i}:x\wedge z\in\gamma_{i}(G)\wedge G\mapsto x\wedge
z\in\gamma_{i-1}(G)\wedge G$ is a natural homomorphism for all $i\geq 2.$

###### Proof.

Similar to the proof of [5, Proposition 5], we have

$|G\wedge G|\prod_{i=2}^{c}|\ker\alpha_{i}|=|G/G^{\prime}\wedge
G/G^{\prime}|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}|.$

Since $|G\wedge G|=|\mathcal{M}(G)||G^{\prime}|$ and using the proof of [17,
Theorem 1.2], we have

$\displaystyle|G\wedge
G|\prod_{i=2}^{c}|\ker\alpha_{i}|=|\mathcal{M}(G)||G^{\prime}|\prod_{i=2}^{c}|\ker\alpha_{i}|$
$\displaystyle=|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}|\leq|\mathcal{M}(G/G^{\prime})|p^{kd}.$

We claim that
$|\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|\leq\prod_{i=2}^{c}|\ker\alpha_{i}|.$
Clearly, the map $\delta_{i}:\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}\rightarrow\gamma_{i}(G)/\gamma_{i+1}(G)\otimes G/Z(G)G^{\prime}$
given by $x\gamma_{i+1}(G)\otimes yG^{\prime}\mapsto x\gamma_{i+1}(G)\otimes
yG^{\prime}Z(G)$ is a natural epimorphism for all $i\geq 2.$ By [19,
Proposition 2.1], we have

$\displaystyle|\mathcal{M}(G)||G^{\prime}||\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|$
$\displaystyle\leq|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}Z(G)|.$

Thus

$\displaystyle|\mathcal{M}(G)||G^{\prime}||\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|\leq|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}Z(G)|$
$\displaystyle=|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}|/|\ker\delta_{i}|=|\mathcal{M}(G)||G^{\prime}|\prod_{i=2}^{c}|\ker\alpha_{i}|/|\ker\delta_{i}|$

and so
$|\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|\leq\prod_{i=2}^{c}|\ker\alpha_{i}|/|\ker\delta_{i}|\leq\prod_{i=2}^{c}|\ker\alpha_{i}|.$
Therefore

$\displaystyle|G\wedge G||\mathrm{Im}\Psi_{2}||\mathrm{Im}\Psi_{3}|$
$\displaystyle\leq|G\wedge
G|\prod_{i=2}^{c}|\ker\alpha_{i}|=|\mathcal{M}(G)||G^{\prime}|\prod_{i=2}^{c}|\ker\alpha_{i}|$
$\displaystyle=|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime}|\leq|\mathcal{M}(G/G^{\prime})|p^{kd}.$

The proof is completed. ∎

We need the following lemma.

######  Lemma 1.6.

Let $G$ be a group of nilpotency class two such that $d(G/Z(G))=d$ is finite.
Then $d(G^{\prime})\leq\frac{1}{2}d(d-1).$

###### Proof.

We can choose a generating set $\\{x_{1}Z(G),\ldots,x_{d}Z(G)\\}$ for $G/Z(G)$
such that $[x_{i},x_{j}]$ is non-trivial for $i\neq j.$ It is clear to see
that $\\{[x_{i},x_{j}]|1\leq i<j\leq d\\}$ generates $G^{\prime},$ as
required. ∎

######  Lemma 1.7.

Let $G$ be a minimal non-abelian $p$-group. If $G$ is isomorphic to $\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ and $m>1,$
then
$\mathcal{M}(G)\cong\mathbb{Z}_{p^{m-1}}\times\mathbb{Z}_{p}\times\mathbb{Z}_{p}.$

###### Proof.

It is clearly obtained by [14, Theorems 49 and 50]. ∎

##  2\. main results

According to [17], the Schur multiplier of a non-abelian finite $p$-group of
order $p^{n}$ with $|G^{\prime}|=p^{k}$ and $d(G)=d$ is bounded by
$p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ The main result of this paper is devoted to
characterize the structure of all finite $p$-groups that attain the mentioned
upper bound. Moreover, we show that all $p$-groups that attain the bound are
capable. Throughout the paper, we say that $|\mathcal{M}(G)|$ attains the
bound provided that $|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$

We begin with the following lemma for the future convenience.

######  Lemma 2.1.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ such that
$|G^{\prime}|=p^{k},$ $d(G)=d,$ and
$G/G^{\prime}\cong\mathbb{Z}_{p^{\alpha_{1}}}\oplus\ldots\oplus\mathbb{Z}_{p^{\alpha_{d}}}$,
where $\alpha_{1}\geq\ldots\geq\alpha_{d}.$ If $|\mathcal{M}(G)|$ attains the
bound. Then

  * $(i)$

$G/G^{\prime}$ is homocyclic.

  * $(ii)$

$2\leq d\leq 3,$ when $k\geq 2.$

###### Proof.

  * $(i)$

By using [17, Corollary 1.3], we have

$\displaystyle|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}\leq
p^{\frac{1}{2}(d-1)(n+k-2-\alpha_{1}-\alpha_{d})+1}.$

Thus $\alpha_{1}=\alpha_{d}$ and so $G/G^{\prime}$ is homocyclic.

  * $(ii)$

By part $(i),$ $G/G^{\prime}$ is homocyclic and so $n=d\alpha_{1}+k.$ Theorem
1.5 and [13, Corollary 2.2.12] imply

$\displaystyle
p^{\frac{1}{2}(d-1)(n+k-2)+1}p^{k}\prod_{i=2}^{c}|\ker\alpha_{i}|=p^{\frac{1}{2}(d-1)(d\alpha_{1}+2k-2)+1}p^{k}\prod_{i=2}^{c}|\ker\alpha_{i}|$
$\displaystyle=|\mathcal{M}(G)||G^{\prime}|\prod_{i=2}^{c}|\ker\alpha_{i}|=|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{c}|(\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime})|$ $\displaystyle\leq p^{\frac{1}{2}d(d-1)\alpha_{1}}p^{kd}.$

Thus $\prod_{i=2}^{c}|\ker\alpha_{i}|\leq p^{d-2}.$ By contrary, let
$G=\langle x_{1},x_{2},\ldots,x_{d}\rangle$ such that $d\geq 4.$ We may assume
that $[x_{1},x_{2}]\gamma_{3}(G)$ and $[x_{1},x_{3}]\gamma_{3}(G)$ are non-
trivial elements in $G^{\prime}/\gamma_{3}(G).$ Thus all elements of sets
$\\{\Psi_{2}(x_{1}G^{\prime}\otimes x_{2}G^{\prime}\otimes
x_{j}G^{\prime})|3\leq j\leq d\\}$ and $\\{\Psi_{2}(x_{1}G^{\prime}\otimes
x_{3}G^{\prime}\otimes x_{j}G^{\prime})|4\leq j\leq d\\}$ are non-trivial.
Clearly, $|\text{Im}(\Psi_{2})|\geq p^{d-2+d-3}.$ Therefore Theorem 1.5
implies

$\begin{array}[]{lcl}p^{d-2+d-3}\leq|\text{Im}\Psi_{2}|\leq\prod_{i=2}^{c}|\ker\alpha_{i}|\leq
p^{d-2}.\end{array}$

We have a contradiction. Thus $d(G)\leq 3.$ Clearly, $2\leq d(G),$ as
required.

∎

######  Proposition 2.2.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ such that
$|G^{\prime}|=p$ and $d(G)=d.$ If $e(G/G^{\prime})>p,$ then $|\mathcal{M}(G)|$
attains the bound if and only if $G\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ where
$m>1.$

###### Proof.

Assume that $|\mathcal{M}(G)|$ attains the bound. If $|G/Z(G)|>p^{2},$ then
$G$ is non-capable, by [11, Theorem C]. Now, Lemma 2.1 $(i)$ implies
$G/G^{\prime}$ is homocyclic of exponent $\alpha_{1}$ for $\alpha_{1}\geq 2.$
Therefore [3, Corollary 7.4] implies $G/G^{\prime}$ is capable, and so by
using [3, Corollary 2.2], we have $1\neq Z^{*}(G)\subseteq
G^{\prime}\cong\mathbb{Z}_{p}.$ Hence $Z^{*}(G)=G^{\prime}.$ By using Lemma
[3, Theorem 4.2] and [13, Theorem 2.5.6 (i)], we have
$|\mathcal{M}(G)|=|\mathcal{M}(G/G^{\prime})|p^{-1}.$ Using [13, Corollary
2.2.12], we have
$|\mathcal{M}(G)|=|\mathcal{M}(G/G^{\prime})|p^{-1}=p^{\frac{1}{2}d(d-1)\alpha_{1}-1}.$
Since $n=\alpha_{1}d+1,$ we have
$|\mathcal{M}(G)|=p^{\frac{1}{2}d(d-1)\alpha_{1}+1}.$ Thus we have a
contradiction. Therefore $|G/Z(G)|=p^{2}.$ Using [2, Lemma 4.2], we have
$G=AZ(G),$ where $A$ is minimal non-abelian. Since $G/G^{\prime}$ is
homocyclic, we have $G/G^{\prime}\cong\mathbb{Z}_{p^{\alpha_{1}}}^{(d)}$ and
$G/Z(G)\cong\mathbb{Z}_{p}\oplus\mathbb{Z}_{p}.$ We claim that $G=A.$ Since
$G=AZ(G),$ we have $G^{\prime}=A^{\prime}=\langle c\rangle$ such that
$A=\langle a,b\rangle$ and $c=[a,b].$ If $Z(G)\subseteq A,$ then $G=A.$ Now
let $Z(G)\nsubseteq A.$ Choose an element $y\in Z(G)\setminus A.$ Thus
$c\otimes yG^{\prime}$ is non-trivial in $G^{\prime}\otimes G/G^{\prime}.$ By
Theorem 1.3, we can see that $c\otimes yG^{\prime}$ is non-trivial in $\ker g$
and so

$\displaystyle|\mathcal{M}(G)|p^{2}\leq|\mathcal{M}(G)||G^{\prime}||\ker
g|=|(G/G^{\prime}\wedge G/G^{\prime})||(G^{\prime}\otimes G/G^{\prime})|$
$\displaystyle\leq p^{\frac{1}{2}d(d-1)\alpha_{1}}p^{d}.$

Thus $|\mathcal{M}(G)|\leq p^{\frac{1}{2}d(d-1)\alpha_{1}+d-2}.$ Now, since
$n=\alpha_{1}d+1,$ by our assumption we have
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+1-2)+1}=p^{\frac{1}{2}d(d-1)\alpha_{1}+1}.$
Hence we have a contradiction. Thus $Z(G)\subseteq A$ and so $G=AZ(G)=A.$ Now,
$A/G^{\prime}$ is homocyclic and $e(G/G^{\prime})\geq p^{2}.$ Therefore $G$
should be isomorphic to one of three groups is given in the Lemma 1.1. First
assume that $G\cong\langle
a,b|a^{p^{m}}=1,b^{p^{m-1}}=1,[a,b]=a^{p^{m-1}},[a,b,a]=[a,b,b]=1\rangle$ and
$m\geq 2.$ If $p$ is odd or $p=2$ and $m\geq 3,$ then Proposition 1.2 implies
$Z^{*}(G)=G^{\prime}$ ($G$ is non-capable). By using Lemma [3, Theorem 4.2]
and [13, Theorem 2.5.6 (i)], we have
$|\mathcal{M}(G)|=|\mathcal{M}(G/G^{\prime})|p^{-1}.$ Now [13, Corollary
2.2.12] implies
$|\mathcal{M}(G)|=|\mathcal{M}(G/G^{\prime})|p^{-1}=p^{\frac{1}{2}2(2-1)(m-1)-1}.$
Since $n=2(m-1)+1,$ we have $|\mathcal{M}(G)|=p^{\frac{1}{2}2(2-1)(m-1)+1}.$
Thus we get a contradiction. If $p=2$ and $m=2,$ then by our assumption $G$
cannot be isomorphic to $D_{8}\cong\langle
a,b|a^{4}=1,b^{2}=1,[a,b]=a^{2},[a,b,a]=[a,b,b]=1\rangle.$ Therefore $G$ is
isomorphic to $Q_{8}$ or $\langle
a,b|a^{p^{\alpha_{1}}}=b^{p^{\alpha_{1}}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$
where $m>1$. By our assumption, $G$ cannot be isomorphic to $Q_{8}$. Hence the
result follows. The converse holds by Lemma 1.7. ∎

######  Proposition 2.3.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ with
$|G^{\prime}|=p^{k},$ $d(G)=d$ and
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ If $k\geq 2$ and $K$ is a
central subgroup of order $p$ contained in $Z(G)\cap G^{\prime},$ then
$|\mathcal{M}(G/K)|$ attains the bound, that means

$|\mathcal{M}(G/K)|=p^{\frac{1}{2}(d-1)(n+k-4)+1}.$

###### Proof.

Let $K$ be a central subgroup of order $p$ contained in $Z(G)\cap G^{\prime}.$
Lemma 2.1 $(i)$ implies $G/G^{\prime}\cong\mathbb{Z}^{(d)}_{p^{m}},$ and so
$(G/G^{\prime}\otimes K)\cong\mathbb{Z}^{(d)}_{p}.$ Using [12, Theorem 4.1],
we have

$\displaystyle|\mathcal{M}(G)||(G^{\prime}\cap K)|$
$\displaystyle\leq|\mathcal{M}(G/K)||\mathcal{M}(K)||(G/G^{\prime}\otimes K)|$
$\displaystyle=|\mathcal{M}(G/K)||(G/G^{\prime}\otimes K)|.$

Thus

$\displaystyle|\mathcal{M}(G)|$
$\displaystyle\leq|\mathcal{M}(G/K)||(G/G^{\prime}\otimes K)|p^{-1}\leq
p^{\frac{1}{2}(d-1)(n+k-4)+1+d-1}$ $\displaystyle=$ $\displaystyle
p^{\frac{1}{2}(d-1)(n+k-2)+1}=|\mathcal{M}(G)|.$

Therefore $|\mathcal{M}(G/K)|=p^{\frac{1}{2}(d-1)(n+k-4)+1},$ as required. ∎

In the next, we show if the order of Schur multiplier of a non-abelian
$p$-group $G$ attains the bound, then $G$ is capable.

######  Lemma 2.4.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ with
$|G^{\prime}|=p^{k}$ and $d(G)=d.$ If $|\mathcal{M}(G)|$ attains the bound,
then $G$ is capable.

###### Proof.

Lemma 2.1 $(i)$ implies $G/G^{\prime}$ is homocyclic. Therefore $G/G^{\prime}$
is capable, by [3, Corollary 7.4] and hence [3, Corollary 2.2] implies
$Z^{*}(G)\subseteq G^{\prime}.$ By contrary, let $G$ be non-capable. Then
there is a normal subgroup $K$ of order $p$ in $Z^{*}(G).$ By using [3,
Theorem 4.2] and [13, Theorem 2.5.6 (i)], we have
$|\mathcal{M}(G)|=|\mathcal{M}(G/K)|p^{-1}.$ Proposition 2.3 implies that
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-4)}$ which is a contradiction, since
by our assumption, we have $|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$
Thus $Z^{*}(G)=1$ and so the result follows from [3, Corollary 2.3]. ∎

######  Proposition 2.5.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ and the nilpotency
class $t$ with $|G^{\prime}|=p^{k},$ $d(G)=d$ and
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ If $k\geq 2$ and $K$ is a
non-trivial central subgroup of order $p^{m}$ contained in $Z(G)\cap
G^{\prime}.$ Then $|\mathcal{M}(G/K)|$ attains the bound, that means
$|\mathcal{M}(G/K)|=p^{\frac{1}{2}(d-1)(n+k-2(m+1))+1}.$

###### Proof.

Let $K$ be a non-trivial central subgroup of order $p^{m}$ contained in
$Z(G)\cap G^{\prime}.$ We have $|G/K|=p^{n-m}$ and $|(G/K)^{\prime}|=p^{k-m}.$
We prove the result by using induction on $m.$ If $m=1,$ then the result holds
by Proposition 2.3. Now let $m\geq 2.$ Consider a normal subgroup $K_{1}$ in
$K$ of order $p^{m-1}$ and using the induction hypothesis, we have

$|\mathcal{M}(G/K_{1})|=p^{\frac{1}{2}(d-1)(n+k-2m)+1}.$

Since $K/K_{1}$ is a normal subgroup of order $p$ in
$Z(G/K_{1})\cap(G^{\prime}/K_{1}),$ Proposition 2.3 implies that

$|\mathcal{M}(G/K)|=|\mathcal{M}(\dfrac{G/K_{1}}{K/K_{1}})|=p^{\frac{1}{2}(d-1)(n+k-2(m+1))+1}.$

And this completes the proof. ∎

The proof of the following corollary is similar to [19, Theorem 1.2]

######  Corollary 2.6.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ and the nilpotency
class $t\geq 3$ with $|G^{\prime}|=p^{k},$ $d(G)=d$ and
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ Then
$|\mathcal{M}(G/\gamma_{i}(G))|$ attains the bound for all $3\leq i\leq t.$

###### Proof.

By Proposition 2.5, we have $|\mathcal{M}(G/\gamma_{t}(G))|$ attains the
bound. We prove induction on $j=t-i+3$ for $3\leq i\leq t.$ If $j=3,$ then
since $k\geq 2,$ by using Proposition 2.5, $|\mathcal{M}(G/\gamma_{t}(G))|$
attains the bound. Using the induction hypothesis,
$|\mathcal{M}(G/\gamma_{i}(G))|$ attains the bound. Since
$\gamma_{i-1}(G)/\gamma_{i}(G)\subseteq Z(G/\gamma_{i}(G))\cap
G^{\prime}/\gamma_{i}(G),$ Proposition 2.5 implies that

$|\mathcal{M}(G/\gamma_{i-1}(G))|=|\mathcal{M}(G/\gamma_{i}(G)/\gamma_{i-1}(G)/\gamma_{i}(G))|$

attains the bound, as required. ∎

######  Lemma 2.7.

Let $G$ be a finite $p$-group of class two. Then $e(G/Z(G))=e(G^{\prime}).$

###### Proof.

Straightforward. ∎

######  Proposition 2.8.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ such that
$|G^{\prime}|\geq p^{2},$ $G/G^{\prime}\cong\mathbb{Z}_{p^{m}}^{(2)}$ and
$G^{\prime}=Z(G)\cong\mathbb{Z}_{p^{k}}$ for $k\geq 2.$ Then
$|\mathcal{M}(G)|$ attains the bound if and only if $G\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p^{m}}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ where
$m\geq 2$ and $p\neq 2.$

###### Proof.

Let $|\mathcal{M}(G)|$ attain the bound. Lemma 2.7 implies
$e(G/Z(G))=e(G^{\prime})=p^{m},$ and so $k=m$ and $d(G)=d(G/G^{\prime})=2.$ By
Lemma 2.4, $G$ is capable. Now by [14, Theorems 50 and 67] $G$ cannot be a
$2$-group. Thus $p>2.$ We may assume that $G=\langle x_{1},x_{2}\rangle$ such
that $x_{1}^{p^{\alpha_{1}}}=x_{2}^{p^{\alpha_{2}}}=1$ for positive integers
$\alpha_{1}$ and $\alpha_{2}.$ First we show that $m\leq\alpha_{1}$ and
$m\leq\alpha_{2}.$ Since
$[x_{1}^{p^{\alpha_{1}}},x_{2}]=[x_{1},x_{2}]^{p^{\alpha_{1}}}=1=[x_{1},x_{2}^{p^{\alpha_{2}}}]=[x_{1},x_{2}]^{p^{\alpha_{2}}}$
and $G^{\prime}=\langle[x_{1},x_{2}]\rangle,$ $e(G^{\prime})=p^{m},$ we have
$m\leq\alpha_{1}$ and $m\leq\alpha_{2}.$ We claim that
$\alpha_{1}=\alpha_{2}.$ By contrary, let $\alpha_{1}\gneqq\alpha_{2}.$ Now we
show that

$x_{1}^{p^{\alpha_{1}-1}}\wedge x_{2}=x_{1}\wedge
x_{2}^{p^{\alpha_{1}-1}}=1_{G\wedge G}.$

Since $\alpha_{1}\gneqq\alpha_{2},$ we have $\alpha_{1}-1\geq\alpha_{2}$ and
so $x_{2}^{p^{\alpha_{1}-1}}=1.$ Thus
$[x_{1}^{p^{\alpha_{1}-1}},x_{2}]=[x_{1},x_{2}^{p^{\alpha_{1}-1}}]=1$ and
hence $x_{1}^{p^{\alpha_{1}-1}}\in Z(G)$ and $x_{2}^{p^{\alpha_{1}-1}}=1.$
Thus

$\displaystyle x_{1}^{p^{\alpha_{1}-1}}\wedge x_{2}$
$\displaystyle=\prod_{i=p^{\alpha_{1}-1}-1}^{0}(^{x_{1}^{i}}(x_{1}\wedge
x_{2}))=\prod_{i=p^{\alpha_{1}-1}-1}^{0}(x_{1}\wedge[x_{1}^{i},x_{2}]x_{2})$
$\displaystyle=\prod_{i=p^{\alpha_{1}-1}-1}^{1}(x_{1}\wedge[x_{1},x_{2}])^{i}\prod_{i=p^{\alpha_{1}-1}-1}^{0}(x_{1}\wedge
x_{2}).$

Clearly,

$\displaystyle\prod_{i=p^{\alpha_{1}-1}-1}^{1}(x_{1}\wedge[x_{1},x_{2}])^{i}$
$\displaystyle=(x_{1}\wedge[x_{1},x_{2}])^{\frac{1}{2}p^{\alpha_{1}-1}(p^{\alpha_{1}-1}-1)}$
$\displaystyle=(x_{1}\wedge[x_{1},x_{2}]^{\frac{1}{2}p^{\alpha_{1}-1}(p^{\alpha_{1}-1}-1)})$
$\displaystyle=(x_{1}\wedge[x_{1},x_{2}^{\frac{1}{2}p^{\alpha_{1}-1}(p^{\alpha_{1}-1}-1)}])=1_{G\wedge
G}.$

Thus

$x_{1}^{p^{\alpha_{1}-1}}\wedge x_{2}=(x_{1}\wedge x_{2})^{p^{\alpha_{1}-1}}.$

Also since $x_{2}^{p^{\alpha_{1}-1}}=1,$ we have

$\displaystyle 1_{G\wedge G}$ $\displaystyle=x_{2}^{p^{\alpha_{1}-1}}\wedge
x_{1}=\prod_{i=p^{\alpha_{1}-1}-1}^{0}(^{x_{2}^{i}}(x_{2}\wedge x_{1}))=$
$\displaystyle\prod_{i=p^{\alpha_{1}-1}-1}^{0}(x_{2}\wedge[x_{1},x_{2}^{i}]x_{1})=\prod_{i=p^{\alpha_{1}-1}-1}^{0}(x_{2}\wedge
x_{1})=(x_{2}\wedge x_{1})^{p^{\alpha_{1}-1}}.$

Thus $(x_{2}\wedge x_{1})^{p^{\alpha_{1}-1}}=1_{G\wedge G}$ and so
$x_{1}^{p^{\alpha_{1}-1}}\wedge x_{2}=(x_{1}\wedge
x_{2})^{p^{\alpha_{1}-1}}=(x_{2}\wedge x_{1})^{-p^{\alpha_{1}-1}}=1_{G\wedge
G}.$ Hence, we have $1\neq x_{1}^{p^{\alpha_{1}-1}}\in Z^{\wedge}(G)$ which is
a contradiction since $G$ is capable. Thus $\alpha_{1}=\alpha_{2}.$
Put $c=[x_{1},x_{2}],$ since $x_{1}^{p^{m}},x_{2}^{p^{m}}\in
Z(G)=G^{\prime}=\langle[x_{1},x_{2}]\rangle,$ we get
$x_{1}^{p^{m}}=c^{p^{\sigma_{1}}}$ and $x_{2}^{p^{m}}=c^{p^{\sigma_{2}}}$ for
positive integers $\sigma_{1},\sigma_{2}.$ We claim that
$\alpha_{1}=\alpha_{2}=m.$ It is enough to show that $\sigma_{1}\geq m$ or
$\sigma_{2}\geq m.$ By contrary, let $\sigma_{1}\lneqq m$ and
$\sigma_{2}\lneqq m.$ First let $\sigma_{1}=0.$ Then $x_{1}^{p^{m}}=c$ and so
$x_{2}^{p^{m}}=c^{p^{\sigma_{2}}}=x_{1}^{p^{m+\sigma_{2}}}.$ Moreover, $1\neq
x_{2}^{p^{m}},$ otherwise $x_{2}^{p^{m}}=c^{p^{\sigma_{2}}}=1$ and so
$\sigma_{2}<m.$ Therefore $|G^{\prime}|<p^{m}$ which is impossible. Since
$x_{2}\wedge x_{2}^{{p^{m}}}=1_{G\wedge G}$ and $x_{1}\wedge
x_{2}^{p^{m}}=x_{1}\wedge x_{1}^{{p^{m+\sigma_{2}}}}=1_{G\wedge G},$ we have
$1\neq x_{2}^{p^{m}}\in Z^{\wedge}(G)$ and so $G$ is non-capable. It is a
contradiction. Thus $\sigma_{1}\neq 0.$ Similarly $\sigma_{2}\neq 0.$ We may
assume that $1\leq\sigma_{1}\leq\sigma_{2}.$ It follows that
$x_{2}^{p^{m}}\in\langle x_{1}^{p^{m}}\rangle$ so
$x_{2}^{p^{m}}=(x_{1}^{p^{m}})^{t}$ for some $t\geq 0.$ If $t=0,$ then
$x_{2}^{p^{m}}=c^{p^{\sigma_{2}}}=1$ and so $\sigma_{2}<m.$ Therefore
$|G^{\prime}|<p^{m}$ which is impossible. Therefore $t\geq 1$ and hence
$x_{1}\wedge x_{2}^{{p^{m}}}=x_{1}\wedge x_{1}^{tp^{m}}=1_{G\wedge
G}=x_{2}\wedge x_{2}^{p^{m}}.$ Thus $x_{1}^{tp^{m}}\in Z^{\wedge}(G),$ which
is a contradiction. Thus $\sigma_{1}\geq m$ or $\sigma_{2}\geq m.$ Hence
$\alpha_{1}=\alpha_{2}=m.$ Therefore $G\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p^{m}}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle.$ Since
$|G^{\prime}|\geq p^{2},$ we have $2\leq m.$ The converse holds by using [14,
Theorem 49]. ∎

Now we are ready to obtain the structure of a $p$-group $G$ of class two such
that $|\mathcal{M}(G)|$ attains the bound.

######  Theorem 2.9.

[18, Theorem 1.1] Let $G$ be a non-abelian finite $p$-group of order $p^{n}$
the nilpotency class $2$ and $|G^{\prime}|=p^{k}.$ Then
$|\mathcal{M}(G)|=p^{\frac{1}{2}(n-k-1)(n+k-2)+1}$ if and only if $G$ is
isomorphic to one of the following groups:

  * $(i)$

$G_{1}\cong E_{1}\times\mathbb{Z}_{p}^{(n-3)},$ where $E_{1}$ is the extra-
special $p$-group of order $p^{3}$ and exponent $p,$ $(p\neq 2).$

  * $(ii)$

$G_{2}\cong\mathbb{Z}_{p}\rtimes\mathbb{Z}_{p}^{(4)},$ $(p\neq 2).$

  * $(iii)$

$G_{3}\cong\langle
x_{1},y_{1},x_{2},y_{2},x_{3},y_{3}|[x_{1},x_{2}]=y_{3},[x_{2},x_{3}]=y_{1},[x_{3},x_{1}]=y_{2},[x_{i},y_{j}]=1,x_{i}^{p}=y_{i}^{p}=1,1\leq
i,j\leq 3\rangle,$ $(p\neq 2).$

######  Theorem 2.10.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ of class two such
that $|G^{\prime}|=p^{k}$ and $d(G)=d.$ Then $|\mathcal{M}(G)|$ attains the
bound if and only if $G$ is isomorphic to one of the following groups:

  * $(i)$

$G_{1}\cong E_{1}\times\mathbb{Z}_{p}^{(n-3)},$ where $E_{1}$ is the extra-
special $p$-group of order $p^{3}$ and exponent $p,$ $(p\neq 2).$

  * $(ii)$

$G_{2}\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle$ is non-
metacyclic of order $p^{2m+1}$ and $m\geq 2.$

  * $(iii)$

$G_{3}\cong\mathbb{Z}_{p}\rtimes\mathbb{Z}_{p}^{(4)},$ $(p\neq 2).$

  * $(iv)$

$G_{4}\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p^{m}}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ of
order $p^{3m}$ in where $m\geq 2,$ and $(p\neq 2).$

  * $(v)$

$G_{5}\cong\langle
x_{1},y_{1},x_{2},y_{2},x_{3},y_{3}|[x_{1},x_{2}]=y_{3},[x_{2},x_{3}]=y_{1},[x_{3},x_{1}]=y_{2},[x_{i},y_{j}]=[y_{j},y_{t}]=1,x_{i}^{p}=y_{i}^{p}=1,1\leq
i,j,t\leq 3\rangle$ and $p\neq 2.$

###### Proof.

Suppose that $|\mathcal{M}(G)|$ attains the bound. First assume that
$|G^{\prime}|=p.$ By Lemma 2.1 $(i),$ $G/G^{\prime}$ is homocyclic. Let
$G/G^{\prime}$ be elementary abelian. Then $d(G)=n-k.$ Theorem 2.9 $(i)$
implies $G\cong E_{1}\times\mathbb{Z}_{p}^{(n-3)},$ where $E_{1}$ is the
extra-special $p$-group of order $p^{3}$ and exponent $p$ for an odd prime
$p.$ Thus $G\cong G_{1}.$ Now assume $G/G^{\prime}$ is not elementary abelian.
Proposition 2.2 implies that $G\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ in where
$m\geq 2.$ Hence $G\cong G_{2}.$ Now suppose that $|G^{\prime}|\geq p^{2}.$ By
using Lemma 2.1 $(ii),$ we have $2\leq d(G)\leq 3.$ Let $d(G)=2.$ Thus
$d(G/Z(G))=2$. Since $G$ is nilpotent of class two, we have
$G^{\prime}\subseteq Z(G).$ We claim that $Z(G)=G^{\prime}.$ By contrary, let
$Z(G)\neq G^{\prime}.$ Since $d(G/G^{\prime})=2$ and $G/G^{\prime}$ is
homocyclic by Lemma 2.1 $(i),$ we have
$G/G^{\prime}\cong\mathbb{Z}_{p^{\alpha}}^{(2)}$ where
$\alpha=\frac{1}{2}(n-k).$ Since $d(G/Z(G))=2,$ $G^{\prime}$ is cyclic, by
using Lemma 1.6. Hence $G^{\prime}\cong\mathbb{Z}_{p^{k}}.$ Now Lemma 2.7
implies that $e(G/Z(G))=e(G^{\prime})=p^{k}.$ Let $G=\langle
x_{1},x_{2}\rangle$ and $G^{\prime}=\langle[x_{1},x_{2}]\rangle.$ Choose an
element $y\in Z(G)\setminus G^{\prime}.$ Thus $[x_{1},x_{2}]\otimes
yG^{\prime}$ is a non-trivial element in $G^{\prime}\otimes G/G^{\prime}.$ By
using Theorem 1.3, we can see that $[x_{1},x_{2}]\otimes yG^{\prime}$ is a
non-trivial element in $\ker g$ and so

$\begin{array}[]{lcl}|\mathcal{M}(G)|p^{k+1}\leq|\mathcal{M}(G)||G^{\prime}||\ker
g|=\\\ |(G/G^{\prime}\wedge G/G^{\prime})||(G^{\prime}\otimes
G/G^{\prime})|\leq
p^{\frac{1}{2}2(2-1)\alpha}p^{2k}=p^{\alpha+2k}.\end{array}$

Thus $|\mathcal{M}(G)|\leq p^{\alpha+k-1}.$ On the other hand, by our
assumption $|\mathcal{M}(G)|=p^{\alpha+k}.$ Therefore we have a contradiction.
Thus $Z(G)=G^{\prime}.$ Since $e(G^{\prime})=e(G/Z(G))=p^{\alpha},$ we have
$|G^{\prime}|=p^{\alpha}.$ So $\alpha=k\geq 2.$ By Proposition 2.8, we have
$G\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p^{m}}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ $2\leq
m,$ and $p\neq 2.$ Thus $G\cong G_{4}.$
Let now $d(G)=3.$ Since $G/G^{\prime}$ has $3$-generators and homocyclic by
Lemma 2.1 $(i)$, we have $G/G^{\prime}\cong\mathbb{Z}_{p^{\alpha_{1}}}^{(3)}$
where $\alpha_{1}=1/3(n-k).$ We know that $G^{\prime}\subseteq Z(G).$ We claim
that $Z(G)=G^{\prime}.$ By contrary, let $Z(G)\neq G^{\prime}.$ We can choose
an element $y\in Z(G)\setminus G^{\prime}.$ Assume that $G=\langle
x_{1},x_{2},x_{3}\rangle$ such that $[x_{1},x_{2}]\neq 1.$ Theorem 1.3 implies
$[x_{1},x_{2}]\otimes yG^{\prime}$ and $([x_{1},x_{2}]\otimes
x_{3}G^{\prime})([x_{2},x_{3}]\otimes x_{1}G^{\prime})([x_{3},x_{1}]\otimes
x_{2}G^{\prime})$ are non-trivial elements in $\ker g$ and hence

$\begin{array}[]{lcl}|\mathcal{M}(G)|p^{k+2}\leq|\mathcal{M}(G)||G^{\prime}||\ker
g|\leq|(G/G^{\prime}\wedge G/G^{\prime})||(G^{\prime}\otimes G/G^{\prime})|\\\
\quad\quad\quad\quad\quad\quad\quad\quad\leq
p^{\frac{1}{2}3(3-1)\alpha_{1}}p^{3k}=p^{3\alpha_{1}+3k}.\end{array}$

Thus $|\mathcal{M}(G)|\leq p^{3\alpha_{1}+3k-k-2}=p^{3\alpha_{1}+2k-2}.$ On
the other hand, by our assumption we have
$|\mathcal{M}(G)|=p^{3\alpha_{1}+2k-1},$ so we have a contradiction. Hence
$Z(G)=G^{\prime}.$ Since $d(G/Z(G))=3,$ Lemmas 1.6 and 2.7 imply
$d(G^{\prime})\leq 3$ and $e(G/Z(G))=e(G^{\prime})=\alpha_{1}.$ Now since
$|\mathcal{M}(G)|=p^{3\alpha_{1}+2k-1},$ we have

$\displaystyle|\ker g|=|(G/G^{\prime}\wedge G/G^{\prime})||(G^{\prime}\otimes
G/G^{\prime})|/|\mathcal{M}(G)||G^{\prime}|$ $\displaystyle\leq
p^{\frac{1}{2}3(3-1)\alpha_{1}}p^{3k}p^{-(3\alpha_{1}+2k-1)}p^{-k}=p$

We claim that $e(G/G^{\prime})=p.$ By contrary, let $x_{1}^{p^{t}}\in
G^{\prime}$ for some $t\geq 2.$ By Theorem 1.3, we have $x_{1}^{p^{t}}\otimes
x_{1}G^{\prime}$ and $([x_{1},x_{2}]\otimes
x_{3}G^{\prime})([x_{2},x_{3}]\otimes x_{1}G^{\prime})([x_{3},x_{1}]\otimes
x_{2}G^{\prime})$ are non-trivial elements in $\ker g.$ Therefore
$p^{2}\leq|\ker g|\leq p$ which is a contradiction Thus $G/G^{\prime}$ is of
exponent $p$ and hence $G^{\prime}=\phi(G)$ and so $d(G)=n-k=3.$ Therefore
$|\mathcal{M}(G)|=p^{\frac{1}{2}(n-k-1)(n+k-2)+1}.$ Using Theorem 2.9, we have
$G\cong G_{3}$ or $G\cong G_{5}.$ The converse follows from Propositions 2.2,
2.8 and Theorem 2.9. The proof is completed. ∎

######  Lemma 2.11.

Let $G$ be isomorphic to $G_{2}$ or $G_{4}.$ Then $Z(G_{2})=\langle
a^{p}\rangle\times\langle b^{p}\rangle\times G_{2}^{\prime}$ and
$Z(G_{4})=G_{4}^{\prime}.$

###### Proof.

The result follows from [14, page 28]. ∎

######  Proposition 2.12.

There exists no non-abelian finite $2$-generators $p$-group $G$ of order
$p^{n}$ of class $t\geq 3$ such that $|G^{\prime}|\geq p^{k}$ for $k\geq 2$
and $|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$

###### Proof.

We prove the result by using induction on $t.$ First we assume that $t=3.$ By
contrary, let there be a group $G$ such that
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ By Lemma 2.1 $(i),$ we have
$G/G^{\prime}\cong\mathbb{Z}_{p^{m}}^{(2)}$ for some $m>0.$ We claim that
$Z(G)\subseteq G^{\prime}.$ Let $G=\langle x,y\rangle$ and $Z(G)\nsubseteq
G^{\prime}.$ Choose an element $g\in Z(G)\setminus G^{\prime}.$ Thus
$[x,y]\gamma_{3}(G)\otimes gG^{\prime}$ is non-trivial in
$G^{\prime}/\gamma_{3}(G)\otimes G/G^{\prime}.$ Using Theorem 1.5, we can see
that $[x,y]\gamma_{3}(G)\otimes gG^{\prime}$ is non-trivial in
$\mathrm{Im}\Psi_{2}$ and so

$\begin{array}[]{lcl}|\mathcal{M}(G)|p^{k+1}\leq|\mathcal{M}(G)||G^{\prime}||\mathrm{Im}\Psi_{2}|\leq\\\
|(G/G^{\prime}\wedge G/G^{\prime})|p^{kd}\leq
p^{\frac{1}{2}2(2-1)m}p^{2k}=p^{m+2k}.\end{array}$

Thus $|\mathcal{M}(G)|\leq p^{m+k-1}.$ On the other hand, by our assumption
$|\mathcal{M}(G)|=p^{m+k}.$ Therefore we have a contradiction. Hence
$Z(G)\subseteq G^{\prime}.$ If $m=1,$ we have $d(G)=n-k=2.$ [15, Theorem 3.1]
implies that $|\mathcal{M}(G)|\leq p^{k}.$ This contradicts our assumption.
Thus $m\geq 2.$ By Proposition 2.5, $|\mathcal{M}(G/Z(G))|$ attains the bound.
We know that $G/Z(G)$ is nilpotent of class two. We claim that
$Z(G/Z(G))=G^{\prime}/Z(G).$ Let $G=\langle x_{1},x_{2}\rangle$ and
$G^{\prime}/Z(G)=\langle[x_{1},x_{2}]Z(G)\rangle\cong\mathbb{Z}_{p^{s}}$ for
some $s\geq 0.$ Choose an element $yZ(G)\in
Z(G/Z(G))\setminus(G/Z(G))^{\prime}.$ Thus $[x_{1},x_{2}]Z(G)\otimes
y(G/Z(G))^{\prime}$ is a non-trivial element in
$(G/Z(G))^{\prime}\otimes(G/Z(G))^{ab}.$ By Lemma 2.1 $(i),$ we have
$(G/Z(G))^{ab}\cong\mathbb{Z}_{p^{\alpha}}^{(2)}.$ By Theorem 1.3, we can see
that $[x_{1},x_{2}]Z(G)\otimes y(G/Z(G))^{\prime}$ is a non-trivial element in
$\ker g$ and so

$\begin{array}[]{lcl}|\mathcal{M}(G/Z(G))|p^{k_{1}+1}\leq|\mathcal{M}(G/Z(G))||G^{\prime}Z(G)/Z(G)||\ker
g|=\\\
|\mathcal{M}((G/Z(G))^{ab})||((G/Z(G))^{\prime}\otimes(G/Z(G))^{ab}|\leq
p^{\frac{1}{2}2(2-1)\alpha}p^{2k_{1}}=p^{\alpha+2k_{1}}.\end{array}$

Thus $|\mathcal{M}(G/Z(G))|\leq p^{\alpha+k_{1}-1}.$ On the other hand, by our
assumption $|\mathcal{M}(G/Z(G))|$ $=p^{\alpha+k_{1}}.$ Therefore we have a
contradiction. Hence $Z(G/Z(G))=G^{\prime}/Z(G).$ Since $d(G/Z(G))=2,$ Theorem
2.10 implies $G/Z(G)\cong G_{2}$ or $G/Z(G)\cong G_{4}.$ Lemma 1.6 implies
$Z(G/Z(G))=G^{\prime}/Z(G)$ is cyclic and so Lemma 2.11 implies that
$G/Z(G)\cong G_{4}$ and $p\neq 2.$ Thus

$G/Z(G)=\langle
xZ(G),yZ(G)|(xZ(G))^{p^{m}}=(yZ(G))^{p^{m}}=([x,y]Z(G))^{p^{m}}=1$
$,[x,[x,y]]Z(G)=[y,[x,y]]Z(G)=1\rangle.$

Without loss of generality, we may assume that $[x,y,x]\neq 1.$ Since
$n=2m+k,$ Theorem 1.5 and [13, Corollary 2.2.12] imply

$\displaystyle
p^{\frac{1}{2}(2-1)(n+k-2)+1}p^{k}|\mathrm{Im}\Psi_{3}|=p^{\frac{1}{2}(2-1)(2m+2k-2)+1}p^{k}|\mathrm{Im}\Psi_{3}|$
$\displaystyle=|\mathcal{M}(G)||G^{\prime}||\mathrm{Im}\Psi_{3}|\leq|\mathcal{M}(G/G^{\prime})|\prod_{i=2}^{3}|(\gamma_{i}(G)/\gamma_{i+1}(G)\otimes
G/G^{\prime})|\leq p^{\frac{1}{2}2(2-1)m}p^{2k}.$

Thus $|\mathrm{Im}\Psi_{3}|\leq 1.$ On the other hand,
$\Psi_{3}(xG^{\prime}\otimes yG^{\prime}\otimes xG^{\prime}\otimes
yG^{\prime})=([[x,y],x]\otimes yG^{\prime})^{2}([y,[x,y]]\otimes
xG^{\prime})^{2}\neq 1.$ Thus $\mathrm{Im}\Psi_{3}$ is non-trivial. Hence we
have a contradiction.

Now let $t>3.$ By contrary, let there exist a group $G$ such that
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$ By using the induction
hypothesis, $\mathcal{M}(G/Z(G))$ cannot attain the bound, which is impossible
by looking Proposition 2.5. Therefore the assumption is false and the result
obtained. ∎

We are ready to obtain the structures of $G$ when
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}.$

######  Theorem 2.13.

Let $G$ be a non-abelian finite $p$-group of order $p^{n}$ such that
$|G^{\prime}|=p^{k}$ and $d(G)=d.$ Then
$|\mathcal{M}(G)|=p^{\frac{1}{2}(d-1)(n+k-2)+1}$ if and only if $G$ is
isomorphic to one of the following groups:

$G_{1}\cong E_{1}\times\mathbb{Z}_{p}^{(n-3)},$ where $E_{1}$ is the extra-
special $p$-group of order $p^{3}$ and exponent $p,$ $(p\neq 2).$

$G_{2}\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle$ is non-
metacyclic of order $p^{2m+1}$ and $m>1.$

$G_{3}\cong\mathbb{Z}_{p}\rtimes\mathbb{Z}_{p}^{(4)},$ $(p\neq 2).$

$G_{4}\cong\langle
a,b|a^{p^{m}}=b^{p^{m}}=c^{p^{m}}=1,[a,b]=c,[a,b,a]=[a,b,b]=1\rangle,$ of
order $p^{3m}$ in where $2\leq m,$ and $(p\neq 2).$

$G_{5}\cong\langle
x_{1},y_{1},x_{2},y_{2},x_{3},y_{3}|[x_{1},x_{2}]=y_{3},[x_{2},x_{3}]=y_{1},[x_{3},x_{1}]=y_{2},[x_{i},y_{j}]=[y_{j},y_{t}]=1,x_{i}^{p}=y_{i}^{p}=1,1\leq
i,j,t\leq 3\rangle$ and $p\neq 2.$

$G_{6}\cong\langle
x_{1},y_{1},x_{2},y_{2},x_{3},y_{3},z|[x_{1},x_{2}]=y_{3},[x_{2},x_{3}]=y_{1},[x_{3},x_{1}]=y_{2},[y_{i},x_{i}]=z,[x_{i},y_{j}]=1,x_{i}^{3}=y_{i}^{3}=z^{3}=1,1\leq
i,j\leq 3\rangle.$

###### Proof.

Suppose that $G$ is nilpotent of class $2$ and $|\mathcal{M}(G)|$ attains the
bound. Then $G$ is isomorphic to one of the groups $G_{1},G_{2},G_{3},G_{4},$
or $G_{5},$ by Theorem 2.10. Now, let $G$ is nilpotent of class at least $3.$
Using Corollary 2.6, $|\mathcal{M}(G/\gamma_{3}(G))|$ attains the bound. Lemma
2.1 $(ii)$ implies that $2\leq d(G)\leq 3.$ Thus by Proposition 2.12, we have
$d(G)=3$ and so $d(G/\gamma_{3}(G))=3.$ By Theorem 2.10, we have
$G/\gamma_{3}(G)\cong G_{3}$ or $G/\gamma_{3}(G)\cong G_{5}.$ Hence
$(G/\gamma_{3}(G))^{ab}$ is elementary abelian. Thus $d(G)=n-k.$ By using [19,
Theorem 1.2] $(iv),$ we have $G\cong G_{6}.$ The converse holds by Theorem
2.10 and [19, Theorem 1.2]. ∎

## References

  * [1] Ya. G. Berkovich, On the order of the commutator subgroups and the Schur multiplier of a finite $p$-group, J. Algebra 144 (1991) 269-272.
  * [2] Y. Berkovich, Groups of Prime Power Order, Vol. 1 (Walter de Gruyter, Berlin, 2008).
  * [3] F. R. Beyl, U. Felgner, and P. Schmid, On groups occurring as center factor groups, J. Algebra 61 (1970) 161-177.
  * [4] N. Blackburn, L. Evens, Schur multipliers of $p$-groups, J. Reine Angew. Math. 309 (1979), 100-113
  * [5] G. Ellis, A bound for the derived and Frattini subgroups of a prime-power, Proc. Amer. Math. Soc., 126 No. 9 (1998) 2513-2523.
  * [6] G. Ellis, On the Schur multiplier of $p$-groups, Comm. Algebra 9 (1999) 4173-4177.
  * [7] G. Ellis, J. Wiegold, A bound on the Schur multiplier of a prime-power group. Bull. Austral. Math. Soc. 60 (1999), no. 2, 191-196.
  * [8] G. Ellis, $q$-crossed On the relation between upper central quotient and lower central series of a group, A.M.S. Soc. 353 (2001) 4219-4234.
  * [9] J. A. Green, On the number of automorphisms of a finite $p$-group, Proc. Roy. Soc. London 237 (1956), 574-581.
  * [10] S. Hatui, Finite $p$-groups having Schur multiplier of maximum order, J. Algebra 492 (2017) 490-497.
  * [11] M. Isaacs, Derived subgroups and centers of capable groups, Proc. Amer. Math. Soc. 129 (2001), 2853-2859.
  * [12] M. R. Jones, Some inequalities for the multiplicator of a finite group, Proc. Amer. Math. Soc. 39 (1973) 450-456.
  * [13] G. Karpilovsky, The Schur mulptiplier, N.S. 2, London Math . Soc. Monogr, 1987.
  * [14] A. Magidin and R. F. Morse, Certain homological functors of $2$-generator $p$-groups of class 2, Contemp. Math. 511 (2010), 127-166.
  * [15] P. Niroomand, On the order of Schur multiplier of non-abelian $p$-groups, J. Algebra 322 (2009) 4479-4482.
  * [16] P. Niroomand, Characterizing finite $p$-groups by their Schur multipliers, C.R. Math. Acad. Sci. Paris, Ser. I 350 (2012), 867-870.
  * [17] P. K. Rai, A note on the order of the Schur multiplier of $p$-groups, available at https://arxiv.org/abs/1606.01493.
  * [18] P. K. Rai, On classification of groups having Schur multiplier of maximum order. Arch. Math. (Basel) 107 (2016), no. 5, 455-460.
  * [19] P. K. Rai, On the order of the Schur multiplier of $p$-groups, available at https://arxiv.org/abs/1705.02520.
  * [20] L. Redei, Das schiefe Produkt in der Gruppentheorie, Comment. Math. Helvet. 20 (1947), 225-267.
  * [21] A. R. Salemkar, M. R. R. Moghaddam, M. Davarpanah, and F. Saeedi, A remark on the Schur multiplier of $p$-groups, Comm. Algebra 35 (2007), 1215-1221.
  * [22] X. Zhou, On the order of the Schur multiplier of finite $p$-groups, Comm. Algebra 22 (1994), 1-8.


# Compressing Lengthy Context With UltraGist

Peitian Zhang1,2 Zheng Liu1 Shitao Xiao1 Ninglu Shao1,2 Qiwei Ye1 Zhicheng
Dou2
1: Beijing Academy of Artificial Intelligence,
2: Gaoling School of Artificial Intelligence, Renmin University of China
{namespace.pt<EMAIL_ADDRESS>
Peitian Zhang and Zheng Liu are the co-first authorsZheng Liu is the
corresponding author

###### Abstract

Compressing lengthy context is a critical but technically challenging problem.
In this paper, we propose a new method called UltraGist, which is
distinguished for its high-quality compression of lengthy context due to the
innovative design of the compression and learning algorithm. UltraGist brings
forth the following important benefits. Firstly, it notably contributes to the
flexibility of compression, as it can be effectively learned to support a
broad range of context lengths and compression ratios. Secondly, it helps to
produce fine-grained compression for the lengthy context, where each small
segment of the context is progressively processed on top of a tailored cross-
attention mechanism. Thirdly, it makes the training process sample-efficient
and thus maximizes the use of training data. Finally, it facilitates the
efficient running of compression for dynamic context, as the compression
result can be progressively generated and hence incrementally updated.
UltraGist is evaluated on a wide variety of tasks associated with lengthy
context, such as document QA and summarization, few-shot learning, multi-
session conversation, et al. Whilst the existing methods fail to handle these
challenging scenarios, our approach is able to preserve a near-lossless
compression performance throughout all the evaluations. Our data, model, and
code have been released at https://github.com/namespace-Pt/UltraGist.

##  1 Introduction

The economical running of large language models (LLMs) is a critical issue for
society. On one hand, it cuts down the monetary cost of using LLMs, which
makes such powerful AI tools more accessible to people’s lives. On the other
hand, it also contributes to the saving of energy, which makes the
corresponding techniques environment-friendly. The economical running of LLMs
can be advanced in different ways, among which the compression of context is
one important perspective [30; 22; 7]. This is because the Transformer
architecture incurs a quadratic time complexity while performing self-
attention. Once the context length can be substantially compressed, there will
be a tremendous reduction of computation and memory consumption.

Despite the preliminary progress, it remains a tough challenge to perform
high-quality compression for LLM’s context. Notably, the existing methods
struggle to handle lengthy context. Most of the time, they are trained to
compress relatively short context with a pre-defined compression ratio [30],
which is insufficient and inflexible to compress the working context for many
real-world applications, such as document-level reading compression, document
summarization, and multi-turn conversation. Besides, the existing methods are
prone to big compression losses, which means the LLM’s performance is likely
to severely degrade based on the compressed context. Even worse, the
compression loss can be magnified when handling longer context or out-of-
domain scenarios. Finally, the compression is statically computed in many
popular methods [22; 30], which means the compression needs to be re-computed
if the context is updated. Such methods are unsuitable for scenarios like
conversations, where new context is dynamically presented.

As discussed, it is non-trivial to perform high-quality compression for
lengthy context, where much of the reason is due to the rigid form of
compression and training. Particularly, the previous methods used to produce a
fixed-size compressed result (e.g., a constant number of compressed hidden
states [30; 15; 23]) for the input context (usually truncated to a uniform
length), based on which the LLM makes its prediction for the target output.
Given such a process, it can only learn to compress one particular context
length by a constant compression ratio. Besides, it is only able to obtain
training loss from the target output. In contrast, the input context, which
accounts for a big amount of the computation during compression, cannot be
utilized as the training objective.

Figure 1: Comparison using an example sentence. The conventional methods (A),
like Gist [30], treat most of the sentence starting from the beginning as
input and the end of it as output. The input is compressed all at once, based
on which the output is predicted. UltraGist (B) partitions the context into
fine-grained segments of an equal window size (S1, S2, S3). Then, it
progressively compresses the entire context based on randomly sampled
compression ratios. Each segment is predicted from its compressed preceding
context (i.e. the grey slots whose brightness indicates the compression
ratio).

To address the above challenges, we propose a new approach called
UltraGist.111Gist [30] is a well-known approach for context compression. The
name is to emphasize the dramatic improvement achieved by our method as high-
quality compression can be made for much longer context. Our approach is
featured for its high-quality compression of lengthy context thanks to the
innovative compression and learning mechanism. In a nutshell, we partition the
lengthy context into fine-grained segments of an equal size $w$, e.g., 1024
tokens, such that the whole context can be progressively compressed based on
each segment’s internal information and the external information from the
compressed preceding context (Figure 1). For each segment, the compression
ratio $\alpha$ is randomly sampled, where the context within the segment is
compressed by a factor of $\alpha$. Based on such a workflow, the learning
process takes place in the form of compression-based language modeling, where
each token is predicted from its compressed preceding context. The above
mechanism brings forth four benefits.

  * •

Firstly, it notably contributes to the compression’s flexibility. On one hand,
the prediction of next token is made based on the compressed preceding
context, where the context lengths differ among tokens. On the other hand,
each token’s preceding context is compressed by a mixture of different ratios
because of the random sampling. Therefore, UltraGist can naturally be learned
to handle a broad range of context lengths and compression ratios during
training.

  * •

Secondly, it helps to produce fine-grained compression for the lengthy
context. Different from many conventional methods which compress the lengthy
context directly, our method is able to effectively preserve the fine-grained
information of context since the small context segments can be progressively
processed with an optimized cross-attention mechanism.

  * •

Thirdly, it substantially improves the sample-efficiency and thus helps to
make full use of the training data, as the training loss can be obtained from
all tokens within each training sample.

  * •

Fourthly, it facilitates the efficient running of compression when dealing
with dynamic context. Thanks to the progressive workflow, the new context,
e.g., a session of dialogue between human and machine, can be incrementally
compressed and added to the existing compression result.

UltraGist is trained with both plain corpus and instruction-tuning data.
Because of the superior sample efficiency, it can quickly establish a strong
compression capability for general applications. The effectiveness of
UltraGist is verified by a wide variety of tasks associated with lengthy
contexts, such as document QA, document summarization, few-shot learning, and
multi-session conversation. Whilst the existing methods fail to handle these
challenging scenarios, our method is able to maintain near-lossless
compression performances throughout all these evaluations. Our data, model,
and source code (anonymously submitted) will be publicly released to
facilitate the future research.

##  2 Related Work

LLMs need to deal with lengthy context in many important applications, like
the reading compression of a book, the summarization of a long document, the
multi-turn conversations with human, et al. To confront such scenarios, a
large body of research has been dedicated to the extension of LLMs’ context.
For example, ALiBi [33] leverages linear-decaying attention biases to achieve
the extrapolation of position encoding. Methods like Position Interpolation
[5], NTK-Aware scaling [1] and ReRoPE [35] make adaptation of RoPE [36] such
that the LLM can handle unseen positions at the inference time. Besides,
recent studies suggest that the LLM’s long-context capability can
significantly benefit from the continual training and fine-tuning after the
modification of ROPE [32; 13]. Considering that training over long-context
data is expensive, people also investigate the cost-effective way of training.
For example, LongLora [6] proposes S2-Attn and leverages LoRA [18] for further
acceleration. PoSE [41] uses skip-wise position indices to train LLMs on 2K
context length as a simulation of 128K. However, fine-tuning operations are
likely to incur losses of LLMs’ general capability. Ultimately, it is an
extremely expensive and resource-consuming option to apply LLMs directly for
processing lengthy context.

Another important line of research is about context compression. Such
techniques aim to compress the context into shorter and more compact forms
where the LLM can generate outputs of equivalent quality. A well-known study
in this area is presented by Gist [30]. It learns to compress the input
prompts based on a certain number of Gist tokens, such that the corresponding
instruction-tuning tasks can be accomplished with shorter context. Apart from
Gist, many related works are proposed following the same spirit. ICAE [15]
fine-tunes the LLM to be a context compressor using LoRA, which produces the
compression result on top of several memory tokens. Meanwhile, AutoCompressor
[7] and CCM [23] also fine-tunes the LLM to generate a fixed number of memory
(or summary) tokens; however, they rely on auto-regression to better confront
longer context and interactive scenarios. In addition to the above methods
which leverages special tokens, there are also research works like LLMLingua
[21; 22], which directly remove unimportant tokens from the context based on
smaller specialist LMs. Despite the above technical progresses, the existing
context compression methods are limited in many perspectives. Notably, they
can hardly handle lengthy context. Most of them can only be applied for a
specific task, rather than being a general approach. Finally, there are still
severe performance losses due to compression.

While our work shares some common principles with the related works, it is
distinguished in many critical perspectives about compression and training.
One notable advantage is the dynamic sampling of compression ratios (i.e.
varying numbers of UltraGist tokens) during the progressive compression
process. Because of such a feature, UltraGist can be learned to flexibly
handle a broad range of context lengths and compression ratios during the
training process. Different from our approach, most of the existing methods
simply learn to compress the uniformly truncated context with a fixed number
of compression tokens [30; 15; 23]. Another important difference is that our
method is able to perform fine-grained compression for each small context
segment on top of its optimized attention mechanism. In contrast, the existing
methods usually append all special tokens to the end of a big-chunk of context
for compression, which is prone to coarse-grained results [15; 7]. Finally,
our progressive compression of fine-grained segments facilitates sample-
efficient training, which significantly improves upon the common practice
where the compression is performed in an encoding-decoding manner [30; 15].
Other auto-regressive approaches also share a similar property [7; 23].
However, due to the coarse-grained partition of context, many of the tokens
can hardly leverage the compressed preceding context during the training
process, since their recent context is largely uncompressed and next tokens
can easily be predicted from it.

##  3 Method

In this section, we start with the preliminaries of context compression and
present the definition of the problem. Then, we elaborate on how lengthy
context is compressed by UltraGist and how the learning process is conducted
based on the training sample.

The LLMs are learned to accomplish arbitrary tasks in the form of language
generation. Formally, it operates as the following probabilistic process of
next token prediction: $\mathrm{P}_{LM}(x_{t}|x_{1},\dots,x_{t-1})$, where
$x_{i}$ is the $i$-th token within the context. Due to the transformer-based
architecture of LLMs, the lengthy context will take a huge computation cost
while performing the next token prediction. To mitigate this problem, the
compression of context becomes a welcomed solution. In our work, the
compression is made on top of the newly added special token, namely the
UltraGist token: <ug>. Suppose $m$ UltraGist tokens are introduced to compress
the context, the language modeling process is modified as:
$\mathrm{P}_{LM}(x_{t}|x_{t-l},\dots,x_{t-1},\text{<ug>}_{1},\dots\text{<ug>}_{m})$,
where $x_{t-l},\dots,x_{t-1}$ are the latest $l$ tokens from the original
context, and $\text{<ug>}_{1},\dots\text{<ug>}_{m}$ are the compression for
the rest of the context before $x_{t-l}$ ($m\ll t-l$). Based on such a
formulation, the learning of compression can be defined as the optimization of
language modeling conditioned on the compressed context with UltraGist.

###  3.1 Compression

We focus on dealing with a lengthy context, e.g., a long document of thousands
of tokens. We argue that the direct compression, as performed by many recent
works [30; 15], is unsuitable for this scenario. This is because lengthy
context is made up of complicated and heterogeneous information, where the
direct compression is likely to incur coarse-grained results. Besides, the
lengthy context can even go beyond the maximum length of LLMs. In this work,
we propose to perform progressive and fine-grained compression for lengthy
context. Particularly, the input context $X=\\{x_{1},\dots,x_{t}\\}$ is
partitioned into small segments of the same window size $w$ (e.g., 1024):

$X=\\{x_{1},\dots,x_{t}\\}\xrightarrow{\mathrm{Partition}}\\{X_{1},\dots
X_{N}\\},~{}X_{i}=\\{x_{(i-1)w+1},\dots,x_{\min\\{iw,t\\}}\\}$ (1)

With the well-partitioned context, the progressive compression takes place
where each segment is processed sequentially. The above workflow facilitates
fine-grained compression of the contextual information. Besides, it relaxes
the requirement on LLMs’ maximum length, as the small segments can fit into
most of the existing models. It also contributes to the efficiency since the
progressive processing results in a linear time complexity w.r.t. the context
length.

Each segment is compressed by a group of UltraGist tokens: <ug>. The UltraGist
tokens are appended to normal tokens in each segment, which jointly summarize
the contextual information of the corresponding segment. To flexibly support
diversified compression requirements, the compression ratio $\alpha_{i}$ for
the $i$-th segment is dynamically determined during the training process. In
our work, $\alpha_{i}$ is uniformly sampled from a scope of candidate ratios:
$\alpha_{i}\sim\mathrm{Uni}(\\{2,4,8,16,32\\})$. After that, the number of
UltraGist tokens $k_{i}$ can be determined accordingly: $k_{i}=w/\alpha_{i}$.
In other words, the information from $w$ normal tokens will be compressed by
$k_{i}$ UltraGist tokens.

The UltraGist tokens are employed to compress each segment based on cross-
attention [26]. Formally, such an operation is defined by the following
formula:

$\mathrm{CrossAttn.}(\underbrace{\text{<ug>}_{1,1},\dots,\text{<ug>}_{i-1,k_{i-1}}}_{\text{UltraGist
tokens from the preceding
context}},x_{i,1},\dots,x_{i,w},\underbrace{\text{<ug>}_{i,1},\dots,\text{<ug>}_{i,k_{i}}}_{\text{UltraGist
tokens for the $i$-th segment}};\mathrm{Mask}).$ (2)

In this place, $x_{i,1},\dots,x_{i,w}$ are the normal tokens within the $i$-th
segment. $\text{<ug>}_{1,1},\dots,\text{<ug>}_{i-1,k_{i-1}}$ are the UltraGist
tokens from the previous segments, which present the compressed information
from the preceding context. $\text{<ug>}_{i,1},\dots,\text{<ug>}_{i,k_{i}}$
are the UltraGist tokens for the $i$-th segment. They are used to query the
previous tokens by cross-attention based on a tailored attention mask
“$\mathrm{Mask}$”.

The computation of cross-attention is explained as follows. For an arbitrary
layer of the LLM, let $\mathbf{H}^{{nt}}\in\mathbb{R}^{w\times D}$ and
$\mathbf{H}^{ug}\in\mathbb{R}^{k_{i}\times D}$ denote the hidden states of
normal tokens and UltraGist tokens, respectively. The hidden states are
projected into queries, keys, and values for cross-attention:

$\displaystyle\boldsymbol{Q}^{nt}=\boldsymbol{W}_{Q}^{nt}\mathbf{H}^{nt},$
$\displaystyle\boldsymbol{K}^{nt}=\boldsymbol{W}_{K}^{nt}\mathbf{H}^{nt},$
$\displaystyle\boldsymbol{V}^{nt}=\boldsymbol{W}_{V}^{nt}\mathbf{H}^{nt},$
$\displaystyle\boldsymbol{Q}^{ug}=\boldsymbol{W}^{ug}_{Q}\mathbf{H}^{ug},$
$\displaystyle\boldsymbol{K}^{ug}=\boldsymbol{W}^{ug}_{K}\mathbf{H}^{ug},$
$\displaystyle\boldsymbol{V}^{ug}=\boldsymbol{W}^{ug}_{V}\mathbf{H}^{ug},$ (3)

where $\boldsymbol{W}_{*}^{nt}$ and $\boldsymbol{W}_{*}^{ug}$ are the
projection matrices to handle normal tokens and UltraGist tokens
($\boldsymbol{W}_{*}^{nt}$ are the LLM’s original parameters, while
$\boldsymbol{W}_{*}^{ug}$ are newly introduced parameters). Based on the
projection results, the following attention computation is conducted:

$\displaystyle\boldsymbol{A}=\mathrm{softmax}\left(\mathrm{Mask}\left(\frac{\\{\boldsymbol{Q}^{nt};\boldsymbol{Q}^{ug}\\}\\{\boldsymbol{K}^{ca};\boldsymbol{K}^{nt};\boldsymbol{K}^{ug}\\}^{T}}{\sqrt{D}}\right)\right),\quad\boldsymbol{V}=\boldsymbol{A}\\{\boldsymbol{V}^{ca};\boldsymbol{V}^{nt};\boldsymbol{V}^{ug}\\}$
$\displaystyle\boldsymbol{}\boldsymbol{O}^{nt}=\boldsymbol{W}^{nt}_{O}\boldsymbol{V}[:w],\quad\boldsymbol{O}^{ug}=\boldsymbol{W}^{ug}_{O}\boldsymbol{V}[w:].$
(4)

In the above equations, $\\{\cdot~{};~{}\cdot\\}$ denotes the matrix
concatenation.
$\boldsymbol{K}^{ca},\boldsymbol{V}^{ca}\in\mathbb{R}^{(\sum_{j=1}^{i-1}{k_{j}})\times
D}$ are the cached projection results for preceding UltraGist tokens. $D$ is
the LLM’s hidden dimension. All tokens are encoded by their relative positions
within the context window while cross-attention is conducted. The output
hidden states, i.e. $\boldsymbol{O}^{nt}$ and $\boldsymbol{O}^{ug}$, are
further processed by other operations (e.g., MLP, LayerNorm) before passed to
the next layer. After the above computation, the keys and values of UltraGist
tokens, i.e. $\boldsymbol{K}^{ug}$ and $\boldsymbol{V}^{ug}$, are generated as
the compression result for the normal tokens. Afterwards, the cached keys and
values of UltraGist tokens are incrementally updated:

$\boldsymbol{K}^{ca}\leftarrow\\{\boldsymbol{K}^{ca};\boldsymbol{K}^{ug}\\},\quad\boldsymbol{V}^{ca}\leftarrow\\{\boldsymbol{V}^{ca};\boldsymbol{V}^{ug}\\}.$
(5)

Figure 2: Illustration of Cross-Attention. $S_{1}$ and $S_{2}$ are the first
two segments, each consisting of 4 tokens. The compression ratio $\alpha$ is
set to 2, where two UltraGist tokens are appended. We use the “stepwise
expansion” style attention mask, where 1) the later UltraGist token can
attention to a larger scope of normal tokens, 2) the UltraGist tokens from
preceding context can all be attended. MHAug indicates the learnable
parameters ($\boldsymbol{W}_{Q}^{ug}$, $\boldsymbol{W}_{K}^{ug}$,
$\boldsymbol{W}_{V}^{ug}$, $\boldsymbol{W}_{O}^{ug}$) for UltraGist tokens’
activations.

It is worth noting the attention mask is tailored for the cross-attention
operation. Particularly, we propose to utilize the “stepwise expansion” style
mask (Figure 2), where the UltraGist tokens can attend to a linearly growing
number of normal tokens in the corresponding segment. In other words, the
first $j\times\alpha_{i}$ normal tokens can be attended by
$\text{<ug>}_{i,j}$. Meanwhile, the UltraGist tokens from preceding context
can always be attended. The above mechanism enables UltraGist tokens to
capture differentiated information of the segment, which contributes to their
joint compression effect. It is empirically more competitive than the common
mask used by previous works [30; 15; 7], where the whole normal tokens can be
attended by the compression module.

###  3.2 Learning

UltraGist is learned to optimize the generation quality with the compressed
context. Formally, the generation of each token is presented as:
$\mathrm{P}_{LM}(x_{i,j}|\text{<ug>}_{1,1},\dots,\text{<ug>}_{i-1,k_{i-1}},x_{i,1}\dots
x_{i,j-1};\boldsymbol{\Theta},\boldsymbol{\Theta}^{ug})$. The generation
process is associated with two sets of parameters. One is the LLM’s original
parameters $\boldsymbol{\Theta}$, which are fixed such that the LLM’s original
capability with short context won’t be affected. The other one is the newly
added learnable parameters, which include the parameters for the projection of
UltraGist tokens’ projection, i.e. $\boldsymbol{W}_{Q}^{ug}$,
$\boldsymbol{W}_{K}^{ug}$, $\boldsymbol{W}_{V}^{ug}$,
$\boldsymbol{W}_{O}^{ug}$ (initialized from the LLM’s original parameters:
$\boldsymbol{W}_{Q}^{nt}$, $\boldsymbol{W}_{K}^{nt}$,
$\boldsymbol{W}_{V}^{nt}$, $\boldsymbol{W}_{O}^{nt}$), and the token embedding
for UltraGist (one shared embedding for UltraGist tokens despite that they are
used multiple times in cross attention). The learning is performed to minimize
the the next-token prediction loss for the entire context:

$\min\limits_{\boldsymbol{\Theta}^{ug}}.\sum_{i=2}^{N}\sum_{j=1}^{w}\mathrm{P}_{LM}(x_{i,j}|\text{<ug>}_{1,1},\dots,\text{<ug>}_{i-1,k_{i-1}},x_{i,1}\dots
x_{i,j-1};\boldsymbol{\Theta},\boldsymbol{\Theta}^{ug}).$ (6)

As presented above, the training loss can be obtained from all tokens within
the context (except the ones in the first segment). Such a property
contributes to a high sample-efficiency, which maximizes the use of training
data and thus facilitates the learning of compression capability. Unlike
recurrent memory methods [7; 4] that compel serial encoding of each segment,
UltraGist allows parallel encoding of adjacent segments. For example, in
Figure 2, the encoding of the first two segments can be computed
simultaneously leveraging the attention mask of the largest gray square.

##  4 Experiment

Our experiment aim to address these research questions: RQ 1. Lengthy
compression capability, whether UltraGist can effectively handle lengthy
context with diversified compression ratios ($\S$4.2)? RQ 2. Generalizability,
whether UltraGist can effectively compress different lengthy tasks ($\S$4.3)?
RQ 3. The cost-effectiveness of UltraGist ($\S$4.4). RQ 4. The effect of each
technical design ($\S$4.5).

###  4.1 Settings

Implementation. Our method is implemented on Llama-2-7B (chat) [38] for the
experiment (an alternative implementation is made with Mistral-7B-Instruct
(v0.2) [20], which is presented in Appendix C). UltraGist is incompatible with
FlashAttention-2 [10] due to its customized attention mask; thus, we use the
scaled dot product attention from PyTorch [29] for acceleration. We use one
8xA800 (80G) machine all our experiments.

Training. The training consists of two phases. In pre-training, we use 2B
tokens from SlimPajama [34]. These tokens are per-source upsampled according
to [13]. In fine-tuning, we leverage LongAlpaca [6], BookSum [25], and 16K
synthetic data from GPT-3.5 (details in Appendix B). All the training samples
are shorter than 10240. The batch size is 8. The learning rate is 5e-5 for
pre-training and 1e-5 for fine-tuning, with linear decay scheduler and no
warmup. As introduced, the LLM’s original parameters are frozen throughout the
training process.

Baselines. We consider the following well-regarded context compression methods
in our experiments, including Gist [30], CCM [23], AutoCompressors [7],
LongLLMLingua [22], ICAE [15]. While Gist is trained on top of Llama-1 [37],
the rest of the baselines are implemented based on LLama-2-7B (chat), which is
the same as our method.

###  4.2 Flexible and Fine-Grained Compression of Lengthy Context

Figure 3: Evaluation results of Topic Retrieval. UltraGist demonstrates
superior compression quality over the baseline methods across a broad range of
context lengths and compression ratios.

The evaluation is first conducted based on Topic Retrieval [8] because of its
flexibility in making the adjustment of context length and compression ratio.
In this benchmark, a user performs conversations with a chatbot on multiple
topics sequentially. By adjusting the number of topics, we can obtain
conversations of different context lengths. In our experiment, we make use of
1, 2, 3, and 10 topics, which results in conversations of 745, 1351, 1977, and
6343 tokens, respectively. The entire conversation is used as the input
context, where the model is asked to retrieve the first topic of the
conversation. Therefore, the retrieval performance can be measured by
accuracy. We vary the compression ratio from $\times$4 to $\times$24 and
report the evaluation results in Figure 3.222CCM is hard coded to compress
each utterance with two ¡COMP¿ tokens. Since the average utterance length is
48 for Topic Retrieval and 32 for MSC, the compression ratio of CCM is
$\times$24 and $\times$16, respectively.

As shown in Figure 3, UltraGist consistently outperforms all baselines with
huge advantages. Notably, it is able to accomplish highly accurate topic
retrieval for all context lengths; most of the time, it can achieve nearly
perfect accuracy (i.e., Accuracy=1). In contrast, all baselines fall behind
and completely break down when the context length is long enough. UltraGist
also exhibits a superior resilience to different compression ratios, as it is
able to stay almost lossless within the $\times$16 compression. However, other
baselines are severely affected when the compression ratio is increased.

Note that both Gist and CCM perform badly in the experiment. In fact, these
two methods can only handle very specific application scenarios given their
training settings: Gist is for compressing short instructions, while CCM is
for compressing daily dialogues. As a result, they are incapable of dealing
with general tasks, such as Topic Retrieval and many other tasks in later
experiments.

Table 1: Evaluation of long-term memory on MSC. Method | Compression Ratio
---|---
$\times$4 |  $\times$8 |  $\times$16 |  $\times$24
Gist | 3.7 | 4.3 | 4.3 | 2.5
CCM | – | – | 21.8 | –
AutoCompressors | 28.8 | 27.3 | 25.0 | 24.0
LongLLMLingua | 22.4 | 19.5 | 17.8 | 15.9
ICAE | 18.1 | 16.6 | 15.3 | 14.5
UltraGist | 39.0 | 36.5 | 33.6 | 32.3

Further evaluations are made based on the multi-session conversation (MSC)
benchmark created by MemGPT [31]. In MSC, each testing case contains a multi-
session conversation between two human speakers. At the end of the
conversation, Speaker 1 asks a question which needs to be answered by very
specific information from one previous conversation turn. The average context
length of MSC is about 2K tokens. The entire conversation is compressed, based
on which the model outputs its answer. The answer’s quality is measured by
ROUGE-L recall. Once again, we vary the compression ratios from $\times$4 to
$\times$24.22footnotemark: 2 As reported in Table 1, UltraGist maintains its
substantial improvements over the baselines across all compression ratios.
Note that MSC is close to the scenario where CCM is trained. However,
UltraGist still outperforms the domain-specific method by a big margin.

###  4.3 High-Quality Compression Across Various Lengthy Tasks

Table 2: Evaluation of various lengthy-context compression tasks (top performances marked in bold). UltraGist is able to produce high-quality compression result throughout all these application scenarios. It significantly outperforms the baseline methods and improves upon the backbone LLM Llama-2-7B. Method | Document Comp. | Example Comp. | Code Comp.
---|---|---|---
NQA | Qasp | HpQA | Musiq | Gov | News | TREC | SSum | LCC | Repo
Llama-2-7B | 18.7 | 19.2 | 25.4 | 9.4 | 27.3 | 25.8 | 61.5 | 40.7 | 52.4 | 43.8
AutoCompressors | 3.3 | 7.8 | 9.6 | 3.9 | 17.5 | 11.3 | 36.2 | 27.8 | 15.2 | 19.2
LongLLMLingua | 13.1 | 21.1 | 20.5 | 4.7 | 23.0 | 25.6 | 34.5 | 39.9 | 54.8 | 50.1
ICAE | 10.1 | 19.5 | 23.7 | 7.2 | 18.6 | 22.4 | 1.0 | 3.2 | 21.0 | 23.2
UltraGist | 22.3 | 37.1 | 43.3 | 18.2 | 26.6 | 25.4 | 63.0 | 40.7 | 56.2 | 51.1

To investigate UltraGist’s compression capability in general scenarios, we
further leverage a variety of real-world tasks associated with lengthy context
from LongBench [3]. One type of tasks is QA and summarization based on one or
multiple compressed long document (Document Comp.), which include the
following datasets: NarriativeQA [24], Qasper [11], HotpotQA [40], Musique
[39], GovReport [19], MultiNews [12]. Another type is about few-shot learning
with compressed in-context examples (Example Comp.), which includes TREC [27]
and SamSum [16]. The third type is about long-range code completion (Code
Comp.), which includes LCC [17] and RepoBench [28]. The context length of
these tasks vary from 1K to 32K. There is no overlap between the tasks’
training sets and our fine-tuning data, thus the evaluation can be interpreted
as a zero-shot evaluation. The results are reported in Table 2.333We exclude
the results of Gist [30] and CCM [23] because the context lengths of these
tasks are far beyond their maximum capacity (as demonstrated by Figure 3).

Firstly, UltraGist achieves superior performances across all scenarios. There
are huge advantages over the baselines in most of the tasks, such as
NarrativeQA, Qasper, HotpotQA, Musique, and TREC. Although some baselines can
achieve comparable performances in a few cases, e.g., LongLLMLingua on
MultiNews summarization, it falls short in other tasks like question answering
and code completion. In fact, none of the baselines can maintain their
effectiveness in general settings.

Secondly, UltraGist improves upon Llama-2-7B, the backbone LLM. In the
experiment, UltraGist compresses the lengthy context into a 4K context window;
while Llama-2-7B directly makes use of the truncated 4K context. Such an
advantage indicates that useful information from Llama-2-7B’s unseen context
can be effectively introduced by UltraGist and hence be utilized. However, the
baseline methods fall far behind Llama-2-7B. It suggests that there are severe
losses of useful information with the baselines such that the compressed
unseen context cannot be utilized by the backbone LLM.

Last but not least, we evaluate UltraGist with the Needle-In-A-Haystack-Search
test (NIHS), which is an extremely challenging task for context compression
methods (to the best of our knowledge, no existing methods can fulfill this
test). As shown in Figure 5, UltraGist can precisely retrieve the needle in
most of the time. It’s also worth noting that the training data of UltraGist
is within 10K. Thus, it suggests that UltraGist can be effectively generalized
to context longer than its training data.

###  4.4 Cost-Effectiveness of UltraGist

Figure 4: Needle-In-A-Haystack test. UltraGist can accurately recall the
hidden needle within the lengthy context (up to 32K).

Figure 5: The perplexity on held-out validation set during training. UltraGist
converges much faster due to its high sample efficiency.

We analyze the cost-effectiveness of UltraGist with multi-turn conversation, a
typical application scenario of context compression where the lengthy context
is dynamically presented [23; 31]. We conduct the pilot experiment based on
data from ShareGPT [2], which contains sufficiently long conversations between
human users and ChatGPT. There are 3 turns of conversation, where the context
length is increased from 765 tokens to 6491 tokens. All methods compress the
historical utterances based on the same compression ratio ($\times$8), where
the new utterance is generated from the compression result. The compression
quality is measured by PPL (perplexity): a lower PPL indicates that the
favorable conversation is more likely to be generated based on the compression
result. The efficiency is measure by TFLOPs (TeraFLOPs): the lower TFLOPs
reflect the higher efficiency. The empirical results are shown in Table 3,
where the following observations can be made.

Firstly, UltraGist leads to the optimal generation quality throughout
conversations of different context lengths. Such an observation is
complementary with the results in Table 1, where conversations of identical
lengths are compressed with different ratios. We can also observe that the PPL
of UltraGist can be consistently reduced whenever longer context is presented.
Such an observation suggests that the historical utterance can be effectively
compressed and utilized by UltraGist. However, it is not the case for many
other baselines, as their PPL can not always be reduced by longer context.

Secondly, UltraGist achieves a competitive efficiency, as it can always
maintain low TFLOPs in the experiment. It can also be observed that the TFLOPs
is increased linearly with the context length thanks to its progressive
compression of context (so do baselines like CCM relying on auto-regressive
processing). In contrast, the static methods, i.e. Gist and LongLLMLingua,
need to re-compute the compression for the dynamic context, which results in a
much quicker growth of TFLOPs.

Table 3: Analysis of cost-effectiveness. UltraGist facilitates the generation of high-quality conversations (lower PPL) while maintaining the efficient running of compression process (lower TFLOPs). Method |  1 Turn (765 tokens) |  2 Turn (3006 tokens) |  3 Turn (6491 tokens)
---|---|---|---
TFLOPs | PPL | TFLOPs | PPL | TFLOPs | PPL
Gist | 8.37 | 8.76 | 44.02 | 9.54 | 131.20 | 10.33
CCM | 8.30 | 7.91 | 33.97 | 7.20 | 75.45 | 7.63
AutoCompressors | 8.41 | 8.32 | 38.18 | 6.09 | 96.63 | 7.51
LongLLMLingua | 8.78 | 5.91 | 48.32 | 4.96 | 145.71 | 4.77
ICAE | 8.78 | 7.72 | 40.62 | 6.55 | 92.37 | 8.13
UltraGist | 8.63 | 4.27 | 34.54 | 3.08 | 75.41 | 2.98

###  4.5 Ablation Studies

We study the impact of each technical factor, including the attention
mechanism in cross-attention, the sampling strategy of compression ratio,
different training stages, and the training method. We leverage the MSC task
[31] for the ablation studies (with ROUGE-L as the performance metric) and
make use of two compression ratios ($\times$4 and $\times$8). The results are
shown in Table 4.

We first explore two optional attention mechanisms besides the “stepwise
expansion” used by UltraGist: “segmentation” where the UltraGist tokens can
sequentially attend to different parts of the context window, and the “full-
coverage” attention where the UltraGist tokens can attend to the entire
context window (more details about the alternatives are presented by Figure 6
in the Appendix). It can be observed that the attention mechanism exerts a
substantial impact on UltraGist’s performance, where the “stepwise expansion”,
which lets each UltraGist token to perceive differentiated local and global
information, notably improves the performance over the other two options.

Table 4: The impact of different technical factors. Factor | Setting | MSC
---|---|---
$\times$4 |  $\times$8
– | Default | 39.0 | 36.5
Attention Mechanism | Segmentation | 36.0 | 33.2
Full-Coverage | 37.2 | 33.7
Sampling of Ratio |  Monotonous ($\times$4) | 38.4 | 31.6
Instance-Random | 36.9 | 34.0
Training Stage | Pretrain Only | 36.0 | 34.8
FT Only | 35.3 | 34.5

We then analyze the dynamic sampling of compression ratio. We make comparison
with two alternatives. One uses a fixed compression ratio of 4, denoted as
Monotonous ($\times$4); the other one samples the compression ratio by
instance rather than by segment (the default setting), denoted as Instance-
Random. We can observe that with the dynamic sampling of compression ratio,
especially the default method where the compression ratio is sampled by
segment, is able to effectively support both scenarios ($\times$4 and
$\times$8). In contrast, the monotonous method can only handle its associated
compression ratio, i.e. $\times$4\. We further investigate the impact of
different training stages. One alternative simply conducts pre-training (Pre-
train Only), while the other one only leverages fine-tuning (FT. Only). It can
be observe that both pre-training and fine-tuning are useful, and the two
training stages jointly result in the superior performance of UltraGist
(Default). Finally, we make investigation for the impact of training method.
We compare UltraGist against the common encoding-decoding approach as used by
many previous methods [30; 15] (the input context is encoded and compressed in
the first place, then the pre-defined target is decoded). We plot the curve of
validation perplexity during the training process in Figure 5. As we can
observe, UltraGist leads to a much quicker reduction of the validation loss,
which reflects its higher sample-efficiency of training.

##  5 Conclusion

This paper introduces UltraGist, a novel approach for the compression of
lengthy context. UltraGist is featured for several critical innovations,
including the progressive fine-grained compression of context, the dynamic
sampling of compression ratios, and the sample-efficient training of
compression. Our experiment demonstrates a series of critical values presented
by UltraGist. Notably, it is able to compress much longer context than the
existing methods. It can substantially reduce the compression loss; for many
tasks, it can even achieve near-lossless compression. Finally, it is
significantly more general than the previous methods, which can effectively
handle various challenging applications.

##  6 Limitations

The current work can be improved from the following perspectives. Firstly, our
primary experiments are performed with 7B models (Llama-2-7B and Mistral-7B)
due to the limitation of resources. As the compression techniques are even
more important for those larger models, it is necessary to make further
explorations in this direction. Secondly, our experiment suggests that the
compression quality can benefit from the increasing of training scale.
Although UltraGist has achieved significant yet preliminary advantages over
the existing methods, it will be a more useful tool if it can be continually
trained. Finally, it is also meaningful to investigate the potential risks of
using context compression, such as hallucinations and the catastrophic
forgetting of LLM’s general skill.

## References

  * [1] Ntk-aware scaled rope, 2023.
  * [2] Sharegpt, 2023.
  * [3] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li.  Longbench: A bilingual, multitask benchmark for long context understanding.  arXiv preprint arXiv:2308.14508, 2023.
  * [4] A. Bulatov, Y. Kuratov, and M. S. Burtsev.  Scaling transformer to 1m tokens and beyond with RMT.  CoRR, abs/2304.11062, 2023.
  * [5] S. Chen, S. Wong, L. Chen, and Y. Tian.  Extending context window of large language models via positional interpolation.  arXiv preprint arXiv:2306.15595, 2023.
  * [6] Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia.  Longlora: Efficient fine-tuning of long-context large language models.  arXiv preprint arXiv:2309.12307, 2023.
  * [7] A. Chevalier, A. Wettig, A. Ajith, and D. Chen.  Adapting language models to compress contexts.  In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3829–3846. Association for Computational Linguistics, 2023.
  * [8] L. Dacheng, S. Rulin, X. Anze, S. Ying, Z. Lianmin, E. G. Joseph, S. Ion, M. Xuezhe, and Z. Hao.  How long can open-source llms truly promise on context length?, June 2023.
  * [9] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang.  Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024.
  * [10] T. Dao.  Flashattention-2: Faster attention with better parallelism and work partitioning.  CoRR, abs/2307.08691, 2023.
  * [11] P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner.  A dataset of information-seeking questions and answers anchored in research papers, 2021.
  * [12] A. Fabbri, I. Li, T. She, S. Li, and D. Radev.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [13] Y. Fu, R. Panda, X. Niu, X. Yue, H. Hajishirzi, Y. Kim, and H. Peng.  Data engineering for scaling language models to 128k context, 2024.
  * [14] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy.  The pile: An 800gb dataset of diverse text for language modeling, 2020.
  * [15] T. Ge, J. Hu, L. Wang, X. Wang, S.-Q. Chen, and F. Wei.  In-context autoencoder for context compression in a large language model, 2024.
  * [16] B. Gliwa, I. Mochol, M. Biesek, and A. Wawer.  Samsum corpus: A human-annotated dialogue dataset for abstractive summarization.  In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, 2019.
  * [17] D. Guo, C. Xu, N. Duan, J. Yin, and J. McAuley.  Longcoder: A long-range pre-trained language model for code completion, 2023.
  * [18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [19] L. Huang, S. Cao, N. Parulian, H. Ji, and L. Wang.  Efficient attentions for long document summarization.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online, June 2021. Association for Computational Linguistics.
  * [20] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  Mistral 7b, 2023.
  * [21] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu.  Llmlingua: Compressing prompts for accelerated inference of large language models.  arXiv preprint arXiv:2310.05736, 2023.
  * [22] H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu.  Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2023.
  * [23] J.-H. Kim, J. Yeom, S. Yun, and H. O. Song.  Compressed context memory for online language model interaction, 2024.
  * [24] T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette.  The narrativeqa reading comprehension challenge, 2017.
  * [25] W. Kryściński, N. Rajani, D. Agarwal, C. Xiong, and D. Radev.  Booksum: A collection of datasets for long-form narrative summarization, 2022.
  * [26] J. Li, D. Li, S. Savarese, and S. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [27] X. Li and D. Roth.  Learning question classifiers.  In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
  * [28] T. Liu, C. Xu, and J. McAuley.  Repobench: Benchmarking repository-level code auto-completion systems, 2023.
  * [29] C. P. Michael Gschwind, Driss Guessous.  Accelerated pytorch 2 transformers.  https://pytorch.org/blog/accelerated-pytorch-2/, 2023.
  * [30] J. Mu, X. L. Li, and N. D. Goodman.  Learning to compress prompts with gist tokens.  CoRR, abs/2304.08467, 2023.
  * [31] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez.  Memgpt: Towards llms as operating systems, 2024.
  * [32] B. Peng, J. Quesnelle, H. Fan, and E. Shippole.  Yarn: Efficient context window extension of large language models.  arXiv preprint arXiv:2309.00071, 2023.
  * [33] O. Press, N. A. Smith, and M. Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [34] D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, June 2023.
  * [35] J. Su.  Rectified rotary position embeddings.  https://github.com/bojone/rerope, 2023.
  * [36] J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu.  Roformer: Enhanced transformer with rotary position embedding.  CoRR, abs/2104.09864, 2021.
  * [37] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [38] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [39] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal.  Musique: Multihop questions via single-hop question composition, 2022.
  * [40] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018.
  * [41] D. Zhu, N. Yang, L. Wang, Y. Song, W. Wu, F. Wei, and S. Li.  Pose: Efficient context window extension of llms via positional skip-wise training.  CoRR, abs/2309.10400, 2023.

##  Appendix A Attention Patterns

Figure 6: Other explored attention mechanisms in Cross-Attention.
“Segmentation”: the UltraGist tokens can sequentially attend to different
parts of the context window; “Full-Coverage”: the UltraGist tokens can attend
to the entire context window.

Besides “Stepwise-Expansion” (Figure 2), we also explore two other attention
mechanisms for cross-attention. They are shown in Figure 6.

##  Appendix B Training Data

In the pre-training phase, we use 2B tokens from SlimPajama. These tokens are
per-source upsampled [13] and mixed with the following portions. commoncrawl
(52.2%); c4 (26.7%); github (5.2%); book (4.2%); arxiv (4.6%); wiki (3.8%);
stackexchange (3.3%). We pack documents in the same source to form the
sequence of length 8192.

Prompt B.1 <Segment 1>
…
<Segment N>

Question: <Question 1 for Segment 1>
Answer: <Answer 1 for Segment 1>
…
Question: <Question 4 for Segment N>
Answer: <Answer 4 for Segment N>

In the fine-tuning phase, we use three datasets. 1) LongAlpaca [6], which
contains long-context QA and summarization data; 2) BookSum [25], which
contains chapter-level summarization of books. 3) Synthesized QA dataset. This
dataset contains 16K long-context question answering instances (13K for books
and 3K for papers). Specifically, we split a given long context (a paper or a
book) into short segments (a chunk with less than 4096 tokens) using the
SemanticTextSplitter444https://github.com/benbrandt/text-splitter. For each
segment, we prompt GPT-3.5-turbo to generate 4 question-answer pairs based on
the segment. We then group continuous segments using Template B, where we can
control the resulted context length by concatenating different number of
segments. The books are randomly sampled from Books3 corpus, and the papers
are sampled from Arxiv, both coming from the Pile [14]. All the above fine-
tuning data are formatted in the manner of multi-turn conversations and limit
the context length up to 10240. To mitigate forgetting, we also include 5000
samples from the pre-train data.

##  Appendix C Results of Mistral

Table 5: Evaluation results of UltraGist (Mistral) on various lengthy-context compression tasks. Method | Document Comp. | Example Comp. | Code Comp.
---|---|---|---
NQA | Qasp | HpQA | Musiq | Gov | News | TREC | SSum | LCC | Repo
Mistral-7B | 21.6 | 29.1 | 37.5 | 18.7 | 31.7 | 23.9 | 71.0 | 43.4 | 57.1 | 54.4
UltraGist (Mistral) | 26.3 | 36.6 | 47.6 | 22.2 | 28.6 | 23.1 | 70.0 | 42.6 | 51.8 | 52.7

We apply UltraGist on Mistral-7B-Instruct (v0.2) as well. The implementation
and training settings are roughly the same as Llama-2. However, we increase
the segment size $w$ to 2048, and train it using 16K-length data for pre-
training and 20K-length data for fine-tuning.

Table 6: Evaluation of long-term memory on MSC. Method | Compression Ratio
---|---
4 | 8 | 16 | 24
Mistral-7B (Oracle) | 42.3
UltraGist (Mistral) | 41.4 | 38.1 | 32.6 | 30.1

We evaluate the model on MSC dataset. The performance of the backbone LLM,
i.e. Mistral-7B-Instruct, is considered as the oracle because it can take in
the entire conversation history without compression. The results are shown in
Table 6. It can be observed that UltraGist is able to achieve fine-grained
compression of the context, exhibiting little compression loss in compaison
with the oracle performance. It also performs well when adopting various
compression ratios (especially at $\times$4 and $\times$8 compression),
validating the flexibility of our approach. However, as the compression ratio
grows to $\times$16 and $\times$24, UltraGist’s performance drops faster than
UltraGist applied on Llama-2 (see Table 1). This may be because Mistral-7B
uses grouped query attention to reduce KV cache size, which results in limited
space for further compression. Therefore, we encourage future research to
investigate the combinative effect of compressing KV head-wise (e.g. GQA),
dimension-wise (e.g. MLA [9]), and sequence-wise (e.g. UltraGist).

In addition, we report the model’s performance on various tasks that require
compressing lengthy context in Table 5. Note that Mistral-7B itself take in
the 32K context without compression. It can be observed that UltraGist
achieves competitive performance against the performance of Mistral-7B when
compressing documents and few-shot examples, verifying its high-quality
compression effect for lengthy context. Surprisingly, it even outperforms
Mistral-7B on several tasks such as NarrativeQA and Qasper, which implies the
high potential of UntraGist to improve the long-context utilization of the
LLM. However, there is information loss when compressing code. We conjecture
this degradation is due to the lack of code-related data during fine-tuning.

##  Appendix D Broader Impact

UltraGist achieves fine-grained and flexible context compression effect, which
may benefit many real-world scenarios involving lengthy context, such as long
document understanding/summarization, and lifelong chatbot with long-term
memory. Besides, it can reduce the KV cache of LLMs and improve inference
efficiency for the backbone LLM, leading to significant resource savings and
contributes to environmental sustainability. As a downside, since UltraGist is
applied to the LLM as a plug-in, it inherits the internal biases of the LLM.
Consequently, there is a risk of generating unreliable or harmful content,
which underscores the need for careful monitoring.