# On the triple tensor product of nilpotent Lie algebras

afsaneh shamsaki  School of Mathematics and Computer Science
Damghan University, Damghan, Iran<EMAIL_ADDRESS>and  Peyman
Niroomand<EMAIL_ADDRESS><EMAIL_ADDRESS>School of Mathematics
and Computer Science
Damghan University, Damghan, Iran

###### Abstract.

In this paper, we give the explicit structure of $\otimes^{3}H$ and
$\wedge^{3}H$ where $H$ is a generalized Heisenberg Lie algebra of rank at
most $2.$ Moreover, for a non-abelian nilpotent Lie algebra $L,$ we obtain an
upper bound for the dimension of $\otimes^{3}L.$

###### Key words and phrases:

Schur multiplier; nilpotent Lie algebra, capable Lie algebra, triple tensor
product

###### 1991 Mathematics Subject Classification:

17B30, 17B05, 17B99

Corresponding Author: Peyman Niroomand

##  1\.  Introduction

It is known from [8, Theorem 3.1] that the dimension of tensor product of a
Lie algebra $L$ of dimension $n$ when $\dim L^{2}=m$ is bounded by
$(n-m)(n-1)+2$ and the structure of the tensor product of a Heisenberg Lie
algebra is given. Let $H$ be a generalized Heisenberg Lie algebra of rank at
most $2.$ In this paper, we complete the explicit structure of the tensor
square and the exterior square of $H.$ Moreover, we give the triple tensor
product and the triple exterior product of $H.$ Finally, we show that
dimension of the triple tensor of a Lie algebra $L$ with derived subalgebra of
dimension $m$ is bounded by $n(n-m)^{2}.$

##  2\.  Preliminary and known results

Let $\mathbb{F}$ be a fixed field and $[,]$ is used to denote the Lie bracket.
For any two Lie algebras $L$ and $K,$ we say there exists an action $L$ on $K$
if an $\mathbb{F}$-bilinear map $L\times K\rightarrow K,$
$(l,k)\mapsto~{}^{l}k$ satisfying

${}^{[l,l^{\prime}]}k=^{l}(^{l^{\prime}}k)-^{l^{\prime}}(^{l}k)\qquad\text{and}\qquad^{l}[k,k^{\prime}]=[^{l}k,k^{\prime}]+[k,^{l}k^{\prime}]$

for all $l,l^{\prime}\in L$ and $k,k^{\prime}\in K.$
The actions are compatible if

(2.1)
$^{{}^{l}k}l^{\prime}=[l^{\prime},^{k}l]\qquad\text{and}\qquad^{{}^{k}l}k^{\prime}=[k^{\prime},^{l}k]$

for all $k,k^{\prime}\in K,$ $l,l^{\prime}\in L.$
Clearly, a Lie algebra $L$ acts on itself by ${}^{l^{\prime}}l=[l^{\prime},l]$
for all $l,l^{\prime}\in L.$ Morover, for all Lie algebras $Q,$ a bilinear
function $\varphi:L\times K\rightarrow Q$ is called a Lie pairing if for all
$l,l^{\prime}\in L$ and $k,k^{\prime}\in K,$ we have

$\displaystyle\varphi([l,l^{\prime}],k)=\varphi(l,^{l^{\prime}}k)-\varphi(l^{\prime},^{l}k),$
$\displaystyle\varphi(l,[k,k^{\prime}])=\varphi(^{k^{\prime}}l,k)-\varphi(^{k}l,k^{\prime}),$
$\displaystyle\varphi(^{k}l,^{l^{\prime}}k^{\prime})=-[\varphi(l,k),\varphi(l^{\prime},k^{\prime})].$

The non-abelian tensor product $L\otimes K$ is the Lie algebra generated by
symbols $l\otimes k$ for all $l\in L$ and $k\in K$ with the following defining
relations

$\displaystyle c(l\otimes k)=cl\otimes k=l\otimes ck,$
$\displaystyle(l+l^{\prime})\otimes k=l\otimes k+l^{\prime}\otimes k,$
$\displaystyle l\otimes(k+k^{\prime})=l\otimes k+l\otimes k^{\prime},$
$\displaystyle[l,l^{\prime}]\otimes
k=l\otimes^{l^{\prime}}k-l^{\prime}\otimes~{}^{l}k,$ $\displaystyle
l\otimes[k,k^{\prime}]=^{k^{\prime}}l\otimes k-^{k}l\otimes k^{\prime},$
$\displaystyle[(l\otimes k),(l\otimes
k^{\prime})]=-^{k}l\otimes~{}^{l^{\prime}}k^{\prime}$

for all $c\in\mathbb{F},$ $l,l^{\prime}\in L$ and $k,k^{\prime}\in K.$ If
$L=K$ and all actions are Lie multiplication, then $L\otimes L$ is called the
non-abelian tensor square of $L.$
The next two propositions are used in the rest of paper.

######  Proposition 2.1.

[3, Proposition 3] There are actions of both $L$ and $K$ on $L\otimes K$ given
by

${}^{l^{\prime}}(l\otimes k)=[l,l^{\prime}]\otimes
k+l\otimes(^{l^{\prime}}k),$ ${}^{k^{\prime}}(l\otimes
k)=(^{k^{\prime}}l)\otimes k+l\otimes[k^{\prime},k]$

for all $l,l^{\prime}\in L$ and $k,k^{\prime}\in K.$

Recall that a Lie algebra $H$ is called a generalized Heisenberg of rank $n$
if $H^{2}=Z(H)$ and $\dim H^{2}=n.$ If $n=1,$ then $H$ is a Heisenberg Lie
algebra that is more well-known. Such Lie algebras are odd dimension and
$H\cong H(m)=\langle x_{i},y_{i},z\mid[x_{i},y_{i}]=z,1\leq i\leq m\rangle.$

######  Proposition 2.2.

[8, Proposition 3.3] Let $H(m)$ be a Heisenberg Lie algebra. Then

$H(m)\otimes H(m)\cong H(m)/H(m)^{2}\otimes H(m)/H(m)^{2}$

for all $m$ such that $m\geq 2.$ In the case $m=1,$ $H(1)\otimes H(1)$ is an
abelian Lie algebra of dimension $6.$

Let $L\square L$ be the submodule of $L\otimes L$ generated by the elements
$l\otimes l.$ Then the exterior square $L\wedge L$ of $L$ is the quotient
$L\wedge L\cong L\otimes L/L\square L.$ For all $l\otimes l^{\prime}\in
L\otimes L,$ we denote the coset $l\otimes l^{\prime}+L\square L$ by $l\wedge
l^{\prime}.$
The Schur multiplier of a Lie algebra $L$ is defined as $\mathcal{M}(L)\cong
R\cap F^{2}/[R,F]$ where $L\cong F/R$ and $F$ is a free Lie algebra. It is
known that the Lie algebra $\mathcal{M}(L)$ is abelian and independent of the
choice of the free Lie algebra $F$ (see [1, 2] for more information).

######  Lemma 2.3.

[5, Lemma 23] Let $L$ be an $n$-dimensional abelian Lie algebra. Then
$\dim\mathcal{M}(L)=\frac{1}{2}n(n-1).$

######  Lemma 2.4.

[6, Lemma 2.8 ] Let $L$ be a nilpotent Lie algebra of class two. Then $L\wedge
L\cong\mathcal{M}(L)\oplus L^{2}.$

Let $A(n)$ be used to denote an abelian Lie algebra of dimension $n.$ Then we
have

######  Lemma 2.5.

[11, Corollary 2.5 and Lemma 3.2]

  * (i).

$A(n)\wedge A(n)\cong\mathcal{M}(A(n)),$

  * (ii).

$H(1)\wedge H(1)\cong A(3),$

  * (iii).

$H(m)\wedge H(m)\cong A(2m^{2}-m)$ for all $m\geq 2.$

The next theorem shows that the exterior product of a Lie algebra $L$ is a
direct summand of $L\otimes L$ when $L/L^{2}$ is of finite dimension.

######  Theorem 2.6.

[9, Theorem 2.5] Let $L/L^{2}$ be a finite dimensional Lie algebra. Then

$L\otimes L\cong L\wedge L\oplus L\square L.$

In the following, the notations $L_{6,22}(\varepsilon),$ $L_{5,8},$
$L_{6,7}^{(2)}(\eta)$ and $L_{1}$ are taken from [4] and [10].

######  Proposition 2.7.

[6, Theorem 2.15] The Schur multiplier of Lie algebras
$L_{6,22}(\varepsilon),$ $L_{5,8},$ $L_{6,7}^{(2)}(\eta),$ and $L_{1}$ are
abelian Lie algebras of dimension $8,6,8$ and $9,$ respectively.

A Lie algebra $L$ is called capable provided that $L\cong H/Z(H)$ for a Lie
algebra $H.$

######  Proposition 2.8.

[10, Proposition 3.1] Let $H$ be a non-capable generalized Heisenberg Lie
algebra of rank $2$ such that $\dim H=n.$ Then

$\displaystyle\dim\mathcal{M}(H)=\frac{1}{2}(n-3)(n-2)-2$

or

$\displaystyle\dim\mathcal{M}(H)=\frac{1}{2}(n-1)(n-4)+1.$

##  3\. main results

We know from Proposition 2.1, $L$ acts on $L\otimes L.$ On the other hand, the
tensor product $L\otimes L$ acts on $L$ by ${}^{t}l=~{}~{}^{\lambda(t)}l$ for
all $t\in L\otimes L$ and $l\in L$ such that $\lambda:L\otimes L\rightarrow L$
is a homomorphism given by $a\otimes b\mapsto[a,b].$ These actions are
compatible and we can construct the triple tensor product
$\otimes^{3}L=(L\otimes L)\otimes L.$
In this section, we give the explicit structure of $L\otimes L=\otimes^{2}L$
and $L\wedge L=\wedge^{2}L$ when $L$ is a generalized Heisenberg of rank $2.$
Moreover, we obtain $\otimes^{3}L$ and $\wedge^{3}L$ when $L$ is a generalized
Heisenberg of rank at most $2$. Also, for a non-abelian nilpotent Lie algebra
$L,$ we give an upper bound for the triple tensor product of $L.$
The following lemmas are useful instruments in the next.

######  Lemma 3.1.

Let $L$ be a Lie algebra of nilpotency class two. Then

  * (i).

$L\otimes L$ acts trivially on $L.$

  * (ii).

$(L\otimes L)\otimes L$ is an abelian Lie algebra.

###### Proof.

(i). We know that $L$ is a Lie algebra of nilpotency class two, so
$\gamma_{3}(L)=0.$ By considering action of $L\otimes L$ on $L,$ we have
${}^{\lambda(a\otimes b)}c=[[a,b],c]=0$ for all $a\otimes b\in L\otimes L$ and
$c\in L.$ Therefore $L\otimes L$ acts trivially on $L.$
(ii). By using the relations of the non-abelian tensor on $(L\otimes L)\otimes
L$ and the part (i), we have

$\displaystyle[(a\otimes b)\otimes c,(a^{\prime}\otimes b^{\prime})\otimes
c^{\prime}]$ $\displaystyle=-(^{c}(a\otimes b))\otimes^{a^{\prime}\otimes
b^{\prime}}c$ $\displaystyle=-^{c}(a\otimes b)\otimes 0=0$

Therefore $[(a\otimes b)\otimes c,(a^{\prime}\otimes b^{\prime})\otimes
c^{\prime}]=0$ for all $(a\otimes b)\otimes c,(a^{\prime}\otimes
b^{\prime})\otimes c^{\prime}\in(L\otimes L)\otimes L,$ and so $(L\otimes
L)\otimes L$ is an abelian Lie algebra. ∎

Let $L$ be a nilpotent Lie algebra of class $k$, $i_{L}:L\rightarrow L$ be the
identity homomorphism and $\gamma_{k}(L)$ be the $k$-th term of the lower
central series of $L$ and $\varphi:\gamma_{k}(L)\rightarrow L$ be a natural
homomorphism. Define homomorphisms $\overline{\varphi}=(\varphi\otimes
i_{L})\otimes i_{L}:(\gamma_{k}(L)\otimes L)\otimes L\rightarrow\otimes^{3}L$
and $\gamma:(L\otimes L)\otimes\gamma_{k}(L)\rightarrow\otimes^{3}L$ given by
$(a\otimes b)\otimes c\mapsto(a\otimes b)\otimes c$. Then

######  Lemma 3.2.

Let $L$ be a non-abelian nilpotent Lie algebra of class $k,$ then
$\mathrm{Im}\gamma\subseteq\mathrm{Im}\overline{\varphi}.$

###### Proof.

By using the relation $k\otimes[l,l^{\prime}]=[l^{\prime},k]\otimes
l-[l,k]\otimes l^{\prime}$ on $L\otimes L,$ we can see that $(a\otimes
b)\otimes[x_{1},\dots,x_{k-1},x_{k}]\in(\gamma_{k}(L)\otimes L)\otimes L$ for
all $a,b,x_{1},\dots,x_{k-1},x_{k}\in L.$ Therefore the result follows. ∎

The next result plays a key role in proving the next theorem.

######  Proposition 3.3.

If $L$ is a nilpotent Lie algebra of class $k,$ then

$(\gamma_{k}(L)\otimes L)\otimes L\overset{(\varphi\otimes i_{L})\otimes
i_{L}}{\longrightarrow}\otimes^{3}L\longrightarrow\otimes^{3}L/\gamma_{k}(L)\rightarrow
0,$

is exact.

###### Proof.

By using Lemma 3.2 and [13, Proposition 1.3], the result follows. ∎

In the following proposition, we are going to determine the structure of
$\otimes^{3}H(m)$ and $\wedge^{3}H(m)$ for all $m\geq 1.$

######  Proposition 3.4.

Let $H(m)$ be a Heisenberg Lie algebra. Then

$\otimes^{3}H(m)\cong\begin{cases}A(12)&\text{if }m=1,\\\
A(2^{3}m^{3})&\text{if }m\geq 2\\\ \end{cases}$

and

$\wedge^{3}H(m)\cong\begin{cases}A(2)&\text{if }m=1,\\\
A(4m^{3}-4m^{2}-m)&\text{if }m\geq 2.\\\ \end{cases}$

###### Proof.

We claim that $(\varphi(L^{2})\otimes L)\otimes L\cong(\varphi(L^{2})\otimes
L^{ab})\otimes L^{ab}.$ It is clear that $\varphi(L^{2})$ and $L$ act
trivially on each other. Hence $(\varphi(L^{2})\otimes L)\otimes
L\cong(\varphi(L^{2})\otimes L^{ab})\otimes L.$ Also, $\varphi(L^{2})\otimes
L^{ab}$ and $L$ act trivially on each other. Thus $(\varphi(L^{2})\otimes
L)\otimes L\cong(\varphi(L^{2})\otimes L^{ab})\otimes L^{ab}.$ By using the
following exact sequence

$L^{2}\otimes L\overset{\varphi\otimes i_{L}}{\longrightarrow}L\otimes
L\longrightarrow L/L^{2}\otimes L/L^{2}\longrightarrow 0,$

we have

(3.1) $\dim L\otimes L=\dim L^{ab}\otimes L^{ab}+\dim\mathrm{Im}\varphi\otimes
i_{L}=\dim L^{ab}\otimes L^{ab}+\dim\varphi(L^{2})\otimes L.$

Let $L\cong H(1).$ Since $H(1)\otimes H(1)\cong A(6)$ and $H(1)^{ab}\otimes
H(1)^{ab}\cong A(4),$ we have $\dim\varphi(L^{2})\otimes L=2$ by using (3.1).
Since $(\varphi(L^{2})\otimes L)\otimes L\cong(\varphi(L^{2})\otimes
L^{ab})\otimes L^{ab}$,

(3.2) $\dim(\varphi(L^{2})\otimes L)\otimes L=(\dim\varphi(L^{2})\otimes
L^{ab})\dim L^{ab}.$

Now, by using (3.2), Proposition 3.3 and Lemma 3.1 (ii), we have

$\dim\otimes^{3}L=\dim\otimes^{3}L^{ab}+\dim(\varphi(L^{2})\otimes
L^{ab})\otimes L^{ab}=12$

and so $\otimes^{3}L\cong A(12).$ Similarly, if $L\cong H(m)$ for all $m\geq
2,$ then $\otimes^{3}L\cong A(2^{3}m^{3}).$
Let $L\cong H(1),$ then $\dim\varphi(L^{2})\wedge
L=\dim\wedge^{2}L-\dim\wedge^{2}L^{ab}=2$ by using the following exact
sequence

$L^{2}\wedge L\overset{\varphi\wedge i_{L}}{\longrightarrow}L\wedge
L\longrightarrow L/L^{2}\wedge L/L^{2}\longrightarrow 0$

and Lemma 2.5. We can easily see $(\varphi(L^{2})\wedge L)\wedge
L\cong(\varphi(L^{2})\wedge L^{ab})\wedge L^{ab}$ and since
$\varphi(L^{2})\wedge L\cong A(2),$ we have $(\varphi(L^{2})\wedge L)\wedge
L\cong A(2)\wedge A(2).$ Thus $(\varphi(L^{2})\wedge L)\wedge L\cong A(1)$ by
using Lemma 2.5. On the other hand, $\wedge^{3}L/L^{2}\cong(A(2)\wedge
A(2))\wedge A(2),$ hence $\wedge^{3}L/L^{2}\cong A(1)$ by using Lemma 2.5 and
[7, Lemma 2.1.7]. Consider the following exact sequence

$(L^{2}\wedge L)\wedge L\overset{(\varphi\wedge i_{L})\wedge
i_{L}}{\longrightarrow}\wedge^{3}L\longrightarrow\wedge^{3}L/L^{2}\rightarrow
0.$

Therefore

$\displaystyle\dim\wedge^{3}L$
$\displaystyle=\dim\wedge^{3}L/L^{2}+\dim\mathrm{Im}(\varphi\wedge i_{L})$
$\displaystyle=\dim\dim\wedge^{3}L/L^{2}+\dim(\varphi(L^{2})\wedge L)\wedge L$
$\displaystyle=2.$

Since $\otimes^{3}L$ is an abelian Lie algebra by using Lemma 3.1 (ii), we
have $\wedge^{3}L\cong A(2).$ Similarly, if $L\cong H(m)$ for all $m\geq 2,$
then $\wedge^{3}L\cong A(4m^{3}-4m^{2}-m).$
∎

The following theorem determines all $n$-dimensional capable generalized
Heisenberg Lie algebras of rank $2.$

######  Theorem 3.5.

[10, Theorem 3.6] Let $H$ be a generalized Heisenberg Lie algebra such that
$\dim H=n$ and $\dim H^{2}=2.$ Then $H$ is capable if and only if $n=5,6$ or
$7$ and $H\cong L_{5,8}=\langle
x_{1},\ldots,x_{5}\big{|}[x_{1},x_{2}]=x_{4},[x_{1},x_{3}]=x_{5}\rangle,$
$H\cong L_{6,22}(\varepsilon)=\langle
x_{1},\ldots,x_{6}\big{|}[x_{1},x_{2}]=x_{5}=[x_{3},x_{4}],[x_{1},x_{3}]=x_{6},[x_{2},x_{4}]=\epsilon
x_{6}\rangle$ where
$\varepsilon\in\mathbb{F}/(\stackrel{{\scriptstyle*}}{{\sim}}),$
$L_{6,7}^{(2)}(\eta)=\langle
x_{1},\ldots,x_{6}\mid[x_{1},x_{2}]=x_{5},[x_{1},x_{3}]=x_{6},[x_{2},x_{4}]=\eta
x_{6},[x_{3},x_{4}]=x_{5}+x_{6}$ where $\eta\in\\{0,\omega\\}$ or $H\cong
L_{1}=\langle
x_{1},\ldots,x_{7}\big{|}[x_{1},x_{2}]=x_{6}=[x_{3},x_{4}],[x_{1},x_{5}]=x_{7}=[x_{2},x_{3}]\rangle.$

The next propositions determine the structure of $\otimes^{2}H,$
$\wedge^{2}H,$ $\otimes^{3}H$ and $\wedge^{3}H$ when $H$ is a generalized
Heisenberg Lie algebra of rank $2.$

######  Proposition 3.6.

Let $H$ be an $n$-dimensional generalized Heisenberg Lie algebra of rank $2.$

  * (i).

If $H$ is non-capable, then

$\otimes^{2}H\cong\begin{cases}A((n-2)^{2})&\text{if }Z^{\wedge}(H)=H^{2},\\\
A((n-2)^{2}+2)&\text{otherwise}\\\ \end{cases}$

and

$\wedge^{2}H\cong\begin{cases}A(\frac{1}{2}(n-3)(n-2))&\text{if
}Z^{\wedge}(H)=H^{2},\\\ A(\frac{1}{2}(n-1)(n-4)+3)&\text{otherwise.}\\\
\end{cases}$

If $H$ is capable, then

$\otimes^{2}H\cong\begin{cases}A(14)&\text{if }H\cong L_{5,8},\\\
A(20)&\text{if }H\cong
L_{6,22}(\varepsilon)~{}~{}\text{or}~{}~{}L_{6,7}^{(2)}(\eta),\\\
A(26)&\text{if }H\cong L_{1}\\\ \end{cases}$

and

$\wedge^{2}H\cong\begin{cases}A(8)&\text{if }H\cong L_{5,8},\\\ A(10)&\text{if
}H\cong L_{6,22}(\varepsilon)~{}~{}\text{or}~{}~{}L_{6,7}^{(2)}(\eta),\\\
A(11)&\text{if }H\cong L_{1}.\\\ \end{cases}$

###### Proof.

(i). First, we obtain the exterior product of $H.$ By invoking Lemma 2.4 and
Proposition 2.8, we have $\wedge^{2}H=\mathcal{M}(H)\oplus H^{2}\cong
A(\frac{1}{2}(n-3)(n-2))$ when $Z^{\wedge}(H)=H^{2}.$ By a similar way, we can
see that $\wedge^{2}H=A(\frac{1}{2}(n-1)(n-4)+3)$ when $Z^{\wedge}(H)\neq
H^{2}.$
Since $H\square H\cong H^{ab}\square H^{ab}\cong A(\frac{1}{2}(n-m)(n-m+1))$
by [9, Lemma 2.3] and using Theorem 2.6, we have $\otimes^{2}H=H\square
H\oplus H\wedge H\cong A(n^{2}-2n+3)$ when $Z^{\wedge}(H)=H^{2}.$ By a similar
way, we can see that $\otimes^{2}H\cong A(n^{2}-2n+5)$ when $Z^{\wedge}(H)\neq
H^{2}.$
(ii). The proof is similar to the part (i). ∎

######  Proposition 3.7.

Let $H$ be $n$-dimensional generalized Heisenberg Lie algebra of rank $2.$

  * (i).

Let $H$ is non-capable. Then

$\otimes^{3}H\cong\begin{cases}A((n-2)^{3})&\text{if }Z^{\wedge}(H)=H^{2},\\\
A((n^{2}-4n+6)(n-2))&\text{otherwise.}\\\ \end{cases}$

and

$\wedge^{3}H\cong\begin{cases}A(\frac{1}{2}(n-2)(n^{2}-6n+7))&\text{if
}Z^{\wedge}(H)=H^{2},\\\ A(\frac{1}{2}n(n^{2}-8n+23)-14)&\text{otherwise.}\\\
\end{cases}$

If $H$ is capable, then

$\otimes^{3}H\cong\begin{cases}A(42)&\text{if }H\cong L_{5,8},\\\
A(80)&\text{if }H\cong
L_{6,22}(\varepsilon)~{}~{}\text{or}~{}~{}L_{6,7}^{(2)}(\eta),\\\
A(130)&\text{if }H\cong L_{1}\\\ \end{cases}$

and

$\wedge^{3}H\cong\begin{cases}A(12)&\text{if }H\cong L_{5,8},\\\
A(20)&\text{if }H\cong
L_{6,22}(\varepsilon)~{}~{}\text{or}~{}~{}L_{6,7}^{(2)}(\eta),\\\
A(39)&\text{if }H\cong L_{1}.\\\ \end{cases}$

###### Proof.

The proof is obtained by a similar way used in the proof of Proposition 3.4. ∎

######  Theorem 3.8.

Let $L$ be an $n$-dimensional non-abelian nilpotent Lie algebra with derived
subalgebra of dimension $m.$ Then

$\dim\otimes^{3}L\leq n(n-m)^{2}.$

In particular, for $m=1$ the equality holds if and only if $L\cong H(1).$

###### Proof.

Let $L$ be an $n$-dimesional nilpotent Lie algebra of nilpotency class $k.$ We
proceed by induction on $n.$ Since $L$ is non-abelian, $n\geq 3.$ If $n=3,$
then $L\cong H(1)$ by the classification of all nilpotent Lie algebras is
given in [4]. By using Proposition 3.4, we have $\otimes^{3}H(1)\cong A(12).$
Therefore the result holds. Since $\gamma_{k}(L)$ is central, $\gamma_{k}(L)$
and $L$ act trivially on each other and so we have $\gamma_{k}(L)\otimes
L\cong\gamma_{k}(L)\otimes L^{ab}$ by using [3, Proposition 5]. By the same
reason and using [3, Proposition 5], we have $(\gamma_{k}(L)\otimes L)\otimes
L\cong(\gamma_{k}(L)\otimes L^{ab})\otimes L^{ab}.$ Consider two cases.
Let $L/\gamma_{k}(L)$ is abelian. Then $L$ is a nilpotant Lie algebra of class
two and $\gamma_{k}(L)=L^{2}.$ Thus
$\dim\otimes^{3}L/\gamma_{k}(L)=\dim\otimes^{3}L/L^{2}=(n-m)^{3}.$ By using
Proposition 3.3, we have

$\displaystyle\dim\otimes^{3}L$
$\displaystyle\leq\dim\otimes^{3}(L/\gamma_{k}(L))+\dim((\gamma_{k}(L)\otimes
L)\otimes L)$ $\displaystyle=(n-m)^{3}+m(n-m)^{2}=n(n-m)^{2}.$

If $L/\gamma_{k}(L)$ is non-abelian, then $\dim
L/\gamma_{k}(L)=n-\dim\gamma_{k}(L)$ and
$\dim(L/\gamma_{k}(L))^{2}=m-\dim\gamma_{k}(L),$ hence

(3.3) $\dim\otimes^{3}(L/\gamma_{k}(L))\leq(n-\dim\gamma_{k}(L))(n-m)^{2}$

by using the induction hypothesis. Now, Proposition 3.3 and (3.3) imply that

$\displaystyle\dim\otimes^{3}L$
$\displaystyle\leq\dim\otimes^{3}(L/\gamma_{k}(L))+\dim((\gamma_{k}(L)\otimes
L)\otimes L)$
$\displaystyle\leq(n-\dim\gamma_{k}(L))(n-m)^{2}+\dim\gamma_{k}(L)(n-m)^{2}$
$\displaystyle=n(n-m)^{2}.$

Let $m=1.$ Then $L\cong H(k)\oplus A(n-2k-1)$ for all $k\geq 1$ by using [12,
Lemma 3.3]. By using Proposition 3.4 the equality holds if and only if $L\cong
H(1).$ ∎

## References

  * [1] Batten, P., Moneyhun, K. and Stitzinger, E.; On characterizing nilpotent Lie algebras by their multipliers. Comm. Algebra 24 (1996), no. 14, 4319–4330.
  * [2] Batten, P. and Stitzinger, E.; On covers of Lie algebras. Comm. Algebra 24 (1996), no. 14, 4301–4317.
  * [3] Ellis, G.; A non-abelian tensor product of Lie algebras. Glasgow Mathematical Journal 33, no. 1 (1991), 101–120.
  * [4] De Graaf, W. A.; Classification of 6-dimensional nilpotent Lie algebras over fields of charac-teristic not 2. J. Algebra 309, no. 2, (2007), 640–653.
  * [5] Moneyhun, K.; Isoclinisms in Lie algebras. Algebras Groups Geom. 11 , no. 1, (1994), 9–22.
  * [6] Johari, F., and Niroomand, P.; Certain functors of nilpotent Lie algebras with the derived subalgebra of dimension two. J. Algebra Appl. 19, no. 01, (2020), 2050012.
  * [7] Johari, F., Parvizi, M., and Niroomand, P. Capability and Schur multiplier of a pair of Lie algebras. J. Geom. Phys. 114, (2017), 184–196.
  * [8] Niroomand, P.; On the tensor square of non-abelian nilpotent finite-dimensional Lie algebras. Linear and Multilinear Algebra 59, no. 8 (2011), 831–836.
  * [9] Niroomand, P.; Some properties on the tensor square of Lie algebras. J. Algebra Appl. 11 (2012), no. 5, 1250085, 6 pp.
  * [10] P. Niroomand, F. Johari and M. Parvizi, Capable Lie algebras with the derived subalgebra of dimension 2 over an arbitrary field. Linear and Multilinear Algebra 67 , no. 3, (2019), 542-554.
  * [11] Niroomand, P., Parvizi, M. and Russo, F. G.; Some criteria for detecting capable Lie algebras. J. Algebra 384 (2013), 36–44.
  * [12] Niroomand, P.; On dimension of the Schur multiplier of nilpotent Lie algebras. Cent. Eur. J. Math. 9 (2011), no. 1, 57–64.
  * [13] Salemkar, A.R., Tavallaee, H., Mohammadzadeh, H. and Edalatzadeh, B.; On the non-abelian tensor product of Lie algebras. Linear and Multilinear Algebra 58, no. 3, (2010), 333–341.


# Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A
Comparative Study

Zooey Nguyen Aitomatic, Inc. (except as noted, all authors are from Aitomatic)
Anthony Annunziata IBM Research Vinh Luong  Sang Dinh  Quynh Le  Anh Hai Ha
Chanh Le  Hong An Phan  Shruti Raghavan  Christopher Nguyen

(April 2024)

###### Abstract

This paper investigates the impact of domain-specific model fine-tuning and of
reasoning mechanisms on the performance of question-answering (Q&A) systems
powered by large language models (LLMs) and Retrieval-Augmented Generation
(RAG). Using the FinanceBench SEC financial filings dataset, we observe that,
for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves
better accuracy than generic models, with relatively greater gains
attributable to fine-tuned embedding models. Additionally, employing reasoning
iterations on top of RAG delivers an even bigger jump in performance, enabling
the Q&A systems to get closer to human-expert quality. We discuss the
implications of such findings, propose a structured technical design space
capturing major technical components of Q&A AI, and provide recommendations
for making high-impact technical choices for such components. We plan to
follow up on this work with actionable guides for AI teams and further
investigations into the impact of domain-specific augmentation in RAG and into
agentic AI capabilities such as advanced planning and reasoning.111This
collaborative work is facilitated by the AI Alliance https://thealliance.ai,
in the interest of open development, open science and open AI. All of the
authors are affiliated with the AI Alliance. Except for specifics disclosed in
this paper, all intellectual properties of the work belong to respective
owners.

Keywords question-answering · fine-tuning · reasoning · retrieval-augmented
generation · embedding models · large language models · SEC financial reports
dataset

##  1 Introduction

AI-powered question-answering (Q&A) systems have emerged as important tools,
alongside established search technologies, to enable quick access to relevant
information and knowledge from large digital sources that are complex and
time-consuming for humans to navigate. Advancements in large language models
(LLMs) have revolutionized the field of Q&A, with models like GPT-3 [3], BERT
[6], and RoBERTa [18] demonstrating remarkable abilities in understanding and
generating human-like text. However, the effectiveness of such models in
handling domain-specific questions that require specialized knowledge is
limited.

Retrieval-augmented generation (RAG) techniques, which combine information
retrieval and generative models [17], have shown promise in boosting the
quality of LLM output in Q&A tasks. RAG systems leverage the strengths of both
retrieval and generation components to provide contextually relevant and
informative responses. While there is a lack of established quantification of
RAG accuracy, early findings suggest that generic RAG does not perform well in
complex domains such as finance. In one instance, RAG based on generic LLMs
such as GPT-4-Turbo fails to answer 81% of the questions derived from
Securities and Exchange Commission (SEC) financial filings [12].

The underperformance of generic LLMs and RAG in domain-specific Q&A has
motivated us to research into and create methods to adapt and extend such
models and techniques. In this paper, we describe and quantify the gains in
accuracy from two major methods: model fine-tuning and iterative reasoning.

Fine-tuning is a way to adapt language models to specific domains and tasks
devlin2019bert, liu2019roberta by training them on domain-specific data and
having them capture the nuances and intricacies of a particular field. In a
typical RAG workflow, there are two principal models that can be considered
for fine-tuning: the Embedding Model, whose tasks are indexing the information
in the corpus and retrieving information relevant to the posed question, and
the Generative Model, whose task is synthesizing an answer. We have performed
experiments and analyses comparing baseline generic-LLM-based RAG and RAG with
one or both of fine-tuned retrieval and fine-tuned generation on the
FinanceBench dataset, and observed promising gains from fine-tuning.

In addition, incorporating iterative reasoning and planning is crucial for
intelligent agents to effectively tackle real-world, complex tasks as
highlighted by [16] and [27]. Motivated by those works and by our first-hand
experience in industrial applications, we have explored integrating the
Observe-Orient-Decide-Act (OODA) loop [2], a well-established iterative
reasoning mechanism, with RAG-based Q&A. That too has shown promising
improvements, especially on highly complex questions.

By investigating the above areas, our main contributions are:

  1. 1.

We identify clear steps in LLM-based Q&A workflows where further innovation
can make large improvements to those workflows’ ultimate accuracy;

  2. 2.

We chart out the beginnings of a structured technical design space for the
primary technical choices for each of such major steps; and

  3. 3.

We quantify the accuracy impacts of certain technical choices, with firm
grounding on a concrete and sufficiently complex real-world dataset, and
highlight technical configurations from which we observe large gains. Such
quantification could serve developers and managers of AI systems in making
informed system-design decisions for quicker success.

The rest of the paper is organized as follows: Section 2 provides an overview
of related work in Q&A AI, focusing on RAG techniques, fine-tuning strategies,
and high-level planning and reasoning. Section 3 describes our framework for
improving on baseline generic RAG, proposes a structured design space for
technical decision-making, and overviews the FinanceBench dataset and the
technical configurations we have experimented with on that data. Section 4
presents results from our experiments, followed by a discussion of our
findings in Section 5. Finally, Section 6 concludes and outlines future
directions.

##  2 Related Work

The introduction of the Transformer architecture [23] laid the foundation for
Q&A AI, enabling the creation of models like BERT [6], RoBERTa [18], and GPT-3
[3], now commonly known as large language models (LLMs) from circa 2021.

One of the key challenges in applying LLMs to Q&A is handling long-form text
and maintaining context over extended sequences. Techniques like Longformer
[1] and Transformer-XL [5] have been developed to address this issue by
introducing attention mechanisms that efficiently process longer texts. These
advancements and works derived from them have paved the way for more effective
retrieval and understanding of relevant information in Q&A systems.

The seminal work by [17] introduced foundational techniques of RAG,
demonstrating its efficacy in knowledge-intensive NLP tasks by augmenting
generative models with retrieved documents to provide contextually rich
answers. Subsequent research has explored various aspects of RAG, such as the
integration of retrieval and generation within LLMs [8], prompt-guided
retrieval augmentation for non-knowledge-intensive tasks [9], and active
retrieval augmented generation [13]. The versatility of RAG systems has been
showcased in their ability to handle diverse question types and data formats.
UnifiedQA [15] demonstrates how advanced RAG systems can cross format
boundaries and address diverse Q&A tasks. Datasets like HybridQA [4] have
further highlighted the challenges and opportunities in dealing with hybrid
data formats and emphasized the need for Q&A systems to effectively navigate
between structured and unstructured data.

Another major R&D area in adapting LLMs to domain-specific contexts is fine-
tuning. Works like BERT [6] and RoBERTa [18] have shown the effectiveness of
fine-tuning in performing tasks in certain non-generic fields. More efficient
fine-tuning approaches, such as adapter layers [10] and model distillation
[21], have further enhanced the practicality of domain-specific LLMs.

As the field of Q&A continues to evolve, there is a growing emphasis on
developing models that can handle complex, multi-hop reasoning. The
integration of in-context learning with RAG [11] and the exploration of
unified frameworks like T5 [20] represent promising directions.

Several authors have emphasized the necessity of iterative reasoning and
planning in achieving real-world-capable intelligence. [16] proposes a
predictive world model architecture that enables abstract representation and
prediction at multiple timescales, addressing uncertainty and enabling more
human-like learning and reasoning. SELF-DISCOVER [27] is a framework that
allows LLMs to self-compose task-specific reasoning structures from atomic
reasoning modules, improving performance by up to 32% on challenging reasoning
benchmarks while requiring 10-40x fewer inference computations compared to
inference-intensive methods. SELF-REFINE [19], an approach for improving
initial outputs from LLMs through iterative feedback and refinement, improves
accuracy by approximately 20% on 7 diverse tasks including mathematical
reasoning using GPT-3.5 and GPT-4. To tackle the challenge of uncertainty in
multi-step reasoning, Self-Evaluation Guided Beam Search for Reasoning [24]
proposes a decoding algorithm integrating self-evaluation guidance via
stochastic beam search, a strategy that can pinpoint failures in logical
reasoning and leads to higher consistency and robustness.

Tree of Thoughts [25], a generalization of Chain of Thought, enables a new
framework for deliberate decision-making by considering multiple different
reasoning paths and self-evaluating choices to decide the next course of
action, as well as looking ahead or backtracking when necessary to make global
choices. For knowledge-intensive reasoning tasks, Knowledge-Augmented
Reasoning Distillation [14] fine-tunes small LMs to generate rationales
obtained from LLMs with augmented knowledge retrieved from an external
knowledge base. Through this approach, significantly smaller models, such as
250M-parameter T5, can outperform fine-tuned large models with 12 times as
many parameters.

PlanBench [22] is an extensible benchmark suite maintained by the automated
planning research community, covering diverse use cases including robotics,
aerial-vehicle navigation, control systems, energy management and integrated-
circuit design. The benchmark facilitates testing the capabilities of LLMs in
planning actions and reasoning about dynamic changes. Currently, state-of-the-
art LLMs are still struggling to be competitive on PlanBench.

##  3 Methodology

In this section, we first present our proposed framework for enhancing Q&A AI
performance. Based on the starting point of a typical RAG workflow, we
identify key areas that could be improved. Following this identification, we
briefly discuss the primary choices and key technical considerations for each
area. We then propose a systematic design space map that can help AI teams be
more deliberate and explicit in enumerating and making their technical choices
when building Q&A AI systems.

Next, we describe the FinanceBench dataset we used to develop and benchmark
models and techniques. We also provide an overview of our evaluation
methodology, which assesses the quality of outputs across different technical
configurations to ensure robustness and effectiveness.

###  3.1 Proposed Framework and Technical Design Space

A typical baseline RAG workflow today comprises the following steps:

  1. 1.

Information Indexing and Retrieval (the “R” step): The information sources are
first indexed, typically into vector embeddings in the most common case of
text-heavy sources. The typical means for indexing is an embedding model that
encodes chunks of content into fixed-sized latent vectors that capture the
semantic essence of such content. The same embedding model is then also used
to search for content that is semantically relevant to a particular question.
This retrieval step is typically based on similarity metrics or semantic
matching algorithms.

  2. 2.

Information Augmentation (the “A” step): Either or both of the indexing and
retrieving steps can be augmented with additional relevant metadata or human-
provided considerations. Metadata can come from the formats and structures of
the sources (e.g., PDF, CSV, etc.), while human-provided considerations may
come from expert knowledge of notable facts, rules and exceptions, and process
heuristics in the concerned domain. In certain domains with very specialized
jargon, it is also desirable to augment the workflow to boost the attention to
certain crucial keywords. The information augmentation would then influence
the relevance rankings of the retrieved candidate content chunks, yielding
more helpful supporting information for the downstream (Generation) step. The
overall effect is to give firmer grounding for that subsequent Generation
step, by helping index the information more appropriately, retrieve
information more relevantly, and add external knowledge that the generative
LLM does not natively have.

  3. 3.

Answer Generation (the “G” step): An LLM synthesizes a coherent and
understandable answer using a combination of the retrieved supporting
information and its own knowledge and generative capabilities. This method
typically yields more accurate and less error-prone results than using the LLM
alone.

Such RAG workflows perform inferencing in one pass-through of these steps,
from retrieving information relevant to the posed question, to augmenting such
information with metadata and user-provided considerations, to generating an
answer to the user.

We have identified, and shall elaborate in this paper, that there are
potential gains in accuracy from the following changes to the RAG workflows:

  1. 1.

A better Retriever: an embedding model that more precisely encodes the latent
meanings and connotations of certain important terms in domain-specific
technical jargon;

  2. 2.

A better Generator: an LLM that “knows” the logics and desired presentational
formats for the specific domain; and

  3. 3.

Iterative, Multi-step Reasoning employing RAG as an informational resource: a
mechanism that combines RAG-based queries with necessity-and-sufficiency
reasoning, task decomposition, and result verification, which are activities
typically performed by humans to ensure accurate outcomes.

Potential gains from other areas, such as from domain-specific Augmenters and
from higher-level Hierarchical Task Planning (HTP) mechanisms, will be
addressed in a future publication (see section 6).

####  3.1.1 Embedding Models for Indexing & Retrieval

Generic Embedding Models

OpenAI’s proprietary text-embedding-ada-002, released in December 2022, became
the state of the art for information retrieval tasks such as text search, code
search, and sentence similarity.. This embedding model offers unification of
capabilities (merging five separate models into a simplified one) and longer
context length (from 2048 to 8192). So far, OpenAI has not made available any
means of fine-tuning text-embedding-ada-002.

BAAI’s bge-large-en, an open-source embedding model, achieved first place on
the Massive Text Embedding Benchmark leaderboard upon its release in August
2023 [26]. BAAI provides fine-tuning capabilities on this embedding model,
which the LlamaIndex library provides an interface for. Its strong performance
on general text-retrieval tasks makes it an ideal candidate for a baseline
embedding model to observe the performance gain achieved by fine-tuning.

Fine-Tuned Embedding Models

Fine-tuning an embedding model alters the mapping of text into a latent vector
space. This text-to-vector mapping facilitates retrieval by identifying text
chunks closest to the query within that space.

Within a given domain, certain terms may map to different meanings than they
do in the colloquial sense, so fine-tuning the embedding model can change how
such terms are associated given the context. For example, in a general
embedding model, the word “pizza” might map closely to “Italy,” “pasta,” and
other food-related concepts. However, in the context of winter sports, a fine-
tuned model would associate it more with terms like “slowing,” “stopping,” and
beginner-level skiing instructions.

In our development and experimentation, we have fine-tuned embedding models
from BAAI’s bge-large-
en model. The process generally involves splitting textual documents via the
LlamaIndex library’s Sentence
Splitter to create semantic chunks, and then using OpenAI’s GPT-3.5 to
generate question-context pairs for each chunk for the synthetic dataset over
which to fine-tune, and then running fine-tune training for 5 epochs. In use
cases where there is already a natural question-and-resolution structure, such
as in CRM systems, we can directly extract question-context pairs without
needing to generate them using an LLM. In the particular use case of financial
analysis covered in this paper, we select a random subset of 100 query-context
pairs from the FinanceBench public dataset for the fine-tuning.

####  3.1.2 Generative Models for Answer Generation

Generic LLMs

OpenAI’s proprietary GPT-3.5, launched in 2022, was a breakthrough in public
adoption of LLM technology. This model has shown remarkable performance across
various NLP tasks, serving as a robust foundation for Q&A AI systems. Since
then, a fast-growing number of LLMs, both commercial and open-source, have
been released, creating a blooming ecosystem of LLM technology with
increasingly diverse capabilities.

OpenAI’s GPT-4, the company’s most advanced LLM to date, accepts multi-modal
input data including both text and images. Google DeepMind’s proprietary
Gemini models, offering many comparable capabilities, are direct competitors
to OpenAI’s GPT models. Startup Anthropic’s Claude models have been widely
adopted for their ability to handle very long contexts.

On the open-source front, Meta’s Llama2 model family has seen the widest
adoption, spawning an ecosystem of AI tools based around it. HuggingFace hosts
a number of alternatives to Llama2, with the Falcon model family being among
the most notable.

Fine-Tuned LLMs

Fine-tuning an LLM requires optimal examples of conversational history
including query-answer pairs. At inference time in a RAG system, the user’s
message is augmented with document texts that the LLM can draw on to answer.
However, fine-tuning an LLM typically focuses solely on query-answer pairs to
isolate the answering behavior to depend only on the query, rather than the
retrieved documents.

In our development and experimentation, we fine-tune on gpt-3.5-turbo-0125
using OpenAI’s fine-tuning API. In the particular use case of financial
analysis covered in this paper, we select a random subset of 100 query-context
pairs directly from the FinanceBench public dataset, then run fine-tune
training for 5 epochs. For consistency we use the same random subset as is
selected in the embedding model fine-tuning discussed above.

####  3.1.3 Iterative Reasoning

OODA Reasoning Paradigm

The Observe-Orient-Decide-Act (OODA) loop is a well-established iterative
reasoning framework emphasizing continuous adaptation and decision-making in
complex environments. Originally developed by military strategist John Boyd
[2], the OODA loop has been widely applied in various domains, including
business, sports, and healthcare [7].

The OODA loop consists of four main stages:

  1. 1.

Observe: Gather information about the environment and the problem at hand;

  2. 2.

Orient: Analyze the collected information, update the understanding of the
situation, and generate potential solutions or actions;

  3. 3.

Decide: Evaluate the potential solutions or actions and select the most
appropriate one based on the current understanding; and

  4. 4.

Act: Execute the selected solution or action and monitor its impact on the
environment.

The iterative nature of the OODA loop allows for continuous refinement and
adaptation based on the feedback received from the environment. By repeatedly
cycling through these four stages, an agent can progressively improve its
understanding of the problem, generate more relevant solutions, and make
better decisions.

In the context of Q&A systems, the OODA loop can be applied to decompose a
hard question into simpler ones more easily satisfied by RAG queries, to
combine and make sense of data points from multiple RAG queries, and to verify
internal consistency. By incorporating the OODA loop, the Q&A system can
iteratively refine its understanding of the question, retrieve more relevant
information, and generate more accurate and contextually appropriate answers.
This iterative process helps overcome the limitations of one-pass input-
process-output data flow, which characterizes many current LLMs and AI
systems.

The integration of the OODA loop into RAG-based Q&A systems involves mapping
the four stages of the loop to the various components of the RAG pipeline. In
the next section, we describe the construction of the OODA mechanism within
our proposed framework.

Figure 1: A typical OODA reasoning loop.

Constructing OODA Mechanism

To clearly guide a generative model to reason through a task, we use the
architecture implemented in Aitomatic’s open-source
OpenSSA222https://github.com/aitomatic/openssa framework, which incorporates
OODA iterations. To execute on a task, a reasoner reasons through it and
returns a conclusion. During the reasoning step, the model iteratively applies
the OODA loop to the task at hand given the instructions and tools provided in
the task description. This process involves continuously gathering data using
the available resources (observe), reflecting on the current context and how
the new information fits within it (orient), making decisions based on the
accumulated knowledge and the goals of the task (decide), and then
implementing those decisions using available tools (act). Iterating through
OODA provides the model a consistent, clear framework for achieving task
objectives, and allows it to adapt its approach as needed based on the
available information.

Figure 2: A specific implementation of OODA applied to question-answering with
RAG. Figure 3: Comparison of pure-RAG and OODA-enabled answers to a
FinanceBench question.

####  3.1.4 The Q&A AI Technical Design Space

Based on the technical considerations discussed above, we propose that AI
developers and managers explicitly analyze a structured technical design space
that outlines the principal components that meaningfully impact their AI
workflows’ accuracy.

If we take into account just the three areas we have identified above for
potential improvement, an example design space materializes as shown in Figure
4.

Figure 4: A structured technical design space capturing high-impact components
within question-answering systems.

As more principal technical components — such as Information Augmentation and
Task Planning, which will be the subjects of our future publications — are
identified as impactful, the design space should expand to include these as
additional dimensions.

In the remaining sections of this paper, we explore specific configurations
within the design space through experimental setups and present results that
quantify the impacts of these technical choices. This exploration aims to
provide AI developers and managers with concrete evidence to guide their
decisions and enhance project success.

###  3.2 Financial Analysis Benchmark Dataset

FinanceBench is a financial-analysis dataset curated and partially open-
sourced by [12]. It sources its data from public company filings with the U.S.
Securities and Exchange Commission (SEC), which typically include key
financial statements (Balance Sheet, Income Statement, and Cash Flows
Statement) and management commentaries on business outlook and risks. For each
filing, one or several analytical questions are crafted, with expert answers
provided by chartered financial analysts (CFAs).

Out of 150 FinanceBench question-answer pairs made publicly available, 141 are
suitable for benchmarking exercises (the remaining 9 are unusable due to
broken/missing document sources). We have categorized these questions into
seven categories of increasing difficulty, as illustrated in Figure 5.

Figure 5: Question difficulty categorizations for FinanceBench.

###  3.3 Evaluation Metrics

To assess the quality of the outputs from our various technical combinations,
we employ a range of evaluation metrics that consider both the accuracy and
relevance of the generated responses.

####  3.3.1 Retrieval Quality Metrics

We use automated retrieval quality metrics from LlamaIndex’s evaluation suite,
which employs an LLM to validate the retrieved documents as they relate to the
query, response, and reference documents.

  * •

Relevancy. Relevancy measures whether or not the response for the query is in
line with the context information. A response scores 0 if it is irrelevant and
1 otherwise.

  * •

Faithfulness. Faithfulness measures whether a response is supported by the
retrieved context, scoring 0 when it is not and 1 when it is.

  * •

Context similarity. Context similarity measures the semantic difference
between the retrieved contexts and the reference contexts. It is calculated by
finding the cosine similarity between the contexts when mapped into an
embedding space. By default, LlamaIndex rounds this score to 0 or 1 if the
cosine is below or above 0.8 respectively.

####  3.3.2 Answer Correctness Metrics

We employ both an automated correctness metric and manual grading to evaluate
whether the final answers produced by the Q&A system given the reference
answer.

  * •

Automated correctness. We use LlamaIndex’s CorrectnessEvaluator, which scores
answers on a 1-5 scale based on their relevance to the reference answer and
provides justifications for each score to ensure consistency. 1 signifies an
irrelevant answer, 2-3 indicate a relevant answer with possible errors, and
4-5 denote a fully correct answer. For consistency with the manually-
calculated binary scores, we convert these scores to percentages by
subtracting 1 and dividing by 4.

  * •

Human correctness. We also conduct human evaluations to assess the quality of
the generated responses, rating answers as correct or incorrect based on the
FinanceBench reference. To observe performance gains in different types of
question-answering, we look at the performance in easier or harder question
categories as described in Section 3.2. This qualitative assessment provides
valuable insights into the real-world applicability and effectiveness of our
proposed framework.

By employing these evaluation metrics, we aim to provide a comprehensive and
rigorous assessment of the performance of our models in handling domain-
specific Q&A tasks. The combination of automated metrics and human evaluations
allows us to thoroughly analyze the strengths and limitations of our proposed
framework and identify areas for further improvement.

##  4 Experiments & Results

We experimented with and evaluated the outputs of the following technical
configurations:

  * •

Generic RAG: bge-large-en with gpt-3.5-turbo-0125.

  * •

RAG with Fine-Tuned Generator: bge-large-en with fine-tuned
gpt-3.5-turbo-0125.

  * •

RAG with Fine-Tuned Retriever: fine-tuned bge-large-en with
gpt-3.5-turbo-0125.

  * •

Fully Fine-Tuned RAG: fine-tuned bge-large-en with fine-tuned
gpt-3.5-turbo-0125.

  * •

Generic RAG with OODA Reasoning: bge-large-en with gpt-3.5-turbo-0125 and OODA
reasoning.

Each system uses the same VectorStoreIndex for the documents, a top-k of 10
documents, and default RAG prompts, all from base LlamaIndex. For the full
reproducible experimental setup, see OpenSSA’s FinanceBench
example333https://github.com/aitomatic/openssa/tree/main/examples/FinanceBench.
Other combinations such as Fine-Tuned RAG with OODA Reasoning shall be
addressed in a future publication (see Section 6).

Our test set consists of the remaining 41 questions in the publicly-available
FinanceBench test of valid questions that were not the 100 selected for
training.

###  4.1 Retrieval Quality Results

Table 1 summarizes the retrieval quality scores achieved by various
configurations across the FinanceBench test set questions. As the OODA
reasoning involves multiple iterative retrievals, it is not directly
comparable to the one-step retrieval processes of these RAG systems, and is
therefore not included in this table.

Configuration | Relevancy | Faithfulness |  Context similarity
---|---|---|---
Generic RAG | 0.317 | 0.700 | 0.894
Fine-Tuned Generator | 0.291 | 0.625 | 0.890
Fine-Tuned Retriever | 0.340 | 0.634 | 0.901
Fully Fine-Tuned RAG | 0.317 | 0.512 | 0.903
Table 1: Retrieval quality metrics of question-answering systems on the
FinanceBench test set.

###  4.2 Answer Correctness Results

Table 2 summarizes the automated scores and human-judged binary-accuracy
scores achieved by various configurations across the FinanceBench test set
questions. Note that the automated correctness scores have been converted from
their original 1-5 scaling to percentages. The OVERALL scores are performed on
the entire test set. The scores for EASIER and HARDER questions are based on
the respective subsets as defined in Section 3.2.

Configuration Human Evaluation EASIER QUESTIONS 0-RETRIEVE 1-COMPARE 2-CALC-
CHANGE Human Evaluation HARDER QUESTIONS 3-CALC-COMPLEX 4-CALC-AND-JUDGE
5-EXPLAIN-FACTORS 6-OTHER-ADVANCED Human Evaluation OVERALL Automated
Correctness OVERALL Generic RAG 55% 16% 37% 44% Fine-Tuned Generator 64% 16%
43% 45% Fine-Tuned Retriever 73% 35% 54% 54% Fully Fine-Tuned RAG 77% 37% 59%
58% Generic RAG with OODA Reasoning 91% 79% 85% 77%

Table 2: Answer correctness metrics of question-answering systems on the
FinanceBench test set.

##  5 Discussion & Analysis

In this section, we analyze the implications of our experimental results,
discuss key findings, and provide recommendations to enhance Q&A AI system
performance through domain-specific fine-tuning and retrieval-augmented
generation (RAG) techniques.

###  5.1 Key Findings

Our experimental results demonstrate several important findings:

Gains in accuracy from RAG with fine-tuning component models

  1. 1.

RAG with fine-tuned retriever, fine-tuned generator, or full fine-tuning
outperforms the generic RAG. Specifically, using the FinanceBench dataset, we
achieved accuracy improvements of up to 20 percentage points over baseline
RAG.

  2. 2.

Fine-tuning the retriever model results in higher accuracy gains compared to
fine-tuned generators. This advantage is significant as fine-tuning embedding
models for retrieval is less costly and less labor-intensive than fine-tuning
LLMs for generation.

Gains in accuracy from RAG with OODA Reasoning

  1. 1.

Integrating iterative reasoning capabilities, such as OODA, with the RAG
engine substantially enhances performance. Specifically, the Generic RAG with
OODA Reasoning configuration achieves an accuracy increase of up to 50
percentage points across the FinanceBench dataset compared to the generic RAG
baseline.

  2. 2.

This finding underscores that OODA, though a domain-agnostic reasoning
mechanism, significantly boosts accuracy in domain-specific tasks when
integrated with relevant informational sources. On the FinanceBench dataset,
the generic OODA configuration outperformed the fully fine-tuned RAG by a
considerable margin of 20-25 percentage points.

####  5.1.1 Implications and Recommendations

Our findings have significant implications for developing Q&A AI systems
across various domains. Based on our experimental results, we propose the
following recommendations:

  1. 1.

Prioritize the fine-tuning of embedding models over language models. Our
approach demonstrates superior performance and offers a more resource-
efficient and scalable solution for addressing real-world challenges by
capitalizing on domain-specific embedding tuning. Fine-tuning embedding models
requires minimal human effort for data acquisition compared to fine-tuning
language models, making it a more practical choice.

  2. 2.

Employ OODA reasoning or other iterative reasoning mechanisms which can
significantly improve informational consistency and enhance the Q&A system’s
ability to combine information from multiple sources.

  3. 3.

Map out a technical design space covering principal components and make
deliberate choices based on empirical evidence: (a) Identify the principal
components that have the most significant impact on the performance of Q&A
systems, such as information indexing and retrieval, answer generation, and
iterative reasoning. (b) Create a structured technical design space that
captures the possible configurations of these components and (c) Make informed
decisions on the optimal configuration for a given domain or task based on
empirical quantitative evidence, such as the results reported in this study.

By following these recommendations, AI teams can leverage the potential of
domain-specific fine-tuning and iterative reasoning to enhance the accuracy,
relevance, and usefulness of generated responses, while maintaining resource
efficiency and scalability.

##  6 Conclusions & Future Directions

In this paper, we identified areas where new or modified techniques can
significantly improve the accuracy of LLM-based RAG in Q&A AI use cases. In
particular, we have investigated the how-tos and the impacts of domain-
specific fine-tuning and iterative reasoning, measuring accuracy gains from
such methods on complex financial-analysis and industrial-equipment datasets.
Promising accuracy improvements of up to 30 percentage points on the generic
RAG baseline suggest there is a lot of value in incorporating these methods in
real-world applications in specialized domains.

Additionally, we have elaborated that designing a high-quality AI system
involves many important technical choices, and that AI developers and managers
can benefit from a structured technical design space map that helps them be
very deliberate and informed in their decision-making.

Looking ahead, we identify several promising directions for future research
and development:

  1. 1.

Custom Augmenters for RAG: Explore the development of domain-specific
augmenters that leverage domain-specific metadata and expert knowledge such as
notable facts, rules and exceptions, and process heuristics, to prioritize and
filter the most pertinent information for a given question.

  2. 2.

Fine-Tuned RAG with OODA Reasoning: Quantify the accuracy gains from combining
fully fine-tuned RAG with iterative OODA loops.

  3. 3.

Practical Guideline for AI Teams: Develop a set of actionable practices for AI
teams seeking to build high-performance Q&A systems. This guideline will be
based on the growing data catalog contributed by the AI Alliance members, the
technical design space mapping framework, and quantitative empirical results
measuring performance of configurations in that design space on datasets from
the catalog. Through the AI Alliance, we all can ensure open development, open
science, and open AI, benefiting the broader AI community.

  4. 4.

Advanced Planning & Reasoning: With our current experimentation showing great
promise from integrating OODA reasoning, we are continuing our research to
explore Hierarchical Task Planning (HTP) and other planning mechanisms that
are well known in financial and industrial domains. Well-performing and
efficient implementations of Planning and Reasoning would push the boundaries
of problem-solving capabilities in mission-critical real-world AI systems.

  5. 5.

Open-Source Industry-Specific Evaluation Datasets: With the proliferation of
applications of AI to various industries, it would be highly beneficial to
establish meaningfully complex benchmark datasets for key industries or
specific domains, especially those with scientific or engineering knowledge
and technical jargon and that significantly differ from general knowledge and
language.

##  7 Acknowledgements

We would like to express our gratitude to the AI Alliance
(https://thealliance.ai) for providing the impetus and platform for
collaboration in open science. We also extend our thanks to the member
organizations of the AI Alliance, their researchers and engineers for their
valuable contributions to this study. Their expertise, insights, and
collaborative spirit have been instrumental in advancing our research.

## References

  * [1] Iz Beltagy, Matthew E. Peters and Arman Cohan  “Longformer: The Long-Document Transformer”, 2020  arXiv:2004.05150 [cs.CL]
  * [2] John Richard Boyd  “Patterns of Conflict”, 1986  URL: http://d-n-i.net/second_level/boyd_military.htm
  * [3] Tom B. Brown et al.  “Language Models are Few-Shot Learners”, 2020  arXiv:2005.14165 [cs.CL]
  * [4] Wenhu Chen et al.  “HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data”, 2020  arXiv:2004.07347 [cs.CL]
  * [5] Zihang Dai et al.  “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, 2019  arXiv:1901.02860 [cs.LG]
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2018  arXiv:1810.04805 [cs.CL]
  * [7] Robert E. Enck  “The OODA Loop”  In _Home Health Care Management & Practice_ 24.3, 2012, pp. 123–124  DOI: 10.1177/1084822312439314
  * [8] Zhangyin Feng et al.  “Retrieval-Generation Synergy Augmented Large Language Models”, 2023  arXiv:2310.05149 [cs.CL]
  * [9] Zhicheng Guo et al.  “Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks”, 2023  arXiv:2305.17653 [cs.CL]
  * [10] Neil Houlsby et al.  “Parameter-Efficient Transfer Learning for NLP”, 2019  arXiv:1902.00751 [cs.LG]
  * [11] Jie Huang et al.  “RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models”, 2024  arXiv:2308.07922 [cs.CL]
  * [12] Pranab Islam et al.  “FinanceBench: A New Benchmark for Financial Question Answering”, 2023  arXiv:2311.11944 [cs.CL]
  * [13] Zhengbao Jiang et al.  “Active Retrieval Augmented Generation”, 2023  arXiv:2305.06983 [cs.CL]
  * [14] Minki Kang et al.  “Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks”, 2023  arXiv:2305.18395 [cs.CL]
  * [15] Daniel Khashabi et al.  “UnifiedQA: Crossing Format Boundaries With a Single QA System”, 2020  arXiv:2005.00700 [cs.CL]
  * [16] Yann LeCun  “A Path Towards Autonomous Machine Intelligence”, 2022  URL: https://openreview.net/pdf?id=BZ5a1r-kVsf
  * [17] Patrick Lewis et al.  “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2021  arXiv:2005.11401 [cs.CL]
  * [18] Yinhan Liu et al.  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019  arXiv:1907.11692 [cs.CL]
  * [19] Aman Madaan et al.  “Self-Refine: Iterative Refinement with Self-Feedback”, 2023  arXiv:2303.17651 [cs.CL]
  * [20] Colin Raffel et al.  “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, 2023  arXiv:1910.10683 [cs.LG]
  * [21] Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf  “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”, 2020  arXiv:1910.01108 [cs.CL]
  * [22] Karthik Valmeekam et al.  “PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change”, 2022  arXiv:2206.10498 [cs.CL]
  * [23] Ashish Vaswani et al.  “Attention Is All You Need”, 2017  arXiv:1706.03762 [cs.CL]
  * [24] Yuxi Xie et al.  “Self-Evaluation Guided Beam Search for Reasoning”, 2023  arXiv:2305.00633 [cs.CL]
  * [25] Shunyu Yao et al.  “Tree of Thoughts: Deliberate Problem Solving with Large Language Models”, 2023  arXiv:2305.10601 [cs.CL]
  * [26] Peitian Zhang et al.  “Retrieve Anything To Augment Large Language Models”, 2023  arXiv:2310.07554 [cs.IR]
  * [27] Pei Zhou et al.  “Self-Discover: Large Language Models Self-Compose Reasoning Structures”, 2024  arXiv:2402.03620 [cs.AI]

## Appendix

### Full set results

We also conducted tests on the full publicly-available FinanceBench valid
question set, which includes questions that were used in fine-tuning training,
and found consistent results with the test set results for answer quality.
Results are shown in table 3 and 4.

Configuration | Relevancy | Faithfulness |  Context similarity
---|---|---|---
Generic RAG | 0.354 | 0.709 | 0.887
Fine-Tuned Generator | 0.355 | 0.673 | 0.887
Fine-Tuned Retriever | 0.426 | 0.738 | 0.901
Fully Fine-Tuned RAG | 0.418 | 0.723 | 0.901
Table 3: Retrieval quality metrics of question-answering systems on the
FinanceBench full set.

Configuration Automated correctness score (1-5) Human evaluation EASIER
QUESTIONS 0-RETRIEVE 1-COMPARE 2-CALC-CHANGE Human evaluation HARDER QUESTIONS
3-CALC-COMPLEX 4-CALC-AND-JUDGE 5-EXPLAIN-FACTORS 6-OTHER-ADVANCED Human
evaluation OVERALL Generic RAG 2.83 63% 18% 43% Fine-Tuned Generator 2.91 65%
20% 45% Fine-Tuned Retriever 3.23 70% 35% 55% Fully Fine-Tuned RAG 3.21 68%
37% 55% Generic RAG with OODA Reasoning 3.91 91% 69% 82%

Table 4: Answer correctness metrics of question-answering systems on the
FinanceBench full set.

### Generated training data

We experimented with fine-tuning on synthetic training data. We first
collected the PDFs from all links listed in the FinanceBench dataset and split
their texts into chunks of 1024 tokens each with LlamaIndex’s
SentenceSplitter. Then, we used LlamaIndex’s RagDatasetGenerator to generate 3
queries and answers per chunk. In total, this created about 15,500 query-
context-answer triplets that could then be used for fine-tuning embeddings
(query-context) and fine-tuning generators (query-answer). For smaller fine-
tuning jobs we used random subsets of this dataset to train.

We found that performance either did not improve or decreased when we used
models fine-tuned on this data. The likeliest explanation, given the success
of fine-tuning on data taken directly from FinanceBench, is that when GPT-3.5
generates queries and answers based on the contexts, it does not do so in the
way that the domain experts that created the FinanceBench data did. These may
lead to questions that rely more heavily on the wording in the context,
answers that do not use prior financial knowledge in addition to the contexts,
or other quirks of generated data that do not correspond to the ways humans
think about the relationships between texts for synthesis tasks. The results
are shown in Table 5

Embedder (+ train size) Generator (+ train size) Automated correctness Generic
Generic 2.83 Generic Fine-tuned on 16k 2.57 Fine-tuned on 16k Generic 2.44
Fine-tuned on 16k Fine-tuned on 16k 2.34 Generic Fine-tuned on 1k 2.95 Fine-
tuned on 1k Generic 2.96 Fine-tuned on 1k Fine-tuned on 1k 2.96

Table 5: Correctness scores of fine-tuning configurations on the FinanceBench
test set.


[]WonjuneKang []DebRoy

# Prompting Large Language Models with Audio for
General-Purpose Speech Summarization

###### Abstract

In this work, we introduce a framework for speech summarization that leverages
the processing and reasoning capabilities of large language models (LLMs). We
propose an end-to-end system that combines an instruction-tuned LLM with an
audio encoder that converts speech into token representations that the LLM can
interpret. Using a dataset with paired speech-text data, the overall system is
trained to generate consistent responses to prompts with the same semantic
information regardless of the input modality. The resulting framework allows
the LLM to process speech inputs in the same way as text, enabling speech
summarization by simply prompting the LLM. Unlike prior approaches, our method
is able to summarize spoken content from any arbitrary domain, and it can
produce summaries in different styles by varying the LLM prompting strategy.
Experiments demonstrate that our approach outperforms a cascade baseline of
speech recognition followed by LLM text processing.

###### keywords:

speech summarization, large language model, speech language model, modality
invariance

##  1 Introduction

Large language models (LLMs) [1, 2, 3] have demonstrated remarkable
capabilities in a wide variety of natural language tasks, such as question
answering, machine translation, language understanding, and text
summarization. Notably, they exhibit the ability to respond to prompts of
tasks that they were not explicitly trained on without any further gradient
updates to their parameters. These characteristics of LLMs, combined with
instruction tuning to align their behavior with human preferences [4, 5, 6],
have allowed users to seamlessly interact with them via text-based prompting.

In this work, we aim to utilize the processing and reasoning capabilities of
LLMs for speech-to-text summarization, the task of condensing the linguistic
content in a speech signal into a text summary. We build on recent work that
extends the capabilities of LLMs to other modalities, in particular speech and
audio. Speech summarization has traditionally been done using a cascade of
automatic speech recognition (ASR) and text summarization [7, 8]. Recently,
some works have proposed systems for end-to-end speech summarization [9, 10,
11], which aim to reduce errors caused by the propagation of mistakes in the
ASR step. However, prior methods are limited by their training data; models
trained on a specific dataset are restricted to producing summaries in that
domain and in the style of that dataset's annotations. Meanwhile, LLMs are
able to summarize text from arbitrary domains and in different styles by
varying their prompts; we would like to leverage these abilities to develop a
system that can perform general-purpose speech summarization.

To this end, we propose an end-to-end pipeline that combines a text LLM with
an audio encoder that allows the LLM to understand and process speech inputs
in the same way as text. Our model is trained using the concept of modality
invariance—the idea that, given certain semantic information in a prompt, the
LLM should provide the same response regardless of the prompt's modality [12].
Specifically, we use an ASR dataset with paired speech-text data; while
keeping the LLM weights frozen, we train the audio encoder to convert speech
inputs into token representations that the LLM can interpret. Then, the end-
to-end system is guided to produce the same output as when text is the input
using next-token prediction loss. We additionally incorporate knowledge
distillation using the response from the corresponding text input as the
teacher model, utilizing feature and logit distillation losses to guide the
model to produce more consistent responses from speech inputs.

Under this framework, every capability of the LLM with text also becomes
possible with speech, and we can perform speech summarization by simply
prompting the LLM to do so. Because our method takes advantage of the innate
abilities of LLMs, it can be used for summarizing spoken content or documents
from any arbitrary domain. Furthermore, it is able to generate summaries in
different styles by varying the prompting strategy to the LLM. Experiments
demonstrate that our proposed system outperforms a baseline cascade of ASR and
LLM-based text processing in terms of response language modeling and speech
summarization on the CNN / DailyMail dataset [13].

In summary, the contributions of this work are as follows: 1) We propose a
framework that allows an LLM to understand and process any speech input, and
leverage its capabilities to perform speech summarization. 2) Our method can
be used to summarize spoken content from any arbitrary domain. 3) We
demonstrate the system's ability to produce summaries in different styles by
adjusting the prompt to the LLM.111Code available at:
https://github.com/wonjune-kang/llm-speech-summarization. Examples of
summaries from various prompts and speech utterances are included on our demo
page: https://llm-speech-summarization.github.io/

##  2 Background and Related Work

###  2.1 Large language models (LLMs) and speech

There has been a large body of recent work on extending the capabilities of
LLMs to speech and audio. Most approaches utilize some sort of audio encoder
to process audio into discrete tokens that can be fed into the LLM as inputs.
These audio tokens can be used to augment the original text-only token
vocabulary of the LLM [14, 15] or trained in a way that aligns them with the
original text token embedding space [16, 12]. Some models [14, 16, 17] are
trained to perform specific tasks such as speech translation, speech
recognition, or speech synthesis, while others [15, 18] are designed to be
treated as general purpose models that can process a broad range of speech
prompts.

The approach we use in this work is most similar to AudioChatLlama [12], which
utilizes the concept of modality invariance between prompts. Given certain
semantic information in a prompt, the idea is that an LLM should provide the
same response regardless of whether the prompt is in audio or text.
AudioChatLlama consists of a trainable audio encoder and a frozen LLM; using
paired speech-text samples, the overall system is guided to produce the same
response whether the input to the LLM is in audio or text. However, it only
relies on next-token prediction loss to align its outputs with those generated
from text inputs, and the authors did not study any temporal alignment
strategies between the two modalities when designing their audio encoder. In
contrast, we also incorporate knowledge distillation by matching the LLM's
token prediction logits and hidden states between text and audio inputs, and
we explore various design choices for the audio encoder so as to induce the
LLM to produce more consistent responses to speech prompts.

###  2.2 Speech summarization

Recent works on speech summarization have largely focused on end-to-end models
[9, 10, 11], which aim to mitigate error propagation from mistakes made during
intermediate steps in traditional cascade models of ASR and text summarization
[7, 8]. However, like most machine learning models, these previous methods are
inherently limited by their training data; models trained on a specific
dataset are only able to produce summaries in that domain and in the style of
that dataset's annotations. Recently, [19] proposed a method to leverage LLMs
as a proxy for human annotators to generate a wider variety of summaries for
training and evaluating summarization models. However, the fundamental problem
still stands; for example, a model trained to summarize instructions using the
How2 dataset [20] would likely not be suitable for summarizing the content of
long meetings. In contrast, our approach performs speech summarization by
leveraging the innate abilities of LLMs, which enables a more general system
that is capable of handling diverse domains and stylistic preferences.

##  3 Proposed Method

###  3.1 System architecture

Our proposed system consists of two components: an LLM and an audio encoder
that translates speech inputs into token embeddings that the LLM can receive
as part of a prompt. We use MiniChat-3B as our LLM, an instruction-tuned
version of MiniMA-3B [21], which was distilled from Llama 2 7B [3]. Although
small for LLM standards, it achieves strong performance on a variety of
benchmarks compared to other 3B parameter models and approaches the
performance of Llama 2 7B, making it a suitable choice for us. Like most other
decoder-only language models, MiniChat takes a token embedding sequence as
input and autoregressively generates a text response.

The backbone of the audio encoder is initialized with a HuBERT-Large model
[22] that was fine-tuned to perform ASR on the Librispeech corpus [23] using
connectionist temporal classification (CTC) loss [24]. It takes as input mono-
channel audio at a 16 kHz sampling rate. Following the HuBERT layers, which
produce 1024-dim representations every 20 ms (50 Hz), we perform temporal
downsampling by a factor of 4 by average pooling with a kernel size of 8 and a
stride of 4. This is followed by a linear projection layer to match the
dimension of the pooled representations with that of the LLM (3072-dim). The
encoder thus produces audio tokens at a rate of 12.5 Hz.

We also explored a downsampling method which involved pooling HuBERT
representations based on the predicted CTC alignments of words from a separate
HuBERT ASR model. Specifically, we performed the same average pooling as
above, but between the CTC-aligned start embedding indices of consecutive
words predicted by the ASR model; this was an effort to more explicitly align
audio tokens with words. We discuss the results of these experiments in
Section 5.

Figure 1: The audio encoder is trained to produce token embeddings such that
the LLM's response to a spoken prompt matches its response to a text prompt
with the same semantic content.

###  3.2 Aligning responses for audio and text inputs

Figure 1 shows the training strategy for the audio encoder and LLM pipeline.
At a high level, it involves aligning the LLM's response to an audio prompt
with the response to a matching text prompt with the same semantic
information. We would like to utilize the capabilities of our LLM without
compromising any of its original performance; therefore, we keep the LLM's
weights frozen during all of our experiments, with the audio encoder being the
only trainable component of our system. Although this framework prevents the
LLM from utilizing any additional information in the speech input beyond the
linguistic content, it allows us to safely preserve its original abilities
without risk of overfitting to any specific tasks or datasets.

Data. We make use of the Librispeech corpus [23], which is often used for ASR
and contains paired audio-text samples of spoken utterances and their
transcripts. For all of the samples in Librispeech, we prompt the LLM using
the text transcripts and save its responses to create a dataset with (audio,
text, response) components for each sample. These responses are then treated
as the ground truth output which the audio encoder is trained to induce from
the LLM. Note that Librispeech consists of utterances that are collected from
audiobooks, which generally do not make for useful prompts when fed into an
LLM without any additional context. However, for our training framework, we
can consider the objective to have been achieved if the audio prompt can
produce the same response as the text prompt, even if the prompt and LLM
response are nonsensical.

Training. We train our system using next-token prediction (NTP) loss,
conditioning the prediction of the response tokens on the input audio tokens.
Formally, let the full token sequence fed into the model be
$\mathbf{x}=(x_{0},...,x_{t},x_{t+1},...x_{T})$, where the first $t$ elements
correspond to the prompt, including the audio tokens and any tokens used in
the LLM's chat structure: $(x_{0},...,x_{t})$. The remaining elements
correspond to the tokens for the LLM's ground truth response to the
corresponding text prompt: $(x_{t+1},...,x_{T})$. Then, the NTP loss is:

$\displaystyle\mathcal{L}_{\mathrm{NTP}}=-\sum_{i=t+1}^{T}\log
p(x_{i}|x_{<i}).$ (1)

We additionally incorporate knowledge distillation at the logit [25] and
feature [26] levels using the outputs and activations of the LLM given the
original text input as the teacher model. These losses are meant to more
explicitly guide the audio encoder to induce predictions and hidden states
from the LLM that match those from the standard text input. For logit
distillation (LD), we perform logit matching between the text and audio
prompts using a soft cross-entropy loss as follows:

$\displaystyle\mathcal{L}_{\mathrm{LD}}=-\sum_{i=t+1}^{T}\sum_{j=1}^{|\mathcal{V}|}\tilde{y}^{(j)}_{i}\log\hat{y}^{(j)}_{i},$
(2)

where $\mathcal{V}$ is the SentencePiece token vocabulary of the LLM, and
$\tilde{y}$ and $\hat{y}$ denote the next-token prediction logits conditioned
on the text and audio prompts, respectively. For feature distillation (FD), we
use the mean-square error (MSE) loss between the hidden states of the LLM when
given the text prompt ($\mathbf{\tilde{h}}$) and audio prompt
($\mathbf{\hat{h}}$) at connector layers $L$ as follows:

$\displaystyle\mathcal{L}_{\mathrm{FD}}=\sum_{l\in
L}\mathrm{MSE}\big{(}\mathbf{\tilde{h}}^{l}_{t+1:T},\mathbf{\hat{h}}^{l}_{t+1:T}\big{)}.$
(3)

The overall training objective is thus:

$\displaystyle\mathcal{L}_{\mathrm{total}}=\lambda_{\mathrm{NTP}}\mathcal{L}_{\mathrm{NTP}}+\lambda_{\mathrm{LD}}\mathcal{L}_{\mathrm{LD}}+\lambda_{\mathrm{FD}}\mathcal{L}_{\mathrm{FD}}$
(4)

with weighting factors $\lambda_{\mathrm{NTP}}$, $\lambda_{\mathrm{LD}}$, and
$\lambda_{\mathrm{FD}}$ for each loss. Note that all losses are computed only
for token indices corresponding to the response portion of the sequence,
$[t+1,T]$.

Figure 2: Speech summarization can be performed by prompting the LLM using any
combination of text and speech inputs.

###  3.3 Performing speech summarization

Once the pipeline consisting of the audio encoder and LLM has been trained,
the LLM can take in any combination of text and speech inputs as part of a
user's prompt. Speech summarization can thus be performed by simply
instructing the LLM to do so in the prompt; Figure 2 shows the inference
process. Because our system leverages the general-purpose abilities of LLMs to
process any arbitrary input, it can summarize spoken content from any domain.
We can further control the style of the desired summary by adjusting the
prompt, such as by instructing the LLM to vary the summary's length or to
emphasize or de-emphasize specific aspects of the content.

Table 1: Perplexity of the LLM on the Librispeech test sets under the ``correct'’ response. Model | Perplexity
---|---
test-clean | test-other
Ground truth text + LLM | 1.608 | 1.595
Cascade: HuBERT ASR + LLM | 1.644 | 1.682
Proposed system | 1.604 | 1.638
NTP + FD losses only | 1.552 | 1.589
NTP + LD losses only | 1.592 | 1.627
NTP loss only | 1.531 | 1.568
CTC-aligned pooling | 1.603 | 1.637
Table 2: Speech summarization performance on the curated CNN / DailyMail test set with speech synthesized using TTS. Metrics are measured against the original summaries from the CNN / DailyMail dataset as well as synthetic ones generated by Llama 2-Chat 7B. Model | CNN / DailyMail Summaries | Llama 2-Chat 7B Summaries
---|---|---
ROUGE-1, 2, L | METEOR | BERTScore | ROUGE-1, 2, L | METEOR | BERTScore
Ground truth text + LLM | 37.31, 15.69, 24.66 | 38.80 | 87.72 | 52.94, 28.49, 36.76 | 44.64 | 91.08
Cascade: HuBERT ASR + LLM |  33.60, 12.97, 22.56 | 34.73 | 86.87 | 49.18, 25.56, 34.86 | 41.20 | 90.13
Proposed system |  33.34, 13.21, 22.52 | 36.46 | 86.83 |  51.12, 27.50, 37.48 | 45.63 | 90.50
NTP + FD losses only | 33.03, 12.81, 22.24 | 36.20 | 86.74 | 51.33, 26.70, 36.74 | 44.85 | 90.28
NTP + LD losses only | 32.37, 12.51, 21.65 | 35.88 | 86.63 | 51.26, 26.93, 36.37 | 45.40 | 90.29
NTP loss only | 31.66, 12.20, 21.03 | 36.04 | 86.51 | 50.47, 26.01, 35.50 | 45.62 | 90.06
CTC-aligned pooling | 32.53, 12.36, 21.70 | 35.83 | 86.70 |  51.60, 27.05, 36.89 | 45.51 | 90.39

##  4 Experimental Setup

###  4.1 Datasets

We trained our system using the Librispeech corpus [23], using the standard
splits for training (960 hours), validation, and testing. To create the LLM
response dataset, we fed reference text into MiniChat and generated the
response using greedy decoding with a maximum decoding length equal to twice
the length of the input prompt.

To evaluate speech summarization, we used the CNN / DailyMail dataset [13].
Because it does not include audio, we synthesized speech for the dataset using
a text-to-speech (TTS) model, StyleTTS 2 [27] trained on the LJ Speech dataset
[28]. We only used the test set, and to keep audio file lengths manageable, we
selected articles that were no more than 1600 characters in length. This
resulted in 886 samples where synthesized speech utterances ranged from 21.3
to 149.3 seconds in length, with an average of 93.9 seconds. While synthesized
speech is preferably avoided, there are unfortunately few speech to summary
datasets available. We opted not to use the popular How2 dataset [20] because
we were interested in evaluating summarization in a domain beyond what that
dataset specifically covers (instructional videos). Furthermore, How2 only
includes pre-extracted acoustic features and not raw waveforms, and thus would
also have required some method to re-synthesize audio.

Additionally, we found the ground truth annotations in CNN / DailyMail to be
ill-suited for evaluating LLM-generated summaries. They consist of
concatenated bullet points that are not continuous pieces of text, and are
thus quite different from the text style produced by LLMs. Because we were
interested in performing and evaluating LLM-style summarization, we created a
new set of reference summaries by prompting Llama 2-Chat 7B [3] to summarize
the CNN / DailyMail articles in 3 sentences or less. We evaluated performance
against both the original summaries and the Llama 2-generated ones.

###  4.2 Training and evaluation

We conducted all experiments on a single NVIDIA V100 32GB GPU. All models were
trained for up to 2.4M steps with a batch size of 1 and gradient accumulation
every 16 steps (effectively 150k steps with a batch size of 16). We used the
AdamW optimizer [29] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, with an initial
learning rate of 5e-5 that linearly decayed down to 5e-6 over the course of
training. We empirically set $\lambda_{\mathrm{NTP}}=0.5$,
$\lambda_{\mathrm{LD}}=0.5$, and $\lambda_{\mathrm{FD}}=1.0$. For feature
distillation, we used connector layers at indices $L=[1,6,12,18,24]$ of the
24-layer LLM.

We selected the checkpoint with the best validation NTP loss for each model.
When performing summarization, we always prompted the LLM to do so in 3
sentences or less and used greedy decoding for consistency. Performance was
evaluated using ROUGE [30], METEOR [31], and BERTScore [32].

##  5 Results

We compare our proposed system against a baseline cascade system of an ASR
model and MiniChat-3B. For a fair comparison, we use the same HuBERT-Large
model as the one used to initialize the backbone of our audio encoder to
perform ASR; it achieves 1.9% and 3.3% word error rates on the test-clean and
test-other splits of Librispeech, respectively. We also perform comparisons
against ablations of our model with different combinations of training
objectives and architectural choices, and include scores from feeding the
ground truth text into the LLM as an additional point of reference.

###  5.1 Response language modeling

As a preliminary experiment, we looked at the LLM's performance on response
language modeling by measuring its perplexity under the ``correct'' response
to prompts on the test sets of Librispeech; i.e., the response generated when
prompted with the ground truth transcript of the audio. The results are shown
in Table 1. All variants of our end-to-end system obtained lower perplexity
than the cascade baseline and comparable values to when the ground truth text
was fed into the LLM. Note that perplexity is an imperfect metric for
measuring language modeling performance [33], and lower values do not
necessarily indicate better performance on downstream tasks. However, we treat
these results as a useful sanity check to demonstrate that our system is able
to process speech inputs coherently.

###  5.2 Speech summarization

Table 2 shows the summarization metrics of the various models on the curated
CNN / DailyMail test set, measured against both the original reference
summaries and the ones generated using Llama 2-Chat. When evaluating against
the original summaries, our proposed method obtained similar scores to the
cascade system, and all systems obtained quite low scores overall. This
indicates a fundamental mismatch between the text style of the original
annotations and that of the summaries generated by the LLM. However, all
models obtained significantly higher scores when evaluating against summaries
generated by Llama 2-Chat, indicating a closer stylistic match to text
generated by another instruction-tuned LLM. Here, our proposed end-to-end
system significantly outperformed the cascade baseline in terms of all
metrics, and it approached the scores obtained from feeding the actual
reference text into the LLM.

When looking at the results of ablations, we found that each of the loss
components used for training the audio encoder contributed to performance in a
meaningful way, with the system trained on all three losses achieving the best
overall performance. Interestingly, using CTC-based word offsets as the basis
to pool embeddings resulted in a slight decrease in scores compared to
standard pooling. This could be a result of using the predicted embedding
offsets as ``hard'' alignments for pooling; if the words predicted by the ASR
model are incorrect, aggregating audio tokens based on these incorrect offsets
could result in a noisier token sequence being provided to the LLM. The issue
could potentially be rectified by using a probabilistic ``soft'' alignment
instead, or by using a separate module for speech-token alignment such as the
Montreal Forced Aligner [34].

We also explored varying the prompt to the LLM to generate summaries from
speech in different styles; Table 3 shows examples of various summaries
generated in this way. These results demonstrate that our framework
successfully leverages the reasoning and processing abilities of LLMs,
allowing it to be used as a general-purpose speech summarization system that
can be applied to any domain or desired summarization style. Additional
examples are included on our demo page.

Table 3: Examples of summaries generated from speech in different styles by varying the user prompt. All prompts begin with: ``Summarize the following article in 3 sentences or less…'' Prompt | Summary
---|---
no other condition |  Juventus is close to winning a fourth straight Serie A title by defeating Lazio 2-0. Carlos Tevez scored his 26th goal of the season, helping Juventus take an early lead. Lazio attempted a late comeback but were unable to score, ending their six-game winning streak.
``without mentioning any names'' |  Juventus's 2-0 victory over Lazio moved them closer to their fourth consecutive Serie A title. Lazio's 15-point lead over second place was reduced to 19 points after the match. Sampdoria drew 0-0 against Cesena, leaving both teams level on 50 points.
``focusing on Lazio'' |  Carlos Tevez scored twice as Juventus defeated Lazio 2-0, moving 15 points ahead of second-place rivals. Lazio's six-game winning streak ended, and the team could be dragged back into third place if their city rivals, Roma, beat Atalanta. Sampdoria drew 0-0 against Cesena, leaving both teams level on 50 points.

##  6 Conclusion

We introduced an end-to-end framework for speech-to-text summarization that
can be applied towards spoken content from any domain. The proposed system
combines an instruction-tuned LLM with an audio encoder that converts speech
into token embeddings that the LLM can interpret, enabling speech
summarization by simply prompting the LLM. Unlike prior methods, our approach
is able to generate summaries in different styles by varying the prompts to
the LLM. The quality of the system's components are a key factor for its
performance; while we performed experiments using relatively small models in
this work, our framework is applicable for any combination of audio encoder
and LLM architectures. Using larger, more powerful models would open up
possibilities for significantly more performant general-purpose speech
summarization systems.

## References

  * [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman _et al._ , ``GPT-4 Technical Report,'' _arXiv preprint arXiv:2303.08774_ , 2023.
  * [2] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos _et al._ , ``PaLM 2 Technical Report,'' _arXiv preprint arXiv:2305.10403_ , 2023.
  * [3] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei _et al._ , ``Llama 2: Open Foundation and Fine-Tuned Chat Models,'' _arXiv preprint arXiv:2307.09288_ , 2023.
  * [4] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester _et al._ , ``Finetuned language models are zero-shot learners,'' in _International Conference on Learning Representations_ , 2022.
  * [5] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin _et al._ , ``Training language models to follow instructions with human feedback,'' _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [6] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus _et al._ , ``Scaling Instruction-Finetuned Language Models,'' _arXiv preprint arXiv:2210.11416_ , 2022.
  * [7] C. Zhu, R. Xu, M. Zeng, and X. Huang, ``A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining,'' in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 194–203.
  * [8] Y. Zhang, A. Ni, T. Yu, R. Zhang, C. Zhu, B. Deb _et al._ , ``An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next,'' in _Findings of the Association for Computational Linguistics: EMNLP 2021_ , 2021, pp. 4426–4433.
  * [9] R. Sharma, S. Palaskar, A. W. Black, and F. Metze, ``End-to-End Speech Summarization using Restricted Self-Attention,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 8072–8076.
  * [10] K. Matsuura, T. Ashihara, T. Moriya, T. Tanaka, A. Ogawa, M. Delcroix, and R. Masumura, ``Leveraging Large Text Corpora for End-to-End Speech Summarization,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [11] K. Matsuura, T. Ashihara, T. Moriya, T. Tanaka, T. Kano, A. Ogawa, and M. Delcroix, ``Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarization,'' in _Proc. Interspeech_ , 2023, pp. 2943–2947.
  * [12] Y. Fathullah, C. Wu, E. Lakomkin, J. Jia, Y. Shangguan, J. Mahadeokar _et al._ , ``AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs,'' _arXiv preprint arXiv:2311.06753_ , 2023.
  * [13] R. Nallapati, B. Zhou, C. dos Santos, Ç. Gulçehre, and B. Xiang, ``Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond,'' in _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , 2016, pp. 280–290.
  * [14] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry _et al._ , ``AudioPaLM: A Large Language Model That Can Speak and Listen,'' _arXiv preprint arXiv:2306.12925_ , 2023.
  * [15] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, ``SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities,'' in _Findings of the Association for Computational Linguistics: EMNLP 2023_ , 2023, pp. 15 757–15 773.
  * [16] Y. Fathullah, C. Wu, E. Lakomkin, J. Jia, Y. Shangguan, K. Li _et al._ , ``Prompting Large Language Models with Speech Recognition Abilities,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 13 351–13 355.
  * [17] J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang _et al._ , ``On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2023, pp. 1–8.
  * [18] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li _et al._ , ``SALMONN: Towards Generic Hearing Abilities for Large Language Models,'' in _International Conference on Learning Representations_ , 2024.
  * [19] J.-W. Jung, R. Sharma, W. Chen, B. Raj, and S. Watanabe, ``AugSumm: Towards Generalizable Speech Summarization Using Synthetic Labels from Large Language Models,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 12 071–12 075.
  * [20] R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze, ``How2: A Large-scale Dataset For Multimodal Language Understanding,'' in _Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL)_. NeurIPS, 2018.
  * [21] C. Zhang, D. Song, Z. Ye, and Y. Gao, ``Towards the Law of Capacity Gap in Distilling Language Models,'' _arXiv preprint arXiv:2311.07052_ , 2023.
  * [22] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [23] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An ASR corpus based on public domain audio books,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [24] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,'' in _Proceedings of the International Conference on Machine Learning_ , 2006, pp. 369–376.
  * [25] G. Hinton, O. Vinyals, and J. Dean, ``Distilling the Knowledge in a Neural Network,'' _arXiv preprint arXiv:1503.02531_ , 2015.
  * [26] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, ``A Comprehensive Overhaul of Feature Distillation,'' in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1921–1930.
  * [27] Y. A. Li, C. Han, V. Raghavan, G. Mischler, and N. Mesgarani, ``StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models,'' _Advances in Neural Information Processing Systems_ , vol. 36, 2023.
  * [28] K. Ito and L. Johnson, ``The LJ Speech Dataset,'' https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [29] I. Loshchilov and F. Hutter, ``Decoupled Weight Decay Regularization,'' in _International Conference on Learning Representations_ , 2019.
  * [30] C.-Y. Lin, ``ROUGE: A Package for Automatic Evaluation of Summaries,'' in _Text Summarization Branches Out_ , 2004, pp. 74–81.
  * [31] M. Denkowski and A. Lavie, ``Meteor Universal: Language Specific Translation Evaluation for Any Target Language,'' in _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , 2014, pp. 376–380.
  * [32] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ``BERTScore: Evaluating Text Generation with BERT,'' in _International Conference on Learning Representations_ , 2020.
  * [33] C. I. Meister and R. Cotterell, ``Language Model Evaluation Beyond Perplexity,'' in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , vol. 1, 2021, pp. 5328–5339.
  * [34] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,'' in _Proc. Interspeech_ , 2017, pp. 498–502.