# Symbolic-Numeric Computation of Integrals in
Successive Galerkin Approximation of
Hamilton-Jacobi-Bellman Equation

Tomoyuki Iori This work was partly supported by JSPS KAKENHI Grant Numbers
JP21K21285 and 22K17855.T. Iori is with the Department of Information and
Physical Sciences, Graduate School of Information Science and Technology,
Osaka University, 1-5 Yamadaoka, Suita, Osaka 565–0871, Japan
<EMAIL_ADDRESS>
###### Abstract

This paper proposes an efficient symbolic-numeric method to compute the
integrals in the successive Galerkin approximation (SGA) of the Hamilton-
Jacobi-Bellman (HJB) equation. A solution of the HJB equation is first
approximated with a linear combination of the Hermite polynomials. The
coefficients of the combination are then computed by iteratively solving a
linear equation, which consists of the integrals of the Hermite polynomials
multiplied by nonlinear functions. The recursive structure of the Hermite
polynomials is inherited by the integrals, and their recurrence relations can
be computed by using the symbolic computation of differential operators. By
using the recurrence relations, all the integrals can be computed from a part
of them that are numerically evaluated. A numerical example is provided to
show the efficiency of the proposed method compared to a standard numerical
integration method.

##  I Introduction

For general nonlinear systems, it is well-known that the optimal feedback
controller with infinite horizon is provided by a solution of the Hamilton-
Jacobi-Bellman (HJB) equation [1]. In the case of linear systems with a
quadratic performance index, the HJB equation reduces to a matrix equation
called the Riccati equation, which can be solved efficiently. However, for
nonlinear cases, the HJB equation is formulated as a nonlinear partial
differential equation (PDE), which is extremely difficult to solve
analytically. Hence, there have been many studies on numerical approximation
techniques of a solution of the HJB equation.

One possibility is to approximate a solution of the HJB equation by the
Taylor-series approximation [2, 3, 4]. By substituting the truncated Taylor-
series expansion of a solution to the HJB equation, a finite number of
algebraic equations for the coefficients of the truncated expansion can be
obtained. Moreover, in [4], a recursive closed form procedure for computing
the coefficients is proposed under several conditions.

The stable manifold method [5] is one of the most promising techniques. In
this method, a sequence of trajectories that converges to a trajectory on the
stable manifold is iteratively computed. After sufficient number of
iterations, the trajectory can be viewed as an approximation of the
Hamiltonian flow on the stable manifold, and thus, it approximately gives the
optimal feedback controller.

Another approach is to solve a linear PDE called the generalized HJB (GHJB)
equation iteratively, which is called a successive approximation approach
(SAA) [6]. As the GHJB equation is still difficult to solve, the GHJB equation
and its solution is further approximated by using multiple basis functions,
which is called the successive Galerkin approximation (SGA). It is proved in
[6] that the solutions of the SGA converges to a solution of the HJB as the
number of basis functions and iterations go to infinity. Several modified
versions of the SAA have been studied [7, 8, 9].

One main issue of the SGA is the computation of integrals associated with the
inner product of a function space. Even if all the integrals can be computed
offline, their computational cost may still be too high to complete within a
reasonable amount of computing power and time. Therefore, it is still
important to seek an efficient method for computing the inner products or the
integrals of nonlinear functions.

In recent years, the symbolic computation of differential operators has been
intensively studied [10, 11, 12, 13], and its applications can be found in
statistics [14, 15], moment problems [16]. In the theory of D-modules [17],
the solutions of PDEs are characterized by an infinite set of differential
operators called an ideal. It can be proven that any ideal is represented by a
finite number of differential operators, and thus, it can be computed
symbolically by using the computer algebra systems (CASs) such as Singular
[18], Macaulay2 [19], and Risa/Asir [20]. In particular, a specific type of
ideals called holonomic ideals and their solutions called holonomic functions
are the main objects manipulated by the symbolic computation. More
specifically, for the integral of a holonomic function, a holonomic ideal that
the integral satisfies can be computed exactly from a holonomic ideal that the
integrand satisfies. This provides a great advantage in the design of
algorithms because the integral of a general nonlinear function cannot be
computed exactly. In addition to differential equations, recurrence equations
can also be treated by the symbolic computation by converting them to
differential equations via the Mellin transform [11, 13].

In this paper, we propose a method to compute recurrence relations satisfied
by the integrals required for the SGA. The expansion of a solution of the HJB
equation with Hermite polynomials, which is holonomic as an infinite sequence
of functions is considered. In addition, the plant dynamics is assumed to be
described by holonomic functions. These problem settings allow us to compute a
set of recurrence relations that are satisfied by the integrals for the SGA.
Once the recurrence equations are computed, any number of the integrals can be
readily computed just by substituting several initial values to the recurrence
equations. Moreover, in contrast to the earlier methods [7, 8], the proposed
method can consider the nonlinearities of the plant dynamics exactly in the
computation of the integrals.

#### Notations

Let ${\boldsymbol{\mathrm{N}}}$ be the set of positive integers and
${\boldsymbol{\mathrm{R}}}$ be the field of real numbers. For
$(i_{1},\dots,i_{p})\in{\boldsymbol{\mathrm{N}}}^{p}$ and
$(x_{1},\dots,x_{n})\in{\boldsymbol{\mathrm{R}}}^{n}$,
$F(i;x)=F(i_{1},\dots,i_{p};x_{1},\dots,x_{n})$ denotes a function that is
defined on a subset of
${\boldsymbol{\mathrm{N}}}^{p}\times{\boldsymbol{\mathrm{R}}}^{n}$. If $p=0$
or $n=0$, $F(i;x)$ or $F(i;x)$ are simply written as $F(i)$ or $F(x)$,
respectively. The symbols ${\mathsf{S}}_{i_{j}},{\mathsf{S}}_{i_{j}}^{-1}\
(j\in\\{1,\dots,p\\})$, and $\partial_{x_{j}}\ (j\in\\{1,\dots,p\\})$ denote
the shift operators for $i_{j}$ and the differential operator for $x_{j}$,
respectively. That is, ${\mathsf{S}}_{i_{j}}\bullet
F(i;x)=F(i_{1},\dots,i_{j}+1,\dots,i_{p};x)$, ${\mathsf{S}}_{i_{j}}\bullet
F(i;x)=F(i_{1},\dots,i_{j}-1,\dots,i_{p};x)$, and $\partial_{x_{j}}\bullet
F(i;x)={\partial{F}/\partial{x_{j}}}(i;x)$, where $\bullet$ denotes the action
of a differential or difference operator on a function. Let
${\boldsymbol{\mathrm{R}}}[x]$ be the set of all polynomials in
$x_{1},\dots,x_{n}$ with coefficients in ${\boldsymbol{\mathrm{R}}}$. The
symbol ${\boldsymbol{\mathrm{R}}}[x]\langle\partial_{x}\rangle$ denotes the
noncommutative ring of differential operators with coefficients in
${\boldsymbol{\mathrm{R}}}[x]$. Similarly,
${\boldsymbol{\mathrm{R}}}[i]\langle{\mathsf{S}}_{i}\rangle$ denotes the
noncommutative ring of difference operators with coefficients in
${\boldsymbol{\mathrm{R}}}[i]$. The symbols
${\boldsymbol{\mathrm{R}}}[x]\langle\partial_{x}\rangle$ and
${\boldsymbol{\mathrm{R}}}[i]\langle{\mathsf{S}}_{i}\rangle$ are also denoted
by ${\mathcal{D}}_{n}$ and ${\mathsf{O}}_{p}$, respectively, if the
indeterminates $x$ and $i$ are clearly specified according to the context. If
${\mathcal{P}}\bullet f=0$ for ${\mathcal{P}}\in{\mathcal{D}}_{n}$,
${\mathcal{P}}$ is said to _annihilate_ a function $f$ and $f$ is a _solution_
of ${\mathcal{P}}$. If ${\mathsf{E}}\bullet f=0$ for
${\mathsf{E}}\in{\mathsf{O}}_{p}$, ${\mathsf{E}}$ is said to _annihilate_ $f$
and $f$ is a _solution_ of ${\mathsf{E}}$.

##  II Problem Setting

Consider the optimal control problem of a nonlinear system:

$\dot{x}=f(x)+g(x)u$ (1)

with a performance index defined as

$J(x,u)\coloneqq\int_{0}^{\infty}q(x(t))+\|u(t)\|^{2}_{R}dt,$ (2)

where $x\in{\boldsymbol{\mathrm{R}}}^{n}$ and
$u\in{\boldsymbol{\mathrm{R}}}^{m}$ are the state and input of the system,
respectively. The nonlinear functions
$f\colon{\boldsymbol{\mathrm{R}}}^{n}\to{\boldsymbol{\mathrm{R}}}^{n}$,
$g\colon{\boldsymbol{\mathrm{R}}}^{n}\to{\boldsymbol{\mathrm{R}}}^{n\times
m}$, and $q\colon{\boldsymbol{\mathrm{R}}}^{n}\to{\boldsymbol{\mathrm{R}}}$
are assumed to consists of holonomic functions, which will be defined in
Section III. It is also assumed that $q(x)\geq 0\
(x\in{\boldsymbol{\mathrm{R}}}^{n})$ and
$R\in{\boldsymbol{\mathrm{R}}}^{m\times m}$ be positive definite.

For a given feedback law $u(x)$, the performance index for the closed loop
trajectory $\phi(t)$ is defined as

$V(x_{0},u(\cdot))\coloneqq\int_{0}^{\infty}q(\phi(t))+\|u(\phi(t))\|^{2}_{R}dt,$
(3)

where $\phi(0)=x_{0}$. The optimal feedback law $u^{*}(x)$ is given as

$u^{*}=-\frac{1}{2}R^{-1}g^{\top}{\frac{\partial{V^{*}}}{\partial{x}}}^{\top},$
(4)

where $V^{*}\colon{\boldsymbol{\mathrm{R}}}^{n}\to{\boldsymbol{\mathrm{R}}}$
is a solution of the HJB equation

$\mathrm{HJB}(V^{*})\coloneqq{\frac{\partial{V^{*}}}{\partial{x}}}f+q-\frac{1}{4}{\frac{\partial{V^{*}}}{\partial{x}}}gR^{-1}g^{\top}{\frac{\partial{V^{*}}}{\partial{x}}}^{\top}=0,$
(5)

such that $V^{*}(0)=0$.

In the SAA, a solution of the HJB equation is iteratively approximated by
solving the GHJB equation defined as

$\mathrm{GHJB}(V,u)\coloneqq{\frac{\partial{V}}{\partial{x}}}(f+gu)+q+\|u\|^{2}_{R}=0.$
(6)

As the GHJB equation is still difficult to solve, the iteration $V^{(l)}$ is
further approximated by using a finite number of basis functions
$\phi_{1}(x),\dots,\phi_{N}(x)$ as

$V^{(l)}(x)\coloneqq\sum_{i=1}^{N}v_{i}^{(l)}\phi_{i}(x)=\Phi^{\top}(x)v^{(l)}\quad\left(v_{i}^{(l)}\in{\boldsymbol{\mathrm{R}}}\right),$
(7)

where $\Phi(x)\coloneqq[\phi_{1}(x)\ \cdots\ \phi_{N}(x)]^{\top}$ and
$v^{(l)}\coloneqq[v_{1}^{(l)}\ \cdots\ \allowbreak v_{N}^{(l)}]^{\top}$. This
also yields a representation of $u^{(l+1)}$ as

$\displaystyle u^{(l+1)}(x)$
$\displaystyle=-\frac{1}{2}R^{-1}g^{\top}(x){\frac{\partial{V^{(l)}}}{\partial{x}}}^{\top}(x)$
(8)
$\displaystyle=-\frac{1}{2}R^{-1}g^{\top}(x){\frac{\partial{\Phi}}{\partial{x}}}^{\top}(x)v^{(l)}.$
(9)

The functional equation $\mathrm{GHJB}(V^{(l)},u^{(l)})=0$ is then
approximated with the following finite number of algebraic equations for
$v_{i}^{(l)}$:

$\left\langle\mathrm{GHJB}(V^{(l)},u^{(l)}),\phi_{k}\right\rangle=0\quad(p=1,\dots,N),$
(10)

where $\langle\cdot,\cdot\rangle$ is an inner product of functions defined as

$\langle
f(x),g(x)\rangle\coloneqq\int_{{\boldsymbol{\mathrm{R}}}^{n}}w(x)f(x)g(x)dx$
(11)

with a weighting function $w(x)>0\ (x\in{\boldsymbol{\mathrm{R}}}^{n})$. By
substituting (7) and (9) into (10), we obtain

$\displaystyle\left\langle\mathrm{GHJB}(V^{(l)},u^{(l)}),\phi_{k}\right\rangle=\sum_{i=1}^{N}v_{i}^{(l)}\left\langle{\frac{\partial{\phi_{i}}}{\partial{x}}}f,\phi_{k}\right\rangle$
(12)
$\displaystyle-\frac{1}{2}\sum_{i=1}^{N}v_{i}^{(l)}\sum_{j=1}^{N}v_{j}^{(l-1)}\left\langle{\frac{\partial{\phi_{i}}}{\partial{x}}}GR^{-1}G^{\top}{\frac{\partial{\phi_{j}}}{\partial{x}}}^{\top},\phi_{k}\right\rangle$
(13) $\displaystyle+\left\langle
q,\phi_{k}\right\rangle+\frac{1}{4}\sum_{i=1}^{N}v_{i}^{(l-1)}\sum_{j=1}^{N}v_{j}^{(l-1)}\left\langle{\frac{\partial{\phi_{i}}}{\partial{x}}}GR^{-1}G^{\top}{\frac{\partial{\phi_{j}}}{\partial{x}}}^{\top},\phi_{k}\right\rangle.$
(14)

Now, let us define

$\begin{gathered}a(i,j,k)\coloneqq\left\langle{\frac{\partial{\phi_{i}}}{\partial{x}}}GR^{-1}G^{\top}{\frac{\partial{\phi_{j}}}{\partial{x}}}^{\top},\phi_{k}\right\rangle,\\\
b(i,k)\coloneqq\left\langle{\frac{\partial{\phi_{i}}}{\partial{x}}}f,\phi_{k}\right\rangle,\quad
c(k)\coloneqq\left\langle q(x),\phi_{k}(x)\right\rangle.\end{gathered}$ (15)

In addition, let $A_{k}$ be the $N\times N$ matrix having $a(i,j,k)$ as its
$(i,j)$-component and $b_{k}$ be the $N$-dimensional vector having $b(i,k)$ as
its $i$-th component. The approximated GHJB equation (10) is then rewritten as
the following linear equations for $v^{(l)}$:

$\mathrm{LE}(v^{(l)},v^{(l-1)})\coloneqq\\\
{v^{(l)}}^{\top}\left(b_{k}-\frac{1}{2}A_{k}v^{(l-1)}\right)+c(k)+\frac{1}{4}\left\|v^{(l-1)}\right\|_{A_{k}}^{2}=0\\\
(k=1,\dots,N).$ (16)

In what follows, for a given $N\in{\boldsymbol{\mathrm{N}}}$, $A_{N}$,
$B_{N}$, and $C_{N}$ denote the sets of the integrals required to define (16),
that is,

$\displaystyle A_{N}\coloneqq\\{a(i,j,k)\mid i,j,k\in\\{1,\dots,N\\}\\},$ (17)
$\displaystyle B_{N}\coloneqq\\{b(i,k)\mid i,k\in\\{1,\dots,N\\}\\},$ (18)
$\displaystyle C_{N}\coloneqq\\{c(k)\mid k\in\\{1,\dots,N\\}\\}.$ (19)

Once the integrals $A_{N}$, $B_{N}$, and $C_{N}$ are computed, the SGA
procedure can be performed by solving the linear equation (16) iteratively,
which is summarized as Algorithm 1.

Algorithm 1 Successive Galerkin approximation

1:Initial guess $v^{(0)}$, sets of integrals $A_{N}$, $B_{N}$, and $C_{N}$,
error tolerance $\epsilon$, max iteration $l_{\mathrm{max}}$

2:Approximation of optimal value function $V^{(l)}(x)=\Phi^{\top}(x)v^{(l)}$

3:$l\leftarrow 0$

4:while $\mathrm{LE}(v^{(l)},v^{(l)})>\epsilon$ and $l<l_{\mathrm{max}}$ do

5: Solve linear equations (16) and compute $v^{(l+1)}$

6: $l\leftarrow l+1$

7:return $V^{(l)}=\Phi^{\top}(x)v^{(l)}$

Although $A_{N}$, $B_{N}$, and $C_{N}$ are never computed again in the
iteration, they are defined as integrals of nonlinear functions, which are
extremely difficult to compute analytically. Hence, we have to rely on
numerical integration methods, which leads to high computational cost.
However, if the basis functions have some recurrence relations such as those
of the orthogonal polynomials, we can expect that the elements of $A_{N}$,
$B_{N}$, and $C_{N}$ may also have similar recursive relations. Such recursive
relations can be computed by using the symbolic computation of differential
operators, which will be briefly introduced in the next section.

##  III Holonomic Functions and Ideals

This section is devoted to introducing some definitions and lemmas related to
the symbolic computation of differential operators, referring to [11, 12]. In
this section, $x=[x_{1}\ \cdots\
x_{n}]^{\top}\in{\boldsymbol{\mathrm{R}}}^{n}$, $s=[s_{1}\ \cdots\
s_{p}]^{\top}\in{\boldsymbol{\mathrm{R}}}^{p}$, and $i=[i_{1}\ \cdots\
i_{p}]^{\top}\in{\boldsymbol{\mathrm{Z}}}^{p}$. Let ${\mathcal{D}}_{n}$ and
${\mathcal{D}}_{n+p}$ denote
${\boldsymbol{\mathrm{R}}}[x]\langle\partial_{x}\rangle$ and
${\boldsymbol{\mathrm{R}}}[x,s]\langle\partial_{x},\partial_{s}\rangle$,
respectively. Furthermore, let ${\mathcal{D}}_{n}\langle
i,{\mathsf{S}}_{i},{\mathsf{S}}_{i}^{-1}\rangle$ denote
${\mathcal{D}}_{n}$-algebra generated by $i$,
${\mathsf{S}}_{i}=[{\mathsf{S}}_{i_{1}}\ \cdots\
{\mathsf{S}}_{i_{p}}]^{\top}$, and
${\mathsf{S}}_{i}^{-1}=[{\mathsf{S}}_{i_{1}}^{-1}\ \cdots\
{\mathsf{S}}_{i_{p}}^{-1}]^{\top}$.

Consider a set of PDEs for an unknown function $f(x)$:

${\mathcal{P}}_{1}\bullet f=\cdots{\mathcal{P}}_{d}\bullet f=0,$ (20)

where ${\mathcal{P}}_{1},\dots,{\mathcal{P}}_{d}\in{\mathcal{D}}_{n}$. For any
solution $f$ of (20) and any differential operator
${\mathcal{Q}}\in{\mathcal{D}}_{n}$, it is obvious that
${\mathcal{Q}}{\mathcal{P}}_{j}\bullet f={\mathcal{Q}}\bullet 0=0$, which
leads to consider a left ideal $I\subset{\mathcal{D}}_{n}$ defined as
$I\coloneqq\\{{\mathcal{Q}}_{1}{\mathcal{P}}_{1}+\cdots{\mathcal{Q}}_{d}{\mathcal{P}}_{d}\in{\mathcal{D}}_{n}\mid{\mathcal{Q}}_{1},\dots,{\mathcal{Q}}_{d}\in{\mathcal{D}}_{n}\\}$.
In this paper, the adjective “left” is omitted because all ideals in this
paper are left ideals.

######  Definition 1 (Holonomic ideal of ${\mathcal{D}}_{n}$)

An ideal $I\subset{\mathcal{D}}_{n}$ is holonomic if the quotient
${\mathcal{D}}_{n}/I$, which can be viewed as a left
${\mathcal{D}}_{n}$-module, has the dimension $n$.

For the definition of the dimension of left ${\mathcal{D}}_{n}$-modules, see
[17, 10, 12]. The notion of holonomic ideals plays an important role in the
symbolic computation of differential operators because of its finiteness [10].

The solutions of holonomic ideals are called holonomic functions and include
most nonlinear functions appears in systems and control theory [22, 23].

######  Definition 2 (Holonomic function)

An analytic function $f(x)$ is said to be holonomic if $f$ is annihilated by a
holonomic ideal $I\subset{\mathcal{D}}_{n}$, that is, ${\mathcal{P}}\bullet
f=0$ for all ${\mathcal{P}}\in I$.

The class of holonomic functions is closed under multiplication and
integration; the product of two holonomic functions and the integral of a
holonomic function are also holonomic[11].

###### Lemma 1

For two holonomic functions $f(x)$ and $g(x)$, their product $(f\cdot g)(x)$
is holonomic.

###### Lemma 2

Suppose a holonomic function $f(x)$ is rapidly decreasing with respect to
$x_{1}$, that is, $\lim_{x_{1}\to\infty}x_{1}^{i}\partial_{x_{1}}^{j}f(x)=0$
for any nonnegative integers $i,j$. Then, the integral
$\int_{-\infty}^{\infty}f(x)dx_{1}$ is holonomic as a function of
$[x_{2},\dots,x_{n}]^{\top}$.

Recurrence relations of infinite sequences can be converted into PDEs and
manipulated by the symbolic computation of differential operators through the
Mellin transform.

######  Definition 3 (Mellin transform between $\partial$ and ${\mathsf{S}}$)

A mapping $\mu\colon{\mathcal{D}}_{n+p}\to{\mathcal{D}}_{n}\langle
i,{\mathsf{S}}_{i},{\mathsf{S}}_{i}^{-1}\rangle$ defined as

$\mu(s_{j})={\mathsf{S}}_{i_{j}},\quad\mu(\partial_{s_{j}})=-i_{j}{\mathsf{S}}_{i_{j}}^{-1}$
(21)

is called the Mellin transform, and a mapping
$\hat{\mu}\colon{\mathcal{D}}_{n}\langle
i,{\mathsf{S}}_{i}\rangle\allowbreak\to{\mathcal{D}}_{n+p}$ defined as

$\hat{\mu}(i_{j})=-\partial_{s_{j}}s_{j}=-s_{j}\partial{s_{j}}-1,\quad\hat{\mu}({\mathsf{S}}_{i_{j}})=s_{j}$
(22)

is called the inverse Mellin transform.

The Mellin transform is related to the multivariate z-transformation
$\mathscr{Z}$ between a multivariate function and an infinite sequence of
functions:

$\displaystyle\mathscr{Z}[f](s,x)$
$\displaystyle\coloneqq\sum_{i_{1}=0}^{\infty}\cdots\sum_{i_{p}=0}^{\infty}f(i;x)s_{1}^{-i_{1}-1}\cdots
s_{p}^{-i_{p}-1},$ (23) $\displaystyle\mathscr{Z}^{-1}[g](i;x)$
$\displaystyle\coloneqq\int_{C}g(s,x)s_{1}^{i_{1}}\cdots s_{p}^{i_{p}}ds,$
(24)

where $C$ is the product of unit circles centered at the origin of the complex
plane. It is readily seen that

$\hat{\mu}(i)\bullet\mathscr{Z}[f]=\mathscr{Z}[if],\quad\hat{\mu}({\mathsf{S}}_{i})\bullet\mathscr{Z}[f]=\mathscr{Z}[{\mathsf{S}}_{i}f].$
(25)

##  IV Recurrence Equations of Integrals

Hereafter, suppose $n=1$ for simplicity and that the $i$-th basis function
$\phi_{i}(x)$ in (7) is the Hermite polynomial $H(i;x)$ of degree $i$. For
multi-dimensional cases, each basis function can be defined as the product of
the Hermite polynomials $H(i_{1};x_{1})H(i_{2};x_{2})\cdots H(i_{n};x_{n})$,
and the following discussions can be applied to with appropriate
modifications, which will be a part of future work.

With the weighting function $w(x)=\exp(-x^{2})$, the z-transforms of integrals
(15) are obtained as follows.

$\displaystyle\mathscr{Z}[a](s,t,v)$
$\displaystyle=\int_{-\infty}^{\infty}\frac{1}{R}\Big{\\{}e^{-x^{2}}g^{2}(x)\mathscr{Z}[\partial_{x}H](s,x)$
(26)
$\displaystyle\qquad\qquad\mathscr{Z}[\partial_{x}H](t,x)\mathscr{Z}[H](u,x)\Big{\\}}dx,$
(27) $\displaystyle\mathscr{Z}[b](s,v)$
$\displaystyle=\int_{-\infty}^{\infty}e^{-x^{2}}\mathscr{Z}[\partial_{x}H](s,x)f(x)\mathscr{Z}[H](u,x)dx,$
(28) $\displaystyle\mathscr{Z}[c](v)$
$\displaystyle=\int_{-\infty}^{\infty}e^{-x^{2}}q(x)\mathscr{Z}[H](u,x)dx.$
(29)

If all the factors of the integrands in (27)–(29) are holonomic, these
z-transforms are also holonomic. Moreover, for each of (27)–(29), some
differential operators that annihilates it can be computed by using the
symbolic computation (for detailed algorithms, see [10, 11, 12]). Accordingly,
we can obtain some recurrence relations form the differential operators via
the Mellin transform (21).

To this end, we next show that the z-transforms of the Hermite polynomials and
their derivatives are holonomic. The Hermite polynomials satisfy the following
difference-differential equations:

$\displaystyle(\partial_{x}^{2}-2x\partial_{x}+2i)\bullet H(i;x)=0,$ (30)
$\displaystyle({\mathsf{S}}_{i}^{2}-2x{\mathsf{S}}_{i}+2(i+1))\bullet
H(i;x)=0.$ (31)

By the inverse Mellin transform (22), the following two differential operators
are obtained.

$\displaystyle(\partial_{x}^{2}-2x\partial_{x}-2s\partial_{s}-2)\bullet\mathscr{Z}[H](s,x)=0,$
(32) $\displaystyle(s^{2}-2xs-2s\partial_{s})\bullet\mathscr{Z}[H](s,x)=0.$
(33)

Moreover, it can be readily verified that (32) and (33) generate a holonomic
ideal by using a CAS. Hence, from Definition 2, $\mathscr{Z}[H](s,x)$ is
holonomic.

###### Lemma 3

The z-transform of the Hermite polynomials $\mathscr{Z}[H](s,x)$ is holonomic.

As the derivative $\partial_{x}H(i;x)$ also appears in (27) and (28), we next
show that $\partial_{x}H(i;x)$ is holonomic. To this end, the following
equality satisfied by $H(i;x)$ is useful:

$\partial_{x}H(i;x)=2iH(i-1;x).$ (34)

From (30), $(\partial_{x}^{2}-2x\partial_{x}+2(i-1))\bullet H(i-1;x)=0$ holds.
As $2i$ and $\partial_{x}^{2}-2x\partial_{x}+2(i-1)$ commute, we have

$\displaystyle 0$
$\displaystyle=2i(\partial_{x}^{2}-2x\partial_{x}+2(i-1))\bullet H(i-1;x)$
(35) $\displaystyle=(\partial_{x}^{2}-2x\partial_{x}+2(i-1))\bullet
2iH(i-1;x)$ (36)
$\displaystyle=(\partial_{x}^{2}-2x\partial_{x}+2(i-1))\bullet\partial_{x}H(i;x).$
(37)

On the other hand, $({\mathsf{S}}_{i}^{2}-2x{\mathsf{S}}_{i}+2i)\bullet
H(i-1;x)=0$ holds from (31). By multiplying the both sides by $2(i+1)(i+2)$
from the left, we obtain

$\displaystyle 0$
$\displaystyle=2(i+1)(i+2)({\mathsf{S}}_{i}^{2}-2x{\mathsf{S}}_{i}+2i)\bullet
H(i-1;x)$ (38)
$\displaystyle=((i+1){\mathsf{S}}_{i}^{2}-2x(i+2){\mathsf{S}}_{i}+2(i+1)(i+2))$
(39) $\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\bullet 2iH(i-1;x),$
(40)
$\displaystyle=((i+1){\mathsf{S}}_{i}^{2}-2x(i+2){\mathsf{S}}_{i}+2(i+1)(i+2))$
(41)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\bullet\partial_{x}H(i;x),$
(42)

where the equalities $(i+2){\mathsf{S}}_{i}^{2}={\mathsf{S}}_{i}^{2}i$ and
$(i+1){\mathsf{S}}_{i}={\mathsf{S}}_{i}i$ are used for deriving the second
line. Eventually, we obtain the differential operators that annihilates
$\mathscr{Z}[\partial_{x}H](s,x)$ as the inverse Mellin transform of (37) and
(42), that is,

$\displaystyle\hat{\mu}(\partial_{x}^{2}-2x\partial_{x}+2(i-1))=\partial_{x}^{2}-2x\partial_{x}-2s\partial_{s}-4,$
(43)
$\displaystyle\hat{\mu}((i+1){\mathsf{S}}_{i}^{2}-2x(i+2){\mathsf{S}}_{i}+2(i+1)(i+2))$
(44)
$\displaystyle=2s^{2}\partial_{s}^{2}+(2s^{2}x-s^{3})\partial_{s}-2s^{2}.$
(45)

We can confirm that the differential operators in (43) and (45) generate a
holonomic ideal by using a CAS and prove the following lemma.

###### Lemma 4

The z-transform $\mathscr{Z}[\partial_{x}H](s,x)$ is holonomic.

As each factor of the integrands in (27)–(29) is holonomic, the integrands are
also holonomic (Lemma 1). Here, the following assumption is made.

###### Assumption 1

All the integrands in (27)–(29) are rapidly decreasing with respect to $x$.

Note that Assumption 1 holds as long as $f$, $g$, and $q$ are of exponential
growth. Hence, (27)–(29) are holonomic (Lemma 2), and the differential
operators annihilating them can be computed by symbolic computation [11].
Consequently, recurrence relations satisfied by $a(i,j,k)$, $b(i,k)$, and
$c(k)$ can be obtained from the differential operators via the Mellin
transform (21). The whole procedure is summarized in Algorithm 2.

Algorithm 2 Derivation of recurrence relations satisfied by $a[i,j,k]$,
$b[i,k]$, and $c[k]$

1:Sets of differential operators
${\mathcal{G}}_{f},{\mathcal{G}}_{g},{\mathcal{G}}_{q}\subset{\mathcal{D}}_{1}$
that annihilate $f(x),g(x),q(x)$, respectively, and
${\mathcal{G}}_{H},{\mathcal{G}}_{\partial H}\subset{\mathcal{D}}_{2}$ that
annihilate $\mathscr{Z}[H](s,x),\mathscr{Z}[\partial_{x}H](s,x)$, respectively

2:Set of difference operators ${\mathsf{G}}_{a}\subset{\mathsf{O}}_{3}$,
${\mathsf{G}}_{b}\subset{\mathsf{O}}_{2}$, and
${\mathsf{G}}_{c}\subset{\mathsf{O}}_{1}$ that annihilate $a(i,j,k)$,
$b(i,k)$, and $c(k)$, respectively

3:Compute sets of differential operators
${\mathcal{J}}_{a}\subset{\mathcal{D}}_{4}$,
${\mathcal{J}}_{b}\subset{\mathcal{D}}_{3}$, and
${\mathcal{J}}_{c}\subset{\mathcal{D}}_{2}$ that annihilate the integrands of
(27), (28), and (29), respectively, from inputs and $\partial_{x}^{2}+2x$ that
annihilates $\exp(-x^{2})$

4:Compute sets of differential operators
${\mathcal{K}}_{a}\subset{\mathcal{D}}_{3}$,
${\mathcal{K}}_{b}\subset{\mathcal{D}}_{2}$, and
${\mathcal{K}}_{c}\subset{\mathcal{D}}$ that annihilates $\mathscr{Z}[a]$,
$\mathscr{Z}[b]$, and $\mathscr{Z}[c]$ from ${\mathcal{J}}_{a}$,
${\mathcal{J}}_{b}$, and ${\mathcal{J}}_{c}$, respectively

5:Compute sets of difference operators ${\mathsf{G}}_{a}$, ${\mathsf{G}}_{b}$,
and ${\mathsf{G}}_{c}$ from ${\mathcal{K}}_{a}$, ${\mathcal{K}}_{b}$, and
${\mathcal{K}}_{c}$, respectively, via Mellin transform (21)

##  V Numerical Example

This section presents a numerical example to show the efficiency of the
proposed method. In the following demonstration, Risa/Asir and Julia are used
to perform symbolic computation and numerical computation, respectively, of
the proposed method. In addition, Maple is used to perform an existing
numerical integration method for comparison. All computations are performed on
a PC(Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz; RAM: 64 GB).

###  V-A Problem setting

Consider the nonlinear optimal control problem of the nonlinear scalar system:

$\dot{x}=\sin(x)+u(x),\quad
J=\int_{0}^{\infty}\frac{1}{2}x^{2}(t)+\frac{1}{2}u^{2}(t)dt.$ (46)

The nonlinear functions $\sin(x)$ and $x^{2}$ are holonomic, which are
annihilated by differential operators $\partial_{x}^{2}+1$ and
$\partial_{x}^{3}$, respectively.

To perform the SGA (Algorithm 1), the following integrals need to be computed:

$\displaystyle a(i,j,k)$ $\displaystyle=\left\langle
2\partial_{x}H(i;x)\partial_{x}H(j;x),H(k;x)\right\rangle,$ (47)
$\displaystyle b(i,k)$
$\displaystyle=\left\langle\partial_{x}H(i;x)\sin(x),H(k;x)\right\rangle,$
(48) $\displaystyle c(k)$ $\displaystyle=\frac{1}{2}\left\langle
x^{2},H(k;x)\right\rangle.$ (49)

###  V-B Symbolic part

To obtain the recurrence relations satisfied by $a(i,j,k)$, $b(i,k)$, and
$c(k)$, Algorithm 2 can be applied to the inputs
${\mathcal{G}}_{f}=\\{\partial_{x}^{2}+1\\}$, ${\mathcal{G}}_{g}=\\{0\\}$,
${\mathcal{G}}_{q}=\\{\partial_{x}^{3}\\}$, ${\mathcal{G}}_{H}$ being the set
of (32) and (33), and ${\mathcal{G}}_{\partial H}$ being the set of (43) and
(45). For example,
${\mathcal{J}}_{a}\subset{\boldsymbol{\mathrm{R}}}[x,s,t,v]\langle\partial_{x},\partial_{s},\partial_{t},\partial_{v}\rangle$
is obtained as the set of 19 differential operators including

$\displaystyle\partial_{x}+2s\partial_{s}^{2}+(2sx-s^{2}+2)\partial_{s}$ (50)
$\displaystyle\quad+2t\partial_{t}+(2tx-t^{2}+2)\partial_{t}+v-2s-2t,$ (51)

where the remaining elements are omitted due to the space limit. It can be
seen that (51) is invariant under the interchange of $s$ and $t$, which is
consistent with the integrand of (27) being invariant under the interchange of
$i=\mu(-s\partial_{s}-1)$ and $j=\mu(-t\partial_{t}-1)$. As the output of
Algorithm 2, we obtain ${\mathsf{G}}_{a}$, ${\mathsf{G}}_{b}$, and
${\mathsf{G}}_{c}$ as listed in Table I.

TABLE I: Difference operators annihilating integrals $a(i,j,k)$, $b(i,k)$, and $c(k)$ ${\mathsf{G}}_{a}$ |  $(j+1)(-j+i+k+1){\mathsf{S}}_{i}+(i+1)(-j+i-k-1){\mathsf{S}}_{j}$
---|---
$(k+1)(-k+i+j){\mathsf{S}}_{j}-(j+2)(-j+i+k){\mathsf{S}}_{k}$
$(-j+i+k+1)(-k-3-j+i){\mathsf{S}}_{k}^{2}+4(k+2)(k+1)(i+j-k-1)$
${\mathsf{G}}_{b}$ |  $i(2i-2k-5){\mathsf{S}}_{i}{\mathsf{S}}_{k}+(i+1){\mathsf{S}}_{k}^{2}-4(k+1)(i+1)(i-k-2)$
$i(2i-2k-5){\mathsf{S}}_{i}^{2}+(i+2)(2i-2k+1){\mathsf{S}}_{k}^{2}+4(i+2)(-k-1+i)(2i^{2}-4ik+2k^{2}-5i+3k-2)$
$(k+1)(-2k+2i-7){\mathsf{S}}_{i}+i(i+1)(-k-1+i){\mathsf{S}}_{k}^{3}+4(i+1)(i^{3}+(-3k-7)i^{2}+(3k^{2}+13k+\frac{53}{4})i-k^{3}-6k^{2}-11k-6){\mathsf{S}}_{k}$
${\mathsf{G}}_{c}$ |  ${\mathsf{S}}_{k}^{3}$

Note that $c(k)$ for this numerical example can be computed analytically as

$c(k)=\begin{cases}\sqrt{\pi}&(k=2)\\\ 0&(\mbox{otherwise}),\end{cases}$ (52)

which obviously satisfies ${\mathsf{S}}_{k}^{3}\bullet c(k)=c(k+3)=0$ for
$k\in{\boldsymbol{\mathrm{N}}}$.

###  V-C Numerical part

From the recurrence relations obtained from the output of Algorithm 2, the
sets of integrals $A_{N}$, $B_{N}$, and $C_{N}$ can be computed recursively.
Note that for a certain set of indices, the coefficients of the shift
operators ${\mathsf{S}}_{i}$, ${\mathsf{S}}_{j}$, and ${\mathsf{S}}_{k}$ in
each difference operator may vanish, which indicates that the difference
operator does not imply any recurrence relation on the set of indices.

For example, the recurrence relation obtained from the second element of
${\mathsf{G}}_{a}$ in Table I:

$(k+1)(-k+i+j)a(i,j+1,k)\\\ -(j+2)(-j+i+k)a(i,j,k+1)=0$ (53)

can be used to compute one of $a(i,j+1,k)$ and $a(i,j,k+1)$ from the other.
However, when the indices $i$, $j$, and $k$ satisfy $i+j-k=0$, the first term
of (53) vanishes regardless of the value $a(i,j+1,k)$, which just indicates
$(j+2)(-j+i+k)a(i,j,k+1)=0$, and does not give any information about the other
values of $A_{N}$. For this reason, to compute all values of $A_{N}$, its
several elements must be computed by using a numerical integration method. In
this example, the following ten values are required:
$a(1,1,1)=a(1,1,2)=a(2,1,2)=a(1,2,2)=a(4,1,1)=a(4,1,2)=a(1,4,1)=a(1,4,2)=0,a(2,1,1)=a(1,2,1)=56.7$.
Similarly, to compute all values of $B_{N}$, the following eight values are
required: $b(1,2)=b(2,1)=b(3,2)=b(4,1)=0,b(1,1)=2.76,\quad
b(1,3)=-2.76,b(3,1)=24.8,\quad b(3,3)=107$.

###  V-D Results and comparison

For $N=10,15$, the computational time of the proposed method is compared to
that of an existing numerical integration method implemented in Maple with
four significant digits. The computational times are summarized in Table II.

TABLE II: Comparison of computational times [s] with numerical integration |  | Proposed | Maple
---|---|---|---
Recurrence relations | $47.7$ | N/A
$N=10$ | $a(i,j,k)$ | $1.42\times 10^{-3}$ | $98.2$
$b(i,k)$ | $3.26\times 10^{-4}$ | $1.43\times 10^{2}$
Total | $47.7$ | $2.44\times 10^{2}$
$N=15$ | $a(i,j,k)$ | $1.51\times 10^{-2}$ | $2.28\times 10^{2}$
$b(i,k)$ | $6.26\times 10^{-4}$ | $3.39\times 10^{2}$
Total | $47.7$ | $5.67\times 10^{2}$

As can be seen, the total times of the proposed method to compute $A_{N}$ and
$B_{N}$ are much smaller than those of the existing method for both cases of
$N=10,15$. In particular, the numerical part of the proposed method is
finished within one second because it just consists of recursive
substitutions.

The SGA (Algorithm 1) is performed with the integrals $A_{N}$, $B_{N}$, and
$C_{N}$ for $N=4,6,\dots,14$. The initial guess $v^{(0)}$ is set to

$v^{(0)}_{k}=\begin{cases}\frac{1+\sqrt{2}}{8}&(k=2)\\\
0&(\mbox{otherwise})\end{cases},$ (54)

which corresponds to $V^{(0)}(x)=v^{(0)}_{2}H(2;x)$ and is equal to the value
function $V(x)=(1+\sqrt{2})x^{2}/2$ of the linearized system $\dot{x}=x+u$
with $J$ in (46) up to a constant. Figure 1 shows $V^{(0)}(x)$ and $V^{*}(x)$
obtained for $N=4,6,\dots,14$, where the constant term of each value function
is adjusted so that $V^{*}(0)=0$.

Figure 1: Value functions

In Fig. 2, $\mathrm{HJB}(V^{(0)}(x))$ and $\mathrm{HJB}(V^{*}(x))$ of the
value functions for $N=4,8,14$ are depicted. As can be seen, the region where
$\mathrm{HJB}(V^{*}(x))\simeq 0$ glows as $N$ increases, which indicates that
$A_{N}$ and $B_{N}$ are computed validly by the proposed method.

Figure 2: Evaluation of HJB equation

Finally, the state and input trajectories with $x(0)=4$ are illustrated in
Fig. 3, which shows that the feedback law obtained from $V^{*}(x)$ with $N=14$
stabilizes the system while suppressing the magnitude of the input.

(a) State

(b) Input

Figure 3: Trajectories with $x(0)=4$

##  VI Conclusion

In this paper, a symbolic-numeric computation method for the integrals in the
successive approximation algorithm has been proposed. The successive
approximation algorithm approximates a solution of Hamilton-Jacobi-Bellman
equation by iteratively solving a linear equation, which is defined with
multiple integrals of nonlinear functions. By using the symbolic computation
of differential operators and the Mellin transform of difference-differential
operators, a set of recurrence relations satisfied by the integrals can be
computed. From a finite number of the integrals computed by numerical
integration, the other integrals can be computed up to any number by using the
recurrence relations.

Future work directions of this study includes the extension of the proposed
method to the case of multi-dimensional systems. Moreover, the approximation
with another orthogonal polynomial such as the Chebyshev polynomials could be
included.

## References

  * [1] J. A. E. Bryson and Y.-C. Ho  Applied Optimal Control, John Wiley & Sons, 1st edition, 1975.
  * [2] D. L. Lukes, “Optimal regulation of nonlinear dynamical systems,”  SIAM Journal on Control, vol. 7, no. 1, pp. 75–100, 1969\.
  * [3] W. L. Garrard, “Suboptimal feedback control for nonlinear systems,”  Automatica, vol. 8, no. 2, pp. 219–221, 1972\.
  * [4] H. Almubarak, N. Sadegh, and D. G. Taylor, “Infinite horizon nonlinear quadratic cost regulator,”  Proceedings of the American Control Conference, vol. 2019-July, pp. 5570–5575, 2019.
  * [5] N. Sakamoto and A. J. van der Schaft, “Analytical approximation methods for the stabilizing solution of the Hamilton-Jacobi equation,”  IEEE Transactions on Automatic Control, vol. 53, no. 10, pp. 2335–2350, 2008.
  * [6] R. W. Beard, G. N. Saridis, and J. T. Wen, “Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation,”  Automatica, vol. 33, no. 12, pp. 2159–2177, 1997.
  * [7] H. Mizuno and K. Fujimoto, “Approximate solution to Hamilton-Jacobi equation based on Chebyshev polynomials,”  Transactions of the Society of Instrument and Control Engineers, vol. 44, no. 2, pp. 113–118, 2008.
  * [8] D. Kalise and K. Kunisch, “Polynomial approximation of high-dimensional HamiltonâJacobiâBellman equations and applications to feedback control of semilinear parabolic PDES,”  SIAM Journal on Scientific Computing, vol. 40, no. 2, pp. A629–A652, 2018.
  * [9] I. Maruta, S. Nishida, and K. Fujimoto, “A Study on Numerical Solutions of Hamilton-Jacobi-Bellman Equations Based on Successive Approximation Approach,”  SICE Journal of Control, Measurement, and System Integration, vol. 13, no. 3, pp. 157–163, 2020.
  * [10] M. Saito, B. Sturmfels, and N. Takayama  Gröbner Deformations of Hypergeometric Differential Equations, Springer-Verlag, 2000.
  * [11] T. Oaku, Y. Shiraki, and N. Takayama, “Algebraic algorithms for D-Modules and numerical analysis,”  Computer Mathematics (Proceedings of ASCM 2003), vol. 10, pp. 23–39, 2003.
  * [12] T. Hibi ed.  Gröbner Bases: Statistics and Software Systems, Springer Japan, 1st edition, 2013.
  * [13] T. Oaku, “Algorithms for $D$-modules, integration, and generalized functions with applications to statistics,”  Advanced Studies in Pure Mathematics, vol. 77, pp. 253–352, 2018.
  * [14] H. Nakayama, K. Nishiyama, M. Noro, K. Ohara, T. Sei, N. Takayama, and A. Takemura, “Holonomic gradient descent and its application to the Fisher-Bingham integral,”  Advances in Applied Mathematics, vol. 47, no. 3, pp. 639–658, 2011.
  * [15] A. Kume and T. Sei, “On the exact maximum likelihood inference of FisherâBingham distributions using an adjusted holonomic gradient method,”  Statistics and Computing, vol. 28, no. 4, pp. 835–847, 2018.
  * [16] F. Bréhard, M. Joldes, and J.-B. Lasserre, “On a moment problem with holonomic functions,” in Proceedings of the 44th International Symposium on Symbolic and Algebraic Computation, pp. 66–73, 2019.
  * [17] S. C. Coutinho  A Primer of Algebraic D-Modules, Cambridge University Press, 1995.
  * [18] W. Decker, G.-M. Greuel, G. Pfister, and H. Schönemann, “Singular 4-2-0 —A computer algebra system for polynomial computations,” 2019.
  * [19] D. R. Grayson and M. E. Stillman, “Macaulay2, a software system for research in algebraic geometry,” 2020.
  * [20] M. Noro, N. Takayama, H. Nakayama, K. Nishiyama, and K. Ohara, “Risa/Asir: A computer algebra system,” 2020.
  * [21] R. W. Beard, G. N. Saridis, and J. T. Wen, “Approximate Solutions to the Time-Invariant,”  Journal of Optimization theory and application, vol. 96, no. 3, pp. 589–626, 1998.
  * [22] C. Koutschan, “Advanced applications of the holonomic systems approach,” Ph.D. dissertation, Johannes Kepler University Linz, 2009\.
  * [23] T. Iori and T. Ohtsuka, “Nonlinear Bayesian filtering via holonomic gradient method with quasi moment generating function,”  Asian Journal of Control, 2022.


# Angle-optimized Text Embeddings

Xianming Li, Jing Li
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

High-quality text embedding is pivotal in improving semantic textual
similarity (STS) tasks, which are crucial components in Large Language Model
(LLM) applications. However, a common challenge existing text embedding models
face is the problem of vanishing gradients, primarily due to their reliance on
the cosine function in the optimization objective, which has saturation zones.
To address this issue, this paper proposes a novel angle-optimized text
embedding model called AnglE. The core idea of AnglE is to introduce angle
optimization in a complex space. This novel approach effectively mitigates the
adverse effects of the saturation zone in the cosine function, which can
impede gradient and hinder optimization processes. To set up a comprehensive
STS evaluation, we experimented on existing short-text STS datasets and a
newly collected long-text STS dataset from GitHub Issues. Furthermore, we
examine domain-specific STS scenarios with limited labeled data and explore
how AnglE works with LLM-annotated data. Extensive experiments were conducted
on various tasks including short-text STS, long-text STS, and domain-specific
STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA)
STS models that ignore the cosine saturation zone. These findings demonstrate
the ability of AnglE to generate high-quality text embeddings and the
usefulness of angle optimization in STS.

##  1 Introduction

The development of text embeddings (Kiros et al., 2015; Hill et al., 2016;
Conneau et al., 2017; Cer et al., 2018; Reimers & Gurevych, 2019; Gao et al.,
2021) is an essential research challenge in the NLP community. Text embeddings
effectively feature key semantic and syntactic information in language, which
broadly affects the performance of downstream tasks, such as text
classification (Li et al., 2021), sentiment analysis (Suresh & Ong, 2021;
Zhang et al., 2022), semantic matching (Grill et al., 2020; Lu et al., 2020),
clustering (Reimers & Gurevych, 2019; Xu et al., 2023), and question-answering
(QA) system (Yue et al., 2021). In particular, text embedding models play a
crucial role in LLMs such as ChatGPT (OpenAI, 2022; 2023), LLaMA (Touvron et
al., 2023a; b), and ChatGLM (Du et al., 2022)-based applications. These LLM-
based applications heavily rely on high-quality text embeddings for tasks such
as vector search, where related documents are retrieved for LLM QA (Asai et
al., 2023).

Figure 1: The saturation zones of the cosine function. The gradient at
saturation zones is close to zero. During backpropagation, if the gradient is
very small, it could kill the gradient and make the network difficult to
learn.

Recent studies (Gao et al., 2021; Jiang et al., 2022; Chuang et al., 2022;
Chanchani & Huang, 2023; Zhuo et al., 2023) have utilized pre-trained language
models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) in
combination with contrastive learning to enhance the quality of text
embeddings. These approaches involve pulling semantically similar samples
together and pushing apart those not (Gao et al., 2021). In these contrastive
models, positive samples that are semantically similar can be generated by
data augmentation, while negative samples that are dissimilar are selected
from different texts within the same mini-batch (in-batch negatives). However,
supervised negatives are underutilized, and the correctness of in-batch
negatives is difficult to guarantee without annotation, which can lead to
performance degradation. Although some models such as (Gao et al., 2021)
optimize hard negative samples, they rely on strict triple formats ($x_{i}$,
$x_{i}^{+}$, $x_{i}^{-}$). While most existing supervised STS datasets only
provide pairs ($x_{i}$, $x_{i}^{+}$) or ($x_{j}$, $x_{j}^{-}$), where
$x_{i}^{+}$ refers to the positive sample of $x_{i}$ while $x_{j}^{-}$ the
negative sample of $x_{j}$. Thus, most contrastive models are used in
unsupervised settings yet might not benefit from human supervision.

For supervised STS (Reimers & Gurevych, 2019; Su, 2022), most efforts to date
employed the cosine function in their training objective to measure the
pairwise semantic similarity. However, the cosine function has saturation
zones, as shown in Figure 1. It can impede the optimization due to the
gradient vanishing issue and hinder the ability to learn subtle distinctions
between texts in backpropagation. Additionally, many STS datasets such as MRPC
111https://www.microsoft.com/en-us/download/details.aspx?id=52398 and QQP
222https://www.quora.com/q/quoradata/ provide binary labels representing
dissimilar ($0$) and similar ($1$), which naturally fall within the saturation
zone of the cosine function. To overcome this challenge, this paper proposes a
novel angle-optimized text embedding. It optimizes not only the cosine
similarity between texts but also the angle to mitigate the negative impact of
the saturation zones of the cosine function on the learning process.
Specifically, it first divides the text embedding into real and imaginary
parts in a complex space. Then, it follows the division rule in complex space
to compute the angle difference between two text embeddings. After
normalization, the angle difference becomes an objective to be optimized. It
is intuitive to optimize the normalized angle difference, because if the
normalized angle difference between two text embeddings is smaller, it means
that the two text embeddings are closer to each other in the complex space,
i.e., their similarity is larger.

In the STS experimental setup, we observed that the majority of existing STS
benchmarks focus on evaluating models on short texts. Unfortunately, there is
a lack of datasets specifically designed to evaluate the STS performance of
models on long texts. Long texts are prevalent in real-world applications such
as financial documents, legal documents, and health reports (Li et al., 2023).
To tackle this challenge, this paper presents a new high-quality long-text STS
dataset. This dataset allows for a more thorough evaluation of model
performance on long texts. Specifically, the dataset is collected from GitHub
Issues with roughly $21$K samples, we use the duplicate issues as the positive
samples and the non-duplicate issues as the negative samples.

We first experimented with both short and long-text datasets and showed that
AnglE outperforms the SOTA STS models in both transfer and non-transfer STS
tasks. For example, AnglE shows an average Spearman correlation of $73.55\%$
in non-transfer STS tasks, compared to $68.03\%$ for SBERT. Then, an ablation
study shows that all components contribute positively to AnglE’s superior
performance. Next, we discuss the domain-specific scenarios with limited
annotated data that are challenging for AnglE-like supervised STS, where it is
observed that AnglE can work well with LLM-supervised data. Finally, we find
that AnglE can benefit downstream retrieval applications and can learn
representations closer to actual representations.

In summary, the contributions of this paper are listed as follows:

$\bullet$ We investigate the negative effects of saturation zone in the cosine
function widely applied in STS and propose a novel angle-optimized text
embedding model to mitigate this issue.

$\bullet$ We extend the existing STS benchmark with a newly collected long-
text dataset from Github Issues to allow a more comprehensive empirical study
in STS.

$\bullet$ We present extensive experiments on STS and demonstrate that AnglE
can substantially improve the text embedding quality in various scenarios.

##  2 Related Work

This section is organized as follows: we first introduce the unsupervised
approaches, then the supervised approaches, and finally give a summary.

#### Unsupervised Approaches

Early studies (Hill et al., 2016; Pagliardini et al., 2018) have demonstrated
the efficacy of augmenting word2vec (Mikolov et al., 2013) with n-gram
embeddings, yielding strong results in text embeddings. Recently, BERT-flow
(Li et al., 2020) has introduced a flow-based approach that maps BERT
embeddings to a standard Gaussian latent space. On the other hand, BERT-
whitening (Su et al., 2021) applies the whitening operation to BERT embeddings
to enhance text embeddings. Furthermore, very recent research (Carlsson et
al., 2020; Zhang et al., 2020; Giorgi et al., 2021; Gao et al., 2021; Yan et
al., 2021; Chuang et al., 2022; Jiang et al., 2022; Zhuo et al., 2023) has
focused on leveraging contrastive objectives to improve the quality of text
embeddings.

#### Supervised Approaches

Supervised text embeddings usually perform better than their unsupervised
counterparts (Gao et al., 2021). Various studies have effectively utilized
supervised datasets to enhance the learning of text embeddings. In particular,
Conneau et al. (2017) introduced a method that leverages supervised Natural
Language Inference (NLI) tasks for this purpose. Building on a transformer
backbone, USE (Cer et al., 2018) incorporates the SNLI dataset to augment
unsupervised training, resulting in improved performance. Furthermore, SBERT
(Reimers & Gurevych, 2019) enhances text embedding by combining BERT with a
siamese architecture.

However, most existing models optimize the cosine similarity but neglect the
negative effect of the saturation zone of the cosine function. To address this
issue, this paper proposes a novel angle-optimized text embedding model to
improve the quality of text embedding.

##  3 Methodology

This section will introduce the components of the proposed angle-optimized
text embedding model, including the input layer, cosine objective, in-batch
negative objective, and angle objective.

###  3.1 Input Layer

For the input sentences, we first apply padding to ensure a consistent length
$l$. Next, we map each word to a continuous $d$-dimensional space to produce
word embeddings $\mathbf{e}_{i}\in\mathbb{R}^{d}$. These word embeddings are
then concatenated to form the model input:
$\mathbf{E}=[\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e}_{l}]\in\mathbb{R}^{l\times
d}$. Subsequently, the model input is passed through an encoder such as BERT
(Devlin et al., 2019), RoBERTa (Liu et al., 2019), and LLaMA (Touvron et al.,
2023a; b) to obtain the contextual representation $\mathbf{X}$.

###  3.2 Cosine Objective

Following the prior study (Su, 2022), we employ the cosine objective function
for end-to-end optimization of cosine similarity between representations, as
follows:

$\mathcal{L}_{cos}=\mathrm{log}\left[1+\sum_{s(\mathbf{X}_{i},\mathbf{X}_{j})>s(\mathbf{X}_{m},\mathbf{X}_{n})}e^{\frac{\mathrm{cos}(\mathbf{X}_{m},\mathbf{X}_{n})-\mathrm{cos}(\mathbf{X}_{i},\mathbf{X}_{j})}{\tau}}\right]$
(1)

where $\tau$ is a temperature hyperparameter, $\mathrm{cos}(\cdot)$ is the
cosine similarity function, and $s(u,v)$ is the similarity between $u$ and
$v$. By optimizing the $\mathcal{L}_{cos}$, we expect the cosine similarity of
the high similarity pair to be greater than that of the low similarity pair.

###  3.3 In-batch Negative Objective

To further improve performance, we integrate the in-batch negative objective
function. Because in-batch negative samples can serve as a data augmentation
technique, which can strengthen the ability to generalize. Unlike existing
contrastive learning methods (Gao et al., 2021; Yan et al., 2021; Jiang et
al., 2022) that generate positive samples through data augmentation, we use
supervised positive samples. Recognizing that there might be identical
sentences within a batch that are not explicitly labeled as positive samples,
causing them to become in-batch negatives, we identify these duplicate
sentences and assign them as positive samples, thereby reducing potential
noise. The formulation for the in-batch negative objective function (ibn) is
as follows:

$\mathcal{L}_{ibn}=-\sum_{b}\sum_{i}^{m}\mathrm{log}\left[\frac{e^{\mathrm{cos}(\mathbf{X}_{b_{i}},\mathbf{X}_{b_{i}}^{+})/\tau}}{\sum_{j}^{N}e^{\mathrm{cos}(\mathbf{X}_{b_{i}},\mathbf{X}_{b_{j}}^{+})/\tau}}\right],$
(2)

where $\tau$ is a temperature hyperparameter, $b$ stands for the $b$-th batch,
$\mathbf{X}_{b_{i}}^{+}$ and $\mathbf{X}_{b_{j}}^{+}$ are the respective
positive samples of $\mathbf{X}_{b_{i}}$ and $\mathbf{X}_{b_{j}}$, $m$
represents the number of positive pairs in $b$-th batch, $N$ is the batch
size, and $\mathrm{cos}(\cdot)$ is the cosine similarity function.

(a)

(b)

Figure 2: (a) Division in complex space. $\Delta\theta$ is the angle
difference between dividend $z$ and divisor $w$ in complex space. (b) Angle
optimization in cosine saturation zones. Even though $\Delta y\approx 0$ could
kill the gradient, the corresponding angle difference in complex space is
still distinct for optimization.

###  3.4 Angle Objective

We found that both the cosine and in-batch negative objectives employ the
cosine function to measure similarity. However, it is important to note that
the cosine function includes saturation zones, which can hinder the
optimization process. We optimize the angle difference in complex space to
mitigate these adverse effects. Figure 2(a) draws the division in complex
space and Figure 2(b) depicts how angle optimization works in cosine
saturation zones. To optimize the angle difference, we initially decompose the
representation $\mathbf{X}$ into its real part $\mathbf{X}^{re}$ and imaginary
part $\mathbf{X}^{im}$. For the pair $(\mathbf{X}_{i},\mathbf{X}_{j})$, their
representations in the complex space are defined as follows:

$\begin{split}\mathbf{z}&=\mathbf{a}+\mathbf{b}i\in\mathbb{C}\\\
\mathbf{w}&=\mathbf{c}+\mathbf{d}i\in\mathbb{C},\end{split}$ (3)

where $\mathbf{a}=\mathbf{X}^{re}_{i}\in\mathbb{R}$,
$\mathbf{b}=\mathbf{X}^{im}_{i}\in\mathbb{R}$,
$\mathbf{c}=\mathbf{X}^{re}_{j}\in\mathbb{R}$, and
$\mathbf{d}=\mathbf{X}^{im}_{j}\in\mathbb{R}$. To compute the angle difference
between $\mathbf{z}$ and $\mathbf{w}$, we calculate division in complex space
in polar coordinates, as follows:

$\begin{split}\frac{\mathbf{z}}{\mathbf{w}}&=\gamma\Delta\theta_{zw}\\\
\gamma&=\frac{r_{\mathbf{z}}}{r_{\mathbf{w}}}=\frac{\sqrt{\mathbf{a}^{2}+\mathbf{b}^{2}}}{\sqrt{\mathbf{c}^{2}+\mathbf{d}^{2}}}\\\
\Delta\theta_{zw}&=\theta_{\mathbf{z}}-\theta_{\mathbf{w}},\end{split}$ (4)

where $r_{\mathbf{z}}$ and $r_{\mathbf{w}}$ represent the magnitudes of
$\mathbf{z}$ and $\mathbf{w}$, while $\theta_{\mathbf{z}}$ and
$\theta_{\mathbf{w}}$ denote the respective angles of $\mathbf{z}$ and
$\mathbf{w}$. For a detailed derivation of this complex space division, please
refer to Appendix A.1. Next, we compute the value of
$\frac{\mathbf{z}}{\mathbf{w}}$ by the division rule in complex space (see
A.2), as follows:

$\frac{\mathbf{z}}{\mathbf{w}}=\frac{\mathbf{a}+\mathbf{b}i}{\mathbf{c}+\mathbf{d}i}=\frac{(\mathbf{a}\mathbf{c}+\mathbf{b}\mathbf{d})+(\mathbf{b}\mathbf{c}-\mathbf{a}\mathbf{d})i}{\mathbf{c}^{2}+\mathbf{d}^{2}}.$
(5)

By employing Eq. 4 and Eq. 5, we can calculate the angle difference between
$\mathbf{z}$ and $\mathbf{w}$ by multiplying both sides by $\frac{1}{\gamma}$,
which can be seen as a normalization operation. In this paper, we determine
the absolute normalized angle difference using the following expression:

$\begin{split}\Delta\theta_{zw}&=\mathrm{abs}(\frac{\mathbf{z}}{\mathbf{w}}\times\frac{1}{\gamma})\\\
&=\mathrm{abs}\left[\frac{(\mathbf{a}\mathbf{c}+\mathbf{b}\mathbf{d})+(\mathbf{b}\mathbf{c}-\mathbf{a}\mathbf{d})i}{\mathbf{c}^{2}+\mathbf{d}^{2}}\times\frac{\sqrt{\mathbf{c}^{2}+\mathbf{d}^{2}}}{\sqrt{\mathbf{a}^{2}+\mathbf{b}^{2}}}\right]\\\
&=\mathrm{abs}\left[\frac{(\mathbf{a}\mathbf{c}+\mathbf{b}\mathbf{d})+(\mathbf{b}\mathbf{c}-\mathbf{a}\mathbf{d})i}{\sqrt{(\mathbf{c}^{2}+\mathbf{d}^{2})(\mathbf{a}^{2}+\mathbf{b}^{2})}}\right].\\\
\end{split}$ (6)

Then, the angle difference can be optimized by the following objective
function:

$\mathcal{L}_{angle}=\mathrm{log}\left[1+\sum_{s(\mathbf{X}_{i},\mathbf{X}_{j})>s(\mathbf{X}_{m},\mathbf{X}_{n})}e^{\frac{\Delta\theta_{ij}-\Delta\theta_{mn}}{\tau}}\right]$
(7)

where $\tau$ is a temperature hyperparameter and $s(u,v)$ is the similarity
between $u$ and $v$. By optimizing the $\mathcal{L}_{angle}$, our objective is
to minimize the normalized angle difference for pairs with high similarity
compared to those with low similarity.

Finally, we combine the aforementioned three objective functions in the
following manner to form the final objective function:

$\mathcal{L}=w_{1}*\mathcal{L}_{cos}+w_{2}*\mathcal{L}_{ibn}+w_{3}*\mathcal{L}_{angle},$
(8)

where $w_{1}$, $w_{2}$, and $w_{3}$ are constants.

##  4 Experiment

###  4.1 Datasets and Evaluation Metrics

#### Existing STS Benchmarks

We mainly evaluate our model on several widely-adopted STS datasets, namely:
MRPC, QQP, QNLI 333https://gluebenchmark.com/, STS 2012-2016 (Agirre et al.,
2012; 2013; 2014; 2015; 2016), SICK-R (Marelli et al., 2014), and STS-B (Cer
et al., 2017). These datasets mainly consist of short text, but real-world
scenarios often involve long text documents. Thus, we introduce a newly long-
text dataset called GitHub Issues Similarity Dataset to comprehensively
evaluate the STS task.

#### GitHub Issues Similarity Dataset

We observed the presence of many duplicate issues on GitHub. Typically, the
maintainers of open source organizations tend to mark these duplicate issues
as closed with a comment like “closing as a duplicate of #id”. Consequently,
these duplicate issues inherently serve as a source of the STS task. It is
also worth noting that most issues contain long texts because of the inclusion
of extensive code within the issues. To compile the dataset, we extracted
duplicated issues from $55$ popular open-source projects (see A.3) on GitHub
using GitHub API 444https://docs.github.com/en/rest. The duplicated issues
were used as positive samples, while the remaining issues were considered
negative samples. Table 4 presents statistics of the GitHub Issues Similarity
Dataset, while Figure 4 shows a violin plot illustrating the token-level text
length distribution. The visualization reveals a substantial number of long
texts. Specifically, the proportion of long texts (token length $>512$) for
the train, validation, and test sets is at $61.03\%$, $60.85\%$, and
$60.50\%$, respectively.

#### Evaluation Metrics

To ensure a fair comparison, we follow previous studies and use Spearman’s
correlation for evaluation. We use SentEval (Conneau & Kiela, 2018) to compute
Spearman’s correlation and report the results in the “all” setting, which is
consistent with the baselines.

[]  []  Split Train Validation Test #Positive $9457$ $774$ $807$ #Negative
$9108$ $773$ $741$ Total $18565$ $1547$ $1548$

Figure 3: Log token length distribution of the GitHub Issue Similarity
Dataset. Figure 4: Statistics of the proposed GitHub Issues Similarity
Dataset. #Positive denotes the count of positive pairs, and #Negative
represents the number of negative pairs.

###  4.2 Implementation Details

In this paper, we use the pre-trained uncased BERT base model (110M
parameters) as the backbone model. For a fair comparison, all BERT-based
baselines also adopt this setting. We set the value of $\tau$ for the cosine
objective and the in-batch negative objective to $0.05$, based on prior
research. Additionally, we determined the value of $\tau$ for the angle
objective to be $1.0$ through grid search. For detailed hyperparameter
settings, see A.4.

###  4.3 Main Results

In this section, we will first introduce the baselines, then the results of
the transfer STS tasks, then the results of the non-transfer STS tasks, and
finally a summary.

#### Baselines

We compare our proposed model with widely used baselines, encompassing both
unsupervised and supervised models. The unsupervised models are average GloVe
(Pennington et al., 2014), BERT-flow (Li et al., 2020), BERT-whitening (Su et
al., 2021), LLaMA2 with avg/max/last pooling (Touvron et al., 2023b), and
contrastive learning models including IS-BERT (Zhang et al., 2020), CT-BERT
(Carlsson et al., 2020), SimCSE (Gao et al., 2021), ConSERT (Yan et al.,
2021), and DiffCSE (Chuang et al., 2022). On the other hand, the chosen
supervised models are InferSent (Conneau et al., 2017), USE (Cer et al.,
2018), SBERT (Reimers & Gurevych, 2019), CoSENT (Su, 2022), SLLaMA(Sentence-
LLaMA), as well as supervised versions of SimCSE and ConSERT.

#### Transfer STS Tasks

For a fair comparison, we train AnglE with the NLI datasets MNLI (Williams et
al., 2018) and SNLI (Bowman et al., 2015) and then transfer it to evaluate
seven STS benchmark datasets. The evaluation results are presented in Table 1.
It is evident that AnglE consistently outperforms the baselines with a gain of
$0.80\%$ in average score compared to the previous SOTA SimCSE. Note that
supervised SBERT and CoSENT show lower results than other unsupervised
contrastive learning models like SimCSE and DiffCSE. This difference might
arise from the difference in data distributions between the training and test
data in the transfer STS tasks. They struggle to effectively generalize to STS
tasks when trained solely with NLI datasets. In contrast, contrastive learning
models exhibit better generalization capabilities due to their alignment and
uniformity features. Because AnglE optimizes both the supervised cosine
objective and the in-batch negative objective. This can allow AnglE to
generalize well in transfer STS tasks. Additionally, the angle optimization in
AnglE mitigates the negative impact of the saturation zone in the cosine
function to produce better performance than other baselines. We observe that
unsupervised LLaMA models show poor performance on STS tasks. This is
primarily because such auto-regressive models are inherently single-
directional, making it challenging to capture semantic dependencies within
tokens. Nevertheless, supervised learning can enhance the performance of LLaMA
on STS tasks. Specifically, SLLaMA, SimCSE-LLaMA, and AnglE-LLaMA achieve
better results than unsupervised LLaMA. Notably, AnglE-LLaMA outperforms
SLLaMA and SimCSE-LLaMA. AnglE-LLaMA demonstrates an impressive gain of
$6.66\%$ compared to SimCSE-LLaMA. This result underscores the effectiveness
of AnglE in improving the text embeddings of LLaMA models.

Model | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICR-R | Avg.
---|---|---|---|---|---|---|---|---
Unsupervised Models
GloVe (avg.) $\dagger$ | $55.14$ | $70.66$ | $59.73$ | $68.25$ | $63.66$ | $58.02$ | $53.76$ | $61.32$
BERT-flow $\ddagger$ | $58.40$ | $67.10$ | $60.85$ | $75.16$ | $71.22$ | $68.66$ | $64.47$ | $66.55$
BERT-whitening $\ddagger$ | $57.83$ | $66.90$ | $60.90$ | $75.08$ | $71.31$ | $68.24$ | $63.73$ | $66.28$
LLaMA-27B-avg $\star$ | $37.35$ | $50.20$ | $38.68$ | $53.20$ | $53.43$ | $48.34$ | $49.45$ | $47.24$
LLaMA-27B-max $\star$ | $35.10$ | $53.00$ | $42.73$ | $49.09$ | $52.62$ | $50.64$ | $49.23$ | $47.49$
LLaMA-27B-last $\star$ | $17.93$ | $20.17$ | $17.26$ | $30.78$ | $40.67$ | $23.72$ | $48.12$ | $28.38$
IS-BERT $\ddagger$ | $56.77$ | $69.24$ | $61.21$ | $75.23$ | $70.16$ | $69.21$ | $64.25$ | $66.58$
CT-BERT $\ddagger$ | $61.63$ | $76.80$ | $68.47$ | $77.50$ | $76.48$ | $74.31$ | $69.19$ | $72.05$
ConSERT-BERT | $64.64$ | $78.49$ | $69.07$ | $79.72$ | $75.95$ | $73.97$ | $67.31$ | $72.74$
DiffCSE-BERT | $72.28$ | $84.43$ | $76.47$ | $83.90$ | $80.54$ | $80.59$ | $71.23$ | $78.49$
SimCSE-BERT | $68.40$ | $82.41$ | $74.38$ | $80.91$ | $78.56$ | $76.85$ | $72.23$ | $76.25$
Supervised Models
InferSent-GloVe $\dagger$ | $52.86$ | $66.75$ | $62.15$ | $72.77$ | $66.87$ | $68.03$ | $65.65$ | $65.01$
USE $\dagger$ | $64.49$ | $67.80$ | $64.61$ | $76.83$ | $73.18$ | $74.92$ | $76.69$ | $71.22$
ConSERT-BERT | $74.07$ | $83.93$ | $77.05$ | $83.66$ | $78.76$ | $81.36$ | $76.77$ | $79.37$
CoSENT-BERT $\star$ | $71.35$ | $77.52$ | $75.05$ | $79.68$ | $76.05$ | $78.99$ | $71.19$ | $75.69$
SBERT $\dagger$ | $70.97$ | $76.53$ | $73.19$ | $79.09$ | $74.30$ | $77.03$ | $72.91$ | $74.89$
SimCSE-BERT | $75.30$ | $84.67$ | $80.19$ | $85.40$ | $80.82$ | $84.25$ | $80.39$ | $81.57$
SLLaMA-27B $\star$ | $63.69$ | $67.40$ | $69.33$ | $71.78$ | $74.60$ | $77.18$ | $66.62$ | $70.08$
SimCSE-LLaMA-27B $\star$ | $65.61$ | $70.98$ | $70.62$ | $74.25$ | $74.72$ | $76.10$ | $67.27$ | $71.36$
AnglE-LLaMA-27B | $\mathbf{75.64}$ | $83.36$ | $78.51$ | $79.85$ | $76.52$ | $81.56$ | $70.71$ | $78.02$
AnglE-BERT | $75.09$ | $\mathbf{85.56}$ | $\mathbf{80.66}$ | $\mathbf{86.44}$ | $\mathbf{82.47}$ | $\mathbf{85.16}$ | $\mathbf{81.23}$ | $\mathbf{82.37}$
Table 1: Text embedding performance on STS tasks. We report the Spearman’s
correlation $\rho\times 100$ of the “all” setting computed by SentEval.
Supervised LLaMA-based models are fine-tuned using the LoRA (Hu et al., 2021)
technique. Results marked with $\dagger$ are obtained from (Reimers &
Gurevych, 2019), while results marked with $\ddagger$ are retrieved from (Gao
et al., 2021). Additionally, results marked with $\star$ denote our own
implementation using official code. For the remaining baselines, we refer to
the corresponding original papers to obtain their results. The T-test
indicates that the improvements from AnglE to SimCSE are significant with a
p-value of $3.71\%$.

#### Non-transfer STS Tasks

To provide a comprehensive analysis, we also evaluate the performance of the
baselines in the non-transfer setting. We train the baselines on the train set
and evaluate them on the test or validation set. Two typical models, SimCSE
and SBERT, representing contrastive and supervised learning, are compared with
our model. The results of the non-transfer STS tasks are listed in Table 2,
where we evaluate the baselines on four short-text datasets (MRPC, STS-B, QQP,
and QNLI) and one long-text dataset (GitHub Issues Similarity Dataset). SimCSE
notably performs poorly compared to SBERT and AnglE in the non-transfer
setting. This is due to the limitation of the small-scale training set, as
there are not enough samples for SimCSE to effectively learn representations.
Furthermore, the datasets only provide pair-supervised data, namely
$(x,x^{+})$ or $(x,x^{-})$, which prevents SimCSE from utilizing its hard
negative objective that relies on triple-supervised data $(x,x^{+},x^{-})$.
This limitation might affect its performance. On the other hand, AnglE
consistently outperforms SBERT, achieving an absolute gain of $5.52\%$. This
can support the idea that angle-optimized text embedding can mitigate the
negative impact of the cosine function, resulting in better performance.
Furthermore, we explore applying the long text model RANbase (86M parameters)
(Li et al., 2023) as the backbone to test the performance on long text. The
results show that AnglE-BERT outperforms AnglE-RAN across all short text
datasets. This advantage might be attributed to the larger parameter size of
BERT and its proficiency in handling short texts. However, we observe a
remarkable shift in long-text STS. AnglE-RAN outperforms AnglE-BERT in this
scenario, suggesting that AnglE-RAN can handle long texts well despite having
fewer parameters.

Model | MRPC | STS-B | QQP | QNLI | GitHub Issues. | Avg.
---|---|---|---|---|---|---
test | test | validation | validation | test
SimCSE-BERT | $48.13$ | $76.27$ | $65.84$ | $33.00$ | $60.38$ | $56.72$
SBERT | $46.19$ | $84.67$ | $73.80$ | $65.98$ | $69.50$ | $68.03$
AnglE-RAN | $58.70$ | $80.23$ | $74.87$ | $63.04$ | $\mathbf{71.25}$ | $69.62$
AnglE-BERT | $\mathbf{62.20}$ | $\mathbf{86.26}$ | $\mathbf{76.54}$ | $\mathbf{72.19}$ | $70.55$ | $\mathbf{73.55}$
Table 2: Results on the STS tasks. All baseline results are our implementation
using the official code. Spearman’s correlation ($\rho\times 100$) serves as
the reported metric.

In short, this evidence suggests AnglE’s superiority in transfer and non-
transfer settings, its ability to produce high-quality text embeddings, and
its robustness and adaptability to different backbones.

###  4.4 Ablation Study

To gain a deeper understanding of AnglE, we conducted an ablation study
examining different objectives and their effects. The results in table 3
indicate that AnglE shows improved performance with all three objectives. In
particular, we observe that AnglE experiences a greater drop in performance
without the angle objective than without the in-batch negative (ibn)
objective. This suggests that angle optimization is more important than ibn in
improving text embedding. Additionally, we find that using the angle objective
alone yields performance close to that of using the cosine objective alone,
demonstrating the effectiveness of angle optimization. We also evaluated five
different pooling strategies and found that the “cls” strategy performed the
best. Finally, we compared the ibn with/without identical sentence pair (ISP)
detection and found that ibn without ISP detection has about $0.18\%$
performance drop than with. This indicates that ibn with ISP detection is
effective.

Model | Spearman’s Correlation
---|---
Objective
AnglE-BERT-all | $\mathbf{86.26}$
\- w/o ibn | $86.00$
\- w/o angle | $85.30$
only cosine | $85.28$
only ibn | $72.48$
only angle | $85.15$
Pooling Strategy
cls | $\mathbf{86.26}$
cls-last-avg | $85.81$
last-avg | $84.15$
last-max | $79.76$
first-last-avg | $81.99$

Table 3: Ablation study of AnglE. The results are Spearman’s correlations on
the STS-B test set.

Table 4: Results of unsupervised and LLM supervised models on the STS-B test
set. For ChatGPT, LLaMA, and ChatGLM, we use the gpt-turbo-3.5, 7B LLaMA2, and
6B ChatGLM, respectively.

Model | Spearman’s
---|---
Unsupervised Models
SimCSE-BERT | $76.85$
ConSERT-BERT | $73.97$
DiffCSE-BERT | $80.59$
LLM-supervised Models
AnglE-BERT + ChatGPT | $81.52$
AnglE-BERT + LLaMA | $79.29$
AnglE-BERT + ChatGLM | $81.11$
AnglE-BERT + Ensemble | $\mathbf{82.01}$

Figure 5: The procedures of the LLM-supervised learning. For the STS task, we
use the prompt “You are a highly smart same-meaning/opposite-meaning sentence-
generating system, your job is to generate {size} synonymous/antonym sentences
of a given input sentence. Input sentence: {text}. Output:” to generate
positive/negative pairs. {size} and {text} are placeholders for the generated
size and the input text, respectively.

###  4.5 Discussion and Analysis

#### Discussion of LLM-supervised Learning

AnglE, a supervised learning model, must be trained on labeled data. However,
the limited availability of domain-supervised data poses a challenge in real-
world applications. To overcome this problem, we propose LLM-supervised
learning. This approach applies LLMs as data annotators to label the pseudo-
supervised data for AnglE training. Figure 5 outlines the procedures involved
in LLM-supervised learning. In this study, we compare the LLM-supervised AnglE
and unsupervised contrastive learning models. To simulate domain application,
we extract all “sentence1” texts from the STS-B train set and employ LLM-
supervised learning to train the AnglE model. Table 4.4 shows the results on
the STS-B test set. It is evident from the results that LLM-supervised AnglE
performs better than unsupervised contrastive baselines, and the ensemble of
LLMs shows the best results. This evidence suggests the effectiveness of LLM-
supervised learning and indicates that it can alleviate the domain-supervised
data scarcity problem.

#### Discussion of Text Retrieval

We also evaluate the performance of the text retrieval task by experimenting
on the test split of the flickr30k dataset (Young et al., 2014). This dataset
consists of five caption texts for each photo, and these texts are similar to
each other. We use the first caption text vector to retrieve the top 5 similar
sentences using faiss 555https://github.com/facebookresearch/faiss. The strict
accuracy 666Strict accuracy means only all the top five sentences retrieved
are equal to the five reference sentences considered correct of AnglE, SimCSE
(supervised), and SBERT are $12.9\%$, $10.4\%$, and $5.2\%$, respectively.
This evidence indicates the effectiveness of using AnglE for the retrieval
task.

#### Analysis of Text Embedding Distribution

Figure 6(a) depicts the density plots of the cosine similarities between
sentence pairs in the STS-B test set to provide an intuitive visualization of
the text embedding quality. Figure 6(b) displays the golden scores for the
same sentence pairs. Analyzing the overall density of cosine similarities, we
find that AnglE’s distribution resembles the golden distribution more closely
than SimCSE (supervised) and SBERT’s. Figure 6(b) illustrates a peak in the
0-1 range; however, only AnglE shows a distinct peak in this range in Figure
6(a). Also, Figure 6(b) portrays a higher peak around $4$ than around $4.8$ in
the 4-5 range, only AnglE demonstrates this feature properly in Figure 6(a).
Notably, the 0-1 and 4-5 ranges in Figure 6(b) represent two saturation zones
of the cosine function. This evidence suggests that AnglE can mitigate the
negative effect of the saturation zone. In conclusion, we can confidently
assert that AnglE produces better text embeddings with a cosine similarity
density closely resembling the actual distribution than the baselines.

(a)

(b)

Figure 6: (a) Density plots of cosine similarities between sentence pairs in
the STS-B test set. The pairs have been categorized into 6 groups, reflecting
the ground truth ratings (where higher ratings indicate a higher degree of
similarity), visually represented on the y-axis. The x-axis represents the
cosine similarity. (b) Density plots of golden scores between sentence pairs
in the STS-B test set.

##  5 Conclusion and Future Work

In this paper, we have presented a novel text embedding model called AnglE,
which optimizes the angle difference in complex space to overcome the adverse
impact of the saturation zone of the cosine function, thereby improving text
embeddings. To comprehensively evaluate the STS tasks, we have introduced the
GitHub Issues Similarity Dataset to evaluate model performance on the long-
text STS task. Furthermore, we have proposed an LLM-supervised learning method
to cope with the scarcity of domain-supervised data. Extensive experimental
results have demonstrated that AnglE outperforms baselines, indicating that
AnglE can handle both short and long-text STS tasks and work effectively in
various scenarios. In future work, we plan to explore the application of AnglE
in real-world scenarios and provide further insights into AnglE.

## References

  * Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre.  SemEval-2012 task 6: A pilot on semantic textual similarity.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pp. 385–393. Association for Computational Linguistics, 2012.
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo.  *SEM 2013 shared task: Semantic textual similarity.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , pp. 32–43, Atlanta, Georgia, USA, 2013. Association for Computational Linguistics.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe.  SemEval-2014 task 10: Multilingual semantic textual similarity.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pp. 81–91. Association for Computational Linguistics, 2014.
  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe.  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pp. 252–263. Association for Computational Linguistics, 2015.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pp. 497–511. Association for Computational Linguistics, 2016.
  * Asai et al. (2023) Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen.  Retrieval-based language models and applications.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts)_ , pp. 41–46. Association for Computational Linguistics, July 2023.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 632–642. Association for Computational Linguistics, 2015.
  * Carlsson et al. (2020) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren.  Semantic re-tuning with contrastive tension.  In _International conference on learning representations_ , 2020.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pp. 1–14. Association for Computational Linguistics, August 2017.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 169–174. Association for Computational Linguistics, 2018.
  * Chanchani & Huang (2023) Sachin Chanchani and Ruihong Huang.  Composition-contrastive learning for sentence embeddings.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , pp. 15836–15848. Association for Computational Linguistics, 2023.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass.  DiffCSE: Difference-based contrastive learning for sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4207–4218. Association for Computational Linguistics, 2022.
  * Conneau & Kiela (2018) Alexis Conneau and Douwe Kiela.  SentEval: An evaluation toolkit for universal sentence representations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), 2018.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 670–680. Association for Computational Linguistics, 2017.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4171–4186, 2019.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pp. 320–335, 2022.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 6894–6910. Association for Computational Linguistics, 2021.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader.  DeCLUTR: Deep contrastive learning for unsupervised textual representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 879–895. Association for Computational Linguistics, 2021.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen.  Learning distributed representations of sentences from unlabelled data.  In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 1367–1377. The Association for Computational Linguistics, 2016.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Jiang et al. (2022) Yuxin Jiang, Linhan Zhang, and Wei Wang.  Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pp. 3021–3035. Association for Computational Linguistics, 2022.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Skip-thought vectors.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015_ , pp. 3294–3302, 2015.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li.  On the sentence embeddings from pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 9119–9130. Association for Computational Linguistics, 2020.
  * Li et al. (2021) Xianming Li, Zongxi Li, Haoran Xie, and Qing Li.  Merging statistical feature via adaptive gate for improved text classification.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 13288–13296, 2021.
  * Li et al. (2023) Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, and Qing Li.  Recurrent attention networks for long-text modeling.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 3006–3019. Association for Computational Linguistics, 2023.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019.  URL http://arxiv.org/abs/1907.11692.
  * Lu et al. (2020) Wenpeng Lu, Xu Zhang, Huimin Lu, and Fangfang Li.  Deep hierarchical encoding model for sentence semantic matching.  _Journal of Visual Communication and Image Representation_ , 71:102794, 2020.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pp. 216–223. European Language Resources Association (ELRA), 2014.
  * Mikolov et al. (2013) Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality.  In _27th Annual Conference on Neural Information Processing Systems 2013._ , pp. 3111–3119, 2013.
  * OpenAI (2022) OpenAI.  Introducing chatgpt, 2022.  URL https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.
  * Pagliardini et al. (2018) Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi.  Unsupervised learning of sentence embeddings using compositional n-gram features.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 528–540. Association for Computational Linguistics, 2018.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1532–1543. Association for Computational Linguistics, 2014.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_ , pp. 3980–3990. Association for Computational Linguistics, 2019.
  * Su (2022) Jianlin Su.  Cosent (1): A more effective sentence vector scheme than sentence bert, Jan 2022.  URL https://kexue.fm/archives/8847.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou.  Whitening sentence representations for better semantics and faster retrieval.  _arXiv preprint arXiv:2103.15316_ , 2021.
  * Suresh & Ong (2021) Varsha Suresh and Desmond Ong.  Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 4381–4394. Association for Computational Linguistics, 2021.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 1112–1122, New Orleans, Louisiana, 2018. Association for Computational Linguistics.
  * Xu et al. (2023) Lingling Xu, Haoran Xie, Zongxi Li, Fu Lee Wang, Weiming Wang, and Qing Li.  Contrastive learning models for sentence representations.  _ACM Trans. Intell. Syst. Technol._ , 14(4), jun 2023.  ISSN 2157-6904.  doi: 10.1145/3593590.  URL https://doi.org/10.1145/3593590.
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu.  Consert: A contrastive framework for self-supervised sentence representation transfer.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , pp. 5065–5075. Association for Computational Linguistics, 2021.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.
  * Yue et al. (2021) Zhenrui Yue, Bernhard Kratzwald, and Stefan Feuerriegel.  Contrastive domain adaptation for question answering using limited text corpora.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 9575–9593. Association for Computational Linguistics, November 2021.
  * Zhang et al. (2022) Han Zhang, Zongxi Li, Haoran Xie, Raymond YK Lau, Gary Cheng, Qing Li, and Dian Zhang.  Leveraging statistical information in fine-grained financial sentiment analysis.  _World Wide Web_ , 25(2):513–531, 2022.
  * Zhang et al. (2020) Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing.  An unsupervised sentence embedding method by mutual information maximization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1601–1610. Association for Computational Linguistics, 2020.
  * Zhuo et al. (2023) Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, and Yi Yang.  WhitenedCSE: Whitening-based contrastive learning of sentence embeddings.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , pp. 12135–12148. Association for Computational Linguistics, 2023.

##  Appendix A Appendix

###  A.1 Complex Space Division in Polar Coordinates

Consider two complex numbers, $\mathbf{z}\in\mathbb{C}$ and
$\mathbf{w}\in\mathbb{C}$, expressed in polar coordinates as follows:

$\begin{split}\mathbf{z}&=\mathbf{r}_{z}\angle\theta_{z}=\mathbf{r}e^{i\theta_{z}}\\\
\mathbf{w}&=\mathbf{r}_{w}\angle\theta_{w}=\mathbf{r}e^{i\theta_{w}}\end{split}$
(9)

The division of $\mathbf{z}$ and $\mathbf{w}$ is as follows:

$\begin{split}\frac{\mathbf{z}}{\mathbf{w}}&=\left(\frac{\mathbf{r}_{z}}{\mathbf{r}_{w}}\right)e^{i\theta_{z}-i\theta_{w}}\\\
&=\left(\frac{\mathbf{r}_{z}}{\mathbf{r}_{w}}\right)\angle(\theta_{z}-\theta_{w})\end{split}$
(10)

In this manner, we have established the division of these complex numbers
$\mathbf{z}$ and $\mathbf{w}$. The outcome is another complex number whose
magnitude corresponds to the ratio of the magnitudes of $\mathbf{z}$ and
$\mathbf{w}$, while its angle represents the difference between the angles of
$\mathbf{z}$ and $\mathbf{w}$.

###  A.2 Division Rule in Complex Space

The division of two complex numbers
$\mathbf{z}=\mathbf{a}+\mathbf{b}i\in\mathbb{C}$ and
$\mathbf{w}=\mathbf{c}+\mathbf{d}i\in\mathbb{C}$ is as follows:

$\begin{split}\frac{\mathbf{z}}{\mathbf{w}}&=\frac{\mathbf{a}+\mathbf{b}i}{\mathbf{c}+\mathbf{d}i}\\\
&=\frac{(\mathbf{a}+\mathbf{b}i)(\mathbf{c}-\mathbf{d}i)}{(\mathbf{c}+\mathbf{d}i)(\mathbf{c}-\mathbf{d}i)}\\\
&=\frac{(\mathbf{a}\mathbf{c}+\mathbf{b}\mathbf{d})+(\mathbf{b}\mathbf{c}-\mathbf{a}\mathbf{d})i}{\mathbf{c}^{2}+\mathbf{d}^{2}},\end{split}$
(11)

where $\mathbf{a}$, $\mathbf{b}$, $\mathbf{c}$, $\mathbf{d}\in\mathbb{R}$.

###  A.3 List of Open-source Projects in GitHub Issue Similarity Dataset

We collected GitHub issues from the following $55$ GitHub repositories.

tensorflow/tensorflow | pytorch/pytorch | huggingface/transformers
---|---|---
keras-team/keras | freeCodeCamp/freeCodeCamp | vuejs/vue
facebook/react | angular/angular | elastic/elasticsearch
numpy/numpy | scikit-learn/scikit-learn | pandas-dev/pandas
psf/requests | scipy/scipy | matplotlib/matplotlib
scrapy/scrapy | opencv/opencv | bumptech/glide
spring-projects/spring-framework | apache/dubbo | apache/superset
apache/airflow | apache/druid | apache/shardingsphere
kubernetes/kubernetes | flutter/flutter | google/jax
twbs/bootstrap | axios/axios | microsoft/vscode
sqlalchemy/sqlalchemy | mwaskom/seaborn | microsoft/TypeScript
microsoft/PowerToys | microsoft/terminal | microsoft/playwright
symfony/symfony | babel/babel | electron/electron
denoland/deno | mrdoob/three.js | mui/material-ui
webpack/webpack | atom/atom | vercel/next.js
ansible/ansible | npm/cli | pallets/flask
tiangolo/fastapi | DefinitelyTyped/DefinitelyTyped | celery/celery
neo4j/neo4j | rust-lang/rust | JuliaLang/julia
golang/go |  |

###  A.4 Hyper-parameters

We determined the values of $w_{1}$, $w_{2}$, and $w_{3}$ by grid searching
for each dataset. For the transfer setting in the main experiment, we set the
max length to $128$ and tested different batch sizes, including $8$, $16$,
$32$, and $50$. Our results indicate that larger batch sizes yield better
results. We used the NVIDIA 3090 Ti GPU to run our experiments. However, due
to the limited GPU memory, we could not exceed a maximum batch size of $50$ in
the main experiment. Therefore, we believe there is still room for improvement
in the performance of AnglE. In future research, we intend to explore the
effects of larger batch sizes using more advanced GPUs, such as the NVIDIA
A100.