# A note on the zeros of finite order Bank–Laine functions

Yueyang Zhang  School of Mathematics and Physics, University of Science and
Technology Beijing, No. 30 Xueyuan Road, Haidian, Beijing, 100083, P.R. China
<EMAIL_ADDRESS>
###### Abstract.

Using Bergweiler and Eremenko’s method of gluing two certain meromorphic
functions along the real axis, we show that, for a dense set of orders
$\rho\in[1,\infty)$, there exists a Bank–Laine function $E$ such that
$\lambda(E)=\rho(E)=\rho$ and $E=w_{1}w_{2}$ with $w_{1}$ and $w_{2}$ being
two entire functions satisfying $c_{1}r^{\rho}\leq n(r,0,w_{i})\leq
c_{2}r^{\rho}$, $i=1,2$, where $c_{1}$ and $c_{2}$ are two positive constants.
This supplies the Bank–Laine functions $E$ constructed recently by the present
author such that the number of the zeros of $E$ has growth behavior
$d_{1}(\log r)^{2}r^{\lambda}\leq n(r,0,E)\leq d_{2}(\log r)^{2}r^{\lambda}$,
where $d_{1}$ and $d_{2}$ are two positive constants.

###### Key words and phrases:

Linear differential equation; Bank–Laine function; Bank–Laine conjecture;
Quasimeromorphic function; Quasiconformal surgery

###### 2010 Mathematics Subject Classification:

Primary 34A20; Secondary 30D15

The author is supported by the Fundamental Research Funds for the Central
Universities (FRF-TP-19-055A1) and a Project supported by the National Natural
Science Foundation of China (12301091)

##  1\. Introduction

An entire function $E(z)$ is called a _Bank–Laine function_ if $E(z)=0$
implies that $E^{\prime}(z)\in\\{-1,1\\}$. In particular, $E(z)$ is called
_special_ if $E(z)=0$ implies that $E^{\prime}(z)=1$. It is known that every
Bank–Laine function is a product of two normalized solutions $w_{1}$ and
$w_{2}$ of the second order differential equation

(1.1) $w^{\prime\prime}+A(z)w=0,$

where $A(z)$ is an entire function; see [9, Proposition 6.4]. All solutions of
equation (1.1) are entire functions. A pair $(w_{1},w_{2})$ of solutions of
(1.1) is called _normalized_ if $w_{1}$ and $w_{2}$ are linearly independent
and the _Wronskian determinant_
$W(w_{1},w_{2})=w_{1}w_{2}^{\prime}-w_{1}^{\prime}w_{2}$, which is always a
nonzero constant, is equal to $1$. Denote $F=w_{2}/w_{1}$. Then $F$ is a
locally univalent meromorphic function and the quotient
$F/F^{\prime}=w_{1}w_{2}$ is a Bank–Laine function, and all Bank–Laine
functions arise in this way. Note that a normalized pair $(w_{1},w_{2})$ can
be recovered from $F$ by the formulas $w_{1}^{2}=1/F^{\prime}$ and
$w_{2}^{2}=F^{2}/F^{\prime}$. Zeros of a Bank–Laine function $E$ are zeros and
poles of $F$, and $E$ is special if and only if $F$ is entire. We refer to [9]
for the basics about the second order differential equation (1.1) and also
about the Nevanlinna theory.

All Bank–Laine functions have order of growth at least $1$. In [4, 5],
Bergweiler and Eremenko constructed entire functions $A$ such that $\rho(A)$
is not an integer and equation (1.1) has two linearly independent solutions
$w_{1}$ and $w_{2}$ such that $\max\\{\lambda(w_{1}),\lambda(w_{1})\\}<\infty$
and one of $w_{1}$ and $w_{2}$ has no zeros. This shows for the first time the
existence of Bank–Laine functions $E$ for which the order $\rho(E)$ is finite
but not integer or half-integer. Bergweiler and Eremenko’s construction
actually disproved _the Bank–Laine conjecture_ concerning the complex
oscillation of the second order differential equation (1.1): _Let $w_{1}$ and
$w_{2}$ be two linearly independent solutions of equation (1.1), then
$\max\\{\lambda(w_{1}),\lambda(w_{2})\\}=\infty$ whenever
$\rho(A)\not\in\mathbb{N}$_. See also [9, Chapter 4]. Bank and Laine initiated
this topic in their two famous papers [1, 2], where they used the Nevanlinna
theory to prove that: If $\rho(A)\not\in\mathbb{N}$, then
$\lambda(E)\geq\rho(A)$; if $\rho(A)<1/2$, then $\lambda(E)=\infty$. The fact
that $\rho(A)=1/2$ implies $\lambda(E)=\infty$ was found by Shen [13] and
Rossi [12] independently, where they actually showed that if
$1/2\leq\rho(A)<1$, then the conclusion $\lambda(E)\geq\rho(A)$ can be
improved to satisfy the inequality

(1.2) $\frac{1}{\rho(A)}+\frac{1}{\lambda(E)}\leq 2.$

The Bank–Laine conjecture has attracted much interest; see the surveys [7, 10]
and references therein.

Using the method of constructing transcendental entire function $A$ by gluing
certain meromorphic functions with infinitely many times developed by
Bergweiler and Eremenko in [5], the present author [15] proves that: For each
$\lambda\in[1,\infty)$ and each $\delta\in[0,1]$, there exists a Bank–Laine
function $E$ such that $E=f_{1}f_{2}$ with $f_{1}$ and $f_{2}$ being two
entire functions such that $\lambda(f_{1})=\delta\lambda$ and
$\lambda(f_{2})=\lambda$, respectively. However, in the case
$\lambda(f_{1})=\lambda(f_{2})$, the zeros of the function $f_{1}$ constructed
in [15] satisfies, for all large $r$,

(1.3) $c_{1}(\log r)^{2}r^{\lambda}\leq n(r,0,f_{1})\leq c_{2}(\log
r)^{2}r^{\lambda},$

where $c_{1}$ and $c_{2}$ are two positive constants. It seems that the method
in [5, 15] does not allow to remove the term $(\log r)^{2}$ in (1.3). In this
paper we shall consider gluing two meromorphic functions along the real axis
as in [4] and prove the following

######  Theorem 1.1.

Let $M$ and $N$ be two positive integers. Then there exists a Bank–Laine
function $E$ of order

(1.4) $\rho=1+\frac{\log^{2}(M/N)}{4\pi^{2}}$

such that $\lambda(E)=\rho(E)=\rho$ and $E=w_{1}w_{2}$ for two entire
functions $w_{1}$ and $w_{2}$ such that $\lambda(w_{1})=\lambda(w_{2})=\rho$.
Moreover, $w_{1}$ and $w_{2}$ both satisfy $c_{1}r^{\rho}\leq n(r,0,w_{i})\leq
c_{2}r^{\rho}$, $i=1,2$, where $c_{1}$ and $c_{2}$ are two positive constants.

In section 2 we actually follow the process in [4] to prove the equivalent
statement: For a dense set of orders $\rho\in(1,\infty)$, there exists an
entire function $A$ of order $\rho(A)=\rho>1$ such that the differential
equation (1.1) has two linearly independent solutions $w_{1}$ and $w_{2}$ such
that $E=w_{1}w_{2}$ satisfies $\lambda(E)=\rho$ and $w_{1}$ and $w_{2}$
satisfy $\lambda(w_{1})=\lambda(w_{2})=\rho$. We can of course choose $M$ and
$N$ suitably so that $w_{2}$ has no zeros and thus the corresponding
Bank–Laine function $E$ is special. We may even choose $M$ and $N$ suitably so
that $w_{1}$ has no zeros.

Bergweiler and Eremenko have started their complicated construction from a
family of locally univalent meromorphic functions $F=f_{2}/f_{1}$ with
$(f_{1},f_{2})$ being a normalized pair of solutions such that
$\lambda(f_{i})<\infty$, $i=1,2$, of the second order differential equation

(1.5)
$f^{\prime\prime}-\left(\frac{1}{4}e^{2z}+\frac{1}{2}b_{1}e^{z}+b_{2}\right)f=0,$

where $b_{1}\not=0$ and $b_{2}$ are two constants. The case $b_{1}=0$
corresponds to equation (1.1) with $A(z)=e^{z}-b$, all nontrivial solutions
such that $\lambda(f)<\infty$ of which are explicitly characterized in [3];
see also [9, Theorem 5.22]. In [17, 16, 14], the present author solves non-
trivial solutions such that $\lambda(f)<\infty$ of equation (1.5). We show in
[14] that equation (1.5) admits two linearly independent solutions $f_{1}$ and
$f_{2}$ such that $\max\\{\lambda(f_{1}),\lambda(f_{2})\\}<\infty$ if and only
if there are two distinct nonnegative integers $k_{1},k_{2}$ such that
$b_{1}=\pm(k_{1}-k_{2})$ and $4b_{2}=4c^{2}=(k_{1}+k_{2}+1)^{2}$. In
particular, it is possible that $\min\\{\lambda(f_{1}),\lambda(f_{2})\\}=0$.
Letting $c=-(k_{1}+k_{2}+1)/2$, then equation (1.5) takes the form

(1.6)
$f^{\prime\prime}-\left[\frac{1}{4}e^{2z}+\frac{1}{2}(k_{2}-k_{1})e^{z}+\frac{(k_{1}+k_{2}+1)^{2}}{4}\right]f=0.$

Then we may write the two linearly independent solutions $f_{1}$ and $f_{2}$
of (1.6) as

(1.7)
$\begin{split}f_{1}=\kappa_{1}e^{-\frac{k_{1}+k_{2}+1}{2}z}e^{-\frac{1}{2}e^{z}}=\left(\sum_{i=0}^{k_{1}}a_{i}e^{iz}\right)e^{-\frac{k_{1}+k_{2}+1}{2}z}e^{-\frac{1}{2}e^{z}},\end{split}$

where

(1.8) $\begin{split}-(k_{1}+1-i)a_{i-1}=(2ic+i^{2})a_{i},\ \ \ \
i=1,\cdots,k_{1}\end{split}$

and

(1.9)
$\begin{split}f_{2}=\kappa_{2}e^{-\frac{k_{1}+k_{2}+1}{2}z}e^{\frac{1}{2}e^{z}}=\left(\sum_{j=0}^{k_{2}}b_{j}e^{jz}\right)e^{-\frac{k_{1}+k_{2}+1}{2}z}e^{\frac{1}{2}e^{z}},\end{split}$

where

(1.10) $\begin{split}(k_{2}+1-j)b_{j-1}=(2jc+j^{2})b_{j},\ \ \ \
j=1,\cdots,k_{2}.\end{split}$

See also [6, Theorem 1.6]. Suppose that $(f_{1},f_{2})$ is a normalized pair
of solutions of (1.6). It is easy to see that $a_{0}$ and $b_{0}$ should
satisfy
$a_{k_{1}}b_{k_{2}}=\frac{k_{1}!k_{2}!}{[(k_{1}+k_{2})!]^{2}}a_{0}b_{0}=1$ and
a simple calculation shows that

(1.11) $\begin{split}a_{i}&=\frac{k_{2}!}{(i+k_{2})!}a_{0},\ \ \
i=1,\cdots,k_{1},\\\ b_{j}&=\frac{(-1)^{j}k_{1}!}{(j+k_{1})!}b_{0},\ \ \
j=1,\cdots,k_{2}.\end{split}$

Let

(1.12)
$\begin{split}F=\frac{f_{2}}{f_{1}}=\left(\frac{\sum_{j=0}^{k_{2}}b_{j}e^{jz}}{\sum_{i=0}^{k_{1}}a_{i}e^{iz}}\right)e^{e^{z}}.\end{split}$

We choose $a_{0}=b_{0}$. Let $A_{0}=B_{0}=1$ and $A_{i}=a_{i}/a_{0}$ and
$B_{j}=b_{j}/b_{0}$. For every pair $(m,n)$ of two non-negative integers $m$
and $n$, we consider the rational function

(1.13)
$\begin{split}R_{m,n}(z)=\frac{P_{n}(z)}{Q_{m}(z)}=\frac{\sum_{j=0}^{2n}B_{j}z^{j}}{\sum_{i=0}^{m}A_{i}z^{i}}.\end{split}$

Thus we may write

(1.14)
$\begin{split}h_{m,n}(z)=R_{m,n}(z)e^{z}=\left(\frac{\sum_{j=0}^{2n}B_{j}z^{j}}{\sum_{i=0}^{m}A_{i}z^{i}}\right)e^{z}.\end{split}$

It follows that
$A_{m}B_{2n}=\frac{m!(2n)!}{[(m+2n)!]^{2}}=\frac{1}{\binom{m+2n}{m}(m+2n)!}$.
Then the meromorphic (entire when $m=0$) function

(1.15)
$\begin{split}g_{m,n}(z)=h_{m,n}(e^{z})=\left(\frac{\sum_{j=0}^{2n}B_{j}e^{jz}}{\sum_{i=0}^{m}A_{i}e^{iz}}\right)e^{e^{z}}\end{split}$

satisfies

(1.16)
$\begin{split}g^{\prime}_{m,n}(z)=\frac{1}{\binom{m+2n}{m}(m+2n)!}\frac{e^{e^{z}+(m+2n+1)z}}{\left(\sum_{i=0}^{m}A_{i}e^{iz}\right)^{2}}.\end{split}$

Since $A_{j}$ are all positive constants, $g_{m,n}(z)$ has the following
properties:

  * (1)

$g^{\prime}_{m,n}(z)\not=0$ for all $z\in\mathbb{C}$;

  * (2)

$g_{m,n}$ is increasing on $\mathbb{R}$, and satisfies $g_{m,n}(x)\to 1$ as
$x\to-\infty$ as well as $g_{m,n}(x)\to+\infty$ as $x\to+\infty$.

Bergweiler and Eremenko’s construction starts from the functions $g_{m,n}$ in
(1.15) with $m=0$ and two or infinitely many different even numbers $2n$. To
prove Theorem 1.1, we consider the generic choice of $m$, $n$ and allow
$g_{m,n}(z)$ to have some poles, i.e., we allow $m\geq 1$.

The associated _Schwarzian differential equation_ to each equation (1.1) is
given by

(1.17)
$\begin{split}S(F):=\left(\frac{F^{\prime\prime}}{F^{\prime}}\right)^{\prime}-\frac{1}{2}\left(\frac{F^{\prime\prime}}{F^{\prime}}\right)^{2}=\frac{F^{\prime\prime\prime}}{F^{\prime}}-\frac{3}{2}\left(\frac{F^{\prime\prime}}{F^{\prime}}\right)^{2}=2A,\end{split}$

where the expression $S(F)$ is called the _Schwarzian derivative_ of $F$. The
Schwarzian derivative is invariant under the linear fractional transformation
of $F$. The general solution of (1.17) is $F=w_{2}/w_{1}$. Note that the
function $g_{m,n}(z)$ in (1.15) produces Bank–Laine functions $E$ of order $1$
such that $E=w_{1}w_{2}$ for two entire functions $w_{1}$ and $w_{2}$
satisfying $c_{1}r\leq n(r,0,w_{i})\leq c_{2}r$, $i=1,2$, where $c_{1}$ and
$c_{2}$ are two positive constants. In [4], Bergweiler and Eremenko also
showed that the entire coefficient $A$ they constructed satisfies
$\rho(A)=\rho$. To do this, they used the fact that the Schwarzian derivative
$S(F)$ can be factored as $S(F)=B(F/F^{\prime})/2$, where

(1.18)
$\begin{split}B(E):=-2\frac{E^{\prime\prime}}{E}+\left(\frac{E^{\prime}}{E}\right)^{2}-\frac{1}{E^{2}},\end{split}$

which is first found in [1, 2]. The general solution of the differential
equation $B(E)=4A$, that is, of the equation

(1.19)
$\begin{split}4A=-2\frac{E^{\prime\prime}}{E}+\left(\frac{E^{\prime}}{E}\right)^{2}-\frac{1}{E^{2}},\end{split}$

is a product of a normalized pair of solutions of (1.1). By using the
arguments in [5] we may also use the identity (1.19) to show that
$\rho(A)=\rho$ for the coefficient $A$ of the second order differential
equation having the two solutions $w_{1}$ and $w_{2}$ in Theorem 1.1.

In the following, we keep the permanent notation: $(w_{1},w_{2})$ is a
normalized pair of solutions of (1.1), the quotient $F=w_{2}/w_{1}$ is a
solution of $S(F)=2A$, and $E=w_{1}w_{2}=F/F^{\prime}$ is a solution of
(1.19).

##  2\. Proof of Theorem 1.1

For every pair $(m,n)$ of two non-negative integers $m$ and $n$, we consider
the rational function $R_{m,n}(z)$ in (1.13). From now on, we fix two distinct
pairs $(m_{1},n_{1})$ and $(m_{2},n_{2})$ and will sometimes omit them from
notation. We shall assume that at least one of $m_{1}$ and $m_{2}$ is nonzero
and at least one of $n_{1}$ and $n_{2}$ is nonzero. Note that
$g_{m_{1},n_{1}}$ and $g_{m_{2},n_{2}}$ are locally univalent meromorphic
functions. The process of gluing two different $g_{m,n}$s is almost the same
as that in [4]: By restricting $g_{m_{1},n_{1}}$ to the upper half-plane
$H^{+}$ and $g_{m_{2},n_{2}}$ to the lower half-plane $H^{-}$, we then paste
them together, using a quasiconformal surgery, producing a meromorphic
function $F$. Then our Bank–Laine function will be $E=F/F^{\prime}$ and thus
$A=B(E)/4$ as in (1.18).

By the properties of $g_{m,n}$ described in the introduction, we see that
there exists an increasing diffeomorphism: $\phi:\mathbb{R}\to\mathbb{R}$ such
that $g_{m_{1},n_{1}}(x)=(g_{m_{2},n_{2}}\circ\phi)(x)$ for all
$x\in\mathbb{R}$. Denote $M=m_{1}+2n_{1}+1$ and $N=m_{2}+2n_{2}+1$ for
simplicity. Let

(2.1)
$\begin{split}k=\frac{M}{N}=\frac{m_{1}+2n_{1}+1}{m_{2}+2n_{2}+1}.\end{split}$

Though $g_{m,n}$ in (1.15) has some poles, we may use the arguments as in [4]
to show that the asymptotic behavior of the diffeomorphism $\phi$ is the
following:

(2.2) $\displaystyle\phi(x)$ $\displaystyle=$ $\displaystyle
x+O(e^{-x/2}),\quad\quad\quad\quad\phi^{\prime}(x)\to 1,\quad x\to+\infty,$
(2.3) $\displaystyle\phi(x)$ $\displaystyle=$ $\displaystyle
kx+c+O(e^{-\delta|x|/2}),\quad\phi^{\prime}(x)\to k,\quad x\to-\infty,$

with

(2.4)
$\begin{split}c=\frac{1}{N}\log\left[\frac{\binom{m_{2}+2n_{2}}{m_{2}}}{\binom{m_{1}+2n_{1}}{m_{1}}}\frac{N!}{M!}\right]\quad\text{and}\quad\delta=\frac{1}{2}\min\\{1,k\\}.\end{split}$

In order to prove (2.2), we note that

(2.5) $\begin{split}\log
g_{m_{1},n_{1}}(x)=e^{x}+O(x)=e^{x}(1+O(xe^{-x})),\quad
x\to+\infty.\end{split}$

The equation $g_{m_{1},n_{1}}(x)=(g_{m_{2},n_{2}}\circ\phi)(x)$ easily implies
that $\frac{2}{3}x\leq\phi(x)\leq 2x$ for large $x$. Thus we also have

(2.6) $\begin{split}\log
g_{m_{2},n_{2}}(\phi(x))&=e^{\phi(x)}(1+O(\phi(x)e^{-\phi(x)}))\\\
&=e^{\phi(x)}(1+O(\phi(x)e^{-2x/3})),\quad x\to+\infty.\end{split}$

Combining the last two equations we obtain

(2.7) $\begin{split}e^{\phi(x)-x}=1+O(xe^{-2x/3}),\quad
x\to+\infty,\end{split}$

from which the first statement in (2.2) easily follows. For the second
statement in (2.2) we use

(2.8)
$\begin{split}\phi^{\prime}=\frac{g^{\prime}_{m_{1},n_{1}}}{g_{m_{1},n_{1}}}\frac{g_{m_{2},n_{2}}\circ\phi}{g^{\prime}_{m_{2},n_{2}}\circ\phi},\end{split}$

so that

(2.9)
$\begin{split}\phi^{\prime}&=e^{Mx-N\phi(x)}\frac{A_{m_{1}}B_{2n_{1}}}{A_{m_{2}}B_{2n_{2}}}\frac{\sum_{i=0}^{m_{2}}A_{i}\phi(x)^{i}}{\sum_{i=0}^{m_{1}}A_{i}x^{i}}\frac{\sum_{j=0}^{2n_{2}}B_{j}\phi(x)^{j}}{\sum_{j=0}^{2n_{1}}B_{j}x^{j}}\\\
&\sim
e^{Mx-N\phi(x)+(m_{2}+2n_{2})\phi(x)-(m_{1}+2n_{1})x}=e^{x-\phi(x)}=1+o(1),\quad
x\to+\infty.\end{split}$

In order to prove (2.3) we note from (1.15) and (1.16) that

(2.10)
$\begin{split}h^{\prime}_{m,n}(x)=\frac{1}{\binom{m+2n}{m}(m+2n)!}\frac{e^{x}x^{m+2n}}{Q_{m}(x)^{2}}.\end{split}$

Since $h_{m,n}(x)$ is analytic on $[0,+\infty)$, then by taking the
derivatives of $h^{\prime}_{m,n}(x)$ successively we find that
$h^{\prime}_{m,n}(0)=\cdots=h^{(m+2n)}_{m,n}(0)=0$. Thus by the formula for
the error term of a Taylor series we have

(2.11)
$\begin{split}h_{m,n}(x)=R_{m,n}(x)e^{x}=1+\frac{x^{m+2n+1}}{\binom{m+2n}{m}(m+2n+1)!}+O(x^{m+2n+2}),\quad
x\to 0.\end{split}$

Hence

(2.12)
$\begin{split}g_{m,n}(x)&=1+\frac{e^{(m+2n+1)x}}{\binom{m+2n}{m}(m+2n+1)!}+O(e^{(m+2n+2)x})\\\
&=1+\frac{e^{(m+2n+1)x}}{\binom{m+2n}{m}(m+2n+1)!}(1+O(e^{x})),\quad
x\to-\infty.\end{split}$

The equation $g_{m_{1},n_{1}}(x)=g_{m_{2},n_{2}}(\phi(x))$ now yields

(2.13)
$\begin{split}\frac{\binom{m_{1}+2n_{1}}{m_{1}}}{\binom{m_{2}+2n_{2}}{m_{2}}}\frac{M!}{N!}e^{N\phi(x)-Mx}=1+O(e^{x})+O(e^{\phi(x)}),\quad
x\to-\infty\end{split}$

and hence

(2.14)
$\begin{split}\phi(x)=\frac{M}{N}x+\frac{1}{N}\log\left[\frac{\binom{m_{2}+2n_{2}}{m_{2}}}{\binom{m_{1}+2n_{1}}{m_{1}}}\frac{N!}{M!}\right]+O(e^{x})+O(e^{\phi(x)}),\quad
x\to-\infty,\end{split}$

which gives the first statement in (2.3). For the second statement in (2.3),
recalling that
$A_{m}B_{2n}=\frac{m!(2n)!}{[(m+2n)!]^{2}}=\frac{1}{\binom{m+2n}{m}(m+2n)!}$,
we use (2.8) to obtain

(2.15)
$\begin{split}\phi^{\prime}&\sim\frac{A_{m_{1}}B_{2n_{1}}}{A_{m_{2}}B_{2n_{2}}}e^{Mx-N\phi(x)}=\frac{A_{m_{1}}B_{2n_{1}}}{A_{m_{2}}B_{2n_{2}}}e^{Mx-N(kx+c+o(1))}\\\
&=\frac{A_{m_{1}}B_{2n_{1}}}{A_{m_{2}}B_{2n_{2}}}e^{-Nc+o(1)}=k+o(1).\end{split}$

Instead of defining a quasiregular function $U$, we now define a
quasimeromorphic function $U$ in the same way as that in [5]. For the
convenience of calculating the number of the zeros and poles of $U$ in the
final part of the proof, below we shall quote directly the process of the
definition in [5].

Let $D=\mathbb{C}\setminus\mathbb{R}_{\leq 0}$, and $p:D\to\mathbb{C}$,
$p(z)=z^{\mu}$, the principal branch of the power. Here $\mu$ is a complex
number to be determined so that $p$ maps $D$ onto the complement $G$ of a
logarithmic spiral $\Gamma$, with

(2.16) $\begin{split}p(x+i0)=p(kx-i0),\quad x<0.\end{split}$

It will be convenient to consider also the map $z\to\mu z$ obtained from $p$
by a logarithmic change of variable: If $w=p(z)$ then $\log w=\mu\log z$. This
shows that, with $a_{-}=\log k-i\pi$ and $a_{+}=i\pi$, we have $\Re(\mu
a_{-})=\Re(\mu a_{+})$; that is, $\Re(\mu(\log k-i\pi))=\Re(\mu i\pi)$.
Moreover, $\Im(i\pi/\mu)=\pi$. A simple computation now yields

(2.17)
$\begin{split}\mu=\Re\mu+i\Im\mu=\frac{2\pi}{4\pi^{2}+\log^{2}k}(2\pi-i\log
k).\end{split}$

The inverse map $h=p^{-1}$ is a conformal homeomorphism: $h:G\to D$. Let
$\Gamma^{\prime}=p(\mathbb{R}_{\geq 0})$. The two logarithmic spirals $\Gamma$
and $\Gamma^{\prime}$ divide the plane into two parts $G^{+}$ and $G^{-}$,
which are images under $p$ of the upper and lower half-planes, respectively.

The function $V$ defined by

(2.18) $V(z)=\left\\{\begin{array}[]{ll}(g_{m_{1},n_{1}}\circ h)(z),&\quad
z\in G^{+},\\\ (g_{m_{2},n_{2}}\circ h)(z),&\quad z\in
G^{-},\end{array}\right.$

is meromorphic in $G^{+}\cup G^{-}$ and has a jump discontinuity on $\Gamma$
and $\Gamma^{\prime}$. In view (2.2), (2.3) and (2.16), this discontinuity can
be removed by a small change in the independent variable. In order to do so,
we consider the strip $\Pi=\\{z:|\Im z|<1\\}$ and define a quasiconformal
homeomorphism $\tau:\mathbb{C}\to\mathbb{C}$, commuting the complex
conjugation, which is the identity outside of $\Pi$ and satisfies

(2.19) $\begin{split}\tau(x)=\phi(x),\quad
x>0,\quad\text{and}\quad\tau(kx)=\phi(x),\quad x<0.\end{split}$

The homeomorphism can be given by an explicit formula: For $y=\Im z\in(-1,1)$,
we set

(2.20) $\tau(x+iy)=\left\\{\begin{array}[]{ll}\phi(x)+|y|(x-\phi(x))+iy,&\quad
x\geq 0,\\\ \phi(x/k)+|y|(x-\phi(x/k)+iy),&\quad x<0.\end{array}\right.$

The Jacobian matrix $D_{\tau}$ of $\tau$ is given for $x>0$ and $0<|y|<1$ by

(2.21)
$D_{\tau}(x+iy)=\left(\begin{array}[]{cc}\phi^{\prime}(x)+|y|(1-\phi(x))&\pm(x-\phi(x))\\\
0&1\\\ \end{array}\right),$

and we see using (2.2) that

(2.22) $D_{\tau}(x+iy)=\left(\begin{array}[]{cc}1&0\\\ 0&1\\\
\end{array}\right),\quad 0<|y|<1,\quad x\to+\infty.$

Similarly, using (2.3) we find that

(2.23) $D_{\tau}(x+iy)=\left(\begin{array}[]{cc}1&\mp c\\\ 0&1\\\
\end{array}\right),\quad 0<|y|<1,\quad x\to-\infty.$

We conclude that $\tau$ is quasiconformal in the plane.

Now we modify $V$ to obtain a function continuous on $\Gamma$ and
$\Gamma^{\prime}$. We define $U:\mathbb{C}\to\mathbb{C}$ by

(2.24) $U(z)=\left\\{\begin{array}[]{ll}(g_{m_{1},n_{1}}\circ h)(z),&\quad
z\in G^{+}\cup\Gamma\cup\Gamma^{\prime}\cup\\{0\\},\\\
(g_{m_{2},n_{2}}\circ\tau\circ h)(z),&\quad z\in G^{-}.\end{array}\right.$

It follows from (2.16) and (2.19) that $U$ is continuous on $\Gamma$ and
$\Gamma^{\prime}$ and quasimeromorphic in the plane. The existence theorem for
solutions of the Beltrami equation [11, § V.1] shows that there exists a
quasiconformal homeomorphism $\psi:\mathbb{C}\to\mathbb{C}$ with the same
Beltrami coefficient as $U$. The function $F=U\circ\psi^{-1}$ is then
meromorphic.

We note that $U$ is meromorphic in $\mathbb{C}\setminus X$, where
$X=p(\Pi^{-})$, and $\Pi^{-}$ is the lower half of $\Pi$. Let
$\Delta=\\{z:|z|>1\\}$. It is easy to see that $X\cap\Delta$ has finite
logarithmic area; that is,

(2.25)
$\begin{split}\int_{X\cap\Delta}\frac{dxdy}{x^{2}+y^{2}}=\int_{\Pi^{-}\cap\Delta}\frac{|p^{\prime}(z)|^{2}}{|p(z)|^{2}}dxdy=|\mu|^{2}\int_{\Pi^{-}\cap\Delta}\frac{dxdy}{x^{2}+y^{2}}<+\infty.\end{split}$

Thus the Beltrami coefficient of $U$ (and hence of $\psi$) satisfies the
hypotheses of the Techmüler–Wittich–Beliskii theorem [11, § V.6]. This theorem
shows that $\psi$ is conformal at $\infty$ and may thus be normalized to
satisfy

(2.26) $\begin{split}\psi(z)\sim z,\quad z\to\infty.\end{split}$

We set $E=F/F^{\prime}$. As $F^{\prime}(z)\not=0$ for all $z\in\mathbb{C}$ by
construction, $E$ is entire. As all zeros and poles of $F$ are simple, all
residues of $F/F^{\prime}$ are equal to $1$ or $-1$, so $E^{\prime}(z)=1$ or
$E^{\prime}(z)=-1$ at every zero of $E$, which implies the Bank–Laine
property.

We begin to estimate the counting functions of the zeros and poles of $F$ as
in [5, 15]. For two positive quantities $S_{1}(r)$ and $S_{2}(r)$, below we
shall use the notation $S_{1}(r)\asymp S_{2}(r)$, provided that there are two
positive constants $c_{1}$ and $c_{2}$ such that $c_{2}S_{2}(r)\leq
S_{1}(r)\leq c_{1}S_{2}(r)$ for all large $r$.

For the meromorphic function $g_{m_{1},n_{1}}$, if $n_{1}\not=0$, then the
counting function $n(r,0,g_{m_{1},n_{1}})$ of the zeros satisfies
$n(r,0,g_{m_{1},n_{1}})\sim\eta r$ with a positive constant $\eta$. Thus the
counting function $\hat{n}(r,0,g_{m_{1},n_{1}})$ of the zeros in the upper
half-pane $H^{+}$ satisfies $\hat{n}(r,0,g_{m_{1},n_{1}})\sim\frac{1}{2}\eta
r$. We note that the image of the circle $\\{z:|z|=r\\}$ under
$h(z)=z^{1/\mu}$ is the part of the logarithmic spiral which connects two
points $-e^{x}$ and $-ke^{x}$ on the negative real axis and intersects the
positive real axis at $r^{1/\Re\mu}=\sqrt{k}e^{x}$. Therefore, when $r$ is
large, the counting function $\hat{n}(r,0,g_{m_{1},n_{1}}\circ h)$ of the
zeros of $g_{m_{1},n_{1}}\circ h$ in $G^{+}$ satisfies

(2.27) $\begin{split}d_{2}r^{1/\Re\mu}\leq\hat{n}(r,0,g_{m_{1},n_{1}}\circ
h)\leq d_{1}r^{1/\Re\mu}\end{split}$

for some positive constants $d_{1}$ and $d_{2}$ both dependent on $k$.
Similarly, if $n_{2}\not=0$, when $r$ is large, the counting function
$\check{n}(r,0,g_{m_{2},n_{2}}\circ h)$ of the zeros of $g_{m_{2},n_{2}}\circ
h$ in $G^{-}$ satisfies

(2.28) $\begin{split}d_{4}r^{1/\Re\mu}\leq\check{n}(r,0,g_{m_{2},n_{2}}\circ
h)\leq d_{3}r^{1/\Re\mu}\end{split}$

for some positive constants $d_{3}$ and $d_{4}$ both dependent on $k$. By the
definition of $\tau(x+iy)$ in (2.20) and the asymptotic behavior for $\phi(x)$
in (2.2) and (2.3), we see that, when $r$ is large, the counting function
$\check{n}(r,g_{m_{2},n_{2}}\circ\tau\circ h)$ of the zeros of
$g_{m_{2},n_{2}}\circ\tau\circ h$ in $G^{-}$ satisfies

(2.29)
$\begin{split}d_{6}r^{1/\Re\mu}\leq\check{n}(r,0,g_{m_{2},n_{2}}\circ\tau\circ
h)\leq d_{5}r^{1/\Re\mu}\end{split}$

for some positive constants $d_{5}$ and $d_{6}$ both dependent on $k$. By
summarizing the above estimates together with the relation in (2.26), we
obtain $n(r,0,F)\asymp r^{1/\Re\mu}$. It follows that $N(r,0,F)\asymp
r^{1/\Re\mu}$. Similarly, we also have $n(r,\infty,F)\asymp r^{1/\Re\mu}$ and
thus $N(r,\infty,F)\asymp r^{1/\Re\mu}$. Since $E=F/F^{\prime}$ and since the
zeros and poles of $F$ are simple, we have

(2.30) $\begin{split}N\left(r,0,E\right)\asymp r^{1/\Re\mu}.\end{split}$

Next we estimate $m(r,1/E)$. We note that there is a set $\Omega_{1}$ with
finite linear measure such that, for all $|z|=r\not\in\Omega_{1}$, the
functions $P_{n}(e^{z})$ and $Q_{n}(e^{z})$ both satisfy

(2.31)
$\begin{split}\exp(-(2n+\varepsilon)r)\leq&\left|P_{n}(e^{z})\right|\leq\exp((2n+\varepsilon)r),\\\
\exp(-(m+\varepsilon)r)\leq&\left|Q_{m}(e^{z})\right|\leq\exp((m+\varepsilon)r),\end{split}$

where $\varepsilon>0$ is a small constant. This implies that

(2.32)
$\begin{split}|g_{m,n}(z)|\leq\exp((m+\varepsilon)r+(2n+\varepsilon)r)\left|\exp(e^{z})\right|\end{split}$

for all $z\in\mathbb{C}$ such that $|z|\not\in\Omega_{1}$. Denote
$L_{i}=\max\\{m_{i},2n_{i}\\}$, $i=1,2$, for simplicity. Clearly, this implies
that there is a set $\Omega_{2}$ with finite linear measure such that

(2.33) $\begin{split}&\
|g_{m_{1},n_{1}}(z)|\leq\exp((L_{1}+\varepsilon)r)\exp\left(e^{\Re
z}\right)\quad\text{for}\quad z\in
H^{+}\quad\text{and}\quad|z|\not\in\Omega_{2},\\\ &\
|g_{m_{2},n_{2}}(z)|\leq\exp((L_{2}+\varepsilon)r)\exp\left(e^{\Re
z}\right)\quad\text{for}\quad z\in
H^{-}\quad\text{and}\quad|z|\not\in\Omega_{2}.\end{split}$

We deduce from (2.24) and (2.33) that there is a set $\Omega_{3}$ with finite
linear measure such that

(2.34) $\begin{split}\log|g_{m_{1},n_{1}}\circ h|\leq
2(L_{1}+\varepsilon)r^{1/\Re\mu}+\exp(r^{1/\Re\mu})\quad\text{for}\quad z\in
G^{+}\quad\text{and}\quad|z|\not\in\Omega_{3}\end{split}$

and, together with the asymptotic behavior for $\phi(x)$ in (2.2) and (2.3)
and also the definition $\tau(x+iy)$ in (2.20), that

(2.35) $\begin{split}\log|g_{m_{2},n_{2}}\circ\tau\circ h|\leq
2(L_{2}+\varepsilon)r^{1/\Re\mu}+\exp(r^{1/\Re\mu})\quad\text{for}\quad z\in
G^{+}\quad\text{and}\quad|z|\not\in\Omega_{3}.\end{split}$

Hence (2.26) yields that

(2.36) $\begin{split}\log|F(z)|\leq\exp((1+o(1))r^{1/\Re\mu})\end{split}$

as $|z|\to\infty$ outside the exceptional set $\Omega_{3}$. Together with the
relation in (2.30), the lemma on the logarithmic derivative [8, 9] now implies
that $E=F/F^{\prime}$ satisfies

(2.37) $\begin{split}m\left(r,\frac{1}{E}\right)=O(r^{1/\Re\mu}),\end{split}$

outside a set of finite linear measure. Thus, by (2.30), the characteristic
function of $E$ satisfies

(2.38) $\begin{split}T\left(r,E\right)\asymp r^{1/\Re\mu}.\end{split}$

In particular, $E$ has finite order $1/\Re\mu$ and we see that $E$ is a
Bank–Laine function satisfying $\lambda(E)=\rho(E)=1/\Re\mu=\rho$, as desired.
Moreover $E$ is the product of two solutions $w_{1}$ and $w_{2}$ of (1.1) such
that $c_{1}r^{1/\Re\mu}\leq n(r,0,w_{i})\leq c_{2}r^{1/\Re\mu}$, $i=1,2$,
where $c_{1}$ and $c_{2}$ are two positive constants. This completes the proof
of Theorem 1.1.

## References

  * [1] S. Bank and I. Laine. On the oscillation theory of $f^{\prime\prime}+Af=0$ where $A$ is entire. Bull. Amer. Math. Soc. 6 (1982), no. 1, 95–98.
  * [2] S. Bank and I. Laine. On the oscillation theory of $f^{\prime\prime}+Af=0$ where $A$ is entire. Trans. Amer. Math. Soc. 273 (1982), no. 1, 351–363.
  * [3] S. Bank and I. Laine. Representations of solutions of periodic second order linear differential equations. J. Reine Angew. Math. 344 (1983), 1–21.
  * [4] W. Bergweiler and A. Eremenko. On the Bank–Laine conjecture. J. Eur. Math. Soc. 19 (2017), no. 6, 1899–1909.
  * [5] W. Bergweiler and A. Eremenko. Quasiconformal surgery and linear differential equations. J. Anal. Math. 137 (2019), no. 2, 751–812.
  * [6] Y. M. Chiang, M. E. Ismail. On value distribution theory of second order periodic ODEs, special functions and orthogonal polynomials. Canad. J. Math. 58 (2006), no. 4, 726–767.
  * [7] G. G. Gundersen. The mathematical work of Ilpo Laine, Proceedings of the Workshop on Complex Analysis and its Applications to Differential and Functional Equations, 1–26, Publ. Univ. East. Finl. Rep. Stud. For. Nat. Sci., 14, Univ. East. Finl., Fac. Sci. For., Joensuu, 2014.
  * [8] W. K. Hayman. Meromorphic functions. Oxford Mathematical Monographs, Clarendon Press, Oxford, 1964.
  * [9] I. Laine. Nevanlinna theory and complex differential equations. De Gruyter Studies in Mathematics, vol. 15, Walter de Gruyter & Co., Berlin, 1993.
  * [10] I. Laine, K. Tohge. The Bank-Laine conjecture–a survey. Some topics on value distribution and differentiability in complex and p-adic analysis, 398–417, Math. Monogr. Ser., 11, Sci. Press Beijing, Beijing, 2008.
  * [11] O. Lehto, K. Virtanen. Quasiconformal mappings in the plane. Springer, New York-Heidelberg, 1973.
  * [12] J. Rossi. Second order differential equations with transcendental coefficients. Proc. Amer. Math. Soc. 97 (1986), no. 1, 61–66.
  * [13] L. C. Shen. Solution to a problem of S. Bank regarding exponent of convergence of zeros of the solutions of differential equation $f^{\prime\prime}+Af=0$. Kexue Tongbao (English Ed.). 30 (1985), no. 12, 1579–1585.
  * [14] Y. Y. Zhang. On the oscillation of certain second-order linear differential equations [J]. Proceedings of the Royal Society of Edinburgh Section A: Mathematics, 53 (2023), no. 6, 1965–1992.
  * [15] Y. Y. Zhang. Zero distribution of finite order Bank–Laine functions, prerint.
  * [16] Y. Y. Zhang. On entire function $e^{p(z)}\int_{0}^{z}\beta(t)e^{-p(t)}dt$ with applications to Tumura–Clunie equations and complex dynamics. Comput. Methods Funct. Theory, 23 (2023), no. 2, 213–235.
  * [17] Y. Y. Zhang, Z. S. Gao and J. L. Zhang. Entire solutions of certain nonlinear differential and delay-differential equations. J. Math. Anal. Appl. 503 (2021), no. 2, paper no. 125349.


# Interactive Audio-text Representation for Automated Audio Captioning with
Contrastive Learning

###### Abstract

Automated Audio captioning (AAC) is a cross-modal task that generates natural
language to describe the content of input audio. Most prior works usually
extract single-modality acoustic features and are therefore sub-optimal for
the cross-modal decoding task. In this work, we propose a novel AAC system
called CLIP-AAC to learn interactive cross-modality representation with both
acoustic and textual information. Specifically, the proposed CLIP-AAC
introduces an audio-head and a text-head in the pre-trained encoder to extract
audio-text information. Furthermore, we also apply contrastive learning to
narrow the domain difference by learning the correspondence between the audio
signal and its paired captions. Experimental results show that the proposed
CLIP-AAC approach surpasses the best baseline by a significant margin on the
Clotho dataset in terms of NLP evaluation metrics. The ablation study
indicates that both the pre-trained model and contrastive learning contribute
to the performance gain of the AAC model.

Index Terms: Automated audio captioning, cross-modal translation, pre-trained
model

##  1 Introduction

Automated Audio Captioning (AAC) aims to generate descriptive sentences for
audio inputs as a cross-modal translation task. It serves as a core module in
many real-world applications, such as producing text labels for sound search
engines [1] and assisting the hearing impaired to access audio content [2].

Most AAC approaches typically adopt encoder-decoder structures [3], where the
audio encoder extracts acoustic features from raw audio inputs, and the text
decoder generates corresponding descriptive captions. Recent work [4] observed
that it is difficult to train a strong encoder for audio inputs because the
supervision only comes from captions, which is quite limited. To overcome such
problem, prior studies [2, 5, 6, 7, 8, 9] proposed transfer learning to pre-
train audio encoders on Audioset [10] for better acoustic features. However,
such acoustic features only contain audio information, which has no text
information as a bridge for decoders and thus leads to misalignment of audio-
text information [11].

In the computer vision field, recent work [12] proposed a novel image-text
pre-trained model, known as CLIP, which was trained with millions of image-
text pairs and can therefore offer better features rich in semantics and
visuals for cross-modal task [13]. The prior study is the source of
inspiration for this work.

In this paper, we propose a novel encoder-decoder method, named CLIP-AAC, to
learn the cross-modality embeddings with both acoustic and textual information
for the AAC task. Specifically, the encoder in CLIP-AAC consists of an audio-
head and a text-head, which are designed to learn the audio embeddings and
text embeddings from audio inputs and corresponding captions. Furthermore, we
propose contrastive learning to enhance the correspondence between audio-text
embeddings. As a result, the extracted features can learn both audio and
textual information, which are then fed into the decoder to generate
descriptive sentences. At the training stage, the caption provides supervision
for both features extraction and token prediction. At the inference stage, as
the caption input is not available, the text-head of the encoder is discarded.
Experimental results show that the proposed CLIP-AAC approach surpasses strong
baselines on the Clotho dataset in terms of NLP evaluation metrics. The
ablation study indicates that both the pre-trained model and contrastive
learning contribute to the performance gain of the CLIP-AAC model.

The rest of this paper is organized as follows. In section 2, we introduce the
proposed CLIP-AAC architecture. In section 3, experimental settings and
evaluation metrics are presented. Section 4 reports and analyzes the result of
the experiments. Section 5 concludes the study.

##  2 CLIP-AAC Architecture

We now introduce the proposed CLIP-AAC architecture, which consists of three
modules: the encoder, the contrastive learning module and the decoder, as
illustrated in Figure 1.

Figure 1: The block diagram of the proposed CLIP-AAC model. The two dashed
boxes denote the audio-head and text-head, respectively. “BN” is the batch
normalization layer, and “MHA” denotes the multi-head attention mechanism.
{$A_{1}$,$A_{2}$,…,$A_{b}$} and {$T_{1}$,$T_{2}$,…,$T_{b}$} are the audio
embeddings and text embeddings extracted from audio-text pairs.

###  2.1 Audio-head and Text-head of Encoder

Given the $N$ audio-text pairs, we denote the AAC dataset as
$D=\\{(a_{i},t_{i})\\},i\in\\{1,2,...,N\\}$, where $a$ is the time-domain
audio clip and $t$ is the caption to describe this audio. In a typical AAC
model, the encoder $Enc$ is designed to convert the audio signal to the audio
embedding $A=Enc(a)$. Such approach usually extracts single-modality acoustic
features and are therefore sub-optimal for the cross-modal decoding task.

To alleviate such an issue, we propose an audio-head $Audio\\_Enc$ and a text-
head $Text\\_Enc$ for the encoder in CLIP-AAC as shown in Figure 1, which take
both audios and captions as inputs to extract corresponding embeddings for
each modality.

For audio-head, we employ the ESResNeXt [14] model which has demonstrated a
powerful ability to learn robust Time-Frequency transformation of audios.
Firstly, the time-domain signal is converted to a log-power spectrogram using
Short-Time Fourier Transform (STFT). Then, we feed the spectrogram into a 2-D
convolutional layer with kernel size of $7\times 7$ and a batch normalization
layer. Next, 4 dual-path blocks [15] are stacked to extract deep
representation. In each dual-path block, the first path contains 3 stacked
bottlenecks [16] with the structure shown in the yellow block. The second path
consists of a max-pooling layer, 2 convolutional layers with kernel size of
$3\times 3$ and $1\times 1$ respectively, and a batch normalization layer. The
outputs of the two paths are multiplied as they have the same shape. Finally,
the hidden features are fed into an average pooling layer and a linear layer
to generate an audio embedding with a length of $L$.

For text-head, the caption input $t$ with any length is firstly converted into
a word embedding. Then, a 12-layer Transformer [17] is employed to extract the
deep features of input embeddings. Next, a further normalization layer is
added after the final attention block [18]. To obtain a text embedding with
the same shape as the audio embedding, another linear layer is employed to map
the features to embeddings with a length of $L$.

The parameters of the CLIP-AAC encoder are initialized by AudioCLIP [19],
which includes three heads of image, audio and text. We remove the image-head
and finetune the text-head and audio-head to adapt the AAC task.

Table 1: BLEUn, METEOR, ROUGEL, CIDEr, SPICE, and SPIDEr in a comparative
study of initialization strategies for the proposed CLIP-AAC encoder. All
models are trained with batch size of 8.

System ID | Initialization | Frozen |  BLEU1 |  BLEU2 |  BLEU3 |  BLEU4 |  ROUGEL | METERO | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---|---|---
1 | Random | No | 0.421 | 0.246 | 0.159 | 0.070 | 0.312 | 0.103 | 0.060 | 0.059 | 0.060
2 | Audio-only | No | 0.480 | 0.285 | 0.183 | 0.114 | 0.326 | 0.132 | 0.217 | 0.081 | 0.149
3 | Audio-Text | Yes | 0.521 | 0.335 | 0.225 | 0.147 | 0.356 | 0.153 | 0.333 | 0.100 | 0.217
4 | Audio-Text | No | 0.561 | 0.365 | 0.245 | 0.161 | 0.372 | 0.168 | 0.394 | 0.115 | 0.254

###  2.2 Contrastive learning

We now introduce contrastive learning to learn the correspondence between
audio embeddings and text embeddings. Specifically, the audio-head
$Audio\\_Enc$ and text-head $Text\\_Enc$ are trained jointly 1) to maximize
the cosine similarity of audio embeddings $A_{i}$ with corresponding text
embeddings $T_{i}$ in a minibatch of $b$ pairs, and 2) to minimize the cosine
similarity of the audio embeddings $A_{i}$ with negative examples. Such
negative examples are remaining $b^{2}-b$ mismatched pairs. As shown in Figure
1, we denote the similarities of matched pairs as purple cells that are
distributed in the diagonal elements of the grid, while the mismatched pairs
are labelled as grey cells in the grid. Such optimization is achieved by
contrastive loss [20] over similarity scores, as shown in Algorithm 1.

Algorithm 1 Pseudocode for Contrastive Loss

1:A minibatch of $b$ audio-text pairs $\\{(a_{i},t_{i})\\}$,
$i\in\\{1,2,...,b\\}$

2:Contrastive loss $L_{cl}$

3:Initialize the parameters of audio-head $Audio\\_Enc$ and text-head
$Text\\_Enc$.

4:Initialize temperature parameter $t$.

5:Extract embeddings from each modality in
minibatch:$A_{i}=Audio\\_Enc(a_{i}),\ A_{i}\in\mathbb{R}^{1\times L}$,
$T_{i}=Text\\_Enc(t_{i}),\ E_{i}\in\mathbb{R}^{1\times L}$,

6:for i in 1, 2, …, b do for k in 1, 2, …, b do Calculate cosine similarity
between $A_{i}$ and $T_{k}$ $\left\langle
A_{i},T_{k}\right\rangle=\frac{A_{i}\cdot(T_{k})^{T}}{\left\|A_{i}\right\|\left\|T_{k}\right\|}$
(1) end forCalculate cross-entropy based contrastive loss for $A_{i}$
$L^{A\rightarrow T}_{i}=-\log\frac{exp(\left\langle
A_{i},T_{i}\right\rangle/t)}{\sum_{k=1}^{b}exp(\left\langle
A_{i},T_{k}\right\rangle/t)}\vspace{-0.1in}$ (2) end for

7:for i in 1, 2, …, b do for k in 1, 2, …, b do Calculate cosine similarity
between $T_{i}$ and $A_{k}$ $\left\langle
T_{i},A_{k}\right\rangle=\frac{T_{i}\cdot(A_{k})^{T}}{\left\|T_{i}\right\|\left\|A_{k}\right\|}$
(3) end forCalculate cross-entropy based contrastive loss for $T_{i}$
$L^{T\rightarrow A}_{i}=-\log\frac{exp(\left\langle
T_{i},A_{i}\right\rangle/t)}{\sum_{k=1}^{b}exp(\left\langle
T_{i},A_{k}\right\rangle/t)}\vspace{-0.1in}$ (4) end for

8:Return the contrastive loss $L_{cl}$
$L_{cl}=\frac{1}{b}\sum^{b}_{i=1}(\lambda L^{A\rightarrow
T}_{i}+(1-\lambda)L_{i}^{T\rightarrow A})\vspace{-0.1in}$ (5)

Given a minibatch of $b$ audio-text pairs, step 1 to 3 in Algorithm 1 explains
the feed forward process of audio and text inputs in the CLIP-AAC encoder. The
$A_{i}$ and $T_{i}$ are the extracted embeddings from audio $a_{i}$ and its
true caption $t_{i}$, respectively. In step 4, for each embedding $A_{i}$, we
calculate its cosine similarities with every text embedding
$\\{T_{1},T_{2},...T_{b}\\}$ in E.q (1). Since only the true-pair
$\left\langle A_{i},T_{i}\right\rangle$ is viewed as positive example, we
calculated the cross-entropy loss for each $A_{i}$ in E.q (2). Similarly, a
symmetric cross-entropy loss for each $T_{i}$ is calculated in E.q (3) and E.q
(4). Finally, the contrastive loss $L_{cl}$ is computed by weight summing
these two losses with a weight $\lambda$. We set $\lambda$ as 0.5 in this
work.

###  2.3 Decoder

The decoder of the proposed CLIP-AAC approach is a typical Transformer [17]
with multi-head attention, which has achieved state-of-the-art performance on
various cross-modal tasks.

As shown in Figure 1, the tokens of the caption are firstly mapped to word
embeddings and then fed into a masked multi-head self-attention layer to
obtain hidden features. Subsequently, such hidden features are sent into
another cross-attention layer to attentively fuse the audio embedding $A$ from
the audio-head in the encoder. Considering the length of the caption, such an
operation (grey block) is only repeated twice. Finally, a linear layer and
softmax function are employed to produce the probability distribution along
with the vocabulary.

During decoding, the $m$-th token $t_{m}$ is predicted based on previous
tokens $\\{t_{0},t_{1},...,t_{m-1}\\}$ and audio embedding $A$, so the output
probability of decoder is denoted as
$P_{\theta}(t_{m}|t_{0},t_{1},...,t_{m-1},A)$. The training objective is to
maximize the log-likelihood for each predicted token via the cross entropy
criterion $L_{ce}$:

$L_{ce}=-\sum_{m=0}^{M}logP_{\theta}(t_{m}|t_{0},t_{1},...,t_{m-1},A)$ (6)

Combined with contrastive loss $L_{cl}$ in E.q 5, we define the total loss
$L_{total}$ for the whole neural network as:

$L_{total}=\alpha L_{cl}+(1-\alpha)L_{ce}$ (7)

where $\alpha$ is the coefficient to trade-off each objective.

##  3 Experiments

###  3.1 Database

We conduct experiments on the Clotho dataset [21]. The Clotho data is
collected from the Freesound archive, and the duration of clips ranges from 15
to 30 seconds. The captions of Clotho are annotated by 5 different annotators
so that each audio clip contains 5 captions for diversity. The length of
captions ranges from eight to twenty words.

Following the previous works [11], we divided the Clotho dataset into a
training set (4884 audio clips with 24420 captions), and a test set (1045
audio clips with 5225 captions). Each audio clip combines one of five captions
as an audio-text training pair. At the inference stage, the average
performances are calculated between predicted captions and all five captions.

Table 2: BLEUn, METEOR, ROUGEL, CIDEr, SPICE, and SPIDEr in a comparative study of the proposed contrastive loss. “$\alpha$” is the coefficient for contrastive loss, and “B.S.” denotes the batch size during training. System ID | $\alpha$ | B.S. |  BLEU1 |  BLEU2 |  BLEU3 |  BLEU4 |  ROUGEL | METERO | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---|---|---
5 | 0 | 8 | 0.551 | 0.358 | 0.241 | 0.158 | 0.369 | 0.162 | 0.368 | 0.109 | 0.239
6 | 0.1 | 8 | 0.556 | 0.359 | 0.240 | 0.156 | 0.367 | 0.165 | 0.379 | 0.115 | 0.247
7 | 0.2 | 4 | 0.556 | 0.365 | 0.246 | 0.160 | 0.371 | 0.163 | 0.372 | 0.109 | 0.241
8 | 0.2 | 8 | 0.561 | 0.365 | 0.245 | 0.161 | 0.372 | 0.168 | 0.394 | 0.115 | 0.254
9 | 0.2 | 16 | 0.572 | 0.379 | 0.257 | 0.169 | 0.379 | 0.171 | 0.407 | 0.118 | 0.263
10 | 0.3 | 8 | 0.561 | 0.366 | 0.248 | 0.165 | 0.374 | 0.166 | 0.380 | 0.114 | 0.247
Table 3: BLEUn, METEOR, ROUGEL, CIDEr, SPICE, and SPIDEr in a comparative study of CLIP-AAC and other competitive techniques. Method |  BLEU1 |  BLEU2 |  BLEU3 |  BLEU4 |  ROUGEL | METERO | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---
GRU Baseline [21] | 0.389 | 0.136 | 0.055 | 0.015 | 0.262 | 0.084 | 0.074 | 0.033 | 0.054
PreCNN Transformer [7] | 0.534 | 0.343 | 0.230 | 0.151 | 0.356 | 0.160 | 0.346 | 0.108 | 0.227
CL4AC [1] | 0.553 | 0.349 | 0.226 | 0.143 | 0.374 | 0.168 | 0.368 | 0.115 | 0.242
AT-CNN10 [4] | 0.556 | 0.363 | 0.242 | 0.159 | 0.368 | 0.169 | 0.377 | 0.115 | 0.246
TL + RLSSR [11] | 0.551 | 0.369 | 0.252 | 0.168 | 0.373 | 0.165 | 0.380 | 0.111 | 0.246
CLIP-AAC (ours) | 0.572 | 0.379 | 0.257 | 0.169 | 0.379 | 0.171 | 0.407 | 0.119 | 0.263

###  3.2 Network Configuration

At the training stage, the network was optimized by Adam [22]. To prevent the
exploding gradients, a small learning rate of $1\times 10^{-5}$ is applied to
the pre-trained encoder and $1\times 10^{-3}$ is applied to the decoder. A
warm-up strategy is applied in the first 5 epochs that linearly increases the
initial learning rate. The training epoch is set to 40 and the default batch
size is 8. In addition, we use label smoothing [23] with $\epsilon=0.1$ and
dropout with a rate of 0.2 to mitigate the over-fitting problem. During
inference, the beam search is applied with a beam size of 3.

###  3.3 Metrics

We report performances on six metrics, including BLEUn [24], METEOR [25],
ROUGEL [26], CIDEr [27], SPICE [28], and SPIDEr [29].

  * •

BLEUn measures the correspondence between generated text and reference text by
computing the precision of $n$-gram in the text.

  * •

METEOR measures the harmonic mean of precision and recalls based on word-level
matches between generated text and reference text.

  * •

ROUGEL computes the F-measures based on the longest common sub-sequence.

  * •

CIDEr both considers term frequency inverse document frequency weights of
n-grams and the cosine similarity between the generated text and reference
text.

  * •

SPICE converts the captions to scene graphs and computes F-score based on
tuple in them.

  * •

SPIDEr is the linear combination of CIDEr and SPICE.

The first three metrics are proposed for machine translation systems but are
also widely used to evaluate AAC systems in the previous works [1, 2, 4, 11].
The last three metrics are specifically used for captioning task [30, 31, 32].
For all metrics, higher score denotes better performance.

##  4 Results

###  4.1 Effect of Initialization strategies for CLIP-AAC encoder

We first analyze the performances of various initialization strategies for the
encoder in the proposed CLIP-AAC approach. As shown in Table 1, system 1 and
system 2 only utilize the audio-head in encoders, while system 3 and system 4
combine the proposed audio-head and text-head in encoders. Specifically,
“Random” means that the encoder in system 1 is initialized randomly. “Audio-
only” denotes that the encoder in system 2 is pre-trained on Audioset. “Audio-
Text” depicts that the encoders in system 3 and 4 are initialized with the
modified AudioCLIP and then fine-tuned on Audioset. Compared with system 3,
the encoder in system 4 is learnable during the training procedure.

We observe from the performances of system 1 that the encoder may not produce
good features in deep architectures with random initialization. Compared with
system 2, system 3 achieves better performance as the proposed encoder with
audio-head and text-head can learn both acoustic and textual information.
System 4 further improves performances by making the encoder trainable on the
Clotho dataset. We adopt the setting of system 4 for the encoder hereafter.

###  4.2 Effect of Contrastive Loss

We further report the effect of proposed contrastive loss $L_{cl}$ on CLIP-AAC
approach. As shown in Table 2, $\alpha$ is the coefficient to trade-off the
contrastive loss $L_{cl}$ and cross-entropy loss $L_{ce}$. $B.S.$ denotes the
batch size used at training stage.

From system 5, 6, 8 and 10, we observe that performances improve as the
coefficient $\alpha$ increases, indicating that the contrastive learning
strategy can learn more audio and text information. We obtain best
performances when $\alpha$ for $L_{cl}$ is set as 0.2. Furthermore, system 7-9
improve performances by increasing the batch size $B.S.$ because a large batch
size can increase the diversity of negative audio-text pairs for training.
Best performances are obtained with $B.S.$ of 16.

Figure 2: Visualization of cosine similarity (after softmax) matrix between
audio embeddings ($A_{1}$ to $A_{8}$) and text embeddings ($T_{1}$ to
$T_{8}$). (A) and (B) denote the system 8 before and after training,
respectively.

To further show the contribution of the contrastive loss, we visualize the
cosine similarity matrix of audio-text embeddings after the softmax function
in system 8. The diagonal elements denote the cosine similarity of true audio-
text pairs in Figure 2. We observe that the proposed pre-trained CLIP-AAC
encoder can catch some correspondence between true audio-text pairs before
training. After training, such correspondence is significantly enhanced by
proposed contrastive learning. Therefore, we draw a conclusion that the CLIP-
AAC encoder can provide high-quality embeddings for cross-modal decoding.

###  4.3 Benchmark against other competitive methods

Table 3 summarizes the comparison between the proposed CLIP-AAC and other
competitive techniques in terms of BLEUn, METEOR, ROUGEL, CIDEr, SPICE, and
SPIDEr. We conduct experiments on the Clotho dataset. Except for the GRU
baseline [21], all encoders in the other five techniques are pre-trained on
AudioSet. The GRU baseline is trained from scratch. We observe that the
proposed CLIP-AAC obtained the best performances for all metrics.

##  5 Conclusions

In this paper, we propose CLIP-AAC to learn interactive cross-modality
representation with both acoustic and textual information for AAC tasks.
Specifically, we introduce an audio-head and text-head in the encoder to
capture audio-text correspondence. Furthermore, we also propose a contrastive
learning strategy to optimize the CLIP framework. Experimental results show
the proposed CLIP-AAC outperforms the best baseline by a large margin in terms
of all metrics.

##  6 Acknowledgements

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG-100E-2018-006). The
computational work for this article was partially performed on resources of
the National Supercomputing Centre, Singapore (https://www.nscc.sg).

## References

  * [1] X. Liu, Q. Huang, X. Mei, T. Ko, H. L. Tang, M. D. Plumbley, and W. Wang, “Cl4ac: A contrastive loss for audio captioning,” _arXiv preprint arXiv:2107.09990_ , 2021.
  * [2] X. Mei, Q. Huang, X. Liu, G. Chen, J. Wu, Y. Wu, J. Zhao, S. Li, T. Ko, H. L. Tang _et al._ , “An encoder-decoder based audio captioning system with transfer and reinforcement learning,” _arXiv preprint arXiv:2108.02752_ , 2021.
  * [3] K. Drossos, S. Adavanne, and T. Virtanen, “Automated audio captioning with recurrent neural networks,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2017, pp. 374–378.
  * [4] X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, “Investigating local and global information for automated audio captioning with transfer learning,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 905–909.
  * [5] X. Xu, Z. Xie, M. Wu, and K. Yu, “The sjtu system for dcase2021 challenge task 6: audio captioning based on encoder pre-training and reinforcement learning,” DCASE2021 Challenge, Tech. Rep, Tech. Rep., 2021.
  * [6] Y. Wu, K. Chen, Z. Wang, X. Zhang, F. Nian, S. Li, and X. Shao, “Audio captioning based on transformer and pre-training for 2020 dcase audio captioning challenge,” _DCASE2020 Challenge, Tech. Rep._ , 2020.
  * [7] K. Chen, Y. Wu, Z. Wang, X. Zhang, F. Nian, S. Li, and X. Shao, “Audio captioning based on transformer and pretrained cnn,” in _Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop_ , 2020, pp. 21–25.
  * [8] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio captioning transformer,” _arXiv preprint arXiv:2107.09817_ , 2021.
  * [9] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [10] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [11] A. Koh, F. Xue, and E. S. Chng, “Automated audio captioning using transfer learning and reconstruction latent space similarity regularization,” _arXiv preprint arXiv:2108.04692_ , 2021.
  * [12] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [13] R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,” _arXiv preprint arXiv:2111.09734_ , 2021.
  * [14] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresne (x) t-fbsp: Learning robust time-frequency transformation of audio,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [15] A. Guzhov, F. Raue, and Hee, “Esresnet: Environmental sound classification based on visual domain models,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 4933–4940.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [18] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [19] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” _arXiv preprint arXiv:2106.13043_ , 2021.
  * [20] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” _arXiv preprint arXiv:2010.00747_ , 2020.
  * [21] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 736–740.
  * [22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [23] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [25] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [26] A. Lavie and A. Agarwal, “Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments,” in _Proceedings of the second workshop on statistical machine translation_ , 2007, pp. 228–231.
  * [27] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * [28] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in _European conference on computer vision_. Springer, 2016, pp. 382–398.
  * [29] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 873–881.
  * [30] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3156–3164.
  * [31] L. Zhou, Y. Kalantidis, X. Chen, J. J. Corso, and M. Rohrbach, “Grounded video description,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6578–6587.
  * [32] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4651–4659.