A Riemann theta function formula with its application to double periodic wave
solutions of nonlinear equations

Engui Fana111 E-mail address<EMAIL_ADDRESS>and Kwok Wing Chowb
a. School of Mathematical Sciences and Key Laboratory of Mathematics for
Nonlinear Science,
Fudan University, Shanghai, 200433, P.R. China
b. Department of Mechanical Engineering, University of Hong Kong, Pokfulam,
Hong Kong

Abstract. Based on a Riemann theta function and Hirota’s bilinear form, a
lucid and straightforward way is presented to explicitly construct double
periodic wave solutions for both nonlinear differential and difference
equations. Once such a equation is written in a bilinear form, its periodic
wave solutions can be directly obtained by using an unified theta function
formula. The relations between the periodic wave solutions and soliton
solutions are rigorously established. The efficiency of our proposed method
can be demonstrated on a class variety of nonlinear equations such as those
considered in this paper, shall water wave equation, (2+1)-dimensional
Bogoyavlenskii-Schiff equation and differential-difference KdV equation.
Keywords: Nonlinear equations; Hirota’s bilinear method; Riemann theta
function; double periodic wave solutions; soliton solutions.
PACS numbers: 11\. 30. Pb; 05. 45. Yv; 02. 30. Gp; 45. 10. -b.

1\. Introduction

The bilinear derivative method developed by Hirota is a powerful and direct
approach to construct exact solution of nonlinear equations. Once a nonlinear
equation is written in bilinear forms by a dependent variable transformation,
then multi-soliton solutions are usually obtained [2]-[7]. It was based on
Hirota forms that Nakamura proposed a convenient way to construct a kind of
quasi-periodic solutions of nonlinear equations [8, 9], where the periodic
wave solutions of the KdV equation and the Boussinesq equation were obtained.
Such a method indeed exhibits some advantages. For example, it does not need
any Lax pairs and Riemann surface for the considered equation, allows the
explicit construction of multi-periodic wave solutions, only relies on the
existence of the Hirota’s bilinear form, as well as all parameters appearing
in Riemann matrix are arbitrary. Recently, further development was made to
investigate the discrete Toda lattice, (2+1)-dimensional Kadomtsev-
Petviashvili equation and Bogoyavlenskii’s breaking soliton equation
[10]-[15]. However, where repetitive recursion and computation must be
preformed for each equation [8]-[15].

The motivation of this paper is to considerably improves the key steps of the
above existing methods. To achieve this aim, we devise a theta function
bilinear formula, which actually provides us a lucid and straightforward way
for applying in a class of nonlinear equations. Once a nonlinear equation is
written in bilinear forms, then the double periodic wave solutions of the
nonlinear equation can be obtained directly by using the formula. Moreover, we
propose a simple and effective method to analyze asymptotic properties of the
periodic solutions. As illustrative examples, we shall construct double
periodic wave solutions to the shall water wave equation, (2+1)-dimensional
Bogoyavlenskii-Schiff equation and differential-difference KdV equation.

The organization of this paper is as follows. In section 2, we briefly
introduce a Hirota bilinear operator and a Riemann theta function. In
particular, we provide a key formula for constructing double periodic wave
solutions for both differential and difference equations. As applications of
our method, in sections 3-5, we construct double periodic wave solutions to
the shall water wave equation, (2+1)-dimensional Bogoyavlenskii-Schiff
equation and differential-difference KdV equation, respectively. In addition,
it is rigorously shown that the double periodic wave solutions tend to the
soliton solutions under small amplitude limits.
2\. Hirota bilinear operator and Riemann theta function

To fix the notations we recall briefly some notions that will be used in this
paper. The Hirota bilinear operators $D_{x},D_{t}$ and $D_{n}$ are defined as
follows:

$\displaystyle D_{x}^{m}D_{t}^{k}f(x,t)\cdot
g(x,t)=(\partial_{x}-\partial_{x^{\prime}})^{m}(\partial_{t}-\partial_{t^{\prime}})^{k}f(x,t)g(x^{\prime},t^{\prime})|_{x^{\prime}=x,t^{\prime}=t}$
$\displaystyle e^{\delta D_{n}}f(n)\cdot
g(n)=e^{\delta(\partial_{n}-\partial_{n}^{\prime})}f(n)g(n^{\prime})|_{n^{\prime}=n}=f(n+\delta)g(n-\delta),$
$\displaystyle{\rm cosh}(\delta D_{n})f(n)\cdot g(n)=\frac{1}{2}(e^{\delta
D_{n}}+e^{-\delta D_{n}})f(n)\cdot g(n),$ $\displaystyle{\rm sinh}(\delta
D_{n})f(n)\cdot g(n)=\frac{1}{2}(e^{\delta D_{n}}-e^{-\delta D_{n}})f(n)\cdot
g(n).$

Proposition 1. The Hirota bilinear operators $D_{x},D_{t}$ and $D_{n}$ have
properties [2]-[7]

$\displaystyle D_{x}^{m}D_{t}^{k}e^{\xi_{1}}\cdot
e^{\xi_{2}}=(\alpha_{1}-\alpha_{2})^{m}(\omega_{1}-\omega_{2})^{k}e^{\xi_{1}+\xi_{2}},$
$\displaystyle e^{\delta D_{n}}e^{\xi_{1}}\cdot
e^{\xi_{2}}=e^{\delta(\nu_{1}-\nu_{2})}e^{\xi_{1}+\xi_{2}},$
$\displaystyle{\rm cosh}(\delta D_{n})e^{\xi_{1}}\cdot e^{\xi_{2}}={\rm
cosh}[\delta(\nu_{1}-\nu_{2})]e^{\xi_{1}+\xi_{2}},$ $\displaystyle{\rm
sinh}(\delta D_{n})e^{\xi_{1}}\cdot e^{\xi_{2}}={\rm
sinh}[\delta(\nu_{1}-\nu_{2})]e^{\xi_{1}+\xi_{2}},$

where $\xi_{j}=\alpha_{j}x+\omega_{j}t+\nu_{j}n+\sigma_{j}$, and $\alpha_{j},\
\omega_{j},\nu_{j},\ \sigma_{j}$, $j=1,2$ are parameters and $n\in\mathbb{Z}$
is a discrete variable. More generally, we have

$None$ $\displaystyle F(D_{x},D_{t},D_{n})e^{\xi_{1}}\cdot
e^{\xi_{2}}=F(\alpha_{1}-\alpha_{2},\omega_{1}-\omega_{2},\exp[\delta(\nu_{1}-\nu_{2})])e^{\xi_{1}+\xi_{2}},$

where $F(D_{x},D_{t},D_{n})$ is a polynomial about operators $D_{x},D_{t}$ and
$D_{n}$. This properties are useful in deriving Hirota’s bilinear form and
constructing periodic wave solutions of nonlinear equations.

In the following, we introduce a general Riemann theta function and discuss
its periodicity, which plays a central role in the construction of periodic
solutions of nonlinear equations. The Riemann theta function reads

$None$ $\vartheta\left[\begin{matrix}\varepsilon\\\
s\end{matrix}\right]({\xi},{\tau})=\sum_{{m}\in\mathbb{{Z}}}\exp\\{2\pi
i({\xi}+{\varepsilon})({m}+{s})-\pi{\tau}({m}+{s})^{2}\\}.$

Here the integer value ${m}\in\mathbb{Z}$, complex parameter
${s},{\varepsilon}\in\mathcal{C}$, and complex phase variables
${\xi}\in\mathcal{C}$; The ${\tau}>0$ which is called the period matrix of the
Riemann theta function.

In the definition of the theta function (2.2), for the case
${s}={\varepsilon}={0}$, hereafter we use
$\vartheta({\xi},{{\tau}})=\vartheta\left[\begin{matrix}0\\\
0\end{matrix}\right]({\xi},{\tau})$ for simplicity. Moreover, we have
$\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})=\vartheta({\xi}+{\varepsilon},{\tau})$.

Definition 1. A function $g(t)$ on $\mathbb{C}$ is said to be quasi-periodic
in $t$ with fundamental periods $T_{1},\cdots,T_{k}\in\mathbb{C}$, if
$T_{1},\cdots,T_{k}$ are linearly dependent over $\mathbb{Z}$ and there exists
a function $G(y_{1},\cdots,y_{k})$, such that

$G(y_{1},\cdots,y_{j}+T_{j},\cdots,y_{k})=G(y_{1},\cdots,y_{j},\cdots,y_{k}),\
\ {\rm for\ all}\ y_{j}\in\mathbb{C},\ j=1,\cdots,k.$
$G(t,\cdots,t,\cdots,t)=g(t).$

In particular, $g(t)$ is called double periodic as $k=2$, and it becomes
periodic with $T$ if and only if $T_{j}=m_{j}T,\ j=1,\cdots,k.$ $\square$

Let’s first see the periodicity of the theta function $\vartheta(\xi,\tau)$.

Proposition 2. [16] The theta function $\vartheta({\xi},{\tau})$ has the
periodic properties

$None$ $\displaystyle\vartheta({\xi}+{1}+i{\tau},{\tau})=\exp(-2\pi
i\xi+\pi\tau)\vartheta({\xi},{\tau}).$

We regard the vectors $1$ and $i\tau_{j}$ as periods of the theta function
$\vartheta({\xi},{\tau})$ with multipliers $1$ and $\exp({-2\pi
i\xi+\pi\tau})$, respectively. Here, $i\tau$ is not a period of theta function
$\vartheta({\xi},{\tau})$, but it is the period of the functions
$\partial^{2}_{\xi}\ln\vartheta({\xi},{\tau})$,
$\partial_{\xi}\ln[\vartheta({\xi}+{e},{\tau})/\vartheta({\xi}+{h},{\tau})]$
and
$\vartheta({\xi}+{e},{\tau})\vartheta({\xi}-{e},{\tau})/\vartheta({\xi}+{h},{\tau})^{2}$.

Proposition 3. The meromorphic functions $f({\xi})$ on $\mathbb{C}$ are as
follow

$\displaystyle(i)\ \ \ \ \
f({\xi})=\partial_{\xi}^{2}\ln\vartheta({\xi},{\tau}),\ \ {\xi}\in\mathbb{C},$
$\displaystyle(ii)\ \ \ \
f({\xi})=\partial_{\xi}\ln\frac{\vartheta({\xi}+{e},{\tau})}{\vartheta({\xi}+{h},{\tau})},\
\ {\xi},\ {e},\ {h}\in\mathbb{C}.$ $\displaystyle(ii)\ \ \ \
f({\xi})=\frac{\vartheta({\xi}+{e},{\tau})\vartheta({\xi}-{e},{\tau})}{\vartheta({\xi},{\tau})^{2}},\
\ {\xi},\ {e},\ {h}\in\mathbb{C}.$

then in all three cases (i)–(iii), it holds that

$None$ $\displaystyle f({\xi}+{1}+i{\tau})=f({\xi}),\ \ \ {\xi}\in\mathbb{C},$

that is, $f(\xi)$ is a double periodic function with $1$ and $i\tau$.

Proof. By using (2.3), it is easy to see that

$\displaystyle\frac{\partial_{\xi}\vartheta({\xi}+{1}+i{\tau},{\tau})}{\vartheta({\xi}+{1}+i{\tau},{\tau})}=-2\pi
i+\frac{\partial_{\xi}\vartheta({\xi},{\tau})}{\vartheta({\xi},{\tau})},$

or equivalently

$None$
$\displaystyle\partial_{\xi}\ln\vartheta({\xi}+{1}+i{\tau},{\tau})=-2\pi
i+\partial_{\xi}\ln\vartheta({\xi},{\tau}).$

Differentiating (2.5) with respective to $\xi$ again immediately proves the
formula (2.4) for the case (i). The formula (2.4) can be proved for the cases
(ii) and (iii) in a similar manner. $\square$

Theorem 1. Suppose that $\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})$ and
$\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})$ are two Riemann theta functions, in which
$\xi=\alpha x+\omega t+\nu n+\sigma$. Then Hirota bilinear operators
$D_{x},D_{t}$ and $D_{n}$ exhibit the following perfect properties when they
act on a pair of theta functions

$None$ $\displaystyle
D_{x}\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})\cdot\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})$
$\displaystyle=\left[\sum_{{\mu=0,1}}\partial_{x}\vartheta\left[\begin{matrix}\varepsilon^{\prime}-\varepsilon\\\
-\mu/2\end{matrix}\right](2{\xi},2{\tau})|_{{\xi}={0}}\right]\vartheta\left[\begin{matrix}\varepsilon^{\prime}+\varepsilon\\\
\mu/2\end{matrix}\right](2{\xi},2{\tau}),$ $None$ $\displaystyle\exp(\delta
D_{n})\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})\cdot\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})$
$\displaystyle=\left[\sum_{{\mu=0,1}}\exp(\delta
D_{n})\vartheta\left[\begin{matrix}\varepsilon^{\prime}-\varepsilon\\\
-\mu/2\end{matrix}\right](2{\xi},2{\tau})|_{{\xi}={0}}\right]\vartheta\left[\begin{matrix}\varepsilon^{\prime}+\varepsilon\\\
\mu/2\end{matrix}\right](2{\xi},2{\tau}),$

where the notation $\sum_{{\mu=0,1}}$ represents two different transformations
corresponding to $\mu=0,1$. The bilinear formula for $t$ is the same as (2.6)
by replacing $\partial_{x}$ with $\partial_{t}$.

In general, for a polynomial operator $F(D_{x},D_{t},D_{n})$ with respect to
$D_{x},D_{t}$ and $D_{n}$, we have the following useful formula

$None$ $\displaystyle
F(D_{x},D_{t},D_{n})\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})\cdot\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})=\left[\sum_{{\mu}}C({\varepsilon^{\prime}},{\varepsilon},{\mu})\right]\vartheta\left[\begin{matrix}\varepsilon^{\prime}+\varepsilon\\\
\mu/2\end{matrix}\right](2{\xi},2{\tau}),$

in which, explicitly

$None$ $\displaystyle
C({\varepsilon},{\varepsilon^{\prime}},{\mu})=\sum_{{m}\in\mathbb{Z}^{N}}F({\mathcal{M}})\exp\left[-2\pi{\tau}({m}-{\mu}/2)^{2}-2\pi
i({m}-{\mu}/2)({\varepsilon^{\prime}}-{\varepsilon})\right].$

where we denote vector ${\mathcal{M}}=(4\pi i({m}-{\mu}/2){\alpha},\ 4\pi
i({m}-{\mu}/2){\omega},\exp[4\pi i({m}-{\mu}/2)\delta\nu]).$

Proof. Making use of Proposition 1, we obtain the relation

$\displaystyle D_{x}\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})\cdot\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})$
$\displaystyle=\sum_{m^{\prime},m\in\mathbb{Z}}{D}_{x}\exp\\{2\pi
im^{\prime}(\xi+\varepsilon^{\prime})-\pi m^{\prime
2}{\tau}\\}\cdot\exp\\{2\pi im(\xi+\varepsilon)-\pi m^{2}{\tau}\\},$
$\displaystyle=\sum_{m^{\prime},m\in\mathbb{Z}}2\pi
i\alpha(m^{\prime}-m)\exp\left\\{2\pi i(m^{\prime}+m)\xi-2\pi
i(m^{\prime}\varepsilon^{\prime}+m\varepsilon)-\pi{\tau}[m^{\prime
2}+m^{2}]\right\\}$

By shifting sum index as $m=l^{\prime}-m^{\prime}$, then

$\displaystyle\Delta=\sum_{l^{\prime},m^{\prime}\in\mathbb{Z}}2\pi
i\alpha(2m^{\prime}-l^{\prime})\exp\left\\{2\pi il^{\prime}\xi-2\pi
i[m^{\prime}\varepsilon^{\prime}+(l^{\prime}-m^{\prime})\varepsilon]-\pi{\tau}[m^{\prime
2}+(l^{\prime}-m^{\prime})^{2}]\right\\}$
$\displaystyle\stackrel{{\scriptstyle l^{\prime}=2l+\mu}}{{=}}\sum_{\mu=0,1}\
\ \sum_{l,m^{\prime}\in\mathbb{Z}}2\pi i\alpha(2m^{\prime}-2l-\mu)\exp\\{4\pi
i\xi(l+\mu/2)$ $\displaystyle\ \ \ \ \ -2\pi
i[m^{\prime}\varepsilon^{\prime}-(m-2l-\mu)\varepsilon]-\pi[m^{\prime
2}+(m^{\prime}-2l-\mu)^{2}]{\tau}\\}$

Finally letting $m^{\prime}=k+l$, we conclude that

$\displaystyle\Delta=\sum_{\mu=0,1}\left[\sum_{k\in\mathbb{Z}}4\pi
i\alpha[k-\mu/2]\exp\\{-2\pi
i(k-\mu/2)(\varepsilon^{\prime}-\varepsilon)-2\pi{\tau}(k-\mu/2)^{2}\\}\right]$
$\displaystyle\ \ \ \ \ \ \ \ \times\left[\sum_{l\in\mathbb{Z}}\exp\\{2\pi
i(l+\mu/2)(2\xi+\varepsilon^{\prime}+\varepsilon)-2\pi{\tau}(l+\mu/2)^{2}\right]$
$\displaystyle=\left[\sum_{{\mu=0,1}}\partial_{x}\vartheta\left[\begin{matrix}\varepsilon^{\prime}-\varepsilon\\\
-\mu/2\end{matrix}\right](2{\xi},2{\tau})|_{{\xi}={0}}\right]\vartheta\left[\begin{matrix}\varepsilon^{\prime}+\varepsilon\\\
\mu/2\end{matrix}\right](2{\xi},2{\tau}),$

by using the following relations

$\displaystyle k+l=(k-\mu/2)+(l+\mu/2),\ \ k-l-\mu=(k-\mu/2)-(l+\mu/2).$

In a similar way, we can prove the formula (2.7). The formula (2.8) follows
from (2.6) and (2.7). $\Box$

Remark 1. The formulae (2.8) and (2.9) show that if the following equations
are satisfied

$None$ $C({\varepsilon},{\varepsilon^{\prime}},{\mu})=0,$

for $\mu=0,1$, then $\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})$ and
$\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})$ are periodic wave solutions of the
bilinear equation

$F(D_{x},D_{t},D_{n})\vartheta\left[\begin{matrix}\varepsilon^{\prime}\\\
0\end{matrix}\right]({\xi},{\tau})\cdot\vartheta\left[\begin{matrix}\varepsilon\\\
0\end{matrix}\right]({\xi},{\tau})=0.$

The formula (2.10) contains two equations which are called constraint
equations. This formula actually provides us an unified approach to construct
double periodic wave solutions for both differential and difference equations.
Once a equation is written bilinear forms, then its periodic wave solutions
can be directly obtained by solving system (2.10).

Theorem 2. Let $C({\varepsilon},{\varepsilon^{\prime}},{\mu})$ and
$F(D_{x},D_{t},D_{n})$ be given in Theorem 1, and make a choice such that
$\varepsilon^{\prime}-\varepsilon=\pm 1/2$. Then

(i) If $F(D_{x},D_{t},D_{n})$ is an even function in the form

$F(-D_{x},-D_{t},-D_{n})=F(D_{x},D_{t},D_{n}),$

then $C({\varepsilon},{\varepsilon^{\prime}},{\mu})$ vanishes automatically
for the case $\mu=1$, namely

$None$ $C({\varepsilon},{\varepsilon^{\prime}},{\mu})=0,\ \ {\rm for}\ \ \
\mu=1.$

(ii) If $F(D_{x},D_{t},D_{n})$ is an odd function in the form

$F(-D_{x},-D_{t},-D_{n})=-F(D_{x},D_{t},D_{n}),$

then $C({\varepsilon},{\varepsilon^{\prime}},{\mu})$ vanishes automatically
for the case $\mu=0$, namely

$None$ $C({\varepsilon},{\varepsilon^{\prime}},{\mu})=0,\ \ {\rm for}\ \mu=0.$

Proof. We are going to consider the case where $F(D_{x},D_{t},D_{n})$ is an
even function and prove the formula (2.11). The formula (2.12) is analogous.
Making transformation ${m}=-{\bar{m}}+{\mu}$, and noting
$F(D_{x},D_{t},D_{n})$ is even, we then deduce that

$\displaystyle
C({\varepsilon},{\varepsilon^{\prime}},{\mu})=\sum_{{\bar{m}}\in\mathbb{Z}}F(-{\mathcal{M}})\exp\left[-2\pi{\tau}({\bar{m}}-{\mu}/2)^{2}+2\pi
i({\bar{m}}-{\mu}/2)({\varepsilon^{\prime}}-{\varepsilon})\right]$
$\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \
=C({\varepsilon},{\varepsilon^{\prime}},{\mu})\exp\left[4\pi
i({\bar{m}}-{\mu}/2)({\varepsilon^{\prime}}-{\varepsilon})\right]$
$\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \
=C({\varepsilon},{\varepsilon^{\prime}},{\mu})\exp\left(\pm 2\pi
i\bar{m}\right)\exp\left(\pm\pi
i\mu\right)=-C({\varepsilon},{\varepsilon^{\prime}},{\mu}),$

which proves the formula (2.11). $\square$

Corollary 1.  Let $\varepsilon_{j}^{\prime}-\varepsilon_{j}=\pm 1/2,\
j=1,\cdots,N$. Assume $F(D_{x},D_{t},D_{n})$ is a linear combination of even
and odd functions

$F(D_{x},D_{t},D_{n})=F_{1}(D_{x},D_{t},D_{n})+F_{2}(D_{x},D_{t},D_{n}),$

where $F_{1}(D_{x},D_{t},D_{n})$ is even and $F_{2}(D_{x},D_{t},D_{n})$ is
odd. In addition, $C({\varepsilon},{\varepsilon^{\prime}},{\mu})$
corresponding (2.9) is given by

$C({\varepsilon},{\varepsilon^{\prime}},{\mu})=C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu})+C_{2}({\varepsilon},{\varepsilon^{\prime}},{\mu}),$

where

$C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu})=\sum_{{m}\in\mathbb{Z}^{N}}F_{1}({\mathcal{M}})\exp\left[-2\pi{\tau}({m}-{\mu}/2)^{2}-2\pi
i({m}-{\mu}/2)({\varepsilon^{\prime}}-{\varepsilon})\right],$
$C_{2}({\varepsilon},{\varepsilon^{\prime}},{\mu})=\sum_{{m}\in\mathbb{Z}^{N}}F_{2}({\mathcal{M}})\exp\left[-2\pi{\tau}({m}-{\mu}/2)^{2}-2\pi
i({m}-{\mu}/2)({\varepsilon^{\prime}}-{\varepsilon})\right].$

Then

$None$ $\displaystyle
C({\varepsilon},{\varepsilon^{\prime}},{\mu})=C_{2}({\varepsilon},{\varepsilon^{\prime}},{\mu})\
\ {\rm for}\ \ \ \mu=1,$ $None$ $\displaystyle
C({\varepsilon},{\varepsilon^{\prime}},{\mu})=C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu}),\
\ {\rm for}\ \mu=0.$

Proof. In a similar to the proof of Theorem 2, shifting sum index as
${m}=-{\bar{m}}+{\mu}$, and using $F_{1}(D_{x},D_{t},D_{n})$ even and
$F_{2}(D_{x},D_{t},D_{n})$ odd, we have

$None$ $\displaystyle
C({\varepsilon},{\varepsilon^{\prime}},{\mu})=C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu})+C_{2}({\varepsilon},{\varepsilon^{\prime}},{\mu})$
$\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \
=\left[C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu})-C_{2}({\varepsilon},{\varepsilon^{\prime}},{\mu})\right]\exp\left(\pm\pi
i\mu\right).$

Then for $\mu=1$, the equation (2.15) gives

$C_{1}({\varepsilon},{\varepsilon^{\prime}},{\mu})=0,$

which implies the formula (2.13). The formula (2.14) is is analogous.
$\square$

The theorem 2 and corollary 1 are very useful to deal with coupled Hirota’s
bilinear equations, which will be seen in the following section 4.
3\. The shall water wave equation

The shall water wave equation takes the form [17]

$None$ $\displaystyle
u_{t}-u_{xxt}-3uu_{t}+3u_{x}\int_{x}^{\infty}u_{t}dx+u_{x}=0,$

which is like to the KdV equation in the family of shall water wave equations.
Hirota and Satsuma obtained soliton solutions of the equation by means of
bilinear method [18]. Here we construct its a double periodic wave solution
and show that the one-soliton solution can be obtained as limiting case of the
double periodic solution.

To apply the Hirota bilinear method for constructing double periodic wave
solutions of the equation (3.1), we consider a variable transformation

$None$ $u=2\partial_{x}^{2}\ln f(x,t).$

Substituting (3.2) into (3.1) and integrating with respect to $x$, we then get
the following Hirota’s bilinear form

$None$ $\displaystyle F(D_{x},D_{t})f\cdot
f=(D_{x}D_{t}+D_{x}^{2}-D_{t}D_{x}^{3}+c)f\cdot f=0,$

where $c$ is an integration constant. In the special case of $c=0$, starting
from the bilinear equation (3.3), it is easy to find its one-soliton solution

$None$ $u_{1}=2\partial_{x}^{2}\ln(1+e^{\eta}),$

with phase variable $\eta=px+\frac{p}{p^{2}-1}t+\gamma$ for every $p$ and
$\gamma$.

Next, we turn to see the periodicity of the solution (3.2), the function $f$
is chosen to be a Riemann theta function, namely,

$None$ $f(x,t)=\vartheta({\xi},{\tau}),$

where phase variable $\xi=\alpha x+\omega t+\sigma.$ With Proposition 3, we
refer to

$None$
$u=2\partial_{x}^{2}\ln\vartheta({\xi},{\tau})=2\alpha^{2}\partial_{\xi}^{2}\ln\vartheta({\xi},{\tau}),$

which shows that the solution $u$ is a double periodic function with two
fundamental periods $1$ and $i{\tau}$.

We introduce the notations by

$None$
$\displaystyle\lambda=e^{-\pi\tau/2},\quad\vartheta_{1}(\xi,\lambda)=\vartheta(2\mathbf{\xi},2\tau)=\sum_{m\in\mathbb{Z}}\lambda^{4m^{2}}\exp(4i\pi
m\xi),$
$\displaystyle\vartheta_{2}(\xi,\lambda)=\vartheta\left[\begin{matrix}0\\\
-1/2\end{matrix}\right](2\mathbf{\xi},2\tau)=\sum_{m\in\mathbb{Z}}\lambda^{(2m-1)^{2}}\exp[2i\pi(2m-1)\xi],$

where the phase variable $\xi=\alpha x+\omega t+\sigma$.

Substituting (3.5) into (3.3), using formula (2.10) and (3.7) leads to a
linear system ( corresponding to $\mu=0$ and $\mu=1$, respectively)

$None$
$\displaystyle[\vartheta_{1}^{\prime\prime}(0,\lambda)\alpha+\vartheta_{1}^{(4)}(0,\lambda)\alpha^{4}]\omega+\vartheta_{1}(0,\lambda)c+\vartheta_{1}^{\prime\prime}(0,\lambda)\alpha^{2}=0,$
$\displaystyle[\vartheta_{2}^{\prime\prime}(0,\lambda)\alpha+\vartheta_{2}^{(4)}(0,\lambda)\alpha^{4}]\omega+\vartheta_{2}(0,\lambda)c+\vartheta_{2}^{\prime\prime}(0,\lambda)\alpha^{2}=0,$

where we have denoted the derivative of $\vartheta_{j}(\xi,\lambda)$ at
$\xi=0$ by notations

$\vartheta_{j}^{(k)}(0,\lambda)=\frac{d^{k}\vartheta_{j}(\xi,\lambda)}{d\xi^{k}}|_{\xi=0},\
\ j=1,2;k=1,2,3,4.$

This system admits an explicit solution $(\omega,c)$. In this way, we obtain
an explicit periodic wave solution (3.6) with parameters $\omega$, $c$ by
(3.8), while other parameters $\alpha,\sigma,\tau,\sigma$ are free.

In summary, double periodic wave (3.6) possesses the following features: (i)
It is is one-dimensional, i.e. there is a single phase variable $\xi$.
Moreover, it has two fundamental periods $1$ and $i\tau$ in phase variable
$\xi$, but it need not to be periodic in $x$ and $t$ . (ii) It can be viewed
as a parallel superposition of overlapping one-soliton waves, placed one
period apart.

In the following, we further consider asymptotic properties of the periodic
wave solution. Interestingly, the relation between the periodic wave solution
(3.6) and the one-soliton solution (3.4) can be established as follows.

Theorem 3. Suppose that the vector $(\omega,c)$ is a solution of the system
(3.8), and for the periodic wave solution (3.6), we let

$None$ $\alpha=\frac{p}{2\pi i},\ \ \sigma=\frac{\gamma+\pi\tau}{2\pi i},$

where the $p$ and $\gamma$ are given in (3.4). Then we have the following
asymptotic properties

$\displaystyle c\longrightarrow 0,\ \ 2\pi
i\xi-\pi\tau\longrightarrow\eta=px+\frac{p}{p^{2}-1}t+\gamma,$
$\displaystyle\vartheta(\xi,\tau)\longrightarrow 1+e^{\eta},\ \ {\rm as}\ \
\lambda\rightarrow 0.$

In other words, the double periodic solution (3.10) tends to the soliton
solution (3.4) under a small amplitude limit, that is,

$None$ $u\longrightarrow u_{1},\ \ {\rm as}\ \ \lambda\rightarrow 0.$

Proof. Here we will directly use the system (3.8) to analyze asymptotic
properties of the periodic solution (3.6). Since the coefficients of system
(3.8) are power series about $\lambda$, its solution $(\omega,c)$ also should
be a series about $\lambda$. We explicitly expand the coefficients of system
(3.8) as follows

$None$
$\displaystyle\vartheta_{1}(0,\lambda)=1+2\lambda^{4}+\cdots,\quad\vartheta_{1}^{\prime\prime}(0,\lambda)=-32\pi^{2}\lambda^{4}+\cdots,$
$\displaystyle\vartheta_{1}^{(4)}(0,\lambda)=512\pi^{4}\lambda^{4}+\cdots,\ \
\vartheta_{2}(0,\lambda)=2\lambda+2\lambda^{9}+\cdots$
$\displaystyle\vartheta_{2}^{\prime\prime}(0,\lambda)=-8\pi^{2}\lambda+\cdots,\
\vartheta_{2}^{(4)}(0,\lambda)=32\pi^{4}\lambda+\cdots.$

Let the solution of the system (3.8) be in the form

$None$
$\displaystyle\omega=\omega_{0}+\omega_{1}\lambda+\omega_{2}\lambda^{2}+\cdots=\omega_{0}+o(\lambda),$
$\displaystyle c=c_{0}+c_{1}\lambda+c_{2}\lambda^{2}+\cdots=c_{0}+o(\lambda).$

Substituting the expansions (3.11) and (3.12) into the system (3.8) (the
second equation is divided by $\lambda$ ) and letting $\lambda\longrightarrow
0$, we immediately obtain the following relations

$\displaystyle c_{0}=0,\ \
(-8\pi^{2}\alpha+32\pi^{4}\alpha^{3})\omega_{0}-8\pi^{2}\alpha^{2}=0,$

which implies

$None$ $c_{0}=0,\ \ w_{0}=\frac{\alpha}{4\pi^{2}\alpha^{2}-1}.$

Combining (3.12) and (3.13) then yields

$c\longrightarrow 0,\ \ 2\pi i\omega\longrightarrow\frac{2\pi i\alpha}{(2\pi
i\alpha)^{2}-1}=\frac{p}{p^{2}-1},\ \ {\rm as}\ \ \lambda\rightarrow 0.$

Hence we conclude

$None$ $\displaystyle\hat{\xi}=2\pi i\xi-\pi\tau=px+2\pi i\omega t+\gamma$
$\displaystyle\quad\longrightarrow px+\frac{p}{p^{2}-1}t+\gamma=\eta,\ \ {\rm
as}\ \ \lambda\rightarrow 0.$

It remains to consider asymptotic properties of the periodic wave solution
(3.6) under the limit $\lambda\rightarrow 0$. By expanding the Riemann theta
function $\vartheta(\xi,\tau)$ and using (3.14), it follows that

$\displaystyle\vartheta(\xi,\tau)=1+\lambda^{2}(e^{2\pi i\xi}+e^{-2\pi
i\xi})+\lambda^{8}(e^{4\pi i\xi}+e^{-4\pi i\xi})+\cdots$ $\displaystyle\ \ \ \
\
=1+e^{\hat{\xi}}+\lambda^{4}(e^{-\hat{\xi}}+e^{2\hat{\xi}})+\lambda^{12}(e^{-2\hat{\xi}}+e^{3\hat{\xi}})+\cdots$
$\displaystyle\ \ \ \ \ \quad\longrightarrow 1+e^{\hat{\xi}}\longrightarrow
1+e^{\eta},\ \ {\rm as}\ \ \lambda\rightarrow 0,$

which together with (3.6) leads to (3.10). Therefore we conclude that the
double periodic solution (3.6) just goes to the one-soliton solution (3.4) as
the amplitude $\lambda\rightarrow 0$. $\square$
4\. The modified Bogoyavlenskii-Schiff equation

We consider (2+1)-dimensional modified Bogoyavlenskii-Schiff equation [19]

$None$ $\displaystyle
u_{t}-4u^{2}u_{z}-2u_{x}\partial_{x}^{-1}(u^{2})_{z}+u_{xxz}=0,$

which was deduced from the Miura transformation [20]. Equation (4.1) is
reduced to the modified KdV equation in the case of $x=z$.

We shall construct a double periodic wave solution to the equation (4.1) by
using Theorem 1 and 2. The equation (4.1) can be described by a coupled system

$None$ $\displaystyle u=\psi_{x},$ $\displaystyle\rho_{xx}+\psi_{x}^{2}+c=0,$
$\displaystyle\psi_{t}+2\psi_{x}\rho_{xz}+\psi_{z}(\rho_{xx}+\psi_{x}^{2}+c)+\psi_{xxz}=0.$

We perform the dependent variable transformations

$None$ $\displaystyle u=\psi_{x}=\partial_{x}\ln\left(\frac{f}{g}\right),\ \
\rho=\ln(fg),$

then equation (4.2) is reduced to the following bilinear form

$None$ $\displaystyle F(D_{x})f\cdot g=(D_{x}^{2}+c)f\cdot g=0,$
$\displaystyle G(D_{t},D_{x},D_{z})f\cdot
g=(D_{t}+D_{x}^{2}D_{z}+cD_{z})f\cdot g=0,$

where $c$ is a constant. The equation (4.4) is a type of coupled bilinear
equations, which is more difficult to be dealt with than the single bilinear
equation (3.3) due to appearance of two functions and two equations. We will
take full advantages of Theorem 2 to reduce the number of constraint
equations.

Now we take into account the periodicity of the solution (4.3), in which we
take $f$ and $g$ as

$None$ $f=\vartheta({\xi}+e,{\tau}),\ \ g=\vartheta({\xi}+h,{\tau}),\ \
{e},{h}\in\mathbb{C},$

where phase variable $\xi=\alpha x+\beta z+\omega t+\sigma.$ By means of
Proposition 3, we find that the solution

$\displaystyle
u=\alpha\partial_{\xi}\ln\frac{\vartheta({\xi}+e,{\tau})}{\vartheta({\xi}+h,{\tau})}$

is a double periodic function with two fundamental periods $1$ and $i\tau$.

In the special case of $c=0$, the equation (4.2) admits one-soliton solution

$None$ $u_{1}=\partial_{x}\ln\frac{1+e^{\eta}}{1-e^{\eta}},$

where $\eta=px+qy-p^{2}qt+\gamma$ for every $p,q$ and $\gamma$.

We take $e=0,\ h=1/2$ in (4.5), and therefore

$None$ $\displaystyle f=\vartheta(\xi,\tau)=\sum_{m\in\mathbb{Z}}\exp({2\pi
in\xi-\pi m^{2}\tau}),$ $\displaystyle g=\vartheta\left[\begin{matrix}1/2\\\
0\end{matrix}\right](\xi,\tau)=\sum_{m\in\mathbb{Z}}\exp({2\pi im(\xi+1/2)-\pi
m^{2}\tau})$ $\displaystyle\ \ \ =\sum_{m\in\mathbb{Z}}(-1)^{m}\exp({2\pi
im\xi-\pi m^{2}\tau}).$

Due to the fact that $F(D_{x})$ is an even function, its constraint equations
in the formula (2.10) vanish automatically for $\mu=1$. Similarly the
constraint equations associated with $G(D_{t},D_{x},D_{z})$ also vanish
automatically for $\mu=0$. Therefore, the Riemann theta function (4.6) is a
solution of the bilinear equation (4.4), provided the following equations

$None$
$\displaystyle\vartheta_{1}^{\prime\prime}(0,\lambda)\alpha^{2}+\vartheta_{1}(0,\lambda)c=0,$
$\displaystyle\vartheta_{2}^{\prime}(0,\lambda)\omega+\vartheta_{2}^{\prime}(0,\lambda)\beta
c+\vartheta_{2}^{\prime\prime\prime}(0,\lambda)\alpha^{2}\beta=0,$

where we introduce the notations by

$\displaystyle\lambda=e^{-\pi\tau/2},\quad\vartheta_{1}(\xi,\lambda)=\vartheta(2\mathbf{\xi},2\tau)=\sum_{m\in\mathbb{Z}}\lambda^{4m^{2}}\exp(4i\pi
m\xi),$
$\displaystyle\vartheta_{2}(\xi,\lambda)=\vartheta\left[\begin{matrix}1/2\\\
-1/2\end{matrix}\right](2\mathbf{\xi},2\tau)=\sum_{m\in\mathbb{Z}}(-1)^{m}\lambda^{(2m-1)^{2}}\exp[2i\pi(2m-1)\xi].$

It is obvious that equation (4.8) admits an explicit solution $\omega$ and
$c$. In this way, a periodic wave solution reads

$None$ $u=\partial_{x}\ln\frac{\vartheta(\xi,\tau)}{\vartheta(\xi+1/2,\tau)},$

where parameters $\omega$ and $c$ are given by (4.11), while other parameters
$\alpha,\beta,\tau,\sigma$ are free. In summary, double periodic wave (4.9)
has the following features: (i) It is one-dimensional and has two fundamental
periods $1$ and $i\tau$ in phase variable $\xi$. (ii) It can be viewed as a
parallel superposition of overlapping one-soliton waves, placed one period
apart.

In the following, we further consider asymptotic properties of the double
periodic wave solution. The relation between the periodic wave solution (4.9)
and the one-soliton solution (4.6) can be established as follows.

Theorem 4. Suppose that the vector $(\omega,c)^{T}$ is a solution of the
system (4.8). In the periodic wave solution (4.9), we choose parameters as

$None$ $\alpha=\frac{p}{2\pi i},\ \ \beta=\frac{q}{2\pi i},\ \
\sigma=\frac{\gamma+\pi\tau}{2\pi i},$

where the $p,q$ and $\gamma$ are the same as those in (4.6). Then we have the
following asymptotic properties

$c\longrightarrow 0,\ \ \xi\longrightarrow\frac{\eta+\pi\tau}{2\pi i},\ \
f\longrightarrow 1+e^{\eta},\ \ g\longrightarrow 1-e^{\eta},\ \ {\rm as}\ \
\lambda\rightarrow 0.$

In other words, the double periodic solution (4.9) tends to the one-soliton
solution (4.6) under a small amplitude limit , that is,

$None$ $u\longrightarrow u_{1},\ \ {\rm as}\ \ \lambda\rightarrow 0.$

Proof. Here we will directly use the system (4.8) to analyze asymptotic
properties of periodic solution (4.9). We explicitly expand the coefficients
of system (4.8) as follows

$None$
$\displaystyle\vartheta_{1}(0,\lambda)=1+2\lambda^{4}+\cdots,\quad\vartheta_{1}^{\prime\prime}(0,\lambda)=-32\pi^{2}\lambda^{4}+\cdots,$
$\displaystyle\vartheta_{2}^{\prime}(0,\lambda)=-4\pi i\lambda+12\pi
i\lambda^{9}+\cdots,\ \
\vartheta_{2}^{\prime\prime\prime}(0,\lambda)=16\pi^{3}i\lambda-48\pi^{3}i\lambda^{9}+\cdots,$

Suppose that the solution of the system (4.8) is of the form

$None$
$\displaystyle\omega=\omega_{0}+\omega_{1}\lambda+\omega_{2}\lambda^{2}+\cdots=\omega_{0}+o(\lambda),$
$\displaystyle c=c_{0}+c_{1}\lambda+c_{2}\lambda^{2}+\cdots=c_{0}+o(\lambda).$

Substituting the expansions (4.12) and (4.13) into the system (4.8) and
letting $\lambda\longrightarrow 0$, we immediately obtain the following
relations

$\displaystyle c_{0}=0,\ \ -4\pi i\omega_{0}+16\pi^{3}i\alpha^{2}\beta=0,\ \ $

which has a solution

$None$ $c_{0}=0,\ \ w_{0}=4\pi^{2}\alpha^{2}\beta.$

Combining (4.13) and (4.14) leads to

$c\longrightarrow 0,\ \ 2\pi i\omega\longrightarrow
8\pi^{3}i\alpha^{2}\beta=-p^{2}q,\ \ {\rm as}\ \ \lambda\rightarrow 0,$

or equivalently

$None$ $\displaystyle\hat{\xi}=2\pi i\xi-\pi\tau=px+qy+2\pi i\omega t+\gamma$
$\displaystyle\quad\longrightarrow px+qy-p^{2}qt+\gamma=\eta,\ \ {\rm as}\ \
\lambda\rightarrow 0.$

It remains to identify that the periodic wave (4.9) possesses the same form
with the one-soliton solution (4.6) under the limit $\lambda\rightarrow 0$.
For this purpose, we start to expand the functions $f$ and $g$ in the form

$f=1+\lambda^{2}(e^{2\pi i\xi}+e^{-2\pi i\xi})+\lambda^{8}(e^{4\pi
i\xi}+e^{-4\pi i\xi})+\cdots.$ $g=1-\lambda^{2}(e^{2\pi i\xi}+e^{-2\pi
i\xi})+\lambda^{8}(e^{4\pi i\xi}+e^{-4\pi i\xi})+\cdots.$

By using (4.13)-(4.15), it follows that

$None$ $\displaystyle
f=1+e^{\hat{\xi}}+\lambda^{4}(e^{-\hat{\xi}}+e^{2\hat{\xi}})+\lambda^{12}(e^{-2\hat{\xi}}+e^{3\hat{\xi}})+\cdots$
$\displaystyle\quad\longrightarrow 1+e^{\hat{\xi}}\longrightarrow 1+e^{\eta},\
\ {\rm as}\ \ \lambda\rightarrow 0;$ $\displaystyle
g=1-e^{\hat{\xi}}+\lambda^{4}(e^{2\hat{\xi}}-e^{-\hat{\xi}})+\lambda^{12}(e^{-2\hat{\xi}}-e^{3\hat{\xi}})+\cdots$
$\displaystyle\quad\longrightarrow 1-e^{\hat{\xi}}\longrightarrow 1-e^{\eta},\
\ {\rm as}\ \ \lambda\rightarrow 0.$

The expression (4.11) follows from (4.16), and thus we conclude that the
double periodic solution (4.9) just goes to the one-soliton solution (4.6) as
the amplitude $\lambda\rightarrow 0$. $\square$
5\. The differential-difference KdV equation

We consider differential-difference KdV equation

$None$
$\displaystyle\frac{d}{dt}\left(\frac{u(n)}{1+u(n)}\right)=u(n-1/2)-u(n+1/2).$

Hirota and Hu have found its soliton solutions and rational solutions [21,
22], among them one-soliton solution reads

$None$ $u_{1}(n)=\frac{(1+e^{\eta+p/2})(1+e^{\eta-p/2})}{(1+e^{\eta})^{2}}-1,$

where $\eta=pn-\sinh(p/2)t+\gamma$ for every $p$ and $\gamma$.

We shall construct a periodic wave solutions to the equation (5.1) by using
Theorem 1. By means of a variable transformation

$None$ $\displaystyle u(n)=\frac{f({n+1/2})f({n-1/2})}{f(n)^{2}}-1,$

the equation (5.1) is reduced to the bilinear equation

$None$
$\displaystyle\left[\sinh(\frac{1}{4}D_{n})D_{t}+2\sinh(\frac{1}{4}D_{n})\sinh(\frac{1}{2}D_{n})+c\right]f(n)\cdot
f(n)=0,$

where $c$ is a constant.

Now we take into account the periodicity of the solution (5.3), in which we
take $f(n)=\vartheta({\xi},{\tau}),$ where phase variable $\xi=\nu n+\omega
t+\sigma.$ Then solution (5.3) is written as

$None$ $\displaystyle u(\xi)\equiv
u(n)=\frac{\vartheta({\xi}+\frac{1}{2}\nu,{\tau})\vartheta({\xi}-\frac{1}{2}\nu,{\tau})}{\vartheta({\xi},{\tau})^{2}}-1.$

By means of Proposition 2, it is easy to deduce that $u_{n}$ is a double
periodic function with two fundamental periods $1$ and $i\tau$.

Substituting (5.5) into (5.4) and using formula (2.10) leads to a linear
system

$None$
$\displaystyle\sinh(\frac{1}{4}D_{n})\vartheta_{1}^{\prime}(0,\lambda)\omega+\vartheta_{1}(0,\lambda)c+\sinh(\frac{1}{4}D_{n})\sinh(\frac{1}{2}D_{n})\vartheta_{1}(0,\lambda)=0,$
$\displaystyle\sinh(\frac{1}{4}D_{n})\vartheta_{2}^{\prime}(0,\lambda)\omega+\vartheta_{2}(0,\lambda)c+\sinh(\frac{1}{4}D_{n})\sinh(\frac{1}{2}D_{n})\vartheta_{2}(0,\lambda)=0,$

where $\vartheta_{1}(\xi,\lambda)$ and $\vartheta_{2}(\xi,\lambda)$ are the
same as those in (3.7) with $\xi=\nu n+\omega t+\sigma.$ By using the solution
$\omega$ and $c$ of system (5.6), a periodic wave solution is obtained by
(5.5).

In the following, we further consider asymptotic properties of the double
periodic wave solution. The relation between the periodic wave solution (5.5)
and the one-soliton solution (5.2) can be established as follows.

Theorem 5. Suppose that the vector $(\omega,c)^{T}$ is a solution of the
system (5.6). In the periodic wave solution (5.5), we choose parameters as

$None$ $\nu=\frac{p}{2\pi i},\ \ \sigma=\frac{\gamma+\pi\tau}{2\pi i},$

where the $p$ and $\gamma$ are the same as those in (5.2). Then we have the
following asymptotic properties

$c\longrightarrow 0,\ \ \xi\longrightarrow\frac{\eta+\pi\tau}{2\pi i},\ \
\vartheta(\xi,\tau)\longrightarrow 1+e^{\eta},\ \ {\rm as}\ \
\lambda\rightarrow 0.$

In other words, the periodic solution (5.5) tends to the one-soliton solution
(5.2) under a small amplitude limit , that is,

$None$ $u(n)\longrightarrow u_{1}(n),\ \ {\rm as}\ \ \lambda\rightarrow 0.$

Proof. Here we will directly use the system (5.6) to analyze asymptotic
properties of periodic solution (5.5). We explicitly expand the coefficients
of system (5.6) as follows

$None$
$\displaystyle\vartheta_{1}(0,\lambda)=1+2\lambda^{4}+\cdots,\quad\sinh(\frac{1}{4}D_{n})\vartheta_{1}^{\prime}(0,\lambda)=8\pi
i\sinh(i\pi\nu)\lambda^{4}+\cdots,$
$\displaystyle\sinh(\frac{1}{4}D_{n})\sinh(\frac{1}{2}D_{n})\vartheta_{1}(0,\lambda)=2\sinh(i\pi\nu)\sinh(2i\pi\nu)\lambda^{4}+\cdots,$
$\displaystyle\vartheta_{2}(0,\lambda)=2\lambda+2\lambda^{9}+\cdots,\ \
\sinh(\frac{1}{4}D_{n})\vartheta_{2}^{\prime}(0,\lambda)=4\pi
i\sinh(i\pi\nu/2)\lambda+\cdots,$
$\displaystyle\sinh(\frac{1}{4}D_{n})\sinh(\frac{1}{2}D_{n})\vartheta_{2}(0,\lambda)=2\sinh(i\pi\nu)\sinh(i\pi\nu/2)\lambda+\cdots.$

Suppose that the solution of the system (5.6) is of the form

$None$
$\displaystyle\omega=\omega_{0}+\omega_{1}\lambda+\omega_{2}\lambda^{2}+\cdots=\omega_{0}+o(\lambda),$
$\displaystyle c=c_{0}+c_{1}\lambda+c_{2}\lambda^{2}+\cdots=c_{0}+o(\lambda).$

Substituting the expansions (5.9) and (5.10) into the system (5.6) and letting
$\lambda\longrightarrow 0$, we immediately obtain the following relations

$\displaystyle c_{0}=0,\ \ 4\pi
i\sinh(i\pi\nu/2)\omega_{0}+2\sinh(i\pi\nu/2)\sinh(i\pi\nu)=0,\ \ $

which implies

$None$ $c_{0}=0,\ \ w_{0}=-\frac{1}{2\pi i}\sinh(i\pi\nu).$

Combining (5.9) and (5.10) leads to

$c\longrightarrow 0,\ \ 2\pi
i\omega\longrightarrow-\sinh(i\pi\nu)=-\sinh(p/2),\ \ {\rm as}\ \
\lambda\rightarrow 0,$

or equivalently

$None$ $\displaystyle\hat{\xi}=2\pi i\xi-\pi\tau=pn+2\pi i\omega t+\gamma$
$\displaystyle\quad\longrightarrow pn-\sinh(p/2)t+\gamma=\eta,\ \ {\rm as}\ \
\lambda\rightarrow 0.$

It remains to consider asymptotic properties of the periodic wave solution
(5.5) under the limit $\lambda\rightarrow 0$. By expanding the Riemann theta
function $\vartheta(\xi,\tau)$, it follows that

$\displaystyle\vartheta(\xi,\tau)=1+e^{\hat{\xi}}+\lambda^{4}(e^{-\hat{\xi}}+e^{2\hat{\xi}})+\lambda^{12}(e^{-2\hat{\xi}}+e^{3\hat{\xi}})+\cdots$
$\displaystyle\ \ \ \ \ \quad\longrightarrow 1+e^{\hat{\xi}}\longrightarrow
1+e^{\eta},\ \ {\rm as}\ \ \lambda\rightarrow 0,$

which together with (5.5) lead to (5.8). Therefore we conclude that the
periodic solution (5.5) just goes to the one-soliton solution (5.2) as the
amplitude $\lambda\rightarrow 0$. $\square$
Acknowledgment

The work described in this paper was supported by grants from Research Grants
Council contracts HKU, the National Science Foundation of China (No.10971031),
Shanghai Shuguang Tracking Project (No.08GG01) and Innovation Program of
Shanghai Municipal Education Commission (No.10ZZ131).

## References

  * [1]
  * [2] R. Hirota and J. Satsuma: Prog. Theor. Phys. 57 (1977) 797.
  * [3] R. Hirota: Direct methods in soliton theory (Springer-verlag, Berlin, 2004).
  * [4] X. B. Hu, C X Li, J. J. C. Nimmo and G. F. Yu, J. Phys. A, 38 (2005) 195.
  * [5] R Hirota and Y. Ohta: J. Phys. Soc. Jpn. 60 (1991) 798\.
  * [6] K W Chow, C K Lam, K Nakkeeran and B Malmed. J. Phys. Soc. Jpn. 77(2008), 054001
  * [7] K. Sawada and T Kotera: Prog. Theor. Phys. 51 (1974) 1355.
  * [8] A. Nakamura, J. Phys. Soc. Jpn. 47, 1701-1705 (1979).
  * [9] A. Nakamura, J. Phys. Soc. Jpn. 48(1980), 1365.
  * [10] H. H. Dai, E. G. Fan and X. G. Geng, arxiv.org/pdf/nlin/0602015
  * [11] Y. Zhang, L. Y. Ye, Y. N. Lv and H. Q. Zhao, J. Phys A, 40 (2007), 5539.
  * [12] Y. C. Hon, E. G. Fan and Z. Y. Qin, Modern Phys Lett B, 22 (2008), 547.
  * [13] E. G. Fan and Y. C. Hon, Phys Rev E, 78 (2008), 036607.
  * [14] E. G. Fan, J. Phys A, 42 (2009), 095206.
  * [15] W. X. Ma, R. G. Zhou, J. Math. Phys, 24 (2009), 1677.
  * [16] H. M. Farkas and I. Kra, Riemann Surfaces, New York, Springer-Verlag, 1992.
  * [17] P. A. Clarkson and E. L. Mansfield, Nonlinearity, 7 (19994), 975.
  * [18] R. Hirota and J. Satsuma, J. Phys. Soc. Jpn. 40 (1976), 611.
  * [19] S. J. Yu, K. Toda, N. Sasa and T. Fukuyama, J. Phys A, 31 (1998), 3337.
  * [20] O. I. Bogoyavenskii, Math. USSR Izv. 36 (1991), 129.
  * [21] R Hirota and J. Satsuma, Progr. Theor. Phys. Suppl. 64 (1976), 64.
  * [22] X. B. Hu and P. A. Clarkson, J. Phys. A 28 (1995) 5009.


Ji Won Yoon1, Seok Min Kim1, Nam Soo Kim1

# MCR-Data2vec 2.0: Improving Self-supervised Speech Pre-training via Model-
level Consistency Regularization

###### Abstract

Self-supervised learning (SSL) has shown significant progress in speech
processing tasks. However, despite the intrinsic randomness in the Transformer
structure, such as dropout variants and layer-drop, improving the model-level
consistency remains under-explored in the speech SSL literature. To address
this, we propose a new pre-training method that uses consistency
regularization to improve Data2vec 2.0, the recent state-of-the-art (SOTA) SSL
model. Specifically, the proposed method involves sampling two different
student sub-models within the Data2vec 2.0 framework, enabling two output
variants derived from a single input without additional parameters.
Subsequently, we regularize the outputs from the student sub-models to be
consistent and require them to predict the representation of the teacher
model. Our experimental results demonstrate that the proposed approach
improves the SSL model's robustness and generalization ability, resulting in
SOTA results on the SUPERB benchmark.

Index Terms: Self-Supervised Learning, Speech Pre-Training, Consistency
Regularization

##  1 Introduction

Self-supervised learning (SSL) has shown great promise in speech processing
[1, 2, 3, 4]. It uses large amounts of unlabeled speech audio data to learn
general speech representations, which can benefit various downstream tasks by
fine-tuning.

Recently, research in speech SSL algorithms has focused on data augmentation
for a noisy scenario to improve the pre-training of SSL models [3, 5, 6, 7, 8,
9]. WavLM [3] employs a masked speech denoising and prediction framework to
pre-train speech representations. Some inputs are artificially simulated to be
noisy or overlapped with masks, and the model predicts pseudo-labels of the
original speech on the masked region. CCC-Wav2vec 2.0 [5] introduces an
augmentation of the original sample and uses its representations to add an
extra cross-contrastive loss to the Wav2vec 2.0 [2] framework. Robust Data2vec
[9] improves the noise robustness of the Data2vec [4] by allowing the model to
have consistent predictions for both original and noisy speech. These studies
mainly aim to make the SSL model more robust to augmentations, which in turn
helps learn better representations.

While augmentation-based approaches are certainly helpful to improve the SSL
model's robustness against the data variation, they may not consider
randomness in the model architecture. For example, most recent SSL models are
based on the Transformer [10] structure, which has intrinsic randomness due to
multiple dropout variants, such as standard dropout [11] for each module,
LayerDrop [12, 13], etc. During the pre-training stage, these dropout variants
randomly discard a portion of layers or neurons, selecting a random _sub-
model_ at each iteration. However, when fine-tuning on a downstream task, all
or a part of the pre-trained SSL model's parameters are commonly frozen [14,
15, 16, 17, 18, 19], causing inconsistency between the pre-training and fine-
tuning. For ease of understanding, assuming that we freeze all of the SSL
model's parameters for the downstream task, the pre-training is performed on a
_sub-model_ that includes sources of randomness. In contrast, the fine-tuning
is conducted on a single _full model_ without randomness. This difference in
the amount of randomness between the pre-training and fine-tuning can create a
gap between the two stages, leading to performance degradation when applying
the SSL model to downstream tasks.

To address this issue, one possible solution is to encourage the sub-models to
produce consistent outputs, regardless of the sources of randomness. If the
sub-models are less affected by the randomness, the gap between the sub-model
(pre-training) and the full model (fine-tuning) can be alleviated. Previous
studies [20, 21, 22, 23] have attempted to improve model-level consistency by
regularizing the output predictions of sub-models to be consistent. However,
since these techniques are mainly designed for supervised or semi-supervised
settings, it is difficult to directly apply them to fully unsupervised pre-
training. Therefore, it is necessary to design a new consistency
regularization framework that can be applied during the pre-training stage of
speech SSL.

In this paper, we introduce MCR-Data2vec 2.0, a new pre-training method for
improving the model-level consistency of Data2vec 2.0 [24], which is the
current state-of-the-art (SOTA) SSL model. Based on the teacher-student scheme
of the Data2vec 2.0 framework, the proposed method randomly samples two
different student sub-models, enabling two output variants derived from a
single input without additional parameters. Due to the dropout variants, the
two sub-models are based on different subsets of layers or neurons. Then, we
regularize the outputs of the student sub-models to be consistent with each
other and require them to predict the same target representation of the
teacher model. Regularizing the student sub-models to produce similar outputs
makes them less affected by randomness. Thus, MCR-Data2vec 2.0 can effectively
reduce the gap between the pre-training and fine-tuning stages, improving the
overall quality of the SSL model.

Figure 1: An overview of MCR-Data2vec 2.0. In the student encoder and decoder,
a crossed-out dotted circle represents dropped subsets of layers or neurons
due to the dropout variants. First, a contextualized representation $y_{n}$ is
generated from the teacher model based on the unmasked training sample
$x_{n}$. The teacher model's weights are a moving average of the student
model's weights. MCR-Data2vec 2.0 randomly samples two different student sub-
models, resulting in two predictions $f_{1}(x_{(n,m)})$ and $f_{2}(x_{(n,m)})$
for the same masked input $x_{(n,m)}$. The masked portions of the training
sample are not encoded [24]. To improve model-level consistency, the outputs
from the student sub-models are regularized to be consistent
($\mathcal{L}_{mcr}$), and the student sub-models predict the same
contextualized target representation for various masked versions of the
training example ($\mathcal{L}_{pred1}$ and $\mathcal{L}_{pred2}$).
$\mathcal{L}_{pred1}$ corresponds to the training objective of the original
Data2vec 2.0.

Through extensive experiments on SUPERB [25, 26], we confirm that the MCR-
Data2vec 2.0 effectively improves the SSL model's robustness and
generalization ability. It achieves SOTA performance on multiple subtasks of
SUPERB, including Phoneme Recognition (PR), Automatic Speech Recognition
(ASR), Keyword Spotting (KS), Intent Classification (IC), Slot Filling (SF),
Emotion Recognition (ER). Furthermore, MCR-Data2vec 2.0 yields considerable
performance improvements for speaker and generation-related tasks compared to
the original Data2vec 2.0.

To summarize, the main contributions of this paper are:

  * •

We propose MCR-Data2vec 2.0, a new consistency regularization framework
designed to improve the recent Data2vec 2.0. _To the best of our knowledge,
this is the first attempt to apply the model-level consistency regularization
during the pre-training of speech SSL._

  * •

By regularizing the outputs of student sub-models to be consistent, the
proposed method can reduce the gap between the pre-training and fine-tuning
stages.

  * •

Through experimental results on SUPERB, we validate that MCR-Data2vec 2.0
significantly improves the overall quality of Data2vec 2.0. Compared to recent
SSL models, our framework achieves promising results on various downstream
tasks.

##  2 Background: Data2vec 2.0

As aforementioned, we use Data2vec 2.0 [24] as our backbone model since it is
one of the most recent SOTA SSL models. Before we describe the proposed
method, it might be beneficial to review some properties of Data2vec 2.0.

###  2.1 Teacher-student Setup

Similar to Data2vec [4], Data2vec 2.0 follows the teacher-student scheme,
where the weights of the teacher $\Delta$ are an exponentially moving average
(EMA) of the student encoder $\theta$ [27]:
$\Delta\leftarrow\tau\Delta+(1-\tau)\theta$. The parameter $\tau$ linearly
increases from a starting value $\tau_{0}$ to a final value $\tau_{e}$ over
$\tau_{n}$ updates, after which the value is kept constant [4, 24].

###  2.2 Contextualized Target Prediction

As shown in Figure 1 (green highlight), Data2Vec 2.0 is based on multi-mask
training that considers $M$ different masked versions of the training sample,
similar to Masked Autoencoders (MAE) [28]. Given a training dataset
$\mathcal{D}=\\{x_{1},x_{2},...,x_{N}\\}$ where $N$ is the number of training
data samples, each sample $x_{n}$ is masked $M$ times. The $m$-th masked
version of sample $x_{n}$ is represented as $x_{(n,m)}$. $y_{n}$ denotes a
contextualized target representation from the teacher model based on the
original sample $x_{n}$.

In the Data2vec 2.0, the teacher model consumes the unmasked training sample
$x_{n}$ to produce the target representation $y_{n}$. Specifically, the output
of the top $K$ blocks of the teacher model is averaged to construct $y_{n}$.
This target representation is contextualized since the teacher model uses a
self-attention mechanism in the Transformer architecture [10]. Each masked
version of sample is fed into the student model, which predicts the same
target representation $y_{n}$ for the different masked versions. When passing
the masked sample $x_{(n,m)}$, the student does not encode masked tokens to
further improve the model efficiency. During the pre-training, Data2vec 2.0 is
trained to minimize the $l_{2}$ loss ($\mathcal{L}_{pred1}$ in Figure 1)
between the student's prediction and the contextualized target representation.

##  3 MCR-Data2vec 2.0

In this section, we introduce our MCR-Data2vec 2.0 framework and the training
algorithm. Based on Data2vec 2.0, we propose a new model-level consistency
regularization to improve the pre-training stage of SSL. The overall framework
of our regularization method is shown in Figure 1.

###  3.1 Student Sub-model Sampling

Unlike Data2vec 2.0, MCR-Data2vec 2.0 samples two different student sub-models
$f_{1}$ and $f_{2}$ from a full single student model $f$. This sub-model
sampling allows two output variants to be derived from a single input without
requiring additional parameters or model structure changes.

Although it is possible to use multiple sub-models for training, this can be
computationally expensive and time-consuming. We experimentally confirm that
MCR-Data2vec 2.0 can achieve considerable improvements using only two sub-
models.

Method | # Params | Content | Semantics | ParaL | Speaker | Generation
---|---|---|---|---|---|---
PR | ASR | KS | QbE | IC | SF | ER | SID | ASV | SE | SS
PER $\downarrow$ | WER $\downarrow$ | Acc $\uparrow$ | MTWV $\uparrow$ | Acc $\uparrow$ | F1 $\uparrow$ | CER $\downarrow$ | Acc $\uparrow$ | Acc $\uparrow$ | EER $\downarrow$ | PESQ $\uparrow$ | STOI $\uparrow$ | SI-SDRi $\uparrow$
HuBERT [1] | 94.70 M | 5.41 | 6.42 | 96.30 | 0.0736 | 98.34 | 88.53 | 25.20 | 64.92 | 81.42 |  5.11 | 2.58 | 93.90 | 9.36
Wav2vec 2.0 [2] | 95.04 M | 5.74 | 6.43 | 96.23 | 0.0233 | 92.35 | 88.30 | 24.77 | 63.43 | 75.18 | 6.02 | 2.55 | 93.90 | 9.77
WavLM [3] | 94.70 M | 4.84 | 6.21 | 96.79 | 0.0870 | 98.63 | 89.38 | 22.86 | 65.94 | 84.51 | 4.69 | 2.58 | 94.01 | 10.37
Data2vec [4] | 93.75 M | 4.69 | 4.94 | 96.56 | 0.0576 | 97.63 | 88.59 | 25.27 | 66.27 | 70.21 | 5.77 | 2.96 | 94.83 | 9.78
CCC-Wav2vec 2.0 [5] | 95.04 M | 5.95 | 6.30 | 96.72 | 0.0673 | 96.47 | 88.08 | 24.34 | 64.17 | 72.84 | 5.61 | 3.06 | 94.94 | 10.86
_Our implementation_ |  |  |  |  |  |  |  |  |  |  |  |  |  |
Data2vec 2.0 [24] | 93.78 M |  3.64 |  4.81 |  96.89 |  0.0841 |  99.00 |  89.67 |  22.09 |  66.66 | 81.43 | 5.59 | 2.98 |  94.85 | 10.41
MCR-Data2vec 2.0 (Ours) | 93.78 M | 3.37 | 4.68 | 97.05 | 0.0595 | 99.21 | 90.04 | 21.73 | 66.99 |  82.40 | 5.36 |  3.01 | 94.94 |  10.62
Table 1: Performance comparison on the SUPERB benchmark. ParaL represents
Paralinguistics aspect of speech. All models were trained using 960 hours of
LibriSpeech and based on the Base setting, consisting of 12 Transformer
blocks. We marked the best in bold and the second-best with underline.

###  3.2 Model-level Consistency Regularization

To train our network, we pass each masked input $x_{(n,m)}$ through the
network twice, resulting in two predictions denoted as $f_{1}(x_{(n,m)})$ and
$f_{2}(x_{(n,m)})$. Note that the two forward passes are based on different
sub-models due to the dropout variants. Thus, the predictions
$f_{1}(x_{(n,m)})$ and $f_{2}(x_{(n,m)})$ are different for the same masked
input $x_{(n,m)}$.

Firstly, the MCR-Data2vec 2.0 requires the student sub-models to predict the
same target representation $y_{n}$ of the teacher model. We minimize the
$l_{2}$ loss between the target representation and the predictions, which can
be formulated as:

$\displaystyle\mathcal{L}_{pred}^{(n,m)}$
$\displaystyle=\mathcal{L}_{pred1}^{(n,m)}+\mathcal{L}_{pred2}^{(n,m)}$
$\displaystyle=(y_{n}-f_{1}(x_{(n,m)}))^{2}+(y_{n}-f_{2}(x_{(n,m)}))^{2}$ (1)

where $\mathcal{L}_{pred1}^{(n,m)}$ corresponds to the training objective of
the original Data2vec 2.0.

Also, the MCR-Data2vec 2.0 aims to encourage the student sub-models to produce
consistent outputs, regardless of randomness in the model architecture. Since
the target of the sub-models is the contextualized representation, both
$f_{1}(x_{(n,m)})$ and $f_{2}(x_{(n,m)})$ are continuous. Thus, we minimize
the $l_{2}$ loss between $f_{1}(x_{(n,m)})$ and $f_{2}(x_{(n,m)})$ to perform
the consistency regularization. The proposed regularization objective
$\mathcal{L}_{mcr}^{(n,m)}$ for the masked sample $x_{n}^{m}$ can be computed
as follows:

$\mathcal{L}_{mcr}^{(n,m)}=(f_{1}(x_{(n,m)})-f_{2}(x_{(n,m)}))^{2}.$ (2)

Although Eq. (3.2) aims to make two different sub-models predict the same
target $y_{n}$, relying solely on this objective function may not sufficiently
reduce the variance (difference) between the outputs of the sub-models. By
minimizing the variance between the outputs of different sub-models via Eq.
(2), we can further improve the model's consistency.

###  3.3 Training Objective

The final objective $\mathcal{L}_{total}^{(n,m)}$ for the sample $x_{(n,m)}$
is given as

$\mathcal{L}_{total}^{(n,m)}=\mathcal{L}_{pred}^{(n,m)}+\lambda\mathcal{L}_{mcr}^{(n,m)}$
(3)

where $\lambda$ is a tunable parameter, and we experimentally set $\lambda$ to
1.

As mentioned earlier, there is the gap between the pre-training and fine-
tuning stages due to the difference in the amount of randomness. By
regularizing the outputs from the two sub-models to be consistent, the sub-
models are less affected by the randomness. Thus, MCR-Data2vec 2.0 can
effectively reduce the inconsistency between the pre-training and fine-tuning,
improving the overall quality of the SSL model.

##  4 Experimental Setup

###  4.1 Pre-training Setup

We conducted experiments using the Base model configuration, which includes 12
Transformer blocks, 768-dimensional hidden states, and 8 attention heads,
resulting in a total of 93.78 M parameters for MCR-Data2vec 2.0. To pre-train
the Data2vec 2.0 Base and MCR-Data2vec 2.0 Base, we used 960 hours of audio
from LibriSpeech [29] and trained it for 400K updates on 8 Quadro RTX 8000
48GB GPUs. The models were implemented using fairseq toolkit [30]. For all
other training configurations, we followed the hyperparameters outlined in
Data2vec 2.0 [24]. The multi-masking strategy for the MCR-Data2vec 2.0 was
also identical to Data2vec 2.0.

###  4.2 Universal Representation Evaluation

To evaluate the effectiveness of our proposed approach, we conducted
experiments using SUPERB [25, 26], a standardized benchmark for pre-trained
models in a range of speech tasks. We evaluated our model on eleven subtasks
of SUPERB, including Phoneme Recognition (PR), Automatic Speech Recognition
(ASR), Keyword Spotting (KS), Query by Example Spoken Term Detection (QbE),
Intent Classification (IC), Slot Filling (SF), Emotion Recognition (ER),
Speaker Identification (SID), Automatic Speaker Verification (ASV), Speech
Enhancement (SE), and Speech Separation (SS). We followed the guidelines
created by SUPERB. Firstly, we employed the same downstream models that were
utilized by SUPERB for each task. Secondly, we froze the pre-trained models
during the fine-tuning. Finally, the downstream models processed the weighted
sum of hidden state results obtained from each layer of the pre-trained model.
For ASR, ASV, SE, and SS, we followed the official configurations of SUPERB.
For the other downstream tasks, we followed the fine-tuning hyperparameter
settings of WavLM [3]. During the fine-tuning, we trained the models on a
single Quadro RTX 8000 48GB GPU.

##  5 Experimental Result

###  5.1 Main Results

We compared the MCR-Data2vec 2.0 with previous SOTA approaches, including
HuBERT [1], Wav2vec 2.0 [2], WavLM [3], Data2vec [4], CCC-Wav2vec 2.0 [5], and
Data2vec 2.0 [24]. Table 1 summarizes the results on SUPERB benchmark. Even
though Data2vec 2.0 achieved promising results compared to previous SSL
methods, MCR-Data2vec 2.0 outperformed the original Data2vec 2.0 in most
configurations, except for the QbE task. For the QbE, the performance of the
MCR-Data2vec 2.0 was worse than that of the Data2vec 2.0. Considering that the
HuBERT Large model performed worse than the HuBERT Base model on the QbE task
[25], higher performance on QbE did not always indicate a better SSL model.

As shown in Table 1, the proposed approach yielded SOTA performances for PR,
ASR, KS, IC, SF, and ER tasks, while Data2vec 2.0 had the second-best results.
By simply regularizing the output of student sub-models to be consistent, the
proposed framework effectively improved the overall quality of the original
Data2vec 2.0. From the results, it is verified that reducing the gap between
the pre-training and fine-tuning stages is important in training the SSL
model.

Additionally, the proposed method showed significant improvements over the
Data2vec 2.0 for speaker and generation tasks, achieving the second-best
performances for SID, SE, and SS tasks. In the case of STOI for SE, MCR-
Data2vec 2.0 yielded the best performance along with CCC-Wav2vec 2.0. It is
important to note that WavLM and CCC-Wav2vec 2.0, which performed well on such
tasks, benefited from data augmentation for the noisy scenario during the pre-
training. WavLM [3] manually simulated noisy/overlapped speech as inputs, and
CCC-Wav2vec 2.0 [5] employed three different data augmentations to make the
SSL model more robust to augmentations. However, MCR-Data2vec 2.0 achieved
comparable results on speaker and generation tasks without using such data
augmentation techniques. This means that our model-level consistency
regularization was supportive in improving the generalization ability and
robustness of the SSL model.

(a) Data2vec 2.0

(b) MCR-Data2vec 2.0

Figure 2: Weight analysis on content and semantic tasks.

###  5.2 Weight Analysis

Following the SUPERB guidelines, we computed a weighted sum of the
representations obtained from each layer of the pre-trained model and then fed
it to the downstream models. Since MCR-Data2vec 2.0 showed considerable
performance improvements, especially for content and semantic tasks, we
analyzed the contribution patterns of the proposed approach. Figure 2 depicts
the weights of the different layers of Data2vec 2.0 and MCR-Data2vec 2.0 for
PR, ASR, KS, IC, and SF tasks, where the higher weight means that the
corresponding layer contributed more to achieving the specific task. From the
results, we found that the patterns of MCR-Data2vec 2.0 were similar to those
of Data2vec 2.0. Even though WavLM [3] reported that the content and semantic
information were primarily encoded in the top layers, both SSL models tended
to leverage information from both middle and top layers. In Data2vec 2.0
framework, the use of information from broader layers could be an important
factor in achieving better performance for content and semantic tasks. In
addition, we can observe that the contribution patterns of MCR-Data2vec 2.0
were not predominantly concentrated on a specific layer compared to those of
Data2vec 2.0. For example, for the ASR task, Data2vec 2.0 heavily relied on
the 10th layer, whereas the patterns of MCR-Data2vec 2.0 were more evenly
distributed across layers 5 to 10. Similarly, for the SF task, Data2vec 2.0
predominantly assigned weights to the 6th and 7th layers, while MCR-Data2vec
2.0 utilized information more uniformly across layers 5 to 8.

###  5.3 $\mathcal{L}_{pred1}$ Loss Curve

To check the effectiveness of the proposed regularization term, we analyzed
the pre-training loss function $L_{pred1}$ (in Eq (3.2)). $L_{pred1}$ measures
the distance between the student sub-model's prediction and the target
representation of the teacher model. It is important to note that, as shown in
Figure 1, both Data2vec 2.0 and MCR-Data2vec 2.0 used $L_{pred1}$ during the
pre-training stage. The loss curves for $\mathcal{L}_{pred1}$ are shown in
Figure 3. The results showed that $\mathcal{L}_{pred1}$ of MCR-Data2vec 2.0
significantly decreased compared to that of Data2vec 2.0. This indicates that
the proposed regularization improved the student model's ability to predict
the target representation of the teacher model.

Figure 3: $\mathcal{L}_{pred1}$ loss curves of Data2vec 2.0 and MCR-Data2vec
2.0, where the total number of updates is 400k.

##  6 Conclusion

We proposed a novel pre-training method, MCR-Data2vec 2.0, to improve the
model-level consistency of speech SSL. The proposed framework could reduce the
gap between the pre-training and fine-tuning stages while improving the
overall quality of the SSL model. From experimental results on the SUPERB
benchmark, it is verified that MCR-Data2vec 2.0 achieved promising
performance, outperforming recent SSL models.

## References

  * [1] W. N. Hsu, B. Bolte, Y. H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [2] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, ``Wav2vec 2.0: a framework for self-supervised learning of speech representations,'' in _Proc. NIPS_ , 2020\.
  * [3] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, ``Wavlm: large-scale self-supervised pre-training for full stack speech processing,'' _JSTSP_ , 2022.
  * [4] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Aulim, ``Data2vec: a general framework for self-supervised learning in speech, vision and language,'' in _Proc. ICML_ , 2022, pp. 1298–1312.
  * [5] V. S. Lodagala, S. Ghosh, and S. Umesh, ``Ccc-wav2vec 2.0: clustering aided cross contrastive self-supervised learning of speech representations,'' in _Proc. IEEE SLT_ , 2022.
  * [6] E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P. Mazaré, M. Douze, and E. Dupoux, ``Data augmenting contrastive learning of speech representations in the time domain,'' in _Proc. IEEE SLT_ , 2021.
  * [7] A. Sriram, M. Auli, and A. Baevski, ``Wav2vec-aug: improved self-supervised training with limited data,'' _arXiv preprint arXiv:2206.13654v1_ , 2022.
  * [8] C. Gao, G. Cheng, and P. Zhang, ``Multi-variant consistency based self-supervised learning for robust automatic speech recognition,'' _arXiv preprint arXiv:2112.12522v2_ , 2022.
  * [9] Q. Zhu, L. Zhou, J. Zhang, S. Liu, Y. Hu, and L. Dai, ``Robust data2vec: noise-robust speech representation learning for asr by combining regression and improved contrastive learning,'' in _Proc. ICASSP_ , 2023.
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Proc. NIPS_ , 2017, pp. 5998–6008.
  * [11] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, ``Dropout: a simple way to prevent neural networks from overfitting,'' _JMLR_ , 2014.
  * [12] A. Fan _et al._ , ``Reducing transformer depth on demand with structured dropout,'' in _Proc. ICLR_ , 2020.
  * [13] G. Huang _et al._ , ``Deep networks with stochastic depth,'' in _Proc. ECCV_ , 2016.
  * [14] L. Tseng, Y. Fu, H. Chang, and H. Lee, ``Mandarin-english code-wwitching speech recognition with self-supervised speech representation models,'' in _Proc. AAAI SAS_ , 2022.
  * [15] S. Lee, S. Kim, J. Lee, E. Song, M. Hwang, and S. Lee, ``Hierspeech: bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis,'' in _Proc. NIPS_ , 2022\.
  * [16] H. Siuzdak, P. Dura, P. Rijn, and N. Jacoby, ``Wavthruvec: latent speech representation as intermediate features for neural speech synthesis,'' in _Proc. INTERSPEECH_ , 2022.
  * [17] B. Thomas, S. Kessler, and S. Karout, ``Efficient adapter transfer of self-supervised speech models for automatic speech recognition,'' in _Proc. ICASSP_ , 2022.
  * [18] N. Pham, A. Waibel, and J. Niehues, ``Adaptive multilingual speech recognition with pretrained models,'' in _Proc. INTERSPEECH_ , 2022.
  * [19] R. Wang _et al._ , ``Lighthubert: lightweight and configurable speech representation learning with once-for-all hidden-unit bert,'' in _Proc. INTERSPEECH_ , 2022.
  * [20] X. Liang, L. Wu, J. Li, Y. Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, and T. Liu, ``R-drop: regularized dropout for neural networks,'' in _Proc. NIPS_ , 2021.
  * [21] K. Zolna, D. Arpit, D. Suhubdy, and Y. Bengio, ``Fraternal dropout,'' in _Proc. ICLR_ , 2018.
  * [22] A. Tarvainen and H. Valpola, ``Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results,'' in _Proc. NIPS_ , 2017.
  * [23] B. Zheng, L. Dong, S. Huang, W. Wang, Z. Chi, S. Singhal, W. Che, T. Liu, X. Song, and F. Wei, ``Consistency regularization for cross-lingual fine-tuning,'' in _Proc. ACL_ , 2021.
  * [24] B. Baevski, A. Babu, W. Hsu, and M. Auli, ``Efficient self-supervised learning with contextualized target representations for vision, speech and language,'' _arXiv preprint arXiv:2212.07525v1_ , 2022.
  * [25] S. Yang _et al._ , ``Superb: speech processing universal performance benchmark,'' in _Proc. INTERSPEECH_ , 2021, pp. 1194–1198.
  * [26] A. Tarvainen and H. Valpola, ``Superb-sg: enhanced speech processing universal performance benchmark for semantic and generative capabilities,'' in _Proc. ACL_ , 2022.
  * [27] J. Grill _et al._ , ``Bootstrap your own latent: A new approach to self-supervised learning,'' in _Proc. NIPS_ , 2020.
  * [28] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, ``Masked autoencoders are scalable vision learners,'' in _Proc. CVPR_ , 2022.
  * [29] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _Proc. ICASSP_ , 2015, pp. 5206–5210.
  * [30] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, ``Fairseq: a fast, extensible toolkit for sequence modeling,'' in _Proc. NAACL_ , 2019, p. 48–53.