# Backward behavior and determining functionals for chevron pattern equations

V. K. Kalantarov1, H. V. Kalantarova2 , and O. Vantzos3 1Department of
Mathematics, Koç University, Rumelifeneri Yolu, Sarıyer, Istanbul, Turkey
2Springer-Verlag, Heidelberg, Germany  3Vantzos Research SMPC, Athens, Greece

###### Abstract.

The paper is devoted to the study of the backward behavior of solutions of the
initial boundary value problem for the chevron pattern equations under
homogeneous Dirichlet’s boundary conditions. We prove that, as
$t\rightarrow\infty$, the asymptotic behavior of solutions of the considered
problem is completely determined by the dynamics of a finite set of
functionals. Furthermore, we provide numerical evidence for the blow-up of
certain solutions of the backward problem in finite time in 1D.

##  1\. Introduction

We consider the system of equations

(1.1) $\displaystyle\tau\partial_{t}A=A+\Delta
A-\phi^{2}A-|A|^{2}A-2ic_{1}\phi\partial_{y}A+i\beta A\partial_{y}\phi,$ (1.2)
$\displaystyle\partial_{t}\phi=-\mathbb{L}\phi-h\phi+\phi\lvert
A\rvert^{2}-c_{2}\mbox{Im}\left[A^{\ast}\partial_{y}A\right],\quad(x,y)\in\Omega,\quad
t>0,$

where the linear operator $\mathbb{L}$ is given by

$\mathbb{L}:=-D_{1}\partial^{2}_{x}-D_{2}\partial^{2}_{y},$

and $\Omega\subset\mathbb{R}^{2}$ is a bounded domain with a sufficiently
smooth boundary $\partial\Omega$. The system is complemented by the following
boundary conditions

(1.3) $A\Big{|}_{\partial\Omega}=0,\quad\phi\Big{|}_{\partial\Omega}=0,\quad
t>0,$

and the initial conditions

(1.4)
$A\Big{|}_{t=0}=A_{0},\quad\phi\Big{|}_{t=0}=\phi_{0},\quad(x,y)\in\Omega.$

In the context of this study,

(1.5) $D_{1}>0,\ D_{2}>0,\ c_{1}\geq 0,\ c_{2}\geq 0,\ h\geq 0,\
\beta\in\mathbb{R}$

are given parameters, the complex valued function $A$, where $A^{\ast}$
denotes its complex conjugate, and the real valued function $\phi$ are unknown
functions. This coupled system of equations was proposed by Rossberg et al.
[26], [27] to model the formation and evolution of patterns in nematic liquid
crystals subject to convection driven by an electric field, sometimes
accompanied with magnetic field. These works were motivated by a number of
experimental studies [18], [25], [24].

Figure 1. Cross-section of a nematic liquid crystal contained between two
parallel transparent plates. Nematic liquid crystals are composed of rod-like
molecules, that are free to flow and orient themselves in 3D space, when
external electric $\vec{E}$ and magnetic fields $\vec{H}$ are applied.

A typical experimental setup used in these investigations consists of a
nematic liquid crystal confined between two planar transparent glass plates,
placed parallel at a distance of approximately $50\mu m$ from each other. Then
AC-voltage of varying frequency is applied across the plates, sometimes in
conjunction with a magnetic field parallel to the plates. The results of the
described paradigmatic experiment, revealed that through use of either
microscopes (with or without polarizers) or via laser diffraction various
pattern formations are observable: Williams domain, Grid-Pattern, chevron
patterns, etc.. The formation of these patterns is governed by experimentally
easily controllable parameters, the amplitude and the frequency of the applied
voltage and the magnitude of magnetic field. The advantage of liquid crystals
over other fluids in these experiments besides its transparency and its phase
transition temperatures being close to the room temperature, which makes
detection of patterns easier, [3], is the fact that their molecules are
locally oriented along desired direction. In [28], it is reported that the
described system has a tendency to form chevron structures after a transition
to defect chaos and that chevron patterns are generally observed when the
isotropy of the system is spontaneously broken, this effect is achieved either
by increasing the voltage or by applying a weak magnetic field.

The model (1.1)-(1.2) is obtained by coupling the real Ginzburg-Landau
equation, where $A$ represents the phase, direction and amplitude of the
periodic pattern and equation (1.2), where $\phi$ represents the angle between
director and the x-axis and also by using symmetries, smallness and regularity
observations from experiments as assumptions. In this model the parameter
$\tau$ is a function of the various physical timescales of the problem,
$\beta$ measures the interaction between the gradient of the director field
and the phase of the rolls, the parameters $D_{1}$ and $D_{2}$ are the
coefficients of the anisotropic diffusion of the director field for the liquid
crystal particles, the dampening parameter $h$ measures the tendency of the
director field to align with the magnetic field $H$, corresponding to $\phi=0$
and the parameters $c_{1}$ and $c_{2}$ control the torque that the director
field and the wave vector of the rolls exert on each other.

In [27], Rossberg et al. generated chevron patterns in simulations of
(1.1)-(1.2). For more numerical studies of the model refer to [29], [21].

The mathematical model (1.1)-(1.2) has also been studied qualitatively. In
[19], the authors prove the following well-posedness result:

######  Theorem 1.1.

([19]) If $c_{1}\in[0,1)$ or $c_{1}\geq 2c_{2}$, $h>0$, then the initial
boundary value problem (1.1)-(1.4), where $A_{0},\phi_{0}\in L^{2}(\Omega)$,
has a unique weak solution

(1.6) $A,\phi\in C([0,T];L^{2}(\Omega));H_{0}^{1}(\Omega)),\quad\forall T>0,\
\ $

such that

(1.7) $\|\phi(t)\|\leq M_{0},\ \ \|A(t)\|\leq M_{0},\ \ \forall t>0,$

and

(1.8) $\int_{0}^{T}\|\nabla\phi(t)\|^{2}dt\leq M_{T},\ \ \int_{0}^{T}\|\nabla
A(t)\|^{2}dt\leq M_{T},\ \forall T>0,$

where $M_{0}$ denotes a generic constant depending only on $\lVert
A_{0}\rVert$, $\lVert\phi_{0}\rVert$ and $\lvert\Omega\rvert$, and where
$M_{T}$ is a generic constant, which depends also on T, besides $\lVert
A_{0}\rVert$, $\lVert\phi_{0}\rVert$ and $\lvert\Omega\rvert$. In other words
this problem generates a continuous semigroup $S(t$), $t\geq 0,$ in the phase
space $V^{0}:=L^{2}(\Omega)\times L^{2}(\Omega).$ Moreover, this semigroup is
bounded dissipative in the phase space $V^{0}$.

In [19], the authors also proved that the problem (1.1)-(1.4) possesses a
global attractor, i.e., there exists a compact, invariant set that attracts
uniformly each bounded set of the phase space $V^{0}$. Furthermore, it is
proved that this semigroup has a global attractor that belongs to
$V^{1}:=H^{1}_{0}(\Omega)\times H^{1}_{0}(\Omega)$. Later, this result is
improved in [20], in which it is proved that under the same assumptions of the
Theorem 1.1 the semigroup
$S(t):V^{0}\rightarrow V^{0}$, $t\geq 0,$ generated by the problem (1.1)-(1.4)
possesses an exponential attractor.

In [20], the authors also prove the global stabilization of the zero steady
state of the system (1.1)-(1.2) by finitely many Fourier modes:

######  Theorem 1.2.

The solution $[A(t),\phi(t)]$ of the problem

(1.9) $\displaystyle\tau\partial_{t}A=A+\Delta
A-\phi^{2}A-|A|^{2}A-2ic_{1}\phi\partial_{y}A+i\beta A\partial_{y}\phi,$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-\mu\sum\limits_{k=1}^{N}(A,w_{k})w_{k}$
(1.10) $\displaystyle\partial_{t}\phi=-\mathbb{L}\phi-h\phi+\phi\lvert
A\rvert^{2}-c_{2}\mbox{Im}\left[A^{\ast}\partial_{y}A\right],$ (1.11)
$\displaystyle A\big{|}_{\partial\Omega}=\phi\big{|}_{\partial\Omega}=0,\ \
A\big{|}_{t=0}=A_{0},\ \phi\big{|}_{t=0}=\phi_{0},$

tends to zero stationary state with an exponential rate in $V^{1}$, if

$\mu\geq 1\quad\mbox{ and }\quad\lambda_{N+1}^{-1}<\delta_{0},$

where $0<\lambda_{1}\leq\lambda_{2}\leq\ldots\leq\lambda_{N+1}\leq\ldots$ are
the eigenvalues of the Laplace operator under the homogeneous Dirichlet
boundary condition,

$\delta_{0}:=2(1-c_{1})/(2+c_{2})\quad\mbox{and}\quad c_{1}<1.$

In this work, our primary objective is to demonstrate that the long time
behavior of solutions to the initial boundary value problem for the system of
equations (1.1)-(1.2) can be uniquely determined by finite number of
determining functionals. Furthermore, we study the backward in time behavior
of solutions to the system (1.1)-(1.2).

After covering some preliminaries in section 2, we devote section 3 of the
paper to the backward in time behavior of solutions to the system (1.1)-(1.2).
There are numerous papers dedicated to the backward behavior of solutions to
nonlinear evolution equations (see, e.g. Bardos and Tartar [2], Dascaliuc
[12], Kukavica and Malcok [22], Guo and Titi [15], Vukadinovic [31] and
references therein). In this section, we establish the backward uniqueness of
the problem (1.1)-(1.4) and show that some solutions of the backward problem
blow up in a finite time. We follow this with further numerical evidence of
finite-time blow up in section 4, using a convex-concave splitting scheme with
explicit tracking of blow up times during each discrete time step.

Section 5 of the paper is dedicated to studying the existence of determining
functionals for the system of chevron pattern equations (1.1)-(1.2). The
intensive study over the last decades of finite dimensional asymptotic
behavior of dissipative dynamical systems generated by nonlinear PDEs has been
inspired by the pioneering works of Foias and Prodi [13] and Ladyzhenskaya
[23]. They proved that asymptotic behavior of solutions to the initial
boundary value problem for 2D Navier-Stokes equations is completely determined
by asymptotic behavior of first $N$ Fourier modes (for $N$ large enough). For
subsequent investigations of the finite-dimensional behavior of dynamical
systems generated by the initial boundary value problem for dissipative PDEs
(existence of finite-dimensional global attractors, inertial manifolds,
exponential attractors, finite number of modes, nodes and local volume
elements determining asymptotic behavior of solutions) we refer to [1], [14],
[16], [30] and the references therein. The concept of determining functionals
(determining interpolant operators) is introduced in [10], [11]. This concept
provided a unified approach for investigating parameters that uniquely
determine the asymptotic behavior of solutions to dissipative PDEs. A further
extension of this unified approach for studying the long-time behavior of
various nonlinear dissipative PDEs was developed in subsequent investigations
of the long-time behaviour of solutions to initial boundary value problems for
Navier-Stokes equations, systems of nonlinear parabolic equations, weakly
damped nonlinear wave equations, nonlinear wave equation with nonlinear
damping, von Karman equations and some other dissipative systems (see, e.g.,
Chueshov [6], [5], [7], Chueshov and Kalantarov [9], Chueshov and Lasiecka
[8], Holst and Titi [17] and references therein.)

The main result of this section is Theorem 5.1, which establishes the
existence of a finite number of functionals that uniquely determine the
asymptotic behavior of solutions to the problem (1.1)-(1.4).

##  2\. Preliminaries

In this section, we introduce the notation that will be used in what follows
and recapitulate some definitions and known results.

  * •

$C$ will denote a generic positive constant,

  * •

$(\cdot,\cdot):=(\cdot,\cdot)_{L^{2}(\Omega)}$,
$\|\cdot\|:=\|\cdot\|_{L^{2}(\Omega)}$,

  * •

$V^{0}:=L^{2}(\Omega)\times L^{2}(\Omega)$ is the Hilbert space with the inner
product

$\langle[u_{1},v_{1}],[u_{2},v_{2}]\rangle_{V^{0}}:=(u_{1},u_{2})+(v_{1},v_{2}),$

  * •

$V^{1}:=H^{1}_{0}\times H^{1}_{0}$ denotes the Hilbert space with the inner
product

$\langle[u_{1},v_{1}],[u_{2},v_{2}]\rangle_{V^{1}}:=(\nabla u_{1},\nabla
u_{2})+(\nabla v_{1},\nabla v_{2}).$

######  Definition 2.1.

( [4] ) Let $V$ and $H$ be reflexive Banach spaces and $V$ be continuously and
densely embedded into $H$. The completeness defect of a set
${\mathcal{L}}=\\{\ell_{j}:j=1,\ldots,N\\}$ of linear functionals on $V$ with
respect to $H$ is the following number

$\epsilon_{{\mathcal{L}}}(V,H):=\sup\\{\|w\|_{H}:w\in
V,\,\ell_{j}(w)=0,\,\ell_{j}\in{\mathcal{L}},\,\|w\|_{V}\leq 1\\}.$

$\epsilon_{{\mathcal{L}}}(V,H)=0$ if and only if the class of functionals
${\mathcal{L}}$ is complete in $V$, i.e., the condition $\ell_{j}(w)=0$ for
all $\ell_{j}\in{\mathcal{L}}$ implies $w=0$.

######  Theorem 2.2.

( [4] ) Let $\epsilon_{\mathcal{L}}=\epsilon_{\mathcal{L}}(V,H)$ be the
completeness defect of a set ${\mathcal{L}=\\{\ell_{j}:j=1,\ldots,N\\}}$ of
linear functionals on $V$ with respect to $H$. Then there exists a constant
$C_{\mathcal{L}}$ such that

$\|w\|_{H}\leq
C_{{\mathcal{L}}}\eta_{{\mathcal{L}}}(w)+\epsilon_{\mathcal{L}}\,\|w\|_{V}\qquad\mbox{for
any}\quad w\in V,$

where $\eta_{{\mathcal{L}}}(w)=\max\\{|\ell_{j}(w)|:j=1,\ldots,N\\}$.

######  Lemma 2.3.

(see, e.g., [30]) Let $g,h$ and $y$ be three positive locally integrable
functions on the interval $(t_{0},\infty)$, such that $y^{\prime}$ (or
equivalently $\frac{dy}{dt}$) is locally integrable on the same interval.
Suppose that the following inequalities are satisfied

$y^{\prime}(t)\leq g(t)y(t)+h(t),\ \ t\in[t_{0},\infty),$
$\int_{t}^{t+r}g(s)ds\leq a_{1},\ \int_{t}^{t+r}h(s)ds\leq a_{2},\
\int_{t}^{t+r}y(s)ds\leq a_{3},\ \ \forall t\in[t_{0},\infty),$

where $r,a_{1},a_{2},a_{3}$ are positive constants.
Then

$y(t+r)\leq(\frac{a_{3}}{r}+a_{2})e^{a_{1}},\ \ t\in[t_{0},\infty).$

######  Theorem 2.4.

(see, e.g. [30]) Suppose that $\mathbf{A}$ is a positive definite, unbounded,
self adjoint linear operator in H with domain $V^{2}:=D(\bf{A})$ and let the
nonlinear operator $\mathbf{B}(\cdot):(0,T)\times V^{1}\rightarrow H,$ with
$V^{1}:=D(\mathbf{A}^{\frac{1}{2}})$ satisfy the condition

(2.1) $\|\mathbf{B}(\mathbf{u}(t))\|_{H}\leq
k(t)\|\mathbf{u}(t)\|_{V^{1}},\quad k(t)\in L^{2}(0,T).$

If

(2.2) $\mathbf{u}\in L^{\infty}(0,T;V^{1})\cap L^{2}(0,T;V^{2})$

is a solution of the equation

(2.3)
$\mathbf{u}^{\prime}(t)+\mathbf{A}\mathbf{u}(t)=\mathbf{B}(\mathbf{u}(t)),\
t\in(0,T),$

that satisfies the condition $\mathbf{u}(T)=0$, then $\mathbf{u}(t)=0,\
\forall t\in[0,T].$

######  Definition 2.5.

A set of continuous linear functionals $\mathcal{L}=\\{l_{j}:j=1,\ldots,N\\}$
on $V$ is said to be a set of asymptotically $(V,H,\mathcal{W})-$determining
functionals for the problem (2.5) if for any two solutions $u,v\in\mathcal{W}$
the condition

$\lim_{t\rightarrow\infty}\int_{t}^{t+1}\lvert
l_{j}(u(\tau))-l_{j}(v(\tau))\rvert^{2}d\tau=0\mbox{ for }j=1,\ldots,N,$

implies that

$\lim_{t\rightarrow\infty}\lVert u(t)-v(t)\rVert_{H}=0.$

######  Theorem 2.6.

([6], Theorem 4.1.) Suppose that

  * •

$\mathbf{A}$ is a positive definite operator with a discrete spectrum in a
Hilbert space $H$, the basis $\\{w_{k}\\}$ of which consists of eigenvectors
of the operator $\mathbf{A}$:

$\mathbf{A}w_{k}=\lambda_{k}w_{k},\ \ (w_{k},w_{j})=\delta_{kj},\
0<\lambda_{1}\leq\lambda_{2}\leq\cdots,\ \
\lim_{k\rightarrow\infty}\lambda_{k}=\infty,$

  * •

the nonlinear operator
$\mathbf{B}(\cdot):D(\mathbf{A}^{\frac{1}{2}})\rightarrow
D(\mathbf{A}^{-\gamma}),\ 0\leq\gamma<\frac{1}{2}$ satisfies the condition

(2.4)
$\|\mathbf{A}^{-\gamma}(\mathbf{B}({\mathbf{u}}_{1})-\mathbf{B}({\mathbf{u}}_{2}))\|\leq
M(\rho)\|\mathbf{A}^{\frac{1}{2}}(\mathbf{u}_{1}-\mathbf{u}_{2})\|,$

for all ${\mathbf{u}}_{j}\in\mathbf{D}({\mathbf{A}}^{\frac{1}{2}})$ such that
$\|\mathbf{A}^{\frac{1}{2}}\mathbf{u_{j}}\|\leq\rho$, $\rho>0,$ and
$M(\rho)>0$ is a continuous function on $\mathbb{R}^{+},$

  * •

the problem

(2.5)
$\begin{cases}\mathbf{u}^{\prime}(t)+\mathbf{A}\mathbf{u}(t)=\mathbf{B}(\mathbf{u}(t)),\
\ t>0,\\\ \mathbf{u}(0)=\mathbf{u}_{0},\end{cases}$

has a unique solution in the class of functions
$\mathcal{W}=C([0,\infty);H)\cap C([0,\infty);D(A^{\frac{1}{2}}))$ and it is
point dissipative , i.e.,

$\|\mathbf{A}^{\frac{1}{2}}\mathbf{u}(t)\|\leq R,\ \ \forall t\geq
t_{0}(u),\quad\forall\mathbf{u}(t)\in\mathcal{W}.$

Suppose also that ${\mathcal{L}}=\\{\ell_{j},\ j=1,\ldots,N\\}$ is a set of
linear continuous functionals defined on $V^{1}=D(\mathbf{A}^{\frac{1}{2}})$.
Then ${\mathcal{L}}$ is a set of asymptotically $(V^{1},H,\mathcal{W})$\-
determining functionals of the problem (2.5) whenever

$\varepsilon_{{\mathcal{L}}}(V^{1},H)<\varepsilon_{0}(\gamma,R)\equiv\sqrt{\frac{1+2\gamma}{1-2\gamma}}\left[(1+2\gamma)M(R)\right]^{\frac{1}{2\gamma-1}}.$

######  Definition 2.7.

We call a pair of functions $[A,\phi]$ a weak solution of the initial boundary
value problem for the system of equations (1.1)-(1.2) under the homogeneous
Dirichlet’s boundary condition, if
$[A,\phi]\in\mathcal{W}=C([0,\infty);V^{0})\cap C((0,\infty);V^{1})$ and the
system of equations (1.1)-(1.2) is satisfied in the sense of distributions.

##  3\. Backward behavior

In this section, we will prove the backward uniqueness of a solution to the
problem (1.1)-(1.4) and demonstrate that some solutions of the backward
problem blow up in finite time. For 1D chevron pattern equations, we will
illustrate that solutions not belonging to the global attractor of the
semigroup generated by the initial boundary value problem blow up in a finite
time $T>0$.

The backward uniqueness of solution to the problem follows from the
corresponding result for an abstract differential equation in a Hilbert space
$H$ of the form (2.5).

Suppose that $[A,\phi]$ and $[\tilde{A},\tilde{\phi}]$ are solutions of the
system (1.1)-(1.2) corresponding to initial data $[A_{0},\phi_{0}]$ and
$[\tilde{A}_{0},\tilde{\phi}_{0}]$ respectively, then
$[a,\Phi]:=[A-\tilde{A},\phi-\tilde{\phi}]$ is a solution of the following
system

(3.1) $\tau\partial_{t}a=a+\Delta
a-\phi^{2}a-\tilde{A}(\phi+\tilde{\phi})\Phi-|A|^{2}a-(Aa^{*}+\tilde{A}^{\ast}a)\tilde{A}\\\
-2ic_{1}[\Phi\partial_{y}A+\tilde{\phi}\partial_{y}a]+i\beta[a\partial_{y}\phi+\tilde{A}\partial_{y}\Phi],$
(3.2)
$\partial_{t}\Phi=-\mathbb{L}\Phi-h\Phi+\Phi|A|^{2}+\tilde{\phi}(Aa^{*}+\tilde{A}^{\ast}a)-c_{2}\mbox{Im}\left[a^{\ast}\partial_{y}A+\tilde{A}^{\ast}\partial_{y}a\right].$

We can write this system in the form (2.5) in a Hilbert space
$V^{0}=L^{2}(\Omega)\times L^{2}(\Omega)$ with
$\mathbf{u}:=\left[\begin{array}[]{c}a\\\ \Phi\\\ \end{array}\right],$

(3.3) $\mathbf{A}:=\left[\begin{array}[]{cc}-\frac{1}{\tau}\Delta&0\\\
0&\mathbb{L}\\\
\end{array}\right],\quad\mathbf{B}(a,\Phi):=\left[\begin{array}[]{c}\frac{1}{\tau}f_{1}(a,\Phi)\\\
f_{2}(a,\Phi)\\\ \end{array}\right],$

where

(3.4)
$f_{1}(a,\Phi):=a-\phi^{2}a-(\phi^{2}-\tilde{\phi}^{2})\tilde{A}-|A|^{2}A+|\tilde{A}|^{2}\tilde{A}\\\
-2ic_{1}[\Phi\partial_{y}A+\tilde{\phi}\partial_{y}a]+i\beta[a\partial_{y}\phi+\tilde{A}\partial_{y}\Phi],$

(3.5)
$f_{2}(a,\Phi):=-h\Phi+\Phi|A|^{2}+\tilde{\phi}(|A|^{2}-|\tilde{A}|^{2})\\\
-c_{2}\mbox{Im}\left[a^{\ast}\partial_{y}A+\tilde{A}^{\ast}\partial_{y}a\right]$

$V^{1}:=D(\mathbf{A}^{\frac{1}{2}})=H_{0}^{1}(\Omega)\times H_{0}^{1}(\Omega)$

and

(3.6) $V^{2}:=D(\mathbf{A})=H^{2}(\Omega)\cap H_{0}^{1}(\Omega)\times
H^{2}(\Omega)\cap H_{0}^{1}(\Omega).$

It is clear that the operator $\mathbf{A}$ is a positive definite self-adjoint
operator in $V^{0}:=L^{2}(\Omega)\times L^{2}(\Omega)$, with the domain
$D(\mathbf{A})$ and the nonlinear term $\mathbf{B}(a,\Phi)$ defined in (3.3)
satisfies condition (2.1). Thus the statement of the Theorem 2.4 holds true
also for the system (1.1)-(1.2). In other words, solution of the initial
boundary value problem for the backward chevron pattern equations, i.e.,
solution of the problem

(3.7) $-\frac{\partial A}{\partial t}=A+\Delta
A-\phi^{2}A-|A|^{2}A-2ic_{1}\phi\frac{\partial A}{\partial y}+i\beta
A\frac{\partial\phi}{\partial y},$ (3.8) $-\frac{\partial\phi}{\partial
t}=D_{1}\frac{\partial^{2}\phi}{\partial
x^{2}}+D_{2}\frac{\partial^{2}\phi}{\partial y^{2}}-h\phi+\phi\lvert
A\rvert^{2}-c_{2}\mbox{Im}\left[A^{\ast}\frac{\partial A}{\partial y}\right],$
(3.9) $A\big{|}_{\partial\Omega}=\phi\big{|}_{\partial\Omega}=0,\ \
A\big{|}_{t=0}=A_{0},\ \phi\big{|}_{t=0}=\phi_{0}$

is unique.
Theorem 2.4 also guarantees backward uniqueness of 1D version of the system
(1.1)-(1.2), i.e., uniqueness of a weak solution to the problem

(3.10)
$\displaystyle\partial_{t}A+A+\partial_{x}^{2}A-\phi^{2}A=|A|^{2}A,\quad
x\in(0,L),\quad t>0,$ (3.11)
$\displaystyle\phi_{t}+D_{1}\partial^{2}_{x}\phi-h\phi+|A|^{2}\phi=0,\quad
x\in(0,L),\quad t>0,$ (3.12) $\displaystyle
A\big{|}_{x=0}=A\big{|}_{x=L}=\phi\big{|}_{x=0}=\phi\big{|}_{x=L}=0,$ (3.13)
$\displaystyle A\big{|}_{t=0}=A_{0},\quad\phi\big{|}_{t=0}=\phi_{0}.$

It is not difficult to see that some solutions of the initial boundary value
problem (3.10)-(3.13) blow up in finite time.

In fact suppose that $[A,\phi]$ is a local solution of the problem
corresponding to the initial data $[A_{0},\phi_{0}]$. Multiplying the equation
(3.10) by $A$ in $L^{2}$ sense we get

$\frac{1}{2}\frac{d}{dt}\|A(t)\|^{2}=-\|A(t)\|^{2}+\|\partial_{x}A(t)\|^{2}+(\phi^{2}(t),|A(t)|^{2})+\int_{0}^{L}|A(x,t)|^{4}dx.$

Thanks to the Cauchy-Schwarz inequality we have

$\int_{0}^{L}|A(x,t)|^{2}dx\leq\sqrt{L}\left(\int_{0}^{L}|A(x,t)|^{4}dx\right)^{\frac{1}{2}}.$

Thus employing the last inequality and the Poincare inequality we obtain that
the function $\Psi(t)=\|A(t)\|^{2}$ satisfies the differential inequality

$\Psi^{\prime}(t)\geq-2\Psi(t)+2\lambda_{1}\Psi(t)+2L^{-1}\Psi^{2}(t).$

It is clear that if $\Psi_{0}:=\Psi(0)=\|A_{0}\|^{2}>(1-\lambda_{1})L$, then
the solution of this inequality tends to infinity in a finite time

$T_{0}\leq\int_{\Psi_{0}}^{\infty}\frac{ds}{2(L^{-1}s^{2}+(\lambda_{1}-1)s)},$

i.e., the solution of the problem (3.10) blows up in a finite time.

##  4\. Numerical Blow-up of the Backward System

To illustrate numerically the blow-up of $\lVert A\rVert$ in the 1D backward
system (3.10), we consider the case where $\phi=0$ and note immediately that
the resulting equation $\partial_{t}A=-\partial^{2}_{x}A+(\lvert
A\rvert^{2}-1)A$ can not be integrated numerically in a straightforward manner
because of the ’wrong’ sign of the Laplacian $\partial^{2}_{x}A$. It is a
basic result in numerical analysis that a direct numerical integration of this
equation would be dominated by spurious high frequency modes.

To progress, we notice that the equation, in its weak form, can be thought of
as the _maximizing $L^{2}$-gradient flow_

$\displaystyle(\partial_{t}A,u)=\mathcal{F}^{\prime}(A)(u),\quad u\in
H^{1}_{0}([0,L]),$
$\displaystyle\mathcal{F}(A)=\frac{1}{2}\lVert\partial_{x}A\rVert^{2}+\frac{1}{4}\lVert\lvert
A\rvert^{2}-1\rVert^{2}\,.$

Given that the main challenge with integrating this gradient flow numerically
is the uncontrollable divergence of the gradient $\partial_{x}A$, we consider
instead the maximizing gradient flow of the functional $\mathcal{F}$ in the
$H^{1}_{\epsilon}$-norm

$\lVert A\rVert_{H^{1}_{\epsilon}}^{2}:=\lVert
A\rVert^{2}+\epsilon\lVert\partial_{x}A\rVert^{2},\quad\epsilon>0,$

which can be written as:

$(\partial_{t}A,u)+\epsilon(\partial_{x}\partial_{t}A,\partial_{x}u)=\mathcal{F}^{\prime}(A)(u)\,.$

This regularization controls the ‘rate of ascent’ of $A$ in $H^{1}$ instead of
$L^{2}$, and so penalizes the rapid divergence of the gradient that
accompanies the introduction of spurious oscillatory modes. In strong form, it
yields the regularized PDE

(4.1)
$\partial_{t}A-\epsilon\partial_{x}^{2}\partial_{t}A=-\partial^{2}_{x}A+(\lvert
A\rvert^{2}-1)A\,.$

Proceeding with the numerical integration of (4.1), we assume that
$A(t^{k})=A^{k}$ and apply _operator splitting_ to do the integration in two
steps: first integrate the non-linear part

(4.2)
$\partial_{t}\hat{A}-(\lvert\hat{A}\rvert^{2}-1)\hat{A}=0,\quad\hat{A}(t^{k})=A^{k},$

to calculate $\hat{A}^{k+1}=\hat{A}(t^{k+1})$, $t^{k+1}=t^{k}+\tau$, and then
the linear part

(4.3) $(id-\epsilon\partial_{x}^{2})\partial_{t}A=-\partial_{x}^{2}A,\quad
A(t^{k})=\hat{A}^{k+1},$

to update $A$ to the next time step.

The first step (4.2) can be integrated exactly:

$\hat{A}(x,t)=\frac{A^{k}(x)}{\sqrt{\lvert A^{k}(x)\rvert^{2}+(1-\lvert
A^{k}(x)\rvert^{2})e^{2(t-t^{k})}}}\,.$

This blows up for $\lvert A^{k}(x)\rvert>1$ at the time

$t=t^{k}+\frac{1}{2}\log\left(\frac{\lvert A^{k}(x)\rvert^{2}}{\lvert
A^{k}(x)\rvert^{2}-1}\right)\,.$

It follows that

(4.4) $\hat{A}^{k+1}=\frac{A^{k}}{\sqrt{\lvert A^{k}\rvert^{2}+(1-\lvert
A^{k}\rvert^{2})e^{2\tau}}}\,,$

as long as the (global) time step $\tau$ is chosen so that

$\tau<\tau_{blow}=\inf_{x}\frac{1}{2}\log\left(\frac{\max(1,\lvert
A^{k}(x)\rvert^{2})}{\max(1,\lvert A^{k}(x)\rvert^{2})-1}\right)\,.$

Having selected a suitable $\tau$, in particular ensuring that
$\tau<\epsilon$, the second step (4.3) can be discretized implicitly

$(id-\epsilon\partial_{x}^{2})\frac{A-\hat{A}^{k+1}}{\tau}=-\partial_{x}^{2}A\\\
\Rightarrow
A^{k+1}=\left(id-(\epsilon-\tau)\partial_{xx}^{2}\right)^{-1}(id-\epsilon\partial_{xx}^{2})\hat{A}^{k+1}\,.$

The operator on the right-hand side is positive definite (for $\tau<\epsilon$)
and is straightforward to calculate numerically with spectral methods (FFT).
Moreover, all the eigenvalues of the operator have modulus smaller than 1,
which reaffirms the control of high frequency modes by the regularization.

In figure 2, we show the results of applying the scheme to the initial
condition $A_{0}(x)=5\sin^{3}(20\pi x/L)+2\sin^{3}(12\pi x/L)-\sin^{3}(4\pi
x/L)$ with $L=10$. We discretize the interval $[0,L]$ with 1000 uniformly
distributed points, and we track the evolution of $A$ up to the time
$t_{blow}$, when the adaptive time step $\tau\approx 0$ up to machine
precision ($\sim 10^{-16}$). We repeat the numerical experiment with different
values for the regularizing parameter
$\epsilon\in\\{0.5,0.1,0.05,0.01,0.005,0.001,0.0005\\}$, with figure 2
corresponding to $\epsilon=0.1$. The results are presented in figure 3, where
we see that the numerical solution blows up at earlier times $t_{blow}$ for
smaller values of $\epsilon$, with a measured rate of
$t_{blow}\sim\epsilon^{0.21}$ as $\epsilon\rightarrow 0$.

Figure 2. Evolution of an oscillating initial condition under the regularized
PDE $\eqref{Aeps}$ with $\epsilon=0.1$, via the operator splitting scheme
(4.2)-(4.3). The evolution is tracked up to the time $t_{blow}$, when the
adaptive time step $\tau\approx 0$ up to machine precision ($\sim 10^{-16}$).

Figure 3. Effect of varying the regularizing parameter $\epsilon$. In the top
plot, we see the norm $\lVert A\rVert$ blowing up at progressively earlier
times for smaller values of $\epsilon$. In the bottom plot, we look at the
log-log plot of blow-up time vs the parameter $\epsilon$, which appears to be
$t_{blow}\sim\epsilon^{0.21}$ as $\epsilon\rightarrow 0$.

##  5\. Determining functionals

In this section, we will show that finitely many linear functionals defined on
the phase space uniquely determine the asymptotic behavior of solutions.
To this end, first we derive a uniform a priory estimates of solutions to the
problem in the phase space $V^{1}:=H^{1}_{0}(\Omega)\times H^{1}_{0}(\Omega)$.
Multiplying the equation (1.1) with $-\Delta A^{\ast}$ in $L^{2}(\Omega)$ and
employing the Cauchy-Schwarz inequality we get

(5.1) $\frac{\tau}{2}\frac{d}{dt}\|\nabla A\|^{2}+\|\Delta
A\|^{2}+(\phi^{2},\lvert\nabla A\rvert^{2})+(|A|^{2},|\nabla A|^{2})\\\
\leq\|\nabla
A\|^{2}+2\lvert(A\partial_{x}A^{\ast},\phi\partial_{x}\phi)\rvert+2\lvert(A\partial_{y}A^{\ast},\phi\partial_{y}\phi)\rvert\\\
+2c_{1}\|\phi\partial_{y}A\|\|\Delta A\|+\lvert\beta\rvert\lVert
A\partial_{y}\phi\rVert\lVert\Delta A\rVert$

Utilizing the inequality

(5.2) $\|\nabla v\|^{2}\leq\|v\|\|\Delta v\|,\ \ \forall v\in
H^{2}(\Omega)\cap H_{0}^{1}(\Omega),$

the Ladyzhenskaya inequality

(5.3) $\|v\|_{L^{4}(\Omega)}^{2}\leq\sqrt{2}\|v\|\|\nabla v\|,\ \ \forall v\in
H_{0}^{1}(\Omega),$

the interpolation inequality,

(5.4) $\lVert\nabla v\rVert_{L^{4}(\Omega)}\leq d_{0}\lVert\Delta
v\rVert^{\frac{1}{2}}\lVert\nabla v\rVert^{\frac{1}{2}},\quad\forall v\in
H^{2}(\Omega)\cap H^{1}_{0}(\Omega),$

the Young’s inequality

(5.5)
$ab\leq\frac{\varepsilon}{p}a^{p}+\frac{1}{q\varepsilon^{\frac{1}{p-1}}}b^{q},\mbox{for
all}\ a,b,\varepsilon>0,\ \mbox{with}\ q=p/(p-1),1<p<\infty,$

we derive the following inequality from (5.1)

(5.6) $\frac{\tau}{2}\frac{d}{dt}\|\nabla A\|^{2}+\|\Delta
A\|^{2}+(\phi^{2},\lvert\nabla A\rvert^{2})+(|A|^{2},|\nabla A|^{2})\\\ \leq
3\lVert\Delta A\rVert^{2}+\frac{1}{4\varepsilon}\lVert
A\rVert^{2}+2|(A\partial_{x}A^{\ast},\phi\partial_{x}\phi)\rvert\\\
+2\lvert(A\partial_{y}A^{\ast},\phi\partial_{y}\phi)\rvert+\frac{c_{1}^{2}}{\varepsilon}\lVert\phi\partial_{y}A\rVert^{2}+\frac{\beta^{2}}{4\varepsilon}\lVert
A\partial_{y}\phi\rVert^{2}.$

We obtain the following estimates for the terms on the right hand side of
(5.6) by using the following inequality

(5.7)
$\nu_{0}\|\Delta\phi\|^{2}\leq\|\mathbb{L}\phi\|^{2}\leq\nu_{1}\|\Delta\phi\|^{2},\quad\nu_{0}>0,\quad\nu_{1}>0,$

in addition to (5.2), (5.3), (5.4) and (5.5):

$2\lvert(A\partial_{x}A^{\ast},\phi\partial_{x}\phi)\rvert+2\lvert(A\partial_{y}A^{\ast},\phi\partial_{y}\phi)\rvert\\\
\leq 4\lVert
A\rVert_{L^{4}(\Omega)}\lVert\phi\rVert_{L^{4}(\Omega)}\lVert\nabla\phi\rVert_{L^{4}(\Omega)}\lVert\nabla
A\rVert_{L^{4}(\Omega)}\quad\quad\quad\quad\quad\quad\\\ \leq 8\lVert
A\rVert^{\frac{1}{2}}\lVert\nabla
A\rVert^{\frac{1}{2}}\lVert\phi\rVert^{\frac{1}{2}}\lVert\nabla\phi\rVert^{\frac{1}{2}}\lVert\nabla\phi\rVert^{\frac{1}{2}}\lVert\Delta\phi\rVert^{\frac{1}{2}}\lVert\nabla
A\rVert^{\frac{1}{2}}\lVert\Delta A\rVert^{\frac{1}{2}}\\\
\leq\frac{\varepsilon}{4}\lVert\Delta\phi\rVert^{2}+\frac{\varepsilon}{4}\lVert\Delta
A\rVert^{2}+C(\varepsilon)\lVert A\rVert^{2}\lVert\phi\rVert^{2}\lVert\nabla
A\rVert^{4}+\frac{\lVert\nabla\phi\rVert^{4}}{4},\quad\quad\quad\ $
$\frac{c_{1}^{2}}{\varepsilon}\lVert\phi\partial_{y}A\rVert^{2}\leq\frac{c_{1}^{2}}{\varepsilon}\lVert\phi\rVert^{2}_{L^{4}(\Omega)}\lVert\nabla
A\rVert^{2}_{L^{4}(\Omega)}\\\
\leq\frac{c_{1}^{2}d_{0}^{2}}{\varepsilon}\lVert\phi\rVert^{2}_{L^{4}(\Omega)}\lVert\Delta
A\rVert\lVert\nabla
A\rVert\leq\frac{c_{1}^{2}d_{0}^{2}}{\varepsilon}\lVert\phi\rVert^{2}_{L^{4}(\Omega)}\lVert
A\rVert^{1/2}\lVert\Delta A\rVert^{3/2}\\\
\leq\frac{3\varepsilon}{4}\lVert\Delta
A\rVert^{2}+\frac{c_{1}^{8}d_{0}^{8}}{4\varepsilon^{7}}\lVert\phi\rVert^{8}_{L^{4}(\Omega)}\lVert
A\rVert^{2}\\\ \leq\frac{3\varepsilon}{4}\lVert\Delta
A\rVert^{2}+C(\varepsilon,c_{1},d_{0})\lVert\phi\rVert^{4}\lVert\nabla\phi\rVert^{4}\lVert
A\rVert^{2}.$

By following the same sequence of steps, we can similarly get

$\displaystyle\frac{\beta^{2}}{4\varepsilon}\lVert
A\partial_{y}\phi\rVert^{2}$ $\displaystyle\leq$
$\displaystyle\frac{\beta^{2}}{4\varepsilon}\lVert
A\rVert^{2}_{L^{4}(\Omega)}\lVert\nabla\phi\rVert^{2}_{L^{4}(\Omega)}$
$\displaystyle\leq$
$\displaystyle\frac{3\varepsilon}{4}\lVert\Delta\phi\rVert^{2}+C(\varepsilon,\beta,d_{0})\lVert
A\rVert^{4}\lVert\nabla A\rVert^{4}\lVert\phi\rVert^{2}.$

Thus we infer from (5.6) that

(5.8) $\frac{\tau}{2}\frac{d}{dt}\|\nabla A\|^{2}+(1-4\varepsilon)\|\Delta
A\|^{2}+(\phi^{2},\lvert\nabla A\rvert^{2})+(|A|^{2},|\nabla A|^{2})\\\
\leq\frac{\varepsilon}{4\nu_{0}}\lVert\mathbb{L}\phi\rVert^{2}+\frac{1}{4\varepsilon}\lVert
A\rVert^{2}+C(\varepsilon)\lVert A\rVert^{2}\lVert\phi\rVert^{2}\lVert\nabla
A\rVert^{4}+\frac{\lVert\nabla\phi\rVert^{4}}{4}\\\
+C(\varepsilon,c_{1},d_{0})\lVert\phi\rVert^{4}\lVert\nabla\phi\rVert^{4}\lVert
A\rVert^{2}+C(\varepsilon,\beta,d_{0})\lVert A\rVert^{4}\lVert\nabla
A\rVert^{4}\lVert\phi\rVert^{2}.$

Next, we multiply the equation (1.2) in $L^{2}(\Omega)$ with
$-\mathbb{L}\phi$:

(5.9)
$\frac{1}{2}\frac{d}{dt}(\mathbb{L}\phi,\phi)+\|\mathbb{L}\phi\|^{2}+h(\mathbb{L}\phi,\phi)\\\
\leq(|\phi||A|^{2},|\mathbb{L}\phi|)+|c_{2}|(|A|\lvert\partial_{y}A\rvert,|\mathbb{L}\phi|)$

Arguing similarly to the calculation of the estimate (5.8) and employing the
interpolation inequality

$\|v\|_{L^{6}(\Omega)}\leq d_{1}\|\nabla
v\|^{\frac{2}{3}}\|v\|^{\frac{1}{3}},$

we obtain:

$\displaystyle(\lvert\phi\rvert\lvert A\rvert^{2},\lvert\mathbb{L}\phi\rvert)$
$\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+\frac{1}{4\varepsilon}\lVert\phi\rVert_{L^{6}}^{2}\lVert
A\rVert_{L^{6}}^{4}$ $\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+\frac{d_{1}^{6}}{4\varepsilon}\lVert\nabla\phi\rVert^{4/3}\lVert\phi\rVert^{2/3}\lVert\nabla
A\rVert^{8/3}\lVert A\rVert^{4/3}$ $\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+C(\varepsilon,d_{1})(\lVert\phi\rVert^{2}\lVert\nabla\phi\rVert^{4}+\lVert
A\rVert^{2}\lVert\nabla A\rVert^{4}),$ $\displaystyle\lvert c_{2}\rvert(\lvert
A\rvert\lvert\partial_{y}A\rvert,\lvert\mathbb{L}\phi\rvert)$
$\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+\frac{c_{2}^{2}}{4\varepsilon}(\lvert
A\rvert^{2},\lvert\partial_{y}A\rvert^{2})$ $\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+\frac{c_{2}^{2}}{4\varepsilon}\lVert
A\rVert_{L^{4}(\Omega)}\lVert\partial_{y}A\rVert^{2}_{L^{4}(\Omega)}$
$\displaystyle\leq$
$\displaystyle\varepsilon\lVert\mathbb{L}\phi\rVert^{2}+\frac{\sqrt{2}c_{2}^{2}d_{0}^{2}}{4\varepsilon}\lVert
A\rVert\lVert\nabla A\rVert^{2}\lVert\Delta A\rVert$ $\displaystyle\leq$
$\displaystyle\varepsilon(\lVert\mathbb{L}\phi\rVert^{2}+\lVert\Delta
A\rVert^{2})+C(\varepsilon,c_{2},d_{0})\lVert A\rVert^{2}\lVert\nabla
A\rVert^{2}.$

By using the above estimates, we infer from (5.9)

(5.10)
$\frac{1}{2}\frac{d}{dt}(\mathbb{L}\phi,\phi)+(1-2\varepsilon)\lVert\mathbb{L}\phi\rVert^{2}-\varepsilon\lVert\Delta
A\rVert^{2}+h(\mathbb{L}\phi,\phi)\\\ \leq
C(\varepsilon,d_{1})(\lVert\phi\rVert^{2}\lVert\nabla\phi\rVert^{4}+\lVert
A\rVert^{2}\lVert\nabla A\rVert^{4})+C(\varepsilon,c_{2},d_{0})\lVert
A\rVert^{2}\lVert\nabla A\rVert^{4}.$

Finally adding (5.8) and (5.10) we get

(5.11) $\frac{d}{dt}\left[\frac{\tau}{2}\|\nabla
A\|^{2}+\frac{1}{2}(\mathbb{L}\phi,\phi)\right]+(1-5\varepsilon)\|\Delta
A\|^{2}\\\
+(1-2\varepsilon-\frac{\varepsilon}{4\nu_{0}})\lVert\mathbb{L}\phi\rVert^{2}+(\phi^{2},\lvert\nabla
A\rvert^{2})+(\lvert A\rvert^{2},\lvert\nabla A\rvert^{2})\\\
\leq\frac{1}{4\varepsilon}\lVert A\rVert^{2}+C(\varepsilon)\lVert
A\rVert^{2}\lVert\phi\rVert^{2}\lVert\nabla
A\rVert^{4}+\frac{\lVert\nabla\phi\rVert^{4}}{4}+C(\varepsilon,c_{1},d_{0})\lVert\phi\rVert^{4}\lVert\nabla\phi\rVert^{4}\lVert
A\rVert^{2}\\\ +C(\varepsilon,\beta,d_{0})\lVert A\rVert^{4}\lVert\nabla
A\rVert^{4}\lVert\phi\rVert^{2}+C(\varepsilon,d_{1})(\lVert\phi\rVert^{2}\lVert\nabla\phi\rVert^{4}+\lVert
A\rVert^{2}\lVert\nabla A\rVert^{4})\\\ +C(\varepsilon,c_{2},d_{0})\lVert
A\rVert^{2}\lVert\nabla A\rVert^{4}$

Since the semigroup generated by the problem is dissipative in $V^{0}$ there
exist $T_{0}>0$ and $M_{0}>0$ such that

$\lVert A(t)\rVert^{2}+\lVert\phi(t)\rVert^{2}\leq M_{0},\quad\forall t\geq
T_{0}.$

Thus, by choosing
$\varepsilon=\min\\{\frac{1}{10},\frac{2\nu_{0}}{8\nu_{0}+1}\\}$ we obtain
from (5.11) that

$\frac{d}{dt}[\tau\lVert\nabla A\rVert^{2}+(\mathbb{L}\phi,\phi)]+\lVert\Delta
A\rVert^{2}\lVert\mathbb{L}\phi\rVert^{2}\\\ \leq\frac{1}{4\varepsilon}\lVert
A\rVert^{2}+C(\varepsilon,M_{0},\beta,d_{0},d_{1})(\lVert\nabla
A\rVert^{4}+\lVert\nabla\phi\rVert^{4}).$

Finally, by using the uniform Gronwall Lemma 2.3, we infer from the last
inequality that there exists $R_{0}>0$, depending only on the parameters of
the system, such that

(5.12) $\|\nabla A\|^{2}+\|\nabla\phi\|^{2}\leq R_{0},\ \ \forall t\geq
T_{1},$

i.e., the semigroup generated by the problem is bounded dissipative in
$V^{1}=H_{0}^{1}(\Omega)\times H_{0}^{1}(\Omega)$.
It remains to show that the condition (2.4) is satisfied, i.e.,

(5.13) $\|f_{j}(a,\Phi)\|_{H^{-\gamma}}\leq M(R_{0})(\|\nabla
a\|+\|\nabla\Phi\|),\ \ j=1,2,$

with some $\gamma\in(0,1)$.

In fact, employing the estimate (5.12), the following inequality (see, e.g.,
[1], Chp. 1)

$\|v\|_{H^{-\delta}(\Omega)}\leq d_{0}\|v\|_{L^{2/(1+\delta)}(\Omega)}$

which is valid for $\delta\in(0,1)$, and the fact that $H^{1}(\Omega)$ is
continuously embedded into $L^{\frac{2}{\delta}}(\Omega)$ we can bound the
last term on the right hand side of (3.4)

$\|i\beta[\tilde{A}\partial_{y}\Phi]\|_{H^{-\delta}(\Omega)}\leq
d_{0}|\beta|\|\tilde{A}\partial_{y}\Phi\|_{L^{2/(1+\delta)}}\\\ \leq
d_{0}|\beta|\|\tilde{A}\|_{L^{2/\delta}}\lVert\partial_{y}\Phi\rVert\leq
Cd_{0}\|\nabla\tilde{A}\|\|\partial_{y}\Phi\|\leq C(R_{0})\|\nabla\Phi\|.$

Similarly we can show that

$\|i\beta[a\partial_{y}\phi]\|_{H^{-\delta}(\Omega)}\leq C(R_{0})\|\nabla
a\|.$

Utilizing the same arguments we can get necessary bounds for all terms on the
right-hand side of (3.4) and (3.5) and obtain the desired estimates (5.13).
Thus thanks to the Theorem 2.6 we have

######  Theorem 5.1.

The set ${\mathcal{L}}$ of functionals on $V^{1}$ is asymptotically
$(V^{1},V^{0},\mathcal{W})$ determining functionals for the problem
(1.1)-(1.4) provided the defect $\varepsilon(V^{1},V^{0})$ is small enough.

######  Remark 5.2.

Similar to the proof of the Theorem 5.1 we can prove that the set
${\mathcal{L}}$ of functionals on $V^{1}$ is asymptotically
$(V^{1},V^{0},\mathcal{W})-$ determining functionals for the problem 1D
problem

(5.14)
$\begin{cases}\tau\partial_{t}A=A+\partial_{x}^{2}A-\phi^{2}A-|A|^{2}A,\
x\in(0,L),t>0,\\\ \partial_{t}\phi=\partial_{x}^{2}\phi-h\phi+\phi\lvert
A\rvert^{2},\ x\in(0,L),t>0,\\\
A\big{|}_{x=0}=A\big{|}_{x=L}=\phi\big{|}_{x=0}=\phi\big{|}_{x=L}=0,\\\
A\big{|}_{t=0}=A_{0},\quad\phi\big{|}_{t=0}=\phi_{0}\end{cases}$

provided the defect $\varepsilon(V^{1},{\bf H})$ is small enough. Let us note
that dissipativity of the semigroup $S(t),t\geq 0$ generated by the problem
(5.14) , as well as existence of and exponential attractor for the semigroup
$S(t),t\geq 0$ is established in [20].

## References

  * [1] A. V. Babin and M. I. Vishik, _Attractors of evolution equations_ , Studies in Mathematics and its Applications, vol. 25, North-Holland Publishing Co., Amsterdam, 1992, Translated and revised from the 1989 Russian original by Babin. MR 1156492
  * [2] C. Bardos and L. Tartar, _Sur l’unicité rétrograde des équations paraboliques et quelques questions voisines_ , Arch. Rational Mech. Anal. 50 (1973), 10–25. MR 338517
  * [3] Á. Buka, T. Börzsönyi, N. Éber, and T. Tóth-Katona, _Patterns in the bulk and at the interface of liquid crystals_ , Coherent Structures in Complex Systems (D. Reguera, J. M. Rubí, and L. L. Bonilla, eds.), Lecture Notes in Physics, vol. 567, Springer, Berlin, Heidelberg, 2001, p. 125.
  * [4] A. Yu. Chebotarev, _Finite-dimensional controllability of systems of Navier-Stokes type_ , Differ. Uravn. 46 (2010), no. 10, 1495–1503. MR 2798716
  * [5] I. D. Chueshov, _A remark on sets of defining elements for reaction-diffusion systems_ , Mat. Zametki 63 (1998), no. 5, 774–784. MR 1683651
  * [6] by same author, _The theory of functionals that uniquely determine the asymptotic dynamics of infinite-dimensional dissipative systems_ , Uspekhi Mat. Nauk 53 (1998), no. 4(322), 77–124. MR 1668050
  * [7] Igor Chueshov, _Dynamics of quasi-stable dissipative systems_ , Universitext, Springer, Cham, 2015. MR 3408002
  * [8] Igor Chueshov and Irena Lasiecka, _Long-time behavior of second order evolution equations with nonlinear damping_ , Mem. Amer. Math. Soc. 195 (2008), no. 912, viii+183. MR 2438025
  * [9] Igor D. Chueshov and Varga K. Kalantarov, _Determining functionals for nonlinear damped wave equations_ , Mat. Fiz. Anal. Geom. 8 (2001), no. 2, 215–227. MR 1859881
  * [10] B. Cockburn, D. A. Jones, and E. S. Titi, _Determining degrees of freedom for nonlinear dissipative equations_ , C. R. Acad. Sci. Paris Sér. I Math. 321 (1995), no. 5, 563–568. MR 1356554
  * [11] by same author, _Estimating the number of asymptotic degrees of freedom for nonlinear dissipative systems_ , Math. Comp. 66 (1997), no. 219, 1073–1087. MR 1415799
  * [12] R. Dascaliuc, _On backward-time behavior of the solutions to the 2-d space periodic navier–stokes equations_ , Annales de l’Institut Henri Poincaré C, Analyse non linéaire, vol. 22, 2005, pp. 385–401.
  * [13] C. Foiaş and G. Prodi, _Sur le comportement global des solutions non-stationnaires des équations de Navier-Stokes en dimension $2$_, Rend. Sem. Mat. Univ. Padova 39 (1967), 1–34. MR 223716
  * [14] C. Foias, O. Manley, R. Rosa, and R. Temam, _Navier-Stokes equations and turbulence_ , Encyclopedia of Mathematics and its Applications, vol. 83, Cambridge University Press, Cambridge, 2001. MR 1855030
  * [15] Y. Guo and E. S. Titi, _On the backward behavior of some dissipative evolution equations_ , Phys. D 306 (2015), 34–47. MR 3367572
  * [16] J. K. Hale and G. Raugel, _Regularity, determining modes and Galerkin methods_ , J. Math. Pures Appl. (9) 82 (2003), no. 9, 1075–1136. MR 2012806
  * [17] M. J. Holst and E. S. Titi, _Determining projections and functionals for weak solutions of the Navier-Stokes equations_ , Recent developments in optimization theory and nonlinear analysis (Jerusalem, 1995), Contemp. Math., vol. 204, Amer. Math. Soc., Providence, RI, 1997, p. 125.
  * [18] S. Kai, K. Hayashi, and Y. Hidaka, _Pattern forming instability in homeotropically aligned liquid crystals_ , The Journal of Physical Chemistry 100 (1996), no. 49, 19007–19016.
  * [19] H. Kalantarova, V. Kalantarov, and O. Vantzos, _Global behavior of solutions to chevron pattern equations_ , J. Math. Phys. 61 (2020), no. 6, 061511, 13. MR 4112645
  * [20] by same author, _Chevron pattern equations: exponential attractor and global stabilization_ , Vietnam J. Math. 49 (2021), no. 3, 901–918. MR 4298702
  * [21] S. Komineas, H. Zhao, and L. Kramer, _Modulated structures in electroconvection in nematic liquid crystals_ , Physical Review E 67 (2003), no. 3, 031701.
  * [22] Igor Kukavica and Mehmet Malcok, _Backward behavior of solutions of the Kuramoto-Sivashinsky equation_ , J. Math. Anal. Appl. 307 (2005), no. 2, 455–464. MR 2142437
  * [23] O. A. Ladyzhenskaya, _Finding minimal global attractors for the Navier-Stokes equations and other partial differential equations_ , Uspekhi Mat. Nauk 42 (1987), no. 6(258), 25–60, 246. MR 933994
  * [24] S. Nasuno, O. Sasaki, S. Kai, and W. Zimmermann, _Secondary instabilities in electroconvection in nematic liquid crystals_ , Physical Review A 46 (1992), no. 8, 4954.
  * [25] H. Richter, A. Buka, and I. Rehberg, _Electrohydrodynamic convection in a homeotropically aligned nematic sample_ , Physical Review E 51 (1995), no. 6, 5886.
  * [26] A. G. Rossberg, _The amplitude formalism for pattern forming systems with spontaneously broken isotropy and some applications_ , Ph.D. thesis, Universität Bayreuth, Fakultät für Mathematik, Physik und Informatik, 1998.
  * [27] A. G. Rossberg, A. Hertrich, L. Kramer, and W. Pesch, _Weakly nonlinear theory of pattern-forming systems with spontaneously broken isotropy_ , Physical review letters 76 (1996), no. 25, 4729.
  * [28] A. G. Rossberg and L. Kramer, _Pattern formation from defect chaos—a theory of chevrons_ , Physica D: Nonlinear Phenomena 115 (1998), no. 1-2, 19–28.
  * [29] H. Sakaguchi and A. Matsuda, _Chevron patterns and defect lattices in an anisotropic model for electroconvection_ , Phys. D 238 (2009), no. 1, 1–8. MR 2571961
  * [30] R. Temam, _Infinite-dimensional dynamical systems in mechanics and physics_ , 2 ed., Applied Mathematical Sciences, vol. 68, Springer-Verlag, New York, 1997.
  * [31] J. Vukadinović, _On the backwards behavior of the solutions of the 2D periodic viscous Camassa-Holm equations_ , J. Dynam. Differential Equations 14 (2002), no. 1, 37–62. MR 1878244


# Fast non-autoregressive inverse folding
with discrete diffusion

John J. Yang
Massachusetts Institute of Technology
<EMAIL_ADDRESS>
&Jason Yim
Massachusetts Institute of Technology
<EMAIL_ADDRESS>&Regina Barzilay
Massachusetts Institute of Technology
<EMAIL_ADDRESS>&Tommi Jaakkola
Massachusetts Institute of Technology
<EMAIL_ADDRESS>
###### Abstract

Generating protein sequences that fold into a intended 3D structure is a
fundamental step in de novo protein design. De facto methods utilize
autoregressive generation, but this eschews higher order interactions that
could be exploited to improve inference speed. We describe a non-
autoregressive alternative that performs inference using a constant number of
calls resulting in a 23 times speed up without a loss in performance on the
CATH benchmark. Conditioned on the 3D structure, we fine-tune ProteinMPNN to
perform discrete diffusion with a purity prior over the index sampling order.
Our approach gives the flexibility in trading off inference speed and accuracy
by modulating the diffusion speed. Code:
https://github.com/johnyang101/pmpnndiff

##  1 Introduction

De novo protein design aims to design proteins from first principles without
modifying an existing protein [7]. This involves designing protein 3D
structures for a desired function such as binding then determining the
sequence that would fold in-vivo into the designed structure. The first step
is the protein structure generation problem for which RFdiffusion has become
state-of-the-art [15]. The aim of this work is to develop a discrete diffusion
model towards the second step of sampling sequences conditioned to fold into a
desired 3D structure, referred to as inverse (protein) folding [6].

Generative models have already become the preferred tool for inverse folding:
ProteinMPNN [3] is a widely used method with successful experimental
validation. However, ProteinMPNN has two limitations. The first is its
autoregressive decoding which scales linearly and can be prohibitively slow
for large proteins. The second is the uniformly random decoding order which is
likely suboptimal given substantial evidence of higher-order interactions
occurring in protein evolution [13].

Inspired by RFdiffusion, in which a pre-trained protein folding model is fine-
tuned with diffusion, we use a pre-trained ProteinMPNN and fine-tune it with
diffusion. We explore multiple variants of discrete diffusion and find the
best configuration to result in equivalent performance as ProteinMPNN on
foldability while using 23 times less compute. Our best diffusion model is
depicted in Figure 1. Our contribution enables exciting possibilities to
extend ProteinMPNN using conditional diffusion models for improvement in both
speed and controllable generation.

The paper is structured as follows. Related work is provided in Appendix A.
Section 2 describes our method of fine-tuning ProteinMPNN with discrete
diffusion. Section 3 evaluates each diffusion variant on the CATH benchmark
[12]. We focus on analyzing tradeoffs in diversity, speed, and designability
of sequence designs. Section 4 concludes with limitations and future
directions.

Figure 1: Overview. Starting from a sequence and structure, the forward
process masks residues on each step. The reverse process unmasks on each step
given only the structure and all residues initializd to [MASK]. Our framework
allows for flexible decoding strategies by striding over intervals of time
$\nabla t$ in each reverse step, i.e. the diagram uses 3 steps to decode the
whole sequence.

##  2 Method

We first provide background on the problem formulation and ProteinMPNN in
section 2.1. Next, section 2.2 describes discrete diffusion and its variants
we implement for fine-tuning ProteinMPNN.

###  2.1 Background

Problem formulation. The task of inverse folding is to design a sequence
$\bm{s}\in\mathcal{V}^{L}$ that folds into a given backbone structure
$\bm{x}\in\mathbb{R}^{L\times 4\times 3}$ where $\mathcal{V}$ is the
vocabulary of 20 amino acids plus a mask token [MASK]and $L$ is the number of
residues. Superscripts $\bm{s}^{(t)}$ will be used to refer to time
$t\in[1,\dots,T]$ while subscripts $s_{i}$ refer to residues $i$ (i.e. index).
Here, $\bm{x}$ refers to the atomic coordinates of the 5 canonical backbone
atoms $\\{N,C_{\alpha},C,C_{\beta},O\\}$. It has been well-documented that
sequences with as low as 30% sequence similarity can fold into the same
structure [10]. Therefore, the goal is to learn a distribution of possible
sequences that fold into a structure, $p(\bm{s}|\bm{x})$.

ProteinMPNN. First described in Dauparas et al. [3], we provide a brief
summary. Each residue is represented as a node in an attributed graph
$\mathcal{G}(\bm{x})=(\mathcal{V}(\bm{x}),\mathcal{E}(\bm{x}))$.
$\mathcal{V}(\bm{x})$ encodes geometric features such as orientation and
sequence index of each residue; $\mathcal{E}(\bm{x})$ constructs a $k$-nearest
neighbor graph based on $C_{\alpha}$ distance and encodes relative geometric
features. The neural network uses message passing to learn embeddings of
protein geometry through a masked language modeling objective that was first
described in Ingraham et al. [8]. Inference is performed by sampling a
uniformly random decoding order then autoregressively decoding residues one by
one.

###  2.2 Discrete diffusion

We provide a overview of discrete diffusion models (D3PM) described in Austin
et al. [1]. Discrete diffusion models are a class of latent variable
generative models that are defined by a fixed forward Markov process
$q(\bm{s}^{(1:T)}|\bm{s}^{(0)})=\prod_{t=1}^{T}q(\bm{s}^{(t)}|\bm{s}^{(t-1)})q(\bm{s}^{(0)})$
from a starting sequence $\bm{s}^{(0)}$ and sequence of increasingly noisy
latent variables $\bm{s}^{(1:T)}=(\bm{s}^{(1)},\dots,\bm{s}^{(T)})$. The goal
is to learn a model that parameterizes the reverse process
$p_{\theta}(\bm{s}^{(0:T)},\bm{x})=\prod_{t=1}^{T}p_{\theta}(\bm{s}^{(t-1)}|\bm{s}^{(t)})$
where,

$\displaystyle
p_{\theta}(\bm{s}^{(t-1)}|\bm{s}^{(t)})=\sum_{\bm{s}^{(0)}}q(\bm{s}^{(t-1)}|\bm{s}^{(t)},\bm{s}^{(0)})p_{\theta}(\bm{s}^{(0)}|\bm{s}^{(t)}).$
(1)

Thus, a model with weights $\theta$ learns how to denoise by predicting
$p_{\theta}(\bm{s}^{(0)}|\bm{s}^{(t)})$. During training, we optimize the
evidence-weighted lower bound (ELBO) between
$q(\bm{s}^{(t-1)}|\bm{s}^{(t)},\bm{s}^{(0)})$ and
$p(\bm{s}_{t-1}|\bm{s}_{t})$. For absorbing state diffusion, the ELBO reduces
to the cross-entropy loss over masked residues,

$\displaystyle\mathcal{L}=\mathbb{E}_{q(\bm{s}^{(0)})}\left[\sum_{t=1}^{T}\frac{T-t+1}{T}\mathbb{E}_{q(\bm{s}^{(t)}|\bm{s}^{(0)})}\left[\sum_{i=1}^{L}\mathds{1}\\{s^{(t)}_{i}=\texttt{[MASK]}\\}\log{p_{\theta}\left(\hat{s}^{(0)}_{i}|\bm{s}^{(t)}\right)}\right]\right].$
(2)

Following Bond-Taylor et al. [2], each inner expectation is re-weighted to
take into account the difficulty of denoising steps. Discrete diffusion
depends on the noising process using a categorical distribution over each
residue
$q(s^{(t)}_{i}|s^{(t-1)}_{i})=\text{Cat}(s^{(t)}_{i};\bm{p}=s^{(t-1)}_{i}\bm{Q}_{t})$
that is parameterized by the probabilities $\bm{p}$ of each category where
$\bm{Q}_{t}$ is a doubly stochastic transition matrix with $s_{i}^{(t-1)}$
represented as a one-hot encoding. Each residue is noised independently. We
can derive a closed form for sampling the $t$-step marginal and reverse step,

$\displaystyle q(s_{i}^{(t)}|s_{i}^{(0)})$
$\displaystyle=\text{Cat}\left(s^{(t)}_{i};\bm{p}=s^{(0)}_{i}\overline{\bm{Q}}_{t}\right),\quad\text{where}\quad\overline{\bm{Q}}_{j}=\prod_{j=1}^{t}\bm{Q}_{j}$
(3) $\displaystyle q(s^{(t-1)}_{i}|s^{(t)}_{i},s_{i}^{(0)})$
$\displaystyle=\text{Cat}\left(s^{(t-1)}_{i};\bm{p}=\frac{s^{(t)}_{i}\bm{Q}_{t}^{\top}\odot
s^{(0)}_{i}\overline{\bm{Q}}_{t-1}}{s^{(0)}_{i}\overline{\bm{Q}}_{t}(s^{(t)}_{i})^{\top}}\right).$
(4)

We choose to use the absorbing state transition matrix,

$\displaystyle\bm{Q}_{t}=(1-\beta_{t})\mathds{I}+\beta_{t}\mathds{1}e_{m}^{\top}$
(5)

where $\beta_{t}$ controls the masking rate, the vocabulary is extended with a
mask token, and $e_{m}$ is a vector with 1 on the index of the mask token and
0 elsewhere. On each forward step, a percentage of the unmasked tokens
transition to mask and stay masked until the final step, where all tokens are
masked. The reverse process performs the opposite with unmasking starting with
residue as [MASK].

###  2.3 Diffusion improvements to ProteinMPNN

ProteinMPNN’s autoregressive sampling can be seen as an absorbing state
diffusion that unmasks one residue on each reverse step with a uniformly
random decoding order. Given this perspective, we sought to extend ProteinMPNN
to be more efficient with discrete diffusion. In the context of inverse
folding, diffusion is conditioned on the structure,
$p_{\theta}(\bm{s}_{t-1}|\bm{s}_{t},\bm{x})$. Borrowing techniques from Bond-
Taylor et al. [2], we first fine-tune ProteinMPNN with non-autoregressive
diffusion training and strided sampling. Second, we utilize more informed
sampling orders based on a purity prior [14].

#### Non-autoregressive diffusion.

We fine-tune ProteinMPNN with absorbing state diffusion where a subset of the
residues are masked/unmasked on each step. We choose a linear schedule for
masking where, for a length $L$ protein, approximately
$\lfloor\frac{t}{T}L\rfloor$ will be masked at time $t$. The $t$-step marginal
probability $\bm{p}=s_{i}^{(0)}\overline{\bm{Q}}_{t}$ in eq. 3 is then,

$s_{i}^{(0)}\overline{\bm{Q}}_{t}=\prod_{j=1}^{t}(1-\beta_{j})s^{(0)}_{i}+\prod_{j=1}^{t}\beta_{j}\mathds{1}e_{m}^{\top}=\left(1-\frac{t}{T}\right)s^{(0)}_{i}+\frac{t}{T}\mathds{1}e_{m}^{\top}$
(6)

where $\beta_{t}$ is set such that $t/T=\prod_{j=1}^{t}\beta_{j}$. Training
proceeds using eq. 3 to noise and $\mathcal{L}$ eq. 2 as the loss. To sample,
all residues are initially masked then $\lfloor\frac{1}{T}L\rfloor$ residues
are unmasked on each step according to a decoding order (to be discussed
soon). Using $T$ steps during sampling can still be prohibitively slow. We
explored efficient sampling where on each step we decode $\lfloor\frac{\nabla
t}{T}L\rfloor$ residues and only require $\lceil\frac{\nabla t}{T}\rceil$
steps where $\nabla t$ is an integer value representing the stride interval. A
depiction of strided sampling is in Figure 1.

#### Purity Prior.

Since structure is correlated with evolutionary couplings [5], we sought to
bias the masking/unmasking order based on couplings strengths, but our
preliminary attempts were not successful. Instead, we utilize a purity prior
to bias the order. The purity of residue index $i\in\\{1,\dots,L\\}$ at step
$t$ is defined as

$\displaystyle\mathfrak{p}(i,t)=\max_{j\in\\{1,\dots,L\\}}p_{\theta}((\bm{s}_{0})_{i}=j|\bm{s}_{t})$
(7)

Purity can be thought of as the model’s confidence in its prediction at a
given index relative to the other indices. We hypothesized ProteinMPNN would
be more confident about jointly predicting coupled residues. On each
forward/reverse step, we perform importance sampling based on $\mathfrak{p}$
to determine the next location to mask/unmask (see Tang et al. [14]).

##  3 Experiments

To evaluate ProteinMPNN diffusion, we report results on the CATH 4.2 Single
Chain benchmark previously reported in ProteinMPNN.

#### Metrics.

Following ProteinMPNN, we sample 8 sequences for each structure in the CATH
test set and calculate the following metrics. The most common metric is
sequence recovery, defined as percentage of correct amino acids relative to
the native sequence, as the primary metrics [6, 4, 19]. Designability is
defined as the RMSD error of a pre-trained protein folding model (we use
ESMFold [11]) to predict the intended structure from the sampled sequence.
Multiple works have found designability to correlate with experimental success
[3, 15, 16], such evidence does not exist for sequence recovery. Therefore, we
designate designability as the main metric to optimize.

In addition, we report diversity as the average pairwise Levenshtein distance
between sequences for a structure. Lastly, speed is the average wall clock
time (in seconds) to sample one sequence per protein in the test set using a
single NVIDIA V100.

#### Diffusion fine-tuning.

We set $T=100$ and use $\nabla t=20$ when using strided sampling for a total
of 5 model calls to sample any sequence. Depending on the protein length, each
model call will have variable wall clock time. Dropout is set to 0.1 during
training. The Adam optimizer is initialized with a learning rate of $10^{-4}$.
We first pre-train ProteinMPNN with the standard next-token prediction
objective as done in Dauparas et al. [3] on the CATH single chain training set
for 200 epochs with batch size of 10,000 tokens. ProteinMPNN is then fine
tuned using the diffusion objective (Equation 2) for 1000 epochs with a batch
size of 5k tokens. Models were trained on NVIDIA RTX A6000-48GB.

###  3.1 Results

Our results are presented in Table 1. Using the same model, we evaluate three
different sampling variants based on whether purity order and strided sampling
$(\nabla t>1)$ are used. Our baseline does not include purity or strided
sampling. Using purity to determine the decoding order results in improvement
over recovery and designability but a drop in diversity. Including strided
sampling gives a 10 times improvement in speed. In comparison to ProteinMPNN,
we find sequence recovery is worse with diffusion but designability remains on
par with a slight drop in diversity. The advantage with diffusion is clear: a
23 times speed up with a small drop in diversity and designability.

Table 1: Performance of discrete diffusion variants. | Purity | $\nabla t$ | Recovery ($\uparrow$) | Diversity ($\uparrow$) | Designability ($\downarrow$) | Speed ($\downarrow$)
---|---|---|---|---|---|---
Diffusion | ✗ | 1 | 33.6 % | 0.624 | 2.855 | 355.7
✓ | 1 | 39.6 % | 0.424 | 2.158 | 329.8
✓ | 20 | 39.9 % | 0.420 | 2.112 | 33.7
ProteinMPNN |  |  | 47.9 % | 0.386 | 2.007 | 768.3

##  4 Discussion

The development of efficient and accurate methods for inverse protein folding
remains a cornerstone task in de novo protein design. We introduced a
diffusion approach that fine-tunes a state-of-the-art inverse folding model to
significantly accelerates the inference process. Our work has several
limitations. First, training ProteinMPNN diffusion without pre-training
performs worse than fine-tuning. The reasons for this are unclear. Second, our
non-autoregressive generation approach does not edit previously generated
residues, making it prone to error accumulation. Ideally, the model would be
able to iteratively refine the whole sequence like in other non-autoregressive
sequence generation works. Third, we utilize a discrete time diffusion process
that is incompatible with continuous time stochastic differential equations
(SDE). An interesting direction would to explore the continuous time
formulations. Lastly, we were not able to find correlations between purity
decoding order with structural characteristics, i.e. residues close in space
should decode together. We plan to investigate further into optimal decoding
orders and analyze its relation with structure.

## References

  * Austin et al. [2021] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg.  Structured denoising diffusion models in discrete state-spaces.  _Advances in Neural Information Processing Systems_ , 34:17981–17993, 2021.
  * Bond-Taylor et al. [2021] Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P. Breckon, and Chris G. Willcocks.  Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes, 2021.
  * Dauparas et al. [2022] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker.  Robust deep learning–based protein sequence design using proteinmpnn.  _Science_ , 378(6615):49–56, 2022.  doi: 10.1126/science.add2187.  URL https://www.science.org/doi/abs/10.1126/science.add2187.
  * Gao et al. [2022] Zhangyang Gao, Cheng Tan, and Stan Z Li.  Pifold: Toward effective and efficient protein inverse folding.  _arXiv preprint arXiv:2209.12643_ , 2022.
  * Hopf et al. [2014] Thomas A Hopf, Charlotta PI Schärfe, João PGLM Rodrigues, Anna G Green, Oliver Kohlbacher, Chris Sander, Alexandre MJJ Bonvin, and Debora S Marks.  Sequence co-evolution gives 3d contacts and structures of protein complexes.  _elife_ , 3:e03430, 2014.
  * Hsu et al. [2022] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives.  Learning inverse folding from millions of predicted structures.  In _International Conference on Machine Learning_ , pages 8946–8970. PMLR, 2022.
  * Huang et al. [2016] Po-Ssu Huang, Scott E Boyken, and David Baker.  The coming of age of de novo protein design.  _Nature_ , 537(7620):320–327, 2016.
  * Ingraham et al. [2019] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola.  Generative models for graph-based protein design.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/f3a4ff4839c56a5f460c88cce3666a2b-Paper.pdf.
  * Jing et al. [2021] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael J. L. Townshend, and Ron Dror.  Learning from protein structure with geometric vector perceptrons, 2021.
  * Krissinel [2007] Evgeny Krissinel.  On the relationship between sequence and structure similarities in proteomics.  _Bioinformatics_ , 23(6):717–723, 2007.
  * Lin et al. [2023] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives.  Evolutionary-scale prediction of atomic-level protein structure with a language model.  _Science_ , 379(6637):1123–1130, 2023.  doi: 10.1126/science.ade2574.  URL https://www.science.org/doi/abs/10.1126/science.ade2574.
  * Orengo et al. [1997] Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton.  Cath–a hierarchic classification of protein domain structures.  _Structure_ , 5(8):1093–1109, 1997.
  * Starr and Thornton [2016] Tyler N Starr and Joseph W Thornton.  Epistasis in protein evolution.  _Protein science_ , 25(7):1204–1218, 2016.
  * Tang et al. [2023] Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.  Improved vector quantized diffusion models, 2023.
  * Watson et al. [2023] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al.  De novo design of protein structure and function with rfdiffusion.  _Nature_ , pages 1–3, 2023.
  * Wicky et al. [2022] BIM Wicky, LF Milles, A Courbet, RJ Ragotte, J Dauparas, E Kinfu, S Tipps, RD Kibler, M Baek, F DiMaio, et al.  Hallucinating symmetric protein assemblies.  _Science_ , 378(6615):56–61, 2022.
  * Wu et al. [2022] Kevin Eric Wu, Kevin K Yang, Rianne van den Berg, James Zou, Alex Xijie Lu, and Ava P Amini.  Protein structure generation via folding diffusion.  2022\.
  * Yang et al. [2023] Kevin K. Yang, Hugh Yeh, and Niccolò Zanichelli.  Masked inverse folding with sequence transfer for protein representation learning.  _bioRxiv_ , 2023.  doi: 10.1101/2022.05.25.493516.  URL https://www.biorxiv.org/content/early/2023/03/19/2022.05.25.493516.
  * Yi et al. [2023] Kai Yi, Bingxin Zhou, Yiqing Shen, Pietro Liò, and Yu Guang Wang.  Graph denoising diffusion for inverse protein folding.  _arXiv preprint arXiv:2306.16819_ , 2023.

##  Appendix A Related work

Advances in inverse folding can be broadly categorized into improving
structure representation or sequence generation. On the structure side, graph-
based architectures combined with geometric features led to significant
improvements over MLP and CNN based models [8, 9, 4]. On the sequence side,
Transformer-based architectures have dominated [8, 6, 18]. However, these
Transformer-based architectures generate sequences autoregressively, leading
to slow inference times and high variance in generation quality. Non-
autoregressive alternatives have shown strong performance on benchmarks such
as perplexity and sequence recovery [4, 19]. However, these benchmarks have
proved unreliable to protein designers. Independent of us, Wu et al. [17]
found that ESM-IF [6] achieving SOTA sequence recovery of 72% but poor
designability results compared to methods with lower sequence recovery.
Multiple works have found designability has been shown to correlate with
experimental validation [3, 15, 16]. Our method improves on Protein-MPNN, a
popular model among protein designers that has demonstrated strong
experimentally validated results, by generating sequences faster and non-
autoregressively [3].


[affiliation=1]UmbertoCappellazzo [affiliation=2]DanieleFalavigna
[affiliation=2]AlessioBrutti

# Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of
Adapters

###### Abstract

Mixture of Experts (MoE) architectures have recently started burgeoning due to
their ability to scale model's capacity while maintaining the computational
cost affordable. Furthermore, they can be applied to both Transformers and
State Space Models, the current state-of-the-art models in numerous fields.
While MoE has been mostly investigated for the pre-training stage, its use in
parameter-efficient transfer learning settings is under-explored. To narrow
this gap, this paper attempts to demystify the use of MoE for parameter-
efficient fine-tuning of Audio Spectrogram Transformers to audio and speech
downstream tasks. Specifically, we propose Soft Mixture of Adapters (Soft-
MoA). It exploits adapters as the experts and, leveraging the recent Soft MoE
method, it relies on a soft assignment between the input tokens and experts to
keep the computational time limited. Extensive experiments across 4 benchmarks
demonstrate that Soft-MoA outperforms the single adapter method and performs
on par with the dense MoA counterpart. We finally present ablation studies on
key elements of Soft-MoA, showing for example that Soft-MoA achieves better
scaling with more experts, as well as ensuring that all experts contribute to
the computation of the output tokens, thus dispensing with the expert
imbalance issue.

###### keywords:

Audio Spectrogram Transformer, efficient fine-tuning, adapters, Mixture of
Experts, Soft Mixture of Adapters

##  1 Introduction

Large pre-trained audio and speech models have exhibited outstanding
performance when fine-tuned with task-specific data [1, 2]. A common practice
entails the adaptation of the whole model to each downstream tasks (i.e., full
fine-tuning) [3, 4]. However, this paradigm has two major limitations: 1)
adapting the entire pre-trained model is expensive and usually demands a
significant volume of training data to avoid overfitting; 2) storing a copy of
the model for each downstream task is often unfeasible and impractical.

Given these shortcomings, current research mainly revolves around learning a
small fraction of task-specific parameters, while keeping the pre-trained
model frozen. This approach is known as parameter-efficient transfer learning
(PETL) and includes several nuances. For example, prompt-tuning methods [5, 6]
introduce trainable task-specific tokens into one or multiple layers. LoRA [7,
8] uses trainable low-rank matrices to approximate the weight matrices.
Adapter-based methods [9, 10, 11] add lightweight modules (adapters) with
bottleneck architecture comprising two fully-connected layers. The adapter can
be inserted after both the multi-head self-attention and fully-connected feed-
forward network blocks (Houlsby) [12], or only after the feed-forward
(Pfeiffer) [10]. Adapters can also be scaled and shifted to modulate the pre-
trained features [13], or their down/up projections can be shared across
different layers and low-dimensional re-scaling coefficients are learned [14].
In the speech field, PETL methods have been investigated and compared in [15,
16].

Very recently, Mixture of Experts (MoE) models have shown remarkable results
in natural language processing, pushing large language models to the limit,
facilitating the effective scaling of Transformers and State Space Models
while concurrently reducing computational costs [17, 18, 19, 20]. MoEs rely on
the idea that sub-modular components, the experts, can specialize in different
inputs and scale the model's capacity. While most works have focused on the
use of MoEs during the pre-training stage, only few works have leveraged MoEs
for efficient fine-tuning [18, 21, 22]. In the latter case, each expert is
usually represented by a single adapter, and the model is referred to as
Mixture of Adapters (MoA). However, these works usually target language-based
tasks, whereas pure audio/speech classification tasks have not been taken into
account before. Therefore, in this paper, we apply MoAs to the Audio
Spectrogram Transformer (AST), a powerful foundation model achieving state-of-
the-art results on various audio/speech tasks [2], and we ask the following
question:

($\mathbf{Q}$) Can we leverage MoAs for the efficient fine-tuning of AST to
audio/speech downstream tasks?

To answer the above research question ($\mathbf{Q}$), we investigate the use
of MoAs for PETL of AST on four popular audio and speech benchmarks.
Specifically, we propose to adapt a recent sparse version of MoE called Soft-
MoE [23] to our PETL setting, whereby each expert only handles a small number
of slots that are the result of a weighted combination of all input tokens. We
call it Soft-MoA, and we compare it with the standard single adapter approach
and with the dense version of MoA that requires each adapter to process all
the input tokens (we refer to it as Dense-MoA). We empirically show that both
Soft and Dense MoA outperform the single adapter approach, both for the
Pfeiffer and Houlsby configuration, leading to accuracy improvement of up to
2.5%; also, Soft-MoA attains performance parity with Dense-MoA while
drastically reducing the training cost. Finally, we demonstrate the
effectiveness of Soft-MoA by carrying out extensive experiments revealing that
1) both Soft and Dense-MoA gains over the single adapter strategy are more
evident when fewer parameters are available, 2) Soft-MoA is robust to ``expert
imbalance'', thus ensuring that all experts are involved in the learning
process, and 3) Soft-MoA attains the best performance accuracy when few slots
($1/2$) and several experts are used rather than the opposite case as multiple
slots tend to learn redundant information.

Figure 1: (a) For each AST layer, the Soft/Dense MoA blocks are inserted
parallel to MHSA and/or FFN sub-layers. (b) Illustration of Dense-MoA, whereby
each expert output, scaled by the router's distribution (thickness of the
arrows), is summed to produce the final output. (c) In Soft-MoA, each expert
only processes a subset of slots (here $2$), and each slot accepts as input a
weighted combination of all input tokens (thickness of the arrows). Note that
the trainable parameters are represented by dashed blocks. Best viewed in
color.

##  2 Methodology

In this section, we first give a brief recap of the AST model and the standard
single adapter approach. In section 2.3, we present the details of the Dense-
MoA and Soft-MoA approaches.

###  2.1 AST model recap

The Audio Spectrogram Transformer (AST) is an attention-based model that
achieves state-of-the-art results on various audio and speech tasks [2, 24].
The AST model receives as input audio spectrograms that are patchified and
then a linear projection is applied to each patch. This results in a sequence
of $L$ tokens of size $d=768$, which we refer to as
$\textbf{X}\in\mathbb{R}^{L\times d}$. AST comprises $12$ attention layers,
each of which is composed of two sub-layers: a multi-head self-attention
(MHSA) and a fully-connected feed-forward (FFN) module.

###  2.2 Adapters

Adapters are light subnetworks that are inserted into every layer of the AST
model. To keep the parameters limited, adapters exploit a bottleneck
architecture. The input sequence of hidden dimension $d$ is first down-
projected into a low-dimensional space with size $r$ (the bottleneck
dimension), and then up-projected back to the original dimension $d$. A non-
linear activation function is also applied in-between the two fully-connected
layers.

While the bottleneck adapter is the most common adapter design, recent works
have also explored convolution-based adapters mainly for vision tasks (e.g.,
Convpass) [25, 26]. In addition to this, adapters usually follow a Pfeiffer
[10] or Houlsby [12] configuration: the former places the adapter parallel or
sequentially to the MHSA or FFN sub-layer, whereas the latter includes the
adapter on both sub-layers.

###  2.3 Dense and Soft MoA

Table 1: Performance evaluations of Dense and Soft-MoA on $4$ benchmarks for the Pfeiffer configuration.  Method | # params | ESC-50 | US8K | GSC | FSC | Avg | Time (ms)
---|---|---|---|---|---|---|---
Full FT | 85.5M | 87.48 | 84.31 | 97.31 | 93.29 | 90.07 | 645
Linear | 9-40K | 75.85 | 77.93 | 41.78 | 27.52 | 55.77 | 226
BitFit | 102K | 86.05 | 82.17 | 85.51 | 63.85 | 79.40 | 513
DPT | 230K | 86.52 | 83.67 | 89.18 | 68.60 | 81.99 | 561
Pref-T | 221K | 82.93 | 81.39 | 83.46 | 55.75 | 75.88 | 529
LoRA | 221K | 86.45 | 83.83 | 93.61 | 76.00 | 84.97 | 525
Bottleneck Adapter
Single | 470K | 88.65 | 83.36 | 93.53 | 78.19 | 85.93 | 513
D-MoA 14 | 535K | 89.55 | 84.30 | 93.89 | 82.43 | 87.54 | 1689
S-MoA 14 | 535K | 89.08 | 84.88 | 93.91 | 82.48 | 87.59 | 626
Convpass Adapter
Single | 491K | 87.93 | 83.38 | 93.47 | 77.62 | 85.60 | 515
D-MoA 14 | 535K | 89.30 | 84.32 | 93.70 | 83.52 | 87.71 | 1727
S-MoA 14 | 535K | 88.43 | 84.29 | 93.36 | 80.36 | 86.61 | 638

Dense-MoA. It encompasses a set of $N$ ``expert'' adapters $E_{1},\dots E_{N}$
and a router network $R$ that learns the optimal distribution over the
adapters for a given input sequence. In its simplest form [27, 18], the router
is a dense fully-connected layer with weights
$\textbf{W}\in\mathbb{R}^{d\times N}$ followed by a softmax function that
takes as input the sequence X and merges the output of each adapter using the
gating scores $g_{1},\dots g_{N}$ to yield the output sequence $\mathbf{Y}$:

$g_{i}=R(\mathbf{X})_{i}=\text{softmax}(\textbf{X}\textbf{W}),$ (1)
$\mathbf{Y}=\sum_{i=1}^{N}g_{i}\cdot E_{i}(\mathbf{X}).$ (2)

If all the $N$ adapters take part in the computation of the output of a given
input (scaled by the router's distribution), then we refer to this as Dense-
MoA (alternatively we can think of this as ensemble MoA). Whereas this
approach would cater to exact computation of gradients and end-to-end-
learning, it would also incur a substantial increase in computational costs
since each input token is computed by every expert rather than a single
expert. To circumvent the above issue, we propose to adapt a recent method
called Soft Mixture of Experts [23] to our PETL setting where each expert is
an adapter, and we call it Soft-MoA. Note that in our setting only the
adapters are actually learned whilst the backbone model is frozen.

Soft-MoA. Rather than feeding all input tokens to each expert, Soft-MoA passes
a different weighted soft combinations of all input tokens to each expert.
Unlike other sparse techniques like Top-$k$ [28] whereby only the $k$ experts
that are assigned the highest router's probability are activated, Soft-MoA
provides fully-differentiable operations, better training stability, and
immunity to ``token dropping'' and ``expert imbalance'' issues [23]. In
practice, each adapter processes $p$ slots, and each slot has a corresponding
$d$-dimensional vector of parameters. These parameters are denoted by
$\mathbf{\Phi}\in\mathbb{R}^{d\times(N\cdot p)}$. The input slots,
$\mathbf{\tilde{X}}$, are computed as the convex combination of all the $L$
input tokens:

$\mathbf{\tilde{X}}=\mathbf{D}^{\top}\mathbf{X},\quad\mathbf{D}_{i,j}=\frac{\text{exp}((\mathbf{X}\mathbf{\Phi})_{i,j}}{\sum_{h=1}^{L}\text{exp}((\mathbf{X}\mathbf{\Phi})_{h,j})}.$
(3)

$\mathbf{D}$ is called the dispatch weights and corresponds to applying a
softmax along the columns of $\mathbf{X}\mathbf{\Phi}$. At this point, each
adapter processes the corresponding slots: $\mathbf{\tilde{Y}}_{i}=E_{\lfloor
i/p\rfloor}(\mathbf{\tilde{X}_{i}})$. Finally, the output tokens $\mathbf{Y}$
are the result of a convex combination of all $(N\cdot p)$ slots:

$\mathbf{Y}=\mathbf{C}\mathbf{\tilde{Y}},\quad\mathbf{C}_{i,j}=\frac{\text{exp}((\mathbf{X}\mathbf{\Phi})_{i,j}}{\sum_{h=1}^{N\cdot
p}\text{exp}((\mathbf{X}\mathbf{\Phi})_{i,h})}.$ (4)

The matrix $\mathbf{C}$ is referred to as the combine weights, and is
equivalent to applying a softmax over the rows of $\mathbf{X}\mathbf{\Phi}$.

We provide an overview of Soft and Dense MoA in Figure 1. We show in Section
3.2 that Soft-MoA outperforms the single adapter approach and remains
competitive with Dense-MoA while trimming down the computational cost.
Finally, for our experiments, following [15] that show that inserting the
adapter in parallel achieves better performance than sequentially, we place
the MoA block parallel to the MHSA layer only (i.e., Pfeiffer) or parallel to
both the MHSA and FFN layers (i.e., Houlsby). Note that the number of slots
$p$ is an hyper-parameter, and we elaborate on its optimal value on Section
3.3.

##  3 Experiments and Discussion

###  3.1 Implementation Details

For our experiments, we mainly follow the implementation details of [15] to
provide a fair comparison.

Datasets. We evaluate the PETL methods on three audio/speech downstream
classification tasks. (1) Audio classification: we use the ESC-50 and
UrbanSound8K (US8K) datasets. ESC-50 [29] consists of $2000$ 5-second-long
environmental audio recordings of $50$ classes. US8K [30] includes $8732$
labeled sound excerpts of urban sounds from $10$ classes. (2) Keyword
spotting: Speech Commands V2 [31] has $105,829$ 1-second recordings of $35$
speech commands. (3) Intent classification: Fluent Speech Commands (FSC) [32]
includes $30043$ English utterances spanning $31$ classes.

PETL baselines. We include two traditional fine-tuning strategies: full fine-
tuning (Full-FT), which finetunes the full pre-trained AST model; and linear
probing, which only fine-tunes the classification head. Following [15] we
include some common PETL baselines: BitFit [33], deep prompt-tuning (DPT) [5],
prefix-tuning (Pref-T) [6] and LoRA [7]. For the analysis of MoA, we take into
account both Bottleneck [12] and Convpass [34] adapters. We report Dense and
Soft-MoA (D/S-MoA) with $14$ or $7$ adapters for the Pfeiffer and Houlsby
configuration, respectively, and we compare them with the standard
implementation using a single adapter per layer (Single).

Training Details. For all experiments we use the AST model pre-trained on
ImageNet-21K [37] and AudioSet [38] provided by the Huggingface Transformers
library [39]. The model has around $85.5$ million parameters, and the hidden
size is $768$. Please refer to [15] for the training details of the baselines
(LoRA, DPT etc.). For MoA experiments, we use AdamW optimizer with cosine
annealing scheduler and weight decay set to 0.1. The number of epochs depends
on each dataset, and it is kept fixed among the all the baselines to foster a
fair comparison. For the ESC-50 and US8K datasets, we run 5-fold and 10-fold
cross validation as suggested in the original papers. Except US8K that does
not provide a validation set by default, for the others we set the hyper-
parameters using the validation set. A detailed list of the hyper-parameters
and the code will be released at the following github
repository.111https://github.com/umbertocappellazzo/PETL_AST

###  3.2 Main Results and Discussion

Table 2: Results of D/S-MoA for the Houlsby configuration. Method | ESC-50 | US8K | GSC | FSC | Avg
---|---|---|---|---|---
Bottleneck Adapter
Single | 88.00 | 82.80 | 91.75 | 78.71 | 85.32
D-MoA 7 | 87.33 | 83.78 | 94.11 | 82.64 | 86.97
S-MoA 7 | 87.13 | 83.77 | 93.67 | 81.41 | 86.50
Convpass Adapter
Single | 87.15 | 82.75 | 92.55 | 77.79 | 85.06
D-MoA 7 | 87.31 | 83.77 | 93.20 | 82.26 | 86.63
S-MoA 7 | 88.13 | 83.87 | 92.69 | 81.69 | 86.60

Figure 2: (Left). The accuracy trend as more parameters are used. (Middle).
The effect of the number of adapters given a fixed parameters budget. (Right).
Adapters contribution to the output tokens for various layers. Results
reported for FSC.

Table 1 presents the performance comparisons between the single adapter
approach and Soft/Dense MoA, as well as some other common PETL methods. The
single method uses one adapter whose bottleneck dimension is $24$, whereas
Soft/Dense-MoA include 14 adapters, each with a dimension of $1$, and one slot
is used for each adapter. From the table we can observe that both MoAs
outperform the single adapter, leading to up to $2.5$ % performance
improvement on average for the Bottleneck case, while for Convpass we notice
that Soft-MoA is slightly worse than Dense-MoA, but still better than the
single adapter. In general, the biggest gain is obtained with the FSC dataset
(up to $5.5$ and $7.6$ %). Indeed, FSC is the more challenging dataset as it
includes longer speech audio data, thus we argue that multiple adapters can
specialize in learning different information, and consequently leading to
better performance. We also notice that the GSC dataset does not benefit much
from the use of MoA architecture, so we surmise that a single adapter already
achieves very competitive performance and so the use of multiple smaller
adapters is not helpful. Another pivotal aspect is the extra computational
cost brought by MoAs, estimated as the average train step time in
milliseconds. Whereas Dense-MoA incurs a considerable increase in time (more
than $3$x with respect to the single adapter), S-MoA, instead, requires only a
limited extra time, while guaranteeing on-par performance.

Finally, we test Soft-MoA's efficacy for the Houlsby configuration, where the
MoA block is also inserted parallel to the FFN sub-layer. Table 2 confirms the
superiority of both MoAs over the single adapter. We note that, in this case,
Convpass and Bottleneck configurations bring about a similar improvement.

###  3.3 Ablation Studies

We now conduct some ablation studies to evaluate the effectiveness of Soft-MoA
under different settings. We focus on the Pfeiffer Bottleneck configuration,
and on the FSC dataset.

Increasing the Parameters Budget. We examine the trend of the methods when we
increase the number of trainable parameters. For the single adapter, we
increase the parameters by making the bottleneck dimension larger, while for
MoAs we keep the bottleneck dimension to $1$ and we increase the number of
adapters. From Figure 2 (Left) we observe that Soft-MoA outperforms the single
adapter, although when many parameters are available the two methods tend to
achieve similar results, thus showing that using a single adapter is
sufficient when scaling the number of parameters is sustainable.

Scaling Up the Number of Adapters. We now investigate how the MoA methods
scale with respect to the number of adapters $N$. Regardless of $N$, we fix
the number of learnable parameters to around $900$K to have a fair comparison.
In this way, we want to figure out if having more adapters with a smaller
bottleneck dimension is better than having a few but ``bigger'' (in terms of
parameters) adapters. The Figure 2 (Middle) shows that Dense-MoA, due to its
intrinsic dense structure, reaches the peak performance when $N=7$, and then
adding more adapters does not lead to additional improvement. On the contrary,
Soft-MoA depends heavily on $N$, and only when this number is large enough
does it attain good performance. This trend is in line with that of the
original Soft MoE paper [23].

Adapters Contribution to the Output Tokens. By design, the computation of the
final output tokens depends on a linear combination of all the adapters'
slots. We want to verify whether all adapters contribute to the output
sequence. We fix one slot per adapter and consider $7$ adapters, and we
approximate the contribution of each adapter by averaging their corresponding
coefficients in the linear combinations for all output tokens. We average over
all the batches of the test set and report the adapter contribution for
different layers in Figure 2 (Right). We see that some adapters have a bigger
impact than others, but all of them contribute to the final output tokens.
Therefore, Soft-MoA does not suffer from to the issue of expert imbalance,
namely few experts monopolize the output contribution, an issue that affects
other routing strategies like Top-$k$ [28, 23].

Optimal Trade-off between Slots and Adapters. The number of slots $p$ is an
important hyper-parameter of Soft-MoA, thus we examine its optimal value. We
notice that if we set the number of slots equal to the number of tokens $L$,
Soft-MoA boils down to Dense-MoA, so it is crucial to keep $p$ small. For our
experiments, depending on the dataset, $L$ is between $100$ and $500$, and
setting $p$ up to $14$ is a reasonable choice. We report the results for FSC
in Table 3 and we see that, with the same number of trainable parameters,
having more adapters with few slots brings better results than having few
adapters but many slots. We speculate that this happens because multiple slots
corresponding to the same adapter might have a tendency to learn similar
concepts and become redundant, whereas using more adapters ends up learning
more diverse information.

Table 3: Optimal trade-off between the # of adapters and slots. Adapters/Slots | Acc
---|---
2/14 | 78.52
4/6 | 80.26
6/4 | 81.65
8/3 | 82.36
12/2 | 83.24
24/1 | 82.87

##  4 Conclusion

In this paper, we propose Soft Mixture of Adapters to efficiently fine-tune
the AST model on various audio/speech downstream tasks. Soft-MoA relies on
multiple adapters that take as input a soft convex combination of all the
input tokens, thus reducing the computational cost of the dense counterpart.
Extensive experiments on $4$ benchmarks show that Soft-MoA performs on par
with Dense-MoA, and it outperforms the single adapter strategy, confirming
itself as a strong method also for parameter-efficient transfer learning
settings. To strengthen our analysis, we carry out ablation studies revealing
that Soft and Dense MoA provide bigger gains over the single adapter when the
parameters budget is limited. We also show that Soft-MoA scales better with
the number of adapters and that it is sufficient to use only $1$ or $2$ slots
to achieve the optimal performance.

## References

  * [1] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in _International Conference on Machine Learning_. PMLR, 2023, pp. 28 492–28 518.
  * [2] Y. Gong, Y.-A. Chung, and J. Glass, ``Ast: Audio spectrogram transformer,'' _arXiv preprint arXiv:2104.01778_ , 2021.
  * [3] E. Tsalera, A. Papadakis, and M. Samarakou, ``Comparison of pre-trained cnns for audio classification using transfer learning,'' _Journal of Sensor and Actuator Networks_ , vol. 10, no. 4, p. 72, 2021.
  * [4] Y. Wang, A. Boumadane, and A. Heba, ``A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,'' _arXiv preprint arXiv:2111.02735_ , 2021.
  * [5] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, ``Visual prompt tuning,'' in _European Conference on Computer Vision_. Springer, 2022, pp. 709–727.
  * [6] X. L. Li and P. Liang, ``Prefix-tuning: Optimizing continuous prompts for generation,'' _Proceedings of ACL_ , 2021.
  * [7] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, ``Lora: Low-rank adaptation of large language models,'' _arXiv preprint arXiv:2106.09685_ , 2021.
  * [8] Y. Zeng and K. Lee, ``The expressive power of low-rank adaptation,'' _ICLR_ , 2024.
  * [9] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, ``Learning multiple visual domains with residual adapters,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [10] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, ``Adapterfusion: Non-destructive task composition for transfer learning,'' _arXiv preprint arXiv:2005.00247_ , 2020.
  * [11] S. Jie, H. Wang, and Z.-H. Deng, ``Revisiting the parameter efficiency of adapters from the perspective of precision redundancy,'' in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 17 217–17 226.
  * [12] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, ``Parameter-efficient transfer learning for nlp,'' in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [13] D. Lian, D. Zhou, J. Feng, and X. Wang, ``Scaling & shifting your features: A new baseline for efficient model tuning,'' _Advances in Neural Information Processing Systems_ , vol. 35, pp. 109–123, 2022.
  * [14] W. Dong, D. Yan, Z. Lin, and P. Wang, ``Efficient adaptation of large vision transformer via adapter re-composing,'' _NeurIPS_ , 2023.
  * [15] U. Cappellazzo, D. Falavigna, A. Brutti, and M. Ravanelli, ``Parameter-efficient transfer learning of audio spectrogram transformers,'' _arXiv preprint arXiv:2312.03694_ , 2023.
  * [16] T.-H. Lin, H.-S. Wang, H.-Y. Weng, K.-C. Peng, Z.-C. Chen, and H.-y. Lee, ``Peft for speech: Unveiling optimal placement, merging strategies, and ensemble techniques,'' _arXiv preprint arXiv:2401.02122_ , 2024.
  * [17] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand _et al._ , ``Mixtral of experts,'' _arXiv preprint arXiv:2401.04088_ , 2024.
  * [18] T. Zadouri, A. Üstün, A. Ahmadian, B. Ermiş, A. Locatelli, and S. Hooker, ``Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning,'' _ICLR_ , 2024.
  * [19] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu _et al._ , ``Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models,'' _arXiv preprint arXiv:2401.06066_ , 2024.
  * [20] M. Pióro, K. Ciebiera, K. Król, J. Ludziejewski, and S. Jaszczur, ``Moe-mamba: Efficient selective state space models with mixture of experts,'' _arXiv preprint arXiv:2401.04081_ , 2024.
  * [21] Y. Wang, S. Mukherjee, X. Liu, J. Gao, A. H. Awadallah, and J. Gao, ``Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models,'' _arXiv preprint arXiv:2205.12410_ , vol. 1, no. 2, p. 4, 2022.
  * [22] A. Mehrish, A. R. Kashyap, L. Yingting, N. Majumder, and S. Poria, ``Adaptermix: Exploring the efficacy of mixture of adapters for low-resource tts adaptation,'' _Interspeech_ , 2023.
  * [23] J. Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby, ``From sparse to soft mixtures of experts,'' _ICLR_ , 2024.
  * [24] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass, ``Ssast: Self-supervised audio spectrogram transformer,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 10, 2022, pp. 10 699–10 709.
  * [25] S. Jie and Z.-H. Deng, ``Convolutional bypasses are better vision transformer adapters,'' _arXiv preprint arXiv:2207.07039_ , 2022.
  * [26] H. Chen, R. Tao, H. Zhang, Y. Wang, W. Ye, J. Wang, G. Hu, and M. Savvides, ``Conv-adapter: Exploring parameter efficient transfer learning for convnets,'' _arXiv preprint arXiv:2208.07463_ , 2022.
  * [27] W. Fedus, B. Zoph, and N. Shazeer, ``Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,'' _The Journal of Machine Learning Research_ , vol. 23, no. 1, pp. 5232–5270, 2022.
  * [28] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, ``Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,'' _arXiv preprint arXiv:1701.06538_ , 2017.
  * [29] K. J. Piczak, ``Esc: Dataset for environmental sound classification,'' in _Proceedings of the 23rd ACM international conference on Multimedia_ , 2015, pp. 1015–1018.
  * [30] J. Salamon, C. Jacoby, and J. P. Bello, ``A dataset and taxonomy for urban sound research,'' in _Proceedings of the 22nd ACM international conference on Multimedia_ , 2014, pp. 1041–1044.
  * [31] P. Warden, ``Speech commands: A dataset for limited-vocabulary speech recognition,'' _arXiv preprint arXiv:1804.03209_ , 2018.
  * [32] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, ``Speech model pre-training for end-to-end spoken language understanding,'' _arXiv preprint arXiv:1904.03670_ , 2019.
  * [33] E. B. Zaken, S. Ravfogel, and Y. Goldberg, ``Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,'' _arXiv preprint arXiv:2106.10199_ , 2021.
  * [34] S. Jie and Z.-H. Deng, ``Convolutional bypasses are better vision transformer adapters,'' _arXiv preprint arXiv:2207.07039_ , 2022.