# Positive solutions of quasilinear elliptic equations with exponential
nonlinearity combined with convection term

Anderson L.A. de Araujo  Universidade Federal de Viçosa, Departamento de
Matemática, Avenida Peter Henry Rolfs, s/n, CEP 36570-900, Viçosa, MG, Brazil
<EMAIL_ADDRESS>and  Luiz F.O Faria  Departamento de Matemática,
Universidade Federal de Juiz de Fora, CEP 30161-970, Juiz de Fora - MG, Brazil
<EMAIL_ADDRESS>
###### Abstract.

We establish the existence of positive solutions for a nonlinear elliptic
Dirichlet problem in dimension $N$ involving the $N$-Laplacian. The
nonlinearity considered depends on the gradient of the unknown function and an
exponential term. In such case, variational methods cannot be applied. Our
approach is based on approximation scheme, where we consider a new class of
normed spaces of finite dimension. As a particular case, we extended the
result achieved by De Araujo and Montenegro [2016] for any $N>2$.

###### Key words and phrases:

Dirichlet problem for the $N$-Laplacian; Galerkin approximation; Trudinger-
Moser inequality; exponential growth; convection term

###### 2010 Mathematics Subject Classification:

35B33; 35J62; 35J66; 35J92; 37L65

Luiz F.O Faria was partially supported by FAPEMIG CEX APQ 02374/17.

##  1\. Introduction

Let $\Omega\subset\mathbb{R}^{N}$ be a smooth bounded domain and $p>1$.
Consider the following problem

(1.1) $\left\\{\begin{array}[]{lll}-\Delta_{p}u=g(x,u,\nabla
u)&\mbox{in}&\Omega,\\\ u=0&\mbox{on}&\partial\Omega.\end{array}\right.$

Here, the operator $-\Delta_{p}:W^{1,p}_{0}(\Omega)\to
W^{-1,p^{\prime}}(\Omega)$, where $\frac{1}{p}+\frac{1}{p^{\prime}}=1$, is
defined by

$\displaystyle\langle-\Delta_{p}u,v\rangle=\int_{\Omega}|\nabla u|^{p-2}\nabla
u\nabla vdx\,\mbox{ for all $u,v\in W^{1,p}_{0}(\Omega)$},$

and the forcing term $g$ has the form of a convection term, that is, it
depends also on the gradient of the unknown function. Due to the presence of
the gradient $\nabla u$ in the term $g(x,u,\nabla u)$, problem (1.1) does not
have, in general, variational structure. This kind of problems are usually
studied by means of topological degree, the method of sub-supersolutions,
fixed point theory, approximation techniques and iterative scheme. For
instance, we would like to cite [2, 5, 25, 13, 8, 12, 14]. In particular, in
[13], via an approximation on finite dimensional subspaces, the authors proved
the existence of a positive solution for the following problem

$None$ $\left\\{\begin{array}[]{lll}-\Delta_{p}u-\mu\Delta_{q}u=g(x,u,\nabla
u)&\mbox{in}&\Omega,\\\ u>0&\mbox{in}&\Omega,\\\
u=0&\mbox{on}&\partial\Omega,\end{array}\right.$

where $\mu\geq 0$, $g:\Omega\times\mathbb{R}\times\mathbb{R}^{N}\to\mathbb{R}$
is a continuous function satisfying the growth condition:

  * $(G)$

$b_{0}|t|^{r_{0}}\leq f(x,t,\xi)\leq b_{1}(1+|t|^{r_{1}}+|\xi|^{r_{2}})$ for
all $(x,t,\xi)\in\Omega\times\mathbb{R}\times\mathbb{R}^{N}$, with constants
$b_{0},b_{1}>0$, $r_{1},r_{2}\in[0,p-1)$, $r_{0}\in[0,p-1)$ if $\mu=0$, and
$r_{0}\in[0,q-1]$ if $\mu>0$.

On the other hand, elliptic problems of the type

(1.2) $\left\\{\begin{array}[]{lcc}-\Delta_{N}v=g(x,v)&\textup{in}&\Omega,\\\
v=0&\textup{on}&\partial\Omega,\end{array}\right.$

where $\Omega\subset\mathbb{R}^{N}$ and $g(x,v)$ is continuous and behaves
like $\exp(\alpha|v|^{N/(N-1)})$ as $|v|\rightarrow+\infty$ have been studied
by many authors, we would like to cite [3, 10, 9, 11, 19, 26, 6]. One of the
main ingredients is the Trudinger-Moser inequality introduced in [22, 28].
Namely, given $u\in W^{1,N}_{0}(\Omega)$, then

(1.3) $e^{\sigma|u|^{\frac{N}{N-1}}}\in L^{1}(\Omega)\,\,\,\mbox{for
every}\,\,\,\sigma>0,$

and there exists a positive constant $L(N)$ which depends on $N$ only, such
that

(1.4) $\sup_{\|u\|_{W^{1,N}_{0}(\Omega)}\leq
1}\int_{\Omega}e^{\sigma|u|^{\frac{N}{N-1}}}dx\leq L(N)|\Omega|\,\,\,\mbox{for
every}\,\,\,\sigma\leq\alpha_{N},$

where $|\Omega|=\int_{\Omega}dx$, $\alpha_{N}=Nw_{N-1}^{\frac{1}{N-1}}$ and
$w_{N-1}$ is the $(N-1)$-dimensional measure of the $(N-1)$-sphere.

In particular, in [6] the authors proved existence of solutions for the
following problem

$\left\\{\begin{array}[]{lll}-\Delta u=\lambda u^{q}+f(u)&\mbox{in}&\Omega,\\\
u>0&\mbox{in}&\Omega,\\\ u=0&\mbox{on}&\partial\Omega,\end{array}\right.$

where $\Omega\subset\mathbb{R}^{2}$, $\lambda>0$ is a small enough parameter,
$0<q<1$ and $f:[0,\infty)\to\mathbb{R}$ is a continuous function satisfying
the growth condition:

  * $(H)$

$0\leq tf(t)\leq C|t|^{r}\exp(\alpha t^{2})$ where $\alpha>0$ and $r>2$.

In [7], still considering $N=2$, the authors proved existence of solutions for
an elliptic system with arguments based in [6] and with nonlinearities
satisfying the growth condition ($H$).

In this work we are concerned with the existence of positive solutions for the
problem:

$None$
$\left\\{\begin{array}[]{lll}-\Delta_{N}u=\lambda(a_{1}u^{r_{1}}+a_{2}|\nabla
u|^{r_{2}})+f(u)&\mbox{in}&\Omega\\\
u=0&\mbox{on}&\partial\Omega,\end{array}\right.$

where $\Omega$ in $\mathbb{R}^{N}$ is a bounded domain with a
$C^{1,\alpha}$-boundary $\partial\Omega$, for some $0<\alpha\leq 1$,
$\lambda>0$ is a parameter, $0<r_{i}<N-1$, for $i=1,2$, $a_{1}>0$, $a_{2}\geq
0$, and $f:[0,\infty)\to\mathbb{R}$ is a nonegative continuous function. The
main assumption on the function $f$ is the following, which will be referred
throughout the paper as $(F)$:

  * $(F)$

$0\leq tf(t)\leq a_{3}t^{r_{3}+1}\exp(\alpha t^{\frac{N}{N-1}})$ where
$a_{3},\alpha>0$, and $r_{3}>N-1$.

Most of the papers, to prove existence results for the problem, assume
Ambrosetti–Rabinowitz conditions (or some additional conditions) to obtain
Palais-Smale or Cerami compactness condition. Notice that in this paper we
don’t need to impose such extra hypotheses.

An interesting problem related to (P), by considering a more general operator,
was treated by [29]. The authors studied a $(N,q)$-Laplacian problem with a
critical Trundinger-Moser nonlinearity as following

$\left\\{\begin{array}[]{lll}-\Delta_{N}u-\Delta_{q}u=\mu|u|^{q-2}u+\lambda|u|^{N-2}ue^{|u|^{N/(N-1)}}&\mbox{in}&\Omega\\\
u=0&\mbox{on}&\partial\Omega,\end{array}\right.$

where $N>q>1$, $\mu\in\mathbb{R}$ and $\lambda>0$. By using a critical point
theorem, based on a cohomological index, they proved the existence of solution
for $\mu$ interacting with the first eigenvalue of the
$(-\Delta_{q}u,W_{0}^{1,q}(\Omega))$ operator and for $\lambda$ sufficiently
large.

Here we extend the results of [6] for the general dimension case $N>1$
($a_{2}=0$). In order to prove the existence of positive solutions for $(P)$,
we borrow some ideas from [6] and [13]. Due to the presence of the
supercritical term $\exp(\alpha|v|^{N/(N-1)})$, along with the convection
term, we had to overcome some problems. For example, in $W_{0}^{1,N}(\Omega)$
we need to assume a Schauder basis instead of the Hilbert basis (like in [6]),
which becomes some additional difficulty. By comparing with [13], due to the
presence of the term $\exp(\alpha|v|^{N/(N-1)})$, a suitable modification on
the approximating approach had to be done. Although in [13] the authors used
the Schauder basis, we could not obtain the necessary estimates for this
approach by considering the approximate spaces used there. To do this, we
consider a new class of normed spaces of finite dimension.

Our main result reads as follows:

######  Theorem 1.1.

Suppose that $f:[0,\infty)\to\mathbb{R}$ is a continuous function satisfying
the assumption $(F)$. Then there exists $\lambda^{*}>0$ such that for every
$\lambda\in(0,\lambda^{*})$ problem $(P)$ admits a (positive) weak solution
$u\in W^{1,N}_{0}(\Omega)$.

##  2\. Preliminary results

The Sobolev space $W^{1,N}_{0}(\Omega)$ is endowed with the norm

$\|u\|_{W^{1,N}_{0}(\Omega)}=\left(\int_{\Omega}|\nabla
u|^{N}dx\right)^{1/N}.$

To prove Theorem 1.1 we approximate $f$ by Lipschitz functions
$f_{k}:\mathbb{R}\to\mathbb{R}$ defined by

(2.1)
$f_{k}(s)=\displaystyle\left\\{\begin{array}[]{lcc}-k[G(-k-\frac{1}{k})-G(-k)],&\textup{if}&s\leq-k,\\\
-k[G(s-\frac{1}{k})-G(s)],&\textup{if}&-k\leq s\leq-\frac{1}{k},\\\
k^{2}s[G(-\frac{2}{k})-G(-\frac{1}{k})],&\textup{if}&-\frac{1}{k}\leq s\leq
0,\\\ k^{2}s[G(\frac{2}{k})-G(\frac{1}{k})],&\textup{if}&0\leq
s\leq\frac{1}{k},\\\ k[G(s+\frac{1}{k})-G(s)],&\textup{if}&\frac{1}{k}\leq
s\leq k,\\\ k[G(k+\frac{1}{k})-G(k)],&\textup{if}&s\geq k,\\\
\end{array}\right.$

where $G(s)=\int_{0}^{s}f(\xi)d\xi$.

The following (approximation) result was proved in [27] and uses the explicit
expression of the sequence (2.1).

######  Lemma 2.1.

Let $f:\mathbb{R}\to\mathbb{R}$ be a continuous function such that $sf(s)\geq
0$ for every $s\in\mathbb{R}$. Then there exists a sequence
$f_{k}:\mathbb{R}\to\mathbb{R}$ of continuous functions satisfying

(i) $sf_{k}(s)\geq 0$ for every $s\in\mathbb{R}$;

(ii) $\forall\,k\in\mathbb{N}$ $\exists c_{k}>0$ such that
$|f_{k}(\xi)-f_{k}(\eta)|\leq c_{k}|\xi-\eta|$ for every
$\xi,\eta\in\mathbb{R}$;

(iii) $f_{k}$ converges uniformly to $f$ in bounded subsets of $\mathbb{R}$.

The sequence $f_{k}$ of the previous lemma has some additional properties.

######  Lemma 2.2.

Let $f:\mathbb{R}\to\mathbb{R}$ be a continuous function satisfying $(F)$ for
every $s\in\mathbb{R}$. Then the sequence $f_{k}$ of Lemma 2.1 satisfies

(i) $\forall\,k\in\mathbb{N}$, $0\leq sf_{k}(s)\leq
C_{1}|s|^{r_{3}}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}})$ for every
$|s|\geq\frac{1}{k}$;

(ii) $\forall\,k\in\mathbb{N}$, $0\leq sf_{k}(s)\leq
C_{2}|s|^{2}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}})$ for every
$|s|\leq\frac{1}{k}$,

where $C_{1}$ and $C_{2}$ are positive constants independent of $k$.

###### Proof.

Everywhere in this proof the constant $a_{3}$ is the one of (2.1).

First step. Suppose that $-k\leq s\leq-\frac{1}{k}$.

By the mean value theorem, there exists $\eta\in(s-\frac{1}{k},s)$ such that

$f_{k}(s)=-k[G(s-\frac{1}{k})-G(s)]=-kG^{\prime}(\eta)(s-\frac{1}{k}-s)=f(\eta)$

and

$sf_{k}(s)=sf(\eta).$

Since $s-\frac{1}{k}<\eta<s<0$ and $f(\eta)<0$, we have $sf(\eta)\leq\eta
f(\eta)$. Therefore,

$\begin{array}[]{rcl}sf_{k}(s)\leq\eta
f(\eta)&\leq&a_{3}|\eta|^{r_{3}}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\\\
&\leq&a_{3}|s-\frac{1}{k}|^{r_{3}}\exp(\alpha\,|s-\frac{1}{k}|^{\frac{N}{N-1}})\\\
&\leq&a_{3}(|s|+\frac{1}{k})^{r_{3}}\exp(\alpha\,(|s|+\frac{1}{k})^{\frac{N}{N-1}})\\\
&\leq&a_{3}(2|s|)^{r_{3}}\exp(\alpha\,(2|s|)^{\frac{N}{N-1}})\\\
&=&a_{3}2^{r_{3}}|s|^{r_{3}}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).\end{array}$

Second step. Assume $\frac{1}{k}\leq s\leq k$.

By the mean value theorem, there exists $\eta\in(s,s+\frac{1}{k})$ such that

$f_{k}(s)=k[G(s+\frac{1}{k})-G(s)]=kG^{\prime}(\eta)(s+\frac{1}{k}-s)=f(\eta)$

and

$sf_{k}(s)=sf(\eta).$

Since $0<s<\eta<s+\frac{1}{k}$ and $f(\eta)>0$, we have $sf(\eta)\leq\eta
f(\eta)$. Therefore,

$\begin{array}[]{rcl}sf_{k}(s)\leq\eta
f(\eta)&\leq&a_{3}|\eta|^{r_{3}}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\\\
&\leq&a_{3}|s+\frac{1}{k}|^{r_{3}}\exp(\alpha\,|s+\frac{1}{k}|^{\frac{N}{N-1}})\\\
&\leq&a_{3}(2|s|)^{r_{3}}\exp(\alpha\,(2|s|)^{\frac{N}{N-1}})\\\
&=&a_{3}2^{r_{3}}|s|^{r_{3}}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).\end{array}$

Third step. Suppose that $|s|\geq k$, then

(2.2)
$f_{k}(s)=\displaystyle\left\\{\begin{array}[]{lcc}-k[G(-k-\frac{1}{k})-G(-k)],&\textup{if}&s\leq-k\\\
k[G(k+\frac{1}{k})-G(k)],&\textup{if}&s\geq k.\\\ \end{array}\right.$

If $s\leq-k$, by the mean value theorem, there exists
$\eta\in(-k-\frac{1}{k},-k)$ such that

$f_{k}(s)=k[G(-k-\frac{1}{k})-G(-k)]=-kG^{\prime}(\eta)(-k-\frac{1}{k}-(-k))=f(\eta)$

and

$sf_{k}(s)=sf(\eta).$

Since $-k-\frac{1}{k}<\eta<-k<0$ and $k<|\eta|<k+\frac{1}{k}$, we conclude
that

(2.3) $\begin{array}[]{rcl}sf_{k}(s)=\frac{s}{\eta}\eta
f(\eta)&\leq&\frac{|s|}{|\eta|}a_{3}|\eta|^{r_{3}}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})=a_{3}|s||\eta|^{{r_{3}}-1}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\\\
&\leq&a_{3}|s|(k+\frac{1}{k})^{{r_{3}}-1}\exp(\alpha\,(k+\frac{1}{k})^{\frac{N}{N-1}})\\\
&\leq&a_{3}|s|(|s|+\frac{1}{k})^{{r_{3}}-1}\exp(\alpha\,(|s|+\frac{1}{k})^{\frac{N}{N-1}})\\\
&\leq&a_{3}|s|(2|s|)^{{r_{3}}-1}\exp(\alpha\,(2|s|)^{\frac{N}{N-1}})\\\
&\leq&a_{3}2^{{r_{3}}-1}|s|^{r_{3}}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).\end{array}$

If $s\geq k$, by the mean value theorem, there exists
$\eta\in(k,k+\frac{1}{k})$ such that

$f_{k}(s)=k[G(k+\frac{1}{k})-G(k)]=kG^{\prime}(\eta)(k+\frac{1}{k}-k)=f(\eta).$

By computations similar to conclude (2.3) one has

$sf_{k}(s)=sf(\eta)=\frac{s}{\eta}\eta
f(\eta)\leq\frac{|s|}{|\eta|}a_{3}|\eta|^{r_{3}}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\leq
a_{3}2^{{r_{3}}-1}|s|^{r_{3}}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).$

Fourth step. Assume $-\frac{1}{k}\leq s\leq\frac{1}{k}$, then

(2.4)
$f_{k}(s)=\displaystyle\left\\{\begin{array}[]{lcc}k^{2}s[G(-\frac{2}{k})-G(-\frac{1}{k})],&\textup{if}&-\frac{1}{k}\leq
s\leq 0\\\ k^{2}s[G(\frac{2}{k})-G(\frac{1}{k})],&\textup{if}&0\leq
s\geq\frac{1}{k}.\\\ \end{array}\right.$

If $-\frac{1}{k}\leq s\leq 0$, by the mean value theorem, there exists
$\eta\in(-\frac{2}{k},-\frac{1}{k})$ such that

$f_{k}(s)=k^{2}s[G(-\frac{2}{k})-G(-\frac{1}{k})]=k^{2}sG^{\prime}(\eta)(-\frac{2}{k}-(-\frac{1}{k}))=-ksf(\eta).$

Therefore

$sf_{k}(s)=-ks^{2}f(\eta)=-k\frac{s^{2}}{\eta}\eta f(\eta)\leq
k\frac{s^{2}}{|\eta|}\eta f(\eta)\leq$ $\leq
a_{3}k|s|^{2}|\eta|^{{r_{3}}-1}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\leq
a_{3}k|s|^{2}(\frac{2}{k})^{{r_{3}}-1}\exp(\alpha\,|\eta|^{\frac{N}{N-1}})\leq
a_{3}2^{{r_{3}}-1}|s|^{2}\exp\left(\alpha\,\left(\frac{2}{k}\right)^{\frac{N}{N-1}}\right)$
(2.5) $\leq a_{3}2^{{r_{3}}-1}|s|^{2}\exp(2^{\frac{N}{N-1}}\alpha)\leq
a_{3}2^{{r_{3}}-1}\exp(2^{\frac{N}{N-1}}\alpha)|s|^{2}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).$

If $0\leq s\leq\frac{1}{k}$, by the mean value theorem, there exists
$\eta\in(\frac{1}{k},\frac{2}{k})$ such that

$f_{k}(s)=k^{2}s[G(\frac{2}{k})-G(\frac{1}{k})]=k^{2}sG^{\prime}(\eta)(\frac{2}{k}-\frac{1}{k})=ksf(\eta).$

By similar computations to conclude (2.5) one obtains

$sf_{k}(s)=ks^{2}f(\eta)=k\frac{s^{2}}{|\eta|}\eta f(\eta)\leq
a_{3}2^{{r_{3}}-1}\exp(2^{\frac{N}{N-1}}\alpha)|s|^{2}\exp(2^{\frac{N}{N-1}}\alpha\,|s|^{\frac{N}{N-1}}).$

The proof of the lemma follows by taking $C_{1}=a_{3}2^{r_{3}}$ ad
$C_{2}=a_{3}2^{{r_{3}}-1}C2^{{r_{3}}-1}\exp(2^{\frac{N}{N-1}}\alpha)$ where
$a_{3}$ is given in $(F)$. ∎

Before concluding this section, we will enunciate a comparison principle due
to Faria, Miyagaki and Motreanu [13, Theorem 2.2].

Consider the Dirichlet problem

(2.6) $\left\\{\begin{array}[]{lll}-\Delta_{p}u-\mu\Delta_{q}u=g(u)&\mbox{ in
}&\Omega\\\ u=0&\mbox{ on }&\partial\Omega,\end{array}\right.$

where $1<q\leq p<+\infty$, $\mu\geq 0$ and $g:\mathbb{R}\rightarrow\mathbb{R}$
is a continuous function.

We recall that $u_{1}\in W^{1,p}(\Omega)$ is a subsolution of problem (2.6) if
$u_{1}\geq 0$ a.e. on $\partial\Omega$ and

$\int_{\Omega}(|\nabla u_{1}|^{p-2}\nabla u_{1}\nabla\varphi+\mu|\nabla
u_{1}|^{q-2}\nabla u_{1}\nabla\varphi)dx\leq\int_{\Omega}g(u_{1})\varphi dx$

for all $\varphi\in W^{1,p}_{0}(\Omega)$ with $\varphi\geq 0$ a.e. in
$\Omega$, while $u_{2}\in W^{1,p}(\Omega)$ is a supersolution of (2.6) if the
reversed inequalities are satisfied with $u_{2}$ in place of $u_{1}$ for all
$\varphi\in W^{1,p}_{0}(\Omega)$ with $\varphi\geq 0$ a.e. in $\Omega$.

######  Theorem 2.3.

Let $g:\mathbb{R}\rightarrow\mathbb{R}$ be a continuous function such that
$t^{1-q}g(t)$ is nonincreasing for $t>0$ if $\mu>0$, and $t^{1-p}g(t)$ is
nonincreasing for $t>0$ if $\mu=0$. Assume that $u_{1}\in W^{1,p}_{0}(\Omega)$
and $u_{2}\in W^{1,p}_{0}(\Omega)$ are a positive subsolution and a positive
supersolution of problem (2.6), respectively. If $u_{i}\in
L^{\infty}(\Omega)\cap C^{1,\alpha}(\Omega)$, $\Delta_{p}u_{i}\in
L^{\infty}(\Omega)$, $u_{i}/u_{j}\in L^{\infty}(\Omega)$ for $i,j=1,2$, then
$u_{2}\geq u_{1}$ in $\Omega$.

##  3\. Approximation problem

For each $n\in\mathbb{N}$, we define the auxiliary problem $(P_{n})$ by

$None$
$\left\\{\begin{array}[]{lll}-\Delta_{N}u=\lambda(a_{1}(u_{+})^{r_{1}}+a_{2}|\nabla
u|^{r_{2}})+f_{n}(u)+\frac{1}{n}&\mbox{in}&\Omega\\\ u>0&\mbox{in}&\Omega\\\
u=0&\mbox{on}&\partial\Omega,\end{array}\right.$

where $f_{n}$ are given by Lemma 2.1 and Lemma 2.2, and $u_{+}=\max\\{u,0\\}$.

To prove Theorem 1.1 we first show the existence of a solution of problem
$(P_{n})$ by using the Galerkin method. We would like to cite [1] as the
seminal paper in this type of approach.

###  3.1. Finite-Dimensional Spaces

Let $\mathcal{B}=\\{w_{1},w_{2},\dots,w_{n},\dots\\}$ be a Schauder basis of
$W^{1,N}_{0}(\Omega)$ (see [15, 4]). For each positive integer $m$, let

$W_{m}=[w_{1},w_{2},\dots,w_{m}]$

be the $m$-dimensional subspace of $W^{1,N}_{0}(\Omega)$ (generated by
$\\{w_{1},w_{2},\dots,w_{m}\\}$) with norm induced from $W^{1,N}_{0}(\Omega)$.
Let $\xi=(\xi_{1},\ldots,\xi_{m})\in\mathbb{R}^{m}$, notice that

$|\xi|_{m}=\|\sum_{j=1}^{m}\xi_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)},$

defines a norm in $\mathbb{R}^{m}$. In fact, let
$\xi^{i}=(\xi^{i}_{1},\ldots,\xi^{i}_{m})\in\mathbb{R}^{m}$, $i=1,2$, and let
$\lambda\in\mathbb{R}$.

  * (i)

$|\xi^{1}+\xi^{2}|_{m}\leq|\xi^{1}|_{m}+|\xi^{2}|_{m}$:

$\displaystyle|\xi^{1}+\xi^{2}|_{m}$ $\displaystyle=$
$\displaystyle\|\sum_{j=1}^{m}\xi^{1}_{j}w_{j}+\sum_{j=1}^{m}\xi^{2}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}$
$\displaystyle\leq$
$\displaystyle\|\sum_{j=1}^{m}\xi^{1}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}+\|\sum_{j=1}^{m}\xi^{2}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}$
$\displaystyle=$ $\displaystyle|\xi^{1}|_{m}+|\xi^{2}|_{m}.$

  * (ii)

$|\lambda\xi^{1}|_{m}=|\lambda||\xi^{1}|_{m}$:

$\displaystyle|\lambda\xi^{1}|_{m}=\|\lambda\sum_{j=1}^{m}\xi^{1}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}=|\lambda|\|\sum_{j=1}^{m}\xi^{1}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}=|\lambda||\xi^{1}|_{m}.$

  * (iii)

$|\xi^{1}|_{m}=0\Leftrightarrow\xi^{1}=0$:

$(\Rightarrow)$
$0=|\xi^{1}|_{m}=\|\sum_{j=1}^{m}\xi^{1}_{j}w_{j}\|_{W^{1,N}_{0}(\Omega)}$
implies $\sum_{j=1}^{m}\xi^{1}_{j}w_{j}=0$. By uniqueness of the
representation (using a Schauder basis) of the null vector, we conclude that
$\xi^{1}=0$.

$(\Leftarrow)$ It is trivial.

By using the above notation, we can identify the normed spaces
$(W_{m},\|\cdot\|_{W^{1,N}_{0}(\Omega)})$ and $(\mathbb{R}^{m},|\cdot|_{m})$
by the isometric linear transformation

(3.1) $v=\sum_{j=1}^{m}\xi_{j}w_{j}\in
V_{m}\mapsto\xi=(\xi_{1},\ldots,\xi_{m})\in\mathbb{R}^{m}.$

The lemma below is a consequence of Brouwer’s Fixed Point Theorem and its
proof can be found in Kesavan [17].

######  Lemma 3.1.

Let $F:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ be a continuous function such
that $\left\langle F(\xi),\xi\right\rangle\geq 0$ for every
$\xi\in\mathbb{R}^{d}$ with $|\xi|=r$ for some $r>0$. Then, there exists
$z_{0}$ in the closed ball $\overline{B}_{r}(0)$ such that $F(z_{0})=0$.

###  3.2. Existence

The following result is concerning the existence result for the auxiliary
problem $(P_{n})$.

######  Lemma 3.2.

There exists $\lambda^{*}>0$ and $n^{*}\in\mathbb{N}$ such that $(P_{n})$
admits a (positive) weak solution $v\in W^{1,N}_{0}(\Omega)\cap
C^{1,\alpha}(\overline{\Omega})$, for some $0<\alpha<1$, for every
$\lambda\in(0,\lambda^{*})$ and $n\geq n^{*}$.

###### Proof.

Let $\mathcal{B}=\\{w_{1},w_{2},\dots,w_{n},\dots\\}$ be a Schauder basis of
$W^{1,N}_{0}(\Omega)$. For each positive integer $m$, let
$W_{m}=[w_{1},w_{2},\dots,w_{m}]$. By using the isometric linear
transformation (3.1), define the function $F:\mathbb{R}^{m}\to\mathbb{R}^{m}$
such that $F(\xi)=(F_{1}(\xi),F_{2}(\xi),\dots,F_{m}(\xi))$, where

$\begin{array}[]{lll}F_{j}(\xi)&=&\displaystyle\int_{\Omega}|\nabla
u|^{N-2}\nabla u\nabla
w_{j}dx-\lambda\left(a_{1}\int_{\Omega}(u_{+})^{r_{1}}w_{j}dx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}w_{j}\right)dx\\\
&&-\displaystyle\int_{\Omega}f_{n}(u_{+})w_{j}-\frac{1}{n}\int_{\Omega}w_{j}dx,\,\,\,\,j=1,\ldots,m.\end{array}$

Therefore,

(3.2) $\left\langle F(\xi),\xi\right\rangle=\int_{\Omega}|\nabla
u|^{N}dx-\lambda\left(a_{1}\int_{\Omega}(u_{+})^{r_{1}}udx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}udx\right)-\int_{\Omega}f_{n}(u_{+})udx-\frac{1}{n}\int_{\Omega}udx.$

Given $u\in W_{m}$, we define

$\Omega^{+}_{n}=\\{x\in\Omega:|u(x)|\geq\frac{1}{n}\\}$

and

$\Omega^{-}_{n}=\\{x\in\Omega:|u(x)|<\frac{1}{n}\\}.$

Thus, we rewrite (3.2) as

$\left\langle F(\xi),\xi\right\rangle=\left\langle
F(\xi),\xi\right\rangle_{P}+\left\langle F(\xi),\xi\right\rangle_{N},$

where

$\left\langle F(\xi),\xi\right\rangle_{P}=\int_{\Omega^{+}_{n}}|\nabla
u|^{N}dx-\lambda\left(a_{1}\int_{\Omega^{+}_{n}}(u_{+})^{r_{1}}udx+a_{2}\int_{\Omega^{+}_{n}}|\nabla
u|^{r_{2}}udx\right)-\int_{\Omega^{+}_{n}}f_{n}(u_{+})u_{+}dx-\frac{1}{n}\int_{\Omega^{+}_{n}}udx$

and

$\left\langle F(\xi),\xi\right\rangle_{N}=\int_{\Omega^{-}_{n}}|\nabla
u|^{N}dx-\lambda\left(a_{1}\int_{\Omega^{-}_{n}}(u_{+})^{r_{1}}udx+a_{2}\int_{\Omega^{-}_{n}}|\nabla
u|^{r_{2}}udx\right)-\int_{\Omega^{-}_{n}}f_{n}(u_{+})u_{+}dx-\frac{1}{n}\int_{\Omega^{-}_{n}}udx.$

Step 1. Since $0<r_{i}<N-1$, for $i=1,2$, then

(3.3)
$\int_{\Omega^{+}_{n}}(u_{+})^{r_{1}+1}dx\leq\int_{\Omega}(u_{+})^{r_{1}+1}dx\leq\int_{\Omega}|u|^{r_{1}+1}dx=\|u\|^{r_{1}+1}_{L^{r_{1}+1}(\Omega)}\leq
C_{1}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{1}+1}.$

By virtue of Lemma 2.2 (i) we get

(3.4)
$\begin{array}[]{rcl}\displaystyle\int_{\Omega^{+}_{n}}f_{n}(u_{+})u_{+}dx&\leq&\displaystyle
C_{1}\int_{\Omega^{+}_{n}}|u|^{r_{3}+1}\exp(2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\\\
&\leq&\displaystyle
a_{3}\left(\int_{\Omega}|u|^{N^{\prime}(r_{3}+1)}dx\right)^{\frac{1}{N^{\prime}}}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N}}\\\
&=&\displaystyle
a_{3}\|u\|_{L^{N^{\prime}(r_{3}+1)}(\Omega)}^{r_{3}+1}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N}},\end{array}$

where $\frac{1}{N}+\frac{1}{N^{\prime}}=1$.

It follows from (3.3) and (3.4) that

(3.5) $\begin{array}[]{rcl}\displaystyle\left\langle
F(\xi),\xi\right\rangle_{P}&\geq&\displaystyle\int_{\Omega^{+}_{n}}|\nabla
u|^{N}dx-\lambda\left(a_{1}C_{1}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{1}+1}+a_{2}\int_{\Omega^{+}_{n}}|\nabla
u|^{r_{2}}udx\right)\\\
&&-C_{3}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{3}+1}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N}}-\displaystyle\frac{C_{4}}{n}\|u\|_{W^{1,N}_{0}(\Omega)},\end{array}$

where $C_{0}$, $C_{1}$ and $C_{3}$ are constants not depending $n$ and $m$.

Step 2. Since $0<r_{i}<N-1$, for $i=1,2$, then

(3.6)
$\int_{\Omega^{-}_{n}}(u_{+})^{r_{1}+1}\leq\int_{\Omega^{-}_{n}}|u|^{r_{1}+1}\leq|\Omega|\frac{1}{n^{r_{1}+1}}.$

By virtue of Lemma 2.2 (ii) we get

(3.7) $\int_{\Omega^{-}_{n}}f_{n}(u_{+})u_{+}\leq
C_{2}\int_{\Omega^{-}_{n}}|u|^{2}\exp(2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\leq
C_{2}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}.$

It follows from (3.6) and (3.7) that

(3.8) $\left\langle
F(\xi),\xi\right\rangle_{N}\geq\int_{\Omega^{-}_{n}}|\nabla
u|^{N}-\lambda\left(a_{1}|\Omega|\frac{1}{n^{r_{1}+1}}+a_{2}\int_{\Omega^{-}_{n}}|\nabla
u|^{r_{2}}udx\right)\\\
-C_{3}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}-\displaystyle|\Omega|\frac{1}{n^{2}}.$

Since

$\int_{\Omega^{+}_{n}}|\nabla u|^{r_{2}}udx+\int_{\Omega^{-}_{n}}|\nabla
u|^{r_{2}}udx=\int_{\Omega}|\nabla u|^{r_{2}}udx$

and

(3.9) $\int_{\Omega}|\nabla u|^{r_{2}}|u|dx\leq\left(\int_{\Omega}|\nabla
u|^{N}dx\right)^{r_{2}/N}\left(\int_{\Omega}|u|^{N/(N-r_{2})}dx\right)^{(N-r_{2})/N}\leq
C\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{2}+1}.$

Thus (3.5), (3.8) and (3.9) imply

(3.10) $\begin{array}[]{rcl}\displaystyle\left\langle
F(\xi),\xi\right\rangle&\geq&\displaystyle\|u\|_{W^{1,N}_{0}(\Omega)}^{N}-\lambda\left(a_{1}C_{1}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{1}+1}+a_{2}C_{2}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{2}+1}\right)\\\
&-&\displaystyle
C_{3}\|u\|_{W^{1,N}_{0}(\Omega)}^{r_{3}+1}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N}}-\displaystyle\frac{C_{4}}{n}\|u\|_{W^{1,N}_{0}(\Omega)}\\\
&-&\displaystyle\lambda
a_{1}|\Omega|\frac{1}{n^{r_{1}+1}}-C_{5}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}-\displaystyle|\Omega|\frac{1}{n^{2}}.\end{array}$

Assume now that $\|u\|_{W^{1,N}_{0}(\Omega)}=r$ for some $r>0$ to be chosen
later. We have

(3.11)
$\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u|^{\frac{N}{N-1}})dx=\int_{\Omega}\exp\left(N2^{\frac{N}{N-1}}\alpha\,r^{\frac{N}{N-1}}\left(\frac{|u|}{\|u\|_{W^{1,N}_{0}(\Omega)}}\right)^{\frac{N}{N-1}}\right)dx$

and in order to apply the Trudinger-Moser inequality (1.4) we must have
$N2^{\frac{N}{N-1}}\alpha\,r^{\frac{N}{N-1}}\leq\alpha_{N}$. Consequently,

$r\leq\frac{1}{2}\left(\frac{\alpha_{N}}{N\alpha}\right)^{\frac{N-1}{N}}.$

Then

$\sup_{\|u\|_{W^{1,N}_{0}(\Omega)}\leq
1}\int_{\Omega}\exp\left(N2^{\frac{N}{N-1}}\alpha\,r^{\frac{N}{N-1}}|u|^{\frac{N}{N-1}}\right)dx\leq
L(N)|\Omega|.$

Hence,

$\begin{array}[]{rcl}\left\langle F(\xi),\xi\right\rangle&\geq&\displaystyle
r^{N}-\lambda(a_{1}C_{1}r^{r_{1}+1}+a_{2}C_{2}r^{r_{2}+1})-C_{3}r^{r_{3}+1}L^{1/N}(N)-\displaystyle\frac{C_{4}}{n}r\\\
&-&\displaystyle\lambda
a_{1}|\Omega|\frac{1}{n^{r_{1}+1}}-C_{5}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}-\displaystyle|\Omega|\frac{1}{n^{2}}.\end{array}$

We need to choose $r$ such that

$r^{N}-C_{3}L(N)^{\frac{1}{N}}r^{r_{3}+1}\geq\frac{r^{N}}{2},$

in other words,

$r\leq\frac{1}{(2C_{3}L(N)^{\frac{1}{N}})^{\frac{1}{r_{3}+1-N}}}.$

Thus, let
$r=\min\left\\{\frac{1}{2(2C_{3}L(N)^{\frac{1}{N}})^{\frac{1}{r_{3}+1-N}}},\frac{1}{2}\left(\frac{\alpha_{N}}{N\alpha}\right)^{\frac{N-1}{N}}\right\\}$,
hence

$\left\langle
F(\xi),\xi\right\rangle\geq\frac{r^{N}}{2}-\lambda(a_{1}C_{1}r^{r_{1}+1}+a_{2}C_{2}r^{r_{2}+1})-\displaystyle\frac{C_{4}}{n}r-\displaystyle\lambda
a_{1}|\Omega|\frac{1}{n^{r_{1}+1}}-C_{5}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}-\displaystyle|\Omega|\frac{1}{n^{2}}.$

Now, defining
$\rho=\frac{r^{N}}{2}-\lambda(a_{1}C_{1}r^{r_{1}+1}+a_{2}C_{2}r^{r_{2}+1})$,
we choose $\lambda^{*}>0$ such that $\rho>0$ for $\lambda<\lambda^{*}$. Since
$0<r_{i}<N-1$, for $i=1,2$, we can choose

$\lambda^{*}=\frac{1}{2}\frac{r^{N}}{2a_{1}C_{1}r^{r_{1}+1}+2a_{2}C_{2}r^{r_{2}+1}}.$

Now we choose $n^{*}\in\mathbb{N}$ such that

$\displaystyle\frac{C_{4}}{n}r+\displaystyle\lambda
a_{1}|\Omega|\frac{1}{n^{r_{1}+1}}+C_{5}\exp(2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{2}}+\displaystyle|\Omega|\frac{1}{n^{2}}<\frac{\rho}{2},$

for every $n\geq n^{*}$. Let $\xi\in\mathbb{R}^{m}$, such that
$|\xi|_{m}:=\left\|\sum_{i=1}^{m}\xi_{i}w_{i}\right\|_{W^{1,N}_{0}(\Omega)}=r$,
then for $\lambda<\lambda^{*}$ and $n\geq n^{*}$ we obtain

(3.12) $\left\langle F(\xi),\xi\right\rangle\geq\frac{\rho}{2}>0.$

Then by Lemma 3.1, for every $m\in\mathbb{N}$ there exists
$y\in\mathbb{R}^{m}$ (with $|y|_{m}\leq r$) such that $F(y)=0$. Therefore,
there exists $u_{m}\in W_{m}$ verifying

(3.13) $\|u_{m}\|_{W^{1,N}_{0}(\Omega)}\leq r,\,\,\mbox{for
every}\,\,m\in\mathbb{N},$

and such that

(3.14) $\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla
u_{m}|^{N-2}\nabla u_{m}\nabla
wdx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u_{m}^{+})^{r_{1}}wdx+a_{2}\int_{\Omega}|\nabla
u_{m}|^{r_{2}}wdx\right)\\\
&&+\displaystyle\int_{\Omega}f_{n}((u_{m})_{+})wdx+\frac{1}{n}\int_{\Omega}wdx,\,\,\forall\,w\in
W_{m}.\end{array}$

Since $W_{m}\subset W^{1,N}_{0}(\Omega)$ $\forall\,m\in\mathbb{N}$ and $r$
does not depend on $m$, then $(u_{m})$ is a bounded sequence in
$W^{1,N}_{0}(\Omega)$. Then, for some subsequence, there exists $u_{n}\in
W^{1,N}_{0}(\Omega)$ (to simplify the notation, until the end of this section
we will omit the subscript $n$ of the variable $u$) such that

(3.15) $u_{m}\rightharpoonup u\,\,\,\mbox{weakly in}\,\,\,W^{1,N}_{0}(\Omega)$

and

(3.16) $u_{m}\to u\,\,\,\mbox{in}\,\,\,L^{N}(\Omega)\,\,\,\mbox{and a.e.
in}\,\,\,\Omega.$

Notice that

(3.17)
$\|u\|_{W^{1,N}_{0}(\Omega)}\leq\liminf_{m\to\infty}\|u_{m}\|_{W^{1,N}_{0}(\Omega)}\leq
r,\,\,\forall\,n\in\mathbb{N},$

and $r$ does not depend on $n$. We claim that

(3.18) $u_{m}\to u\,\,\,\mbox{in}\,\,\,W^{1,N}_{0}(\Omega).$

Using the fact that $\mathcal{B}=\\{w_{1},w_{2},\dots,w_{n},\dots\\}$ is a
Schauder basis of $W^{1,N}_{0}(\Omega)$, for every $u\in W^{1,N}_{0}(\Omega)$
there exists a unique sequence $(\alpha_{n})_{n\geq 1}$ in $\mathbb{R}$ such
that $u=\sum_{j=1}^{\infty}\alpha_{j}w_{j}$, so

(3.19) $\psi_{m}:=\sum_{j=1}^{m}\alpha_{j}w_{j}\rightarrow u\,\,\mbox{ in
}W_{0}^{1,N}(\Omega)\,\,\mbox{ as }m\rightarrow\infty.$

Using as test function $w=(u_{m}-\psi_{m})\in W_{m}$ in (3.14), we get

(3.20) $\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla
u_{m}|^{N-2}\nabla
u_{m}\nabla(u_{m}-\psi_{m})dx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u_{m}^{+})^{r_{1}}(u_{m}-\psi_{m})dx+a_{2}\int_{\Omega}|\nabla
u_{m}|^{r_{2}}(u_{m}-\psi_{m})dx\right)\\\
&&\displaystyle+\int_{\Omega}f_{n}((u_{m})_{+})(u_{m}-\psi_{m})dx+\frac{1}{n}\int_{\Omega}(u_{m}-\psi_{m})dx.\end{array}$

By continuity of $f_{n}$, (3.15), (3.16), (3.19) and hypothesis $(F)$, we get

(3.21)
$\lim_{m\rightarrow\infty}\frac{1}{n}\int_{\Omega}(u_{m}-\psi_{m})dx=0,$
(3.22) $\lim_{m\rightarrow\infty}\displaystyle
a_{1}\int_{\Omega}(u_{m}^{+})^{r_{1}}(u_{m}-\psi_{m})dx=0,$ (3.23)
$\lim_{m\rightarrow\infty}a_{2}\int_{\Omega}|\nabla
u_{m}|^{r_{2}}(u_{m}-\psi_{m})dx=0,$

and

(3.24)
$\lim_{m\rightarrow\infty}\int_{\Omega}f_{n}((u_{m})_{+})(u_{m}-\psi_{m})dx=0.$

Notice that (3.21), (3.22) and (3.23) are immediately. Let us verigy (3.24).
By continuity of $f_{n}$ and (3.16) we obtain

$f_{n}((u_{m})_{+})^{N^{\prime}}\to f_{n}(u_{+})^{N^{\prime}}\,\,\,\mbox{ a.e.
in}\,\,\,\Omega$

and by Lemma 2.1 and (3.13), we obtain

$\begin{array}[]{rcl}\displaystyle\int_{\Omega}f_{n}((u_{m})_{+})^{N^{\prime}}dx&\leq&\displaystyle
c_{n}^{N^{\prime}}\int_{\Omega}|u_{m}|^{N^{\prime}}=\|u_{m}\|_{L^{N^{\prime}}(\Omega)}^{N^{\prime}}\leq
c_{n}^{N^{\prime}}C\|u_{m}\|^{N^{\prime}}_{W^{1,N}_{0}(\Omega)}\leq
c_{n}^{N^{\prime}}Cr^{N^{\prime}}.\\\ \end{array}$

Hence, [16, Theorem 13.44] leads to

(3.25) $f_{n}((u_{m})_{+})\to f_{n}(u_{+})\,\,\,\mbox{ weakly in
}\,\,\,L^{N^{\prime}}(\Omega).$

Applying (3.16), (3.19) and (3.25), we conclude that (3.24) holds.

By (3.13) and (3.15), we obtain

(3.26) $\lim_{m\rightarrow\infty}\displaystyle\int_{\Omega}|\nabla
u_{m}|^{N-2}\nabla u_{m}\nabla(u-\psi_{m})dx=0.$

By $\eqref{eq113}-\eqref{eq116}$ and (3.26), we obtain

(3.27) $\lim_{m\rightarrow\infty}\displaystyle\int_{\Omega}|\nabla
u_{m}|^{N-2}\nabla u_{m}\nabla(u_{m}-u)dx=0.$

Now it is sufficient to apply the $(S_{+})-$ property of $-\Delta_{p}$ (see,
e.g., [23, Proposition 3.5.]) for obtaining (3.18).

Let $k\in\mathbb{N}$, then for every $m\geq k$ we obtain

$\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla u_{m}|^{N-2}\nabla
u_{m}\nabla
w_{k}dx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u_{m}^{+})^{r_{1}}w_{k}dx+a_{2}\int_{\Omega}|\nabla
u_{m}|^{r_{2}}w_{k}dx\right)+\int_{\Omega}f_{n}((u_{m})_{+})w_{k}dx\\\
&&\displaystyle+\frac{1}{n}\int_{\Omega}w_{k}dx,\,\,\,\forall\,w_{k}\in
W_{k}.\end{array}$

Lettin $m\rightarrow\infty$, on accout of (3.18) we arrive at

$\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla u|^{N-2}\nabla u\nabla
w_{k}dx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u^{+})^{r_{1}}w_{k}dx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}w_{k}dx\right)+\int_{\Omega}f_{n}(u_{+})w_{k}dx\\\
&&\displaystyle+\frac{1}{n}\int_{\Omega}w_{k}dx,\,\,\,\forall\,w_{k}\in
W_{k}.\end{array}$

Since $[W_{k}]_{k\in\mathbb{N}}$ is dense in $W^{1,N}_{0}(\Omega)$ we conclude
that

$\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla u|^{N-2}\nabla u\nabla
wdx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u^{+})^{r_{1}}wdx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}wdx\right)+\int_{\Omega}f_{n}(u_{+})wdx\\\
&&\displaystyle+\frac{1}{n}\int_{\Omega}wdx,\,\,\,\forall\,w\in
W^{1,N}_{0}(\Omega).\end{array}$

Furthermore, $u\geq 0$ in $\Omega$. In fact, since $u_{-}\in
W^{1,N}_{0}(\Omega)$ then from (3.2) we obtain

$\begin{array}[]{lll}\displaystyle\int_{\Omega}|\nabla u|^{N-2}\nabla u\nabla
u_{-}dx&=&\displaystyle\lambda\left(a_{1}\int_{\Omega}(u^{+})^{r_{1}}u_{-}dx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}u_{-}dx\right)+\int_{\Omega}f_{n}(u_{+})u_{-}dx\\\
&&\displaystyle+\frac{1}{n}\int_{\Omega}u_{-}dx.\end{array}$

Hence

$-\|u_{-}\|_{W^{1,N}_{0}(\Omega)}^{N}=\lambda\left(a_{1}\int_{\Omega}(u^{+})^{r_{1}}u_{-}dx+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}u_{-}dx\right)+\int_{\Omega}f_{n}(u_{+})u_{-}dx+\frac{1}{n}\int_{\Omega}u_{-}dx\geq
0,$

because $\int_{\Omega}f_{n}(u_{+})u_{-}dx=0$. Then $u_{-}\equiv 0$ a.e. in
$\Omega$.

The first inequality in hypothesis $(F)$ and the equation in $(P_{n})$
guarantee that $u\not=0$. Here the presence of $\frac{1}{n}>0$ is needed.
Next, we observe that hypothesis $(F)$ allows us to refer to [18, Theorem 7.1]
from which we infer that $u\in L^{\infty}(\Omega)$. Furthermore, the
regularity result up to the boundary in [20, Theorem 1] and [21, p. 320]
ensures that $u\in C^{1,\beta}(\overline{\Omega})$ with some $\beta\in(0,1)$.
We also note that we may apply the strong maximum principle in [24, Theorem
5.4.1]. We are thus in a position to apply [24, Theorem 5.4.1] concluding that
$u>0$ in $\Omega$ because we know that $u\geq 0$, $u\not=0$, thereby $u$ is a
solution of problem $(P_{n})$. This completes the proof. ∎

######  Remark 3.3.

To apply [18, Theorem 7.1] and infer that $u\in L^{\infty}(\Omega)$, notice
that it is necessary to consider the approximating functions $f_{n}$ (given by
Lemma 2.1) instead of $f$. In fact, since

$F_{n}(v,p)=\lambda(a_{1}v^{r_{1}}+a_{2}|p|^{r_{2}})+f_{n}(v)+\frac{1}{n}$

satisfies the inequality $(7.2)$ in [18], because

$sign(v).F_{n}(v,p)\leq\lambda(a_{1}|v|^{r_{1}}+a_{2}|p|^{r_{2}})+c_{n}|v|+\frac{1}{n}.$

While $F(v,p)=\lambda(a_{1}v^{r_{1}}+a_{2}|p|^{r_{2}})+f(v)$ does not
necessarily satisfy such a hypothesis, in fact

$sign(v).F(v,p)\leq\lambda(a_{1}|v|^{r_{1}}+a_{2}|p|^{r_{2}})+a_{3}|u|^{r_{3}}\exp(\alpha|v|^{\frac{N}{N-1}}).$

##  4\. Proof of the main result

In this section we will prove Theorem 1.1. Consider the following problem

(4.1) $\left\\{\begin{array}[]{lll}-\Delta_{N}v=\lambda
a_{1}v^{r_{1}}&\mbox{in}&\Omega\\\ v>0&\mbox{in}&\Omega\\\
v=0&\mbox{on}&\partial\Omega\end{array}\right.$

where $\lambda,a_{1}$ and $r_{1}$ were given in Theorem 1.1. This problem
admits a solution $v_{0}\in C^{1}_{0}(\overline{\Omega})$, see for instace
[13, Lemma 4.1]. The function $v_{0}$ allows us to bound from below the
solutions of $(P_{n})$.

For each $\lambda\in(0,\lambda^{*})$ and $n\in\mathbb{N}$ sufficiently large,
by using Lemma 3.2, we get that equation $(P_{n})$ has a weak solution
$u_{n}\in W^{1,N}_{0}(\Omega)\cap C^{1,\alpha}(\overline{\Omega})$ for some
$\alpha\in(0,1)$.

In view of (3.17), we can argue as for (3.18), to find a subsequence
$n\rightarrow\infty$ such that the corresponding sequence $\\{u_{n}\\}$ is
strongly convergent:

(4.2) $u_{n}\to u\,\,\,\mbox{in}\,\,\,W^{1,N}_{0}(\Omega).$

In fact, for some subsequence, there exists $u\in W^{1,N}_{0}(\Omega)$ such
that

(4.3) $u_{n}\rightharpoonup u\,\,\,\mbox{weakly in}\,\,\,W^{1,N}_{0}(\Omega)$

and, by Sobolev embedding for $1\leq s<+\infty$,

$u_{n}\rightarrow u\,\,\mbox{in}\,\,L^{s}(\Omega)\,\,\mbox{and}\,\,\mbox{a.e.
in}\,\,\Omega.$

Notice that

(4.4) $\begin{array}[]{c}\displaystyle\int_{\Omega}|\nabla u_{n}|^{N-2}\nabla
u_{n}\nabla w\geq\lambda\,a_{1}\int_{\Omega}(u_{n})^{r_{1}}w,\forall w\in
W^{1,N}_{0}(\Omega)\mbox{ with }w\geq 0.\\\ \end{array}$

Since

$u_{n}\rightarrow u\,\,\mbox{a.e. in}\,\,\Omega,$

we have

(4.5) $f_{n}(u_{n}(x))\rightarrow f(u(x))\,\,\mbox{a.e. in}\,\,\Omega,$

by the uniform convergence of Lemma 2.1 ($iii$).

By Lemma 2.2

$\begin{array}[]{rcl}\displaystyle\int_{\Omega}f_{n}(u_{n})^{N^{\prime}}dx&=&\displaystyle\int_{\Omega_{n}^{+}}f_{n}(u_{n})^{N^{\prime}}dx+\int_{\Omega_{n}^{-}}f_{n}(u_{n})^{N^{\prime}}dx,\\\
\end{array}$
$\begin{array}[]{rcl}\displaystyle\int_{\Omega^{+}_{n}}f_{n}(u_{n})^{N^{\prime}}dx&\leq&\displaystyle
C_{1}^{\frac{N}{N-1}}\int_{\Omega^{+}_{n}}|u_{n}|^{(r_{3}-1)\frac{N}{N-1}}\exp(\frac{N}{N-1}2^{\frac{N}{N-1}}\alpha|u_{n}|^{\frac{N}{N-1}})dx\\\
&\leq&\displaystyle
C_{1}^{\frac{N}{N-1}}\left(\int_{\Omega}|u_{n}|^{(r_{3}-1)\frac{N}{N-2}}dx\right)^{\frac{N-2}{N-1}}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u_{n}|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N-1}}\\\
&=&\displaystyle
C_{1}^{\frac{N}{N-1}}\|u_{n}\|_{L^{(r_{3}-1)\frac{N}{N-2}}(\Omega)}^{(r_{3}-1)\frac{N}{N-1}}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u_{n}|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N-1}}\\\
&\leq&\displaystyle
C\|u_{n}\|_{{}_{W^{1,N}_{0}(\Omega)}}^{(r_{3}-1)\frac{N}{N-1}}\left(\int_{\Omega}\exp(N2^{\frac{N}{N-1}}\alpha|u_{n}|^{\frac{N}{N-1}})dx\right)^{\frac{1}{N-1}}\end{array}$

and

$\int_{\Omega^{-}_{n}}f_{n}(u_{n})^{N^{\prime}}dx\leq
C_{2}^{\frac{N}{N-1}}\int_{\Omega^{-}_{n}}|u_{n}|^{\frac{N}{N-1}}\exp(\frac{N}{N-1}2^{\frac{N}{N-1}}\alpha|u_{n}|^{\frac{N}{N-1}})dx$
$\leq
C_{2}^{\frac{N}{N-1}}\exp(\frac{N}{N-1}2^{\frac{N}{N-1}}\alpha)|\Omega|\frac{1}{n^{\frac{N}{N-1}}}.$

Since $\|u_{n}\|_{{}_{W^{1,N}_{0}(\Omega)}}\leq r$, by the estimates before,
we obtain

$\displaystyle\int_{\Omega}f_{n}(u_{n})^{N^{\prime}}dx\leq C,$

for each $n$. Since $f_{n}(u_{n}(x))\rightarrow f(u(x))\,\,\mbox{a.e.
in}\,\,\Omega$, [16, Theorem 13.44] leads to

(4.6) $f_{n}(u_{n})\to f(u)\,\,\,\mbox{ weakly in
}\,\,\,L^{N^{\prime}}(\Omega).$

Recall from (3.2) that, forall $w\in W^{1,N}_{0}(\Omega)$,

(4.7) $\int_{\Omega}|\nabla u_{n}|^{N-2}\nabla u_{n}\nabla
w=\lambda\left(a_{1}\int_{\Omega}(u_{n})^{r_{1}}w+a_{2}\int_{\Omega}|\nabla
u_{n}|^{r_{2}}w\right)+\int_{\Omega}f_{n}(u_{n})w+\frac{1}{n}\int_{\Omega}w.$

Taking $w=u_{n}-u$ in (4.7), we obtain

(4.8) $\lim_{m\rightarrow\infty}\displaystyle\int_{\Omega}|\nabla
u_{n}|^{N-2}\nabla u_{n}\nabla(u_{n}-u)dx=0.$

Now it is sufficient to apply the $(S_{+})-$ property of $-\Delta_{p}$ for
obtaining (4.2).

Then from (4.2) and (4.6) and the fact that $u_{n}$ solves $(P_{n})$, by
passing to limit when $n\to+\infty$ we get that

(4.9) $\int_{\Omega}|\nabla u|^{N-2}\nabla u\nabla
w=\lambda\left(a_{1}\int_{\Omega}(u)^{r_{1}}w+a_{2}\int_{\Omega}|\nabla
u|^{r_{2}}w\right)+\int_{\Omega}f(u)w,\,\,\,\forall\,w\in
W^{1,N}_{0}(\Omega).$

Now, we are going to check that $u>0$ in $\Omega$. Notice that, by (4.1) and
(4.4), for each $n$ sufficiently large $u_{n}$ is a supersolution and $v_{0}$
is a subsolution of Problem (4.1). In order to apply Theorem 2.3, we need to
check that $\frac{u_{n}}{v_{0}},\frac{v_{0}}{u_{n}}\in L^{\infty}(\Omega)$.
This follows by using Hopf boundary point lemma (in the strong maximum
principle for both Dirichlet problems (4.1) and $(P_{n})$ with corresponding
solutions $v_{0}$ and $u_{n}$), regularity up to the boundary and L’Hôpital
theorem (see [13] for details). Therefore, $u_{n}(x)\geq v_{0}(x)>0$ for all
$x\in\Omega$. Thus, by passing to the limit, we conclude that $u$ is a
positive solution of problem $(P)$ and the proof of the theorem is thus
complete. ∎

## References

  * [1] C. O. Alves and D. G. de Figueiredo, _Nonvariational Elliptic Systems via Galerkin Methods_ , D. Haroske, T. Runst and H. J. Schmeisser (eds.) Function Spaces, Differential Operators and Nonlinear Analysis. The Hans Triebel Anniversary Volume, 2003
  * [2] H. Amann, M.G. Crandall, On some existence theorems for semi-linear elliptic equations, Indiana Univ. Math. J. 27 (1978) 779-790.
  * [3] F. V. Atkinson and L. A. Peletier, _Elliptic equations with critical growth_. Math. Inst. Univ. Leiden, Rep. 21 (1986)
  * [4] H. Brézis, _Functional analysis, Sobolev spaces and partial differential equations_ , (Springer, 2011).
  * [5] H. Brézis, R.E.L. Turner, On a class of superlinear elliptic problems, Comm. Partial Differential Equations 2 (1977) 601–614.
  * [6] A.L.A. de Araujo, M. Montenegro, Existence of solution for a general class of elliptic equations with exponential growth, Ann. Mat. 195 (2016) 1737–1748.
  * [7] A.L.A. de Araujo, M. Montenegro, Existence of solution for a nonvariational elliptic system with exponential growth in dimension two. J. Differential Equations, v. 264 (2018) 2270–2286.
  * [8] D.G. de Figueiredo, M. Girardi and M. Matzeu, Semilinear elliptic equations with dependence on the gradient via mountain-pass techniques, Dier. Integr. Equ., 17 (2004), 119-126.
  * [9] de Figueiredo, D.G., do Ó, J.M., Ruf, B.: On an inequality by N. Trudinger and J. Moser and related elliptic equations. Commun. Pure Appl. Math. 55, 135–152 (2002)
  * [10] de Figueiredo, D.G., Miyagaki, O.H., Ruf, B.: Elliptic equations in R2 with nonlinearities in the critical growth range. Calc. Var. Partial Differ. Equ. 3, 139–153 (1995)
  * [11] L.R. de Freitas, Multiplicity of solutions for a class of quasilinear equations with exponential critical growth. Nonlinear Anal. TMA 95, 607–624 (2014)
  * [12] F. Faraci, D. Motreanu and D. Puglisi, Positive solutions of quasi-linear elliptic equations with dependence on the gradient, Calc. Var. Partial Dierential Equations, 54 (2015), 525-538.
  * [13] L.F.O Faria, O.H. Miyagaki, D. Motreanu, Comparison and positive solutions for problemswith (p, q)-Laplacian and convection term. Proc. Edinb. Math. Soc. 57(2), 687-698 (2014).
  * [14] L. F. O. Faria, O. H. Miyagaki, D. Motreanu and M. Tanaka, Existence results for nonlinear elliptic equations with Leray-Lions operator and dependence on the gradient, Nonlinear Anal., 96 (2014), 154-166.
  * [15] S. Fučík, O. John, J. Nečas, _On the existence of Schauder bases in Sobolev spaces_. Comment. Math. Univ. Carol. 13, 163–175 (1972)
  * [16] E. Hewitt and K. Stromberg, Real and Abstract Analysis, SpringerVerlag, 1975.
  * [17] S. Kesavan, _Topics in functional analysis and applications_ , John Wiley and Sons, (1989).
  * [18] O. A. Ladyzhenskaya and N. N. Ural’tseva, _Linear and quasilinear elliptic equations_ (Academic Press, 1968).
  * [19] N.Lam, G. Lu, Elliptic equations and systems with subcritical and critical exponential growth without the Ambrosetti–Rabinowitz condition. J. Geom. Anal. 24, 118–143 (2014)
  * [20] G. M. Lieberman, Boundary regularity for solutions of degenerate elliptic equations, _Nonlinear Anal._ 12 (1988) 1203–1219.
  * [21] G. M. Lieberman, The natural generalization of the natural conditions of Ladyzhenskaya and Ural’tseva for elliptic equations, _Comm. Partial Differential Equations_ 16 (1991), 311–361.
  * [22] J. Moser, _A sharp form of an inequality by Trudinger_. Indiana Univ. Math. J. 20 1077–1092 (1971)
  * [23] D. Motreanu, V.V. Motreanu AND N.S. Papageorgiou, _Multiple constant sign and nodal solutions for Nonlinear Neumann eigenvalue problems_ , Ann. Sc. Norm. Sup. Pisa Cl. Sci. (5), X (2011), 1–27.
  * [24] P. Pucci AND J. Serrin, _The maximum principle_ , (Birkhäuser Verlag, 2007).
  * [25] D. Ruiz, A priori estimates and existence of positive solutions for strongly nonlinear problems, J. Differential Equations 199 (2004) 96–114.
  * [26] E.A.B. Silva, S.H.M. Soares, Liouville-Gelfand type problems for the N-Laplacian on bounded domains of $R^{N}$. Annali della Scuola Normale Superiore di Pisa. Classe di Scienze 4, 1–30 (1999)
  * [27] W. A. Strauss, _On weak solutions of semilinear hyperbolic equations_ , An. Acad. Brasil. Ciênc. 42 645–651 (1970)
  * [28] N. S. Trudinger, _On the imbeddings into Orlicz spaces and applications_. J. Math. Mech. 17 473–484 (1967)
  * [29] Y. Yang and K. Perera, (N, q)-Laplacian problems with critical TrudingerMoser nonlinearities, Bull. London Math. Soc. 48 (2016), 260–270.


# Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch

###### Abstract

Recent research in speech processing exhibits a growing interest in
unsupervised and self-supervised representation learning from unlabelled data
to alleviate the need for large amounts of annotated data. We investigate
several popular pre-training methods and apply them to Flemish Dutch. We
compare off-the-shelf English pre-trained models to models trained on an
increasing amount of Flemish data. We find that the most important factors for
positive transfer to downstream speech recognition tasks include a substantial
amount of data and a matching pre-training domain. Ideally, we also finetune
on an annotated subset in the target language. All pre-trained models improve
linear phone separability in Flemish, but not all methods improve Automatic
Speech Recognition. We experience superior performance with wav2vec 2.0 and we
obtain a 30% WER improvement by finetuning the multilingually pre-trained
XLSR-53 model on Flemish Dutch, after integration into an HMM-DNN acoustic
model.

Index Terms—  speech recognition, self-supervised learning, pre-training,
cross-lingual

##  1 Introduction

Building a good speech recogniser typically requires a large amount of
annotated data from a specific language. Obtaining high-quality labelled data
is a costly and time-intensive process, and for many languages this remains a
big issue. However, even in a highly resourced language like English, recent
work has shown impressive results in Automatic Speech Recognition (ASR) by
pre-training on unlabelled data and transferring that knowledge to regular
speech recognition models [1, 2, 3] or even completely unsupervised speech
recognition [4]. This paradigm shift towards unlabelled data is of great
significance as untranscribed recordings of speech are much easier to acquire.

Self-supervised learning is a clever way to learn general information from
data without requiring any labels. Recently many successful methods have
emerged for self-supervised representation learning from speech. The general
idea is to implicitly learn the global structure and local characteristics
that are inherently present in speech. Depending on the task, both local
information, such as the pronunciation of a specific phoneme, and more global
information, such as speaker traits and recording properties, can be useful.
By pre-training a network with a well-chosen objective function, these
relevant attributes about the input speech can be captured and summarised in
rich feature vectors. This improves several downstream tasks like speech
recognition and typically reduces the amount of required data, since the
principal characteristics are already extracted and more easily accessible.
Moreover, the structure of a speech waveform is to some extent general and
language-independent, which explains the improvements with these features in
low-resource languages [5, 6].

The objective function used in self-supervised learning techniques is the
driving force behind the extraction of powerful speech representations. In
fact, the self-supervised objective has more impact on the learned
representation than architectural differences between methods [7]. In
Autoregressive Predictive Coding (APC) [8, 9, 10, 11], the objective is to
predict a frame a few steps ahead, given the information up to that point.
Another branch of research focuses on predicting the current frame given past
and future context, by reconstructing several masked frames [12, 13], similar
to the Masked Language Modeling (MLM) approach in Natural Language Processing
[14]. Finally, Contrastive Predictive Coding (CPC) [15] is a popular technique
in representation learning, where the objective is to predict the future in
the latent space and a contrastive loss is applied to maximise mutual
information. CPC has been successfully applied to speech recognition [1, 16,
17, 18] and has shown to be able to learn robust and cross-lingual speech
representations [5, 6, 19]. We refer to the literature for other related work
in self-supervised and unsupervised representation learning [20, 21, 22, 23,
24, 25, 26, 27].

Following the widespread improvements in ASR as a result of self-supervised
pre-training, this paper will focus on Flemish Dutch, a medium-resourced
language. Flemish is the language spoken in Flanders, the Dutch-speaking part
of Belgium. It is closely related to the Dutch variant spoken in The
Netherlands, but there are still many noticeable differences [28]. A few
seconds of speech suffice to distinguish the two variants. Although
geographically relatively small, Flemish Dutch is diverse with several
dialects, roughly corresponding to the five provinces in Flanders, though
natives will observe even finer detail. Furthermore, Dutch belongs to the
family of West-Germanic languages, like English and German, which makes it a
very interesting language to examine whether pre-training on English leads to
strong improvements in Dutch. While there is some overlap in phones, there are
also several vowels and diphthongs that do not occur in English.

In this work, we compare several popular self-supervised pre-training methods
when applied to Flemish. First of all, we look at the applicability of off-
the-shelf models that are pre-trained on English and assess the
transferability to Flemish. This would be convenient for several research
domains and technological applications. Additionally it would eliminate the
need for large computational resources necessary to pre-train these models,
which scales with the model size (e.g. the high-capacity wav2vec 2.0 model
[1]) and for many models also with the amount of data. Second, we examine the
importance of matching the pre-training language to the target language as
opposed to the amount of data used in pre-training. To this end, we compare
pre-trained models in English and Netherlands Dutch to models trained on
Flemish Dutch. Recent work [5] has shown that low-resource languages can
greatly benefit from higher-resource languages when they are more similar due
to positive transfer, but cross-lingual representation learning degrades the
performance on high-resource languages due to interference. Furthermore, self-
supervised pre-training has shown to improve robustness and reduce the
degradation on out-of-domain data [29, 30]. We show that simply augmenting the
finetuning data leads to a strong speech recognition improvement in noisy and
reverberated environments. Finally, we investigate ASR improvements with the
recent wav2vec 2.0 model [1, 5] and study several pre-training and finetuning
scenario’s with an increasing amount of data, yielding substantial reduction
of Word Error Rates (WER) compared to the baselines.

We evaluate the models in terms of linear phone separability by reporting the
classification accuracy of an external linear classifier. The classifier is
trained to predict Flemish Dutch phones from the features extracted from each
model. For the ASR experiments, we report the results of an HMM-DNN hybrid
model [31] where the DNN is trained with the learned features.

##  2 Models

The procedure consists of three separate phases: 1) pre-training a model on
data without labels, 2) optionally finetuning the model on a labelled set with
transcripts, 3) extracting the learned features to perform a downstream
evaluation task.

###  2.1 Self-Supervised Pre-training

We start with a short description of the investigated pre-training techniques
and refer to the corresponding papers for more details. Table 1 gives an
overview of all models.

Table 1: Shallow overview and comparison of all pre-training techniques. Model |  Feature encoder |  Aggregator |  Objective |  Output dimension |  # Parameters
---|---|---|---|---|---
APC |  Filterbank |  GRU |  Reconstruct future frame |  512 |  4.1M
Mockingjay |  Filterbank |  Bidirectional Transformer |  Reconstruct masked frame |  768 |  21.3M
CPC |  CNN |  LSTM |  Identify future feature |  256 |  1.8M
wav2vec |  CNN |  CNN |  Identify future feature |  512 |  32.5M
wav2vec 2.0 |  CNN |  Transformer |  Identify quantised future feature |  768 (base), 1024 (large) |  95.0M (base), 317.3M (large)

####  2.1.1 APC

In APC, autoregressive models encode the temporal information in the past
sequence of frames, for example with Gated Recurrent Units (GRU). A future
frame, $n$ steps ahead of the current frame, is linearly predicted from the
autoregressive outputs. The model is then trained with an L1 reconstruction
loss on the predicted frame. We use a model with 3 GRU layers and predict 5
steps ahead [8, 10]. The outputs of the last GRU layer are extracted as
features for the downstream task.

####  2.1.2 Mockingjay

While APC conditions its prediction on past context only, Mockingjay leverages
both past and future context to predict a frame that has been masked out. The
encoder is a deep bidirectional Transformer [32] that learns contextualised
representations, which are extracted from the last layer. These
representations are linearly mapped to predict the masked frames, and the
model is trained with a reconstruction loss between the predicted and true
frames. We use the base model with 3 Transformer blocks in the encoder [12,
33].

####  2.1.3 CPC

CPC directly applies a stack of strided convolutional layers to the raw
waveform to encode the sequence in the latent space. An autoregressive model
(the aggregator) then looks at the representations of the past sequence and
its output is mapped to predict the latent representations for several steps
in the future. The loss is not reconstructive, but contrastive: given the
aggregator output, the model has to distinguish the correct sample out of a
bunch with distractors from windows more distant in time or from different
sequences. We use the modified CPC approach [6] where the encoder exists of 5
CNN layers, the autoregressive model is an LSTM and the prediction network is
a 1-layer Transformer network. The model predicts 1 to 12 steps in the future,
with a separate projection layer for every step, and is trained with 10
distractors. The outputs of the autoregressive model are the extracted
features.

####  2.1.4 wav2vec

Wav2vec is built on CPC but uses a fully convolutional model. The
autoregressive model is replaced by a context network consisting of 12
convolutional layers. Two additional linear transformations increase the
capacity of the encoder (this architecture is called wav2vec large in the
corresponding paper [16]). The outputs of the context network are the feature
vectors [34].

####  2.1.5 wav2vec 2.0

Wav2vec 2.0 combines ideas from wav2vec [16], vq-wav2vec [17] and MLM. The
encoder computes latent speech representations from the raw waveform with 7
temporal convolution blocks. A certain proportion of the latent features is
masked before feeding to the aggregator, which is a Transformer network. At
the same time, a quantisation module maps the latent feature vectors to
discretised versions. The final training objective is then to distinguish the
true quantised representation for a masked time step, given the aggregator
output [1]. We differentiate between the base and large architecture of the
model, which contain respectively 12 and 24 Transformer blocks in the
aggregator. The contextual features at the output of the aggregator are
extracted for downstream tasks [34]. We duplicate them in time to mimic a
stride of 10ms instead of 20ms.

The wav2vec 2.0 model can be finetuned on a labelled set. To this end, an
extra linear layer is added on top of the context network and a CTC loss is
applied with the transcription characters as targets. The encoder is frozen
during finetuning. Finetuning is done after the pre-training is completed.

Finally, XLSR-53 is a large wav2vec 2.0 model pre-trained on 53 languages
simultaneously [5]. The authors have shown that the quantised speech
representations can express connections between languages when trained in a
multilingual setup.

Due to limited resources, we pre-train wav2vec 2.0 base models for 100k
updates and finetune for 500k updates, and we don’t pre-train our own wav2vec
2.0 large models but only finetune existing pre-trained models.

###  2.2 Downstream Feature Evaluation

####  2.2.1 Phone Classification

We train an external phone classifier consisting of just one linear layer and
a softmax layer [33], with as input the features extracted from the pre-
trained models. All pre-trained features are compared to the baseline of
80-dimensional log-mel filterbank features, including second order delta
features and mean-variance normalisation. For every utterance, there is a
phone label every 10ms, corresponding to the stride of the input features. The
classifier is trained with a cross-entropy loss. We report the accuracy of the
classifier of predicting the correct phone label for every 10ms window,
instead of using the most voted phone during its entire duration, because the
learned representations should contain phonetic information even at the start
of a phone.

For English experiments we use the phone labels from [15], which have been
generated by forced alignment with Kaldi [31] using pre-trained models on
LibriSpeech, and mapped to 41 classes. For Flemish experiments we use the
phone labels provided in the Corpus Gesproken Nederlands (Section 3.1.1). The
phone sequences have been computed by forced alignment on the manually checked
orthographic transcripts with SPRAAK [35], and have been partly manually
checked as well. There are 49 distinct phone classes [36].

####  2.2.2 ASR

We train a baseline HMM-DNN model with Kaldi [31] on MFCC features. The HMM-
GMM models triphones and includes LDA, MLLT and fMLLR transformations. It is
trained on MFCC features to compute alignments and build a phonetic tree with
one state per phone. For the pre-trained models, we reuse the alignments and
tree from the MFCC model and only train the DNN model with the extracted
features as input. We make a distinction between a large DNN model containing
14 TDNN-F layers [37] (similar to the Switchboard recipe) and a small DNN
model with only 3 TDNN-F layers. We leave out iVector extraction and speed
perturbation, and remove the delta layers for pre-trained features. We decode
with a pruned trigram language model and use a lexicon of 100k words. We
report Word Error Rates based on the Levenshtein distance, but make a
correction for inconsistencies in compounding (which occur frequently in
Dutch).

##  3 Data

###  3.1 Flemish Dutch datasets

####  3.1.1 Labelled data

Corpus Gesproken Nederlands (CGN) [36] \- also called Spoken Dutch Corpus - is
a manually annotated speech database of around 900 hours of Dutch, of which
270 hours correspond to Flemish Dutch. CGN contains both phonetical and word-
level transcriptions and segmentations. The labelled data can be used for
finetuning, for ASR model training and for the proposed evaluation procedures.
We make the distinction between three training sets of data, based on the type
of speech.

VL-train-clean This set contains 35h of prepared, read speech by professional
readers. This corresponds to component O of CGN.

VL-train-other This set contains several types of speech, including read
speech (VL-train-clean), news reports, interviews, lectures, sports
commentary, etc. This set holds 145h of data from components
B,F,G,H,I,J,K,L,M,N,O of CGN.

VL-train-all This set contains all components from the CGN database and
corresponds to 270 hours of speech. The difference with VL-train-other is the
inclusion of narrowband telephone speech (8kHz resampled to 16kHz) and
spontaneous conversational speech, which correspond to respectively components
C,D and component A of CGN.

In a similar way, we make a distinction between VL-test-clean (4h) and VL-
test-other (15h, including the 4h from VL-test-clean). There is no overlap in
speakers with the train sets.

For phone classification experiments in English, we use the train-clean-100
set of LibriSpeech [38] and use the train-test split and phone labels from
[15].

####  3.1.2 Unlabelled data

We have created a dataset of 450h of unlabelled data for unsupervised
experiments in Flemish Dutch, by extracting audio from online available
resources. We refer to this set as VL-unsup. This set consists of 200h of data
from recordings in the Flemish parliament, 100h of audio from broadcast TV
news and 150h of audio from TV talkshows. For pre-training, we use this set
and the labelled sets without the transcriptions.

###  3.2 Pre-trained models

For some experiments, we use off-the-shelf available pre-trained models for
APC, Mockingjay, CPC, wav2vec and wav2vec 2.0 [33, 34]. These models have been
pre-trained on English audiobooks from LibriSpeech (LS-960) [38], LibriLight
(LL-60k) [39] or both, i.e. LibriVox (LV-60k) [40]. The XLSR-53 model is
trained on 56k hours of data from 53 different languages. The XLSR data
originates from CommonVoice [41], Multilingual LibriSpeech [40], and BABEL
[42]. It includes around 1.6k hours of Dutch [5] of which we recon only a very
small part is Flemish Dutch (a few hours in CommonVoice). We also use a
wav2vec 2.0 model pre-trained on the Dutch part of VoxPopuli (VP-NL-4.5k)
[43], which contains 4.5k hours of Netherlands Dutch speech recordings from
the European parliament.

##  4 Discussion

###  4.1 Phone classification

####  4.1.1 Applicability of off-the-shelf models to Flemish Dutch

First, we perform phone classification as explained in Section 2.2.1. For
experiments in English, we train and test the classifier on a train-test split
of LibriSpeech train-clean-100. For experiments in Flemish, we either train a
classifier on VL-train-clean and test on VL-test-clean, or train on VL-train-
other and evaluate on VL-test-other. Table 2 shows the phone classification
accuracies with features extracted from English pre-trained models that are
online available (see Section 3.2).

Table 2: Linear phone classification accuracy (%) with features extracted from off-the-shelf models pre-trained on English. We evaluate classification on English and Flemish. Model | English | Flemish
---|---|---
LS-tc100 | VL-test-clean | VL-test-other
Baseline | 48.0 | 48.5 | 39.3
APC | 72.7 | 71.4 | 60.1
Mockingjay | 68.1 | 71.4 | 59.1
CPC | 71.3 | 71.7 | 60.5
wav2vec | 78.4 | 73.3 | 62.4
wav2vec 2.0 (base) | 75.1 | 71.7 | 58.8

The relative improvements with respect to the baseline as a result of pre-
training are consistent across both languages. The accuracy on VL-test-clean
is of a similar magnitude as the accuracy on LS-tc100, which can be explained
by the fact that both sets contain rather easy, clean speech. On VL-test-clean
and VL-test-other, we see absolute accuracy improvements of more than 20%.
This shows that the pre-training techniques improve linear phone separability,
even when the target language differs from the pre-training language.

####  4.1.2 Language Matching

Second, we examine the effect of matching the domain (i.e. the language, but
also the type of speech) of the pre-training speech to the target speech. We
pre-train models on Flemish Dutch data, compare them to other pre-trained
models, and investigate the effect of finetuning wav2vec 2.0 on a Flemish
subset. Table 3 reports phone classification accuracies for pre-training and
finetuning on several datasets.

Table 3: Phone classification accuracy (PCA) percentage when training a classifier on VL-train-clean and testing on VL-test-clean (’clean’), and when training a classifier on VL-train-other and testing on VL-test-other (’other’). Model |  Pre-training |  Finetuning | PCA
---|---|---|---
clean |  other
Baseline (Filterbank) |  – |  – |  48.5 |  39.3
APC |  LS-960 |  – |  71.4 |  60.1
VL-train-clean |  – |  66.9 |  54.8
VL-train-other |  – |  67.6 |  57.1
VL-unsup |  – |  73.3 |  63.3
VL-train-all + VL-unsup |  – |  73.3 |  63.0
Mockingjay |  LS-960 |  – |  71.4 |  59.1
VL-train-clean |  – |  65.2 |  53.5
CPC |  LL-60k |  – |  71.7 |  60.5
VL-train-clean |  – |  72.6 |  55.6
VL-train-other |  – |  69.3 |  59.5
VL-unsup |  – |  66.8 |  57.4
|  VL-train-all + VL-unsup |  – |  67.5 |  58.1
wav2vec |  LS-960 |  – |  73.3 |  62.4
wav2vec 2.0 base |  LS-960 |  – |  71.7 |  58.8
VP-NL-4.5k |  – |  64.5 |  50.0
VL-train-other |  – |  47.4 |  36.1
VL-train-all + VL-unsup |  – |  54.7 |  43.9
LS-960 |  VL-train-other |  83.6 |  76.2
|  VP-NL-4.5k |  VL-train-other |  83.6 |  76.1
|  VL-train-other |  VL-train-other |  81.3 |  74.1
|  VL-train-all + VL-unsup |  VL-train-other |  82.2 |  75.0
wav2vec 2.0 large |  LS-960 |  – |  55.9 |  45.2
LV-60k |  – |  24.6 |  14.3
XLSR-53 |  – |  34.4 |  21.8
VP-NL-4.5k |  – |  58.2 |  45.7
LS-960 |  VL-train-other |  81.2 |  73.4
LV-60k |  VL-train-other |  85.0 |  76.6
XLSR-53 |  VL-train-other |  86.4 |  79.1
|  VP-NL-4.5k |  VL-train-other |  84.12 |  76.3

For APC, we notice an improvement over the English pre-trained model when we
match the training and target language and use a sufficient amount of data
(but still less than LibriSpeech). For CPC, we experienced converging
difficulties and a high sensitivity to the number of training cases. We see an
improvement over the pre-trained model on VL-test-clean when only training on
VL-train-clean. This might suggest that domain matching is important for CPC.
The LibriLight pre-trained model is trained on much more data (60k hours),
which can explain the strong performance on VL-test-other.

For wav2vec 2.0, the base models trained on Flemish data and the pre-trained
model on VoxPopuli perform worse than the LibriSpeech model, despite matching
language (VL) or using more data in a related language (VP). The former is
most likely explained by sub-optimal training, the latter could be explained
by the fact that the VoxPopuli parliament recordings reflect different
acoustic conditions to certain components with clean speech in CGN. Finetuning
on Flemish leads to very high phone classification accuracies for all models.

For the large high-capacity wav2vec 2.0 models, we note low accuracies without
finetuning. Other works corroborate this finding and ascribe it to the
problem-agnostic pre-training, and have shown that certain audio features are
more easily accessible from the middle layers in very deep transformer models
than the output layer [4, 30]. Finetuning (with graphemic transcripts)
alleviates this discrepancy and gives an improved accuracy over the base
models with the large models. It seems that the large model also benefits more
from a large amount of pre-training data, as the LibriVox and XLSR model show.

The XLSR-53 model reports the highest score. It is trained on a similar amount
of data as the LibriVox model, but the training data contains Dutch, German
and English. We also note that the (Flemish) Dutch stops, which are prevoiced
voiced stops, differ from the aspirated voiceless stops in English. This is
also better covered in the XLSR-53 set.

###  4.2 ASR Results

####  4.2.1 Clean ASR

Table 4 reports WER results on VL-test-other of HMM-DNN ASR experiments with a
large DNN and features from different models, as described in Section 2.2.2.
Every ASR model (including the baseline model) is trained on VL-train-other.

Table 4: ASR experiments with large DNN ASR model, reporting WER on VL-test-other. Model |  Pre-training |  Finetuning |  WER
---|---|---|---
Baseline (MFCC) |  – |  – |  15.10
APC |  LS-960 |  – |  16.02
VL-train-all + VL-unsup |  – |  16.20
CPC |  LS-960 |  – |  15.03
wav2vec |  LS-960 |  – |  14.89
wav2vec 2.0 base |  LS-960 |  – |  13.84
VL-train-other |  – |  14.44
VL-train-all + VL-unsup |  – |  13.52
|  LS-960 |  VL-train-other |  11.42
|  VL-train-other |  VL-train-other |  13.41
|  VL-train-all + VL-unsup |  VL-train-other |  11.76
wav2vec 2.0 large |  LS-960 |  – |  14.33
LV-60k |  – |  14.72
VP-NL-4.5k |  – |  16.32
XLSR-53 |  – |  13.40
LS-960 |  VL-train-other |  10.87
LV-60k |  VL-train-other |  12.65
|  XLSR-53 |  VL-train-other |  10.61

The improvements compared to the baseline with MFCC are small, if any, except
for wav2vec 2.0. In contrast to the phone classification experiments, we see
an improvement over the LibriSpeech model (960h) when we pre-train the base
model on a comparable amount of Flemish (720h) without finetuning, which
supports the idea that the combination of a matching pre-training language and
a large amount of data is key.

The poor performance of the large VoxPopuli model can possibly be explained by
a poor transfer from Netherlands Dutch to Flemish and different acoustic
conditions compared to the test set. Also, contrarily to phone classification,
the large wav2vec 2.0 model pre-trained on LibriSpeech outperforms the model
pre-trained on LibriVox.

Analogous to the phone classification experiments, the XLSR-53 model - which
has a large variability in its training data - yields a significant WER
improvement, manifesting a positive cross-lingual transfer to Flemish, and the
best results are obtained when finetuning a model on an annotated Flemish
subset. We obtain almost 30% relative WER improvement when finetuning XLSR-53
on Flemish, compared to the baseline. The difference with the LibriSpeech
model is however small. We postulate that a large wav2vec 2.0 model trained on
more Flemish data would equal or improve this result.

####  4.2.2 Effect of amount of pre-training and finetuning data

We quantitatively examine the effect of an increasing amount of data used for
pre-training wav2vec 2.0 base models (unlabelled) or finetuning XLSR-53
(labelled). We shuffle all available data from all sets. We also evaluate
finetuning on different types of speech (from different sets). For the
unlabelled dataset, this distinction is not trivial. Table 5 shows the
results. The ASR model is always trained on VL-train-other.

Table 5: WER with DNN ASR model in function of the amount of Flemish data used
for pre-training or finetuning.

10h |  30h |  50h |  100h |  150h |  250h |  350h |  500h |  700h
---|---|---|---|---|---|---|---|---
31.87 |  20.85 |  16.76 |  15.55 |  15.74 |  14.76 |  14.34 |  14.73 |  13.52
(a) Unlabelled data for pre-training a base wav2vec 2.0 model (no finetuning),
large ASR DNN.

0h |  1h |  10h |  20h |  30h |  50h |  90h |  150h |  250h
---|---|---|---|---|---|---|---|---
27.75 |  13.84 |  12.08 |  11.32 |  11.19 |  10.71 |  10.61 |  10.53 |  10.50
(b) Labelled data for finetuning XLSR-53, small ASR DNN.

VL set |  No FT (0h) |  Clean (29h) |  Other (128h) |  All (248h)
---|---|---|---|---
WER |  13.40 |  12.35 |  10.61 |  10.58
(c) Different sets of CGN for finetuning XLSR-53, large ASR DNN.

For pre-training, it is necessary to have a considerable amount of data to
improve upon the baseline, and more Flemish data gives improvements. It seems
that the learned audio representations include acoustic details aside from
more abstract phoneme qualities, as the WER on 150h of shuffled data is higher
than when pre-training on an equal amount of matched data (VL-train-other in
Table 4). This might suggest a high dependency on acoustic conditions. For
finetuning, the data should match the type or conditions of the test set for
optimal results, and the improvements saturate with more data. Note that a
small DNN suffices after finetuning, but a large DNN is required when the XLSR
model is not finetuned (first column of Table 5(b)). This is in line with the
poor phone classification results of the large models without finetuning.

####  4.2.3 ASR in noisy environments

We investigate the robustness of wav2vec 2.0 to noisy and reverberated speech
by replicating the VL-test-other set in 4 different scenario’s: filtered with
RIRs (rev), with added noise at a certain SNR (noise1: 5-20dB, noise2: 0-15dB)
and both (rev \+ noise3: 5-15dB). We use noises from multiple sources (NTT
Noise-DB, CHIME2, NoiseX, DEMAND, Humming) and RIRs from the Aachen Impulse
Response Database. We compare wav2vec 2.0 large models pre-trained on
LibriVox: without finetuning, finetuned on VL-train-other (’clean’) and
finetuned on a fourfold augmented VL-train-other (’aug’) by adding noise and
reverberation in Table 6.

Table 6: WER with large DNN ASR on augmented VL-test-other with LibriVox pre-trained large wav2vec 2.0 models. Model |  FT | WER
---|---|---
clean |  rev |  noise1 |  noise2 |  rev + noise3
MFCC |  – |  15.10 |  28.36 |  20.58 |  26.26 |  39.21
w2v2 |  – |  14.71 |  27.12 |  19.96 |  25.28 |  39.19
w2v2 |  clean |  12.43 |  22.60 |  16.31 |  20.61 |  33.32
w2v2 |  aug |  12.13 |  18.08 |  14.64 |  17.39 |  24.43

Finetuning on augmented data gives strong improvements over finetuning on
clean data in the reverberated and noisy settings, with 3-9% absolute WER
reduction. More so, there is even a slight improvement in the clean setting as
well, probably because of more finetuning data.

##  5 Conclusion

Pre-trained features on English speech transfer well to Flemish Dutch in terms
of improving linear phone separability. These self-supervised pre-trained
models are readily available and easy to use. Matching the pre-training and
target language further improves results, but either matching the type of
speech or using a larger amount of data is necessary. The recently proposed
wav2vec 2.0 model appears superior, especially when finetuned on data from the
target language. Finally, we obtain the best results with the large
multilingually trained XLSR-53 model and see nearly 30% improvement in WER by
finetuning the XLSR-53 model on Flemish, compared to the baseline. We show the
importance of matching pre-training and target language and acoustic
conditions.

##  6 Acknowledgements

This research received funding from the Flemish Government under the
”Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

## References

  * [1] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
  * [2] Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli,  “Self-training and pre-training are complementary for speech recognition,” 2020.
  * [3] Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu,  “Pushing the limits of semi-supervised learning for automatic speech recognition,” 2020.
  * [4] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  “Unsupervised speech recognition,” 2021.
  * [5] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli,  “Unsupervised cross-lingual representation learning for speech recognition,” 2020.
  * [6] Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux,  “Unsupervised pretraining transfers well across languages,” 2020,  Code: https://github.com/facebookresearch/CPC_audio.
  * [7] Yu-An Chung, Yonatan Belinkov, and James Glass,  “Similarity analysis of self-supervised speech representations,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 3040–3044.
  * [8] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass,  “An unsupervised autoregressive model for speech representation learning,” 2019.
  * [9] Yu-An Chung and James Glass,  “Generative pre-training for speech with autoregressive predictive coding,” 2020.
  * [10] Yu-An Chung, Hao Tang, and James Glass,  “Vector-quantized autoregressive predictive coding,” 2020,  Code: https://github.com/iamyuanchung/VQ-APC.
  * [11] Yu-An Chung and James Glass,  “Improved speech representations with multi-target autoregressive predictive coding,” 2020.
  * [12] Andy T. Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020.
  * [13] Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Shang-Wen Li, and Hung yi Lee,  “Audio ALBERT: A lite BERT for self-supervised learning of audio representation,” 2021.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  * [15] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,” 2019.
  * [16] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,” 2019.
  * [17] Alexei Baevski, Steffen Schneider, and Michael Auli,  “vq-wav2vec: Self-supervised learning of discrete speech representations,” 2020.
  * [18] Alexei Baevski, Michael Auli, and Abdelrahman Mohamed,  “Effectiveness of self-supervised pre-training for speech recognition,” 2020.
  * [19] Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron Van den Oord,  “Learning robust and multilingual speech representations,” 2020.
  * [20] Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aaron Van den Oord,  “Unsupervised speech representation learning using wavenet autoencoders,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 12, pp. 2041–2053, Dec 2019.
  * [21] Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Ricard Marxer, and James Glass,  “A convolutional deep markov model for unsupervised speech representation learning,” 2020.
  * [22] Andy T. Liu, Shang-Wen Li, and Hung yi Lee,  “TERA: Self-supervised learning of transformer encoder representation for speech,” 2020.
  * [23] Alexander H. Liu, Yu-An Chung, and James Glass,  “Non-autoregressive predictive coding for learning speech representations from local dependencies,” 2020.
  * [24] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio,  “Multi-task self-supervised learning for robust speech recognition,” 2020.
  * [25] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio,  “Learning problem-agnostic speech representations from multiple self-supervised tasks,” 2019.
  * [26] Dongwei Jiang, Wubo Li, Miao Cao, Ruixiong Zhang, Wei Zou, Kun Han, and Xiangang Li,  “Speech SIMCLR: Combining contrastive and reconstruction objective for self-supervised speech representation learning,” 2020.
  * [27] Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff,  “Deep contextualized acoustic representations for semi-supervised speech recognition,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020.
  * [28] Hans Van de Velde, M. Kissine, E. Tops, S. van der Harst, and R. van Hout,  “Will Dutch become Flemish? Autonomous developments in Belgian Dutch,”  in Multilingua 29, 2010.
  * [29] Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli,  “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,”  2021\.
  * [30] Danni Ma, Neville Ryant, and Mark Liberman,  “Probing acoustic representations for phonetic properties,” 2021.
  * [31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely,  “The Kaldi speech recognition toolkit,”  in IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. Dec. 2011, IEEE Signal Processing Society,  IEEE Catalog No.: CFP11SRW-USB.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,” 2017.
  * [33] Andy T. Liu and Yang Shu-wen,  “S3PRL: The self-supervised speech pre-training and representation learning toolkit,” 2020.
  * [34] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  * [35] Kris Demuynck, Tom Laureys, and Steven Gillis,  “Automatic generation of phonetic transcriptions for large speech corpora,”  in Proceedings International Conference on Spoken Language Processing, 2002, vol. 1, pp. 333–336.
  * [36] Nelleke Oostdijk,  “The Spoken Dutch Corpus: Overview and first evaluation,”  Proceedings of LREC-2000, Athens, vol. 2, 01 2000.
  * [37] Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur,  “Semi-orthogonal low-rank matrix factorization for deep neural networks,”  in Proc. Interspeech 2018, 2018, pp. 3743–3747.
  * [38] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [39] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux,  “Libri-Light: A benchmark for ASR with limited or no supervision,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7669–7673,  https://github.com/facebookresearch/libri-light.
  * [40] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert,  “MLS: A large-scale multilingual dataset for speech research,”  Interspeech 2020, Oct 2020.
  * [41] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber,  “Common Voice: A massively-multilingual speech corpus,” 2020.
  * [42] Mark J. F. Gales, Kate M. Knill, Anton Ragni, and Shakti P. Rath,  “Speech recognition and keyword spotting for low-resource languages: Babel project research at cued.,”  in SLTU, 2014, pp. 16–23.
  * [43] Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux,  “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” 2021,  Code: https://github.com/facebookresearch/voxpopuli.