# A note on $\Gamma$-convergence of Tikhonov functionals
for nonlinear inverse problems

Alexey Belenkin
Department of Numerical Mathematics
Saarland University
Saarbrücken  Germany
<EMAIL_ADDRESS>
&Michael Hartz
Department of Mathematics
Saarland University
Saarbrücken  Germany
<EMAIL_ADDRESS>
&Thomas Schuster
Department of Numerical Mathematics
Saarland University
Saarbrücken  Germany
<EMAIL_ADDRESS>

###### Abstract

We consider variational regularization of nonlinear inverse problems in Banach
spaces using Tikhonov functionals. This article addresses the problem of
$\Gamma$-convergence of a family of Tikhonov functionals and assertions of the
convergence of their respective infima. Such questions arise, if model
uncertainties, inaccurate forward operators, finite dimensional approximations
of the forward solutions and / or data, etc. make the evaluation of the
original functional impossible and, thus, its minimizer not computable. But
for applications it is of utmost importance that the minimizer of the
replacement functional approximates the original minimizer. Under certain
additional conditions this is satisfied if the approximated functionals
converge to the original functional in the sense of $\Gamma$-convergence. We
deduce simple criteria in different topologies which guarantee
$\Gamma$-convergence as well as convergence of minimizing sequences.

_K_ eywords Gamma-convergence $\cdot$ convergence of minimizing sequences
$\cdot$ variational regularization $\cdot$ Tikhonov functional $\cdot$ equi-
mild coercivity

MSC 2010: 58E50, 65J22

##  1 Introduction

Nonlinear inverse problems arise in a large variety of applications ranging
from medicine, natural science and engineering. Such problems are usually
modeled as operator equations

$F(x)=y$ (1.1)

for a continuous, nonlinear mapping $F:\operatorname{dom}(F)\subseteq X\to Y$,
where $\operatorname{dom}(F)$ denotes the domain of $F$ and $X$, $Y$ are
Banach spaces. Problems like (1.1) usually are ill-posed in the sense that
they are not continuously invertible. The stable solution of such equations by
regularization methods has been widely studied over the last decades and
published in textbooks such as [9, 12, 13, 18, 20]. An important class of
regularization methods relies on the minimization of _Tikhonov functionals_

$T(x):=\frac{1}{p}\left\|{F(x)-y^{\delta}}\right\|_{Y}^{p}+\alpha\,\Omega(x)$
(1.2)

in $\operatorname{dom}(F)$, which is well-known as _variational
regularization_. Here $y^{\delta}\in Y$ denotes noise-contaminated data with
noise level $\delta>0$, i.e., $\left\|{y^{\delta}-y}\right\|_{Y}\leq\delta$,
$\Omega:\operatorname{dom}(F)\to\mathbb{R}_{+}$ is a functional acting as
penalty term and $\alpha>0$ serves as regularization parameter and balances
the weighting of the data fitting term
$\left\|{F(x)-y^{\delta}}\right\|_{Y}^{p}$ and the penalty term $\Omega(x)$.
The existence of minimizers, the stability and regularization property of $T$
and different parameter choice rules for $\alpha$ depending on $\delta$ and /
or $y^{\delta}$ have been studied in various settings and are very well
understood.

In practical applications and implementations of variational regularization
methods however $T(x)$ as well as its minimizer(s) are barely accessible,
since approximations in finite dimensional subspaces of $X$, $Y$,
respectively, e.g. by discretization, are necessary or the forward problem
$F(x)$ is too expensive or just impossible to be evaluated exactly. This is,
e.g., the case if only a viscosity or weak solution of a corresponding PDE are
well defined rather than a classical one (see, e.g., [21] for an application
in tensor tomography). A further problem of computing $T(x)$ or $F(x)$ exactly
are model uncertainties that have to be taken into account (see [18]). That
means, instead of computing a minimizer of $T$ in applications one rather
computes a minimizer of

$T_{\delta}(x):=\frac{1}{p}\left\|{F_{\delta}(x)-y^{\delta}}\right\|_{Y}^{p}+\alpha(\delta)\,\Omega(x)$
(1.3)

for some $\delta>0$ including noise as well as model inaccuracies
$F_{\delta}$, which are the result of processes mentioned above. In some
situations it might be more convenient to write $T_{n}$ instead of
$T_{\delta}$, e.g. to model a discrete approximation process. Standard theory
in inverse problems tries to find parameter choice rules $\alpha(\delta)$ such
that a minimizer $x_{\delta}$ of $T_{\delta}$, if existent, converges to a
solution $x^{\dagger}$ of (1.1). Articles such as [16, 17] investigate the
combination of regularization and discretization. In general the analysis for
such convergence results demands for specific asymptotic behaviors of the
noise level, the model or discretization error, or assumptions on the
nonlinearity of the problem such as the tangential cone condition, all of
which are difficult or impossible to check for specific applications. In this
article we rather address the question under which conditions we can guarantee
that a minimizer $x_{\delta}$ is _close_ to a minimizer of $T$ in the sense
that

$\min_{x\in\operatorname{dom}(F)}T(x)=\lim_{\delta\to
0}\inf_{x\in\operatorname{dom}(F)}T_{\delta}(x)\;,$ (1.4)

where we replace $\delta\to 0$ by $n\to\infty$ if we want to address a
discrete process. Such assertions can be achieved by proving that
$T_{\delta}\to T$ as $\delta\to 0$ ($T_{n}\to T$ as $n\to\infty$) in the sense
of _$\mathit{\Gamma}$ -convergence_. So we are more interested in the
question, whether $x_{\delta}$ is a good replacement for a (potentially) not
accessible minimizer of $T$ rather than convergence to an exact solution of
(1.1), a question which is of large interest from a practical point of view.
For this purpose $\Gamma$-convergence seems to be superior to pointwise or
uniform convergence of $T_{\delta}$ to $T$.

The concept of $\Gamma$-convergence was introduced by Ennio De Giorgi in the
1970s and can be seen as a generalization of G-convergence, a notion of
convergence for Green functions. $\Gamma$-convergence is often used for
homogenization problems, dimension reduction, and transitions from discrete to
continuous models because it helps to capture the relevant behavior of
solutions of a family of minimum problems

$\min\left\\{T_{\varepsilon}(u);\,u\in X_{\varepsilon}\right\\}$

in a problem

$\min\left\\{T(u);\,u\in X\right\\}$

that does not depend on the parameter $\varepsilon$ and typically presents a
limiting case for the parameter $\varepsilon$. Standard references on
$\Gamma$-convergence are the textbooks [5, 8]. The following properties of
$\Gamma$-convergence make this idea especially intriguing (see [5]):

  * •

the limit functional is always lower semicontinuous;

  * •

the convergence is stable under additive continuous perturbations;

  * •

under some suitable conditions on the the family
$\left\\{T_{\varepsilon}\right\\}$ like, e.g., equi-coercivity, also the
convergence of their minimum problems is guaranteed, that is

    * –

the limit functional has a minimum point,

    * –

the infima of $\left\\{T_{\varepsilon}\right\\}$ converge to the minimum value
of $T$, and

    * –

the minimizers of $\left\\{T_{\varepsilon}\right\\}$ converge (up to
subsequences) to a minimizer of $T$.

The last point can be understood as an extension of the following result from
calculus of variations to families of functionals: A coercive and lower
semicontinuous functional attains its infimum and has a minimum point. A very
prominent example, where $\Gamma$-convergence arises in image processing, is
the approximation of the Mumford-Shah-functional by the Ambrosio-Tortorelli-
functional, see [2]. In this article a fairly general setting is considered,
existing results are more specific. For weak-to-weak sequentially continuous,
exact forward operators similar results were achieved in [20], but not in the
setting of $\Gamma$-convergence. In [6] $\Gamma$-convergence with respect to
the weak∗ topology was mentioned for exact bounded linear operators. The
outcome in this work goes beyond these results.

#### Outlook and Main Results

In Section 2, we give a short introduction to the concept of
$\Gamma$-convergence. This includes convergence results for minima and
minimizers of a sequence of functionals, which are presented in [5] and [8].
We also recall a few basics from the theory of Banach spaces. In Section 3, we
define the functionals in question and propose conditions that will allow the
theory of $\Gamma$-convergence to be applied to those functionals. We work in
a Banach space setting and consider the norm, weak, and weak∗ topologies to
highlight the differences between the respective conditions in these
topologies. In Section 4 we present examples to demonstrate how the
theoretical results from the previous section apply to typical inverse
problems settings like integral equations of first kind and a parameter
identification problem for an elliptic boundary value problem. The main
achievements of this article are the presentation of criteria for a family of
Tikhonov functionals in a fairly general setting to satisfy
$\Gamma$-convergence to some limit functional as well as an infimal property
as (1.4).

##  2 Preliminaries

In this section, we give the necessary basics of the theory of
$\Gamma$-convergence and also list a few results from functional analysis and
topology about compactness in Banach spaces that will be needed later on.

###  2.1 Banach spaces

The Tikhonov functionals we study are defined on (subsets of) Banach spaces.
In this section, we collect some basic facts from functional analysis that we
require. For further background, we refer the reader to the books [7, 14, 19].

Every Banach space is naturally equipped with the norm topology. However, it
is known from variational minimization that compactness plays an important
role for minimization problems. Since norm compact sets are scarce in infinite
dimensions, we consider in addition to the norm topology two other topologies
on a Banach space $X$. These are the weak topology, and, provided that $X$ is
a dual space, the weak∗ topology.

First, we recall the definition of the weak topology.

######  Definition 2.1.

Let $X$ be a normed space over $\mathbb{K}=\mathbb{R}$ or
$\mathbb{K}=\mathbb{C}$. The _(continuous) dual space_ of $X$ is the space

$X^{*}=\left\\{x^{*}:X\to\mathbb{K};x^{*}\text{ is linear and
continuous}\right\\},$

equipped with the norm

$\left\|{x^{*}}\right\|=\sup\left\\{\left|{x^{*}(x)}\right|;x\in X\text{ with
}\left\|{x}\right\|\leq 1\right\\}.$

The _weak topology_ on $X$ is the coarsest topology on $X$ for which all
elements of $X^{*}$ are continuous.

The weak topology on $X$ can be understood as the locally convex topology on
$X$ induced by the semi-norms

$p_{x^{*}}(x)=\left|{x^{*}(x)}\right|\quad(x\in X),$

where $x^{*}$ ranges over all elements on $X^{*}$. As usual, topological
properties that hold with respect to the weak topology are said to hold
_weakly_. For more explanation of this construction, we refer the reader to
[7, Chapter 5] or [14, Chapter 2].

The weak topology is at most coarser than the norm topology, and hence may
admit more compact sets. This principle works best in reflexive spaces, whose
definition we now recall.

######  Definition 2.2.

Let $X$ be a normed space. The dual space $(X^{*})^{*}$ of $X^{*}$ is called
the _bidual_ of $X$ and is denoted by $X^{**}$. The map

$i:X\to X^{**},\quad i(x)(x^{*})=x^{*}(x)\quad(x\in X,x^{*}\in X^{*}),$

is called the _canonical embedding_ from $X$ into $X^{**}$. A normed space $X$
is _reflexive_ if the canonical embedding $i:X\to X^{**}$ is surjective.

Examples of reflexive spaces are $L^{p}$ spaces for $1<p<\infty$, see, for
instance, [14, Theorem 1.11.10].

The importance of reflexivity for compactness in the weak topology is
explained by the following theorem.

######  Theorem 2.3.

A normed space is reflexive if and only if its closed unit ball is compact in
the weak topology.

###### Proof.

Cf. [14, 2.8.2 Theorem, p.245]. ∎

We also mention in passing another important result about weak compactness,
namely the Eberlein–Šmulian theorem. It shows in particular that a subset of a
Banach space is weakly compact if and only if it is sequentially weakly
compact, see, for instance, [14, Theorem 2.8.6].

Theorem 2.3 shows that in non-reflexive spaces such as $L^{\infty}$ spaces,
the weak topology is less useful. When working with a dual space, this issue
can sometimes be circumvented by using the weak∗ topology, whose definition is
given as follows.

######  Definition 2.4.

Let $X$ be a normed space. The weak∗ topology on $X^{*}$ is the coarsest
topology on $X^{*}$ for which the evaluation maps

$X^{*}\to\mathbb{K},\quad x^{*}\mapsto x^{*}(x),$

are continuous for all $x\in X$.

The weak∗ topology on $X^{*}$ can be understood as the locally convex topology
on $X^{*}$ induced by the semi-norms

$p_{x}(x^{*})=\left|{x^{*}(x)}\right|\quad(x^{*}\in X^{*}),$

where $x$ ranges over all elements on $X$. For more explanation of this
construction, we once again refer to [7, Chapter 5] or [14, Chapter 2].

The crucial result about compactness with respect to the weak∗ topology is the
following result, which is a generalization of one implication in Theorem 2.3.

######  Theorem 2.5 (Banach–Alaoglu).

Let $X$ be a normed space. Then the closed unit ball of the dual space $X^{*}$
is compact in the weak∗ topology.

###### Proof.

Cf. [14, 2.6.18 Theorem, p.229]. ∎

The weak and weak∗ topologies are never metrizable in the context of infinite
dimensional spaces. However, if $X$ is separable, then the unit ball of
$X^{*}$ is metrizable in the weak∗ topology; see for example [7, Theorem
V.5.1]. Moreover, if $X$ is separable and reflexive, then the unit ball of $X$
is metrizable in the weak topology; see the discussion following [7, Theorem
V.5.1].

Finally, we mention the following basic consequence of the Hahn–Banach
theorem.

######  Theorem 2.6.

If $M$ is a convex subset of a normed space, then its norm closure coincides
with its weak closure. In particular, $M$ is weakly closed if and only if it
is norm closed.

###### Proof.

Cf. [14, 2.5.16 Theorem, p.216]. ∎

The analogue of Theorem 2.6 for the weak∗ topology is false if $X$ is not
reflexive. Indeed, if $X$ is not reflexive, then the kernel of any element of
$X^{**}\setminus i(X)$ is a norm closed subspace of $X^{*}$ that is not weak∗
closed (see [7, Theorem IV.3.1 and Theorem V.1.3]).

###  2.2 Semi-continuity and coercivity

In this subsection, we recall the notions of semi-continuity and coercivity of
functionals on topological spaces. These guarantee the existence of minima and
will be used throughout the paper.

In the following we denote the set of the extended real numbers by
$\overline{\mathbb{R}}:=\left[-\infty,+\infty\right]:=\mathbb{R}\cup\left\\{+\infty,-\infty\right\\}$.

######  Definition 2.7.

Let $X$ be a topological space, $x\in X$, and let $\mathcal{N}(x)$ denote the
set of all open neighborhoods of $x$ in $X$. A functional
$f:X\to\overline{\mathbb{R}}$ is said to be _lower semicontinuous at $x\in
X$_, if for every $t\in\mathbb{R}$, with $t<f(x)$ , there exists
$U\in\mathcal{N}(x)$ such that $t<f(y)$ for every $y\in U$. We say that $f$ is
_lower semicontinuous (l.s.c)_ on $X$ if $f$ is lower semicontinuous at each
point $x\in X$.

The notion of upper semicontinuity is obtained by replacing $<$ with $>$ in
the previous definition.

If $X$ is a metric space, then a function $f:X\to\mathbb{R}$ is lower
semicontinuous at $x\in X$ if and only if

$f(x)\leq\liminf_{j\to\infty}f\left(x_{j}\right)\;,$

for every sequence $\left(x_{j}\right)_{j\in\mathbb{N}}$ converging to $x$ in
$X$.

We will make use of the following well known result.

######  Lemma 2.8.

Let $X$ be a normed space.

  1. (a)

The norm on $X$ is weakly lower semicontinuous.

  2. (b)

The norm on $X^{*}$ is weak∗ lower semicontinuous.

###### Proof.

See, for instance, [14, Theorem 2.5.21] and [14, Theorem 2.6.14]. ∎

The following concept is crucial in the calculus of variations. Recall that a
subset $A$ of a topological space is countably compact if every countable open
cover of $A$ has a finite subcover. Clearly, every compact set is countably
compact.

######  Definition 2.9.

Let $X$ be topological space. We say that a functional
$f:X\to\overline{\mathbb{R}}$ is _coercive_ on $X$ if the closure of its
sublevel set $\left\\{f\leq t\right\\}:=\left\\{x\in X;f(x)\leq t\right\\}$ is
countably compact for every $t\in\mathbb{R}$. This is equivalent to the
existence of closed countably compact sets $K_{t}\subseteq X$ such that
$\left\\{f\leq t\right\\}\subseteq K_{t}$ for every $t\in\mathbb{R}$. A
functional $f:X\to\overline{\mathbb{R}}$ is _mildly coercive_ if there exists
a non-empty countably compact set $K\subseteq X$ such that $\inf_{x\in
X}f(x)=\inf_{x\in K}f(x)$.

######  Remark 2.10.

If $f:X\to\overline{\mathbb{R}}$ is coercive, then $f$ is also mildly
coercive. In fact, if $f\neq+\infty$, then there exists $t\in\mathbb{R}$ such
that $\left\\{f\leq t\right\\}$ is not empty, and we can take $K$ as the
closure of this set in $X$. In the case $f=+\infty$ we may take any countably
compact subset of $X$ as $K$. The converse is in general not true. An example
of a non-coercive, mildly coercive function is given by any periodic function
$f:\mathbb{R}^{n}\to\mathbb{R}$.

The following theorem is a well known result from the calculus of variations.

######  Theorem 2.11.

Let $X$ be a topological space. If $f:X\to\overline{\mathbb{R}}$ is coercive
and lower semicontinuous, then

  1. (a)

$f$ has a minimum point in $X$;

  2. (b)

if $\left(x_{j}\right)_{j\in\mathbb{N}}$ is a minimizing sequence of $f$ in
$X$ and $x$ is a cluster point of $\left(x_{j}\right)_{j\in\mathbb{N}}$, then
$x$ is a minimum point of $f$ in $X$;

  3. (c)

if $f$ is not identically $+\infty$, then every minimizing sequence for $f$
has a cluster point.

###### Proof.

Cf. [8, Theorem 1.15, p.13]. ∎

###  2.3 $\Gamma$-convergence

Here we give a short introduction to $\Gamma$-convergence following the book
by Dal Maso [8]. We also list some useful properties for applied problems.
Another reference is the book by Braides [5].

######  Definition 2.12.

Let $\left(X,\tau\right)$ be a topological space, $x\in X$, and
$\left(f_{j}\right)_{j\in\mathbb{N}}$ a sequence of functionals with
$f_{j}:X\to\overline{\mathbb{R}}$ for all $j\in\mathbb{N}$. The
_$\mathit{\Gamma}$ -lower limit_ and _$\mathit{\Gamma}$ -upper limit_ of the
sequence $\left(f_{j}\right)_{j\in\mathbb{N}}$ at $x$ are defined by

$\left(\Gamma\operatorname{-}\liminf_{j\to\infty}f_{j}\right)(x)=\sup_{U\in\mathcal{N}(x)}\liminf_{j\to\infty}\inf_{y\in
U}f_{j}(y)\;,$
$\left(\Gamma\operatorname{-}\limsup_{j\to\infty}f_{j}\right)(x)=\sup_{U\in\mathcal{N}(x)}\limsup_{j\to\infty}\inf_{y\in
U}f_{j}(y)\;,$

where $\mathcal{N}(x)$ is the set of all open neighborhoods of $x$ in $X$. If
there exists a functional $f_{\infty}:X\to\overline{\mathbb{R}}$ such that

$\left(\Gamma\operatorname{-}\liminf_{j\to\infty}f_{j}\right)(x)=\left(\Gamma\operatorname{-}\limsup_{j\to\infty}f_{j}\right)(x)=f_{\infty}(x)$

for all $x\in X$, then we write
$f_{\infty}=\Gamma\operatorname{-}\smash[b]{\lim\limits_{j\to\infty}}f_{j}$
and we say that the sequence $\left(f_{j}\right)_{j\in\mathbb{N}}$
_$\mathit{\Gamma}$ -converges_ to $f_{\infty}$ (in $X$) or that $f_{\infty}$
is the _$\mathit{\Gamma}$ -limit_ of $\left(f_{j}\right)_{j\in\mathbb{N}}$ (in
$X$).

######  Remark 2.13.

If $X$ is a metric space, then $\left(f_{j}\right)_{j\in\mathbb{N}}$
$\Gamma$-converges to $f_{\infty}:X\to\overline{\mathbb{R}}$ if and only if
the following conditions are satisfied:

  1. (a)

for every $x\in X$ and for every sequence
$\left(x_{j}\right)_{j\in\mathbb{N}}$ in $X$ converging to $x$, the inequality

$f_{\infty}(x)\leq\liminf_{j\to\infty}f_{j}\left(x_{j}\right)\;$

holds, and

  2. (b)

for every $x\in X$ there exists a sequence
$\left(x_{j}\right)_{j\in\mathbb{N}}$ in $X$ converging to $x$ such that

$f_{\infty}(x)\geq\limsup_{j\to\infty}f_{j}\left(x_{j}\right)\;.$

A proof of the equivalence can be found in [8, Proposition 8.1, p.86]. It
turns out that in our setting, the characterization in Definition 2.12 is
somewhat easier to deal with. In addition, using Definition 2.12 allows us to
deal with weak and weak∗ topologies, which are typically not metrizable.

######  Remark 2.14.

Actually, the above equivalence, as well as all other results from [8,
Proposition 8.1], also hold true in a more general class of spaces, called
sequential spaces. These are topological spaces whose topology is given by
sequentially open sets. This generalization can be easily proven by using the
fact that every sequential space is a quotient of some first-countable space,
which was shown in [11].

######  Remark 2.15.

The $\Gamma$-lower limit and the $\Gamma$-upper limit of a sequence of
functionals $\left(f_{j}\right)_{j\in\mathbb{N}}$ from a topological space $X$
into $\overline{\mathbb{R}}$ are both lower semicontinuous on $X$; see [8,
Proposition 6.8].

It is helpful to recall the relationship between $\Gamma$-convergence and
pointwise or uniform convergence.

######  Proposition 2.16.

Let $X$ be a topological space and let $\left(f_{j}\right)_{j\in\mathbb{N}}$
be a sequence of functionals from $X$ into $\overline{\mathbb{R}}$.

  1. (a)

The following inequalities hold:

$\Gamma\operatorname{-}\liminf_{j\to\infty}f_{j}\leq\liminf_{j\to\infty}f_{j}\;,\quad\Gamma\operatorname{-}\limsup_{j\to\infty}f_{j}\leq\limsup_{j\to\infty}f_{j}\;.$

In particular, if $\left(f_{j}\right)_{j\in\mathbb{N}}$
$\mathit{\Gamma}$-converges to $f_{\infty}$ and converges pointwise to $f$,
then $f_{\infty}\leq f$.

  2. (b)

If each $f_{j}$ is lower semicontinuous and
$\left(f_{j}\right)_{j\in\mathbb{N}}$ converges uniformly to $f$, then
$\left(f_{j}\right)_{j\in\mathbb{N}}$ $\Gamma$-converges to $f$.

###### Proof.

See, for instance, [8, Proposition 5.1] for part (a) and [8, Proposition 5.2,
Remark 5.3] for part (b). ∎

We will now give the central theorem that makes $\Gamma$-convergence useful
and important for applications. For this we need following definitions.

######  Definition 2.17.

Let $X$ be a topological space. We say that a sequence
$\left(f_{j}\right)_{j\in\mathbb{N}}$ of functionals from $X$ into
$\overline{\mathbb{R}}$ is _equi-coercive_ if for every $t\in\mathbb{R}$ there
exists a closed countably compact set $K_{t}\subseteq X$ of $X$ such that
$\left\\{f_{j}\leq t\right\\}\subseteq K_{t}$ for every $j\in\mathbb{N}$. The
sequence $\left(f_{j}\right)_{j\in\mathbb{N}}$ is _equi-mildly coercive_ if
there exists a non-empty countably compact set $K\subseteq X$ such that
$\inf_{x\in X}f_{j}(x)=\inf_{x\in K}f_{j}(x)$ for all $j\in\mathbb{N}$.

######  Definition 2.18.

Let $f:X\to\overline{\mathbb{R}}$ be a functional and let $\varepsilon>0$. An
_$\varepsilon$ -minimizer_ of $f$ in $X$ is a point $x\in X$ such that

$f(x)\leq\max\left\\{\inf_{y\in
X}f(y)+\varepsilon\,,\,-\frac{1}{\varepsilon}\right\\}\;.$

The following theorem can be seen as a generalization of Theorem 2.11 to
sequences of functionals.

######  Theorem 2.19.

Let $X$ be a topological space and let $\left(f_{j}\right)_{j\in\mathbb{N}}$
be a sequence of functionals from $X$ into $\overline{\mathbb{R}}$. Suppose
that $\left(f_{j}\right)_{j\in\mathbb{N}}$ is equi-mildly coercive and
$\mathit{\Gamma}$-converges to $f_{\infty}$.

  1. (a)

The functional $f_{\infty}$ attains its minimum on $X$ and

$\min_{x\in X}f_{\infty}(x)=\lim_{j\to\infty}\inf_{x\in X}f_{j}(x)\;.$

  2. (b)

Let $\left(x_{j}\right)_{j\in\mathbb{N}}$ be a sequence in $X$ such that
$x_{j}$ is an $\varepsilon_{j}$-minimizer of $f_{j}$ in $X$ for every
$j\in\mathbb{N}$, where $\left(\varepsilon_{j}\right)_{j\in\mathbb{N}}$ is a
sequence of positive real numbers converging to 0. Then any cluster point
$x_{\infty}$ of $\left(x_{j}\right)_{j\in\mathbb{N}}$ is a minimizer of
$f_{\infty}$ in $X$, and

$f_{\infty}(x_{\infty})=\lim_{j\to\infty}f_{j}\left(x_{j}\right)\;.$

  3. (c)

If moreover $\left(f_{j}\right)_{j\in\mathbb{N}}$ is equi-coercive and
$f_{\infty}$ is not identically $+\infty$, then every sequence
$\left(x_{j}\right)_{j\in\mathbb{N}}$ as in (b) has a cluster point.

###### Proof.

Part (a) is proved in [8, Theorem 7.4, p.69].

In the setting of (b), [8, Corollary 7.20, p.81] shows that any cluster point
$x_{\infty}$ of $\left(x_{j}\right)_{j\in\mathbb{N}}$ is a minimizer of
$f_{\infty}$. Moreover, either

$\displaystyle-\infty<\inf_{x\in X}f_{j}(x)$ $\displaystyle\leq
f_{j}\left(x_{j}\right)\leq\inf_{x\in X}f_{j}(x)+\varepsilon_{j}$ or
$\displaystyle-\infty=\inf_{x\in X}f_{j}(x)$ $\displaystyle\leq
f_{j}{x_{j}}\leq-\frac{1}{\varepsilon_{j}}$

for all $j\in\mathbb{N}$. By part (a), both sides of these inequalities
converge to $\min_{x\in X}f_{\infty}(x)=f_{\infty}\left(x_{\infty}\right)$.

In the setting of (c), let $t=\min_{x\in X}f_{\infty}(x)+1$ if $\min_{x\in
X}f_{\infty}(x)\neq-\infty$ and let $t\in\mathbb{R}$ be arbitrary otherwise.
By equi-coercivity, there exists a closed countably compact set $K\subseteq X$
such that $\left\\{f_{j}\leq t\right\\}\subseteq K$ for all $j\in\mathbb{N}$.
Part (a) implies that for all but finitely many $j\in\mathbb{N}$, we have
$x_{j}\in\left\\{f_{j}\leq t\right\\}\subseteq K$; whence
$\left(x_{j}\right)_{j\in\mathbb{N}}$ has a cluster point. ∎

The notion of $\Gamma$-convergence can be extended to families indexed by a
continuous parameter in a straightforward manner, as given in [5, Section 1.9,
p.37]. Moreover, many examples for application of $\Gamma$-convergence are
given in [5].
A more detailed investigation and deeper results, such as the relation between
$\Gamma$-convergence and G-convergence or topological set convergence in the
sense of Kuratowski, can be found in the book by Dal Maso [8].

##  3 $\mathbf{\Gamma}$-convergence and coerciveness of Tikhonov functionals

In this section, we consider the Tikhonov functional arising from the equation
$F(x)=y$. As usually only inexact data $y^{\delta}$ are available instead of
$y$, we will be approximating the exact Tikhonov functional by Tikhonov
functionals arising from the inexact data and a suitable sequence of
operators. Our aim is to apply Theorem 2.19 and derive convergence of minimum
values and minimum points of inexact functionals toward the minima and
minimizers of the exact functional, respectively. To this end, we postulate
conditions under which we will be able to prove $\Gamma$-convergence and equi-
mild coercivity of the functionals. Since both concepts depend on the topology
of the underlying space, we will consider the most commonly used ones, that is
the norm topology, the weak topology, and the weak∗ topology. At the end we
will compare the different conditions for those topologies.

###  3.1 The setting

Let $X,Y$ be Banach spaces. On $X$, we will consider three different
topologies: the norm topology, the weak topology, and, provided that $X$ is a
dual space, the weak∗ topology. Let $\tau$ be any of these three topologies.

We assume the following setup:

  * •

Let $F:X\to Y$ be an operator (not necessarily linear) with domain
$\emptyset\neq\operatorname{dom}(F)\subseteq X$ such that for all $y\in Y$,
the function

$X\to\mathbb{R}\cup\left\\{+\infty\right\\},\quad
x\mapsto\begin{cases}\left\|{F(x)-y}\right\|_{Y}&,\text{ if
}x\in\operatorname{dom}(F)\;,\\\ +\infty&,\text{ otherwise,}\end{cases}$

is $\tau$-lower semicontinuous on $X$ and norm upper semicontinuous on
$\operatorname{dom}(F)$.

  * •

Let $\Omega:X\to\mathbb{R}\cup\left\\{+\infty\right\\}$ be a functional with
the following properties:

    * –

its effective domain $\operatorname{dom}\left(\Omega\right)=\left\\{x\in
X;\Omega(x)<\infty\right\\}$ satisfies
$\operatorname{dom}\left(\Omega\right)\supseteq\operatorname{dom}(F)$;

    * –

$\Omega$ is $\tau$-lower semicontinuous on $X$ and norm-upper semicontinuous
on $\operatorname{dom}\left(\Omega\right)$.

######  Remark 3.1.

Note that the two norm-upper semicontinuity assumptions can be replaced by
norm continuity since $\tau$-lower semicontinuity implies norm-lower
semicontinuity. Moreover, if $\operatorname{dom}(F)$ is $\tau$-closed, then
$\tau$-lower semicontinuity of $x\mapsto\left\|{F(x)-y}\right\|_{Y}$ on
$\operatorname{dom}(F)$ implies $\tau$-lower semicontinuity on $X$ of the
first function above.

Let $y\in Y$ and $\alpha\in[0,\infty)$. Let $p\in[1,\infty)$ and define the
target Tikhonov functional as

$T:X\to\mathbb{R}\cup\left\\{+\infty\right\\},\quad
x\mapsto\begin{cases}\frac{1}{p}\left\|{F(x)-y}\right\|_{Y}^{p}+\alpha\,\Omega(x)&,\text{
if }x\in\operatorname{dom}(F)\;,\\\ +\infty&,\text{ otherwise}\;.\end{cases}$

We define approximations to $T$ in the following way.

  * •

For $n\in\mathbb{N}$, let $F_{n}:\operatorname{dom}\left(F_{n}\right)\to Y$ be
operators with
$\operatorname{dom}\left(F_{n}\right)\subseteq\operatorname{dom}(F)$ for all
$n\in\mathbb{N}$.

  * •

Let the sequence of domains
$\left(\operatorname{dom}\left(F_{n}\right)\right)_{n\in\mathbb{N}}$ be
increasing, i.e.
$\operatorname{dom}\left(F_{n}\right)\subseteq\operatorname{dom}\left(F_{n+1}\right)$
for all $n\in\mathbb{N}$, with the property that
$\bigcup_{n=1}^{\infty}\operatorname{dom}\left(F_{n}\right)$ is norm dense in
$\operatorname{dom}(F)$.

  * •

Assume that $\left(F_{n}\right)_{n\in\mathbb{N}}$ converges to $F$ locally
uniformly with respect to $\tau$ on the $\tau$-closure
$\overline{\operatorname{dom}(F)}^{\tau}$, meaning that for each
$x\in\overline{\operatorname{dom}(F)}^{\tau}$ there exists a $\tau$-open
neighborhood $U$ of $x$ such that

$\lim_{n\to\infty}\sup_{x^{\prime}\in
U\cap\operatorname{dom}\left(F_{n}\right)}\left\|{F_{n}\left(x^{\prime}\right)-F\left(x^{\prime}\right)}\right\|_{Y}=0\;.$

(Note that by the density assumption on
$\bigcup_{n=1}^{\infty}\operatorname{dom}\left(F_{n}\right)$, the set
$U\cap\operatorname{dom}\left(F_{n}\right)$ is not empty for sufficiently
large $n$.)

Moreover, let $\left(y_{n}\right)_{n\in\mathbb{N}}$ be a sequence in $Y$ with
$\lim_{n\to\infty}\left\|{y_{n}-y}\right\|_{Y}=0$ and let
$\left(\alpha_{n}\right)_{n\in\mathbb{N}}$ be a sequence in
$\left(0,\infty\right)$ converging to $\alpha$. We define the approximating
Tikhonov functionals by

$T_{n}:X\to\mathbb{R}\cup\left\\{+\infty\right\\},\quad
x\mapsto\begin{cases}\frac{1}{p}\left\|{F_{n}(x)-y_{n}}\right\|_{Y}^{p}+\alpha_{n}\,\Omega(x)&,\text{
if }x\in\operatorname{dom}\left(F_{n}\right)\;,\\\ +\infty&,\text{
otherwise}\;.\end{cases}$

######  Remark 3.2.

Let $X$ be equipped with the weak topology. Theorem 2.6 shows that if
$\operatorname{dom}(F)$ is convex, then $\operatorname{dom}(F)$ is weakly
closed if and only if it is norm closed. Moreover, Lemma 2.8 implies that if
$\operatorname{dom}(F)$ is weakly closed and $F$ is weak-to-weak continuous,
then the map $x\mapsto\left\|{F(x)-y}\right\|_{Y}$ is weakly lower
semicontinuous. Again by Theorem 2.6, if each
$\operatorname{dom}\left(F_{n}\right)$ is convex, then so is the increasing
union $\bigcup_{n\in\mathbb{N}}\operatorname{dom}\left(F_{n}\right)$, hence
this union is norm dense in $\operatorname{dom}(F)$ if and only if it is
weakly dense.

###  3.2 $\Gamma$-convergence of Tikhonov functionals

In this part, we will prove that
$\Gamma\operatorname{-}\lim_{n\to\infty}T_{n}=T$ under suitable assumptions,
that is we will prove that both the $\Gamma$-lower and $\Gamma$-upper limits
from Definition 2.12 of our approximating Tikhonov functionals are equal to
the target Tikhonov functional. As it turns out, the definition in terms of
neighborhoods is slightly easier to deal with in our setting and allows us to
handle all three mentioned topologies in a similar way.

######  Theorem 3.3.

Assume the setup of Subsection 3.1. Then $\left(T_{n}\right)_{n\in\mathbb{N}}$
$\mathit{\Gamma}$-converges to $T$ with respect to $\tau$.

###### Proof.

In each part, we have to show the lim inf inequality
$T\leq\Gamma\operatorname{-}\liminf_{n\to\infty}T_{n}$ and the lim sup
inequality $\Gamma\operatorname{-}\limsup_{n\to\infty}T_{n}\leq T$. We start
by showing the lim sup inequality in the norm topology. Since every open
neighborhood in the weak or the weak∗ topology is also an open neighborhood in
the norm topology, this also establishes the lim sup inequality in the weak
and in the weak∗ topology, see [8, Proposition 6.3].

To show the lim sup inequality in the norm topology, let $x\in X$. If
$x\notin\operatorname{dom}(F)$, then $T(x)=\infty$, so the inequality
trivially holds. Let $x\in\operatorname{dom}(F)$. For $k\in\mathbb{N}$ with
$k\geq 1$, let $B_{1/k}(x)$ be the open ball with radius $\frac{1}{k}$
centered at $x$. In the definition of the $\Gamma$-upper limit, it suffices to
consider open neighborhoods of $x$ of the form $B_{1/k}(x)$, that is,

$\Gamma\operatorname{-}\limsup_{n\to\infty}T_{n}(x)=\lim_{k\to\infty}\limsup_{n\to\infty}\inf_{x^{\prime}\in
B_{1/k}(x)}T_{n}\left(x^{\prime}\right)\;;$

see for instance [8, Remark 4.3]. Let $k\in\mathbb{N}$ with $k\geq 1$. Since
$\bigcup_{n=1}^{\infty}\operatorname{dom}\left(F_{n}\right)$ is norm dense in
$\operatorname{dom}(F)$, there exist $n_{0}\in\mathbb{N}$ and $\xi_{k}\in
B_{1/k}(x)\cap\operatorname{dom}\left(F_{n_{0}}\right)$. If $n\geq n_{0}$,
then $\xi_{k}\in\operatorname{dom}\left(F_{n}\right)$, so the convergence
assumption on $\left(F_{n}\right)_{n\in\mathbb{N}}$, which in particular
implies pointwise convergence on
$\bigcup_{n=1}^{\infty}\operatorname{dom}\left(F_{n}\right)$, yields
$\lim_{n\to\infty}\left\|{F_{n}\left(\xi_{k}\right)-F\left(\xi_{k}\right)}\right\|_{Y}=0$.
Hence

$\lim_{n\to\infty}T_{n}\left(\xi_{k}\right)=T\left(\xi_{k}\right)$

and so

$\limsup_{n\to\infty}\inf_{x^{\prime}\in
B_{1/k}(x)}T_{n}(x)\leq\limsup_{n\to\infty}T_{n}\left(\xi_{k}\right)=T\left(\xi_{k}\right)\;.$

Since $\lim_{k\to\infty}\left\|{\xi_{k}-x}\right\|_{X}=0$ and
$\xi_{k}\in\operatorname{dom}(F)$ for all $k$, we find that
$\limsup_{k\to\infty}T\left(\xi_{k}\right)\leq T(x)$ and so

$\Gamma\operatorname{-}\limsup_{n\to\infty}T_{n}(x)=\lim_{k\to\infty}\limsup_{n\to\infty}\inf_{x^{\prime}\in
B_{1/k}(x)}T_{n}(x)\leq\limsup_{k\to\infty}T\left(\xi_{k}\right)\leq T(x)\;,$

which completes the first step.

It remains to show the lim inf inequality in all three cases. For all
$n\in\mathbb{N}$ define

$\displaystyle S_{n}:X\to\mathbb{R}\cup\left\\{+\infty\right\\},$
$\displaystyle\quad
x\mapsto\begin{cases}\frac{1}{p}\left\|{F_{n}(x)-y_{n}}\right\|_{Y}^{p}&,\text{
if }x\in\operatorname{dom}\left(F_{n}\right)\;,\\\ +\infty&,\text{
otherwise}\;,\end{cases}$ and $\displaystyle
R_{n}:X\to\mathbb{R}\cup\left\\{+\infty\right\\},$ $\displaystyle\quad
x\mapsto\begin{cases}\alpha_{n}\,\Omega(x)&,\text{ if
}x\in\operatorname{dom}\left(\Omega\right)\;,\\\ +\infty&,\text{
otherwise}\;.\end{cases}$

Then $T_{n}\geq S_{n}+R_{n}$ for all $n\in\mathbb{N}$ and so by a basic
inequality for the $\Gamma\operatorname{-}\liminf$ of a sum (see [8,
Proposition 6.17]), we find that

$\Gamma\operatorname{-}\liminf_{n\to\infty}T_{n}\geq\Gamma\operatorname{-}\liminf_{n\to\infty}\left(S_{n}+R_{n}\right)\geq\Gamma\operatorname{-}\liminf_{n\to\infty}S_{n}+\Gamma\operatorname{-}\liminf_{n\to\infty}R_{n}\;.$

To deal with the second summand, observe that since
$\left(\alpha_{n}\right)_{n\in\mathbb{N}}$ tends to $\alpha$, we have

$\left(\Gamma\operatorname{-}\liminf_{n\to\infty}R_{n}\right)(x)=\sup_{U\in\mathcal{N}(x)}\liminf_{n\to\infty}\inf_{x^{\prime}\in
U}R_{n}\left(x^{\prime}\right)=\alpha\,\sup_{U\in\mathcal{N}(x)}\inf_{x^{\prime}\in
U}\Omega\left(x^{\prime}\right)=\alpha\,\Omega(x)\;,$

where that last equality follows from the lower semicontinuity of
$x\mapsto\Omega(x)$.

It remains to show that

$\left(\Gamma\operatorname{-}\liminf_{n\to\infty}S_{n}\right)(x)\geq\begin{cases}\frac{1}{p}\left\|{F(x)-y}\right\|_{Y}^{p}&,\text{
if }x\in\operatorname{dom}(F)\;,\\\ +\infty&,\text{ otherwise}\;.\end{cases}$

Since the function $t\mapsto\frac{1}{p}t^{p}$ is continuous and increasing, it
suffices to consider the case $p=1$, as the $\Gamma$-lower limit is preserved
by applying continuous increasing functions, see [8, Proposition 6.16]. If $x$
is not in the $\tau$-closure of $\operatorname{dom}(F)$, then there exists a
$\tau$-open neighborhood $U$ of $x$ that has an empty intersection with the
closure of $\operatorname{dom}(F)$, and so
$\Gamma\operatorname{-}\liminf_{n\to\infty}{T_{n}}(x)=+\infty$, so the
inequality holds in this case.

Let $x$ be in the $\tau$-closure of $\operatorname{dom}(F)$, $\varepsilon>0$,
and $t\in\mathbb{R}$ with $\left\|{F(x)-y}\right\|_{Y}>t$. Here, we use the
convention $\left\|{F(x)-y}\right\|_{Y}=+\infty$ for
$x\notin\operatorname{dom}(F)$. For all
$x^{\prime}\in\operatorname{dom}\left(F_{n}\right)$ and all $n\in\mathbb{N}$,
the triangle inequality yields

$\left\|{F_{n}\left(x^{\prime}\right)-y_{n}}\right\|_{Y}\geq\left\|{F\left(x^{\prime}\right)-y}\right\|_{Y}-\left(\left\|{F_{n}\left(x^{\prime}\right)-F\left(x^{\prime}\right)}\right\|_{Y}+\left\|{y-y_{n}}\right\|\right)_{Y}\;.$

The convergence assumptions on $\left(F_{n}\right)_{n\in\mathbb{N}}$ and on
$\left(y_{n}\right)_{n\in\mathbb{N}}$ show that there exist
$n_{0}\in\mathbb{N}$ and an open neighborhood $V_{1}$ of $x$ such that for all
$n\geq n_{0}$ and all $x^{\prime}\in
V_{1}\cap\operatorname{dom}\left(F_{n}\right)$, each of the two summands in
parentheses is less than $\varepsilon$. By lower semicontinuity of
$x\mapsto\left\|{F(x)-y}\right\|_{Y}$, we may further find another open
neighborhood $V_{2}$ of $x$ and achieve that

$\left\|{F\left(x^{\prime}\right)-y}\right\|_{Y}>t$

for all $x^{\prime}\in V_{2}\cap\operatorname{dom}(F)$. Therefore, setting
$V:=V_{1}\cap V_{2}$, we have

$\displaystyle\Gamma\operatorname{-}\liminf_{n\to\infty}S_{n}(x)$
$\displaystyle=\sup_{U\in\mathcal{N}(x)}\liminf_{n\to\infty}\inf_{x^{\prime}\in
U\cap\operatorname{dom}\left(F_{n}\right)}\left\|{F_{n}\left(x^{\prime}\right)-y_{n}}\right\|_{Y}$
$\displaystyle\geq\liminf_{n\to\infty}\inf_{x^{\prime}\in
V\cap\operatorname{dom}\left(F_{n}\right)}\left\|{F_{n}\left(x^{\prime}\right)-y_{n}}\right\|_{Y}$
$\displaystyle\geq t-2\,\varepsilon\;.$

Since $\varepsilon>0$ and $t<\left\|{F(x)-y}\right\|_{Y}$ were arbitrary, the
desired inequality follows. ∎

######  Remark 3.4.

Examination of the proof shows that the assumption of locally uniform
convergence of $\left(F_{n}\right)_{n\in\mathbb{N}}$ can be weakened to

$\inf_{U\in\mathcal{N}(x)}\limsup_{n\to\infty}\sup_{x^{\prime}\in
U\cap\operatorname{dom}\left(F_{n}\right)}\left\|{F_{n}\left(x^{\prime}\right)-F\left(x^{\prime}\right)}\right\|_{Y}=0\quad\text{
for all }x\in\operatorname{dom}(F)\;.$

Equivalently, for all $x\in\operatorname{dom}(F)$ and all $\varepsilon>0$,
there exist an open neighborhood $U$ of $x$ and $n_{0}\in\mathbb{N}$ such that

$\left\|{F_{n}\left(x^{\prime}\right)-F\left(x^{\prime}\right)}\right\|_{Y}<\varepsilon\quad\text{
for all }n\geq n_{0}\text{ and all }x^{\prime}\in
U\cap\operatorname{dom}\left(F_{n}\right)\;.$

This differs from local uniform convergence in that the neighborhood $U$ may
depend on $\varepsilon$.

We require the following elementary lemma.

######  Lemma 3.5.

Let $\left(f_{j}\right)_{j\in\mathbb{N}}$ be a sequence of functionals from a
topological space $X$ onto $\overline{\mathbb{R}}$ that
$\mathit{\Gamma}$-converges to
$f:X\to\overline{\mathbb{R}}$. Let $\left(\lambda_{j}\right)_{j\in\mathbb{N}}$
be a sequence in $\left(0,\infty\right)$ converging to
$\lambda\in\left[0,\infty\right]$. Then

$\Gamma\operatorname{-}\lim_{j\to\infty}\left(\lambda_{j}\,f_{j}\right)(x)=\lambda\,f(x)$

whenever the right-hand side is not of the form $0\cdot\pm\infty$ or
$\infty\cdot 0$.

###### Proof.

We use the following basic fact: If $\left(\phi_{j}\right)_{j\in\mathbb{N}}$
is a sequence in $\overline{\mathbb{R}}$, then

$\liminf_{j\to\infty}\lambda_{j}\,\phi_{j}\geq\lambda\,\liminf_{j\to\infty}\phi_{j}$
(3.1)

whenever the right-hand side is defined. Indeed, if
$-\infty<t<\liminf_{j\to\infty}\phi_{j}$, then $\phi_{j}\geq t$ for all but
finitely many $j\in\mathbb{N}$ and so

$\liminf_{j\to\infty}\lambda_{j}\,\phi_{j}\geq\liminf_{j\to\infty}\lambda_{j}\,t=\lambda\,t,$

provided the right-hand side is not of the form $\infty\cdot 0$.
Distinguishing the cases $\lambda=0,\lambda=+\infty$ and
$\lambda\in\left(0,\infty\right)$, Inequality (3.1) follows.

Let $x\in X$ and $U\in\mathcal{N}(x)$. Then Inequality (3.1) shows that

$\liminf_{j\to\infty}\inf_{x^{\prime}\in
U}\left(\lambda_{j}\,f_{j}\right)\left(x^{\prime}\right)\geq\lambda\,\liminf_{j\to\infty}\inf_{x^{\prime}\in
U}f_{j}\left(x^{\prime}\right),$ (3.2)

provided the right-hand side is defined. The definition of the $\Gamma$-lower
limit shows that if $f(x)\in\mathbb{R}\setminus\left\\{0\right\\}$, then there
exists $V\in\mathcal{N}(x)$ with $\liminf_{j\to\infty}\inf_{x^{\prime}\in
U}f_{j}\left(x^{\prime}\right)\in\mathbb{R}\setminus\left\\{0\right\\}$ for
all $U\in\mathcal{N}(x)$ with $U\subseteq V$. So if $\lambda\,f(x)$ is
defined, then (3.2) holds for all $U\in\mathcal{N}(x)$ with $U\subseteq V$.
Taking the supremum over all such $U$ in (3.2) gives

$\Gamma\operatorname{-}\liminf_{j\to\infty}\left(\lambda_{j}\,f_{j}\right)(x)\geq\lambda\,f(x).$

A similar argument shows that

$\Gamma\operatorname{-}\limsup_{j\to\infty}\left(\lambda_{j}\,f_{j}\right)(x)\leq\lambda\,f(x)$

when the right-hand side is defined. Combining both inequalities gives the
result. ∎

######  Corollary 3.6.

Assume the setup of Subsection 3.1 and that $\alpha>0$. Then
$\left(\frac{1}{\alpha_{n}}T_{n}\right)_{n\in\mathbb{N}}$
$\mathit{\Gamma}$-converges to $\frac{1}{\alpha}T$ in the topology $\tau$.

If $\alpha=0$, then we have the following convergence instead:

######  Proposition 3.7.

Assume the conditions of Theorem 3.3 and let $\alpha=0$. If additionally we
have that

$\lim_{n\to\infty}\frac{\left\|{y_{n}-y}\right\|_{Y}}{\alpha_{n}^{1/p}}=0\;,$

and

$\lim_{n\to\infty}\frac{\left\|{F_{n}(x)-F(x)}\right\|_{Y}}{\alpha_{n}^{1/p}}=0$

for each $x\in F^{-1}(y)$, then
$\left(\frac{1}{\alpha_{n}}T_{n}\right)_{n\in\mathbb{N}}$
$\mathit{\Gamma}$-converges to

$\tilde{T}:X\to\mathbb{R}\cup\left\\{+\infty\right\\},\quad
x\mapsto\begin{cases}\Omega(x)&,\text{ if }x\in F^{-1}(y)\;,\\\
+\infty&,\text{ otherwise}\;,\end{cases}$

in the considered topology $\tau$.

###### Proof.

For any $x\notin F^{-1}(y)$ we have that $T(x)\neq 0$, so Lemma 3.5 implies
that
$\Gamma\operatorname{-}\lim_{n\to\infty}\frac{1}{\alpha_{n}}T_{n}(x)=+\infty$
for $x\notin F^{-1}(y)$.

Let now $x\in F^{-1}(y)$. The lim sup inequality
$\Gamma\operatorname{-}\limsup_{n\to\infty}\frac{1}{\alpha_{n}}T_{n}\leq\tilde{T}$
follows from

$\displaystyle\left(\Gamma\operatorname{-}\limsup_{n\to\infty}\frac{1}{\alpha_{n}}T_{n}\right)(x)$
$\displaystyle=\sup_{U\in\mathcal{N}(x)}\limsup_{n\to\infty}\inf_{x^{\prime}\in
U\cap\operatorname{dom}\left(F_{n}\right)}\frac{1}{p\
\alpha_{n}}\left\|{F_{n}\left(x^{\prime}\right)-y_{n}}\right\|_{Y}^{p}+\Omega\left(x^{\prime}\right)$
$\displaystyle\leq\limsup_{n\to\infty}\frac{1}{p\
\alpha_{n}}\left\|{F_{n}(x)-y_{n}}\right\|_{Y}^{p}+\Omega(x)$
$\displaystyle=\frac{1}{p}\left(\limsup_{n\to\infty}\frac{\left\|{F_{n}(x)-F(x)}\right\|_{Y}}{\alpha_{n}^{1/p}}+\frac{\left\|{y-y_{n}}\right\|_{Y}}{\alpha_{n}^{1/p}}\right)^{p}+\Omega(x)$
$\displaystyle=\Omega(x)\;.$

The lim inf inequality
$\tilde{T}\leq\Gamma\operatorname{-}\liminf_{n\to\infty}\frac{1}{\alpha_{n}}T_{n}$
also follows easily from the lower semicontinuity of $\Omega$:

$\displaystyle\Gamma\operatorname{-}\liminf_{n\to\infty}\frac{1}{\alpha_{n}}T_{n}(x)\geq\Gamma\operatorname{-}\liminf_{n\to\infty}\Omega(x)=\Omega(x).$

Combining both inequalities yields the result. ∎

This result is similar to one by Burger, see [6, Lemma 3.4]. However, our
initial assumptions on the setting are slightly more general, and therefore
his result is a specific case of the proposition above.

###  3.3 Equi-mild coercivity of Tikhonov functionals

In this part, we will investigate whether our Tikhonov functionals are equi-
mildly coercive, that is if there exists a suitable compact set (in the
relevant topology) such that all the functionals have their global infimum on
that set.

######  Proposition 3.8.

Assume the setup of Subsection 3.1 and let $\alpha>0$. In addition, assume
that the sublevel sets $\left\\{\Omega\leq t\right\\}$ are relatively
countably compact in the topology $\tau$ for every $t\in\mathbb{R}$. Then
$\left(T_{n}\right)_{n\in\mathbb{N}}$ is equi-coercive in the considered
topology $\tau$.

###### Proof.

Let $t\in\mathbb{R}$. Since $\alpha>0$, there exists $\delta>0$ with
$\alpha_{n}>\delta>0$ for all $n\in\mathbb{N}$. Thus, we have
$T_{n}(x)\geq\delta\,\Omega(x)$ for every
$x\in\operatorname{dom}\left(F_{n}\right)$. Hence

$\left\\{T_{n}\leq
t\right\\}\subseteq\left\\{\Omega\leq\frac{t}{\delta}\right\\}$

for all $n\in\mathbb{N}$. Therefore, the equi-coercivity follows from the
assumed relative countable compactness of the sublevel sets of $\Omega$ w.r.t.
the topology $\tau$. ∎

######  Corollary 3.9.

Assume the setup of Subsection 3.1, let $\alpha>0$ and assume that the
sublevel sets $\left\\{\Omega\leq t\right\\}$ are bounded for every
$t\in\mathbb{R}$.

  1. (a)

If $X$ is a reflexive Banach space, then $\left(T_{n}\right)_{n\in\mathbb{N}}$
is equi-coercive in the weak topology.

  2. (b)

If $X$ is a dual space, then $\left(T_{n}\right)_{n\in\mathbb{N}}$ is equi-
coercive in the weak∗ topology.

###### Proof.

This is immediate from Proposition 3.8 and the Banach–Alaoglu theorem (Theorem
2.5, see also Theorem 2.3). ∎

######  Remark 3.10.

If $\alpha=0$ and if there exists a bounded sequence of minimizers of
$\left(T_{n}\right)_{n\in\mathbb{N}}$, then
$\left(T_{n}\right)_{n\in\mathbb{N}}$ is equi-mildly coercive in the weak∗
topology (assuming $X$ is a dual space). This again follows from the
Banach–Alaoglu theorem (Theorem 2.5).

Moreover, if minimizers are convergent in norm, then
$\left(T_{n}\right)_{n\in\mathbb{N}}$ is equi-mildly coercive in the norm
topology. Indeed, if $m_{n}\in X$ is a minimizer of $T_{n}$ for all
$n\in\mathbb{N}$ such that $\left(m_{n}\right)_{n\in\mathbb{N}}$ converges to
some $m\in X$ in norm, then

$K=\left\\{m_{n};n\in\mathbb{N}\right\\}\cup\left\\{m\right\\}$

is norm compact, because a convergent sequence, together with its limit point,
forms a compact set in any metric space. Clearly,

$\inf_{x\in K}T_{n}(x)=\inf_{x\in X}T_{n}(x)\quad\text{ for all
}n\in\mathbb{N}\;.$

######  Remark 3.11.

Proposition 3.8 is also true for
$\left(\frac{1}{\alpha_{n}}T_{n}\right)_{n\in\mathbb{N}}$, even for
$\alpha=0$, since $\frac{1}{\alpha_{n}}T_{n}\geq\Omega$ for all
$n\in\mathbb{N}$.

##  4 Examples

In this section we aim to illustrate the usefulness of the above theoretical
results by applying it to the following examples.

######  Example 4.1.

First we consider an integral equation of first kind with continuous kernel.
To this end let $X=Y=Z=L^{\infty}\left(\left[0,1\right]\right)$, which is the
dual space of $L^{1}\left(\left[0,1\right]\right)$, and consider the
corresponding weak-$*$ topology on $L^{\infty}\left(\left[0,1\right]\right)$.
Let $K:\left[0,1\right]\times\left[0,1\right]\to\mathbb{R}$ be continuous. Let

$\displaystyle F:L^{\infty}\left(\left[0,1\right]\right)\to
L^{\infty}\left(\left[0,1\right]\right),\quad F(x)(t)$
$\displaystyle=\int_{0}^{1}K\left(s,t\right)x(s)\,\mathrm{d}s\;,$
$\displaystyle\Omega:L^{\infty}\left(\left[0,1\right]\right)\to\left[0,\infty\right],\quad\Omega(x)$
$\displaystyle=\left\|{x}\right\|_{L^{\infty}}\;.$

Obviously, both mappings are (norm-to-norm) continuous. Furthermore, $F$ is
also weak∗-to-weak∗ continuous, since it is the adjoint operator of the
integral operator on $L^{1}\left(\left[0,1\right]\right)$ with kernel
$\tilde{K}\left(s,t\right):=K\left(t,s\right)$. Hence
$x\mapsto\left\|{F(x)-y}\right\|$ and $\Omega$ are both weak∗ lower
semicontinuous by Lemma 2.8. Therefore, Theorem 3.3 applies to this example.

######  Example 4.2.

Parameter identifications for PDEs represent an important class of inverse
problems. We consider the elliptic boundary value problem

$\displaystyle-\Delta u+c\,u$ $\displaystyle=$ $\displaystyle f\qquad\mbox{in
}D\;,$ (4.1a) $\displaystyle u$ $\displaystyle=$ $\displaystyle
0\qquad\mbox{on }\partial D\;.$ (4.1b)

Here, $D\subset\mathbb{R}^{d}$ denotes an open, convex, bounded domain with
smooth boundary $\partial D$, and $d\in\left\\{1,2,3\right\\}$. Equation
(4.1a) can be seen as stationary Schrödinger equation with potential $c(x)$.
The forward (direct) problem means to compute the solution $u(x)$ for given
functions $c$ and $f$. The inverse problem consists of calculating the source
term $f$ from $u$ with given potential $c$. The mathematical formulation of
this inverse problem is

$F(f)=u^{\mathrm{meas}}\;,$ (4.2)

where $F:\operatorname{dom}(F)\subseteq X\to Y$ maps the parameter $f$ to the
unique (weak) solution $u$ of (4.1) and $u^{\mathrm{meas}}$ are the measured
data. We define $X:=L^{2}(D)$ and $\operatorname{dom}(F):=\\{f\in
L^{2}(D);\left\|{f}\right\|_{X}\leq\rho\\}$, where $\rho>0$. To ensure the
existence and uniqueness of $u$ we furthermore assume $c\in L^{\infty}(D)$ and
$c\geq 0$ a.e. Since $u$ is to be a weak solution of (4.1) we introduce the
Sobolev space $H_{0}^{1}(D)$, which is the closure of the space
$C_{0}^{\infty}(D)$ of infinitely differentiable functions with compact
support w.r.t. the $H^{1}$-norm. As usual, we equip $H_{0}^{1}(D)$ with the
scalar product

$\left\langle u,v\right\rangle_{H_{0}^{1}(D)}:=\int_{D}\nabla u\cdot\nabla
v\,\mathrm{d}x\;,\qquad u,v\in H_{0}^{1}(D)\;.$

On this space its induced norm is equivalent to the $H^{1}$-norm. We note that
$H_{0}^{1}(D)$ is continuously embedded in $L^{2}(D)$ ([1, Theorem 4.12]).
Hence $u$ solves the variational problem

$a_{c}(u,v)=\ell_{f}(v)\qquad\mbox{for all }v\in H_{0}^{1}(D)$ (4.3)

with the symmetric bilinear form $a_{c}:H^{1}_{0}(D)\times
H^{1}_{0}(D)\to\mathbb{R}$

$a_{c}(u,v)=\int_{D}\left(\nabla u\cdot\nabla v+c\,u\,v\right)\,\mathrm{d}x$

and the bounded, linear functional on $H_{0}^{1}(D)$

$\ell_{f}(v)=\int_{D}f\>\\!v\,\mathrm{d}x\;,\qquad v\in H_{0}^{1}(D)\;.$

We have that $a_{c}$ is $H_{0}^{1}$-coercive since

$a_{c}(u,u)\geq\left\|{u}\right\|_{H_{0}^{1}(D)}^{2}\qquad\mbox{for all }u\in
H_{0}^{1}(D)\;.$

It is also continuous on $H^{1}_{0}(D)\times H^{1}_{0}(D)$ since for all
$u,v\in H_{0}^{1}(D)$ we have

$\displaystyle a_{c}(u,v)$ $\displaystyle\leq\left|{\left\langle
u,v\right\rangle_{H_{0}^{1}(D)}}\right|+\left\|{c\,u\,v}\right\|_{L^{1}(D)}$
$\displaystyle\leq\left\|{u}\right\|_{H_{0}^{1}(D)}\left\|{v}\right\|_{H_{0}^{1}(D)}+\left\|{c}\right\|_{L^{2}(D)}\left\|{u\,v}\right\|_{L^{2}(D)}$
$\displaystyle\leq\left\|{u}\right\|_{H_{0}^{1}(D)}\left\|{v}\right\|_{H_{0}^{1}(D)}+\left\|{c}\right\|_{L^{2}(D)}\left\|{u}\right\|_{L^{4}(D)}\left\|{v}\right\|_{L^{4}(D)}$
$\displaystyle\leq\left(1+\left\|{c}\right\|_{L^{2}(D)}\,C_{S}^{2}\right)\left\|{u}\right\|_{H_{0}^{1}(D)}\left\|{v}\right\|_{H_{0}^{1}(D)}\;,$

where $C_{S}$ is a constant depending only on dimension $d$ and domain $D$. In
the last step we used the Sobolev Embedding Theorem, see e.g. [1, Theorem
4.12]. The Lax–Milgram theorem is thus applicable and guarantees that the
linear mapping $A_{c}:H_{0}^{1}(D)\to\left(H_{0}^{1}(D)\right)^{*}=H^{-1}(D)$
given by $A_{c}(u):=a_{c}(u,\cdot)$ is continuously invertible. Since
$f\mapsto\ell_{f}$ is linear and bounded as a mapping from $L^{2}(D)$ to
$(H^{1}_{0}(D))^{*}$, this yields continuity of $f\mapsto
A_{c}^{-1}(\ell_{f})=u$ as a mapping from $L^{2}(D)$ to $H^{1}_{0}(D)$. From
Theorem 4 (in §6.3) and Theorem 6 (in §6.2) of [10] we even have that, by our
assumptions to $D$, the weak solution $u\in H^{2}(D)$ and depends continuously
on $f$. We define $Y:=L^{2}(D)$. Since the inclusion $H^{1}_{0}(D)\subset
L^{2}(D)$ is continuous, the forward operator $F:f\mapsto u$ is therefore
norm-to-norm continuous. Moreover, since $F$ is linear, it is also weak-to-
weak continuous ([14, Theorem 2.5.11]). From this, as well as from norm and
weak closedness of $\operatorname{dom}(F)$, it follows that the Tikhonov
functional $T:\operatorname{dom}(F)\to\mathbb{R},$

$T(f):=\frac{1}{2}\left\|{F(f)-u^{\mathrm{meas}}}\right\|^{2}_{Y}+\frac{\alpha}{2}\left\|{f}\right\|_{X}^{2}$
(4.4)

satisfies the continuity assumptions of Subsection 3.1 in the norm and in the
weak topology.
Minimizing $T$ yields a stable regularization method for (4.2), see, e.g.,
[9]. Since in practical applications the boundary value problem (4.1) has to
be solved numerically, only approximations of the forward solver $F(f)$ can be
computed, e.g., by using the Finite Element method. This method relies on the
Galerkin approximation for the weak solution $u$ by choosing finite
dimensional, conformal Finite Element spaces $V_{n}\subset H^{1}_{0}(D)$,
$\dim\left(V_{n}\right)<\infty$, $V_{n}\subset V_{n+1}$, and

$\overline{\bigcup_{n\in\mathbb{N}}V_{n}}=H^{1}_{0}(D)\;,$

and subsequently solving the variational problems to find $u_{n}\in V_{n}$
such that

$a_{c}(u_{n},\varphi)=\ell_{f}(\varphi)\qquad\mbox{for all }\varphi\in
V_{n}\;.$ (4.5)

Again the symmetry, continuity and coercivity of $a_{c}$ yields a unique
solution $u_{n}\in V_{n}$. This leads to the fact that in applications one
minimizes the Tikhonov functional $T_{n}:\operatorname{dom}(F)\to\mathbb{R},$

$T_{n}(f):=\frac{1}{2}\left\|{F_{n}(f)-u^{\mathrm{meas}}}\right\|^{2}_{Y}+\frac{\alpha}{2}\left\|{f}\right\|_{X}^{2}\;,$
(4.6)

where $F_{n}(f):=u_{n}$ is the unique solution of (4.5). Standard settings for
$V_{n}$ are piecewise linear B-splines for which the convergence

$\left\|{u_{n}-u}\right\|_{L^{2}(D)}\leq
C\,n^{-2}\left\|{f}\right\|_{L^{2}(D)}$ (4.7)

holds true for $n\to\infty$, where $C>0$ is a constant only depending on $D,d$
and $c$. The proof of this estimate relies on the Céa-Lemma and the Theorem of
Aubin-Nitsche (see [3, 15] and [4, Folgerung 7.7]). Higher rates are obtained
by using higher order splines and for $u$ being of sufficient regularity.
Since $\operatorname{dom}\left(F_{n}\right)=\operatorname{dom}(F)$, the
estimate (4.7) and the definition of $\operatorname{dom}(F)$ immediately yield
uniform convergence $F_{n}\to F$ as $n\to\infty$ in $X$.

All prerequisites in Subsection 3.1 are thus satisfied and Theorem 3.3 is
valid for this example in both topologies, the norm topology and the weak
topology. Furthermore, we have that $\left(T_{n}\right)_{n\in\mathbb{N}}$ is
equi-coercive in the weak topology of $X$ (Proposition 3.8). This leads to the
important fact that from any minimizing sequences
$\left(f_{k,n}\right)_{k\in\mathbb{N}}$ of $T_{n}$ one can construct a weakly
convergent minimizing sequence $\left(f_{k(n),n}\right)_{n\in\mathbb{N}}$ of
$T$ with a monotonically increasing function $k(n)$ (by Theorem 2.19), meaning
that any minimizer of $T_{n}$ is in this sense close to a minimizer of $T$,
which is very important from a practical point of view.

##  5 Conclusion

In this article we presented simple criteria like local uniform convergence
and equi-mild coercivity to obtain $\Gamma$-convergence for families of
Tikhonov functionals for nonlinear, continuous (maybe ill-posed) operator
equations in Banach spaces and convergence of their corresponding minimal
sequences. The topologies addressed in this article for convergence and
continuity are the norm, weak and weak∗ topologies. The results are of
importance for practical applications, where an exact evaluation of the
original Tikhonov functional is not possible, since, e.g., the forward
operator can not be exactly evaluated or forward solutions and / or
measurement data are represented in finite dimensional subspaces. Then our
results guarantee that, under mild conditions, the minimizer of the
approximated functional is close to a minimizer of the original functional.

## References

  * [1] R. Adams and J. J. Fournier.  Sobolev spaces.  Elsevier, 2003.
  * [2] L. Ambrosio and V. M. Tortorelli.  Approximation of functional depending on jumps by elliptic functional via $\Gamma$-convergence.  Communications on Pure and Applied Mathematics, 43(8):999–1036, 1990.
  * [3] J. Aubin.  Behavior of the error of the approximate solutions of boundary value problems for linear elliptic operators by Galerkin’s and finite difference methods.  Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 21(4):599–637, 1967.
  * [4] D. Braess.  Finite Elemente: Theorie, schnelle Löser und Anwendungen in der Elastizitätstheorie.  Springer, 2013.
  * [5] A. Braides.  Gamma-convergence for Beginners.  Oxford University Press, New York, 2005.
  * [6] M. Burger.  Variational Regularization in Inverse Problems and Machine Learning.  arXiv:2112.04591, 2021.
  * [7] J. B. Conway.  A Course in Functional Analysis, volume 96 of Graduate Texts in Mathematics.  Springer-Verlag, New York, second edition, 1990.
  * [8] G. Dal Maso.  An Introduction to $\mathit{\Gamma}$-Convergence.  Springer New York, New York, 1993.
  * [9] H. W. Engl, M. Hanke, and A. Neubauer.  Regularization of Inverse Problems.  Mathematics and Its Applications. Kluwer Academic Publishers, Dordrecht, 1996.
  * [10] L. C. Evans.  Partial Differential Equations, volume 19 of Graduate Studies in Mathematics.  American Mathematical Soc., 2010.
  * [11] S. P. Franklin.  Spaces in which sequences suffice.  Fundamenta Mathematicae, 57(1):107–115, 1965.
  * [12] C. Groetsch.  The theory of Tikhonov regularization for Fredholm equations.  Pitman Publishing Limited, Boston, 1984.
  * [13] A. K. Louis.  Inverse und schlecht gestellte Probleme.  Teubner, Stuttgart, 1989.
  * [14] R. E. Megginson.  An Introduction to Banach Space Theory.  Graduate Texts in Mathematics. Springer-Verlag New York, 1998.
  * [15] J. Nitsche.  Ein Kriterium für die Quasi-Optimalität des Ritzschen Verfahrens.  Numerische Mathematik, 11(4):346–348, 1968.
  * [16] R. Plato and G. Vainikko.  On the Regularization of Projection Methods for Solving Ill-Posed Problems.  Numer. Math., 57:63–79, 1990.
  * [17] C. Pöschl, E. Resmerita, and O. Scherzer.  Discretization of variational regularization in Banach spaces.  Inverse Problems, 26(10), 2010.  ID 105017.
  * [18] A. Rieder.  Keine Probleme mit inversen Problemen (No Problems with inverse Problems).  Vieweg, Wiesbaden, 2003.
  * [19] W. Rudin.  Functional Analysis.  International Series in Pure and Applied Mathematics. McGraw-Hill Inc., New York, second edition, 1991.
  * [20] T. Schuster, B. Kaltenbacher, B. Hofmann, and K. Kazimierski.  Regularization Methods in Banach Spaces.  de Gruyter, 2012.
  * [21] L. Vierus and T. Schuster.  Well-defined forward operators in dynamic diffractive tensor tomography using viscosity solutions of transport equations.  Electron. Trans. Numer. Anal., 57:80–100, 2022.


# TSNAT: Two-Step Non-Autoregressvie Transformer Models
for Speech Recognition

###### Abstract

The autoregressive (AR) models, such as attention-based encoder-decoder models
and RNN-Transducer, have achieved great success in speech recognition. They
predict the output sequence conditioned on the previous tokens and acoustic
encoded states, which is inefficient on GPUs. The non-autoregressive (NAR)
models can get rid of the temporal dependency between the output tokens and
predict the entire output tokens in at least one step. However, the NAR model
still faces two major problems. On the one hand, there is still a great gap in
performance between the NAR models and the advanced AR models. On the other
hand, it’s difficult for most of the NAR models to train and converge. To
address these two problems, we propose a new model named the two-step non-
autoregressive transformer(TSNAT), which improves the performance and
accelerating the convergence of the NAR model by learning prior knowledge from
a parameters-sharing AR model. Furthermore, we introduce the two-stage method
into the inference process, which improves the model performance greatly. All
the experiments are conducted on a public Chinese mandarin dataset ASIEHLL-1.
The results show that the TSNAT can achieve a competitive performance with the
AR model and outperform many complicated NAR models.

Index Terms: Autoregressive, Non-Autoregressive, Transformer, Two-Step, Speech
Recognition

##  1 Introduction

End-to-end models have achieved great success in speech recognition,
especially attention-based encoder-decoder models [1, 2, 3, 4] and transducer-
based models [5, 6, 7, 8, 9]. Most of these end-to-end models generate the
target sequence in an autoregressive fashion, which predicts the next token
conditioned on the previously generated tokens and the acoustic encoded
sequence. The autoregressive characteristic makes the inference process must
be carried out step by step, which cannot be implemented in parallel and
results in a large latency. By contrast, the non-autoregressive model (NAR)
can get rid of the temporal dependency and directly generate the target
sequences based on the encoded acoustic states in at least one step.

Although the Non-autoregressive models can perform inference very efficiently,
it still faces two significant problems. On the one hand, there is still a
great gap in performance between the advanced autoregressive (AR) model and
the non-autoregressive model. Chen et.al proposed two kinds of iterative
inference methods to alleviate the problem [10]. Too many iterations have a
negative impact on the speed of inference. Besides, some researchers also
utilized a CTC Model to generate the preliminary predictions and then correct
the previous prediction by a non-autoregressive decoder [11, 12, 13]. However,
it will introduce some new problems. It is hard to eliminate accumulated error
caused by the CTC Models and carry out the frame-wise operation of the CTC
module in parallel. On the other hand, it is difficult for most of the non-
autoregressive transformer models to train and converge. To our knowledge,
there are three major ways to alleviate this problem. Firstly, some works try
to introduce an auxiliary CTC loss to accelerate the training and convergence
[12, 14]. Secondly, more iterations in the training process also bring the
improvement on the performance [10, 15], which is very simple and
straightforward. Thirdly, instead of learning from a sequence partially or
fully filled with <MASK> [10, 15], some works try to improve training
efficiency by providing a preliminary sequence with the information of target
sequence to the non-autoregressive transformer decoder [11, 12, 13, 14]. These
methods are either time-consuming or difficult to implement.

The traditional NAR model predicts the output tokens conditional on the
acoustic encoded states and a sequence that is completely empty and does not
contain any prior knowledge. In order to make the training match the
inference, the NAR model tries to learn from the sequence fully filled with
<MASK> in the training process, which hinders the improvement of training
speed and performance. To address this problem, inspired by the dual-mode ASR
(streaming and non-streaming) [16] and two-pass end-to-end models [17, 18,
19], we propose a model named two-step non-autoregressive transformer (TSNAT).
The TSNAT utilizes an AR model to accelerate the NAR model convergence and
improve the performance. Our model consists of an optional frond-end block, an
acoustic encoder, and a dual-mode transformer decoder. The dual-mode
transformer decoder can model the context in both an autoregressive and a non-
autoregressive way. Different from the traditional method that transfer
knowledge into the NAR model from a pre-trained AR model [20], we train an AR
model and a NAR model from scratch simultaneously. The NAR model can learn
some linguistic prior knowledge from the AR model depending on the dual-mode
transformer decoder. During the inference, our model can perform two-step
decoding without depending on the external attention decoder. All the
experiments are conducted on a public Chinese mandarin dataset ASIEHLL-1. The
results show that the TSNAT can achieve a competitive performance with the AR
model and outperform other NAR models.

(a) The Structure of Two-Step Non-Autoregressive Transformer

(b) The First-Step Inference Graph

Figure 1: (a) illustrates the structure of our proposed two-step non-
autoregressive transformer (TSNAT) and the inference process. The TSNAT model
consists of an optional front-end block, an acoustic encoder, and a dual-mode
transformer decoder. The inference can be divided into two-step, pre-selection
and rescoring. The dual-mode decoder first generates n-best hypothesizes in a
non-autoregressive and then rescores these hypotheses in an autoregressive
way. (b) illustrates an output probability graph of the non-autoregressive
model. The first red circle of each column means the end-of-sentence token
<EOS>. The other black circle represents different tokens of vocabulary. Each
path that starts with the blank circle and ends with a red circle in the graph
represents a possible hypothesis.

The remainder of this paper is organized as follows. Section 2 describes our
proposed model. Section 3 presents our experimental setup and results. The
conclusions will be given in Section 4.

##  2 Two-Step Non-Autoregressive Transformer

Our proposed two-step non-autoregressive transformer, as shown in Fig.1(a),
consists of three components, an optional convolutional front end block, an
acoustic encoder, and a dual-mode transformer decoder.

###  2.1 The Convolutional Front End and Acoustic Encoder

The convolutional front end block generally consists of two 2D-Convolution
layers and an output project layer [4]. The convolution layer is mainly
responsible for the processing and down-sampling of the low-level acoustic
features. The last output project layer will project the processed acoustic
feature into the dimension that the encoder required. The convolutional front
end block also can be replaced with the linear project layers with splicing
frame operation [21].

We utilize the transformer encoder [22, 4] as the acoustic encoder, which
contains a positional embedding, $N$ repetitive multi-head self-attention
layers (MHA), and feed-forward network layer (FFN). The sine-cosine positional
embedding proposed by [22] is applied for all the experiments in this paper.
Besides, the model also applies residual connection and layer normalization.

The multi-head self-attention layers depend on all contexts to compute the
attention weights from different perspectives, which makes it powerful to
model long-range temporal information.

$\begin{split}\text{SA}_{i}(\bm{Q},\bm{K},\bm{V})&=\text{Softmax}((\bm{Q}_{t})\bm{K}^{\top}/\sqrt{d_{k}})\bm{V}\\\
\text{MHA}(\bm{Q},\bm{K},\bm{V})&=\text{Concat}(SA_{0},SA_{1},...,SA_{H})\end{split}$
(1)

where $\bm{Q}$, $\bm{K}$ and $\bm{V}$ indicate the query, key, and value
matrix respectively. $d_{k}$ is the last dimension of $\bm{K}$. The multi-head
attention concatenates $H$ output vectors of the self-attention mechanism.
Besides, both the input of self-attention and the output of multi-head
attention should equip with an independent weight matrix respectively. The
feed-forward network (FFN) contains two linear layers and a gated liner unit
(GLU) [23] activation function [7].

$\text{FFN}(x)=\text{GLU}(xW_{1}+b_{1})W_{2}+b_{2}$ (2)

where parameters $W_{*}$ and $b_{*}$ are learnable weight matrix and bias
respectively.

###  2.2 The Dual-Mode Transformer Decoder

The major difference between the autoregressive model and the non-
autoregressive model focus on the structure of the decoder. The AR models
force themselves to learn the linguistic dependencies by blocking out the
future information. This characteristic makes the AR model predict the output
sequence step by step. The conditional probability $P_{AR}(Y|X)$ can be
expressed as:

$P_{AR}(Y|X)=P(y_{1}|X)\prod_{i=2}^{L}P(y_{i}|y_{<i},X)$ (3)

where $X$ is the acoustic encoded sequence with length $T$ and $y_{i}$ denotes
the $i$-th token of predicted sequence with length $L$. However, the non-
autoregressive models get rid of the dependencies on the previously predicted
tokens. Each step in the inference process is independent of each other, which
makes the inference step can be implemented in parallel and greatly improves
the reasoning speed of the model. The conditional probability $P_{NAR}(Y|X)$
can be rewritten as:

$P_{NAR}(Y|X)=\prod_{i=1}^{L}P(y_{i}|X)$ (4)

Both the AR decoder and the NAR decoder can apply the transformer as the basic
structure. There are two significant differences between these two kinds of
decoders. On the one hand, the AR transformer needs to apply the masking
operation to the previous output to model the linguistic dependencies, but the
NAR transformer needs to model the bidirectional dependencies between the
output tokens without any masking operations. On the other hand, the AR
decoder adopts all the tokens of vocabulary and some special tokens as the
modeling units, while the NAR decoder only requires one <MASK> token. Their
modeling units are not coincident.

Inspired by the dual-mode ASR [16], which models the streaming ASR and non-
streaming ASR by sharing an encoder, we propose a novel method that unifies
the AR and NAR decoder into one dual-mode transformer decoder (DMTD). We
assume that the AR decoder will accelerate the training and convergence of the
NAR decoder. The linguistic dependencies learned by the AR decoder might be
able to provide some prior knowledge for the NAR decoder, which will make the
training of the NAR decoder more efficient than guessing based on the empty
sequence. The self-attention of DMTD applies to the masking-optional
operation. During training, the encoder forwards once, and the decoder
forwards twice, once in autoregressive mode and once in non-autoregressive
mode. During the AR forward, the DMTD will adopt the truth token sequences
that begin with the begin-of-sentence token <BOS> as the input, and then mask
future information of self-attention and calculate the cross-entropy (CE) loss
$\mathcal{L}_{AR}$. For the NAR forward, the decoder will utilize a sequence
filled with <MASK> as the input and calculate the CE loss $\mathcal{L}_{NAR}$.
The final loss is equal to the weighted sum of $\mathcal{L}_{AR}$ and
$\mathcal{L}_{NAR}$.

$\mathcal{L}=(1-\alpha)\mathcal{L}_{NAR}+\alpha\mathcal{L}_{AR}$ (5)

where $\alpha$ dictates the weight of AR loss. The model is optimized by these
two losses jointly.

###  2.3 The Two Step Inference

Most of the non-autoregressive models just select the token which has the
highest probability at each position and concatenates them from left to right
as the final predicted result [10, 14, 15]. This inference method can be
regarded as a greedy search. As shown in Fig.1(b), we consider the conditional
probability matrix generated by the decoder as a graph, which contains
numerous possible hypothesize. Each hypothesis starts with any blank circle in
the first column and ends with a red circle (end-of-sentence token <EOS>). We
could select a better hypothesis than the one predicted by greedy search by
depending on the external language model or the second attention decoder as
[17, 19, 18]. Now that the dual-mode transformer decoder is able to be trained
in the AR and NAR fashion simultaneously, the decoder can perform dual-mode
inference naturally.

The inference process can be divided into two steps, pre-selection and
rescoring. During the first pre-selection, the dual-mode decoder will generate
the conditional probability matrix from a full-mask sequence in a NAR way and
then select the $N$-best hypothesizes. In order to avoid that the model tends
to choose shorter sentences, we utilize the length-normalized scores. The
selection of $N$-best hypothesizes from the probability graph contains only
simple addition operations, and does not take much time and computing
resources. During the second step, the dual-mode decoder will insert begin-of-
sentence token <BOS> into the head of the $N$-best hypothesizes. Then it
utilizes the processed candidates as the input and predict them in an AR
fashion. Due to the transformer can carry out efficient computing in parallel,
the entire inference process can be finished in two step.

##  3 Experiments and Results

###  3.1 Dataset

In this work, all experiments are conducted on a public Mandarin speech corpus
AISHELL-1111https://openslr.org/33/.The training set contains about 150 hours
of speech (120,098 utterances) recorded by 340 speakers. The development set
contains about 20 hours (14,326 utterances) recorded by 40 speakers. And about
10 hours (7,176 utterances / 36109 seconds) of speech is used as the test set.
The speakers of different sets are not overlapped.

###  3.2 Experimental Setup

For all experiments, we use 80-dimensional FBANK features computed on a 25ms
window with a 10ms shift. We choose 4234 characters (including a padding
symbol <PAD>, an unknown token <UNK>, a begin-of-sentence token <BOS>) and an
end-of-sentence token <EOS> as modeling units.

Our model consists of 12 encoder blocks and 6 decoder blocks. There are 4
heads in multi-head attention. The 2D convolution front end utilizes two-layer
time-axis CNN with ReLU activation, stride size 2, channels 384, and kernel
size 3. Both the output size of the multi-head attention and the feed-forward
layers are 384. The hidden size of the feed-forward layers is 768. We adopt an
Adam optimizer with warmup steps 12000 and the learning rate scheduler
reported in [22]. After 100 epochs, we average the parameters saved in the
last 20 epochs. We also use the time mask and frequency mask method proposed
in [24] instead of speed perturbation.

We use the character error rate (CER) to evaluate the performance of different
models. For evaluating the inference speed of different models, we decode
utterances one by one to compute real-time factor (RTF) on the test set. The
RTF is the time taken to decode one second of speech. All experiments are
conducted on a GeForce GTX TITAN X 12G GPU.

###  3.3 Results

####  3.3.1 Comparison of The Model With Different AR Weights $\alpha$

This section compares the models with the different AR weights $\alpha$. When
the weight $\alpha$ is equal to 0, the NAR model is equivalent to the one that
adopts an empty sequence as the input without any prior knowledge. If $\alpha$
is equal to 1, the model will be completely transformed into an AR model. When
the weight is between 0 and 1, the model can perform the two-step inference.
For all one-step inference in this section (marked as OneStep), the NAR models
($\alpha\in[0,1)$) apply greedy search and the AR ($\alpha=1.0$) models apply
the beam search with width 10. The two-step inference of this section (marked
as TwoStep) rescores the 10-best candidate sequences. As shown in Table 1,
it’s obvious that the hybrid AR and NAR training ($\alpha\in(0,1)$) can
improve the performance of the NAR model. We guess that the NAR decoder can
get some prior knowledge from the AR decoder by sharing the parameters, which
reduces the difficulty of training the NAR model from scratch. In the case of
one-step inference, there is still a great performance gap between the AR
model($\alpha=1.0$) and the NAR model ($\alpha\in[0,1)$). However, the
introduced two-step inference method can improve the performance of the NAR
model greatly and achieve comparable results with the AR model.

Table 1: Comparison of the model with different AR weights $\alpha$ (CER %). Weight $\alpha$ | 0.0 | 0.3 | 0.5 | 0.7 | 0.9 | 1.0
---|---|---|---|---|---|---
Dev | OneStep | 6.5 | 5.9 | 5.8 | 5.8 | 6.5 | 5.3
TwoStep | - | 5.6 | 5.5 | 5.4 | 5.6 | -
Test | OneStep | 7.2 | 6.5 | 6.5 | 6.4 | 6.5 | 6.0
TwoStep | - | 6.2 | 6.1 | 6.0 | 6.2 | -

####  3.3.2 The influence of the $N$-best Sequences on The Model Performance

This section pays attention to the inference of the $N$-best sequence on the
NAR model performance. We select the different number of candidate sequences.
Under the condition that $N$ is equal to 1, the two-step inference makes no
sense. Therefore, we replace it with the result of the greedy search. The
results in Table 2 indicate that the more candidate sequences, the better
performance it archives, which is consistent with the results previously
reported [19]. When the number of candidate sequences is greater than 10, the
performance tends to be stable. However, with the increase of $N$, the model
requires more computing resources, which leads to an increase in latency and
real-time-factor (RTF). Taken together, we will select the 10-best candidates
for the second-step rescoring by balancing the model performance and the
inference speed.

Table 2: Comparison of the influence of the $N$-best sequences on the model performance (CER %). $N$ | 1 | 5 | 10 | 20 | 50
---|---|---|---|---|---
Dev | 5.8 | 5.4 | 5.4 | 5.4 | 5.3
Test | 6.4 | 6.0 | 5.9 | 5.9 | 5.9
RTF | 0.0054 | 0.0167 | 0.0173 | 0.0181 | 0.0220

####  3.3.3 Comparison with Other Non-Autoregressive Models

To further study the upper limit of model performance, we readjust the
parameter configuration of the model and named three models of different
sizes, named TSNAT-Small, TSNAT-Middle and TSNAT-Big. The three models have
the same depth (12 encoder blocks and 6 decoder blocks) and are trained under
the same conditions. Their differences focus on the dimensions of the model
($d_{model}$) and the hidden size of the feed-forward network ($d_{ff}$), as
shown in Table 3.

Table 3: The Parameter Configuration of The Model. Mode | TSNAT-Small | TSNAT-Middle | TSNAT-Big
---|---|---|---
$d_{model}$ | 384 | 512 | 512
$d_{ff}$ | 384 | 512 | 512
#Params | 34M | 59M | 87M
Table 4: Comparison with other non-autoregressive models (CER %). All models
in this table use SpecAugment to improve the performance.

Model | LM | Dev | Test | RTF
---|---|---|---|---
A-FMLM(K=1) [10] | w/o | 6.2 | 6.7 | -
Insertion-NAT [25] | w/o | 6.1 | 6.7 | -
LASO-big [15] $\lozenge$ | w/o | 5.8 | 6.4 | -
CASS-NAT [26] $\lozenge$ | w | 5.3 | 5.8 | -
CTC-enhanced NAR [13] $\lozenge$ | w/o | 5.3 | 5.9 | -
ST-NAT [14] | w/o | 6.9 | 7.7 | 0.0056
ST-NAT [14] | w | 6.4 | 7.0 | 0.0292
TSNAT-Small (34M) | w/o | 5.8 | 6.4 | 0.0054
\+ Two Step Inference | w/o | 5.4 | 5.9 | 0.0173
TSNAT-Middle (59M) | w/o | 5.4 | 6.0 | 0.0063
\+ Two Step Inference | w/o | 5.2 | 5.7 | 0.0176
TSNAT-Big (87M) | w/o | 5.3 | 6.0 | 0.0077
\+ Two Step Inference | w/o | 5.1 | 5.6 | 0.0185

  * $\lozenge$

These models additionally use speed-perturb to augment the speech data.

Compared with the other non-autoregressive models, we proposed TSNAT model can
achieve quite competitive performance. To our best knowledge, the TSNAT-Big
with two-step inference achieves the state-of-the-art (SOTA) performance
without depending on any external language models. The results also show that
the two-step inference is faster than ST-NAT with a neural language model
because the two-step inference of the dual-mode transformer decoder can be
implemented in parallel without iteration step by step.

####  3.3.4 The Analysis of Attention Weights in The Dual-Mode Decoder

(a) Self-Attention of NAR Mode

(b) Self-Attention of AR Mode

(c) EncDec-Attention of NAR Mode

(d) EncDec-Attention of AR Mode

Figure 2: The Visualization of Attention Weights in The Dual-Mode Decoder. We
extract the self-attention weights and encoder-decoder attention weights from
the last block of the dual-mode transformer decoder. The pictures come from
the test set, and its corresponding sentence ID is ’BAC009S0764W0121’.

We want to find out the difference between the AR and NAR modes of the dual-
mode transformer decoder. Therefore, we extract the attention weights of these
two modes from the last decoder block. It’s clear that the self-attention
mechanism of the NAR model focuses on the whole input sequence, while the
self-attention mechanism of the AR mode pays attention to the previous
sequence. This is consistent with their own characteristics. Besides, the
encoder-decoder attention weights of the NAR mode are more dispersed than the
one under the AR mode, which also shows the autoregressive model has stronger
modeling ability.

##  4 Conclusions

Compared with the autoregressive model for speech recognition, the non-
autoregressive model can get rid of the temporal dependency and predict the
entire tokens in at least one step. However, the non-autoregressive model
still faces two problems. On the one hand, there is still a great performance
gap between the advanced autoregressive (AR) model and the non-autoregressive
(NAR) transformer model. On the other hand, it’s difficult for most of the
non-autoregressive models to train and converge. In this paper, we try to
address these problems from two aspects. Firstly, we propose a parameters-
sharing training method to improve the performance of the NAR model by
learning some valuable knowledge from the AR model. Then we make full use of
the dual-mode decoder by introducing the two-step inference method. We conduct
our experiments on a public Chinese mandarin dataset ASIEHLL-1. The results
show that our proposed model can achieve comparable performance with the AR
model and outperform other typical NAR models.

## References

  * [1] C. J. K, B. Dzmitry, S. Dmitriy, C. Kyunghyun, and B. Yoshua, “Attention-based models for speech recognition,” in _Advances in neural information processing systems_ , 2015, pp. 577–585.
  * [2] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 4960–4964.
  * [3] K. Suyoun, H. Takaaki, and W. Shinji, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 4835–4839.
  * [4] D. Linhao, X. Shuang, and X. Bo, “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5884–5888.
  * [5] A. Graves, “Sequence transduction with recurrent neural networks,” _arXiv preprint arXiv:1211.3711_ , 2012.
  * [6] Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang _et al._ , “Streaming end-to-end speech recognition for mobile devices,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6381–6385.
  * [7] Z. Tian, J. Yi, J. Tao, Y. Bai, and Z. Wen, “Self-Attention Transducers for End-to-End Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 4395–4399.
  * [8] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7829–7833.
  * [9] C.-F. Yeh, J. Mahadeokar, K. Kalgaonkar, Y. Wang, D. Le, M. Jain, K. Schubert, C. Fuegen, and M. L. Seltzer, “Transformer-transducer: End-to-end speech recognition with self-attention,” _arXiv preprint arXiv:1910.12977_ , 2019\.
  * [10] N. Chen, S. Watanabe, J. Villalba, and N. Dehak, “Non-autoregressive transformer automatic speech recognition,” _arXiv preprint arXiv:1911.04908_ , 2019.
  * [11] Y. Higuchi, S. Watanabe, N. Chen, T. Ogawa, and T. Kobayashi, “Mask ctc: Non-autoregressive end-to-end asr with ctc and mask predict,” _arXiv preprint arXiv:2005.08700_ , 2020.
  * [12] Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa, and T. Kobayashi, “Improved mask-ctc for non-autoregressive end-to-end asr,” _arXiv preprint arXiv:2010.13270_ , 2020.
  * [13] X. Song, Z. Wu, Y. Huang, C. Weng, D. Su, and H. Meng, “Non-autoregressive transformer asr with ctc-enhanced decoder input,” _arXiv preprint arXiv:2010.15025_ , 2020.
  * [14] Z. Tian, J. Yi, J. Tao, Y. Bai, S. Zhang, and Z. Wen, “Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition,” in _Proc. Interspeech 2020_ , 2020, pp. 5026–5030. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-2086
  * [15] Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen, and S. Zhang, “Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition,” in _Proc. Interspeech 2020_ , 2020, pp. 3381–3385. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-1600
  * [16] J. Yu, W. Han, A. Gulati, C.-C. Chiu, B. Li, T. N. Sainath, Y. Wu, and R. Pang, “Dual-mode asr: Unify and improve streaming asr with full-context modeling,” _Proceedings of ICLR_ , 2021.
  * [17] T. N. Sainath, R. Pang, D. Rybach, Y. He, R. Prabhavalkar, W. Li, M. Visontai, Q. Liang, T. Strohman, Y. Wu _et al._ , “Two-pass end-to-end speech recognition,” _arXiv preprint arXiv:1908.10992_ , 2019.
  * [18] K. Hu, T. N. Sainath, R. Pang, and R. Prabhavalkar, “Deliberation model based two-pass end-to-end speech recognition,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7799–7803.
  * [19] Z. Tian, J. Yi, Y. Bai, J. Tao, S. Zhang, and Z. Wen, “One in a hundred: Select the best predicted sequence from numerous candidates for streaming speech recognition,” _arXiv preprint arXiv:2010.14791_ , 2020.
  * [20] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: Fast, robust and controllable text to speech,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 3171–3180.
  * [21] Y. zhao, J. Li, X. Wang, and Y. Li, “The speechtransformer for large-scale mandarin chinese speech recognition,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 7095–7099.
  * [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [23] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 933–941.
  * [24] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.
  * [25] Y. Fujita, S. Watanabe, M. Omachi, and X. Chan, “Insertion-based modeling for end-to-end automatic speech recognition,” _arXiv preprint arXiv:2005.13211_ , 2020.
  * [26] R. Fan, W. Chu, P. Chang, and J. Xiao, “Cass-nat: Ctc alignment-based single step non-autoregressive transformer for speech recognition,” _arXiv preprint arXiv:2010.14725_ , 2020.