# A Nesterov type algorithm with double Tikhonov regularization: fast
convergence of the function values and strong convergence to the minimal norm
solution

Mikhail Karapetyants University of Vienna, Faculty of Mathematics, Oskar-
Morgenstern-Platz 1, A-1090 Vienna, Austria, email:
<EMAIL_ADDRESS>Research done during the visit of the
author at Technical University of Cluj-Napoca. Szilárd Csaba László Technical
University of Cluj-Napoca, Department of Mathematics, Memorandumului 28, Cluj-
Napoca, Romania, e-mail<EMAIL_ADDRESS>This work was
supported by a grant of the Ministry of Research, Innovation and Digitization,
CNCS-UEFISCDI, project number PN-III-P1-1.1-TE-2021-0138, within PNCDI III.

Abstract. We investigate the strong convergence properties of a Nesterov type
algorithm with two Tikhonov regularization terms in connection to the
minimization problem of a smooth convex function $f.$ We show that the
generated sequences converge strongly to the minimal norm element from
$\operatorname*{argmin}f$. We also show that from a practical point of view
the Tikhonov regularization does not affect Nesterov’s optimal convergence
rate of order $\mathcal{O}(n^{-2})$ for the potential energies $f(x_{n})-\min
f$ and $f(y_{n})-\min f$, where $(x_{n}),\,(y_{n})$ are the sequences
generated by our algorithm. Further, we obtain fast convergence to zero of the
discrete velocity, but also some estimates concerning the value of the
gradient of the objective function in the generated sequences.

Key Words. inertial algorithm, convex optimization, Tikhonov regularization,
strong convergence

AMS subject classification. 34G25, 47J25, 47H05, 90C26, 90C30, 65K10

##  1 Introduction

Let $\mathcal{H}$ be a Hilbert space endowed with the scalar product
$\langle\cdot,\cdot\rangle$ and norm $\|\cdot\|$ and consider the optimization
problem

$\inf_{x\in\mathcal{H}}f(x)$ (1)

where $f:\mathcal{H}\longrightarrow\mathbb{R}$ is a convex, continuously
Fréchet differentiable function, with $L$-Lipschitz continuous gradient, whose
set of minimizers $\operatorname*{argmin}f$ is nonempty.

We associate to the optimization problem (1) the following inertial-gradient
type algorithm. Let $x_{0},x_{1}\in\mathcal{H}$ and for all $k\geq 1$ set

$\left\\{\begin{array}[]{lll}y_{k}=x_{k}+b_{k-1}(x_{k}-x_{k-1})-c_{k}x_{k}\\\
x_{k+1}=y_{k}-s{\nabla}f(y_{k})-s\epsilon_{k}y_{k}.\end{array}\right.$ (2)

We assume that $s\in\left(0,\frac{1}{L}\right)$ and the sequence
$(c_{k})_{k\geq 1}$ is nonnegative for $k$ big enough and satisfies
$\lim_{k\to+\infty}c_{k}=0$. Further, we assume that $(\epsilon_{k})_{k\geq
1}$ is a non-increasing positive sequence that satisfies
$\lim_{k\to+\infty}\epsilon_{k}=0.$ Observe that in case the inertial
parameter $(b_{k})_{k\geq 0}$ satisfies $\lim_{k\to+\infty}b_{k}=1,$ then
Algorithm (2) has the form of the famous Nesterov algorithm, (see [22, 15] and
also [11, 18]), with two Tikhonov regularization terms. Indeed, the terms
$c_{k}x_{k}$ and $\epsilon_{k}y_{k}$ in Algorithm (2) play the role of
Tikhonov regularization terms, consequently our aim is to obtain the strong
convergence of the generated sequences to the element of minimal norm from
$\operatorname*{argmin}f,$ (see [1, 3, 6, 8, 9, 10, 12, 14, 16, 17, 19, 20,
25, 26]) and at the same time to preserve the optimality of Nesterov algorithm
concerning the convergence rate of order $\mathcal{O}(k^{-2})$ for the
potential energy $f(x_{k})-\min f$, (see [22, 15] ). Our analysis reveals that
the inertial parameter and the Tikhonov regularization parameters are strongly
correlated. This fact is in concordance with some recent results from the
literature concerning the strong convergence of the trajectories of some
continuous second order dynamical systems to a minimal norm minimizer of a
convex function or to the minimal norm zero of a maximally monotone operator
[1, 2, 6, 7, 10, 12, 13, 14, 19]. Concerning the discrete case, that is, the
case of inertial algorithms that converge strongly to the minimal norm
solution of a convex optimization problem, there are only few results in the
literature, see [10, 20] and also those refer to proximal inertial algorithms
obtained via implicit discretizations of some second order continuous
dynamical systems, (see [24, 19]).

Indeed, in [10] the following inertial-proximal algorithm was considered in
connection to the optimization problem (1):
$x_{0},x_{1}\in\mathcal{H},\,x_{k+1}={\rm
prox}_{f}\left(x_{k}+\left(1-\frac{\alpha}{k}\right)(x_{k}-x_{k-1})-\frac{c}{k^{2}}x_{k}\right),$
where $\alpha>3,\,c>0$ and
$\operatorname*{prox}\nolimits_{f}:\mathcal{H}\rightarrow\mathcal{H},\quad\operatorname*{prox}\nolimits_{f}(x)=\operatorname*{argmin}_{y\in\mathcal{H}}\left(f(y)+\frac{1}{2}\|y-x\|^{2}\right),$
denotes the proximal point operator of the convex function $f$. Due to our
best knowledge this is the first inertial algorithm in the literature for
which both strong convergence results for the generated sequences and fast
convergence of the potential energy $f(x_{k})-\min f$ and discrete velocity
$\|x_{k}-x_{k-1}\|$ were obtained. However, from practical point of view, it
is not natural that the minimizers of a smooth function to be approximated via
proximal, i.e. backward, steps. Another drawback of this algorithm is that
does not assure the full strong convergence of the generated sequences to the
minimum norm minimizer $x^{*}$. Indeed, according to [10] only the strong
convergence result $\liminf_{k\to+\infty}\|x_{k}-x^{*}\|=0$ is provided. In
order to overcome these deficiencies in [20] the author assumed that the
objective function in (1) is proper, convex and lower semicontinuous only and
associated to this optimization problem the following inertial-proximal
algorithm: $x_{0},x_{1}\in\mathcal{H},\,x_{k+1}={\rm
prox}_{\lambda_{k}f}\left(x_{k}+\left(1-\frac{\alpha}{k^{q}}\right)(x_{k}-x_{k-1})-\frac{c}{k^{p}}x_{k}\right),$
where $\alpha,\,q,\,c,\,p>0$ and $(\lambda_{k})$ is a sequence of positive
real numbers. According to [20], in case the stepsize $\lambda_{k}\equiv 1$
and $0<q<1,\,1<p<q+1$ the full convergence of the generated sequences to the
minimum norm minimizer $x^{*}$ is obtained, i.e.
$\lim_{k\to+\infty}\|x_{k}-x^{*}\|=0$. Further, the fast convergence of the
potential energy $f(x_{k})-\min f$ and discrete velocity $\|x_{k}-x_{k-1}\|$
were shown.

In concordance to the results emphasized above, the main goal of this paper is
to obtain similar results for gradient type inertial algorithms. Unfortunately
our parameters in Algorithm (2) will not have such simple forms as the
parameters in [10] or [20] and this is due to the fact that we cannot use a
discrete Lyapunov function of similar form as the ones considered in [10, 20],
instead we have to construct a new discrete Lyapunov function suitable for our
analysis. Therefore, the forms of the given sequences $(b_{k})_{k\geq
0},\,(c_{k})_{k\geq 1}$ are crucial in order to obtain our results. More
precisely, for given Tikhonov regularization parameter $(\epsilon_{k})_{k\geq
1}$ and a fixed stepsize $s$ consider the sequence $(q_{k})_{k\geq 0}$ which
after an index $k$ big enough, satisfies

$(Q)\,\,\,\,(1-s\epsilon_{k+1})^{2}q_{k+1}^{2}-(1-s\epsilon_{k})^{2}q_{k}^{2}-2sq_{k+1}+s(1-s\epsilon_{k})^{2}q_{k}\leq
0,\,q_{k}\geq\frac{2s}{(1-s\epsilon_{k})^{2}}.$

Then, the inertial parameter $(b_{k})_{k\geq 0}$ and the regularization
parameter $(c_{k})_{k\geq 1}$ from Algorithm (2) are defined via the
conditions

$(B)\,\,\,\left\\{\begin{array}[]{lll}b_{k-1}=0,\mbox{ if }k=1\mbox{ or
}(1-s\epsilon_{k-1})(1-s\epsilon_{k})q_{k-1}q_{k}=0\\\ \\\ \displaystyle
b_{k-1}=\frac{(q_{k-1}-s)((1-s\epsilon_{k-1})^{2}q_{k-1}-2s)}{(1-s\epsilon_{k-1})(1-s\epsilon_{k})q_{k-1}q_{k}},\mbox{
otherwise}\end{array}\right.$

and

$(C)\,\,\,\left\\{\begin{array}[]{lll}c_{k}=0,\mbox{ if }k=1\mbox{ or
}(1-s\epsilon_{k-1})(1-s\epsilon_{k})q_{k-1}q_{k}=0\\\ \\\ \displaystyle
c_{k}=\frac{2s}{(1-s\epsilon_{k-1})(1-s\epsilon_{k})^{2}q_{k}}\left(\frac{s}{q_{k-1}}-\frac{s^{2}\epsilon_{k}}{q_{k-1}}-s(\epsilon_{k-1}-\epsilon_{k})\right),\mbox{
otherwise}.\end{array}\right.$

Note that despite of the complex form of these parameters, from a practical
perspective, Algorithm (2) can easily be implemented.

A comprehensive analysis of the above conditions will be carried out in
section 4. Here we just underline that in case we specify the parameters as
$\epsilon_{k}=\frac{c}{k^{p}},\,c,p>0$ and we take
$q_{k}=ak^{q},\,a>0,\,0<q<1$ then (Q) is satisfied for every fixed stepsize
$s\in\left(0,\frac{1}{L}\right)$ and the main result of the paper can be
summarized in the following theorem.

######  Theorem 1.

For $p<2q$ let $(x_{k})_{k\geq 0}$ and $(y_{k})_{k\geq 1}$ be the sequences
generated by Algorithm (2). Then, $(x_{k})$ and $(y_{k})$ converge strongly to
$x^{*}$, where $\\{x^{*}\\}=\operatorname*{pr}_{\operatorname*{argmin}f}(0)$
is the minimum norm minimizer of our objective function $f.$ Further,
$f(x_{k})-\min f=\mathcal{O}\left(k^{-p}\right),\mbox{ as }k\to+\infty,\mbox{
and }f(y_{k})-\min f=\mathcal{O}\left(k^{-p}\right),\mbox{ as }k\to+\infty.$
Additionally, $\|{\nabla}f(x_{k})\|=\ o\left(k^{-\frac{p}{2}}\right)\mbox{ as
}k\to+\infty,\,\|{\nabla}f(y_{k})\|=\ o\left(k^{-\frac{p}{2}}\right)\mbox{ as
}k\to+\infty\mbox{ and }\|x_{k}-x_{k-1}\|\ =\
o\left(k^{-\frac{p}{2}}\right)\mbox{ as }k\to+\infty.$

The paper is organized as follows. In the next section we present some
preliminary results and notions that we need to carry out our analysis. In
section 3 we prove the main result of the paper. We obtain strong convergence
of the sequences generated by Algorithm (2) and also fast convergence of the
potential energy and discrete velocity. In section 4 we consider the
parameters in a simple form and discuss the conditions these parameters must
satisfy in order to obtain the results presented at section 3. Further, in
section 5 via some numerical experiments we show that Algorithm (2) indeed
assures the convergence of the generated sequences to a minimal norm solution
and also that both Tikhonov regularization terms are indispensable in order to
obtain this result. Finally, we conclude our paper with some future research
plans.

##  2 Preliminary results

In order to obtain strong convergence for the sequence $(x_{k})$ generated by
Algorithm (2) we need some preliminary results. The first one is the Descent
Lemma [21].

######  Lemma 2.

Let $f:\mathcal{H}\longrightarrow\mathbb{R}$ be a smooth function, with
$L-$Lipschitz continuous gradient. Then,

$f(x)\leq f(y)+\langle{\nabla}f(y),x-y\rangle+\frac{L}{2}\|y-x\|^{2},\,\forall
x,y\in\mathcal{H}.$

Further, we need the following property of smooth, convex functions, see [21].

######  Lemma 3.

Let $f:\mathcal{H}\longrightarrow\mathbb{R}$ be a convex smooth function, with
$L-$Lipschitz continuous gradient. Then,

$\frac{1}{2L}\|{\nabla}f(y)-{\nabla}f(x)\|^{2}+\langle{\nabla}f(y),x-y\rangle+f(y)\leq
f(x),\mbox{ for all }x,y\in\mathcal{H}.$

Our first original result is a modified descent lemma, which in particular
contains Lemma 1 from [5], however has a considerable simplified proof.

######  Lemma 4.

Let $f:\mathcal{H}\longrightarrow\mathbb{R}$ be a convex smooth function, with
$L-$Lipschitz continuous gradient and let $s>0.$ Then,

$f(y-s{\nabla}f(y))\leq
f(x)+\langle{\nabla}f(y),y-x\rangle+\left(\frac{L}{2}s^{2}-s\right)\|{\nabla}f(y)\|^{2}-\frac{1}{2L}\|{\nabla}f(y)-{\nabla}f(x)\|^{2},\,\forall
x,y\in\mathcal{H}.$ (3)

Assume further that $s\in\left(0,\frac{1}{L}\right].$ Then,

$f(y-s{\nabla}f(y))\leq
f(x)+\langle{\nabla}f(y),y-x\rangle-\frac{s}{2}\|{\nabla}f(y)\|^{2}-\frac{s}{2}\|{\nabla}f(y)-{\nabla}f(x)\|^{2},\,\forall
x,y\in\mathcal{H}.$ (4)

###### Proof.

Indeed, by taking $x=y-s{\nabla}f(y)$ in Lemma 2, we get

$f(y-s{\nabla}f(y))\leq
f(y)+\left(\frac{L}{2}s^{2}-s\right)\|{\nabla}f(y)\|^{2},\,\forall
y\in\mathcal{H}.$ (5)

From Lemma 3 we have

$f(y)\leq
f(x)+\langle{\nabla}f(y),y-x\rangle-\frac{1}{2L}\|{\nabla}f(y)-{\nabla}f(x)\|^{2},\,\forall
x,y\in\mathcal{H}.$ (6)

Combining (5) and (6) we get

$f(y-s{\nabla}f(y))\leq
f(x)+\langle{\nabla}f(y),y-x\rangle+\left(\frac{L}{2}s^{2}-s\right)\|{\nabla}f(y)\|^{2}-\frac{1}{2L}\|{\nabla}f(y)-{\nabla}f(x)\|^{2},\,\forall
x,y\in\mathcal{H},$

which is nothing else that (3).

Assume that $0<s\leq\frac{1}{L}.$ Then,

$\frac{L}{2}s^{2}-s\leq-\frac{s}{2}\mbox{ and }-\frac{1}{2L}\leq-\frac{s}{2},$

hence (3) leads to (4), that is

$f(y-s{\nabla}f(y))\leq
f(x)+\langle{\nabla}f(y),y-x\rangle-\frac{s}{2}\|{\nabla}f(y)\|^{2}-\frac{s}{2}\|{\nabla}f(y)-{\nabla}f(x)\|^{2},\,\forall
x,y\in\mathcal{H}.$

∎

We continue the present section by emphasizing the main idea behind the
Tikhonov regularization, which will assure strong convergence results for the
sequence generated our algorithm (2) to a minimizer of minimal norm of the
objective function $f$. By $\overline{x}_{k}$ we denote the unique solution of
the strongly convex minimization problem

$\displaystyle\min_{x\in\mathcal{H}}\left(f(x)+\frac{\epsilon_{k}}{2}\|x\|^{2}\right).$

We know, (see for instance [8]), that
$\lim\limits_{k\to+\infty}\overline{x}_{k}=x^{\ast}$, where
$x^{\ast}=\operatorname*{argmin}\limits_{x\in\operatorname*{argmin}f}\|x\|$ is
the minimal norm element from the set $\operatorname*{argmin}f.$ Obviously,
$\\{x^{*}\\}=\operatorname*{pr}_{\operatorname*{argmin}f}0$ and we have the
inequality $\|\overline{x}_{k}\|\leq\|x^{\ast}\|$ (see [12]).

Since $\overline{x}_{k}$ is the unique minimizer of the strongly convex
function $f_{k}(x)=f(x)+\frac{\epsilon_{k}}{2}\|x\|^{2},$ obviously one has

${\nabla}f_{k}(\overline{x}_{k})={\nabla}f(\overline{x}_{k})+\epsilon_{k}\overline{x}_{k}=0.$
(7)

Further, from Lemma A.1 c) from [20] we have

$\left\|\overline{x}_{k+1}-\overline{x}_{k}\right\|\leq\min\left(\frac{\epsilon_{k}-\epsilon_{k+1}}{\epsilon_{k+1}}\|\overline{x}_{k}\|,\frac{\epsilon_{k}-\epsilon_{k+1}}{\epsilon_{k}}\|\overline{x}_{k+1}\|\right).$
(8)

Note that since $f_{k}$ is strongly convex, from the gradient inequality we
have

$f_{k}(y)-f_{k}(x)\geq\langle{\nabla}f_{k}(x),y-x\rangle+\frac{\epsilon_{k}}{2}\|x-y\|^{2},\mbox{
for all }x,y\in\mathcal{H}.$ (9)

In particular

$f_{k}(x)-f_{k}(\overline{x}_{k})\geq\frac{\epsilon_{k}}{2}\|x-\overline{x}_{k}\|^{2},\mbox{
for all }x\in\mathcal{H}.$ (10)

Moreover, observe that for all $x,y\in\mathcal{H}$, one has

$f(x)-f(y)=(f_{k}(x)-f_{k}(\overline{x}_{k}))+(f_{k}(\overline{x}_{k})-f_{k}(y))+\frac{\epsilon_{k}}{2}(\|y\|^{2}-\|x\|^{2})\leq
f_{k}(x)-f_{k}(\overline{x}_{k})+\frac{\epsilon_{k}}{2}\|y\|^{2}.$ (11)

Note that ${\nabla}f_{k}(x)={\nabla}f(x)+\epsilon_{k}x$, consequently if
${\nabla}f$ is L-Lipschitz continuous then the Lipschitz constant of the
gradient of $f_{k}$ is $L+\epsilon_{k}.$

Hence, if we apply Lemma 4 to $f_{k}$ we get that for all
$s\in\left(0,\frac{1}{L+\epsilon_{k}}\right]$ one has

$f_{k}(y-s{\nabla}f_{k}(y))\leq
f_{k}(x)+\langle{\nabla}f_{k}(y),y-x\rangle-\frac{s}{2}\|{\nabla}f_{k}(y)\|^{2}-\frac{s}{2}\|{\nabla}f_{k}(y)-{\nabla}f_{k}(x)\|^{2},\,\forall
x,y\in\mathcal{H}.$ (12)

Now, we can rewrite Algorithm (2) in a more convenable equivalent form, by
using the strongly convex function $f_{k}.$ Indeed, since
${\nabla}f_{k}(x)={\nabla}f(x)+\epsilon_{k}x$, Algorithm (2) can equivalently
be written as: $x_{0},x_{1}\in\mathcal{H}$ and for all $k\geq 1$

$\left\\{\begin{array}[]{lll}y_{k}=x_{k}+b_{k-1}(x_{k}-x_{k-1})-c_{k}x_{k}\\\
x_{k+1}=y_{k}-s{\nabla}f_{k}(y_{k}).\end{array}\right.$ (13)

##  3 Strong convergence

In this section we provide sufficient conditions such that the sequences
generated by (13) converge strongly to the minimum norm minimizer of $f$ and
at the same time fast convergence of the function values in the generated
sequences and also fast convergence of the discrete velocity to zero are
obtained. Moreover, we also show some pointwise estimates for the gradient of
the objective function.

In order to obtain our general result concerning the strong convergence of the
sequences generated by the algorithm (13) we need to use (12), hence we adjust
the indexes in algorithm (13) as follows.

Assume that $0<s<\frac{1}{L}$ and let $k_{0}\in\mathbb{N}$ such that the
following assumption holds.

$(S)\,\,\,\,s\in\left(0,\frac{1}{L+\epsilon_{k_{0}}}\right]\subseteq\left(0,\frac{1}{L+\epsilon_{k}}\right],\mbox{
for all }k\geq k_{0}.$

Note that such index $k_{0}$ exists, since $\epsilon_{k}$ is nonincreasing and
$\epsilon_{k}\to 0$ as $k\to+\infty.$
Further, since $\epsilon_{k}$ is nonincreasing there exists $k_{1}\geq k_{0}$
such that $1-s\epsilon_{k}>0$ for all $k\geq k_{1}.$

Consider the sequence $(q_{k})_{k\geq 0}$ which after an index $k_{2}\geq
k_{1}$, satisfies the condition (Q), that is

$(Q)\,\,\,\,(1-s\epsilon_{k+1})^{2}q_{k+1}^{2}-(1-s\epsilon_{k})^{2}q_{k}^{2}-2sq_{k+1}+s(1-s\epsilon_{k})^{2}q_{k}\leq
0,\,q_{k}\geq\frac{2s}{(1-s\epsilon_{k})^{2}}$

for all $k\geq k_{2}.$ Note that $q_{k}\geq 2s>0$ for all $k\geq k_{2}.$

Let $\overline{k}=k_{2}+1$ and observe that
$(1-s\epsilon_{k-1})(1-s\epsilon_{k})q_{k-1}q_{k}>0$ for all
$k\geq\overline{k}$, consequently the sequences $b_{k}$ and $c_{k}$ defined at
(B) and (C) have the following forms:

$b_{k-1}=\frac{(q_{k-1}-s)((1-s\epsilon_{k-1})^{2}q_{k-1}-2s)}{(1-s\epsilon_{k-1})(1-s\epsilon_{k})q_{k-1}q_{k}},\mbox{
for all }k\geq\overline{k}$

and

$c_{k}=\frac{2s}{(1-s\epsilon_{k-1})(1-s\epsilon_{k})^{2}q_{k}}\left(\frac{s}{q_{k-1}}-\frac{s^{2}\epsilon_{k}}{q_{k-1}}-s(\epsilon_{k-1}-\epsilon_{k})\right),\mbox{
for all }k\geq\overline{k}.$

The following general result holds.

######  Theorem 5.

For a sequence $(q_{k})$ satisfying (Q) and the stepsize $s$ satisfying (S),
consider the sequences $(b_{k})_{k\geq 0}$ and $(c_{k})_{k\geq 1}$ defined at
(B) and (C) and let $(x_{k})_{k\geq 0},\,(y_{k})_{k\geq 1}$ be the sequences
generated by Algorithm (2). Assume that the sequence
$\left(\frac{q_{k}\epsilon_{k}}{q_{k-1}\epsilon_{k-1}}\right)_{k\geq\overline{k}}$
is bounded, the sequence $(q_{k}^{2}\epsilon_{k})_{k\geq\overline{k}-1}$ is
increasing, further $\lim_{k\to+\infty}q_{k}^{2}\epsilon_{k}=+\infty$ and
$\lim_{k\to+\infty}\frac{q_{k}(\epsilon_{k}-\epsilon_{k+1})}{\epsilon_{k}}=0$.

Then, $(x_{k})$ converges strongly to $x^{*}$, where
$\\{x^{*}\\}=\operatorname*{pr}_{\operatorname*{argmin}f}(0)$ is the minimum
norm minimizer of our objective function $f.$ Moreover $\|x_{k}-y_{k}\|\ =\
o\left(\sqrt{\epsilon_{k}}\right)\mbox{ as }k\to+\infty,$ hence $(y_{k})$ also
converges strongly to $x^{*}.$

Further, the following estimates hold.

$f_{k}(x_{k})-f_{k}(\overline{x}_{k})=o(\epsilon_{k})\mbox{ as }k\to+\infty,$
$f(x_{k})-\min f=\mathcal{O}\left(\epsilon_{k}\right),\mbox{ as
}k\to+\infty\mbox{ and }f(y_{k})-\min
f=\mathcal{O}\left(\epsilon_{k}\right),\mbox{ as }k\to+\infty,$
$\|x_{k}-x_{k-1}\|\ =\ o\left(\sqrt{\epsilon_{k}}\right)\mbox{ as
}k\to+\infty,$

and

$\|{\nabla}f(x_{k})\|=\ o\left(\sqrt{\epsilon_{k}}\right)\mbox{ as
}k\to+\infty\mbox{ and }\|{\nabla}f(y_{k})\|=\
o\left(\sqrt{\epsilon_{k}}\right)\mbox{ as }k\to+\infty.$

###### Proof.

Assume that $k\geq\overline{k}$. We take $y=y_{k},\,x=x_{k}$ in (12) and we
get

$f_{k}(x_{k+1})\leq
f_{k}(x_{k})+\langle{\nabla}f_{k}(y_{k}),y_{k}-x_{k}\rangle-\frac{s}{2}\|{\nabla}f_{k}(y_{k})\|^{2}-\frac{s}{2}\|{\nabla}f_{k}(y_{k})-{\nabla}f_{k}(x_{k})\|^{2},\,\forall
k\geq\overline{k}.$ (14)

Now we take $y=y_{k},\,x=x^{*}$ in (12) and taking into account that
${\nabla}f(x^{*})=0$ we get

$f_{k}(x_{k+1})\leq
f_{k}(x^{*})+\langle{\nabla}f_{k}(y_{k}),y_{k}-x^{*}\rangle-\frac{s}{2}\|{\nabla}f_{k}(y_{k})\|^{2}-\frac{s}{2}\|{\nabla}f_{k}(y_{k})-\epsilon_{k}x^{*}\|^{2},\,\forall
k\geq\overline{k}.$ (15)

Consider the sequence $(p_{k})_{k\geq\overline{k}}$ defined by

$p_{k}=\frac{(1-s\epsilon_{k})^{2}q_{k}^{2}}{2s}-q_{k},$ (16)

for all $k\geq\overline{k}.$ Note that due to assumption $(Q)$ one has
$p_{k}\geq 0$ for all $k\geq\overline{k}.$

We multiply (14) with $p_{k}$ and (15) with $q_{k}$ and add to get

$\displaystyle(p_{k}+q_{k})(f_{k}(x_{k+1})-f(x^{*}))-$ $\displaystyle
p_{k}(f_{k}(x_{k})-f(x^{*}))\leq$ (17)
$\displaystyle\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle$
$\displaystyle-\frac{s}{2}p_{k}\|{\nabla}f_{k}(y_{k})-{\nabla}f_{k}(x_{k})\|^{2}-\frac{s}{2}q_{k}\|{\nabla}f_{k}(y_{k})\|^{2}-\frac{q_{k}}{2}\epsilon_{k}(s\epsilon_{k}-1)\|x^{*}\|^{2},$

for all $k\geq\overline{k}.$

Now by neglecting the nonpositive terms from the right hand side of (17) we
obtain

$\displaystyle(p_{k}+q_{k})(f_{k}(x_{k+1})-f(x^{*}))-$ $\displaystyle
p_{k}(f_{k}(x_{k})-f(x^{*}))-q_{k}\frac{\epsilon_{k}}{2}\|x^{*}\|^{2}\leq$
(18)
$\displaystyle\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle,$

for all $k\geq\overline{k}.$

Further, by using the fact that $(\epsilon_{k})$ is nonincreasing we have

$f_{k}(x_{k+1})=f_{k+1}(x_{k+1})+\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\|x_{k+1}\|^{2}\geq
f_{k+1}(x_{k+1}),$

consequently it holds

$\displaystyle(p_{k}+q_{k})(f_{k}(x_{k+1})-f(x^{*}))-p_{k}(f_{k}(x_{k})-f(x^{*}))-q_{k}\frac{\epsilon_{k}}{2}\|x^{*}\|^{2}=(p_{k}+q_{k})f_{k}(x_{k+1})-p_{k}f_{k}(x_{k})$
(19) $\displaystyle-
q_{k}f_{k}(x^{*})=(p_{k}+q_{k})(f_{k}(x_{k+1})-f_{k+1}(\overline{x}_{k+1}))+(p_{k}+q_{k})f_{k+1}(\overline{x}_{k+1})-p_{k}(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))$
$\displaystyle-
p_{k}f_{k}(\overline{x}_{k})-q_{k}f_{k}(x^{*})\geq(p_{k}+q_{k})(f_{k+1}(x_{k+1})-f_{k+1}(\overline{x}_{k+1}))-(p_{k-1}+q_{k-1})(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))$
$\displaystyle+(p_{k-1}+q_{k-1}-p_{k})(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))+p_{k}(f_{k+1}(\overline{x}_{k+1})-f_{k}(\overline{x}_{k}))+q_{k}(f_{k+1}(\overline{x}_{k+1})-f_{k}(x^{*})).$

In one hand, according to (10) one has
$f_{k}(\overline{x}_{k+1})-f_{k}(\overline{x}_{k})\geq\frac{\epsilon_{k}}{2}\|\overline{x}_{k+1}-\overline{x}_{k}\|^{2}$
hence

$\displaystyle
p_{k}(f_{k+1}(\overline{x}_{k+1})-f_{k}(\overline{x}_{k}))=p_{k}\left(f_{k}(\overline{x}_{k+1})-f_{k}(\overline{x}_{k})+\frac{\epsilon_{k+1}-\epsilon_{k}}{2}\|\overline{x}_{k+1}\|^{2}\right)$
(20) $\displaystyle\geq
p_{k}\frac{\epsilon_{k}}{2}\|\overline{x}_{k+1}-\overline{x}_{k}\|^{2}+p_{k}\frac{\epsilon_{k+1}-\epsilon_{k}}{2}\|\overline{x}_{k+1}\|^{2}\geq
p_{k}\frac{\epsilon_{k+1}-\epsilon_{k}}{2}\|\overline{x}_{k+1}\|^{2}.$

On the other hand, by using the gradient inequality we get

$\displaystyle
q_{k}(f_{k+1}(\overline{x}_{k+1})-f_{k}(x^{*}))=q_{k}\left(f_{k}(\overline{x}_{k+1})-f_{k}(x^{*})+\frac{\epsilon_{k+1}-\epsilon_{k}}{2}\|\overline{x}_{k+1}\|^{2}\right)$
(21) $\displaystyle\geq q_{k}\epsilon_{k}\langle
x^{*},\overline{x}_{k+1}-x^{*}\rangle+q_{k}\frac{\epsilon_{k+1}-\epsilon_{k}}{2}\|\overline{x}_{k+1}\|^{2}.$

Hence, combining (18), (19), (20) and (21) we obtain

$\displaystyle(p_{k}+q_{k})(f_{k+1}(x_{k+1})-f_{k+1}(\overline{x}_{k+1}))-(p_{k-1}+q_{k-1})(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))$
(22)
$\displaystyle+(p_{k-1}+q_{k-1}-p_{k})(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))\leq
q_{k}\epsilon_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\|\overline{x}_{k+1}\|^{2}$
$\displaystyle+\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle,\mbox{
for all }k\geq\overline{k}.$

Now, according to the form of $p_{k}$ and condition $(Q)$ one has

$p_{k-1}+q_{k-1}-p_{k}=\frac{(1-s\epsilon_{k-1})^{2}q_{k-1}^{2}}{2s}-\frac{(1-s\epsilon_{k})^{2}q_{k}^{2}}{2s}+q_{k}\geq
s\frac{(1-s\epsilon_{k-1})^{2}}{2s}q_{k-1}=(p_{k-1}+q_{k-1})\frac{s}{q_{k-1}},$

for all $k\geq\overline{k}.$ Consequently, (22) leads to

$\displaystyle(p_{k}+q_{k})(f_{k+1}(x_{k+1})-f_{k+1}(\overline{x}_{k+1}))-(p_{k-1}+q_{k-1})(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))$
(23)
$\displaystyle+(p_{k-1}+q_{k-1})\frac{s}{q_{k-1}}(f_{k}(x_{k})-f_{k}(\overline{x}_{k}))\leq
q_{k}\epsilon_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\|\overline{x}_{k+1}\|^{2}$
$\displaystyle+\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle,\mbox{
for all }k\geq\overline{k}.$

For all $k\geq\overline{k}$, consider now the sequence

$\eta_{k}=\frac{(p_{k}+q_{k})y_{k}-p_{k}x_{k}}{(1-\frac{s}{q_{k-1}})(1-s\epsilon_{k})q_{k}}=\frac{(1-s\epsilon_{k})q_{k}}{2s(1-\frac{s}{q_{k-1}})}y_{k}-\left(\frac{(1-s\epsilon_{k})q_{k}}{2s(1-\frac{s}{q_{k-1}})}-\frac{1}{(1-\frac{s}{q_{k-1}})(1-s\epsilon_{k})}\right)x_{k}.$

Recall that $y_{k}=x_{k}+b_{k-1}(x_{k}-x_{k-1})-c_{k}x_{k}$, hence, by using
Algorithm (13) one has

$\displaystyle\eta_{k+1}$
$\displaystyle=\frac{(1-s\epsilon_{k+1})q_{k+1}((1+b_{k}-c_{k+1})x_{k+1}-b_{k}x_{k})}{2s(1-\frac{s}{q_{k}})}-\left(\frac{(1-s\epsilon_{k+1})q_{k+1}}{2s(1-\frac{s}{q_{k}})}-\frac{1}{(1-\frac{s}{q_{k}})(1-s\epsilon_{k+1})}\right)x_{k+1}$
$\displaystyle=\left(\frac{(1-s\epsilon_{k+1})q_{k+1}(b_{k}-c_{k+1})}{2s(1-\frac{s}{q_{k}})}+\frac{1}{(1-\frac{s}{q_{k}})(1-s\epsilon_{k+1})}\right)y_{k}-\frac{(1-s\epsilon_{k+1})q_{k+1}b_{k}}{2s(1-\frac{s}{q_{k}})}x_{k}$
$\displaystyle-\left(\frac{(1-s\epsilon_{k+1})q_{k+1}(b_{k}-c_{k+1})}{2s(1-\frac{s}{q_{k}})}+\frac{1}{(1-\frac{s}{q_{k}})(1-s\epsilon_{k+1})}\right)s{\nabla}f_{k}(y_{k})$
$\displaystyle=\left(1-\frac{s}{q_{k-1}}\right)\eta_{k}-\frac{(1-s\epsilon_{k})q_{k}}{2}{\nabla}f_{k}(y_{k}),\mbox{
for all }k\geq\overline{k}.$ (24)

In what follows we show that

$\displaystyle\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle\leq$
(25)
$\displaystyle\left(1-\frac{s}{q_{k-1}}\right)\|\eta_{k}-x^{*}\|^{2}-\|\eta_{k+1}-x^{*}\|^{2}+\frac{s}{q_{k-1}}\|x^{*}\|^{2},\mbox{
for all }k\geq\overline{k}.$

Indeed, by using (3) we get

$\displaystyle\left(1-\frac{s}{q_{k-1}}\right)\|\eta_{k}-x^{*}\|^{2}-\|\eta_{k+1}-x^{*}\|^{2}+\frac{s}{q_{k-1}}\|x^{*}\|^{2}$
$\displaystyle=\left(1-\frac{s}{q_{k-1}}\right)\|\eta_{k}\|^{2}-\|\eta_{k+1}\|^{2}-2\left\langle\left(1-\frac{s}{q_{k-1}}\right)\eta_{k}-\eta_{k+1},x^{*}\right\rangle$
$\displaystyle=\left(1-\frac{s}{q_{k-1}}\right)\frac{s}{q_{k-1}}\|\eta_{k}\|^{2}+\left\langle\left(1-\frac{s}{q_{k-1}}\right)\eta_{k},(1-s\epsilon_{k})q_{k}{\nabla}f_{k}(y_{k})\right\rangle-\frac{(1-s\epsilon_{k})^{2}q_{k}^{2}}{4}\|{\nabla}f_{k}(y_{k})\|^{2}$
$\displaystyle-\langle(1-s\epsilon_{k})q_{k}{\nabla}f_{k}(y_{k}),x^{*}\rangle=\left\langle{\nabla}f_{k}(y_{k}),(p_{k}+q_{k})y_{k}-p_{k}x_{k}+(s\epsilon_{k}-1)q_{k}x^{*}-\frac{s}{2}(p_{k}+q_{k}){\nabla}f_{k}(y_{k})\right\rangle$
$\displaystyle+\left(1-\frac{s}{q_{k-1}}\right)\frac{s}{q_{k-1}}\|\eta_{k}\|^{2}.$

Consequently, by denoting
$E_{k}=(p_{k}+q_{k})(f_{k+1}(x_{k+1})-f_{k+1}(\overline{x}_{k+1}))+\|\eta_{k+1}-x^{*}\|^{2}$,
(23) and (25) lead to

$\displaystyle
E_{k}-E_{k-1}+\frac{s}{q_{k-1}}E_{k-1}\leq\frac{s}{q_{k-1}}\|x^{*}\|^{2}+q_{k}\epsilon_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\|\overline{x}_{k+1}\|^{2},$
(26)

for all $k\geq\overline{k}.$

Consider now the sequence
$\pi_{k}=\frac{1}{\prod_{i=\overline{k}}^{k}\left(1-\frac{s}{q_{i-1}}\right)}.$
Note that $(\pi_{k})_{k\geq\overline{k}}$ is well defined, positive and
increasing since by the hypotheses we have $q_{k-1}\geq 2s$ for all
$k\geq\overline{k}.$ Further, since $\frac{1}{1-\frac{s}{q_{i-1}}}\leq 2$ we
have that $\pi_{k}\leq 2^{k-\overline{k}+1}.$

Now, by multiplying (26) with $\pi_{k}$ we obtain

$\displaystyle\pi_{k}E_{k}-\pi_{k-1}E_{k-1}$
$\displaystyle\leq\frac{s}{q_{k-1}}\pi_{k}\|x^{*}\|^{2}+q_{k}\epsilon_{k}\pi_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\pi_{k}\|\overline{x}_{k+1}\|^{2}$
(27)
$\displaystyle=(\pi_{k}-\pi_{k-1})\|x^{*}\|^{2}+q_{k}\epsilon_{k}\pi_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\pi_{k}\|\overline{x}_{k+1}\|^{2},$

for all $k>\overline{k}.$

By summing up (27) from $k=\overline{k}+1$ to $k=n>\overline{k}+1$ we obtain

$\displaystyle\pi_{n}E_{n}$
$\displaystyle\leq\pi_{n}\|x^{*}\|^{2}+\sum_{k=\overline{k}+1}^{n}q_{k}\epsilon_{k}\pi_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+\sum_{k=\overline{k}+1}^{n}(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\pi_{k}\|\overline{x}_{k+1}\|^{2}+C,$
(28)

for some $C>0.$

Next we show that

$\frac{\pi_{n}\|x^{*}\|^{2}+\sum_{k=\overline{k}+1}^{n}q_{k}\epsilon_{k}\pi_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle+\sum_{k=\overline{k}+1}^{n}(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\pi_{k}\|\overline{x}_{k+1}\|^{2}+C}{\pi_{n}}=o(q_{n}^{2}\epsilon_{n})\mbox{
as }n\to+\infty.$

Indeed, according to the hypotheses $q_{n}^{2}\epsilon_{n}\to+\infty$ as
$n\to+\infty$ and we know that $(\pi_{n})$ is increasing, hence
$\frac{\pi_{n}\|x^{*}\|+C}{\pi_{n}}=o(q_{n}^{2}\epsilon_{n})\mbox{ as
}n\to+\infty.$

Further, since $\left(\frac{q_{k}\epsilon_{k}}{q_{k-1}\epsilon_{k-1}}\right)$
is bounded, $(q_{k}^{2}\epsilon_{k}\pi_{k})$ is increasing and
$\lim_{n\to+\infty}q_{n}^{2}\epsilon_{n}\pi_{n}=+\infty$, by using the fact
that $\lim_{n\to+\infty}\langle x^{*},x^{*}-\overline{x}_{n+1}\rangle=0$, via
the Cesàro-Stolz theorem we get

$\displaystyle\lim_{n\to+\infty}\frac{\sum_{k=\overline{k}+1}^{n}q_{k}\epsilon_{k}\pi_{k}\langle
x^{*},x^{*}-\overline{x}_{k+1}\rangle}{q_{n}^{2}\epsilon_{n}\pi_{n}}$
$\displaystyle=\lim_{n\to+\infty}\frac{q_{n}\epsilon_{n}\pi_{n}\langle
x^{*},x^{*}-\overline{x}_{n+1}\rangle}{q_{n}^{2}\epsilon_{n}\pi_{n}-q_{n-1}^{2}\epsilon_{n-1}\pi_{n-1}}=\lim_{n\to+\infty}\frac{\langle
x^{*},x^{*}-\overline{x}_{n+1}\rangle}{q_{n}-\frac{q_{n-1}^{2}\epsilon_{n-1}\pi_{n-1}}{q_{n}\epsilon_{n}\pi_{n}}}$
$\displaystyle=\lim_{n\to+\infty}\frac{\langle
x^{*},x^{*}-\overline{x}_{n+1}\rangle}{\frac{q_{n}^{2}\epsilon_{n}-q_{n-1}^{2}\epsilon_{n-1}}{q_{n}\epsilon_{n}}+s\frac{q_{n-1}\epsilon_{n-1}}{q_{n}\epsilon_{n}}}\leq\lim_{n\to+\infty}\frac{\langle
x^{*},x^{*}-\overline{x}_{n+1}\rangle}{s\frac{q_{n-1}\epsilon_{n-1}}{q_{n}\epsilon_{n}}}=0.$

Finally, according to the hypotheses
$\lim_{n\to+\infty}\frac{q_{n}(\epsilon_{n}-\epsilon_{n+1})}{\epsilon_{n}}=0$,
hence for some $M>0$ one has

$\displaystyle\lim_{n\to+\infty}\frac{\sum_{k=\overline{k}+1}^{n}(p_{k}+q_{k})\frac{\epsilon_{k}-\epsilon_{k+1}}{2}\pi_{k}\|\overline{x}_{k+1}\|^{2}}{q_{n}^{2}\epsilon_{n}\pi_{n}}$
$\displaystyle=\frac{1}{4s}\lim_{n\to+\infty}\frac{(1-s\epsilon_{n})^{2}q_{n}^{2}(\epsilon_{n}-\epsilon_{n+1})\pi_{n}\|\overline{x}_{n+1}\|^{2}}{q_{n}^{2}\epsilon_{n}\pi_{n}-q_{n-1}^{2}\epsilon_{n-1}\pi_{n-1}}=$
$\displaystyle=\frac{1}{4s}\lim_{n\to+\infty}\frac{(1-s\epsilon_{n})^{2}(\epsilon_{n}-\epsilon_{n+1})\|\overline{x}_{n+1}\|^{2}}{\epsilon_{n}-\frac{q_{n-1}^{2}\epsilon_{n-1}\pi_{n-1}}{q_{n}^{2}\pi_{n}}}$
$\displaystyle=\frac{1}{4s}\lim_{n\to+\infty}\frac{(1-s\epsilon_{n})^{2}(\epsilon_{n}-\epsilon_{n+1})\|\overline{x}_{n+1}\|^{2}}{\frac{q_{n}^{2}\epsilon_{n}-q_{n-1}^{2}\epsilon_{n-1}}{q_{n}^{2}}+s\frac{q_{n-1}\epsilon_{n-1}}{q_{n}^{2}}}$
$\displaystyle\leq
M\lim_{n\to+\infty}\frac{q_{n}(\epsilon_{n}-\epsilon_{n+1})}{\epsilon_{n}}=0.$

Consequently, from (28) we get $E_{n}=o(q_{n}^{2}\epsilon_{n})\mbox{ as
}n\to+\infty$, which, taking into account the form of $E_{n}$, leads to
$(p_{n}+q_{n})(f_{n+1}(x_{n+1})-f_{n+1}(\overline{x}_{n+1}))=o(q_{n}^{2}\epsilon_{n})\mbox{
as }n\to+\infty$. In other words,
$f_{n}(x_{n})-f_{n}(\overline{x}_{n})=o(\epsilon_{n})\mbox{ as }n\to+\infty$
and by using (11) we get

$f(x_{n})-\min f=\mathcal{O}(\epsilon_{n})\mbox{ as }n\to+\infty.$

In order to show strong convergence, we use (10) and we get

$\displaystyle\lim_{n\to+\infty}\|x_{n}-x^{*}\|^{2}$ $\displaystyle\leq
2\lim_{n\to+\infty}\left(\|x_{n}-\overline{x}_{n}\|^{2}+\|\overline{x}_{n}-x^{*}\|^{2}\right)$
$\displaystyle\leq
4\lim_{n\to+\infty}\frac{f_{n}(x_{n})-f_{n}(\overline{x}_{n})}{\epsilon_{n}}+2\lim_{n\to+\infty}\|\overline{x}_{n}-x^{*}\|^{2}=0.$

Concerning the rates of convergence for the discrete velocity
$\|x_{n}-x_{n-1}\|$ we conclude the following. From the definition of $E_{n}$
and the fact that $E_{n}\ =\ o\left(q_{n}^{2}\epsilon_{n}\right)\mbox{ as
}n\to+\infty$ we have that

$\|\eta_{n}-x^{*}\|\ =\ o\left(q_{n}\sqrt{\epsilon_{n}}\right)\mbox{ as
}n\to+\infty.$

Now, using the definition of $\eta_{n}$ and the fact that
$y_{n}=x_{n}+b_{n-1}(x_{n}-x_{n-1})-c_{n}x_{n}$ we derive

$\displaystyle\eta_{n}-x^{*}\ $
$\displaystyle=\frac{(1-s\epsilon_{n})q_{n}}{2s(1-\frac{s}{q_{n-1}})}y_{n}-\left(\frac{(1-s\epsilon_{n})q_{n}}{2s(1-\frac{s}{q_{n-1}})}-\frac{1}{(1-\frac{s}{q_{n-1}})(1-s\epsilon_{n})}\right)x_{n}-x^{*}$
$\displaystyle=\frac{(1-s\epsilon_{n})q_{n}}{2s(1-\frac{s}{q_{n-1}})}b_{n-1}(x_{n}-x_{n-1})+\left(\frac{1}{(1-\frac{s}{q_{n-1}})(1-s\epsilon_{n})}-\frac{(1-s\epsilon_{n})q_{n}}{2s(1-\frac{s}{q_{n-1}})}c_{n}\right)x_{n}-x^{*}$

Now, since $(q_{n}^{2}\epsilon_{n})$ is increasing, and $(\epsilon_{n})$ is
non-increasing we deduce that $(q_{n})$ is increasing. Further, since
$\lim_{n\to+\infty}q_{n}^{2}\epsilon_{n}=+\infty$ and
$\lim_{n\to+\infty}\epsilon_{n}=0$ we obtain that
$\lim_{n\to+\infty}q_{n}=+\infty.$ Consequently, for $n$ big enough one has

$b_{n-1}=\frac{(q_{n-1}-s)((1-s\epsilon_{n-1})^{2}q_{n-1}-2s)}{(1-s\epsilon_{n-1})(1-s\epsilon_{n})q_{n-1}q_{n}}\leq\frac{(1-s\epsilon_{n-1})q_{n-1}}{(1-s\epsilon_{n})q_{n}}<1$

and

$\lim_{n\to+\infty}\frac{c_{n}}{\sqrt{\epsilon_{n}}}=\lim_{n\to+\infty}\frac{2s}{q_{n}\sqrt{\epsilon_{n}}}\frac{1}{(1-s\epsilon_{n-1})(1-s\epsilon_{n})^{2}}\left(\frac{s}{q_{n-1}}-\frac{s^{2}\epsilon_{n}}{q_{n-1}}-s(\epsilon_{n-1}-\epsilon_{n})\right)=0.$

Consequently, by using the fact that $(x_{n})$ is bounded we have

$\lim_{n\to+\infty}\frac{\left(\frac{1}{(1-\frac{s}{q_{n-1}})(1-s\epsilon_{n})}-\frac{(1-s\epsilon_{n})q_{n}}{2s(1-\frac{s}{q_{n-1}})}c_{n}\right)x_{n}-x^{*}}{q_{n}\sqrt{\epsilon_{n}}}=0,$

which combined with the fact that
$\lim_{n\to+\infty}\frac{\eta_{n}-x^{*}}{q_{n}\sqrt{\epsilon_{n}}}=0$ yields

$\lim_{n\to+\infty}\frac{(1-s\epsilon_{n})}{2s(1-\frac{s}{q_{n-1}})}b_{n-1}\frac{x_{n}-x_{n-1}}{\sqrt{\epsilon_{n}}}=0.$

But $(b_{n})$ is bounded and according to our hypothese cannot go to 0 as
$n\to+\infty$, hence $\|x_{n}-x_{n-1}\|=o(\sqrt{\epsilon_{n}})$ as
$n\to+\infty.$

From here we deduce at once that $\|y_{n}-x_{n}\|=o(\sqrt{\epsilon_{n}})$ as
$n\to+\infty,$ hence in particular

$\lim_{n\to+\infty}y_{n}=x^{*}.$

From (3) we have
$\left(1-\frac{s}{q_{n-1}}\right)\eta_{n}-\eta_{n+1}=\frac{(1-s\epsilon_{n})q_{n}}{2}{\nabla}f_{n}(y_{n})$
and using the fact that $\|\eta_{n}\|\ =\
o\left(q_{n}\sqrt{\epsilon_{n}}\right)\mbox{ as }n\to+\infty$ we obtain that
$\|{\nabla}f_{n}(y_{n})\|=o(\sqrt{\epsilon_{n}})$ as $n\to+\infty,$ and
further that $\|{\nabla}f(y_{n})\|=o(\sqrt{\epsilon_{n}})$ as $n\to+\infty.$
By using the $L-$Lipschitz continuity of ${\nabla}f$ we get
$\|{\nabla}f(x_{n})-{\nabla}f(y_{n})\|\leq L\|y_{n}-x_{n}\|$ which combined
with the facts that $\|y_{n}-x_{n}\|=o(\sqrt{\epsilon_{n}})$ as $n\to+\infty$
and $\|{\nabla}f(y_{n})\|=o(\sqrt{\epsilon_{n}})$ as $n\to+\infty$ lead to
$\|{\nabla}f(x_{n})\|=o(\sqrt{\epsilon_{n}})$ as $n\to+\infty$.

Finally, by using Lemma 2 we get

$f(y_{n})-\min f\leq f(x_{n})-\min
f+\|{\nabla}f(x_{n})\|\|y_{n}-x_{n}\|+\frac{L}{2}\|y_{n}-x_{n}\|^{2}$

hence, from the facts that $f(x_{n})-\min f=\mathcal{O}(\epsilon_{n})$,
$\|y_{n}-x_{n}\|=o(\sqrt{\epsilon_{n}})$ and
$\|{\nabla}f(x_{n})\|=o(\sqrt{\epsilon_{n}})$ s $n\to+\infty$ we obtain that

$f(y_{n})-\min f=\mathcal{O}(\epsilon_{n})\mbox{ as }n\to+\infty.$

∎

##  4 Particular choice of the parameter sequences $(q_{k})$ and
$(\epsilon_{k})$

Let us consider a specific choice of the sequences $(q_{k})$ and
$(\epsilon_{k})$ being polynomial type, namely, $q_{k}=ak^{q}$,
$\epsilon_{k}=\frac{c}{k^{p}}$, where $1\geq q>0$, $p>0$ and $a$ and $c$ are
positive real numbers. Let us fix $0<s<\frac{1}{L}.$ Then from condition (S)
we have $s\leq\frac{1}{L+\epsilon_{k_{0}}}$ for some $k_{0}\in\mathbb{N}$, and
it is an easy computation that

$k_{0}=\mbox{int}\left(\frac{cs}{1-Ls}\right)^{\frac{1}{p}}+1,$

where $\mbox{int}(x)$ denotes the integer part of $x.$

Now we compute the index $k_{1}\geq k_{0}$ such that $1-s\epsilon_{k}>0$ for
all $k\geq k_{1}.$ Note that $1-s\epsilon_{k}>0$ whenever
$k\geq\mbox{int}(cs)^{\frac{1}{p}}+1$, consequently one can take
$k_{1}=k_{0}.$

Condition $(Q)$ in this case becomes: after an index $k_{2}\geq k_{1}$ it
holds that

$\left(1-\frac{sc}{(k+1)^{p}}\right)^{2}a^{2}(k+1)^{2q}-\left(1-\frac{sc}{k^{p}}\right)^{2}a^{2}k^{2q}-2sa(k+1)^{q}+s\left(1-\frac{sc}{k^{p}}\right)^{2}ak^{q}\
\leq\ 0$

and

$ak^{q}\ \geq\ \frac{2s}{(1-\frac{sc}{k^{p}})^{2}}$

for all $k\geq k_{2}$. Note that the second condition if always fulfilled
starting from $k$ large enough due to $q$ and $p$ being positive.

Now consider the case $q<1$. Since

$\left(1-\frac{sc}{(k+1)^{p}}\right)^{2}a^{2}(k+1)^{2q}-\left(1-\frac{sc}{k^{p}}\right)^{2}a^{2}k^{2q}=\mathcal{O}(k^{2q-1})\mbox{
as }k\to+\infty$

and

$-2sa(k+1)^{q}+s\left(1-\frac{sc}{k^{p}}\right)^{2}ak^{q}=-sak^{q}+\mathcal{O}(k^{q-1})+\mathcal{O}(k^{q-p})\mbox{
as }k\to+\infty$

we obtain that

$\displaystyle\left(1-\frac{sc}{(k+1)^{p}}\right)^{2}a^{2}(k+1)^{2q}-\left(1-\frac{sc}{k^{p}}\right)^{2}a^{2}k^{2q}-2sa(k+1)^{q}+s\left(1-\frac{sc}{k^{p}}\right)^{2}ak^{q}=-sak^{q}$
$\displaystyle+\mathcal{O}(k^{2q-1})+\mathcal{O}(k^{q-1})+\mathcal{O}(k^{q-p})\mbox{
as }k\to+\infty,$

hence there exists an index $k_{2}\geq k_{1}$ such that (Q) holds.

Now, if $q=1$ we obtain

$\displaystyle\left(1-\frac{sc}{(k+1)^{p}}\right)^{2}a^{2}(k+1)^{2}-\left(1-\frac{sc}{k^{p}}\right)^{2}a^{2}k^{2}-2sa(k+1)+s\left(1-\frac{sc}{k^{p}}\right)^{2}ak=(2a^{2}-as)k$
$\displaystyle+\mathcal{O}(k^{1-p})+\mathcal{O}(1)\mbox{ as }k\to+\infty,$

hence (Q) holds provided $2a^{2}-as<0$, that is $a<\frac{s}{2}.$

Concerning the sequence $(b_{k})_{k\geq 0}$ in this particular case condition
(B) becomes:

$(Bp)\,\,\,\left\\{\begin{array}[]{lll}b_{k-1}=0,\mbox{ if
}k\in\left\\{1,(cs)^{\frac{1}{p}},(cs)^{\frac{1}{p}}+1\right\\}\\\ \\\
\displaystyle
b_{k-1}=\frac{k^{p}(a(k-1)^{q}-s)(a((k-1)^{p}-cs)^{2}(k-1)^{q}-2s(k-1)^{2p})}{a^{2}(k-1)^{q+p}k^{q}((k-1)^{p}-cs)(k^{p}-cs)},\mbox{
otherwise}.\end{array}\right.$

Note that $b_{k}\to 1$ as $k\to+\infty.$

Further, condition (C) becomes:

$(Cp)\,\,\,\left\\{\begin{array}[]{lll}c_{k}=0,\mbox{ if
}k\in\left\\{1,(cs)^{\frac{1}{p}},(cs)^{\frac{1}{p}}+1\right\\}\\\ \\\
\displaystyle
c_{k}=\frac{2s^{2}k^{p}((k-1)^{p}k^{p}-c(k-1)^{p}-ac(k-1)^{q}k^{p}+ac(k-1)^{q+p})}{a^{2}(k-1)^{q}k^{q}((k-1)^{p}-cs)(k^{p}-cs)^{2}},\mbox{
otherwise}.\end{array}\right.$

Note that $c_{k}>0$ for $k$ big enough, further $(c_{k})$ is nonincreasing and
$c_{k}\to 0$ as $k\to+\infty.$ Hence, indeed in this case the term
$c_{k}x_{k}$ in Algorithm (2) plays the role of a Tikhonov regularization
term. More precisely, Algorithm (2) reads as: $x_{0},x_{1}\in\mathcal{H}$ and
for all $k\geq 1$

$\left\\{\begin{array}[]{llll}\displaystyle y_{k}=x_{k},\mbox{ if
}k\in\left\\{1,(cs)^{\frac{1}{p}},(cs)^{\frac{1}{p}}+1\right\\}\\\
\displaystyle
y_{k}=x_{k}+\frac{k^{p}(a(k-1)^{q}-s)(a((k-1)^{p}-cs)^{2}(k-1)^{q}-2s(k-1)^{2p})}{a^{2}(k-1)^{q+p}k^{q}((k-1)^{p}-cs)(k^{p}-cs)}(x_{k}-x_{k-1})\\\
\displaystyle\,\,\,\,\,\,\,\,\,\,-\frac{2s^{2}k^{p}((k-1)^{p}k^{p}-c(k-1)^{p}-ac(k-1)^{q}k^{p}+ac(k-1)^{q+p})}{a^{2}(k-1)^{q}k^{q}((k-1)^{p}-cs)(k^{p}-cs)^{2}}x_{k},\mbox{
otherwise}\\\
x_{k+1}=y_{k}-s{\nabla}f(y_{k})-\frac{cs}{k^{p}}y_{k}.\end{array}\right.$ (29)

Note that from a numerical point of view Algorithm (29) can easily be
implemented. In this particular case, we have the following result.

######  Theorem 6.

Let $0<q<1$ and $0<p<2q$ and for a fixed the stepsize
$s\in\left(0,\frac{1}{L}\right)$ consider the sequences
$(x_{k})_{k\in\mathbb{N}},\,(y_{k})_{k\in\mathbb{N}}$ generated by Algorithm
(29). Then, $(x_{k})$ and $(y_{k})$ converge strongly to $x^{*}$, where
$\\{x^{*}\\}=\operatorname*{pr}_{\operatorname*{argmin}f}(0)$ is the minimum
norm minimizer of our objective function $f.$

Further,

$f_{k}(x_{k})-f_{k}(\overline{x}_{k})=o(k^{-p})\mbox{ as }k\to+\infty,$
$f(x_{k})-\min f=\mathcal{O}\left(k^{-p}\right),\mbox{ as }k\to+\infty\mbox{
and }f(y_{k})-\min f=\mathcal{O}\left(k^{-p}\right),\mbox{ as }k\to+\infty,$
$\|x_{k}-x_{k-1}\|\ =\ o\left(k^{-\frac{p}{2}}\right)\mbox{ as }k\to+\infty$

and

$\|{\nabla}f(x_{k})\|\ =\ o\left(k^{-\frac{p}{2}}\right)\mbox{ as
}k\to+\infty\mbox{ and }\|{\nabla}f(y_{k})\|\ =\
o\left(k^{-\frac{p}{2}}\right)\mbox{ as }k\to+\infty$

###### Proof.

We only need to show that the following conditions from the hypotheses of
Theorem 5 hold:

  * •

the sequence
$\frac{q_{k}\epsilon_{k}}{q_{k-1}\epsilon_{k-1}}=\frac{k^{q-p}}{(k-1)^{q-p}}$
is bounded if we take the starting index big enough;

  * •

the sequence $q_{k}^{2}\epsilon_{k}=a^{2}ck^{2q-p}$ is increasing after a
starting index big enough;

  * •

$\lim_{k\to+\infty}q_{k}^{2}\epsilon_{k}=\lim_{k\to+\infty}a^{2}ck^{2q-p}\ =\
+\infty$;

  * •

$\lim_{k\to+\infty}\frac{q_{k}(\epsilon_{k}-\epsilon_{k+1})}{\epsilon_{k}}=\lim_{k\to+\infty}\frac{ak^{q}\left(\frac{1}{k^{p}}-\frac{1}{(k+1)^{p}}\right)}{\frac{1}{k^{p}}}\
=\ 0$.

First of all, the sequence $\frac{k^{q-p}}{(k-1)^{q-p}}$ is indeed bounded if
we take the starting index big enough since
$\lim_{k\to+\infty}\frac{k^{q-p}}{(k-1)^{q-p}}=1$. Secondly, the sequence
$a^{2}ck^{2q-p}$ is increasing when $2q>p$ and
$\lim_{k\to+\infty}a^{2}ck^{2q-p}\ =\ +\infty$ when $2q>p$. Finally,

$\lim_{k\to+\infty}\frac{ak^{q}\left(\frac{1}{k^{p}}-\frac{1}{(k+1)^{p}}\right)}{\frac{1}{k^{p}}}\
=\lim_{k\to+\infty}\frac{ak^{q}}{k+1}\frac{(k+1)^{p}-k^{p}}{(k+1)^{p-1}}=0,$

since $q<1.$ ∎

######  Remark 7.

We emphasize that Algorithm (29) can be seen as a Nesterov type algorithm with
two Tikhonov regularization terms. Indeed, the extrapolation parameter
$(b_{k})$ goes to 1 as $k\to+\infty$ such as in the Nesterov algorithm.
Further, the terms $c_{k}x_{k}$ and $\epsilon_{k}y_{k}$ can be thought as
Tikhonov regularization terms since both $c_{k}$ and $\epsilon_{k}$ are
nonnegative and nonincreasing sequences (after $k$ big enough), and goes to 0
as $k\to+\infty.$ Unfortunately we could not allow the case $q=1$ and $p=2$ in
our algorithm. Nevertheless, if $p$ is close to 2, (and $q$ is close to 1),
then from a numerical perspective the convergence rates obtained for the
potential energy $f(x_{k})-\min f$ and discrete velocity $\|x_{k}-x_{k-1}\|$
are as good as the rates obtained for the famous Nesterov algorithm, see [11].
Moreover, our algorithm assures the strong convergence of the generated
sequences to the minimum norm minimizer a feature that makes it unique in the
literature.

##  5 Numerical experiments

In this section we consider some numerical experiments in order to sustain the
theoretical results obtained in Theorem 6. To this purpose, let us consider
the objective function

$f:\mathbb{R}^{2}\mapsto\mathbb{R},\ f(x,y)=(ax+by)^{2},$

where $a,b\in\mathbb{R}\setminus\\{0\\}$. Then obviously $f$ is smooth and
convex and its gradient is Lipschitz continuous, having Lipschitz constant
$L=2\sqrt{2}\sqrt{(a^{2}+b^{2})\max(a^{2},b^{2})}.$ Observe that the minimal
value of $f$ is 0 and the set $\operatorname*{argmin}f$ is
$\left\\{\left(x,-\frac{a}{b}x\right):x\in\mathbb{R}\right\\}$, further
clearly $(0,0)$ is the minimizer of minimal norm. For simplicity in the
following experiments concerning Algorithm (29) we take everywhere
$q_{k}=k^{\frac{4}{5}}$ and $s=0.1$, (which always will satisfy
$s<\frac{1}{L}$), and fix the starting points $x_{0}=(1,-1)$ and
$x_{1}=(-1,1).$

In our first experiment we fix $a=0.1$ and $b=100$ and in Algorithm (29) we
set $\epsilon_{k}=\frac{1}{k^{p}}$, where $p\in\\{0.3,0.6,0.9,1.2,1.5\\}$.
Further, we consider the case when there is no Tikhonov regularization, that
is the case of a Nesterov type algorithm by taking $\epsilon_{k}\equiv 0$ and
$c_{k}\equiv 0$. In order to show the theoretical rates obtained in Theorem 6
for the discrete velocity $\|x_{k}-x_{k-1}\|$ and the potential energy
$f(x_{k})-\min f$ we also represent the values $1/k$ and $f(x_{1})/k^{2}$,
respectively.

So we run Algorithm (29) for 20 iterations, the results are shown in Figure 1.

Figure 1: Different choices of $\epsilon_{k}$

Observe that indeed, our algorithm has a similar (even better) behavior as the
Nesterov type algorithm, the convergence rates for the discrete velocity and
potential energy are of order $o(1/k)$ and $O(1/k^{2})$, respectively.
Further, the Tikhonov regularization does not affect the optimal rates, even
more, while we increase $p$ these rates become better.

In our second experiment for $a=1,\,b=5$ we show the influence of the Tikhonov
regularization terms $\epsilon_{k}y_{k}$ and $c_{k}x_{k}$ on the behaviour of
the iterates of the algorithm. In the next figures we represent the first
component of the iterates $x_{k}$ with red meanwhile the second component will
be represented with blue.

First, we analyze what happens if we renounce to both Tikhonov regularization
terms. So let us put both $c_{k}\equiv 0$ and $\epsilon_{k}\equiv 0$ in
Algorithm (29). According to Figure 2 in this case there is no convergence to
the minimal norm element.

Figure 2: $\epsilon_{k}y_{k}\equiv 0,\,c_{k}x_{k}\equiv 0$

Next we show that in order to have convergence to the minimal norm element the
presence of both Tikhonov regularization terms are essential. To this purpose,
we take $\epsilon_{k}=1/k^{\frac{3}{2}}$, (and the corresponding $c_{k}$), in
order to show convergence to the minimum norm minimizer and also
$\epsilon_{k}\equiv 0$ to show that in this case our algorithm does not
converge anymore to the minimum norm minimizer, see Figure 3(a). Note that in
case $\epsilon_{k}\equiv 0$ the parameter $c_{k}$ in Algorithm (29) becomes
$c_{k}=\frac{2s^{2}}{(k-1)^{q}k^{q}}$, hence the term $c_{k}x_{k}$ in the
formulation of $y_{k}$ still has the role of a Tikhonov regularization term.
Further, we consider the case $c_{k}\equiv 0$, but
$\epsilon_{k}=1/k^{\frac{3}{2}}$, see Figure 3(b).

(a) The absence of the term $\epsilon_{k}y_{k}$

(b) The absence of the term $c_{k}x_{k}$

Figure 3: Dropping one of the Tikhonov regularization terms in Algorithm (29)
we do not have convergence to the minimum norm solution anymore.

As we can see, in the absence of one of the Tikhonov regularization terms we
do not have the convergence to the element of the minimal norm. Hence,
according to the last two figures the presence of double Tikhonov
regularization terms in our algorithm is fully justified.

##  6 Conclusions, perspectives

Due to our best knowledge, Algorithm (2) and in particular Algorithm (29) are
the first inertial gradient type algorithms considered in the literature that
assure strong convergence to the minimum norm minimizer of a smooth convex
function and also fast convergence of the function values and discrete
velocity. As we have emphasized in the paper these algorithms can be seen as
Nesterov type algorithms with two Tikhonov regularization terms. Despite of
the complex structure of the inertial parameter and one of the Tikhonov
regularization parameters our algorithms can easily be implemented, therefore
are suitable for use in practical problems arising in image processing and
machine learning. As a future related research we mention here the forward-
backward algorithms with Tikhonov regularization associated to the
minimization problem having in its objective the sum of a proper convex lower
semicontinuous function and a smooth convex function with Lipschitz continuous
gradient. In our opinion similar results to those provided in Theorem 6 can be
obtained. Indeed, the success of such research is promising taking into
account that in [20] strong convergence of an inertial-proximal algorithm to
the minimal norm minimizer of a proper convex and lower semicontinuous
function is shown, meanwhile in the present paper we obtained similar results
for inertial gradient type algorithm in connection to a smooth convex
optimization problem.

##  7 Declarations

Availability of data and materials

In this manuscript only the datasets generated by authors were analysed.

Competing interests

The authors have no competing interests.

## References

  * [1] C.D. Alecsa, S.C. László, Tikhonov regularization of a perturbed heavy ball system with vanishing damping, SIAM J. OPTIM. 31(4), 2921-2954 (2021)
  * [2] H. Attouch, A. Balhag, Z. Chbani, H. Riahi, Damped inertial dynamics with vanishing Tikhonov regularization: Strong asymptotic convergence towards the minimum norm solution, Journal of Differential Equations 311, 29-58 (2022)
  * [3] H. Attouch, L.M. Briceño-Arias, P.L. Combettes, A strongly convergent primal-dual method for nonoverlapping domain decomposition, Numerische Mathematik 133(3), 443-470 (2016)
  * [4] H. Attouch, A. Cabot, Z. Chbani, H. Riahi, Inertial Forward–Backward Algorithms with Perturbations: Application to Tikhonov Regularization. J Optim Theory Appl 179, 1–36 (2018)
  * [5] H. Attouch, Z. Chbani, J. Fadili, H. Riahi, First-order optimization algorithms via inertial systems with Hessian driven damping, Mathematical Programming, 2020, https://doi.org/10.1007/s10107-020-01591-1
  * [6] H. Attouch, Z. Chbani, H. Riahi, Combining fast inertial dynamics for convex optimization with Tikhonov regularization, J. Math. Anal. Appl 457, 1065-1094 (2018)
  * [7] H. Attouch, Z. Chbani, H. Riahi, Accelerated gradient methods with strong convergence to the minimum norm minimizer: a dynamic approach combining time scaling, averaging, and Tikhonov regularization, https://arxiv.org/pdf/2211.10140.pdf (2022)
  * [8] H. Attouch, R. Cominetti, A dynamical approach to convex minimization coupling approximation with the steepest descent method, Journal of Differential Equations 128(2), 519-540 (1996)
  * [9] H. Attouch, M.-O. Czarnecki, Asymptotic Control and Stabilization of Nonlinear Oscillators with Non-isolated Equilibria, J. Differential Equations 179, 278-310 (2002)
  * [10] H. Attouch, S.C. László, Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution, https://arxiv.org/abs/2104.11987 (2021)
  * [11] H. Attouch, J. Peypouquet, The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than $1/k^{-2}$, SIAM J. Optim. 26(3), 1824-1834 (2016)
  * [12] R.I. Boţ, E.R. Csetnek, S.C. László, Tikhonov regularization of a second order dynamical system with Hessian damping, Math. Program. 189, 151–186 (2021)
  * [13] R.I. Boţ, E.R. Csetnek, S.C. László, On the strong convergence of continuous Newton-like inertial dynamics with Tikhonov regularization for monotone inclusions, https://www.researchgate.net/publication/369118075, (2023)
  * [14] R.I. Boţ, S.M. Grad, D. Meier, M. Staudigl, Inducing strong convergence of trajectories in dynamical systems associated to monotone inclusions with composite structure, Adv. Nonlinear Anal. 10, 450–476 (2021)
  * [15] A. Chambolle, C. Dossal, it On the convergence of the iterates of the “fast iterative shrinkage/ thresholding algorithm”, J. Optim. Theory Appl. 166(3), 968–982 (2015)
  * [16] R. Cominetti, J. Peypouquet, S. Sorin, Strong asymptotic convergence of evolution equations governed by maximal monotone operators with Tikhonov regularization, J. Differential Equations 245, 3753-3763 (2008)
  * [17] M.A. Jendoubi, R. May, On an asymptotically autonomous system with Tikhonov type regularizing term, Archiv der Mathematik 95 (4), 389-399 (2010)
  * [18] S.C. László, Convergence rates for an inertial algorithm of gradient type associated to a smooth nonconvex minimization, Mathematical Programming 190, 285–329 (2021)
  * [19] S.C. László, On the strong convergence of the trajectories of a Tikhonov regularized second order dynamical system with asymptotically vanishing damping, Journal of Differential Equations 362, 355-381 (2023)
  * [20] S.C. László, On the convergence of an inertial proximal algorithm with a Tikhonov regularization term, https://arxiv.org/abs/2302.02115, 2023
  * [21] Y. Nesterov , Introductory lectures on convex optimization: a basic course. Kluwer Academic Publishers, Dordrecht, 2004
  * [22] Y. Nesterov, A method of solving a convex programming problem with convergence rate $O(1/k^{2})$, Soviet Math. Dokl. 27, 372-376 (1983)
  * [23] B. Shi, S.S. Du, M.I. Jordan, Weijie J. Su, Understanding the Acceleration Phenomenon via High-Resolution Differential Equations, https://arxiv.org/abs/1810.08907
  * [24] W. Su, S. Boyd, E.J. Candès, A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights, Journal of Machine Learning Research 17(153), 1-43 (2016)
  * [25] A.N. Tikhonov, Doklady Akademii Nauk SSSR 151 (1963) 501-504, (Translated in ”Solution of incorrectly formulated problems and the regularization method”), Soviet Mathematics 4 (1963) 1035-1038)
  * [26] A.N. Tikhonov, V.Y. Arsenin, Solutions of Ill-Posed Problems, Winston, New York, (1977)