# New Douglas-Rashford Splitting Algorithms for Generalized DC Programming
with Applications in Machine Learning

Yonghong Yao 111School of Mathematical Sciences, Tiangong University, Tianjin
300387, China; and Center for Advanced Information Technology, Kyung Hee
University, Seoul 02447, South Korea; e-mail<EMAIL_ADDRESS>, Lateef O.
Jolaoso222School of Mathematical Sciences, University of Southampton, SO17
1BJ, United Kingdom; e-mail<EMAIL_ADDRESS>, Yekini
Shehu333(Corresponding Author) School of Mathematical Sciences, Zhejiang
Normal University, Jinhua 321004, People’s Republic of China; e-mail:
<EMAIL_ADDRESS>, Jen-Chih Yao444Center for General Education, China
Medical University, Taichung 40402, Taiwan, Academy of Romanian Scientists,
Bucharest, Romania; e-mail<EMAIL_ADDRESS>
###### Abstract

In this work, we propose some new Douglas-Rashford splitting algorithms for
solving a class of generalized DC (difference of convex functions) in real
Hilbert spaces. The proposed methods leverage the proximal properties of the
nonsmooth component and a fasten control parameter which improves the
convergence rate of the algorithms. We prove the convergence of these methods
to the critical points of nonconvex optimization under reasonable conditions.
We evaluate the performance and effectiveness of our methods through
experimentation with three practical examples in machine learning. Our
findings demonstrated that our methods offer efficiency in problem-solving and
outperform state-of-the-art techniques like the DCA (DC Algorithm) and ADMM.

Keywords: Douglas-Rachford splitting algorithm; DC programming; Nonconvex
optimization; Machine learning.

2010 MSC classification: 65K05, 90C26, 90C30.

##  1 Introduction

Let us consider the following class of generalized DC programming in a real
Hilbert space $H$:

$\min_{x\in H}\\{p(x)=f(x)+g(x)-h(x)\\},$ (1)

where $f,g:H\rightarrow(-\infty,+\infty]$ are proper, convex, and lower
semicontinuous (not necessarily smooth) functions, and
$h:H\rightarrow(-\infty,+\infty]$ is a convex and smooth function. The DC
programming (1) was first presented by Tao et al. [41], which has received
attention due to its applications in image processing [46], compressed sensing
[31], statistics and machine learning [19, 27, 28, 35], dimensionality
reduction [16] and multiple-input-multiple-output (MIMO) [53]. For example, in
the application of DC programming (1) in machine learning, the function $f$
stands for a loss function that denotes the data fidelity and $g-h$ represents
a regularizer that induces some expected structures in the solution [29, 30].

The DCA (DC algorithm) is one of the most prominent methods for solving DC
programming (1). The DCA linearizes the concave part of the DC programming (1)
at the current iteration and obtains the next iteration via a convex
subproblem. Further studies on the DCA have been given in [20, 32, 52]. In
[11, Algorithm 1], Chuang et al. introduced the following unified Douglas-
Rachdford splitting algorithm to solve DC programming (1):

$\displaystyle\left\\{\begin{array}[c]{ll}&y_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-x_{n}\|^{2}\Big{\\}},\\\
&z_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2y_{n}-x_{n}+\beta\nabla
h(y_{n}))\|^{2}\Big{\\}},\\\
&x_{n+1}=x_{n}+\kappa_{n}(z_{n}-y_{n}),\end{array}\right.$ (5)

with $\beta>0$ and $\kappa_{n}\in(0,2)$, and obtained, under certain
conditions, that $\underset{n\rightarrow\infty}{\lim}\|y_{n}-x\|=0$,
$\underset{n\rightarrow\infty}{\lim}x_{n}=y$, where $x$ is a stationary point
of DC programming (1) and ${\rm prox}_{\beta f}(y)=x$. In [4], Bian and Zhang
extended the three-operator splitting algorithm of Davis-Yin [15] from solving
the optimization problem of the sum of three convex functions to solving
nonconvex optimization problems. This three-operator in [15] could be seen as
a slight extension of the Douglas-Rachford splitting algorithm [18] and the
generalized forward-backward splitting algorithm in [3, 38].

Motivated by the forward-backward splitting algorithm with deviations proposed
in [39], Hu and Dong [24] proposed the following weakly convergent three-
operator splitting algorithm with deviations for solving DC programming (1):

$\displaystyle\left\\{\begin{array}[c]{ll}&s_{n}=x_{n}+u_{n}-\beta Lv_{n},\\\
&y_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-s_{n}\|^{2}\Big{\\}},\\\
&t_{n}=y_{n}-\frac{\beta L}{2}v_{n},\\\ &q_{n}=y_{n}+v_{n},\\\
&z_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2t_{n}-s_{n}+\beta\nabla
h(q_{n}))\|^{2}\Big{\\}},\\\
&x_{n+1}=x_{n}+\kappa_{n}(z_{n}-y_{n}),\end{array}\right.$ (12)

where the vectors $u_{n+1}$ and $v_{n+1}$ are chosen such that

$\frac{\kappa_{n+1}}{2-\kappa_{n+1}}\|u_{n+1}\|^{2}+\kappa_{n+1}\beta
L\|v_{n+1}\|^{2}\leq\zeta_{n}l_{n}^{2}$ (13)

with

$l_{n}^{2}=\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}+\frac{1}{2-\kappa_{n}}u_{n}\|^{2}.$
(14)

Consequently, both one-step and two-step inertial three-operator splitting
algorithms are deduced in [24, Algorithm 2, Algorithm 3] from the deviation
vector $u_{n}$. Numerical comparisons of [24, Algorithm 2, Algorithm 3] with
unified Douglas-Rachdford splitting algorithm (5) using DC regularized sparse
recovery problems showed that [24, Algorithm 2, Algorithm 3] outperformed
unified Douglas-Rachdford splitting algorithm (5) in terms of cpu time and
number of iterations.

Contribution. Our aim in this paper is to continue the approach of Chuang et
al.[11] and Hu and Dong [24] by designing a new splitting algorithm to solve
DC programming (1) with the following contributions:

  * •

we introduce new fast Douglas-Rachford splitting algorithms with the aim of
solving DC programming (1) in Hilbert spaces, which are also extensions of the
unified Douglas-Rachford splitting algorithm proposed in (5);

  * •

in our proposed algorithms, we relax the strict norm conditions (13) and (14)
used in [24] in proving the convergence of the methods to critical point of
the nonconvex optimization problem;

  * •

we provide some numerical experiments and implement the proposed methods for
solving real-life problems in machine learning. The results of the experiments
indicate the accuracy and efficiency of the proposed methods over popular
methods such as the DC algorithm and ADMM in the literature.

Organization. We structure the rest of the paper as follows: in Section 2, we
put in place some basic concepts and lemmas while in Section 3, we introduce
our proposed algorithms and weak convergence analysis. Section 4 deals with
numerical experiments where we implement the proposed methods in three DC
models in machine learning. Some final remarks and future considerations are
given in Section 5.

##  2 Preliminaries

In the convergence analysis of this paper, we shall denote the weak
convergence by the symbol ”$\rightharpoonup$” and denote the strong
convergence by the symbol ”$\rightarrow$”.

The following definitions and basic concepts can be found in [3].

######  Definition 2.1.

A mapping $T:H\to H$ is called

  * (i)

nonexpansive if $\|Tx-Ty\|\leq\|x-y\|,$ for all $x,y\in H;$

  * (ii)

$L$-Lipschitz continuous if there exists $L>0$ such that $\|Tx-Ty\|\leq
L\|x-y\|$ for all $x,y\in H.$

We define the domain of a function $f:H\rightarrow(-\infty,+\infty]$ to be
${\rm dom}f:=\\{x\in H:f(x)<+\infty\\}$ and say that $f$ is proper if ${\rm
dom}f\neq\emptyset$.

######  Definition 2.2.

Suppose $f:H\rightarrow(-\infty,+\infty]$ is proper. The subdifferential of
$f$ at $x$ is defined by

$\partial f(x):=\\{u\in H:f(z)\geq f(x)+\langle z-x,u\rangle,\forall z\in
H\\}.$

We say that $f$ is subdifferentiable at $x$ if $\partial f(x)\neq\emptyset$.
In this case, the elements of $\partial f(x)$ are termed the subgradients of
$f$ at $x$.

######  Definition 2.3.

An operator $A:H\to 2^{H}$ is said to be monotone if for any $x,y\in H,$

$\langle x-y,u-v\rangle\geq 0,\forall u\in Ax,v\in Ay.$ (15)

The Graph of $A$ is defined by

$Gr(A):=\\{(x,u)\in H\times H:u\in Ax\\}.$

If $Gr(A)$ is not properly contained in the graph of any other monotone
mapping, then $A$ is maximal monotone operator. It is well-known that for each
$x\in H$, and $\lambda>0$, there is a unique $z\in H$ such that
$x\in(I+\lambda A)z$. Furthermore, $A$ is $\rho$-strongly monotone (with
$\rho>0$) if

$\langle x-y,u-v\rangle\geq\rho\|u-v\|^{2},\forall u\in Ax,v\in Ay.$

######  Definition 2.4.

  1. (i)

A function $f$ is called convex if ${\rm dom}f$ is a convex set and if
$\forall x,y\in{\rm dom}f,\delta\in[0,1]$, we have

$f(\delta x+(1-\delta)y)\leq\delta f(x)+(1-\delta)f(y).$

  2. (ii)

A function $f$ is said to be $\rho$-strongly convex with $\rho>0$ if
$f-\frac{\rho}{2}\|.\|^{2}$ is convex. Thus, $\forall x,y\in{\rm
dom}f,\delta\in[0,1]$,

$f(\delta x+(1-\delta)y)\leq\delta
f(x)+(1-\delta)f(y)-\frac{\rho}{2}\theta(1-\theta)\|x-y\|^{2}.$

Moreover, if $f$ is convex, we have

$f(x)\geq f(y)+\langle u,x-y\rangle,x,y\in{\rm dom}f,$

with $u\in\partial f(y)$ is arbitrary. Also, if $f$ is $\rho$-strongly convex
with $\rho>0$, we have

$f(x)\geq f(y)+\langle u,x-y\rangle+\frac{\rho}{2}\|x-y\|^{2},x,y\in{\rm
dom}f,$

with $u\in\partial f(y)$ is arbitrary.

######  Definition 2.5.

Given $\beta>0$ and $f:H\rightarrow(-\infty,+\infty]$ a proper, lower
semicontinuous and convex function. The proximal operator of $f$ with $\beta$
is defined by

${\rm prox}_{\beta f}(x):=\underset{u\in H}{\rm
argmin}\Big{\\{}f(u)+\frac{1}{2\beta}\|u-x\|^{2}\Big{\\}}$

for each $x\in H$.

Using the first-order optimality condition of the minimization $\underset{u\in
H}{\min}\Big{\\{}f(u)+\frac{1}{2\beta}\|u-x\|^{2}\Big{\\}}$, we have that
${\rm prox}_{\beta f}(x)=(I+\beta\partial f)^{-1}(x)$, where $I$ is the
identity on $H$.

######  Lemma 2.6.

([3, Example 22.4]) Suppose $\rho>0$ and $f:H\rightarrow(-\infty,+\infty]$ is
a proper, lower-semicontinuous and convex function. If $f$ is $\rho$-strongly
convex, then $\partial f$ is $\rho$-strongly monotone.

######  Lemma 2.7.

([3, Proposition 16.36]) Suppose $f:H\rightarrow(-\infty,+\infty]$ is a
proper, lower-semicontinuous and convex function. If $\\{v_{n}\\}$ and
$\\{x_{n}\\}$ are sequences in $H$ with $(x_{n},v_{n})\in Gr(\partial f)$ for
each $n\in\mathbb{N}$, $x_{n}\rightarrow x$ and $v_{n}\rightharpoonup v$, then
$(x,v)\in Gr(\partial f)$.

######  Lemma 2.8.

If $x,y\in H$, we have

  * (i)

$2\langle
x,y\rangle=\|x\|^{2}+\|y\|^{2}-\|x-y\|^{2}=\|x+y\|^{2}-\|x\|^{2}-\|y\|^{2}.$

  * (ii)

Let $x,y\in H$ and $a\in\mathbb{R}$. Then

$\displaystyle\|(1-a)x+ay\|^{2}=(1-a)\|x\|^{2}+a\|y\|^{2}-(1-a)a\|x-y\|^{2}.$

######  Lemma 2.9.

([36]) Let $K$ be a nonempty subset of a Hilbert space $H$ and let
$\\{x_{n}\\}$ be a bounded sequence in $H$. Assume the following two
conditions are satisfied:

  * •

$\lim_{n\to\infty}\|x_{n}-x\|$ exists for each $x\in K$,

  * •

every weak cluster point of $\\{x_{n}\\}$ belongs to $K$.

Then $\\{x_{n}\\}$ converges weakly to a point in $K$.

######  Definition 2.10.

We say that $x$ is a stationary point of $p$ in DC programming (1) if

$0\in\partial f(x)+\partial g(x)-\nabla h(x).$

We shall denote by $\Omega$, the set of all the stationary points of $p$ in DC
programming (1) and consequently assume that $\Omega\neq\emptyset$ throughout
this paper.

In DC programming (1), assume for the rest of this paper that $\beta>0$ is
given, $f$ and $g$ are $\rho$-strongly convex, and $h$ is a smooth convex
function having a Lipschitz continuous gradient with constant $L>0$. The
following results have been established in [24, Proposition 3.1] regarding the
set of stationary points of DC programming (1).

######  Lemma 2.11.

If $2\rho>L$ in DC programming (1), then the stationary point of $p$ is
unique.

######  Lemma 2.12.

([11, Proposition 3.1]) Suppose $\beta>0$. Then $x\in\Omega$ if and only if
there exists $y\in H$ such that

$\displaystyle\left\\{\begin{array}[c]{ll}x&={\rm prox}_{\beta f}(y),\\\
x&={\rm prox}_{\beta f}(2x-y+\beta\nabla h(x)).\end{array}\right.$

In order to obtain our convergence analysis of the proposed Algorithm 1, we
give the following conditions.

######  Assumption 2.13.

Suppose the iterative parameters in Algorithm 1 fulfill these conditions:

  1. (i)

$2\rho>L$;

  2. (ii)

$0<a\leq\kappa_{n}\leq b<2$.

Going by Lemma 2.12, given $\beta>0$, we set

$\Gamma_{\beta}:=\\{y:{\rm prox}_{\beta f}(y)=x,{\rm prox}_{\beta
f}(2x-y+\beta\nabla h(x))=x,x\in\Omega\\}.$

##  3 Proposed Algorithm and Its Convergence

In this section, we introduce the proposed algorithms and obtain associated
weak convergence results to the critical point of DC programming (1).

Algorithm 1 First Version of Douglas-Rachford Algorithm

1:Initialization: Choose $\beta>0,\theta\geq 0$, and $x_{0},v_{0}\in H$
arbitrarily. Set $n=0.$

2:Iteration Step: Given the iterates $x_{n},v_{n}$, compute
$\displaystyle\left\\{\begin{array}[c]{ll}&u_{n}=\frac{1}{1+\theta}x_{n}+\frac{\theta}{1+\theta}v_{n},\\\
&y_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-u_{n}\|^{2}\Big{\\}},\\\
&z_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2y_{n}-u_{n}+\beta\nabla
h(y_{n}))\|^{2}\Big{\\}},\\\ &x_{n+1}=u_{n}+\kappa_{n}(z_{n}-y_{n}),\\\
&v_{n+1}=\frac{1}{1+\theta}x_{n+1}+\frac{\theta}{1+\theta}v_{n}.\end{array}\right.$
(22) Set $n\leftarrow n+1,$ and go to Iteration Step if the stopping criterion
is not met.

######  Remark 3.1.

If $\theta=0$ in Algorithm 1, then $u_{n}=x_{n}$ and $v_{n+1}=x_{n+1}$.
Consequently, we obtain [11, Algorithm 1]. Therefore, [11, Algorithm 1] is
recovered from Algorithm 1 when $\theta=0$. It suffices to obtain the weak
convergence of Algorithm 1 when $\theta>0$ in Theorem 3.3 below.

We can easily obtain this lemma by following same arguments as in [11, Theorem
3.1]. For the sake of completeness and to make our results reader-friendly, we
include the complete proof.

######  Lemma 3.2.

Suppose $\beta,\kappa>0$, for each $u\in H$, let

$\displaystyle\left\\{\begin{array}[c]{ll}&y=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-u\|^{2}\Big{\\}},\\\
&z=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2y-u+\beta\nabla
h(y))\|^{2}\Big{\\}},\\\ &x=u+\kappa(z-y).\end{array}\right.$ (26)

Then for any $\bar{x}\in\Omega$, there exists $\bar{y}\in\Gamma_{\beta}$ such
that

$\displaystyle\|x-\bar{y}\|^{2}$ $\displaystyle\leq$
$\displaystyle\|u-\bar{y}\|^{2}-\beta\kappa(2\rho-L)\Big{(}\|y-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}\Big{)}$
(27) $\displaystyle-\kappa(2-\kappa)\|z-y\|^{2}.$

###### Proof.

Using the first-order optimality condition on $y={\rm prox}_{\beta f}(u)$ and
$z={\rm prox}_{\beta g}(2y-u+\beta\nabla h(y))$, we obtain

$\frac{1}{\beta}(u-y)\in\partial f(y),$ (28)

and

$\nabla h(y)+\frac{1}{\beta}(2y-z-u)\in\partial g(z)$ (29)

respectively. Now for any $\bar{x}\in\Omega$, there exists a
$\bar{y}\in\Gamma_{\beta}$ such that

$\displaystyle\frac{1}{\beta}(\bar{y}-\bar{x})\in\partial f(\bar{x})$ (30)

and

${\rm and}\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{1}{\beta}(\bar{x}-\bar{y})+\nabla h(\bar{x})\in\partial g(\bar{x}).$
(31)

By observing that $f$ is $\rho$-strongly convex, we have from Lemma 2.6 that
$\partial f$ is $\rho$-strongly monotone. Consequently,

$\langle\zeta_{y}-\zeta_{\bar{x}},y-\bar{x}\rangle\geq\rho\|y-\bar{x}\|^{2},\leavevmode\nobreak\
\leavevmode\nobreak\ \forall\zeta_{y}\in\partial
f(y),\zeta_{\bar{x}}\in\partial f(\bar{x}).$ (32)

If we apply (32) in (28) and (30), we obtain

$\rho\|y-\bar{x}\|^{2}\leq\frac{1}{\beta}\langle
y-\bar{x},u-y-\bar{y}+\bar{x}\rangle$

which implies that

$(2+2\beta\rho)\|y-\bar{x}\|^{2}\leq 2\langle y-\bar{x},u-\bar{y}\rangle.$
(33)

Similarly, let us apply the property that $g$ is $\rho$-strongly convex in
(29) and (31). Then we obtain

$\displaystyle\rho\|z-\bar{x}\|^{2}\leq\langle z-\bar{x},\nabla h(y)-\nabla
h(\bar{x})+\frac{1}{\beta}(2y-z-u)-\frac{1}{\beta}(\bar{x}-\bar{y})\rangle.$
(34)

If we rearrange (34), noting the Cauchy-Schwarz inequality and Lipschitz
continuity of $\nabla h$, we get

$\displaystyle 2\beta\rho\|z-\bar{x}\|^{2}$ $\displaystyle\leq$ $\displaystyle
2\beta\langle z-\bar{x},\nabla h(y)-\nabla
h(\bar{x})+\frac{1}{\beta}(2y-z-u)-\frac{1}{\beta}(\bar{x}-\bar{y})\rangle$
(35) $\displaystyle=$ $\displaystyle 2\beta\langle z-\bar{x},\nabla
h(y)-\nabla h(\bar{x})\rangle+2\langle z-\bar{x},2y-z-u-\bar{x}\rangle$
$\displaystyle\leq$ $\displaystyle 2\beta L\|z-\bar{x}\|\|y-\bar{x}\|+2\langle
z-\bar{x},2y-z-u-\bar{x}\rangle$ $\displaystyle\leq$ $\displaystyle\beta
L\|z-\bar{x}\|^{2}+\beta L\|y-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}$
$\displaystyle+\|2y-z-u+\bar{y}-\bar{x}\|^{2}-\|2z-2y+u-\bar{y}\|^{2}.$

Therefore, we have from (35) that

$\displaystyle\|2z-2y+u-\bar{y}\|^{2}$ $\displaystyle\leq$
$\displaystyle(1+\beta L-2\beta\rho)\|z-\bar{x}\|^{2}+\beta
L\|y-\bar{x}\|^{2}$ (36) $\displaystyle+\|2y-z-u+\bar{y}-\bar{x}\|^{2}.$

Now, consider (noting (33))

$\displaystyle\|2y-z-u+\bar{y}-\bar{x}\|^{2}$ $\displaystyle=$
$\displaystyle\|(y-z)+(y-\bar{x})-(u-\bar{y})\|^{2}$ (37) $\displaystyle=$
$\displaystyle\|y-z\|^{2}+\|y-\bar{x}\|^{2}+\|u-\bar{y}\|^{2}+2\langle
y-\bar{x},y-z\rangle$ $\displaystyle-2\langle u-y,y-z\rangle-2\langle
u-\bar{y},y-\bar{x}\rangle$ $\displaystyle=$ $\displaystyle
2\|y-z\|^{2}+2\|y-\bar{x}\|^{2}+\|u-\bar{y}\|^{2}-\|z-\bar{x}\|^{2}$
$\displaystyle-2\langle u-\bar{y},y-z\rangle-2\langle
u-\bar{y},y-\bar{x}\rangle$ $\displaystyle\leq$ $\displaystyle
2\|y-z\|^{2}+2\|y-\bar{x}\|^{2}+\|u-\bar{y}\|^{2}-\|z-\bar{x}\|^{2}$
$\displaystyle-2\langle u-\bar{y},y-z\rangle-(2+2\beta\rho)\|y-\bar{x}\|^{2}$
$\displaystyle=$ $\displaystyle
2\|y-z\|^{2}+\|u-\bar{y}\|^{2}-\|z-\bar{x}\|^{2}-2\beta\rho\|y-\bar{x}\|^{2}$
$\displaystyle-2\langle u-\bar{y},y-\bar{x}\rangle-2\langle
u-\bar{y},\bar{x}-z\rangle$ $\displaystyle\leq$ $\displaystyle
2\|y-z\|^{2}+\|u-\bar{y}\|^{2}-\|z-\bar{x}\|^{2}$
$\displaystyle-(2+4\beta\rho)\|y-\bar{x}\|^{2}+2\langle
u-\bar{y},z-\bar{x}\rangle.$

We next estimate the term $2\langle u-\bar{y},z-\bar{x}\rangle$ in (37). Now,
if we look at (34) and we rearrange, we obtain

$\displaystyle 2\langle u-\bar{y},z-\bar{x}\rangle$ $\displaystyle\leq$
$\displaystyle 2\beta\langle z-\bar{x},\nabla h(y)-\nabla
h(\bar{x})\rangle+2\langle z-\bar{x},y-z\rangle$ (38) $\displaystyle+2\langle
z-\bar{x},y-\bar{x}\rangle-2\beta\rho\|z-\bar{x}\|^{2}$ $\displaystyle\leq$
$\displaystyle\beta L\|z-\bar{x}\|^{2}+\beta
L\|y-\bar{x}\|^{2}+\|y-\bar{x}\|^{2}-\|z-y\|^{2}$
$\displaystyle-\|z-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}+\|y-\bar{x}\|^{2}-\|z-y\|^{2}$
$\displaystyle-2\beta\rho\|z-\bar{x}\|^{2}$ $\displaystyle=$
$\displaystyle(\beta L-2\beta\rho)\|z-\bar{x}\|^{2}+(2+\beta
L)\|y-\bar{x}\|^{2}$ $\displaystyle-2\|z-y\|^{2}.$

Plugging (38) into (37), we get

$\displaystyle\|2y-z-u+\bar{y}-\bar{x}\|^{2}$ $\displaystyle\leq$
$\displaystyle 2\|y-z\|^{2}+\|u-\bar{y}\|^{2}-\|z-\bar{x}\|^{2}$ (39)
$\displaystyle-(2+4\beta\rho)\|y-\bar{x}\|^{2}+(\beta
L-2\beta\rho)\|z-\bar{x}\|^{2}$ $\displaystyle+(2+\beta L)\|y-\bar{x}\|^{2}$
$\displaystyle-2\|z-y\|^{2}$ $\displaystyle=$
$\displaystyle\|u-\bar{y}\|^{2}+(\beta L-2\beta\rho-1)\|z-\bar{x}\|^{2}$
$\displaystyle+(\beta L-4\beta\rho)\|y-\bar{x}\|^{2}.$

Now, let us plug (39) into (36) to obtain

$\displaystyle\|2z-2y+u-\bar{y}\|^{2}$ $\displaystyle\leq$
$\displaystyle(1+\beta L-2\beta\rho)\|z-\bar{x}\|^{2}+\beta
L\|y-\bar{x}\|^{2}$ $\displaystyle+\|u-\bar{y}\|^{2}+(\beta
L-2\beta\rho-1)\|z-\bar{x}\|^{2}$ $\displaystyle+(\beta
L-4\beta\rho)\|y-\bar{x}\|^{2}$ $\displaystyle=$
$\displaystyle\|u-\bar{y}\|^{2}-2\beta(2\rho-L)\Big{(}\|y-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}\Big{)}.$

We then obtain from $x=u+\kappa(z-y)$ that (noting immediate last inequality)

$\displaystyle\|x-\bar{y}\|^{2}$ $\displaystyle=$
$\displaystyle\|u+\kappa(z-y)-\bar{y}\|^{2}$ $\displaystyle=$
$\displaystyle\|\Big{(}1-\frac{\kappa}{2}\Big{)}(u-\bar{y})+\frac{\kappa}{2}(2z-2y+u-\bar{y})\|^{2}$
$\displaystyle=$
$\displaystyle\Big{(}1-\frac{\kappa}{2}\Big{)}\|u-\bar{y}\|^{2}+\frac{\kappa}{2}\|2z-2y+u-\bar{y}\|^{2}$
$\displaystyle-\frac{\kappa}{2}\Big{(}1-\frac{\kappa}{2}\Big{)}\|(2z-2y+u-\bar{y})-(u-\bar{y})\|^{2}$
$\displaystyle=$
$\displaystyle\Big{(}1-\frac{\kappa}{2}\Big{)}\|u-\bar{y}\|^{2}+\frac{\kappa}{2}\|2z-2y+u-\bar{y}\|^{2}$
$\displaystyle-\kappa(2-\kappa)\|z-y\|^{2}$ $\displaystyle\leq$
$\displaystyle\Big{(}1-\frac{\kappa}{2}\Big{)}\|u-\bar{y}\|^{2}+\frac{\kappa}{2}\|u-\bar{y}\|^{2}$
$\displaystyle-\beta\kappa(2\rho-L)\Big{(}\|y-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}\Big{)}-\kappa(2-\kappa)\|z-y\|^{2}$
$\displaystyle=$
$\displaystyle\|u-\bar{y}\|^{2}-\beta\kappa(2\rho-L)\Big{(}\|y-\bar{x}\|^{2}+\|z-\bar{x}\|^{2}\Big{)}$
$\displaystyle-\kappa(2-\kappa)\|z-y\|^{2}.$

∎

We now give the following weak convergence result of Algorithm 1 using Lemma
3.2.

######  Theorem 3.3.

The sequence $\\{x_{n}\\}$ generated by Algorithm 1 converges weakly to a
point in $\Gamma_{\beta}$ under Assumption 2.13.

###### Proof.

We shall only consider the case $\theta>0$ in our convergence analysis since
the case $\theta=0$ has already been considered in [11, Algorithm 1] with
convergence results given in [11, Theorem 3.1]. Now by Lemma 2.8 (ii), we have
from Algorithm 1 that (putting $\bar{y}=y$ and $\bar{x}=x$ in Lemma 3.2)

$\displaystyle\|u_{n}-y\|^{2}$ $\displaystyle=$
$\displaystyle\big{\|}\frac{1}{1+\theta}(x_{n}-y)+\frac{\theta}{1+\theta}(v_{n}-y)\big{\|}^{2}$
(40) $\displaystyle=$
$\displaystyle\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\frac{\theta}{1+\theta}\|v_{n}-y\|^{2}-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}\qquad$

and

$\displaystyle\|v_{n+1}-y\|^{2}$ $\displaystyle=$
$\displaystyle\frac{\theta}{1+\theta}\|v_{n}-y\|^{2}+\frac{1}{1+\theta}\|x_{n+1}-y\|^{2}-\frac{\theta}{(1+\theta)^{2}}\|x_{n+1}-v_{n}\|^{2}.\qquad$
(41)

If we plug (40) into (27) in Lemma 3.2, we have

$\displaystyle\|x_{n+1}-y\|^{2}$ $\displaystyle\leq$
$\displaystyle\|u_{n}-y\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
(42) $\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\frac{\theta}{1+\theta}\|v_{n}-y\|^{2}$
$\displaystyle-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}.$

We then obtain from (41) and (42) that

$\displaystyle\|x_{n+1}-y\|^{2}$ $\displaystyle+$
$\displaystyle\theta\|v_{n+1}-y\|^{2}\leq\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\frac{\theta}{1+\theta}\|v_{n}-y\|^{2}$
(43)
$\displaystyle-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}+\frac{\theta}{1+\theta}\|x_{n+1}-y\|^{2}+\frac{\theta^{2}}{1+\theta}\|v_{n}-y\|^{2}$
$\displaystyle-\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n+1}-v_{n}\|^{2}.$

Therefore,

$\displaystyle\frac{1}{1+\theta}\|x_{n+1}-y\|^{2}+\theta\|v_{n+1}-y\|^{2}+\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n+1}-v_{n}\|^{2}$
(44) $\displaystyle\leq$
$\displaystyle\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\theta\|v_{n}-y\|^{2}-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}$
$\displaystyle-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}+\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}$
$\displaystyle-\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\theta\|v_{n}-y\|^{2}+\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}$
$\displaystyle-\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}.$

Define

$\displaystyle c_{n}:$ $\displaystyle=$
$\displaystyle\frac{1}{1+\theta}\|x_{n}-y\|^{2}+\theta\|v_{n}-y\|^{2}$ (45)
$\displaystyle+\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}.$

Then, (44) becomes

$\displaystyle c_{n+1}$ $\displaystyle\leq$ $\displaystyle
c_{n}-\frac{\theta^{2}}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
(46)
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}.$

Thus, $\\{c_{n}\\}$ is monotone non-increasing and $\\{c_{n}\\}$ is
convergent. Moreover, $\\{c_{n}\\}$ is bounded. Furthermore, we obtain from
(46) that

$\underset{n\rightarrow\infty}{\lim}\|x_{n}-v_{n-1}\|=0,\leavevmode\nobreak\
\leavevmode\nobreak\ \underset{n\rightarrow\infty}{\lim}\|x_{n}-v_{n}\|=0.$

Furthermore, we have from (46) that

$\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}\leq
c_{n}-c_{n+1},$

which implies that

$\underset{n\rightarrow\infty}{\lim}\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}=0.$

Since $2\rho>L$ and $0<a\leq\kappa_{n}\leq b<2$, we get

$\underset{n\rightarrow\infty}{\lim}\|y_{n}-x\|=0,\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm and}\leavevmode\nobreak\ \leavevmode\nobreak\
\underset{n\rightarrow\infty}{\lim}\|z_{n}-x\|=0.$ (47)

Let us define

$s_{n}:=\|v_{n}-x_{n}\|^{2}+2\langle v_{n}-x_{n},x_{n}-y\rangle$

and

$t_{n}:=-\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n-1}\|^{2}.$

Since $\\{x_{n}\\}$ is bounded and
$\underset{n\rightarrow\infty}{\lim}\|x_{n}-v_{n}\|=0$, we have
$\underset{n\rightarrow\infty}{\lim}s_{n}=0$ and $t_{n}\rightarrow 0$, as
$n\rightarrow\infty$. Noting that

$\displaystyle\|v_{n}-y\|^{2}$ $\displaystyle=$
$\displaystyle\|v_{n}-x_{n}\|^{2}+2\langle v_{n}-x_{n},x_{n}-y\rangle$
$\displaystyle+\|x_{n}-y\|^{2}$

and

$\displaystyle s_{n}$ $\displaystyle=$
$\displaystyle\|v_{n}-x_{n}\|^{2}+2\langle v_{n}-x_{n},x_{n}-y\rangle$
$\displaystyle=$ $\displaystyle\|v_{n}-y\|^{2}-\|x_{n}-y\|^{2}.$

We then obtain

$c_{n}-\theta s_{n}+\theta
t_{n}=\frac{\theta^{2}+\theta+1}{1+\theta}\|x_{n}-y\|^{2}.$

By the existence of the limit $\underset{n\rightarrow\infty}{\lim}c_{n}$, we
get $\underset{n\rightarrow\infty}{\lim}\|x_{n}-y\|$ exists. We conclude by
Lemma 2.9 that $\\{x_{n}\\}$ converges weakly to a point in $\Gamma_{\beta}$.
Consequently, $\\{v_{n}\\}$ also converges weakly to a point in
$\Gamma_{\beta}$. This completes the proof.

∎

######  Remark 3.4.

One can obtain from (46) that

$\frac{\theta}{(1+\theta)^{2}}\|x_{n}-v_{n}\|^{2}\leq c_{n}-c_{n+1}$

and thus

$\frac{\theta}{(1+\theta)^{2}}\sum_{j=0}^{n}\|x_{j}-v_{j}\|^{2}\leq\sum_{j=0}^{n}(c_{n}-c_{n+1})\leq
c_{0},$

which implies that (if $x_{0}=v_{0}$)

$(n+1)\frac{\theta}{(1+\theta)^{2}}\min_{0\leq j\leq n}\|x_{j}-v_{j}\|^{2}\leq
c_{0}=\frac{1+\theta(1+\theta)}{1+\theta}\|x_{0}-y\|^{2}.$

Consequently, we have

$\min_{0\leq j\leq n}\|x_{j}-v_{j}\|\leq\mathcal{O}(\frac{1}{\sqrt{n+1}}).$

In the same way, one can obtain

$\min_{0\leq j\leq
n}\|x_{j}-v_{j-1}\|\leq\mathcal{O}(\frac{1}{\sqrt{n+1}}),\min_{0\leq j\leq
n}\|z_{j}-y_{j}\|\leq\mathcal{O}(\frac{1}{\sqrt{n+1}}).$

We design another version of Douglas-Rachford splitting algorithm for DC
programming (1) and obtain weak convergence results.

Algorithm 2 Second Version of Douglas-Rachford Algorithm

1:Initialization: Choose $\alpha_{n}\in[0,1)$, $\beta>0$, and $x_{0},v_{0}\in
H$ arbitrarily. Set $n=0.$

2:Iteration Step: Given the iterates $x_{n},v_{n}$, compute
$\displaystyle\left\\{\begin{array}[c]{ll}&u_{n}=(1-\alpha_{n})x_{n}+\alpha_{n}v_{n},\\\
&y_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-u_{n}\|^{2}\Big{\\}},\\\
&z_{n}=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2y_{n}-u_{n}+\beta\nabla
h(y_{n}))\|^{2}\Big{\\}},\\\ &x_{n+1}=u_{n}+\kappa_{n}(z_{n}-y_{n}),\\\
&v_{n+1}=(1-\alpha_{n})v_{n}+\alpha_{n}x_{n}.\end{array}\right.$ (53) Set
$n\leftarrow n+1,$ and go to Iteration Step if the stopping criterion is not
met.

Since our Algorithm 2 reduces to [11, Algorithm 1] when $\alpha_{n}=0$ for
each $n\in\mathbb{N}$ for which the convergence results have been obtained in
[11]. It suffices to only consider the case $0<a\leq\alpha_{n}\leq b<1$ in our
convergence result below for Algorithm 2.

######  Theorem 3.5.

The sequence $\\{x_{n}\\}$ generated by Algorithm 2 converges weakly to a
point in $\Gamma_{\beta}$ under Assumption 2.13 when either
$0<a\leq\alpha_{n}\leq b<1$ or $\alpha_{n}=0$ for each $n\in\mathbb{N}$.

###### Proof.

We can easily obtain from Algorithm 2 and Lemma 3.2 that (with $\bar{y}=y$)

$\displaystyle\|x_{n+1}-y\|^{2}$ $\displaystyle\leq$
$\displaystyle\|u_{n}-y\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
(54) $\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}.$

By Lemma 2.8(ii), we have

$\displaystyle\|u_{n}-y\|^{2}$ $\displaystyle=$
$\displaystyle\big{\|}(1-\alpha_{n})(x_{n}-y)+\alpha_{n}(v_{n}-y)\big{\|}^{2}$
(55) $\displaystyle=$
$\displaystyle(1-\alpha_{n})\|x_{n}-y\|^{2}+\alpha_{n}\|v_{n}-y\|^{2}-\alpha_{n}(1-\alpha_{n})\|x_{n}-v_{n}\|^{2}\qquad$

and

$\displaystyle\|v_{n+1}-y\|^{2}$ $\displaystyle=$
$\displaystyle\|(1-\alpha_{n})(v_{n}-y)+\alpha_{n}(x_{n}-y)\|^{2}$ (56)
$\displaystyle=$
$\displaystyle(1-\alpha_{n})\|v_{n}-y\|^{2}+\alpha_{n}\|x_{n}-y\|^{2}-\alpha_{n}(1-\alpha_{n})\|x_{n}-v_{n}\|^{2}.\qquad$

Plugging (55) into (54), we obtain

$\displaystyle\|x_{n+1}-y\|^{2}$ $\displaystyle\leq$
$\displaystyle(1-\alpha_{n})\|x_{n}-y\|^{2}+\alpha_{n}\|v_{n}-y\|^{2}$ (57)
$\displaystyle-\alpha_{n}(1-\alpha_{n})\|x_{n}-v_{n}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
$\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2}.$

Denote

$a_{n}:=\|x_{n}-y\|^{2}+\|v_{n}-y\|^{2},\quad\forall n\in\mathbb{N}.$ (58)

Then, summing (56) and (57) gives

$\displaystyle a_{n+1}$ $\displaystyle\leq$ $\displaystyle
a_{n}-2\alpha_{n}(1-\alpha_{n})\|x_{n}-v_{n}\|^{2}-\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}$
(59) $\displaystyle-\kappa_{n}(2-\kappa_{n})\|z_{n}-y_{n}\|^{2},$

which, by $\alpha_{n}\in[0,1)$, and $\kappa_{n}\in(0,2)$, implies that
$\\{a_{n}\\}$ is a monotone non-increasing and $\\{a_{n}\\}$ is convergent.
Moreover, $\\{a_{n}\\}$ is bounded. We then obtain from the definition of
$\\{a_{n}\\}$ that both $\\{x_{n}\\}$ and $\\{v_{n}\\}$ are bounded.
Furthermore, $\\{u_{n}\\}$, $\\{y_{n}\\}$ and $\\{z_{n}\\}$ are also bounded.

We also obtain from (59) that

$\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}\leq
a_{n}-a_{n+1},$

which implies that

$\underset{n\rightarrow\infty}{\lim}\beta\kappa_{n}(2\rho-L)\Big{(}\|y_{n}-x\|^{2}+\|z_{n}-x\|^{2}\Big{)}=0.$

Since $2\rho>L$ and $0<a\leq\kappa_{n}\leq b<2$, we get

$\underset{n\rightarrow\infty}{\lim}\|y_{n}-x\|=0,\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm and}\leavevmode\nobreak\ \leavevmode\nobreak\
\underset{n\rightarrow\infty}{\lim}\|z_{n}-x\|=0.$ (60)

By the condition that
$\underset{n\rightarrow\infty}{\liminf}\kappa_{n}(2-\kappa_{n})>0$, we also
have from (59) that

$\underset{n\rightarrow\infty}{\lim}\|z_{n}-y_{n}\|=0.$

Also from (59),

$\underset{n\rightarrow\infty}{\lim}2\alpha_{n}(1-\alpha_{n})\|x_{n}-v_{n}\|^{2}=0.$

Hence,

$\underset{n\rightarrow\infty}{\lim}\|x_{n}-v_{n}\|=0.$

We have from $v_{n+1}=(1-\alpha_{n})v_{k}+\alpha_{n}x_{n}$ that

$v_{n+1}-v_{n}=\alpha_{n}(x_{n}-v_{n})\rightarrow 0,n\rightarrow\infty.$

So,

$\displaystyle\|x_{n+1}-x_{n}\|\leq\|x_{n+1}-v_{n+1}\|+\|v_{n+1}-v_{n}\|+\|x_{n}-v_{n}\|\rightarrow
0,n\rightarrow\infty.$

Since $\\{x_{n}\\}$, $\\{y_{n}\\}$ are $\\{z_{n}\\}$ are bounded, let $v\in H$
be a weak cluster point of $\\{x_{n}\\}$. Then there exists
$\\{x_{n_{j}}\\}\subset\\{x_{n}\\}$ such that $x_{n_{j}}\rightharpoonup v,$
$j\to\infty$. Then $x_{n_{j}+1}\rightharpoonup v,$ $j\to\infty$,
$z_{n_{j}}\rightharpoonup v,$ $j\to\infty$ and $u_{n_{j}}\rightharpoonup v,$
$j\to\infty$. Replacing $n$ with ${n_{j}}$ in (28) and (29), we have from (60)
and Lemma 2.7 that

$\frac{1}{\beta}(v-x)\in\partial f(x)\leavevmode\nobreak\ \leavevmode\nobreak\
{\rm and}\leavevmode\nobreak\ \leavevmode\nobreak\ \nabla
h(x)+\frac{1}{\beta}(x-v)\in\partial g(x).$

This implies that ${\rm prox}_{\beta f}(v)=x$ and ${\rm prox}_{\beta
g}(2x-v+\beta\nabla h(x))=x$. We then obtain that $v\in\Gamma_{\beta}$.

We now show that $\\{x_{n}\\}$ converges weakly to an element in
$\Gamma_{\beta}$. Define

$\displaystyle c_{n}:=\|v_{n}-x_{n}\|^{2}+2\langle
v_{n}-x_{n},x_{n}-y\rangle.$

Since $v_{n}-x_{n}\rightarrow 0$, as $n\rightarrow\infty$ and $\\{x_{n}\\}$ is
bounded, we have that $c_{n}\rightarrow 0$, as $n\rightarrow\infty$. Observe
also that

$\displaystyle\|v_{n}-y\|^{2}=\|v_{n}-x_{n}\|^{2}+2\langle
v_{n}-x_{n},x_{n}-y\rangle+\|x_{n}-y\|^{2}$

which implies that

$c_{n}=\|v_{n}-y\|^{2}-\|x_{n}-y\|^{2}.$

Combine it with the definition of $a_{n}$ to have

$\|x_{n}-y\|^{2}=\frac{1}{2}\big{(}a_{n}-c_{n}\big{)}.$

Since $\lim_{n\rightarrow\infty}a_{n}$ exists, then
$\lim_{k\rightarrow\infty}\|x_{n}-y\|$ exists. By Lemma 2.9, we conclude that
$\\{x_{n}\\}$ converges weakly to a point in $\Gamma_{\beta}$. The proof is
complete. ∎

######  Remark 3.6.

The proposed Algorithm 2 can be viewed as an extension of an inertial-type
algorithm. In particular, the following is a special case of (53) in Algorithm
2:

$\displaystyle\left\\{\begin{array}[c]{ll}&u_{n-1}=v_{n}+\theta_{n-1}(v_{n}-v_{n-1}),\\\
&y_{n-1}=\underset{v\in H}{\rm
argmin}\Big{\\{}f(v)+\frac{1}{2\beta}\|v-u_{n-1}\|^{2}\Big{\\}},\\\
&z_{n-1}=\underset{v\in H}{\rm
argmin}\Big{\\{}g(v)+\frac{1}{2\beta}\|v-(2y_{n-1}-u_{n-1}+\beta\nabla
h(y_{n-1}))\|^{2}\Big{\\}},\\\
&x_{n}=u_{n-1}+\kappa_{n-1}(z_{n-1}-y_{n-1}),\\\
&v_{n+1}=(1-\alpha_{n})v_{n}+\alpha_{n}x_{n}\end{array}\right.$ (66)

where $\theta_{n}:=\frac{1-2\alpha_{n}}{\alpha_{n}}\in(-1,+\infty)$ and
$0<a\leq\alpha_{n}\leq b<1$. To see this, suppose $\bar{u}_{n}:=v_{n}-x_{n}$.
Then the first update in (53) implies

$u_{n}=v_{n+1}+\theta_{n}(v_{n+1}-v_{n})=x_{n}+\bar{u}_{n}+(1+\theta_{n})(v_{n+1}-v_{n}).$
(67)

We obtain from the last update in (53) that
$v_{n+1}-v_{n}=-\alpha_{n}\bar{u}_{n}.$ If we substitute this into (67)
together with $\theta_{n}=\frac{1-2\alpha_{n}}{\alpha_{n}}$, we have

$\displaystyle u_{n}$ $\displaystyle=$ $\displaystyle
x_{n}+(1-\alpha_{n}-\theta_{n}\alpha_{n})\bar{u}_{n}$ $\displaystyle=$
$\displaystyle x_{n}+(1-\alpha_{n}-\theta_{n}\alpha_{n})(v_{n}-x_{n})$
$\displaystyle=$ $\displaystyle(1-\alpha_{n})x_{n}+\alpha_{n}v_{n}.$

Since $\theta_{n}=\frac{1-2\alpha_{n}}{\alpha_{n}}\in(-1,+\infty)$, we have
$\alpha_{n}=\frac{1}{2+\theta_{n}}\in(0,1)\subset[0,1).$ Therefore, (66) is a
special case of (53).

##  4 Numerical Experiments

The numerical experiments is devoted to the implementation of the proposed
algorithms in solving real-life problems. Three examples are considered where
we applied the algorithms to solve optimization models in machine learning and
compare their performance with other methods in the literature such as [11,
Algorithm 1] and [24, Algorithm 2] and the popular ADMM method. All codes are
written on Python using Jupyter Notebook. Also, the datasets used are source
from open source such as the UCI database.

###  4.1 Tuning of parameters

It is well known that control parameters are very important to the performance
of the iterative algorithms. Hence, we start the numerical section by studying
the behaviour of the proposed algorithms to change in the control parameters.
For this purpose, we tune the values of the parameters $\theta$ and $\beta$ in
the Douglas-Rashford splitting Algorithm 1. We consider the following
optimization model:

$\underset{x\in\mathbb{R}^{n}}{\min}\leavevmode\nobreak\ f(x)+g(x)-h(x)$ (68)

where $f:\mathbb{R}^{n}\to[+\infty,-\infty)$ is the quadratic function defined
by $f(x)=\frac{1}{2}x^{\top}Qx+c^{\top}x+d,$ with $Q\in\mathbb{R}^{M\times N}$
is a positive definite matrix, $c\in\mathbb{R}^{N}$ is a constant vector and
$d\in\mathbb{R}$ is a constant scalar; $g:\mathbb{R}^{n}\to\mathbb{R}$ is the
$\ell_{1}$ regularization term and $h:\mathbb{R}^{n}\to\mathbb{R}$ is the
Huber loss function given by

$h(x)=\begin{cases}\frac{1}{2}x^{2},\qquad&\|x\|_{2}\leq\delta,\\\
\delta(|x|-\frac{1}{2}\delta),&\mbox{otherwise}\end{cases}$

and $\delta$ being a constant scalar. This type of model is very common in
machine learning. For example, in robust classification with sparsity
regularization (see, e.g. [52]), problem (68) is used to learn a robust
classifier that is also sparse. The Huber loss function is used in this case
to handle noisy data, the quadratic term is used to regularize the model
complexity and the $\ell_{1}$ regularization term is used to induce sparsity
in the model parameters. Also problem (68) is used in to reconstruct a signal
from noisy measurements while promoting sparsity in signal processing (see,
e.g. [8]). The quadratic fidelity term is used to measure the fidelity of the
reconstructed signal, the $\ell_{1}$ regularization term is used to enforce
sparsity and the Huber loss function is used to handle outliers or noise in
the measurement.

The proximal operator of the quadratic term is calculated analytically, which
is given by

${\rm prox}_{\beta
f}(x)=\left(Q+\frac{1}{\beta}I\right)^{-1}\left(\frac{1}{\beta}x-c\right),$

where $I$ is the identity operator (see Appendix for details). To handle the
inverse of the matrix $Q$ in the case where it is a non-square matrix, we used
the pseudo-inverse $Q^{+}$ of $Q$ which is calculated by
$Q^{+}=(Q^{\top}Q)^{-1}Q^{\top},$ where $Q^{+}$ is the transpose of $Q$. This
allows us to handle non-square of $Q$ properly. In the first experiment, we
take $\kappa_{n}=\kappa=0.009,\delta=0.001$, $\max\\_\mbox{iter}=1000.$ The
algorithm is stopped if $Err=\|x_{n+1}-x_{n}\|^{2}<10^{-5}$ or
$\max\\_\mbox{iter}$ is reached. In the first instance, we choose
$\theta=0.0005,0.05,0.5,1,5$ and vary the values of $(M,N)$ as
$(100,25),\leavevmode\nobreak\ (200,100),\leavevmode\nobreak\ (250,150)$ and
$(300,200)$. The starting points $x_{0}$ and $v_{0}$ are generated randomly.
The results of the experiments are shown in Table 1 and Figure 1.

Furthermore, taking $\theta=0.0005,$ we test the algorithm for
$\beta=0.0001,0.001,0.1,1,10$ and vary the value of $(M,N)$ as
$(100,25),\leavevmode\nobreak\ (200,100),\leavevmode\nobreak\ (300,150),$
(400,200), (500,300) and $(600,400).$ The results are shown in Table 2 and 2.

Figure 1:  Comparing the performance of proposed Douglas-Rashford splitting
algorithm for various values of $\theta$ in Example 4.1 using various
dimension of $(M,N)$. Instance 1: (100,25); Instance 2: (200,100), Instance 3:
(300,150), Instance 4: (400, 200), Instance 5: (500,300) and Instance 6:
(600,400).

Figure 2:  Comparing the performance of proposed Douglas-Rashford splitting algorithm for various values of $\beta$ in Example 4.1 using various dimension of $(M,N)$. Instance 1: (100,25); Instance 2: (200,100), Instance 3: (300,150), Instance 4: (400, 200), Instance 5: (500,300) and Instance 6: (600,400). $(M,N)$ | $\theta=0.0005$ | $\theta=0.05$ | $\theta=0.5$ | $\theta=1$ | $\theta=5$
---|---|---|---|---|---
| Iter/Time | Iter/Time | Iter/Time | Iter/Time | Iter/Time
(100,25) | 201/ 0.0712 | 177/0.0532 | 206/0.0473 | 250/0.0626 | 406/0.1098
(200,100) | 227/0.2670 | 244/0.3138 | 258/0.3298 | 304/0.4152 | 579/0.6904
(300,150) | 253/0.276 | 233/0.2830 | 288/0.3451 | 324/0.3999 | 622/0.8319
(400,250) | 275/0.5952 | 259/0.5378 | 311/0.6616 | 333/4.2368 | 706/1.5850
(600,300) | 242/2.7268 | 254/2.5572 | 273/2.8729 | 333/4.2368 | 720/9.0847
(600,400) | 264/6.1672 | 262/5.8376 | 289/6.8894 | 344/8.6013 | 786/26.6056
Table 1: Results of varying parameter $\theta$ in the proposed Douglas-Rashford splitting algorithm 1. $(M,N)$ | $\beta=0.0001$ | $\beta=0.001$ | $\beta=0.1$ | $\beta=1$ | $\beta=10$
---|---|---|---|---|---
| Iter/Time | Iter/Time | Iter/Time | Iter/Time | Iter/Time
(100,25) | 2/0.0013 | 163/0.0584 | 162/0.0373 | 158/0.0313 | 164/0.0629
(200,100) | 2/0.0020 | 528/0.3301 | 216/0.1566 | 199/0.1415 | 203/0.1254
(300,150) | 374/0.8984 | 554/1.2979 | 214/0.5333 | 205/0.4861 | 219/0.6669
(400,250) | 453/1.1677 | 611/1.7133 | 235/0.6431 | 214/0.5807 | 228/ 0.6119
(600,300) | 881/9.2771 | 623/6.1512 | 235/2.3229 | 228/2.2761 | 247/2.9510
(600,400) | 1000/17.0568 | 718/12.6837 | 245/4.8480 | 236/4.2068 | 259/5.3359
Table 2: Results of varying stepsize $\beta$ in the proposed Douglas-Rashford
splitting algorithm 1.

In the initial experiment, where we assess the performance of the algorithm
across a range of $\theta$ values, we observe some distinct trends in the
behaviour of the algorithm. Notably, we find that the effectiveness of the
algorithm diminishes as $\theta$ exceeds a critical threshold $(\theta>1)$,
signaling a clear degradation in performance. This decline is further worsen
with increase in the values of both $M$ and $N$, providing evidence of an
overestimation in the control parameter. Conversely, a noteworthy observation
emerges when the value of $\theta$ is smaller. The algorithm exhibits enhanced
performance with smaller values of $\theta$, even with larger values dimension
of $M$ and $N$. Additionally, upon closer examination, we identify that the
algorithm reaches its best performance when $\theta$ is set to 0.05,
underscoring the importance of optimal parameter selection in maximizing the
effectiveness of the proposed algorithm.

In the subsequent experiment, dedicated to evaluating the performance of the
algorithm across different values of the stepsize $\beta$, some intriguing
behaviour emerge. Particularly, for smaller values of $N$ and $M$, we observe
improved algorithmic performance with decreased $\beta$, notably with
$\beta=0.0001$. However, as the values of $M$ and $N$ increase, a drastic
deterioration in algorithmic efficacy is noted for smaller $\beta$ values,
indicative of an underestimation in stepsize. Interestingly, regardless of the
specific values of $M$ and $N$, the algorithm consistently achieves better
performance with large value of $\beta$. It is noted that the best performance
of the algorithm is achieved when $\beta=1.$

###  4.2 Regularized least squares problem with the logarithmic regularizer

Next, we shall consider the regularized least square (RLS) model with
logarithmic regularizer given as follows:

$\min_{w\in\mathbb{R}^{N}}\mathcal{J}(w)=\frac{1}{2}\|Aw-b\|^{2}+\sum_{i=1}^{N}(\mu\log(|w_{i}|+\epsilon)-\mu\log\epsilon),$
(69)

where $A\in\mathbb{R}^{m\times N},b\in\mathbb{R}^{m}$, $\epsilon>0$ is a
constant and $\mu>0$ is the regularization parameter. RLS with logarithmic
regularizer is very common in machine learning, mainly for preventing
overfitting in models, see, e.g. [22, 5, 33]. It is easy to see that the model
(69) is a special case of DC programming (1) with
$f(w)=\frac{1}{2}\|Aw-b\|^{2}$, $g(w)=\frac{\mu}{\epsilon}\|w\|_{1}$ and
$h(w)=\sum_{i=1}^{N}\mu\Big{(}\frac{|w_{i}|}{\epsilon}-\log(|w_{i}|+\epsilon)+\log\epsilon\Big{)}$.
The first-order optimization condition of the first subproblem in Algorithm 1
and Algorithm 2 give ([24])

$A^{*}(Ay_{n}-b)+\frac{1}{\beta}(y_{n}-u_{n})=0,$

which yields

$(\beta A^{*}A+I)y_{n}=\beta A^{*}b+u_{n},$

and can be solved effectively by the LU factorization method, the conjugate
gradient method and so on [6].

In our experiments, we test the performance of the proposed algorithms for
finding the minimizer of (69) and compare the results with other methods in
the literature. The matrices $A\in\mathbb{R}^{m\times N}$ and
$b\in\mathbb{R}^{m}$ are generated randomly with i.i.d. standard Gaussian
entries, and then normalized so that the columns of $A$ have unit norms. Also
the vector $b$ is generated randomly such that $b$ is non-zero vector. The
proposed Algorithm 1 and 2 are compared with [37] (DCA) and [11, Algorithm 1]
(GDCP). We initialize the algorithms by generating the starting points
randomly, $\mu=0.001,\epsilon=0.5,$
$max\\_iter=1000,\beta=0.04,\kappa_{n}=\frac{n}{n+10},$ $\theta=0.9$ for Alg
1, $\alpha_{n}=\frac{1}{n+1}$ for Alg 2 and $\alpha=0.09$ for GDCP. We stopped
the algorithms when

$Err=\frac{\|x_{n}-x_{n-1}\|}{\max\\{1,\|x_{n}\|\\}}<10^{-5}$

and record the number of iteration and CPU time taken for each algorithm. The
results of the experiments are recorded in Table 3 and Figure 3.

$(m,N)$ | Alg 1 | Alg 2 | DCA | GDCP
---|---|---|---|---
| Iter/Time | Iter/Time | Iter/Time | Iter/Time
(100,50) | 156/0.0708 | 212/0.0398 | 331/0.0785 | 1000/0.1567
(200,128) | 160/0.1762 | 217/0.1728 | 343/0.2659 | 1000/0.8564
(521,304) | 168/1.0966 | 228/1.4425 | 366/1.8522 | 760/3.8603
(700,500) | 169/6.3734 | 226/7.8930 | 365/12.1768 | 763/26.9791
(1000,700) | 171/11.5587 | 231/19.5223 | 369/30.5246 | 760/63.9168
(1500,1000) | 174/29.7912 | 236/47.7586 | 378/72.9514 | 759/145.9198
Table 3: Comparison of numerical results for Example 4.2.

Figure 3:  Graphical illustration of values of Err against number of
iterations by each method in Example 4.2. Instance 1: (100,50); Instance 2:
(250,128), Instance 3: (521,304), Instance 4: (700, 500), Instance 5:
(1000,700) and Instance 6: (1500,1000).

The numerical results show that efficiency and accuracy of the proposed
algorithms. Both proposed methods consistently have the best performance than
the comparing methods in terms of number of iterations and time of execution
irrespective of the size of the data size. This is a desirable result which
give credence to the importance of the proposed methods.

###  4.3 Support vector machines (SVM) with L1 regularization

Given a dataset $\\{(x_{i},y_{i})\\}_{i=1}^{n}$ where $x_{i}$ represents the
number of the features and $y_{i}$ is the corresponding label
$(+1\leavevmode\nobreak\ \mbox{or}\leavevmode\nobreak\ -1)$ of the $i$-th
sample, the primal form of the SVM optimization model can be written as

$\underset{w,b}{\mbox{minimize}}\leavevmode\nobreak\
\frac{1}{2}\|w\|_{2}^{2}+C\sum_{i=1}^{n}\leavevmode\nobreak\
\max\big{(}0,1-y_{i}(w^{\top}x_{i}+b)\big{)}+\lambda\|w\|_{1}$ (70)

where $w$ is the weight of vector, $b$ is the bias term, $\lambda$ is the
regularization parameter for the L1 penalty, $C$ is the regularization
parameter for the hinge loss function. The first term of the model represents
a convex function which deals with sparsity in the model, the second term is
the hinge loss function penalizing misclassification and the third term is the
L1 regularization term dealing with large weights. The objective function is a
good example of a generalized DC programming problem written as

$\min_{w,b}\leavevmode\nobreak\ f(w,b)+g(w)-h(w),$

where $f(w,b)=\|w\|_{2}^{2}+C\sum_{i=1}^{n}\max(0,1-y_{i}(w^{\top}x_{i}+b)),$
$g(w)=\lambda\|w\|_{1}$ and $h(w)=\frac{1}{2}\|w\|_{2}^{2}$.

The SVM was introduced by Vapnik [12, 48] as a kernel-based model used for
classification and regression tasks in machine learning. Its exceptional
generalization ability, optimal solution, and discriminative power have
garnered significant attention from the data mining, pattern recognition, and
machine learning communities in recent times. SVM has been used as a powerful
tool for addressing real-world binary classification problems. Researches have
demonstrated the superiority of SVMs over alternative supervised learning
techniques [47, 25, 49]. Due to their robust theoretical foundations and
remarkable generalization capabilities, SVMs have risen to prominence as one
of the most widely adopted classification methods in recent times. A detailed
theoretical explanation and formulation of the SVM can be found in, for
instance, [48, 5, 33, 22].

#### Related works

Considerable research works have been dedicated to exploring the applications
of DC programming across various domains including operational research,
machine learning, and economics. An intriguing application of DC programming
was presented in the 2006 paper of Argiou et al [1] titled ”A DC programming
algorithm for kernel selection”. The author discussed a greedy algorithm aimed
at learning kernels from a convex hull comprising basic kernels. While this
methodology had been previously introduced, it was confined to a finite set of
basic kernels. Additionally, the authors noted that the applicability of their
approach was constrained by the non-convex nature of a critical maximization
step involved in the learning process. However, they discovered that the
optimization problem could be reformulated as a DC program without solving the
problem. Another noteworthy paper discussed in this direction is the paper by
Thi et al [42]. The authors introduced an innovative application of DC
programming within the context of SVM algorithms. Their focus lay in the
optimal selection of representative features from data, a pivotal task in
enhancing the efficiency and interpretability of machine learning models. The
authors equate this problem to minimizing a zero norm function over step-k
feature vectors. The zero-norm function counts the number of non-zero elements
in the feature vector, essentially quantifying the sparsity of the selected
features. This innovative approach opens up new avenues for enhancing the
interpretability and performance of machine learning models in various
applications. In the paper [43], the authors leveraged DC decomposition
techniques to address a challenging optimization problem within the context of
machine learning. Specifically, they employed a DC decomposition-based
algorithm known as DCA (DC Algorithm) to efficiently find local minima of the
objective function. This algorithm iteratively updates the solution by solving
convex subproblems and performing a DC decomposition-based update step,
converging towards a local minimum of the objective function. The authors
applied this approach to ten datasets, some of which were particularly sparse.
In the thesis [34], the author conducted a comprehensive survey focusing on
the unified DC programming method for feature selection within the framework
of semi-supervised SVM and multiclass SVM. The primary algorithm emphasized in
the paper is based on the subgradient optimality principle. The unified DC
programming method discussed in the thesis aims to identify the most relevant
subset of features from high-dimensional data while optimizing the performance
of the SVM classifier, and demonstrating its applicability and efficacy across
different problem domains. In the more recent thesis by Ho [51], the author
delved into the application of DC programming for learning a kernel tailored
to a specific supervised learning task. Notably, what sets this approach apart
from others in the literature is its capability to handle an infinite set of
basic kernels. Other related results can be found in [44, 45].

#### Experimental setup

We begin by describing the datasets used in the experiments. We consider the
following two datasets which are source from the UCI machine learning database
and Kaggle:

  * •

Dataset1: Bank-Note-Authentication:
https://archive.ics.uci.edu/dataset/267/banknote+authentication

  * •

Dataset2: Creditcard: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud

Dataset1 contains 1,372 rows with 5 columns consisting of 4 features and 1
target attribute/class classifying whether a given banknote is authentic or
not. The data were extracted from images that were taken from genuine and
forged back note specimen and has been used in many research works in machine
learning. The target class contains two variables; namely, 0 which represents
genuine note and 1 which represents fake note. More so, the data contains a
balance ratio of both classes which is 55:45 (genuine:fake) and has no missing
value. Hence, there is no need for a data cleaning purpose, which means that,
we implemented the data directly in our experiment. Details of the dataset can
be found in Table 4.

Dataset2 contains transaction made by credit cards in September 2013 by
European cardholders within two days. It contains 234,807 transaction out of
which 492 are fraud which indicates a highly imbalanced dataset. Hence, we
carried out a feature engineering task to avoid wrong classification of the
model due to the imbalance in the dataset. We used the undersampling technique
by reducing the number of samples in the majority class to match the number of
samples in the minority class. This helps to balance the dataset and makes it
less skewed towards the majority class. Then we used the ’resample’ function
from scikit-learn to randomly selects a subset of samples from the majority
class without replacement, ensuring that each sample is selected only once.
The number of samples selected is equal to the number of samples in the
minority class, ensuring a balance class distribution. This is then used to
form a new dataset which is used for our experiment. Details of the original
imbalanced dataset and the balance dataset after undersampling can be seen in
Figure 4 and 5.

In this experiments, we compare the performance of Algorithm 1 and 2 with the
ADMM [40], DCA [37] and GDCP [11]. In the model, we choose the following
general parameters for implementing the algorithm: $C=1,$ $\lambda=0.001,$ for
Algorithm 1, 2, ADMM, DCA and GDCP, we choose
$\theta=0.01,\beta=0.001,\kappa_{n}=0.3,\alpha_{n}=\frac{1}{10(n+1)}$ and the
declared parameters used in the original papers of the comparing methods. The
maximum allowed iteration is 2000 and we also set the algorithms to stop if
$\|x_{k+1}-x_{k}\|<10^{-4}.$

Table 4: Description of dataset1. | variance | skewness | curtosis | entropy | class
---|---|---|---|---|---
count | 1372 | 1372 | 1372 | 1372 | 1372
mean | 0.433735 | 1.92235 | 1.39763 | -1.19166 | 0.444606
std | 2.84276 | 5.86905 | 4.31003 | 2.10101 | 0.497103
min | -7.0421 | -13.7731 | -5.2861 | -8.5482 | 0
25% | -1.773 | -1.7082 | -1.57498 | -2.41345 | 0
50% | 0.49618 | 2.31965 | 0.61663 | -0.58665 | 0
75% | 2.82147 | 6.81462 | 3.17925 | 0.39481 | 1
max | 6.8248 | 12.9516 | 17.9274 | 2.4495 | 1

Figure 4: Highly imbalanced dataset2 (Left) and Balanced dataset2 after
undersampling (Right)

Figure 5: Correlation matrix of features of imbalanced dataset2 (Left) and
correlation matrix of features of balanced dataset2 after undersampling
(Right).

#### Experimental results

We divided the training data into four subsets with varying ratios of testing
to training data: 10% testing and 90% training, 20% testing and 80% training,
30% testing and 70% training, and 40% testing and 60% training using the
train_test_split function from sklearn in python. For dataset1, we carried out
preprocssing technique to ensure that the data is standardized and suitable
for training the model. For dataset2, we carried out undersampling feature
selection to ensure that the model does not wrongly classify the testing data.
We train each of the algorithm using the designated training data preference
specified above. Consistent starting points and parameters are applied to
ensure fair comparison among algorithms. The performance metrics are recorded
for each algorithm to assess their effectiveness. These include accuracy,
execution time, precision, MAE, MSE and RMSE scores which are defined below:

  * (i)

Accuracy: This measures the proportion of correctly classified instances out
of the total number of instances. It is the ratio of the number of correct
predictions to the total number of predictions.

  * (ii)

Precision: This measures the proportion of true positive predictions among all
positive predictions made by the model. It is calculated as the ratio of true
positive to sum of true positive and false positive. A desirable high
precision score indicates that the model makes fewer false predictions.

  * (iii)

Mean absolute error (MAE): This is use to measures the average absolute
difference between the predicted values and the actual values, with the
formula given by

$MAE=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|$

where $y_{i}$ is the actual value and $\hat{y}_{i}$ is the predicted value.

  * (iv)

Mean square error (MSE): This is the average squared difference between the
predicted values and the actual values. It penalizes larger errors more
heavily that smaller errors due to squaring and it is sensitive to outliers.
It’s formula is given by

$MSE=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}.$

  * (v)

Root mean square error(RMSE): This represents the average magnitude of the
errors in the same units as the target variable. It provides a measure of
spread of the errors. It’s formula is given by

$RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}.$

The experimental results for dataset1 can be found in Table 5 \- 8. We also
compare the MAE, MSE and RMSE scores for each algorithm for the split dataset
to observed the performance of the algorithm for predicting the value of the
test data. The results are shown in Figure 6 \- 9. Similarly, the results of
the experiments for dataset2 can be found in Table 9 \- 12 and Figure 10 \-
13.

Table 5: Experimental results of algorithms for 10% split of dataset1. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9855 | 0.9348 | 0.9348 | 0.9348 | 0.8768
Precision | 0.9841 | 1.0 | 1.0 | 1.0 | 1.0
Time | 0.2941 | 0.2497 | 0.2508 | 0.2826 | 0.2540
MAE | 0.0290 | 0.1304 | 0.1304 | 0.1304 | 0.2464
MSE | 0.0580 | 0.2609 | 0.2609 | 0.2609 | 0.4928
RMSE | 0.2408 | 0.5108 | 0.5108 | 0.5108 | 0.7020
Table 6: Experimental results of algorithms for 20% split of dataset1. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9818 | 0.9345 | 0.9345 | 0.9345 | 0.8981
Precision | 0.9747 | 0.9902 | 0.9902 | 0.9902 | 0.9592
Time | 0.2889 | 0.2599 | 0.2625 | 0.2858 | 0.2772
MAE | 0.0364 | 0.1309 | 0.1309 | 0.2464 | 0.2036
MSE | 0.0727 | 0.2618 | 0.2618 | 0.4928 | 0.4073
RMSE | 0.2697 | 0.5117 | 0.5117 | 0.7020 | 0.6382
Table 7: Experimental results of algorithms for 30% split of dataset1. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9836 | 0.9399 | 0.9399 | 0.9399 | 0.9290
Precision | 0.9881 | 0.9874 | 0.9874 | 0.9874 | 0.9863
Time | 0.2695 | 0.2156 | 0.2208 | 0.2635 | 0.2641
MAE | 0.0328 | 0.1202 | 0.1202 | 0.1202 | 0.1421
MSE | 0.0656 | 0.2404 | 0.2404 | 0.2404 | 0.2815
RMSE | 0.2561 | 0.4903 | 0.4903 | 0.4903 | 0.5331
Table 8: Experimental results of algorithms for 40% split of dataset1. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9878 | 0.9466 | 0.9466 | 0.9466 | 0.9150
Precision | 0.9752 | 0.9768 | 0.9768 | 0.9768 | 0.9808
Time | 0.5294 | 0.3004 | 0.3068 | 0.4760 | 0.3936
MAE | 0.0583 | 0.1068 | 0.1608 | 0.2036 | 0.1699
MSE | 0.1165 | 0.2136 | 0.2136 | 0.4073 | 0.3399
RMSE | 0.3413 | 0.4621 | 0.4621 | 0.6382 | 0.5829

Figure 6: Comparison of MAE, MSE and RMSE for 10% split of dataset1.

Figure 7: Comparison of MAE, MSE and RMSE for 20% split of dataset1.

Figure 8: Comparison of MAE, MSE and RMSE for 30% split of dataset1.

Figure 9: Comparison of MAE, MSE and RMSE for 40% split of dataset1. Table 9: Experimental results of algorithms for 10% split of dataset2. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9368 | 0.8316 | 0.9050 | 0.9052 | 0.7473
Precision | 0.9512 | 0.7454 | 0.8696 | 0.8696 | 0.6727
Time | 2.7984 | 2.5741 | 2.2906 | 2.6751 | 2.7383
MAE | 0.1263 | 0.3368 | 0.1895 | 0.1895 | 0.5052
MSE | 0.2526 | 0.6737 | 0.3789 | 0.3789 | 1.0105
RMSE | 0.5026 | 0.8206 | 0.6156 | 0.6156 | 1.0052
Table 10: Experimental results of algorithms for 20% split of dataset2. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9316 | 0.9211 | 0.8684 | 0.8474 | 0.8632
Precision | 0.9468 | 0.9271 | 0.8158 | 0.7931 | 0.8142
Time | 2.8183 | 2.5693 | 2.3673 | 2.4144 | 2.7594
MAE | 0.1368 | 0.1579 | 0.2632 | 0.3053 | 0.2737
MSE | 0.2737 | 0.3158 | 0.5263 | 0.6105 | 0.5474
RMSE | 0.5231 | 0.5620 | 0.7255 | 0.7813 | 0.7398
Table 11: Experimental results of algorithms for 30% split of dataset2. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9087 | 0.9018 | 0.9298 | 0.8596 | 0.7895
Precision | 0.8868 | 0.9116 | 0.9640 | 0.8471 | 0.9298
Time | 2.8389 | 2.0428 | 2.3072 | 2.3861 | 2.5579
MAE | 0.1824 | 0.1965 | 0.1404 | 0.2807 | 0.4211
MSE | 0.3649 | 0.3930 | 0.2807 | 0.5614 | 0.8421
RMSE | 0.6040 | 0.6269 | 0.5298 | 0.7493 | 0.9177
Table 12: Experimental results of algorithms for 40% split of dataset2. | ADMM | Alg1 | Alg2 | DCA | GDCP
---|---|---|---|---|---
Accuracy | 0.9076 | 0.8179 | 0.8285 | 0.8258 | 0.8073
Precision | 0.9508 | 0.7963 | 0.8082 | 0.8131 | 0.7873
Time | 2.9516 | 2.5901 | 2.5424 | 2.6523 | 2.9498
MAE | 0.1847 | 0.3641 | 0.3430 | 0.3482 | 0.3852
MSE | 0.3694 | 0.7282 | 0.6861 | 0.6964 | 0.7704
RMSE | 0.6078 | 0.8533 | 0.8282 | 0.8346 | 0.88775

Figure 10: Comparison of MAE, MSE and RMSE for 10% split of dataset2

Figure 11: Comparison of MAE, MSE and RMSE for 20% split of dataset2

Figure 12: Comparison of MAE, MSE and RMSE for 30% split of dataset2

Figure 13: Comparison of MAE, MSE and RMSE for 40% split of dataset2

#### Discussion of results

The results of the experiments for dataset1 reveal that Algorithm 1 and 2
consistently demonstrate better convergence rates compared to the other
algorithms across all four splits of the dataset. While the ADMM algorithm
achieves the better accuracy and precision scores, it suffers from longer
execution times compared to the others. Specifically, Algorithm 1 and 2
exhibit better accuracy and precision scores than both the DCA and GDCP
methods. Notably, the GDCP method displays the worst performance with lowest
accuracy and highest MAE, MSE, and RMSE scores among all methods evaluated.
These findings underscore the effectiveness of the proposed Algorithm 1 and 2
in achieving rapid convergence and competitive performance in accuracy and
precision metrics when compared to other algorithms in the experiment.
However, the ADMM method, while boasting superior accuracy and precision, lags
in execution time efficiency. This behaviour is supported by findings in the
literature which had shown the ADMM having poor convergence rate.

For dataset2, the proposed Algorithm 1 and 2 also showed to have the best rate
of convergence than other methods in the experiments. However, a nuanced
observation reveals that for the (10:90) and (40:60) splits of the dataset,
the accuracy and precision scores of the ADMM and DCA methods surpass those of
the proposed algorithms. Despite achieving commendable accuracy and precision
scores, the ADMM method continues to exhibit the poorest execution time among
the evaluated methods. Furthermore, consistent with the findings in the
literature, the DCA algorithm showcases promising performance, particularly in
feature selection tasks. Also, the GDCP method consistently underperforms
across all splits of the dataset, displaying the lowest accuracy and highest
error metrics among all methods considered. These observations underscore the
importance of considering various performance metrics and dataset
characteristics when evaluating algorithm efficacy. While Algorithm 1 and 2
demonstrate strong convergence rates, the ADMM and DCA methods exhibit
competitive accuracy and precision scores, but at the expense of longer
execution times. Additionally, this findings align with prior research
indicating the efficiency of the DCA algorithm in feature selection tasks. In
conclusion, we noted that all five methods consistently yield high values of
MSE, and RMSE scores. This outcomes may arise from the algorithms reaching the
maximum iteration limit of 2000, where the resulting point computed is not a
critical point of the optimization model.

##  5 Final Remarks

We have proved in this paper that new two variants of the unified Douglas-
Rachford splitting algorithm converge weakly to a critical point of a
generalized DC programming in Hilbert spaces. We recovered the unified
Douglas-Rachford splitting algorithm from our proposed algorithms. Numerical
illustrations from machine learning showed that our algorithms have practical
implementations, are efficient, and outperform some other related algorithms
for solving this class of generalized DC programming. As part of the future
project, we study the accelerated versions of our proposed algorithms.

## References

  * [1] Argyriou, A., Hauser, R., Micchelli, C. and M. Pontil A.: DC-programming algorithm for kernel selection. Proceedings of the 23rd International Conference on Machine Learning (ICML ’06), 2006.
  * [2] Banert, S., Rudzusika, J., Oktem, O., Adler, J.: Accelerated forward-backward optimization using deep learning, https://arxiv.org/abs/2105.05210.
  * [3] Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. 2nd ed. Springer, New York, 2017.
  * [4] Bian, F., Zhang, X.: A three-operator splitting algorithm for nonconvex sparsity regularization, Siam J. Sci. Comput. 43, (2021).
  * [5] Bishop CM.: Pattern Recognition and Machine Learning. New York : Springer, 2006
  * [6] Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1-122 (2011).
  * [7] Ceng, L.C., Ansari, Q.H., Schaible, S.: Hybrid extragradient-like methods for generalized mixed equilibrium problems, systems of generalized equilibrium problems and optimization problems. J. Glob. Optim. 53, 69-96 (2012).
  * [8] Candès, E.J. and Wakin, M.B.: An Introduction To Compressive Sampling, IEEE Signal Processing Magazine, 25, 21-30 (208).
  * [9] Ceng, L.C., Coroian, I., Qin, X., Yao, J.C.: A general viscosity implicit iterative algorithm for split variational inclusions with hierarchical variational inequality constraints. Fixed Point Theory 20, 469-482 (2019).
  * [10] Ceng, L.C., Li, X., Qin, X.: Parallel proximal point methods for systems of vector optimization problems on Hadamard manifolds without convexity. Optimization 69, 357-383 (2020).
  * [11] Chuang, C.S., He, H., Zhang, Z.: A unified Douglas-Rachford algorithm for generalized DC programming. J. Glob. Optim. 82, 331-349 (2022).
  * [12] Cortes C and Vapnik, VN.: Support vector networks, Machine Learning, 20, 273–297, (1995).
  * [13] Cui, F., Tang, Y., Yang, Y.: An inertial three-operator splitting algorithm with applications to image inpainting. Appl. Set-Valued Anal. Optim. 1, 113-134 (2019).
  * [14] Cui, H.H., Ceng, L.C.: Convergence of over-relaxed contraction-proximal point algorithm in Hilbert spaces. Optimization 66, 793-809 (2017).
  * [15] Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications, Set-Valued Variat. Anal. 25, 829-858 (2017).
  * [16] Ding, M., Song, X. Yu, B.: An Inexact Proximal DC Algorithm with Sieving Strategy for Rank Constrained Least Squares Semidefinite Programming. J. Sci. Comput. 91, 75 (2022).
  * [17] Dong, Q.L., Huang, J.Z., Li, X.H., Cho, Y.J., Rassias, Th.M.: MiKM: multi-step inertial Krasnosel’skii–Mann algorithm and its applications. J. Glob. Optim. 73, 801-824 (2019).
  * [18] Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two or three space variables. Trans. Am. Math. Soc. 82, 421-439 (1956).
  * [19] Gaudioso, M., Gorgone, E., Hiriart-Urruty, J.B.: Feature selection in SVM via polyhedral k-norm. Optim. Lett. 14, 19-36 (2020).
  * [20] Gotoh, JY, Takeda, A., Tono, K.: DC formulations and algorithms for sparse optimization problems. Math. Program. 169, 141-176 (2018).
  * [21] Guan, J.L., Ceng, L.C., Hu, B.: Strong convergence theorem for split monotone variational inclusion with constraints of variational inequalities and fixed point problems. J. Inequal. Appl. 311, (2018).
  * [22] Hastie, T. and Tibshirani, R. and Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer Series in Statistics, 2001.
  * [23] He, L., Cui, Y.L., Ceng L.C., Zhao, T.Y., Wang, D.Q., Hu, H.Y.: Strong convergence for monotone bilevel equilibria with constraints of variational inequalities and fixed points using subgradient extragradient implicit rule. J. Inequal. Appl. 146, (2021).
  * [24] Hu, Z., Huang, Dong, Q.L.: A three-operator splitting algorithm with deviations for generalized DC programming. Appl. Numer. Math. 191, 62-74 (2023).
  * [25] Huang, G.B., Mao, K.Z., Siew, C.K., Huang D.S.: Fast Modular Network Implementation for Support Vector Machines. IEEE Trans Neural Netw., 16(6), Issue 6, p1651 (2005).
  * [26] Latif, A., Ceng, L.C., Al-Mezel, S.A.: Some iterative methods for convergence with hierarchical optimization and variational inclusions. J. Nonlinear Convex Anal. 17, 735-755 (2016).
  * [27] Le Thi, H.A., Pham Dinh, T., Le, H.M., Vo, X.T.: DC approximation approaches for sparse optimization. European J. Oper. Res. 244(1), 26-46 (2015).
  * [28] Li, G., Yang, L., Wu, Z., Wu, G.: D.C. programming for sparse proximal support vector machines. Inform. Sciences 547, 187-201 (2021).
  * [29] Liu, T., Pong, T.K., Takeda, A.: A refined convergence analysis of pDCAe with applications to simultaneous sparse recovery and outlier detection. Comput. Optim. Appl. 73, 69-100 (2019).
  * [30] Lou, Y., Yan, M.: Fast L1-L2 Minimization via a Proximal Operator. SIAM J. Sci. Comput. 74, 767-785 (2018).
  * [31] Lou, Y., Yin, P., He, Q., Xin, J.: Minimization of $\ell_{1-2}$ for compressed sensing. SIAM J. Sci. Comput. 37, A536-A563 (2015).
  * [32] Lu, Z., Zhou, Z.: Nonmonotone enhanced proximal DC algorithms for a class of structured nonsmooth DC programming. SIAM J. Optim. 29(4), 2725-2752 (2019).
  * [33] Murphy, KP.: Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
  * [34] Nguyen, M.C.: DC programming and DCA for some classes of problems in machine learning and data mining. Other [cs.OH]. Université de Lorraine, 2014. English. NNT: 2014LORR0080.tel-01750803.
  * [35] Nouiehed, M., Pang, J.S., Razaviyayn, M.: On the pervasiveness of difference-convexity in optimization and statistics. Math. Program. 174, 195-222 (2019).
  * [36] Opial. Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591-597 (1967).
  * [37] Pham D.T., Souad, E.B.: Algorithms for solving a class of nonconvex optimization problems. Methods of subgradients. In: Hiriart-Urruty, J.B. (ed.) Fermat Days 85: Mathematics for Optimization. NorthHolland Mathematics Studies, vol. 129, pp. 249–271. North-Holland, Amsterdam (1986) North-Holland Mathematics Studies, vol. 129, pp. 249–271. North-Holland, Amsterdam (1986)
  * [38] Raguet, H., Fadili, J., Peyre, G.: Generalized forward-backward splitting. SIAM J. Imaging Sci. 6(3), 1199-1226 (2011).
  * [39] Sadeghi, H., Banert, S., Giselsson, P.: Forward-backward splitting with deviations for monotone inclusions. https://arxiv.org/abs/2112.00776.
  * [40] Sun, T., Yin, P., Cheng, L., Jiang, H.: Alternating direction method of multipliers with difference of convex functions. Adv. Comput. Math. 44, 723–744 (2018).
  * [41] Tao, P.D., Souad, E.B.: Algorithms for solving a class of nonconvex optimization problems: methods of subgradients. Math. Optim. 129, 249-271 (1986).
  * [42] Thi, H.A.L., Le, H.M., Nguyen, V.V., Dinh, T.P.: A DC programming approach for feature selection in support vector machines learning. Adv Data Anal Classif 2, 259–278 (2008).
  * [43] Thi H.A.L., Nguyen, M.C., Dinh, T.P.: A DC programming approach for finding communities in networks, Neural Comput. 26(12), 2827-54 (2014).
  * [44] Thi, H.A.L., Le, H.M., Phan, D.N., Tran, B.: Novel DCA based algorithms for a special class of nonconvex problems with application in machine learning, Appl. Mathem. Comp., 409 (15), 125904 (2021).
  * [45] Yao, C., Jiang, X.: A globally convergent difference-of-convex algorithmic framework and application to log-determinant optimization problems. arXiv:2306.02001.
  * [46] Tu, K., Zhang, H., Gao, H. et al. A hybrid Bregman alternating direction method of multipliers for the linearly constrained difference-of-convex problems. J. Glob. Optim. 76, 665-693 (2020).
  * [47] Ukey, K.P., Alvi, A.S.: Text classification using support vector machine. J. Softw Eng. Appl., 5(12), 55–58 (2012).
  * [48] Vapnik, VN.: The Nature of Statistical Learning Theory. Springer-Verlag, New York, 1995.
  * [49] Wang, J., Zhao, C., Huo, Z., Qiao, Y., Sima, H.: High quality proposal feature generation for crowded pedestrian detection. Pattern Recognition, 128(5), 108605 (2022).
  * [50] Wen, J., Fang, X., Cui, J., Fei, L., Yan, K., Chen, Y., Xu, Y.: Robust Sparse Linear Discriminant Analysis, IEEE Trans. Circ. Syst Video Tech, 29, 390-403 (2019).
  * [51] Ho, V.T.: Advanced machine learning techniques based on DC programming and DCA. Machine Learning [cs.LG]. Université de Lorraine, 2017. English. NNT : 2017LORR0289.tel-01810274
  * [52] Wen, B., Chen, X., Pong, T.K.: A proximal difference-of-convex algorithm with extrapolation. Comput. Optim. Appl. 69, 297-324 (2018).
  * [53] Wu, P., Cheng, J.: Nonconvex regularized gradient projection sparse reconstruction for massive MIMO channel estimation. IEEE T. Commun. 69(11), 7722-7735 (2021).