k\leq{\mathcal{M}}}\frac{1}{k^{\eta}}\left|(k-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,k}-k^{\ast}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\right|>x\right\\}$
$\displaystyle=0.$

Since $(k-k^{\ast})^{2}/t^{\eta}$ is increasing on
$[(1-\delta){\mathcal{M}},{\mathcal{M}}]$ we get that

$\displaystyle\lim_{\delta\rightarrow
0}\limsup_{m\rightarrow\infty}P\left\\{\sup_{(1-\delta){\mathcal{M}}\leq
k\leq{\mathcal{M}}}\frac{1}{k^{\eta}}\left((k-k^{\ast})^{2}A_{m}+2(k-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,k}\right)\right.$
$\displaystyle\left.=\frac{1}{{\mathcal{M}}^{\eta}}\left[({\mathcal{M}}-k^{\ast})^{2}A_{m}+2({\mathcal{M}}-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\right]\right\\}=1.$

Next we write

$\displaystyle P$
$\displaystyle\left\\{\frac{1}{{\mathcal{M}}^{\eta}}\left[({\mathcal{M}}-k^{\ast})^{2}A_{m}+2({\mathcal{M}}-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\right]\leq{\mathcal{c}}{\mathcal{n}}^{1-\eta}\right\\}$
$\displaystyle=P\left\\{2({\mathcal{M}}-k^{\ast})[\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}]{\mathcal{M}}^{-3/2}\leq\left[{\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\eta}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right]{\mathcal{M}}^{-3/2+\eta}\right\\}.$

Using (C.13) we get

${\mathcal{M}}^{-1/2}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\overset{{\mathcal{D}}}{\rightarrow}\bar{s}_{1}{\mathcal{N}},$

where ${\mathcal{N}}$ denotes a standard normal random variable, $\bar{s}_{1}$
is defined in (3.13) and $\mathbf{D}$ is the covariance matrix in (C.12).
Also,

$\left[{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\eta}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right]{\mathcal{M}}^{-3/2+\eta}\rightarrow-2x,$

and thus we conclude

$\displaystyle\lim_{m\rightarrow\infty}$ $\displaystyle
P\left\\{2({\mathcal{M}}-k^{\ast})[\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}]{\mathcal{M}}^{-3/2}\leq\left[{\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\eta}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right]{\mathcal{M}}^{-3/2+\eta}\right\\}$
$\displaystyle=P\left\\{\bar{s}_{1}{\mathcal{N}}\leq-x\right\\}=1-\Phi(x/\bar{s}_{1}).$

We assume now that $t^{\ast}=\infty$ and $\bar{u}=0$ also holds. We modify the
definition of ${\mathcal{M}}$ as

${\mathcal{M}}=k^{\ast}+\left(\frac{{\mathcal{c}}}{A_{m}}{\mathcal{n}}^{1-\eta}(k^{\ast})^{\eta}\right)^{1/2}+R\quad\;\mbox{with}\quad\;R=\frac{x}{A_{m}}(k^{\ast})^{1/2}.$

Proceeding as in the previous case we get that for any $0<\delta<1$

(C.14) $\displaystyle\lim_{m\rightarrow\infty}$ $\displaystyle
P\left\\{\tau_{m}>{\mathcal{M}}\right\\}$
$\displaystyle=\lim_{m\rightarrow\infty}\left\\{\max_{(1-\delta){\mathcal{M}}\leq
k\leq{\mathcal{M}}}\frac{1}{k^{\eta}}\left((k-k^{\ast})^{2}A_{m}+2(k-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m.k}\right)\leq{\mathcal{c}}{\mathcal{n}}^{1-\eta}\right\\}$
$\displaystyle=\lim_{m\rightarrow\infty}P\left\\{2({\mathcal{M}}-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\leq\left({\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\eta}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right){\mathcal{M}}^{\eta}\right\\}$
$\displaystyle=\lim_{m\rightarrow\infty}P\left\\{2(k^{\ast})^{-1/2}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\leq\left({\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\tau}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right)\frac{{\mathcal{M}}^{\eta}}{({\mathcal{M}}-k^{\ast})(k^{\ast})^{1/2}}\right\\}.$

Using the definitions of ${\mathcal{M}}$ and $R$ we have

$\left({\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{{\mathcal{M}}^{\eta}}({\mathcal{M}}-k^{\ast})^{2}A_{m}\right)\frac{{\mathcal{M}}^{\eta}}{({\mathcal{M}}-k^{\ast})(k^{\ast})^{1/2}}\rightarrow-2x.$

As in the proof of (C.13) we have

$(k^{\ast})^{-1/2}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\overset{{\mathcal{D}}}{\rightarrow}\bar{s}_{2}{\mathcal{N}},$

where ${\mathcal{N}}$ is a standard normal random variable and $\bar{s}_{2}$
is defined in (3.14). Now we conclude

$\lim_{m\rightarrow\infty}P\\{\tau_{m}>{\mathcal{M}}\\}=1-\Phi(x/\bar{s}_{2}),$

where $\Phi$ is the standard normal distribution function.
In the final case we write

$k^{\ast}=u_{m}{\mathcal{n}}\;\;\;\;\mbox{with}\;\;\;u_{m}\rightarrow\bar{u}\in(0,1).$

In this case, we define

${\mathcal{M}}=k^{\ast}+x(k^{\ast})^{1/2}.$

We write

$\max_{1\leq
k\leq{\mathcal{M}}}\frac{{\mathcal{D}}_{m,k}}{{\mathcal{n}}(k/{\mathcal{n}})^{\eta}}=\max\left(U_{1,m},U_{2,m}\right),$
$U_{1,m}=\max_{1\leq k\leq
k^{\ast}}\frac{{\mathcal{D}}_{m,k}}{{\mathcal{n}}(k/{\mathcal{n}})^{\eta}}$

and

$U_{2,m}=\max_{k^{\ast}+1\leq
k\leq{\mathcal{M}}}\frac{{\mathcal{D}}_{m,k}}{{\mathcal{n}}(k/{\mathcal{n}})^{\eta}}.$

We use the decomposition in (C.10) and we note

$\left(\frac{1}{k^{\ast}}\right)^{1/2}\max_{k^{\ast}+1\leq
k\leq{\mathcal{M}}}\|{\mathcal{r}}_{m,k}(\hat{\mbox{\boldmath${\theta}$}}_{m})-{\mathcal{r}}_{m,k^{\ast}}(\hat{\mbox{\boldmath${\theta}$}}_{m})\|=O_{P}(1).$

It follows from the proof of Theorem 3.1 that

$\left(\max_{1\leq k\leq
k^{\ast}}\frac{{\mathcal{D}}_{m,k}}{k^{\ast}(k/k^{\ast})^{\eta}},\left(\frac{1}{k^{\ast}}\right)^{1/2}{\mathcal{r}}_{m,k^{\ast}}(\hat{\mbox{\boldmath${\theta}$}}_{m})\right)\overset{{\mathcal{D}}}{\rightarrow}\left(\sup_{0\leq
s\leq
1}\frac{\mbox{\boldmath${\Gamma}$}(s)^{\top}\mathbf{D}^{-1}\mbox{\boldmath${\Gamma}$}(s)}{s^{\eta}},\mbox{\boldmath${\Gamma}$}(1)\right),$

where $\\{\mbox{\boldmath${\Gamma}$}(s),s\geq 0\\}$ is a Gaussian process with
$E\mbox{\boldmath${\Gamma}$}(s)=\mathbf{0}$ and
$E\mbox{\boldmath${\Gamma}$}(s)\mbox{\boldmath${\Gamma}$}^{\top}(t)=\min(t,s)\mathbf{D}.$
We write

$U_{2,m}=\max_{1\leq j\leq
x(k^{\ast})^{1/2}}\left(\frac{k^{\ast}+j}{m}\right)^{1-\eta}\left({\mathcal{r}}_{m,k^{\ast}}+j\mbox{\boldmath${\Delta}$}+[{\mathcal{r}}_{m,k^{\ast}}-{\mathcal{r}}_{m,k}]\right)^{\top}\hat{\mathbf{D}}_{m}^{-1}\left({\mathcal{r}}_{m,k^{\ast}}+j\mbox{\boldmath${\Delta}$}+[{\mathcal{r}}_{m,k^{\ast}}-{\mathcal{r}}_{m,k}]\right)^{\top}.$

Thus we get

$\left(U_{m,1},U_{m,2}\right)\overset{{\mathcal{D}}}{\rightarrow}\left(\sup_{0<t\leq
1}\frac{\bar{u}^{1-\eta}}{t^{\eta}}\mbox{\boldmath${\Gamma}$}^{\top}\mathbf{D}^{-1}\mbox{\boldmath${\Gamma}$}(t),\bar{u}^{1-\eta}\sup_{0\leq
t\leq
x}(t\mbox{\boldmath${\Delta}$}+\mbox{\boldmath${\Gamma}$}(1))^{\top}\mathbf{D}^{-1}(t\mbox{\boldmath${\Delta}$}+\mbox{\boldmath${\Gamma}$}(1))\right).$

Next we assume that (3.9) holds. We replace (C.10) and (C.11) with

${\mathcal{r}}_{m,k}(\hat{\mbox{\boldmath${\theta}$}}_{m})=(k-k^{\ast})\mbox{\boldmath${\Upsilon}$}+\tilde{\mathbf{m}}_{m,k},$
$\tilde{\mathbf{m}}_{m,k}={\mathcal{r}}_{m,k^{\ast}}(\hat{\mbox{\boldmath${\theta}$}}_{m})+[{\mathcal{r}}_{m,k}(\hat{\mbox{\boldmath${\theta}$}}_{m})-{\mathcal{r}}_{m,k^{\ast}}(\hat{\mbox{\boldmath${\theta}$}}_{m})-(k-k^{\ast})\mbox{\boldmath${\Upsilon}$}],$
${\mathcal{D}}_{m,k}=(k-k^{\ast})^{2}B_{m}+2(k-k^{\ast})\mbox{\boldmath${\Upsilon}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\tilde{\mathbf{m}}_{m,k}+\tilde{\mathbf{m}}_{m,k}^{\top}\hat{\mathbf{D}}_{m}^{-1}\tilde{\mathbf{m}}_{m,k}.$

Following the same arguments as in the case of (3.2), we get that if
$\bar{u}=0$, then

$\displaystyle\lim_{m\rightarrow\infty}P\\{\tau_{m}>\tilde{{\mathcal{M}}}\\}$
$\displaystyle=\lim_{m\rightarrow\infty}P\left\\{2(\tilde{{\mathcal{M}}}-k^{\ast})[\mbox{\boldmath${\Upsilon}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\tilde{\mathbf{m}}_{m,\tilde{{\mathcal{M}}}}]\tilde{{\mathcal{M}}}^{-3/2}\leq\left[{\mathcal{c}}{\mathcal{n}}^{1-\eta}-\frac{1}{\tilde{{\mathcal{M}}}^{\eta}}\left(\tilde{{\mathcal{M}}}-k^{\ast}\right)^{2}B_{m}\right]\tilde{{\mathcal{M}}}^{-3/2+\eta}\right\\},$

and the definition of $\tilde{{\mathcal{M}}}$ depends on if
$\tilde{t}^{\ast}<\infty$ is finite or not. If $\tilde{t^{\ast}}<\infty$ we
use

$\tilde{{\mathcal{M}}}=k^{\ast}+\tilde{u}_{{\mathcal{n}}}\left({\mathcal{n}}^{1-\eta}\frac{{\mathcal{c}}}{B_{m}}\right)^{1/(2-\eta)}+\tilde{R},$

with

$\tilde{R}=x{(\tilde{u}^{\ast}+\tilde{t}^{\ast})^{3/2}}\left(\left({\mathcal{n}}^{1-\eta}\frac{{\mathcal{c}}}{B_{m}}\right)^{1/(2-\eta)}\right)^{1/2}.$

Using the definition of $\bar{{\mathcal{M}}}$ we get

(C.15)
$\lim_{m\rightarrow\infty}\left[{\mathcal{c}}{\mathcal{n}}^{1-\tau}-\frac{1}{\tilde{{\mathcal{M}}}^{\tau}}\left(\tilde{{\mathcal{M}}}-k^{\ast}\right)^{2}B_{m}\right]\tilde{{\mathcal{M}}}^{-3/2+\tau}=-2x.$

Applying (B.19) and Lemma B.10 with the independence of the approximating
Gaussian processes we conclude

(C.16)
$(\tilde{{\mathcal{M}}}-k^{\ast})[\mbox{\boldmath${\Upsilon}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\tilde{\mathbf{m}}_{m,\tilde{{\mathcal{M}}}}]\tilde{{\mathcal{M}}}^{-3/2}\overset{{\mathcal{D}}}{\rightarrow}\bar{s}_{1}{\mathcal{N}},$

where ${\mathcal{N}}$ is a standard normal random variable and $\bar{s}_{1}$
is defined in (3.13). The covariance matrix $\mathbf{D}$ is defined in (C.12).
If $\tilde{t}^{\ast}=\infty$ and $\bar{u}=0$, then we use

${\mathcal{M}}=k^{\ast}+\left(\frac{{\mathcal{c}}}{B_{m}}{\mathcal{n}}^{1-\eta}(k^{\ast})^{\eta}\right)^{1/2}+\tilde{R},\quad\;\mbox{with}\quad\;\tilde{R}=\frac{x}{B_{m}}(k^{\ast})^{1/2}.$

We still have (C.15) but (C.16) is replaced with

$(\tilde{{\mathcal{M}}}-k^{\ast})[\mbox{\boldmath${\Upsilon}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\tilde{\mathbf{m}}_{m,\tilde{{\mathcal{M}}}}]\tilde{{\mathcal{M}}}^{-3/2}\overset{{\mathcal{D}}}{\rightarrow}\bar{s}_{2}{\mathcal{N}},$

where $\bar{s}_{2}$ is given in (3.14). As in the previous case, if
$0<\bar{u}<1$, then we use again ${\mathcal{M}}=k^{\ast}+x(k^{\ast})^{1/2}$.
Since the sum of gradients (after removing the mean) of the log likelihood
function of the observations after the change is negligible, we get that

$\max_{1\leq
k\leq{\mathcal{M}}}\frac{{\mathcal{D}}_{m}(k)}{g_{m}(k)}\overset{{\mathcal{D}}}{\rightarrow}\bar{u}^{1-\eta}\max\left(\sup_{0<t\leq
1}\frac{1}{t^{\eta}}\mbox{\boldmath${\Gamma}$}^{\top}(t)\mathbf{D}^{-1}\mbox{\boldmath${\Gamma}$}(t),\sup_{0\leq
t\leq
x}(t\mbox{\boldmath${\Upsilon}$}+\mbox{\boldmath${\Gamma}$}(1)^{\top}\mathbf{D}^{-1}(t\mbox{\boldmath${\Upsilon}$}+\mbox{\boldmath${\Gamma}$}(t))\right),$

where recall that $\\{\mbox{\boldmath${\Gamma}$}(t),t\geq 0\\}$ is a Gaussian
process $E\mbox{\boldmath${\Gamma}$}(t)=\mathbf{0}$ and
$E\mbox{\boldmath${\Gamma}$}(t)\mbox{\boldmath${\Gamma}$}^{\top}(s)=\min(t,s)\mathbf{D}$.

Next we consider the Rényi type detector. First we investigate the case when
after the change we have an (asymptotically) stationary sequence. Assume that
$k^{\ast}\leq r$, i.e. the change occurred before the monitoring started. By
definition,

$P\\{\bar{\tau}_{m}=r\\}=P\left\\{\frac{{\mathcal{D}}_{m}(r)}{\bar{g}_{m}(r)}\geq
1\right\\}.$

The proof is based again on (C.9) and (C.10). It follows from Lemma B.9 that

$\frac{1}{(k^{\ast})^{1/2}}\|{\mathcal{r}}_{m,k^{\ast}}\|=O_{P}(1)$

and if $k^{\ast}\rightarrow\infty$ and $r-k^{\ast}\rightarrow\infty$, then

$\left(\frac{1}{(k^{\ast})^{1/2}}{\mathcal{r}}_{m,k^{\ast}},\frac{1}{(r-k^{\ast})^{1/2}}\left[{\mathcal{r}}_{m,r}(\hat{\mbox{\boldmath${\theta}$}}_{m})-{\mathcal{r}}_{m,k^{\ast}}(\hat{\mbox{\boldmath${\theta}$}}_{m})-(r-k^{\ast})\mbox{\boldmath${\Delta}$}\right]\right)\overset{{\mathcal{D}}}{\rightarrow}\left(\mathbf{N}_{1},\mathbf{N}_{2}\right)$

where $\mathbf{N}_{1},\mathbf{N}_{2}$ are independent standard normal random
variables with covariance matrices $\mbox{\boldmath${\Sigma}$}_{1}$ and
$\mbox{\boldmath${\Sigma}$}_{2}$ defined in (3.5) and (3.10), respectively.
Thus we get

$\frac{1}{r^{1/2}}\left((r-k^{\ast})\mbox{\boldmath${\Delta}$}+\mathbf{m}_{m,r}\right)\overset{{\mathcal{D}}}{\rightarrow}{\mathcal{a}}^{1/2}\mathbf{N}_{1}+{\mathcal{b}}+\mathbf{N}_{2}.$

If ${\mathcal{a}}<\infty$, then we use

${\mathcal{M}}=k^{\ast}+xr^{1/2}.$

It follows from the proof of Theorem 3.1 that for all $K>0$

$\displaystyle\left\\{\frac{1}{r^{1/2}}{\mathcal{r}}_{m,rt}(\hat{\mbox{\boldmath${\theta}$}}_{m}),\frac{1}{r^{1/2}}\left({\mathcal{r}}_{m,k^{\ast}}+[{\mathcal{r}}_{m,xr^{1/2}}-{\mathcal{r}}_{m,k^{\ast}}-xr^{1/2}]\right),t\geq
0,x\geq 0\right\\}$ $\displaystyle\hskip
56.9055pt\overset{{\mathcal{D}}[0,{\mathcal{a}}]\times[0,K]}{\longrightarrow}\left\\{\mbox{\boldmath${\Gamma}$}_{1}(t),\mbox{\boldmath${\Gamma}$}_{1}({\mathcal{a}})+x\mbox{\boldmath${\Delta}$}\right\\}$

where $\\{\mbox{\boldmath${\Gamma}$}_{1}(t),t\geq 0\\}$ is Gaussian process
with $E\mbox{\boldmath${\Gamma}$}_{1}(t)=\mathbf{0}$,
$E\mbox{\boldmath${\Gamma}$}_{1}(t)\mbox{\boldmath${\Gamma}$}_{1}^{\top}(s)=\min(t,s)\mbox{\boldmath${\Sigma}$}_{1}$
with $\mbox{\boldmath${\Sigma}$}_{1}$ defined in (3.5). Thus we conclude

$\displaystyle\sup_{r\leq k\leq{\mathcal{M}}}$
$\displaystyle\frac{{\mathcal{D}}_{m}(k)}{g_{m}(k)}=\frac{1}{{\mathcal{c}}}\max\left(\max_{r\leq
k\leq
k^{\ast}}\frac{{\mathcal{D}}_{m}(k)}{(k/r)^{\eta}r},\;\max_{k^{\ast}<k\leq{\mathcal{M}}}\frac{{\mathcal{D}}_{m}(k)}{(k/r)^{\eta}r}\right)$
$\displaystyle\overset{{\mathcal{D}}}{\rightarrow}\frac{1}{{\mathcal{c}}{\mathcal{a}}^{\eta}}\max\left(\sup_{1\leq
t\leq{\mathcal{a}}}\mbox{\boldmath${\Gamma}$}_{1}^{\top}(t)\mathbf{D}^{-1}\mbox{\boldmath${\Gamma}$}_{1}(t),\;\max_{0\leq
s\leq
x}(\mbox{\boldmath${\Gamma}$}_{1}({\mathcal{a}})+s\mbox{\boldmath${\Delta}$})^{\top}\mathbf{D}^{-1}(\mbox{\boldmath${\Gamma}$}_{1}({\mathcal{a}})+s\mbox{\boldmath${\Delta}$}).\right).$

For the final case ${\mathcal{a}}=\infty$ we use

${\mathcal{M}}=k^{\ast}+\left(\frac{{\mathcal{c}}}{A_{m}}\frac{(k^{\ast})^{\eta}}{r^{\eta-1}}\right)^{1/2}+x(k^{\ast})^{1/2}.$

We observe that

(C.17)
$\frac{k^{\ast}}{((k^{\ast})^{\eta}/r^{\eta-1})^{1/2}}=\left(\left(\frac{k^{\ast}}{r}\right)^{2-\eta}r\right)^{1/2}\rightarrow\infty$

and

(C.18)
$\frac{(k^{\ast})^{\eta}/r^{\eta-1}}{k^{\ast}}=\left(\frac{k^{\ast}}{r}\right)^{\eta-1}\rightarrow\infty.$

As in the previous cases,

$\displaystyle\lim_{m\rightarrow\infty}$ $\displaystyle
P\\{\bar{\tau}_{m}>{\mathcal{M}}\\}$
$\displaystyle=\lim_{m\rightarrow\infty}P\left\\{2({\mathcal{M}}-k^{\ast})\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}<\left[{\mathcal{c}}r^{1-\eta}-\frac{({\mathcal{M}}-k^{\ast})^{2}A_{m}}{{\mathcal{M}}^{\eta}}\right]{\mathcal{M}}^{\eta}\right\\}$
$\displaystyle=\lim_{m\rightarrow\infty}P\left\\{2{\mathcal{M}}^{-1/2}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}<\left[{\mathcal{c}}r^{1-\eta}-\frac{({\mathcal{M}}-k^{\ast})^{2}A_{m}}{{\mathcal{M}}^{\eta}}\right]\frac{1}{{\mathcal{M}}-k^{\ast}}{\mathcal{M}}^{-1/2+\eta}\right\\}.$

We obtain from (C.17) and (C.18) that

$\left[{\mathcal{c}}r^{1-\eta}-\frac{({\mathcal{M}}-k^{\ast})^{2}A_{m}}{{\mathcal{M}}^{\eta}}\right]\frac{1}{{\mathcal{M}}-k^{\ast}}{\mathcal{M}}^{-1/2+\eta}\rightarrow-2x.$

We get from (C.13) that

${\mathcal{M}}^{-1/2}\mbox{\boldmath${\Delta}$}^{\top}\hat{\mathbf{D}}_{m}^{-1}\mathbf{m}_{m,{\mathcal{M}}}\overset{{\mathcal{D}}}{\rightarrow}\bar{s}_{2}{\mathcal{N}},$

where ${\mathcal{N}}$ is a standard normal random variable and $\bar{s}_{2}$
is defined in (3.14). We now conclude

$\lim_{m\rightarrow\infty}P\\{\bar{\tau}_{m}>{\mathcal{M}}\\}=1-\Phi(x/\bar{s}_{2}).$

The proof when (3.9) holds is essentially the same so the details are omitted.
∎


# DisControlFace: Disentangled Control for Personalized Facial Image Editing

Haozhe Jia1 Yan Li2 Hengfei Cui2 Di Xu1 Changpeng Yang1
Yuwang Wang3 Tao Yu3
1Huawei Cloud Computing Technologies Co., Ltd 2Northwestern Polytechnical
University
3Tsinghua University

###### Abstract

In this work, we focus on exploring explicit fine-grained control of
generative facial image editing, all while generating faithful and consistent
personalized facial appearances. We identify the key challenge of this task as
the exploration of disentangled conditional control in the generation process,
and accordingly propose a novel diffusion-based framework, named
DisControlFace, comprising two decoupled components. Specifically, we leverage
an off-the-shelf diffusion reconstruction model as the backbone and freeze its
pre-trained weights, which helps to reduce identity shift and recover editing-
unrelated details of the input image. Furthermore, we construct a parallel
control network that is compatible with the reconstruction backbone to
generate spatial control conditions based on estimated explicit face
parameters. Finally, we further reformulate the training pipeline into a
masked-autoencoding form to effectively achieve disentangled training of our
DisControlFace. Our DisControlNet can perform robust editing on any facial
image through training on large-scale 2D in-the-wild portraits and also
supports low-cost fine-tuning with few additional images to further learn
diverse personalized priors of a specific person. Extensive experiments
demonstrate that DisControlFace can generate realistic facial images
corresponding to various face control conditions, while significantly
improving the preservation of the personalized facial details.

Figure 1: Our DisControlFace can realistically edit the input portrait
corresponding to different explicit parametric controls and faithfully
generate personalized facial details of an individual. Our model also supports
other editing tasks, e.g., inpainting and semantic manipulations.

##  1 Introduction

Facial image editing has long been a hot research topic in the fields of
computer vision and computer graphics, where the key challenge is to
effectively achieve fine-grained controllable generation of realistic facial
images while preserving personalized facial priors.

3D Morphable Models (3DMMs) [3, 17] have been widely employed to represent
variations in facial shape and texture [4, 30, 17, 20, 56, 57, 58, 59],
whereas their ability to capture personalized facial features is limited and
the performance highly depends on the quality and diversity of the 3D face
training data. On top of this, subsequent learning-based explicit face models
[50, 63, 14, 7, 29, 12, 56, 57, 58, 59] achieve controllable generations of
dynamic and expressive facial animations by capturing the nuances of facial
features under different expressions, poses, and lighting conditions,
nevertheless, can neither generate realistic facial appearances that
correspond to the animated 3D face geometries nor model refined geometric
details in non-facial regions, e.g., hair, eyes, and mouth. Follow-up efforts
integrate explicit facial modeling with implicit 3D-aware representations like
Neural Radiance Fields (NeRFs) to reconstruct animated realistic head avatars
[15, 66, 41, 31, 40, 24, 42, 34, 33, 39, 51, 52], which however, heavily rely
on 3D consistent data such as monocular portrait videos and tend to exhibit
limited generalization.

In contrast, generative face models enable single image reconstruction and
editing due to the superior capability in learning rich face priors from in-
the-wild portraits. Recent GAN-based approaches [6, 5, 10, 19, 2, 23, 37, 55,
62] incorporate explicit 3D facial priors and implicit neural representations
to achieve directed generations of high-resolution, realistic, and view-
consistent facial images without the need of 3D face scans or portrait videos.
However, those methods mainly provide implicit or limited explicit controls of
face generations and rarely delve into preserving personalized facial details
during editing.

More recently, the diffusion-based framework [22] has emerged as the
predominant choice for various generation tasks, owing to its impressive
performance and diverse conditioning options. Some approaches have also shown
promising enhancements in face reconstruction and various face editing tasks,
such as face relighting [43], semantic attributes manipulation [44], and
explicit appearance control [13]. Unfortunately, it can be seen that when
editing and modifying some specific facial attributes, other facial attributes
often occur unexpected and uncontrollable changes, leading to an incoherent
and identity-altered generated face. This prevalent issue can be attributed to
that these generative face models struggle to effectively perform disentangled
control in the generation process.

In this work, we propose a novel diffusion-based generative framework, namely,
DisControlFace, to explicitly control facial image editing while faithfully
preserving facial identity. Specifically, to provide a strong identity
preservation and make the subsequent disentangled control possible, we adopt
pre-trained Diff-AE [11] with frozen weights as a deterministic reconstruction
backbone rather than training from scratch like existing conditional diffusion
models. Next, we specially construct a Face-ControlNet tailed for the Diff-AE
backbone, which generates multi-scale, spatial-aware control conditions
corresponding to different explicit facial parameters, e.g., FLAME and
lighting parameters, for the diffusion generation process. In additions, we
further introduce mask-autoencoding strategy to disrupt the spatial
information of the face inherent in the semantic encoder of Diff-AE, and
thereby effectively achieving disentangled training of the model. We employ
large-scale 2D in-the-wild portraits to train our DisControlFace to learn a
robust and generalized capability in explicit face editing, On this basis, it
is also easy to further adopt a low-cost personalized fine-tuning to enable
the generic model to generate the edited images with faithful facial details
of a specific person. Our approach not only achieves state-of-the-art (SOTA)
qualitative and quantitative results for personalized facial image editing,
but also generates realistic and faithful facial appearance of specific
individuals in image inpainting and semantic attributes manipulations (see
Figure 1).

Our contributions can be summarized as follows:

  * •

We propose a novel generation framework for synthesizing personalized
portraits corresponding to the editing of multiple explicit facial properties
only trained with 2D in-the-wild images.

  * •

We achieve disentangled conditional control for diffusion-based generation
process by constructing a decoupled conditioning mechanism and designing a
masked-autoencoding-based training strategy, which allows fine-grained control
of explicit face editing while generating consistent personalized facial
appearance.

  * •

Our generation framework can handle various related tasks including explicit
face editing, image inpainting, and semantic attributes manipulation, also
supports low-cost face animation through personalized fine-tuning.

##  2 Related Work

Generative Face Modeling. Various GAN-based models [35, 27, 28, 49, 6, 38]
have been proposed to synthesize realistic facial images by learning the
underlying data priors from large-scale in-the-wild images [49, 6, 38].
However, when it comes to precise control and interpretability, those
generative face models fall short compared to explicit parametric models.
Given this, several approaches [5, 55, 62, 54] go a step further by
integrating explicit parameters and GANs, which can simultaneously generate
highly realistic and coherent portraits and achieve fine-grained control of
facial appearance. Recently, diffusion models [22, 53] have gained recognition
for their superior ability to learn data distributions compared to GANs, and
thus have been widely adopted to generate realistic and diverse images in
various generation tasks, including facial image synthesis [44, 44].
DiffusionRig [13], a closely related method, introduces pixel-aligned physical
properties rendered from explicit parameters estimated by DECA [14] to
denoising diffusion process to generate photorealistic facial images
corresponding to target pose, expression, and lighting conditions, which
however, highly relies on a personalized fine-tuning (around 20 images) to
learn specific facial priors. Similar problems widely exist in most
conditional diffusion face models, as can be observed where identity shifts
and unexpected attributes alterations may occur during face reconstruction and
editing. This can be attributed to the lack of disentangled control
capabilities when conditioning diffusion models with various semantic or
physical information. DisControlFace overcomes this challenge by constructing
a parallel control network for explicit face editing and accordingly designing
an effective masked-autoencoding training strategy.

Conditional Diffusion Model. Conditional Denoising Diffusion Implicit Model
(DDIM)[53] enables the Denoising Diffusion Probabilistic Model (DDPM) [22] to
generate content consistent with specific control signals though various
conditioning manners. Most existing models encode various control information
into global conditional vectors, which can be text embedding [45] or semantic
embedding [44, 43]. To achieve spatial-aware and precise control of the
generation, some approaches (e.g., DiffusionRig [13] and SR3 [48] )
concatenate various spatial conditions and denoised images together as the
input of the U-Net noise predictor in each denoising step. However, this form
of conditioning requires the U-Net to have a unique input layer, resulting in
the model having limited generalization and making it hard to reuse existing
well-trained diffusion models. Besides, some other methods specially construct
spatial conditioning branches to extract spatial-aligned conditional features
and insert them into the U-Net [65, 61, 43]. ControlNet [65] has been widely
employed to add various spatial visual guidance (e.g., edge maps, pose maps,
depth maps, etc.) to text-to-image generation models such as Stable Diffusion
[45]. Whereas, it may not suitable for deterministic reconstruction or editing
tasks since there still exists uncertainty and randomness in the generation
controlled by visual guidance and text prompts. In contrast, our Face-
ControlNet is specially designed to be compatible with a deterministic
reconstruction backbone, aiming to better address facial image editing (see
detailed discussions in the supplement).

Learning Personalized Facial Priors. A crucial step of personalized facial
image synthesis is effectively extracting the facial prior of the specific
person and further injecting this identity prior into the generative models.
Existing methods address this issue by fine-tuning the model using a few
specific individual portraits in various settings [36, 25, 16, 47, 13], or
designing special learning strategies, such as identity penalty, face
recognition loss, and face embedding editing [32, 1, 64, 60]. Considering that
even facial attributes like pose and expression of the same person often
exhibit a high degree of complexity and diversity, it is more reasonable to
utilize additional portraits to learn more faithful and personalized facial
representations. In this work, we further focus on learning the personalized
features corresponding to various facial appearance control conditions, aiming
to enhance the identity consistency and realism during face attributes
editing.

##  3 Preliminaries

###  3.1 3D Morphable Face Models

FLAME [30] , a popular 3DMM model, can be expressed as
$M(\boldsymbol{\beta},\boldsymbol{\theta},\boldsymbol{\psi}):\mathbb{R}^{\lvert\boldsymbol{\beta}\rvert\times\lvert\boldsymbol{\theta}\rvert\times\lvert\boldsymbol{\psi}\rvert}\rightarrow\mathbb{R}^{3N}$,
which takes shape $\boldsymbol{\beta}$, pose $\boldsymbol{\theta}$, and
expression $\boldsymbol{\psi}$ as inputs and outputs a face mesh with $N$
vertices. On this basis, some off-the-shelf 3D face estimators, such as DECA
[14] and EMOCA [7], achieve 3D face reconstruction by regressing individual-
specific FLAME parameters from in-the-wild images. We utilize EMOCA to obtain
a face mesh corresponding to the input image with enhanced expression
consistency, and render it to pixel-aligned explicit conditions. Specifically,
given a single image $I$, the coarse branch of EMOCA estimate its
corresponding $\boldsymbol{\beta}$, $\boldsymbol{\theta}$,
$\boldsymbol{\psi}$, albedo $\boldsymbol{\alpha}$, spherical harmonic (SH)
illumination coefficient $\boldsymbol{l}$, and camera $c$, which can be
expressed as
$E_{c}(I)\rightarrow(\boldsymbol{\beta},\boldsymbol{\theta},\boldsymbol{\psi},\boldsymbol{\alpha},\boldsymbol{l},\boldsymbol{c})$.
The detailed branch further outputs a detail vector $\delta$ and computes the
displacement map $D$ in UV space, which is specifically expressed as
$E_{d}(I)\rightarrow\boldsymbol{\delta}$ and
$F_{d}(\boldsymbol{\delta},\boldsymbol{\psi},\boldsymbol{\theta}_{jaw})\rightarrow
D$.

Figure 2: Pipeline overview. Our DisControlNet consists of a Diff-AE
reconstruction backbone and a Face-ControlNet control network. Diff-AE
backbone is frozen with pre-trained weights to inherit the near-exact
reconstruction ability, consequently aiding in identity preservation and
subsequent fine-grained control of editing. Then, a Face-ControlNet tailored
for the Diff-AE backbone is constructed, which takes pixel-aligned snapshots
rendered from estimated explicit parameters as inputs and generates multi-
scale control features to condition the diffusion generation. Moreover, a
training strategy based on masked-autoencoding is accordingly designed to
enable the model to learn disentangled control for personalized face editing.

###  3.2 Diff-AE

Diff-AE [44] reformulates the traditional diffusion generation model into an
autoencoder for near-exact reconstruction and attribute manipulation of facial
image. Specifically, Diff-AE uses a semantic encoder to generate a
512-dimentional latent code $z$ with global facial semantics of the input
image $x_{0}$. Then, $z$ can be introduced to the reverse deterministic
generative process of DDIM to obtain a noisy map $x_{T}$ which captures the
stochastic variations of $x_{0}$. Last, a conditional DDIM model decodes
$(z,x_{T})$ to achieve a deterministic reconstruction of $x_{0}$. By linearly
modifying the semantic latent code $z$, Diff-AE can manipulate diverse global
facial attributes, e.g., age, gender, and hairstyle.

##  4 DisControlFace

In the pursuit of an ideal personalized facial image editing, we seek to
construct a generic model to achieve fine-grained facial control while
maintaining facial identity of the input portrait, whereafter utilizing a low-
cost personalized fine-tuning to further improve the faithfulness and
consistency of the edited facial appearance.

To this end, we propose a novel framework, namely DisControlFace (shown in
Figure 2), models identity preservation and controllable editing by two
decoupled components. Specifically, we adopt a frozen Diff-AE reconstruction
backbone to learn strong facial priors and recover editing-unrelated details
of the input image (Sec. 4.1). Furthermore, we construct a parallel Face-
ControlNet to control explicit face editing (Sec. 4.2). We further design a
disentangled training strategy for the constructed decoupled branches (Sec.
4.3). Our DisControlFace also supports low-cost personalized fine-tuning using
few additional portraits of specific individuals (Sec. 4.4). Finally, we
introduce the inference editing (Sec. 4.5). We now delve into the details of
each major component of DisControlFace and please see the supplement for more
details.

###  4.1 Deterministic Reconstruction Backbone

Different from most previous methods, which train the whole conditional-DDIM
generation model from scratch, we adopt Diff-AE [44] (introduced in Sec. 3.2)
as the reconstruction backbone of our DisControlFace and freeze its pre-
trained weights in training, which endows the model to inherit and maintain
the deterministic face reconstruction capability of Diff-AE. Considering that
the semantic encoder of Diff-AE extracts global facial semantics, including
identity, background information as well as other high-level semantic
attributes, which are expected to be maintained in personalized editing, this
setting can enable the model to preserve the identity of the input portrait to
the fullest extent and makes it possible to train Face-ControlNet to learn
fine-grained and disentangled control of face editing.

###  4.2 Face-ControlNet

Compared to directly adopting the explicit face parameters as non-spatial
control conditions, generating pixel-aligned conditional maps with those
parameters is more conducive and compatible to convolution-based visual
representation, which also helps enable fine-grained spatial control for the
denoising diffusion process. In this work, we specially construct Face-
ControlNet, a parallel control network compatible with the adopted Diff-AE
reconstruction backbone to perform a disentangled spatial conditioning for
diffusion-based generation. Specifically, considering U-Net [46] excels at
extracting spatial features from images for various vision tasks, we construct
Face-ControlNet as a U-Net, which takes pixel-aligned visual guidance map as
the input and generates multi-scale deep features for spatial-aware
conditioning. Then we feed back the spatial-condition features outputted by
each stage of the U-Net’s decoder back to Diff-AE to provide multi-scale
conditional control. The overall architecture is shown in Figure 2 and please
see detailed settings and more discussions in the supplement.

we endow the model with capabilities in changing the shape, pose, expression,
and lighting of the input image through an explicit parametric rendering.
Specifically, we use EMOCA [7] to predict FLAME parameters (including shape
$\beta$, pose $\theta$, and expression $\psi$), SH illumination parameter $l$,
and camera parameter $c$ from a input portrait. Different from those previous
work [13, 18, 43], here we also adopt the person-specific detail vector
$\delta$ estimated by EMOCA, which can be combined with $\theta$ and $\psi$ to
generate the expression-dependent displacement map for refining the face
geometry with animatable wrinkle details. To avoid introducing unrelated
facial appearance priors caused by inaccurate and unrealistic appearance
estimation, we set the albedo map $\alpha$ to a constant gray value, thereby
focusing on controlling the edit of shape, pose, expression, and lighting. We
first render these explicit parameters into a surface normal snapshot $n$:

$n=\mathcal{R}(\mathcal{G}(\mathcal{M}(\beta,\theta,\psi)),c,\delta)$ (1)

where the FLAME model $\mathcal{M}$ is used to calculate the 3D face mesh,
$\mathcal{G}$ and $\mathcal{R}$ indicate normal calculation function and the
Lambertian reflectance renderer, respectively. By means of this, $n$ can
reflect the fine-grained geometry of the input face and is compatible with
pose parameter $\theta$ and expression parameter $\psi$. Furthermore, we also
generate a shading shape snapshot $s$ to illustrate the lighting conditions
associated with the SH illumination parameter $l$:

$s=\mathcal{R}(\mathcal{M}(\beta,\theta,\psi),\alpha,l,c,\delta)$ (2)

Finally, we concatenate $n$ and $s$ along the channel dimension as the input
visual guidance map of Face-ControlNet.

###  4.3 Disentangled Training with Masked Diff-AE

From the perspective of conditional DDIM, DisControlFace can be regarded as a
denosing diffusion process that is simultaneously conditioned by an identity-
inherent global code and multi-scale explicit-control feature maps (see Figure
2). Unfortunately, due to the decoupling setting of DisControlFace, where the
pre-trained Diff-AE backbone can already allow near-exact image
reconstruction, only limited gradients can be generated during error back-
propagation, which are far from sufficient to effectively train Face-
ControlNet. Consequently, we find that it is infeasible to train Face-
ControlNet in a traditional conditional DDIM generation form. To address this
issue, we introduce the idea of Masked Autoencoders (MAE) [21] to the training
of DisControlFace and further formulate the current pipeline as a masked
diffusion autoencoder (Masked Diff-AE). Concretely, we divide the input image
$x_{0}$ into regular non-overlapping patches and randomly mask different
portions of patches to obtain a masked image $x_{0}^{m}$ at different
timesteps. By means of this, the semantically latent code $z^{m}$ encoded by
the semantic encoder $\mathcal{E}_{\eta}$ of Diff-AE only contains fragmented
and incomplete content and spatial information of $x_{0}$. Meanwhile, the
spatial-condition features $f$ generated by Face-ControlNet
$\mathcal{F}_{\phi}$ comprise the fine-grained face shape as well as the
camera parameter and lighting condition of $x_{0}$, which can help to recover
the masked face regions in $x_{0}^{m}$ in each random denoising timestep. The
overall training objective can thereby be parameterized as:

$\mathcal{L}=\mathbb{E}_{x_{0},x_{0}^{m},t,\epsilon}[\|\epsilon-\epsilon_{\theta}(x_{t},t,z^{m},f)\|_{2}^{2}]$
(3)

where $\epsilon_{\theta}$ is the U-Net of Diff-AE which predicts the noise
$\epsilon\sim\mathcal{N}(0,\mathbf{I})$ added in noisy image $x_{t}$.
Throughout the generalized training, we only train $\mathcal{F}_{\phi}$ and
freeze $\epsilon_{\theta}$ and $\mathcal{E}_{\eta}$ with the pre-trained
weights.

###  4.4 Personalized Portraits Fine-tuning

At this point, our DisControlFace is able to explicitly change pose,
expression, and lighting of a facial image without significant identity shift.
However, considering the personalized semantic information encoded from a
single facial image is still limited by the fixed expression, motion,
lighting, and background information, it is meaningful to inject the model
with diverse facial appearance priors of the same person. Given this, we
additionally perform a personalized fine-tuning for the DisControlFace, where
we collect a small set of in-the-wild portraits of a specific person as the
fine-tuning dataset.

We divide personalized fine-tuning into two stages. First, we fine-tune
$\mathcal{E}_{\eta}$ and $\epsilon_{\theta}$ following the original training
mechanism of Diff-AE [44]. Then, we freeze $\mathcal{E}_{\eta}$ and
$\epsilon_{\theta}$ and fine-tune $\mathcal{F}_{\phi}$ using the proposed
masked-Diff-AE disentangled training introduced in Sec. 4.3. Following
previous work [13], we estimate the FLAME parameters all portraits of a
specific person and adopt the mean shape parameter to compute the shading
shape snapshot and the surface normal snapshot in the second fine-tuning
stage. Benefit from the powerful face prior extraction capability of the Diff-
AE backbone and the generalized face editing control capability of Face-
ControlNet, in practice, we only need to conduct fine-tuning for few epochs in
both stages. Please see more details in the supplement.

Figure 3: Comparison to SOTA methods in one-shot editing on FFHQ [27]. For
each selected image, we use EMOCA [7] to estimate the corresponding explicit
parameters, then synthesize the edited images using different methods with the
updated parameters of pose, expression, and lighting. We additionally provide
the rendered shading shapes in the second row as references. As can be seen,
our DisControlFace can edit images that match well with the target control
information while faithfully synthesizing facial appearances with minimal
identity shift.

###  4.5 Inference Editing

In practice, we first use EMOCA to predict all explicit parameters (mentioned
in Sec. 4.2) of the input portrait $x_{0}$, then we modify the pose parameters
$\theta$, expression parameter $\psi$, and SH light parameter $l$ by manually
setting target values or directly transferring these parameters from a
reference portrait. After this, we calculate the rendered shading shape
snapshot $s$ and surface normal snapshot $n$ based on the modified parameters,
and further generate the corresponding control conditions using Face-
ControlNet. On the other hand, we generate the global latent code $z_{s}$ of
the masked $x_{0}$ using the semantic encoder $\mathcal{E}_{\eta}$ of Diff-AE,
where we design an intuitive masking strategy for denoising inference.
Concretely, for each timestep $t$, we generate the masked image $x^{m}_{t}$ by
randomly masking the patches of $x_{0}$ with a linear ratio
$\rho_{t}=0.75-0.5(T-t)/T$, where the number of the inference denoising steps
$T$ is set to 20 in this work. Under this setting, we can generate $x^{m}_{t}$
with high masking ratios $\rho_{t}$ to emphasize the facial control of the
intermediate denoising result $z_{t-1}$ in the early stage of the inference,
and then gradually decrease $\rho_{t}$ to inject the semantic information into
$z_{t-1}$ to recover the identity.

##  5 Experiments

Datasets. We train and evaluate the generic model on FFHQ [27], a public
dataset with 70k in-the-wild facial images. For personalized fine-tuned model,
we follow DiffurionRig [13] to collect several in-the-wild images of different
celebrities from the internet for fine-tuning and evaluation. Various
comparisons and ablation studies are presented in Sec. 5.1 and 5.2,
respectively. Besides, we further evaluate the well trained generic model on
an out-of-domain dataset CeleA-HQ [26] and adopt the monocular RGB video
sequences provided in NeRFace [15] to achieve a high-quality personalized
fine-tuning. Please see the accompanying video and supplement for additional
results.

###  5.1 Comparison

Baselines. We compare our methods against three SOTA methods for parametric
face image synthesis and editing: HeadNeRF [24], GIF [18], and DiffusionRig
[13]. Among these, HeadNeRF is a NeRF-based head model, while GIF and
DiffusionRig are both generative face models built upon GAN and diffusion
model, respectively.

One-shot editing results. We first evaluate our DisControlFace in an one-shot
face editing scenario, to be specific, we adopt generic model to perform face
editing for in-the-wild facial images in FFHQ dataset. The qualitative
comparison against baselines are provided in Figure 3. For all methods, we
visualize the editing results of pose, expression, and lighting on three
identities at a resolution of $256^{2}$. In order to intuitively measure the
editing accuracy, we also give the rendered snapshot $s$ of the shading shape
corresponding to the modified explicit parameters (second row of Figure 3). We
can find that our method outperforms all baselines on identity preservation
and editing accuracy. Specifically, HeadNeRF cannot generate realistic facial
appearance as well as the background of the original image. GIF has a good
control ability of parametric face editing, however, is completely unable to
preserve the face identity. DiffusionRig, another diffusion-based method, can
achieve overall good editing results, nevertheless, still change the face
identity by some degree (e.g., eyes, lip color, and hair) and struggles to
generate the details consistent with the snapshot $s$ (e.g., mouth shape and
smile lines). Compared to the other methods, our method can not only
synthesize images with consistent identity, but perform robust editing with
accurate details.

Figure 4: Comparison to DiffusionRig [13] in Few-shot editing. We fine-tune both models with 5 portrait images of Taylor Swift and visualize the editing results of another image in pose, expression, and lighting. Our DisControlFace can effectively preserve the facial priors and generate realistic and natural edited images. | ID $\uparrow$ | Shape$\downarrow$ | Pose $\downarrow$ | Exp $\downarrow$ | Light $\downarrow$
---|---|---|---|---|---
GIF [18] | 0.22 | 3.0 | 5.6 | 5.0 | 0.40
DiffusionRig [13] | 0.24 | 4.3 | 4.2 | 2.8 | 0.36
Ours | 0.31 | 2.8 | 4.5 | 2.9 | 0.31
DiffusionRig† [13] | 0.59 | $\setminus$ | 4.9 | 1.3 | 0.31
Ours† | 0.81 | $\setminus$ | 2.1 | 1.4 | 0.28
Table 1: Quantitative comparisons against SOTA methods using identity
consistency (ID) and DECA re-inference errors on shape, pose, expression, and
lighting. † represents fine-tuning the model with person-specific images.

Table 1 (top part) provides the quantitative editing results of all methods on
1000 images of FFHQ. Following previous work [18, 9, 13], we apply DECA re-
inference on edited images and calculate the Root Mean Square Error (RMSE)
between the input and re-inferred face vertices as well as spherical harmonics
to evaluate the editing accuracy on shape, pose, expression, and lighting. We
additionally measure the identity consistency (ID) by computing the cosine
similarity between the deep features generated by ArcFace [8] of the original
and edited images. The results indicate that our method achieve the overall
best performance on all metrics. Note that DisControlFace outperforms other
methods by a large margin in ID, which demonstrates the superiority of our
method in identity preservation during one-shot editing.

Figure 5: Ablation study on person-specific detail vector $\delta$. Adopting
the detail vector $\delta$ estimated by EMOCA [7] can help to generate the
control condition with refined face geometry, which further allows faithful
editing with consistent facial details. Figure 6: Ablation study on number of
images used in personalized fine-tuning. Fine-tuning with 5 images already
enables our model to generate satisfying editing results and using more images
can further refine the facial details.

Few-shot editing results. We further compare our DisControlFace against
DiffusionRig [13] in few-shot face editing using personalized fine-tuned
models. Qualitative comparison is first presented in Figure 4, where we fine-
tune both models with 5 portraits and edit another image of the same celebrity
with explicit parameters set manually. We can observe that both our model and
DiffusionRig generate edited images consistent with target explicit face
parameters, while our DisControlFace is superior to DiffusionRig in preserving
personalized details, e.g., hair color and makeup. Meanwhile, DiffusionRig
struggles to generate natural and realistic mouth and teeth that correspond to
the target control conditions. In addition, quantitative results obtained by
both models fine-tuned with 20 portraits are given in Table 1 (bottom part),
which indicate the proposed DisControlFace gains overall better results on all
metrics and outperforms DiffusionRig by a significant gap in ID and pose
errors. Since we fix the shape parameter in the face editing of the same
person, here we do not compare the shape errors.

Figure 7: Synthesis results in semantic manipulation. We follow Diff-AE [11]
to manipulate the global facial attributes (age, wavy hair, and eyeglasses) by
linearly editing $z$. Owing to the disentangled control mechanism, our model
can simultaneously perform semantic manipulation and explicit editing of the
facial image.

###  5.2 Ablation Study

Person-specific detail vector $\delta$. When using the estimated FLAME
parameters to calculate the 3D face mesh, we additionally adopt the person-
specific detail vector $\delta$ to generate refined expression-dependent face
geometry. Figure 5 demonstrates that using $\delta$ can generate a rendered
snapshot with refined face geometry, which thereby helps to recover the
detailed expressions (e.g., wrinkles) of the edited face.

Number of images used in personalized fine-tuning. It is expected that
utilizing more images for personalized fine-tuning can benefit the model in
learning stronger facial priors of a person. Therefore, although our
DisControlFace can basically preserve the identity in one-shot face editing,
it is still meaningful to explore the editing performance using different
numbers of fine-tuning images. Figure 6 illustrates that fine-tuning the model
with 5 images can already visibly improve the synthesis quality of the edited
image, and adding the fine-tuning images to 10 and 20 can further slightly
enhance the personalized details of the face.

Figure 8: Visualization of conditions mixing where the global semantic
conditioning and explicit spatial conditioning are separately extracted from
two images of different persons.

###  5.3 Applications

Semantic manipulation. Since we adopt Diff-AE [11] as the reconstruction
backbone, our DisControlFace inherits the capability of changing the global
facial semantics. Therefore, compared to previous diffusion-based model such
as DiffusionRig [13], our model can simultaneously perform semantic
manipulation and explicit editing of the input portrait. The visualized
results are shown in Figure 7, which once again demonstrate the effectiveness
of the proposed disentangled control mechanism (see more in the supplement).

Conditions mixing. To further verify the disentangled conditioning of our
DisControlFace, we show the synthesis results by mixing the semantic condition
$z$ from one image and the control condition $f$ from another image in Figure
8. The visualized results demonstrate that the semantic encoder provides the
global information associate with the facial appearance and background while
Face-ControlNet controls the shape, pose, expression, and lighting of the
edited image.

Image inpainting. Benefit from the masked autoencoding characteristic of the
proposed disentangled training, our model also enables zero-shot image
inpainting. Figure 9 shows the results of inpainting on center-masked facial
images as well as the subsequent editing.

Figure 9: Zero-shot inpainting and subsequent editing on an out-of-domain
dataset CeleA-HQ [26]. DisControlFace can generate coherent and similar facial
appearance with original images

##  6 Limitations and Conclusion

Limitations and future work. We adopt EMOCA to estimate explicit face
parameters of the input image and generate spatial-aware conditions
representing control information based on them. However, EMOCA struggles to
model the detailed geometry of eyeballs as well as some extreme expressions,
which could hinder the model from recovering the corresponding facial details
when generating edited images.

Furthermore, our editing framework is constructed as a denoising diffusion
pipeline, which therefore is still unable to compete with GANs in terms of
generation speed. In the future, with the development of fast sampling
algorithms, we expect the generation time of our model to further decrease.

Conclusion. We have presented DisControlFace, a novel diffusion framework for
personalized facial image editing using unstructured 2D imagery. By means of a
deterministic reconstruction backbone and a parallel control network as well
as an effective disentangled training strategy, our model excels in explicit
and fine-grained control while generating identity-consistent facial details
in face editing. This may boost a series of related applications including
various semantic and explicit facial image editing, zero-shot and personalized
image inpainting, and low-cost face animation.

## References

  * [1] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 843–852, 2023.
  * [2] Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein.  Generative neural articulated radiance fields.  Advances in Neural Information Processing Systems (NeurIPS), 35:19900–19916, 2022.
  * [3] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157–164. 2023.
  * [4] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics (TVCG), 20(3):413–425, 2013.
  * [5] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16123–16133, 2022.
  * [6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5799–5809, 2021.
  * [7] Radek Daněček, Michael J Black, and Timo Bolkart.  Emoca: Emotion driven monocular face capture and animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20311–20322, 2022.
  * [8] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4690–4699, 2019.
  * [9] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5154–5163, 2020.
  * [10] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong.  Gram: Generative radiance manifolds for 3d-aware image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10673–10683, 2022.
  * [11] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  volume 34, pages 8780–8794, 2021.
  * [12] Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe-Henri Gosselin, Christian Theobalt, and Louis Chevallier.  Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12819–12829, 2021.
  * [13] Zheng Ding, Xuaner Zhang, Zhihao Xia, Lars Jebe, Zhuowen Tu, and Xiuming Zhang.  Diffusionrig: Learning personalized priors for facial appearance editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12736–12746, 2023.
  * [14] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (ToG), 40(4):1–13, 2021.
  * [15] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8649–8658, 2021.
  * [16] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  ACM Transactions on Graphics (TOG), 42(4):1–13, 2023.
  * [17] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter.  Morphable face models-an open framework.  In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 75–82. IEEE, 2018.
  * [18] Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael J Black, and Timo Bolkart.  Gif: Generative interpretable faces.  In Proceedings of the 2020 International Conference on 3D Vision (3DV), pages 868–878. IEEE, 2020.
  * [19] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.  arXiv preprint arXiv:2110.08985, 2021.
  * [20] Shalini Gupta, Kenneth R Castleman, Mia K Markey, and Alan C Bovik.  Texas 3d face recognition database.  In Proceedings of the IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI), pages 97–100. IEEE, 2010.
  * [21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022.
  * [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840–6851, 2020.
  * [23] Fangzhou Hong, Zhaoxi Chen, LAN Yushi, Liang Pan, and Ziwei Liu.  Eva3d: Compositional 3d human generation from 2d image collections.  In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2022.
  * [24] Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang.  Headnerf: A real-time nerf-based parametric head model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20374–20384, 2022.
  * [25] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [26] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [27] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401–4410, 2019.
  * [28] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8110–8119, 2020.
  * [29] Biwen Lei, Jianqiang Ren, Mengyang Feng, Miaomiao Cui, and Xuansong Xie.  A hierarchical representation network for accurate and detailed face reconstruction from in-the-wild images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 394–403, 2023.
  * [30] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Transactions on Graphics (ToG), 36(6):194–1, 2017.
  * [31] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt.  Neural actor: Neural free-view synthesis of human actors with pose control.  ACM Transactions on Graphics (TOG), 40(6):1–16, 2021.
  * [32] Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and SY Kung.  3d-fm gan: Towards 3d-controllable face manipulation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 107–125. Springer, 2022.
  * [33] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4460–4470, 2019.
  * [34] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [35] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7588–7597, 2019.
  * [36] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or.  Mystyle: A personalized generative prior.  ACM Transactions on Graphics (TOG), 41(6):1–10, 2022.
  * [37] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Unsupervised learning of efficient geometry-aware neural articulated representations.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 597–614. Springer, 2022.
  * [38] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 20002–20013, 2021.
  * [39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 165–174, 2019.
  * [40] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5865–5874, 2021.
  * [41] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14314–14323, 2021.
  * [42] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 523–540. Springer, 2020.
  * [43] Puntawat Ponglertnapakorn, Nontawat Tritrong, and Supasorn Suwajanakorn.  Difareli: Diffusion face relighting.  2023\.
  * [44] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10619–10629, 2022.
  * [45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022.
  * [46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241. Springer, 2015.
  * [47] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22500–22510, 2023.
  * [48] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 45(4):4713–4726, 2022.
  * [49] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 20154–20166, 2020.
  * [50] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted facial geometry reconstruction using image-to-image translation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1576–1585, 2017.
  * [51] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In Advances in neural information processing systems (NeurIPS), volume 33, pages 7462–7473, 2020.
  * [52] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2437–2446, 2019.
  * [53] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [54] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu.  Next3d: Generative neural texture rasterization for 3d-aware head avatars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20991–21002, 2023.
  * [55] Keqiang Sun, Shangzhe Wu, Zhaoyang Huang, Ning Zhang, Quan Wang, and Hongsheng Li.  Controllable 3d face synthesis with conditional generative occupancy fields.  In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 16331–16343, 2022.
  * [56] Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt.  Fml: Face model learning from videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10812–10822, 2019.
  * [57] Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt, et al.  Learning complete 3d morphable face models from images and videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3361–3371, 2021.
  * [58] Luan Tran, Feng Liu, and Xiaoming Liu.  Towards high-fidelity nonlinear 3d face morphable model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1126–1135, 2019.
  * [59] Luan Tran and Xiaoming Liu.  On learning 3d face morphable model from in-the-wild images.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(1):157–171, 2019.
  * [60] Dani Valevski, Danny Wasserman, Yossi Matias, and Yaniv Leviathan.  Face0: Instantaneously conditioning a text-to-image model on a face.  arXiv preprint arXiv:2306.06638, 2023.
  * [61] Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang.  Disco: Disentangled control for referring human dance generation in real world.  arXiv preprint arXiv:2307.00040, 2023.
  * [62] Yue Wu, Yu Deng, Jiaolong Yang, Fangyun Wei, Qifeng Chen, and Xin Tong.  Anifacegan: Animatable 3d-aware face image generation for video avatars.  In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 36188–36201, 2022.
  * [63] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 601–610, 2020.
  * [64] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng.  Inserting anybody in diffusion models via celeb basis.  arXiv preprint arXiv:2306.00926, 2023.
  * [65] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836–3847, 2023.
  * [66] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges.  Im avatar: Implicit morphable head avatars from videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13545–13555, 2022.

Figure 10: Detailed architecture of the proposed DisControlFace. Please zoom
in to see details.

##  7 Appendix

In this supplement, we present:

  * •

Section 7.1: Implementation details, including further descriptions of the
training and inference processes as well as detailed architecture of the
proposed model.

  * •

Section 7.2: Model analysis, including the discussion about the difference
between Face-ControlNet and ControlNet, and the validation of the necessity of
Masked Diff-AE training.

  * •

Section 7.3: Additional experiments, including additional ablation study on
different inference masking strategies and more visual results.

  * •

Section 7.4: More discussions about limitations and future work.

###  7.1 Implementation Details

All detailed configurations of generic training and fine-tuning are shown in
Table 2. The structural details of DisControlFace is presented in Figure 10.
Specifically, for Diff-AE backbone [44], we adopted its officially released
hyperparameters and pre-trained weights111https://github.com/phizaz/diffae.
corresponding to FFHQ256. Our Face-ControlNet mirrors the structure of the
Conditional DDIM (U-Net) in Diff-AE, which however, customizes the input layer
by setting the input channel number to 6, allowing it to take the concatenated
snapshots as input. Meanwhile, we fuse the 2D feature maps outputted by the
input layers of Conditional DDIM and Face-ControlNet by pixel-wise summation
and feed the fused feature maps into the subsequent layers of Face-ControlNet,
which tends to improve the quality of the generation images. Finally, to
further provide fine-grained conditioning for the diffusion generation
process, we add multi-scale features outputted by the middle and all decoder
blocks of the Face-ControlNet ($f_{A}$ to $f_{M}$ in Figure 10) back to the
corresponding blocks of Diff-AE.

| generic training | fine-tuning
---|---|---
Image size | 256
Patch size | 16
Image normalization | [-1, 1]
Masking ratio | sample from $U(0.25,0.75)$
Optimizer | AdamW (no weight decay)
Loss | MSE loss
EMA decay | 0.9999 | $\setminus$
Learning rate | 1e-4 | 1e-5
Batch size | 32 | 4
Denoising steps | 1000
Epochs | 200 | 10
Device | 8 V100s | 1 V100
Table 2: Training and fine-tuning configurations.

###  7.2 Model Analysis

Face-ControlNet v.s ControlNet

Figure 11: Generation examples of ControlNet [65] \+ Stable Diffusion 1.5
[45]. Here we adopt soft HED Boundary as the visual guidance of ControlNet and
the inference prompt is ”oil painting of handsome old man, masterpiece”.

Our Face-ControlNet is inspired from ControlNet [65], a popular deep network
which has been widely employed to add various spatial visual guidance (e.g.,
edge maps, pose maps, depth maps, etc.) to Stable Diffusion (SD) [45] for
text-to-image generation. However, there exists many differences between two
models. First, ControlNet is specially designed for SD which can generate
specific content based on the input prompts, while there still exists
uncertainty and randomness in the generation controlled by visual guidance and
text prompts. As shown in Figure 11, given a specific inference prompt and a
HED Boundary map of a portrait image, SD combined with ControlNet generates
images matching the input HED Boundary but with diverse details and semantics.
In contrast to this, this paper focuses on reconstruction and editing of the
input portrait image. Given this, our Face-ControlNet is designed to be
compatible with a deterministic reconstruction backbone, aiming to better
address facial image editing while recovering all edit-unrelated details. Also
due to this, different from ControlNet, it is infeasible to train Face-
ControlNet with the traditional training strategy of diffusion model. Second,
we construct Face-ControlNet as a U-Net instead of adopting zero convolutions
introduced in ControlNet. With this setting, we can endow Face-ControlNet with
strong capability of U-shape models in extracting spatial deep features from
pixel-aligned visual guidance map (two snapshots in this work). In practice,
we find that constructing deep decoder indeed can obtain better generation
quality than using zero convolutions.

Effectiveness of Masked Diff-AE training

Figure 12: The necessity of masked-autoencoding training for FaceControlNet.

In this paper, we achieve a disentangled training of the proposed Face-
ControlNet by reformulating the traditional training pipeline as a masked
diffusion autoencoder (Masked Diff-AE). To demonstrate the necessity of the
proposed Maksed Diff-AE training, we separately train Face-ControlNet with and
without random masking for the input image of semantic encoder of the Diff-AE
backbone. Figure 12 shows that both training strategies enable the model to
reconstruct the input image. However, only the model trained in the form of
Masked Diff-AE can generate images with novel poses. This result supports the
point we mentioned in the main paper that only limited gradients can be
generated during error back-propagation since the pre-trained Diff-AE backbone
can already allow near-exact image reconstruction, which are far from
sufficient to effectively train Face-ControlNet. By means of our Masked Diff-
AE training, the semantic encoder of Diff-AE can generate the semantic latent
code containing fragmented and incomplete content and spatial information of
the input image, which facilitates Face-ControlNet to recover the masked face
regions and thereby learning explicit face control.

###  7.3 Additional Experiments

Additional ablation study on different masking strategies in inference. Here
we explore how the masking strategy used in inference affects the generation
performance. The comparison is shown in Figure 13, we can observe that when
setting masking ratio to 25% for all inference steps, the edited image can not
match the control signal very well which can be attributed to strong
deterministic reconstruction in this setting. Meanwhile, setting masking ratio
to 75% for all inference steps slightly harm the semantics recovering.
Besides, we can see the other three masking strategies can achieve better
editing results where the proposed linear masking ratio can perform overall
best editing with accurate face control and good preservation of facial
semantics.

Figure 13: Ablation study on different masking strategies in inference. We set
the inference denoising steps to 20 for all masking strategies. Strategy A:
the masking ratio is set to 25% for all 20 steps; Strategy B: the masking
ratio is set to 75% for all 20 steps; Strategy C: the masking ratio is set to
25% and 75% for the first 10 steps and last 10 steps; Strategy D: the masking
ratio is set to 75% and 25% for the first 10 steps and last 10 steps: Strategy
D: the linear masking ratio introduced in the Section 4.5 of the main paper.
Figure 14: More results in semantic manipulation. Figure 15: More few-shot
editing results. For Barack Obama and Taylor Swift, we use 20 portrait images
for fine-tuning. For the person from NeRFace dataset, we sample 500 frames
from a monocular video to fine-tune the model. Figure 16: zero-shot editing
results on CelebA-HQ dataset [26].

More visual results. In this section, we provide additional visual results of
our DisControlFace:

  * •

Figure 14 visualizes semantic manipulations on Obama, Taylor, and a person in
NeRFace dataset.

  * •

Figure 15 shows more few-shot editing results.

  * •

Figure 16 provides zero-shot editing results on CeleA-HQ dataset [26].

###  7.4 Additional Limitations and Future Work

it would be more reasonable to perform random masking only on the face-related
patches Our DisControlFace has a parallel editing control network besides the
U-Net noise predictor and performs denoising diffusion process in image space,
which results in the model being able to generate images with limited
resolutions. Potential future improvements includes introducing a light-weight
super-resolution network to the model or extending the model to a latent
diffusion version.

In the proposed Masked Diff-AE training, we randomly mask some patches of the
whole image and take the masked image as the input of the semantic encoder of
Diff-AE backbone to achieve an effective disentangled training of Face-
ControlNet. Meanwhile, only performing random masking on face-related patches
is expected to further improve the consistency of the background region of the
input portrait with different editing applied, which however might slightly
increase the training and inference time. Corresponding explorations and
experiments can serve as another potential valuable future work.


# Solution estimates and stability tests for linear neutral differential
equations

Leonid Berezansky  Elena Braverman  Dept. of Math., Ben-Gurion University of
the Negev, Beer-Sheva 84105, Israel  Dept. of Math. and Stats., University of
Calgary,2500 University Drive N.W., Calgary, AB, Canada T2N 1N4; e-mail
<EMAIL_ADDRESS>phone 1-(403)-220-3956, fax 1-(403)–282-5150
(corresponding author)

###### Abstract

Explicit exponential stability tests are obtained for the scalar neutral
differential equation

$\dot{x}(t)-a(t)\dot{x}(g(t))=-\sum_{k=1}^{m}b_{k}(t)x(h_{k}(t)),$

together with exponential estimates for its solutions.

Estimates for solutions of a non-homogeneous neutral equation are also
obtained, they are valid on every finite segment, thus describing both
asymptotic and transient behavior. For neutral differential equations,
exponential estimates are obtained here for the first time. Both the
coefficients and the delays are assumed to be measurable, not necessarily
continuous functions.

###### keywords:

linear neutral differential equations, exponential stability tests, explicit
solution estimates, variable delays and coefficients AMS subject
classification: 34K20, 34K25, 34K06

††journal: Applied Mathematics Letters

##  1 Introduction

In recent papers [3, 4] we obtained new exponential stability conditions for a
scalar linear neutral differential equation

$\dot{x}(t)-a(t)\dot{x}(g(t))=-b(t)x(h(t)),~{}~{}t\geq t_{0}.$ (1.1)

Here the functions $a,b,g,h$ are assumed to be Lebesgue measurable, $b$ is
essentially bounded on $[t_{0},\infty)$, $a$ satisfies

$|a(t)|\leq A_{0}<1,~{}~{}t\geq t_{0}.$ (1.2)

The condition on the delay in the neutral term

$mes~{}E=0\Longrightarrow mes~{}g^{-1}(E)=0,$ (1.3)

where $mes~{}E$ is the Lebesgue measure of the set $E$, guarantees that
$u(g(t))$ is properly defined and is Lebesgue measurable for any measurable
$u$. The delays in both terms of (1.1) are variable but bounded: for some
$\tau>0$ and $\sigma>0$, $0\leq t-g(t)\leq\sigma$, $0\leq t-h(t)\leq\tau$ for
$t\geq t_{0}$. All the functions are considered in the space $L_{\infty}$ of
Lebesgue measurable essentially bounded functions with the essential supremum
norm $\|\cdot\|_{J}$ on a certain segment
$J\subset[t_{0}-\max\\{\tau,\sigma\\},\infty)$.

######  Proposition 1.

[3] If at least one of the following conditions holds

$0<b_{0}\leq
b(t),~{}\|b\|_{[t_{0},\infty)}\tau\leq\frac{1}{e}\,,~{}\|a\|_{[t_{0},\infty)}+\|b\|_{[t_{0},\infty)}\left\|\frac{a}{b}\right\|_{[t_{0},\infty)}<1;$
(1.4)
$\left\|\frac{a}{b_{1}}\right\|_{[t_{0},\infty)}\frac{\|b\|_{[t_{0},\infty)}}{1-\|a\|_{[t_{0},\infty)}}+\left\|\frac{b-b_{1}}{b_{1}}\right\|_{[t_{0},\infty)}<1,\mbox{
where ~{}}b_{1}(t):=\min\left\\{b(t),\frac{1}{\tau e}\right\\};$ (1.5)
$\displaystyle\|b\|_{[t_{0},\infty)}\left(\left\|\frac{a}{b}\right\|_{[t_{0},\infty)}+\left\|\left(t-h(t)-\frac{1}{\|b\|_{[t_{0},\infty)}e}\right)^{+}\right\|_{[t_{0},\infty)}\right)<1-\|a\|_{[t_{0},\infty)},$
(1.6)

where $u^{+}=\max\\{u,0\\}$, equation (1.1) is uniformly exponentially stable.

######  Proposition 2.

[4, Theorem 1(a)] Assume that $0\leq a_{0}\leq a(t)\leq A_{0}<1$, $0<b_{0}\leq
b(t)\leq B_{0}$ and $\displaystyle\tau B_{0}+\frac{\sigma
A_{0}B_{0}^{2}(1-a_{0})}{(1-A_{0})^{2}b_{0}}<1-A_{0}$. Then equation (1.1) is
uniformly exponentially stable.

In the present paper, we consider a generalization of (1.1) to the case of
several delays in the non-neutral part

$\dot{x}(t)-a(t)\dot{x}(g(t))=-\sum_{k=1}^{m}b_{k}(t)x(h_{k}(t)).$ (1.7)

We refer the reader to the review of known stability results for neutral
equations in [3]. For a particular case of constant coefficients and delays
sharp results are obtained in [2]. The recent paper [5] contains exponential
estimates for solutions of delay differential equations without the neutral
term ($a(t)\equiv 0$ in (1.7)), see also [10]. The purpose of the present
paper is two-fold.

  1. 1.

Investigate stability of (1.7) and get sufficient conditions which involve all
the delays in the non-neutral part. Among other results, we obtain a new
exponential stability test which has a very simple form and can be applied to
a wide class of neutral equations.

  2. 2.

Develop explicit exponential estimates for solutions of (1.7) and its non-
homogeneous version, dependent on the right-hand side and the initial
functions. These inequalities are valid on every finite segment, thus both
describing asymptotic and transient behavior. For neutral differential
equations, exponential estimates are obtained here for the first time.

Note that our assumptions refer to a half-line $[t_{0},\infty)$, which is
essential for exponential stability tests. However, once all the parameters
are considered on a finite segment $[t_{0},t_{1}]$, solution estimates on this
interval remain valid. Only few asymptotic formulas for solutions of neutral
differential equations are known, see [6, 12, 13] and references therein, and
the present paper fills this gap.

The paper is organized as follows. Section 2 includes some definitions and
auxiliary results, among them an estimate for the fundamental function of an
equation with a non-delay term. Section 3 is the main part of the paper where
we obtain stability tests and develop solution estimates for equation (1.7)
and its non-homogeneous version. Section 4 presents illustrating examples and
discussion.

##  2 Preliminaries

We consider scalar delay differential equation (1.7) where, similarly to
(1.1), $a,b_{k},g,h_{k}$ are Lebesgue measurable, $a$ satisfies (1.2) and
$b_{0}\leq b_{k}(t)\leq B_{k}$ for $t\geq t_{0}\geq 0$, implication (1.3)
holds and there are $\sigma>0$, $\tau_{k}>0$ such that $0\leq
t-g(t)\leq\sigma$, $0\leq t-h_{k}(t)\leq\tau_{k}$ for $t\geq t_{0}$,
$i=1,\dots,m$.

Along with (1.7), we consider an initial value problem for a non-homogeneous
equation

$\dot{x}(t)-a(t)\dot{x}(g(t))+\sum_{k=1}^{m}b_{k}(t)x(h_{k}(t))=f(t),~{}t\geq
t_{0},~{}$ (2.1) $x(t)=\varphi(t),~{}t\leq
t_{0},~{}\dot{x}(t)=\psi(t),~{}t<t_{0},$ (2.2)

where $f:[t_{0},\infty)\rightarrow{\mathbb{R}}$ is a Lebesgue measurable
locally essentially bounded function,
$\varphi:(-\infty,t_{0}]\rightarrow{\mathbb{R}}$ and
$\psi:(-\infty,t_{0})\rightarrow{\mathbb{R}}$ are Borel measurable bounded
functions.

Further, we assume that the above conditions hold for (1.7) and (2.1)-(2.2)
without mentioning it, as well as similar conditions for all other neutral
equations considered in the paper.

######  Definition 1.

A locally absolutely continuous on $[t_{0},\infty)$ function
$x:{\mathbb{R}}\rightarrow{\mathbb{R}}$ is called a solution of problem
(2.1)-(2.2) if it satisfies equation (2.1) for almost all $t\in[t_{0},\infty)$
and the equalities in (2.2) for $t\leq t_{0}$. For each $s\geq t_{0}$, the
solution $X(t,s)$ of the problem

$\dot{x}(t)-a(t)\dot{x}(g(t))+\sum_{k=1}^{m}b_{k}(t)x(h_{k}(t))=0,~{}x(t)=0,~{}\dot{x}(t)=0,~{}t<s,~{}x(s)=1$
(2.3)

is called the fundamental function of equation (1.7). We assume $X(t,s)=0$ for
$0\leq t<s$.

######  Definition 2.

We will say that equation (1.7) is uniformly exponentially stable if there
exist $M>0$ and $\gamma>0$ such that the solution of problem (2.1)-(2.2) with
$f\equiv 0$ has the estimate $\displaystyle|x(t)|\leq
Me^{-\gamma(t-t_{0})}\sup_{t\in(-\infty,t_{0}]}(|\varphi(t)|+|\psi(t)|)$,
$t\geq t_{0}$, where $M$ and $\gamma$ do not depend on $t_{0}\geq 0$,
$\varphi$ and $\psi$. The fundamental function $X(t,s)$ of equation (1.7) has
an exponential estimate if it satisfies $\displaystyle|X(t,s)|\leq
M_{0}e^{-\gamma_{0}(t-s)}$ for some $t_{0}\geq 0$, $M_{0}>0$, $\gamma_{0}>0$
and $t\geq s\geq t_{0}$.

For a fixed bounded interval $J=[t_{0},t_{1}]$, consider the space
$L_{\infty}[t_{0},t_{1}]$ of all essentially bounded on $J$ functions with the
norm $\|y\|_{J}=\operatorname*{{\rm ess}\,sup}_{t\in J}|y(t)|$, also
$\|f\|_{[t_{0},\infty)}=\operatorname*{{\rm ess}\,sup}_{t\geq t_{0}}|f(t)|$,
$I$ is the identity operator. Define a linear bounded operator on the space
$L_{\infty}[t_{0},t_{1}]$ as
$\displaystyle(Sy)(t)=\left\\{\begin{array}[]{ll}a(t)y(g(t)),&g(t)\geq
t_{0},\\\ 0,&g(t)<t_{0}.\\\ \end{array}\right.$ Note that there exists a
unique solution of problem (2.1)-(2.1), see, for example, [1], and it can be
presented as

$\begin{array}[]{ll}x(t)=&\displaystyle
X(t,t_{0})x_{0}+\int_{t_{0}}^{t}X(t,s)[(I-S)^{-1}f](s)ds+\int_{t_{0}}^{t_{0}+\sigma}X(t,s)[(I-S)^{-1}(a(\cdot)\psi(g(\cdot)))](s)ds\\\
&\displaystyle-\sum_{k=1}^{m}\int_{t_{0}}^{t_{0}+\tau_{k}}X(t,s)[(I-S)^{-1}(b_{k}(\cdot)\varphi(h_{k}(\cdot)))](s)ds,\end{array}$
(2.4)

where $\psi(g(t))=0$ for $g(t)\geq t_{0}$, $\varphi(h_{k}(t))=0$ for
$h_{k}(t)\geq t_{0}$, and in $L_{\infty}[t_{0},t_{1}]$, for any $t_{1}>t_{0}$,

$\|(I-S)^{-1}\|\leq\frac{1}{1-\|a\|_{[t_{0},\infty)}}.$ (2.5)

Let us start with a uniform estimate

$|Y(t,s)|\leq K,~{}~{}t\geq s\geq t_{0}$ (2.6)

for the fundamental function $Y(t,s)$ of the equation with a non-delay term

$\dot{y}(t)-a_{0}(t)\dot{y}(g(t))=c(t)y(t)-\sum_{k=0}^{m}d_{k}(t)y(h_{k}(t)),~{}t\geq
t_{0},$ (2.7)

where $0\leq t-g(t)\leq\sigma$, $t-h_{k}(t)\leq\tau_{k}$, $k=0,\dots,m$.
Denote $\displaystyle d(t):=\sum_{k=0}^{m}d_{k}(t)$.

######  Lemma 1.

If $\|a_{0}\|_{[t_{0},\infty)}<1$, there is an $\alpha_{0}>0$ such that
$d(t)-c(t)\geq\alpha_{0}$ and

$K_{0}:=\left(\frac{\|c\|_{[t_{0},\infty)}+\sum_{k=0}^{m}\|d_{k}\|_{[t_{0},\infty)}}{1-\|a_{0}\|_{[t_{0},\infty)}}\right)\left(\left\|\frac{a_{0}}{d-c}\right\|_{[t_{0},\infty)}+\sum_{k=0}^{m}\tau_{k}\left\|\frac{d_{k}}{d-c}\right\|_{[t_{0},\infty)}\right)<1$
(2.8)

then the fundamental function $Y(t,s)$ of (2.7) satisfies (2.6) with
$K=(1-K_{0})^{-1}.$

###### Proof.

For brevity of notations, we set $y(t)=Y(t,t_{0})$. Then, $y$ satisfies (2.7),
where the initial value is $y(t_{0})=1$, with the zero initial functions. Let
$J=[t_{0},t_{1}]$, where $t_{1}>t_{0}$ is arbitrary. Equality (2.7) implies
the estimate, due to (2.5),

$\|\dot{y}\|_{J}\leq\left(\frac{\|c\|_{[t_{0},\infty)}+\sum_{k=0}^{m}\|d_{k}\|_{[t_{0},\infty)}}{1-\|a_{0}\|_{[t_{0},\infty)}}\right)\|y\|_{J}.$
(2.9)

Further, since
$d(t)y(t)-\sum_{k=0}^{m}d_{k}(t)y(h_{k}(t))=\sum_{k=0}^{m}d_{k}(t)[y(t)-y(h_{k}(t))]$,
from (2.7),

$\dot{y}(t)=-[d(t)-c(t)]y(t)+a_{0}(t)\dot{y}(g(t))+\sum_{k=0}^{m}d_{k}(t)\int_{h_{k}(t)}^{t}\dot{y}(\xi)d\xi.$

Integrating from $t_{0}$ to $t$, we get

$\displaystyle y(t)=$ $\displaystyle
e^{-\int_{t_{0}}^{t}[d(\xi)-c(\xi)]d\xi}+\int_{t_{0}}^{t}e^{-\int_{s}^{t}[d(\xi)-c(\xi)]d\xi}[d(s)-c(s)]\times$
$\displaystyle\times\left[\frac{a_{0}(s)}{d(s)-c(s)}\dot{y}(g(s))+\sum_{k=0}^{m}\frac{d_{k}(s)}{d(s)-c(s)}\int_{h_{k}(s)}^{s}\dot{y}(\xi)d\xi\right]ds.$

Therefore, by (2.9) and the definition of $K_{0}$ in (2.8), we have

$\|y\|_{J}\leq
1+\left(\left\|\frac{a_{0}}{d-c}\right\|_{[t_{0},\infty)}+\sum_{k=0}^{m}\tau_{k}\left\|\frac{d_{k}}{d-c}\right\|_{[t_{0},\infty)}\right)\|\dot{y}\|_{J}\leq
1+K_{0}\|y\|_{J}.$

Then $\|Y(t,t_{0})\|_{J}\leq(1-K_{0})^{-1}$, and the expression in the right-
hand side does not depend on $t_{1}$.

Hence $\displaystyle\|Y(t,t_{0})\|_{[t_{0},\infty)}\leq(1-K_{0})^{-1}$. Again,
the same inequality holds with $t_{0}$ replaced by any $s\geq t_{0}$. Thus
estimate (2.6) holds. ∎

##  3 Main Results

We start with an exponential estimate which later will be used to analyze
exponential stability.

######  Theorem 1.

Assume that there exist constants $\lambda>0$ and $\alpha>0$ such that

$p(t):=\sum_{k=1}^{m}e^{\lambda(t-h_{k}(t))}b_{k}(t)+\lambda
a(t)e^{\lambda(t-g(t))}-\lambda\geq\alpha,~{}t\geq
t_{0},~{}e^{\lambda\sigma}\|a\|_{[t_{0},\infty)}<1,$ (3.1)
$M_{1}:=\frac{\lambda+\sum\limits_{k=1}^{m}e^{\lambda\tau_{k}}\|b_{k}\|_{[t_{0},\infty)}+\lambda
e^{\lambda\sigma}\|a\|_{[t_{0},\infty)}}{1-e^{\lambda\sigma}\|a\|_{[t_{0},\infty)}}\left(\left\|\frac{a}{p}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!(1+\lambda\sigma)e^{\lambda\sigma}+\sum_{k=1}^{m}\left\|\frac{b_{k}}{p}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!e^{\lambda\tau_{k}}\tau_{k}\right)<1.$
(3.2)

Then for the solution of problem (2.1), the following estimate is valid

$\begin{array}[]{ll}|x(t)|\leq&\displaystyle
M_{0}e^{-\lambda(t-t_{0})}\left[|x(t_{0})|+\frac{e^{\lambda\sigma}-1}{\lambda(1-\|a\|_{[t_{0},\infty)})}\|a\|_{[t_{0},\infty)}\|\psi\|_{[t_{0}-\sigma,t_{0}]}\right.\vspace{2mm}\\\
&\displaystyle\left.+\sum_{k=1}^{m}\frac{e^{\lambda\tau_{k}}-1}{\lambda(1-\|a\|_{[t_{0},\infty)})}\|b_{k}\|_{[t_{0},\infty)}\|\varphi\|_{[t_{0}-\tau_{k},t_{0}]}\right]+\frac{M_{0}}{\lambda(1-\|a\|_{[t_{0},\infty)})}\|f\|_{[t_{0},t]},\end{array}$
(3.3)

where $M_{0}:=(1-M_{1})^{-1}$.

###### Proof.

Consider first the case $f\equiv 0$. After the substitution
$x(t)=e^{-\lambda(t-t_{0})}z(t)$ into (2.1), we get

$\dot{z}(t)-a(t)e^{\lambda(t-g(t))}\dot{z}(g(t))=\lambda z(t)-\lambda
e^{\lambda(t-g(t))}a(t)z(g(t))-\sum_{k=1}^{m}e^{\lambda(t-h_{k}(t))}b_{k}(t)z(h_{k}(t)).$
(3.4)

Equation (3.4) has the form of (2.7) with

$a_{0}(t)=a(t)e^{\lambda(t-g(t))},~{}c(t)=\lambda,~{}d_{0}(t)=\lambda
a(t)e^{\lambda(t-g(t))},~{}h_{0}(t)=g(t),~{}d_{k}(t)=e^{\lambda(t-h_{k}(t))}b_{k}(t),$

$k=1,\dots,m$. Again, $\displaystyle d(t)=\sum_{k=0}^{m}d_{k}(t)$, in (3.1) we
have $p=d-c$.

Then

$\left\|\frac{a_{0}}{d-c}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\leq\left\|\frac{a}{p}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!e^{\lambda\sigma},\left\|\frac{d_{0}}{d-c}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\leq\left\|\frac{a}{p}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\lambda
e^{\lambda\sigma},\left\|\frac{d_{k}}{d-c}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!\leq\left\|\frac{b_{k}}{p}\right\|_{[t_{0},\infty)}\\!\\!\\!\\!e^{\lambda\tau_{k}},k=1,\dots,m,$
$\frac{\|c\|_{[t_{0},\infty)}+\sum_{k=0}^{m}\|d_{k}\|_{[t_{0},\infty)}}{1-\|a_{0}\|_{[t_{0},\infty)}}\leq\frac{\lambda+\sum_{k=1}^{m}e^{\lambda\tau_{k}}\|b_{k}\|_{[t_{0},\infty)}+\lambda
e^{\lambda\sigma}\|a\|_{[t_{0},\infty)}}{1-e^{\lambda\sigma}\|a\|_{[t_{0},\infty)}}.$

Let $Z(t,s)$ be the fundamental function of (3.4). Inequalities (3.1) and
(3.2) imply (2.8). By Lemma 1, $|Z(t,s)|\leq M_{0}$. If $X(t,s)$ is a
fundamental function of (2.1) then for $X(t,s)$ the exponential equality
$X(t,s)=e^{-\lambda(t-s)}Z(t,s)$ is valid. Hence $|X(t,s)|\leq
M_{0}e^{-\lambda(t-s)}$.

By (2.4), the solution $x$ of (2.1)-(2.2) satisfies

$\displaystyle|x(t)|\leq$
$\displaystyle|X(t,t_{0})|\,|x_{0}|+\int_{t_{0}}^{t_{0}+\sigma}|X(t,s)|\,\|(I-S)^{-1}\|\,|a(s)|\,|\psi(g(s))|ds$
$\displaystyle+\sum_{k=1}^{m}\int_{t_{0}}^{t_{0}+\tau_{k}}|X(t,s)|\,\|(I-S)^{-1}\|\,|b_{k}(s)|\,|\varphi(h_{k}(s))|\,ds$
$\displaystyle\leq$ $\displaystyle
M_{0}e^{-\lambda(t-t_{0})}|x(t_{0})|+\frac{M_{0}}{\lambda\left(1-\|a\|_{[t_{0},\infty)}\right)}\|a\|_{[t_{0},\infty)}\left(e^{-\lambda(t-t_{0}-\sigma)}-e^{-\lambda(t-t_{0})}\right)\|\psi\|_{[t_{0}-\sigma,t_{0}]}$
$\displaystyle+\frac{M_{0}}{\lambda\left(1-\|a\|_{[t_{0},\infty)}\right)}\sum_{k=1}^{m}\|b_{k}\|_{[t_{0},\infty)}\left(e^{-\lambda(t-t_{0}-\tau_{k})}-e^{-\lambda(t-t_{0})}\right)\|\varphi\|_{[t_{0}-\tau_{k},t_{0}]},$

which implies (3.3) with $f\equiv 0$.

For the general case we apply (2.4), the estimate for $X(t,s)$ and the
inequalities

$\left|\int_{t_{0}}^{t}X(t,s)(I-S)^{-1}(f(s))\,ds\right|\leq\frac{M_{0}}{\lambda\left(1-\|a\|_{[t_{0},\infty)}\right)}\|f\|_{[t_{0},t]}.$

∎

From continuity of $p$ in $\lambda$, where $p$ is defined in (3.1), Theorem 1
immediately implies the following exponential stability test.

######  Theorem 2.

Assume that for some $\alpha>0$, $\displaystyle
b(t):=\sum_{k=1}^{m}b_{k}(t)\geq\alpha,~{}\|a\|_{[t_{0},\infty)}<1$ and

$\left(\frac{\sum_{k=1}^{m}\|b_{k}\|_{[t_{0},\infty)}}{1-\|a\|_{[t_{0},\infty)}}\right)\left(\left\|\frac{a}{b}\right\|_{[t_{0},\infty)}+\sum_{k=1}^{m}\tau_{k}\left\|\frac{b_{k}}{b}\right\|_{[t_{0},\infty)}\right)<1.$

Then equation (1.7) is uniformly exponentially stable.

######  Corollary 1.

Assume that $\displaystyle b_{k}>0$ are constants,
$\displaystyle\|a\|_{[t_{0},\infty)}<\frac{1}{2}$ and
$\displaystyle\sum_{k=1}^{m}b_{k}\tau_{k}<1-2\|a\|_{[t_{0},\infty)}.$ Then
equation (1.7) is uniformly exponentially stable.

Consider (1.1) which is equation (1.7) for $m=1$.

######  Corollary 2.

Assume that for some $\alpha>0$, $b(t)\geq\alpha$ for any $t\geq t_{0}$,
$\|a\|_{[t_{0},\infty)}<1$, and

$\left(\left\|\frac{a}{b}\right\|_{[t_{0},\infty)}+\tau\right)\|b\|_{[t_{0},\infty)}<1-\|a\|_{[t_{0},\infty)}.$

Then equation (1.1) is uniformly exponentially stable.

##  4 Examples and Discussion

To compare known exponential stability results obtained in [3, 4] and the test
obtained in Theorem 2, consider equation (1.1).

Let us illustrate our results with an example where Theorem 2 implies
exponential stability, while Propositions 1 and 2 fail. If anyone of
assumptions (1.4)-(1.6) holds, conditions of Corollary 2 also hold. However it
is possible (see Example 1) that all conditions of Proposition 1 fail, but
conditions of Corollary 2 hold. The paper [3] also studies equation (1.7) with
several delay terms. Exponential stability tests in [3] depend on the greatest
delay $\tau=\max\\{\tau_{k}\\}$ with the assumption that $b_{k}(t)\geq 0$.
Theorem 1 depends on all delays $\tau_{k}$ without any assumption on the sign
of $b_{k}(t)$.

Conditions in Proposition 2, unlike in Corollary 2, depend on the delay of a
neutral term. So for small $\sigma$ Proposition 2 is better than Corollary 2,
but for large $\sigma$, Corollary 2 is better (see again Example 1 below).

######  Example 1.

Consider the equation

$\dot{x}(t)-0.15\dot{x}(t-\sigma)=-x\left(t-\frac{1}{e}-0.1\sin{t}\right),~{}~{}\sigma>0.$
(4.1)

Here $t-h(t)=\frac{1}{e}+0.1\sin t\leq\tau:=\frac{1}{e}+0.1>\frac{1}{e}$.
Conditions (1.4)-(1.6) of Proposition 1 fail, but conditions of Corollary 1
($m=1$) hold. Hence by Corollary 1 equation (4.1) is exponentially stable.
Assumptions of Proposition 2 are satisfied for equation (4.1) if
$\sigma<2.165$. Thus, for $\sigma>2.2$, say, $\sigma=3$, the results of
Corollary 2 lead to exponential stability of (4.1) while Propositions 1 and 2
cannot establish this result.

We omit here comparison with other known stability results since [3] contains
this part. Most of these results are for autonomous equations, or equations
with a non-delay term, see for example [8, 9, 11]. As we mentioned earlier, we
are not aware of exponential estimates for solutions for neutral differential
equations. Exponential estimates for solutions of delay differential equations
without a neutral term can be found in recent paper [5]. The present paper
partially generalizes the results obtained in [5] to the neutral case.

Let us illustrate exponential estimates for a solution of equation (4.1) with
either constant or variable $\sigma$.

######  Example 2.

Consider the initial value problem

$\begin{array}[]{l}\displaystyle\dot{x}(t)-0.15\dot{x}(t-0.5)=-x\left(t-\frac{1}{e}-0.1\sin{t}\right),~{}t\geq
0,\\\ x(t)=\cos t,\dot{x}(t)=\sin{2t}+2,~{}t<0,x(0)=1.\end{array}$ (4.2)

We apply Theorem 1, where $m=1$, $a(t)=0.15$, $b(t)=1$, $\sigma=0.5$,
$\displaystyle\frac{1}{e}-0.1\approx 0.2679\leq
t-h(t)\leq\tau=\frac{1}{e}+0.1\approx 0.4679$. Thus inequality (3.1) holds for
$\lambda=0.1$ with $M_{1}\approx 0.96,M_{0}\leq 25.61$. Hence for the
fundamental function of the equation in (4.2) we have an estimate
$|X(t,s)|\leq 25.61e^{-0.1(t-s)}$, $t\geq s\geq 0$. Next, we have for the
initial conditions $\psi(t)=\sin{2t}+2$, $\varphi(t)=\cos t$, $\|\psi\|=3$,
$\|\varphi\|=1$. By (3.3), for the solution of problem (4.2) we have the
following estimate

$|x(t)|\leq 42.4e^{-0.1t},$ (4.3)

see the comparison to the numerical solution in Fig. 1, left. Next, consider
variable $\sigma$

$\begin{array}[]{l}\displaystyle\dot{x}(t)-0.15\dot{x}(t-2.7-0.3\cos
t)=-x\left(t-\frac{1}{e}-0.1\sin{t}\right),~{}t\geq 0,\\\ x(t)=\cos
t,\dot{x}(t)=\sin{2t}+2,~{}t<0,x(0)=1.\end{array}$ (4.4)

All the parameters are as above, only instead of $\sigma=0.5$ we have $2.4\leq
t-\sigma(t)\leq 3$. Inequality (3.1) holds for $\lambda=0.06$ with $M_{0}\leq
25.5$. In this case,

$|x(t)|\leq 54.5e^{-0.06t},$ (4.5)

see the comparison to the numerical solution in Fig. 1, right.

Figure 1: The absolute value of the numerical solution of (4.2) compared to
estimate (4.3) (left) and the numerical solution of (4.4) compared to estimate
(4.5) (right), with the logarithmic scale in $x$.

Finally, let us state possible directions for further research extending the
results of the present paper.

  1. 1.

Obtain explicit estimates of solutions for nonlinear neutral differential
equations.

  2. 2.

Extend the estimates of solutions to a vector DDE, or to higher order neutral
equations, for recent results on third order neutral equations see [7].
Consider other types of neutral equations, such as equations with a
distributed delay, and stochastic differential equations.

  3. 3.

In this paper, we presented pointwise estimates. It would be interesting to
obtain estimates in an integral form.

  4. 4.

In Corollary 1 the right-hand side is equal to $1-2\|a\|$ implying
$\|a\|<\frac{1}{2}$. Can we extend the results to $\|a\|\in(0.5,1)$?

  5. 5.

Derive exponential estimates dependent on the neutral delay
$\sigma=\operatorname*{{\rm ess}\,sup}_{t\geq t_{0}}(t-g(t))$ for problem
(2.1)-(2.2).

## Acknowledgment

The second author acknowledges the support of NSERC, the grant
RGPIN-2020-03934.

## References

  * [1] N. V. Azbelev and P. M. Simonov, Stability of Differential Equations with Aftereffect. Stability and Control: Theory, Methods and Applications, 20. Taylor $\&$ Francis, London, 2003\.
  * [2] A. S. Balandin and V. V. Malygina, On the exponential stability of linear differential-difference equations of neutral type (Russian), Izv. Vyssh. Uchebn. Zaved. Mat. 2007, no. 7, 17–27; translation in Russian Math. (Iz. VUZ) 51 (2007), 15–24.
  * [3] L. Berezansky and E. Braverman, On stability of linear neutral differential equations with variable delays, Czechoslovak Math. J. 69(144) (2019), 863–891.
  * [4] L. Berezansky and E. Braverman, A new stability test for linear neutral differential equations, Appl. Math. Lett. 81 (2018), 79–85.
  * [5] L. Berezansky and E. Braverman, Solution estimates for linear differential equations with delay, Appl. Math. Comput. 372 (2020), 124962, 10 pp.
  * [6] A. Domoshnitsky, M. Gitman and R. Shklyar, Stability and estimate of solution to uncertain neutral delay systems, Bound. Value Probl. 2014, 2014:55, 14 pp.
  * [7] A. Domoshnitsky, I. Volinsky, S. Levi and S. Shemesh, Stability of third order neutral delay differential equations, AIP Conference Proceedings 2159, 020002 (2019); https://doi.org/10.1063/1.5127464.
  * [8] K. Gopalsamy, A simple stability criterion for linear neutral differential systems, Funkcialaj Ekvacioj 28 (1985), 33–38.
  * [9] K. Gopalsamy, Stability and Oscillations in Delay Differential Equations of Population Dynamics. Mathematics and its Applications, 74, Kluwer Academic Publishers Group, Dordrecht, 1992.
  * [10] V. L. Kharitonov and D. Hinrichsen, Exponential estimates for time delay systems, Systems Control Lett. 53 (2004), 395–405.
  * [11] V. Kolmanovskii and A. Myshkis, Introduction to the Theory and Applications of Functional-Differential Equations, _M_ athematics and its Applications, 463, Kluwer Academic Publishers, Dordrecht, 1999\.
  * [12] I.-G. E. Kordonis, Ch. G. Philos and I. K. Purnaras, On the behavior of solutions of linear neutral integrodifferential equations with unbounded delay, Georgian Math. J. 11 (2004), 337–348.
  * [13] I. I. Matveeva, On the exponential stability of the solutions of neutral type linear periodic systems with variable delay (Russian), Sib. Elektron. Mat. Izv. 16 (2019), 748–756.