* [27] C. B. Thorn, _Derivational model from field theory II_ , Phys. Rev. D17(1978)1037, and references therein.
  * [28] G. Veneziano, _Construction of a crossing-symmetric, Regge behaved amplitude for linearly rising trajectories_ , Nuovo Cimento A57 190(1968).
  * [29] For comprehensive reviews of string theory see
e.g. J. Polchinski, _String Theory, vol.I-II_ , Cambridge University Press
1998.

  * [30] C. B. Thorn, _Derivational model from field theory II_ , Phys. Rev. D17(1978)1037, and references therein.
  * [31] H. Bethe, E. Salpeter (1951). ”A Relativistic Equation for Bound-State Problems”. Physical Review 84 (6): 1232. Bibcode:1951PhRv…84.1232S. doi:10.1103/PhysRev.84.1232.
  * [32] S. Fubini and G. Furlan, Physicsi, 229. S. Weinberg, Phys. Rev. 150, 1313. L. Susskind, Phys. Rev. 165, 1535. L. Susskind, “Infinite Momentum Frames and Particle Dynamics”, in Lectures in Theoretical Physics, Vol. XI-D: Mathematical Methods in Theoretical Physics, K. Mahanthappa and W. Brittin, ed.; Gordon and Breach, New York.


# On $\lambda-$ Pseudo bi-starlike functions related with Fibonacci numbers

Kaliyappan Vijaya  Gangadharan Murugusundaramoorthy and Hatun Özlem Güney
K.Vijaya– School of Advanced Sciences, Vellore Institute of Techlogy,
Vellore-632014, India<EMAIL_ADDRESS>G.Murugusundaramoorthy– School of
Advanced Sciences, Vellore Institute of Technology, Vellore-632014, India.
<EMAIL_ADDRESS>H.Özlem Güney– Dicle University, Faculty of Science,
Department of Mathematics, Diyarbakır-TÜRKİYE<EMAIL_ADDRESS>
###### Abstract

In this paper we define a new subclass $\lambda$-bi-pseudo-starlike functions
of $\Sigma$ related to shell-like curves connected with Fibonacci numbers and
determine the initial Taylor-Maclaurin coefficients $|a_{2}|$ and $|a_{3}|$
for $f\in\mathcal{PSL}_{\Sigma}^{\lambda}(\tilde{p}(z)).$ Further we determine
the Fekete-Szegö result for the function class
$\mathcal{PSL}_{\Sigma}^{\lambda}(\tilde{p}(z))$ and for special cases,
corollaries are stated which some of them are new and have not been studied so
far.

###### keywords:

Analytic functions, bi-univalent, shell-like curve, Fibonacci numbers,
starlike functions.

AMS classification 30C45 (see
https://mathscinet.ams.org/mathscinet/msc/msc2020.html)

##  1 Introduction

Let $\mathcal{A}$ denote the class of functions $f$ which are analytic in the
open unit disk
$\mathbb{U}=\left\\{z\,:\,\,z\in\mathbb{C}\,\text{and}\,|z|<1\right\\}.$ Also
let $\mathcal{S}$ denote the class of functions in $\mathcal{A}$ which are
univalent in $\mathbb{U}$ and normalized by the conditions
$f(0)=f^{\prime}(0)-1=0$ and are of the form:

$f\left(z\right)=z+\sum_{n=2}^{\infty}a_{n}z^{n}.$ (1)

The Koebe one quarter theorem [4] ensures that the image of $\mathbb{U}$ under
every univalent function $f\in\mathcal{A}$ contains a disk of radius
$\frac{1}{4}.$ Thus every univalent function $f$ has an inverse $f^{-1}$
satisfying

$f^{-1}(f(z))=z,\ (z\in\mathbb{U})~{}{\rm and}~{}~{}f(f^{-1}(w))=w\
(|w|<r_{0}(f),\ r_{0}(f)\geq\frac{1}{4}).$

A function $f\in\mathcal{A}$ is said to be bi-univalent in $\mathbb{U}$ if
both $f$ and $f^{-1}$ are univalent in $\mathbb{U}.$ Let $\Sigma$ denote the
class of bi-univalent functions defined in the unit disk $\mathbb{U}.$ Since
$f\in\Sigma$ has the Maclaurian series given by (1), a computation shows that
its inverse $g=f^{-1}$ has the expansion

$g(w)=f^{-1}(w)=w-a_{2}w^{2}+(2a_{2}^{2}-a_{3})w^{3}+\cdots.$ (2)

One can see a short history and examples of functions in the class $\Sigma$ in
[14]. Several authors have introduced and investigated subclasses of bi-
univalent functions and obtained bounds for the initial coefficients (see
[2],[3],[10],[14],[15],[16]). For a brief historical account and for several
interesting examples of functions in the class $\Sigma;$ see the pioneering
work on this subject by Srivastava et al. [14], which actually revived the
study of bi-univalent functions in recent years. From the work of Srivastava
et al. [14], we choose to recall the following examples of functions in the
class $\Sigma:$

$\frac{z}{1-z},\quad-\log(1-z)\qquad\text{and}\qquad\frac{1}{2}\log\left(\frac{1+z}{1-z}\right).$

We notice that the class $\Sigma$ is not empty. However, the Koebe function is
not a member of $\Sigma.$ An analytic function $f$ is subordinate to an
analytic function $F$ in $\mathbb{U}$, written as $f\prec
F~{}(z\in\mathbb{U}),$ provided there is an analytic function $\omega$ defined
on $\mathbb{U}$ with $\omega(0)=0$ and $\left|\omega(z)\right|<1$ satisfying
$f(z)=F(\omega(z)).$ It follows from Schwarz Lemma that

$f(z)\prec F(z)\quad\Longleftrightarrow f(0)=F(0)~{}{\rm
and}~{}~{}f(\mathbb{U})\subset F(\mathbb{U})~{},z\in\mathbb{U}$

(for details see [4], [9]). We recall important subclasses of $\mathcal{S}$ in
geometric function theory such that if $f\in\mathcal{A}$ and

$\frac{zf^{\prime}(z)}{f(z)}\prec p(z)\quad{\rm and}\quad
1+\frac{zf^{\prime\prime}(z)}{f^{\prime}(z)}\prec p(z)$

where $p(z)=\frac{1+z}{1-z}$, then we say that $f$ is starlike and convex,
respectively. These functions form known classes denoted by $\mathcal{S^{*}}$
and $\mathcal{C}$, respectively. Recently, in [13], Sokół introduced the class
$\mathcal{SL}$ of shell-like functions as the set of functions
$f\in\mathcal{A}$ which is described in the following definition:

######  Definition 1.1.

The function $f\in\mathcal{A}$ belongs to the class $\mathcal{SL}$ if it
satisfies the condition that

$\frac{zf^{\prime}(z)}{f(z)}\prec\tilde{p}(z)$

with

$\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau z-\tau^{2}z^{2}},$

where $\tau=(1-\sqrt{5})/2\approx-0.618$.

It should be observed $\mathcal{SL}$ is a subclass of the starlike functions
$\mathcal{S^{*}}.$

The function $\tilde{p}$ is not univalent in $\mathbb{U}$, but it is univalent
in the disc $|z|<(3-\sqrt{5})/2\approx 0.38$. For example,
$\tilde{p}(0)=\tilde{p}(-1/2\tau)=1$ and $\tilde{p}(e^{\mp
i\arccos(1/4)})=\sqrt{5}/5$, and it may also be noticed that

$\frac{1}{|\tau|}=\frac{|\tau|}{1-|\tau|},$

which shows that the number $|\tau|$ divides $[0,1]$ such that it fulfils the
golden section. The image of the unit circle $|z|=1$ under $\tilde{p}$ is a
curve described by the equation given by

$(10x-\sqrt{5})y^{2}=(\sqrt{5}-2x)(\sqrt{5}x-1)^{2},$

which is translated and revolved trisectrix of Maclaurin. The curve
$\tilde{p}(re^{it})$ is a closed curve without any loops for $0<r\leq
r_{0}=(3-\sqrt{5})/2\approx 0.38$. For $r_{0}<r<1$, it has a loop, and for
$r=1$, it has a vertical asymptote. Since $\tau$ satisfies the equation
$\tau^{2}=1+\tau,$ this expression can be used to obtain higher powers
$\tau^{n}$ as a linear function of lower powers, which in turn can be
decomposed all the way down to a linear combination of $\tau$ and $1$. The
resulting recurrence relationships yield Fibonacci numbers $u_{n}$:

$\tau^{n}=u_{n}\tau+u_{n-1}.$

In [12] Raina and Sokół showed that

$\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau
z-\tau^{2}z^{2}}=\left(t+\frac{1}{t}\right)\frac{t}{1-t-t^{2}}$
$=\frac{1}{\sqrt{5}}\left(t+\frac{1}{t}\right)\left(\frac{1}{1-(1-\tau)t}-\frac{1}{1-\tau
t}\right)$
$=\left(t+\frac{1}{t}\right)\sum_{n=1}^{\infty}u_{n}t^{n}=1+\sum_{n=1}^{\infty}(u_{n-1}+u_{n+1})\tau^{n}z^{n},$
(3)

where

$u_{n}=\frac{(1-\tau)^{n}-\tau^{n}}{\sqrt{5}},\tau=\frac{1-\sqrt{5}}{2}\,\,\,(n=1,2,\ldots).$
(4)

This shows that the relevant connection of $\tilde{p}$ with the sequence of
Fibonacci numbers ${u_{n}}$, such that
$u_{0}=0,\,u_{1}=1,\,u_{n+2}=u_{n}+u_{n+1}$ for $n=0,1,2,\cdots$. And they got

$\displaystyle\tilde{p}(z)$ $\displaystyle=$ $\displaystyle
1+\sum_{n=1}^{\infty}\tilde{p}_{n}z^{n}$ (5) $\displaystyle=$ $\displaystyle
1+(u_{0}+u_{2})\tau
z+(u_{1}+u_{3})\tau^{2}z^{2}+\sum_{n=3}^{\infty}(u_{n-3}+u_{n-2}+u_{n-1}+u_{n})\tau^{n}z^{n}$
$\displaystyle=$ $\displaystyle 1+\tau
z+3\tau^{2}z^{2}+4\tau^{3}z^{3}+7\tau^{4}z^{4}+11\tau^{5}z^{5}+\cdots.$

Let $\mathcal{P}(\beta)$, $0\leq\beta<1$, denote the class of analytic
functions $p$ in $\mathbb{U}$ with $p(0)=1$ and $Re\\{p(z)\\}>\beta$.
Especially, we will use $\mathcal{P}$ instead of $\mathcal{P}(0).$

######  Theorem 1.2.

([6]) The function $\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau
z-\tau^{2}z^{2}}$ belongs to the class $\mathcal{P}(\beta)$ with
$\beta=\sqrt{5}/10\approx 0.2236$.

Now we recall the following lemma which will be relevant for our study .

######  Lemma 1.3.

([11]) Let $p\in\mathcal{P}$ with $p(z)=1+c_{1}z+c_{2}z^{2}+\cdots$, then

$|c_{n}|\leq 2\qquad for\qquad n\geq 1.$ (6)

In this present work, we introduce a new subclass of ${\Sigma}$ associated
with shell-like functions connected with Fibonacci numbers and obtain the
initial Taylor coefficients $|a_{2}|$ and $|a_{3}|$ for this function class.
Also, we give bounds for the Fekete-Szegö functional $|a_{3}-\mu a_{2}^{2}|$
for this class.

##  2 Bi-Univalent function class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$

In this section, we introduce a new subclass of ${\Sigma}$ associated with
shell-like functions connected with Fibonacci numbers and obtain the initial
Taylor coefficients $|a_{2}|$ and $|a_{3}|$ for the function class by
subordination. Firstly, let $p(z)=1+p_{1}z+p_{2}z^{2}+\cdots$, and
$p\prec\tilde{p}.$ Then there exists an analytic function $u$ such that
$|u(z)|<1$ in $\mathbb{U}$ and $p(z)=\tilde{p}(u(z))$. Therefore, the function

$h(z)=\frac{1+u(z)}{1-u(z)}=1+c_{1}z+c_{2}z^{2}+\ldots$ (1)

is in the class $\mathcal{P}(0)$. It follows that

$u(z)=\frac{c_{1}z}{2}+\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\frac{z^{2}}{2}+\left(c_{3}-c_{1}c_{2}+\frac{c_{1}^{3}}{4}\right)\frac{z^{3}}{2}+\cdots$
(2)

and

$\displaystyle\tilde{p}(u(z))$ $\displaystyle=$ $\displaystyle
1+\frac{\tilde{p}_{1}c_{1}z}{2}+\left\\{\frac{1}{2}\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\tilde{p}_{1}+\frac{c_{1}^{2}}{4}\tilde{p}_{2}\right\\}z^{2}$
(3)
$\displaystyle+\left\\{\frac{1}{2}\left(c_{3}-c_{1}c_{2}+\frac{c_{1}^{3}}{4}\right)\tilde{p}_{1}+\frac{1}{2}c_{1}\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\tilde{p}_{2}+\frac{c_{1}^{3}}{8}\tilde{p}_{3}\right\\}z^{3}+\cdots.$

And similarly, there exists an analytic function $v$ such that $|v(w)|<1$ in
$\mathbb{U}$ and $p(w)=\tilde{p}(v(w))$. Therefore, the function

$k(w)=\frac{1+v(w)}{1-v(w)}=1+d_{1}w+d_{2}w^{2}+\ldots$ (4)

is in the class $\mathcal{P}(0)$. It follows that

$v(w)=\frac{d_{1}w}{2}+\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\frac{w^{2}}{2}+\left(d_{3}-d_{1}d_{2}+\frac{d_{1}^{3}}{4}\right)\frac{w^{3}}{2}+\cdots$
(5)

and

$\displaystyle\tilde{p}(v(w))$ $\displaystyle=$ $\displaystyle
1+\frac{\tilde{p}_{1}d_{1}w}{2}+\left\\{\frac{1}{2}\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\tilde{p}_{1}+\frac{d_{1}^{2}}{4}\tilde{p}_{2}\right\\}w^{2}$
(6)
$\displaystyle+\left\\{\frac{1}{2}\left(d_{3}-d_{1}d_{2}+\frac{d_{1}^{3}}{4}\right)\tilde{p}_{1}+\frac{1}{2}d_{1}\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\tilde{p}_{2}+\frac{d_{1}^{3}}{8}\tilde{p}_{3}\right\\}w^{3}+\cdots.$

The class $\mathcal{L}_{\lambda}(\alpha)$ of $\lambda$-pseudo-starlike
functions of order $\alpha$ was introduced and investigated by Babalola [1]. A
function $f\in\mathcal{A}$ is in the class $\mathcal{L}_{\lambda}(\alpha)$ if
it satisfies

$\Re\left(\frac{z(f^{\prime}(z))^{\lambda}}{f(z)}\right)>\alpha,\quad(0\leq\alpha<1)$

where $\lambda\geq 1,\lambda\in\mathbb{R}$ and $z\in\mathbb{U}$. In [1] it was
showed that all pseudo-starlike functions are Bazilevič functions of type
$\left(1-1/\lambda\right)$ and of order $\alpha^{1/\lambda}$ and univalent in
open unit disk $\mathbb{U}$. Recently Joshi et al. [5] defined the bi-pseudo-
starlike functions class and obtained the bounds for the initial coefficients
$|a_{2}|$ and $|a_{3}|$. In this paper we define a new class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z)),$$\lambda$-bi-pseudo-starlike
functions of $\Sigma$ related to shell-like curves connected with Fibonacci
numbers and determine the bounds for the initial Taylor-Maclaurin coefficients
of $|a_{2}|$ and $|a_{3}|.$ Further we consider the Fekete-Szegö problem in
this class and the special cases are stated as corollaries which are new and
have not been studied so far.

######  Definition 2.1.

For $\lambda\geq 1$ and $\lambda\in\mathbb{R},$ a function $f\in\Sigma$ of the
form (1) is said to be in the class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$ if the following
subordination hold:

$\frac{z(f^{\prime}(z))^{\lambda}}{f(z)}\prec\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau
z-\tau^{2}z^{2}}$ (7)

and

$\frac{w(g^{\prime}(w))^{\lambda}}{g(w)}\prec\tilde{p}(w)=\frac{1+\tau^{2}w^{2}}{1-\tau
w-\tau^{2}w^{2}}$ (8)

where $\tau=(1-\sqrt{5})/2\approx-0.618$ where $z,w\in\mathbb{U}$ and $g$ is
given by (2).

Specializing the parameter $\lambda=1$ and $\lambda=2,$ we have the following
remarks, respectively:

######  Remark 2.2.

[8] For $\lambda=1$ a function $f\in\Sigma$ is in the class
$\mathcal{PSL}^{1}_{\Sigma}(\tilde{p}(z))\equiv\mathcal{SL}_{\Sigma}(\tilde{p}(z))$
if the following conditions are satisfied:

$\frac{zf^{\prime}(z)}{f(z)}\prec\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau
z-\tau^{2}z^{2}}$ (9)

and

$\frac{wg^{\prime}(w)}{g(w)}\prec\tilde{p}(w)=\frac{1+\tau^{2}w^{2}}{1-\tau
w-\tau^{2}w^{2}}$ (10)

where $\tau=(1-\sqrt{5})/2\approx-0.618$ where $z,w\in\mathbb{U}$ and $g$ is
given by (2).

######  Remark 2.3.

For $\lambda=2$ a function $f\in\Sigma$ is in the class
$\mathcal{PSL}^{2}_{\Sigma}(\tilde{p}(z))\equiv\mathcal{GSL}_{\Sigma}(\tilde{p}(z))$
if the following conditions are satisfied:

$\left(f^{\prime}(z)\frac{zf^{\prime}(z)}{f(z)}\right)\prec\tilde{p}(z)=\frac{1+\tau^{2}z^{2}}{1-\tau
z-\tau^{2}z^{2}}$ (11)

and

$\left(g^{\prime}(w)\frac{wg^{\prime}(w)}{g(w)}\right)\prec\tilde{p}(w)=\frac{1+\tau^{2}w^{2}}{1-\tau
w-\tau^{2}w^{2}}$ (12)

where $\tau=(1-\sqrt{5})/2\approx-0.618$ where $z,w\in\mathbb{U}$ and $g$ is
given by (2).

In the following theorem we determine the initial Taylor coefficients
$|a_{2}|$ and $|a_{3}|$ for the function class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z)).$ Later we will reduce these
bounds to other classes for special cases.

######  Theorem 2.4.

Let $f$ given by (1) be in the class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z)),$ then

$|a_{2}|\leq\frac{|\tau|}{\sqrt{(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau}}$
(13)

and

$|a_{3}|\leq\frac{|\tau|\left[(2\lambda-1)^{2}-2(5\lambda^{2}-4\lambda+1)\tau\right]}{(3\lambda-1)\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]},$
(14)

where $\lambda\geq 1.$

######  Proof 2.5.

Let $f\in\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$ and $g=f^{-1}.$
Considering (7) and (8), we have

$\frac{z(f^{\prime}(z))^{\lambda}}{f(z)}=\tilde{p}(u(z))$ (15)

and

$\frac{w(g^{\prime}(w))^{\lambda}}{g(w)}=\tilde{p}(v(w)),$ (16)

where $\tau=(1-\sqrt{5})/2\approx-0.618$ where $z,w\in\mathbb{U}$ and $g$ is
given by (2). Since

$\frac{z(f^{\prime}(z))^{\lambda}}{f(z)}=1+(2\lambda-1)a_{2}z+[(3\lambda-1)a_{3}+\left(2\lambda^{2}-4\lambda+1\right)a_{2}^{2}]z^{2}+\dots$

and

$\frac{w(g^{\prime}(w))^{\lambda}}{g(w)}=1-(2\lambda-1)a_{2}w+[\left(2\lambda^{2}+2\lambda-1\right)a_{2}^{2}-(3\lambda-1)a_{3}]w^{2}+\dots.$

Thus we have

$\displaystyle
1+(2\lambda-1)a_{2}z+[(3\lambda-1)a_{3}+\left(2\lambda^{2}-4\lambda+1\right)a_{2}^{2}]z^{2}+\dots$
$\displaystyle=$ $\displaystyle
1+\frac{\tilde{p}_{1}c_{1}}{2}z+\left[\frac{1}{2}\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\tilde{p}_{1}+\frac{c_{1}^{2}}{4}\tilde{p}_{2}\right]z^{2}$
$\displaystyle+\left[\frac{1}{2}\left(c_{3}-c_{1}c_{2}+\frac{c_{1}^{3}}{4}\right)\tilde{p}_{1}+\frac{1}{2}c_{1}\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\tilde{p}_{2}+\frac{c_{1}^{3}}{8}\tilde{p}_{3}\right]z^{3}+\cdots.$
(18)

and

$\displaystyle
1-(2\lambda-1)a_{2}w+[\left(2\lambda^{2}+2\lambda-1\right)a_{2}^{2}-(3\lambda-1)a_{3}]w^{2}+\dots,$
(19) $\displaystyle=$ $\displaystyle
1+\frac{\tilde{p}_{1}d_{1}w}{2}+\left[\frac{1}{2}\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\tilde{p}_{1}+\frac{d_{1}^{2}}{4}\tilde{p}_{2}\right]w^{2}$
$\displaystyle+\left[\frac{1}{2}\left(d_{3}-d_{1}d_{2}+\frac{d_{1}^{3}}{4}\right)\tilde{p}_{1}+\frac{1}{2}d_{1}\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\tilde{p}_{2}+\frac{d_{1}^{3}}{8}\tilde{p}_{3}\right]w^{3}+\cdots.$

It follows from (2.5) and (19) that

$(2\lambda-1)a_{2}=\frac{c_{1}\tau}{2},$ (20)
$(3\lambda-1)a_{3}+\left(2\lambda^{2}-4\lambda+1\right)a_{2}^{2}=\frac{1}{2}\left(c_{2}-\frac{c_{1}^{2}}{2}\right)\tau+\frac{c_{1}^{2}}{4}3\tau^{2},$
(21)

and

$-(2\lambda-1)a_{2}=\frac{d_{1}\tau}{2},$ (22)
$\left(2\lambda^{2}+2\lambda-1\right)a_{2}^{2}-(3\lambda-1)a_{3}=\frac{1}{2}\left(d_{2}-\frac{d_{1}^{2}}{2}\right)\tau+\frac{d_{1}^{2}}{4}3\tau^{2}.$
(23)

From (20) and (22), we have

$c_{1}=-d_{1},$ (24)

and

$a_{2}^{2}=\frac{(c_{1}^{2}+d_{1}^{2})}{8(2\lambda-1)^{2}}\tau^{2}.$ (25)

Hence

$|a_{2}|\leq\frac{|\tau|}{2\lambda-1}.$ (26)

Now, by summing (21) and (23), we obtain

$2(2\lambda-1)^{2}a_{2}^{2}=\frac{1}{2}(c_{2}+d_{2})\tau-\frac{1}{4}(c_{1}^{2}+d_{1}^{2})\tau+\frac{3}{4}(c_{1}^{2}+d_{1}^{2})\tau^{2}.$
(27)

Substituting (25) in (27), we have

$2\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]a_{2}^{2}=\frac{1}{2}(c_{2}+d_{2})\tau^{2}.$
(28)

Therefore, using Lemma (1.3) we obtain

$|a_{2}|\leq\frac{|\tau|}{\sqrt{(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau}}.$
(29)

Now, so as to find the bound on $|a_{3}|,$ let’s subtract from (21) and (23).
So, we find

$2(3\lambda-1)a_{3}-2(3\lambda-1)a_{2}^{2}=\frac{1}{2}\left(c_{2}-d_{2}\right)\tau.$
(30)

Hence, we get

$2(3\lambda-1)|a_{3}|\leq 2|\tau|+2(3\lambda-1)|a_{2}|^{2}.$ (31)

Then, in view of (29), we obtain

$|a_{3}|\leq\frac{|\tau|\left[(2\lambda-1)^{2}-2(5\lambda^{2}-4\lambda+1)\tau\right]}{(3\lambda-1)\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}.$

By taking the parameter $\lambda=1$ and $\lambda=2$ in the above theorem, we
have the following the initial Taylor coefficients $|a_{2}|$ and $|a_{3}|$ for
the function classes $\mathcal{SL}_{\Sigma}(\tilde{p}(z))$ and
$\mathcal{GSL}_{\Sigma}(\tilde{p}(z)),$ respectively.

######  Corollary 2.6.

[8] Let $f$ given by (1) be in the class
$\mathcal{SL}_{\Sigma}(\tilde{p}(z)),$ then

$|a_{2}|\leq\frac{|\tau|}{\sqrt{1-2\tau}}$ (32)

and

$|a_{3}|\leq\frac{|\tau|(1-4\tau)}{2(1-2\tau)}.$ (33)

######  Corollary 2.7.

Let $f$ given by (1) be in the class $\mathcal{GSL}_{\Sigma}(\tilde{p}(z)),$
then

$|a_{2}|\leq\frac{|\tau|}{\sqrt{9-21\tau}}$ (34)

and

$|a_{3}|\leq\frac{|\tau|(9-26\tau)}{5(9-21\tau)}.$ (35)

##  3 Fekete-Szegö inequality for the function class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$

Fekete and Szegö [7] introduced the generalized functional $|a_{3}-\mu
a_{2}^{2}|,$ where $\mu$ is some real number. Due to Zaprawa [17],in the
following theorem we determine the Fekete-Szegö functional for
$f\in\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$.

######  Theorem 3.1.

Let $f$ given by (1) be in the class
$\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$ and $\mu\in\mathbb{R},$ then

$|a_{3}-\mu
a_{2}^{2}|\leq\left\\{\begin{array}[]{ll}\frac{|\tau|}{4(3\lambda-1)},&\mbox{
$0\leq|h(\mu)|\leq\frac{|\tau|}{4(3\lambda-1)}$},\\\
4|h(\mu)|,&\,\,\,\,\,\,\,\,\,\,\,\mbox{
$|h(\mu)|\geq\frac{|\tau|}{4(3\lambda-1)}$},\end{array}\right.$

where

$h(\mu)=\frac{(1-\mu)\tau^{2}}{4\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}.$
(1)

Proof. From (28) and (30) we obtain

$a_{3}-\mu
a_{2}^{2}=(1-\mu)\frac{\tau^{2}(c_{2}+d_{2})}{4\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}+\frac{\tau(c_{2}-d_{2})}{4(3\lambda-1)}$
(2)
$=\left(\frac{(1-\mu)\tau^{2}}{4\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}+\frac{\tau}{4(3\lambda-1)}\right)c_{2}\\\
\\\
+\left(\frac{(1-\mu)\tau^{2}}{4\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}-\frac{\tau}{4(3\lambda-1)}\right)d_{2}.$

So we have

$a_{3}-\mu
a_{2}^{2}=\left(h(\mu)+\frac{|\tau|}{4(3\lambda-1)}\right)c_{2}+\left(h(\mu)-\frac{|\tau|}{4(3\lambda-1)}\right)d_{2},$
(3)

where

$h(\mu)=\frac{(1-\mu)\tau^{2}}{4\left[(2\lambda-1)^{2}-(10\lambda^{2}-11\lambda+3)\tau\right]}.$
(4)

Then, by taking modulus of (3), we conclude that

$|a_{3}-\mu
a_{2}^{2}|\leq\left\\{\begin{array}[]{ll}\frac{|\tau|}{4(3\lambda-1)},&\mbox{
$0\leq|h(\mu)|\leq\frac{|\tau|}{4(3\lambda-1)}$},\\\
4|h(\mu)|,&\,\,\,\,\,\,\,\,\,\,\,\mbox{
$|h(\mu)|\geq\frac{|\tau|}{4(3\lambda-1)}.$}\end{array}\right.$

Taking $\mu=1$, we have the following corollary.

######  Corollary 3.2.

If $f\in\mathcal{PSL}^{\lambda}_{\Sigma}(\tilde{p}(z))$, then

$|a_{3}-a_{2}^{2}|\leq\frac{|\tau|}{4(3\lambda-1)}.$ (5)

By specializing the parameter $\lambda=1$ and $\lambda=2$ in the above
theorem, we have the following the Fekete-Szegö inequalities for the function
classes $\mathcal{SL}_{\Sigma}(\tilde{p}(z))$ and
$\mathcal{GSL}_{\Sigma}(\tilde{p}(z)),$ respectively.

######  Corollary 3.3.

[8] Let $f$ given by (1) be in the class $\mathcal{SL}_{\Sigma}(\tilde{p}(z))$
and $\mu\in\mathbb{R},$. then we have

$|a_{3}-\mu a_{2}^{2}|\leq\left\\{\begin{array}[]{ll}\frac{|\tau|}{8},&\mbox{
$0\leq|h(\mu)|\leq\frac{|\tau|}{8}$},\\\
4|h(\mu)|,&\,\,\,\,\,\,\,\,\,\,\,\mbox{
$|h(\mu)|\geq\frac{|\tau|}{8}$},\end{array}\right.$

where

$h(\mu)=\frac{(1-\mu)\tau^{2}}{4\left[1-2\tau\right]}.$ (6)

Further if $\mu=1$ we get

$|a_{3}-a_{2}^{2}|\leq\frac{|\tau|}{8}.$

######  Corollary 3.4.

Let $f$ given by (1) be in the class $\mathcal{GSL}_{\Sigma}(\tilde{p}(z))$
and $\mu\in\mathbb{R},$ then we have

$|a_{3}-\mu a_{2}^{2}|\leq\left\\{\begin{array}[]{ll}\frac{|\tau|}{20},&\mbox{
$0\leq|h(\mu)|\leq\frac{|\tau|}{20}$},\\\
4|h(\mu)|,&\,\,\,\,\,\,\,\,\,\,\,\mbox{
$|h(\mu)|\geq\frac{|\tau|}{20}$},\end{array}\right.$

where

$h(\mu)=\frac{(1-\mu)\tau^{2}}{4\left[9-21\tau\right]}.$ (7)

Further if $\mu=1$ we get

$|a_{3}-a_{2}^{2}|\leq\frac{|\tau|}{20}.$

Acknowledgements:The authors thank the referees of this paper for their
insightful suggestions and corrections to improve the paper in present form.

## References

  * [1] Paper1 Babalola K.O. On $\lambda$-pseudo-starlike functions J. Class. Anal. 3(2) 2013 137-147
  * [2] Paper2 Brannan D.A., Clunie J. and Kirwan W.E. Coefficient estimates for a class of starlike functions Canad. J. Math. 22 1970 476-485
  * [3] Paper3 Brannan D.A. and Taha T.S. On some classes of bi-univalent functions Studia Univ.Babes-Bolyai Math. 31(2) 1986 70-77
  * [4] Book4 Duren P.L. Univalent Functions In: Grundlehren der Mathematischen Wissenschaften,Band 259 New York, Berlin, Heidelberg and Tokyo, Springer-Verlag 1983
  * [5] Paper5 Joshi S., Joshi S. and Pawar H. On some subclasses of bi-univalent functions associated with pseudo-starlike function J.Egyptian Math.Soc. 24 2016 522-525
  * [6] Paper6 Dziok J., Raina R.K. and Sokół J. On $\alpha-$convex functions related to a shell-like curve connected with Fibonacci numbers Appl. Math. Comp. 218 2011 996-1002
  * [7] Paper7 Fekete M. and Szegö G. Eine Bemerkung über ungerade schlichte Functionen J. London Math. Soc. 8 1933 85-89
  * [8] Paper8 Güney H.Ö., Murugusundaramoorthy G. and Sokół J. Subclasses of bi-univalent functions related to shell-like curves connected with Fibonacci numbers Commun. Fac. Sci. Univ. Ank. Ser. A1-Math. Stat. 68(2) 2019 1909-1921
  * [9] Book9 Miller S.S. and Mocanu P.T. Differential Subordinations Theory and Applications P Series of Monographs and Text Books in Pure and Applied Mathematics Marcel Dekker, New York 2000
  * [10] Paper10 Lewin M. On a coefficient problem for bi-univalent functions Proc. Amer.Math. Soc. 18 1967 63-68
  * [11] Book11 Pommerenke Ch. Univalent Functions Math. Math, Lehrbucher, Vandenhoeck and Ruprecht, Göttingen 1975
  * [12] Paper12 Raina R.K. and Sokół J. Fekete-Szegö problem for some starlike functions related to shell-like curves Math. Slovaca 66 2016 135-140
  * [13] Paper13 Sokół J. On starlike functions connected with Fibonacci numbers Folia Scient. Univ. Tech. Resoviensis 175 1999 111-116
  * [14] Paper14 Srivastava H.M., Mishra A.K. and Gochhayat P. Certain subclasses of analytic and bi-univalent functions Appl. Math. Lett. 23(10) 2010 1188-1192
  * [15] Paper15 Xu Q.-H., Gui Y.-C. and Srivastava H.M. Coefficient estimates for a certain subclass of analytic and bi-univalent functions Appl. Math. Lett. 25 2012 990-994
  * [16] Paper16 Li X-F. and Wang A-P Two new subclasses of bi-univalent functions Inter. Math. Forum, 7(30) 2012 1495-1504
  * [17] Paper17 Zaprawa P. On the Fekete-Szegö problem for classes of bi-univalent functions Bull. Belg. Math. Soc. Simon Stevin 21(1) 2014 69-78
  * [18]


# Catastrophic Forgetting in the Context of Model Updates

Hillary Sanders & Rich Harang

###### Abstract

A large obstacle to deploying deep learning models in practice is the process
of updating models post-deployment (ideally, frequently). Deep neural networks
can cost many thousands of dollars to train. When new data comes in the
pipeline, you can train a new model from scratch (randomly initialized
weights) on all existing data. Instead, you can take an existing model and
fine-tune (continue to train) it on new data. The former is costly and slow.
The latter is cheap and fast, but catastrophic forgetting generally causes the
new model to ’forget’ how to classify older data well. There are a plethora of
complicated techniques to keep models from forgetting their past learnings.
Arguably the most basic is to mix in a small amount of past data into the new
data during fine-tuning: also known as ’data rehearsal’. In this paper, we
compare various methods of limiting catastrophic forgetting and conclude that
if you can maintain access to a portion of your past data (or tasks), data
rehearsal is ideal in terms of overall accuracy across all time periods, and
performs even better when combined with methods like Elastic Weight
Consolidation (EWC). Especially when the amount of past data (past ’tasks’) is
large compared to new data, the cost of updating an existing model is far
cheaper and faster than training a new model from scratch.

##  1 Introduction

Catastrophic Forgetting - the tendency of deep neural networks to ’forget’
previously learned information upon learning new information - has been
studied since 1989[3]. This is most clearly demonstrated when models are given
separate tasks to learn sequentially, but the effect is also at play whenever
a model is learning any type of information sequentially, where that
information changes in distribution over time. Real-world applications of
machine learning very often have new (training) data coming in over time. In
order to release new models trained on this new information, machine learning
developers can retrain an entire model from scratch (randomly initialized
weights) using all existing training data, but this is computationally very
costly. Another option is to take an existing model trained on past data, and
simply fine-tune it on the new data that has come in. But new data is coming
from (generally) a slightly different ’distribution’ than old data, and so
especially when this change in distribution is large, the catastrophic
forgetting effect becomes clear upon fine-tuning.

In the malware detection space, the distribution of malicious and benign
content is always changing. New malware is always being produced, as well as
new benign content. As a result, a model trained on data up until time $t$
will generally perform extremely well on a holdout set from before $t$ (coming
from the same parent distribution as the training data), but it’s accuracy
will, on average, decay validated against $t+n$ holdout sets with increasing
$n$.

Figure 1: Model performance on holdout validation data over time (timestamp
value is based on the first-seen time of a particular sample). Model was
trained on training data up until 2019-03. While performance on holdout
validation data is high in the months overlapping with training data, the
model’s accuracy quickly decays on holdout validation data coming from after
2019-03.

Simply fine-tuning existing models with new data, however, results in models
that perform well on data similar to the fine-tuned data, but perform poorly
on past data (which we still want our models to be able to classify well).
Figure 2 contrasts a model that has been sequentially fine-tuned on monthly
data (colored lines) vs a model that has been fully retrained from scratch.
Note how the red line (sequentially fine-tuned) does more poorly on past
months, than the black line (retrained from scratch).

Figure 2: Here, a base model trained on data up until 2019-03 (the dark teal
line) is sequentially fine-tuned on data from the following months: 2019-04
(light teal), then 2019-05 (grey), then 2019-06 (light pink), and finally
2019-07 (the red line). The black line represents a model that was retrained
from scratch (with randomly initialized weights) on all data up until 2019-07.
While the black line is far more computationally costly to produce each month,
note how it performs better on older data than its sequentially fine-tuned
comparison model, the red line.

If you can get sequentially fine-tuned models (red line) to match or beat
performance of a model trained all at once (black line) by minimizing
catastrophic forgetting, then you have a path to deploying accurate models
very quickly and very cheaply. In this paper, we compare various methods of
minimizing this catastrophic forgetting effect. Arguably the most basic
approach, one that was suggested in the 90s by X [cite], is to simply mix in
data associated with past ’tasks’ (older data) into the fine-tuning dataset.
We also tested approaches that don’t require access to past data: model
averaging, L2 regularization on weight movement, and Elastic Weight
Consolidation (EWC) regularization [2]. Our results showed that by far the
most effective approach is data rehearsal.

##  2 Related Work

Catastrophic Forgetting was introduced as an issue in 1989[3] by McCloskey and
Cohen. In 1995 Robins [4] discussed a model training set up where each new
item (task) is a single sample, but instead of training a model by
sequentially learning on each item individually, items were learned alongside
3 past items during each epoch. This was termed ’sweep rehearsal’, and helped
improve forgetting. This concept generalizes to situations in which each item
(new ’task’) is composed of many samples, and data associated with old tasks
is mixed in with new task data during model training.

Data rehearsal requires access to past data, which isn’t always possible. Much
work has been done on the subject of limiting catastrophic forgetting when you
don’t have permanent access to past task data. Atkinson et. al.[1] (among
others) use a psuedo-rehearsal system to generate (via a deep generative
neural network) items representative of previous tasks.

##  3 Experiments

Our experimental setup was as follows. We experimented with our PE malware
detection model, which is deep neural network consisting of five large fully
connected layers (sizes 1024, 768, 512, 512, 512), followed by output layers
(primarily the ‘is_malware’ output). We grabbed data and extracted features
for 12 separate months (each treated as a ’task’ in our set up), consisting of
about four million training samples each (about a million leftover for
validation and testing). The first five months were used to train an initial
’base’ model. To test a method, the base model was fine-tuned on data from the
sixth month (using the chosen method, e.g. L2 regularization, EWC, etc), the
resulting model was trained on the seventh month, that resulting model was
trained on the eight month, and that resulting model was trained on the ninth
month. The final three months were kept a time-split validation holdout. We
chose to apply the methods sequentially on four separate months in order to
better simulate real-world model updates (if a method cannot be applied
iteratively and still work, it is of much less use). Unless otherwise stated,
all models were trained for ten epochs during each training or fine-tuning
session. As a comparison, we also trained a model on the first nine months of
data all at once. Although theoretically a sequentially updated model could
perform better, the basic idea is that a sequentially updated model with no
catastrophic forgetting will perform the same as a model that was trained all
at once (the latter being far more cost-prohibitive to train for each new
model update).

How to best evaluate a model is dependant on what the model is to be used for.
We chose to use average AUC on holdout test sets across all twelve months as
our main metric (using the final sequentially updated model that has seen the
first nine months of data) . We also separately show (see Table 1): average
AUC across the first five ‘base’ months, the four ‘update’ months, and the
final three ‘future’ months.

###  3.1 Model Averaging

Simply training multiple models per time period and averaging the results
seems like a potentially reasonable approach. It would result in a much larger
and harder to deploy ensemble model, but each additional model would be fast
to train. Our model averaging results, however, were very poor. Averaging the
predictions of multiple separate models, each trained on one month’s worth of
data (for the same number of epochs as the sequentially fine-tuned model,
making training costs equal) performed much worse than a sequentially fine-
tuned model.

Figure 3: Model averaging results are shown in blue. (Note that the y-limits
in the Experiments figures are different than those shown earlier in the
paper.)

###  3.2 Regularization

One option to limit ‘forgetting’ is to suppress the movement of parameters
during fine-tuning. You can do this naively, like via $L2$ regularization, or
you can use more complicated methods like Elastic Weight Consolidation
(EWC)[2] to penalize the movement of parameters that are estimated to be
important to past tasks.

####  3.2.1 L2 Regularization

Figure 3 shows $L2$ regularization: an $L2$ loss penalty is added to each
parameter with respect to the current parameter’s value $\theta_{c}$ during
fine-tuning, and the previous model’s parameter value $\theta_{p}$. The
further away you move from the previous model’s parameter value, the higher
the penalty:$(\theta_{c}-\theta_{p})^{2}$. Training loss (denoted
$L_{c}^{`}(\theta)$ in the equation below) is simply standard training loss of
the current model $L_{c}(\theta)$, plus the $L2$ regularization penalty:

$L_{c}^{`}(\theta)=L_{c}(\theta)+\sum_{i}^{n}{(\theta_{i,c}-\theta_{i,p})^{2}}$
(1)

Figure 4: L2 regularization. The colored lines represent the original base
model trained up until 2019-03, and then sequentially fine-tuned using
different levels of $L2$ regularization.

As expected, forgetting is reduced but learning is reduced too. There’s a very
clear trade-off, with no happy medium that attains the performance of the
trained-from-scratch model.

####  3.2.2 EWC Regularization

Elastic Weight Consolidation (EWC)[2] is a regularization method developed by
researchers at Deep Mind in 2017. EWC is similar to $L2$ regularization, but
each parameter’s movement penalty is scaled by the parameter movement squared
($(\theta_{c}-\theta_{p})^{2}$: standard $L2$ regularization), multiplied by
EWC’s estimate of parameter importance to past tasks (in our case, the ability
to classify past data). The idea is to think about your base model as a
Bayesian prior of the parameter estimates - then fine tuning is just applying
more data to approximate the posterior. Essentially, you apply regularization
that simulates the prior distribution. EWC assumes that the prior is normally
distributed: with mean given by the base model’s parameters, and variance
given by the inverse of the Fisher Information matrix diagonal $F$. Instead of
minimizing loss during fine-tuning with respect to your new training data, you
minimize an estimate of your loss with respect to both your past $p$ and
current $c$ data:

$L_{p,c}(\theta)=L_{c}(\theta)+\lambda
L_{p}(\theta)=L_{c}(\theta)+\sum_{i}^{n}{\frac{\lambda}{2}*F_{i,p}*(\theta_{i,c}-\theta_{i,p})^{2}}$
(2)

So what we have here is $L2$ regularization, scaled by a scaling parameter
$\frac{\lambda}{2}$ and by this $F_{i}$ value for each parameter $\theta_{i}$.

Elastic Weight Consolidation (EWC)[2] is similar to $L2$ regularization, but
each parameter’s movement penalty is scaled by the movement squared (standard
$L2$ regularization), multiplied by EWC’s estimate of parameter importance to
classifying past data. EWC estimates this using the diagonal of the Hessian of
the negative log likelihood with respect to parameters (where this loss
corresponds to accuracy on past task(s)). This requires saving second partial
derivatives of your model’s loss with respect to its parameters during each
training round, but doesn’t require saving the data used to train the model
each round.

The specifics of the formula appear to come from using Taylor Series / Laplace
Approximation. Loss with current parameters on current data (fine-tuning data)
is known, but estimating loss with current parameters on past data is not
known. Without access to past data during fine-tuning, but _with_ access to
saved partial derivatives of loss on past data with previous parameters
$\theta_{p}$, you can estimate the loss $L_{p}$ on past data with current
parameters $\theta_{c}$ via Taylor Series:

$L_{p}(\theta_{c})=L_{p}(\theta_{p})+\frac{\partial
L_{p}}{\partial\theta_{p}}(\theta_{c}-\theta_{p})+\frac{1}{2}\frac{\partial^{2}L_{p}}{\partial\theta_{p}^{2}}(\theta_{c}-\theta_{p})^{2}+...$
(3)

Because we’re minimizing loss through gradient descent, we only care about
terms that aren’t constant with respect to $\theta_{c}$. So the first term on
the right hand side of of the equation we can drop. The second term we can
assume is 0 because $L_{p}$ was at a local minimum with previous parameters
$\theta_{p}$ (before fine-tuning), and the fourth term ($...$) we’re choosing
to drop as “small change”. What we really want to estimate is the loss of both
current and previous data on current parameters $\theta_{c}$, so adding
$L_{c}(\theta_{c})$ to both sides of our equation leads us to:

$L_{p,c}(\theta_{c})=L_{c}(\theta_{c})+\frac{1}{2}\frac{\partial^{2}L_{p}}{\partial\theta_{p}^{2}}(\theta_{c}-\theta_{p})^{2}+constant$
(4)

Replacing $\frac{1}{2}$ with the scaling term $\frac{\lambda}{2}$, leads us to
an equation that looks very similar to the one the EWC paper shows. The
formula we arrived at represents fisher information $F$ as
$\frac{\partial^{2}L_{p}}{\partial\theta_{p}^{2}}$. But is that what Fisher
Information is?

Well, with your model’s (negative log likelihood) loss as $L_{p}$, the Fisher
diagonal is defined as $E(-\frac{\partial
L_{p}}{\partial\theta}*-\frac{\partial
L_{p}}{\partial\theta})=E(\frac{\partial L_{p}}{\partial\theta}*\frac{\partial
L_{p}}{\partial\theta})$: the negatives coming from the ’negative’ log
likelihood and then cancelling out.111It’s interesting to note that when loss
is at a local minimum and thus $E(\frac{\partial L_{p}}{\partial\theta})=0$,
$E(\frac{\partial L_{p}}{\partial\theta}*\frac{\partial
L_{p}}{\partial\theta})$ is the same as the $variance(\frac{\partial
L_{p}}{\partial\theta})$.

Under certain regularity constraints (which don’t hold particularly true in
this case), the Fisher diagonal value _is_ equal to the negative expected
second partial derivative of the Loss $L$ with respect to $\theta$:
$-E(-\frac{\partial^{2}L}{\partial\theta^{2}})=E(\frac{\partial^{2}L}{\partial\theta^{2}})$
\- i.e., the diagonal of the Hessian of the negative log-likelihood (loss)
with respect to parameters: what we had in our derivation formula. If you
switch out $E(\frac{\partial^{2}L}{\partial\theta^{2}})$ for $E(\frac{\partial
L_{p}}{\partial\theta}*\frac{\partial L_{p}}{\partial\theta})$, you arrive at
the formula Deep Mind proposed.222the derivation was not published alongside
the paper, so this is our guess at how they derived their formula! 333A more
intuitive way to think about this is that $variance(\frac{\partial
L}{\partial\theta})$ might be a nice way to estimate how sensitive your model
is to changes in $\theta$. When loss is greatly affected by small changes to a
given parameter $\theta_{i}$ (i.e. $variance(\frac{\partial
L}{\partial\theta_{i}})$) is high), then the ‘confidence’ that $\theta_{i}$
should not be changed is high, so the variance in the prior distribution
should be small (tight). The added regularization term simulates this
confidence, so that parameters the model is very sensitive to don’t end up
being changed much.

While the derivation uses the Hessian form of Fisher Information, in practice,
we found that we had to use the $E(\frac{\partial
L}{\partial\theta}*\frac{\partial L}{\partial\theta})$ version to attain any
sort of good performance. The reason is that if, for any parameter $i$, your
$F_{i,p}$ term is less than zero, then the resulting regularization term will
just keep on pushing $\theta_{i}$ further and further away from $\theta_{i,p}$
(the previous value), which will tend to wreck any model (including ours). The
$E(\frac{\partial L}{\partial\theta}*\frac{\partial L}{\partial\theta})$ form
of fisher $F$ is guaranteed to be positive, while the
$E(\frac{\partial^{2}L}{\partial\theta^{2}})$ form, not so much. In practice,
we found that about a third of our
$E(\frac{\partial^{2}L}{\partial\theta^{2}})$ estimates tended to be negative,
making that approach unusable.

Figure 5: Elastic Weight Consolidation

EWC regularization performed better than basic L2 regularization, but not by
an outstanding amount. We theorize that the larger and more complicated a
model is, the more error is introduced by the taylor-series estimations and
approximations involved in EWC.

###  3.3 Data Rehearsal

Robins [4] discussed the concept of rehearsal in 1995, shortly after the
concept of catastrophic forgetting was introduced. Simply put: you mix in data
from past tasks while training on new ones. It works really well, but it does
mean you need to maintain access to your older data (or at least an _iid_
(independent and identically distributed, i.e. random) subsample of it), which
isn’t always possible. Mixing in old data also increases the amount of
training data you have, so it takes longer to fine-tune a model one epoch.

Initially, to make comparisons to other methods equal, we fixed the fine-
tuning epoch sizes to 4 million, and tested out various proportions from past
data (see Figure 6). For example, a 50% rehearsal indicates that during fine-
tuning, 2 million samples come from the new month’s data, and 2 million
samples come from past data (sampled uniformly unless otherwise stated) -
meaning that the model misses out on much new data each epoch.

Figure 6: Various forms of data rehearsal with fixed number of training
samples each epoch. Each form mixes in varying amounts of past data, but all
train for the same number of epochs on the same total number of samples.

Not fixing the fine-tuning epoch size (i.e. allow it to increase with added
past data) yielded far better results (see Figure 7). However, doing so
increases the computational cost of each model update - for example, if you
rehearse old data at a rate of 50%, epoch size jumps to 8 million, doubling
the computational cost of training. This is still generally much cheaper
computationally than retraining the model from scratch on all available data,
but is no longer a computationally fair comparison to the other methods
presented above.

Figure 7: Rehearsal without epoch size limits (i.e. fine-tuning updates are
slower amd more costly). 50% rehearsal (2x time to train, light-pink) achieved
the best average AUC.

###  3.4 Conclusion

Figure 8: Various forms of model updates compared. For each method category
(e.g. L2 regularization), the version with the best validation accuracy was
selected (e.g. L2 with a certain Lambda value).

Our experiments concluded that if you do have access to past data, rehearsal
is an excellent way to minimize catastrophic forgetting. Furthermore, if
you’re willing to increase model fine-tuning time, it is much more effective
(and easy to implement). It also can be combined with regularization
approaches, like EWC (which resulted in the best results overall).

Figure 9: When data rehearsal is combined with EWC (aqua), the sequentially
fine-tuned model very nearly matches the comparison model that is trained on
all data all at once (black), and does much better than the baseline of
sequential fine-tuning (red). Note that this version of data rehearsal did not
fix epoch sizes, and so fine-tuning time is approximately twice that of normal
(red’s time), plus extra to estimate EWC damages (cost depends on how many
samples you chose to estimate this with).

## Acknowledgements

Thanks to Sophos for supporting this research.

## References

  * [1] Craig Atkinson, Brendan McCane, Lech Szymanski and Anthony Robins  “Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting”  In _Neurocomputing_ 428 Elsevier BV, 2021, pp. 291–307  DOI: 10.1016/j.neucom.2020.11.050
  * [2] James Kirkpatrick et al.  “Overcoming catastrophic forgetting in neural networks”, 2017  arXiv:1612.00796 [cs.LG]
  * [3] Michael McCloskey and Neal J. Cohen  “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem” 24, Psychology of Learning and Motivation  Academic Press, 1989, pp. 109–165  DOI: https://doi.org/10.1016/S0079-7421(08)60536-8
  * [4] ANTHONY ROBINS  “Catastrophic Forgetting, Rehearsal and Pseudorehearsal”  In _Connection Science_ 7.2 Taylor & Francis, 1995, pp. 123–146  DOI: 10.1080/09540099550039318