# Supplemental Material : Lifelong Generative Modelling Using Dynamic
Expansion Graph Model

Fei Ye and Adrian G. Bors
Department of Computer Science, University of York, York YO10 5GH, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Contents

  1. A The proof of Theorem 1
  2. B The proof of Theorem 2
  3. C The proof of Lemma 1
  4. D The proof of Theorem 3
  5. E The proof of Lemma 2
  6. F Generalize existing ELBOs to LELBO
    1. F.1 Importance sampling
    2. F.2 Hierarchical Variational Inference
  7. G Forgetting behaviour of other generative models
    1. G.1 Generative Adversarial Nets (GANs)
    2. G.2 Energy-based GANs
  8. H Risk bound estimation from finite samples
  9. I Self-Supervised VAEs
  10. J The main objective function, algorithm and pipeline for DEGM
    1. J.1 Derivation of ${\mathcal{L}_{MELBO}}$
    2. J.2 The pipeline and algorithm of DEGM
    3. J.3 The component selection for model evaluation during the testing phase
    4. J.4 Limitations of the proposed theoretical framework
    5. J.5 The difference between related work and DEGM
  11. K Unsupervised lifelong learning benchmark
    1. K.1 Network architecture and hyperaprameter setting
    2. K.2 Results on cross-domain setting (COFMI)
    3. K.3 Results on Split MNIST and Split Fashion
  12. L Additional experiment results
    1. L.1 Quantitative evaluation
    2. L.2 Results for generalization bounds
    3. L.3 Evaluation by using other criterion
    4. L.4 Ablation study
    5. L.5 The number of parameters used in various methods
    6. L.6 Visual results

##  Appendix A A The proof of Theorem 1

###### Assumption 1

Let $\mathcal{X}$ be a metric space that satisfies
$\mathcal{L}(a,b)\leq\mathcal{L}(a,c)+\mathcal{L}(c,b)$ where the loss
function $\mathcal{L}(\cdot)$ is a metric and $a,b,c\in\mathcal{X}$.

Based on Assumption 1, we provide the detailed proof as follows :

Proof.

Let $\mathcal{P}_{i}$ and ${\tilde{\mathcal{P}}}_{i}$ be two domains over
$\mathcal{X}$. Then for
$h_{{\mathcal{P}_{i}}}^{\star}=\arg{\min_{h\in\mathcal{H}}}{{\mathcal{R}}_{{\mathcal{P}_{i}}}}(h,f_{{\mathcal{P}_{i}}})$
and
$h_{\tilde{\mathcal{P}}_{i}}^{\star}=\arg{\min_{h\in\mathcal{H}}}{{\mathcal{R}}_{\tilde{\mathcal{P}}_{i}}}(h,f_{\tilde{\mathcal{P}}_{i}})$
where $f_{\tilde{\mathcal{P}}_{i}}\in\mathcal{H}$ is the ground truth function
(identity function under the encoder-decoding process) for
$\tilde{\mathcal{P}}_{i}$.

Then according to the triangle inequality property of $\mathcal{L}$, applied
twice, we have :

$\displaystyle{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,{f_{\mathcal{P}_{i}}}\big{)}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}{h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}{h^{*}_{{\tilde{\mathcal{P}}}_{i}},h_{{\mathcal{P}_{i}}}^{*}}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}{h_{{\mathcal{P}_{i}}}^{*}},{f_{\mathcal{P}_{i}}}\big{)}$
(1)

Eq. (1) holds because, after applying twice the triangle inequality,
$\mathcal{L}(a,b)\leq\mathcal{L}(a,c)+\mathcal{L}(c,d)+L(d,b)$ where $a,b,c,d$
are $h({\bf x}),{f_{\mathcal{P}_{i}}}({\bf
x}),{h^{*}_{{\tilde{\mathcal{P}}}_{i}}}({\bf
x}),h_{{\mathcal{P}_{i}}}^{*}({\bf x})$ and ${\bf x}$ is sampled from the same
domain ${\mathcal{P}}_{i}$. According to the definition of discrepancy
distance (See Definition 2 of the paper), we define the discrepancy distance
between ${\mathcal{P}}_{i}$ and ${\tilde{\mathcal{P}}}_{i}$ as :

$\displaystyle
disc_{\mathcal{L}}\big{(}{\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i}\big{)}={\sup_{\left({h,h^{\prime}}\right)\in{\cal
H}}}\Big{|}{{{\mathbb{E}}_{{\bf{x}}\sim{\mathcal{P}_{i}}}}\left[{{\cal
L}\left({h^{\prime}\left({\bf{x}}\right),h\left({\bf{x}}\right)}\right)}\right]-{{\mathbb{E}}_{{\bf{x}}\sim{\tilde{\mathcal{P}}_{i}}}}\left[{{\cal
L}\left({h^{\prime}\left({\bf{x}}\right),h\left({\bf{x}}\right)}\right)}\right]}\Big{|}.$
(2)

We rewrite the above equation as :

$\displaystyle
disc_{\mathcal{L}}\big{(}{\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i}\big{)}={\sup_{\left({h,h^{\prime}}\right)\in{\cal
H}}}\big{|}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\left(h,h^{\prime}\right)-{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\left(h,h^{\prime}\right)\big{|}.$
(3)

We consider $h^{\prime}$ to be $h^{*}_{{\tilde{\mathcal{P}}}_{i}}$ in Eq. (3)
and we have :

$\displaystyle{\sup_{\left({h,h^{\prime}}\right)\in
H}}\Big{|}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\left(h,h^{\prime}\right)-{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\left(h,h^{\prime}\right)\Big{|}\geq\Big{|}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}-{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}\Big{|}$
(4)

We also know that :

$\displaystyle{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}\leq\Big{|}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}-{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}\Big{|}+{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}$
(5)

Therefore, we can replace the first term of the right hand side of Eq. (1) by
the right hand side of Eq. (5), resulting in :

$\displaystyle\vspace{-30pt}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\left(h,{f_{\mathcal{P}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\tilde{\mathcal{P}}}_{i}}}\big{(}{h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}}\big{)}+\Big{|}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}-{{\mathcal{R}}_{{\tilde{{\mathcal{P}}}_{i}}}}\big{(}h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}\Big{|}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},{f_{\mathcal{P}_{i}}}\big{)}$
(6)

Then the second term, representing the absolute value of the difference, in
RHS of Eq. (6) can be replaced by
$disc_{\mathcal{L}}({\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i})$ from Eq.
(4)), since the discrepancy distance between two distributions is an upper
bound to this absolute value, resulting in :

$\displaystyle\vspace{-30pt}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,{f_{\mathcal{P}_{i}}}\big{)}$
$\displaystyle\leq{{\mathcal{R}}_{{\tilde{\mathcal{P}}}_{i}}}\big{(}{h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}}\big{)}+disc_{\mathcal{L}}({\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i})+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},{f_{\mathcal{P}_{i}}}\big{)}$
(7)

From Definition 3 from the paper, we know that
$disc_{\mathcal{L}}({\mathcal{P}}_{i},{\tilde{\mathcal{P}}}_{i})\leq
disc^{\star}_{\mathcal{L}}({\mathcal{P}}_{i},{\tilde{\mathcal{P}}}_{i})$. This
allows use to replace
$disc_{\mathcal{L}}({\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i})$ by using
$disc^{\star}_{\mathcal{L}}({\mathcal{P}}_{i},{\tilde{\mathcal{P}}}_{i})$ in
Eq. (7), resulting in :

$\displaystyle\vspace{-30pt}{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h,{f_{\mathcal{P}_{i}}}\big{)}$
$\displaystyle\leq{{\mathcal{R}}_{{\tilde{\mathcal{P}}}_{i}}}\big{(}{h,h^{*}_{{\tilde{\mathcal{P}}}_{i}}}\big{)}+disc^{\star}_{\mathcal{L}}\big{(}{\mathcal{P}_{i}},{\tilde{\mathcal{P}}}_{i}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},h^{*}_{{\tilde{\mathcal{P}}}_{i}}\big{)}+{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}\big{(}h^{*}_{{\mathcal{P}}_{i}},{f_{\mathcal{P}_{i}}}\big{)}$
(8)

Eq. (8) proves Theorem 1 and a similar proof can be found in Theorem 8 from
Mansour et al. (2009).

##  Appendix B B The proof of Theorem 2

Theorem 2.

For a given sequence of tasks $\\{\mathcal{T}_{1},\dots,\mathcal{T}_{t}\\}$,
we derive a GB between the target distribution and the evolved source
distribution during the $t$-th task learning :

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{t}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right),$
(9)

where ${\mathcal{P}_{(1:t)}}$ is the mixture distribution
$\\{{\mathcal{P}}_{1}\otimes{\mathcal{P}}_{2},\dots,\otimes{\mathcal{P}}_{t}\\}$.
As it can be seen in Eq. (9), the performance on the target domain is largely
depending on the discrepancy term even if ${\cal M}$ minimizes the source risk
well. In the following, we provide an analytical bound that considers all
previously learnt distributions.

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\rm
Err}^{a}+{\rm Err}^{d},$ (10)

where ${\rm Err}^{d}\geq 0$ evaluates the difference on the two risk terms,
expressed by:

$\displaystyle\sum\nolimits_{k=1}^{t-1}\Big{\\{}{{{\mathcal{R}}_{{{\mathbb{P}}^{(t-k)}}}}\left({h,h_{{{\mathbb{P}}^{(t-l)}}}^{*}}\right)}-{{\mathcal{R}}_{{{\mathbb{P}}^{(t-1-k)}}\otimes{{\tilde{\mathcal{P}}}_{(t-k)}}}}\left({h,h_{{{\mathbb{P}}^{(t-1-k)}}\otimes{{\tilde{\mathcal{P}}}_{(t-k)}}}^{*}}\right)\Big{\\}}\,,$
(11)

where
${\mathcal{R}}_{{\mathbb{P}}^{0}\otimes{\tilde{\mathcal{P}}}_{1}}(h,h^{*}_{{\mathbb{P}}^{0}\otimes{\tilde{\mathcal{P}}}_{1}})={{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}})$.
${\rm Err}^{a}$ is the accumulated error term expressed by :

$\displaystyle\sum\limits_{k=1}^{t-2}\Big{\\{}{\mathcal{R}}_{A}\left({{{\mathbb{P}}^{(t-1-k)}}\otimes{{\tilde{\mathcal{P}}}_{(t-k)}},{{\mathbb{P}}^{(t-k)}}}\right)\Big{\\}}+{\mathcal{R}}_{A}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right).$
(12)

From Eq. (12), we can observe that while learning more tasks ($t$ is
increased) during the lifelong learning, the gap on the GB for ${\mathcal{M}}$
tends to be large since ${\rm Err}^{a}$ increases. This can also explain why
GR fails when learning a long sequence of tasks Ramapuram et al. (2020); Ye
and Bors (2020c). Additionally, the term
${\mathcal{R}}_{A}({{{\mathbb{P}}^{(t-1-k)}}\otimes{{\tilde{\mathcal{P}}}_{(t-k)}},{{\mathbb{P}}^{(t-k)}}})$
and ${\rm Err}^{d}$ tend to be small when the discrepancy
$disc^{*}_{\mathcal{L}}({\mathbb{P}^{i}},{\mathbb{P}^{(i-1)}}\otimes{\tilde{\mathcal{P}}_{i}})$
is equal to 0 in each task learning ($i=1,\dots,t$). This is achieved by the
optimal generator distribution that approximates
${\mathbb{P}^{(i-1)}}\otimes{\tilde{\mathcal{P}}_{i}}$ exactly in each task
learning (See also other explanation in Proposition 3 in Appendix I).

Proof.

Firstly, we can derive the bound according to Theorem 1 :

$\displaystyle{{{\mathcal{R}}_{{{\cal P}_{(i:t)}}}}}\left({h,{f_{{{\cal
P}_{(i:t)}}}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}^{*}}\right)+disc_{\mathcal{L}}^{\star}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+\varepsilon\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal P}}_{t}}}\right)$ (13)

where ${\mathcal{P}_{(1:t)}}$ represents the mixture distribution
$\\{{\mathcal{P}}_{1}\otimes{\mathcal{P}}_{2},\dots,\otimes{\mathcal{P}}_{t}\\}$.
Let $\rho_{(1:t)}({\bf x})$ represent the density function for
${\mathcal{P}_{(1:t)}}$ and $\rho_{(i)}({\bf x})$ the density function for
$\mathcal{P}_{i}$. Since $\mathcal{P}_{(1:t)}$ is the mixture distribution and
its density is expressed by $\rho_{(1:t)}({\bf
x})=\frac{1}{t}\sum\nolimits_{i=1}^{t}\rho_{(i)}({\bf x})$. We know that
${{{\mathcal{R}}_{{{\cal P}_{(i:t)}}}}}({h,{f_{{{\cal P}_{(i:t)}}}}})$ can be
rewritten as the integral form
$\int{{\rho_{(i:t)}}({\bf{x}}){\mathcal{L}}(h,{f_{{P_{(1:t)}}}})}\,\mathrm{d}{\bf{x}}$.
We then take $\rho_{(1:t)}({\bf
x})=\frac{1}{t}\sum\nolimits_{i=1}^{t}\rho_{(i)}({\bf x})$ in this integral
form, resulting in :

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{t}{\int{{\rho_{(i)}}({\bf{x}}){\mathcal{L}}(h,{f_{{P_{(1:t)}}}})}{\mkern
1.0mu}{\rm{d}}{\bf{x}}}$ (14)

We assume that ${\mathcal{P}}_{i}$ is independent from ${\mathcal{P}_{j}}$,
where $i\neq j$, which is a reasonable assumption, since each task is
associated with a different dataset. Therefore, the true labeling function
$f_{\mathcal{P}_{(1:t)}}$ can be represented by $f_{{{\mathcal{P}}_{i}}}$
under the target distribution ${\mathcal{P}}_{i}$ of the $i$-th task. Then we
rewrite Eq. (14) as the expectation form
$\frac{1}{t}\sum\nolimits_{i=1}^{t}{{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}(h,{f_{{{\mathcal{P}}_{i}}}})}$.

Based on the above results, Eq. (13) is rewritten as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{{{\mathcal{R}}_{{{\mathcal{P}}_{i}}}}(h,{f_{{{\mathcal{P}}_{i}}}})}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}^{*}}\right)$ (15)
$\displaystyle+disc_{\mathcal{L}}^{\star}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+\varepsilon\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal P}}_{t}}}\right)$

In the following, we provide the derivations for ${\rm Err}^{a}$ and ${\rm
Err}^{d}$ from Eq. (10). We consider to take
${{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}}}$ and
${{{\mathbb{P}}^{t-1}}}$ as the target and source domains, respectively. This
is reasonable choice since we allow the ${{{\mathbb{P}}^{t-1}}}$ (the
generator distribution of the model) to approximate
${{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}}}$. We derive the
bound as :

$\displaystyle{{\mathcal{R}}_{{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}}}}\left({h,h_{{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}}}^{*}}\right)\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}}^{*}}\right)+disc_{\cal
L}^{\star}\left({{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}},{{\mathbb{P}}^{t-1}}}\right)+\varepsilon\left({{{\mathbb{P}}^{t-2}}\otimes{{\tilde{\mathcal{P}}}_{t-1}},{{\mathbb{P}}^{t-1}}}\right)$
(16)

We then consider to take
${{{\mathbb{P}}^{t-3}}\otimes{{\tilde{\mathcal{P}}}_{t-2}}}$ and
${{{\mathbb{P}}^{t-2}}}$ as the target and source domains, respectively, and
we derive the bound as :

$\displaystyle{{\mathcal{R}}_{{{\mathbb{P}}^{t-3}}\otimes{{\tilde{\mathcal{P}}}_{t-2}}}}\left({h,h_{{{\mathbb{P}}^{t-3}}\otimes{{\tilde{\mathcal{P}}}_{t-2}}}^{*}}\right)\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-2}}}}\left({h,h_{{{\mathbb{P}}^{t-2}}}^{*}}\right)+disc_{\cal
L}^{\star}\left({{{\mathbb{P}}^{t-3}}\otimes{{\tilde{\mathcal{P}}}_{t-2}},{{\mathbb{P}}^{t-2}}}\right)+\varepsilon\left({{{\mathbb{P}}^{t-3}}\otimes{{\tilde{\mathcal{P}}}_{t-2}},{{\mathbb{P}}^{t-2}}}\right)$
(17)

According to the Inductive Inference, we have :

$\displaystyle\dots{\\\ }\dots{\\\ }$
$\displaystyle{{\mathcal{R}}_{{{\mathbb{P}}^{1}}\otimes{{\tilde{\mathcal{P}}}_{2}}}}\left({h,h_{{{\mathbb{P}}^{1}}}^{*}}\right)\leq{{\mathcal{R}}_{{{\mathbb{P}}^{2}}}}\left({h,h_{{{\mathbb{P}}^{2}}}^{*}}\right)+disc_{\cal
L}^{\star}\left({{{\mathbb{P}}^{1}}\otimes{{\tilde{\mathcal{P}}}_{2}},{{\mathbb{P}}^{2}}}\right)+\varepsilon\left({{{\mathbb{P}}^{1}}\otimes{{\tilde{\mathcal{P}}}_{2}},{{\mathbb{P}}^{2}}}\right)$
(18)

$\displaystyle{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}\left({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}}\right)\leq{{\mathcal{R}}_{{{\mathbb{P}}^{1}}}}\left({h,h_{{{\mathbb{P}}^{1}}}^{*}}\right)+disc_{\cal
L}^{\star}\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)+\varepsilon\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)$
(19)

We then sum up all the above derivations in the inequality from Eq. (15) to
Eq. (19), resulting in :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)}\Big{\\}}+{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}\left({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}}\right)+{\sum\limits_{k=1}^{t-2}\Big{\\{}{{{\mathcal{R}}_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}^{*}}\right)}}\Big{\\}}\leq{\sum\limits_{k=1}^{t-2}\Big{\\{}{{{\mathcal{R}}_{{{\mathbb{P}}^{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-k}}}^{*}}\right)}}\Big{\\}}$
(20)
$\displaystyle+{\mathcal{R}}_{{\mathbb{P}}^{1}}\left(h,h^{*}_{{\mathbb{P}}^{1}}\right)+{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+\sum\limits_{k=1}^{t-2}\Big{\\{}disc_{\cal
L}^{\star}\left({{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}},{{\mathbb{P}}^{t-k}}}\right)+\varepsilon\left({{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}},{{\mathbb{P}}^{t-k}}}\right)\Big{\\}}$
$\displaystyle+disc_{\mathcal{L}}^{\star}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+\varepsilon\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+disc_{\cal
L}^{\star}\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)+\varepsilon\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)$

Then we move the second and third term in the left hand side to the right hand
side in Eq.(20), resulting in :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)}\Big{\\}}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\mathcal{R}}_{{\mathbb{P}}^{1}}\left(h,h^{*}_{{\mathbb{P}}^{1}}\right)-{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}\left({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}}\right)$
(21)
$\displaystyle+{\sum\limits_{k=1}^{t-2}\Big{\\{}{{{\mathcal{R}}_{{{\mathbb{P}}^{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-k}}}^{*}}\right)}-{{\mathcal{R}}_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}^{*}}\right)}\Big{\\}}$
$\displaystyle+\sum\limits_{k=1}^{t-2}\left(disc_{\cal
L}^{\star}\left({{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}},{{\mathbb{P}}^{t-k}}}\right)\right.$
$\displaystyle\left.+\varepsilon\left({{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}},{{\mathbb{P}}^{t-k}}}\right)\right)+disc_{\mathcal{L}}^{\star}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal P}}_{t}}}\right)$
$\displaystyle+\varepsilon\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+disc_{\cal
L}^{\star}\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)+\varepsilon\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)$

Then we can rewrite Eq. (21) as :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)}\Big{\\}}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\mathcal{R}}_{{\mathbb{P}}^{1}}\left(h,h^{*}_{{\mathbb{P}}^{1}}\right)-{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}\left({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}}\right)$
(22)
$\displaystyle+\underbrace{{\sum\limits_{k=1}^{t-1}\Big{\\{}{{{\mathcal{R}}_{{{\mathbb{P}}^{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-k}}}^{*}}\right)}-{{\mathcal{R}}_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}}\left({h,h_{{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}}}^{*}}\right)}\Big{\\}}}_{\rm{Err}^{a}}$
$\displaystyle+\underbrace{\sum\limits_{k=1}^{t-2}\Big{\\{}{\mathcal{R}}_{A}\left({{{\mathbb{P}}^{t-1-k}}\otimes{{\tilde{\mathcal{P}}}_{t-k}},{{\mathbb{P}}^{t-k}}}\right)\Big{\\}}+{\mathcal{R}}_{A}\left({{{\cal
P}_{(1:t)}},{{\mathbb{P}}^{(t-1)}}\otimes{{\tilde{\cal
P}}_{t}}}\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{1}},{{\mathbb{P}}^{1}}}\right)}_{\rm{Err}^{d}}$

where
${\mathcal{R}}_{{\mathbb{P}}^{0}\otimes{\tilde{\mathcal{P}}}_{1}}(h,h^{*}_{{\mathbb{P}}^{0}\otimes{\tilde{\mathcal{P}}}_{1}})={{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{1}}}}({h,h_{{{\tilde{\mathcal{P}}}_{1}}}^{*}})$
and this proves Theorem 2.

##  Appendix C C The proof of Lemma 1

According to the bound on the KL divergence :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{{\mathbb{E}_{{{\mathcal{P}}_{i}}}}KL(q_{\omega^{t}}({\bf{z}}\mid{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))}$
$\displaystyle\leq{{\mathbb{E}}_{{{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}}}KL(q_{\omega^{t}}({\bf{z}}\mid{\bf{\tilde{x}}}^{t})\mid\mid
p({\bf{z}}))$ (23)
$\displaystyle+\Big{|}{{{\mathbb{E}}_{{{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}}}KL(q_{\omega^{t}}({\bf{z}}\mid{\bf{\tilde{x}}}^{t})\mid\mid
p({\bf{z}}))-\frac{1}{t}\sum\limits_{i=1}^{t}{{\mathbb{E}_{{{\mathcal{P}}_{i}}}}KL(q_{\omega^{t}}({\bf{z}}\mid{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))}}\Big{|}$

We also know that $\mathcal{L}_{ELBO}\left({\bf x};\\{\theta,\omega\\}\right)$
is expressed as :

$\displaystyle\mathcal{L}_{ELBO}\left({\bf
x};\\{\theta,\omega\\}\right):={\mathbb{E}_{{q_{\omega}}\left({{\bf z}\mid{\bf
x}}\right)}}\left[{\log{p_{\theta}}\left({{\bf x}\mid{\bf
z}}\right)}\right]-KL\left[{{q_{\omega}}\left({{\bf z}\,|\,{\bf
x}}\right)\mid\mid p\left({\bf z}\right)}\right]\,,$ (24)

When the decoder models a Gaussian distribution, ${\log{p_{\theta}}\left({{\bf
x}\,|\,{\bf z}}\right)}$ can be represented as :

$\displaystyle{\log{p_{\theta}}\left({{\bf x}\,|\,{\bf
z}}\right)}=-\frac{1}{{2\sigma_{\theta}^{2}\left({\bf{z}}\right)}}{\left\|{{\bf{x}}-{\mu_{\theta}}\left({\bf{z}}\right)}\right\|^{2}}-\frac{1}{2}\log
2\pi\sigma_{\theta}^{2}\left({\bf{z}}\right)$ (25)

where ${\sigma_{\theta}\left({\bf{z}}\right)}$ and
${{\mu_{\theta}}\left({\bf{z}}\right)}$ are the variance and mean of Gaussian
distribution, obtained by the decoder. ${\left\|{\cdot}\right\|^{2}}$
represents the reconstruction error (square loss). We implement the decoder by
a Gaussian distribution with the fixed variance
${\mathcal{N}}({\mu_{\theta}}\left({\bf{z}}\right),\sigma{\bf{I}})$ where
${{\mu_{\theta}}\left({\bf{z}}\right)}$ is a deep convolutional neural network
and ${\bf{I}}$ is the identity matrix. Therefore, Eq. (25) is represented by
the fixed variance $\sigma$ :

$\displaystyle{\log{p_{\theta}}\left({{\bf x}\,|\,{\bf
z}}\right)}=-\frac{1}{{2\sigma^{2}}}{\left\|{{\bf{x}}-{\mu_{\theta}}\left({\bf{z}}\right)}\right\|^{2}}-\frac{1}{2}\log
2\pi\sigma^{2}$ (26)

Since $h$ is the hypothesis of the model ($\mathcal{M}^{t}$), implemented as a
encoding-decoding process, we have

${\mathcal{L}_{ELBO}}({\bf{x}}_{i}^{T};h)=-\frac{1}{{2\sigma^{2}}}{\mathcal{L}}(h({\bf{x}}_{i}^{T}),{f_{{{\mathcal{P}}_{i}}}}({\bf{x}}_{i}^{T}))-\frac{1}{2}\log
2\pi\sigma^{2}-{{KL}}(q_{\omega^{t}}({\bf{z}}\mid{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))$

.

Then we focuse on the negative ELBO

$-{\mathcal{L}_{ELBO}}({\bf{x}}_{i}^{T};h)=\frac{1}{{2\sigma^{2}}}{\mathcal{L}}(h({\bf{x}}_{i}^{T}),{f_{{{\mathcal{P}}_{i}}}}({\bf{x}}_{i}^{T}))+\frac{1}{2}\log
2\pi\sigma^{2}+{{KL}}(q_{\omega^{t}}({\bf{z}}\mid{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))$

.

And we know that
${\mathcal{R}}_{\mathcal{P}_{i}}(h,f_{\mathcal{P}_{i}})={\mathbb{E}}_{{\bf
x}\sim\mathcal{P}_{i}}{\mathcal{L}}(h({\bf
x}^{T}_{i}),f_{\mathcal{P}_{i}}({\bf x}^{T}_{i}))$ and we have :

$\displaystyle{\mathbb{E}}_{{\bf
x}\sim\mathcal{P}_{i}}\left[-\mathcal{L}_{ELBO}({\bf
x}^{T}_{i};h)\right]={\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}\left\\{\frac{1}{{2\sigma^{2}}}{\mathcal{L}}(h({\bf
x}^{T}_{i}),f_{\mathcal{P}_{i}}({\bf x}^{T}_{i}))+KL(q_{\omega^{t}}({\bf
z}\mid{\bf x}^{T}_{i})\mid\mid p({\bf z}))\right\\}+\frac{1}{2}\log
2\pi\sigma^{2}$ (27)

We observe that $\frac{1}{2}\log 2\pi\sigma^{2}$ and $\frac{1}{{2\sigma^{2}}}$
are constants. In order to simplify the notations, we set
$\sigma=\frac{1}{{\sqrt{2}}}$. Therefore, Eq. (27) is rewritten as :

$\displaystyle{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}\left[-\mathcal{L}_{ELBO}({\bf
x}^{T}_{i};h)\right]={\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}\\{{\mathcal{L}}(h({\bf
x}^{T}_{i}),f_{\mathcal{P}_{i}}({\bf x}^{T}_{i}))+KL(q_{\omega^{t}}({\bf
z}\mid{\bf x}^{T}_{i})\mid\mid p({\bf z}))\\}+\frac{1}{2}\log\pi$ (28)

Eq. (28) can be seen as the average ELBO for all samples. We then take Eq.
(23) in Eq.(7) in the paper, we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}\\{{\mathcal{L}}(h({\bf
x}^{T}_{i}),f_{\mathcal{P}_{i}}({\bf
x}^{T}_{i}))+{{KL}}(q_{\omega^{t}}({\bf{z}}|{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))\\}\Big{\\}}$ $\displaystyle\leq{{\mathbb{E}}_{{\bf
x}^{t}\sim{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\Big{\\{}{\mathcal{L}}\left({h({\tilde{\bf
x}}^{t}),h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}({\tilde{\bf
x}}^{t})}\right)$ (29)
$\displaystyle+{{{KL}}(q_{\omega^{t}}({\bf{z}}\mid{\bf{\tilde{x}}}^{t})\mid\mid
p({\bf{z}}))}\Big{\\}}$ $\displaystyle+\left|KL_{1}-KL_{2}\right|\
+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)$

It notes that we can add the constant $\frac{1}{2}\log\pi$ in both sides of
Eq. (29). According to Eq. (28), we can rewrite Eq. (29) as :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}[-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}}]$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\left[-{\mathcal{L}_{ELBO}\left({\bf\tilde{x}}^{t};h\right)}\right]+\left|KL_{1}-KL_{2}\right|+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)\,.$
(30)

This proves Lemma 1. We can further replace the latest term in the right hand
side of Eq. (30) by ${\rm Err}^{a}+{\rm Err}^{d}$ (See details in the proof of
Theorem 2), resulting in :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}[-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}}]$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\left[-{\mathcal{L}_{ELBO}\left({\bf\tilde{x}}^{t};h\right)}\right]+\left|KL_{1}-KL_{2}\right|+{\rm
Err}^{a}+{\rm Err}^{d}.$ (31)

##  Appendix D D The proof of Theorem 3

Firstly, ${\mathcal{R}}_{C}$ can be easily proved since the task was trained
only once and we simply derive the bound between the training sets and testing
sets.

For the components that are trained more than once, we firstly consider the
$c^{\prime}_{i}$-th component and $a(i,j)$-th task. This can be easily
generalized to other components and other tasks. We firstly consider to take
${\mathcal{P}}_{a(i,j)}$ and ${\mathbb{P}}^{0}_{a(i,j)}$ as the target and
source distribution and we have a bound :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}}_{a(i,j)}}}\left({h_{c^{\prime}_{i}},{f_{{\mathcal{P}}_{a(i,j)}}}}\right)\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{0}}}\left({h_{c^{\prime}_{i}},{h^{*}_{{\mathbb{P}}_{a(i,j)}^{0}}}}\right)+disc_{\mathcal{L}}^{\star}\left({{\mathcal{P}}_{a(i,j)},{\mathbb{P}}_{a(i,j)}^{0}}\right)+\varepsilon\left({{\mathcal{P}}_{a(i,j)},{\mathbb{P}}_{a(i,j)}^{0}}\right)$
(32)

We can observe that ${\mathbb{P}}^{0}_{a(i,1)}$ represent the training set
${\tilde{\mathcal{P}}}_{a(i,j)}$. Then we consider to take
${{\mathbb{P}}_{a(i,j)}^{0}}$ and ${{\mathbb{P}}_{a(i,j)}^{1}}$ as the target
and source domains, respectively. Then we have the bound as :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{0}}}\left({h_{c^{\prime}_{i}},{f_{{\mathbb{P}}_{a(i,j)}^{0}}}}\right)\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{1}}}\left({h_{c^{\prime}_{i}},{h^{*}_{{\mathbb{P}}_{a(i,j)}^{1}}}}\right)+disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{0},{\mathbb{P}}_{a(i,j)}^{1}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{0},{\mathbb{P}}_{a(i,j)}^{1}}\right)$
(33)

Similarly, we have the following bounds :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{1}}}\left({h_{c^{\prime}_{i}},{h^{*}_{{\mathbb{P}}_{a(i,j)}^{1}}}}\right)\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{2}}}\left({h_{c^{\prime}_{i}},{h^{*}_{{\mathbb{P}}_{a(i,j)}^{2}}}}\right)+disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{1},{\mathbb{P}}_{a(i,j)}^{2}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{1},{\mathbb{P}}_{a(i,j)}^{2}}\right)$
(34)

$\displaystyle\dots{\\\ }$
$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-2}}}\left({h_{c^{\prime}_{i}},{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-2}}}}\right)\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}^{*}}\right)+disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{c(i,j)-2},{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{c(i,j)-2},{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}\right)$
(35)

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}}\left({h_{c^{\prime}_{i}},{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)-1}}}}\right)\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}^{*}}\right)+disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{c(i,j)-1},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{c(i,j)-1},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)$
(36)

Then we sum up all the above relationships, resulting in :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}_{a(i,j)}}}}\left({h_{c^{\prime}_{i}},{f_{{\mathcal{P}_{a(i,j)}}}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}^{*}}\right)$
(37)
$\displaystyle+\sum\limits_{k=-1}^{c(i,j)-1}{\Big{\\{}{disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)}\Big{\\}}8i}$

where we also use ${\mathbb{P}}^{-1}_{a(i,j)}$ represent
${\mathcal{P}}_{a(i,j)}$. RHS of Eq. (37) is an upper bound to the target risk
for a single task ${\mathcal{P}}_{a(i,j)}$ modelled by using the
$c^{\prime}_{i}$-th component. In the following, we consider all components
$C^{\prime}$ that are trained more than once :

$\displaystyle\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{{\mathcal{R}}_{{\mathcal{P}_{a(i,j)}}}}\left({h_{c^{\prime}_{i}},{f_{{\mathcal{P}_{a(i,j)}}}}}\right)}$
$\displaystyle\leq\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}\left\\{{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}^{*}}\right)\right.$
(38)
$\displaystyle\left.{+\sum\limits_{k=-1}^{c(i,j)-1}{\left({disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)+\varepsilon\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)}\right)}}\right\\}$

RHS of Eq. (38) is still an upper bound to the target risk of tasks modelled
by the components that trained more than once. Therefore,
${\mathcal{R}}_{A^{\prime}}$ in the paper, can be expressed by RHS of Eq.
(38), which proves Theorem 3.

Based on the results from Theorem 3, we provide the additional analysis of the
results of Theorem 3 in the following. Firstly, we rewrite Eq.(11) of the
paper by :

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{|C|}\Big{\\{}{\mathcal{R}}_{{{\mathcal{P}}_{a_{i}}}}\left(h_{c_{i}},f_{{\mathcal{P}}_{a_{i}}}\right)\Big{\\}}+\frac{1}{t}\sum\nolimits_{i=1}^{|C^{\prime}|}\sum\nolimits_{j=1}^{{{\tilde{a}}_{i}}}\Big{\\{}\mathcal{R}_{{\mathcal{P}}_{a(i,j)}}\left({h_{c^{\prime}_{i}},{f_{{{\mathcal{P}}_{a(i,j)}}}}}\right)\Big{\\}}\leq\sum\nolimits_{i=1}^{|C|}\Big{\\{}\mathcal{R}_{{\tilde{\mathcal{P}}_{a_{i}}}}\left(h_{c_{i}},h^{*}_{\tilde{\mathcal{P}}_{a_{i}}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{a_{i}}},{\tilde{\mathcal{P}}_{a_{i}}}}\right)\Big{\\}}$
(39)
$\displaystyle+\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}\left\\{{{\mathcal{R}}_{\mathbb{P}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}^{*}}\right){+\sum\limits_{k=-1}^{c(i,j)-1}{\left({\mathcal{R}}_{A}\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)\right)}}\right\\}\,.$

We consider an extreme case where ${\bf M}$ only has a single component after
LLL, ($|C^{\prime}|=1$ and $|C|=0$). Then the first term in RHS of Eq. (39)
disappears and resulting in :

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{|C^{\prime}|}\sum\nolimits_{j=1}^{{{\tilde{a}}_{i}}}\mathcal{R}_{{\mathcal{P}}_{a(i,j)}}\left({h_{c^{\prime}_{i}},{f_{{{\mathcal{P}}_{a(i,j)}}}}}\right)\leq\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}\left\\{{{\mathcal{R}}_{\mathbb{P}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},h_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}^{*}}\right){+\sum\limits_{k=-1}^{c(i,j)-1}{\Big{\\{}{\mathcal{R}}_{A}\left({{\mathbb{P}}_{a(i,j)}^{k},{\mathbb{P}}_{a(i,j)}^{k+1}}\right)\Big{\\}}}}\right\\}\,.$
(40)

In this case, learning early tasks ($a(i,j)$ is small) tends to increase the
number of error terms more than when learning more recent tasks ($a(i,j)$ is
large), This is caused by the number of accumulated error terms
${\mathcal{R}}_{A}(\cdot)$ controlled by the times of GR processes
$c(i,j)=t-a(i,j)$. In the opposite case where the number of components ($K$)
is equal to the number of tasks ($t$), $|C^{\prime}|=0$ and the mixture model
has not accumulated errors. The GB for this case is :

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{|C|}{\mathcal{R}}_{{{\mathcal{P}}_{a_{i}}}}\left(h_{c_{i}},f_{{\mathcal{P}}_{a_{i}}}\right)\leq\sum\nolimits_{i=1}^{|C|}\Big{\\{}\mathcal{R}_{{\tilde{\mathcal{P}}_{a_{i}}}}\left(h_{c_{i}},h^{*}_{\tilde{\mathcal{P}}_{a_{i}}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{a_{i}}},{\tilde{\mathcal{P}}_{a_{i}}}}\right)\Big{\\}}\,.$
(41)

where $|C|=K=t$. Then the lifelong learning problem is transformed to be the
generalization problem under the generative modelling. This motivates us to
propose a novel dynamic mixture model which would not accumulate errors during
LLL.

##  Appendix E E The proof of Lemma 2

Similarly to the proof for Theorem 3, we firstly consider the components that
are trained only once :

$\displaystyle{\mathcal{R}}_{C}$
$\displaystyle=\sum\limits_{i=1}^{{|C|}}\Big{\\{}{{\mathcal{R}}_{{\tilde{\mathcal{P}}_{a_{i}}}}}\left({h_{c_{i}},h^{*}_{\tilde{\mathcal{P}}_{a_{i}}}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{a_{i}}},{\tilde{\mathcal{P}}_{a_{i}}}}\right)\Big{\\}},$
(42)

Then we add the KL divergence term and $D_{diff}$ term in Eq. (42), resulting
in :

$\displaystyle{\mathcal{R}}_{C}$
$\displaystyle=\sum\limits_{i=1}^{|C|}\Big{\\{}{{\mathcal{R}}_{{\tilde{\mathcal{P}}_{a_{i}}}}}\left({h_{c_{i}},h^{*}_{\tilde{\mathcal{P}}_{a_{i}}}}\right)+{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{a_{i}}}KL\left(p_{c_{i}}({\bf
z}\mid{\bf x}^{S}_{a_{i}})\mid\mid p({\bf z})\right)+D_{diff}\left({\bf
x}^{T}_{a_{i}},{\bf
x}^{S}_{a_{i}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{a_{i}}},{\tilde{\mathcal{P}}_{a_{i}}}}\right)\Big{\\}},$
(43)

where $p_{c_{i}}({\bf z}\mid\cdot)$ represents the variational distribution
modelled by the inference model of the $c_{i}$-th component.
$D_{diff}(\cdot,\cdot)$ is defined as :

$\displaystyle D_{diff}({\bf x}^{T}_{a_{i}},{\bf
x}^{S}_{a_{i}})=\Big{|}{{{\mathbb{E}}_{{{\mathcal{P}}_{a_{i}}}}}KL(q_{\omega_{c_{i}}}({\bf{z}}\mid{\bf{x}}_{i}^{T})\mid\mid
p({\bf{z}}))-{{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}}KL(q_{\omega_{c_{i}}}({\bf{z}}\mid{\bf{x}}_{i}^{S})\mid\mid
p({\bf{z}}))}\Big{|}$ (44)

We can rewrite Eq. (43) as the negative ELBO form :

$\displaystyle{\mathcal{R}}_{C}$
$\displaystyle=\sum\limits_{i=1}^{|C|}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{-\mathcal{L}_{ELBO}\left({\bf
x}^{S}_{a_{i}};h_{c_{i}}\right)\\}+D_{diff}\left({\bf x}^{T}_{a_{i}},{\bf
x}^{S}_{a_{i}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{a_{i}}},{\tilde{\mathcal{P}}_{a_{i}}}}\right)\Big{\\}},$
(45)

where ${\mathcal{L}}_{ELBO}(\cdot;h_{c_{i}})$ represents the ELBO estimated by
the $c_{i}$-th component. Secondly, we consider the components that are
trained more than once :

$\displaystyle{{\mathcal{R}}_{A^{\prime}}}$
$\displaystyle=\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}}\right)+{\mathcal{R}}_{A}\left({{{\mathcal{P}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\},$
(46)

We then add the KL divergence term and $D_{diff}$ term in eq.(46), resulting
in :

$\displaystyle{{\mathcal{R}}_{A^{\prime}}}$
$\displaystyle=\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h_{c^{\prime}_{i}},{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}}\right)+{\mathbb{E}}_{{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}KL\left(p_{c^{\prime}_{i}}({\bf
z}\mid{\bf x}^{t}_{a(i,j)})\mid\mid p({\bf z})\right)\right.$ (47)
$\displaystyle\left.+D_{diff}\left({\bf x}^{T}_{a(i,j)},{\bf
x}^{t}_{a(i,j)}\right)+{\mathcal{R}}_{A}\left({{{\mathcal{P}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\},$

where the conditional distribution $p_{c^{\prime}_{i}}({\bf z}|{\bf
x}^{t}_{a(i,j)})$ is modelled by the inference model of the
$c^{\prime}_{i}$-th component. Then we rewrite Eq. (47) as the negative ELBO
form :

$\displaystyle{{\mathcal{R}}_{A^{\prime}}}$
$\displaystyle=\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-\mathcal{L}_{ELBO}\left({\bf
x}^{t}_{a(i,j)};h_{c^{\prime}_{i}}\right)+D_{diff}\left({\bf
x}^{T}_{a(i,j)},{\bf
x}^{t}_{a(i,j)}\right)+{\mathcal{R}}_{A}\left({{{\mathcal{P}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\},$
(48)

We summarize all KL divergence and $D_{diff}$ terms :

$\displaystyle\sum\limits_{i=1}^{|C|}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{a_{i}}}KL\left(p_{c_{i}}({\bf
z}\mid{\bf x}^{S}_{a_{i}})\mid\mid p({\bf z})\right)+D_{diff}\left({\bf
x}^{T}_{a_{i}},{\bf x}^{S}_{a_{i}}\right)\Big{\\}}$ (49)
$\displaystyle+\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}KL\left(p_{c^{\prime}_{i}}({\bf
z}\mid{\bf x}^{t}_{a(i,j)})\mid\mid p({\bf z})\right)+D_{diff}\left({\bf
x}^{T}_{a(i,j)},{\bf x}^{t}_{a(i,j)}\right)\right\\},$

We also know that :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{{\mathcal{P}}}_{i}}KL\left(p({\bf
z}\mid{\bf x}^{T}_{i})\mid\mid p({\bf z})\right)$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{|C|}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{a_{i}}}KL\left(p({\bf
z}\mid{\bf x}^{S}_{a_{i}})\mid\mid p({\bf z})\right)+D_{diff}\left({\bf
x}^{T}_{a_{i}},{\bf x}^{S}_{a_{i}}\right)\Big{\\}}$ (50)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}KL\left(p({\bf
z}\mid{\bf x}^{t}_{a(i,j)})\mid\mid p({\bf z})\right)+D_{diff}\left({\bf
x}^{T}_{a(i,j)},{\bf x}^{t}_{a(i,j)}\right)\right\\}$

where we omit the subscript for $p{({\bf z}\mid\cdot)}$ for simplicity. We
then consider the inequality from Eq. (50) into Eq. (11) of the paper and we
have :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\mathcal{P}}_{a_{i}}}\Big{\\{}-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{a_{i}};h_{c_{i}}\right)}}\Big{\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\mathbb{E}}_{{\mathcal{P}}_{a(i,j)}}\Big{\\{}-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{a(i,j)};h_{c^{\prime}_{i}}\right)}}\Big{\\}}\leq\frac{1}{t}\sum\limits_{i=1}^{|C|}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{a_{i}}}\left\\{-{{{\cal
L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h_{c_{i}}\right)}\right\\}\Big{\\}}$
(51)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{|{A^{\prime}_{c^{\prime}_{i}}}|}\Big{\\{}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{{-{{\cal
L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h_{c^{\prime}_{i}}\right)}\right\\}\Big{\\}}+\frac{1}{t}\\{{\mathcal{R}}^{II}_{A^{\prime}}+{{\mathcal{R}}_{C}^{II}}+D_{diff}*\\}$

This proves Lemma 2. We should also observe that $D_{diff}*$ is expressed by :

$\displaystyle D_{diff}*=\sum\limits_{i=1}^{|C|}\Big{\\{}D_{diff}\left({\bf
x}^{T}_{a_{i}},{\bf
x}^{S}_{a_{i}}\right)\Big{\\}}+\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{D_{diff}\left({\bf
x}^{T}_{a(i,j)},{\bf x}^{t}_{a(i,j)}\right)\right\\},$ (52)

Based on the above results, in the following, we derive the risk bound of the
mixture model to NLL :

$\displaystyle\frac{1}{t}{\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\mathcal{P}}_{a_{i}}}\Big{\\{}-\log{p_{c_{i}}\left({\bf
x}^{T}_{a_{i}}\right)}}\Big{\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}\Big{\\{}-{\log
p_{c^{\prime}_{i}}\left({\bf
x}^{T}_{a(i,j)}\right)}}\Big{\\}}\leq\frac{1}{t}\sum\limits_{i=1}^{|C|}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{a_{i}}}\left\\{-{{{\cal
L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h_{c_{i}}\right)}\right\\}\Big{\\}}$
(53)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{|{A^{\prime}_{c^{\prime}_{i}}}|}\Big{\\{}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{{-{{\cal
L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h_{c^{\prime}_{i}}\right)}\right\\}\Big{\\}}+\frac{1}{t}\Big{\\{}{\mathcal{R}}^{II}_{A^{\prime}}+{{\mathcal{R}}_{C}^{II}}+D_{diff}*\Big{\\}}$

where $\log p_{c_{i}}(\cdot)$ represents the sample log-likelihood (model
likelihood) under the $c_{i}$-th component.

##  Appendix F F Generalize existing ELBOs to LELBO

In this section, we generalize previous well know ELBOs to LELBO.

###  F.1 Importance sampling

The main idea of Importance Weighted Autoencoder (IWELBO) Burda et al. (2015)
is to allow the recognition network to generate multiple samples during the
optimization leading to a better modelling of the posterior probabilities. The
corresponding ELBO for sampling $K^{\prime}$ samples is defined as :

$\displaystyle{{\mathcal{L}}_{{ELBO}_{K^{\prime}}}}\left({\bf{x}};{\mathcal{M}}\right)={{\mathbb{E}}_{{{\bf{z}}_{1}},..,{{\bf{z}}_{K^{\prime}}}\sim
q\left({{\bf{z}}|{\bf{x}}}\right)}}\left[{\log\frac{1}{K^{\prime}}\sum\limits_{i=1}^{K^{\prime}}{\frac{{p\left({{\bf{x}},{{\bf{z}}_{i}}}\right)}}{{q\left({{{\bf{z}}_{i}}|{\bf{x}}}\right)}}}}\right]$
(54)

where $K^{\prime}$ is the number of weighted samples and $K^{\prime}=1$ is
equivalent to the standard ELBO. In order to calculate $w_{i}=p({\bf x},{\bf
z}_{i})/q({\bf z}_{i}|{\bf x})$ in practice, we rewrite $w_{i}$ as $\exp({\log
w_{i}})$. By calculating the right hand side of Eq. (54) requires to estimate
each individual ${\mathbb{E}}_{z_{i}}\log w_{i}$ which is a standard ELBO.

In the following, we extend this IWELBO to the LLL setting. From Lemma 1, we
know that :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{\mathcal{L}_{ELBO}\left({\bf\tilde{x}}^{t};h\right)}\Big{]}$
(55) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

where ${\mathcal{L}_{ELBO}(\cdot)}$ has the form according to Kingma and
Welling (2013). We can rewrite the above equation, by considering importance
sampling, as :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\log p\left({\bf
x}^{T}_{i}\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{{\mathbb{E}}_{{\bf{z}}\sim
q\left({{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log\frac{{p\left({{{\tilde{\bf{x}}}^{t}},{\bf{z}}}\right)}}{{q\left({{\bf{z}}\mid{\bf{x}}}\right)}}}\right]\Big{]}$
(56) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

According to $-\log
p({\bf{x}})\leq-{{\mathcal{L}}_{{ELBO}_{K^{\prime}+1}}}({\bf{x}};{\mathcal{M}})\leq-{{\mathcal{L}}_{{ELBO}_{K^{\prime}}}}({\bf{x}};{\mathcal{M}})$
Burda et al. (2015), we have
$-{{\mathcal{L}}_{{ELBO}_{K^{\prime}+1}}}({\tilde{\bf{x}}^{t}};{\mathcal{M}})\leq-{{\mathcal{L}}_{{ELBO}_{K^{\prime}}}}({{\tilde{\bf{x}}}^{t}};{\mathcal{M}})$,
based on the assumation that ${\mathbb{P}}^{t-1}$ is fixed. We note that
$\mathcal{M}$ represents the model and $h$ is the hypothesis of $\mathcal{M}$.
We assume that when the $h^{\star}$ is an optimal solution for $-\log
p({\tilde{\bf{x}}}^{t})$ and we have $-\log
p({\tilde{\bf{x}}}^{t})\leq-{\mathcal{L}}_{ELBO}({\tilde{\bf{x}}}^{t};h^{\star})\leq-{\mathcal{L}}_{{ELBO}_{K^{\prime}}}({\tilde{\bf{x}}}^{t};h)$.
In the following, we rewrite Eq. (55) by using $h^{\star}$, resulting in :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h^{\star}\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{\mathcal{L}_{ELBO}\left({\bf\tilde{x}}^{t};h^{\star}\right)}\Big{]}$
(57) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

We observe that $|KL_{1}-KL_{2}|$ of Eq. (57) is estimated by $h^{\star}$
($\\{\theta_{\star},\omega_{\star}\\}$ are the corresponding model
parameters). We can replace the first term in RHS of Eq. (57) by using
${\mathcal{L}}_{{ELBO}_{K^{\prime}}}({\tilde{\bf{x}}}^{t};h)$, resulting in :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h^{\star}\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{\mathcal{L}}_{{ELBO}_{K^{\prime}}}({\tilde{\bf{x}}}^{t};h)\Big{]}$
(58) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

LHS of Eq. (58) is an upper bound to
$\frac{1}{t}{\sum\nolimits_{i=1}^{t}}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}[-\log p_{\theta^{\star}}({\bf x}_{i}^{T})]$
and we rewrite Eq. (58) as :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-\log p_{\theta^{\star}}({\bf
x}_{i}^{T})\Big{]}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{\mathcal{L}}_{{ELBO}_{K^{\prime}}}({\tilde{\bf{x}}}^{t};h)\Big{]}$
(59) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

We then decompose the first term in RHS of Eq. (59), and we have :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\log p_{\theta^{\star}}\left({\bf
x}^{T}_{i}\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\left[-{\mathbb{E}_{{{\bf{z}}_{1}},\dots,{{\bf{z}}_{K^{\prime}}}\sim
q\left({{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log\frac{1}{K^{\prime}}\sum\limits_{i=1}^{K^{\prime}}{\frac{{p_{\theta}\left({{{\tilde{\bf{x}}}^{t}},{{\bf{z}}_{i}}}\right)}}{{q_{\omega}\left({{{\bf{z}}_{i}}\mid{\bf{x}}}\right)}}}}\right]\right]$
(60) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

If we do not consider ${\rm Err}^{a}$ and ${\rm Err}^{d}$ as in Lemma 1 (See
details in Appendix C), Eq. (60) can be :

$\displaystyle\frac{{\rm{1}}}{t}{\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\left[-{\log p\left({\bf
x}^{T}_{i}\right)}\right]}$ $\displaystyle\leq{\mathbb{E}}_{{\tilde{\bf
x}}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\left[-{\mathbb{E}_{{{\bf{z}}_{1}}\dots,{\bf{z}}_{K^{\prime}}\sim
q\left({{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log\frac{1}{K’}\sum\limits_{i=1}^{K’}{\frac{{p\left({{{\tilde{\bf{x}}}^{t}},{{\bf{z}}_{i}}}\right)}}{{q\left({{{\bf{z}}_{i}}\mid{\bf{x}}}\right)}}}}\right]\right]$
(61)
$\displaystyle+\left|KL_{1}-KL_{2}\right|+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)\,.$

where we omit the subscript (the model’s parameters) for Eq. (61) for the sake
of simplification. It notes that $h\in\mathcal{H}$ is the model and its
parameters are $\\{\theta,\omega\\}$ optimized by Eq.(60). We call RHS of Eq.
(60) as $\mathcal{L}_{{LELBO}_{K^{\prime}}}$ and when $K^{\prime}=1$,
$\mathcal{L}_{{LELBO}_{K^{\prime}}}$ is equal to $\mathcal{L}_{LELBO}$ (RHS of
Eq. (56)). Based on the assumption that
${\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}$ is fixed, we have
$\mathcal{L}_{{LELBO}_{K^{\prime}+1}}\leq\mathcal{L}_{{LELBO}_{K^{\prime}}}$.
We can observe that the tightness of ELBO on the marginal log-likelihood of
the source distribution ${\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}$
can not guarantee a tight GB on the marginal log-likelihood of the target
distribution. However, the tightness of GB is largely depending on the
discrepancy distance between the evolved source and target distribution.

###  F.2 Hierarchical Variational Inference

In this section, we review the Hierarchical latent variable model and extend
it to IELBO.

Auxiliary Deep Generative Models (ADGM) Maal$\o$e et al. (2016)

ADGM is a classical hierarchical latent variable model. ADGM introduces an
auxiliary variable $\bf a$ into the variational distribution $q({\bf a},{\bf
z}\,|\,{\bf x})=q({\bf z}\,|\,{\bf a},{\bf x})q({\bf a}\,|\,{\bf x})$ and its
ELBO is expressed as :

$\displaystyle\log
p\left({\bf{x}}\right)=\log\iint{p\left({{\bf{x}},{\bf{a}},{\bf{z}}}\right)d{\bf{a}}d{\bf{z}}}\geq{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\bf{x}}}\right)p\left({{\bf{x}}\mid{\bf{z}}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\bf{x}}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\bf{x}}}\right)}}}\right]={\mathcal{L}}_{ADGM}\left({\mathcal{M}},{\bf
x}\right)\,.$ (62)

We decompose $\mathcal{L}_{ADGM}({\bf x};{\mathcal{M}})$ as :

$\displaystyle{{\cal
L}_{ADGM}}\left({\mathcal{M}},{\bf{x}}\right)={{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log
p\left({{\bf{x}}\mid{\bf{z}}}\right)}\right]+{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\bf{x}}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\bf{x}}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\bf{x}}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\bf{x}}}\right)}}}\right]$
(63)

We consider the following inequality :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}{{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\bf{x}}_{i}^{T}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\bf{x}}_{i}^{T}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\bf{x}}_{i}^{T}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\bf{x}}_{i}^{T}}\right)}}}\right]}\leq{\mathbb{E}}_{\tilde{{\bf
x}}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{i}}{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{{\bf{x}}^{t}}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\tilde{\bf{x}}^{t}}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\tilde{\bf{x}}^{t}}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\tilde{\bf{x}}^{t}}}\right)}}}\right]$
(64) $\displaystyle+\left|{\mathbb{E}}_{{\tilde{\bf
x}}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{i}}{{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\tilde{\bf{x}}^{t}}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\tilde{\bf{x}}^{t}}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\tilde{\bf{x}}^{t}}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\tilde{\bf{x}}^{t}}}\right)}}}\right]-\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim\mathcal{P}_{i}}{{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\bf{x}}_{i}^{T}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\bf{x}}_{i}^{T}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\bf{x}}_{i}^{T}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\bf{x}}_{i}^{T}}\right)}}}\right]}}\right|\,.$

We name the second term of RHS of Eq. (64) as $|{\bf E}^{*}_{1}-{\bf
E}^{*}_{2}|$. By considering Lemma 1, we have :

$\displaystyle\left(1/t\right){\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\Big{[}-{\mathcal{L}_{ADGM}\left({\bf
x}^{T}_{i}\right)}\Big{]}}$ $\displaystyle\leq{\mathbb{E}}_{\tilde{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\Big{[}-{{\mathbb{E}}_{q\left({{\bf{a}},{\bf{z}}\mid{\tilde{\bf{x}}^{t}}}\right)}}\left[{\log\frac{{p\left({{\bf{a}}\mid{\bf{z}},{\tilde{\bf{x}}}^{t}}\right)p\left({{\tilde{\bf{x}}^{t}}\mid{\bf{z}}}\right)p\left({\bf{z}}\right)}}{{q\left({{\bf{a}}\mid{\tilde{\bf{x}}^{t}}}\right)q\left({{\bf{z}}\mid{\bf{a}},{\tilde{\bf{x}}^{t}}}\right)}}}\right]\Big{]}$
(65) $\displaystyle+\left|{\bf E}^{*}_{1}-{\bf E}^{*}_{2}\right|+{\rm
Err}^{a}+{\rm Err}^{d}.$

Importance Weighted Hierarchical Variational Inference (IWHVI) Sobolev and
Vetrov (2019).

IWHVI introduces a multisample generalization for the hierarchical variable
model by using two auxiliary distributions,
$q({\bf{\varsigma}}\,|\,{\bf{z}})$, $\tau({\bf{\psi}}\,|\,{\bf{z}},{\bf{x}})$
and its ELBO is expressed as :

$\displaystyle\log
p\left({\bf{x}}\right)\geq{{\mathbb{E}}_{q\left({{\bf{z}}\mid{\bf{x}}}\right)}}\log\frac{{p\left({{\bf{x}},{\bf{z}}}\right)}}{{q\left({{\bf{z}}\mid{\bf{x}}}\right)}}\geq{{\mathbb{E}}_{q\left({{\bf{z}},{{\bf{\psi}}_{0}}\mid{\bf{x}}}\right)}}{{\mathbb{E}}_{\tau\left({{{\bf{\psi}}_{1:K}}\mid{\bf{z}},{\bf{x}}}\right)}}{{\mathbb{E}}_{q\left({{{\bf{\zeta}}_{1:L}}\mid{\bf{z}}}\right)}}\log\frac{{p\left({{\bf{x}}\mid{\bf{z}}}\right)\frac{1}{L}\sum\limits_{k=1}^{L}{\frac{{p\left({{\bf{z}},{{\bf{\zeta}}_{k}}}\right)}}{{p\left({{{\bf{\zeta}}_{k}},{\bf{z}}}\right)}}}}}{{\frac{1}{{K+1}}\sum\limits_{k=0}^{K}{\frac{{q\left({{\bf{z}},{{\bf{\psi}}_{k}}\mid{\bf{x}}}\right)}}{{\tau\left({{{\bf{\psi}}_{k}}\mid{\bf{z}},{\bf{x}}}\right)}}}}}={\mathcal{L}}_{IWHVI}\left({\mathcal{M}},{\bf
x}\right)\,.$ (66)

From Eq. (66), we have $-\log p(\bf x)\leq-\mathcal{L}_{ELBO}({\bf
x};{\mathcal{M}})\leq-{\mathcal{L}}_{IWHVI}({\bf x};{\mathcal{M}})$.
Therefore, according to Lemma 1, we generalize Eq. (66) to LLL setting :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\\{-{\log p\left({\bf x}^{T}_{i}\right)}\\}}$
$\displaystyle\leq{\mathbb{E}}_{\tilde{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\\{-{\mathcal{L}}_{IWHVI}\left(\tilde{\bf
x}^{t};h\right)\\}$ (67) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm
Err}^{a}+{\rm Err}^{d}\,.$

Additionally, Eq (67) can be extended to the IWELBO bound :

$\displaystyle\left(1/t\right){\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\\{-{\log p\left({\bf x}^{T}_{i}\right)}\\}}$
$\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\left\\{-{\mathbb{E}}\log\frac{1}{{K^{\prime}}}\sum\limits_{i=1}^{K^{\prime}}{\frac{{p\left({{\bf{x}}|{{\bf{z}}_{i}}}\right)\frac{1}{L}\sum\limits_{j=1}^{L}{\frac{{p\left({{{\bf{z}}_{i}},{{\bf{\zeta}}_{\left({i.j}\right)}}}\right)}}{{q\left({{{\bf{\zeta}}_{\left({i.j}\right)}}\mid{{\bf{z}}_{i}}}\right)}}}}}{{\frac{1}{{K+1}}\sum\limits_{k=0}^{K}{\frac{{q\left({{{\bf{z}}_{i}},{{\bf{\psi}}_{\left({i,k}\right)}}\mid{\bf{x}}}\right)}}{{\tau\left({{{\bf{\psi}}_{\left({i,k}\right)}}\mid{\bf{x}},{{\bf{z}}_{i}}}\right)}}}}}}\right\\}$
(68) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}\,.$

We call the first term of RHS of Eq. (68) as
${\mathcal{L}}_{{LIWHVI}_{K^{\prime}}}$. Each
${{{\bf{\psi}}_{\left({i,0}\right)}}}$ is sampled from
$q({{{\bf{\psi}}}}\mid{\bf x})$ and each
${{{\bf{\psi}}_{\left({i,k}\right)}}}$ is sampled from
${\tau\left({{{\bf{\psi}}}\mid{\bf{z}}_{i},{\bf{x}}}\right)}$.
${{{\bf{\zeta}}_{\left({i.j}\right)}}}$ is sampled from
$q({{{\bf{\zeta}}}}\mid{\bf z}_{i})$ and ${\bf z}_{i}$ is sampled from $q({\bf
z}\mid{\bf x},{{{\bf{\psi}}_{\left({i,0}\right)}}})$. Based on the assumption
that ${\mathbb{P}}^{t-1}$ is fixed and $|KL_{1}-KL_{2}|=0$, we have
${\mathcal{L}}_{{LIWHVI}_{K^{\prime}}}\geq{\mathcal{L}}_{{LIWHVI}_{K^{\prime}+1}}$.

Ladder Variational Autoencoders (LVA) Sønderby et al. (2016).

LVA introduces a new inference model into the VAE framework, which allows to
generate the latent variable ${\bf z}_{i}$ in each $i$-th layer and its
approximate posterior is expressed as :

$\displaystyle
q\left({{\bf{z}}\mid{\bf{x}}}\right)=q\left({{{\bf{z}}_{1}}\mid{\bf{x}}}\right)\prod\limits_{i=2}^{L}{q\left({{{\bf{z}}_{i}}\mid{{\bf{z}}_{i-1}}}\right)}\,.$
(69)

We can also extend ELBO frm LVAE to the LLL setting, resulting in :

$\displaystyle\frac{1}{t}{\sum\nolimits_{i=1}^{t}{\mathbb{E}}_{{\bf
x}^{T}_{i}\sim{\mathcal{P}_{i}}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}\\}}$ $\displaystyle\leq{\mathbb{E}}_{{\bf
x}^{t}\sim{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}\\{-{\mathcal{L}_{LVA}\left({\bf\tilde{x}}^{t};h\right)}\\}$
(70) $\displaystyle+\left|KL_{1}-KL_{2}\right|+{\rm Err}^{a}+{\rm Err}^{d}.$

We use the approximate posterior (Eq. (69)) in $|KL_{1}-KL_{2}|$.

##  Appendix G G Forgetting behaviour of other generative models

In this section, we extend the proposed GB to other types of generative models
including GANs and Energy-based GANs.

###  G.1 Generative Adversarial Nets (GANs)

The discrepancy distance has been used in GANs Adlam et al. (2019) but its
usage has not been explored within the LLL setting. Following from Adlam et
al. (2019), we define
${{\mathcal{L}}_{\mathcal{H}}}{\rm{=\\{}}{\mathcal{L}}(h({\bf{x}}),h^{\prime}({\bf{x}})):h,h^{\prime}\in{\mathcal{H}}{\rm{\\}}}$
as the family of discriminators which is used in the discrepancy distance (See
Definition 2 in the paper). The errors of the model on the target distribution
are bounded, as in Adlam et al. (2019) :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}_{i}}}}\left(h,{f_{\mathcal{P}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{1}}}\left({h,{f_{\mathcal{P}_{i}}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathcal{P}_{1}},{{\mathbb{P}}^{i}}}\right)\,,$
(71)

where we assume that ${f_{\mathcal{P}_{i}}}$ is the true labeling function for
the $i$-th task. The detailed proof can be found in Adlam et al. (2019).
${\mathbb{P}}^{1}$ represents the generator distribution of a GAN model
trained on the $i$-th task learning. However, Eq. (71) is only applied for a
single domain. In the following, we generalize Eq. (71) to the LLL setting,
inspired by the proposed GB.

In order to overcome the forgetting, GANs are trained in a Self-Supervised (GR
process) fashion in which the generator and discriminator are retrained on its
generations. In this case, the previously generated samples and samples from
the current task are treated as real images while the generated images during
the training are treated as fake images. In the following, we derive a GB
between the target distribution and the evolved source distribution.

###### Proposition 1

Let a GAN model be trained on a sequence of $t$ tasks by using the GR process,
then we derive a GB for this GAN between the target and the evolved source
distribution during the $t$-th task learning :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}_{\left(1:t\right)}}}}\left(h,{f_{\mathcal{P}_{\left(1:t\right)}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t}}}}\left({h,{f_{\mathcal{P}_{\left(1:t\right)}}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathcal{P}_{\left(1:t\right)}},{{\mathbb{P}}^{t}}}\right)\,,$
(72)

where ${\mathcal{P}}_{(1:t)}$ represents the mixture distribution formed by
samples draw from $\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$.
${\mathbb{P}}^{t}$ represents the generator distribution of a GAN model
trained on the $t$-th task learning. In the following, we derive a GB to
exhibit how a GAN model lose previously learnt knowledge for each task
learning.

###### Proposition 2

For a given a sequence of $t$ tasks, a GAN model is trained with the GR
process. We derive a GB on the risk of all tasks between the target
distribution and the evolved distribution at the $t$-th task learning :

$\displaystyle\sum\limits_{k=1}^{t}{{{\cal R}_{{{\cal
P}_{k}}}}\left({h,{f_{{{\cal
P}_{k}}}}}\right)}{\rm{}}\leq\sum\limits_{k=1}^{t}{\left\\{{{{\cal
R}_{{\mathbb{P}}_{k}^{t}}}\left({h,{f_{{\mathcal{P}}_{k}}}}\right)+\sum\limits_{j=k}^{t}{\\{disc_{\cal
L}^{\star}\left({{\mathbb{P}}_{k}^{j-1},{\mathbb{P}}_{k}^{j}}\right)\\}}+disc_{\cal
L}^{\star}\left({{\mathbb{P}}_{k}^{k-1},{{\cal P}_{i}}}\right){\mkern
1.0mu}}\right\\}},$ (73)

Eq. (73) explicitly measures the degenerated performance for each task
learning, caused by the discrepancy distance between two distributions
${\mathbb{P}}^{j-1}_{i}$ and ${\mathbb{P}}^{j}_{i}$. From Eq. (73), we find
that as learning more tasks ($t$ is increased), The generalization performance
of the GAN model would be gradually degenerated since the discrepancy distance
terms are increased (The second term in the right hand side of Eq. (73).

Proof.

In order to measure the forgetting behaviour of a GAN for each task learning,
we need to define the individual approximation distribution related to each
task. Let us define the approximation distribution ${\mathbb{P}}^{j}_{i}$
formed by the sampling process ${\bf x}\sim{\mathbb{P}}^{j}$ if
$I_{\mathcal{T}}({\bf x})=i$. ${\mathbb{P}}^{j}_{i}$ is the probabilistic
representation of the generated data related to the $i$-th task where $j$
represents that the GAN model has been trained on $j$ number of tasks. we use
${\mathbb{P}}^{(i-1)}_{i}$ represent ${\tilde{\mathcal{P}}}_{i}$ for
simplicity. Therefore, for the $i$-th task, we can have the following bound :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}^{(i-1)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{i}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{i}_{i},{\mathbb{P}}^{(i-1)}_{i}}\right)\,,$
(74)

and

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}^{i}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{(i+1)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{(i+1)}_{i},{\mathbb{P}}^{i}_{i}}\right)\,,$
(75)

In the following, we treat ${\mathbb{P}}^{(i+1)}_{i}$ as the target
distribution and ${\mathbb{P}}^{(i+2)}_{i}$ as the source distribution, we
have :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}^{(i+1)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{(i+2)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{(i+2)}_{i},{\mathbb{P}}^{(i+1)}_{i}}\right)\,,$
(76)

We repeat this process, resulting in :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}^{((i+2)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{((i+3)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{(i+3)}_{i},{\mathbb{P}}^{(i+2)}_{i}}\right)$
(77) $\displaystyle\dots{\\\ }\dots{\\\
}{{\mathcal{R}}_{{\mathbb{P}}^{t-1}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{t}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{t}_{i},{\mathbb{P}}^{t-1}_{i}}\right)$

We then sum up all inequalities, resulting in :

$\displaystyle{{\mathcal{R}}_{{\mathbb{P}}^{(i-1)}_{i}}}\left({h,f_{{\mathcal{P}}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{t}_{i}}}\left({h,{f_{{\mathcal{P}}_{i}}}}\right)+\sum\limits_{j=i}^{t}{disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{i}^{j-1},{\mathbb{P}}_{i}^{j}}\right)}\,,$
(78)

We can observe that the left hand side of Eq. (78) is also an upper bound to
the target risk of the model at the $i$-th task :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}}_{i}}}\left(h,{f_{\mathcal{P}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{(i-1)}_{i}}}\left({h,{f_{{\mathcal{P}}_{i}}}}\right)+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{(i-1)}_{i},{\mathcal{P}}_{i}}\right)\,,$
(79)

By comparing Eq. (79) and Eq. (78), we have a GB for the $i$-th task :

$\displaystyle{{\mathcal{R}}_{{\mathcal{P}}_{i}}}\left(h,{f_{\mathcal{P}_{i}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{\mathbb{P}}^{t}_{i}}}\left({h,{f_{{\mathcal{P}}_{i}}}}\right)+\sum\limits_{j=i}^{t}\\{{disc_{\mathcal{L}}^{\star}\left({{\mathbb{P}}_{i}^{j-1},{\mathbb{P}}_{i}^{j}}\right)}\\}+dis{c^{\star}_{\mathcal{L}}}\left({{\mathbb{P}}^{i-1}_{i},{\mathcal{P}}_{i}}\right)\,,$
(80)

Then we can easily obtain a GB for all tasks based on Eq. (80).

$\displaystyle\sum\limits_{k=1}^{t}{{{\cal R}_{{{\cal
P}_{k}}}}\left({h,{f_{{{\cal
P}_{k}}}}}\right)}{\rm{}}\leq\sum\limits_{k=1}^{t}{\left\\{{{{\cal
R}_{{\mathbb{P}}_{k}^{t}}}\left({h,{f_{{\mathcal{P}}_{k}}}}\right)+\sum\limits_{j=k}^{t}{\\{disc_{\cal
L}^{\star}\left({{\mathbb{P}}_{k}^{j-1},{\mathbb{P}}_{k}^{j}}\right)\\}}+disc_{\cal
L}^{\star}\left({{\mathbb{P}}_{k}^{k-1},{{\cal P}_{i}}}\right){\mkern
1.0mu}}\right\\}},$ (81)

This proves Proposition 2.

###  G.2 Energy-based GANs

Energy function was firstly used in GANs Zhao et al. (2017) in 2016, called
EBGAN. Different from GANs, EBGAN introduces using a discriminator consisting
of an auto-encoder to calculate the energy value which is the reconstruction
loss. It shows that EBGAN assigns higher energy to fake images and low energy
to real images. We implement $h\in{\mathcal{H}}$ as the discriminator. EBGAN
in lifelong learning can be seen as an advanced Self-Supervised VAEs without
the KL divergence term since its generator can produce sharper images than
VAEs. Then we can derive a GB for EBGAN, similar to that defined by Theorem 2.

$\displaystyle\frac{1}{t}\sum\nolimits_{i=1}^{t}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right),$
(82)

Different from Theorem 2, ${{{\mathbb{P}}^{t-1}}}$ is approximated by the
generator of EBGAN trained on the $(t-1)$ number of tasks. Since EBGAN does
not have the KL regularization and therefore the generator tends to generate
more realistic samples when comparing with the generator of VAEs. In addition
to the EBGAN, the proposed theory framework can be used to analyze the
forgetting behaviour for the existing generative models Chen et al. (2016a);
Dziugaite et al. (2015); Ye and Bors (2021c, d, b, 2020b) and lifelong
learning approaches Seff et al. (2017); Ye and Bors (2021f, e); Yoon et al.
(2017); Ye and Bors (2021g, a, 2020c, 2020a). We will investigate these
methods in the future work.

##  Appendix H H Risk bound estimation from finite samples

In this section, we introduce how to estimate the risk bound from finite
samples.

###### Definition 1

(Rademacher complexity).

Let $\mathcal{H}$ represent a hypothesis class, For a given unlabeled sample
$U=\\{{{\bf{x}}_{i}}\\}_{i=1}^{m}$, the Rademacher complexity of $\mathcal{H}$
with respect to the sample $U$ is defined as follows :

$\displaystyle{{\mathop{\rm
Re}\nolimits}_{U}}\left(\mathcal{H}\right)=\mathop{\mathbb{missing}}{E}\limits_{\mathcal{K}}\left[{\mathop{\sup}\limits_{h\in\mathcal{H}}\frac{2}{m}\sum\limits_{i=1}^{m}{{{\mathcal{K}}_{i}}h\left({{{\bf{x}}_{i}}}\right)}}\right]$
(83)

where $\mathcal{K}_{i}$ is an independent uniform random variable within
$\\{-1,+1\\}$. The Rademacher complexity for the whole hypothesis class is
defined as :

$\displaystyle{{\mathop{\rm
Re}\nolimits}_{n}}\left(\mathcal{H}\right)={\mathbb{E}_{U\sim\left({{D}}\right)^{n}}}{{\mathop{\rm
Re}\nolimits}_{U}}\left(\mathcal{H}\right)$ (84)

Lemma 3

Let $\mathcal{H}$ represent a hypothesis class and let
${\mathcal{L}_{\mathcal{H}}}=\\{\mathcal{X}\to\mathcal{L}(h^{\prime}({\bf{x}}),h({\bf{x}})),(h^{\prime},h)\in\mathcal{H}\\}$.
Let $\mathcal{L}:\mathcal{X}\times\mathcal{X}\to{\mathbb{R}_{+}}$ be a loss
function, satisfying
$\forall({\bf{x}},{\bf{x^{\prime}}})\in\mathcal{X},\mathcal{L}({\bf{x}},{\bf{x^{\prime}}})>M$
and $M$ is the positive value. Let $\mathcal{P}$ be a distribution over
$\mathcal{X}$ and let $\tilde{\mathcal{P}}$ be the corresponding empirical
distribution formed by a sample $U=\\{{{\bf x}_{1},\dots,{\bf x}_{m}}\\}$.
Then for any $\delta\in(0,1)$, with the probability $1-\delta$, we have :

$\displaystyle
disc_{\mathcal{L}}\left({\mathcal{P},{\tilde{\mathcal{P}}}}\right)\leq{{\mathop{\rm
Re}\nolimits}_{U}}\left({{\mathcal{L}_{\mathcal{H}}}}\right)+3M\sqrt{\frac{{\log\frac{2}{\delta}}}{{2m}}}$
(85)

From Lemma 3, we can generalize the bound in case of more general loss
function
$\forall({\bf{x^{\prime}}},{\bf{x}})\in{\mathcal{X}^{2}},{\mathcal{L}_{q}}({\bf{x^{\prime}}},{\bf{x}})=|{\bf{x^{\prime}}}-{\bf{x}}{|^{q}}$
for some $q$. In the following, we show how to estimate the discrepancy
distance by using the finite number of samples according to Mansour et al.
(2009).

###### Corollary 1

Let $\mathcal{P}$ and $\mathbb{P}$ represent two domains over $\mathcal{X}$,
respectively. Let $U_{\mathcal{P}}$ and $U_{\mathbb{P}}$ represent samples of
size $m_{\mathcal{P}}$ and $m_{\mathbb{P}}$, drawn independently from
$U_{\mathcal{P}}$ and $U_{\mathbb{P}}$. Let $\hat{\mathcal{P}}$ and
$\hat{\mathbb{P}}$ represent the empirical distributions for $U_{\mathcal{P}}$
and $U_{\mathbb{P}}$. let
${\mathcal{L}}({\bf{x^{\prime}}},{\bf{x}})=|{\bf{x^{\prime}}}-{\bf{x}}{|^{2}}$
be a loss function, satisfying
$\forall({\bf{x}},{\bf{x^{\prime}}})\in\mathcal{X},\mathcal{L}({\bf{x}},{\bf{x^{\prime}}})>M$
and $M$ is the positive value. Then with probability $1-\delta$, we have :

$\displaystyle disc_{\mathcal{L}}\left({\mathcal{P},\mathbb{P}}\right)$
$\displaystyle\leq
disc_{\mathcal{L}}\left({\hat{\mathcal{P}},\hat{\mathbb{P}}}\right)+4q\left({{{\mathop{\rm
Re}\nolimits}}_{{U_{\mathcal{P}}}}}\left(\mathcal{H}\right)+{{{\mathop{\rm
Re}\nolimits}}_{{U_{\mathbb{P}}}}}\left(\mathcal{H}\right)\right)$ (86)
$\displaystyle+3M\left({\sqrt{\frac{{\log\left({\frac{4}{\delta}}\right)}}{{2{m_{\mathcal{P}}}}}}+\sqrt{\frac{{\log\left({\frac{4}{\delta}}\right)}}{{2{m_{\mathbb{P}}}}}}}\right)$

We use $disc_{\mathcal{L}}^{\star}({\mathcal{P},\mathbb{P}})$ to represent the
right hand side of eq.(86).

Corollary 1 can allow us to estimate the risk bound by using finite samples.
In the following, we derive the risk bound for the model based on the
empirical data distribution.

##  Appendix I I Self-Supervised VAEs

Self-Supervised technologies have been used in the semi-supervised learning
Zhai et al. (2019), multi-task learning Doersch and Zisserman (2017) and
generative models Chen et al. (2019); Tran et al. (2019); Cao et al. (2020).
In the context of lifelong learning, self-supervised learning is used to
overcome catastrophic forgetting Seff et al. (2017) that this paper focuses
on.

The learning process of the Self-Supervised VAEs is training a model
$\mathcal{M}$ on the samples generated by its decoder $g_{\theta}(.)$ and the
samples from the current task. For instance, at the $i$-th task learning, we
get samples from the mixture distribution
$\mathbb{P}^{(i-1)}\otimes{\tilde{\mathcal{P}}}_{i}$ where
$\mathbb{P}^{(i-1)}$ is the distribution for ${\mathcal{M}}^{(i-1)}$, which
was trained with the $(i-1)$-th task. We assume that at the $i$-th task
learning, we can generate a joint sample $({\bf x}^{\prime}_{(i-1)},{\bf
x}^{S}_{i})$ from $\mathbb{P}^{(i-1)}$ and ${\tilde{\mathcal{P}}}_{i}$,
respectively. Then the model can be represented by
${p_{\theta}}({{\bf{x}}^{\prime}_{(i-1)}},{\bf{x}}_{i}^{S},{{\bf{z}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})={p_{\theta}}({{\bf{x}}^{\prime}_{(i-1)}},{\bf{x}}_{i}^{S}|{{\bf{z}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})p({{\bf{z}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})$
where ${{\bf{z}}^{\prime}_{(i-1)}}$ and ${\bf{z}}_{i}^{S}$ are two latent
variables corresponding to different observed variables. We aim to maximize
the sample log-likelihood, expressed as :

$\displaystyle\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}},{\bf{x}}_{i}^{S})$
$\displaystyle=\log\int{\int{{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}},{\bf{x}}_{i}^{S}|{{{\bf{z}}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})p({{{\bf{z}}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})d{{{\bf{z^{\prime}}}}_{(i-1)}}d{\bf{z}}_{i}^{S}}}$
(87)
$\displaystyle\geq\int{\int{\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}},{\bf{x}}_{i}^{S}|{{{\bf{z}}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})p({{{\bf{z}}}^{\prime}_{(i-1)}},{\bf{z}}_{i}^{S})d{{{\bf{z}}}^{\prime}_{(i-1)}}d{\bf{z}}_{i}^{S}}}$

Since ${\bf x}^{\prime}_{(i-1)}$ and ${\bf x}^{S}_{i}$ are independent, we can
rewrite Eq.(87) as :

$\displaystyle\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}})+\log{p_{\theta}}({\bf{x}}_{i}^{S})\geq\int{\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}}|{{{\bf{z}}}^{\prime}_{(i-1)}})p({{{\bf{z}}}^{\prime}_{(i-1)}})d{{{\bf{z}}}^{\prime}_{(i-1)}}+}\int{\log{p_{\theta}}({\bf{x}}_{i}^{S}|{\bf{z}}_{i}^{S})p({\bf{z}}_{i}^{S})d{\bf{z}}_{i}^{S}}$
(88)

By considering to use the variational distributions Kingma and Welling (2013),
we rewrite eq.(88) as :

$\displaystyle\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}})+\log{p_{\theta}}({\bf{x}}_{i}^{S})$
$\displaystyle\geq{\mathbb{E}_{{q_{{\omega^{i}}}}({{{\bf{z}}}^{\prime}_{(i-1)}}|{{{\bf{x}}}^{\prime}_{(i-1)}})}}[\log{p_{\theta}}({{{\bf{x}}}^{\prime}_{(i-1)}}|{{{\bf{z}}}^{\prime}_{(i-1)}})]+KL[{q_{{\omega^{i}}}}({{{\bf{z}}}^{\prime}_{(i-1)}}|{{{\bf{x}}}^{\prime}_{(i-1)}})||p({{{\bf{z}}}^{\prime}_{(i-1)}})]$
(89)
$\displaystyle+{\mathbb{E}_{{q_{{\omega^{i}}}}({\bf{z}}_{i}^{S}|{\bf{x}}_{i}^{S})}}[\log{p_{\theta}}({\bf{x}}_{i}^{S}|{\bf{z}}_{i}^{S})]+KL[{q_{{\omega^{i}}}}({\bf{z}}_{i}^{S}|{\bf{x}}_{i}^{S})||p({\bf{z}}_{i}^{S})]$

where $p({{{\bf{z}}}^{\prime}_{(i-1)}})$ and $p({\bf{z}}_{i}^{S})$ are normal
distributions. Eq.(89) is used for training the VAEs model in a Self-
Supervised fashion at the $i$-th task learning. In order to simplify the loss
function used for VAEs under LLL, we uniformly generate samples from a mixture
distribution ${\mathbb{P}}^{i}\otimes{\tilde{\mathcal{P}}}_{i}$. Then the loss
function can be defined as :

$\displaystyle\log{p_{\theta}}({\tilde{\bf{x}}^{i}})\geq{{\mathbb{E}}_{{q_{{\omega^{i}}}}({\bf{z}}|{{\tilde{\bf{x}}}^{i}})}}[\log{p_{\theta}}({{\tilde{\bf{x}}}^{i}}|{\bf{z}})]+KL[{q_{{\omega^{i}}}}({\bf{z}}|{{\tilde{\bf{x}}}^{i}})||p({\bf{z}})]$
(90)

where ${\tilde{\bf x}}^{i}$ is sampled from
${\mathbb{P}}^{i}\otimes{\tilde{\mathcal{P}}}_{i}$. Based on this objective
function, we analyze how the Self-Supervised VAEs can achieve an optimal
solution in lifelong generative modelling.

###### Proposition 3

Let
${\tilde{\mathcal{P}}}_{(1:t)}=\\{{\tilde{\mathcal{P}}}_{1}\otimes{\tilde{\mathcal{P}}}_{2},\dots,\otimes{\tilde{\mathcal{P}}}_{t}\\}$
be a mixture distribution. The Self-Supervised VAEs approximates
${\tilde{\mathcal{P}}}_{(1:t)}$ exactly when VAEs approximates each target
distribution exactly in each task learning.

When satisfying Proposition 3, the GB from Theorem 2 in the paper is redefined
as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{{{\mathcal{R}}_{{\mathcal{P}_{i}}}}}\left({h,{f_{{\mathcal{P}_{i}}}}}\right)$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{{\mathcal{R}}_{{\tilde{\mathcal{P}}_{i}}}}}\left(h,h^{*}_{\tilde{\mathcal{P}}_{i}}\right)\\}+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}}},{{\tilde{\mathcal{P}}_{(1:t)}}}\right).$
(91)

From eq.(91), the lifelong learning problem is transformed into the multiple
source-target generalization problem. In the following, we assume that the
random variables $\\{{\bf x}^{T}_{1},\dots,{\bf x}^{T}_{t}\\}$ are sampled
from $\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ and $\\{{\bf
x}^{S}_{1},\dots,{\bf x}^{S}_{t}\\}$ are sampled from
$\\{{\tilde{\mathcal{P}}}_{1},\dots,{\tilde{\mathcal{P}}}_{t}\\}$. We can
define the KL divergence as follows :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathbb{E}}_{{{\mathcal{P}}_{i}}}}KL\left({p\left({{\bf{z}}|{\bf{x}}_{i}^{T}}\right)||p\left({\bf{z}}\right)}\right)}\Big{\\}}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathbb{E}}_{{{\tilde{\mathcal{P}}}_{i}}}}KL\left({p\left({{\bf{z}}|{\bf{x}}_{i}^{S}}\right)||p\left({\bf{z}}\right)}\right)}\Big{\\}}$
(92)
$\displaystyle+\left|{\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathbb{E}}_{{{\mathcal{P}}_{i}}}}KL\left({p\left({{\bf{z}}|{\bf{x}}_{i}^{T}}\right)||p\left({\bf{z}}\right)}\right)}\Big{\\}}-\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{{{\mathbb{E}}_{{{\tilde{\mathcal{P}}}_{i}}}}KL\left({p\left({{\bf{z}}|{\bf{x}}_{i}^{S}}\right)||p\left({\bf{z}}\right)}\right)}\Big{\\}}}\right|$

where the latest term in RHS of Eq. (92) is defined as $D_{diff}({\bf
x}^{T}_{i},{\bf x}^{S}_{i})$. According to the definition of NLL, we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-\log
p\left({\bf x}^{T}_{i}\right)\\}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}\\}.$ (93)

In the following, we consider to sum up Eq. (92) and Eq. (91), resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}\\}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{S}_{i};h\right)}+D_{diff}\left({\bf x}^{T}_{i},{\bf
x}^{S}_{i}\right)\\}\Big{\\}}+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}}},{{\tilde{\mathcal{P}}_{(1:t)}}}\right).$
(94)

Eq. (94) is derived based on the formulation of ELBO (See details in Lemma 1
of the paper). It can be observed that LHS of Eq. (94) is RHS of Eq. (93) and
we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\log
p\left({\bf x}^{T}_{i}\right)}\\}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}\Big{\\{}-{\mathcal{L}_{ELBO}\left({\bf
x}^{S}_{i};h\right)}+D_{diff}\left({\bf x}^{T}_{i},{\bf
x}^{S}_{i}\right)\Big{\\}}+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}}},{{\tilde{\mathcal{P}}_{(1:t)}}}\right).$
(95)

Eq. (95) explicitly defines a GB for the VAE mode that is trained on all
training sets or the generator distribution that approximates the distribution
of all training samples, exactly.

Proof.

We begin with the first task in which $\mathcal{M}$ is trained on
$\tilde{\mathcal{P}}_{1}$ and the optimal generator distribution
$\mathbb{P}^{1}=\tilde{\mathcal{P}}_{1}$. In the second task learning,
${\mathcal{M}}^{1}$ is trained on the mixture distribution
$\mathbb{P}^{1}\otimes{\tilde{\mathcal{P}}}_{2}$. Since
$\mathbb{P}^{1}=\tilde{\mathcal{P}}_{1}$, we replace
$\mathbb{P}^{1}\otimes{\tilde{\mathcal{P}}}_{2}$ by
${\tilde{\mathcal{P}}}_{1}\otimes{\tilde{\mathcal{P}}}_{2}$ and the optimal
generator distribution
$\mathbb{P}^{2}={\tilde{\mathcal{P}}}_{1}\otimes{\tilde{\mathcal{P}}}_{2}$. By
using the mathematical recursion, we have
$\mathbb{P}^{t}={\tilde{\mathcal{P}}}_{(1:t)}$ if VAEs approximates the target
distribution exactly in each task learning.

###### Proposition 4

For a given sequence of tasks
$\\{{\mathcal{T}}_{1},\dots,{\mathcal{T}}_{t}\\}$, the sample log-likelihood
of a single model $\bf{M}$ evaluated on the target distribution
$\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ during the $t$-th task
learning is bounded by the accumulated errors and the generalization errors.

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{t}^{T}}\right)}\right\\}}$
$\displaystyle\leq{\mathbb{E}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}^{t}};h\right)}\\}+\underbrace{\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)}_{\text{Accumulated
error}}$ (96)
$\displaystyle+\underbrace{\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)}_{\text{Generalization
error}}$

We have the accumulated error ${\rm Err}^{c}$ and the generalization error
term ${\rm Err}^{h}$ :

$\displaystyle{\rm
Err}^{c}=\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)$
(97)

$\displaystyle{\rm
Err}^{h}=\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)$
(98)

Then, Eq. (96) is rewritten as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{t}^{T}}\right)}\right\\}}$
$\displaystyle\leq{\mathbb{E}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}^{t}};h\right)}\\}+{\rm
Err}^{c}+{\rm Err}^{h}$ (99)

Eq. (99) explicitly defines a GB for the negative sample log-likelihood for
$\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ where ${\rm Err}^{c}$ and
${\rm Err}^{h}$ are two error terms. We show that the gap on this GB depends
not only on ${\rm Err}^{c}$ but also on ${\rm Err}^{h}$.

Proof.

Since the model $\mathcal{M}$ is trained on a sequence of distributions
$\\{{\tilde{\mathcal{P}}},\dots,{\tilde{\mathcal{P}}}_{t}\\}$, we would like
to derive the GB between the model distribution and
$\\{{\tilde{\mathcal{P}}},\dots,{\tilde{\mathcal{P}}}_{t}\\}$ as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{{{\mathcal{R}}_{{\tilde{\mathcal{P}}_{i}}}}}\left({h,{f_{{\tilde{\mathcal{P}}_{i}}}}}\right)$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right),$
(100)

where ${\tilde{\mathcal{P}}}_{(1:t)}$ represents the mixture distribution
$\\{{\tilde{\mathcal{P}}}_{1}\otimes\dots\otimes{\tilde{\mathcal{P}}}_{t}\\}$.
As similar to the proof from Lemma 1 of the paper, we add the KL divergence
term in both sides in eq.(100), resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{{\mathcal{R}}_{{\tilde{\mathcal{P}}_{i}}}}}\left({h,{f_{{\tilde{\mathcal{P}}_{i}}}}}\right)+KL\left({q\left({{\bf{z}}|{\bf{x}}_{i}^{S}}\right)||p\left({\bf{z}}\right)}\right)\\}$
$\displaystyle\leq{{\mathcal{R}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\left({h,h_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}^{*}}\right)+{{\mathbb{E}}_{{{\mathbb{P}}^{t-1}\otimes{\tilde{\mathcal{P}}}_{t}}}}KL(p({\bf{z}}|{\bf{\tilde{x}}}^{t})||p({\bf{z}}))$
(101)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}\\{+{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right),$

We can rewrite Eq.(101) According to ELBO, resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{S}};h\right)\\}$
$\displaystyle\leq{\mathbb{E}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}^{t}};h\right)}\\}+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right),$
(102)

Secondly, we consider an ideal situation in which the model $\mathcal{M}$
approximates the learning target distribution exactly in each task learning.
Then, we can derive the GB for the risk between
$\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ and
$\\{{\tilde{\mathcal{P}}}_{1},\dots,{\tilde{\mathcal{P}}}_{t}\\}$ :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{T}};h\right)\\}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{S}};h\right)}\\}+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right),$
(103)

We also know that the left hand side of Eq. (103) is a bound for
$\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{t}^{T}}\right)}\right\\}}$, expressed as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{t}^{T}}\right)}\right\\}}\leq\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{\mathcal{P}_{i}}\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{T}};h\right)\\}$
(104)

By comparing eq.(104) and eq.(103), we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{t}^{T}}\right)}\right\\}}\leq\frac{1}{t}\sum\limits_{i=1}^{t}\\{{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{S}};h\right)}\\}\\}+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)$
(105)

Then we add the last two terms in the right hand side of eq.(103) to both
sides of eq.(102), resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}\\{{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{i}^{S}};h\right)\\}\\}+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)\leq{\mathbb{E}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}^{t}};h\right)}\\}$
(106)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)$

It can be observed that the right hand side of eq.(105) is equal to the left
hand side of eq.(106). By comparing eq.(105) and eq.(106), we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}{\left\\{{-\log
p\left({{\bf{x}}_{i}^{T}}\right)}\right\\}}$
$\displaystyle\leq{\mathbb{E}}_{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}\\{{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}^{t}};h\right)}\\}+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}^{t}}\right)}\\}$
(107)
$\displaystyle+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{{{\mathbb{P}}^{t-1}}\otimes{{\tilde{\mathcal{P}}}_{t}}}}\right)+\frac{1}{t}\sum\limits_{i=1}^{t}\\{{{D_{diff}}\left({{\bf{x}}_{i}^{S},{\bf{x}}_{i}^{T}}\right)}\\}+{\mathcal{R}}_{A}\left({{\tilde{\mathcal{P}}_{(1:t)}},{\mathcal{P}}_{(1:t)}}\right)$

This proves Proposition 4.

###### Proposition 5

For a given mixture model $\bf{M}$, The sample log-likelihood of $\bf{M}$
evaluated on the target distribution
$\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ during the $t$-th task
learning is bounded by :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
(108)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal P}}_{(1:t)}}}\right)$

From eq.(108), it can be observed that the generalization error term
$\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal P}}_{(1:t)}}}\right)$ is the same for both the
single model $\mathcal{M}$ and the mixture model $\bf{M}$. This means that the
mixture model can obtain a tight GB by reducing the accumulated error term
during lifelong learning.

Proof.

Firstly, we consider to take
$\\{{\tilde{\mathcal{P}}_{1}},\dots,{\tilde{\mathcal{P}}_{t}}\\}$ as the
target distribution and we can derive the GB for $\bf{M}$ as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C|}{\\{{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}\left({h,{f_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}}\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{{\mathcal{R}}_{\tilde{\mathcal{P}}_{a(i,j)}}}\left({h,{f_{\tilde{\mathcal{P}}_{a(i,j)}}}}\right)\\}}}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h,{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}}\right)\right.$
(109)
$\displaystyle\left.+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\\{{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}\left({h,{f_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}}\right)\\}}$

We then add the KL divergence in both sides of eq.(109), resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C|}{\\{{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}\left({h,{f_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}}\right)+KL\left({q\left({{\bf{z}}|{\bf{x}}_{{a_{i}}}^{S}}\right)||p\left({\bf{z}}\right)}\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{{\mathcal{R}}_{\tilde{\mathcal{P}}_{a(i,j)}}}\left({h,{f_{\tilde{\mathcal{P}}_{a(i,j)}}}}\right)+KL\left({q\left({{\bf{z}}|{\bf{x}}_{a(i,j)}^{S}}\right)||p\left({\bf{z}}\right)}\right)\\}}}\leq$
(110)
$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}\left\\{{{\mathcal{R}}_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}\left({h,{f_{{\mathbb{P}}_{a(i,j)}^{c(i,j)}}}}\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)+KL\left({q\left({{\bf{z}}|{\bf{x}}_{a(i,j)}^{t}}\right)||p\left({\bf{z}}\right)}\right)\right\\}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\\{{{\mathcal{R}}_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}\left({h,{f_{{{\tilde{\mathcal{P}}}_{{a_{i}}}}}}}\right)\\}+KL\left({q\left({{\bf{z}}|{\bf{x}}_{a_{i}}^{S}}\right)||p\left({\bf{z}}\right)}\right)}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$

We rewrite eq.(110) as the ELBO from as :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a(i,j)}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{S}};h\right)\\}}}\leq$
(111)
$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$

In the following, we consider to take
$\\{{\mathcal{P}}_{1},\dots,{\mathcal{P}}_{t}\\}$ as the target distribution
and $\\{{\tilde{\mathcal{P}}}_{1},\dots,{\tilde{\mathcal{P}}}_{t}\\}$ as the
source distribution. Then we have :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\tilde{\mathcal{P}}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{S}_{i};h\right)}+D_{diff}\left({\bf x}^{T}_{i},{\bf
x}^{S}_{i}\right)\\}+{\mathcal{R}}_{A}\left({{\mathcal{P}_{(1:t)}}},{{\tilde{\mathcal{P}}_{(1:t)}}}\right).$
(112)

We then add the latest two term of RHS of Eq. (112) into both sides of Eq.
(111), resulting in :

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal
P}}_{(1:t)}}}\right)\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a(i,j)}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{S}};h\right)\\}}}\leq$
(113)
$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal P}}_{(1:t)}}}\right)$

It can be noted that LHS of Eq. (113) is equal to RHS of Eq. (112), therefore,
RHS of Eq. (113) is also an upper bound to LHS of Eq. (112).

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\mathcal{L}_{ELBO}\left({\bf
x}^{T}_{i};h\right)}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
(114)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal P}}_{(1:t)}}}\right)$

Eventually, we know that LHS of Eq. (114) is an upper bound to NLL, then RHS
of Eq. (114) is also an upper bound to NLL.

$\displaystyle\frac{1}{t}\sum\limits_{i=1}^{t}{\mathbb{E}}_{{\mathcal{P}}_{i}}\\{-{\log
p\left({\bf x}^{T}_{i}\right)}$
$\displaystyle\leq\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}\sum\limits_{j=1}^{{\tilde{a}}_{i}}{\mathbb{E}}_{{\mathbb{P}}^{c(i,j)}_{a(i,j)}}\left\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a(i,j)}}^{t}};h\right)+{\mathcal{R}}_{A}\left({{{\tilde{\mathcal{P}}}_{a(i,j)}},{\mathbb{P}}_{a(i,j)}^{c(i,j)}}\right)\right\\}$
(115)
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{|C|}{\mathbb{E}}_{{\tilde{\mathcal{P}}_{a_{i}}}}{\\{-{\mathcal{L}_{ELBO}}\left({{\bf{x}}_{{a_{i}}}^{S}};h\right)\\}}+\frac{1}{t}\sum\limits_{i=1}^{|C^{\prime}|}{\sum\limits_{j=1}^{{{\tilde{a}}_{i}}}{\\{{D_{diff}}\left({{\bf{x}}_{a(i,j)}^{S},{\bf{x}}_{a(i,j)}^{t}}\right)\\}}}$
$\displaystyle+\frac{1}{t}\sum\limits_{i=1}^{t}{{D_{diff}}\left({{\bf{x}}_{i}^{T},{\bf{x}}_{i}^{S}}\right)}+{{\mathcal{R}}_{A}}\left({{{\cal
P}_{(1:t)}},{{\widetilde{\cal P}}_{(1:t)}}}\right)$

This proves Proposition 5.

###### Proposition 6

Let $\mathcal{P}_{i}$ be a target distribution and let ${\mathbb{P}}^{i}$ be
an approximation distribution that partly overlaps with $\mathcal{P}_{i}$. Let
$\\{\theta^{*},\omega^{*}\\}$ be the optimal parameters for a VAE model
trained on samples ${\bf x}^{\prime}$ drawn from ${\mathbb{P}}^{i}$. Then
${\mathcal{L}}_{ELBO}(\\{\theta^{*},\omega^{*}\\},{\bf x})$ is not a tight
ELBO for $\log p_{\theta}({\bf x})$ by using the model’s parameters
$\\{\theta^{*},\omega^{*}\\}$.

Proof

Let $\\{\theta^{\prime},\omega^{\prime}\\}$ be an optimal parameter set for a
VAE model trained on real samples ${\bf x}$ drawn from ${\mathcal{P}}_{i}$ and
we have ${\mathcal{L}}_{ELBO}(\\{\theta^{\prime},\omega^{\prime}\\},{\bf
x})>{\mathcal{L}}_{ELBO}(\\{{\tilde{\theta}},{\tilde{\omega}}\\},{\bf
x}),{\tilde{\theta}}\neq\theta^{\prime},{\tilde{\omega}}\neq\omega^{\prime}$.
Since the model’s parameters $\\{\theta^{*},\omega^{*}\\}$ are optimized by
maximizing ELBO on samples draw from ${\mathbb{P}}_{i}$ and are not equal to
$\\{\theta^{\prime},\omega^{\prime}\\}$. Therefore we have
${\mathcal{L}}_{ELBO}(\\{\theta^{\prime},\omega^{\prime}\\},{\bf
x})>{\mathcal{L}}_{ELBO}(\\{{\theta^{*}},{\omega^{*}}\\},{\bf x})$ and
${\mathcal{L}}_{ELBO}(\\{{\theta^{*}},{\omega^{*}}\\},{\bf x})$ is not a tight
ELBO to the real sample log-likelihood $\log p_{\theta}({\bf x})$.

Figure 1: The target risk for a single model under MNIST, Fashion, IFashion
lifelong learning.

In order to investigate the results for Proposition 6, we train a single model
with GR under MNIST, Fashion, IFashion (MIF) lifelong learning (The detailed
setting is provided in Appendix L.2). We also train a single model that can
access the real training samples from previous tasks during LLL and this model
can be seen to be optimal for the real sample log-likelihood. We evaluate the
average target risk (estimated by the negative ELBO on all testing samples)
for each training epoch and the results are reported in Fig. 1 where ”Single”
and ”Single*” represent the single model with GR and the single model that can
access the real training samples from previous tasks, respectively. It notes
that ”Single” is trained on the approximation distribution instead of the
distributon of real training sets. It shows that, as learning more tasks, a
VAE model with GR process gradually loses performance on the real data
distribution. This demonstrates that the optimal parameters
$\\{\theta^{*},\omega^{*}\\}$ achieved from the approximation distribution are
not optimal for the real sample log-likelihood, which empirically proves
Proposition 6.

##  Appendix J J The main objective function, algorithm and pipeline for DEGM

In this section, we introduce additional information for the proposed DEGM.

###  J.1 Derivation of ${\mathcal{L}_{MELBO}}$

As illustrated in Fig.2 from the paper, where we show the graph structure
implementing DEGM, we have $K$ number of basic nodes in DEGM. When building a
new specific node for learning a new task, we derive the main objective
function showing as follows.

Theorem 4.

When a new specific node ($(t+1)$-th node) is built for learning the
$(t+1)$-th task. This specific node connected with all basic nodes can be seen
as a sub-graph model which can be trained by a valid lower bound (ELBO).

Proof.

We start by considering the KL divergence Doersch (2016) :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]={\mathbb{E}_{z\sim
Q\left({\bf{z}}\right)}}\left[{\log Q\left({\bf{z}}\right)-\log
p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$ (116)

where $Q({\bf z})$ is the variational distribution. We can rewrite the above
equation as :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$
$\displaystyle={\mathbb{E}_{z\sim Q\left({\bf{z}}\right)}}\left[{\log
Q\left({\bf{z}}\right)-\log p\left({{\bf{x}}|{\bf{z}}}\right)-\log
p\left(z\right)}\right]+\log p\left({\bf{x}}\right)$ (117)

And we have :

$\displaystyle\log
p\left({\bf{x}}\right)-{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$
$\displaystyle={\mathbb{E}_{z\sim Q\left({\bf{z}}\right)}}\left[{\log
p\left({{\bf{x}}|{\bf{z}}}\right)}\right]-{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({\bf{z}}\right)}\right]$
(118)

where the right hand side is also called evidence lower bound (ELBO). We
particularly focus on the KL term
${{KL}}\left[{Q\left({\bf{z}}\right)||p\left({\bf{z}}\right)}\right]$ which
has the following from :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({\bf{z}}\right)}\right]=\int{q\left({\bf{z}}\right)}\log\frac{{q\left({\bf{z}}\right)}}{{p\left({\bf{z}}\right)}}d{\bf{z}}$
(119)

where $q({\bf z})$ is the density of $Q({\bf x})$. Since we have $K$
components and we consider the $q({\bf z})$ to be mixture density function
$q\left({\bf{z}}\right)=\sum\nolimits_{i=1}^{w}{{\pi_{i}}{q_{{{\tilde{\omega}}_{\mathcal{SG}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}$
where $\pi_{i}$ is the weight. We then rewrite Eq. (119) as :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({\bf{z}}\right)}\right]$
$\displaystyle=\int{\left({\sum\nolimits_{i=1}^{K}{{\pi_{i}}{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}\right)}\log\frac{{q\left({\bf{z}}\right)}}{{p\left({\bf{z}}\right)}}d{\bf{z}}$
(120)
$\displaystyle=\sum\nolimits_{i=1}^{K}{{\pi_{i}}}\int{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)\log\frac{{q\left({\bf{z}}\right)}}{{p\left({\bf{z}}\right)}}d{\bf{z}}}$

We then add the term
${q_{{{\tilde{\omega}}_{\mathcal{SG}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)/{q_{{{\tilde{\omega}}_{\mathcal{SG}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)$
to Eq. (120), resulting in :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({\bf{z}}\right)}\right]$
$\displaystyle=\sum\nolimits_{i=1}^{K}{{\pi_{i}}}\int{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)\log\\{\frac{{q\left({\bf{z}}\right)}}{{p\left({\bf{z}}\right)}}\times\frac{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}\\}d{\bf{z}}$
(121)

We rewrite the above equation as :

$\displaystyle\sum\nolimits_{i=1}^{K}{{\pi_{i}}}\int{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)\times\log\\{\frac{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}{{p\left({\bf{z}}\right)}}\times\frac{{q\left({\bf{z}}\right)}}{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}\\}d{\bf{z}}=$
(122)
$\displaystyle\sum\nolimits_{i=1}^{K}{{\pi_{i}}}\int{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)\times\left({\log\frac{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}{{p\left({\bf{z}}\right)}}+\log\frac{{q\left({\bf{z}}\right)}}{{{q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)}}}\right)d{\bf{z}}$

Then we can rewrite the above equation as KL terms :

$\displaystyle{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$
$\displaystyle=\sum\nolimits_{i=1}^{K}\left({{\pi_{i}}{{KL}}\left[{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)||p\left({\bf{z}}\right)}\right]}\right)$
(123)
$\displaystyle-\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)||p\left({\bf{z}}\right)}\right]}\right)}$

where
${q_{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}}\circ{q_{{{\omega^{\prime}}_{(t+1)}}}}\left({{\bf{z}}|{\bf{x}}}\right)$
is the density form of
${Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}$.
We take the expression of
$KL\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$
to Eq. (118), resulting in :

$\displaystyle\log p\left({\bf
x}\right)-KL\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]$
$\displaystyle={\mathbb{E}_{z\sim Q\left({\bf{z}}\right)}}\left[{\log
p\left({{\bf{x}}|{\bf{z}}}\right)}\right]-\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}||p({\bf
z})\right]}\right)}$ (124)
$\displaystyle+\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}||Q\left({\bf{z}}\right)}\right]}\right)}$

where we move the last term of the right hand side to the left hand side,
resulting in :

$\displaystyle\log
p\left({\bf{x}}\right)-{{KL}}\left[{Q\left({\bf{z}}\right)||p\left({{\bf{z}}|{\bf{x}}}\right)}\right]-\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}||Q\left({\bf{z}}\right)}\right]}\right)}=$
(125) $\displaystyle{\mathbb{E}_{z\sim Q\left({\bf{z}}\right)}}\left[{\log
p\left({{\bf{x}}|{\bf{z}}}\right)}\right]-\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}||p\left({\bf{z}}\right)}\right]}\right)}$

We know that KL terms are always larger or equal to zero and the right hand
side of Eq. (125) is a lower bound to the sample log-likelihood. Finally, the
objective function for training DEGM is to maximize this lower bound :

$\displaystyle{\mathbb{E}_{z\sim Q\left({\bf{z}}\right)}}\left[{\log
p\left({{\bf{x}}|{\bf{z}}}\right)}\right]-\sum\nolimits_{i=1}^{K}{\left({{\pi_{i}}{{KL}}\left[{{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}||p\left({\bf{z}}\right)}\right]}\right)}$
(126)

where in the first term, ${\bf z}$ is sampled from the mixture distribution
$Q\left({\bf{z}}\right)=\sum\nolimits_{i=1}^{K}{{\pi_{i}}{Q_{{{{\tilde{\omega}}_{\mathcal{GI}(i)}}}\,\circ\,{\omega^{\prime}}_{(t+1)}}\left({{\bf{z}}|{\bf{x}}}\right)}}$.
In order to reuse the parameters and transferable information from all basic
nodes, we consider the following implementations for the decoder. When
calculating the first term, we build an input layer on the top of the decoder.
This input layer is used as the identity function and is connected with each
sub-decoder $g_{{\tilde{\theta}}_{\mathcal{GI}(i)}}(\bf z)$ of each basic
node, represented as one layer or module in a single decoder. Then we obtain
the intermediate data representation $\tilde{\bf
x}=\sum\nolimits_{i=1}^{K}\pi_{i}g_{{{\tilde{\theta}}_{{\mathcal{GI}}(i)}}}({\bf
z})$ which is used as the input for the newly created sub-decoder
$g_{\theta^{\prime}_{(t+1)}}(\tilde{\bf x})$. Therefore, we treat the
intermediate data representation ${\tilde{\bf x}}$ as the information between
two layers in a single decoder and do not consider ${\tilde{\bf x}}$ to be the
random variable. In this case we rewrite the decoding distribution $p({\bf
x}|{\bf z})$ as :

$p({\bf x}|{\bf
z})={p_{{\theta^{\prime}_{(t+1)}}\circ\\{{\tilde{\theta}}_{\mathcal{GI}(1)},\dots,{\tilde{\theta}}_{\mathcal{GI}(K)}\\}}}\left({\bf
x}\,|\,{\bf z}\right)$

. Therefore, we rewrite Eq. (126) as :

$\displaystyle\mathcal{L}_{MELBO}\left({\bf x};\mathcal{M}_{(t+1)}\right)$
$\displaystyle={\mathbb{E}_{Q(\bf{z})}}\left[{\log{p_{{\theta^{\prime}_{(t+1)}}\circ\\{{\tilde{\theta}}_{\mathcal{GI}(1)},\dots,{\tilde{\theta}}_{\mathcal{GI}(K)}\\}}}\left({{\bf{x}}\mid{\bf
z}}\right)}\right]$ (127)
$\displaystyle-\sum\limits_{i=1}^{K}\left(\pi_{i}{{{KL}}\left[{{q_{{\tilde{\omega}}_{{\cal
G}{\cal I}(i)}}\circ
q_{{\omega^{\prime}}_{(t+1)}}}\left({{{\bf{z}}}\mid{\bf{x}}}\right)\mid\mid
p\left({\bf{z}}_{i}\right)}\right]}\right)$

Eq.(127) shows that we can implement existing variational inference mechanisms
such as using a more expressive posterior Maal$\o$e et al. (2016); Kingma et
al. (2016); Sobolev and Vetrov (2019), important sampling Burda et al. (2015)
and Semi-Implicit Variational Inference Yin and Zhou (2018); Molchanov et al.
(2019) in our framework to enable for lifelong learning.

Input: All training databases

Output: The model’s parameters

1for  _$i <taskCount$_ do

2 if  _$i==1$_ then

3 Build a basic component $B_{1}$ and added to ${\mathcal{G}}$ ;

4 $isBasic=True$;

5

6 end if

7

8 for  _$index <batchCount$_ do

9 ${\bf x}_{batch}\sim Q^{S}_{i}$ a batch of images sampled from the $i$-th
task;

10 if  _isBasic == True_ then

11 Update $\\{{\omega_{i}},{\theta_{i}}\\}$ by maximizing
${\mathcal{L}}_{ELBO}({\bf x}_{batch};{\mathcal{M}_{i}})$;

12

13 end if

14 else

15 Update $\\{{\omega^{\prime}_{i}},{\theta^{\prime}_{i}}\\}$ by maximizing
${\mathcal{L}}_{MELBO}({\bf x}_{batch};{\mathcal{M}_{i}})$;

16

17 end if

18

19 end for

20

Expansion mechanism

;

21 ${\bf x}_{new}\sim Q_{i+1}^{S}$ collect 1000 number of samples from the
next task;

22

Calculate the importance of each basic node to the new task

;

23 for  _$k <K$_ do

24 $ks_{k}=\left|\mathcal{L}_{ELBO}(B_{k})-{\mathbb{E}}_{{\bf x}\sim{\bf
x}_{new}}\mathcal{L}_{ELBO}({\bf x};B_{k})\right|$;

25

26 end for

27 $\mathcal{K}=\\{ks_{1},\dots,ks_{K}\\}$;

28 if  _$\min\\{{\mathcal{K}}\\}\leq\tau$_ then

29

The construction of the basic node

;

30 $V(i+1,\mathcal{GI}(j))=0$, $j=1,\dots,K$;

31 Build a basic component $B_{(K+1)}$ and added to ${\mathcal{G}}$ ;

32 $isBasic=True$;

33 $K=K+1$;

34

35 end if

36 else

37

The construction of the sub-graph structure

;

38
$V(i+1,\mathcal{GI}(i))=(w^{*}-ks_{i})/\sum\nolimits_{j=1}^{K}(w^{*}-ks_{j})$,
$w^{*}=\sum\nolimits_{j=1}^{K}ks_{j}$, $i=1,\dots,K$;

39
$\sum\nolimits_{j=1}^{K}\pi_{i}f_{{{\tilde{\omega}}_{{\mathcal{GI}(j)}}}}\circ{f_{{{\omega^{\prime}}_{(i+1)}}}}({\bf
x})$ build the inference model ;

40 ${\bf z}=\sum\nolimits_{j=1}^{K}\pi_{j}{\bf z}_{j}$ obtain the latent
representation;

41 $\tilde{\bf
x}=\sum\nolimits_{j=1}^{K}\pi_{j}g_{{\tilde{\theta}}_{{\mathcal{GI}(j)}}}({\bf
z})$ obtain the intermediate representations from sub-decoder of all basic
nodes;

42 $g_{\theta^{\prime}_{(i+1)}}(\tilde{\bf x})$ build a sub-decoder ;

43 Add ${\mathcal{M}}_{i+1}$ in ${\mathcal{S}}$;

44 $isBasic=False$;

45

46 end if

47

48 end for

Algorithm 1 The training algorithm for DEGM

###  J.2 The pipeline and algorithm of DEGM

In order to conveniently understand the learning procedure of DEGM, we show
the processing pipeline in Figure 2. Once the $t$-th task was finished, we get
a group of samples, denoted as ${\tilde{Q}}_{(t+1)}^{S}$ from
$\mathcal{T}_{t+1}$. Then we calculate the edge values, represented by
$\\{\pi_{1},\dots,\pi_{K}\\}$. If $\min({\mathcal{K}})>\tau$, then we build a
basic node which will be added into $\mathcal{G}$ for learning the $(t+1)$-th
task, otherwise, we build a specific node based on members of $\mathcal{G}$
for learning the $(t+1)$-th task. From this pipeline, we observe that the edge
values regularize the latent representations and the intermediate data
representations in the inference and generation process, respectively. We also
provide the pseudocode in Algorithm 1.

Figure 2: The processing pipeline of the proposed DEGM.

###  J.3 The component selection for model evaluation during the testing phase

Let us consider DEGM model with $t$ trained nodes after LLL. We introduce the
cluster assignment ${\bf{u}}$ in the DEGM and the probability density of DEGM
on $n$ samples is represented by :

$\displaystyle p\left({\bf
x}\right)=\prod\nolimits_{i=1}^{n}\sum\nolimits_{j=1}^{t}p\left({\bf
x}_{i}\mid{\bf u}_{(i,j)}\right)p\left({\bf u}_{(i,j)}\right),\;{\bf
u}_{(i,j)}\in\\{0,1\\}\,.$ (128)

We particularly focus on the posterior $p({{\bf{u}}_{(i,j)}}|{{\bf{x}}_{i}})$
which can be rewritten by the Bayes’ theorem :

$\displaystyle p\left({\bf u}_{(i,j)}\,|\,{\bf x}_{i}\right)$
$\displaystyle=\frac{p\left({\bf x}_{i}\,|\,{\bf u}_{(i,j)}\right)p\left({\bf
u}_{(i,j)}\right)}{p\left({\bf x}_{i}\right)}=\frac{p\left({\bf
x}_{i}\,|\,{\bf
u}_{(i,j)}\right)p\left({{{\bf{u}}_{(i,j)}}}\right)}{{\sum\nolimits_{k=1}^{t}p\left({\bf
u}_{(i,k)}\right)p\left({\bf x}_{i}\,|\,{\bf u}_{(i,k)}\right)}},$ (129)

where the prior is $p({\bf u}_{(i,j)})=1/t$. Since $\log_{a}$ is a function
monotonic increasing if $a>1$, we can replace each $p({\bf x}_{i}\,|\,{\bf
u}_{(i,j)})$ by $\log p({\bf x}_{i}\,|\,{\bf u}_{(i,j)})$ estimated by
$\mathcal{L}_{ELBO}(,{\bf x}_{i};{\cal M}_{j})$ for the elements of
$\mathcal{G}$ on the sample ${\bf x}$, and by $\mathcal{L}_{MELBO}({\bf
x}_{i};\mathcal{M}_{j})$ from (127) for elements of $\mathcal{S}$, as
described in Section 5.3 of the paper. This selection process can allow DEGM
to infer a related component without having task labels.

###  J.4 Limitations of the proposed theoretical framework

The primary limitation of the proposed theoretical framework is that it only
supports the Gaussian decoder since ELBO can be decomposed by a negative
reconstruction error term ($\mathcal{L}$ loss) and the KL divergence term. Our
future work will focus on extending the proposed theoretical framework for
other types of distributions for the decoder such as Bernoulli distribution.

One potential negative societal impacts of our work is that the proposed DEGM
would cause data leakage. For example, in many lifelong learning tasks, one
potential requirement is to allow the data for each task to appear once for
privacy protection.

###  J.5 The difference between related work and DEGM

In this section, we discuss the difference between several related works and
DEGM. Firstly, our work is related to Rao et al. (2019) that expands its
network architectures on the inference models only. Rao et al. (2019) also
uses the generative replay mechanism, which would lead to the degenerated
performance on previously learnt samples due to the network forgetting. This
is not the case in DEGM since we do not use the generative replay mechanism.
To compare with CN-DPM Lee et al. (2020) that also uses the expansion
mechanism only, DEGM can reuse prior knowledge when learning novel samples by
forming a graph structure in which the independent basic nodes are used to
model entirely different tasks while the specific nodes are built based on the
information flow from basic nodes for learning related tasks. However, in both
studies Lee et al. (2020) and Rao et al. (2019) perform task-free continual
learning, which is not the main task in our work since we address a more
general lifelong learning problem where the task boundaries are provided only
at the training stage.

Our work is more related to LIMix Ye and Bors (2021e) that introduces a
mixture model with an infinite number of components. LIMix has only a single
shared encoder and decoder, which has limitations when learning several
entirely different tasks since the shared parameters preserve the knowledge
from the first task only. DEGM can have several basic nodes which aim to learn
several entirely different tasks and can be used for transferring knowledge
when learning novel tasks. Additionally, LIMix introduces a new theoretical
analysis for the forgetting behaviour of the model under lifelong learning,
which, however, is only applied in the supervised learning task. The proposed
theoretical framework in our work studies the forgetting behaviour of the
model under the unsupervised lifelong learning setting.

##  Appendix K K Unsupervised lifelong learning benchmark

###  K.1 Network architecture and hyperaprameter setting

For the general network architecture for VAEs, the inference and generator
models are implemented by two fully connected networks where each network
contains two layers and each layer has 200 hidden units. We also extend VAEs
by using two stochastic layers where the latent dimension for the first and
second stochastic layer is 100 and 50, respectively. The VAE with two
stochastic layers is named as ELBO-GR*. If VAE uses only a single stochastic
layer, then the latent dimension is 100. When the important sampling is used
in various models such as VAEs, we call it as IWELBO-GR-$K^{\prime}$ where
$K^{\prime}$ represents the number of weighted samples used in the objective
function. The GPU used for the experiment is GeForce GTX 1080. The operating
system is Ubuntu 18.04.5.

For the DEGM model, the inference model and the generator in DEGM are
implemented by four sub-models and each sub-model has only one layer with 200
units. Therefore, the inference and generator model in each specific node are
two sub-models built based on all basic components members. As similar to a
single model, DEGM can be extended with the important sampling framework. For
implementation, if a basic component is used for training, we can easily use
the IWELBO objective function (See Eq. (54) in Appendix F.1). If a specific
node is used for training, we rewrite the objective function as the IWELBO
bound form as :

$\displaystyle{{\mathcal{L}}_{{MELBO}_{K^{\prime}}}}\left({\bf{x}};{\mathcal{M}}\right)={{\mathbb{E}}_{{{\bf{z}}_{1}},..,{{\bf{z}}_{K^{\prime}}}\sim
Q\left({{\bf{z}}}\right)}}\left[{\log\frac{1}{K^{\prime}}\sum\limits_{i=1}^{K^{\prime}}{\frac{{p\left({{\bf{x}},{{\bf{z}}_{i}}}\right)}}{{Q\left({{{\bf{z}}_{i}}}\right)}}}}\right]$
(130)

The above equation is extended by the IWELBO bound with Eq. (118).

CN-DPM is originally designed for the task-free task. In order allow CN-DPM in
a more general lifelong unsupervised learning task, we implement a variant of
CN-DPM, namely CN-DPM*. We replace the Dirichlet process in CN-DPM* by
dynamically creating a new component after the task switch since the Dirichlet
process can allow CN-DPM to be a large model. As similar to CN-DPM, CN-DPM*
has a single shared module which is frozen after the first task learning and a
new component will be built based on this shared module. In order to set the
optimal setting for LIMix Ye and Bors (2021e), the number of components in
LMix matches the number of tasks while each component models a certain task
only.

We train various models using Adam Kingma and Ba (2015) optimization algorithm
with the learning rate of 0.0001. It notes that the decoder models Bernoulli
distribution and we use the binary cross entropy as the reconstruction error
term in ELBO. The batch size and the number of training epochs for each task
learning are 64 and 500, respectively.

###  K.2 Results on cross-domain setting (COFMI)

We search the threshold $\tau$ of 35-40 for Split MNIST/Fashion and 80-100 for
COFMI, respectively. DEGM based methods has four basic nodes and one specific
node after Split MNIST and Split Fashion lifelong learning setting.

We report the results under COFMI in Table 1. The weight matrix
${\mathcal{V}}$ of DEGM-IWELBO-50 under COFMI lifelong learning is shown in
Fig. 3. DEGM-IWELBO-50, DEGM-IWELBO-5, DEGM-ELBO has 4 basic and 1 specific
nodes after COFMI lifelong learning.

Figure 3: $\bf V$ of DEGM-IWVAE-50 under COFMI lifelong learning. ”C1” represents the first component and ”C4” is a basic node that connects previous three basic nodes (”C1”, ”C2”, ”C3”).  Mwthods | Caltech 101 | OMNIGLOT | Fashion | MNIST | IFashion | Average
---|---|---|---|---|---|---
ELBO-GR | -163.68 | -136.97 | -247,91 | -101.75 | -237,03 | -177,47
IWELBO-GR-50 | -153.65 | -131.37 | -243.62 | -97,29 | -234.58 | -172.10
IWELBO-GR-5 | -166.05 | -134.07 | -245.78 | -99.43 | -235.73 | -176.21
ELBO-GR* | -175.10 | -140.05 | -247.54 | -102.73 | -237.06 | -180.50
IWELBO-GR*-50 | -215.16 | -144.42 | -246.35 | -102.82 | -236.12 | -188.97
CN-DPM*-IWELBO-50 | -136.22 | -150.31 | -259.12 | -131.34 | -243.97 | -184.19
LIMix-IWELBO-50 | -137.32 | -150.79 | 258.69 | -131.25 | -243.50 | -184.32
DEGM-ELBO | -137.72 | -116.07 | -233.32 | -122.33 | -234.62 | -168.81
DEGM-IWELBO-50 | -133.93 | -112.15 | -230.63 | -107.44 | -232.17 |  -163.27
DEGM-IWELBO-5 | -137.18 | -113.25 | -231.88 | -109.21 | -233.43 | -164.99
Table 1: The estimation of the sample log-likelihood under COFMI lifelong
learning.

###  K.3 Results on Split MNIST and Split Fashion

In this section, we perform five independent runs for Split MNIST and Split
Fashion. After the training, we calculate the average result as well as the
standard deviation (std). The results for Split MNIST and Split Fashion are
reported in Table 2 and Table 3, respectively.

Mwthods | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Average
---|---|---|---|---|---|---
ELBO-GR | -97.97 | -98.16 | -98.70 | -97.91 | -98.39 | -98.23 $\pm$ (0.28 std)
IWELBO-GR-50 | -93.61 | -93.69 | -93.66 | -93.64 | -93.27 | -93.57 $\pm$ (0.15 std)
IWELBO-GR-5 | -95.96 | -95.68 | -95.95 | -95.68 | -95.73 | -95.80 $\pm$ (0.12 std)
ELBO-GR* | -97.92 | -98.79 | -97.55 | -98.45 | -99.11 | -98.36 $\pm$ (0.56 std)
IWELBO-GR*-50 | -91.15 | -91.33 | -91.24 | -91.23 | -91.18 | -91.23 $\pm$ (0.06 std)
CN-DPM*-IWELBO-50 | -95.94 | -96.35 | -95.90 | -95.74 | -95.63 | -95.91 $\pm$ (0.24 std)
LIMix-IWELBO-50 | -95.57 | -95.80 | -95.84 | -95.84 | -95.63 | -95.74 $\pm$ (0.11 std)
DEGM-ELBO | -93.85 | -87.03 | -93.15 | -93.71 | -93.51 | -92.25 $\pm$ (2.62 std)
DEGM-IWELBO-50 | -89.41 | -89.58 | -89.60 | -85.78 | -85.83 |  -88.04 $\pm$ (1.82 std)
DEGM-IWELBO-5 | -91.54 | -91.70 | -91.42 | -91.21 | -91.32 | -91.44 $\pm$ (0.17 std)
Table 2: Five independent runs for the estimation of the sample log-likelihood under Split MNIST setting. Mwthods | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Average
---|---|---|---|---|---|---
ELBO-GR | -241.00 | -240.38 | -240.26 | -240.82 | -240.44 | -240.58 $\pm$ (0.28 std)
IWELBO-GR-50 | -236.48 | -236.65 | -236.98 | -236.70 | -236.49 | -236.66 $\pm$ (0.18 std)
IWELBO-GR-5 | -237.86 | -238.19 | -238.31 | -238.00 | -238.00 | -238.08 $\pm$ (0.15 std)
ELBO-GR* | -242.54 | -242.27 | -244.76 | -242.38 | -247.61 | -243.91 $\pm$ (2.06 std)
IWELBO-GR*-50 | -236.71 | -236.77 | -237.04 | -236.88 | -237.08 | -236.90 $\pm$ (0.14 std)
CN-DPM*-IWELBO-50 | -237.60 | -237.13 | -237.81 | -237.38 | -237.43 | -237.47$\pm$ (0.22 std)
LIMix | -237.28 | -237.50 | -237.39 | -237.37 | -237.87 | -237.48$\pm$ (0.20 std)
DEGM-ELBO | -240.59 | -237.68 | -239.45 | -237.60 | -237.39 | -238.54 $\pm$ (1.26 std)
DEGM-IWELBO-50 | -234.17 | -232.72 | -234.13 | -233.15 | -234.63 |  -233.76 $\pm$ (0.70 std)
DEGM-IWELBO-5 | -236.51 | -234.46 | -236.59 | 235.00 | -237.08 | -235.93 $\pm$ (1.00 std)
Table 3: Five independent runs for the estimation of the sample log-likelihood
under Split Fashion setting.

##  Appendix L L Additional experiment results

###  L.1 Quantitative evaluation

Baselines:

We introduce several baselines and one baseline is the proposed mixture model
but we use few epochs (five epochs) for training when the new node is built on
other basic nodes, namely DEGM-1. The second baseline is dynamically creating
a new VAE to adapt to a new task, namely DEGM-2, which is a strong baseline
and would achieve the best performance for each task.

\-  Hyperparameter setting and GPUs The SGD optimization algorithm for each
model is using the Adam Kingma and Ba (2015), with a learning rate of 0.0002
while the other hyperparameters are set to their default. The GPU used for the
experiment is GeForce GTX 1080.

Setting.

To measure the reconstruction quality, we calculate Squared Loss (SL) as the
criterion. We train various models under MNIST LeCun et al. (1998), SVHN
Netzer et al. (2011), Fashion Xiao et al. (2017), InverseFashion (IFashion),
Rated MNIST (RMNIST) and Cifar10 Krizhevsky and Hinton (2009), (MSFIRC)
lifelong learning setting. We also evaluate the performance of various models
under CelebA Liu et al. (2015), CACD Chen et al. (2014), 3D-Chair Aubry et al.
(2014), Ommiglot Lake et al. (2015), ImageNet* Krizhevsky et al. (2012), Car
Yang et al. (2015), Zappos Yu and Grauman (2017), CUB Wah et al. (2010)
(CCCOSCZC) lifelong learning setting.

In the following, we provide details about the setting in the experiments. For
lifelong learning setting of MSFIRC and CCCOSCZC, we train model for each task
for 20 training epochs. The SGD optimization algorithm for each model is using
the Adam Kingma and Ba (2015) where the learning rate is 0.0002 and other
hyperparameters are default. In the following, we provide the details for
datasets. For CelebA, CACD, we randomly choose 10000 samples as the testing
set and the remaining samples are used as the training set. For 3D-chair, we
randomly choose 1000 samples as the testing set and the remaining samples as
the training set. For ImageNet, we randomly choose 10000 and 50000 samples as
the testing set and the training set, respectively. For CUB, we randomly
choose 1000 samples as the testing set and the remaining samples as the
training set.

We consider the threshold of 400 and 600 for DEGM on the MSFIRC and CCCOSCZC,
respectively. We report the results in Table 4 and Table 5, respectively. The
interference can be observed by BE and CN-DPM* which gives the degenerated
performance on the last task since Cifar10 is different from the first task.
In contrast, DEGM avoids interference by building a basic component to learn
Cifar10 database.

(a) ${\bf V}$ of DEGM after the MSFIRC lifelong learning. (b) $\bf{V}$ of DEGM
after the CCCOSCZC lifelong learning.