# Generative learning for deep networks

Boris Flach
<EMAIL_ADDRESS>
&Alexander Shekhovtsov
<EMAIL_ADDRESS>
&Ondrej Fikar
<EMAIL_ADDRESS>

Czech Technical University in Prague

###### Abstract

Learning, taking into account full distribution of the data, referred to as
generative, is not feasible with deep neural networks (DNNs) because they
model only the conditional distribution of the outputs given the inputs.
Current solutions are either based on joint probability models facing
difficult estimation problems or learn two separate networks, mapping inputs
to outputs (recognition) and vice-versa (generation). We propose an
intermediate approach. First, we show that forward computation in DNNs with
logistic sigmoid activations corresponds to a simplified approximate Bayesian
inference in a directed probabilistic multi-layer model. This connection
allows to interpret DNN as a probabilistic model of the output and all hidden
units given the input. Second, we propose that in order for the recognition
and generation networks to be more consistent with the joint model of the
data, weights of the recognition and generator network should be related by
transposition. We demonstrate in a tentative experiment that such a coupled
pair can be learned generatively, modelling the full distribution of the data,
and has enough capacity to perform well in both recognition and generation.

##  1 Introduction

Neural networks with logistic sigmoid activations may look very similar to
Bayesian networks of the same structure with logistic conditional
distributions, aka sigmoid belief networks [11]. However, hidden units in NNs
are deterministic and take on real values while hidden units in Bayesian
networks are binary random variables with an associated distribution. Given
enough capacity and training data, both models can estimate the posterior
distribution of their output arbitrary well. Besides somewhat different
modelling properties, there is a principled difference that with stochastic
models it is possible to pose, at least theoretically, a number of inference
and learning problems with missing data by marginalizing over latent and
unobserved variables. Unfortunately, even forward inference in Bayesian
networks requires methods such as sampling or optimization of a variational
approximation. Likewise, in more tightly coupled graphical models such as deep
Boltzmann machine (DBM) [16] or deep belief networks (DBN) [5, 9] practically
all computations needed e.g. for computing marginal and posterior
probabilities are not tractable in the sense that approximations typically
involve sampling or optimization.

In this paper we propose stacked (deep) conditional independent model (DCIM).
There are two views how the model can be defined. One, as a Bayesian network
with logistic conditional distributions. The other, just assuming that
conditional probabilities of a general Bayesian network factor over the parent
nodes up to a normalising factor. With binary units this necessary implies
that conditional probabilities are logistic. It is noteworthy that we find the
same form of conditional probabilities in most of the neural probabilistic
models: restricted Boltzmann machine, DBM [16], DBN [5], etc. When the units
are arranged in layers, as in a typical DNN, the layered Bayesian network can
be viewed as a Markov chain with a state space of all variables in a layer. In
this view, all necessary assumptions can be summarized in one property, termed
strong conditional independence that the forward conditional transition
probabilities between the layers factorize over the dimensions of both the
input and output state spaces of the transition up to a normalising factor.
Making only this assumption, we show that a simple approximate Bayesian
inference in DCIM recovers main constructive elements of DNNs: sigmoid and
softmax activations with its real-valued variables corresponding to
expectations of binary random variables. With this interpretation we can view
DNN as a way of performing approximate inference very efficiently. To our
knowledge, this relationship has not been established before. Under this
approximation conditional likelihood learning of a DCIM is equivalent to that
of DNN and can be performed by back-propagation.

Our second objective is to propose an alternative, generative learning
approach for DNNs. In a number of recent works [6, 13, 15, 10] and in the
prior work [4, 5] a pair of deterministic recognition (encoder) and a
stochastic generator (decoder) networks is trained. Let us denote
${\boldsymbol{x}}^{0}$ the input of the recognition network, to be specific,
an image, and ${\boldsymbol{x}}^{d}$ (random variables at layer $d$) the
output of the recognition network, the latent state. Although the two networks
often are taken to have a symmetric structure [4, 6], their parameters are
decoupled. The stochastic generator network can typically only generate
samples, but cannot directly evaluate its posterior distribution or the
gradient thereof, requiring variational or sampling-based approximations. The
methods proposed in [6, 13] assume that samples generated from the same latent
state $x^{d}$ must fall close together in the image space. This prohibits
using categorical latent spaces such as digit class in MNIST because digits of
the same class naturally may look differently. Instead, a model’s own
continuous latent space is used such that fixing a point in it defines both
the class and the shape of the digit. Works [6, 13] are thus restricted to
unsupervised learning.

Given full training data pairs of
$({\boldsymbol{x}}^{0},{\boldsymbol{x}}^{d})$, the recognition network could
learn a distribution $p({\boldsymbol{x}}^{d}\mid{\boldsymbol{x}}^{0})$ and the
generator network could in principle learn a distribution
$q({\boldsymbol{x}}^{0}\mid{\boldsymbol{x}}^{d})$. With our link between DNN
and DCIM, we ask the question when the two DCIMs, modelling the two
conditional distributions $p({\boldsymbol{x}}^{0}\mid{\boldsymbol{x}}^{d})$
and $q({\boldsymbol{x}}^{d}\mid{\boldsymbol{x}}^{0})$, are consistent, i.e.,
correspond to some implicitly modelled joint distribution
$p^{*}({\boldsymbol{x}}^{0,\dots,d})$ of the data and all hidden layers. It is
the case when $p({\boldsymbol{x}}^{1,\dots
d}\mid{\boldsymbol{x}}^{0})/q({\boldsymbol{x}}^{0,\dots
d-1}\mid{\boldsymbol{x}}^{d})=A({\boldsymbol{x}}^{0})B({\boldsymbol{x}}^{d})$
for some functions $A,B$. While this cannot be strictly satisfied with our
strong conditional independence assumptions, we observe that most of the terms
in the ratio cancel when we set the weights of recognition network $p$ to be
transposed of those of network $q$. Both models therefore can be efficiently
represented by one and the same DNN, share its parameters and can be learned
simultaneously by using an estimator similar to pseudo-likelihood. We further
use the link between DNN and DCIM to approximately compute the posterior
distribution in the generator network
$q({\boldsymbol{x}}^{d}\mid{\boldsymbol{x}}^{k})$ given a sample of an inner
layer ${\boldsymbol{x}}^{k}$. The approximation is reasonable when this
posterior is expected to have a single mode, such as when reconstructing an
image from lower level features. We thus can fully or partially avoid sampling
in the generator model. We demonstrate in a tentative experiment that such a
coupled pair can be learned simultaneously, modelling the full distribution of
the data, and has enough capacity to perform well in both recognition and
generation.

##### Outline

In Section 2.1 we consider strongly conditional independent model with two
layers and derive the NN from it. This model will serve as a building block of
DCIM formally introduced in Section 2.2. In Section 2.3 we consider coupled
pairs of DCIMs and propose a novel approach for generative learning. In
Section 2.4 we discuss more connections with related work. In Section 3 we
propose our proof-of-concept experiments of generative learning using only DNN
inference or sampling only a single layer.

##  2 Deep conditional independent models

###  2.1 An approximation for conditional independent models

Let $\boldsymbol{x}=(x_{1},\ldots,x_{n})$ and
$\boldsymbol{y}=(y_{1},\ldots,y_{m})$ be two collections of discrete random
variables with a joint distribution
$p(\boldsymbol{x},\boldsymbol{y})=p(\boldsymbol{y}\mid\boldsymbol{x})\>p(\boldsymbol{x})$.
The conditional distribution $p(\boldsymbol{y}\mid\boldsymbol{x})$ is strongly
conditional independent if it factors as
$p(\boldsymbol{y}\mid\boldsymbol{x})=\frac{1}{Z(\boldsymbol{x})}\prod_{i,j}g_{ij}(x_{i},y_{j})$.
Without loss of generality, it can be written as

$p(\boldsymbol{y}\mid\boldsymbol{x})=\frac{1}{Z(\boldsymbol{x})}\exp\sum_{i,j}u_{ij}(x_{i},y_{j})=\prod_{j=1}^{m}p(y_{j}\mid\boldsymbol{x})=\prod_{j=1}^{m}\frac{1}{Z_{j}(\boldsymbol{x})}\exp\sum_{i=1}^{m}u_{ij}(x_{i},y_{j}).$
(1)

The functions $u_{ij}$ are arbitrary and $Z_{j}({\boldsymbol{x}})$ denote the
corresponding normalization constants. While sampling from
$p(\boldsymbol{y}\mid\boldsymbol{x})$ is easy, computing $p(\boldsymbol{y})$
by marginalizing over $\boldsymbol{x}$ is not tractable even if
$p(\boldsymbol{x})$ factorizes over the components of $\boldsymbol{x}$.

We consider the following approximation for the marginal distribution
$p({\boldsymbol{y}})$:

$p(\boldsymbol{y})\approx\frac{1}{Z}\exp\sum_{ij}\sum_{x_{i}}p(x_{i})u_{ij}(x_{i},y_{j}).$
(2)

One of the possible ways to derive it, is to consider Taylor expansions for
moments of functions of random variables. By using a linear embedding
$\boldsymbol{\Phi}$ such that
$\sum_{i,j}u_{ij}(x_{i},y_{j})=\left<\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y}),\boldsymbol{u}\right>$,
where $\boldsymbol{u}$ is a vector with components $u_{i,j}(k,l)$, we can
write $p(\boldsymbol{y}\mid\boldsymbol{x})$ in the form

$p(\boldsymbol{y}\mid\boldsymbol{x})=\frac{\exp\left<\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y}),\boldsymbol{u}\right>}{\sum_{\boldsymbol{y}^{\prime}}\exp\left<\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y}^{\prime}),\boldsymbol{u}\right>}.$
(3)

The marginal distribution $p({\boldsymbol{y}})$ is obtained from

${\mathbb{E}}_{\boldsymbol{x}\sim
p(\boldsymbol{x})}p(\boldsymbol{y}\mid\boldsymbol{x})={\mathbb{E}}_{\boldsymbol{x}\sim
p(\boldsymbol{x})}\frac{\exp\left<\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y}),\boldsymbol{u}\right>}{\sum_{\boldsymbol{y}^{\prime}}\exp\left<\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y}^{\prime}),\boldsymbol{u}\right>}.$
(4)

Taking the first order Taylor expansion w.r.t. the random variable
$\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y})$ for fixed
${\boldsymbol{y}}$ around

$\overline{\Phi}(\boldsymbol{y})={\mathbb{E}}_{\boldsymbol{x}\sim
p(\boldsymbol{x})}\boldsymbol{\Phi}(\boldsymbol{x},\boldsymbol{y})$ (5)

and computing its expectation w.r.t. $\boldsymbol{x}$ gives (2) as the
constant term, while the first order term vanishes.

######  Example 1.

Assume that random variables in the two collections
${\boldsymbol{x}}=(x_{1},\ldots,x_{n})$ and
${\boldsymbol{y}}=(y_{1},\ldots,y_{m})$ are $\\{0,1\\}$-valued. Any function
$u_{ij}(x_{i},y_{j})$ of binary variables can be written as
$u_{ij}(x_{i},y_{j})=y_{j}W_{ji}x_{i}+b_{j}y_{j}+c_{i}x_{i}+d$. Terms
$c_{i}x_{i}+d$ cancel in the normalization of
$p(\boldsymbol{y}\mid\boldsymbol{x})$ and thus can be omitted. The
approximation (2) reads then $\hat{p}(\boldsymbol{y})=\prod_{j}\hat{p}(y_{j})$
with

$\hat{p}(y_{j})=\frac{e^{(\left<\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\right>+b_{j})y_{j}}}{e^{\left<\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\right>+b_{j}}+1},$
(6)

where $\boldsymbol{w}^{j}$ denotes the $j$-th row of the matrix $W$ and
$\bar{\boldsymbol{x}}={\mathbb{E}}_{\boldsymbol{x}\sim
p(\boldsymbol{x})}{\boldsymbol{x}}$. This in turn leads to the following
approximation for the expectation of $\boldsymbol{y}$:

$\bar{y}_{j}=\hat{p}(y_{j}{=}1)=\frac{1}{1+e^{-\langle\boldsymbol{w}^{j},{\boldsymbol{x}}\rangle-
b_{j}}}=\mathcal{S}(\langle\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\rangle+b_{j}),$
(7)

where $\mathcal{S}$ is the logistic (sigmoid) function.

If $\pm 1$ encoding is used instead to represent states of ${\boldsymbol{x}}$
and ${\boldsymbol{y}}$, then the corresponding expressions read

$\hat{p}(y_{j})=\frac{\exp\bigl{[}y_{j}\bigl{(}\left<\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\right>+b_{j}\bigr{)}\bigr{]}}{2\cosh\bigl{[}\left<\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\right>+b_{j}\bigr{]}},$
(8)

and

$\bar{y}_{j}=\tanh[\left<\boldsymbol{w}^{j},\bar{\boldsymbol{x}}\right>+b_{j}].$
(9)

######  Remark 1.

Let $p(\boldsymbol{x},\boldsymbol{y})$ be a joint model such that
$p(\boldsymbol{y}\mid\boldsymbol{x})$ is strongly conditional independent as
above. Then $p(\boldsymbol{x}\mid\boldsymbol{y})$ is not in general
conditional independent. If both conditional distributions are strongly
conditional independent, then the joint model is a restricted Boltzmann
machine or, what is the same, an MRF on a bipartite graph.

###  2.2 Deep conditional independent models

Let
$\mathcal{X}=\bigl{(}\boldsymbol{x}^{0},\boldsymbol{x}^{1},\ldots,\boldsymbol{x}^{d}\bigr{)}$
be a sequence of collections of binary valued random variables

$\boldsymbol{x}^{k}=\bigl{\\{}x_{i}^{k}=\pm
1\bigm{|}i=1,\ldots,n_{k}\bigr{\\}}.$ (10)

The next observation highlights the difficulties that we run into if
considering the deep Boltzmann machine model.

######  Observation 1.

Let us assume that the joint distribution $p(\mathcal{X})$ is a deep Boltzmann
machine, i.e. an MRF on a layered, $d$-partite graph. Then neither the forward
conditionals $p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})$ nor the backward
conditionals $p(\boldsymbol{x}^{k-1}\mid\boldsymbol{x}^{k})$ are
(conditionally) independent. This can be seen as follows. The joint
distribution of such a model can be written w.l.o.g. as

$p(\mathcal{X})=\frac{1}{Z}\exp\Bigr{[}\sum_{k=0}^{d}\left<\boldsymbol{b}^{k},\boldsymbol{x}^{k}\right>+\sum_{k=0}^{d}\left<\boldsymbol{x}^{k},W^{k}\boldsymbol{x}^{k-1}\right>\Bigr{]}$
(11)

and it follows that

$p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})=\frac{F(\boldsymbol{x}^{k})}{Z(\boldsymbol{x}^{k-1})}\exp\Bigr{[}\left<\boldsymbol{b}^{k},\boldsymbol{x}^{k}\right>+\left<\boldsymbol{x}^{k},W^{k}\boldsymbol{x}^{k-1}\right>\Bigr{]}$
(12)

where $F(\boldsymbol{x}^{k})$ results from marginalisation over
$\boldsymbol{x}^{k+1},\ldots,\boldsymbol{x}^{d}$ and can be arbitrarily
complex.

We will therefore consider a similar, but different class, for which the
forward conditionals are (conditionally) independent.

######  Definition 1.

A joint distribution $p(\mathcal{X})$ for a sequence of collections of binary
valued random variables (10) is a deep conditional independent model (DCIM) if
it has the form

$p(\boldsymbol{x}^{0},\ldots,\boldsymbol{x}^{n})=p(\boldsymbol{x}^{0})\prod_{k=1}^{d}p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})$
(13)

and its forward conditionals $p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})$
are strongly conditional independent, i.e. have the form (1) for all
$k=1,2,\ldots,d$.

A DCIM can be thus seen as an inhomogeneous Markov chain model with high
dimensional state spaces $\boldsymbol{x}^{k}\in\mathcal{X}^{k}$, however with
an yet unspecified distribution for $\boldsymbol{x}^{0}$. Given such a model
and a realisation of $\boldsymbol{x}^{0}$, the posterior conditional
distribution $p(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})$ can be computed by
applying the approximation (2) recursively (see Example. 1). This leads to the
recursion

$\bar{x}_{i}^{k}=\tanh\bigl{(}\left<\boldsymbol{w}^{k}_{i},\bar{\boldsymbol{x}}^{k-1}\right>+b^{k}_{i}\bigr{)}\text{,\hskip
5.0pt$k=1,2\ldots,d-1$},$ (14)

where $\bar{\boldsymbol{x}}^{k}$ denotes the expectation of
$\boldsymbol{x}^{k}$ w.r.t. $p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{0})$.
Finally, we obtain

$p(x^{d}_{i}\mid\boldsymbol{x}^{0})=\frac{\exp(x^{d}_{i}a^{d}_{i})}{2\cosh(a^{d}_{i})}\text{\hskip
5.0ptwith\hskip
5.0pt}a^{d}_{i}=\left<\boldsymbol{w}^{d}_{i},\bar{\boldsymbol{x}}^{d-1}\right>+b^{d}_{i}.$
(15)

It is obvious that the (approximate) computation of
$p(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})$ for a DCIM is exactly the
forward computation for a corresponding DNN. Hence, discriminative learning of
a DCIM by maximizing the conditional likelihood
$p({\boldsymbol{x}}^{d}\mid{\boldsymbol{x}}^{0})$ of training samples
$({\boldsymbol{x}}^{d},{\boldsymbol{x}}^{0})$ means to learn the corresponding
DNN with the same loss.

With 0-1 state variables the model is equivalent to a Bayesian network (a
directed graphical model), in which the set of parents of a variable
$x^{k}_{j}$ is the preceding layer ${\boldsymbol{x}}^{k-1}$ (or its subset,
depending on the structure of the weights) and the conditional distributions
are of the form (1), _i.e_., logistic. Such networks were proposed by Neal
[11] under the name sigmoid belief networks as a simpler alternative to
Boltzmann machines. When we derive the model from the strong conditional
independence assumption, it establishes a link to deep Boltzmann machines
[16], which are graphical models factorizing over a $d$-partite graph.

###  2.3 Generative learning of deep networks

To learn a DCIM generatively from a given i.i.d. training data $\mathcal{T}$,
it is necessary to specify the joint model
$p(\boldsymbol{x}^{0},\ldots,\boldsymbol{x}^{d})$ and to choose an appropriate
estimator. In theory, the DCIM model can be completed to a joint model by
specifying a prior distribution $p(\boldsymbol{x}^{0})$ over images and then
the maximum likelihood estimator can be applied. It is however not realistic
to model this complex multi-modal distribution in a closed form. We propose to
circumvent this problem by applying the following bidirectional conditional
likelihood estimator:

$\frac{1}{\lvert\mathcal{T}\rvert}\sum_{(\boldsymbol{x}^{0},\boldsymbol{x}^{d})\in\mathcal{T}}\bigl{[}\log
p_{\theta}(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})+\log
p_{\theta}(\boldsymbol{x}^{0}\mid\boldsymbol{x}^{d})\bigr{]}\rightarrow\max_{\theta},$
(16)

because it avoids to model $p(\boldsymbol{x}^{0})$ and $p(\boldsymbol{x}^{d})$
explicitly.

From the definition of a DCIM, and particularly from the fact that a DCIM is
also a Markov chain model, follows that its reverse conditional distribution
factorises similar as (13), i.e.

$p(\boldsymbol{x}^{0},\ldots,\boldsymbol{x}^{d-1}\mid\boldsymbol{x}^{d})=\prod_{k=0}^{d-1}p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k+1}).$
(17)

This holds for any $p(\boldsymbol{x}^{0})$ completing the DCIM to a joint
distribution. Unfortunately, however, the reverse conditionals
$p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k+1})$ of a DCIM are not
(conditionally) independent. This follows from an argument similar to the one
used in Observation 1. In order to resolve the problem, we propose to consider
pairs of tightly connected DCIMs: a forward model and a reverse (i.e.
backward) model

$p(\boldsymbol{x}^{d},\ldots,\boldsymbol{x}^{1}\mid\boldsymbol{x}^{0})=\prod_{k=1}^{d}p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})\text{\hskip
10.00002ptand\hskip
10.00002pt}q(\boldsymbol{x}^{0},\ldots,\boldsymbol{x}^{d-1}\mid\boldsymbol{x}^{d})=\prod_{k=0}^{d-1}q(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k+1})$
(18)

such that the forward conditionals of $p$ and the backward conditionals of $q$
are strongly (conditional) independent. Such a pair would give rise to a
single joint distribution if

$\frac{p(\boldsymbol{x}^{d},\ldots,\boldsymbol{x}^{1}\mid\boldsymbol{x}^{0})}{q(\boldsymbol{x}^{0},\ldots,\boldsymbol{x}^{d-1}\mid\boldsymbol{x}^{d})}=\frac{A(\boldsymbol{x}^{d})}{B(\boldsymbol{x}^{0})}$
(19)

holds for any realisation of $\mathcal{X}$ for some functions $A,B$. We
already know that this is impossible to achieve while retaining conditional
independence of both, the forward and backward conditionals. Therefore, we
propose to choose the parameters for two models such that the equation is
fulfilled as close as possible. Substituting (1) for the binary case in the
lhs of (19), gives

$\frac{\prod_{k=0}^{d-1}\tilde{Z}(\boldsymbol{x}^{k+1})}{\prod_{k=1}^{d}Z(\boldsymbol{x}^{k-1})}\frac{\exp\Bigl{[}\sum_{k=1}^{d}\left<\boldsymbol{x}^{k},W^{k}\boldsymbol{x}^{k-1}+\boldsymbol{b}^{k}\right>\Bigr{]}}{\exp\Bigl{[}\sum_{k=0}^{d-1}\left<\boldsymbol{x}^{k},V^{k}\boldsymbol{x}^{k+1}+\boldsymbol{a}^{k}\right>\Bigr{]}}\overset{?}{=}\frac{A(\boldsymbol{x}^{d})}{B(\boldsymbol{x}^{0})},$
(20)

where $Z_{k}$ and $\tilde{Z}_{k}$ denote the partition functions of
$p(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k-1})$ and
$q(\boldsymbol{x}^{k}\mid\boldsymbol{x}^{k+1})$ respectively. It is readily
seen that all terms in the exponentials cancel out if $(V^{k-1})^{T}=W^{k}$
holds for the weights and $\boldsymbol{b}^{k}=\boldsymbol{a}^{k-1}$ holds for
the biases. The remaining, not cancelling terms in the lhs of (19) are then
the partition functions of the conditional distributions.

Summarising, a pair of such DCIMs share the same structure as well as their
weights and biases. They are therefore represented by a single DNN and can be
learned simultaneously by the estimator (16), which now reads as

$\frac{1}{\lvert\mathcal{T}\rvert}\sum_{(\boldsymbol{x}^{0},\boldsymbol{x}^{d})\in\mathcal{T}}\bigl{[}\log
p_{\theta}(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})+\log
q_{\theta}(\boldsymbol{x}^{0}\mid\boldsymbol{x}^{d})\bigr{]}\rightarrow\max_{\theta}.$
(21)

Since both, the forward and backward conditionals are strongly independent,
approximation (2) can be applied for computing the probabilities
$p(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})$ and
$q(\boldsymbol{x}^{0}\mid\boldsymbol{x}^{d})$.

######  Remark 2.

If the model consists of one layer only ($d=1$), then the pair of
$p(\boldsymbol{x}^{1}\mid\boldsymbol{x}^{0})$ and
$q(\boldsymbol{x}^{0}\mid\boldsymbol{x}^{1})$ define a single joint
distribution, which is an RBM, i.e. an MRF on a bipartite graph. The estimator
(16) becomes the pseudo-likelihood estimator.

###  2.4 Relations to other models

We briefly discuss relations between the proposed model and other models not
mentioned in the introduction.

The connection between Bayesian networks and neural networks with injected
noise, used in [7, 13] is different from ours. It relates two equivalent
representations of stochastic models, while we relate stochastic and
deterministic models.

Similar to our work, Hinton et al. [5] uses a constraint that generator and
recognition networks are related by transposition of weights in the
unsupervised, layer-by-layer pre-training phase, motivated by their study of
deep Boltzmann machine. For the supervised fine-tuning the two models are
decoupled.

The proposed model is of course related to classical auto-encoders [14] at
least technically. Our learning approach is generative, but in contrast to
auto-encoders it is supervised. Moreover, the “decoding” part (the reverse
model) is in our case constrained to have the same parameters as the
“encoding” part (the forward model).

Our model and the proposed generative learning approach are undoubtedly
related to generative adversarial networks (GAN), [3],[12] proposed by
Goodfellow et al. Similar to them, the reverse part of our model aims at
generating data. But in contrast to them our model uses no specific “noise
space” and conditions the image distribution on the class. The randomness in
the generating part comes solely through the reverse Markov chain. We believe
that this should suffice for modelling rich distributions in the input space.
It is however to expect that the approximation (2) used for learning might
impose limitations on the expressive power of this model part as compared with
GANs (see experiments). Another difference is that our model is learned
jointly with a “twin”, i.e. the forward part, rather than in competition with
an “adversarial”.

##  3 Experiments

###  3.1 MNIST dense

The first experiment is a simple proof of concept. We trained a small network
(three fully connected layers with 512, 512, 10 nodes) on the standard MNIST
dataset [8] for both learning approaches: the discriminative and the
generative one. Sigmoid activation was chosen for all but the last layer, for
which we used soft-max activation, i.e. considering its nodes as categorical
variable. A standard optimiser without drop-out and any other regularisers was
used for searching the maximum of the objective function. Fig. 1 shows the
training and validation accuracies for both approaches as functions of the
iterations. It is clearly seen that the combined objective (16) of the
generative learning imposes only a rather insignificant drop of the
classification accuracy. Fig. 1 also shows the training and validation loss of
the the backward model.

To further asses the reverse model learned by the generative approach, we
sampled 10 images for each of the classes from both models. The results are
shown in Fig. 2. For each sampled image we also applied the forward model to
classify it. The classes with the highest probability along with values of
that probability are shown on top of each sampled image. It is clearly seen
that the generative learning approach has found a set of model parameters that
yields simultaneously good classification accuracy and the ability to generate
images of average digits. On the other hand, it is also visible from Fig. 2
that the generative model part learned only the mean shapes of the digits.

Figure 1: Learning simple network on MNIST. Left: accuracy for discriminative
and generative learning (dashed - training, solid: validation). Right: loss of
the reverse model in generative learning (dashed - training, solid:
validation).

Figure 2: Images sampled form the reverse model part along with their
classification and probability values (see text). Each row shows images
sampled from one class. Left: images obtained by recursive sampling through
the net starting from the class layer till the image layer. Right: images
obtained by sampling through one layer only and then recursively computing
probabilities till the image layer.

###  3.2 MNIST CNN

The second experiment intends to analyse the behaviour of the proposed
generative learning approach for convolutional neural networks. We learned a
deep CNN ( eight layers) with architecture ((3,1,32), (3,1,32), (2,2,32),
(3,1,64), (3,1,64), (2,2,64), (50), (10)) on MNIST data. The components of the
triplets denote the window size, stride and kernel number for convolutional
layers. The singletons denote the size of dense layers. Sigmoid activation was
used for all but the last layer. When using the objective (21), the net
achieves 0.989 validation accuracy for the forward part and 0.23 loss for the
reverse model part. We observed neither over-fitting nor “vanishing gradient”
effects. We conjecture that the reverse learning task serves as a strong
regulariser. The left tableaux in Fig. 3 shows images sampled from the reverse
model part. Again, for better visibility, we sampled from the the distribution
$q(\boldsymbol{x}^{d-1}\mid\boldsymbol{x}^{d})$ and then recursively computed
the probabilities down to the image layer by applying the approximation (2).
It is clearly seen that the learned generative model part is not able to
capture the multi-modal image distribution. We conjecture that one of the
possible reasons is the simplicity of the approximation (2), especially when
applied for learning of the reverse model part.

To analyse the problem, we considered a somewhat different learning objective
for this model part

$\frac{1}{\lvert\mathcal{T}\rvert}\sum_{(\boldsymbol{x}^{0},\boldsymbol{x}^{d})\in\mathcal{T}}\bigl{[}\log
p_{\theta}(\boldsymbol{x}^{d}\mid\boldsymbol{x}^{0})+\log
q_{\theta}(\boldsymbol{x}^{0}\mid\boldsymbol{x}^{d-1})\bigr{]}\rightarrow\max_{\theta}.$
(22)

where $\boldsymbol{x}^{d-1}$ is sampled from
$p_{\theta}(\boldsymbol{x}^{d-1}\mid\boldsymbol{x}^{0})$ for the current
$\theta$. The model learned with this objective achieved 0.990 validation
accuracy and 0.14 validation loss for the reverse model part. Images sampled
from this model in the same way as described above, are shown in the right
tableaux of Fig. 3. It is clearly seen that this model captured modes in the
image distribution, i.e. different writing styles of the digits.

Figure 3: Images sampled from generatively learned CNNs on MNIST data. For
better visibility the images were obtained by sampling through one layer only
and then recursively computing probabilities till the image layer. Left:
learning method as described in Sec. 2.3. Right: alternative learning method
described in Sec. 3.2.

All experiments presented in this paper were carried out by using Keras [2] on
top of Tensorflow [1].

##  4 Conclusions

We proposed a class of probabilistic models that leads to a new statistical
interpretation of DNNs in which all neurons of the net become random
variables. In contrast to Boltzmann machines, this class allows for an
efficient approximation when computing the distribution of the output
variables conditioned on a realisation of the input variables. The computation
itself becomes identical to the recursive forward computation for a DNN.

The second objective of the paper, to design a generative learning approach
for DCIMs, has been reached only partially. The proposed approach and its
variants do allow to learn the forward and backward model parts
simultaneously. However, the achievable expressive power of the backward part
of the model is still inferior to GANs.

Notwithstanding this, we believe that the presented approach opens interesting
research directions. One of them is to analyse the proposed approximation and
to search for better ones, leading to new and better activation functions.
Another question in that direction is a statistical interpretation of the ReLu
activation as the expectation of a possibly continuous random variable.

A different direction calls for generalising DCIMs by introducing interactions
between neurons in layers. This would lead to deep CRFs. We envision to learn
such models, at least discriminatively, by using e.g. recent results for
efficient marginal approximations of log-supermodular CRF.

#### Acknowledgements

A. Shekhovtsov was supported by Toyota Motor Europe HS, O. Fikar was supported
by Czech Technical University in Prague under grant SGS17/185/OHK3/3T/13 and
B. Flach was supported by Czech Science Foundation under grant 16-05872S.

## References

  * Abadi et al. [2015] Abadi, M. et al. (2015).  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Chollet et al. [2015] Chollet, F. et al. (2015).  Keras.  https://github.com/fchollet/keras.
  * Goodfellow et al. [2014] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. (2014).  Generative adversarial nets.  In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672–2680.
  * Hinton et al. [1995] Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995).  The wake-sleep algorithm for unsupervised neural networks.  Science, 268:1158.
  * Hinton et al. [2006] Hinton, G. E., Osindero, S., and Teh, Y. W. (2006).  A fast learning algorithm for deep belief networks.  Neural Computation, 18(7):1527–1554.
  * Kingma and Welling [2013] Kingma, D. P. and Welling, M. (2013).  Auto-encoding variational bayes.  CoRR, abs/1312.6114.
  * Kingma and Welling [2014] Kingma, D. P. and Welling, M. (2014).  Efficient gradient-based inference through transformations between bayes nets and neural nets.  In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, pages II–1782–II–1790. JMLR.org.
  * LeCun and Cortes [2010] LeCun, Y. and Cortes, C. (2010).  MNIST handwritten digit database.
  * Lee et al. [2009] Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009).  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations.  In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 609–616, New York, NY, USA. ACM.
  * Mnih and Gregor [2014] Mnih, A. and Gregor, K. (2014).  Neural Variational Inference and Learning in Belief Networks.  In Proceedings of the 31st International Conference on Machine Learning, Cycle 2, volume 32 of JMLR Proceedings, pages 1791–1799. JMLR.org.
  * Neal [1992] Neal, R. M. (1992).  Connectionist learning of belief networks.  Artif. Intell., 56(1):71–113.
  * Radford et al. [2016] Radford, A., Metz, L., and Chintala, S. (2016).  Unsupervised representation learning with deep convolutional generative adversarial networks.  In ICLR.
  * Rezende et al. [2014] Rezende, D. J., Mohamed, S., and Wierstra, D. (2014).  Stochastic backpropagation and approximate inference in deep generative models.  In Jebara, T. and Xing, E. P., editors, Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1278–1286. JMLR Workshop and Conference Proceedings.
  * Rumelhart et al. [1986] Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986).  Learning representations by back-propagating errors.  Nature, 323:533–536.
  * Salakhutdinov [2015] Salakhutdinov, R. (2015).  Learning deep generative models.  Annual Review of Statistics and Its Application, 2:361–385.
  * Salakhutdinov and Hinton [2009] Salakhutdinov, R. and Hinton, G. E. (2009).  Deep Boltzmann Machines.  In AISTATS, pages 448–455.