# Generative vs. discriminative modeling under the lens of uncertainty
quantification

Elouan Argouarc’h, François Desbouvries,
SAMOVAR
Télécom SudParis, Institut Polytechnique de Paris
91120 Palaiseau France
{elouan.argouarch<EMAIL_ADDRESS>
&Eric Barat, Eiji Kawasaki,
CEA-List
Université Paris-Saclay
91120 Palaiseau France
{eric.barat<EMAIL_ADDRESS>

###### Abstract

Learning a parametric model from a given dataset indeed enables to capture
intrinsic dependencies between random variables via a parametric conditional
probability distribution and in turn predict the value of a label variable
given observed variables. In this paper, we undertake a comparative analysis
of generative and discriminative approaches which differ in their construction
and the structure of the underlying inference problem. Our objective is to
compare the ability of both approaches to leverage information from various
sources in an epistemic uncertainty aware inference via the posterior
predictive distribution. We assess the role of a prior distribution, explicit
in the generative case and implicit in the discriminative case, leading to a
discussion about discriminative models suffering from imbalanced dataset. We
next examine the double role played by the observed variables in the
generative case, and discuss the compatibility of both approaches with semi-
supervised learning. We also provide with practical insights and we examine
how the modeling choice impacts the sampling from the posterior predictive
distribution. With regard to this, we propose a general sampling scheme
enabling supervised learning for both approaches, as well as semi-supervised
learning when compatible with the considered modeling approach. Throughout
this paper, we illustrate our arguments and conclusions using the example of
affine regression, and validate our comparative analysis through
classification simulations using neural network based models.

##  1 Introduction

The statistical learning tasks of classification and regression [74][32] are
paramount in many scientific fields and have gained increasing interest in the
big-data and machine learning era. Elaborate methods and tools [5][9] enable
to leverage flexible parametric models in order to capture intrinsic
dependencies between related variables, and in turn predict the value of a
variable of interest given observed values of the others.

Many statistical learning methods, both historical [1][6][51] and recent
[52][44][52], can be understood as building an approximate of the conditional
probability distribution of a variable of interest (which we refer to as label
in this paper) given the value of an observed variable. This is usually
achieved by considering a parameterized model and adjusting the parameters by
minimizing a loss function computed on a dataset comprised of recorded couples
of observations and labels.

In this context, two main approaches are often opposed and compared
[56][73][52, §9.4] [48][20][77]. The discriminative modeling consists in
parameterizing a distribution over the label given the observations as in
[42][41][29] for example; while the generative modeling techniques
parameterize a distribution over the observed variable given the label, see
[66][61][62] [50] for some applications.

In many learning tasks, the observations are high-dimensional random variables
(rv) when compared to the usually low-dimensional labels. Therefore,
discriminative models are much more convenient to work with compared to
generative models as the later involve modeling a high-dimensional conditional
distribution. However, recent developments in generative probabilistic
modeling [27][60][70] which now enable to capture intrinsic distributions have
paved the way towards a renewed appeal of the generative modeling approach
[59][37][78].

When predicting a label of interest using a parametric model (be it generative
or discriminative) from an observation, committing to a unique value can lead
to high imprecision as there are two sources of uncertainty that one needs to
account for [16][39][36]. Uncertainty aware modeling aims at computing or
estimating confidence/credible intervals associated to a prediction, but this
task of uncertainty quantification remains challenging [2]. The aleatoric
uncertainty is the source of uncertainty which results from the stochastic
nature of the unknown (or at least with untractable probability density
function (pdf)) Data Generating Process (DGP) which, as we assume, generates
the observations from the labels. However, using an approximate parametric
model also induces an additional uncertainty about the predicted labels, which
is referred to as epistemic [35].

Bayesian modeling methods for uncertainty quantification are perhaps amongst
the most promising approaches [26]: the model parameters are treated as
random, and are marginalized-out to obtain the predicted law of label given
the observation as well as the dataset. This so-called posterior predictive
distribution (ppd) will from now on be the distribution of interest, and our
aim throughout this paper is to compute (or at least sample from) this
distribution.

Such techniques include building conjugate models for which, by construction,
the model parameter posterior distribution is easy to sample from, see e.g.
[65] [30]. In the case of more elaborate (typically neural network (NN) based)
models, prior conjugacy can be leveraged to some extent in methods such as
Bayesian last layer [21], but the ppd can no longer be sampled from with a
straightforward procedure. Fortunately, Bayesian methods still enable to
sample from this distribution [53], be it via MCMC methods [11][75][38] or
variational inference [28][23][69]. Finally, approximate methods such as
posterior Bootstrap [55][49][22][54] do indeed provide with theoretically
grounded approaches for Bayesian model averaging, or bagging, uncertainty
quantification.

Our aim in this paper is to compare the generative and discriminative
approaches under the scope of Bayesian uncertainty quantification.

The rest of our paper is organized as follows. In section 2 we first provide
with a precise description of both generative and discriminative
constructions. Next in section 3, by analysing the ppd, we explain different
behaviours of the two modeling approaches in an epistemic uncertainty aware
inference. More specifically, in section 3.4, we focus our attention on the
role of a specific distribution which can be understood as a prior
distribution associated to the ppd, and analyze the ability of each approach
to infer using prior information. By doing so, we give clues as to why
discriminative models can suffer from imbalanced datasets, while the
generative ones do not, and confirm this analysis via both illustrative and
quantitative simulations. In order to sample from the ppd, especially in the
generative case, we provide in section 3.5 with a general sampling algorithm
which is based on a Gibbs scheme and which can easily be applied to both
approaches. Finally in section 4, we specifically discuss the dependency of
model parameters to observations, and conclude on the compatibility of each
approach with the task of semi-supervised learning which aims at inferring the
model parameters from both labeled and unlabeled datasets. We propose to
leverage the corresponding Gibbs sampling scheme to perform Bayesian semi-
supervised learning in the generative case. We finally perform simulations in
the context of image classification in order to illustrate the arguments of
this paper in different learning scenarios.

##  2 Supervised learning: Context and objective

Let $(X,Y)$ be a couple of rv related via a DGP $\mathcal{P}_{Y|X}$ which
describes the probability distribution of $Y$ given $X$. The task of
prediction consists in retrieving information about an unknown $x_{0}$ which
(is assumed to have) generated an observed $y_{0}$ via the DGP:
$y_{0}\sim\mathcal{P}_{Y|X}(Y|X=x_{0}).$ In this paper, we use the specific
nomenclature of a classification problem and we denote $Y$ as _observation_
(from the DGP) and, $X$ as the corresponding _label_ (though it is not
necessarily categorical). The Bayes formula tells us that, once the value
$y_{0}$ is observed, the information about $x_{0}$ is encapsulated in the
_posterior_ distribution $\mathcal{P}_{X|Y}$ with probability density
function111throughout this paper, the pdf should be understood w.r.t. the
appropriate measure, depending on the nature (continuous, discrete or mixed
continuous-discrete) of the underlying variables. (pdf) given by:

$p_{X|Y}(x_{0}|y_{0})=\frac{p_{Y|X}(y_{0}|x_{0})\pi_{X_{0}}(x_{0})}{p_{Y_{0}}(y_{0})},\text{where
}p_{Y_{0}}(y_{0})=\int p_{Y|X}(y_{0}|x)\pi_{X_{0}}(x)\mathrm{d}x,$ (1)

where $p_{Y|X}({\bf.}|{\bf.})$ is the conditional pdf associated with the DGP
and $\pi_{X_{0}}({\bf.})$ is the pdf associated with the distribution which
describes our _prior_ knowledge about $x_{0}$ (in this paper we denote prior
distributions with letter $\Pi$ and their pdf with $\pi$). This posterior
distribution can be used to obtain pointwise Bayes predictors by minimizing
the expectation of a well-designed loss function $l$ [43]:
$x_{0}^{*}=\arg\min_{x_{0}}\mathbbm{E}_{X|Y}\mathopen{}\left[{l(X,x_{0})}\,\negthickspace\right]\mathclose{};$
but in essence, this posterior distribution describes our inability to commit
to a singular value of $x_{0}$. This source of uncertainty is induced by the
random nature of the DGP and is referred to as _aleatoric_.

If both of these pdf can be evaluated, then (1) can be computed at least up to
the constant denominator $p_{Y_{0}}(y_{0})=\int
p_{Y|X}(y_{0}|x)\pi_{X_{0}}(x)\mathrm{d}x$. In many situations however, the
pdf associated with the DGP is intractable and consequently (1) cannot be
evaluated, not even up to a constant. This situation occurs either when (i) we
only dispose of a dataset $\mathcal{D}$ (defined in the next paragraph)
generated from and the DGP (and so its pdf) is otherwise simply unknown; or
(ii) when the DGP is only available via its stochastic simulation procedure
which enables to obtain (and augment [18][47]) a dataset $\mathcal{D}$ but its
pdf is implicit [13]; historical approaches in this setting include the
Approximate Bayesian Computation (ABC) methods [14]. In this paper we consider
the first setting and suppose that we dispose of $\mathcal{D}$ generated from
the DGP but that we no longer have access to the random sampling mechanism of
the DGP making ABC unfeasible. A possible approach to cope with this
shortcoming is to resort to an approximation of the intractable posterior
using a conditional probability distribution $\mathcal{P}_{\theta}$ where
parameter $\theta$ is inferred using observed couples
$\mathcal{D}\stackrel{{\scriptstyle\Delta}}{{=}}\\{(x_{i},y_{i})|x_{i}\sim\mathcal{P}^{\mathcal{D}}_{X},y_{i}\sim\mathcal{P}_{Y|X}(Y|X=x_{i})\\}_{i=1}^{|\mathcal{D}|}$.
We denote $\mathcal{P}_{X}^{\mathcal{D}}$ the probability distribution which
effectively generated the values in dataset $\mathcal{D}$ but we stress here
that this probability distribution is not necessarily the same as the prior
distribution $\Pi_{X_{0}}$ (this particular point and its consequences are
discussed in details in section 3.4). This general formulation includes the
usual tasks of statistical parametric learning: we talk about regression
(resp. classification) when $X$ is continuous (resp. categorical).

Since a unique parameter estimate of $\theta$ (such as Maximum Likelihood
Estimates (MLE) or Maximum A Posteriori (MAP) estimates) can be stained with
high imprecision if $\mathcal{D}$ is not representative enough of the DGP, we
rather consider $\theta$ to be a hidden rv, assume a prior knowledge described
by distribution $\Pi_{\Theta}$, and approximate (1) with the ppd [25]:

$p(x_{0}|y_{0},\mathcal{D})=\int
p(x_{0},\theta|y_{0},\mathcal{D})\mathrm{d}{\theta}.$ (2)

This pdf indeed accounts for the _epistemic_ uncertainty which is the
uncertainty about the unknown parameter $\theta$ (induced by the finite number
of recorded samples in $\mathcal{D}$) propagated to $x_{0}$ predicted by this
model. The ppd (2) is computed by integrating out $\theta$ in the joint pdf:

$p(x_{0},\theta|y_{0},\mathcal{D})=p(x_{0}|y_{0},\theta)p(\theta|y_{0},\mathcal{D}),$
(3)

and so
$p(x_{0}|y_{0},\mathcal{D})=\mathbbm{E}_{\theta}[p(x_{0}|y_{0},\theta)|y_{0},\mathcal{D}]$
which explains the denomination: it is the average of (posterior)
_predictions_ $p(x_{0}|y_{0},\theta)$ over probable models under the
_posterior_ $p(\theta|y_{0},\mathcal{D})$. Ultimately, computing the posterior
pdf (2), or sampling from the distribution if exact computation of the
expectation by integration is unfeasible, would allow to identify the probable
(aleatoric) values of $x_{0}$ that might have generated $y_{0}$ via the DGP,
while accounting for the modeling (epistemic) uncertainty.

#### Illustrating running example

Throughout this paper, we propose to illustrate arguments and conclusions on
the continued example of affine regression. Though it serves as an
illustrative example, this specific application to affine modeling is, in
itself, relevant since affine relationship between variables of interest are
most frequent in many science fields and are in many cases, the first
considered dependency hypothesis. Let $(X,Y)\in\mathbb{R}\times\mathbb{R}$ be
two real-valued rv (assumed to be) related via an unknown DGP. For
illustration purposes, we consider a toy setting where the DGP is of the form:

$Y=\alpha_{1}X+\alpha_{0}+\sigma_{Y|X}\epsilon\iff
p_{Y|X}(y|x)=\mathcal{N}(y;\alpha_{1}x+\alpha_{0},\sigma^{2}_{Y|X}).$ (4)

We dispose of recorded data $\mathcal{D}$ produced by the DGP, and where the
distribution of the $x$ values is $\mathcal{P}_{X}^{\mathcal{D}}$. The prior
knowledge on $x_{0}$ is given by the distribution
$\pi_{X_{0}}(x_{0})=\mathcal{N}(x_{0};\mu_{X_{0}},\sigma_{X_{0}}^{2})$. We
chose the prior and the DGP to be conjugated such that the posterior
distribution reads:

$p_{X|Y}(x_{0}|y_{0})=\mathcal{N}(x_{0};(\frac{\alpha_{1}^{2}}{\sigma^{2}_{Y|X}}+\frac{1}{\sigma_{X_{0}}^{2}})^{-1}(\frac{\alpha_{1}(y_{0}-\alpha_{0})}{\sigma^{2}_{Y|X}}+\frac{\mu_{X_{0}}}{\sigma_{X_{0}}^{2}}),(\frac{\alpha_{1}^{2}}{\sigma^{2}_{Y|X}}+\frac{1}{\sigma_{X_{0}}^{2}})^{-1});$
(5)

and so this posterior distribution can be used to assess the quality of the
inference when comparing the ppd $p(x_{0}|y_{0},\mathcal{D})$ to it; but we
otherwise suppose the DGP unavailable. An example of this setting is
illustrated in figure 1.

Figure 1: Supervised learning setting

###  2.1 Generative versus Discriminative modeling

In the previous section we explained the general principle of modeling the
posterior pdf (1), and we emphasized on the role of the ppd (2), which
accounts for the epistemic modeling uncertainty. However, we have not
explained precisely yet how the modeling is carried out. In fact, using a
parametric conditional probability distribution $\mathcal{P}_{\theta}$, we can
either model the unknown DGP with $\mathcal{P}_{\theta}(Y|X)$ and deduce the
corresponding posterior via the Bayes formula, or model the posterior directly
with $\mathcal{P}_{\theta}(X|Y)$. In the literature, the first approach is
classically referred to as _Generative_ (since it models the data _generating_
process), while the second one is called _Discriminative_ (since it makes
sense in particular in the classification setting, where the model directly
computes the label probabilities which enable to _discriminate_ samples via
their respective classes). The first approach is called generative modeling
but in the (deep) Machine Learning literature, generative modeling [8] can
also refer to the task building a parametric probability distribution which is
generative in the sense that it can be sampled from easily and is designed to
resemble a probability which produced recorded data. In this paper and unless
stated otherwise, generative modeling refers to the approach which consists in
building an approximate of the posterior distribution of interest (1) via
modeling the unknown likelihood. These two approaches differ in their
philosophy: the first one models only what is unknown, i.e. the generative
process, while the second one directly models the function of interest, i.e.
the posterior pdf. Figure 2 provides with an illustration of the difference
between the two approaches.

Figure 2: Illustration of the difference between generative and discriminative
modeling approaches (right)

These approaches yield two different equations:

$\displaystyle p(x_{0}|y_{0},\theta)\stackrel{{\scriptstyle
G}}{{=}}\frac{p_{\theta}(y_{0}|x_{0})\pi_{X_{0}}(x_{0})}{\int
p_{\theta}(y_{0}|x)\pi_{X_{0}}(x)\mathrm{d}x}$ (6) $\displaystyle
p(x_{0}|y_{0},\theta)\stackrel{{\scriptstyle D}}{{=}}p_{\theta}(x_{0}|y_{0}),$
(7)

in which superscripts $G$ and $D$ respectively stand for generative and
discriminative; this notation will be used throughout the rest of this paper.
In both equations, $p_{\theta}$, be it $p_{\theta}(y_{0}|x_{0})$ in the
generative case or $p_{\theta}(x_{0}|y_{0})$ in the discriminative case, is
the conditional pdf associated with $\mathcal{P}_{\theta}$, and is what is
effectively computed with model associated with parameter $\theta$.

Figure 3 displays a graphical representation of both models and explains how
the modeling choice affects the dependence between all the rv. We build upon
these two figures by writing the full joint pdf of all rv.

$y_{0}$$x_{0}$$y_{1}$$y_{|\mathcal{D}|}$$x_{1}$$x_{|\mathcal{D}|}$$\theta$……$y_{0}$

$y_{0}$$x_{0}$$y_{1}$$y_{|\mathcal{D}|}$$x_{1}$$x_{|\mathcal{D}|}$$\theta$……$y_{0}$

Figure 3: Graphical models compared: Generative (left) versus Discriminative
(right). The grey (resp. white) nodes are observed (resp. latent) variables.

This comparison of graphical models allow us to deduce the joint distribution
of all the rv of interest:

$\displaystyle p(x_{0},y_{0},\theta,\mathcal{D})\stackrel{{\scriptstyle
G}}{{=}}\pi_{\Theta}(\theta)\pi_{X_{0}}(x_{0})p_{\theta}(y_{0}|x_{0})\prod_{(x_{i},y_{i})\in\mathcal{D}}p^{\mathcal{D}}_{X}(x_{i})p_{\theta}(y_{i}|x_{i}),$
(8) $\displaystyle p(x_{0},y_{0},\theta,\mathcal{D})\stackrel{{\scriptstyle
D}}{{=}}\pi_{\Theta}(\theta)p_{Y_{0}}(y_{0})p_{\theta}(x_{0}|y_{0})\prod_{(x_{i},y_{i})\in\mathcal{D}}p^{\mathcal{D}}_{Y}(y_{i})p_{\theta}(x_{i}|y_{i}).$
(9)

The difference between generative and discriminative modeling has already been
established in the literature. In the context of classification, [56] compares
the Naive Bayes Classifier (which is a generative model) and logistic
regression (which is discriminative) in term of (asymptotic) classification
error and conclude in favor of the generative approach when working with a
small amount of training data.

#### Illustrative running example

We now illustrate the difference between both modeling approaches by
performing homoskedastic affine regression with unknown variance Gaussian
Noise where model parameters are the coefficients of the affine transform as
well as the variance of the unknown noise so
$\theta=\\{\beta_{1},\beta_{0},\sigma^{2}\\}$. Both the DGP (4) and the
posterior (5) belong to the considered parametric family of conditional
probability, so both modeling approaches correspond to a well-specified
inference problem. We first describe the generative approach
$y_{0}\stackrel{{\scriptstyle
G}}{{=}}\beta_{1}x_{0}+\beta_{0}+\sigma\epsilon,\epsilon\sim\mathcal{N}(0,1)\iff
p_{\theta}(y_{0}|x_{0})\stackrel{{\scriptstyle
G}}{{=}}\mathcal{N}(y_{0};\beta_{1}x_{0}+\beta_{0},\sigma^{2})$. Once again,
thanks to the conjugacy of the prior $\pi_{X_{0}}(x_{0})$ and the previous
$p_{\theta}(y_{0}|x_{0})$, the posterior pdf $p(x_{0}|y_{0},\theta)$ admits a
closed-form expression:

$p(x_{0}|y_{0},\theta)\stackrel{{\scriptstyle
G}}{{=}}\mathcal{N}(x_{0};(\frac{\beta_{1}^{2}}{\sigma^{2}}+\frac{1}{\sigma_{X_{0}}^{2}})^{-1}(\frac{\beta_{1}(y_{0}-\beta_{0})}{\sigma^{2}}+\frac{\mu_{X_{0}}}{\sigma_{X_{0}}^{2}}),(\frac{\beta_{1}^{2}}{\sigma^{2}}+\frac{1}{\sigma_{X_{0}}^{2}})^{-1}).$
(10)

We now describe the discriminative modeling approach with the same
parameterized $\mathcal{P}_{\theta}$:

$x_{0}\stackrel{{\scriptstyle
D}}{{=}}\beta_{1}y_{0}+\beta_{0}+\sigma\epsilon,\epsilon\sim\mathcal{N}(0,1)\iff
p_{\theta}(x_{0}|y_{0})\stackrel{{\scriptstyle
D}}{{=}}\mathcal{N}(x_{0};\beta_{1}y_{0}+\beta_{0},\sigma^{2}).$ (11)

###  2.2 Handling multiple observations

The bayesian philosophy behind equation (1) is to assume (i) prior knowledge
on $x_{0}$ (before observation) in the form of a prior $\Pi_{X_{0}}$ and (ii)
an observation model. This observation model is assumed to produce $y_{0}$
from $x_{0}$ but it might not necessarily be true in practice. In this case,
we say that the observation model is mispecified w.r.t. the observation. In
our case, we considered the observation model to be the DGP
$\mathcal{P}_{Y|X}$, so (1) is a well specified Bayesian setting. Then, after
observing (one or) several observations:
$y_{0,1},...,y_{0,N_{0}}\stackrel{{\scriptstyle
iid}}{{\sim}}\mathcal{P}_{Y|X}(Y|X=x_{0})$. We deduce the posterior pdf:

$\displaystyle p_{X|Y_{0,1},...,Y_{0,N_{0}}}(x_{0}|y_{0,1},...,y_{0,N_{0}})$
$\displaystyle=\frac{\pi_{X_{0}}(x_{0})\prod_{n=1}^{N_{0}}p_{Y|X}(y_{0,n}|x_{0})}{\int\pi_{X_{0}}(x_{0})\prod_{n=1}^{N_{0}}p_{Y|X}(y_{n}|x_{0})\mathrm{d}y_{1}...\mathrm{d}y_{N_{0}}}.$
(12)

We often describe this equation as a form of Bayesian updating: we update the
prior knowledge with the observations. In section 3.4, we will discuss the
role of the prior $\Pi_{X_{0}}$ with regard to both modeling approaches; but
in this section, we first specifically examine whether or not each approach
enables to easily handle multiple observations in the inference of $x_{0}$.

Equations (6) and (7) explain how we can predict the value of $x_{0}$ from a
unique observed value of $y_{0}$ using model $\theta$ for respectively the
generative and the discriminative approach. In this case, both approaches
enable computation of the posterior $p(x_{0}|y_{0},\theta)$ as both equations
are tractable (at least up to a constant). By contrast, when we observe not
only a single observation but rather a collection of observations from the DGP
which originate from the same unknown value of interest, as in (12), then the
generative approach allows to handle this situation with a tractable
equivalent of (6), while the discriminative one does not.

Indeed, under a generative modeling, we can easily rewrite equation (6) as:

$p(x_{0}|y_{0,1},...,y_{0,N_{0}},\theta)\stackrel{{\scriptstyle
G}}{{=}}\frac{\pi_{X_{0}}(x_{0})\prod_{n=1}^{N_{0}}p_{\theta}(y_{0,n}|x_{0})}{p(y_{0,1},...,y_{0,N_{0}}|\theta)};$
(13)

and this formula can be computed up to its constant denominator (w.r.t.
$x_{0}$). On the other hand, with a discriminative modeling, equation (7)
becomes:

$p(x_{0}|y_{0,1},...,y_{0,N_{0}},\theta)\stackrel{{\scriptstyle
D}}{{=}}\frac{\prod_{n=1}^{N_{0}}p_{Y_{n}}(y_{0,n})p_{\theta}(x_{0}|y_{0,n})}{p(x_{0}|\theta)^{N_{0}-1}p_{Y_{1},...,Y_{N_{0}}}(y_{0,1},...,y_{0,N_{0}})}.$
(14)

However, factor $p(x_{0}|\theta)\stackrel{{\scriptstyle D}}{{=}}\int
p_{\theta}(x_{0}|y)p_{Y_{0}}(y)\mathrm{d}y$ is always intractable since
$p_{Y_{0}}(y)$ given by (1) is defined implicitely by the unknown DGP.
Therefore, (14) cannot be evaluated, not even up to a constant, when
$N_{0}>1$. Finally, only the generative approach allows to conveniently deal
with multiple observations. In order to carry on with the comparison of both
approaches, we only consider the case of a unique observation $y_{0}$, but,
concerning the generative modeling, the equations of the rest of this paper
still hold with multiple observations.

##  3 Supervised Epistemic Uncertainty via the ppd

We now discuss how the epistemic uncertainty is accounted for in each
approach, be it generative or discriminative. To that end we analyze how the
modeling choice impacts the ppd and more precisely how it can be sampled from.
We proceed in three steps: first we analyze the model posterior distribution
$p(\theta|y_{0},\mathcal{D})$ (see §3.1), we then deduce the joint
distribution
$p(x_{0},\theta|y_{0},\mathcal{D})\stackrel{{\scriptstyle\eqref{joint}}}{{=}}p(x_{0}|y_{0},\theta)p(\theta|y_{0},\mathcal{D})$
(see §3.2) and we finally come to its (other) marginal of interest, i.e. the
ppd (2) (see §3.3).

###  3.1 model posterior: $p(\theta|y_{0},\mathcal{D})$ or
$p(\theta|\mathcal{D})?$

In this section we look at the posterior distribution over model $\theta$
given the observation $y_{0}$ and the recorded dataset $\mathcal{D}$. Using
Bayes rule, it can be written as:

$p(\theta|y_{0},\mathcal{D})=\frac{p(y_{0}|\theta,\xcancel{\mathcal{D}})}{p(y_{0}|\mathcal{D})}p(\theta|\mathcal{D}),\text{
where
}p(\theta|\mathcal{D})\propto\pi_{\Theta}(\theta)\prod_{(x_{i},y_{i})\in\mathcal{D}}p(x_{i},y_{i}|\theta).$
(15)

It is important to note here that in the previous equation we can cancel out
$\mathcal{D}$ since for any variables involved in figure 3, we have
$p(.|.,\theta,\mathcal{D})=p(.|.,\theta)$. By glancing at these two equations,
we can already see that the probable values of $\theta$ under this posterior
correspond to models for which the elements of $\mathcal{D}$, the couples
$(x_{i},y_{i})$, are likely. In this section, we discuss the impact of $y_{0}$
on this distribution and conclude on whether or not this observation carries
information for inference of $\theta$ depending on the modeling approach. To
that hand, we start by leveraging equations (8) and (9) to deduce:

$\displaystyle p(y_{0}|\mathcal{D})\stackrel{{\scriptstyle G}}{{=}}\int
p(y_{0}|\theta)p(\theta|\mathcal{D})\mathrm{d}\theta\text{ where
}p(y_{0}|\theta)\stackrel{{\scriptstyle G}}{{=}}\int
p_{\theta}(y_{0}|x)\pi_{X_{0}}(x)\mathrm{d}x,$ (16) $\displaystyle
p(y_{0}|\theta)\stackrel{{\scriptstyle
D}}{{=}}p(y_{0}|\mathcal{D})\stackrel{{\scriptstyle D}}{{=}}p_{Y_{0}}(y_{0}).$
(17)

On the one hand, with a generative approach, $p(y_{0}|\theta)$ indeed depends
on $\theta$, so $y_{0}$ indeed carries information for inferring of $\theta$
since the two rv are not independent. We can moreover analyze how the
information is carried by $y_{0}$ to a posteriori models. Probable generative
models $\theta$ under posterior $p(\theta|y_{0},\mathcal{D})$ produce, with
high probability, the value $y_{0}$ for unknown values $x$ distributed under
the prior $\Pi_{X_{0}}$. The posterior distribution of models $\theta$
therefore effectively depends on $y_{0}$. We finally conclude that the role of
$y_{0}$ in the ppd inference is twofold: (i) in conjonction to the prior
$\Pi_{X_{0}}$ it indeed carries information to probable (epistemic) models
$\theta$ and (ii) it carries information to probable (aleatoric) $x_{0}$
values via posterior models $\theta$.

On the other hand, under a discriminative approach, factors $p(y_{0}|\theta)$
and $p(y_{0}|\mathcal{D})$ reduce to $p_{Y_{0}}(y_{0})$ (see (17)) so $\theta$
and $y_{0}$ are independent rv and finally the posterior over $\theta$ reduces
to $p(\theta|\mathcal{D})$. Let us analyze why observation $y_{0}$ does not
carry any information to a posteriori models. The information carried by
$y_{0}$ to a discriminative model $\theta$ is that it should produce, with
high probability, unknown values $x$ for $y_{0}$. However, this is nothing but
saying that $p_{\theta}(.|Y=y_{0})$ is a probability distribution, which we
already know by construction of a discriminative model using
$\mathcal{P}_{\theta}$ a conditional probability distribution. So, by contrast
with the generative approach, in the discriminative approach, the role of
$y_{0}$ is solely aleatoric, i.e. to infer $x_{0}$ via probable discriminative
models which do not depend on $y_{0}$.

###  3.2 Joint pdf

We now derive the joint pdf $p(x_{0},\theta|y_{0},\mathcal{D})$ given by
equation (3) for both generative and discriminative modeling approaches. In
the generative case (6), as explained before, the first factor in the joint
pdf is $p(x_{0}|y_{0},\theta)\stackrel{{\scriptstyle
G}}{{=}}p_{\theta}(y_{0}|x_{0})\pi_{X_{0}}(x_{0})/p(y_{0}|\theta)$. In
general, this expression can only be computed up to a normalizing constant
since $p(y_{0}|\theta)=\int p_{\theta}(y_{0}|x)\pi_{X_{0}}(x)\mathrm{d}x$
might be intractable. However, this denominator is a constant w.r.t. $x_{0}$
but it indeed depends on $\theta$ so it must not be treated as a constant in
the joint pdf; so, with regard to the joint pdf, the first factor cannot be
computed. Moreover, the second factor is $p(\theta|y_{0},\mathcal{D})\propto
p(y_{0}|\theta)p(\theta|\mathcal{D})$, and as we have explained before in the
previous section 3.1 indeed depends on $y_{0}$. For the same reason,
$p(y_{0}|\theta)$ is intractablethe second factor in the joint pdf cannot be
computed, not even up to a constant, either. Conveniently, both factors are
intractable because of the same factor $p(y_{0}|\theta)$ which appears in the
denominator of the first and in the numerator of the second. So, even though
none of the two factors can be computed individually, the intractable terms
cancel out by multiplication and the joint pdf can be computed up to a
constant (w.r.t. both $x_{0}$ and $\theta$) as:

$\displaystyle p(x_{0},\theta|y_{0},\mathcal{D})$
$\displaystyle\stackrel{{\scriptstyle
G}}{{=}}\frac{\pi_{X_{0}}(x_{0})p_{\theta}(y_{0}|x_{0})}{p(y_{0}|\theta)}\frac{p(\theta|\mathcal{D})p(y_{0}|\theta)}{p(y_{0}|\mathcal{D})}\stackrel{{\scriptstyle
G}}{{\propto}}\pi_{X_{0}}(x_{0})p_{\theta}(y_{0}|x_{0})p(\theta|\mathcal{D})$
(18) $\displaystyle\text{where }p(\theta|\mathcal{D})$
$\displaystyle\stackrel{{\scriptstyle
G}}{{\propto}}\pi_{\Theta}(\theta)\prod_{(x_{i},y_{i})\in\mathcal{D}}p_{\theta}(y_{i}|x_{i}).$
(19)

In the discriminative setting, the first factor in the joint pdf (3) reads
$p(x_{0}|y_{0},\theta)=p_{\theta}(x_{0}|y_{0})$ (see again equation (7)). This
quantity is directly computed in a normalized way by model
$\mathcal{P}_{\theta}$. Moreover, as we pointed out in the previous section
3.1, the second factor reduces to $p(\theta|\mathcal{D})$ which can be
computed up to a constant. So, unlike in the generative case, both factors can
be computed and the joint pdf therefore reads:

$\displaystyle p(x_{0},\theta|y_{0},\mathcal{D})$
$\displaystyle\stackrel{{\scriptstyle
D}}{{=}}p_{\theta}(x_{0}|y_{0})p(\theta|\mathcal{D})$ (20)
$\displaystyle\text{where }p(\theta|\mathcal{D})$
$\displaystyle\stackrel{{\scriptstyle
D}}{{\propto}}\pi_{\Theta}(\theta)\prod_{(x_{i},y_{i})\in\mathcal{D}}p_{\theta}(x_{i}|y_{i}).$
(21)

So in both modeling cases, the joint pdf can be computed (at least up to a
constant).

###  3.3 The ppd

Recall that the ppd (2) is obtained by marginalizing out the rv $\theta$ in
the joint distribution (3). The consequence is twofold: first its pdf is
obtained by integrating the joint pdf w.r.t. variable $\theta$; and second,
sampling from the joint distribution provides $x_{0}$ samples which are
distributed under the ppd. In this section, we discuss the first point.

By integrating (18) and (20) w.r.t. $\theta$, we obtain expressions for the
ppd (2) in both cases:

$\displaystyle p(x_{0}|y_{0},\mathcal{D})\stackrel{{\scriptstyle
G}}{{\propto}}\pi_{X_{0}}(x_{0})\int
p_{\theta}(y_{0}|x_{0})p(\theta|\mathcal{D})\mathrm{d}\theta$ (22)
$\displaystyle p(x_{0}|y_{0},\mathcal{D})\stackrel{{\scriptstyle D}}{{=}}\int
p_{\theta}(x_{0}|y_{0})p(\theta|\mathcal{D})\mathrm{d}\theta$ (23)

In practice, these two equations can only be used when exact computation of
the integral is feasible. Nonetheless, they remain relevant as we can analyze
them both to grasp a difference between the two approaches which provides with
another interpretation of the ppd. The first formula, in the generative case,
corresponds to Bayesian inference using the prior and the marginal likelihood
$p(y_{0}|x_{0},\mathcal{D})=\int
p_{\theta}(y_{0}|x_{0})p(\theta|\mathcal{D})\mathrm{d}\theta$; while the
second formula, in the discriminative case, corresponds to an averaging of
posterior predictions. So, in both cases, the ppd is an averaging of the
quantity $p_{\theta}$ (which is what is effectively computed with model
$\theta$) w.r.t. $p(\theta|\mathcal{D})$. This is to be contrasted with the
original definition of the ppd, defined as the average of predictions w.r.t.
$p(\theta|y_{0},\mathcal{D})$. These two definitions only coincide in the
discriminative case since the model computes directly the prediction (see (7))
and the posterior distribution $p(\theta|y_{0},\mathcal{D})$ reduces to
$p(\theta|\mathcal{D})$ (see section 3.1). As a consequence, this means that
$\int p(x_{0}|y_{0},\theta)p(\theta|\mathcal{D})\mathrm{d}\theta$ is an exact
construction of the ppd only in the discriminative case.

###  3.4 Explicit or Implicit prior

A main difference between the two approaches lies in the role of the $x_{0}$
marginal in the joint pdf $p(x_{0},y_{0}|\mathcal{D})$. This distribution is
of particular interest as it can be considered as a prior
$p(x_{0}|\mathcal{D})$ in the ppd $p(x_{0}|y_{0},\mathcal{D})$ which is the
object of interest in both modeling approach:

$p(x_{0}|y_{0},\mathcal{D})\propto
p(x_{0}|\mathcal{D})p(y_{0}|x_{0},\mathcal{D}).$ (24)

We can again leverage both equations (8) and (9) to deduce:

$\displaystyle p(x_{0}|\mathcal{D})\stackrel{{\scriptstyle
G}}{{=}}p(x_{0}|\theta)\stackrel{{\scriptstyle G}}{{=}}\pi_{X_{0}}(x_{0});$
(25) $\displaystyle p(x_{0}|\mathcal{D})\stackrel{{\scriptstyle D}}{{=}}\int
p(x_{0}|\theta)p(\theta|\mathcal{D})\mathrm{d}\theta\text{ where
}p(x_{0}|\theta)\stackrel{{\scriptstyle D}}{{=}}\int
p_{\theta}(x_{0}|y)p_{Y_{0}}(y)\mathrm{d}y.$ (26)

These two equations allow to understand that the $x_{0}$ marginal does not
play the same role in the generative and discriminative cases. While in the
former setting this immutable marginal distribution describes prior knowledge
and does not depend on $\mathcal{D}$ (see equation (25)); in the latter
setting, this marginal distribution is the result of an intricate interaction
between the dataset $\mathcal{D}$, the prior distribution $\Pi_{X_{0}}$ and
the DGP $\mathcal{P}_{Y|X}$. On the one hand, in the generative approach, it
corresponds to the prior $\Pi_{X_{0}}$ which can be specified according to the
problem at hand and, in itself, may provide significant information about the
value of interest $x_{0}$. In practice, this prior distribution can also play
a role of regularization and may as well be understood as a safeguard since it
can effectively constrain the prediction to a specific region of the space
[76][71][64] but more importantly, the prior distribution is often the result
of an elicitation effort [67] which consists in of (i) obtaining prior
information and (ii) transcribing this knowledge into a probability
distribution. On the other hand with a discriminative approach, this $x_{0}$
marginal has a very different role. The relevance of equation (26) first lie
in the fact that it highlights the systematic intractablity of pdf
$p(x_{0}|\mathcal{D})$. Indeed, it can never be computed (even if exact
integration was feasible) since it ultimately involves computing the pdf
$p_{Y|X}$ in $p_{Y_{0}}(y)=\int p_{Y|X}(y|x)\pi_{X_{0}}(x)\mathrm{d}x$, which
is unknown by assumption. This intractability does not pose any practical
issue since the computation of the $x_{0}$ marginal $p(x_{0}|\mathcal{D})$
(26) is not required for computing the joint pdf (20) (and consequently not
for computing the pdf of (or sampling from) the ppd). But this also means that
the discriminative construction does not allow to leverage any information
encapsulated in, or any practical property induced by, a prior distribution
during the inference. We now rewrite the expression of equation (26) as:

$p(x_{0}|\mathcal{D})\stackrel{{\scriptstyle D}}{{=}}\int\int\int
p_{\theta}(x_{0}|y)p(\theta|\mathcal{D})p_{Y|X}(y|x)\pi_{X_{0}}(x)\mathrm{d}x\mathrm{d}y\mathrm{d}\theta\stackrel{{\scriptstyle
D}}{{=}}\int p(x_{0}|y,\mathcal{D})p_{Y_{0}}(y)\mathrm{d}y.$ (27)

So, this distribution has a pdf $p(x_{0}|\mathcal{D})$ which indeed depends on
(i) $\mathcal{D}$ via the unknown $\theta$ and is indirectly related to (ii)
the prior $\Pi_{X_{0}}$ and (iii) the DGP $\mathcal{P}_{Y|X}$ via
$\mathcal{P}_{Y_{0}}$. As a consequence, this distribution will attribute high
probability mass to the $x$ values which have high probability under
$p(x_{0}|y,\mathcal{D})$ for some value $y\sim\mathcal{P}_{Y_{0}}$. As such,
an implicit density estimation mechanism $x_{1},...,x_{|\mathcal{D}|}$ of
$\mathcal{D}$ shifts the distribution $p(x_{0}|\mathcal{D})$ away from
$\Pi_{X_{0}}$ and towards the regions of high probability under
$\mathcal{P}_{X}^{\mathcal{D}}$. This implicit density estimation mechanism
appears clearly in the limiting case where the aleatoric uncertainty increases
since we observe that pdf $p(x_{0}|\mathcal{D})$ becomes
$p(x_{0}|x_{1},...,x_{|\mathcal{D}|})$. Conversely, when the aleatoric
uncertainty decreases, this pdf is, under assumptions of identifibility and
invertibility, $\pi_{X_{0}}(x_{0})$. We will illustrate this effect in both
contexts of regression (using the running example) and classification. As a
consequence, the ppd in the discriminative approach indeed does not provide
with an approximation of (1) with prior $\Pi_{X_{0}}$. It instead provides
with an approximation of:

$\frac{p_{X}^{\mathcal{D}}(x_{0})p_{Y|X}(y_{0}|x_{0})}{\int
p_{X}^{\mathcal{D}}(x)p_{Y|X}(y_{0}|x)\mathrm{d}x}.$ (28)

Consequently, a mismatch between the prior $\Pi_{X_{0}}$ and the distribution
$\mathcal{P}_{X}^{\mathcal{D}}$, which effectively generated the $x_{i}$
values in $\mathcal{D}$, will result in a mismatch between the target
posterior (1) and the ppd (2). Subsequently, only the regions of space which
are well represented by the $x_{i}$ values in dataset $\mathcal{D}$ will have
high probability mass under the marginal $p(x_{0}|\mathcal{D})$, and hence,
under the ppd $p(x_{0}|y_{0},\mathcal{D})$. Though this argument relates
$\mathcal{D}$ to the posterior $p(x_{0}|y_{0},\mathcal{D})$ (via the
distribution $p(x_{0}|\mathcal{D})$), we consider that this argument is not
related to epistemic uncertainty as (i) the effect does not vanish when the
number of recorded observation, i.e. the size of $\mathcal{D}$ increases; and
(ii) the same effect can be observed when considering
$p(x_{0}|y_{0},\theta^{*})$ where $\theta^{*}$ is a unique parameter (such as
MLE or MAP) estimate.
Finally, in the discriminative case, it is of particular interest to study the
distribution $p(x_{0}|\mathcal{D})$ as it corresponds to the average
prediction over observations $y_{0}$ since
$\mathbbm{E}_{y_{0}\sim\mathcal{P}_{Y_{0}}}\mathopen{}\left[{p(x_{0}|y_{0},\mathcal{D})}\,\negthickspace\right]\mathclose{}\stackrel{{\scriptstyle
D}}{{=}}p(x_{0}|\mathcal{D})$. This, together with the probability mass of
$p(x_{0}|\mathcal{D})$ which favors the regions of $x$ values in
$\mathcal{D}$, tells us that a discriminative model will favor the regions
which are well represented in the dataset. In a classification task, the
dominant labels will be predicted more often than the others, thus explaining
that discriminative models indeed suffer from imbalanced dataset. We further
emphasize this precise point using the illustrative running example, a
classification example provided in supplementary materials (see section
3.4.1), as well as in quantitative simulations in section 5.

#### Illustrative running example

We now leverage the affine regression example to illustrate the effects of the
implicit prior on the ppd in the discriminative modeling approach. We first
display an empirical approximation of the distribution $p(x_{0}|\mathcal{D})$.
To that end, using equation (27), we obtain samples from this distribution via
the two step sampling procedure $y\sim\mathcal{P}_{Y_{0}}$ and $x_{0}\sim
p(x_{0}|y,\mathcal{D})$ (the second sampling step is detailed in the next
paragraph 3.5). Of course in practice, the first sampling step cannot be
conducted as sampling from $p_{Y_{0}}(y)=\int
p_{Y|X}(y|x)\pi_{X_{0}}(x)\mathrm{d}x$ requires sampling from the DGP which we
recall is unknown by hypothesis, but in our example, we do resort to this
sampling procedure for illustration purposes.

Figure 4: Empirical estimate of $p(x_{0}|\mathcal{D})$ in the discriminative
setting: samples are obtained via $y\sim\mathcal{P}_{Y_{0}}$ (top) followed by
$x_{0}\sim p(x_{0}|y,\mathcal{D})$ (bottom). This distribution corresponds to
a trade-off between $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$ (right)

In figure 4, an empirical estimate of $p(x_{0}|\mathcal{D})$ is obtained via
the described two-step sampling procedure (upper-left and lower-left) and is
plotted against prior $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$. We
see that, in the discriminative setting, $p(x_{0}|\mathcal{D})$ (which we
recall acts as a prior in (24)) indeed corresponds to a trade-off between the
two distributions and is shifted towards $\mathcal{P}_{X}^{\mathcal{D}}$ via
an implicit density estimation mechanism from the $x$-values in $\mathcal{D}$.
In this example, we can also visualize how the DGP affects the balance between
$\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$ which we now illustrate in
the next figure 5.

Figure 5: Varying degrees of aleatoric uncertainty in the DGP yield the
distribution $p(x_{0}|\mathcal{D})$ to shift between $\Pi_{X_{0}}$ and
$\mathcal{P}_{X}^{\mathcal{D}}$.

In this figure, on the one hand, we see that for lower values of
$\sigma_{X|Y}$ (the noise standard deviation in the DGP (4)) the distribution
$p(x_{0}|\mathcal{D})$ gets closer to $\Pi_{X_{0}}$; while, on the other hand,
this distribution gets closer to $\mathcal{P}_{X}^{\mathcal{D}}$ for larger
values of $\sigma_{X|Y}^{2}$. This example seems to hint that when the DGP is
stained with high (resp. low) aleatoric uncertainty, the distribution
$p(x_{0}|\mathcal{D})$ leans more towards $\mathcal{P}_{X}^{\mathcal{D}}$
(resp. $\Pi_{X_{0}}$). In section 3.4.1, we provide with another example of
classification and observe a similar effect.

As we have mentionned before, it therefore follows that the ppd in the
discriminative case provides with an approximation of (28), which leads to a
visible mismatch between the ppd and the true (unknown) posterior when the
prior $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$ are different
probability distributions, which we now illustrate. To that end, we compare
this pdf to an histogram of samples from the ppd $p(x_{0}|y_{0},\mathcal{D})$
with large $\mathcal{D}$ to remove the effect of epistemic uncertainty and we
observe that they perfectly match. Conversely, when the prior $\Pi_{X_{0}}$
and $\mathcal{P}_{X}^{\mathcal{D}}$ are the same probability distribution then
the ppd indeed approximates the true ppd. This is illustrated in figure 6.

Figure 6: The $x_{0}$ marginal is inferred on $\mathcal{D}$: a mismatch
between the prior $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$ results in
a mislead approximation (first line). A discriminative approach is accurate in
the case where these two distributions match (second line)

We now illustrate a more problematic issue related to the same mechanism.
Because the dataset shifts the distribution $p(x_{0}|\mathcal{D})$, which acts
as a prior in the ppd, toward $\mathcal{P}_{X}^{\mathcal{D}}$ via an implicit
density estimation mechanism of $\mathcal{P}_{X}^{\mathcal{D}}$ with
$p(x_{0}|\mathcal{D})$ the ppd will only assign high probability to the
regions of space which are assigned high probability under
$p(x_{0}|\mathcal{D})$. The consequence of this in the case of affine modeling
is that we are not able to predict accurately outside of the support induced
by $\mathcal{D}$ as the ppd attributes little to no mass to the true value of
$x_{0}$. A discriminative affine model cannot extrapolate to regions outside
of the support of $\mathcal{D}$ and this conclusion argues, for once, in
disfavor of a discriminative approach since an affine model, amongst all
models, is expected to extrapolate well. Conversely, as a result of the
explicit prior, the generative approach does not suffer from the same
shortcoming and we observe that the affine generative model indeed produces a
ppd which assigns high probability to the true value of $x_{0}$ and
approximates the true unknown posterior. This is illustrated in figure 7.

Figure 7: Illustration of inaccurate discriminative prediction when outside of
the support of $\mathcal{D}$

####  3.4.1 Classification example

Figure 8: Contour plot of the DGP used in the classification example

In the previous example of regression, we illustrated how a discriminative
modeling approach builds an implicit prior via a density estimation mechanism,
resulting in a poor approximation in the case of a mismatch between
$\mathcal{P}_{X}^{\mathcal{D}}$ and the desired prior $\Pi_{X_{0}}$. We also
illustrated how, in a discriminative model, the dataset $\mathcal{D}$, the
prior $\Pi_{X_{0}}$ and the DGP interact to yield $p(x_{0}|\mathcal{D})$
which, via an implicit density estimation mechanism, is a trade-off between
$\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$. We now also illustrate this
specific effect on a classification problem where $X$ is a Categorical
variable taking value $c=1,...,C$.

We consider a 2-dimensional example with $C=3$ where the DGP reads:
$p_{Y|X=c}(y)=\mathcal{N}({\scriptstyle
y;r_{Y|X}\mathopen{}\left[{\mathrm{Re}(\omega^{c}),\mathrm{Im}(\omega^{c})}\,\negthickspace\right]\mathclose{}^{T},I_{2}})$
with $\omega=e^{2i\pi/3}$. In this DGP, the aleatoric uncertainty can be
controlled via the value of $r_{Y|X}$ which describes the distance of each
Gaussian class to the origin ($r\\_$ stands for radius). Indeed, the further
the different classes are from each other, the easier it is to accurately
classify a sample $y\sim\mathcal{P}_{Y_{0}}$ to its according unknown label.
The goal of this section is to illustrate the effect of the roles of
distributions $\mathcal{P}_{X}^{\mathcal{D}}$ and $\Pi_{X_{0}}$ in the
discriminative approach. We therefore select the two distributions
$\mathcal{P}_{X}^{\mathcal{D}}$ from be different to one another:
$\mathrm{Pr}(X=c)=(4-c)/6$ for $X\sim\Pi_{X_{0}}$ versus
$\mathrm{Pr}(X=c)=c/6$ for $X\sim\mathcal{P}_{X}^{\mathcal{D}}$ for labels
$c=1,2,3$. The distribution $\mathcal{P}_{X}^{\mathcal{D}}$ defines the
frequencies of classes, and together with the DGP can produce a toy dataset
$\mathcal{D}$ which is illustrated in the next figure 9.

Figure 9: Dataset $\mathcal{D}$ with class distribution
$\mathcal{P}_{X}^{\mathcal{D}}$

As a discriminative model, we use a multinomial Logistic classifier which
class probability reads:

$\mathrm{Pr}_{\theta}(X=c|Y)=\frac{\exp(\beta_{c}^{T}Y+\beta_{c,0})}{\sum_{c=1}^{C}\exp(\beta_{c}^{T}Y+\beta_{c,0})};$
(29)

where
$\theta=\mathopen{}\left\\{{\beta_{1},\beta_{1,0},...,\beta_{C},\beta_{C,0}}\,\negthickspace\right\\}\mathclose{}$
with $\beta_{c},\beta_{c,0}\in\mathbbm{R}^{2}\times\mathbbm{R}$. Similarly to
the continued regression example, we will display an empirical estimate of
distribution $p(x_{0}|\mathcal{D})$ (26) obtained by the sampling procedure:
$y\sim\mathcal{P}_{Y_{0}}$ (via the unknown DGP which we use only for
illustrative purposes), $\theta\sim p(\theta|\mathcal{D})$ and
$x_{0}\sim\mathrm{Categorical}(\mathrm{Pr}_{\theta}(X=1|Y),\mathrm{Pr}_{\theta}(X=2|Y),\mathrm{Pr}_{\theta}(X=3|Y))$
given by equation (29).

A prior distribution over parameters $\Pi_{\theta}$ which would be conjugate
to this logistic model would lead to a posterior $p(\theta|\mathcal{D})$
available in close form, but unfortunately, such a conjugate prior does not
exists amongst the usual probability distributions. We therefore use a simple
Gaussian prior over parameter $\theta$ and resort to sampling from the ppd
using an Metropolis-Hastings MCMC algorithm, though a Gibbs sampling scheme is
also available in this setting [34]. In the next figure, we display empirical
estimates of $p(x_{0}|\mathcal{D})$ against $\Pi_{X_{0}}$ and
$\mathcal{P}_{X}^{\mathcal{D}}$.

Figure 10: Varying degrees of aleatoric uncertainty in the DGP yield the
distribution $p(x_{0}|\mathcal{D})$ to shift between $\Pi_{X_{0}}$ and
$\mathcal{P}_{X}^{\mathcal{D}}$.

This figure once again illustrates that distribution $p(x_{0}|\mathcal{D})$
(which we recall acts as a prior in the ppd (24)) corresponds to a trade-off
between $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$. With a very similar
interpretation to that of figure 5, this figure also seems to hint that the
dynamic of the DGP dictates the trade-off between the two distributions: high
(resp. low) aleatoric uncertainty shifts $p(x_{0}|\mathcal{D})$ more towards
$\mathcal{P}_{X}^{\mathcal{D}}$ (resp. $\Pi_{X_{0}}$).

###  3.5 Gibbs sampling from the ppd

We now come to sampling from the ppd. Samples from the ppd can be obtained in
two distinct ways. If exact computation of expectation w.r.t.
$p(\theta|\mathcal{D})$ is feasible, then (22) and (23) give tractable
expressions (possibly up to a constant) for the ppd. However, the integrals in
these equations admit closed form expressions only in specific cases when
using conjugate models, and exact integration is, more often than not,
unfeasible making the posterior pdf intractable. So in practice, we rather
resort to sampling via the joint distribution since, as we explained before,
sampling from the joint distribution (3) produce samples $x_{0}$ that are
distributed according to the ppd (2).

With that regard, in both modeling cases, the joint pdf, (18) and (20), can be
computed (at least up to a constant), and so we can sample from the joint
distribution via the pdf. More conveniently, the discriminative approach
yields a specific factorization of its joint pdf (20) which enables a
sequential sampling procedure with $\theta\sim p(\theta|\mathcal{D})$ and
$x_{0}\sim p_{\theta}(x_{0}|y_{0})$. It is therefore motivated to construct
models for which the posterior probability distribution of parameters By
contrast, the generative approach does not induce the same factorization and
does not benefit from the same convenient sequential sampling scheme. In this
section, we propose a general scheme for sampling from the ppd which can be
applied to both modeling approach.

We now propose a scheme that enables to sample from the joint distribution
(3), and therefore from its $x_{0}$ marginal which is nothing but the ppd (2).
This sampling scheme is based on the notorious Gibbs sampling [10][24] which
is an MCMC algorithm [68] that applies specifically to a joint distribution
$p(u,v)$ with a markov transition
$q_{t}(u^{(t+1)},v^{(t+1)}|u^{(t)},v^{(t)})=p(u^{(t+1)}|v^{(t+1)})p(v^{(t+1)}|u^{(t)})$.
This transition leaves the joint distribution $p(u,v)$ invariant since the
Gibbs algorithm can be seen as two steps of Metropolis Hastings [12]
transition where the acceptance probability is $1$. We apply the principle of
Gibbs sampling to the joint distribution $p(x_{0},\theta|y_{0},\mathcal{D})$
in the generative (resp. discriminative) setting. First, conditionally on the
current model $\theta^{(t-1)}$, $x_{0}$ is distributed according to the
posterior for that model. So the first step of the markov transition is to
draw $x_{0}^{(t)}$ from equation (6) (resp. (7)) with $\theta^{(t-1)}$. Then,
conditionally on the current value of $x_{0}^{(t)}$, $\theta$ is distributed
according to $p(\theta|x_{0}^{(t)},y_{0},\mathcal{D})$ and so the second step
of the markov transition consists in sampling $\theta^{(t)}\sim
p(\theta|\mathcal{D}_{+}^{(t)})$ with the analogous of (19) (resp. (21)) where
$\mathcal{D}_{+}^{(t)}=\mathcal{D}\cup\\{(x_{0}^{(t)},y_{0})\\}$. We summarize
this Gibbs sampler in algorithm 1 (for the moment readers should disregard the
red parts of the algorithm, as they are related to the semi-supervised
learning task covered in section 4).

Though this algorithm is written in a similar fashion in both modeling
approaches, the conclusions with regard to the different behaviours of the two
modeling approaches presented in the previous sections still hold as they are
the result of a structural difference between the generative and
discriminative approach. This algorithm will be especially useful in the semi-
supervised setting, which we describe in the next section 4.

In the case of multiple observations $y_{0,1},...,y_{0,N_{0}}$ as in section
2.2, the previous algorithm can be effortlessly adjusted in the generative
case (recall that the discriminative case is not compatible with multiple
observations, see section 2.2). Indeed in this setting, at time $t$, we first
sample $x_{0}^{(t)}$ according to (13) for current model $\theta^{(t-1)}$; and
then the dataset $\mathcal{D}$ is augmented into
$\mathcal{D}_{+}^{(t)}=\mathcal{D}\bigcup_{i=1}^{N_{0}}\\{(x_{0}^{(t)},y_{0,i})\\}$.

#### Illustrative running example

We now come back to the continued example of affine modeling to provide an
example of the Gibbs algorithm mechanism. We assume prior knowledge over
parameter $\theta$ in the form of
$\pi_{\Theta}(\theta)=\mathcal{N}(\beta;\mu_{\beta},\Sigma_{\beta})\mathrm{I}\Gamma(\sigma^{2};\lambda,\eta)$
where $\mu_{\beta}\in\mathbb{R}^{2}$ and $\Sigma_{\beta}\in\mathbb{R}^{2\times
2}$ is a covariance matrix, $\mathrm{I}\Gamma$ is the pdf associated with an
inverse Gamma distribution and $\lambda,\eta>0$ are respectively the shape and
scale parameters of the corresponding Gamma distribution. Unfortunately, the
posterior $p(\theta|\mathcal{D})$ does not admit a closed form expression;
but, at least, this choice of conjugate priors allows for both conditionals to
be tractable. We first explicit these two conditionals pdf in the generative
(resp. discriminative) setting:

$\displaystyle p(\beta|\sigma^{2},\mathcal{D})\stackrel{{\scriptstyle
G}}{{=}}\mathcal{N}(\beta;(\frac{\mathbf{X}^{T}\mathbf{X}}{\sigma^{2}}+\Sigma_{\beta}^{-1})^{-1}(\frac{\mathbf{X}^{T}\mathbf{Y}}{\sigma^{2}}+\Sigma_{\beta}\mu_{\beta}),(\frac{\mathbf{X}^{T}\mathbf{X}}{\sigma^{2}}+\Sigma_{\beta}^{-1})^{-1}),$
(30) $\displaystyle\text{ where }\mathbf{X}\stackrel{{\scriptstyle
G}}{{=}}\begin{bmatrix}x_{1},1\\\ ...\\\
x_{|\mathcal{D}|},1\end{bmatrix},\mathbf{Y}\stackrel{{\scriptstyle
G}}{{=}}\begin{bmatrix}y_{1}\\\ ...\\\ y_{|\mathcal{D}|}\end{bmatrix};$
$\displaystyle p(\sigma^{2}|\beta,\mathcal{D})\stackrel{{\scriptstyle
G}}{{=}}\mathrm{I}\Gamma(\sigma^{2};\lambda+\frac{|\mathcal{D}|}{2},\eta+\sum_{i=1}^{|\mathcal{D}|}\frac{(y_{i}-\beta_{1}x_{i}-\beta_{0})^{2}}{2}).$
(31) $\displaystyle p(\beta|\sigma^{2},\mathcal{D})\stackrel{{\scriptstyle
D}}{{=}}\mathcal{N}(\beta;(\frac{\mathbf{Y}^{T}\mathbf{Y}}{\sigma^{2}}+\Sigma_{\beta}^{-1})^{-1}(\frac{\mathbf{Y}^{T}\mathbf{X}}{\sigma^{2}}+\Sigma_{\beta}\mu_{\beta}),(\frac{\mathbf{Y}^{T}\mathbf{Y}}{\sigma^{2}}+\Sigma_{\beta}^{-1})^{-1}),$
(32) $\displaystyle\text{ where }\mathbf{Y}\stackrel{{\scriptstyle
D}}{{=}}\begin{bmatrix}y_{1},1\\\ ...\\\
y_{|\mathcal{D}|},1\end{bmatrix},\mathbf{X}\stackrel{{\scriptstyle
D}}{{=}}\begin{bmatrix}x_{1}\\\ ...\\\ x_{|\mathcal{D}|}\end{bmatrix};$
$\displaystyle p(\sigma^{2}|\beta,\mathcal{D})\stackrel{{\scriptstyle
D}}{{=}}\mathrm{I}\Gamma(\sigma^{2};\lambda+\frac{|\mathcal{D}|}{2},\eta+\sum_{i=1}^{|\mathcal{D}|}\frac{(x_{i}-\beta_{1}y_{i}-\beta_{0})^{2}}{2}).$
(33)

So in both modeling cases, the posterior distribution $p(\theta|\mathcal{D})$
can be sampled from using a Gibbs scheme by sequentially sampling these two
conditionnals. Then, from a Gibbs sampling of affine models, we can almost
effortlessly obtain samples from the ppd by including the additional step of
sampling $x_{0}$ from $p(x_{0}|y_{0},\theta)$ for the current model parameters
within the Gibbs sequential Markovian transition. Again, in supplementary
material, we summarize this Gibbs sampler in the specific case of affine
homoskedastic modelling, see algorithm 2 (readers should disegard the steps in
red for now, as they are related to semi-supervised learning which we now
discuss).

##  4 Bayesian Semi-Supervised learning

In this section, we now build upon the equations, arguments and conclusions
presented in the previous sections to tackle the problem of semi-supervised
learning. As we have mentionned before, supervised learning techniques use the
observed variables and their corresponding labels to build a model which
capture the dependency between two rv and which can be used to make
predictions about the label conditionally on the value of observed rv.
Conversely, unsupervised learning [33][31] take interest in learning pattern
in a data distribution without considering the notion of associated labels.
Structure can be represented by data clusters obtained by K-means [46][45],
graph-based clustering methods such as spectral clustering [57] or Louvain
method [7], or via maximum-likelihood in a mixture probability distribution
model [15]. In the beginning of this paper, we explained that the Generative
approach for modeling the unknown posterior, should not be confused with the
tasks and techniques of Generative modeling. These techniques can also be
considered as unsupervised learning as they enable to capture the structure
from univariate data such that the corresponding probabilistic model can be
sampled from easily in order to obtain observations which are approximately
distributed according to the dataset distribution. Most popular methods
include Variational AutoEncoders [40], Generative Adversarial Networks [27],
Normalizing Flows [60] and Diffusion models [70] but this is beyond the scope
of this paper. Semi-supervised learning does however lie within the scope of
this paper as it aims to obtain a conditional model to predict label from
observations, but the goal is to infer the model from both a labeled dataset
and unlabeled observations. In this section we now build upon the previous
arguments and discuss the compatibility of both learning approach with this
learning task.

###  4.1 The learning task

In section 2 and onward, we presented the general task of learning a model for
the posterior (1) using a set of _labeled_ observations $\mathcal{D}$, and how
to predict about an $x_{0}$ given a corresponding observation $y_{0}$ with the
ppd, which we now refer to as a supervised learning task. However, in many
statistical learning settings, we also dispose of _unlabeled_ observations.
They corresponds to values $\widetilde{y}_{j}$, which we know (or assume) are
produced by the DGP, but for an unknown values $\widetilde{x}_{j}$ for which
we assume prior knowledge $\Pi_{\widetilde{X}_{j}}$:
$\mathcal{Y}=\\{\widetilde{y}_{j}|\exists\widetilde{x_{j}}\sim\Pi_{\widetilde{X}_{j}},\widetilde{y}_{j}\sim\mathcal{P}_{Y|X}(Y|X=\widetilde{x_{j}})\\}_{j=1}^{|\mathcal{Y}|}$.
When the observations in $\mathcal{D}$ and $\mathcal{Y}$ cover different
regions of the observation space, and/or when $\mathcal{Y}$ has a significant
amount of elements, then the unlabeled observations $\mathcal{Y}$ may contain
significant or non-negligible information [58]. In this context, a semi-
supervised learning task aims at inferring a model from both labeled and
unlabeled observations. This question has risen in importance in importance
where we dispose of a lot of unlabeled observations, but where the labelling
tasks is expensive (as it is the case when the labelling needs to be conducted
by a human operator).

Figure 11: Affine semi-supervised regression setting

The ppd (2) then becomes:

$p(x_{0}|y_{0},\mathcal{D},\mathcal{Y})=\int_{\Theta}p(x_{0}|y_{0},\theta)p(\theta|y_{0},\mathcal{D},\mathcal{Y})\mathrm{d}\theta,$
(34)

and we aim to compute this pdf, or sample this distribution if exact
computation of the integral is not feasible. This formulation is more general
than the one described in section 2 and it reduces to supervised learning in
the case where $\mathcal{Y}=\varnothing$.

Throughout this section, we will carry on using the continued example of
affine modeling to illustrate the arguments and conclusions. To that end, we
suppose that, in addition to $\mathcal{D}$, we also dispose of unlabeled
observations $\widetilde{y}_{j}$ produced from the DGP (4) via an unknown
label $\widetilde{x}_{j}$ for which we suppose prior knowledge in the form of
a prior $\Pi_{\widetilde{X}}$ which is supposed to be the same for all
$j=1,...,|\mathcal{Y}|$ and which we consider to be Gaussian
$\pi_{\widetilde{X}}(\widetilde{x}_{j})=\mathcal{N}(\widetilde{x}_{j};\mu_{\widetilde{X}},\sigma^{2}_{\widetilde{X}})$.
The semi-supervised setting is illustrated in figure 11.

###  4.2 Both modeling confronted to the semi-supervised learning task

We now confront the two modeling approaches to the specific problem of semi-
supervised learning by analysing the model posterior which reads:

$p(\theta|y_{0},\mathcal{D},\mathcal{Y})=p(\theta|\mathcal{D})\frac{p(y_{0}|\theta)p(\mathcal{Y}|\theta)}{p(y_{0},\mathcal{Y}|\mathcal{D})}.$
(35)

We first explain that the discriminative approach does not allow for Bayesian
semi-supervised learning. To see this, recall the conclusion of section 3.1:
when we do not know $x_{0}$, the posterior over $\theta$ does not depend on
$y_{0}$ and so this observation does not carry any information to the
discriminative models. Therefore, the same applies for the elements of
$\mathcal{Y}$: since we do not know the label $\widetilde{x}_{j}$, the
unlabeled observation does not bring any information on $\theta$ as the
posterior over models does not depend on $\widetilde{y}_{j}$. Finally, the
model posterior (35) reduces to $p(\theta|\mathcal{D})$, (34) reduces to (23)
and all the other equations in section concerning the discriminative modeling
approach remain unchanged.

Conversely, the generative approach indeed allows for semi-supervised
learning. In section 3.1, we explained that, even though we do not know the
value of $x_{0}$, the posterior distribution over models still depends on the
observation $y_{0}$ indirectly through the prior $\Pi_{X_{0}}$. With a similar
argument, we understand that the unlabeled data $\mathcal{Y}$ indeed carry
information on model $\theta$. We write:
$p(\mathcal{Y}|\theta)\stackrel{{\scriptstyle
G}}{{=}}\prod_{j=1}^{|\mathcal{Y}|}\int
p_{\theta}(\widetilde{y}_{j}|\widetilde{x}_{j})\pi_{\widetilde{X}_{j}}(\widetilde{x}_{j})\mathrm{d}\widetilde{x}_{j}$.
So, even though we do not know the label $\widetilde{x}_{j}$, probable
generative models $\theta$ under (35) produce, with high probability, the
value of $\widetilde{y}_{j}$ for some unknown label distributed under the
prior $\Pi_{\widetilde{X}_{j}}$. In a classification setting, this term can
indeed be computed as a tractable finite sum [4][37], but in general, this
term is only available in integral form making the joint pdf
$p(x_{0},\theta|y_{0},\mathcal{D},\mathcal{Y})\stackrel{{\scriptstyle
G}}{{\propto}}\pi_{X_{0}}(x_{0})p_{\theta}(y_{0}|x_{0})p(\theta|\mathcal{D})p(\mathcal{Y}|\theta)$,
intractable, not even up to a constant. This raises the question of sampling
from the ppd since its pdf is intractable. In the next section, we propose to
use a variation of the Gibbs algorithm presented in section 3.5, and which
allows to sample from this ppd.

###  4.3 A Gibbs sampling algorithm for semi-supervised learning

In this section we extend the previous Gibbs sampling algorithm presented in
section 3.5 for sampling from the ppd (34) in the case of generative semi-
supervised learning. To that end, we apply the Gibbs mechanism of sequentially
sampling the conditional distributions in the joint distribution
$p(x_{0},\widetilde{x}_{1},...,\widetilde{x}_{|\mathcal{Y}|},\theta|y_{0},\mathcal{D},\mathcal{Y})$.
Firstly, conditionally on the current value of $\theta^{(t-1)}$, the labels
$x_{0},\widetilde{x}_{1},...,\widetilde{x}_{|\mathcal{Y}|}$ are independent
and each distributed according to its own posterior distribution. So the first
step of the Gibbs markovian transition is to sample $x_{0}^{(t)}\sim
p(x_{0}|y_{0},\theta^{(t-1)})$ with equation (6) (as in section 3.5) and
$\widetilde{x}_{j}^{(t)}\sim
p(\widetilde{x}_{j}|\widetilde{y}_{j},\theta^{(t-1)})$. Secondly,
conditionally on the current label values
$x_{0}^{(t)},\widetilde{x}_{1}^{(t)},...,\widetilde{x}_{|\mathcal{Y}|}^{(t)}$,
the model parameters are distributed according to
$p(\theta|\mathcal{D},x_{0}^{(t)},y_{0},\widetilde{x}_{1}^{(t)},\widetilde{y}_{1},...,\widetilde{x}_{|\mathcal{Y}|}^{(t)},\widetilde{y}_{|\mathcal{Y}|})$.
So the second step of the Gibbs markovian transition is to sample
$\theta^{(t)}\sim p(\theta|\mathcal{D}_{+}^{(t)})$ analogous of equation (19)
where
$\mathcal{D}_{+}^{(t)}=\mathcal{D}\cup(x_{0}^{(t)},y_{0})\cup\\{(\widetilde{x}^{(t)}_{j},\widetilde{y}_{j})\\}_{j=1}^{|\mathcal{Y}|}$.
We summarize this Gibbs mechanism in algorithm 1 and we highlight in red the
steps which are effectively responsible for semi-supervised learning.

0:
$y_{0},\mathcal{D},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{Y}}$,
number of steps $T$

$\theta^{(0)}\sim p(\theta|\mathcal{D})$

for $t=1$ to $T$ do

$x_{0}^{(t)}\sim p(x_{0}|y_{0},\theta^{(t-1)})$ with (6) (resp. (7)

for all $\widetilde{y}_{j}\in\mathcal{Y}$ do

sample $\widetilde{x}_{j}^{(t)}\sim
p(\widetilde{x}_{j}|\widetilde{y}_{j},\theta^{(t-1)})=\frac{p_{\theta^{(t-1)}}(\widetilde{y}_{j}|\widetilde{x}_{j})\pi_{\widetilde{X}_{j}}(\widetilde{x}_{j})}{p(\widetilde{y}_{j}|\theta^{(t-1)})}$

end for

set
$\mathcal{D}_{+}^{(t)}=\mathcal{D}\cup(x_{0}^{(t)},y_{0})$$\cup\\{(\widetilde{x}^{(t)}_{j},\widetilde{y}_{j})\\}_{j=1}^{|\mathcal{Y}|}$
and $\theta^{(t)}\sim p(\theta|\mathcal{D}_{+}^{(t)})$ with (19) (resp. (21))

end for

return $x_{0}^{(T)}$

Algorithm 1 Gibbs sampling from
$p(x_{0}|y_{0},\mathcal{D},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{Y}})$
in the generative (resp. discriminative) setting

In the case of semi-supervised learning setting, this Gibbs algorithm is all
the more crucial. Indeed, while in the supervised context the Gibbs approach
was only an alternative option to sampling from the joint distribution which
pdf (18) could be computed up to a constant; in the case of semi-supervised
learning however, it is possible that this joint pdf cannot be evaluated, not
even up to a constant and the Gibbs approach is therefore very convenient for
sampling the corresponding ppd.

We have written the Gibbs algorithm with including $\mathcal{Y}$ to perform
semi-supervised learning for both modeling approach but of course, as we have
mentionned before, the semi-supervised ppd (34) reduces to the supervised ppd
(2) in the discriminative setting, so this Gibbs algorithm, even though it
involves $\mathcal{Y}$, is not able to leverage any information from the
unlabeled observations in this modeling approach. We therefore would like to
stress that the semi-supervised learning is not enabled by the Gibbs procedure
itself but rather by using a generative modeling instead of a discriminative
one which induces different conditional dependency between all the rv. We
proposed the Gibbs algorithm as a way to sample the joint distributions
$p(x_{0},\theta|y_{0},\mathcal{D},\mathcal{Y})$ in the case of generative
modeling (with possibly $\mathcal{D}=\varnothing$ in the supervised setting)
which is particularly convenient to use because of the conditional
independence (w.r.t. $\theta$) of labels
$x_{0},\widetilde{x}_{1},...,\widetilde{x}_{|\mathcal{Y}|}$. However, this
Gibbs scheme it is only a possible approach for sampling from the
corresponding ppd which effectively depends on $\mathcal{Y}$.

We now come back to the continued example of affine modeling and use it to
illustrate the practical use of the Gibbs sampling algorithm which we use to
illustrate, respectively, the compatibility and incompatibility between the
generative and discriminative approaches and the semi-supervised learning. In
algorithm SM3.1 we first explicit how to build upon the supervised Gibbs
sampling algorithm of the supervised ppd presented in section 3.5 to obtain a
Gibbs sampling scheme of the semi-supervised ppd.

In this case, the generative and discriminative approach yield two different
equations for the posterior $p(\widetilde{x}_{j}|\widetilde{y}_{j},\theta)$:

$\displaystyle
p(\widetilde{x}_{j}|\widetilde{y}_{j},\beta,\sigma^{2})\stackrel{{\scriptstyle
G}}{{=}}\mathcal{N}(\widetilde{x}_{j};(\frac{1}{\sigma^{2}}+\frac{1}{\sigma_{\widetilde{X}}^{2}})^{-1}(\frac{\beta_{1}(\widetilde{y}_{j}-\beta_{0})}{\sigma^{2}}+\frac{\mu_{\widetilde{X}}}{\sigma_{\widetilde{X}}^{2}}),(\frac{1}{\sigma^{2}}+\frac{1}{\sigma_{\widetilde{X}}^{2}})^{-1}),$
(36) $\displaystyle
p(\widetilde{x}_{j}|\widetilde{y}_{j},\beta,\sigma^{2})\stackrel{{\scriptstyle
D}}{{=}}\mathcal{N}(\widetilde{x}_{j};\beta_{1}\widetilde{y}_{j}+\beta_{0},\sigma^{2}).$
(37)

The Gibbs sampling procedure for semi-supervised learning of affine
homoskedastic is summarized in algorithm 2. Again, the steps highlighted in
red are indeed responsible for leveraging information from the unlabeled
observations.

0: $y_{0},\mathcal{D}$, $\mathcal{Y}$, number of steps $T$

${\sigma^{2}}^{(0)}\sim\mathrm{I}\Gamma(\sigma^{2};\lambda,\eta)$

$\beta^{(0)}\sim p(\beta|{\sigma^{2}}^{(0)},\mathcal{D})$ with (30)

for $t=1$ to $T$ do

$x_{0}^{(t)}\sim p(x_{0}|y_{0},\beta^{(t-1)},{\sigma^{2}}^{(t-1)})$ with (10)
(resp. (11)

for all $\widetilde{y}_{j}\in\mathcal{Y}$ do

$\widetilde{x}_{j}^{(t)}\sim
p(\widetilde{x}_{j}|\widetilde{y}_{j},\beta^{(t-1)},{\sigma^{2}}^{(t-1)})$
with (36) (resp. (37))

end for

set
$\mathcal{D}_{+}^{(t)}=\mathcal{D}\cup(x_{0}^{(t)},y_{0})$$\cup\\{(\widetilde{x}^{(t)}_{j},\widetilde{y}_{j})\\}_{j=1}^{|\mathcal{Y}|}$,
${\sigma^{2}}^{(t)}\sim p(\sigma^{2}|\beta^{(t-1)},\mathcal{D}_{+}^{(t)})$
with (31) (resp. (33))

$\beta^{(t)}\sim p(\beta|{\sigma^{2}}^{(t)},\mathcal{D}_{+}^{(t)})$ with
(30)(resp. (32))

end for

return $x_{0}^{(T)}$

Algorithm 2 Gibbs sampling from $p(x_{0}|y_{0},\mathcal{D},$$\mathcal{Y}$$)$
using a generative (resp. discriminative) homoskedastic affine model

We will now illustrate whether or not the modeling approach enables leveraging
information in $\mathcal{Y}$ to contribute in reducing the epistemic
uncertainty. We first provide empirical evidence that the discriminative
approach is unable to leverage information in unlabeled observations to reduce
the epistemic uncertainty. To that end, we consider a setting where
$\mathcal{D}$ contains few points leading to high epistemic uncertainty to be
reduced; and we apply the previous Gibbs sampling algorithm SM3.1 in the
discriminative case with and without unlabeled observations $\mathcal{Y}$ and
visually observe that the resulting samples seem to follow the same
distribution. This empirical illustration is presented in figure 12.

Figure 12: Samples from the supervised and semi-supervised Gibbs algorithm in
affine discriminative modeling

We further perform a Kolmogorov-Smirnof statistical test where the null
hypothesis is $H_{0}:$ the samples obtained from the supervised and semi-
supervised Gibbs sampling algorithm are from the same (unknown) distribution.
This test works from two set of iid samples; but the samples from the Gibbs
sampling algorithm are correlated samples so we first extract (almost)
uncorrelated samples sub-sampling the Markov chain at every each integrated
auto-correlation time steps. The Kolmogorov-Smirnof test yields a $p$-value of
$0.566$ and we cannot reject the null hypothesis with low error probability;
which indicates that the data i.e. the two sets of de-correlated samples is
consistent with the null hypothesis i.e. that they originate from the same
underlying probability distribution.

Conversely, in the generative modeling approach, we go back to the setting
presented in the previous figure 11, and we compare the supervised and semi-
supervised ppds, from which we obtain samples via the corresponding Gibbs
sampling algorithm 2 with and without $\mathcal{Y}$. The results are presented
in the next figure 13(a). We compare the empirical distributions (built an
histogram of the samples) and we visually observe that the semi-supervised ppd
provides with a better of the true unknown posterior than the supervised ppd.
To go further than a visual interpretation, we compare the calibration curves
of the both ppds. The calibration curve allows to assess the quality of an
approximation and is computed from the pdf of the target posterior, in our
case the true unknown posterior and from samples from the approximating
distribution, in our case the (supervised or semi-supervised) ppds. It is
built by computing, for values $\alpha\in[0,1]$, the $\alpha$-highest density
region of the target distribution using the pdf and computing the proportion
of samples which land in that region. We can therefore conclude that the
unlabeled observation were taken into account during the inference and indeed
contributed to reducing the epistemic uncertainty.

(a) Unlabeled observations $\mathcal{Y}$ indeed contribute to reducing the
generative epistemic uncertainty.

(b) Calibration curves of supervised and semi-supervised ppds.

Figure 13: Semi-supervised generative affine modeling

###  4.4 Parallel inference

In this section we propose to re-discuss the problems of supervised and semi-
supervised learning by considering them as solving two (or several) posterior
inferences at the same time. To that end, we now denote
$\mathcal{X}=\\{\widetilde{x}_{j}|\widetilde{y}_{j}\sim\mathcal{P}_{Y|X}(Y|X=\widetilde{x}_{j}\\}_{j=1}^{|\mathcal{Y}|}$
the set of unknown labels associated to the unlabeled observations. On the one
hand, $\mathcal{X}$ are not necessarily label values of interest in the
initial problem (that of inferring $x_{0}$ via $y_{0}$), it is nonetheless an
unknown rv related to $\mathcal{Y}$ via the same unknown DGP and can be
associated to an inference problem (again, possibly irrelevant in the context
of the initial problem). On the other hand, both $x_{0}$ and $\mathcal{X}$ can
be values of interest. Indeed, in many learning instances, we dispose of a
training dataset (the set $\mathcal{D}$ in our paper) to infer the probable
models; and given another set of observations (the so-called testing dataset),
our aim is to predict for each of them the associated label (which is
different from one observation to another, as opposed to section 2.2, where
one common label produced several observations). In this case, all the
observations in the testing dataset play an epistemic role in the generative
case. More precisely, the observations of the testing dataset act as unlabeled
observations resulting in an underlying semi-supervised learning setting. This
is illustrated via quantitative simulations in the next section 5.

In this context, since we are trying to infer $x_{0}$ from $y_{0}$ and
$\mathcal{X}$ from $\mathcal{Y}$ via probable models $\theta$ using the same
observations $\mathcal{D}$, the two inference should not be treated as
independent problems in both modelling approaches. However, a main difference
between the two modeling approaches can be understood when considering two (or
several) inference problems at once.

In the discriminative setting, we have that
$p(x_{0},\mathcal{X}|y_{0},\mathcal{Y},\mathcal{D})\stackrel{{\scriptstyle
D}}{{=}}p(x_{0}|y_{0},\mathcal{D})p(\mathcal{X}|\mathcal{Y},\mathcal{D})$ so
the inference of $x_{0}$ (resp $\mathcal{X}$) does not depend on $\mathcal{Y}$
(resp $y_{0}$). As such, each inference problem can be solved via sampling
from its corresponding ppd. Conversely, in the generative setting, all the
observations $y_{0},\mathcal{Y}$ act as unlabelled observations in both
inferences and the two problems should not be treated independently as all the
unlabeled observations contribute to reducing the epistemic uncertainty. As a
result,
$p(x_{0},\mathcal{X}|y_{0},\mathcal{Y},\mathcal{D})\stackrel{{\scriptstyle
G}}{{\neq}}p(x_{0}|y_{0},\mathcal{D})p(\mathcal{X}|\mathcal{Y},\mathcal{D})$
and both inference can not be solved by sampling from each corresponding ppd,
and this modelling approach instead calls for a sampling from the joint
distribution. Finally, note that the previous semi-supervised algorithm, which
we proposed as a way to obtain samples from the posterior
$p(x_{0}|y_{0},\mathcal{D},\mathcal{Y})$ was constructed by applying the Gibbs
sequential sampling mechanism to the joint distribution
$p(x_{0},\mathcal{X},\theta|y_{0},\mathcal{Y},\mathcal{D})$ and as such,
produced desired samples from $p(x_{0}|y_{0},\mathcal{D},\mathcal{Y})$ but
also, as a byproduct, samples from
$p(\mathcal{X}|\mathcal{Y},\mathcal{D},y_{0})$, effectively solving both
inference problems at once. Again of course, since the Gibbs is only a tool
for solving the inference problem, the underlying structure of dependency
between rv is preserved. So in the discriminative case these distributions
respectively reduce to $p(x_{0}|y_{0},\mathcal{D})$ and
$p(\mathcal{X}|\mathcal{Y},\mathcal{D})$.

##  5 Simulations

Throughout the paper, we leveraged the example of affine homoskedastic
modeling in the case of univariate regression to illustrate the arguments of
this paper. As we mentionned before, this illustrating example can be relevant
to some readers as modeling affine dependencies is a most frequent problem in
many scientific fields. We also used this example as it enables, with
appropriate choice of priors, to run a straightforward Gibbs sampler for both
the generative and discriminative modeling approaches. However, considering
models which enable such convenient sampling procedures with closed-form
$p(\theta|\mathcal{D})$ (or all its conditionals in a Gibbs scheme) indeed
heavily restricts the choice of model $\mathcal{P}_{\theta}$. In this section
we now consider conditional models $\mathcal{P}_{\theta}$ which are defined
using NN functions with tractable pdf, but for which we are not able to elicit
a prior $\Pi_{\theta}$ over parameters $\theta$ such that the posterior
distribution $p(\theta|\mathcal{D})$ admits a closed form expression, and we
resort to approximate sampling from that posterior using Stochastic Gradient
Langevin Dynamics [72] [19].

In this section, we tackle the problem of classification in which $X$ is hence
a Categorical rv. We evaluate generative and discriminative models both
defined via NN functions (we describe the specific structure hereafter) and
with similar number of parameters. We proceed to assess the classification
accuracy of a generative versus discriminative model for $\theta\sim
p(\theta|\mathcal{D},y_{0})$ via Gibbs sampling. We consider three different
scenarios which we now describe.

Following the idea described in section 4.4, we now consider two distincts
sets: the training dataset $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$, and a
testing dataset $\\{(x_{0,j},y_{0,j})\\}_{j=1}^{M}$. All the labels $x_{0,j}$
are distributed according to $\Pi_{X_{0}}$, and in the following we address
three scenarios, which differ on the size of the testing set, and the possible
discrepancy between $\Pi_{X_{0}}$ and $\mathcal{P}_{X}^{\mathcal{D}}$.

Scenario 1: Identical priors and sizes. We first consider the scenario where
the label distribution from the training dataset coincides with the prior. So
$\mathcal{P}_{X}^{\mathcal{D}}=\Pi_{X_{0}}$, and as such, couples
$(x_{i},y_{i})$ (which belong to training dataset $\mathcal{D}$) and
$(x_{0,j},y_{0,j})$ (which belong to the testing dataset) have the same
distributions. This corresponds to the most frequent situation in practice and
we use this setting as a baseline. We consider the dataset $\mathcal{D}$ and
the set $\\{y_{0,j}\\}$ of unlabeled observations to be of the same size, i.e.
$N=M$.

Figure 14: Different prior distributions in scenario 2.

Scenario 2: Imbalanced dataset (different priors, same sizes). We then
consider the scenario where $\mathcal{P}_{X}^{\mathcal{D}}\neq\Pi_{X_{0}}$, so
couples $(x_{i},y_{i})$ and $(x_{0,j},y_{0,j})$ do not have the same
distributions (and indeed are quite different - see figure 14). We still set
$N=M$.

Scenario 3: Few labeled samples (same priors, different sizes). We finally
consider the scenario where $\mathcal{P}_{X}^{\mathcal{D}}=\Pi_{X_{0}}$, but
$N\ll M$, so we dispose of a few labeled observations in $\mathcal{D}$, and of
a large amount of observed values $y_{0,j}$ for which we want to infer the
corresponding label $x_{0,j}$. In this setting, the low number of observations
in $\mathcal{D}$ will hinder the prediction accuracy of both models, but as we
have explained before, the large amount of unlabeled observations act as an
unlabeled dataset in the generative case.

We consider both the classification datasets of MNIST and of FashionMNIST, for
which we reduce the dimension of observations via a Principal Component
Analysis [63] in order to keep $95\%$ of explained variation. We compare a
discriminative model which is a fully connected NN with 4 hidden-layers of 256
units to a generative model which is built using a combination of invertible
conditional Normalizing Flows layers [17] and stochastic ones [3]. We sample
from the joint distribution via Gibbs sampling with $T=10$ steps. The results
are provided in the next table 1 and for each dataset and scenario, we
consider 10 independent runs and we display the average classification
accuracy as well as the standard deviation.

Dataset | MNIST | FashionMNIST
---|---|---
Model. | Disc. | Gen. | Disc. | Gen.
Scenario 1 | $0.9628\pm 0.0014$ | $\mathbf{0.9749\pm 0.0145}$ | $\mathbf{0.8767\pm 0.0009}$ | $0.8652\pm 0.0094$
Scenario 2 | $0.9520\pm 0.0007$ | $\mathbf{0.9774\pm 0.0213}$ | $0.7000\pm 0.0057$ | $\mathbf{0.8482\pm 0.0172}$
Scenario 3 | $0.9349\pm 0.0018$ | $\mathbf{0.9690\pm 0.0237}$ | $0.7618\pm 0.0008$ | $\mathbf{0.8365\pm 0.0187}$
Table 1: Results of classification accuracy in percentages of Discriminative
and Generative modeling approaches

We now analyze the results of this experiment. Comparing the results for the
first scenario tells us that the generative modeling can indeed be on par in
terms of classification accuracy when compared to its discriminative
counterpart. This scenario can be used as a baseline experiment for the two
following scenarios. In the second scenario, the distribution of labels in the
dataset, $\mathcal{P}_{X}^{\mathcal{D}}$, is different from $\Pi_{X_{0}}$,
leading to a situation of imbalanced dataset. In this situation, we notice
that the discriminative model indeed suffers in term of accuracy as it favors
the dominant classes of the dataset. As we have explained, the generative
approach does not suffer from such dataset imbalance, or at least not as much,
which is confirmed in this experiment. This is in accordance with the
discussion of section 3.4. Finally, in the third scenario, the lower number of
labeled observation in $\mathcal{D}$ hinders, as expected, the classification
accuracy of the discriminative model as compared to its generative
counterpart, as the latter indeed leverages the unlabeled observations in the
inference of probable models, which indeed contributes to reduce the modeling
epistemic uncertainty, as discussed in section 4.2.

## Conclusion

Throughout this paper, we discussed the epistemic uncertainty quantification
in generative and discriminative models, and draw several conclusions. On the
one hand, discriminative models are an easy-to-use tool since they can be
parameterized easily and directly approximate the posterior. Moreover, if they
can be sampled from easily, then one can use a straightforward two step
procedure for sampling from the ppd, which indeed enables to quantify the
epistemic uncertainty. However, by nature discriminative models do not take
into account the information contained in the prior distribution, which is
replaced by an implicit prior inferred on the dataset. As a result, they
suffer from imbalanced datasets. Finally they cannot be conveniently used in
the context of inferring from multiple observations, and they cannot leverage
information from unlabeled data.

On the other hand, generative models are perhaps less convenient to use as
they usually require a more sophisticated structure and require an additional
inference step, in addition to the prior distribution, to sample from the
corresponding posterior. Yet by construction they do enable to leverage
information from all available sources, making them an appealing tool, in
particular in a semi-supervised context. In practice, the two-step procedure
for sampling from the ppd is no longer available; but our general purpose
Gibbs sampling based algorithm indeed enables to sample from the ppd (and thus
perform epistemic uncertainty quantification) while taking into account both
the labeled and unlabeled observations.

## References

  * [1] Alexander C Aitken.  On least squares and linear combination of observations.  Proceedings of the Royal Society of Edinburgh, 55:42–48, 1936.
  * [2] Ahmed Alaa and Mihaela Van Der Schaar.  Discriminative jackknife: Quantifying uncertainty in deep learning via higher-order influence functions.  In International Conference on Machine Learning, pages 165–174. PMLR, 2020.
  * [3] Elouan Argouarc’h, François Desbouvries, Eric Barat, Eiji Kawasaki, and Thomas Dautremer.  Discretely indexed flows, 2022.
  * [4] Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov.  Semi-conditional normalizing flows for semi-supervised learning.  arXiv preprint arXiv:1905.00505, 2019.
  * [5] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind.  Automatic differentiation in machine learning: a survey.  Journal of machine learning research, 18(153):1–43, 2018.
  * [6] Joseph Berkson.  Application of the logistic function to bio-assay.  Journal of the American statistical association, 39(227):357–365, 1944.
  * [7] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre.  Fast unfolding of communities in large networks.  Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
  * [8] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks.  Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models.  IEEE transactions on pattern analysis and machine intelligence, 2021.
  * [9] Léon Bottou, Frank E Curtis, and Jorge Nocedal.  Optimization methods for large-scale machine learning.  SIAM review, 60(2):223–311, 2018.
  * [10] George Casella and Edward I George.  Explaining the gibbs sampler.  The American Statistician, 46(3):167–174, 1992.
  * [11] Changyou Chen, David Carlson, Zhe Gan, Chunyuan Li, and Lawrence Carin.  Bridging the gap between stochastic gradient mcmc and stochastic optimization.  In Artificial Intelligence and Statistics, pages 1051–1060. PMLR, 2016.
  * [12] Siddhartha Chib and Edward Greenberg.  Understanding the metropolis-hastings algorithm.  The american statistician, 49(4):327–335, 1995.
  * [13] Kyle Cranmer, Johann Brehmer, and Gilles Louppe.  The frontier of simulation-based inference.  Proceedings of the National Academy of Sciences, 117(48):30055–30062, 2020.
  * [14] Katalin Csilléry, Michael GB Blum, Oscar E Gaggiotti, and Olivier François.  Approximate bayesian computation (abc) in practice.  Trends in ecology & evolution, 25(7):410–418, 2010.
  * [15] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the royal statistical society: series B (methodological), 39(1):1–22, 1977.
  * [16] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? does it matter?  Structural safety, 31(2):105–112, 2009.
  * [17] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803, 2016.
  * [18] Conor Durkan, George Papamakarios, and Iain Murray.  Sequential neural methods for likelihood-free inference.  arXiv preprint arXiv:1811.08723, 2018.
  * [19] Alain Durmus and Éric Moulines.  Nonasymptotic convergence analysis for the unadjusted Langevin algorithm.  The Annals of Applied Probability, 27(3):1551 – 1587, 2017.
  * [20] Ethan Fetaya, Jörn-Henrik Jacobsen, Will Grathwohl, and Richard Zemel.  Understanding the limitations of conditional generative models.  arXiv preprint arXiv:1906.01171, 2019.
  * [21] Felix Fiedler and Sergio Lucia.  Improved uncertainty quantification for neural networks with bayesian last layer.  IEEE Access, 2023.
  * [22] Edwin Fong, Simon Lyddon, and Chris Holmes.  Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap.  In International Conference on Machine Learning, pages 1952–1962. PMLR, 2019.
  * [23] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  * [24] A. E Gelfand.  Gibbs sampling.  Journal of the American statistical Association, 95(452):1300–1304, 2000.
  * [25] A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin.  Bayesian data analysis.  Chapman and Hall/CRC, 1995.
  * [26] Zoubin Ghahramani.  Probabilistic machine learning and artificial intelligence.  Nature, 521(7553):452–459, 2015.
  * [27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [28] Alex Graves.  Practical variational inference for neural networks.  Advances in neural information processing systems, 24, 2011.
  * [29] David Greenberg, Marcel Nonnenmacher, and Jakob Macke.  Automatic posterior transformation for likelihood-free inference.  In International Conference on Machine Learning, pages 2404–2414. PMLR, 2019.
  * [30] Lauren A Hannah, David M Blei, and Warren B Powell.  Dirichlet process mixtures of generalized linear models.  Journal of Machine Learning Research, 12(6), 2011.
  * [31] Trevor Hastie, Robert Tibshirani, Jerome Friedman, Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  Unsupervised learning.  The elements of statistical learning: Data mining, inference, and prediction, pages 485–585, 2009.
  * [32] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  The elements of statistical learning: data mining, inference, and prediction, volume 2.  Springer, 2009.
  * [33] Geoffrey Hinton and Terrence J Sejnowski.  Unsupervised learning: foundations of neural computation.  MIT press, 1999.
  * [34] Chris Holmes and Leonhard Knorr-Held.  Bayesian auxiliary variable models for binary and multinomial regression.  Bayesian Analysis, 1:145–168, 03 2006.
  * [35] Ziyi Huang, Henry Lam, and Haofeng Zhang.  Quantifying epistemic uncertainty in deep learning.  arXiv preprint arXiv:2110.12122, 2021.
  * [36] Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  Machine learning, 110(3):457–506, 2021.
  * [37] Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson.  Semi-supervised learning with normalizing flows.  In International Conference on Machine Learning, pages 4615–4630. PMLR, 2020.
  * [38] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson.  What are bayesian neural network posteriors really like?  In International conference on machine learning, pages 4629–4640. PMLR, 2021.
  * [39] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Advances in neural information processing systems, 30, 2017.
  * [40] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [41] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  pages 32–33, 2009.
  * [42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [43] Erich L Lehmann and George Casella.  Theory of point estimation.  Springer Science & Business Media, 2006.
  * [44] Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou.  A survey of convolutional neural networks: analysis, applications, and prospects.  IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021.
  * [45] Aristidis Likas, Nikos Vlassis, and Jakob J Verbeek.  The global k-means clustering algorithm.  Pattern recognition, 36(2):451–461, 2003.
  * [46] Stuart Lloyd.  Least squares quantization in pcm.  IEEE transactions on information theory, 28(2):129–137, 1982.
  * [47] Jan-Matthis Lueckmann, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H Macke.  Likelihood-free inference with emulator networks.  In Symposium on Advances in Approximate Bayesian Inference, pages 32–53. PMLR, 2019.
  * [48] Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke.  Benchmarking simulation-based inference.  In International conference on artificial intelligence and statistics, pages 343–351. PMLR, 2021.
  * [49] Simon Lyddon, Stephen Walker, and Chris C Holmes.  Nonparametric learning from bayesian models with randomized objective functions.  Advances in neural information processing systems, 31, 2018.
  * [50] Radek Mackowiak, Lynton Ardizzone, Ullrich Kothe, and Carsten Rother.  Generative classifiers as a basis for trustworthy image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2971–2981, 2021.
  * [51] James N Morgan and John A Sonquist.  Problems in the analysis of survey data, and a proposal.  Journal of the American statistical association, 58(302):415–434, 1963.
  * [52] Kevin P Murphy.  Machine learning: a probabilistic perspective.  MIT press, 2012.
  * [53] Radford M Neal.  Bayesian learning for neural networks, volume 118.  Springer Science & Business Media, 2012.
  * [54] Michael A Newton, Nicholas G Polson, and Jianeng Xu.  Weighted bayesian bootstrap for scalable posterior distributions.  Canadian Journal of Statistics, 49(2):421–437, 2021.
  * [55] Michael A Newton and Adrian E Raftery.  Approximate bayesian inference with the weighted likelihood bootstrap.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 56(1):3–26, 1994.
  * [56] Andrew Ng and Michael Jordan.  On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.  Advances in neural information processing systems, 14, 2001.
  * [57] Andrew Ng, Michael Jordan, and Yair Weiss.  On spectral clustering: Analysis and an algorithm.  Advances in neural information processing systems, 14, 2001.
  * [58] Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell.  Text classification from labeled and unlabeled documents using em.  Machine learning, 39:103–134, 2000.
  * [59] Augustus Odena.  Semi-supervised learning with generative adversarial networks.  arXiv preprint arXiv:1606.01583, 2016.
  * [60] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  Journal of Machine Learning Research, 22(57):1–64, 2021.
  * [61] George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation.  Advances in neural information processing systems, 30, 2017.
  * [62] George Papamakarios, David Sterratt, and Iain Murray.  Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows.  In The 22nd international conference on artificial intelligence and statistics, pages 837–848. PMLR, 2019.
  * [63] Karl Pearson.  Liii. on lines and planes of closest fit to systems of points in space.  The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559–572, 1901.
  * [64] Juho Piironen and Aki Vehtari.  Sparsity information and regularization in the horseshoe and other shrinkage priors.  Electronic Journal of Statistics, 11, 07 2017.
  * [65] Adrian E Raftery, David Madigan, and Jennifer A Hoeting.  Bayesian model averaging for linear regression models.  Journal of the American Statistical Association, 92(437):179–191, 1997.
  * [66] Michael Revow, Christopher KI Williams, and Geoffrey E Hinton.  Using generative models for handwritten digit recognition.  IEEE transactions on pattern analysis and machine intelligence, 18(6):592–606, 1996.
  * [67] Christian P Robert et al.  The Bayesian choice: from decision-theoretic foundations to computational implementation, volume 2.  Springer, 2007.
  * [68] G. O. Roberts and J.S. Rosenthal.  General state space Markov chains and MCMC algorithms.  Probability Surveys, 1:20 – 71, 2004.
  * [69] Tim GJ Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal.  Tractable function-space variational inference in bayesian neural networks.  Advances in Neural Information Processing Systems, 35:22686–22698, 2022.
  * [70] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  * [71] Harald Steck and Tommi Jaakkola.  On the dirichlet prior and bayesian regularization.  Advances in neural information processing systems, 15, 2002.
  * [72] Y Teh, Alexandre Thiéry, and Sebastian Vollmer.  Consistency and fluctuations for stochastic gradient langevin dynamics.  Journal of Machine Learning Research, 17, 2016.
  * [73] Ilkay Ulusoy and Christopher M Bishop.  Generative versus discriminative methods for object recognition.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 258–265. IEEE, 2005.
  * [74] Vladimir Vapnik.  The nature of statistical learning theory.  Springer science & business media, 2013.
  * [75] Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swikatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin.  How good is the bayes posterior in deep neural networks really?  arXiv preprint arXiv:2002.02405, 2020.
  * [76] Peter M Williams.  Bayesian regularization and pruning using a laplace prior.  Neural computation, 7(1):117–143, 1995.
  * [77] Chenyu Zheng, Guoqiang Wu, Fan Bao, Yue Cao, Chongxuan Li, and Jun Zhu.  Revisiting discriminative vs. generative classifiers: Theory and implications.  In International Conference on Machine Learning, pages 42420–42477. PMLR, 2023.
  * [78] Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin Adric Dunn, and David A Klindt.  Score-based generative classifiers.  In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.