Optimal Bayesian Estimation of a Regression Curve, a Conditional Density and a
Conditional Distribution
A.G. Nogales

Dpto. de Matemáticas, Universidad de Extremadura

Avda. de Elvas, s/n, 06006–Badajoz, SPAIN.

e-mail<EMAIL_ADDRESS>
> Abstract. In this paper several related estimation problems are addressed
> from a Bayesian point of view and optimal estimators are obtained for each
> of them when some natural loss functions are considered. Namely, we are
> interested in estimating a regression curve. Simultaneously, the estimation
> problems of a conditional distribution function, or a conditional density,
> or even the conditional distribution itself, are considered. All these
> problems are posed in a sufficiently general framework to cover continuous
> and discrete, univariate and multivariate, parametric and non-parametric
> cases, without the need to use a specific prior distribution. The loss
> functions considered come naturally from the quadratic error loss function
> comonly used in estimating a real function of the unknown parameter. The
> cornerstone of the mentioned Bayes estimators is the posterior predictive
> distribution. Some examples are provided to illustrate these results.

  * AMS Subject Class. (2010): Primary 62F15, 62G07 Secondary 62Jxx

  * Key words and phrases:  Bayesian estimation of a regression curve, posterior predictive distribution.

##  1 Introduction.

In Statistics, the expression _the probability of an event_ $A$ (written
$P_{\theta}(A)$) is, in general, ambiguous as it depends of the unknown
parameter $\theta$. Before conducting the experiment, a Bayesian statistician,
provided with the prior distribution, possesses a natural candidate: the prior
predictive probability of $A$ since it is the prior mean of the probabilities
of $A$. However, according to Bayesian philosophy, after performing the
experiment and having observed the data $\omega$, a reasonable estimation is
the posterior predicitive probability of $A$ given $\omega$ because it is the
posterior mean of the probabilities of $A$ given $\omega$. It can be shown not
only that this is the Bayes estimator of the probability $P_{\theta}(A)$ of
$A$ for the squared error loss function but also that the posterior predictive
distribution is the Bayes estimator of the sampling probability distribution
$P_{\theta}$ for the squared variation total loss function and that the
posterior predictive density is the Bayes estimator of its density for the
$L^{1}$-squared loss function. Notice that these loss functions should be
considered natural in the sense that they are derived directly from the
quadratic error loss function commonly used in the estimation of a real
function of the parameter. Nogales (2021) contains accurate statements and
proofs of these results which are nothing but a functional generalization of
Theorem 1.1 (more specifically of its Corollary 1.2.(a)) of Lehmann et al.
(1998, p. 228) which obtains the Bayes estimator of a real function of the
parameter for the squared error loss function.

This article is about estimating a regression curve and some other related
problems like the estimation of a conditional density or a conditional
distribution function or even of the conditional distribution itself, ever
from a Bayesian point of view. Therefore, this paper should be considered as
the conditional counterpart of Nogales (2021) and the results below become the
functional extension of Lehmann et al. (1998, Theorem 1.1) for the conditional
case. So it is not surprising that the posterior predictive distribution is
the keystone in the estimation problems to be discussed below. Some examples
are presented in Section 7 to illustrate the results. See Nogales (2021), and
the references therein, for other examples of determination of the posterior
predictive distribution. But in practice, the explicit evaluation of the
posterior predictive distribution could be cumbersome and its simulation may
become preferable. Gelman et al. (2014) is also a good reference for such
simulation methods and, hence, for the computation of the Bayes estimators of
the conditional density and the regression curve.

The posterior predictive distribution has been presented as the base for
Predictive Inference, which seeks to make inferences about a new unknown
observation from the previous random sample (in contrast to the greater
emphasis that statistical inference makes on the estimation and contrast of
parameters since its mathematical foundations in the early twentieth century,
see Geisser (1993) or Gelman et al. (2014)). With that idea in mind, it has
also been used in other areas such as model selection, testing for
discordancy, goodness of fit, perturbation analysis or classification (see
addtional fields of application in Geisser (1993) and Rubin (1984)), but never
as a possible solution for the Bayesian problems of estimating an
unconditional or conditional density. The reader is referred to the references
within Nogales (2021) for other uses of the posterior predictive distribution
in Bayesian statistics.

In what follows we place ourselves in a general framewok for Bayesian
inference, as described in Barra (1971).

##  2 The framework.

Let
$(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
be a Bayesian statistical experiment and
$X_{i}:(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})\rightarrow(\Omega_{i},\mathcal{A}_{i})$,
$i=1,2$, two statistics. Consider the Bayesian experiment image of
$(X_{1},X_{2})$

$(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2},\\{P_{\theta}^{(X_{1},X_{2})}\colon\theta\in(\Theta,\mathcal{T},Q)\\}).$

In the next, we shall suppose that
$P^{(X_{1},X_{2})}(\theta,A_{12}):=P_{\theta}^{(X_{1},X_{2})}(A_{12})$,
$\theta\in\Theta,\ A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}$, is a Markov
kernel. Let us write $R_{\theta}=P_{\theta}^{(X_{1},X_{2})}$.

The Bayesian experiment corresponding to a sample of size $n$ of the joint
distribution of $(X_{1},X_{2})$ is

$\big{(}(\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n},\big{\\{}R_{\theta}^{n}\colon\theta\in(\Theta,\mathcal{T},Q)\big{\\}}\big{)}\qquad\hbox{(1)}$

We write $R^{n}(\theta,A^{\prime}_{12,n})=R_{\theta}^{n}(A^{\prime}_{12,n})$
for $A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$ and

$\Pi_{12,n}:=Q\otimes R^{n},$

for the joint distribution of the parameter and the sample, i.e.

$\Pi_{12,n}(A^{\prime}_{12,n}\times
T)=\int_{T}R_{\theta}^{n}(A^{\prime}_{12,n})dQ(\theta),\quad
A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n},T\in\mathcal{T}.$

The corresponding prior predictive distribution $\beta_{12,n,Q}^{*}$ is

$\beta_{12,n,Q}^{*}(A^{\prime}_{12,n})=\int_{\Theta}R_{\theta}^{n}(A^{\prime}_{12,n})dQ(\theta),\quad
A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}.$

The posterior distribution is a Markov kernel

$R_{n}^{*}:((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n})\mbox{$\succ$$\longrightarrow$}(\Theta,\mathcal{T})$

such that, for all
$A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$ and
$T\in\mathcal{T}$,

$\Pi_{12,n}(A^{\prime}_{12,n}\times
T)=\int_{T}R_{\theta}^{n}(A^{\prime}_{12,n})dQ(\theta)=\int_{A^{\prime}_{12,n}}R_{n}^{*}(x^{\prime},T)d\beta_{12,n,Q}^{*}(x^{\prime}).$

Let us write $R_{n,x^{\prime}}^{*}(T):=R_{n}^{*}(x^{\prime},T)$.

The posterior predictive distribution on
$\mathcal{A}_{1}\times\mathcal{A}_{2}$ is the Markov kernel

${R_{n}^{*}}^{R}:((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n})\mbox{$\succ$$\longrightarrow$}(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})$

defined, for $x^{\prime}\in(\Omega_{1}\times\Omega_{2})^{n}$, by

${R_{n}^{*}}^{R}(x^{\prime},A_{12}):=\int_{\Theta}R_{\theta}(A_{12})dR_{n,x^{\prime}}^{*}(\theta)$

It follows that, with obvious notations,

$\int_{\Omega_{1}\times\Omega_{2}}f(x)d{R_{n,x^{\prime}}^{*}}^{\\!\\!\\!\\!R}(x)=\int_{\Theta}\int_{\Omega_{1}\times\Omega_{2}}f(x)dR_{\theta}(x)dR_{n,x^{\prime}}^{*}(\theta)$

for any non-negative or integrable real random variable (r.r.v. for short)
$f$.

We can also consider the posterior predictive distribution on
$(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$ defined as the Markov kernel

${R_{n}^{*}}^{R^{n}}:((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n})\mbox{$\succ$$\longrightarrow$}((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n})$

such that

${R_{n}^{*}}^{R^{n}}(x^{\prime},A^{\prime}_{12,n}):=\int_{\Theta}R_{\theta}^{n}(A^{\prime}_{12,n})dR_{n,x^{\prime}}^{*}(\theta)$

According to Theorem 1 of Nogales (2021), it is the Bayes estimator of the
distribution $R_{\theta}^{n}$ for the squared total variation function, i.e.,

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times\Theta}\sup_{A_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}}\left|{R_{n,x^{\prime}}^{*}}^{\\!\\!\\!\\!R^{n}}(A_{12,n})-R_{\theta}^{n}(A_{12,n})\right|^{2}d\Pi_{12,n}(x^{\prime},\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times\Theta}\sup_{A_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}}\left|M(x^{\prime},A_{12,n})-R_{\theta}^{n}(A_{12,n})\right|^{2}d\Pi_{12,n}(x^{\prime},\theta),$

for every Markov kernel
$M:(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\mbox{$\succ$$\longrightarrow$}(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$.

It can be readily checked that

$\left[{R_{n,x^{\prime}}^{*}}^{\\!\\!\\!\\!R^{n}}\right]^{\pi^{\prime}_{1}}={R_{n,x^{\prime}}^{*}}^{\\!\\!\\!\\!R},$

where
$\pi^{\prime}_{1}(x^{\prime}):=x^{\prime}_{1}:=(x^{\prime}_{11},x^{\prime}_{21})$
for $x^{\prime}\in(\Omega_{1}\times\Omega_{2})^{n}$. So, Theorem 2 of Nogales
(2021) shows that

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times\Theta}\sup_{A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}}\left|{R_{n,x^{\prime}}^{*}}^{\\!\\!\\!\\!R}(A_{12})-R_{\theta}(A_{12})\right|^{2}d\Pi_{12,n}(x^{\prime},\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times\Theta}\sup_{A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}}\left|M(x^{\prime},A_{12})-R_{\theta}(A_{12})\right|^{2}d\Pi_{12,n}(x^{\prime},\theta),$

for every Markov kernel
$M:(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\mbox{$\succ$$\longrightarrow$}(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})$.

We introduce some notations for
$(x^{\prime},x,\theta)\in(\Omega_{1}\times\Omega_{2})^{n}\times(\Omega_{1}\times\Omega_{2})\times\Theta$:

$\displaystyle\pi^{\prime}(x^{\prime},x,\theta):=x^{\prime},\quad\pi^{\prime}_{i}(x^{\prime},x,\theta):=x^{\prime}_{i}:=(x^{\prime}_{i1},x^{\prime}_{i2}),\quad
1\leq i\leq n,$
$\displaystyle\pi(x^{\prime},x,\theta):=x,\quad\pi_{i}(x^{\prime},x,\theta):=x_{i},\quad
i=1,2,$ $\displaystyle q(x^{\prime},x,\theta):=\theta.$

Let us consider the probability space

$\displaystyle((\Omega_{1}\times\Omega_{2})^{n}\times(\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T},\Pi_{n}),\qquad\hbox{(2)}$

where

$\Pi_{n}(A^{\prime}_{12,n}\times A_{12}\times
T)=\int_{T}R_{\theta}(A_{12})R_{\theta}^{n}(A^{\prime}_{12,n})dQ(\theta),$

when $A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$,
$A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}$ and $T\in\mathcal{T}$.

So, for a r.r.v. $f$ on
$((\Omega_{1}\times\Omega_{2})^{n}\times(\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T})$,

$\int
fd\Pi_{n}=\int_{\Theta}\int_{(\Omega_{1}\times\Omega_{2})^{n}}\int_{(\Omega_{1}\times\Omega_{2})}f(x^{\prime},x,\theta)dR_{\theta}(x)dR_{\theta}^{n}(x^{\prime})dQ(\theta)\qquad{\rm(3)}$

provided that the integral exists. Moreover, for a r.r.v. $h$ on
$((\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T})$,

$\int
hd\Pi_{n}=\int_{\Theta}\int_{\Omega_{1}\times\Omega_{2}}h(x,\theta)dR_{\theta}(x)dQ(\theta)=\int_{\Omega_{1}\times\Omega_{2}}\int_{\Theta}h(x,\theta)dR^{*}_{1,x}(\theta)d\beta^{*}_{12,1,Q}(x).$

The following proposition is straightforward.

######  Proposition 1.

$\displaystyle\Pi_{n}^{(\pi^{\prime},q)}(A^{\prime}_{12.n}\times
T)=\Pi_{12,n}(A^{\prime}_{12,n}\times
T)=\int_{T}R_{\theta}^{n}(A^{\prime}_{12,n})dQ(\theta)=\int_{A^{\prime}_{12,n}}R^{*}_{n,x^{\prime}}(T)d\beta_{12,n,Q}^{*}(x^{\prime})$
$\displaystyle\Pi_{n}^{(\pi,q)}(A_{12}\times T)=\Pi_{12,1}(A_{12}\times
T)=\int_{T}R_{\theta}(A_{12})dQ(\theta)=\int_{A_{12}}R^{*}_{1,x}(T)d\beta_{12,1,Q}^{*}(x)$
$\displaystyle\Pi_{n}^{q}=Q,\quad\Pi_{n}^{(\pi^{\prime},q)}=\Pi_{12,n},\quad\Pi_{n}^{\pi^{\prime}}=\beta_{12,n,Q}^{*},\quad\Pi_{n}^{(\pi,q)}=\Pi_{12,1},\quad\Pi_{n}^{\pi}=\beta_{12,1,Q}^{*}$
$\displaystyle\Pi_{n}^{\pi^{\prime}|q=\theta}=R_{\theta}^{n},\quad\Pi_{n}^{\pi|q=\theta}=R_{\theta},\quad\Pi_{n}^{q|\pi^{\prime}=x^{\prime}}=R^{*}_{n,x^{\prime}},\quad\Pi_{n}^{q|\pi=x}=R^{*}_{1,x^{\prime}}$
$\displaystyle P_{\theta}^{X_{1}}=R_{\theta}^{\pi_{1}},\quad
P_{\theta}^{X_{2}|X_{1}=x_{1}}=R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}},$
$\displaystyle
E_{P_{\theta}}(X_{2}|X_{1}=x_{1})=E_{R_{\theta}}(\pi_{2}|\pi_{1}=x_{1}).$

In particular, the probability space (2) contains all the basic ingredients of
the Bayesian experiment (1), i.e., the prior distribution, the sampling
probabilities, the posterior distributions and the prior predictive
distribution. In addition, it becomes the natural framework to address the
estimation problems of this paper, as we can see in the next.

##  3 Bayes estimator of the conditional distribution.

An estimator of the conditional distribution $P_{\theta}^{X_{2}|X_{1}}$ from a
$n$-sized sample of the joint distribution of $(X_{1},X_{2})$ is a Markov
kernel

$M:((\Omega_{1}\times\Omega_{2})^{n}\times\Omega_{1},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$

so that, being observed
$x^{\prime}=((x^{\prime}_{11},x^{\prime}_{21}),\dots,(x^{\prime}_{1n},x^{\prime}_{2n}))\in(\Omega_{1}\times\Omega_{2})^{n}$,
$M(x^{\prime},x_{1},\cdot)$ is a probability measure on $\mathcal{A}_{2}$ that
is considered as an estimation of the conditional distribution
$P_{\theta}^{X_{2}|X_{1}=x_{1}}$ for a given $x_{1}\in\Omega_{1}$.

From a Bayesian point of view, the Bayes estimator of the conditional
distribution $P^{X_{2}|X_{1}}=R^{\pi_{2}|\pi_{1}}$ is a Markov kernel

$M:((\Omega_{1}\times\Omega_{2})^{n}\times\Omega_{1},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$

minimizing the Bayes risk

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times\Theta}\int_{\Omega_{1}}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M(x^{\prime},x_{1},A_{2})-R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2})\big{|}^{2}dR_{\theta}^{\pi_{1}}(x_{1})d\Pi_{12,n}(x^{\prime},\theta)=$
$\displaystyle\int_{\Theta}\int_{(\Omega_{1}\times\Omega_{2})^{n}}\int_{\Omega_{1}}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M(x^{\prime},x_{1},A_{2})-R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2})\big{|}^{2}dR_{\theta}^{\pi_{1}}(x_{1})dR_{\theta}^{n}(x^{\prime})dQ(\theta)=$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n}\times(\Omega_{1}\times\Omega_{2})\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M(x^{\prime},x_{1},A_{2})-R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2})\big{|}^{2}d\Pi_{n}(x^{\prime},x,\theta).$

The next result yields the Bayes estimator of the conditional distribution
$P_{\theta}^{X_{2}|X_{1}}$ from the posterior predictive distribution.

######  Theorem 1.

Suppose that the $\sigma$-field $\mathcal{A}_{2}$ is separable. Then, the
conditional distribution of $\pi_{2}$ given $\pi_{1}=x_{1}$ with respect to
the posterior predictive distribution
${R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}$,

$M_{n}^{*}(x^{\prime},x_{1},A_{2}):=\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{\pi_{2}|\pi_{1}=x_{1}}(A_{2}),$

is the Bayes estimator of the conditional distribution $R^{\pi_{2}|\pi_{1}}$
for the squared total variation loss function, i.e.,

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M_{n}^{*}(x^{\prime},x_{1},A_{2})-R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2})\big{|}^{2}d\Pi_{n}(x^{\prime},x,\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M(x^{\prime},x_{1},A_{2})-R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2})\big{|}^{2}d\Pi_{n}(x^{\prime},x,\theta)$

for any estimator $M$ of the conditional distribution $R^{\pi_{2}|\pi_{1}}$.

Fix an event $A_{2}\in\mathcal{A}_{2}$ and define

$H_{A_{2}}(x^{\prime},x_{1},\theta):=P_{\theta}^{X_{2}|X_{1}=x_{1}}(A_{2})=R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(A_{2}).$

Jensen’s inequality could help to achieve a proof of the theorem if we can
prove the following result.

######  Lemma 1.

$E_{\Pi_{n}}(H_{A_{2}}|\pi^{\prime}=x^{\prime},\pi_{1}=x_{1})=M_{n}^{*}(x^{\prime},x_{1},A_{2}),$

##  4 Bayes estimator of the conditional density.

When the joint distribution $R_{\theta}=P_{\theta}^{(X_{1},X_{2})}$ has a
density $f_{\theta}$ with respect to the product of two $\sigma$-finite
measures $\mu_{1}$ and $\mu_{2}$ on $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$,
resp., the conditional density is

$f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f_{\theta}(x_{1},x_{2})}{f_{\theta,X_{1}}(x_{1})}$

for almost every $x_{1}$, where $f_{\theta,X_{1}}(x_{1})$ stands for the
marginal density of $X_{1}$.

An estimator of the conditional density $f_{\theta}^{X_{2}|X_{1}}$ from a
$n$-sized sample of the joint distribution of $(X_{1},X_{2})$ is a map

$m:((\Omega_{1}\times\Omega_{2})^{n}\times\Omega_{1}\times\Omega_{2},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1}\times\mathcal{A}_{2})\rightarrow(\mathbb{R},\mathcal{R})$

so that, being observed
$x^{\prime}=((x^{\prime}_{11},x^{\prime}_{21}),\dots,(x^{\prime}_{1n},x^{\prime}_{2n}))\in(\Omega_{1}\times\Omega_{2})^{n}$,
$m(x^{\prime},x_{1},\cdot)$ is considered as an estimation of the conditional
density $f_{\theta}^{X_{2}|X_{1}=x_{1}}$ of $X_{2}$ given $X_{1}=x_{1}$.

It is well known (see, for instance, Ghosal et al. (2017, p.126)) that, given
two probability measures $P_{1}$ and $P_{2}$ on a measurable space
$(\Omega,\mathcal{A})$ having densities $p_{1}$ and $p_{2}$ with respect to a
$\sigma$-finite measure $\mu$,

$\sup_{A\in\mathcal{A}}|P_{1}(A)-P_{2}(A)|=\frac{1}{2}\int_{\Omega}|p_{1}-p_{2}|d\mu.$

So the Bayesian estimation of the conditional distribution
$P_{\theta}^{X_{2}|X_{1}=x_{1}}=R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}$ for the
squared total variation loss function corresponds to the Bayesian estimation
of its density $f_{\theta}^{X_{2}|X_{1}=x_{1}}$ for the $L^{1}$-squared loss
function. Hence, according to Theorem 1, the Bayes estimator of the
conditional density $f_{\theta}^{X_{2}|X_{1}=x_{1}}$ for the $L^{1}$-squared
loss function is the $\mu_{2}$-density
${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}$ of the conditional
distribution

$\left[{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{\pi_{2}|\pi_{1}=x_{1}}.$

Notice that

$\displaystyle{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}(A_{1}\times
A_{2})=\int_{\Theta}R_{\theta}(A_{1}\times
A_{2})dR^{*}_{n,x^{\prime}}(\theta)=$
$\displaystyle\int_{\Theta}\int_{A_{1}\times
A_{2}}f_{\theta}(x_{1},x_{2})d(\mu_{1}\times\mu_{2})(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)=$
$\displaystyle\int_{A_{1}\times
A_{2}}\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d(\mu_{1}\times\mu_{2})(x_{1},x_{2})$

where $r^{*}_{n,x^{\prime}}(\theta)$ denotes the $Q$-density of the posterior
distribution $R^{*}_{n,x^{\prime}}$. So $r^{*}_{n,x^{\prime}}(\theta)$ is of
the form $K(x^{\prime})f_{n,\theta}(x^{\prime})$, where

$f_{n,\theta}(x^{\prime}):=\prod_{i=1}^{n}f_{\theta}(x^{\prime}_{i})$

is the $(\mu_{1}\times\mu_{2})^{n}$-density of $R_{\theta}^{n}$. Hence the
$\mu_{1}\times\mu_{2}$-density of the posterior predictive distribution
${R^{*}_{x^{\prime}}}^{\\!\\!R}$ is

$f^{*}_{n,x^{\prime}}(x_{1},x_{2}):=\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta),$

and its first marginal is

$f^{*}_{n,x^{\prime},1}(x_{1}):=\int_{\Omega_{2}}\int_{\Theta}f_{\theta}(x_{1},t)r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(t).$

So, we have proved the following result.

######  Theorem 2.

Suppose that $\mathcal{A}_{2}$ is separable. The Bayes estimator of the
conditional density $f_{\theta}^{X_{2}|X_{1}}$ for the $L^{1}$-squared loss
function is the $\mu_{2}$-density

${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f^{*}_{n,x^{\prime}}(x_{1},x_{2})}{f^{*}_{n,x^{\prime},1}(x_{1})}=\frac{\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)}{\int_{\Omega_{2}}\int_{\Theta}f_{\theta}(x_{1},t)r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(t)}$

of the conditional distribution
$\left[{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{\pi_{2}|\pi_{1}}$ of
$\pi_{2}$ given $\pi_{1}$ with respect to the posterior predictive
distribution ${R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}$, i.e.

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\left(\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)\right)^{2}d\Pi_{n}(x^{\prime},x,\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\left(\int_{\Omega_{2}}\left|m(x^{\prime},x_{1},t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)\right)^{2}d\Pi_{n}(x^{\prime},x,\theta),$

for any estimator $m$ of the conditional density.

##  5 Bayes estimator of the conditional distribution function.

When $X_{2}$ is a r.r.v. we may be interested in the estimation of the
conditional distribution function of $X_{2}$ given $X_{1}=x_{1}$

$F_{\theta}(x_{1},t):=P_{\theta}(X_{2}\leq
t|X_{1}=x_{1})=R_{\theta}^{\pi_{2}|\pi_{1}=x_{1}}(]-\infty,t]).$

An estimator of such a conditional distribution function from a $n$-sized
sample of $R_{\theta}$ is a map of the form

$F:(x^{\prime},x_{1},t)\in(\Omega_{1}\times\mathbb{R})^{n}\times\Omega_{1}\times\mathbb{R}\mapsto
F(x^{\prime},x_{1},t):=M(x^{\prime},x_{1},]-\infty,t])\in[0,1]$

for a Markov kernel

$M:((\Omega_{1}\times\mathbb{R})^{n}\times\Omega_{1},(\mathcal{A}_{1}\times\mathcal{R})^{n}\times\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\mathbb{R},\mathcal{R})$

An optimal estimator of the conditional distribution function $F_{\theta}$ for
the $L^{\infty}$-squared loss function from a Bayesian point of view (i.e. a
Bayes estimator) is an estimator $F_{n}^{*}$ minimizing the Bayes risk, i.e.,
such that

$\displaystyle\int_{(\Omega_{1}\times\mathbb{R})^{n+1}\times\Theta}\sup_{t\in\mathbb{R}}|F_{n}^{*}(x^{\prime},x_{1},t)-F_{\theta}(x_{1},t)|^{2}d{\Pi_{n}}(x^{\prime},x,\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\mathbb{R})^{n+1}\times\Theta}\sup_{t\in\mathbb{R}}|F(x^{\prime},x_{1},t)-F_{\theta}(x_{1},t)|^{2}d{\Pi_{n}}(x^{\prime},x,\theta)$

for any estimator $F$ of the conditional distribution function $F_{\theta}$.

A natural candidate is the conditional distribution function for the posterior
predictive distribution as it is stated in the next theorem.

######  Theorem 3.

The posterior predictive conditional distribution function

$F_{n}^{*}(x^{\prime},x_{1},t):=\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{\pi_{2}|\pi_{1}=x_{1}}\\!\\!(]-\infty,t])$

is the Bayes estimator of the conditional distribution function $F_{\theta}$
for the $L^{\infty}$-squared loss function.

##  6 Bayes estimator of a regression curve.

Now suppose that $X_{2}$ is an squared-integrable r.r.v. So
$(\Omega_{2},\mathcal{A}_{2})=(\mathbb{R},\mathcal{R})$. The regression curve
of $X_{2}$ given $X_{1}$ is the map $x_{1}\in\Omega_{1}\mapsto
r_{\theta}(x_{1}):=E_{\theta}(X_{2}|X_{1}=x_{1})$. An estimator of the
regression curve $r_{\theta}$ from a sample of size $n$ of the joint
distribution of $(X_{1},X_{2})$ is a statistic

$m:(x^{\prime},x_{1})\in(\Omega_{1}\times\mathbb{R})^{n}\times\Omega_{1}\longmapsto
m(x^{\prime},x_{1})\in\mathbb{R},$

so that, being observed $x^{\prime}\in(\Omega_{1}\times\mathbb{R})^{n}$,
$m(x^{\prime},\cdot)$ is the estimation of $r_{\theta}$.

From a frequentist point of view, the simplest way to evaluate the error in
estimating an unknown regression curve is to use the expectation of the
quadratic deviation (see Nadaraya (1989, p. 120)):

$\displaystyle
E_{\theta}\left[\int_{\Omega_{1}}(m(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}dP_{\theta}^{X_{1}}(x_{1})\right]=$
$\displaystyle\int_{(\Omega_{1}\times\mathbb{R})^{n}}\int_{\Omega_{1}}(m(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}dR_{\theta}^{\pi_{1}}(x_{1})dR_{\theta}^{n}(x^{\prime}).$

From a Bayesian point of view, the Bayes estimator of the regression curve
$r_{\theta}$ should minimize the Bayes risk (i.e. the prior mean of the
expectation of the quadratic deviation)

$\displaystyle\int_{\Theta}\int_{(\Omega_{1}\times\mathbb{R})^{n}}\int_{\Omega_{1}}(m(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}dR_{\theta}^{\pi_{1}}(x_{1})dR_{\theta}^{n}(x^{\prime})dQ(\theta)=$
$\displaystyle
E_{\Pi_{n}}\big{[}(m(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}\big{]}.$

The following result solves the problem of estimation of the regression curve
from a Bayesian point of view.

######  Theorem 4.

The regression curve of $\pi_{2}$ on $\pi_{1}$ with respect to the posterior
predictive distribution ${R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}$

$m_{n}^{*}(x^{\prime},x_{1}):=E_{{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}}(\pi_{2}|\pi_{1}=x_{1})$

is the Bayes estimator of the regression curve
$r_{\theta}(x_{1}):=E_{\theta}(X_{2}|X_{1}=x_{1})$ for the squared error loss
function, i.e.,

$E_{\Pi_{n}}[(m_{n}^{*}(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}]\leq
E_{\Pi_{n}}[(m_{n}(x^{\prime},x_{1})-r_{\theta}(x_{1}))^{2}]$

for any other estimator $m_{n}$ of the regression curve $r_{\theta}$.

######  Remark.

(Estimation of the regression curve when densities are available) According to
the previous results, when $R_{\theta}$ has density $f_{\theta}$ with respect
to the product $\mu_{1}\times\mu_{2}$ of two $\sigma$-finite measures, the
$\mu_{2}$-density ${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}$ of
the conditional distribution
$\left[{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{\pi_{2}|\pi_{1}=x_{1}}$
is

${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f^{*}_{n,x^{\prime}}(x_{1},x_{2})}{f^{*}_{n,x^{\prime},1}(x_{1})}=\frac{\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)}{\int_{\mathbb{R}}\int_{\Theta}f_{\theta}(x_{1},t)r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(t)},$

which is the Bayes estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}}$. Hence, the Bayes estimator of the regression curve
can be computed as

$\displaystyle
m_{n}^{*}(x^{\prime},x_{1}):=E_{{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}}(\pi_{2}|\pi_{1}=x_{1})=$
$\displaystyle\int_{\mathbb{R}}x_{2}\cdot{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2})d\mu_{2}(x_{2})=$
$\displaystyle\frac{\int_{\mathbb{R}}x_{2}\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(x_{2})}{\int_{\mathbb{R}}\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(x_{2})}.$

##  7 Examples.

######  Example 1.

Let us suppose that, for $\theta,\lambda,x_{1}>0$,
$P_{\theta}^{X_{1}}=G(1,\theta^{-1})$,
$P_{\theta}^{X_{2}|X_{1}=x_{1}}=G(1,(\theta x_{1})^{-1})$ and
$Q=G(1,\lambda^{-1})$, where $G(\alpha,\beta)$ denotes the gamma distribution
of parameters $\alpha,\beta>0$. Hence the joint density of $X_{1}$ and $X_{2}$
is

$f_{\theta}(x_{1},x_{2})=\theta^{2}x_{1}\exp\\{-\theta
x_{1}(1+x_{2})\\}I_{]0,\infty[^{2}}(x_{1},x_{2}).$

So, the density of $R_{\theta}^{n}$ is

$f_{n,\theta}(x^{\prime})=\theta^{2n}\cdot\prod_{i=1}^{n}x^{\prime}_{i1}\cdot\exp\big{\\{}-\theta\textstyle\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})\big{\\}}\cdot
I_{]0,\infty[^{2n}}(x^{\prime}),$

and the posterior $Q$-density given $x^{\prime}$ is

$\frac{dR^{*}_{n,x^{\prime}}(\theta)}{dQ}=:r^{*}_{n,x^{\prime}}(\theta)=K(x^{\prime})f_{n,\theta}(x^{\prime})$

where
$K(x^{\prime})=[\int_{0}^{\infty}f_{n,\theta}(x^{\prime})dQ(\theta)]^{-1}$.

So, the posterior predictive density given $x^{\prime}\in]0,\infty[^{2n}$ is

$\displaystyle
f^{*}_{n,x^{\prime}}(x)=\int_{0}^{\infty}f_{\theta}(x)r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)=$
$\displaystyle\lambda
K(x^{\prime})x_{1}\prod_{i=1}^{n}x^{\prime}_{i1}\int_{0}^{\infty}\theta^{2n+2}\exp\left\\{-\theta[\lambda+x_{1}(1+x_{2})+\textstyle\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]\right\\}d\theta\cdot
I_{]0,\infty[^{2}}(x).$

Since

$\int_{0}^{\infty}\theta^{n}\exp\\{-a\theta\\}d\theta=\frac{n!}{a^{n+1}},$

we have that

$f^{*}_{n,x^{\prime}}(x)=\frac{(2n+2)!\lambda
K(x^{\prime})x_{1}\prod_{i=1}^{n}x^{\prime}_{i1}}{[\lambda+x_{1}(1+x_{2})+\textstyle\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{(2n+3)}}$

and its first marginal is

$f^{*}_{n,x^{\prime},1}(x_{1})=\int_{\mathbb{R}}f^{*}_{n,x^{\prime}}(x_{1},x_{2})dx_{2}=\int_{0}^{\infty}\frac{A}{(Bt+C)^{m}}dt=\frac{A}{(m-1)BC^{m-1}}$

where

$m=2n+3,\quad A=(2n+2)!\lambda
K(x^{\prime})x_{1}\prod_{i=1}^{n}x^{\prime}_{i1},\quad
B=x_{1},\quad\hbox{and}\quad
C=\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2}).$

So

$f^{*}_{n,x^{\prime},1}(x_{1})=\frac{(2n+2)!\lambda
K(x^{\prime})x_{1}\prod_{i=1}^{n}x^{\prime}_{i1}}{(2n+2)x_{1}[\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+2}}.$

The Bayes estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2})=\theta x_{1}\exp\\{-\theta
x_{1}x_{2}\\}I_{]0,\infty[}(x_{2})$ is, for $x_{1},x_{2}>0$,

$\displaystyle{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f^{*}_{n,x^{\prime}}(x_{1},x_{2})}{f^{*}_{n,x^{\prime},1}(x_{1})}=$
$\displaystyle(2n+2)x_{1}\frac{[\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+2}}{[\lambda+x_{1}(1+x_{2})+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+3}}=$
$\displaystyle\frac{(2n+2)x_{1}a_{n}(x^{\prime},x_{1})^{2n+2}}{(x_{1}x_{2}+a_{n}(x^{\prime},x_{1}))^{2n+3}}$

where

$a_{n}(x^{\prime},x_{1})=\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2}).$

The Bayes estimator of the conditional distribution function

$F_{\theta}(x_{1},t):=P_{\theta}(X_{2}\leq t|X_{1}=x_{1})$

is, for $t>0$,

$\displaystyle\begin{split}F_{n}^{*}(x^{\prime},x_{1},t)&=\int_{0}^{t}(2n+2)x_{1}\frac{[\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+2}}{[\lambda+x_{1}(1+x_{2})+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+3}}dx_{2}\\\
&=\int_{0}^{t}\frac{(2n+2)x_{1}a_{n}(x^{\prime},x_{1})^{2n+2}}{(x_{1}x_{2}+a_{n}(x^{\prime},x_{1}))^{2n+3}}dx_{2}\\\
&=a_{n}(x^{\prime},x_{1})^{2n+2}\left(\frac{1}{a_{n}(x^{\prime},x_{1})^{2n+2}}-\frac{1}{(x_{1}t+a_{n}(x^{\prime},x_{1}))^{2n+2}}\right)\\\
&=1-\left(1+\frac{x_{1}t}{a_{n}(x^{\prime},x_{1})}\right)^{-2n-2}.\end{split}$

The Bayes estimator of the regression curve
$r_{\theta}(x_{1}):=E_{\theta}(X_{2}|X_{1}=x_{1})=\frac{1}{\theta x_{1}}$ is,
for $x_{1}>0$,

$m^{*}_{n}(x^{\prime},x_{1})=\int_{0}^{\infty}x_{2}\cdot{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2})dx_{2}=\frac{\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})}{(2n+1)x_{1}}=\frac{a_{n}(x^{\prime},x_{1})}{(2n+1)x_{1}}.$

######  Example 2.

Let us suppose that $X_{1}$ has a Bernoulli distribution of unknown parameter
$\theta\in]0,1[$ (i.e. $P_{\theta}^{X_{1}}=Bi(1,\theta)$) and, given
$X_{1}=k_{1}\in\\{0,1\\}$, $X_{2}$ has distribution $Bi(1,1-\theta)$ when
$k_{1}=0$ and $Bi(1,\theta)$ when $k_{1}=1$, i.e.
$P_{\theta}^{X_{2}|X_{1}=k_{1}}=Bi(1,k_{1}+(1-2k_{1})(1-\theta))$. We can
think of tossing a coin with probability $\theta$ of getting heads ($=1$) and
making a second toss of this coin if it comes up heads on the first toss, or
tossing a second coin with probability $1-\theta$ of making heads if the first
toss is tails ($=0$). Consider the uniform distribution on $]0,1[$ as the
prior distribution $Q$.

So, the joint probability function of $X_{1}$ and $X_{2}$ is

$\displaystyle\begin{split}f_{\theta}(k_{1},k_{2})&=\theta^{k_{1}}(1-\theta)^{1-k_{1}}[k_{1}+(1-2k_{1})(1-\theta)]^{k_{2}}[1-k_{1}-(1-2k_{1})(1-\theta)]^{1-k_{2}}\\\
&=\begin{cases}\theta(1-\theta)\hbox{ if }k_{2}=0,\\\ (1-\theta)^{2}\hbox{ if
}k_{1}=0,k_{2}=1,\\\ \theta^{2}\hbox{ if
}k_{1}=1,k_{2}=1.\end{cases}\end{split}$

The probability function of $R_{\theta}^{n}$ is

$f_{n,\theta}(k^{\prime})=\prod_{i=1}^{n}f_{\theta}(k^{\prime}_{i})=\theta^{a_{n}(k^{\prime})}(1-\theta)^{b_{n}(k^{\prime})}$

for
$k^{\prime}=(k^{\prime}_{1},\dots,k^{\prime}_{n})=(k^{\prime}_{11},k^{\prime}_{12},\dots,k^{\prime}_{n1},k^{\prime}_{n2})\in\\{0,1\\}^{2n}$,
where

$a_{n}(k^{\prime})=n_{+0}(k^{\prime})+2n_{11}(k^{\prime}),\quad
b_{n}(k^{\prime})=n_{+0}(k^{\prime})+2n_{01}(k^{\prime}),$

being $n_{j_{1}j_{2}}(k^{\prime})$ the number of indices $i\in\\{1,\dots,n\\}$
such that $(k^{\prime}_{i1},k^{\prime}_{i2})=(j_{1},j_{2})$ and
$n_{+j}=n_{0j}+n_{1j}$ for $j=0,1$. Notice that
$a_{n}(k^{\prime})+b_{n}(k^{\prime})=2n$.

Hence, the posterior $Q$-density given $k^{\prime}$ is

$r^{*}_{n,k^{\prime}}(\theta)=\frac{dR^{*}_{n,k^{\prime}}}{dQ}(\theta)=K(k^{\prime})f_{n,\theta}(k^{\prime})=K(k^{\prime})\theta^{a_{n}(k^{\prime})}(1-\theta)^{b_{n}(k^{\prime})}$

where

$K(k^{\prime})=\left[\int_{0}^{1}f_{n,\theta}(k^{\prime})dQ(\theta)\right]^{-1}=\frac{1}{B(a_{n}(k^{\prime})+1,b_{n}(k^{\prime})+1)},$

where $B(\alpha,\beta)$ stands for the Beta function.

So, the posterior predictive density given $k^{\prime}\in\\{0,1\\}^{2n}$ is

$\displaystyle\begin{split}f^{*}_{n,k^{\prime}}(k_{1},k_{2})&=\int_{0}^{1}f_{\theta}(k_{1},k_{2})r^{*}_{n,k^{\prime}}(\theta)dQ(\theta)\\\
&=K(k^{\prime})\int_{0}^{1}\theta^{a_{n}(k^{\prime})+k_{1}}(1-\theta)^{b_{n}(k^{\prime})+k_{2}}d\theta\\\
&=K(k^{\prime})B(a_{n}(k^{\prime})+k_{1}+1,b_{n}(k^{\prime})+k_{2}+1),\end{split}$

and its first marginal is

$\displaystyle
f^{*}_{n,k^{\prime},1}(k_{1})=K(k^{\prime})[B(a_{n}(k^{\prime})+k_{1}+1,b_{n}(k^{\prime})+1)+B(a_{n}(k^{\prime})+k_{1}+1,b_{n}(k^{\prime})+2)].$

The Bayes estimator of the conditional probability function

$f_{\theta}^{X_{2}|X_{1}=k_{1}}(k_{2})=[k_{1}+(1-2k_{1})(1-\theta)]^{k_{2}}[1-k_{1}-(1-2k_{1})(1-\theta)]^{1-k_{2}}$

is

$\displaystyle{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=k_{1}}(k_{2}):=\frac{f^{*}_{n,k^{\prime}}(k_{1},k_{2})}{f^{*}_{n,k^{\prime},1}(k_{1})}=\begin{cases}\frac{2n+2}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+3}&\hbox{
if }k_{1}=k_{2}=0,\vspace{1ex}\\\
\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+3}&\hbox{
if }k_{1}=0,k_{2}=1,\vspace{1ex}\\\
\frac{2n+3}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+4}&\hbox{ if
}k_{1}=1,k_{2}=0,\vspace{1ex}\\\
\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+4}&\hbox{
if }k_{1}=k_{2}=1.\end{cases}$

The Bayes estimator of the conditional mean
$r_{\theta}(k_{1}):=E_{\theta}(X_{2}|X_{1}=k_{1})=\theta^{k_{1}}(1-\theta)^{1-k_{1}}$
is, for $k_{1}=0,1$,

$m^{*}_{n}(k^{\prime},k_{1})={f^{*}_{n,k^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=k_{1}}(1)=\begin{cases}\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+3}&\hbox{
if }k_{1}=0,\vspace{1ex}\\\
\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+4}&\hbox{
if }k_{1}=1.\end{cases}$

######  Example 3.

Let $(X_{1},X_{2})$ have bivariate normal distribution with density

$\displaystyle
f_{\theta}(x):=\frac{1}{2\pi\sigma^{2}\sqrt{1-\rho^{2}}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[(x_{1}-\theta)^{2}-2\rho(x_{1}-\theta)(x_{2}-\theta)+(x_{2}-\theta)^{2}]\right\\}$
$\displaystyle=\frac{1}{2\pi\sigma^{2}\sqrt{1-\rho^{2}}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[x_{1}^{2}+x_{2}^{2}-2\rho
x_{1}x_{2}-2(1-\rho)(x_{1}+x_{2})\theta+2(1-\rho)\theta^{2}]\right\\},$

where $\sigma>0$ and $\rho\in]-1,1[$ are supposed to be kown. So

$\displaystyle R_{\theta}=N_{2}\left(\left(\begin{array}[]{c}\theta\\\
\theta\end{array}\right),\sigma^{2}\left(\begin{array}[]{cc}1&\rho\\\
\rho&1\end{array}\right)\right),\qquad
X_{1},X_{2}\sim_{\theta}N(\theta,\sigma^{2})$ $\displaystyle
P_{\theta}^{X_{2}|X_{1}=x_{1}}=N((1-\rho)\theta+\rho
x_{1},\sigma^{2}\sqrt{1-\rho^{2}}),\qquad
E_{\theta}(X_{2}|X_{1}=x_{1})=(1-\rho)\theta+\rho x_{1}.$

Hence, for
$x^{\prime}=(x^{\prime}_{1},\dots,x^{\prime}_{n})=(x^{\prime}_{11},x^{\prime}_{12},\dots,x^{\prime}_{n1},x^{\prime}_{n2})\in(\mathbb{R}^{2})^{n}$,

$\displaystyle
f_{n,\theta}(x^{\prime})=\prod_{i=1}^{n}f_{\theta}(x^{\prime}_{i})=$
$\displaystyle\frac{1}{[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}\textstyle\sum_{i=1}^{n}\left[(x^{\prime}_{i1}-\theta)^{2}-2\rho(x^{\prime}_{i1}-\theta)(x^{\prime}_{i2}-\theta)+(x^{\prime}_{i2}-\theta)^{2}\right]\right\\}.$

Let us consider the prior distribution $Q=N(\mu,\tau^{2})$ whose density is

$g(\theta)=\frac{1}{\tau\sqrt{2\pi}}\exp\left\\{-\frac{1}{2\tau^{2}}(\theta-\mu)^{2}\right\\}.$

The posterior $Q$-density is

$\displaystyle
r^{*}_{n,x^{\prime}}(\theta):=\frac{dR^{*}_{n,x^{\prime}}}{dQ}(\theta)=K_{1}(x^{\prime})f_{n,\theta}(x^{\prime})=$
$\displaystyle\frac{K_{1}(x^{\prime})}{[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[2n(1-\rho)\theta^{2}-2(1-\rho)s_{1}(x^{\prime})\theta+s_{2}(x^{\prime})-2\rho
p(x^{\prime})]\right\\}$

where

$\displaystyle
s_{1}(x^{\prime}):=\sum_{i}(x^{\prime}_{i1}+x^{\prime}_{i2}),\quad
s_{2}(x^{\prime}):=\sum_{i}({x^{\prime}_{i1}}^{2}+{x^{\prime}_{i2}}^{2}),\quad
p(x^{\prime})=\sum_{i}x^{\prime}_{i1}x^{\prime}_{i2},$ $\displaystyle
K_{1}(x^{\prime})=\left[\int_{\mathbb{R}}f_{n,x^{\prime}}(\theta)dQ(\theta)\right]^{-1}.$

Note that, writing
$c_{n}(\rho,\sigma,\tau)=[\tau\sqrt{2\pi}(2\pi\sigma^{2}\sqrt{1-\rho^{2}})^{n}]^{-1}$,

$\displaystyle\int_{\mathbb{R}}f_{n,x^{\prime}}(\theta)dQ(\theta)=$
$\displaystyle
c_{n}(\rho,\sigma,\tau)\int_{\mathbb{R}}\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[2n(1-\rho)\theta^{2}-2(1-\rho)s_{1}(x^{\prime})\theta+s_{2}(x^{\prime})-2\rho
p(x^{\prime})]-\frac{1}{2\tau^{2}}(\theta-\mu)^{2}]\right\\}d\theta$
$\displaystyle=c_{n}(\rho,\sigma,\tau)\int_{\mathbb{R}}\exp\\{-(A_{1}\theta^{2}-B_{1}(x^{\prime})\theta+C_{1}(x^{\prime}))\\}d\theta$
$\displaystyle=\frac{\exp\left\\{-\left(C_{1}(x^{\prime})-\frac{B_{1}^{2}(x^{\prime})}{4A_{1}}\right)\right\\}}{\tau\sqrt{2\pi}[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n}}\int_{\mathbb{R}}\exp\left\\{-A_{1}\left(\theta-\frac{B_{1}(x^{\prime})}{2A_{1}}\right)^{2}\right\\}d\theta$
$\displaystyle=\frac{\exp\left\\{-\left(C_{1}(x^{\prime})-\frac{B_{1}^{2}(x^{\prime})}{4A_{1}}\right)\right\\}}{\tau\sqrt{2A_{1}}[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n}}$

where

$\displaystyle A_{1}=\frac{n}{\sigma^{2}(1+\rho)}+\frac{1}{2\tau^{2}},\quad
B_{1}(x^{\prime})=\frac{s_{1}(x^{\prime})}{\sigma^{2}(1+\rho)}+\frac{\mu}{\tau^{2}},\quad
C_{1}(x^{\prime})=\frac{s_{2}(x^{\prime})-2\rho
p(x^{\prime})}{2{\sigma^{2}(1-\rho^{2})}}+\frac{\mu^{2}}{2\tau^{2}}.$

Hence

$K_{1}(x^{\prime})=\tau\sqrt{2A_{1}}[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n}\exp\left\\{C_{1}(x^{\prime})-\frac{B_{1}^{2}(x^{\prime})}{4A_{1}}\right\\}$

The posterior predictive density given $x^{\prime}\in(\mathbb{R}^{2})^{n}$ is

$\displaystyle
f_{n,x^{\prime}}^{*}(x):=\int_{\mathbb{R}}f_{\theta}(x)r_{n,x^{\prime}}^{*}(\theta)g(\theta)d\theta=\frac{K_{1}(x^{\prime})}{\tau\sqrt{2\pi}[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n+1}}\cdot$
$\displaystyle\int_{\mathbb{R}}\exp\left\\{-\frac{[2(n+1)(1-\rho)\theta^{2}-2(1-\rho)(x_{1}+x_{2}+s_{1}(x^{\prime}))\theta+(x_{1}^{2}+x_{2}^{2}+s_{2}(x^{\prime}))-2\rho(x_{1}x_{2}+p(x^{\prime}))]}{2\sigma^{2}(1-\rho^{2})}\right.$
$\displaystyle\left.-\frac{(\theta^{2}-2\mu\theta+\mu^{2})}{2\tau^{2}}\right\\}d\theta=$
$\displaystyle
K_{2}(x^{\prime})\int_{\mathbb{R}}\exp\\{-(A_{2}\theta^{2}-B_{2}(x,x^{\prime})\theta+C_{2}(x,x^{\prime}))\\}d\theta=$
$\displaystyle
K_{2}(x^{\prime})\exp\left\\{-\left(C_{2}(x,x^{\prime})-\frac{B_{2}^{2}(x,x^{\prime})}{4A_{2}}\right)\right\\}\int_{\mathbb{R}}\exp\left\\{-A_{2}\left(\theta-\frac{B_{2}(x,x^{\prime})}{2A_{2}}\right)^{2}\right\\}d\theta=$
$\displaystyle
K_{2}(x^{\prime})\sqrt{\frac{\pi}{A_{2}}}\exp\left\\{-\left(C_{2}(x,x^{\prime})-\frac{B_{2}^{2}(x,x^{\prime})}{4A_{2}}\right)\right\\}$

where

$\displaystyle
K_{2}(x^{\prime})=\frac{K_{1}(x^{\prime})}{\tau\sqrt{2\pi}[2\pi\sigma^{2}\sqrt{1-\rho^{2}}]^{n+1}}=\frac{\sqrt{A_{1}}\exp\left\\{C_{1}(x^{\prime})-\frac{B_{1}^{2}(x^{\prime})}{4A_{1}}\right\\}}{2\pi^{3/2}\sigma^{2}\sqrt{1-\rho^{2}}},\quad
A_{2}=\frac{(n+1)(1+\rho)}{\sigma^{2}}+\frac{1}{2\tau^{2}}$ $\displaystyle
B_{2}(x.x^{\prime})=\frac{x_{1}+x_{2}+s_{1}(x^{\prime})}{\sigma^{2}(1+\rho)}+\frac{\mu}{\tau^{2}},\quad
C_{2}(x,x^{\prime})=\frac{x_{1}^{2}+x_{2}^{2}+s_{2}(x^{\prime})-2\rho(x_{1}x_{2}+p(x^{\prime}))}{2\sigma^{2}(1-\rho^{2})}+\frac{\mu^{2}}{\tau^{2}}$

We can write

$\displaystyle
C_{2}(x,x^{\prime})-\frac{B_{2}^{2}(x,x^{\prime})}{4A_{2}}=A_{3}x_{1}^{2}+A_{3}x_{2}^{2}+B_{3}x_{1}x_{2}+C_{3}(x^{\prime})(x_{1}+x_{2})+D_{3}(x^{\prime})$

where

$\displaystyle
A_{3}=\frac{1}{2\sigma^{2}(1-\rho^{2})}-\frac{\tau^{2}}{\sigma^{2}(1+\rho)^{2}[4(n+1)(1+\rho)\tau^{2}+2\sigma^{2}]},$
$\displaystyle
B_{3}=-\frac{\rho}{\sigma^{2}(1-\rho^{2})}-\frac{\tau^{2}}{\sigma^{2}(1+\rho)^{2}[2(n+1)(1+\rho)\tau^{2}+\sigma^{2}]},$
$\displaystyle
C_{3}(x^{\prime})=-\frac{\tau^{2}s_{1}(x^{\prime})+\mu\sigma^{2}(1+\rho)}{\sigma^{2}(1+\rho)^{2}[2(n+1)(1+\rho)\tau^{2}+\sigma^{2}]},$
$\displaystyle
D_{3}(x^{\prime})=\frac{\tau^{2}s_{2}(x^{\prime})-2\rho\tau^{2}p(x^{\prime})+2\mu^{2}\sigma^{2}(1-\rho^{2})}{2\tau^{2}\sigma^{2}(1-\rho^{2})}-\frac{\tau^{4}s_{1}(x^{\prime})^{2}+\mu^{2}\sigma^{4}(1+\rho)^{2}+2\mu\sigma^{2}(1+\rho)s_{1}(x^{\prime})}{\tau^{2}\sigma^{2}(1+\rho)^{2}[4(n+1)(1+\rho)\tau^{2}+2\sigma^{2}]}$

It is readily shown that $A_{3}>0$. It follows that the posterior predictive
density $f^{*}_{n,x^{\prime}}$ is the density of a normal bivariate
distribution $N_{2}(m(x^{\prime}),\Sigma)$ where

$m(x^{\prime})=\binom{m_{1}(x^{\prime})}{m_{2}(x^{\prime})},\quad\Sigma=\sigma_{1}^{2}\left(\begin{array}[]{cc}1&\rho_{1}\\\
\rho_{1}&1\end{array}\right)$

being

$\rho_{1}=\frac{B_{3}}{2A_{3}},\quad\sigma_{1}^{2}=\frac{2A_{3}}{4A_{3}^{2}-B_{3}^{2}},\quad
m_{1}(x^{\prime})=m_{2}(x^{\prime})=-\frac{C_{3}(x^{\prime})}{2(1-\rho_{1})}.$

It is easy to see that, as can be expected, $|\rho_{1}|<1$ and
$\sigma_{1}^{2}>0$. If we denote

$a_{n}(\rho,\sigma,\tau):=2(n+1)(1+\rho)+\frac{\sigma^{2}}{\tau^{2}}$

We can write

$\displaystyle\rho_{1}=-\frac{a_{n}(\rho,\sigma,\tau)+\frac{1-\rho}{1+\rho}}{a_{n}(\rho,\sigma,\tau)-\frac{1-\rho}{1+\rho}}\cdot\rho,\quad\sigma_{1}^{2}=\frac{a_{n}(\rho,\sigma,\tau)}{a_{n}(\rho,\sigma,\tau)-\frac{1-\rho}{1+\rho}}\cdot\sigma^{2},$
$\displaystyle
m_{1}(x^{\prime})=m_{2}(x^{\prime})=\frac{s_{1}(x^{\prime})+(1+\rho)\frac{\sigma^{2}}{\tau^{2}}\mu}{2(1-\rho_{1})(1+\rho)^{2}\sigma^{2}a_{n}(\rho,\sigma,\tau)}$

It follows that the conditional distribution

$\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}\right)^{\pi_{2}|\pi_{1}=x_{1}}\\!\\!:=N\big{(}(1-\rho_{1})m_{1}(x^{\prime})+\rho_{1}x_{1},\sigma_{1}^{2}(1-\rho_{1}^{2})\big{)}$

is the Bayes estimator of the conditional distribution

$P_{\theta}^{X_{2}|X_{1}=x_{1}}=N\big{(}(1-\rho)\theta+\rho
x_{1},\sigma^{2}(1-\rho^{2})\big{)}$

for the squared total variation function, and its density
${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!\pi_{2}|\pi_{1}=x_{1}}$ is the Bayes
estimator of the conditional density

$f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2})=\frac{1}{\sigma\sqrt{2\pi(1-\rho^{2})}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[x_{2}-(1-\rho)\theta-\rho
x_{1}]^{2}\right\\}$

for the $L^{1}$-squared loss function.

Moreover, its mean

$E_{{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}}(\pi_{2}|\pi_{1}=x_{1})=(1-\rho_{1})m_{1}(x^{\prime})+\rho_{1}x_{1}$

is the Bayes estimator of the regression curve

$E_{\theta}(X_{2}|X_{1}=x_{1})=(1-\rho)\theta+\rho x_{1}$

for the squared error loss function.

##  8 Acknowledgements.

This paper has been supported by the Junta de Extremadura (Spain) under the
grant Gr18016.

##  9 References.

  * •

Barra, J.R. (1971) Notions Fondamentales de Statistique Mathématique, Dunod,
Paris.

  * •

Bogachev, V.I. (2007) Measure Theory, Vol. II, Springer, Berlin.

  * •

Florens, J.P., Mouchart, M., Rolin, J.M. (1990) Elements of Bayesian
Statistics, Marcel Dekker, New York.

  * •

Geisser, S. (1993) Predictive Inference: An Introduction, Springer Science+
Business Media, Dordrecht.

  * •

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.
(2014) Bayesian Data Analysis, 3rd ed., CRC Press.

  * •

Ghosal, S., Vaart, A.v.d. (2017) Fundamentals of Noparametric Bayesian
Inference, Cambridge University Press, Cambridge UK.

  * •

Lehmann, E.L., Casella, G. (1998) Theory of Point Estimation, Second Edition,
Springer, New York.

  * •

Nadaraya, E.A. (1989) Nonparametric Estimation of Probability Densities and
Regression Curves, Kluwer Academic Publishers, Dordrecht.

  * •

Nogales, A.G. (2021) On Bayesian Estimation of Densities and Sampling
Distributions: the Posterior Predictive Distribution as the Bayes Estimator,
to appear in Statistica Neerlandica.

  * •

Rubin, D.B. (1984) Bayesianly justifiable and relevant frequency calculations
for the applied statisticians, The Annals of Statistics, 12(4), 1151-1172.