The Mixtures and the Neural Critics: On the Pointwise Mutual Information
Profiles of Fine Distributions

Paweł Czyż∗ 1,2 Frederic Grabowski∗ 3

Julia E. Vogt4,5 Niko Beerenwinkel† 1,5 Alexander Marx† 2,4

1Department of Biosystems Science and Engineering, ETH Zurich 2ETH AI Center,
ETH Zurich 3Institute of Fundamental Technological Research, Polish Academy of
Sciences 4Department of Computer Science, ETH Zurich 5SIB Swiss Institute of
Bioinformatics

###### Abstract

Mutual information quantifies the dependence between two random variables and
remains invariant under diffeomorphisms. In this paper, we explore the
pointwise mutual information profile, an extension of mutual information that
maintains this invariance. We analytically describe the profiles of
multivariate normal distributions and introduce the family of _fine
distributions_ , for which the profile can be accurately approximated using
Monte Carlo methods. We then show how fine distributions can be used to study
the limitations of existing mutual information estimators, investigate the
behavior of neural critics used in variational estimators, and understand the
effect of experimental outliers on mutual information estimation. Finally, we
show how fine distributions can be used to obtain model-based Bayesian
estimates of mutual information, suitable for problems with available domain
expertise in which uncertainty quantification is necessary.

## 1 Introduction

Mutual information (MI) is a non-parametric statistical measure used to
determine the dependency between two random variables (r.v.) and numerous
approaches have been proposed to estimate it (Kay, 1992; Kraskov et al., 2004;
Cellucci et al., 2005; Belghazi et al., 2018). In recent work, Czyż et al.
(2023) proposed a family of distributions for which ground-truth MI is
analytically tractable to evaluate the performance of selected estimators on
different distributions. They concluded that there is no universally acclaimed
state-of-the-art estimator, with different methods performing best under
different assumptions on the data-generating process, which had been often
implicitly assumed. In parametric statistics, generative models are used to
explicitly state the assumptions about the data-generating process. Although
such model-based estimators have been used in settings with discrete r.v.
(Hutter, 2001; Brillinger, 2004), the only model-based MI estimator for
continuous r.v. is canonical correlation analysis (CCA), proposed by Kay
(1992) and Brillinger (2004). As CCA assumes that the joint distribution is
multivariate normal, we consider the development of more advanced model-based
MI estimators to be an important open problem.

A second and related problem is the construction of new distributions with
known ground-truth MI, which can be used to assess the performance of
different estimators under varying distributional assumptions. Czyż et al.
(2023) used multivariate normal and Student distributions (and their
transformations) with analytically tractable MI to model the sparsity of
interactions and effects of long tails. However, these distributions are far
from universal: they are not suitable to model communication channels in which
experimental failures result in measurement outliers or distributions used in
biology (Grabowski et al., 2019) and physics (Carrara & Ernst, 2023), which
involve both continuous and discrete r.v. Moreover, reparametrizations with
diffeomorphisms do not change the pointwise mutual information profile (the
distribution of pointwise mutual information), limiting the expressivity of
constructed distributions.

In this work, we address both problems: we propose a family of _fine
distributions_ for which the ground-truth MI can be reliably estimated using
Monte Carlo methods with arbitrarily high accuracy. The proposed distributions
include normalizing flows (Kobyzev et al., 2021; Papamakarios et al., 2021)
and their finite mixtures, allow for discrete and continuous random variables,
and can be used to investigate the phenomena involving outliers resulting from
failed experimental measurements. Moreover, we show how they can be used to
create expressive generative models which provide model-based mutual
information estimates, beyond the linear case of CCA.

##### Contributions

We introduce a novel family of fine distributions, for which mutual
information and the pointwise mutual information profile can be accurately
determined using Monte Carlo methods (Sec. 2). In Sec. 3 we demonstrate three
different applications of fine distributions: first, we show how to construct
low-dimensional distributions posing significant challenges to existing mutual
information estimators (Sec. 3.1). Second, we use the introduced family of
fine distributions to study the robustness of mutual information estimators to
inliers and outliers (Sec. 3.2). Third, we show how to apply fine
distributions to understand the biases and sample efficiency of variational
estimator employing neural critics (Sec. 3.3). Finally, we demonstrate how the
introduced family of fine distributions can be used to provide model-based
mutual information estimates (Sec. 4), allowing for Bayesian inference of
mutual information.

Figure 1: First two panels: samples from a bivariate normal distribution and a
transformed distribution. Both distributions have the same PMI profile (blue
histogram in the fourth panel). Third panel: mixture distribution with
distinct PMI profile, which cannot be obtained as a transformation of
multivariate normal distribution due to a different PMI profile (pink
histogram in the fourth panel). All three distributions have the same mutual
information, marked with the black line in the fourth panel.

## 2 Theoretical framework

In this paper, we consider random variables valued in smooth manifolds without
boundary (Lee, 2012, Ch. 1) equipped with given reference measures. In
particular, our results apply to both categorical and continuous random
variables, as the considered manifolds include open subsets of the Euclidean
space $\mathbb{R}^{n}$ (equipped with the Lebesgue measure) as well as zero-
dimensional discrete spaces $\\{1,\dotsc,m\\}$ (equipped with the counting
measure). For a given pair of smooth manifolds $\mathcal{X}$ and
$\mathcal{Y}$, we equip their product $\mathcal{X}\times\mathcal{Y}$ with the
product measure and define the set
$\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ to consist of all
probability measures $P_{XY}$ on $\mathcal{X}\times\mathcal{Y}$ such that the
measures $P_{XY}$, $P_{X}(A)=P_{XY}(A\times\mathcal{Y})$ and
$P_{Y}(B)=P_{XY}(\mathcal{X}\times B)$ have smooth and positive PDFs (or PMFs)
$p_{XY}$, $p_{X}$ and $p_{Y}$ with respect to the reference measures on
$\mathcal{X}\times\mathcal{Y}$, $\mathcal{X}$ and $\mathcal{Y}$, respectively.
The primary reason for studying
$\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ is a convenient formula
for pointwise mutual information (Pinsker & Feinstein, 1964, Ch. 2):

######  Definition 1 (Pointwise Mutual Information).

Let $X$ and $Y$ be random variables valued in $\mathcal{X}$ and $\mathcal{Y}$,
respectively, such that
$P_{XY}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$. The pointwise
mutual information (PMI) is defined111We use the natural logarithm, meaning
that all quantities are measured in nats. as

$\mathrm{PMI}_{XY}(x,y)=\log\frac{p_{XY}(x,y)}{p_{X}(x)\,p_{Y}(y)}.$

Note that from smoothness and positivity of all the densities it follows that
PMI is a smooth, hence measurable, function, s.t. the following distribution
exists:

######  Definition 2 (PMI profile).

The pointwise mutual information profile222Although we are not aware of a
prior formal definition and studies of the PMI profile, histograms of
approximate PMI between words have been studied before in the computational
linguistics community (Allen & Hospedales, 2019).  $\mathrm{Prof}_{XY}$ is
defined as the distribution of the random variable $T=\mathrm{PMI}_{XY}(X,Y).$

The mutual information is the first moment of the profile,
$\mathbf{I}\\!\left(X;Y\right)=\mathbb{E}_{T\sim\mathrm{Prof}_{XY}}[T],$ and
is known to be invariant under diffeomorphisms (Kraskov et al., 2004,
Appendix). More generally, in Appendix A.1 we show that the whole profile is
invariant (see Fig. 1):

######  Theorem 3.

Let $P_{XY}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ and
$f\colon\mathcal{X}\to\mathcal{X}$ and $g\colon\mathcal{Y}\to\mathcal{Y}$ be
diffeomorphisms. Then for $X^{\prime}=f(X)$ and $Y^{\prime}=g(Y)$ it holds
that
$P_{X^{\prime}Y^{\prime}}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$
and $\mathrm{Prof}_{XY}=\mathrm{Prof}_{X^{\prime}Y^{\prime}}.$

In Appendix A.2 we prove the following three results, characterizing the
profiles of simple distributions:

######  Proposition 4.

Let $X$ and $Y$ be r.v. with joint distribution
$P_{XY}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$. Then,
$\mathbf{I}\\!\left(X;Y\right)=0$ if and only if
$\mathrm{Prof}_{XY}=\delta_{0}$ is the Dirac measure with a single atom at
$0$.

######  Proposition 5.

If $X$ and $Y$ are discrete r.v. with
$P_{XY}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$, then the PMI
profile is discrete:

$\mathrm{Prof}_{XY}=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}p_{XY}(x,y)\,\delta_{\mathrm{PMI}_{XY}(x,y)}.$

######  Theorem 6.

Let $X$ and $Y$ be r.v. such that the joint distribution
$P_{XY}\in\mathcal{P}\\!\left({\mathbb{R}^{m},\mathbb{R}^{n}}\right)$ is
multivariate normal. If $k=\min(m,n)$ and $\rho_{1},\rho_{2},\dotsc,\rho_{k}$
are canonical correlations (Jendoubi & Strimmer, 2019) between $X$ and $Y$,
then the profile $\mathrm{Prof}_{XY}$ is a generalized $\chi^{2}$
distribution, namely the distribution of the variable

$T=\mathbf{I}\\!\left(X;Y\right)+\sum_{i=1}^{k}\frac{\rho_{i}}{2}(Q_{i}-Q^{\prime}_{i}),$

where $Q_{i}$ and $Q^{\prime}_{i}$ are i.i.d. variables sampled according to
the $\chi^{2}_{1}$ distribution. In particular, $\mathrm{Prof}_{XY}$ is
symmetric around its median, which coincides with the mean
$\mathbf{I}\\!\left(X;Y\right)=-\frac{1}{2}\sum_{i=1}^{k}\log\left(1-\rho_{i}^{2}\right),$
and all the moments of this distribution exist; its variance is equal to
$\rho_{1}^{2}+\cdots+\rho_{k}^{2}$.

However, analytic formulae governing PMI profiles are not known except for the
simplest cases above and their reparametrizations by diffeomorphisms due to
Theorem 3. Below we show how to construct a family of distributions with
diverse PMI profiles, for which the PMI profile can be approximated
numerically.

######  Definition 7.

We call a distribution
$P_{XY}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ _fine_ if it
is possible to sample $(X,Y)\sim P_{XY}$ and for every point
$(x,y)\in\mathcal{X}\times\mathcal{Y}$ we can analytically evaluate the
densities $p_{XY}(x,y)$, $p_{X}(x)$ and $p_{Y}(y)$.

For example, discrete distribution as well as multivariate normal and Student
distributions are fine. With the advancements in probabilistic computation, we
can expect that the list of known fine distributions will grow over time.

The properties are chosen so that we can estimate the PMI profile and mutual
information with Monte Carlo approaches. Namely, observe that it is easy to
sample $T\sim\mathrm{Prof}_{XY}$ by sampling a data point $(x,y)$ and
evaluating $t=\mathrm{PMI}_{XY}(x,y)$. Then, MI can be approximated with a
Monte Carlo estimate of the integral
$\mathbf{I}\\!\left(X;Y\right)=\mathbb{E}[T].$ Assuming
$\mathbf{I}\\!\left(X;Y\right)<\infty$, the Monte Carlo estimator of the
mutual information is guaranteed to be unbiased. For a detailed discussion of
Monte Carlo standard error (MCSE) under different regularity conditions see
Flegal et al. (2008).

Analogously, to estimate the PMI profile, we can approximate it with a
histogram: for bin $B\subset\mathbb{R}$ one can introduce its indicator
function $\mathbf{1}_{B}$ and integrate $\mathbb{E}[\mathbf{1}_{B}(T)].$ Its
cumulative density function can be approximated with an empirical sample using
the expectations $\mathbb{E}[\mathbf{1}_{(-\infty,a_{n}]}(T)]$ for a given
sequence $(a_{n})$. As the characteristic functions are bounded between $0$
and $1$, the Monte Carlo estimator of both quantities is unbiased and has
standard error bounded from above by $1/\sqrt{4n}$ due to the inequality of
Popoviciu (1935).

We can enlarge the family of known fine distributions with two constructions,
with proofs given in Appendix A.4. First, we can transform known fine
distributions with diffeomorphisms with tractable Jacobians, such as
normalizing flows (Papamakarios et al., 2021; Kobyzev et al., 2021):

######  Proposition 8.

If $P_{XY}$ is fine and $f$ and $g$ are diffeomorphisms with tractable
Jacobians, then $P_{f(X)g(Y)}$ is fine.

As noted in Theorem 3, the above construction does not change the PMI profile.
However, it is possible to obtain more expressive profiles by using finite
mixtures of fine distributions (see Fig. 1):

######  Proposition 9.

Consider a mixture distribution $P_{X^{\prime}Y^{\prime}}$ with weights
$w_{1},\dotsc,w_{K}$ and component distributions
$P_{X_{1}Y_{1}},\dotsc,P_{X_{K}Y_{K}}$. If all component distributions are
fine, then also the mixture distribution is fine.

Since Gaussian mixture models are universal approximators of smooth densities,
and they belong to the fine distribution family, in principle one could use
these constructions to approximate any distribution. In practice, the
evaluation time of PMI grows linearly with the number of components and can
become costly for a high number of components (or when large normalizing flow
architectures are used). We consider estimating mutual information of mixture
distributions an important problem due to their counterintuitive ability to
create and destroy information (Haussler & Opper, 1997; Kolchinsky & Tracey,
2017):

######  Proposition 10.

Consider r.v. $(X_{k},Y_{k})$ such that
$\mathbf{I}\\!\left(X_{k};Y_{k}\right)<\infty$ for $k=1,\dotsc,K$ and their
mixture $(X^{\prime},Y^{\prime})$ with weights $w_{1},\dotsc,w_{K}$. The
following inequalities apply:

$0\leq\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\right)\leq\log
K+\sum_{k=1}^{K}w_{k}\,\mathbf{I}\\!\left(X_{k};Y_{k}\right).$

Moreover, these inequalities are tight:

  1. 1.

There exists a mixture such that
$\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\right)=\log K$ even though
$\mathbf{I}\\!\left(X_{k};Y_{k}\right)=0$ for all $k$.

  2. 2.

There exists a mixture such that
$\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\right)=0$ even though
$\mathbf{I}\\!\left(X_{k};Y_{k}\right)>0$ for all $k$.

For completeness, we include a proof in Appendix A.6.

## 3 Case studies

In this section, we apply fine distributions to three distinct problems. In
Sec. 3.1 we demonstrate how they can be used to extend existing benchmarks of
mutual information estimators. In Sec. 3.2 we show how fine can be used in
experimental sciences to investigate the robustness of mutual information to
outliers and inliers. In Sec. 3.3 we apply fine to investigate the biases and
sample efficiency of variational estimators employing neural critics.

###  3.1 Extending the Benchmarks

Figure 2: Samples from the proposed distributions. Distributions X and AI
represent one-dimensional variables $X$ and $Y$. Distributions Waves and
Galaxy plot two-dimensional $X$ variable using spatial coordinates, while one-
dimensional $Y$ variable is represented by color. The rightmost plot presents
estimates according to different mutual information algorithms using
independently generated data sets with $N=5\,000$ points each, compared to the
ground-truth mutual information of the distribution (dotted line).

In this section, we show how expressive the fine distributions are by
visualizing them and demonstrate how they can be applied to investigate the
limits of the applicability of mutual information estimators. Czyż et al.
(2023) benchmarked mutual information estimators using r.v. $(X,Y)$
distributed according to multivariate normal and Student distributions (for
which mutual information is analytically tractable Arellano-Valle et al.
(2013)) and their transformations $(f(X),g(Y))$, where $f$ and $g$ are
continuous injective functions (which ensures that
$\mathbf{I}\\!\left(f(X);g(Y)\right)=\mathbf{I}\\!\left(X;Y\right)$).

However, transforming only multivariate normal and Student distributions does
not ensure that the family of distributions is diverse enough. In particular,
if $f$ and $g$ were diffeomorphisms, the diversity of the PMI profiles would
be very limited due to Theorem 3. The proposed family of fine distributions
can be used as a more expressive alternative to the multivariate normal and
Student distributions. Although the analytical formula for ground-truth mutual
information of fine distributions is not available, it can be reliably
estimated with a Monte Carlo estimator.

To illustrate this application, we implemented four low-dimensional
distributions in TensorFlow Probability on JAX (Dillon et al., 2017; Bradbury
et al., 2018) assuming that $X$ and $Y$ are valued in Euclidean spaces.
Additionally, in Appendix B, we consider cases involving discrete variables.
We visualise samples from the distributions considered in Fig. 2 and defer the
detailed description to Appendix C.1; the X distribution is a mixture of two
bivariate normal distributions. The marginal distributions $P_{X}$ and $P_{Y}$
are normal, although the joint distribution $P_{XY}$ is not. The AI
distribution is a mixture of six bivariate normal distributions, illustrating
how expressive fine distributions can be. To illustrate that fine
distributions are more expressive than mixtures of normal distributions, we
consider a two-dimensional $X$ variable and a one-dimensional $Y$ variable.
The Waves distribution was created using twelve multivariate normal
distributions in which the $X$ variable has been transformed by a
diffeomorphism. The Galaxy distribution was created using two multivariate
normal distributions in which the $X$ variable was transformed with the spiral
diffeomorphism (Czyż et al., 2023).

Analogous distributions could be also defined in higher dimensions or
transformed via continuous injections, which preserve mutual information. We
do not aim to create a rigorous benchmark in this work, so we restricted our
attention only to these four distributions, and for each of them we sampled
ten data sets with $N=5\,000$ points and applied five estimators: the
histogram-based estimator (Cellucci et al., 2005; Darbellay & Vajda, 1999),
the popular KSG estimator (Kraskov et al., 2004), canonical correlation
analysis (Kay, 1992; Brillinger, 2004) and two neural estimators: InfoNCE
(Oord et al., 2018) and MINE (Belghazi et al., 2018) (see Appendix C.2 for
hyperparameters used). The estimates are shown in Fig. 2.

Even though the considered problems are low-dimensional and do not encode more
information than 1.5 nats, they pose a considerable challenge for the
estimators. The performance of the KSG estimator is consistently the best one,
with the Waves task not solved by any estimator. The CCA estimator, excelling
at distributions that are close to multivariate normal (Czyż et al., 2023), is
not able to capture any information at all. This suggests that fine
distributions can provide a rich set of distributions that can be used to test
mutual information estimators.

###  3.2 Modeling Outliers

Figure 3: Left: increasing the contamination level $\alpha$ with inlier noise
distribution (covariance of $N_{\tilde{Y}}$ is of similar scale as the signal
distribution $P_{XY}$). Middle: increasing the contamination level $\alpha$
with outlier noise distribution (noise covariance $N_{\tilde{Y}}$ has $5^{2}$
larger scale than the signal distribution $P_{XY}$). Right: increasing the
variance of the noisy normal distribution for constant contamination of
$20\%$. Note that outliers (corresponding to very small or large covariance of
$n(y)$) have less impact than inliers (corresponding to covariance of $n(y)$
similar to the covariance of the signal distribution $P_{XY}$).

In this section, we use fine distributions to study the effect of inliers and
outliers on mutual information estimation. Consider an electric circuit or a
biological system modeled as a communication channel $p_{Y\mid X}(y\mid x)$.
The researcher controls the input variable $X$, which results in a
distribution $P_{X}$, and subsequently measures the outcome variable $Y$. The
mutual information $\mathbf{I}\\!\left(X;Y\right)$ is then estimated based on
the collected experimental samples (Nałȩcz-Jawecki et al., 2023).

However, every experimental system can suffer from occasional failures. We
model the output of a failing system with a noise distribution with a PDF
$n(y)$. If the probability of system failure does not depend on input value
$x$, the communication channel becomes a mixture:

$p_{Y^{\prime}\mid X}(y\mid x)=(1-\alpha)p_{Y\mid X}(y\mid x)+\alpha n(y).$

By multiplying both sides by the density $p_{X}(x)$, the distribution of the
channel inputs provided by the scientist, we arrive at the mixture
distribution $P_{XY^{\prime}}$ with PDF

$p_{XY^{\prime}}(x,y)=(1-\alpha)p_{XY}(x,y)+\alpha n(y)p_{X}(x).$

If the system failure is unnoticed, one can only measure the r.v.
$Y^{\prime}$, rather than $Y$. It is therefore of interest to understand how
much $\mathbf{I}\\!\left(X;Y\right)$ and
$\mathbf{I}\\!\left(X;Y^{\prime}\right)$ can differ under realistic
assumptions on the noise $n(y)$ and whether the standard mutual information
estimation techniques are robust to it. In Appendix A.3 we prove the following
upper bound:

######  Proposition 11.

Consider variables $X$, $Y$ and $Y^{\prime}$, such that

$p_{XY^{\prime}}(x,y)=(1-\alpha)p_{XY}(x,y)+\alpha n(y)p_{X}(x)$

with contamination level $\alpha\in[0,1]$. Then,

$\mathbf{I}\\!\left(X;Y^{\prime}\right)\leq(1-\alpha)\,\mathbf{I}\\!\left(X;Y\right).$

However, we can use fine distributions to evaluate this quantity exactly. We
consider a setting with a two-dimensional input variable $X$ and two-
dimensional output variables $Y$ (perfect output) and $Y^{\prime}$
(contaminated output). As the joint density $p_{XY}$ we used a multivariate
normal with unit scale and correlations
$\mathrm{corr}(X_{1},Y_{1})=\mathrm{corr}(X_{2},Y_{2})=0.8$ and for the noise
$n(y)$ we used a multivariate normal distribution with covariance
$\sigma^{2}I_{2}$. If $\sigma^{2}\approx 1$ this results in inliers, where the
noise distribution is hard to distinguish from the signal. For $\sigma^{2}\ll
1$ the system failures are all close to $0$, while outliers are present for
$\sigma^{2}\gg 1$.

In Fig. 3 we present the results of three experiments: in the first two, we
changed the contamination level $\alpha\in[0,0.5]$ for $\sigma^{2}=1$ (inlier
noise) and $\sigma^{2}=5^{2}$ (outlier noise) respectively. In the third
experiment, we fixed $\alpha=0.2$ and varied
$\sigma^{2}\in\left[2^{-7},2^{8}\right]$. We see that the inlier noise results
in a slightly faster decrease of mutual information, while the outlier noise
decreases almost linearly following the upper bound
$(1-\alpha)\mathbf{I}\\!\left(X;Y\right)$. Interestingly, in this low-
dimensional setting, the KSG, MINE, and InfoNCE estimators reliably estimate
the mutual information $\mathbf{I}\\!\left(X;Y^{\prime}\right)$, which can
significantly differ from $\mathbf{I}\\!\left(X;Y\right)$. Although CCA would
be the preferred method to estimate $\mathbf{I}\\!\left(X;Y\right)$ without
any noise (Czyż et al., 2023), even a small number of outliers ($\alpha=5\%$)
can result in unreliable estimates.

###  3.3 Understanding Neural Critics

Figure 4: Left: PDF of the considered distribution. Middle: neural critic and
PMI values. Right: normalized neural critic and PMI profiles.  Figure 5:
Estimation of mutual information using a function approximating PMI as a
function of sample size for Monte Carlo (MC), InfoNCE, Donsker-Varadhan and
NWJ losses. From left to right: true PMI function is used, a constant bias is
added, a functional bias is added. The rightmost plot: true PMI function is
used in a high-dimensional problem.

Variational estimators of mutual information optimize a critic
$f\colon\mathcal{X}\times\mathcal{Y}\to\mathbb{R}$ to obtain an approximate
lower bound on mutual information. For example, the Donsker–Varadhan
variational approximation (Belghazi et al., 2018)

$I_{\text{DV}}=\sup_{f}\left(\mathbb{E}_{P_{XY}}[f]-\log\mathbb{E}_{P_{X}\otimes
P_{Y}}[\exp f]\right)$

is a lower bound and $I_{\text{DV}}=\mathbf{I}\\!\left(X;Y\right)$ whenever
$f=\mathrm{PMI}_{XY}+c$ for any real number $c$. Analogously, the lower bound
of Nguyen et al. (2007),

$I_{\text{NWJ}}=\sup_{f}\left(\mathbb{E}_{P_{XY}}[f]-\mathbb{E}_{P_{X}\otimes
P_{Y}}[\exp(f-1)]\right),$

becomes tight for $f=\mathrm{PMI}_{XY}+1$. Oord et al. (2018) propose a
variational approximation loss which uses a batch of
$(x_{i},y_{i})_{i=1,\dotsc,n}$ samples from $P_{XY}$ to estimate

$I_{\text{NCE}}(f)=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}\log\frac{\exp
f(x_{i},y_{i})}{\frac{1}{n}\sum_{j=1}^{n}\exp f(x_{i},y_{j})}\right]$

Here, if $f(x,y)=\mathrm{PMI}_{XY}(x,y)+c(x),$ where $c$ is any function, then
$I_{\text{NCE}}(f)\to\mathbf{I}\\!\left(X;Y\right)$ as $n\to\infty$.

In practice, only a finite sample is available, so expectation values are only
approximate (and any particular finite-sample estimate does not need to be a
lower bound on mutual information) and critic $f$ is modeled via some
parametric family, for example, a neural network, which has to be learned from
the data available. It is therefore of interest to understand the finite-
sample behavior of neural critics and variational approximations.

Do learned critics capture PMI well? We simulated $N=5\,000$ data points from
a mixture of four bivariate normal distributions (Fig. 4) with
$\mathbf{I}\\!\left(X;Y\right)=0.36$ and fitted the neural critic (see
Appendix C) to half of the data retaining the latter half as the test data
set, on which the final loss value was evaluated, yielding
$I_{\text{NWJ}}=0.33$, $I_{\text{DV}}=0.32$, $I_{\text{NCE}}=0.35$, which have
about $10\%$ error compared to the ground-truth. This difference can be
attributed to the critic being misspecified or the estimation error due to
evaluation on a finite batch. In Fig. 4 we plot the critic and the PMI. As the
Donsker–Varadhan estimator estimates PMI only up to an additive constant, we
normalized the critic and the PMI plots to have zero mean. Analogously, we
removed the functional degree of freedom $c(x)$ from the InfoNCE estimator by
removing the mean calculated along the $y$ dimension. Overall we saw a
mismatch, suggesting that neural critics do not capture the PMI function of
the distribution well. However, we can compare the PMI profile with the
histogram of the values predicted by the critic in Fig. 4 (shifting the PMI
profile and the histogram to have mean 0 for the Donsker–Varadhan estimator).
For InfoNCE it is not possible to compare the PMI profile with the critic
values due to the functional degree of freedom $c(x)$. Generally, we see
little discrepancy between the (shifted) PMI profile and learned values. This
suggests that although the neural critics may not learn the PMI function
properly in regions with low density, the PMI profile can be still
approximated well.

How robust are the estimators to a misspecified critic? In the next experiment
(Fig. 5), we evaluated $I_{\text{DV}}$, $I_{\text{NWJ}}$,
$I_{\text{InfoNCE}}$, and the simple Monte Carlo estimator (MC) for increasing
sample sizes where we provided them with an oracle critic $f$. Using a perfect
critic, $f=\mathrm{PMI}_{XY}$, we saw that all estimators had similar
performance on this problem. When a constant bias was added,
$f=\mathrm{PMI}_{XY}+c$, the Monte Carlo and NWJ estimators became biased.
Finally, InfoNCE was the only unbiased estimator when a functional degree of
freedom was added, $f(x,y)=\mathrm{PMI}_{XY}(x,y)+\sin(x^{2})$.

In low dimensions, variational approximations do not provide worse mutual
information values than the Monte Carlo estimator, when the correct critic
function is used, and are more robust to misspecification due to additional
degrees of freedom. However, this additional robustness results in a bias in
higher dimensions. We simulated a data set using a multivariate normal
distribution with 25 strongly interacting components,
$\mathrm{corr}(X_{i},Y_{i})=0.8$, which results in
$\mathbf{I}\\!\left(X;Y\right)\approx 12.8$. Although the Monte Carlo
estimator and NWJ estimator (which are not robust to additional degrees of
freedom) provided estimates close to the ground-truth (with Monte Carlo
slightly outperforming NWJ in both bias and variance), Donsker–Varadhan
estimator resulted in a strong positive bias for small batch sizes and InfoNCE
had a negative bias with very low variance.

## 4 Model-based estimation

Figure 6: From left to right: sample from the true data distribution, data
distribution sampled using a single MCMC sample, posterior distribution of the
mutual information (black line denotes ground-truth value), and posterior
distribution of the PMI profiles (black curve denotes the ground-truth value).
Top row: well-specified model for the AI distribution. Bottom row:
misspecified model for the Galaxy distribution results in biased inferences
with miscalibrated posterior. Posterior predictive checking can be used to
diagnose unreliable estimates.

As the final application of fine distributions, we consider the problem of
model-based estimation of mutual information. Consider a parametric
statistical model $\\{P_{\theta}\mid\theta\in\Theta\\}$ for $(X,Y)$ such that
for every value of the parameter vector $\theta$ the mutual information
contained in $P_{\theta}$ is known. That is, we define
$\mathbf{I}(P_{\theta}):=\mathbf{I}\\!\left(X;Y\right)$, where $(X,Y)\sim
P_{\theta}$.

If the data are modeled as i.i.d. r.v. $(X_{i},Y_{i})\sim P_{\theta}$, one can
estimate $\theta$ and use $\mathbf{I}(P_{\theta})$ as an estimate of the
mutual information. For example, Brillinger (2004) interprets the CCA
estimator (Kay, 1992) as the estimate $\mathbf{I}(P_{\hat{\theta}})$, where
$\hat{\theta}$ is the maximum likelihood estimate in the model in which
$P_{\theta}$ is a multivariate normal distribution. Analogously, if $X_{i}$
and $Y_{i}$ are discrete r.v., one can model $\theta$ as a probability matrix
and assume $P_{\theta}$ to be the multinomial distribution. Brillinger (2004)
proposes to use the maximum likelihood estimate $\hat{\theta}$ also in this
case to provide the estimate $\mathbf{I}(P_{\hat{\theta}})$.

More generally, Hutter (2001) observes that if the Dirichlet prior is used for
$\theta$, the Bayesian posterior $P(\theta\mid
X_{1},Y_{1},\dotsc,X_{N},Y_{N})$ will also be a Dirichlet distribution and
provides an approximation via moments to the posterior distribution of mutual
information.

We note that one can construct a Bayesian alternative for the CCA estimator by
using a prior on the covariance matrix (Lewandowski et al., 2009; Legramanti
et al., 2020) and Markov chain Monte Carlo (Gelman et al., 2013, Ch. 11) to
provide samples $\theta_{1},\dotsc,\theta_{M}$ from the posterior
$P(\theta\mid X_{1},Y_{1},\dotsc,X_{N},Y_{N})$ and construct a sample-based
approximation to the posterior on mutual information,
$\mathbf{I}(P_{\theta_{1}}),\dotsc,\mathbf{I}(P_{\theta_{M}})$.

More generally, consider a statistical model
$\\{P_{\theta}\mid\theta\in\Theta\\}$ such that all $P_{\theta}$ are fine
distributions. One can again find maximum likelihood $\hat{\theta}$ or, if
prior $P(\theta)$ is used, apply Bayesian inference algorithms to construct a
sample $\theta_{1},\dotsc,\theta_{M}$ from the posterior. Although the exact
values for $\mathbf{I}(P_{\hat{\theta}})$ and
$\mathbf{I}(P_{\theta_{1}}),\dotsc,\mathbf{I}(P_{\theta_{M}})$ are not
available, they can be approximated as in Sec. 2. Since fine distributions
include mixture models and normalizing flows, this approach can be used as a
general technique for building model-based mutual information estimators,
where the generative model can be constructed using domain knowledge.
Moreover, this is the first Bayesian estimator of the PMI profile: as all
distributions $P_{\theta_{m}}$ are fine, one can construct $M$ histograms (or
CDFs) approximating the profile.

To illustrate this approach we implemented a sparse Gaussian mixture model
(see Appendix D) in NumPyro (Phan et al., 2019) and used the NUTS sampler
(Hoffman & Gelman, 2014) to obtain a Bayesian posterior conditioned on 500
data points from the AI distribution (see Fig. 6). We see that the posterior
is concentrated around the ground-truth mutual information value and the
ground-truth PMI profile is well-approximated by the posterior samples.

However, this method relies on a model of the true data-generating
distribution. As observed in Sec. 3.1, the CCA estimator can yield unreliable
estimates when the model is misspecified, that is, the true data-generating
process $P_{XY}$ does not belong to the assumed multivariate normal family
$P_{\theta}$ (Watson & Holmes, 2016). To illustrate the risks of using
misspecified models, we applied the same Gaussian mixture model to 500 points
from the Galaxy distribution (Fig. 6). We see that a data sample simulated
from the model looks substantially different from the observed data, meaning
that the model did not capture the distribution well. In particular, most of
the probability mass of the Bayesian posterior is far from the ground-truth
mutual information. Similarly, the posterior on the PMI profile is biased. We
therefore recommend using model-checking techniques such as posterior
predictive checks (Gelman et al., 2013, Ch. 6) and discriminator-based
validation (Sankaran & Holmes, 2023, Sec. 4) to understand the deficiencies of
the model. To mitigate the risk of overfitting, which can bias the model (see
Appendix D), we recommend cross-validation (Piironen & Vehtari, 2017).

## 5 Conclusion

In this article, we have studied pointwise mutual information profiles,
determining them analytically for multivariate normal distributions (Theorem
6), and proposed the family of fine distributions, which include multivariate
normal and Student distributions, mixture models and normalizing flows, for
which the profile can be approximated using Monte Carlo methods. We showed how
fine distributions can be used to provide novel benchmark tasks to test mutual
information estimators and calculate mutual information transmitted through a
communication channel in the presence of inliers and outliers (which can be
used in the experimental design in electrical and biological sciences). Fine
distributions allowed us to study how well neural critics approximate
pointwise mutual information and we conclude that the error in low-dimensional
settings can originate from neural critic misspecification, rather than the
formula used to provide a variational approximation. Finally, we showed how
Bayesian estimates of mutual information between continuous r.v. can be
performed using fine distributions. Additionally, the proposed method provides
estimates of the pointwise mutual information profiles, which can be more
accurate than the ones obtained using neural critics and variational lower
bounds. Although this approach is not universal, we find it suitable for
problems with precise domain knowledge available (which can be used to
construct the generative distribution $P_{\theta}$ and provide the prior
$P(\theta)$) and in which uncertainty quantification is desired.

##### Limitations and Further Research

In Sec. 3.2, we study a channel in which the probability of failure $\alpha$
is constant and independent of the input variable $X$. However, in both
biological and electric systems this assumption may be broken: for large
values of $x$ the system may result in an outlier more easily. However, if
$\alpha$ depends on $x$, it is not straightforward to use fine distributions
to model these situations. Although we propose fine distributions as a basis
for model-based Bayesian inference of mutual information, model
misspecification (Watson & Holmes, 2016) as well as underfitting and
overfitting can bias inference (Piironen & Vehtari, 2017). We therefore
recommend following Gelman et al. (2020) to carefully validate the model.
Overfitting and model misspecification may be more difficult to detect
especially in high-dimensional situations or when more expressive models are
constructed, for example using normalizing flows, as Bayesian inference for
models involving neural networks is known to be challenging (Izmailov et al.,
2021). Finally, fine distributions could be used to study sample size
requirements and optimality of different estimation techniques under different
distributional assumptions similarly as in Czyż et al. (2023). However, in
this work we do not construct a formal benchmark (cf. Sec. 3.1), leaving it
for future work. In particular, fine distributions provide a general framework
for problems involving both continuous and discrete r.v. and it would be
interesting to consider computational examples using other manifolds, for
example by using computational geometry frameworks such as geomstats (Miolane
et al., 2020).

#### Acknowledgments

FG was supported by the Norwegian Financial Mechanism GRIEG-1 grant operated
by Narodowe Centrum Nauki (National Science Centre, Poland)
2019/34/H/NZ6/00699. PC and AM were supported by a fellowship from the ETH AI
Center. We would like to thank Paweł Nałȩcz-Jawecki and Julia Kostin for
helpful suggestions on the manuscript.

## References

  * Allen & Hospedales (2019) Carl Allen and Timothy Hospedales.  Analogies explained: Towards understanding word embeddings.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 223–231. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/allen19a.html.
  * Arellano-Valle et al. (2013) R.B. Arellano-Valle, J.E. Contreras-Reyes, and M.G. Genton.  Shannon entropy and mutual information for multivariate skew-elliptical distributions.  _Scandinavian Journal of Statistics_ , 49:42–62, 2013.
  * Belghazi et al. (2018) Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm.  Mutual information neural estimation.  In _International conference on machine learning_ , pp. 531–540. PMLR, 2018.
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brillinger (2004) David R. Brillinger.  Some data analyses using mutual information.  _Brazilian Journal of Probability and Statistics_ , 18(2):163–182, 2004.  ISSN 01030752, 23176199.  URL http://www.jstor.org/stable/43601047.
  * Carrara & Ernst (2023) Nick Carrara and Jesse Ernst.  Using Monte Carlo tree search to calculate mutual information in high dimensions, 2023.
  * Cellucci et al. (2005) Christopher J Cellucci, Alfonso M Albano, and Paul E Rapp.  Statistical validation of mutual information calculations: Comparison of alternative numerical algorithms.  _Physical review E_ , 71(6):066208, 2005.
  * Czyż et al. (2023) Paweł Czyż, Frederic Grabowski, Julia E. Vogt, Niko Beerenwinkel, and Alexander Marx.  Beyond normal: On the evaluation of mutual information estimators.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Darbellay & Vajda (1999) Georges A Darbellay and Igor Vajda.  Estimation of the information by an adaptive partitioning of the observation space.  _IEEE Transactions on Information Theory_ , 45(4):1315–1321, 1999.
  * Dillon et al. (2017) Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matthew D. Hoffman, and Rif A. Saurous.  Tensorflow distributions.  _CoRR_ , abs/1711.10604, 2017.  URL http://arxiv.org/abs/1711.10604.
  * Flegal et al. (2008) James M. Flegal, Murali Haran, and Galin L. Jones.  Markov Chain Monte Carlo: Can we trust the third significant figure?  _Statistical Science_ , 23(2):250 – 260, 2008\.  doi: 10.1214/08-STS257.  URL https://doi.org/10.1214/08-STS257.
  * Frühwirth-Schnatter & Malsiner-Walli (2019) Sylvia Frühwirth-Schnatter and Gertraud Malsiner-Walli.  From here to infinity: sparse finite versus Dirichlet process mixtures in model-based clustering.  _Advances in Data Analysis and Classification_ , 13(1):33–64, Mar 2019.  ISSN 1862-5355.  doi: 10.1007/s11634-018-0329-y.  URL https://doi.org/10.1007/s11634-018-0329-y.
  * Gao et al. (2017) Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath.  Estimating mutual information for discrete-continuous mixtures.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ef72d53990bc4805684c9b61fa64a102-Paper.pdf.
  * Gelman et al. (2013) A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin.  _Bayesian Data Analysis, Third Edition_.  Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013\.  ISBN 9781439840955.  URL https://books.google.pl/books?id=ZXL6AQAAQBAJ.
  * Gelman et al. (2020) Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák.  Bayesian workflow, 2020.
  * Grabowski et al. (2019) Frederic Grabowski, Paweł Czyż, Marek Kochańczyk, and Tomasz Lipniacki.  Limits to the rate of information transmission through the MAPK pathway.  _Journal of The Royal Society Interface_ , 16(152):20180792, 2019.  doi: 10.1098/rsif.2018.0792.  URL https://royalsocietypublishing.org/doi/abs/10.1098/rsif.2018.0792.
  * Haussler & Opper (1997) David Haussler and Manfred Opper.  Mutual information, metric entropy and cumulative relative entropy risk.  _The Annals of Statistics_ , 25(6):2451 – 2492, 1997.  doi: 10.1214/aos/1030741081.  URL https://doi.org/10.1214/aos/1030741081.
  * Hoffman & Gelman (2014) Matthew D. Hoffman and Andrew Gelman.  The No-U-Turn Sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo.  _Journal of Machine Learning Research_ , 15(47):1593–1623, 2014.  URL http://jmlr.org/papers/v15/hoffman14a.html.
  * Hutter (2001) Marcus Hutter.  Distribution of mutual information.  In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), _Advances in Neural Information Processing Systems_ , volume 14. MIT Press, 2001.  URL https://proceedings.neurips.cc/paper_files/paper/2001/file/fb2e203234df6dee15934e448ee88971-Paper.pdf.
  * Imhof (1961) J. P. Imhof.  Computing the distribution of quadratic forms in normal variables.  _Biometrika_ , 48(3/4):419–426, 1961.  ISSN 00063444.  URL http://www.jstor.org/stable/2332763.
  * Izmailov et al. (2021) Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson.  What are Bayesian neural network posteriors really like?  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 4629–4640. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/izmailov21a.html.
  * Jendoubi & Strimmer (2019) Takoua Jendoubi and Korbinian Strimmer.  A whitening approach to probabilistic canonical correlation analysis for omics data integration.  _BMC Bioinformatics_ , 20(1):15, Jan 2019.  ISSN 1471-2105.  doi: 10.1186/s12859-018-2572-9.  URL https://doi.org/10.1186/s12859-018-2572-9.
  * Kay (1992) J. Kay.  Feature discovery under contextual supervision using mutual information.  In _IJCNN International Joint Conference on Neural Networks_ , volume 4, pp. 79–84, 1992.  doi: 10.1109/IJCNN.1992.227286.
  * Kobyzev et al. (2021) I. Kobyzev, S. D. Prince, and M. A. Brubaker.  Normalizing flows: An introduction and review of current methods.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_, 43(11):3964–3979, nov 2021.  ISSN 1939-3539.  doi: 10.1109/TPAMI.2020.2992934.
  * Kolchinsky & Tracey (2017) Artemy Kolchinsky and Brendan D. Tracey.  Estimating mixture entropy with pairwise distances.  _Entropy_ , 19(7), 2017.  ISSN 1099-4300.  doi: 10.3390/e19070361.  URL https://www.mdpi.com/1099-4300/19/7/361.
  * Kraskov et al. (2004) Alexander Kraskov, Harald Stögbauer, and Peter Grassberger.  Estimating mutual information.  _Physical Review E_ , 69(6):066138, 2004.
  * Lee (2012) J.M. Lee.  _Introduction to Smooth Manifolds_.  Graduate Texts in Mathematics. Springer, 2nd edition, 2012.  ISBN 9781441999825.  URL https://doi.org/10.1007/978-1-4419-9982-5.
  * Legramanti et al. (2020) Sirio Legramanti, Daniele Durante, and David B Dunson.  Bayesian cumulative shrinkage for infinite factorizations.  _Biometrika_ , 107(3):745–752, 05 2020.  ISSN 0006-3444.  doi: 10.1093/biomet/asaa008.  URL https://doi.org/10.1093/biomet/asaa008.
  * Lewandowski et al. (2009) Daniel Lewandowski, Dorota Kurowicka, and Harry Joe.  Generating random correlation matrices based on vines and extended onion method.  _Journal of Multivariate Analysis_ , 100(9):1989–2001, 2009.  ISSN 0047-259X.  doi: https://doi.org/10.1016/j.jmva.2009.04.008.  URL https://www.sciencedirect.com/science/article/pii/S0047259X09000876.
  * Marx et al. (2021) Alexander Marx, Lincen Yang, and Matthijs van Leeuwen.  Estimating conditional mutual information for discrete-continuous mixtures using multi-dimensional adaptive histograms.  In _Proceedings of the SIAM International Conference on Data Mining (SDM)_ , pp. 387–395, 2021.
  * Miolane et al. (2020) Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec.  Geomstats: A Python package for Riemannian geometry in machine learning.  _Journal of Machine Learning Research_ , 21(223):1–9, 2020.  URL http://jmlr.org/papers/v21/19-027.html.
  * Murphy (2023) Kevin P. Murphy.  _Probabilistic Machine Learning: Advanced Topics_.  MIT Press, 2023.  URL http://probml.github.io/book2.
  * Nałȩcz-Jawecki et al. (2023) Paweł Nałȩcz-Jawecki, Paolo Armando Gagliardi, Marek Kochańczyk, Coralie Dessauges, Olivier Pertz, and Tomasz Lipniacki.  The MAPK/ERK channel capacity exceeds 6 bit/hour.  _PLOS Computational Biology_ , 19(5):1–21, 05 2023.  doi: 10.1371/journal.pcbi.1011155.  URL https://doi.org/10.1371/journal.pcbi.1011155.
  * Nguyen et al. (2007) XuanLong Nguyen, Martin J Wainwright, and Michael Jordan.  Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization.  In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), _Advances in Neural Information Processing Systems_ , volume 20. Curran Associates, Inc., 2007.  URL https://proceedings.neurips.cc/paper_files/paper/2007/file/72da7fd6d1302c0a159f6436d01e9eb0-Paper.pdf.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Papamakarios et al. (2021) George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _Journal of Machine Learning Research_ , 22(57):1–64, 2021.  URL http://jmlr.org/papers/v22/19-1028.html.
  * Petersen & Pedersen (2012) K. B. Petersen and M. S. Pedersen.  The matrix cookbook, November 2012.  URL http://www2.compute.dtu.dk/pubdb/pubs/3274-full.html.  Version 20121115.
  * Phan et al. (2019) Du Phan, Neeraj Pradhan, and Martin Jankowiak.  Composable effects for flexible and accelerated probabilistic programming in numpyro.  _arXiv preprint arXiv:1912.11554_ , 2019.
  * Piironen & Vehtari (2017) Juho Piironen and Aki Vehtari.  Comparison of Bayesian predictive methods for model selection.  _Statistics and Computing_ , 27(3):711–735, May 2017.  ISSN 1573-1375.  doi: 10.1007/s11222-016-9649-y.  URL https://doi.org/10.1007/s11222-016-9649-y.
  * Pinsker & Feinstein (1964) M.S. Pinsker and A. Feinstein.  _Information and Information Stability of Random Variables and Processes_.  Holden-Day series in time series analysis. Holden-Day, 1964.  ISBN 9780816268047.
  * Politis (1991) Dimitris N Politis.  _On the entropy of a mixture distribution_.  Purdue University. Department of Statistics, 1991.
  * Polyanskiy & Wu (2022) Y. Polyanskiy and Y. Wu.  _Information Theory: From Coding to Learning_.  Cambridge University Press, 2022.  Book draft.
  * Popoviciu (1935) Tiberiu Popoviciu.  Sur les équations algébriques ayant toutes leurs racines réelles.  _Mathematica (Cluj)_ , 9:129–145, 1935.
  * Sankaran & Holmes (2023) Kris Sankaran and Susan P. Holmes.  Generative models: An interdisciplinary perspective.  _Annual Review of Statistics and Its Application_ , 10(1):325–352, 2023.  doi: 10.1146/annurev-statistics-033121-110134.  URL https://doi.org/10.1146/annurev-statistics-033121-110134.
  * Watson & Holmes (2016) James Watson and Chris Holmes.  Approximate Models and Robust Decisions.  _Statistical Science_ , 31(4):465 – 489, 2016\.  doi: 10.1214/16-STS592.  URL https://doi.org/10.1214/16-STS592.

APPENDIX

##  Appendix A Technical results

###  A.1 Proof of the invariance of the pointwise mutual information profile

Recall a well-known result (Kraskov et al., 2004, Appendix):

######  Lemma 12 (Invariance of PMI).

Let $X^{\prime}=f(X)$ and $Y^{\prime}=g(Y)$, where $f$ and $g$ are
diffeomorphisms. Then for every $x^{\prime}$ and $y^{\prime}$ we have

$\mathrm{PMI}_{X^{\prime}Y^{\prime}}(x^{\prime},y^{\prime})=\mathrm{PMI}_{XY}(x,y),$

where $x=f^{-1}(x^{\prime})$ and $y=g^{-1}(y^{\prime})$.

###### Proof.

From

$p_{X^{\prime}Y^{\prime}}(x^{\prime},y^{\prime})=p_{XY}(x,y)\left|\det
D\left(f^{-1}\times g^{-1}\right)(x^{\prime},y^{\prime})\right|$

and analogous quantities we conclude that $p_{X^{\prime}Y^{\prime}}$ as well
as $p_{X^{\prime}}$ and $p_{Y^{\prime}}$ are smooth and everywhere positive
functions, so that $\mathrm{PMI}_{X^{\prime}Y^{\prime}}$ is well-defined. As
$D(f^{-1}\times g^{-1})(x^{\prime},y^{\prime})$ is a block matrix with
$Df^{-1}(x^{\prime})$ and $Dg^{-1}(y^{\prime})$ blocks on the diagonal and
other blocks zero, we have $\det D(f^{-1}\times
g^{-1})(x^{\prime},y^{\prime})=\det Df^{-1}(x^{\prime})\cdot\det
Dg^{-1}(y^{\prime})$. ∎

Now we can prove: See 3

###### Proof.

From the proof of Lemma 12 we conclude that
$P_{X^{\prime}Y^{\prime}}\in\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$.
Then, we note the profile is the pushforward measure

$\mathrm{Prof}_{XY}:=(\mathrm{PMI}_{XY})_{\\#}P_{XY}.$

Now let $B\subseteq\mathbb{R}$ be any set in the Borel $\sigma$-algebra and
$\mathbf{1}_{B}$ be its characteristic function. Using the change of variables
formula for pushforward measure and invariance of PMI:

$\displaystyle\mathrm{Prof}_{X^{\prime}Y^{\prime}}(B)$
$\displaystyle=\int\mathbf{1}_{B}(t)\,\mathrm{d}((\mathrm{PMI}_{X^{\prime}Y^{\prime}})_{\\#}P_{X^{\prime}Y^{\prime}})(t)$
$\displaystyle=\int\mathbf{1}_{B}\left(\mathrm{PMI}_{X^{\prime}Y^{\prime}}(x^{\prime},y^{\prime})\right)\,\mathrm{d}P_{X^{\prime}Y^{\prime}}(x^{\prime},y^{\prime})$
$\displaystyle=\int\mathbf{1}_{B}\left(\mathrm{PMI}_{X^{\prime}Y^{\prime}}(f(x),g(y))\right)\,\mathrm{d}P_{XY}(x,y)$
$\displaystyle=\int\mathbf{1}_{B}\left(\mathrm{PMI}_{XY}(x,y)\right)\,\mathrm{d}P_{XY}(x,y)$
$\displaystyle=\mathrm{Prof}_{XY}(B)$

∎

###  A.2 Pointwise Mutual Information Profiles

The following result shows that the distributions in all
$\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ classes with zero
mutual information have the same profile:

See 4

###### Proof.

If $\mathrm{Prof}_{XY}=\delta_{0}$, then the expected value is
$\mathbf{I}\\!\left(X;Y\right)=0$. To prove the converse, if
$\mathbf{I}\\!\left(X;Y\right)=0$, then $X$ and $Y$ are independent. Hence,
$p_{XY}(x,y)=p_{X}(x)p_{Y}(y)$ at every point $(x,y)$ and
$\mathrm{PMI}_{XY}(x,y)=0$ everywhere. ∎

The following result characterizes the PMI profiles for discrete r.v.: See 5

###### Proof.

The measure $P_{XY}$ is discrete and given by

$P_{XY}=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}p_{XY}(x,y)\delta_{(x,y)},$

so its pushforward by the $\mathrm{PMI}_{XY}$ function has the form

$\mathrm{Prof}_{XY}=(\mathrm{PMI}_{XY})_{\\#}P_{XY}=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}p_{XY}(x,y)\delta_{\mathrm{PMI}_{XY}(x,y)}.$

∎

See 6

###### Proof.

Without loss of generality assume that $m\leq n$. As the PMI profile is
invariant to diffeomorphisms (Theorem 3), we can also assume that variables
$X$ and $Y$ have been whitened by applying canonical correlation analysis
(Jendoubi & Strimmer, 2019), that is $\mathbb{E}[X]=0$, $\mathbb{E}[Y]=0$ and
the covariance matrix is given by

$\Sigma=\begin{pmatrix}I_{m}&\Sigma_{XY}\\\
\Sigma_{XY}^{T}&I_{n}\end{pmatrix}=\begin{pmatrix}I_{m}&R&0\\\ R&I_{m}&0\\\
0&0&I_{n-m}\end{pmatrix}$

where

$\Sigma_{XY}=\begin{pmatrix}R&0_{m\times(n-m)}\end{pmatrix}$

is an $m\times n$ matrix with the last $n-m$ columns being zero vectors and
$R=\mathrm{diag}(\rho_{1},\dotsc,\rho_{m})$ being the $m\times m$ diagonal
matrix representing canonical correlations.

We will write the inverse in the block form

$\Sigma^{-1}=\begin{pmatrix}\Lambda_{X}&\Lambda_{XY}\\\
\Lambda_{XY}^{T}&\Lambda_{Y}\end{pmatrix}=\begin{pmatrix}\Lambda_{X}&\tilde{R}&0\\\
\tilde{R}&\Lambda_{X}&0\\\ 0&0&I_{n-m}\end{pmatrix}$

where the blocks have been calculated using the formula from Petersen &
Pedersen (2012, Sec. 9.1):

$\displaystyle\Lambda_{X}$
$\displaystyle=(I_{m}-\Sigma_{XY}\Sigma_{XY}^{T})^{-1}=\mathrm{diag}\left(u_{1},\dotsc,u_{m}\right)$
$\displaystyle\Lambda_{Y}$
$\displaystyle=(I_{n}-\Sigma_{XY}^{T}\Sigma_{XY})=\mathrm{diag}\left(u_{1},\dotsc,u_{m},1,\dotsc,1\right)$
$\displaystyle\Lambda_{XY}$
$\displaystyle=-\Sigma_{XY}\Lambda_{Y}=\begin{pmatrix}\tilde{R}&0_{m\times(n-m)}\end{pmatrix},$

where
$\tilde{R}=-\mathrm{diag}\left(u_{1}\rho_{1},\dotsc,u_{m}\rho_{m}\right)$ and
$u_{i}=1/\left(1-\rho_{i}^{2}\right)$.

We define a quadratic form

$\displaystyle s(x,y)$
$\displaystyle=x^{T}\Lambda_{X}x+y^{T}\Lambda_{Y}y+2x^{T}\Lambda_{XY}y$
$\displaystyle=\sum_{i=1}^{m}u_{i}\left(x_{i}^{2}+y_{i}^{2}-2\rho_{i}x_{i}y_{i}\right)+\sum_{j=m+1}^{n}y_{j}^{2}$

which can be used to calculate log-PDFs:

$\displaystyle\log p_{XY}(x,y)$
$\displaystyle=-\frac{1}{2}s(x,y)-\frac{1}{2}\log\det\Sigma-\frac{m+n}{2}\log
2\pi,$ $\displaystyle\log p_{X}(x)$
$\displaystyle=-\frac{1}{2}x^{T}x-\frac{m}{2}\log 2\pi,$ $\displaystyle\log
p_{Y}(y)$ $\displaystyle=-\frac{1}{2}y^{T}y-\frac{n}{2}\log 2\pi.$

Hence,

$\mathrm{PMI}_{XY}(x,y)=\frac{x^{T}x+y^{T}y-s(x,y)}{2}-\frac{1}{2}\log\det\Sigma.$

We recognize the last summand as

$\mathbf{I}\\!\left(X;Y\right)=\frac{1}{2}\log\left(\frac{\det I_{m}\cdot\det
I_{n}}{\det\Sigma}\right)=-\frac{1}{2}\log\det\Sigma.$

Define quadratic form

$\displaystyle q(x,y)$
$\displaystyle=2\big{(}\mathrm{PMI}_{XY}(x,y)-\mathbf{I}\\!\left(X;Y\right)\big{)}=x^{T}x+y^{T}y-s(x,y)$
$\displaystyle=\sum_{i=1}^{m}\bigg{(}(1-u_{i})\left(x_{i}^{2}+y_{i}^{2}\right)+2\rho_{i}u_{i}x_{i}y_{i}\bigg{)},$

which has a corresponding matrix

$Q=\begin{pmatrix}K&F&0\\\ F&K&0\\\ 0&0&0\end{pmatrix},$

where

$K=\mathrm{diag}(1-u_{1},\dotsc,1-u_{m})$

and

$F=\mathrm{diag}(\rho_{1}u_{1},\dotsc,\rho_{m}u_{m}).$

We are interested in the distribution of

$q(X,Y)=\begin{pmatrix}X^{T}&Y^{T}\end{pmatrix}Q\begin{pmatrix}X\\\
Y\end{pmatrix},$

where $(X,Y)\sim\mathcal{N}(0,\Sigma)$.

Imhof (1961) presents a general approach to evaluating the distributions of
such quadratic forms. Consider a r.v.

$Z=\begin{pmatrix}\eta\\\ \epsilon\\\
\xi\end{pmatrix}\sim\mathcal{N}(0,I_{m+n})$

which is split into blocks of sizes $m$, $m$ and $n-m$. We will construct a
linear transformation $A$ such that

$\begin{pmatrix}X\\\ Y\end{pmatrix}=A\begin{pmatrix}\eta\\\ \epsilon\\\
\xi\end{pmatrix}.$

Then, the distribution of $q(X,Y)$ is the distribution of

$Z^{T}A^{T}QAZ,\qquad Z\sim\mathcal{N}(0,I_{m+n}).$

We will construct $A$ as

$A=\begin{pmatrix}P_{-}&P_{+}&0\\\ -P_{-}&P_{+}&0\\\ 0&0&I_{n-m}\end{pmatrix}$

where

$\displaystyle
P_{-}=\mathrm{diag}\left(\sqrt{\frac{1-\rho_{1}}{2}},\cdots,\sqrt{\frac{1-\rho_{m}}{2}}\right),\qquad
P_{+}=\mathrm{diag}\left(\sqrt{\frac{1+\rho_{1}}{2}},\cdots,\sqrt{\frac{1+\rho_{m}}{2}}\right).$

We calculate

$AA^{T}=\begin{pmatrix}P_{-}^{2}+P_{+}^{2}&P_{+}^{2}-P_{-}^{2}&0\\\
P_{+}^{2}-P_{-}^{2}&P_{-}^{2}+P_{+}^{2}&0\\\
0&0&I_{n-m}\end{pmatrix}=\begin{pmatrix}I_{m}&R&0\\\ R&I_{m}&0\\\
0&0&I_{n-m}\end{pmatrix}=\Sigma$

and

$A^{T}QA=\begin{pmatrix}2P_{-}^{2}(K-F)&0&0\\\ 0&2P_{+}^{2}(K+F)&0\\\
0&0&0\end{pmatrix},$

where

$2P_{-}^{2}(K-F)=\mathrm{diag}\left(-\rho_{1},\dotsc,-\rho_{m}\right),\qquad
2P_{+}^{2}(K+F)=\mathrm{diag}\left(\rho_{1},\dotsc,\rho_{m}\right).$

Hence, the distribution of $q(X,Y)$ is the same as the distribution of

$\sum_{i=1}^{m}\rho_{i}(-\eta_{i}^{2}+\epsilon_{i}^{2})+\sum_{j=1}^{n-m}0\cdot\xi_{j}^{2},$

where $(\eta_{,}\epsilon,\xi)\sim\mathcal{N}(0,I_{m+n})$. To summarize, let
$Q_{1},\dotsc,Q_{m},Q^{\prime}_{1},\dotsc,Q^{\prime}_{m}$ be i.i.d. random
variables distributed according to the $\chi^{2}_{1}$ distribution. The
quadratic form $q(X,Y)$ has the distribution the same as

$\sum_{i=1}^{m}\rho_{i}(Q_{i}-Q^{\prime}_{i}),$

which can also be written as

$q(X,Y)\sim\sum_{i=1}^{m}\left(\rho_{i}\chi^{2}_{1}-\rho_{i}\chi^{2}_{1}\right).$

Note that this distribution is symmetric around $0$. We can now reconstruct
the profile from $q(X,Y)$:

$\mathrm{Prof}_{XY}=\mathbf{I}\\!\left(X;Y\right)+\sum_{i=1}^{m}\left(\frac{\rho_{i}}{2}\chi^{2}_{1}-\frac{\rho_{i}}{2}\chi^{2}_{1}\right),$

which is symmetric around $\mathbf{I}\\!\left(X;Y\right)$ and, in agreement
with Proposition 4, degenerates to the atomic distribution $\delta_{0}$ if and
only if $\mathbf{I}\\!\left(X;Y\right)=0$, which is equivalent to $\rho_{i}=0$
for all $i$.

As a linear combination of independent $\chi^{2}_{1}$ variables, profile has
all finite moments. Using the fact that the variance of $\chi^{2}_{1}$
distribution is $2$, and quadratic scaling of variance, each term has variance
$2\cdot(\rho_{i}/2)^{2}=\rho_{i}^{2}/2$. As variances of independent variables
are additive, we can sum up all the $2m$ terms to obtain
$\rho_{1}^{2}+\cdots+\rho_{m}^{2}$. ∎

######  Proposition 13.

With fixed mutual information, the variance of the PMI profile is maximized
when $\rho_{1}^{2}=\rho_{2}^{2}=\dotsc=\rho_{m}^{2}$.

###### Proof.

Let $a_{i}=1-\rho_{i}^{2}$. To maximize the variance we equivalently have to
minimize $a_{1}+\dotsc+a_{m}$ preserving given constraint on mutual
information and $a_{i}\in(0,1]$.

The constraint on mutual information takes the form

$\mathbf{I}\\!\left(X;Y\right)=-\frac{1}{2}\sum_{i=1}^{m}\log\left(1-\rho_{i}^{2}\right)=-\frac{1}{2}\log\left(a_{1}\cdots
a_{m}\right).$

Hence, the product $a_{1}\cdots a_{m}$ has to be constant. Denote this
constant by $A^{m}$ for $A\in(0,1]$ as well.

Let $a_{1},\dotsc,a_{m}$ be any minimum of $a_{1}+\dotsc+a_{m}$ under the
constraints $a_{1}\cdots a_{m}=A^{m}$ and $a_{i}\in(0,1]$. From the inequality
between arithmetic and geometric means we note that

$\frac{a_{1}+\dotsc+a_{m}}{m}\geq\sqrt[m]{a_{1}\cdots a_{m}}=A,$

where the equality holds only if $a_{1}=\dotsc=a_{m}=A$. Hence, this is the
unique minimum under constraints provided. It follows that
$\rho_{1}^{2}=\cdots=\rho_{m}^{2}$. Writing $\rho^{2}$ for the common value,
we have

$\rho^{2}=1-\exp\left(-2\mathbf{I}\\!\left(X;Y\right)/m\right)$

and

$V=m\rho^{2}=m\left(1-\exp\left(-2\mathbf{I}\\!\left(X;Y\right)/m\right)\right).$

The mutual information can also be written as function of variance

$\mathbf{I}\\!\left(X;Y\right)=-\frac{1}{2}m\log\left(1-\rho^{2}\right)=-\frac{1}{2}m\log(1-V/m).$

∎

######  Proposition 14.

With fixed non-zero mutual information the variance of the PMI profile is
minimized when $\rho_{i}\neq 0$ for exactly one $i$.

###### Proof.

Let $a_{i}\in(0,1]$ be any numbers. We have

$(1-a_{1})(1-a_{2})\geq 0$

which is equivalent to

$1+a_{1}a_{2}\geq a_{1}+a_{2}$

where the equality holds if and only if $a_{1}=1$ or $a_{2}=1$.

Using the principle of mathematical induction one can prove a more general
inequality:

$\displaystyle a_{1}a_{2}\cdots a_{m}+(m-1)$
$\displaystyle=1+a_{1}(a_{2}\cdots a_{m})+(m-2)$ $\displaystyle\geq
a_{1}+\big{(}a_{2}\cdots a_{m}+(m-2)\big{)}$ $\displaystyle\geq
a_{1}+a_{2}+\big{(}a_{3}\cdots a_{m}+(m-3)\big{)}$ $\displaystyle~{}\vdots$
$\displaystyle\geq a_{1}+a_{2}+\cdots+a_{m}.$

Let us analyze when the equality can hold. To obtain equality in the first
step, we need $a_{1}=1$ or $a_{2}\cdots=a_{m}=1$, which, given the constraints
$a_{i}\in(0,1]$ would mean that $a_{2}=\cdots a_{m}=1$. Reasoning inductively,
one proves that equality holds only when at least $m-1$ among these numbers
are $1$.

Let us apply the above reasoning to the numbers $a_{i}=1-\rho_{i}^{2}$ and
note that we are solving a maximization problem $a_{i}\in(0,1]$ under a
constraint

$a_{1}\cdots a_{m}=P.$

Using the argument above we note that all the maxima for the above problem are
permutations of the sequence $P,1,1,\dotsc,1$. This proves that at most one
$\rho_{i}^{2}\neq 0$, what it turns results in at most one $\rho_{i}\neq 0$. ∎

###  A.3 Proof of the failing channel inequality

In this section we prove:

See 11

###### Proof.

Let $Z\sim\mathrm{Bernoulli}(1-\alpha)$ be an auxiliary variable. We have
$(X,Y^{\prime})\mid Z=1\sim P_{XY}$ and $(X,Y^{\prime})\mid Z=0\sim
P_{X}\otimes N_{Y}$.

From the data processing inequality and chain rule we conclude that

$\mathbf{I}\\!\left(X;Y^{\prime}\right)\leq\mathbf{I}\\!\left(X;Y^{\prime},Z\right)=\mathbf{I}\\!\left(X;Z\right)+\mathbf{I}\\!\left(X;Y^{\prime}\mid
Z\right).$

Now note that $X$ and $Z$ are independent, so
$\mathbf{I}\\!\left(X;Z\right)=0$.

Hence,

$\displaystyle\mathbf{I}\\!\left(X;Y^{\prime}\right)$
$\displaystyle\leq\mathbf{I}\\!\left(X;Y^{\prime}\mid Z\right)$
$\displaystyle=\alpha\mathbf{D}_{\mathrm{KL}}\left(P_{XY^{\prime}\mid
Z=0}\parallel P_{X\mid Z=0}\otimes P_{Y^{\prime}\mid
Z=0}\right)+(1-\alpha)\mathbf{D}_{\mathrm{KL}}\left(P_{XY^{\prime}\mid
Z=1}\parallel P_{X\mid Z=1}\otimes P_{Y^{\prime}\mid Z=1}\right)$
$\displaystyle=\alpha\mathbf{D}_{\mathrm{KL}}\left(P_{XY^{\prime}\mid
Z=0}\parallel P_{X}\otimes
N_{Y}\right)+(1-\alpha)\mathbf{D}_{\mathrm{KL}}\left(P_{XY}\parallel
P_{X}\otimes P_{Y}\right)$
$\displaystyle=\alpha\mathbf{D}_{\mathrm{KL}}\left(P_{X}\otimes N_{Y}\parallel
P_{X}\otimes N_{Y}\right)+(1-\alpha)\mathbf{I}\\!\left(X;Y\right)$
$\displaystyle=(1-\alpha)\mathbf{I}\\!\left(X;Y\right).$

∎

###  A.4 Constructing new fine distributions

In this section we prove Proposition 8 and Proposition 9.

See 8

###### Proof.

From the proof of Lemma 12 and the assumption of tractability of the Jacobians
of $f$ and $g$ we obtain the tractability of the formulae for the densities
$p_{f(X)g(Y)}$, $p_{f(X)}$ and $p_{g(Y)}$. Sampling $(f(X),g(Y))$ amounts to
sampling $(X,Y)$ and then transforming the sample using $f$ and $g$. ∎

See 9

###### Proof.

We can evaluate the densities $p_{X^{\prime}Y^{\prime}}(x,y)$,
$p_{X^{\prime}}(x)$ and $p_{Y^{\prime}}(y)$ of the mixture distribution using
the weighted sums of $p_{X_{k}Y_{k}}(x,y)$, $p_{X_{k}}(x)$ and $p_{Y_{k}}(y)$,
respectively. To sample $(X^{\prime},Y^{\prime})\sim P_{X^{\prime}Y^{\prime}}$
from the mixture distribution we can sample an auxiliary variable
$Z\sim\mathrm{Categorical}(K;w_{1},\dotsc,w_{K})$. Then, we have
$\big{(}(X^{\prime},Y^{\prime})\mid Z=k\big{)}=(X_{k},Y_{k})$, so that we can
sample from $P_{X_{k}Y_{k}}$. ∎

###  A.5 Approximation with smooth densities

In this section, we prove the following result:

######  Theorem 15.

Let $X$ and $Y$ be random variables with joint distribution $P_{XY}$ and
finite mutual information $\mathbf{I}\\!\left(X;Y\right)<\infty$. Then, there
exists a sequence of random variables $(X_{k},Y_{k})$ for $k=1,2,\dotsc$ such
that:

  1. 1.

Each $P_{X_{k}Y_{k}}\in\mathcal{P}\\!\left({m,n}\right)$.

  2. 2.

Distributions $P_{X_{k}Y_{k}}$ weakly converge to $P_{XY}$ as $k\to\infty$.

  3. 3.

Mutual information is preserved:

$\lim_{k\to\infty}\mathbf{I}\\!\left(X_{k};Y_{k}\right)=\mathbf{I}\\!\left(X;Y\right).$

The proof will automatically follow from two lemmata.

######  Lemma 16.

Let $X$ be a random variable on $\mathbb{R}^{k}$. Let $N$ be a r.v.
independent of $X$ distributed according to the multivariate normal
distribution $\mathcal{N}(0,\sigma^{2}I)$ with $\sigma>0$. Then the
distribution of $X+N$ has a PDF which is everywhere smooth and positive.

###### Proof.

Let $P_{X}$ be the distribution of $X$ and $f$ be the PDF of the multivariate
normal distribution $\mathcal{N}(0,\sigma^{2}I)$, which is bounded, smooth,
and everywhere positive. The convolution of $P_{X}$ and normal density has a
PDF

$(f*P_{X})(x)=\int f(x-y)\,\mathrm{d}P_{X}(y)=\mathbb{E}[f(x-X)].$

From the dominated convergence theorem it follows that all the partial
derivatives of $\partial_{j}(f*P_{X})$ exist and are given by
$(\partial_{j}f)*P_{X}$. From the principle of mathematical induction we
conclude that $f*P_{X}$ is smooth.

Positivity follows from the fact that the random variable $f(x-X)$ is strictly
positive, so that its expected value is strictly positive as well. ∎

The second lemma is the following:

######  Lemma 17.

Assume that $I(X;Y)<\infty$. Let $X_{k}=X+N_{k}$ and $Y_{k}=Y+M_{k}$ where
$N_{k}$ and $M_{k}$ are independent noise variables distributed according to
$\mathcal{N}(0,k^{-2}I)$. Then

$\lim_{k\to\infty}\mathbf{I}\\!\left(X_{k};Y_{k}\right)=\mathbf{I}\\!\left(X;Y\right).$

###### Proof.

Observe that for every $k$ the equality
$\mathbf{I}\\!\left(X_{k};Y_{k}\right)\leq\mathbf{I}\\!\left(X;Y\right)$
holds. This is a simple corollary of the data processing inequality applied to
the variables $(X_{k},N_{k})$ and $(Y_{k},M_{k})$. Then, note that
$P_{X_{k}Y_{k}}$ converges weakly to $P_{XY}$. As mutual information is
assumed to be finite, it is given by the Kullback–Leibler divergence, which is
lower semicontinuous. This proves the reverse inequality. ∎

###  A.6 Creating and destroying information with mixtures

Let $A=(0,1)$ and $B=(1,2)$ be two disjoint intervals of unit length. We
define two pairs of random variables:

$\displaystyle(X_{1},Y_{1})\sim\mathrm{Uniform}(A\times
A),\qquad(X_{2},Y_{2})\sim\mathrm{Uniform}(B\times B).$

Note that

$\mathbf{I}\\!\left(X_{1};Y_{1}\right)=\mathbf{I}\\!\left(X_{2};Y_{2}\right)=0.$

If $(X,Y)\sim 0.5P_{X_{1}Y_{1}}+0.5P_{X_{2}Y_{2}}$ is distributed according to
a mixture, we have

$p_{XY}(x,y)=\frac{1}{2}\mathbf{1}[(x,y)\in A\times A\cup B\times B]$

and

$p_{X}(x)=\frac{1}{2}\mathbf{1}[x\in A\cup B],\qquad
p_{Y}(y)=\frac{1}{2}\mathbf{1}[x\in A\cup B].$

Hence,

$\mathrm{PMI}_{XY}(x,y)=\log 2\cdot\mathbf{1}[(x,y)\in A\times A\cup B\times
B]$

and

$\mathbf{I}\\!\left(X;Y\right)=\frac{1}{2}\log 2+\frac{1}{2}\log 2=\log 2.$

Figure 7: Top row: two distributions with zero MI can be mixed to obtain a
distribution with non-zero MI. Bottom row: two distributions with non-zero MI
can be mixed to obtain a distribution with zero MI.

For a second example, demonstrating vanishing mutual information, recall the
distribution constructed as above:

$p_{XY}(x,y)=\frac{1}{2}\mathbf{1}[(x,y)\in A\times A\cup B\times B]$

and a symmetric one

$p_{UV}(x,y)=\frac{1}{2}\mathbf{1}[(x,y)\in A\times B\cup B\times A].$

We have

$\mathbf{I}\\!\left(X;Y\right)=\mathbf{I}\\!\left(U;V\right)=\log 2.$

On the other hand, the mixture distribution

$(Z,T)\sim 0.5P_{XY}+0.5P_{UV}$

has zero mutual information, $\mathbf{I}\\!\left(Z;T\right)=0$, as

$\displaystyle p_{ZT}(x,y)$
$\displaystyle=\frac{1}{4}\mathbf{1}[(x,y)\in(A\cup B)\times(A\cup B)]$
$\displaystyle=\frac{1}{2}\mathbf{1}[x\in A\cup
B]\cdot\frac{1}{2}\mathbf{1}[y\in A\cup B]$ $\displaystyle=p_{Z}(x)\cdot
p_{T}(y).$

Note that similar examples can be constructed in the fine distribution family
by using mixtures of multivariate normal distributions.

This demonstrates that mixtures can create and destroy information. There is
however an upper bound on the amount of information mixtures can create (see
Haussler & Opper (1997) and Kolchinsky & Tracey (2017)): See 10

###### Proof.

The examples provide explicit constructions of distributions with the
specified properties. To prove the upper bound, consider a variable
$Z\sim\mathrm{Categorical}(K;w_{1},\dotsc,w_{K})$. The random variables
corresponding to the mixture distribution, $(X^{\prime},Y^{\prime})$, have
conditional distributions

$(X^{\prime},Y^{\prime})\mid Z=k\sim P_{X_{k}Y_{k}}.$

From the data processing inequality and chain rule we have

$\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\right)\leq\mathbf{I}\\!\left(X^{\prime};Y^{\prime},Z\right)=\mathbf{I}\\!\left(X^{\prime};Z\right)+\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\mid
Z\right).$

As $Z$ is discrete, the first summand,
$\mathbf{I}\\!\left(X^{\prime};Z\right)$, is bounded from above by the entropy
$H(Z)$ (Polyanskiy & Wu, 2022, Th. 3.4(e)), which cannot exceed $\log K$
(Polyanskiy & Wu, 2022, Th. 1.4(b)). The second summand can be written as

$\displaystyle\mathbf{I}\\!\left(X^{\prime};Y^{\prime}\mid Z\right)$
$\displaystyle=\sum_{k=1}^{K}P(Z=k)\,\mathbf{D}_{\mathrm{KL}}\left(P_{X^{\prime}Y^{\prime}\mid
Z=k}\parallel P_{X^{\prime}\mid Z=k}\otimes P_{Y^{\prime}\mid Z=k}\right)$
$\displaystyle=\sum_{k=1}^{K}w_{k}\,\mathbf{D}_{\mathrm{KL}}\left(P_{X_{k}Y_{k}}\parallel
P_{X_{k}}\otimes
P_{Y_{k}}\right)=\sum_{k=1}^{K}w_{k}\,\mathbf{I}\\!\left(X_{k};Y_{k}\right).$

∎

##  Appendix B Distributions involving discrete variables

The formalism in Section 2 is applicable to both continuous and discrete
random variables, although in Section 3.1 we focus on the distributions in
which both $X$ and $Y$ are continuous. If $X$ and $Y$ are discrete, mutual
information $\mathbf{I}\\!\left(X;Y\right)$ can be calculated analytically
from the joint probability matrix and there exist numerous approaches to
estimate it from collected samples (Hutter, 2001; Brillinger, 2004).

In this section we consider the mixed case, in which one variable is
continuous and the other one is discrete. For example, Carrara & Ernst (2023)
describe a particle physics experiment in which $X$ is an 18-dimensional
random variable, but $Y$ is binary. Grabowski et al. (2019) consider a cell
transmitting information through the MAPK signalling pathway, assuming the
input signal $X$ to be discrete and the measured response $Y$ to be
continuous.

Yet, there are only a few distributions $P_{XY}$ with known ground-truth
mutual information assuming this discrete-continuous case. Gao et al. (2017,
Sec. 5) describe a discrete random variable $X$ which is uniformly sampled
from the set $\mathcal{X}=\\{0,\dotsc,m-1\\}$ and the continuous $Y$ variable
is sampled as

$(Y\mid X=x)\sim\mathrm{Uniform}(x,x+2),$

which is therefore distributed on $\mathcal{Y}=(0,m+1)$. Gao et al. (2017)
prove that mutual information in this case is

$\mathbf{I}\\!\left(X;Y\right)=\log m-\frac{m}{m-1}\log 2$

for $m\geq 2$ and $\mathbf{I}\\!\left(X;Y\right)=0$ for $m=1$ and consider a
multivariate analogue of this distribution, in which pairs of variables
$(X_{k},Y_{k})$ for $k=1,\dotsc,K$ are sampled independently using the above
procedure. Then, they are concatenated into multivariate vectors
$(X_{1},\dotsc,X_{K})$ and $(Y_{1},\dotsc,Y_{K})$ with $K$ times larger mutual
information.

Figure 8:  Left: samples from the $P_{XY}$ distribution, colored by the value
of the binary variable $Y$. Right: the PMI profile of $P_{XY}$ distribution.

We will show how to relate the bivariate example to the framework of fine
(multivariate case can be constructed analogously). Note that the joint
distribution $P_{XY}$ is not strictly in
$\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$, as it is supported on
the one-dimensional manifold $\mathcal{M}\subset\mathcal{X}\times\mathcal{Y}$
with $m$ connected components (cf. Politis (1991) and Marx et al. (2021)), on
which

$p_{XY}(x,y)=\frac{1}{2m}\mathbf{1}[y\in(x,x+2)],$

but it is possible to extend the definitions of Section 2, so that this
technical difficulty is resolved. The marginal distributions are tractable and
admit PDFs:

$\displaystyle p_{X}(x)$ $\displaystyle=1/m,$ $\displaystyle p_{Y}(y)$
$\displaystyle=\sum_{x=0}^{m-1}p_{XY}(x,y).$

Although $p_{Y}$ is not smooth on $\mathcal{Y}$ (at integer points), this
technical difficulty can also be resolved as this set is of measure zero.
Hence, although this distribution is not strictly fine, it can still be
modelled using the introduced framework.

Next, Gao et al. (2017, Sec. 5) consider the zero-inflated Poissonization of
the exponential random variable valued in $\mathcal{X}=\\{x\in\mathbb{R}\mid
x\geq 0\\}$, with: $X\sim\mathrm{Exp}(1)$ and $Y$ being a discrete random
variable valued in the set of non-negative integers
$\mathcal{Y}=\\{0,1,2,\dotsc\\}$:

$(Y\mid X=x)\sim p\,\delta_{0}+(1-p)\,\mathrm{Poisson}(x).$

They show that

$\mathbf{I}\\!\left(X;Y\right)=(1-p)\left(2\log
2-\gamma-\sum_{k=0}^{\infty}\log k\cdot 2^{-k}\right),$

where $\gamma$ is the Euler–Mascheroni constant.

To use fine distributions in this case, one has to formally extend the
definition of $\mathcal{P}\\!\left({\mathcal{X},\mathcal{Y}}\right)$ as
$\mathcal{X}$ is a manifold with boundary (Lee, 2012, Ch. 1). The joint
probability distribution is then given by

$p_{XY}(x,y)=p_{X}(x)\cdot p_{Y\mid X}(y\mid
x)=e^{-x}\cdot\left(p\cdot\mathbf{1}[y=0]+(1-p)\frac{e^{-x}x^{y}}{y!}\right)$

and the PMF function of the $Y$ variable is also analytically known:

$p_{Y}(y)=p\cdot\mathbf{1}[y=0]+(1-p)\cdot 2^{-(1+y)}$

Hence, the framework of fine distributions, with minor modifications, can
accommodate the above distributions.

However, it also allows one to create more expressive distributions, for which
analytical formulae for ground-truth mutual information are not available, but
can be approximated with the Monte Carlo methods as explained in Section 2:
consider a continuous random variable $X$ and a discrete random variable $Y$.
To introduce a dependency between $X$ and $Y$ variable, we can use a mixture
of distributions in which the component variables are independent, i.e.,
$P_{X_{k}Y_{k}}=P_{X_{k}}\otimes P_{Y_{k}}$. Therefore, we consider a
graphical model $X\leftarrow Z\rightarrow Y$, in which the distributions
$P_{X_{k}}=P_{X\mid Z=k}$ are known and have tractable PDFs. The distributions
of $P_{Y_{k}}=P_{Y\mid Z_{k}}$ are given by probability tables. Monte Carlo
estimators of Section 2 can then be used to estimate
$\mathbf{I}\\!\left(X;Y\right)$ with high accuracy. Note also that this
general procedure includes the case $X\leftarrow Y$ by setting $Z=Y$.

We illustrate it in a simple example with $\mathcal{X}=\mathbb{R}^{2}$,
$\mathcal{Y}=\\{0,1\\}$ and $K=3$ components.

The first component will model a cluster in the $\mathcal{X}$ space, strongly
associated with $Y=1$ value. For $P_{X_{1}}$ we use a bivariate Student
distribution centered at $(1,1)$ with isotropic dispersion $\Omega=0.2\cdot
I_{2}$ and 8 degrees of freedom. We take $P_{Y_{1}}$ to be the Bernoulli
variable with probability $P(Y_{1}=1)=0.95$.

Analogously, we define a second cluster, strongly associated with $Y=0$ value:
$P_{X_{2}}$ is a bivariate Student distribution with the same dispersion
matrix, but centered at $(-1,-1)$ and with 8 degrees of freedom. Then, $Y_{2}$
is a Bernoulli variable with $P(Y_{2}=1)=0.05$.

We then define a third component using a bivariate normal distribution
centered at $(0,0)$ and with covariance matrix

$\Sigma=0.1\begin{pmatrix}1&0.95\\\ 0.95&1\end{pmatrix}.$

This component is not informative of $Y$, that is $P(Y_{3}=1)=0.5$.

We used weights $w_{1}=w_{2}=1/4$ and $w_{3}=1/2$, what resulted in the
distribution visualised in Fig. 8. We estimated both the profile and the
mutual information using $N=10^{6}$ samples and obtained
$\mathbf{I}\\!\left(X;Y\right)=0.224$ with MCSE of $5.1\cdot 10^{-4}$.

In principle, the above construction can be used to generate realistic high-
dimensional data sets (e.g., audio or image) with known ground-truth mutual
information, by assuming the generative model $Y\to X$ (i.e., $Z=Y$) and
modelling each $P_{X_{k}}$ using a normalizing flow or an autoregressive model
(Murphy, 2023, Ch. 22) trained on an auxiliary data set with fixed label
$Y_{k}=y_{k}$. Hence, at least in principle, one could obtain highly-
expressive generative process $p_{X\mid Y}(x\mid y)$ with tractable
probability and sampling. Pairing this with an arbitrary probability vector
$p_{Y}(y)$ one can obtain $p_{XY}(x,y)$ and $p_{X}(x)$ even for high-
dimensional data sets, so that Monte Carlo estimator can be used to determine
the ground-truth mutual information. However, we anticipate possible practical
difficulties with scaling up the proposed approach to high-dimensional data
and we leave empirical investigation of this topic to future work.

##  Appendix C Experimental details

###  C.1 New distributions

We constructed the X distribution as a mixture of bivariate normal
distributions with equal weights, zero mean and covariance matrices specified
by

$\Sigma_{\pm}=0.3\begin{pmatrix}1&\pm 0.9\\\ \pm 0.9&1\end{pmatrix}.$

Note that the marginal distributions of each of component distributions is
$\mathcal{N}(0,0.3^{2})$ and subsequently their mixture has exactly the same
marginal distributions. This is therefore an interesting example of a
distribution in which the joint probability distribution is not multivariate
normal, although the marginal distributions of $X$ and $Y$ variables are
normal individually.

The AI distribution was constructed as an equally-weighted mixture of six
bivariate normal distributions with equal weights and the following
parameters:

$\displaystyle\mu_{1}$ $\displaystyle=(1,0)$ $\displaystyle\Sigma_{1}$
$\displaystyle=\mathrm{diag}(0.01,0.2)$ $\displaystyle\mu_{2}$
$\displaystyle=(1,1)$ $\displaystyle\Sigma_{2}$
$\displaystyle=\mathrm{diag}(0.05,0.001)$ $\displaystyle\mu_{3}$
$\displaystyle=(1,-1)$ $\displaystyle\Sigma_{3}$
$\displaystyle=\mathrm{diag}(0.05,0.001)$ $\displaystyle\mu_{4}$
$\displaystyle=(-0.8,-0.2)$ $\displaystyle\Sigma_{4}$
$\displaystyle=\mathrm{diag}(0.03,0.001)$ $\displaystyle\mu_{5}$
$\displaystyle=(-1.2,0)$ $\displaystyle\Sigma_{5}$
$\displaystyle=\begin{pmatrix}0.04&0.085\\\ 0.085&0.2\end{pmatrix}$
$\displaystyle\mu_{6}$ $\displaystyle=(-0.4,0)$ $\displaystyle\Sigma_{6}$
$\displaystyle=\begin{pmatrix}0.04&-0.085\\\ -0.085&0.2\end{pmatrix}$

The Galaxy distribution was constructed as an equally-weighted mixture of
isotropic multivariate normal distributions with $\mu_{\pm}=\pm(1,1,1)$ and
unit covariance matrix and the $X$ variable was transformed using the spiral
diffeomorphism with $v=0.5$ (cf. Czyż et al. (2023)).

The Waves distribution was created as an equally-weighted mixture of 12
multivariate normal distributions with equal covariance matrices
$\Sigma=\mathrm{diag}(0.1,1,0.1)$ and mean vectors

$\mu_{i}=(x,0,x\,\mathrm{mod}\,4),\quad i\in\\{0,1,\dotsc,11\\}.$

This construction results in a distribution where different vertical
components of the $X$ variable are assigned $Y$ values calculated modulo 4.
Then, we transformed the $X$ variable with a continuous injection

$f(x_{1},x_{2})=(x_{1}+5\sin(3x_{2}),x_{2}),$

which does not change the mutual information. Finally, we applied the affine
mappings

$a_{1}(x)=0.1x-0.8,\quad a_{2}(y)=0.5y,$

to make the range of the typical values comparable with other distributions.

To estimate the ground-truth mutual information we used the Monte Carlo
approach described in Section 2 with $N=200\,000$ samples.

###  C.2 Estimator hyperparameters

Czyż et al. (2023, Appendix E.4) study the effects of hyperparameters on
mutual information estimators. We decided to use the histogram-based estimator
(Cellucci et al., 2005; Darbellay & Vajda, 1999) with a fixed number of 10
bins per dimension and the popular KSG estimator (Kraskov et al., 2004) with
$k=10$ neighbors. Canonical correlation analysis (Kay, 1992; Brillinger, 2004)
does not have any hyperparameters. Finally, we employed neural estimators
(InfoNCE (Oord et al., 2018) and MINE (Belghazi et al., 2018)) with the neural
critic being a ReLU network with 16 and 8 hidden neurons, as it obtained
competitive performance in the benchmark of Czyż et al. (2023, Appendix E.4).

As a preprocessing strategy, we followed Czyż et al. (2023, Appendix E.3) and
transformed all samples to have zero empirical mean and unit variance along
each dimension.

##  Appendix D Gaussian Mixture Models

Recall from Sec. 4 that for Bayesian estimation of mutual information consists
of the following steps:

  1. 1.

Propose a parametric generative model of the data,
$P_{\theta}:=P(X,Y\mid\theta)$, and assume a prior $P(\theta)$ on the
parameter space.

  2. 2.

Use a Markov chain Monte Carlo method to obtain a sample
$\theta^{(1)},\dotsc,\theta^{(m)}$ from the posterior $P(\theta\mid
X_{1},Y_{1},\dotsc,X_{N},Y_{N})$.

  3. 3.

Estimate mutual information (and the PMI profile) for each $\theta^{(m)}$
using the Monte Carlo method described in Sec. 2.

  4. 4.

Validate the findings using e.g., posterior predictive checks and cross-
validation.

We consider the following sparse Gaussian mixture model with $K=10$
components:

$\displaystyle\pi$
$\displaystyle\sim\mathrm{Dirichlet}(K;1/K,1/K,\dotsc,1/K),$ $\displaystyle
Z_{n}\mid\pi$ $\displaystyle\sim\mathrm{Categorical}(\pi),$ $\displaystyle
n=1,\dotsc,N,$ $\displaystyle\mu_{k}$
$\displaystyle\sim\mathcal{N}\left(0,3^{2}I_{D}\right),$ $\displaystyle
k=1,\dotsc,K,$ $\displaystyle\Sigma_{k}$
$\displaystyle\sim\mathrm{ScaledLKJ}(1,1),$ $\displaystyle k=1,\dotsc,K,$
$\displaystyle(X_{n},Y_{n})\mid Z_{n},\\{\mu_{k},\Sigma_{k}\\}$
$\displaystyle\sim\mathcal{N}(\mu_{Z_{n}},\Sigma_{Z_{n}}),$ $\displaystyle
n=1,\dotsc,N.$

Sampling a single covariance matrix $\Sigma$ from
$\mathrm{ScaledLKJ}(\sigma,\eta)$ distribution corresponds to sampling the
correlation matrix $R$ from the Lewandowski-Kurowicka-Joe (LKJ) distribution
(Lewandowski et al., 2009):

$p(R)\propto(\det R)^{\eta-1},$

sampling the scale parameters

$\lambda_{1},\lambda_{2},\dotsc,\lambda_{D}\sim\mathrm{HalfCauchy}(\mathrm{scale}{=}\sigma),$

and then constructing the covariance matrix as
$\Sigma_{ij}=R_{ij}\lambda_{i}\lambda_{j}$.

The sparse Dirichlet prior is a finite-dimensional alternative to the
Dirichlet process, which truncates the number of occupied clusters depending
on the data (Frühwirth-Schnatter & Malsiner-Walli, 2019). In particular, the
_a priori_ expected number of clusters depends on the number of data points to
be observed.

To perform Markov chain Monte Carlo sampling, we implemented the model in
NumPyro (Phan et al., 2019), with local latent variables $Z_{n}$ marginalized
out and performed sampling using the NUTS sampler (Hoffman & Gelman, 2014). We
used 2000 warm-up steps and collected 800 samples.

The $m$th sample is therefore given by

$\theta^{(m)}=\left(\pi^{(m)},\left(\mu_{k}^{(m)},\Sigma_{k}^{(m)}\right)_{k=1,\dotsc,K}\right)$

which is then used to parametrize a Gaussian mixture distribution
$P_{\theta^{(m)}}$.

Finally, using Monte Carlo method described in Sec. 2 (with 100,000 samples)
we then estimated mutual information $\mathbf{I}(P_{\theta^{(m)}})$ and the
profile.

We used the above procedure to perform inference in the model applied to the
X, AI, Waves and Galaxy distributions, changing the number of data points
$N\in\\{125,250,500,1000\\}$. We visualise the observed sample, a single
posterior predictive sample and posterior on mutual information and the PMI
profile in Fig. 9, Fig. 10, Fig. 11 and Fig. 12.

Figure 9:  Gaussian mixture model fitted to the X distribution with 125, 250,
500 and 1000 samples.

Figure 10:  Gaussian mixture model fitted to the AI distribution with 125,
250, 500 and 1000 samples.

Figure 11:  Gaussian mixture model fitted to the Waves distribution with 125,
250, 500 and 1000 samples.

Figure 12:  Gaussian mixture model fitted to the Galaxy distribution with 125,
250, 500 and 1000 samples.