# On the Effectiveness of Hybrid Mutual Information Estimation

Marco Federici  David Ruhe  Patrick Forré

###### Abstract

Estimating the mutual information from samples from a joint distribution is a
challenging problem in both science and engineering. In this work, we realize
a variational bound that generalizes both discriminative and generative
approaches. Using this bound, we propose a hybrid method to mitigate their
respective shortcomings. Further, we propose Predictive Quantization (PQ): a
simple generative method that can be easily combined with discriminative
estimators for minimal computational overhead. Our propositions yield a
tighter bound on the information thanks to the reduced variance of the
estimator. We test our methods on a challenging task of correlated high-
dimensional Gaussian distributions and a stochastic process involving a system
of free particles subjected to a fixed energy landscape. Empirical results
show that hybrid methods consistently improved mutual information estimates
when compared to the corresponding discriminative counterpart.

Machine Learning, Information Theory

##  1 Introduction

$\log p({\mathbf{x}})p({\mathbf{y}})$

$\log p({\mathbf{x}})p({\mathbf{y}}|Q_{2}({\mathbf{x}}))$

$\log p({\mathbf{x}})p({\mathbf{y}}|Q_{4}({\mathbf{x}}))$

$\log p({\mathbf{x}})p({\mathbf{y}}|Q_{8}({\mathbf{x}}))$

$\log p({\mathbf{x}})p({\mathbf{y}}|{\mathbf{x}})$

Figure 1: Predictive Quantization (PQ). This figure shows how a simple
quantization function $Q({\mathbf{x}})$ can be used to obtain proposals that
lie closer to a target joint distribution (on the right) than the product of
the marginals (on the left). We can use these intermediate distributions to
split the problem of estimating mutual information into two parts, effectively
reducing the variance of popular discriminative estimators. This results in a
tighter estimate of the variational bound. Vertical dashed lines are used to
indicate the quantized regions. Note that as the number of partitions
increases, the proposal approaches the joint distribution.

Mutual Information quantifies the amount of information gained about one
random variable by observing another one (Shannon, 1948; MacKay et al., 2003).
As such, it is a measure of their mutual dependence. Estimating the dependency
of two random variables is a problem found ubiquitously in science and
engineering. Examples are independent component analysis (Hyvärinen & Oja,
2000; Bach & Jordan, 2002), neuroscience (Palmer et al., 2015), Bayesian
experimental design (Ryan et al., 2016). Further, mutual information
estimation plays a crucial role in self-supervised learning, which in recent
years has gained significant attention from the machine learning community
(Hjelm et al., 2018; Zbontar et al., 2021) in the form of information
optimization. Information optimization, such as in the information bottleneck
method (minimization) (Tishby et al., 2000; Alemi et al., 2016) or deep
representation learning (Bengio et al., 2013; Hjelm et al., 2018; van den Oord
et al., 2018), can be conducted without explicitly modeling the underlying
distributions. Instead, through directly bounding the mutual information, one
can obtain tractable objective functions that can be optimized using flexible
function estimators.

Mutual information estimation, the core focus of this work, is a challenging
task since one usually has access only to samples from the underlying
probability distributions, and not their densities (Paninski, 2003; McAllester
& Stratos, 2020). Classical estimators (Kraskov et al., 2004; Gao et al.,
2015) typically break down when the data is higher-dimensional and can be non-
differentiable, making them unsuitable for gradient-based information
optimization (Hjelm et al., 2018).

Variational mutual information estimation considers a lower bound on the true
information. Therefore, maximizing this bound using a flexible variational
family yields accurate information estimates. By identifying a bound that
unifies the generative and discriminative methods as categorized by Song &
Ermon (2020) and Poole et al. (2019), we aim to address their respective
shortcomings. That is, generative approaches lack flexibility and
discriminative approaches suffer from unfavorable bias-variance trade-offs for
large estimates. The unification of these techniques provides a method that
can provide more accurate estimation.

Further, we specify a simple yet effective generative method called
_Predictive Quantization_ (PQ) that makes use of a quantized (discrete)
representation of the data, such as a clustering or a set of external
attributes, to approximate quantities (such as a marginal entropy) that are
usually intractable.

Our contributions, therefore, include the following:

  1. 1.

We introduce a novel family of hybrid mutual information estimators which
generalizes both generative and discriminative approaches.

  2. 2.

We show theoretically and empirically that there is a clear advantage to
combining generative and discriminative mutual information estimators since
they have complementary strengths and weaknesses.

  3. 3.

We design Predictive Quantization (PQ): a simple generative method that can to
improve a wide range of recently proposed discriminative estimators.

Experimentally, we test our models by estimating mutual information between
the dimensions of a mixture of correlated multi-dimensional Gaussian
distributions and compare the estimates of temporal information for a
stochastic process of spatially correlated moving particles subjected to a
fixed energy landscape.

##  2 Mutual Information Estimation

Given two random variables ${\mathbf{x}}$ and ${\mathbf{y}}$ with support
${\mathbb{X}}$, ${\mathbb{Y}}$ and joint probability density
$p({\mathbf{x}},{\mathbf{y}})$, the mutual information between ${\mathbf{x}}$
and ${\mathbf{y}}$ is defined as Kullback-Leibler (KL) divergence between
$p({\mathbf{x}},{\mathbf{y}})$ and the product of the two marginal
distributions111Unless otherwise specified, expectations are computed with
respect to $p({\mathbf{x}},{\mathbf{y}})$. See Appendix A for notational
details.:

$\displaystyle I({\mathbf{x}};{\mathbf{y}})$
$\displaystyle\overset{\operatorname{def}}{=}{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||p({\mathbf{x}})p({\mathbf{y}}))$
$\displaystyle=\mathbb{E}\left[\log\frac{p({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right].$
(1)

In most applications of interest, we can sample ${\bm{x}},{\bm{y}}\sim
p({\mathbf{x}},{\mathbf{y}})$ to compute a Monte Carlo estimate. However, some
of the densities in Equation 1 are usually unknown or intractable. For this
reason, a common strategy involves the introduction of a variational
distribution $q({\mathbf{x}},{\mathbf{y}})$ to obtain a lower bound on mutual
information, which can be used for either estimation or maximization:

$\displaystyle I({\mathbf{x}};{\mathbf{y}})$
$\displaystyle=\mathbb{E}\left[\log\left(\frac{q({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\frac{p({\bm{x}},{\bm{y}})}{q({\bm{x}},{\bm{y}})}\right)\right]$
$\displaystyle=\mathbb{E}\left[\log\frac{q({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right]+\underbrace{{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||q({\mathbf{x}},{\mathbf{y}}))}_{\text{Variational
Gap}}$
$\displaystyle\geq\mathbb{E}\left[\log\frac{q({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right]$
$\displaystyle\overset{\operatorname{def}}{=}I_{q}({\mathbf{x}},{\mathbf{y}}),$
(2)

where $q({\mathbf{x}},{\mathbf{y}})$ is a joint density in a set
${\mathcal{Q}}$ of attainable variational densities with positive support on
${\mathbb{X}}\times{\mathbb{Y}}$. Note that the bound in Equation 2 is tight
only when $p({\mathbf{x}},{\mathbf{y}})\in\mathcal{Q}$. Therefore, accurate
mutual information estimation requires access to a flexible variational
family. The KL divergence between the joint distribution
$p({\mathbf{x}},{\mathbf{y}})$ and its variational approximation
$q({\mathbf{x}},{\mathbf{y}})$ is commonly referred to as the _variational
gap_ , which we seek to minimize for accurate estimation.

Previous work (Poole et al., 2019; Song & Ermon, 2020) has identified two
categories of approaches for maximizing the ratio
$I_{q}({\mathbf{x}};{\mathbf{y}})$: generative and discriminative methods. In
the following sections, we recover both these approaches using a
parameterization for $q({\mathbf{x}},{\mathbf{y}})$ consisting of a normalized
proposal distribution (generative) and an unnormalized density ratio
(discriminative). On one hand, discriminative approaches (Belghazi et al.,
2018; Nguyen et al., 2010) directly model the entire ratio, but this comes at
the cost of considerable bias (van den Oord et al., 2018) or high variance for
large values of mutual information (McAllester & Stratos, 2020). Generative
approaches (Barber & Agakov, 2003), on the other hand, model the components in
Equation 2 using learnable normalized densities. However, access to flexible
parameterized distributions can be hard to attain, or they are expensive to
optimize. These issues usually become more severe for high-dimensional
${\mathbf{x}}$ and ${\mathbf{y}}$.

| Proposal | Energy
---|---|---
Generative | $r_{\theta}({\mathbf{x}},{\mathbf{y}})$ | $f_{k}({\mathbf{x}},{\mathbf{y}})=k$
Discriminative | $p({\mathbf{x}})p({\mathbf{y}})$ | $f_{\phi}({\mathbf{x}},{\mathbf{y}})$
Hybrid (Ours) | $r_{\theta}({\mathbf{x}},{\mathbf{y}})$ | $f_{\phi}({\mathbf{x}},{\mathbf{y}})$
Table 1: Summary of the modeling choices for the main families of approaches
for mutual information estimation with respect to the expression in Equation
6. Our proposed hybrid approach can exploit the flexibility of discriminative
models while mitigating the variance because of the use of more expressive
proposals.
$p({\mathbf{x}},{\mathbf{y}})$$q({\mathbf{x}},{\mathbf{y}})$$r({\mathbf{x}},{\mathbf{y}})$$p({\mathbf{x}})p({\mathbf{y}})$$I({\mathbf{x}};{\mathbf{y}})={\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||p({\mathbf{x}})p({\mathbf{y}}))$$KL(p({\mathbf{x}},{\mathbf{y}})||q({\mathbf{x}},{\mathbf{y}}))$$I_{q}({\mathbf{x}},{\mathbf{y}})$$I_{r}({\mathbf{x}},{\mathbf{y}})$${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$$KL(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
Figure 2: Visualization of the additive decomposition of the terms of Equation
2 and Equation 6. The mutual information lower-bound for the implicitly
defined distribution $q({\mathbf{x}},{\mathbf{y}})$ (in blue) can be seen as
the sum of the lower-bound for the proposal distribution
$r({\mathbf{x}},{\mathbf{y}})$ (in orange) and the estimation of the
corresponding variational gap (in red).

##  3 A Generalized Variational Approach

In the following, we set up a parametrization for
$q({\mathbf{x}},{\mathbf{y}})$ that allows us to derive popular mutual
information estimators as special cases and propose new generalized hybrid
estimators. First, note that any joint probability distribution
$q({\mathbf{x}},{\mathbf{y}})$ can be written as a normalized exponential:

$\displaystyle q({\mathbf{x}},{\mathbf{y}})$
$\displaystyle=\frac{e^{F({\mathbf{x}},{\mathbf{y}})}}{Z_{F}},\ \ \text{with
}Z_{F}\overset{\operatorname{def}}{=}\iint
e^{F({\bm{x}},{\bm{y}})}d{\bm{x}}d{\bm{y}}.$ (3)

The function $F:{\mathbb{X}}\times{\mathbb{Y}}\to{\mathbb{R}}$ is commonly
referred to as the negative energy function and $Z_{F}$ as its normalization
constant222The energy needs to satisfy mild integrability constraints.. The
main advantage is that there are few restrictions on $F$, a property that has
fruitfully been exploited by energy-based models (Song & Kingma, 2021;
Grathwohl et al., 2019; Haarnoja et al., 2017; Song et al., 2020; Hyvärinen &
Dayan, 2005; Ngiam et al., 2011).

The main disadvantage, however, is that the normalization constant $Z_{F}$ is
generally intractable due to computationally expensive integration over the
support ${\mathbb{X}}\times{\mathbb{Y}}$. To address this problem, we compute
a Monte Carlo estimate of $Z_{F}$ by sampling from a proposal distribution
$r({\mathbf{x}},{\mathbf{y}})$ from an attainable family of normalized
densities ${\mathcal{R}}$:

$\displaystyle Z_{F}$
$\displaystyle=\iint\frac{r({\bm{x}},{\bm{y}})}{r({\bm{x}},{\bm{y}})}e^{F({\bm{x}},{\bm{y}})}d{\bm{x}}d{\bm{y}}$
$\displaystyle=\mathbb{E}_{r}\left[e^{F({\bm{x}},{\bm{y}})-\log
r({\bm{x}},{\bm{y}})}\right].$ (4)

Lastly, we can reparameterize the energy such that
$F({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}\log
r({\mathbf{x}},{\mathbf{y}})+f({\mathbf{x}},{\mathbf{y}})$ to obtain

$\displaystyle
q({\mathbf{x}},{\mathbf{y}})=r({\mathbf{x}},{\mathbf{y}})\frac{e^{f({\mathbf{x}},{\mathbf{y}})}}{Z_{f}},\
\ \text{with }Z_{f}=\mathbb{E}_{r}[e^{f({\bm{x}},{\bm{y}})}].$ (5)

Intuitively, $q({\mathbf{x}},{\mathbf{y}})$ is obtained by transforming the
proposal $r({\mathbf{x}},{\mathbf{y}})\in{\mathcal{R}}$ with a critic function
$f\in{\mathcal{F}}$, and re-normalizing to obtain a valid density. We denote
the family of variational distributions obtained with this procedure as
${\mathcal{Q}}\overset{\operatorname{def}}{=}{\mathcal{R}}_{\mathcal{F}}$ to
underline the dependency on the chosen proposals ${\mathcal{R}}$ and critics
${\mathcal{F}}$.

Using this parameterization in Equation 2, we obtain a general bound that
includes both a generative and a discriminative component:

$\displaystyle I_{q}({\mathbf{x}},{\mathbf{y}})$
$\displaystyle=\underbrace{\mathbb{E}\left[\log\frac{r({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right]}_{I_{r}({\mathbf{x}},{\mathbf{y}})}$
$\displaystyle\ \ \
+\underbrace{\mathbb{E}[f({\bm{x}},{\bm{y}})]-\log\mathbb{E}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]}_{{{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}}.$
(6)

The entire decomposition is visualized in Figure 2. The total information
$I({\mathbf{x}};{\mathbf{y}})$ (first row) decomposes through Equation 2 into
the lower-bound $I_{q}({\mathbf{x}};{\mathbf{y}})$ and respective variational
gap
${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
(second row). $I_{q}({\mathbf{x}};{\mathbf{y}})$ is then further split into
the terms of Equation 6 (third row). The first component
$I_{r}({\mathbf{x}};{\mathbf{y}})$ is a mutual information lower bound
determined by the proposal $r({\mathbf{x}},{\mathbf{y}})\in{\mathcal{R}}$.
This quantity differs from the target mutual information
$I({\mathbf{x}};{\mathbf{y}})$ by the variational gap for
$r({\mathbf{x}},{\mathbf{y}})$:
${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
(fourth row). I.e.,
$I({\mathbf{x}};{\mathbf{y}})=I_{r}({\mathbf{x}};{\mathbf{y}})+{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$.

The second component in Equation 6 consists of the comparison between the
value of the critic $f$ of samples from the true joint to samples from the
proposal. This expression can be seen as the Donsker-Varadhan representation
of the aforementioned variational gap of the proposal (Donsker & Varadhan,
1983):

$\displaystyle{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))\geq{{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))},$
(7)

in which the inequality is tight when ${\mathcal{F}}$ contains
$f^{*}_{k}({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}\log\frac{p({\mathbf{x}},{\mathbf{y}})}{r({\mathbf{x}},{\mathbf{y}})}+k$
with $k\in{\mathbb{R}}$ (Poole et al., 2019). Note that the variational gap
for $q$ is always smaller than the variational gap for $r$ whenever
${\mathcal{F}}$ contains at least a constant ($k$) function:

$\displaystyle\exists
f\in\mathcal{F},\,f:{\mathbb{X}}\times{\mathbb{Y}}\to\\{k\\}\implies\max_{f\in{\mathcal{F}}}I_{q}({\mathbf{x}};{\mathbf{y}})\geq
I_{r}({\mathbf{x}};{\mathbf{y}})$ (8)

Since critics are usually modeled using flexible function approximators, this
is a reasonable assumption, and, in practice, the family of transformed
densities ${\mathcal{R}}_{\mathcal{F}}$ is much larger than original proposals
${\mathcal{R}}$.

As summarized in Table 1, previous approaches for mutual information
estimation can be seen as instances of the parameterization reported in
Equation 2 obtained by either restricting
$r({\mathbf{x}},{\mathbf{y}})=p({\mathbf{x}})p({\mathbf{y}})$ (discriminative
methods) or using a constant critic function (generative methods). In the
following sections, we analyze the strengths and weaknesses of each method to
design a novel hybrid approach that takes advantage of the best
characteristics of both.

###  3.1 The Discriminative Approach: a Bias-Variance Trade-Off

In this section we focus on discriminative methods, which we recover by
setting $r({\mathbf{x}},{\mathbf{y}})=p({\mathbf{x}})p({\mathbf{y}})$, causing
$I_{r}({\mathbf{x}},{\mathbf{y}})$ to vanish from Equation 6. On the one hand,
modeling density ratios directly with neural network critics allow for great
flexibility (Gutmann & Hyvärinen, 2010; van den Oord et al., 2018). On the
other, the efficacy of techniques to estimate
${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$
heavily depends on bias-variance trade-offs of the chosen approximation to
evaluate the normalization constant (Poole et al., 2019; McAllester & Stratos,
2020; Song & Ermon, 2020). In practice, Monte Carlo estimation of the log-
normalization constant $\log Z_{f}$ with a limited number of samples yields
biased results caused by the log-expectation (Belghazi et al., 2018; van den
Oord et al., 2018). For this reason, we consider a looser lower-bound of
${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
corresponding to the dual representation of the KL-divergence (Donsker &
Varadhan, 1983):

$\displaystyle{{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$
$\displaystyle\ \ \
\geq\mathbb{E}_{p}[f({\bm{x}},{\bm{y}})]-\mathbb{E}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]+1,$
(9)

where the inequality is tight when the normalization constant $Z_{f}=1$.

Estimating Equation 9 using Monte Carlo samples suffers from less bias at the
cost of a higher variance (Nguyen et al., 2010; Poole et al., 2019; Guo et
al., 2022). In fact, the variance of
${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$
for a critic $f\in{\mathcal{F}}$ is mostly determined by the variance of
$e^{f({\bm{x}},{\mathbf{y}})}$, which can be bounded from below by an
exponential of the KL-divergence between the implicitly defined
$q({\mathbf{x}},{\mathbf{y}})$ and original proposal
$r({\mathbf{x}},{\mathbf{y}})$:

$\displaystyle\mathbb{V}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]$
$\displaystyle\geq
Z_{f}^{2}{\chi^{2}}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
$\displaystyle\geq
Z^{2}_{f}\left(e^{{\mathrm{KL}}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}-1\right).$
(10)

Here, ${\chi^{2}}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
denotes the Pearson chi-squared divergence between the variational
distribution and the proposal. Intuitively, the variance increases the more
the critic changes the proposal. Whenever the critic is constant on the whole
support ${\mathbb{X}}\times{\mathbb{Y}}$, $q({\mathbf{x}},{\mathbf{y}})$
matches $r({\mathbf{x}},{\mathbf{y}})$, rendering the variance and value of
${{\mathrm{KL}}_{f}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$
zero. In this setting, the variational gap is completely determined by the
proposal, since
$I_{q}({\mathbf{x}};{\mathbf{y}})=I_{r}({\mathbf{x}};{\mathbf{y}})$. In
contrast, when $q({\mathbf{x}},{\mathbf{y}})=p({\mathbf{x}},{\mathbf{y}})$
(i.e., the critic is optimal
$f({\mathbf{x}},{\mathbf{y}})=f^{*}({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}\log\frac{p({\mathbf{x}},{\mathbf{y}})}{r({\mathbf{x}},{\mathbf{y}})}$),
the variational gap for $q({\mathbf{x}},{\mathbf{y}})$ is zero, but the
variance is still determined by an exponential of the variational gap for
$r({\mathbf{x}},{\mathbf{y}})$. In the case in which
$r({\mathbf{x}},{\mathbf{y}})=p({\mathbf{x}})p({\mathbf{y}})$, the variance
grows exponentially with the amount of information to estimate (McAllester &
Stratos, 2020; Song & Ermon, 2020).

###  3.2 On the Generative Approach

When setting $f({\mathbf{x}},{\mathbf{y}})=k$, the second term of Equation 6
vanishes. We are now in the generative setting. Considering
$I_{r}({\mathbf{x}};{\mathbf{y}})$, computing and optimizing the ratio between
the proposal and the product of the marginals requires access to a flexible
family of distributions with known (or approximate) densities. Depending on
the modeling choices for $r({\mathbf{x}},{\mathbf{y}})$, the computation may
require estimating up to two entropy and one cross-entropy term:

$\displaystyle I_{r}({\mathbf{x}};{\mathbf{y}})$
$\displaystyle=\mathbb{E}[\log
r({\mathbf{x}},{\mathbf{y}})]+H({\mathbf{x}})+H({\mathbf{y}}).$ (11)

Barber & Agakov (2003) and McAllester & Stratos (2020) model a joint proposal
as the product of one of the marginals and a conditional proposal
$r_{\mathbf{x}}({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}p({\mathbf{x}})r({\mathbf{y}}|{\mathbf{x}})$.
This reduces the computation to the estimation of only one entropy and a
cross-entropy term:

$\displaystyle I_{r_{\mathbf{x}}}({\mathbf{x}};{\mathbf{y}})$
$\displaystyle=\mathbb{E}[\log r({\mathbf{y}}|{\mathbf{x}})]+H({\mathbf{y}}).$
(12)

Popular modeling choices for joint and conditional proposals include
transforming simple densities using normalizing flows (Rezende & Mohamed,
2015; Kingma et al., 2016; Dinh et al., 2017) or using variational lower-
bounds (Kingma & Welling, 2014). Despite recent advances (Durkan et al., 2019;
Ho et al., 2020), generative approaches tend to show a trade-off between
flexibility and computational costs, which may limit their effectiveness in
high-dimensional settings.

###  3.3 A Hybrid Method

We can combine the advantages of both methods by (i) considering proposals
$r({\mathbf{x}},{\mathbf{y}})$ such that
${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))<{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||p({\mathbf{x}})p({\mathbf{y}}))$
to lower the variance for the estimates of
${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$.
Furthermore, we choose a flexible $f({\mathbf{x}},{\mathbf{y}})$, e.g., a
neural network, that refines the proposal.

Both terms of Equation 6 are now nonzero. Note that if we already had access
to flexible density estimators, we can now simply include them as
$r({\mathbf{x}},{\mathbf{y}})$, where $f({\mathbf{x}},{\mathbf{y}})$ corrects
for any lack of flexibility. One can therefore apply an iterative method:

  1. 1.

Pick the best proposal $\hat{r}({\mathbf{x}},{\mathbf{y}})\in{\mathcal{R}}$ to
minimize the variational gap. This is equivalent to maximizing
$I_{r}({\mathbf{x}},{\mathbf{y}})$:

$\displaystyle\hat{r}({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}\operatorname*{arg\,max}_{r\in{\mathcal{R}}}I_{r}({\mathbf{x}};{\mathbf{y}})$
(13)

  2. 2.

Learn the ratio between the best proposal and the joint distribution:

$\displaystyle\hat{f}({\mathbf{x}},{\mathbf{y}})=\operatorname*{arg\,max}_{f\in{\mathcal{F}}}{{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||\hat{r}({\mathbf{x}},{\mathbf{y}}))}$
(14)

In practice, these two terms can be jointly optimized for fixed distributions
of ${\mathbf{x}}$ and ${\mathbf{y}}$ as long as the objective in Equation 14
is treated as constant w.r.t the proposal.

Since our hybrid approach includes a flexible $f({\mathbf{x}},{\mathbf{y}})$,
we design in the next section a simple family of proposals ${\mathcal{R}}_{Q}$
that are guaranteed to lie closer to the joint distribution
$p({\mathbf{x}},{\mathbf{y}})$ than the product of marginals, but yield simple
and efficient computation of $I_{r_{Q}}({\mathbf{x}},{\mathbf{y}})$.

##  4 Predictive Quantization

input $p({\mathbf{x}},{\mathbf{y}})$, $Q({\mathbf{x}})$,
$s_{\psi}(Q({\mathbf{x}})|{\mathbf{y}})$

repeat

$\bar{\bm{x}}\sim p(Q({\mathbf{x}}))$

$({\bm{x}}_{i},{\bm{y}}_{i})_{i=1}^{B}\stackrel{{\scriptstyle B}}{{\sim}}\
$$p({\mathbf{x}},{\mathbf{y}})$$\
p({\mathbf{x}},{\mathbf{y}}|Q({\mathbf{x}})=\bar{\bm{x}})$.

$\hat{I}({\mathbf{x}};{\mathbf{y}})\leftarrow$
DiscriminativeMIEstimate$(({\bm{x}}_{i},{\bm{y}}_{i})_{i=1}^{B};\phi)$

$\hskip 70.0001pt+H(Q({\mathbf{x}}))+\log
s_{\psi}(\bar{x}|({\bm{y}}_{i})_{i=1}^{B})$

$\phi\leftarrow\eta\frac{\partial\hat{I}({\mathbf{x}};{\mathbf{y}})}{\partial\phi}$

$\psi\leftarrow\eta\frac{\partial\hat{I}({\mathbf{x}};{\mathbf{y}})}{\partial\psi}$

until converged

Algorithm 1 Hybrid Mutual Information Estimate with PQ

We are interested in proposal distributions that we can easily sample (to
estimate $Z_{f}$) and whose $I_{r}({\mathbf{x}},{\mathbf{y}})$ we can
effectively approximate. We define a joint distribution
$r_{Q}({\mathbf{x}},{\mathbf{y}})$ that factorizes as

$\displaystyle
r_{Q}({\mathbf{x}},{\mathbf{y}})\overset{\operatorname{def}}{=}p({\mathbf{x}})p({\mathbf{y}}|Q({\mathbf{x}})),$
(15)

where $Q$ maps to some discrete quantization
$\bar{\mathbf{x}}=Q({\mathbf{x}})$, e.g., a cluster index. This proposal
allows tractable estimation of $I_{r}({\mathbf{x}};{\mathbf{y}})$ since

$\displaystyle I_{r_{Q}}({\mathbf{x}};{\mathbf{y}})$
$\displaystyle=\mathbb{E}\left[\log\frac{p({\bm{y}}|\bar{{\bm{x}}})}{p({\bm{y}})}\right]=I(\bar{{\bm{x}}};{\mathbf{y}})$
(16) $\displaystyle=\mathbb{E}\left[\log
p(\bar{{\bm{x}}}|{\bm{y}})\right]+H(\bar{{\bm{x}}})$
$\displaystyle\geq\mathbb{E}\left[\log
s_{\psi}(\bar{{\bm{x}}}|{\bm{y}})\right]+H(\bar{{\bm{x}}}).$

Here, $s_{\phi}({\mathbf{x}}|{\mathbf{y}})$ is a variational approximation to
$p(\bar{{\mathbf{x}}}|{\mathbf{y}})$. This form of the proposal has three key
advantages:

  1. 1.

Since $p(\bar{{\mathbf{x}}})$ is discrete, we can obtain good estimates of
$H(\bar{\bm{x}})$ for reasonable numbers of quantization intervals. Further,
we easily sample $r_{Q}({\mathbf{x}},{\mathbf{y}})$.

  2. 2.

$s_{\psi}(\bar{{\mathbf{x}}}|{\mathbf{y}})$ can be parametrized as a
categorical distribution, obtaining a simple prediction task of
$\bar{{\mathbf{x}}}$ given ${\mathbf{y}}$. We can amortize the inference using
a deep neural network.

  3. 3.

${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r_{Q}({\mathbf{x}},{\mathbf{y}}))<{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||p({\mathbf{x}})p({\mathbf{y}}))$
for any $Q$ such that $I({\mathbf{y}};Q({\mathbf{x}}))>0$.

The last advantage is quickly identified by observing

$\displaystyle{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r_{Q}({\mathbf{x}},{\mathbf{y}}))$
$\displaystyle\stackrel{{\scriptstyle
Eq.\ref{eq:mi_lbound}}}{{=}}I({\mathbf{x}};{\mathbf{y}})-I_{r_{Q}}({\mathbf{y}};{\mathbf{x}})$
$\displaystyle\stackrel{{\scriptstyle
Eq.\ref{eq:quantized_info}}}{{=}}I({\mathbf{x}};{\mathbf{y}})-I({\mathbf{y}};Q({\mathbf{x}}))$
$\displaystyle\leq I({\mathbf{x}};{\mathbf{y}}).$ (17)

Generally, we want to choose $Q({\mathbf{x}})$ so that
$I({\mathbf{y}};\bar{{\mathbf{x}}})$ is large, causing
$r_{Q}({\mathbf{x}},{\mathbf{y}})$ to be closer to the optimal proposal
(visualized in Figure 1). On the other hand, the number of quantization
intervals should be much smaller than the number of samples we have to
estimate the information. This is because estimates of discrete entropy
$H(\bar{\bm{x}})$ become less reliable for a large number of quantization
intervals. Examples of viable quantization functions could include clustering
algorithms or handcrafted functions of ${\mathbf{x}}$, e.g., a weak label from
weakly supervised tasks (Ratner et al., 2017).

The PQ generative estimator can be easily integrated with existing
discriminative mutual information estimators by sampling batches
$({\bm{x}}_{i},{\bm{y}}_{i})_{i=1}^{B}$ in which each ${\bm{x}}_{i}$ lies in
the same quantized region ($\,\forall
i,j\in[B],\,Q({\bm{x}}_{i})=Q({\bm{x}}_{j})$) and adding the estimation of
$I(Q({\mathbf{x}});{\mathbf{y}})$ to the original discriminative estimate for
$KL(p({\mathbf{x}},{\mathbf{y}})||r_{Q}({\mathbf{x}},{\mathbf{y}}))$. In
Algorithm 1 we underline (in green) the simple modifications required to
integrate PQ with a given discriminative estimator that produces samples from
the marginal $p({\mathbf{x}})p({\mathbf{y}})$ by shuffling the
$({\bm{x}}_{i},{\bm{y}}_{i})$ pair within a batch. By sampling each batch
conditioned on one quantized value $\bar{\bm{x}}$, the shuffling operation
becomes equivalent to sampling from
$p({\mathbf{x}})p({\mathbf{y}}|\bar{\bm{x}})$ instead.

| $r({\mathbf{x}},{\mathbf{y}})$ | $I_{r}({\mathbf{x}};{\mathbf{y}})$ | Requirements
---|---|---|---
BA (Barber & Agakov, 2003) | $p({\mathbf{x}})r_{\theta}({\bm{y}}|{\bm{x}})$ | $\mathbb{E}[\log r_{\theta}({\bm{y}}|{\bm{x}})]+H({\mathbf{y}})$ |  $H({\mathbf{y}})$ known
DoE (McAllester & Stratos, 2020) | $p({\mathbf{x}})r_{\theta}({\mathbf{y}}|{\mathbf{x}})$ | $\mathbb{E}[\log r_{\theta}({\bm{y}}|{\bm{x}})-\log s_{\xi}({\bm{y}})]$ |  $p({\mathbf{y}})$ fixed
GM (Song & Ermon, 2020) | $r_{\theta}({\bm{x}},{\bm{y}})$ | $\mathbb{E}[\log r_{\theta}({\bm{x}},{\bm{y}})-\log s_{\psi}({\bm{x}})-\log s_{\xi}({\bm{y}})]$ |  $p({\mathbf{x}})$, $p({\mathbf{y}})$ fixed
PQ (Ours) | $p({\mathbf{x}})p({\mathbf{y}}|\bar{\mathbf{x}}))$ | $\mathbb{E}[\log s_{\psi}(\bar{\bm{x}}|{\bm{y}})]+H(\bar{\mathbf{x}}))$ | $\bar{\mathbf{x}}=Q({\mathbf{x}})$
Table 2: Overview of the main generative approaches to mutual information
estimation and their requirements.

##  5 Related Work

Recent work on variational mutual information estimation (Poole et al., 2019)
provides an overview of tractable and scalable objectives for estimating and
optimizing Mutual Information (MI), identifying some of the characteristics
and limitations of the two approaches. The authors analyze the trade-off
between bias and variance for discriminative MI estimators focusing on critic
architectures and the techniques used to estimate the normalization constant.
They propose an interpolation between discriminative estimators with a low
variance but high bias (van den Oord et al., 2018), and high-variance low-bias
estimators (Nguyen et al., 2010).

Song & Ermon (2020) show that discriminative approaches such as Mutual
Information Neural Estimation (MINE) (Belghazi et al., 2018) and the Nguyen-
Wainright-Jordan (NWJ) method (Nguyen et al., 2010) exhibit variance that
grows exponentially with the true amount of underlying information, making
them less suitable for estimation of large information quantities. The authors
propose to control the bias-variance trade-off of the aforementioned
estimators by clipping the critic’s values during training. Other closely
related work focuses on the statistical limitation of MI estimators
(McAllester & Stratos, 2020). The authors underline that the exponential
variance scaling affects any variational lower bound and focus on generative
mutual information estimation based on the difference of entropies (DoE).

Other works consider factorizing the discriminative approaches using the sum
of conditional and marginal MI terms (Sordoni et al., 2021), expressing the
density ratio as a sum of several simpler critics (Rhodes et al., 2020), and
using Fenchel-Legendre transform for MI estimation and optimization (Guo et
al., 2022) to reduce the dependency on large-batch training and increase bound
tightness.

Our work further unifies these families of estimators, addressing the
exponential scaling issue of discriminative estimators by bringing the
distribution used to estimate the normalization constant closer to the joint
distribution of the two variables. The effectiveness of the combination of
normalized (generative) and unnormalized (discriminative) distributions has
been shown in the context of energy-based models Gao et al. (2020); Xiao et
al. (2021). We further propose Predictive Quantization (PQ) as a simple yet
effective generative estimator inspired by the literature on discrete mutual
information estimation (Cover & Thomas, 2006; Gao et al., 2017) that can be
easily combined with any of the aforementioned discriminative estimators to
effectively reduce their variance.

##  6 Empirical Evaluation

We test our proposed hybrid models on a particularly challenging correlated
mixture of normal distributions, and a discrete-time particle simulation task.
These tasks have been selected to satisfy the following criteria.

  1. 1.

Known True Mutual Information. Some of the estimators considered in this
analysis are lower bounds, while others tend to overestimate the true value of
mutual information. For this reason, we exclusively selected benchmarks in
which the true value of mutual information is known.

  2. 2.

Controllable Mutual Information. We consider tasks where the amount of
information can be tuned by, e.g., a higher dimensionality or increasing the
number of particles. We designed our experiments to make the difference
between benchmarked estimators explicit by varying the batch size and target
mutual information.

$(a)$

$(b)$

Figure 3: (a) Blue: the estimated mutual information in nats of several
discriminative estimators. Orange: mutual information estimates using our PQ
method. Green: results obtained by including a learned conditional normal
proposal. Hybrid estimators improve upon their discriminative counterparts. We
further indicate the parts of the total information contributed by
$I_{r}({\mathbf{x}};{\mathbf{y}})$ and
${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$,
respectively. (b) Bias and variance of the included discriminative estimators
(denoted by colors) as a function of batch size. For all estimators, we report
a lowered bias using PQ.

###  6.1 Models and Optimization

We evaluate the performance of several discriminative mutual information
estimators that differ for the computation of the normalization constant, the
parameterization of the energy function, and their optimization (see Appendix
B for details regarding each estimator). For each discriminative model, we
consider three proposals:

  1. 1.

The product of the marginals $p({\mathbf{x}})p({\mathbf{y}})$. This
corresponds to using no generative component.

  2. 2.

The product of the marginal $p({\mathbf{x}})$ and a conditional Normal
proposal
$\mathcal{N}({\mathbf{y}}|\bm{\mu}_{\theta}({\mathbf{x}}),\bm{\sigma}^{2}_{\theta}({\mathbf{x}}))$
(BA, DoE). The functions $\bm{\mu}_{\theta}({\mathbf{x}})$ and
$\bm{\sigma}^{2}_{\theta}({\mathbf{x}})$ are parametrized by neural networks.
Note that mutual information estimation using this approach requires
estimating the entropy of the marginal distribution $p({\mathbf{y}})$.

  3. 3.

The product of $p({\mathbf{x}})$ and $p({\mathbf{y}}|Q({\mathbf{x}}))$ defined
in section 4. This requires specifying a fixed quantization function
$Q({\mathbf{x}})$ and a classifier $s_{\psi}(Q({\mathbf{x}})|{\mathbf{y}})$.

For a fair comparison, all the architectures used in this analysis consist of
simple Multi-Layer Perceptrons (MLP), ReLU activations (Nair & Hinton, 2010),
and the same neural architecture. All models are trained using Adam (Kingma &
Ba, 2014) with a learning rate of 5e-4 for a total of 100,000 iterations.

$(a)$

$(b)$

Figure 4: (a) Shows the mutual information estimates of several approaches on
the particle dataset. Using PQ (orange), all estimators yield better estimates
closer to the true information than the baselines (blue). (b) Bias-variance
analysis of the number of clusters used for PQ. Note that 0 clusters
correspond to no generative estimator.

###  6.2 Tasks and Results

#### Mixture of Correlated Normals

Following previous work (Poole et al., 2019; Song & Ermon, 2019; Guo et al.,
2022), we create a dataset by sampling $100\,000$ points from a correlated
distribution. Instead of using a two-dimensional correlated normal
distribution, which simple generative estimators can easily fit, we use a
mixture of 4 correlated normal distributions. The location and scale of each
component are chosen such that both marginals $p({\mathbf{x}})$,
$p({\mathbf{y}})$ and conditionals $p({\mathbf{y}}|{\mathbf{x}})$,
$p({\mathbf{x}}|{\mathbf{y}})$ are bimodal distributions. The log-density of
the product and joint distribution is visualized in Figure 1. Each pair of
dimensions shares $\approx 1.37$ nats of information. We stacked 5 independent
versions to reach an amount of mutual information that is challenging enough
to underline issues of the models in this analysis. We define the quantization
as a component-wise indicator function
$Q_{2}({\mathbf{x}})=\mathbb{I}({\mathbf{x}}>0)$, separating positive and
negative values of ${\mathbf{x}}$ for each dimension.

Figure 3(a) shows the amount of information estimated by several combinations
of generative and discriminative estimators for a fixed batch size of 64 at
the end of the training procedure. We note that for all analyzed models,
adding a generative component results in improved (i.e., less biased) mutual
information estimates compared to the purely discriminative baselines (in
blue). This is observed for both estimators that underestimate and
overestimate the information and holds for both normal proposals (in green)
and proposals that use $r({\mathbf{y}}|Q_{2}({\mathbf{x}}))$ (orange).
However, it is worth mentioning that $r({\mathbf{y}}|Q_{2}({\mathbf{x}}))$
does not require access to the entropy $H({\mathbf{y}})$. We further indicate
the parts of the total information contributed by
$I_{r}({\mathbf{x}};{\mathbf{y}})$ and
${{\mathrm{KL}}_{f}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}$,
respectively.

Figure 3(b) reports the effect of increasing the batch size on the bias and
variance of the mutual information estimators. The addition of a generative
component has a similar effect to an increased batch size. In particular, for
estimations that suffer from high variance (such as MINE and NWJ) the adding
the PQ generative component is comparable to increasing the batch size by a
factor of 64. Estimators based on InfoNCE that are characterized by lower
variance can benefit from the addition of a generative component to lower
their bias. Further analysis of the effect of the number of samples from the
proposal are reported in Section D.2.

#### Discrete Time Multi-Particle Simulation

Stochastic processes characterize relevant problems in scientific discovery.
For example, it can be of interest in a dynamical system (e.g., fluid dynamics
or molecular dynamics) estimating how much information gets propagated through
time. We simulate a dataset that aims to resemble the characteristics of a
system of particles moving in a fixed energy landscape following discrete-time
overdamped Langevin dynamics (Ermak & Yeh, 1974):
${\mathbf{x}}_{t}={\mathbf{x}}_{t-1}-\epsilon\nabla
U({\mathbf{x}}_{t-1})+\sqrt{2\epsilon/\beta}\bm{\eta},$ where $\nabla
U({\mathbf{x}}_{t})$ refers to the gradient of an energy $U({\mathbf{x}}_{t})$
evaluated at a position ${\mathbf{x}}_{t}$, and $\bm{\eta}$ is time-
independent noise sampled from a unit Normal $\mathcal{N}(\bm{0},\mathbf{1})$.
When the system is in equilibrium, ${\mathbf{x}}_{t}$ follows the Boltzmann
distribution $p({\mathbf{x}}_{t})\propto e^{-\beta U({\mathbf{x}}_{t})}$.
Therefore, it is possible to compute a good approximation of the entropy.
Similarly, the conditional entropy of the transition distribution can be
computed as a function of $\epsilon$ and $\beta$. As a result, in this
simplified system, we can compute a faithful approximation of mutual
information between the positions of a particle at consecutive time steps. We
simulate a collection of five independent particles on a two-dimensional
energy landscape characterized by multiple wells to create a ten-dimensional
trajectory vector ${\mathbf{x}}_{t}$ for which
$I({\mathbf{x}}_{t};{\mathbf{x}}_{t+1})\approx 10.561$ nats. Furthermore, to
introduce spurious correlation, we first concatenate 10-dimensional time-
independent Normal noise and 10 constant dimensions before applying an
invertible non-linear function. This last operation increases the
dimensionality to 30 without effectively changing the value of mutual
information. Further details on the dataset creation procedure can be found in
Section D.3.

To produce the quantization function used for this experiment, we first
project the 30-dimensional feature space onto 10 principal components using
Temporal Independent Component Analysis (TICA) (Molgedey & Schuster, 1994).
TICA ensures that particles that correlate highly through time, have similar
representations. Secondly, we apply a K-nearest neighbors clustering to obtain
$Q({\mathbf{x}})$ for all particles and time-steps of the dataset. The
rationale behind this choice of the quantization function is to make sure that
$Q({\mathbf{x}})$ captures temporal information by grouping those particles.
Thereby, it yields a proposal distribution
$r_{Q}({\mathbf{x}}_{t+1},{\mathbf{x}}_{t})$ that improves
$I({\mathbf{x}}_{t};Q({\mathbf{x}}_{t+1}))$.

Figure 4(a) shows the mutual information result for discriminative (blue) and
hybrid models using PQ as the generative components (orange). Overall, we
observe that including a generative component yields improved mutual
information estimates of the tested estimators. Figure 4(b) shows the effect
of increasing the number of quantization intervals $Q({\mathbf{x}})$ on the
bias-variance trade-off. In this case, this number is provided by the clusters
for the TICA quantization. Increasing the number of clusters results in a
variance reduction for most of the estimators, except for InfoNCE-based
estimators. This is because InfoNCE estimators are characterized by low
variance and high bias, and the increase in variance is due to the estimation
of the PQ generative component. We also notice that the bias for some
estimators tends to increase for large numbers of quantized values. We
hypothesize that this effect is due to the induced scarcity of the samples
from $p({\mathbf{y}}|Q({\mathbf{x}}))$.

##  7 Conclusion

We introduced a hybrid approach for mutual information estimates that
generalizes discriminative and generative methods and combines advantages of
both approaches. On top of that, we propose Predictive Quantization (PQ): a
simple generative method that can be used to improve discriminative
estimators. These contributions analytically yield improved mutual information
estimates with lower variances. Theoretical results were confirmed
experimentally on two challenging mutual information estimation tasks.

#### Limitations and Future Work

Although the hybrid approaches proposed in this work help to address
limitations of generative and discriminative approaches in the literature,
they introduce additional complexity due to the interplay between the two
components. Nevertheless, we believe that simple (or non-parametric) proposals
could be used together with discriminative models to maximize information
between learned representations in future work.

## References

  * Alemi et al. (2016) Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K.  Deep variational information bottleneck.  _CoRR_ , abs/1612.00410, 2016.
  * Bach & Jordan (2002) Bach, F. R. and Jordan, M. I.  Kernel independent component analysis.  _Journal of machine learning research_ , 3(Jul):1–48, 2002.
  * Barber & Agakov (2003) Barber, D. and Agakov, F. V.  The im algorithm: A variational approach to information maximization.  In _NIPS_ , pp. 201–208, 2003.  URL http://papers.nips.cc/paper/2410-information-maximization-in-noisy-channels-a-variational-approach.
  * Belghazi et al. (2018) Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D.  Mutual information neural estimation.  In Dy, J. and Krause, A. (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 531–540. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/belghazi18a.html.
  * Bengio et al. (2013) Bengio, Y., Courville, A., and Vincent, P.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Cover & Thomas (2006) Cover, T. M. and Thomas, J. A.  _Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)_.  Wiley-Interscience, USA, 2006.  ISBN 0471241954.
  * Dinh et al. (2017) Dinh, L., Sohl-Dickstein, J., and Bengio, S.  Density estimation using real NVP.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=HkpbnH9lx.
  * Donsker & Varadhan (1983) Donsker, M. D. and Varadhan, S. S.  Asymptotic evaluation of certain markov process expectations for large time. iv.  _Communications on pure and applied mathematics_ , 36(2):183–212, 1983.
  * Durkan et al. (2019) Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G.  Neural spline flows.  _Advances in neural information processing systems_ , 32, 2019.
  * Ermak & Yeh (1974) Ermak, D. L. and Yeh, Y.  Equilibrium electrostatic effects on the behavior of polyions in solution: polyion-mobile ion interaction.  _Chemical Physics Letters_ , 24(2):243–248, 1974\.
  * Gao et al. (2020) Gao, R., Nijkamp, E., Kingma, D. P., Xu, Z., Dai, A. M., and Wu, Y. N.  Flow contrastive estimation of energy-based models.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pp. 7515–7525. Computer Vision Foundation / IEEE, 2020.  doi: 10.1109/CVPR42600.2020.00754.  URL https://openaccess.thecvf.com/content_CVPR_2020/html/Gao_Flow_Contrastive_Estimation_of_Energy-Based_Models_CVPR_2020_paper.html.
  * Gao et al. (2015) Gao, S., Ver Steeg, G., and Galstyan, A.  Efficient estimation of mutual information for strongly dependent variables.  In _Artificial intelligence and statistics_ , pp. 277–286. PMLR, 2015.
  * Gao et al. (2017) Gao, W., Kannan, S., Oh, S., and Viswanath, P.  Estimating mutual information for discrete-continuous mixtures.  In _NIPS_ , 2017.
  * Gibbs & Su (2002) Gibbs, A. L. and Su, F. E.  On Choosing and Bounding Probability Metrics.  _Interdisciplinary Science Reviews_ , 70(3):419–435, December 2002.  doi: 10.1111/j.1751-5823.2002.tb00178.x.
  * Grathwohl et al. (2019) Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K.  Your classifier is secretly an energy based model and you should treat it like one.  _arXiv preprint arXiv:1912.03263_ , 2019.
  * Guo et al. (2022) Guo, Q., Chen, J., Wang, D., Yang, Y., Deng, X., Huang, J., Carin, L., Li, F., and Tao, C.  Tight mutual information estimation with contrastive fenchel-legendre optimization.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=M-seILmeISn.
  * Gutmann & Hyvärinen (2010) Gutmann, M. and Hyvärinen, A.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pp. 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * Haarnoja et al. (2017) Haarnoja, T., Tang, H., Abbeel, P., and Levine, S.  Reinforcement learning with deep energy-based policies.  In _International conference on machine learning_ , pp. 1352–1361. PMLR, 2017.
  * Hjelm et al. (2018) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y.  Learning deep representations by mutual information estimation and maximization.  _arXiv preprint arXiv:1808.06670_ , 2018.
  * Hjelm et al. (2019) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y.  Learning deep representations by mutual information estimation and maximization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.  URL https://openreview.net/forum?id=Bklr3j0cKX.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Hyvärinen & Dayan (2005) Hyvärinen, A. and Dayan, P.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Hyvärinen & Oja (2000) Hyvärinen, A. and Oja, E.  Independent component analysis: algorithms and applications.  _Neural networks_ , 13(4-5):411–430, 2000.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In Bengio, Y. and LeCun, Y. (eds.), _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.  URL http://arxiv.org/abs/1312.6114.
  * Kingma et al. (2016) Kingma, D. P., Salimans, T., and Welling, M.  Improving variational inference with inverse autoregressive flow.  _CoRR_ , abs/1606.04934, 2016.  URL http://arxiv.org/abs/1606.04934.
  * Kraskov et al. (2004) Kraskov, A., Stögbauer, H., and Grassberger, P.  Estimating mutual information.  _Physical review E_ , 69(6):066138, 2004.
  * MacKay et al. (2003) MacKay, D. J., Mac Kay, D. J., et al.  _Information theory, inference and learning algorithms_.  Cambridge university press, 2003.
  * McAllester & Stratos (2020) McAllester, D. and Stratos, K.  Formal limitations on the measurement of mutual information.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 875–884. PMLR, 2020.
  * Molgedey & Schuster (1994) Molgedey, L. and Schuster, H. G.  Separation of a mixture of independent signals using time delayed correlations.  _Physical review letters_ , 72(23):3634, 1994\.
  * Nair & Hinton (2010) Nair, V. and Hinton, G. E.  Rectified linear units improve restricted boltzmann machines.  In _Icml_ , 2010.
  * Ngiam et al. (2011) Ngiam, J., Chen, Z., Koh, P. W., and Ng, A. Y.  Learning deep energy models.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pp. 1105–1112, 2011.
  * Nguyen et al. (2010) Nguyen, X., Wainwright, M. J., and Jordan, M. I.  Estimating divergence functionals and the likelihood ratio by convex risk minimization.  _IEEE Transactions on Information Theory_ , 56(11):5847–5861, 2010.
  * Palmer et al. (2015) Palmer, S. E., Marre, O., Berry, M. J., and Bialek, W.  Predictive information in a sensory population.  _Proceedings of the National Academy of Sciences_ , 112(22):6908–6913, 2015.
  * Paninski (2003) Paninski, L.  Estimation of entropy and mutual information.  _Neural computation_ , 15(6):1191–1253, 2003\.
  * Poole et al. (2019) Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G.  On variational bounds of mutual information.  In _International Conference on Machine Learning_ , pp. 5171–5180. PMLR, 2019.
  * Ratner et al. (2017) Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., and Ré, C.  Snorkel: Rapid training data creation with weak supervision.  In _Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases_ , volume 11, pp. 269. NIH Public Access, 2017.
  * Rezende & Mohamed (2015) Rezende, D. J. and Mohamed, S.  Variational inference with normalizing flows.  In Bach, F. R. and Blei, D. M. (eds.), _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pp. 1530–1538. JMLR.org, 2015.  URL http://proceedings.mlr.press/v37/rezende15.html.
  * Rhodes et al. (2020) Rhodes, B., Xu, K., and Gutmann, M. U.  Telescoping density-ratio estimation.  _Advances in neural information processing systems_ , 33:4905–4916, 2020.
  * Ryan et al. (2016) Ryan, E. G., Drovandi, C. C., McGree, J. M., and Pettitt, A. N.  A review of modern computational algorithms for bayesian optimal design.  _International Statistical Review_ , 84(1):128–154, 2016.
  * Shannon (1948) Shannon, C. E.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423, 1948.
  * Song & Ermon (2019) Song, J. and Ermon, S.  Understanding the limitations of variational mutual information estimators.  _arXiv preprint arXiv:1910.06222_ , 2019.
  * Song & Ermon (2020) Song, J. and Ermon, S.  Understanding the limitations of variational mutual information estimators.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=B1x62TNtDS.
  * Song & Kingma (2021) Song, Y. and Kingma, D. P.  How to train your energy-based models.  _arXiv preprint arXiv:2101.03288_ , 2021.
  * Song et al. (2020) Song, Y., Garg, S., Shi, J., and Ermon, S.  Sliced score matching: A scalable approach to density and score estimation.  In _Uncertainty in Artificial Intelligence_ , pp. 574–584. PMLR, 2020.
  * Sordoni et al. (2021) Sordoni, A., Dziri, N., Schulz, H., Gordon, G., Bachman, P., and Des Combes, R. T.  Decomposed mutual information estimation for contrastive representation learning.  In _International Conference on Machine Learning_ , pp. 9859–9869. PMLR, 2021.
  * Tishby et al. (2000) Tishby, N., Pereira, F. C. N., and Bialek, W.  The information bottleneck method.  _CoRR_ , physics/0004057, 2000.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.
  * Xiao et al. (2021) Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A.  VAEBM: A symbiosis between variational autoencoders and energy-based models.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=5m3SEczOV8L.
  * Zbontar et al. (2021) Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S.  Barlow twins: Self-supervised learning via redundancy reduction.  In _International Conference on Machine Learning_ , pp. 12310–12320. PMLR, 2021.

##  Appendix A Notation and conventions

  1. 1.

Random variables

We use ${\mathbf{x}}$ to denote random variables, while ${\bm{x}}$ refers to
their realization. We use $p({\bm{x}})$ as a shorthand for
$p({\mathbf{x}}={\bm{x}})$ and $p({\mathbf{x}}|{\bm{y}})$ for
$p({\mathbf{x}}|{\mathbf{y}}={\bm{y}})$.

  2. 2.

Expectations

We use $\mathbb{E}_{{\bm{x}}\sim p({\mathbf{x}})}[f({\bm{x}})]$ to denote the
expected value of $f({\bm{x}})$ with respect to the density $p({\mathbf{x}})$
for both discrete and continuous random variables. We omit the subscript
$\mathbb{E}[f({\bm{x}})]\overset{\operatorname{def}}{=}\mathbb{E}_{{\bm{x}}\sim
p({\mathbf{x}})}[f({\bm{x}})]$ for brevity. When no subscript is specified,
the expectation is considered with respect to the joint density $p$.

  3. 3.

Kullback-Leibler divergence We use
${\mathrm{KL}}(p({\mathbf{x}})||q({\mathbf{x}}))\overset{\operatorname{def}}{=}\mathbb{E}\left[\log\frac{p({\bm{x}})}{q({\bm{x}})}\right]$
to denote the Kullback-Leibler (KL) divergence between the densities
$p({\mathbf{x}})$ and $q({\mathbf{x}})$.

  4. 4.

Mutual information and entropy

We use the notation $I({\mathbf{x}};{\mathbf{y}})$ to indicate the value of
mutual information between the random variables ${\mathbf{x}}_{1}$ and
${\mathbf{y}}$ (see definition in equation 1).

We use $H({\mathbf{x}})\overset{\operatorname{def}}{=}\mathbb{E}[-\log
p({\bm{x}})]$ to denote the entropy of ${\mathbf{x}}$ for both discrete and
continuous (differential entropy) random variables.

##  Appendix B Discriminative Mutual Information estimators

We report a summary of the objectives and parametrized critic architectures
$g_{\phi}({\bm{x}},{\bm{y}})$ for a batch
$({\bm{x}}_{i},{\bm{y}}_{i})_{i=1}^{B}$ of samples from
$p({\mathbf{x}},{\mathbf{y}})$ and with $({\bm{y}}_{i,j}^{\prime})_{j=1}^{K}$
representing $K$ samples from the proposal $r({\mathbf{y}}|{\bm{x}}_{i})$ for
each ${\bm{x}}_{i}$.

  * •

NWJ (Nguyen et al., 2010):

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i})-\frac{1}{e\
BK}\sum_{i=1}^{B}\sum_{j=1}^{K}e^{g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})+1}$
(18)

  * •

MINE (Belghazi et al., 2018)

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i})-\log\frac{1}{BK}\sum_{i=1}^{B}\sum_{j=1}^{K}e^{g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})}$
(19)

A running average of
$\frac{1}{BK}\sum_{i=1}^{B}\sum_{j=1}^{K}e^{g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})+1}$
is used for training.

  * •

InfoNCE (van den Oord et al., 2018)

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i})-\frac{1}{B}\sum_{i=1}^{B}\log\frac{1}{K}\sum_{j=1}^{K}e^{g_{\phi}({\bm{x}}_{i})^{T}{\bm{y}}_{i,j}^{\prime}},$
(20)

with ${\bm{y}}_{i,j}={\bm{y}}_{j}$ and $B=K$.

  * •

JS (Hjelm et al., 2019; Poole et al., 2019) Mutual information estimation is
performed analogously to NWJ, the critic is optimized by minimizing

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}-s^{+}(-g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i}))-\frac{1}{BK}\sum_{i=1}^{B}\sum_{j=1}^{K}-s^{+}(g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})),$
(21)

in which $s^{+}$ corresponds to the softplus function.

  * •

NWJ+InfoNCE (Poole et al., 2019) This estimator uses an interpolation of the
baseline used in NWJ and InfoNCE:

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}\left[g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i})-\log
B({\bm{x}}_{i})\right]-\frac{1}{BK}\sum_{i=1}^{B}\sum_{j=1}^{K}\frac{e^{g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})}}{B({\bm{x}}_{i})}+1,$
(22)

with
$B_{\alpha}({\bm{x}}_{i})\overset{\operatorname{def}}{=}\alpha+(1-\alpha)\frac{1}{K}\sum_{j=1}^{K}e^{g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime})}$.
Throughout our experiments, we use $\alpha=0.5$

  * •

SMILE(Song & Ermon, 2020) Mutual Information is estimated equivalently to MINE
with the exception of value clipping $CLIP$ inside the exponential

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i})-\log\frac{1}{BK}\sum_{i=1}^{B}\sum_{j=1}^{K}e^{CLIP(g_{\phi}({\bm{x}}_{i},{\bm{y}}_{i,j}^{\prime}),\tau,\tau)}.$
(23)

The parameters $\phi$ are updated using the same objective as JS.

Throughout the experiments, we used the value $\tau=5$.

##  Appendix C Proofs

###  C.1 Composing $I({\mathbf{x}};{\mathbf{y}})$ from
$I_{r}({\mathbf{x}};{\mathbf{y}})$ and
${\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}})$

$\displaystyle
I_{r}({\mathbf{x}};{\mathbf{y}})+{\mathrm{KL}}(p({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
$\displaystyle=\mathbb{E}\left[\log\frac{r({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right]+\mathbb{E}\left[\log\frac{p({\bm{x}},{\bm{y}})}{r({\bm{x}},{\bm{y}})}\right]$
$\displaystyle=\mathbb{E}\left[\frac{p({\bm{x}},{\bm{y}})}{p({\bm{x}})p({\bm{y}})}\right]$
$\displaystyle=I({\mathbf{x}};{\mathbf{y}})$ (24)

###  C.2 Variance

We report the proof to show the exponential growth of the variance of the
normalization constant for completeness. Similar proofs can be found in
(McAllester & Stratos, 2020; Song & Ermon, 2020).

$\displaystyle\mathbb{V}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]$
$\displaystyle\geq\mathbb{E}_{r}\left[\left(e^{f({\bm{x}},{\bm{y}})}\right)^{2}\right]-\mathbb{E}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]^{2}$
$\displaystyle=Z^{2}\mathbb{E}_{r}\left[\left(\frac{r({\bm{x}},{\bm{y}})e^{f({\bm{x}},{\bm{y}})}}{r({\bm{x}},{\bm{y}})Z}\right)^{2}\right]-Z^{2}$
$\displaystyle=Z^{2}\iint
r({\bm{x}},{\bm{y}})\left[\left(\frac{q({\mathbf{x}},{\mathbf{y}})}{r({\bm{x}},{\bm{y}})}\right)^{2}-1\right]d{\bm{x}}d{\bm{y}}$
$\displaystyle\overset{\operatorname{def}}{=}Z^{2}{\chi^{2}}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))$
$\displaystyle\geq
Z^{2}\left(e^{{\mathrm{KL}}(q({\mathbf{x}},{\mathbf{y}})||r({\mathbf{x}},{\mathbf{y}}))}-1\right),$
(25)

A proof for the upper bound shown in the last step can be found in (Gibbs &
Su, 2002) (Theorem 5).

###  C.3 An Improved Bound

Consider a set of attainable critics $\mathcal{F}$. When we include at least a
constant function $f_{k}:{\mathbb{X}}\times{\mathbb{Y}}\to\\{k\\}$, where $k$
is a constant, our lower bound is at least as good as a fully generative
approach.

$\displaystyle\max_{(r,f)\in\mathcal{R}\times\mathcal{F}}I_{q}({\mathbf{x}},{\mathbf{y}})$
$\displaystyle=\max_{(r,f)\in\mathcal{R}\times\mathcal{F}}\left[I_{r}({\mathbf{x}},{\mathbf{y}})+{\mathrm{KL}}_{f}(p({\bm{x}},{\bm{y}})||r({\bm{x}},{\bm{y}}))\right]$
(26)
$\displaystyle=\max_{(r,f)\in\mathcal{R}\times\mathcal{F}}\left[I_{r}({\mathbf{x}};{\mathbf{y}})+\mathbb{E}[f({\bm{x}},{\bm{y}})]-\log\mathbb{E}_{r}\left[e^{f({\bm{x}},{\bm{y}})}\right]\right]$
(27)
$\displaystyle\stackrel{{\scriptstyle{f_{k}\in\mathcal{F}}}}{{\geq}}\max_{r\in\mathcal{R}}\left[I_{r}({\mathbf{x}};{\mathbf{y}})+k-\log\mathbb{E}_{r}\left[e^{k}\right]\right]$
(28)
$\displaystyle=\max_{r\in\mathcal{R}}\left[I_{r}({\mathbf{x}};{\mathbf{y}})\right]$
(29)

##  Appendix D Experimental Details

We include additional details on the models and datasets for reproducibility
purposes. A full version of the code and dataset will be published upon
acceptance.

###  D.1 Architectures

Generative and discriminative architectures used for the experiments on the
Normal Mixture dataset consist of MLPs with hidden layers of 256 and 128
hidden units with ReLU activations. The same architecture is used for joint
and separable critic architectures, and for parametrizing learnable
conditional distribution $r_{\theta}({\mathbf{y}}|{\mathbf{x}})$. The
parametrization for the variational distribution
$s_{\phi}(Q({\mathbf{x}})|{\mathbf{y}})$ used in the PQ model uses a simpler
architecture consisting of one hidden layer of 128 units.

###  D.2 Normal Mixture

Each dimension of the samples of ${\mathbf{x}}$ and ${\mathbf{y}}$ are drawn
from a mixture of four correlated Normal distributions with mixing proportions
$\pi=[1/4,1/4,1/4,1/4]$, covariance $\Sigma=\begin{bmatrix}1&0.95\\\
0.95&1\end{bmatrix}$ and mean $\mu_{1}=[\epsilon+\delta,-\epsilon+\delta]$,
$\mu_{2}=[-\epsilon-\delta,\epsilon-\delta]$,
$\mu_{3}=[\epsilon-\delta,-\epsilon-\delta]$ and
$\mu_{4}=[-\epsilon+\delta,\epsilon+\delta]$. The values of bias and variance
reported in the figures are computed on the last epoch of the end of the
training procedure.

###  D.3 Discrete-time Multi-particle Simulation

The trajectories for the discrete-time multi-particle simulation task are
generated using the log-density of a mixture of Normal as an energy function
(Figure 5). For simulation, we use $\epsilon=0.05$ and $\beta=0.3$. Since the
equilibrium distribution generated using discrete timesteps slightly differs
from the Boltzmann distribution, we re-estimate the entropy by binning 2M
samples into a 100$\times$100 grid. Each trajectory is obtained by considering
100.000 consecutive samples after an initial burn-in of 100.000 samples
starting from a fixed initialization to ensure convergence to the equilibrium
distribution. After concatenating the trajectories for the 5 particles, we
concatenate 10 independent samples from a unit Normal distribution and 10
constant zero dimensions to each timestep. Finally, we apply to layers of
randomly initialized affine autoregressive transformations, obtaining a
correlated 30-dimensional vector for each timestep. The values of bias and
variance reported in the figures are computed on the last 10 epochs of the
training procedure to mitigate the effect of the high variance.

Figure 5: Visualization of the 2D energy $U({\mathbf{x}})$ and a portion of
simulated trajectory for one particle used to generate the dataset.

##  Appendix E Additional Experimental Results

Figure 6: Effect of increasing the number of negative samples on mutual
information estimation for a fixed batch-size of 64. Note that the MINE
estimator switches from overestimating ($<4$ negatives) to underestimating
($>4$ negatives).

Figure Figure 6 shows the effect of increasing the number of samples from
$r({\mathbf{y}}|{\mathbf{x}})$ used to estimate the normalization constant for
several discriminative mutual information estimators and their hybrid
counterparts. We can observe a similar effect to the one obtained a batch-size
(Figure 3). The introduction of the PQ generative model has a considerable
effect on reducing the variance of the estimators: using PQ together with the
MINE or NWJ has a similar effect to increasing the number of negatives by a
factor x64.