# PQMass: Probabilistic Assessment of the Quality of
Generative Models using Probability Mass Estimation

Pablo Lemos  Sammy Sharief  Nikolay Malkin  Laurence Perreault-Levasseur
Yashar Hezaveh

###### Abstract

We propose a comprehensive sample-based method for assessing the quality of
generative models. The proposed approach enables the estimation of the
probability that two sets of samples are drawn from the same distribution,
providing a statistically rigorous method for assessing the performance of a
single generative model or the comparison of multiple competing models trained
on the same dataset. This comparison can be conducted by dividing the space
into non-overlapping regions and comparing the number of data samples in each
region. The method only requires samples from the generative model and the
test data. It is capable of functioning directly on high-dimensional data,
obviating the need for dimensionality reduction. Significantly, the proposed
method does not depend on assumptions regarding the density of the true
distribution, and it does not rely on training or fitting any auxiliary
models. Instead, it focuses on approximating the integral of the density
(probability mass) across various sub-regions within the data space.

Machine Learning, ICML

##  1 Introduction

Generative modeling has become an important and ubiquitous topic in machine
learning. The goal of generative modeling is to approximate a distribution
given a set of samples from the distribution. Generative machine learning has
witnessed the development of a succession of methods for distribution
approximation, including variational autoencoders (VAEs, Kingma & Welling,
2013), generative adversarial networks (GANs, Goodfellow et al., 2014),
normalizing flows (Rezende & Mohamed, 2015), and score-based (diffusion)
generative models (Ho et al., 2020b).

With advancements in generative models, evaluating their performance using
rigorous, clearly defined metrics and criteria has become increasingly
essential. Disambiguating true from modeled distributions is especially
pertinent in light of the growing emphasis on AI safety within the community,
as well as in scientific domains where stringent standards of rigor and
uncertainty quantification are needed for the adoption of machine learning
methods. When evaluating generative models, we are interested in three
qualitative properties (Stein et al., 2023; Jiralerspong et al., 2023):
Fidelity refers to the quality and realism of individual outputs generated by
a model. It assesses how indistinguishable each generated sample is from real
data. Diversity pertains to the range and variety of outputs a model can
produce. For example, a model that misses a mode in the training data,
producing no samples of the given mode, has lower diversity. Novelty refers to
the ability of a model to generate new, previously unseen samples that are not
replicas of the training data yet are coherent and meaningful within the
context of the task. Essentially, it means appropriately interpolating the
regions in between the training data in higher-density regions.

There have been two classes of methods for evaluating generative models:
sample-based methods, which compare generated samples to true samples
directly, and likelihood-based methods, which make use of the likelihood of
the data under the model. It has been observed that likelihood-based methods,
such as data negative log-likelihood (NLL), show too much variance to be
useful in practice and are ‘saturated’ on standard benchmarks, i.e., they do
not correlate well with sample fidelity, particularly in high-dimensional
settings (Theis et al., 2015; Nalisnick et al., 2018; Nowozin et al., ; Yazici
et al., 2020; Le Lan & Dinh, 2021). On the other hand, most sample-based
methods cannot measure the combination of fidelity, diversity, and novelty.
For example, the Fréchet Inception Distance (FID; Heusel et al., 2017) and the
Inception Score (IS; Salimans et al., 2016; Sajjadi et al., 2018) measure
fidelity and diversity, but not novelty, while precision and recall (Salimans
et al., 2018) measure only fidelity and diversity, respectively; and
authenticity (Alaa et al., 2022) measures only novelty. Recently, the Feature
Likelihood Score (FLS; Jiralerspong et al., 2023) was proposed as a measure
that captures fidelity, novelty, and diversity. However, FLS essentially
relies on the approximation of the underlying distribution by a Gaussian
mixture model and consequently requires feature extraction and compression for
high-dimensional models using an auxiliary model.

In this work, we propose PQMass, a general statistical framework for
evaluating the quality of generative models that measures the combination of
fidelity, diversity, and novelty and scales well to high dimensions without
the need for dimensionality reduction. PQMass allows us to accurately estimate
the relative probability of a generative model given a test dataset by
examining the _probability mass_ of the data over appropriately chosen
regions.

The main idea motivating PQMass is that given a sub-region of the entire
space, the number of data points lying in the region follows a binomial
distribution with a parameter equal to the region’s true probability mass
(which is unknown and should be marginalized). By extension, given a partition
of the space into regions, the counts of points falling into the regions
follow a multinomial distribution. PQMass uses two-sample tests on these
multinomial distributions’ parameters for prudently selected regions defined
by Voronoi cells. This approach gives us a p-value, measuring the quality of a
generative model (the probability of a model given the training data). It also
allows the comparison of various competing models by analyzing the ratio of
their probabilities.

Since PQMass works with the _integrals_ of the true and estimated probability
density functions over regions, it does not make any assumptions about the
true underlying density field. PQMass is also computationally efficient and
scales well with the dimensionality of the data, allowing us to directly
evaluate generative models of high-dimensional data, such as images, without
the need for feature extraction or compression. Finally, PQMass is general and
flexible, allowing the evaluation of generative models on any type of data,
including images and time series.

Our contributions are the following: In § 2, we introduce the theoretical
framework for PQMass. In § 3, we introduce an algorithmic instantiation of the
framework. In § 4, we present experiments on synthetic and real-world datasets
and compare PQMass with prior methods.

##  2 Theoretical Framework

Figure 1: An illustration of the statistic computed by PQMass. Left: We start
with two sets of samples $\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$ and
$\left\\{y_{i}\right\\}_{i=1...m}\sim q(y)$, shown as red and blue points,
respectively. For a given region $R\subseteq X$, the fraction of samples in
$R$ for each set of samples follows a binomial distribution, with parameters
$n$ and $\mathbb{P}_{p}(R)$ for the red samples, and $m$ and
$\mathbb{P}_{q}(R)$ for the blue samples. We can use this binomial
distribution to estimate the probability that both binomial distributions are
the same. Right: If we can build non-overlapping regions
$\left\\{R_{i}\right\\}_{i=1...n_{R}}$, for example, using a Voronoi
tesselation, the distribution of samples in the regions follows a multinomial
distribution, for each set of samples. PQMass performs hypothesis tests on
equality of the two multinomial distributions to obtain a p-value.

Our problem statement is the following. We have two sets of samples
$\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$ and
$\left\\{y_{i}\right\\}_{i=1...m}\sim q(y)$, where both $p$ and $q$ have
support in some region $X\subseteq\mathbb{R}^{d}$. We are interested in
testing the statistical hypothesis that $p=q$, that is, the two samples came
from the same distribution.

###  2.1 Measuring Equivalence

We recall an elementary fact from the foundations of probability: two
probability distributions $p(x)$ and $q(x)$ over a measurable region
$X\subseteq\mathbb{R}^{d}$ are equivalent if they are equal as measures, i.e.,

$\mathbb{P}_{p}(R)=\mathbb{P}_{q}(R)\qquad\text{$\forall R\subseteq X$, $R$
measurable}.$ (1)

Following this fact, we can measure whether two distributions are equivalent
by estimating, using samples drawn from these distributions, whether the mass
they assign to a large number of finite, arbitrary regions is equal.

We will now introduce a method for estimating this probability. We start with
the following simple observation:

######  Lemma 2.1.

Given a set of samples $\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$, the
probability in a region $R\subseteq X$ can be approximated by the proportion
of samples in $R$:

$\mathbb{P}_{p}(R)=\mathbb{E}_{x\sim p(x)}\left[\mathds{1}\left(x\in
R\right)\right]\approx\frac{1}{n}|\\{i:x_{i}\in R\\}|.$ (2)

This estimate is unbiased, and in the limit of infinite samples, this
approximation becomes exact (i.e., the estimator is consistent).

###### Proof.

We start by defining the random variable $X$ as

$X=\begin{cases}1&\text{if }x\in R\\\ 0&\text{otherwise}\end{cases}$ (3)

We have that

$\mathbb{E}_{x\sim p(x)}\left[X\right]=\mathbb{P}_{p}(R)$ (4)

We can approximate this expectation by the empirical mean of the samples:

$\mathbb{E}_{x\sim
p(x)}\left[X\right]\approx\frac{1}{n}\sum_{i=1}^{n}X_{i}=\frac{1}{n}\sum_{i=1}^{n}\mathds{1}\left(x_{i}\in
R\right)$ (5)

which is the proportion of samples in $R$. ∎

In the following, we will use the notation

$k({\bf x},R)\equiv\sum_{i=1}^{n}\mathds{1}\left(x_{i}\in R\right).$ (6)

Here, $k({\bf x},R)$ is the number of samples of the entire dataset, ${\bf
x}$, in a region $R$. We can approximate the probability mass in a region $R$
by the proportion of samples in ${\bf x}$ that lie in $R$. We can now use this
approximation to measure whether two sets of samples are equivalent, by
measuring whether the proportion of samples in a large number of finite
regions is the same for both sets of samples.

A simple, non-probabilistic method for assessing the equivalence of two
datasets might involve directly comparing their respective proportions.
Nonetheless, this paper will demonstrate a framework for conducting a more
comprehensive analysis. This approach aims to quantify the probability that
the two datasets are equivalent, thereby providing a rigorous statistical
measure of their similarity.

###  2.2 Quantifying the Probability in a Region: A Frequentist Approach

We will now introduce a method for measuring this proportion. First, we notice
the following important property:

######  Lemma 2.2.

Given a set of samples $\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$, the number
of samples in a region $R\subseteq X$ follows a binomial distribution:

$k({\bf x},R)\sim\mathcal{B}\left(n,\mathbb{P}_{p}(R)\right)$ (7)

This means that, given two sets of samples
$\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$ and
$\left\\{y_{i}\right\\}_{i=1...m}\sim q(y)$, we can estimate the value of
$\mathbb{P}_{p}(R)$ and $\mathbb{P}_{q}(R)$, using $\hat{p}_{R}\equiv k({\bf
x},R)/n$ and $\hat{q}_{R}\equiv k({\bf y},R)/n$ as estimators respectively.

We can easily extend this result to any number of non-overlapping regions
$\left\\{R_{i}\right\\}_{i=1...n_{R}}$, by noticing that the number of samples
in each region follows a multinomial distribution:

$\left\\{k({\bf
x},R_{i})\right\\}_{i=1...n_{R}}\sim\mathcal{M}\left(n,\left\\{\mathbb{P}_{p}(R_{i})\right\\}_{i=1...n_{R}}\right).$
(8)

Therefore, given a set of regions $\left\\{R_{i}\right\\}_{i=1...n_{R}}$, we
can estimate the probability that both sets of samples are equivalent by
measuring whether the multinomial distributions are the same. We first discuss
how to approach this problem from a frequentist perspective. We will discuss
the Bayesian approach in the next subsection.

In the frequentist approach, there are multiple ways to measure whether two
multinomial distributions are the same (see, e.g., Anderson et al., 1974;
Zelterman, 1987; Plunkett & Park, 2019; Bastian et al., 2024). In this paper,
we use one of the simplest methods, the Pearson $\chi^{2}$ test (Rao, 1948).
We will explore alternative methods in future work. For reviews on this test,
see (Rao, 2002). We start by defining the expected counts for each region:

$\hat{N}_{x,i}\equiv n\hat{p}_{R_{i}},\qquad\hat{N}_{y,i}\equiv
m\hat{p}_{R_{i}},$ (9)

where:

$\hat{p}_{R_{i}}\equiv\frac{k({\bf x},R_{i})+k({\bf y},R_{i})}{n+m}.$ (10)

We can then define the $\chi^{2}_{\rm{PQM}}$ statistic using the definition of
the Pearson chi-squared:

$\chi^{2}_{\rm{PQM}}\equiv\sum_{i=1}^{n_{R}}\left[\frac{(k({\bf
x},R_{i})-\hat{N}_{x,i})^{2}}{\hat{N}_{x,i}}+\frac{(k({\bf
y},R_{i})-\hat{N}_{y,i})^{2}}{\hat{N}_{y,i}}\right].$ (11)

This statistic follows a $\chi^{2}$ distribution with $n_{R}-1$ degrees of
freedom. Therefore, we can calculate the p-value of the test as

$\text{p-value}(\chi^{2}_{\rm{PQM}})\equiv\int_{-\infty}^{\chi^{2}_{\rm{PQM}}}\chi^{2}_{n_{R}-1}(z)dz.$
(12)

This p-value provides an estimate of the probability that the two sets of
samples are equivalent.

With that, we can summarize this section into the main theoretical result of
this paper:

Given any sampling distribution, or generative model, if two sets of samples
are generated from the same distribution, then the statistic $\chi^{2}_{\rm
PQM}$ follows a chi-square distribution with $n_{R}-1$ degrees of freedom.

We can then use this result to quantify how different the two sets of samples
are.

###  2.3 Quantifying the Probability in a Region: A Bayesian Approach

We can also use a Bayesian approach to quantify the probability in a set of
regions. Once again, we know that the proportion of samples in a set of
regions $\left\\{R_{i}\right\\}_{i=1...n_{R}}$ follows a multinomial
distribution. Therefore, we can use the closed-form likelihood of the
multinomial distribution to calculate the probability of the proportion of
samples in a region $R$ given a set of samples
$\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$:

$\mathcal{L}_{x}\equiv\mathbb{P}(\left\\{x_{i}\right\\}_{i=1...n}|\mathbb{P}_{p},\left\\{R_{i}\right\\}_{i=1...n_{R}})=\\\
{n\choose k({\bf x},R_{i}),...,k({\bf
x},R_{n_{R}})}\prod_{i}^{n_{R}}\mathbb{P}_{p}(R_{i})^{k({\bf x},R_{i})},$ (13)

and similarly for $\mathcal{L}_{y}$. In a Bayesian approach, we can use the
posterior predictive distribution (PPD) to calculate the probability of
finding the number of samples, $k({\bf y},R)$, in a region $R$ given the
number of samples from the other set, $k({\bf x},R)$, in the same region:

$\mathbb{P}(\left\\{y_{i}\right\\}_{i=1...m}|\left\\{x_{i}\right\\}_{i=1...n},\left\\{R_{i}\right\\}_{i=1...n_{R}})=\\\
\int{\mathrm{d}}\lambda_{1}{\mathrm{d}}\lambda_{2}...{\mathrm{d}}\lambda_{n_{R}}\mathbb{P}(\left\\{y_{i}\right\\}_{i=1...m}|\left\\{\lambda_{i}\right\\}_{i=1...n_{R}},\left\\{R_{i}\right\\}_{i=1...n_{R}})\\\
\cdot\mathbb{P}(\left\\{\lambda_{i}\right\\}_{i=1...n_{R}}|\left\\{x_{i}\right\\}_{i=1...n},\left\\{R_{i}\right\\}_{i=1...n_{R}})$
(14)

Assuming a uniform prior $\lambda_{i}\sim\mathcal{U}(0,1)$, we can rewrite the
integrand as a product of two likelihoods:

$\mathbb{P}(\left\\{y_{i}\right\\}_{i=1...m}|\left\\{\lambda_{i}\right\\}_{i=1...n_{R}},\left\\{R_{i}\right\\}_{i=1...n_{R}})=\\\
\mathbb{P}(\left\\{x_{i}\right\\}_{i=1...n}|\left\\{R_{i}\right\\}_{i=1...n_{R}})^{-1}\\\
\cdot\int{\mathrm{d}}\lambda_{1}{\mathrm{d}}\lambda_{2}...{\mathrm{d}}\lambda_{n_{R}}\mathbb{P}(\left\\{y_{i}\right\\}_{i=1...m}|\left\\{\lambda_{i}\right\\}_{i=1...n_{R}},\left\\{R_{i}\right\\}_{i=1...n_{R}})\\\
\cdot\mathbb{P}(\left\\{\left\\{x_{i}\right\\}_{i=1...n}|\lambda_{i}\right\\}_{i=1...n_{R}}\left\\{R_{i}\right\\}_{i=1...n_{R}})$
(15)

Given that we have a closed-form likelihood, it is possible to calculate the
integrand numerically. This is still, however, an integral over variable
$\lambda$ with a dimensionality equal to the number of regions. One can
numerically approximate this integral using Monte Carlo integration or methods
such as nested sampling (Skilling, 2006; Buchner, 2023; Lemos et al., 2023a).
We show an experiment using this Bayesian approach with a low number of
regions in § B, but otherwise, given the numerical cost of this method, we
will focus on the frequentist approach in this paper, and leave the Bayesian
approach for future work.

##  3 Algorithmic Instantiation

Algorithm 1 The frequentist version of our algorithm

1: Input: Two sets of samples $\left\\{x_{i}\right\\}_{i=1...n}\sim p(x)$ and
$\left\\{y_{i}\right\\}_{i=1...m}\sim q(y)$, a distance metric $d:X\times
X\rightarrow\mathbb{R}$, a number of regions $n_{R}$

2: Output: A p-value

3: Procedure:

4: $x^{\rm ref}_{i=1\ldots n_{R}}\sim\left\\{x_{i}\right\\}_{i=1\ldots n}$

5: ${\rm counts}_{x}\leftarrow\\{0\\}_{i=1\ldots n_{R}}$

6: ${\rm counts}_{y}\leftarrow\\{0\\}_{i=1\ldots n_{R}}$

7: for $i=1$ to $n$ do

8: $idx\leftarrow{\rm argmin}_{j=1\ldots n_{R}}d(x_{i},x^{\rm ref}_{j})$

9: ${\rm counts}_{x}[idx]\leftarrow{\rm counts}_{x}[idx]+1$

10: end for

11: for $i=1$ to $m$ do

12: $idx\leftarrow{\rm argmin}_{j=1\ldots n_{R}}d(y_{i},x^{\rm ref}_{j})$

13: ${\rm counts}_{y}[idx]\leftarrow{\rm counts}_{y}[idx]+1$

14: end for

15: $\chi_{\rm PQM}^{2}\leftarrow$ calculate from Eq. 11

16: p-value $\leftarrow$ calculate from Eq. 12

17: return $\chi_{\rm PQM}^{2}$, p-value

We now introduce an algorithmic instantiation of our framework. The key
element in the algorithm is the choice of the regions
$\left\\{R_{i}\right\\}_{i=1...n_{R}}$. We follow Lemos et al. (2023a) in
using the Tests of Accuracy with Random Points (TARP) framework. We combine
definition 7 and remark 2 of (Lemos et al., 2023a) in the following way:

######  Definition 3.1.

Given a distance metric $d:U\times U\rightarrow R$, every pair of points
$\left(x_{c},x_{r}\right)\sim U$ defines a TARP region as:

$\left\\{x\in U\ |\ d(\theta,x_{c})\leq d(x_{r},x_{c})\right\\}$ (16)

In this case, however, we cannot randomly select regions from $U$. The reason
for that is that the derivation of § 2.2 assumes that the different regions do
not overlap, which will not be the case if we randomly select regions from
$U$. Therefore, we need to generate regions that do not overlap. We do this by
noticing the following:

######  Lemma 3.2.

A set of distinct points $\left\\{x^{\rm ref}_{i}\right\\}_{i=1...n}$ defines
a set of non-overlapping TARP regions, through the Voronoi tessellation of the
space $U$.

$R_{i}\equiv\left\\{x\in U\ |\ d(x,x^{\rm ref}_{i})<d(x,x^{\rm
ref}_{j})\forall i\neq j\right\\}$ (17)

i.e., the region $R_{i}$ is defined by the points that are closest to the
reference point that defines that region, $x^{\rm ref}_{i}$, than to any other
reference point.

The $R_{i}$ partition the entire space into non-overlapping regions.111To be
precise, there exists a set of points that are equidistant to two or more
reference points. In this case, we could assign the point to the region of the
reference point with the lowest index. However, this is an insignificant
point, as the boundaries of the regions have measure zero if the points are
drawn from a continuous distribution. We show an illustration of the Voronoi
tessellation in the right panel of Fig. 1, and the frequentist version of our
algorithm in Algorithm 1.

##  4 Experiments

###  4.1 Null Test

Figure 2: Null test. We generate two sets of samples from the same Gaussian
mixture model in $100$ dimensions. We then measure the $\chi^{2}_{\rm PQM}$
value of our test ($n_{R}=100$) and repeat this process $500$ times,
resampling every time. We can see that the $\chi^{2}_{\rm PQM}$ value is
distributed as a chi-squared distribution with $n_{R}-1$ degrees of freedom,
as expected.

We start by testing PQMass on a null test, where we know that the two sets of
samples are equivalent. As our generative model, we use a Gaussian mixture
model in $100$ dimensions, with $20$ components. We then repeat the following
process $200$ times: We generate $5000$ samples from the Gaussian mixture
model and then measure the $\chi^{2}_{\rm PQM}$ value of our test, with
$n_{R}=100$. We show the results in Fig. 2. We can see that the $\chi^{2}_{\rm
PQM}$ value is distributed as a chi-squared distribution with $n_{R}-1$
degrees of freedom, as expected from the discussion of § 2.2.

This test, while simple, is important, as it shows that the proposed method is
not biased towards rejecting the null hypothesis. More importantly, it shows
our main result introduced in § 2.2 in action. We show more such null tests
for complex distributions in § A.

###  4.2 Gaussian Mixture Model

Figure 3:  Gaussian Mixture Model. We show the results for PQMass on the Y
axis, and the number of dropped modes on the X axis. We repeat the comparison
$20$ times, each time changing the reference points, and report the standard
deviation. We can see that the $\chi^{2}_{\rm PQM}$ value increases as we drop
more modes, as expected.

For our second test, we re-use the Gaussian Mixture Model from § 4.1. We use
our test, dropping some of the modes from the Gaussian Mixture Model. For each
number of modes, we first generate $5000$ samples and then repeat our test
$20$ times with $n_{R}=100$, changing the random reference points each time.
We show the results in Fig. 3. We can see that the $\chi^{2}_{\rm PQM}$ value
increases as we drop more modes, as expected. This shows that PQMass can
detect when a generative model is missing modes and, therefore, has poor
fidelity and diversity.

###  4.3 Validation of Sampling Methods

Table 1: Comparison of various sampling methods on a mixture model in $D=2$ and a funnel in $D=10$. We show the results for PQMass and compare them to the Wasserstein distance, the MMD with an RBF kernel, and the Jensen-Shannon divergence.  | Mixture Model ($D=2$) | Funnel ($D=10$)
---|---|---
Model |  $\chi^{2}_{{\rm PQM}}$ ($n_{R}=100$) | $\mathcal{W}_{2}$ | RBF MMD | JSD |  $\chi^{2}_{{\rm PQM}}$ ($n_{R}=100$) | $\mathcal{W}_{2}$ | RBF MMD
MCMC | $1621.06\pm 82.40$ | $8.94$ | $0.0151$ | $0.42$ | $855.40\pm 49.60$ | $17.41$ | $0.0566$
FAB | $678.22\pm 88.43$ | $5.77$ | $0.0038$ | $0.39$ | $96.69\pm 13.52$ | $22.08$ | $0.0032$
GGNS | $109.32\pm 12.36$ | $3.2$ | $0.0011$ | $0.35$ | $472.99\pm 18.00$ | $32.20$ | $0.0358$
Figure 4:  Differences in various sample-based metrics as a function of the
dimensionality of the data. We use a mixture of $10$ equally weighted
Gaussians with varying numbers of dimensions. For each dimensionality,
calculate the value of both metrics when comparing samples from the same
distribution (blue) and from different distributions where one of them is
missing one mode (orange), repeating the test various times to get an estimate
of the noise. Ideally, we want our sample-based metrics to be able to detect
that a mode has been dropped. Figure 5:  Computational cost of various sample-
based metrics, as a function of the dimensionality of the data. For the same
experiment as in Fig. 4, we show the time required to calculate the value of
each metric, both as a function of the dimensionality of the data (left) and
the number of samples (right).

In this section, we show how we can use PQMass to study the performance of
sampling algorithms as long as we have access to true samples from the
probability distribution. Sampling methods are often evaluated on synthetic
tasks, such as sampling from a Gaussian mixture model, for which we can
generate true samples. Sample-based metrics that are commonly used to evaluate
sampling methods include the Wasserstein distance, the Sinkhorn distance
(Cuturi, 2013), the Kulback-Leibler divergence (Kullback & Leibler, 1951), the
Jensen-Shannon divergence (Lin, 1991), among others. While these sample-based
metrics work well on low-dimensional settings, they are not well suited for
high-dimensional settings, as they require the calculation of integrals over
the space of samples, or a density estimation step, which is not feasible in
high dimensions.

On the other hand, the proposed method is only based on calculating distances
between points and can, therefore, be easily applied to high-dimensional
settings. Furthermore, the method does not make any assumptions about the
underlying distribution and, therefore, can be applied to any distribution as
long as we have access to true samples.

First, we compare PQMass to other sample-based metrics on a low-dimensional
task, where we can use other traditionally used sample-based metrics. We use
the two-dimensional Gaussian mixture model used in Midgley et al. (2022),
which has been used for benchmarking various sampling methods. We generate
samples using Flow annealed importance sampling bootstrap (FAB, Midgley et
al., 2022) Markov Chain Monte Carlo (Metropolis et al., 1953), for which we
use the emcee implementation (Foreman-Mackey et al., 2013), and Gradient-
Guided Nested Sampling (GGNS, Skilling, 2006; Lemos et al., 2023b). For each
method, we generate $10,000$ samples and compare them with $10,000$ samples
from the true distribution. We calculate the Wasserstein distance, the Mean
Maximum Discrepancy (MMD, Gretton et al., 2012) with a radial basis function
(RBF) kernel and the Jensen-Shannon divergence (after performing a step of
kernel density estimation); and compare them to the $\chi_{\rm PQM}^{2}$ value
of our test, with $n_{R}=100$. For PQMass, we repeat this process $20$ times,
each time changing the reference points, and report the standard deviation. We
show the results in Table 1. We can see that PQMass is in good agreement with
the other sample-based metrics and that the $\chi^{2}_{\rm PQM}$ value is
correlated with the other sample-based metrics.

We then repeat this experiment on a higher dimensional task: Neal’s funnel
distribution (Neal, 2003), a 10-dimensional distribution that is commonly used
to benchmark sampling methods. We repeat the same experiment as before, except
we do not use the Jensen-Shannon divergence, as it requires a density
estimation step, which is not feasible in high dimensions. We see that PQMass
is in good agreement with the MMD, while the Wasserstein distance follows a
different pattern. We show samples from each model in § D, which show that,
visually, PQMass captures the best-performing sampling methods.

There are, of course, multiple other methods for sampling, both deep learning
based (such as Zhang & Chen, 2021; Lahlou et al., 2023; Richter et al., 2023,
amongst many others) and non-deep learning based (e.g., Del Moral et al.,
2006; Hoffman et al., 2014), that could have been added to Table 1.
Furthermore, hyperparameter tuning could be used to improve the performance of
each sampling method. The purpose of this table is not to compare different
sampling methods but rather to show that PQMass can be used to evaluate
sampling methods.

Finally, we study the scaling of PQMass with the dimensionality of the data.
We use a mixture of $10$ equally weighted Gaussians with varying numbers of
dimensions. For each dimensionality, we generate $5,000$ samples and compare
them with $5,000$ samples from the same distribution. We calculate the
$\chi^{2}_{\rm PQM}$ value of our test, with $n_{R}=100$, as well as other
sample-based metrics. We then repeat the experiment, dropping one mode and
comparing the value of each metric. We do not compute the Kulback-Leibler or
Jensen-Shannon divergence, as they require a density estimation step, which is
not feasible in high dimensions. We show the results in Fig. 4. We can see
that while $\mathcal{W}_{2}$ and linear MMD get increasingly noisier as the
dimensionality increases, PQMass remains stable. MMD with an RBF kernel can
also consistently highlight the difference between the two, however the
computational cost of radial basis MMD scales far worse with dimensionality
than PQMass. We show this in Fig. 5. We can see that the computational cost of
PQMass scales much better with both dimensionality and number of samples than
the Wasserstein and the MMD with an RBF kernel. MMD with a linear kernel is
the cheapest computationally, but as shown in Fig. 4, it is too noisy to be
usable in complex problems.

###  4.4 Time Series

Figure 6: Time Series. We show the results for PQMass on the Y-axis and the
amplitude of the signal on the X-axis. We repeat the comparison $5000$ times,
each time generating a time series with $A=0$, and a time series with $A\neq
0$. We can see that the $\chi^{2}_{\rm PQM}$ value increases as $A$ grows, as
expected. We also show the values of $\chi^{2}_{\rm PQM}$ corresponding to the
$3\sigma$ and $5\sigma$ significance levels of detection as black dashed
lines.  Figure 7: Examples of time series with $A=0$ (left), and with
$A\approx 0.12$ (right). PQMass can detect the signal with $5\sigma$
significance, a signal that is invisible to the naked eye (at least, to those
of the authors).

For our next experiment, we show the flexibility of PQMass by applying it to a
very different type of dataset: Time series data. We design the following
experiment: We observe a noisy time series of fixed length, and we want to
know if there is an underlying signal hiding in the noise. We generate the
time series as follows: We generate the signal as:

$y(t)=A\cdot\cos(t)+\eta(t),$ (18)

where $A$ is the amplitude of the signal and $\eta(t)$ is i.i.d. Gaussian
noise with zero mean and unit variance. If $A=0$, then the time series is just
noise. If $A\neq 0$, then there is a signal hiding in the noise. For each
observation, we generate $100$ datapoints between $t=0$ and $t=10$. We then
repeat the following process $5000$ times: We generate a time series with
$A=0$ and a time series with $A\neq 0$. We then compare the two time series
using PQMass, with $n_{R}=100$. We show the results in Fig. 6 for varying
values of $A$. We can see that the $\chi^{2}_{\rm PQM}$ value increases as $A$
grows, as expected. The plot also the values of $\chi^{2}_{\rm PQM}$
corresponding to the $3\sigma$ and $5\sigma$ significance levels of detection.
We see that, for this experiment, we can detect the signal with $5\sigma$
significance for $A\approx 0.12$, a signal that is invisible to the naked eye.
We show an example of a time series with this amplitude, compared to one
without signal in Fig. 7.

This experiment serves to show the versatility of PQMass. Because we make no
assumptions about the underlying distribution, we can apply PQMass to any type
of data as long as we have access to samples. Detecting signals in noisy time
series is a common problem in multiple disciplines, such as astronomy (Zackay
et al., 2021; Aigrain & Foreman-Mackey, 2023), finance (Chan, 2004; Sezer et
al., 2020), and anomaly detection (Ren et al., 2019; Shaukat et al., 2021).
Existing methods rely on assumptions about the underlying distribution.
PQMass, on the other hand, can detect that the observed signal is not
consistent with samples of random noise with no assumptions on the generative
process. We leave the application of PQMass to these domains for future work.

###  4.5 Training a Generative Model

In this section, we track the value of our metric as we train two generative
models, a variational autoencoder (VAE; Kingma & Welling, 2013) and a
denoising diffusion model (Ho et al., 2020a; Song et al., 2021) with
hyperparameters detailed in § E. We train on the
MNIST222http://yann.lecun.com/exdb/mnist/ dataset and compare generated
samples with the MNIST test set. After each epoch of training is completed, we
generate $2000$ samples and compute $\chi^{2}_{PMQ}$ with $n_{R}=100$. We
execute this experiment over the first $100$ epochs to highlight the early
stages of training for the models and repeat the experiment twice for
robustness. We see in Fig. 8 that as training continues, the metric begins to
stabilize with fluctuations due to the unique samples generated at each epoch.
For the VAEs, we see that the samples can capture high-level structures, but
they struggle to capture the complex features. Whereas the diffusion models,
the generated samples are similar to the true distribution as seen by their
low $\chi^{2}_{PQM}$ values, which plateau at around $\chi^{2}_{PQM}\sim 99$
as expected.

Figure 8: After each training epoch for the different models is completed, we
generate samples and compute the $\chi^{2}$ metric to assess the quality
against the true distribution. We perform this experiment twice, Run 1 and Run
2 for each model respectively, and see $\chi^{2}_{PMQ}$ values agree with each
other even as they generate unique samples each time. The diffusion model can
achieve $\chi^{2}_{PMQ}$ near ideal ($\chi^{2}_{PMQ}$ equal to the number of
degrees of freedom $n_{R}-1$, in this case $99$) , indicating its ability to
learn the underlying distribution.

The use of the metric for the generative model gives the ability to track the
quality of the model’s performance relative to the true distribution,
preserving the integrity of the underlying data structure and without any loss
of dimensionality. Therefore, given samples from the true distribution (i.e. a
test set), this approach can be used as a method to indicate performance
throughout the training of a generative model.

###  4.6 Image Datasets

Figure 9: Correlation of human error rate in identifying real images with
various sample-based metrics (results of prior methods taken from Jiralerspong
et al. (2023)). The result for PQMass is the $\chi^{2}_{\rm PQM}$ value
averaged over 20 samples. PQMass estimates the fidelity and diversity of
generative models well despite not relying on a feature representation, unlike
FLS and FID.

In this section, we test PQMass on image datasets. We use samples from Stein
et al. (2023), which contains samples from various image datasets, and various
generative models. We focus on the CIFAR-10
dataset333https://www.cs.toronto.edu/~kriz/cifar.html. Following (Stein et
al., 2023), we first use the human error rate as a measure of the fidelity and
diversity of the generative models. We compare generated samples from each of
the models, to the test data for CIFAR-10. We repeat the comparison $20$
times, each time varying the reference points. We show the results for
CIFAR-10 in Fig. 9. We find that our chi-squared values strongly correlate
with the human error rate, indicating that PQMass can measure the fidelity and
diversity of generative models.

One key strength of PQMass is its scaling to high-dimensional problems. Unlike
alternative methods such as FID and FLS, PQMass does not require using a
feature extractor to reduce the dimensionality of the problem. Therefore, we
can use PQMass to compare generative models on high-dimensional problems and
do not risk potential biases introduced by the feature extractor.

###  4.7 Novelty

Figure 10: Novelty. PQMass (Y axis) when a fraction of the generated samples
are replaced by samples taken from the training set (X axis). We repeat the
comparison $20$ times, each time changing the reference points, and report the
standard deviation. We show the value of $\chi^{2}_{\rm PQM}$ in the left
panel, for the test (red) and train (blue), and the p-value for the training
data on the left. We see that, as memorization increases, the gap between the
train and test $\chi^{2}_{\rm PQM}$ increases and the p-value goes down.

One important aspect of generative models is their novelty: We want to know if
the generative model generates samples that are novel or if it simply copies
the training data. While PQMass can measure the fidelity and diversity of
generative models, it cannot directly measure their novelty. There are two
approaches we can follow to detect memorization: The first one is to follow
Jiralerspong et al. (2023), in looking at the generalization gap for PQMass,
i.e., how our metric changes when comparing the generated samples to the
training data and when comparing the generated samples to the validation data.
The second one is to look at the value of $\chi^{2}_{\rm PQM}$ on the training
data directly. Indeed, we know that a value that is too low is indicative of
overfitting. For a large number of regions (i.e., degrees of freedom), we know
the chi-square distribution is approximately Gaussian. Therefore, we can use
the symmetry of this distribution (as shown in Fig. 2, to get a p-value that
penalizes suspiciously low values of $\chi^{2}_{\rm PQM}$ by considering the
mirror reflection of the chi-square value around the maximum:

$\text{p-value}_{{\rm
overfit}}(\chi^{2}_{\rm{PQM}})\equiv\int_{-\infty}^{2n_{R}-\chi^{2}_{\rm
PQM}}\chi^{2}_{n_{R}-1}(z)dz.$ (19)

To study this, we repeat the ‘copycat’ experiment of (Jiralerspong et al.,
2023): We pick one of the models used in § 4.6, and repeat our test, replacing
some fraction of the generated samples with samples from the training data. We
then calculate the $\chi^{2}_{\rm PQM}$ value of our test, first comparing the
generated samples to the training data and then comparing the generated
samples to the validation data.

We show the results for PFGMPP in Fig. 10. The left panel shows that the gap
between $\chi^{2}_{\rm PQM}$ when compared to the training data and
$\chi^{2}_{\rm PQM}$ value when comparing to the validation data is larger the
more samples we replace. The right panel shows the two-sided p-value, which
clearly goes down as the number of copied samples increases. This shows that
both of these methods (generalization gap and two-sided p-value) can be used
to detect memorization in generative models.

##  5 Conclusions

In this paper, we have introduced a new method for quantifying the probability
that two sets of samples are drawn from the same probability distribution.
PQMass is based on comparing the probability mass of the two sets of samples
in a set of non-overlapping regions. It relies only on the calculation of
distances between points and, therefore, can be applied to high-dimensional
problems. It does not require training or fitting any models. Furthermore, it
does not require any assumptions about the underlying distribution and,
therefore, can be applied to any type of data as long as we have access to
true samples. We have shown that PQMass can be used to evaluate sampling
methods and track the performance of generative models as they train.

We have shown the performance of PQMass on a variety of synthetic tasks, as
well as on comparing sampling methods, comparing generative models of images,
and detecting hidden signals in time series. Given the versatility and low
computational cost of PQMass, it can serve as a valuable tool for evaluating
the quality and performance of generative models and sampling methods.

## Acknowledgments

This research was made possible by a generous donation by Eric and Wendy
Schmidt with the recommendation of the Schmidt Futures Foundation. This work
is also supported by the Simons Collaboration on ”Learning the Universe”. The
work is in part supported by computational resources provided by Calcul Quebec
and the Digital Research Alliance of Canada. Y.H. and L.P. acknowledge support
from the Canada Research Chairs Program, the National Sciences and Engineering
Council of Canada through grants RGPIN-2020- 05073 and 05102, and the Fonds de
recherche du Québec through grants 2022-NC-301305 and 300397. N.M.
acknowledges funding from CIFAR, Genentech, Samsung, and IBM.

We thank Adam Coogan, Sebastian Wagner-Carena, Joey Bose, Gauthier Gidel, Alex
Tong, Alexandra Volokhova and Marco Jiralespong for very inspiring
discussions, and feedback on early versions of this work. We also thank Marco
Jiralespong for sharing useful data.

## Reproducibility Statement

An implementation of PQMass, along with notebooks to reproduce the results
from the experiments will be released upon publication.

## Impact Statement

As a robust and efficient algorithm for measuring discrepancies between sets
of samples in high dimensions, PQMass can be used in the assessment of ML
models as well as in scientific or industrial applications. The ability of
PQMass to distinguish machine-generated and naturally occurring data
distributions has implications for the detection of synthetic content, which
is especially important in light of the growing emphasis on the interaction of
AI systems with intellectual property, human rights, and social welfare.

## References

  * Aigrain & Foreman-Mackey (2023) Aigrain, S. and Foreman-Mackey, D.  Gaussian process regression for astronomical time series.  _Annual Review of Astronomy and Astrophysics_ , 61:329–371, 2023.
  * Alaa et al. (2022) Alaa, A., Breugel, B., Saveliev, E., and Schaar, M.  How faithful is your synthetic data.  _Sample-Level Metrics for Evaluating and Auditing Generative Models. arXiv_ , 2022.
  * Anderson et al. (1974) Anderson, D. A., McDonald, L. L., and Weaver, K. D.  Tests on categorical data from the unionintersection principle.  _Annals of the Institute of Statistical Mathematics_ , 26:203–213, 1974.
  * Bastian et al. (2024) Bastian, P., Dette, H., and Koletzko, L.  Testing equivalence of multinomial distributions—a constrained bootstrap approach.  _Statistics & Probability Letters_, 206:109999, 2024.
  * Buchner (2023) Buchner, J.  Nested sampling methods.  _Statistic Surveys_ , 17:169–215, 2023.
  * Chan (2004) Chan, N. H.  _Time series: applications to finance_.  John Wiley & Sons, 2004.
  * Cuturi (2013) Cuturi, M.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26, 2013.
  * Del Moral et al. (2006) Del Moral, P., Doucet, A., and Jasra, A.  Sequential monte carlo samplers.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 68(3):411–436, 2006.
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., and Goodman, J.  emcee: the mcmc hammer.  _Publications of the Astronomical Society of the Pacific_ , 125(925):306, 2013.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gretton et al. (2012) Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. (2020a) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 6840–6851. Curran Associates, Inc., 2020a.  URL https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf.
  * Ho et al. (2020b) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020b.
  * Hoffman et al. (2014) Hoffman, M. D., Gelman, A., et al.  The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.  _J. Mach. Learn. Res._ , 15(1):1593–1623, 2014\.
  * Jiralerspong et al. (2023) Jiralerspong, M., Bose, A. J., and Gidel, G.  Feature likelihood score: Evaluating generalization of generative models using samples.  _arXiv preprint arXiv:2302.04440_ , 2023.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-Encoding Variational Bayes.  _arXiv e-prints_ , art. arXiv:1312.6114, December 2013.  doi: 10.48550/arXiv.1312.6114.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kullback & Leibler (1951) Kullback, S. and Leibler, R. A.  On information and sufficiency.  _The annals of mathematical statistics_ , 22(1):79–86, 1951.
  * Lahlou et al. (2023) Lahlou, S., Deleu, T., Lemos, P., Zhang, D., Volokhova, A., Hernández-Garcıa, A., Ezzine, L. N., Bengio, Y., and Malkin, N.  A theory of continuous generative flow networks.  In _International Conference on Machine Learning_ , pp. 18269–18300. PMLR, 2023.
  * Le Lan & Dinh (2021) Le Lan, C. and Dinh, L.  Perfect density models cannot guarantee anomaly detection.  _Entropy_ , 23(12):1690, 2021.
  * Lemos et al. (2023a) Lemos, P., Coogan, A., Hezaveh, Y., and Perreault-Levasseur, L.  Sampling-based accuracy testing of posterior estimators for general inference.  _arXiv preprint arXiv:2302.03026_ , 2023a.
  * Lemos et al. (2023b) Lemos, P., Malkin, N., Handley, W., Bengio, Y., Hezaveh, Y., and Perreault-Levasseur, L.  Improving gradient-guided nested sampling for posterior inference.  _arXiv preprint arXiv:2312.03911_ , 2023b.
  * Lin (1991) Lin, J.  Divergence measures based on the shannon entropy.  _IEEE Transactions on Information theory_ , 37(1):145–151, 1991.
  * Metropolis et al. (1953) Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E.  Equation of state calculations by fast computing machines.  _The journal of chemical physics_ , 21(6):1087–1092, 1953.
  * Midgley et al. (2022) Midgley, L. I., Stimper, V., Simm, G. N., Schölkopf, B., and Hernández-Lobato, J. M.  Flow annealed importance sampling bootstrap.  _arXiv preprint arXiv:2208.01893_ , 2022.
  * Nalisnick et al. (2018) Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B.  Do deep generative models know what they don’t know?  _arXiv preprint arXiv:1810.09136_ , 2018.
  * Neal (2003) Neal, R. M.  Slice sampling.  _The annals of statistics_ , 31(3):705–767, 2003\.
  * (29) Nowozin, S., Cseke, B., Tomioka, R., and GAN.  Training generative neural samplers using variational divergence minimization.  In _Proc. Adv. Neural Inf. Process. Syst._ , pp. 271–279.
  * Plunkett & Park (2019) Plunkett, A. and Park, J.  Two-sample test for sparse high-dimensional multinomial distributions.  _Test_ , 28:804–826, 2019.
  * Rao (2002) Rao, C.  Karl pearson chi-square test the dawn of statistical inference.  _Goodness-of-fit tests and model validity_ , pp. 9–24, 2002.
  * Rao (1948) Rao, C. R.  Tests of significance in multivariate analysis.  _Biometrika_ , 35(1/2):58–79, 1948.
  * Ren et al. (2019) Ren, H., Xu, B., Wang, Y., Yi, C., Huang, C., Kou, X., Xing, T., Yang, M., Tong, J., and Zhang, Q.  Time-series anomaly detection service at microsoft.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pp. 3009–3017, 2019.
  * Rezende & Mohamed (2015) Rezende, D. and Mohamed, S.  Variational inference with normalizing flows.  In _International conference on machine learning_ , pp. 1530–1538. PMLR, 2015.
  * Richter et al. (2023) Richter, L., Berner, J., and Liu, G.-H.  Improved sampling via learned diffusions.  _arXiv preprint arXiv:2307.01198_ , 2023.
  * Sajjadi et al. (2018) Sajjadi, M. S., Bachem, O., Lucic, M., Bousquet, O., and Gelly, S.  Assessing generative models via precision and recall.  _Advances in neural information processing systems_ , 31, 2018.
  * Salimans et al. (2016) Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Salimans et al. (2018) Salimans, T., Zhang, H., Radford, A., and Metaxas, D.  Improving gans using optimal transport.  _arXiv preprint arXiv:1803.05573_ , 2018.
  * Sezer et al. (2020) Sezer, O. B., Gudelek, M. U., and Ozbayoglu, A. M.  Financial time series forecasting with deep learning: A systematic literature review: 2005–2019.  _Applied soft computing_ , 90:106181, 2020.
  * Shaukat et al. (2021) Shaukat, K., Alam, T. M., Luo, S., Shabbir, S., Hameed, I. A., Li, J., Abbas, S. K., and Javed, U.  A review of time-series anomaly detection techniques: A step to future perspectives.  In _Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 1_ , pp. 865–877. Springer, 2021.
  * Skilling (2006) Skilling, J.  Nested sampling for general bayesian computation.  2006\.
  * Song et al. (2021) Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=PxTIG12RRHS.
  * Stein et al. (2023) Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, J. E. T., and Loaiza-Ganem, G.  Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.  _arXiv preprint arXiv:2306.04675_ , 2023.
  * Szegedy et al. (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2818–2826, 2016.
  * Theis et al. (2015) Theis, L., Oord, A. v. d., and Bethge, M.  A note on the evaluation of generative models.  _arXiv preprint arXiv:1511.01844_ , 2015.
  * Yazici et al. (2020) Yazici, Y., Foo, C.-S., Winkler, S., Yap, K.-H., and Chandrasekhar, V.  Empirical analysis of overfitting and mode drop in gan training.  In _2020 IEEE International Conference on Image Processing (ICIP)_ , pp. 1651–1655. IEEE, 2020.
  * Zackay et al. (2021) Zackay, B., Venumadhav, T., Roulet, J., Dai, L., and Zaldarriaga, M.  Detecting gravitational waves in data with non-stationary and non-gaussian noise.  _Physical Review D_ , 104(6):063034, 2021.
  * Zelterman (1987) Zelterman, D.  Goodness-of-fit tests for large sparse multinomial distributions.  _Journal of the American Statistical Association_ , 82(398):624–629, 1987.
  * Zhang & Chen (2021) Zhang, Q. and Chen, Y.  Path integral sampler: a stochastic control approach for sampling.  _arXiv preprint arXiv:2111.15141_ , 2021.

##  Appendix A Null Tests

Figure 11: Null tests, similar to the one shown in Fig. 2, but for two
different distributions: Samples from MNIST (left) and a VAE trained on MNIST
(right). The main result of our paper holds for samples from any distribution
as long as we are comparing two sets of independent samples from the same
distribution. Note that, on MNIST, the fit is not perfect due to the limited
number of data points.

The main result of this paper, shown in § 2.2, was shown in practice in § 4.1
for a high-dimensional mixture of Gaussians. In this section, we show that it
works in even more complex and high-dimensional distributions. First, we test
it on the MNIST data: Using the MNIST observations themselves as samples from
some underlying probability distribution. We put together the MNIST train and
test data, leading to $70,000$ images. We then split them into two subsets of
$35,000$ images each. To repeat the null test of § 4.1, we need independent
samples at every iteration (note that this is different from the error bars
reported in the rest of the experiments, which just arise from repeating with
random tesselations, but always the same samples). Therefore, for each
iteration, we take $1,000$ images from each dataset, and perform the PQM test
with $n_{R}=100$, leading to 35 data points. We plot their histogram in the
left panel of Fig. 11. While noisy, due to the very few data points, we see
that the histogram does follow a chi-squared distribution with $n_{R}-1$
degrees of freedom, as expected.

For our second experiment, we use one of the MNIST generative models described
in § 4.5. In this case, we can generate as many samples as we want. Because we
are comparing the generative model to itself, the similarity of the generated
samples to the MNIST train or test data is irrelevant for this particular
test.

##  Appendix B Bayesian Experiment

Figure 12: Log-probability of various models (translationally shifted by $l$)
given a test dataset from a ground truth distribution. This probability is
calculated using the Bayesian method described in Eq. 15. The ground truth
distribution corresponds to $l=0$.

In this experiment, we numerically test the validity of the Bayesian method
described in Eq. 15. This involved integrating the product of two multinomial
distributions to determine the probability of a model given the training data.
The setup mirrors that of the previous Gaussian Mixture Model experiment: we
created a ’ground-truth’ distribution, a Gaussian Mixture Model in 100
dimensions with 20 components. The means and variances of these components
were randomly selected. From this mixture, we drew 5000 samples to form the
test set.

We then introduced a shift to this distribution along the diagonal direction
(characterized by a vector of ones in 100 dimensions) multiplied by a scalar
magnitude $l$. The resulting models, parameterized by $l$, were considered as
potential generative models, aligning with the ground truth when $l=0$. Note
that this is an arbitrary choice to construct a family of competing models.

Three non-overlapping regions were then defined by selecting three random
points and constructing a Voronoi tessellation. We formed a regularly spaced
grid over the values of $\lambda_{1}$ and $\lambda_{2}$ (representing the
probability mass in the first two regions), ensuring that $\lambda_{3}$ was
chosen to allow the combination of the three values to be appropriately
normalized.

We drew samples for each model (defined by different $l$ values on a regularly
spaced grid) and calculated the model’s probability with Eq. 15. This was
achieved by computing the appropriately normalized multinomial distributions
for the test data and the model data within the three regions and then
integrating their product. This integration was done by summing across all
grid points in the probability mass and multiplying by the volume of each grid
point’s probability.

Fig. 12 shows the results: the left panel illustrates the log probability,
while the right panel presents the (un-normalized) probability of the models
as a function of the single parameter $l$. As anticipated, the correct model
($l=0$) has a high probability, with the probability dropping as $l$ diverges
from this value. Employing a greater number of smaller regions could enhance
the constraining power; however, this would require a sample-based integration
method over the unknown $\lambda$ values.

##  Appendix C Ablation Study

Model | Default | $n_{R}=50$ | $n_{R}=200$ | L1 Distance | Inception
---|---|---|---|---|---
ACGAN-Mod | $12407\pm 10$ | $10410\pm 808$ | $14125\pm 995$ | $12336\pm 944$ | $7891\pm 1009$
LOGAN | $713\pm 72$ | $408\pm 69$ | $1180\pm 95$ | $728\pm 108$ | $5750\pm 535$
BigGAN-Deep | $268\pm 30$ | $135\pm 16$ | $498\pm 36$ | $238\pm 32$ | $615\pm 74$
iDDPM-DDIM | $238\pm 22$ | $143\pm 17$ | $470\pm 41$ | $244\pm 31$ | $533\pm 78$
MHGAN | $234\pm 39$ | $123\pm 19$ | $470\pm 48$ | $194\pm 18$ | $628\pm 64$
StyleGAN-XL | $230\pm 13$ | $119\pm 15$ | $477\pm 23$ | $203\pm 23$ | $199\pm 19$
StyleGAN2-ada | $207\pm 26$ | $128\pm 25$ | $417\pm 48$ | $197\pm 23$ | $259\pm 30$
PFGMPP | $177\pm 22$ | $108\pm 18$ | $335\pm 28$ | $175\pm 16$ | $239\pm 28$
Table 2: Ablation study for CIFAR-10. We repeat the experiment of § 4.6,
varying the number of reference points, as well as changing the distance
metric to L1. We show the results for the frequentist version of PQMass,
sorted in order of decreasing $\chi^{2}_{\rm PQM}$ for the default version. We
find that the results are robust to the choice of the number of reference
points and the distance metric. Any observed differences are within the
standard deviation of the experiment.

In this section, we study the effect of varying the two hyperparameters of our
experiment: The number of reference points $n_{R}$, and the distance metric
(which in all experiments in the main text is L2). We repeat the experiment on
CIFAR10 described in § 4.6, varying the number of reference points, as well as
changing the distance metric to L1. We also show the results when using the
Inception-v3 feature extractor (Szegedy et al., 2016), as done in previous
work such as (Jiralerspong et al., 2023). We find that the order of the models
is robust to the choice of the number of reference points and the distance
metric, as well as to the use of a feature extractor. Any observed differences
are within the standard deviation of the experiment. We show the results in
Table 2.

##  Appendix D Samples for Validation

Figure 13: Samples from our the various sampling methods introduced in § 4.3,
and their corresponding PQMass values.

In § 4.3, we showed the performance of PQMass, comparing samples from various
sampling algorithms to true distribution samples. Fig. 13 shows the samples
from each algorithm and the underlying distribution. For the Gaussian Mixture
Model (top), we see that MCMC is missing a mode, while in FAB the samples are
too noisy; for Neal’s funnel (bottom), we see that FAB’s samples look
indistinguishable from true samples, while GGNS look similar but more spread
out, and GFN and MCMC fail to model this distribution. In both cases, the
results from $\chi^{s}_{\rm PQM}$ correlate well with the similarity to true
samples we can see by eye.

##  Appendix E Generative Model Hyperparameters

In this section, we detail the hyper-parameters used for the respective models
in § 4.5. We first note that we use the MNIST data with the channel set to 1,
since it is grayscale, and the spatial dimensions are $28\times 28$. For the
variational autoencoder (VAE), the batch size is defined to be 512. For the
encoder, the first layer maps from 784 dimensions to 512, followed by a second
layer, reducing it further to 256 dimensions. The latent space was represented
with 20 dimensions, as determined by the output of the subsequent layers. The
Decoder architecture was the same, progressively reconstructing the data from
the 20-dimensional latent space back to the original 784-dimensional space. We
utilize the Adam optimizer with the learning rate initially set to 0.001. We
also include a scheduler to reduce the learning rate every 50 steps by 0.1.
For the denoising diffusion model, we use the package
score_models444https://github.com/AlexandreAdam/torch_score_models. Our model
utilizes the NCSN++ architecture with Variance Preserving Stochastic
Differential Equation (SDE). The batch size is defined to be 256 with a
learning rate of 0.001 and exponential moving average decay of 0.999.