arXiv

Li et al.

[*]Corresponding author<EMAIL_ADDRESS>
0Year 0Year 0Year

#  Sharp detection of low-dimensional structure in probability measures via
dimensional logarithmic Sobolev inequalities

Matthew T.C. Li  Center for Computational Science and Engineering, MIT, USA
Tiangang Cui  School of Mathematics and Statistics, University of Sydney,
Australia  Fengyi Li  Center for Computational Science and Engineering, MIT,
USA  Youssef Marzouk  Center for Computational Science and Engineering, MIT,
USA  Olivier Zahm  Université Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK,
38000, Grenoble, France

(2024; 2024; Date; Date; Date)

###### Abstract

Identifying low-dimensional structure in high-dimensional probability measures
is an essential pre-processing step for efficient sampling. We introduce a
method for identifying and approximating a target measure $\pi$ as a
perturbation of a given reference measure $\mu$ along a few significant
directions of $\mathbb{R}^{d}$. The reference measure can be a Gaussian or a
nonlinear transformation of a Gaussian, as commonly arising in generative
modeling. Our method extends prior work on minimizing majorizations of the
Kullback–Leibler divergence to identify optimal approximations within this
class of measures. Our main contribution unveils a connection between the
_dimensional_ logarithmic Sobolev inequality (LSI) and approximations with
this ansatz. Specifically, when the target and reference are both Gaussian, we
show that minimizing the dimensional LSI is equivalent to minimizing the KL
divergence restricted to this ansatz. For general non-Gaussian measures, the
dimensional LSI produces majorants that uniformly improve on previous
majorants for gradient-based dimension reduction. We further demonstrate the
applicability of this analysis to the squared Hellinger distance, where
analogous reasoning shows that the dimensional Poincaré inequality offers
improved bounds.

###### keywords:

Bayesian inference, dimensional logarithmic Sobolev inequality, dimensional
Poincaré inequality, feature detection, gradient-based dimension reduction,
reaction coordinates.

##  1 Introduction

Sampling from a given probability measure $\pi$ on $\mathbb{R}^{d}$ becomes
increasingly challenging as the dimension $d$ increases [39, 38, 2, 35, 41,
40]. Efficient sampling algorithms therefore often rely on exploiting inherent
low-dimensional structure present in $\pi$. Our motivation lies in identifying
such a structure where $\pi$ is close, in a sense we make precise below, to
another known probability measure $\mu$ on $\mathbb{R}^{d}$ which is itself
easy to sample from. Typically, $\mu$ takes the form of a product measure, a
Gaussian measure, or a nonlinear transformation of it, such as those used in
GANs or other generative models [26]. Depending on the application, $\mu$ can
be either given (_e.g._ , via expert knowledge) or $\mu$ can constructed in
some tractable class of measure (_e.g._ , by minimizing some distance from
$\pi$ to $\mu$).

In many realistic scenarios, $\pi$ is close to $\mu$ in that both measures
essentially differ only along a few significant dimensions of
$\mathbb{R}^{d}$. This suggests that $\pi$ can be effectively approximated by
a probability measure $\widetilde{\pi}$ defined as

$\mathrm{d}\widetilde{\pi}(x)\propto\ell_{r}(U_{r}^{\top}x)\mathrm{d}\mu(x),$
(1.1)

where $\ell_{r}:\mathbb{R}^{r}\rightarrow\mathbb{R}_{+}$ is a low-dimensional
function of $r\ll d$ variables and $U_{r}\in\mathbb{R}^{d\times r}$ a matrix,
both of which need to be determined. We refer to $\widetilde{\ell}_{r}$ as the
_profile function_ and to $x\mapsto U_{r}^{\top}x$ as the _linear features_ of
$\widetilde{\pi}$. Such structured approximation — which we interchangeably
describe as _dimension reduction_ for the probability measure $\pi$ — is
particularly relevant in at least the following contexts:

  * •

In Bayesian inverse problems [44, 31], $\pi$ can be the posterior measure
defined as $\mathrm{d}\pi(x)\propto\ell(x)\mathrm{d}\mu(x)$, where $\mu$
denotes the prior measure of the model parameters $x\in\mathbb{R}^{d}$, and
$\ell$ the likelihood function associated with observed data. The
approximation (1.1) replaces the likelihood $\ell$ with a _ridge function_
$x\mapsto\widetilde{\ell}_{r}(U_{r}^{\top}x)$, where the linear features
$U_{r}$ thus capture low-dimensionality in the _prior-to-posterior update_ of
a Bayesian inverse problem (that is, the directions of the parameter $x$ for
which data are most informative of).

  * •

In generative modeling, the probability density $\mu$ corresponding to
empirical samples $\\{X_{i}\\}_{i=1}^{n_{\mathrm{train}}}$ in $\mathbb{R}^{d}$
is modeled as the push-forward of a lower-dimensional measure $\nu$ in
$\mathbb{R}^{\kappa}$ with $\kappa\ll d$ latent coordinates. In other words,
$\mu=T_{\\#}\nu$, where the function $T:\mathbb{R}^{\kappa}\to\mathbb{R}^{d}$
is learned from samples [26]. It is of further interest to consider
_conditional sampling_ with a likelihood model specified in the nominal $d$
dimensional parameter space [37] — this is equivalent to sampling the Bayesian
posterior $\mathrm{d}\pi(z)=(\ell\circ T)(z)\mathrm{d}\nu(z)$ in the latent
space. The approximation (1.1) replaces the composed likelihood $\ell\circ T$
with a ridge function that captures low-dimensional prior-to-posterior
updates, further complementing the dimensionality reduction brought on by the
latent space representation.

  * •

In molecular dynamics, $\mathrm{d}\pi(x)\propto\exp(-\beta V(x))\mathrm{d}x$
can be a Boltzmann distribution associated with a molecular system with energy
potential $x\mapsto V(x)$ with respect to the Lebesgue measure. Such
distributions are typically multimodal, where each mode corresponds to a
physical conformation. If a measure $\widetilde{\pi}$ as in (1.1) approximates
well $\pi$ with $\mu$ being a unimodal measure (_e.g._ Gaussian), then
features $U_{r}$ have given a physical interpretation as _reaction
coordinates_ which captures the multimodality of $\pi$. Finding good reaction
coordinates is an active research field [36].

With some abuse of notation, we denote the measures in (1.1) by
$\widetilde{\pi}(U_{r},\ell_{r})$. A natural idea for determining optimal
approximations within this class is to consider minimizing an Amari
$\alpha$-divergence [3]

$\min_{\begin{subarray}{c}U_{r}\in\mathbb{R}^{d\times r}\\\
U_{r}^{\top}U_{r}=I_{r}\end{subarray}}\,\min_{\ell_{r}:\mathbb{R}^{r}\to\mathbb{R}_{+}}D_{\alpha}(\pi\,||\,\widetilde{\pi}(U_{r},\ell_{r})),$
(1.2)

where choosing $\alpha\in\mathbb{R}$ interpolates the Kullback-Leibler
divergence ($\alpha=1$), the squared Hellinger distance ($\alpha=1/2$), the
$\chi^{2}$ divergence ($\alpha=2$), and their reverse formulations. For any
fixed $U_{r}$, the optimal profile function $\ell_{r}^{*}=\ell_{r}^{*}(U_{r})$
depends on $\alpha$ and $U_{r}$, but is known in closed-form, see e.g. [34,
Thm 1]. Problem (1.2) thus reduces to optimizing
$D_{\alpha}(\pi\,||\,\widetilde{\pi}(U_{r},\ell_{r}^{*}(U_{r})))$ for the
features $U_{r}$. Nevertheless, this optimization requires access to
(gradients of the) normalizing constants of
$\widetilde{\pi}(U_{r},\ell_{r}^{*}(U_{r}))$ and $\pi$, which becomes
computationally intractable for higher dimensional settings.

To circumvent this problem, Zahm et al. [48] proposed to _majorize_ (1.2) with
bounds of the form

$\min_{\ell_{r}:\mathbb{R}^{r}\to\mathbb{R}_{+}}D_{\alpha}(\pi\,||\,\widetilde{\pi}(U_{r},\ell_{r}))\leq\mathcal{J}_{\alpha}(U_{r}).$
(1.3)

The function $U_{r}\mapsto\mathcal{J}_{\alpha}(U_{r})$, which implicitly
depends on $\pi$, should be easier to compute numerically and to minimize
compared to the left-hand side of (1.3). Although minimizing this surrogate
loss yields features $U_{r}^{*}$ which may be sub-optimal compared to the
global minimizer of the $\alpha$-divergence, this procedure furnishes
$\widetilde{\pi}(U_{r}^{*},\ell_{r}^{*}(U_{r}^{*}))$ with _computationally
certifiable_ approximation error. Accordingly, one can then select $r$ to
achieve any desired error tolerance
$D_{\alpha}(\pi\,||\,\widetilde{\pi}(U_{r}^{*},\ell_{r}^{*}(U_{r}^{*})))\leq\epsilon$
by ensuring $\mathcal{J}_{\alpha}(U_{r}^{*})\leq\epsilon$. In many instances
it suffices to choose $r\ll d$ to achieve $\epsilon\ll 1$.

The construction of these majorizations hinges on functional inequalities
satisfied by the reference measure $\mu$, and only requires the Radon-Nikodym
derivative $\mathrm{d}\pi/\mathrm{d}\mu$ to be a smooth function. The seminal
contribution of Zahm et al. [48] related dimension reduction with the
Kullback-Leibler (KL) divergence to the _logarithmic Sobolev inequality_ (LSI,
[28]), while afterwards [20, 17, 25] related the squared Hellinger distance to
the _Poincaré inequality_. Both constructions are generalized in [34], which
related each Amari $\alpha$-divergence to a corresponding _$\phi$ -Sobolev
inequality_, see [18, 6]. Remarkably, in all these works the minimizers
$U_{r}^{*}$ of $U_{r}\mapsto\mathcal{J}_{\alpha}(U_{r})$ are identical, and
are given by the matrix containing the dominant eigenvectors of

$H(\pi||\mu)=\mathbb{E}_{\pi}\left[\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}\mu}\right)\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}\mu}\right)^{\top}\right],$
(1.4)

which is the _Fisher information matrix of $\pi$ relative to $\mu$_. While
these functional inequalities seemingly suggest that the relative Fisher
information features are quasi-optimal for all $\alpha$, it remains unclear
whether better majorants can be obtained for specific $\alpha$ using more
specialized functional inequalities.

Our main contribution is to show that the _dimensional logarithmic Sobolev
inequality_ [10, 22] permits us to obtain better majorants for (and only for)
the KL divergence corresponding to $\alpha=1$. Assuming $\mu$ to be the
standard Gaussian measure in $\mathbb{R}^{d}$, this functional inequality
improves upon the LSI by explicitly accounting for the dimension $d$, and
formally recovers the latter in the limit $d\to\infty$. In our Theorem 4, we
show how to exploit the dimensional LSI in order to obtain both a majorant
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ and a minorant
$\mathcal{J}_{\mathrm{KL}}^{\uparrow}(\cdot)$ of the KL divergence so that

$\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})\leq\min_{\ell_{r}:\mathbb{R}^{r}\to\mathbb{R}_{+}}D_{\mathrm{KL}}(\pi\,||\,\widetilde{\pi}(U_{r},\ell_{r}))\leq\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r}).$
(1.5)

(We suppress their implicit dependence on $\pi$ for notational levity.) While
the minorant $\mathcal{J}_{\mathrm{KL}}^{\uparrow}(\cdot)$ is not used to
construct $U_{r}$, it provides a useful certificate of the sharpness of
inequality (1.5) at minimizers of the majorant. Notably, the majorant
$\mathcal{J}^{\downarrow}_{\mathrm{KL}}(\cdot)$ satisfies
$\mathcal{J}^{\downarrow}_{\mathrm{KL}}(U_{r})\leq\mathcal{J}_{1}(U_{r})$ for
all $U_{r}$ and thus uniformly improves upon the bound (1.3) obtained in [48].
The critical points of $\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ and
$\mathcal{J}_{1}(\cdot)$, however, do not necessarily coincide.

Heuristically, the motivation for pursuing this new choice of functional
inequality stems from the observation that the LSI is satisfied by ‘infinite
dimensional’ distributions (this is implied by standard tensorization
arguments, see e.g. [8]). Since our intended application is dimension
_reduction_ , intuitively the dimensional LSI should be the more appropriate
tool. In fact, we discover that the dimensional LSI _instrinically_ captures
dimension reduction for probability measures: our Proposition 7 shows that the
dimensional majorant $\mathcal{J}^{\downarrow}_{\mathrm{KL}}(U_{r})$ is
equivalent to the KL divergence (1.5) whenever $\pi$ is a Gaussian measure.
When $\pi$ corresponds to a non-Gaussian measure, we show that the majorant in
(1.5) is strictly tighter than the majorant obtained from the LSI, with the
difference being exponentially better in certain instances.

Evaluating and optimizing the majorant
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ requires computing

$\displaystyle H(\pi)$
$\displaystyle=\mathbb{E}_{\pi}\left[\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)^{\top}\right]\quad\text{and}\quad
M(\pi)=\mathbb{E}_{\pi}[XX^{\top}],$ (1.6)

where $H(\pi)$ is the Fisher information matrix of $\pi$, relative to the
Lebesgue measure $\mathrm{d}x$ of $\mathbb{R}^{d}$, and $M(\pi)$ is the second
moment matrix of $\pi$. Evaluating the minorant
$\mathcal{J}_{\mathrm{KL}}^{\uparrow}(\cdot)$ requires in addition the
computation of $m(\pi)=\mathbb{E}_{\pi}[X]$, the mean of the measure $\pi$.
However, unlike for the LSI majorant $\mathcal{J}_{1}(\cdot)$, there is no
closed-form expression for the global minimizer of
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$. Nevertheless, we observe
numerically that first order gradient descent methods converge to the same
minimizer irrespective of the initialization. We therefore conjecture — but
cannot prove — that $\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ is
_benignly non-convex_ [42] on the Grassmann manifold
$\\{U_{r}\in\mathbb{R}^{d\times r}:U_{r}^{\top}U_{r}=I_{r}\\}$, and we leave
this as a problem of independent interest left for future work.

We also consider adapting the reference measure $\mu$ to be the best Gaussian
approximation to $\pi$, with mean $m(\pi)$ and covariance $C(\pi)$, given by
$\mu=\mathcal{N}(m(\pi),\,C(\pi))$. In Theorem 8 we show that this choice,
combined with the dimensional LSI, leads to a majorization whose global
minimizer is known in closed form. Intriguingly, this bound suggests that
relative deviations between $\pi$ and its best Gaussian approximation are best
captured by the relative changes between the Fisher information matrix
$H(\pi)$ and the precision matrix $C(\pi)^{-1}$, in that

$\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{r}^{\downarrow}))\leq\frac{1}{2}\sum_{k=r+1}^{d}\ln(\lambda_{k}(H(\pi),C(\pi)^{-1})),$
(1.7)

where $\lambda_{k}(A,B)$ is the $k$-th largest eigenvalue of the generalized
eigenvalue problem $Av=\lambda Bv$, and the columns of $U_{r}^{\downarrow}$
contain the first $r$ generalized eigenvectors.

When considering Bayesian inverse problems specifically, [21] shows that the
LSI can be used to determine features $V_{r}^{*}$ along which the posterior
$\pi_{X\mid Y}$ differs most from the prior $\mu_{X}$ for the _average_
realization of data $Y$. In Theorem 9 we show that the dimensional LSI further
improves on these bounds. This recovers the same features in [21], with
$V_{r}^{*}$ corresponding to the leading eigenvectors of
$H_{\mathrm{df}}(\pi_{X\mid Y}):=\mathbb{E}_{X,Y}[\nabla_{x}\ln\pi_{x\mid
y}\nabla_{x}\ln\pi_{x\mid y}^{\top}]$, but offers the significantly improved
certificate

$\mathbb{E}_{Y}\left[\operatorname{D_{\mathrm{KL}}}(\pi_{X\mid
Y}\,||\,\widetilde{\pi}_{X\mid
Y}(V_{r}^{*})\right]\leq\frac{1}{2}\sum_{k=r+1}^{d}\ln(1+\lambda_{k}(H_{\mathrm{df}}(\pi_{X\mid
Y})),$ (1.8)

where $\lambda_{k}(\cdot)$ denotes the $k$-th largest eigenvalue of
$H_{\mathrm{df}}(\pi_{X\mid Y})$. Practically, (1.8) implies that
significantly fewer features are necessary to attain a certifable error level
than previously believed. The form of the certificate (1.8) also highlights a
connection to the _expected information gain_ (EIG) appearing in optimal
experimental design [1, 29]. The application of functional inequalities to
optimal experimental design is also a topic of recent interest [33, 16].

Lastly, we also explore structured approximations in (1.2) with the squared
Hellinger distance corresponding to $\alpha=1/2$. Motivated by the same
heuristic considerations behind exploring the dimensional LSI, we show in
Theorem 13 that the _dimensional Poincaré inequality_ proposed in [7] permits
us to obtain improved majorizations over the bounds obtained from the standard
Poincaré inequality. However, the form of this improved majorization is
implicit, which is difficult to optimize directly. Instead, we consider its
use in obtaining improved certificates for the optimal features $U_{r}^{*}$.
Intriguingly, leveraging this improvement requires the use of _lower bounds_
on the approximation error, which presents significant computational
challenges. We leave addressing these issues, as well as potential
applications of the dimensional Poincaré to more general ridge-based
regression (see e.g. [12, 47]), to future work.

##  2 Dimension Reduction with the KL Divergence

###  2.1 Certifiable bounds using the logarithmic Sobolev inequality

We recall some properties of the (KL divergence) optimal profile function from
[48, §2.1]. For any matrix $U_{r}\in\mathbb{R}^{d\times r}$ with $r\leq d$,
and with $\widetilde{\pi}$ as in (1.1), the minimizer of
$\ell_{r}\mapsto\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi})=\int\log(\frac{\mathrm{d}\pi}{\mathrm{d}\widetilde{\pi}})\mathrm{d}\pi$
is the conditional expectation

$\ell^{\mathrm{KL}}_{r}(\theta_{r})=\mathbb{E}_{X\sim\mu}\left[\left.\frac{\mathrm{d}\pi}{\mathrm{d}\mu}(X)\,\right|\,U_{r}^{\top}X=\theta_{r}\right].$
(2.1)

We denote by
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})=\widetilde{\pi}^{\mathrm{KL}}(\cdot\mid
U_{r})$ the resulting probability measure given by

$\mathrm{d}\widetilde{\pi}^{\mathrm{KL}}(x\mid
U_{r})\propto\ell^{\mathrm{KL}}_{r}(U_{r}^{\top}x)\mathrm{d}\mu(x),$ (2.2)

Let us emphasize that, by construction, the law of $U_{r}^{\top}\widetilde{X}$
and of $U_{r}^{\top}X$ are the same for any $U_{r}$, where
$\widetilde{X}\sim\widetilde{\pi}^{\mathrm{KL}}_{r}(U_{r})$ and $X\sim\pi$. In
other words, the KL optimal profile (2.1) ensures that the marginal laws of
$\widetilde{\pi}^{\mathrm{KL}}_{r}(U_{r})$ and $\pi$ are the same along the
linear feature $U_{r}$. Using the above notations, problem (1.2) becomes

$\min_{\begin{subarray}{c}U_{r}\in\mathbb{R}^{d\times r}\\\
U_{r}^{\top}U_{r}=I_{r}\end{subarray}}\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r})).$
(2.3)

In the following we assume that $\mu=\mathcal{N}(0,I_{d})$ is the standard
normal measure. In order to derive a majorant for
$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}))$,
and we employ _logarithmic Sobolev inequality_ (LSI).

######  Proposition 1 (Gaussian LSI).

The entropy $\textsc{Ent}_{\mu}(f)=\int f\ln f\mathrm{d}\mu-\int
f\mathrm{d}\mu\ln\int f\mathrm{d}\mu$ of any smooth positive function
$f:\mathbb{R}^{d}\to\mathbb{R}_{+}$ with respect to the standard Gaussian
$\mu=\mathcal{N}(0,I_{d})$ satisfies

$\frac{1}{2\int f\mathrm{d}\mu}\,\left\|\int\nabla
f\mathrm{d}\mu\right\|_{2}^{2}\leq\textsc{Ent}_{\mu}(f)\leq\frac{1}{2}\int\|\nabla\ln
f\|_{2}^{2}f\mathrm{d}\mu.$ (2.4)

We refer to the upper bound as the _logarithmic Sobolev inequality (LSI)_ ,
and to the lower bound as the _reverse logarithmic Sobolev inequality (reverse
LSI)_.

The logarithmic Sobolev inequality first appeared in the seminal paper of
Gross [28] — we are unaware of the original reference for the reverse
inequality. Both bounds are formulated for general Markov diffusion operators
in [8, Ch. 5], which we translate in Appendix C.1 to obtain the above
proposition. Note the choice of $f=\mathrm{d}\pi/\mathrm{d}\mu$ in (2.4)
permits us to bound the divergence
$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\mu)$ following

$\frac{1}{2}\left\|\mathbb{E}_{\pi}\left[\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}\mu}\right)\right]\right\|_{2}^{2}\leq\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\mu)\leq\frac{1}{2}\mathbb{E}_{\pi}\left[\left\|\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}\mu}\right)\right\|_{2}^{2}\right],$

The next theorem further motivates the connection between dimension reduction
for the KL divergence and the logarithmic Sobolev inequality, and we leave the
proof to Appendix D.1. While this strategy was originally proposed in [48,
Corr. 2.10], the formulation of the lower bound is a new contribution.

######  Theorem 2.

Consider the probability measure
$\mathrm{d}\pi(x)\propto\ell(x)\mathrm{d}\mu(x)$ for some smooth
$\ell:\mathbb{R}^{d}\to\mathbb{R}_{+}$ and standard Gaussian
$\mu=\mathcal{N}(0,I_{d})$. Then, for any matrix $U_{r}\in\mathbb{R}^{d\times
r}$ with $r\leq d$ orthonormal columns, the measure
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})$, as in (2.2), satisfies

$\frac{1}{2}\|(I_{d}-U_{r}U_{r}^{\top})m(\pi)\|_{2}^{2}\leq\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}))\leq\frac{1}{2}\operatorname{tr\,}\left((I_{d}-U_{r}U_{r}^{\top})H(\pi||\mu)\right),$
(2.5)

where $m(\pi)=\mathbb{E}_{\pi}[X]$ is the mean of $\pi$ and
$H(\pi||\mu)=\mathbb{E}_{X\sim\pi}[\nabla\ln\ell(X)\nabla\ln\ell(X)^{\top}]$
is the Fisher information matrix of $\pi$ relative to $\mu$.

Minimizing the right-hand side of (2.5) corresponds corresponds to
maximimizing $U_{r}\mapsto\operatorname{tr\,}(U_{r}^{\top}H(\pi||\mu)U_{r})$
and yields the matrix $U_{r}^{*}$ whose columns contain the $r$ leading
eigenvectors of $H(\pi||\mu)$, see [48]. This is computationally tractable to
obtain, requiring only (an empirical estimate of) the relative Fisher
information matrix. The approximation of $\pi$ by
$\widetilde{\pi}^{\mathrm{KL}}(U_{r}^{*})$ is therefore furnished with the
error certificates

$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}^{*}))\leq\frac{1}{2}\sum_{k=r+1}^{d}\lambda_{k}(H(\pi||\mu)),$
(2.6)

where $\lambda_{k}(H)$ denotes the $k$-th largest eigenvalue of $H(\pi||\mu)$.
While $U_{r}^{*}$ may not be the globally optimal minimum of (2.3), this
result _certifies_ that its approximation error is no larger than
$\frac{1}{2}\sum_{k=r+1}^{d}\lambda_{k}(H(\pi||\mu))$, which can be made
arbitrarily small as $r\to d$. When the spectrum of $H$ decays rapidly, it
suffices to choose rank $r\ll d$ to obtain an accurate approximation. The
lower bound (2.5) provides a guarantee of the smallest achievable
approximation error, which may also inform the practitioner’s choice of $r$.

###  2.2 Improved bounds using dimensional inequalities

We now introduce a functional inequality which is uniformly sharper than the
LSI for Gaussian measures. We defer its proof to Appendix C.2. In the
following, we employ the notation $v^{\otimes 2}=vv^{T}$.

######  Proposition 3 (Dimensional Gaussian LSI).

For standard Gaussian $\mu=\mathcal{N}(0,I_{d})$ on $\mathbb{R}^{d}$, and for
any smooth function $f:\mathbb{R}^{d}\to\mathbb{R}_{+}$ such that $\int
f\mathrm{d}\mu=1$, we have the _dimensional logarithmic Sobolev_ and the
_reverse dimensional logarithmic Sobolev_ inequalities

$\displaystyle\textsc{Ent}_{\mu}\left(f\right)$
$\displaystyle\leq{\frac{1}{2}\int\|x\|_{2}^{2}f\mathrm{d}\mu-\frac{d}{2}+\frac{1}{2}\ln\det\left(\int\left(\nabla\ln
f-x\right)^{\otimes 2}f\mathrm{d}\mu\right),}$ (2.7)
$\displaystyle\textsc{Ent}_{\mu}(f)$
$\displaystyle\geq\frac{1}{2}\int\|x\|_{2}^{2}f\mathrm{d}\mu-\frac{d}{2}-\frac{1}{2}\ln\det\left(\int
x^{\otimes 2}\,f\mathrm{d}\mu-\left(\int xf\mathrm{d}\mu\right)^{\otimes
2}\right).$ (2.8)

The Gaussian dimensional LSI was first proposed by Bakry and Ledoux [10, Prop.
2], but the formulations in Proposition 3 are attributed to Dembo [22]111We
were made aware of this reference from [24], but could not obtain a copy
ourselves., who strengthened their proof technique in a natural manner (see
Appendix C.2). By concavity $\ln\det(S)\leq\operatorname{tr\,}S-d$, which
shows that (2.7) and (2.8) are uniformly tighter than the bounds in
Proposition 1. Due to the explicit dependence on dimension, however, these
inequalities do not tensorize. Note that the constants in Proposition 3 are
sharp for functions given by (translations of) Gaussian kernels [10, §2], just
as with the LSI.

######  Remark 1.

We are only aware of the following classes of measures satisfying the
dimensional LSI in the Euclidean geometry: Gaussian measures (with possibly
singular covariances), push-forwards of Gaussian measures by Lipschitz
functions, and perturbations of Gaussians measures in the sense of Holley-
Stroock [30]. Formulations of the dimensional logarithmic Sobolev inequalities
exist for measures on other manifolds, but we leave the interested reader to
e.g., [8, Thm 6.7.3] for further details.

Improved bounds for the KL divergence between $\pi$ and its approximation
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})$ are obtained using the dimensional LSI;
we leave the proof to Appendix D.2.

######  Theorem 4.

Consider the standard Gaussian measure $\mu=\mathcal{N}(0,I_{d})$ on
$\mathbb{R}^{d}$. Then, for any matrix $U_{r}\in\mathbb{R}^{d\times r}$ with
$r\leq d$ orthonormal columns, the measure
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})$, as in (2.2), satisfies

$\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})\leq
D_{\mathrm{KL}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}))\leq\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r}),$
(2.9)

where

$\displaystyle\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})=\frac{\operatorname{tr\,}(M(\pi))-d+\ln\det\left(H(\pi)\right)}{2}-\frac{\operatorname{tr\,}(U_{r}^{\top}M(\pi)U_{r})-r+\ln\det(U_{r}^{\top}H(\pi)^{-1}U_{r})}{2},$
(2.10)
$\displaystyle\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})=\frac{\operatorname{tr\,}(M(\pi))-d-\ln\det(C(\pi))}{2}-\frac{\operatorname{tr\,}(U_{r}^{\top}M(\pi)U_{r})-r-\ln\det(U_{r}^{\top}C(\pi)^{-1}U_{r})}{2}.$
(2.11)

Here
$H(\pi)=\mathbb{E}_{\pi}[\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}x})\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}x})^{\top}]$
denotes the Fisher information matrix of $\pi$ and
$M(\pi)=\mathbb{E}_{\pi}[XX^{\top}]$ is the second moment matrix of $\pi$, and
$C(\pi)=M(\pi)-\mathbb{E}_{\pi}[X]\mathbb{E}_{\pi}[X]^{\top}$ is the
covariance matrix of $\pi$.

######  Remark 2.

Let $U_{\perp}$ denote any orthogonal completion to $U_{r}$ such that
$[U_{r},U_{\perp}]\in\mathbb{R}^{d\times d}$ is unitary. Using standard matrix
determinant identities (see Appendix B) we have the equivalent representations

$\displaystyle\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})$
$\displaystyle=\frac{\operatorname{tr\,}(U_{\perp}^{\top}M(\pi)U_{\perp})-(d-r)+\ln\det(U_{\perp}^{\top}H(\pi)U_{\perp})}{2}$
(2.12) $\displaystyle\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})$
$\displaystyle=\frac{\operatorname{tr\,}(U_{\perp}^{\top}M(\pi)U_{\perp})-(d-r)-\ln\det(U_{\perp}^{\top}C(\pi)U_{\perp})}{2},$
(2.13)

which do not require computing matrix inverses to evaluate. Interestingly, the
above expressions resemble, but fundamentally differ from, the _log-
determinant divergence_ (or _Burg divergence_)
$D_{\ell,d}(X,Y)=\operatorname{tr\,}(XY^{-1})-d-\ln\det(XY^{-1})$ between two
symmetric positive definite matrices $X,Y\in\mathbb{R}^{d\times d}$, see [32].

Just as the dimensional LSI improves on the LSI, the inequalities in Theorem 4
improve on those from Theorem 2, meaning that

$\frac{1}{2}\|(I_{d}-U_{r}U_{r}^{\top})m(\pi)\|_{2}^{2}\leq\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})\qquad\text{and}\qquad\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})\leq\frac{1}{2}\operatorname{tr\,}\left((I_{d}-U_{r}U_{r}^{\top})H(\pi||\mu)\right).$

To demonstrate this, it suffises to apply
$\ln\det(S)\leq\operatorname{tr\,}(S)-d$ on (2.12) and (2.13) and observe that
the relative Fisher information matrix $H(\pi||\mu)$ with
$\mu=\mathcal{N}(0,I_{d})$ decomposes as

$\displaystyle H(\pi||\mu)$
$\displaystyle\overset{\eqref{eq:defH}}{=}\mathbb{E}_{\pi}\left[\left(\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)+x\right)\left(\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)+x\right)^{\top}\right]$
$\displaystyle\overset{\eqref{eq:defHandM}}{=}H(\pi)+\left(\int\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)x^{\top}+x\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\right)^{\top}\mathrm{d}\pi\right)+M(\pi)$
$\displaystyle=H(\pi)-2I_{d}+M(\pi),$ (2.14)

where the last step is obtained by an integration by parts. Theorem 4 sharpens
the error certificates from Theorem 2 for all features by an _exponential_
factor (the “logdet” term). Minimizing
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ thus results in features with
tighter certifiable approximation guarantees compared to minimizing the
majorant in (2.5), as shown in Appendix A.

We show in the following proposition that the majorization (2.10)
monotonically decreases over sequences of nested features
$\textrm{span}(U_{r})\subseteq\textrm{span}(U_{r+1})$. This implies
$\min_{U_{r+1}}\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r+1})\leq\min_{U_{r}}\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})$,
and thus one always obtains features with improved error certificates when
increasing the feature rank $r$, as intuitively desired. We defer the proof to
Appendix D.3. However, unlike the minimizers
$\\{U_{1}^{*},U_{2}^{*},\ldots\\}$ of the LSI bound (2.5) which satisfy
$\textrm{span}(U_{r}^{*})\subseteq\textrm{span}(U_{r+1}^{*})$, the minimizers
of $\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ are not necessarily nested.

######  Proposition 5.

Let $\\{u_{k}\\}_{k=1}^{d}$ denote any orthonormal basis of $\mathbb{R}^{d}$
and consider the sequence $\\{U_{r}=(u_{1},\ldots,u_{r})\in\mathbb{R}^{d\times
r}\\}_{1\leq r\leq d}$. Then, we have
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r+1})\leq\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})$
for all $r<d$. As a consequence, for all $r<d$ we have

$\min_{\begin{subarray}{c}U_{r+1}\in\mathbb{R}^{d\times(r+1)}\\\
U_{r+1}^{\top}U_{r+1}=I_{r+1}\end{subarray}}\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r+1})\leq\min_{\begin{subarray}{c}U_{r}\in\mathbb{R}^{d\times
r}\\\
U_{r}^{\top}U_{r}=I_{r}\end{subarray}}\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r}).$
(2.15)

There are additional numerical considerations to minimizing
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$, however. When viewed as a
function over Euclidean vectors in $\mathbb{R}^{d\times r}$, this objective is
non-convex due to the orthogonality constraint $U_{r}^{\top}U_{r}=I_{r}$.
Alternatively, when considered as optimization over the Grassmann manifold
$\mathrm{Gr}(d,r)$ — the manifold of $r$ dimensional subspaces equipped with
the canonical Euclidean metric [23] — it is also impossible for this function
to be globally geodesically convex, as such smooth functions cannot exist on
compact manifolds [14, Corr 11.10]. Both observations suggest that
(Riemannian) gradient descent may stagnate at local minima. Despite this, our
numerical experiments in Pymanopt [46] — with structured $M(\pi)$ and $H(\pi)$
arising from statistical problems, or more generally with random $M(\pi)\succ
0$ and $H(\pi)\succ 0$ — suggests that Riemannian gradient descent [23]
converges to the same solution for all initial conditions. This leads us to
the following.

######  Conjecture 6.

The function
$\bar{\mathcal{J}}_{\mathrm{KL}}^{\downarrow}:\mathrm{Gr}(d,r)\rightarrow\mathbb{R}$
defined by
$\bar{\mathcal{J}}_{\mathrm{KL}}^{\downarrow}(\mathrm{range}\\{U_{r}\\})=\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})$
as in (2.10) for all orthonormal matrix $U_{r}\in\mathbb{R}^{d\times r}$, is
benignly non-convex on the Grassmann manifold $\mathrm{Gr}(d,r)$.

Benign non-convexity refers to non-convex functions for which every local
minima is a global minima; see e.g. [42, 45] for a curated list of examples.
Geometrically, all saddle points (if any) must therefore have at least one
escape direction, providing a mechanism for first-order descent methods to
reach the global minimizer almost surely. While there has been significant
interest in the optimization community on discovering such functions, to our
knowledge the proof techniques for demonstrating benign non-convexity on the
Grassmannian require knowing the global minima in closed form, see e.g. [4,
5]. We are unable to prove such a characterization, and we can only derive
necessary and sufficient conditions for first order critical points in
Appendix A.

We now revisit the central question concerning the choice of functional
inequality for dimension reduction. Evidently, the use of the dimensional LSI
improves upon the LSI: Theorem 4 shows that sharper error certificates are
obtained which are also computationally straightforward to minimize, assuming
Conjecture 6 holds. In fact, the following proposition suggests that this is,
in some sense, the _optimal_ choice of functional inequality for dimension
reduction under the KL divergence. Specifically, minimizing majorant (2.10) is
equivalent to minimizing the KL divergence when the target is Gaussian, i.e.,
the dimensional LSI instrinsically encodes dimension reduction for Gaussian
measures. Accordingly, Theorem 4 can be interpreted as the extension of
dimension reduction towards non-Gaussian distributions.

######  Proposition 7.

If $\pi$ is a non-singular Gaussian measure then
$\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})=\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}))$
for all matrices $U_{r}\in\mathbb{R}^{d\times r}$ with orthonormal columns.

###### Proof.

Let $\pi=\mathcal{N}(m,C)$ be a Gaussian measure with mean $m$ and covariance
$C\succeq 0$. For any orthonormal matrix $U=[U_{r},\,\,U_{\perp}]$ the measure
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})$ defined as in (2.2) is also Gaussian,
but with mean $U_{r}U_{r}^{\top}m$ and covariance
$U\mathrm{diag}(U_{r}^{\top}CU_{r},I_{\perp})U^{\top}$. Direct computation
shows that

$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}_{r})=\ln\det(U_{r}^{\top}CU_{r})-\ln\det
C-d+r+\operatorname{tr\,}U_{\perp}^{\top}CU_{\perp}+\operatorname{tr\,}U_{\perp}^{\top}mm^{\top}U_{\perp}.$

Combining this with the observation
$H(\pi)=\mathbb{E}_{\pi}[C^{-1}(X-m)(X-m)^{\top}C^{-1}]=C^{-1}$ and applying
the standard identity $C=M(\pi)-mm^{\top}$ completes the proof. ∎

####  2.2.1 Numerical Example: Linear Gaussian inverse problems

We consider linear Gaussian inverse problems — applications to a broader class
of Bayesian inverse problems are discussed in §2.5. The reference measure
$\mu$ is the prior distribution of the model parameters $X$, assumed isotropic
Gaussian in $\mathbb{R}^{d}$, and we assume data generated by the model
$Y=AX+\epsilon$, where $A\in\mathbb{R}^{m\times d}$ and
$\epsilon\sim\mathcal{N}(0,I_{m})$. The posterior measure of $X|Y=y$ is the
Gaussian measure $\pi$ with mean $m(\pi)=(I_{d}+A^{\top}A)^{-1}A^{\top}y$ and
covariance $C(\pi)=(I_{d}+A^{\top}A)^{-1}$. For any feature
$U_{r}\in\mathbb{R}^{d\times r}$, the approximation
$\widetilde{\pi}^{\mathrm{KL}}(U_{r})$ is also Gaussian with mean
$U_{r}U_{r}^{\top}m(\pi)$ and covariance
$I_{d}+U_{r}U_{r}^{\top}(C(\pi)-I_{d})U_{r}U_{r}^{\top}$.

Cui et al. [19, Lemma 2.2] considered dimension reduction for Gaussian target
measures under the assumption $\mu_{X\mid
Y}=\mathbb{E}_{\widetilde{\pi}^{\mathrm{KL}}}[X]$. Provided this holds, they
show that the $r$ leading eigenvectors of the relative Fisher information
matrix $H$ are globally optimal for (1.2) with the KL divergence. However,
since $Y\neq 0$ almost surely, these features are not provably optimal for the
typical linear Gaussian inverse problem.

Instead, the globally optimal features can be determined by minimizing
$U_{r}\mapsto\mathcal{J}^{\downarrow}(U_{r})$. We evaluate the stability of
this objective when the Fisher information matrix and the second moment matrix
are empirically estimated from samples. While we use the standard Monte Carlo
for $\widehat{H}(\pi||\mu)$ and $\widehat{M}(\pi)$, we use the estimator
$\widehat{H}(\pi):=\widehat{H}(\pi||\mu)-\widehat{M}(\pi)+2I_{n_{x}}$ rather
than the Monte Carlo estimate of $\mathbb{E}_{\pi}[\nabla\ln\pi^{\otimes 2}]$
as we observed this produced more stable numerical results.

Figure 1 compares the performance of the estimator for a linear Gaussian
inverse problem of size $n_{x}=m=50$. The solid black curve depicts the result
of minimizing (2.10) with analytical $H(\pi)$ and $M(\pi)$, whereas the solid
gray curve depicts the loss corresponding to (2.6). The dashed lines show the
result of minimizing (2.10) using empirically estimated matrices of sample
size $n_{s}$ — observe that they under predict the true error due to
overfitting. Instead, when evaluated on a ‘testing’ set of $\widehat{H}(\pi)$
and $\widehat{M}(\pi)$ computed with $200$ samples, the solid coloured curves
show that the estimated $\widehat{U}_{r}^{\downarrow}$ are close to the global
minimizer.

Figure 1: Comparison of approximation error certificates for a Gaussian target
measure of size $n_{x}=n_{y}=50$. The globally optimally features
$U_{r}^{\mathrm{opt}}$ are obtained by minimizing (2.10), and their achieved
KL error is shown in black. The certificate obtained from the LSI majorization
is shown in gray. The minimizers of the (empirically estimated) dimensional
LSI majorant is shown in by the coloured curves, where solid lines correspond
to testing error, and dashed lines correspond to training error.

####  2.2.2 Numerical Example: Rosenbrock

Consider the two-dimensional Rosenbrock distribution $\pi=T_{\\#}\mu$ with
pushforward $T(z_{1},z_{2})=[z_{1}-0.5,~{}(z_{1}-0.5)^{2}+\sqrt{0.2}z_{2}]$
and standard Gaussian $\mu$. Figure 2 shows the probability density of $\pi$,
alongside its approximations $\widetilde{\pi}^{\mathrm{KL}}(U_{1}^{*})$ with
feature $U_{1}^{*}$, obtained by minimizing the majorant (2.5), and
$\widetilde{\pi}^{\mathrm{KL}}(U_{1}^{\downarrow})$ with feature
$U_{1}^{\downarrow}$ obtained by minimizing the majorant (2.10). The
computation of each approximation via (2.2) uses Gauss-Hermite quadratures.
The relative Fisher information matrix $H(\pi||\mu)$ and moment matrix
$M(\pi)$ are similarly computed with quadrature, yielding the Fisher
information matrix $H(\pi)$ via (2.14). Note that the tail behaviour of the
approximation differs significantly depending on the feature.

Figure 2:  (Left) Probability density of target measure. (Centre) Approximate
measure using feature $U_{1}^{*}$ determined minimizing (2.5). (Right)
Approximate measure using feature $U_{1}^{\downarrow}$ minimizing (2.9).

Each candidate feature $U_{1}\in\mathbb{R}^{2}$ can be parametrized by a
unique angle $\theta\in[0,\pi)$ by
$U_{1}(\theta)=(\cos(\theta),\sin(\theta))$. For each $\theta$, we compute
$\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{1}(\theta)))$,
once again using numerical quadratures; this corresponds to the solid blue
curve in Figure 3. The shaded gray region corresponds to the majorization and
minorization provided in Theorem 2. Minimizing the majorant yields
$\theta(U_{1}^{*})$, depicted by the dashed yellow line. In contrast, the
shaded blue region correponds to the majorization and minorization from
Theorem 4. The minimizer $\theta(U_{1}^{\downarrow})$ of (2.10) is shown by
the dashed red line.

We observe that the bounds corresponding to the dimensional LSI are
significantly tighter than those corresponding to the (non-dimensional) LSI,
consistent with our analysis. Furthermore, we observe the benign non-convex
property of $\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ as states in
Conjecture 6. Surprisingly, we also observe that the dimensional LSI
$\theta(U_{1}^{\downarrow})$ minimizer coincides with the global minimizer of
the KL divergence, whereas the LSI minimizer $\theta(U_{1}^{*})$ is nearly the
worse possible feature.

Figure 3: (Rosenbrock) KL divergence
$\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{1}(\theta)))$
with $U_{1}(\theta)=(\cos(\theta),\sin(\theta))$ for $\theta\in[0,\pi]$ (blue
line) and its bounds obtain either by the LSI of Theorem 2 (gray shade) or by
the dimensional LSI of Theorem 4 (blue shade).

###  2.3 Choosing the reference measure $\mu$ as the best Gaussian
approximation to $\pi$

As a motivation, let us note that for _isotropic_ measure $\pi$ so that
$M(\pi)=I_{d}$, the upper bound in (2.9) of Theorem 4 simplifies as

$\displaystyle\mathcal{J}_{\mathrm{KL}}^{\downarrow}(U_{r})$
$\displaystyle=\frac{1}{2}\ln\det(H(\pi))-\frac{1}{2}\ln\det(U_{r}^{\top}H(\pi)U_{r}).$

In addition, if $\pi$ is _centered_ so that $m(\pi)=0$, the lower bound
vanishes $\mathcal{J}_{\mathrm{KL}}^{\uparrow}(U_{r})=0$ for any $U_{r}$. In
this case, the minimizer of $\mathcal{J}_{\mathrm{KL}}^{\downarrow}(\cdot)$ is
known in closed form: it is the matrix $U_{r}^{\downarrow}$ with columns
containing the $r$ leading eigenvectors of the Fisher information matrix
$H(\pi)$. The bound (2.9) evaluated at this $U_{r}^{\downarrow}$ then
simplifies as

$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}^{\downarrow}))\leq\frac{1}{2}\sum_{k=r+1}^{d}\ln(\lambda_{k}(H(\pi))),$

where $\lambda_{k}(H(\pi))$ is the $k$-th largest eigenvalue of $H(\pi)$.
Recalling (2.14), we have $H(\pi)=H(\pi||\gamma)+I_{d}$ so that
$U_{r}^{\downarrow}=U_{r}^{*}$, where $U_{r}^{*}$ is the minimizer of the LSI
majorant (2.5). This further implies that we have
$\lambda_{k}(H(\pi))=1+\lambda_{k}(H(\pi||\mu))$, and using the inequality
$\ln(1+t)\leq t$ demonstrates that the above sharpens (2.6).

The same reasoning can be generalized to non-isotropic/non-centered measures
$\pi$ by changing the reference measure $\mu$ to be the Gaussian measure
$\mu=\mathcal{N}(m(\pi),C(\pi))$. This choice is natural since it is the best
Gaussian approximation to $\pi$, in the sense of minimizing
$\mu\mapsto\operatorname{D_{\mathrm{KL}}}(\pi||\mu)$ over the set of Gaussian
measures. The proof is given in Appendix D.4.

######  Theorem 8.

Let $\pi$ be a probability measure on $\mathbb{R}^{d}$ with mean $m(\pi)$ and
non-singular covariance matrix $C(\pi)$. Consider the Gaussian measure
$\mu=\mathcal{N}(m(\pi),C(\pi))$ with same mean and covariance. Then, for any
$U_{r}\in\mathbb{R}^{d\times r}$ such that $U_{r}^{\top}C(\pi)U_{r}=I_{r}$,
the approximation $\widetilde{\pi}^{\mathrm{KL}}(U_{r})$ as in (2.2) satisfies

$\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(U_{r}))\leq\frac{1}{2}\ln\det(C(\pi)H(\pi))-\frac{1}{2}\ln\det(V_{r}^{\top}H(\pi)V_{r}),$
(2.16)

where $V_{r}=C(\pi)U_{r}$ and where
$H(\pi)=\mathbb{E}_{\pi}[\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}x})\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}x})^{\top}]$
is the Fisher information matrix of $\pi$. In particular, the minimizer the
above right-hand side is given by
$U_{r}^{\downarrow}=C(\pi)^{-1}V_{r}^{\downarrow}$ where
$V_{r}^{\downarrow}=[v_{1},\ldots,v_{r}]$ contains the $r$ largest generalized
eigenvectors of the matrix pair $(H(\pi),C(\pi)^{-1})$, meaning
$H(\pi)v_{k}=\lambda_{k}(H(\pi))C(\pi)^{-1}v_{k},$ with the orthogonality
constraint $v_{i}^{\top}C(\pi)^{-1}v_{j}=\delta_{i,j}$. In this case, we
obtain

$\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{r}^{\downarrow}))\leq\frac{1}{2}\sum_{k=r+1}^{d}\ln(\lambda_{k}(H(\pi))).$
(2.17)

Theorem 8 suggests that the deviations between $\pi$ and its best Gaussian
approximation $\mu=\mathcal{N}(m(\pi),C(\pi))$ are captured by the relative
deviations — in the sense of generalized Rayleigh quotients — of the Fisher
information matrix $H(\pi)$ and the precision matrix $C(\pi)^{-1}$. Note that
for Gaussian target measures $\pi=\mathcal{N}(m,C)$, we have $H(\pi)=C^{-1}$
so that the right-hand side of (2.16) vanishes for any $U_{r}$. This is not
surprising because, in this case,
$\pi=\mu=\widetilde{\pi}^{\mathrm{KL}}(U_{r})$ for any $U_{r}$.

###  2.4 Numerical example: Rosenbrock with best Gaussian reference measure
$\mu$

We revisit the two-dimensional Rosenbrock distribution from Section 2.2.2 with
the reference measure $\mu=\mathcal{N}(m(\pi),C(\pi))$. Figure 4 depicts the
density of the optimal Gaussian reference (note it’s anisotropic covariance
structure), as well as an approximation $\widetilde{\pi}^{\mathrm{KL}}(U_{1})$
with randomly chosen feature $U_{1}$. Observe that the approximation retains
the tail properties from the reference measure.

In Figure 5 we compare the approximation properties when using the standard
reference measure $\mu=\mathcal{N}(0,I)$ compared to the optimal Gaussian
reference $\mu=\mathcal{N}(m(\pi),C(\pi))$. The red line shows that the
optimal reference achieves uniformly lower approximation error than with the
standard reference measure shown by the blue line. However, in comparing the
performances of the majorants, we note that the majorant in Theorem 8 (shown
by the upper red envelope) suggests to select the globally sub-optimal feature
direction at $\theta=180$ degrees. In contrast, the minimizer of the majorant
from Theorem 4 (upper blue envelope) certifies a lower approximation error,
while also coinciding with the globally optimal direction, as previously
discussed in Section 2.2.2. It is an interesting research direction to
understand when one should or should not construct approximations based on the
standard reference measure.

Figure 4:  (Left) Probability density of target measure. (Centre) Optimal
Gaussian prior with mean $m(\pi)$ and covariance $C(\pi)$. (Right) Approximate
measure with feature $U_{r}$ using optimal Gaussian prior. Figure 5:
(Rosenbrock) KL divergence
$\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{1}(\theta)))$
with $U_{1}(\theta)=(\cos(\theta),\sin(\theta))$ for $\theta\in[0,\pi]$ (red
line) with the optimal Gaussian reference measure
$\mu=\mathcal{N}(m(\pi),C(\pi))$ and its bounds obtained by the dimensional
LSI of Theorem 8 (red shade). We include for comparison the true approximation
error (blue line) and dimensional LSI bounds (blue shade) in Theorem 4 when
constructing an approximation using the standard reference measure
$\mu=\mathcal{N}(0,1)$.

###  2.5 Application to Bayesian inverse problems: the data-free setting

For Bayesian inverse problems, the target measure is the posterior
$\mathrm{d}\pi^{y}(x)\propto\ell^{y}(x)\mathrm{d}\mu(x)$, where the likelihood
function $x\mapsto\ell^{y}(x)$ is proportional to the probability density of
observing a given data $y$ knowing $x$. As noticed in [43, 48, 20], in many
practical settings the data $y$ only provide information about a few linear
features of $x$. Choosing the prior $\mu$ as the reference measure is natural
here, and $U_{r}$ can be interpreted as the directions which captures the
prior-to-posterior update. The methodology in §2.1 and §2.2 provides an
algorithm towards circumventing the impact of high-dimensionality in posterior
sampling.

The _data-free_ approach proposed in [21] consists of seeking for low-
dimensional features $V_{r}\in\mathbb{R}^{d\times r}$ which are informed by
the _average_ realization of data. Formally, these features are characterized
by

$\min_{\begin{subarray}{c}V_{r}\in\mathbb{R}^{d\times r}\\\
V_{r}^{\mathsf{T}}V_{r}=I_{r}\end{subarray}}\,\mathbb{E}_{Y}\left[\,\min_{\ell_{r}^{Y}:\,\mathbb{R}^{r}\to\mathbb{R}_{+}}\operatorname{D_{\mathrm{KL}}}(\pi^{Y}\,||\,\widetilde{\pi}(V_{r},\ell_{r}^{Y}))\right]\quad\text{for
}\mathrm{d}\widetilde{\pi}(x\mid
V_{r},\ell_{r}^{y})\propto\ell_{r}^{y}(V_{r}^{\mathsf{T}}x)\mathrm{d}\mu(x).$
(2.18)

In contrast to (1.2), optimality is quantified with respect to the _averaged_
KL divergence with respect to data $Y$, with marginal law given by
$\mathrm{d}\rho(y)\propto(\int\ell^{y}(x)\mathrm{d}\mu(x))\mathrm{d}y$. The
authors demonstrate an application of the logarithmic Sobolev inequality to
derive a majorization to (2.18). We show in Theorem 9 that a tighter
majorization can be attained by leveraging the dimensional LSI instead. The
proofs of these statements are left to Appendix D.5.

######  Theorem 9.

Let $\mathrm{d}\pi^{Y}(x)\propto\ell^{Y}(x)\mathrm{d}\mu(x)$ be the posterior
distribution of the random vector $X\sim\mu$ conditioned on $Y$, where
$\mu=\mathcal{N}(0,I_{d})$ is the standard Gaussian prior and $\ell^{y}(x)$ is
the likelihood function of observing $Y=y$ given $X=x$. For any matrix
$V_{r}\in\mathbb{R}^{d\times r}$ with $r\leq d$ orthogonal columns, the
measure $\mathrm{d}\widetilde{\pi}^{\mathrm{KL},y}(x\mid
V_{r})\propto\ell^{\mathrm{KL},y}_{r}(V_{r}^{\top}x)\mathrm{d}\mu(x)$ with
$\ell^{\mathrm{KL},y}_{r}(\theta_{r})=\mathbb{E}_{X\sim\mu}[\ell^{y}(X)\mid
V_{r}^{\top}X=\theta_{r}]$ satisfies

$\mathbb{E}_{Y}\left[\operatorname{D_{\mathrm{KL}}}(\pi^{Y}||\widetilde{\pi}^{\mathrm{KL},Y}(V_{r}))\right]\leq\frac{1}{2}\ln\det\left(V_{\perp}^{\top}(I_{d}+H_{\mathrm{df}})V_{\perp}\right),$
(2.19)

where $V_{\perp}\in\mathbb{R}^{d\times(d-r)}$ denotes any orthogonal
completion to $V_{r}$ and where $H_{\mathrm{df}}\in\mathbb{R}^{d\times d}$ is
the _data-free diagnostic matrix_ defined by

$H_{\mathrm{df}}=\mathbb{E}_{X,Y}[\nabla_{x}\ln\ell^{Y}(X)\nabla_{x}\ln\ell^{Y}(X)^{\top}].$
(2.20)

In particular, the minimizer $V_{r}^{\downarrow}$ of the right-hand side of
(2.19) is given by $V_{r}^{\downarrow}=[v_{1},\ldots,v_{r}]$, where $v_{k}$ is
the $k$-th largest eigenvector of $H_{\mathrm{df}}$. Denoting by
$\lambda_{k}(H_{\mathrm{df}})$ the corresponding eigenvalue, (2.19) yields

$\mathbb{E}_{Y}\left[\operatorname{D_{\mathrm{KL}}}(\pi^{Y}\,||\,\widetilde{\pi}^{\mathrm{KL}}\,{}^{Y}(V_{r}))\right]\leq\frac{1}{2}\sum_{k=r+1}^{d}\ln(1+\lambda_{k}(H_{\mathrm{df}})).$
(2.21)

The nomenclature ‘data-free’ refers to the fact that $H_{\mathrm{df}}$ only
requires an expectation over the joint law of $(X,Y)$ and does not depend on
the observed realization $y$ of the data. Samples from the joint $(X,Y)$ can
be easily obtained via Gibb’s algorithm, by first sampling $X\sim\mu$ and then
$Y\mid X\sim\pi^{X}$ where
$\mathrm{d}\pi^{x}(y)\propto\ell^{y}(x)\mathrm{d}y$.

Compared to the majorant
$\mathbb{E}_{Y}[\operatorname{D_{\mathrm{KL}}}(\pi^{Y}||\widetilde{\pi}^{\mathrm{KL},Y}(V_{r}))]\leq\frac{1}{2}\operatorname{tr\,}((I_{d}-V_{r}V_{r}^{\top})H_{\mathrm{df}})$
obtained by [21, §3.3], (2.19) is strictly tighter by
$\ln\det(I+A)\leq\operatorname{tr\,}(A)$. While both majorants are minimized
by the same feature $V_{r}^{\downarrow}$, the certificate of Cui and Zahm only
ensures that

$\mathbb{E}_{Y}\operatorname{D_{\mathrm{KL}}}(\pi\,||\,\widetilde{\pi}^{\mathrm{KL}}(V_{r}^{\downarrow}))\leq\frac{1}{2}\sum_{k=r+1}^{d}\lambda_{k}(H_{\mathrm{df}}),$

whereas Theorem 9 guarantees that the approximation error can be significantly
less. In some instances this can be _exponentially_ better, such as for small
$r$ when the sum of the trailing eigenvalues is large. Consequently,
significantly fewer dimensions $r$ suffice to achieve the same error tolerance
than previously considered.

######  Remark 3.

Proposition 7 extends to the data-free setting, that is,
$\mathbb{E}_{Y}\left[\operatorname{D_{\mathrm{KL}}}(\pi^{Y}||\widetilde{\pi}^{\mathrm{KL},Y}(V_{r}))\right]=\frac{1}{2}\ln\det\left(V_{\perp}^{\top}(I_{d}+H_{\mathrm{df}})V_{\perp}\right),$
for any $V_{r}$ whenever $\pi^{Y}$ is Gaussian.

######  Remark 4.

As noticed in [13],
$\mathbb{E}_{Y}\operatorname{D_{\mathrm{KL}}}(\pi^{Y}||\widetilde{\pi}^{\mathrm{KL},Y}(V_{r}))=I(X;Y)-I(V_{r}^{\top}X;Y)$
where $I(X,Y)$ is the _mutual information_ between $X$ and $Y$. Thus, the
bound (2.19) yields a lower-bound on the mutual information between
$V_{r}^{\top}X$ and $Y$ given by

$I(V_{r}^{\top}X;Y)\geq
I(X;Y)-\frac{1}{2}\ln\det\left(V_{\perp}^{\top}(I_{d}+H_{\mathrm{df}})V_{\perp}\right).$

######  Remark 5.

The reverse dimensional LSI (2.8) results in a lower bound (not shown)
involving the _data-free lower diagnostic matrix_

$G_{\mathrm{df}}=\mathbb{E}_{Y}[\mathbb{E}_{X\mid
Y}[\nabla_{x}\ln\ell^{Y}(X)]\mathbb{E}_{X\mid
Y}[\nabla_{x}\ln\ell^{Y}(X)]^{\top}].$

This is difficult to compute on account of the nested expectations with
respect to the law of $X\mid Y$. A looser bound can be obtained by lower
bounding this matrix, in the Loewner sense, by
$G^{\prime}_{\mathrm{df}}=\mathbb{E}_{X,Y}[\nabla_{x}\ln\ell^{Y}(X)]\mathbb{E}_{X,Y}[\nabla_{x}\ln\ell^{Y}(X)]^{\top}$.
However, $G^{\prime}_{\mathrm{df}}=0$ under the commonly used data likelihood
model $Y\mid X\sim\mathcal{N}(g(X),~{}\sigma_{\mathrm{obs}}^{2})$, with
parameter-to-observation model $g$, resulting in a vacuous bound.

####  2.5.1 Applications to Inverse Problems with Generative Modeling Priors

Recent research has focused on Bayesian inverse problems relying on generative
modeling priors such as GANs [27]. This adopts an empirical Bayes philosophy
wherein the prior is constructed from training samples and is assumed to be
supported on a low-dimensional manifold. Mathematically, the prior is codified
as the pushforward of a low-dimensional latent variable $Z$ taking values in
$\mathbb{R}^{\kappa},~{}\kappa\ll d$, so that $\mu_{X}=\phi_{\\#}\mu_{Z}$ for
some $\phi:\mathbb{R}^{\kappa}\to\mathbb{R}^{d}$. Without loss of generality,
the measure $\mu_{Z}$ can be taken to be standard Gaussian. This procedure
significantly reduces the complexity of sampling — it suffices to sample the
latent posterior

$\mathrm{d}\pi_{Z}^{y}(z)\propto\ell^{y}(\phi(z))\mathrm{d}\mu_{Z}(z),$ (2.22)

and posterior samples in the model space $X$ can be obtained by the
pushforward condition $\pi^{y}_{X}=\phi_{\\#}\pi^{y}_{Z}$.

Figure 6: (Left) GAN decoder applied to prior sample $Z\sim\mu_{Z}$. (Right)
GAN decoder applied to same prior sample but projected onto features
$V_{r}^{\downarrow}$. Increasing $r$ shows which features (in the nominal
model space) are informed by data.

We demonstrate that additional dimension reduction can be achieved by
accounting for interactions between data, likelihood models, and generative
priors. In particular, we consider a geophysical inverse problem in which the
nominal parameter space $X$ are discretizations of the subsurface taking
values in $3600$ dimensions. Since realizations of the subsurface exhibit
regular, repeated layered structures, high-fidelity prior samples can be drawn
from a trained GAN prior with only a $300$ dimensional latent space (see
Figure 6). Using this GAN prior we compute the error certificate (2.19) and
plot it in Figure 7 as a function of rank $r$. This figure shows that data
only inform relatively few latent coordinates — e.g., it suffices consider
$r=50$ features to obtain an expected KL error of $0.1$ nats. Moreover, domain
practitioners can visualize which features are “data-informed” by visualizing
the convergence of the GAN output with increasing $r$, as depicted in Figure
6.

Figure 7: The data-free approximation error certificate (2.21) versus rank $r$
for the geophysical inverse problem with generative GAN prior.

##  3 Dimension reduction with the squared Hellinger distance

It is of interest to consider alternative error metrics in (1.2) to the KL
divergence. Notably, [20, 17] considered the squared Hellinger distance
$\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\mu)=\frac{1}{2}\int(\sqrt{\mathrm{d}\pi/\mathrm{d}x}-\sqrt{\mathrm{d}\mu/\mathrm{d}x})^{2}\mathrm{d}x$.
Straightforward calculation permits us to relate this distance to the variance
$\operatorname{Var}_{\mu}(\sqrt{\mathrm{d}\pi/\mathrm{d}\mu})$ following

$\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\mu)=1-\sqrt{1-\operatorname{Var}_{\mu}\left(\sqrt{\frac{\mathrm{d}\pi}{\mathrm{d}\mu}}\right)}.$
(3.1)

As the Poincaré inequality provides gradient-based bounds on variances, it is
natural choice to employ it in this context to control the squared Hellinger
distance (just as the logarithmic Sobolev inequality naturally relates to the
KL divergence). The Poincaré inequality, however, similarly exhibits a
“dimension-free” property as the LSI, see e.g. [18, §3], and thus one may
wonder whether dimensional refinements can be leveraged to obtain improved
majorizations. In this section, we discuss such refinements to the Poincaré
inequality for the purpose of dimension reduction in the squared Hellinger
distance. Our presentation largely mirrors Section 2.

###  3.1 Certifiable bounds using the Poincaré Inequality

Let us recall the Poincaré inequality for the standard Gaussian measure
$\mu=\mathcal{N}(0,I_{d})$.

######  Proposition 10 (Theorem 4.7.2. in [8]).

The standard Gaussian measure $\mu=\mathcal{N}(0,I_{d})$ satisfies

$\left\|\int\nabla
f(x)\mathrm{d}\mu\right\|_{2}^{2}\leq\operatorname{Var}_{\mu}(f)\leq\int\|\nabla
f\|_{2}^{2}\mathrm{d}\mu.$ (3.2)

for any smooth function $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$. We refer to
the upper (resp. lower) bound as the (resp. reverse) _Poincaré inequality_.

By letting $f=\sqrt{\mathrm{d}\pi/\mathrm{d}\mu}$ in (3.2), direct calculation
permits to bound the squared Hellinger distance (3.1) from $\pi$ to
$\mu=\mathcal{N}(0,I_{d})$ as follow

$1-\sqrt{\left(1-\left\|\mathbb{E}_{\pi}\left[X\sqrt{\frac{\mathrm{d}\mu}{\mathrm{d}\pi}}\right]\right\|^{2}\right)_{+}}\leq\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\mu)\leq
1-\sqrt{\left(1-\frac{\operatorname{tr\,}(H(\pi||\mu)}{4})\right)_{+}},$

where
$H(\pi||\mu)=\mathbb{E}_{\pi}[\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})^{\top}]$
is the Fisher information matrix of $\pi$ relative to $\mu$ and
$(t)_{+}:=\max(t,0)$. The above left-hand side involves the term
$\mathbb{E}_{\pi}[X\sqrt{\frac{\mathrm{d}\mu}{\mathrm{d}\pi}}]$ which,
compared to (2.5), is different to the mean of $\pi$. Its computation is,
however, difficult when $\pi$ is known up to a normalizing constant which
makes it not useable in practice.

With another judicious choice of test function $f$, the same Poincaré
inequality yields a tractable upper-bound for the squared Hellinger distance
between $\pi$ and a structured approximation $\widetilde{\pi}$ as in (1.1).

######  Theorem 11 (Adapted from [34]).

Let $\pi$ be a probability measures on $\mathbb{R}^{d}$ and let
$\mu=\mathcal{N}(0,I_{d})$ be the standard Gaussian measure. Then, for every
matrix $U_{r}\in\mathbb{R}^{d\times r}$ with $r<d$ orthogonal columns, the
minimizer of
$\ell_{r}\mapsto\operatorname{d_{\mathrm{Hell}}^{2}}(\pi\,||\,\widetilde{\pi}(U_{r},\ell_{r}))$
where $\mathrm{d}\widetilde{\pi}(x\mid
U_{r},\ell_{r})\propto\ell_{r}(U_{r}^{\top}x)\mathrm{d}\mu(x)$ is

$\ell^{\mathrm{Hell}}_{r}(\theta_{r}\mid
U_{r})=\mathbb{E}_{X\sim\mu}\left[\left.\sqrt{\frac{\mathrm{d}\pi}{\mathrm{d}\mu}}(X)\,\right|\,U_{r}^{\top}X=\theta_{r}\right]^{2}.$
(3.3)

Furthermore, the resulting measure
$\widetilde{\pi}^{\mathrm{Hell}}(U_{r})=\widetilde{\pi}(U_{r},\ell^{\mathrm{Hell}}_{r}(U_{r}))$
satisifies

$\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\widetilde{\pi}^{\mathrm{Hell}}(U_{r}))\leq
1-\sqrt{\left(1-\frac{\operatorname{tr\,}(H(\pi||\mu))-\operatorname{tr\,}(U_{r}^{\top}H(\pi||\mu)U_{r})}{4}\right)_{+}},$
(3.4)

where
$H(\pi||\mu)=\mathbb{E}_{\pi}[\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})^{\top}]$
is the Fisher information matrix of $\pi$ relative to $\mu$.

The matrix $U_{r}^{*}=[u_{1},\ldots,u_{r}]$ whose columns are the $r$ leading
eigenvectors of $H(\pi||\mu)$ is a minimizer of the right-hand side of (3.4),
just as with the LSI bound for the KL divergence (c.f. (2.6)). (In fact, [34]
show these features certify dimension reduction for _all_ Amari
$\alpha$-divergences with $\alpha\in(0,1]$.) This furnishes
$\widetilde{\pi}^{\mathrm{Hell}}(U_{r}^{*})$ with the error certificate in the
squared Hellinger distance

$\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\widetilde{\pi}^{\mathrm{Hell}}(U_{r}^{*}))\leq
1-\sqrt{\left(1-\frac{1}{4}\sum_{k=r+1}^{d}\lambda_{k}(H(\pi||\mu))\right)_{+}}.$
(3.5)

Whereas the reverse LSI yields the lower bound in Prop. 1, the reverse
Poincaré inequality in (3.2) does not lead to a computable minorization for
the squared Hellinger distance without explicit reference to $Z_{\pi}$.

###  3.2 Dimensional Improvements to the Poincaré Inequality

There are several formulations of dimensional Poincaré inequalities for the
standard Gaussian $\mu$ in the literature, including:

  * •

the Bobkov-Ledoux [11] inequality

$\operatorname{Var}_{\mu}(f)\leq 6\int\|\nabla
f\|_{2}^{2}\mathrm{d}\mu-6\int\frac{\|\langle\nabla
f,x\rangle\|_{2}^{2}}{d+\|x\|_{2}^{2}}\mathrm{d}\mu,$

  * •

the Bonnefont-Joulin-Ma inequality [9]

$\operatorname{Var}_{\mu}(f)\leq\frac{d(d+3)}{d-1}\int\frac{\|\nabla
f\|_{2}^{2}}{1+\|x\|_{2}^{2}}\mathrm{d}\mu,$

  * •

and the first inequality of Bolley-Gentil-Guillin [7], who obtain

$\operatorname{Var}_{\mu}(f)\leq\int\|\nabla
f\|_{2}^{2}\mathrm{d}\mu-\frac{1}{2d}\left(\int(\|x\|_{2}^{2}-d)f\mathrm{d}\mu\right)^{2}$

by linearizing the dimensional logarithmic Sobolev inequality.222Linearizing
refers to the choice of test function $f_{\epsilon}=1+\epsilon g$, where $\int
g\mathrm{d}\mu=0$. Substituting into (2.7), scaling $\epsilon\to 0$, and
matching terms of order $\epsilon^{2}$ recovers the desired inequality. A
similar exercise applied to the LSI (2.4) recovers the Poincaré inequality
(3.2).

These inequalities are not mutually comparable as one does not imply the
other.

Although we do not explicitly document this, the Bobkov-Ledoux and the
Bonnefont-Joulin-Ma inequalities produce worse majorizations than (3.4)
obtained from the (non-dimensional) Poincaré inequality. We speculate this is
due to their large constants — whereas (3.2) is sharp for $f(x)=x$, it is
unclear whether the prefactors of both these inequalities can be further
optimized. Furthermore, it can be shown that the Bolley-Gentil-Guillin
inequality produces a majorant which explicitly requires the normalizing
constant of $\pi$, and is thus of commensurate difficulty to compute when
$\pi$ is known up to a constant.

Instead, we rely on another dimensional improvement to the Poincaré inequality
proposed by Bolley, Gentil, and Guillin.

######  Proposition 12 (Theorem 4.3 in [7]).

The variance of a smooth function $f:\mathbb{R}^{d}\to\mathbb{R}$ with respect
to the standard Gaussian $\mu=\mathcal{N}(0,I_{d})$ satisfies the _dimensional
Poincaré inequality_

$\operatorname{Var}_{\mu}(f)\leq\int\|\nabla
f\|_{2}^{2}\mathrm{d}\mu-\int\frac{\|f-\int f\mathrm{d}\mu-\langle\nabla
f,x\rangle\|_{2}^{2}}{d+\|x\|_{2}^{2}}\mathrm{d}\mu.$ (3.6)

######  Remark 6.

We are not aware of a _reverse_ dimensional Poincaré inequality. The proof
technique used in [8, Thm 6.7.3] to derive the reverse dimensional LSI relies
on an algebraic identity specific to the LSI entropy function $x\mapsto x\ln
x$, which cannot be adapted here. Alternatively, linearizing the reverse
dimensional LSI merely recovers the (non-dimensional) reverse Poincaré
inequality in Proposition 10.

Evidently (3.6) improves on the Poincaré inequality, although the difference
vanishes in the limit as $d\to\infty$. Note that the dimensional Poincaré is
also tight for the test function $f(x)=x$.

We apply the dimensional Poincaré inequality to obtain a majorization for the
squared Hellinger approximation error. The proof can be found in Appendix D.6.

######  Theorem 13.

Under the same setting as Theorem 11, we have

$\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\,\widetilde{\pi}^{\mathrm{Hell}}(U_{r}))\leq
1-\sqrt{\left(1-\frac{\operatorname{tr\,}(H(\pi||\mu))-\operatorname{tr\,}(U_{r}^{\top}H(\pi||\mu)U_{r})}{4}+\delta_{r}(U_{r},\,\operatorname{d_{\mathrm{Hell}}^{2}}(\pi,\,\widetilde{\pi}^{\mathrm{Hell}}(U_{r})))\right)_{+}}$
(3.7)

with

$\delta_{r}(U_{r},y)=\frac{(1-(1-y)^{2}-\frac{1}{2}\operatorname{tr\,}(M(\pi))+\frac{1}{2}\operatorname{tr\,}(U_{r}^{\top}M(\pi)U_{r})+\frac{1}{2}(d-r))^{2}}{\operatorname{tr\,}(M(\pi))-\operatorname{tr\,}(U_{r}^{\top}M(\pi)U_{r})+d-r},$
(3.8)

where
$H(\pi||\mu)=\mathbb{E}_{\pi}[\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})\nabla\ln(\frac{\mathrm{d}\pi}{\mathrm{d}\mu})^{\top}]$
is the Fisher information matrix of $\pi$ relative to $\mu$ and
$M(\pi)=\mathbb{E}_{\pi}[XX^{\top}]$ the second moment matrix.

######  Remark 7.

For isotropic measure $\pi$ such that $M(\pi)=I_{d}$ we obtain

$\epsilon_{r}(U_{r},y)=\frac{(1-(1-y)^{2}))^{2}}{2(d-r)}.$

Observe that the function $\mathbb{R}^{d\times
r}\times\mathbb{R}\ni(U_{r},y)\mapsto\delta_{r}(U_{r},y)\in\mathbb{R}$ defined
in (3.8) is strictly positive — the numerator is evidently positive, while the
denominator is positive since $M(\pi)\succ 0$. By inspection, this implies
that the majorant (3.7) improves upon the majorant (3.4). Unfortunately, the
squared Hellinger loss appears in both the left and right hand side of (3.7),
and is therefore difficult to optimize to determine linear features $U_{r}$.
Instead, one might hope to _evaluate_ (3.7) with the features $U_{r}^{*}$
minimizing (3.4) to obtain a tighter certificate for the approximation error.
Surprisingly, to do so one cannot simply bootstrap the upper bound from (3.4)
to further bound the right hand side of (3.7)— instead, one requires a  _lower
bound_ on the approximation error in order to leverage the improvements from
(3.7). Although non-vacuous lower bounds can be obtained from applying the
reverse Poincaré inequality (3.2), we do not expect such bounds to be easily
computable. It therefore remains an open question as to how we can make use of
the improvements brought on by the dimensional Poincaré inequality, and we
leave exploring this direction to future work.

##  Appendix A Necessary and sufficient conditions for first order optimality
of (2.10)

Encoding the orthogonality constraint into (2.10) with Lagrange multiplier
$S=S^{\top}\in\mathbb{R}^{r\times r}$ and ignoring constant terms yields the
Lagrangian

$L(U_{r},S)=\frac{\operatorname{tr\,}\left(U_{r}^{\top}M(\pi)U_{r}\right)+\ln\det\left(U_{r}^{\top}H(\pi)^{-1}U_{r}\right)+\operatorname{tr\,}\left(S(I_{r}-U_{r}^{\top}U_{r})\right)}{2}.$

First-order optimality asserts that

$\nabla_{U_{r}}L(U_{r},S)=M(\pi)U_{r}+H(\pi)^{-1}U_{r}\left(U_{r}^{\top}H(\pi)^{-1}U_{r}\right)^{-1}-U_{r}S=0$
(A.1)

for all orthonormal $U_{r}$. Left multiplying (A.1) by $U_{r}^{\top}$ thus
recovers $S=U_{r}^{\top}M(\pi)U_{r}+I_{d},$ and (A.1) is equivalently

$U_{r}=(I_{d}-U_{r}U_{r}^{\top})M(\pi)U_{r}+H(\pi)^{-1}U_{r}\left(U_{r}^{\top}H(\pi)^{-1}U_{r}\right)^{-1}.$
(A.2)

Re-arranging (A.2) also gives the alternative fixed point condition

$U_{r}=H(\pi)\left(I_{d}-(I_{d}-U_{r}U_{r}^{\top})M(\pi)\right)U_{r}(U_{r}^{\top}H(\pi)^{-1}U_{r}).$
(A.3)

We are not aware of any closed form solutions to (A.2) or (A.3).

Observe that left multiplying (A.3) by primal feasible $U_{r}^{\top}$ and
$U_{\perp}^{\top}$ recovers

$\displaystyle
U_{r}^{\top}\left(H(\pi)-H(\pi)M(\pi)+H(\pi)U_{r}U_{r}^{\top}M(\pi)\right)U_{r}(U_{r}^{\top}H(\pi)^{-1}U_{r})$
$\displaystyle=I_{r}$ (A.4) $\displaystyle
U_{\perp}^{\top}\left(H(\pi)-H(\pi)M(\pi)+H(\pi)U_{r}U_{r}^{\top}M(\pi)\right)U_{r}$
$\displaystyle=0_{d-r,r}$ (A.5)

However, when $r>1$, (A.4) implies that
$U_{r}^{\top}\left(H(\pi)-H(\pi)M(\pi)+H(\pi)U_{r}U_{r}^{\top}M(\pi)\right)U_{r}$
is symmetric, as it is the inverse of a symmetric matrix. This means

$U_{r}^{\top}(H(\pi)M(\pi)-M(\pi)H(\pi))U_{r}=(U_{r}^{\top}H(\pi)U_{r})(U_{r}^{\top}M(\pi)U_{r})-(U_{r}^{\top}M(\pi)U_{r})(U_{r}^{\top}H(\pi)U_{r}),$

or, expanding $H(\pi)$ as (2.14) and condensing notation via the matrix
commutator $[A,B]=AB-BA$, we have that

$\displaystyle U_{r}^{\top}[H(\pi||\mu),M(\pi)]\,U_{r}$
$\displaystyle=[U_{r}^{\top}H(\pi||\mu)U_{r},\,U_{r}^{\top}M(\pi)U_{r}]$ (A.6)
$\displaystyle
U_{\perp}^{\top}(H(\pi||\mu)+M(\pi)+H(\pi||\mu)M(\pi)-M(\pi)^{2})U_{r}$
$\displaystyle=U_{\perp}^{\top}(H-M(\pi))U_{r}U_{r}^{\top}M(\pi)U_{r}.$ (A.7)

To reiterate: $U_{r}$ is a critical point of (2.10) if and only if it
satisfies (A.6) and (A.7). While (A.6) is satisfied if $U_{r}$ contains any
$r$ eigenvectors of $H(\pi||\mu)$ or $M(\pi)$, these are in general these are
not the only solutions. We examine both of these cases individually:

  1. 1.

suppose $U_{r}$ contains any $r$ eigenvectors of the second moment matrix
$M(\pi)$, so that $M(\pi)U_{r}=U_{r}S$ where
$S=\mathrm{diag}(s_{1},\ldots,s_{r})$. Then (A.7) becomes

$U_{\perp}^{\top}H(\pi||\mu)U_{r}=0_{d-r,r}\,.$

This condition cannot be satisfied as vectors cannot be simultaneously
orthogonal, $M(\pi)$-orthogonal, and $H(\pi||\mu)$-orthogonal, unless any
pairs of these matrices commute.

  2. 2.

suppose $U_{r}$ contains any $r$ eigenvectors of $H(\pi||\mu)$ with
$H(\pi||\mu)U_{\perp}=U_{\perp}T_{\perp}$. Then (A.7) becomes

$(I_{\perp}+T_{\perp})U_{\perp}^{\top}M(\pi)U_{r}-U_{\perp}^{\top}M(\pi)^{2}U_{r}=-(U_{\perp}^{\top}M(\pi)U_{r})(U_{r}^{\top}M(\pi)U_{r}).$

Numerically, we observe for Gaussian target distributions, that this condition
is not met. In other words, the leading eigenvectors of the relative Fisher
information matrix, while globally optimal for (2.5), are not even local
minima of (2.10) in general.

Note that this does not contradict Lemma 2.2 in [43]. In their setting, they
assume the mean of $\pi^{\mathrm{KL}}_{r}$ matches the mean of $\pi$ — in
general, this condition is not true. However, when this criteria is met, then
it can be shown that (A.7) is also satisfied.

These results suggest that numerical optimization is required in general to
obtain critical points of (2.10).

######  Remark 8.

Observe that (A.4) allows the component $(U_{r}^{\top}H(\pi)^{-1}U_{r})$
appearing in (2.10) to be replaced by a function involving only $H(\pi)$ and
$M$ at optimality, which does not require computing matrix inverses. This may
offer computational speedups, but we do not investigate this further in the
current work.

##  Appendix B Auxiliary Matrix Determinant Identity

Sylvester’s determinant theorem allows us to write

$\displaystyle\det\left(U_{\perp}^{\top}H(\pi)U_{\perp}\right)$
$\displaystyle=\det\left(U_{\perp}^{\top}(H(\pi)-I_{d})U_{\perp}+I_{\perp}\right)=\det\left((H(\pi)-I_{d})(U_{\perp}U_{\perp}^{\top})+I_{d}\right)$
$\displaystyle=\det\left((H(\pi)-I_{d})(I_{d}-U_{r}U_{r}^{\top})+I_{d}\right)=\det\left(H(\pi)-H(\pi)\,U_{r}U_{r}^{\top}-(I_{d}-U_{r}U_{r}^{\top})+I_{d}\right)$
$\displaystyle=\det\left(H(\pi)-H(\pi)\,U_{r}U_{r}^{\top}+U_{r}U_{r}^{\top}\right)=\det\left(H(\pi)\right)\det\left(I_{d}-U_{r}U_{r}^{\top}+H(\pi)^{-1}U_{r}U_{r}^{\top}\right)$
$\displaystyle=\det\left(H(\pi)\right)\det\left(I_{d}+(H(\pi)^{-1}U_{r}-U_{r})U_{r}^{\top}\right)=\det\left(H(\pi)\right)\det\left(I_{r}+U_{r}^{\top}(H(\pi)^{-1}U_{r}-U_{r})\right)$
$\displaystyle=\det\left(H(\pi)\right)\det\left(U_{r}^{\top}H(\pi)^{-1}U_{r}\right),$

assuming $H(\pi)$ is invertible.

##  Appendix C Derivation of Functional Inequalities

###  C.1 Gaussian Logarithmic Sobolev Inequality

Numerous derivations of the Gaussian logarithmic Sobolev inequality are known,
but we prefer the presentation of [8] as this also conveniently encapsulates
the _reverse_ LSI. The following is a condensed version of [8, Thm 5.5.2],
which is sufficient to obtain Theorem 1.

######  Theorem 14 (Local Logarithmic Sobolev Inequality).

A Markov diffusion process $\\{X_{t},~{}t\geq 0\\}$ satisfies a curvature
dimension inequality $\textsc{CD}(0,\infty)$ if and only if it satisfies the
_local logarithmic Sobolev inequality_

$P_{t}(f\ln f)-P_{t}f\ln P_{t}f\leq tP_{t}\left(\frac{\Gamma f}{f}\right)$
(C.1)

and the _reverse local logarithmic Sobolev inequality_

$P_{t}(f\ln f)-P_{t}f\ln P_{t}f\geq t\,\frac{\Gamma(P_{t}f)}{P_{t}f}$ (C.2)

for smooth positive functions $f:\mathbb{R}^{d}\to\mathbb{R}_{+}$.

We emphasize that the presentation as _local_ logarithmic Sobolev
inequalities, where local refers to the law of the diffusion processes at time
$t<\infty$, is key to obtaining the reverse inequality. On the contrary, it is
customary to present the ergodic ($t\to\infty$) formulation of the logarithmic
Sobolev inequality, but at the expense of obtaining a vacuous reverse
inequality.

The central idea towards converting Eqs (C.1) and (C.2) into Theorem 1 is to
specialize to the $d$-dimensional _heat process_

$\mathrm{d}X_{t}=\mathrm{d}W_{t},\quad X_{0}=x_{0},$ (C.3)

where $W_{t}$ denotes $d$-dimensional Brownian motion. While the heat process
does not have an invariant measure, it suffices to observe that the law of
$X_{t}$ at any time $t>0$ is isotropic Gaussian with mean $x_{0}$ and marginal
variance $t$. It is straightforward to verify that this Markov diffusion
process satisfies the $\mathrm{CD}(0,d)$ curvature dimension inequality333also
synonymously referred to as the _Bakry-Émery criterion_.; see [8, Definition
1.16.1] for precise details. In addition, note that if a diffusion process
satisfies $\textsc{CD}(\rho,n)$, then it also satisfies
$\textsc{CD}(\rho^{\prime},n^{\prime})$ for any $n^{\prime}\geq n$ and
$\rho^{\prime}\leq\rho$, hence the heat process is a valid choice for the
local logarithmic Sobolev inequality.

The _Markov semigroup_ operator of the heat process is given by
$P_{t}f(x):=\mathbb{E}[f(X_{t})\mid
X_{0}=x]=\mathbb{E}_{Z\sim\mathcal{N}(0,I_{d})}[f(x+tZ)],$ while its _carré du
champ_ operator is

$\Gamma f:=\Gamma(f,f)=\frac{1}{2}\|\nabla f\|_{2}^{2}.$ (C.4)

Accordingly, fix $t=1$ and, without loss of generality, consider the initial
condition $x_{0}=0$ (alternatively, we can choose the test functions
$g(\cdot)=f(\cdot-x_{0})$). Then, the law of $X_{1}|X_{0}=x_{0}$ is the $d$
dimensional isotropic Gaussian $\mu$, and we have
$\textsc{Ent}_{\mu}(f)=\textsc{Ent}_{P_{1}}(f)$, which matches the left hand
side of both (C.1) and (C.2). Substituting the carré du champ (C.4) into (C.1)
and simplifying via the logarithmic derivative obtains the desired upper bound
(2.4). To obtain the reverse inequality, note
$P_{1}f(x_{0})=\mathbb{E}_{Z\sim\mu}[f(x_{0}+Z)]$ so that
$\nabla_{x}P_{1}f(x_{0})=\mathbb{E}_{Z\sim\mu}[\nabla_{z}f(x_{0}+Z)]$.
Evaluating the carré du champ $\Gamma P_{1}f(0)=\frac{1}{2}\|\mu(\nabla
f)\|_{2}^{2}$ obtains (C.2) and concludes the proof.

###  C.2 Dimensional Gaussian Logarithmic Sobolev Inequality

A proof of the dimensional logarithmic Sobolev inequality (and its reverse)
for general Markov diffusion processes satisfying specific curvature-dimension
inequalities is provided in [8, Thm 6.7.4]. Specifying to the heat process, as
above, then recovers the (weaker) dimensional Gaussian LSI of [10].

However, instead we choose to follow the original presentation of [10], later
improved by Dembo [22], as this approach clarifies the dependence on the
Gaussianity of the reference measure. Neither reference documents the reverse
inequality, but these authors were undoubtedly aware of its existence. Since
we were unable to find this result explicitly in the literature, we provide
its statement and proof below.

The proof proceeds by transforming the Gaussian LSI into Euclidean form by
choosing the test function
$f(x)=((2\pi)^{-d/2}\exp(-\frac{1}{2}\|x\|_{2}^{2}))^{-1}g(x)$, with
$g:\mathbb{R}^{d}\to\mathbb{R}_{+}$ normalized as $\int g(x)\mathrm{d}x=1$ (so
that $\int f\mathrm{d}\mu=1$). Applied to the inequalities in (2.4), this
yields the _Euclidean logarithmic Sobolev inequality_

$\int g(x)\ln g(x)\mathrm{d}x\leq\frac{1}{2}\int\frac{\|\nabla
g(x)\|_{2}^{2}}{g(x)}\mathrm{d}x-\frac{d}{2}\ln(2\pi)-d,$ (C.5)

as well as the _reverse Euclidean logarithmic Sobolev inequality_

$\int g(x)\ln g(x)\mathrm{d}x\geq\frac{1}{2}\left\|\int
xg(x)\mathrm{d}x\right\|_{2}^{2}-\frac{d}{2}\ln(2\pi)-\frac{1}{2}\int
g\|x\|_{2}^{2}\mathrm{d}x,$ (C.6)

after noting that $\int\partial_{k}g(x)\mathrm{d}x=0$ for all $1\leq k\leq d$.

The goal is to bootstrap this to obtain dimensional improvements to the
Euclidean LSI. The original insight of [10] is to recognize that the test
functions $g$ are probability density functions. This suggests considering
$g(x)=(T^{-1})_{\\#}f(x)=|\det\nabla T(x)|f(T(x))$ for
$f:\mathbb{R}^{d}\to\mathbb{R}_{+}$ and $\int f(x)\mathrm{d}x=1$, i.e., the
pushforward of a probability density $f$ under diffeomorphisms
$T^{-1}:\mathbb{R}^{d}\to\mathbb{R}^{d}$, and optimizing the bounds with
respect to $T$ (or $T^{-1}$). Specifically, Bakry and Ledoux considered the
map $T(x)=\sigma x$, optimizing over the scalar parameter $\sigma>0$. Dembo
[22] generalized this construction to maps $T(x)=Ax$ with symmetric positive
definite matrices $A=A^{\top}\succ 0$.

Applying this choice of test function into (C.5), we obtain

$\int f(x)\ln
f(x)\mathrm{d}x\leq\frac{1}{2}\operatorname{tr\,}\left(A^{2}\left(\int\frac{\nabla
f(x)^{\otimes
2}}{f(x)}\mathrm{d}x\right)\right)-\frac{d}{2}\ln(2\pi)-d-\frac{1}{2}\ln\det(A^{2}),$

and straightforward computation shows that $A^{-2}=\int(\nabla\ln
f(x))^{\otimes 2}f(x)\mathrm{d}x$ optimizes the bound. This yields the
_dimensional_ Euclidean LSI

$\int f(x)\ln f(x)\mathrm{d}x\leq\frac{1}{2}\ln\det\left(\int\frac{\nabla
f(x)^{\otimes 2}}{f(x)}\mathrm{d}x\right)-\frac{d}{2}\ln(2\pi e).$ (C.7)

The same exercise applied to (C.6) yields

$\int f(x)\ln f(x)\mathrm{d}x\geq\frac{1}{2}\left\|A^{-1}\int
xf(x)\mathrm{d}x\right\|_{2}^{2}-\frac{d}{2}\ln(2\pi)-\frac{1}{2}\int\|A^{-1}x\|_{2}^{2}f(x)\mathrm{d}x-\ln\det
A.$

For this lower bound, the optimal matrix is $A^{2}=\int x^{\otimes
2}f(x)\mathrm{d}x-\left(\int xf(x)\mathrm{d}x\right)^{\otimes 2}$, from which
we obtain the _dimensional_ reverse Euclidean LSI

$\int f(x)\ln f(x)\mathrm{d}x\geq-\frac{1}{2}\ln\det\left(\int x^{\otimes
2}f(x)\mathrm{d}x-\left(\int xf(x)\mathrm{d}x\right)^{\otimes
2}\right)-\frac{d}{2}\ln(2\pi e).$ (C.8)

It remains to convert these inequalities for the Gaussian measure by
considering test functions
$f(x)=g(x)(2\pi)^{d/2}\exp(-\frac{1}{2}\|x\|_{2}^{2})$, normalized $\int
f(x)\mathrm{d}x=1$ so that $\int g\mathrm{d}\mu=1$. Substituting this into
(C.7) obtains (2.7), and (C.8) obtains (2.8), by applying the Gaussian
integration by parts identity $\int
x_{k}g(x)\mathrm{d}\mu(x)=\int\partial_{k}g(x)\mathrm{d}\mu(x)$.

##  Appendix D Miscellaneous Proofs

###  D.1 Proof of Lower Bound in Theorem 2

We only detail the proof of the lower bound in Theorem 2 below since the upper
bound is extensively documented in [48].

###### Proof.

Without loss of generality, we assume that
$\ell:\mathbb{R}^{d}\rightarrow\mathbb{R}_{+}$ is normalized such that
$\int\ell\mathrm{d}\mu=1$. Given a matrix $U_{r}\in\mathbb{R}^{d\times r}$
with orthogonal columns and $x_{r}\in\mathbb{R}^{r}$, we consider the function
$f:\mathbb{R}^{d-r}\rightarrow\mathbb{R}$ defined by

$f(x_{\perp})=\frac{\ell(U_{r}x_{r}+U_{\perp}x_{\perp})}{\widetilde{\ell}^{\mathrm{KL}}_{r}(x_{r})},$
(D.1)

where $\widetilde{\ell}^{\mathrm{KL}}_{r}$ as in (2.1) and $U_{\perp}$ is any
orthogonal complement of $U_{r}$. By definition of
$\widetilde{\ell}^{\mathrm{KL}}_{r}$, we have $\int f\mathrm{d}\mu_{\perp}=1$.
Thus, the LSI (2.4) for the $(d-r)$ dimensional Gaussian measure $\mu_{\perp}$
yields

$\frac{1}{2}\left\|\int\nabla f\mathrm{d}\mu_{\perp}\right\|^{2}\leq\int f\ln
f\mathrm{d}\mu_{\perp}.$

Next, we integrate of the above inequality with respect to $x_{r}$ against the
probability measure
$\mathrm{d}\pi_{r}(x_{r})=\widetilde{\ell}^{\mathrm{KL}}_{r}(x_{r})\mathrm{d}\mu_{r}(x_{r})$,
where $\mu_{r}$ is the $r$-dimensional isotropic Gaussian measure (recall that
$\int\ell^{\mathrm{KL}}_{r}\mathrm{d}\mu_{r}=\int\ell\mathrm{d}\mu=1$). The
right-hand side becomes

$\displaystyle\int\left(\int f\ln
f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}$
$\displaystyle=\int\ell(x)\ln\frac{\ell(x)}{\widetilde{\ell}^{\mathrm{KL}}_{r}(U_{r}^{\top}x)}\mathrm{d}\mu(x)=\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{r})).$
(D.2)

Using Jensen’s inequality, the left-hand side satisfies

$\displaystyle\int\left(\frac{1}{2}\left\|\int\nabla
f\mathrm{d}\mu_{\perp}\right\|^{2}\right)\mathrm{d}\pi_{r}$
$\displaystyle\geq\frac{1}{2}\left\|\int\left(\int\nabla
f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}\right\|^{2}=\frac{1}{2}\left\|U_{\perp}^{\top}\int\nabla\ln\ell\mathrm{d}\pi\right\|^{2}.$

It remains to show that

$\int\nabla\ln\ell\mathrm{d}\pi=\int\nabla\ell\mathrm{d}\mu=-\int\ell\nabla\ln\left(\frac{\mathrm{d}\mu}{\mathrm{d}x}\right)\mathrm{d}\mu=\int
x\ell\mathrm{d}\mu=\mathbb{E}_{\pi}[X],$

where we have used integration by parts. Combining the above relations yields
(2.5). ∎

###  D.2 Proof of Theorem 4

###### Proof.

As in the proof of Theorem 1, we consider the function
$f:\mathbb{R}^{d-r}\rightarrow\mathbb{R}$ defined as in (D.1) with $\ell$
assumed to be normalized such that $\int\ell\mathrm{d}\mu=1$. Applying the
dimensional LSI (2.7) with $\mu_{\perp}$ yields

$\displaystyle\int f\ln f\mathrm{d}\mu_{\perp}$
$\displaystyle\leq\frac{1}{2}\int|x_{\perp}|^{2}f\mathrm{d}\mu_{\perp}-\frac{d-r}{2}+\frac{1}{2}\ln\det\left(\int\left(\nabla\ln
f-x_{\perp}\right)^{\otimes 2}f\mathrm{d}\mu_{\perp}\right)$ (D.3)

Next, we integrate the inequality above with respect to $x_{r}$ against the
probability measure
$\mathrm{d}\pi_{r}=\widetilde{\ell}^{\mathrm{KL}}_{r}\mathrm{d}\mu_{r}$, where
$\mu_{r}$ is the $r$-dimensional isotropic Gaussian measure. As before,
following (D.2), the left-hand side of (D.3) becomes $\int\left(\int f\ln
f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}=\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{r}))$.
The first term in the right-hand of (D.3) side becomes

$\displaystyle\int\left(\frac{1}{2}\int|x_{\perp}|^{2}f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}$
$\displaystyle=\frac{1}{2}\int|U_{\perp}^{\top}x|^{2}\ell\mathrm{d}\mu=\frac{1}{2}\int|U_{\perp}^{\top}x|^{2}\mathrm{d}\pi.$

Finally, using Jensen’s inequality (noting that $S\mapsto\ln\det(S)$ is
concave), the last term in the right-hand of (D.3) side satisfies

$\displaystyle\int\frac{1}{2}\ln\det\left(\int\left(\nabla\ln
f-x_{\perp}\right)^{\otimes 2}f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}$
$\displaystyle\leq\frac{1}{2}\ln\det\left(\int\left(\int\left(\nabla\ln
f-x_{\perp}\right)^{\otimes
2}f\mathrm{d}\mu_{\perp}\right)\mathrm{d}\pi_{r}\right)$
$\displaystyle=\frac{1}{2}\ln\det\left(\int\left(U_{\perp}^{\top}(\nabla\ln\ell-x)\right)^{\otimes
2}\mathrm{d}\pi\right).$

Injecting the three above relations in (D.3) yields the right-bound of (2.9).
The lower bound is obtained in a similar way. This concludes the proof. ∎

###  D.3 Proof of Proposition 5

We require the following lemma.

######  Lemma 15.

Let $v\in\mathbb{R}^{d}$ such that $v^{\top}v>0$ and $W\in\mathbb{R}^{d\times
k}$, $1\leq k<d$, such that $W^{\top}W\succeq 0$. Then we have the inequality

$\ln\det v^{\top}v+\ln\det
W^{\top}W\leq\ln\det\begin{pmatrix}v^{\top}v&v^{\top}W\\\
W^{\top}v&W^{\top}v\end{pmatrix}.$ (D.4)

###### Proof.

Re-write (D.4) as

$\ln\det\begin{pmatrix}v^{\top}v&\\\
&I_{k}\end{pmatrix}+\ln\det\begin{pmatrix}1&\\\
&W^{\top}W\end{pmatrix}\leq\ln\det\begin{pmatrix}v^{\top}v&v^{\top}W\\\
W^{\top}v&W^{\top}v\end{pmatrix},$

where $I_{k}$ denotes the $k\times k$ identity matrix. Re-arranging then gives

$\ln\det\begin{pmatrix}1&\theta^{\top}\\\
\theta&I_{k}\end{pmatrix}=\ln\det\left(I_{k+1}+\begin{pmatrix}0\\\
\theta\end{pmatrix}\begin{pmatrix}0&\theta^{\top}\end{pmatrix}\right)\geq 0$

with $\theta=\frac{1}{\sqrt{v^{\top}v}}(W^{\top}W)^{-\frac{1}{2}}W^{\top}v$.
But since $\det\left(I_{k+1}+\begin{pmatrix}0\\\
\theta\end{pmatrix}\begin{pmatrix}0&\theta^{\top}\end{pmatrix}\right)=1+\|\theta\|_{2}^{2}$,
this concludes the proof. ∎

We now demonstrate that $U_{r}\mapsto\mathcal{J}^{\downarrow}_{d,r}(U_{r})$ is
monotonically decreasing for nested sequences.

###### Proof.

Denoting $U_{r}=\begin{pmatrix}u_{1}&\ldots u_{r}\end{pmatrix}$ and
$U_{\perp,r}=\begin{pmatrix}u_{r+1}&\ldots&u_{d}\end{pmatrix}\in\mathbb{R}^{d\times(d-r)}$
for $1\leq r<d$, then

$\mathcal{J}^{\downarrow}(U_{r})=\frac{1}{2}\operatorname{tr\,}U_{\perp,r}^{\top}M(\pi)U_{\perp,r}-\frac{d-r}{2}+\frac{1}{2}\ln\det\left(U_{\perp,r}^{\top}H(\pi)U_{\perp,r}\right)$
(D.5)

and

$\mathcal{J}^{\downarrow}(U_{r+1})=\frac{1}{2}\operatorname{tr\,}U_{\perp,r+1}^{\top}M(\pi)U_{\perp,r+1}-\frac{d-r-1}{2}+\frac{1}{2}\ln\det\left(U_{\perp,r+1}^{\top}H(\pi)U_{\perp,r+1}\right).$

Demonstrating
$\mathcal{J}^{\downarrow}(U_{r+1})\leq\mathcal{J}^{\downarrow}(U_{r})$ amounts
to the inequality

$1-u_{r+1}^{\top}M(\pi)u_{r+1}+\ln\det\left(U_{\perp,r+1}^{\top}H(\pi)U_{\perp,r+1}\right)\leq\ln\det\left(U_{\perp,r}^{\top}H(\pi)U_{\perp,r}\right).$

The key idea is to recognize that by defining the orthonormal sub-matrix

$V=\begin{pmatrix}u_{1}&\ldots&u_{r}&u_{r+2}&\ldots&u_{d}\end{pmatrix}\in\mathbb{R}^{d\times(d-1)},$

we have

$0\leq\mathcal{J}^{\downarrow}(V)=\frac{u_{r+1}^{\top}M(\pi)u_{r+1}-1}{2}+\frac{1}{2}\ln\det\left(u_{r+1}^{\top}H(\pi)u_{r+1}\right),$

where non-negativity is ensured since

$\operatorname{D_{\mathrm{KL}}}(\pi\,\|\,\widetilde{\pi}^{\mathrm{KL}}(U^{\downarrow}_{d-1}))\leq\min_{U_{d-1}}\mathcal{J}^{\downarrow}(U_{d-1})\leq\mathcal{J}^{\downarrow}(V).$

It therefore suffices to show that

$\ln\det\left(u_{r+1}^{\top}H(\pi)u_{r+1}\right)+\ln\det\left(U_{\perp,r+1}^{\top}H(\pi)U_{\perp,r+1}\right)\leq\ln\det\left(U_{\perp,r}^{\top}H(\pi)U_{\perp,r}\right).$

But this follows from applying Lemma 15 with $v=H(\pi)^{-\frac{1}{2}}u_{r+1}$
and $W=H(\pi)^{-\frac{1}{2}}U_{\perp,r+1}$. ∎

###  D.4 Proof of Theorem 8

###### Proof.

Let $T(x)=C(\pi)^{-1/2}(x-m(\pi))$ be the affine map such that
$\overline{\pi}=T_{\sharp}\pi$ is centered and isotropic, that is,
$m(\overline{\pi})=0$ and $M(\overline{\pi})=I_{d}$. Applying Theorem 4 to
$\overline{\pi}$ yields

$\operatorname{D_{\mathrm{KL}}}(\overline{\pi}||\overline{\widetilde{\pi}^{\mathrm{KL}}}(\overline{U_{r}}))\leq\frac{1}{2}\ln\det(H(\overline{\pi}))-\frac{1}{2}\ln\det(\overline{U_{r}}^{\top}H(\overline{\pi})\overline{U_{r}}),$
(D.6)

for any matrix $\overline{U_{r}}\in\mathbb{R}^{d\times r}$ with orthogonal
columns. Here
$\overline{\widetilde{\pi}^{\mathrm{KL}}}(\overline{U_{r}})=\overline{\widetilde{\pi}^{\mathrm{KL}}}(\cdot\mid\overline{U_{r}})$
is defined by
$\mathrm{d}\overline{\widetilde{\pi}^{\mathrm{KL}}}(\overline{x}\mid\overline{U_{r}})=\overline{\ell^{\mathrm{KL}}_{r}}(\overline{U_{r}}^{\top}\overline{x})\mathrm{d}\overline{\mu}(\overline{x}),$
where $\overline{\mu}=\mathcal{N}(0,I_{d})$ and
$\overline{\ell^{\mathrm{KL}}_{r}}(\theta_{r})=\mathbb{E}_{\overline{X}\sim\overline{\mu}}[\frac{\mathrm{d}\overline{\pi}}{\mathrm{d}\overline{\mu}}(\overline{X})\mid\overline{U_{r}}^{\top}\overline{X}=\theta_{r}]$.
Next, we use a change of variable $x=T^{-1}(\overline{x})$ to show that (D.6)
yields (2.16). We can write

$\displaystyle\overline{\ell^{\mathrm{KL}}_{r}}(\overline{U_{r}}^{\top}\overline{x})$
$\displaystyle=\mathbb{E}_{\overline{X}\sim\overline{\mu}}\left[\left.\frac{\mathrm{d}\overline{\pi}}{\mathrm{d}\overline{\mu}}(\overline{X})\right|\overline{U_{r}}^{\top}\overline{X}=\overline{U_{r}}^{\top}\overline{x}\right]$
$\displaystyle=\mathbb{E}_{X\sim\mu}\left[\left.\frac{\mathrm{d}\pi}{\mathrm{d}\mu}(X)\right|\overline{U_{r}}^{\top}T(X)=\overline{U_{r}}^{\top}T(x)\right]$
$\displaystyle=\mathbb{E}_{X\sim\mu}\left[\left.\frac{\mathrm{d}\pi}{\mathrm{d}\mu}(X)\right|\overline{U_{r}}^{\top}C(\pi)^{-1/2}X=\overline{U_{r}}^{\top}C(\pi)^{-1/2}x\right]=\ell^{\mathrm{KL}}_{r}(U_{r}^{\top}x),$

where

$U_{r}=C(\pi)^{-1/2}\overline{U_{r}}.$

We deduce that
$\mathrm{d}\overline{\widetilde{\pi}^{\mathrm{KL}}}(\overline{U_{r}})=\ell^{\mathrm{KL}}_{r}(U_{r}^{\top}T^{-1}(\cdot))\mathrm{d}(T_{\sharp}\mu)=T_{\sharp}\widetilde{\pi}^{\mathrm{KL}}(U_{r})$
so that

$\operatorname{D_{\mathrm{KL}}}(\overline{\pi}||\overline{\widetilde{\pi}^{\mathrm{KL}}}(\overline{U_{r}}))=\operatorname{D_{\mathrm{KL}}}(T_{\sharp}\pi||T_{\sharp}\widetilde{\pi}^{\mathrm{KL}}(U_{r}))=\operatorname{D_{\mathrm{KL}}}(\pi||\widetilde{\pi}^{\mathrm{KL}}(U_{r})),$

for any $U_{r}$ such that $U_{r}^{\top}C(\pi)U_{r}=I_{r}$. We conclude that
the left-hand side of (D.6) and (2.16) are the same. Furthermore, because
$\mathrm{d}\overline{\pi}(\overline{x})=\mathrm{d}\pi(T^{-1}(\overline{x}))|\det\nabla
T^{-1}(\overline{x})|$ and
$T^{-1}(\overline{x})=C(\pi)^{1/2}\overline{x}+m(\pi)$, we can write

$H(\overline{\pi})=\mathbb{E}_{\overline{\pi}}\left[C(\pi)^{1/2}\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\circ
T^{-1}\right)\nabla\ln\left(\frac{\mathrm{d}\pi}{\mathrm{d}x}\circ
T^{-1}\right)^{\top}C(\pi)^{1/2}\right]=C(\pi)^{1/2}H(\pi)C(\pi)^{1/2}.$

We conclude that the right-hand side of (D.6) and (2.16) are the same.
Finally, evaluating (2.16) at $U_{r}=U_{r}^{\downarrow}$ yields (2.17) and
concludes the proof. ∎

###  D.5 Proof of Theorem 9

###### Proof.

By direct application of Theorem 4 we have the relation

$\operatorname{D_{\mathrm{KL}}}(\pi^{Y}\,||\,\widetilde{\pi}^{Y}(V_{r}))\leq\frac{1}{2}\operatorname{tr\,}(V_{\perp}^{\top}M^{Y}V_{\perp})-\frac{d-r}{2}+\frac{1}{2}\ln\det\left(V_{\perp}^{\top}(2I_{d}+H^{Y}-M^{Y})V_{\perp}\right),$

where $M^{Y}:=\mathbb{E}_{X\sim\pi^{Y}}[XX^{\top}]$ and
$H^{Y}:=\mathbb{E}_{X\sim\pi^{Y}}[\nabla\ln\ell(X)\nabla\ln\ell(X)^{\top}]$
denote matrices which are measurable with respect to the random variable $Y$.
It remains to to integrate both sides of the inequality with respect to the
law $\rho$ of this random variable. However, recognizing that
$\mathbb{E}_{Y\sim\rho}[M^{Y}]=I_{d}$ and
$\mathbb{E}_{Y\sim\rho}[H^{Y}]=H^{\mathrm{df}}$, applying Jensen’s inequality
to the concave matrix-valued function $S\mapsto\ln\det S$ concludes the proof.
∎

###  D.6 Proof of Theorem 13

The following lemma is required for the proof of Theorem 13.

######  Lemma 16.

For all $\mu$-integrable functions $f$ and $g$ we have the inequality
$\frac{\mathbb{E}_{X\sim\mu}[f(X)]^{2}}{\mathbb{E}_{X\sim\mu}[g(X)]}\leq\mathbb{E}_{X\sim\mu}\left[\frac{f(X)^{2}}{g(X)}\right].$

###### Proof.

This follows from Jensen’s inequality as $(x,y)\mapsto\frac{x^{2}}{y}$ is
jointly convex444A bivariate function $(x,y)\mapsto f(x,y)$ is _jointly
convex_ if for all $\lambda\in[0,1]$, and for all pairs $(x_{1},y_{1})$ and
$(x_{2},y_{2})$, it satisfies the inequality $f(\lambda
x_{1}+(1-\lambda)x_{2},\lambda y_{1}+(1-\lambda)y_{2})\leq\lambda
f(x_{1},y_{1})+(1-\lambda)f(x_{2},y_{2})$. (see [15, §3.2.6]). ∎

The proof of Theorem 13 then proceeds as follows.

###### Proof.

Without loss of generality, assume $\ell:\mathbb{R}^{d}\to\mathbb{R}^{+}$ is
normalized so that $\mathrm{d}\pi=\ell\mathrm{d}\mu$, or equivalently,
$\int\ell\mathrm{d}\mu=1$. Given a matrix $U_{r}\in\mathbb{R}^{d\times r}$
with orthonormal columns and $x_{r}\in\mathbb{R}^{r}$, consider the function
$f:\mathbb{R}^{d-r}\to\mathbb{R}$ given by

$f(x_{\perp})=\sqrt{\ell(U_{r}x_{r}+U_{\perp}x_{\perp})}$

with $U_{\perp}$ as any orthogonal complement of $U_{r}$. By construction we
have $\int f^{2}\mathrm{d}\mu=1$. For this choice of test function, the
dimensional Poincaré inequality (3.6) for the $(d-r)$ dimensional Gaussian
measure $\mu_{\perp}$ yields

$\int f^{2}\mathrm{d}\mu_{\perp}-\left(\int
f\mathrm{d}\mu_{\perp}\right)^{2}\leq\int|\nabla\ln
f|^{2}f^{2}\mathrm{d}\mu_{\perp}-\int\frac{(1-\frac{1}{f}\int
f\mathrm{d}\mu_{\perp}-\langle\nabla\ln
f,x_{\perp}\rangle)^{2}}{d-r+|x_{\perp}|^{2}}f^{2}\mathrm{d}\mu_{\perp}.$
(D.7)

We next integrate the above inequality with respect to $x_{r}$ with law
$\mu_{r}$ given by the $r$-dimensional isotropic Gaussian measure. By (3.3),
the left-hand side of (D.7) becomes
$1-\int\ell_{r}^{\mathrm{Hell}}\mathrm{d}\mu_{r}=1-(1-d_{\mathrm{Hell}}^{2}(\pi\,,\,\tilde{\pi}_{r}^{\mathrm{Hell}}(U_{r})))^{2}$.
The first term in right hand of (D.7) becomes $\int|\nabla\ln
f|^{2}f^{2}\mathrm{d}\mu=\int\frac{1}{4}|U_{\perp}\nabla\ln\ell|^{2}\mathrm{d}\pi,$
after recognizing that $\mathrm{d}\pi=f^{2}\mathrm{d}\mu$. Meanwhile, applying
Lemma 16 to the latter term in the right-hand side of (D.7) shows that

$-\int\frac{(1-\frac{1}{f}\int f\mathrm{d}\mu_{\perp}-\langle\nabla\ln
f,x_{\perp}\rangle)^{2}}{d-r+|x_{\perp}|^{2}}\mathrm{d}\pi\leq-\frac{(1-\mathbb{E}_{\pi}[\frac{1}{\sqrt{\ell}}\int\sqrt{\ell}\mathrm{d}\mu_{\perp}]-\frac{1}{2}\mathbb{E}_{\pi}[\langle
U_{\perp}^{\top}\nabla\ln\ell,U_{\perp}^{\top}x\rangle)^{2}}{d-r+\mathbb{E}_{\pi}[|U_{\perp}^{\top}x|^{2}]}.$
(D.8)

This can be further simplified, first by noting that

$1-\mathbb{E}_{\pi}\left[\frac{1}{\sqrt{\ell}}\int\sqrt{\ell}\mathrm{d}\mu_{\perp}\right]\overset{\eqref{eq:optHell}}{=}1-(1-d_{\mathrm{Hell}}^{2}(\pi\,,\,\tilde{\pi}_{r}^{\mathrm{Hell}}(U_{r})))^{2},$

and secondly by noting that

$\int\nabla\ln\ell(x)x^{\top}\mathrm{d}\pi(x)=\int
xx^{\top}\mathrm{d}\pi(x)-I,$

from applying Stein’s identity (i.e., integration by parts) with the standard
Gaussian $\mu$. ∎

## Acknowledgments

ML and YM thank Brent Wheelock, Dimitar Trenev, Kevin Daly, Grant Seastream,
and Tuan Tran at ExxonMobil for the code and discussions related to Bayesian
inference for geophysical applications with generative modeling priors. ML
also thanks Nicolas Boumal for discussions on the ManOpt forum related to
optimization on matrix manifolds.

## Funding

ML and YMM acknowledge support from the US Department of Energy, Office of
Advanced Scientific Computing Research, under grant DE-SC0023187 and from the
ExxonMobil Technology and Engineering Company. YMM further acknowledges
support from the US Department of Energy, Office of Advanced Scientific
Computing Research, under grants DE-SC0021226 and DE-SC0023188. TC is
supported by the ARC Discovery Project DP210103092. OZ acknowledges support
from the ANR JCJC project MODENA (ANR-21-CE46-0006-01).

## References

  * AGG [16] Alen Alexanderian, Philip J. Gloor, and Omar Ghattas.  On Bayesian A- and D-Optimal Experimental Designs in Infinite Dimensions.  Bayesian Analysis, 11(3):671–695, September 2016.
  * ALPW [24] C Andrieu, A Lee, S Power, and Andi Q Wang.  Explicit convergence bounds for metropolis markov chains: isoperimetry, spectral gaps and profiles.  The Annals of Applied Probability, 2024.
  * Ama [09] Shun-Ichi Amari.  $\alpha$-divergence is unique, belonging to both $f$-divergence and Bregman divergence classes.  IEEE Transactions on Information Theory, 55(11):4925–4931, Nov 2009.
  * AS [21] Kwangjun Ahn and Felipe Suarez.  Riemannian Perspective on Matrix Factorization, February 2021\.
  * AV [23] Foivos Alimisis and Bart Vandereycken.  Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Riemannian Steepest Descent, September 2023.
  * BG [10] François Bolley and Ivan Gentil.  Phi-entropy inequalities for diffusion semigroups.  Journal de Mathématiques Pures et Appliquées, 93(5):449–473, May 2010.
  * BGG [18] François Bolley, Ivan Gentil, and Arnaud Guillin.  Dimensional improvements of the logarithmic Sobolev, Talagrand and Brascamp–Lieb inequalities.  The Annals of Probability, 46(1):261–301, Jan 2018.
  * BGL [14] Dominique Bakry, Ivan Gentil, and Michel Ledoux.  Analysis and Geometry of Markov Diffusion Operators.  Springer International Publishing, 2014.
  * BJM [16] Michel Bonnefont, Aldéric Joulin, and Yutao Ma.  Spectral gap for spherically symmetric log-concave probability measures, and beyond.  Journal of Functional Analysis, 270(7):2456–2482, Apr 2016.
  * BL [06] Dominique Bakry and Michel Ledoux.  A logarithmic Sobolev form of the Li-Yau parabolic inequality.  Revista Matemática Iberoamericana, 22(2):683–702, Sep 2006.
  * BL [09] Sergey G. Bobkov and Michel Ledoux.  Weighted Poincaré-type inequalities for Cauchy and other convex measures.  The Annals of Probability, 37(2):403–427, Mar 2009.
  * BMPZ [21] Daniele Bigoni, Youssef Marzouk, Clémentine Prieur, and Olivier Zahm.  Nonlinear dimension reduction for surrogate modeling using gradient information.  arXiv:2102.10351 [cs, math], February 2021.
  * BMZ [22] Ricardo Baptista, Youssef Marzouk, and Olivier Zahm.  Gradient-based data and parameter dimension reduction for bayesian models: an information theoretic perspective.  (arXiv:2207.08670), Jul 2022.  arXiv:2207.08670 [math, stat].
  * Bou [23] Nicolas Boumal.  An Introduction to Optimization on Smooth Manifolds.  Cambridge University Press, 2023.
  * BV [04] Stephen Boyd and Lieven Vandenberghe.  Convex Optimization.  Cambridge University Press, 2004.
  * CABZ [24] Qiao Chen, Élise Arnaud, Ricardo Baptista, and Olivier Zahm.  Coupled Input-Output Dimension Reduction: Application to Goal-Oriented Bayesian Experimental Design and Global Sensitivity Analysis.  (preprint), 2024.
  * CDZ [22] Tiangang Cui, Sergey Dolgov, and Olivier Zahm.  Scalable conditional deep inverse rosenblatt transports using tensor trains and gradient-based dimension reduction.  Journal of Computational Physics, 485:112103, 2022.
  * Cha [04] Djalil Chafaï.  Entropies, convexity, and functional inequalities.  Kyoto Journal of Mathematics, 44(2), Jan 2004.  arXiv: math/0211103.
  * CMM+ [14] Tiangang Cui, James Martin, Youssef M. Marzouk, Antti Solonen, and Alessio Spantini.  Likelihood-informed dimension reduction for nonlinear inverse problems.  Inverse Problems, 30(11):114015, Nov 2014.  arXiv: 1403.4680.
  * CT [22] Tiangang Cui and Xin T. Tong.  A unified performance analysis of likelihood-informed subspace methods.  Bernoulli, 28(4):2788–2815, 2022.
  * CZ [21] Tiangang Cui and Olivier Zahm.  Data-free likelihood-informed dimension reduction of Bayesian inverse problems.  Inverse Problems, 37(4):045009, Mar 2021.
  * Dem [90] Amir Dembo.  Information inequalities and uncertainty principles. Technical report no. 75.  Technical report, Department of Statistics, Stanford University, 1990\.
  * EAS [98] Alan Edelman, Tomás A. Arias, and Steven T. Smith.  The Geometry of Algorithms with Orthogonality Constraints.  SIAM Journal on Matrix Analysis and Applications, 20(2):303–353, January 1998.
  * ES [24] Alexandros Eskenazis and Yair Shenfeld.  Intrinsic dimensional functional inequalities on model spaces.  Journal of Functional Analysis, 286(7):110338, 2024.
  * FDUZ [23] Rafael Flock, Yiqiu Dong, Felipe Uribe, and Olivier Zahm.  Certified coordinate selection for high-dimensional bayesian inversion with laplace prior.  2023\.
  * GPAM+ [14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672–2680. Curran Associates, Inc., 2014.
  * GPM+ [14] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Networks, June 2014.
  * Gro [75] Leonard Gross.  Logarithmic Sobolev inequalities.  American Journal of Mathematics, 97(4):1061–1083, 1975.
  * GW [21] Omar Ghattas and Karen Willcox.  Learning physics-based models from data: Perspectives from inverse problems and model reduction.  Acta Numerica, 30:445–554, May 2021.
  * HS [87] Richard Holley and Daniel Stroock.  Logarithmic Sobolev inequalities and stochastic Ising models.  Journal of Statistical Physics, 46(5):1159–1194, March 1987.
  * KS [05] Jari Kaipio and Erkki Somersalo.  Statistical and Computational Inverse Problems.  Springer, Dordrecht, 2005.
  * KSD [06] Brian Kulis, Mátyás Sustik, and Inderjit Dhillon.  Learning low-rank kernel matrices.  In Proceedings of the 23rd International Conference on Machine Learning - ICML ’06, pages 505–512, Pittsburgh, Pennsylvania, 2006. ACM Press.
  * LBM [24] Fengyi Li, Ayoub Belhadji, and Youssef Marzouk.  Nonlinear Bayesian optimal experimental design using logarithmic Sobolev inequalities, February 2024.
  * LMZ [24] Matthew T. C. Li, Youssef Marzouk, and Olivier Zahm.  Principal feature detection via $\phi$-Sobolev inequalities.  Bernoulli, 2024.  to appear.
  * MV [18] Oren Mangoubi and Nisheeth K. Vishnoi.  Dimensionally tight bounds for second-order Hamiltonian Monte Carlo.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 6030–6040, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * PBL+ [19] Loucas Pillaud-Vivien, Francis Bach, Tony Lelièvre, Alessandro Rudi, and Gabriel Stoltz.  Statistical Estimation of the Poincar{\’e} constant and Application to Sampling Multimodal Distributions, November 2019.
  * PRO [22] Dhruv V. Patel, Deep Ray, and Assad A. Oberai.  Solution of Physics-based Bayesian Inverse Problems with Deep Generative Priors.  Computer Methods in Applied Mechanics and Engineering, 400:115428, October 2022.
  * PST [12] Natesh S. Pillai, Andrew M. Stuart, and Alexandre H. Thiéry.  Optimal scaling and diffusion limits for the Langevin algorithm in high dimensions.  Ann. Appl. Probab., 22(6):2320–2356, 2012.
  * RR [98] Gareth O. Roberts and Jeffrey S. Rosenthal.  Optimal scaling of discrete approximations to Langevin diffusions.  J. R. Stat. Soc. Ser. B Stat. Methodol., 60(1):255–268, 1998.
  * RvH [15] Patrick Rebeschini and Ramon van Handel.  Can local particle filters beat the curse of dimensionality?  Ann. Appl. Probab., 25(5):2809–2866, 2015.
  * SBBA [08] M. Snyder, Thomas Bengtsson, Peter Bickel, and L. Anderson.  Obstacles to high-dimensional particle filtering.  Mon Weather Rev., page 4629–4640, 2008.
  * SQW [16] Ju Sun, Qing Qu, and John Wright.  When Are Nonconvex Problems Not Scary?, Apr 2016.
  * SSC+ [15] Alessio Spantini, Antti Solonen, Tiangang Cui, James Martin, Luis Tenorio, and Youssef Marzouk.  Optimal Low-rank Approximations of Bayesian Linear Inverse Problems.  SIAM Journal on Scientific Computing, 37(6):A2451–A2487, Jan 2015\.
  * Stu [10] A. M. Stuart.  Inverse problems: A Bayesian perspective.  Acta Numerica, 19:451–559, May 2010.
  * Sun [21] Ju Sun.  Provable nonconvex methods/algorithms.  https://sunju.org/research/nonconvex/, 2021.
  * TKW [16] J. Townsend, N. Koep, and S. Weichwald.  PyManopt: a Python toolbox for optimization on manifolds using automatic differentiation.  Journal of Machine Learning Research, 17(137):1–5, 2016.
  * VPZ [23] Romain Verdière, Clémentine Prieur, and Olivier Zahm.  Diffeomorphism-based feature learning using Poincaré inequalities on augmented input space.  2023\.
  * ZCL+ [22] Olivier Zahm, Tiangang Cui, Kody Law, Alessio Spantini, and Youssef Marzouk.  Certified dimension reduction in nonlinear Bayesian inverse problems.  Mathematics of Computation, 91(336):1789–1835, Apr 2022.