scale inverse problems. Developing the analysis of these methods, following
the approach in this paper, constitutes an interesting direction for futher
study.

###  6.2 Open Technical Problems

Several open technical problems remain unsolved in the work we have presented;
we list them here.

  1. 1.

We study lower bounds on $C_{t}$ in Lemma 3.15 for the affine invariant
Wasserstein gradient flow with $P_{t}=C_{t}.$ We obtain strong results only in
dimension $N_{\theta}=1.$ When $N_{\theta}>1$, determining whether, and if so
when, $C_{t}$ can become singular is an interesting question for further
study, in particular under a logconcavity assumption such as Definition 3.13.

  2. 2.

For the affine invariant Stein gradient flow, study of convergence and
convergence rates for general kernel functions is of interest.

  3. 3.

For the Gaussian approximate Fisher-Rao gradient flow Eq. 82, in particular
under the logconcavity assumption Definition 3.13, we prove a global
convergence rate of at least

$e^{-\alpha\min\\{\frac{1}{\beta},\lambda_{0,\min}\\}t}$

in Proposition 4.10. However, the numerical study showed far superior behavior
on one problem class. A sharp bound is of interest. We study this problem from
the perspective of local convergence rate in Proposition 4.11 for
$N_{\theta}=1$. Studying this problem for $N_{\theta}>1$ is of future
interest.

  4. 4.

For the Gaussian approximate Fisher-Rao gradient flow Eq. 82, under the
logconcave assumption such as Definition 3.13, we prove that $C_{t}$ is
bounded below and above in Proposition 4.10. Furthermore, studying whether
$C_{t}$ can become singular or infinity for the general posterior case is of
future interest.

Acknowledgments YC acknowledge the support from the Air Force Office of
Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning
and Physics-Based Modeling and Simulation). DZH and AMS are supported by the
generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures
program; AMS is also supported by the Office of Naval Research (ONR) through
grant N00014-17-1-2079 and by a Department of Defense Vannevar Bush Faculty
Fellowship. SR is supported by Deutsche Forschungsgemeinschaft (DFG) -
Project-ID 318763901 - SFB1294.

## References

  * [1] S.-I. Amari, Natural gradient works efficiently in learning, Neural computation, 10 (1998), pp. 251–276.
  * [2] S.-i. Amari, Information Geometry and its Applications, vol. 194, Springer, 2016.
  * [3] L. Ambrosio, N. Gigli, and G. Savaré, Gradient flows: in Metric Spaces and in the Space of Probability Measures, Springer Science & Business Media, 2005.
  * [4] D. Applebaum, Lévy processes and stochastic calculus, Cambridge university press, 2009.
  * [5] N. Ay, J. Jost, H. V. Lê, and L. Schwachhöfer, Information geometry and sufficient statistics, Probability Theory and Related Fields, 162 (2015), pp. 327–364.
  * [6] N. Ay, J. Jost, H. V. Lê, and L. Schwachhöfer, Information Geometry, vol. 64, Springer, 2017.
  * [7] D. Bakry and M. Émery, Diffusions hypercontractives, in Seminaire de probabilités XIX 1983/84, Springer, 1985, pp. 177–206.
  * [8] D. Bakry, I. Gentil, M. Ledoux, et al., Analysis and geometry of Markov diffusion operators, vol. 103, Springer, 2014.
  * [9] M. Bauer, M. Bruveris, and P. W. Michor, Uniqueness of the Fisher–Rao metric on the space of smooth densities, Bulletin of the London Mathematical Society, 48 (2016), pp. 499–506.
  * [10] B. M. Bell, The iterated Kalman smoother as a Gauss–Newton method, SIAM Journal on Optimization, 4 (1994), pp. 626–636.
  * [11] B. M. Bell and F. W. Cathey, The iterated Kalman filter update as a Gauss-Newton method, IEEE Transactions on Automatic Control, 38 (1993), pp. 294–297.
  * [12] J.-D. Benamou and Y. Brenier, A computational fluid mechanics solution to the monge-kantorovich mass transfer problem, Numerische Mathematik, 84 (2000), pp. 375–393.
  * [13] J. Bertoin, Lévy processes, vol. 121, Cambridge university press Cambridge, 1996.
  * [14] R. Bhatia, T. Jain, and Y. Lim, On the bures–wasserstein distance between positive definite matrices, Expositiones Mathematicae, 37 (2019), pp. 165–191.
  * [15] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, Variational inference: A review for statisticians, Journal of the American statistical Association, 112 (2017), pp. 859–877.
  * [16] D. Blömker, C. Schillings, P. Wacker, and S. Weissmann, Well posedness and convergence analysis of the ensemble Kalman inversion, Inverse Problems, 35 (2019), p. 085007.
  * [17] N. M. Boffi and E. Vanden-Eijnden, Probability flow solution of the Fokker-Planck equation, arXiv preprint arXiv:2206.04642, (2022).
  * [18] S. Brooks, A. Gelman, G. Jones, and X.-L. Meng, Handbook of Markov chain Monte Carlo, CRC press, 2011.
  * [19] M. Burger, M. Erbar, F. Hoffmann, D. Matthes, and A. Schlichting, Covariance-modulated optimal transport and gradient flows, arXiv preprint arXiv:2302.07773, (2023).
  * [20] E. Calvello, S. Reich, and A. M. Stuart, Ensemble Kalman methods: a mean field perspective, arXiv preprint arXiv:2209.11371, (2022).
  * [21] S. Cao and D. Z. Huang, Bayesian calibration for large-scale fluid structure interaction problems under embedded/immersed boundary framework, International Journal for Numerical Methods in Engineering, (2022).
  * [22] J. A. Carrillo, Y.-P. Choi, C. Totzeck, and O. Tse, An analytical framework for consensus-based global optimization method, Mathematical Models and Methods in Applied Sciences, 28 (2018), pp. 1037–1066.
  * [23] N. N. Cencov, Statistical decision rules and optimal inference, American Mathematical Soc., 2000.
  * [24] N. Chada and X. Tong, Convergence acceleration of ensemble Kalman inversion in nonlinear settings, Mathematics of Computation, 91 (2022), pp. 1247–1280.
  * [25] N. K. Chada, Y. Chen, and D. Sanz-Alonso, Iterative ensemble Kalman methods: a unified perspective with some new variants, arXiv preprint arXiv:2010.13299, (2020).
  * [26] L.-P. Chaintron and A. Diez, Propagation of chaos: a review of models, methods and applications, arXiv preprint arXiv:2106.14812, (2021).
  * [27] P. Chen and O. Ghattas, Projected Stein variational gradient descent, Advances in Neural Information Processing Systems, 33 (2020), pp. 1947–1958.
  * [28] Y. Chen, B. Hosseini, H. Owhadi, and A. M. Stuart, Solving and learning nonlinear PDEs with Gaussian processes, Journal of Computational Physics, 447 (2021), p. 110668.
  * [29] Y. Chen, D. Z. Huang, J. Huang, S. Reich, and A. M. Stuart, Derivative-free multimodal sampling: Fisher-Rao Gradient Flow and Gaussian mixture approximation, in preparation, (2022).
  * [30] Y. Chen and W. Li, Optimal transport natural gradient for statistical manifolds with continuous sample space, Information Geometry, 3 (2020), pp. 1–32.
  * [31] Y. Chen and D. S. Oliver, Ensemble randomized maximum likelihood method as an iterative ensemble smoother, Mathematical Geosciences, 44 (2012), pp. 1–26.
  * [32] Y. Chen, H. Owhadi, and A. Stuart, Consistency of empirical bayes and kernel flow for hierarchical parameter estimation, Mathematics of Computation, 90 (2021), pp. 2527–2578.
  * [33] S. Chewi, T. Le Gouic, C. Lu, T. Maunu, and P. Rigollet, SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence, Advances in Neural Information Processing Systems, 33 (2020), pp. 2098–2109.
  * [34] L. Chizat and F. Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport, Advances in neural information processing systems, 31 (2018).
  * [35] N. Chopin, O. Papaspiliopoulos, et al., An introduction to sequential Monte Carlo, Springer, 2020.
  * [36] S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statistical Science, 28 (2013), pp. 424–446.
  * [37] J. Coullon and R. J. Webber, Ensemble sampler for infinite-dimensional inverse problems, Statistics and Computing, 31 (2021), pp. 1–9.
  * [38] T. Cui, K. J. Law, and Y. M. Marzouk, Dimension-independent likelihood-informed mcmc, Journal of Computational Physics, 304 (2016), pp. 109–137.
  * [39] F. Daum, J. Huang, and A. Noushin, Exact particle flow for nonlinear filters, in Signal Processing, Sensor Fusion, and Target Recognition XIX, I. Kadar, ed., vol. 7697, International Society for Optics and Photonics, SPIE, 2010, pp. 92 – 110, https://doi.org/10.1117/12.839590.
  * [40] P. Del Moral, A. Doucet, and A. Jasra, Sequential Monte Carlo samplers, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68 (2006), pp. 411–436.
  * [41] G. Detommaso, T. Cui, Y. Marzouk, A. Spantini, and R. Scheichl, A Stein variational Newton method, Advances in Neural Information Processing Systems, 31 (2018).
  * [42] Z. Ding and Q. Li, Ensemble Kalman inversion: mean-field limit and convergence analysis, Statistics and Computing, 31 (2021), pp. 1–21.
  * [43] M. P. Do Carmo and J. Flaherty Francis, Riemannian Geometry, vol. 6, Springer, 1992.
  * [44] A. Doucet, A. M. Johansen, et al., A tutorial on particle filtering and smoothing: Fifteen years later, Handbook of Nonlinear Filtering, 12 (2009), p. 3.
  * [45] A. Duncan, N. Nüsken, and L. Szpruch, On the geometry of Stein variational gradient descent, arXiv preprint arXiv:1912.00894 (To appear in JMLR), (2019).
  * [46] M. M. Dunlop and G. Stadler, A gradient-free subspace-adjusting ensemble sampler for infinite-dimensional bayesian inverse problems, arXiv preprint arXiv:2202.11088, (2022).
  * [47] A. A. Emerick and A. C. Reynolds, Investigation of the sampling performance of ensemble-based methods with a simple reservoir model, Computational Geosciences, 17 (2013), pp. 325–350.
  * [48] G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, Journal of Geophysical Research: Oceans, 99 (1994), pp. 10143–10162.
  * [49] D. Foreman-Mackey, D. W. Hogg, D. Lang, and J. Goodman, EMCEE: The MCMC hammer, Publications of the Astronomical Society of the Pacific, 125 (2013), p. 306.
  * [50] D. Friedman, The functional equation f (x+ y)= g (x)+ h (y), The American Mathematical Monthly, 69 (1962), pp. 769–772.
  * [51] T. Friedrich, Die fisher-information und symplektische strukturen, Mathematische Nachrichten, 153 (1991), pp. 273–296, https://doi.org/https://doi.org/10.1002/mana.19911530125, https://onlinelibrary.wiley.com/doi/abs/10.1002/mana.19911530125.
  * [52] T. Galy-Fajou, V. Perrone, and M. Opper, Flexible and efficient inference with particles for the variational Gaussian approximation, Entropy, 23 (2021), p. 990, https://doi.org/10.3390/e23080990.
  * [53] T. Galy-Fajou, V. Perrone, and M. Opper, Flexible and efficient inference with particles for the variational gaussian approximation, Entropy, 23 (2021), p. 990.
  * [54] A. Garbuno-Inigo, F. Hoffmann, W. Li, and A. M. Stuart, Interacting Langevin diffusions: Gradient structure and ensemble Kalman sampler, SIAM Journal on Applied Dynamical Systems, 19 (2020), pp. 412–441.
  * [55] A. Garbuno-Inigo, N. Nüsken, and S. Reich, Affine invariant interacting Langevin dynamics for Bayesian inference, SIAM Journal on Applied Dynamical Systems, 19 (2020), pp. 1633–1658.
  * [56] A. L. Gibbs and F. E. Su, On choosing and bounding probability metrics, International Statistical Review, 70 (2002), pp. 419–435.
  * [57] O. Gonzalez and A. M. Stuart, A First Course in Continuum Mechanics, vol. 42, Cambridge University Press, 2008.
  * [58] J. Goodman and J. Weare, Ensemble samplers with affine invariance, Communications in applied mathematics and computational science, 5 (2010), pp. 65–80.
  * [59] L. Gross, Logarithmic sobolev inequalities, American Journal of Mathematics, 97 (1975), pp. 1061–1083.
  * [60] A. Halder and T. T. Georgiou, Gradient flows in uncertainty propagation and filtering of linear gaussian systems, in 2017 IEEE 56th Annual Conference on Decision and Control (CDC), IEEE, 2017, pp. 3081–3088.
  * [61] A. Halder and T. T. Georgiou, Gradient flows in filtering and Fisher-Rao geometry, in 2018 Annual American Control Conference (ACC), IEEE, 2018, pp. 4281–4286.
  * [62] D. Z. Huang, J. Huang, S. Reich, and A. M. Stuart, Efficient derivative-free bayesian inference for large-scale inverse problems, arXiv preprint arXiv:2204.04386, (2022).
  * [63] D. Z. Huang, T. Schneider, and A. M. Stuart, Iterated Kalman methodology for inverse problems, Journal of Computational Physics, 463 (2022), p. 111262.
  * [64] M. A. Iglesias, K. J. Law, and A. M. Stuart, Ensemble Kalman methods for inverse problems, Inverse Problems, 29 (2013), p. 045001.
  * [65] T. Isaac, N. Petra, G. Stadler, and O. Ghattas, Scalable and efficient algorithms for the propagation of uncertainty from data through inference to prediction for large-scale problems, with application to flow of the antarctic ice sheet, Journal of Computational Physics, 296 (2015), pp. 348–368.
  * [66] K. Itô, P. Henry Jr, et al., Diffusion processes and their sample paths: Reprint of the 1974 edition, Springer Science & Business Media, 1996.
  * [67] A. H. Jazwinski, Stochastic processes and filtering theory, Courier Corporation, 2007.
  * [68] R. Jordan, D. Kinderlehrer, and F. Otto, The variational formulation of the Fokker–Planck equation, SIAM journal on mathematical analysis, 29 (1998), pp. 1–17.
  * [69] S. J. Julier and J. K. Uhlmann, New extension of the Kalman filter to nonlinear systems, in Signal processing, sensor fusion, and target recognition VI, vol. 3068, International Society for Optics and Photonics, 1997, pp. 182–193.
  * [70] S. J. Julier, J. K. Uhlmann, and H. F. Durrant-Whyte, A new approach for filtering nonlinear systems, in Proceedings of 1995 American Control Conference-ACC’95, vol. 3, IEEE, 1995, pp. 1628–1632.
  * [71] R. E. Kalman, A new approach to linear filtering and prediction problems, J. Basic Eng. Mar, 82 (1960), pp. 35–45.
  * [72] R. E. Kalman and R. S. Bucy, New Results in Linear Filtering and Prediction Theory, Journal of Basic Engineering, 83 (1961), pp. 95–108, https://doi.org/10.1115/1.3658902, https://doi.org/10.1115/1.3658902, https://arxiv.org/abs/https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/83/1/95/5503549/95_1.pdf.
  * [73] S. Karlin and J. McGregor, The classification of birth and death processes, Transactions of the American Mathematical Society, 86 (1957), pp. 366–400.
  * [74] M. Khan and W. Lin, Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models, in Artificial Intelligence and Statistics, PMLR, 2017, pp. 878–887.
  * [75] M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava, Fast and scalable bayesian deep learning by weight-perturbation in adam, in International Conference on Machine Learning, PMLR, 2018, pp. 2611–2620.
  * [76] H. Kim, D. Sanz-Alonso, and A. Strang, Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors, arXiv preprint arXiv:2205.09322, (2022).
  * [77] A. Korba, A. Salim, M. Arbel, G. Luise, and A. Gretton, A non-asymptotic analysis for Stein variational gradient descent, Advances in Neural Information Processing Systems, 33 (2020), pp. 4672–4682.
  * [78] M. Lambert, S. Bonnabel, and F. Bach, The continuous-discrete variational kalman filter (CD-VKF), in 2022 IEEE 61st Conference on Decision and Control (CDC), 2022, pp. 6632–6639, https://doi.org/10.1109/CDC51059.2022.9992993.
  * [79] M. Lambert, S. Bonnabel, and F. Bach, The recursive variational gaussian approximation (r-vga), Statistics and Computing, 32 (2022), pp. 1–24.
  * [80] M. Lambert, S. Chewi, F. Bach, S. Bonnabel, and P. Rigollet, Variational inference via Wasserstein gradient flows, arXiv preprint arXiv:2205.15902, (2022).
  * [81] V. Laschos and A. Mielke, Geometric properties of cones with applications on the hellinger–kantorovich space, and a new distance on the space of probability measures, Journal of Functional Analysis, 276 (2019), pp. 3529–3576.
  * [82] R. S. Laugesen, P. G. Mehta, S. P. Meyn, and M. Raginsky, Poisson’s equation in nonlinear filtering, SIAM Journal on Control and Optimization, 53 (2015), pp. 501–525.
  * [83] B. Leimkuhler, C. Matthews, and J. Weare, Ensemble preconditioning for markov chain monte carlo simulation, Statistics and Computing, 28 (2018), pp. 277–290.
  * [84] W. Li and J. Zhao, Wasserstein information matrix, Information Geometry, (2023), https://doi.org/10.1007/s41884-023-00099-9, https://doi.org/10.1007/s41884-023-00099-9.
  * [85] F. Liese and I. Vajda, On divergences and informations in statistics and information theory, IEEE Transactions on Information Theory, 52 (2006), pp. 4394–4412.
  * [86] W. Lin, M. E. Khan, and M. Schmidt, Fast and simple natural-gradient variational inference with mixture of exponential-family approximations, in International Conference on Machine Learning, PMLR, 2019, pp. 3992–4002.
  * [87] M. Lindsey, J. Weare, and A. Zhang, Ensemble markov chain monte carlo with teleporting walkers, SIAM/ASA Journal on Uncertainty Quantification, 10 (2022), pp. 860–885.
  * [88] Q. Liu, Stein variational gradient descent as gradient flow, Advances in neural information processing systems, 30 (2017).
  * [89] Q. Liu and D. Wang, Stein variational gradient descent: A general purpose Bayesian inference algorithm, Advances in neural information processing systems, 29 (2016).
  * [90] Z. Liu, A. M. Stuart, and Y. Wang, Second order ensemble langevin method for sampling and inverse problems, arXiv preprint arXiv:2208.04506, (2022).
  * [91] I. Lopez-Gomez, C. Christopoulos, H. L. Langeland Ervik, O. R. Dunbar, Y. Cohen, and T. Schneider, Training physics-based machine-learning parameterizations with gradient-free ensemble Kalman methods, Journal of Advances in Modeling Earth Systems, 14 (2022), p. e2022MS003105.
  * [92] J. Lu, Y. Lu, and J. Nolen, Scaling limit of the Stein variational gradient descent: The mean field regime, SIAM Journal on Mathematical Analysis, 51 (2019), pp. 648–671.
  * [93] Y. Lu, J. Lu, and J. Nolen, Accelerating Langevin sampling with birth-death, arXiv preprint arXiv:1905.09863, (2019).
  * [94] Y. Lu, D. Slepčev, and L. Wang, Birth-death dynamics for sampling: Global convergence, approximations and their asymptotics, arXiv preprint arXiv:2211.00450, (2022).
  * [95] L. Malagò, L. Montrucchio, and G. Pistone, Wasserstein Riemannian geometry of positive definite matrices, Information Geometry, 1 (2018), pp. 137–179, https://doi.org/10.1007/s41884-018-0014-4.
  * [96] D. Maoutsa, S. Reich, and M. Opper, Interacting particle solutions of fokker–planck equations through gradient–log–density estimation, Entropy, 22 (2020), p. 802.
  * [97] J. Martens, New insights and perspectives on the natural gradient method, The Journal of Machine Learning Research, 21 (2020), pp. 5776–5851.
  * [98] J. Martin, L. C. Wilcox, C. Burstedde, and O. Ghattas, A stochastic Newton MCMC method for large-scale statistical inverse problems with application to seismic inversion, SIAM Journal on Scientific Computing, 34 (2012), pp. A1460–A1487.
  * [99] H. P. McKean, Propagation of chaos for a class of non-linear parabolic equations, Stochastic Differential Equations (Lecture Series in Differential Equations, Session 7, Catholic Univ., 1967), (1967), pp. 41–57.
  * [100] K. P. Murphy, Machine learning: a probabilistic perspective, MIT press, 2012.
  * [101] J. A. Nelder and R. Mead, A simplex method for function minimization, The computer journal, 7 (1965), pp. 308–313.
  * [102] Y. Ollivier, Online natural gradient as a Kalman filter, Electronic Journal of Statistics, 12 (2018), pp. 2930–2961.
  * [103] Y. Ollivier, The extended Kalman filter is a natural gradient descent in trajectory space, arXiv preprint arXiv:1901.00696, (2019).
  * [104] M. Opper and C. Archambeau, The variational Gaussian approximation revisited, Neural computation, 21 (2009), pp. 786–792.
  * [105] F. Otto, The geometry of dissipative evolution equations: The porous medium equation, Communications in Partial Differential Equations, 26 (2001), pp. 101–174, https://doi.org/10.1081/PDE-100002243, https://doi.org/10.1081/PDE-100002243, https://arxiv.org/abs/https://doi.org/10.1081/PDE-100002243.
  * [106] G. A. Pavliotis, Stochastic processes and applications: diffusion processes, the Fokker-Planck and Langevin equations, vol. 60, Springer, 2014\.
  * [107] J. Pidstrigach and S. Reich, Affine-invariant ensemble transform methods for logistic regression, arXiv preprint arXiv:2104.08061, (2021).
  * [108] H. Poincaré, Sur les équations aux dérivées partielles de la physique mathématique, American Journal of Mathematics, (1890), pp. 211–294.
  * [109] M. Quiroz, D. J. Nott, and R. Kohn, Gaussian variational approximation for high-dimensional state space models, arXiv preprint arXiv:1801.07873, (2018).
  * [110] C. R. Rao, Information and the accuracy attainable in the estimation of statistical parameters, in Breakthroughs in statistics, Springer, 1992, pp. 235–247.
  * [111] C. E. Rasmussen, Gaussian processes in machine learning, in Summer school on machine learning, Springer, 2003, pp. 63–71.
  * [112] S. Reich, A dynamical systems framework for intermittent data assimilation, BIT Numerical Mathematics, 51 (2011), pp. 235–249.
  * [113] F. Rellich and J. Berkowitz, Perturbation theory of eigenvalue problems, CRC Press, 1969.
  * [114] T. Salimans, H. Zhang, A. Radford, and D. Metaxas, Improving GANs using optimal transport, arXiv preprint arXiv:1803.05573, (2018).
  * [115] F. Santambrogio, $\\{$Euclidean, metric, and Wasserstein$\\}$ gradient flows: an overview, Bulletin of Mathematical Sciences, 7 (2017), pp. 87–154.
  * [116] D. Sanz-Alonso, A. M. Stuart, and A. Taeb, Inverse problems and data assimilation, arXiv preprint arXiv:1810.06191, (2018).
  * [117] S. Sarkka, On unscented Kalman filtering for state estimation of continuous-time nonlinear systems, IEEE Transactions on automatic control, 52 (2007), pp. 1631–1641.
  * [118] T. Schneider, S. Lan, A. Stuart, and J. Teixeira, Earth system modeling 2.0: A blueprint for models that learn from observations and targeted high-resolution simulations, Geophysical Research Letters, 44 (2017), pp. 12–396.
  * [119] Z. Shen, Z. Wang, S. Kale, A. Ribeiro, A. Karbasi, and H. Hassani, Self-consistency of the fokker-planck equation, arXiv preprint arXiv:2206.00860, (2022).
  * [120] Y. Song, S. Garg, J. Shi, and S. Ermon, Sliced score matching: A scalable approach to density and score estimation, in Uncertainty in Artificial Intelligence, PMLR, 2020, pp. 574–584.
  * [121] H. W. Sorenson, Kalman Filtering: Theory and Application, IEEE, 1985\.
  * [122] A. Srivastava, I. Jermyn, and S. Joshi, Riemannian analysis of probability density functions with applications in vision, in 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2007, pp. 1–8.
  * [123] A. N. Subrahmanya, A. A. Popov, and A. Sandu, An ensemble variational Fokker-Planck method for data assimilation, arXiv preprint arXiv:2111.13926, (2021).
  * [124] A.-S. Sznitman, Topics in propagation of chaos, in Ecole d’été de probabilités de Saint-Flour XIX—1989, Springer, 1991, pp. 165–251.
  * [125] A. Takatsu, Wasserstein geometry of gaussian measures, Osaka Journal of Mathematics, 48 (2011), pp. 1005–1026.
  * [126] N. G. Trillos, B. Hosseini, and D. Sanz-Alonso, From optimization to sampling through gradient flows, arXiv preprint arXiv:2302.11449 (To appear in Notices of AMS), (2023).
  * [127] C. Villani, Optimal transport: old and new, vol. 338, Springer, 2009\.
  * [128] C. Villani, Topics in optimal transportation, vol. 58, American Mathematical Soc., 2021.
  * [129] M. J. Wainwright, M. I. Jordan, et al., Graphical models, exponential families, and variational inference, Foundations and Trends® in Machine Learning, 1 (2008), pp. 1–305.
  * [130] E. A. Wan and R. Van Der Merwe, The unscented Kalman filter for nonlinear estimation, in Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), Ieee, 2000, pp. 153–158.
  * [131] Y. Wang, P. Chen, and W. Li, Projected wasserstein gradient descent for high-dimensional bayesian inference, SIAM/ASA Journal on Uncertainty Quantification, 10 (2022), pp. 1513–1532.
  * [132] Y. Wang, P. Chen, M. Pilanci, and W. Li, Optimal neural network approximation of wasserstein gradient direction via convex optimization, arXiv preprint arXiv:2205.13098, (2022).
  * [133] Y. Wang and W. Li, Information Newton’s flow: second-order optimization method in probability space, arXiv preprint arXiv:2001.04341, (2020).
  * [134] Y. Wang and W. Li, Accelerated information gradient flow, Journal of Scientific Computing, 90 (2022), pp. 1–47.
  * [135] S. Weissmann, N. K. Chada, C. Schillings, and X. T. Tong, Adaptive Tikhonov strategies for stochastic ensemble Kalman inversion, arXiv preprint arXiv:2110.09142, (2021).
  * [136] K.-V. Yuen, Bayesian methods for structural dynamics and civil engineering, John Wiley & Sons, 2010.
  * [137] H. Yumei, W. Xuezhi, P. Quan, H. Zhentao, and B. Moran, Variational bayesian Kalman filter using natural gradient, Chinese Journal of Aeronautics, 35 (2022), pp. 1–10.
  * [138] G. Zhang, J. Martens, and R. B. Grosse, Fast convergence of natural gradient descent for over-parameterized neural networks, Advances in Neural Information Processing Systems, 32 (2019).

##  Appendix A Unique Property of the KL Divergence Energy

###  A.1 Proof of Proposition 2.1

######  Proof A.1.

First note that we already know that the KL divergence satisfies the desired
property, by Eq. 6. To show the uniqueness, we consider $\rho$ and $\rho_{\rm
post}$ to be continuous density functions, positive everywhere. We start by
calculating the first variation of the $f$-divergence:

$\frac{\delta D_{f}}{\delta\rho}=f^{\prime}\Bigl{(}\frac{\rho}{\rho_{\rm
post}}\Bigr{)}-\int f^{\prime}\Bigl{(}\frac{\rho}{\rho_{\rm post}}\Bigr{)}{\rm
d}\theta.$

We assume that this first variation is invariant under scaling of $\rho_{\rm
post}$ by a positive real $c$ in order to determine constraints that this
imposes on $f$. We thus have that

$f^{\prime}\Bigl{(}\frac{\rho(\theta)}{\rho_{\rm post}(\theta)}\Bigr{)}-\int
f^{\prime}\Bigl{(}\frac{\rho}{\rho_{\rm post}}\Bigr{)}{\rm
d}\theta=f^{\prime}\Bigl{(}\frac{c\rho(\theta)}{\rho_{\rm
post}(\theta)}\Bigr{)}-\int f^{\prime}\Bigl{(}\frac{c\rho}{\rho_{\rm
post}}\Bigr{)}{\rm d}\theta,$

and hence that

(102) $f^{\prime}\Bigl{(}\frac{\rho(\theta)}{\rho_{\rm
post}(\theta)}\Bigr{)}-f^{\prime}\Bigl{(}\frac{c\rho(\theta)}{\rho_{\rm
post}(\theta)}\Bigr{)}=\int f^{\prime}\Bigl{(}\frac{\rho}{\rho_{\rm
post}}\Bigr{)}{\rm d}\theta-\int f^{\prime}\Bigl{(}\frac{c\rho}{\rho_{\rm
post}}\Bigr{)}{\rm d}\theta,$

for any $\theta\in\mathbb{R}^{d}$ and $c>0$. Because $\rho$ and $\rho_{\rm
post}$ integrate to $1$ and they are continuous, there exists
$\theta^{\dagger}$ such that $\rho(\theta^{\dagger})/\rho_{\rm
post}(\theta^{\dagger})=1$. Setting $\theta=\theta^{\dagger}$ in the above
identity, we obtain

(103) $g(c):=f^{\prime}(1)-f^{\prime}(c)=\int
f^{\prime}\Bigl{(}\frac{\rho}{\rho_{\rm post}}\Bigr{)}{\rm d}\theta-\int
f^{\prime}\Bigl{(}\frac{c\rho}{\rho_{\rm post}}\Bigr{)}{\rm d}\theta.$

Combining (102), (103) to eliminate the integrated terms we obtain

(104) $f^{\prime}\Bigl{(}\frac{\rho(\theta)}{\rho_{\rm
post}(\theta)}\Bigr{)}-f^{\prime}\Bigl{(}\frac{c\rho(\theta)}{\rho_{\rm
post}(\theta)}\Bigr{)}=g(c),$

where $c$ is an arbitrary positive number. Note, furthermore, that $g(c)$ is
continuous since $f$ is continuously differentiable. Since $\rho$ and
$\rho_{\rm post}$ are arbitrary, we can write (104) equivalently as

(105) $f^{\prime}(y)-f^{\prime}(cy)=g(c),$

for any $y,c\in\mathbb{R}_{+}$. Let $h:\mathbb{R}\to\mathbb{R}$ such that
$h(z)=f^{\prime}(\exp(z))$. Then, we can equivalently formulate (105) as

(106) $h(z_{1})-h(z_{2})=r(z_{1}-z_{2}),$

for any $z_{1},z_{2}\in\mathbb{R}$ and $r:\mathbb{R}\to\mathbb{R}$ such that
$r(t)=g(\exp(-t))$.

We can show $r$ is linear function. Setting $z_{1}=z_{2}$ in (106) shows that
$r(0)=0.$ Note also that, again by (106),

$r(z_{1}-z_{2})+r(z_{2}-z_{3})=h(z_{1})-h(z_{3})=r(z_{1}-z_{3}).$

Hence, since $z_{1},z_{2}$ and $z_{3}$ are arbitrary, we deduce that for any
$x,y\in\mathbb{R}$, it holds that

(107) $r(x)+r(y)=r(x+y).$

Furthermore $r$ is continuous since $f$ is assumed continuously
differentiable. With the above conditions, it is a standard result in
functional equations that $r(x)$ is linear. Indeed, as a sketch of proof, by
(107) we can first deduce $r(n)=nr(1)$ for $n\in\mathbb{Z}$. Then, by setting
$x,y$ to be dyadic rationals, we can deduce
$r(\frac{i}{2^{k}})=\frac{i}{2^{k}}r(1)$ for any $i,k\in\mathbb{Z}$. Finally
using the continuity of the function $r$, we get $r(x)=xr(1)$ for any
$x\in\mathbb{R}$. For more details see [50].

Using the fact that $r$ is linear and the equation (106), we know that $h$ is
an affine function and thus $f^{\prime}(\exp(z))=az+b$ for some
$a,b\in\mathbb{R}$. Equivalently, $f^{\prime}(y)=a\log(y)+b$. Using the
condition $f(1)=0$, we get $f(y)=ay\log(y)+(b-a)(y-1)$. Plugging this $f$ into
the formula for $D_{f}$, we get

$D_{f}[\rho||\rho_{\rm post}]=a\text{KL}[\rho||\rho_{\rm post}],$

noting that the affine term in $f(y)$ has zero contributions in the formula
for $D_{f}$. The proof is complete.

##  Appendix B Proofs of Affine Invariant Gradient Flows

###  B.1 Preliminaries

In this section, we present some lemmas that are useful for our proofs in
later sections. The first lemma concerns the change-of-variable formula under
the gradient and divergence operators.

######  Lemma B.1.

Consider any invertible affine mapping from $\theta\in$ to $\tilde{\theta}\in$
defined by $\tilde{\theta}=\varphi(\theta)=A\theta+b$. For any differentiable
scalar field $f:$ and vector field $g:$, we have

$\nabla_{\theta}f(\theta)=A^{T}\nabla_{\tilde{\theta}}\tilde{f}(\tilde{\theta})\quad\text{and}\quad\nabla_{\theta}\cdot
g(\theta)=\nabla_{\tilde{\theta}}\cdot(A\tilde{g}(\tilde{\theta})),$

where $\tilde{f}(\tilde{\theta}):=f(A^{-1}(\tilde{\theta}-b))$ and
$\tilde{g}(\tilde{\theta}):=g(A^{-1}(\tilde{\theta}-b))$.

######  Proof B.2.

Note that $\tilde{f}(\tilde{\theta})=f(\theta)$ and
$\tilde{g}(\tilde{\theta})=g(\theta)$. By direct calculations, we get

$\displaystyle[\nabla_{\theta}f(\theta)]_{i}=\frac{\partial
f(\theta)}{\partial\theta_{i}}=\sum_{j}\frac{\partial
f(\theta)}{\partial\tilde{\theta}_{j}}\frac{\partial\tilde{\theta}_{j}}{\partial\theta_{i}}=\sum_{j}\frac{\partial\tilde{f}(\tilde{\theta})}{\partial\tilde{\theta}_{j}}A_{ji}=[A^{T}\nabla_{\tilde{\theta}}\tilde{f}(\tilde{\theta})]_{i},$
$\displaystyle\nabla_{\theta}\cdot g(\theta)=\sum_{i}\frac{\partial
g_{i}(\theta)}{\partial\theta_{i}}=\sum_{i}\sum_{j}\frac{\partial
g_{i}(\theta)}{\partial\tilde{\theta}_{j}}\frac{\partial\tilde{\theta}_{j}}{\partial\theta_{i}}=\sum_{i}\sum_{j}\frac{\partial\tilde{g}_{i}(\tilde{\theta})}{\partial\tilde{\theta}_{j}}A_{ji}=\nabla_{\tilde{\theta}}\cdot(A\tilde{g}(\tilde{\theta})).$

This completes the proof.

Since $\tilde{f}(\tilde{\theta})=f(\theta)$ and
$\tilde{g}(\tilde{\theta})=g(\theta)$, we can also summarize the result in
Lemma B.1 as $\nabla_{\theta}f=A^{T}\nabla_{\tilde{\theta}}f$ and
$\nabla_{\theta}\cdot g=\nabla_{\tilde{\theta}}\cdot(Ag)$.

The second lemma concerns the change-of-variable formula for the first
variation of functional.

######  Lemma B.3.

Consider any invertible mapping from $\theta\in$ to $\tilde{\theta}\in$
defined by $\tilde{\theta}=\varphi(\theta)$. Then, for any differentiable
functional $\mathcal{E}$ in probability space, it holds that

$\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}=\Bigl{\langle}\frac{\delta{\mathcal{E}}}{\delta\rho},{\sigma}\Bigr{\rangle},$

for every $\sigma\in T_{\rho}\mathcal{P}$ and
$\tilde{\sigma}=\varphi\\#\sigma\in T_{\tilde{\rho}}\mathcal{P}$. Here,
$\tilde{\mathcal{E}}$ and $\tilde{\rho}$ are defined in Definition 3.1.
Consequently, it follows that

$\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\tilde{\theta})=\frac{\delta\mathcal{E}}{\delta\rho}(\theta).$

.

######  Proof B.4.

By the definition of the first variation, for any $\tilde{\sigma}\in
T_{\tilde{\rho}}\mathcal{P}$, we have that

$\displaystyle\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}$
$\displaystyle=\lim_{\epsilon\rightarrow
0}\frac{\tilde{\mathcal{E}}(\tilde{\rho}+\epsilon\tilde{\sigma})-\tilde{\mathcal{E}}(\tilde{\rho}+\epsilon\tilde{\sigma})}{\epsilon}$
$\displaystyle=\lim_{\epsilon\rightarrow
0}\frac{{\mathcal{E}}(\varphi^{-1}\\#\tilde{\rho}+\epsilon\varphi^{-1}\\#\tilde{\sigma})-{\mathcal{E}}(\varphi^{-1}\\#\tilde{\rho}+\epsilon\varphi^{-1}\\#\tilde{\sigma})}{\epsilon}$
$\displaystyle=\lim_{\epsilon\rightarrow
0}\frac{{\mathcal{E}}(\rho+\epsilon\sigma)-{\mathcal{E}}(\rho+\epsilon\sigma)}{\epsilon}$
$\displaystyle=\Bigl{\langle}\frac{\delta{\mathcal{E}}}{\delta\rho},{\sigma}\Bigr{\rangle}.$

Here in the second equality, we have used the fact that
$\tilde{\mathcal{E}}(\tilde{\rho})=\mathcal{E}(\varphi^{-1}\\#\tilde{\rho})$
as in Definition 3.1; moreover, $\tilde{\sigma}=\varphi\\#\sigma$ by
definition.

Now, we write out the integral explicitly:

$\displaystyle\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}=\int\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\tilde{\theta})\tilde{\sigma}(\tilde{\theta})\mathrm{d}\tilde{\theta}=\int\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\varphi(\theta))\sigma(\theta)\mathrm{d}\theta,$

where in the last equality, we used the change-of-variable
$\tilde{\theta}=\varphi(\theta)$ and the fact that
$\tilde{\sigma}(\tilde{\theta})=\sigma(\varphi^{-1}(\tilde{\theta}))|\nabla_{\tilde{\theta}}\varphi^{-1}(\tilde{\theta})|$.
Combining the above relations, we obtain

$\int\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\varphi(\theta))\sigma(\theta)\mathrm{d}\theta=\Bigl{\langle}\frac{\delta{\mathcal{E}}}{\delta\rho},{\sigma}\Bigr{\rangle}=\int\frac{\delta{\mathcal{E}}}{\delta\rho}(\theta)\sigma(\theta)\mathrm{d}\theta,$

which holds for any $\sigma\in T_{\rho}\mathcal{P}$. We deduce that

$\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\varphi(\theta))=\frac{\delta\mathcal{E}}{\delta\rho}(\theta).$

Here we used the fact that the integration of any element in
$T_{\rho}^{*}\mathcal{P}$ is 0 so the above identity holds exact. The proof is
complete noticing that $\tilde{\theta}=\varphi(\theta)$.

###  B.2 Proof of Proposition 3.4

######  Proof B.5.

From the definition, it suffices to show the equivalence of

  * (a)

$\varphi\\#(\nabla\mathcal{E}(\rho))=\nabla\tilde{\mathcal{E}}(\tilde{\rho})$
for any $\mathcal{E}$;

  * (b)

$\varphi^{\\#}g=g$.

From (b) to (a): For any $\sigma\in T_{\rho}\mathcal{P}$, by the definition of
the pushforward and pull-back operators, we have

$(\varphi^{\\#}g)_{\rho}(\nabla\mathcal{E}(\rho),\sigma)=g_{\tilde{\rho}}(\varphi\\#(\nabla\mathcal{E}(\rho)),\tilde{\sigma}),$

where $\tilde{\sigma}=\varphi\\#\sigma\in T_{\tilde{\rho}}\mathcal{P}$. Thus,
by (b) we get
$g_{\rho}(\nabla\mathcal{E}(\rho),\sigma)=g_{\tilde{\rho}}(\varphi\\#(\nabla\mathcal{E}(\rho)),\tilde{\sigma})$.

Using the definition of the gradient operator and Lemma B.3, we have

$g_{\rho}(\nabla\mathcal{E}(\rho),\sigma)=\Bigl{\langle}\frac{\delta\mathcal{E}}{\delta\rho},\sigma\Bigr{\rangle}=\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}.$

Combining the two relations above leads to
$\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}=g_{\tilde{\rho}}(\varphi\\#(\nabla\mathcal{E}(\rho)),\tilde{\sigma})$.
Then, using the definition of the gradient operator, we get
$\varphi\\#(\nabla\mathcal{E}(\rho))=\nabla\tilde{\mathcal{E}}(\tilde{\rho})$.

From (a) to (b): similar as above, we have the relation

$g_{\rho}(\nabla\mathcal{E}(\rho),\sigma)=\Bigl{\langle}\frac{\delta\mathcal{E}}{\delta\rho},\sigma\Bigr{\rangle}=\Bigl{\langle}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}},\tilde{\sigma}\Bigr{\rangle}=g_{\tilde{\rho}}(\nabla\tilde{\mathcal{E}}(\tilde{\rho}),\tilde{\sigma}).$

By (a), it holds that

$g_{\tilde{\rho}}(\nabla\tilde{\mathcal{E}}(\tilde{\rho}),\tilde{\sigma})=g_{\tilde{\rho}}(\varphi\\#(\nabla\mathcal{E}(\rho)),\tilde{\sigma})=(\varphi^{\\#}g)_{\rho}(\nabla\mathcal{E}(\rho),\sigma),$

where in the last equality we used the definition of the pull-back operator.

Combining the two relations above and noticing the fact that $\mathcal{E}$ and
$\sigma$ are arbitrary, we get $\varphi^{\\#}g=g$. This completes the proof.

###  B.3 Proof of LABEL:{prop:_affine_invariance_of_mean-field_dynamics}

By definition, $f$ and $h$ satisfy the equation

(108) $-\nabla_{\theta}\cdot(\rho
f)+\frac{1}{2}\nabla_{\theta}\cdot(\nabla_{\theta}\cdot(hh^{T}\rho))=-M(\rho)^{-1}\frac{\delta\mathcal{E}(\rho;\rho_{\rm
post})}{\delta\rho}.$

For a fixed $\mathcal{E}$, one can write $f=f(\theta;\rho,\rho_{\rm
post}),h=h(\theta;\rho,\rho_{\rm post})$.

Consider an invertible affine transformation
$\tilde{\theta}=\varphi(\theta)=A\theta+b$, and correspondingly
$\tilde{\rho}=\varphi\\#\rho,\tilde{\rho}_{\rm post}=\varphi\\#\rho_{\rm
post}$. Under this new coordinate, we obtain the corresponding equation to
determine the drift $\tilde{f}$ and diffusion coefficient $\tilde{h}$:

(109)
$-\nabla_{\tilde{\theta}}\cdot(\tilde{\rho}\tilde{f})+\frac{1}{2}\nabla_{\tilde{\theta}}\cdot(\nabla_{\tilde{\theta}}\cdot(\tilde{h}\tilde{h}^{T}\tilde{\rho}))=-M(\tilde{\rho})^{-1}\frac{\delta\mathcal{E}(\tilde{\rho};\tilde{\rho}_{\rm
post})}{\delta\tilde{\rho}}.$

This identity can be satisfied by many $\tilde{f}$, $\tilde{h}.$ Our goal is
to show that the choice

(110) $\displaystyle\tilde{f}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post})$ $\displaystyle=Af(\theta;\rho,\rho_{\rm post})$
$\displaystyle\tilde{h}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm post})$
$\displaystyle=Ah(\theta;\rho,\rho_{\rm post})$

satisfies the equation Eq. 109.

First, by the change-of-variable formula, we have

(111)
$\tilde{\rho}(\tilde{\theta})=\rho(\varphi^{-1}(\tilde{\theta}))|\nabla_{\tilde{\theta}}\varphi^{-1}(\tilde{\theta})|=\rho(\theta)|A^{-1}|.$

For the $\tilde{f}$ in Eq. 110, we calculate

(112)
$\displaystyle-\nabla_{\tilde{\theta}}\cdot(\tilde{\rho}(\tilde{\theta})\tilde{f}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post}))$
$\displaystyle=-\nabla_{\theta}\cdot(A^{-1}\tilde{\rho}(\tilde{\theta})\tilde{f}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post}))$
$\displaystyle=-\nabla_{\theta}\cdot(\rho(\theta)f(\theta;\rho,\rho_{\rm
post}))|A^{-1}|,$

where in the first equality, we used the change-of-variable formula for the
divergence operator (LABEL:remark:_change_of_variable_grad_and_div); in the
second equality, we used Eq. 110 and Eq. 111. Similarly, for the $\tilde{h}$
in Eq. 110, we can get

(113)
$\frac{1}{2}\nabla_{\tilde{\theta}}\cdot(\nabla_{\tilde{\theta}}\cdot(\tilde{h}\tilde{h}^{T}\tilde{\rho}))=\frac{1}{2}\nabla_{\theta}\cdot(\nabla_{\theta}\cdot(hh^{T}\rho))|A^{-1}|.$

Here we note that the left hand side is evaluated at $\tilde{\theta}$ while
the right hand side is evaluated at $\theta$, similar to Eq. 112.

Thus, for the choice of $\tilde{f}$ and $\tilde{h}$ in Eq. 110, it holds that

(114)
$-\nabla_{\tilde{\theta}}\cdot(\tilde{\rho}\tilde{f})+\frac{1}{2}\nabla_{\tilde{\theta}}\cdot(\nabla_{\tilde{\theta}}\cdot(\tilde{h}\tilde{h}^{T}\tilde{\rho}))=-M(\rho)^{-1}\frac{\delta\mathcal{E}(\rho;\rho_{\rm
post})}{\delta\rho}|A^{-1}|,$

where we used the fact that $f$ and $h$ satisfy Eq. 108. The right hand side
is evaluated at $\theta$.

Now, in order for Eq. 109 to hold, it suffices to show

(115) $\left(M(\rho)^{-1}\frac{\delta\mathcal{E}(\rho;\rho_{\rm
post})}{\delta\rho}\right)(\theta)|A^{-1}|=\left(M(\tilde{\rho})^{-1}\frac{\delta\mathcal{E}(\tilde{\rho};\tilde{\rho}_{\rm
post})}{\delta\tilde{\rho}}\right)(\tilde{\theta}).$

Using the condition that the gradient flow in probability space is affine
invariant, we have
$\varphi\\#(\nabla\mathcal{E}(\rho))=\nabla\tilde{\mathcal{E}}(\tilde{\rho})$,
or equivalently,

(116) $\left(M(\rho)^{-1}\frac{\delta\mathcal{E}(\rho;\rho_{\rm
post})}{\delta\rho}\right)(\theta)|A^{-1}|=\left(M(\tilde{\rho})^{-1}\frac{\delta\tilde{\mathcal{E}}(\tilde{\rho};\rho_{\rm
post})}{\delta\tilde{\rho}}\right)(\tilde{\theta}),$

where we used the definition of the gradient and the push forward operator.
Here $\tilde{\mathcal{E}}=\varphi\\#\mathcal{E}$. Using the assumption, we
have

$\tilde{\mathcal{E}}(\tilde{\rho};\rho_{\rm
post})=\mathcal{E}(\varphi^{-1}\\#\tilde{\rho};\rho_{\rm
post})=\mathcal{E}(\tilde{\rho};\varphi\\#\rho_{\rm
post})=\mathcal{E}(\tilde{\rho};\tilde{\rho}_{\rm post}).$

Combining it with Eq. 116, we get the desired Eq. 115. The proof is complete.

###  B.4 Proof of Proposition 3.7

######  Proof B.6.

Let $\phi$ be chosen so that

$f(\theta;\rho,\rho_{\rm
post}):=P(\theta,\rho)\nabla_{\theta}\phi(\theta;\rho,\rho_{\rm post})$

satisfies the identity

(117) $-\nabla_{\theta}\cdot(\rho f)=\rho\bigl{(}\log\rho_{\rm
post}-\log\rho\bigr{)}-\rho\mathbb{E}_{\rho}[\log\rho_{\rm post}-\log\rho].$

Consider an invertible affine transformation
$\tilde{\theta}=\varphi(\theta)=A\theta+b$, and correspondingly
$\tilde{\rho}=\varphi\\#\rho,\tilde{\rho}_{\rm post}=\varphi\\#\rho_{\rm
post}$. Using the proof of LABEL:{prop:_affine_invariance_of_mean-
field_dynamics}, which can be found in Section B.3, we know that
$\tilde{f}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post})=Af(\theta;\rho,\rho_{\rm post})$ satisfies the identity

(118)
$-\nabla_{\tilde{\theta}}\cdot(\tilde{\rho}\tilde{f})=\tilde{\rho}\bigl{(}\log\tilde{\rho}_{\rm
post}-\log\tilde{\rho}\bigr{)}-\tilde{\rho}\mathbb{E}_{\tilde{\rho}}[\log\tilde{\rho}_{\rm
post}-\log\tilde{\rho}].$

Notice that

(119) $\displaystyle\tilde{f}(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post})$
$\displaystyle=AP(\theta,\rho)\nabla_{\theta}\phi(\theta;\rho,\rho_{\rm
post})$
$\displaystyle=AP(\theta,\rho)A^{T}\nabla_{\tilde{\theta}}\phi(\theta;\rho,\rho_{\rm
post})$
$\displaystyle=P(\tilde{\theta},\tilde{\rho})\nabla_{\tilde{\theta}}\phi(\theta;\rho,\rho_{\rm
post}),$

where in the second equality, we used the change-of-variable formula from
LABEL:remark:_change_of_variable_grad_and_div. In the last equality, we used
the assumed condition on $P$.

Now using the uniqueness of the solution to the equation (35), we get

(120) $\phi(\theta;\rho,\rho_{\rm
post})=\phi(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm post})\quad\text{up
to constants}.$

We can use the above result to prove that the mean-field dynamics is affine
invariant. By definition, it suffices to show

$AP(\theta,\rho)\nabla_{\theta}\phi(\theta;\rho,\rho_{\rm
post})=P(\tilde{\theta},\tilde{\rho})\nabla_{\tilde{\theta}}\phi(\tilde{\theta};\tilde{\rho},\tilde{\rho}_{\rm
post}).$

This is readily true by combining the calculation in (119) and the condition
(120). The proof is complete.

###  B.5 Proof of Proposition 3.8

######  Proof B.7.

By Proposition 3.4, it suffices to show that the gradient flows for any
$\mathcal{E}$ is affine invariant. We write down the form of the corresponding
gradient flow as follows:

(121) $\displaystyle\frac{\partial\rho_{t}(\theta)}{\partial
t}=\nabla_{\theta}\cdot\Bigl{[}\rho_{t}P(\theta,\rho_{t})\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\Bigr{]}.$

Consider $\tilde{\theta}=\varphi(\theta)=A\theta+b$ and
$\tilde{\rho}_{t}=\varphi\\#\rho_{t}$ for an invertible affine transformation
$\varphi$. Then, it suffices to show that under the assumption
$P(\tilde{\theta},\tilde{\rho})=AP(\theta,\rho)A^{T}$, one has

(122) $\displaystyle\frac{\partial\tilde{\rho}_{t}(\tilde{\theta})}{\partial
t}=\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}P(\tilde{\theta},\tilde{\rho}_{t})\nabla_{\tilde{\theta}}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{{\rho}}}\Bigr{|}_{\tilde{\rho}=\tilde{\rho}_{t}}\Bigr{]}.$

Here, the transformed energy functional $\tilde{\mathcal{E}}$ is defined via
Definition 3.1.

First, for the left hand side, by definition, we have

(123)
$\tilde{\rho}_{t}(\tilde{\theta})=\rho_{t}(\varphi^{-1}(\tilde{\theta}))|\nabla_{\tilde{\theta}}\varphi^{-1}(\tilde{\theta})|=\rho_{t}(\theta)|A^{-1}|.$

For the right hand side, by Lemma B.3, it holds that

(124)
$\nabla_{\tilde{\theta}}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}(\tilde{\theta})=\nabla_{\tilde{\theta}}\left(\frac{\delta\mathcal{E}}{\delta\rho}(\theta)\right)=A^{-T}\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}(\theta),$

where in the last equality, we used the change-of-variable formula mentioned
in LABEL:remark:_change_of_variable_grad_and_div. Therefore, we can write the
right hand side of Eq. 122 as

(125)
$\displaystyle\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}P(\tilde{\theta},\tilde{\rho}_{t})\nabla_{\tilde{\theta}}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{{\rho}}}\Bigr{|}_{\tilde{\rho}=\tilde{\rho}_{t}}\Bigr{]}$
$\displaystyle=\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}P(\tilde{\theta},\tilde{\rho}_{t})A^{-T}\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta{\rho}}\Bigr{|}_{\rho=\rho_{t}}\Bigr{]}$
$\displaystyle=\nabla_{\theta}\cdot\Bigl{[}\tilde{\rho}_{t}A^{-1}P(\tilde{\theta},\tilde{\rho}_{t})A^{-T}\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta{\rho}}\Bigr{|}_{\rho=\rho_{t}}\Bigr{]}$
$\displaystyle=\nabla_{\theta}\cdot\Bigl{[}\rho_{t}A^{-1}P(\tilde{\theta},\tilde{\rho}_{t})A^{-T}\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta{\rho}}\Bigr{|}_{\rho=\rho_{t}}\Bigr{]}\cdot|A^{-1}|,$

where in the second equality, we used the change-of-variable formula for the
divergence operator (LABEL:remark:_change_of_variable_grad_and_div), and in
the third equality, we used Eq. 123. Based on Eq. 123, Eq. 125 and Eq. 121, a
sufficient condition for Eq. 122 to hold is
$A^{-1}P(\tilde{\theta},\tilde{\rho})A^{-T}=P(\theta,\rho)$, or equivalently,
$P(\tilde{\theta},\tilde{\rho})=AP(\theta,\rho)A^{T}$. This completes the
proof.

###  B.6 Proof of Proposition 3.9

######  Proof B.8.

Consider the invertible affine transformation
$\tilde{\theta}=\varphi(\theta)=A\theta+b$ and correspondingly
$\tilde{\rho}=\varphi\\#\rho,\tilde{\rho}_{\rm post}=\varphi\\#\rho_{\rm
post}$. Using LABEL:eq:aMFD-AI-sigma, we get

$\displaystyle
D(\tilde{\theta},\tilde{\rho})=\frac{1}{2}h(\tilde{\theta},\tilde{\rho})h(\tilde{\theta},\tilde{\rho})^{T}=\frac{1}{2}Ah(\theta,\rho)h(\theta,\rho)^{T}A^{T}=AD(\theta,\rho)A^{T}.$

Similarly, it holds that $d(\tilde{\theta},\tilde{\rho})=Ad(\theta,\rho)$.
Based on these relations, we can calculate as follows:

(126) $\displaystyle Af(\theta,\rho,\rho_{\rm post})$
$\displaystyle\quad=AP(\theta,\rho)\nabla_{\theta}\log\rho_{\rm
post}(\theta)+A(D(\theta,\rho)-P(\theta,\rho))\nabla_{\theta}\log\rho(\theta)-Ad(\theta,\rho)$
$\displaystyle\quad=AP(\theta,\rho)A^{T}\nabla_{\tilde{\theta}}\log\tilde{\rho}_{\rm
post}(\tilde{\theta})+\bigl{(}AD(\theta,\rho)-AP(\theta,\rho)\bigr{)}A^{T}\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta})-Ad(\theta,\rho)$
$\displaystyle\quad=P(\tilde{\theta},\tilde{\rho})\nabla_{\tilde{\theta}}\log\tilde{\rho}_{\rm
post}(\tilde{\theta})+\bigl{(}D(\tilde{\theta},\tilde{\rho})-P(\tilde{\theta},\tilde{\rho})\bigr{)}\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta})-d(\tilde{\theta},\tilde{\rho})$
$\displaystyle\quad=f(\tilde{\theta},\tilde{\rho},\tilde{\rho}_{\rm post}).$

The first equality is by definition. In the second equality, we used
$A^{T}\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta})=\nabla_{{\theta}}\log{\rho}({\theta})$,
which is due to Eq. 111 and LABEL:remark:_change_of_variable_grad_and_div. In
the third equality, we used the condition in Proposition 3.8. With this
result, the mean-field equation is affine invariant (Definition 3.5). The
proof is complete.

###  B.7 Proof of Proposition 3.10

######  Proof B.9.

The proof is similar to that in Section B.5. For any energy functional
$\mathcal{E}$, the gradient flow has the form

(127) $\displaystyle\frac{\partial\rho_{t}(\theta)}{\partial
t}=\nabla_{\theta}\cdot\Bigl{[}\rho_{t}(\theta)\int\kappa(\theta,\theta^{\prime},\rho_{t})\rho_{t}(\theta^{\prime})P(\theta,\theta^{\prime},\rho_{t})\left(\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\right)(\theta^{\prime})\mathrm{d}\theta^{\prime}\Bigr{]}.$

Consider $\tilde{\theta}=\varphi(\theta)=A\theta+b$ and
$\tilde{\rho}_{t}=\varphi\\#\rho_{t}$ for an invertible affine transformation
$\varphi$. Then, it suffices to show that under the assumed condition, one has

(128) $\displaystyle\frac{\partial\tilde{\rho}_{t}(\tilde{\theta})}{\partial
t}$
$\displaystyle=\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}(\tilde{\theta})\int\kappa(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})\tilde{\rho}_{t}(\tilde{\theta}^{\prime})P(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})\left(\nabla_{\tilde{\theta}}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}\Bigr{|}_{\tilde{\rho}=\tilde{\rho}_{t}}\right)(\tilde{\theta}^{\prime})\mathrm{d}\tilde{\theta}^{\prime}\Bigr{]}.$

For the right hand side of Eq. 128, we have

(129)
$\displaystyle\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}(\tilde{\theta})\int\kappa(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})\tilde{\rho}_{t}(\tilde{\theta}^{\prime})P(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})\left(\nabla_{\tilde{\theta}}\frac{\delta\tilde{\mathcal{E}}}{\delta\tilde{\rho}}\Bigr{|}_{\tilde{\rho}=\tilde{\rho}_{t}}\right)(\tilde{\theta}^{\prime})\mathrm{d}\tilde{\theta}^{\prime}\Bigr{]}$
$\displaystyle=$
$\displaystyle\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}(\tilde{\theta})\int\kappa(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})\tilde{\rho}_{t}(\tilde{\theta}^{\prime})P(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho}_{t})A^{-T}\left(\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\right)(\varphi^{-1}(\tilde{\theta}^{\prime}))\mathrm{d}\tilde{\theta}^{\prime}\Bigr{]}$
$\displaystyle=$
$\displaystyle\nabla_{\tilde{\theta}}\cdot\Bigl{[}\tilde{\rho}_{t}(\tilde{\theta})\int\kappa(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t}){\rho}_{t}(\theta^{\prime})P(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t})A^{-T}\left(\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\right)(\theta^{\prime})\mathrm{d}\theta^{\prime}\Bigr{]}$
$\displaystyle=$
$\displaystyle\nabla_{\theta}\cdot\Bigl{[}\tilde{\rho}_{t}(\tilde{\theta})\int\kappa(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t}){\rho}_{t}(\theta^{\prime})A^{-1}P(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t})A^{-T}\left(\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\right)(\theta^{\prime})\mathrm{d}\theta^{\prime}\Bigr{]}$
$\displaystyle=$ $\displaystyle(\nabla_{\theta}\cdot\mathsf{f})\cdot|A^{-1}|$
$\displaystyle\mathsf{f}=\rho_{t}(\theta)\int\kappa(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t}){\rho}_{t}(\theta^{\prime})A^{-1}P(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho}_{t})A^{-T}\left(\nabla_{\theta}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{t}}\right)(\theta^{\prime})\mathrm{d}\theta^{\prime},$

where in the first equality, we used Eq. 124; in the second equality, we
changed of coordinates in the integral
$\tilde{\theta}^{\prime}=\varphi(\theta^{\prime})$; in the third equality, we
used the change-of-variable formula for the divergence operator
(LABEL:remark:_change_of_variable_grad_and_div), and in the last equality, we
used Eq. 123.

By Eq. 123, Eq. 129 and Eq. 127, a sufficient condition for Eq. 128 to hold is

$\kappa(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho})A^{-1}P(\tilde{\theta},\varphi(\theta^{\prime}),\tilde{\rho})A^{-T}=\kappa(\theta,\theta^{\prime},\rho)P(\theta,\theta^{\prime},\rho),$

or equivalently,

$\kappa(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho})P(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho})=\kappa(\theta,\theta^{\prime},\rho)AP(\theta,\theta^{\prime},\rho)A^{T},$

where $\tilde{\theta}^{\prime}=\varphi(\theta^{\prime})$. This completes the
proof.

###  B.8 Proof of Proposition 3.11

######  Proof B.10.

Consider the invertible affine transformation
$\tilde{\theta}=\varphi(\theta)=A\theta+b$ and correspondingly
$\tilde{\rho}=\varphi\\#\rho,\tilde{\rho}_{\rm post}=\varphi\\#\rho_{\rm
post}$. By direct calculations, we get

(130) $\displaystyle Af(\theta,\rho,\rho_{\rm post})$
$\displaystyle\quad=\int\kappa(\theta,\theta^{\prime},\rho)AP(\theta,\theta^{\prime},\rho)\nabla_{\theta^{\prime}}\bigl{(}\log\rho(\theta^{\prime})-\log\rho_{\rm
post}(\theta^{\prime})\bigr{)}\rho(\theta^{\prime})\mathrm{d}\theta^{\prime}$
$\displaystyle\quad=\int\kappa(\theta,\theta^{\prime},\rho)AP(\theta,\theta^{\prime},\rho)A^{T}\bigl{(}\nabla_{\tilde{\theta}}\log\tilde{\rho}_{\rm
post}(\tilde{\theta}^{\prime})-\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta}^{\prime})\bigr{)}\rho(\theta^{\prime})\mathrm{d}\theta^{\prime}$
$\displaystyle\quad=\int\kappa(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho})P(\tilde{\theta},\tilde{\theta}^{\prime},\tilde{\rho})\bigl{(}\nabla_{\tilde{\theta}}\log\tilde{\rho}_{\rm
post}(\tilde{\theta}^{\prime})-\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta}^{\prime})\bigr{)}\tilde{\rho}(\tilde{\theta}^{\prime})\mathrm{d}\tilde{\theta}^{\prime}$
$\displaystyle\quad=f(\tilde{\theta},\tilde{\rho},\tilde{\rho}_{\rm post}).$

The first equality is by definition. In the second equality, we used
$A^{T}\nabla_{\tilde{\theta}}\log\tilde{\rho}(\tilde{\theta})=\nabla_{{\theta}}\log{\rho}({\theta})$,
which is due to Eq. 111 and LABEL:remark:_change_of_variable_grad_and_div. In
the third equality, we used the the relation
$\rho(\theta^{\prime})=\tilde{\rho}(\tilde{\theta}^{\prime})|A|$ due to Eq.
111, and ${\rm d}\tilde{\theta}^{\prime}=|A|{\rm d}\theta^{\prime}$; we also
used the condition in Proposition 3.10. With this result, the mean-field
equation is affine invariant (Definition 3.5). The proof is complete.

##  Appendix C Proofs of the Convergence of Affine Invariant Gradient Flows

###  C.1 Proof of Proposition 3.12

######  Proof C.1.

The Fisher-Rao gradient flow of the KL divergence Eq. 25 can be solved
analytically using the variation of constants formula as follows. First note
that

$\displaystyle\frac{\partial\log\rho_{t}(\theta)}{\partial t}=\log\rho_{\rm
post}(\theta)-\log\rho_{t}(\theta)-\mathbb{E}_{\rho_{t}}[\log\rho_{\rm
post}(\theta)-\log\rho_{t}(\theta)],$

so that

$\displaystyle\frac{\partial e^{t}\log\rho_{t}(\theta)}{\partial
t}=e^{t}\log\rho_{\rm post}(\theta)-e^{t}\mathbb{E}_{\rho_{t}}[\log\rho_{\rm
post}(\theta)-\log\rho_{t}(\theta)].$

Thus

$\log\rho_{t}(\theta)=(1-e^{-t})\log\rho_{\rm
post}(\theta)+e^{-t}\log\rho_{0}(\theta)-\int_{0}^{t}e^{\tau-t}\mathbb{E}_{\rho_{\tau}}[\log\rho_{\rm
post}(\theta)-\log\rho_{\tau}(\theta)]\mathrm{d}\tau.$

It follows that there exists some constant $Z_{t}$ such that

(131)
$\displaystyle\rho_{t}(\theta)=\frac{1}{Z_{t}}\rho_{0}(\theta)^{e^{-t}}\rho_{\rm
post}(\theta)^{1-e^{-t}},\quad\frac{\rho_{t}(\theta)}{\rho_{\rm
post}(\theta)}=\frac{1}{Z_{t}}\left(\frac{\rho_{0}(\theta)}{\rho_{\rm
post}(\theta)}\right)^{e^{-t}}.$

In the following, we first obtain the following lower bound on $Z_{t}$:

(132) $\displaystyle Z_{t}\geq e^{-Ke^{-t}(1+B)},$

where the constants $K,B$ are from Eq. 61 and Eq. 62. In fact, using our
assumptions Eq. 61 and Eq. 62, we have

$\displaystyle Z_{t}$
$\displaystyle=\int\left(\frac{\rho_{0}(\theta)}{\rho_{\rm
post}(\theta)}\right)^{e^{-t}}\rho_{\rm
post}(\theta)\mathrm{d}\theta\geq\int\left(e^{-K(1+|\theta|^{2})}\right)^{e^{-t}}\rho_{\rm
post}(\theta)\mathrm{d}\theta$ $\displaystyle=\int
e^{-Ke^{-t}(1+|\theta|^{2})}\rho_{\rm post}(\theta)\mathrm{d}\theta\geq
e^{\int-Ke^{-t}(1+|\theta|^{2})\rho_{\rm post}(\theta)\mathrm{d}\theta}\geq
e^{-Ke^{-t}(1+B)},$

where in the second to last inequality, we used Jensen’s inequality and the
fact that $e^{x}$ is convex.

By plugging Eq. 62 and Eq. 132 into Eq. 131, we get

(133) $\displaystyle\frac{\rho_{t}(\theta)}{\rho_{\rm
post}(\theta)}=\frac{1}{Z_{t}}\left(\frac{\rho_{0}(\theta)}{\rho_{\rm
post}(\theta)}\right)^{e^{-t}}\leq
e^{Ke^{-t}(1+B)}e^{Ke^{-t}(1+|\theta|^{2})}=e^{Ke^{-t}(2+B+|\theta|^{2})}.$

Using Eq. 133, we get the following upper bound on the KL divergence:

(134) $\displaystyle\begin{split}&{\rm
KL}\Bigl{[}\rho_{t}(\theta)\Big{\|}\rho_{\rm post}(\theta)\Bigr{]}\\\
&\quad=\int\rho_{t}(\theta)\log\frac{\rho_{t}(\theta)}{\rho_{\rm
post}(\theta)}d\theta\leq\int\rho_{t}(\theta)\log(e^{Ke^{-t}(2+B+|\theta|^{2})})d\theta\\\
&\quad=\int\rho_{t}(\theta)Ke^{-t}(2+B+|\theta|^{2})d\theta=Ke^{-t}\left((2+B)+\int|\theta|^{2}\rho_{t}(\theta)d\theta\right).\end{split}$

For the last integral in LABEL:e:KLub1, using Eq. 131 and the Hölder
inequality, we can rewrite it as

(135)
$\displaystyle\begin{split}&\phantom{{}={}}\frac{1}{Z_{t}}\int|\theta|^{2}\rho_{0}(\theta)^{e^{-t}}\rho_{\rm
post}(\theta)^{1-e^{-t}}d\theta=\frac{1}{Z_{t}}\int(|\theta|^{2}\rho_{0}(\theta))^{e^{-t}}(|\theta|^{2}\rho_{\rm
post}(\theta))^{1-e^{-t}}d\theta\\\
&\leq\frac{1}{Z_{t}}\left(\int|\theta|^{2}\rho_{0}(\theta)\mathrm{d}\theta\right)^{e^{-t}}\left(\int(|\theta|^{2}\rho_{\rm
post}(\theta))d\theta\right)^{1-e^{-t}}\leq Be^{Ke^{-t}(1+B)},\end{split}$

where for the last inequality we used Eq. 132 and our assumption Eq. 62.

Combining LABEL:e:KLub1 and LABEL:e:KLub2 together, for $t\geq\log((1+B)K)$,
we have

(136) $\displaystyle{\rm KL}\Bigl{[}\rho_{t}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\leq Ke^{-t}(2+B+Be^{Ke^{-t}(1+B)})\leq(2+B+eB)Ke^{-t}.$

This completes the proof of Proposition 3.12.

###  C.2 Proof of Proposition 3.14

######  Proof C.2.

Thanks to the assumptions and the logarithmic Sobolev inequality, we obtain
the decay of the KL divergence

(137) $\displaystyle\partial_{t}\mathrm{KL}[\rho_{t}\|\rho_{\rm post}]$
$\displaystyle=-\int\rho_{t}[\nabla_{\theta}\log(\frac{\rho_{t}}{\rho_{\rm
post}})]^{T}P[\nabla_{\theta}\log(\frac{\rho_{t}}{\rho_{\rm
post}})]\mathrm{d}\theta$
$\displaystyle\leq-\lambda\int\rho_{t}[\nabla_{\theta}\log(\frac{\rho_{t}}{\rho_{\rm
post}})]^{T}[\nabla_{\theta}\log(\frac{\rho_{t}}{\rho_{\rm
post}})]\mathrm{d}\theta$
$\displaystyle\leq-2\lambda\alpha\mathrm{KL}[\rho_{t}\|\rho_{\rm post}].$

Then we have

(138) $\displaystyle\frac{1}{2}\lVert\rho_{t}-\rho_{\rm
post}\rVert^{2}_{1}\leq\mathrm{KL}[\rho_{t}\|\rho_{\rm
post}]\leq\mathrm{KL}[\rho_{0}\|\rho_{\rm post}]e^{-2\alpha\lambda t}.$

###  C.3 Proof of Lemma 3.15

######  Proof C.3.

The evolution equation for $C_{t}$, under the affine invariant Wasserstein
gradient flow Eq. 47 with preconditioner $P(\rho_{t})=C_{t}$, is

(139) $\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}$
$\displaystyle=\int\frac{\partial\rho_{t}}{\partial t}(\theta-m_{t})(\theta-
m_{t})^{T}\mathrm{d}\theta-\int\rho_{t}\frac{\mathrm{d}m_{t}}{\mathrm{d}t}(\theta-
m_{t})^{T}\mathrm{d}\theta-\int\rho_{t}(\theta-
m_{t})\frac{\mathrm{d}m_{t}}{\mathrm{d}t}^{T}\mathrm{d}\theta$
$\displaystyle=\int\frac{\partial\rho_{t}}{\partial t}(\theta-m_{t})(\theta-
m_{t})^{T}\mathrm{d}\theta$
$\displaystyle=\int\nabla_{\theta}\cdot\Bigl{[}\rho_{t}C_{t}(\nabla_{\theta}\log\rho_{t}-\nabla_{\theta}\log\rho_{\rm
post})\Bigr{]}(\theta-m_{t})(\theta-m_{t})^{T}\mathrm{d}\theta$
$\displaystyle=2C_{t}+C_{t}(\int\rho_{t}(\nabla_{\theta}\log\rho_{\rm
post})(\theta-m_{t})^{T}\mathrm{d}\theta+\int\rho_{t}(\theta-
m_{t})(\nabla_{\theta}\log\rho_{\rm post})^{T}\mathrm{d}\theta)C_{t}.$

Since the parametrized family $C_{t}$ of Hermitian matrices is continuously
differentiable in $t$, we can parameterize its eigenvalues as
$\lambda_{1}(t),\lambda_{2}(t),\cdots,\lambda_{N_{\theta}}(t)$ such that they
are continuously differentiable in $t$ [113, Chapter I, Section 5]. We remark
these eigenvalues may not preserve any ordering. The corresponding normalized
eigenvectors are $u_{1}(t),u_{2}(t),\cdots,u_{N_{\theta}}(t)$. At time $t=0$,
since $\lambda_{0,\min}$, we have
$\lambda_{1}(0),\lambda_{2}(0),\cdots\lambda_{N_{\theta}}(0)\geq\lambda_{0,\min}$.

If $\lambda_{i}(t)$ is an eigenvalue with multiplicity one, then $u_{i}(t)$ is
also differentiable. By taking time derivative on both sides of
$\lambda_{i}(t)u_{i}(t)=C_{t}u_{i}(t)$, and using $u_{i}(t)^{T}u_{i}(t)=1$, we
get

(140)
$\displaystyle\partial_{t}\lambda_{i}(t)u_{i}(t)+\lambda_{i}(t)\partial_{t}u_{i}(t)=\partial_{t}C_{t}u_{i}(t)+C_{t}\partial_{t}u_{i}(t),\quad
u_{i}(t)^{T}\partial_{t}u_{i}(t)=0.$

By multiplying $u_{i}$ on both sides of the first relation in Eq. 140, we get
the following variational formula for the eigenvalues

(141)
$\displaystyle\partial_{t}\lambda_{i}(t)=u_{i}(t)^{T}\partial_{t}C_{t}(t)u_{i}(t).$

If $\lambda_{i}(t)$ is not a simple eigenvalue, $u_{i}(t)$ may not be
continuous at time $t$. Thanks to [113, Chapter I, Section 5], there exists
subsequent limit: $t_{m}\rightarrow t$,
$\lim_{m\rightarrow\infty}u_{i}(t_{m})=\widetilde{u}_{i}$, and
$C_{t}\widetilde{u}_{i}=\lambda_{i}(t)\widetilde{u}_{i}$. Moreover, we still
have
$\partial_{t}\lambda_{i}(t)=\widetilde{u}_{i}(t)^{T}\partial_{t}C_{t}\widetilde{u}_{i}(t)$.
For simplicity of notations, in the rest of this proof, we will not
distinguish $u_{i}(t)$ and $\widetilde{u}_{i}(t)$. And simply denote them as
$u_{i}$.

By multiplying $u_{i}$ on both sides of Eq. 139, we get

(142)
$\displaystyle\partial_{t}\lambda_{i}(t)=2\lambda_{i}(t)+2\lambda_{i}(t)\int\rho_{t}(u_{i}^{T}\nabla_{\theta}\log\rho_{\rm
post})(\theta-m_{t})^{T}u_{i}\mathrm{d}\theta.$

Through Taylor expansion, we have

(143) $\displaystyle u_{i}^{T}(\nabla_{\theta}\log\rho_{\rm
post}(\theta)-\nabla_{\theta}\log\rho_{\rm post}(m_{t}))$
$\displaystyle\quad\quad\quad=\int_{0}^{1}u_{i}(t)^{T}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(m_{t}+\tau(\theta-m_{t}))(\theta-m_{t})\mathrm{d}\tau.$

Bringing LABEL:eq:eks-taylor into Eq. 142 leads to

(144) $\displaystyle\partial_{t}\lambda_{i}(t)$ $\displaystyle=$
$\displaystyle
2\lambda_{i}(t)+2\lambda_{i}(t)\int_{0}^{1}\int\rho_{t}u_{i}^{T}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta-m_{t})(\theta-m_{t})^{T}u_{i}\mathrm{d}\theta\mathrm{d}\tau$
$\displaystyle=$ $\displaystyle
2\lambda_{i}(t)+2\lambda_{i}(t)\int_{0}^{1}\int\rho_{t}u_{i}^{T}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta-m_{t})(\theta-m_{t})^{T}u_{i}\mathrm{d}\theta\mathrm{d}\tau$
$\displaystyle\geq$ $\displaystyle
2\lambda_{i}(t)-2\lambda_{i}(t)\int_{0}^{1}\sqrt{\int\rho_{t}(u_{i}^{T}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta-m_{t}))^{2}\mathrm{d}\theta\int\rho_{t}((\theta-
m_{t})^{T}u_{i})^{2}\mathrm{d}\theta}\mathrm{d}\tau$ $\displaystyle\geq$
$\displaystyle
2\lambda_{i}(t)-2\lambda_{i}(t)\int_{0}^{1}\sqrt{\int\rho_{t}\lVert\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}\rVert_{2}^{2}{\rm tr}\\{(\theta-m_{t})(\theta-
m_{t})^{T}\\}\mathrm{d}\theta\lambda_{i}}\mathrm{d}\tau$ $\displaystyle\geq$
$\displaystyle 2\lambda_{i}(t)-2\lambda_{i}(t)^{3/2}\beta\sqrt{{\rm
tr}(C_{t})}.$

Here we use Cauchy’s inequality in the third equation and, in the fourth
equation, we use the following two facts:

$\displaystyle\int\rho_{t}(u_{i}^{T}\nabla_{\theta}\nabla_{\theta}$
$\displaystyle\log\rho_{\rm post}(\theta-m_{t}))^{2}\mathrm{d}\theta$
$\displaystyle\leq\int\rho_{t}\lVert
u_{i}\rVert_{2}^{2}\lVert\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}\rVert_{2}^{2}\lVert\theta-m_{t}\rVert_{2}^{2}\mathrm{d}\theta$
$\displaystyle=\int\rho_{t}\lVert\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}\rVert_{2}^{2}{\rm tr}\\{(\theta-m_{t})(\theta-
m_{t})^{T}\\}\mathrm{d}\theta;$ $\displaystyle\int\rho_{t}((\theta-m_{t})^{T}$
$\displaystyle u_{i})^{2}\mathrm{d}\theta=\lambda_{i}.$

When $N_{\theta}=1$, we have ${\rm tr}(C_{t})=\lambda_{i}$. Therefore, if
$\lambda_{i}\leq\frac{1}{\beta}$, we have $\partial_{t}\lambda_{i}(t)\geq 0$,
which leads to that $C_{t}\geq\min\\{\frac{1}{\beta},\lambda_{0,\min}\\}$.
When $N_{\theta}>1$ and ${\rm tr}\\{C_{t}\\}\leq K$, if
$\lambda_{i}\leq\frac{1}{\beta^{2}K}$, we have $\partial_{t}\lambda_{i}(t)\geq
0$, which leads to that
$C_{t}\succeq\min\\{\frac{1}{\beta^{2}K},\lambda_{0,\min}\\}$.

##  Appendix D Proofs for Gaussian Approximate Gradient Flows

###  D.1 Preliminaries

We start with the following Stein’s identities concerning the Gaussian density
function $\rho_{a}.$

######  Lemma D.1.

Assume $\theta\sim\mathcal{N}(m,C)$ with density
$\rho_{a}(\theta)=\rho_{a}(\theta;m,C)$, we have

(145)
$\displaystyle\nabla_{m}\rho_{a}(\theta)=-\nabla_{\theta}\rho_{a}(\theta)\quad\text{and}\quad\nabla_{C}\rho_{a}(\theta)=\frac{1}{2}\nabla_{\theta}\nabla_{\theta}\rho_{a}(\theta).$

Furthermore, for any scalar field $f:\mathbb{R}^{N_{\theta}}\to\mathbb{R}$ and
vector field $g:\mathbb{R}^{N_{\theta}}\to\mathbb{R}^{N_{\theta}}$, we have

(146)
$\displaystyle\mathbb{E}_{\rho_{a}}[\nabla_{\theta}g(\theta)]=\nabla_{m}\mathbb{E}_{\rho_{a}}[g(\theta)]=\mathrm{Cov}[g(\theta),\theta]C^{-1},$
$\displaystyle\mathbb{E}_{\rho_{a}}[\nabla_{\theta}\nabla_{\theta}f(\theta)]=\mathrm{Cov}[\nabla_{\theta}f(\theta),\theta]C^{-1}=-C^{-1}\mathbb{E}_{\rho_{a}}\Bigl{[}\bigl{(}C-(\theta-m)(\theta-m)^{T}\bigr{)}f\Bigr{]}C^{-1}.$

######  Proof D.2.

For Gaussian density function $\rho_{a}$, we have

$\displaystyle\nabla_{m}\rho_{a}(\theta)$
$\displaystyle=\nabla_{m}\frac{1}{\sqrt{|2\pi
C|}}\exp\Bigl{\\{}-\frac{1}{2}(\theta-m)C^{-1}(\theta-m)\Bigr{\\}}$
$\displaystyle=C^{-1}(\theta-m)\rho_{a}(\theta)=-\nabla_{\theta}\rho_{a}(\theta),$
$\displaystyle\nabla_{C}\rho_{a}(\theta)$
$\displaystyle=\rho_{a}(\theta)\Bigl{(}-\frac{1}{2}\frac{\partial\log|C|}{\partial
C}-\frac{1}{2}\frac{\partial(\theta-m)^{T}C^{-1}(\theta-m)}{\partial
C}\Bigr{)}$
$\displaystyle=-\frac{1}{2}\rho_{a}(\theta)\Bigl{(}C^{-1}-C^{-1}(\theta-m)(\theta-m)^{T}C^{-1}\Bigr{)}=\frac{1}{2}\nabla_{\theta}\nabla_{\theta}\rho_{a}(\theta).$

For any scalar field $f(\theta)$ and vector field $g(\theta)$, we have

(147) $\displaystyle\mathbb{E}_{\rho_{a}}[\nabla_{\theta}g(\theta)]$
$\displaystyle=\int\nabla_{\theta}g(\theta)\rho_{a}(\theta)\mathrm{d}\theta=-\int
g(\theta)\nabla_{\theta}\rho_{a}(\theta)^{T}\mathrm{d}\theta=\nabla_{m}\mathbb{E}_{\rho_{a}}[g(\theta)]$
$\displaystyle=\int
g(\theta)(\theta-m)^{T}C^{-1}\rho_{a}(\theta)\mathrm{d}\theta=\mathrm{Cov}[g(\theta),\theta]C^{-1},$
$\displaystyle\mathbb{E}_{\rho_{a}}[\nabla_{\theta}\nabla_{\theta}f(\theta)]$
$\displaystyle=\int\nabla_{\theta}\nabla_{\theta}f(\theta)\rho_{a}(\theta)\mathrm{d}\theta=-\int\nabla_{\theta}f(\theta)\nabla_{\theta}\rho_{a}(\theta)^{T}\mathrm{d}\theta$
$\displaystyle=\int\nabla_{\theta}f(\theta)(\theta-m)^{T}C^{-1}\rho_{a}(\theta)\mathrm{d}\theta=\mathrm{Cov}[\nabla_{\theta}f(\theta),\theta]C^{-1}$
$\displaystyle=\int
f(\theta)\nabla_{\theta}\nabla_{\theta}\rho_{a}(\theta)\mathrm{d}\theta=-C^{-1}\mathbb{E}_{\rho_{a}}\Bigl{[}\bigl{(}C-(\theta-m)(\theta-m)^{T}\bigr{)}f\Bigr{]}C^{-1}.$

The following lemma is proved in [79, Theorem 1]:

######  Lemma D.3.

Consider the KL divergence

$\displaystyle{\rm KL}\Bigl{[}\rho_{a}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}=-\frac{1}{2}\log\bigl{|}C\bigr{|}-\int\rho_{a}(\theta)\log\rho_{\rm
post}(\theta)\mathrm{d}\theta+{\rm const}.$

For fixed $\rho_{\rm post}$ the minimizer of this divergence over the space
$\mathcal{P}^{G}$, so that, for $a_{\star}=(m_{\star},C_{\star})$ and
$\rho_{a_{\star}}(\theta)=\mathcal{N}(m_{\star},C_{\star})$, it follows that

$\displaystyle\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}=0\quad\textrm{and}\quad
C_{\star}^{-1}=-\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}.$

###  D.2 Consistency of Riemannian Perspective in Section 4.1.2

Here we prove Proposition 4.1, Proposition 4.2, and Lemma 4.3, which
identifies specific gradient flows, within the paper, that satisfy the
assumptions required for application of Proposition 4.2.

######  Proof D.4 (Proof of Proposition 4.1).

Recall from (65) that any element in $T_{\rho_{a_{t}}}\mathcal{P}^{G}$ is
given in the form $\nabla_{a}\rho_{a_{t}}\cdot\sigma$. Thus we have, for
$\sigma_{t}\in T_{\rho_{t}}\mathcal{P}$ in Eq. 71, it holds that

(148) $\displaystyle
g_{\rho_{a_{t}}}(\sigma_{t},\nabla_{a}\rho_{a_{t}}\cdot\sigma)$
$\displaystyle=g_{\rho_{a_{t}}}(-$
$\displaystyle=-\Bigl{\langle}\frac{\delta\mathcal{E}}{\delta\rho}\Bigr{|}_{\rho=\rho_{a_{t}}},\nabla_{a}\rho_{a_{t}}\cdot\sigma\Bigr{\rangle}$
$\displaystyle=-\Bigl{\langle}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}},\sigma\Bigr{\rangle}=\mathfrak{g}_{{a_{t}}}(-\mathfrak{M}(a_{t})^{-1}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}},\sigma)$
$\displaystyle=g_{\rho_{a_{t}}}\Bigl{(}-\nabla_{a}\rho_{a_{t}}\cdot\mathfrak{M}(a_{t})^{-1}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}},\nabla_{a}\rho_{a_{t}}\cdot\sigma\Bigr{)}.$

Combining Eq. 148 and the definition Eq. 70 leads to

(149) $\displaystyle
P^{G}\sigma_{t}=-\nabla_{a}\rho_{a_{t}}\mathfrak{M}(a_{t})^{-1}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}}.$

By plugging (149) into (72), we get

(150) $\displaystyle\nabla_{a}\rho_{a_{t}}\cdot\frac{\partial a_{t}}{\partial
t}=\frac{\partial\rho_{a_{t}}}{\partial
t}=-\nabla_{a}\rho_{a_{t}}\cdot\mathfrak{M}(a_{t})^{-1}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}}.$

Since we may choose $\nabla_{a}\rho_{a_{t}}$ so that it has non-zero values in
each one of its entries, we can remove $\nabla_{a}\rho_{a_{t}}$ on both sides
of Eq. 150, and get Eq. 68.

######  Proof D.5 (Proof of Proposition 4.2).

The mean and covariance evolution equations of Eq. 72 are

(151)
$\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=\int-P^{G}\sigma_{t}(\theta,\rho_{a_{t}})\theta\mathrm{d}\theta,$
$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}=\int-P^{G}\sigma_{t}(\theta,\rho_{a_{t}})(\theta-
m_{t})(\theta-m_{t})^{T}\mathrm{d}\theta,$

where $\rho_{a_{t}}=\mathcal{N}(m_{t},C_{t})$. For any $f(\theta)\in
T^{*}_{\rho_{a_{t}}}\mathcal{P}^{G}$, we have

$\displaystyle\bigl{\langle}P^{G}\sigma_{t}(\theta,\rho_{a_{t}}),f(\theta)\bigr{\rangle}$
$\displaystyle=g_{\rho_{a_{t}}}(P^{G}\sigma_{t}(\theta,\rho_{a_{t}}),$
$\displaystyle=g_{\rho_{a_{t}}}(\sigma_{t}(\theta,\rho_{a_{t}}),$

Using assumption Eq. 75 and taking $f(\theta)$ to be linear and quadratic
functions of $\theta$, LABEL:eq:mC-Riemannian become

(152)
$\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=\int-\sigma_{t}(\theta,\rho_{a_{t}})\theta\mathrm{d}\theta,$
$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}=\int-\sigma_{t}(\theta,\rho_{a_{t}})(\theta-
m_{t})(\theta-m_{t})^{T}\mathrm{d}\theta,$

delivering LABEL:eq:mC-Momentum. This indicates that the mean and covariance
evolution equations of the gradient flow Eq. 72 obtained from Riemannian
perspective are the same as the closed system obtained by the moment closure
approach. Combining Proposition 4.1 completes the proof.

######  Proof D.6 (Proof of Lemma 4.3).

Using the calculation in the proof of Lemma D.1, the tangent space of the
Gaussian density manifold at $\rho_{a}$ with $a=[m,C]$ is

(153) $\displaystyle T_{\rho_{a}}\mathcal{P}^{G}$ $\displaystyle={\rm
span}\bigl{\\{}\rho_{a}[C^{-1}(\theta-m)]_{i},\,\rho_{a}[C^{-1}(\theta-m)(\theta-m)^{T}C^{-1}-C^{-1}]_{ij}\bigr{\\}}$
$\displaystyle={\rm
span}\bigl{\\{}\rho_{a}(\theta_{i}-\mathbb{E}_{\rho_{a}}[\theta_{i}]),\,\rho_{a}(\theta_{i}\theta_{j}-\mathbb{E}_{\rho_{a}}[\theta_{i}\theta_{j}])\bigr{\\}},$

and $1\leq i,j\leq N_{\theta}$.

For the Fisher-Rao metric, we have

(154) $\displaystyle\M{}^{\rm FR}(\rho_{a})^{-1}{\rm
span}\bigl{\\{}\theta_{i},\theta_{i}\theta_{j}\bigr{\\}}={\rm
span}\bigl{\\{}\rho_{a}(\theta_{i}-\mathbb{E}_{\rho_{a}}[\theta_{i}]),\rho_{a}(\theta_{i}\theta_{j}-\mathbb{E}_{\rho_{a}}[\theta_{i}\theta_{j}])\bigr{\\}}=T_{\rho_{a}}\mathcal{P}^{G}.$

For the affine invariant Wasserstein metric with preconditioner $P$
independent of $\theta$, we have

(155) $\displaystyle\quad={\rm
span}\bigl{\\{}\nabla_{\theta}\cdot(\rho_{a}(\theta)P(\rho_{a})e_{i}),\,\nabla_{\theta}\cdot(\rho_{a}(\theta)P(\rho_{a})e_{i}\theta_{j})\bigr{\\}}$
$\displaystyle\quad={\rm
span}\bigl{\\{}\nabla_{\theta}\cdot(\rho_{a}(\theta)(b^{\prime}+A^{\prime}\theta))\quad\forall
b^{\prime}\in$ $\displaystyle\quad={\rm span}\bigl{\\{}\rho_{a}(\theta)[{\rm
tr}(A^{\prime})-(\theta-m)^{T}C^{-1}(b^{\prime}+A^{\prime}\theta)]\quad\forall
b^{\prime}\in$ $\displaystyle\quad=T_{\rho_{a}}\mathcal{P}^{G}.$

Here $e_{i}$ is the $i$-th unit vector.

For the affine invariant Stein metric with preconditioner $P$ independent of
$\theta$ and with a bilinear kernel
$\kappa(\theta,\theta^{\prime},\rho)=(\theta-m)^{T}A(\rho)(\theta^{\prime}-m)+b(\rho)$
($b\neq 0$, $A$ nonsingular), we have

(156) $\displaystyle\quad={\rm
span}\Bigl{\\{}\nabla_{\theta}\cdot\Bigl{(}\rho_{a}(\theta)P(\rho_{a})\int\kappa\rho_{a}(\theta^{\prime})e_{i}\mathrm{d}\theta^{\prime}\Bigr{)},\nabla_{\theta}\cdot\Bigl{(}\rho_{a}(\theta)P(\rho_{a})\int\kappa\rho_{a}(\theta^{\prime})e_{i}\theta_{j}^{\prime}\mathrm{d}\theta^{\prime}\Bigr{)}\Bigr{\\}}$
$\displaystyle\quad={\rm
span}\Bigl{\\{}\nabla_{\theta}\cdot\Bigl{(}\rho_{a}(\theta)P(\rho_{a})be_{i}\Bigr{)},\nabla_{\theta}\cdot\Bigl{(}\rho_{a}(\theta)P(\rho_{a})[(\theta-m)^{T}Ae_{j}c_{jj}e_{i}+be_{i}m_{j}]\Bigr{)}\Bigr{\\}}$
$\displaystyle\quad={\rm
span}\Bigl{\\{}\nabla_{\theta}\cdot(\rho_{a}(\theta)(b^{\prime}+A^{\prime}\theta))\quad\forall
b^{\prime}\in$ $\displaystyle\quad=T_{\rho_{a}}\mathcal{P}^{G}.$

###  D.3 Proof of Proposition 4.5

######  Proof D.7.

Following Section 4.1.3, we consider the invertible affine transformation
$\varphi(\theta)=A\theta+b$ in . For Gaussian density space where $a=(m,C)$,
we have a corresponding invertible affine transformation in , such that
$\tilde{a}=A^{G}a+b^{G}$, where $A^{G}$ and $b^{G}$ depend only on $A$ and $b$
and are defined by the identities $\tilde{m}=Am+b$ and $\tilde{C}=ACA^{T}.$
Furthermore, the corresponding transformation tangent vector $\sigma\in$ in
$T_{\rho_{a}}\mathcal{P}^{G}$ is $\tilde{\sigma}=A^{G}\sigma\in$.

From the definition Eq. 66 and Proposition 3.4, we have

$\displaystyle\langle\mathfrak{M}(a)\sigma_{1},\sigma_{2}\rangle$
$\displaystyle=g_{\rho_{a}}(\nabla_{a}\rho_{a}\cdot\sigma_{1},\nabla_{a}\rho_{a}\cdot\sigma_{2})=g_{\varphi\\#\rho_{a}}(\varphi\\#\nabla_{a}\rho_{a}\cdot\sigma_{1},\varphi\\#\nabla_{a}\rho_{a}\cdot\sigma_{2})$
$\displaystyle=g_{\rho_{\tilde{a}}}(\nabla_{\tilde{a}}\rho_{\tilde{a}}\cdot\tilde{\sigma}_{1},\nabla_{\tilde{a}}\rho_{\tilde{a}}\cdot\tilde{\sigma}_{2})=\langle\mathfrak{M}(\tilde{a})\tilde{\sigma}_{1},\tilde{\sigma}_{2}\rangle,$

which leads to $\mathfrak{M}(a)=A^{G^{T}}\mathfrak{M}(\tilde{a})A^{G}$.
Following the definition of $\tilde{a}_{t}$, we have

$\displaystyle\frac{\partial\tilde{a}_{t}}{\partial
t}=-A^{G}\mathfrak{M}(a_{t})^{-1}\left.\frac{\partial\mathcal{E}(\rho_{a})}{\partial
a}\right|_{a=a_{t}}$
$\displaystyle\quad=-A^{G}\mathfrak{M}(a_{t})^{-1}A^{G^{T}}\left.\frac{\partial\tilde{\mathcal{E}}(\rho_{\tilde{a}})}{\partial\tilde{a}}\right|_{\tilde{a}=\tilde{a}_{t}}$
$\displaystyle\quad=-\mathfrak{M}(\tilde{a}_{t})^{-1}\left.\frac{\partial\tilde{\mathcal{E}}(\rho_{\tilde{a}})}{\partial\tilde{a}}\right|_{\tilde{a}=\tilde{a}_{t}}.$

###  D.4 Proof of Lemma 4.6

######  Proof D.8.

Define $e_{t}:=\theta_{t}-m_{t}$,
$\mathsf{A}_{t}=\mathsf{A}(\rho_{t},\rho_{\rm post})$, and
$\mathsf{b}_{t}=\mathsf{b}(\rho_{t},\rho_{\rm post})$. Taking expectation on
both sides of (77), we obtain that the evolution of $m_{t}$ and $e_{t}$ are
the ODEs:

$\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=\mathsf{b}_{t}\quad\mathrm{and}\quad\frac{\mathrm{d}e_{t}}{\mathrm{d}t}=\mathsf{A}_{t}e_{t}.$

To solve the ODE for $e_{t}$, let $B_{t}:$ denote the solution of the ODE
system

$\frac{\mathrm{d}B_{t}}{\mathrm{d}t}=\mathsf{A}_{t}B_{t}\quad\textrm{and}\quad
B_{0}=$

With this, one has the explicit formula of the solution $e_{t}=B_{t}e_{0}$. It
implies that $e_{t}\sim\mathcal{N}(0,C_{t})$ where
$C_{t}=B_{t}C_{0}B_{t}^{T}$. Consequently, the law of $\theta_{t}$ is Gaussian
and $C_{t}$ satisfies the ODE

$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}$
$\displaystyle=\frac{\mathrm{d}B_{t}}{\mathrm{d}t}C_{0}B_{t}^{T}+B_{t}C_{0}\frac{\mathrm{d}B_{t}}{\mathrm{d}t}^{T}=\mathsf{A}_{t}C_{t}+C_{t}\mathsf{A}_{t}^{T}.$

###  D.5 Proof of Proposition 4.7

######  Proof D.9.

Let $\varphi:\theta\rightarrow\tilde{\theta}$ denote an invertible affine
transformation in , where $\varphi(\theta)=A\theta+b$ with $A\in$, $b\in$, and
$A$ invertible. The affine invariance property of the mean and covariance
evolution equation Eq. 78 leads to the following relation

(157) $\displaystyle\mathsf{b}(\varphi\\#\rho_{a},\varphi\\#\rho_{\rm
post})=A\mathsf{b}(\rho_{a},\rho_{\rm post}),$
$\displaystyle\mathsf{A}(\varphi\\#\rho_{a},\varphi\\#\rho_{\rm
post})=A\mathsf{A}(\rho_{a},\rho_{\rm post})A^{-1}.$

Let $f(\theta_{t};\rho_{a_{t}},\rho_{\rm
post})=\mathsf{A}(\rho_{a_{t}},\rho_{\rm
post})(\theta_{t}-m_{t})+\mathsf{b}(\rho_{a_{t}},\rho_{\rm post})$. Using
Equation 157, we get

$\displaystyle Af(\theta_{t};\rho_{a_{t}},\rho_{\rm post})$
$\displaystyle=A\mathsf{A}(\rho_{a_{t}},\rho_{\rm
post})(\theta_{t}-m_{t})+A\mathsf{b}(\rho_{a_{t}},\rho_{\rm post})$
$\displaystyle=A\mathsf{A}(\rho_{a_{t}},\rho_{\rm
post})A^{-1}A(\theta_{t}-m_{t})+A\mathsf{b}(\rho_{a_{t}},\rho_{\rm post})$
$\displaystyle=f(\varphi(\theta_{t});\varphi\\#\rho_{t},\varphi\\#\rho_{\rm
post}).$

Therefore, the mean-field equation is affine invariant. The proof is complete.

##  Appendix E Proofs of the Convergence of Gaussian Approximate Gradient
Flows

###  E.1 Proof of Proposition 4.8

######  Proof E.1.

Under the Gaussian posterior assumption Eq. 97 and the fact that

$\frac{\mathrm{d}C_{t}^{-1}}{\mathrm{d}t}=-C_{t}^{-1}\frac{\mathrm{d}C_{t}}{\mathrm{d}t}C_{t}^{-1},$

the Gaussian approximate Fisher-Rao gradient flow Eq. 82 becomes

(158)
$\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=C_{t}C_{\star}^{-1}\bigl{(}m_{\star}-m_{t}\bigr{)},$
$\displaystyle\frac{\mathrm{d}C_{t}^{-1}}{\mathrm{d}t}=-C_{t}^{-1}+C_{\star}^{-1}.$

The covariance update equation has analytical solution

(159) $\displaystyle C_{t}^{-1}=(1-e^{-t})C_{\star}^{-1}+e^{-t}C_{0}^{-1}.$

We reformulate Eq. 98a as

$m_{\star}-m_{t}=e^{-t}C_{t}C_{0}^{-1}\bigl{(}m_{\star}-m_{0}\bigr{)}.$

Computing its time derivative leads to

(160) $\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}$
$\displaystyle=-e^{-t}C_{t}C_{0}^{-1}(m_{0}-m_{\star})+e^{-t}\frac{\mathrm{d}C_{t}}{\mathrm{d}t}C_{0}^{-1}(m_{0}-m_{\star})$
$\displaystyle=-e^{-t}C_{t}C_{0}^{-1}(m_{0}-m_{\star})+e^{-t}(C_{t}-C_{t}C_{\star}^{-1}C_{t})C_{0}^{-1}(m_{0}-m_{\star})$
$\displaystyle=C_{t}C_{\star}^{-1}e^{-t}C_{t}C_{0}^{-1}(m_{\star}-m_{0})$
$\displaystyle=C_{t}C_{\star}^{-1}\bigl{(}m_{\star}-m_{t}\bigr{)}.$

###  E.2 Proof of Convergence for Gaussian Posterior

In this section we prove Proposition 4.9. Under the assumption Eq. 97, for
$\theta_{t}\sim\mathcal{N}(m_{t},C_{t})$ with density $\rho_{a_{t}}$, we have

(161)
$\displaystyle\mathbb{E}_{\rho_{a_{t}}}\bigl{[}\nabla_{\theta}\log\rho_{\rm
post}(\theta_{t})\bigr{]}=-C_{\star}^{-1}(m_{t}-m_{\star}),\quad\mathbb{E}_{\rho_{a_{t}}}\bigl{[}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta_{t})\bigr{]}=-C_{\star}^{-1}.$

Here $m_{\star}$ and $C_{\star}$ are the posterior mean and covariance given
in Eq. 9. We have explicit expressions for the Gaussian approximate gradient
flows Eqs. 96, 82, and LABEL:eq:Gaussian-Wasserstein:

(162) $\displaystyle\begin{split}\text{Gaussian approximate}&\text{ gradient
flow:}\\\
&\partial_{t}{m}_{t}=-C_{\star}^{-1}(m_{t}-m_{\star}),\quad\partial_{t}{C}_{t}=\frac{1}{2}C_{t}^{-1}-\frac{1}{2}C_{\star}^{-1}.\\\
\text{Gaussian approximate}&\text{ Fisher-Rao gradient flow:}\\\
&\partial_{t}{m}_{t}=-C_{t}C_{\star}^{-1}(m_{t}-m_{\star}),\quad\partial_{t}{C}_{t}=C_{t}-C_{t}C_{\star}^{-1}C_{t}.\\\
\text{Gaussian approximate}&\text{ Wasserstein gradient flow:}\\\
&\partial_{t}{m}_{t}=-C_{\star}^{-1}(m_{t}-m_{\star}),\quad\partial_{t}{C}_{t}=2\I-C_{t}C^{-1}_{\star}-C^{-1}_{\star}C_{t}.\end{split}$

For the dynamics of $m_{t}$ for both Gaussian approximate gradient flow and
Gaussian approximate Wasserstein gradient flow, we have

(163) $\displaystyle m_{t}-m_{\star}=e^{-tC_{\star}^{-1}}(m_{0}-m_{\star}).$

By taking the $2-$norm on both sides, using $\|\cdot\|_{2}$ to denote both the
vector and induced matrix norms, and recalling that the largest eigenvalue of
$C_{\star}$ is $\lambda_{\star,\max}$, we obtain

(164)
$\displaystyle\|m_{t}-m_{\star}\|_{2}\leq\|e^{-tC_{\star}^{-1}}\|_{2}\|m_{0}-m_{\star})\|_{2}\leq
e^{-t/\lambda_{\star,\max}}\|m_{0}-m_{\star}\|_{2}=\mathcal{O}(e^{-t/\lambda_{\star,\max}}).$

The bound can be achieved, when $m_{0}-m_{\star}$ has nonzero component in the
$C_{\star}$ eigenvector direction corresponding to $\lambda_{\star,\max}$.

For the Gaussian approximate Fisher-Rao gradient flow, thanks to the explicit
formula Eq. 98a, we find that

(165) $\displaystyle\|m_{t}-m_{\star}\|_{2}\leq
e^{-t}\max\\{\|C_{\star}\|_{2},\|C_{0}\|_{2}\\}\|C_{0}^{-1}\|_{2}\|m_{0}-m_{\star}\|_{2}=\mathcal{O}(e^{-t}).$

The bound can be achieved, when $m_{0}-m_{\star}$ is nonzero.

Next, we analyze the dynamics of the covariance matrix $C_{t}$. Our
initialization $C_{0}=\lambda_{0}$, commutes with $C_{\star},C_{\star}^{-1}$.
It follows that $C_{t}$ commutes with $C_{\star},C_{\star}^{-1}$ for any
$t\geq 0$ and all gradient flows in Eq. 162, since $0$ is the unique solution
of the evolution ordinary differential equation of
$C_{t}C_{{\star}}-C_{{\star}}C_{t}$. So we can diagonalize
$C_{t},C_{t}^{-1},C_{\star},C_{\star}^{-1}$ simultaneously, and write down the
dynamics of the eigenvalues of $C_{t}$. For any eigenvalue $\lambda_{t}$ of
$C_{t}$, it satisfies the following differential equations,

(166) $\displaystyle\begin{split}\text{Gaussian approximate gradient
flow:}\quad&\partial_{t}\lambda_{t}=\frac{1}{2\lambda_{t}}-\frac{1}{2\lambda_{\star}},\\\
\text{Gaussian approximate Fisher-Rao gradient
flow:}\quad&\partial_{t}{\lambda}_{t}=\lambda_{t}-\lambda^{2}_{t}\lambda_{\star}^{-1},\\\
\text{Gaussian approximate Wasserstein gradient
flow:}\quad&\partial_{t}{\lambda}_{t}=2-\frac{2\lambda_{t}}{\lambda_{\star}},\end{split}$

where $\lambda_{\star}$ is the corresponding eigenvalue of $C_{\star}$. From
Eq. 166, we know that $\lambda_{t}$ is bounded between $\lambda_{0}$ and
$\lambda_{\star}$. Moreover, the ordinary differential equations in Eq. 166
can be solved explicitly.

For the Gaussian approximate gradient flow

(167)
$\displaystyle\lambda_{t}-\lambda_{\star}=(\lambda_{0}-\lambda_{\star})e^{-\frac{t}{2\lambda_{\star}^{2}}-\frac{\lambda_{t}-\lambda_{0}}{\lambda_{\star}}},\quad|\lambda_{t}-\lambda_{\star}|=\mathcal{O}(e^{-t/2\lambda_{\star}^{2}}).$

Since the largest eigenvalue of $C_{\star}$ is $\lambda_{\star,\max}$, we
conclude that

$\|C_{t}-C_{\star}\|_{2}=\mathcal{O}(e^{-t/2\lambda_{\star,\max}^{2}}).$

For the Gaussian approximate Fisher-Rao gradient flow,

(168)
$\displaystyle\lambda_{t}=\frac{\lambda_{\star}}{1+\left(\frac{\lambda_{\star}}{\lambda_{0}}-1\right)e^{-t}},\quad|\lambda_{t}-\lambda_{\star}|=\mathcal{O}(e^{-t}).$

It follows that

$\|C_{t}-C_{\star}\|_{2}=\mathcal{O}(e^{-t}).$

For the Gaussian approximate Wasserstein gradient flow

(169)
$\displaystyle\lambda_{t}=\lambda_{\star}+e^{-2t/\lambda_{\star}}(\lambda_{0}-\lambda_{\star}),\quad|\lambda_{t}-\lambda_{0}|=\mathcal{O}(e^{-2t/\lambda_{\star}}).$

Since the large eigenvalue of $C_{\star}$ is $\lambda_{\star,\max}$, we
conclude that

$\|C_{t}-C_{\star}\|_{2}=\mathcal{O}(e^{-2t/\lambda_{\star,\max}}).$

###  E.3 Proof of Convergence for Logconcave Posterior

Before proving Proposition 4.10, we first show that the covariance matrices
$C_{t}$ are well conditioned for all positive times $t$, under the gradient
flows Eqs. 96, 82, and LABEL:eq:Gaussian-Wasserstein.

######  Lemma E.2.

Under assumption Eq. 64, if the initial covariance matrix satisfies
$\lambda_{0,\min}$, then for Eqs. 96, 82, and LABEL:eq:Gaussian-Wasserstein,
we have for all $t\geq 0$,

$\min\\{\lambda_{0,\min},1/\beta\\}$

######  Proof E.3.

Following the proof of Section C.3, we can parametrize eigenvalues of $C_{t}$
as $\lambda_{1}(t),\lambda_{2}(t),\cdots,\lambda_{N_{\theta}}(t)$ such that
they are continuous differentiable in $t$. The corresponding normalized
eigenvectors are $u_{1}(t),u_{2}(t),\cdots,u_{N_{\theta}}(t)$. At time $t=0$,
since $\lambda_{0,\min}$, we have
$\lambda_{1}(0),\lambda_{2}(0),\cdots\lambda_{N_{\theta}}(0)\geq\lambda_{0,\min}$.
We have the following variational formula for the eigenvalues

(170) $\displaystyle\partial_{t}\lambda_{i}=u_{i}^{T}\partial_{t}C_{t}u_{i}.$

For the Gaussian approximate gradient flow Eq. 96, using assumption Eq. 64, we
have

(171) $\displaystyle\frac{1}{2}C_{t}^{-1}-\frac{1}{2}\beta$

By plugging Eq. 171 into Eq. 170 and noticing
$u_{i}(t)^{T}C_{t}^{-1}u_{i}(t)=1/\lambda_{i}(t)$, we get

(172)
$\displaystyle\frac{1}{2\lambda_{i}(t)}-\frac{\beta}{2}\leq\partial_{t}{\lambda}_{i}(t)\leq\frac{1}{2\lambda_{i}(t)}-\frac{\alpha}{2},\quad\lambda_{0,\min}\leq\lambda_{i}(0)\leq\lambda_{0,\max}.$

It follows that if $\lambda_{i}(t)\geq 1/\alpha$, then
$\partial_{t}{\lambda}_{i}(t)\leq 0$; if $\lambda_{i}(t)\leq 1/\beta$, then
$\partial_{t}{\lambda}_{i}(t)\geq 0$. Recall from our choice of $C_{0}$, we
have $\lambda_{0,\min}\leq\lambda_{i}(t)\leq\lambda_{0,\max}$. We conclude
that
$\min\\{\lambda_{0,\min},1/\beta\\}\leq\lambda_{i}(t)\leq\max\\{\lambda_{0,\max},1/\alpha\\}$.

For the Gaussian approximate Fisher-Rao gradient flow in Eq. 82, we have

(173) $\displaystyle C_{t}-\beta
C_{t}^{2}\preceq\partial_{t}C_{t}=C_{t}+C_{t}\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)]C_{t}\preceq C_{t}-\alpha C_{t}^{2}.$

By plugging Eq. 173 into Eq. 170 and noticing
$u_{i}(t)^{T}C^{2}_{t}u_{i}(t)=\lambda^{2}_{i}(t)$, we get

(174)
$\displaystyle\lambda_{i}(t)-\beta\lambda_{i}(t)^{2}\leq\partial_{t}{\lambda}_{i}(t)\leq\lambda_{i}(t)-\alpha\lambda_{i}(t)^{2},\quad\lambda_{0,\min}\leq\lambda_{i}(0)\leq\lambda_{0,\max}.$

If $\lambda_{i}(t)\geq 1/\alpha$, then $\partial_{t}{\lambda}_{i}(t)\leq 0$;
if $\lambda_{i}(t)\leq 1/\beta$, then $\partial_{t}{\lambda}_{i}(t)\geq 0$. We
conclude that
$\min\\{\lambda_{0,\min},1/\beta\\}\leq\lambda_{i}(t)\leq\max\\{\lambda_{0,\max},1/\alpha\\}$.

For the Gaussian approximate Wasserstein gradient flow in LABEL:eq:Gaussian-
Wasserstein, we have

(175) $\displaystyle\partial_{t}C_{t}=2$

Plugging Eq. 175 into Eq. 170 and noticing

(176) $\displaystyle
u_{i}(t)^{T}\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)]C_{t}u_{i}(t)=\lambda_{i}(t)u_{i}(t)^{T}\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)]u_{i}(t),$ (177) $\displaystyle-\beta\leq
u_{i}(t)^{T}\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)]u_{i}(t)\leq-\alpha,$

we get

(178) $\displaystyle
2-2\beta\lambda_{i}(t)\leq\partial_{t}{\lambda}_{i}(t)\leq
2-2\alpha\lambda_{i}(t),\quad\lambda_{0,\min}\leq\lambda_{i}(0)\leq\lambda_{0,\max}.$

If $\lambda_{i}(t)\geq 1/\alpha$, then $\partial_{t}{\lambda}_{i}(t)\leq 0$;
if $\lambda_{i}(t)\leq 1/\beta$, then $\partial_{t}{\lambda}_{i}(t)\geq 0$. We
conclude that
$\min\\{\lambda_{0,\min},1/\beta\\}\leq\lambda_{i}(t)\leq\max\\{\lambda_{0,\max},1/\alpha\\}$.

######  Proof E.4 (Proof of Proposition 4.10).

The time derivative of the KL divergence is

(179)
$\displaystyle\begin{split}&\phantom{{}={}}\partial_{t}{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}=\int\partial_{t}\rho_{a_{t}}(\theta)\log\frac{\rho_{a_{t}}(\theta)}{\rho_{\rm
post}(\theta)}\mathrm{d}\theta\\\
&=\partial_{t}{m}_{t}^{T}\int\partial_{m_{t}}\mathcal{N}(m_{t},C_{t})\log\frac{\rho_{a_{t}}(\theta)}{\rho_{\rm
post}(\theta)}\mathrm{d}\theta+{\rm
tr}\Bigl{[}\partial_{t}{C}_{t}\int\partial_{C_{t}}\mathcal{N}(m_{t},C_{t})\log\frac{\rho_{a_{t}}(\theta)}{\rho_{\rm
post}(\theta)}\mathrm{d}\theta\Bigr{]}\\\
&=-\partial_{t}{m}_{t}^{T}\int\nabla_{\theta}\rho_{a_{t}}(\theta)\log\frac{\rho_{a_{t}}(\theta)}{\rho_{\rm
post}(\theta)}\mathrm{d}\theta+\frac{1}{2}{\rm
tr}\Bigl{[}\partial_{t}{C}_{t}\int\nabla_{\theta}\nabla_{\theta}\rho_{a_{t}}(\theta)\log\frac{\rho_{a_{t}}(\theta)}{\rho_{\rm
post}(\theta)}\mathrm{d}\theta\Bigr{]},\end{split}$

where in the last equality we use Eq. 145. After an integration by parts, we
can further simplify LABEL:e:dtKL as

(180)
$\displaystyle\partial_{t}{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}=$
$\displaystyle-\partial_{t}m_{t}^{T}\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]$ $\displaystyle{}-\frac{1}{2}{\rm
tr}\Bigl{[}\partial_{t}C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}.$

We first discuss the Gaussian approximate Wasserstein gradient flow. For
logconcave posterior distribution satisfying Eq. 64,
${\mathrm{KL}}[\cdot\|\rho_{\rm post}(\theta)]$ is $\alpha$-convex

(181) $\displaystyle{\mathrm{KL}}[\mu_{t}\|\rho_{\rm
post}(\theta)]\leq(1-t){\mathrm{KL}}[\mu_{0}\|\rho_{\rm
post}(\theta)]+t{\mathrm{KL}}[\mu_{1}\|\rho_{\rm post}(\theta)]-\frac{\alpha
t(1-t)}{2}W^{2}_{2}(\mu_{0},\mu_{1}),$

where $\mu_{t}$ is the geodesic from $\mu_{0}$ to $\mu_{1}$ under the
Wasserstein metric. See [3, Chapter 9.4]. The Gaussian space is geodesically
closed under the Wasserstein metric: if $\mu_{0},\mu_{1}$ are two Gaussian
distributions, every measure on the geodesic $\mu_{t}$ from $\mu_{0}$ to
$\mu_{1}$ is still a Gaussian distribution. It follows from the convexity that
the minimizer $\rho_{a_{\star}}$ of ${\mathrm{KL}}[\rho_{a}\|\rho_{\rm
post}(\theta)]$ is unique.

Employing LABEL:eq:Gaussian-Wasserstein in Eq. 180 leads to

$\displaystyle\partial_{t}{\mathrm{KL}}$
$\displaystyle\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}$ $\displaystyle=$
$\displaystyle-\left\|\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\right\|_{2}^{2}-\frac{1}{2}{\rm tr}\Bigl{[}(2$
$\displaystyle+C_{t}\mathbb{E}_{\rho_{a_{t}}}\bigl{[}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]})(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}$ $\displaystyle=$
$\displaystyle-\left\|\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\right\|_{2}^{2}$ $\displaystyle-{\rm
tr}\Bigl{[}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}.$

It has been proven in [80, Appendix D] that when
$-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm post}(\theta)\succeq\alpha I$,
we have

(182)
$\displaystyle\begin{split}&\left\|\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\right\|_{2}^{2}\\\ &+{\rm
tr}\Big{[}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}\\\ &\geq
2\alpha\left({\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}-{\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\right).\end{split}$

It follows that the gradient flow under the Wasserstein metric converges
exponentially fast,

(183) $\displaystyle{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\leq e^{-2\alpha
t}{\mathrm{KL}}\Bigl{[}\rho_{a_{0}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}+(1-e^{-2\alpha
t}){\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]},$

where $\rho_{a_{0}}\sim\mathcal{N}(m_{0},C_{0})$ is the initial condition and
$\rho_{a_{\star}}\sim\mathcal{N}(m_{\star},C_{\star})$ is the unique global
minimizer of Eq. 9. Next, we discuss the Gaussian approximate gradient flow
Eq. 96. Employing Eq. 96 in Eq. 180 leads to

(184)
$\displaystyle\begin{split}\partial_{t}{\mathrm{KL}}&\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}=-\left\|\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\right\|_{2}^{2}\\\ &-\frac{1}{4}{\rm
tr}\Bigl{[}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}.\end{split}$

From Lemma E.2, we have the upper bound of $C_{t}$:
$C_{t}\preceq\max\\{1/\alpha,\lambda_{0,\max}\\}$. In particular for any
symmetric matrix $A$, we have $AC_{t}A\preceq
A\max\\{1/\alpha,\lambda_{0,\max}\\}A$, and ${\rm
tr}[AC_{t}A]\preceq\max\\{1/\alpha,\lambda_{0,\max}\\}{\rm tr}[A^{2}]$. By
comparing Eq. 184 and LABEL:e:lowb1, we have

(185)
$\displaystyle\begin{split}&\left\|\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\right\|_{2}^{2}+\frac{1}{4}{\rm
tr}\Bigl{[}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])^{2}\Bigr{]}\\\
&\geq\frac{\alpha}{\max\\{1,4/\alpha,4\lambda_{0,\max}\\}}\left({\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}-{\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\right).\end{split}$

It then follows that the gradient flow converges exponentially fast:

(186) $\displaystyle{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\leq
e^{-Kt}{\mathrm{KL}}\Bigl{[}\rho_{a_{0}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}+(1-e^{-Kt}){\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]},$

where $K=2\alpha/\max\\{1,4/\alpha,4\lambda_{0,\max}\\}$.

Next, we discuss the Gaussian approximate Fisher-Rao gradient flow Eq. 82.
Using Eq. 82 in Eq. 180 leads to

(187)
$\displaystyle\begin{split}&\phantom{{}={}}\partial_{t}{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}=-\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]^{T}C_{t}\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\\\ &-\frac{1}{2}{\rm
Tr}\Bigl{[}C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)])C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}.\end{split}$

From Lemma E.2, we have the lower bound of $C_{t}$:
$C_{t}\succeq\min\\{1/\beta,\lambda_{0,\min}\\}$. In particular for any vector
$u$, $u^{T}C_{t}u\geq\min\\{1/\beta,\lambda_{0,\min}\\}\|u\|_{2}^{2}$; for any
positive definite symmetric matrix $A$, we have ${\rm
tr}[C_{t}A]\geq\min\\{1/\beta,\lambda_{0,\min}\\}{\rm tr}A$. By comparing
LABEL:e:ngdflow and LABEL:e:lowb1, we have

(188)
$\displaystyle\begin{split}&\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]^{T}C_{t}\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right]\\\ +&\frac{1}{2}{\rm
tr}\Bigl{[}C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)])C_{t}(C_{t}^{-1}+\mathbb{E}_{\rho_{a_{t}}}\left[\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\right])\Bigr{]}\\\
\geq&\alpha\min\\{1/\beta,\lambda_{0,\min}\\}\left({\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}-{\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\right).\end{split}$

Then, it follows that the Gaussian approximate Fisher-Rao gradient flow
converges exponentially fast:

(189) $\displaystyle{\mathrm{KL}}\Bigl{[}\rho_{a_{t}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}\leq
e^{-Kt}{\mathrm{KL}}\Bigl{[}\rho_{a_{0}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}+(1-e^{-Kt}){\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]},$

where $K=\alpha\min\\{1/\beta,\lambda_{0,\min}\\}$.

To prove Eq. 100, we first prove the following lemma.

######  Lemma E.5.

KL divergence can be used to upper bound Wasserstein distance: for any
Gaussian measure $\rho_{a}$, it holds that

(190) $\displaystyle\alpha
W_{2}^{2}(\rho_{a},\rho_{a_{\star}})\leq{\mathrm{KL}}\Bigl{[}\rho_{a}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]}-{\mathrm{KL}}\Bigl{[}\rho_{a_{\star}}(\theta)\Big{\|}\rho_{\rm
post}(\theta)\Bigr{]},$

where $a_{\star}=(m_{\star},C_{\star})$ and
$\rho_{a_{\star}}\sim\mathcal{N}(m_{\star},C_{\star})$ is the unique global
minimizer of Eq. 9.

We take $\mu_{0}=\rho_{a_{\star}}$ and $\mu_{1}=\rho_{a}$ in Eq. 181, and
$\mu_{t}$ the geodesic flow from $\rho_{a_{\star}}$ to $\rho_{a}$. Let
$f(t)={\mathrm{KL}}[\mu_{t}\|\rho_{\rm
post}(\theta)]-{\mathrm{KL}}[\rho_{a_{\star}}\|\rho_{\rm post}(\theta)]$. Then
Eq. 181 implies that $f(t)$ is convex

(191) $\displaystyle f\bigl{(}(1-t)c_{1}+tc_{2}\bigr{)}$
$\displaystyle\leq(1-t)f(c_{1})+tf(c_{2})-\frac{\alpha
W^{2}_{2}(\mu_{c_{1}},\mu_{c_{2}})}{2}t(1-t)$ (192)
$\displaystyle=(1-t)f(c_{1})+tf(c_{2})-\frac{\alpha(c_{2}-c_{1})^{2}W^{2}_{2}(\rho_{a},\rho_{a_{\star}})}{2}t(1-t),$

for any $0\leq c_{1}\leq c_{2}\leq 1$. It follows that $f(t)$ is convex and by
taking $t=1/2$ and $c_{1},c_{2}\rightarrow c$

(193) $\displaystyle
f^{\prime\prime}(c)=\frac{f(c_{1})+f(c_{2})-2f((c_{1}+c_{2})/2)}{(c_{1}-c_{2})^{2}/4}\geq\alpha
W_{2}^{2}(\rho_{a},\rho_{a_{\star}}).$

Since $\rho_{a_{\star}}$ is the minimizer of
${\mathrm{KL}}[\rho_{a}\|\rho_{\rm post}(\theta)]$, we have that $f(0)=0$ and
$f^{\prime}(0)=0$. By integrating Eq. 193, we get

(194) $\displaystyle{\mathrm{KL}}[\rho_{a}\|\rho_{\rm
post}(\theta)]-{\mathrm{KL}}[\rho_{a_{\star}}\|\rho_{\rm
post}(\theta)]=f(1)\geq\frac{\alpha}{2}W_{2}^{2}(\rho_{a},\rho_{a_{\star}}).$

Equation 100 follows from combining Eq. 194 and Eq. 99,

(195) $\displaystyle
W_{2}^{2}(\rho_{a_{t}},\rho_{a_{\star}})\leq\frac{2}{\alpha}\left({\mathrm{KL}}[\rho_{a_{t}}\|\rho_{\rm
post}(\theta)]-{\mathrm{KL}}[\rho_{a_{\star}}\|\rho_{\rm
post}(\theta)]\right)$ (196)
$\displaystyle\leq\frac{2e^{-Kt}}{\alpha}\left({\mathrm{KL}}[\rho_{a_{0}}\|\rho_{\rm
post}(\theta)]-{\mathrm{KL}}[\rho_{a_{\star}}\|\rho_{\rm
post}(\theta)]\right).$

###  E.4 Proof of Proposition 4.11

Let $\rho_{a_{\star}}\sim\mathcal{N}(m_{\star},C_{\star})$ denote the unique
minimizer of Eq. 2, noting that this is also the unique stationary point of
Gaussian approximate Fisher-Rao gradient flow Eq. 82. It satisfies (See Lemma
D.3 for definition of $a_{\star}$)

(197) $\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}=0\quad\textrm{and}\quad\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}=C_{\star}^{-1}.$

For $N_{\theta}=1$, the linearized Jacobian matrix of the ODE system Eq. 82
around the minimizer $\rho_{a_{\star}}$ is

(198)
$\displaystyle\begin{bmatrix}-1&-\frac{1}{2}\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})\bigr{]}\\\
-\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-
m_{\star})\bigr{]}C_{\star}&-\frac{1}{2}-\frac{1}{2}\mathbb{E}_{\rho_{a_{\star}}}[-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})^{2}]\end{bmatrix}.$

Define

(199) $\displaystyle A_{1}$
$\displaystyle:=\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})\bigr{]},$ $\displaystyle A_{2}$
$\displaystyle:=\mathbb{E}_{\rho_{a_{\star}}}[-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})^{2}]\geq 0.$

Using the Cauchy inequality and Eq. 197, we have

(200) $\displaystyle
A_{2}C_{\star}^{-1}=A_{2}\mathbb{E}_{\rho_{a_{\star}}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}\geq A_{1}^{2}.$

The two eigenvalues of Eq. 198 satisfy

$\displaystyle\lambda_{1}$
$\displaystyle=\frac{-(\frac{3}{2}+\frac{1}{2}A_{2})-\sqrt{(\frac{1}{2}-\frac{1}{2}A_{2})^{2}+2A_{1}^{2}C_{\star}}}{2}\leq-1$
$\displaystyle\lambda_{2}$
$\displaystyle=\frac{-(\frac{3}{2}+\frac{1}{2}A_{2})+\sqrt{(\frac{1}{2}-\frac{1}{2}A_{2})^{2}+2A_{1}^{2}C_{\star}}}{2},$
$\displaystyle=-\frac{1+A_{2}-A_{1}^{2}C_{\star}}{(\frac{3}{2}+\frac{1}{2}A_{2})+\sqrt{(\frac{1}{2}-\frac{1}{2}A_{2})^{2}+2A_{1}^{2}C_{\star}}}\leq-\frac{1}{3+A_{2}}.$

##### Step 1

We will first prove that

$A_{2}\leq
4\log(\frac{\beta}{\alpha})+\frac{2}{\sqrt{\pi}}\sqrt{\log(\frac{\beta}{\alpha})}+4+\frac{4}{\sqrt{\pi}}.$

Without loss of generality, we assume $m_{\star}=0$, which can be achieved
through change of variable. Considering only the right half of the integration
defining $A_{2}$, we have

$\displaystyle\int_{0}^{+\infty}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\theta^{2}\frac{1}{\sqrt{2\pi
C_{\star}}}e^{-\frac{1}{2}\frac{\theta^{2}}{C_{\star}}}\mathrm{d}\theta$
$\displaystyle=$
$\displaystyle\int_{0}^{A}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\theta^{2}\frac{1}{\sqrt{2\pi
C_{\star}}}e^{-\frac{1}{2}\frac{\theta^{2}}{C_{\star}}}\mathrm{d}\theta+\int_{A}^{+\infty}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\theta^{2}\frac{1}{\sqrt{2\pi
C_{\star}}}e^{-\frac{1}{2}\frac{\theta^{2}}{C_{\star}}}\mathrm{d}\theta$
$\displaystyle\leq$
$\displaystyle\frac{A^{2}}{C_{\star}}+\int_{A}^{+\infty}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\theta^{2}\frac{1}{\sqrt{2\pi
C_{\star}}}e^{-\frac{1}{2}\frac{\theta^{2}}{C_{\star}}}\mathrm{d}\theta\quad\textrm{
Using }\lx@cref{creftype~refnum}{eq:stationary}\textrm{ and replace
}\theta\textrm{ as }A$ $\displaystyle\leq$
$\displaystyle\frac{A^{2}}{C_{\star}}+\beta\int_{A}^{+\infty}\frac{\theta^{3}}{A}\frac{1}{\sqrt{2\pi
C_{\star}}}e^{-\frac{1}{2}\frac{\theta^{2}}{C_{\star}}}\mathrm{d}\theta\quad\textrm{
Using }\theta\geq A\textrm{ and }\lx@cref{creftype~refnum}{a:logconcave}$
$\displaystyle=$ $\displaystyle\frac{A^{2}}{C_{\star}}+\frac{2\beta
C_{\star}^{2}}{A\sqrt{2\pi
C_{\star}}}(\frac{A^{2}}{2C_{\star}}+1)e^{-\frac{A^{2}}{2C_{\star}}}$
$\displaystyle\leq$ $\displaystyle
2\frac{A^{2}}{2C_{\star}}+\frac{\beta}{\sqrt{\pi}\alpha}(\frac{A}{\sqrt{2C_{\star}}}+\frac{\sqrt{2C_{\star}}}{A})e^{-\frac{A^{2}}{2C_{\star}}}\qquad\textrm{
Using }C_{\star}\leq\frac{1}{\alpha}$ $\displaystyle\leq$
$\displaystyle(2+\frac{2}{\sqrt{\pi}})(\log(\frac{\beta}{\alpha})+1);$

here we have set
$\frac{A^{2}}{2C_{\star}}=\max\\{\log(\frac{\beta}{\alpha}),1\\}$. Therefore,
the local convergence rate is faster than

(201)
$\displaystyle\lambda_{2}\leq-\frac{1}{(7+\frac{4}{\sqrt{\pi}})\bigl{(}1+\log(\frac{\beta}{\alpha})\bigr{)}}\qquad\textrm{or}-\lambda_{2}\geq\mathcal{O}(1/(1+\log(\frac{\beta}{\alpha}))).$

##### Step 2

Next, we will construct an example to show the bound is sharp. The basic idea
is to construct a sequence of posterior density functions with $\beta$,
$\alpha$ and their ratio $\frac{\beta}{\alpha}$ converging to $+\infty$ and
$-\lambda_{2}$ satisfying Eq. 201.

We consider the following sequence of posterior density functions $\rho_{\rm
post}(\theta)$,

(202) $\displaystyle-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)=\int
H(x)\frac{e^{-\frac{(\theta-x)^{2}}{2\sigma^{2}}}}{\sigma\sqrt{2\pi}}{\rm
d}x\qquad H(x)=\begin{cases}\beta&\gamma-\sigma<x<\gamma+\sigma\\\
\alpha&\textrm{ Otherwise}\end{cases},$

which is parameterized by $\gamma>0$ and $\gamma\rightarrow 0$. We further set
$\sigma=\gamma^{1.5}$ and determine $\alpha$ and $\beta$ based on the
stationery condition (197) at the minimizer $m_{\star}$ and $C_{\star}$, such
that

(203) $\displaystyle m_{\star}=0\quad
C_{\star}=-\frac{\gamma^{2}}{2\log\gamma}-\gamma^{3}\quad\textrm{and}\quad\alpha=\frac{1}{(-\log\gamma)C_{\star}}.$

We have $\alpha$ for any $\gamma$. Since the first stationery point condition
in Eq. 197 can always hold by shifting $\nabla_{\theta}\log\rho_{\rm
post}(\theta)$ by a constant. The convariance condition in Eq. 197 becomes

(204) $\displaystyle{C_{\star}}^{-1}$ $\displaystyle=\int
H(x)\frac{1}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{(x-m_{\star})^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x$
$\displaystyle=\alpha+(\beta-\alpha)\int_{\gamma-\sigma}^{\gamma+\sigma}\frac{1}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{(x-m_{\star})^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x.$

Given any $C_{\star}\leq\frac{1}{\alpha}$, we can adjust $\beta$ to satisfy
Eq. 204.

Next, we will estimate the leading order of $\alpha$, $\beta$, $A_{1}$ and
$A_{2}$ in Eq. 199.

Let denote
$m_{\star\sigma}=\frac{xC_{\star}+m_{\star}\sigma^{2}}{\sigma^{2}+C_{\star}}$
and $C_{\star\sigma}=\frac{\sigma^{2}C_{\star}}{\sigma^{2}+C_{\star}}$, we
have

$\displaystyle-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\mathcal{N}(m_{\star},C_{\star})=\int
H(x)\frac{e^{-\frac{(\theta-
m_{\star\sigma})^{2}}{2C_{\star\sigma}}}}{\sqrt{2\pi
C_{\star\sigma}}}\frac{1}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{(x-m_{\star})^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x.$

Bringing this to Eq. 199 leads to

(205)
$\begin{split}&A_{1}=\mathbb{E}_{\rho_{\star}}\bigl{[}-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})\bigr{]}=\int
H(x)\frac{m_{\star\sigma}-m_{\star}}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{(x-m_{\star})^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x,\\\
&A_{2}=\mathbb{E}_{\rho_{\star}}[-\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)(\theta-m_{\star})^{2}]=\int
H(x)\frac{C_{\star\sigma}+(m_{\star\sigma}-m_{\star})^{2}}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{(x-m_{\star})^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x.\end{split}$

With the definitions in Eq. 203, we have the following estimation about
Gaussian integration

(206)
$\displaystyle\int_{\gamma-\sigma}^{\gamma+\sigma}x^{n}e^{-\frac{x^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x=2\sqrt{\gamma}\gamma^{n+2}(1+\mathcal{O}(\log\gamma\sqrt{\gamma})).$

Bringing the definition of $\sigma$ and $C_{\star}$ and using change-of-
variable with $y=\frac{x}{\gamma}$, Eq. 206 becomes

$\displaystyle\int_{\gamma-\sigma}^{\gamma+\sigma}x^{n}e^{-\frac{x^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x=\gamma^{n+1}\int_{1-\sqrt{\gamma}}^{1+\sqrt{\gamma}}y^{n}e^{y^{2}\log\gamma}{\rm
d}y.$

Bringing the following Taylor expansion leads to Eq. 206

$\displaystyle\gamma^{n+1}\int_{1-\sqrt{\gamma}}^{1+\sqrt{\gamma}}y^{n}e^{y^{2}\log\gamma}{\rm
d}y\geq$ $\displaystyle
2\sqrt{\gamma}\gamma^{n+1}(1-\sqrt{\gamma})^{n}e^{(1+\sqrt{\gamma})^{2}\log\gamma}$
$\displaystyle=$ $\displaystyle
2\sqrt{\gamma}\gamma^{n+2}(1+\mathcal{O}(2\log\gamma\sqrt{\gamma})),$
$\displaystyle\gamma^{n+1}\int_{1-\sqrt{\gamma}}^{1+\sqrt{\gamma}}y^{n}e^{y^{2}\log\gamma}{\rm
d}y\leq$ $\displaystyle
2\sqrt{\gamma}\gamma^{n+1}(1+\sqrt{\gamma})^{n}e^{(1-\sqrt{\gamma})^{2}\log\gamma}$
$\displaystyle=$ $\displaystyle
2\sqrt{\gamma}\gamma^{n+2}(1-\mathcal{O}(2\log\gamma\sqrt{\gamma})).$

Then the estimation for $A_{1}$, $A_{2}$ from Eq. 205 and the convariance
condition in Eq. 197 become

(207) $\displaystyle A_{1}$
$\displaystyle=\frac{(\beta-\alpha)C_{\star}}{\sigma^{2}+C_{\star}}\int_{\gamma-\sigma}^{\gamma+\sigma}x\frac{1}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{x^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x$
$\displaystyle=\frac{(\beta-\alpha)C_{\star}}{\sigma^{2}+C_{\star}}\frac{2\gamma^{2.5}\sqrt{-\log\gamma}}{\sqrt{\pi}}(1+\mathcal{O}(\log\gamma\sqrt{\gamma}))$
$\displaystyle=\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{2.5}(-\log\gamma)^{0.5}+\mathcal{O}\Bigl{(}\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{3}(-\log\gamma)^{1.5}\Bigr{)},$
$\displaystyle A_{2}$ $\displaystyle=\alpha
C_{\star}+(\beta-\alpha)\int_{\gamma-\sigma}^{\gamma+\sigma}\frac{C_{\star\sigma}+m_{\star\sigma}^{2}}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{x^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x$ $\displaystyle=\alpha
C_{\star}+(\beta-\alpha)\frac{2\sqrt{-\log\gamma}\gamma^{3.5}(1+2\gamma\log\gamma)(1+\gamma+2\gamma\log\gamma)}{\sqrt{\pi}}(1+\mathcal{O}(\log\gamma\sqrt{\gamma}))$
$\displaystyle=\alpha
C_{\star}+\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{3.5}(-\log\gamma)^{0.5}+\mathcal{O}\Bigl{(}\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{4}(-\log\gamma)^{1.5}\Bigr{)},$
$\displaystyle{C_{\star}}^{-1}$
$\displaystyle=\alpha+(\beta-\alpha)\int_{\gamma-\sigma}^{\gamma+\sigma}\frac{1}{\sqrt{2\pi(\sigma^{2}+C_{\star})}}e^{-\frac{x^{2}}{2(\sigma^{2}+C_{\star})}}{\rm
d}x$
$\displaystyle=\alpha+(\beta-\alpha)2\gamma^{1.5}\frac{\sqrt{-\log\gamma}}{\sqrt{\pi}}(1+\mathcal{O}(\log\gamma\sqrt{\gamma}))$
$\displaystyle=\alpha+\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{1.5}(-\log\gamma)^{0.5}+\mathcal{O}\Bigl{(}\frac{2(\beta-\alpha)}{\sqrt{\pi}}\gamma^{2}(-\log\gamma)^{1.5}\Bigr{)}.$

Combining Eq. 207 and Eq. 203 leads to

(208)
$\displaystyle\alpha=\mathcal{O}(\frac{1}{\gamma^{2}})\qquad\frac{\beta}{\alpha}=\mathcal{O}(\frac{(-\log\gamma)^{0.5}}{\gamma^{1.5}})\qquad\beta-\alpha=\mathcal{O}(\frac{(-\log\gamma)^{0.5}}{\gamma^{3.5}})$
(209) $\displaystyle A_{1}=\mathcal{O}(\frac{-\log\gamma}{\gamma})\qquad
A_{2}=\mathcal{O}(-\log\gamma)$ (210) $\displaystyle
1-\frac{A_{1}^{2}C_{\star}}{A_{2}}=\frac{A_{2}C_{\star}^{-1}-A_{1}^{2}}{A_{2}C_{\star}^{-1}}=\frac{\mathcal{O}\Bigl{(}(-\log\gamma)/\gamma^{2}\Bigr{)}}{\mathcal{O}\Bigl{(}(-\log\gamma)^{2}/\gamma^{2}\Bigr{)}}=\mathcal{O}(\frac{1}{-\log\gamma}).$

Finally, the large eigenvalue of Eq. 198 becomes

$\displaystyle-\frac{1+A_{2}-A_{1}^{2}C_{\star}}{\frac{3}{2}+\frac{1}{2}A_{2}+\sqrt{(\frac{1}{2}A_{2}-\frac{1}{2})^{2}+2A_{1}^{2}C_{\star}}}$
$\displaystyle\geq-\frac{1+A_{2}-A_{1}^{2}C_{\star}}{1+A_{2}}$
$\displaystyle=-\frac{\frac{1}{A_{2}}+(1-\frac{A_{1}^{2}C_{\star}}{A_{2}})}{1+\frac{1}{A_{2}}}=-\mathcal{O}(\frac{1}{-\log\gamma}).$

Here we use $A_{2}\geq 0$ and $1-\frac{A_{1}^{2}C_{\star}}{A_{2}}\geq 0$.
Equation 208 $\log(\frac{\beta}{\alpha})=\mathcal{O}(-\log\gamma)$ indicates
that for the constructed logconcave density, the local convergence rate is
slower than $-\mathcal{O}(1/\log(\frac{\beta}{\alpha})).$

###  E.5 Proof of Proposition 4.12

Consider the special case, where $\theta\in$ and
$\Phi_{R}(\theta)=\sum_{k=0}^{2K+1}a_{2k}\theta^{2k}$ with $a_{4K+2}>0$.
Assume $\rho_{a}(\theta)\sim\mathcal{N}(m,C)$, then we have

(211) $\displaystyle\mathbb{E}_{\rho_{a}}\bigl{[}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}=$
$\displaystyle-\sum_{k=1}^{2K+1}2ka_{2k}\mathbb{E}_{\rho_{a}}[\theta^{2k-1}]$
$\displaystyle=$
$\displaystyle-\sum_{k=1}^{2K+1}2ka_{2k}\sum_{i=0}^{k-1}\binom{2k-1}{2i+1}m^{2i+1}C^{k-i-1}\frac{(2k-2i-2)!}{2^{k-i-1}(k-i-1)!}$
$\displaystyle=$
$\displaystyle-m\sum_{k=1}^{2K+1}2ka_{2k}\sum_{i=0}^{k-1}\binom{2k-1}{2i+1}m^{2i}C^{k-i-1}\frac{(2k-2i-2)!}{2^{k-i-1}(k-i-1)!}.$

Define

(212) $\displaystyle
f(C)=\sum_{k=1}^{2K+1}2k(2k-1)a_{2k}C^{k-1}\frac{(2k-2)!}{2^{k-1}(k-1)!}.$

We have

(213)
$\displaystyle\mathbb{E}_{\rho_{a}}\bigl{[}\nabla_{\theta}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}_{m=0}=\frac{\partial\mathbb{E}_{\rho_{a}}\bigl{[}\nabla_{\theta}\log\rho_{\rm
post}(\theta)\bigr{]}}{\partial m}|_{m=0}=-f(C).$

We choose $\\{a_{2k}\\}_{k=1}^{2K+1}$ such that

$2k(2k-1)\frac{(2k-2)!}{2^{k-1}(k-1)!}a_{2k}=\binom{2K+1}{k}(-1)^{2K+1-k}$

Then we have

$1-f(C)C=-(C-1)^{2K+1}$

For the gradient flows resulting from our assumption on $\Phi_{R}$, initialize
at $m_{0}=0$ and with $C_{0}$ very close to $1$. For the Gaussian approximate
gradient floww Eq. 96, we have

(214) $\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=0,$
$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}=\frac{1}{2C_{t}}(1-f(C_{t})C_{t})=-\frac{(C_{t}-1)^{2K+1}}{2C_{t}}.$

For the Gaussian approximate Fisher-Rao gradient flow Eq. 82, we have

(215) $\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=0,$
$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}=C_{t}(1-f(C_{t})C_{t})=-C_{t}(C_{t}-1)^{2K+1}.$

For the Gaussian approximate Wasserstein gradient flow LABEL:eq:Gaussian-
Wasserstein, we have

(216) $\displaystyle\frac{\mathrm{d}m_{t}}{\mathrm{d}t}=0,$
$\displaystyle\frac{\mathrm{d}C_{t}}{\mathrm{d}t}=2(1-f(C_{t})C_{t})=-2(C_{t}-1)^{2K+1}.$

In all cases the convergence rate to $C_{t}=1$ is
$\mathcal{O}(t^{-\frac{1}{2K}})$.

##  Appendix F Numerical Integration

In this section, we discuss how we compute the reference values of
$\mathbb{E}\theta$, $\mathrm{Cov}\theta$, and
$\mathbb{E}(\cos(\omega^{T}\theta+b)$ for logconcave posterior and Rosenbrock
posterior in Section 5. For any Gaussian distribution, we have

$\displaystyle\int\cos(\omega^{T}\theta+b)\mathcal{N}(m,C)\mathrm{d}\theta=\exp\\{-\frac{1}{2}\omega^{T}C\omega\\}\cos(\omega^{T}m+b).$

We can rewrite the logconcave function,

$\displaystyle\Phi_{R}(\theta)=\frac{1}{2}\frac{({\theta^{(1)}}-{\theta^{(2)}}/\sqrt{\lambda})^{2}}{10/\lambda}+\frac{{\theta^{(2)}}^{4}}{20}.$

Therefore,

$\displaystyle\int
e^{-\Phi_{R}(\theta)}\mathrm{d}{\theta^{(1)}}=\sqrt{20\pi/\lambda}e^{-{\theta^{(2)}}^{4}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}{\theta^{(1)}}\mathrm{d}{\theta^{(1)}}=\sqrt{20\pi/\lambda}\frac{{\theta^{(2)}}}{\sqrt{\lambda}}e^{-{\theta^{(2)}}^{4}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}{\theta^{(1)}}^{2}\mathrm{d}{\theta^{(1)}}=\sqrt{20\pi/\lambda}(\frac{{\theta^{(2)}}^{2}}{\lambda}+\frac{10}{\lambda})e^{-{\theta^{(2)}}^{4}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}\cos({\omega^{(1)}}{\theta^{(1)}}+{\omega^{(2)}}{\theta^{(2)}}+b)\mathrm{d}{\theta^{(1)}}$
$\displaystyle\quad\quad\quad=\sqrt{20\pi/\lambda}e^{-\frac{5}{\lambda}{\omega^{(1)}}^{2}}\cos({\omega^{(1)}}{\theta^{(2)}}/\sqrt{\lambda}+{\omega^{(2)}}{\theta^{(2)}}+b)e^{-{\theta^{(2)}}^{4}/20}.$

We can rewrite the Rosenbrock function as

$\Phi_{R}(\theta)=\frac{1}{2}\frac{({\theta^{(2)}}-{\theta^{(1)}}^{2})^{2}}{10/\lambda}+\frac{(1-{\theta^{(1)}})^{2}}{20}.$

Therefore,

$\displaystyle\int
e^{-\Phi_{R}(\theta)}\mathrm{d}{\theta^{(2)}}=\sqrt{20\pi/\lambda}e^{-(1-{\theta^{(1)}})^{2}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}{\theta^{(2)}}\mathrm{d}{\theta^{(2)}}=\sqrt{20\pi/\lambda}{\theta^{(1)}}^{2}e^{-(1-{\theta^{(1)}})^{2}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}{\theta^{(2)}}^{2}\mathrm{d}{\theta^{(2)}}=\sqrt{20\pi/\lambda}({\theta^{(1)}}^{4}+\frac{10}{\lambda})e^{-(1-{\theta^{(1)}})^{2}/20},$
$\displaystyle\int
e^{-\Phi_{R}(\theta)}\cos({\omega^{(2)}}{\theta^{(2)}}+{\omega^{(1)}}{\theta^{(1)}}+b)\mathrm{d}{\theta^{(2)}}$
$\displaystyle\quad\quad\quad=\sqrt{20\pi/\lambda}e^{-\frac{5}{\lambda}{\omega^{(2)}}^{2}}\cos({\omega^{(2)}}{\theta^{(1)}}^{2}+{\omega^{(1)}}{\theta^{(1)}}+b)e^{-(1-{\theta^{(1)}})^{2}/20}.$

We have

$\int\int
e^{-\Phi_{R}(\theta)}\mathrm{d}{\theta^{(1)}}\mathrm{d}{\theta^{(2)}}=\frac{20\pi}{\sqrt{\lambda}}\qquad\mathbb{E}[\theta]=\begin{bmatrix}1\\\
11\end{bmatrix}\qquad\mathrm{Cov}[\theta]=\begin{bmatrix}10&20\\\
20&\frac{10}{\lambda}+240\end{bmatrix}.$

Other 2D integrations can be addressed by first performing 1D integration with
respect to one variable analytically, and then the second 1D integrations are
computed numerically with $10^{7}$ uniform points.