Super-resolution structured illumination microscopy: past, present and future

Kirti Prakash1,2,*, , Benedict Diederich3,4, Stefanie Reichelt5, Rainer
Heintzmann3,4,6,*, Lothar Schermelleh7,*

1 National Physical Laboratory, TW11 0LW Teddington, UK
2 Department of Paediatrics, Wellcome–MRC Cambridge Stem Cell Institute,
University of Cambridge, Cambridge, UK
3 Leibniz Institute of Photonic Technology, Albert-Einstein-Straße 9, 07745
Jena, Germany
4 Institute of Physical Chemistry and Abbe Center of Photonics, Helmholtzweg
4, Friedrich-Schiller-University, Jena, Germany
5 CRUK Cambridge Research Institute, Robinson Way, Cambridge CB2 0RE, UK
6 Faculty of Physics and Astronomy, Friedrich-Schiller-University, Jena,
Germany
7 Micron Advanced Bioimaging Unit, Department of Biochemistry, University of
Oxford, Oxford OX1 3QU, UK
kirtiprakash25, beniroquai, StefanieReiche6, HeintzmannLab, LSchermelleh

*<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
## Abstract

Structured illumination microscopy (SIM) has emerged as an essential technique
for 3D and live-cell super-resolution imaging. However, to date, there has not
been a dedicated workshop or journal issue covering the various aspects of
SIM, from bespoke hardware and software development and the use of commercial
instruments to biological applications. This special issue aims to recap
recent developments as well as outline future trends. In addition to SIM, we
cover related topics such as complementary super-resolution microscopy
techniques, computational imaging, visualisation and image processing methods.

## Introduction

Fluorescence light microscopy is a core technique in life sciences that has
contributed to countless major discoveries. However, the diffraction limit,
first described by Ernst Abbe in 1873 [1], has restricted the optical
resolution to about half the wavelength of the light used, i.e. 200-300 nm in
the lateral directions (x and y), and to about the wavelength of light, i.e.
500-800 nm along the optical axis (z). In the past two decades, several super-
resolution microscopy (SRM) approaches have been developed that allow to
overcome this barrier and push the spatial resolution to the 10-100 nm range,
thus closing the gap to electron microscopy [71, 25, 70]. The SRM techniques
can be further divided into single-molecule localisation microscopy (SMLM)
[43], that includes techniques like photoactivatable localisation microscopy
(PALM) [7, 29] and stochastic optical reconstruction microscopy (STORM) [67],
stimulated emission depletion (STED) [58, 27, 3] microscopy and structured
illumination microscopy (SIM) [23, 20]. In recognition of these breakthrough
developments Eric Betzig, Stefan Hell and William E. Moerner were awarded the
Nobel prize for Chemistry in 2014 [6].

[frametitle=Frequently asked questions in super-resolution structured
illumination microscopy field] The following questions have not always been
agreed upon in the super-resolution/SIM field, and with this special issue, we
hope to address some of these:

  1. 1.

What is super-resolution microscopy and should diffraction-limited linear SIM
be classified as ‘super-resolution’?

  2. 2.

Should High-NA TIRF-SIM, which can achieve lateral resolution down to 84 nm,
be considered as diffraction limited?

  3. 3.

Can non-linear SIM become broadly applicable and live-cell compatible?

  4. 4.

Do you need ‘switching’ of states for non-linear super-resolution?

  5. 5.

How can information about single-molecule detection be best combined with the
knowledge of the illumination structure?

  6. 6.

Do high quality SIM images require reconstruction in Fourier space?

  7. 7.

Can SIM be used for deep tissue imaging?

  8. 8.

How can the fundamental limitation of SIM, i.e. generating sufficient stripe
contrast in densely labelled and/or extended biological structures due to out-
of-focus light, be addressed?

  9. 9.

Should image scanning microscopy be considered a form of SIM and what forms of
structured illumination could be used other than stripes?

  10. 10.

Can SIM be used to improve the resolution of (Rayleigh scattering)
transmission microscopy?

  11. 11.

How does sparse illumination compare to dense illumination in linear and non-
linear SIM?

  12. 12.

Can we generate “true” super-resolution images from simple instruments
enhanced with machine-learning-based algorithms?

  13. 13.

Can research-grade super-resolution (SIM) microscopes be built cost-
efficiently?

Recent SRM developments have focussed on combining localisation-based
microscopy with modulated illumination to push the resolution (precision)
towards a few nanometers as in MINFLUX and SIMFLUX [5, 13, 64, 62]. However,
an increase in resolution requires an increase in local light dosage, which
increases photobleaching/phototoxicity (SIM, STED, SMLM) or requires the need
to introduce spatial and temporal sparsity (MINFLUX) making imaging comparably
slow. Moreover, most SRM techniques mentioned above rely primarily on
chemically fixed (i.e. dead) cells, are restricted in the imaged sample volume
(2D, single plane, small field-of-views) or are restricted to imaging sparse
single entities such as vesicles or single molecules.

Super-resolution linear SIM is a notable exception [24, 59]. By making use of
frequency mixing when exciting samples with a patterned illumination followed
by computational unmixing and reconstruction, SIM achieves a 2-fold resolution
increase over conventional diffraction-limited fluorescence microscopy in 2D
or 3D (Figure 1). While the numerical resolution improvement is moderate
compared to other SRM techniques, it pushes SIM into an application sweet spot
with most macromolecular structures and their dynamics falling in the size
range of 100-300 nm [69, 77, 73]. By not asking for the highest spatial
resolution, SIM is less demanding in terms of photon budget and therefore more
compatible with live-cell imaging [39, 30, 72, 53, 31, 81]. In addition, it
offers volumetric imaging with relatively large field-of-views and comparably
high temporal resolution which makes it suitable for high-content and live-
cell imaging [84, 83, 50]. Taken together, the uniquely balanced combination
of properties have made SIM remarkably successful in a wide range of
biological applications and promoting new discoveries [17, 56, 34, 10, 48, 16,
51, 55, 32, 76, 47, 19, 41, 57, 77, 79, 49].

## Recent and future developments in SIM

SIM has been available in commercial instruments for more than 10 years and
has been successfully established in many labs and core facilities around the
world. However, wider dissemination of SIM has been curtailed by the
complexity of instruments and its proneness to reconstruction artefacts when
sample properties, system calibration and parameter settings are not carefully
matched [15]. Thus, to lower the activation energy for research labs to
venture into super-resolution SIM, many recent developments aim at further
performance/resolution enhancement and ‘democratising’ SIM by

  * •

using non-linear SIM approaches [26, 21, 63, 42] or by combinations with
single-molecule imaging [5, 65, 18, 13, 37]

  * •

exploiting correlative and combinatorial fluorescence imaging approaches with
multifocal [22, 2], 2-photon [33, 75, 45] and light-sheet microscopy [38, 11]

  * •

implementing the technique into smaller and more cost-efficient setups [28,
80, 40]

  * •

making the method more robust against artefacts using alternative illumination
schemes [68, 4, 52] and/or intelligent data processing [82, 8]

  * •

increasing its application range e.g. by the implementation of adaptive optics
[14, 9, 35], cryo-imaging [60, 79]

Important advancements are being made to improve and simplify the
instrumentation. For instance, the newly designed optical setups using an
open-sources toolbox [80] or waveguide-based photonic chips [82] or open-
source software development [54] may reduce the cost of SR imaging enabling
SIM on smaller and more affordable devices. Moreover, next-generation sCMOS
cameras (for example, the new 500 fps sCMOS by Photometrics) offer increased
frame rates and higher sensitivity with reduced noise levels to enable faster
high-quality SIM imaging for observing cellular dynamics [12, 46, 74]. For a
maximal imaging speed, the rolling shutter schemes can be combined with SIM
data acquisition [78].
In parallel, the rapidly evolving field of computational microscopy is paving
the way to circumvent the resolution limit without the necessity of
specialised hardware. Such computational techniques not only reduce the
complexity of the instrumentation but can also potentially reduce aberration
artefacts [82, 8, 61, 80, 40, 66]. Adaptive optics (AO) works well to correct
sample aberrations. However, it does not solve the problem of reduced
illumination pattern contrast in the presence of significant out of focus
background. One solution here is offered by combining SIM with light sheet
illumination, 2-photon, or photoswitching methods.
Recent advancements in intelligent microscopy and machine learning (ML)
assisted denoising are now being used for improving SIM reconstructions. For
example, using deep-learning models, the apparent axial resolution of SIM has
been improved by a factor of two [36, 44, 8]. By combining physical data
acquisition with simple image processing algorithms, one recover previously
inaccessible information. At the same time, this brings up new problems of
validation of the results from ML methods. New standards have to be developed,
where computational scientists and mathematicians can help to bring together
state-of-the-art algorithms, such as GPU-parallelization for faster data
processing or machine-learning-based data interpretation.

## The wider-reaching social implications

SRM has evolved into a highly interdisciplinary field requiring experts from
physics, engineering, chemistry, biology and computer sciences to drive
innovations for increasing spatial and temporal resolution and ultimately
biological applicability. The global advances in super-resolution microscopy
need to be synchronized to inspire researchers in the field of microscopy to
combine their methods with techniques from different areas.

SRM and SIM, in particular, have become important tools of basic science
discovery. However, there are still considerable activation barriers for the
researcher to use SRM, as instruments are not yet ‘turn-key’, and using SIM
(or any other SRM method) to its full potential requires considerable
knowledge. Furthermore, the use of SRM to study human pathologies and for
medical diagnosis is still in its infancy, but will foreseeably play a
considerable role in the future. New SIM modalities, implementation of AO,
machine-learning, and the intelligent algorithms will help to democratise SIM
and increase it’s usage in basic research and biomedical application.

Super-resolution microscopy is very often an expensive undertaking, thus
limiting its broader application. In this meeting, we plan to kick-off the
idea of an open-source “openSIM” system inspired by its derivatives in the
light-sheet community. Open discussions where ideas and requirements for a
prototype are gathered are a first and significant step to make science not
only affordable but available. This special issue shows current progress in
SIM development and application.

## Acknowledgements

The special issue is part of Theo Murphy international scientific meeting
organised by the Royal Society called SIMposium: recent advancements in
structured illumination microscopy. We thank Mike Shaw for comments on the
manuscript. A Twitter () discussion on this topic which can be followed here
or with hashtag SIMposium.

## References

  * 1. E. Abbe.  Beitrage zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung.  Archiv für Mikroskopische Anatomie, 9:413–420, 1873.
  * 2. S. Abrahamsson, H. Blom, A. Agostinho, D. C. Jans, A. Jost, M. Müller, L. Nilsson, K. Bernhem, T. J. Lambert, R. Heintzmann, et al.  Multifocus structured illumination microscopy for fast volumetric super-resolution imaging.  Biomedical optics express, 8(9):4135–4140, 2017.
  * 3. S. C. Baer.  Method and apparatus for improving resolution in scanned optical system, Feb. 2 1999.  US Patent 5,866,911.
  * 4. G. Ball, J. Demmerle, R. Kaufmann, I. Davis, I. M. Dobbie, and L. Schermelleh.  Simcheck: a toolbox for successful super-resolution structured illumination microscopy.  Scientific reports, 5(1):1–12, 2015.
  * 5. F. Balzarotti, Y. Eilers, K. C. Gwosch, A. H. Gynnå, V. Westphal, F. D. Stefani, J. Elf, and S. W. Hell.  Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes.  Science, 355(6325):606–612, 2017.
  * 6. E. Betzig, S. W. Hell, and W. E. Moerner.  The nobel prize in chemistry 2014.  Nobel Media AB, 2014.
  * 7. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess.  Imaging intracellular fluorescent proteins at nanometer resolution.  Science, 313(5793):1642–1645, 2006.
  * 8. M. Boland, E. A. Cohen, S. Flaxman, and M. A. Neil.  Improving axial resolution in sim using deep learning.  arXiv preprint arXiv:2009.02264, 2020.
  * 9. M. Booth, D. Andrade, D. Burke, B. Patton, and M. Zurauskas.  Aberrations and adaptive optics in super-resolution microscopy.  Microscopy, 64(4):251–261, 2015.
  * 10. A. C. Brown, S. Oddos, I. M. Dobbie, J.-M. Alakoskela, R. M. Parton, P. Eissmann, M. A. Neil, C. Dunsby, P. M. French, I. Davis, et al.  Remodelling of cortical actin where lytic granules dock at natural killer cell immune synapses revealed by super-resolution microscopy.  PLoS Biol, 9(9):e1001152, 2011.
  * 11. B.-J. Chang, V. D. P. Meza, and E. H. Stelzer.  csilsfm combines light-sheet fluorescence microscopy and coherent structured illumination for a lateral resolution below 100 nm.  Proceedings of the National Academy of Sciences, 114(19):4869–4874, 2017.
  * 12. B.-C. Chen, W. R. Legant, K. Wang, L. Shao, D. E. Milkie, M. W. Davidson, C. Janetopoulos, X. S. Wu, J. A. Hammer, Z. Liu, et al.  Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution.  Science, 346(6208), 2014.
  * 13. J. Cnossen, T. Hinsdale, R. Ø. Thorsen, M. Siemons, F. Schueder, R. Jungmann, C. S. Smith, B. Rieger, and S. Stallinga.  Localization microscopy at doubled precision with patterned illumination.  Nature methods, 17(1):59–63, 2020.
  * 14. D. Débarre, E. J. Botcherby, M. J. Booth, and T. Wilson.  Adaptive optics for structured illumination microscopy.  Optics express, 16(13):9290–9305, 2008.
  * 15. J. Demmerle, C. Innocent, A. J. North, G. Ball, M. Müller, E. Miron, A. Matsuda, I. M. Dobbie, Y. Markaki, and L. Schermelleh.  Strategic and practical guidelines for successful structured illumination microscopy.  Nature protocols, 12(5):988–1010, 2017.
  * 16. P. Eswaramoorthy, P. W. Winter, P. Wawrzusin, A. G. York, H. Shroff, and K. S. Ramamurthi.  Asymmetric division and differential gene expression during a bacterial developmental program requires diviva.  PLoS Genet, 10(8):e1004526, 2014.
  * 17. R. D. Gray, D. Albrecht, C. Beerli, M. Huttunen, G. H. Cohen, I. J. White, J. J. Burden, R. Henriques, and J. Mercer.  Nanoscale polarization of the entry fusion complex of vaccinia virus drives efficient fusion.  Nature microbiology, 4(10):1636–1644, 2019.
  * 18. L. Gu, Y. Li, S. Zhang, Y. Xue, W. Li, D. Li, T. Xu, and W. Ji.  Molecular resolution imaging by repetitive optical selective exposure.  Nature methods, 16(11):1114–1118, 2019.
  * 19. Y. Guo, D. Li, S. Zhang, Y. Yang, J.-J. Liu, X. Wang, C. Liu, D. E. Milkie, R. P. Moore, U. S. Tulu, et al.  Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales.  Cell, 175(5):1430–1442, 2018.
  * 20. M. G. Gustafsson.  Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy.  Journal of microscopy, 198(2):82–87, 2000.
  * 21. M. G. Gustafsson.  Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution.  Proceedings of the National Academy of Sciences, 102(37):13081–13086, 2005.
  * 22. B. Hajj, J. Wisniewski, M. El Beheiry, J. Chen, A. Revyakin, C. Wu, and M. Dahan.  Whole-cell, multicolor superresolution imaging using volumetric multifocus microscopy.  Proceedings of the National Academy of Sciences, 111(49):17480–17485, 2014.
  * 23. R. Heintzmann and C. G. Cremer.  Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating.  In Optical Biopsies and Microscopic Techniques III, volume 3568, pages 185–196. International Society for Optics and Photonics, 1999.
  * 24. R. Heintzmann and M. G. Gustafsson.  Subdiffraction resolution in continuous samples.  Nature Photonics, 3(7):362–364, 2009.
  * 25. R. Heintzmann and T. Huser.  Super-resolution structured illumination microscopy.  Chemical reviews, 117(23):13890–13908, 2017.
  * 26. R. Heintzmann, T. M. Jovin, and C. Cremer.  Saturated patterned excitation microscopy—a concept for optical resolution improvement.  JOSA A, 19(8):1599–1609, 2002.
  * 27. S. W. Hell and J. Wichmann.  Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy.  Optics letters, 19(11):780–782, 1994.
  * 28. Ø. I. Helle, F. T. Dullo, M. Lahrberg, J.-C. Tinguely, O. G. Hellesø, and B. S. Ahluwalia.  Structured illumination microscopy using a photonic chip.  Nature Photonics, 14(7):431–438, 2020.
  * 29. S. T. Hess, T. P. Girirajan, and M. D. Mason.  Ultra-high resolution imaging by fluorescence photoactivation localization microscopy.  Biophysical journal, 91(11):4258, 2006.
  * 30. L. M. Hirvonen, K. Wicker, O. Mandula, and R. Heintzmann.  Structured illumination microscopy of a living cell.  European Biophysics Journal, 38(6):807–812, 2009.
  * 31. X. Huang, J. Fan, L. Li, H. Liu, R. Wu, Y. Wu, L. Wei, H. Mao, A. Lal, P. Xi, et al.  Fast, long-term, super-resolution imaging with hessian structured illumination microscopy.  Nature biotechnology, 36(5):451–459, 2018.
  * 32. B. Hübner, M. Lomiento, F. Mammoli, D. Illner, Y. Markaki, S. Ferrari, M. Cremer, and T. Cremer.  Remodeling of nuclear landscapes during human myelopoietic cell differentiation maintains co-aligned active and inactive nuclear compartments.  Epigenetics & chromatin, 8(1):1–21, 2015.
  * 33. M. Ingaramo, A. G. York, P. Wawrzusin, O. Milberg, A. Hong, R. Weigert, H. Shroff, and G. H. Patterson.  Two-photon excitation improves multifocal structured illumination microscopy in thick scattering tissue.  Proceedings of the National Academy of Sciences, 111(14):5254–5259, 2014.
  * 34. G. Jacquemet, A. Stubb, R. Saup, M. Miihkinen, E. Kremneva, H. Hamidi, and J. Ivaska.  Filopodome mapping identifies p130cas as a mechanosensitive regulator of filopodia stability.  Current Biology, 29(2):202–216, 2019.
  * 35. N. Ji.  Adaptive optical fluorescence microscopy.  Nature methods, 14(4):374–380, 2017.
  * 36. L. Jin, B. Liu, F. Zhao, S. Hahn, B. Dong, R. Song, T. C. Elston, Y. Xu, and K. M. Hahn.  Deep learning enables structured illumination microscopy with low light levels and enhanced speed.  Nature communications, 11(1):1–7, 2020.
  * 37. P. Jouchet, C. Cabriel, N. Bourg, M. Bardou, C. Poüs, E. Fort, and S. Lévêque-Fort.  Nanometric axial localization of single fluorescent molecules with modulated excitation.  BioRxiv, page 865865, 2019.
  * 38. P. J. Keller, A. D. Schmidt, A. Santella, K. Khairy, Z. Bao, J. Wittbrodt, and E. H. Stelzer.  Fast, high-contrast imaging of animal development with scanned light sheet–based structured-illumination microscopy.  Nature methods, 7(8):637–642, 2010.
  * 39. P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, and M. G. Gustafsson.  Super-resolution video microscopy of live cells by structured illumination.  Nature methods, 6(5):339–342, 2009.
  * 40. M. Lachetta, H. Sandmeyer, A. Sandmeyer, J. S. am Esch, T. Huser, and M. Müller.  Simulating digital micromirror devices for patterning coherent excitation light in structured illumination microscopy.  bioRxiv, 2020.
  * 41. C. Lesterlin, G. Ball, L. Schermelleh, and D. J. Sherratt.  Reca bundles mediate homology pairing between distant sisters during dna break repair.  Nature, 506(7487):249–253, 2014.
  * 42. D. Li, L. Shao, B.-C. Chen, X. Zhang, M. Zhang, B. Moses, D. E. Milkie, J. R. Beach, J. A. Hammer, M. Pasham, et al.  Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics.  Science, 349(6251), 2015.
  * 43. K. A. Lidke, B. Rieger, T. M. Jovin, and R. Heintzmann.  Superresolution by localization of quantum dots using blinking statistics.  Optics express, 13(18):7052–7062, 2005.
  * 44. C. Ling, C. Zhang, M. Wang, F. Meng, L. Du, and X. Yuan.  Fast structured illumination microscopy via deep learning.  Photonics Research, 8(8):1350–1359, 2020.
  * 45. F. Liu, Q. Li, S. Jiang, L. Zhou, J. Zhang, and H. Zhang.  Two-photon structured illumination microscopy imaging using fourier ptychography scheme.  Optics Communications, page 126872, 2021.
  * 46. T.-L. Liu, S. Upadhyayula, D. E. Milkie, V. Singh, K. Wang, I. A. Swinburne, K. R. Mosaliganti, Z. M. Collins, T. W. Hiscock, J. Shea, et al.  Observing the cell in its native state: Imaging subcellular dynamics in multicellular organisms.  Science, 360(6386), 2018.
  * 47. U. Manor, S. Bartholomew, G. Golani, E. Christenson, M. Kozlov, H. Higgs, J. Spudich, and J. Lippincott-Schwartz.  A mitochondria-anchored isoform of the actin-nucleating spire protein regulates mitochondrial division.  Elife, 4:e08828, 2015.
  * 48. K. McArthur, L. W. Whitehead, J. M. Heddleston, L. Li, B. S. Padman, V. Oorschot, N. D. Geoghegan, S. Chappaz, S. Davidson, H. San Chin, et al.  Bak/bax macropores facilitate mitochondrial herniation and mtdna efflux during apoptosis.  Science, 359(6378), 2018.
  * 49. V. Mennella, B. Keszthelyi, K. McDonald, B. Chhun, F. Kan, G. C. Rogers, B. Huang, and D. Agard.  Subdiffraction-resolution fluorescence microscopy reveals a domain of the centrosome critical for pericentriolar material organization.  Nature cell biology, 14(11):1159–1168, 2012.
  * 50. J. Mertz.  Optical sectioning microscopy with planar or structured illumination.  Nature methods, 8(10):811, 2011.
  * 51. E. Miron, R. Oldenkamp, J. M. Brown, D. M. Pinto, C. S. Xu, A. R. Faria, H. A. Shaban, J. D. Rhodes, C. Innocent, S. de Ornellas, et al.  Chromatin arranges in chains of mesoscale domains with nanoscale functional topography independent of cohesin.  Science advances, 6(39):eaba8811, 2020.
  * 52. Y. Mo, F. Feng, H. Mao, J. Fan, and L. Chen.  Structured illumination microscopy artifacts caused by illumination scattering.  bioRxiv, pages 2021–01, 2021.
  * 53. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac.  Structured illumination microscopy using unknown speckle patterns.  Nature Photonics, 6(5):312–315, 2012.
  * 54. M. Müller, V. Mönkemöller, S. Hennig, W. Hübner, and T. Huser.  Open-source image reconstruction of super-resolution structured illumination microscopy data in imagej.  Nature communications, 7(1):1–6, 2016.
  * 55. S. Murugesan, J. Hong, J. Yi, D. Li, J. R. Beach, L. Shao, J. Meinhardt, G. Madison, X. Wu, E. Betzig, et al.  Formin-generated actomyosin arcs propel t cell receptor microcluster movement at the immune synapse.  Journal of Cell Biology, 215(3):383–399, 2016.
  * 56. M. Nozumi, F. Nakatsu, K. Katoh, and M. Igarashi.  Coordinated movement of vesicles and actin bundles during nerve growth revealed by superresolution microscopy.  Cell reports, 18(9):2203–2216, 2017.
  * 57. F. Ochs, G. Karemore, E. Miron, J. Brown, H. Sedlackova, M.-B. Rask, M. Lampe, V. Buckle, L. Schermelleh, J. Lukas, et al.  Stabilization of chromatin topology safeguards genome integrity.  Nature, 574(7779):571–574, 2019.
  * 58. V. Okhonin.  A method of examination of sample microstructure. SU 1374922 A1. Priority 10.04.1986. DOI: 10.13140/2.1.2588.1922.  1991\.
  * 59. K. O’Holleran and M. Shaw.  Optimized approaches for optical sectioning and resolution enhancement in 2d structured illumination microscopy.  Biomedical optics express, 5(8):2580–2590, 2014.
  * 60. M. A. Phillips, M. Harkiolaki, D. M. S. Pinto, R. M. Parton, A. Palanca, M. Garcia-Moreno, I. Kounatidis, J. W. Sedat, D. I. Stuart, A. Castello, et al.  Cryosim: super-resolution 3d structured illumination cryogenic fluorescence microscopy for correlated ultrastructural imaging.  Optica, 7(7):802–812, 2020.
  * 61. K. Prakash.  Laser-free super-resolution microscopy.  bioRxiv, 2020.
  * 62. K. Prakash.  At the molecular resolution with minflux?  2021\.
  * 63. E. H. Rego, L. Shao, J. J. Macklin, L. Winoto, G. A. Johansson, N. Kamps-Hughes, M. W. Davidson, and M. G. Gustafsson.  Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution.  Proceedings of the National Academy of Sciences, 109(3):E135–E143, 2012.
  * 64. L. Reymond, T. Huser, V. Ruprecht, and S. Wieser.  Modulation-enhanced localization microscopy.  Journal of Physics: Photonics, 2(4):041001, 2020.
  * 65. L. Reymond, J. Ziegler, C. Knapp, F.-C. Wang, T. Huser, V. Ruprecht, and S. Wieser.  Simple: Structured illumination based point localization estimator with enhanced precision.  Optics express, 27(17):24578–24590, 2019.
  * 66. C. T. Russell and M. Shaw.  mmsim: An open toolbox for accessible structured illumination microscopy.  bioRxiv, 2021.
  * 67. M. J. Rust, M. Bates, and X. Zhuang.  Stochastic optical reconstruction microscopy (storm) provides sub-diffraction-limit image resolution.  Nature methods, 3(10):793, 2006.
  * 68. L. Schaefer, D. Schuster, and J. Schaffer.  Structured illumination microscopy: artefact analysis and reduction utilizing a parameter optimization approach.  Journal of microscopy, 216(2):165–174, 2004.
  * 69. L. Schermelleh, P. M. Carlton, S. Haase, L. Shao, L. Winoto, P. Kner, B. Burke, M. C. Cardoso, D. A. Agard, M. G. Gustafsson, et al.  Subdiffraction multicolor imaging of the nuclear periphery with 3d structured illumination microscopy.  Science, 320(5881):1332–1336, 2008.
  * 70. L. Schermelleh, A. Ferrand, T. Huser, C. Eggeling, M. Sauer, O. Biehlmaier, and G. P. Drummen.  Super-resolution microscopy demystified.  Nature cell biology, 21(1):72–84, 2019.
  * 71. L. Schermelleh, R. Heintzmann, and H. Leonhardt.  A guide to super-resolution fluorescence microscopy.  Journal of Cell Biology, 190(2):165–175, 2010.
  * 72. L. Shao, P. Kner, E. H. Rego, and M. G. Gustafsson.  Super-resolution 3d microscopy of live whole cells using structured illumination.  Nature methods, 8(12):1044–1046, 2011.
  * 73. M. Shaw, A. Bella, and M. G. Ryadnov.  Creim: Coffee ring effect imaging model for monitoring protein self-assembly in situ.  The journal of physical chemistry letters, 8(19):4846–4851, 2017\.
  * 74. M. Shaw, L. Zajiczek, and K. O’Holleran.  High speed structured illumination microscopy in optically thick samples.  Methods, 88:11–19, 2015.
  * 75. C. J. Sheppard, M. Castello, G. Tortarolo, G. Vicidomini, and A. Diaspro.  Image formation in image scanning microscopy, including the case of two-photon excitation.  JOSA A, 34(8):1339–1350, 2017.
  * 76. J.-H. Sir, A. R. Barr, A. K. Nicholas, O. P. Carvalho, M. Khurshid, A. Sossick, S. Reichelt, C. D’Santos, C. G. Woods, and F. Gergely.  A primary microcephaly protein complex forms a ring around parental centrioles.  Nature genetics, 43(11):1147–1153, 2011.
  * 77. D. Smeets, Y. Markaki, V. J. Schmid, F. Kraus, A. Tattermusch, A. Cerase, M. Sterr, S. Fiedler, J. Demmerle, J. Popken, et al.  Three-dimensional super-resolution microscopy of the inactive x chromosome territory reveals a collapse of its active nuclear compartment harboring distinct xist rna foci.  Epigenetics & chromatin, 7(1):1–27, 2014.
  * 78. L. Song, H.-W. Lu-Walther, R. Förster, A. Jost, M. Kielhorn, J. Zhou, and R. Heintzmann.  Fast structured illumination microscopy using rolling shutter cameras.  Measurement Science and Technology, 27(5):055401, 2016.
  * 79. K. F. Sonnen, L. Schermelleh, H. Leonhardt, and E. A. Nigg.  3d-structured illumination microscopy provides novel insight into architecture of human centrosomes.  Biology open, 1(10):965–976, 2012.
  * 80. H. Wang, R. Lachmann, B. Marsikova, R. Heinzmann, and B. Diederich.  Ucsim2: 2d structured illumination microscopy using uc2.  bioRxiv, pages 2021–01, 2021.
  * 81. Y. Wu and H. Shroff.  Faster, sharper, and deeper: structured illumination microscopy for biological imaging.  Nature methods, 15(12):1011–1019, 2018.
  * 82. L.-H. Yeh, S. Chowdhury, and L. Waller.  Computational structured illumination for high-content fluorescence and phase microscopy.  Biomedical optics express, 10(4):1978–1998, 2019.
  * 83. A. G. York, P. Chandris, D. Dalle Nogare, J. Head, P. Wawrzusin, R. S. Fischer, A. Chitnis, and H. Shroff.  Instant super-resolution imaging in live cells and embryos via analog image processing.  Nature methods, 10(11):1122–1126, 2013.
  * 84. A. G. York, S. H. Parekh, D. Dalle Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff.  Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy.  Nature methods, 9(7):749–754, 2012.

Figure 1: (Caption next page.) Figure 1: Biological super-resolution imaging
with 3D-SIM. (A) Mouse C127 mammary epithelial cell nucleus replication
labelled with 5-ethenyl-2’-deoxyuridine (EdU, red) for 15 min before fixation
with formaldehyde. The thymidine analogue EdU is incorporated into newly
synthesized DNA of S-phase cells (here mid-to-late S phase) and detected via
click-chemistry with Alexa Fluor 594 azide. DNA is labelled with 4’,
6-diamidino-2-phenylindole (DAPI, cyan). Single z-section of an image stack is
shown with conventional wide-field illumination (top, left), structured
illumination (1 of 15 raw images acquired per z-plane with laterally shifted
and rotated stripes; bottom, left), and after 3D-SIM reconstruction (right).
Note that 3D-SIM resolves higher-order domain organisation of chromatin and
DNA-free interchromatin regions (inset), as well as the location of nuclear
pores in the peripheral chromatin layer visible as DAPI void dots in the
central region of the nucleus. Scale bar: 5 µm and 1 µm (inset). (B)
Corresponding frequency distribution of the DAPI signal in Fourier
(reciprocal) space. Concentric rings indicate the respective spatial
resolution in µm. Spots in the raw SI frequency plot (arrowheads) correspond
to first and second order stripes in the image (generated by a three-beam
interference; only coarse first order stripes are visible in the image). (C)
Orthogonal cross section and corresponding frequency distribution of the same
dataset. The arrowheads indicate the position of the z-section shown in (A).
Note the 2-fold extended frequency distribution in the reconstructed data in
both lateral and axial direction, that includes the filling-in of the missing
frequencies along the z-axis of the wide-field frequency plot.


(i) Störmer-Verlet scheme and computation of the constrained learned set
${\mathcal{D}}_{{\bm{H}}_{{\bm{\lambda}}^{\,i}}}$. Let $i$ be the index of the
sequence $\\{{\bm{\lambda}}^{\,i},i=0,1,\ldots,i_{\hbox{{\ppppcarac max}}}\\}$
of the Lagrange multipliers that are computed using Eq. (6.34) with
${\bm{\lambda}}^{\,0}={\hbox{\bf 0}}_{N_{r}}$. For $m=0,1,\ldots,M_{s}$ (with
$M_{s}>1$ an integer), let $t_{m}=m\,\Delta t$ be the time sampling, which is
such that $t_{M_{s}}=t_{s}$ with $t_{s}=M_{s}\,\Delta t$. Let
$\Delta{\bm{W}}_{m+1}^{{\hbox{{\ppppcarac wien}}}}={\bm{W}}^{\hbox{{\ppppcarac
wien}}}(t_{m+1})-{\bm{W}}^{\hbox{{\ppppcarac wien}}}(t_{m})$ be the Gaussian,
second-order, centered, ${\mathbb{R}}^{\nu}$-valued random variable such that
$E\\{\Delta{\bm{W}}_{m+1}^{{\hbox{{\ppppcarac
wien}}}}\otimes\Delta{\bm{W}}_{m+1}^{{\hbox{{\ppppcarac wien}}}}\\}=\Delta
t\,[I_{\nu}]$. Let $\\{\theta_{\ell},\ell=1,\ldots,N\\}$ be $N$ independent
realizations in $\Theta$. For $m=0,1,\ldots,M_{s}-1$, let
$\Delta{\mathbb{W}}_{m+1}^{\ell}=\Delta{\bm{W}}_{m+1}^{{\hbox{{\ppppcarac
wien}}}}(\theta_{\ell})$ be the realization $\theta_{\ell}$ of
$\Delta{\bm{W}}_{m+1}^{{\hbox{{\ppppcarac wien}}}}$. Following the choice of
$({\bm{u}}_{0},{\hbox{\bf{v}}}_{0})$ defined in Theorem 4, let
${\bm{u}}_{0}^{1},\ldots,{\bm{u}}_{0}^{N}$ in ${\mathbb{R}}^{\nu}$ such that
for $k=1,\ldots,n_{\hbox{{\pppppcarac MC}}}$ and for $j=1,\ldots,N_{d}$, we
take ${\bm{u}}_{0}^{\ell}={\bm{\eta}}_{d}^{j}$ with $\ell=j+(k-1)\times
N_{d}$. Let ${\hbox{\bf{v}}}_{0}^{1},\ldots,{\hbox{\bf{v}}}_{0}^{N}$ in
${\mathbb{R}}^{\nu}$ be $N$ independent realizations of the
${\mathbb{R}}^{\nu}$-valued random variable ${\bm{V}}_{G}$ also defined in
Theorem 4. For $\ell=1,\ldots,N$, the realizations
$\Delta{\mathbb{W}}_{m+1}^{\ell}$, ${\bm{u}}_{0}^{\ell}$, and
${\hbox{\bf{v}}}_{0}^{\ell}$ are independent of
$\\{{\bm{\lambda}}^{\,i}\\}_{i}$. For $i\in\\{0,1,\ldots,i_{\hbox{{\ppppcarac
max}}}\\}$ and for $\ell\in\\{1,\ldots,N\\}$, we introduce the realizations
${\mathbb{U}}_{m}^{i,\ell}={\bm{U}}_{\\!{\bm{\lambda}}^{\,i}}(t_{m}\,;\theta_{\ell})$
and
${\mathbb{V}}_{m}^{i,\ell}={\bm{V}}_{\\!{\bm{\lambda}}^{\,i}}(t_{m}\,;\theta_{\ell})$.
For $m\in\\{0,1,\ldots,M_{s}-1\\}$, the Störmer-Verlet scheme applied to
realization $\theta_{\ell}$ of Eqs. (6.27) to (6.29) yields the following
recurrence,

$\displaystyle{\mathbb{U}}_{m+1/2}^{i,\ell}={\mathbb{U}}_{m}^{i,\ell}+\frac{\Delta
t}{2}\,{\mathbb{V}}_{m}^{i,\ell}\,,$ (6.37)
$\displaystyle{\mathbb{V}}_{m+1}^{i,\ell}=\frac{1-\gamma}{1+\gamma}\,{\mathbb{V}}_{m}^{i,\ell}+\frac{\Delta
t}{1+\gamma}\,{\bm{L}}_{{\bm{\lambda}}^{\,i-1}}({\mathbb{U}}_{m+1/2}^{i,\ell})+\frac{\sqrt{f_{0}}}{1+\gamma}\,\Delta{\mathbb{W}}_{m+1}^{\ell}\,,$
(6.38)
$\displaystyle{\mathbb{U}}_{m+1}^{i,\ell}={\mathbb{U}}_{m+1/2}^{i,\ell}+\frac{\Delta
t}{2}\,{\mathbb{V}}_{m+1}^{i,\ell}\,,$ (6.39)

with the initial condition

${\mathbb{U}}_{0}^{i,\ell}={\bm{u}}_{0}^{\ell}\quad,\quad{\mathbb{V}}_{0}^{i,\ell}={\hbox{\bf{v}}}_{0}^{\ell}\,,$
(6.40)

in which $\gamma=f_{0}\,\Delta t/4$ and where, using Eq. (6.30),

${\bm{L}}_{{\bm{\lambda}}^{\,i-1}}({\bm{u}})=\frac{1}{\zeta({\bm{u}})}\,\nabla_{\\!{\bm{u}}}\zeta({\bm{u}})-[\nabla_{\\!{\bm{u}}}{\bm{h}}^{c}({\bm{u}})]\,{\bm{\lambda}}^{\,i-1}$
(6.41)

Using Eq. (5.17), for $\alpha\in\\{1,\ldots,\nu\\}$ and
$r\in\\{1,\ldots,N_{r}\\}$, the entry
$[\nabla_{\\!{\bm{u}}}{\bm{h}}^{c}({\bm{u}})]_{\alpha r}$ of matrix
$[\nabla_{\\!{\bm{u}}}{\bm{h}}^{c}({\bm{u}})]\in{\mathbb{M}}_{\nu,N_{r}}$ is
written as

$[\nabla_{\\!{\bm{u}}}{\bm{h}}^{c}({\bm{u}})]_{\alpha r}=\frac{2}{\nu
s^{2}}([V]^{T}\tilde{\bm{q}}_{\hbox{{\ppppcarac
targ}}}^{r}-u_{\alpha})\,\exp\bigg{(}-\frac{1}{\nu
s^{2}}\,\|\,[V]^{T}\tilde{\bm{q}}_{\hbox{{\ppppcarac
targ}}}^{r}-{\bm{u}}\,\|^{2}\bigg{)}\,.$ (6.42)

The recurrence defined by Eqs. (6.37) to (6.42) allows
${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i}}}$ to be calculated as

${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i}}}=\\{{\bm{\eta}}_{{\bm{\lambda}}^{\,i}}^{1},\ldots,{\bm{\eta}}_{{\bm{\lambda}}^{\,i}}^{N}\\}\quad,\quad{\bm{\eta}}_{{\bm{\lambda}}^{\,i}}^{\ell}={\bm{U}}_{\\!{\bm{\lambda}}^{\,i}}(t_{s}\,;\theta_{\ell})={\mathbb{U}}_{M_{s}}^{i,\ell}\,.$
(6.43)

(ii) Summary of the algorithm. The algorithm for calculating
${\bm{\lambda}}^{\hbox{{\ppppcarac sol}}}$ and the
${\mathcal{D}}_{{\bm{H}}^{\hbox{{\ppppcarac
post}}}}=\\{{\bm{\eta}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{\eta}}_{\hbox{{\ppppcarac post}}}^{N}\\}$ with
${\bm{\eta}}_{\hbox{{\ppppcarac
post}}}^{\ell}={\bm{\eta}}_{{\bm{\lambda}}^{\hbox{{\ppppcarac sol}}}}^{\ell}$
for $\ell=1,\ldots,N$ is summarized in Algorithm 1. We then obtained the $N$
independent realizations ${\bm{\eta}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{\eta}}_{\hbox{{\ppppcarac post}}}^{N}$ of the
posterior ${\mathbb{R}}^{\nu}$-valued random variable
${\bm{H}}_{\hbox{{\ppppcarac post}}}$. Then, the realizations
${\bm{q}}_{\hbox{{\ppppcarac post}}}^{1},\ldots,{\bm{q}}_{\hbox{{\ppppcarac
post}}}^{N}$ of the posterior observations ${\bm{Q}}_{\hbox{{\ppppcarac
post}}}$ and the realizations ${\bm{w}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{w}}_{\hbox{{\ppppcarac post}}}^{N}$ of the posterior
control parameter ${\bm{W}}_{\hbox{{\ppppcarac post}}}$ are computed using Eq.
(2.4), that is to say,

${\bm{Q}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{q}}}+[\Phi_{q}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}\quad,\quad{\bm{W}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{w}}}+[\Phi_{w}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}\,.$ (6.44)

Algorithm 1 Algorithm for calculating ${\bm{\lambda}}^{\hbox{{\ppppcarac
sol}}}$ and ${\mathcal{D}}_{{\bm{H}}_{\hbox{{\pppppcarac
post}}}}=\\{{\bm{\eta}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{\eta}}_{\hbox{{\ppppcarac post}}}^{N}\\}$.

1:Data:$N_{d}$,
${\mathcal{D}}_{d}=\\{{\bm{\eta}}_{d}^{1},\ldots,{\bm{\eta}}_{d}^{N_{d}}\\}$,
$N$, $i_{\hbox{{\ppppcarac max}}}$, $M_{s}$, $t_{s}$, $\Delta t$, $f_{0}$,
$\gamma=f_{0}\,\Delta t/4$

2:Init:
$\,\,\Delta{\mathbb{W}}_{m+1}^{\ell},\ell\in\\{1,\ldots,N\\},m\in\\{1,\ldots,M_{s}-1\\}$,
$\,\,{\bm{u}}_{0}^{\ell}$ and ${\hbox{\bf{v}}}_{0}^{\ell}$ for
$\ell\in\\{1,\ldots,N\\}$, ${\bm{\lambda}}^{\,0}={\hbox{\bf 0}}_{N_{r}}$

3:for $i=1:i_{\hbox{{\ppppcarac max}}}$ do

4: for $\ell=1:N\,(loop\,in\,parallel\,computation)$ do

5:
${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i}}}=\\{{\bm{\eta}}_{{\bm{\lambda}}^{\,i}}^{1},\ldots,{\bm{\eta}}_{{\bm{\lambda}}^{\,i}}^{N}\\}$
from Eq. (6.43), using Eqs. (6.37) to (6.42) and
${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i-1}}}$
(${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,0}}}$ not used for $i=1$)

6: end for

7: $\nabla_{\\!{\bm{\lambda}}}\Gamma({\bm{\lambda}}^{\,i})$ and
$[\Gamma{\,{}^{\prime\prime}}({\bm{\lambda}}^{\,i})]$ using Eqs. (6.19) and
(6.20), and ${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i}}}$

8: ${\hbox{{err}}}(i)$ using Eq. (6.36)

9: ${\bm{\lambda}}^{\,i+1}={\bm{\lambda}}^{\,i}-\alpha_{{\hbox{{\ppppcarac
relax}}}}(i)\,[\Gamma{\,{}^{\prime\prime}}({\bm{\lambda}}^{\,i})]^{-1}\,\nabla_{\\!{\bm{\lambda}}}\Gamma({\bm{\lambda}}^{\,i})$
using Eq. (6.34) with a relaxation factor $\alpha_{{\hbox{{\ppppcarac
relax}}}}(i)\in]0\,,1]$

10: ${\bm{\lambda}}^{\,i}\leftarrow{\bm{\lambda}}^{\,i+1}$

11:
${\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i-1}}}\leftarrow{\mathcal{D}}_{{\bm{H}}_{\\!{\bm{\lambda}}^{\,i}}}$

12:end for

13:${\bm{\lambda}}^{\hbox{{\ppppcarac
sol}}}={\bm{\lambda}}^{i_{\hbox{{\ppppppcarac sol}}}}$, $i_{\hbox{{\ppppcarac
sol}}}=\arg\,\min_{i}{\hbox{{err}}}(i)\,\,$ from Eq. (6.35)

14:${\mathcal{D}}_{{\bm{H}}_{\hbox{{\pppppcarac
post}}}}\leftarrow{\mathcal{D}}_{{\bm{H}}_{{\bm{\lambda}}^{\hbox{{\ppppppcarac
sol}}}}}$

##  7 Numerical illustration

We consider a supervised case. The training set
$D_{d}=\\{{\bm{x}}^{1},\ldots,{\bm{x}}^{N_{d}}\\}$ with
${\bm{x}}^{j}\\!\\!=\\!({\bm{q}}_{d}^{j},{\bm{w}}_{d}^{j})\in{{{\mathbb{R}}^{n_{x}}}}\\!\\!=\\!{{{\mathbb{R}}^{n_{q}}}}\\!\times\\!{{{\mathbb{R}}^{n_{w}}}}$
is made up of $N_{d}$ independent realizations of random variable
${\bm{X}}=({\bm{Q}},{\bm{W}})$, which are generated as explained in A for
which $n_{x}=430\,098$, $n_{q}=10\,098$, $n_{w}=420\,000$, and
$N_{d}\in\\{100,200,300,400\\}$. The target set $D_{\hbox{{\ppppcarac targ}}}$
is made up of $N_{r}$ independent realizations ${\bm{q}}_{\hbox{{\ppppcarac
targ}}}^{r}\in{{{\mathbb{R}}^{n_{q}}}}$ of random variable
${\bm{Q}}_{\hbox{{\ppppcarac targ}}}$ for which
$N_{r}\in[50\,,N_{\hbox{{\ppppcarac targ}}}]$ with $N_{\hbox{{\ppppcarac
targ}}}\in\\{100,200,300,400\\}$. The ${{{\mathbb{R}}^{n_{q}}}}$-valued random
variable ${\bm{Q}}$ corresponds to the finite element discretization of a
${\mathbb{R}}^{3}$-valued random field
$\\{{\mathbb{U}}({\bm{\omega}})=({\mathbb{U}}_{1}({\bm{\omega}}),{\mathbb{U}}_{2}({\bm{\omega}}),{\mathbb{U}}_{3}({\bm{\omega}})\,,{\bm{\omega}}\in\overline{\Omega}\\}$
and the ${{{\mathbb{R}}^{n_{w}}}}$-valued random variable ${\bm{W}}$ is a
nonlinear transformation of the finite element discretization of a
${\mathbb{M}}_{6}^{+}$-valued random field
$\\{[{\bm{G}}({\bm{\omega}})]\,,{\bm{\omega}}\in\Omega\\}$, in which $\Omega$
is the open bounded set of ${\mathbb{R}}^{3}$ defined in A.1, and where
${\bm{Q}}$ and ${\bm{W}}$ are constructed in A.2. Regarding the presentation
of the results, and having to limit the number of figures, the probability
density functions and the convergence curves when they will be relative to
${\bm{Q}}$, will be limited to $3$ components, denoted by
$Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac obs}}},2}$, and
$Q_{{\hbox{{\pppcarac obs}}},3}$ that are also defined at the end of A.2.

###  7.1 Training set

The training set is generated as explained in A.2 with the stochastic boundary
value problem defined in A.1. For illustration, Fig. 2 shows one realization
$\theta\in\Theta$ of the components $(1,1)$, $(1,2)$, and $(4,4)$ of the
${\mathbb{M}}^{+}_{6}$-valued random field
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3})]$
in the plane $\omega_{3}=0.095774$ and Fig. 3 shows the corresponding
realization of the components $k=1,2,3$ of the real-valued random field
$(\omega_{1},\omega_{2})\mapsto{\mathbb{U}}_{k}(\omega_{1},\omega_{2},\omega_{3})$
in the plane $\omega_{3}=0.095774$.

(a)
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3};\theta)]_{11}$

(b)
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3};\theta)]_{12}$

(c)
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3};\theta)]_{44}$

Figure 2: For the training set, example of one realization $\theta\in\Theta$
of the components $(1,1)$, $(1,2)$, and $(4,4)$ of the random field
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3})]$
in the plane $\omega_{3}=0.095774$.

(a)
$(\omega_{1},\omega_{2})\mapsto{\mathbb{U}}_{1}(\omega_{1},\omega_{2},\omega_{3};\theta)$

(b)
$(\omega_{1},\omega_{2})\mapsto{\mathbb{U}}_{2}(\omega_{1},\omega_{2},\omega_{3};\theta)$

(c)
$(\omega_{1},\omega_{2})\mapsto{\mathbb{U}}_{3}(\omega_{1},\omega_{2},\omega_{3};\theta)$

Figure 3: For the training set, example of one realization $\theta\in\Theta$
of the components $1$, $2$, and $3$ of the random field
$(\omega_{1},\omega_{2})\mapsto{\mathbb{U}}(\omega_{1},\omega_{2},\omega_{3})$
in the plane $\omega_{3}=0.095774$.

###  7.2 Reduced representation

The reduced representation is constructed by using a PCA of
${\bm{X}}=({\bm{Q}},{\bm{W}})$ as explained in Section 2. With
$\varepsilon_{\hbox{{\pppppcarac PCA}}}=0.0001$, for $N_{d}=100$, $200$,
$300$, and $400$, we have, respectively, $\nu=99$, $192$, $271$, and $331$.
For $N_{d}=100$, Fig. 4 displays the graph of the eigenvalues
$\alpha\mapsto\kappa_{\alpha}$ of $[\widehat{C}_{\bm{X}}]$ and the graph of
the error function $\nu\mapsto{\hbox{{\pcarac err}}}_{\hbox{{\pppppcarac
PCA}}}(\nu\,;N_{d})$ defined by Eq. (2.3).

(a) Graph of $\alpha\mapsto\kappa_{\alpha}$

(b) Graph of $\nu\mapsto{\hbox{{\pcarac err}}}_{\hbox{{\pppppcarac
PCA}}}(\nu\,;N_{d})$

Figure 4: Principal component analysis of the training set performed for
$N_{d}=100$. Graph of the eigenvalues $\alpha\mapsto\kappa_{\alpha}$ of
$[\widehat{C}_{\bm{X}}]$ and error function $\nu\mapsto{\hbox{{\pcarac
err}}}_{\hbox{{\pppppcarac PCA}}}(\nu\,;N_{d})$ defined by Eq. (2.3).

###  7.3 Numerical values of the algorithm parameters for computing the
constrained learned set.

In all the calculations and for any value of ${\bm{\lambda}}$, the number $N$
of the independent realizations
$\\{{\bm{\eta}}_{\bm{\lambda}}^{1},\ldots,{\bm{\eta}}_{\bm{\lambda}}^{N}\\}$
of the constrained learned set ${\mathcal{D}}_{{\bm{H}}_{\bm{\lambda}}}$ (see
Section 6.7) is fixed to the value $N=1\,000$ (this value has been obtained
from a convergence analysis with respect to $N$). The parameters of the
Störmer-Verlet scheme are $M_{s}=30$, $\Delta t=0.2188$, and $f_{0}=4$. Due to
the choice $f_{0}=4$, the stationary regime of the ISDE is obtained from
instant $21\times\Delta t$ and the realizations is extracted at
$30\times\Delta t$.

###  7.4 Iterative algorithm for computing ${\bm{\lambda}}^{\hbox{{\ppppcarac
sol}}}$.

For computing the solution ${\bm{\lambda}}^{\hbox{{\ppppcarac sol}}}$ of the
Lagrange multiplier, Algorithm 1 is used. For $N_{d}=100$ and $N_{r}=20$, Fig.
5 displays the graph of the relaxation factor
$i\mapsto\alpha_{\hbox{{\ppppcarac relax}}}(i)$ and the graph of the error
function $i\mapsto{\hbox{{err}}}(i)$ defined by Eq. (6.36). It can be seen a
fast convergence of the algorithm as a function of the iteration number. Fig.
6 shows the graph of function $r\mapsto b^{c}_{r}$ defined by Eq. (5.19),
representing the components of vector
${\bm{b}}^{c}=(b_{1}^{c},\ldots,b_{N_{r}}^{c})$, and the graph of function
$r\mapsto\lambda^{\hbox{{\ppppcarac sol}}}_{r}$, representing the components
of vector ${\bm{\lambda}}^{\hbox{{\ppppcarac
sol}}}=(\lambda_{1}^{\hbox{{\ppppcarac
sol}}},\ldots,\lambda_{N_{r}}^{\hbox{{\ppppcarac sol}}})$ defined by Eq.
(6.35).

(a) Graph of $i\mapsto\alpha_{\hbox{{\ppppcarac relax}}}(i)$

(b) Graph of $i\mapsto{\hbox{{err}}}(i)$

Figure 5: Iterative algorithm for calculating
${\bm{\lambda}}^{\hbox{{\ppppcarac sol}}}$ with the constrained learned set
for $N_{d}=100$ and $N_{r}=20$. Graph of the relaxation factor
$i\mapsto\alpha_{\hbox{{\ppppcarac relax}}}(i)$ defined in Algorithm 1 and
graph of the error function $i\mapsto{\hbox{{err}}}(i)$ defined by Eq. (6.36).

(a) Graph of $r\mapsto b^{c}_{r}$

(b) Graph of $r\mapsto\lambda^{\hbox{{\ppppcarac sol}}}_{r}$

Figure 6: Constraint and optimal Lagrange multiplier estimated with the
constrained learned set for $N_{d}=100$ and $N_{r}=20$. Graph of function
$r\mapsto b^{c}_{r}$ defined by Eq. (5.19) and graph of function
$r\mapsto\lambda^{\hbox{{\ppppcarac sol}}}_{r}$ with
${\bm{\lambda}}^{\hbox{{\ppppcarac sol}}}$ defined by Eq. (6.35).

###  7.5 Posterior probability measure of ${\bm{Q}}_{\hbox{{\ppppcarac
post}}}$ estimated with the constrained learned set

For $N_{d}=100$, the convergence with respect to $N_{r}$ of the posterior
probability measure of ${\bm{Q}}_{\hbox{{\ppppcarac post}}}(N_{r})$ estimated
with the constrained learned set has been analyzed by studying, for $k=1,2,3$,
the mean-square norm $|||Q_{{\hbox{{\pppcarac
obs}}},k}(N_{r})|||=\\{E\\{Q_{{\hbox{{\pppcarac
obs}}},k}(N_{r})^{2}\\}\\}^{1/2}$ of random component $Q_{{\hbox{{\pppcarac
obs}}},k}(N_{r})$ of ${\bm{Q}}_{\hbox{{\ppppcarac post}}}(N_{r})$ (which
depends on $N_{r}$). For $k=1,2,3$, Fig. 7 shows the graph of function
$N_{r}\mapsto|||Q_{{\hbox{{\pppcarac obs}}},k}(N_{r})|||$ as well as the
corresponding value for the training set and for the reference, which are both
independent of $N_{r}$. This figure shows the good convergence with respect to
$N_{r}$, which is reached for $N_{r}=20$. Always for $N_{d}=100$, Fig. 8 (a),
(b), and (c) related to $N_{r}=20$, and Fig. 8 (d), (e), and (f) related to
$N_{r}=100$, display the probability density functions of the random variables
$Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac obs}}},2}$, and
$Q_{{\hbox{{\pppcarac obs}}},3}$, estimated with the training set, with the
constrained learned set (the posterior), and the reference. Figs. 7 and 8 show
that the posteriors are close to the targets (this good result holds for all
the components of ${\bm{Q}}$). For this supervised framework, it can be seen
that the proposed method performs very well for the quantity of interest
${\bm{Q}}$ for which a target has been given, which means that there would
also be very good behavior of the method if used in an unsupervised setting.

(a) $N_{r}\mapsto|||Q_{{\hbox{{\pppcarac obs}}},1}(N_{r})|||$

(b) $N_{r}\mapsto|||Q_{{\hbox{{\pppcarac obs}}},2}(N_{r})|||$

(c) $N_{r}\mapsto|||Q_{{\hbox{{\pppcarac obs}}},3}(N_{r})|||$

Figure 7: Convergence of the mean-square norm with respect to $N_{r}$ for
$N_{d}=100$. Graph of $N_{r}\mapsto|||Q_{{\hbox{{\pppcarac
obs}}},1}(N_{r})|||$, $|||Q_{{\hbox{{\pppcarac obs}}},2}(N_{r})|||$, and
$|||Q_{{\hbox{{\pppcarac obs}}},3}(N_{r})|||$ estimated with the training set
(top black line), with the constrained learned set (the posterior) (blue
line), and the reference (red thick line).

(a) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{r}=20$

(b) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{r}=20$

(c) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{r}=20$

(d) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{r}=100$

(e) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{r}=100$

(f) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{r}=100$

Figure 8: For $N_{d}=100$, for $N_{r}=20$ (Figs. (a), (b), and (c)) and for
$N_{r}=100$ (Figs. (d), (e), and (f)), probability density functions of the
random variables $Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac
obs}}},2}$, and $Q_{{\hbox{{\pppcarac obs}}},3}$, estimated with the training
set (black line), with the constrained learned set (the posterior) (blue
line), and the reference (red thick line).

###  7.6 Posterior probability measure of ${\bm{W}}_{\hbox{{\ppppcarac
post}}}$ estimated with the constrained learned set

Fig. 9 is related to the standard deviation fields
$(\omega_{1},\omega_{2})\mapsto\sigma_{G11}(\omega_{1},\omega_{2},\omega_{3})$,
$\sigma_{G12}(\omega_{1},\omega_{2},\omega_{3})$, and
$\sigma_{G44}(\omega_{1},\omega_{2},\omega_{3})$ in the plane
$\omega_{3}=0.095774$ of the components $(1,1)$, $(1,2)$, and $(4,4)$ of the
random field
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3})]$
for the training set with $N_{d}=100$, for the transformation of the posterior
${\bm{W}}_{\hbox{{\ppppcarac post}}}$ (see A.2) computed with the constrained
learned set for which $N_{d}=100$ and $N_{r}=20$, and finally, for the
reference.

(a) Training set,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G11}(\omega_{1},\omega_{2},\omega_{3})$

(b) Training set,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G12}(\omega_{1},\omega_{2},\omega_{3})$

(c) Training set,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G44}(\omega_{1},\omega_{2},\omega_{3})$

(d) Posterior,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G11}(\omega_{1},\omega_{2},\omega_{3})$

(e) Posterior,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G12}(\omega_{1},\omega_{2},\omega_{3})$

(f) Posterior,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G44}(\omega_{1},\omega_{2},\omega_{3})$

(g) Reference,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G11}(\omega_{1},\omega_{2},\omega_{3})$

(h) Reference,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G12}(\omega_{1},\omega_{2},\omega_{3})$

(i) Reference,
$(\omega_{1},\omega_{2})\mapsto\sigma_{G44}(\omega_{1},\omega_{2},\omega_{3})$

Figure 9: For the training set with $N_{d}=100$ (Figs. (a, b, and c), for the
posterior estimated with the constrained learned set with $N_{d}=100$ and
$N_{r}=20$ (Figs. (d, e, and f), and for the reference (Figs. (g,h, and i),
standard deviation fields
$(\omega_{1},\omega_{2})\mapsto\sigma_{G11}(\omega_{1},\omega_{2},\omega_{3})$,
$\sigma_{G12}(\omega_{1},\omega_{2},\omega_{3})$, and
$\sigma_{G44}(\omega_{1},\omega_{2},\omega_{3})$ in the plane
$\omega_{3}=0.095774$ of the components $(1,1)$, $(1,2)$, and $(4,4)$ of the
random field
$(\omega_{1},\omega_{2})\mapsto[{\bm{G}}(\omega_{1},\omega_{2},\omega_{3})]$.

Since there is no target for the control variable, we cannot directly compare
${\bm{W}}_{\hbox{{\ppppcarac post}}}$ (that is to say the random field
$\\{[{\bm{G}}_{\hbox{{\ppppcarac
post}}}({\bm{\omega}})],{\bm{\omega}}\in\Omega\\}$) with a target. In the case
of the numerical illustration that we present, the map ${\bm{f}}$ such that
${\bm{Q}}={\bm{f}}({\bm{W}})$ is known numerically, that is, for
${\bm{w}}^{j}_{d}$ given, we have calculated
${\bm{q}}^{j}_{d}={\bm{f}}({\bm{w}}_{d}^{j})$ with the boundary value problem
to generate the training set. This situation is particular and is not that of
the general framework that we have given ourselves. Nevertheless, this
particular situation allows us to use another method to qualify the quality of
the probability measure of ${\bm{W}}_{\hbox{{\ppppcarac post}}}$ estimated
with the constrained learned set, as follows. The proposed constrained-
learned-set algorithm allows for computing the realizations
$\\{{\bm{\eta}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{\eta}}_{\hbox{{\ppppcarac post}}}^{N}\\}$ of the
posterior random variable ${\bm{H}}_{\hbox{{\ppppcarac post}}}$, which
constitute the points of the constrained learned set. Using Eq. (6.44), the
realizations ${\bm{q}}_{\hbox{{\ppppcarac
post}}}^{1},\ldots,{\bm{q}}_{\hbox{{\ppppcarac post}}}^{N}$ and
${\bm{w}}_{\hbox{{\ppppcarac post}}}^{1},\ldots,{\bm{w}}_{\hbox{{\ppppcarac
post}}}^{N}$ are computed by the equations ${\bm{Q}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{q}}}+[\Phi_{q}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}$ and ${\bm{W}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{w}}}+[\Phi_{w}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}$. We can then compare ${\bm{Q}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{q}}}+[\Phi_{q}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}$ with ${\bm{Q}}^{\hbox{{\ppppcarac
QA}}}={\bm{f}}({\bm{W}}_{\hbox{{\ppppcarac post}}})$ in which
${\bm{W}}_{\hbox{{\ppppcarac
post}}}=\underline{{\bm{w}}}+[\Phi_{w}]\,[\kappa]^{1/2}\,{\bm{H}}_{\hbox{{\ppppcarac
post}}}$ and where mapping ${\bm{f}}$ is evaluated with the computational
model.

For $N_{r}=100$, the convergence of ${\bm{Q}}^{\hbox{{\ppppcarac
QA}}}={\bm{f}}({\bm{W}}_{\hbox{{\ppppcarac post}}})$ with respect to $N_{d}$
has been analyzed by studying, for $k=1,2,3$, the mean-square norm
$|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac
obs}}},k}|||=\\{E\\{(Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac
obs}}},k})^{2}\\}\\}^{1/2}$ of random component $Q^{\hbox{{\ppppcarac
QA}}}_{{\hbox{{\pppcarac obs}}},k}$ of ${\bm{Q}}^{\hbox{{\ppppcarac QA}}}$
(which depends on $N_{d}$). For $k=1,2,3$, Table 1 yields the values of
$|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},k}|||$. The expected
convergence can be viewed with respect to $N_{d}$ (this result is consistent
with the convergence of the pdf’s shown in Fig. 10).

Table 1: For $N_{r}=100$, convergence of the mean-square norm of $|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},1}|||$, $|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},2}|||$, and $|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},3}|||$ as a function of $N_{d}$. $N_{d}$ | 100 | 200 | 300 | 400
---|---|---|---|---
$|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},1}|||$ | $7.59\times 10^{-3}$ | $8.15\times 10^{-3}$ | $8.12\times 10^{-3}$ | $8.20\times 10^{-3}$
$|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},2}|||$ | $2.44\times 10^{-2}$ | $2.60\times 10^{-2}$ | $2.63\times 10^{-2}$ | $2.67\times 10^{-2}$
$|||Q^{\hbox{{\ppppcarac QA}}}_{{\hbox{{\pppcarac obs}}},3}|||$ | $5.11\times 10^{-3}$ | $5.46\times 10^{-3}$ | $6.35\times 10^{-3}$ | $5.56\times 10^{-3}$

(a) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{d}=100$

(b) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{d}=100$

(c) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{d}=100$

(d) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{d}=200$

(e) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{d}=200$

(f) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{d}=200$

(g) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{d}=300$

(h) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{d}=300$

(i) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{d}=300$

(j) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},1}}(q)$ for $N_{d}=400$

(k) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},2}}(q)$ for $N_{d}=400$

(l) pdf $q\mapsto p_{Q_{{\hbox{{\ppppcarac obs}}},3}}(q)$ for $N_{d}=400$

Figure 10: For $N_{r}=100$, convergence analysis in $N_{d}$, for $N_{d}=100$
(Figs. (a), (b), and (c)), for $N_{d}=200$ (Figs. (d), (e), and (f)), for
$N_{d}=300$ (Figs. (g), (h), and (i)), and for $N_{d}=400$ (Figs. (j), (k),
and (l)), probability density functions of the random variables
$Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac obs}}},2}$, and
$Q_{{\hbox{{\pppcarac obs}}},3}$, for ${\bm{Q}}$ estimated with the training
set (black line), for ${\bm{Q}}^{\hbox{{\ppppcarac
QA}}}={\bm{f}}({\bm{W}}_{\hbox{{\ppppcarac post}}})$ estimated with the
posterior learned set of ${\bm{W}}_{\hbox{{\ppppcarac post}}}$ (blue line),
and for ${\bm{Q}}_{\hbox{{\ppppcarac targ}}}$ corresponding to the reference
(red thick line).

In addition to the convergence analysis of the mean-square norm with respect
to $N_{d}$, Fig. 10 displays the probability density functions of the random
variables $Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac obs}}},2}$,
and $Q_{{\hbox{{\pppcarac obs}}},3}$, for ${\bm{Q}}$ estimated with the
training set, for ${\bm{Q}}^{\hbox{{\ppppcarac
QA}}}={\bm{f}}({\bm{W}}_{\hbox{{\ppppcarac post}}})$ estimated with
${\bm{W}}_{\hbox{{\ppppcarac post}}}$, and for ${\bm{Q}}_{\hbox{{\ppppcarac
targ}}}$ corresponding to the reference. Similarly to the convergence of the
mean-square norm, this figure shows a clear convergence with respect to
$N_{d}$. For $N_{d}=100$ and $N_{r}=20$ or even $N_{r}=100$, compared to the
reference, the posterior ${\bm{Q}}^{\hbox{{\ppppcarac QA}}}$ evaluated with
${\bm{W}}_{\hbox{{\ppppcarac post}}}$ thanks to the knowledge of ${\bm{f}}$,
is less good than the prediction of the posterior ${\bm{Q}}_{\hbox{{\ppppcarac
post}}}$. This is mainly due to the use of the reduced representation for a
problem in high dimension. Fig. 10 shows that the prediction can be improved
by increasing the value of $N_{d}$, that is to say, by increasing the value of
$\nu$, which requires to increase the number of points in the training set and
consequently, which can induce potential difficulties if the numerical cost
for constructing each point of the training set is high. However, the
presented numerical illustration shows that the proposed method allows for
integrating,with a good quality, a target set of realizations (i.e. data) in a
supervised model, which is defined only by a small number of points in a
training set and for which the target set of realizations are specified only
for the quantities of interest (output) and not for the control variable
(input). Finally, it should be noted that, when the training set is generated
using a stochastic boundary value problem, there is also another method as we
have proposed and validated in [39, 51]. It consists, for the generation of
the constrained learned set, to introduce an additional scalar constraint to
minimize the norm of the residue of the partial differential equations of the
boundary value problem. This procedure can be implemented without difficulty
in the methodology presented in this paper, involving only one additional
component in the vector-valued function ${\bm{h}}^{C}$ and the vector
${\bm{b}}^{c}$.

##  8 Conclusion

In this paper, we have presented a novel functional approach that makes it
possible to take into account a target set of realizations in the Kullback-
Leibler minimum principle for constructing a posterior probability measure
from a prior probability measure defined by a given training set of
realizations. This approach thus allows for integrating a target set of
realizations in a supervised model, which is defined only by a small number of
points in a training set. It consists in constructing and analyzing a weak
formulation of the Fourier transform of the probability measure
(characteristic function) of the observed quantities of interest and to derive
from it a finite representation of the functional constraint. On the basis of
the positive Hermitian form associated with the Fourier transform of the
probability measure, we have constructed and analyzed the properties of a
functional family of functions, which only depends on the target set of the
given realizations. These properties have allowed us to show the existence and
the uniqueness of the posterior probability measure constructed by using the
Kullback-Leibler minimum principle. The numerical aspects have been detailed
in order to facilitate the implementation of the algorithms. The presented
numerical illustration that is in high dimension demonstrates the efficiency
and the robustness of the proposed method.

##  Appendix A Generation of the training set, target set, and numerical
values of the parameters

The training set $D_{d}=\\{{\bm{x}}^{1},\ldots,{\bm{x}}^{N_{d}}\\}$ with
${\bm{x}}^{j}\\!\\!=\\!({\bm{q}}_{d}^{j},{\bm{w}}_{d}^{j})\in{{{\mathbb{R}}^{n_{x}}}}\\!\\!=\\!{{{\mathbb{R}}^{n_{q}}}}\\!\times\\!{{{\mathbb{R}}^{n_{w}}}}$
is made up of $N_{d}$ independent realizations of random variable
${\bm{X}}=({\bm{Q}},{\bm{W}})$, which are generated by using a stochastic
computational model corresponding to the finite element discretization of a
stochastic elliptic boundary value problem for which $n_{x}=430\,098$,
$n_{q}=10\,098$, and $n_{w}=420\,000$. The target set $D_{\hbox{{\ppppcarac
targ}}}=\\{{\bm{q}}^{1}_{\hbox{{\ppppcarac
targ}}},\ldots,{\bm{q}}^{N_{r}}_{\hbox{{\ppppcarac targ}}}\\}$ is generated
using the stochastic computational model with another values of the parameters
(see A.3).

###  A.1 Definition of the stochastic boundary value problem

Let $\Omega=]\,0\,,1\,[\,\times\,]\,0\,,0.2\,[\,\times\,]\,0\,,0.1\,[\,m^{3}$
be the bounded open set of ${\mathbb{R}}^{3}$, with generic point
${\bm{\omega}}=(\omega_{1},\omega_{2},\omega_{3})$, and with boundary
$\partial\Omega=\Gamma_{0}\cup\Gamma_{1}\cup\Gamma_{2}$ in which
$\Gamma_{0}=\\{\omega_{1}=1\,,\,0\leq\omega_{2}\leq
0.2\,,\,0\leq\omega_{3}\leq 0.1\\}$,
$\Gamma_{1}=\\{\omega_{1}=0\,,\,0\leq\omega_{2}\leq
0.2\,,\,0\leq\omega_{3}\leq 0.1\\}$, and
$\Gamma_{2}=\partial\Omega\backslash\\{\Gamma_{0}\cup\Gamma_{1}\\}$. Let be
$\overline{\Omega}=\Omega\cup\partial\Omega$. The outward unit normal to
$\partial\Omega$ is denoted by
${\mathbb{n}}=({\mathbb{n}}_{1},{\mathbb{n}}_{2},{\mathbb{n}}_{3})$. We use
the usual convention of summation on repeated Latin indices. Domain $\Omega$
is occupied by a heterogeneous and anisotropic elastic random medium for which
the elastic properties are defined by the fourth-order tensor-valued non-
Gaussian random field
${\mathbb{A}}=\bigg{\\{}\\{{\mathbb{A}}_{kmnq}({\bm{\omega}})\\}_{kmnq},{\bm{\omega}}\in\Omega\bigg{\\}}$.
Let ${\mathbb{U}}=({\mathbb{U}}_{1},{\mathbb{U}}_{2},{\mathbb{U}}_{3})$ be the
${\mathbb{R}}^{3}$-valued displacement random field defined in $\Omega$. A
Dirichlet condition ${\mathbb{U}}={\hbox{\bf 0}}$ is given on $\Gamma_{0}$
while a Neumann condition is given on $\Gamma_{1}\cup\Gamma_{2}$. The
stochastic boundary value problem is written, for $k=1,2,3$ and almost surely,
as

$\displaystyle-\frac{\partial{\mathbb{\sigma}}_{km}}{\partial\omega_{m}}$
$\displaystyle=0\,\,\,\,\hbox{in}\,\,\,\Omega\,,$ (A.1)
$\displaystyle{\mathbb{U}}_{k}$
$\displaystyle=0\,\,\,\,\hbox{on}\,\,\,\Gamma_{0}\,,$ (A.2)
$\displaystyle{\mathbb{\sigma}}_{km}\,{\mathbb{n}}_{m}$
$\displaystyle=p_{k}\,\,\hbox{on}\,\,\,\Gamma_{1}\,,$ (A.3)
$\displaystyle{\mathbb{\sigma}}_{km}\,{\mathbb{n}}_{m}$
$\displaystyle=0\,\,\,\,\hbox{on}\,\,\,\Gamma_{2}\,,$ (A.4)

in which the stress tensor ${\mathbb{\sigma}}$ is related to the strain tensor
${\mathbb{\epsilon}}$ by
${\mathbb{\epsilon}}_{nq}=(\partial{\mathbb{U}}_{n}/\partial\omega_{q}+\partial{\mathbb{U}}_{q}/\partial\omega_{n})/2$
by the constitutive equation,
${\mathbb{\sigma}}_{km}({\bm{\omega}})={\mathbb{A}}_{kmnq}({\bm{\omega}})\,{\mathbb{\epsilon}}_{nq}({\mathbb{U}}({\bm{\omega}}))$.
For $k=1,2,3$, the applied stresses $p_{k}$ on $\Gamma_{1}$ are defined as
follows:

$p_{1}=0$ on $\Gamma_{1}$, except:

$\quad p_{1}=-1.8\times 10^{8}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,0\leq\omega_{2}\leq
0.02\,,\,0\leq\omega_{3}\leq 0.1\\}$.

$\quad p_{1}=+9.0\times 10^{7}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,0.18\leq\omega_{2}\leq
0.2\,,\,0\leq\omega_{3}\leq 0.1\\}$.

$p_{2}=0$ on $\Gamma_{1}$, except:

$\quad p_{2}=+1.0\times 10^{7}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,\\{0\leq\omega_{2}\leq
0.02\\}\cup\\{0.18\leq\omega_{2}\leq 0.20\\}\,,\,0\leq\omega_{3}\leq 0.02\\}$.

$\quad p_{2}=-1.5\times 10^{7}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,\\{0\leq\omega_{2}\leq
0.02\\}\cup\\{0.18\leq\omega_{2}\leq 0.20\\}\,,\,0.08\leq\omega_{3}\leq
0.1\\}$.

$p_{3}=0$ on $\Gamma_{1}$, except:

$\quad p_{3}=-2.40\times 10^{7}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,0\leq\omega_{2}\leq
0.02\,,\,0\leq\omega_{3}\leq 0.1\\}$.

$\quad p_{3}=+2.64\times 10^{7}\,N/m^{2}$ for
${\bm{\omega}}\in\\{\omega_{1}=0\,,\,0.18\leq\omega_{2}\leq
0.2\,,\,0\leq\omega_{3}\leq 0.1\\}$.

Using the matrix representation in Voigt notation, the random elasticity field
is rewritten, for $k$, $m$, $n$, and $q$ in $\\{1,2,3\\}$, as
$[{\bm{A}}({\bm{\omega}})]_{{\hbox{\pbf{i}}}{\hbox{\pbf{j}}}}={\mathbb{A}}_{kmnq}({\bm{\omega}})$
with ${\hbox{\bf{i}}}=(k,m)$ with $1\leq k\leq m\leq 3$ and
${\hbox{\bf{j}}}=(n,q)$ with $1\leq n\leq q\leq 3$ in which indices i and j
belong to $\\{1,\ldots,6\\}$. The ${\mathbb{M}}_{6}^{+}$-valued random field
$\\{[{\bm{A}}({\bm{\omega}})],{\bm{\omega}}\in\Omega\\}$ is a non-Gaussian,
second order, and statistically homogeneous. Its mean function is the given
${\bm{\omega}}$-independent matrix
$[\,\underline{{\bm{A}}}\,]=E\\{[{\bm{A}}({\bm{\omega}})]\\}\in{\mathbb{M}}_{6}^{+}$
corresponding to a homogeneous isotropic elastic material whose Young modulus
is $10^{10}\,N/m^{2}$ and Poisson coefficient $0.15$ (note that the
fluctuations around the mean are those of a heterogeneous anisotropic elastic
material). The non-Gaussian ${\mathbb{M}}_{6}^{+}$-valued random field
$\\{[{\bm{A}}({\bm{\omega}})]\,,{\bm{\omega}}\in\Omega\\}$ is constructed
using the stochastic model [76, 77, 68] of random elasticity fields for
heterogeneous anisotropic elastic media that are isotropic in statistical mean
and exhibit anisotropic statistical fluctuations, for which the
parameterization consists of spatial-correlation lengths and of a positive-
definite lower bound. The random field
$\\{[{\bm{A}}({\bm{\omega}})],{\bm{\omega}}\in\Omega\\}$ is written as,

$[{\bm{A}}({\bm{\omega}})]=\frac{1}{1+\epsilon}\,[\underline{{\mathbb{L}}}]^{T}\,\bigg{(}\epsilon\,[I_{6}]+[{\bm{G}}({\bm{\omega}})]\bigg{)}\,[\underline{{\mathbb{L}}}]\quad,\quad\forall\,{\bm{\omega}}\in\Omega\,,$
(A.5)

in which $[\underline{{\mathbb{L}}}]$ is the upper triangular $(6\times 6)$
real matrix such that
$[\,\underline{{\bm{A}}}\,]=[\underline{{\mathbb{L}}}]^{T}[\underline{{\mathbb{L}}}]$,
where $\epsilon$ is a given positive number (which can be chosen arbitrarily
small), and where
$\\{[{\bm{G}}({\bm{\omega}})],{\bm{\omega}}\in{\mathbb{R}}^{3}\\}$ is a
${\mathbb{M}}^{+}_{6}$-valued random field (by construction), defined on
$(\Theta,{\mathcal{T}},{\mathcal{P}})$, indexed by ${\mathbb{R}}^{3}$. Then
$[{\bm{G}}]$ is homogeneous, mean-square continuous, and such that
$E\\{[{\bm{G}}({\bm{\omega}}))]\\}=[I_{6}]$ for all
${\bm{\omega}}\in{\mathbb{R}}^{3}$. Note that the lower bound
$\epsilon\,[\,\underline{{\bm{A}}}\,]/(1+\epsilon)$ used in Eq. (A.5) could be
replaced by a more general lower bound $[A_{b}]$ in ${\mathbb{M}}_{6}^{+}$ as
proposed in [75, 68]. For all ${\bm{\omega}}$ fixed in ${\mathbb{R}}^{3}$, the
${\mathbb{M}}^{+}_{6}$-valued random variable $[{\bm{G}}({\bm{\omega}})]$ has
been constructed by using the Maximum Entropy Principle under the following
available information, $E\\{[{\bm{G}}({\bm{\omega}})]\\}\\!=\\![I_{6}]$ and
$E\\{\log(\det[{\bm{G}}({\bm{\omega}})])\\}\\!=\\!b_{G}$ with
$|b_{G}|\,<\\!+\infty$, which has been introduced in order that the random
matrix $[{\bm{G}}({\bm{\omega}})]^{-1}$ (that exists almost surely) be such
that $E\\{\|[{\bm{G}}({\bm{\omega}})]^{-1}\|^{2}\\}\leq$
$E\\{\|[{\bm{G}}({\bm{\omega}})]^{-1}\|_{F}^{2}\\}<+\infty$. In this
construction, for all ${\bm{\omega}}$ fixed in ${\mathbb{R}}^{3}$,
$[{\bm{G}}({\bm{\omega}})]=[g\big{(}\,\\{{\mathcal{G}}_{mn}({\bm{\omega}}),1\\!\leq\\!m\\!\leq\\!n\\!\leq\\!6\\}\,\big{)}]$
is a ${\mathbb{M}}_{6}^{+}$-valued nonlinear function $[g(.)]$ of
$6\times(6+1)/2=21$ independent normalized Gaussian real-valued random
variables denoted by $\\{{\mathcal{G}}_{mn}({\bm{\omega}}),1\leq m\leq n\leq
6\\}$ and such that $E\\{{\mathcal{G}}_{mn}({\bm{\omega}})\\}=0$ and
$E\\{{\mathcal{G}}_{mn}({\bm{\omega}})^{2}\\}=1$. The spatial correlation
structure of random field $\\{[{\bm{G}}({\bm{\omega}})],$
${\bm{\omega}}\in{\mathbb{R}}^{3}\\}$ is introduced by considering $21$
independent real-valued random fields
$\\{{\mathcal{G}}_{mn}({\bm{\omega}}),{\bm{\omega}}\in{\mathbb{R}}^{3}\\}$ for
$1\leq m\leq n\leq 6$, corresponding to $21$ independent copies of a unique
normalized Gaussian homogeneous mean-square continuous real-valued random
field $\\{{\mathcal{G}}({\bm{\omega}}),{\bm{\omega}}\in{\mathbb{R}}^{3}\\}$
whose normalized spectral measure is given and has a support that is
controlled by three spatial correlation lengths $L_{c1}=L_{c2}=L_{c3}=0.4$.
Note that this Gaussian field ${\mathcal{G}}$ can be replaced by a non-
Gaussian field for taking into account uncertainties in the spectral measure
[78]. The constant $b_{G}$ is eliminated in favor of a hyperparameter
$\delta_{G}>0$, which allows for controlling the level of statistical
fluctuations of $[{\bm{G}}({\bm{\omega}})]$, defined by
$\delta_{G}=(E\\{\|[{\bm{G}}({\bm{\omega}})]-[I_{6}]\|_{F}^{2}\\}/6)^{1/2}$,
which is independent of ${\bm{\omega}}$ and such that $\delta_{G}=0.6$.

###  A.2 Stochastic computational model for generating the training set
$D_{d}$ and observed quantities of interest

The stochastic boundary value problem defined by Eqs. (A.1) to (A.4) is
discretized by the finite element method. Domain $\Omega$ is meshed with
$50\\!\times\\!10\\!\times\\!5=2\,500$ finite elements using $8$-nodes finite
elements. There are $3\,366$ nodes and $10\,098$ dofs (degrees of freedom).
The displacements are locked at all the $66$ nodes belonging to surface
$\Gamma_{0}$ and therefore, there are $198$ zero Dirichlet conditions. There
are $8$ integration points in each finite element. Consequently, there are
$N_{p}=20\,000$ integration points
${\bm{\omega}}^{1},\ldots,{\bm{\omega}}^{N_{p}}$. The
${{{\mathbb{R}}^{n_{w}}}}$-valued random variable ${\bm{W}}$ is generated as
follows. For all $p=1,\ldots,N_{p}$, let
$[{\bm{G}}_{p}^{\rm{log}}]=\log_{M}([{\bm{G}}({\bm{\omega}}^{p})])\in{\mathbb{M}}_{6}$
in which $\log_{M}$ is the logarithm of positive-definite matrices. The
${{{\mathbb{R}}^{n_{w}}}}$-valued random variable ${\bm{W}}$ is then defined
as the vector that is the reshaping of the upper triangular part of the
$N_{p}$ matrices $\\{\,[{\bm{G}}_{p}^{\rm{log}}],p=1,\ldots,N_{p}\\}$.We then
have $n_{w}=21\times N_{p}=420\,000$. The finite element discretization of
random field
$\\{{\mathbb{U}}({\bm{\omega}}),{\bm{\omega}}\in\overline{\Omega}\,\\}$ is the
${{{\mathbb{R}}^{n_{q}}}}$-valued random variable ${\bm{Q}}$ with
$n_{q}=10\,098$. Consequently ${\bm{X}}=({\bm{Q}},{\bm{W}})$ is a random
variable with values in ${{{\mathbb{R}}^{n_{x}}}}$ with
$n_{x}=n_{q}+n_{w}=430\,098$. The stochastic computational model is then
represented by a stochastic linear matrix equation that is solved by using the
Monte Carlo numerical simulation method yielding the training set
$D_{d}=\\{{\bm{x}}^{1},\ldots,{\bm{x}}^{N_{d}}\\}$ in which
${\bm{x}}^{j}\\!\\!=\\!({\bm{q}}_{d}^{j},{\bm{w}}_{d}^{j})\in{{{\mathbb{R}}^{n_{x}}}}\\!\\!=\\!{{{\mathbb{R}}^{n_{q}}}}\\!\times\\!{{{\mathbb{R}}^{n_{w}}}}$
is a realization of random variable ${\bm{X}}=({\bm{Q}},{\bm{W}})$, the
computed realizations being independent. For studying the convergence
properties, the considered values of $N_{d}$ are
$N_{d}\in\\{100,200,300,400\\}$.

The components of the quantity of interest ${\bm{Q}}$, which will be observed
for presenting the results, are the $3$ components denoted by
$Q_{{\hbox{{\pppcarac obs}}},1}$, $Q_{{\hbox{{\pppcarac obs}}},2}$, and
$Q_{{\hbox{{\pppcarac obs}}},3}$ that correspond to the $3$ dofs along
directions $\omega_{1}$, $\omega_{2}$, and $\omega_{3}$ of the finite element
node of coordinates $(0,0,0.1)$ (located at top corner in which the
displacements are significant and result from tension, torsion, and two
bendings contributions).

###  A.3 Target set of realizations

The target set $D_{\hbox{{\ppppcarac targ}}}=\\{{\bm{q}}_{\hbox{{\ppppcarac
targ}}}^{1},\ldots{\bm{q}}_{\hbox{{\ppppcarac targ}}}^{N_{r}}\\}$ is generated
using the stochastic boundary value problem defined in Section A.1 for which
the elasticity matrix $[\underline{{\bm{A}}}^{\hbox{{\ppppcarac targ}}}]$ is
the one of a homogeneous and isotropic elastic material with a Young modulus
$9\times 10^{9}\,N/m^{2}$ and a Poisson coefficient $\nu=0.15$. The level of
statistical fluctuations of the random field $\\{{\bm{G}}^{\hbox{{\ppppcarac
targ}}}({\bm{\omega}}),{\bm{\omega}}\in{\mathbb{R}}^{3}\\}$ is
$\delta_{G}^{\hbox{{\ppppcarac targ}}}=0.3$. In order to analyze the
convergence with respect to $N_{r}$, we have considered, in consistency with
the values of $N_{d}$, the intervals $N_{r}\in[50\,,N_{\hbox{{\ppppcarac
targ}}}]$ with $N_{\hbox{{\ppppcarac targ}}}\in\\{100,200,300,400\\}$.

## Conflict of interest

The author declares that he has no conflict of interest.

## References

  * [1] J. M. Bernardo, A. F. M. Smith, Bayesian Theory, John Wiley & Sons, Chichester, 2000.
  * [2] M. C. Kennedy, A. O’Hagan, Bayesian calibration of computer models, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63 (3) (2001) 425–464.  doi:10.1111/1467-9868.00294.
  * [3] J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, Vol. 65, John Wiley & Sons, 2005.
  * [4] P. Congdon, Bayesian Statistical Modelling, Vol. 704, John Wiley & Sons, 2007\.
  * [5] B. P. Carlin, T. A. Louis, Bayesian Methods for Data Analysis, Chapman and Hall/CRC, 2008.
  * [6] J. E. Gentle, Computational statistics, Springer, New York, 2019.  doi:10.1007/978-0-387-98144-4.
  * [7] J. Marin, P. Pudlo, C. Robert, R. Ryder, Approximate Bayesian computational methods, Statistics and Computing 22 (6) (2012) 1167–1180.  doi:10.1007/s11222-011-9288-2.
  * [8] G. Givens, J. Hoeting, Computational Statistics, 2nd Edition, John Wiley and Sons, Hoboken, New Jersey, 2013.
  * [9] S. L. Scott, A. W. Blocker, F. V. Bonassi, H. A. Chipman, E. I. George, R. E. McCulloch, Bayes and big data: The consensus Monte Carlo algorithm, International Journal of Management Science and Engineering Management 11 (2) (2016) 78–88.  doi:10.1080/17509653.2016.1142191.
  * [10] R. Ghanem, D. Higdon, H. Owhadi, Handbook of Uncertainty Quantification, Vol. 1 to 3, Springer, Cham, Switzerland, 2017.  doi:10.1007/978-3-319-12385-1.
  * [11] J. Kaipio, E. Somersalo, Statistical and Computational Inverse Problems, Vol. 160, Springer Science & Business Media, 2005.  doi:10.1007/b138659.
  * [12] Y. M. Marzouk, H. N. Najm, L. A. Rahn, Stochastic spectral methods for efficient Bayesian solution of inverse problems, Journal of Computational Physics 224 (2) (2007) 560–586.  doi:10.1016/j.jcp.2006.10.010.
  * [13] A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numerica 19 (2010) 451–559.  doi:10.1017/S0962492910000061.
  * [14] C. Soize, A computational inverse method for identification of non-Gaussian random fields using the Bayesian approach in very high dimension, Computer Methods in Applied Mechanics and Engineering 200 (45-46) (2011) 3083–3099.  doi:10.1016/j.cma.2011.07.005.
  * [15] H. G. Matthies, E. Zander, B. V. Rosić, A. Litvinenko, O. Pajonk, Inverse problems in a Bayesian setting, in: Computational Methods for Solids and Fluids, Vol. 41, Springer, 2016, pp. 245–286.  doi:10.1007/978-3-319-27996-1\\_10.
  * [16] I. Bilionis, N. Zabaras, Bayesian uncertainty propagation using Gaussian processes, in: R. Ghanem, D. Higdon, O. Houman (Eds.), Handbook of Uncertainty Quantification, Springer, Cham, Switzerland, 2017, Ch. 15, pp. 555–600.
  * [17] M. Dashti, A. M. Stuart, The Bayesian approach to inverse problems, in: R. Ghanem, D. Higdon, O. Houman (Eds.), Handbook of Uncertainty Quantification, Springer, Cham, Switzerland, 2017, Ch. 10, pp. 311–428.  doi:10.1007/978-3-319-12385-1\\_7.
  * [18] M. Arnst, B. Abello Álvarez, J.-P. Ponthot, R. Boman, Itô-SDE MCMC method for Bayesian characterization of errors associated with data limitations in stochastic expansion methods for uncertainty quantification, Journal of Computational Physics 349 (2017) 59–79.  doi:10.1016/j.jcp.2017.08.005.
  * [19] A. Spantini, T. Cui, K. Willcox, L. Tenorio, Y. Marzouk, Goal-oriented optimal approximations of Bayesian linear inverse problems, SIAM Journal on Scientific Computing 39 (5) (2017) S167–S196.  doi:10.1137/16M1082123.
  * [20] U. Picchini, A. Samson, Coupling stochastic em and approximate Bayesian computation for parameter inference in state-space models, Computational Statistics 33 (1) (2018) 179–212.  doi:10.1007/s00180-017-0770-y.
  * [21] G. Perrin, C. Soize, Adaptive method for indirect identification of the statistical properties of random fields in a Bayesian framework, Computational Statistics 35 (1) (2020) 111–133.  doi:10.1007/s00180-019-00936-5.
  * [22] Y. Shen, D. Cornford, M. Opper, C. Archambeau, Variational markov chain Monte Carlo for Bayesian smoothing of non-linear diffusions, Computational Statistics 27 (1) (2012) 149–176.  doi:10.1007/s00180-011-0246-4.
  * [23] N. Depraetere, M. Vandebroek, A comparison of variational approximations for fast inference in mixed logit models, Computational Statistics 32 (1) (2017) 93–125.  doi:10.1007/s00180-015-0638-y.
  * [24] A. Golightly, D. J. Wilkinson, Bayesian sequential inference for nonlinear multivariate diffusions, Statistics and Computing 16 (4) (2006) 323–338.  doi:10.1007/s11222-006-9392-x.
  * [25] P. Fearnhead, Exact and efficient Bayesian inference for multiple changepoint problems, Statistics and Computing 16 (2) (2006) 203–213.  doi:10.1007/s11222-006-8450-8.
  * [26] M. Neil, M. Tailor, D. Marquez, Inference in hybrid Bayesian networks using dynamic discretization, Statistics and Computing 17 (3) (2007) 219–233.  doi:10.1007/s11222-007-9018-y.
  * [27] R. Sambasivan, S. Das, S. K. Sahu, A Bayesian perspective of statistical machine learning for big data, Computational Statistics 35 (3) (2020) 893–930.  doi:10.1007/s00180-020-00970-8.
  * [28] C. Soize, R. Ghanem, C. Desceliers, Sampling of Bayesian posteriors with a non-Gaussian probabilistic learning on manifolds from a small dataset, Statistics and Computing 30 (5) (2020) 1433–1457.  doi:10.1007/s11222-020-09954-6.
  * [29] H. Owhadi, C. Scovel, T. Sullivan, On the brittleness of Bayesian inference, SIAM Review 57 (4) (2015) 566–582.  doi:10.1137/130938633.
  * [30] S. Kullback, R. A. Leibler, On information and sufficiency, The Annals of Mathematical Statistics 22 (1) (1951) 79–86.  doi:10.1214/aoms/1177729694.
  * [31] J. N. Kapur, H. K. Kesavan, Entropy Optimization Principles with Applications, Academic Press, San Diego, 1992.
  * [32] T. M. Cover, J. A. Thomas, Elements of Information Theory, Second Edition, John Wiley & Sons, Hoboken, 2006.
  * [33] N. Vasconcelos, P. Ho, P. Moreno, The Kullback-Leibler kernel as a framework for discriminant and localized representations for visual recognition, Proceedings of the European Conference on Computer Vision (2004) 430–441doi:10.1007/978-3-540-24672-5\\_34.
  * [34] W. Zhang, S. Shan, X. Chen, W. Gao, Local Gabor binary patterns based on Kullback-Leibler divergence for partially occluded face recognition, IEEE Signal Processing Letters 14 (11) (2007) 875–878.  doi:10.1109/LSP.2007.903260.
  * [35] O. Cappé, A. Garivier, O.-A. Maillard, R. Munos, G. Stoltz, et al., Kullback-Leibler upper confidence bounds for optimal sequential allocation, The Annals of Statistics 41 (3) (2013) 1516–1541.  doi:10.1214/13.AOS1119.
  * [36] N. Saleem, G. Ijaz, Low rank sparse decomposition model based speech enhancement using gammatone filterbank and Kullback-Leibler divergence, International Journal of Speech Technology 21 (2) (2018) 217–231.  doi:10.1007/s10772-018-9500-2.
  * [37] S. Filippi, O. Cappé, A. Garivier, Optimism in reinforcement learning and Kullback-Leibler divergence, Proceedings of the 48th Annual Allerton IEEE Conference on Communication, Control, and Computing (2010) 115–122doi:10.1109/ALLERTON.2010.5706896.
  * [38] C. Soize, R. Ghanem, Physics-constrained non-Gaussian probabilistic learning on manifolds, International Journal for Numerical Methods in Engineering 121 (1) (2020) 110–145.  doi:10.1002/nme.6202.
  * [39] C. Soize, R. Ghanem, Probabilistic learning on manifolds constrained by nonlinear partial differential equations for small datasets, Computer Methods in Applied Mechanics and Engineering 380 (2021) 113777.  doi:10.1016/j.cma.2021.113777.
  * [40] C. Soize, R. Ghanem, Data-driven probability concentration and sampling on manifold, Journal of Computational Physics 321 (2016) 242–258.  doi:10.1016/j.jcp.2016.05.044.
  * [41] G. Perrin, C. Soize, N. Ouhbi, Data-driven kernel representations for sampling with an unknown block dependence structure under correlation constraints, Computational Statistics & Data Analysis 119 (2018) 139–154.  doi:10.1016/j.csda.2017.10.005.
  * [42] C. Farhat, R. Tezaur, T. Chapman, P. Avery, C. Soize, Feasible probabilistic learning method for model-form uncertainty quantification in vibration analysis, AIAA Journal 57 (11) (2019) 4978–4991.  doi:10.2514/1.J057797.
  * [43] R. Ghanem, C. Soize, L. Mehrez, V. Aitharaju, Probabilistic learning and updating of a digital twin for composite material systems, International Journal for Numerical Methods in Engineering (2020).  doi:10.1002/nme.6430.
  * [44] J. Guilleminot, J. E. Dolbow, Data-driven enhancement of fracture paths in random composites, Mechanics Research Communications 103 (2020) 103443.  doi:10.1016/j.mechrescom.2019.103443.
  * [45] C. Soize, R. Ghanem, Probabilistic learning on manifolds, Foundations of Data Science 2 (3) (2020) 279–307.  doi:10.3934/fods.2020013.
  * [46] M. Arnst, C. Soize, K. Bulthies, Computation of sobol indices in global sensitivity analysis from small data sets by probabilistic learning on manifolds, International Journal for Uncertainty Quantification 11 (2) (2021) 1–23.  doi:10.1615/Int.J.UncertaintyQuantification.2020032674.
  * [47] C. Soize, R. Ghanem, Probabilistic learning on manifolds (plom) with partition, International Journal for Numerical Methods in Engineering 123 (1) (2022) 268–290.  doi:10.1002/nme.6856.
  * [48] R. Ghanem, C. Soize, Probabilistic nonconvex constrained optimization with fixed number of function evaluations, International Journal for Numerical Methods in Engineering 113 (4) (2018) 719–741.  doi:10.1002/nme.5632.
  * [49] R. Ghanem, C. Soize, C. Safta, X. Huan, G. Lacaze, J. C. Oefelein, H. N. Najm, Design optimization of a scramjet under uncertainty using probabilistic learning on manifolds, Journal of Computational Physics 399 (2019) 108930.  doi:10.1016/j.jcp.2019.108930.
  * [50] E. Capiez-Lernout, C. Soize, Nonlinear stochastic dynamics of detuned bladed disks with uncertain mistuning and detuning optimization using a probabilistic machine learning tool, International Journal of Non-Linear Mechanics 143 (2022) 104023.  doi:10.1016/j.ijnonlinmec.2022.104023.
  * [51] C. Soize, Probabilistic learning inference of boundary value problem with uncertainties based on Kullback-Leibler divergence under implicit constraints, Computer Methods in Applied Mechanics and Engineering Available online (2022) 1–35.  doi:10.1016/j.cma.2022.115078.
  * [52] C. Soize, The Fokker-Planck Equation for Stochastic Dynamical Systems and its Explicit Steady State Solutions, Vol. Series on Advances in Mathematics for Applied Sciences: Vol 17, World Scientific, Singapore, 1994.  doi:10.1142/2347.
  * [53] G. H. Golub, C. F. Van Loan, Matrix Computations, Second Edition, Johns Hopkins University Press, Baltimore and London, 1993.
  * [54] I. M. Gelfand, N. I. Vilenkin, Generalized Functions. Volume 4. Applications of Harmonic Analysis, Vol. 380, AMS Chelsea Publishing, 1964.
  * [55] J. Dieudonné, Treatise on Analysis, Vol. 6, Academic Press, New York, 1978.
  * [56] C. Soize, Mathematical Methods in Signal Analysis (in French, Méthodes Mathématiques en Analyse du Signal), Masson, Paris, 1993.
  * [57] A. Bowman, A. Azzalini, Applied Smoothing Techniques for Data Analysis: The Kernel Approach With S-Plus Illustrations, Vol. 18, Oxford University Press, Oxford: Clarendon Press, New York, 1997.  doi:10.1007/s001800000033.
  * [58] P. Krée, C. Soize, Mathematics of Random Phenomena, Reidel Pub. Co, 1986, (first published by Bordas in 1983 and also published by Springer Science & Business Media in 2012).
  * [59] C. Soize, Polynomial chaos expansion of a multimodal random vector, SIAM-ASA Journal on Uncertainty Quantification 3 (1) (2015) 34–60.  doi:10.1137/140968495.
  * [60] C. Soize, Construction of probability distributions in high dimension using the maximum entropy principle. applications to stochastic processes, random fields and random matrices, International Journal for Numerical Methods in Engineering 76 (10) (2008) 1583–1611.  doi:10.1002/nme.2385.
  * [61] R. Neal, MCMC using hamiltonian dynamics, in: S. Brooks, A. Gelman, G. Jones, X.-L. Meng (Eds.), Handbook of Markov Chain Monte Carlo, Chapman and Hall-CRC Press, Boca Raton, 2011, Ch. 5, pp. 1–51.  doi:10.1201/b10905-6.
  * [62] M. Girolami, B. Calderhead, Riemann manifold Langevin and Hamiltonian Monte Carlo methods, Journal of the Royal Statistics Society 73 (2) (2011) 123–214.  doi:10.1111/j.1467-9868.2010.00765.x.
  * [63] C. Robert, G. Casella, Monte Carlo Statistical Methods, Springer Science & Business Media, 2005.  doi:10.1007/978-1-4757-4145-2.
  * [64] D. G. Luenberger, Optimization by Vector Space Methods, John Wiley and Sons, New York, 2009.
  * [65] N. Agmon, Y. Alhassid, R. D. Levine, An algorithm for finding the distribution of maximal entropy, Journal of Computational Physics 30 (2) (1979) 250–258.  doi:10.1016/0021-9991(79)90102-5.
  * [66] C. T. Kelley, Solving Nonlinear Equations With Newton’s Method, SIAM, 2003.  doi:10.1137/1.9780898718898.
  * [67] A. Batou, C. Soize, Calculation of Lagrange multipliers in the construction of maximum entropy distributions in high stochastic dimension, SIAM/ASA Journal on Uncertainty Quantification 1 (1) (2013) 431–451.  doi:10.1137/120901386.
  * [68] C. Soize, Uncertainty Quantification. An Accelerated Course with Advanced Applications in Computational Engineering, Springer, New York, 2017.  doi:10.1007/978-3-319-54339-0.
  * [69] P. Kloeden, E. Platen, Numerical Solution of Stochastic Differentials Equations, Springer-Verlag, Heidelberg, 1992.
  * [70] D. Talay, L. Tubaro, Expansion of the global error for numerical schemes solving stochastic differential equations, Stochastic analysis and applications 8 (4) (1990) 483–509.  doi:10.1080/07362999008809220.
  * [71] D. Talay, Stochastic Hamiltonian systems: exponential convergence to the invariant measure, and discretization by the implicit Euler scheme, Markov Process and Related Fields 8 (2) (2002) 163–198.
  * [72] E. Hairer, C. Lubich, G. Wanner, Geometric numerical integration illustrated by the Störmer-Verlet method, Acta Numerica 12 (2003) 399–450.  doi:10.1017/S0962492902000144.
  * [73] K. Burrage, I. Lenane, G. Lythe, Numerical methods for second-order stochastic differential equations, SIAM Journal on Scientific Computing 29 (1) (2007) 245–264.  doi:10.1137/050646032.
  * [74] C. Soize, I. E. Poloskov, Time-domain formulation in computational dynamics for linear viscoelastic media with model uncertainties and stochastic excitation, Computers & Mathematics with Applications 64 (11) (2012) 3594–3612.  doi:10.1016/j.camwa.2012.09.010.
  * [75] J. Guilleminot, C. Soize, Stochastic model and generator for random fields with symmetry properties: application to the mesoscopic modeling of elastic random media, Multiscale Modeling & Simulation (A SIAM Interdisciplinary Journal) 11 (3) (2013) 840–870.  doi:10.1137/120898346.
  * [76] C. Soize, Non Gaussian positive-definite matrix-valued random fields for elliptic stochastic partial differential operators, Computer Methods in Applied Mechanics and Engineering 195 (1-3) (2006) 26–64.  doi:10.1016/j.cma.2004.12.014.
  * [77] C. Soize, Tensor-valued random fields for meso-scale stochastic model of anisotropic elastic microstructure and probabilistic analysis of representative volume element size, Probabilistic Engineering Mechanics 23 (2-3) (2008) 307–323.  doi:10.1016/j.probengmech.2007.12.019.
  * [78] C. Soize, Stochastic elliptic operators defined by non-Gaussian random fields with uncertain spectrum, The American Mathematical Society Journal Theory of Probability and Mathematical Statistics 105 (2021) 113–136.  doi:10.1090/tpms/1159.