* de Pater . (2016) 16depaterde Pater, I., Sault, RJ., Butler, B., DeBoer, D. Wong, MH.  201606\.  Peering through Jupiter’s clouds with radio spectral imaging Peering through Jupiter’s clouds with radio spectral imaging.  Science3521198-1201.  10.1126/science.aaf2210
  * Del Genio . (2009) 09delgenioDel Genio, AD., Achterberg, RK., Baines, KH., Flasar, FM., Read, PL., Sánchez-Lavega, A. Showman, AP.  2009\.  Saturn Atmospheric Structure and Dynamics, In: Saturn from Cassini-Huygens Saturn Atmospheric Structure and Dynamics, In: Saturn from Cassini-Huygens.  Dougherty, M. K., Esposito, L. W., & Krimigis, S. M. (), Saturn from Cassini-Huygens Saturn from cassini-huygens ( 113-159).  Springer.  10.1007/978-1-4020-9217-6_6
  * del Río-Gaztelurrutia . (2018) 18gaztelurrutiadel Río-Gaztelurrutia, T., Sánchez-Lavega, A., Antuñano, A., Legarreta, J., García-Melendo, E., Sayanagi, KM.Barry, T.  201803\.  A planetary-scale disturbance in a long living three vortex coupled system in Saturn’s atmosphere A planetary-scale disturbance in a long living three vortex coupled system in Saturn’s atmosphere.  Icarus302499-513.  10.1016/j.icarus.2017.11.029
  * Delahaye . (2021) 20geisaDelahaye, T., Armante, R., Scott, NA., Jacquinet-Husson, N., Chédin, A., Crépeau, L.Yurchenko, S.  202107\.  The 2020 edition of the GEISA spectroscopic database The 2020 edition of the GEISA spectroscopic database.  Journal of Molecular Spectroscopy380111510.  10.1016/j.jms.2021.111510
  * Encrenaz (2003) 03encrenazEncrenaz, T.  200302\.  ISO observations of the giant planets and Titan: what have we learnt? ISO observations of the giant planets and Titan: what have we learnt?  Plan. & Space Sci.51289-103.  10.1016/S0032-0633(02)00145-9
  * Ferris Benson (1980) 80ferrisFerris, JP. Benson, R.  198005\.  Diphosphine is an intermediate in the photolysis of phosphine to phosphorus and hydrogen Diphosphine is an intermediate in the photolysis of phosphine to phosphorus and hydrogen.  Nature285156.  10.1038/285156a0
  * Feuchtgruber . (1997) 97feuchtgruberFeuchtgruber, H., Lellouch, E., de Graauw, T., Bezard, B., Encrenaz, T. Griffin, M.  199709\.  External supply of oxygen to the atmospheres of giant planets External supply of oxygen to the atmospheres of giant planets.  Nature389159-162.
  * L. Fletcher . (2021) 21fletcher_epscFletcher, L., Hammel, H., Milam, S., de Pater, I., Fouchet, T., Orton, G.Glasse, A.  202109\.  The JWST Giant Planet Atmospheres Programme The JWST Giant Planet Atmospheres Programme.  European Planetary Science Congress European planetary science congress ( EPSC2021-39).  10.5194/epsc2021-39
  * LN. Fletcher (2023) 23fletcher_dataFletcher, LN.  2023\.  Supporting Data for JWST Saturn Atmosphere [Data]. Supporting Data for JWST Saturn Atmosphere [Data].  https://github.com/JWSTGiantPlanets/saturn-atmosphere-miri 10.5281/zenodo.7891589
  * LN. Fletcher . (2011) 11fletcher_vimsFletcher, LN., Baines, KH., Momary, TW., Showman, AP., Irwin, PGJ., Orton, GS.Merlet, C.  201108\.  Saturn’s tropospheric composition and clouds from Cassini/VIMS 4.6-5.1 $\mu$m nightside spectroscopy Saturn’s tropospheric composition and clouds from Cassini/VIMS 4.6-5.1 $\mu$m nightside spectroscopy.  Icarus214510-533.  10.1016/j.icarus.2011.06.006
  * LN. Fletcher . (2017) 17fletcher_QPOFletcher, LN., Guerlet, S., Orton, GS., Cosentino, RG., Fouchet, T., Irwin, PGJ.Morales-Juberías, R.  201711\.  Disruption of Saturn’s quasi-periodic equatorial oscillation by the Great Northern Storm Disruption of Saturn’s quasi-periodic equatorial oscillation by the Great Northern Storm.  Nature Astronomy1765-770.  10.1038/s41550-017-0271-5
  * LN. Fletcher, Gustafsson Orton (2018) 18fletcher_ciaFletcher, LN., Gustafsson, M. Orton, GS.  201803\.  Hydrogen Dimers in Giant-planet Infrared Spectra Hydrogen Dimers in Giant-planet Infrared Spectra.  Astrophys. J. Supplement23524.  10.3847/1538-4365/aaa07a
  * LN. Fletcher . (2016) 16fletcherFletcher, LN., Irwin, PGJ., Achterberg, RK., Orton, GS. Flasar, FM.  201601\.  Seasonal variability of Saturn’s tropospheric temperatures, winds and para-H2 from Cassini far-IR spectroscopy Seasonal variability of Saturn’s tropospheric temperatures, winds and para-H2 from Cassini far-IR spectroscopy.  Icarus264137-159.  10.1016/j.icarus.2015.09.009
  * LN. Fletcher . (2008) 08fletcher_polesFletcher, LN., Irwin, PGJ., Orton, GS., Teanby, NA., Achterberg, RK., Bjoraker, GL.Flasar, FM.  200801\.  Temperature and Composition of Saturn’s Polar Hot Spots and Hexagon Temperature and Composition of Saturn’s Polar Hot Spots and Hexagon.  Science31979-82.  10.1126/science.1149514
  * LN. Fletcher . (2007) 07fletcher_tempFletcher, LN., Irwin, PGJ., Teanby, NA., Orton, GS., Parrish, PD., Calcutt, SB.Taylor, FW.  2007\.  Characterising Saturn’s Vertical Temperature Structure from Cassini/CIRS Characterising Saturn’s Vertical Temperature Structure from Cassini/CIRS.  Icarus189457-478.
  * LN. Fletcher, Kaspi . (2020) 20fletcher_beltzoneFletcher, LN., Kaspi, Y., Guillot, T. Showman, AP.  2020\.  How Well Do We Understand the Belt/Zone Circulation of Giant Planet Atmospheres? How Well Do We Understand the Belt/Zone Circulation of Giant Planet Atmospheres?  Space Science Reviews216230.  10.1007/s11214-019-0631-9
  * LN. Fletcher, Orton . (2018) 18fletcher_polesFletcher, LN., Orton, GS., Sinclair, JA., Guerlet, S., Read, PL., Antuñano, A.Calcutt, SB.  201809\.  A hexagon in Saturn’s northern stratosphere surrounding the emerging summertime polar vortex A hexagon in Saturn’s northern stratosphere surrounding the emerging summertime polar vortex.  Nature Communications93564.  10.1038/s41467-018-06017-3
  * LN. Fletcher, Orton, Teanby Irwin (2009) 09fletcher_ph3Fletcher, LN., Orton, GS., Teanby, NA. Irwin, PGJ.  200908\.  Phosphine on Jupiter and Saturn from Cassini/CIRS Phosphine on Jupiter and Saturn from Cassini/CIRS.  Icarus202543-564.  10.1016/j.icarus.2009.03.023
  * LN. Fletcher, Orton, Teanby, Irwin Bjoraker (2009) 09fletcher_ch4Fletcher, LN., Orton, GS., Teanby, NA., Irwin, PGJ. Bjoraker, GL.  200902\.  Methane and its isotopologues on Saturn from Cassini/CIRS observations Methane and its isotopologues on Saturn from Cassini/CIRS observations.  Icarus199351-367.  10.1016/j.icarus.2008.09.019
  * LN. Fletcher, Sromovsky . (2020) 20fletcher_saturnFletcher, LN., Sromovsky, L., Hue, V., Moses, JI., Guerlet, S., West, RA. Koskinen, T.  202012\.  Saturn’s Seasonal Atmosphere at Northern Summer Solstice Saturn’s Seasonal Atmosphere at Northern Summer Solstice.  arXiv e-printsarXiv:2012.09288.
  * LN. Fletcher . (2012) 12fletcher_spireFletcher, LN., Swinyard, B., Salji, C., Polehampton, E., Fulton, T., Sidher, S.Lim, T.  201203\.  Sub-millimetre spectroscopy of Saturn’s trace gases from Herschel/SPIRE Sub-millimetre spectroscopy of Saturn’s trace gases from Herschel/SPIRE.  Astron. Astrophys.539A44.  10.1051/0004-6361/201118415
  * Fouchet . (2008) 08fouchetFouchet, T., Guerlet, S., Strobel, D., Simon-Miller, A., Bézard, B. Flasar, F.  2008\.  An equatorial oscillation in Saturn’s middle atmosphere. An equatorial oscillation in Saturn’s middle atmosphere.  Nature4537192200–202.
  * Friedson Moses (2012) 12friedsonFriedson, AJ. Moses, JI.  201204\.  General circulation and transport in Saturn’s upper troposphere and stratosphere General circulation and transport in Saturn’s upper troposphere and stratosphere.  Icarus218861-875.  10.1016/j.icarus.2012.02.004
  * García-Melendo . (2011) 11garciaGarcía-Melendo, E., Pérez-Hoyos, S., Sánchez-Lavega, A. Hueso, R.  201109\.  Saturn’s zonal wind profile in 2004-2009 from Cassini ISS images and its long-term variability Saturn’s zonal wind profile in 2004-2009 from Cassini ISS images and its long-term variability.  Icarus21562-74.  10.1016/j.icarus.2011.07.005
  * Gardner . (2023) 23gardnerGardner, JP., Mather, JC., Abbott, R., Abell, JS., Abernathy, M., Abney, FE.Zondag, E.  202304\.  The James Webb Space Telescope Mission The James Webb Space Telescope Mission.  arXiv e-printsarXiv:2304.04869.  10.48550/arXiv.2304.04869
  * Giles . (2015) 15gilesGiles, RS., Fletcher, LN. Irwin, PGJ.  201509\.  Cloud structure and composition of Jupiter’s troposphere from 5- $\mu$ m Cassini VIMS spectroscopy Cloud structure and composition of Jupiter’s troposphere from 5- $\mu$ m Cassini VIMS spectroscopy.  Icarus257457-470.  10.1016/j.icarus.2015.05.030
  * Gordon . (2022) 22gordonGordon, IE., Rothman, LS., Hargreaves, RJ., Hashemi, R., Karlovets, EV., Skinner, FM.Yurchenko, SN.  202201\.  The HITRAN2020 molecular spectroscopic database The HITRAN2020 molecular spectroscopic database.  Journal of Quantitative Spectroscopy and Radiative Transfer277107949.  10.1016/j.jqsrt.2021.107949
  * Guerlet . (2010) 10guerletGuerlet, S., Fouchet, T., Bézard, B., Moses, JI., Fletcher, LN., Simon-Miller, AA. Michael Flasar, F.  201010\.  Meridional distribution of CH3C2H and C4H2 in Saturn’s stratosphere from CIRS/Cassini limb and nadir observations Meridional distribution of CH3C2H and C4H2 in Saturn’s stratosphere from CIRS/Cassini limb and nadir observations.  Icarus209682-695.  10.1016/j.icarus.2010.03.033
  * Guerlet . (2009) 09guerletGuerlet, S., Fouchet, T., Bézard, B., Simon-Miller, AA. Flasar, FM.  2009\.  Vertical and meridional distribution of ethane, acetylene and propane in Saturn’s stratosphere from CIRS/Cassini limb observations Vertical and meridional distribution of ethane, acetylene and propane in Saturn’s stratosphere from CIRS/Cassini limb observations.  Icarus203214-232.
  * Guerlet . (2015) 15guerletGuerlet, S., Fouchet, T., Vinatier, S., Simon, AA., Dartois, E. Spiga, A.  201508\.  Stratospheric benzene and hydrocarbon aerosols detected in Saturn’s auroral regions Stratospheric benzene and hydrocarbon aerosols detected in Saturn’s auroral regions.  A&A580A89.  10.1051/0004-6361/201424745
  * Guerlet . (2014) 14guerletGuerlet, S., Spiga, A., Sylvestre, M., Indurain, M., Fouchet, T., Leconte, J.Forget, F.  201408\.  Global climate modeling of Saturn’s atmosphere. Part I: Evaluation of the radiative transfer model Global climate modeling of Saturn’s atmosphere. Part I: Evaluation of the radiative transfer model.  Icarus238110-124.  10.1016/j.icarus.2014.05.010
  * Hesman . (2012) 12hesmanHesman, BE., Bjoraker, GL., Sada, PV., Achterberg, RK., Jennings, DE., Romani, PN.Irwin, PGJ.  201211\.  Elusive Ethylene Detected in Saturn’s Northern Storm Region Elusive Ethylene Detected in Saturn’s Northern Storm Region.  Astrophys. J.760124.  10.1088/0004-637X/760/1/24
  * Holton (2004) 04holtonHolton, J.  2004\.  An Introduction to Dynamic Meteorology An Introduction to Dynamic Meteorology.  Academic press.
  * Howett . (2007) 07howett_nh4shHowett, CJA., Carlson, RW., Irwin, PGJ. Calcutt, SB.  200701\.  Optical constants of ammonium hydrosulfide ice and ammonia ice Optical constants of ammonium hydrosulfide ice and ammonia ice.  Journal of the Optical Society of America B Optical Physics24126-136.  10.1364/JOSAB.24.000126
  * Hue . (2015) 15hueHue, V., Cavalié, T., Dobrijevic, M., Hersant, F. Greathouse, TK.  201509\.  2D photochemical modeling of Saturn’s stratosphere. Part I: Seasonal variation of atmospheric composition without meridional transport 2D photochemical modeling of Saturn’s stratosphere. Part I: Seasonal variation of atmospheric composition without meridional transport.  Icarus257163-184.  10.1016/j.icarus.2015.04.001
  * Hue . (2016) 16hueHue, V., Greathouse, TK., Cavalié, T., Dobrijevic, M. Hersant, F.  201603\.  2D photochemical modeling of Saturn’s stratosphere. Part II: Feedback between composition and temperature 2D photochemical modeling of Saturn’s stratosphere. Part II: Feedback between composition and temperature.  Icarus267334-343.  10.1016/j.icarus.2015.12.007
  * Hueso . (2018) 18huesoHueso, R., Juaristi, J., Legarreta, J., Sánchez-Lavega, A., Rojas, JF., Erard, S.Le Sidaner, P.  2018Jan.  The Planetary Virtual Observatory and Laboratory (PVOL) and its integration into the Virtual European Solar and Planetary Access (VESPA) The Planetary Virtual Observatory and Laboratory (PVOL) and its integration into the Virtual European Solar and Planetary Access (VESPA).  Plan. & Space Sci.15022-35.  10.1016/j.pss.2017.03.014
  * Hurley . (2012) 12hurleyHurley, J., Fletcher, LN., Irwin, PGJ., Calcutt, SB., Sinclair, JA. Merlet, C.  201212\.  Latitudinal variation of upper tropospheric NH3 on Saturn derived from Cassini/CIRS far-infrared measurements Latitudinal variation of upper tropospheric NH3 on Saturn derived from Cassini/CIRS far-infrared measurements.  Plan. & Space Sci.73347-363.  10.1016/j.pss.2012.08.003
  * Irwin . (2008) 08irwinIrwin, P., Teanby, N., de Kok, R., Fletcher, L., Howett, C., Tsang, C.Parrish, P.  2008\.  The NEMESIS planetary atmosphere radiative transfer and retrieval tool The NEMESIS planetary atmosphere radiative transfer and retrieval tool.  Journal of Quantitative Spectroscopy and Radiative Transfer10961136–1150.
  * Irwin (2022) 22irwin_nemesisIrwin, PGJ.  2022\.  NEMESIS Retrieval Code [Software]. NEMESIS Retrieval Code [Software].  https://github.com/nemesiscode/radtrancode 10.5281/zenodo.5816724
  * Irwin . (2022) 22irwinIrwin, PGJ., Teanby, NA., Fletcher, LN., Toledo, D., Orton, GS., Wong, MH.Dobinson, J.  202206\.  Hazy Blue Worlds: A Holistic Aerosol Model for Uranus and Neptune, Including Dark Spots Hazy Blue Worlds: A Holistic Aerosol Model for Uranus and Neptune, Including Dark Spots.  Journal of Geophysical Research (Planets)1276e07189.  10.1029/2022JE007189
  * Jones . (2023) 23jonesJones, OC., Álvarez-Márquez, J., Sloan, GC., Kavanagh, PJ., Argyriou, I., Labiano, A.García Marín, M.  202301\.  Observations of the Planetary Nebula SMP LMC 058 with the JWST MIRI Medium Resolution Spectrometer Observations of the Planetary Nebula SMP LMC 058 with the JWST MIRI Medium Resolution Spectrometer.  submitted to MNRASarXiv:2301.13233.  10.48550/arXiv.2301.13233
  * Karkoschka Tomasko (1993) 93karkoschkaKarkoschka, E. Tomasko, MG.  1993\.  Saturn’s upper atmospheric hazes observed by the Hubble Space Telescope Saturn’s upper atmospheric hazes observed by the Hubble Space Telescope.  Icarus106428-441.  10.1006/icar.1993.1183
  * King . (2023) 23king_pipelineKing, OR., Harkett, J., Fletcher, LN. Melin, H.  2023\.  Custom JWST/MIRI Processing Pipeline [Software]. Custom JWST/MIRI Processing Pipeline [Software].  https://github.com/JWSTGiantPlanets/pipelines/ 10.5281/zenodo.7891560
  * Koskinen . (2016) 16koskinenKoskinen, TT., Moses, JI., West, RA., Guerlet, S. Jouchoux, A.  201608\.  The detection of benzene in Saturn’s upper atmosphere The detection of benzene in Saturn’s upper atmosphere.  Geophys. Res. Lett.437895-7901.  10.1002/2016GL070000
  * Labiano . (2021) 21labianoLabiano, A., Argyriou, I., Álvarez-Márquez, J., Glasse, A., Glauser, A., Patapis, P.Vandenbussche, B.  202112\.  Wavelength calibration and resolving power of the JWST MIRI Medium Resolution Spectrometer Wavelength calibration and resolving power of the JWST MIRI Medium Resolution Spectrometer.  Astron. Astrophys.656A57.  10.1051/0004-6361/202140614
  * Laraia . (2013) 13laraiaLaraia, AL., Ingersoll, AP., Janssen, MA., Gulkis, S., Oyafuso, F. Allison, M.  201309\.  Analysis of Saturn’s thermal emission at 2.2-cm wavelength: Spatial distribution of ammonia vapor Analysis of Saturn’s thermal emission at 2.2-cm wavelength: Spatial distribution of ammonia vapor.  Icarus226641-654.  10.1016/j.icarus.2013.06.017
  * Li . (2017) 17liLi, C., Ingersoll, A., Janssen, M., Levin, S., Bolton, S., Adumitroaie, V.Williamson, R.  201706\.  The distribution of ammonia on Jupiter from a preliminary inversion of Juno microwave radiometer data The distribution of ammonia on Jupiter from a preliminary inversion of Juno microwave radiometer data.  Geophys. Res. Lett.445317-5325.  10.1002/2017GL073159
  * Martonchik . (1984) 84martonchikMartonchik, JV., Orton, GS. Appleby, JF.  198402\.  Optical properties of NH3 ice from the far infrared to the near ultraviolet Optical properties of NH3 ice from the far infrared to the near ultraviolet.  Applied Optics23541-547.
  * MAST Archive (2022) MAST1247_MIRIMAST Archive.  2022\.  JWST MIRI/MRS Saturn Observations 1247 [Data]. JWST MIRI/MRS Saturn Observations 1247 [Data].  http://dx.doi.org/10.17909/wjpz-7383 10.17909/wjpz-7383
  * Moses . (2015) 15mosesMoses, JI., Armstrong, ES., Fletcher, LN., Friedson, AJ., Irwin, PGJ., Sinclair, JA. Hesman, BE.  201511\.  Evolution of stratospheric chemistry in the Saturn storm beacon region Evolution of stratospheric chemistry in the Saturn storm beacon region.  Icarus261149-168.  10.1016/j.icarus.2015.08.012
  * Moses . (2000) 00mosesMoses, JI., Bézard, B., Lellouch, E., Gladstone, GR., Feuchtgruber, H. Allen, M.  2000\.  Photochemistry of Saturn’s Atmosphere. I. Hydrocarbon Chemistry and Comparisons with ISO Observations Photochemistry of Saturn’s Atmosphere. I. Hydrocarbon Chemistry and Comparisons with ISO Observations.  Icarus143244-298.  10.1006/icar.1999.6270
  * Moses . (2023) 23mosesMoses, JI., Brown, ZL., Koskinen, TT., Fletcher, LN., Serigano, J., Guerlet, S.Müller-Wodarg, I.  202302\.  Saturn’s atmospheric response to the large influx of ring material inferred from Cassini INMS measurements Saturn’s atmospheric response to the large influx of ring material inferred from Cassini INMS measurements.  Icarus391115328.  10.1016/j.icarus.2022.115328
  * Moses . (2005) 05mosesMoses, JI., Fouchet, T., Bézard, B., Gladstone, GR., Lellouch, E. Feuchtgruber, H.  200508\.  Photochemistry and diffusion in Jupiter’s stratosphere: Constraints from ISO observations and comparisons with other giant planets Photochemistry and diffusion in Jupiter’s stratosphere: Constraints from ISO observations and comparisons with other giant planets.  Journal of Geophysical Research (Planets)110E08001.  10.1029/2005JE002411
  * Moses Greathouse (2005) 05moses_satMoses, JI. Greathouse, TK.  200509\.  Latitudinal and seasonal models of stratospheric photochemistry on Saturn: Comparison with infrared data from IRTF/TEXES Latitudinal and seasonal models of stratospheric photochemistry on Saturn: Comparison with infrared data from IRTF/TEXES.  Journal of Geophysical Research (Planets)110E09007.  10.1029/2005JE002450
  * Moses . (2007) 07mosesMoses, JI., Liang, MC., Yung, YL. Shia, RL.  200703\.  Two-Dimensional Photochemical Modeling of Hydrocarbon Abundances on Saturn Two-Dimensional Photochemical Modeling of Hydrocarbon Abundances on Saturn.  Lunar and Planetary Science Conference Lunar and planetary science conference ( 38, 2196).
  * Moses Poppe (2017) 17mosesMoses, JI. Poppe, AR.  201711\.  Dust ablation on the giant planets: Consequences for stratospheric photochemistry Dust ablation on the giant planets: Consequences for stratospheric photochemistry.  Icarus29733-58.  10.1016/j.icarus.2017.06.002
  * Noll Larson (1990) 90nollNoll, KS. Larson, HP.  1990\.  The spectrum of Saturn from 1990–2230 cm-1: abundances of AsH3, CH3D, CO, GeH4, and PH3 The spectrum of Saturn from 1990–2230 cm-1: abundances of AsH3, CH3D, CO, GeH4, and PH3.  Icarus89168–189.
  * OPAL Archive (2022) MAST_OPALOPAL Archive.  2022\.  Outer Planets Legacy Program Archive [Data]. Outer Planets Legacy Program Archive [Data].  https://archive.stsci.edu/prepds/opal/ 10.17909/T9G593
  * Orton . (2008) 08orton_qxoOrton, G., Yanamandra-Fisher, P., Fisher, B., Friedson, A., Parrish, P., Nelson, J.Tollestrup, E.  200805\.  Semi-annual oscillations in Saturn’s low-latitude stratospheric temperatures Semi-annual oscillations in Saturn’s low-latitude stratospheric temperatures.  Nature453196-198.
  * Pérez-Hoyos . (2016) 16perez-hoyosPérez-Hoyos, S., Sanz-Requena, JF., Sánchez-Lavega, A., Irwin, PGJ. Smith, A.  201610\.  Saturn’s tropospheric particles phase function and spatial distribution from Cassini ISS 2010-11 observations Saturn’s tropospheric particles phase function and spatial distribution from Cassini ISS 2010-11 observations.  Icarus2771-18.  10.1016/j.icarus.2016.04.022
  * Pirraglia . (1981) 81pirragliaPirraglia, JA., Conrath, BJ., Allison, MD. Gierasch, PJ.  198108\.  Thermal structure and dynamics of Saturn and Jupiter Thermal structure and dynamics of Saturn and Jupiter.  Nature2925825677-679.  10.1038/292677a0
  * Read . (2009) 09readRead, PL., Conrath, BJ., Fletcher, LN., Gierasch, PJ., Simon-Miller, AA. Zuchowski, LC.  2009Dec.  Mapping potential vorticity dynamics on saturn: Zonal mean circulation from Cassini and Voyager data Mapping potential vorticity dynamics on saturn: Zonal mean circulation from Cassini and Voyager data.  Plan. & Space Sci.5714-151682-1698.  10.1016/j.pss.2009.03.004
  * Roman . (2013) 13romanRoman, MT., Banfield, D. Gierasch, PJ.  201307\.  Saturn’s cloud structure inferred from Cassini ISS Saturn’s cloud structure inferred from Cassini ISS.  Icarus22593-110.  10.1016/j.icarus.2013.03.015
  * Sánchez-Lavega . (2020) 20sanchez_hexSánchez-Lavega, A., García-Muñoz, A., del Río-Gaztelurrutia, T., Pérez-Hoyos, S., Sanz-Requena, JF., Hueso, R.Peralta, J.  202005\.  Multilayer hazes over Saturn’s hexagon from Cassini ISS limb images Multilayer hazes over Saturn’s hexagon from Cassini ISS limb images.  Nature Communications112281.  10.1038/s41467-020-16110-1
  * Sayanagi . (2013) 13sayanagiSayanagi, KM., Dyudina, UA., Ewald, SP., Fischer, G., Ingersoll, AP., Kurth, WS.West, RA.  201303\.  Dynamics of Saturn’s great storm of 2010-2011 from Cassini ISS and RPWS Dynamics of Saturn’s great storm of 2010-2011 from Cassini ISS and RPWS.  Icarus223460-478.  10.1016/j.icarus.2012.12.013
  * Serigano . (2022) 22seriganoSerigano, J., Hörst, SM., He, C., Gautier, T., Yelle, RV., Koskinen, TT.Radke, MJ.  202206\.  Compositional Measurements of Saturn’s Upper Atmosphere and Rings From Cassini INMS: An Extended Analysis of Measurements From Cassini’s Grand Finale Orbits Compositional Measurements of Saturn’s Upper Atmosphere and Rings From Cassini INMS: An Extended Analysis of Measurements From Cassini’s Grand Finale Orbits.  Journal of Geophysical Research (Planets)1276e07238.  10.1029/2022JE007238
  * Simon . (2023) 23simonSimon, AA., Hedman, MM., Nicholson, PD., Tiscareno, MS., Showalter, MR., McDonald, T. Callos, S.  202302\.  Hubble Detects the Start of a New Saturn Ring Spoke Season Hubble Detects the Start of a New Saturn Ring Spoke Season.  Geophys. Res. Lett.503e2022GL101904.  10.1029/2022GL101904
  * Sinclair . (2013) 13sinclairSinclair, JA., Irwin, PGJ., Fletcher, LN., Moses, JI., Greathouse, TK., Friedson, AJ.Merlet, C.  201307\.  Seasonal variations of temperature, acetylene and ethane in Saturn’s atmosphere from 2005 to 2010, as observed by Cassini-CIRS Seasonal variations of temperature, acetylene and ethane in Saturn’s atmosphere from 2005 to 2010, as observed by Cassini-CIRS.  Icarus225257-271.  10.1016/j.icarus.2013.03.011
  * Sromovsky . (2013) 13sromovskySromovsky, LA., Baines, KH. Fry, PM.  201309\.  Saturn’s Great Storm of 2010-2011: Evidence for ammonia and water ices from analysis of VIMS spectra Saturn’s Great Storm of 2010-2011: Evidence for ammonia and water ices from analysis of VIMS spectra.  Icarus226402-418.  10.1016/j.icarus.2013.05.043
  * Sromovsky . (2021) 21sromovskySromovsky, LA., Baines, KH. Fry, PM.  202107\.  Evolution of Saturn’s north polar color and cloud structure between 2012 and 2017 inferred from Cassini VIMS and ISS observations Evolution of Saturn’s north polar color and cloud structure between 2012 and 2017 inferred from Cassini VIMS and ISS observations.  Icarus362114409.  10.1016/j.icarus.2021.114409
  * Sromovsky . (2016) 16sromovskySromovsky, LA., Baines, KH., Fry, PM. Momary, TW.  201609\.  Cloud clearing in the wake of Saturn’s Great Storm of 2010-2011 and suggested new constraints on Saturn’s He/H2 ratio Cloud clearing in the wake of Saturn’s Great Storm of 2010-2011 and suggested new constraints on Saturn’s He/H2 ratio.  Icarus276141-162.  10.1016/j.icarus.2016.04.031
  * Stam . (2001) 01stamStam, DM., Banfield, D., Gierasch, PJ., Nicholson, PD. Matthews, K.  2001\.  Near-IR Spectrophotometry of Saturnian Aerosols-Meridional and Vertical Distribution Near-IR Spectrophotometry of Saturnian Aerosols-Meridional and Vertical Distribution.  Icarus152407-422.  10.1006/icar.2001.6641
  * Sung . (2013) 13sungSung, K., Toon, GC., Mantz, AW. Smith, MAH.  201311\.  FT-IR measurements of cold C3H8 cross sections at 7-15 $\mu$m for Titan atmosphere FT-IR measurements of cold C3H8 cross sections at 7-15 $\mu$m for Titan atmosphere.  Icarus22621499-1513.  10.1016/j.icarus.2013.07.028
  * Sylvestre . (2015) 15sylvestreSylvestre, M., Guerlet, S., Fouchet, T., Spiga, A., Flasar, FM., Hesman, B. Bjoraker, GL.  201509\.  Seasonal changes in Saturn’s stratosphere inferred from Cassini/CIRS limb observations Seasonal changes in Saturn’s stratosphere inferred from Cassini/CIRS limb observations.  Icarus258224-238.  10.1016/j.icarus.2015.05.025
  * Tennyson . (2016) 16tennysonTennyson, J., Yurchenko, SN., Al-Refaie, AF., Barton, EJ., Chubb, KL., Coles, PA.Zak, E.  201609\.  The ExoMol database: Molecular line lists for exoplanet and other hot atmospheres The ExoMol database: Molecular line lists for exoplanet and other hot atmospheres.  Journal of Molecular Spectroscopy32773-94.  10.1016/j.jms.2016.05.002
  * Vasavada Showman (2005) 05vasavadaVasavada, AR. Showman, AP.  200508\.  Jovian atmospheric dynamics: an update after Galileo and Cassini Jovian atmospheric dynamics: an update after Galileo and Cassini.  Reports on Progress in Physics681935-1996.  10.1088/0034-4885/68/8/R06
  * Wells . (2015) 15wellsWells, M., Pel, JW., Glasse, A., Wright, GS., Aitink-Kroes, G., Azzollini, R.Wright, D.  201507\.  The Mid-Infrared Instrument for the James Webb Space Telescope, VI: The Medium Resolution Spectrometer The Mid-Infrared Instrument for the James Webb Space Telescope, VI: The Medium Resolution Spectrometer.  PASP127646.  10.1086/682281
  * Westphal . (1992) 92westphalWestphal, JA., Baum, WA., Ingersoll, AP., Barnet, CD., de Jong, EM., Danielson, GE. Caldwell, J.  1992\.  Hubble Space Telescope observations of the 1990 equatorial disturbance on Saturn - Images, albedos, and limb darkening Hubble Space Telescope observations of the 1990 equatorial disturbance on Saturn - Images, albedos, and limb darkening.  Icarus100485-498.  10.1016/0019-1035(92)90112-K
  * Wright . (2023) 23wrightWright, GS., Rieke, GH., Glasse, A., Ressler, M., Marín, MG., Aguilar, J.Wolff, S.  2023may.  The Mid-infrared Instrument for JWST and Its In-flight Performance The mid-infrared instrument for jwst and its in-flight performance.  Publications of the Astronomical Society of the Pacific1351046048003.  https://dx.doi.org/10.1088/1538-3873/acbe66 10.1088/1538-3873/acbe66
  * Yanamandra-Fisher . (2001) 01yanamandraYanamandra-Fisher, PA., Orton, GS., Fisher, BM. Sánchez-Lavega, A.  2001\.  NOTE: Saturn’s 5.2-$\mu$m Cold Spots: Unexpected Cloud Variability NOTE: Saturn’s 5.2-$\mu$m Cold Spots: Unexpected Cloud Variability.  Icarus150189-193.  10.1006/icar.2000.6580

## Supplemental Material

Figures 18-21 show comparisons of the discrete features observed in the
November-2022 MIRI/MRS observations to HST observations of the corresponding
latitudes in September 2022. Discrete features will have drifted substantially
in the $\sim 7$ weeks between observations, so these four figures are meant to
show the longitudinal context for the more focussed MRS maps. They allow the
reader to observe whether any similar vortices or other structures are seen in
the same latitude band as those observed by MRS - see the main text for a full
discussion of these features.

Figures 22 and 23 show residuals between the NEMESIS models and the MRS
measurements, where blue regions imply underfitting of the data, and red
regions imply overfitting. These figures were used to identify problems with
wavelength calibration, the overlap between MIRI channels, or residuals
implying missing spectral bands in our line database.

Figure 24 shows a single spectrum at $None$N, and the best-fitting NEMESIS
model with multiple scattering of reflected sunlight and thermal emission. We
then remove parameters from the model to show their individual effects on the
models - removing one of the two clouds; turning off reflected sunlight, and
removing aerosols entirely. This figure confirms that all models converge
longward of $\sim 6.5$ $\mu$m, such that longer wavelength MIRI channels can
be modelled without the inclusion of aerosols or multiple scattering.

Finally, Fig. 25 compares Cassini/CIRS and MIRI/MRS data to central-meridian
scans from ESO’s VISIR instrument on the VLT between 2016 and 2022 [Blake .
(2022)]. CIRS and MRS data were convolved to match the VISIR filter functions
with telluric contamination. As described in the main text, the VISIR
observations reproduce the banded structure of Saturn revealed by Cassini and
JWST, and highlight the challenges associated with calibrating ground-based
datasets.

Figure 18: Comparison of the $None$N, $None$W warm feature on Saturn with HST
observations using a range of different filters. Each subplot shows a
planetographic cylindrical map projection, with a $None$ latitude range (i.e.
from $None$N to $None$N for this figure). The variability in zonal wind speed
with latitude and time difference between the HST and JWST observations makes
it difficult to match specific discrete features between the observations.
Figure 19: Comparison of the $None$N, $None$W 5-$\mu$m warm feature with HST
observations. See Fig 18 for more detail.  Figure 20: Comparison of the
$None$N, $None$W 5 $\mu$m cool feature with HST observations. See Fig 18 for
more detail. Note that the anticyclonic vortex at $None$N observed in the HST
data is not seen in the MIRI observation. Figure 21: Comparison of the
$None$N, $None$W 5 $\mu$m warm feature with HST observations. See Fig 18 for
more detail. Figure 22: Brightness temperature residuals in Kelvin between
model and data as a function of latitude and wavelength after stage 2 and 3 of
our multi-stage retrieval scheme. Blue regions imply underfitting, red regions
imply overfitting. Possible causes of discrepancies are noted in the
appropriate locations, including overlap regions between adjacent MIRI
subbands, and persistent wavelength calibration issues. Figure 23: Brightness
temperature residuals in Kelvin before and after the addition of pseudolines
of C3H8 from 13sung. Blue regions imply underfitting, and two of the three
propane bands in this range are clearly visible in panel (a), increasing in
strength with latitude due to the warmer polar temperatures. Residuals are
much improved in panel (b), when propane is included. Figure 24: Examples of
forward models to the 4.9-7.3 $\mu$m spectral range ($None$N) using different
approaches to modelling aerosols. The best fit with multiple scattering of
scattered sunlight and thermal emission by our two-tier cloud model is shown
by the black curve, the effects of removing either the upper (blue) or lower
(orange) clouds is also shown. If we turn off reflected sunlight but retain
thermal scattering, we get the green line. Turning off scattering entirely
produces the red forward model (i.e., a purely absorbing cloud). With no
aerosols in the model, we get the high brightness temperatures of the purple
line. Beyond the CH4 emission at 6.5 $\mu$m, the different models are largely
indistinguishable from one another, supporting the exclusion of scattering and
aerosols at longer wavelengths.  Figure 25: Comparison of VLT/VISIR
observations from 2016-2022 to MIRI/MRS observations in 2022 and Cassini/CIRS
in 2017. Ground-based observations were scaled to match a low-latitude CIRS
average by 22blake, which causes the mismatch at 7.9 $\mu$m where the low-
latitudes are observed to have cooled with time. Nevertheless, the large-scale
asymmetries and belt/zone contrasts are captured by the VISIR data, with the
strongest brightness temperature gradients co-located with the peaks of
eastward jets (vertical dotted lines in each panel).


# Out-of-Distribution Detection using BiGAN and MDL

Mojtaba Abolfazli 1
<EMAIL_ADDRESS>
&Mohammad Zaeri Amirani 1
<EMAIL_ADDRESS>
Anders Høst-Madsen 1
<EMAIL_ADDRESS>
&June Zhang 1
<EMAIL_ADDRESS>
&Andras Bratincsak 2
<EMAIL_ADDRESS>

1 Department of Electrical and Computer Engineering
2 Department of Pediatrics, John A. Burns School of Medicine
University of Hawaii at Manoa, Honolulu, HI, USA

###### Abstract

We consider the following problem: we have a large dataset of normal data
available. We are now given a new, possibly quite small, set of data, and we
are to decide if these are normal data, or if they are indicating a new
phenomenon. This is a novelty detection or out-of-distribution detection
problem. An example is in medicine, where the normal data is for people with
no known disease, and the new dataset people with symptoms. Other examples
could be in security. We solve this problem by training a bidirectional
generative adversarial network (BiGAN) on the normal data and using a Gaussian
graphical model to model the output. We then use universal source coding, or
minimum description length (MDL) on the output to decide if it is a new
distribution, in an implementation of Kolmogorov and Martin-Löf randomness. We
apply the methodology to both MNIST data and a real-world electrocardiogram
(ECG) dataset of healthy and patients with Kawasaki disease, and show better
performance in terms of the ROC curve than similar methods.

##  1 Introduction

The basic problem we consider is the following: we have a large dataset
$\mathbf{x}_{i}$, $i=1\ldots N$ of ‘normal’ data available for training. For
example, this could be ECG data for people with no known heart disease. We are
then given a test set $\mathbf{x}_{i}^{\prime}$, $i=1\ldots M$ with $M\ll N$
and wants to determine if this belongs to the class of normal data, or if this
should be considered a new class of data. If $M=1$ this is anomaly detection
or outlier detection; if $M>1$ this can be called novelty detection, based on
the terminology in Pimentel et al. (2014) or group anomaly detection, as in
Chalapathy et al. (2018).

Alternatively, if we consider class identity to be solely based on the data
distribution, then we can view novelty detection as an out-of-distribution
(OOD) or distribution shift test based on Nalisnick et al. (2019); Rabanser et
al. (2019).

When $M=1$, the only real possibility of detecting anomalous data is to check
if it fits with normal data, i.e., if it was likely to have been generated by
the distribution underlying the normal data. When $M>1$, more approaches to
analysis open up. One can still check the likelihood of the test data under
the distribution of the normal data. But since we have more test data samples
available, one can try to find new patterns in data that could be an
expression of a new underlying model. This can be expressed as finding an
alternative distribution for the test data.

The goal of our paper is to perform OOD detection on high-dimensional data,
$\mathbf{x}_{i}^{\prime}$, $i=1\ldots M>1$, particularly when there are only
subtle differences between the test and training data. Our method uses
Rissanen’s minimum description length (MDL) Rissanen (1978, 1983), which
explicitly accounts for the complexity of the probabilistic model to avoid
overfitting.

We focus on subtle changes as these are typical of for example adversarial
attacks or medical data. Adversarial attackers want to mimic normal data to go
undetected. And in for example ECG data, many heart conditions are difficult
to detect in a standard 12-lead electrocardiogram (ECG). The indications is in
subtle changes of relationship between multiple variables.

##  2 Related Work

The problem of novelty or OOD detection using deep generative models has
attracted much attention in recent years due to the ability of these networks
to estimate the distribution of training data. The central approach to this
problem is that these models give lower likelihood to OOD data than in-
distribution data. This approach was initially suggested in Bishop (1994).
However, further investigations in recent years show that generative models
may give higher likelihood to OOD data than in-distribution data in high
dimensional cases Choi et al. (2018); Nalisnick et al. (2018). This phenomenon
makes their applicability for novelty or OOD detection limited. Choi et al.
(2018) use an ensembles of generative models to tackle this issue. In
Nalisnick et al. (2019), a method based on typicality set in information
theory was suggested to detect OOD data based on the closeness of the
information content to the empirical entropy of training data. In a similar
work Morningstar et al. (2021), multiple statistics from in-distribution data
are extracted to differentiate between in-distribution and OOD data. A
parameter-free OOD score similar to likelihood-ratio was proposed in Serrà et
al. (2019). A combination of Rao’s score test Rao (1948) and typicality test
Nalisnick et al. (2019) was suggested in Bergamin et al. (2022). Chalapathy et
al. (2018) proposed an approach based on measuring the deviation from in-
distribution data.

All these approaches more or less are based on how well data fits with the
default distribution, that is essentially based on the likelihood, eq. (1)
below. As explained after (1) this has an inherent limitation. Our approach is
based on finding an alternative model for the OOD data through MDL.

##  3 Problem Statement

Let the data be $\mathbf{x}\in\mathbb{R}^{n}$. We model the normal data
through a probability distribution $P(\mathbf{x})$. In an ideal setting, this
model is given; but in practical scenarios, the model is learned from training
data. How this is done is the main topic of the paper. We will start by
discussing the problem for known $P(\mathbf{x}$) and then generalize to
unknown $P(\mathbf{x})$.

###  3.1 $P(\mathbf{x})$ Known

Given a set of test data $\mathbf{x}_{i}^{\prime}$, $i=1\ldots M$, the OOD
problem can be stated as the hypothesis test

$\displaystyle H_{0}$ $\displaystyle:\\{\mathbf{x}_{i}^{\prime}\\}\sim P$
$\displaystyle H_{1}$
$\displaystyle:\\{\mathbf{x}_{i}^{\prime}\\}\sim\widehat{P}$

with $\widehat{P}\neq P$, with $\widehat{P}$ _not_ known. One approach is to
check how well the data fits $P$ through the log-likelihood of
$\\{\mathbf{x}_{i}^{\prime}\\}$ under the (known) distribution $P$,

$\ell(\\{\mathbf{x}_{i}^{\prime}\\})=-\sum_{i=1}^{M}\log_{2}P(\mathbf{x}_{i}^{\prime}),$
(1)

as was done in Nalisnick et al. (2019). We consider base-2 logarithm instead
of natural logarithm so that we can interpret the log-likelihood as binary
codelength via Kraft’s Inequality .

However, it is possible that $\\{\mathbf{x}_{i}^{\prime}\\}\sim\widehat{P}$
and $\widehat{P}\neq P$, but $\\{\mathbf{x}_{i}^{\prime}\\}$ fits perfectly
with $P$. One example is that $P$ is iid across the $n$-components of
$\mathbf{x}$, and $\widehat{P}$ has the same marginals as $P$ but a different
joint distribution. It is therefore necessary to have an alternative
distribution $\widehat{P}$, and since it is unknown, it has to be estimated
from data. One such test in one dimension is the Kolmogorov-Smirnoff test
Corder & Foreman (2014), which compares the CDF $F$ and the empirical CDF of
$\\{\mathbf{x}_{i}^{\prime}\\}$, $\hat{F}$. However, this is difficult to
generalize to large dimensions. Instead a probability density $\widehat{P}$
can be estimated from the test data. It could be a parametric or non-
parametric estimation (Vapnik, 1998, Chapter 7), Rissanen et al. (1992). For a
parametric estimation, $\widehat{P}_{\theta}$, we can choose the best
hypothesis based on the the (generalized) log-likelihood test (GLRT). We
choose $H_{0}$ if

$\ell(\\{\mathbf{x}_{i}^{\prime}\\})>\min_{\theta}-\sum_{i=1}^{M}\log_{2}\widehat{P}_{\theta}(\mathbf{x}_{i}^{\prime})+\tau$

otherwise, we choose $H_{1}$. Here $\tau$ is the threshold in the hypothesis
test that can adjusted to obtain a desired false alarm probability. If
hypothesis $H_{1}$ is chosen, then the test data, $\mathbf{x}^{\prime}$, is
considered to be from a different distribution than the training data
$\mathbf{x}$. However, if a complicated model $\widehat{P}_{\theta}$ with many
parameters is used, it can easily lead to overfitting, which results in a poor
ROC (receiver operating characteristic) curve. Our solution is to instead
consider a large class of alternative models $\\{\widehat{P}_{\theta}\\}$,
which includes both simple models and very complex models such as non-
parametric models like empirical pdfs Rissanen et al. (1992). The complexity
of the model is also accounted for through MDL principle Rissanen (1983). This
leads to the MDL criterion where we choose $H_{0}$ if

$\displaystyle\ell(\\{\mathbf{x}_{i}^{\prime}\\})>\min_{\widehat{P}_{\theta}}\min_{\theta}-\sum_{i=1}^{M}\log_{2}\widehat{P}_{\theta}(\mathbf{x}_{i}^{\prime})+L(\widehat{P}_{\theta})+\tau.$
(2)

Otherwise, we choose $H_{1}$. The term $L(\widehat{P}_{\theta})$ is the
description length of the model $\widehat{P}_{\theta}$ within the class
$\\{\widehat{P}_{\theta}\\}$. We can interpret the LHS and RHS of (2) as
codelengths. This criterion is a special case of atypicality as defined in
Høst-Madsen et al. (2019); Sabeti & Høst-Madsen (2019)

######  Definition 3.1.

A test set, ${\mathbf{x}^{\prime}_{i}}$ is (atypical) out of distribution if
it can be described (coded) with fewer bits in itself rather than using the
(optimum) code for (typical) training set, ${\mathbf{x}_{i}}$.

A downside to using (2) is that it is difficult to use the method on complex
data which do not have ”nice” distributions. An equivalent, but more
practical, approach to (atypical) out of distribution detection is based on
the observation that if data is encoded by an optimum coder, the resulting
bitstream is iid uniform, i.e., totally random Cover & Thomas (2006). In other
words, the output of an optimum coder can not be compressed any further. If it
can be further compressed, it is therefore OOD. From a theoretical point of
view, this approach is directly an application of the theory of randomness
developed by Kolmogorov and Martin-Löf Li & Vitányi (2008); Nies (2009); Cover
& Thomas (2006). Random sequences can be characterized through Kolmogorov
complexity. A sequence of bits $\\{x_{n},n=1,\ldots,\infty\\}$ is random (i.e,
iid uniform) if the Kolmogorov complexity of the sequence satisfies
$K(x_{1},\ldots,x_{n})\geq n-c$ for some constant $c$ and for all $n$ Li &
Vitányi (2008). The sequence is incompressible if $K(x_{1},\ldots,x_{n}|n)\geq
n$ for all $n$, and a finite sequence is algorithmically random if
$K(x_{1},\ldots,x_{n}|n)\geq n$ Cover & Thomas (2006). Thus, we characterize a
set of data as OOD if the output of the optimum coder is not random according
to Martin-Löf’s definition.

For real-valued data, coding into a bitstream is not feasible or useful. A
better approach is to transform data into an iid Gaussian random variables
(i.e., $\mathcal{N}(0,\Sigma)$), and then use a coder for the Gaussian
distribution. The transformation is always possible for continuous random
variables Sabeti & Høst-Madsen (2019)

######  Proposition 3.2.

For any continuous random variable $\mathbf{X}$ there exists an
$n$-dimensional iid uniform random variable $\mathbf{U}$ and an invertible
$\check{\mathbf{F}}$ so that $\mathbf{X}=\check{\mathbf{F}}^{-1}(\mathbf{U})$.

IID uniform random variables can be converted into an iid Gaussian
distribution by using the normal CDF $\Phi$ on each $U_{i}$. We will call the
combined invertible map $\mathcal{E}:\mathbb{R}^{n}\to\mathbb{R}^{n}$. Unlike
(2), this approach leads to an algorithm that can be applied to complex high-
dimensional data: Algorithm 1.

Algorithm 1 Known $P(\mathbf{x})$

1:Input: $\\{\mathbf{x}^{\prime}_{i}$, $i=1\ldots M\\}$

2:Calculate $\mathbf{z}_{i}^{\prime}=\mathcal{E}(\mathbf{x}_{i})$.

3:Code the sequence $\\{\mathbf{z}^{\prime}_{i}$, $i=1\ldots M\\}$ with an iid
Gaussian coder with length $L_{1}$.

4:Code the sequence $\\{\mathbf{z}^{\prime}_{i}$, $i=1\ldots M\\}$ with a
universal vector Gaussian coder with length $L_{2}$.

5:if $L_{2}+\tau<L_{1}$ then

6: $\\{\mathbf{x}_{i}^{\prime}$, $i=1\ldots M\\}$ is OOD.

7:end if

How to code multivariate Gaussian data will be discussed in Section 4. A
limitation is that the universal coder (a coder that does not know the
underlying data distribution apriori) is also Gaussian. This is because really
the only ”nice” multivariate distribution is Gaussian. However, one could
easily combine this with a coder that codes each component of
$\mathbf{z}^{\prime}_{i}$ with an MDL based histogram as in Rissanen et al.
(1992); coders can always be combined by choosing the shortest codelength as
in (2). We did not pursue this direction because we think the most interesting
signs of distribution shift can be seen in the high dimensional structure.

In practice, we generally do not know $P(\mathbf{x})$, and in the following we
will therefore generalize Algorithm 1 to this case.

###  3.2 $P(\mathbf{x})$ Unknown

When $P(\mathbf{x})$ is not known apriori, it first has to be estimated from a
set of training data $\mathbf{x}_{i}$, $i=1\ldots N$. Then to use Algorithm 1
we need the (invertible) map $\mathcal{E}$. In one dimension one can simply
use the empirical CDF, but when the data is high-dimensional, we need a more
sophisticated approach. In this paper we choose to use BiGAN Donahue et al.
(2016); Dumoulin et al. (2016). We will discuss alternatives below.

Figure 1:  BiGAN structure from Donahue et al. (2016)

Shown in Figure 1, BiGAN has a generator map
$\mathcal{G}:\mathbb{R}^{m}\to\mathbb{R}^{n}$ and an encoder map
$\mathcal{E}:\mathbb{R}^{n}\to\mathbb{R}^{m}$. Here $\mathbb{R}^{m}$, $m\leq
n$, is the feature or latent space, on which the distribution is specified to
be $\mathcal{N}(\mathbf{0},\mathbf{I})$, as is usual in GAN. Extending the
usual GAN architecture, the discriminator now trains to jointly discriminate
in data and latent space (tuples $(\mathbf{x},\mathcal{E}(\mathbf{x}))$ versus
$(\mathcal{G}(\mathbf{z}),\mathbf{z}))$.

If the training of the BiGAN is perfect, the mappings $\mathcal{G}$ and
$\mathcal{E}$ are each others inverses according to Donahue et al. (2016).
However, in practical training scenarios, the solutions for $\mathcal{G}$ and
$\mathcal{E}$ do not necessarily converge to the the global optimum. That
means $\mathbf{x}\neq\mathcal{G}(\mathcal{E}(\mathbf{x}))$. Since MDL is based
on lossless coding, that is, reproducing data exactly from the encoding, this
is an issue.

Our solution is that we first encode $\mathbf{z}=\mathcal{E}(\mathbf{x})$, as
in in Algorithm 1, and subsequently also encode the residual, or
reconstruction error, $\mathbf{x}-\mathcal{G}(\mathbf{z})$. We will detail how
this joint coding is done. Notice that MDL Rissanen (1983) is based on
quantizing the real-valued data and then letting the quantization step
converge to zero. We need to carefully derive how this works for the joint
coding.

Consider coding of a _single_ data $\mathbf{x}$. We represent each component
of $\mathbf{z}=\mathcal{E}(\mathbf{x})$ in fixed point with $r_{z}$ digits
after ’.’ and unlimited number of digits prior as in Rissanen (1983). The
number of bits required to encode $\mathbf{z}$ is then

$\displaystyle L_{z}(\mathbf{z})$
$\displaystyle=-\log_{2}\int_{\mathbf{z}}^{\mathbf{z}+2^{-r_{z}}}P_{z}(\mathbf{t})d\mathbf{t}\approx-\log_{2}(P_{z}(\mathbf{z})2^{-mr_{z}})$
$\displaystyle=-\log_{2}P_{z}(\mathbf{z})+mr_{z}.$ (3)

Next the coder encodes $\mathbf{x}-\mathcal{G}(\hat{\mathbf{z}})$ with $r$
bits precision, where $\hat{\mathbf{z}}$ is the quantized version of
$\mathcal{E}(\mathbf{x})$. The number of bits required is

$\displaystyle L(x)$
$\displaystyle=-\log_{2}\int_{\mathbf{x}}^{\mathbf{x}+2^{-r}}P(\mathbf{t}-\mathcal{G}(\hat{\mathbf{z}}))d\mathbf{t}$
$\displaystyle\approx-\log_{2}(P(\mathbf{x}-\mathcal{G}(\hat{\mathbf{z}}))2^{-nr})$
$\displaystyle=-\log_{2}P(\mathbf{x}-\mathcal{G}(\hat{\mathbf{z}}))+nr.$ (4)

we have here assumed that $\hat{\mathbf{z}}$ does not change inside the
quantization cube $[\mathbf{x},\mathbf{x}+2^{-r}]$. This is true if the number
of digits $r$ is large relative to $r_{z}$, which is reasonable as even if let
$r_{z}\to\infty$, we do not get lossless coding due the non-invertibility of
$\mathcal{E}$, but $r\to\infty$ gives lossless coding. Here $mr_{z}$ and $nr$
do not depend on which coder is used, and therefore cancels out: In the
criterion (2) we always compare two codelengths and the $mr_{z}+nr$ in both
sides cancel out. We can therefore let $r,r_{z}\to\infty$ and we end up with
the codelength

$\displaystyle-\log_{2}P_{z}(\mathbf{z})-\log_{2}P(\mathbf{x}-\mathcal{G}(\mathbf{z}))$
(5)

as codelength $L_{1}$. The remaining question is what to use for the
distribution $P(\mathbf{x}-\mathcal{G}({\mathbf{z}}))$. Since the idea is that
$\mathcal{E}$ should capture all structure in the data, we suggest to simply
use an iid Gaussian distribution,
$\mathbf{x}-\mathcal{G}({\mathbf{z}})\sim\mathcal{N}(\mu\mathbf{1},\sigma^{2}\mathbf{I})$,
with either estimated $\mu$, $\sigma^{2}$ for training coding, or using MDL
for $\mu,\sigma^{2}$ in the universal coder.

The method is described in Algorithm 2.

Algorithm 2 Unknown $P(\mathbf{x})$

1:Training phase

2:Input: $\\{\mathbf{x}_{i}$, $i=1\ldots N\\}$

3:Train BiGAN on $\mathbf{x}_{i}$ with output $\mathcal{G},\mathcal{E}$.

4:Estimate the covariance matrix of $\\{\mathcal{E}(\mathbf{x}_{i})$,
$i=1\ldots N\\}$ as described in Section 4.

5:Estimate the scalars $\mu$ and $\sigma^{2}$ of
$\\{\mathbf{x}_{i}-\mathcal{G}(\mathcal{E}(\mathbf{x}_{i}))$, $i=1\ldots N\\}$

6:

7:Test phase

8:Input: $\\{\mathbf{x}^{\prime}_{i}$, $i=1\ldots M\\}$

9:Calculate $\mathbf{z}_{i}^{\prime}=\mathcal{E}(\mathbf{x}_{i})$.

10:Code the data $\\{\mathbf{z}^{\prime}_{i}$, $i=1\ldots M\\}$ with the
estimated covariance matrix from training using a multivariate Gaussian coder,
and
$\\{\mathbf{x}_{i}^{\prime}-\mathcal{G}(\mathcal{E}(\mathbf{x}_{i}^{\prime}))$,
$i=1\ldots M\\}$ with $\mu,\sigma^{2}$ from training using a univariate
Guassian coder. Combine using (5) resulting in codelength $L_{1}$.

11:Code the data $\\{\mathbf{z}^{\prime}_{i}$, $i=1\ldots M\\}$ with a
universal multivariate Gaussian coder as described in Section 4 and
$\\{\mathbf{x}_{i}^{\prime}-\mathcal{G}(\mathcal{E}(\mathbf{x}_{i}^{\prime}))$,
$i=1\ldots M\\}$ with a universal univariate Gaussian coder (Cover & Thomas,
2006, Section 13.2). Sum the two codelengths together to obtain $L_{2}$.

12:if $L_{2}+\tau<L_{1}$ then

13: $\\{\mathbf{x}_{i}^{\prime}$, $i=1\ldots M\\}$ is out-of-distribution.

14:end if

The reason we code $\\{\mathbf{z}_{i}^{\prime}\\}$ with an estimated
covariance matrix in step 7 is that the distribution of $\mathcal{E}(x)$ might
not be perfectly $\mathcal{N}(\mathbf{0},\mathbf{I})$, and then the universal
coder would always be better if we did not encode with the estimated
covariance matrix.

The main issue with BiGAN in our context is that the reconstruction error
$\mathbf{x}-\mathcal{G}(\mathcal{E}(\mathbf{x}))$ can be large with an ”ugly”
distribution for both in-distribution and OOD data. We did experiment with
autoencoders as alternatives. A key feature required of methods in our context
is the ability to prescribe the distribution in the latent space, and not all
autoencoders have that ability. We tried to alternative methods that have that
ability: Adversarial AutoEncoder (AAE) Makhzani et al. (2015), and Variational
AutoEncoder (VAE) Kingma & Welling (2013). Both of these were better at
reconstruction, i.e., $\mathbf{x}-\mathcal{G}(\mathcal{E}(\mathbf{x}))$ was
significantly smaller. However, the distribution of $\mathcal{E}(x)$ were very
far from Gaussian, where BiGAN gives something that is close to Gaussian, and
in the end BiGAN therefore had better performance.

##  4 Universal Multivariate Gaussian Coder

A multivariate Gaussian distribution with zero mean is completely
characterized by its covariance matrix $\boldsymbol{\Sigma}$. When the
covariance matrix is not known, it has to be estimated from data. A simple
solution to estimate $\boldsymbol{\Sigma}$ is the maximum likelihood, which is
empirical covariance matrix $\mathbf{S}$. However, the empirical covariance
matrix is an overparametrized solution especially when the data is generated
under a sparse Gaussian graphical model; that is $\boldsymbol{\Sigma}^{-1}$ is
sparse Dempster (1972). An efficient method for finding
$\boldsymbol{\Sigma}^{-1}$, especially when a sparse structure is of interest,
is to use regularized log-likelihood solution Friedman et al. (2008); Yuan &
Lin (2007)

$\underset{\boldsymbol{\Omega}\succ 0}{\text{max}}\hskip
5.69046pt\mbox{logdet}\hskip
2.84544pt\boldsymbol{\Omega}-\mbox{tr}(\mathbf{S}\boldsymbol{\Omega})-\lambda\|\boldsymbol{\Omega}\|_{1},$
(6)

where $\boldsymbol{\Omega}=\boldsymbol{\Sigma}^{-1}$ is the precision matrix,
$\lambda$ is the regularization term, and $\|\boldsymbol{\Omega}\|_{1}$ is
$\ell_{1}$ norm. The parameter $\lambda$ determines the sparsity of
$\boldsymbol{\Omega}$ and needs to be tuned. The sparsity pattern of
$\boldsymbol{\Omega}$ can be captured by a graph called conditional
independence graph $G(V,E)$, where $V$ is the set of vertices and $E$ is the
set of edges. The graph $G$ contains an edge between vertex $i$ and vertex
$j$, i.e., $(i,j)\in E$, if $\boldsymbol{\Omega}_{ij}\neq 0$.

To pick the best value of penalty term $\lambda$, different model selection
techniques have been proposed in the literature. A detailed overview of
existing methods can be found in Abolfazli et al. (2021a). We use the approach
proposed in Abolfazli et al. (2021b) to select $\lambda$ and hence find graph
$G$. Unlike other approaches that may only consider the number of edges from
$G$ (i.e., number of nonzero elements in $\boldsymbol{\Omega}$) to account for
model complexity, this approach considers the whole structure of $G$ for a
more accurate model complexity. This is achieved by adopting MDL principle.
For every value of $\lambda$ we first encode the structure associated with it,
i.e., $G_{\lambda}$ and then we encode the data with the resulting sparse
structure; the model that minimizes the summation of these two terms is the
best model

$\min_{\lambda}L(G_{\lambda})+L(\mathbf{x}|G_{\lambda}).$ (7)

In (7), $L(G_{\lambda})$ is the description length of the conditional
independence graph, $G_{\lambda}$, associated with $\lambda$, and
$L(\mathbf{x}|G_{\lambda})$ is the description length of data, $\mathbf{x}$,
when encoded with $G_{\lambda}$. We use graph statistics such as graph motifs
and degree distribution to encode $G_{\lambda}$. The detail on how to compute
$L(G_{\lambda})$ can be found in Abolfazli et al. (2021b).

To compute $L(\mathbf{x}|G_{\lambda})$, we need to encode the data with
respect to $G_{\lambda}$. Based on the empirical covariance matrix
$\mathbf{S}$, one can estimate the covariance matrix,
$\widehat{\boldsymbol{\Sigma}}$, using Dempster’s covariance selection
Dempster (1972) that satisfies
$\widehat{\boldsymbol{\Sigma}}_{ij}=\mathbf{S}_{ij}$ when $(i,j)\in E$, and
$\widehat{\boldsymbol{\Sigma}}^{-1}_{ij}=0$ when $(i,j)\notin E$.

Once we estimate $\widehat{\boldsymbol{\Sigma}}$, we use predictive MDL to
compute $L(\mathbf{x}|G_{\lambda})$

$L(\mathbf{x}|G_{\lambda})=-\sum_{i=0}^{N-1}\log_{2}f\left(\mathbf{x}_{i+1}|\hat{\boldsymbol{\theta}}(\mathbf{x}_{1},\ldots,\mathbf{x}_{i})\right),$
(8)

where $f(\cdot|\cdot)$ is the conditional pdf and
$\hat{\boldsymbol{\theta}}(\mathbf{x}_{1},\ldots,\mathbf{x}_{i})$ denote the
maximum likelihood estimate of parameters, which is
$\widehat{\boldsymbol{\Sigma}}$ obtained under $G_{\lambda}$. To encode real-
valued data, we assume a fixed-point representation with a (large) finite
number, $r$, bits after the period, and an unlimited number of bits before the
period as described in Section 3.2. Note that since there is no estimate of
$\widehat{\boldsymbol{\Sigma}}$ for the first few sample, we encode them with
an arbitrary default distribution.

##  5 Experiments

The proposed MDL method is examine with the digital image dataset MNIST Deng
(2012) as well as a real-world ECG dataset Bratincsák et al. (2020).

###  5.1 MNIST dataset

We used the ImageDataGenerator function of the keras Python library to perturb
the MNIST dataset to create several synthetic OOD dataset. The corresponding
perturbations to generate synthetic OOD datasets are listed in Table
LABEL:tab:scenarios.

Table 1: Scenarios for OOD detection. Rotation and shearing values are in degree, width and height shift in fraction, zoom and brightness in range. | Perturbation type | Value/Range value
---|---|---
Case–1 | Rotation | $5$
Case–2 | Shearing | $20$
Case–3 |  $[$ Width shift , Height shift$]$ | $[0.02,0.02]$
Case–4 | Zooming | $[0.8,1.2]$
Case–5 | Zooming | $[1,1.1]$
Case–6 | Zooming | $[0.9,1]$
Case–7 | Brightness | $[0.2,2]$
Case–8 | Brightness | $[1,2]$
Case–9 | Brightness | $[0.2,1]$
Case–10 | Gaussian noise | $\mu=0,\sigma=0.05$

As it can be seen in Figure 2 it is hard to distinguish between in-
distribution (unperturbed) and OOD (perturbed) data by visual inspection. In
fact, each of samples in the OOD test set individually can be considered as a
normal sample but their collective behavior is OOD. Our goal is exactly to
detect such subtle changes.

In our experiment, 60,0000 examples in the original MNIST dataset were used
for training and validation, and 10,000 original test images were used as in-
distribution test data. All the modified datasets were considered as OOD in
testing.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Figure 2: Sample plot for original test images and corresponding images in OOD
test cases. Note that the differences between the normal data and the OOD data
is very subtle.

After training BiGAN, we transfer both training and test datasets into the
latent space of size $20$. For the comparison scenarios, the AAE and VAE
autoencoders as described in Chalapathy et al. (2018) were used to transfer
the data into the latent space of the same dimension. Table 2 provides the
details of the employed BiGAN, AAE, and VAE hyperparameters in this
experiment. For the sake of fairness, we employ two fully connected hidden
layers of size 512, each followed by a Leaky Relu and Batch normalization
layers for the encoder and decode/generator networks in all three BiGAN, AAE,
and VAE schemes. The same architecture is used for the discrimination network
in BiGAN and AAE. The sigmoid activation function in the last layer is used
for the decoder/generator networks in all three schemes and in the
discriminator network for BiGAN and AAE.

Table 2: BiGAN, AAE, and VAE networks and training hyperparameters Hyperparameter | Value
---|---
Number of epochs | $50,000$
Batch size | $128$
Initial learning rate | $0.002$
Learning rate decacy |  $0.01$ after $25,000$ epochs
$\ell_{2}$-regulizer | $2.5\cdot 10^{-4}$
Initializing network weight | $\mathcal{N}(0,0.2)$
Bath normalization momentum | $0.8$
LeakyRelu | $\alpha=0.2$

Afterward, we encode the data using Algorithm 2, which entails the coding of
data in latent space and encoding of reconstruction error.

We used graphical lasso Friedman et al. (2008) to solve (6). It is possible to
use any regularized estimator of Gaussian graphical models. For regularization
hyperparamter $\lambda$, we consider $20$ candidate values within the range
$[0.1,1]$ with logarithmic step size. The results are given for one of the
graph coders, i.e., coding with number of common neighbors and degree
distribution. For details of coding, an interested reader can refer to
Abolfazli et al. (2021b).

To perform the experiments, we split the data into batches of size $M$ for
each of the OOD test cases and also the original in-distribution test data and
compute $L_{2}-L_{1}$ based on 2 for each batch. Next, we can obtain ROC curve
by plotting $L_{2}-L_{1}$ of the OOD data versus $L_{2}-L_{1}$ of the in-
distribution test data. Table 3 shows the area under ROC curve (AUROC) for
each test case across different values of test set size $M$ for Algorithm 2
verses the distance-based ($\ell_{2}$) approach proposed in Chalapathy et al.
(2018). We implemented the distance-based approach for three different
generative networks: BiGAN, AAE, and VAE. The results indicate that our
approach performed best in all cases except Case–8. Even for Case–8, our
approach still had decent detection performance. The performance of Algorithm
2 improves with test set size.

It might seem odd that we get AUROC less than $0.5$ (i.e., random guessing) in
some cases for $\ell_{2}$-BiGAN, $\ell_{2}$-AAE, and $\ell_{2}$-VAE. The
reason is that in these cases, the OOD data somehow has smaller reconstruction
error than the in-distribution data even though the generative model has never
seen the OOD data before. This phenomenon has also been observed in the
literature Choi et al. (2018); Rabanser et al. (2019); Nalisnick et al.
(2019).

Note that we can not improve the performance of the $\ell_{2}$-based approach
by ”flipping-the-output,” i.e., decide the opposite of what the test says,
since this does not apply to all the test cases such as Case–2, 4, 8, 10, and
in real world we don’t know what is the type of OOD data.

Table 3: AUROC for different test cases and testset size $M$. The best value
for each case is boldfaced. Algorithm 2 denotes our approach, and $\ell_{2}$
refer to method in Chalapathy et al. (2018) under different architectures.

| $\mathbf{M=50}$ | $\mathbf{M=100}$ | $\mathbf{M=200}$
---|---|---|---
|  Algorithm 2 |  $\ell_{2}$-BiGAN |  $\ell_{2}$-AAE |  $\ell_{2}$-VAE |  Algorithm 2 |  $\ell_{2}$-BiGAN |  $\ell_{2}$-AAE |  $\ell_{2}$-VAE |  Algorithm 2 |  $\ell_{2}$-BiGAN |  $\ell_{2}$-AAE |  $\ell_{2}$-VAE
Case–1 | $\mathbf{0.633}$ | $0.167$ | $0.015$ | $0.003$ | $\mathbf{0.693}$ | $0.076$ | $0.001$ | $0.000$ | $\mathbf{0.769}$ | $0.032$ | $0.000$ | $0.000$
Case–2 | $\mathbf{0.709}$ | $0.685$ | $0.505$ | $0.501$ | $\mathbf{0.791}$ | $0.765$ | $0.501$ | $0.475$ | $\mathbf{0.873}$ | $0.839$ | $0.502$ | $0.511$
Case–3 | $\mathbf{0.592}$ | $0.192$ | $0.002$ | $0.000$ | $\mathbf{0.638}$ | $0.108$ | $0.000$ | $0.000$ | $\mathbf{0.687}$ | $0.054$ | $0.000$ | $0.000$
Case–4 | $\mathbf{0.930}$ | $0.928$ | $0.539$ | $0.464$ | $\mathbf{0.987}$ | $0.980$ | $0.606$ | $0.503$ | $\mathbf{1.000}$ | $\mathbf{1.000}$ | $0.573$ | $0.477$
Case–5 | $\mathbf{0.707}$ | $0.176$ | $0.002$ | $0.001$ | $\mathbf{0.815}$ | $0.090$ | $0.000$ | $0.000$ | $\mathbf{0.914}$ | $0.038$ | $0.000$ | $0.000$
Case–6 | $\mathbf{0.840}$ | $0.645$ | $0.317$ | $0.167$ | $\mathbf{0.933}$ | $0.711$ | $0.254$ | $0.074$ | $\mathbf{0.989}$ | $0.775$ | $0.174$ | $0.021$
Case–7 | $\mathbf{0.740}$ | $0.666$ | $0.139$ | $0.139$ | $\mathbf{0.846}$ | $0.722$ | $0.050$ | $0.067$ | $\mathbf{0.917}$ | $0.774$ | $0.012$ | $0.012$
Case–8 | $0.896$ | $0.903$ | $0.956$ | $\mathbf{0.982}$ | $0.971$ | $0.965$ | $0.993$ | $\mathbf{0.998}$ | $0.997$ | $0.997$ | $\mathbf{1.000}$ | $\mathbf{1.000}$
Case–9 | $\mathbf{0.988}$ | $0.197$ | $0.000$ | $0.000$ | $\mathbf{1.000}$ | $0.111$ | $0.000$ | $0.000$ | $\mathbf{1.000}$ | $0.058$ | $0.000$ | $0.000$
Case–10 | $\mathbf{0.904}$ | $0.712$ | $0.845$ | $0.663$ | $\mathbf{0.967}$ | $0.802$ | $0.923$ | $0.725$ | $\mathbf{0.998}$ | $0.873$ | $0.978$ | $0.807$

###  5.2 ECG dataset

We were provided ECG data of 23522 subjects, 423 subjects were diagnosed with
the Kawasaki disease and the rest are considered healthy subjects. Kawasaki
disease causes inflammation in blood vessels in children under the age of five
years old McCrindle et al. (2017). However, it can be very difficult for
medical professionals to diagnose using since ECG data between healthy and
diseased subjects can be very similar.

For each patient, a total of 176 Heart Rate Variation (HRV) features were
automatically extracted from 12 leads ECG signals Bratincsák et al. (2020). We
used $90\%$ of the healthy (normal data) subjects were used for training. The
remaining healthy subjects and diseased subjects were used as test set. A
batch of healthy subjects should be detected as an in-distribution data
whereas a batch of diseased subjects should be detected as an OOD sample.

To test the performance of Algorithm 2, we randomly pick $50$ samples from
each of in-distribution test data and OOD data (i.e., Kawasaki data). We
repeat this experiment $500$ times. We used BiGAN, where the dimension of the
latent space is $40$. The ROC curve is obtained by changing the threshold
$\tau$. Figure 3 represents ROC curve for this experiment. The corresponding
AUROC is equal to $0.9996$. This shows that our approach can distinguish
Kawasaki subjects as OOD data with high performance.

Figure 3: ROC curve of detecting Kawasaki subjects as OOD data.

##  6 Conclusion, Limitations, and Future work

We have developed a method for out-of-distribution detection based on MDL. We
have shown that it performs well on both MNIST and real-world ECG data.

There are two future directions to consider. First, from our experiments, we
learned that BiGAN is far from ideal: the reconstruction error
$\mathbf{x}-\mathcal{G}(\mathcal{E}(\mathbf{x}))$ is quite large, and has a
strange distribution. For future work, we will consider other deep generative
models; we did experiment with AAE and VAE but they do resulted in additional
issues compared with BiGAN. Second, we only considered multivariate Gaussian
distributions as the alternative models in this paper. An advantage of our
MDL/coding approach is that we can substitute a single coder with a
combination of multiple coders. For future work, we will use such an approach
to consider alternative models that are mixtures of multivariate Gaussians,
and in more generality non-parametric distributions.

## Acknowledgements

The research was funded in part by the NSF grant CCF-1908957.

## References

  * Abolfazli et al. (2021a) Abolfazli, M., Høst-Madsen, A., Zhang, J., and Bratincsak, A.  Graph coding for model selection and anomaly detection in gaussian graphical models.  In _ISIT’2021, Melbourne, Australia, July 12-20, 2021_ , 2021a.
  * Abolfazli et al. (2021b) Abolfazli, M., Host-Madsen, A., Zhang, J., and Bratincsak, A.  Graph compression with application to model selection.  _arXiv preprint arXiv:2110.00701_ , 2021b.
  * Bergamin et al. (2022) Bergamin, F., Mattei, P.-A., Havtorn, J. D., Senetaire, H., Schmutz, H., Maaløe, L., Hauberg, S., and Frellsen, J.  Model-agnostic out-of-distribution detection using combined statistical tests.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 10753–10776. PMLR, 2022.
  * Bishop (1994) Bishop, C. M.  Novelty detection and neural network validation.  _IEE Proceedings-Vision, Image and Signal processing_ , 141(4):217–222, 1994.
  * Bratincsák et al. (2020) Bratincsák, A., Kimata, C., Limm-Chan, B. N., Vincent, K. P., Williams, M. R., and Perry, J. C.  Electrocardiogram standards for children and young adults using Z-scores.  _Circulation: Arrhythmia and Electrophysiology_ , 13(8):e008253, 2020.
  * Chalapathy et al. (2018) Chalapathy, R., Toth, E., and Chawla, S.  Group anomaly detection using deep generative models.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pp. 173–189. Springer, 2018.
  * Choi et al. (2018) Choi, H., Jang, E., and Alemi, A. A.  Waic, but why? generative ensembles for robust anomaly detection.  _arXiv preprint arXiv:1810.01392_ , 2018.
  * Corder & Foreman (2014) Corder, G. W. and Foreman, D. I.  _Nonparametric Statistics: A Step-by-Step Approach_.  John Wiley & Sons, 2014.
  * Cover & Thomas (2006) Cover, T. and Thomas, J.  _Information Theory, 2nd Edition_.  John Wiley, 2006.
  * Dempster (1972) Dempster, A. P.  Covariance selection.  _Biometrics_ , 28(1):157–175, 1972.  ISSN 0006341X, 15410420.  URL http://www.jstor.org/stable/2528966.
  * Deng (2012) Deng, L.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Donahue et al. (2016) Donahue, J., Krähenbühl, P., and Darrell, T.  Adversarial feature learning.  _arXiv preprint arXiv:1605.09782_ , 2016.
  * Dumoulin et al. (2016) Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., and Courville, A.  Adversarially learned inference.  _arXiv preprint arXiv:1606.00704_ , 2016.
  * Friedman et al. (2008) Friedman, J., Hastie, T., and Tibshirani, R.  Sparse inverse covariance estimation with the graphical lasso.  _Biostatistics_ , 9(3):432–441, 2008.
  * Høst-Madsen et al. (2019) Høst-Madsen, A., Sabeti, E., and Walton, C.  Data discovery and anomaly detection using atypicality.  _IEEE Transactions on Information Theory_ , 65(9), September 2019.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li & Vitányi (2008) Li, M. and Vitányi, P.  _An Introduction to Kolmogorov Complexity and Its Applications_.  Springer, third edition, 2008.
  * Makhzani et al. (2015) Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B.  Adversarial autoencoders.  _arXiv preprint arXiv:1511.05644_ , 2015.
  * McCrindle et al. (2017) McCrindle, B. W., Rowley, A. H., Newburger, J. W., Burns, J. C., Bolger, A. F., Gewitz, M., Baker, A. L., Jackson, M. A., Takahashi, M., Shah, P. B., et al.  Diagnosis, treatment, and long-term management of kawasaki disease: a scientific statement for health professionals from the american heart association.  _Circulation_ , 135(17):e927–e999, 2017.
  * Morningstar et al. (2021) Morningstar, W., Ham, C., Gallagher, A., Lakshminarayanan, B., Alemi, A., and Dillon, J.  Density of states estimation for out of distribution detection.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 3232–3240. PMLR, 2021.
  * Nalisnick et al. (2018) Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B.  Do deep generative models know what they don’t know?  _arXiv preprint arXiv:1810.09136_ , 2018.
  * Nalisnick et al. (2019) Nalisnick, E., Matsukawa, A., Teh, Y. W., and Lakshminarayanan, B.  Detecting out-of-distribution inputs to deep generative models using typicality, 2019.  URL https://arxiv.org/abs/1906.02994.
  * Nies (2009) Nies, A.  _Computability and Randomness_.  Oxford University Press, 2009.
  * Pimentel et al. (2014) Pimentel, M. A., Clifton, D. A., Clifton, L., and Tarassenko, L.  A review of novelty detection.  _Signal Processing_ , 99:215–249, 2014.  ISSN 0165-1684.  doi: https://doi.org/10.1016/j.sigpro.2013.12.026.  URL https://www.sciencedirect.com/science/article/pii/S016516841300515X.
  * Rabanser et al. (2019) Rabanser, S., Günnemann, S., and Lipton, Z.  Failing loudly: An empirical study of methods for detecting dataset shift.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/846c260d715e5b854ffad5f70a516c88-Paper.pdf.
  * Rao (1948) Rao, C. R.  Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation.  In _Mathematical Proceedings of the Cambridge Philosophical Society_ , volume 44, pp. 50–57. Cambridge University Press, 1948.
  * Rissanen (1978) Rissanen, J.  Modeling by shortest data description.  _Automatica_ , pp. 465–471, 1978.
  * Rissanen (1983) Rissanen, J.  A universal prior for integers and estimation by minimum description length.  _The Annals of Statistics_ , (2):416–431, 1983\.
  * Rissanen et al. (1992) Rissanen, J., Speed, T., and Yu, B.  Density estimation by stochastic complexity.  _IEEE Transactions on Information Theory_ , 38(2):315–323, 1992.  doi: 10.1109/18.119689.
  * Sabeti & Høst-Madsen (2019) Sabeti, E. and Høst-Madsen, A.  Data discovery and anomaly detection using atypicality for real-valued data.  _Entropy_ , pp. 219, Feb. 2019.  Available at https://doi.org/10.3390/e21030219.
  * Serrà et al. (2019) Serrà, J., Álvarez, D., Gómez, V., Slizovskaia, O., Núñez, J. F., and Luque, J.  Input complexity and out-of-distribution detection with likelihood-based generative models.  In _International Conference on Learning Representations_ , 2019.
  * Vapnik (1998) Vapnik, V. N.  _Statistical Learning Theory_.  John Wiley, 1998.
  * Yuan & Lin (2007) Yuan, M. and Lin, Y.  Model selection and estimation in the gaussian graphical model.  _Biometrika_ , 94(1):19–35, 2007.