* [59] A. Avgoustidis and I. Zavala, _Warped Wilson Line DBI Inflation_ , _JCAP_ 01 (2009) 045 [0810.5001].
  * [60] T.R. Taylor and C. Vafa, _R R flux on Calabi-Yau and partial supersymmetry breaking_ , _Phys. Lett. B_ 474 (2000) 130 [hep-th/9912152].
  * [61] K. Becker, M. Becker, M. Haack and J. Louis, _Supersymmetry breaking and alpha-prime corrections to flux induced potentials_ , _JHEP_ 06 (2002) 060 [hep-th/0204254].
  * [62] O. DeWolfe and S.B. Giddings, _Scales and hierarchies in warped compactifications and brane worlds_ , _Phys. Rev. D_ 67 (2003) 066008 [hep-th/0208123].
  * [63] B. de Wit, D.J. Smit and N.D. Hari Dass, _Residual Supersymmetry of Compactified D=10 Supergravity_ , _Nucl. Phys. B_ 283 (1987) 165.
  * [64] J. Polchinski, _Dirichlet Branes and Ramond-Ramond charges_ , _Phys. Rev. Lett._ 75 (1995) 4724 [hep-th/9510017].
  * [65] F. Denef, _Les Houches Lectures on Constructing String Vacua_ , _Les Houches_ 87 (2008) 483 [0803.1194].
  * [66] M. Cicoli, J.P. Conlon and F. Quevedo, _Systematics of String Loop Corrections in Type IIB Calabi-Yau Flux Compactifications_ , _JHEP_ 01 (2008) 052 [0708.1873].
  * [67] M. Cicoli, J.P. Conlon and F. Quevedo, _General Analysis of LARGE Volume Scenarios with String Loop Moduli Stabilisation_ , _JHEP_ 10 (2008) 105 [0805.1029].
  * [68] S. AbdusSalam, S. Abel, M. Cicoli, F. Quevedo and P. Shukla, _A systematic approach to Kähler moduli stabilisation_ , _JHEP_ 08 (2020) 047 [2005.11329].
  * [69] M. Cicoli, F. Quevedo, R. Savelli, A. Schachner and R. Valandro, _Systematics of the $\alpha$’ expansion in F-theory_, _JHEP_ 08 (2021) 099 [2106.04592].
  * [70] E. Palti, _The Swampland: Introduction and Review_ , _Fortsch. Phys._ 67 (2019) 1900037 [1903.06239].
  * [71] R.C. Smith, _Notes on the swampland (4): The distance conjecture for arbitrary calabi-yau manifolds, the emergence proposal, and the de sitter conjecture_ , Oct, 2019.
  * [72] C. Vafa, _The String landscape and the swampland_ , hep-th/0509212.
  * [73] H. Ooguri and C. Vafa, _On the Geometry of the String Landscape and the Swampland_ , _Nucl. Phys. B_ 766 (2007) 21 [hep-th/0605264].
  * [74] D. Klaewer and E. Palti, _Super-Planckian Spatial Field Variations and Quantum Gravity_ , _JHEP_ 01 (2017) 088 [1610.00010].
  * [75] F. Baume and E. Palti, _Backreacted Axion Field Ranges in String Theory_ , _JHEP_ 08 (2016) 043 [1602.06517].
  * [76] I. Valenzuela, _Backreaction Issues in Axion Monodromy and Minkowski 4-forms_ , _JHEP_ 06 (2017) 098 [1611.00394].
  * [77] S. Bielleman, L.E. Ibanez, F.G. Pedro, I. Valenzuela and C. Wieck, _Higgs-otic Inflation and Moduli Stabilization_ , _JHEP_ 02 (2017) 073 [1611.07084].
  * [78] R. Blumenhagen, I. Valenzuela and F. Wolf, _The Swampland Conjecture and F-term Axion Monodromy Inflation_ , _JHEP_ 07 (2017) 145 [1703.05776].
  * [79] A. Hebecker, P. Henkenjohann and L.T. Witkowski, _Flat Monodromies and a Moduli Space Size Conjecture_ , _JHEP_ 12 (2017) 033 [1708.06761].
  * [80] M. Cicoli, D. Ciupke, C. Mayrhofer and P. Shukla, _A Geometrical Upper Bound on the Inflaton Range_ , _JHEP_ 05 (2018) 001 [1801.05434].
  * [81] A. Font, A. Herráez and L.E. Ibáñez, _The Swampland Distance Conjecture and Towers of Tensionless Branes_ , _JHEP_ 08 (2019) 044 [1904.05379].
  * [82] P. Agrawal, G. Obied, P.J. Steinhardt and C. Vafa, _On the Cosmological Implications of the String Swampland_ , _Phys. Lett. B_ 784 (2018) 271 [1806.09718].
  * [83] F. Denef, A. Hebecker and T. Wrase, _de Sitter swampland conjecture and the Higgs potential_ , _Phys. Rev. D_ 98 (2018) 086004 [1807.06581].
  * [84] S.K. Garg and C. Krishnan, _Bounds on Slow Roll and the de Sitter Swampland_ , _JHEP_ 11 (2019) 075 [1807.05193].
  * [85] H. Ooguri, E. Palti, G. Shiu and C. Vafa, _Distance and de Sitter Conjectures on the Swampland_ , _Phys. Lett. B_ 788 (2019) 180 [1810.05506].
  * [86] M. Dine and N. Seiberg, _Is the Superstring Weakly Coupled?_ , _Phys. Lett. B_ 162 (1985) 299.
  * [87] S. Weinberg, _Gauge Hierarchies_ , _Phys. Lett. B_ 82 (1979) 387.
  * [88] M.E. Peskin and D.V. Schroeder, _An Introduction to quantum field theory_ , Addison-Wesley, Reading, USA (1995).
  * [89] S. Dimopoulos and H. Georgi, _Softly Broken Supersymmetry and SU(5)_ , _Nucl. Phys. B_ 193 (1981) 150.
  * [90] G. ’t Hooft, _Naturalness, chiral symmetry, and spontaneous chiral symmetry breaking_ , _NATO Sci. Ser. B_ 59 (1980) 135.
  * [91] L. Susskind, _Dynamics of Spontaneous Symmetry Breaking in the Weinberg-Salam Theory_ , _Phys. Rev. D_ 20 (1979) 2619.
  * [92] S. Dimopoulos, L. Susskind and S. Raby, _Technicolour_ , _AIP Conf. Proc._ 59 (1980) 407.
  * [93] E. Farhi and L. Susskind, _A Technicolored G.U.T._ , _Phys. Rev. D_ 20 (1979) 3404.
  * [94] C.P. Burgess, _The Cosmological Constant Problem: Why it’s hard to get Dark Energy from Micro-physics_ , in _100e Ecole d’Ete de Physique: Post-Planck Cosmology_ , pp. 149–197, 2015, DOI [1309.4133].
  * [95] S. Weinberg, _Anthropic Bound on the Cosmological Constant_ , _Phys. Rev. Lett._ 59 (1987) 2607.
  * [96] J.D. Barrow and F.J. Tipler, _The Anthropic Cosmological Principle_ , Oxford U. Pr., Oxford (1988).
  * [97] S. Weinberg, _Theories of the cosmological constant_ , in _Conference on Critical Dialogs in Cosmology_ , pp. 195–203, 8, 1996 [astro-ph/9610044].
  * [98] H. Martel, P.R. Shapiro and S. Weinberg, _Likely values of the cosmological constant_ , _Astrophys. J._ 492 (1998) 29 [astro-ph/9701099].
  * [99] J. Garriga and A. Vilenkin, _On likely values of the cosmological constant_ , _Phys. Rev. D_ 61 (2000) 083502 [astro-ph/9908115].
  * [100] N. Kaloper and A. Padilla, _Sequestering the Standard Model Vacuum Energy_ , _Phys. Rev. Lett._ 112 (2014) 091304 [1309.6562].
  * [101] N. Kaloper and A. Padilla, _Vacuum Energy Sequestering: The Framework and Its Cosmological Consequences_ , _Phys. Rev. D_ 90 (2014) 084023 [1406.0711].
  * [102] N. Kaloper and A. Padilla, _Sequestration of Vacuum Energy and the End of the Universe_ , _Phys. Rev. Lett._ 114 (2015) 101302 [1409.7073].
  * [103] N. Kaloper, A. Padilla, D. Stefanyszyn and G. Zahariade, _Manifestly Local Theory of Vacuum Energy Sequestering_ , _Phys. Rev. Lett._ 116 (2016) 051302 [1505.01492].
  * [104] N. Kaloper, A. Padilla and D. Stefanyszyn, _Sequestering effects on and of vacuum decay_ , _Phys. Rev. D_ 94 (2016) 025022 [1604.04000].
  * [105] N. Kaloper and A. Padilla, _Vacuum Energy Sequestering and Graviton Loops_ , _Phys. Rev. Lett._ 118 (2017) 061303 [1606.04958].
  * [106] A. Adams, J. McGreevy and E. Silverstein, _Decapitating tadpoles_ , hep-th/0209226.
  * [107] F. Niedermann and A. Padilla, _Gravitational Mechanisms to Self-Tune the Cosmological Constant: Obstructions and Ways Forward_ , _Phys. Rev. Lett._ 119 (2017) 251306 [1706.04778].
  * [108] E.A. Bergshoeff, D.Z. Freedman, R. Kallosh and A. Van Proeyen, _Pure de Sitter Supergravity_ , _Phys. Rev. D_ 92 (2015) 085040 [1507.08264].
  * [109] E. Dudas, S. Ferrara, A. Kehagias and A. Sagnotti, _Properties of Nilpotent Supergravity_ , _JHEP_ 09 (2015) 217 [1507.07842].
  * [110] I. Bandos, L. Martucci, D. Sorokin and M. Tonin, _Brane induced supersymmetry breaking and de Sitter supergravity_ , _JHEP_ 02 (2016) 080 [1511.03024].
  * [111] S. Nagy, A. Padilla and I. Zavala, _The Super-Stückelberg procedure and dS in pure supergravity_ , _Proc. Roy. Soc. Lond. A_ 476 (2020) 20200035 [1910.14349].
  * [112] S. Bansal, S. Nagy, A. Padilla and I. Zavala, _Unimodular vs Nilpotent Superfield Approach to Pure dS Supergravity_ , _JHEP_ 01 (2021) 146 [2010.13758].
  * [113] G.R. Dvali, G. Gabadadze and M. Porrati, _Metastable gravitons and infinite volume extra dimensions_ , _Phys. Lett. B_ 484 (2000) 112 [hep-th/0002190].
  * [114] E. Witten, _The Cosmological constant from the viewpoint of string theory_ , in _4th International Symposium on Sources and Detection of Dark Matter in the Universe (DM 2000)_ , pp. 27–36, 3, 2000 [hep-ph/0002297].
  * [115] Y. Aghababaie, C.P. Burgess, S.L. Parameswaran and F. Quevedo, _Towards a naturally small cosmological constant from branes in 6-D supergravity_ , _Nucl. Phys. B_ 680 (2004) 389 [hep-th/0304256].
  * [116] C.P. Burgess and L. van Nierop, _Technically Natural Cosmological Constant From Supersymmetric 6D Brane Backreaction_ , _Phys. Dark Univ._ 2 (2013) 1 [1108.0345].
  * [117] C.P. Burgess and L. van Nierop, _Large Dimensions and Small Curvatures from Supersymmetric Brane Back-reaction_ , _JHEP_ 04 (2011) 078 [1101.0152].
  * [118] F. Niedermann and R. Schneider, _Fine-tuning with Brane-Localized Flux in 6D Supergravity_ , _JHEP_ 02 (2016) 025 [1508.01124].
  * [119] F. Niedermann and R. Schneider, _SLED Phenomenology: Curvature vs. Volume_ , _JHEP_ 03 (2016) 130 [1512.03800].
  * [120] R. Bousso and J. Polchinski, _Quantization of four form fluxes and dynamical neutralization of the cosmological constant_ , _JHEP_ 06 (2000) 006 [hep-th/0004134].
  * [121] S. Ashok and M.R. Douglas, _Counting flux vacua_ , _JHEP_ 01 (2004) 060 [hep-th/0307049].
  * [122] F. Denef and M.R. Douglas, _Distributions of flux vacua_ , _JHEP_ 05 (2004) 072 [hep-th/0404116].
  * [123] M. Demirtas, M. Kim, L. McAllister, J. Moritz and A. Rios-Tascon, _Exponentially Small Cosmological Constant in String Theory_ , _Phys. Rev. Lett._ 128 (2022) 011602 [2107.09065].
  * [124] M. Demirtas, M. Kim, L. McAllister, J. Moritz and A. Rios-Tascon, _Small cosmological constants in string theory_ , _JHEP_ 12 (2021) 136 [2107.09064].
  * [125] I. Zlatev, L.-M. Wang and P.J. Steinhardt, _Quintessence, cosmic coincidence, and the cosmological constant_ , _Phys. Rev. Lett._ 82 (1999) 896 [astro-ph/9807002].
  * [126] Planck collaboration, _Planck 2018 results. VI. Cosmological parameters_ , _Astron. Astrophys._ 641 (2020) A6 [1807.06209].
  * [127] H.E.S. Velten, R.F. vom Marttens and W. Zimdahl, _Aspects of the cosmological “coincidence problem”_ , _Eur. Phys. J. C_ 74 (2014) 3160 [1410.2509].
  * [128] F.C. Adams and G. Laughlin, _A Dying universe: The Long term fate and evolution of astrophysical objects_ , _Rev. Mod. Phys._ 69 (1997) 337 [astro-ph/9701131].
  * [129] J. Garriga, M. Livio and A. Vilenkin, _The Cosmological constant and the time of its dominance_ , _Phys. Rev. D_ 61 (2000) 023503 [astro-ph/9906210].
  * [130] S. Weinberg, _The Cosmological constant problems_ , in _4th International Symposium on Sources and Detection of Dark Matter in the Universe (DM 2000)_ , pp. 18–26, 2, 2000 [astro-ph/0005265].
  * [131] S.A. Bludman, _Vacuum energy: If not now, then when?_ , _Nucl. Phys. A_ 663 (2000) 865 [astro-ph/9907168].
  * [132] A. Barreira and P.P. Avelino, _Anthropic versus cosmological solutions to the coincidence problem_ , _Phys. Rev. D_ 83 (2011) 103001 [1103.2401].
  * [133] C. Armendariz-Picon, V.F. Mukhanov and P.J. Steinhardt, _A Dynamical solution to the problem of a small cosmological constant and late time cosmic acceleration_ , _Phys. Rev. Lett._ 85 (2000) 4438 [astro-ph/0004134].
  * [134] C. Armendariz-Picon, V.F. Mukhanov and P.J. Steinhardt, _Essentials of k essence_ , _Phys. Rev. D_ 63 (2001) 103510 [astro-ph/0006373].
  * [135] M. Malquarti, E.J. Copeland and A.R. Liddle, _K-essence and the coincidence problem_ , _Phys. Rev. D_ 68 (2003) 023512 [astro-ph/0304277].
  * [136] R.J. Scherrer, _Phantom dark energy, cosmic doomsday, and the coincidence problem_ , _Phys. Rev. D_ 71 (2005) 063519 [astro-ph/0410508].
  * [137] P.P. Avelino, _The Coincidence problem in linear dark energy models_ , _Phys. Lett. B_ 611 (2005) 15 [astro-ph/0411033].
  * [138] G. D’Amico, N. Kaloper, A. Padilla, D. Stefanyszyn, A. Westphal and G. Zahariade, _An étude on global vacuum energy sequester_ , _JHEP_ 09 (2017) 074 [1705.08950].
  * [139] F. Niedermann and A. Padilla, _Gravitational Mechanisms to Self-Tune the Cosmological Constant: Obstructions and Ways Forward_ , _Phys. Rev. Lett._ 119 (2017) 251306 [1706.04778].
  * [140] B. Coltman, Y. Li and A. Padilla, _Cosmological consequences of Omnia Sequestra_ , _JCAP_ 06 (2019) 017 [1903.02829].
  * [141] I. Iben and A. Renzini, _Asymptotic giant branch evolution and beyond_ , _Ann. Rev. Astron. Astrophys._ 21 (1983) 271.
  * [142] M. Livio, _How rare are extraterrestrial civilizations and when did they emerge?_ , _Astrophys. J._ 511 (1999) 429 [astro-ph/9808237].
  * [143] M.L. Graesser, S.D.H. Hsu, A. Jenkins and M.B. Wise, _Anthropic distribution for cosmological constant and primordial density perturbations_ , _Phys. Lett. B_ 600 (2004) 15 [hep-th/0407174].
  * [144] A. Aguirre, _The Cold big bang cosmology as a counter example to several anthropic arguments_ , _Phys. Rev. D_ 64 (2001) 083508 [astro-ph/0106143].
  * [145] C. Armendariz-Picon, V.F. Mukhanov and P.J. Steinhardt, _Essentials of k essence_ , _Phys. Rev. D_ 63 (2001) 103510 [astro-ph/0006373].
  * [146] K. Choi and S.H. Im, _Realizing the relaxion from multiple axions and its UV completion with high scale supersymmetry_ , _JHEP_ 01 (2016) 149 [1511.00132].
  * [147] D.E. Kaplan and R. Rattazzi, _Large field excursions and approximate discrete symmetries from a clockwork axion_ , _Phys. Rev. D_ 93 (2016) 085007 [1511.01827].
  * [148] G. Dvali, _Large hierarchies from attractor vacua_ , _Phys. Rev. D_ 74 (2006) 025018 [hep-th/0410286].
  * [149] G. Dvali, _Three-form gauging of axion symmetries and gravity_ , hep-th/0507215.
  * [150] N. Kaloper and L. Sorbo, _A Natural Framework for Chaotic Inflation_ , _Phys. Rev. Lett._ 102 (2009) 121301 [0811.1989].
  * [151] N. Kaloper, A. Lawrence and L. Sorbo, _An Ignoble Approach to Large Field Inflation_ , _JCAP_ 03 (2011) 023 [1101.0026].
  * [152] T. Banks, M. Dine, P.J. Fox and E. Gorbatov, _On the possibility of large axion decay constants_ , _JCAP_ 06 (2003) 001 [hep-th/0303252].
  * [153] J.E. Kim, H.P. Nilles and M. Peloso, _Completing natural inflation_ , _JCAP_ 01 (2005) 005 [hep-ph/0409138].
  * [154] N. Arkani-Hamed, L. Motl, A. Nicolis and C. Vafa, _The String landscape, black holes and gravity as the weakest force_ , _JHEP_ 06 (2007) 060 [hep-th/0601001].
  * [155] J. Brown, W. Cottrell, G. Shiu and P. Soler, _Fencing in the Swampland: Quantum Gravity Constraints on Large Field Inflation_ , _JHEP_ 10 (2015) 023 [1503.04783].
  * [156] C. Cheung and G.N. Remmen, _Naturalness and the Weak Gravity Conjecture_ , _Phys. Rev. Lett._ 113 (2014) 051601 [1402.2287].
  * [157] T. Rudelius, _Constraints on Axion Inflation from the Weak Gravity Conjecture_ , _JCAP_ 09 (2015) 020 [1503.00795].
  * [158] G.F. Giudice and M. McCullough, _A Clockwork Theory_ , _JHEP_ 02 (2017) 036 [1610.07962].
  * [159] L.F. Abbott, _A Mechanism for Reducing the Value of the Cosmological Constant_ , _Phys. Lett. B_ 150 (1985) 427.
  * [160] J.D. Brown and C. Teitelboim, _Dynamical Neutralization of the Cosmological Constant_ , _Phys. Lett. B_ 195 (1987) 177.
  * [161] J.D. Brown and C. Teitelboim, _Neutralization of the Cosmological Constant by Membrane Creation_ , _Nucl. Phys. B_ 297 (1988) 787.
  * [162] E. Silverstein and A. Westphal, _Monodromy in the CMB: Gravity Waves and String Inflation_ , _Phys. Rev. D_ 78 (2008) 106003 [0803.3085].
  * [163] L. McAllister, E. Silverstein and A. Westphal, _Gravity Waves and Linear Inflation from Axion Monodromy_ , _Phys. Rev. D_ 82 (2010) 046003 [0808.0706].
  * [164] N. Kaloper and A. Lawrence, _London equation for monodromy inflation_ , _Phys. Rev. D_ 95 (2017) 063526 [1607.06105].
  * [165] A. Padilla, _Monodromy inflation and an emergent mechanism for stabilising the cosmological constant_ , _JHEP_ 01 (2019) 175 [1806.04740].
  * [166] B.K. El-Menoufi, S. Nagy, F. Niedermann and A. Padilla, _Quantum corrections to vacuum energy sequestering (with monodromy)_ , _Class. Quant. Grav._ 36 (2019) 215014 [1903.07612].
  * [167] N. Kaloper, _Irrational Monodromies of Vacuum Energy_ , _JHEP_ 11 (2019) 106 [1806.03308].
  * [168] J. Polchinski and A. Strominger, _New vacua for type II string theory_ , _Phys. Lett. B_ 388 (1996) 736 [hep-th/9510227].
  * [169] K.-M. Lee and E.J. Weinberg, _Decay of the True Vacuum in Curved Space-time_ , _Phys. Rev. D_ 36 (1987) 1088.
  * [170] B.S. Acharya, F. Benini and R. Valandro, _Fixing moduli in exact type IIA flux vacua_ , _JHEP_ 02 (2007) 018 [hep-th/0607223].
  * [171] L.E. Ibanez and M. Montero, _A Note on the WGC, Effective Field Theory and Clockwork within String Theory_ , _JHEP_ 02 (2018) 057 [1709.02392].
  * [172] S. Bielleman, L.E. Ibanez and I. Valenzuela, _Minkowski 3-forms, Flux String Vacua, Axion Stability and Naturalness_ , _JHEP_ 12 (2015) 119 [1507.06793].
  * [173] F. Carta, F. Marchesano, W. Staessens and G. Zoccarato, _Open string multi-branched and Kähler potentials_ , _JHEP_ 09 (2016) 062 [1606.00508].
  * [174] E. Bergshoeff, R. Kallosh, T. Ortin, D. Roest and A. Van Proeyen, _New formulations of D = 10 supersymmetry and D8 - O8 domain walls_ , _Class. Quant. Grav._ 18 (2001) 3359 [hep-th/0103233].
  * [175] A. Saltman and E. Silverstein, _A New handle on de Sitter compactifications_ , _JHEP_ 01 (2006) 139 [hep-th/0411271].
  * [176] L.J. Romans, _Massive N=2a Supergravity in Ten-Dimensions_ , _Phys. Lett. B_ 169 (1986) 374.
  * [177] O. DeWolfe, A. Giryavets, S. Kachru and W. Taylor, _Type IIA moduli stabilization_ , _JHEP_ 07 (2005) 066 [hep-th/0505160].
  * [178] G.F. Giudice and M. McCullough, _Comment on ”Disassembling the Clockwork Mechanism”_ , 1705.10162.
  * [179] L. Randall and R. Sundrum, _A Large mass hierarchy from a small extra dimension_ , _Phys. Rev. Lett._ 83 (1999) 3370 [hep-ph/9905221].
  * [180] M. Cicoli, S. De Alwis, A. Maharana, F. Muia and F. Quevedo, _De Sitter vs Quintessence in String Theory_ , _Fortsch. Phys._ 67 (2019) 1800079 [1808.08967].
  * [181] T.W. Grimm, E. Palti and I. Valenzuela, _Infinite Distances in Field Space and Massless Towers of States_ , _JHEP_ 08 (2018) 143 [1802.08264].
  * [182] R. Blumenhagen, D. Kläwer, L. Schlechter and F. Wolf, _The Refined Swampland Distance Conjecture in Calabi-Yau Moduli Spaces_ , _JHEP_ 06 (2018) 052 [1803.04989].
  * [183] B. Heidenreich, M. Reece and T. Rudelius, _Emergence of Weak Coupling at Large Distance in Quantum Gravity_ , _Phys. Rev. Lett._ 121 (2018) 051601 [1802.08698].
  * [184] A. Landete and G. Shiu, _Mass Hierarchies and Dynamical Field Range_ , _Phys. Rev. D_ 98 (2018) 066012 [1806.01874].
  * [185] S.-J. Lee, W. Lerche and T. Weigand, _Tensionless Strings and the Weak Gravity Conjecture_ , _JHEP_ 10 (2018) 164 [1808.05958].
  * [186] R. Blumenhagen, _Large Field Inflation/Quintessence and the Refined Swampland Distance Conjecture_ , _PoS_ CORFU2017 (2018) 175 [1804.10504].
  * [187] A. Font, A. Herráez and L.E. Ibáñez, _The Swampland Distance Conjecture and Towers of Tensionless Branes_ , _JHEP_ 08 (2019) 044 [1904.05379].
  * [188] G. Obied, H. Ooguri, L. Spodyneiko and C. Vafa, _De Sitter Space and the Swampland_ , 1806.08362.
  * [189] D. Andriot, _New constraints on classical de Sitter: flirting with the swampland_ , _Fortsch. Phys._ 67 (2019) 1800103 [1807.09698].
  * [190] A. Hebecker, T. Skrzypek and M. Wittner, _The $F$-term Problem and other Challenges of Stringy Quintessence_, _JHEP_ 11 (2019) 134 [1909.08625].
  * [191] A. Banerjee, H. Cai, L. Heisenberg, E.O. Colgáin, M.M. Sheikh-Jabbari and T. Yang, _Hubble sinks in the low-redshift swampland_ , _Phys. Rev. D_ 103 (2021) L081305 [2006.00244].
  * [192] Y. Akrami, R. Kallosh, A. Linde and V. Vardanyan, _The Landscape, the Swampland and the Era of Precision Cosmology_ , _Fortsch. Phys._ 67 (2019) 1800075 [1808.09440].
  * [193] E.J. Copeland, M. Sami and S. Tsujikawa, _Dynamics of dark energy_ , _Int. J. Mod. Phys. D_ 15 (2006) 1753 [hep-th/0603057].
  * [194] V. Mukhanov and S. Winitzki, _Introduction to quantum effects in gravity_ , Cambridge University Press (6, 2007).
  * [195] M. Abramowitz and I. Stegun, _Handbook of Mathematical Functions: With Formulas, Graphs, and Mathematical Tables_ , Applied mathematics series, Dover Publications (1965).
  * [196] J. Khoury and A. Weltman, _Chameleon fields: Awaiting surprises for tests of gravity in space_ , _Phys. Rev. Lett._ 93 (2004) 171104 [astro-ph/0309300].
  * [197] J. Khoury and A. Weltman, _Chameleon cosmology_ , _Phys. Rev. D_ 69 (2004) 044026 [astro-ph/0309411].
  * [198] A. Padilla, E. Platts, D. Stefanyszyn, A. Walters, A. Weltman and T. Wilson, _How to Avoid a Swift Kick in the Chameleons_ , _JCAP_ 03 (2016) 058 [1511.05761].
  * [199] A. Bedroya and C. Vafa, _Trans-Planckian Censorship and the Swampland_ , _JHEP_ 09 (2020) 123 [1909.11063].
  * [200] M. Dine and Y. Yu, _Comments on the Transplanckian Censorship Conjecture_ , 2110.07300.
  * [201] M. Dine, J.A.P. Law-Smith, S. Sun, D. Wood and Y. Yu, _Obstacles to Constructing de Sitter Space in String Theory_ , _JHEP_ 02 (2021) 050 [2008.12399].
  * [202] M.P. Hertzberg, S. Kachru, W. Taylor and M. Tegmark, _Inflationary Constraints on Type IIA String Theory_ , _JHEP_ 12 (2007) 095 [0711.2512].
  * [203] S.K. Garg, C. Krishnan and M. Zaid Zaz, _Bounds on Slow Roll at the Boundary of the Landscape_ , _JHEP_ 03 (2019) 029 [1810.09406].
  * [204] B. Valeixo Bento, D. Chakraborty, S.L. Parameswaran and I. Zavala, _Dark Energy in String Theory_ , _PoS_ CORFU2019 (2020) 123 [2005.10168].
  * [205] A. Hebecker, T. Skrzypek and M. Wittner, _The $F$-term Problem and other Challenges of Stringy Quintessence_, _JHEP_ 11 (2019) 134 [1909.08625].
  * [206] H. Ooguri and C. Vafa, _Non-supersymmetric AdS and the Swampland_ , _Adv. Theor. Math. Phys._ 21 (2017) 1787 [1610.01533].
  * [207] C.P. Burgess, M. Cicoli, D. Ciupke, S. Krippendorf and F. Quevedo, _UV Shadows in EFTs: Accidental Symmetries, Robustness and No-Scale Supergravity_ , _Fortsch. Phys._ 68 (2020) 2000076 [2006.06694].
  * [208] M. Cicoli, S. de Alwis and A. Westphal, _Heterotic Moduli Stabilisation_ , _JHEP_ 10 (2013) 199 [1304.1809].
  * [209] M. Cicoli, G. Dibitetto and F.G. Pedro, _New accelerating solutions in late-time cosmology_ , _Phys. Rev. D_ 101 (2020) 103524 [2002.02695].
  * [210] M. Cicoli, G. Dibitetto and F.G. Pedro, _Out of the Swampland with Multifield Quintessence?_ , _JHEP_ 10 (2020) 035 [2007.11011].
  * [211] L.A. Boyle, R.R. Caldwell and M. Kamionkowski, _Spintessence! New models for dark matter and dark energy_ , _Phys. Lett. B_ 545 (2002) 17 [astro-ph/0105318].
  * [212] M. Brinkmann, M. Cicoli, G. Dibitetto and F.G. Pedro, _To appear_ , .
  * [213] F.J. Dyson, _Divergence of perturbation theory in quantum electrodynamics_ , _Phys. Rev._ 85 (1952) 631.
  * [214] T. Aoyama, M. Hayakawa, T. Kinoshita and M. Nio, _Tenth-Order QED Contribution to the Electron g-2 and an Improved Value of the Fine Structure Constant_ , _Phys. Rev. Lett._ 109 (2012) 111807 [1205.5368].
  * [215] D. Hanneke, S. Fogwell and G. Gabrielse, _New Measurement of the Electron Magnetic Moment and the Fine Structure Constant_ , _Phys. Rev. Lett._ 100 (2008) 120801 [0801.1134].
  * [216] C.M. Bender and T.T. Wu, _Anharmonic oscillator_ , _Phys. Rev._ 184 (1969) 1231.
  * [217] T.I. Banks and C.M. Bender, _Anharmonic oscillator with polynomial self-interaction_ , _J. Math. Phys._ 13 (1972) 1320.
  * [218] C.M. Bender and T.T. Wu, _Anharmonic oscillator. 2: A Study of perturbation theory in large order_ , _Phys. Rev. D_ 7 (1973) 1620.
  * [219] S. Graffi, V. Grecchi and B. Simon, _Borel summability: Application to the anharmonic oscillator_ , _Phys. Lett. B_ 32 (1970) 631.
  * [220] E. Palti, C. Vafa and T. Weigand, _Supersymmetric Protection and the Swampland_ , _JHEP_ 06 (2020) 168 [2003.10452].
  * [221] A. Hebecker, _Lectures on Naturalness, String Landscape and Multiverse_ , 2008.10625.
  * [222] M. Cicoli, F. Quevedo, R. Savelli, A. Schachner and R. Valandro, _Systematics of the $\alpha$’ expansion in F-theory_, _JHEP_ 08 (2021) 099 [2106.04592].
  * [223] S. Kachru, R. Kallosh, A.D. Linde and S.P. Trivedi, _De Sitter vacua in string theory_ , _Phys. Rev. D_ 68 (2003) 046005 [hep-th/0301240].
  * [224] M. Cicoli, S. De Alwis, A. Maharana, F. Muia and F. Quevedo, _De Sitter vs Quintessence in String Theory_ , _Fortsch. Phys._ 67 (2019) 1800079 [1808.08967].
  * [225] K. Becker, M. Becker, M. Haack and J. Louis, _Supersymmetry breaking and alpha-prime corrections to flux induced potentials_ , _JHEP_ 06 (2002) 060 [hep-th/0204254].
  * [226] M. Berg, M. Haack and E. Pajer, _Jumping Through Loops: On Soft Terms from Large Volume Compactifications_ , _JHEP_ 09 (2007) 031 [0704.0737].
  * [227] D. Ciupke, J. Louis and A. Westphal, _Higher-Derivative Supergravity and Moduli Stabilization_ , _JHEP_ 10 (2015) 094 [1505.03092].
  * [228] R. Blumenhagen, M. Cvetic, S. Kachru and T. Weigand, _D-Brane Instantons in Type II Orientifolds_ , _Ann. Rev. Nucl. Part. Sci._ 59 (2009) 269 [0902.3251].
  * [229] J.J. Blanco-Pillado, C.P. Burgess, J.M. Cline, C. Escoda, M. Gomez-Reino, R. Kallosh et al., _Racetrack inflation_ , _JHEP_ 11 (2004) 063 [hep-th/0406230].
  * [230] R. Kallosh and A.D. Linde, _Landscape, the scale of SUSY breaking, and inflation_ , _JHEP_ 12 (2004) 004 [hep-th/0411011].
  * [231] V. Balasubramanian, P. Berglund, J.P. Conlon and F. Quevedo, _Systematics of moduli stabilisation in Calabi-Yau flux compactifications_ , _JHEP_ 03 (2005) 007 [hep-th/0502058].
  * [232] J.P. Conlon, F. Quevedo and K. Suruliz, _Large-volume flux compactifications: Moduli spectrum and D3/D7 soft supersymmetry breaking_ , _JHEP_ 08 (2005) 007 [hep-th/0505076].
  * [233] M.T. Grisaru, A.E.M. van de Ven and D. Zanon, _Two-Dimensional Supersymmetric Sigma Models on Ricci Flat Kahler Manifolds Are Not Finite_ , _Nucl. Phys. B_ 277 (1986) 388.
  * [234] E. Witten, _New Issues in Manifolds of SU(3) Holonomy_ , _Nucl. Phys. B_ 268 (1986) 79.
  * [235] C.P. Burgess, C. Escoda and F. Quevedo, _Nonrenormalization of flux superpotentials in string theory_ , _JHEP_ 06 (2006) 044 [hep-th/0510213].
  * [236] M. Berg, M. Haack and B. Kors, _Loop corrections to volume moduli and inflation in string theory_ , _Phys. Rev. D_ 71 (2005) 026005 [hep-th/0404087].
  * [237] P. Candelas, X.C. De La Ossa, P.S. Green and L. Parkes, _A Pair of Calabi-Yau manifolds as an exactly soluble superconformal theory_ , _Nucl. Phys. B_ 359 (1991) 21.
  * [238] B.R. Greene and M.R. Plesser, _Duality in Calabi-Yau Moduli Space_ , _Nucl. Phys. B_ 338 (1990) 15.
  * [239] P. Candelas, P.S. Green and T. Hubsch, _Rolling Among Calabi-Yau Vacua_ , _Nucl. Phys. B_ 330 (1990) 49.
  * [240] M. Berg, M. Haack and B. Kors, _String loop corrections to Kahler potentials in orientifolds_ , _JHEP_ 11 (2005) 030 [hep-th/0508043].
  * [241] M. Berg, M. Haack and E. Pajer, _Jumping Through Loops: On Soft Terms from Large Volume Compactifications_ , _JHEP_ 09 (2007) 031 [0704.0737].
  * [242] V. Balasubramanian, P. Berglund, J.P. Conlon and F. Quevedo, _Systematics of moduli stabilisation in Calabi-Yau flux compactifications_ , _JHEP_ 03 (2005) 007 [hep-th/0502058].
  * [243] X. Gao, A. Hebecker, S. Schreyer and G. Venken, _Loops, Local Corrections and Warping in the LVS and other Type IIB Models_ , 2204.06009.
  * [244] D. Ciupke, J. Louis and A. Westphal, _Higher-Derivative Supergravity and Moduli Stabilization_ , _JHEP_ 10 (2015) 094 [1505.03092].
  * [245] M.D. Freeman and C.N. Pope, _Beta Functions and Superstring Compactifications_ , _Phys. Lett. B_ 174 (1986) 48.
  * [246] R. Blumenhagen, M. Cvetic, S. Kachru and T. Weigand, _D-Brane Instantons in Type II Orientifolds_ , _Ann. Rev. Nucl. Part. Sci._ 59 (2009) 269 [0902.3251].
  * [247] D. Baumann, A. Dymarsky, S. Kachru, I.R. Klebanov and L. McAllister, _D3-brane Potentials from Fluxes in AdS/CFT_ , _JHEP_ 06 (2010) 072 [1001.5028].
  * [248] M. Kim, _D-Instanton Superpotential In String Theory_ , 2201.04634.
  * [249] J.J. Blanco-Pillado, C.P. Burgess, J.M. Cline, C. Escoda, M. Gomez-Reino, R. Kallosh et al., _Inflating in a better racetrack_ , _JHEP_ 09 (2006) 002 [hep-th/0603129].
  * [250] J.P. Conlon, _The de Sitter swampland conjecture and supersymmetric AdS vacua_ , _Int. J. Mod. Phys. A_ 33 (2018) 1850178 [1808.05040].
  * [251] A. Westphal, _de Sitter string vacua from Kahler uplifting_ , _JHEP_ 03 (2007) 102 [hep-th/0611332].
  * [252] S. Ashok and M.R. Douglas, _Counting flux vacua_ , _JHEP_ 01 (2004) 060 [hep-th/0307049].
  * [253] F. Denef and M.R. Douglas, _Distributions of flux vacua_ , _JHEP_ 05 (2004) 072 [hep-th/0404116].
  * [254] F. Denef and M.R. Douglas, _Distributions of nonsupersymmetric flux vacua_ , _JHEP_ 03 (2005) 061 [hep-th/0411183].
  * [255] W. Taylor and Y.-N. Wang, _The F-theory geometry with most flux vacua_ , _JHEP_ 12 (2015) 164 [1511.03209].
  * [256] M. Demirtas, M. Kim, L. McAllister, J. Moritz and A. Rios-Tascon, _Small cosmological constants in string theory_ , _JHEP_ 12 (2021) 136 [2107.09064].
  * [257] R. Kallosh and A.D. Linde, _Landscape, the scale of SUSY breaking, and inflation_ , _JHEP_ 12 (2004) 004 [hep-th/0411011].
  * [258] Planck collaboration, _Planck 2018 results. VI. Cosmological parameters_ , _Astron. Astrophys._ 641 (2020) A6 [1807.06209].
  * [259] Planck collaboration, _Planck 2018 results. X. Constraints on inflation_ , _Astron. Astrophys._ 641 (2020) A10 [1807.06211].
  * [260] L. Amendola et al., _Cosmology and fundamental physics with the Euclid satellite_ , _Living Rev. Rel._ 21 (2018) 2 [1606.00180].
  * [261] F. Denef, A. Hebecker and T. Wrase, _de Sitter swampland conjecture and the Higgs potential_ , _Phys. Rev. D_ 98 (2018) 086004 [1807.06581].
  * [262] S.K. Garg and C. Krishnan, _Bounds on Slow Roll and the de Sitter Swampland_ , _JHEP_ 11 (2019) 075 [1807.05193].
  * [263] H. Ooguri, E. Palti, G. Shiu and C. Vafa, _Distance and de Sitter Conjectures on the Swampland_ , _Phys. Lett. B_ 788 (2019) 180 [1810.05506].
  * [264] J. Louis, M. Rummel, R. Valandro and A. Westphal, _Building an explicit de Sitter_ , _JHEP_ 10 (2012) 163 [1208.3208].
  * [265] M. Cicoli, S. Krippendorf, C. Mayrhofer, F. Quevedo and R. Valandro, _D-Branes at del Pezzo Singularities: Global Embedding and Moduli Stabilisation_ , _JHEP_ 09 (2012) 019 [1206.5237].
  * [266] M. Cicoli, A. Maharana, F. Quevedo and C.P. Burgess, _De Sitter String Vacua from Dilaton-dependent Non-perturbative Effects_ , _JHEP_ 06 (2012) 011 [1203.1750].
  * [267] M. Cicoli, D. Klevers, S. Krippendorf, C. Mayrhofer, F. Quevedo and R. Valandro, _Explicit de Sitter Flux Vacua for Global String Models with Chiral Matter_ , _JHEP_ 05 (2014) 001 [1312.0014].
  * [268] J. Blåbäck, U. Danielsson and G. Dibitetto, _Fully stable dS vacua from generalised fluxes_ , _JHEP_ 08 (2013) 054 [1301.7073].
  * [269] A.P. Braun, M. Rummel, Y. Sumitomo and R. Valandro, _De Sitter vacua from a D-term generated racetrack potential in hypersurface Calabi-Yau compactifications_ , _JHEP_ 12 (2015) 033 [1509.06918].
  * [270] N. Cribiori, R. Kallosh, A. Linde and C. Roupec, _de Sitter Minima from M theory and String theory_ , _Phys. Rev. D_ 101 (2020) 046018 [1912.02791].
  * [271] I. Antoniadis, Y. Chen and G.K. Leontaris, _Logarithmic loop corrections, moduli stabilisation and de Sitter vacua in string theory_ , _JHEP_ 01 (2020) 149 [1909.10525].
  * [272] C. Crinò, F. Quevedo and R. Valandro, _On de Sitter String Vacua from Anti-D3-Branes in the Large Volume Scenario_ , _JHEP_ 03 (2021) 258 [2010.15903].
  * [273] M. Cicoli, I.n.G. Etxebarria, F. Quevedo, A. Schachner, P. Shukla and R. Valandro, _The Standard Model quiver in de Sitter string compactifications_ , _JHEP_ 08 (2021) 109 [2106.11964].
  * [274] M. Berg, M. Haack and B. Kors, _String loop corrections to Kahler potentials in orientifolds_ , _JHEP_ 11 (2005) 030 [hep-th/0508043].
  * [275] G. von Gersdorff and A. Hebecker, _Radius stabilization by two-loop Casimir energy_ , _Nucl. Phys. B_ 720 (2005) 211 [hep-th/0504002].
  * [276] I. Antoniadis, Y. Chen and G.K. Leontaris, _Perturbative moduli stabilisation in type IIB/F-theory framework_ , _Eur. Phys. J. C_ 78 (2018) 766 [1803.08941].
  * [277] J.M. Maldacena and C. Nunez, _Supergravity description of field theories on curved manifolds and a no go theorem_ , _Int. J. Mod. Phys. A_ 16 (2001) 822 [hep-th/0007018].
  * [278] S.R. Green, E.J. Martinec, C. Quigley and S. Sethi, _Constraints on String Cosmology_ , _Class. Quant. Grav._ 29 (2012) 075006 [1110.0545].
  * [279] F.F. Gautason, D. Junghans and M. Zagermann, _On Cosmological Constants from alpha’-Corrections_ , _JHEP_ 06 (2012) 029 [1204.0807].
  * [280] K. Dasgupta, R. Gwyn, E. McDonough, M. Mia and R. Tatar, _de Sitter Vacua in Type IIB String Theory: Classical Solutions and Quantum Corrections_ , _JHEP_ 07 (2014) 054 [1402.5112].
  * [281] D. Kutasov, T. Maxfield, I. Melnikov and S. Sethi, _Constraining de Sitter Space in String Theory_ , _Phys. Rev. Lett._ 115 (2015) 071305 [1504.00056].
  * [282] C. Quigley, _Gaugino Condensation and the Cosmological Constant_ , _JHEP_ 06 (2015) 104 [1504.00652].
  * [283] M. Dine, J.A.P. Law-Smith, S. Sun, D. Wood and Y. Yu, _Obstacles to Constructing de Sitter Space in String Theory_ , _JHEP_ 02 (2021) 050 [2008.12399].
  * [284] M. Montero, T. Van Riet and G. Venken, _A dS obstruction and its phenomenological consequences_ , _JHEP_ 05 (2020) 114 [2001.11023].
  * [285] I. Zlatev, L.-M. Wang and P.J. Steinhardt, _Quintessence, cosmic coincidence, and the cosmological constant_ , _Phys. Rev. Lett._ 82 (1999) 896 [astro-ph/9807002].
  * [286] H.E.S. Velten, R.F. vom Marttens and W. Zimdahl, _Aspects of the cosmological “coincidence problem”_ , _Eur. Phys. J. C_ 74 (2014) 3160 [1410.2509].
  * [287] F. Cunillera and A. Padilla, _A stringy perspective on the coincidence problem_ , _JHEP_ 10 (2021) 055 [2105.03426].
  * [288] M.P. Hertzberg, S. Kachru, W. Taylor and M. Tegmark, _Inflationary Constraints on Type IIA String Theory_ , _JHEP_ 12 (2007) 095 [0711.2512].
  * [289] S.K. Garg, C. Krishnan and M. Zaid Zaz, _Bounds on Slow Roll at the Boundary of the Landscape_ , _JHEP_ 03 (2019) 029 [1810.09406].
  * [290] B. Valeixo Bento, D. Chakraborty, S.L. Parameswaran and I. Zavala, _Dark Energy in String Theory_ , _PoS_ CORFU2019 (2020) 123 [2005.10168].
  * [291] C.M. Will, _The Confrontation between General Relativity and Experiment_ , _Living Rev. Rel._ 17 (2014) 4 [1403.7377].
  * [292] E.O. Colgáin and H. Yavartanoo, _Testing the Swampland: $H_{0}$ tension_, _Phys. Lett. B_ 797 (2019) 134907 [1905.02555].
  * [293] A. Banerjee, H. Cai, L. Heisenberg, E.O. Colgáin, M.M. Sheikh-Jabbari and T. Yang, _Hubble sinks in the low-redshift swampland_ , _Phys. Rev. D_ 103 (2021) L081305 [2006.00244].
  * [294] A.G. Riess et al., _A Comprehensive Measurement of the Local Value of the Hubble Constant with 1 km/s/Mpc Uncertainty from the Hubble Space Telescope and the SH0ES Team_ , 2112.04510.
  * [295] M. Cicoli, F. Muia and F.G. Pedro, _Microscopic Origin of Volume Modulus Inflation_ , _JCAP_ 12 (2015) 040 [1509.07748].
  * [296] M. Cicoli, F.G. Pedro and G. Tasinato, _Natural Quintessence in String Theory_ , _JCAP_ 07 (2012) 044 [1203.6655].
  * [297] Y. Akrami, R. Kallosh, A. Linde and V. Vardanyan, _The Landscape, the Swampland and the Era of Precision Cosmology_ , _Fortsch. Phys._ 67 (2019) 1800075 [1808.09440].
  * [298] P. Agrawal, G. Obied, P.J. Steinhardt and C. Vafa, _On the Cosmological Implications of the String Swampland_ , _Phys. Lett. B_ 784 (2018) 271 [1806.09718].
  * [299] F. Ferrer and J.A. Grifols, _Constraints on the mass of the superlight gravitino from the muon anomaly_ , _Phys. Rev. D_ 56 (1997) 7466 [hep-ph/9704444].
  * [300] A. Brignole, F. Feruglio and F. Zwirner, _Signals of a superlight gravitino at $e^{+}e^{-}$ colliders when the other superparticles are heavy_, _Nucl. Phys. B_ 516 (1998) 13 [hep-ph/9711516].
  * [301] A. Brignole, F. Feruglio, M.L. Mangano and F. Zwirner, _Signals of a superlight gravitino at hadron colliders when the other superparticles are heavy_ , _Nucl. Phys. B_ 526 (1998) 136 [hep-ph/9801329].
  * [302] M. Kawasaki, K. Kohri, T. Moroi and A. Yotsuyanagi, _Big-Bang Nucleosynthesis and Gravitino_ , _Phys. Rev. D_ 78 (2008) 065011 [0804.3745].
  * [303] C.P. Burgess, M. Cicoli, F. Quevedo and M. Williams, _Inflating with Large Effective Fields_ , _JCAP_ 11 (2014) 045 [1404.6236].
  * [304] C.P. Burgess, M. Cicoli, S. de Alwis and F. Quevedo, _Robust Inflation from Fibrous Strings_ , _JCAP_ 05 (2016) 032 [1603.06789].
  * [305] M. Cicoli, M. Goodsell and A. Ringwald, _The type IIB string axiverse and its low-energy phenomenology_ , _JHEP_ 10 (2012) 146 [1206.0819].
  * [306] N. Arkani-Hamed, L. Motl, A. Nicolis and C. Vafa, _The String landscape, black holes and gravity as the weakest force_ , _JHEP_ 06 (2007) 060 [hep-th/0601001].
  * [307] S. Dimopoulos, S. Kachru, J. McGreevy and J.G. Wacker, _N-flation_ , _JCAP_ 08 (2008) 003 [hep-th/0507205].
  * [308] N. Kaloper and L. Sorbo, _Of pngb quintessence_ , _JCAP_ 04 (2006) 007 [astro-ph/0511543].
  * [309] K. Choi, _String or M theory axion as a quintessence_ , _Phys. Rev. D_ 62 (2000) 043509 [hep-ph/9902292].
  * [310] N. Kaloper and L. Sorbo, _Where in the String Landscape is Quintessence_ , _Phys. Rev. D_ 79 (2009) 043528 [0810.5346].
  * [311] S. Panda, Y. Sumitomo and S.P. Trivedi, _Axions as Quintessence in String Theory_ , _Phys. Rev. D_ 83 (2011) 083506 [1011.5877].
  * [312] M. Kamionkowski, J. Pradler and D.G.E. Walker, _Dark energy from the string axiverse_ , _Phys. Rev. Lett._ 113 (2014) 251302 [1409.0549].
  * [313] Y. Olguin-Trejo, S.L. Parameswaran, G. Tasinato and I. Zavala, _Runaway Quintessence, Out of the Swampland_ , _JCAP_ 01 (2019) 031 [1810.08634].
  * [314] Particle Data Group collaboration, _Review of Particle Physics_ , _PTEP_ 2020 (2020) 083C01.
  * [315] D.J. Kapner, T.S. Cook, E.G. Adelberger, J.H. Gundlach, B.R. Heckel, C.D. Hoyle et al., _Tests of the gravitational inverse-square law below the dark-energy length scale_ , _Phys. Rev. Lett._ 98 (2007) 021101 [hep-ph/0611184].
  * [316] T. Damour and J.F. Donoghue, _Equivalence Principle Violations and Couplings of a Light Dilaton_ , _Phys. Rev. D_ 82 (2010) 084033 [1007.2792].
  * [317] B.S. Acharya, A. Maharana and F. Muia, _Hidden Sectors in String Theory: Kinetic Mixings, Fifth Forces and Quintessence_ , _JHEP_ 03 (2019) 048 [1811.10633].
  * [318] M. Cicoli, C.P. Burgess and F. Quevedo, _Anisotropic Modulus Stabilisation: Strings at LHC Scales with Micron-sized Extra Dimensions_ , _JHEP_ 10 (2011) 119 [1105.2107].
  * [319] C.P. Burgess and F. Quevedo, _Axion Homeopathy: Screening Dilaton Interactions_ , 2110.10352.
  * [320] C.P. Burgess, D. Dineen and F. Quevedo, _Yoga Dark Energy: Natural Relaxation and Other Dark Implications of a Supersymmetric Gravity Sector_ , 2111.07286.
  * [321] W. Buchmuller, K. Hamaguchi, O. Lebedev and M. Ratz, _Maximal temperature in flux compactifications_ , _JCAP_ 01 (2005) 004 [hep-th/0411109].
  * [322] L. Anguelova, V. Calo and M. Cicoli, _LARGE Volume String Compactifications at Finite Temperature_ , _JCAP_ 10 (2009) 025 [0904.0051].
  * [323] J.P. Conlon, R. Kallosh, A.D. Linde and F. Quevedo, _Volume Modulus Inflation and the Gravitino Mass Problem_ , _JCAP_ 09 (2008) 011 [0806.0809].
  * [324] J.P. Hsu, R. Kallosh and S. Prokushkin, _On brane inflation with volume stabilization_ , _JCAP_ 12 (2003) 009 [hep-th/0311077].
  * [325] L. Aparicio, M. Cicoli, S. Krippendorf, A. Maharana, F. Muia and F. Quevedo, _Sequestered de Sitter String Scenarios: Soft-terms_ , _JHEP_ 11 (2014) 071 [1409.1931].
  * [326] C. Pitrou, A. Coc, J.-P. Uzan and E. Vangioni, _Precision big bang nucleosynthesis with improved Helium-4 predictions_ , _Phys. Rept._ 754 (2018) 1 [1801.08023].
  * [327] S. Davidson, M. Losada and A. Riotto, _A New perspective on baryogenesis_ , _Phys. Rev. Lett._ 84 (2000) 4284 [hep-ph/0001301].
  * [328] T. He, S. Kachru and A. Westphal, _Gravity waves and the LHC: Towards high-scale inflation with low-energy SUSY_ , _JHEP_ 06 (2010) 065 [1003.4265].
  * [329] S. Hellerman, N. Kaloper and L. Susskind, _String theory and quintessence_ , _JHEP_ 06 (2001) 003 [hep-th/0104180].
  * [330] M. Cicoli, M. Kreuzer and C. Mayrhofer, _Toric K3-Fibred Calabi-Yau Manifolds with del Pezzo Divisors for String Compactifications_ , _JHEP_ 02 (2012) 002 [1107.0383].
  * [331] M. Cicoli, D. Ciupke, C. Mayrhofer and P. Shukla, _A Geometrical Upper Bound on the Inflaton Range_ , _JHEP_ 05 (2018) 001 [1801.05434].
  * [332] M. Cicoli, F. Quevedo and R. Valandro, _De Sitter from T-branes_ , _JHEP_ 03 (2016) 141 [1512.04558].
  * [333] M. Cicoli, C. Mayrhofer and R. Valandro, _Moduli Stabilisation for Chiral Global Models_ , _JHEP_ 02 (2012) 062 [1110.3333].
  * [334] M. Cicoli, S. Krippendorf, C. Mayrhofer, F. Quevedo and R. Valandro, _D3/D7 Branes at Singularities: Constraints from Global Embedding and Moduli Stabilisation_ , _JHEP_ 07 (2013) 150 [1304.0022].
  * [335] M. Cicoli, I.n. Garcìa-Etxebarria, C. Mayrhofer, F. Quevedo, P. Shukla and R. Valandro, _Global Orientifolded Quivers with Inflation_ , _JHEP_ 11 (2017) 134 [1706.06128].
  * [336] U.H. Danielsson and T. Van Riet, _What if string theory has no de Sitter vacua?_ , _Int. J. Mod. Phys. D_ 27 (2018) 1830007 [1804.01120].
  * [337] M. Cicoli, A. Schachner and P. Shukla, _Systematics of type IIB moduli stabilisation with odd axions_ , 2109.14624.
  * [338] M. Cicoli, V. Guidetti, N. Righi and A. Westphal, _Fuzzy Dark Matter Candidates from String Theory_ , 2110.02964.
  * [339] J. Shelton, W. Taylor and B. Wecht, _Nongeometric flux compactifications_ , _JHEP_ 10 (2005) 085 [hep-th/0508133].
  * [340] I. Benmachiche and T.W. Grimm, _Generalized N=1 orientifold compactifications and the Hitchin functionals_ , _Nucl. Phys. B_ 748 (2006) 200 [hep-th/0602241].
  * [341] E. Plauschinn, _Non-geometric backgrounds in string theory_ , _Phys. Rept._ 798 (2019) 1 [1811.11203].
  * [342] R.J. Scherrer and A.A. Sen, _Thawing quintessence with a nearly flat potential_ , _Phys. Rev. D_ 77 (2008) 083515 [0712.3450].
  * [343] S. Dutta and R.J. Scherrer, _Hilltop Quintessence_ , _Phys. Rev. D_ 78 (2008) 123525 [0809.4441].
  * [344] R.J. Hardwick, V. Vennin, C.T. Byrnes, J. Torrado and D. Wands, _The stochastic spectator_ , _JCAP_ 10 (2017) 018 [1701.06473].
  * [345] G.E. Uhlenbeck and L.S. Ornstein, _On the Theory of the Brownian Motion_ , _Phys. Rev._ 36 (1930) 823.
  * [346] Supernova Cosmology Project collaboration, _Measurements of $\Omega$ and $\Lambda$ from 42 high redshift supernovae_, _Astrophys. J._ 517 (1999) 565 [astro-ph/9812133].
  * [347] Supernova Search Team collaboration, _Observational evidence from supernovae for an accelerating universe and a cosmological constant_ , _Astron. J._ 116 (1998) 1009 [astro-ph/9805201].
  * [348] Planck collaboration, _Planck 2018 results. VI. Cosmological parameters_ , _Astron. Astrophys._ 641 (2020) A6 [1807.06209].
  * [349] E.J. Copeland, S. Ghataore, F. Niedermann and A. Padilla, _Generalised scalar-tensor theories and self-tuning_ , 2111.11448.
  * [350] E. Dudas, S. Ferrara, A. Kehagias and A. Sagnotti, _Properties of Nilpotent Supergravity_ , _JHEP_ 09 (2015) 217 [1507.07842].
  * [351] I. Bandos, L. Martucci, D. Sorokin and M. Tonin, _Brane induced supersymmetry breaking and de Sitter supergravity_ , _JHEP_ 02 (2016) 080 [1511.03024].
  * [352] S. Nagy, A. Padilla and I. Zavala, _The Super-Stückelberg procedure and dS in pure supergravity_ , _Proc. Roy. Soc. Lond. A_ 476 (2020) 20200035 [1910.14349].
  * [353] S. Bansal, S. Nagy, A. Padilla and I. Zavala, _Unimodular vs Nilpotent Superfield Approach to Pure dS Supergravity_ , _JHEP_ 01 (2021) 146 [2010.13758].
  * [354] B.A. Campbell, M.J. Duncan, N. Kaloper and K.A. Olive, _Gravitational dynamics with Lorentz Chern-Simons terms_ , _Nucl. Phys. B_ 351 (1991) 778.
  * [355] B. Zwiebach, _Curvature Squared Terms and String Theories_ , _Phys. Lett. B_ 156 (1985) 315.
  * [356] B. Zumino, _Gravity Theories in More Than Four-Dimensions_ , _Phys. Rept._ 137 (1986) 109.
  * [357] S. Cecotti, S. Ferrara, L. Girardello and M. Porrati, _Lorentz Chern-simons Terms in $N=1$ Four-dimensional Supergravity Consistent With Supersymmetry and String Compactification_, _Phys. Lett. B_ 164 (1985) 46.
  * [358] D.J. Gross and J.H. Sloan, _The Quartic Effective Action for the Heterotic String_ , _Nucl. Phys. B_ 291 (1987) 41.
  * [359] L. Alvarez-Gaume, P.H. Ginsparg, G.W. Moore and C. Vafa, _An O(16) x O(16) Heterotic String_ , _Phys. Lett. B_ 171 (1986) 155.
  * [360] K. Becker and L.-S. Tseng, _Heterotic flux compactifications and their moduli_ , _Nucl. Phys. B_ 741 (2006) 162 [hep-th/0509131].
  * [361] C. Charmousis and A. Padilla, _The Instability of Vacua in Gauss-Bonnet Gravity_ , _JHEP_ 12 (2008) 038 [0807.2864].
  * [362] D.J. Gross and E. Witten, _Superstring Modifications of Einstein’s Equations_ , _Nucl. Phys. B_ 277 (1986) 1.
  * [363] J. Brown, W. Cottrell, G. Shiu and P. Soler, _On Axionic Field Ranges, Loopholes and the Weak Gravity Conjecture_ , _JHEP_ 04 (2016) 017 [1504.00659].
  * [364] G. Shiu, P. Soler and W. Cottrell, _Weak Gravity Conjecture and extremal black holes_ , _Sci. China Phys. Mech. Astron._ 62 (2019) 110412 [1611.06270].
  * [365] E. Bergshoeff, D. Freedman, R. Kallosh and A. Van Proeyen, _Construction of the de Sitter supergravity_ , in _About Various Kinds of Interactions: Workshop in honour of Professor Philippe Spindel_ , 2, 2016 [1602.01678].
  * [366] J.P. Conlon and F. Revello, _Moduli Stabilisation and the Holographic Swampland_ , _LHEP_ 2020 (2020) 171 [2006.01021].
  * [367] J.P. Conlon, S. Ning and F. Revello, _Exploring The Holographic Swampland_ , 2110.06245.
  * [368] A. Michelson, _Light Waves and Their Uses_ , Chicago. University. The decennial publications. 2d series, University of Chicago Press (1903).
  * [369] R.H.L. Kelvin, _I. nineteenth century clouds over the dynamical theory of heat and light_ , _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ 2 (1901) 1 [https://doi.org/10.1080/14786440109462664].
  * [370] J. Louis and A. Micu, _Type 2 theories compactified on Calabi-Yau threefolds in the presence of background fluxes_ , _Nucl. Phys. B_ 635 (2002) 395 [hep-th/0202168].
  * [371] S. Hosono, A. Klemm and S. Theisen, _Lectures on mirror symmetry_ , _Lect. Notes Phys._ 436 (1994) 235 [hep-th/9403096].
  * [372] E. Palti, G. Tasinato and J. Ward, _WEAKLY-coupled IIA Flux Compactifications_ , _JHEP_ 06 (2008) 084 [0804.1248].


# Multi-view self-supervised learning for multivariate variable-channel time
series

###### Abstract

Labeling of multivariate biomedical time series data is a laborious and
expensive process. Self-supervised contrastive learning alleviates the need
for large, labeled datasets through pretraining on unlabeled data. However,
for multivariate time series data, the set of input channels often varies
between applications, and most existing work does not allow for transfer
between datasets with different sets of input channels. We propose learning
one encoder to operate on all input channels individually. We then use a
message passing neural network to extract a single representation across
channels. We demonstrate the potential of this method by pretraining our model
on a dataset with six EEG channels and then fine-tuning it on a dataset with
two different EEG channels. We compare models with and without the message
passing neural network across different contrastive loss functions. We show
that our method, combined with the TS2Vec loss, outperforms all other methods
in most settings.

Index Terms—  Self-supervised learning, Message passing neural networks,
Multi-view learning, Multivariate time series, Sleep staging

##  1 Introduction

In recent years, self-supervised learning has shown promising results in the
fields of computer vision and natural language processing [1, 2]. Self-
supervised learning relies on inherent patterns within the data to enable
pretraining on large, unlabeled datasets, thus facilitating the transfer of
learned structures to smaller labeled datasets, usually called the downstream
tasks. Obtaining ground truth scoring for biomedical signals such as
electroencephalography (EEG) often requires the expertise of multiple
professionals, rendering label acquisition a challenging and expensive
endeavor [3]. Consequently, self-supervised learning methods are particularly
interesting for biomedical time series data.

Many self-supervised learning methods use _contrastive learning_ to pretrain
the networks. Contrastive learning relies on having both positive and negative
pairs, where the positive pairs are encouraged to be close and the negative
pairs distant in representation space [4]. Non-contrastive self-supervised
learning tasks include the reconstruction of masked input pixels and loss
functions that only require positive views. In this work, we focus on
_contrastive self-supervised learning_.

Previous work on contrastive pretraining for time series data uses various
different strategies to create positive pairs. Broadly speaking, we divide the
strategies into three categories. The first category uses augmentations such
as masking, scaling, or random additive noise. The second category uses
contrastive predictive coding (CPC), where an autoregressive model is used to
predict future samples. A closely related strategy uses a combination of
masking and CPC to reconstruct masked out segments within the current
sequence. The third category relies on data that inherently contains multiple
views, such as multiple channels or different modalities. We refer to the
third strategy as a multi-view strategy.

Previous significant work on contrastive pretraining for time series data
includes Eldele et al. [5], who use augmentations such as permutations and
scaling. Furthermore, they use a temporal contrasting strategy similar to CPC
to predict future augmented samples. Zhang et al. [6] use similar
augmentations but create a separate encoder in the frequency domain and
encourage time and frequency representations to be close. Yue et al. [7] use
random cropping and masking to augment the input signal as well as a new
hierarchical time series loss to train their model, which they call TS2Vec.
BErt-like Neurophysiological Data Representation (BENDR) by Kostas et al. [8]
comprises a convolutional encoder that tokenizes raw input EEG, and a
transformer that contextualizes the tokens. The network is then trained using
a combination of CPC and masking. Kiyasseh et al. [9] and Deldari et al. [10]
both leverage the multi-view strategy for creating positive pairs. Kiyasseh et
al. [9] investigate contrastive pretraining for electrocardiography (ECG).
They use both neighboring samples in time and different channels as positive
pairs. Finally, Deldari et al. [10] use different sensor modalities as
positive pairs and present a new loss, COCOA, tailored for contrastive
learning in settings with more than one view. We focus our work on the multi-
view strategy for multivariate time series data.

A significant challenge for self-supervised learning applied on multivariate
time series is that the number of channels may vary from application to
application. The varying number of channels makes it difficult to transfer
between tasks with different channels [11], and few of the current methods
have a principled way of handling this issue. The mentioned previous work
either pretrain and fine-tune on the same dataset [5, 7, 10], or discard
excess channels or zero-pad missing channels during fine-tuning and/or
pretraining [6, 8]. The work most closely related to ours is SeqCLR by
Mohsenvand et al. [12]. SeqCLR is a single encoder that works separately on
all channels individually. The encoder is pretrained using augmentations.
During fine-tuning, the outputs of all input channels are concatenated and
used as input to the classifier.

We propose a channel agnostic network that generalizes between datasets with
varying sets of input channels with no further preprocessing. We learn a
single-channel encoder and add a message passing neural network (MPNN) after
the encoder to extract the optimal combination of the individual channel
representations. We use the different channels of the multivariate time series
to create the positive pairs during pretraining. We demonstrate the use of the
MPNN by pretraining on an EEG dataset with six channels and fine-tuning on an
EEG dataset with two different channels, and compare different loss functions
in the pretraining phase. Our results show that when combined with the TS2Vec
loss, our method outperforms all other methods on most sample sizes.

##  2 Methods

###  2.1 Channel agnostic setup

We use a convolutional encoder to extract representations from the raw EEG
signals. Our encoder follows the architecture in BENDR [8] with the exception
that we take only one channel as input. Given an input
$\boldsymbol{X}\in\mathbb{R}^{N\times C\times T_{\text{in}}}$ with $N$ samples
of raw EEG with $C$ channels, and each with a length of $T_{\text{in}}$, we
take each channel, ${\boldsymbol{x}}^{c}$, and apply the same encoder
$\operatorname{H}_{\boldsymbol{\theta}}$ to obtain the representation
${\boldsymbol{h}}^{c}$:

${\boldsymbol{h}}^{c}=\operatorname{H}_{\boldsymbol{\theta}}\left\lparen{\boldsymbol{x}}^{c}\right\rparen,\quad{\boldsymbol{x}}^{c}\in\mathbb{R}^{N\times
1\times T_{\text{in}}},\quad{\boldsymbol{h}}^{c}\in\mathbb{R}^{N\times L\times
T_{\text{\text{out}}}},$ (1)

where $L$ is the output dimension of the encoder and $T_{\text{out}}$ is the
length after downsampling in the encoder. This setup is visualized in Figure 1
for $C=6$ channels. Based on these $C$ representations, we use two different
methods for creating the different positive views ${\boldsymbol{z}}^{v}$ for
contrastive learning.

In the first approach, we simply use each representation
${\boldsymbol{h}}^{c}$ for each view:

${\boldsymbol{z}}^{v}={\boldsymbol{h}}^{c},\quad(c,v)\in\\{(1,1),\dots,(C,C)\\}.$
(2)

This approach results in $V$ positive views,
${\boldsymbol{z}}^{v}\in\mathbb{R}^{N\times L\times T_{\text{out}}}$. The $V$
positive views form $C(C-1)/2$ positive pairs per datapoint to use for
contrastive learning.

When fine-tuning on a downstream task with $C^{d}$ channels, this approach
produces $C^{d}$ representations,
$[{\boldsymbol{z}}^{1},\dots,{\boldsymbol{z}}^{C^{d}}]$. To obtain a single
representation across all channels, i.e., to use for classification, we add a
linear layer of size $C^{d}\times 1$, which combines the $C^{d}$
representations into one representation. The linear layer is optimized during
fine-tuning.

In the second approach, for each new batch, we randomly divide the $C$
representations into two groups, ${\boldsymbol{g}}_{1}$ and
${\boldsymbol{g}}_{2}$, with $C_{1}$, and $C_{2}=C-C_{1}$ representations
respectively, where $C_{1},C_{2}\geq 2$. Figure 2 shows an example
partitioning. The partitioning (both exact split and partition size) of the
representations is chosen randomly for each batch.

In both groups, we form a fully connected graph and use identical message
passing neural networks (MPNN) to extract the intra-group context for both
groups individually:

${\boldsymbol{z}}^{v}={\boldsymbol{z}}^{g_{v}}=\operatorname{MPNN}_{\phi}({\boldsymbol{g}}_{v}),v\in[1,2].$
(3)

This approach results in $V=2$ positive views,
${\boldsymbol{z}}^{v}\in\mathbb{R}^{N\times L\times T_{\text{out}}}$. The two
positive views form one positive pair for each data point to use for
contrastive learning.

Fig. 1: We apply the same encoder $\operatorname{H}_{\boldsymbol{\theta}}$ to
each of the $C$ input channels ${\boldsymbol{x}}^{c}$ to obtain $C$
representations ${\boldsymbol{h}}^{c}$. Here, the setup is shown for $C=6$
channels.

###  2.2 Message passing neural network

As stated in eq. (3), the MPNN is used to extract the intra-group context for
each of the two groups. MPNNs were originally formalized in [13] and we follow
their definition. An MPNN acts on graphs, and in our case, we form a fully
connected directed graph within both of the groups, ${\boldsymbol{g}}_{v}$.
This means that the input graph consists of $C_{v}$ vertices with vertex
features ${\boldsymbol{h}}$.

The MPNN consists of two phases, the message passing phase, and the readout
phase. The message passing phase takes place in $K$ rounds, defined by the
message passing networks $\operatorname{M}_{\phi_{k}}$, and an update
operation $\operatorname{U}_{k}$. In each round, we compute the message and
update the state for all ${\boldsymbol{h}}$ in ${\boldsymbol{g}}_{v}$:

$\displaystyle\boldsymbol{m}_{k+1}^{{\boldsymbol{h}}}$
$\displaystyle=\frac{1}{C_{v}-1}\quad\sum_{\mathclap{{\boldsymbol{h}}^{\prime}\in{\boldsymbol{g}}_{v}\backslash\\{{\boldsymbol{h}}\\}}}\operatorname{M}_{\phi_{k}}\left({\boldsymbol{h}}_{k},{\boldsymbol{h}}_{k}^{\prime}\right)$
(4) $\displaystyle{\boldsymbol{h}}_{k+1}$
$\displaystyle=\operatorname{U}_{k}\left\lparen{\boldsymbol{h}}_{k},\boldsymbol{m}_{k+1}^{\boldsymbol{h}}\right\rparen.$

We define
$\operatorname{U}_{k}\left\lparen{\boldsymbol{h}},\boldsymbol{m}^{\boldsymbol{h}}\right\rparen={\boldsymbol{h}}+\boldsymbol{m}^{\boldsymbol{h}}$
and use a neural network for each $\operatorname{M}_{\phi_{k}}$.
$\operatorname{M}_{\phi_{k}}$ acts on the concatenation of
${\boldsymbol{h}}_{k}$ and ${\boldsymbol{h}}_{k}^{\prime}$. Finally, the
readout phase computes the final representation across the graph according to:

${\boldsymbol{z}}^{{\boldsymbol{g}}_{v}}=\operatorname{R}_{\phi_{R}}\left\lparen\frac{1}{C_{v}}\sum_{{\boldsymbol{h}}\in{\boldsymbol{g}}_{v}}{\boldsymbol{h}}_{K}\right\rparen,$
(5)

where $\operatorname{R}_{\phi_{R}}$ is a neural network.

Since the same $\operatorname{M}_{\phi_{k}}$ is applied to all nodes at round
$k$ and $\operatorname{R}_{\phi_{R}}$ simply operates on the mean across all
final hidden states, the $\operatorname{MPNN}_{\phi}$ is able to compute the
intra-graph representation on graphs of arbitrary sizes. This enables us to
choose the size of the input graphs during pretraining randomly. Furthermore,
for a downstream task with $C^{d}$ channels, we can simply produce the
representations $[{\boldsymbol{h}}^{1},\dots,{\boldsymbol{h}}^{C^{d}}]$, form
a fully connected graph ${\boldsymbol{g}}$, and use the pretrained
$\operatorname{MPNN}_{\phi}$ to compute one final representation
${\boldsymbol{z}}^{\boldsymbol{g}}$.

Fig. 2: In our second multi-view setting, we split the $C$ representations
into two random groups consisting of at least two channels each. Here, group 1
consists of $C_{1}=2$ randomly chosen representations and group 2 ccomprises
the remaining $C_{2}=4$ representations.

###  2.3 Contrastive losses

For each of the two settings, we pretrain a neural network with three
different contrastive losses.
NT-Xent loss [1]: Given a batch of $N$ samples consisting of $V$ views, the
loss is computed pairwise for each pair of flattened views,
$\mathbf{z}^{v}\in\mathbb{R}^{N\times L\cdot T_{\text{out}}}$ and
$\mathbf{z}^{w}\in\mathbb{R}^{N\times L\cdot T_{\text{out}}}$, also referred
to as the positive pairs. The remaining $2N-1$ samples across both views are
used as negative examples. Let $\operatorname{s}_{{\tau\left\lparen
i,j\right\rparen}}^{\left\lparen
w,v\right\rparen}=\frac{{\boldsymbol{z}}_{i}^{w}\cdot{\boldsymbol{z}}_{j}^{v}}{\tau\|{\boldsymbol{z}}_{i}^{w}\|\cdot\|{\boldsymbol{z}}_{j}^{v}\|}$
denote the $\tau$-scaled cosine similarity between ${\boldsymbol{z}}_{i}^{w}$
and ${\boldsymbol{z}}_{j}^{v}$. The loss for one positive pair then becomes:

$\small\ell^{(w,v)}_{i}=\ln\frac{\exp\left\lparen\operatorname{s}_{{\tau\left\lparen
i,i\right\rparen}}^{\left\lparen
w,v\right\rparen}\right\rparen}{\sum\limits_{j}^{N}\exp\left\lparen\operatorname{s}_{{\tau\left\lparen
i,j\right\rparen}}^{\left\lparen
w,v\right\rparen}\right\rparen+\sum\limits_{j\neq
i}^{N}\exp\left\lparen\operatorname{s}_{{\tau\left\lparen
i,j\right\rparen}}^{\left\lparen w,w\right\rparen}\right\rparen}$ (6)

We compute $\ell^{(w,v)}$ for all positive pairs in the batch and average over
them:

$\small\mathcal{L}_{\text{NT-
Xent}}^{(w,v)}=-\frac{1}{N}\sum_{i}^{N}\ell_{i}^{(w,v)}$ (7)

This operation is repeated for all combinations of views:

$\small\mathcal{L}_{\text{NT-Xent}}=\frac{1}{V(V-1)}\sum_{v}^{V}\sum_{w\neq
v}^{V}\mathcal{L}^{(w,v)}_{\text{NT-Xent}}$ (8)

TS2Vec loss [7]: The TS2Vec loss also takes each combination of positive
pairs. However, instead of flattening the tokens produced by the encoder, the
loss takes the temporal relations in the representations into account. This is
done by constructing two different versions of the negative examples and using
these to compute a temporal loss and an instance loss, respectively. In the
temporal loss, the negative examples are the remaining time stamps within the
same sequence, $i$. In the instance loss, the negative examples are the
remaining sequences in the batch at the same time stamp, $t$:

$\displaystyle\ell t_{(i,t)}^{(w,v)}$
$\displaystyle=\ln\frac{\exp\left(\mathbf{z}_{i,t}^{w}\cdot\mathbf{z}_{i,t}^{v}\right)}{\sum\limits_{t^{\prime}}^{T_{\text{out}}}\exp\left(\mathbf{z}_{i,t}^{w}\cdot\mathbf{z}_{i,t^{\prime}}^{v}\right)+\sum\limits_{t^{\prime}\neq
t}^{T_{\text{out}}}\exp\left(\mathbf{z}^{w}_{i,t}\cdot\mathbf{z}^{w}_{i,t^{\prime}}\right)}$
(9) $\displaystyle\ell i_{(i,t)}^{(w,v)}$
$\displaystyle=\ln\frac{\exp\left(\mathbf{z}_{i,t}^{w}\cdot\mathbf{z}_{i,t}^{v}\right)}{\sum\limits_{j}^{N}\left(\exp\mathbf{z}^{w}_{i,t}\cdot\mathbf{z}_{j,t}^{v}\right)+\sum\limits_{j\neq
i}^{N}\exp\left(\mathbf{z}^{w}_{i,t}\cdot\mathbf{z}^{w}_{j,t}\right)}$

The temporal loss and instance loss are added to form the dual loss,
$\mathcal{L}^{(w,v)}_{\text{dual}}=-\frac{1}{2NT}\sum\limits_{i}^{N}\sum\limits_{t}^{T_{\text{out}}}\left(\ell
t_{(i,t)}^{(w,v)}+\ell i_{(i,t)}^{(w,v)}\right)$. The loss is then computed
hierarchically by iteratively applying a maxpool operation across the temporal
dimension of the representations and recomputing the dual loss to form
$\mathcal{L}^{(w,v)}_{\text{TS2Vec}}$ [7]. Finally, this loss is also computed
for all combinations of views:

$\small\mathcal{L}_{\text{TS2Vec}}=\frac{1}{V\cdot(V-1)}\sum_{v}^{V}\sum_{w\neq
v}^{V}\mathcal{L}_{\text{TS2Vec}}^{(w,v)}.$ (10)

COCOA loss [10]: The COCOA loss is meant to reduce the computational
complexity associated with NT-Xent when contrasting more than two views and
also acts on flattened versions of ${\boldsymbol{z}}$. The loss separately
computes the cross-view correlation (i.e., correlation between the positive
pairs) as:

$\small\mathcal{L}_{\text{C}}^{i}=\sum_{v}^{V}\sum_{w\neq
v}^{V}\exp\left\lparen 1/\tau-\operatorname{s}_{{\tau\left\lparen
i,i\right\rparen}}^{\left\lparen w,v\right\rparen}\right\rparen$ (11)

and the intra-view discriminator. The intra-view discriminator computes the
correlation between the negative examples. The negative examples are only
taken from the corresponding view, $v$, of the remaining examples in the
batch:

$\small\mathcal{L}_{\text{D}}^{v}=\frac{1}{N}\sum_{i}^{N}\sum_{j\neq
i}^{N}\exp\left\lparen\operatorname{s}_{{\tau\left\lparen
i,j\right\rparen}}^{\left\lparen v,v\right\rparen}\right\rparen.$ (12)

The cross-view correlation and intra-view discriminator are then combined into
the final loss:

$\small\mathcal{L}_{\text{COCOA}}=\sum_{i}^{N}\mathcal{L}_{\text{C}}^{i}+\lambda\sum_{v}^{V}\mathcal{L}_{\text{D}}^{v}$
(13)

##  3 Experimental setup

The implementation is available at
https://github.com/theabrusch/Multiview_TS_SSL.

###  3.1 Data

For pretraining, we use the Physionet Challenge 2018 (PC18) dataset [14, 15],
which is a dataset annotated for sleep staging. We use the EEG data from the
994 subjects of the training set to pretrain the models. The dataset contains
the following six EEG channels; F3-M2, F4-M1, C3-M2, C4-M1, O1-M2, and O2-M1.
All of the data is resampled from 200 Hz to 100 Hz. We split the subjects
0.8/0.2 for training and validating and then segment the entire dataset into
30s windows with no overlap. This results in 710,942 windows for pretraining
and 178,569 windows for tracking the validation performance.

For fine-tuning, we use the SleepEDFx dataset [15, 16]. The dataset contains
153 nights of sleep recordings from 78 subjects, and the data is annotated for
sleep staging. Sleep staging gives rise to the following five classes; wake,
N1, N2, N3 and R (the last four are different phases of sleep). The aim is to
predict the sleep stage for windows of length 30s. We use the EEG data, which
contains two channels; Fpz-Cz and Pz-Oz, sampled at 100Hz. We split the
subjects 0.6/0.2/0.2 for training, validating, and testing. The splits are
kept fixed throughout all experiments. Again, we segment the data into 30s
windows with no overlap, yielding 122,016 and 37,379 windows available for
training and validation and 36,955 windows for testing. In practice, we
downsample the number of training and validation windows to simulate a setting
with only a few labels available for fine-tuning. This process is described in
Section 3.4.

All windows in the pretraining and fine-tuning datasets are standardized, so
each channel has zero mean and a standard deviation of one.

###  3.2 Model architecture

We follow [8] and use 6 convolutional blocks consisting of a 1D convolution, a
dropout layer, a group normalization layer, and a GELU activation function.
The kernel width is 3 in the first layer and 2 in the remaining 5 layers, and
the stride is set to the same value as the width. We use 256 kernels for all
intermediate layers and set the output dimension of the final layer to 64.
Finally, we add a readout layer with kernel width and stride set to 1. This
gives an output dimension ${\boldsymbol{h}}^{c}\in\mathbb{R}^{N\times
L=64\times T_{\text{out}}=33}$.

For the MPNN, we use a single linear layer followed by a dropout layer and a
ReLU activation layer for all $\operatorname{M}_{\phi_{k}}$. The linear layer
only acts on the second dimension of ${\boldsymbol{h}}^{c}$, i.e. the same
weights are applied at all time steps $t\in T_{\text{out}}$. Thus, since it
takes in two hidden states at a time, the dimension of the weights are $2\cdot
64\times 64$. For $\operatorname{R}_{\phi_{R}}$, we use two linear layers
separated by a dropout layer and a ReLU activation function.

###  3.3 Pretraining setup

During pretraining, all of our models are trained for 10 epochs. We use the
AdamW optimizer with a learning rate of $10^{-3}$ and a weight decay of
$10^{-2}$. We apply a dropout rate of 10% between all layers in the network.
All of the pretrained models are trained using a batch size of 64.

We benchmark our results against BENDR [8] and SeqCLR [12]. For BENDR, we use
the original hyperparameters with no additional fine-tuning. Since our input
to the model is smaller than what was used in the original paper, it is likely
that more optimal masking parameters exist. Following their code, the
pretraining is stopped if the network learns to precisely reconstruct tokens.
This happens in our version of BENDR after 5900 iterations (around halfway
into the first epoch).

For SeqCLR, we pretrain a version of their recurrent neural network
(SeqCLR_R), since this is reported to show the best results on the sleep
staging dataset. We pretrain on windows of size 30s since this yielded better
results than the 20s reported in the paper. We adjust their augmentations to
our sampling frequency and input size. All implementation details are in the
Git repository.

###  3.4 Fine-tuning

When fine-tuning without the MPNN, we use a linear layer across all
${\boldsymbol{z}}^{c}$ to obtain one representation
${\boldsymbol{z}}\in\mathbb{R}^{N\times 64\times 33}$ for classification. With
the MPNN setup, we simply use the pretrained $\operatorname{MPNN}_{\phi}$ to
obtain one representation across all channels. Subsequently, we average pool
along the time dimension to obtain $T=4$ and flatten the representation, i.e.,
${\boldsymbol{z}}^{\text{final}}\in\mathbb{R}^{N\times 4\cdot 64=256}$. We
then use a single linear layer followed by a softmax operation that classifies
each window.

All of our models are fine-tuned with a learning rate of $5\cdot 10^{-4}$. We
use the AdamW optimizer with a weight decay of $10^{-2}$. The batch size is
set to 32. As we are generally interested in settings with few labels
available for fine-tuning, we test the model by sampling a balanced set from
the full dataset available for fine-tuning. We sample 10, 25, 50, 100, 200,
500, and 1000 data points per class respectively, and compare the performance
of each of the pretrained models with the same models trained from scratch. We
sample the same number of data points from the validation set. All models are
fine-tuned for a maximum of 40 epochs, using early stopping on the validation
loss with a patience of 7 epochs. We do this for both of the following
settings: one where we optimize the entire network during fine-tuning and one
where we only optimize the final linear layer(s) during fine-tuning.

Since the BENDR encoder acts on a fixed input dimension, it is less trivial to
fine-tune on a dataset with a different set of input channels. We insert the
channels of the fine-tuning dataset at the position of the closest channel in
the pretraining dataset. Therefore, we insert the channel Fpz-Cz at the
position of both the F3-M3 and the F4-M1 channels, and insert the channel Pz-
Oz in the same position as O1-M1 and O2-M1. We insert zeros at the positions
of the remaining channels.

##  4 Results and discussion

We run the fine-tuning experiments for five different seeds (i.e., both the
data sampling and model initialization are reseeded five times) and report the
averaged scores.

Figure 3 (top) shows the results when optimizing the entire network during
fine-tuning. Table 1 shows a subset of the sample sizes and also includes the
results from BENDR and the different versions of SeqCLR. All scores reported
are balanced accuracy scores. Since it is a five-class problem, the chance
level is 20%. The table shows that the two networks trained from scratch yield
similar results. However, the MPNN model has a lower score for fewer samples
and higher scores for more samples compared to the non-MPNN model. This is
likely due to the higher amount of trainable parameters in the MPNN model,
which makes it more likely to overfit on small sample sizes. It is also clear
that all of the pretraining schemes improve the score across all sample sizes.
Both BENDR and SeqCLR_R improve the results over the models trained from
scratch, but at a lower margin compared to the remaining models. We also
pretrained a model with the same encoder architecture as our own models, but
using the SeqCLR augmentations for contrastive learning. This model showed
similar results as the SeqCLR_R model, indicating that the multi-view
pretraining strategy is beneficial when transferring between tasks with
variable input channels.

Comparing the pretrained MPNN models to the pretrained non-MPNN models, the
picture is less clear. Where the MPNN model trained with the COCOA loss
outperforms the non-MPNN model on smaller sample sizes, the MPNN+NT-Xent model
performs worse than its non-MPNN counterpart on all sample sizes. Nonetheless,
both Table 1 and Figure 3 clearly demonstrate that the MPNN model trained with
the TS2Vec loss outperforms all other pretraining schemes at all sample sizes.
This is especially the case for the smallest sample size where the margin to
the second highest score is 12.5%.

Fig. 3: Balanced accuracy scores when optimizing the entire network (top) and freezing the encoder (bottom) during fine-tuning. Scores are averaged across 5 seeds. Model | Pretraining | Samples per class
---|---|---
|  | 10 | 50 | 100 | 1000
BENDR | BENDR | $.284$ | $.494$ | $.543$ | $.657$
SeqCLR_R | NT-Xent | $.308$ | $.460$ | $.559$ | $.643$
SeqCLR_R | Scratch | $.234$ | $.362$ | $.398$ | $.564$
Wo. MPNN | COCOA | $.416$ | $.549$ | $.598$ | $.682$
Wo. MPNN | NT-Xent | $.443$ | $.581$ | $.622$ | $.698$
Wo. MPNN | TS2Vec | $.485$ | $.605$ | $.625$ | $.719$
Wo. MPNN | Scratch | $.268$ | $.355$ | $.426$ | $.624$
W. MPNN | COCOA | $.479$ | $.622$ | $.631$ | $.675$
W. MPNN | NT-Xent | $.388$ | $.550$ | $.595$ | $.689$
W. MPNN | TS2Vec | $\mathbf{.610}$ | $\mathbf{.656}$ | $\mathbf{.666}$ | $\mathbf{.725}$
W. MPNN | Scratch | $.225$ | $.335$ | $.441$ | $.643$
Table 1: Balanced accuracy scores after optimizing the entire network during
fine-tuning averaged across 5 seeds.

It therefore seems that the MPNN clearly improves the pretraining when
combined with a loss that explicitly considers the temporal relation in the
data.

Model | Pretraining | Samples per class
---|---|---
|  | 10 | 50 | 100 | 1000
BENDR | BENDR | $.201$ | $.215$ | $.227$ | $.263$
SeqCLR_R | NT-Xent | $.256$ | $.305$ | $.308$ | $.577$
SeqCLR_R | Scratch | $.278$ | $.325$ | $.338$ | $.363$
Wo. MPNN | COCOA | $.346$ | $.463$ | $.488$ | $.505$
Wo. MPNN | NT-Xent | $.355$ | $.493$ | $.522$ | $.563$
Wo. MPNN | TS2Vec | $.403$ | $.555$ | $.573$ | $\mathbf{.658}$
Wo. MPNN | Scratch | $.196$ | $.203$ | $.204$ | $.273$
W. MPNN | COCOA | $.414$ | $.467$ | $.466$ | $.477$
W. MPNN | NT-Xent | $.379$ | $.458$ | $.462$ | $.459$
W. MPNN | TS2Vec | $\mathbf{.483}$ | $\mathbf{.584}$ | $\mathbf{.601}$ | $.644$
W. MPNN | Scratch | $.203$ | $.214$ | $.224$ | $.260$
Table 2: Balanced accuracy scores after freezing the encoder and the MPNN
during fine-tuning averaged across 5 seeds.

Table 2 and Figure 3 (bottom) show the results when freezing the encoder and
MPNN during fine-tuning. Since we only optimize linear layers for these
results, it is clear that the representations learned during pretraining for
almost all models are transferable to a dataset with completely different
channels. However, the BENDR results are comparable to the results of the
randomly initialized models. The table shows that while the MPNN+COCOA loss
and MPNN+NT-Xent models achieve a slightly higher score than their non-MPNN
counterparts on the smallest sample size, they perform worse on all other
sample sizes. The MPNN+TS2Vec model again outperforms all other models for
sample sizes smaller than 200 samples per class, whereas the non-MPNN+TS2Vec
model achieves the highest performance for 200 samples or more per class.

Thus, the results indicate that the pretrained MPNN helps in optimally
combining the two channels for smaller sample sizes. When more data is
available, the non-MPNN models are able to learn a better combination using
the linear layer that is also optimized during fine-tuning. When fine-tuning
on two channels, the complexity of inter-channel interactions is limited. We
hypothesize that the pretrained MPNN is even more useful when fine-tuning on
datasets with more than two channels and thus increased inter-channel
complexity.

Finally, we reiterate that the pretraining of BENDR was not optimized for our
dataset. It is therefore likely that more optimal pretraining settings exist.
However, the results still demonstrate the issue with existing pretraining
schemes, where transferring between datasets with varying input channels is
non-trivial. The methods presented here alleviate this issue.

##  5 Conclusions

Self-supervised learning for multivariate time series suffers from the
limitation that the input variables may vary from pretraining task to
downstream task. Therefore, we proposed a channel-agnostic pretraining scheme
applying the same encoder to all incoming channels and combining the channels
using an MPNN. We compared our method to a network trained without the MPNN
and the corresponding models with no pretraining and repeated our experiments
for three different contrastive loss functions. We demonstrated the capability
of the model by pretraining on a dataset with six EEG channels using a multi-
view strategy for contrastive learning and fine-tuning on a dataset with two
different EEG channels. We also compared to a model pretrained using
augmentations for contrastive learning.

Our results showed that the MPNN model trained with a TS2Vec multi-view loss
outperformed all other methods at all sample sizes when the entire network was
optimized during fine-tuning. The same pattern was repeated when freezing the
pretrained network during fine-tuning, although the MPNN+TS2Vec model was
slightly outperformed by the non-MPNN+TS2Vec model at larger sample sizes. Our
results demonstrated the potential of MPNNs combined with the multi-view
strategy in creating a channel-agnostic pretraining scheme allowing for great
flexibility when transferring between variable-channel datasets.

## References

  * [1] Ting Chen et al.,  “A simple framework for contrastive learning of visual representations,”  in Proceedings of the 37th International Conference on Machine Learning, 2020.
  * [2] Jacob Devlin et al.,  “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  * [3] Magdy Younes,  “The case for using digital eeg analysis in clinical sleep medicine,”  Sleep Science and Practice, vol. 1, no. 1, 2017.
  * [4] R. Hadsell et al.,  “Dimensionality reduction by learning an invariant mapping,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2006\.
  * [5] Emadeldeen Eldele et al.,  “Time-series representation learning via temporal and contextual contrasting,”  in Proceedings the 30th International Joint Conference on Artificial Intelligence, 2021.
  * [6] Xiang Zhang et al.,  “Self-supervised contrastive pre-training for time series via time-frequency consistency,”  in Advances in Neural Information Processing Systems, 2022.
  * [7] Zhihan Yue et al.,  “TS2Vec: Towards universal representation of time series,”  Proceedings of the AAAI Conference on Artificial Intelligence, 2022\.
  * [8] Demetres Kostas et al.,  “BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data,”  Frontiers in Human Neuroscience, vol. 15, 2021.
  * [9] Dani Kiyasseh et al.,  “Clocs: Contrastive learning of cardiac signals across space, time, and patients,”  in International Conference on Machine Learning, 2021.
  * [10] Shohreh Deldari et al.,  “COCOA: Cross modality contrastive learning for sensor data,”  Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 6, no. 3, 2022.
  * [11] Antoine Guillot and Valentin Thorey,  “Robustsleepnet: Transfer learning for automated sleep staging at scale,”  IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, 2021.
  * [12] Mostafa Neo Mohsenvand et al.,  “Contrastive representation learning for electroencephalogram classification,”  in Proceedings of the Machine Learning for Health NeurIPS Workshop, 2020.
  * [13] Justin Gilmer et al.,  “Neural message passing for quantum chemistry,”  in Proceedings of the 34th International Conference on Machine Learning, 2017.
  * [14] Mohammad M. Ghassemi et al.,  “You snooze, you win: the physionet/computing in cardiology challenge 2018,”  in Computing in Cardiology Conference, 2018, vol. 45.
  * [15] A. L. Goldberger et al.,  “PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals,”  Circulation, vol. 101, no. 23, 2000.
  * [16] B. Kemp et al.,  “Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg,”  IEEE Transactions on Biomedical Engineering, vol. 47, no. 9, 2000\.