$\xi_{f,\rm t,s}$ parameters in DHOST theories in Section 9.11.

In Horndeski theories, we obtain $\xi_{f}=\gamma=0.485^{+0.967}_{-0.708}$ and
$\xi_{\rm t}=5.151^{+6.112}_{-4.016}$ at the $1\sigma$ level, and
$-1.216<\gamma<2.175$ and $-2.098<\xi_{\rm t}$ at the $95\%$ confidence level.
The $\gamma$ constraint in Horndeski theories obtained here is not directly
comparable to those obtained from existing studies by, e.g., Gil-Marín et al.
(2017b) because we simultaneously vary the $\xi_{\rm t}$ parameter in
Horndeski theories.

  12. 12.

We have shown that the anisotropic component of the 3PCF contributes
significantly to the constraints on the shape of the non-linear velocity field
in Section 9.10. In particular, the constraints on the parameters $\xi_{\rm
t}$ and $\xi_{\rm s}$ are $\sim 35$ and $\sim 20$ times better when the
anisotropic component is added than when only the isotropic component is
considered. This result strongly supports the main claim of this paper that
the anisotropic three-point statistics should be considered to test the non-
linearity of modified gravity theories.

Below is a summary of some of the concerns and future enhancements to the
results of this paper.

  1. 1.

In order to encourage the future development of the anisotropic 3PCF analysis,
we comment on the situation beyond the assumptions used to derive the non-
linear effects of DHOST theories that we focus on in this paper (see Section
2.2). First, our analysis can be applied to other modified gravity theories,
such as $f(R)$ gravity models and brane-world models. In addition, it should
also be possible to constrain effects such as the CDM-baryon relative velocity
and massive neutrinos, which give rise to characteristic non-linear behaviour.
The calculations of DHOST theories in this paper assume minimal coupling
between the metric field and the scalar field, Gaussianity of the initial
conditions, and the quasi-static limit, but we need additional correction
terms if these assumptions are removed. In addition, since DHOST theories
modify the shift term from GR, we cannot exclude the possibility of shift
bias, which we do not consider in a $\Lambda$CDM model. In the presence of
shift bias, we cannot use the $\xi_{\rm s}$ and $\xi_{\rm t}$ parameters to
constrain DHOST theories, but we expect the $E_{\rm s}$ and $E_{\rm t}$
parameters constrained at each redshift to remain valid (Section 9.12).

  2. 2.

We also comment on some improvements in our analysis of the anisotropic 3PCF
(see Section 3.5). First, as more mock catalogues are created in the future,
increasing the number of multipoles in the 3PCF to be considered should
improve the results of this work (e.g., Byun & Krause, 2022). Second, as shown
in Figure 10, we can dramatically improve the current parameter constraints by
using the theoretical model of the 3PCF, which is applicable to small scales
(see Section 7.4). Third, although we have used the shape of the linear power
spectrum calculated by an $\Lambda$CDM model in a high-$z$ region in this
work, it needs to be calculated in the framework of DHOST theories in the
future (e.g., Hiramatsu & Yamauchi, 2020). Fourth, we have calculated the
Gaussian function describing the damping effect of the BAO signal for a
$\Lambda$CDM model, but we also need to constrain this function itself.
Finally, we have neglected the Alcock-Paczyński (AP) effect in this work; the
analysis method of the anisotropic 3PCF that includes the AP effect has been
established by Sugiyama et al. (2021) using the Patchy mock and should be
straightforward to apply to actual galaxy data. We hope that addressing these
issues will further improve our results.

Finally, in Appendix A we provide the software package that can reproduce all
the results obtained in this paper, HITOMI. The aim of HITOMI is to make
available all the programs we have used to complete the anisotropic 3PCF
analysis, from downloading the SDSS DR12 galaxy data, measuring the 2PCFs and
3PCFs, computing the theoretical models, calculating the covariance matrices,
the window function corrections, MCMC analysis, and producing figures and
tables. This makes it easier for any user to see how partial improvements to
HITOMI, e.g. improved 3PCF model calculations, feed through to the final
parameter constraints. Furthermore, by replacing the BOSS galaxy data used in
HITOMI, our analysis can be easily applied to future galaxy surveys such as
DESI (DESI Collaboration et al., 2016), Euclid (Laureijs et al., 2011), and
PFS (Takada et al., 2014).

## Acknowledgements

NSS acknowledges financial support from JSPS KAKENHI Grant Number 19K14703.
Numerical computations were carried out on Cray XC50 at Center for
Computational Astrophysics, National Astronomical Observatory of Japan. The
work of SH was supported by JSPS KAKENHI Grants No. JP21H01080. The work of TK
was supported by JSPS KAKENHI Grant No. JP20K03936 and MEXT-JSPS Grant-in-Aid
for Transformative Research Areas (A) “Extreme Universe”, No. JP21H05182 and
No. JP21H05189. The work of DY was supported in part by JSPS KAKENHI Grants
No. 19H01891, No. 22K03627. SS acknowledges the support for this work from
NSF-2219212. SS is supported in part by World Premier International Research
Center Initiative (WPI Initiative), MEXT, Japan. H-JS is supported by the U.S.
Department of Energy, Office of Science, Office of High Energy Physics under
DE-SC0019091 and DE-SC0023241. This project has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation program (grant agreement 853291). FB is a University
Research Fellow.

## Data Availability

The data underlying this article are available at the SDSS data base
(https://www.sdss.org/dr12/).

## References

  * Abbott et al. (2017a) Abbott B. P., et al., 2017a, Phys. Rev. Lett., 119, 161101
  * Abbott et al. (2017b) Abbott B. P., et al., 2017b, Astrophys. J. Lett., 848, L13
  * Agarwal et al. (2021) Agarwal N., Desjacques V., Jeong D., Schmidt F., 2021, J. Cosmology Astropart. Phys., 2021, 021
  * Aghanim et al. (2020) Aghanim N., et al., 2020, Astron. Astrophys., 641, A6
  * Alam et al. (2015) Alam S., et al., 2015, Astrophys. J. Suppl., 219, 12
  * Alam et al. (2017) Alam S., et al., 2017, Mon. Not. Roy. Astron. Soc., 470, 2617
  * Alam et al. (2021a) Alam S., et al., 2021a, Phys. Rev. D, 103, 083533
  * Alam et al. (2021b) Alam S., et al., 2021b, J. Cosmology Astropart. Phys., 2021, 050
  * Albrecht & Steinhardt (1982) Albrecht A., Steinhardt P. J., 1982, Phys. Rev. Lett., 48, 1220
  * Alcock & Paczyński (1979) Alcock C., Paczyński B., 1979, Nature, 281, 358
  * Alvarez et al. (2014) Alvarez M., et al., 2014, arXiv e-prints, p. arXiv:1412.4671
  * Amendola et al. (2018) Amendola L., Kunz M., Saltas I. D., Sawicki I., 2018, Phys. Rev. Lett., 120, 131101
  * Amendola et al. (2020) Amendola L., Bettoni D., Pinho A. M., Casas S., 2020, Universe, 6, 20
  * Anderson et al. (2014) Anderson L., et al., 2014, Mon. Not. Roy. Astron. Soc., 441, 24
  * Arai & Nishizawa (2018) Arai S., Nishizawa A., 2018, Phys. Rev. D, 97, 104038
  * Arai et al. (2022) Arai S., et al., 2022, arXiv e-prints, p. arXiv:2212.09094
  * Audren et al. (2013) Audren B., Lesgourgues J., Benabed K., Prunet S., 2013, JCAP, 1302, 001
  * Aviles et al. (2018) Aviles A., Rodriguez-Meza M. A., De-Santiago J., Cervantes-Cota J. L., 2018, JCAP, 11, 013
  * Babichev & Deffayet (2013) Babichev E., Deffayet C., 2013, Class. Quant. Grav., 30, 184001
  * Babichev & Lehébel (2018) Babichev E., Lehébel A., 2018, JCAP, 12, 027
  * Babichev et al. (2016) Babichev E., Koyama K., Langlois D., Saito R., Sakstein J., 2016, Class. Quant. Grav., 33, 235014
  * Baker et al. (2017) Baker T., Bellini E., Ferreira P. G., Lagos M., Noller J., Sawicki I., 2017, Phys. Rev. Lett., 119, 251301
  * Baker et al. (2021) Baker T., et al., 2021, Rev. Mod. Phys., 93, 015003
  * Baldauf et al. (2012) Baldauf T., Seljak U., Desjacques V., McDonald P., 2012, Phys. Rev. D, 86, 083540
  * Baldauf et al. (2015a) Baldauf T., Mirbabayi M., Simonović M., Zaldarriaga M., 2015a, Phys. Rev. D, 92, 043514
  * Baldauf et al. (2015b) Baldauf T., Mercolli L., Mirbabayi M., Pajer E., 2015b, J. Cosmology Astropart. Phys., 2015, 007
  * Barreira et al. (2014) Barreira A., Li B., Baugh C., Pascoli S., 2014, JCAP, 08, 059
  * Barreira et al. (2016) Barreira A., Sánchez A. G., Schmidt F., 2016, Phys. Rev. D, 94, 084022
  * Barriga & Gaztañaga (2002) Barriga J., Gaztañaga E., 2002, MNRAS, 333, 443
  * Barriga & Gaztanaga (2002) Barriga J., Gaztanaga E., 2002, Mon. Not. Roy. Astron. Soc., 333, 443
  * Bartolo et al. (2013) Bartolo N., Bellini E., Bertacca D., Matarrese S., 2013, JCAP, 03, 034
  * Baumgart & Fry (1991) Baumgart D. J., Fry J. N., 1991, ApJ, 375, 25
  * Bean et al. (1983) Bean A. J., Ellis R. S., Shanks T., Efstathiou G., Peterson B. A., 1983, MNRAS, 205, 605
  * Bellini & Zumalacarregui (2015) Bellini E., Zumalacarregui M., 2015, Phys. Rev. D, 92, 063522
  * Bellini et al. (2015) Bellini E., Jimenez R., Verde L., 2015, JCAP, 05, 057
  * Bellini et al. (2016) Bellini E., Cuesta A. J., Jimenez R., Verde L., 2016, JCAP, 02, 053
  * Beltran Jimenez et al. (2016) Beltran Jimenez J., Piazza F., Velten H., 2016, Phys. Rev. Lett., 116, 061101
  * Ben Achour et al. (2016a) Ben Achour J., Crisostomi M., Koyama K., Langlois D., Noui K., Tasinato G., 2016a, JHEP, 12, 100
  * Ben Achour et al. (2016b) Ben Achour J., Langlois D., Noui K., 2016b, Phys. Rev. D, 93, 124005
  * Berezhiani & Khoury (2014) Berezhiani L., Khoury J., 2014, JCAP, 09, 018
  * Bernardeau & Brax (2011) Bernardeau F., Brax P., 2011, J. Cosmology Astropart. Phys., 2011, 019
  * Bernardeau et al. (2002) Bernardeau F., Colombi S., Gaztanaga E., Scoccimarro R., 2002, Phys. Rept., 367, 1
  * Bernardeau et al. (2012) Bernardeau F., Van de Rijt N., Vernizzi F., 2012, Phys. Rev. D, 85, 063509
  * Bernardeau et al. (2013) Bernardeau F., Van de Rijt N., Vernizzi F., 2013, Phys. Rev. D, 87, 043530
  * Beutler et al. (2014) Beutler F., et al., 2014, MNRAS, 443, 1065
  * Beutler et al. (2016) Beutler F., Blake C., Koda J., Marin F., Seo H.-J., Cuesta A. J., Schneider D. P., 2016, Mon. Not. Roy. Astron. Soc., 455, 3230
  * Beutler et al. (2017) Beutler F., et al., 2017, Mon. Not. Roy. Astron. Soc., 466, 2242
  * Bharadwaj et al. (2020) Bharadwaj S., Mazumdar A., Sarkar D., 2020, MNRAS, 493, 594
  * Bianchi et al. (2015) Bianchi D., Gil-Marín H., Ruggeri R., Percival W. J., 2015, MNRAS, 453, L11
  * Blas et al. (2011) Blas D., Lesgourgues J., Tram T., 2011, JCAP, 07, 034
  * Blas et al. (2013) Blas D., Garny M., Konstandin T., 2013, JCAP, 09, 024
  * Blas et al. (2016a) Blas D., Garny M., Ivanov M. M., Sibiryakov S., 2016a, JCAP, 07, 028
  * Blas et al. (2016b) Blas D., Garny M., Ivanov M. M., Sibiryakov S., 2016b, JCAP, 07, 052
  * Bolton et al. (2012) Bolton A. S., et al., 2012, Astron. J., 144, 144
  * Borisov & Jain (2009) Borisov A., Jain B., 2009, Phys. Rev. D, 79, 103506
  * Bose & Koyama (2016) Bose B., Koyama K., 2016, JCAP, 08, 032
  * Bose & Taruya (2018) Bose B., Taruya A., 2018, JCAP, 10, 019
  * Bose et al. (2017) Bose B., Koyama K., Hellwing W. A., Zhao G.-B., Winther H. A., 2017, Phys. Rev. D, 96, 023519
  * Bose et al. (2018) Bose B., Koyama K., Lewandowski M., Vernizzi F., Winther H. A., 2018, JCAP, 04, 063
  * Bose et al. (2020a) Bose B., Byun J., Lacasa F., Moradinezhad Dizgah A., Lombriser L., 2020a, JCAP, 02, 025
  * Bose et al. (2020b) Bose B., Cataneo M., Tröster T., Xia Q., Heymans C., Lombriser L., 2020b, Mon. Not. Roy. Astron. Soc., 498, 4650
  * Bouchet et al. (1992) Bouchet F. R., Juszkiewicz R., Colombi S., Pellat R., 1992, Astrophys. J. Lett., 394, L5
  * Bouchet et al. (1995) Bouchet F. R., Colombi S., Hivon E., Juszkiewicz R., 1995, A&A, 296, 575
  * Brinckmann & Lesgourgues (2019) Brinckmann T., Lesgourgues J., 2019, Physics of the Dark Universe, 24, 100260
  * Burrage et al. (2019) Burrage C., Dombrowski J., Saadeh D., 2019, JCAP, 10, 023
  * Byun & Krause (2022) Byun J., Krause E., 2022, MNRAS,
  * Cabass et al. (2022a) Cabass G., Ivanov M. M., Philcox O. H. E., Simonović M., Zaldarriaga M., 2022a, Phys. Rev. D, 106, 043506
  * Cabass et al. (2022b) Cabass G., Ivanov M. M., Philcox O. H. E., Simonović M., Zaldarriaga M., 2022b, Phys. Rev. Lett., 129, 021301
  * Calles et al. (2020) Calles J., Castiblanco L., Noreña J., Stahl C., 2020, JCAP, 07, 033
  * Capozziello & Francaviglia (2008) Capozziello S., Francaviglia M., 2008, Gen. Rel. Grav., 40, 357
  * Carlstrom et al. (2019) Carlstrom J., et al., 2019, in Bulletin of the American Astronomical Society. p. 209 (arXiv:1908.01062)
  * Castiblanco et al. (2019) Castiblanco L., Gannouji R., Noreña J., Stahl C., 2019, JCAP, 07, 030
  * Cataneo & Rapetti (2018) Cataneo M., Rapetti D., 2018, Int. J. Mod. Phys. D, 27, 1848006
  * Cataneo et al. (2019) Cataneo M., Lombriser L., Heymans C., Mead A., Barreira A., Bose S., Li B., 2019, Mon. Not. Roy. Astron. Soc., 488, 2121
  * Chiang et al. (2015) Chiang C.-T., Wagner C., Sánchez A. G., Schmidt F., Komatsu E., 2015, J. Cosmology Astropart. Phys., 2015, 028
  * Chiba et al. (2020) Chiba T., Chibana F., Yamaguchi M., 2020, JCAP, 06, 003
  * Chibana et al. (2019) Chibana F., Kimura R., Yamaguchi M., Yamauchi D., Yokoyama S., 2019, JCAP, 10, 049
  * Child et al. (2018) Child H. L., Slepian Z., Takada M., 2018, arXiv e-prints, p. arXiv:1811.12396
  * Clarkson et al. (2019) Clarkson C., de Weerd E. M., Jolicoeur S., Maartens R., Umeh O., 2019, MNRAS, 486, L101
  * Coulton et al. (2023) Coulton W. R., et al., 2023, ApJ, 943, 178
  * Creminelli & Vernizzi (2017) Creminelli P., Vernizzi F., 2017, Phys. Rev. Lett., 119, 251302
  * Creminelli & Zaldarriaga (2004) Creminelli P., Zaldarriaga M., 2004, JCAP, 10, 006
  * Creminelli et al. (2013) Creminelli P., Noreña J., Simonović M., Vernizzi F., 2013, JCAP, 12, 025
  * Creminelli et al. (2014a) Creminelli P., Gleyzes J., Simonović M., Vernizzi F., 2014a, JCAP, 02, 051
  * Creminelli et al. (2014b) Creminelli P., Gleyzes J., Hui L., Simonović M., Vernizzi F., 2014b, J. Cosmology Astropart. Phys., 2014, 009
  * Creminelli et al. (2018) Creminelli P., Lewandowski M., Tambalo G., Vernizzi F., 2018, JCAP, 12, 025
  * Creminelli et al. (2019) Creminelli P., Tambalo G., Vernizzi F., Yingcharoenrat V., 2019, JCAP, 10, 072
  * Crisostomi & Koyama (2018) Crisostomi M., Koyama K., 2018, Phys. Rev. D, 97, 021301
  * Crisostomi et al. (2016) Crisostomi M., Koyama K., Tasinato G., 2016, JCAP, 04, 044
  * Crisostomi et al. (2019) Crisostomi M., Lewandowski M., Vernizzi F., 2019, Phys. Rev. D, 100, 024025
  * Crisostomi et al. (2020) Crisostomi M., Lewandowski M., Vernizzi F., 2020, Phys. Rev. D, 101, 123501
  * Crocce & Scoccimarro (2008) Crocce M., Scoccimarro R., 2008, Phys. Rev. D, 77, 023533
  * Cusin et al. (2018a) Cusin G., Lewandowski M., Vernizzi F., 2018a, JCAP, 04, 005
  * Cusin et al. (2018b) Cusin G., Lewandowski M., Vernizzi F., 2018b, JCAP, 04, 061
  * D’Amico et al. (2022a) D’Amico G., Lewandowski M., Senatore L., Zhang P., 2022a, arXiv e-prints, p. arXiv:2201.11518
  * D’Amico et al. (2022b) D’Amico G., Donath Y., Lewandowski M., Senatore L., Zhang P., 2022b, arXiv e-prints, p. arXiv:2206.08327
  * DESI Collaboration et al. (2016) DESI Collaboration et al., 2016, arXiv e-prints, p. arXiv:1611.00036
  * Dalal et al. (2010) Dalal N., Pen U.-L., Seljak U., 2010, JCAP, 11, 007
  * Dawson et al. (2013) Dawson K. S., et al., 2013, Astron. J., 145, 10
  * Dawson et al. (2016) Dawson K. S., et al., 2016, Astron. J., 151, 44
  * Deffayet et al. (2011) Deffayet C., Gao X., Steer D. A., Zahariade G., 2011, Phys. Rev. D, 84, 064039
  * Desjacques & Seljak (2010) Desjacques V., Seljak U., 2010, Classical and Quantum Gravity, 27, 124011
  * Desjacques et al. (2018a) Desjacques V., Jeong D., Schmidt F., 2018a, JCAP, 12, 035
  * Desjacques et al. (2018b) Desjacques V., Jeong D., Schmidt F., 2018b, Phys. Rept., 733, 1
  * Desjacques et al. (2018c) Desjacques V., Jeong D., Schmidt F., 2018c, J. Cosmology Astropart. Phys., 2018, 035
  * Dima & Vernizzi (2018) Dima A., Vernizzi F., 2018, Phys. Rev. D, 97, 101302
  * Dinda (2018) Dinda B. R., 2018, JCAP, 06, 017
  * Dodelson & Schneider (2013) Dodelson S., Schneider M. D., 2013, Phys. Rev., D88, 063537
  * Doi et al. (2010) Doi M., et al., 2010, Astron. J., 139, 1628
  * Doré et al. (2014) Doré O., et al., 2014, arXiv e-prints, p. arXiv:1412.4872
  * Dvali et al. (2000) Dvali G. R., Gabadadze G., Porrati M., 2000, Phys. Lett. B, 485, 208
  * Efstathiou & Jedrzejewski (1994) Efstathiou G., Jedrzejewski R. I., 1994, Adv. Space Res., 3, 379
  * Eggemeier et al. (2019) Eggemeier A., Scoccimarro R., Smith R. E., 2019, Phys. Rev. D, 99, 123514
  * Eggemeier et al. (2021) Eggemeier A., Scoccimarro R., Smith R. E., Crocce M., Pezzotta A., Sánchez A. G., 2021, Phys. Rev. D, 103, 123550
  * Eisenstein & Hu (1998) Eisenstein D. J., Hu W., 1998, Astrophys. J., 496, 605
  * Eisenstein et al. (2007a) Eisenstein D. J., Seo H.-j., White M. J., 2007a, Astrophys. J., 664, 660
  * Eisenstein et al. (2007b) Eisenstein D. J., Seo H.-j., Sirko E., Spergel D., 2007b, Astrophys. J., 664, 675
  * Eisenstein et al. (2011) Eisenstein D. J., et al., 2011, Astron. J., 142, 72
  * Esposito et al. (2019) Esposito A., Hui L., Scoccimarro R., 2019, Phys. Rev. D, 100, 043536
  * Ezquiaga & Zumalacárregui (2017) Ezquiaga J. M., Zumalacárregui M., 2017, Phys. Rev. Lett., 119, 251304
  * Feldman et al. (1994) Feldman H. A., Kaiser N., Peacock J. A., 1994, ApJ, 426, 23
  * Feldman et al. (2001) Feldman H. A., Frieman J. A., Fry J. N., Scoccimarro R., 2001, Phys. Rev. Lett., 86, 1434
  * Ferreira (2019) Ferreira P. G., 2019, Ann. Rev. Astron. Astrophys., 57, 335
  * Fosalba & Gaztanaga (1998) Fosalba P., Gaztanaga E., 1998, Mon. Not. Roy. Astron. Soc., 301, 503
  * Fosalba et al. (2005) Fosalba P., Pan J., Szapudi I., 2005, ApJ, 632, 29
  * Frieman & Gaztañaga (1999) Frieman J. A., Gaztañaga E., 1999, ApJ, 521, L83
  * Frieman & Gaztanaga (1994) Frieman J. A., Gaztanaga E., 1994, ApJ, 425, 392
  * Frigo & Johnson (2005) Frigo M., Johnson S. G., 2005, Proceedings of the IEEE, 93, 216
  * Frusciante & Perenon (2020) Frusciante N., Perenon L., 2020, Phys. Rept., 857, 1
  * Fry (1984) Fry J. N., 1984, ApJ, 279, 499
  * Fry (1994) Fry J. N., 1994, Phys. Rev. Lett., 73, 215
  * Fry & Peebles (1980) Fry J. N., Peebles P. J. E., 1980, ApJ, 238, 785
  * Fry & Scherrer (1994) Fry J. N., Scherrer R. J., 1994, Astrophys. J., 429, 36
  * Fry & Seldner (1982) Fry J. N., Seldner M., 1982, ApJ, 259, 474
  * Fujita & Vlah (2020) Fujita T., Vlah Z., 2020, J. Cosmology Astropart. Phys., 2020, 059
  * Fukugita et al. (1996) Fukugita M., Ichikawa T., Gunn J. E., Doi M., Shimasaku K., Schneider D. P., 1996, Astron. J., 111, 1748
  * Gagrani & Samushia (2017) Gagrani P., Samushia L., 2017, MNRAS, 467, 928
  * Ganz & Noui (2020) Ganz A., Noui K., 2020, arXiv e-prints, p. arXiv:2007.01063
  * Gaztañaga & Scoccimarro (2005) Gaztañaga E., Scoccimarro R., 2005, MNRAS, 361, 824
  * Gaztañaga et al. (2005) Gaztañaga E., Norberg P., Baugh C. M., Croton D. J., 2005, MNRAS, 364, 620
  * Gaztañaga et al. (2009) Gaztañaga E., Cabré A., Castander F., Crocce M., Fosalba P., 2009, MNRAS, 399, 801
  * Gelman & Rubin (1992) Gelman A., Rubin D. B., 1992, Statistical Science, 7, 457
  * Gil-Marín et al. (2011) Gil-Marín H., Schmidt F., Hu W., Jimenez R., Verde L., 2011, J. Cosmology Astropart. Phys., 2011, 019
  * Gil-Marín et al. (2012) Gil-Marín H., Wagner C., Fragkoudi F., Jimenez R., Verde L., 2012, J. Cosmology Astropart. Phys., 2012, 047
  * Gil-Marín et al. (2014) Gil-Marín H., Wagner C., Noreña J., Verde L., Percival W., 2014, J. Cosmology Astropart. Phys., 2014, 029
  * Gil-Marín et al. (2015a) Gil-Marín H., Noreña J., Verde L., Percival W. J., Wagner C., Manera M., Schneider D. P., 2015a, MNRAS, 451, 539
  * Gil-Marín et al. (2015b) Gil-Marín H., et al., 2015b, MNRAS, 452, 1914
  * Gil-Marín et al. (2017a) Gil-Marín H., Percival W. J., Verde L., Brownstein J. R., Chuang C.-H., Kitaura F.-S., Rodríguez-Torres S. A., Olmstead M. D., 2017a, MNRAS, 465, 1757
  * Gil-Marín et al. (2017b) Gil-Marín H., Percival W. J., Verde L., Brownstein J. R., Chuang C.-H., Kitaura F.-S., Rodríguez-Torres S. A., Olmstead M. D., 2017b, Mon. Not. Roy. Astron. Soc., 465, 1757
  * Gleyzes et al. (2015a) Gleyzes J., Langlois D., Piazza F., Vernizzi F., 2015a, JCAP, 02, 018
  * Gleyzes et al. (2015b) Gleyzes J., Langlois D., Piazza F., Vernizzi F., 2015b, Phys. Rev. Lett., 114, 211101
  * Goldstein et al. (2022) Goldstein S., Esposito A., Philcox O. H. E., Hui L., Hill J. C., Scoccimarro R., Abitbol M. H., 2022, Phys. Rev. D, 106, 123525
  * Goroff et al. (1986) Goroff M. H., Grinstein B., Rey S. J., Wise M. B., 1986, ApJ, 311, 6
  * Gott et al. (1991) Gott J. R. I., Gao B., Park C., 1991, ApJ, 383, 90
  * Greig et al. (2013) Greig B., Komatsu E., Wyithe J. S. B., 2013, MNRAS, 431, 1777
  * Groth & Peebles (1977) Groth E. J., Peebles P. J. E., 1977, Astrophys. J., 217, 385
  * Gualdi & Verde (2020) Gualdi D., Verde L., 2020, J. Cosmology Astropart. Phys., 2020, 041
  * Gunn et al. (1998) Gunn J. E., et al., 1998, Astron. J., 116, 3040
  * Gunn et al. (2006) Gunn J. E., et al., 2006, Astron. J., 131, 2332
  * Guo & Jing (2009) Guo H., Jing Y. P., 2009, ApJ, 698, 479
  * Guo et al. (2014) Guo H., Li C., Jing Y. P., Börner G., 2014, ApJ, 780, 139
  * Guth (1981) Guth A. H., 1981, Phys. Rev., D23, 347
  * Hahn (2005) Hahn T., 2005, Computer Physics Communications, 168, 78
  * Hale-Sutton et al. (1989) Hale-Sutton D., Fong R., Metcalfe N., Shanks T., 1989, MNRAS, 237, 569
  * Hamilton (1997) Hamilton A. J. S., 1997, in Ringberg Workshop on Large Scale Structure Ringberg, Germany, September 23-28, 1996. (arXiv:astro-ph/9708102), doi:10.1007/978-94-011-4960-0_17
  * Hamilton (2000) Hamilton A. J. S., 2000, MNRAS, 312, 257
  * Hand et al. (2017) Hand N., Li Y., Slepian Z., Seljak U., 2017, J. Cosmology Astropart. Phys., 2017, 002
  * Hartlap et al. (2007) Hartlap J., Simon P., Schneider P., 2007, A&A, 464, 399
  * Hashimoto et al. (2017) Hashimoto I., Rasera Y., Taruya A., 2017, Phys. Rev. D, 96, 043526
  * Heavens et al. (2017) Heavens A. F., Sellentin E., de Mijolla D., Vianello A., 2017, MNRAS, 472, 4244
  * Hellwing et al. (2013) Hellwing W. A., Li B., Frenk C. S., Cole S., 2013, Mon. Not. Roy. Astron. Soc., 435, 2806
  * Hernández-Aguayo et al. (2019) Hernández-Aguayo C., Hou J., Li B., Baugh C. M., Sánchez A. G., 2019, Mon. Not. Roy. Astron. Soc., 485, 2194
  * Hiramatsu (2022) Hiramatsu T., 2022, J. Cosmology Astropart. Phys., 2022, 035
  * Hiramatsu & Yamauchi (2020) Hiramatsu T., Yamauchi D., 2020, Phys. Rev. D, 102, 083525
  * Hirano et al. (2018) Hirano S., Kobayashi T., Tashiro H., Yokoyama S., 2018, Phys. Rev. D, 97, 103517
  * Hirano et al. (2019a) Hirano S., Kobayashi T., Yamauchi D., Yokoyama S., 2019a, Phys. Rev. D, 99, 104051
  * Hirano et al. (2019b) Hirano S., Kobayashi T., Yamauchi D., 2019b, Phys. Rev. D, 99, 104073
  * Hirano et al. (2020) Hirano S., Kobayashi T., Yamauchi D., Yokoyama S., 2020, Phys. Rev. D, 102, 103505
  * Hivon et al. (1995) Hivon E., Bouchet F. R., Colombi S., Juszkiewicz R., 1995, A&A, 298, 643
  * Hockney & Eastwood (1981) Hockney R. W., Eastwood J. W., 1981, Computer Simulation Using Particles
  * Hoffmann et al. (2015) Hoffmann K., Bel J., Gaztañaga E., Crocce M., Fosalba P., Castander F. J., 2015, MNRAS, 447, 1724
  * Hoffmann et al. (2018) Hoffmann K., Gaztañaga E., Scoccimarro R., Crocce M., 2018, MNRAS, 476, 814
  * Horndeski (1974) Horndeski G. W., 1974, Int. J. Theor. Phys., 10, 363
  * Hu & Sawicki (2007) Hu W., Sawicki I., 2007, Phys. Rev. D, 76, 064004
  * Ishak (2019) Ishak M., 2019, Living Rev. Rel., 22, 1
  * Ivanov & Sibiryakov (2018) Ivanov M. M., Sibiryakov S., 2018, JCAP, 07, 053
  * Ivanov et al. (2020) Ivanov M. M., Simonović M., Zaldarriaga M., 2020, J. Cosmology Astropart. Phys., 2020, 042
  * Ivanov et al. (2022) Ivanov M. M., Philcox O. H. E., Nishimichi T., Simonović M., Takada M., Zaldarriaga M., 2022, Phys. Rev. D, 105, 063512
  * Ivanov et al. (2023) Ivanov M. M., Philcox O. H. E., Cabass G., Nishimichi T., Simonović M., Zaldarriaga M., 2023, Phys. Rev. D, 107, 083515
  * Jain & Bertschinger (1996) Jain B., Bertschinger E., 1996, Astrophys. J., 456, 43
  * Jain et al. (2016) Jain R. K., Kouvaris C., Nielsen N. G., 2016, Phys. Rev. Lett., 116, 151103
  * Jing (2005) Jing Y. P., 2005, ApJ, 620, 559
  * Jing & Boerner (1997) Jing Y. P., Boerner G., 1997, A&A, 318, 667
  * Jing & Borner (1998) Jing Y. P., Borner G., 1998, Astrophys. J., 503, 37
  * Jing & Börner (2004) Jing Y. P., Börner G., 2004, ApJ, 607, 140
  * Jing & Zhang (1989) Jing Y.-P., Zhang J.-L., 1989, ApJ, 342, 639
  * Jing et al. (1991) Jing Y. P., Mo H. J., Boerner G., 1991, A&A, 252, 449
  * Jolicoeur et al. (2017) Jolicoeur S., Umeh O., Maartens R., Clarkson C., 2017, J. Cosmology Astropart. Phys., 2017, 040
  * Jolicoeur et al. (2018) Jolicoeur S., Umeh O., Maartens R., Clarkson C., 2018, J. Cosmology Astropart. Phys., 2018, 036
  * Kaiser (1987) Kaiser N., 1987, Mon. Not. Roy. Astron. Soc., 227, 1
  * Kamalinejad & Slepian (2020) Kamalinejad F., Slepian Z., 2020, arXiv e-prints, p. arXiv:2011.00899
  * Karagiannis et al. (2018) Karagiannis D., Lazanu A., Liguori M., Raccanelli A., Bartolo N., Verde L., 2018, MNRAS, 478, 1341
  * Karagiannis et al. (2022) Karagiannis D., Maartens R., Randrianjanahary L. F., 2022, J. Cosmology Astropart. Phys., 2022, 003
  * Kase & Tsujikawa (2019) Kase R., Tsujikawa S., 2019, Int. J. Mod. Phys. D, 28, 1942005
  * Kase & Tsujikawa (2020a) Kase R., Tsujikawa S., 2020a, JCAP, 11, 032
  * Kase & Tsujikawa (2020b) Kase R., Tsujikawa S., 2020b, Phys. Rev. D, 101, 063511
  * Kayo et al. (2004) Kayo I., et al., 2004, PASJ, 56, 415
  * Kehagias & Riotto (2013) Kehagias A., Riotto A., 2013, Nucl. Phys. B, 873, 514
  * Kimura et al. (2012) Kimura R., Kobayashi T., Yamamoto K., 2012, Phys. Rev. D, 85, 024023
  * Kimura et al. (2018) Kimura R., Suyama T., Yamaguchi M., Yamauchi D., Yokoyama S., 2018, Publ. Astron. Soc. Jap., 70, Publications of the Astronomical Society of Japan, Volume 70, Issue 5, 1 October 2018, L5, https://doi.org/10.1093/pasj/psy083
  * Kitaura et al. (2016) Kitaura F.-S., et al., 2016, Mon. Not. Roy. Astron. Soc., 456, 4156
  * Klypin et al. (2016) Klypin A., Yepes G., Gottlober S., Prada F., Hess S., 2016, Mon. Not. Roy. Astron. Soc., 457, 4340
  * Kobayashi (2019) Kobayashi T., 2019, Rept. Prog. Phys., 82, 086901
  * Kobayashi et al. (2011) Kobayashi T., Yamaguchi M., Yokoyama J., 2011, Prog. Theor. Phys., 126, 511
  * Kobayashi et al. (2015) Kobayashi T., Watanabe Y., Yamauchi D., 2015, Phys. Rev. D, 91, 064013
  * Kobayashi et al. (2022) Kobayashi Y., Nishimichi T., Takada M., Miyatake H., 2022, Phys. Rev. D, 105, 083517
  * Koyama (2016) Koyama K., 2016, Rept. Prog. Phys., 79, 046902
  * Koyama & Sakstein (2015) Koyama K., Sakstein J., 2015, Phys. Rev. D, 91, 124066
  * Koyama et al. (2009) Koyama K., Taruya A., Hiramatsu T., 2009, Phys. Rev. D, 79, 123512
  * Koyama et al. (2018) Koyama K., Umeh O., Maartens R., Bertacca D., 2018, JCAP, 07, 050
  * Kreisch & Komatsu (2018) Kreisch C. D., Komatsu E., 2018, JCAP, 12, 030
  * Kulkarni et al. (2007) Kulkarni G. V., Nichol R. C., Sheth R. K., Seo H.-J., Eisenstein D. J., Gray A., 2007, MNRAS, 378, 1196
  * Kuruvilla & Porciani (2020) Kuruvilla J., Porciani C., 2020, J. Cosmology Astropart. Phys., 2020, 043
  * Lange et al. (2022) Lange J. U., Hearin A. P., Leauthaud A., van den Bosch F. C., Guo H., DeRose J., 2022, MNRAS, 509, 1779
  * Langlois (2017) Langlois D., 2017, in 52nd Rencontres de Moriond on Gravitation. pp 221–228 (arXiv:1707.03625)
  * Langlois (2019) Langlois D., 2019, Int. J. Mod. Phys. D, 28, 1942006
  * Langlois & Noui (2016) Langlois D., Noui K., 2016, JCAP, 02, 034
  * Langlois et al. (2017) Langlois D., Mancarella M., Noui K., Vernizzi F., 2017, JCAP, 05, 033
  * Langlois et al. (2018) Langlois D., Saito R., Yamauchi D., Noui K., 2018, Phys. Rev. D, 97, 061501
  * Langlois et al. (2020) Langlois D., Noui K., Roussille H., 2020, arXiv e-prints, p. arXiv:2012.10218
  * Laureijs et al. (2011) Laureijs R., et al., 2011, arXiv e-prints, p. arXiv:1110.3193
  * Lazanu & Liguori (2018) Lazanu A., Liguori M., 2018, J. Cosmology Astropart. Phys., 2018, 055
  * Lazanu et al. (2016) Lazanu A., Giannantonio T., Schmittfull M., Shellard E. P. S., 2016, Phys. Rev. D, 93, 083517
  * Lazeyras et al. (2016) Lazeyras T., Wagner C., Baldauf T., Schmidt F., 2016, J. Cosmology Astropart. Phys., 2016, 018
  * Levi & Vlah (2016) Levi M., Vlah Z., 2016, arXiv e-prints, p. arXiv:1605.09417
  * Lewandowski (2020) Lewandowski M., 2020, JCAP, 08, 044
  * Lewandowski & Senatore (2017) Lewandowski M., Senatore L., 2017, JCAP, 08, 037
  * Lewandowski & Senatore (2020) Lewandowski M., Senatore L., 2020, JCAP, 03, 018
  * Lewandowski et al. (2015) Lewandowski M., Perko A., Senatore L., 2015, JCAP, 05, 019
  * Liguori et al. (2010) Liguori M., Sefusatti E., Fergusson J. R., Shellard E. P. S., 2010, Advances in Astronomy, 2010, 980523
  * Linde (1982) Linde A. D., 1982, Phys. Lett., B108, 389
  * Lombriser & Lima (2017) Lombriser L., Lima N. A., 2017, Phys. Lett. B, 765, 382
  * Lombriser & Taylor (2016) Lombriser L., Taylor A., 2016, JCAP, 03, 031
  * Ma & Fry (2000) Ma C.-P., Fry J. N., 2000, ApJ, 543, 503
  * Maldacena (2003) Maldacena J. M., 2003, JHEP, 05, 013
  * Marin (2011) Marin F., 2011, Astrophys. J., 737, 97
  * Marín et al. (2008) Marín F. A., Wechsler R. H., Frieman J. A., Nichol R. C., 2008, ApJ, 672, 849
  * Marin et al. (2013) Marin F. A., et al., 2013, Mon. Not. Roy. Astron. Soc., 432, 2654
  * Martin (2012) Martin J., 2012, Comptes Rendus Physique, 13, 566
  * Matarrese et al. (1997) Matarrese S., Verde L., Heavens A. F., 1997, Mon. Not. Roy. Astron. Soc., 290, 651
  * Matsubara (2004) Matsubara T., 2004, ApJ, 615, 573
  * Matsubara (2008) Matsubara T., 2008, Phys. Rev., D77, 063530
  * Mazumdar et al. (2020) Mazumdar A., Bharadwaj S., Sarkar D., 2020, MNRAS, 498, 3975
  * McBride et al. (2011a) McBride C. K., Connolly A. J., Gardner J. P., Scranton R., Newman J. A., Scoccimarro R., Zehavi I., Schneider D. P., 2011a, ApJ, 726, 13
  * McBride et al. (2011b) McBride C. K., Connolly A. J., Gardner J. P., Scranton R., Scoccimarro R., Berlind A. A., Marín F., Schneider D. P., 2011b, ApJ, 739, 85
  * McCullagh et al. (2016) McCullagh N., Jeong D., Szalay A. S., 2016, MNRAS, 455, 2945
  * McDonald (2009) McDonald P., 2009, JCAP, 11, 026
  * Melville & Noller (2020) Melville S., Noller J., 2020, Phys. Rev. D, 101, 021502
  * Moradinezhad Dizgah et al. (2021) Moradinezhad Dizgah A., Biagetti M., Sefusatti E., Desjacques V., Noreña J., 2021, J. Cosmology Astropart. Phys., 2021, 015
  * Moresco et al. (2017b) Moresco M., et al., 2017b, A&A, 604, A133
  * Moresco et al. (2017a) Moresco M., et al., 2017a, Astron. Astrophys., 604, A133
  * Moresco et al. (2021) Moresco M., Veropalumbo A., Marulli F., Moscardini L., Cimatti A., 2021, ApJ, 919, 144
  * Mueller et al. (2018) Mueller E.-M., Percival W., Linder E., Alam S., Zhao G.-B., Sánchez A. G., Beutler F., Brinkmann J., 2018, Mon. Not. Roy. Astron. Soc., 475, 2122
  * Munshi & McEwen (2020) Munshi D., McEwen J. D., 2020, Mon. Not. Roy. Astron. Soc., 498, 5299
  * Munshi & Regan (2017) Munshi D., Regan D., 2017, J. Cosmology Astropart. Phys., 2017, 042
  * Munshi et al. (2020a) Munshi D., McEwen J. D., Kitching T., Fosalba P., Teyssier R., Stadel J., 2020a, JCAP, 05, 043
  * Munshi et al. (2020b) Munshi D., Namikawa T., Kitching T. D., McEwen J. D., Takahashi R., Bouchet F. R., Taruya A., Bose B., 2020b, Mon. Not. Roy. Astron. Soc., 493, 3985
  * Namikawa et al. (2018) Namikawa T., Bouchet F. R., Taruya A., 2018, Phys. Rev. D, 98, 043530
  * Namikawa et al. (2019) Namikawa T., Bose B., Bouchet F. R., Takahashi R., Taruya A., 2019, Phys. Rev. D, 99, 063511
  * Nan et al. (2018) Nan Y., Yamamoto K., Hikage C., 2018, J. Cosmology Astropart. Phys., 2018, 038
  * Narikawa et al. (2013) Narikawa T., Kobayashi T., Yamauchi D., Saito R., 2013, Phys. Rev. D, 87, 124006
  * Nichol et al. (2006) Nichol R. C., et al., 2006, MNRAS, 368, 1507
  * Nishimichi et al. (2007) Nishimichi T., Kayo I., Hikage C., Yahata K., Taruya A., Jing Y. P., Sheth R. K., Suto Y., 2007, PASJ, 59, 93
  * Nojiri & Odintsov (2011) Nojiri S., Odintsov S. D., 2011, Phys. Rep., 505, 59
  * Nojiri et al. (2017) Nojiri S., Odintsov S. D., Oikonomou V. K., 2017, Phys. Rep., 692, 1
  * Noller (2020) Noller J., 2020, Phys. Rev. D, 101, 063524
  * Noller & Nicola (2019) Noller J., Nicola A., 2019, Phys. Rev. D, 99, 103502
  * Noller & Nicola (2020) Noller J., Nicola A., 2020, Phys. Rev. D, 102, 104045
  * Oddo et al. (2020) Oddo A., Sefusatti E., Porciani C., Monaco P., Sánchez A. G., 2020, J. Cosmology Astropart. Phys., 2020, 056
  * Oddo et al. (2021) Oddo A., Rizzo F., Sefusatti E., Porciani C., Monaco P., 2021, J. Cosmology Astropart. Phys., 2021, 038
  * Okada et al. (2013) Okada H., Totani T., Tsujikawa S., 2013, Phys. Rev. D, 87, 103002
  * Ostrogradsky (1850) Ostrogradsky M., 1850, Mem. Acad. St. Petersbourg, 6, 385
  * Pace et al. (2021) Pace F., Battye R. A., Bellini E., Lombriser L., Vernizzi F., Bolliet B., 2021, J. Cosmology Astropart. Phys., 2021, 017
  * Pan & Szapudi (2005) Pan J., Szapudi I., 2005, MNRAS, 362, 1363
  * Pan et al. (2007) Pan J., Coles P., Szapudi I., 2007, MNRAS, 382, 1460
  * Pardede et al. (2022) Pardede K., Rizzo F., Biagetti M., Castorina E., Sefusatti E., Monaco P., 2022, J. Cosmology Astropart. Phys., 2022, 066
  * Peacock & Nicholson (1991) Peacock J. A., Nicholson D., 1991, MNRAS, 253, 307
  * Pearson & Samushia (2018) Pearson D. W., Samushia L., 2018, MNRAS, 478, 4500
  * Peebles (1975) Peebles P. J. E., 1975, ApJ, 196, 647
  * Peebles (1980) Peebles P. J. E., 1980, The large-scale structure of the universe
  * Peebles & Groth (1975) Peebles P. J. E., Groth E. J., 1975, ApJ, 196, 1
  * Peebles & Yu (1970) Peebles P. J. E., Yu J. T., 1970, Astrophys. J., 162, 815
  * Peirone et al. (2019) Peirone S., Benevento G., Frusciante N., Tsujikawa S., 2019, Phys. Rev. D, 100, 063509
  * Peloso & Pietroni (2013) Peloso M., Pietroni M., 2013, JCAP, 05, 031
  * Peloso & Pietroni (2014) Peloso M., Pietroni M., 2014, JCAP, 04, 011
  * Percival & White (2009) Percival W. J., White M., 2009, Mon. Not. Roy. Astron. Soc., 393, 297
  * Percival et al. (2014) Percival W. J., et al., 2014, Mon. Not. Roy. Astron. Soc., 439, 2531
  * Percival et al. (2022) Percival W. J., Friedrich O., Sellentin E., Heavens A., 2022, MNRAS, 510, 3207
  * Perenon et al. (2019) Perenon L., Bel J., Maartens R., de la Cruz-Dombriz A., 2019, JCAP, 06, 020
  * Perlmutter et al. (1999) Perlmutter S., et al., 1999, Astrophys. J., 517, 565
  * Philcox & Ivanov (2022) Philcox O. H. E., Ivanov M. M., 2022, Phys. Rev. D, 105, 043517
  * Philcox et al. (2022) Philcox O. H. E., Ivanov M. M., Cabass G., Simonović M., Zaldarriaga M., Nishimichi T., 2022, Phys. Rev. D, 106, 043530
  * Pollack et al. (2012) Pollack J. E., Smith R. E., Porciani C., 2012, MNRAS, 420, 3469
  * Predehl et al. (2021) Predehl P., et al., 2021, A&A, 647, A1
  * Rampf & Wong (2012) Rampf C., Wong Y. Y. Y., 2012, J. Cosmology Astropart. Phys., 2012, 018
  * Raveri (2020) Raveri M., 2020, Phys. Rev. D, 101, 083524
  * Reid et al. (2016) Reid B., et al., 2016, Mon. Not. Roy. Astron. Soc., 455, 1553
  * Riess et al. (1998) Riess A. G., et al., 1998, Astron. J., 116, 1009
  * Rizzo et al. (2023) Rizzo F., Moretti C., Pardede K., Eggemeier A., Oddo A., Sefusatti E., Porciani C., Monaco P., 2023, J. Cosmology Astropart. Phys., 2023, 031
  * Rodríguez-Torres et al. (2016) Rodríguez-Torres S. A., et al., 2016, MNRAS, 460, 1173
  * Ross et al. (2012) Ross A. J., et al., 2012, Mon. Not. Roy. Astron. Soc., 424, 564
  * Ruggeri et al. (2018) Ruggeri R., Castorina E., Carbone C., Sefusatti E., 2018, J. Cosmology Astropart. Phys., 2018, 003
  * Saito et al. (2009) Saito S., Takada M., Taruya A., 2009, Phys. Rev. D, 80, 083528
  * Saito et al. (2014) Saito S., Baldauf T., Vlah Z., Seljak U., Okumura T., McDonald P., 2014, Phys. Rev. D, 90, 123522
  * Saito et al. (2015) Saito R., Yamauchi D., Mizuno S., Gleyzes J., Langlois D., 2015, JCAP, 06, 008
  * Sakstein (2015a) Sakstein J., 2015a, Phys. Rev. D, 92, 124045
  * Sakstein (2015b) Sakstein J., 2015b, Phys. Rev. Lett., 115, 201101
  * Sakstein & Jain (2017) Sakstein J., Jain B., 2017, Phys. Rev. Lett., 119, 251303
  * Sakstein et al. (2016) Sakstein J., Wilcox H., Bacon D., Koyama K., Nichol R. C., 2016, JCAP, 07, 019
  * Sakstein et al. (2017a) Sakstein J., Kenna-Allison M., Koyama K., 2017a, JCAP, 03, 007
  * Sakstein et al. (2017b) Sakstein J., Babichev E., Koyama K., Langlois D., Saito R., 2017b, Phys. Rev. D, 95, 064013
  * Saltas & Christensen-Dalsgaard (2022) Saltas I. D., Christensen-Dalsgaard J., 2022, A&A, 667, A115
  * Saltas & Lopes (2019) Saltas I. D., Lopes I., 2019, Phys. Rev. Lett., 123, 091103
  * Saltas et al. (2018) Saltas I. D., Sawicki I., Lopes I., 2018, JCAP, 05, 028
  * Salzano et al. (2017) Salzano V., Mota D. F., Capozziello S., Donahue M., 2017, Phys. Rev. D, 95, 044038
  * Samushia et al. (2021) Samushia L., Slepian Z., Villaescusa-Navarro F., 2021, MNRAS, 505, 628
  * Sato (1981) Sato K., 1981, Mon. Not. Roy. Astron. Soc., 195, 467
  * Schmittfull et al. (2015) Schmittfull M., Baldauf T., Seljak U., 2015, Phys. Rev. D, 91, 043530
  * Scoccimarro (1997) Scoccimarro R., 1997, Astrophys. J., 487, 1
  * Scoccimarro (2000) Scoccimarro R., 2000, ApJ, 544, 597
  * Scoccimarro (2015) Scoccimarro R., 2015, Phys. Rev. D, 92, 083532
  * Scoccimarro & Couchman (2001) Scoccimarro R., Couchman H. M. P., 2001, MNRAS, 325, 1312
  * Scoccimarro & Frieman (1996) Scoccimarro R., Frieman J., 1996, Astrophys. J. Suppl., 105, 37
  * Scoccimarro & Frieman (1999) Scoccimarro R., Frieman J. A., 1999, ApJ, 520, 35
  * Scoccimarro et al. (1998) Scoccimarro R., Colombi S., Fry J. N., Frieman J. A., Hivon E., Melott A., 1998, Astrophys. J., 496, 586
  * Scoccimarro et al. (1999) Scoccimarro R., Couchman H. M. P., Frieman J. A., 1999, Astrophys. J., 517, 531
  * Scoccimarro et al. (2001a) Scoccimarro R., Feldman H. A., Fry J. N., Frieman J. A., 2001a, Astrophys. J., 546, 652
  * Scoccimarro et al. (2001b) Scoccimarro R., Sheth R. K., Hui L., Jain B., 2001b, ApJ, 546, 20
  * Scoccimarro et al. (2004) Scoccimarro R., Sefusatti E., Zaldarriaga M., 2004, Phys. Rev. D, 69, 103513
  * Scoccimarro et al. (2012) Scoccimarro R., Hui L., Manera M., Chan K. C., 2012, Phys. Rev. D, 85, 083002
  * Sebastiani et al. (2017) Sebastiani L., Vagnozzi S., Myrzakulov R., 2017, Adv. High Energy Phys., 2017, 3156915
  * Sefusatti (2009) Sefusatti E., 2009, Phys. Rev. D, 80, 123002
  * Sefusatti & Komatsu (2007) Sefusatti E., Komatsu E., 2007, Phys. Rev. D, 76, 083004
  * Sefusatti et al. (2006) Sefusatti E., Crocce M., Pueblas S., Scoccimarro R., 2006, Phys. Rev. D, 74, 023522
  * Sefusatti et al. (2010) Sefusatti E., Crocce M., Desjacques V., 2010, MNRAS, 406, 1014
  * Sefusatti et al. (2012) Sefusatti E., Crocce M., Desjacques V., 2012, MNRAS, 425, 2903
  * Sellentin & Heavens (2016) Sellentin E., Heavens A. F., 2016, MNRAS, 456, L132
  * Senatore & Trevisan (2018) Senatore L., Trevisan G., 2018, JCAP, 05, 019
  * Senatore & Zaldarriaga (2015) Senatore L., Zaldarriaga M., 2015, JCAP, 02, 013
  * Sharp et al. (1984) Sharp N. A., Bonometto S. A., Lucchin F., 1984, A&A, 130, 79
  * Sherwin & Zaldarriaga (2012) Sherwin B. D., Zaldarriaga M., 2012, Phys. Rev. D, 85, 103523
  * Shirasaki et al. (2021) Shirasaki M., Sugiyama N. S., Takahashi R., Kitaura F.-S., 2021, Phys. Rev. D, 103, 023506
  * Slepian & Eisenstein (2015) Slepian Z., Eisenstein D. J., 2015, MNRAS, 454, 4142
  * Slepian & Eisenstein (2016) Slepian Z., Eisenstein D. J., 2016, MNRAS, 455, L31
  * Slepian & Eisenstein (2017) Slepian Z., Eisenstein D. J., 2017, Mon. Not. Roy. Astron. Soc., 469, 2059
  * Slepian & Eisenstein (2018) Slepian Z., Eisenstein D. J., 2018, MNRAS, 478, 1468
  * Slepian et al. (2017a) Slepian Z., et al., 2017a, MNRAS, 468, 1070
  * Slepian et al. (2017b) Slepian Z., et al., 2017b, Mon. Not. Roy. Astron. Soc., 469, 1738
  * Slepian et al. (2018) Slepian Z., et al., 2018, Mon. Not. Roy. Astron. Soc., 474, 2109
  * Smee et al. (2013) Smee S. A., et al., 2013, AJ, 146, 32
  * Smith et al. (2002) Smith J. A., et al., 2002, Astron. J., 123, 2121
  * Smith et al. (2008) Smith R. E., Sheth R. K., Scoccimarro R., 2008, Phys. Rev. D, 78, 023523
  * Song & Percival (2009) Song Y.-S., Percival W. J., 2009, JCAP, 10, 004
  * Song et al. (2015a) Song Y.-S., et al., 2015a, Phys. Rev. D, 92, 043522
  * Song et al. (2015b) Song Y.-S., Taruya A., Oka A., 2015b, J. Cosmology Astropart. Phys., 2015, 007
  * Sotiriou & Faraoni (2010) Sotiriou T. P., Faraoni V., 2010, Rev. Mod. Phys., 82, 451
  * Springel (2005) Springel V., 2005, MNRAS, 364, 1105
  * Starobinsky (1980) Starobinsky A. A., 1980, Phys. Lett., B91, 99
  * Sugiyama & Futamase (2013) Sugiyama N. S., Futamase T., 2013, Astrophys. J., 769, 106
  * Sugiyama & Spergel (2014) Sugiyama N. S., Spergel D. N., 2014, JCAP, 02, 042
  * Sugiyama et al. (2018) Sugiyama N. S., Shiraishi M., Okumura T., 2018, MNRAS, 473, 2737
  * Sugiyama et al. (2019) Sugiyama N. S., Saito S., Beutler F., Seo H.-J., 2019, MNRAS, 484, 364
  * Sugiyama et al. (2020) Sugiyama N. S., Saito S., Beutler F., Seo H.-J., 2020, MNRAS, 497, 1684
  * Sugiyama et al. (2021) Sugiyama N. S., Saito S., Beutler F., Seo H.-J., 2021, Mon. Not. Roy. Astron. Soc., 501, 2862
  * Sunyaev & Zeldovich (1970) Sunyaev R. A., Zeldovich Y. B., 1970, Astrophys. Space Sci., 7, 3
  * Szapudi et al. (2001) Szapudi I., Postman M., Lauer T. R., Oegerle W., 2001, ApJ, 548, 114
  * Takada & Hu (2013) Takada M., Hu W., 2013, Phys. Rev. D, 87, 123504
  * Takada & Jain (2003) Takada M., Jain B., 2003, MNRAS, 340, 580
  * Takada et al. (2014) Takada M., et al., 2014, Publ. Astron. Soc. Jap., 66, R1
  * Takahashi et al. (2020) Takahashi R., Nishimichi T., Namikawa T., Taruya A., Kayo I., Osato K., Kobayashi Y., Shirasaki M., 2020, ApJ, 895, 113
  * Takushima et al. (2014) Takushima Y., Terukina A., Yamamoto K., 2014, Phys. Rev. D, 89, 104007
  * Takushima et al. (2015) Takushima Y., Terukina A., Yamamoto K., 2015, Phys. Rev. D, 92, 104033
  * Taruya (2016) Taruya A., 2016, Phys. Rev. D, 94, 023504
  * Taruya et al. (2010) Taruya A., Nishimichi T., Saito S., 2010, Phys. Rev. D, 82, 063522
  * Taruya et al. (2011) Taruya A., Saito S., Nishimichi T., 2011, Phys. Rev. D, 83, 103527
  * Taruya et al. (2014a) Taruya A., Koyama K., Hiramatsu T., Oka A., 2014a, Phys. Rev. D, 89, 043509
  * Taruya et al. (2014b) Taruya A., Nishimichi T., Bernardeau F., Hiramatsu T., Koyama K., 2014b, Phys. Rev. D, 90, 123515
  * Taylor & Joachimi (2014) Taylor A., Joachimi B., 2014, Mon. Not. Roy. Astron. Soc., 442, 2728
  * Taylor et al. (2013) Taylor A., Joachimi B., Kitching T., 2013, Mon. Not. Roy. Astron. Soc., 432, 1928
  * Tellarini et al. (2015) Tellarini M., Ross A. J., Tasinato G., Wands D., 2015, J. Cosmology Astropart. Phys., 2015, 004
  * Tellarini et al. (2016) Tellarini M., Ross A. J., Tasinato G., Wands D., 2016, J. Cosmology Astropart. Phys., 2016, 014
  * Toth et al. (1989) Toth G., Hollosi J., Szalay A. S., 1989, ApJ, 344, 75
  * Tram et al. (2016) Tram T., Fidler C., Crittenden R., Koyama K., Pettinari G. W., Wands D., 2016, JCAP, 05, 058
  * Traykova et al. (2019) Traykova D., Bellini E., Ferreira P. G., 2019, JCAP, 08, 035
  * Tsedrik et al. (2023) Tsedrik M., Moretti C., Carrilho P., Rizzo F., Pourtsidou A., 2023, MNRAS, 520, 2611
  * Tseliakhovich & Hirata (2010) Tseliakhovich D., Hirata C., 2010, Phys. Rev. D, 82, 083520
  * Umeh et al. (2019) Umeh O., Koyama K., Maartens R., Schmidt F., Clarkson C., 2019, JCAP, 05, 020
  * Valageas et al. (2017) Valageas P., Taruya A., Nishimichi T., 2017, Phys. Rev. D, 95, 023504
  * Valogiannis & Bean (2019) Valogiannis G., Bean R., 2019, Phys. Rev. D, 99, 063526
  * Valogiannis et al. (2020) Valogiannis G., Bean R., Aviles A., 2020, JCAP, 01, 055
  * Verde et al. (1998) Verde L., Heavens A. F., Matarrese S., Moscardini L., 1998, Mon. Not. Roy. Astron. Soc., 300, 747
  * Verde et al. (2000) Verde L., Wang L., Heavens A. F., Kamionkowski M., 2000, MNRAS, 313, 141
  * Verde et al. (2002) Verde L., et al., 2002, MNRAS, 335, 432
  * Wang et al. (2004) Wang Y., Yang X., Mo H. J., van den Bosch F. C., Chu Y., 2004, MNRAS, 353, 287
  * Weinberg (1989) Weinberg S., 1989, Rev. Mod. Phys., 61, 1
  * Welling et al. (2016) Welling Y., van der Woude D., Pajer E., 2016, J. Cosmology Astropart. Phys., 2016, 044
  * Wilson et al. (2017) Wilson M. J., Peacock J. A., Taylor A. N., de la Torre S., 2017, MNRAS, 464, 3121
  * Woodard (2015) Woodard R. P., 2015, Scholarpedia, 10, 32243
  * Yamamoto et al. (2017) Yamamoto K., Nan Y., Hikage C., 2017, Phys. Rev. D, 95, 043528
  * Yamauchi & Sugiyama (2022) Yamauchi D., Sugiyama N. S., 2022, Phys. Rev. D, 105, 063515
  * Yamauchi et al. (2017a) Yamauchi D., Yokoyama S., Takahashi K., 2017a, Phys. Rev. D, 95, 063530
  * Yamauchi et al. (2017b) Yamauchi D., Yokoyama S., Tashiro H., 2017b, Phys. Rev. D, 96, 123516
  * Yankelevich & Porciani (2019) Yankelevich V., Porciani C., 2019, MNRAS, 483, 2078
  * Yoo & Seljak (2013) Yoo J., Seljak U., 2013, Phys. Rev. D, 88, 103520
  * Yoo et al. (2011) Yoo J., Dalal N., Seljak U., 2011, JCAP, 07, 018
  * Yoshikawa et al. (2020) Yoshikawa K., Tanaka S., Yoshida N., Saito S., 2020, ApJ, 904, 159
  * Yuan et al. (2022) Yuan S., Garrison L. H., Eisenstein D. J., Wechsler R. H., 2022, MNRAS, 515, 871
  * Zel’Dovich (1970) Zel’Dovich Y. B., 1970, A&A, 500, 13
  * Zumalacárregui & García-Bellido (2014) Zumalacárregui M., García-Bellido J., 2014, Phys. Rev. D, 89, 064046
  * d’Amico et al. (2020) d’Amico G., Gleyzes J., Kokron N., Markovic K., Senatore L., Zhang P., Beutler F., Gil-Marín H., 2020, J. Cosmology Astropart. Phys., 2020, 005
  * de Carvalho et al. (2020) de Carvalho E., Bernui A., Xavier H. S., Novaes C. P., 2020, MNRAS, 492, 4469
  * de Rham & Matas (2016) de Rham C., Matas A., 2016, JCAP, 06, 041
  * de Rham & Melville (2018) de Rham C., Melville S., 2018, Phys. Rev. Lett., 121, 221101
  * de Weerd et al. (2020) de Weerd E. M., Clarkson C., Jolicoeur S., Maartens R., Umeh O., 2020, J. Cosmology Astropart. Phys., 2020, 018

##  Appendix A HITOMI

In order to improve the reproducibility of the results of this paper, we
publish the complete set of program codes we used under the name HITOMI. The
languages used in it are c++ and python. Users can download the source files
from the following link: https://github.com/naonori/hitomi.git. In particular,
to reproduce the results of this paper, refer to the DEMO section of the
linked page. There, it explains how to measure the 2PCF and 3PCF from the BOSS
DR12 data, compute the theoretical model including the window function
correction, compute the covariance matrix from the Patchy mocks, combine them
to perform the Fisher and MCMC analyses, and finally summarise the obtained
results in figures and tables. To illustrate these things, we recorded a video
of us running the program and uploaded it to YouTube. The text editor used for
this is vim.

HITOMI requires several external programs such as montepython (Brinckmann &
Lesgourgues, 2019), CLASS (Blas et al., 2011), CUBA (Hahn, 2005), GSL
141414http://www.gnu.org/software/gsl/, FFTW (Frigo & Johnson, 2005), and
FFTLog (Hamilton, 2000). We have written a script with the code to install the
external programs needed to run HITOMI on the Cray XC50 at the Center for
Computational Astrophysics of the National Astronomical Observatory of Japan.
A video recording of the use of this script is available at the following
link: https://www.youtube.com/watch?v=vlP7XIXZsUM. Of course, users of other
PC clusters will have to install HITOMI according to their environment.
Nevertheless, our installation instructions will be helpful to users as a
demonstration.

HITOMI not only reproduces the results of this paper but also offers various
options. For example, it can measure both the power spectrum and the
bispectrum. HITOMI also provides the codes to simplify the 3PCF and bispectrum
measurements for simulations with periodic boundary conditions with a global
line-of-sight direction. It is also possible to measure the 2PCF and 3PCF
(power spectrum and bispectrum) after the reconstruction of the galaxy
distribution and compute the corresponding reconstructed models (Eisenstein et
al., 2007b; Shirasaki et al., 2021). Although not yet implemented, in the
future, we plan to release a code to compute the bispectrum covariance matrix
of galaxies based on perturbation theory, as was done by Sugiyama et al.
(2020). We also plan to release a code that performs an anisotropic BAO
analysis using the anisotropic 3PCF, as in Sugiyama et al. (2021).

It is possible to modify parts of the HITOMI code, e.g., the theoretical
calculation of the 3PCF, to investigate how the results propagate to the final
parameter constraint results. It is also possible to replace the BOSS DR12
galaxy data with data from other galaxies or galaxy clusters, e.g. PFS (Takada
et al., 2014), DESI (DESI Collaboration et al., 2016), Euclid (Laureijs et
al., 2011), SPHEREx (Doré et al., 2014), CMB-S4 (Carlstrom et al., 2019), and
eROSITA (Predehl et al., 2021), to perform data analysis of the 3PCF or
bispectrum.


# Trigonometric Extension of the Geometric Correction Factor: Prototype for
adding precision to adaptive ray tracing in ENZO

Collin M. Cunningham  Collin M. Cunningham1: Under the supervision of John H.
Wise, Ph.D.2
1Center for Computational Cosmology - School of Physics - Georgia Institute of
Technology, Atlanta Georgia, USA
2Center for Relativistic Astrophysics - School of Physics - Georgia Institute
of Technology, Atlanta Georgia, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In this paper, we describe a method designed to add precision to radiation
simulations in the adaptive mesh refinement cosmological hydrodynamics code
ENZO. We build upon the geometric correction factor described in ENZO+MORAY:
radiation hydrodynamics adaptive mesh refinement simulations with adaptive ray
tracing (Wise and Abel 2011) which accounts for partial coverage of a ray’s
solid angle with a cube. Because of this geometric mismatch in the methods to
approximate this, there are artifacts in the radiation field. Here, we address
the two-dimensional extension, which acts as a sufficient estimate of the
three-dimensional case and, in practice, the Hierarchical Equal Area
isoLatitude Pixelization of the sphere (HEALPix) (Gorski 2005). We will
demonstrate the value of an extension to the geometric correction factor and
lay the groundwork for a future implementation to ENZO to improve simulations
of radiation from point sources.

##  1 Introduction

In ENZO, rays are generated from point sources of radiation, which then travel
radially outward. The ray normals are calculated from HEALPix (Gorski 2005).
Some number of photons are associated with each ray. Photons are subsequently
deposited into grid cells based on the geometric overlap between the volume
and the computational grid. These absorbed photons ionize and heat gas.
Because of the geometric mismatch between the volumes associated with the rays
and a Cartesian grid, there are artifacts in the radiation field. We are able
to remedy many of these instances by adding precision to the radiation field
in a rather simple way: a Trigonometric extension of the geometric Correction
Factor (herein TegCF). A prototype for TegCF is the foundation of this paper,
which will serve as an indication of the value of implementing a full version.

We begin by simplifying this problem in several ways. Firstly, we will only be
considering two-dimensional space, mapping rays onto the Cartesian plane.
Therefore, instead of each ray having a volume, each ray will have an area.
This allows us to use a series of trigonometric calculations to deduce the
proportion of photons to deposit, wherein lies the mechanics of TegCF. Also,
we assume we are far from the point-source of radiation; thus, rays will be
parallel. This reduction in complexity suffices as a reasonable approximation
of the in precision added by TegCF, because the current method in ENZO (Wise
Abel 2011) operates under these assumptions as well on a given cell in two
dimensions. Therefore, we can directly compare examples of the two methods in
small discrete cases.

Currently, only the cells with which the ray directly intersects can accept
radiation. By depositing radiation in neighboring cells, we can add precision
in these calculations and retroactively get a better estimate of how much
radiation to deposit in the intersectional cell. The accuracy added to the
intersectional cell arises from the fact that the current method uses an
approximation of area and does not compute it directly. This will, in turn,
fix some of the artifacts in computations and make radiation simulations
generally more accurate.

##  2 The TegCF method

In this section, we outline the explicit methodology behind TegCF. We limit
our discussion to the intersection of the ray with one cell. This rectangle
area is called a pixel (figure 1). Also, we assume the ray enters on the left
side. As is customary in Cartesian coordinates, let each cell have a width and
height of $1$ (for implementation, this will be replaced by some $\Delta c$).
We name the height of the point where the ray enters the cell $Q$, such that
$Q\in[0,1)$. Each ray will also have an angle $\phi$, where $\phi$ is the
angle between the ray and the $x$-axis, $\phi\in[0,\pi/2)$. Let the width of
the ray be $L_{pix}$ upon entering the cell and remain constant until the ray
exits the cell. We limit width by $L_{pix}\leq 1$ ($=\Delta c/2$ for
implementation). This clearly does not address many possible ways a ray can
enter a cell. The extension from this to all possible entry cases is given at
the end of this section.

We want to turn our attention to the covering fraction of neighboring cells,
labeled 1 through 9 in figure 1. Let $f_{i}$ denote the covering fraction of
cell $i$. Given the assumptions above, note that $f_{3}=f_{7}=0$. This follows
from the restrictions on $\phi$.

Figure 1: Illustration of our simplification to single cell, single ray. The
grey shaded area is called a pixel.

Covering fractions in general will be found using the following formula:

$f_{i}=\frac{A_{i}}{\mathrm{d}r\times L_{pix}}\ ,$

where $\mathrm{d}r$ is the distance travelled by the ray-vector from its entry
point in cell 5 to its exit point on the boundary of either cell 2 or 6, and
$A_{i}$ is the area of the overlap of the pixel and the cell.

###  2.1 Cell covering areas

We will briefly address how we found the $A_{i}:i=1,2,4,6,8,9$. The
accompanying source code for the prototype gives further insight. The
following aggregate variables are given for brevity.

Commonly used variables
---
$W$ | $L_{pix}/2$
$Y$ | $W\tan\phi$
$Z$ | $W\sec\phi$
$A$ | $W\cos\phi$
$U$ | $W\sin\phi$
$L$ | $\mathrm{d}r$

Some of these variables are not used in the formulas given here, but we give
them anyway as an aide for parsing the protoype’s code.

####  2.1.1 Cell 1

$Z+Q$ must be greater than 1, else the area is zero.

  * •

$Q+A\leq 1$: We have a triangle with sides $Q+Z-1$ and $(Q+Z-1)\cot\phi$. The
area is

$A_{1}=(Q+Z-1)^{2})\cot\phi/2$

  * •

$Q+A>1$: We find this area by subtracting out the extension of the upper side
of the ray from the previous iteration. The area is

$A_{1}=\frac{1}{2}[(Q+Z-1)^{2})\cot\phi-(Q+A-1)^{2}(\cot\phi+\tan\phi)]$

####  2.1.2 Cell 2

There are two primary scenarios here, the one in which the ray exits through
cell 2 and when it exits through cell 6. Each of these cases has two sub
cases, one for the case of a triangle and the other for a quadrilateral.

  * •

$\tan\phi+Q\geq 1$: $Z+Q\leq 1$:

$A_{2}=W^{2}\cot\phi/2$

  * •

$\tan\phi+Q\geq 1$: $Z+Q>1$:

$A_{2}=W^{2}\cot\phi/2-(Q+Z-1)^{2}\cot\phi/2$

  * •

$\tan\phi+Q<1$: $Z+Q\leq 1$: Let $h=(1-U)\tan\phi+Q+Z-1$.

$A_{2}=h^{2}(\cot\phi+\tan\phi)/2$

  * •

$\tan\phi+Q<1$: $Z+Q>1$:

$A_{2}=h^{2}(\cot\phi+\tan\phi)/2-(Q+Z-1)^{2}\cot\phi/2$

####  2.1.3 Cell 4

  * •

$Q+A\geq 1$: The covering area of Cell 4 is a triangle with height $(1-Q)$ and
base length $(1-Q)\cot\phi$. Therefore, we have a covering area of

$A_{4}=(1-Q)^{2}\cot\phi/2$

  * •

$Q+A<1$: $Z+Q\leq 1$: We again have a triangle but with sides $W$ and $Y$
implying

$A_{4}=WY/2$

  * •

$Q+A<1$: $Z+Q>1$: The remaining case results in a quadrilateral. We can deduce
that $B=W\sin\phi$, $C=1-A-Q$, $E=C\cot\phi$. This results in

$A_{4}=(AB+CE)/2+C(B-E)$

####  2.1.4 Cell 6

These cases correspond to where the ray exits as in cell 2. There is a
simplification by subtracting out cell 9 in the case where the ray exits
through cell 6.

  * •

$\tan\phi+Q\leq 1$:

$A_{6}=YW/2-A_{9}$

  * •

$\mathrm{d}r_{x}+U>1$ (checks whether there is any overlap in the case where
the ray exits through cell 2):

$A_{6}=(\mathrm{d}r_{x}+W\sin\phi-1)^{2}(\cot\phi+\tan\phi)/2$

####  2.1.5 Cell 8

The area is nonzero if and only if $Q-A<0$. Let $\beta=(Z-Q)\cot\phi$,
$R=Q\tan\phi$, $\eta=\beta-R$.

  * •

$\tan\phi+Q-Z>0$:

$A_{8}=\eta^{2}\sin\phi\cos\phi/2$

  * •

$\tan\phi+Q-Z\leq 0$:

$A_{8}=\eta^{2}\sin\phi\cos\phi/2-(\beta-1)|\tan\phi+Q-Z|/2$

####  2.1.6 Cell 9

The area is nonzero if and only if $\tan\phi+Q-Z<0$.

  * •

$\tan\phi+Q-A>0$:

$A_{9}=|\tan\phi+Q-Z|((Z-Q)\cot\phi-1)/2$

  * •

$\tan\phi+Q-A\leq 0$: Let $\iota=|\tan\phi+Q-A|+$
$|\tan\phi+Q-Z|$

$A_{9}=\frac{W\iota\sin\phi-\tan\phi(\tan\phi+Q-A)^{2}}{2}$

Now we can retroactively find the the exact area of the intersectional cell.

$A_{5}=\mathrm{d}r\times 2W-\sum A_{i}$

###  2.2 Generalities

Not all rays will hit a cell on the left side, nor will it necessarily have an
angle between 0 and $\pi/2$. We need to be able to calculate covering
fractions for rays entering on any side and with negative $\phi$ values.
Luckily, this is simple with translations.

At each place a ray intersects with a cell, the above method outputs a value
for each of the cells, so we will represent this as a $3\times 3$ matrix.We
can translate this matrix using flips and rotations to get the covering
fractions for any possible ray. Rotations adjust for rays entering top, bottom
and right sides; flips adjust for negative $\phi$ values
($R(\mathbf{C}_{4},\mathbb{Z}_{2})$).

##  3 Single ray single cell

We will compare the TegCF prototype to the the current method in the case of
the intersection of a single ray with a single cell. Firstly, observe that
figure 2 is the covering fraction of the pixel in a 9 cell region. This is a
ray with normal $\hat{n}=[1/\sqrt{2},1/\sqrt{2}]$ where the ray enters the
cell at the bottom corner, $Q=0$, with $L_{pix}=1$. Here the covering fraction
given by the current method defaults to 1. Observe that the TegCF prototype
allocates the radiation more accurately and adds smoothness.

Figure 2: Left - current method; right - TegCF:
$\hat{n}=[0.7071,0.7071],Q=0,L_{pix}=1.0$. Color indicates covering percentage
of pixel. Figure 2 Covering Fractions
---
$f_{1}$ | 0.0
$f_{2}$ | 0.0884
$f_{4}$ | 0.0884
$f_{5}$ | 0.6464
$f_{6}$ | 0.0884
$f_{8}$ | 0.0884
$f_{9}$ | 0.0
$f_{\text{ENZO}}$ | 1.0

Figures 3 and 4 depict two more comparisons of the two methods. Let
$f_{\text{ENZO}}$ denote the approximation given by ENZO in the corresponding
tables. Both these examples demonstrate how TegCF is lossless with regard to
depositing photons, whereas in the current method, there is an inherent loss
of photons for any pixel where the area does not overlap over a certain
proportion. Also, note the large difference between the approximation,
$f_{\text{ENZO}}$, and the exact covering fraction, $f_{5}$. Cases similar to
this cause non-negligable discrepancies in simulations.

Figure 3: Left - current method; right - prototype:
$\hat{n}=[0.9950,0.0995],Q=0,L_{pix}=0.4$.  Figure 3 Covering Fractions
---
$f_{1}$ | 0.0
$f_{2}$ | 0.0
$f_{4}$ | 0.0050
$f_{5}$ | 0.6157
$f_{6}$ | 0.0050
$f_{8}$ | 0.3706
$f_{9}$ | 0.0037
$f_{\text{ENZO}}$ | 0.5625
Figure 4: Left - current method; right - prototype:
$\hat{n}=[0.8944,0.4472],Q=0.8,L_{pix}=1.0$.  Figure 4 Covering Fractions
---
$f_{1}$ | 0.1173
$f_{2}$ | 0.2708
$f_{4}$ | 0.0894
$f_{5}$ | 0.5224
$f_{6}$ | 0.0
$f_{8}$ | 0.0
$f_{9}$ | 0.0
$f_{\text{ENZO}}$ | 0.16

##  4 Single ray multi-cell

Here we extend our testing to a single ray spanning a $10\times 10$
computational grid. We will give two cases for comparison. In figure 5, we
examine the extension of the case from figure 2 of the previous section.
Again, $L_{pix}=1$, and $\hat{n}=[1/\sqrt{2},1/\sqrt{2}]$, but it now passes
over several cells. Now to further compare the two, we will take the sum of
the covering fractions in all cells. We have $F_{\text{ENZO}}=10$ and
$F_{\text{TegCF}}=9.645$. The discrepancy here results from the computational
grid cutting short the cells [11, 10], [10,11], [-1,0], [0, -1]. With these
additions, the totals are equal, but the allocation is much smoother in with
TegCF.

Figure 5: Left - current method; right - prototype:
$\hat{n}=[0.7071,0.7071],L_{pix}=1.0$.

Figure 6 shows an example wherein TegCF greatly outperforms its counterpart in
precision. The ray has total width $L_{pix}=1$ and a normal of [0.1961,
0.9805]. $F_{\text{ENZO}}=6.0$, $F_{\text{TegCF}}=8.7742$. Thus, there is a
loss of a full $2.7742$ pixels which should have been deposited.

Figure 6: Left - current method; right - prototype:
$\hat{n}=[0.1961,0.9805],L_{pix}=1.0$.

##  5 Multi-ray multi-cell

These are examples of what radiation simulations could look like on a small-
scale. We assume we are far from the point source, giving us parallel rays. We
again give two cases for comparison. The “checkerboard” in figure 7 is the
extension of figures 2 and 5, with rays spaced $2\Delta c$ apart. This
demonstrates how smoothness will counteract artifacts in renderings of
radiation. With the correction described in section 4 (but on all sides), the
sums of the covering fractions are equal (i.e. lossless in a worst case
scenario).

Figure 7: Left - current method; right - prototype:
$\hat{n}=[0.1961,0.9805],L_{pix}=1.0$.

Our next case is the most drastic difference. It is almost complete coverage
using the prototype with almost vertical rays, which is a common occurrence in
ENZO. See figure 8 for the illustration. As you can see, even on a very small
scale, we have artifacts on the left. In addition to this, there is a massive
loss in information. The sum of the covering fractions from the ENZO method is
$F_{\text{ENZO}}=15.11663$, and the sum using the TegCF method is
$F_{\text{TegCF}}=49.9378$. There is over triple the coverage using the new
method. These rays are identical, but the difference between the two methods
is extraordinary.

Figure 8: Left - current method; right - prototype:
$\hat{n}=[0.1000,0.0100],L_{pix}=1.0$.

##  6 Summary

It should be clear from these examples that the singular case, single ray
multi-cell, and multi-ray multi-cell cases can vary greatly with this
extension of the geometric correction factor. In this paper we have
demonstrated the effectiveness of this extension and its potential value in
accurate adaptive ray tracing methods used for radiation simulations. There is
room for a large improvement in ENZO without noticeable computation costs. Not
only will this counter the artifacts currently displayed, it will add
precision and smoothness to general radiative calculations. This prototype was
developed in Python, but the key elements should transition over to the C++
source code of ENZO smoothly because of its inclusiveness.

##  7 References

Abel T. & Wandelt B. 2001, MNRAS, 330, L53
Gorski K. M., Hivon E., Banday A. J., Wandelt B. D.,
Hansen F. K., Reinecke M., Bartelmann M., 2005,
ApJ, 622, 759
Wise J. & Abel T. 2011, MNRAS 414, 3458-3491


MukhtarMohamed Oli DanyiLiu HaoTang SharonGoldwater

# Orthogonality and isotropy of speaker and phonetic information
in self-supervised speech representations

###### Abstract

Self-supervised speech representations can hugely benefit downstream speech
technologies, yet the properties that make them useful are still poorly
understood. Two candidate properties related to the geometry of the
representation space have been hypothesized to correlate well with downstream
tasks: (1) the degree of orthogonality between the subspaces spanned by the
speaker centroids and phone centroids, and (2) the isotropy of the space,
i.e., the degree to which all dimensions are effectively utilized. To study
them, we introduce a new measure, Cumulative Residual Variance (CRV), which
can be used to assess both properties. Using linear classifiers for speaker
and phone ID to probe the representations of six different self-supervised
models and two untrained baselines, we ask whether either orthogonality or
isotropy correlate with linear probing accuracy. We find that both measures
correlate with phonetic probing accuracy, though our results on isotropy are
more nuanced.

###### keywords:

model analysis, representational geometry

##  1 Introduction

Self-supervised speech representations have made a huge impact on downstream
speech technologies, yet the properties that make their representations useful
are still poorly understood. Benchmarks indicate that both phone and speaker
labels are, to a large degree, linearly separable in the representations of
popular recent models [1], and beyond this, a number of studies have compared
the extent to which these labels are recoverable from the representations of
different models [1, 2] or across different layers of the same model [3, 4].
However, these analyses say little about how such information is represented,
beyond just assessing the linear separability of classes. Here, we address
this question using a geometric approach—an approach that is widely used for
analyzing self-supervised models of text (e.g., [5, 6, 7, 8, 9, 10]) as well
as high-dimensional brain imaging data (e.g., [11, 12, 13]), but has received
only a little attention in the speech technology community [14, 15, 16, 17].

To assist our analysis, we develop a new measure for analyzing high-
dimensional distributions, the Cumulative Residual Variance (CRV). When
applied to datasets ${X}$ and ${Y}$ embedded in the same high dimensional
space, the CRV of ${X}$ with respect to ${Y}$, denoted ${X}$\${Y}$, provides a
quantitative measure of the degree to which the principal components of ${Y}$
are orthogonal to those of ${X}$. Meanwhile, ${X}$\${X}$ is a measure of the
isotropy of ${X}$—the degree to which ${X}$ effectively utilizes all
dimensions of the embedding space, i.e., has uniform covariance [18].

Using this measure, we draw on two previous lines of work that suggest
potentially fruitful analyses. First, we build on a recent study which
analyzed LSTM models trained using two different loss functions and
demonstrated that speaker and phonetic information were represented in
orthogonal subspaces [17]. The CRV measure allows us to better quantify
orthogonality, and we use it to analyze several additional models with a
variety of architectures, loss functions, and training data. In experiments on
English LibriSpeech, we show that, unlike randomly initialized (untrained)
models, all trained models have a high degree of orthogonality between the
speaker and phonetic subspaces. In addition, for all six trained models, the
accuracy of a phone classifier trained on the model representations is
significantly correlated with the CRV between the two subspaces.

Next, we explore whether and how the isotropy of the representational space
might predict phone or speaker classification accuracy. It has been argued in
the NLP literature that higher isotropy is desirable in an embedding space
(e.g., [6, 9] and see review in [19]). However, we did not find strong
evidence for this hypothesis: when we computed the isotropy and phone (or
speaker) classification accuracy for different layers of each model, we found
a statistically significant correlation in only two out of six trained models.
On the other hand, we did find a strong and consistent correlation between
phone classification accuracy and the isotropy of the phone class centroids.
This suggests that having evenly distributed centroids is more important for
classification accuracy in these models than having evenly distributed frame
representations.

Figure 1: Evaluating orthogonality as cosine similarities between principal
components (a-c) versus using residual variance (d). Isotropy can be evaluated
with self residual variance (e).

##  2 Isotropy and orthogonality

In NLP, most researchers have argued that representations with greater
isotropy are desirable [6, 9, 10, 20]; but see [21]. However, Rudman et al.
[18] noted that the measures of ``isotropy" used in much of that work do not
match its mathematical definition—that is, the extent to which the covariance
matrix is proportional to the identity matrix. They introduced (and
demonstrated the correctness of) a new measure called IsoScore, and later used
it to show that isotropy is in fact negatively correlated with task
performance in several BERT models [19]. Meanwhile, we know of only one study
of isotropy in models of speech [22], which found a strong positive
correlation between IsoScore and word discrimination performance in supervised
acoustic word embedding models. Here, we explore whether isotropy can predict
either phone or speaker classification performance in self-supervised
representations.

As noted above, IsoScore [18] is one way to measure isotropy. IsoScore ranges
from 0 (minimally isotropic) to 1 (maximally isotropic), and can be
interpreted as the approximate proportion of the dimensions that are uniformly
utilized. Computing the IsoScore for a point cloud
${X}\subseteq\mathbb{R}^{d}$ starts by applying PCA, then finding the
Euclidean distance between the length-normalized vector of eigenvalues
$\Lambda$ (the diagonal of the covariance matrix) and the diagonal of the
identity matrix in $\mathbb{R}^{d}$. This distance is then normalized and
rescaled to fall between 0 and 1.

While IsoScore has properties that can be desirable (e.g., it allows direct
comparisons between spaces of different dimensionality on the same 0-1 scale),
it is not the only possible measure of isotropy. For example, Del Giudice [23]
discusses estimators of ``Effective Dimensionality'' which normalize $\Lambda$
to create a probability distribution, then calculate its entropy to measure
deviance from uniformity, and return a value interpreted as the number (rather
than proportion) of dimensions uniformly utilized.

Apart from isotropy, orthogonality is also a desirable property when learning
representations, since encoding different kinds of information in orthogonal
dimensions or subspaces would allow them to be easily disentangled. In fact,
there have been attempts in representation learning to enforce such
orthogonality to enable disentanglement [24, 25]. There is also evidence that
human brains encode different aspects of the same item in orthogonal coding
axes, thereby minimizing interference and maximizing robustness [26, 27].
However, [17] (henceforth, Liu et al.) is the only work we know of to explore
orthogonality in either supervised or unsupervised speech models. We describe
their method, and how we build on it, in more detail below.

##  3 Measuring orthogonality

Before evaluating the orthogonality between speaker and phonetic encoding, we
first follow Liu et al. in identifying speaker directions and phonetic
directions. Phonetic directions are found by aggregating the frame-level
representations for each of the 39 phones (based on forced alignment) to
obtain their centroids, and then applying principal component analysis to the
centroids. The 39 principal components found represent the _phonetic
directions_ , along which the variance between the centroids is maximized. The
same method is used to obtain speaker directions using the speakers in the
dataset.

Our next step diverges from Liu et al.'s: while they looked at the cosine
similarities between the phonetic and speaker directions (§3.1), we propose a
new measure that quantifies orthogonality with a single numerical value
(§3.2).

###  3.1 Cosine similarity between principal directions

For each pair of a speaker and a phonetic direction, Liu et al. computed the
degree of orthogonality by taking the absolute value of their cosine
similarity, with 0 being perfectly orthogonal and 1 being perfectly aligned.
Figs. 1a-c present the pairwise similarity for representations extracted from
(a) the second LSTM layer of the same CPC-big model used by Liu et al. (from
[28]); (b) the same layer of a randomly initialized CPC model that has not
been trained, and (c) log Mel features. Confirming Liu et al.'s results, Fig.
1a shows very low similarities between any pair of phonetic and speaker
directions, indicating the two types of information are largely encoded
orthogonally.

While the similarity matrix gives some indication of the relationship between
the directions encoding speaker and phonetic information, it can be difficult
to summarize with a single number: we need to consider the degree of alignment
between every pair of directions in order to fully capture the degree of
orthogonality between speaker and phonetic encoding. In addition, alignment
between principal directions with large eigenvalues means overall lower
orthogonality than alignment between principal directions with small
eigenvalues, but the matrix does not reflect the amounts of variance in each
principal direction.

###  3.2 Cumulative Residual Variance (CRV)

We propose Cumulative Residual Variance (CRV) as a quantitative measure of
orthogonality between datasets ${X}$ and ${Y}$ embedded in $\mathbb{R}^{d}$.
CRV satisfies the two desiderata mentioned above: (1) it captures the
interaction between every pair of principal directions and (2) it weights the
contribution from each principal direction in proportion to its relative
explained variance. Here, we set ${X}$ and ${Y}$ to be the speaker and phone
centroids (or vice versa), so the number of data points $n_{X}$ and $n_{Y}$ is
less than the dimensionality $d$, and each dataset only spans a subspace of
$\mathbb{R}^{d}$. However, this need not be true in general; for example CRV
could be applied to the sets of frame-level representations from two different
speakers or two different phones.

In short, the CRV of ${Y}$ with respect to ${X}$, written as ${Y}$\${X}$,
evaluates how much variance is preserved in ${Y}$ as the principal directions
of ${X}$ are collapsed one by one.111Note that CRV is an asymmetrical distance
measure. Like KL divergence, it could be symmetrised as ${X}$\${Y}$ \+
${Y}$\${X}$, if desired.  As in Liu et al., ``collapsing" a direction $v$ from
a dataset ${Y}$ refers to the operation of projecting ${Y}$ onto the subspace
orthogonal to $v$, i.e. ${Y}^{\prime}={Y}-({Y}v)v^{\top}$. Collapsing $v$
affects any principal direction of ${Y}$ that is not orthogonal to $v$, which
addresses the first desideratum. We evaluate the effect of the collapsing
operation by computing the residual variance in ${Y}^{\prime}$, as given by
PCA. The larger the residual variance ${Y^{\prime}}$ has, the more orthogonal
${Y}$ is to $v$.

The residual variances computed in this way can be plotted as in Fig. 1d,
where for any given $x$-axis value, its $y$ value is the proportion of
variance remaining in ${Y}$ after collapsing the minimum number of top
principal directions of ${X}$ such that at least $x$ proportion of ${X}$'s
variance has been removed. CRV is then computed from this plot as the area
under the curve (AUC), to yield a single numerical value. In this way, the
effect of collapsing each direction is weighted by the variance explained by
that direction, hence CRV also satisfies our second desideratum.

In Fig. 1d, we plot residual variance of the phone centroids with respect to
the explained variance in the speaker centroids for representations from a
trained and an untrained CPC222Though CPC is a loss function, with a slight
abuse, we refer to a randomly initialized CPC-big in [28] as untrained CPC. as
well as for log Mel features. We can see that the relative magnitude of the
AUC is CPC, followed by log Mel and untrained CPC. While the strong
orthogonality in the trained CPC is consistent with Fig. 1a, the relative
degree of orthogonality between log Mel and untrained CPC is less salient from
Fig. 1b-c. There are more dark spots in Fig. 1c, indicating more pairs of
aligned speaker and phonetic directions in log Mel, but this should have less
effect on overall orthogonality as compared to the top left corner of Fig. 1b,
which shows that the first two speaker and phonetic directions of the
untrained CPC are very strongly aligned. This is properly reflected in Fig.
1d.

###  3.3 Evaluating isotropy with Self-CRV

A byproduct of CRV is _self-CRV_ , or ${Y}$\${Y}$, which evaluates the degree
of isotropy of ${Y}$ in $\mathbb{R}^{d}$. If ${Y}$ is highly anisotropic, its
variance will be concentrated around a few directions. This results in a
residual variance curve with a small AUC, as illustrated in Fig. 1e for
untrained CPC.

Self-CRV is closely related to IsoScore, both being functions of the
eigenvalues of the dataset. However, IsoScore measures isotropy as a
percentage of representation dimensions, whereas self-CRV accounts for the
absolute number of isotropic dimensions and is comparable across models with
different dimensions as long as the number of data points in ${Y}$ is smaller
than the dimensionality of all models (as in our subspace analyses). After
multiplying IsoScore by model dimension, we found a Spearman's rank
correlation of 1 between it and self-CRV.

##  4 Experimental Setup

#### Models

In addition to CPC-big, we measured orthogonality and isotropy in five pre-
trained Transformer-based English self-supervised speech models: HuBERT (base-
ls960) [29], wav2vec 2.0 (base-960h) [30], WavLM (base) [31], WavLM+ (base-
plus) [31], and Data2Vec (base-960h) [32]. Apart from the architecture, CPC-
big differs from the Transformer-based models in its dimensionality (512 vs.
768), number of layers (5 CNN followed by 4 LSTM vs. 7 CNN followed by 12
Transformer blocks), frame rate (10ms vs. 20ms) and amount of training data
(6k hr vs. 960 hr for all others except WavLM+, which used 96k hr). To
determine the degree of orthogonality and isotropy in these models before
training, we also tested representations extracted from a HuBERT model and a
CPC-big model with just random initialization and no training. Since the
Transformer models we tested have mostly the same architecture and are
distinguished by the training methods and objective, the untrained HuBERT is
representative of the other Transformer models. Finally, 40-dimensional log
Mel features are used as a baseline.

#### Dataset

We perform our analysis on the dev-clean subset of LibriSpeech [33], which
matches the language (English) and genre (read speech) of the pre-trained
models and was also used in Liu et al.333We hope in future to examine how much
these results generalize, by extending the analyses to other genres and
languages, either using different pre-trained models or by testing these
models on other data. Dev-clean contains 40 speakers, each contributing at
least eight minutes of speech. We used half of dev-clean for training
classifiers and half for testing, with different splits depending on the
scenario, as described below.

#### Probing classifiers

Our analysis focuses on speaker information and phonetic information, due to
their influence on a variety of downstream speech tasks. We train logistic
regression classifiers to predict the speaker (or phone) label based on a
single representation frame. In previous work, frames are typically pooled
across phones [4, 34] or utterances (for speaker ID) [1]; but like Liu et al.
we use individual frames, so we can analyze how both types of information sit
in the same set of embeddings.

For speaker classification, we obtain speaker labels from the LibriSpeech
metadata and train the probing classifier on a random half of each speaker’s
utterances, using the other half for testing. For phone classification, we
obtain the phone labels from forced alignments with Kaldi. We evaluated phone
accuracy in two ways: shared speakers (as in Liu et al.), where the same
speakers appear in both training and test, and the more standard across-
speaker, where we trained on data from a random half of the speakers and
tested on the other half. In practice, the measures are very strongly
correlated and don't differ much, so in this paper we only report across-
speaker phone accuracy.

#### Computing CRV, IsoScore, and correlations

We computed CRV and IsoScore for each layer of each model by first encoding
the utterances from LibriSpeech dev-clean to obtain the representations. We
then computed the phone and speaker centroids and CRV values as described in
§3. In particular, Ph\Spk measures the orthogonality of the phone and speaker
subspaces, and Ph\Ph and Spk\Spk measure the isotropy of the phone and speaker
subspaces, respectively. We computed the IsoScore using a random sample of
250,000 frames. Finally, for correlations between classifier accuracy and CRV
or IsoScore, we computed Spearman (rank) correlation, since it is less
sensitive to outliers and we have no reason to believe that correlations will
be linear.

##  5 Results and Discussion

log Mel

HuBERT

WavLM

WavLM+

wav2vec 2.0

data2vec

rand. HuBERT

CPC-big

rand. CPC-big

Figure 2: Layerwise results for all models, showing (in columns from left to
right): Phone and speaker classiciation accuracy; CRV orthogonality measures
(Ph\Spk and Spk\Ph); and self-CRV measures (Ph\Ph and Spk\Spk).

Figure 3: Correlations between orthogonality or isotropy measures and probing
accuracies. Marker styles are as in Fig. 2.

###  5.1 Layerwise Classification Accuracy

Fig. 2 (lst column) shows the results of our probing classifiers for phones
(top) and speakers (bottom), across all layers of each model444Due to space
constraints, not all results can be displayed in the paper. The complete
spreadsheet of our results, and the code for computing CRV can be found at
https://github.com/uililo/cumulative-residual-variance.. For phones, our
findings align closely with those of [3, 4, 34], despite analyzing frame-wise
rather than pooling the representations for each phone token. That is, for
wav2vec2 (and data2vec) the highest probing accuracies are in the late middle
layers, while for HuBERT-family models (HuBERT, WavLM, WavLM+), accuracy
remains high through the final layers.

To the best of our knowledge, previous studies have only reported speaker
probing accuracy across all the layers of HuBERT [35]. Extending the layerwise
analysis of speaker information to the other widely-used SSL models, we find
far more variation here than with phone accuracy, perhaps because all of these
models, despite being self-supervised, are designed with ASR in mind. We see
especially poor linear separability in the later layers of wav2vec2 and
data2vec, where speaker accuracy is even worse than the randomly initialized
ones. We speculate that the rising pattern of speaker accuracy in the randomly
initialized models may be because the model incorporates more context in later
layers, allowing the model to effectively average features over the whole
utterance.

###  5.2 Geometry of the phone and speaker subspaces

Layer-wise CRV and self-CRV results for all models are shown in Fig. 2,
columns 2 and 3. Like the CPC model studied by Liu et al., all trained
Transformer models have high Ph\Spk orthogonality. Interestingly, untrained
HuBERT (unlike untrained CPC) also reaches a somewhat high Ph\Spk value in the
final layers, although still lower than the trained models. The trained models
also show high isotropy in the phone and speaker centroids (Ph\Ph and
Spk\Spk), though as with probing accuracy, the difference between trained and
untrained models is much more striking for the phonetic measure, suggesting
that model training reorganizes the representational geometry of the phonetic
information more than the speaker information.

We then computed rank correlations $\rho$ between each of the four CRV
measures and the speaker or phone accuracies. The most striking correlation is
between Ph\Ph and phone accuracy, as shown in Figs. 3a (all models) and 3b
(trained models only). Pooling all datapoints together, $\rho$ = 0.94, and the
trained models individually each have $\rho$ from 0.69 to 0.9 (all values
$p<0.05$). In contrast, we only found statistically significant correlations
between Spk\Spk and speaker accuracy (Fig. 3c) in wav2vec2 and data2vec, and
no significant correlation when pooling the results from all trained models.

For the orthogonality measures, we found significant correlations between
Ph\Spk and phone accuracy (Fig. 3d) in each of the trained models ($\rho=$
0.54-0.78 for Transformer models, 1.0 for CPC), as well as in the pooled data
($\rho=$ 0.54), although the correlations are weaker than for Ph\Ph.
Correlations between Spk\Ph and speaker accuracy are even weaker, reaching
significance on the pooled data, but not for any individual model.

Altogether, our results suport Liu et al.'s claim that orthogonality between
the phonetic and speaker subspace is relevant for extracting phonetic
information, but also suggest that isotropy of the phonetic space may be even
more critical. It is less clear why the geometry of speaker information is
less correlated to speaker classification, and to what extent this result is
due to model training that is implicitly focused on ASR performance.

###  5.3 Isotropy of the frame representation space

Finally, we evaluated the isotropy of frame representations (rather than the
centroids). For this, we used IsoScore, which (1) has a rank correlation of 1
with self-CRV as isotropy measures of speaker or phone centroids, and (2) is
easier to compute than self-CRV when applied to a large number of
representations. The IsoScore values were low, ranging from 0.18 to near 0
across models and layers, similar to the range found by Rudman et al. [18] for
contextualized word embedding models. Also, the IsoScore values for untrained
HuBERT were comparable to those of the trained models. We find a statistically
significant ($p<0.05$) positive correlation with phone probing accuracy in
HuBERT and WavLM, and when pooling results from all trained models; but for
speaker probing accuracy we found negative correlations in the same two
models, and no significant pooled correlation. These mixed results suggest
that isotropy of the representation space itself is not necessarily a good
predictor of model performance, especially if different tasks are considered.

##  6 Conclusion

This paper introduced the Cumulative Residual Variance as a new way to analyze
the representational geometry of high-dimensional spaces, and used CRV and
IsoScore to examine whether orthogonality or isotropy can predict phone or
speaker probing accuracy in self-supervised speech models. We did not find
strong evidence that isotropy of the frame representations is meaningful, but
we did show that phone probing accuracy is correlated with the degree of
orthogonality between the subspaces defined by the phone and speaker
centroids, and even more strongly with the isotropy of the phone centroids
themselves. These findings suggest that geometric analyses may be a productive
route for future study, particularly if they can be more closely connected to
theoretical analyses such as those of [13]. For instance, [13] highlights the
relevance of four different geometric properties, including the distance
between class centroids (related to our subspace isotropy measure) as well as
the isotropy of the individual class manifolds (i.e., phones or speakers). We
hope that our work may inspire further exploration of these connections.

##  7 Acknowledgements

This work was supported in part by the UKRI Centre for Doctoral Training in
Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the
University of Edinburgh, and also received support from Google.

## References

  * [1] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech Processing Universal PERformance Benchmark,'' in _Proc. Interspeech_ , 2021.
  * [2] D. Ma, N. Ryant, and M. Liberman, ``Probing acoustic representations for phonetic properties,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [3] A. Pasad, J.-C. Chou, and K. Livescu, ``Layer-wise analysis of a self-supervised speech representation model,'' in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2021, pp. 914–921.
  * [4] A. Pasad, B. Shi, and K. Livescu, ``Comparative layer-wise analysis of self-supervised speech models,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023.
  * [5] T. Mikolov, W.-t. Yih, and G. Zweig, ``Linguistic regularities in continuous space word representations,'' in _Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics_ , 2013, pp. 746–751.
  * [6] X. Cai, J. Huang, Y. Bian, and K. Church, ``Isotropy in the contextual embedding space: Clusters and manifolds,'' in _International conference on learning representations_ , 2020.
  * [7] T. A. Chang, Z. Tu, and B. K. Bergen, ``The geometry of multilingual language model representations,'' _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2022.
  * [8] E. Hernandez and J. Andreas, ``The low-dimensional linear geometry of contextualized word representations,'' _Proceedings of the 25th Conference on Computational Natural Language Learning_ , 2021.
  * [9] J. Mu, S. Bhat, and P. Viswanath, ``All-but-the-top: Simple and effective postprocessing for word representations,'' _Proceedings of the International Conference on Learning Representations_ , 2017.
  * [10] W. Timkey and M. Van Schijndel, ``All bark and no bite: Rogue dimensions in transformer language models obscure representational quality,'' _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * [11] N. Kriegeskorte and R. A. Kievit, ``Representational geometry: integrating cognition, computation, and the brain,'' _Trends in cognitive sciences_ , vol. 17, no. 8, pp. 401–412, 2013.
  * [12] S. Saxena and J. P. Cunningham, ``Towards the neural population doctrine,'' _Current opinion in neurobiology_ , vol. 55, pp. 103–111, 2019.
  * [13] B. Sorscher, S. Ganguli, and H. Sompolinsky, ``Neural representational geometry underlies few-shot concept learning,'' _Proceedings of the National Academy of Sciences_ , 2022.
  * [14] C. Stephenson, J. Feather, S. Padhy, O. Elibol, H. Tang, J. McDermott, and S. Chung, ``Untangling in invariant speech recognition,'' _Advances in neural information processing systems_ , vol. 32, 2019.
  * [15] B. M. Abdullah, M. Mosbach, I. Zaitova, B. Möbius, and D. Klakow, ``Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Study,'' in _Proc. Interspeech_ , 2021.
  * [16] K. Choi and E. J. Yeo, ``Opening the black box of wav2vec feature encoder,'' _arXiv preprint_ , 2022.
  * [17] O. D. Liu, H. Tang, and S. Goldwater, ``Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces,'' in _Proc. Interspeech_ , 2023.
  * [18] W. Rudman, N. Gillman, T. Rayne, and C. Eickhoff, ``Isoscore: Measuring the uniformity of embedding space utilization,'' _Findings of the Association for Computational Linguistics_ , 2022.
  * [19] W. Rudman and C. Eickhoff, ``Stable anisotropic regularization,'' _International Conference on Learning Representations_ , 2024.
  * [20] B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li, ``On the sentence embeddings from pre-trained language models,'' _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * [21] Y. Ding, K. Martinkus, D. Pascual, S. Clematide, and R. Wattenhofer, ``On isotropy calibration of transformers,'' _Proceedings of Workshop on Insights from Negative Results in NLP_ , 2021.
  * [22] B. M. Abdullah and D. Klakow, ``Analyzing the representational geometry of acoustic word embeddings,'' _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , 2022.
  * [23] M. Del Giudice, ``Effective dimensionality: A tutorial,'' _Multivariate behavioral research_ , vol. 56, no. 3, pp. 527–542, 2021.
  * [24] M. H. Sarhan, N. Navab, A. Eslami, and S. Albarqouni, ``Fairness by learning orthogonal disentangled representations,'' in _European Conference on Computer Vision_. Springer, 2020.
  * [25] J. Cha and J. Thiyagalingam, ``Orthogonality-enforced latent space in autoencoders: An approach to learning disentangled representations,'' in _Proceedings of the 40th International Conference on Machine Learning_ , 2023.
  * [26] T. Flesch, K. Juechems, T. Dumbalska, A. Saxe, and C. Summerfield, ``Orthogonal representations for robust context-dependent task performance in brains and neural networks,'' _Neuron_ , vol. 110, no. 7, 2022.
  * [27] A. Libby and T. J. Buschman, ``Rotational dynamics reduce interference between sensory and memory representations,'' _Nature neuroscience_ , vol. 24, no. 5, pp. 715–726, 2021.
  * [28] T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, A. Baevski, E. Dunbar, and E. Dupoux, ``The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,'' _arXiv preprint arXiv:2011.11588_ , 2020.
  * [29] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [30] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [31] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [32] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``Data2vec: A general framework for self-supervised learning in speech, vision and language,'' in _International Conference on Machine Learning_. PMLR, 2022, pp. 1298–1312.
  * [33] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2015.
  * [34] P. C. English, J. Kelleher, and J. Carson-Berndsen, ``Domain-informed probing of wav2vec 2.0 embeddings for phonetic features,'' in _Proceedings of SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , 2022.
  * [35] H.-J. Chang, A. H. Liu, and J. Glass, ``Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering,'' in _Proc. INTERSPEECH 2023_ , 2023, pp. 2983–2987.