all those at the University of Arizona for their hospitality during the
associated secondment at the commencement of this work. NSB and IP were funded
by the NASA/STScI GO grant JWST-GO-02260.001. CJC acknowledges support from
the Science & Technology Facilities Council (STFC) Consolidated Grant
ST/W000997/1 & RA acknowledges funding from the Science & Technology
Facilities Council (STFC) through Consolidated Grant ST/W000857/1. GB has
received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 Framework Programme (grant agreement no. 853022, PEVAP).

This work was performed using resources provided by the Cambridge Service for
Data Driven Discovery (CSD3) operated by the University of Cambridge Research
Computing Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using
Tier-2 funding from the Engineering and Physical Sciences Research Council
(capital grant EP/T022159/1), and DiRAC funding from the Science and
Technology Facilities Council (www.dirac.ac.uk).

This work is also based on observations made with the NASA/ ESA/CSA James Webb
Space Telescope. The data were obtained from the Mikulski Archive for Space
Telescopes at the Space Telescope Science Institute, which is operated by the
Association of Universities for Research in Astronomy, Inc., under NASA
contract NAS 5-03127 for JWST. The observations are associated with JWST GO
Cycle 1 program ID 2260. The JWST data used in this paper can be found in
MAST: 10.17909/dhmh-fx64. The synthetic detector images corresponding to the
favoured $r_{\rm in}=0.1\,r_{G}$ models (with and without a cavity) plus those
of the simulated standard star are available via a Zenodo repository at
10.5281/zenodo.10409615. Other models can be shared upon request to the
corresponding author.

This work also benefited from the Core2disk-III residential program of
Institut Pascal at Université Paris-Saclay, with the support of the program
“Investissements d’avenir” ANR-11-IDEX-0003-01.

umPy (van der Walt et al., 2011), SciPy (Virtanen et al., 2020), Matplotlib
(Hunter, 2007), MOCASSIN (Ercolano et al., 2003, 2005, 2008), MIRISim
(Klaassen et al., 2021), JWST Pipeline (Bushouse et al., 2023), CASA (McMullin
et al., 2007)

## References

  * Agra-Amboage et al. (2014) Agra-Amboage, V., Cabrit, S., Dougados, C., et al. 2014, A&A, 564, A11, doi: 10.1051/0004-6361/201220488
  * Alexander et al. (2014) Alexander, R., Pascucci, I., Andrews, S., Armitage, P., & Cieza, L. 2014, in Protostars and Planets VI, ed. H. Beuther, R. S. Klessen, C. P. Dullemond, & T. Henning, 475, doi: 10.2458/azu_uapress_9780816531240-ch021
  * Alexander et al. (2023) Alexander, R., Rosotti, G., Armitage, P. J., et al. 2023, MNRAS, 524, 3948, doi: 10.1093/mnras/stad1983
  * Alexander (2008) Alexander, R. D. 2008, MNRAS, 391, L64, doi: 10.1111/j.1745-3933.2008.00556.x
  * Alexander et al. (2006) Alexander, R. D., Clarke, C. J., & Pringle, J. E. 2006, MNRAS, 369, 216, doi: 10.1111/j.1365-2966.2006.10293.x
  * Appelgren et al. (2023) Appelgren, J., Lambrechts, M., & van der Marel, N. 2023, arXiv e-prints, arXiv:2303.07297, doi: 10.48550/arXiv.2303.07297
  * Argyriou et al. (2023) Argyriou, I., Glasse, A., Law, D. R., et al. 2023, A&A, 675, A111, doi: 10.1051/0004-6361/202346489
  * Arulanantham et al. (2021) Arulanantham, N., France, K., Hoadley, K., et al. 2021, AJ, 162, 185, doi: 10.3847/1538-3881/ac1593
  * Arulanantham et al. (2023) Arulanantham, N., Gronke, M., Fiorellino, E., et al. 2023, arXiv e-prints, arXiv:2301.01761.  https://arxiv.org/abs/2301.01761
  * Asplund et al. (2021) Asplund, M., Amarsi, A. M., & Grevesse, N. 2021, A&A, 653, A141, doi: 10.1051/0004-6361/202140445
  * Badnell (2006a) Badnell, N. R. 2006a, A&A, 447, 389, doi: 10.1051/0004-6361:20053269
  * Badnell (2006b) —. 2006b, ApJS, 167, 334, doi: 10.1086/508465
  * Bai (2017) Bai, X.-N. 2017, ApJ, 845, 75, doi: 10.3847/1538-4357/aa7dda
  * Bai et al. (2016) Bai, X.-N., Ye, J., Goodman, J., & Yuan, F. 2016, ApJ, 818, 152, doi: 10.3847/0004-637X/818/2/152
  * Bajaj et al. (2024) Bajaj, N. S., Pascucci, I., Gorti, U., et al. 2024, AJ, 167, 127, doi: 10.3847/1538-3881/ad22e1
  * Baldovin-Saavedra et al. (2012) Baldovin-Saavedra, C., Audard, M., Carmona, A., et al. 2012, A&A, 543, A30, doi: 10.1051/0004-6361/201118329
  * Ballabio et al. (2020) Ballabio, G., Alexander, R. D., & Clarke, C. J. 2020, MNRAS, 496, 2932, doi: 10.1093/mnras/staa1767
  * Banzatti et al. (2019) Banzatti, A., Pascucci, I., Edwards, S., et al. 2019, ApJ, 870, 76, doi: 10.3847/1538-4357/aaf1aa
  * Blandford & Payne (1982) Blandford, R. D., & Payne, D. G. 1982, MNRAS, 199, 883, doi: 10.1093/mnras/199.4.883
  * Brickhouse et al. (2010) Brickhouse, N. S., Cranmer, S. R., Dupree, A. K., Luna, G. J. M., & Wolk, S. 2010, ApJ, 710, 1835, doi: 10.1088/0004-637X/710/2/1835
  * Bushouse et al. (2023) Bushouse, H., Eisenhamer, J., Dencheva, N., et al. 2023, JWST Calibration Pipeline, 1.11.2, Zenodo, doi: 10.5281/zenodo.8140011
  * Cahill et al. (2019) Cahill, E., Whelan, E. T., Huélamo, N., & Alcalá, J. 2019, MNRAS, 484, 4315, doi: 10.1093/mnras/stz280
  * Clarke & Alexander (2016) Clarke, C. J., & Alexander, R. D. 2016, MNRAS, 460, 3044, doi: 10.1093/mnras/stw1178
  * Clarke et al. (2001) Clarke, C. J., Gendrin, A., & Sotomayor, M. 2001, MNRAS, 328, 485, doi: 10.1046/j.1365-8711.2001.04891.x
  * de Valon et al. (2020) de Valon, A., Dougados, C., Cabrit, S., et al. 2020, A&A, 634, L12, doi: 10.1051/0004-6361/201936950
  * Dullemond et al. (2007) Dullemond, C. P., Hollenbach, D., Kamp, I., & D’Alessio, P. 2007, in Protostars and Planets V, ed. B. Reipurth, D. Jewitt, & K. Keil, 555, doi: 10.48550/arXiv.astro-ph/0602619
  * Edwards et al. (2006) Edwards, S., Fischer, W., Hillenbrand, L., & Kwan, J. 2006, ApJ, 646, 319, doi: 10.1086/504832
  * Emsenhuber et al. (2023) Emsenhuber, A., Burn, R., Weder, J., et al. 2023, arXiv e-prints, arXiv:2301.04656, doi: 10.48550/arXiv.2301.04656
  * Ercolano et al. (2005) Ercolano, B., Barlow, M. J., & Storey, P. J. 2005, MNRAS, 362, 1038, doi: 10.1111/j.1365-2966.2005.09381.x
  * Ercolano et al. (2003) Ercolano, B., Barlow, M. J., Storey, P. J., & Liu, X. W. 2003, MNRAS, 340, 1136, doi: 10.1046/j.1365-8711.2003.06371.x
  * Ercolano et al. (2009) Ercolano, B., Clarke, C. J., & Drake, J. J. 2009, ApJ, 699, 1639, doi: 10.1088/0004-637X/699/2/1639
  * Ercolano & Owen (2010) Ercolano, B., & Owen, J. E. 2010, MNRAS, 406, 1553, doi: 10.1111/j.1365-2966.2010.16798.x
  * Ercolano & Owen (2016) —. 2016, MNRAS, 460, 3472, doi: 10.1093/mnras/stw1179
  * Ercolano et al. (2021) Ercolano, B., Picogna, G., Monsch, K., Drake, J. J., & Preibisch, T. 2021, MNRAS, 508, 1675, doi: 10.1093/mnras/stab2590
  * Ercolano et al. (2008) Ercolano, B., Young, P. R., Drake, J. J., & Raymond, J. C. 2008, ApJS, 175, 534, doi: 10.1086/524378
  * Erkal et al. (2022) Erkal, J., Manara, C. F., Schneider, P. C., et al. 2022, A&A, 666, A188, doi: 10.1051/0004-6361/202244254
  * Espaillat et al. (2021) Espaillat, C., Calvet, N., & McClure, M. 2021, EUV vs. X-ray Photoevaporation of Protoplanetary Disks, JWST Proposal. Cycle 1, ID. #1676
  * Espaillat et al. (2013) Espaillat, C., Ingleby, L., Furlan, E., et al. 2013, ApJ, 762, 62, doi: 10.1088/0004-637X/762/1/62
  * Espaillat et al. (2023) Espaillat, C. C., Thanathibodee, T., Pittman, C. V., et al. 2023, arXiv e-prints, arXiv:2311.07739.  https://arxiv.org/abs/2311.07739
  * Fang et al. (2023) Fang, M., Wang, L., Herczeg, G. J., et al. 2023, arXiv e-prints, arXiv:2305.07929, doi: 10.48550/arXiv.2305.07929
  * Ferreira (1997) Ferreira, J. 1997, A&A, 319, 340, doi: 10.48550/arXiv.astro-ph/9607057
  * Ferreira & Pelletier (1995) Ferreira, J., & Pelletier, G. 1995, A&A, 295, 807
  * Flores-Rivera et al. (2023) Flores-Rivera, L., Flock, M., Kurtovic, N. T., et al. 2023, A&A, 670, A126, doi: 10.1051/0004-6361/202141664
  * Font et al. (2004) Font, A. S., McCarthy, I. G., Johnstone, D., & Ballantyne, D. R. 2004, ApJ, 607, 890, doi: 10.1086/383518
  * Francis & van der Marel (2020) Francis, L., & van der Marel, N. 2020, ApJ, 892, 111, doi: 10.3847/1538-4357/ab7b63
  * Gaia Collaboration et al. (2022) Gaia Collaboration, Vallenari, A., Brown, A. G. A., et al. 2022, arXiv e-prints, arXiv:2208.00211.  https://arxiv.org/abs/2208.00211
  * Galavis et al. (1995) Galavis, M. E., Mendoza, C., & Zeippen, C. J. 1995, A&AS, 111, 347
  * Glasse et al. (2015) Glasse, A., Rieke, G. H., Bauwens, E., et al. 2015, PASP, 127, 686, doi: 10.1086/682259
  * Glassgold et al. (1997) Glassgold, A. E., Najita, J., & Igea, J. 1997, ApJ, 480, 344, doi: 10.1086/303952
  * Glassgold et al. (2007) Glassgold, A. E., Najita, J. R., & Igea, J. 2007, ApJ, 656, 515, doi: 10.1086/510013
  * Gordon et al. (2019) Gordon, K. D., Boyer, M. L., Muzerolle, J., Sloan, G. C., & Volk, K. 2019, Absolute Flux Calibration (G Dwarfs), JWST Proposal. Cycle 1, ID. #1538
  * Gorti et al. (2015) Gorti, U., Hollenbach, D., & Dullemond, C. P. 2015, ApJ, 804, 29, doi: 10.1088/0004-637X/804/1/29
  * Gressel et al. (2020) Gressel, O., Ramsey, J. P., Brinch, C., et al. 2020, ApJ, 896, 126, doi: 10.3847/1538-4357/ab91b7
  * Grevesse et al. (1992) Grevesse, N., Noels, A., & Sauval, A. J. 1992, in ESA Special Publication, Vol. 348, Coronal Streamers, Coronal Loops, and Coronal and Solar Wind Composition, ed. C. Mattok, 305–308
  * Grevesse & Sauval (1998) Grevesse, N., & Sauval, A. J. 1998, Space Sci. Rev., 85, 161, doi: 10.1023/A:1005161325181
  * Güdel et al. (2010) Güdel, M., Lahuis, F., Briggs, K. R., et al. 2010, A&A, 519, A113, doi: 10.1051/0004-6361/200913971
  * Güdel et al. (2018) Güdel, M., Eibensteiner, C., Dionatos, O., et al. 2018, A&A, 620, L1, doi: 10.1051/0004-6361/201834271
  * Hartigan et al. (1995) Hartigan, P., Edwards, S., & Ghandour, L. 1995, ApJ, 452, 736, doi: 10.1086/176344
  * Hendler et al. (2018) Hendler, N. P., Pinilla, P., Pascucci, I., et al. 2018, MNRAS, 475, L62, doi: 10.1093/mnrasl/slx184
  * Henning et al. (2017) Henning, T. K., Abergel, A., Absil, O., et al. 2017, MIRI EC Protoplanetary and Debris Disks Survey, JWST Proposal. Cycle 1, ID. #1282
  * Herczeg et al. (2007) Herczeg, G. J., Najita, J. R., Hillenbrand, L. A., & Pascucci, I. 2007, ApJ, 670, 509, doi: 10.1086/521545
  * Hollenbach & Gorti (2009) Hollenbach, D., & Gorti, U. 2009, ApJ, 703, 1203, doi: 10.1088/0004-637X/703/2/1203
  * Hollenbach et al. (1994) Hollenbach, D., Johnstone, D., Lizano, S., & Shu, F. 1994, ApJ, 428, 654, doi: 10.1086/174276
  * Hollenbach & McKee (1989) Hollenbach, D., & McKee, C. F. 1989, ApJ, 342, 306, doi: 10.1086/167595
  * Huélamo et al. (2015) Huélamo, N., de Gregorio-Monsalvo, I., Macias, E., et al. 2015, A&A, 575, L5, doi: 10.1051/0004-6361/201424404
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Igea & Glassgold (1999) Igea, J., & Glassgold, A. E. 1999, ApJ, 518, 848, doi: 10.1086/307302
  * Kashyap & Drake (2000) Kashyap, V., & Drake, J. J. 2000, Bulletin of the Astronomical Society of India, 28, 475
  * Kingdon & Ferland (1996) Kingdon, J. B., & Ferland, G. J. 1996, ApJS, 106, 205, doi: 10.1086/192335
  * Klaassen et al. (2016) Klaassen, P. D., Mottram, J. C., Maud, L. T., & Juhasz, A. 2016, MNRAS, 460, 627, doi: 10.1093/mnras/stw989
  * Klaassen et al. (2021) Klaassen, P. D., Geers, V. C., Beard, S. M., et al. 2021, MNRAS, 500, 2813, doi: 10.1093/mnras/staa3416
  * Komaki et al. (2021) Komaki, A., Nakatani, R., & Yoshida, N. 2021, ApJ, 910, 51, doi: 10.3847/1538-4357/abe2af
  * Kunitomo et al. (2020) Kunitomo, M., Suzuki, T. K., & Inutsuka, S.-i. 2020, MNRAS, 492, 3849, doi: 10.1093/mnras/staa087
  * Lahuis et al. (2007) Lahuis, F., van Dishoeck, E. F., Blake, G. A., et al. 2007, ApJ, 665, 492, doi: 10.1086/518931
  * Lee et al. (2021) Lee, C.-F., Tabone, B., Cabrit, S., et al. 2021, ApJ, 907, L41, doi: 10.3847/2041-8213/abda38
  * Lesur (2021) Lesur, G. R. J. 2021, A&A, 650, A35, doi: 10.1051/0004-6361/202040109
  * Liffman (2003) Liffman, K. 2003, PASA, 20, 337, doi: 10.1071/AS03019
  * Liu et al. (2014) Liu, C.-F., Shang, H., Walter, F. M., & Herczeg, G. J. 2014, ApJ, 786, 99, doi: 10.1088/0004-637X/786/2/99
  * López-Martínez & Gómez de Castro (2015) López-Martínez, F., & Gómez de Castro, A. I. 2015, MNRAS, 448, 484, doi: 10.1093/mnras/stu2690
  * Louvet et al. (2018) Louvet, F., Dougados, C., Cabrit, S., et al. 2018, A&A, 618, A120, doi: 10.1051/0004-6361/201731733
  * Manara et al. (2023) Manara, C. F., Ansdell, M., Rosotti, G. P., et al. 2023, in Astronomical Society of the Pacific Conference Series, Vol. 534, Protostars and Planets VII, ed. S. Inutsuka, Y. Aikawa, T. Muto, K. Tomida, & M. Tamura, 539, doi: 10.48550/arXiv.2203.09930
  * Martel & Lesur (2022) Martel, É., & Lesur, G. 2022, A&A, 667, A17, doi: 10.1051/0004-6361/202142946
  * McGinnis et al. (2018) McGinnis, P., Dougados, C., Alencar, S. H. P., Bouvier, J., & Cabrit, S. 2018, A&A, 620, A87, doi: 10.1051/0004-6361/201731629
  * McLaughlin & Bell (2000) McLaughlin, B. M., & Bell, K. L. 2000, Journal of Physics B Atomic Molecular Physics, 33, 597, doi: 10.1088/0953-4075/33/4/301
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 376, Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell, 127
  * Najita et al. (2010) Najita, J. R., Carr, J. S., Strom, S. E., et al. 2010, ApJ, 712, 274, doi: 10.1088/0004-637X/712/1/274
  * Nakatani et al. (2018a) Nakatani, R., Hosokawa, T., Yoshida, N., Nomura, H., & Kuiper, R. 2018a, ApJ, 857, 57, doi: 10.3847/1538-4357/aab70b
  * Nakatani et al. (2018b) —. 2018b, ApJ, 865, 75, doi: 10.3847/1538-4357/aad9fd
  * Nazari et al. (2024) Nazari, P., Tabone, B., Ahmadi, A., et al. 2024, arXiv e-prints, arXiv:2402.18631.  https://arxiv.org/abs/2402.18631
  * Nemer & Goodman (2023) Nemer, A., & Goodman, J. 2023, arXiv e-prints, arXiv:2311.01802, doi: 10.48550/arXiv.2311.01802
  * Nisini et al. (2018) Nisini, B., Antoniucci, S., Alcalá, J. M., et al. 2018, A&A, 609, A87, doi: 10.1051/0004-6361/201730834
  * Nomura et al. (2007) Nomura, H., Aikawa, Y., Tsujimoto, M., Nakagawa, Y., & Millar, T. J. 2007, ApJ, 661, 334, doi: 10.1086/513419
  * Olofsson et al. (2011) Olofsson, J., Benisty, M., Augereau, J. C., et al. 2011, A&A, 528, L6, doi: 10.1051/0004-6361/201016074
  * Owen et al. (2012) Owen, J. E., Clarke, C. J., & Ercolano, B. 2012, MNRAS, 422, 1880, doi: 10.1111/j.1365-2966.2011.20337.x
  * Owen et al. (2010) Owen, J. E., Ercolano, B., Clarke, C. J., & Alexander, R. D. 2010, MNRAS, 401, 1415, doi: 10.1111/j.1365-2966.2009.15771.x
  * Pascucci et al. (2021) Pascucci, I., Alexander, R., Ballabio, G., et al. 2021, Caught in the act of dispersing their disks? MIRI MRS can tell, JWST Proposal. Cycle 1, ID. #2260
  * Pascucci et al. (2023) Pascucci, I., Cabrit, S., Edwards, S., et al. 2023, in Astronomical Society of the Pacific Conference Series, Vol. 534, Protostars and Planets VII, ed. S. Inutsuka, Y. Aikawa, T. Muto, K. Tomida, & M. Tamura, 567, doi: 10.48550/arXiv.2203.10068
  * Pascucci et al. (2014) Pascucci, I., Ricci, L., Gorti, U., et al. 2014, ApJ, 795, 1, doi: 10.1088/0004-637X/795/1/1
  * Pascucci & Sterzik (2009) Pascucci, I., & Sterzik, M. 2009, ApJ, 702, 724, doi: 10.1088/0004-637X/702/1/724
  * Pascucci et al. (2011) Pascucci, I., Sterzik, M., Alexander, R. D., et al. 2011, ApJ, 736, 13, doi: 10.1088/0004-637X/736/1/13
  * Pascucci et al. (2020) Pascucci, I., Banzatti, A., Gorti, U., et al. 2020, ApJ, 903, 78, doi: 10.3847/1538-4357/abba3c
  * Pelan & Berrington (1995) Pelan, J., & Berrington, K. A. 1995, A&AS, 110, 209
  * Pelletier & Pudritz (1992) Pelletier, G., & Pudritz, R. E. 1992, ApJ, 394, 117, doi: 10.1086/171565
  * Pfalzner et al. (2022) Pfalzner, S., Dehghani, S., & Michel, A. 2022, ApJ, 939, L10, doi: 10.3847/2041-8213/ac9839
  * Picogna et al. (2021) Picogna, G., Ercolano, B., & Espaillat, C. C. 2021, MNRAS, 508, 3611, doi: 10.1093/mnras/stab2883
  * Picogna et al. (2019) Picogna, G., Ercolano, B., Owen, J. E., & Weber, M. L. 2019, MNRAS, 487, 691, doi: 10.1093/mnras/stz1166
  * Pohl et al. (2017) Pohl, A., Sissa, E., Langlois, M., et al. 2017, A&A, 605, A34, doi: 10.1051/0004-6361/201630234
  * Preibisch et al. (2005) Preibisch, T., Kim, Y.-C., Favata, F., et al. 2005, ApJS, 160, 401, doi: 10.1086/432891
  * Rab et al. (2023) Rab, C., Weber, M. L., Picogna, G., Ercolano, B., & Owen, J. E. 2023, ApJ, 955, L11, doi: 10.3847/2041-8213/acf574
  * Rab et al. (2022) Rab, C., Weber, M., Grassi, T., et al. 2022, A&A, 668, A154, doi: 10.1051/0004-6361/202244362
  * Rapson et al. (2015) Rapson, V. A., Sargent, B., Germano Sacco, G., et al. 2015, ApJ, 810, 62, doi: 10.1088/0004-637X/810/1/62
  * Rigliaco et al. (2013) Rigliaco, E., Pascucci, I., Gorti, U., Edwards, S., & Hollenbach, D. 2013, ApJ, 772, 60, doi: 10.1088/0004-637X/772/1/60
  * Rodenkirch et al. (2020) Rodenkirch, P. J., Klahr, H., Fendt, C., & Dullemond, C. P. 2020, A&A, 633, A21, doi: 10.1051/0004-6361/201834945
  * Rosotti (2023) Rosotti, G. P. 2023, New A Rev., 96, 101674, doi: 10.1016/j.newar.2023.101674
  * Rota et al. (2024) Rota, A. A., Meijerhof, J. D., van der Marel, N., et al. 2024, arXiv e-prints, arXiv:2401.05798, doi: 10.48550/arXiv.2401.05798
  * Rybicki & Lightman (1979) Rybicki, G. B., & Lightman, A. P. 1979, Radiative processes in astrophysics
  * Sacco et al. (2014) Sacco, G. G., Kastner, J. H., Forveille, T., et al. 2014, A&A, 561, A42, doi: 10.1051/0004-6361/201322273
  * Sacco et al. (2012) Sacco, G. G., Flaccomio, E., Pascucci, I., et al. 2012, ApJ, 747, 142, doi: 10.1088/0004-637X/747/2/142
  * Sarafidou et al. (2023) Sarafidou, E., Gressel, O., Picogna, G., & Ercolano, B. 2023, arXiv e-prints, arXiv:2310.01985, doi: 10.48550/arXiv.2310.01985
  * Saraph & Tully (1994) Saraph, H. E., & Tully, J. A. 1994, A&AS, 107, 29
  * Savage & Sembach (1996) Savage, B. D., & Sembach, K. R. 1996, ARA&A, 34, 279, doi: 10.1146/annurev.astro.34.1.279
  * Sellek et al. (2020) Sellek, A. D., Booth, R. A., & Clarke, C. J. 2020, MNRAS, 498, 2845, doi: 10.1093/mnras/staa2519
  * Sellek et al. (2021) Sellek, A. D., Clarke, C. J., & Booth, R. A. 2021, MNRAS, 506, 1, doi: 10.1093/mnras/stab1693
  * Sellek et al. (2022) Sellek, A. D., Clarke, C. J., & Ercolano, B. 2022, MNRAS, 514, 535, doi: 10.1093/mnras/stac1148
  * Sellek & et al. (in prep.) Sellek, A. D., & et al. in prep.
  * Shu et al. (1993) Shu, F. H., Johnstone, D., & Hollenbach, D. 1993, Icarus, 106, 92, doi: 10.1006/icar.1993.1160
  * Simon et al. (2016) Simon, M. N., Pascucci, I., Edwards, S., et al. 2016, ApJ, 831, 169, doi: 10.3847/0004-637X/831/2/169
  * Somigliana et al. (2020) Somigliana, A., Toci, C., Lodato, G., Rosotti, G., & Manara, C. F. 2020, MNRAS, 492, 1120, doi: 10.1093/mnras/stz3481
  * Suzuki & Inutsuka (2009) Suzuki, T. K., & Inutsuka, S.-i. 2009, ApJ, 691, L49, doi: 10.1088/0004-637X/691/1/L49
  * Suzuki et al. (2016) Suzuki, T. K., Ogihara, M., Morbidelli, A., Crida, A., & Guillot, T. 2016, A&A, 596, A74, doi: 10.1051/0004-6361/201628955
  * Szulágyi et al. (2012) Szulágyi, J., Pascucci, I., Ábrahám, P., et al. 2012, ApJ, 759, 47, doi: 10.1088/0004-637X/759/1/47
  * Thanathibodee et al. (2023) Thanathibodee, T., Espaillat, C., Calvet, N., & Franco-Hernandez, R. 2023, Searching for Evidence of EUV Photoevaporation in Actively Dispersing Protoplanetary Disks, JWST Proposal. Cycle 2, ID. #3983
  * van der Walt et al. (2011) van der Walt, S., Colbert, S. C., & Varoquaux, G. 2011, Computing in Science and Engineering, 13, 22, doi: 10.1109/MCSE.2011.37
  * Verner et al. (1996) Verner, D. A., Ferland, G. J., Korista, K. T., & Yakovlev, D. G. 1996, ApJ, 465, 487, doi: 10.1086/177435
  * Verner & Yakovlev (1995) Verner, D. A., & Yakovlev, D. G. 1995, A&AS, 109, 125
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Burovski, E., et al. 2020, scipy/scipy: SciPy 1.5.3, v1.5.3, Zenodo, Zenodo, doi: 10.5281/zenodo.4100507
  * Wang et al. (2019) Wang, L., Bai, X.-N., & Goodman, J. 2019, ApJ, 874, 90, doi: 10.3847/1538-4357/ab06fd
  * Wang & Goodman (2017) Wang, L., & Goodman, J. 2017, ApJ, 847, 11, doi: 10.3847/1538-4357/aa8726
  * Wardle & Koenigl (1993) Wardle, M., & Koenigl, A. 1993, ApJ, 410, 218, doi: 10.1086/172739
  * Weber et al. (2020) Weber, M. L., Ercolano, B., Picogna, G., Hartmann, L., & Rodenkirch, P. J. 2020, MNRAS, 496, 223, doi: 10.1093/mnras/staa1549
  * Weder et al. (2023) Weder, J., Mordasini, C., & Emsenhuber, A. 2023, arXiv e-prints, arXiv:2304.12380, doi: 10.48550/arXiv.2304.12380
  * Wölfer et al. (2023) Wölfer, L., Facchini, S., van der Marel, N., et al. 2023, A&A, 670, A154, doi: 10.1051/0004-6361/202243601
  * Xie & et al. (in prep.) Xie, C., & et al. in prep.
  * Xu et al. (2021) Xu, Z., Herczeg, G. J., Johns-Krull, C. M., & France, K. 2021, ApJ, 921, 181, doi: 10.3847/1538-4357/ac1739
  * Yan & Babb (2024) Yan, P.-G., & Babb, J. F. 2024, ApJ, 961, 43, doi: 10.3847/1538-4357/ad0e77

##  Appendix A Modifications to MIRISim

As the latest version of MIRISim was released prior to the launch of JWST, the
Calibration Data Products (CDP) it uses are from the ground-based testing, and
for several effects have been superseded by inflight measurements. For a
better direct comparison with the data we therefore downloaded the new CDP
files for the following calibrations: DISTORTION (9B.05.07), FLAT (08.01.00),
FRINGE (Channels 3&4 A/B/C only: 8P.02.00/9B.00.00/8P.01.06), PHOTOM
(9B.04.08) and PSF (09.02.00) and modified MIRISim to read these files instead
of its defaults.

We also modified MIRISim to set the observation time keywords in the output
file headers to the date of the real observations (as otherwise the pipeline
would default to using the ground-based calibration files when analyzing our
synthetic images). Moreover, as of version 1.11 of the pipeline, the “EXPMID”
keyword is required to calculate the time-dependent photometric calibration
and we modified MIRISim to include this in the headers.

In testing our method we found that when MIRISim interpolates an input FITS
file (as used for the line emission) onto its internal grid, flux on scales
smaller than a pixel - as expected for an unresolved wind - was not well-
conserved. We therefore implemented in MIRISim our own 2-dimensional histogram
interpolation method - where the luminosity in each cell in the input grid is
placed into the nearest neighbour cell in the MIRISim grid (or shared equally
between cells in the case of a tie) - which we found corrected this issue
adequately.

We also discovered that MIRISim did not apply the PSF convolution to extended
sources when simulating MIRI-MRS. We thus include in our interpolation a step
to upsample the PSF over the field of view at the resolution of the input FITS
file, and apply the convolution to the data before the histogram rebinning is
done. This is done in each wavelength channel of the input file. Finally, the
contribution of the convolved, rebinned emission from each channel is added to
all wavelength channels of MIRI-MRS according to the line spread function. An
equivalent PSF upsampling and convolution method was applied to the disk
model.

##  Appendix B Cross sections of the wind for different parameters

Figure 15 shows that for a lower mass-loss rate, the wind is more highly
ionized. Collisions with electrons thus dominate over neutral H throughout the
emitting region for all lines of interest. The [Ar II] emission is comparably
extended to that of the [Ne II]

Figure 16 shows that for a larger $r_{\rm in}$, the emitting regions are
generally very thin and close to the inner radial boundary of the wind where
the wind is ionized. This means that electrons dominate over neutral H
throughout the emitting region for all lines of interest. The [Ar II] emission
is now slightly more extended than that of the [Ne II]

Figure 15: As with Figure 1 but for a gas density 10 times lower. Figure 16:
As with Figure 1 but for $r_{\rm in}=r_{G}$. Note the change in scale of the
panels to accommodate the larger scales on which the wind is ionized.

##  Appendix C Effect of collisional excitation by neutral H

In Figures 1, 15 and 16 we compare the critical densities (at the average gas
temperature indicated in the first panel) for the excitation of the mid-IR
noble gas lines by neutral H and electrons to the densities of these species
determined from the photoionization radiative transfer.

For mass-loss rates $\lesssim 5\times 10^{-9}\,M_{\odot}\,\mathrm{yr^{-1}}$
(Figure 15), we can see that the low gas densities and moderate ionization
levels mean that neutral H never reaches the critical density, whereas the
electrons do within the inner $3-5\,\mathrm{au}$ of the wind. This is a clear
indication that in the inner parts of the wind, electrons are the more
important colliders. The neutral H only starts to dominate once the X-ray
ionization falls low enough, which happens outside $\sim 10\,\mathrm{au}$,
while the majority of the line emission (assuming only electrons) comes from
inside $10\,\mathrm{au}$; even then the first species for which neutral H
comes a significant collider is Ne III which presents the most compact
emission under these conditions as it predominantly comes from the highly
ionization EUV-heated region. Therefore, for all but the highest mass-loss
rates we can be sure that the lack of neutral H as a collider cannot
significantly affect our results as it only increases the excitation
significantly in already very faint regions.

At the higher mass-loss rates that we prefer (Figure 1), however, the denser
wind has somewhat lower ionization levels, since recombination is more
effective and the ionizing radiation is more easily attenuated. Now, the
collisional excitation with neutral H can dominate over that with electrons on
similar scales to where the wind emission is mostly coming from. The Ar lines
are the least affected since the emitting extent is limited more by the
ionization: for [Ar II] most emission originates within $\sim 3\mathrm{au}$
while neutral H becomes important beyond $\sim 6\,\mathrm{au}$. Consequently,
the Ar II/III luminosities should be robust predictions, and the Ar II
emission should remain compact. On the other hand, since the critical
densities of the Ne lines due to neutral H are much closer to those with
electrons than for the Ar lines, neutral H dominates most of the emitting
region of the wind, save for the very inner edge where the photoionization by
EUV is still significant.

In these cases, the line emission may be boosted by a factor up to
$\frac{n_{\rm H}}{n_{\rm e}}/\frac{n_{\rm crit,H}}{n_{\rm crit,e}}$. However
this is an upper limit: for example in the case of the [Ne III] line the
collisions with neutral H bring most of the emitting region into LTE, when the
excitation (and hence line emission) saturates and is no longer sensitive to
the collider densities. Moreover, the broader inclusion of such additional
pathways for collisionally-excited emission would lead to additional cooling
(Sellek & et al., in prep.), thus lowering the temperature of the outer
regions of the wind somewhat. This would both reduce the line excitation and
raise the critical densities with neutral H of the lines of interest, thus
reducing the additional emission with respect to what might be estimated from
these figures.

A similar picture is seen for a larger $r_{\rm in}$ (Figure 16), but here the
emission (assuming only electrons) was generally limited to the EUV-ionized
region. This means that the X-ray-ionized regions contribute relatively less
to the overall line flux than at smaller $r_{\rm in}$ and hence boosting the
excitation in this region would have a more limited effect.

It is overall therefore likely that in denser environments neutral H can be an
important collider and now that rates are available (Yan & Babb, 2024), this
should be considered more explicitly and self-consistently in future works in
order to disentangle some of the opposing effects described above, at least
for the densest winds. This may in turn affect the accuracy with which our
preferred model of a dense wind - with a high mass-loss rate and small inner
radius - agrees with the data. However, since a lot of our conclusions are
based on ruling out the rest of the parameter space where neutral H is less
significant on the basis of line ratios, we can satisfactorily argue that they
will still not describe the data if neutral H is considered. Moreover, some of
its effects would only strength our conclusions: for example, since in the
dense winds collisional excitation by neutral H affects Ne II much more than
Ar II, it should increase the emitting extent of [Ne II] with respect to [Ar
II] and hence the dense winds would remain the most consistent with extended
[Ne II] emission but compact [Ar II].


11institutetext: University of Glasgow, Glasgow, Scotland, UK
22institutetext: Cancer Research UK Scotland Institute, Glasgow, Scotland, UK
33institutetext: University College London, London, England, UK
33email<EMAIL_ADDRESS>
# Synthetic Privileged Information Enhances Medical Image Representation
Learning

Lucas Farndale✉ 112 2 https://orcid.org/0009-0003-3667-2001 Chris Walsh 1122
https://orcid.org/0000-0002-6890-2401 Robert Insall 112233
https://orcid.org/0000-0003-4898-040X Ke Yuan 1122
https://orcid.org/0000-0002-2318-1460

###### Abstract

Multimodal self-supervised representation learning has consistently proven to
be a highly effective method in medical image analysis, offering strong task
performance and producing biologically informed insights. However, these
methods heavily rely on large, paired datasets, which is prohibitive for their
use in scenarios where paired data does not exist, or there is only a small
amount available. In contrast, image generation methods can work well on very
small datasets, and can find mappings between unpaired datasets, meaning an
effectively unlimited amount of paired synthetic data can be generated. In
this work, we demonstrate that representation learning can be significantly
improved by synthetically generating paired information, both compared to
training on either single-modality (up to $4.4\times$ error reduction) or
authentic multi-modal paired datasets (up to $5.6\times$ error reduction).

###### Keywords:

Image Generation Pathology Knowledge Distillation

##  1 Introduction

Recent work [1, 2, 3] has established the efficacy of integrating multiple
sources of data to improve representation learning for histopathology. It has
been demonstrated that integrating additional information, either advanced,
high content information or something more routine, can greatly enhance the
features that are learned by models, as it can act as an additional source of
implicit supervision. This is particularly notable for privileged information
– information that is available during training but not inference – which can
be distilled into models to greatly improve single-modality downstream task
performance where the privileged information is not available. Paired datasets
can be used to distil knowledge from the privileged data to the primary data,
producing much more biologically informative representations and improving
performance on a variety of a priori unseen tasks during evaluation by only
training a small classifier head [1].

(a)

(b)

Figure 1: (a) Schematic of synthetic paired data being generated and passed to
the self-supervised model (b) Comparison of standard multimodal self-
supervised approaches against our proposed method using synthetically
generated data. Images from [4].

A major drawback of these approaches, however, is the reliance on large
datasets where data are paired one-to-one during training. In many cases, such
as histology, the privileged data of interest only exist in small datasets or
in distinct, unpaired datasets which have not been previously amenable to
these approaches. Image generation methods typically require orders of
magnitude less data than self-supervised methods, with high quality image
generation being possible with hundreds of examples (e.g. [4]), compared to
hundreds of thousands for self-supervised learning [5]. Furthermore, cycleGAN
[6] does not require a one-to-one pairing of images and can achieve excellent
results for image generation. We may therefore construct arbitrarily large,
paired datasets using models which were trained on few or no paired samples.

In this work, we demonstrate that the benefits of training on large, paired
datasets can be replicated using synthetically generated paired data for
privileged self-supervised training. We show that high quality, biologically
informed self-supervised models can be trained that benefit from privileged
information, but only require a single data source for training and inference
by using that input to generate a synthetic privileged pair. The key
contributions of this work are:

  * •

We demonstrate that synthetically generated data can be used as privileged
information in place of authentic source data, including where there is little
or no real paired data, to improve model performance;

  * •

We show that these models are more robust to distribution shift than
unprivileged models, and encode meaningful biological features;

  * •

We demonstrate that models trained with synthetic data closely mimic the
representations learned from real data, and learn extra information compared
to training with a single modality;

  * •

We demonstrate the extra benefit from distilling synthetically generated data
when the real dataset size is limited.

##  2 Background and Related Work

Self-Supervised Learning (SSL) is a midpoint between supervised and
unsupervised, where inherent structural properties of data are leveraged to to
create inductional biases which enable the model to learn without labels. The
goal of this is to produce a vector representation of the image which has a
much lower dimension than the original image but contains the useful semantics
of the image needed for downstream applications. Recent work has extensively
applied SSL in a wide variety of domains and tasks [7], and state-of-the-art
SSL models have been shown to outperform both supervised and unsupervised
models in many settings.

Privileged Information is any information that is available during training
but not during inference. In a medical imaging context, this is true of an
enormous corpus of scientific data, which has already been generated in one
condition, but which would be impractical or expensive to generate again for
future studies or to create a larger dataset. We are primarily motivated by
histopathology, where vast quantities of brightfield H&E stain datasets are
available, but comparatively few for immunohistochemistry (IHC) stains and
fewer again for state-of-the-art data such as spatial -omics, multiplex IHC,
super-resolution microscopy or expert pixel-level annotations.

As Siamese networks will generally learn to encode only features which are
shared between branches, if the inputs are different then only semantic
content shared between inputs is learned. This means that useful features can
be neglected if they are not shared. For example, it was shown that pairing
thresholded immunofluorescence (IF) images with H&E images significantly
decreased model downstream performance on the H&E images because the model
neglected many details, such as fine-grained tissue structure, in these images
that were not detectable in the IF, which were mostly black [1].

TriDeNT [1] was developed to address this issue by only neglecting features
that are not present in the primary image. This means that features may still
be neglected by the encoder of the privileged input, but only the encoder of
the primary input is of interest for downstream tasks. TriDeNT works by
utilising an additional primary input branch to generate a third
representation, meaning that there are three concurrent joint-embedding latent
spaces. The model must then balance retaining features that are present in
both primary branches with those shared by the privileged information, and was
shown to retain both feature sets, significantly improving downstream
performance.

Image Generation methods have been extensively applied in medical imaging,
both generating realistic images from a random seed [8], or virtually
translating between modalities [9] or stains [10, 11]. Some image methods
require paired data, such as pix2pix [12], but others, such as cycleGAN [6],
can find mappings between unpaired datasets. This is especially useful in the
context of histology, as the vast majority of additional stains are not from
the same slice as their paired H&E stains, and often are from different
sections of the tissue block. This means it is hard to determine which parts
of the images should be aligned when comparing different stains, as their
morphologies can be very different. It has been shown that CycleGANs can be
used to generate realistic IHC stains directly from H&E stains without
requiring corresponding input/target pairs [13, 10], and we demonstrate here
that these synthetically generated stains can be used to improve the
performance of self-supervised models using TriDeNT. These stains usually have
the advantage of being perfectly spatially registered and aligned.

It has been shown that the representations from generative models perform
poorly when evaluated on downstream tasks [14], with even support vector
machines outperforming state-of-the-art generative models [15]. This is due to
the models being highly optimised for the specific task of image generation,
and therefore having representations which contain information, such as
precise spatial locations of features, which are not useful for downstream
tasks such as tissue type classification. It is therefore essential for
representation learning to distil the useful information from these models’
representations without over-constraining the model to replicate all of the
features present, which reduces the quality of representations.

(a)

(b)

|  |  | PanNuke | IHC (cycleGAN) | IHC (pix2pix) | SHIFT
---|---|---|---|---|---|---
Loss | Method | Privileged | NCT | Camelyon | NCT | Camelyon | NCT | Camelyon | NCT | Camelyon
VICReg | TriDeNT | ✓ |  0.9359 |  0.9149 |  0.9252 |  0.9263 |  0.9184 |  0.8960 |  0.9383 | 0.6294
Siamese | ✓ |  0.9044 |  0.8952 |  0.9234 |  0.9462 |  0.9250 |  0.8994 |  0.9236 | 0.5975
✗ NCT | 0.8937 | 0.7056 | 0.8937 | 0.7056 | 0.8937 | 0.7056 | 0.8937 |  0.7056
✗ Real | $0.7301^{\dagger}$ | 0.4636 | 0.8434 | 0.5360 | 0.8434 | 0.5360 | 0.8008 |  0.6387
InfoNCE | TriDeNT | ✓ |  0.9287 |  0.8719 |  0.9331 |  0.9033 |  0.9302 |  0.8992 |  0.9115 |  0.6430
Siamese | ✓ |  0.9103 |  0.8889 |  0.9260 |  0.9356 |  0.9259 |  0.9224 |  0.9299 | 0.6007
✗ NCT | 0.9047 | 0.6205 | 0.9047 | 0.6205 | 0.9047 | 0.6205 | 0.9047 | 0.6205
✗ Real | $0.7199^{\dagger}$ | 0.5852 | 0.8623 | 0.5600 | 0.8623 | 0.5600 | 0.7921 |  0.7056
CrossEntropy | Supervised | ✗ | 0.9245 | 0.8440 | 0.9245 | 0.8440 | 0.9245 | 0.8440 | 0.9245 | 0.8440

(c)

|  |  | PanNuke | IHC (cycleGAN) | SHIFT
---|---|---|---|---|---
|  |  | NCT | Camelyon | NCT | Camelyon | NCT | Camelyon
Loss | Method | Privileged | Synthetic | Real | Synthetic | Real | Synthetic | Real | Synthetic | Real | Synthetic | Real | Synthetic | Real
VICReg | TriDeNT | ✓ |  0.9359 | $0.7337^{\dagger}$ |  0.9149 | 0.5306 |  0.9252 | 0.8650 |  0.9263 | 0.6266 |  0.9383 | 0.7624 |  0.6294 | 0.5339
Siamese | ✓ |  0.9044 | $0.6000^{\dagger}$ |  0.8952 | 0.4831 |  0.9234 | 0.8256 |  0.9462 | 0.8614 |  0.9236 | 0.7439 |  0.5975 | 0.5120
InfoNCE | TriDeNT | ✓ |  0.9287 | $0.7530^{\dagger}$ |  0.8719 | 0.6908 |  0.9331 | 0.8722 |  0.9033 | 0.8304 |  0.9115 | 0.7998 |  0.6430 | 0.5653
Siamese | ✓ |  0.9103 | $0.5289^{\dagger}$ |  0.8889 | 0.7134 |  0.9260 | 0.8370 |  0.9356 | 0.8423 |  0.9299 | 0.7464 | 0.6007 |  0.6601

(d)

Figure 2: (a) Guided-GradCAM [16] maps for samples from the NCT dataset.
Predicted labels for each classifier are shown, as is the true label. Labels
are adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus
(MUC), muscle (MUS), normal mucosa (NORM), stroma (STR), and tumour (TUM). See
Figure S1 for examples of all classes; (b) Breakdown of performance on the NCT
evaluation task by class; (c) Results for models trained on the NCT dataset
with synthetically generated privileged information as pairs for the NCT
patches. Values marked with “†” are from [1]; (d) Classification performance
on NCT and Camelyon of models trained on the PanNuke/IHC/SHIFT datasets
compared to the synthetically generated images paired with the NCT dataset.

##  3 Methodology and Datasets

In all tasks we follow the implementation details of [1]. We compare TriDeNT
to Siamese baselines, with and without privileged information, and also to
supervised learning. Encoders are ResNet-50 [17] trained for 100 epochs with a
warmup-cosine learning rate, with maximum value $10^{-4}$. Three-layer dense
projection heads of width 8192 with batch normalisation were used, and
classifier heads were single dense layers with softmax activations. For full
details of datasets used, see Table S1. We employ three evaluation tasks:
tissue classification on the NCT dataset [18], metastasis detection on the
Camelyon17 dataset [19], and malignancy detection on the PanNuke dataset [20].
We use existing image generation models trained on PanNuke, SHIFT [4] and a
pix2pix model and a cycleGAN both trained on the IHC dataset detailed in [10].
The pix2pix model uses paired image-to-image translation, while the cycleGAN
uses unpaired images, allowing the necessity for paired data in model training
to be assessed.

##  4 Results

###  4.1 Synthetically Generated Segmentations as Privileged Data Improves
Downstream Classification Performance

In [14], it was shown that standard Siamese unprivileged models failed to
learn sub-nuclear features from histopathology images. This was addressed by
pairing these images with machine-generated nuclear segmentations, which
prompted the model to learn features related specifically to the nuclei. This
was then further explored in [1], where it was shown that training models
using TriDeNT on the Pannuke dataset [20] with the segmentations as privileged
information greatly improves their downstream task performance. Here we use
the setting of [14], generating segmentation maps using HoVer-Net [21], and
study the effect of using this as privileged information for TriDeNT. Note
that PanNuke contains train/validation/test folds of approximately 2000 images
each, meaning the SSL training dataset size is substantially increased by
generating synthetically generated data from the NCT dataset.

The results in Figure 2 demonstrate that distilling information from the
synthetically generated nuclear segmentations greatly improves the performance
of the model, and synthetically generated privileged information is found to
be effectively distilled into the primary models. Notably, unprivileged
learning on PanNuke or the NCT dataset performed poorly at distinguishing
stroma from muscle, which requires subtle biological features to be learned.
Models with synthetically generated privileged information performed
considerably better. Encoders trained with privileged information retain
features of both the background/connective tissue (using information from only
the H&E patches) and the nuclei (using information from both the H&E patches
and the segmentation masks), enabling them to make accurate predictions across
different tissue types. Figures 2(a) and S1 demonstrate that the TriDeNT model
with synthetically generated privileged information learns to accurately
identify the morphology of the nuclei in the images, making it easier to
identify the phenotypic change between muscle and stroma. It was previously
observed that unprivileged Siamese models tend to focus on the background,
rather than the potentially more informative features of the nuclei [1].
Privileged training with segmentation masks produced models which focused more
on nuclei, this is equally true of synthetically generated privileged
information. We find that even providing only the segmented nuclei as
privileged information achieves improvements, although to a lesser extent than
also providing the type annotations, as shown in Table S2.

We note that there is a marked performance improvement for models with
synthetically generated privileged information on the Camelyon17 task, which
assesses the ability of the model to both detect metastasis and transfer to
out-of-distribution datasets. Performance on this task is a good indicator of
whether meaningful, generalisable biological features have been learned, as
opposed to features specific to the train set or simply the contours of an
image. It is known that privileged information causes in models to learn
representations which are less prone to overfitting on small changes in
irrelevant features such as colour, brightness, or blur, which can easily
arise between datasets [1]. Instead, models must focus on more useful semantic
features shared between primary and privileged branches, such as the nuclei in
Figures 2(a) and S1, which naturally gives rise to representations which
transfer well to out-of-distribution data. We find that this is equally true
for synthetic privileged information.

We also find that training on synthetically generated training sets offers
significant improvements compared to using privileged information from the
real source dataset, as shown in Figure 2(d). This is due to the synthetic
dataset being either larger (compared to PanNuke and SHIFT) or being more
curated to encompass a distribution of samples (compared to IHC). This
demonstrates that not only is using synthetic privileged information a good
method to enhance performance on tasks related to the dataset the synthetic
privileged information is generated _for_ , but it can also offer better
results on tasks relating to the dataset the synthetic privileged information
was generated _from_.

As above, we see significant performance improvements for both pix2pix and
cycleGAN models, showing that no real paired data is needed to achieve good
results. It is notable that the synthetic SHIFT dataset improves performance
on the NCT task, although this does not generalise well to the Camelyon
evaluation task. The SSL performance is improved in spite of the generator
making many verifiable errors, such as predicting fluorescence in patches
labelled “background” or “debris”, which makes little biological sense. This
demonstrates that the addition of any useful privileged information can
enhance performance, and synthetically generated data does not need to be
perfect, with even models trained on very small datasets being suitable.

Supervised | Method | Privileged | Synthetic | Real
---|---|---|---|---
✗ | TriDeNT | ✓ |  0.7508 | 0.5545
Siamese | ✓ | 0.5523 | 0.6399
✗ | 0.5313 | 0.5264
✓ | TriDeNT | ✓ |  0.9159 | 0.9080
Siamese | ✓ | 0.8931 | 0.8274
✗ | 0.8914 | 0.8506

(a)

(b)

Figure 3: (a) Evaluation of representations from models trained on the real
PanNuke dataset and the synthetically generated nuclear segmentations paired
with the NCT dataset. (b) UMAP projections, coloured by PanNuke label.

###  4.2 Training With Synthetically Generated Privileged Data Outperforms
Training on the Real Source Dataset

Another key use case for synthetically generated privileged information is
where only a small amount of data is available. In Figure 3(a), we evaluate
models trained on the small PanNuke dataset by performing unsupervised
$k$-means clustering with 2 clusters to determine binary labels. Trained with
synthetically generated privileged information, TriDeNT was able to accurately
make completely unsupervised classifications. For all other methods,
performance is little better than random, implying that neither the PanNuke
train set nor unprivileged training on the larger dataset is sufficient for
good performance. This can be intuitively seen from Figure 3(b), where TriDeNT
differentiates between the two classes more than unprivileged methods, with
qualitative similarities between the projections of real and synthetic models.
This shows that this approach could be used to better analyse small datasets
from rare diseases or underrepresented populations, which may not have
sufficient labels for classifier training, either for basic research or in a
clinical setting. Supervised evaluation produces similar results, with the
TriDeNT model trained on NCT with synthetically generated PanNuke patches
outperforming training directly on the real PanNuke dataset.

It is notable that models performed well which were trained with synthetically
generated privileged information from small datasets such as PanNuke and
SHIFT, which have generator train sets of 2205 and 665 images respectively.
Figure 2(d) demonstrates that the reduction in error from training on the
larger synthetic dataset compared to the real smaller dataset is more than
400% and 200% for PanNuke and SHIFT respectively.

##  5 Discussion and Conclusions

###  5.1 Synthetically Generated Privileged Data Improves Model Performance in
Both In- and Out-of-Distribution Settings

We have established that the performance of single-input models is
significantly improved by training with synthetically generated paired data.
This can be considered as a form of implicit multi-objective learning, as the
image generation model has been optimised to perform a different task to the
self-supervised pretext task, and this provides the commensurate benefits.
Despite this, the required computational power at any point is no different to
standard Siamese/TriDeNT learning, as the images are generated prior to the
self-supervised training. In many cases, these generative models are already
available and could be used to create images without being trained. This means
that these methods can be accessible to clinicians and researchers without
large-scale resources, and enable the application of self-supervised
privileged training to datasets which are otherwise intractable due to a lack
of paired data.

###  5.2 Clinical Impact and Future Work

This work was primarily motivated by two use cases: unpaired datasets, and
small/rare datasets. Many state-of-the-art datasets do not have accompanying
routine data, and this approach allows insights from those data to be
integrated into routine data. Furthermore, many datasets are very limited in
size or scope, particularly for rare diseases or those affecting
underrepresented populations. This approach could enable retrospective
reanalysis of routine data from rare samples using insights from current
state-of-the-art data, meaning routine data from both rare and common cases
can be fully utilised. This is especially prescient, given that TriDeNT was
shown to significantly outperform comparable methods for few-shot applications
[1].

We have limited the scope of this study to the case of a single paired input,
but further improvements could be made with the addition of multiple paired
datasets. This could be easily achieved by applying several generative models
to the same large primary dataset. We have also not considered training with
the real data shuffled into the synthetically generated dataset, which could
further improve performance.

## Acknowledgements

LF is supported by the MRC grant MR/W006804/1, CW and RI acknowledge support
from CRUK A24450. RI is supported by EPSRC EP/S0300875/1 and Wellcome
221786/Z/20/Z. KY acknowledges support from EP/R018634/1, BB/V016067/1. We
acknowledge support from A. Ammar, J. Hay, H. Morgan, C. Nixon, J. Quinn, J.
Edwards and the Glasgow Tissue Research Facility.

## References

  * [1] Lucas Farndale, Robert Insall, and Ke Yuan.  Trident: Triple deep network training for privileged knowledge distillation in histopathology.  arXiv preprint arXiv:2312.02111, 2023.
  * [2] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  * [3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [4] Erik A Burlingame, Mary McDonnell, Geoffrey F Schau, Guillaume Thibault, Christian Lanciault, Terry Morgan, Brett E Johnson, Christopher Corless, Joe W Gray, and Young Hwan Chang.  Shift: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning.  Scientific reports, 10(1):1–14, 2020.
  * [5] Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave.  Are large-scale datasets necessary for self-supervised pre-training?  arXiv preprint arXiv:2112.10740, 2021.
  * [6] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [7] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4037–4058, 2020.
  * [8] Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof.  Diffusion models in medical imaging: A comprehensive survey.  Medical Image Analysis, page 102846, 2023.
  * [9] Qing Lyu and Ge Wang.  Conversion between ct and mri images using diffusion and score-matching models.  arXiv preprint arXiv:2209.12104, 2022.
  * [10] Christopher D Walsh, Joanne Edwards, and Robert H Insall.  Ensuring accurate stain reproduction in deep generative networks for virtual immunohistochemistry.  arXiv preprint arXiv:2204.06849, 2022.
  * [11] Kevin de Haan, Yijie Zhang, Jonathan E Zuckerman, Tairan Liu, Anthony E Sisk, Miguel FP Diaz, Kuang-Yu Jen, Alexander Nobori, Sofia Liou, Sarah Zhang, et al.  Deep learning-based transformation of h&e stained tissues into special stains.  Nature communications, 12(1):4884, 2021.
  * [12] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [13] Zhaoyang Xu, Xingru Huang, Carlos Fernández Moro, Béla Bozóky, and Qianni Zhang.  Gan-based virtual re-staining: a promising solution for whole slide image analysis.  arXiv preprint arXiv:1901.04059, 2019.
  * [14] Lucas Farndale, Robert Insall, and Ke Yuan.  More from less: Self-supervised knowledge distillation for information-sparse histopathology data.  arXiv preprint arXiv:2303.10656, 2023.
  * [15] Adalberto Claudio Quiros, Nicolas Coudray, Anna Yeaton, Wisuwat Sunhem, Roderick Murray-Smith, Aristotelis Tsirigos, and Ke Yuan.  Adversarial learning of cancer tissue representations.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24, pages 602–612. Springer, 2021.
  * [16] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Jakob Nikolas Kather, Niels Halama, and Alexander Marx.  100,000 histological images of human colorectal cancer and healthy tissue, April 2018.
  * [19] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al.  From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.  IEEE Transactions on Medical Imaging, 2018.
  * [20] Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benes, Ali Khuram, and Nasir Rajpoot.  Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification.  In European Congress on Digital Pathology, pages 11–19. Springer, 2019.
  * [21] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot.  Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.  Medical Image Analysis, 58:101563, 2019.
  * [22] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou.  A visual–language foundation model for pathology image analysis using medical twitter.  Nature Medicine, pages 1–10, 2023.

## Supplementary Material

Table S1: Datasets

Name |  Tissue Type |  Train/Test Split | Training | Evaluation | Purpose | Task
---|---|---|---|---|---|---
NCT [18] | Colorectal | 100,000/7,177 | ✓ | ✓ |  Cancer Tissue Analysis | Tissue Type Classification
Note: Nine tissue types: adipose (ADI), background (BACK), debris (DEB),
lymphocytes (LYM), mucus (MUC), smooth
muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR),
colorectal adenocarcinoma epithelium (TUM)
Camelyon17 [19] | Lymph Node | 179,394/146,722 | ✗ | ✓ |  Out-of Distribution Metastasis Detection |  Out-of-Distribution Binary Classification
Note: Train set from 3 centers, validation set from center 4, and test set
from center 5
PanNuke [20] | Pan-Cancer | 2,205/4,373 | ✓ | ✓ | Nuclear Segmentation | Neoplastic Cell Detection
Note: As in [22, 1], patches are labelled as benign if no neoplastic cells are
present, and malignant if the patch contains at least
10 cells and 30% of the cells are neoplastic.
Synthetic IHC [10] | Breast | 93,952/- | ✓ | ✗ |  Virtual Restaining Brightfield AE1/AE3 | -
Note: Originally 23,488 $448\times 448$px patches, which were split into 4
smaller patches
SHIFT [4] | Prostate | 665/11,593 | ✓ | ✗ |  Virtual Restaining IF pan-CK | -
Note: All 12,258 patches used in SSL training. Generator trained on only 665.
Only 4 patients’ data in dataset.

Table S2: Results for models trained with synthetically generated nuclear
segmentations as privileged information for the NCT dataset. Unlike in the
results in the main text, these segmentations simply mask the background,
leaving the nuclei. There are no type annotations. Evaluation is performed on
the NCT and Camelyon datasets.

Loss | Method | Privileged | NCT | Camelyon
---|---|---|---|---
VICReg | TriDeNT | ✓ |  0.9280 |  0.8330
Siamese | ✓ | 0.7778 | 0.7871
✗ | 0.8937 | 0.7056
InfoNCE | TriDeNT | ✓ |  0.9334 | 0.6859
Siamese | ✓ | 0.9058 |  0.8878
✗ | 0.9047 | 0.6205

Fig. S1: Representative samples from each NCT tissue class with Guided-GradCAM
activation maps. True and predicted labels are shown for each classifier.