* Izotov et al. (2020) Izotov YI, Schaerer D, Worseck G, Verhamme A, Guseva NG, et al. 2020. MNRAS 491:468–482
  * Izotov et al. (2017) Izotov YI, Thuan TX, Guseva NG. 2017. MNRAS 471:548–561
  * Izotov et al. (2021) Izotov YI, Worseck G, Schaerer D, Guseva NG, Chisholm J, et al. 2021. MNRAS 503:1734–1752
  * Izotov et al. (2018) Izotov YI, Worseck G, Schaerer D, Guseva NG, Thuan TX, et al. 2018. MNRAS 478:4851–4865
  * Izumi et al. (2021) Izumi T, Matsuoka Y, Fujimoto S, Onoue M, Strauss MA, et al. 2021. ApJ 914:36
  * Japelj et al. (2017) Japelj J, Vanzella E, Fontanot F, Cristiani S, Caminha GB, et al. 2017\. MNRAS 468:389–403
  * Jaskot & Ravindranath (2016) Jaskot AE, Ravindranath S. 2016. ApJ 833:136
  * Jiang et al. (2021) Jiang L, Kashikawa N, Wang S, Walth G, Ho LC, et al. 2021. Nature Astronomy 5:256–261
  * Jones et al. (2021) Jones LH, Barger AJ, Cowie LL. 2021. arXiv e-prints :arXiv:2101.04669
  * Jones et al. (2020) Jones T, Sanders R, Roberts-Borsani G, Ellis RS, Laporte N, et al. 2020\. ApJ 903:150
  * Jung et al. (2019) Jung I, Finkelstein SL, Dickinson M, Hutchison TA, Larson RL, et al. 2019\. ApJ 877:146
  * Jung et al. (2020) Jung I, Finkelstein SL, Dickinson M, Hutchison TA, Larson RL, et al. 2020\. ApJ 904:144
  * Kakiichi et al. (2021) Kakiichi K, Ellis RS, Hennawi J, Katz H, Nakajima K, et al. 2021. NIRSpec Integral Field Spectroscopy of LyC-Leaking Galaxies. JWST Proposal. Cycle 1
  * Kakiichi et al. (2018) Kakiichi K, Ellis RS, Laporte N, Zitrin A, Eilers AC, et al. 2018. MNRAS 479:43–63
  * Kakiichi & Gronke (2021) Kakiichi K, Gronke M. 2021. ApJ 908:30
  * Kartaltepe et al. (2021) Kartaltepe J, Casey CM, Bagley M, Bongiorno A, Capak PL, et al. 2021. COSMOS-Webb: The Webb Cosmic Origins Survey. JWST Proposal. Cycle 1
  * Katz et al. (2018) Katz H, Kimm T, Haehnelt M, Sijacki D, Rosdahl J, Blaizot J. 2018. MNRAS 478:4986–5005
  * Kawamata et al. (2018) Kawamata R, Ishigaki M, Shimasaku K, Oguri M, Ouchi M, Tanigawa S. 2018\. ApJ 855:4
  * Kendrew et al. (2015) Kendrew S, Scheithauer S, Bouchet P, Amiaux J, Azzollini R, et al. 2015\. PASP 127:623
  * Khaire et al. (2016) Khaire V, Srianand R, Choudhury TR, Gaikwad P. 2016. MNRAS 457:4051–4062
  * Kikuchihara et al. (2020) Kikuchihara S, Ouchi M, Ono Y, Mawatari K, Chevallard J, et al. 2020. ApJ 893:60
  * Kulkarni et al. (2019) Kulkarni G, Worseck G, Hennawi JF. 2019. MNRAS 488:1035–1065
  * Labbe et al. (2021) Labbe I, Bezanson R, Atek H, Brammer G, Coe D, et al. 2021. UNCOVER: Ultra-deep NIRCam and NIRSpec Observations Before the Epoch of Reionization. JWST Proposal. Cycle 1
  * Labbé et al. (2015) Labbé I, Oesch PA, Illingworth GD, van Dokkum PG, Bouwens RJ, et al. 2015. ApJS 221:23
  * Lagache et al. (2018) Lagache G, Cousin M, Chatzikos M. 2018. A&A 609:A130
  * Lam et al. (2019) Lam D, Bouwens RJ, Labbé I, Schaye J, Schmidt KB, et al. 2019. A&A 627:A164
  * Lapi et al. (2017) Lapi A, Mancuso C, Celotti A, Danese L. 2017. ApJ 835:37
  * Laporte et al. (2019) Laporte N, Katz H, Ellis RS, Lagache G, Bauer FE, et al. 2019. MNRAS 487:L81–L85
  * Laporte et al. (2021a) Laporte N, Meyer RA, Ellis RS, Robertson BE, Chisholm J, Roberts-Borsani GW. 2021a. MNRAS 505:3336–3346
  * Laporte et al. (2017) Laporte N, Nakajima K, Ellis RS, Zitrin A, Stark DP, et al. 2017. ApJ 851:40
  * Laporte et al. (2021b) Laporte N, Zitrin A, Ellis RS, Fujimoto S, Brammer G, et al. 2021b. MNRAS
  * Larson et al. (2018) Larson RL, Finkelstein SL, Pirzkal N, Ryan R, Tilvi V, et al. 2018. ApJ 858:94
  * Lewis et al. (2020) Lewis JSW, Ocvirk P, Aubert D, Sorce JG, Shapiro PR, et al. 2020. MNRAS 496:4342–4357
  * Lilly et al. (2017) Lilly SJ, Bordoloi R, Kashino D, Simcoe RA. 2017. Exploring the End of Cosmic Reionization. JWST Proposal. Cycle 1
  * Liu et al. (2016) Liu C, Mutch SJ, Angel PW, Duffy AR, Geil PM, et al. 2016. MNRAS 462:235–249
  * Livermore et al. (2017) Livermore RC, Finkelstein SL, Lotz JM. 2017. ApJ 835:113
  * Livermore et al. (2018) Livermore RC, Trenti M, Bradley LD, Bernard SR, Holwerda BW, et al. 2018\. ApJ 861:L17
  * Lotz et al. (2017) Lotz JM, Koekemoer A, Coe D, Grogin N, Capak P, et al. 2017. ApJ 837:97
  * Lovell et al. (2021) Lovell CC, Vijayan AP, Thomas PA, Wilkins SM, Barnes DJ, et al. 2021. MNRAS 500:2127–2145
  * Ma et al. (2016) Ma X, Hopkins PF, Kasen D, Quataert E, Faucher-Giguère CA, et al. 2016\. MNRAS 459:3614–3619
  * Ma et al. (2020) Ma X, Quataert E, Wetzel A, Hopkins PF, Faucher-Giguère CA, Kereš D. 2020. MNRAS 498:2001–2017
  * Madau (1995) Madau P. 1995. ApJ 441:18
  * Madau (2017) Madau P. 2017. ApJ 851:50
  * Madau & Dickinson (2014) Madau P, Dickinson M. 2014. ARA&A 52:415–486
  * Madau & Haardt (2015) Madau P, Haardt F. 2015. ApJ 813:L8
  * Madau et al. (1999) Madau P, Haardt F, Rees MJ. 1999. ApJ 514:648–659
  * Mainali et al. (2017) Mainali R, Kollmeier JA, Stark DP, Simcoe RA, Walth G, et al. 2017. ApJ 836:L14
  * Mainali et al. (2018) Mainali R, Zitrin A, Stark DP, Ellis RS, Richard J, et al. 2018. MNRAS 479:1180–1193
  * Maiolino et al. (2015) Maiolino R, Carniani S, Fontana A, Vallini L, Pentericci L, et al. 2015\. MNRAS 452:54–68
  * Malkan et al. (2021) Malkan MA, Alavi A, Atek H, Baronchelli I, Battisti AJ, et al. 2021. PASSAGE-Parallel Application of Slitless Spectroscopy to Analyze Galaxy Evolution. JWST Proposal. Cycle 1
  * Marchi et al. (2017) Marchi F, Pentericci L, Guaita L, Ribeiro B, Castellano M, et al. 2017\. A&A 601:A73
  * Marchi et al. (2018) Marchi F, Pentericci L, Guaita L, Schaerer D, Verhamme A, et al. 2018\. A&A 614:A11
  * Mashian et al. (2016) Mashian N, Oesch PA, Loeb A. 2016. MNRAS 455:2101–2109
  * Mason et al. (2019) Mason CA, Naidu RP, Tacchella S, Leja J. 2019. MNRAS 489:2669–2676
  * Mason et al. (2015) Mason CA, Trenti M, Treu T. 2015. ApJ 813:21
  * Mason et al. (2018) Mason CA, Treu T, Dijkstra M, Mesinger A, Trenti M, et al. 2018. ApJ 856:2
  * Matsuoka et al. (2016) Matsuoka Y, Onoue M, Kashikawa N, Iwasawa K, Strauss MA, et al. 2016. ApJ 828:26
  * Matsuoka et al. (2018) Matsuoka Y, Strauss MA, Kashikawa N, Onoue M, Iwasawa K, et al. 2018. ApJ 869:150
  * Matthee et al. (2017) Matthee J, Sobral D, Best P, Khostovan AA, Oteo I, et al. 2017. MNRAS 465:3637–3655
  * Mauerhofer et al. (2021) Mauerhofer V, Verhamme A, Blaizot J, Garel T, Kimm T, et al. 2021. A&A 646:A80
  * McGreer et al. (2015) McGreer ID, Mesinger A, D’Odorico V. 2015. MNRAS 447:499–505
  * McLeod et al. (2016) McLeod DJ, McLure RJ, Dunlop JS. 2016. MNRAS 459:3812–3824
  * McLeod et al. (2015) McLeod DJ, McLure RJ, Dunlop JS, Robertson BE, Ellis RS, Targett TA. 2015. MNRAS 450:3032–3044
  * McQuinn (2016) McQuinn M. 2016. ARA&A 54:313–362
  * Meneghetti et al. (2017) Meneghetti M, Natarajan P, Coe D, Contini E, De Lucia G, et al. 2017. MNRAS 472:3177–3216
  * Meštrić et al. (2020) Meštrić U, Ryan-Weber EV, Cooke J, Bassett R, Sawicki M, et al. 2020. MNRAS 494:4986–5007
  * Meyer et al. (2020) Meyer RA, Kakiichi K, Bosman SEI, Ellis RS, Laporte N, et al. 2020. MNRAS 494:1560–1578
  * Meyer et al. (2021) Meyer RA, Laporte N, Ellis RS, Verhamme A, Garel T. 2021. MNRAS 500:558–564
  * Mitra et al. (2015) Mitra S, Choudhury TR, Ferrara A. 2015. MNRAS 454:L76–L80
  * Mitra et al. (2018) Mitra S, Choudhury TR, Ferrara A. 2018. MNRAS 473:1416–1425
  * Monsalve et al. (2017) Monsalve RA, Rogers AEE, Bowman JD, Mozdzen TJ. 2017. ApJ 847:64
  * Morishita (2021) Morishita T. 2021. ApJS 253:4
  * Morishita et al. (2020) Morishita T, Stiavelli M, Trenti M, Treu T, Roberts-Borsani GW, et al. 2020. ApJ 904:50
  * Morishita et al. (2018) Morishita T, Trenti M, Stiavelli M, Bradley LD, Coe D, et al. 2018. ApJ 867:150
  * Mortlock et al. (2011) Mortlock DJ, Warren SJ, Venemans BP, Patel M, Hewett PC, et al. 2011. Nature 474:616–619
  * Mostardi et al. (2015) Mostardi RE, Shapley AE, Steidel CC, Trainor RF, Reddy NA, Siana B. 2015\. ApJ 810:107
  * Naidu et al. (2018) Naidu RP, Forrest B, Oesch PA, Tran KVH, Holden BP. 2018. MNRAS 478:791–799
  * Naidu et al. (2020) Naidu RP, Tacchella S, Mason CA, Bose S, Oesch PA, Conroy C. 2020. ApJ 892:109
  * Nakajima et al. (2016) Nakajima K, Ellis RS, Iwata I, Inoue AK, Kusakabe H, et al. 2016. ApJ 831:L9
  * Nakajima et al. (2020) Nakajima K, Ellis RS, Robertson BE, Tang M, Stark DP. 2020. ApJ 889:161
  * Norgaard-Nielsen & Perez-Gonzalez (2017) Norgaard-Nielsen HU, Perez-Gonzalez PG. 2017. The MIRI HUDF Deep Imaging Survey. JWST Proposal. Cycle 1
  * Oesch et al. (2021) Oesch P, Bouwens R, Brammer G, Chisholm J, Fudamoto Y, et al. 2021. FRESCO: The First Reionization Epoch Spectroscopic COmplete Survey. JWST Proposal. Cycle 1
  * Oesch et al. (2018) Oesch PA, Bouwens RJ, Illingworth GD, Labbé I, Stefanon M. 2018. ApJ 855:105
  * Oesch et al. (2016) Oesch PA, Brammer G, van Dokkum PG, Illingworth GD, Bouwens RJ, et al. 2016. ApJ 819:129
  * Oesch et al. (2015) Oesch PA, van Dokkum PG, Illingworth GD, Bouwens RJ, Momcheva I, et al. 2015. ApJ 804:L30
  * Onodera et al. (2020) Onodera M, Shimakawa R, Suzuki TL, Tanaka I, Harikane Y, et al. 2020. ApJ 904:180
  * O’Shea et al. (2015) O’Shea BW, Wise JH, Xu H, Norman ML. 2015. ApJ 807:L12
  * Ouchi et al. (2020) Ouchi M, Ono Y, Shibuya T. 2020. ARA&A 58:617–659
  * Ouchi et al. (2010) Ouchi M, Shimasaku K, Furusawa H, Saito T, Yoshida M, et al. 2010. ApJ 723:869–894
  * Pahl et al. (2020) Pahl AJ, Shapley A, Faisst AL, Capak PL, Du X, et al. 2020. MNRAS 493:3194–3211
  * Pahl et al. (2021) Pahl AJ, Shapley A, Steidel CC, Chen Y, Reddy NA. 2021. MNRAS 505:2447–2467
  * Parsa et al. (2018) Parsa S, Dunlop JS, McLure RJ. 2018. MNRAS 474:2904–2923
  * Pelliccia et al. (2020) Pelliccia D, Strait V, Lemaux BC, Bradac M, Coe D, et al. 2020. arXiv e-prints :arXiv:2011.08857
  * Pentericci et al. (2016) Pentericci L, Carniani S, Castellano M, Fontana A, Maiolino R, et al. 2016\. ApJ 829:L11
  * Pentericci et al. (2018) Pentericci L, Vanzella E, Castellano M, Fontana A, De Barros S, et al. 2018. A&A 619:A147
  * Planck Collaboration et al. (2020) Planck Collaboration, Aghanim N, Akrami Y, Ashdown M, Aumont J, et al. 2020. A&A 641:A6
  * Postman et al. (2012) Postman M, Coe D, Benítez N, Bradley L, Broadhurst T, et al. 2012\. ApJS 199:25
  * Puchwein et al. (2019) Puchwein E, Haardt F, Haehnelt MG, Madau P. 2019. MNRAS 485:47–68
  * Reddy et al. (2016) Reddy NA, Steidel CC, Pettini M, Bogosavljević M, Shapley AE. 2016\. ApJ 828:108
  * Rieke et al. (2017) Rieke G, Alberts S, Lyu J, Morrison J, Shivaei I. 2017. MIRI in the Hubble Ultra-Deep Field. JWST Proposal. Cycle 1
  * Rieke et al. (2015a) Rieke GH, Ressler ME, Morrison JE, Bergeron L, Bouchet P, et al. 2015a. PASP 127:665
  * Rieke et al. (2015b) Rieke GH, Wright GS, Böker T, Bouwman J, Colina L, et al. 2015b. PASP 127:584
  * Rieke et al. (2019) Rieke M, Arribas S, Bunker A, Charlot S, Finkelstein S, et al. 2019. BAAS 51:45
  * Rieke et al. (2005) Rieke MJ, Kelly D, Horner S. 2005. Overview of James Webb Space Telescope and NIRCam’s Role. In Cryogenic Optical Systems and Instruments XI, eds. JB Heaney, LG Burriesci, vol. 5904 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series
  * Rigby et al. (2017) Rigby JR, Bayliss M, Fischer TC, Florian M, Gladders MD, et al. 2017. TEMPLATES: Targeting Extremely Magnified Panchromatic Lensed Arcs and Their Extended Star Formation. JWST Proposal ID 1355. Cycle 0 Early Release Science
  * Roberts-Borsani et al. (2016) Roberts-Borsani GW, Bouwens RJ, Oesch PA, Labbe I, Smit R, et al. 2016\. ApJ 823:143
  * Roberts-Borsani et al. (2020) Roberts-Borsani GW, Ellis RS, Laporte N. 2020. MNRAS 497:3440–3450
  * Robertson et al. (2010) Robertson BE, Ellis RS, Dunlop JS, McLure RJ, Stark DP. 2010. Nature 468:49–55
  * Robertson et al. (2014) Robertson BE, Ellis RS, Dunlop JS, McLure RJ, Stark DP, McLeod D. 2014\. ApJ 796:L27
  * Robertson et al. (2015) Robertson BE, Ellis RS, Furlanetto SR, Dunlop JS. 2015. ApJ 802:L19
  * Rojas-Ruiz et al. (2020) Rojas-Ruiz S, Finkelstein SL, Bagley MB, Stevans M, Finkelstein KD, et al. 2020. ApJ 891:146
  * Rutkowski et al. (2016) Rutkowski MJ, Scarlata C, Haardt F, Siana B, Henry A, et al. 2016. ApJ 819:81
  * Rutkowski et al. (2017) Rutkowski MJ, Scarlata C, Henry A, Hayes M, Mehta V, et al. 2017. ApJ 841:L27
  * Salmon et al. (2020) Salmon B, Coe D, Bradley L, Bouwens R, Bradač M, et al. 2020. ApJ 889:189
  * Salmon et al. (2018) Salmon B, Coe D, Bradley L, Bradač M, Strait V, et al. 2018. ApJ 864:L22
  * Salpeter (1955) Salpeter EE. 1955. ApJ 121:161
  * Salvador-Solé et al. (2017) Salvador-Solé E, Manrique A, Guzman R, Rodríguez Espinosa JM, Gallego J, et al. 2017. ApJ 834:49
  * Schaerer et al. (2021) Schaerer D, Atek H, Chisholm J, Flury SR, Guseva NG, et al. 2021. LyC22 - Deep spectroscopic insights on star-forming galaxies 2.2 Gyr after the Big Bang. JWST Proposal. Cycle 1
  * Schechter (1976) Schechter P. 1976. ApJ 203:297–306
  * Schmidt et al. (2016) Schmidt KB, Treu T, Bradač M, Vulcani B, Huang KH, et al. 2016. ApJ 818:38
  * Schmidt et al. (2014) Schmidt KB, Treu T, Trenti M, Bradley LD, Kelly BC, et al. 2014. ApJ 786:57
  * Secunda et al. (2020) Secunda A, Cen R, Kimm T, Götberg Y, de Mink SE. 2020. ApJ 901:72
  * Seiler et al. (2018) Seiler J, Hutter A, Sinha M, Croton D. 2018. MNRAS 480:L33–L37
  * Shapley et al. (2021) Shapley AE, Sanders R, Berg D, Bouwens R, Brammer G, et al. 2021. The AURORA Survey: First Direct Metallicity Calibrations at High Redshift. JWST Proposal. Cycle 1
  * Shapley et al. (2016) Shapley AE, Steidel CC, Strom AL, Bogosavljević M, Reddy NA, et al. 2016. ApJ 826:L24
  * Sharma et al. (2016) Sharma M, Theuns T, Frenk C, Bower R, Crain R, et al. 2016. MNRAS 458:L94–L98
  * Shen et al. (2020) Shen X, Hopkins PF, Faucher-Giguère CA, Alexander DM, Richards GT, et al. 2020. MNRAS 495:3252–3275
  * Shipley et al. (2018) Shipley HV, Lange-Vagle D, Marchesini D, Brammer GB, Ferrarese L, et al. 2018. ApJS 235:14
  * Shivaei et al. (2018) Shivaei I, Reddy NA, Siana B, Shapley AE, Kriek M, et al. 2018. ApJ 855:42
  * Shull et al. (2012) Shull JM, Harness A, Trenti M, Smith BD. 2012. ApJ 747:100
  * Siana et al. (2015) Siana B, Shapley AE, Kulas KR, Nestor DB, Steidel CC, et al. 2015. ApJ 804:17
  * Smit et al. (2014) Smit R, Bouwens RJ, Labbé I, Zheng W, Bradley L, et al. 2014. ApJ 784:58
  * Smith et al. (2020) Smith BM, Windhorst RA, Cohen SH, Koekemoer AM, Jansen RA, et al. 2020\. ApJ 897:41
  * Sobacchi & Mesinger (2015) Sobacchi E, Mesinger A. 2015. MNRAS 453:1843–1854
  * Song et al. (2016a) Song M, Finkelstein SL, Ashby MLN, Grazian A, Lu Y, et al. 2016a. ApJ 825:5
  * Song et al. (2016b) Song M, Finkelstein SL, Livermore RC, Capak PL, Dickinson M, Fontana A. 2016b. ApJ 826:113
  * Stanway & Eldridge (2018) Stanway ER, Eldridge JJ. 2018. MNRAS 479:75–93
  * Stanway & Eldridge (2019) Stanway ER, Eldridge JJ. 2019. A&A 621:A105
  * Stanway et al. (2016) Stanway ER, Eldridge JJ, Becker GD. 2016. MNRAS 456:485–499
  * Stark (2016) Stark DP. 2016. ARA&A 54:761–803
  * Stark et al. (2017) Stark DP, Ellis RS, Charlot S, Chevallard J, Tang M, et al. 2017. MNRAS 464:469–479
  * Stark et al. (2015a) Stark DP, Richard J, Charlot S, Clément B, Ellis R, et al. 2015a. MNRAS 450:1846–1855
  * Stark et al. (2013) Stark DP, Schenker MA, Ellis R, Robertson B, McLure R, Dunlop J. 2013\. ApJ 763:129
  * Stark et al. (2015b) Stark DP, Walth G, Charlot S, Clément B, Feltre A, et al. 2015b. MNRAS 454:1393–1403
  * Stefanon et al. (2021) Stefanon M, Bouwens RJ, Labbé I, Illingworth GD, Gonzalez V, Oesch PA. 2021. arXiv e-prints :arXiv:2103.16571
  * Stefanon et al. (2017) Stefanon M, Bouwens RJ, Labbé I, Muzzin A, Marchesini D, et al. 2017\. ApJ 843:36
  * Stefanon et al. (2019) Stefanon M, Labbé I, Bouwens RJ, Oesch P, Ashby MLN, et al. 2019. ApJ 883:99
  * Steidel et al. (2018) Steidel CC, Bogosavljević M, Shapley AE, Reddy NA, Rudie GC, et al. 2018. ApJ 869:123
  * Strait et al. (2020) Strait V, Bradač M, Coe D, Bradley L, Salmon B, et al. 2020. ApJ 888:124
  * Strom et al. (2021) Strom AL, Rudie GC, Chen Y, Law DR, Maseda M, et al. 2021. CECILIA: A direct-method metallicity calibration for Cosmic Noon through the Epoch of Reionization. JWST Proposal. Cycle 1
  * Sun & Furlanetto (2016) Sun G, Furlanetto SR. 2016. MNRAS 460:417–433
  * Tamura et al. (2019) Tamura Y, Mawatari K, Hashimoto T, Inoue AK, Zackrisson E, et al. 2019\. ApJ 874:27
  * Tang et al. (2020) Tang M, Stark D, Chevallard J, Charlot S, Endsley R, Congiu E. 2020\. arXiv e-prints :arXiv:2012.04697
  * Tanvir et al. (2018) Tanvir NR, Laskar T, Levan AJ, Perley DA, Zabl J, et al. 2018. ApJ 865:107
  * The HERA Collaboration et al. (2021a) The HERA Collaboration, Abdurashidova Z, Aguirre JE, Alexander P, Ali Z, et al. 2021a. arXiv e-prints :arXiv:2108.07282
  * The HERA Collaboration et al. (2021b) The HERA Collaboration, Abdurashidova Z, Aguirre JE, Alexander P, Ali ZS, et al. 2021b. arXiv e-prints :arXiv:2108.02263
  * Tilvi et al. (2020) Tilvi V, Malhotra S, Rhoads JE, Coughlin A, Zheng Z, et al. 2020. ApJ 891:L10
  * Tilvi et al. (2016) Tilvi V, Pirzkal N, Malhotra S, Finkelstein SL, Rhoads JE, et al. 2016\. ApJ 827:L14
  * Topping et al. (2021) Topping MW, Shapley AE, Stark DP, Endsley R, Robertson B, et al. 2021\. ApJ 917:L36
  * Trenti et al. (2011) Trenti M, Bradley LD, Stiavelli M, Oesch P, Treu T, et al. 2011. ApJ 727:L39
  * Treu et al. (2015) Treu T, Schmidt KB, Brammer GB, Vulcani B, Wang X, et al. 2015. ApJ 812:114
  * Treu et al. (2017) Treu TL, Abramson LE, Bradac M, Brammer G, Fontana A, et al. 2017. Through the Looking GLASS: A JWST Exploration of Galaxy Formation and Evolution from Cosmic Dawn to Present Day. JWST Proposal ID 1324. Cycle 0 Early Release Science
  * Ucci et al. (2021) Ucci G, Dayal P, Hutter A, Yepes G, Gottlöber S, et al. 2021. MNRAS 506:202–214
  * Vanzella et al. (2016) Vanzella E, de Barros S, Vasei K, Alavi A, Giavalisco M, et al. 2016. ApJ 825:41
  * Vanzella et al. (2018) Vanzella E, Nonino M, Cupani G, Castellano M, Sani E, et al. 2018. MNRAS 476:L15–L19
  * Vasei et al. (2016) Vasei K, Siana B, Shapley AE, Quider AM, Alavi A, et al. 2016. ApJ 831:38
  * Vielfaure et al. (2020) Vielfaure JB, Vergani SD, Japelj J, Fynbo JPU, Gronke M, et al. 2020. A&A 641:A30
  * Vijayan et al. (2021) Vijayan AP, Lovell CC, Wilkins SM, Thomas PA, Barnes DJ, et al. 2021. MNRAS 501:3289–3308
  * Villasenor et al. (2021) Villasenor B, Robertson B, Madau P, Schneider E. 2021. submitted
  * Vogelsberger et al. (2020) Vogelsberger M, Nelson D, Pillepich A, Shen X, Marinacci F, et al. 2020\. MNRAS 492:5167–5201
  * Walter et al. (2016) Walter F, Decarli R, Aravena M, Carilli C, Bouwens R, et al. 2016. ApJ 833:67
  * Wang et al. (2020) Wang F, Davies FB, Yang J, Hennawi JF, Fan X, et al. 2020. ApJ 896:23
  * Wang et al. (2021) Wang F, Yang J, Fan X, Hennawi JF, Barth AJ, et al. 2021. ApJ 907:L1
  * Wang et al. (2019) Wang F, Yang J, Fan X, Wu XB, Yue M, et al. 2019. ApJ 884:30
  * Watson et al. (2015) Watson D, Christensen L, Knudsen KK, Richard J, Gallazzi A, Michałowski MJ. 2015. Nature 519:327–330
  * Wechsler & Tinker (2018) Wechsler RH, Tinker JL. 2018. ARA&A 56:435–487
  * Wells et al. (2015) Wells M, Pel JW, Glasse A, Wright GS, Aitink-Kroes G, et al. 2015. Publications of the Astronomical Society of the Pacific 127:646–664
  * Wilkins et al. (2017) Wilkins SM, Feng Y, Di Matteo T, Croft R, Lovell CC, Waters D. 2017\. MNRAS 469:2517–2530
  * Wilkins et al. (2016) Wilkins SM, Feng Y, Di-Matteo T, Croft R, Stanway ER, et al. 2016. MNRAS 458:L6–L9
  * Williams et al. (2018) Williams CC, Curtis-Lake E, Hainline KN, Chevallard J, Robertson BE, et al. 2018. ApJS 236:33
  * Williams et al. (2021a) Williams CC, Maseda M, Tacchella S, Alberts S, Bhatawdekar R, et al. 2021a. UDF medium band survey: Using H-alpha emission to reconstruct Ly-alpha escape during the Epoch of Reionization. JWST Proposal. Cycle 1
  * Williams et al. (2021b) Williams CC, Oesch P, Barrufet L, Bezanson R, Bowler RAA, et al. 2021b. PANORAMIC - A Pure Parallel Wide Area Legacy Imaging Survey at 1-5 Micron. JWST Proposal. Cycle 1
  * Willmer et al. (2020) Willmer CNA, Hainline KN, Curtis-Lake E, Jades Team. 2020. Guitarra, a Simulator for the JWST/NIRCam. In Uncovering Early Galaxy Evolution in the ALMA and JWST Era, eds. E da Cunha, J Hodge, J Afonso, L Pentericci, D Sobral, vol. 352
  * Willott et al. (2017a) Willott CJ, Abraham RG, Albert L, Bradac M, Brammer G, et al. 2017a. CANUCS: The CAnadian NIRISS Unbiased Cluster Survey. JWST Proposal. Cycle 1
  * Willott et al. (2017b) Willott CJ, Arribas S, Ferruit P, Jakobsen P, Maiolino R. 2017b. Cosmic reionization, metal enrichment and host galaxies from quasar spectroscopy. JWST Proposal. Cycle 1
  * Windhorst et al. (2017) Windhorst RA, Alpaslan M, Ashcraft T, Broadhurst TJ, Coe D, et al. 2017\. JWST Medium-Deep Fields - Windhorst IDS GTO Program. JWST Proposal. Cycle 1
  * Wright et al. (2015) Wright GS, Wright D, Goodson GB, Rieke GH, Aitink-Kroes G, et al. 2015\. PASP 127:595
  * Xu et al. (2016) Xu H, Wise JH, Norman ML, Ahn K, O’Shea BW. 2016. ApJ 833:84
  * Yan et al. (2011) Yan H, Yan L, Zamojski MA, Windhorst RA, McCarthy PJ, et al. 2011. ApJ 728:L22
  * Yang et al. (2020) Yang J, Wang F, Fan X, Hennawi JF, Davies FB, et al. 2020. ApJ 897:L14
  * Yang et al. (2019) Yang J, Wang F, Fan X, Yue M, Wu XB, et al. 2019. AJ 157:236
  * Yue et al. (2018) Yue B, Castellano M, Ferrara A, Fontana A, Merlin E, et al. 2018. ApJ 868:115
  * Yue et al. (2016) Yue B, Ferrara A, Xu Y. 2016. MNRAS 463:1968–1979
  * Yung et al. (2019a) Yung LYA, Somerville RS, Finkelstein SL, Popping G, Davé R. 2019a. MNRAS 483:2983–3006
  * Yung et al. (2020a) Yung LYA, Somerville RS, Finkelstein SL, Popping G, Davé R, et al. 2020a. MNRAS 496:4574–4592
  * Yung et al. (2020b) Yung LYA, Somerville RS, Popping G, Finkelstein SL. 2020b. MNRAS 494:1002–1017
  * Yung et al. (2019b) Yung LYA, Somerville RS, Popping G, Finkelstein SL, Ferguson HC, Davé R. 2019b. MNRAS 490:2855–2879
  * Zackrisson et al. (2017) Zackrisson E, Binggeli C, Finlator K, Gnedin NY, Paardekooper JP, et al. 2017. ApJ 836:78
  * Zackrisson et al. (2013) Zackrisson E, Inoue AK, Jensen H. 2013. ApJ 777:39
  * Zackrisson et al. (2011) Zackrisson E, Rydberg CE, Schaerer D, Östlin G, Tuli M. 2011. ApJ 740:13
  * Zheng et al. (2012) Zheng W, Postman M, Zitrin A, Moustakas J, Shu X, et al. 2012. Nature 489:406–408
  * Zheng et al. (2017) Zheng W, Zitrin A, Infante L, Laporte N, Huang X, et al. 2017. ApJ 836:210
  * Zhu et al. (2019) Zhu H, Avestruz C, Gnedin NY. 2019. ApJ 882:152
  * Zitrin et al. (2015) Zitrin A, Labbé I, Belli S, Bouwens R, Ellis RS, et al. 2015. ApJ 810:L12

Table 1: Schechter (1976) Luminosity Function $\phi(M_{\mathrm{UV}})$ Parameters | $\phi^{\star}$ | $M^{\star}$ |  |
---|---|---|---|---
Redshift | (Mpc-3 mag-1) | (mag) | $\alpha$ | Reference
$z\sim 6$ | $5.1^{+1.2}_{-1.0}\times 10^{-4}$ | $-20.93\pm 0.09$ | $-1.93\pm 0.08$ | Bouwens et al. (2021a)
$z\sim 7$ | $1.9^{+0.8}_{-0.6}\times 10^{-4}$ | $-21.15\pm 0.13$ | $-2.06\pm 0.11$ | Bouwens et al. (2021a)
$z\sim 8$ | $9^{+9}_{-5}\times 10^{-5}$ | $-20.93\pm 0.28$ | $-2.23\pm 0.20$ | Bouwens et al. (2021a)
$z\sim 9$ | $2.1^{+1.4}_{-0.9}\times 10^{-5}$ | $-21.15(fixed)$ | $-2.33\pm 0.19$ | Bouwens et al. (2021a)
$z\sim 10$ | $4.2^{+4.5}_{-2.2}\times 10^{-6}$ | $-21.19(fixed)$ | $-2.38\pm 0.28$ | Oesch et al. (2018)
Table 2: Double Power-Law Luminosity Function $\phi_{\mathrm{DPL}}(M_{\mathrm{UV}})$ Parameters | $\phi^{\star}_{\mathrm{DPL}}$ | $M^{\star}_{\mathrm{DPL}}$ |  |  |
---|---|---|---|---|---
Redshift | (Mpc-3 mag-1) | (mag) | $\alpha$ | $\beta$ | Reference
$z\sim 6$ | $3.02^{+0.70}_{-0.45}\times 10^{-4}$ | $-21.03^{+0.09}_{-0.03}$ | $-2.08^{+0.07}_{-0.06}$ | $-4.57^{+0.09}_{-0.10}$ | Harikane et al. (2021)
$z\sim 7$ | $8.91^{+2.31}_{-1.83}\times 10^{-4}$ | $-20.12^{+0.12}_{-0.10}$ | $-1.89^{+0.10}_{-0.09}$ | $-3.81^{+0.10}_{-0.13}$ | Harikane et al. (2021)
$z\sim 8$ | $4.83^{+2.25}_{-2.25}\times 10^{-4}$ | $-19.80\pm 0.26$ | $-1.96\pm 0.15$ | $-3.98\pm 0.14$ | Bowler et al. (2020)
$z\sim 9$ | $2.85^{+1.39}_{-1.39}\times 10^{-4}$ | $-19.67\pm 0.33$ |  $-2.1$ (fixed) | $-3.75\pm 0.22$ | Bowler et al. (2020)
Table 3: Luminosity Density $\rho_{\mathrm{UV}}$

| $\log_{10}\rho_{\mathrm{UV}}\,^{\rm a}$ | $\rho_{\mathrm{UV}}(>L^{\star})/$ | $\rho_{\mathrm{UV}}(>0.1L^{\star})/$ | $\log_{10}\rho^{\mathrm{DPL}}_{\mathrm{UV}}\,{}^{\rm a}$ | $\rho^{\mathrm{DPL}}_{\mathrm{UV}}(>L^{\star})/$ | $\rho^{\mathrm{DPL}}_{\mathrm{UV}}(>0.1L^{\star})/$
---|---|---|---|---|---|---
Redshift | $(>0.01L^{\star})$ | $\rho_{\mathrm{UV}}(>0.01L^{\star})$ | $\rho_{\mathrm{UV}}(>0.01L^{\star})$ | $(>0.01L^{\star})$ | $\rho^{\mathrm{DPL}}_{\mathrm{UV}}(>0.01L^{\star})$ | $\rho^{\mathrm{DPL}}_{\mathrm{UV}}(>0.01L^{\star})$
$z\sim 6$ | $26.25^{\rm b}$ | $0.066$ | $0.49$ | $26.27$ | $0.048$ | $0.45$
$z\sim 7$ | $26.05$ | $0.046$ | $0.42$ | $26.20$ | $0.106$ | $0.57$
$z\sim 8$ | $25.83$ | $0.027$ | $0.32$ | $25.87$ | $0.084$ | $0.52$
$z\sim 9$ | $25.41$ | $0.019$ | $0.28$ | $25.72$ | $0.068$ | $0.44$
$z\sim 10$ | $24.80$ | $0.016$ | $0.25$ | … | … | …

a(erg s-1 Hz-1 Mpc${}^{-3})$; b To convert $\rho_{UV}$ to a star formation
rate density $\dot{\rho}_{\star}$ [$M_{\odot}$ yr-1 Mpc-3], a useful
approximation is $\log_{10}\dot{\rho}_{\star}\approx\log_{10}\rho_{UV}-28.1$.
This conversion assumes a 100 Myr-old, constant star formation rate stellar
population without dust (Eldridge et al., 2017).


# Learning Large-scale Universal User Representation with Sparse Mixture of
Experts

Caigao Jiang  Siqiao Xue  James Zhang  Lingyue Liu  Zhibo Zhu  Hongyan Hao

###### Abstract

Learning user sequence behaviour embedding is very sophisticated and
challenging due to the complicated feature interactions over time and high
dimensions of user features. Recent emerging foundation models, e.g., BERT and
its variants, encourage a large body of researchers to investigate in this
field. However, unlike natural language processing (NLP) tasks, the parameters
of user behaviour model come mostly from user embedding layer, which makes
most existing works fail in training a universal user embedding of large
scale. Furthermore, user representations are learned from multiple downstream
tasks, and the past research work do not address the seesaw phenomenon. In
this paper, we propose SUPERMOE, a generic framework to obtain high quality
user representation from multiple tasks. Specifically, the user behaviour
sequences are encoded by MoE transformer, and we can thus increase the model
capacity to billions of parameters, or even to trillions of parameters. In
order to deal with seesaw phenomenon when learning across multiple tasks, we
design a new loss function with task indicators. We perform extensive offline
experiments on public datasets and online experiments on private real-world
business scenarios. Our approach achieves the best performance over state-of-
the-art models, and the results demonstrate the effectiveness of our
framework.

Machine Learning, ICML

##  1 Introduction

Recent works have demonstrated that the pre-trained model plays a critical
role on a wide range of applications, e.g., (Devlin et al., 2018; Dosovitskiy
et al., 2020; Riquelme et al., 2021; Bommasani et al., 2021b; Geng et al.,
2022; Sun et al., 2019; Qiu et al., 2020; Khan et al., 2021; Wu et al., 2020;
Xiao et al., 2021; Zeng et al., 2021). To improve the efficiency and
effectiveness of these models, many researchers attempt to exploit transformer
in order to capture chronological pattern and dynamics of user intentions
(Zeng et al., 2021; Xue et al., 2021). With the remarkable achievements of
pre-trained models, especially BERT-based models (Qiu et al., 2021), the
transformer backbone has been utilized to address user data sparsity and cold-
start problems in downstream applications (Yuan et al., 2020; Zhang et al.,
2020a). In addition, DNN-based self-supervised learning (SSL) model is
designed to improve semantic representations for highly-skewed data
distribution, with inadequate explicit user feedback in user behaviour
sequence interactions via unlabeled data (Yao et al., 2021; Shin et al., 2021;
Zhang et al., 2020b).

However, the existing pre-trained model suffers from many difficulties in
achieving good user representations, e.g., only a few behaviour channels are
used in the model due to the huge sizes of vocabularies and the resulting low
training efficiency. In AETN (Zhang et al., 2020a), only three behaviour
channels are utilized, yielding sub-optimal user representations. Therefore,
the motivations of our work are threefold, supported by our practical
observations in online production system. Firstly, most of model parameters
come from feature embedding of ID and categorical features, which usually
dominate GPU memory usage (Lian et al., 2021). For example, the number of user
IDs are often in the scale of billions, resulting in parameter size of
$\textit{numberIDs}\times\textit{embeddingDIMs}$. Secondly, the front
embedding layer accounts for the majority of the model’s size, while the rest
of model layers are extremely computationally expensive. Consequently,
training feature embedding layer and main neural networks simultaneously and
synchronously for model of large scale is challenging, which calls for
efficient model training algorithm for sparsity. Finally, there are multiple
training objectives no matter in model pre-training stage or in fine-tuning
stage, which often causes pre-trained user embedding models with sub-optimal
performance when using simple bottom-shared mechanism for the reason of seesaw
phenomenon (Tang et al., 2020) and negative transfer (Ma et al., 2018; Chen et
al., 2019).

Model | Large channels | Sequential | Temporal | Multi-task learning | Scalability up to Trillions
---|---|---|---|---|---
MTL(Tang et al., 2020) | $\surd$ | $\times$ | $\times$ | $\surd$ | $\times$
PERSIA(Lian et al., 2021) | $\surd$ | $\times$ | $\times$ | $\times$ | $\surd$
MTSSL(Yao et al., 2021) | $\surd$ | $\times$ | $\times$ | $\times$ | $\times$
BERT(Sun et al., 2019) | $\times$ | $\surd$ | $\times$ | $\surd$ | $\surd$
AETN(Zhang et al., 2020a) | $\times$ | $\surd$ | $\times$ | $\surd$ | $\times$
OURS | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
Table 1: Advantages and limitations of the proposed model and the other models

In this paper, we propose SUPERMOE, a general framework for user sequence
behaviour representation and prediction using sparse MoE transformer.
Intuitively, transformer demonstrates the importance of capturing long range
dependencies and pairwise or higher order interactions between elements
(Bommasani et al., 2021a). The sparse gating mechanism, such as MoE, has shown
its great advantages in multi-objective learning in user recommendation
systems. Therefore, embedding the gating function in transformer would be a
good alternative to conventional models in user representation learning. The
comparison of advantages and limitations of the proposed model and the other
models is listed in Table 1.

Our contributions can thus be summarized as follows: 1) We propose a sparse
MoE transformer model to deal with huge amount of user behaviour sequence data
with high dimensions. 2) We propose a novel multi-task optimization algorithm
in order to address seesaw problem and negative transfer problem across
multiple tasks. 3) We devise a novel method to split feature projection layer
in order to address the issue of GPU memory explosion, which successfully
integrates hundreds of behaviour channels into model training. 4) Our method
significantly outperforms existing user behaviour representation learning
methods.

##  2 Problem Statement

Generally, we denote a typical one-channel user behaviour sequence as
$s=[s_{1},s_{2},...,s_{i},...,s_{N}]$, where $s_{i}$ indicates the $i^{th}$
user behaviour for this channel, which has length of $N$. A multi-channel user
behaviour sequence is denoted as
$S=\\{[s^{j}_{1},s^{j}_{2},...,s^{j}_{i}...,s^{j}_{N}]\\}$, and
$[s^{j}_{1},s^{j}_{2},...,s^{j}_{i}...,s^{j}_{N}]$ is the $j^{th}$ channel of
user behaviour sequence corresponding to $M$ behaviour channels. Each instance
$S$ in each task contains a userID $u\in U$, and three types of sequence
channels, namely, category channel $S_{category}$, ID channel $S_{ID}$ and
dense channel $S_{dense}$. Therefore, given a set of $N$ tasks
$T=\\{t_{1},t_{2},...,t_{n}\\}$ with corresponding supervised label
$Y=\\{y_{1},y_{2},...,y_{n}\\}$, our goal is to learn the base user
representations across these tasks in order to apply them to downstream
applications. Following the two-stage training paradigm (Devlin et al., 2018),
we pre-train a base model firstly on the huge pre-training dataset and then
fine-tune a new model on downstream target dataset with parameters initialized
as the pre-trained model. After the training, our base representation model
should be able to produce universal representation $\mathcal{H}$ to serve all
downstream tasks.

##  3 Methodology

Figure 1: The SuperMoE framework consists of three different stages. In the
multi-channel feature projection stage, all the channel features are embedded
as dense vectors. During the pre-training stage, a series of masked channel
prediction(MCP) tasks are utilized in order to achieve general user
representations. The upper right shows the finetune stage, which freezes the
parameters of pre-training model as an initialization. The upper left depicts
a standard MoE transformer unit with dynamic routing mechanism.

###  3.1 User Embedding Pre-training Framework

Pre-training Tasks. Similar to the pre-training task in (Devlin et al., 2018),
a new user representation pre-training task is designed to cater to the
attribution of user behaviour data, i.e., masked channel prediction (MCP)
task. Slightly different from masked language modeling (MLM) task in NLP, not
all of the features are masked due to multi-channel problem in user behaviour
data which would produce too many feature vocabularies. Theoretically, in the
MCP task, some channel elements in the behaviour sequence are randomly masked
with special token $[MASK]$ at pre-training stage. Therefore, an MCP task of
one feature channel is elaborated as
$input=[s_{1},s_{2},...,[MASK]_{i}...,s_{N}]$, with $label=[MASK]_{i}$.
However, only a few channels are selected to be MCP tasks due to our belief
that the more important a user behaviour sequence is, the more likely the
sequence is selected as MCP task. In order to preserve essential information
of user behaviours, we choose user ID, location, time interval, payment tool,
product, trade amount, super position model (SPM) trace, click, and conversion
etc.

Pre-training Objectives. Formally, we denote $s_{mask}$ as the probability of
the estimated activity, and the probability $p(s_{mask};\Theta)$ is
represented by the product of the conditional distributions over the masked
sequence:

$p(s_{mask};\Theta)=\prod_{i=1}^{N}p(s_{mask}|s_{1},s_{2},...,[MASK]...,s_{N};\Theta)$
(1)

Our objective is to maximize $p(s_{mask};\Theta)$, which is equivalent to
minimizing the following loss function:

$L_{mcp}^{i}=-\frac{1}{|S_{i}|}\sum_{j\in
S_{i}}-logp(\hat{s_{j}}=s_{j})\textrm{ , }$ (2)

where $S_{i}$ is the set of positions of masked elements of the $i^{th}$ MCP
task, and $\hat{s_{j}}$ and $s_{j}$ are the predicted user behaviour and the
ground-truth behaviour, respectively. Notably, user behaviours are of very
different statistical characteristics from NLP or CV, e.g., the click and
conversion task are sequential tasks. Hence, we propose a new training
objective function:

$L_{k}(\Theta_{k})=\frac{1}{\sum_{i}{\delta_{k}^{i}}}\sum_{i}{\delta_{k}^{i}loss_{k}(\widehat{s}_{k}^{i}(\Theta_{k}),s_{k}^{i})}\textrm{
, }$ (3)

where $\delta$ is the indicator of training samples among $k$ tasks.

Pre-training Model Framework. The main architecture ingredients of pre-
training model are a stack of MoE transformers. Basically, our MoE
transformer’s backbone has a simple structure which consists of a multi-
channel feature projection (MFP) layer, a MoE multi-head self-attention (MoE-
MSA) layer and two MoE feed-forward network (MoE-FFN) layers. MFP layer takes
the following form:

$y_{mpf}=[s_{category}*w_{category},\ split(s_{ID}*w_{ID}),\
s_{dense}]\textrm{ , }$ (4)

where $[\cdot]$ means the concatenation operator of all vectors. Each MFP
layer in the encoder block is followed by a layer normalization and nonlinear
activation layer. The operator $split(\cdot)$ is a model parallel operation,
implemented by the whale framework (Xianyan Jia, 2022). Note that the
splitting of MFP layer addresses the issue of GPU memory explosion, which
successfully integrate hundreds of behaviour channels into model training. An
MoE-MSA layer takes the output of MFP $y_{m}pf$ as input, formulated as:

$y_{msa}=softmax(\frac{(qw^{q}G_{q}(q))(kw^{k}G_{k}(k))^{T}}{\sqrt{d_{k}}})(vw^{v}G_{v}(v))\textrm{
, }$ (5)

where $q,k,v$ is the output of an MFP layer, and $y_{msa}$ is output of an
MoE-MSA layer, connected by two MoE-FNN layers. Lastly, the point-wise MoE-
FNN(Fedus et al., 2021) can be formulated as:

$y_{ffn}=\sum_{e=1}^{E}G_{e}(x)\cdot\textit{FFN}_{e}(x)\textrm{ , }$ (6)

with $\textit{FFN}_{e}(x)=w_{o_{e}}\cdot\textit{Relu}(w_{i_{e}}\cdot
x)$,$G_{e}(x)=softmax(\textit{TopK}(h_{e}(x),k))$ , where $w_{o}$ and $w_{i}$
are the standard feed-forward networks with the same parameters. We choose top
1 strategy (Fedus et al., 2021) for $\textit{TopK}(\cdot)$ function. In
summary, $y_{ffn}$ is the output of backbone of an MoE transformer. Formally,
a series of MoE transformer blocks can be described as:

$y_{moe}=\textit{MoETransformer}([s_{category},s_{ID},s_{dense}])\textrm{ , }$
(7)

where
$\textit{MoETransformer}=$$\textit{MoE}_{FFN}(\textit{MoE}_{MSA}(\textit{MFP}(\cdot)))$.
The overall pre-training architecture is shown in Figure 1.

###  3.2 User Embedding Fine-tuning Framework

After pre-training, we adapt the learned user representations to specific
downstream tasks, instead of using pre-trained representations directly, which
is somehow unrelated to our defacto targets in production environment.
Therefore, we need to develop a new model to fine-tune our user behaviour
model across multiple downstream tasks with a unified framework. Assuming that
we have restored and initialized the parameters of the previous pre-trained
model, the fine-tuning model shares the parameters of the pre-trained part,
and a linear classification layer is placed on the top of the final output
without activation function. Denoting $h_{o}$ as the output of the final MoE-
transformer, we have:

$y_{i}=\textit{Tower}_{i}(\textit{MaxPooling}(h_{o}))\textrm{ , }$ (8)

and the $Tower_{i}$ is a linear classification layer of the $i^{th}$ fine-
tuning task. Note that the user representation
$\mathcal{H}=\textit{MaxPooling}(h_{o})$. The overall architecture of our
fine-tuning framework is shown in Figure 1.

###  3.3 Multi-task Training Optimization

In order to address seesaw and negative transfer problems and to improve
learning from multiple tasks, such as regression and classification, we
leverage a multi-task optimization strategy, i.e., jointly optimize across
multiple tasks, which can be applied in both pre-training stage and fine-
tuning stage. Mathematically, we get $k$ training objectives from equation
($5$), and therefore, the total loss can be formulated as:

$\begin{split}&Loss(\Theta)=\lambda_{1}*l_{1}(\widehat{s}_{1}(\Theta_{1}),s_{1})+\\\
&\lambda_{2}*l_{2}(\widehat{s}_{2}(\Theta_{2}),s)+...+\lambda_{k}*l_{k}(\widehat{s}_{k}(\Theta_{k}),s_{k})\textrm{
, }\end{split}$ (9)

where $Loss(\Theta)$ denotes the total loss and $\alpha_{k}$ is the
regularization strength of the $k^{th}$ loss. Recall that our objective is
actually to maximize Area Under Curve (AUC) score, we consider the following
bi-level optimization problem:

$Max\ AUC_{val}(\theta_{\lambda},\lambda)\ \
~{}~{}s.t.\theta_{\lambda}=\arg\min_{\Theta}Loss(\Theta,\lambda)\textrm{ , }$
(10)

where $AUC_{val}$ is the AUC score on validation dataset while training.
However, $AUC_{val}(\theta_{\lambda},\lambda)$ is non-differentiable with the
indicator function $\mathbb{I}(f(\lambda,x_{i}^{+})<f(\lambda,x_{j}^{-}))$,
and $x_{i}^{+}$ and $x_{j}^{-}$ are the positive and negative samples,
respectively. We therefore employ
$max\\{0,1-(f(\lambda,x_{i}^{+})-f(\lambda,x_{j}^{-}))\\}$ as a differentiable
convex surrogate of the above indicator function.

##  4 Experimental Methodology

In this section, we demonstrate the online and offline performance of SUPERMOE
in generating general embedding for user behaviour sequence. We evaluate our
model in four different real world test datasets, and one for public and three
for private datasets respectively.

Table 2: Overall AUC performance for different models Model | SIUPD | PAYTOOL | MCP |  FORTUNE
---|---|---|---|---
| Category1 Category2 | Category1 Category2 Category3 Category4 Category5 | subscription | CTR CVR
MMOE | 80.758 79.172 | 87.691 55.016 92.166 61.019 90.581 | 67.843 | 80.988 90.765
PLE | 81.819 79.798 | 87.762 55.269 92.803 61.267 91.924 | 68.351 | 81.719 91.751
BERT | 83.021 80.319 | 88.908 56.081 93.217 62.832 93.657 | 70.092 | 82.683 92.014
AETN | 82.828 80.774 | 89.293 55.961 93.229 62.706 92.899 | 70.055 | 82.952 91.817
OURS |  83.453 80.971 |  89.598 56.192 93.461 63.574 94.356 | 71.218 |  83.791 92.331

###  4.1 Experiment Settings

####  4.1.1 Dataset description

We evaluate the performance of our model on four different downstream
applications, i.e., SIUPD, Paytool, MCP, and Fortune. SIUPD dataset comes from
the IJCAI17 contest
111https://tianchi.aliyun.com/dataset/dataDetail?dataId=58, which contains
139,6245 users’ shopping logs on Alipay platform. Paytool is a user payment
preference dataset, which describes the behaviour of using payment tools for
online users. In MCP dataset, we use 103 channels of subscription and
redemption behaviour sequences for users. Fortune dataset includes users
”impression$\rightarrow$click” and ”click$\rightarrow$purchase” behaviours.
All these four datasets are split into training/test sets with the ratio of
$0.8$/$0.2$. The statistics of the datasets can be found in Table 3.

Table 3: Dataset Descriptions Dataset | Training | Test | Channels | AverageLength
---|---|---|---|---
SIUPD | 16M | 4M | 11 | 150
Paytool | 240M | 60M | 12 | 128
MCP | 80M | 20M | 103 | 128
Fortune | 32M | 8M | 786 | 128

####  4.1.2 Baselines

We fine-tune and evaluate our model against four other representative models:
MMOE(Ma et al., 2018), a classical multi-task recommendation model, PLE(Tang
et al., 2020), an extension of MMOE with multiple progressive extraction
layers, BERT(Devlin et al., 2018), a well-famed sequence model widely used in
large scale representation learning, especially in NLP and AETN(Zhang et al.,
2020a), a user representation learning model, which combines multi-head
attention and Denoising Autoencoder(DAE) model to generate user embeddings.

###  4.2 Offline Evaluation Results

In order to show the advantages of our model, we conduct the following
intrinsic experiments to evaluate offline and online performances.

####  4.2.1 Offline Model Performance

In this section, we present the results of offline model performance in the
downstream tasks. Table 2 summarizes the overall AUC scores of different
models across all datasets. Taking the evaluation results of SIUPD dataset as
an example, it is obvious that our model improves the baseline method MMoE by
gains of 2.7 and 1.8, respectively, in two combined tasks, for the reason that
our model utilizes more abundant chronological user behaviours to address the
behaviour sparsity issue. Moreover, we outperform the other two sequential
models with gains of 0.63 and 0.19, respectively, benefiting from of our
multi-task optimization. Similar performances can be observed in other three
datasets. It is worth mentioning that our methods all achieve the state-of-
the-art performances with significant gains.

####  4.2.2 Offline Embedding Performance

To evaluate the user embedding quality and efficiency of our model, we conduct
six different experiments for comparison, and analyze the effects of different
embedding methods, as well as different model capacities. We select the user’s
payment switching task in PAYTOOL dataset to report AUC score, Recall@85 and
Recall@50 respectively. The results are illustrated in Table 4. Notably, all
sequential embedding methods are better than PLE-only model, which
demonstrates the advantage of user embedding. Furthermore, our embedding is
more effective than other two sequential models, which takes the same model
size of 1 billion. We also investigate the performance of different model
capacities, and it can be seen in Table 4 that MoE with 20 billions parameters
performs much better than MoE with 1 billion, which generates gains of 0.67
AUC, 2.01 recall@85, and 1.39 recall@50, respectively.

Table 4: Embedding Evaluation in PAYTOOL Model | AUC Score | Recall@85 | Recall@50
---|---|---|---
PLE | 92.183 | 19.583 | 46.581
PLE+BERT | 94.067 | 28.751 | 50.673
PLE+AETN | 94.143 | 29.033 | 50.894
PLE+MoE1B | 95.721 | 30.628 | 53.766
PLE+MoE10B | 96.169 | 31.193 | 54.938
PLE+MoE20B | 96.395 | 32.640 | 55.174

####  4.2.3 Online A/B Testing

To further investigate the quality and effectiveness of our user embeddings,
we conduct two A/B testing experiments against online baseline model.
”Online1” experiment is a payment switching scenario operating on real-world
Alipay platform. In this experiment, our model brings on gains of 13.41% pv,
1.97% in conversion and 21.36% GMV. In addition, our model achieves gains of
4.95%,9.11% and 25.19%, respectively, in ”Online2” experiment, which is a fund
subscription and redemption scenario. These results are summarized in Table 5.

Table 5: Online Comparison of Different Models Scenario | Models | PV | PVCVR | GMV
---|---|---|---|---
Online1 | PLE+BERT | 0 | 0 | 0
OURS | 13.41% | 1.97% | 21.36%
Online2 | PLE+BERT | 0 | 0 | 0
OURS | 4.95% | 9.11% | 25.19%

##  5 Conclusions

In this paper, we investigated the utilization of multi-layer MoE networks as
a practical way to massively increase model capacity and to deal with seesaw
phenomenon and negative transfer problem. To complete this research, we
introduce an user behaviour representation pre-training and fine-tuning model
using sparse MoE. We have shown that it is possible to learn large scale user
embeddings, while capturing ubiquitous high order correlations using sparse
MoE, with our meticulous model architecture. Moreover, we formulated a bi-
level optimization method in order to address multi-task optimization.
Extensive empirical experiments demonstrated the overwhelming superiority of
our method on various real-world datasets comparing to other state-of-the-art
methods.

## References

  * Bommasani et al. (2021a) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021a.
  * Bommasani et al. (2021b) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021b.
  * Chen et al. (2019) Chen, Q., Zhao, H., Li, W., Huang, P., and Ou, W.  Behavior sequence transformer for e-commerce recommendation in alibaba.  In _Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data_ , pp. 1–4, 2019.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fedus et al. (2021) Fedus, W., Zoph, B., and Shazeer, N.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv preprint arXiv:2101.03961_ , 2021.
  * Geng et al. (2022) Geng, S., Liu, S., Fu, Z., Ge, Y., and Zhang, Y.  Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5).  _arXiv preprint arXiv:2203.13366_ , 2022.
  * Khan et al. (2021) Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M.  Transformers in vision: A survey.  _ACM Computing Surveys (CSUR)_ , 2021.
  * Lian et al. (2021) Lian, X., Yuan, B., Zhu, X., Wang, Y., He, Y., Wu, H., Sun, L., Lyu, H., Liu, C., Dong, X., et al.  Persia: A hybrid system scaling deep learning based recommenders up to 100 trillion parameters.  _arXiv preprint arXiv:2111.05897_ , 2021.
  * Ma et al. (2018) Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1930–1939, 2018.
  * Qiu et al. (2020) Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., and Huang, X.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ , 63(10):1872–1897, 2020.
  * Qiu et al. (2021) Qiu, Z., Wu, X., Gao, J., and Fan, W.  U-bert: Pre-training user representations for improved recommendation.  In _Proc. of the AAAI Conference on Artificial Intelligence. Menlo Park, CA, AAAI_ , pp. 1–8, 2021.
  * Riquelme et al. (2021) Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N.  Scaling vision with sparse mixture of experts.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Shin et al. (2021) Shin, K., Kwak, H., Kim, K.-M., Kim, S. Y., and Ramstrom, M. N.  Scaling law for recommendation models: Towards general-purpose user representations.  _arXiv preprint arXiv:2111.11294_ , 2021.
  * Sun et al. (2019) Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pp. 1441–1450, 2019.
  * Tang et al. (2020) Tang, H., Liu, J., Zhao, M., and Gong, X.  Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations.  In _Fourteenth ACM Conference on Recommender Systems_ , pp. 269–278, 2020.
  * Wu et al. (2020) Wu, C., Wu, F., Qi, T., Lian, J., Huang, Y., and Xie, X.  Ptum: Pre-training user model from unlabeled user behaviors via self-supervision.  _arXiv preprint arXiv:2010.01494_ , 2020.
  * Xianyan Jia (2022) Xianyan Jia, Le Jiang, A. W. W. X. Z. S. J. Z. X. L. L. C. Y. L. Z. Z. X. L. W. L.  Whale: Efficient giant model training over heterogeneous GPUs.  In _2022 USENIX Annual Technical Conference (USENIX ATC 22)_ , Carlsbad, CA, July 2022. USENIX Association.  URL https://www.usenix.org/conference/atc22/presentation/jia-xianyan.
  * Xiao et al. (2021) Xiao, C., Xie, R., Yao, Y., Liu, Z., Sun, M., Zhang, X., and Lin, L.  Uprec: User-aware pre-training for recommender systems.  _arXiv preprint arXiv:2102.10989_ , 2021.
  * Xue et al. (2021) Xue, S., Shi, X., Hao, H., Ma, L., Zhang, J., Wang, S., and Wang, S.  A graph regularized point process model for event propagation sequence.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–7, 2021.  doi: 10.1109/IJCNN52387.2021.9533830.
  * Yao et al. (2021) Yao, T., Yi, X., Cheng, D. Z., Yu, F., Chen, T., Menon, A., Hong, L., Chi, E. H., Tjoa, S., Kang, J., et al.  Self-supervised learning for large-scale item recommendations.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pp. 4321–4330, 2021.
  * Yuan et al. (2020) Yuan, F., He, X., Karatzoglou, A., and Zhang, L.  Parameter-efficient transfer from sequential behaviors for user modeling and recommendation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pp. 1469–1478, 2020\.
  * Zeng et al. (2021) Zeng, Z., Xiao, C., Yao, Y., Xie, R., Liu, Z., Lin, F., Lin, L., and Sun, M.  Knowledge transfer via pre-training for recommendation: A review and prospect.  _Frontiers in big Data_ , pp. 4, 2021.
  * Zhang et al. (2020a) Zhang, J., Bai, B., Lin, Y., Liang, J., Bai, K., and Wang, F.  General-purpose user embeddings based on mobile app usage.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 2831–2840, 2020a.
  * Zhang et al. (2020b) Zhang, S., Yin, H., Chen, T., Hung, Q. V. N., Huang, Z., and Cui, L.  Gcn-based user representation learning for unifying robust recommendation and fraudster detection.  In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_ , pp. 689–698, 2020b.