* Kastner et al. (2010) Kastner, J.H.; Buchanan, C.; Sahai, R.; Forrest, W.J.; Sargent, B.A. The Dusty Circumstellar Disks of B[e] Supergiants in the Magellanic Clouds. Astron. J. 2010, 139, 1993-2002. https://doi.org/10.1088/0004-6256/139/5/1993
  * Mehner et al. (2016) Mehner, A.; de Wit, W. J.; Groh, J. H.; Oudmaijer, R. D.; Baade, D.; Rivinius, T.; Selman, F.; Boffin, H. M. J.; Martayan, C. VLT/MUSE discovers a jet from the evolved B[e] star MWC 137. Astron. Astrophys. 2016, 585, A81. https://doi.org/10.1051/0004-6361/201527180
  * Clark et al. (2005) Clark, J.S.; Negueruela, I.; Crowther, P.A.; Goodwin, S.P. On the massive stellar population of the super star cluster Westerlund 1. Astron. Astrophys. 2005, 434, 949-969. https://doi.org/10.1051/0004-6361:20042413
  * Clark et al. (2013) Clark, J.S.; Ritchie, B.W.; Negueruela, I. The circumstellar environment and evolutionary state of the supergiant B[e] star Wd1-9. Astron. Astrophys. 2013, 560, A11. https://doi.org/10.1051/0004-6361/201321412
  * Liermann et al. (2010) Liermann, A.; Kraus, M.; Schnurr, O.; Borges Fernandes, M. The 13Carbon footprint of B[e] supergiants. Mon. Not. R. Astrono. Soc. 2010, 408, L6-L10. https://doi.org/10.1111/j.1745-3933.2010.00915.x
  * Kraus (2009) Kraus, M. The pre- versus post-main sequence evolutionary phase of B[e] stars. Constraints from 13CO band emission. Astron. Astrophys. 2009, 494, 253-262. https://doi.org/10.1051/0004-6361:200811020
  * Ekström et al. (2012) Ekström, S.; Georgy, C.; Eggenberger, P.; Meynet, G.; Mowlavi, N.; Wyttenbach, A.; Granada, A.; Decressin, T.; Hirschi, R.; Frischknecht, U.; Charbonnel, C.; Maeder, A. Grids of stellar models with rotation. I. Models from 0.8 to 120 $M_{\odot}$ at solar metallicity (Z = 0.014). Astron. Astrophys. 2012, 537, A146. https://doi.org/10.1051/0004-6361/201117751
  * Berkhuijsen et al. (1988) Berkhuijsen, E.M.; Humphreys, R.M.; Ghigo, F.D.; Zumach, W. A catalogue of the brightest stars in the field of M 31. Astron. Astrophys. Suppl. 1988, 76, 65-99.
  * Massey et al. (1986) Massey, P.; Armandroff, T.E.; Conti, P.S. Massive stars in M31. Astron. J. 1986, 92, 1303-1333. https://doi.org/10.1086/114264
  * Freedman (1984) Freedman, W.L. The Young Stellar Content of Nearby Resolved Galaxies. Ph.D. Thesis 1984, University of Toronto (Canada).
  * Humphreys et al. (1990) Humphreys, R.M.; Massey, P.; Freedman, W.L. Spectroscopy of Luminous Blue Stars in M31 and M33. Astron. J. 1990, 99, 84. https://doi.org/10.1086/115315
  * Massey et al. (2006) Massey, P.; Olsen, K.A.G.; Hodge, P.W.; Strong, S.B.; Jacoby, G.H.; Schlingman, W.; Smith, R.C. A Survey of Local Group Galaxies Currently Forming Stars. I. UBVRI Photometry of Stars in M31 and M33. Astron. J. 2006, 131, 2478-2496. https://doi.org/10.1086/503256
  * Massey et al. (2007a) Massey, P.; Olsen, K.A.G.; Hodge, P.W.; Jacoby, G.H.; McNeill, R.T.; Smith, R.C.; Strong, S.B. A Survey of Local Group Galaxies Currently Forming Stars. II. UBVRI Photometry of Stars in Seven Dwarfs and a Comparison of the Entire Sample. Astron. J. 2007, 133, 2393-2417. https://doi.org/10.1086/513319
  * Massey et al. (2007b) Massey, P.; McNeill, R.T.; Olsen, K.A.G.; Hodge, P.W.; Blaha, C.; Jacoby, G.H.; Smith, R.C.; Strong, S.B. A Survey of Local Group Galaxies Currently Forming Stars. III. A Search for Luminous Blue Variables and Other H$\alpha$ Emission-Line Stars. Astron. J. 2007, 134, 2474-2503. https://doi.org/10.1086/523658
  * Massey et al. (2000) Massey, P.; Waterhouse, E.; DeGioia-Eastwood, K. The Progenitor Masses of Wolf-Rayet Stars and Luminous Blue Variables Determined from Cluster Turnoffs. I. Results from 19 OB Associations in the Magellanic Clouds.Astron. J. 2000, 119, 2214-2241. https://doi.org/10.1086/301345
  * Clark et al. (2012) Clark, J.S.; Castro, N.; Garcia, M.; Herrero, A.; Najarro, F.; Negueruela, I.; Ritchie, B.W.; Smith, K.T. On the nature of candidate luminous blue variables in M 33. Astron. Astrophys. 2012, 541, A146. https://doi.org/10.1051/0004-6361/201118440
  * Humphreys et al. (2014) Humphreys, R.M.; Weis, K.; Davidson, K.; Bomans, D.J.; Burggraf, B. Luminous and Variable Stars in M31 and M33. II. Luminous Blue Variables, Candidate LBVs, Fe II Emission Line Stars, and Other Supergiants. Astrophys. J. 2014, 790, 48. https://doi.org/10.1088/0004-637X/790/1/48
  * Humphreys et al. (2017) Humphreys, R.M.; Gordon, M.S.; Martin, J.C.; Weis, K.; Hahn, D. Luminous and Variable Stars in M31 and M33. IV. Luminous Blue Variables, Candidate LBVs, B[e] Supergiants, and the Warm Hypergiants: How to Tell Them Apart. Astrophys. J. 2017, 836, 64. https://doi.org/10.3847/1538-4357/aa582e
  * Gvaramadze et al. (2010) Gvaramadze, V.V.; Kniazev, A.Y.; Fabrika, S. Revealing evolved massive stars with Spitzer. Mon. Not. R. Astrono. Soc. 2010, 405, 1047-1060. https://doi.org/10.1111/j.1365-2966.2010.16496.x
  * Wachter et al. (2010) Wachter, S.; Mauerhan, J.C.; Van Dyk, S.D.; Hoard, D.W.; Kafka, S.; Morris, P.W. A Hidden Population of Massive Stars with Circumstellar Shells Discovered with the Spitzer Space Telescope. Astron. J. 2010, 139, 2330-2346. https://doi.org/10.1088/0004-6256/139/6/2330
  * Aadland et al. (2018) Aadland, E.; Massey, P.; Neugent, K.F.; Drout, M.R. Shedding Light on the Isolation of Luminous Blue Variables. Astron. J. 2018, 156, 294. https://doi.org/10.3847/1538-3881/aaeb96
  * Campagnolo et al. (2018) Campagnolo, J.C.N.; Borges Fernandes, M.; Drake, N.A.; Kraus, M.; Guerrero, C.A.; Pereira, C.B. Detection of new eruptions in the Magellanic Clouds luminous blue variables R 40 and R 110. Astron. Astrophys. 2018, 613, A33. https://doi.org/10.1051/0004-6361/201731785
  * Cutri et al. (2003) Cutri, R.M.; Skrutskie, M.F.; van Dyk, S.; Beichman, C.A.; Carpenter, J.M.; Chester, T.; Cambresy, L.; Evans, T.; Fowler, J.; Gizis, J.; Howard, E.; Huchra, J.; Jarrett, T.; Kopan, E.L.; Kirkpatrick, J.D.; Light, R.M.; Marsh, K.A.; McCallon, H.; Schneider, S.; Stiening, R.; Sykes, M.; Weinberg, M.; Wheaton, W.A.; Wheelock, S.; Zacarias, N. VizieR Online Data Catalog: 2MASS All-Sky Catalog of Point Sources. VizieR On-line Data Catalog 2003, II/246.
  * Cutri, & et al. (2012) Cutri, R.M.; et al. VizieR Online Data Catalog: WISE All-Sky Data Release. VizieR On-line Data Catalog 2012, II/311.
  * Worthey, & Lee (2011) Worthey, G.; Lee, H. An Empirical UBV RI JHK Color-Temperature Calibration for Stars. Astrophys. J. Suppl. 2011, 193, 1. https://doi.org/10.1088/0067-0049/193/1/1
  * Morris et al. (1997) Morris, P.W.; Voors, R.H.M.; Lamers, H.J.G.L.M.; Eenens, Ph.R.J. Near-Infrared Spectra of LBV; Be and B[e] Stars: Does Axisymmetry Provide a Morphological Link? In Luminous Blue Variables: Massive Stars in Transition. 1997, Eds. A. Nota and H. Lamers, ASP Conference Series, 120, San Francisco: Astronomical Society of the Pacific, 20.
  * Bonanos et al. (2009) Bonanos, A.Z.; Massa, D..; Sewilo, M.; Lennon, D.J.; Panagia, N.; Smith, L.J.; Meixner, M.; Babler, B.L.; Bracker, S.; Meade, M.R.; Gordon, K.D.; Hora, J.L.; Indebetouw, R.; Whitney, B.A. Spitzer SAGE Infrared Photometry of Massive Stars in the Large Magellanic Cloud. Astron. J. 2009, 138, 1003-1021. https://doi.org/10.1088/0004-6256/138/4/1003
  * Stahl et al. (1989) Stahl, O.; Smolinski, J.; Wolf, B.; Zickgraf, F.-J. High-Dispersion Spectroscopy of the B[e] Supergiant S 111. In Physics of luminous blue variables. 1989, Eds. K. Davidson, A.F.J. Moffat, and H.J.G.L.M. Lamers. Dordrecht and Boston, MA, Kluwer Academic Publishers, Astrophysics and Space Science Library. 157, 295
  * Miroshnichenko et al. (2011) Miroshnichenko, A.S.; Manset, N.; Polcaro, F.; Rossi, C.; Zharikov, S. The B[e] phenomenon in the Milky Way and Magellanic Clouds. In Active OB stars: structure, evolution, mass loss, and critical limits. 2011, Proceedings of the International Astronomical Union, IAU Symposium, 272, 260-264. https://doi.org/10.1017/S1743921311010507
  * Evans et al. (2011) Evans, C.J.; Taylor, W.D.; Hénault-Brunet, V.; Sana, H.; de Koter, A.; Simón-Díaz, S.; Carraro, G.; Bagnoli, T.; Bastian, N.; Bestenlehner, J.M.; Bonanos, A.Z.; Bressert, E.; Brott, I.; Campbell, M.A.; Cantiello, M.; Clark, J.S.; Costa, E.; Crowther, P.A.; de Mink, S.E.; Doran, E. Dufton, P.L.; Dunstall, P.R.; Friedrich, K.; Garcia, M.; Gieles, M.; Gräfener, G.; Herrero, A.; Howarth, I.D.; Izzard, R.G.; Langer, N.; Lennon, D.J.; Maíz Apellániz, J.; Markova, N.; Najarro, F.; Puls, J.; Ramirez, O.H.; Sabín-Sanjulián, C.; Smartt, S.J.; Stroud, V.E.; van Loon, J.Th.; Vink, J.S.; Walborn, N.R. The VLT-FLAMES Tarantula Survey. I. Introduction and observational overview. Astron. Astrophys. 2011, 530, A108. https://doi.org/10.1051/0004-6361/201116782
  * Evans et al. (2015) Evans, C.J.; Kennedy, M.B.; Dufton, P.L.; Howarth, I.D.; Walborn, N.R.; Markova, N.; Clark, J.S.; de Mink, S.E.; de Koter, A.; Dunstall, P.R.; Hénault-Brunet, V.; Maíz Apellániz, J.; McEvoy, C.M.; Sana, H.; Simón-Díaz, S.; Taylor, W.D.; Vink, J.S. The VLT-FLAMES Tarantula Survey. XVIII. Classifications and radial velocities of the B-type stars. Astron. Astrophys. 2015, 574, A13. https://doi.org/10.1051/10.1051/0004-6361/201424414
  * Kalari et al. (2014) Kalari, V.M.; Vink, J.S.; Dufton, P.L.; Evans, C.J.; Dunstall, P.R.; Sana, H.; Clark, J.S.; Ellerbroek, L.; de Koter, A.; Lennon, D.J.; Taylor, W.D. The VLT-FLAMES Tarantula Survey. XV. VFTS 822: A candidate Herbig B[e] star at low metallicity. Astron. Astrophys. 2014, 564, L7. https://doi.org/10.1051/0004-6361/201323289
  * Kamath et al. (2015) Kamath, D.; Wood, P.R.; Van Winckel, H. Optically visible post-AGB stars, post-RGB stars and young stellar objects in the Large Magellanic Cloud. Mon. Not. R. Astrono. Soc. 2015, 454, 1468-1502. https://doi.org/10.1093/mnras/stv1202
  * González-Fernández et al. (2015) González-Fernández, C.; Dorda, R.; Negueruela, I.; Marco, A. A new survey of cool supergiants in the Magellanic Clouds. Astron. Astrophys. 2015, 578, A3. https://doi.org/10.1051/0004-6361/201425362
  * Hernández et al. (2004) Hernández, J.; Calvet, N.; Briceño, C.; Hartmann, L.; Berlind, P. Spectral Analysis and Classification of Herbig Ae/Be Stars. Astron. J. 2004, 127, 1682-1701. https://doi.org/10.1086/381908
  * Cardelli et al. (1989) Cardelli, Jason A.; Clayton, Geoffrey C.; Mathis, John S. The Relationship between Infrared, Optical, and Ultraviolet Extinction. Astrophys. J. 1989, 345, 245. https://doi.org/10.1086/167900
  * Whitney et al. (2008) Whitney, B.A.; Sewilo, M.; Indebetouw, R.; Robitaille, T.P.; Meixner, M.; Gordon, K.; Meade, M.R.; Babler, B.L.; Harris, J.; Hora, J.L.; Bracker, S.; Povich, M.S.; Churchwell, E.B.; Engelbracht, C.W.; For, B.-Q.; Block, M.; Misselt, K.; Vijh, U.; Leitherer, C.; Kawamura, A., Blum, R.D.; Cohen, M.; Fukui, Y.; Mizuno, A.; Mizuno, N.; Srinivasan, S.; Tielens, A.G.G.M.; Volk, K.; Bernard, J.-P.; Boulanger, F.; Frogel, J.A.; Gallagher, J.; Gorjian, V.; Kelly, D.; Latter, W.B.; Madden, S.; Kemper, F.; Mould, J.R.; Nota, A.; Oey, M.S.; Olsen, K.A.; Onishi, T.; Paladini, R.; Panagia, N.; Perez-Gonzalez, P.; Reach, W.; Shibai, H.; Sato, S.; Smith, L.J.; Staveley-Smith, L.; Ueta, T.; Van Dyk, S.; Werner, M.; Wolff, M.; Zaritsky, D. Spitzer Sage Survey of the Large Magellanic Cloud. III. Star Formation and 1000 New Candidate Young Stellar Objects. Astron. J. 2008, 136, 18-43. https://doi.org/10.1088/0004-6256/136/1/18
  * Levato et al. (2014) Levato, H.; Miroshnichenko, A.S.; Saffe, C. New objects with the B[e] phenomenon in the Large Magellanic Cloud. Astron. Astrophys. 2014, 568, A28. https://doi.org/10.1051/0004-6361/201423846
  * Dunstall et al. (2012) Dunstall, P.R.; Fraser, M.; Clark, J.S.; Crowther, P.A.; Dufton, P.L.; Evans, C.J.; Lennon, D.J.; Soszyński, I.; Taylor, W.D.; Vink, J.S. The VLT-FLAMES Tarantula Survey. V. The peculiar B[e]-like supergiant, VFTS698, in 30 Doradus. Astron. Astrophys. 2012, 542, A50. https://doi.org/10.1051/0004-6361/201218872
  * Campbell et al. (2010) Campbell, M.A.; Evans, C.J.; Mackey, A.D.; Gieles, M.; Alves, J.; Ascenso, J.; Bastian, N.; Longmore, A.J. VLT-MAD observations of the core of 30 Doradus. Mon. Not. R. Astrono. Soc. 2010, 405, 421-435. https://doi.org/10.1111/j.1365-2966.2010.16447.x
  * Evans et al. (2004) Evans, C.J.; Howarth, I.D.; Irwin, M.J.; Burnley, A.W.; Harries, T.J. A 2dF survey of the Small Magellanic Cloud. Mon. Not. R. Astrono. Soc. 2004, 353, 601-623. https://doi.org/10.1111/j.1365-2966.2004.08096.x
  * Graus et al. (2012) Graus, A.S.; Lamb, J.B.; Oey, M.S. Discovery of New, Dust-poor B[e] Supergiants in the Small Magellanic Cloud. Astrophys. J. 2012, 759, 10. https://doi.org/10.1088/0004-637X/759/1/10
  * Bonanos et al. (2010) Bonanos, A.Z.; Lennon, D.J.; Köhlinger, F.; van Loon, J.Th.; Massa, D.L.; Sewilo, M.; Evans, C.J.; Panagia, N.; Babler, B.L.; Block, M.; Bracker, S.; Engelbracht, C.W.; Gordon, K.D.; Hora, J.L.; Indebetouw, R.; Meade, M.R.; Meixner, M.; Misselt, K.A.; Robitaille, T.P.; Shiao, B. Whitney, B.A. Spitzer SAGE-SMC Infrared Photometry of Massive Stars in the Small Magellanic Cloud. Astron. J. 2010, 140, 416-429. https://doi.org/10.1088/0004-6256/140/2/416
  * Wisniewski et al. (2007) Wisniewski, J.P.; Bjorkman, K.S.; Bjorkman, J.E.; Clampin, M. Discovery of a New Dusty B[e] Star in the Small Magellanic Cloud. Astrophys. J. 2007, 670, 1331-1336. https://doi.org/10.1086/522330
  * Meyssonnier, & Azzopardi (1993) Meyssonnier, N.; Azzopardi, M. A new catalogue of H-alpha emission-line stars and small nebulae in the Small Magellanic Cloud. Astron. Astrophys. Suppl. 1993, 102, 451-593.
  * Keller et al. (1999) Keller, S.C.; Wood, P.R.; Bessell, M.S. Be stars in and around young clusters in the Magellanic Clouds. Astron. Astrophys. Suppl. 1999, 134, 489-503. https://doi.org/10.1051/aas:1999151
  * Keller et al. (2008) Keller, L.D.; Sloan, G.C.; Forrest, W.J.; Ayala, S.; D’Alessio, P.; Shah, S.; Calvet, N.; Najita, J.; Li, A.; Hartmann, L.; Sargent, B.; Watson, D.M.; Chen, C.H. PAH Emission from Herbig Ae/Be Stars. Astrophys. J. 2008, 684, 411-429. https://doi.org/10.1086/589818
  * Peeters et al. (2002) Peeters, E.; Hony, S.; Van Kerckhoven, C.; Tielens, A.G.G.M.; Allamandola, L.J.; Hudgins, D.M.; Bauschlicher, C.W. The rich 6 to 9 $\mu$m spectrum of interstellar PAHs. Astron. Astrophys. 2002, 390, 1089-1113. https://doi.org/10.1051/0004-6361:20020773
  * Ruffle et al. (2015) Ruffle, P.M.E.; Kemper, F.; Jones, O.C.; Sloan, G.C.; Kraemer, K.E.; Woods, P.M.; Boyer, M.L.; Srinivasan, S.; Antoniou, V.; Lagadec, E.; Matsuura, M.; McDonald, I.; Oliveira, J.M.; Sargent, B.A.; Sewiło, M.; Szczerba, R.; van Loon, J.Th.; Volk, K.; Zijlstra, A.A. Spitzer infrared spectrograph point source classification in the Small Magellanic Cloud. Mon. Not. R. Astrono. Soc. 2015, 451, 3504-3536. https://doi.org/10.1093/mnras/stv1106
  * Whelan et al. (2013) Whelan, D.G.; Lebouteiller, V.; Galliano, F.; Peeters, E.; Bernard-Salas, J.; Johnson, K.E.; Indebetouw, R.; Brandl, B.R. An In-depth View of the Mid-infrared Properties of Point Sources and the Diffuse ISM in the SMC Giant H II Region, N66. Astrophys. J. 2013, 771, 16. https://doi.org/10.1088/0004-637X/771/1/16
  * Kamath et al. (2014) Kamath, D.; Wood, P.R.; Van Winckel, H. Optically visible post-AGB/RGB stars and young stellar objects in the Small Magellanic Cloud: candidate selection, spectral energy distributions and spectroscopic examination. Mon. Not. R. Astrono. Soc. 2014, 439, 2211-2270. https://doi.org/10.1093/mnras/stt2033
  * Heydari-Malayeri (1990) Heydari-Malayeri, M. Discovery of a low mass B[e] supergiant in the Small magellanic Cloud. Astron. Astrophys. 1990, 234, 233.
  * Corral (1996) Corral, L.J. LBV-Type Stars in M33. Astron. J. 1996, 112, 1450. https://doi.org/10.1086/118113
  * Fabrika, & Sholukhova (1999) Fabrika, S.; Sholukhova, O. A survey of blue - H$\alpha$ objects in the galaxy M 33. Astron. Astrophys. Suppl. 1999, 140, 309-326. https://doi.org/10.1051/aas:1999425
  * Kraus et al. (2014) Kraus, M.; Cidale, L.S.; Arias, M.L.; Oksala, M.E.; Borges Fernandes, M. Discovery of the First B[e] Supergiants in M 31. Astrophys. J. Lett. 2014, 780, L10. https://doi.org/10.1088/2041-8205/780/1/L10
  * Sholukhova et al. (2015) Sholukhova, O.; Bizyaev, D.; Fabrika, S.; Sarkisyan, A.; Malanushenko, V.; Valeev, A. New luminous blue variables in the Andromeda galaxy. Mon. Not. R. Astrono. Soc. 2015, 447, 2459-2467. https://doi.org/10.1093/mnras/stu2597
  * Kourniotis et al. (2018) Kourniotis, M.; Kraus, M.; Arias, M.L.; Cidale, L.; Torres, A.F. On the evolutionary state of massive stars in transition phases in M33. Mon. Not. R. Astrono. Soc. 2018, 480, 3706-3717. https://doi.org/10.1093/mnras/sty2087
  * Sholukhova et al. (2017) Sholukhova, O.; Fabrika, S.; Valeev, A. Classification of LBV-Star Candidates in the Galaxy M31. In Stars: From Collapse to Collapse. 2017, Eds. Yu. Yu. Balega, D. O. Kudryavtsev, I. I. Romanyuk, and I. A. Yakunin. ASP Conference Series, 510, San Francisco: Astronomical Society of the Pacific, 468.
  * Humphreys et al. (2013) Humphreys, R.M.; Davidson, K.; Grammer, S.; Kneeland, N.; Martin, J.C.; Weis, K.; Burggraf, B. Luminous and Variable Stars in M31 and M33. I. The Warm Hypergiants and Post-red Supergiant Evolution. Astrophys. J. 2013, 773, 46. https://doi.org/10.1088/0004-637X/773/1/46
  * Humphreys et al. (2017) Humphreys, R.M.; Davidson, K.; Hahn, D.; Martin, J.C.; Weis, K. Luminous and Variable Stars in M31 and M33. V. The Upper HR Diagram. Astrophys. J. 2017, 844, 40. https://doi.org/10.3847/1538-4357/aa7cef
  * Gordon et al. (2016) Gordon, M.S.; Humphreys, R.M.; Jones, T.J. Luminous and Variable Stars in M31 and M33. III. The Yellow and Red Supergiants and Post-red Supergiant Evolution. Astrophys. J. 2016, 825, 50. https://doi.org/10.3847/0004-637X/825/1/50
  * Fabrika et al. (2005) Fabrika, S.; Sholukhova, O.; Becker, T.; Afanasiev, V.; Roth, M.; Sanchez, S.F. Crowded field 3D spectroscopy of LBV candidates in M 33. Astron. Astrophys. 2005, 437, 217-226. https://doi.org/10.1051/0004-6361:20035824
  * Stahl et al. (1985) Stahl, O.; Wolf, B.; de Groot, M.; Leitherer, C. Atlas of hig-dispersion spectra of peculiar emission-line stars in the Magellanic Clouds. Astron. Astrophys. Suppl. 1985, 61, 237-258.
  * Sholukhova et al. (2004) Sholukhova, O.; Fabrika, S.; Roth, M.; Becker, T. B 416 – a B[e]-SUPERGIANT in Interacting Binary? Baltic Astronomy 2004, 13, 156-158.
  * Sandage (1983) Sandage, A. The brightest stars in nearby galaxies. II. The color-magnitude diagram for the brightest red and blue stars in M101. Astron. J. 1983, 88, 1569-1578. https://doi.org/10.1086/113447
  * Sandage (1984a) Sandage, A.The brightest stars in nearby galaxies. III. The color-magnitude diagram for the brightest red and blue stars in M 81 and Holmberg IX. Astron. J. 1984, 89, 621-629. https://doi.org/10.1086/113558
  * Sandage (1984b) Sandage, A. The brightest stars in nearby galaxies. IV. The color-magnitude diagram for the brightest red and blue stars in NGC 2403. Astron. J. 1984, 89, 630-635. https://doi.org/10.1086/113559
  * Zickgraf, & Humphreys (1991) Zickgraf, F.-J.; Humphreys, R.M. A Stellar Content Survey of NGC 2403 and M81. Astron. J. 1991, 102, 113. https://doi.org/10.1086/115860
  * Grammer, & Humphreys (2013) Grammer, S.; Humphreys, R.M. The Massive Star Population in M101. I. The Identification and Spatial Distribution of the Visually Luminous Stars. Astron. J. 2013, 146, 114. https://doi.org/10.1088/0004-6256/146/5/114
  * Humphreys (1980) Humphreys, R.M. Studies of luminous stars in nearby galaxies. VII - The brightest blue stars in the spiral galaxies M101 and NGC 2403. Astrophys. J. 1980, 241, 598-601. https://doi.org/10.1086/158374
  * Humphreys, & Aaronson (1987) Humphreys, R.M.; Aaronson, M. The Visually Brightest Early-Type Supergiants in the Spiral Galaxies NGC 2403, M81, and M101. Astron. J. 1987, 94, 1156. https://doi.org/10.1086/114553
  * Sholukhova et al. (1998a) Sholukhova, O.N.; Fabrika, S.N.; Vlasyuk, V.V.; Dodonov, S.N. Spectroscopy of stars in the galaxy M 81. Astronomy Letters 1998, 24, 507-515.
  * Sholukhova et al. (1998b) Sholukhova, O.N.; Fabrika, S.N.; Vlasyuk, V.V. Spectroscopy of stars in the Galaxy NGC 2403. Astronomy Letters 1998, 24, 603-610.
  * Grammer et al. (2015) Grammer, S.H.; Humphreys, R.M.; Gerke, J. The Massive Star Population in M101. III. Spectra and Photometry of the Luminous and Variable Stars. Astron. J. 2015, 149, 152. https://doi.org/10.1088/0004-6256/149/5/152
  * Humphreys et al. (2019) Humphreys, R.M.; Stangl, S.; Gordon, M.S.; Davidson, K.; Grammer, S.H. Luminous and Variable Stars in NGC 2403 and M81. Astron. J. 2019, 157, 22. https://doi.org/10.3847/1538-3881/aaf1ac
  * Pustilnik et al. (2017) Pustilnik, S.A.; Makarova, L.N.; Perepelitsyna, Y.A.; Moiseev, A.V.; Makarov, D.I. The extremely metal-poor galaxy DDO 68: the luminous blue variable, H$\alpha$ shells and the most luminous stars. Mon. Not. R. Astrono. Soc. 2017, 465, 4985-5002. https://doi.org/10.1093/mnras/stw3049
  * Spetsieri et al. (2018) Spetsieri, Z.T.; Bonanos, A.Z.; Kourniotis, M.; Yang, M.; Lianou, S.; Bellas-Velidis, I.; Gavras, P.; Hatzidimitriou, D.; Kopsacheili, M.; Moretti, M.I.; Nota, A.; Pouliasis, E.; Sokolovsky, K.V. Massive star population of the Virgo Cluster galaxy NGC4535. Discovery of new massive variable candidates with the Hubble Space Telescope. Astron. Astrophys. 2018, 618, A185. https://doi.org/10.1051/0004-6361/201833290
  * Spetsieri et al. (2019) Spetsieri, Z.T.; Bonanos, A.Z.; Yang, M.; Kourniotis, M.; Hatzidimitriou, D. The HST Key Project galaxies NGC 1326A, NGC 1425 and NGC 4548: New variable stars and massive star population. Astron. Astrophys. 2019, in press. arXiv:1907.07140
  * Ababakr et al. (2017) Ababakr, K.M.; Oudmaijer, R.D.; Vink, J.S. A statistical spectropolarimetric study of Herbig Ae/Be stars. Mon. Not. R. Astrono. Soc. 2017, 472, 854-868. https://doi.org/10.1093/mnras/stx1891
  * Bik & Thi (2004) Bik, A.; Thi, W.F. Evidence for an inner molecular disk around massive Young Stellar Objects. Astron. Astrophys. 2004, 427, L13-L16. https://doi.org/10.1051/0004-6361:200400088
  * Blum et al. (2004) Blum, R.D.; Barbosa, C.L.; Damineli, A.; Conti, P.S.; Ridgway, S. Accretion Signatures from Massive Young Stellar Objects. Astrophys. J. 2004, 617, 1167-1176. https://doi.org/10.1086/425680
  * Ilee et al. (2013) Ilee, J.D.; Wheelwright, H.E.; Oudmaijer, R.D.; de Wit, W.J.; Maud, L.T.; Hoare, M.G.; Lumsden, S.L.; Moore, T.J.T.; Urquhart, J.S.; Mottram, J.C. CO bandhead emission of massive young stellar objects: determining disc properties. Mon. Not. R. Astrono. Soc. 2013, 429, 2960-2973. https://doi.org/10.1093/mnras/sts537
  * Ilee et al. (2014) Ilee, J.D.; Fairlamb, J.; Oudmaijer, R.D.; Mendigutía, I.; van den Ancker, M.E.; Kraus, S.; Wheelwright, H.E. Investigating the inner discs of Herbig Ae/Be stars with CO bandhead and Br$\gamma$ emission. Mon. Not. R. Astrono. Soc. 2014, 445, 3723-3736. https://doi.org/10.1093/mnras/stu1942
  * Ilee et al. (2018) Ilee, J.D.; Oudmaijer, R.D.; Wheelwright, H.E.; Pomohaci, R. Blinded by the light: on the relationship between CO first overtone emission and mass accretion rate in massive young stellar objects. Mon. Not. R. Astrono. Soc. 2018, 477, 3360-3368. https://doi.org/10.1093/mnras/sty863
  * Esteban, & Fernandez (1998) Esteban, C.; Fernandez, M. S266: a ring nebula around a Galactic B[e] supergiant? Mon. Not. R. Astrono. Soc. 1998, 298, 185-192. https://doi.org/10.1046/j.1365-8711.1998.01610.x
  * Gvaramadze, & Menten (2012) Gvaramadze, V.V.; Menten, K.M. Discovery of a parsec-scale bipolar nebula around MWC 349A. Astron. Astrophys. 2012, 541, A7. https://doi.org/10.1051/0004-6361/201218841
  * Cohen et al. (1985) Cohen, M.; Bieging, J.H.; Dreher, J.W.; Welch, W.J. The binary system MWC 349. Astrophys. J. 1985, 292, 249-256. https://doi.org/10.1086/163154
  * Marchiano et al. (2012) Marchiano, P.; Brandi, E.; Muratore, M.F.; Quiroga, C.; Ferrer, O.E.; García, L.G. The spectroscopic orbits and physical parameters of GG Carinae. Astron. Astrophys. 2012, 540, A91. https://doi.org/10.1051/0004-6361/201117715
  * Miroshnichenko et al. (2003) Miroshnichenko, A.S.; Levato, H.; Bjorkman, K.S.; Grosso, M. Properties of galactic B[e] supergiants II. HDE 327083. Astron. Astrophys. 2003, 406, 673-683. https://doi.org/10.1051/0004-6361:20030824
  * Wolf, & Stahl (1985) Wolf, B.; Stahl, O. The absorption spectrum of the Be star MWC 300. Astron. Astrophys. 1985, 148, 412-416.
  * Miroshnichenko et al. (2004) Miroshnichenko, A.S.; Levato, H.; Bjorkman, K.S.; Grosso, M.; Manset, N.; Men’shchikov, A.B.; Rudy, R.J.; Lynch, D.K.; Mazuk, S.; Venturini, C.C.; Puetter, R.C.; Perry, R.B. Properties of galactic B[e] supergiants. III. MWC 300. Astron. Astrophys. 2004, 417, 731-743. https://doi.org/10.1051/0004-6361:20034181
  * Miroshnichenko, & Corporon (1999) Miroshnichenko, A.; Corporon, P. Revealing the nature of the B[e] star MWC 342. Astron. Astrophys. 1999, 349, 126-134.
  * Millour et al. (2009) Millour, F.; Chesneau, O.; Borges Fernandes, M.; Meilland, A.; Mars, G.; Benoist, C.; Thiébaut, E.; Stee, P.; Hofmann, K.-H.; Baron, F.; Young, J.; Bendjoya, P.; Carciofi, A.; Domiciano de Souza, A.; Driebe, T.; Jankov, S.; Kervella, P.; Petrov, R.G.; Robbe-Dubois, S.; Vakili, F.; Waters, L.B.F.M.; Weigelt, G. A binary engine fuelling HD 87643’s complex circumstellar environment. Determined using AMBER/VLTI imaging. Astron. Astrophys. 2009, 507, 317-326. https://doi.org/10.1051/0004-6361/200811592
  * Plets et al. (1995) Plets, H.; Waelkens, C.; Trams, N.R. The peculiar binary supergiant 3 Puppis. Astron. Astrophys. 1995, 293, 363-370. Astronomy and Astrophysics, Vol. , p. (1995)
  * Meilland et al. (2010) Meilland, A.; Kanaan, S.; Borges Fernandes, M.; Chesneau, O.; Millour, F.; Stee, Ph.; Lopez, B. Resolving the dusty circumstellar environment of the A[e] supergiant HD 62623 with the VLTI/MIDI. Astron. Astrophys. 2010, 512, A73. https://doi.org/10.1051/0004-6361/200913640
  * de Wit et al. (2014) de Wit, W.J.; Oudmaijer, R.D.; Vink, J.S. Dusty Blue Supergiants: News from High-Angular Resolution Observations. Advances in Astronomy 2014, 2014, id.270848. https://doi.org/10.1155/2014/270848
  * Hendry et al. (2008) Hendry, M.A.; Smartt, S.J.; Skillman, E.D.; Evans, C.J.; Trundle, C.; Lennon, D.J.; Crowther, P.A.; Hunter, I. The blue supergiant Sher 25 and its intriguing hourglass nebula. Mon. Not. R. Astrono. Soc. 2008, 388, 1127-1142. https://doi.org/10.1111/j.1365-2966.2008.13347.x
  * Smith et al. (2013) Smith, N.; Arnett, W.D.; Bally, J.; Ginsburg, A.; Filippenko, A.V. The ring nebula around the blue supergiant SBW1: pre-explosion snapshot of an SN 1987A twin. Mon. Not. R. Astrono. Soc. 2013, 429, 1324-1341. https://doi.org/10.1093/mnras/sts418
  * Smith, & Tombleson (2015) Smith, N.; Tombleson, R. Luminous blue variables are antisocial: their isolation implies that they are kicked mass gainers in binary evolution. Mon. Not. R. Astrono. Soc. 2015, 447, 598-617. https://doi.org/10.1093/mnras/stu2430
  * Richardson, & Mehner (2018) Richardson, N.D.; Mehner, A. The 2018 Census of Luminous Blue Variables in the Local Group. Research Notes of the American Astronomical Society 2018, 2, 121. https://doi.org/10.3847/2515-5172/aad1f3
  * Langer, & Heger (1998) Langer, N.; Heger, A. B[e] Supergiants: what is Their Evolutionary Status? In B[e] stars. 1998, Eds. A.M. Hubert, C. Jaschek. Dordrecht ; Boston : Kluwer Academic Publishers, Astrophysics and space science library. 233, 235.
  * Pasquali et al. (2000) Pasquali, A.; Nota, A.; Langer, N.; Schulte-Ladbeck, R. E.; Clampin, M. R4 and Its Circumstellar Nebula: Evidence for a Binary Merger? Astron. J. 2000, 119, 1352-1358. https://doi.org/10.1086/301257
  * Martin, & Humphreys (2017) Martin, J.C.; Humphreys, R.M. Multi-epoch BVRI Photometry of Luminous Stars in M31 and M33. Astron. J. 2017, 154, 81. https://doi.org/10.3847/1538-3881/aa7e2e
  * Stanek et al. (1999) Stanek, K.Z.; Kaluzny, J.; Krockenberger, M.; Sasselov, D.D.; Tonry, J.L.; Mateo, M. Detached Eclipsing Binaries and Cepheids. III. Variables in the Field M31C. Astron. J. 1999, 117, 2810-2830. https://doi.org/10.1086/300899
  * Vilardell et al. (2006) Vilardell, F.; Ribas, I.; Jordi, C. Eclipsing binaries suitable for distance determination in the Andromeda galaxy. Astron. Astrophys. 2006, 459, 321-331. https://doi.org/10.1051/0004-6361:20065667
  * Bartlett et al. (2019) Bartlett, E.S.; Clark, J.S.; Negueruela, I. CI Camelopardalis: The first sgB[e]-high mass X-ray binary twenty years on: A supernova imposter in our own Galaxy? Astron. Astrophys. 2019, 622, A93. https://doi.org/10.1051/0004-6361/201834315
  * Filliatre, & Chaty (2004) Filliatre, P.; Chaty, S. The Optical/Near-Infrared Counterpart of the INTEGRAL Obscured Source IGR J16318-4848: An sgB[e] in a High-Mass X-Ray Binary? Astrophys. J. 2004, 616, 469-484. https://doi.org/10.1086/424869
  * Lau et al. (2016) Lau, R.M.; Kasliwal, M.M.; Bond, H.E.; Smith, N.; Fox, O.D.; Carlon, R.; Cody, A.M.; Contreras, C.; Dykhoff, D.; Gehrz, R.; Hsiao, E.; Jencson, J.; Khan, R.; Masci, F.; Monard, L.A.G.; Monson, A.J.; Morrell, N.; Phillips, M.; Ressler, M.E. Rising from the Ashes: Mid-infrared Re-brightening of the Impostor SN 2010da in NGC 300. Astrophys. J. 2016, 830, 142. https://doi.org/10.3847/0004-637X/830/2/142
  * Lau et al. (2017) Lau, R.M.; Heida, M.; Kasliwal, M.M.; Walton, D.J. First Detection of Mid-infrared Variability from an Ultraluminous X-Ray Source Holmberg II X-1. Astrophys. J. Lett. 2017, 838, L17. https://doi.org/10.1088/2041-8205/780/1/L10
  * Villar et al. (2016) Villar, V.A.; Berger, E.; Chornock, R.; Margutti, R.; Laskar, T.; Brown, P.J.; Blanchard, P.K.; Czekala, I.; Lunnan, R.; Reynolds, M.T. The Intermediate Luminosity Optical Transient SN 2010da: The Progenitor, Eruption, and Aftermath of a Peculiar Supergiant High-mass X-Ray Binary. Astrophys. J. 2016, 830, 11. https://doi.org/10.3847/0004-637X/830/1/11


[2em]2em1em

# Decoupling Exploration and Exploitation for Unsupervised Pre-training with
Successor Features ††thanks: This work was supported by the Australian
Research Council through Discovery Early Career Researcher Award DE200100245
and Linkage Project LP210301046.

JaeYoon Kim  Junyu Xuan  Christy Liang  Farookh Hussain

###### Abstract

Unsupervised pre-training has been on the lookout for the virtue of a value
function representation referred to as successor features (SFs), which
decouples the dynamics of the environment from the rewards. It has a
significant impact on the process of task-specific fine-tuning due to the
decomposition. However, existing approaches struggle with local optima due to
the unified intrinsic reward of exploration and exploitation without
considering the linear regression problem and the discriminator supporting a
small skill sapce. We propose a novel unsupervised pre-training model with SFs
based on a non-monolithic exploration methodology. Our approach pursues the
decomposition of exploitation and exploration of an agent built on SFs, which
requires separate agents for the respective purpose. The idea will leverage
not only the inherent characteristics of SFs such as a quick adaptation to new
tasks but also the exploratory and task-agnostic capabilities. Our suggested
model is termed Non-Monolithic unsupervised Pre-training with Successor
features (NMPS), which improves the performance of the original monolithic
exploration method of pre-training with SFs. NMPS outperforms Active Pre-
training with Successor Features (APS) in a comparative experiment.

###### Index Terms:

Unsupervised pre-training, Successor features, Non-monolithic exploration

##  I Introduction

Unsupervised pre-training leverages previously acquired behaviour without
requiring an extrinsic reward, enabling an agent to rapidly adapt to new
downstream tasks during fine-tuning. It has contributed significantly to
Computer Vision (CV) [1] and Natural Language Processing (NLP) [2], overcoming
the weak point of supervised reinforcement learning which is limited to a
single task.

As skill-based unsupervised reinforcement learning (RL) has been researched,
one of current mainstream unsupervised pre-training RLs involves an agent
maximizing the mutual information between an encoded state and a policy skill
to learn an explicit skill vector. This approach takes advantage of algorithms
such as VIC [3], DIAYN [4], and VALOR [5]. However, during fine-tuning, these
algorithms may suffer from issues related to inferior generalization and
sluggish task inference due to the inefficient interpolation relying on latent
behaviour spaces learnt during unsupervised pre-training.

To address these issues, research on the value function using Successor
Features (SFs) has been conducted to decouple the dynamics of environment from
the reward [6, 7]. Using SFs for value function approximation makes it
possible to relearn only the applicable component in case of changes to
dynamics or reward. VISR [8] takes advantage of SFs focusing on ’task
inference’ for fine-tuning after it is pre-trained. Although it has strengths
against VIC and DIAYN in terms of certain issues, it also has the weakness of
inefficient exploration.

Figure 1: An example of noise-based monolithic exploration (left) and non-
monolithic exploration (right). In the noise-based monolithic exploration,
noise and the agent’s action play roles in exploration and exploitation,
respectively. The final action, ’Action’, taken by the agent in the
environment is the result of adding the action with a noise. In the non-
monolithic exploration, the exploitation agent and the exploration agent work
for their own purposes with the help of a mode-switching controller. The mode-
switching controller considers the state of one or both agents.

APS [9], one of the competence-based algorithms mentioned in [10], utilizes
combined intrinsic reward with rewards of VISR and Active Pre-Training (APT)
[11], allowing ‘task inference’ and ‘exploration’, respectively. Even though
the mutual information exploration method of APS is considered a recent
leading method, its performance is not sufficient since its combined intrinsic
reward interferes with the original intention of the respective intrinsic
reward from VISR or APT. In addtion, the competence-based approaches such as
APS have a chronic issue supporting small skill spaces due to the requirement
of a precise discriminator [10]. The linear reward-regression problem
mentioned in [8] gets worse since these interference can ultimately cause
damage to the role of classifier or discriminator for fine-tuning the pre-
trained RL agent.

In our research, we propose a novel unsupervised pre-training model for the
agent based on SFs ensuring the decomposition of ‘task inference’ and
‘exploration’ to improve performance. In the methodology, the agent that
conducts ‘task inference’ should primarily focus on the ‘exploitation’ without
‘exploration’. To avoid a single agent that conducts both ‘exploitation’ and
‘exploration’, there should be discrete agents responsible for each task.

Since a RL agent built on noise-based monolithic exploration typically is not
suitable for our research, we focus on separating ‘exploitation’ and
‘exploration’ using a non-monolithic exploration methodology, as shown in
Fig.1. There should be several answers to build a decoupled exploration
methodology that allows an agent using SFs to switch between two modes,
exploitation and exploration, during unsupervised pre-training. The questions
that arise are: How can such a methodology be conducted for unsupervised pre-
training of SFs to switch an exploration mode? How can separate agents be
trained during the pre-training of the decoupled exploration methodology? Is
it possible for an agent using SFs to overcome a chronic issue supporting
small skill spaces resulting from the requirement of a precise discriminator?
The contributions of our research are three-folded:

  * •

The development of a decoupled exploration methodology based on non-monolithic
exploration methodology to improve the performance of a monolithic agent using
successor features during unsupervised pre-training.

  * •

An optimized training method of separate agents during the pre-training phase
of decoupled exploration methodology for the agent using successor features.

  * •

The ability taking advantage of a competence-based algorithm as a separate
exploration agent for greater flexibility and generalization of discriminator
of exploitation agent.

Table I: Key Notations.

Symbol |  Meaning
---|---
$t$ |  action step
$s$ |  current state
$s^{\prime}$ |  next state
$a$ |  The final action to an environment
$a^{exploit}_{t}$ |  The action of $\pi^{exploit}$
$a^{explor}_{t}$ |  The action of $\pi^{explor}$
z |  Encoded observation (or state)
w |  Task vector
$r^{exploitation}\;or\;r^{exploit}$ |  Intrinsic reward of APS exploitation
$r^{exploration}\;or\;r^{explor}$ |  Intrinsic reward of APS exploration
$Exploit$ |  The agent of exploitation
$Explor$ |  The agent of exploration
$\pi^{exploit}$ |  The policy of Exploit
$\pi^{explor}$ |  The policy of Explor
$\phi^{exploit}$ |  The feature of Exploit
$\phi^{explor}$ |  The feature of Explor
$Q^{\text{exploit}}$ |  The state-action value function of Exploit
$Action$ |  The action, chosen by Homeo, to an environment
$D_{promise}(t-k,t)$ |  The value promise discrepancy
$D_{promise}^{explot}$ |  The value promise discrepancy of $\pi^{exploit}$
$f_{homeo}$ |  The function of homeostasis of the policy of Exploit, $\pi^{exploit}$
$\rho$ |  The preset value of target rate, i.e. the average number of switches of the reference model, which is selected among (0.1; 0.01; 0.001; 0.0001) for pre-training
$D_{R}^{\\#}$ |  The data of a common replay buffer and its encoded state through the state encoder of $\\#$ which is Exploit or Explor

##  II PRELIMINARY KNOWLEDGE

###  II-A Successor features

[12] further extends the successor representation of [13] to a continuous
space as

$\displaystyle\begin{aligned}
r(s,a_{t},s^{\prime})&=\phi(s,a,s^{\prime})^{T}\textbf{w}\\\
Q^{\pi}(s,a)&=E^{\pi}\biggl{[}\sum^{\infty}_{i=t}\gamma^{i-t}\phi_{i+1}\;|\;S_{t}=s,\;A_{t}=a\biggl{]}^{T}\textbf{w}\\\
&=\psi^{\pi}(s,a)^{T}\textbf{w}\end{aligned}$ (1)

where w is a task vector, $r(s,a_{t},s^{\prime})$ is a reward function, s is a
state and $\phi_{t}=\phi(S_{t},A_{t},S^{{}^{\prime}}_{t})$ and
$\psi^{\pi}(s,a)$ are a successor feature of policy $\pi$. Generalized Policy
Improvement (GPI) based on SFs has also been claimed to have a transfer
reinforcement learning framework [14, 15].

Figure 2: The architecture of our proposed pre-training model, NMPS, (right)
compared with Active Pre-training with Successor Features (left)

###  II-B Non-monolithic exploration

As described in [16, 17], [18] claims the importance of ‘when’ to explore,
which is a non-monolithic exploration approach, as opposed to ‘how’ to explore
which is typically associated with monolithic exploration. As its switching
method, it adopts ‘Homeostasis’ [19], which is abbreviated as ’Homeo’, using
the difference of value function between k steps which is called the ‘value
promise discrepancy’ as

$D_{promise}(t-k,t):=\;\big{|}V(s_{t-k})-\sum_{i=0}^{k-1}\gamma^{i}R_{t-i}-\gamma^{k}V(s_{t})\big{|}$
(2)

where $V(s)$ denotes the agent’s value estimate at state s, $R$ denotes the
reward and $\gamma$ denotes a discount factor.

##  III Related work

The research in [20] makes use of SFs to transfer previous knowledge to
similar environments in simulated and real experiments. As the importance of
SFs is magnified, [21] researches visual semantic planning to predict a
sequence of actions for a desired goal state by using a deep successor
representation. [22] shows that although SFs in transfer learning promote
learning speed, there is a restriction in transferring knowledge between tasks
when an SF representation relies on the task’s optimal policy. UVFAs, which is
$V(s,g,\theta)$ using parameters $\theta$, claims the importance of value
function approximation of adjacent states as well as adjacent goals [23]. USFA
[6] combines UVFAs, SFs and GPI to support multi-task deep reinforcement
learning.

In order to overcome ’linear regression problem’, VISR takes advantage of SFs
and the behavioral mutual information (BMI) [4, 24], which constructs a reward
function as [8] to support fast task inference as follows:

$\log q(w|s)=\phi(s)^{T}\textbf{w}$ (3)

where q is a BMI discriminator.

Since APS makes use of the strong points of both VISR and APT, the intractable
conditional entropy $-H(s|z)$ can be lower bounded. For the variational lower
bound of APS, the intrinsic reward for the pre-training is the sum of
$\text{log}q(w|s)$ of VISR and $r_{t}^{APT}(s)$ of APT.

$\displaystyle\begin{split}r_{APS}(s,a,s^{\prime})&=r_{APS}^{exploitation}(s,a,s^{\prime})+r_{APS}^{exploration}(s,a,s^{\prime})\\\
&=\phi(s)^{T}w+\log\Biggl{(}1+\frac{1}{k}\sum_{h^{(j)}\in
N_{k}(h)}\big{\|}h-h^{(j)}\big{\|}_{n_{h}}^{n_{h}}\Biggr{)}\\\ &=\log
q(w|s)+r_{t}^{APT}(s)\end{split}$ (4)

where $h=\phi(s^{\prime})$ and $N_{k}(\cdot)$ denotes the k nearest neighbors.

| Reward |  | $D_{R}^{\\#}$ |  | T. f or s |  |
---|---|---|---|---|---|---|---
The variants of NMPS | Exploit | Explor |  | Exploit | Explor |  | Et | Er |  | Action
$NMPS\\_X\\_sep^{ex}$ | $r^{exploit}$ | $r^{explor}$ |  | $D_{R}^{exploit}$ | $D_{R}^{explor}$ |  | ✓ | ✓ |  | Homeo
$NMPS\\_X\\_sep^{e*}$ | $r^{explor}$ |  | $D_{R}^{exploit}$ | $D_{R}^{explor}$ |  | ✓ | x |  | Homeo
$NMPS\\_X\\_exploit^{ex}$ | $r^{explor}$ |  | $D_{R}^{exploit}$ | common |  | ✓ | ✓ |  | Homeo
$NMPS\\_X\\_exploit^{e*}$ | $r^{explor}$ |  | $D_{R}^{exploit}$ | common |  | ✓ | x |  | Homeo
$NMPS\\_X\\_explor^{ex}$ | $r^{explor}$ |  | common | $D_{R}^{explor}$ |  | ✓ | ✓ |  | Homeo
$NMPS\\_X\\_explor^{e*}$ | $r^{explor}$ |  | common | $D_{R}^{explor}$ |  | ✓ | x |  | Homeo
$NMPS\\_D\\_sep^{ex}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | ✓ |  | Homeo
$NMPS\\_D\\_sep^{e*}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | x |  | Homeo
$NMPS\\_D\\_sep^{ex\\_D}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | ✓ |  | DIAYN
$NMPS\\_D\\_sep^{e*\\_D}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | x |  | DIAYN
$NMPS\\_D\\_sep^{ex\\_D\\_A10}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | ✓ |  | DIAYN
$NMPS\\_D\\_sep^{e*\\_D\\_A10}$ | DIAYN |  | $D_{R}^{exploit}$ | $D_{R}^{DIAYN}$ |  | ✓ | x |  | DIAYN
Table II: Comparing all variants of NMPS based on 4 factors that are ‘Reward’,
‘$D_{R}^{\\#}$’, ‘The train of feature or skill (’T. f or s’ marked in the
table)’ and ‘Action’ on the model notation of
$NMPS\\_\bigl{\\{}X\;or\;D\bigr{\\}}\\_\bigl{\\{}sep,\;exploit\;or\;explor\bigr{\\}}\\_\bigl{\\{}e,\;x\;or\;*\bigr{\\}}$.

##  IV Our methodology

An agent using SFs computes $\psi^{\pi}(s,a)$ during unsupervised pre-
training. Then, during supervised fine-tuning, the agent can find a
$\textbf{w}\in R^{d}$ by solving a regression problem through
$r(s,a_{t},s^{\prime})$ of Eq. (1). Afterward, it computes based on
$Q^{\pi}(s,a)$ of Eq. (1). However, APS cannot exactly solve a regression
problem during supervised fine-tuning with $\psi^{\pi}(s,a)$ computed based on
$r_{APS}(s,a,s^{\prime})$ of Eq. (4) during unsupervised pre-training.

The discrete agents using SFs, which are Exploit and Explor, are required to
divide the united intrinsic reward of APS, which is Eq. (4), to overcome an
implicit issue that violates the linear reward-regression problem of Eq. (1)
during task-specific fine-tuning. After pre-training, Exploit is used for
fine-tuning and Explor is discarded.

Since our model utilizes discrete agents, we explore various model variants to
enhance its performance across different domains. Table II shows all evaluated
models, which are composed of 4 independent factors, to find the optimal model
in each domain. There are 4 groups: $NMPS\\_X\\_sep$, $NMPS\\_X\\_exploit$,
$NMPS\\_X\\_explor$, $NMPS\\_D\\_sep$ in NMPS model. Among them, the standard
type is $NMPS\\_X\\_sep^{ex}$.

  * •

In the ‘Exploit’ of ‘Reward’ column, the intrinsic rewards of exploitation
agent which is abbreviated as ’Exploit’ are $r^{explitation}_{APS}$ of APS. In
the ‘Explor’ of ‘Reward’ column, the intrinsic rewards of exploration agent
which is abbreviated as ’Explor’ are $r^{exploration}_{APS}$ of APS or the
reward of DIAYN with X and D of model notation, respectively. DIAYN in
‘Reward’ also means that Explor is DIAYN.

  * •

‘common’ in ‘$D_{R}^{\\#}$’ means that the active $D_{R}^{\\#}$ is used for
both agents in common. Furthermore, ‘sep(erate)’, ‘exploit’ and ‘explor’ of
model notation means the use of $D_{R}^{\\#}$ of both agents, only
$D_{R}^{exploit}$ of exploitation agent and only $D_{R}^{explor}$ of
exploration, respectively.

  * •

‘✓’ and ‘x’ in ‘T. f or s’ means that the feature or skill of Explor is
trained or not. ‘e’, ‘x’ or ‘*’ of model notation means that the feature of
Exploit or the feature or skill of Explor is trained or not.

  * •

‘Homeo’ in ‘Action’ means that the final action is determined in Exploit or
Explor depending on the decision of Homeo. ‘DIAYN’ in ‘Action’ with ‘D’ of
notation means that the only exploration agent of $NMPS\\_D\\_sep^{ex}$,
DIAYN, decides Action. $NMPS\\_D\\_sep^{\\#\\#\\_D}$ denotes that the default
dimensions of feature of Exploit and skill of Explor are 10 and 16,
respectively. ’A10’ in $NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$ denotes that all
dimensions of feature of Exploit and skill of Explor are 10.

The details of four independent factors are explained in the following
section.

###  IV-A The model with a non-monolithic exploration

We adopt the non-monolithic model used in [18] as a reference method inspiring
our model, as shown in Fig.2 for the decoupled exploration methodology that
our research seeks. In our adopted pre-training model, the decision of Action
is decided by Homeo using the value promise discrepancy of Exploit, based on
$Q^{\text{exploit}}$, as an input:

$\begin{split}D_{\text{promise}}^{\text{exploit}}&=D_{\text{promise}}^{\text{exploit}}(t-k,t)\\\
Action&=\begin{cases}a_{t}^{exploit}\sim\pi^{exploit},&\text{for
}f_{homeo}(D_{promise}^{exploit})=0\\\
a_{t}^{explor}\sim\pi^{explor},&\text{for
}f_{homeo}(D_{promise}^{exploit})\not=0\\\ \end{cases}\end{split}$ (5)

where $f_{homeo}(D_{promise})$ with the input of
$D_{promise}(t-k,t)^{exploit}$ is Homeo for the decision of exploration mode
switching in which its result value are based on Bernoulli distribution and
$D_{promise}(t-k,t)^{exploit}$ is the value promise discrepancy of
$\pi^{exploit}$.

Then, we expect that the entropy of an Explor agent using $\pi^{explor}$ is
greater than that of an Exploit agent using $\pi^{exploit}$ as follows:

$\textsc{$\textit{H}\Bigl{(}\pi^{explor}$}\Bigr{)}>\textsc{$\textit{H}\Bigl{(}\pi^{exploit}$}\Bigr{)}$
(6)

where $\textsc{$\textit{H}\Bigl{(}\cdot$}\Bigr{)}$ denotes the overall entropy
of a policy. Since Exploit using SFs is used during fine-tuning and Explor
takes a supportive role for the exploration of Exploit, $\pi^{explor}$ with
$r_{APS}^{exploration}(s,a,s^{\prime})$ in Eq. 4 is discarded when a pre-
training is finished. Only $\pi^{exploit}$ with
$r_{APS}^{exploitation}(s,a,s^{\prime})$ in Eq. 4 is used for fine-tuning.

###  IV-B The optimized training method of separate agents

In this research, our implementation utilizes an off-policy, DDPG, used as a
base algorithm of [10] for all agents. A data from $D_{R}^{\\#}$ of either a
single agent or both agents can be used to train both agents and determine
whether they will synchronize as follows:

$(...,\textbf{w},state,...)\sim
D_{R}^{exploit}\;\text{or/and}\;D_{R}^{explor}$ (7)

where (…, w, state, …) is the data loaded from $D_{R}^{\\#}$ of both or one of
two agents according to the case of ‘and’ or ‘or’ to be initialized for pre-
training and w is a task vector. In the case of ‘and’ of Eq. (7), each agent
is separately trained with its own $D_{R}^{\\#}$. Both agents regarding ‘or’
of Eq. (7) are trained with one of $D_{R}^{\\#}$ in common (see Table II for
more details).

In addition, to maximize the effect of exploration for the model, the other
variance of Explor using a feature or a skill can be considered (see ’T. f or
s’ of Table II for more details). If a feature or skill of Explor is not
trained, the feature or skill of Explor with its initial value is used for the
agent during pre-training. In the case of feature, the following result of
action-value function, $Q$, can be expected:

$\displaystyle\begin{aligned} Q^{Explor}_{\phi_{Trained}^{Explor}}\neq
Q^{Explor}_{\phi_{NonTrained}^{Explor}}\\\ \end{aligned}$ (8)

where $\phi_{Trained}^{Explor}$ and $\phi_{NonTrained}^{Explor}$ are the
trained and the nontrained feature of Explor, respectively. This relies on the
fact that Explor will be discarded when the pre-training is finished.

Algorithm 1 The main algorithm of NMPS

1:Initialize:

2: Set the value of the duration step of exploration mode Set the value of
$\rho$ in (0.1; 0.01; 0.001; 0.0001) for Homeo Set whether the feature or
skill of Explor is
or not for Train_Both_Agents

3:procedure
$\textit{Evaluate}\\_\pi^{exploit}$($D_{promise}\;\text{of}\;\pi^{exploit}$)

4: Compute the output of Homeo with $D_{promise}$ of

5: $\pi^{exploit}$

6:end procedure

7:procedure Train_Both_Agents($...,\;\textbf{w},\;state,...$)

8: Train the feature of $\pi^{exploit}$

9: Train $\pi^{exploit}$ with $r_{APS}^{exploitation}(s,a,s^{\prime})$

10: Train the feature or skill of $\pi^{explor}$ according to

11: the initialization procedure

12: Train $\pi^{explor}$ with $r_{APS}^{exploration}(s,a,s^{\prime})$

13:end procedure

14:for $t=0,\ldots,T-1$ do

15: if starting mode then

16: The output of Homeo $\leftarrow$ exploration mode

17: else

18: Compute $D_{promise}$ of $\pi^{exploit}$

19: $Evaluate\\_\pi^{exploit}(D_{promise}\;\text{of}\;\pi^{exploit})$

20: end if

21: $\textbf{w}\sim$ exploitation agent

22: if output of Homeo == exploitation mode then

23: action $\leftarrow\pi^{exploit}$

24: else

25: action $\leftarrow\pi^{explor}$

26: end if

27: if training step of $\pi^{exploit}$ and $\pi^{explor}$ then

28: $Train\\_Both\\_Agents\bigl{(}(...,\textbf{w},state,...)$

29: $\sim$ $D_{R}^{exploit}$ or/and $D_{R}^{explor}\bigr{)}$

30: end if

31: Execute action, observe reward (for evaluation) and

32: next_state

33:end for

###  IV-C The greater flexibility and generalization of discriminator

We consider that since Eq. (1) shows that a successor feature and a task
vector are a pair to produce a reward, the combination of two factors is
crucial to synchronize the two agents. In this research, the decomposition of
‘task inference’ and ‘exploration’ extends to the combination of
$\pi^{exploit}$ using SFs and $\pi^{explor}$ of the competence-based
algorithms for the purpose of improving exploration performance. The
competence-based algorithm maximizes the mutual information
$I(\textbf{z};\textbf{w})$ between the encoded observation z and skill w to
learn an explicit skill vector as follows:

$I(\textbf{z};\textbf{w})=H(\textbf{z})-H(\textbf{z};\textbf{w})=H(\textbf{w})-H(\textbf{w};\textbf{z}).$
(9)

Each competence-based algorithm has a different performance in an environment
since a competence-based algorithm has a different algorithm based on a
different decomposition of the mutual information mentioned above. As a
variant of our model, we leverage a competence-based algorithm for
exploration. This approach enhances the flexibility and generalization of the
Exploit’s discriminator, allowing it to overcome the limitations associated
with supporting small skill spaces (see $NMPS\\_D$ of Table II). In the model,
the reward of Explor is the reward of DIAYN, which is
$NMPS\\_D\\_sep^{\\#\\#}$. Action can also be the action of DIAYN instead of
the decision of Homeo, which is $NMPS\\_D\\_sep^{\\#\\#\\_D\\_\\#\\#\\#}$. We
also expect the following entropy comparison for each agent of NMPS:

$\textsc{$\textit{H}\Bigl{(}\pi^{explor}_{competence-
based\;algorithm}$}\Bigr{)}>\textsc{$\textit{H}\Bigl{(}\pi^{exploit}$}\Bigr{)}.\vspace*{-8.5mm}$
(10)

0pt

(a) Walker

0pt

(b) Jaco Arm

0pt

(c) Quadruped

Figure 3: The comparison result of fine-tuning of NMPS (the best one), APS,
DIAYN and SMM on Walker (a), Jaco Arm (b) and Quadruped (c) by using the
intrinsic reward of APS or DIAYN for Explor of NMPS.

0pt

(a) Walker

0pt

(b) Jaco Arm

0pt

(c) Quadruped

Figure 4: The comparison result of fine-tuning of all $NMPS\\_X$s of NMPS and
APS on Walker (a) and Jaco Arm (b, smoothed line) and most NMPS variants and
APS on Quadruped (c) by using the intrinsic reward of APS or DIAYN for Explor
of NMPS.

##  V Experiments

The open source code111https://github.com/rll-research/url_benchmark for pre-
training and fine-tuning of a competence-based approach evaluated in this
research is used for the comparison, which follows a state-based agent. Our
code is available on the following github222https://github.com/jangikim2/NMPS.
The hyper-parameters and Homeo used in this experiment are explained in ’Table
I’ and ’Algorithm I’ of Appendix in ’supplementary document.pdf’ on the above
github. The evaluation of this research has been conducted in three domains:
Walker, Quadruped and Jaco Arm. Three domains are based on the DeepMind
Control Suite (DMC), which is a collection of continuous control tasks.

The discrete agents of $NMPS\\_X$ of our model are based on APS implemented in
open source code. Both Exploit and Explor are implemented with $\log q(w|s)$
and $r_{t}^{APT}(s)$ of Eq. 4, respectively. That is to say, in $NMPS\\_X$,
each agent has a separate APS intrinsic reward. Meanwhile, in $NMPS\\_D$,
DIAYN in competence-based algorithm is used as an Explor instead of Explor of
$NMPS\\_X$.

NMPS is implemented on the source code with XI-intra(100, informed, $p^{*}$,
X)333The section 3.1 of [18] where ‘XI’ means an intrinsic reward, ‘100’ is an
explore duration, ‘informed’ is a trigger type, $p^{*}$ is the exploit
duration parameterized by a target rate $\rho$ and ‘X’ is the starting mode of
explore. Instead of RND used in [18], this research takes advantage of DDPG
since all reference algorithms are based on DDPG in the reference code1.
Therefore, the design of NMPS in this research hinges on off-policy.

Since NMPS is applied to only pre-training, the original code of fine-tuning
in the reference code is utilized for the fine-tuning without a modification.
After fine-tuning, Exploit of NMPS is compared with APS, DIAYN and SMM [25].
On the other hand, Explor of NMPS is discarded after pre-training.

Algorithm 1 shows the explanation of main algorithm of our non-monolithic
exploration model during the pre-training phase. During the pre-training phase
of $NMPS\\_X\\_exploit$ or $NMPS\\_X\\_explor$, the two agents, Exploit and
Explor, use a shared replay buffer that is generated by an active agent among
the discrete agents. Meanwhile, for pre-training of $NMPS\\_D$, separate
replay buffers are used for each agent.
$NMPS\\_D\\_sep^{\\#\\#\\_D\\_\\#\\#\\#}$ does not need Homeo since only
Explor is involved in Action.

The most important tuned parameter in our source code is the target rate,
$\rho$, for Homeo. The best result of each NMPS among 4 values of target rate
$\rho$, which are (0.1; 0.01; 0.001; 0.0001), has been selected. The total
number of pre-training steps in the reference code is 2M steps. We make a
starting mode during 100K steps for exploration by using only the Explor. The
snapshot of pre-training made in 1M steps is used for fine-tuning. A task
vector used in both agents is generated by Exploit.

The best results of NMPS and other models (APS, SMM and DIAYN) are compared in
Section V-A, while all NMPSs and APS are evaluated in Section V-B. In Section
V-C, four factors of NMPS are analysed.

###  V-A Using the intrinsic reward of APS and DIAYN

Fig. 3 presents a comparison of the best performance achieved by NMPS, APS,
DIAYN and SMM in all three domains. According to the results, DIAYN and SMM
perform worse than or similarly to APS in Walker and Jaco Arm domains.
However, in only Quadruped domain, DIAYN shows remarkable performance in the
early stage. Hence, in the experiment on Quadruped domain, NMPS employs DIAYN
as Explor for taking advantage of the performance of DIAYN. Overall, due to
the expected effect of aforementioned four factors of NMPS, NMPS shows the
best performance among all competence-based algorithms.

  * •

Walker, (a) of Fig. 3: $NMPS\\_X\\_exploit^{e*}$, which makes the training of
feature different from $NMPS\\_X\\_exploit^{ex}$, outperforms all other models
across all steps. The performance of APS is simialr to that of SMM. DIAYN, a
skill discovery algorithm, has the worst performance in Walker domain.

  * •

Jaco Arm, (b) of Fig. 3: The performance of $NMPS\\_X\\_exploit^{ex}$ is
better than that of APS after around 0.25M steps and is better than those of
SMM and DIAYN (the worst) over all steps. SMM and DIAYN exhibit poor
performance even when compared to APS. The skill discovery algorithm DIAYN is
also not competitive in Jaco Arm.

  * •

Quadruped, (c) of Fig. 3: Although DIAYN exhibits remarkable performance in
the early stage, $NMPS\\_D\\_sep^{ex\\_D\\_A10}$ shows the best performance
after around 0.68M steps. APS has the third best performance among 4 models.
SMM has the worst performance after around 0.6M steps in Quadruped.

In Walker, $NMPS\\_X\\_exploit^{e*}$ shows overwhelming performance across all
steps. It dramatically improves the performance of APS, which is similar to
that of DIAYN, since it has three components which are decoupling the combined
intrinsic reward of APS, not training feature of Explor and using
$D_{R}^{exploit}$ in common for two agents. The above first two components
boosts its exploration by empowering much entropy to Explor. On the other
hand, Exploit can hold the privilege of ’task inference’ for fine-tuning.
Using $D_{R}^{exploit}$ in common can help two agents to synchronize them by
using the common data.

In Jaco Arm, the performance of $NMPS\\_X\\_exploit^{ex}$ is behind in the
very early stage. The decoupled agents causes lagging behind APS during the
stage. However, after entering the steady state after 0.25M steps, it
outperforms APS. Ensuring training feature of Explor makes the stable
operation of Explore in the domain.

In Quadruped, $NMPS\\_D\\_sep^{ex\\_D\\_A10}$, a variant type, exhibits the
best performance in the Quadruped domain, even though $\pi^{explor}_{DIAYN}$
feeds all data into $\pi^{exploit}$ during its pre-training. In the early
stage of fine-tuning, its Exploit is behind against DIAYN since it has an
exploitation agent, $\pi^{exploit}$, with $r^{exploit}$ during its pre-
training. However, during the early stage, the performance of Exploit is
better than APS and SMM since the data of $\pi^{explor}_{DIAYN}$ is also in
charge of the performance of $\pi^{exploit}$. Although its performance is not
enough compared with DIAYN in the early stage, it eventually improves and
becomes better than DIAYN, whose performance starts to stabilize after around
0.68M steps. The reason is that SFs of Exploit gives rise to the exact ’task
inference’ during the steady state of DIAYN.

###  V-B The performance of all NMPSs against that of APS

$NMPS\\_X$ in Waker and Jaco Arm are compared with APS. All NMPSs in Quadruped
are compared with APS.

  * •

Walker, (a) of Fig. 4: $NMPS\\_X$ is superior to APS across all steps. The
overall performance of each variant type is as follows.

$NMPS\\_X\\_explor>NMPS\\_X\\_exploit>$

$NMPS\\_X\\_sep>APS$

  * •

Jaco Arm, (b) of Fig. 4: Since the performance difference between $NMPS\\_X$
and APS is small, the graph shows a smoothed line. However, all variants of
$NMPS\\_X$ exhibit better performance than APS after at least 0.7M steps. The
overall performance of each variant type is as follows.

$NMPS\\_X\\_exploit>NMPS\\_X\\_sep>$

$NMPS\\_X\\_explor>APS$

  * •

Quadruped, (c) of Fig. 4: Most NMPS variants are superior to APS.
$NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$ shows the best performance. The overall
performance order of each variant is as follows.

$NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}>NMPS\\_D\\_sep^{\\#\\#\\_D}>$

$NMPS\\_X\\_explor>NMPS\\_D\\_sep>$

$NMPS\\_X\\_exploit>APS>NMPS\\_X\\_sep$

In Walker, $NMPS\\_X$ has the strength against APS. Especially, in this
domain, the performance of $NMPS\\_X\\_sep$ is behind against that of other
types, $NMPS\\_X\\_explor$ and $NMPS\\_X\\_exploit$.

In Jaco Arm, even though the difference of performance between $NMPS\\_X$ and
APS is small, all $NMPS\\_X$s are superior to APS. The performance of
$NMPS\\_X\\_explor$ is behind that of other two types, $NMPS\\_X\\_exploit$
and $NMPS\\_X\\_sep$.

In Quadruped, although most $NMPS\\_X$ variants are superior to APS,
$NMPS\\_D$ improves the performance of $NMPS\\_X$ further. The difference
between the performance of $NMPS\\_D$ and that of $NMPS\\_X$ is big. In the
performance difference, the potential of $NMPS\\_D$ is remarkable for how to
use an agent with SFs. $NMPS\\_D\\_sep^{\\#\\#\\_D\\_\\#\\#\\#}$ shows high
performance among all NMPSs. Among them, the performance of
$NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$ is highest. We pay attention to the
dimension of feature and skill of $NMPS\\_D\\_sep^{\\#\\#\\_D\\_\\#\\#\\#}$.
In addition, it is noticeable that the performance of $NMPS\\_D\\_sep^{e*}$ is
better than that of $NMPS\\_D\\_sep^{\\#\\#\\_D}$. If the dimension of feature
and skill of NMPS is not the same as that of practically optimized feature and
skill such as $NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$, the performance of
$NMPS\\_D\\_sep^{\\#\\#\\_D}$ is not enough to overcome even that of
$NMPS\\_D\\_sep^{e*}$.

###  V-C The analysis of four factors of NMPS

Although NMPS using decoupled intrinsic rewards gives rise to the performance
improvement of APS, $NMPS\\_X\\_sep$ is not enough to overcome APS as shown in
Quadruped of Fig. 4.

In Walker and Jaco Arm, $NMPS\\_X\\_exploit$ verifies a potential
competitiveness across two domains. Furthermore, it is noteworthy that
synchronizing both Exploit and Explor through $NMPS\\_X\\_exploit$ and
$NMPS\\_X\\_explor$ is significant to boost the performance of normal type,
$NMPS\\_X\\_sep$.

The effect of the factor regarding ”The train of feature or skill” depends on
the domain, as the purpose of this factor introduces a significant variance
into NMPS.

Based on the performance results in Quadruped of Fig. 4, an agent with SFs can
overcome the limitation of supporting only small skill spaces of a
discriminator by leveraging the superior performance of another competence-
based algorithm in the environment.

##  VI Discussion

###  VI-A Decoupling a monolithic exploration agent with SFs

The variants of NMPS demonstrate the contribution of NMPS. Most NMPS variants
outperform APS in all environments. Decoupling exploitation and exploration in
the original monolithic algorithm with SFs improves its performance. The mode-
switching controller (Homeo in this research) for both agents impacts the rate
of communication with the environment. As a result, agents with SFs based on
NMPS show better performance than APS during fine-tuning.

###  VI-B The factors to maximize the performance of NMPS

In this research, several factors are found to influence the performance of
decoupled unsupervised pre-training. These factors include the factor of mode
switching controller, i.e. the target rate $\rho$, and four factors related to
the variants of NMPS. The Homeo controller’s target rate, $\rho$, is found to
be a crucial factor. Furthermore, V-B shows that the performance difference
among the variants of $NMPS\\_X$ and $NMPS\\_D$ are due to the four factors
related to their design. These factors result in performance differences among
the four groups of NMPS across different environments. Notably, even though
there are the agents of same type, taking advantage of $D_{R}^{\\#}$ and
${}^{\prime}T.\;f\;or\;s^{\prime}$ makes a different performance. They are
other breakthroughs for $NMPS\\_X$ and $NMPS\\_D$ to overcome their own
limitation.

###  VI-C The vulnerable point of competence-based approach

We believe that the performance of DIAYN in Quadruped domain is better than
that of APS due to the feature dimension of APS to some extent since the
feature dimension of APS, 10, is smaller than the skill dimension of DIAYN
which is 16.

Interestingly, the performance of $NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$ based on
the feature dimension of APS is better than that of
$NMPS\\_D\\_sep^{\\#\\#\\_D}$ and DIAYN. The extreme non-monolithic operation
of Explor of $NMPS\\_D\\_sep^{\\#\\#\\_D\\_\\#\\#\\#}$ has the full charge of
exploration as well as exploitation. Finally,
$NMPS\\_D\\_sep^{\\#\\#\\_D\\_A10}$ suggests a new methodology in order to
overcome the linear reward-regression issue of agent using SFs.

##  VII Conclusion

In summary, this research proposes a novel unsupervised pre-training method
with SFs, NMPS. To improve performance, it incorporates a non-monolithic
exploration methodology with greater flexibility, generalization of the
discriminator, and optimized training methods for each agent’s feature or
skill.

Further research is needed to enhance our model for an agent which is not
based on features or skills for Explor.

## References

  * [1] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, and M. Gheshlaghi Azar, “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [2] S. Ilić, E. Marrese-Taylor, J. A. Balazs, and Y. Matsuo, “Deep contextualized word representations for detecting sarcasm and irony,” _arXiv preprint arXiv:1809.09795_ , 2018.
  * [3] K. Gregor, D. J. Rezende, and D. Wierstra, “Variational intrinsic control,” _arXiv preprint arXiv:1611.07507_ , 2016.
  * [4] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning skills without a reward function,” _arXiv preprint arXiv:1802.06070_ , 2018.
  * [5] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel, “Variational option discovery algorithms,” _arXiv preprint arXiv:1807.10299_ , 2018.
  * [6] D. Borsa, A. Barreto, J. Quan, D. Mankowitz, R. Munos, H. van Hasselt, D. Silver, and T. Schaul, “Universal successor features approximators,” _arXiv preprint arXiv:1812.07626_ , 2018.
  * [7] X. Han and F. Wu, “Meta reinforcement learning with successor feature based context,” _arXiv preprint arXiv:2207.14723_ , 2022.
  * [8] S. Hansen, W. Dabney, A. Barreto, T. Van de Wiele, D. Warde-Farley, and V. Mnih, “Fast task inference with variational intrinsic successor features,” _arXiv preprint arXiv:1906.05030_ , 2019.
  * [9] H. Liu and P. Abbeel, “Aps: Active pretraining with successor features,” in _International Conference on Machine Learning_. PMLR, 2021, Conference Proceedings, pp. 6736–6747.
  * [10] M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel, “Urlb: Unsupervised reinforcement learning benchmark,” _arXiv preprint arXiv:2110.15191_ , 2021.
  * [11] H. Liu and P. Abbeel, “Behavior from the void: Unsupervised active pre-training,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 18 459–18 473, 2021.
  * [12] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver, “Successor features for transfer in reinforcement learning,” _arXiv preprint arXiv:1606.05312, 2016_ , 2016.
  * [13] P. Dayan, “Improving generalization for temporal difference learning: The successor representation,” _Neural computation_ , vol. 5, no. 4, pp. 613–624, 1993.
  * [14] A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Žídek, and R. Munos, “Transfer in deep reinforcement learning using successor features and generalised policy improvement,” _arXiv preprint arXiv:1901.10964_ , 2019.
  * [15] C. Ma, D. R. Ashley, J. Wen, and Y. Bengio, “Universal successor features for transfer reinforcement learning,” _arXiv preprint arXiv:2001.04025_ , 2020\.
  * [16] S. J. Gershman and B. G. Tzovaras, “Dopaminergic genes are associated with both directed and random exploration,” _Neuropsychologia_ , vol. 120, pp. 97–104, 2018.
  * [17] R. B. Ebitz, B. J. Sleezer, H. P. Jedema, C. W. Bradberry, and B. Y. Hayden, “Tonic exploration governs both flexibility and lapses,” _PLoS computational biology_ , vol. 15, no. 11, p. e1007475, 2019.
  * [18] M. Pislar, D. Szepesvari, G. Ostrovski, D. Borsa, and T. Schaul, “When should agents explore?” _arXiv preprint arXiv:2108.11811_ , 2021.
  * [19] G. G. Turrigiano and S. B. Nelson, “Homeostatic plasticity in the developing nervous system,” _Nature reviews neuroscience_ , vol. 5, no. 2, pp. 97–107, 2004.
  * [20] J. Zhang, J. T. Springenberg, J. Boedecker, and W. Burgard, “Deep reinforcement learning with successor features for navigation across similar environments,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, Conference Proceedings, pp. 2371–2378.
  * [21] Y. Zhu, D. Gordon, E. Kolve, D. Fox, L. Fei-Fei, A. Gupta, R. Mottaghi, and A. Farhadi, “Visual semantic planning using deep successor representations,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, Conference Proceedings, pp. 483–492.
  * [22] L. Lehnert, S. Tellex, and M. L. Littman, “Advantages and limitations of using successor features for transfer in reinforcement learning,” _arXiv preprint arXiv:1708.00102_ , 2017.
  * [23] T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value function approximators,” in _International Conference on Machine Learning_ , 2015, Conference Proceedings, pp. 1312–1320.
  * [24] D. Warde-Farley, T. Van de Wiele, T. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih, “Unsupervised control through non-parametric discriminative rewards,” _arXiv preprint arXiv:1811.11359_ , 2018.
  * [25] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, “Efficient exploration via state marginal matching,” _arXiv preprint arXiv:1906.05274_ , 2019.