| (GHz) |  | (${}^{{}^{\prime\prime}}$) | transmissiona | (K) |
|  ⋆⋆93.9798 | IRAM-30m | 26.18 | 0.7 | $2.83\times 10^{-4}$ | high
|  ⋆⋆140.9677 | IRAM-30m | 17.45 | 0.65 | $1.75\times 10^{-3}$ | high
| 234.9357 | IRAM-30m | 10.47 | 0.4 | $9.96\times 10^{-3}$ | 19.0 hr
|  ⋆⋆281.9142 | IRAM-30m | 8.73 | 0.35 | $1.44\times 10^{-2}$ | 15.3 hr
| 563.6590 | SOFIA-GREAT | 26.18 | 0.72 | $3.22\times 10^{-3}$ | high
| 610.5880 | SOFIA-GREAT | 18.2 | 0.94 | $3.06\times 10^{-3}$ | high
| 891.9210 | SOFIA-GREAT | 18.2 | 0.99 | $2.88\times 10^{-3}$ | high
| 938.7622 | SOFIA-GREAT | 18.2 | 0.98 | $2.10\times 10^{-3}$ | high
PN | 985.5877 | SOFIA-GREAT | 18.2 | 0.17 | $2.03\times 10^{-3}$ | high
| 1032.3968 | SOFIA-GREAT | 18.2 | 0.7 | $1.93\times 10^{-3}$ | high
| 1079.1886 | SOFIA-GREAT | 18.2 | 0.81 | $1.88\times 10^{-3}$ | high
| 1219.4526 | SOFIA-GREAT | 18.2 | 0.71 | $1.8\times 10^{-3}$ | high
| 1266.1675 | SOFIA-GREAT | 18.2 | 0.95 | $1.63\times 10^{-3}$ | high
| 1312.8613 | SOFIA-GREAT | 18.2 | 0.88 | $1.56\times 10^{-3}$ | high
| 1359.5331 | SOFIA-GREAT | 18.2 | 0.96 | $1.55\times 10^{-3}$ | high
| 1452.8076 | SOFIA-GREAT | 14 | 0.96 | $2.27\times 10^{-3}$ | high
| 1499.4088 | SOFIA-GREAT | 14 | 0.98 | 2.25$\times 10^{-3}$ | high
| 1824.8703 | SOFIA-GREAT | 14 | 0.93 | $1.84\times 10^{-3}$ | high
| 108.7072 | IRAM-30m | 22.63 | 0.65 | $1.13\times 10^{-5}$ | high
| 108.9984 | IRAM-30m | 22.57 | 0.65 | $5.1\times 10^{-4}$ | high
| 109.0454 | IRAM-30m | 22.56 | 0.65 | $4.59\times 10^{-4}$ | high
| 109.2062 | IRAM-30m | 22.53 | 0.64 | $2.24\times 10^{-4}$ | high
| 109.2714 | IRAM-30m | 22.51 | 0.65 | $4.89\times 10^{-5}$ | high
| 109.2812 | IRAM-30m | 22.51 | 0.64 | $6.12\times 10^{-4}$ | high
| 152.3891 | IRAM-30m | 16.14 | 0.6 | $5.01\times 10^{-3}$ | high
|  ⋆152.6570 | IRAM-30m | 16.11 | 0.6 | $5.64\times 10^{-4}$ | high
|  ⋆152.6803 | IRAM-30m | 16.11 | 0.6 | $5.15\times 10^{-4}$ | high
|  ⋆152.8555 | IRAM-30m | 16.09 | 0.61 | $2.36\times 10^{-4}$ | high
|  ⋆152.8881 | IRAM-30m | 16.09 | 0.61 | $6.73\times 10^{-4}$ | high
| 152.9532 | IRAM-30m | 16.08 | 0.61 | $2.59\times 10^{-5}$ | high
PO | 239.7043 | IRAM-30m | 10.26 | 0.01 | $1.34\times 10^{-2}$ | 10.6 hr
| 239.9490 | IRAM-30m | 10.25 | 0.02 | $6.19\times 10^{-4}$ | high
| 239.9581 | IRAM-30m | 10.25 | 0.01 | $6.02\times 10^{-4}$ | high
| 240.1411 | IRAM-30m | 10.24 | 0.01 | $1.59\times 10^{-4}$ | high
| 240.1525 | IRAM-30m | 10.24 | 0.01 | $4.62\times 10^{-4}$ | high
| 240.2683 | IRAM-30m | 10.24 | 0.02 | $7.17\times 10^{-6}$ | high
| 283.3487 | IRAM-30m | 8.68 | 0.03 | $1.51\times 10^{-2}$ | 19.8 hr
| 283.5868 | IRAM-30m | 8.67 | 0.32 | $7.97\times 10^{-4}$ | high
| 283.5932 | IRAM-30m | 8.67 | 0.32 | $7.85\times 10^{-4}$ | high
| 283.7776 | IRAM-30m | 8.67 | 0.32 | $6.68\times 10^{-4}$ | high
| 283.7854 | IRAM-30m | 8.67 | 0.33 | $9.04\times 10^{-4}$ | high
| 283.9125 | IRAM-30m | 8.66 | 0.33 | $1.04\times 10^{-5}$ | high
| 79.9033 | IRAM-30m | 30.79 | 0.63 | $4.04\times 10^{-6}$ | high
HCP | 159.8025 | IRAM-30m | 15.39 | 0.55 | $1.68\times 10^{-5}$ | high
| 279.6349 | IRAM-30m | 8.80 | 0.33 | $7.51\times 10^{-6}$ | high
| 266.9445 | IRAM-30m | 9.22 | 0.23 | $1.06\times 10^{-4}$ | high
PH3 | 533.7946 | SOFIA-GREAT | 18.2 | 0.95 | $5.53\times 10^{-6}$ | high
| 533.8153 | SOFIA-GREAT | 18.2 | 0.95 | $4.90\times 10^{-6}$ | high

Note. — a ATRAN (https://atran.arc.nasa.gov/cgi-bin/atran/atran.cgi) program
is used for the calculation of the atmospheric transmission.

b We use a frequency resolution of $1$ km/s and a signal to noise ratio $\geq$
3 with IRAM-30m.

c We use a frequency resolution of 1 km/s and a signal to noise ratio $3$ with
SOFIA.

∗Observed by Fontani et al. (2016).

∗∗ Observed by Mininni et al. (2018)

Figure 27: Spatial distribution of physical parameters: (a) H2 density, (b)
In-fall velocity, and (c) Dust temperatures for hot corino (IRAS4A) and hot
core (Sagittarius B2) are taken from Mottram et al. (2013) and Rolffs et al.
(2010) respectively.  Figure 28: The line profile of the transition of PH3,
which could be observed with the IRAM-30m toward IRAS4A is shown with the
RATRAN model. Abundance of PH3 is taken from the CMMC model of hot corino
noted in Table 6.

## References

  * Adams et al. (1990) Adams, N. G., McIntosh, B. J., & Smith, D. 1990, A&A, 232, 443
  * Agúndez et al. (2014) Agúndez, M., Cernicharo, J., Decin, L., Encrenaz, P., & Teyssier, D. 2014, ApJ, 790, L27, doi: 10.1088/2041-8205/790/2/L27
  * Agúndez et al. (2007) Agúndez, M., Cernicharo, J., & Guélin, M. 2007, ApJ, 662, L91, doi: 10.1086/519561
  * Agúndez et al. (2008) Agúndez, M., Cernicharo, J., Pardo, J. R., Guélin, M., & Phillips, T. G. 2008, A&A, 485, L33, doi: 10.1051/0004-6361:200810193
  * Altwegg et al. (2016) Altwegg, K., Balsiger, H., Bar-Nun, A., et al. 2016, Science Advances, 2, e1600285, doi: 10.1126/sciadv.1600285
  * Anicich (1993) Anicich, V. G. 1993, ApJS, 84, 215, doi: 10.1086/191752
  * Aota & Aikawa (2012) Aota, T., & Aikawa, Y. 2012, ApJ, 761, 74, doi: 10.1088/0004-637X/761/1/74
  * Bains et al. (2020) Bains, W., Petkowski, J. J., Seager, S., et al. 2020, arXiv e-prints, arXiv:2009.06499.  https://arxiv.org/abs/2009.06499
  * Becke (1993) Becke, A. D. 1993, The Journal of Chemical Physics, 98, 5648, doi: 10.1063/1.464913
  * Bergner et al. (2019) Bergner, J. B., Öberg, K. I., Walker, S., et al. 2019, ApJ, 884, L36, doi: 10.3847/2041-8213/ab48f9
  * Bernal et al. (2021) Bernal, J. J., Koelemay, L. A., & Ziurys, L. M. 2021, ApJ, 906, 55, doi: 10.3847/1538-4357/abc87b
  * Boogert et al. (2015) Boogert, A. C. A., Gerakines, P. A., & Whittet, D. C. B. 2015, ARA&A, 53, 541, doi: 10.1146/annurev-astro-082214-122348
  * Bregman et al. (1975) Bregman, J., Lester, D., & Rank, D. 1975, The Astrophysical Journal, 202, L55
  * Cancès et al. (1997) Cancès, E., Mennucci, B., & Tomasi, J. 1997, Journal of Chemical Physics, 107, 3032, doi: 10.1063/1.474659
  * Caux et al. (2011) Caux, E., Kahane, C., Castets, A., et al. 2011, A&A, 532, A23, doi: 10.1051/0004-6361/201015399
  * Chang et al. (2021) Chang, Q., Zheng, X.-L., Zhang, X., et al. 2021, Research in Astronomy and Astrophysics, 21, 039, doi: 10.1088/1674-4527/21/2/39
  * Chantzos et al. (2020) Chantzos, J., Rivilla, V. M., Vasyunin, A., et al. 2020, A&A, 633, A54, doi: 10.1051/0004-6361/201936531
  * Charnley & Millar (1994) Charnley, S. B., & Millar, T. J. 1994, Monthly Notices of the Royal Astronomical Society, 270, 570, doi: 10.1093/mnras/270.3.570
  * Cheung et al. (1968) Cheung, A. C., Rank, D. M., Townes, C. H., Thornton, D. D., & Welch, W. J. 1968, Phys. Rev. Lett., 21, 1701, doi: 10.1103/PhysRevLett.21.1701
  * Das et al. (2015) Das, A., Majumdar, L., Sahu, D., et al. 2015, ApJ, 808, 21, doi: 10.1088/0004-637X/808/1/21
  * Das et al. (2018) Das, A., Sil, M., Gorai, P., Chakrabarti, S. i. K., & Loison, J. C. 2018, ApJS, 237, 9, doi: 10.3847/1538-4365/aac886
  * De Beck et al. (2013) De Beck, E., Kamiński, T., Patel, N. A., et al. 2013, A&A, 558, A132, doi: 10.1051/0004-6361/201321349
  * Draine (1978) Draine, B. T. 1978, ApJS, 36, 595, doi: 10.1086/190513
  * Dubernet et al. (2013) Dubernet, M. L., Alexander, M. H., Ba, Y. A., et al. 2013, A&A, 553, A50, doi: 10.1051/0004-6361/201220630
  * Dunning (1989) Dunning, Thom H., J. 1989, J. Chem. Phys., 90, 1007, doi: 10.1063/1.456153
  * Ferland et al. (2017) Ferland, G. J., Chatzikos, M., Guzmán, F., et al. 2017, Rev. Mexicana Astron. Astrofis., 53, 385\.  https://arxiv.org/abs/1705.10877
  * Ferrero et al. (2020) Ferrero, S., Zamirri, L., Ceccarelli, C., et al. 2020, ApJ, 904, 11, doi: 10.3847/1538-4357/abb953
  * Fletcher et al. (2009) Fletcher, L. N., Orton, G. S., Teanby, N. A., & Irwin, P. G. J. 2009, Icarus, 202, 543, doi: 10.1016/j.icarus.2009.03.023
  * Fontani et al. (2016) Fontani, F., Rivilla, V. M., Caselli, P., Vasyunin, A., & Palau, A. 2016, ApJ, 822, L30, doi: 10.3847/2041-8205/822/2/L30
  * Fontani et al. (2019) Fontani, F., Rivilla, V. M., van der Tak, F. F. S., et al. 2019, MNRAS, 489, 4530, doi: 10.1093/mnras/stz2446
  * Frisch et al. (2013) Frisch, M. J., Trucks, G. W., Schlegel, H. B., et al. 2013, Gaussian 09 Revision D.01
  * Gorai et al. (2020) Gorai, P., Bhat, B., Sil, M., et al. 2020, ApJ, 895, 86, doi: 10.3847/1538-4357/ab8871
  * Gorai et al. (2017a) Gorai, P., Das, A., Das, A., et al. 2017a, ApJ, 836, 70, doi: 10.3847/1538-4357/836/1/70
  * Gorai et al. (2017b) Gorai, P., Das, A., Majumdar, L., et al. 2017b, Molecular Astrophysics, 6, 36, doi: 10.1016/j.molap.2017.01.004
  * Gorai et al. (2020) Gorai, P., Sil, M., Das, A., et al. 2020, ACS Earth and Space Chemistry, 4, 920–946, doi: 10.1021/acsearthspacechem.0c00098
  * Greaves et al. (2020) Greaves, J. S., Richards, A. M. S., Bains, W., et al. 2020, Nature Astronomy, doi: 10.1038/s41550-020-1174-4
  * Guelin et al. (1990) Guelin, M., Cernicharo, J., Paubert, G., & Turner, B. E. 1990, A&A, 230, L9
  * Halfen et al. (2008) Halfen, D. T., Clouthier, D. J., & Ziurys, L. M. 2008, ApJ, 677, L101, doi: 10.1086/588024
  * Hincelin et al. (2015) Hincelin, U., Chang, Q., & Herbst, E. 2015, A&A, 574, A24, doi: 10.1051/0004-6361/201424807
  * Hogerheijde & van der Tak (2000) Hogerheijde, M. R., & van der Tak, F. F. S. 2000, A&A, 362, 697.  https://arxiv.org/abs/astro-ph/0008169
  * Jarosewich (1990) Jarosewich, E. 1990, Meteoritics, 25, 323, doi: 10.1111/j.1945-5100.1990.tb00717.x
  * Jenkins (2009) Jenkins, E. B. 2009, ApJ, 700, 1299, doi: 10.1088/0004-637X/700/2/1299
  * Jiménez-Serra et al. (2018) Jiménez-Serra, I., Viti, S., Quénard, D., & Holdship, J. 2018, The Astrophysical Journal, 862, 128, doi: 10.3847/1538-4357/aacdf2
  * Jiménez-Serra et al. (2016) Jiménez-Serra, I., Vasyunin, A. I., Caselli, P., et al. 2016, ApJ, 830, L6, doi: 10.3847/2041-8205/830/1/L6
  * Jura & York (1978) Jura, M., & York, D. G. 1978, ApJ, 219, 861, doi: 10.1086/155847
  * Kaye & Strobel (1983) Kaye, J. A., & Strobel, D. F. 1983, Geophys. Res. Lett., 10, 957, doi: 10.1029/GL010i010p00957
  * Keene et al. (1983) Keene, J., Blake, G. A., & Phillips, T. G. 1983, ApJ, 271, L27, doi: 10.1086/184086
  * Lefloch et al. (2016) Lefloch, B., Vastel, C., Viti, S., et al. 2016, MNRAS, 462, 3937, doi: 10.1093/mnras/stw1918
  * Lodders (2003) Lodders, K. 2003, ApJ, 591, 1220, doi: 10.1086/375492
  * Lord (1992) Lord, S. D. 1992, A new software tool for computing Earth’s atmospheric transmission of near-and far-infrared radiation, Vol. 103957 (Ames Research Center)
  * Maciá (2005) Maciá, E. 2005, Chemical Society Reviews, 34, 691
  * Maciá et al. (1997) Maciá, E., Hernández, M., & Oró, J. 1997, Origins of Life and Evolution of the Biosphere, 27, 459
  * Mathis et al. (1977) Mathis, J. S., Rumpl, W., & Nordsieck, K. H. 1977, ApJ, 217, 425, doi: 10.1086/155591
  * Mauersberger et al. (1988) Mauersberger, R., Henkel, C., & Wilson, T. L. 1988, A&A, 205, 235
  * McElroy et al. (2013) McElroy, D., Walsh, C., Markwick, A. J., et al. 2013, A&A, 550, A36, doi: 10.1051/0004-6361/201220465
  * Milam et al. (2008) Milam, S. N., Halfen, D. T., Tenenbaum, E. D., et al. 2008, ApJ, 684, 618, doi: 10.1086/589135
  * Millar (1991) Millar, T. J. 1991, A&A, 242, 241
  * Millar et al. (1987) Millar, T. J., Bennett, A., & Herbst, E. 1987, Monthly Notices of the Royal Astronomical Society, 229, 41P, doi: 10.1093/mnras/229.1.41P
  * Mininni et al. (2018) Mininni, C., Fontani, F., Rivilla, V. M., et al. 2018, MNRAS, 476, L39, doi: 10.1093/mnrasl/sly026
  * Mottram et al. (2013) Mottram, J. C., van Dishoeck, E. F., Schmalzl, M., et al. 2013, A&A, 558, A126, doi: 10.1051/0004-6361/201321828
  * Nguyen et al. (2020) Nguyen, T., Oba, Y., Shimonishi, T., Kouchi, A., & Watanabe, N. 2020, ApJ, 898, L52, doi: 10.3847/2041-8213/aba695
  * Noll & Marley (1997) Noll, K. S., & Marley, M. S. 1997, in Astronomical Society of the Pacific Conference Series, Vol. 119, Planets Beyond the Solar System and the Next Generation of Space Missions, ed. D. Soderblom, 115
  * Öberg et al. (2007) Öberg, K. I., Fuchs, G. W., Awad, Z., et al. 2007, ApJ, 662, L23, doi: 10.1086/519281
  * Ossenkopf & Henning (1994) Ossenkopf, V., & Henning, T. 1994, A&A, 291, 943
  * Pasek (2019) Pasek, M. A. 2019, Icarus, 317, 59, doi: 10.1016/j.icarus.2018.07.011
  * Puzzarini et al. (2014) Puzzarini, C., Biczysko, M., Bloino, J., & Barone, V. 2014, ApJ, 785, 107, doi: 10.1088/0004-637X/785/2/107
  * Reid et al. (2009) Reid, M. J., Menten, K. M., Zheng, X. W., Brunthaler, A., & Xu, Y. 2009, The Astrophysical Journal, 705, 1548, doi: 10.1088/0004-637x/705/2/1548
  * Rivilla et al. (2016) Rivilla, V. M., Fontani, F., Beltrán, M. T., et al. 2016, ApJ, 826, 161, doi: 10.3847/0004-637X/826/2/161
  * Rivilla et al. (2018) Rivilla, V. M., Jiménez-Serra, I., Zeng, S., et al. 2018, MNRAS, 475, L30, doi: 10.1093/mnrasl/slx208
  * Rivilla et al. (2020) Rivilla, V. M., Drozdovskaya, M. N., Altwegg, K., et al. 2020, MNRAS, 492, 1180, doi: 10.1093/mnras/stz3336
  * Rolffs et al. (2010) Rolffs, R., Schilke, P., Comito, C., et al. 2010, A&A, 521, L46, doi: 10.1051/0004-6361/201015106
  * Röllig et al. (2007) Röllig, M., Abel, N. P., Bell, T., et al. 2007, A&A, 467, 187, doi: 10.1051/0004-6361:20065918
  * Schöier et al. (2005) Schöier, F. L., van der Tak, F. F. S., van Dishoeck, E. F., & Black, J. H. 2005, A&A, 432, 369, doi: 10.1051/0004-6361:20041729
  * Shimonishi et al. (2020) Shimonishi, T., Das, A., Sakai, N., et al. 2020, ApJ, 891, 164, doi: 10.3847/1538-4357/ab6e6b
  * Shirley (2015) Shirley, Y. L. 2015, Publications of the Astronomical Society of the Pacific, 127, 299–310, doi: 10.1086/680342
  * Sil et al. (2018) Sil, M., Gorai, P., Das, A., et al. 2018, ApJ, 853, 139, doi: 10.3847/1538-4357/aa984d
  * Smith et al. (1989) Smith, D., McIntosh, B. J., & Adams, N. G. 1989, J. Chem. Phys., 90, 6213, doi: 10.1063/1.456337
  * Smith et al. (2004) Smith, I. W. M., Herbst, E., & Chang, Q. 2004, Monthly Notices of the Royal Astronomical Society, 350, 323, doi: 10.1111/j.1365-2966.2004.07656.x
  * Snellen et al. (2020) Snellen, I. A. G., Guzman-Ramirez, L., Hogerheijde, M. R., Hygate, A. P. S., & van der Tak, F. F. S. 2020, Re-analysis of the 267-GHz ALMA observations of Venus: No statistically significant detection of phosphine.  https://arxiv.org/abs/2010.09761
  * Sousa-Silva et al. (2020) Sousa-Silva, C., Seager, S., Ranjan, S., et al. 2020, Astrobiology, 20, 235, doi: 10.1089/ast.2018.1954
  * Tarrago et al. (1992) Tarrago, G., Lacome, N., Lévy, A., et al. 1992, Journal of Molecular Spectroscopy, 154, 30, doi: 10.1016/0022-2852(92)90026-K
  * Tenenbaum et al. (2007) Tenenbaum, E. D., Woolf, N. J., & Ziurys, L. M. 2007, ApJ, 666, L29, doi: 10.1086/521361
  * Tenenbaum & Ziurys (2008) Tenenbaum, E. D., & Ziurys, L. M. 2008, The Astrophysical Journal, 680, L121, doi: 10.1086/589973
  * Thorne et al. (1983) Thorne, L. R., Anicich, V. G., & Huntress, W. T. 1983, Chemical Physics Letters, 98, 162, doi: 10.1016/0009-2614(83)87120-6
  * Thorne et al. (1984) Thorne, L. R., Anicich, V. G., Prasad, S. S., & Huntress, W. T., J. 1984, ApJ, 280, 139, doi: 10.1086/161977
  * Tomasi et al. (2005) Tomasi, J., Mennucci, B., & Cammi, R. 2005, Chemical Reviews, 105, 2999, doi: 10.1021/cr9904009
  * Turner et al. (2015) Turner, A. M., Abplanalp, M. J., Chen, S. Y., et al. 2015, Phys. Chem. Chem. Phys., 17, 27281, doi: 10.1039/C5CP02835C
  * Turner & Bally (1987) Turner, B. E., & Bally, J. 1987, ApJ, 321, L75, doi: 10.1086/185009
  * Turner et al. (1990) Turner, B. E., Tsuji, T., Bally, J., Guelin, M., & Cernicharo, J. 1990, ApJ, 365, 569, doi: 10.1086/169511
  * van der Tak et al. (2007) van der Tak, F. F. S., Black, J. H., Schöier, F. L., Jansen, D. J., & van Dishoeck, E. F. 2007, A&A, 468, 627, doi: 10.1051/0004-6361:20066820
  * Villanueva et al. (2020) Villanueva, G., Cordiner, M., Irwin, P., et al. 2020, No phosphine in the atmosphere of Venus.  https://arxiv.org/abs/2010.14305
  * Visscher et al. (2006) Visscher, C., Lodders, K., & Fegley, Bruce, J. 2006, ApJ, 648, 1181, doi: 10.1086/506245
  * Wakelam & Herbst (2008) Wakelam, V., & Herbst, E. 2008, ApJ, 680, 371, doi: 10.1086/587734
  * Wakelam et al. (2015) Wakelam, V., Loison, J. C., Herbst, E., et al. 2015, ApJS, 217, 20, doi: 10.1088/0067-0049/217/2/20
  * Wenger et al. (2000) Wenger, M., Ochsenbein, F., Egret, D., et al. 2000, A&AS, 143, 9, doi: 10.1051/aas:2000332
  * Wilson & Pauls (1979) Wilson, T. L., & Pauls, T. 1979, A&A, 73, L10
  * Yamaguchi et al. (2011) Yamaguchi, T., Takano, S., Sakai, N., et al. 2011, PASJ, 63, L37, doi: 10.1093/pasj/63.5.L37
  * Ziurys (1987) Ziurys, L. M. 1987, ApJ, 321, L81, doi: 10.1086/185010
  * Ziurys et al. (2018) Ziurys, L. M., Schmidt, D. R., & Bernal, J. J. 2018, ApJ, 856, 169, doi: 10.3847/1538-4357/aaafc6


# Neural Task Representations as Weak Supervision for Model Agnostic Cross-
Lingual Transfer

Sujay Kumar Jauhar
Microsoft Research AI
Redmond, WA, USA
<EMAIL_ADDRESS>
&Michael Gamon
Microsoft Research AI
Redmond, WA, USA
<EMAIL_ADDRESS>
&Patrick Pantel
Facebook Inc.
Seattle, WA, USA
<EMAIL_ADDRESS>This work was done while the third author was at Microsoft
Research AI.

###### Abstract

Natural language processing is heavily Anglo-centric, while the demand for
models that work in languages other than English is greater than ever. Yet,
the task of transferring a model from one language to another can be expensive
in terms of annotation costs, engineering time and effort. In this paper, we
present a general framework for easily and effectively transferring neural
models from English to other languages. The framework, which relies on task
representations as a form of weak supervision, is model and task agnostic,
meaning that many existing neural architectures can be ported to other
languages with minimal effort. The only requirement is unlabeled parallel
data, and a loss defined over task representations. We evaluate our framework
by transferring an English sentiment classifier to three different languages.
On a battery of tests, we show that our models outperform a number of strong
baselines and rival state-of-the-art results, which rely on more complex
approaches and significantly more resources and data. Additionally, we find
that the framework proposed in this paper is able to capture semantically rich
and meaningful representations across languages, despite the lack of direct
supervision.

##  1 Introduction

Recent advances in Natural Language Processing (NLP) and Deep Learning have
enabled systems to achieve human parity in several key research areas such as
Speech Recognition (?), Machine Translation (?) and Machine Comprehension (?),
while advancing the state-of-the-art for many other tasks. Nevertheless, much
of this research revolves around models, methods and datasets that are Anglo-
centric. It is estimated111https://en.wikipedia.org/wiki/English-
speaking_world that only about 350 million people are native English speakers,
while another 500 million to 1 billion speak it as a second language. This
accounts for at most 20% of the world’s population. With language technologies
making inroads into the digital lives of people, it is becoming increasingly
imperative to build NLP applications that can understand the other 80% of the
world.

Building such systems from scratch can be expensive and time-consuming.
Performant NLP models often rely on vast amounts of high quality annotated
data, which come at the cost of annotator time, effort and money. Therefore,
much of the research community’s efforts towards building tools for other
languages have relied on transferring existing English models to other
languages.

Previous efforts have relied on Machine Translation (MT) to translate training
or test data from English to a target language (?; ?). Others have
additionally considered bilingual dictionaries to directly transfer features
(?), while some have explored methods such as structural correspondence
learning (?), and cross-lingual representation learning (?).

Building state-of-the-art MT systems requires expertise and vast amounts of
training data. The best commercial APIs such as Google
Translate222https://cloud.google.com/translate/ and Microsoft
Translator333https://www.microsoft.com/en-us/translator/translatorapi.aspx
charge on a per-character basis, and the long-term costs for model transfer
can become prohibitive. Meanwhile, bilingual dictionaries can be equally
expensive to build if done manually, or contain significant noise, if induced
automatically.

In this paper we propose a novel framework to transfer an existing neural
English model to another language with minimal cost and effort. The framework
leverages _task representations_ – the layer in a neural network model
immediately preceding the prediction layer, which we hypothesize captures an
abstract semantic representation of the prediciton problem – as a form of weak
supervision in unlabeled data. The result is a model that learns a language
agnostic neural architecture, jointly with language specific embeddings.

Specifically, the framework:  (i) Is model and task agnostic, and thus
applicable to a wide range of existing neural architectures.  (ii) Needs no
target language training data, no translation system or bilingual dictionary –
only a parallel corpus.  (iii) And has the sole modelling requirement of
defining a loss over task representations, thereby greatly reducing the
engineering effort involved in model transfer.

Our approach is especially useful when a high quality MT system is not
available for a target language or specialized domain. However, we also show
in our experiments that it can be used in conjunction with an MT based
transfer approach, to produce state-of-the-art results.

We test our framework on transferring an English sentiment classifier, and
evaluate on 3 different target languages using a public dataset. Our
experiments show that our approach not only outperforms several strong
baselines (including an MT system trained on the same parallel data) and
previous approaches, but even outperforms the previous state-of-the-art on one
language, while relying on a much simpler method and requiring significantly
fewer resources. Additionally, our approach is able to learn semantically
meaningful cross-lingual representations without the need for any direct
supervision.

##  2 Related Work

There are several research areas related to work in this paper. Cross-lingual
model transfer can be more generally perceived as an instance of domain
adaptation. Previous work in this area has explored techniques such as
structural correspondence learning (?), and feature sharing via augmentation
(?). Domain adaptation for deep learning has also been tackled, for example
with stacked de-noising autoencoders (?), a gradient reversal layer for domain
differentiation (?) and generalizations of Convolutional Neural Networks that
capture consistencies between domains (?).

(?) study the transferability of neural network components in the context of
image recognition. They suggest that higher layers tend to be more specialized
and domain specific, and therefore less generalizable. (?) investigate the
same problem, but looking more specifically at domain adaptation in NLP. They
find that all but the embedding layer tend to be too domain specific to be
practically transferable. Nevertheless what we propose in our framework for
cross-lingual transfer is the opposite: specifically, sharing the higher
layers of a network while maintaining language specific embeddings. Our
results demonstrate that this approach is, in fact, successful for our
evaluation task of sentiment classification.

Sharing information across domains also pertains to Multi-Task Learning. The
deep learning community’s work in this area can be broadly separated into two
approaches: hard, and soft parameter sharing. In hard parameter sharing,
models share a common architecture with some task-specific layers (?; ?),
while in soft parameter sharing tasks have their own sets of parameters that
are constrained by some shared penalty (?; ?). Our work fits in the first
category, with a shared model that is complemented by language specific
embeddings.

Several efforts have focused specifically on cross-lingual model transfer for
NLP tasks such as part-of-speech tagging (?), speech recognition (?) and
semantic role labeling (?). A number of researchers have also looked at
sentiment classification, such as (?), (?), (?) and (?). While we evaluate our
approach of model transfer on sentiment classification, our paper proposes a
general framework capable of tackling many different NLP problems in a unified
way.

Different techniques have been used for cross-lingual transfer, including co-
training (?), structural correspondence learning (?), and matrix factorization
(?; ?). More recently, neural approaches have also tackled the problem of
model transfer with approaches such as layer sharing across languages (?), and
adversarial training (?).

The closest to our work, in terms of modeling paradigm, is prior research on
label projection (?; ?), feature projection (?) and weak supervision (?; ?).
Our work differs from these methods by being a neural framework that
integrates task featurization, model learning and cross-lingual transfer in a
joint schema, while being flexible enough to accommodate a wide range of
target applications.

We also note research on bilingual embeddings, which relates to model transfer
by seeking to embed units from different languages in a joint space. Previous
solutions include modeling with canonical correlation analysis (?), matrix
factorization (?) and neural networks (?). The popular Word2Vec model (?), for
example, has been modified to work for bilingual word embeddings (?; ?), while
others have tackled longer units of representations (?). (?) provides a
comparison of several of these approaches. Notably, these efforts typically
only deal with the first embedding layer of neural networks, rather than full
model transfer – which is the problem tackled in this paper.

Figure 1: The setup of the data and model depicting our general framework for
transferring an English model to French. English embeddings are red, while
French embeddings are gold. The shared black-box architecture is purple, and
the predictive output layer is blue.

##  3 Model Transfer by Representation Projection

To describe our general framework for cross-lingual neural model transfer we
begin by providing the intuition behind the framework, then formalizing it.
Finally we show how it can be applied to easily transfer an English sentiment
classification model to other languages.

###  3.1 General Framework

Many neural NLP models can be decomposed into three distinct components. The
first is an embedding layer, which converts linguistic units (words,
characters, etc.) into dense representations. The last is a prediction layer,
– often a softmax function – used to produce a probability distribution over
the space of output labels. Sandwiched in-between these two is some task-
appropriate model architecture. Our general framework is agnostic to the
choice of this architecture, which can have an arbitrary network configuration
and number of layers. Therefore, let us assume, for the sake of this
exposition, that it is a black-box. In particular, we draw attention to the
last of these layers (i.e. the one immediately preceding the prediction layer)
and call this the _task representation layer_. These components are depicted
in Figure 1.

Our approach for model transfer relies on two key hypotheses. First, the
black-box network and prediction layer can be shared across languages; this
means that the only difference between an English and French model (for
example) is the language specific embeddings (shown by color in Figure 1).
Second, the task representation layer captures all the information required to
make a successful prediction.

Now consider a model trained on English labeled examples, and some parallel
data. An indication of successful model transfer is that the French model
predicts the same _thing_ on the French side of parallel data as the English
model does on the English side. We do not really care what this _thing_ is. It
can be an actual label (?), in which case the approach can be referred to as
_label projection_. But we can also try to produce the same task
representation in both languages (_representation projection_), a softer form
of weak supervision. This is the core of our framework. We will show in our
experiments (see Section 4) that soft representation projection is
significantly better than label projection.

We now formalize these intuitions. Consider a task $\mathbb{T}$ with some
labeled data $D_{L}=\\{(x_{i},y_{i})\mid 0\leq i\leq N\\}$, where $x_{i}$ are
English inputs and $y_{i}$ are outputs that take on $K$ possible values.
Without loss of generality let us assume that the inputs
$x_{i}=\\{e_{i1},...,e_{il}\\}$ are sequences of English words444Our framework
is agnostic to the granularity of units, and is applicable regardless of this
choice.. Moreover we have some parallel data $D_{P}=\\{(e_{j},f_{j})\mid 0\leq
j\leq M\\}$, where $e_{j}=\\{e_{j1},...,e_{jl_{1}}\\}$ and
$f_{j}=\\{f_{j1},...,f_{jl_{2}}\\}$ are parallel English and French sentences
respectively.

Let us define English embeddings $U=\\{\vec{u_{i}}\mid\forall e_{i}\in
V_{E}\\}$ such that there is a vector for every word in vocabulary $V_{E}$.
Similarly let us define $V=\\{\vec{v_{i}}\mid\forall f_{i}\in V_{F}\\}$ for
vocabulary $V_{F}$. To enable a shared model architecture, we require that
dimensions $d$ of vectors $\vec{u_{i}}$ and $\vec{v_{i}}$ be the same. We
denote a mapping of the English sequence $e_{j}=\\{e_{j1},...,e_{jm}\\}$ to a
sequence of vectors as $\vec{\vec{u_{j}}}$ and a French sequence
$f_{j}=\\{f_{j1},...,f_{jn}\\}$ to a sequence of vectors $\vec{\vec{v_{j}}}$.
We define a black-box model $\mathcal{M}$ with parameters
$\theta_{\mathcal{M}}$ that takes as input, a sequence of embeddings, and
yields a task representation. Specifically, for an English input $x_{i}$:

$\mathcal{R}_{x_{i}}^{\mathbb{T}}=\mathcal{M}(\vec{\vec{u_{i}}};\theta_{\mathcal{M}})$
(1)

Finally we consider a prediction layer $\mathcal{\pi}$ with parameters
$\theta_{\mathcal{\pi}}$ that yields a probability distribution over the $K$
output variables:

$\widehat{\mathcal{\pi}_{i}^{k}}=\frac{\mathcal{\pi}_{k}(\mathcal{R}_{x_{i}}^{\mathbb{T}};\theta_{\mathcal{\pi}})}{\displaystyle\sum_{j=1}^{K}\mathcal{\pi}_{j}(\mathcal{R}_{x_{i}}^{\mathbb{T}};\theta_{\mathcal{\pi}})}$
(2)

where $\pi_{k}$ is the $k^{\text{th}}$ neuron of the layer, and we use the
shorthand $\widehat{\mathcal{\pi}_{i}^{k}}$ to denote $P(\hat{y_{i}}=k)$. Our
framework then optimizes two losses.

Labeled Loss: Given that we have labeled English data, we optimize the
following loss for the combined network:

$\mathcal{L}_{D_{L}}=\displaystyle\sum_{i=1}^{N}\sum_{k=1}^{K}\Delta_{L}\left(\widehat{\mathcal{\pi}_{i}^{k}},y_{i}\right)$
(3)

where $\Delta_{L}$ is a loss function defined between
$\widehat{\mathcal{\pi}_{i}^{k}}$ and the response variable $y_{i}$. For
example, in the binary case $\Delta_{L}$ might be a cross-entropy loss.

Unlabeled Loss: On the parallel data we use the English task representations
generated by the model as weak supervision for the French side. Specifically:

$\mathcal{L}_{D_{P}}=\displaystyle\sum_{j=1}^{M}\Delta_{P}\left(\mathcal{R}_{e_{j}}^{\mathbb{T}},\mathcal{R}_{f_{j}}^{\mathbb{T}}\right)$
(4)

where $\Delta_{P}$ is some loss function between task representations yielded
on parallel inputs. Since task representations are vectors, the mean-squared
error between them – for example – might be an appropriate loss.

Then jointly, our final optimization problem is
$\mathcal{L}=\mathcal{L}_{D_{L}}+\alpha\mathcal{L}_{D_{P}}$, where $\alpha$ is
a hyperparameter that controls the mixing strength between the two loss
components.

Notice that with this framework there is no requirement for MT, since neither
training nor test data is ever translated. Nor are any other resources, such
as a pivot lexicon or bilingual dictionary, used. The only requirement is
paralled data and the definition of a loss function $\mathcal{L}_{D_{P}}$. The
model architecture $mathcal{M}$ and the labeled loss $\mathcal{L}_{D_{L}}$ are
properties one would have defined anyway for an English only model.

Figure 2: The hierarchical RNN model architecture that we use to predict
sentiment polarity. A sentence RNN is used to convert sequences of word
embeddings into sentence representations, which are in turn input into a
Review-level RNN to generate a task representation.

###  3.2 Training and Prediction

With well defined loss functions $\Delta_{L}$ and $\Delta_{P}$, training
consists of back-propagating errors through the network and updating the
parameters of the model. We distinguish two separate scenarios: two-stage
training, and joint training.

In two-stage training we first train the labeled loss given by Equation 3 on
the labeled English data by finding
$U^{*},\theta_{\mathcal{M}}^{*},\theta_{\pi}^{*}=\displaystyle\arg\max_{U,\theta_{\mathcal{M}},\theta_{\pi}}\mathcal{L}_{D_{L}}$.
We then freeze the English embeddings $U$, and the shared model parameters
$\theta_{\mathcal{M}}$ and $\theta_{\pi}$. In the second stage we only update
the French embeddings $V$ on the parallel data using the unlabeled projection
loss given by Equation 4 by optimizing
$V^{*}=\displaystyle\arg\max_{V}\mathcal{L}_{D_{P}}$.

In contrast, for joint training, we do not freeze any parameters and make
joint updates. That is, we optimize
$U^{*},V^{*},\theta_{\mathcal{M}}^{*},\theta_{\pi}^{*}=\displaystyle\arg\max_{U,V\theta_{\mathcal{M}},\theta_{\pi}}\mathcal{L}$.

At prediction time, all we do is use the model consisting of the parameters
$V^{*},\theta_{\mathcal{M}}^{*},\theta_{\pi}^{*}$. That is, we swap out the
English embeddings of the full task model for French ones; this is depicted on
the right-hand side of Figure 1.

###  3.3 Example Model Transfer: Sentiment Classification

We now show how the general framework we have described can be used to easily
transfer an English sentiment classifier to other languages. All we need is to
define the model architecture $\mathcal{M}$ and the two loss functions
$\mathcal{L}_{D_{L}}$ and $\mathcal{L}_{D_{P}}$.

But first a note on data granularity. In our exposition of Section 3.1, we
referred to inputs as sequences of words. However, in sentiment classification
we now consider reviews, which are sequences of sentences – which are in turn,
sequences of words. Because our labeled data is chunked into review-sized
inputs, the parallel data only needs to be aligned at paragraph, rather than
sentence or word level.

Our model architecture $\mathcal{M}$ is a hierarchical Recurrent Neural
Network (RNN) that is depicted in Figure 2. Hierarchical RNN models for
document-level classification have previously proven to be effective (?; ?).
For reference, our architecture achieves an F1 score of 0.92 when trained and
tested in English on our benchmark dataset (see Section 4.2). The model
consists of a Sentence RNN which converts sequences of words into sentence
representations, and a Review-level RNN which converts a sequence of sentence
representations into a task representation. The variant of RNN we use is Gated
Recurrent Units (GRUs) (?).

Given the binary nature of the prediction task, the prediction layer is simply
a sigmoid layer with one output neuron that computes the probability of a
positive label:
$\widehat{\mathcal{\pi}_{i}^{+}}=\sigma(\theta_{\pi}^{T}\cdot\mathcal{R}_{x_{i}}^{\mathbb{T}})$.
The labeled loss is a cross-entropy loss:

$\mathcal{L}_{D_{L}}=-\displaystyle\sum_{i=1}^{N}\left(y_{i}\log\widehat{\mathcal{\pi}_{i}^{+}}+(1-y_{i})\log(1-\widehat{\mathcal{\pi}_{i}^{+}})\right)$
(5)

On the parallel side, the unlabeled loss is a mean-squared error loss:

$\mathcal{L}_{D_{P}}=\displaystyle\sum_{j=1}^{M}\frac{1}{d^{\mathbb{T}}}\sum_{i=1}^{d^{\mathbb{T}}}\left(\mathcal{R}_{e_{j}}^{\mathbb{T}}(i)-\mathcal{R}_{f_{j}}^{\mathbb{T}}(i)\right)^{2}$
(6)

where $d^{\mathbb{T}}$ is the dimension of the task representation
$\mathcal{R}^{\mathbb{T}}$, and $\mathcal{R}^{\mathbb{T}}(i)$ denotes its
$i^{\text{th}}$ dimension.

##  4 Evaluation

We evaluate the transfer of the sentiment classifier we proposed in Section
3.3 to three different languages on a publicly available dataset. We also show
how the model may be combined with a simple translation baseline to yield
state-of-the-art results. We begin by providing implementation details and
describing the data.

###  4.1 Implementation Details

We implement our framework in Keras555https://keras.io/ and
TensorFlow666https://www.tensorflow.org/. The code will be made available by
December 2018 at https://github.com/Microsoft/CLModelTransfer.

In our standard model, we set the embedding size to 64, and the encoding size
of both sentence-level and review-level GRUs to 256. We use padding and
truncation to normalize all input data to a uniform 30 sentences, each with 20
words. We set the dropout factor to 0.5 (?) for all but the embedding layer.
In two-stage training, we train the sentiment classifier on English labeled
data for 10 epochs, then freeze its parameters, followed by 10 epochs of
training in the parallel data. In the joint training setting, we pre-train the
sentiment classifier on English labeled data for 4 epochs, followed by 12
epochs of joint training over labeled and parallel data. The optimizer we use
is RMSProp (?). We additionally experimented with varying several of the model
hyperparameters777We can report that the following did not affect model
performance, or only affected it adversely: using bidirectional instead of
unidirectional GRUs, adding Gaussian Noise between layers, varying the mixing
factor $\alpha$, and greatly changing the number of training epochs.. A study
of model size is presented in Section 5.1.

###  4.2 Data

We evaluate on the multilingual review dataset from (?), which consists of
Amazon product reviews for books, DVDs and music in 4 different languages:
English, German, French and Japanese. Reviews labeled $>3$ stars are
considered positive, while those with $<3$ stars are tagged as negative. The
reviews across languages are not aligned in any way.

We use the test portions of the data (consisting of a balanced set of 6000
reviews in each of the three languages) to measure precision, recall, F1 and
accuracy in German, French and Japanese. On the English side, we use all
available non-test instances from the dataset as labeled training data. This
amounts to a total of 111,220 reviews. We apply the logic described above to
convert “star-ratings” into binary positive or negative labels.

We use the Europarl corpus (?) for English-German and English-French language
pairs to obtain unlabeled parallel data. For each of these, the corpus
contains approximately 2 million sentence pairs. In the English-Japanese
language pair we use a parallel subtitle corpus (?), which consists of 3.2
million sentence pairs. The subtitle corpus consists of much shorter elements,
which are sometimes text fragments or even individual words.

While all the parallel data is aligned at _sentence_ level, our model does not
actually need such fine-grained alignments. Thus we mimic paragraph-level
alignment by batching contiguous groups of sentences into pseudo paragraphs.
We vary the group size randomly between 15 and 30, and ignore internal
alignment structure. As described in Section 4.1, padding and truncation are
used to normalize the size of parallel pseudo paragraphs into consistent
inputs.

###  4.3 Baselines

We compare against a number of baselines, which we divide into three broad
categories.

|  |  |  | DE | FR | JP
---|---|---|---|---|---|---
| Model | Var | Prec | Rec | F1 | Acc (%) | Prec | Rec | F1 | Acc (%) | Prec | Rec | F1 | Acc (%)
| TFTrans | LR | TrnT | 0.670 | 0.906 | 0.770 | 73.0 | 0.736 | 0.847 | 0.788 | 77.2 | 0.574 | 0.957 | 0.718 | 62.4
| TstT | 0.759 | 0.774 | 0.766 | 76.4 | 0.779 | 0.704 | 0.739 | 75.2 | 0.572 | 0.888 | 0.695 | 61.1
| NN | TrnT | 0.727 | 0.852 | 0.785 | 76.6 | 0.774 | 0.747 | 0.760 | 76.5 | 0.604 | 0.797 | 0.688 | 63.8
Trans- | TstT | 0.814 | 0.651 | 0.723 | 75.1 | 0.819 | 0.624 | 0.708 | 74.3 | 0.640 | 0.516 | 0.572 | 61.3
lation | MSTrans | LR | TrnT | 0.810 | 0.840 | 0.825 | 82.2 | 0.836 | 0.794 | 0.814 | 81.9 | 0.637 | 0.93 | 0.756 | 70.0
| TstT | 0.804 | 0.879 | 0.840 | 83.2 | 0.873 | 0.770 | 0.818 | 82.9 | 0.766 | 0.784 | 0.775 | 77.2
| NN | TrnT | 0.870 | 0.854 | 0.862 | 86.3 | 0.909 | 0.728 | 0.809 | 82.8 | 0.787 | 0.754 | 0.770 | 77.5
| TstT | 0.909 | 0.830 | 0.868* | 87.4* | 0.920 | 0.759 | 0.832* | 84.6* | 0.857 | 0.669 | 0.751 | 77.8*
No | Bi-SG | - | 0.595 | 0.450 | 0.512 | 57.2 | 0.685 | 0.229 | 0.344 | 56.2 | 0.507 | 0.726 | 0.597 | 51.1
Trans- | Label | 2-Stg | 0.583 | 0.784 | 0.669 | 61.1 | 0.595 | 0.884 | 0.711 | 64.1 | 0.807 | 0.386 | 0.522 | 64.7
lation | Projection | Joint | 0.567 | 0.961 | 0.713 | 61.4 | 0.625 | 0.924 | 0.746 | 68.5 | 0.820 | 0.462 | 0.591 | 68.0
| CL-SCL |  | - | 78.0 | - | 78.4 | - | 73.1
Prior | CR-RL |  | - | 78.1 | - | 77.3 | - | 72.9
Work | Bi-PV |  | - | 80.2 | - | 81.3 | - | 74.2
| Bi-DRL |  | - | 84.3 | - | 83.5 | - | 76.2
This | Repr. | 2-Stg | 0.708 | 0.912 | 0.797 | 76.8 | 0.703 | 0.918 | 0.796 | 76.6 | 0.773 | 0.711 | 0.741 | 75.2
Paper | Projection | Joint | 0.796 | 0.816 | 0.806 | 80.4 | 0.782 | 0.842 | 0.811 | 80.4 | 0.735 | 0.846 | 0.786* | 77.0
Table 1: Results on the Amazon Multilingual Reviews Dataset. Models are
divided into 4 groups, and the best results from each group are highlighted in
bold. Overall best results are additionally marked with *.

#### Translation Baselines

We build models over two translation systems. The first is a commercial
solution from Microsoft (MSTrans). This system is trained on orders of
magnitude more data than is available to us, and is highly tuned. So, in order
to make a fairer comparison, we also train a state-of-the-art Neural Machine
Translation (NMT) architecture on the same parallel data available to us
(TFTrans). We use an encoder-decoder architecture, which consists of 3 layers
of bidirectional LSTMs in both encoder and decoder; the embedding size is set
to 128, while the dimensions of the hidden representations in the encoders are
set to 512.

We distinguish 2 variants of translations. Training-time Translation (TrnT),
which translates the English training data into another language and then
trains a sentiment model in that language; and Test-time Translation (TstT),
which trains an English sentiment model and uses it to classify reviews that
are translated into English at test time.

Finally we also experiment with 2 modeling approaches. The first is a neural
model (NN) based on the architecture in Figure 2. The second is a logistic
regressor with L2 regularization that is trained on uni-, bi- and tri-gram
features.

This yields a total of 8 translation baselines: the cross-product of 2 types
each of translation, training/testing variants, and modeling approaches.

#### No Translation Baselines

We also build 3 baselines that do not rely on translation. The first leverages
bilingual embeddings. Specifically we train a bilingual skip-gram model (Bi-
SG) on our parallel data sets based on (?), and use the resulting English side
embeddings as fixed features in training the English sentiment model.

The second and third models are based on Label Projection. This is similar to
our architecture for model transfer, but instead of using task representations
as soft supervision for the unlabeled loss, we use the predicted label on the
English side of the parallel data as hard supervision. We distinguish 2
variants of this approach: 2-stage training, and joint training (see Section
3.2).

| DE | FR | JP
---|---|---|---
| Prec | Rec | F1 | Acc (%) | Prec | Rec | F1 | Acc (%) | Prec | Rec | F1 | Acc (%)
MSTranslator (TrnT) | 0.870 | 0.854 | 0.862 | 86.3 | 0.909 | 0.728 | 0.809 | 82.8 | 0.787 | 0.754 | 0.770 | 77.5
\+ Repr. Projection (Joint) | 0.873 | 0.860 | 0.866 | 86.7 | 0.906 | 0.772 | 0.834 | 84.6 | 0.790 | 0.803 | 0.797 | 79.5
MSTranslator (TstT) | 0.909 | 0.830 | 0.868 | 87.4 | 0.920 | 0.759 | 0.832 | 84.6 | 0.857 | 0.669 | 0.751 | 77.8
\+ Repr. Projection (Joint) | 0.903 | 0.847 | 0.874 | 87.8 | 0.909 | 0.792 | 0.846 | 85.6 | 0.852 | 0.697 | 0.767 | 78.8
Table 2: Results of combining our approach with the MSTranslator NN model.
Statistically significant improvements are shown in bold.

#### Prior Work Baselines

Finally, we also compare against prior work. Here we cite (?), who have the
current state-of-the-art on the Multilingual Amazon Reviews dataset. All
previous work on this dataset only reports accuracy scores, so we are only
able to cite these numbers. (?) report results from a number of different
models that we also provide as reference points. Notably, all these approaches
require some form of machine translation - either at training time, test time,
or both.

##### CL-SCL:

The cross-lingual structural correspondence learning approach by (?) that set
the original benchmark results for the Multilingual Reviews dataset.

##### CR-RL:

A bilingual embedding method from (?) that shares knowledge across languages
by having vectors that are part language-dependent, part language-independent.

##### Bi-PV:

An application of (?), who propose learning bilingual paragraph vectors by
treating paragraph encodings as context.

##### Bi-DRL:

The bilingual document representation learning model by (?), that is the
current state-of-the-art. It leverages monolingual and bilingual constraints
in order to jointly train a classifier, but requires both training and test
data to be translated.

###  4.4 Results

The results of our evaluation are presented in Table 1. There are several
interesting findings here. First, the NN model based on MSTrans produces very
strong results. In fact, both TrnT and TstT variants of this simple baseline
outperform the previous state-of-the-art (Bi-DRL) across all 3 languages. Note
that Bi-DRL also uses a commercial translation engine (Google) at both
training and test time. These improvements can be attributed to the progress
that MT has made; (?) report results from an MT baseline using Google
translate, and the results from 2010 are significantly worse than effectively
the same baseline in our work.

However, high quality translation comes at a cost. Translating our English
training set into _one_ other language costs approximately $1000, based on
Microsoft Translator pricing; costs are similar for Google’s translation
service. Translation at test time is even worse in the long run, since costs
are at least amortized in TrnT models. Costs can easily sky-rocket, if
translation of large datasets into many languages is considered.

Meanwhile, using a state-of-the-art NMT architecture trained on publicly
available data (TFTrans) produces significantly worse results. Because of the
discrepancies between training and test data, introduced by poorer
translations, we notice that LR can sometimes outperform NN models. This is in
contrast to MSTrans, where higher translation quality means that NN yields
better results. The performance of TrnT and TstT are also reversed vis-à-vis
MSTrans and TFTrans; while TrnT performs slightly better in the case of
TFTrans, TstT performs better for MSTrans.

The No Translation baselines are worse still. Label projection uses hard
supervision that affects model performance dramatically. This is likely
because the binary labels we use as weak supervision in parallel data do not
necessarily _mean_ anything, and therefore cause the model to learn parameters
that cannot generalize to actually meaningful test reviews. Meanwhile, Bi-SG
is little better than random. We hypothesize that cross-lingual embeddings –
if trained in isolation – are not able to mesh with higher layers of neural
networks, which are often fragilely co-ordinated with one another (?).

Meanwhile, our approach – especially under joint training – performs
significantly better. We outperform the No Translation and TFTrans baselines
by large margins. The differences are statistically significant, measured by
Fisher’s Exact Test ($\text{p-value}=0.01$). We also outperform all prior
work, except the state-of-the-art Bi-DRL model. In fact, in Japanese we even
outperform Bi-DRL and rival MSTrans-NN on accuracy, and produce the best F1
score. This shows the robustness of our approach, which is less susceptible to
language divergence than translation-based models.

In summary, our simple yet flexible framework produces results that are
significantly better than other resource reliant, complex models, which
require translation or curated dictionaries. We even rival state-of-the-art
approaches that rely on high quality commercial translation engines, which
have recourse to orders of magnitude more data.

###  4.5 Model Composition with Translation

While we are not able to outperform the MSTrans baseline, we examine if we can
improve on it by combining our approach with translation. We test this by
training a simple interpolation between our joint model and the MSTrans-NN
models on training data. We then evaluate on the test sets. The results are
presented in Table 2.

This reveals that we are able to improve upon 2 of the 3 evaluation languages.
The differences on German – while higher numerically – are not statistically
significant. The results for French and Japanese are state-of-the-art. Thus
our approach, while capable even without a translation engine, can also be
used in combination with a commercial translator to yield state-of-the-art
results.

As a reference, the empirical upper-bound yielded by native training and
testing – that is, using train and test data of the same language from the
dataset – is about 92% for German and French, and 90% for Japanese. An
interesting avenue for future research is to explore better model composition
techniques that could potentially close the gap with the upper-bound.

##  5 Model Analysis

Next we analyze our models both quantitatively and qualitatively.

###  5.1 Effects of Model Size

We first consider the effects of model size on cross-lingual transfer. We vary
the embedding dimensions from 48 to 256 in increments of 16, and encoding
dimensions from 256 to 1024 in increments of 64. For each size setting we run
a total of 10 experiments and take the average accuracy across runs. The
results of this experiment are presented in Figure 3.

Figure 3: Performance by model size. Results are the average % accuracy over
10 runs.

They show that there is a general downward trend in accuracy with larger
models. This is likely due to two factors. The first is that we have limited
labeled training data, and with more parameters, overfitting is always an
issue. Second, the unlabeled loss (Equation 4) is effectively using a vector
(noisily generated by the sentiment model on unlabeled data) as weak
supervision. The larger this vector, the more _noise_ it may contain, which in
turn leads to less reliable supervision.

Thus our framework for model transfer is better suited to more compact models,
unless bigger training datasets are available.

###  5.2 Cross-Lingual Word Association

We finally consider the embeddings learned via representation projection.
Table 3 shows a few sentiment bearing English words, and their nearest French
neighbors (by vector cosine distance on their respective embeddings) in our
joint model.

EN Word | Nearest FR Neighbors
---|---
excellent | honneur, essentiel, heureux, formidable, fantastique,
m’amène, ravie, remarquable, bienvenue, heureuse
terrible | ridicule, gaspillage, désolé, déchets, déception,
décevant, déçu, honte, désolée, pire
useful | complet, fonctionnera, civils, appropriée, utile,
digne, utilité, pertinent, pertinents, requises
cheat | désole, mensonger, déchirer, insultant, erronés,
déficitaire, floue, comprendrais, imprégné, incohérent
Table 3: The 10 closest French words (by embedding distance) to sentiment
bearing English words.

We can see that positive (or negative) sentiment terms in English are close to
positive (or negative) terms in French. Note that nearest neighbors are not
necessarily accurate translations – the sentiment prediction task does not
really require translations; it is sufficient to identify words that echo the
same sentiment. Our approach is thus able to pick up on sentiment similarity
across languages, without direct supervision and only using the weak fuzzy
signal from representation projection.

##  6 Conclusion and Future Work

In this paper, we have shown a general framework for transferring neural
models from one language to another. The framework leverages the novel idea of
representation projection: using intermediate task representations as a form
of weak predictive supervision. Notably our task agnostic framework requires
little by way of additional resources (a parallel corpus is sufficient) and
engineering (the only requirement is a loss function over task representations
in parallel data). We show how this framework can be used to easily adapt an
existing English neural model for sentiment classification to other languages.
We test our approach on three target languages and evaluate on a public
dataset of product reviews. The results reveal that our model is able to
outperform a number of strong baselines, and rival more complex models that
have access to significantly more training resources. When combined with a
commercial translation-based model, we produce state-of-the-art results.
Finally, our approach is able to learn embeddings that transfer the semantics
of sentiment polarity across languages, without the need of direct
supervision.

There are several avenues of future work that we plan to tackle. Since what we
have proposed in this paper is a framework rather than a single model, we hope
to investigate it’s application to other problems and domains. Specifically we
hope to tackle NLP tasks at different levels of granularity, such as sentence
or word level classification problems. Our goal is to also extend the
framework in order to accommodate more complex structured prediction problems,
such as part-of-speech tagging and dependency parsing, as well as tackle
multiple problems at once in a multi-task manner. Finally, we’re also excited
by the possibility of applying the framework to multi-modal (instead of multi-
lingual) transfer, where – for example – one might transfer sentiment
understanding to images, without the need for explicit image annotations.

## References

  * [Blitzer, McDonald, and Pereira 2006] Blitzer, J.; McDonald, R.; and Pereira, F.  2006\.  Domain adaptation with structural correspondence learning.  In Proceedings of the 2006 conference on empirical methods in natural language processing, 120–128.  Association for Computational Linguistics.
  * [Chen et al. 2016] Chen, X.; Sun, Y.; Athiwaratkun, B.; Cardie, C.; and Weinberger, K.  2016\.  Adversarial deep averaging networks for cross-lingual sentiment classification.  arXiv preprint arXiv:1606.01614.
  * [Cho et al. 2014] Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y.  2014\.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078.
  * [Das and Petrov 2011] Das, D., and Petrov, S.  2011\.  Unsupervised part-of-speech tagging with bilingual graph-based projections.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, 600–609.  Association for Computational Linguistics.
  * [Daumé III 2009] Daumé III, H.  2009\.  Frustratingly easy domain adaptation.  arXiv preprint arXiv:0907.1815.
  * [Faruqui and Dyer 2014] Faruqui, M., and Dyer, C.  2014\.  Improving vector space word representations using multilingual correlation.  In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 462–471.
  * [Ganin and Lempitsky 2014] Ganin, Y., and Lempitsky, V.  2014\.  Unsupervised domain adaptation by backpropagation.  arXiv preprint arXiv:1409.7495.
  * [Glorot, Bordes, and Bengio 2011] Glorot, X.; Bordes, A.; and Bengio, Y.  2011\.  Domain adaptation for large-scale sentiment classification: A deep learning approach.  In Proceedings of the 28th international conference on machine learning (ICML-11), 513–520.
  * [Gouws, Bengio, and Corrado 2015] Gouws, S.; Bengio, Y.; and Corrado, G.  2015\.  Bilbowa: Fast bilingual distributed representations without word alignments.  In International Conference on Machine Learning, 748–756.
  * [Hassan et al. 2018] Hassan, H.; Aue, A.; Chen, C.; Chowdhary, V.; Clark, J.; Federmann, C.; Huang, X.; Junczys-Dowmunt, M.; Lewis, W.; Li, M.; et al.  2018\.  Achieving human parity on automatic chinese to english news translation.  arXiv preprint arXiv:1803.05567.
  * [Hu, Peng, and Qiu 2017] Hu, M.; Peng, Y.; and Qiu, X.  2017\.  Mnemonic reader for machine comprehension.  arXiv preprint arXiv:1705.02798.
  * [Huang et al. 2013] Huang, J.-T.; Li, J.; Yu, D.; Deng, L.; and Gong, Y.  2013\.  Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers.  In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, 7304–7308.  IEEE.
  * [Klementiev, Titov, and Bhattarai 2012] Klementiev, A.; Titov, I.; and Bhattarai, B.  2012\.  Inducing crosslingual distributed representations of words.  Proceedings of COLING 2012 1459–1474.
  * [Koehn 2005] Koehn, P.  2005\.  Europarl: A parallel corpus for statistical machine translation.  In MT summit, volume 5, 79–86.
  * [Kozhevnikov and Titov 2014] Kozhevnikov, M., and Titov, I.  2014\.  Cross-lingual model transfer using feature representation projection.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, 579–585.
  * [Lee 2013] Lee, D.-H.  2013\.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on Challenges in Representation Learning, ICML, volume 3, 2.
  * [Long and Wang 2015] Long, M., and Wang, J.  2015\.  Learning multiple tasks with deep relationship networks.  arXiv preprint arXiv:1506.02117.
  * [Long et al. 2015] Long, M.; Cao, Y.; Wang, J.; and Jordan, M. I.  2015\.  Learning transferable features with deep adaptation networks.  arXiv preprint arXiv:1502.02791.
  * [Luong, Pham, and Manning 2015] Luong, T.; Pham, H.; and Manning, C. D.  2015\.  Bilingual word representations with monolingual quality in mind.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 151–159.
  * [Mihalcea, Banea, and Wiebe 2007] Mihalcea, R.; Banea, C.; and Wiebe, J.  2007\.  Learning multilingual subjective language via cross-lingual projections.  In Proceedings of the 45th annual meeting of the association of computational linguistics, 976–983.
  * [Mikolov et al. 2013] Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J.  2013\.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * [Misra et al. 2016] Misra, I.; Shrivastava, A.; Gupta, A.; and Hebert, M.  2016\.  Cross-stitch networks for multi-task learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3994–4003.
  * [Mou et al. 2016] Mou, L.; Meng, Z.; Yan, R.; Li, G.; Xu, Y.; Zhang, L.; and Jin, Z.  2016\.  How transferable are neural networks in nlp applications?  arXiv preprint arXiv:1603.06111.
  * [Padó and Lapata 2009] Padó, S., and Lapata, M.  2009\.  Cross-lingual annotation projection for semantic roles.  Journal of Artificial Intelligence Research 36(1):307–340.
  * [Pan et al. 2011] Pan, J.; Xue, G.-R.; Yu, Y.; and Wang, Y.  2011\.  Cross-lingual sentiment classification via bi-view non-negative matrix tri-factorization.  In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 289–300.  Springer.
  * [Pham, Luong, and Manning 2015] Pham, H.; Luong, T.; and Manning, C.  2015\.  Learning distributed representations for multilingual text sequences.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 88–94.
  * [Prettenhofer and Stein 2010] Prettenhofer, P., and Stein, B.  2010\.  Cross-language text classification using structural correspondence learning.  In Proceedings of the 48th annual meeting of the association for computational linguistics, 1118–1127.  Association for Computational Linguistics.
  * [Pryzant et al. 2017] Pryzant, R.; Chung, Y.; Jurafsky, D.; and Britz, D.  2017\.  JESC: Japanese-English Subtitle Corpus.  ArXiv e-prints.
  * [Shi et al. 2015] Shi, T.; Liu, Z.; Liu, Y.; and Sun, M.  2015\.  Learning cross-lingual word embeddings via matrix co-factorization.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, 567–572.
  * [Shi, Mihalcea, and Tian 2010] Shi, L.; Mihalcea, R.; and Tian, M.  2010\.  Cross language text classification by model translation and semi-supervised learning.  In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, 1057–1067.  Association for Computational Linguistics.
  * [Srivastava et al. 2014] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R.  2014\.  Dropout: A simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research 15(1):1929–1958.
  * [Tieleman and Hinton 2012] Tieleman, T., and Hinton, G.  2012\.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  COURSERA: Neural networks for machine learning 4(2):26–31.
  * [Upadhyay et al. 2016] Upadhyay, S.; Faruqui, M.; Dyer, C.; and Roth, D.  2016\.  Cross-lingual models of word embeddings: An empirical comparison.  arXiv preprint arXiv:1604.00425.
  * [Wan 2009] Wan, X.  2009\.  Co-training for cross-lingual sentiment classification.  In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-volume 1, 235–243.  Association for Computational Linguistics.
  * [Xiao and Guo 2014] Xiao, M., and Guo, Y.  2014\.  Semi-supervised matrix completion for cross-lingual text classification.  In AAAI, 1607–1614.
  * [Xiong et al. 2016] Xiong, W.; Droppo, J.; Huang, X.; Seide, F.; Seltzer, M.; Stolcke, A.; Yu, D.; and Zweig, G.  2016\.  Achieving human parity in conversational speech recognition.  arXiv preprint arXiv:1610.05256.
  * [Yang and Hospedales 2016] Yang, Y., and Hospedales, T. M.  2016\.  Trace norm regularised deep multi-task learning.  arXiv preprint arXiv:1606.04038.
  * [Yang et al. 2016] Yang, Z.; Yang, D.; Dyer, C.; He, X.; Smola, A.; and Hovy, E.  2016\.  Hierarchical attention networks for document classification.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1480–1489.
  * [Yosinski et al. 2014] Yosinski, J.; Clune, J.; Bengio, Y.; and Lipson, H.  2014\.  How transferable are features in deep neural networks?  In Advances in neural information processing systems, 3320–3328.
  * [Zhang et al. 2014] Zhang, Z.; Luo, P.; Loy, C. C.; and Tang, X.  2014\.  Facial landmark detection by deep multi-task learning.  In European Conference on Computer Vision, 94–108.  Springer.
  * [Zhou, Wan, and Xiao 2016a] Zhou, X.; Wan, X.; and Xiao, J.  2016a.  Attention-based lstm network for cross-lingual sentiment classification.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 247–256.
  * [Zhou, Wan, and Xiao 2016b] Zhou, X.; Wan, X.; and Xiao, J.  2016b.  Cross-lingual sentiment classification with bilingual document representation learning.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, 1403–1412.