* Cojocaru et al. (2017) Cojocaru, R., Rebassa-Mansergas, A., Torres, S., & García-Berro, E. 2017, MNRAS, 470, 1442
  * Copperwheat et al. (2011) Copperwheat, C. M., Marsh, T. R., Littlefair, S. P., et al. 2011, MNRAS, 410, 1113
  * Cyburt et al. (2010) Cyburt, R. H., Amthor, A. M., Ferguson, R., et al. 2010, ApJS, 189, 240
  * D’Antona & Tailo (2022) D’Antona, F. & Tailo, M. 2022, in Astrophysics and Space Science Library, Vol. 465, Astrophysics and Space Science Library, ed. S. Bhattacharyya, A. Papitto, & D. Bhattacharya, 201–244
  * Deloye et al. (2007) Deloye, C. J., Taam, R. E., Winisdoerffer, C., & Chabrier, G. 2007, MNRAS, 381, 525
  * Deng et al. (2021) Deng, Z.-L., Li, X.-D., Gao, Z.-F., & Shao, Y. 2021, ApJ, 909, 174
  * Echevarría et al. (2007) Echevarría, J., Michel, R., Costero, R., & Zharikov, S. 2007, A&A, 462, 1069
  * Echevarría et al. (2008) Echevarría, J., Smith, R. C., Costero, R., Zharikov, S., & Michel, R. 2008, MNRAS, 387, 1563
  * El-Badry et al. (2021) El-Badry, K., Rix, H.-W., Quataert, E., Kupfer, T., & Shen, K. J. 2021, MNRAS, 508, 4106
  * Ergma & Sarna (1996) Ergma, E. & Sarna, M. J. 1996, MNRAS, 280, 1000
  * Fedorova & Ergma (1989) Fedorova, A. V. & Ergma, E. V. 1989, Ap&SS, 151, 125
  * Ferguson et al. (2005) Ferguson, J. W., Alexander, D. R., Allard, F., et al. 2005, ApJ, 623, 585
  * Freedman et al. (2014) Freedman, R. S., Lustig-Yaeger, J., Fortney, J. J., et al. 2014, ApJS, 214, 25
  * Freedman et al. (2008) Freedman, R. S., Marley, M. S., & Lodders, K. 2008, ApJS, 174, 504
  * Freytag et al. (1996) Freytag, B., Ludwig, H. G., & Steffen, M. 1996, A&A, 313, 497
  * Fuller et al. (1985) Fuller, G. M., Fowler, W. A., & Newman, M. J. 1985, ApJ, 293, 1
  * Gänsicke et al. (2009) Gänsicke, B. T., Dillon, M., Southworth, J., et al. 2009, MNRAS, 397, 2170
  * Garraffo et al. (2018) Garraffo, C., Drake, J. J., Alvarado-Gomez, J. D., Moschou, S. P., & Cohen, O. 2018, ApJ, 868, 60
  * Goliasch & Nelson (2015) Goliasch, J. & Nelson, L. 2015, ApJ, 809, 80
  * Gomez et al. (2021) Gomez, S., Torres, M. A. P., Jonker, P. G., et al. 2021, MNRAS, 502, 48
  * Gossage et al. (2023) Gossage, S., Kalogera, V., & Sun, M. 2023, ApJ, 950, 27
  * Green et al. (2018a) Green, M. J., Hermes, J. J., Marsh, T. R., et al. 2018a, MNRAS, 477, 5646
  * Green et al. (2020) Green, M. J., Marsh, T. R., Carter, P. J., et al. 2020, MNRAS, 496, 1243
  * Green et al. (2019) Green, M. J., Marsh, T. R., Steeghs, D., et al. 2019, MNRAS, 485, 1947
  * Green et al. (2018b) Green, M. J., Marsh, T. R., Steeghs, D. T. H., et al. 2018b, MNRAS, 476, 1663
  * Hachisu et al. (1996) Hachisu, I., Kato, M., & Nomoto, K. 1996, ApJ, 470, L97
  * Heinke et al. (2013) Heinke, C. O., Ivanova, N., Engel, M. C., et al. 2013, ApJ, 768, 184
  * Henyey et al. (1965) Henyey, L., Vardya, M. S., & Bodenheimer, P. 1965, ApJ, 142, 841
  * Hernandez et al. (2021) Hernandez, M. S., Schreiber, M. R., Parsons, S. G., et al. 2021, MNRAS, 501, 1677
  * Hernandez et al. (2022a) Hernandez, M. S., Schreiber, M. R., Parsons, S. G., et al. 2022a, MNRAS, 512, 1843
  * Hernandez et al. (2022b) Hernandez, M. S., Schreiber, M. R., Parsons, S. G., et al. 2022b, MNRAS, 517, 2867
  * Herwig (2000) Herwig, F. 2000, A&A, 360, 952
  * Hurley et al. (2000) Hurley, J. R., Pols, O. R., & Tout, C. A. 2000, MNRAS, 315, 543
  * Hurley et al. (2002) Hurley, J. R., Tout, C. A., & Pols, O. R. 2002, MNRAS, 329, 897
  * Iglesias & Rogers (1993) Iglesias, C. A. & Rogers, F. J. 1993, ApJ, 412, 752
  * Iglesias & Rogers (1996) Iglesias, C. A. & Rogers, F. J. 1996, ApJ, 464, 943
  * Imada et al. (2018) Imada, A., Isogai, K., Yanagisawa, K., & Kawai, N. 2018, PASJ, 70, 79
  * Irwin (2004) Irwin, A. W. 2004, The FreeEOS Code for Calculating the Equation of State for Stellar Interiors
  * Istrate et al. (2014) Istrate, A. G., Tauris, T. M., & Langer, N. 2014, A&A, 571, A45
  * Itoh et al. (1996) Itoh, N., Hayashi, H., Nishikawa, A., & Kohyama, Y. 1996, ApJS, 102, 411
  * Jermyn et al. (2023) Jermyn, A. S., Bauer, E. B., Schwab, J., et al. 2023, ApJS, 265, 15
  * Jermyn et al. (2021) Jermyn, A. S., Schwab, J., Bauer, E., Timmes, F. X., & Potekhin, A. Y. 2021, ApJ, 913, 72
  * Kalomeni et al. (2016) Kalomeni, B., Nelson, L., Rappaport, S., et al. 2016, ApJ, 833, 83
  * Knigge et al. (2011) Knigge, C., Baraffe, I., & Patterson, J. 2011, ApJS, 194, 28
  * Kupfer et al. (2018) Kupfer, T., Korol, V., Shah, S., et al. 2018, MNRAS, 480, 302
  * Kupfer et al. (2016) Kupfer, T., Steeghs, D., Groot, P. J., et al. 2016, MNRAS, 457, 1828
  * Lagos et al. (2020) Lagos, F., Schreiber, M. R., Parsons, S. G., Gänsicke, B. T., & Godoy, N. 2020, MNRAS, 499, L121
  * Langanke & Martínez-Pinedo (2000) Langanke, K. & Martínez-Pinedo, G. 2000, Nuclear Physics A, 673, 481
  * Lasota et al. (2008) Lasota, J. P., Dubus, G., & Kruk, K. 2008, A&A, 486, 523
  * Lee et al. (2022) Lee, Y., Kim, S. C., Moon, D.-S., et al. 2022, ApJ, 925, L22
  * Li et al. (2019) Li, Z., Chen, X., Chen, H.-L., & Han, Z. 2019, ApJ, 871, 148
  * Lin et al. (2011) Lin, J., Rappaport, S., Podsiadlowski, P., et al. 2011, ApJ, 732, 70
  * Littlefield et al. (2013) Littlefield, C., Garnavich, P., Applegate, A., et al. 2013, AJ, 145, 145
  * Liu et al. (2021) Liu, W.-M., Jiang, L., & Chen, W.-C. 2021, ApJ, 910, 22
  * Liu et al. (2022) Liu, W. M., Yungelson, L., & Kuranov, A. 2022, A&A, 668, A80
  * Maccarone et al. (2023) Maccarone, T. J., Kupfer, T., Najera Casarrubias, E., et al. 2023, arXiv e-prints, arXiv:2302.12318
  * Marsh (1999) Marsh, T. R. 1999, MNRAS, 304, 443
  * Maxted et al. (2014) Maxted, P. F. L., Bloemen, S., Heber, U., et al. 2014, MNRAS, 437, 1681
  * McAllister et al. (2019) McAllister, M., Littlefair, S. P., Parsons, S. G., et al. 2019, MNRAS, 486, 5535
  * McDonald & Zijlstra (2015) McDonald, I. & Zijlstra, A. A. 2015, MNRAS, 448, 502
  * Nagel et al. (2009) Nagel, T., Rauch, T., & Werner, K. 2009, A&A, 499, 773
  * Nasser et al. (2001) Nasser, M. R., Solheim, J. E., & Semionoff, D. A. 2001, A&A, 373, 222
  * Nebot Gómez-Morán et al. (2011) Nebot Gómez-Morán, A., Gänsicke, B. T., Schreiber, M. R., et al. 2011, A&A, 536, A43
  * Nelemans et al. (2001) Nelemans, G., Portegies Zwart, S. F., Verbunt, F., & Yungelson, L. R. 2001, A&A, 368, 939
  * Nelemans et al. (2010) Nelemans, G., Yungelson, L. R., van der Sluys, M. V., & Tout, C. A. 2010, MNRAS, 401, 1347
  * Nelson et al. (2004) Nelson, L. A., Dubeau, E., & MacCannell, K. A. 2004, ApJ, 616, 1124
  * Nomoto et al. (2007) Nomoto, K., Saio, H., Kato, M., & Hachisu, I. 2007, ApJ, 663, 1269
  * Oda et al. (1994) Oda, T., Hino, M., Muto, K., Takahara, M., & Sato, K. 1994, Atomic Data and Nuclear Data Tables, 56, 231
  * Pala et al. (2022) Pala, A. F., Gänsicke, B. T., Belloni, D., et al. 2022, MNRAS, 510, 6110
  * Pala et al. (2020) Pala, A. F., Gänsicke, B. T., Breedt, E., et al. 2020, MNRAS, 494, 3799
  * Pala et al. (2017) Pala, A. F., Gänsicke, B. T., Townsley, D., et al. 2017, MNRAS, 466, 2855
  * Parsons et al. (2023) Parsons, S. G., Hernandez, M. S., Toloza, O., et al. 2023, MNRAS, 518, 4579
  * Parsons et al. (2015) Parsons, S. G., Schreiber, M. R., Gänsicke, B. T., et al. 2015, MNRAS, 452, 1754
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34
  * Paxton et al. (2019) Paxton, B., Smolec, R., Schwab, J., et al. 2019, ApJS, 243, 10
  * Podsiadlowski et al. (2003) Podsiadlowski, P., Han, Z., & Rappaport, S. 2003, MNRAS, 340, 1214
  * Podsiadlowski et al. (2002) Podsiadlowski, P., Rappaport, S., & Pfahl, E. D. 2002, ApJ, 565, 1107
  * Potekhin & Chabrier (2010) Potekhin, A. Y. & Chabrier, G. 2010, Contributions to Plasma Physics, 50, 82
  * Ramsay et al. (2018) Ramsay, G., Green, M. J., Marsh, T. R., et al. 2018, A&A, 620, A141
  * Rappaport et al. (1983) Rappaport, S., Verbunt, F., & Joss, P. C. 1983, ApJ, 275, 713
  * Rebassa-Mansergas et al. (2011) Rebassa-Mansergas, A., Nebot Gómez-Morán, A., Schreiber, M. R., Girven, J., & Gänsicke, B. T. 2011, MNRAS, 413, 1121
  * Reimers (1975) Reimers, D. 1975, Memoires of the Societe Royale des Sciences de Liege, 8, 369
  * Rodríguez-Gil et al. (2009) Rodríguez-Gil, P., Torres, M. A. P., Gänsicke, B. T., et al. 2009, A&A, 496, 805
  * Roelofs et al. (2005) Roelofs, G. H. A., Groot, P. J., Marsh, T. R., et al. 2005, MNRAS, 361, 487
  * Roelofs et al. (2006) Roelofs, G. H. A., Groot, P. J., Nelemans, G., Marsh, T. R., & Steeghs, D. 2006, MNRAS, 371, 1231
  * Rogers & Nayfonov (2002) Rogers, F. J. & Nayfonov, A. 2002, ApJ, 576, 1064
  * Sarkar et al. (2023a) Sarkar, A., Ge, H., & Tout, C. A. 2023a, MNRAS, 520, 3187
  * Sarkar et al. (2023b) Sarkar, A., Ge, H., & Tout, C. A. 2023b, MNRAS, 519, 2567
  * Sarkar & Tout (2022) Sarkar, A. & Tout, C. A. 2022, MNRAS, 513, 4169
  * Saumon et al. (1995) Saumon, D., Chabrier, G., & van Horn, H. M. 1995, ApJS, 99, 713
  * Schaller et al. (1992) Schaller, G., Schaerer, D., Meynet, G., & Maeder, A. 1992, A&AS, 96, 269
  * Scherbak & Fuller (2023) Scherbak, P. & Fuller, J. 2023, MNRAS, 518, 3966
  * Schreiber (2013) Schreiber, M. R. 2013, Science, 340, 932
  * Schreiber et al. (2021) Schreiber, M. R., Belloni, D., Gänsicke, B. T., Parsons, S. G., & Zorotovic, M. 2021, Nature Astronomy, 5, 648
  * Schreiber & Gänsicke (2002) Schreiber, M. R. & Gänsicke, B. T. 2002, A&A, 382, 124
  * Schreiber et al. (2010) Schreiber, M. R., Gänsicke, B. T., Rebassa-Mansergas, A., et al. 2010, A&A, 513, L7
  * Schreiber & Hessman (1998) Schreiber, M. R. & Hessman, F. V. 1998, MNRAS, 301, 626
  * Schreiber & Lasota (2007) Schreiber, M. R. & Lasota, J. P. 2007, A&A, 473, 897
  * Schreiber et al. (2016) Schreiber, M. R., Zorotovic, M., & Wijnen, T. P. G. 2016, MNRAS, 455, L16
  * Shen (2015) Shen, K. J. 2015, ApJ, 805, L6
  * Shen & Bildsten (2007) Shen, K. J. & Bildsten, L. 2007, ApJ, 660, 1444
  * Soethe & Kepler (2021) Soethe, L. T. T. & Kepler, S. O. 2021, MNRAS, 506, 3266
  * Sokolovsky et al. (2022) Sokolovsky, K. V., Strader, J., Swihart, S. J., et al. 2022, ApJ, 934, 142
  * Solheim (2010) Solheim, J. E. 2010, PASP, 122, 1133
  * Sparks & Sion (2021) Sparks, W. M. & Sion, E. M. 2021, ApJ, 914, 5
  * Staude et al. (2001) Staude, A., Schwope, A. D., & Schwarz, R. 2001, A&A, 374, 588
  * Taam & Spruit (1989) Taam, R. E. & Spruit, H. C. 1989, ApJ, 345, 972
  * Thorstensen et al. (2002) Thorstensen, J. R., Fenton, W. H., Patterson, J. O., et al. 2002, ApJ, 567, L49
  * Timmes & Swesty (2000) Timmes, F. X. & Swesty, F. D. 2000, ApJS, 126, 501
  * Toonen & Nelemans (2013) Toonen, S. & Nelemans, G. 2013, A&A, 557, A87
  * Tutukov et al. (1985) Tutukov, A. V., Fedorova, A. V., Ergma, E. V., & Yungelson, L. R. 1985, Soviet Astronomy Letters, 11, 52
  * Van & Ivanova (2019) Van, K. X. & Ivanova, N. 2019, ApJ, 886, L31
  * Van et al. (2019) Van, K. X., Ivanova, N., & Heinke, C. O. 2019, MNRAS, 483, 5595
  * van Roestel et al. (2022) van Roestel, J., Kupfer, T., Green, M. J., et al. 2022, MNRAS, 512, 5440
  * Wang (2018) Wang, B. 2018, Research in Astronomy and Astrophysics, 18, 049
  * Wang et al. (2021) Wang, B., Chen, W.-C., Liu, D.-D., et al. 2021, MNRAS, 506, 4654
  * Warner (1995) Warner, B. 1995, Cambridge Astrophysics Series Vol. 28, Cataclysmic Variable Stars (Cambridge Univ. Press, Cambridge)
  * Wolf et al. (2013) Wolf, W. M., Bildsten, L., Brooks, J., & Paxton, B. 2013, ApJ, 777, 136
  * Wong & Bildsten (2021) Wong, T. L. S. & Bildsten, L. 2021, ApJ, 923, 125
  * Woudt et al. (2012) Woudt, P. A., Warner, B., de Budé, D., et al. 2012, MNRAS, 421, 2414
  * Yamaguchi et al. (2023) Yamaguchi, N., El-Badry, K., Rodriguez, A. C., et al. 2023, MNRAS
  * Yaron et al. (2005) Yaron, O., Prialnik, D., Shara, M. M., & Kovetz, A. 2005, ApJ, 623, 398
  * Yu et al. (2019) Yu, Z., Thorstensen, J. R., Rappaport, S., et al. 2019, MNRAS, 489, 1023
  * Yungelson (2008) Yungelson, L. R. 2008, Astronomy Letters, 34, 620
  * Zorotovic & Schreiber (2022) Zorotovic, M. & Schreiber, M. 2022, MNRAS, 513, 3587
  * Zorotovic et al. (2010) Zorotovic, M., Schreiber, M. R., Gänsicke, B. T., & Nebot Gómez-Morán, A. 2010, A&A, 520, A86
  * Zorotovic et al. (2016) Zorotovic, M., Schreiber, M. R., Parsons, S. G., et al. 2016, MNRAS, 457, 3867


# NewsBERT: Distilling Pre-trained Language Model for
Intelligent News Application

Chuhan Wu1 Fangzhao Wu2 Yang Yu3 Tao Qi1 Yongfeng Huang1 Qi Liu3
1Department of Electronic Engineering & BNRist, Tsinghua University, Beijing
100084, China
2Microsoft Research Asia, Beijing 100080, China
3University of Science and Technology of China, Hefei 230027, China
{wuchuhan15, wufangzhao<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Pre-trained language models (PLMs) like BERT have made great progress in NLP.
News articles usually contain rich textual information, and PLMs have the
potentials to enhance news text modeling for various intelligent news
applications like news recommendation and retrieval. However, most existing
PLMs are in huge size with hundreds of millions of parameters. Many online
news applications need to serve millions of users with low latency tolerance,
which poses great challenges to incorporating PLMs in these scenarios.
Knowledge distillation techniques can compress a large PLM into a much smaller
one and meanwhile keeps good performance. However, existing language models
are pre-trained and distilled on general corpus like Wikipedia, which has gaps
with the news domain and may be suboptimal for news intelligence. In this
paper, we propose NewsBERT, which can distill PLMs for efficient and effective
news intelligence. In our approach, we design a teacher-student joint learning
and distillation framework to collaboratively learn both teacher and student
models, where the student model can learn from the learning experience of the
teacher model. In addition, we propose a momentum distillation method by
incorporating the gradients of teacher model into the update of student model
to better transfer the knowledge learned by the teacher model. Thorough
experiments on two real-world datasets with three tasks show that NewsBERT can
empower various intelligent news applications with much smaller models.

##  1 Introduction

Pre-trained language models (PLMs) like BERT Devlin et al. (2019) and GPT
Radford et al. (2019) have achieved remarkable success in various NLP
applications Liu et al. (2019); Yang et al. (2019). These PLMs are usually in
huge size with hundreds of millions of parameters Qiu et al. (2020). For
example, the BERT-Base model contains about 110M parameters and 12 Transformer
Vaswani et al. (2017) layers, which may raise a high demand of computational
resources in model training and inference. However, many online applications
need to provide services for a large number of concurrent users and the
tolerance of latency is often low, which hinders the deployment of large-scale
PLMs in these systems Sanh et al. (2019).

In recent years, online news websites such as MSN News and Google News have
gained huge popularity for users to digest digital news Wu et al. (2019b).
These news websites usually involve a series of intelligent news applications
like automatic news topic classification Wu et al. (2019c), news headline
generation Tan et al. (2017) and news recommendation Okura et al. (2017); Wang
et al. (2018); Wu et al. (2019b, a, d, 2021b). In these applications, text
modeling is a critical technique because news articles usually contain rich
textual content Wang et al. (2020a). Thus, these applications would benefit a
lot from the powerful language understanding ability of PLMs if they could be
incorporated in an efficient way, which further has the potential to improve
the news reading experience of millions of users Wu et al. (2020).

Knowledge distillation is a technique that can compress a cumbersome teacher
model into a lighter-weight student model by transferring useful knowledge
Hinton et al. (2015); Kim and Rush (2016). It has been employed to compress
many huge pre-trained language models into much smaller versions and meanwhile
keep most of the original performance Sanh et al. (2019); Sun et al. (2019);
Wang et al. (2020c); Jiao et al. (2020). For example, Sanh et al. Sanh et al.
(2019) proposed a DistilBERT approach, which learns the student model from the
soft target probabilities of the teacher model by using a distillation loss
with softmax-temperature Jang et al. (2016), and they regularized the hidden
state directions of the student and teacher models to be aligned. Jiao et al.
Jiao et al. (2020) proposed TinyBERT, which is an improved version of
DistilBERT. In addition to the distillation loss, they proposed to regularize
the token embeddings, hidden states and attention heatmaps of both student and
teacher models to be aligned via the mean squared error loss. These methods
usually learn the teacher and student models successively, where the student
can only learn from the results of the teacher model. However, the learning
experience of the teacher may also be useful for the learning of student model
Zhang et al. (2018), which is not considered by existing methods. In addition,
the corpus for pre-training and distilling general language models (e.g.,
WikiPedia) may also have some domain shifts with news corpus, which may not be
optimal for intelligent news applications.

In this paper, we propose a NewsBERT approach that can distill PLMs for
various intelligent news applications. In our approach, we design a teacher-
student joint learning and distillation framework to collaboratively learn
both teacher and student models in news intelligence tasks by sharing the
parameters of top layers, and meanwhile distill the student model by
regularizing the output soft probabilities and hidden representations. In this
way, the student model can learn from the teacher’s learning experience to
better imitate the teacher model, and the teacher can also be aware of the
learning status of the student model to enhance student teaching. In addition,
we propose a momentum distillation method by using the gradients of the
teacher model to boost the gradients of student model in a momentum way, which
can better transfer useful knowledge learned by the teacher model to enhance
the learning of student model. We conduct extensive experiments on two real-
world datasets that involve three news intelligence tasks. The results
validate that our proposed NewsBERT approach can consistently improve the
performance of these tasks using much smaller models and outperform many
baseline methods for PLM distillation.

The main contributions of this work include:

  * •

We propose a NewsBERT approach to distill pre-trained language models for
intelligent news applications.

  * •

We propose a teacher-student joint learning and distillation framework to
collaboratively learn both teacher and student models by sharing knowledge in
their learning process.

  * •

We propose a momentum distillation method by using the gradient of the teacher
model to boost the learning of student model in a momentum manner.

  * •

Extensive experiments on real-world datasets validate that our method can
effectively improve the model performance in various intelligent news
applications in an efficient way.

##  2 Related Work

In recent years, many researchers explore to use knowledge distillation
techniques to compress large-scale PLMs into smaller ones Tang et al. (2019);
Sanh et al. (2019); Sun et al. (2019); Mirzadeh et al. (2020); Sun et al.
(2020); Wang et al. (2020c); Jiao et al. (2020); Wang et al. (2020b); Xu et
al. (2020); Wu et al. (2021a). For example, Tang et al. Tang et al. (2019)
proposed a BiLSTMSOFT method that distills the BERT model into a single layer
BiLSTM using the distillation loss in downstream tasks. Sanh et al. Sanh et
al. (2019) proposed a DistilBERT approach, which distills the student model at
the pre-training stage using the distillation loss and a cosine embedding loss
that aligns the hidden states of teacher and student models. Sun et al. Sun et
al. (2019) proposed a patient knowledge distillation method for BERT
compression named BERT-PKD, which distills the student model by learning from
teacher’s output soft probabilities and hidden states produced by intermediate
layers. Wang et al. Wang et al. (2020c) proposed MiniLM, which employs a deep
self-attention distillation method that uses the KL-divergence loss between
teacher’s and student’s attention heatmaps computed by query-key inner-product
and the value relations computed by value-value inner-product. Jiao et al.
Jiao et al. (2020) proposed TinyBERT, which distills the BERT model at both
pre-training and fine-tuning stages by using the distillation loss and the MSE
loss between the embeddings, hidden states and attention heatmaps. There are
also a few works that explore to distill pre-trained language models for
specific downstream tasks such as document retrieval Lu et al. (2020); Chen et
al. (2020). For example, Lu et al. Lu et al. (2020) proposed a TwinBERT
approach for document retrieval, which employs a two-tower architecture with
two separate language models to encode the query and document, respectively.
They used the distillation loss function to compress the two BERT models into
smaller ones. These methods usually train the teacher and student models
successively, i.e., distilling the student model based on a well-tuned teacher
model. However, the useful experience evoked by the teacher’s learning process
cannot be exploited by the student and the teacher is also not aware of the
student’s learning status. In addition, the corpus for pre-training and
distilling these language models usually has some domain shifts with news
texts. Thus, it may not be optimal to apply the off-the-shelf distilled
language models to intelligent news applications. In this work, we propose a
NewsBERT method to distill pre-trained language models for intelligent news
applications, which can effectively reduce the computational cost of PLMs and
meanwhile achieve promising performance. We propose a teacher-student joint
learning and distillation framework, where the student model can exploit the
useful knowledge produced by the learning process of the teacher model. In
addition, we propose a momentum distillation method that integrates the
gradient of the teacher model into the student model gradient as a momentum to
boost the learning of the student.

Figure 1: The framework of our NewsBERT approach in a typical news
classification task.

##  3 NewsBERT

In this section, we introduce our NewsBERT approach that can distill PLMs for
intelligent news applications. We will first introduce the teacher-student
joint learning and distillation framework of NewsBERT by using the news
classification task as a representative example, then introduce our proposed
momentum distillation method, and finally introduce how to learn NewsBERT in
more complicated tasks like news recommendation.

###  3.1 Teacher-Student Joint Learning and Distillation Framework

The overall framework of our NewsBERT approach in a typical news
classification task is shown in Fig. 1. It contains a teacher model with a
parameter set $\Theta_{t}$ and a student model with a parameter set
$\Theta_{s}$. The teacher is a strong but large-scale PLM (e.g., BERT) with
heavy computational cost, and the goal is to learn the light-weight student
model that can keep most of the teacher’s performance. Different from existing
knowledge distillation methods that first learn the teacher model and then
distill the student model from the fixed teacher model, in our approach we
jointly learn the teacher and student models and meanwhile distilling useful
knowledge from the teacher model. Both teacher and student language models
contain an embedding layer and several Transformer Vaswani et al. (2017)
layers. We assume that the teacher model has $NK$ Transformer Vaswani et al.
(2017) layers on the top of the embedding layer and the student model contains
$N$ Transformer layers on the embedding layer. Thus, the inference speed of
the student model is approximately $K$ times faster than the teacher. We first
use the teacher and student models to separately process the input news text
(denoted as $x$) through their Transformer layers and obtain the hidden
representation of each token. We use a shared attentive pooling Yang et al.
(2016) layer (with parameter set $\Theta_{p}$) to convert the hidden
representation sequences output by the teacher and student models into unified
news embeddings, and finally use a shared dense layer (with parameter set
$\Theta_{d}$) to predict the classification probability scores based on the
news embedding. By sharing the parameters of the top pooling and dense layers,
the student model can get richer supervision information from the teacher, and
the teacher can also be aware of student’s learning status. Thus, the teacher
and student can be reciprocally learned by sharing useful knowledge encoded by
them, which is helpful for learning a strong student model.

Next, we introduce the knowledge distillation details of our approach. We
assume the $i$-th Transformer layer in the student model corresponds to the
layers $[(i-1)K+1,...,iK]$ in the teacher model. We call the stack of these
$K$ layers in the teacher model as a “block”. Motivated by Sun et al. (2019),
we apply a hidden loss to align the hidden representations given by each layer
in the student model and its corresponding block in the teacher model, which
can help the student better learn from the teacher. We denote the token
representations output by the embedding layers in the teacher and student
models as $\mathbf{E}^{t}$ and $\mathbf{E}^{s}$, respectively. The hidden
representations produced by the $i$-th layer in the student model are denoted
as $\mathbf{H}^{s}_{i}$, and the hidden representations given by the
corresponding block in the teacher model are denoted as $\mathbf{H}^{t}_{iK}$.
The hidden loss function applied to these layers is formulated as follows:

$\mathcal{L}^{l}_{hidden}(x,\Theta_{t};\Theta_{s})=\mathrm{MSE}(\mathbf{E}^{t},\mathbf{E}^{s})+\sum_{i=1}^{N}\mathrm{MSE}(\mathbf{H}^{t}_{iK},\mathbf{H}^{s}_{i}),$
(1)

where MSE stands for the Mean Squared Error loss function. In addition, since
the pooling layer is shared between student and teacher, we expect the unified
news embeddings learned by the pooling layers in the teacher and student
models (denoted as $\mathbf{h}^{t}$ and $\mathbf{h}^{s}$ respectively) to be
similar. Thus, we propose to apply an additional hidden loss to these
embeddings, which is formulated as follows:

$\mathcal{L}^{p}_{hidden}(x,\Theta_{t};\Theta_{s},\Theta_{p})=\mathrm{MSE}(\mathbf{h}^{t},\mathbf{h}^{s}).$
(2)

Besides, to encourage the student model to make similar predictions with the
teacher model, we use the distillation loss function to regularize the output
soft labels. We denote the soft labels predicted by the teacher and student
models as $\hat{y}_{t}$ and $\hat{y}_{s}$, respectively. The distillation loss
is formulated as:

$\mathcal{L}_{distill}(x,\Theta_{t};\Theta_{s},\Theta_{p},\Theta_{d})=\mathrm{CE}(\hat{y}_{t}/t,\hat{y}_{s}/t),$
(3)

where CE stands for the cross-entropy function and $t$ is the temperature
value. The overall loss function for distillation is a summation of the hidden
losses and the distillation loss, which is formulated as:

$\mathcal{L}_{d}(x,\Theta_{t};\Theta_{s},\Theta_{p},\Theta_{d})=\mathcal{L}^{l}_{hidden}+\mathcal{L}^{p}_{hidden}+\mathcal{L}_{distill}.$
(4)

Since the original teacher and student models are task-agnostic, both teacher
and student models need to receive task-specific supervision signals from the
task labels (denoted as $y$) to tune their parameters. Thus, the unified loss
function $\mathcal{L}^{s}$ for training the student model is the summation of
the overall distillation loss and the classification loss, which is written as
follows:

$\mathcal{L}^{s}(x,\Theta_{t};\Theta_{s},\Theta_{p},\Theta_{d})=\mathcal{L}_{d}(x,\Theta_{t};\Theta_{s},\Theta_{p},\Theta_{d})+\mathrm{CE}(\hat{y}_{s},y).$
(5)

Since we do not expect the teacher to be influenced by the student too
heavily, the loss function $\mathcal{L}^{t}$ for training the teacher model is
only the classification loss, which is computed as follows:

$\mathcal{L}^{t}(x;\Theta_{t},\Theta_{p},\Theta_{d})=\mathrm{CE}(\hat{y}_{t},y).$
(6)

By jointly optimizing the loss functions of the teacher and student models via
backward propagation, we can obtain a light-weight student model that can
generate task-specific news representations for inferring the labels in
downstream tasks as the teacher model.

###  3.2 Momentum Distillation

In our approach, each Transformer layer in the student model corresponds to a
block in the teacher model and we expect they have similar behaviors in
learning hidden text representations. To help the student model better imitate
the teacher model, we propose a momentum distillation method that can inject
the gradients of the teacher model into the student model as a gradient
momentum to boost the learning of the student model. We denote the gradients
of the $j$-th layer in the $i$-th block of the teacher model as
$\mathbf{g}^{t}_{i,j}$, which is computed by optimizing the teacher’s training
loss $\mathcal{L}^{t}$ via backward propagation. The gradients of the $k$-th
layer in the student model is denoted as $\mathbf{g}^{s}_{k}$, which is
derived from $\mathcal{L}^{s}$. We use the average of the gradients from each
layer in the $i$-th block of the teacher model as the overall gradients of
this block (denoted as $\mathbf{g}^{t}_{i}$), which is formulated as:

$\mathbf{g}^{t}_{i}=\frac{1}{K}\sum_{j=1}^{K}{\mathbf{g}^{t}_{i,j}}.$ (7)

Motivated by the momentum mechanism Qian (1999); He et al. (2020), we combine
the block gradients $\mathbf{g}^{t}_{i}$ with the gradients of the
corresponding layer in the student model in a momentum manner, which is
formulated as follows:

$\mathbf{g}^{s}_{k}=\beta\mathbf{g}^{t}_{k}+(1-\beta)\mathbf{g}^{s}_{k},$ (8)

where $\beta$ is a momentum hyperparameter that controls the strength of the
gradient momentum of the teacher model. In this way, the teacher’s gradients
are explicitly injected into the student model, which may have the potential
to better guide the learning of the student by pushing each layer in the
student model to have similar function with the corresponding block in the
teacher model.

Figure 2: The framework of incorporating NewsBERT in personalized news
recommendation.

###  3.3 Applications of NewsBERT for News Intelligence

In this section, we briefly introduce the applications of NewsBERT in other
news intelligence scenarios like personalized news recommendation. An
illustrative framework of news recommendation is shown in Fig. 2, which is a
two-tower framework. The input is a sequence with a user’s $T$ historical
clicked news (denoted as $[D_{1},D_{2},...,D_{T}]$) and a candidate news
$D_{c}$, and the output is the click probability score $\hat{y}$ which can be
further used for personalized news ranking and display. We first use a shared
NewsBERT model to encode each clicked news and the candidate news into their
hidden representations $[\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{T}]$
and $\mathbf{h}_{c}$. Then, we use a user encoder to capture user interest
from the representations of clicked news and obtain a user embedding
$\mathbf{u}$. The final click probability score is predicted by matching the
user embedding $\mathbf{u}$ and $\mathbf{h}_{c}$ via a click predictor, which
can be implemented by the inner product function. In this framework, teacher
and student NewsBERT models are used to generate news embeddings separately,
while the user encoder and click predictor are shared between the teacher and
student models to generate the prediction scores, which are further
constrained by the distillation loss function. In addition, the MSE hidden
losses are simultaneously applied to all news embeddings generated by the
shared NewsBERT model and the user embedding $\mathbf{u}$ generated by the
user encoder, which can encourage the student model to be similar with the
teacher model in supporting user interest modeling.

##  4 Experiments

###  4.1 Datasets and Experimental Settings

We conduct experiments on two real-world datasets. The first dataset is MIND
Wu et al. (2020), which is a large-scale public news recommendation dataset.
It contains the news impression logs of 1 million users on the Microsoft News
website during 6 weeks (from 10/12/2019 to 11/22/2019). We used this dataset
for learning and distilling our NewsBERT model in the news topic
classification and personalized news recommendation tasks. The logs of the
first 5 weeks were used for training and validation, and the rest were
reserved for test. Since many news may appear in multiple dataset splits, in
the news topic classification task we only used the news that do not appear in
the training and validation sets for test. The second dataset is a news
retrieval dataset (named as NewsRetrieval), which was sampled from the logs of
Bing search engine from 07/31/2020 to 09/13/2020. It contains the search
queries of users and the corresponding clicked news. On this dataset, we
finetuned models distilled on MIND to measure their cross-task performance in
news retrieval. We used the logs in the first month for training, the logs in
the next week for validation, and the rest for test. The statistics of the two
datasets are summarized in Table 1.

MIND
---
# Users | 1,000,000 | # News | 161,013
# News categories | 20 | # Impressions | 15,777,377
# Click behaviors | 24,155,470 | Avg. # words per news title | 11.52
NewsRetrieval
# Queries | 1,990,942 | # News | 1,428,779
Avg. # words per query | 11.83 | Avg. # words per news text | 596.09
|  |  |

Table 1: Statistics of MIND and NewsRetrieval.

In our experiments, motivated by Chi et al. (2020), we used the first 8 layers
of the pre-trained UniLM Bao et al. (2020) model as the teacher model111We
used the UniLM V2 model., and we used the parameters of its first 1, 2 or 4
layers to initialize the student models with different capacities. In the news
recommendation task, the user encoder was implemented by an attentive pooling
layer, and the click predictor was implemented by inner product. The query
vectors in all attentive pooling layers were 256-dimensional. We used Adam
Bengio and LeCun (2015) as the model optimizer, and the learning rate was
3e-6. The temperature value $t$ was set to 1. The batch size was 32. The
dropout Srivastava et al. (2014) ratio was 0.2. The gradient momentum
hyperparameter $\beta$ was set to 0.1 and 0.15 in the news topic
classification task and the news recommendation task, respectively. These
hyperparamters were tuned on the validation set. Since the topic categories in
MIND are imbalanced, we used accuracy and macro-F1 score (denoted as macro-F)
as the metrics for the news topic classification task. Following Wu et al.
(2020), we used the AUC, MRR, nDCG@5 and nDCG@10 scores to measure the
performance of news recommendation models. On the news retrieval task, we used
AUC as the main metric. We independently repeated each experiment 5 times and
reported the average results.

Model | Topic classification |  News Recommendation | Speedup
---|---|---|---
| Accuracy | Macro-F | AUC | MRR | nDCG@5 | nDCG@10 |
Glove | 71.13 | 49.71 | 67.92 | 33.09 | 36.03 | 41.80 | -
BERT-12 | 73.68 | 51.44 | 69.78 | 34.56 | 37.90 | 43.45 | 1.0x
BERT-8 | 73.95 | 51.56 | 70.04 | 34.70 | 38.09 | 43.79 | 1.5x
UniLM-12 | 74.54 | 51.75 | 70.53 | 35.29 | 38.61 | 44.29 | 1.0x
UniLM-8 | 74.69 | 52.10 | 70.72 | 35.40 | 38.74 | 44.41 | 1.5x
UniLM-4 | 73.53 | 51.20 | 69.64 | 34.38 | 37.65 | 43.38 | 3.0x
UniLM-2 | 72.96 | 50.76 | 68.96 | 33.52 | 36.74 | 42.48 | 6.0x
UniLM-1 | 72.32 | 50.37 | 68.02 | 33.14 | 36.09 | 41.87 | 12.0x
TwinBERT-4* | 73.59 | 51.24 | 69.78 | 34.48 | 37.76 | 43.47 | 3.0x
TwinBERT-2* | 72.98 | 50.84 | 69.12 | 33.67 | 36.89 | 42.60 | 6.0x
TwinBERT-1* | 72.40 | 50.45 | 68.32 | 33.40 | 36.34 | 42.09 | 12.0x
TinyBERT-6 | 73.54 | 50.80 | 69.77 | 34.54 | 37.88 | 43.44 | 2.0x
TinyBERT-4 | 73.17 | 50.39 | 69.39 | 33.84 | 37.50 | 43.10 | 9.4x
TinyBERT-4* | 73.76 | 51.12 | 69.90 | 34.52 | 37.77 | 43.48 | 3.0x
TinyBERT-2* | 73.15 | 50.94 | 69.35 | 33.80 | 37.42 | 43.06 | 6.0x
TinyBERT-1* | 72.55 | 50.50 | 68.40 | 33.46 | 36.39 | 42.15 | 12.0x
NewsBERT-4 | 74.45 | 51.78 | 70.31 | 34.89 | 38.32 | 43.95 | 3.0x
NewsBERT-2 | 74.10 | 51.26 | 69.89 | 34.50 | 37.75 | 43.50 | 6.0x
NewsBERT-1 | 73.49 | 50.65 | 68.97 | 33.54 | 36.77 | 42.51 | 12.0x
|  |  |  |  |  |  |

Table 2: Performance comparisons of different methods. * Means using the UniLM
model for distillation. The results of best performed teacher and student
models are highlighted.

###  4.2 Performance Evaluation

In this section, we compare the performance of our NewsBERT approach with many
baseline methods, including: (1) Glove Pennington et al. (2014), which is a
widely used pre-trained word embedding. We used Glove to initialize the word
embeddings in a Transformer Vaswani et al. (2017) model for news topic
classification and the NRMS Wu et al. (2019d) model for news recommendation.
(2) BERT Devlin et al. (2019), a popular PLM with bi-directional Transformers.
We compare the performance of the 12-layer BERT-Base model or its first 8
layers. (3) UniLM Bao et al. (2020), a unified language model for natural
language understanding and generation, which is the teacher model in our
approach. We also compare its 12-layer version and its variant using the first
1, 2, 4, or 8 layers. (4) TwinBERT Lu et al. (2020), a method to distill PLMs
for document retrieval. For fair comparison, we used the same UniLM model as
our approach. (5) TinyBERT Jiao et al. (2020), which is a state-of-the-art
two-stage knowledge distillation method for PLM compression. We compare the
performance of the officially released 4-layer and 6-layer TinyBERT models
distilled from BERT-Base and the performance of student models with 1, 2, and
4 layers distilled from the UniLM model.

Table 2 shows the performance of all the compared methods in news topic
classification and news recommendation tasks. From the results, we have the
following observations. First, compared with the Glove baseline, the methods
based on PLMs achieve better performance. It shows that contextualized word
representations generated by PLMs are more informative in language modeling.
Second, by comparing the results of BERT and UniLM (both 8- and 12-layer
versions), we find UniLM-based models perform better in both tasks. It shows
that UniLM is stronger than BERT in modeling news texts, and thereby we used
UniLM for learning and distilling our models. Third, compared with BERT-12 and
UniLM-12, their variants using the first 8 layers perform better. This may be
because the top layers in PLMs are adjusted to fitting the self-supervision
tasks (e.g., masked token prediction) while the hidden representations of
intermediate layers have better generalization ability, which is also
validated by Chi et al. (2020). Fourth, compared with TwinBERT, the results of
TinyBERT and NewsBERT are usually better. This may be because the TwinBERT
method only distills the teacher model based on the output soft labels, while
the other two methods can also align the hidden representations learned by
intermediate layers, which can help the student model better imitate the
teacher model. Fifth, our NewsBERT approach outperforms all other compared
baseline methods, and our further t-test results show the improvements are
significant at $p<0.01$ (by comparing the models with the same number of
layers). This is because our approach employs a teacher-student joint learning
and distillation framework where the student can learn from the learning
process of the teacher, which is beneficial for the student to extract useful
knowledge from the teacher model. In addition, our approach uses a momentum
distillation method that can inject the gradients of teacher model into the
student model in a momentum way, which can help each layer in the student
model to better imitate the corresponding part in the teacher model. Thus, our
approach can achieve better performance than other distillation methods.
Sixth, NewsBERT can achieve satisfactory and even comparable results with the
original PLM. For example, there is only a 0.24% accuracy gap between
NewsBERT-4 and the teacher model in the topic classification task. In
addition, the size of student models is much smaller than the original
12-layer model, and their training or inference speed is much faster (e.g.,
about 12.0x speedup for the one-layer NewsBERT). Thus, our approach has the
potential to empower various intelligent news applications in an efficient
way.

Figure 3: Cross-task performance in news retrieval.

Next, to validate the generalization ability of our approach, we evaluate the
performance of NewsBERT in an additional news retrieval task. We used the
NewsBERT model learned in the news recommendation task, and we finetuned it
with the labeled news retrieval data in a two-tower framework used by TwinBERT
Lu et al. (2020). We compared its performance with several methods, including
fine-tuning the general UniLM model or the TwinBERT and TinyBERT models
distilled in the news recommendation task. The results are shown in Fig. 3,
from which we have several findings. First, directly fine-tuning the generally
pre-trained UniLM model is worse than using the models distilled in the news
recommendation task. This is probably because that language models are usually
pre-trained on general corpus like Wikipedia, which has some domain shifts
with the news domain. Thus, generally pre-trained language models may not be
optimal for intelligent news applications. Second, our NewsBERT approach also
achieves better cross-task performance than TinyBERT and TwinBERT. It shows
that our approach is more suitable in distilling PLMs for intelligent news
applications than these methods.

###  4.3 Effectiveness of Teacher-Student Joint Learning and Distillation
Framework

In this section, we conduct experiments to validate the advantage of our
proposed teacher-student joint learning and distillation framework over
conventional methods that learn teacher and student models successively Hinton
et al. (2015). We first compare the performance of the student models under
our framework and their variants learned in a disjoint manner. The results are
shown in Fig. 4. We find that our proposed joint learning and distillation
framework can consistently improve the performance of student models with
different capacities. This is because in our approach the student model can
learn from the useful experience evoked by the learning process of the teacher
model, and the teacher model is also aware of the student’s learning status.
However, in the disjoint learning framework, student can only learn from the
results of a passive teacher. Thus, learning teacher and student models
successively may not be optimal for distilling a high-quality student model.

(a) Topic classification.

(b) News recommendation.

Figure 4: Influence of the teacher-student joint learning and distillation
framework on the student model.

(a) Topic classification.

(b) News recommendation.

Figure 5: Influence of the teacher-student joint learning and distillation
framework on the teacher model.

We also explore the influence of the teacher-student joint learning and
distillation framework on the teacher model. We compare the performance of the
original UniLM-8 model and its variants that serve as the teacher model for
distilling different student models. The results are shown in Fig. 5. We find
a very interesting phenomenon that the performance of some teacher models is
better than the original UniLM-8 model that does not participate in the joint
learning and distillation framework. This may be because the teacher model may
also benefit from the useful knowledge encoded by the student model. These
results show that our teacher-student joint learning and distillation
framework can help learn the teacher and student models reciprocally, which
may improve both of their performance.

###  4.4 Ablation Study

In this section, we conduct experiments to validate the effectiveness of
several core techniques in our approach, including the hidden loss, the
distillation loss and the momentum distillation method. We compare the
performance of NewsBERT and its variants with one of these components removed.
The results are shown in Fig. 6. We find that the momentum distillation method
plays a critical role in our method because the performance declines
considerably when it is removed. This may be because the gradients of teacher
model condense the knowledge and experience obtained from its learning
process, which can better teach the student model to have similar function
with the teacher model and thereby yields better performance. In addition, the
distillation loss function is also important for our approach. This is because
the distillation loss regularizes the output of the student model to be
similar with the teacher model, which encourages the student model to behave
similarly with the teacher model. Besides, the hidden loss functions are also
useful for our approach. It may be because the hidden loss functions can align
the hidden representations learned by the teacher and student models, which is
beneficial for the student model to imitate the teacher.

(a) Topic classification.

(b) News recommendation.

Figure 6: Effect of each core component in NewsBERT.

###  4.5 Hyperparameter Analysis

In this section, we conduct experiments to study the influence of the gradient
momentum hyperparameter $\beta$ on the model performance. We vary the value of
$\beta$ from 0 to 0.3, and the results are shown in Fig. 7. We observe that
the performance is not optimal when the value of $\beta$ is too small. This is
because the gradient momentum is too weak under a small $\beta$, and the
useful experience from the teacher model cannot be effectively exploited.
However, the performance starts to decline when $\beta$ is relatively large
(e.g., $\beta>0.2$). This is because the gradients of the teacher model
inevitably have some inconsistency with the gradients of the student model,
and a large gradient momentum may lead the student model updates deviate the
appropriate direction. Thus, a moderate selection of $\beta$ from 0.1 to 0.2
is recommended.

(a) Topic classification.

(b) News recommendation.

Figure 7: Influence of the gradient momentum hyperparameter $\beta$.

##  5 Conclusion

In this paper, we propose a knowledge distillation approach named NewsBERT to
compress pre-trained language models for intelligent news applications. We
propose a teacher-student joint learning and distillation framework to
collaboratively train both teacher and student models, where the student model
can learn from the learning experience of the teacher model and the teacher
model is aware of the learning of student model. In addition, we propose a
momentum distillation method that combines the gradients of the teacher model
with the gradients of the student model in a momentum way, which can boost the
learning of student model by injecting the knowledge learned by the teacher.
We conduct extensive experiments on two real-world datasets with three
different news intelligence tasks. The results show that our NewsBERT approach
can effectively improve the performance of these tasks with considerably
smaller models.

## Acknowledgments

This work was supported by the National Natural Science Foundation of China
under Grant numbers 82090053, 61862002, and Tsinghua-Toyota Research Funds
20213930033.

## References

  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020.  Unilmv2: Pseudo-masked language models for unified language model pre-training.  In _ICML_ , pages 642–652. PMLR.
  * Bengio and LeCun (2015) Yoshua Bengio and Yann LeCun. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Chen et al. (2020) Xuanang Chen, Ben He, Kai Hui, Le Sun, and Yingfei Sun. 2020.  Simplified tinybert: Knowledge distillation for document retrieval.  _arXiv preprint arXiv:2009.07531_.
  * Chi et al. (2020) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020.  Infoxlm: An information-theoretic framework for cross-lingual language model pre-training.  _arXiv preprint arXiv:2007.07834_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , pages 4171–4186.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , pages 9729–9738.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  Tinybert: Distilling BERT for natural language understanding.  In _EMNLP Findings_ , pages 4163–4174.
  * Kim and Rush (2016) Yoon Kim and Alexander M Rush. 2016.  Sequence-level knowledge distillation.  In _EMNLP_ , pages 1317–1327.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2020) Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020.  Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval.  In _CIKM_ , pages 2645–2652.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020.  Improved knowledge distillation via teacher assistant.  In _AAAI_ , volume 34, pages 5191–5198.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017.  Embedding-based news recommendation for millions of users.  In _KDD_ , pages 1933–1942. ACM.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_ , pages 1532–1543.
  * Qian (1999) Ning Qian. 1999.  On the momentum term in gradient descent learning algorithms.  _Neural networks_ , 12(1):145–151.
  * Qiu et al. (2020) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020\.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ , pages 1–26.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _JMLR_ , 15(1):1929–1958.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for bert model compression.  In _EMNLP-IJCNLP_ , pages 4314–4323.
  * Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020\.  Mobilebert: a compact task-agnostic bert for resource-limited devices.  In _ACL_ , pages 2158–2170.
  * Tan et al. (2017) Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017.  From neural sentence summarization to headline generation: A coarse-to-fine approach.  In _IJCAI_ , volume 17, pages 4109–4115.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling task-specific knowledge from bert into simple neural networks.  _arXiv preprint arXiv:1903.12136_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wang et al. (2020a) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020a.  Fine-grained interest matching for neural news recommendation.  In _ACL_ , pages 836–845.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018.  Dkn: Deep knowledge-aware network for news recommendation.  In _WWW_ , pages 1835–1844.
  * Wang et al. (2020b) Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020b.  Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers.  _arXiv preprint arXiv:2012.15828_.
  * Wang et al. (2020c) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020c.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _NeurIPS_.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning.  In _IJCAI_ , pages 3863–3869.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Npa: Neural news recommendation with personalized attention.  In _KDD_ , pages 2576–2584.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Mingxiao An, Yongfeng Huang, and Xing Xie. 2019c.  Neural news recommendation with topic-aware news representation.  In _ACL_ , pages 1154–1159.
  * Wu et al. (2019d) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019d.  Neural news recommendation with multi-head self-attention.  In _EMNLP_ , pages 6390–6395.
  * Wu et al. (2021a) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021a.  One teacher is enough? pre-trained language model distillation from multiple teachers.  In _Findings of ACL_ , pages 4408–4413.
  * Wu et al. (2021b) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021b.  Empowering news recommendation with pre-trained language models.  In _SIGIR_ , pages 1652–1656. ACM.
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020.  Mind: A large-scale dataset for news recommendation.  In _ACL_ , pages 3597–3606.
  * Xu et al. (2020) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020.  Bert-of-theseus: Compressing bert by progressive module replacing.  In _EMNLP_ , pages 7859–7869.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _NeurIPS_ , volume 32, pages 5753–5763.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _NAACL-HLT_ , pages 1480–1489.
  * Zhang et al. (2018) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018.  Deep mutual learning.  In _CVPR_ , pages 4320–4328.