# Quality-aware News Recommendation

Chuhan Wu† Fangzhao Wu‡ Tao Qi† Yongfeng Huang†
†Department of Electronic Engineering, Tsinghua University, Beijing 100084,
China
‡Microsoft Research Asia, Beijing 100080, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

News recommendation is a core technique used by many online news platforms.
Recommending high-quality news to users is important for keeping good user
experiences and news platforms’ reputations. However, existing news
recommendation methods mainly aim to optimize news clicks while ignore the
quality of news they recommended, which may lead to recommending news with
uninformative content or even clickbaits. In this paper, we propose a quality-
aware news recommendation method named QualityRec that can effectively improve
the quality of recommended news. In our approach, we first propose an
effective news quality evaluation method based on the distributions of users’
reading dwell time on news. Next, we propose to incorporate news quality
information into user interest modeling by designing a content-quality
attention network to select clicked news based on both news semantics and
qualities. We further train the recommendation model with an auxiliary news
quality prediction task to learn quality-aware recommendation model, and we
add a recommendation quality regularization loss to encourage the model to
recommend higher-quality news. Extensive experiments on two real-world
datasets show that QualityRec can effectively improve the overall quality of
recommended news and reduce the recommendation of low-quality news, with even
slightly better recommendation accuracy.

##  1 Introduction

News recommendation is important for online news platforms to provide users
with personalized news reading services to alleviate their information
overload Wu et al. (2020c). Predicting whether a user will click on a
candidate news is a core task in news recommendation Wu et al. (2021). For
example, Okura et al. (2017) proposed to use a GRU network to model user
interest from clicked news, and match it with candidate news for click
prediction. Wu et al. (2019c) proposed to use multi-head self-attention
networks to model the contexts of click behaviors for user modeling, and
evaluate the relevance between user interest and candidate news for click
prediction. Wang et al. (2020) proposed a fine-grained interest matching
method based on a 3-D CNN model to predict news click scores. These methods
are dedicated to optimizing news clicks while ignoring the quality of news
they recommend. In fact, there are inevitably some low-quality news on news
platforms that are not detected by human editors or automatic quality check
systems Shu et al. (2018). These news may have uninformative and even harmful
content, and frequently recommending them will heavily injure user experience
and the reputation of online news platforms Urban and Schweiger (2014).

Figure 1: Examples of a high-quality news and a low-quality news with their
dwell time distributions.

To address the above challenge, in this work we study quality-aware news
recommendation. Instead of using manually annotated quality ratings to
evaluate news quality, we observe that users’ implicit feedback like the dwell
time of news reading can be good indications of news quality. Fig. 1 shows an
example of a high-quality news and a low-quality news as well as the
distributions of users’ dwell time on them. We can see that the high-quality
news has a formal and informative title as well as detailed information of the
baseball player in the news body. By contrast, the title of the low-quality
news is written in a clickbaity style and the body is also informal and less
informative. We can see that users’ dwell time on the high-quality news is
usually long and few users leave the news webpage quickly after click, which
means that most users are attracted by the content of this news after they
click it. In contrast, most users close the webpage of the low-quality news
very quickly, indicating that they are disappointed at its content. Thus, the
distribution of dwell time derived from users’ implicit feedback can provide
abundant and accessible clues for measuring news quality.

In this paper, we propose a news quality-aware news recommendation method
named QualityRec, which can consider the quality factor of news to enhance
news recommendation. In our approach, we first propose an effective method to
measure news quality based on the distributions of users’ dwell time on each
news. We then design a content-quality joint attention network to consider
both semantic and quality information of news when selecting important click
behaviors to model user interest in a quality-aware way. Finally, we propose
to train the model with an auxiliary news quality prediction task to learn a
quality-aware news model, and we further introduce an additional
regularization function to encourage the model to recommend news with high
qualities. Extensive experiments on two real-world datasets show that
QualityRec can effectively improve the average recommendation quality and
greatly reduce the recommendation chances of low-quality news, with even a
slightly better recommendation accuracy.

The contributions of this paper are as follows:

  * •

To our best knowledge, this is the first work that studies the problem of
quality-aware news recommendation.

  * •

We propose an effective method to measure news quality based on users’
implicit dwell time feedback.

  * •

We design a content-quality attention network to consider both semantic and
quality information of news in user interest modeling.

  * •

We develop a quality-aware model training framework with an auxiliary news
quality prediction task and a quality regularizer to learn quality-enhanced
recommendation models.

##  2 Related Work

News recommendation techniques have been extensively studied over years Wu et
al. (2021). Most existing news recommendation methods rely on click signals to
model user interest and train recommendation models Okura et al. (2017); Wang
et al. (2018); Wu et al. (2019c); Wang et al. (2020); Liu et al. (2020); Hu et
al. (2020a, b); Qi et al. (2021a); Zhang et al. (2021b, a). For example, Okura
et al. (2017) proposed to use a GRU network to learn user interest embedding
from clicked news, and use the inner product between user embedding and news
embedding to predict click scores. Wang et al. (2018) used a candidate-aware
attention network to select clicked news according to their relevance to
candidate news for user modeling, and compute click scores from the
concatenation of news and user embeddings. Wu et al. (2019c) proposed to use
multi-head self-attention networks to capture the relatedness between clicked
news for user modeling, and evaluate the inner product between user and news
embeddings for click prediction. Wang et al. (2020) explored using a 3-D CNN
network to match clicked news and candidate news for click prediction based on
their fine-grained semantic relations. Qi et al. (2021b) proposed a
popularity-aware news recommendation method that considers popularity
information of clicked news and candidate news in both user modeling and click
prediction. However, these methods can only optimize news clicks while ignore
the quality of news they recommend, which may lead to recommending some low-
quality news such as clickbaits that attract users to click.

Figure 2: The framework of QualityRec.

There are a few methods that consider user engagement signals beyond clicks to
improve news recommendation Yi et al. (2014); Wu et al. (2020a, b). For
example, Yi et al. (2014) proposed to weight a user’s clicked news in interest
modeling based on this user’s reading dwell time on them. Wu et al. (2020b)
proposed a news recommendation method based on click preference and reading
satisfaction derived from the normalized reading speed of users to model user
interest and train the recommendation models. However, these methods only
consider each individual user’s feedback on a news, which cannot effectively
indicate the quality of news. Thus, it is difficult for these methods to
surpass the recommendation of low-quality news. Different from existing
methods, we propose an effective method to measure news quality based on
implicit user feedback. We propose to incorporate news quality information
into both user modeling and model training, which can effectively promote
high-quality news and surpass low-quality news in the recommendation results.

##  3 QualityRec

In this section, we introduce the details of our proposed QualityRec method
for quality-aware news recommendation, as shown in Fig. 2. We will first
introduce how to model news quality with users’ implicit feedback, then
explain how to incorporate quality information into user modeling, and finally
introduce how to train a quality-aware news recommendation model.

###  3.1 News Quality Modeling

Effective modeling of news quality is a prerequisite for quality-aware news
recommendation. However, it is very challenging due to the lack of labeled
data and the diverse and dynamic characteristics of online news information
Louis and Nenkova (2013); Wu et al. (2020c). Prior work has found that the
implicit feedback of users on news such as dwell time can be indications of
user satisfaction on news content Lu et al. (2018); Wu et al. (2020b), which
has the potential to reflect news quality. However, the satisfaction of each
individual user cannot well indicate news quality due to the variety of
personal preferences. An intuitive way is using the average dwell time of
users on a specific news as its quality measurement. However, the average
dwell time is heavily influenced by outliers and cannot reflect the ratios of
short dwell time, which is usually important for detecting low-quality news Wu
et al. (2020a). Thus, average dwell time may be suboptimal in news quality
modeling. To solve this problem, we propose to use the distribution of dwell
time on news to measure its quality. Since the length dwell time may have some
randomness, we first quantize each continuous dwell time value $d_{i}$ into a
discrete one with the logarithm function as follows:

$\hat{d}_{i}=\lfloor\log_{2}(1+d_{i})\rfloor.$ (1)

We denote the quantized dwell time of a news as
$[\hat{d}_{1},\hat{d}_{2},...,\hat{d}_{C}]$, where $C$ is number of clicks on
it. We represent the dwell time distribution of this news as a vector
$\mathbf{t}$, and its $j-th$ element $t_{j}$ can be computed as follows:

$t_{j}=\frac{1}{C}\sum_{k}{I(\hat{d}_{k}=j)},$ (2)

where $I(\cdot)$ is an indicator function. Since the vector $\mathbf{t}$
contains the distribution information of dwell time on a news, it is more
informative than average dwell time in modeling news quality.

In addition, since we observe that the length of dwell time is approximately
lognormal, the following formulation111The element index $i$ starts from 0.
can be used if we need to directly derive a quality score $q$ from
$\mathbf{t}$:

$q=\sum_{i}i*t_{i},$ (3)

which means that the news quality score is higher if a higher ratio of clicks
is associated with longer dwell time. In this way, we can obtain a simple yet
effective method for measuring news quality from implicit user feedback.

###  3.2 Quality-aware User Modeling

We then introduce the quality-aware user modeling method in QualityRec.
Existing user modeling methods usually only consider the feedback of each
individual user into interest modeling, while the influence of the quality of
clicked news on user interest modeling and subsequent news recommendation
cannot be fully considered. Thus, in our approach we explore to consider
quality information in user modeling. We denote the historical clicked news of
a user as $[D_{1},D_{2},...,D_{N}]$, where $N$ is the history length. Based on
our proposed quality modeling method, we first obtain the dwell time
distributions of clicked news, which are denoted as
$[\mathbf{t}_{1},\mathbf{t}_{2},...,\mathbf{t}_{N}]$. We use a dense layer to
learn hidden representations of them (denoted as
$[\mathbf{h}^{d}_{1},\mathbf{h}^{d}_{2},...,\mathbf{h}^{d}_{N}]$) to better
capture quality information. To model the content information of clicked news,
following prior work Wu et al. (2019c) we use a news model to encode the
titles of clicked news into their hidden semantic representations, which are
denoted as $[\mathbf{h}^{t}_{1},\mathbf{h}^{t}_{2},...,\mathbf{h}^{t}_{N}]$.
Without loss of generality, we implement the news model with a Transformer
Vaswani et al. (2017) encoder with the attention pooling mechanism. We then
use a user model to capture the relatedness between user behaviors for user
interest modeling, and it outputs a sequence of contextualized click behavior
representations $[\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{N}]$. We also
implement the user model with a Transformer. Finally, to select both
informative and qualified news for user interest modeling, we propose a
content-quality joint attention network to pay different attention to clicked
news according to both their semantic and quality information. More
specifically, the content attention score $\alpha^{c}_{i}$ and the quality
attention score $\alpha^{q}_{i}$ of the $i$-th clicked news are computed as
follows:

$\alpha^{c}_{i}=\mathbf{v}^{c}\tanh(\mathbf{W}^{c}\mathbf{h}_{i}+\mathbf{b}^{c}),$
(4)
$\alpha^{q}_{i}=\mathbf{v}^{q}\tanh(\mathbf{W}^{q}\mathbf{h}_{i}+\mathbf{b}^{q}),$
(5)

where $\mathbf{v}^{c}$, $\mathbf{v}^{q}$, $\mathbf{W}^{c}$, $\mathbf{W}^{q}$,
$\mathbf{b}^{c}$, and $\mathbf{b}^{q}$ are model parameters. The normalized
content-quality joint attention score $\alpha_{i}$ is formulated as follows:

$\alpha_{i}=\frac{\exp(\alpha^{c}_{i}+\alpha^{q}_{i})}{\sum_{j=1}^{N}\exp(\alpha^{c}_{j}+\alpha^{q}_{j})}.$
(6)

The final quality-aware user interest embedding $\mathbf{u}$ is a weighted
summation of contextualized click behavior representations, i.e.,
$\mathbf{u}=\sum_{i=1}^{N}\alpha_{i}\mathbf{h}_{i}$. The user embedding is
further used for predicting a click score $\hat{y}$ based on its relevance to
the candidate news embedding $\mathbf{r}^{c}$, which is learned from the
candidate news title222We only use news titles for click prediction because
users cannot see the main content before click. through the news encoder and
an additional dense layer. The click score is finally used for personalized
news ranking.

###  3.3 Model Training

We then introduce the quality-aware training framework of QualityRec. The main
task is the click prediction task. Following many existing works Wu et al.
(2019b) we use the negative sampling method to construct training samples from
impressions, and we use crossentropy loss (denoted as $\mathcal{L}_{c}$) to
train the click prediction model. However, the click prediction task is not
aware of news quality signals, and therefore we introduce two additional loss
functions to train the model.

The first one is a quality prediction loss, which aims to predict the quality
score $q$ of a candidate news derived from its dwell time distribution. Since
the quality of news depends on both its title and main body, we apply the news
encoder to the concatenation of news title and body to learn a hidden news
representation $\mathbf{h}^{c}$. We apply a dense layer to learn quality
prediction task-specific news representations from $\mathbf{h}^{c}$, and use
another dense layer to predict the quality score $\hat{q}$. To train the
quality prediction model, we use the mean absolute error as the loss function
(denoted as $\mathcal{L}_{q}$), which is formulated as follows:

$\mathcal{L}_{q}=\frac{1}{Q}|\hat{q}-q|,$ (7)

where $Q$ is the maximum value of $q$ in the training dataset (for
normalization purpose).333We use the quality score as the prediction target
rather than the raw dwell time distributions because we find it is quite
difficult to predict dwell time distributions from news content due to the
uncertainty of user behaviors.

The second one is a recommendation quality regularization. It encourages the
model to give high click scores for high-quality news and demote low-quality
news. To achieve this goal, we design a regularization function
$\mathcal{L}_{r}$ as follows:

$\mathcal{L}_{r}=\frac{\hat{y}}{\epsilon+q},$ (8)

where $\epsilon$ is a small value (i.e., 1e-6) to avoid the click score being
divided by a zero quality score. In this loss, if the quality of top ranked
news is lower, the regularization loss will be larger. Thus, the model will be
encouraged to recommend news with higher quality by optimizing this loss. The
unified loss $\mathcal{L}$ for joint model training is a weighted summation of
the above three kinds of losses:

$\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{q}+\mu\mathcal{L}_{r},$ (9)

where $\lambda$ and $\mu$ are hyperparameters that control the intensity of
the two auxiliary losses.

##  4 Experiments

###  4.1 Datasets and Experimental Settings

We use two real-world news recommendation datasets for conducting our
experiments. The first one is drawn from Wu et al. (2020b), which is a dataset
collected from Microsoft News (denoted as News). It contains 500,000 news
impressions of 285,563 users from 10/12/2019 to 11/13/2019. The second one is
a proprietary dataset collected by our self from a commercial news feeds
platform (denoted as Feeds.444Anonymized for double-blind review. It contains
500,000 impression logs of 337,286 users between 06/23/2019 and 07/20/2019. On
both datasets, the impressions in the last week are used for test, and the
rest are used for training and validation. More detailed statistical
information of the two datasets is shown in Table 1. In addition, we show the
distributions of quality score on both datasets in Fig. 3.

| News | Feeds
---|---|---
#news | 95,628 | 112,052
#users | 285,563 | 337,286
#impressions | 500,000 | 500,000
#clicks | 790,588 | 853,291
avg. title len. | 11.65 | 11.28
avg. dwell time | 293.17 | 267.98
avg. quality score | 6.719 | 6.297
|  |
Table 1: Statistical information of datasets.

(a) News. (b) Feeds.

Figure 3: Distributions of quality scores.

Methods | AUC($\uparrow$) | MRR($\uparrow$) | nDCG@5($\uparrow$) | nDCG@10($\uparrow$) | QS@5($\uparrow$) | QS@10($\uparrow$) | LQ@5($\downarrow$) | LQ@10($\downarrow$)
---|---|---|---|---|---|---|---|---
EBNR | 0.6086 | 0.2025 | 0.2068 | 0.2531 | 6.757 | 6.751 | 0.00638 | 0.0137
DKN | 0.6020 | 0.1981 | 0.2001 | 0.2479 | 6.760 | 6.755 | 0.00634 | 0.0136
NAML | 0.6192 | 0.2093 | 0.2133 | 0.2616 | 6.764 | 6.757 | 0.00632 | 0.0135
NPA | 0.6223 | 0.2114 | 0.2150 | 0.2644 | 6.758 | 6.750 | 0.00640 | 0.0137
NRMS | 0.6263 | 0.2140 | 0.2182 | 0.2673 | 6.761 | 6.755 | 0.00636 | 0.0136
FIM | 0.6301 | 0.2167 | 0.2211 | 0.2709 | 6.756 | 6.748 | 0.00642 | 0.0139
HieRec | 0.6376 | 0.2198 | 0.2275 | 0.2771 | 6.732 | 6.724 | 0.00649 | 0.0141
DTW | 0.6310 | 0.2135 | 0.2220 | 0.2718 | 6.772 | 6.764 | 0.00624 | 0.0134
CPRS | 0.6475 | 0.2231 | 0.2321 | 0.2830 | 6.781 | 6.770 | 0.00616 | 0.0131
QualityRec | 0.6481 | 0.2235 | 0.2323 | 0.2834 | 6.844 | 6.841 | 0.00460 | 0.0109
|  |  |  |  |  |  |  |

Table 2: Recommendation accuracy and quality on News. “$\uparrow$”: higher is
better. “$\downarrow$”: lower is better.

Methods | AUC($\uparrow$) | MRR($\uparrow$) | nDCG@5($\uparrow$) | nDCG@10($\uparrow$) | QS@5($\uparrow$) | QS@10($\uparrow$) | LQ@5($\downarrow$) | LQ@10($\downarrow$)
---|---|---|---|---|---|---|---|---
EBNR | 0.6564 | 0.2443 | 0.2650 | 0.3246 | 6.593 | 6.576 | 0.00198 | 0.00469
DKN | 0.6528 | 0.2395 | 0.2606 | 0.3208 | 6.585 | 6.572 | 0.00201 | 0.00474
NAML | 0.6672 | 0.2498 | 0.2702 | 0.3279 | 6.606 | 6.588 | 0.00186 | 0.00444
NPA | 0.6680 | 0.2501 | 0.2706 | 0.3284 | 6.604 | 6.587 | 0.00189 | 0.00448
NRMS | 0.6696 | 0.2511 | 0.2721 | 0.3294 | 6.607 | 6.590 | 0.00188 | 0.00446
FIM | 0.6714 | 0.2524 | 0.2735 | 0.3319 | 6.602 | 6.585 | 0.00191 | 0.00452
HieRec | 0.6765 | 0.2565 | 0.2770 | 0.3359 | 6.599 | 6.581 | 0.00194 | 0.00461
DTW | 0.6704 | 0.2518 | 0.2739 | 0.3301 | 6.611 | 6.594 | 0.00185 | 0.00439
CPRS | 0.6787 | 0.2582 | 0.2798 | 0.3375 | 6.620 | 6.605 | 0.00179 | 0.00416
QualityRec | 0.6801 | 0.2594 | 0.2816 | 0.3389 | 6.696 | 6.688 | 0.00134 | 0.00318
|  |  |  |  |  |  |  |

Table 3: Recommendation accuracy and quality on Feeds. “$\uparrow$”: higher is
better. “$\downarrow$”: lower is better.

In our experiments, following Wu et al. (2019c); Wang et al. (2020) we use
GloVe Pennington et al. (2014) to initialize word embeddings in the news
model. The hidden dimension in the news and user models is 400, and the
Transformers in them have 20 attention heads. We use Adam Kingma and Ba (2015)
as the model optimizer and the learning rate is 0.0001. The intensity of
dropout applied to each layer is 0.2. The loss weights $\lambda$ and $\mu$ are
2.0 and 0.5, respectively.555More details of hyperparameter are included in
Appendix. To measure recommendation accuracy, we use AUC, MRR, nDCG@5 and
nDCG@10. To measure the quality of recommended news, we use two types of
metrics. The first one is the average quality score of top K ranked news
(denoted as QS@K). The second one is the number of news with the lowest 5%
quality score that appear in the top K recommendation results (denoted as
LQ@K). We report the results under K=5 and K=10. We repeat each experiment 5
times and report the average scores.

###  4.2 Performance Evalution

First, we evaluate the performance of different methods in terms of their
recommendation accuracy and the quality of top recommended news (shorten as
recommendation quality). We compare QualityRec with the following baseline
methods: (1) EBNR Okura et al. (2017), an embedding-based news recommendation
method with a GRU network for user modeling; (2) DKN Wang et al. (2018), using
knowledge-aware CNN for news modeling and candidate-aware attention for user
modeling; (3) NAML Wu et al. (2019a), using CNN and attention networks for
news modeling and an attention network for user modeling; (4) NPA Wu et al.
(2019b), using personalized attention mechanism for both news and user
modeling; (5) NRMS Wu et al. (2019c), using multi-head self-attention
mechanism for news and user modeling; (6) FIM Wang et al. (2020), a fine-
grained interest matching method based on 3-D CNN for news recommendation. (7)
HieRec Qi et al. (2021c), a hierarchical interest modeling and matching method
for news recommendation. (8) DTW Yi et al. (2014), using dwell time to weight
clicked news in user modeling. We use NRMS as the basic model. (9) CPRS Wu et
al. (2020b), a user modeling method with click preference and reading
satisfaction for news recommendation. The results on the News and Feeds
datasets are shown in Tables 2 and 3, respectively. We have third main
observations from the results. First, the methods based on click signals only
usually have lower recommendation quality than those that consider user
engagement signals (i.e., DTW, CPRS, and QualityRec). This may be because
optimizing click-through rate may lead to recommending some news that entice
users to click (i.e, clickbaits), which usually have low content quality.
Second, QualityRec can more effectively improve recommendation quality than
DTW and CPRS, and can even slightly boost recommendation accuracy. This shows
that the engagement signals of each individual user are not suitable for
indicating news quality, and incorporating quality signals in an appropriate
way can also help target user interest. Third, we find that QualityRec is
especially effective in reducing the frequency of recommending low-quality
news. It shows that our approach may effectively mitigate the negative
influence of low-quality news on user experience.

(a) Recommendation accuracy. (b) Recommendation quality.

Figure 4: Influence of news quality measurements.

Low Predicted Quality Score | High Predicted Quality Score
---|---
Wild & wonderful treasures of SEMA 2019 – US edition | The NFL Hot Seat: Sean McVay ran out of magic, and Jared Goff isn’t helping
100 Incredible Christmas Tree Decorating Ideas | Hundreds rally at Texas Governor’s Mansion in support of death row inmate Rodney Reed
14 Celebs Over 50 Who Are In The Best Shape Of Their Lives | Anthony Bourdain’s Chef’s Knife Sold for Over $230,000
The 22 Wildest Celebrity Hair Transformations of the Last Decade | Free Rodney Reed Petition Surpasses 2 Million Signatures to Stop Execution of Texas Death Row Inmate
25 Biggest Grocery Store Mistakes Making You Gain Weight | Denny Hamlin seals title bid with clinching Phoenix victory
Top 30 Mistakes Diyers Make When Remodeling a Kitchen | ’Beautiful boys’: Victims in Mexico ambush remembered as funerals
Best Nostalgic Pics of Leonardo Dicaprio | Amazon Is Accused of Forcing Up Prices in Antitrust Complaint
30 Style Shortcuts for No-hassle Christmas Decor | An Arctic blast is coming to the eastern part of the US
30 home trends that ruled the last decade | Winners and losers in College Football Playoff first rankings include Big Ten and Clemson
These Are The World’s Most Beautiful Small Towns | Monday Measure: Why Clemson deserves playoff inclusion (and more respect)

Table 4: The news with the lowest or the highest predicted quality scores.

###  4.3 Effectiveness of News Quality Modeling

Next, we verify the effectiveness of the news quality modeling method we
proposed. We first compare the recommendation accuracy and quality under
different quality modeling methods, including: (a) using average dwell time of
a news as quality measurement; (b) using the average of the logarithm of dwell
time as quality measurement; (c) using the distribution of quantized dwell
time. We conducted experiments on the News dataset and the results are shown
in Fig. 4.666The following experimental results are all on News. We can see
that using the average dwell time yields the worst performance. This is
because average dwell time can be heavily influenced by outlier values and can
be noisy for quality modeling. In addition, using the average of dwell time
logarithm is also suboptimal. This is because it also losses the distribution
information of dwell time and may be affected by extreme values. In addition,
we use the Pearson correlation coefficient (PCC) to evaluate the quality
prediction results under different quality label construction methods, i.e.,
(a), (b) and the quality score computing method in our approach. The PCC score
of them are 0.22, 0.36 and 0.48, respectively, which shows that the quality
scores labeled by our proposed method may be less noisy.777We provide further
discussions on the correlations between them and the quality scores rated by
users in Appendix.

We also show the top 10 news with the lowest or highest quality scores
predicted by our approach in Table 4.888The bodies of these news are not shown
due to space limit, and they can be retrieved by common search engines. We can
see that the titles of predicted low-quality news are all written in a
clickbaity style, and it is unsuitable to recommend them to users frequently.
By contrast, the predicted high-quality news have attractive and informative
titles, which may better meet users’ interest. This result shows that
QualityRec not only can serve as a news recommender, but also can be used as a
news quality evaluator to help news platforms identify potential high-quality
news and filter low-quality ones like clickbaits for better news quality
moderation.

###  4.4 Ablation Study

We then present several ablation studies on the quality-aware user modeling
method and the loss functions for model training. We remove one of them from
the model and compare the recommendation accuracy and quality in Fig. 5.999We
remove the quality attention score to deactivate quality-aware user modeling.
We find that the quality-aware user modeling method can improve both accuracy
and quality, which is because quality signals are both useful for targeting
user interest and improving recommendation quality. In addition, the quality
prediction loss has notable contributions to recommendation quality, which
shows that learning a quality-aware news model can help quality-aware
recommendation. Moreover, we find that the recommendation quality loss leads
to some accuracy sacrifices but great quality improvements. It shows that
regularizing the model to recommend high-quality news will lose some clicks,
but maybe actually beneficial for user experience.

(a) Recommendation accuracy. (b) Recommendation quality.

Figure 5: Effectiveness of the quality-aware user modeling technique and
different training losses.

###  4.5 Hyperparameter Analysis

Finally, we discuss the influence of the two loss coefficients (i.e.,
$\lambda$ and $\mu$) in Eq. (9). We vary the quality prediction loss weight
$\lambda$ from 0 to 4.0 and the recommendation quality loss weight $\mu$ from
0 to 1. The results are shown in Fig. 6. We can see that when the quality
prediction loss weight $\lambda$ is either too small or too large, the
recommendation accuracy decreases, and the recommendation quality is also
suboptimal. In addition, we find that the accuracy loss becomes bigger when
$\mu$ is larger, while the quality improvements become marginal when $\mu$ is
larger than 0.5. Thus, we prefer to choose $\lambda=2.0$ and $\mu=0.5$ to
balance recommendation accuracy and quality.

(a) Recommendation accuracy. (b) Recommendation quality.

Figure 6: Influence of the hyperparameters $\lambda$ and $\mu$.

##  5 Conclusion

In this paper, we propose a quality-aware news recommendation method named
QualityRec, which can recommend higher-quality news to users to improve their
experience. In our approach, we propose a news quality measurement based on
the distribution of dwell time. In addition, we propose a quality-aware user
modeling method by using both quality and semantic information to select
clicked news for user modeling. Besides, we propose a quality-aware model
training method with an auxiliary quality prediction task to learn quality-
aware news models, and a recommendation quality regularization loss to
encourage the model to recommend higher-quality news. Extensive experiments on
two real-world datasets show that QualityRec can effectively improve
recommendation quality and even slightly boost accuracy. It can not only serve
as a recommendation algorithm, but also can be used as a news quality
evaluator to help online news platforms to control news qualities.

## References

  * Hu et al. (2020a) Linmei Hu, Chen Li, Chuan Shi, Cheng Yang, and Chao Shao. 2020a.  Graph neural news recommendation with long-term and short-term interest modeling.  _Information Processing & Management_, 57(2):102142.
  * Hu et al. (2020b) Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, and Ming Zhou. 2020b.  Graph neural news recommendation with unsupervised preference disentanglement.  In _ACL_ , pages 4255–4264.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Liu et al. (2020) Danyang Liu, Jianxun Lian, Shiyin Wang, Ying Qiao, Jiun-Hung Chen, Guangzhong Sun, and Xing Xie. 2020.  Kred: Knowledge-aware document representation for news recommendations.  In _RecSys_ , pages 200–209.
  * Louis and Nenkova (2013) Annie Louis and Ani Nenkova. 2013.  What makes writing great? first experiments on article quality prediction in the science journalism domain.  _TACL_ , 1:341–352.
  * Lu et al. (2018) Hongyu Lu, Min Zhang, and Shaoping Ma. 2018.  Between clicks and satisfaction: Study on multi-phase user preferences and satisfaction for online news reading.  In _SIGIR_ , pages 435–444.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017.  Embedding-based news recommendation for millions of users.  In _KDD_ , pages 1933–1942.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_ , pages 1532–1543.
  * Qi et al. (2021a) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021a.  Personalized news recommendation with knowledge-aware interactive matching.  In _SIGIR_ , pages 61–70.
  * Qi et al. (2021b) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021b.  Pp-rec: News recommendation with personalized user interest and time-aware news popularity.  In _ACL_ , pages 5457–5467.
  * Qi et al. (2021c) Tao Qi, Fangzhao Wu, Chuhan Wu, Peiru Yang, Yang Yu, Xing Xie, and Yongfeng Huang. 2021c.  Hierec: Hierarchical user interest modeling for personalized news recommendation.  In _ACL_ , pages 5446–5456.
  * Shu et al. (2018) Kai Shu, Suhang Wang, Thai Le, Dongwon Lee, and Huan Liu. 2018.  Deep headline generation for clickbait detection.  In _ICDM_ , pages 467–476. IEEE.
  * Urban and Schweiger (2014) Juliane Urban and Wolfgang Schweiger. 2014.  News quality from the recipients’ perspective: Investigating recipients’ ability to judge the normative quality of news.  _Journalism Studies_ , 15(6):821–840.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020.  Fine-grained interest matching for neural news recommendation.  In _ACL_ , pages 836–845.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018.  Dkn: Deep knowledge-aware network for news recommendation.  In _WWW_ , pages 1835–1844.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning.  In _IJCAI_ , pages 3863–3869.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Npa: Neural news recommendation with personalized attention.  In _KDD_ , pages 2576–2584.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c.  Neural news recommendation with multi-head self-attention.  In _EMNLP-IJCNLP_ , pages 6390–6395.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021.  Personalized news recommendation: A survey.  _arXiv preprint arXiv:2106.08934_.
  * Wu et al. (2020a) Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2020a.  Neural news recommendation with negative feedback.  _CCF Transactions on Pervasive Computing and Interaction_ , 2(3):178–188.
  * Wu et al. (2020b) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020b.  User modeling with click preference and reading satisfaction for news recommendation.  In _IJCAI_ , pages 3023–3029.
  * Wu et al. (2020c) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020c.  Mind: A large-scale dataset for news recommendation.  In _ACL_ , pages 3597–3606.
  * Yi et al. (2014) Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu, and Suju Rajan. 2014.  Beyond clicks: dwell time for personalization.  In _RecSys_ , pages 113–120.
  * Zhang et al. (2021a) Qi Zhang, Qinglin Jia, Chuyuan Wang, Jingjie Li, Zhaowei Wang, and Xiuqiang He. 2021a.  Amm: Attentive multi-field matching for news recommendation.  In _SIGIR_ , pages 1588–1592.
  * Zhang et al. (2021b) Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021b.  UNBERT: user-news matching BERT for news recommendation.  In _IJCAI_ , pages 3356–3362.

##  Appendix A Appendix

###  A.1 Experimental Environment

Our experimental environment is built on a Linux server with Ubuntu 16.04
operation system. The version of Python is 3.8. The machine has 4 Tesla V100
GPUs with 32GB memory. We use Keras 2.2.4 with tensorflow backend 1.15 to
implement the models and experiments.

###  A.2 Hyperparameter Settings

The complete hyperparameter settings of our approach are listed in Table 5.

Hyperparameters | Values
---|---
token embedding dimension | 300
hidden dimension | 400
attention head | 20
Transformer layer | 1
$\lambda$ | 2.0
$\mu$ | 0.5
dropout | 0.2
max title length | 30
max body length | 200
max history length | 50
optimizer | Adam
learning rate | 1e-4
batch size | 32
max epoch | 4
|
Table 5: Hyperparameter settings.

###  A.3 Preprocessing

Since the quality score of news with few clicks may be noisy, we filter the
news with fewer than 10 clicks in the quality prediction task. In addition, we
limit the maximum dwell time to 4096 to filter some extremely long dwell time.

###  A.4 Notes on Quality Modeling Method

To further verify the effectiveness of our proposed quality modeling method,
we invite a group of volunteers to rate the quality of a random set of news
(the ratings are from 1 to 5). The results show that the average ratings given
by human have a 0.64 PCC score with our quality scores, while only have 0.21
and 0.32 PCC scores with the quality score computing methods (a) and (b) in
Section 4.3. It shows that our proposed quality modeling method has
consistency with the news quality evaluated by human users.


# Attention Mechanism with Energy-Friendly Operations

Yu Wana,b Baosong Yangb∗ Dayiheng Liub Rong Xiaob† Derek F. Wonga
Haibo Zhangb Boxing Chenb Lidia S. Chaoa
aNLP2CT Lab, University of Macau
<EMAIL_ADDRESS><EMAIL_ADDRESS>
bAlibaba Group
{yangbaosong.ybs,liudayiheng.ldyh,birong.xr,zhanhui.zhb,
<EMAIL_ADDRESS>Equal contribution. Work was done when interning
at DAMO Academy, Alibaba Group. Corresponding author.

###### Abstract

Attention mechanism has become the dominant module in natural language
processing models. It is computationally intensive and depends on massive
power-hungry multiplications. In this paper, we rethink variants of attention
mechanism from the energy consumption aspects. After reaching the conclusion
that the energy costs of several energy-friendly operations are far less than
their multiplication counterparts, we build a novel attention model by
replacing multiplications with either selective operations or additions.
Empirical results on three machine translation tasks demonstrate that the
proposed model, against the vanilla one, achieves competitable accuracy while
saving 99% and 66% energy during alignment calculation and the whole attention
procedure. Code is available at: https://github.com/NLP2CT/E-Att.

##  1 Introduction

Attention mechanism (ATT, Bahdanau et al., 2015; Vaswani et al., 2017; Yang et
al., 2018) has demonstrated huge success in a variety of natural language
processing tasks Su et al. (2018); Kitaev and Klein (2018); Tan et al. (2018);
Yang et al. (2019a); Devlin et al. (2019); Zhang et al. (2020). The module
learns hidden representations of a sequence by serving each word as a query to
attend to all keys in the target sentence, then softly assembling their
values. It is a de-facto standard to achieve this via performing linear
projections and dot products on representations of queries and keys Vaswani et
al. (2017), resulting in large amount of multiplications. In spite of its
promising quality, such kind of paradigm may be not the preferred solution
from the energy consumption aspect Horowitz (2014); Raffel et al. (2020). How
to build a high energy-efficient ATT still remains a great challenge.

Operation (FP32) | ASIC | FPGA
---|---|---
Addition | 0.9 | 0.4
Multiplication | 3.7 | 18.8
Table 1: Energy cost ($10^{-12}Joule$) of addition and multiplication on
ASIC/FPGA chips You et al. (2020). The representative of ASIC chip is Google’s
TPU, while Microsoft cloud employs FPGA chips.

Our work starts from in-depth investigations on approaches in ATT context with
respect to model compression Hinton et al. (2015); Jiao et al. (2020) and
complexity optimization Yang et al. (2019b); Raganato et al. (2020); Beltagy
et al. (2020); Tay et al. (2021). These approaches can potentially alleviate
the problem of high energy consumption in ATT. Nevertheless, intentions of all
these methods are not exactly from the energy-friendly perspective, thus
overlooking the origin of energy consumed, i.e., basic arithmetic operations
in electric equipments. Massive multiplications still remain, consuming far
more energy than its additive counterpart on modern devices (Table 1, Li et
al., 2020).

To this end, we propose to approach this problem from a new direction –
replacing massive multiplications in ATT with cheaper operations. Concretely,
we propose a novel energy-efficient attention mechanism (E-ATT). It equips
binarized selective operations instead of linear projections over input hidden
states, and measures attentive scores using $L_{1}$ distance rather than dot-
product. Consequently, E-ATT abandons most of multiplications to reach the
goal of energy cost reduction.

We examine our method with Transformer model Vaswani et al. (2017), and
conduct experiments on three machine translation tasks. Compared with
conventional ATT, our E-ATT can save more than 99% energy of the vanilla
alignment calculation and around 66% energy of the whole attention model. In
the meanwhile, our models yield acceptable translation qualities across
language pairs. Extensive analyses also demonstrate that E-ATT can
functionally model semantic alignments without using multiplications.

##  2 Preliminary

#### Conventional Attention Mechanism

Given input representations $\mathbf{X}\in\mathbb{R}^{l_{1}\times d}$ and
$\mathbf{Y}\in\mathbb{R}^{l_{2}\times d}$ with $l_{1}$, $l_{2}$ being sequence
length, and $d$ is the input dimensionality. Note $l_{1}$ and $l_{2}$ may be
equal for self-attention pattern, and represent lengths of target and source
sequence in cross-attention. ATT first projects the inputs into three
representations111For simplicity, we omit the bias in linear projections, as
well as splits and concatenations in multi-head mechanism.:

$\displaystyle\mathbf{Q}$
$\displaystyle=\mathbf{X}\mathbf{W}_{Q},~{}~{}~{}[\mathbf{K};\mathbf{V}]=\mathbf{Y}[\mathbf{W}_{K};\mathbf{W}_{V}],$
(1)

where
$\\{\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V}\\}\in\mathbb{R}^{d\times d}$
are trainable parameters. $\mathbf{Q}\in\mathbb{R}^{l_{1}\times d}$,
$\\{\mathbf{K},\mathbf{V}\\}\in\mathbb{R}^{l_{2}\times d}$ are query, key and
value representations, respectively. The attention alignment is calculated
with dot-product multiplication and softmax activation:

$\displaystyle\mathbf{M}_{ij}\propto\exp(\frac{\mathbf{Q}_{i}{\mathbf{K}}_{j}^{\top}}{\sqrt{d}})\in\mathbb{R}^{l_{1}\times
l_{2}}.$ (2)

Then, the output is derived by multiplying attention weights with value
representation $\hat{\mathbf{V}}$:

$\displaystyle\mathbf{O}$
$\displaystyle=\mathbf{M}{\mathbf{V}}\in\mathbb{R}^{l_{1}\times d}.$ (3)

As seen, matrix multiplications are massively exploited into conventional ATT.

#### Related Work

Several related approaches potentially alleviate the power-hungry drawback of
ATT. One direction relies on model compression by pruning redundant parameters
Denton et al. (2014); Wang et al. (2016); Zhuang et al. (2018) or distilling
the learned knowledge from a large model to a smaller one Hinton et al.
(2015); Yim et al. (2017), which still maintains multiplicative operations.
Another direction aims at reducing the computational complexity of attention
module, e.g. linearly projecting input (Dense, Tay et al., 2021), or randomly
initializing and training attention weights (RandInit, Tay et al., 2021). To
give a full comparison of energy consumption of these approaches, we conduct
the number of multiplicative operations and energy costs across modules in
Table 2. As seen, vanilla ATT Vaswani et al. (2017) involves the most
multiplicative operations, and requires the most energy than other methods.
Although Dense and RandInit significantly reduce the energy consumption, Tay
et al. (2021) point out that these approaches fail to be employed into cross-
attention networks, since neither linear transition nor randomly initialized
matrix is able to exactly model alignment information across languages.

Model | Alignment | Attention
---|---|---
# mul | $\Delta_{A}(\%)$ | # mul | $\Delta_{A}(\%)$
Vanilla | $2ld^{2}+l^{2}d$ | 100.00 | $3ld^{2}+2l^{2}d$ | 100.00
Dense | $ld^{2}+l^{2}d$ | 51.10 | $2ld^{2}+2l^{2}d$ | 67.59
RandInit | $0$ | 0.00 | $ld^{2}+l^{2}d$ | 33.80
1-5 E-ATT | $0$ | 0.44 | $ld^{2}+l^{2}d$ | 34.10

Table 2: Calls of multiplication (mul) and energy consumption ratio on ASIC
chip ($\Delta_{A}$) of vanilla ATT, Dense, RandInit, and our model.
“Alignment” and “Attention” indicate the statistics are conducted at the level
of alignment calculation and the whole attention model, respectively. $l$ and
$d$ are sequential length and model size. Please refer to Appendix A for more
details.

##  3 Energy-Efficient Attention Mechanism

In this section, we describe E-ATT by pertinently reducing the multiplicative
operations of ATT, including selective operation and $L_{1}$ distance.

###  3.1 Feature Selection with Discreteness

Since the linear transitions of queries and keys (Equation 1) involve massive
multiplications within conventional ATT, we propose to modify them with
binarized quantization Liu et al. (2018); Qin et al. (2020). Concretely, the
inputs $\mathbf{X}$ and $\mathbf{Y}$ are turned into discrete value with a
threshold function $f(\cdot)$:

$\displaystyle f(x)=\begin{cases}1&x>\tau,\\\ 0&\text{otherwise,}\end{cases}$
(4)

where $\tau$ and $d$ are threshold and hidden size, respectively. The derived
representations $\tilde{\mathbf{X}}=f(\mathbf{X})$ and
$\tilde{\mathbf{Y}}=f(\mathbf{Y})$ contain discrete features composing of
zeros and ones. Since this procedure is undifferentiable, we need to predefine
a pattern of gradient calculation for $\mathbf{X}$ when receiving back-
propagated gradient $\mathbf{Z}$. Wu et al. (2018) pointed out that, when
simulating the back-propagated progress across discrete activations, those
patterns which peak at the medium of domain reveal better training
stabilization and model performance. We thus use a modified Gaussian function
during back-propagation following Wu et al. (2018):

$\displaystyle\nabla\mathbf{X}=\sqrt{\frac{2}{\pi}}e^{-2(\mathbf{Z}-\tau)^{2}},$
(5)

and the same procedure is applied for $\mathbf{Y}$. Then given parameters
$\mathbf{W}_{Q},\mathbf{W}_{K}\in\mathbb{R}^{d\times d}$, we derive query and
key representations $\mathbf{Q},\mathbf{K}$ by applying masked selection
function:

$\displaystyle\tilde{\mathbf{Q}}$
$\displaystyle=g(\tilde{\mathbf{X}},\tilde{\mathbf{W}_{Q})}\in\mathbb{R}^{l_{1}\times
d\times d},$ (6) $\displaystyle\tilde{\mathbf{K}}$
$\displaystyle=g(\tilde{\mathbf{Y}},\tilde{\mathbf{W}_{K})}\in\mathbb{R}^{l_{2}\times
d\times d},$ (7) $\displaystyle\mathbf{Q}$
$\displaystyle=\sum_{i=1}^{d}\tilde{\mathbf{Q}}_{\cdot,i,\cdot};~{}~{}~{}\mathbf{K}=\sum_{i=1}^{d}\tilde{\mathbf{K}}_{\cdot,i,\cdot},$
(8)

where $\tilde{\mathbf{W}_{Q}}\in\mathbb{R}^{l_{1}\times d\times d}$ and
$\tilde{\mathbf{W}_{K}}\in\mathbb{R}^{l_{2}\times d\times d}$ are derived by
tiling $\mathbf{W}_{Q},\mathbf{W}_{K}$ with $l_{1}$ and $l_{2}$ times,
respectively. $g(\cdot,\cdot)$ represents indexed feature selection defined as
follows:

$\displaystyle
g(\mathbf{U},\mathbf{P})=\begin{cases}\mathbf{U}_{i,j,\cdot}&\mathbf{P}_{i,j}=1,\\\
\mathbf{0}&\text{otherwise.}\end{cases}$ (9)

###  3.2 Pairwise $L_{1}$ Distance

As the dot-product multiplication can be viewed as similarity calculation
between $\mathbf{Q}$ and $\mathbf{K}$, we argue that other similarity
estimation methods can play this role as well. Accordingly, we further propose
to use pairwise $L_{1}$ distance instead, which does not require any
multiplication. Attention score calculation in Equation 2 is then modified as:

$\displaystyle\mathbf{M}_{ij}\propto\exp(-\frac{||{\mathbf{Q}}_{i}-{\mathbf{K}}_{j}||_{1}}{\sqrt{d}}),$
(10)

where $||\cdot||_{1}$ denotes the $L_{1}$ norm of inputted vector. Here we use
negative $L_{1}$ value to ensure that larger distance contributes lower
attention score.

##  4 Experiments

###  4.1 Dataset Preprocessing

In this paper we evaluate our approach with three widely used machine
translation datasets: IWSLT’15 English -Vietnamese (En-Vi), WMT’14 English -
German (En-De) and WMT’17 Chinese - English (Zh-En). All datasets are
segmented into subwords by byte-pair encoding (BPE, Sennrich et al., 2016)
with 32k merge operations. Specially, for the former two tasks, we apply joint
BPE for both source and target languages.

Dataset | Train | Dev | Test
---|---|---|---
En-Vi | 13.3K | 1,553 | 1,268
En-De | 4.50M | 3,000 | 3,003
Zh-En | 20.6M | 2,002 | 2,001
Table 3: Dataset statistics. Each cell represents the number of examples. K:
thousand, M: million.

###  4.2 Experimental Setting

We apply Transformer-Base Vaswani et al. (2017) setting for all experiments.
The model dimensionality is 512, and 6 layers are engaged in both encoder and
decoder side. The inner-connection dimensionality for feedforward block is
2,048, and the number of heads in multi-head attention networks is 8. We share
the source embedding, target embedding and target softmax projection weight
for En-Vi task, and share the latter two matrices for En-De. We modify the
learning rate schedule as:
$lr=0.001\cdot\min\left(\frac{t}{8000},1,(\frac{20000}{t})^{0.5}\right),$
where $t$ denotes the current step. Across all tasks, we determine the
threshold $\tau$ as 1.0.

For both baseline and our model, En-Vi, En-De and Zh-En tasks take 50k, 150k
and 200k updates, and each batch contains 4,096, 32,768 and 32,768 source
tokens. The dropout ratio is determined as 0.3, 0.1 and 0.1, respectively. All
experiments are conducted over 4 NVIDIA V100 GPUs. For each task, we choose
the best model over dev set, defining beam size as 4, 4, 10 and decoding alpha
as 1.5, 0.6 and 1.5, respectively.

Attention mechanism | En-Vi | En-De | Zh-En |  ASIC (%) |  FPGA (%)
---|---|---|---|---|---
Vanilla | 30.26 $\pm$ 0.07 | 27.60 $\pm$ 0.04 | 24.28 $\pm$ 0.08 | 100.00 | 100.00
E-ATT | 29.48 $\pm$ 0.08 | 27.45 $\pm$ 0.04 | 24.23 $\pm$ 0.06 | 34.10 | 33.83

Table 4: Averaged BLEU scores (%) upon test set on IWSLT’15 En-Vi, WMT’14 En-
De and WMT’17 Zh-En tasks over 5 independent runs. E-ATT gives comparable
results against conventional ATT, reducing the energy cost at 65.90%/66.17% on
ASIC/FPGA chip in attention procedure. Since the energy cost is difficult to
be empirically evaluated, we report the theoretical values following the
common practice Chen et al. (2020); You et al. (2020).

###  4.3 Experimental Results

As shown in Table 4, vanilla model achieves best performance over all
translation tasks. However, replacing conventional attention networks with
E-ATT does not lead to significant performance drop, with small decrease of
0.15$\sim$0.78 BLEU score. Besides, after referring the statistics from Table
1 and 2, our E-ATT module takes 34.10%/33.83% energy of conventional ATT.
These results reveal that, E-ATT can achieve competitive translation quality,
and more importantly, significantly reduce the energy consumption of
attention.

###  4.4 Ablation Study

Model |  BLEU (%)
---|---
Vanilla | 28.12
Replace with discrete selection | 27.51
Replace with $L_{1}$ distance | 28.05
1-2 E-ATT | 27.45
Table 5: Model performance with component replacements over En-Vi dev set.
Using $L_{1}$ distance as similarity measurement does not harm model
performance.

We further conduct ablation experiments on En-Vi task. As seen in Table 5,
using discrete feature selection instead of linear transition slightly harms
performance, with 0.61 BLEU score decrease. Besides, replacing dot-product
attention with $L_{1}$ distance does not significantly affect model
performance. We can conclude that: 1) the performance gap between E-ATT and
vanilla model mainly stems from the usage of discrete feature selection; and
2) $L_{1}$ distance can measure the similarity of vectorized representations
and give modest performance compared to baseline.

##  5 Analyses

###  5.1 Hybrid Attention Networks

- | Vanilla | Dense | RandInit | E-ATT
---|---|---|---|---
Vanilla | 28.12 | 19.92 | 19.31 | 27.72
Dense | 27.48 | 19.43 | 19.21 | 27.60
RandInit | 27.36 | 18.98 | 18.83 | 27.48
E-ATT | 28.08 | 19.85 | 19.67 | 27.45

Table 6: BLEU score (%) of different model hybrids with modifying self-
attention (horizontal) and cross-attention (vertical) network upon En-Vi dev
set. E-ATT can achieve good performance when applied as cross-attention
modules, whereas Dense or RandInit can not.

We conduct a series of experiments involving hybrids of attention networks
among vanilla ATT, Dense, RandInit, and E-ATT module in Table 6. As shown, the
conventional attention network performs the best among all models. Our module
performs well when served as either self-attention or cross-attention modules.
Besides, for all cases applying Dense/RandInit as cross-attention modules,
models perform significantly worse, identical with the findings in Tay et al.
(2021). On the contrary, E-ATT module can give better performance with
marginal performance drop comparing with baseline, indicating that E-ATT
module is capable of providing adequate semantic alignments across languages
for translation. Besides, it is encouraging to see that our method works
compatibly with other modules with marginal performance drop.

###  5.2 Knowledge Distillation

$0$$20$$40$$60$$80$$100$$19$$24$$29$Energy consumption ratio (%)BLEU (%)ATTE-
ATT Figure 1: Performance and energy consumption of different models with
knowledge distillation on En-Vi dev set. We regard the energy consumption of
ATT baseline as 1, and accumulatively halve the dimensionality of model till
untrainable (from 512 to 64). Energy consumption is estimated over ASIC. E-ATT
requires far less energy to meet up the baseline performance.

Knowledge distillation is a representative of model compression approach
Hinton et al. (2015); Tang et al. (2019). We further conduct experiments on
ATT models with various dimensionalities compressed by knowledge distillation.
Figure 1 shows the energy consumption and performance of different models with
modified dimensionality $d$. As seen, by accumulatively halving $d$ from 512,
both ATT and E-ATT progressively loses the quality. However, E-ATT can give a
better trade-off between model performance and energy consumption than
knowledge distillation methods.

###  5.3 Case Study

Figure 2: Case study from WMT’17 Zh-En dev set. E-ATT performs well on cross-
lingual alignments.

We visualize the averaged attention values over one case from WMT’17 Zh-En dev
set. As seen, our model can give good aligned information, where preposition
phrase "around 50 years ago" is arranged at the end of sentence in English,
while its aligned phrase is at the front in Chinese. This reveals that, our
E-ATT can perform well on modeling the cross-lingual alignments.

###  5.4 Binarization Statistics

$0$$20$$40$$60$$80$$100$$19$$24$$29$1234560204060LayerRatio(%)ATTE-ATTEnc-
SelfDec-SelfDec-Crs-QueryDec-Crs-Key Figure 3: Ratio of nonzero values in the
representations of E-ATT. Enc-Self: encoder self-attention ; Dec-Self: decoder
self-attention; Dec-Crs-Query/Key: query/key representation for decoder cross-
attention. Query representations in cross-attention are the most active.

We further collect the ratio of nonzero values $\rho$ for each attention
module in Figure 3, we can see that it increases with the number of encoder
layers, denoting that more information is arranged into attentive calculation
at higher layer of source side. However, for decoder E-ATT, the ratio meets
its peak at middle layers, revealing that decoder E-ATT are most active at the
middle term of semantic processing. Interestingly, ratio in the query of
cross-attention modules, which align source and target semantics, is higher
for the layer closer to output. As the binarized key representation of each
cross-attention module is equivalent, higher ratio of nonzero values in query
representation means that, E-ATT at higher decoder layer provides more
information for cross-lingual alignments, thus enrich the information for
translation.

##  6 Discussion and Conclusion

In this paper, we empirically investigate the high energy-consumption problem
in ATT. We argue that the alignment modeling procedure can be achieved with
additions other than multiplications, thus reducing the energy costs.
Extensive analyses suggest that: 1) Binarized representations marginally harm
the feature extraction procedure; and 2) $L_{1}$ distance can be efficiently
exploited to measure alignment among queries and keys. Compared to baseline,
our approach can yield considerable quality of translations, and significantly
save energy in attention mechanism. Although we have shown the superiority of
E-ATT, considering the whole Transformer block222A Transformer block consists
of a multi-head attention layer and a feed-forward layer. Please refer to
Appendix A for more calculation details., the use of E-ATT brings 17% energy
reduction. We hope this work can attract more researches on energy-efficient
models. It is worth to further design techniques that reduce the energy cost
of other modules in Transformer.

## Acknowledgements

The authors would like to send great thanks to all reviewers and meta-reviewer
for their insightful comments. This work was supported in part by the Science
and Technology Development Fund, Macau SAR (Grant No. 0101/2019/A2), the
Multi-year Research Grant from the University of Macau (Grant No.
MYRG2020-00054-FST), and Alibaba Group through Alibaba Research Intern
Program.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural Machine Translation by Jointly Learning to Align and Translate.  In _International Conference on Learning Representations (ICLR)_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The Long-Document Transformer.  _arXiv preprint arXiv:2004.05150_.
  * Chen et al. (2020) Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. 2020.  AdderNet: Do We Really Need Multiplications in Deep Learning?  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1468–1477.
  * Denton et al. (2014) Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014.  Exploiting Linear Structure within Convolutional Networks for Efficient Evaluation.  In _Neural Information Processing Systems (NIPS)_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling The Knowledge in A Neural Network.  _arXiv preprint arXiv:1503.02531_.
  * Horowitz (2014) Mark Horowitz. 2014.  1.1 Computing’s Energy Problem (and What We Can Do About It).  In _IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC)_.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for Natural Language Understanding.  In _Empirical Methods in Natural Language Processing (EMNLP): Findings_.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency Parsing with A Self-Attentive Encoder.  In _Association for Computational Linguistics (ACL)_.
  * Li et al. (2020) Zhengjie Li, Yufan Zhang, Jian Wang, and Jinmei Lai. 2020.  A Survey of FPGA Design for AI Era.  _Journal of Semiconductors_ , 41(2):021402.
  * Liu et al. (2018) Xuan Liu, Di Cao, and Kai Yu. 2018.  Binarized LSTM Language Model.  In _North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_.
  * Qin et al. (2020) Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. 2020.  Binary Neural Networks: A Survey.  _Pattern Recognition_ , 105:107281.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21:140:1–140:67.
  * Raganato et al. (2020) Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020.  Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation.  In _Empirical Methods in Natural Language Processing (EMNLP): Findings_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural Machine Translation of Rare Words with Subword Units.  In _Association for Computational Linguistics (ACL)_.
  * Song et al. (2021) Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. 2021\.  AdderSR: Towards Energy Efficient Image Super-Resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Su et al. (2018) Jinsong Su, Jiali Zeng, Deyi Xiong, Yang Liu, Mingxuan Wang, and Jun Xie. 2018.  A Hierarchy-to-Sequence Attentional Neural Machine Translation Model.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 26(3):623–632.
  * Tan et al. (2018) Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018.  Deep Semantic Role Labeling with Self-attention.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI)_.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling Task-Specific Knowledge from BERT into Simple Neural Networks.  _arXiv preprint arXiv:1903.12136_.
  * Tay et al. (2021) Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2021\.  Synthesizer: Rethinking Self-Attention for Transformer Models.  In _Proceedings of the 38th International Conference on Machine Learning (ICML)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  In _Advances in Neural Information Processing Systems (NIPS)_.
  * Wang et al. (2016) Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. 2016.  CNNpack: Packing Convolutional Neural Networks in the Frequency Domain.  In _Advances in Neural Information Processing Systems (NIPS)_.
  * Wu et al. (2018) Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. 2018.  Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks.  _Frontiers in Neuroscience_ , 12:331.
  * Yang et al. (2019a) Baosong Yang, Jian Li, Derek F Wong, Lidia S Chao, Xing Wang, and Zhaopeng Tu. 2019a.  Context-aware self-attention networks.  In _Proceedings of the AAAI conference on artificial intelligence (AAAI)_ , pages 387–394.
  * Yang et al. (2018) Baosong Yang, Zhaopeng Tu, Derek F Wong, Fandong Meng, Lidia S Chao, and Tong Zhang. 2018.  Modeling localness for self-attention networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4449–4458.
  * Yang et al. (2019b) Baosong Yang, Longyue Wang, Derek F Wong, Lidia S Chao, and Zhaopeng Tu. 2019b.  Convolutional self-attention networks.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4040–4045.
  * Yim et al. (2017) Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. 2017.  A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * You et al. (2020) Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. 2020.  ShiftAddNet: A Hardware-Inspired Deep Network.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Zhang et al. (2020) Biao Zhang, Deyi Xiong, Jun Xie, and Jinsong Su. 2020.  Neural Machine Translation With GRU-Gated Attention Model.  _IEEE Transactions on Neural Networks and Learning Systems_ , 31(11):4688–4698.
  * Zhuang et al. (2018) Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. 2018.  Discrimination-Aware Channel Pruning for Deep Neural Networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_.

##  Appendix A Energy Ratio Calculation

We calculate the energy cost following the common practice Chen et al. (2020);
You et al. (2020). Note that, we follow suggestions in Song et al. (2021) to
omit the energy calculation of activate functions, such as relu and softmax,
because they are specially designed over some modern AI chips, which requires
far less energy than additive operation.

For the common case where input $\mathbf{X}\in\mathbb{R}^{l_{1}\times d}$ is
projected into $\mathbf{Q}\in\mathbb{R}^{l_{1}\times d}$ with
$\mathbf{W}\in\mathbb{R}^{d\times d}$:

$\displaystyle\mathbf{Q}=\mathbf{X}\mathbf{W}^{\top}+b,$ (11)

the number of multiplicative operations is $l_{1}\times d\times d+l_{1}\times
d=l_{1}d^{2}+l_{1}d$. For the number of additive operations, it is also
$l_{1}d^{2}+l_{1}d$. Specially, if $d\gg 1$, we can omit the term $l_{1}d$ for
simplicity.

We calculate the energy cost from three levels: the alignment calculation
which is used to measure the attention score, the whole attention model which
is the core of our work, and the widely used Transformer Vaswani et al. (2017)
block, which contains a multi-head attention layer and a feedforward layer. We
simply set the sequence length of inputs $\mathbf{X}$ and $\mathbf{Y}$ to $l$.
Then:

#### Alignment Calculation

Two linear prjections are arranged to obtain query and key representations,
yielding $2ld^{2}$ additive/multiplicative operations. Both query and key
representations are used to derive attention logits. In dot-product, the
number of required multiplicative operations is $l\times d\times l=l^{2}d$.
The total numbers of additive/multiplicative operations are both
$2ld^{2}+l^{2}d$.

#### Attention Model

In order to obtain value representations, attention model requires $ld^{2}$
additive/multiplicative operations. Besides, applying weighted sum over value
representations with attention weights requires $l^{2}d$
multiplicative/additive operations. Overall, the numbers of
multiplicative/additive operations in the whole attention model are
$3ld^{2}+2l^{2}d$.

####  Transformer Block

Although representations in multi-head attention are splitted into $h$ heads,
in which dimension is $d_{h}$ ($d=h\times d_{h}$), the number of
multiplicative/additive operations is also $h\times l\times d_{h}\times
l=l^{2}hd_{h}=l^{2}d$. There are one output linear transition for the output
of multiple heads and two additional linear transitions in feedforward layer,
resulting in $ld^{2}+4ld^{2}+4ld^{2}=9ld^{2}$ additional
additive/multiplicative operations. The overall multiplication operations in a
Transformer Block is $9ld^{2}+3ld^{2}+2l^{2}d=12ld^{2}+2l^{2}d$.

Calculation Level | $\Delta_{A}$ | $\Delta_{F}$
---|---|---
Alignment Calculation | 0.45 | 0.05
Attention Model | 34.09 | 33.83
Transformer Block | 83.17 | 83.10
Table 7: Energy consumption ratio (%) at each calculation level on En-Vi
translation task compared to conventional Transformer design.

Same as the steps above, we can calculate the required energy consumption of
other modules. For example, considering our proposed E-ATT, the numbers of
additive and multiplicative operations are $2ld+l^{d}$ and $0$ in alignment
calculation, $ld^{2}+2ld+2l^{2}d$ and $ld^{2}+l^{2}d$ in attention model,
$10ld^{2}+2ld+2l^{2}d$ and $10ld^{2}+l^{2}d$ in Transformer block.

In this way, we can calculate the ratio of energy cost referring to the
statistics in Table 1. For example, the ratio between E-ATT and conventional
attention model on ASIC chip is:

$\displaystyle\Delta_{A}=$
$\displaystyle\frac{0.9(ld^{2}+2ld+2l^{2}d)+3.7(ld^{2}+l^{2}d)}{0.9(3ld^{2}+2l^{2}d)+3.7(3ld^{2}+2l^{2}d)}$
(12)

Similarly, that on FPGA chip is:

$\displaystyle\Delta_{F}=$
$\displaystyle\frac{0.4(ld^{2}+2ld+2l^{2}d)+18.8(ld^{2}+l^{2}d)}{0.4(3ld^{2}+2l^{2}d)+18.8(3ld^{2}+2l^{2}d)}$
(13)

For the IWSLT’15 En-Vi task, with $d$ being 512, the averaged length of
dataset is $\bar{l}=22$. We can get the result $\Delta_{A}$ = 34.09%,
$\Delta_{F}$ = 33.83%, thus the energy reduction ratio is $1-\Delta_{A}$ =
65.91%, $1-\Delta_{F}$ = 66.17%. Similarly, energy consumption ratios at the
level of alignment calculation are 99.55% and 99.95%. Those of Transformer
block are 83.17% and 83.10%, respectively.