# Short-term forecasting of Amazon rainforest fires based on ensemble
decomposition model

Ramon Gomes da Silva1 Matheus Henrique Dal Molin Ribeiro1,2 Viviana Cocco
Mariani3,4 and Leandro dos Santos Coelho1,3
1 Pontifical Catholic University of Parana, Industrial & Systems Engineering
Graduate Program, Curitiba, PR – Brazil
2 Federal Technological University of Parana, Department of Mathematics, Pato
Branco, PR – Brazil
3 Federal University of Parana, Dept. of Electrical Eng., Curitiba, PR –
Brazil
4 Pontifical Catholic University of Parana, Mechanical Engineering Graduate
Program, Curitiba, PR – Brazil  The authors would like to thank National
Council of Scientific and Technological Development of Brazil - CNPq (Grants
number: 303906/2015-4-PQ and 404659/2016-0-Univ), PRONEX ‘Fundação Araucária’
042/2018, and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior \-
Brazil – CAPES (Finance Code: 001) for financial support of this work.

###### Abstract

Accurate forecasting is important for decision-makers. Recently, the Amazon
rainforest is reaching record levels of the number of fires, a situation that
concerns both climate and public health problems. Obtaining the desired
forecasting accuracy becomes difficult and challenging. In this paper were
developed a novel heterogeneous decomposition-ensemble model by using Seasonal
and Trend decomposition based on Loess in combination with algorithms for
short-term load forecasting multi-month-ahead, to explore temporal patterns of
Amazon rainforest fires in Brazil. The results demonstrate the proposed
decomposition-ensemble models can provide more accurate forecasting evaluated
by performance measures. Diebold-Mariano statistical test showed the proposed
models are better than other compared models, but it is statistically equal to
one of them.

##  1 Introduction

The Amazon rainforest represents over half of the planet’s remaining
rainforests and comprises the largest and most biodiverse tract of tropical
rainforest in the world. Brazil contains the majority part of the Amazon
rainforest, with 60% of it (see illustration in Figure 1), followed by Peru
with 13%, Colombia with 10%, and with minor amounts in Venezuela, Ecuador,
Bolivia, Guyana, Suriname, and French Guiana.

Figure 1: Brazilian Amazon rainforest area

In the last years, the fire spots in Amazon has brought attention to world
climate discussions and also public health problems in the region [1].
Therefore, many studies have been developed discussing the impacts of the fire
and how to control it, however, there is a lack of studies to forecast the
number of fire spots in the region. Seasonal and Trend decomposition based on
Loess (STL) showed to be effective and popular in time series forecasting,
mainly to seasonal series. Moreover, ensemble learning approaches increasing
the accuracy and efficiency of the models which learns different data
patterns, combining potentialities of each base (weak) model making them
efficient.

In this respect, the objective of this paper is to develop a heterogeneous
ensemble learning model by using STL decomposition with machine learning
models algorithms to train the components generated by decomposition for fire
spots cases forecasting in the Brazilian Amazon rainforest multi-month-ahead
(one and two-months ahead). The time series is split into three different
components and train each component of the STL decomposition with $k$-Nearest
Neighbor ($k$-NN), Multivariate Adaptive Regression Splines (MARS), Support
Vector Regression (SVR) with kernel Radial, Boosted Generalized Linear Model
(GLMBoost), Cubist and Multi-Layer Perceptron (MLP). Then, the predictions of
the components are summed giving one thousand combinations by Grid-Search. The
combination chosen for one-month-ahead by its performance is composed by SVR
with kernel Radial, MARS, and GLMBoost, respectively, and for two-months-ahead
is composed by Cubist, $k$-NN and MLP, respectively.

The main contributions of this study are: (i) to develop a novel heterogeneous
decomposition-ensemble learning model by using STL decomposition combined with
machine learning models algorithms; (ii) to compare between heterogeneous and
homogeneous decomposition and non-decomposed models to evaluate the
performance of the proposed model; (iii) to realize comparisons between the
predictions of the models of the multi-step-ahead; and (iv) to present a
relevant novel to the environmental field, as well to time series forecasting
multi-step-ahead using STL decomposition-ensemble model.

The remainder of this paper is structured as follows. Section 2 describes the
dataset adopted and presents the methodology applied in this paper. Section 3
shows the results and discussions. Finally, Section 4 concludes the paper.

##  2 Material & Methodology

The dataset analyzed in this paper refers to active fire spots in Brazilian
Amazon rainforest area collected from the database of Fire Program of Brazil’s
National Institute for Space Research (INPE) [2]. The dataset consists of 255
observations monthly from June 1998 to August 2019. To determine system lags
to create inputs, simulations were conducted varying the lag from 1 to 10,
which lag equals to 10 (where the training data started since April 1999)
presented better results. Moreover, the dataset was split into training and
test sets in the proportion of 70% and 30%, respectively. In Table 1 is
presented a summary of the statistical indicators of the dataset, which are
the Maximum (Max), Minimum (Min), Mean, Median and Standard Deviation (Std).

Variable | Samples | # of | Statistical indicator
---|---|---|---
Samples | Max | Min | Mean | Median | Std
Fire spots | All set | 245 | 73141 | 70 | 9427 | 3131 | 13249.01
| Training set | 171 | 73141 | 70 | 10273 | 3175 | 14788.60
| Test set | 74 | 36569 | 379 | 7473 | 2792 | 8477.56
Table 1: Summary of the statistical indicators of the dataset

The experiments discussed in this paper were developed using R statistical
software. The dataset was shared into three components: seasonal, trend and
remainder, based on STL decomposition [3], illustrated on Figure 2.

Figure 2: STL decomposition

Each component was trained using the following six models: $k$-NN [4], MARS
[5], SVR with kernel Radial [6], GLMBoost [7], Cubist [8] and MLP [9]. A time-
slice validation and Principal Components Analysis (PCA) [10] pre-processing
were applied in the training process. For the simulations with STL
decomposition, PCA needed 6 components to capture 95 percent of the variance,
while for simulations without STL decomposition PCA needed 7 components to
capture 95 percent of the variance of the signal.

To forecast one (1) and two-months-ahead (2) the applied structures are
defined as,

$\displaystyle\hat{y}(t+h)$ $\displaystyle=$ $\displaystyle
f\left\\{y(t+h-1),y(t+h-2),\ldots,y(t+h-n_{y})\right\\},$ (1)
$\displaystyle\hat{y}(t+h)$ $\displaystyle=$ $\displaystyle
f\left\\{\hat{y}(t+h-1),y(t+h-2),\ldots,y(t+h-n_{y})\right\\},$ (2)

where $f$ is a function that maps the fire spots data, $\hat{y}(t+h)$ is the
forecast fire spots in horizon $h=1,2$ at time $t$ ($1,\ldots,245$),
$y(t+h-1)$, $\hat{y}(t+h-1)$ are the previous observed and predicted fire
spots, $y(t+h-n_{y})$ is the previous observed fire spots at lag $n_{y}=10$.

The predictions obtained from the trained components are recomposed by a
simple summation of the extrapolated trend and seasonal components. For one-
month-ahead, the chosen combination, due to its performance, was SVR with
kernel Radial for the seasonal component, MARS for trend component and
GLMBoost for the remainder, denominated STL-Ensemble-1. And for two-months-
ahead, the chosen combination was composed by Cubist, $k$-NN and MLP, for
seasonal, trend and remainder component, respectively, namely STL-Ensemble-2.

Moreover, the performance of the proposed STL-Ensemble-1 is compared to the
combination with all components trained using only MARS, SVR with kernel
radial, and GLMBoost namely STL-MARS, STL-SVR, and STL-GLMBoost, and it is
compared with the same algorithms applied in dataset without decomposition. In
the same way, STL-Ensemble-2 is compared with the combinations using $k$-NN,
Cubist, and MLP, namely STL-KNN, STL-CUBIST, and STL-MLP, as well as the
models without decomposition. In order, these comparisons were made to show
that the heterogeneous decomposition performed better than homogeneous and
even without decomposing.

To evaluate the performance of the various forecasting ensemble models,
including the proposed model, measures were adopted. The metrics used in this
study are relative root mean square error (RRMSE) (3) and coefficient of
determination ($R^{2}$) (4), where, $y_{i}$ is the observed value,
$\hat{y}_{i}$ is the predicted value and $\overline{y}$ is the mean of
$y_{i}$, that are identified by,

$\text{RRMSE}=\frac{\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}}}{\frac{1}{n}\sum_{i=1}^{n}y_{i}},$
(3)

$R^{2}=1-\frac{\sum_{i=1}^{n}\left[y_{i}-\hat{y}_{i}\right]^{2}}{\sum_{i=1}^{n}\left[y_{i}-\overline{y}\right]^{2}}$
(4)

Moreover, a Diebold-Mariano test was conducted to evaluate the statistical
difference of the proposed models from the other models.

##  3 Results

The hyperparameters of the models used in this paper, presented in Table 2,
were defined by using a Grid Search. In Figure 3 are depicted the STL-
Ensemble-1 model fire spots prediction values (blue dotted line), STL-
Ensemble-2 model fire spots prediction values (red dotted line) with the
observed time series (black line). The analysis reveals that in both training
and test datasets the models learned and performed accurately following series
trends. Furthermore, as reported in Table 3 the performance measures also
reflect which in the horizon of one-month-ahead STL-Ensemble-1 model results
(in bold) clearly performed better than models cited in Section 2, and in the
horizon of two-months-ahead STL-Ensemble-2 (in bold) had a better performance
compared to the other models, both exhibiting the importance of combined
methodology. Diebold-Mariano test (DM) was conducted comparing the STL-
Ensemble-1 with each of the other models of its horizon, the same for STL-
Ensemble-2, presented in Table 4. DM test shows that STL-Ensemble-1 and STL-
Ensemble-2 are better than the models compared, but STL-Ensemble-1 error is
statistically equal to STL-MARS error once its $p$-value is greater than 0.1.

Model | Components | Control Hyperparameters | Model | Components | Control Hyperparameters
---|---|---|---|---|---
# of Neighbors | # of Boosting Iterations
$k$-NN | Seasonal | 9 | GLMBoost | Seasonal | 250
| Trend | 5 |  | Trend | 50
| Remainder | 11 |  | Remainder | 100
| Nondecomposed | 7 |  | Nondecomposed | 250
|  | # of Terms | Product Degree |  |  | # of Committees | # of Instances
MARS | Seasonal | 9 | 1 | Cubist | Seasonal | 1 | 5
| Trend | 3 | 1 |  | Trend | 1 | 5
| Remainder | 9 | 1 |  | Remainder | 1 | 0
| Nondecomposed | 2 | 1 |  | Nondecomposed | 10 | 0
|  | Sigma | Cost | Kernel |  |  | # of Hidden Units
SVR | Seasonal | 0.0996 | 4 | Radial | MLP | Seasonal | 9
| Trend | 0.9212 | 2 | Radial |  | Trend | 9
| Remainder | 0.0881 | 0.25 | Radial |  | Remainder | 1
| Nondecomposed | 0.2105 | 2 | Radial |  | Nondecomposed | 1

Table 2: Control hyperparameters for the models Figure 3: Fire spots
prediction for STL-Ensemble model

Horizon | Metric | STL-Ensemble-1 | STL-MARS | STL-SVR | STL-GLMBoost | MARS | SVR | GLMBoost
---|---|---|---|---|---|---|---|---
One-month- | RRMSE | 0.3197 | 0.3903 | 0.3470 | 0.5961 | 0.6661 | 0.6540 | 0.6395
ahead | $R^{2}$ | 0.9132 | 0.8957 | 0.9000 | 0.6818 | 0.5911 | 0.6742 | 0.5542
|  | STL-Ensemble-2 | STL-KNN | STL-CUBIST | STL-MLP |  $k$-NN | CUBIST | MLP
Two-months- | RRMSE | 0.6311 | 1.4046 | 3.5327 | 1.7753 | 0.6641 | 0.7482 | 0.8575
ahead | $R^{2}$ | 0.8186 | 0.7236 | 0.5425 | 0.3504 | 0.6490 | 0.5856 | 0.0581

Table 3: Performance measures of the models on test set

Horizon |  |  | STL-Ensemble-1
---
vs
STL-MARS
| STL-Ensemble-1
---
vs
STL-SVR
| STL-Ensemble-1
---
vs
STL-GLMBoost
| STL-Ensemble-1
---
vs
MARS
| STL-Ensemble-1
---
vs
SVR
| STL-Ensemble-1
---
vs
GLMBoost
One-month- | DM | -0.8191 | -1.8836 | -3.2631 | -3.4462 | -2.5824 | -3.8813
ahead |  $p$-value | 0.4154 | 0.0636 | 0.0016 | 0.0009 | 0.0118 | 0.00024
Horizon |  |  | STL-Ensemble-2
---
vs
STL-KNN
| STL-Ensemble-2
---
vs
STL-CUBIST
| STL-Ensemble-2
---
vs
STL-MLP
| STL-Ensemble-2
---
vs
$k$-NN
| STL-Ensemble-2
---
vs
CUBIST
| STL-Ensemble-2
---
vs
MLP
Two-months- | DM | -6.1844 | -9.3799 | -7.8515 | -5.1354 | -8.3402 | -17.1809
ahead |  $p$-value | 9.8633e-10 | 6.4036e-20 | 1.2964e-14 | 3.5252e-07 | 3.1566e-16 | 1.1203e-56

Table 4: Diebold-Mariano tests

##  4 Conclusion

In this paper, heterogeneous decomposition-ensemble (STL-Ensemble-1 and STL-
Ensemble-2) models were proposed to short-term forecasting of Brazilian Amazon
rainforest fire cases multi-month-ahead. The STL-Ensemble-1 model was compared
with six different approaches, namely as STL-MARS, STL-SVR, STL-GLMBoost,
MARS, SVR and GLMBoost, while STL-Ensemble-2 model was compared to others,
namely as STL-KNN, STL-CUBIST, STL-MLP, $k$-NN, Cubits, and MLP. The models
were evaluated by RRMSE and $R^{2}$ performance measures, and our proposed
models yield better forecast accuracy than other models. Further, the Diebold-
Mariano test was conducted and indicated that STL-Ensemble-1 and STL-
Ensemble-2 were more accurate than the compared, yet the STL-Ensemble-1 error
is statistically equal to STL-MARS error. Finally, we can conclude that our
proposed models can help government agency policies to predict and prevent
fire spots in Amazon rainforest. As future research propositions are intended
to perform a multi-step-ahead forecast with a wider horizon, using different
algorithms and decomposition methods.

## References

  * [1] F. Machado-Silva, R. Libonati, T. F. M. de Lima, R. B. Peixoto, J. R. A. França, M. A. F. M. Magalhães, F. L. M. Santos, J. A. Rodrigues, and C. C. DaCamara.  Drought and fires influence the respiratory diseases hospitalizations in the Amazon.  Ecological Indicators, 109:105817, 2020.
  * [2] Brazil.  Programa de queimadas - Instituto Nacional de Pesquisa Espaciais (INPE). [Fire program - National Institute for Space Research (INPE)], 2019\.  http://queimadas.dgi.inpe.br/queimadas/portal. Access in 22 oct. 2019 (in Portuguese).
  * [3] R. B. Cleveland, W. S. Cleveland, J. E McRae, and I. Terpenning.  STL: a seasonal-trend decomposition.  Journal of Official Statistics, 6(1):3–73, 1990.
  * [4] T. M. Cover and P. Hart.  Nearest neighbor pattern classification.  IEEE Transactions on Information Theory, 13(1):21–27, 1967.
  * [5] J. H. Friedman.  Multivariate adaptive regression splines.  The Annals of Statistics, 19(1):1–67, 1991.
  * [6] C. Cortes and V. Vapnik.  Support-vector networks.  Machine Learning, 20(3):273–297, 1995.
  * [7] P. Bühlmann and B. Yu.  Boosting with the L2 loss: regression and classification.  Journal of the American Statistical Association, 98(462):324–339, 2003.
  * [8] J. R. Quinlan.  Proceedings of the 5th Australian Joint Conference on Artificial Intelligence.  In Learning with Continuous Classes, pages 343–348. World Scientific, 1992.
  * [9] D. E. Rumelhart, J. L. McClelland, and P.D.P. Research Group.  Parallel Distributed Processing, Vol. 1: Foundations, vol. 1.  The MIT Press, Cambridge, MA, London, England, 1987.
  * [10] J. E. Jackson.  A user’s guide to principal components.  John Wiley & Sons, New York, USA, 1991.


# Neural News Recommendation with Collaborative News Encoding and Structural
User Encoding

Zhiming Mao1,2, Xingshan Zeng3, Kam-Fai Wong1,2
1The Chinese University of Hong Kong, Hong Kong, China
2MoE Key Laboratory of High Confidence Software Technologies, China
3Huawei Noah’s Ark Lab, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Automatic news recommendation has gained much attention from the academic
community and industry. Recent studies reveal that the key to this task lies
within the effective representation learning of both news and users. Existing
works typically encode news title and content separately while neglecting
their semantic interaction, which is inadequate for news text comprehension.
Besides, previous models encode user browsing history without leveraging the
structural correlation of user browsed news to reflect user interests
explicitly. In this work, we propose a news recommendation framework
consisting of collaborative news encoding (CNE) and structural user encoding
(SUE) to enhance news and user representation learning. CNE equipped with
bidirectional LSTMs encodes news title and content collaboratively with cross-
selection and cross-attention modules to learn semantic-interactive news
representations. SUE utilizes graph convolutional networks to extract cluster-
structural features of user history, followed by intra-cluster and inter-
cluster attention modules to learn hierarchical user interest representations.
Experiment results on the MIND dataset validate the effectiveness of our model
to improve the performance of news recommendation111Our code is released at
https://github.com/Veason-silverbullet/NNR.

##  1 Introduction

Online news applications, such as CNN News and MSN News, have become more and
more people’s first choices to obtain the latest news (Das et al., 2007). With
a deluge of news generated every day, an efficient news recommendation system
should push relevant news to users to satisfy their diverse personalized
interests (IJntema et al., 2010).

From the perspective of representation learning (Bengio et al., 2013),
existing works mainly study how to effectively encode news and users into
discriminative representations (Okura et al., 2017; Wu et al., 2019a; Wang et
al., 2020). News encoders typically extract semantic representations of news
from the textual spans (e.g., news title and content). User encoders are
employed to learn the representation of a user from her browsing history. News
recommendation models predict the matching probabilities between candidate
news and users by measuring the similarity of their representations.

Figure 1:  (a) An example of user browsing history. (b) An example of news
title-content semantic interaction.

Existing news recommendation models typically encode news title and content
separately and encode users’ browsing histories without explicit structural
modeling. We argue that these encodings restrict the power of the news and
user representations. To enhance news and user encoding, this work is
established based on the two aspects of news and user representation learning:

($1$) Encoding the semantic interaction between news title and content: Title
and content play different roles in news, but they are complementary. News
title distills the information of content, while content interprets the
details of title, as shown in Figure 1(b). Previous works treat news title and
content as two separate textual features, leading to a “semantic encoding
dilemma”. This dilemma is dyadic as: (i) Although a news title is much shorter
than content, the performance of title-encoding is empirically better than
content-encoding (Wu et al., 2020). This can be attributed to the crucial
information that the human-summarized title naturally represents; (ii) News
titles are always subjective and rhetorical to attract potential readers. This
leads to a severe textual data sparsity problem. News titles with unseen
terminology, metaphor and ambiguity make it difficult to comprehend news with
limited title wording (Shree et al., 2019). For example in Figure 1(b), the
word “curse” is a metaphor, which cannot be resolved by the training corpus or
the title itself, due to its unique semantic occurrence. News encoders must
turn to the content to interpret the semantics of the word “curse” (i.e., “has
never made the playoffs” and “mean Alabama is toast”). However, news encoders
proposed by previous works either extract features solely from the title, or
encode title and content separately, then perform concatenation or attention
fusion on them (Zhu et al., 2019; Wu et al., 2019a). Such separate encodings
of title and content without leveraging their semantic interaction are
inadequate for news text comprehension.

($2$) Encoding the user-interest-news correlation with hierarchical cluster-
structure: While a user usually has diverse interests in news topics, her
browsed news with the same topic is often linked by some logical correlation.
For example in Figure 1(a), the news N3, N5 and N6 are labeled as sports news
and logically correlated with the topic “football”, forming a virtual user
interest cluster. None of the single news can precisely represent the overall
user interest in “football”. However, refined user interest in “football” can
be encoded by aggregating the news N3, N5 and N6. With aspects of user
interests encoded within specific clusters, overall user representations can
be aggregated by leveraging the correlation among interest clusters. Previous
works typically formulate user history as an ordered linear sequence of news.
Based on this sequential formulation, recurrent neural networks (Okura et al.,
2017; An et al., 2019) and attention networks (Zhu et al., 2019; Wu et al.,
2019a, b, c) are proposed to encode user history. These encoding methods
viewing user history as a sequence of news cannot explicitly model the
hierarchical user-interest-news correlation. Compared to linear sequences,
hierarchical clusters are more suitable to represent a user’s diverse
interests. User history can be structurally formulated into certain interest
clusters, as correlated news shares information in a specific interest
cluster. Encoding user history with hierarchical cluster-structure is more
precise to represent the correlation of news and user interests.

To address the above issues, in this work, we propose collaborative news
encoding (CNE) and structural user encoding (SUE) to learn semantic-
interactive news representations and hierarchical user representations. We
conduct experiments on the MIND dataset (Wu et al., 2020), showing the
encoding effectiveness of our proposed model. Experiments and further analyses
validate that (i) CNE can enhance news encoding by exploiting the word-level
semantic interaction between news title and content with cross-selective and
cross-attentive mechanisms; (ii) SUE utilizes hierarchical cluster graphs to
model the correlation of a user’s browsed news, which can extract more precise
user interest representations; (iii) our model significantly outperforms
existing state-of-the-art news recommendation models on the real-world MIND
dataset.

##  2 Related Work

News recommendation is not only an important research task in NLP (Wu et al.,
2020) but also a core component of industrial personalized news service (Okura
et al., 2017). Conventional collaborative filtering (CF) approaches (Wang and
Blei, 2011) exploit the interaction relationship between news and users. Since
news only lasts for a short period, CF-based methods suffer from severe cold-
start problem. To tackle this, content-based methods used handcrafted features
to encode news and users (Li et al., 2010; Son et al., 2013; Bansal et al.,
2015). In recent years, deep neural models have achieved superior performance
in news recommendation. Many studies pinpointed that this improvement came
from the fine-grained news and user representations, which were extracted by
deep neural networks (Wu et al., 2019a, c; Wang et al., 2020).

For news representation learning, existing works used convolutional neural
networks (CNN) (An et al., 2019), knowledge-aware CNN (Wang et al., 2018),
personalized attention networks (Wu et al., 2019b), and multi-head self-
attention networks (Wu et al., 2019c) to extract features from news title text
as news representations. Zhu et al. (2019) employed parallel CNNs to encode
news title and content respectively and then concatenated them into a unified
representation. Wu et al. (2019a) encoded news title and content separately
and incorporated them with multi-view attention.

For user representation learning, Okura et al. (2017) used GRU to encode user
history by ordered timestamp. An et al. (2019) utilized RNN to learn short-
term user representations from the browsing history, combined with long-term
user embeddings. Various attention networks are also widely used to attend to
important news in user history (Wu et al., 2019a, b; Zhu et al., 2019). Wu et
al. (2019c) employed multi-head self-attention (Vaswani et al., 2017) to
capture deep interaction of user browsed news. These works formulated user
history as an ordered linear sequence of news, to which recurrent or attention
models were applied without modeling the structural correlation of user
browsed news. Hu et al. (2020) formulated news and user jointly with a
bipartite graph and disentangled user preferences with routing mechanism,
which however implicitly relied on the manually-set latent preference factor.

##  3 Methodology

Our model is composed of the Collaborative News Encoding (CNE) module
presented in Section 3.1 and Structural User Encoding (SUE) module presented
in Section 3.2. CNE and SUE extract representations of candidate news and
users respectively. The overall model architecture is illustrated in Figure 2.
Finally, Section 3.3 will describe the click predictor and details of model
training.

###  3.1 Collaborative News Encoding

####  3.1.1 Cross-selective Encoding

The news encoder is employed to learn semantic news representations from news
title and content. Given the title word sequence
$x^{t}=[x^{t}_{1},x^{t}_{2},...,x^{t}_{N}]$ and content word sequence
$x^{c}=[x^{c}_{1},x^{c}_{2},...,x^{c}_{M}]$, they are mapped to word
embeddings $W^{t}=[w^{t}_{1},w^{t}_{2},...,w^{t}_{N}]$ and
$W^{c}=[w^{c}_{1},w^{c}_{2},...,w^{c}_{M}]$, where $N$ and $M$ are the word
sequence lengths. For simplicity, we only formulate the title encoding part of
our model (denoted by superscript $t$). The content encoding formula is
symmetric as a counterpart (denoted by superscript $c$) and omitted.

First, two parallel bidirectional LSTMs are employed to extract the sequential
features from the title and content word embeddings respectively.

$\overrightarrow{h}^{t}_{i}=\overrightarrow{LSTM}^{t}(w^{t}_{i},\overrightarrow{h}^{t}_{i-1},\overrightarrow{c}^{t}_{i-1})$
(1)
$\overleftarrow{h}^{t}_{i}=\overleftarrow{LSTM}^{t}(w^{t}_{i},\overleftarrow{h}^{t}_{i+1},\overleftarrow{c}^{t}_{i+1})$
(2)

where $\\{h^{t}\\}$ and $\\{c^{t}\\}$ are LSTM hidden states and cell states.
The i-th title sequential feature is fused as
$h^{t}_{i}=[\overrightarrow{h}^{t}_{i};\overleftarrow{h}^{t}_{i}]$, where
$[\cdot;\cdot]$ denotes vector concatenation. We consider the global semantic
information of title (content) preserved in its LSTM cell states and
concatenate the last forward $\overrightarrow{c}^{t}_{N}$ and backward
$\overleftarrow{c}^{t}_{1}$ as the semantic memory vector $m^{t}$.

$m^{t}=[\overrightarrow{c}^{t}_{N};\overleftarrow{c}^{t}_{1}]$ (3)

To facilitate semantic interaction between title and content, we design a
gated cross-selective network, inspired by Geng et al. (2020). Concretely, we
utilize the semantic memory vector $m^{c(t)}$ to perform feature recalibration
(Hu et al., 2018) on the sequential features $\\{h^{t(c)}\\}$ by a sigmoid
gate function. The motivation behind this gate function is to utilize the
memory vector of content (title) $m^{c(t)}$ to cross-select important semantic
information from the $i$-th title (content) sequential feature $h^{t(c)}_{i}$.

$Gate^{t}_{i}=\sigma(W_{g}^{h}h^{t}_{i}+W_{g}^{m}m^{c}+b_{g})$ (4)
$\tilde{h}^{t}_{i}=Gate^{t}_{i}\odot{h^{t}_{i}}$ (5)

where $\sigma$ is sigmoid activation, $\odot$ denotes element-wise
multiplication. $\tilde{h}^{t}_{i}$ is the cross-selective feature of the i-th
title sequential feature $h^{t}_{i}$ interacting with the content memory
vector $m^{c}$. It is the first stage of collaborative title-content semantic
interaction.

####  3.1.2 Cross-attentive Encoding

Based on the cross-selective sequential feature $\\{\tilde{h}\\}$, a two-phase
attention module is designed to learn cross-attentive representations of title
and content. First, we employ self-attention layers to learn the self-
attentive representation of the sequential $\\{\tilde{h}\\}$.

$\alpha^{t}_{self}=softmax(v^{\mathsf{T}}tanh(W\tilde{h}^{t}+b))$
$r^{t}_{self}=\sum_{i=1}^{N}{\alpha^{t}_{self,i}\tilde{h}^{t}_{i}}$ (6)

Then we employ the self-attentive representation $r_{self}$ as a query, and
the $\\{\tilde{h}\\}$ as key-value pairs to build cross-attention
layers222Practically, we employ the scaled dot-product attention proposed by
Vaswani et al. (2017).
$Attention(Q,K)=softmax(\frac{\bar{Q}\bar{K}^{T}}{\sqrt{d}})$, where
$\bar{Q}=QW^{Q}$, $\bar{K}=KW^{K}$. The same attention functions are also
applied in Eq. (10) and (12).. It is the second stage of collaborative title-
content semantic interaction.

$\alpha^{t}_{cross}=Attention(r^{c}_{self},\\{\tilde{h}^{t}\\})$
$r^{t}_{cross}=\sum_{i=1}^{N}{\alpha^{t}_{cross,i}\tilde{h}^{t}_{i}}$ (7)

Figure 2:  The overall architecture of our model. The graph construction is
based on the user history in Figure 1(a).

We compute element-wise summation (denoted as $\oplus$) of the self-attentive
representation $r_{self}$ and cross-attentive representation $r_{cross}$ to
derive the title and content semantic-interactive representations, i.e.,
$r^{t}$ and $r^{c}$. Finally, we concatenate $r^{t}$ and $r^{c}$ as the
collaborative news representation $r^{n}$.

$r^{n}=[r^{t};r^{c}]=[r^{t}_{self}\oplus{r^{t}_{cross}};r^{c}_{self}\oplus{r^{c}_{cross}}]$
(8)

###  3.2 Structural User Encoding

####  3.2.1 Cluster-based Encoding of User History

The user encoder is employed to learn user interest representations from their
browsing histories. To formulate the cluster-structure of user interests, we
construct a hierarchical cluster graph in two steps:

Intra-cluster Subgraph $\mathcal{G}^{1}$. We construct an original cluster
graph333https://en.wikipedia.org/wiki/Cluster_graph with the topic category
label of news (e.g., “Sports” and “Travel” in Figure 1(a)). We build the
subgraph $\mathcal{G}^{1}=(V_{n},E_{n})$ by treating the browsed news as nodes
$\\{V_{n}\\}$ and adding bidirectional edges $\\{E_{n}\\}$ to those nodes,
which share the same category labels. Each news node of $\\{V_{n}\\}$ is
associated with its embedding $r^{n}$ in Eq. (8). Each cluster contains
multiple browsed news with a specific topic, reflecting an aspect of user
interests.

Inter-cluster Subgraph $\mathcal{G}^{2}$. Besides intra-cluster refinement of
user interests, modeling inter-cluster correlation is also essential to
leverage the overall information of user history. For each cluster $C_{i}$ in
$\mathcal{G}^{1}$, we add a new cluster proxy node $V_{p}^{i}$. We build the
subgraph $\mathcal{G}^{2}=(\\{V_{n},V_{p}\\},\\{E_{p}^{1},E_{p}^{2}\\})$ by
adding bidirectional edges $\\{E_{p}^{1}\\}$ to those news nodes $\\{V_{n}\\}$
and proxy nodes $\\{V_{p}\\}$ within the same clusters and fully connecting
$\\{V_{p}\\}$ by bidirectional edges $\\{E_{p}^{2}\\}$. The node embedding of
$\\{V_{p}\\}$ is initiated as zero-embedding $r^{p}$. News node information
among clusters aggregates via cluster proxy nodes.

The hierarchical cluster graph $\mathcal{G}$ consists of intra- and inter-
cluster subgraphs: $\mathcal{G}=\\{\mathcal{G}^{1},\mathcal{G}^{2}\\}$. With
$d$-dimensional node embedding vectors $\\{r^{n}_{i}\\}_{i=1}^{|V_{n}|}$ and
$\\{r^{p}_{i}\\}_{i=1}^{|V_{p}|}$, we define the history feature matrix as
$H^{0}=[r^{n};r^{p}]\in{\mathbb{R}^{(|V_{n}|+|V_{p}|)\times{d}}}$. For graph
$\mathcal{G}$, we denote its normalized adjacency matrix as $\tilde{A}$ and
degree matrix as $\tilde{D}$. We use graph convolutional networks (GCN) (Kipf
and Welling, 2017) to extract structural representations of user history. To
mitigate the over-smoothing issue of deep GCN (Chen et al., 2020), we add
residual connections to adjacent GCN layers, following Li et al. (2019).

$H^{l+1}=ReLU(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}H^{l}W^{l})+H^{l}$
(9)

where $W^{l}$ is a trainable matrix. The GCN extracts structural features on
graph $\mathcal{G}$, refining specific user interest representations within
clusters and aggregating overall user history information among clusters. We
train GCN of $L$ layers and derive the structural user history representation
from the news node embeddings as
$r^{h}=\\{H_{i}^{L}\\}_{i=1}^{|V_{n}|}\in{\mathbb{R}^{|V_{n}|\times{d}}}$.

####  3.2.2 Intra-cluster Attention

Given the $|C|$ interest clusters implied by $|V_{n}|$ user’s browsed news,
there are $|C_{i}|$ news in cluster $C_{i}$. The structural user history
representation $r^{h}$ can be viewed as
$r^{h}=\\{r^{h}_{i}\\}^{|C|}_{i=1}=\Big{\\{}\\{r^{h}_{i,j}\\}^{|C_{i}|}_{j=1}\Big{\\}}^{|C|}_{i=1}$.
To derive intra-cluster features associated with candidate news, we design an
intra-cluster attention layer, regarding the candidate news representation
$r^{n}_{can}$ as a query, and the $j$-th intra-cluster feature $r^{h}_{i,j}$
of cluster $C_{i}$ as a key-value pair.

$\alpha^{intra}_{i}=Attention(r^{n}_{can},\\{r^{h}_{i}\\})$
$r^{intra}_{cl,i}=\sum_{j=1}^{|C_{i}|}{\alpha^{intra}_{i,j}r^{h}_{i,j}}$ (10)

The intra-cluster feature $r^{intra}_{cl,i}$ attends to the node-level
features $\\{r^{h}_{i,j}\\}_{j=1}^{|C_{i}|}$ of cluster $C_{i}$, associated
with the candidate news representation $r^{n}_{can}$. The $r^{intra}_{cl,i}$
refines the $i$-th user interest representation within the cluster $C_{i}$ of
graph $\mathcal{G}$.

####  3.2.3 Inter-cluster Attention

Before inter-cluster modeling, a nonlinear transformation is performed to
project the $r^{intra}_{cl,i}$, which is originally a linear combination of
node-level features in cluster $C_{i}$, into cluster-level feature spaces.

$\tilde{r}^{intra}_{cl,i}=ReLU(\tilde{W}r^{intra}_{cl,i}+\tilde{b})+r^{intra}_{cl,i}$
(11)

To derive inter-cluster features associated with candidate news, we design an
inter-cluster attention layer, regarding the candidate news representation
$r^{n}_{can}$ as a query, and the $i$-th intra-cluster feature
$\tilde{r}^{intra}_{cl,i}$ of graph $\mathcal{G}$ as a key-value pair.

$\alpha^{inter}=Attention(r^{n}_{can},\\{\tilde{r}^{intra}_{cl}\\})$
$r^{inter}_{cl}=\sum_{i=1}^{|C|}{\alpha^{inter}_{i}\tilde{r}^{intra}_{cl,i}}$
(12)

The inter-cluster feature $r^{inter}_{cl}$ attends to the cluster-level
features $\\{\tilde{r}^{intra}_{cl,i}\\}_{i=1}^{|C|}$ of graph $\mathcal{G}$,
associated with the candidate news representation $r^{n}_{can}$. With intra-
cluster and inter-cluster attention, $r^{inter}_{cl}$ hierarchically
aggregates user interest representations within the cluster graph
$\mathcal{G}$. $r^{inter}_{cl}$ is adopted as the user representation $r^{u}$,
i.e., $r^{u}=r^{inter}_{cl}$.

###  3.3 Click Predictor and Model Training

Given the news and user representations $r^{n}$ and $r^{u}$, the click
predictor is employed to predict the probability that user $u$ clicks on news
$n$. Motivated by the previous works (Wang et al., 2018; Wu et al., 2019a), we
compute the dot-product $\hat{y}_{n,u}$ of $r^{n}$ and $r^{u}$, i.e.,
$\hat{y}_{n,u}=\left\langle r^{n},r^{u}\right\rangle$, as the unnormalized
matching score of news $n$ and user $u$.

Following common practice of previous works (Huang et al., 2013; Wang et al.,
2020), we employ negative sampling strategy to model training. For each user
click-impression, i.e., the $i$-th impression log that user $u$ had clicked on
news $n$, we compute its unnormalized matching score as $\hat{y}^{+}_{i}$.
Besides, we randomly sample $K$ pieces of news, which are not clicked by the
user $u$. Unnormalized matching scores $\hat{y}^{-}_{i,j}$ are computed for
these $K$ negative samples, where $j=1,...,K$. By such means, it can be
reformulated as a $(K+1)$-way classification problem. We employ softmax
function to derive the normalized matching probabilities and sum up the
negative log-likelihood of positive samples over the training dataset
$\mathcal{D}$, as model training loss $\mathcal{L}$.

$\mathcal{L}=-\sum_{i=1}^{|\mathcal{D}|}{log\frac{exp(\hat{y}^{+}_{i})}{exp(\hat{y}^{+}_{i})+\sum_{j=1}^{K}{exp(\hat{y}^{-}_{i,j})}}}$
(13)

# users | 200,000 | # users in train set | 189,532
---|---|---|---
# news | 78,520 | # news in train set | 75,858
# training logs | 594,433 | # positive samples | 902,330
Avg. title len | 11.67 | Avg. content len | 41.01
Table 1: Statistics of the $200$K-MIND dataset.

##  4 Experiment Setup

###  4.1 Dataset and Experiment Settings

We conduct experiments on the MIND dataset (Wu et al., 2020). MIND is a large-
scale English news recommendation dataset built from real-world MSN news and
anonymized user click logs444https://msnews.github.io. Since the MIND is quite
large-scale, following existing works (Wu et al., 2019a, b, c; Wang et al.,
2020)555These works used the MSN news dataset with 10K sampled users, as
training on the full MIND dataset with 1 million users is very expensive in
GPU time., we randomly sample $200$K users’ click logs out of $1$ million
users from the user behavior logs of MIND training and validation sets. Since
the MIND test set is not labeled, we half-split the original validation set
into experimental validation and test sets. We employ the abstract column
texts in MIND as the news content texts666Detailed MIND dataset format at
https://github.com/msn
ews/msnews.github.io/blob/master/assets/doc/introduction.md. Detailed
statistics of the $200$K-MIND dataset are shown in Table 1.

We truncate news title and content with the maximum length of $32$ and $128$
respectively. The number of news in user browsing history is capped at $50$.
Following (An et al., 2019; Wu et al., 2019a, b, c), we perform negative
sampling with the sampling ratio $K$ of $4$ (see Section 3.3). The word
embedding is initialized from the pretrained $300$-dimensional Glove embedding
(Pennington et al., 2014). The number of GCN layers in SUE is set as $L=4$
(investigated in Section 5.4). We use Adam optimizer (Kingma and Ba, 2015)
with the learning rate of $1$e-$4$ to train our model with the dropout rate of
$0.2$. The area under the ROC curve (AUC), mean reciprocal rank (MRR), and
normalized discounted cumulative gain (nDCG@$5$ and nDCG@$10$) are adopted as
ranking metrics to evaluate model performance. We set the batch size to $64$
and conduct early stopping if the validation AUC score had not improved over
$5$ epochs. We independently repeat each experiment for $10$ times and report
the average performance scores.

###  4.2 Comparison Methods

We compare our model with state-of-the-art general and news-specific
recommendation methods.

General Recommendation Methods. General methods utilize handcrafted features
to learn news and user representations. We use the TF-IDF features extracted
from news and user history with the one-hot news and user IDs as input
features for the experiments. The general methods include ($1$) LibFM (Rendle,
2012), a factorization machine estimating the sparse feature interaction
between news and users; ($2$) DSSM (Huang et al., 2013), a deep structured
semantic model, regarding the user history as a query and candidate news as
key documents; ($3$) Wide&Deep (Cheng et al., 2016), a framework consisting of
wide channels with a linear model and deep channels with a neural model.

Neural News Recommendation Methods. We compete with the state-of-the-art
neural models, which are specifically designed for news recommendation,
including ($1$) DAE-GRU Okura et al. (2017), encoding news with a denoising
autoencoder and users with a gated recurrent network; ($2$) DFM (Lian et al.,
2018), a deep fusion model using multi-channel inception blocks to capture
various interaction among news features; ($3$) DKN (Wang et al., 2018),
utilizing knowledge-aware CNNs to fuse knowledge encoding and textual encoding
of news title; ($4$) LSTUR (An et al., 2019), encoding news title with a CNN
network, while jointly modeling long-term user preferences and short-term user
interests with a GRU network; ($5$) NAML (Wu et al., 2019a), utilizing CNN
networks to encode title and content texts, while encoding the category and
subcategory topics with dense layers. The text and topic representations are
incorporated by multi-view attention. NAML uses an attention network as its
user encoder; ($6$) NPA (Wu et al., 2019b), attending to important words and
news articles by personalized attention networks built with user embeddings;
($7$) NRMS (Wu et al., 2019c), utilizing effective multi-head self-attention
networks (Vaswani et al., 2017) to extract fine-grained representations from
the news title and user history respectively; ($8$) FIM (Wang et al., 2020),
encoding news titles with dilated convolution networks and modeling the
interaction between candidate news and user history with $3$D convolutional
matching networks.

Variants of Our Model. To further verify the efficacy of our model design, we
also experiment with the ablation variants of our model by respectively
removing the cross-selection module (CNE w/o CS), cross-attention module (CNE
w/o CA), GCN layers (SUE w/o GCN), and hierarchical cluster attention module
(SUE w/o HCA).

##  5 Experiment Results and Analyses

Methods |  AUC |  MRR | nDCG@5 | nDCG@10
---|---|---|---|---
LibFM | 61.16 | 27.88 | 30.06 | 36.44
DSSM | 64.74 | 30.12 | 33.22 | 39.50
Wide&Deep | 64.62 | 29.87 | 32.71 | 39.11
DAE-GRU | 65.98 | 31.48 | 34.93 | 41.12
DFM | 64.63 | 29.80 | 32.82 | 39.29
DKN | 66.20 | 31.25 | 34.23 | 40.92
LSTUR | 68.10 | 32.87 | 36.46 | 42.69
NAML |  68.63† | 33.16 | 36.79 | 43.07
NPA | 67.34 | 32.59 | 35.98 | 42.28
NRMS | 68.61 |  33.46† |  37.02† |  43.30†
FIM | 68.44 | 32.95 | 36.58 | 42.97
CNE-SUE |  69.55⋆ |  33.70⋆ |  37.54⋆ |  43.79⋆
CNE w/o CS | 69.39 | 33.52 | 37.30 | 43.62
CNE w/o CA | 69.48 | 33.61 | 37.39 | 43.68
SUE w/o GCN | 69.31 | 33.48 | 37.25 | 43.53
SUE w/o HCA | 69.40 | 33.52 | 37.37 | 43.65
Table 2: Performance comparison results († denotes the highest baseline
scores, ⋆ denotes that the performance improvements over all baseline methods
are validated by Student’s unpaired t-test with $p$-value $<0.01$).

###  5.1 Main Comparison Results

Table 2 shows the performance comparison results. Our model CNE-SUE achieves
the highest performance consistently in all evaluation metrics. Detailed
observations can be obtained as follows.

General recommendation methods yield much lower performance than most neural
news recommendation methods. This is due to that deep neural models can learn
refined representations adaptively, which are more effective than general
feature engineering with fixed handcrafted features.

In all evaluation metrics, our model CNE-SUE outperforms all comparison
methods by significant margins ($+0.92\%$ AUC, $+0.24\%$ MRR, $+0.52\%$
nDCG@$5$, and $+0.49\%$ nDCG@$10$ compared to the best baseline performance).
This performance improvement derives from the collaborative news
representations and structural user representations extracted by our model.
Specifically, CNE can extract more accurate news semantics by leveraging the
title-content semantic interaction, compared to title-encoding (e.g., LSTUR,
NRMS, and FIM) and separate encodings of title and content (e.g., NAML). SUE
modeling diverse user interests with hierarchical cluster structure is more
powerful than the comparison methods formulating user history as a linear
sequence of news, which employ recurrent neural networks (e.g., LSTUR) or
attention networks (e.g., NPA, NAML, and NRMS).

From table 2, we can observe varying degrees of performance decreases on the
ablation variants compared to our full model. It suggests the usefulness of
different components in our model. CNE w/o CA performs the best among all
variants. This is because the news representations learned by CNE are composed
of self- and cross-attentive representations (refer to Eq. (8)), and the
remaining self-attention can achieve suboptimal performance. Removing GCN
layers leads to the most significant impact on performance, indicating the
efficacy of structural modeling on user history.

###  5.2 Effectiveness of Collaborative Encoding

News Enc. |  AUC |  MRR | nDCG@5 | nDCG@10
---|---|---|---|---
CNN | 68.31 | 33.07 | 36.65 | 42.92
KCNN | 65.27 | 30.72 | 33.67 | 40.28
Per-CNN | 68.37 | 33.12 | 36.68 | 42.90
MHSA | 68.26 | 33.07 | 36.59 | 42.83
NAML |  68.63† |  33.16† |  36.79† |  43.07†
CNE | 69.21 | 33.32 | 37.16 | 43.44
NAML-T | 68.49 | 33.21 | 36.89 | 43.11
NAML-C | 67.22 | 32.08 | 35.62 | 41.76
CNE-T | 68.25 | 33.18 | 36.80 | 43.12
CNE-C | 67.96 | 32.65 | 36.34 | 42.44
Table 3: Ablation study of news encoders (KCNN denotes knowledge-aware CNN in
DKN, Per-CNN denotes CNN with personalized attention in NPA, MHSA denotes
multi-head self-attention networks in NRMS, $*$-T(C) denotes title (content)
encoding only).

We conduct ablation experiments on news encoders. For fair comparison and
excluding the influence of SUE, all ablation models apply the same attention
user encoders. We also examine the title and content encodings. Table 3 shows
the ablation results777We did not include the ablation of FIM, because the FIM
news encoder produces special hierarchical 3D-sized representations, which are
incompatible with other ablation encoders..

From the ablation results, we can observe that CNE significantly outperforms
other existing news encoding methods. NAML is also competitive, as it can
incorporate informative representations from news texts and topic categories.
Table 3 also shows that title-encoding ($*$-T) is much more effective than
content-encoding ($*$-C), though content texts are theoretically more
informative. This confirms the “semantic encoding dilemma” in news encoding
and may explain why many existing works (e.g., LSTUR, NPA, NRMS, and FIM)
employ title-encoding only. Comparing NAML to NAML-T, there is no significant
performance enhancement. In contrast, CNE achieves much higher scores than the
individual title and content encodings (CNE-T and CNE-C). It validates the
necessity of encoding news title and content with word-level semantic
interaction to enhance news representation learning.

###  5.3 Effectiveness of Structural Encoding

User Enc. |  AUC |  MRR | nDCG@5 | nDCG@10
---|---|---|---|---
LSTUR | 68.10 | 32.87 | 36.46 | 42.69
ATT | 68.31 | 33.07 | 36.65 | 42.92
Per-ATT | 66.97 | 32.22 | 35.48 | 41.82
Can-ATT | 68.48 | 33.23 | 36.73 | 43.04
MHSA |  68.75† |  33.34† |  36.97† |  43.26†
SUE | 69.03 | 33.53 | 37.26 | 43.48
Table 4: Ablation study of user encoders (ATT denotes vanilla attention
networks, Per-ATT denotes personalized attention in NPA, Can-ATT denotes
candidate-aware attention in DKN, MHSA denotes multi-head self-attention
networks in NRMS).

We conduct ablation experiments on user encoders. For fair comparison and
excluding the influence of CNE, all ablation models apply the same CNN title
encoders. Table 4 shows the ablation results.

According to Table 4, MHSA performs much better than other baseline user
encoders. This is because MHSA (Vaswani et al., 2017) can model the
correlation of each pair of news in user history. It validates the necessity
of modeling the correlation of historical news in user encoding. Moreover,
Table 4 shows that SUE significantly outperforms MHSA. This is because the
manner of encoding historical news correlation with hierarchical clusters in
SUE is more fine-grained than MHSA. Concretely, modeling intra-cluster news
interaction is more effective to reflect aspects of user interests, while
modeling inter-cluster user interests interaction is more effective to encode
overall user representations. The ablation results indicate that structural
modeling of the hierarchical user-interest-news correlation can effectively
enhance user encoding.

Figure 3:  The performance of our model on validation set with respect to the
number of GCN layers (the trend of ndcg@10 is similar to ndcg@5 and hence
omitted).

###  5.4 Parameter Analysis

We investigate the influence of the number of GCN layers $L$ in our model.
Figure 3 shows the results. The model performance on validation set increases
and reaches a peak, as $L$ increases from $1$ to $4$. This is because equipped
with deeper GCN, the model can capture more fine-grained information of user
browsing behaviors by modeling higher-order interaction of browsed news.
Nevertheless, as $L$ continues to increase, the model performance begins to
decline. This may be because deep GCN always suffers from the over-smoothing
issue (Chen et al., 2020). As GCN becomes too deep, the user history
representations $r^{h}$ tend to be indistinguishable and impair the ultimate
user representations. Herein, $L=4$ is optimal for our model.

(a) Attention weights of our model over the title words.

(b) Attention weights of our model over the content words.

Figure 4:  Attention weight visualization on the news $N6$ (darker colors
denote higher attention weights).

###  5.5 Case Study

We then probe into how our model processes news texts. According to Eq. (6),
(7), and (8), we define the word attention weights of title (content) as
$\alpha^{t(c)}=(\alpha^{t(c)}_{self}+\alpha^{t(c)}_{cross})/2$, where
$\alpha^{t(c)}\in{[0,1]}$. As shown in Figure 4, we visualize our model’s
output title (content) attention weights $\alpha^{t(c)}$ over the title
(content) words of the news $N6$ in Figure 1.

From Figure 4(a), we observe that our model mainly attends to the words
“curse” and “CFP”, which contain the core information of the news $N6$. It
validates that our model can distill the most informative words from the news
title. As the content visualization shown in Figure 4(b), our model mostly
attends to the words “Alabama” and “toast”, which interpret the specific
semantics of the word “curse” in the context of the news $N6$. Besides, our
model also attends to the important contextual words, such as “initial”,
“rankings”, and “trend”. These title and content attention weights indicate
that our model can accurately encode the news $N6$.

Figure 5:  The AUC scores of different models with respect to the number of
user interest clusters.

###  5.6 Analysis on User Interest Modeling

We analyze how our model performs with different numbers of user interest
clusters $|C|$ (refer to Section 3.2). The results are shown in Figure 5. When
$|C|=1$, our model slightly underperforms NAML. This is because our model is
overfitted to represent single user interest with cluster graphs. All models’
performance increases with the growth of $|C|$. This is because the models can
learn more precise user representations as more news information is
incorporated. In cases of $|C|>1$, our model consistently outperforms all
baselines. It validates the usefulness of encoding user history with
hierarchical cluster-structure in cases of modeling diverse user interests.
Moreover, the performance of all models decreases when $|C|$ becomes too large
(i.e., $|C|>8$). This reveals the challenge of predicting a user’s news-
clicking behavior when her browsing history covers too many kinds of news
topics.

##  6 Conclusion

In this work, we present a neural news recommendation model with collaborative
news encoding and structural user encoding. CNE leverages the title-content
semantic interaction to enhance news encoding. SUE exploits the correlation of
browsed news and represents user interests with hierarchical cluster graphs to
enhance user encoding. Experiment results show that our model achieves
significant performance enhancement compared to the existing state-of-the-art
methods. We also further analyze our model and validate its effectiveness.

## Acknowledgements

We appreciate some insightful comments from the anonymous reviewers. The
research described in this paper is partially supported by Hong Kong RGC-GRF
#14204118 and Hong Kong RSFS #3133237.

## References

  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019.  Neural news recommendation with long- and short-term user representations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 336–345, Florence, Italy. Association for Computational Linguistics.
  * Bansal et al. (2015) Trapit Bansal, Mrinal Das, and Chiranjib Bhattacharyya. 2015.  Content driven user profiling for comment-worthy recommendations of news and blog articles.  In _Proceedings of the 9th ACM Conference on Recommender Systems_ , page 195–202, New York, USA. Association for Computing Machinery.
  * Bengio et al. (2013) Y. Bengio, A. Courville, and P. Vincent. 2013.  Representation learning: A review and new perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(8):1798–1828.
  * Chen et al. (2020) Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020.  Measuring and relieving the over-smoothing problem for graph neural networks from the topological view.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(04):3438–3445.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & deep learning for recommender systems.  In _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems_ , page 7–10, New York, NY, USA. Association for Computing Machinery.
  * Das et al. (2007) Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007.  Google news personalization: Scalable online collaborative filtering.  In _Proceedings of the 16th International Conference on World Wide Web_ , page 271–280, New York, NY, USA. Association for Computing Machinery.
  * Geng et al. (2020) Xinwei Geng, Longyue Wang, Xing Wang, Bing Qin, Ting Liu, and Zhaopeng Tu. 2020\.  How does selective mechanism improve self-attention networks? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2986–2995, Online. Association for Computational Linguistics.
  * Hu et al. (2018) J. Hu, L. Shen, and G. Sun. 2018.  Squeeze-and-excitation networks.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7132–7141.
  * Hu et al. (2020) Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, and Ming Zhou. 2020.  Graph neural news recommendation with unsupervised preference disentanglement.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4255–4264, Online. Association for Computational Linguistics.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM International Conference on Information & Knowledge Management_, page 2333–2338, New York, NY, USA. Association for Computing Machinery.
  * IJntema et al. (2010) Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom. 2010.  Ontology-based news recommendation.  In _Proceedings of the 2010 EDBT/ICDT Workshops_ , New York, NY, USA. Association for Computing Machinery.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Li et al. (2019) Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. 2019.  Deepgcns: Can gcns go as deep as cnns?  In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010.  A contextual-bandit approach to personalized news article recommendation.  In _Proceedings of the 19th International Conference on World Wide Web_ , page 661–670, New York, NY, USA. Association for Computing Machinery.
  * Lian et al. (2018) Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. 2018.  Towards better representation learning for personalized news recommendation: a multi-channel deep fusion approach.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 3805–3811. International Joint Conferences on Artificial Intelligence Organization.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017.  Embedding-based news recommendation for millions of users.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , page 1933–1942, New York, NY, USA. Association for Computing Machinery.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Rendle (2012) Steffen Rendle. 2012.  Factorization machines with libFM.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 3(3):57:1–57:22.
  * Shree et al. (2019) Jaya Shree, Emily Liu, Andrew Gordon, and Jerry Hobbs. 2019.  Deep natural language understanding of news text.  In _Proceedings of the First Workshop on Narrative Understanding_ , pages 19–27, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Son et al. (2013) Jeong-Woo Son, A-Yeong Kim, and Seong-Bae Park. 2013.  A location-based news article recommendation with explicit localized semantic analysis.  In _Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’13, page 293–302, New York, NY, USA. Association for Computing Machinery.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Wang and Blei (2011) Chong Wang and David Blei. 2011.  Collaborative topic modeling for recommending scientific articles.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 448–456.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020.  Fine-grained interest matching for neural news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 836–845, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018.  Dkn: Deep knowledge-aware network for news recommendation.  In _Proceedings of the 2018 World Wide Web Conference_ , WWW ’18, page 1835–1844, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 3863–3869. International Joint Conferences on Artificial Intelligence Organization.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Npa: Neural news recommendation with personalized attention.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, page 2576–2584, New York, NY, USA. Association for Computing Machinery.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c.  Neural news recommendation with multi-head self-attention.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6389–6394, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020.  MIND: A large-scale dataset for news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3597–3606, Online. Association for Computational Linguistics.
  * Zhu et al. (2019) Qiannan Zhu, Xiaofei Zhou, Zeliang Song, Jianlong Tan, and Li Guo. 2019.  Dan: Deep attention neural network for news recommendation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):5973–5980.