An Effective Non-Autoregressive Model for Spoken Language Understanding

Lizhi Cheng

Department of Computer Science and Engineering,
Shanghai Jiao Tong University

PR China

Weijia Jia
[1]

BNU-UIC Institute of Artificial Intelligence and Future Networks Beijing Normal University (BNU Zhuhai),
Guangdong Key Lab of AI and Multi-Modal Data Processing,
BNU-HKBU United International College
PR China

Wenmian Yang
[1]

Department of Computer Science and Engineering,
Shanghai Jiao Tong University
PR China

Spoken Language Understanding (SLU), a core component of the task-oriented dialogue system, expects a shorter inference latency due to the impatience of humans. Non-autoregressive SLU models clearly increase the inference speed but suffer uncoordinated-slot problems caused by the lack of sequential dependency information among each slot chunk. To gap this shortcoming, in this paper, we propose a novel non-autoregressive SLU model named Layered-Refine Transformer, which contains a Slot Label Generation (SLG) task and a Layered Refine Mechanism (LRM). SLG is defined as generating the next slot label with the token sequence and generated slot labels. With SLG, the non-autoregressive model can efficiently obtain dependency information during training and spend no extra time in inference. LRM predicts the preliminary SLU results from Transformer's middle states and utilizes them to guide the final prediction. Experiments on two public datasets indicate that our model significantly improves SLU performance (1.5% on Overall accuracy) while substantially speed up (more than 10 times) the inference process over the state-of-the-art baseline.

<concept_desc>Computing methodologies Natural language processing</concept_desc>

[500]Computing methodologies Natural language processing

§ INTRODUCTION

With the widespread application of intelligent voice assistants, e.g., Apple Siri,task-oriented dialogue systems have received more attention. Working as the spoken interface between users and machines, Spoken Language Understanding (SLU) plays a critical role in the task-oriented dialogue system. A typical SLU task mainly includes two subtasks, i.e., Intent Detection (ID) and Slot Filling (SF) <cit.>. Given by an utterance expressed in natural language from the user, ID aims to identify the intent of the user (e.g., GetWeather), and SF aims to fill the slot for each token in the utterance (e.g., location, time). Generally, ID works on sentence-level and is treated as a semantic classification task, while SF is a sequence labeling task focusing on token-level. A simple example of SLU is shown in Figure <ref>.

An example of SLU.

Recent studies find that ID and SF are closely related and positively promote the predicting performance of each other. Specifically, some joint models <cit.> apply a joint loss function to link two tasks, or utilize hidden states of one task in the other <cit.>. <cit.>  propagate the token-level intent results to the SF task, achieving the state-of-the-art (SOTA) performance. However, most of the work relies on the RNN-based framework, which models the entire sequence dependency in an autoregressive way, leading to redundant computation and inevitable high latency.

An intuitive idea for accelerating the predicting process is to utilize the self-attention (i.e., Transformer) <cit.> instead of the RNN as the basic framework in a non-autoregressive way.
Although using a self-attention based framework does increase the calculation speed, this method will reduce the prediction performance of the SF task <cit.>.
The reason is that SF is a sequence labeling task whose results utilize the “Inside–Outside–Beginning (IOB)” tagging format. Therefore, SF heavily depends on the strongness of the sequential dependency information among each slot chunk (proved by <cit.>). However, the slot labels are predicted independently and simultaneously in self-attention based methods, which reduces the strongness of the sequential dependency and causes the uncoordinated slot problem. Figure <ref> shows an example of the uncoordinated slot problem, where  should be followed by , but  is predicted  by mistake. Therefore, how to maintain the non-autoregressive efficiency when inferring while increase the sequential dependency information to avoid the uncoordinated problem in SLU tasks is a major challenge.

An example of uncoordinated slot problem.

Although <cit.> tries to handle this problem via a two-pass mechanism called SlotRefine, this method still has the following weakness:
First, SlotRefine supposes all the uncoordinated slot problems are caused by wrong `I-tags' and only utilize `B-tags' to correct the `I-tags' in its second pass period.
However, as shown in Figure <ref>, when predicting, an incorrect `B-tag' follows by a correct `I-tag' also happens. In this case, SlotRefine cannot utilize `I-tags' to correct the `B-tags'.
Second, the two-pass mechanism needs to characterize text and predict two times in both the training and testing process, which inevitably leads to lower efficiency.
Therefore, how to use the prediction results for interactive feedback to improve the overall performance while avoiding inferring the entire model repeatedly is also a significant challenge.

To solve the above challenges, in this paper, we propose a Layered-Refine Transformer (LR-Transformer) framework for SLU, which containing a Slot Label Generation (SLG) auxiliary task and a Layered Refine Mechanism (LRM) based on the Transformer.
Specifically, SLG is an auxiliary task defined as predicting the next SF label according to the utterance sequence and generated SF labels, which is an autoregressive process like the machine translation. We jointly train the SLG and the original sequence labeling-based SLU tasks by multi-task learning and share their encoder. By SLG, the shared encoder learns more sequential dependency information from the decoder through the cross attention mechanism in the Transformer and further improves sequence labeling-based SLU tasks. Notable, SLG is only employed in the training process and consumes no extra inference time. Thus our model is still a fast non-autoregressive model.
LRM modifies the hidden states between Transformer layers according to the intermediate predicted ID and SF results to help the final prediction.
More specifically, LRM predicts the preliminary results of ID and SF by the hidden states of the former Transformer layer, merging the result embeddings with the hidden states and inputting them into the next Transformer layer. We have further proved in experiments that LRM only needs to be employed once and does not need to be used between every Transformer layer, so the cost is much smaller than running the entire model twice.

The main contributions of this paper are presented as follows:

* We design the SLG as auxiliary multitasking of SLU, which increases the sequential dependency information of the model while consuming no extra inference time.

* We propose the LRM, which improves the overall performance of SLU tasks by the interaction of predicted results between Transformer layers. Compared with running the entire model twice, the cost of LRM is much smaller.

* Experimental results on two public datasets show our model is superior to both existing SOTA autoregressive and non-autoregressive models in terms of speed and performance, indicating that our model has great potential for real-world application.

§ RELATED WORK

In this section, we introduce some joint learning and multi-task learning approaches utilized in SLU.

In SLU, intent detection is usually seen as a semantic classification problem to predict the intent label, and slot filling is mainly regarded as a sequence labeling task.
Early studies <cit.> usually regard ID and SF as two separate tasks and utilize pipeline approaches to manage these two tasks. These methods typically suffer from error propagation due to their independent models and prove less effective than the joint models.

Recently, some work finds that ID and SF are closely related. <cit.> propose a slot-gated model that learns the relationship between intent and slot by the gate mechanism. Inspired by <cit.>, some bi-directional networks are proposed <cit.>, which dig into the correlation between ID and SF deeper and model the relationship between them more explicitly. In the above work, the interrelated connections between ID and SF are established. Besides, <cit.> utilize a hierarchical capsule neural network structure encapsulating the hierarchical relationship among utterance, slot, and intent. <cit.> propose a Stack-Propagation framework that uses result information of ID to guide the SF task but ignores the impact of SF results on the ID task. Although Stack-Propagation reaches the SOTA performance on two SLU datasets, it surfers a long inference latency caused by the heavy and complex framework. <cit.> then concern about the impact of SF results in their model but still suffering the long inference latency. Inspired by <cit.>, <cit.> propose a portable framework RPFSLU, which contains a two-round predicting period. RPFSLU utilizes the semantic information of the first round prediction results to guide the second round prediction by a represent learning process. However, although RPFSLU is portable, it is not a lightweight model due to its two-round prediction process.

The above methods mostly use the autoregressive models (e.g., LSTM and GRU <cit.>) that suffer inevitable long inference latency. Inspired by the well performance of Transformer in orther tasks
<cit.>, <cit.> then propose a non-autoregressive joint model named SlotRefine for SLU, which speeds up the predicting process and encounters the uncoordinated-slot problem. SlotRefine tries to handle this problem by a two-pass refine mechanism and get some effect. However, it still suffers uncoordinated slot problems caused by the wrong `I-tag' and not efficient enough during inference due to its two-pass mechanism that needs to run the whole model twice.

Meanwhile, some work <cit.> tries to enhance the performance via multi-task learning.
These multi-task learning methods link the two tasks implicitly via applying a joint loss function.
<cit.> propose an impressive multi-task learning approach for multi-turn SLU by consolidating context memory with a dialogue logistic inference task called DLI. DLI needs no extra labeled data and only needs to be carried out during training, which inspired us a lot for designing SLG.

Compared with previous work, our LR-Transformer model obtains the sequential dependency information via auxiliary multitasking and utilizes the interaction of the prediction results between ID and SF via the Transformer's middle states.
Thus, our model has high predicting accuracy and fast inference speed, which indicates that our model has great potential for industrial application.

§ METHOD

In this section, we first introduce the basic model of our LR-Transformer. Then, we descirbe the detail of our Slot Label Generation task. Finally, we introduce the Layered Refine Mechanism.

General framework.

§.§ Problem Formulation and Basic Model

In this section, we introduce the problem formulation and describe our basic model in detail.

The input of SLU tasks is an utterance composed by a token sequence $\textbf{X}=\{x_1,...,x_n\}$, where $n$ donates the sequence length.
Given $\textbf{X}$ as input, our tasks are composed of Intent Detection (ID) and Slot Filling (SF).
Specifically, ID is a semantic classification task to predict the intent label for the whole utterance, while SF is a sequence labeling task to give each token in the sequence a slot label. In our model, the intent label and all slot labels are predicted simultaneously.

Following previous non-autoregressive models, we employ the multi-head Transformer encoder <cit.> as our basic model.
<cit.> describe the Transformer framework in great detail, so we do not need to introduce it again. The only difference is that we utilize self-attention with relative position representations <cit.> to address the sequential information.

Akin to the operation in BERT <cit.>, we first insert a special token `CLS' in the beginning of the token sequence, which is utilized to predict the label of the intent. Given the new sequence $\textbf{X}=\{x_{cls},x_1,...,x_n\}$ as input, the Transformer encoder returns the hidden states sequence $\textbf{H}=\{\textbf{h}_{cls},\textbf{h}_1,...,\textbf{h}_n \in \mathbb{R}^{ d_{model} }\}$ as output, where $d_{model}$ is the input and output dimension of the Transformer layer. Then, the prediction of ID and SF are calculated as:

\begin{equation}
\begin{aligned}
\textbf{y}^I &= \mathrm{softmax} ( \textbf{W}^I \cdot \textbf{h}_{cls} + \textbf{b}^I )\\
\textbf{y}^S_j &= \mathrm{softmax} ( \textbf{W}^S \cdot (\textbf{h}_j \oplus \textbf{h}_{cls}) + \textbf{b}^S )
\end{aligned}
\label{eq:getres}
\end{equation}

where $\textbf{y}^I \in \mathbb{R}^{d_{i}} $ and $\textbf{y}^S = \{ \textbf{y}^S_1,..., \textbf{y}^S_n \in \mathbb{R}^{d_{s}} \}$ donate the results of ID and SF, $d_{i}$ and $d_s$ are the the categories of the intent label and slot labels,
$\textbf{W}^I \in \mathbb{R}^{ d_{i} \times d_{model} }$ and $\textbf{W}^S \in \mathbb{R}^{ d_{s} \times 2d_{model}}$ are fully connected matrices,
$\textbf{b}^I \in \mathbb{R}^{d_{i}}$ and $\textbf{b}^S \in \mathbb{R}^{d_{s}}$ are bias vectors, and $\oplus$ donates the concatenation operation.

The objective of our basic model can be formulated as:

\begin{equation}
p\left(\textbf{y}^{I}, \textbf{y}^{S} \mid \textbf{X}\right)=p\left(\textbf{y}^{I} \mid \textbf{X}\right) \cdot \prod_{j}^{n} p\left(\textbf{y}_{j}^{S} \mid \textbf{X}, \textbf{y}^{I}\right)
\end{equation}

The joint loss function of SLU is defined as:

\begin{equation}
\begin{aligned}
\mathcal{L}_{SLU} &=-\log P(\textbf{y}^{I} \mid x_{1}, \ldots, x_{n}) \\
&-\sum^{n}_{j=1} \log P(\textbf{y}_{j}^{S} \mid x_{1}, \ldots, x_{n})
\end{aligned}
\end{equation}

§.§ Slot Label Generation

In this section, we introduce the Slot Label Generation (SLG) task in detail.

As we aforementioned, the SF task heavily depends on the strongness of the sequential dependency information among each slot chunk. Although we have tried to obtain dependency information via self-attention with relative position representations <cit.>, it is not enough for the sequence labeling task like SF.

To obtain more sequential dependency information for the model, we design the SLG as auxiliary multitasking, jointly training the original sequence labeling-based SLU tasks and sharing their encoder.

Specifically, given the token sequence $\textbf{X}=\{x_{cls},x_1,...,x_n\}$ and previous predicted slot labels $\{ \textbf{y}^G_{1},..., \textbf{y}^G_{j-1} \}$, SLG generates the next slot label  $\textbf{y}^G_j$ by an autoregressive way. The objective of our SLG can be formulated as:

\begin{equation}
\begin{aligned}
p( \textbf{y}^G \mid \textbf{X})= \prod_{j=1}^{n} p(\textbf{y}_{j}^{G} \mid \textbf{X}, \textbf{y}_{1}^{G},...,\textbf{y}_{j-1}^{G}  )
\end{aligned}
\end{equation}

where $\textbf{y}^G = \{ \textbf{y}^G_1,..., \textbf{y}^G_n \in \mathbb{R}^{d_{s}} \}$.

In practice, we extend our basic model with a Transformer decoder <cit.> to construct the model architecture of SLG. The complete framework of SLG works as a sequence to sequence (seq2seq) model. Since the Transformer-based seq2seq model has been widely used, we do not describe it in more detail.

Moreover, to enhance the prediction consistency between the SLU task and the SLG task, we further design a consistency loss function based on the cross entropy.
The final loss function of SLG is defined as:

\begin{equation}
\begin{aligned}
\mathcal{L}_{SLG}
&= (1-\alpha) \cdot -\log P( \textbf{y}^{G} \mid\textbf{X} ) +\alpha \cdot H(  \textbf{y}^{G} ,\textbf{y}^{S} )\\
&= (\alpha-1) \cdot \log P( \textbf{y}^{G} \mid\textbf{X} ) +\alpha \cdot H(  \textbf{y}^{G} ,\textbf{y}^{S} )
\label{eq_slgloss}
\end{aligned}
\end{equation}

where $H$ is the cross-entropy function, $\alpha$ is a hyper-parameter and $\textbf{y}^{S}$ is the predicted labels of SF obtained by Eq.<ref>.

By SLG, the shared encoder can learn more sequential dependency information from the decoder through the cross attention mechanism in Transformer and further improve sequence labeling-based SLU tasks.

The loss function of overall multi-task learning is defined as:

\begin{equation}
\mathcal{L}= \mathcal{L}_{SLU}+\lambda \mathcal{L}_{SLG}
\label{eq:jloss}
\end{equation}

where $\lambda$ is a hyper-parameter.

Notably, the SLG task is only carried out during the training process and
costs no extra time for inference. Therefore, our SLU model is still a non-autoregressive model, which is very efficient.

§.§ Layered Refine Mechanism

In this section, we introduce the Layered Refine Mechanism (LRM) for the Transformer in detail.

LRM architecture.

Previous work <cit.> has widely proved that ID and SF are closely related. Taking advantage of the correlation between these two tasks, especially utilizing one task's results in the other, can effectively enhance the overall performance.
However, since the non-autoregressive approach predicts the results of ID and SF simultaneously, we can not directly employ these results in a one-pass prediction process like Stack-Propagation <cit.> do.
Although we can utilize a two-pass mechanism to generate the first-pass results and guide the second-pass prediction via them, it is a trade-off between autoregression and non-autoregression, which costs much time.

Considering Transformer contains a multi-layer architecture, and each encoder layer of the Transformer has the same structure. We design the LRM which works between two Transformer layers and utilizes middle states of the Transformer to guide the final prediction.

Specifically, we first predict a preliminary results of ID $\tilde{\textbf{y}}^I$ and SF $\tilde{\textbf{y}}^S=\{ \tilde{\textbf{y}}^S_1,...,\tilde{\textbf{y}}^S_n \}$ by Eq.<ref> according to hidden states $\textbf{H}^k=\{\textbf{h}^k_{cls},\textbf{h}^k_1,...,\textbf{h}^k_n \}$ from the $k$-th Transformer layer.
Then, we embed $\tilde{\textbf{y}}^I$ into $\textbf{e}^{I} \in \mathbb{R}^{d_e}$ and $\tilde{\textbf{y}}^S=\{ \tilde{\textbf{y}}^S_1,...,\tilde{\textbf{y}}^S_n \}$ into $\textbf{e}^{S}=\{ \textbf{e}^S_1,...,\textbf{e}^S_n \in \mathbb{R}^{d_e}  \}$ by the embedding layer, respectively, where $d_e$ is the embedding size and we set $d_e=d_{model}$ in this paper.

Since the SF returns a sequence of result embedding vectors, we further calculate the weighted  average of those embedding vectors via an attention mechanism  to obtain an utterance-level result embedding vector $\textbf{e}^S_0 \in \mathbb{R}^{d_e}$ by

\begin{equation}
\textbf{e}^S_0=\sum_{j=1}^{n} \alpha_{j} \cdot \textbf{e}^S_j
\end{equation}

where $\alpha_j$ is the weight of $\textbf{e}^S_j$ obtained by

\begin{equation}
\alpha_{j}=\frac{\operatorname{exp}(\textbf{e}^S_{j})}{\sum_{k=1}^{n} \operatorname{exp}(\textbf{e}^S_{k})}
\end{equation}

By the above calculation, we obtain result embedding vectors $\textbf{e}^{I}$, $\textbf{e}^{S}$ and $\textbf{e}^{S}_0$, which contain semantic information from the preliminary results of ID and SF. Subsequently, we merge these vectors with the former output and obtain a new hidden states sequence  $\textbf{H}'=\{\textbf{h}'_{cls},\textbf{h}'_1,...,\textbf{h}'_n \}$ by

\begin{align}
\textbf{h}_{cls}'&= \textbf{h}^k_{cls} + \textbf{e}^{I} + \textbf{e}^{S}_0\\
\textbf{h}_j'&= \textbf{h}^k_j+\textbf{e}^{S}_j
\end{align}

We use $\textbf{H}'$ as input of the $k+1$-th Transformer layer.

LRM can incorporate the bidirectional semantic information from one task to the other by propagating the combination of former output and preliminary results so that  ID and SF become more accurate.
The complete Markov chain process can be simplified as follow:

\begin{equation}
\begin{aligned}
p(&\textbf{y}^{I}, \textbf{y}^{S}  \mid \textbf{X}) =p(\tilde{\textbf{y}}^{I} \mid \textbf{X}) \cdot  p(\tilde{\textbf{y}}^{S} \mid \textbf{X}, \tilde{\textbf{y}}^{I})\\
&\cdot  p(\textbf{y}^{I} \mid \textbf{X}, \tilde{\textbf{y}}^{I}, \tilde{\textbf{y}}^{S} ) \cdot  p(\textbf{y}^{S} \mid \textbf{X}, \tilde{\textbf{y}}^{I}, \tilde{\textbf{y}}^{S}, \textbf{y}^{I} )
\end{aligned}
\end{equation}

LRM is a portable plugin for the Transformer, which can be utilized between two Transformer encoder layer intervals. Actually, using LRM once in the whole Transformer structure is enough because it already considers the interaction between ID and SF. On the contrary, overusing LRM  will make the Transformer layers lose their own feature and negatively affect the final performance because we use the SLU classifier in LRM. Also, LRM is not suitable for the autoregressive SLG task because it uses all SLU prediction labels simultaneously.

Besides, the network structure of LRM is mainly composed of a fully connected layer as the SLU classifier and an embedding layer for result embedding. Therefore, our LRM is very lightweight and consumes very little inference time.

§ EXPERIMENT

In this section, we demonstrate the effectiveness of .
We first introduce datasets, the necessary hyper-parameters, and the baselines used in our experiments.
Then, we compare the performance of our framework with baselines and analyze the experiment results.
Subsequently, we analyze the error caused by the uncoordinated problem and carry out an ablation study to verify the effeteness of SLG and LRM.
Finally, we combine our model with the pre-trained model and analyze the effect.

§.§ Experimental Settings and Baselines

Dataset                     ATIS          SNIPS

Vocabulary Size             722           11241

Avg. tokens per utterance        11.28         9.05

Intent categories                 21            7

Slot categories                  120           72

Training set size                4478          13084

Validation set size              500           700

Test set size                     893           700

Dataset statistics.

To evaluate the efficiency of our proposed model, we conduct experiments on two public datasets, i.e., ATIS (Airline Travel Information Systems <cit.>) and SNIPS (collected by Snips personal voice assistant <cit.>). Compared
with ATIS, the SNIPS dataset is more complex due to its large vocabulary size, cross-domain intents, and more out-of-vocabulary words.
The statistics of ATIS and SNIPS are shown in Table <ref>.

Evaluation Metrics:
Following previous work, we evaluate the SLU performance of ID by accuracy and the performance of SF by the F1 score.
Besides, we utilize overall accuracy to indicate the proportion of utterance in the corpus whose slots and intent are both correctly predicted. Usually, a higher intent accuracy and F1 score also lead to higher overall accuracy. However, this does not always happen, e.g., when the prediction contains more mistakes, but most mistakes are from the same utterances.

Set up:
Following previous work, we use Adam <cit.> to optimize the parameters in our model and adopted the suggested learning rate of 0.001. The batch size is set to 32 according to the size of training data.

When tuning hyper-parameters, we repeat the model 5 times and select the parameters with the best average performance on the validation set as the optimal.

We first select the hyper-parameters used in our basic model. To select Transformer input and output size $d_{model}$, and the size of inner-layer in the feed-forward network of Transformer (we call it $d_{ff}$ in the following part), we leverage the grid search. Specifically, we determine $d_{model}$ in the range of $\{128,\ 256,\ 512 \}$ and $d_{ff}$ in the range of $\{ 128,\ 256,\ 512,\ 768\}$. We finally choose $d_{model}$ as 128 and $d_{ff}$ as 512 as the optimal. For other hyper-parameters of Transformer, following <cit.>, we set both encoder and decoder layers as 6, the number of attention heads as 8, and the dropout ratio as 0.3. We utilize LRM once between the second and the third Transformer layer.

Then, we choose the hyper-parameters $\alpha$ used in Eq.<ref> and $\lambda$ used in Eq.<ref>. We first fix $\lambda$ as 1 and select $\alpha$ in the range of (0,0.5] with the step 0.05. Subsequently, with the selected $\alpha$, we select $\lambda$ in range of (0,1] with the step 0.25. We finally get the optimal when $\alpha$  is 0.35 and $\lambda$ is 0.75. We will introduce the influence of $\lambda$ and $\alpha$ in our ablation study.

2*Model      3c|ATIS        3cSNIPS

2-7     1cIntent     1cSlot     Overall     1cIntent     1cSlot     Overall

7cAutoregressive Models

Joint Seq <cit.>              92.6           94.2              80.7            96.9           87.3              73.2

Attention-BiRNN <cit.>       91.1           94.2              78.9            96.7           87.8              74.1

Slot-Gated <cit.>        93.6           94.8              82.2            97.0           88.8              75.5

SF-ID <cit.>           97.8           95.8              86.8            97.4           92.2              80.6

Stack-Propagation <cit.>               96.9           95.9              86.5            98.0           94.2              86.9

7cNon-autoregressive Models

SlotRefine <cit.>      97.1           96.0              86.9            97.4           93.5              84.4

Basic model      96.8          95.2          85.6          96.1          92.8          82.1

98.2$\uparrow$      96.1$\uparrow$      87.2$\uparrow$      98.4$\uparrow$      94.8$\uparrow$      88.4$\uparrow$

SLU performance on ATIS and SNIPS datasets.
The numbers with $\uparrow$ indicate that the improvement of our model over all baselines is statistically significant with $p < 0.05$ under t-test.

We compare our model with the existing baselines, including:

* Joint Seq <cit.>: A GRU <cit.> based model with a multi-task modeling approach.

* Attention-BiRNN <cit.>: A LSTM <cit.> based encoder-decoder model with an intent attention mechanism.

* Slot-gated <cit.>: A LSTM based joint model together with a slot-gated mechanism as a special gate function.

* SF-ID <cit.>: A LSTM based joint model with cross-impact calculating between two tasks.

* Stack-Propagation <cit.>: A LSTM based joint model with stack-propagation framework and token-level ID. This model has already guided SF by ID results, which is the state-of-the-art of the joint model.

* Basic model: The encoder of Transformer framework with relative position representations <cit.>.

* SlotRefine <cit.>: A Transformer based non-autoregressive model with a two-pass refine mechanism.

For Joint Seq, Attention BiRNN, Slot-gated, SF-ID, and Stack-Propagation, we adopt the reported results from <cit.>.
For SlotRefine <cit.>), since the benchmark in their original paper is calculated nonstandardly according to their open-source code, we re-implemented the model (all hyper-parameters strictly identical as <cit.>) and obtained the results. Note that, for SlotRefine, basic model, and LR-Transformer, we repeat the experiment  5 times and report the average as the final results.

§.§ Result and Analysis

In this section, we show the results of our experiments and do some analysis.

SLU Performance:
The experiment results of the proposed models on ATIS and SNIPS datasets are shown in Table <ref>.

The results show that our model significantly outperforms all the baselines and achieves the best performance in all three metrics.
Compared with the prior non-autoregressive model SlotRefine, our model enhances the performance by 1.1%(ID), 0.1%(SF), and 0.3%(Overall) on ATIS and 1.0%(ID), 1.3%(SF), and 4.0%(Overall) on SNIPS. Compared with the SOTA baseline Stack-Propagation,  also achieve improvement by 1.3%(ID), 0.2%(SF), and 0.7%(Overall) on ATIS and 0.4%(ID), 0.6%(SF), and 1.5%(Overall) on SNIPS.
This indicates the effectiveness of our .

Notably, without SLG and LRM, our basic model performs worse than both SlotRefine and Stack-Propagation, but  outperforms both of them with SLG and LRM. We attribute this enhancement to the fact that our SLG task effectively obtains the sequential dependency information, and LRM directly takes the explicit result information into consideration, which grasps the relationship between the intent and slots. We will conduct experiments for the ablation study in section 4.4 to further verify this idea.

Speed Up:
The inference time of SLU models is shown in Table <ref>. All the models in this experiment are conducted with a single TITAN Xp GPU.
From the table, we can obviously find that our model achieves significant speedup ($\times$10.53 on ATIS; $\times$10.24 on SNIPS) against the autoregressive SOTA model Stack-Propagation because all slot labels are calculated simultaneously in our non-autoregressive method.

2*Model     2c|ATIS              2cSNIPS
1cLatency      Speedup     1cLatency      Speedup

Stack-Propagation          138.12ms       1.00$\times$      143.44ms      1.00$\times$

SlotRefine                 32.04ms        4.31$\times$      41.12ms       3.48$\times$

w/o LRM        12.93ms       10.68$\times$      13.82ms      10.38$\times$

13.11ms       10.53$\times$      14.01ms      10.24$\times$

Latency of SLU models. “Latency” is the average inference time without minibatching. “Speedup” is compared against the existing SOTA model Stack-Propagation <cit.>.

More importantly, compared with the existing non-autoregressive model SlotRefine, our model reduces nearly 60% inference latency on SNIPS and more than 65% on ATIS.
The reason for this phenomenon is that SlotRefine needs to run the whole model twice, including the embedding layer, Transformer, and the classifier. As a comparison, our model is lightweight and obtains the final SLU results with a one-period prediction. Thus, although our model contains more transformer layers, it is still significantly faster than SlotRefine.

From Table <ref>, we can also find that utilizing LRM
consumes only 3% extra inference time compared to without LRM.
This indicates that LRM is a lightweight approach, which generates negligible time cost.

§.§ Error Analysis

In this section, we will analysis the error caused by uncoordinated slots.

The number of uncoordinated slots on the validation set of SNIPS during training.

Case study for the SF task.

We first visualize the number decrease of uncoordinated slots in the training process. As shown in Figure <ref>, the number of uncoordinated slots drops slow and inefficient for the basic model.
For our LR-Transformer, the number of uncoordinated slots drops significantly faster. Ten epochs training of our model is already better than 100 epochs training of the basic model on this problem.
Compared with SlotRefine, our model also drops much faster and achieves better convergence after 5 epochs of training.

Model      slot error       Unc. error    BI error    IB error

Basic model             169        57(33.7%)      31        26

SlotRefine                 127        28(22.0%)      8         20

LR-Transformer           117       13(11.1%)      7       6

The statistics of slot error on the validation set of SNIPS after training 100 epochs.

To further analyze the error in SF tasks in detail, we show the statistics of slot error (i.e., incorrect slots) on the validation set of SNIPS after training 100 epochs. Specifically, we define the errors caused by the uncoordinated slot problem as "Unc. error". The uncoordinated slots includes two cases, i.e., correct `B-tag' followed wrong `I-tag' and wrong `B-tag' following correct `I-tag'. We define the first case as "BI error" and the second case as "IB error.", respectively.

The statistics show that the uncoordinated slot problem composes a big part of all slot errors. Without any approach to solving this problem, our basic model encounters 57 uncoordinated slots, composing 33.7% of all slot errors. In these uncoordinated slots, 31 are caused by the "BI error," while 26 are caused by the "IB error." The proportion of the "BI error" and the "IB error" is almost close.

Compared with the basic model, our LR-Transformer reduces 44 uncoordinated slots. The proportion of uncoordinated slots in all incorrect slot labels drops from 33.7% to 11.1%. Moreover, our LR-Transformer efficiently reduces both the BI error and the IB error simultaneously. The reducing proportion between the BI error and the IB error of our model is also close. This is mainly because our SLG effectively obtains sequential dependency while LRM considers both B-tag slots and I-tag slots.
As a comparison, SlotRefine correct most uncoordinated slots caused by the "BI error" but still suffering "IB error." For SlotRefine, the "IB error" proportion is much higher than "BI error."

2*Model      3c|ATIS        3cSNIPS

2-7     1cIntent     1cSlot     Overall     1cIntent     1cSlot     Overall

Basic model <cit.>       96.8          94.8          85.3          96.1          92.8          82.1

w/o SLG       $97.7_{0.047}$          $95.6_{0.123}$           $86.1_{0.082}$           $98.0_{0.094}$           $93.6_{0.125}$           $84.9_{0.216}$

w/o LRM*        $97.0_{0.094}$           $95.9_{0.169}$           $86.2_{0.216}$           $97.6_{0.124}$           $94.1_{0.249}$           $86.3_{0.205}$

w/o LRM        $97.1_{0.047}$          $95.9_{0.094}$           $86.3_{0.094}$           $97.7_{0.047}$          $94.2_{0.047}$           $86.5_{0.094}$

*     ${98.1}_{0.047}$      ${96.1}_{0.163}$      ${87.1}_{0.141}$      ${98.7}_{0.081}$      ${94.6}_{0.169}$      ${88.2}_{0.169}$

$\textbf{98.2}_{0.047}$      $\textbf{96.1}_{0.047}$     $\textbf{87.2}_{0.081}$      $\textbf{98.8}_{0.047}$      $\textbf{94.8}_{0.094}$      $\textbf{88.4}_{0.081}$

Performance (mean and standard deviation of the model repeated 5 times) comparison of each module in our model. Models with * represent not using consistency loss in Eq.<ref>, i.e., $\alpha = 0$ in Eq.<ref>.

We provide an example for the case study. In Figure <ref>, we notice that the basic model suffers a serious uncoordinated slot problem, including both two cases, i.e., "BI error" and "IB error," respectively.
SlotRefine solves "BI error" but predicts incorrect slot labels in the "IB error" case due to the error propagation from the wrong B-tag label. Our model solves problems of both cases and predicts a correct slot label sequence.

Above all, our LR-Transformer indeed remedies the problem of the uncoordinated slots, leading to better performance on SF.

§.§ Ablation Study

In this section, we do an ablation study to verify the effectiveness of SLG and LRM in detail.

Effect of SLG:
As shown in Table <ref>, compared with the basic model, the  w/o LRM (i.e., basic model + SLG) enhances the performance with a large margin. Specifically, SLG brings enhancement by 0.3% for ID, 1.1% for SF, and 1.0% for overall on ATIS while 1.6% for ID, 1.4% for SF, and 4.4% for overall on SNIPS.
We attribute this improvement to the autoregressive structure of SLG, which brings sequential solid dependency information, making prediction more accurate.

Moreover, comparing LR-Transformer with LR-Transformer* and LR-Transformer w/o LRM with LR-Transformer w/o LRM*, we find that utilizing consistency loss function in Eq.<ref> brings a slight enhancement. More importantly, the consistency loss effectively reduces the standard deviation, especially on SF and overall. This is mainly because utilizing consistency loss can enhance the prediction consistency between the SLU task and the SLG task, which makes the model more stable. Moreover, compared with SF, the standard deviation on ID tasks is much lower. We consider the reason is ID contains fewer categories and easier to predict.

Impact of $\alpha$ on the SF performance on the SNIPS validation set.

We further conduct experiments to study the impact of $\alpha$ on the SF performance. In this experiment, we keep the other settings and change the $\alpha$ from 0 to 0.5. According to the results shown in Figure <ref>, we find that the F1 scores change little when $\alpha$ increases from 0 to 0.35, but the stand deviation drops continuously. Then, when $\alpha$ is larger than 0.4, both the F1 score and the stand deviation get worse rapidly. We consider the large weight of $\alpha$ makes the wrong prediction in SLU excessively affect the SLG task and bring negative effect since the predicted labels of SLU are not the same as the ground-truth labels.

Impact of $\lambda$ on the SF performance on the SNIPS validation set.

Similar to $\alpha$, we also conduct experiments to study the impact of $\lambda$ on the SF performance. As shown in Figure <ref>, the F1 score is enhanced notably when $\lambda$ changed from 0 to 0.25, which indicates the effect of our SLG module. Then, the F1 score continues to rise until $\lambda$ reaches 0.75 and begins to drop with the increase of $\lambda$. This observation indicates that we should make a suitable balance between the SLU and SLG  in our final framework.

2*Model      3c|ATIS        3cSNIPS

2-7     1cIntent     1cSlot     Overall     1cIntent     1cSlot     Overall

BERT-SLU <cit.>           97.5           96.1              88.2            98.6           97.0              92.8

BERT + Stack-Propagation <cit.>               97.5           96.1              88.6            99.0           97.0              92.9

BERT + SlotRefine  <cit.>      97.7           96.1              88.6            99.0           97.0              92.9

98.2          96.1          87.2          98.4          94.8          88.4

BERT+SLG      98.3$\uparrow$      96.2$\uparrow$      88.7$\uparrow$      99.1$\uparrow$      97.1$\uparrow$      93.1$\uparrow$

SLU performance of Bert-based models on ATIS and SNIPS datasets. The numbers with $\uparrow$ indicate that the improvement of our model over all baselines is statistically significant with $p < 0.05$ under t-test.

Effect of LRM:
By comparing  w/o SLG (i.e., basic model + LRM) with the basic model, we find that LRM enhances the performance by 0.9%(ID), 0.8%(SF), and 0.8%(Overall) on ATIS and 1.9%(ID), 0.8%(SF), and 2.8%(Overall) on SNIPS, which shows that utilizing LRM alone still enhances SLU performance on both two tasks. We attribute the improvement to the direct utilization of ID and SF preliminary results. The semantic information from both slot labels and ID labels enhances the performance when predicting.

SF performance comparison of using LRM in different Transformer encoder intervals on SNIPS validation set. 1-2 represents the interval of the 1st and the 2nd Transformer encoder layer, and so on.

As we introduced in section <ref>, LRM can be utilized in each two Transformer encoder layer intervals. Motivated by finding the best place to operate LRM, we conduct the experiment to compare SF performance on the validation set of SNIPS when using LRM in different Transformer encoder intervals. The experiment result is shown in Figure <ref>.

SF performance comparison of different LRM usage count on SNIPS validation set.

We further conduct experiments to evaluate SF performance for different usage count of LRM. As shown in Figure <ref>, the F1 score on the validation set achieves a satisfying level when employing LRM once or twice and drops significantly with increasing usage count.
This phenomenon indicates that employing LRM one time is enough, and overuse LRM will negatively influence SLU prediction. We have introduced the reason in section 3.3, using LRM once in the whole Transformer structure already considers the interaction between ID and SF. Overusing LRM  will make the Transformer layers lose their own feature and negatively affect the final performance due to the utilization of the SLU classifier in LRM.

§.§ Effect of Pretraining

In this section, we conduct experiments to evaluate the ability of our model to combining the pretrained model.

Recently, some work builds their models based on large-scale pre-trained model BERT <cit.>, which utilized billions of external corpus and tremendous model parameters. Since the number of BERT parameters is much more than ours, it is unfair to compare the performance of our model with them directly. Thus, we also combine our approaches with BERT to highlight the effectiveness of .

To make better use of the pre-trained results of BERT, we retain the weights of the original BERT model and do not change the inner structure of BERT. Therefore, we only combine our SLG module with BERT. In practice, we employ BERT as the encoder and utilize the same Transformer decoder as we introduced in section 3.2. Then we fine-tune the complete model on the SLU dataset.

As shown in Table <ref>, our BERT+SLG outperforms all previous BERT-based models on all evaluation metrics. Compared with BERT-SLU, SLG brings an enhancement of 0.8%(ID), 0.1%(SF), and 0.5%(Overall) on ATIS dataset and 0.5%(ID), 0.1%(SF), and 0.3%(Overall) on SNIPS dataset. All of these improvements of our model are statistically significant with $p<0.05$ under t-test. The results show that our SLG approach is still useful for BERT-based models. This phenomenon also indicates that our SLG module is portable and effective.

2*Model     2c|ATIS              2cSNIPS
1cLatency      Speedup     1cLatency      Speedup

BERT+Stack-Propagation          220.11ms       1.00$\times$      225.87ms      1.00$\times$

BERT-SLU            48.90ms       4.50$\times$      49.59ms      4.55$\times$

BERT+SlotRefine                 97.81ms        2.25$\times$      99.19ms       2.27$\times$

13.11ms       -      14.01ms      -

BERT+SLG         48.89ms       4.50$\times$      49.61ms      4.55$\times$

Latency of Bert-based SLU models. “Latency” is the average inference time without minibatching. “Speedup” is compared against the existing SOTA model BERT+Stack-Propagation.

We also record the inference time for each SLU model when employing BERT.
As shown in Table <ref>, our BERT+SLG also achieves significant speedup ($\times$4.50 on ATIS; $\times$4.55 on SNIPS) against the autoregressive SOTA model Stack-Propagation. Meanwhile, comparing with the existing non-autoregressive model SlotRefine, our model can reduce 50% inference time, as our model does not need to run BERT two times.
It is worth noting that our model consumes almost the same time as BERT-SLU <cit.> because our SLG module works only in the training period and wastes no extra time during inference.

From Table <ref>, we further find that the improvement of BERT-based models on the SNIPS dataset is much more obvious than that on the ATIS dataset. Considering that SNIPS contains much more Out-of-Vocabulary (OOV) tokens, we attribute the improvement of BERT to alleviating the OOV problem by the WordPiece encoding <cit.>. Besides, although BERT can bring enhancement, the trade-off for this enhancement is 3.77 times of inference latency when comparing BERT+SLG with LR-Transformer in Table <ref>.

To sum up, our designed SLG task is well combined with the pre-trained model BERT, and BERT+SLG outperforms all existing models on both performance and inference latency.

§ CONCLUSION

In this paper, we propose a fast and accurate non-autoregressive model: . To address the sequential dependency information among tokens, we design the SLG task, which effectively enhances the performance of SF through multi-task learning and costs no extra inference time.
We also design a Layered Refine Mechanism, which guides the final prediction via the interaction of predicted results between Transformer layers. LRM explicitly introduces the correlation between ID and SF into the model with a little cost of time.
Experiments on two public datasets indicate that our model significantly improves performance while substantially accelerate the inference speed.

§ ACKNOWLEDGEMENTS

This work is supported by Guangdong Key Lab of AI and Multi-modal Data Processing, Chinese National Research Fund (NSFC) Project No. 61872239; BNU-UIC Institute of Artificial Intelligence and Future Networks funded by Beijing Normal University (Zhuhai) and AI-DS Research Hub, BNU-HKBU United International College (UIC), Zhuhai, Guangdong, China.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Bai, Zhou, Zhang, and ZongBai
et al2019]
authorpersonHe Bai, personYu Zhou,
personJiajun Zhang, and personChengqing Zong.
Memory Consolidation for Contextual Spoken Language
Understanding with Dialogue Logistic Inference. In
booktitleProceedings of the 57th Annual Meeting of the
Association for Computational Linguistics. pages5448–5453.

[Chen, Zhuo, and WangChen
et al2019]
authorpersonQian Chen, personZhu Zhuo,
and personWen Wang. year2019.
Bert for joint intent classification and slot
journalarXiv preprint arXiv:1902.10909

[Chen, Hakanni-Tür, Tur, Celikyilmaz, Guo,
and DengChen et al2016]
authorpersonYun-Nung Chen, personDilek
Hakanni-Tür, personGokhan Tur, personAsli
Celikyilmaz, personJianfeng Guo, and personLi
Deng. year2016.
Syntax or semantics? knowledge-guided joint
semantic frame parsing. In booktitle2016 IEEE Spoken
Language Technology Workshop (SLT). IEEE, pages348–355.

[Cheng, Yang, and JiaCheng
et al2021]
authorpersonLizhi Cheng, personWenmian
Yang, and personWeijia Jia. year2021.
A Result based Portable Framework for Spoken
Language Understanding. In booktitle2021 IEEE International
Conference on Multimedia and Expo (ICME). IEEE.

[Cho, van Merriënboer, Gulcehre, Bahdanau,
Bougares, Schwenk, and BengioCho et al2014]
authorpersonKyunghyun Cho, personBart van
Merriënboer, personCaglar Gulcehre, personDzmitry
Bahdanau, personFethi Bougares, personHolger
Schwenk, and personYoshua Bengio.
Learning Phrase Representations using RNN
Encoder–Decoder for Statistical Machine Translation. In
booktitleProceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP).

[Coucke, Saade, Ball, Bluche, Caulier, Leroy,
Doumouro, Gisselbrecht, Caltagirone, Lavril, et alCoucke
et al2018]
authorpersonAlice Coucke, personAlaa Saade,
personAdrien Ball, personThéodore Bluche,
personAlexandre Caulier, personDavid Leroy,
personClément Doumouro, personThibault
Gisselbrecht, personFrancesco Caltagirone,
personThibaut Lavril, et al
Snips voice platform: an embedded spoken language
understanding system for private-by-design voice interfaces.
journalarXiv preprint arXiv:1805.10190

[Devlin, Chang, Lee, and ToutanovaDevlin
et al2019]
authorpersonJacob Devlin, personMing-Wei
Chang, personKenton Lee, and personKristina
Toutanova. year2019.
BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. In
booktitleProceedings of the 2019 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers).

[Goo, Gao, Hsu, Huo, Chen, Hsu, and ChenGoo
et al2018]
authorpersonChih-Wen Goo, personGuang Gao,
personYun-Kai Hsu, personChih-Li Huo,
personTsung-Chieh Chen, personKeng-Wei Hsu, and
personYun-Nung Chen. year2018.
Slot-gated modeling for joint slot filling and
intent prediction. In booktitleProceedings of the 2018
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 2 (Short Papers).

[Haihong, Niu, Chen, and SongHaihong
et al2019]
authorpersonE Haihong, personPeiqing Niu,
personZhongfu Chen, and personMeina Song.
A novel bi-directional interrelated model for joint
intent detection and slot filling. In booktitleProceedings
of the 57th Annual Meeting of the Association for Computational
Linguistics. pages5467–5471.

[Hakkani-Tür, Tür, Celikyilmaz, Chen,
Gao, Deng, and WangHakkani-Tür et al2016]
authorpersonDilek Hakkani-Tür,
personGökhan Tür, personAsli Celikyilmaz,
personYun-Nung Chen, personJianfeng Gao,
personLi Deng, and personYe-Yi Wang.
Multi-domain joint semantic frame parsing using
bi-directional rnn-lstm.. In booktitleProceedings of The
17th Annual Meeting of the International Speech Communication Association
(INTERSPEECH 2016). pages715–719.

[Hemphill, Godfrey, and DoddingtonHemphill
et al1990]
authorpersonCharles T Hemphill, personJohn J
Godfrey, and personGeorge R Doddington.
The ATIS spoken language systems pilot corpus. In
booktitleSpeech and Natural Language: Proceedings of a
Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.

[Hochreiter and SchmidhuberHochreiter and
authorpersonSepp Hochreiter and
personJürgen Schmidhuber. year1997.
Long Short-Term Memory.
journalNeural Computation (year1997),

[Kingma and BaKingma and Ba2014]
authorpersonDiederik P Kingma and
personJimmy Ba. year2014.
Adam: A method for stochastic optimization.
journalarXiv preprint arXiv:1412.6980

[Kurata, Xiang, Zhou, and YuKurata
et al2016]
authorpersonGakuto Kurata, personBing
Xiang, personBowen Zhou, and personMo Yu.
Leveraging Sentence-level Information with Encoder
LSTM for Semantic Slot Filling. In booktitleProceedings
of the 2016 Conference on Empirical Methods in Natural Language Processing.

[Li, Du, Zhou, Zhou, Zeng, Xiao, and ZhuLi
et al2021a]
authorpersonBei Li, personQuan Du,
personTao Zhou, personShuhan Zhou,
personXin Zeng, personTong Xiao, and
personJingbo Zhu. year2021a.
ODE Transformer: An Ordinary Differential
Equation-Inspired Model for Neural Machine Translation.
journalarXiv preprint arXiv:2104.02308

[Li, Wang, Liu, Du, Xiao, Zhang, and ZhuLi
et al2021b]
authorpersonBei Li, personZiyang Wang,
personHui Liu, personQuan Du, personTong
Xiao, personChunliang Zhang, and personJingbo
Zhu. year2021b.
Learning Light-Weight Translation Models from Deep
Transformer. In booktitleProceedings of the AAAI Conference
on Artificial Intelligence, Vol. volume35.

[Liu and LaneLiu and Lane2016]
authorpersonBing Liu and personIan Lane.
Attention-Based Recurrent Neural Network Models for
Joint Intent Detection and Slot Filling. In
booktitleInterspeech 2016. pages685–689.

[Liu, Lin, Jia, Zhou, and ZhaoLiu
et al2020]
authorpersonTianyi Liu, personXiangyu Lin,
personWeijia Jia, personMingliang Zhou, and
personWei Zhao. year2020.
Regularized Attentive Capsule Network for
Overlapped Relation Extraction. In booktitleProceedings of
the 28th International Conference on Computational Linguistics.

[Liu, Meng, Zhang, Zhou, Chen, and XuLiu
et al2019]
authorpersonYijin Liu, personFandong Meng,
personJinchao Zhang, personJie Zhou,
personYufeng Chen, and personJinan Xu.
CM-Net: A Novel Collaborative Memory Network for
Spoken Language Understanding. In booktitleProceedings of
the 2019 Conference on Empirical Methods in Natural Language Processing and
the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP). pages1050–1059.

[Mesnil, Dauphin, Yao, Bengio, Deng,
Hakkani-Tur, He, Heck, Tur, Yu, et alMesnil et al2014]
authorpersonGrégoire Mesnil, personYann
Dauphin, personKaisheng Yao, personYoshua Bengio,
personLi Deng, personDilek Hakkani-Tur,
personXiaodong He, personLarry Heck,
personGokhan Tur, personDong Yu, et al
Using recurrent neural networks for slot filling in
spoken language understanding.
journalIEEE/ACM Transactions on Audio, Speech, and
Language Processing (year2014), pages530–539.

[Peng and YaoPeng and Yao2015]
authorpersonBaolin Peng and personKaisheng
Yao. year2015.
Recurrent neural networks with external memory for
language understanding.
journalarXiv preprint arXiv:1506.00195

[Pentyala, Liu, and DreyerPentyala
et al2019]
authorpersonShiva Pentyala, personMengwen
Liu, and personMarkus Dreyer.
Multi-Task Networks With Universe, Group, and Task
Feature Learning. In booktitleProceedings of the 57th
Annual Meeting of the Association for Computational Linguistics.

[Qin, Che, Li, Wen, and LiuQin
et al2019]
authorpersonLibo Qin, personWanxiang Che,
personYangming Li, personHaoyang Wen, and
personTing Liu. year2019.
A Stack-Propagation Framework with Token-Level
Intent Detection for Spoken Language Understanding. In
booktitleProceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLP-IJCNLP).

[Qin, Liu, Wanxiang Che, and LiuQin
et al2021]
authorpersonLibo Qin, personTailu Liu,
personBinging Kang Wanxiang Che, and personTing
Liu. year2021.
A Co-Interactive Transformer for Joint Slot Filling
and Intent Detection. In booktitleICASSP.

[Ren, Liu, Tan, Zhao, Zhao, and LiuRen
et al2020]
authorpersonYi Ren, personJinglin Liu,
personXu Tan, personSheng Zhao, personZhou
Zhao, and personTie-Yan Liu. year2020.
A Study of Non-autoregressive Model for Sequence
journalarXiv preprint arXiv:2004.10454

[Schuster and NakajimaSchuster and
authorpersonMike Schuster and
personKaisuke Nakajima. year2012.
Japanese and korean voice search. In
booktitle2012 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). IEEE, pages5149–5152.

[Shaw, Uszkoreit, and VaswaniShaw
et al2018]
authorpersonPeter Shaw, personJakob
Uszkoreit, and personAshish Vaswani.
Self-Attention with Relative Position
Representations. In booktitleProceedings of the 2018
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 2 (Short Papers).

[Tur and De MoriTur and De Mori2011]
authorpersonGokhan Tur and personRenato
De Mori. year2011.
booktitleSpoken language understanding: Systems for
extracting semantic information from speech.
publisherJohn Wiley & Sons.

[Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and PolosukhinVaswani et al2017]
authorpersonAshish Vaswani, personNoam
Shazeer, personNiki Parmar, personJakob Uszkoreit,
personLlion Jones, personAidan N Gomez,
personŁukasz Kaiser, and personIllia
Polosukhin. year2017.
Attention is all you need. In
booktitleAdvances in neural information processing
systems. pages5998–6008.

[Wu, Ding, Lu, and XieWu
et al2020]
authorpersonDi Wu, personLiang Ding,
personFan Lu, and personJian Xie.
SlotRefine: A Fast Non-Autoregressive Model for
Joint Intent Detection and Slot Filling. In
booktitleProceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP).

[Yang, Jia, Zhou, and LuoYang
et al2019]
authorpersonWenmian Yang, personWeijia Jia,
personXIaojie Zhou, and personYutao Luo.
Legal Judgment Prediction via Multi-Perspective
Bi-Feedback Network. In booktitleProceedings of the
Twenty-Eighth International Joint Conference on Artificial Intelligence
(IJCAI-19). pages4085–4091.

[Yao, Peng, Zhang, Yu, Zweig, and ShiYao
et al2014]
authorpersonKaisheng Yao, personBaolin
Peng, personYu Zhang, personDong Yu,
personGeoffrey Zweig, and personYangyang Shi.
Spoken language understanding using long short-term
memory neural networks. In booktitle2014 IEEE Spoken
Language Technology Workshop (SLT). IEEE, pages189–194.

[Zhang, Li, Du, Fan, and YuZhang
et al2019]
authorpersonChenwei Zhang, personYaliang
Li, personNan Du, personWei Fan, and
personPhilip S Yu. year2019.
Joint Slot Filling and Intent Detection via Capsule
Neural Networks. In booktitleProceedings of the 57th Annual
Meeting of the Association for Computational Linguistics.

[Zhang and WangZhang and Wang2016]
authorpersonXiaodong Zhang and
personHoufeng Wang. year2016.
A Joint Model of Intent Determination and Slot
Filling for Spoken Language Understanding.. In
booktitleProceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence. pages2993–2999.


# Wikipedia graph mining: dynamic structure of collective memory

Volodymyr Miz, Kirell Benzi, Benjamin Ricaud, and Pierre Vandergheynst  Ecole
Polytechnique Fédérale de LausanneLausanneSwitzerland<EMAIL_ADDRESS>
(2018)

###### Abstract.

Wikipedia is the biggest encyclopedia ever created and the fifth most visited
website in the world. Tens of millions of people surf it every day, seeking
answers to various questions. Collective user activity on its pages leaves
publicly available footprints of human behavior, making Wikipedia an excellent
source for analysis of collective behavior.

In this work, we propose a new method to analyze and retrieve collective
memories, the way social groups remember and recall the past. We use the
Hopfield network model as an artificial memory abstraction to build a
macroscopic collective memory model. To reveal memory patterns, we analyze the
dynamics of visitors activity on Wikipedia and its Web network structure. Each
pattern in the Hopfield network is a cluster of Wikipedia pages sharing a
common topic and describing an event that triggered human curiosity during a
finite period of time. We show that these memories can be remembered with good
precision using a memory network. The presented approach is scalable and we
provide a distributed implementation of the algorithm.

Collective memory, Graph Algorithm, Hopfield Network, Wikipedia, Web Logs
Analysis

††copyright: acmcopyright††doi: 10.475/123_4††isbn:
123-4567-24-567/08/06††conference: ; August 19–23; London, United
Kingdom††journalyear: 2018††price: 15.00††ccs: Information systems Wrappers
(data mining)††ccs: Information systems Web log analysis††ccs: Networks
Network dynamics††ccs: Human-centered computing Wikis

##  1\. Introduction

Over recent years, the Web has significantly affected the way people learn,
interact in social groups, store and share information. Apart from being an
essential part of modern life, social networks, online services, and knowledge
bases generate a massive amount of logs, containing traces of global online
activity on the Web. A large-scale example of such publicly available
information is the Wikipedia knowledge base and its history of visitors
activity. This data is a great source for collective human behavior analysis
at scale. Due to this reason, the analysis of the Wikipedia in this area has
become popular over the recent years (Tinati et al., 2016), (García-Gavilanes
et al., 2017), (Kanhabua et al., 2014).

Collective memory (Halbwachs, 2013) is an interesting social phenomenon of
human behavior. Studying this concept is a way to enhance our understanding of
a common view of events in social groups and identify the events that
influence remembering of the past. Early research on collective memory relied
on interviews and self-reports that led to a limited number of subjects and
biased results (Stone et al., 1999). The availability of the Web activity data
opened new opportunities toward systematic studies at a much larger scale
(Ferron, 2012), (Graus et al., 2017), (García-Gavilanes et al., 2017).
Nonetheless, the general nature of collective memory formation and its
modeling remain open questions. Can we model collective and individual memory
formation similarly? Is it possible to find collective memories and behavior
patterns inside a collaborative knowledge base? In this work, we adopt a data-
driven approach to shed some light on these questions.

An essential part of the Web visitors activity data is the underlying human-
made graph structure that was initially introduced to facilitate navigation.
The combination of the activity dynamics and structure of Web graphs inspires
an idea of their similarity to biological neural networks. A good example of
such network is the human brain. Numerous neurons in the brain constitute a
biological dynamic neural network, where dynamics are expressed in terms of
neural spikes. This network is in charge of perception, decision making,
storing memories, and learning.

During learning, neurons in our brain self-organize and form strongly
connected groups called neural assemblies (Allport, 1985). These groups
express similar activation patterns in response to a specific stimuli. When
learning is completed, and the stimuli applied once again, reactions of the
assemblies correspond to consistent dynamic activity patterns, i.e. memories.
Synaptic plasticity mechanisms govern this self-organization process.

Hebbian learning theory (Hebb, 2005) proposes an explanation of this self-
organization and describes the basic rules that guide the network design. The
theory implies that simultaneous activation of a pair of neurons leads to an
increase in the strength of their connection. In general,the Hebbian theory
implies that the neural activity transforms brain networks. This assumption
leads to an interesting question. Can temporal dynamics cause a self-
organization process in the Wikipedia Web network, similar to the one, driven
by neural spikes in the brain?

To answer this question, we introduce a collective memory modeling approach
inspired by the theory of learning in the brain, in particular, a content-
addressable memory system, the Hopfield network (Hopfield, 1982). In our
experiments, we use a dynamic network of Wikipedia articles, where the
dynamics comes from the visitors activity on each page, i.e. the number of
visits per page per hour. We assume that the Wikipedia network can self-
organize similar to a Hopfield network with a modified Hebbian learning rule
and learn collective memory patterns under the influence of visitors activity
similar to neurons in the brain. The results of our experiments demonstrate
that memorized patterns correspond to groups of collective memories,
containing clusters of linked pages that have a closely related meaning. A
topic of a cluster corresponds to a real world event that triggered the
interest of Wikipedia visitors during a finite period of time. In addition,
the collective memory gives access to the time lapse when the event occured.
We also show that our collective memory model is able to recall an event
recorded in the collective memory, given only a part of the event cluster.
Here the term _recall_ means that we recover a missing fraction of the visitor
activity signal in a memory cluster.

Contributions. Our contributions are as follows:

  * •

We propose a novel collective memory learning framework, inspired by
artificial models of individual memory – the Hebbian learning theory and the
Hopfield network model.

  * •

We formalize our findings into an content-addressable model of collective
memories. So far, collective memory (Halbwachs, 2013) has been considered just
as a concept that lacks a general model of memory formation. In the
experiments we demonstrate that given a modified Hebbian learning rule,
Wikipedia hyperlinks network can self-organize and gain properties reminiscent
of an associative memory.

  * •

We develop a graph algorithm for collective memory extraction. Computations
are local on the graph, allowing to build an efficient implementation. We
provide a distributed implementation of the algorithm to show that it can
handle dense graphs and massive amounts of time-series data.

  * •

We present graph visualizations as an interactive tool for collective memory
studies.

The rest of this paper organized as follows. In Section 2, we give an overview
of related works on large-scale collective memory research and analysis. Then,
we present a graph algorithm and a community detection approach in Section 4.
Section 3 describes the dataset and data preprocessing. We then discuss the
results of our experiments and evaluate discovered memory properties in
Section 6. Finally, information about the data, the code and tools used for
the study as well as online visualizations are provided in Section 8.

##  2\. Related work

The term Collective memory first appeared in the book of Maurice Halbwachs in
1925 (Halbwachs, 2013). He proposed a concept of a social group that shares a
set of collective memories that exist beyond a memory of each member and
affects understanding of the past by this social group. Halbwachs’s hypothesis
influenced a range of studies in sociology (Assmann and Czaplicka, 1995),
(Barash, 2016), psychology (Coman et al., 2009), (Ferron and Massa, 2012),
cognitive sciences (Ferron, 2012), and, only recently, in computer science (Au
Yeung and Jatowt, 2011), where authors extract collective memories using LDA
(Blei et al., 2003), applied to a collection of news articles.

Despite the fact that Wikipedia is the largest ever created encyclopedia of
public knowledge and the fifth most visited website in the world, the studies
on collective memory considered the Wikipedia visitor activity data only
recently. The idea of regarding Wikipedia as a global memory space was first
introduced by Pentzold in 2009 (Pentzold, 2009). Then, it was followed by a
range of collective memory studies based on various Wikipedia data archives.

Analyzing Wikipedia page views, Kanhabua et al. (Kanhabua et al., 2014)
proposed a collective memory model, investigating 5500 events from 11
categories. The authors proposed a remembering score based on a combination of
time-series analysis and location information. The focus of the work is on the
four types of events: aviation accidents, earthquakes, hurricanes, and
terrorist attacks. The work presents extensive experimental results, however,
it limited to particular types of collective memories.

Traumatic events such as attacks and bombings have also been investigated in
(Ferron and Massa, 2011), (Ferron, 2012) based on the Wikipedia edit activity
data. The authors investigate the difference between traumatic and
nontraumatic events using natural language processing techniques. The study is
limited to a certain type of events.

Another case study (García-Gavilanes et al., 2017) focuses on memory-
triggering patterns to understand collective memories. The collective memory
model is inferred from the Wikipedia visitors activity. The work considers
only a case of aircraft incidents reported in English Wikipedia. The authors
try to build a general mathematical model and explain the phenomenon of
collective memory, extracted from Wikipedia, based on that single topic.

Popularity and celebrities represent another focus point of public interest.
The Wikipedia hourly visits on the pages of celebrities was used to
investigate fame levels of tennis players (Yucesoy and Barabási, 2016). The
authors, though, did not tackle collective memories and aimed to quantify the
relationship between performance and popularity of the athletes.

To the best of our knowledge, we are the first to apply a content-addressable
memory model to the Wikipedia viewership data for collective memory analysis.
We build our collective memory model based on the assumption that it is
similar to a memory of an individual. For the first time, we demonstrate that
Wikipedia Web network and its dynamics can gain properties reminiscent of
associative memory. Unlike the results of previous works, our model is not
limited to particular classes of events and does not require labeled data. We
do not analyze the content of the pages and only rely on the collective
viewership dynamics in the network that makes the presented model tractable.

Figure 1.  a) Data structure built from Wikipedia data, with time-series
associated to nodes of the Wikipedia hyperlink graph. b) Focus on the time-
series signals (number of visits per hour) residing on the vertices of the
graph.

##  3\. Dataset

We use the dataset described in (Benzi, 2017). This dataset is based on two
Wikipedia SQL dumps: English language articles and user visit counts per page
per hour. The original datasets are publicly available on the Wikimedia
website (Foundation, 2016).

The Wikipedia network of pages is first constructed using the data from
article dumps that contain information about the references (edges) between
the pages (nodes)111Note that Wikipedia is continuously updating. Some links
that existed at the moment we made the dump may have been removed from current
versions of the pages. To check consistency with past versions, one can use
the dedicated search tool at http://wikipedia.ramselehof.de/wikiblame.php..
Time-series are then associated to each node (Fig. 1), corresponding to the
visits history from 02:00, 23 September 2014 to 23:00, 30 April 2015.

The graph contains 116 016 Wikipedia pages (out of a total 4 856 639 pages)
with 6 573 475 hyperlinks. Most of the Wikipedia pages remain unvisited.
Therefore, only the pages that have a number of visits higher than 500 at
least once in the recording are kept. The time-series associated to the nodes
have a length of $T=5278$ hours.

Preprocessing. To identify potential core events of collective memories and
reduce further the processing time, we select nodes that have bursts of
visitor activity, i.e. spikes in the signal. This is done on a monthly basis
for two reasons. Firstly, it reduces the processing and the needs for memory
as each month can be processed independently. Secondly, it allows for a study
of the evolution of the collective memories on a monthly basis, by
investigating the trained Hopfield networks (see next section).

For each month, we define the burstiness $b$ of a page as the number and
length of the peaks of visits over time. We denote by $x_{i}$ the time-series
giving the number of visits per hour for a node labeled $i$, during month $m$.
To define a burst of visits we compute the mean $\mu$ and the standard
deviation $\sigma$ of $x_{i}$. We select values that are above $n\sigma+\mu$,
where $n$ is a tunable activity rate parameter ($n=5$ here). The burstiness of
the page associated to node $i$ is defined by $b=\sum_{t=0}^{T_{m}-1}k[t]$
where $T_{m}$ is the length of the time-series for month $m$ and

(1) $k[t]=\begin{cases}1,&\text{if $x_{i}[t]>n\sigma+\mu$},\\\
0,&\text{otherwise}.\end{cases}$

For each month, we discard the pages that have a burstiness smaller or equal
to $5$.

##  4\. Collective memory learning

Our goal is to create a self-organized memory from the Wikipedia data. This
memory is not a standard neural network but more a concept network or a
network made of pieces of information. Indeed, the Wikipedia pages replace
neurons in the network. Hence it can be seen as a memory connecting high-level
concepts. The self-organized structure is shaped as a Hopfield network with
additional constraints, relating it to the concept of associative memory. The
learning process follows the Hebbian rules, adapted to our particular dataset.

Hopfield network. Our approach is based on a Hopfield model of artificial
memory (Hopfield, 1982). A Hopfield network is a recurrent neural network
serving as an associative memory system. It allows for storing and recalling
patterns. Let $N$ be the number of nodes in the Hopfield network. Starting
from an initial partial memory pattern $P_{0}\in\mathbb{R}^{N}$, the action of
recalling a learned pattern is done by the following iterative computation:

(2) $P_{j+1}=f_{\theta}(WP_{j}),$

where $W$ is the weight matrix of the Hopfield network. The function
$f_{\theta}:\mathbb{R}^{N}\to\mathbb{R}^{N}$ is a nonlinear thresholding
function (step function giving values $\\{-1,1\\}$) that binarize the vector
entries. The value $\theta$ is the threshold (same for all entries). In our
case, we build a network per month so $W$ is associated to a particular month.
For each $j\geq 0$, $P_{j}$ is a matrix of (binarized) time-series where each
row is associated to a node of the network and each column corresponds to an
hour of the month considered. We stop the iteration when the iterative process
has converged to a stable solution ($P_{j+1}=P_{j}$). Note that $P_{0}$ is a
binary matrix, obtained from the time-series using the function defined in Eq.
(1).

Dimensionality reduction. The learning process has been modified in order to
cope with a large amount of data. We recall that the number of pages has
already been reduced by keeping only the ones with bursts of activity.
However, it is still a large number of neurons and weights for a Hopfield
network. Therefore, instead of training a fully connected network, we reduce
the number of links in the following manner. We consider only hyperlink
connections between pages and we learn their associated weight. In this way,
nodes in the network are linked by human-made connections and it can be seen
as a sort of pre-trained, constrained, network. This is a strong assumption as
no link in the Hopfield network can be created between pages that are not
related by a hyperlink.

Hebbian learning. We use a synaptic-plasticity-inspired computational model
(Hebb, 2005) in the proposed graph learning algorithm to compute weights. The
main idea is that a co-activation of two neurons results in the enforcement of
a connection (synapse) between them. We do not take causality of activations
into account. For two neurons $i$ and $j$, their respective activity (number
of visits) over time $x_{i}$ and $x_{j}$ are compared to determine if they are
co-active or not at each time step. We introduce the following similarity
measure ${\rm Sim\\{i,j,t\\}}$ between node $i$ and $j$ at time $t$, where
${\rm Sim\\{i,j,t\\}}=0$ if $x_{i}[t]=x_{j}[t]=0$ and otherwise:

(3) ${\rm
Sim}\\{i,j,t\\}=\frac{\min(x_{i}[t],x_{j}[t])}{\max(x_{i}[t],x_{j}[t])}\in[0,1].$

This function compares the ratio of the number of visits, putting more
emphasis on the pages receiving a similar amount of visits.

For each time step $t$, the edge weight $w_{ij}$ between $i$ and $j$ is
updated by the following amount:

(4) $\Delta w_{ij}=\begin{cases}+Sim\\{i,j\\},&\text{if $Sim\\{i,j\\}$ $>$
$\lambda$},\\\ 0,&\text{otherwise},\end{cases}$

where $\lambda=0.5$ is a threshold parameter.

The computations are tractable because 1) they are local on the graph, i.e.
weight updates depend on a node and its first order neighborhood, 2) weight
updates are iterative, and 3) a weight update occurs only between the
connected nodes and not among all possible combinations of nodes. These three
facts allow us to build a distributed model to speed up computations. For this
purpose, we use a graph-parallel Pregel-like abstraction, implemented in the
GraphX framework (Gonzalez et al., 2014), (Xin et al., 2013).

Figure 2. Weighted degree distribution in log-log scale. Linearity in log-log
scale corresponds to power-law behavior $P(k)\sim k^{-\gamma}$. Power-law
exponent $\gamma=3.81$ for the initial graph (blue), and $\gamma=2.85$ for the
learned graph (red). The decrease of the exponent in the learned graph
indicates that the weight distribution becomes smoother and the number of
greedy hubs drops with the learning.

##  5\. Graph visualization and community detection

The structure of each learned network shows patterns corresponding to
communities of correlated nodes. We analyze all the monthly Hopfield networks,
extract communities and visualize them in the following sections. We use a
heuristic method based on modularity optimization (Louvain method) (Blondel et
al., 2008) for community detection. Colors in the graph depict and highlight
the detected communities. A resolution parameter controls the size of
communities. To represent the graph in 2D space for visualization, we use a
force-directed layout (Jacomy et al., 2014). The resulting spatial layout
reflects the organization of the network. An interactive version of all
visualizations presented in this paper is available online (Miz, 2017b).

Figure 3. Edge weight distribution in the learned graph in log scale. The
learned graph has skewed heavy-tailed weight distribution.

##  6\. Experiments and results

Using the approach presented in Section 4, we intend to learn collective
memories and describe their properties. After the learning, we inspect
different resulting networks on different time scales. We extract strongly
connected clusters of nodes from these networks and discuss their features.
Also, we investigate the temporal behavior of the memories.

A 7 months network. We first learn the graph using the 7-month Wikipedia
activity dataset. We start from the initial graph of Wikipedia pages connected
with hyperlinks, described in Section 3. The Hopfield network, the result of
the learning process, is a network $G=(V,E,W)$, where $V$ is the set of
Wikipedia pages, $E$ is the set of references between the pages, and $W$ is
the set of weights, reflecting the similarity of activity patterns between
articles.

After the learning, the majority of weights are 0 (6 297 977, 95.8%). Only 275
498 edges (4.2%) have strictly positive weights. For a better visualization,
we prune out the edges that have 0-weight. We also delete disconnected nodes
that remain after edge pruning and remove small disconnected components of the
graph. The number of remaining nodes is 35 839 (31% of the initial number).
Figure 4 shows snapshots of the initial Wikipedia graph before learning (a)
and the learned graph after weight update and pruning (b).

The initial and learned Wikipedia graphs have statistically heterogeneous
connectivity (Fig. 2) and weight patterns (Fig. 3) that correspond to skewed
heavy-tailed distributions.

The degree distribution of the initial graph has a larger power-law exponent
$\gamma=3.81$ (Fig. 2, blue) than the learned one ($\gamma=2.85$). This shows
that the initial Wikipedia graph is dominated by large hubs that attract most
of the connections to numerous low-degree neighbors. These hubs correspond to
general topics in the Wikipedia network. They often link broad topics such as,
for instance, the “United States” page, with a large number of hyperlinks
pointing to it from highly diverse subjects.

We have applied a community detection algorithm, described in Section 5, to
both graphs. The initial Wikipedia graph (Fig. 4) is dense and cluttered with
a significant number of unused references, while the learned graph reveals
smaller and more separated communities. This is confirmed by the community
size distribution of the graphs (Fig. 5). The number of communities and their
size change after learning. Initially, the small number of large communities
dominate the graph (blue), while after the learning (red) we see a five times
increase in the number of communities. Moreover, as a result of the learning,
the size of the communities decreases by one order of magnitude. The
modularity of the learned graph is 25% higher, giving the first evidence of
the creation of associative structures.

The analysis of each community of nodes in the learned graph gives a glimpse
of the events that occurred during the 7-month period. Each cluster is a group
of pages related to a common topic such as a championship, a tournament, an
awards ceremony, a world-level contest, an attack, an incident, or a popular
festive events such as Halloween or Christmas. The graph contains the summary
of events that occurred during the period of interest, hence its name of
collective memory.

Before going deeper in the clusters analysis and the information they contain,
we investigate the evolution of the graph structure over time.

|
---|---
(a) Initial | (b) Learned
Figure 4. Wikipedia graph of hyperlinks (left) and learned Hopfield network
(right). Colors correspond to the detected communities. The learned graph is
much more modular than the initial one, with a larger number of smaller
communities. The layout is force-directed.

Monthly networks. To obtain a better, fine-grained view of the collective
memories, we focus on a smaller time-scale. Indeed, events attracting the
attention of Wikipedia users for a period longer than a week or two are rare.
Therefore, we split our dataset into months. Monthly graphs are smaller,
compared to the 7-months graph, and contain 10 000 nodes on average. However,
the properties and distributions of monthly graphs are similar to the 7-months
one, described above.

Figure 5. Community size distribution of the initial Wikipedia graph of
hyperlinks (blue) and the learned Hopfield network (red). The total number of
communities: 32 for the initial graph, 172 for the learned one.

Short-term (monthly) graphs allow to understand and visualize the dynamics of
the memory formation in the long-term (7 months) graph. To give an example of
the memory evolution, we discuss the cluster of the USA National Football
League championship and the collective memories that are mostly related to the
previous NFL seasons (Fig. 6). NFL is one of the most popular sports leagues
in the USA and its triggers a lot of interest on Wikipedia. Thanks to the high
number of visits on this topic we were able to spot a cluster related to the
NFL on each of the monthly graphs. Figure 6 shows information about the NFL
clusters. The top part of the figure contains the learned graphs, for each
month, where the NFL cluster is highlighted in red. The final game of the 2014
season, Super Bowl XLIX, had been played on February 1, 2015. This explains
the increase in size until February where it reaches its maximum. The activity
collapses after this event and the cluster disappears. For the sake of
interpretability, we extracted 30 NFL team pages from the original cluster
(485 pages) to show the details of the evolution in time as a table on Fig. 6.
This fraction of the nodes reflects overall dynamics in the entire cluster.
Each row describes the hourly activity of a page, while the columns split the
plot into months. The sum of visits for the selected pages is plotted as a red
line on the bottom.

Figure 6. Evolution of the National Football League 2014-2015 championship
cluster. We show 30 NFL teams from the main cluster. Top: the monthly learned
graph with the NFL cluster highlighted in red. Middle table: visitors activity
per hour on the NFL teams’ Wikipedia pages in greyscale (the more visits, the
darker). Bottom: timeline, in red, of the overall visitor activity in the
cluster.

The dynamics of the detected cluster reflects the real timeline of the NFL
championship. The spiking nature of the overall activity corresponds to
weekends when most of the games were played. Closer to the end of the
championship, the peaks become stronger, following the increasing interest of
fans. We see the highest splash of the activity on 1 February, when the final
game was played.

We want to emphasize that this cluster, as well as all the others, was
obtained in a completely unsupervised manner. Football team pages were
automatically connected together in a cluster having “NFL” as a common topic.
Moreover, the cluster is not formed by one Wikipedia page and its direct
neighbors, it involves many pages with distances of several hops on the graph.

The NFL championship case is an example of a periodic (yearly) collective
memory. The interest increases over the months until the expected final event.
Accidents and incidents are different types of events as they appear suddenly,
without prior activity. Our proposed learning method allows for the detection
of these kinds of collective memories as well. We provide examples of three
accidents to demonstrate the behavior of the collective memory in case of an
unexpected event.

We pick three core events among 172 detected and discuss them to show the
details of memory detection by our approach. Figure 7 shows the extracted
clusters from the learned graph (top) and the overall timeline of the
clusters’ activity (bottom).

Charlie Hebdo shooting. 7 January 2015. This terrorist attack is an example of
an unexpected event. The cluster emerged over a period of 72 hours, following
the attack. All pages in the cluster are related to the core event.
Strikingly, a look at the title of the pages is sufficient to get a precise
summary of what the event is about. There is a sharp peak of activity on the
first day of the attack, slowly decaying over the following week.

Germanwings flight 9525 crash. 24 March 2015. This cluster not only involves
pages describing the crash or providing more information about it, but also
the pages of similar events that happened in the past. It includes, for
example, a page enumerating airlane crashes and the page of a previous crash
that happened in December 2014, the Indonesia AirAsia Flight 8501 crash. As a
result, the memory of the event is connected to the memory of the Flight 8501
crash, that is why we can see an increase of visits in December. This is an
example where our associative memory approach not only connects pages together
but also events.

Ferguson unrest. Second wave. November 24, 2014 – December 2, 2014. This is an
example of an event that has official beginning and end dates. A sharp
increase in the activity at the beginning of protests highlights the main
event. This moment triggers the core cluster emergence. We also see that the
cluster becomes active once again at the end of the unrest.

|  |
---|---|---
|  |
(a) Germanwings 9525 crash | (b) Ferguson unrest | (c) Charlie Hebdo attack
Figure 7. Collective memory clusters (top) and overall activity of visitors in
the clusters (bottom).

Eventually, we conclude our exploration of the clusters of the learned graphs
by providing on Table 1 a list of handpicked page titles inside each cluster
that refer to previous events and related subjects. The connected events
occurred outside of the 7-months period we analyze. This illustrates the
associative features of our method. Firstly, pages are grouped by events with
the help of visitors activity. Secondly, events themselves are connected
together by this activity. Memory is made of groups of concepts tightly
connected and these groups are in turn connected together through concepts
they share.

Table 1. Collective memories triggered by core events.

Charlie Hebdo attack | Germanwings 9525 crash | Ferguson unrest
---|---|---
Porte de Vincennes hostage crisis | Inex-Adria Aviopromet Flight 1308 | Shooting of Tamir Rice
Al-Qaeda | Pacific Southwest Airlines Flight 1771 | Shooting of Amadou Diallo
Islamic terrorism | SilkAir Flight 185 | Sean Bell Shooting Incident
Hezbollah | Suicide by pilot | Shooting of Oscar Grant
2005 London bombings | Aviation safety | 1992 Los Angeles riots
Anders Behring Breivik | Air France Flight 296 | O.J. Simpson murder case
Jihadism | Air France Flight 447 | Shooting of Trayvon Martin
2015 Baga massacre | Airbus | Attack on Reginald Denny

Recalling memories. In this section, our goal is to test the hypothesis that
the proposed method, as a memory, allows recalling events from partial
information. We emulate recall processes using the Hopfield network approach,
described in Section 4. We show that the learned graph structure can recover a
memory (cluster of pages and its activations) from an incomplete input.

We create incomplete patterns for the Hopfield network by selecting randomly a
few pages contained in a chosen cluster. We built the input matrix $P_{0}$
setting to $(-1)$ (inactive state) all the time-series except for the few
pages selected. We then apply iteratively Eq. (2).

The results of the experiment are illustrated with an example on Figure 8. We
selected the cluster associated to the Charlie Hebdo Attack. From the list of
pages, we select a subset of it, here $80\%$. We applied the learned graph for
the month of January, when the memory was detected. After the recall (Fig. 8,
right), most of the cluster is recovered at the correct position in time. Note
that the model forgets a part of the activity, plotted in light red. This
missing part is made of pages that are active outside of the time of the
event, giving the evidence that they are not directly related (or weakly
related) to the event.

To evaluate the performance of the remembering, we mute different fractions of
nodes in memory clusters and compute recall errors. We define the recall
accuracy $A$ as the ratio of correctly recalled activations over the number of
initial activations. The error is given by $1-A$. Figure 10 shows the results
of the evaluation. We consider three types of errors. First, we measure the
accuracy of a recalled pattern over 7 months (green). Second, the accuracy of
a recalled pattern over a period of 72 hours after the start of the event
(blue). And third, a relaxed accuracy of the recall during the same period of
time, with a correct recovery if an initially active node is active at least
once in the recalled pattern (red). As expected, the quality of the recalled
memory increases with the quality of the input pattern. The better performance
of the recall in the 72 hours zone is due to the memory effect that focuses on
the most active part of the cluster and forgets small isolated activity spikes
of the individual pages scattered over the 7 months.

---
Figure 8. Recall of an event from a partial pattern (Charlie Hebdo attack). The red vertical lines define the start of the event and its most active part, ending 72 hours from the start. Left: full activity over time of the pages in the cluster. Middle: pattern with 20% of it set inactive. Right: result of the recall using the Hopfield network model of associative memory. In light red are shown the difference with the original pattern (the forgotten activity). |  |  |  |  |  |
---|---|---|---|---|---|---
October | November | December | January | February | March | April
Figure 9. Recalled activity over time for each monthly learned graph.

In Figure 9, we show the results of the recall process for each monthly
learned graph when the entire time-series are given at the input. The output
of the graph is summed up to create a global curve of activity over the 7
months. The initial global activity is subtracted from the output curve to
obtain the final curve plotted on the figure. Red areas on the plot correspond
to a low activity in the output, hence an absence of memory in the zone. Green
areas represent positive recalls. The memory of each of the monthly graph
corresponds to its learning period. This demonstrates the global effectiveness
of the learning and the recall process.

Figure 10. Plot of the recall error rate with three different error measures.

##  7\. Conclusions and future work

In this paper, we propose a new method that allows learning and remembering
collective memories in an unsupervised manner. To extract memories, the method
analyses the Wikipedia Web network and hourly viewership history of its
articles. The collective memories are summaries of the events that raised the
interest of Wikipedia visitors. The approach is able to perform an efficient
knowledge retrieval from the Wikipedia encyclopedia allowing to highlight the
changing interests of visitors over time and to shed some light on their
collective behavior.

This approach is able to handle large-scale datasets. We have noted
experimentally a high robustness of the method to the tuning of the
parameters. This will be investigated further in future work.

This work opens new avenues for dynamic graph-structured data analysis. For
example, the proposed approach could be used in a framework for automated
event detection, monitoring, and filtering in network structured visitor
activity streams. The resulting framework is also of interest for the
understanding of the human memory and the simulation of an artificial memory.

##  8\. Tools, implementation, code and online visualizations

All learned graphs (overall September-April, monthly activity, and localized
events) are available online (Miz, 2017b) to foster further exploration and
analysis. For graph visualization we used the open source software package
Gephi (Bastian et al., 2009) and layout ForceAtlas2 (Jacomy et al., 2014). We
used Apache Spark GraphX (Xin et al., 2013), (Gonzalez et al., 2014) for graph
learning implementation and graph analysis. The presented results can be
reproduced using the code for Wikipedia dynamic graph learning, written in
Scala (Miz, 2017a). The dataset is available on Zenodo (Kirell et al., 2017).

###### Acknowledgements.

We would like to thank Michaël Defferrard and Andreas Loukas for fruitful
discussions and useful suggestions. The research leading to these results has
received funding from the European Union’s H2020 Framework Programme
(H2020-MSCA-ITN-2014) under grant agreement no 642685 MacSeNet.

## References

  * (1)
  * Allport (1985) DA Allport. 1985\.  Distributed memory, modular subsystems and dysphasia.  Current perspectives in dysphasia (1985), 32–60.
  * Assmann and Czaplicka (1995) Jan Assmann and John Czaplicka. 1995.  Collective memory and cultural identity.  New German Critique 65 (1995), 125–133.
  * Au Yeung and Jatowt (2011) Ching-man Au Yeung and Adam Jatowt. 2011.  Studying how the past is remembered: towards computational history through large scale text mining. In Proceedings of the 20th ACM international conference on Information and knowledge management. ACM, 1231–1240.
  * Barash (2016) Jeffrey Andrew Barash. 2016\.  Collective Memory and the Historical Past.  University of Chicago Press.
  * Bastian et al. (2009) Mathieu Bastian, Sebastien Heymann, Mathieu Jacomy, et al. 2009\.  Gephi: an open source software for exploring and manipulating networks.  Icwsm 8 (2009), 361–362.
  * Benzi (2017) Kirell Maël Benzi. 2017\.  From recommender systems to spatio-temporal dynamics with network science.  EPFL, Chapter 5, 97–98.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  Journal of machine Learning research 3, Jan (2003), 993–1022.
  * Blondel et al. (2008) Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008.  Fast unfolding of communities in large networks.  Journal of statistical mechanics: theory and experiment 2008, 10 (2008), P10008.
  * Coman et al. (2009) Alin Coman, Adam D Brown, Jonathan Koppel, and William Hirst. 2009\.  Collective memory from a psychological perspective.  International Journal of Politics, Culture, and Society IJPS 22, 2 (2009), 125–141.
  * Ferron (2012) Michela Ferron. 2012\.  Collective memories in Wikipedia.  Ph.D. Dissertation. University of Trento.
  * Ferron and Massa (2011) Michela Ferron and Paolo Massa. 2011.  Studying collective memories in Wikipedia.  Journal of Social Theory 3, 4 (2011), 449–466.
  * Ferron and Massa (2012) Michela Ferron and Paolo Massa. 2012.  Psychological processes underlying Wikipedia representations of natural and manmade disasters. In Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration. ACM, 2.
  * Foundation (2016) Wikimedia Foundation. 2016\.  Wikimedia Downloads: Database tables as sql.gz and content as XML files.  (May 2016).  https://dumps.wikimedia.org/other/ Accessed: 2016-May-16.
  * García-Gavilanes et al. (2017) Ruth García-Gavilanes, Anders Mollgaard, Milena Tsvetkova, and Taha Yasseri. 2017.  The memory remains: Understanding collective memory in the digital age.  Science Advances 3, 4 (2017), e1602368.
  * Gonzalez et al. (2014) Joseph E Gonzalez, Reynold S Xin, Ankur Dave, Daniel Crankshaw, Michael J Franklin, and Ion Stoica. 2014.  GraphX: Graph Processing in a Distributed Dataflow Framework.. In OSDI, Vol. 14. 599–613.
  * Graus et al. (2017) David Graus, Daan Odijk, and Maarten de Rijke. 2017.  The birth of collective memories: Analyzing emerging entities in text streams.  arXiv preprint arXiv:1701.04039 (2017).
  * Halbwachs (2013) Maurice Halbwachs. 2013\.  Les cadres sociaux de la mémoire.  Albin Michel.
  * Hebb (2005) Donald Olding Hebb. 2005\.  The organization of behavior: A neuropsychological theory.  Psychology Press.
  * Hopfield (1982) John J Hopfield. 1982\.  Neural networks and physical systems with emergent collective computational abilities.  Proceedings of the national academy of sciences 79, 8 (1982), 2554–2558.
  * Jacomy et al. (2014) Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian. 2014.  ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software.  PloS one 9, 6 (2014), e98679.
  * Kanhabua et al. (2014) Nattiya Kanhabua, Tu Ngoc Nguyen, and Claudia Niederée. 2014\.  What triggers human remembering of events? A large-scale analysis of catalysts for collective memory in Wikipedia. In Digital Libraries (JCDL), 2014 IEEE/ACM Joint Conference on. IEEE, 341–350.
  * Kirell et al. (2017) Benzi Kirell, Miz Volodymyr, Ricaud Benjamin, and Vandergheynst Pierre. 2017.  Wikipedia time-series graph.  (Sept. 2017).  https://doi.org/10.5281/zenodo.886484
  * Miz (2017a) V. Miz. 2017a.  Wikipedia graph mining. GraphX implementation.  https://github.com/mizvol/WikiBrain. (2017).
  * Miz (2017b) V. Miz. 2017b.  Wikipedia graph mining. Visualization.  (2017).  https://goo.gl/Xgb7QG
  * Pentzold (2009) Christian Pentzold. 2009\.  Fixing the floating gap: The online encyclopaedia Wikipedia as a global memory place.  Memory Studies 2, 2 (2009), 255–272.
  * Stone et al. (1999) Arthur A Stone, Christine A Bachrach, Jared B Jobe, Howard S Kurtzman, and Virginia S Cain. 1999.  The science of self-report: Implications for research and practice.  Psychology Press.
  * Tinati et al. (2016) Ramine Tinati, Markus Luczak-Roesch, and Wendy Hall. 2016\.  Finding Structure in Wikipedia Edit Activity: An Information Cascade Approach. In Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 1007–1012.
  * Xin et al. (2013) Reynold S Xin, Joseph E Gonzalez, Michael J Franklin, and Ion Stoica. 2013.  Graphx: A resilient distributed graph system on spark. In First International Workshop on Graph Data Management Experiences and Systems. ACM, 2.
  * Yucesoy and Barabási (2016) Burcu Yucesoy and Albert-László Barabási. 2016\.  Untangling performance from success.  EPJ Data Science 5, 1 (2016), 17.