# Session-based Recommendation with Hypergraph Attention Networks

Jianling Wang Texas A&M University, {jlwang, zhuziwei<EMAIL_ADDRESS>Kaize Ding22footnotemark: 2 Arizona State University<EMAIL_ADDRESS>Ziwei
Zhu11footnotemark: 1 James Caverlee11footnotemark: 1

###### Abstract

Session-based recommender systems aim to improve recommendations in short-term
sessions that can be found across many platforms. A critical challenge is to
accurately model user intent with only limited evidence in these short
sessions. For example, is a flower bouquet being viewed meant as part of a
wedding purchase or for home decoration? Such different perspectives greatly
impact what should be recommended next. Hence, this paper proposes a novel
session-based recommendation system empowered by hypergraph attention
networks. Three unique properties of the proposed approach are: (i) it
constructs a hypergraph for each session to model the item correlations
defined by various contextual windows in the session simultaneously, to
uncover item meanings; (ii) it is equipped with hypergraph attention layers to
generate item embeddings by flexibly aggregating the contextual information
from correlated items in the session; and (iii) it aggregates the dynamic item
representations for each session to infer the general purpose and current
need, which is decoded to infer the next interesting item in the session.
Through experiments on three benchmark datasets, we find the proposed model is
effective in generating informative dynamic item embeddings and providing more
accurate recommendations compared to the state-of-the-art.

##  1 Introduction

Recommendation systems are ubiquitous, acting as an essential component in
online platforms to help users discover items of interest. In many practical
scenarios, a recommendation system needs to infer the next interaction for a
user based only on the short-term prior interactions within a particular
session. Previous approaches that learn a static model of users [10, 11, 22,
31, 34] or rely on long-term user behaviors [9, 17, 28, 30] are not well-
suited for such scenarios and could lead to poor predictive power. In
contrast, session-based recommendation has attracted increasing attention,
with recent approaches showing promising performance in inferring user
interests in these (often anonymous) short-term sessions [16, 19, 21, 23, 32].

Figure 1: An example with three sessions. The same flower bouquet appears in
each session, but with a different purpose in each. The righthand side shows
contextual windows of size 2 and 3 including this flower bouquet. Items in the
same contextual window are correlated; and different contextual windows may
have different levels of importance for characterizing an item.

A critical issue is how items are treated in such session-based recommendation
approaches. The individual items can reveal user intent, but they only provide
limited evidence. To illustrate, consider the example in Figure 1. In
different sessions, the same flower bouquet can be viewed differently, i.e.,
as part of a wedding party purchase, an option for home decoration, or a gift
for Mother’s Day. However, if we independently consider Flower Bouquet A, it
might be viewed as exactly the same item across sessions. In a sense, the
meaning of an item (and what it reveals about user intent) could be inferred
from contextual windows, each of which contains a set of consecutive items
showing up together within a session. In Session $C$, bouquet $A$ is likely
for a wedding party since the user clicks it right between other wedding-
related items, while the bouquet in Session $A$ may be a Mother’s Day gift
since it is clicked along with items specified for “Mom.”

Hence, we propose to exploit these contextual windows to model session-wise
item representations that can robustly capture user intent with only limited
evidence available in short sessions. However, there are several key
challenges in eliciting the user intent signal among items from the contextual
windows in each session: (1) conventional graph structures and graph neural
networks [2, 3, 7, 8, 15] are designed to model the pairwise connections
between items, which are not always sufficient since we need to consider
contextual windows connecting various numbers of items ranging from two to
many. For example, for Session $B$ in Figure 1, the pairwise linkage between
bouquet $A$ and $B$ is not enough to reveal that bouquet $A$ is meant for home
decoration. But such evidence could be inferred by analyzing the triadic
relations among both the bouquets and a vase as defined by Contextual Window
$1$. To this end, we adopt the hypergraph [1, 4, 6, 29] structure to model the
correlations amongst items in different contextual windows within each
session. In a session hypergraph, each node denotes an item, and a hyperedge
connects the collection of items that show up together within a contextual
window. This hypergraph structure supports capturing correlations among items
defined by contextual windows, which could be arbitrary-order depending on the
usage scenario; (2) while propagating and aggregating the user intent evidence
within a session hypergraph, some items are informative, but others may not
be. Moreover, different contextual windows may bring in different levels of
evidence for how items are represented. Thus, a key challenge is how to
carefully highlight the informative items on each hyperedge and also emphasize
the evidence from hyperedges with larger impacts.

To tackle the aforementioned challenges, we propose SHARE: a Session-based
Hypergraph Attention Network for REcommendation. Specifically, the proposed
SHARE has three unique characteristics:

  * •

First, it captures contextual information with sliding windows, whereby items
appearing in the same window are connected with a hyperedge. By applying
multiple sliding windows on the session sequence, it is able to model a
session considering contextual windows of varying sizes simultaneously in the
hypergraph structure.

  * •

Second, it incorporates a carefully-designed hypergraph attention network to
extract the user intent evidence from contextual windows, which is able to pay
more attention on the informative items (nodes) and also emphasize the
evidence from contextual windows (hyperedges) with larger impacts.

  * •

Third, the session-wise item embeddings resulting from a stack of hypergraph
attention layers can then be fed into a self-attention layer to infer both the
general intent and current interests in the session, which are decoded jointly
to generate the next-item recommendation for this session.

With experiments on three benchmark datasets for session-based recommendation,
we show that such a hypergraph-based approach is better suited than
conventional graphs for modeling item correlations within sessions. The
proposed SHARE is effective in predicting the next interesting items, and
significantly outperforms the state-of-the-art in session-based
recommendation.

##  2 Related Work

#### Session-based Recommendation.

Given sequential user behaviors in a session, session-based recommendation
systems aim to infer the subsequent behavior. Since these sessions usually
exclude login information and take place in the short-term (e.g., less than a
half hour) where user intent is local to the session, previous works that
elicit user preferences from long-term historic behaviors or generate user
embeddings based on their identification [11, 28, 30] are not well-suited to
providing recommendations for these anonymous short-term sessions. Though
matrix factorization-based and neighbor-based methods can be applied for
session-based recommendation [18, 24], they ignore the transitional patterns
between items. Thus, FPMC [23] has been proposed to extend matrix
factorization with Markov Chains, which is able to predict the next item
considering the transition from the last item. With the advancement of neural
networks, different neural mechanisms have been applied to handle sequential
session data. In [12, 13], the authors adopt recurrent neural networks to
learn sequential patterns from all sessions and infer the next items with the
output of the last layer. NARM [16] utilizes an attention layer to aggregate
items in the session and capture the main purpose of each session.
Furthermore, STAMP [19] designs another attention component to emphasize the
short-term interest in the session. To capture the local dependencies between
items, SR-GNN [32] applies the conventional graph neural network to model the
transitions between items in a session graph. In contrast, we propose to
utilize hypergraphs and an attention mechanism to model item correlations and
user intent with the contextual windows in session-based recommendation.

#### Graph-based Recommendation.

Since graphs can be a good fit for modeling the interactions between item-
item, user-user, or user-item, there are works in developing recommendation
systems centering around different graph structures. Translation-based
recommendation systems [9, 20] treat all the items as nodes and users as the
connections between items consumed subsequently. These models embed users and
items into a similar space by minimizing the translation loss, which requires
clear user identification (which is often not available in session-based
scenarios). Recently, several efforts apply newly introduced Graph Neural
Networks (GNNs) for recommendation [2, 31, 33], in which different GNN models
have been designed for representation learning in graph structured data. GCMC
[2] consists of neural graph autoencoders to reconstruct the user-item rating
graph. And in NGCF [31], the authors propose to construct a user-item
bipartite graph and utilize multiple graph neural layers to capture the multi-
hop collaborative signals between users and items. In social recommendation,
the relations between users can be exploited with the social graph [5, 32].
For example, DiffNet [5] adopts a Graph Convolutional Network to model the
diffusion of user embeddings among their social connections. Unlike SHARE,
these methods are built on conventional graph structures (not hypergraphs).
And they are not designed for the special characteristics of session-based
recommendation, like sequential user actions in short-lived sessions without
user identification.

##  3 The Proposed Model - SHARE

In this section, we propose a novel session-based recommendation model with
Hypergraph Attention Networks to exploit the contextual windows within each
individual session. In the following, we begin with the problem setup and then
structure the design of SHARE around three research questions: (i) Given that
items falling into the same contextual window are correlated, how do we
construct a hypergraph for each session to model the correlations among items
from a variety of contextual windows simultaneously? (ii) Considering these
contextual windows, how do we update an item embedding with the user intent
evidence that propagates in and across the contextual windows? (iii) With
these session-wise item embeddings, how can we infer the next interesting item
by extracting both the general interest and current need in the session?

###  3.1 Problem Setup.

In session-based recommendation [16, 21, 32], given the sequence of items
which have been interacted within a session, the goal is to predict the next
item. Let $\textbf{I}=\\{i_{1},i_{2},...,i_{N}\\}$ denote the set of $N$
unique items in the system. These items start with a set of embeddings
$\\{\textbf{i}_{1},\textbf{i}_{2}\,$…, $\textbf{i}_{N}\\}$, each of which is a
trainable embedding associated with the item ID. An anonymous session $s$
consisting of a sequence of $t$ actions (i.e., the items interacted within the
session) can be denoted by $\textbf{s}=[i_{s,1}$, $i_{s,2}$,…, $i_{s,t}]$, in
which the items are sorted in chronological order and $i_{s,p}$ represents the
$p_{th}$ item interacted within the session. A session-based recommendation
system should predict the next possible action (i.e., $i_{s,t+1}$) based on
the previous $t$ actions. That is, we want to generate the preference scores
for all the items in I based on the sequence of actions in session $s$. Then
the top-K preferred items can be treated as candidates for recommendation.

###  3.2 Session Hypergraph Construction.

With a sequence of interacted items in session $s$, we propose to model the
session as a hypergraph before exploiting the item correlations in different
(possibly, overlapping) contextual windows. Let
$\mathcal{G}_{s}=\\{\mathcal{V}_{s},\mathcal{E}_{s}\\}$ denote the hypergraph
constructed from session $s$, in which the node set $\mathcal{V}_{s}$ consists
of all the unique items appearing in this session. Each hyperedge
$e\in\mathcal{E}_{s}$ will connect all the items falling into the specific
contextual window. As in Figure 2, we can apply a sliding window with size $w$
on the item sequence for the session to identify all of the contextual windows
of size $w$ in this session, with which the items appearing in the same window
can be treated as items falling into the same contextual window and thus will
be connected with a hyperedge. Hence, to exploit contextual windows of varying
sizes in the session, we can apply sliding windows of varying sizes on the
item sequence. Let $\mathcal{E}_{s}^{w}$ represent the collection of all the
hyperedges constructed with such a sliding window of size $w$ on session $s$.
Then we gather hyperedges based on different sliding windows together to be
the set of hyperedges $\mathcal{E}_{s}$ for session $s$ with
$\mathcal{E}_{s}=\mathcal{E}_{s}^{2}\cup\mathcal{E}_{s}^{3}\cup...\mathcal{E}_{s}^{W}$,
in which $W$ is the maximum size of contextual windows that we consider in the
model. We repeat this process to construct the unique hypergraph for each of
the sessions.

Figure 2: The structure of SHARE: It applies multiple sliding windows to
capture the contextual information to construct a hypergraph. With a well-
designed HGAT network, it is able to generate the item embeddings revealing
their meanings in the specific session. The sequence of session-wise item
embeddings is fed into the self-attention layer to generate an embedding for
the session, which is decoded for the preference scores on items.

###  3.3 Hypergraph Attention Networks.

With the hypergraph structure, we want to learn the representations for nodes
(items) considering the correlations among nodes (items) defined by different
hyperedges. In the following, we introduce a novel hypergraph attention
network (HGAT) which is able to aggregate the user intent evidence that
propagates in and across contextual windows while updating the node
representations in a hypergraph. There are recent works extending neural
networks from a conventional graph to a hypergraph by generalizing the
convolution operation [1, 6, 29], with which the neighboring node features
will be firstly aggregated to the common hyperedges and then propagated to the
node. They usually treat nodes equally while characterizing the hyperedge or
use a pre-defined weight matrix to model the importance of information
propagated via different hyperedges. However, in session hypergraphs, for
nodes belonging to the same contextual window, some of them may be informative
in revealing user intent, but others may not be. And the evidence propagating
via different hyperedges (contextual windows) may bring in different levels of
impacts to an item. Thus, we require a new representation learning process for
hypergraphs to capture these special needs.

To tackle this problem, we propose to generalize the attention mechanism for
hypergraphs so that we can highlight the informative items on each hyperedge
and also emphasize the evidence from hyperedges with larger impacts. In the
following, we will explain the node representation learning process with the
proposed hypergraph attention layer (see Figure 2) in two steps:

Node to Hyperedge. Since information can propagate among neighboring nodes via
hyperedges, they are the key factor for node representation learning in
hypergraph. With the special hypergraph structure, instead of directly
updating each node with the the neighboring node information, we firstly treat
each hyperedge as an interlayer between nodes and aggregate all the
information propagating via the hyperedge.

We take the graph $\mathcal{G}_{s}$ constructed by session $s$ as an example.
The operation can be applied on hypergraphs for other sessions. Let
$\\{\textbf{n}_{1}^{(0)},\textbf{n}_{2}^{(0)},...,\textbf{n}_{p}^{(0)}\\}=\\{\textbf{i}_{1},\textbf{i}_{2},...,\textbf{i}_{p}\\}$
denote the node input for the first HGAT layer, which is the initial item
embeddings for the set of unique items $\\{i_{1},i_{2},...,i_{p}\\}$ appearing
in $\mathcal{G}_{s}$. Since some nodes on a hyperedge are informative but
others may not be, we should pay varying attention on the information from
these nodes while aggregating them together. Let $\textbf{m}_{t\sim j}^{(1)}$
denote the information propagating via hyperedge $\epsilon_{j}$ from node $t$
on the $(1)_{th}$ HGAT layer. We can aggregate the information from each of
the nodes connected by $\epsilon_{j}$ with the attention operation to generate
the representation $\textbf{e}_{j}^{(1)}$ as:

(3.1) $\displaystyle\textbf{e}_{j}^{(1)}$
$\displaystyle=\sum_{t\in\mathcal{N}_{j}}\textbf{m}_{t\sim
j}^{(1)}\quad\quad\textbf{m}_{t\sim
j}^{(1)}=\alpha_{jt}\textbf{W}_{1}^{(1)}\textbf{n}_{t}^{(0)}$
$\displaystyle\alpha_{jt}$
$\displaystyle=\frac{S(\hat{\textbf{W}}_{1}^{(1)}\textbf{n}_{t}^{(0)},\textbf{u}^{(1)})}{\sum_{f\in\mathcal{N}_{j}}S(\hat{\textbf{W}}_{1}^{(1)}\textbf{n}_{f}^{(0)},\textbf{u}^{(1)})}$

in which $\mathcal{N}_{j}$ denotes all the nodes connected by hyperedge
$\epsilon_{j}$ and $\textbf{u}^{(1)}$ represents a trainable node-level
context vector for the $(1)_{th}$ HGAT layer. $\textbf{W}_{1}^{(1)}$ and
$\hat{\textbf{W}}_{1}^{(1)}$ are the transform matrices and $\alpha_{jt}$
denotes the attention score of node t on hyperedge $\epsilon_{j}$. We use a
function $S(\cdot,\cdot)$ to calculate the similarity between the node
embedding and context vector. Empirically, we use Scaled Dot-Product Attention
to calculate the attention scores [14, 26], which is defined as:

(3.2) $S(\textbf{a},\textbf{b})=\frac{\textbf{a}^{T}\textbf{b}}{\sqrt{D}}$

where $D$ is the dimension size and can be used for normalization while
calculating the similarity scores.

Hyperedge to Node. To update the embedding for a node, we need to aggregate
the contents from all its connected hyperedges. Similarly, we utilize the
attention mechanism to perform the aggregation in order to model the
significance of different hyperedges. Let $\textbf{m}_{j\rightarrow t}^{(1)}$
denote information (user intent evidence) from hyperedge $\epsilon_{j}$ to
node $t$. Given the set of hyperedges $\mathcal{Y}_{t}$ which are connected to
node $t$, its update embedding is calculated as:

(3.3) $\displaystyle\textbf{n}_{t}^{(1)}$
$\displaystyle=\sum_{j\in\mathcal{Y}_{t}}\textbf{m}_{j\rightarrow
t}^{(1)}\quad\quad\textbf{m}_{j\rightarrow
t}^{(1)}=\beta_{tj}\textbf{W}_{2}^{(1)}\textbf{e}_{j}^{(1)}$
$\displaystyle\beta_{tj}$
$\displaystyle=\frac{S(\hat{\textbf{W}}_{2}\textbf{e}_{j}^{(1)},\textbf{W}_{3}^{(1)}\textbf{n}_{t}^{(0)})}{\sum_{f\in\mathcal{Y}_{t}}S(\hat{\textbf{W}}_{2}^{(1)}\textbf{e}_{f}^{(1)},\textbf{W}_{3}^{(1)}\textbf{n}_{t}^{(0)})}$

where $\textbf{W}_{2}^{(1)}$, $\hat{\textbf{W}}_{2}^{(1)}$ and
$\textbf{W}_{3}^{(1)}$ are the trainable matrices used to transform the vector
before calculating the attention scores for the $(1)_{th}$ HGAT layer.
$\beta_{tj}$ indicates the impact of hyperedge $\epsilon_{j}$ on node $t$. As
in the last step, we use the Scaled Dot-Product Attention formula defined by
Equation 3.2 to calculate $S$. The resulting $\textbf{n}_{t}^{(1)}$ can be
treated as the updated embedding for note $t$ by aggregating information from
its neighboring region in the hypergraph.

High-order Propagation. While a single HGAT layer can capture the information
from direct neighbors, we construct a Hypergraph Attention Network by stacking
multiple HGAT layers to model the multi-hop high-order information propagation
in the hypergraph. In this Hypergraph Attention Network, the output of the
$(l-1)^{th}$ HGAT layer is the input for the $l^{th}$ layer. Thus the outputs
(i.e., node embeddings) from the last layer ($L$) inherit the contextual
information from all the previous layers and can be used to characterize the
items in this session. For each of the session hypergraph, such an Hypergraph
Attention Network is able to generate session-wise item embeddings to reflect
user intent in the session, by highlighting the informative items on each
hyperedge and also emphasizing the evidence from hyperedges with larger
impacts.

###  3.4 Next-Item Prediction.

To infer the next interesting item, our goal is to generate an embedding which
can encode both the general interest and the current need of the session. We
adopt the idea from self-attention [14, 26] to achieve this goal. Since the
general interest can be uncovered by the aggregation of all the items and the
current need is revealed by the last item, we want to aggregate the items in
the session while paying more attention on those items which are highly
correlated to the last one. Thus, we can treat the last item in the session as
the query and the sequence of items in the session as both keys and values,
leading to the design of the self-attention layer in Figure 2.

For a session interacted with $[i_{s,1},i_{s,2},...,i_{s,t}]$ sequentially, we
will lookup the corresponding node embeddings from the output of the
hypergraph attention networks to get
$[\textbf{n}_{s,1}^{(L)},\textbf{n}_{s,2}^{(L)},...,\textbf{n}_{s,t}^{(L)}]$.
We will transform the item embeddings with $\textbf{W}_{Q}$, $\textbf{W}_{K}$
and $\textbf{W}_{V}$ to generate the query vectors, key vectors, and value
vectors correspondingly. Then we can aggregate the sequence of embeddings
with:

(3.4) $\displaystyle\textbf{h}_{s}$ $\displaystyle=\sum_{i\leq
t}\sigma_{ti}\textbf{W}_{V}\textbf{n}_{s,i}^{(L)}$ $\displaystyle\sigma_{ti}$
$\displaystyle=\frac{S(\textbf{W}_{Q}\textbf{n}_{s,t}^{(L)},\textbf{W}_{K}\textbf{n}_{s,i}^{(L)})}{\sum_{j\leq
t}S(\textbf{W}_{Q}\textbf{n}_{s,t}^{(L)},\textbf{W}_{K}\textbf{n}_{s,j}^{(L)})}$

in which the attention scores $S$ is defined in Equation 3.2. According to
observations in previous research [14, 32], the order of items is less likely
to be related to the general interest in short-term sequences and could
introduce noise in modeling such short-term sequences. We omit the order
information while handling the item sequences in session-based recommendation.

With this carefully-designed self-attention layer, the resulting
$\textbf{h}_{s}$ encodes both the general interest and the current need for
session $s$. Then we compute the multiplication between $\textbf{h}_{s}$ and
the latent factor of item $v$ to predict the preference score of session $s$
on $v$ using $p_{sv}=\textbf{h}_{s}^{T}\textbf{i}_{v}$. Let
$\textbf{p}_{s}=[p_{s1},p_{s2},...,p_{sN}]$ denote the predicted preference
scores of session $s$ on all of the $N$ items in the system, which will be
processed with a Softmax layer to generate the final scores such that:
$\hat{\textbf{y}}_{s}=softmax(\textbf{p}_{s})$. We use a one-hot vector
$\textbf{y}_{s}=[y_{s1},y_{s2},...,y_{sN}]$ to denote the ground-truth item of
session $s$. During the training phrase, we calculate the cross-entropy loss
for all training sessions $S_{train}$ with $\mathcal{L}=-\sum_{s\in
S_{train}}\sum_{v=1}^{N}y_{sv}\log{\hat{y}_{sv}}$. Then we can train the model
using back-propagation. In the testing phase, given an unseen session, we can
construct a new hypergraph and feed it into the hypergraph attention network.
And then we can calculate its preference scores on all the items and recommend
the items ranked among the top.

##  4 Experiments

In this section, we conduct several experiments to evaluate how the proposed
SHARE model performs in session-based recommendation.

|  | # Sessions
---
(Training)
| # Sessions
---
(Testing)
# Items |  | Avg. Length
---
of Sessions
$YooChoose_{1/64}$ | 369,859 | 55,898 | 16,766 | 6.16
$YooChoose_{1/4}$ | 5,917,745 | 55,898 | 29,618 | 5.71
$Diginetica$ | 719,470 | 60,858 | 43,097 | 5.12

Table 1: Dataset Statistics.

###  4.1 Data.

We adopt two public datasets that have been widely used to evaluate session-
based recommendation: YooChoose and Diginetica [13, 16, 21, 32]. Yoochoose
contains sessions of click events from an online retailer in Europe and was
originally published as part of the 2015 RecSys Challenge. Diginetica contains
sessions of product transaction data from an online retailer and was released
as part of the 2016 CIKM Cup.

Datasets |  | S-Pop | MF | FPMC | KNN | GRU4Rec+ | NARM | STAMP | RPNet | SR-GNN | SHARE
---|---|---|---|---|---|---|---|---|---|---|---
$YooChoose_{1/64}$ | Hit@20 | 30.44 | 31.31 | 45.62 | 51.60 | 67.84 | 68.32 | 68.74 | 69.13 | 70.57$\dagger$ | 71.51$\ast$
MRR@20 | 18.35 | 12.08 | 15.01 | 21.81 | 29.00 | 28.63 | 29.67 | 30.24 | 30.94$\dagger$ | 31.45$\ast$
$YooChoose_{1/4}$ | Hit@20 | 27.08 | 3.40 | 51.86 | 52.31 | 69.11 | 69.73 | 70.44 | 70.71 | 71.36$\dagger$ | 72.25$\ast$
MRR@20 | 17.75 | 1.57 | 17.50 | 21.70 | 29.22 | 29.23 | 30.00 | 31.03 | 31.89$\dagger$ | 32.11$\ast$
$Diginetica$ | Hit@20 | 21.06 | 5.24 | 26.53 | 35.75 | 46.16 | 49.70 | 45.64 | 47.79 | 50.73$\dagger$ | 52.73$\ast$
MRR@20 | 13.68 | 1.98 | 6.95 | 11.57 | 14.69 | 16.17 | 14.32 | 17.66$\dagger$ | 17.59 | 18.05$\ast$

Table 2: Comparison of Different Models. All the results are in percentage
(%). The best performing method in each column is boldfaced, and the second
best method is marked with $\dagger$. $\ast$ indicates that the improvement of
the best result is statistically significant compared with the next-best
result with $p<0.05$.

We keep sessions with length longer than 1 and items which appear in at least
5 sessions. For Yoochoose, we test on all the sessions happening on the last
day in the dataset while training all the sessions before that. Only items
appearing in the training set are considered. As for Diginetica, we split the
dataset and use the sessions happening in the last 7-days for testing.
Furthermore, we adopt a standard sequence preprocessing method used in
previous work [19, 25, 32] to generate the session sequences and labels. Then
we get the final Diginetica dataset as described in Table 1. However, it is
not necessary to train on the entire set of training sequences from YooChoose
since it is extremely large and training only on a fraction of recent sessions
can lead to better prediction performance based on the experimental results in
[25]. As in [16, 19, 32], we sort all of the training sequences generated from
YooChoose, and retrieve the most recent 1/64 and 1/4 to be the training
samples in $YooChoose_{1/64}$ and $YooChoose_{1/4}$ (listed in Table 1). Note
that $YooChoose_{1/64}$ and $YooChoose_{1/4}$ share the same set of testing
samples. In addition, for fair comparison, the training samples and testing
samples in all of the three datasets are exactly the same as in [16, 19, 21,
32].

###  4.2 Experimental Setup

####  4.2.1 Evaluation Metrics.

We aim to evaluate how each model performs in predicting the next item in each
session. Thus, the experiments follow the leave-one-out setting with one
ground-truth item to be tested for each session in the test set. As in
previous works for session-based recommendation [16, 21, 32], we adopt both
Mean Reciprocal Rank (MRR@K) and Hit Rate (Hit@K) as evaluation metrics. Given
the ranked list of items predicted for each session, Hit@K measures the
probability that the ground-truth item is within the top-K. Let $r_{s}$ denote
the ranking of the ground-truth item for session $s$. Then, $Hit_{s}@K=1$ if
$r_{s}\leq K$ and $Hit_{s}@K=0$ otherwise. As for MRR@K, it measures the
average ranking of the ground-truth items among the lists. That is
$MRR_{s}@K=\frac{1}{r_{s}}$ if $r_{s}\leq K$ otherwise $MRR_{s}@K=0$. Then we
take the average values of MRR and Hit Rate over all the sessions in the test
set and report the results.

Methods | $YooChoose_{1/64}$ | $YooChoose_{1/4}$ | $Diginetica$
---|---|---|---
Hit | MRR | Hit | MRR | Hit | MRR
@10 | @20 | @10 | @20 | @10 | @20 | @10 | @20 | @10 | @20 | @10 | @20
(1) w/o Hypergraph | 59.79 | 70.45 | 29.46 | 30.21 | 60.13 | 70.79 | 29.77 | 30.52 | 37.57 | 50.52 | 16.17 | 17.06
(2) \+ GAT | 60.25 | 70.88 | 29.77 | 30.59 | 60.60 | 71.22 | 30.54 | 31.29 | 38.36 | 51.54 | 17.75 | 17.61
(3) \+ HyperGCN | 60.93 | 71.29 | 30.65 | 31.37 | 60.98 | 71.73 | 30.58 | 31.33 | 39.32 | 52.39 | 16.83 | 17.73
SHARE | 61.13$\ast$ | 71.51$\ast$ | 30.67$\ast$ | 31.45$\ast$ | 61.52$\ast$ | 72.25$\ast$ | 30.96$\ast$ | 32.11$\ast$ | 39.52$\ast$ | 52.73$\ast$ | 17.12$\ast$ | 18.05$\ast$

Table 3: Ablation Test Result. All the results are in percentage (%). $\ast$
indicates that the improvement of the best result is statistically significant
compared with the next-best result with $p<0.05$.

####  4.2.2 Baselines.

  * •

S-Pop: This simple baseline recommends the most popular items based on their
popularity in the current session. Ties are broken up using the popularity
values based on the whole training set.

  * •

KNN [24]: This method recommends items which are most similar to items clicked
in the current session. Each item is represented with a binary vector
indicating all the sessions it appears in and cosine similarity is used to
define their similarity.

  * •

MF [22]: This matrix factorization-based model is trained with Bayesian
personalized ranking loss.

  * •

FPMC [23]: Building on top of both Matrix Factorization and Markov Chains, it
is able to infer the next item based on sequential behaviors.

  * •

GRU4Rec+ [12]: This baseline utilizes Gated Recurrent Unit (GRU) to model
sequential patterns, with a new ranking loss functions tailored to RNNs in
session-based recommendation.

  * •

NARM [16]: Besides learning the sequential behavior with GRU, it also includes
an attention layer to extract the session’s main purpose, which are combined
together to infer the preference scores.

  * •

STAMP [19]: This model combines both general interests from long-term memory
and current interests from the short-term memory of the last-clicks in
session-based recommendation.

  * •

RPNet [21]: Considering the repeat consumption in each session, this model can
recommend unclicked items in the explore mode while recommending repeated
items in the repeat mode.

  * •

SR-GNN [32]: This is the state-of-the-art for session-based recommendation
with graph neural networks. It models session sequences as graph-structured
data and designs a graph neural network to capture the transition between
items. Recommendation is made with the composition of the whole session and
the last click.

####  4.2.3 Parameter Tuning.

All of the experiments are conducted on a server machine equipped with a 12 GB
Titan Xp GPU. We use Adam as the optimizer with a learning rate set to be
0.001 for all datasets. The batch size for $YooChoose_{1/64}$ is 100. Since
there are more training sessions in both $YooChoose_{1/4}$ and Diginetica, we
set their batch size to be 300. For a fair comparison, the dimension of the
item embedding is set to be 100 as in the baseline methods. We grid search for
the dropout rate in {0.1, 0.2, 0.3, 0.4, 0.5, 0.6} and L2 regularization in
{$10^{-5}$, $10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$}. The dropout is set to
be 0.3 and the L2 regularization is set to be $10^{-6}$ for all the datasets.
We train each model for 50 epochs or until its performance does not improve
for the validation set after 5 epochs. We fine-tune the maximum size of the
sliding window in {2, 3, 4, 5, 6} and the number of HGAT layer in {1, 2, 3, 4}
for different datasets. As in previous works, we randomly sample 10% of the
training sessions as validation for parameter tuning.

###  4.3 Overall Evaluation.

To compare SHARE with the baseline models, we summarize the overall results in
Table 2. As in previous work [16, 19, 32], we report MRR and Hit Rate at K=20
for the overall comparison with the baselines. We find that SHARE outperforms
all of the baselines under each of the metrics for session-based
recommendation over all of the datasets.

Starting from the simplest method, since MF is unable to capture sequential
information, it performs poorly in session-based recommendation. Integrating
with Markov Chains, FPMC can improve MF, but it is still not a good fit for
modeling the short-term user behaviors and performs worse than KNN. With
advances in neural networks, we observe that GRU4Rec+ achieves much better
results than the non-neural models in sequential pattern modeling. However, in
comparison to GRU4Rec+, we see that NARM and STAMP are better suited to
extract the general interests from each short-term anonymous session by
aggregating items with weighted attention scores. Emphasizing the item clicked
most recently helps STAMP achieve better performance than NARM. Furthermore,
by predicting and modeling the repeat consumption in sessions, RPNet can
outperform the attention-based and GRU-based models in this recommendation
scenario.

As the state-of-the-art for session-based recommendation, SR-GNN outperforms
RPNet, verifying the effectiveness of modeling item transitions with
appropriate graph structure to obtain accurate user representations in
sessions. However, the conventional graph structure is insufficient in
capturing the correlations amongst items across different contextual windows.
Since SHARE is specifically designed with this challenge in mind, we observe
that it achieves significant improvements compared with SR-GNN. Further, we
observe that the performance of SHARE on $YooChoose_{1/64}$ is even better
than SR-GNN on $YooChoose_{1/4}$. Though $YooChoose_{1/64}$ and
$YooChoose_{1/4}$ contain different numbers of training sessions, they are
tested on exactly the same set of sessions. That is, with much less training
data, SHARE can recommend more accurately than the state-of-the-art model
trained with more data.

Figure 3: Size of contextual windows vs. Hit Rate.

###  4.4 Ablation Analysis.

To examine the design of the proposed model, we conduct an ablation test to
compare SHARE with three of its variants and report the results in Table 3. In
Model (1), we remove the Hypergraph attention network from SHARE, meaning that
a set of static item embeddings are directly fed into the self-attention layer
for different sessions. Instead of utilizing the hypergraph structure, Model
(2) constructs a conventional graph for each session based on pairwise item
co-occurrence within the session and use a Graph Attention Networks (GAT) [27]
to generate the dynamic item embeddings. In Model (3), we train a hypergraph
convolution networks [6] instead of HGAT network to generate the dynamic item
embeddings. To compare them under different conditions, we report their
performance under both $K=10$ and $K=20$. Overall, SHARE can outperform all of
its variants for both K values, indicating the effectiveness of its design.

Without the hypergraph component, though it includes the self-attention layer
to learn the sequential patterns, Model (1) achieves the weakest results since
it does not consider the item correlations within each session and uses the
same embedding for an item across different sessions. Then in Model (2),
building on top of Model (1), we adopt the conventional graph structure and
GAT networks [27] to obtain dynamic item representations considering the
consecutive transitions of items in each session. This model can outperform
model (1), illustrating the necessity of modeling the in-session item
correlations with a graph-based structure. Next, to take advantage of the
hypergraph structure for modeling the complex item correlations in each
session, we construct a hypergraph for each session following the same
procedure as in SHARE. But in Model (3), we generate the item embedding with
hypergraph convolutional networks [6], which aggregates information from the
neighboring hyperedges with the convolutional operation. It can achieve better
performance than using conventional graphs, which shows the effectiveness of
the hypergraph structure and the necessity of modeling the contextual windows
for item representation learning. However, it still falls behind SHARE since
it does not consider the informativeness of nodes and the different impacts
from contextual windows.

Figure 4: Number of HGAT Layers vs. Hit Rate.

###  4.5 Study of SHARE.

In this section, we conduct further studies on three core factors in SHARE:
size of the contextual window, depth of information propagation and length of
the sessions. In the following, due to the limit of space, as
$YooChoose_{1/64}$ and $YooChoose_{1/4}$ shares the same set of testing
sessions, we show the results trained with $YooChoose_{1/4}$ to uncover the
patterns for YooChoose.

Size of contextual window. We first examine the performance of SHARE by
varying the maximum size of contextual windows that we consider while
constructing the hypergraphs (in Section 3.2). We use the 1-layer HGAT network
to avoid the influence of high-order information propagation. Note that
maximum size = 1 is similar to the case that no hypergraph is constructed for
each session, meaning that items use static embeddings across sessions. As in
Figure 3, for YooChoose, at the beginning, the performance of SHARE improves
as contextual windows with larger size are incorporated into the hypergraphs.
In Diginetica, the best choice of window size is smaller that in YooChoose.
The reason could be that the length of sessions is shorter in Diginetica, and
thus we need only consider smaller contextual windows. These observations show
that contextual windows can have significant contribution on characterizing
items in each session and the proposed framework is effective in modeling
various contextual windows simultaneously.

High-order Propagation. We can stack multiple HGAT layers to model the
information flowing among items with high-order connections in hypergraph. To
visualize the high-order information propagation, while setting the maximum
window size to be constant (i.e., 2), we show the resulting Hit Rate in Figure
4 by varying the number of HGAT layers in SHARE. In each dataset, starting
with a single HGAT layer, we find that the performance is improved by stacking
one more layer, indicating the importance of modeling the information via
high-order connections. For Diginetica, stacking more than two layers will
worsen the performance since the sessions in this dataset are short in
general. Using too many layers will bring in noise for the representation
learning process. Meanwhile, the optimized number of layers for YooChoose is
larger than that for Diginetica. Thus, we conclude that SHARE can capture both
the direct and high-order connections among items in the hypergraphs and lead
to accurate recommendation.

Short vs. Long Sessions. To fully understand how SHARE performs in modeling
sessions under different circumstances, we group the sessions based on their
length and test how SHARE and SR-GNN perform for each group of sessions. As in
Figure 5, SHARE provides better recommendations for all session lengths in
comparison with SR-GNN. Both models can make more accurate predictions for
short action sequences than the longer ones, since they omit the order
information which may be important for long sessions with adequate user
behaviors. However, since the percentage of long sessions is low in real-world
datasets for session-based recommendation, the main purpose is to boost the
recommendation for sessions with fewer items. Indeed, the proposed framework
is suitable for such practical scenarios (i.e., short-term sessions)
characterized by only a limited number of sequential actions.

Figure 5: Session Length vs. Recommendation Quality.

##  5 Conclusion and Future Work

To recommend the next interesting items in short-term sessions, we are
motivated to exploit the correlations among items within various contextual
windows in each session to better model their dynamic meanings across
sessions. In this work, we propose a novel session-based recommendation system
– SHARE – which is empowered by the hypergraph structure and hypergraph
attention networks. With experiments on three real-world benchmark datasets,
we find that the proposed SHARE is able to generate effective session-wise
item embeddings and thus provide more accurate recommendation compared with
baseline models. While in this work we mainly studying how to generate
session-wise item representations, in the future we want to explore how to
capture the item dynamics along time and over different domains in session-
based recommendation. Meanwhile, we are also interested in extending the
proposed hypergraph attention networks for representation learning in other
research areas.

## References

  * [1] S. Bai, F. Zhang, and P. H. Torr, Hypergraph convolution and hypergraph attention, arXiv preprint arXiv:1901.08150, (2019).
  * [2] R. v. d. Berg, T. N. Kipf, and M. Welling, Graph convolutional matrix completion, arXiv preprint arXiv:1706.02263, (2017).
  * [3] K. Ding, J. Li, R. Bhanushali, and H. Liu, Deep anomaly detection on attributed networks, in SDM, 2019.
  * [4] K. Ding, J. Wang, J. Li, D. Li, and H. Liu, Be more with less: Hypergraph attention networks for inductive text classification, in EMNLP, 2020\.
  * [5] W. Fan, Y. Ma, Q. Li, Y. He, E. Zhao, J. Tang, and D. Yin, Graph neural networks for social recommendation, in WWW, 2019.
  * [6] Y. Feng, H. You, Z. Zhang, R. Ji, and Y. Gao, Hypergraph neural networks, in AAAI, 2019.
  * [7] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, Neural message passing for quantum chemistry, in ICML, 2017.
  * [8] W. Hamilton, Z. Ying, and J. Leskovec, Inductive representation learning on large graphs, in NeurIPS, 2017.
  * [9] R. He, W.-C. Kang, and J. McAuley, Translation-based recommendation, in RecSys, 2017.
  * [10] X. He, Z. He, X. Du, and T.-S. Chua, Adversarial personalized ranking for recommendation, in SIGIR, 2018.
  * [11] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, Neural collaborative filtering, in WWW, 2017.
  * [12] B. Hidasi and A. Karatzoglou, Recurrent neural networks with top-k gains for session-based recommendations, in CIKM, 2018.
  * [13] B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, Session-based recommendations with recurrent neural networks, in ICLR, 2016.
  * [14] W.-C. Kang and J. McAuley, Self-attentive sequential recommendation, in ICDM, 2018.
  * [15] T. N. Kipf and M. Welling, Semi-supervised classification with graph convolutional networks, arXiv preprint arXiv:1609.02907, (2016).
  * [16] J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, and J. Ma, Neural attentive session-based recommendation, in CIKM, 2017.
  * [17] J. Li, Y. Wang, and J. McAuley, Time interval aware self-attention for sequential recommendation, in WSDM, 2020.
  * [18] G. Linden, B. Smith, and J. York, Amazon. com recommendations: Item-to-item collaborative filtering, IEEE Internet computing, 7 (2003), pp. 76–80.
  * [19] Q. Liu, Y. Zeng, R. Mokhosi, and H. Zhang, Stamp: short-term attention/memory priority model for session-based recommendation, in KDD, 2018\.
  * [20] R. Pasricha and J. McAuley, Translation-based factorization machines for sequential recommendation, in RecSys, 2018.
  * [21] P. Ren, Z. Chen, J. Li, Z. Ren, J. Ma, and M. de Rijke, Repeatnet: A repeat aware neural recommendation machine for session-based recommendation, in AAAI, 2019.
  * [22] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, Bpr: Bayesian personalized ranking from implicit feedback, in UAI, 2009.
  * [23] S. Rendle, C. Freudenthaler, and L. Schmidt-Thieme, Factorizing personalized markov chains for next-basket recommendation, in WWW, 2010.
  * [24] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, Item-based collaborative filtering recommendation algorithms, in WWW, 2001.
  * [25] Y. K. Tan, X. Xu, and Y. Liu, Improved recurrent neural networks for session-based recommendations, in Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, 2016.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in NeurIPS, 2017.
  * [27] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, Graph attention networks, 2018.
  * [28] J. Wang and J. Caverlee, Recurrent recommendation with local coherence, in WSDM, 2019.
  * [29] J. Wang, K. Ding, L. Hong, H. Liu, and J. Caverlee, Next-item recommendation with sequential hypergraphs, in SIGIR, 2020.
  * [30] J. Wang, R. Louca, D. Hu, C. Cellier, J. Caverlee, and L. Hong, Time to shop for valentine’s day: Shopping occasions and sequential recommendation in e-commerce, in WSDM, 2020.
  * [31] X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua, Neural graph collaborative filtering, in SIGIR, 2019.
  * [32] S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, Session-based recommendation with graph neural networks, in AAAI, 2019.
  * [33] L. Zheng, C.-T. Lu, F. Jiang, J. Zhang, and P. S. Yu, Spectral collaborative filtering, in RecSys, 2018.
  * [34] Z. Zhu, J. Wang, and J. Caverlee, Improving top-k recommendation via jointcollaborative autoencoders, in WWW, 2019.


# Building a great multi-lingual teacher with sparsely-gated mixture of
experts for speech recognition

###### Abstract

The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity
with a little computational complexity. In this work, we investigate how
multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up
with a simple routing algorithm in order to achieve better accuracy. More
specifically, we apply the sparsely-gated MoE technique to two types of
networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer
(T-T). We demonstrate through a set of ASR experiments on multiple language
data that the MoE networks can reduce the relative word error rates by 16.3%
and 4.6% with the S2S-T and T-T, respectively. Moreover, we thoroughly
investigate the effect of the MoE on the T-T architecture in various
conditions: streaming mode, non-streaming mode, the use of language ID and the
label decoder with the MoE.

Index Terms: Multi-lingual speech recognition, End-to-end model, Mixture of
experts, Transformers

##  1 Introduction

There has been a great deal of attention to multi-lingual automatic speech
recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. Recent advances of end-to-
end (E2E) ASR techniques have simplified multi-lingual ASR with a single
network [5, 6, 7, 12, 13, 14, 17, 18]. Such a multi-lingual ASR model has a
potential of improving accuracy on low resource language and code-switching
data [9, 13]. The majority of multi-lingual ASR work focuses on the efficient
use of multi-lingual training data through self-supervised learning [12, 15,
19], combination of unsupervised and supervised learning [10, 11, 18, 20],
meta-transfer learning [12], multi-task learning [3, 4, 8], adapter networks
[9, 21] or configurable network [14]. Nowadays a large amount of multiple
language data is usually available for multi-lingual ASR. Yet, the capacity of
conventional E2E ASR networks will not be sufficient to digest large-scale
multi-lingual information including many domains [16, 17]. In [17], Xiao et
al. built a gigantic dense network with a massive amount of data. Such a dense
model is proven to be powerful but results in an explosion in training and
inference costs. Recently, sparsely gated mixture of experts (MoE) networks
have been proposed as a computationally efficient way to increase the model
capacity [22, 23]. In this work, we investigate the effect of the MoE on
multi-lingual speech recognition accuracy.

In [24, 25], You et al. utilized a variant of MoEs to the mere stacks of the
Transformer layers in the Connectionist Temporal Classification (CTC)
framework for ASR. They reported a prominent improvement by modifying the loss
terms and routing algorithm for mono-lingual ASR tasks. Performance on the
multi-lingual data would be limited because of the simple Transformer
architecture trained based on the CTC loss. For multi-lingual ASR, Gaur et al.
investigated the effect of the MoE with the RNN-T network [13]. They proposed
the mixture of informed experts (MIE) by specializing the experts on data from
a particular language. Despite of a relatively small LSTM-based model, they
achieved an improvement in word error rate (WER) of up to 9% relative on
lower-resource languages while maintaining performance on high resource
languages. It is worth noting that [14] attempts at minimizing the multi-
lingual E2E network size by manually hardcoding assignment between a specific
expert and language.

In this work, we apply the MoE to two types of popular E2E Transformers,
sequence-to-sequence Transformer (S2S-T) and Transformer Transducer (T-T), for
multi-lingual ASR. Moreover, we further increase the capacity of those E2E
Transformer networks in contrast to the small RNN-T [13] or fixed routing
mechanism based on language input [14]. We investigate the effect of
recognition accuracy with respect to the network capacity increased with the
MoE technique.

In section 2, we review the Switch Transformer architecture that will be
adopted in this work. Section 3 describes the end-to-end ASR networks with the
MoE for our studies. Section 4 describes ASR experiments on multiple
languages. We conclude this work and describe our future plan in section 5.

##  2 Sparsely-gated mixture of experts (MoE)

Figure 1: Schematic diagram of MoE Transformer encoder

As the name indicates, the MoE layer typically consists of a set of $N$
_expert networks_ and routes an input representation $\mathbf{x}$ to the
$K$-best experts out of $N$ experts based on the _gating network_ output [22].
Denoting $G(\mbox{$\mathbf{x}$})_{i}$ and $E_{i}(\mbox{$\mathbf{x}$})$ as the
$i$-th best output of the gating network and the output of the $i$-th best
expert network on input $\mathbf{x}$, the MoE layer output $\mathbf{y}$ can be
expressed in the weighted sum of the experts as

$\mbox{$\mathbf{y}$}=\sum_{i}^{K}G(\mbox{$\mathbf{x}$})_{i}E_{i}(\mbox{$\mathbf{x}$})\vspace{-1.1em}$
(1)

Depending on the sparsity of $G(\mbox{$\mathbf{x}$})$, computation of the MoE
can be saved.

In contrast to [22], Fedus et al. [23] adopted a simple switching strategy,
routing the data to only a single expert instead of using the multiple
experts. They applied a simple switching router to the Feed-Forwarding Network
(FFN) layer. Figure 1 shows a block chart of the MoE with a switching router,
also known as Switch Transformer. Since only the single expert is selected,
Switch Transformer does not increase computational complexity much. An only
additional cost will be computing the gating function.

A set of experts or each expert can be assigned to each device and feed-
forwarding computation is performed on the selected expert. The data samples
are routed to the expert with the highest router probability. Here, we follow
[23] to control the number of the data samples computed with each expert
through the _capacity factor_ as

$\displaystyle\text{expert capacity}=\left(\frac{\text{samples per
batch}}{\text{number of experts}}\right)\times\text{capacity
factor}\vspace{-1.1em}$

When the capacity factor is greater than 1.0, buffer will be created to handle
the case that the input data are not perfectly balanced across experts. If the
input data samples are unevenly distributed to the expert, some of data
samples will not be assigned to any experts and data overflows occur. The
overflown data will be passed directly to the next layer through the residual
connection without being processed by any experts. In other words, the data
will be dropped at the expert layer in the case that too many samples are
routed to a single expert. A larger capacity factor keeps the data overflow
from happening, but increasing the capacity factor will lead to the
inefficient use of computation and memory. In the following experiment, we set
the capacity factor to 1.5.

To balance a data load across the experts, the auxiliary loss is typically
added [22, 23]. Following [23], we compute the auxiliary loss given a batch
$\mathcal{B}$ and $T$ samples as:

$\displaystyle\text{aux. loss}=\alpha N\cdot\sum_{i=1}^{N}f_{i}\cdot
P_{i}\vspace{-1.1em}$ (2)

where $f_{i}$ is the fraction of samples dispatched to expert $i$,

$\displaystyle
f_{i}=\frac{1}{T}\sum_{\mbox{$\mathbf{x}$}\in\mathcal{B}}\mathbb{1}\\{\mathop{\rm
argmax}G(\mbox{$\mathbf{x}$}),i\\}\vspace{-1.1em}$ (3)

$P_{i}$ indicates the fraction of the router probability allocated for expert
$i$ and is computed by averaging the probability of routing sample
$\mathbf{x}$ to expert $i$ as,

$\displaystyle
P_{i}=\frac{1}{T}\sum_{\mbox{$\mathbf{x}$}\in\mathcal{B}}G(\mbox{$\mathbf{x}$})_{i}\vspace{-1.1em}$
(4)

In this work, we set $\alpha=\text{0.01}$.

We also introduce the _switching jitters_ , a small variable to control the
multiplicative fraction to the gate input as
$\mbox{$\mathbf{x}$}*=\text{uniform}(1-\epsilon,1+\epsilon)$. We use
$\epsilon=\text{0.01}$ in all the experiments reported in the next section.

##  3 End-to-end speech recognition network

In this section, we first describe the S2S Transformer network and explain how
the MoE is applied to the S2S network. We then describe the T-T network with
the MoE.

###  3.1 Sequence-to-sequence Transformer (S2S-T)

Figure 2 depicts the network architecture of the S2S-T for end-to-end ASR. As
illustrated in Figure 2, the S2S-T consists of the convolution subsampling
layer, the audio encoder, the label token embedding layer and the decoder.
Each encoder block is composed of two sub-layers: the self-attention module
followed by the MoE FFN module. We employ Multi-Headed Attention (MHA) used in
[26] here. Each decoder layer consists of three sub-layers: self-attention,
encoder-decoder attention and the MoE FFN module. As shown in Figure 2, each
Transformer layer has residual connections around each of the sub-layers,
layer normalization and dropout. Note that we employ the pre-layer norm
residual units, passing the data before layer normalization. The S2S-T network
used here does not mask attention input so that the attention module can
observe the whole context.

The FFN module is conventionally composed of the linear layer, a nonlinear
activation function followed by the linear layer without the MoE layer, as
shown on the left side in Figure 1. In contrast to normal practice, we
introduce the MoE FFN module, as depicted in the right block in Figure 1. We
also apply ReLU activation and dropout between the MoE layers.

Figure 2: MoE Sequence-to-sequence Transformer (S2-T)

###  3.2 Transformer Transducer (T-T)

The S2S-T can achieve better accuracy in many sequence-to-sequence
classification tasks. However, maintaining accuracy with low latency in a
streaming task is still a challenge [27]. On the contrary, the T-T network is
suitable for the streaming task [28]. For that reason, we also investigate the
effect of the MoE on the T-T network.

The main difference between the T-T and S2S network is that the transducer
network provides a probability distribution over the label space at each time
step while the S2S model attends over the whole input for every prediction in
the output label sequence. This is why the use of the T-T is much more
straightforward than the adoption of the S2S network in the streaming tasks.
The output label of the transducer network includes an additional null label
to indicate the lack of output at that time step in a similar manner with the
CTC framework [29]. In contrast to CTC, the transducer networks can condition
the label distribution on the label history [30].

Figure 3 shows the T-T architecture adopted in this work.

Figure 3: MoE Transformer Transducer (T-T)

As illustrated in Figure 3, our T-T network consists of the audio encoder,
label decoder and joint network. The audio encoder block is composed of the
convolution subsampling layer followed by a stack of the Transformer blocks
with the relative positional embedding [26]. Each encoder layer contains two
sub-layers: the MHA layer and the feed-forward network layer. Layer
normalization is applied prior to the MHA layer. We use the relative position
differences between input elements for multi-headed attention. In contrast to
absolute position representations, the relative position representation is
invariant to the total sequence length. This attention mechanism allows us to
control the history length and latency of recognition output by masking the
attention score of the left and right frames. To encourage gradient flow, we
put a residual connection between the layer normalization input and dropout
output for each sub-layer. In this work, we replace the conventional FFN layer
with the MoE FFN layer. Our label decoder is composed of the embedding layer
followed by a stack of LSTM layers [31]. In the following experiment with T-T,
we also consider replacing the LSTM projection layer in the label decoder with
the MoE FFN module. Similar to the Transformer layer, we perform pre-layer
normalization and post-dropout for the LSTM layer and the MoE FFN module.
Residual connections are added to both layers to propagate gradient
information. The joint network linearly combines the audio encoder output at
every time step and the label decoder output given the previous non-blank
output label sequence. The combined output is fed to the ReLU activation
function, a linear layer and the Softmax layer. The transducer loss is
efficiently computed by using the forward algorithm [30].

##  4 Experiment

###  4.1 Datasets

The dataset used for training consists of 10 languages totaling approximately
75 thousand hours. As shown in Table 1, our training dataset is comprised of
data from English, Spanish, Romanian, German, Italian, French, Portuguese,
Dutch, Polish and Greek. It is worth noting here that we report the
experimental results for multiple variants of English (US and UK), Spanish
(ES-ES and ES-MX) and Romanian (RO). Table 2 tabulates the statistics of our
test set for each locale: number of utterances and number of words. The
dataset of each locale also contains not only various speakers but also
different speech recognition tasks such as command-and-control tasks in
mobile, office and car scenarios, Cortana phrases, dictation, conversational
speech in telecommunication and so on. We consider the English test, Spanish
test and Romanian test as a high-resource task, mid-resource task and low-
resource task, respectively.

For training a network, we further split the whole training data set into
training and validation sets in order to determine the convergence. The amount
of training data for each language is different as shown in Table 1. We, thus,
sample the lower resource data more frequently to balance the language data
distribution during training. For all the experiments reported here, we used a
80-dimensional log filter-bank energy feature extracted at an interval of
every 10ms. 10014 unique BPE [32] tokens are used for covering vocabulary of
10 languages.

Language | Hrs
---|---
English (EN) | 38585
Spanish (ES) | 7584
German (DE) | 4893
French (FR) | 5955
Italian (IT) | 6580
Portuguese (PT) | 4302
Polish (PL) | 2112
Greek (EL) | 2190
Romanian (RO) | 1899
Dutch (NL) | 880
Table 1: Duration of Training data: 10 language data Language | Code | No. utterances | No. words
---|---|---|---
English | EN-US | 219965 | 1374748
EN-UK | 16743 | 79191
Spanish | ES-MX | 15279 | 124742
ES-ES | 19275 | 150980
Romanian | RO | 16626 | 365828
Table 2: Test dataset details

###  4.2 Experiments

Model | #experts | #params | Lang. ID | EN-US | EN-UK | RO | ES-ES | ES-MX | Overall |
---|---|---|---|---|---|---|---|---|---|---
S2S Transformer | N/A | 98.9M | No | 10.7 | 9.44 | 23.1 | 15.0 | 15.3 | 13.43 | ${}_{1\text{st}}$
MoE S2S Transformer | 24 | 677M | No | 9.68 | 9.31 | 18.9 | 14.1 | 14.7 | 11.92 | ${}_{2\text{nd}}$
72 | 1.88B | No | 9.25 | 7.83 | 18.2 | 12.3 | 13.6 | 11.24 | ${}_{3\text{rd}}$
120 | 3.09B | No | 9.00 | 7.83 | 18.5 | 13.6 | 13.6 | 11.29 | ${}_{4\text{th}}$
Table 3: WERs of sequence-to-sequence Transformers (S2S-T) with respect to the number of experts. Model | #exprts | #params | Streaming | Lang. ID | EN-US | EN-UK | RO | ES-ES | ES-MX | Overall |
---|---|---|---|---|---|---|---|---|---|---|---
Audio | Label |
T-T | N/A | N/A | 87.3M | Yes | No | 11.2 | 9.15 | 23.3 | 16.4 | 17.6 | 14.02 | ${}_{1\text{st}}$
87.7M | Yes | Yes | 10.8 | 8.90 | 22.8 | 15.1 | 15.6 | 13.45 | ${}_{2\text{nd}}$
|  |  | 87.7M | No | Yes | 9.76 | 7.54 | 18.7 | 11.0 | 11.5 | 11.46 | ${}_{3\text{rd}}$
MoE T-T | 24 | N/A | 521M | Yes | Yes | 9.58 | 8.57 | 21.4 | 14.5 | 14.8 | 12.87 | ${}_{4\text{th}}$
No | No | 9.83 | 7.74 | 17.5 | 11.3 | 12.2 | 11.35 | ${}_{5\text{th}}$
|  |  |  | No | Yes | 9.52 | 7.34 | 16.9 | 10.7 | 11.3 | 10.93 | ${}_{6\text{th}}$
|  | 24 | 621M | No | Yes | 9.51 | 7.38 | 16.8 | 10.5 | 11.1 | 10.93 | ${}_{7\text{th}}$
Deeper T-T | N/A | N/A | 144M | Yes | Yes | 10.0 | 7.86 | 20.4 | 13.0 | 13.7 | 12.18 | ${}_{8\text{th}}$
Deeper MoE T-T | 24 | N/A | 1.01B | Yes | Yes | 10.1 | 8.18 | 19.0 | 12.8 | 13.1 | 12.04 | ${}_{9\text{th}}$
72 | N/A | 2.82B | Yes | Yes | 9.67 | 7.88 | 19.3 | 12.5 | 13.3 | 11.79 | ${}_{10\text{th}}$
| 24 | N/A | 1.01B | No | Yes | 9.17 | 6.89 | 14.8 | 9.50 | 10.1 | 10.17 | ${}_{11\text{th}}$
| 72 | N/A | 2.82B | No | Yes | 9.19 | 6.99 | 14.8 | 9.42 | 10.1 | 10.17 | ${}_{12\text{th}}$
Table 4: WERs with various Transformer Transducer (T-T) architectures.

####  4.2.1 S2S-T model

The S2S-T baseline model has 18 encoder layers and 6 decoder layers. The
dropout probability is 0.1 and the hidden size is 2048 in the encoder and
decoder. Each MHA layer has 8 heads. The number of FLOPs for the MoE S2S-T
network is matched to that of the S2S-T baseline network. The MoE FFN is
applied to every two Transformer layers. All the models were trained with the
AdamW optimizer. The learning rate schedule and batch size were kept the same
for all the experiments. Training was done by using 24 GPUs with three ND A100
v4 Azure Virtual Machines (VMs) or three NCv3-series VMs.

Table 3 shows the word error rate (WER) on each locale and the overall WERs
for each model architecture: the conventional S2S-T model and its MoE versions
with 24 experts, 72 experts and 120 experts. Here, the overall WER is computed
by averaging the WERs weighted with the number of the test words in each
locale set. In Table 3, the second column and the third column indicate the
number of experts and the number of trainable parameters, respectively.
Language ID information is not used for generating the numbers in Table 3. It
is clear from Table 3 that the accuracy can be improved by applying the MoE
FFN layer to each Transformer layer. In these experiments, the S2S-T model
with 72 experts provides the best accuracy although the difference between 72
and 120 experts is not that significant. The dropout probability of each MoE
FFN is set to 0.1 in the case of 24 experts and 72 experts while it is set to
0.4 in the case of the 120 experts. The dropout of 0.1 for the 120 experts
provided the worse result because of the overfitting issue.

####  4.2.2 T-T model

The convolution sampling layer for the T-T is composed of three sub-layers:
two convolutional neural network (CNN) layers with kernel size (3,3) and
strides (2,2) and a linear layer with the output size of 512. The first CNN
has a single input channel and 512 output channels, and the second CNN has 512
input channels and 512 output channels. Our baseline audio encoder is composed
of 18 identical layers. Each layer contains two sub-layers: the
512-dimensional attention layer with 8 heads and the FFN module with the
dimension size of 2048. In the case of the streaming mode, the attention
layers only attend 18 left-contexts and 4 right-contexts. In the non-streaming
mode, whole utterance frames are attended by the network. Our baseline label
decoder is composed of a 320-dimensional embedding and two layers of the
unidirectional LSTM with 1024 hidden units. The hidden size of the MoE FFN
module for the LSTM layer is 1024. The joint network contains two linear
layers to project each input to a 512-dimensional vector and the output layer
to project to 10,015 tokens including the blank symbol.

Table 4 shows the WER obtained with different types of the T-T models on each
locale and the overall WER weighted-averaged over all the locales. Here, we
also investigate the effect of language ID information on the model. As the
language ID feature, we use the one-hot vector in the same way as [12]. It is
clear from the 1st and 2nd rows in Table 4 that the use of the language ID
improves recognition accuracy for the non-MoE T-T. It is also clear from the
2nd and 3rd rows in Table 4 that the non-streaming T-T model can reduce the
overall WER by 14.7% relative to the streaming model. By comparing the numbers
in the 2nd row with those in the 4th row and the WERs in the 3rd row with
those in the 6th row, we can see that recognition accuracy can be improved by
applying the MoE to the audio encoder only. Observe that the use of input
language ID information can also improve recognition accuracy in the case of
the MoE network by contrasting the WERs in the 5th row to those in the 6th
row. We can see from the WERs in the 6th and 7th rows that the MoE FFN module
in the LSTM layer did not make a big difference. To investigate the effect of
the MoE model for a larger capacity, we further increased the number of the
encoder layers from 18 to 36 (labeled as Deeper T-T in Table 4). From Table 4,
it is clear that the overall WER can be reduced from 13.45% to 12.18% by
increasing the number of the audio encoder layers even without the MoE layer.
It is also clear that the use of the MoE can further improve recognition
accuracy. In the case of the audio encoder with 18 Transformer layers, the MoE
layer reduces the relative WER by 4.3% and 4.6% for streaming and non-
streaming modes, respectively. Moreover, the MoE layer reduces the relative
WER by 3.2% for the deeper audio encoder network. However, the gain obtained
with the MoE T-T is not as large as that with the MoE S2S-T. From the fact
that the use of the MoE in the label decoder has no impact on recognition
accuracy, we draw a conclusion that the T-T architecture does not combine
audio and label information well. In fact, the S2S-T architecture can fuse
audio and language information better by attending over the audio context
vectors in each label decoder block and bypassing information to the end
through the residual connections.

##  5 Conclusion

In this work, we have scaled up the capacity of the multi-lingual ASR network
with sparsely gated MoE. The MoE method used in this work increases
computational complexity only for gating calculation; an increase in
computational complexity is trivial. We have also demonstrated through a set
of ASR experiments on the multiple locales that the MoE can improve the
recognition accuracy. The MoE gives a larger gain with S2S Transformer than
that with the T-T network. We are going to show how the multi-lingual MoE
model can be compressed with knowledge distillation in [33, 34].

##  6 Acknowledgements

The authors would like to thank Long Wu, Rui Jiang, Liyang Lu, Peidong Wang,
Tianyu Wu, Nayuki Kanda, Yao Qian, Jinyu Li, Saeed Maleki, Baihan Huang,
Weixing Zhang, Jesse Benson, Tim Harris, Yuan Yu and Mengchen Liu for
technical discussions. We also would like to thank Ed Lin, Michael Zeng,
Xuedong Huang and Yuan Yu for their project support.

## References

  * [1] T. Schultz and A. Waibel, “Language-independent and language-adaptive acoustic modeling for speech recognition,” _Speech Communication_ , vol. 35, no. 1, pp. 31–52, 2001.
  * [2] A. Ghoshal, P. Swietojanski, and S. Renals, “Multilingual training of deep neural networks,” in _Proc. ICASSP_ , 2013.
  * [3] J. Cui, B. Kingsbury, B. Ramabhadran, A. Sethy, K. Audhkhasi, X. Cui, E. Kislal, L. Mangu, M. Nussbaum-Thom, M. Picheny, , Z. Tüske, P. Golik, R. Schlüter, H. Ney, M. J. F. Gales, K. M. Knill, A. Ragni, H. Wang, and P. Woodland, “Multilingual representations for low resource speech recognition and keyword search,” in _Proc. ASRU_. IEEE, 2015, pp. 259–266.
  * [4] D. Chen and B. K.-W. Mak, “Multitask learning of deep neural networks for low-resource speech recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 7, pp. 1172–1183, 2015.
  * [5] S. Watanabe, T. Hori, and J. R. Hershey, “Language independent end-to-end architecture for joint language identification and speech recognition,” in _Proc. ASRU_. IEEE, 2017, pp. 265–271.
  * [6] S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao, “Multilingual speech recognition with a single end-to-end model,” in _Proc. ICASSP_. IEEE, 2018, pp. 4904–4908.
  * [7] A. Waters, N. Gaur, P. Haghani, P. Moreno, and Z. Qu, “Leveraging language id in multilingual end-to-end speech recognition,” in _Proc. ASRU_. IEEE, 2019, pp. 928–935.
  * [8] S. Kim and M. L. Seltzer, “Towards language-universal end-to-end speech recognition,” in _Proc. ICASSP_ , 2018.
  * [9] A. Kannan, A. Datta, T. N. Sainath, E. Weinstein, B. Ramabhadran, Y. Wu, A. Bapna, Z. Chen, and S. Lee, “Large-scale multilingual speech recognition with a streaming end-to-end model,” _arXiv preprint arXiv:1909.05330_ , 2019\.
  * [10] C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech representation learning with labeled and unlabeled data,” 2021.
  * [11] C. Wang, Y. Wu, S. Liu, J. Li, Y. Qian, K. Kumatani, and F. Wei, “Unispeech at scale: An empirical study of pre-training method on large-scale speech recognition dataset,” _CoRR_ , vol. arXiv:2107.05233, 2021.
  * [12] R. Lahiri, K. Kumatani, E. Sun, and Y. Qian, “Multilingual speech recognition using knowledge transfer across learning processes,” _CoRR_ , vol. arXiv:2110.07909, 2021.
  * [13] N. Gaur, B. Farris, P. Haghani, I. Leal, P. J. Moreno, M. Prasad, B. Ramabhadran, and Y. Zhu, “Mixture of informed experts for multilingual speech recognition,” in _Proc. ICASSP_ , 2021.
  * [14] L. Zhou, J. Li, E. Sun, and S. Liu, “A configurable multilingual model is all you need to recognize all languages,” in _Proc. ASRU_ , 2021.
  * [15] M. Karimi, C. Liu, K. Kumatani, Y. Qian, T. Wu, and J. Wu, “Deploying self-supervised learning in the wild for hybrid automaticspeech recognition,” in _Submitted to ICASSP 2022_ , 2022.
  * [16] B. Li, R. Pang, T. N. Sainath, A. Gulati, Y. Zhang, J. Qin, P. Haghani, W. R. Huang, M. Ma, and J. Bai, “Scaling end-to-end models for large-scale multilingual asr,” _CoRR_ , vol. arXiv:2104.14830, 2021.
  * [17] A. Xiao, W. Zheng, G. Keren, D. Le, F. Zhang, C. Fuegen, O. Kalinli, Y. Saraf, and A. Mohamed, “Scaling asr improves zero and few shot learning,” _CoRR_ , vol. arXiv:2111.05948, 2021.
  * [18] J. Bai, B. Li, Y. Zhang, A. Bapna, N. Siddhartha, K. C. Sim, and T. N. Sainath, “Joint unsupervised and supervised training for multilingual ASR,” _CoRR_ , vol. arXiv:2111.08137, 2021.
  * [19] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _CoRR_ , vol. abs/2006.13979, 2020. [Online]. Available: https://arxiv.org/abs/2006.13979
  * [20] S. Raghavan and K. Shubham, “Hybrid unsupervised and supervised multitask learning for speech recognition in low resource languages,” in _Proc. Workshop on Machine Learning in Speech and Language Processing_ , 2021.
  * [21] W. Hou, Y. Wang, S. Gao, and T. Shinozaki, “Meta-adapter: Efficient cross-lingual adaptation with meta-learning,” in _Proc. ICASSP_ , 2021.
  * [22] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in _ICLR_ , 2017. [Online]. Available: https://openreview.net/pdf?id=B1ckMDqlg
  * [23] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” _CoRR_ , vol. arXiv:2101.03961, 2021.
  * [24] Z. You, S. Feng, D. Su, and D. Yu, “SpeechMoE: Scaling to large acoustic models with dynamic routing mixture of experts,” in _Proc. Interspeech_ , 2021.
  * [25] ——, “SpeechMoE2: Mixture-of-experts model with improved routing,” _CoRR_ , vol. arXiv:2111.11831, 2021.
  * [26] J. Li, Y. Wu, Y. Gaur, C. Wang, R. Zhao, and S. Liu, “On the comparison of popular end-to-end models for large scale speech recognition,” in _Proc. Interspeech_ , 2020.
  * [27] H. Inaguma, Y. Gaur, L. Lu, J. Li, and Y. Gong, “Minimum latency training strategies for streaming sequence-to-sequence ASR,” in _Proc. ICASSP_ , 2020.
  * [28] Q. Zhang, H. Lu, H. Sak, A. Tripathia, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,” in _Proc. ICASSP_ , 2020.
  * [29] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proc. ICML_ , 2006.
  * [30] A. Graves, “Sequence transduction with recurrent neural networks,” in _Proc. ICML_ , 2012.
  * [31] H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in _Proc. Interspeech 2014_ , 2014, pp. 338–342.
  * [32] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1715–1725. [Online]. Available: https://aclanthology.org/P16-1162
  * [33] J. H. M. Wong, M. J. F. Gales, and Y. Wang, “General sequence teacher–student learning,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 11, pp. 1725–1736, 2019.
  * [34] K. Kumatani, D. Dimitriadis, Y. Gaur, R. Gmyr, E. S. Eskimez, J. Li, and M. Zeng, “Sequence-level self-learning with multi-task learning framework,” in _Proc. Interspeech_ , October 2020.