# HyperFormer: Learning Expressive Sparse Feature Representations via
Hypergraph Transformer

Kaize Ding*†, Albert Jiongqian Liang‡, Bryan Perrozi‡, Ting Chen‡, Ruoxi
Wang‡, Lichan Hong‡, Ed H. Chi‡, Huan Liu†, Derek Zhiyuan Cheng‡ †Arizona
State University, {kaize.ding<EMAIL_ADDRESS>
‡Google, {jiongqian, hubris, iamtingchen, ruoxi, lichan, edchi,
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Learning expressive representations for high-dimensional yet sparse features
has been a longstanding problem in information retrieval. Though recent deep
learning methods can partially solve the problem, they often fail to handle
the numerous sparse features, particularly those tail feature values with
infrequent occurrences in the training data. Worse still, existing methods
cannot explicitly leverage the correlations among different instances to help
further improve the representation learning on sparse features since such
relational prior knowledge is not provided. To address these challenges, in
this paper, we tackle the problem of representation learning on feature-sparse
data from a graph learning perspective. Specifically, we propose to model the
sparse features of different instances using hypergraphs where each node
represents a data instance and each hyperedge denotes a distinct feature
value. By passing messages on the constructed hypergraphs based on our
Hypergraph Transformer (HyperFormer), the learned feature representations
capture not only the correlations among different instances but also the
correlations among features. Our experiments demonstrate that the proposed
approach can effectively improve feature representation learning on sparse
features.

Sparse Features; Hypergraph; Graph Neural Networks

*Work was done as an intern at Google.

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††price: 15.00††doi: 10.1145/3539618.3591999††isbn:
978-1-4503-9408-6/23/07††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Machine learning††ccs: Information systems Information
retrieval

##  1\. Introduction

As one prevailing line of research for dealing with sparse features,
researchers try to model the cross features in either the raw feature level
(rendle2010factorization, ) or the embedding level (wang2021dcn, ) to improve
the representation expressiveness. Despite their success, existing methods
still have a major bottleneck of capturing the following relational
information within data: (1) Instance Correlations. Most of the existing
efforts assume training instances are independently and identically
distributed (i.i.d.), while different instances in the data may share
correlated behavior/feature patterns (zheng2022hien, ). Since the features of
an instance could be extremely sparse, leveraging the knowledge from other
instances is highly beneficial to improve the representation quality of sparse
features in a collective way. Yet, how to model the correlations between
different data instances without prior knowledge remains unexplored in this
field; (2) Feature Correlations. In general, different feature values are
usually correlated (e.g., two different feature values are commonly shared by
many instances), which should not be neglected for learning expressive feature
representations. In real-world systems, as the data scale grows continuously
in a power-law distribution, a large number of features appear very few times
in the training set, also known as tail features. As a result, existing
methods such as feature interaction learning approaches that rely on feature
co-occurrence will lose their efficacy on tail features due to their rarity in
the data (guo2021dual, ). How to better capture feature correlations is a key
to improve the representation learning of sparse features. Based on those two
limitations, one natural research question to ask is that – given the input
data with high-dimensional and sparse features, is there a natural way to
explicitly model both instance correlations and feature correlations
simultaneously?

To answer this question, we go beyond the existing learning paradigm and study
relational representation learning on data with sparse features from a graph
learning perspective. Due to the fact that the relationships among instances
are naturally high-order rather than pair-wise, e.g., a group of users sharing
the same feature ”location”, instead of using simple graph, we adopt
hypergraph (zhou2006learning, ) to model the high-order correlations among
data instances. Specifically, we take feature values as the proxy (i.e.,
hyperedges) to connect different data instances (i.e., nodes) with the same
feature values. In order to facilitate learning representations on each
hypergraph constructed from the feature-sparse data, we further develop a
plug-and-play model – Hypergraph Transformer (HyperFormer), which serves as an
embedding module and is compatible to be trained together with arbitrary
prediction model for task-agnostic sparse predictive analytics. HyperFormer
iteratively aggregates the information from hyperedges to nodes and vice
versa, which allows multi-hop message-passing on the constructed hypergraphs
and captures the instance correlations as well as feature correlations
simultaneously. The resulted feature representations can improve the
predictive power of different models on feature-sparse data. Our experiments
on (i) CTR prediction and (ii) top-K item recommendation tasks demonstrate
that HyperFormer is generalizable across different tasks, and further enhances
state-of-the-art approaches of representation learning for sparse features.

##  2\. Related Work

Figure 1. Illustration of the proposed HyperFormer.

Learning with Sparse Features has been a classic yet challenging problem in
information retrieval and recommender system. A prevailing line of research
tries to model the cross features in either the raw feature level or the
embedding level. Compared to conventional approaches (rendle2010factorization,
; juan2016field, ; pan2018field, ), deep learning models have shown their
superiority for handling high-dimensional sparse features (cheng2016wide, ;
shan2016deep, ; wang2021dcn, ; ding2019feature, ). Methods such as Wide&Deep
(cheng2016wide, ), Deep Crossing (shan2016deep, ), PNN (qu2016product, ), DCN
(wang2017deep, ), AutoInt (song2019autoint, ), Fi-GNN (li2019fi, ), DCN-v2
(wang2021dcn, ) have been proposed to automatically model the cross features.
However, existing methods are not able to explicitly capture the correlations
between instances and are also ineffective to handle the tail features that
appear rarely in the data.

Graph Neural Networks (GNNs) generally follow the neighborhood aggregation
scheme (kipf2016semi, ; hamilton2017inductive, ; velickovic2017graph, ;
ding2022data, ), learning the latent node representation via message passing
among local or high-order neighbors in the graph (klicpera2019predict, ;
ding2022meta, ; ding2023structural, ). More recently, GNNs have been actively
explored to improve the performance of CTR prediction (li2019fi, ;
li2021graphfm, ; guo2021dual, ; zheng2022hien, ) by capturing the interactions
between features. Our approach leverages the idea of hypergraph
(zhou2006learning, ; feng2019hypergraph, ; wang2020next, ; ding2020more, ;
wang2021session, ; jeong2022nothing, ) and we build HyperFormer to perform
representation learning on sparse features. In addition, our work focuses on
developing a new embedding module rather than learning the cross-features, and
our plug-and-play model is compatible to be trained with any feature
interaction learning methods for making final predictions.

##  3\. Methodology

Problem Definition. For the sake of simplicity, we only consider sparse
features and use multi-hot representation for sparse features, where each
sparse feature is also called a field. The input of our problem is a high-
dimensional sparse vector $\mathbf{x}\in\mathrm{R}^{N}$ of multi-hot
representation, where $N$ is the total number of sparse feature values. Also,
$x_{i}=0$ means the $i$-th feature value does not exist in the instance and
$x_{i}=1$ means otherwise. The objective is to learn a low-dimensional
embedding vector $\mathbf{e}\in\mathrm{R}^{d}$ that represents the raw input
features in the latent space.

Existing works apply an embedding layer to project the input features into a
low dimensional feature vector, which is commonly implemented by looking up
from an embedding table
$\mathbf{F}=[\mathbf{f}_{1},\mathbf{f}_{2},...,\mathbf{f}_{N}]\in\mathbb{R}^{N\times
d}$ and concatenating the retrieved embeddings into a dense real-value vector.
Correspondingly, $\mathbf{f}_{k}$ can be regarded as the dense representation
for feature $x_{k}$. In this paper, we argue that existing methods cannot
explicitly consider the correlations between instances and the correlations
between features, leading to the feature representations less expressive.

###  3.1. Feature Hypergraph

In this paper, we propose to alleviate the feature sparsity issue through
relational representation learning. Since each specific feature can appear in
multiple data instances, it can be naturally utilized as a bridge to capture
instance correlations as well as feature correlations. For example, a group of
users sharing the same feature values for “location” or “age”. Such
correlations among instances are inherently high-order rather than pair-wise,
thus we propose to build feature hypergraph to model the input data and try to
enable message-passing on it to capture desired relational information.
Specifically, we define the feature hypergraph as follows:

###### Definition 3.1.

Feature Hypergraph: A feature hypergraph is defined as a graph
$G=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}=\\{v_{1},\dots,v_{n}\\}$
represents the set of nodes in the graph, and
$\mathcal{E}=\\{e_{1},\dots,e_{m}\\}$ represents the set of hyperedges.
Specifically, each node represents a data instance and each hyperedge
represents a unique feature value. Correspondingly, for any hyperedge $e$, it
can connect arbitrary number of nodes/instances (i.e., $\sigma(e)\geq 1$).

Scalability Extension. Considering the fact that the scale of training data
could be extremely large in practice, it is almost impossible to build a
single feature hypergraph to handle all the data instances. To counter this
issue, we propose to construct in-batch hypergraph based on data instances in
the batch to further support mini-batch training. In Figure 1, we illustrate
the steps for constructing the in-batch hypergraphs. For each batch, we
randomly sample a batch of instances and update the hypergraph structure based
on the data samples in the batch. From our experiments, the in-batch feature
hypergraph is also effective for capturing the desired data dependencies and
achieves satisfying performance improvements.

###  3.2. Hypergraph Transformer

To support representation learning on the constructed feature hypergraphs, we
further propose a new model Hypergraph Transformer (HyperFormer) in this
paper, which adopts the Transformer-like architecture (vaswani2017attention, )
to exploit the hypergraph structure to encode both the instance correlations
and feature correlations. Apart from conventional GNN models, each layer in
HyperFormer learns the representations with two different hypergraph-guided
message-passing functions, capturing high-order instance correlations and
feature correlations simultaneously. Formally, a Transformer layer can be
defined as:

(1) $\displaystyle\mathbf{H}^{l}$
$\displaystyle=\textsc{TF}_{edge}\Big{(}\mathbf{Q}_{edge}^{l}=\mathbf{H}^{l-1},\mathbf{K}_{edge}^{l}=\mathbf{F}^{l-1},\mathbf{V}_{edge}^{l}=\mathbf{F}^{l-1}\Big{)},$
$\displaystyle\mathbf{F}^{l}$
$\displaystyle=\textsc{TF}_{node}\Big{(}\mathbf{Q}_{node}^{l}=\mathbf{F}^{l-1},\mathbf{K}_{node}^{l}=\mathbf{H}^{l},\mathbf{V}_{node}^{l}=\mathbf{H}^{l}\Big{)},$

where
$\textsc{TF}\Big{(}\mathbf{Q},\mathbf{K},\mathbf{V}\Big{)}=\text{FFN}\Big{[}\text{softmax}(\frac{\mathbf{Q}\mathbf{K}^{\mathrm{T}}}{\sqrt{d}})\mathbf{V}\Big{]}$
denotes the Transformer-like attention mechanism. In essence,
$\textsc{TF}_{edge}$ is a message-passing function that aggregates information
from hyperedges to nodes and $\textsc{TF}_{node}$ is another message-passing
function that aggregates information from nodes to hyperedges. Specifically,
we first look up from the feature embedding table $\mathbf{F}$ to initialize
hyperedge representations and the initial node representation of each instance
is computed by concatenating all its feature representations. Without loss of
generality, we describe the two message-passing functions in a single
HyperFormer layer $l$ as follows:

Feature-to-Instance Message-Passing. With all the hyperedges representations
$\\{\mathbf{f}_{j}^{l-1}\arrowvert\forall e_{j}\in\mathcal{E}_{i}\\}$, we
first apply an feature-to-instance (edge-to-node) message-passing to learn the
next-layer representation $\mathbf{h}_{i}^{l}$ of node $v_{i}$. Specifically,
we set the node representation from the last HyperFormer layer $l-1$ as the
query. The representations of the connected hyperedges can be projected into
keys and values. Formally, the similarity between the query and key can be
calculated as:

(2) $\displaystyle\alpha_{ij}$
$\displaystyle=\frac{\exp((\mathbf{h}_{i}^{l-1}\mathbf{W}^{Q}_{edge})^{\mathrm{T}}\mathbf{k}_{j})}{\sum_{e_{p}\in\mathcal{E}_{i}}\exp((\mathbf{h}_{i}^{l-1}\mathbf{W}^{Q}_{edge})^{\mathrm{T}}\mathbf{k}_{p})},\quad\mathbf{k}_{p}=\mathbf{f}_{p}^{l-1}\mathbf{W}^{K}_{edge},$

in which $\mathbf{W}^{K}_{edge}$ is the projection matrix for the key of the
feature-to-instance transformer. Then the next layer node representation can
be computed as:

(3)
$\mathbf{h}_{i}^{l}=\sigma\bigg{(}\sum_{e_{j}\in\mathcal{E}_{i}}\alpha_{ij}\mathbf{f}_{j}^{l-1}\mathbf{W}^{V}_{edge}\bigg{)},$

where $\sigma$ is the non-linearity such as ReLU and $\mathbf{W}^{V}_{edge}$
is a trainable projection matrix for the value.

Instance-to-Feature Message-Passing. With all the updated node
representations, we again apply an instance-to-feature (node-to-edge) message-
passing based on the Transformer layer to learn the next-layer representation
of hyperedge $e_{j}$. Similarly, this process can be formally expressed as:

(4)
$\mathbf{f}_{j}^{l}=\sigma\bigg{(}\sum_{v_{k}\in\mathcal{V}_{j}}\beta_{jk}\mathbf{h}_{k}^{l}\mathbf{W}^{V}_{node}\bigg{)},$

where $\mathbf{f}_{j}^{l}$ is the output representation of hyperedge $e_{j}$
and $\mathbf{W}^{V}_{node}$ is the projection matrix. $\beta_{jk}$ denotes the
attention score of hyperedge $e_{j}$ on node $v_{k}$, which can be computed
by:

(5) $\displaystyle\beta_{jk}$
$\displaystyle=\frac{\exp((\mathbf{f}_{j}^{l-1}\mathbf{W}^{Q}_{node})^{\mathrm{T}}\mathbf{k}_{k})}{\sum_{v_{p}\in\mathcal{V}_{j}}\exp(\mathbf{f}_{j}^{l-1}\mathbf{W}^{Q}_{node})^{\mathrm{T}}\mathbf{k}_{p})},\quad\mathbf{k}_{p}=\mathbf{h}_{p}^{l}\mathbf{W}^{K}_{node},$

where $\mathbf{W}^{Q}_{node}$ and $\mathbf{W}^{K}_{node}$ are the projection
matrices for the query and key of the instance-to-feature message-passing. By
stacking multiple HyperFormer layers, we are able to capture high-order
instance correlations and feature correlations. The feature representations
learned from the last HyperFormer $\mathbf{F}^{L}$ can be directly plugged
into any model architecture as the feature embedding layer and improve the
prediction performance on the downstream tasks.

##  4\. Experiments

To evaluate the effectiveness of the proposed approach, we conduct our
experiments on two real-world tasks that often suffer from the feature
sparsity issue: (i) click-through-rate (CTR) prediction and (ii) top-k item
recommendation.

###  4.1. Task1: CTR Prediction

Click-through-rate (CTR) prediction is a task that predicts how likely a user
is going to click an advertisement. Typically, an instance sample is
represented by high-dimensional and sparse features, such as user profile, ad
attributes, and contextual features such as time, platform, and geographic
location. We first try to evaluate the effectiveness of HyperFormer for CTR
prediction.

Datasets. For the task of CTR prediction, we adopt two public real-world
benchmark datasets in which the features are extremely sparse and the
statistics for those datasets can be found in Table 1. To adopt the
MovieLens-1M dataset for CTR prediction, we follow (wang2021dcn, ;
song2019autoint, ) to transform the original user ratings into binary values.
The dataset is divided into 8:1:1 for training, validation, and testing,
respectively. The Criteo dataset is widely adopted for CTR prediction that
includes 45 million users’ ad clicks on display ads over a 7-day period. As in
(wang2021dcn, ; song2019autoint, ), we use the data from the first 6 days for
training and randomly split the data from the last day into validation and
test sets.

Table 1. Dataset Statistics.

CTR Prediction | Dataset | #Sample | #Field | #Feature
---|---|---|---|---
MovieLen-1M | 0.94M | 7 | 3,529
Criteo | 45.84M | 39 | 998,960
Item Recommendation | Dataset | #User | #Item | #Features
Amazon-Movie | 19,873 | 10,176 | 8,504
Bookcrossing | 48,999 | 193,765 | 5,100

Baselines. We include the following baselines methods for CTR prediction:
Logistic Regression (LR) and Factorization Machine (FM)
(rendle2010factorization, ). Different neural extensions of FM, including
Neural Factorization Machine (NFM) (he2017neural, ), xDeepFM (lian2018xdeepfm,
), and HoFM (blondel2016higher, ). AutoInt (song2019autoint, ) is designed to
automatically learn the feature interaction with self-attention. DCN-v2
(wang2021dcn, ) is an improved deep & cross network that models the explicit
and bounded-degree feature interactions. To show the flexibility and
effectiveness of HyperFormer, we integrate it into two representative
baselines AutoInt and DCN-v2 then report their performance in the experiments.

General Comparison. We evaluate the performance of different methods based on
two widely-used metrics for CTR prediction: AUC and LogLoss in Table 2. FM is
able to model the second-order feature interaction and thus outperforms LR,
which can only learn from raw feature input. With the power of deep neural
networks, xDeepFM and NFM can improve the performance of FM in both datasets
by incorporating non-linear transformations and interactions among features.
AutoInt further improves on NFM by adaptively modeling feature interactions
using an attention mechanism. DCN-v2 is also shown to be an effective approach
for CTR. More importantly, our experimental results demonstrate the
effectiveness of HyperFormer, as it improves the performance of the two
representative CTR models for both AUC and LogLoss.

Further Analysis on Tail Features. Feature values usually follow a power-law
distribution and those tail features only appear a few times among all the
data examples. Without enough learning signals, it is hard for the low-
frequency features to obtain informative embeddings, resulting in low CTR
prediction accuracy for data samples that contain those low-frequency
features. Our HyperFormer is proposed to address this issue by modeling the
correlations between features through hypergraph message passing. To examine
whether HyperFormer achieves this goal, we first sort and slice all the
feature values in MovieLens-1M by frequency. Then we retrieve the test inputs
that contain each set of feature values for evaluation. We compare the CTR
performance of DCN-v2 with and without HyperFormer in Figure 2. Our results
show that HyperFormer enables DCN-v2 to achieve better performance on samples
with low-frequency features, as measured by both LogLoss and AUC. This
demonstrates that HyperFormer is effective in enhancing the quality of feature
embeddings for tail features, leading to more accurate CTR predictions for
items with tail features.

Table 2. AUC and Logloss on CTR prediction.

Method | Movielens-1M |  | Criteo
---|---|---|---
| AUC | LogLoss |  | AUC | LogLoss
LR | $0.7716$ | $0.4424$ |  | $0.7820$ | $0.4695$
FM | $0.8252$ | $0.3998$ |  | $0.7836$ | $0.4700$
NFM | $0.8357$ | $0.3883$ |  | $0.7957$ | $0.4562$
xDeepFM | $0.8286$ | $0.4108$ |  | $0.8009$ | $0.4517$
HoFM | $0.8304$ | $0.4013$ |  | $0.8004$ | $0.4508$
AutoInt | $0.8456$ | $0.3797$ |  | $0.8061$ | $0.4455$
DCN-v2 | $0.8402$ | $0.3811$ |  | $0.8045$ | $0.4462$
AutoInt+HyperFormer | $\mathbf{0.8462}$ | $\mathbf{0.3770}$ |  | $\mathbf{0.8072}$ | $\mathbf{0.4444}$
DCN-v2+HyperFormer | $\mathbf{0.8471}$ | $\mathbf{0.3755}$ |  | $\mathbf{0.8061}$ | $\mathbf{0.4453}$

Figure 2. CTR performance across different instance groups.

###  4.2. Task2: Top-K Item Recommendation

In many real-world recommendation systems, the goal is to retrieve the Top-K
relevant items for a given user. Indeed, the input of these top-k
recommendation frameworks also suffers from the same issue as the CTR task in
that the user and item features are also extremely sparse and follow the long-
tailed distribution. To further demonstrate the generalizability of
HyperFormer, we evaluate its performance on the relational representation
learning for recommendation systems.

Datasets. In the experiment, we adopt two benchmark datasets for evaluation.
Amazon-Movie consists of product reviews and metadata for the ”Movie” category
spanning from May 1996 to July 2014 (mcauley2015image, ). Bookcrossing
(ziegler2005improving, ) collects the user-item ratings within the community,
including both user demographic and age as well as item features such as
Title, Author, Year, and Publisher. After removing the inactive users and
items, we obtain the final datasets as summarized in Table 1. We randomly
sample 70% data for model training, 10% for validation and 20% for testing.

Baselines. Due to its high efficiency and flexibility, Two-tower model with
separate user and item towers is widely adopted as the fundamental learning
architecture for large-scale top-k item retrieval (yi2019sampling, ;
yao2021self, ; wang2021cross, ; yu2021dual, ; zhang2021model, ). Specifically,
the high-dimensional user and item features are input to the corresponding
towers, and the preference scores are typically computed by a dot-product
between user and item embeddings encoded by the corresponding towers. To solve
the feature imbalance issue, DAT (yu2021dual, ) was proposed to extend each
tower with an extra learnable vector to memorize the cross-tower information.
Recently, Yao et.al proposed SSL (yao2021self, ) to leverage latent feature
correlations in a two-tower model by augmenting the data and incorporating an
auxiliary self-supervised learning task.

Figure 3. Top-k Recommendation performance comparison.

General Comparison. To evaluate HyperFormer on top-K item recommendation, we
plug it into a two-tower recommendation model and compare it with the baseline
methods above that were proposed to address the feature sparsity issue. We use
NDCG@10 and Recall@10 as evaluation metrics and summarize the results for both
Amazon-Movie and Bookcrossing in Figure 3. By introducing the category
alignment loss and an extended vector capturing the cross-tower information,
DAT can significantly improve two-tower in top-k recommendation for both
datasets. We find that SSL significantly outperforms DAT in Bookcrossing but
falls behind DAT in Amazon-Movie. However, the proposed HyperFormer
consistently outperforms all other methods in both datasets, showcasing its
effectiveness in feature representation learning. The advantage across both
CTR prediction and top-k recommendation highlights the generalizability of
HyperFormer and its potential to address feature sparsity in various real-
world tasks.

##  5\. Conclusion

In this paper, we focus on the problem of representation learning on high-
dimensional sparse features. We propose to build feature hypergraphs to model
the instance correlations and feature correlations explicitly. The proposed
Hypergraph Transformer further enables message-passing on the constructed
feature hypergraphs, resulting in more informative feature representations
that encode instance correlations and feature correlations within the data.
The evaluation of different methods on click-through-rate prediction and item
recommendation demonstrate the effectiveness of our approach in capturing the
relational information within data for learning informative feature
representations.

## Acknowledgments

This material is based upon work supported by, or in part by, the NSF grant
2229461.

## References

  * [1] Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata.  Higher-order factorization machines.  NeurIPS, 2016.
  * [2] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al.  Wide & deep learning for recommender systems.  In RecSys, 2016.
  * [3] Kaize Ding, Yichuan Li, Jundong Li, Chenghao Liu, and Huan Liu.  Feature interaction-aware graph neural networks.  arXiv preprint arXiv:1908.07110, 2019.
  * [4] Kaize Ding, Jianling Wang, James Caverlee, and Huan Liu.  Meta propagation networks for few-shot semi-supervised learning on graphs.  In AAAI, 2022.
  * [5] Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu.  Be more with less: Hypergraph attention networks for inductive text classification.  In EMNLP, 2020.
  * [6] Kaize Ding, Yancheng Wang, Yingzhen Yang, and Huan Liu.  Eliciting structural and semantic global knowledge in unsupervised graph contrastive learning.  In AAAI, 2023.
  * [7] Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu.  Data augmentation for deep graph learning: A survey.  SIGKDD Explorations, 2022.
  * [8] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao.  Hypergraph neural networks.  In AAAI, 2019.
  * [9] Wei Guo, Rong Su, Renhao Tan, Huifeng Guo, Yingxue Zhang, Zhirong Liu, Ruiming Tang, and Xiuqiang He.  Dual graph enhanced embedding neural network for ctr prediction.  In KDD, 2021.
  * [10] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In NeurIPS, 2017.
  * [11] Xiangnan He and Tat-Seng Chua.  Neural factorization machines for sparse predictive analytics.  In SIGIR, 2017.
  * [12] Ujun Jeong, Kaize Ding, Lu Cheng, Ruocheng Guo, Kai Shu, and Huan Liu.  Nothing stands alone: Relational fake news detection with hypergraph neural networks.  In BigData, 2022.
  * [13] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin.  Field-aware factorization machines for ctr prediction.  In RecSys, 2016.
  * [14] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [15] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann.  Predict then propagate: Graph neural networks meet personalized pagerank.  In ICLR, 2019.
  * [16] Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang.  Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction.  In CIKM, 2019.
  * [17] Zekun Li, Shu Wu, Zeyu Cui, and Xiaoyu Zhang.  Graphfm: Graph factorization machines for feature interaction modeling.  arXiv preprint arXiv:2105.11866, 2021.
  * [18] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun.  xdeepfm: Combining explicit and implicit feature interactions for recommender systems.  In KDD, 2018.
  * [19] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel.  Image-based recommendations on styles and substitutes.  In SIGIR, 2015.
  * [20] Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu.  Field-weighted factorization machines for click-through rate prediction in display advertising.  In TheWebConf, 2018.
  * [21] Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang.  Product-based neural networks for user response prediction.  In ICDM, 2016.
  * [22] Steffen Rendle.  Factorization machines.  In ICDM, 2010.
  * [23] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao.  Deep crossing: Web-scale modeling without manually crafted combinatorial features.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 255–262, 2016.
  * [24] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang.  Autoint: Automatic feature interaction learning via self-attentive neural networks.  In CIKM, 2019.
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [26] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In ICLR, 2018.
  * [27] Jianling Wang, Kaize Ding, Liangjie Hong, Huan Liu, and James Caverlee.  Next-item recommendation with sequential hypergraphs.  In SIGIR, 2020.
  * [28] Jianling Wang, Kaize Ding, Ziwei Zhu, and James Caverlee.  Session-based recommendation with hypergraph attention networks.  In SDM, 2021.
  * [29] Jinpeng Wang, Jieming Zhu, and Xiuqiang He.  Cross-batch negative sampling for training two-tower recommenders.  In SIGIR, 2021.
  * [30] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang.  Deep & cross network for ad click predictions.  In KDD. 2017.
  * [31] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi.  Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems.  In TheWebConf, 2021.
  * [32] Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H Chi, Steve Tjoa, Jieqi Kang, et al.  Self-supervised learning for large-scale item recommendations.  In CIKM, 2021.
  * [33] Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi.  Sampling-bias-corrected neural modeling for large corpus item recommendations.  In RecSys, 2019.
  * [34] Yantao Yu, Weipeng Wang, Zhoutian Feng, and Daiyue Xue.  A dual augmented two-tower model for online large-scale recommendation.  DLP-KDD Workshop, 2021.
  * [35] Yin Zhang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Lichan Hong, and Ed H Chi.  A model of two tales: Dual transfer learning framework for improved long-tail item recommendation.  In TheWebConf, 2021.
  * [36] Zuowu Zheng, Changwang Zhang, Xiaofeng Gao, and Guihai Chen.  Hien: hierarchical intention embedding network for click-through rate prediction.  In SIGIR, 2022.
  * [37] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf.  Learning with hypergraphs: Clustering, classification, and embedding.  NeurIPS, 2006.
  * [38] Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen.  Improving recommendation lists through topic diversification.  In WWW, 2005.