# UFIN: Universal Feature Interaction Network for Multi-Domain Click-Through
Rate Prediction

Zhen Tian  Gaoling School of Artificial Intelligence
Renmin University of China
Beijing, China
<EMAIL_ADDRESS>Changwang Zhang  Poisson Lab, Huawei
Beijing, China
<EMAIL_ADDRESS>Wayne Xin Zhao  Gaoling School of Artificial
Intelligence
Renmin University of China
Beijing, China
<EMAIL_ADDRESS>Xin Zhao  Poisson Lab, Huawei
Beijing, China
<EMAIL_ADDRESS>Ji-Rong Wen  Gaoling School of Artificial Intelligence
Renmin University of China
Beijing, China
<EMAIL_ADDRESS>Zhao Cao  Poisson Lab, Huawei
Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Click-Through Rate (CTR) prediction, which aims to estimate the probability of
a user clicking on an item, is a key task in online advertising. Numerous
existing CTR models concentrate on modeling the feature interactions within a
solitary domain, thereby rendering them inadequate for fulfilling the
requisites of multi-domain recommendations in real industrial scenarios. Some
recent approaches propose intricate architectures to enhance knowledge sharing
and augment model training across multiple domains. However, these approaches
encounter difficulties when being transferred to new recommendation domains,
owing to their reliance on the modeling of ID features (_e.g.,_ _item_id_). To
address the above issue, we propose the Universal Feature Interaction Network
(UFIN) approach for CTR prediction. UFIN exploits textual data to learn
universal feature interactions that can be effectively transferred across
diverse domains. For learning universal feature representations, we regard the
text and feature as two different _modalities_ and propose an encoder-decoder
network founded on a Large Language Model (LLM) to enforce the transfer of
data from the text modality to the feature modality. Building upon the above
foundation, we further develop a mixture-of-experts (MoE) enhanced adaptive
feature interaction model to learn transferable collaborative patterns across
multiple domains. Furthermore, we propose a multi-domain knowledge
distillation framework to enhance feature interaction learning. Based on the
above methods, UFIN can effectively bridge the semantic gap to learn common
knowledge across various domains, surpassing the constraints of ID-based
models. Extensive experiments conducted on eight datasets show the
effectiveness of UFIN, in both multi-domain and cross-platform settings. Our
code is available at https://github.com/RUCAIBox/UFIN.

###### Index Terms:

Universal Feature Interaction, CTR Prediction, Natural Language
Recommendation, Large Language Models

##  I Introduction

Click-Through Rate (CTR) prediction, which aims to predict the probability of
a user clicking on an item, is an important task for online advertising and
recommender systems. Various approaches have been proposed for effective CTR
prediction [1, 2, 3, 4]. These methods mainly focus on accurately modeling the
complicated feature interactions to capture the underlying collaborative
patterns. Most of the existing approaches concentrate on single-domain
prediction, where each model is solely trained to serve the CTR prediction of
a single scenario. However, in large-scale corporate enterprises, numerous
business domains frequently necessitate CTR prediction to augment user
contentment and enhance commercial revenue. For instance, in the case of
e-commerce enterprises, the advertising scenarios encompass a wide array of
options and manifest notable disparities, encompassing domains such as motion
pictures, literary works, electronic devices, and culinary delights, among
others. Merely mixing all the data and training a single shared CTR model
cannot yield satisfactory results across all domains owing to the substantial
distribution variance among diverse scenarios (_domain seesaw phenomenon_
[5]). The domain-specific modeling paradigm severely restricts the efficient
utilization of extensive user behavior data in business scenarios.

Some recent studies [6, 5, 7, 8] propose conducting multi-domain CTR
predictions. The core idea of these approaches is to introduce a shared neural
network for learning the common knowledge across diverse domains, while
simultaneously integrating multiple domain-specific sub-networks to capture
the distinct characteristics of each domain. Although somewhat efficacious,
the majority of these methods rely on modeling the ID features (_e.g.,_
_item_id_) to develop the CTR prediction. A major obstacle of this paradigm is
the limited transferability of the learned model to new recommendation
scenarios, even when the underlying data structures remain unchanged.

Inspired by recent advancements in natural language recommendations [9], our
objective is to devise a novel approach to learn universally applicable
collaborative patterns by surpassing the constraints of ID features. Our
fundamental concept entails transforming raw features, such as the _location_
of an item, into textual data and employing Large Language Models (LLMs) to
acquire transferable representations. While previous attempts have
demonstrated the promise of this approach for certain recommendation tasks
[10, 11], there remain several critical challenges to address in the context
of multi-domain CTR predictions. First, the textual semantic space is not
directly conducive to the task of CTR prediction [12]. In comparison to
traditional feature interaction based methods, LLMs encounter difficulties in
capturing collaborative patterns, thereby resulting in suboptimal model
performance. Second, due to substantial distribution variance across different
domains, effectively leveraging the collaborative knowledge from the source
domain to enhance the target domain proves to be a formidable task. For
instance, the interaction between features _user_ and _title_ proves most
valuable in movie recommendations, yet its efficacy diminishes in the context
of beauty recommendations.

To address these issues, in this paper, we propose the Universal Feature
Interaction Network (UFIN) for multi-domain CTR prediction. UFIN exploits
textual data to acquire knowledge of universal feature interactions that can
be effectively transferred across diverse domains. To learn universal feature
representations, we devise a prompt to convert the raw features into text and
subsequently generate a set of _universal features_ to capture the general
attributes of interactions. Notably, we regard the text and feature
representations as two _modalities_ and devise an encoder-decoder network
founded on a Large Language Model (LLM) to enforce the conversion of data from
the text modality to the feature modality. This scheme can be denoted as ”raw
features $\Rightarrow$ text $\Rightarrow$ universal features”. For learning
universal feature interactions, we develop an MoE-enhanced adaptive feature
interaction model, which can learn the generalized collaborative patterns from
diverse domains. To further enhance the acquisition of collaborative
knowledge, we propose a multi-domain knowledge distillation framework to
supervise the training of our approach. Through these aforementioned
mechanisms, UFIN can effectively bridge the semantic gap to learn common
knowledge across various recommendation domains, surpassing the limitations of
ID-based models.

The paper’s main contributions are summarized as follows:

$\bullet$ We propose a novel Universal Feature Interaction Network (UFIN) for
CTR prediction, intelligently acquiring the collaborative patterns across
diverse domains.

$\bullet$ To the best of our knowledge, UFIN is the first deep CTR model to
harness Large Language Models (LLMs) to adaptively learn the feature
interactions for recommendations, thereby obtaining universal feature
representations from textual data. This empowers UFIN to proficiently bridge
the semantic gap across various domains.

$\bullet$ We propose a multi-domain knowledge distillation framework for
enhancing the feature interaction learning. This motivates UFIN to
proficiently acquire the collaborative knowledge from diverse domains, thereby
improving the model performance.

$\bullet$ We conduct extensive experiments on eight widely used datasets. UFIN
outperforms a number of competitive baselines in both multi-domain and cross-
platform settings, demonstrating the effectiveness of our approach.

##  II Methodology

In this section, we present a universal feature interaction network for multi-
domain CTR predictions, named UFIN. Unlike previous works, our goal is to
learn the universal feature interactions that are able to effectively
transferred to new recommendation domains.

Figure 1: The overall framework of UFIN is designed as an encoder-decoder
architecture, followed by a feature interaction network. The encoder-decoder
accomplishes the transformation from text to feature modality, yielding a
collection of universal features. The feature interaction network consists of
multiple adaptive learning experts, and each expert automatically learns the
underlying true interaction orders within each given domain. Further, a
semantic gating router is incorporated to adaptively integrate all experts for
learning the _universal feature interactions_ across diverse domains.

###  II-A Problem Formulation

The CTR prediction task is to estimate the probability of a user clicking on
an item. An instance of the CTR task can be denoted as a tuple $(\bm{x},y)$,
where the vector $\bm{x}$ includes the user, item and context features, and
the label $y\in\\{0,1\\}$ represents the whether the item is clicked or not.
The feature vector $\bm{X}$ contains multiple fields
$\\{\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{m}\\}$, where $m$ is the number of
feature fields, and $\bm{x}_{j}$ is a one-hot identifier (ID) vector of the
$j$-th feature field. In this way, the CTR dataset of a single domain $D$ can
be formulated as a set of _instances_ $\\{(\bm{X}_{i},y_{i})\\}_{i=1}^{N}$.
Real-world recommender applications often have to deal with multiple business
domains. Specifically, multi-domain CTR models need to make predictions for
$L$ domains, $\\{D_{1},D_{2},\cdots,D_{L}\\}$. Compared to single-domain CTR
prediction, multi-domain CTR prediction often poses greater challenges, as
different domains may have different feature fields and data distributions.

Most of multi-domain CTR prediction approaches adopt the embedding-based
feature interaction learning paradigm. Since the input feature is usually
sparse and high-dimensional, most CTR models employ an embedding layer to map
these one-hot features into low-dimensional vectors, _i.e.,_ for the feature
vector $\bm{x}_{j}$, the corresponding embedding $\bm{e}_{j}$ is obtained by
an embedding look-up operation. As such, the features can be represented as a
list of embeddings $\bm{E}=\\{\bm{e}_{1},\bm{e}_{2},..,\bm{e}_{m}\\}$. Based
on $\bm{E}$, most approaches conduct feature interactions $\mathcal{F}$ to
capture the collaborative patterns between users and items. The main goal of
the CTR prediction is to learn the prediction function
$\hat{y}=\mathcal{F}(\bm{E})$.

###  II-B Overview of UFIN

As shown in Figure 1, UFIN is designed with an encoder-decoder architecture
followed by an interaction network. The core idea of UFIN is to consider the
text and features as two _modalities_ for learning universal feature
representations and universal feature interactions. Based on this idea, we
formulate the general feature form as the natural language text and employ an
MoE-enhanced large language model (LLM) as the encoder to transform the text
modality into a latent space. Afterwards, the decoder performs a mapping to
the feature modality, generating the _universal features_. As such, UFIN can
capture the generalized attributes of interactions, bridging the semantic gap
between different domains. For learning universal feature interactions, we
derive adaptive feature interactions based on the universal features to
acquire transferable collaborative knowledge. To learn the common
collaborative patterns of different domains, we incorporate them into an MoE
model to enhance the knowledge sharing. Furthermore, we propose a multi-domain
knowledge distillation framework to enhance feature interaction learning.

In what follows, we introduce the details of universal feature representation
learning (Section II-C) and universal feature interaction learning (Section
II-D).

###  II-C Universal Feature Representation Learning

To deliver transferable recommendations, we first map the features from
various domains to a common semantic space. Prior studies rely on the
embedding look-up operation to acquire the representations for the given
feature IDs. These approaches have two significant limitations. Firstly, it
relinquishes the innate semantic information of features, thereby greatly
impairing the model’s transferability. Secondly, these models cannot be
transferred to alternative platforms that possess distinct feature fields (for
instance, transitioning from _Amazon_ to _MovieLens_).

To learn transferable feature representations, we adopt natural language text
as the universal data form, which is derived from raw features through a
prompt. As increasingly more evidence shows [12], text and feature
representations can be regarded as two _modalities_ that can be mutually
transformed. Based on this idea, we employ an MoE-enhanced LLM as the encoder
to enforce the text modality into a latent space and develop a decoder that
performs a mapping to the feature modality, thereby generating the universal
feature representations (_universal features_). This approach can be expressed
as ”raw features $\Rightarrow$ text $\Rightarrow$ universal features”,
providing a means to bridge the semantic gap across different domains.

####  II-C1 Feature Textualization

The first step in learning universal feature representations is to transform
the raw features into textual data, as described by a prompt. As previous work
shows [13], an effective prompt should consist of _personalized_ fields for
different users and items. For this purpose, we design a prompt, that includes
the user profile, item description and contextual information to conduct such
transformation. As shown in Figure 1, given the user features (_i.e.,_ _male_
, _student_), item features (_i.e.,_ _jacket_ , _HOUONE_) and context features
(_i.e.,_ _09:21_), the transformed textual data is shown as:

> There is a user, whose gender is male, and occupation is student. The
> product is a jacket and its name is “HOUONE”. The system time is 09:21.

In our prompt, different types (_i.e.,_ user-side, item-side and context-side)
of raw features (_i.e.,_ $\\{\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{m}\\}$) are
sequentially summarized into a natural language sentence, where the
descriptions of different sides are separated by the period `"."`, and the
features are separated by the comma `","`. As such, a CTR instance can be
denoted as $\\{\bm{w}_{1},\bm{w}_{2},...,\bm{w}_{n}\\}$, where $n$ is the
number of words in a sentence. In this way, we can obtain a universal data
form (_i.e.,_ natural language text) to represent the CTR instances across
various domains or platforms. Note that we can also design other types of
prompts to conduct such transformation. Unlike prior work [12], the format of
the prompt does not have an obvious effect in our framework, while the
semantic information of the prompt, specifically the features it contains, has
a large influence on the performance, which will be explored in the Section
III-C7.

####  II-C2 Textual Feature Encoding

Given the universal data form in the text modality, our goal is to transform
it into feature modality to obtain the universal features. Most multimodal
works [14] (_e.g.,_ text-to-image) employ an encoder-decoder architecture to
align the representations of different modalities. Following them, we first
employ an MoE-enhanced LLM as the encoder to project the textual data into a
common latent space.

LLM based Textual Encoding. Motivated by the recent advances in large language
models (LLMs [15]), which show excellent language modeling capacity, we adopt
FLAN-T5 [16] to learn the latent representations of the text. Given the words
of textual data $\\{\bm{w}_{1},\bm{w}_{2},...,\bm{w}_{n}\\}$, we feed them
into the LLM, and we have:

$\displaystyle\\{\bm{v}_{1},\bm{v}_{2},...,\bm{v}_{n}\\}$
$\displaystyle=\textrm{LLM}(\\{\bm{w}_{1},\bm{w}_{2},...,\bm{w}_{n}\\}),$ (1)
$\displaystyle\bm{s}$
$\displaystyle=\textrm{LayerNorm}(\sum_{j=1}^{n}\bm{v}_{j}),$ (2)

where $\\{\bm{v}_{1},...,\bm{v}_{n}\\}\in\mathbb{R}^{d_{V}\times n}$ is the
last hidden state of LLM, $d_{V}$ is the state dimension and $\bm{s}$ is the
latent representation. Unlike existing studies [10, 11], we employ sum pooling
to preserve token-level semantics of features and apply LayerNorm [17] to
adjust the semantic distributions. Note that the LLM is solely for text
encoding, which is not tuned during training. Therefore, we can cache the last
hidden state $\\{\bm{v}_{1},...,\bm{v}_{n}\\}$ to ensure the efficiency of our
approach.

Multi-Domain Semantic Fusion. In the above, we obtain semantic representations
from LLMs. However, recent study [10] found that the original semantic space
of PLMs is not suitable for the recommendation task. To address this issue, a
commonly used approach is to employ a neural network to learn the appropriate
semantic space for enhancing the representations. Since different domains
usually correspond to varying semantic contexts, merely learning a shared
semantic space for all domains will suffer from the domain seesaw phenomenon
that degrades the model capacity. Inspired by the recent study [10], our idea
is to learn an independent semantic space for each domain and adaptively
combine them based on the semantic context.

To achieve this purpose, we employ a single-layer perception (SLP) to learn a
suitable semantic space for each domain, and incorporate them into an MoE
model to enhance the domain fusion. Specifically, given $L$ domains, we
introduce $L$ experts, each learning in a different subspaces, and combine
them through a gating router:

$\displaystyle\bm{z}$
$\displaystyle=\sum_{j=1}^{L}\sigma(\bm{W}_{j}\bm{s}+\bm{b}_{j})\cdot g_{j},$
(3) $\displaystyle\bm{g}$ $\displaystyle=\mathrm{Softmax}(\bm{W}_{g}\bm{s}),$
(4)

where $\bm{W}_{j}\in\mathbb{R}^{d_{V}\times d_{V}}$ and
$\bm{b}_{j}\in\mathbb{R}^{d_{V}}$ are the weight and bias of the $j$-th
expert, $\sigma$ is the activation function, $g_{j}$ is the $j$-th combination
weight of the gating router calculated by Eq. (4),
$\bm{W}_{g}\in\mathbb{R}^{L\times d_{V}}$ is the router weight, and $\bm{z}$
is the enhanced representations.

Fusing Anonymous Features. As there exists numerous anonymous features (such
as the identifiers of the user/item) that lack semantic information, we
refrain from including them in our prompt template. Nevertheless, these
features may play a significant role, particularly in situations where
semantic features are limited or nonexistent. For instance, in the case of the
Amazon dataset, only _user_id_ features are accessible on the user side. To
generate more comprehensive predictions, we expand our methodology to
encompass these features. There are many ways to achieve this purpose, and we
follow the existing ID-Text fusion work [10], which employs a distinct
embedding for each anonymous feature and merges them with the textual
representations:

$\displaystyle\tilde{\bm{z}}$
$\displaystyle=\bm{z}+\sum_{k=1}^{c}\bm{U}_{k}\bm{h}_{k},$ (5)

where $\\{\bm{h}_{1},...,\bm{h}_{c}\\}\in\mathbb{R}^{d_{A}\times c}$
represents the anonymous embeddings, $c$ is the number of anonymous fields,
and $\bm{U}_{k}\in\mathbb{R}^{d_{V}\times d_{A}}$ is the projection matrix.
Note that the anonymous features are only auxiliary representations and are
not used unless specified. For efficiency considerations, we do not employ
other complex mechanisms (e.g., self-attention [18]), which will be studied in
our future work.

####  II-C3 Universal Feature Generation

With the above textual encoding procedure, we can obtain the universal
representation of the instances. Previous works [19] directly feed the textual
representations into a feedforward network (_e.g.,_ MLP) to make predictions,
resulting in suboptimal performance. As the recent study shows [20], it is
challenging for an MLP to capture effective collaborative patterns compared
with the feature-wise interactions (_e.g.,_ FM [21]). Our proposed solution,
by contrast, entails harnessing textual data to generate _universal features_
that transcend various domains, thereby capturing the collective patterns that
are commonly observed. To illustrate, we anticipate generating the universal
attribute _”amusing”_ from the textual expressions _”This movie is hilarious”_
and _”This book is whimsical”_ within the realms of movie and book
recommendations. Based on this idea, we develop a decoder that conducts a
transformation to map the latent representation of textual data to the feature
modality, shown as:

$\displaystyle\tilde{\bm{e}}_{j}$
$\displaystyle=\textrm{LayerNorm}(\bm{V}_{j}\tilde{\bm{z}}),$ (6)

where $j\in\\{1,2,...,n_{u}\\}$ and $n_{u}$ is the field number of universal
features. Here we incorporate a set of projection matrices
$\\{\bm{V}_{j}\in\mathbb{R}^{d\times d_{V}}\\}_{j=1}^{n_{u}}$ to generate a
set of universal features
$\tilde{\bm{E}}=\\{\tilde{\bm{e}}_{1},\tilde{\bm{e}}_{2},...,\tilde{\bm{e}}_{n_{u}}\\}$,
each measuring different aspects from different representation subspaces. As
such, we use the generated universal representations for subsequent feature
interaction modeling.

###  II-D Universal Feature Interaction Learning

The core of our proposed UFIN is to learn the universal feature interactions
for intelligently acquiring the generalized collaborative knowledge across
diverse domains. To this end, our approach is to model the adaptive feature
interactions based on the generated universal features to capture the common
collaborative patterns. Furthermore, to promote feature interaction learning,
we introduce a framework for distilling knowledge to guide the model’s
learning process and subsequently enhance its performance.

####  II-D1 Adaptive Feature Interaction Learning at a Single Domain

For learning universal feature interactions, an important issue is how to
accurately model the interactions orders/forms within each domain, as
different domains typically correspond to varying feature relationships.
Traditional methods manually design a maximal order and further remove the
useless interactions from them, _e.g.,_ FM [21] empirically enumerates all
second-order feature interaction terms. These approaches not only results in
inaccurate modeling of the underlying true feature interactions in real-world
scenarios but also limits model transferability. As a promising approach,
recent study EulerNet [22] proposes to model the _adaptive_ feature
interactions, _i.e.,_ the interaction forms are automatically learned from
data, allowing for arbitrary orders and a flexible number of terms. Given the
input features
$\tilde{\bm{E}}=\\{\tilde{\bm{e}}_{1},...,\tilde{\bm{e}}_{n_{u}}\\}$ (See Eq.
(6)), the adaptive feature interaction learning function of EulerNet [22] is
formulated as:

$\mathcal{F}(\tilde{\bm{E}};\mathcal{A})=\bm{w}^{\top}\sum_{\bm{\alpha}\in\mathcal{A}}\tilde{\bm{e}}_{1}^{\alpha_{1}}\odot\tilde{\bm{e}}_{2}^{\alpha_{2}}\odot...\odot\tilde{\bm{e}}_{n_{u}}^{\alpha_{n_{u}}},$
(7)

where $\bm{\alpha}=[\alpha_{1},\alpha_{2},...,\alpha_{n_{u}}]$ denotes the
learnable order parameter of each feature, $\mathcal{A}$ is the parameter set
of all the learnable orders, and $\bm{w}$ is a transition vector for
generating a scalar result. For a given domain, the underlying true feature
interaction forms are automatically learned from the parameter $\mathcal{A}$,
_e.g.,_ the interactions of FMs [21] can be learned by
$\mathcal{A}=\\{\bm{\alpha}|\sum_{j=1}^{m}\alpha_{j}=2,\forall\alpha_{j}\in\\{0,1\\}\\}$.
Previous works [23, 24] face challenges in achieving this, because when the
embedding $\bm{e}_{j}$ contains negative values, the order $\alpha_{j}$ must
be set to an integer value to avoid invalid operation (_e.g.,_ $(-1)^{0.5}$).
As a solution, EulerNet [22] leverages Euler’s formula to learn the feature
interactions in a complex vector space that enables the efficient learning of
arbitrary-order feature interactions, without additional restrictions (_e.g.,_
_non-negative_ embedding or _integer_ order).

In our case, we utilize EulerNet [22] to adaptively learn the underlying true
feature interaction orders/forms within each given domain. We mainly transfer
the knowledge of interaction orders (_i.e.,_ $\mathcal{A}$) to enhance
predictions in a target domain.

####  II-D2 Multi-Domain Feature Interaction Learning

In the above, we have discussed the feature interaction learning within a
single domain. In this section, we aim to generalize the methodology for
adapting the modeling of feature interactions across multiple domains.
Intuitively, we can train distinct models independently to adapt to the
distribution of each domain. However, this approach fails to grasp the
interconnectedness between diverse domains, leading to challenges in the cold-
start scenario. Our objective is to acquire knowledge of the universal feature
interactions that encompass the shared collaborative patterns between
different domains. Our proposed solution entails the introduction of multiple
sets of interaction orders (_i.e.,_ $\mathcal{A}$), each of which learns the
underlying true feature interaction orders for a single domain, and shares
some of them across different domains to acquire knowledge of the common
collaborative patterns.

In practice, we employ an MoE model to implement our idea. Given $L$ domains,
we introduce $L$ experts, each expert $j$ is implemented by EulerNet [22] with
the learnable order parameters $\mathcal{A}_{j}$. All experts share the input
embedding $\tilde{\bm{E}}=\\{\tilde{\bm{e}}_{1},...,\tilde{\bm{e}}_{n_{u}}\\}$
(See Eq. (6)), combined by a gating router based on the semantic
representation $\tilde{\bm{z}}$ (See Eq. (5)):

$\displaystyle\zeta$
$\displaystyle=\sum_{j=1}^{L}\mathcal{F}({\tilde{\bm{E}}};\mathcal{A}_{j})\cdot\tilde{\bm{g}}_{j},$
(8) $\displaystyle\tilde{\bm{g}}$
$\displaystyle=\mathrm{TopK}\Big{(}\mathrm{Softmax}(\tilde{\bm{W}}_{g}\tilde{\bm{z}})\Big{)},$
(9)

where $\mathrm{TopK}(\cdot)$ retains only the top-K elements of the input
tensor and sets all other elements as zero, $\mathcal{A}_{j}$ (_learnable
parameter_) is the interaction order of $j$-th expert, $\mathcal{F}(\cdot)$ is
the feature interaction function (See Eq. (7)) of EulerNet [22],
$\tilde{\bm{g}}_{j}$ is the $j$-th combination weight,
$\tilde{\bm{W}}_{g}\in\mathbb{R}^{L\times d_{V}}$ is the router weight, and
$\zeta$ is the output logits. For an instance in a given domain $u$, we use
its corresponding semantic representations $\tilde{\bm{z}}$ to select $K$
experts with the order sets
$\mathcal{S}_{u}=\\{\mathcal{A}^{u}_{j}\\}_{j=1}^{K}$ for collaboratively
learning the feature interactions. Formally we set $K>\lceil{L/2}\rceil$, and
we have the following finding:

######  Theorem 1.

Given $L$ domains $\mathcal{D}=\\{D_{1},D_{2},...,D_{L}\\}$ and $L$ experts
$\mathcal{A}=\\{\mathcal{A}_{1},\mathcal{A}_{2},...,\mathcal{A}_{L}\\}$, each
domain $D_{u}$ select $K$ experts
$\mathcal{S}_{u}=\\{\mathcal{A}_{1}^{u},\mathcal{A}_{2}^{u},...,\mathcal{A}_{K}^{u}\\}$
from $\mathcal{A}$, _i.e.,_ $\mathcal{S}_{u}\subseteq\mathcal{A}$. If
$K>\lceil{L/2}\rceil$, then for any given domain $D_{u}$ and $D_{v}$, the set
of the selected experts $\mathcal{S}_{u}$ and $\mathcal{S}_{v}$ must have a
same element, _i.e.,_ $\mathcal{S}_{u}\cap\mathcal{S}_{v}\neq\emptyset,\forall
u\neq v$. See proofs in Section VI.

It demonstrates that for any given domains $u$ and $v$, there exist at least
one shared expert that learns the common feature interactions. Therefore, our
approach can capture the common feature interactions between arbitrary domain
pairs, thereby capable of learning the generalized feature relationship across
all domains. Finally, we apply the sigmoid function on the logits (See Eq.
(8)) to obtain the prediction:

$\displaystyle\hat{y}=\textrm{Sigmoid}(\zeta).$ (10)

As mentioned in the work [6], a good multi-domain CTR model should contain the
features that depict the domain-specific information. For this purpose, we can
further incorporate a _feature adaptor_ , which takes as input the domain-
specific features, to precisely capture the distinct characteristics of each
domain. Following existing multi-domain methods [6], we add the output logits
of the feature adaptor (_i.e.,_ $\zeta_{f}$) to our approach (_i.e.,_ $\zeta$)
for prediction:

$\displaystyle\hat{y}=\textrm{Sigmoid}(\zeta+\zeta_{f}).$ (11)

It allows the flexibility to choose any method. Here we employ the simplest
regression model (_e.g.,_ LR [25]) as the feature adaptor to ensure the
efficiency of the model.

####  II-D3 Knowledge Distillation Enhanced Training

With the above approaches, UFIN is able to learn the universal feature
interactions based on the representations generated by LLMs. However, as
existing study shows [12], it is challenging for the representations of LLMs
to capture the feature co-occurrence correlation that results in the poor
performance compared with traditional collaborative models. To facilitate our
approach in capturing the underlying collaborative patterns, we propose a
multi-domain knowledge distillation framework that promotes feature
interaction learning.

Multi-Domain Guided Distillation. In the knowledge distillation framework, for
each domain, we pre-train a feature interaction model as the _guided network_
to learn the domain-specific collaborative patterns. As such, multiple guided
networks from different domains are incorporated as teacher model to supervise
the training of our approach. It allows flexibility to choose any teacher
model for each domain, and we specifically employ the EulerNet [22] as the
teacher with consideration for the consistency. Following existing studies
[26, 27], we use MSE loss to align the output logits between the guided
network and our approach:

$\displaystyle\mathcal{L}_{KD}=\sum_{p=1}^{M}\sum_{i=1}^{N_{p}}||\zeta^{G}_{p,i}-\zeta_{p,i}||^{2},$
(12)

where $\zeta^{G}_{p,i}$ and $\zeta_{p,i}$ (see Eq. (8)) are the logits of the
guided network and our proposed UFIN for the $i$-th instance in the $p$-th
domain. Note that the guided networks are only used for auxiliary training,
which are discard during inference, thus ensuring the transferability of our
approach.

Multi-Task Learning. To further promote the utilization of LLMs in the CTR
prediction task, we incorporate the commonly adopted binary cross-entropy
loss:

$\displaystyle\mathcal{L}_{CTR}=\sum_{p=1}^{M}\sum_{i=1}^{N_{p}}\bigg{(}y_{i}^{p}\log(\hat{y}_{i}^{p})+(1-y_{i}^{p})\log(1-\hat{y}_{i}^{p})\bigg{)},$
(13)

where $y_{i}^{p}$ and $\hat{y}_{i}^{p}$ are the ground-truth label and
predicted result of $i$-th instance in the $p$-th domain. As for the model
training, we adopt a multi-task training strategy to jointly optimize the
knowledge distillation loss and the binary classification loss:

$\displaystyle\mathcal{L}=\mathcal{L}_{KD}+\mathcal{L}_{CTR}.$ (14)

###  II-E Discussion

In the literature, a number of CTR models have been proposed. To better
highlight the novelty and difference of our approach, we make a brief
comparison of different CTR methods. For ID-based methods, such as STAR [6],
they rely on the ID features to develop the prediction, which impairs the
inherent semantic of features that makes them incapable of being applied in
the new platforms. While for semantic methods, such as P5 [28], attempt to
leverage the world knowledge of LLMs to uplift the prediction. The primary
challenge of these approaches lies in their inability to capture collaborative
signals, resulting in poor performance. Although CTRL [12] proposes a
contrastive learning framework to transfer the knowledge of LLMs to a domain-
specific collaborative model. Due to the limited capacity of the collaborative
model, it cannot adequately learn the common knowledge across different
domains and suffers the seesaw phenomenon (See Section III-B). In contrast,
our model is more _universal_ in the application of multi-domain or cross-
platform settings, naturally integrating the semantic knowledge of LLMs and
collaborative knowledge of interactions. The overall comparison is presented
in Table I.

TABLE I: Comparison of different CTR methods.

Methods | Transfer Learning | Knowledge Patterns
---|---|---
Multi-Domain | Cross-Platform | Collaborative | Semantic
STAR [6] | ✔ | ✗ | ✔ | ✗
CTRL [12] | ✗ | ✗ | ✔ | ✔
P5 [28] | ✔ | ✔ | ✗ | ✔
UFIN (ours) | ✔ | ✔ | ✔ | ✔

##  III EXPERIMENTS

In this section, we first introduce the experimental settings and then give
the results and analysis.

###  III-A Experimental Settings

We present the experimental settings, including the datasets, baseline
approaches, the details of hyper-parameters, and evaluation metrics.

####  III-A1 Datasets

It is worth mentioning that the majority of alternative public datasets cannot
be utilized to evaluate our model due to their inclusion of solely anonymous
features and the absence of semantic features. To evaluate the performance of
our model, we conduct experiments on the
Amazon111https://nijianmo.github.io/amazon/index.html and
MovieLens-1M222https://grouplens.org/datasets/movielens datasets in both
multi-domain setting and cross-platform setting. The statistics of datasets
are summerized in Table II.

  * •

Amazon [29] is a widely-used dataset for recommender systems research. We keep
the five-core datasets and select seven subsets for the multi-domain setting
(_i.e.,_ _”Movies and TV”_ , _”Books”_ , _”Electronics”_ , _”Office Products”_
, _”Musical Instruments”_ , _”Toys and Games”_ and _”Grocery and Gourmet
Food”_).

  * •

MovieLens-1M is a movie recommendation dataset, which does not contain
overlapped users or items with Amazon. We use this dataset to evaluate the
performance in a cross-platform setting.

TABLE II: The statistics of datasets. Dataset |  $\\#$ Users |  $\\#$ Items |  $\\#$ Interactions
---|---|---|---
Amazon | 1,002,827 | 2,530,874 | 7,427,505
\- Movies | 295,908 | 40,792 | 2,141,592
\- Books | 585,167 | 191,826 | 4,077,731
\- Electronics | 184,876 | 26,336 | 780,698
\- Food | 14,552 | 5,474 | 86,518
\- Instruments | 1,429 | 736 | 7,835
\- Office | 4,890 | 1,805 | 36,572
\- Toys | 19,231 | 9,142 | 116,559
MovieLens-1M | 6,041 | 3,669 | 739,012

####  III-A2 Compared Models

We compare the UFIN with twelve state-of-the-art methods, including single-
domain methods, multi-domain methods and semantic methods.

Single-Domain Methods:

  * •

DeepFM [30] combines traditional second-order factorization machines with a
feed-forward neural network (MLP), utilizing a shared embedding layer.

  * •

DCNV2 [31] is the state-of-the-art model which captures the high-order feature
interactions by taking kernel product operation of concatenated feature
vectors.

  * •

xDeepFM [32] introduces the Compressed Interaction Network (CIN), which forms
the cornerstone of the xDeepFM model. It employs the outer product of stacked
feature matrices at the vector-wise level. Additionally, a feed-forward neural
network is integrated to enhance the feature representations.

  * •

EulerNet [22] utilize the Euler’s formula to map the feature representations
from the real vector space in a _complex vector space_ , which provides a way
to cast the feature interaction orders to the linear coefficients. As such, it
can automatically learn the arbitrary-order feature interactions from data.

Multi-Domain Methods:

  * •

SharedBottom [33] is a multi-task model that parameterizes the bottom layers.
In our implementation, we apply SharedBottom to share the embedding layer,
while each domain retains its distinct fully-connected network that remains
unshared.

  * •

MMoE [34] implicitly captures task relationships in multi-task learning,
accommodating diverse label spaces for different tasks. We extend MMoE to the
context of multi-domain CTR prediction, configuring each expert as a fully-
connected network. The count of experts matches the domain count, and MMoE
additionally incorporates gating networks for each domain. These networks
process input features and yield softmax gates, dynamically combining experts
with varying weights.

  * •

HMOE [35] extends MMoE to scenario-aware experts using a gradient cutting
trick to explicitly encode scenario correlations.

  * •

AESM2 [36] introduces an innovative expert selection algorithm that
automatically identifies scenario-/task-specific and shared experts for each
input.

  * •

STAR [6] proposes a star topology architecture, which consists of a shared
center network and multiple domain-specific networks for the adaptation of
multi-domain distributions.

  * •

PEPNet [5] introduces an embedding personalized network to align multi-domain
feature representations. The common knowledge from different domains can be
learned in a shared Gated-Net.

Semantic Methods:

  * •

P5 [28] is a semantic-based recommendation model that transforms diverse
recommendation tasks into text generation tasks through prompt learning,
leveraging T5 [37] as the underlying model.

  * •

CTRL [12] proposes to combine the collaborative signals and semantic knowledge
in a contrastive learning framework. Specifically, it utilizes the knowledge
of LLMs to enhance the learning of a collaborative. Only the collaborative
model will online.

For our approach, since there are no effective user-side textual features
(_e.g.,_ only _user_id_ available on the Amazon dataset), we incorporate the
_user_id_ features into our approach (See Section II-C2). We introduce two
versions of our approach: (1) UFINt denotes the model using only item text;
(2) UFINt+f denotes the model using item text and integrating a feature
adaptor (See Section II-D2).

####  III-A3 Implementation Details

The dataset is randomly split into 80% for training, 10% for validation and
10% for testing. Following the work [31], we convert the ratings of 4-5 as
label 1; ratings of 1-2 as label 0; and remove the ratings of 3. Following the
work [5], we separately train each single-domain model in each domain to
report their best results. For multi-domain evaluation, we mix all train data
of the Amazon dataset to train the multi-domain models and semantic models,
and evaluate their performance in each given domain. For cross-platform
evaluation, we first pre-train each semantic model on the mixed Amazon
dataset, and then fine-tune them on the MovieLens-1M dataset.

The baseline implementation follows FuxiCTR [38]. For each method, the grid
search is applied to find the optimal settings. The embedding size is 16,
learning rate is tuned in {1e-3, 1e-4, 1e-5}, and batch_size is 1024. The
optimizer is Adam [39]. The $L_{2}$ penalty weight is in {1e-3, 1e-5, 1e-7}.
The default settings for MLP components are: (1) the architecture of hidden
layer is $400\times 400\times 400$; (2) the dropout rate is 0.2. For DeepFM,
the hidden layer of MLP is in {$128\times 128\times 128$, $256\times 256\times
256$, $400\times 400\times 400$}. For xDeepFM, the depth of CIN is in {1, 2,
3, 4, 5} and the hidden size is in {100, 200, 400}. For DCNV2, the depth of
CrossNet is in {1, 2, 3, 4, 5}. For EulerNet, the number of Euler interaction
layer is in {1, 2, 3, 4, 5}, and the number of order vectors is in {5, 10, 20,
30, 40}. For SharedBottom, the hidden layer of MLP for each domain is in
{$128\times 128\times 128$, $256\times 256\times 256$, $400\times 400\times
400$}. For MMoE, the numer of experts is in $\\{2,4,8,10\\}$, the hidden layer
of gating network is in {$64\times 64\times 64$, $128\times 128\times 128$,
$400\times 400\times 400$}. For HMoE, the numer of experts is in
$\\{2,4,8,10\\}$, the hidden layer of scenario network is in {$64\times
64\times 64$, $128\times 128\times 128$, $400\times 400\times 400$}. For
AMSE2, the numer of experts is in $\\{2,4,8,10\\}$, the layer number is in
$\\{1,2,3,4,5\\}$. For STAR, the hidden layer of the auxiliary networks and
topology network is in {$128\times 128\times 128$, $256\times 256\times 256$,
$400\times 400\times 400$}. For PEPNet, the gate hideen dim is in
$\\{16,32,48,64,80\\}$, and hidden units of PPNet is in {$128\times 128\times
128$, $256\times 256\times 256$, $400\times 400\times 400$}. For P5, we use T5
as the base model. For CTRL, we try to use RoBERTa and T5 as the semantic
model. The collaborative model is AutoInt, the depth, number of head and
attention size is 2, 2, 40 respectively. For our approach, the number of
experts in the semantic fusion MoE and feature interaction MoE is equal to the
number of domains (_i.e.,_ $L=7$). The universal feature fields $n_{u}$ is set
to $7$ and dimension $d$ is set to 16. The $K$ value of $TopK$ function is 5.
The SLP hidden size is $128$. For the EulerNet expert, the layer number is 1,
and the number of order vectors is 7.

####  III-A4 Evaluation Metrics

We use two popular metrics to evaluate the model performance: AUC and LogLoss.

$\bullet$ AUC [40] (Area Under the Curve) is a metric used to evaluate the
performance of binary classification or ranking models, particularly in tasks
where the prediction involves ranking items. It quantifies the ability of a
model to correctly rank higher the positive instances compared to negative
instances. A higher AUC value indicates better model performance.

In the context of ranking tasks, AUC is calculated using the following
formula:

$\text{AUC}=\frac{\sum_{(i,j)\in S}\mathbbm{1}(s_{i}>s_{j})}{|S|}$

Where $S$ is the set of all pairs of instances $(i,j)$ where $i$ is a positive
instance and $j$ is a negative instance, $s_{i}$ and $s_{j}$ represent the
predicted scores or ranks of instances $i$ and $j$, respectively.
$\mathbbm{1}(s_{i}>s_{j})$ is the indicator function that equals 1 if the
predicted score of $i$ is greater than that of $j$, and 0 otherwise. $|S|$
denotes the total number of pairs in the set $S$. In this context, AUC
measures the probability that a randomly chosen positive instance will be
ranked higher than a randomly chosen negative instance according to the
model’s predictions.

$\bullet$ LogLoss [41], also known as logarithmic loss or cross-entropy loss,
is a common evaluation metric used in binary classification and ranking tasks.
It measures the difference between predicted probabilities and actual binary
labels. In the context of ranking tasks, LogLoss is often used to assess the
quality of predicted relevance scores for different items. In a ranking task,
the LogLoss formula can be expressed as:

$\text{LogLoss}=-\frac{1}{N}\sum_{i=1}^{N}\Big{(}y_{i}\cdot\log(p_{i})+(1-y_{i})\cdot\log(1-p_{i})\Big{)}$

where $N$ is the number of instances (samples), $y_{i}$ is the true binary
label (0 or 1) for the $i$-th instance, and $p_{i}$ is the predicted
probability of the positive class for the $i$-th instance. LogLoss penalizes
larger differences between predicted probabilities and true labels. It
encourages the model to make confident predictions that are close to the true
labels. A lower LogLoss value indicates better alignment between predicted
probabilities and actual labels, reflecting better model performance.

###  III-B Overall Performance

TABLE III: Performance comparison of different CTR models. “Improv.” indicates
the relative improvement ratios of our proposed approach over the best
baseline. “*” denotes that statistical significance for $p<0.01$ compare to
the best baseline. Note that a higher AUC or lower Logloss at 0.001-level is
regarded significant, as stated in previous studies [42, 1, 30].

Evaluation | Dataset | Metric | Single-Domain Methods | Multi-Domain Methods | Semantic Methods | Improv.
---|---|---|---|---|---|---
DeepFM | xDeepFM | DCNV2 | EulerNet | SharedBottom | MMoE | HMoE | AESM2 | STAR | PEPNet | P5 | CTRL | UFINt | UFINt+f
Multi- Domain | Movies | AUC | 0.8568 | 0.8565 | 0.8537 | 0.8550 | 0.8465 | 0.8440 | 0.8445 | 0.8477 | 0.8448 | 0.8456 | 0.6440 | 0.8435 | 0.8515 | 0.8582∗ | $+0.16\%$
LogLoss | 0.2670 | 0.2738 | 0.2698 | 0.2685 | 0.2752 | 0.2751 | 0.2755 | 0.2726 | 0.2738 | 0.2746 | 0.3760 | 0.2736 | 0.2741 | 0.2649∗ | $+0.79\%$
Books | AUC | 0.8107 | 0.8094 | 0.8118 | 0.8166 | 0.8044 | 0.8095 | 0.8097 | 0.8108 | 0.8057 | 0.8080 | 0.5740 | 0.8058 | 0.8130 | 0.8215∗ | $+0.60\%$
LogLoss | 0.2486 | 0.2492 | 0.2484 | 0.2458 | 0.2528 | 0.2509 | 0.2500 | 0.2489 | 0.2507 | 0.2505 | 0.3332 | 0.2549 | 0.2514 | 0.2437∗ | $+0.85\%$
Electronics | AUC | 0.7493 | 0.7504 | 0.7508 | 0.7524 | 0.7437 | 0.7455 | 0.7420 | 0.7438 | 0.7277 | 0.7386 | 0.5762 | 0.7376 | 0.7337 | 0.7532 | $+0.11\%$
LogLoss | 0.3207 | 0.3192 | 0.3186 | 0.3181 | 0.3318 | 0.3217 | 0.3227 | 0.3218 | 0.3272 | 0.3272 | 0.3751 | 0.3218 | 0.3336 | 0.3185 | $-$
Food | AUC | 0.7198 | 0.7208 | 0.7204 | 0.7188 | 0.6833 | 0.7103 | 0.7117 | 0.7154 | 0.7139 | 0.7135 | 0.5899 | 0.7188 | 0.7231 | 0.7322∗ | $+1.58\%$
LogLoss | 0.2938 | 0.2902 | 0.2944 | 0.3064 | 0.2891 | 0.2841 | 0.2829 | 0.2822 | 0.2810 | 0.2836 | 0.3658 | 0.3034 | 0.2903 | 0.2774∗ | $+1.28\%$
Instruments | AUC | 0.5811 | 0.5879 | 0.5789 | 0.6056 | 0.6137 | 0.6275 | 0.6301 | 0.6587 | 0.6618 | 0.6780 | 0.5623 | 0.5978 | 0.6635 | 0.6768 | $-$
LogLoss | 0.4921 | 0.4751 | 0.4514 | 0.3070 | 0.2115 | 0.1809 | 0.1717 | 0.1714 | 0.1711 | 0.1722 | 0.2494 | 0.4929 | 0.1720 | 0.1798 | $-$
Office | AUC | 0.7391 | 0.7405 | 0.7425 | 0.7401 | 0.7187 | 0.7334 | 0.7524 | 0.7522 | 0.7788 | 0.7554 | 0.5755 | 0.7432 | 0.7590 | 0.7812∗ | $+0.31\%$
LogLoss | 0.2334 | 0.2503 | 0.2077 | 0.2353 | 0.2116 | 0.2038 | 0.2018 | 0.2050 | 0.1974 | 0.2024 | 0.2743 | 0.2721 | 0.2179 | 0.1945∗ | $+1.47\%$
Toys | AUC | 0.7781 | 0.7821 | 0.7856 | 0.7820 | 0.7620 | 0.7666 | 0.7799 | 0.7806 | 0.7725 | 0.7717 | 0.5772 | 0.7623 | 0.7768 | 0.7964∗ | $+1.37\%$
LogLoss | 0.2196 | 0.2177 | 0.2207 | 0.2277 | 0.2554 | 0.2164 | 0.2141 | 0.2116 | 0.2124 | 0.2145 | 0.2937 | 0.2218 | 0.2262 | 0.2061∗ | $+2.59\%$
Cross- Platform |  MovieLens-1M | AUC | 0.8973 | 0.8969 | 0.8989 | 0.9018 | 0.8969 | 0.8967 | 0.8978 | 0.8963 | 0.8964 | 0.8970 | 0.7840 | 0.8970 | 0.9024 | 0.9029∗ | $+0.12\%$
LogLoss | 0.3166 | 0.3189 | 0.3147 | 0.3086 | 0.3165 | 0.3183 | 0.3183 | 0.3206 | 0.3167 | 0.3151 | 0.4482 | 0.3155 | 0.3096 | 0.3053∗ | $+1.07\%$
Efficiency |  Amazon | Latency (ms) | 0.0238 | 0.0345 | 0.0270 | 0.0323 | 0.0333 | 0.0357 | 0.1111 | 0.1250 | 0.0400 | 0.0370 | 6.5737 | 0.0714 | 0.0385 | 0.0476 | $-$
MovieLens-1M | Latency (ms) | 0.0247 | 0.0352 | 0.0264 | 0.0349 | 0.0307 | 0.0394 | 0.1289 | 0.1470 | 0.0564 | 0.0477 | 6.5737 | 0.0852 | 0.0453 | 0.0594 | $-$

We compare the proposed approach with twelve baseline methods on seven multi-
domain datasets and one cross-platform dataset. The overall performance is
presented in Table III, and we have the following observations:

(1) Single-domain methods (_i.e.,_ DeepFM [30], xDeepFM [32], DCNV2 [31] and
EulerNet [22]) perform well on the Movies and Books datasets, but exhibit
subpar performance on sparse domains with less interactions (_i.e.,_
Instruments and Office). This is consistent with the observations reported in
PEPNet [5], where single-domain methods exhibit varying performance across
different domains.

(2) Multi-domain methods (_i.e.,_ SharedBottom [33], MMoE [34], HMoE [35],
AESM2 [36], STAR [6] and PEPNet [5]) achieve comparable performance on the
Instruments and Office datasets, showing the effectiveness of multi-domain
information sharing.

(3) For semantic methods, P5 [28] performs poorly across all datasets due to
the limitation of LLMs in capturing collaborative signals. Besides, CTRL [12]
achieves better performance by aligning the representations between LLMs and
collaborative models, but it becomes less effective on the Instruments
dataset, showing its limitation of capturing the relatedness between different
domains.

(4) Our proposed approach achieves the best performance in almost all the
cases. It signifies that our suggested universal feature interactions are
better suited for the adaptation of multi-domain distributions. Further, the
cross-platform evaluation results show that our approach can be effectively
transferred to a new platform. Besides, UFINt+f has a great improvement over
UFINt, showing the effectiveness of incorporating feature adaptor to learn
domain-specific collaborative patterns.

Regarding the efficiency of the model, it can be observed that the latency of
single-domain models (_i.e.,_ DeepFM [30], xDeepFM [32], DCNV2 [31] and
EulerNet [22]) is relatively small due to their simplistic architecture. In
the case of SharedBottom [33], MMoE [34], STAR [6], and PEPNet [5], they are
more efficient as their learning strategies are simpler. Conversely, the
latency of AESM2 [36] and HMoE [35] is relatively large due to their intricate
architecture and learning algorithm. For semantic models, the latency of P5
[28] is much larger since the training process of PLMs is extremely time-
consuming. In contrast, CTRL [12] and our approach are much more efficient.
This is because we can cache the textual representations of the LLMs, and only
the lightweight feature interaction backbone needs to be deployed in the
inference stage, thereby preserving the efficient online inference akin to
traditional recommendation models.

###  III-C Further Analysis

####  III-C1 Zero-shot Learning Analysis

To show the transfer learning ability of our approach, we evaluate the zero-
shot performance of five methods (_i.e.,_ EulerNet [22], P5 [28], CTRL [12],
STAR [6] and our proposed UFIN), and compare the results to the best
performance of fully-trained single-domain methods. In this setting, we train
the model on three pre-trained datasets (_i.e.,_ _Movies_ , _Books_ and
_Electronics_) and directly test them on two downstream datasets (_i.e.,_
_Instruments_ and _Toys_) without further training. The downstream datasets
retain only the interactions involving overlapped users from the pre-trained
datasets. As shown in Figure 2, UFIN achieves the best zero-shot performance
on both datasets. On the Instruments dataset, UFIN performs even better than
the fully-trained single-doamin methods. These results demonstrate the strong
transferability and inductive capability of our approach in learning general
knowledge across different domains.

Figure 2: Performance comparison under the zero-shot setting.

####  III-C2 Cross-Platform Learning Analysis

In this part, we investigate the effectiveness of the proposed LLM based
representation approach. Specially, we examine whether the model pre-trained
on the Amazon datasets performs better than those without pre-training. We
compare the performance of the following representation methods: (1) Embedding
Look-up: use an embedding vector to represent each feature, (2) PLM based
encoding: encode the textual data with PLMs (_i.e.,_ BERT [43], T5 [37] and
FLAN-T5 [16]).

Experimental results are shown in Table IV. We observe that PLM-based methods
perform better when pre-training is applied, whereas the Embedding Look-up
method is negatively affected by pre-training. It indicates that natural
language is more promising as the general representations across different
scenarios. Furthermore, the approach with T5 [37] and FLAN-T5 [16] largely
outperforms other methods, showing the excellent language modeling capacity of
LLMs.

TABLE IV: Cross-platform performance on the MovieLens-1M dataset.

Representation Methods | w/o. Pre-train | w. Pre-train | Effect.
---|---|---|---
AUC | Log Loss | AUC | Log Loss
Embedding Look-up | 0.8997 | 0.3102 | 0.8990 | 0.3109 | $-0.08\%$
BERT [43] | 0.9001 | 0.3089 | 0.9009 | 0.3084 | $+0.09\%$
T5 [37] | 0.9008 | 0.3105 | 0.9027 | 0.3054 | $+0.21\%$
FLAN-T5 [16]) | 0.9013 | 0.3074 | 0.9030 | 0.3050 | $+0.19\%$

####  III-C3 Ablation Study

In this part, we analyze the impact of each proposed technique or component on
the model performance. We propose four variants as: (1) _w/o KD_ removing the
knowledge distillation procedure, (2) _w/o MoE-E_ removing the MoE model of
the encoder ($\bm{z}=\bm{s}$ in Eq. (3)), (3) _w/o UniF_ without generating
universal features ($\bm{e}_{j}$ in Eq.(6)), _i.e.,_ directly feeding the
latent vector $\tilde{\bm{z}}$ (Eq. (6)) into an MLP, and (4) _w/o Uid_
without fusing _user_id_ features (See Section II-C2). We show the results of
ablation study in Figure 3(a). We observe that all the proposed components are
effective to improve the model performance.

Besides, we explore the effect of the feature interaction experts (Section
II-D2). We show the results of five methods (_i.e.,_ MLP, AutoInt [42], CIN
[32], CrossNet [31], EulerNet [22]) in Figure 3(b). We can observe that choice
of EulerNet achieves the best performance, indicating that adaptive feature
interaction learning is the key component to improve the model capacity for
domain adaptation.

(a) Components (b) Feature Interaction Expert

Figure 3: Ablation study of UFIN on the Amazon dataset.

####  III-C4 Visualizing the Universal Features

In the literature of CTR predictions, there are few works that can transfer
knowledge across different platforms. To gain a deeper comprehension of the
actual knowledge being transmuted, we visually represent the ubiquitous
feature representations (See Eq. (6)) on the datasets of Amazon Movies, Amazon
Books, and MovieLens-1M by projecting them onto a two-dimensional plane using
t-SNE [44]. As shown in Figure 4, we can observe that the distribution of
MovieLens-1M is more closely aligned with Amazon_Movies than Amazon_Books.
This is reasonable since the contexts of MovieLens-1M and Amazon_Movies are
similar. Figure 4 further presents a case of two closely aligned interactions
(highlight in red circle) in the Amazon_Movies and MovieLens-1M dataset.
Notably, both sentences encompass the keyword _thriller_ , thereby
demonstrating our approach’s ability to capture the collective attributes
across disparate scenarios through general textual semantics.

Figure 4: Visualization of the universal features.

####  III-C5 Visualizing the Universal Feature Interactions

As discussed in Section II-D, UFIN captures the common feature interactions to
learn the relatedness between different domains. To verify the feature
interactions learned in UFIN, we visualize the interaction orders (i.e.,
$\mathcal{A}_{j}$ in Eq.(8)) of the Top-1 expert for the Amazon_movies,
MovieLens in Figure 5(a) and (b) respectively. We can observe that their
learned feature interactions exhibit substantial differences. We further
visualize one of their shared experts in Figure 5(c). Notably, the orders
learned in the shared expert exhibit some intersections with those in the
domain-specific expert of Amazon_movies (i.e.,
$\bm{\alpha}_{2},\bm{\alpha}_{6},\bm{\alpha}_{7}$) and MovieLens (i.e.,
$\bm{\alpha}_{3},\bm{\alpha}_{5},\bm{\alpha}_{6}$). These shared feature
interactions enhance the transferability of our model and enable it to capture
more generalized collaborative knowledge for CTR predictions.

(a) (a) Amazon_movies

(b) (b) MovieLens

(c) (c) Shared Expert

Figure 5: Visualization of the feature interactions.

####  III-C6 Impact of the Selected Experts Number $K$

In the expert selection part, the $K$ value of TopK function (See Eq.(9)) can
balance the number of domain shared experts. To analyze the influence of $K$,
we vary $K$ in the range of 1 to 7 and report the results in Figure 6. We can
observe that the performance on the MovieLens dataset increases as $K$
increases from 1 to 4, indicating that the common feature interactions learned
in the shared experts are valuable. Whereas on the Amazon_Movies dataset, UFIN
achieves the best performance as $K$ increases to 5. However, the model
performance decreases when $K$ exceeds 5. This indicates that adding more
shared experts may hinder the learning of domain-specific knowledge that hurt
the model performance.

(a) AUC (b) LogLoss

Figure 6: Impact of the selected experts number.

####  III-C7 The analysis of the Prompt Design

In this section, we explore the impact of different prompts construction
methods on training UFIN. Specifically, we consider several rules for
constructing prompts: 1) Omit auxiliary text descriptions and directly combine
feature fields and values using ”:”; 2) Mask the feature fields with a
meaningless unified word “Field”; 3) Remove certain feature field;

> Base: There is a user, whose gender is male, and occupation is student.
> There is a movie, its title is “Leon: The Professional”, its genre is Crime
> Drama Romance Thriller, and it is released at 1994.
>
> Prompt-1: gender: male; occupation: student; title: “Leon: The
> Professional”; genre: Crime Drama Romance Thriller; release year: 1994.
>
> Prompt-2: There is a user, whose Field is male, and Field is student. There
> is a movie, its Field is “Leon: The Professional”, its Field is Crime Drama
> Romance Thriller, and it is Field at 1994.
>
> Prompt-3: There is a user, whose gender is male, and occupation is student.
> There is a movie, its title is “Leon: The Professional”, its genre is Crime
> Drama Romance Thriller.

We utilize three types of prompts to train UFIN, and the results on both the
Amazon and MovieLens-1M datasets are depicted in Figures 7 and 8
correspondingly. We can observe that on both the Amazon and MovieLens-1M
datasets, the performance disparity of the model between Prompt-1, Prompt-2,
and the base on both the Amazon and MovieLens-1M datasets is negligible,
indicating that our model is not sensitive to the design or format of the
template content. Regarding Prompt-3 on the Amazon dataset, we eliminated the
_title_ (w.o. title) and _description_ (w.o. desc) fields. The analysis from
Figure 7 reveals that eliminating either the _title_ or _description_ results
in decreased model performance, implying the importance of the semantic
information contained within these text features. Besides, the removal of the
_description_ field leads to a greater decline in model performance. This
disparity is due to the _description_ containing more substantial information
compared to the _title_. On the MovieLens-1M dataset, we remove the fields
_release year_ (_i.e.,_ w.o. Year) and _genre_ (_i.e.,_ w.o. genre). As shown
in Figure 8, we can observe that the model’s performance changes little after
removing _release year_ , but there is a sharp decline in the model’s
performance after removing the _genre_ field. This is because compared to the
movie _genre_ , _release year_ contains fewer personalized item information.
These findings suggest that our approach demonstrates minimal correlation with
the design of the prompt text but exhibits a strong relationship with the
quantity of information (_personalized features_) it encompasses.

(a) AUC (b) LogLoss

Figure 7: Performance on Amazon w.r.t. different prompts.

(a) AUC (b) LogLoss

Figure 8: Performance on MovieLen-1M w.r.t. different prompts.

##  IV Related work

###  IV-A Feature Interaction Learning.

Predicting the probability of users clicking ads or items is critical in
online advertising [45] and recommender systems. The key to achieve a great
CTR performance is to learning effective feature interactions. Typically, most
of existing CTR models [46, 47, 48, 49] adopt the embedding look-up and
feature interaction learning framework to capture the underlying correlation
between users and items. For example, FM [21] introduces low-dimensional
vectors to represent the features and uses inner product of them to capture
second-order feature interactions. Besides, FwFM [48] and FmFM [50] propose
incorporating field information to improve the pair-wise feature interactions.
However, these approaches only focus on learning the low-order feature
interactions, which largely degrades the model capacity and results in the
suboptimal performance.

Recently, a number of approaches [32, 31] are proposed to model the high-order
feature interactions. Among them, xDeepFM [32] conducts multi-layer
convolution operations on the concatenated vectors to model high-order
interactions; DCNV2 [31] performs the kernel product on the concatenated
feature vectors to construct higher-order feature interactions. These
approaches have largely raised the performance bar of CTR predictions.
Typically, these approaches follow a common paradigm to model the feature
interactions: they first pre-define a maximal order and only consider
conducting feature interactions within the pre-defined orders. Though
effective to some extend, the orders of feature interactions are empirically
designed, which cannot accurately capture the underlying feature relationship
in the real-world scenarios. Further, due to the exponential growth of high-
order feature combinations, such approaches often set a small order, which
cannot scale to the high-order cases in the industrial scenarios.

Considering the above limitations, some approaches [22, 23, 24] are proposed
to automatically learn the feature interaction orders from the data. The core
idea of these approaches is to encode the features into a special vector space
(_e.g.,_ logarithm vector space). As such, the complicated feature
interactions are converted to the simple linear combinations and the orders
are cast into the learnable coefficients, enabling the adaptive learning of
the interaction orders. However, previous studies mainly focus on learning the
orders within a single-domain, and seldom investigate the transferability of
the order information across different domains or tasks.

###  IV-B Multi-Domain CTR predictions.

Traditional CTR prediction models are developed for modeling the user’s
interest within a single domain, _i.e.,_ they are training using examples
collected from a single domain and serving the prediction of a single task.
However, in large-scale real-world industrial platforms, the user behavior
data are often collected from multi-domains. Merely mixing all the data and
training a single shared CTR model cannot yield satisfactory results across
all domains owing to the substantial distribution variance among diverse
scenarios, which is called domain seesaw phenomenon in recommender systems.
Such paradigm severely restricts the efficient utilization of extensive user
behavior data in business scenarios.

To address these issues, a number of approaches [6, 5, 7, 8] are proposed to
enhance the information sharing and improve the multi-domain performance.
Among them, SharedBottom [33] introduces a unique embedding layer to learn a
commonly shared feature representations for different domains. Besides, MMoE
[34] extends the multi-task learning (MTL) methods into multi-domain
scenarios, by regarding each domain as a specific task; HMoE [35] extends MMoE
to scenario-aware experts using a gradient cutting trick to explicitly encode
scenario correlations. Despite the progress, these approaches cannot
effectively exploit the domain relationship and suffer from the degradation of
model capability.

As a promising research direction, some recent studies [35, 36, 6, 5] propose
to explicitly introduce the domain-shared parameters to compactly learn the
common knowledge across different domains. The core idea of these approaches
is to introduce a shared neural network for learning the common knowledge
across diverse domains, while simultaneously integrating multiple domain-
specific sub-networks to capture the distinct characteristics of each domain.
For example, STAR [6] proposes a star topology architecture, which consists of
a shared center network and multiple domain-specific networks for the
adaptation of multi-domain distributions; PEPNet [5] introduces an embedding
personalized network to align multi-domain feature representations. Despite
the progress, these approaches still rely on explicit embedding look-up
operation for the given feature IDs, which impairs the inherent semantic of
features and makes it difficult to be applied in different platforms.

###  IV-C Semantic CTR Prediction Models.

With the development of natural language processing (NLP), the large language
models (LLMs [15]) have shown excellent language modeling capacity in various
downstream tasks, which promises researchers to apply LLMs into the
recommendation tasks. In the context of CTR predictions, the large language
models (LLMs) have two common application methods in CTR prediction models:
scoring/ranking function and feature engineering encoder.

For the first kind of application, the LLMs mainly serve as the learning
backbone to generate predictive results. For example, P5 [28] converts
different recommendation task into text generation and employs T5 [37] model
to generate the result. Besides, M6-Rec [13] uses the M6 [19] model to deliver
the recommendation in a prompting learning framework. These approaches exploit
the textual data as a general input form, which can effectively bridge the
semantic gap between different domains, enabling them to be effectively
transferred to a new domain or platform.

Despite the progress, compared to the traditional CTR prediction models, LLMs
cannot effectively capture the collaborative patterns that severely limits the
model capacity. As a promising approach, a recent study [12] proposes a
contrastive learning framework to align the knowledge between LLMs and
collaborative models. Due to the limited capacity of collaborative model, it
cannot adequately learn the common knowledge across different domains. In
contrast, our solution is to utilize the LLMs to generate universal features
from textual data, learning the collective attributes of different domains. As
such, we can learn the universal feature interactions to capture the
generalized collaborative patterns across diverse domains, naturally
integrating the world knowledge of LLMs and the collaborative knowledge
between users and items.

##  V Conclusion

In this paper, we propose the Universal Feature Interaction Network (UFIN) for
multi-domain CTR prediction. Unlike previous approaches that heavily rely on
modeling ID features for developing the CTR predictions, our approach leverage
the textual data to learn the universal feature interactions. Specifically, we
regard the text and features as two modalities that can be mutually converted.
As such, we employ a LLM-based encoder-decoder architecture to transform the
data from text modality to feature modality, obtaining universal feature
representations. Building upon this foundation, we design an adaptive feature
interaction model enhanced by a mixture-of-experts (MoE) architecture for
capturing the generlized feature interactions across different domains. To
effectively learn the collaborative patterns across different domains, we
propose a multi-domain knowledge distillation framework to improve the
training of our approach.

As future work, we will explore how to better integrate anonymous features
with generated universal feature representations. In addition, we will also
consider incorporating the Conversion Rate (CVR) prediction task into our
approach to capture more effective correlation between different tasks.

##  VI Proofs of Theorem 1

###### Proof.

By contradiction, we assume there exists two domains $\mathcal{D}_{u}$ and
$\mathcal{D}_{v}$ ($u\neq v$) that satisfies
$\mathcal{S}_{u}\cap\mathcal{S}_{v}=\emptyset$.

According to the principle of inclusion-exclusion, we have:

$\displaystyle|\mathcal{S}_{u}\cup\mathcal{S}_{v}|$
$\displaystyle=|\mathcal{S}_{u}|+|\mathcal{S}_{v}|-|\mathcal{S}_{u}\cap\mathcal{S}_{v}|$
$\displaystyle=K+K-0$ $\displaystyle=2K>2\times\lceil{L/2}\rceil\geq L$

On the other hand, since $\mathcal{S}_{u}\subseteq\mathcal{A}$ and
$\mathcal{S}_{v}\subseteq\mathcal{A}$, we have
$\mathcal{S}_{u}\cup\mathcal{S}_{v}\subseteq\mathcal{A}$.

Therefore, we have:

$\displaystyle|\mathcal{S}_{u}\cup\mathcal{S}_{v}|\leq|\mathcal{A}|=L.$

Since the assumption leads to a contradiction, our initial assumption that
$\exists u\neq v,\mathcal{S}_{u}\cap\mathcal{S}_{v}=\emptyset$ must be False.

Therefore, $\forall u\neq v,\mathcal{S}_{u}\cap\mathcal{S}_{v}\neq\emptyset$.
∎

## References

  * [1] H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir _et al._ , “Wide & deep learning for recommender systems,” in _Proceedings of the 1st workshop on deep learning for recommender systems_ , 2016, pp. 7–10.
  * [2] X. He and T.-S. Chua, “Neural factorization machines for sparse predictive analytics,” in _Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval_ , 2017, pp. 355–364.
  * [3] Z. Li, W. Cheng, Y. Chen, H. Chen, and W. Wang, “Interpretable click-through rate prediction through hierarchical attention,” in _Proceedings of the 13th International Conference on Web Search and Data Mining_ , 2020, pp. 313–321.
  * [4] Y. Shan, T. R. Hoens, J. Jiao, H. Wang, D. Yu, and J. Mao, “Deep crossing: Web-scale modeling without manually crafted combinatorial features,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 255–262.
  * [5] J. Chang, C. Zhang, Y. Hui, D. Leng, Y. Niu, and Y. Song, “Pepnet: Parameter and embedding personalized network for infusing with personalized prior information,” _arXiv preprint arXiv:2302.01115_ , 2023.
  * [6] X.-R. Sheng, L. Zhao, G. Zhou, X. Ding, B. Dai, Q. Luo, S. Yang, J. Lv, C. Zhang, H. Deng _et al._ , “One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 4104–4113.
  * [7] Y. Zhang, X. Wang, J. Hu, K. Gao, C. Lei, and F. Fang, “Scenario-adaptive and self-supervised model for multi-scenario personalized recommendation,” in _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, 2022, pp. 3674–3683.
  * [8] J. Zhou, X. Cao, W. Li, L. Bo, K. Zhang, C. Luo, and Q. Yu, “Hinet: Novel multi-scenario & multi-task learning with hierarchical information extraction,” _arXiv preprint arXiv:2303.06095_ , 2023.
  * [9] J. Lin, X. Dai, Y. Xi, W. Liu, B. Chen, X. Li, C. Zhu, H. Guo, Y. Yu, R. Tang, and W. Zhang, “How can recommender systems benefit from large language models: A survey,” 2023.
  * [10] Y. Hou, S. Mu, W. X. Zhao, Y. Li, B. Ding, and J.-R. Wen, “Towards universal sequence representation learning for recommender systems,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 585–593.
  * [11] J. Li, M. Wang, J. Li, J. Fu, X. Shen, J. Shang, and J. McAuley, “Text is all you need: Learning language representations for sequential recommendation,” _arXiv preprint arXiv:2305.13731_ , 2023.
  * [12] X. Li, B. Chen, L. Hou, and R. Tang, “Ctrl: Connect tabular and language model for ctr prediction,” _arXiv preprint arXiv:2306.02841_ , 2023.
  * [13] Z. Cui, J. Ma, C. Zhou, J. Zhou, and H. Yang, “M6-rec: Generative pretrained language models are open-ended recommender systems,” _arXiv preprint arXiv:2205.08084_ , 2022.
  * [14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [15] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [16] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma _et al._ , “Scaling instruction-finetuned language models,” _arXiv preprint arXiv:2210.11416_ , 2022.
  * [17] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [19] J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia _et al._ , “M6: A chinese multimodal pretrainer,” _arXiv preprint arXiv:2103.00823_ , 2021.
  * [20] S. Rendle, W. Krichene, L. Zhang, and J. Anderson, “Neural collaborative filtering vs. matrix factorization revisited,” in _Fourteenth ACM conference on recommender systems_ , 2020, pp. 240–248.
  * [21] S. Rendle, “Factorization machines,” in _2010 IEEE International conference on data mining_. IEEE, 2010, pp. 995–1000.
  * [22] Z. Tian, T. Bai, W. X. Zhao, J.-R. Wen, and Z. Cao, “Eulernet: Adaptive feature interaction learning via euler’s formula for ctr prediction,” _arXiv preprint arXiv:2304.10711_ , 2023.
  * [23] W. Cheng, Y. Shen, and L. Huang, “Adaptive factorization network: Learning adaptive-order feature interactions,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 3609–3616.
  * [24] S. Cai, K. Zheng, G. Chen, H. Jagadish, B. C. Ooi, and M. Zhang, “Arm-net: Adaptive relation modeling network for structured data,” in _Proceedings of the 2021 International Conference on Management of Data_ , 2021, pp. 207–220.
  * [25] M. Richardson, E. Dominowska, and R. Ragno, “Predicting clicks: estimating the click-through rate for new ads,” in _Proceedings of the 16th international conference on World Wide Web_ , 2007, pp. 521–530.
  * [26] J. Zhu, J. Liu, W. Li, J. Lai, X. He, L. Chen, and Z. Zheng, “Ensembled ctr prediction via knowledge distillation,” in _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, 2020, pp. 2941–2958.
  * [27] Z. Tian, T. Bai, Z. Zhang, Z. Xu, K. Lin, J.-R. Wen, and W. X. Zhao, “Directed acyclic graph factorization machines for ctr prediction via knowledge distillation,” in _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_ , 2023, pp. 715–723.
  * [28] S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang, “Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5),” in _Proceedings of the 16th ACM Conference on Recommender Systems_ , 2022, pp. 299–315.
  * [29] J. Ni, J. Li, and J. McAuley, “Justifying recommendations using distantly-labeled reviews and fine-grained aspects,” in _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_ , 2019, pp. 188–197.
  * [30] H. Guo, R. Tang, Y. Ye, Z. Li, and X. He, “Deepfm: a factorization-machine based neural network for ctr prediction,” _arXiv preprint arXiv:1703.04247_ , 2017.
  * [31] R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi, “Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 1785–1797.
  * [32] J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, and G. Sun, “xdeepfm: Combining explicit and implicit feature interactions for recommender systems,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1754–1763.
  * [33] S. Ruder, “An overview of multi-task learning in deep neural networks,” _arXiv preprint arXiv:1706.05098_ , 2017.
  * [34] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1930–1939.
  * [35] H. Tang, J. Liu, M. Zhao, and X. Gong, “Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations,” in _Proceedings of the 14th ACM Conference on Recommender Systems_ , 2020, pp. 269–278.
  * [36] X. Zou, Z. Hu, Y. Zhao, X. Ding, Z. Liu, C. Li, and A. Sun, “Automatic expert selection for multi-scenario and multi-task search,” in _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2022, pp. 1535–1544.
  * [37] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [38] J. Zhu, J. Liu, S. Yang, Q. Zhang, and X. He, “Open benchmarking for click-through rate prediction,” ser. CIKM ’21, 2021, p. 2759–2769.
  * [39] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [40] J. M. Lobo, A. Jiménez-Valverde, and R. Real, “Auc: a misleading measure of the performance of predictive distribution models,” _Global ecology and Biogeography_ , vol. 17, no. 2, pp. 145–151, 2008.
  * [41] A. Buja, W. Stuetzle, and Y. Shen, “Loss functions for binary class probability estimation and classification: Structure and applications,” _Working draft, November_ , vol. 3, p. 13, 2005.
  * [42] W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang, “Autoint: Automatic feature interaction learning via self-attentive neural networks,” in _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 1161–1170.
  * [43] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [44] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [45] D. Agarwal, B. Long, J. Traupman, D. Xin, and L. Zhang, “Laser: A scalable response prediction platform for online advertising,” in _Proceedings of the 7th ACM international conference on Web search and data mining_ , 2014, pp. 173–182.
  * [46] C. Xu and M. Wu, “Learning feature interactions with lorentzian factorization machine,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 6470–6477.
  * [47] W. Lu, Y. Yu, Y. Chang, Z. Wang, C. Li, and B. Yuan, “A dual input-aware factorization machine for ctr prediction,” in _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 3139–3145.
  * [48] J. Pan, J. Xu, A. L. Ruiz, W. Zhao, S. Pan, Y. Sun, and Q. Lu, “Field-weighted factorization machines for click-through rate prediction in display advertising,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 1349–1357.
  * [49] M. Blondel, A. Fujino, N. Ueda, and M. Ishihata, “Higher-order factorization machines,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [50] Y. Sun, J. Pan, A. Zhang, and A. Flores, “Fm2: Field-matrixed factorization machines for recommender systems,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 2828–2837.