# Multi-Scale User Behavior Network for Entire Space
Multi-Task Learning

Jiarui Jin1, Xianyu Chen1, Weinan Zhang1,∗, Yuanbo Chen2, Zaifan Jiang2,
Zekun Zhu2, Zhewen Su2, Yong Yu1 1Shanghai Jiao Tong University, 2Alibaba
GroupChina jinjiarui97, xianyujun, wnzhang<EMAIL_ADDRESS>yuanbo.cyb,zaifan.jzf, zekun.zhu<EMAIL_ADDRESS>
(2022)

###### Abstract.

Modelling the user’s multiple behaviors is an essential part of modern
e-commerce, whose widely adopted application is to jointly optimize click-
through rate (CTR) and conversion rate (CVR) predictions. Most of existing
methods are designed to encode the dependence between the behavior paths
“observation $\rightarrow$ click” and “click $\rightarrow$ conversion” by
combining the two-tower architecture with multi-task learning techniques to
optimize multiple ranking objectives. However, such systems overlook the
effect of two key characteristics of the user’s behaviors: for each item list,
(i) _contextual dependence_ refers to that the user’s behaviors on any item
are not purely determinated by the item itself but also are influenced by the
user’s previous behaviors (e.g., clicks, purchases) on other items in the same
sequence; (ii) _multiple time scales_ means that users are likely to click
frequently but purchase periodically. To this end, we develop a new multi-
scale user behavior network named Hierarchical rEcurrent Ranking On the Entire
Space (HEROES) which incorporates the contextual information to estimate the
user multiple behaviors in a multi-scale fashion. Concretely, we introduce a
hierarchical framework, where the lower layer models the user’s engagement
behaviors while the upper layer estimates the user’s satisfaction behaviors.
The proposed architecture can automatically learn a suitable time scale for
each layer to capture the dynamic user’s behavioral patterns. Besides the
architecture, we also introduce the Hawkes process to form a novel recurrent
unit which can not only encode the items’ features in the context but also
formulate the excitation or discouragement from the user’s previous behaviors.
We further show that HEROES can be extended to build unbiased ranking systems
through combinations with the survival analysis technique. Extensive
experiments over three large-scale industrial datasets demonstrate the
superiority of our model compared with the state-of-the-art methods.

Multi-Scale User Modeling; Multi-Task Learning

††conference: The 45th International ACM SIGIR Conference on Research and
Development in Information Retrieval; October 17 - 22, 2022; Georgia,
USA††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of
the 31st ACM International Conference on Information and Knowledge Management;
October 17–21, 2022; Atlanta, GA, USA††booktitle: Proceedings of the 31st ACM
International Conference on Information and Knowledge Management (CIKM ’22),
October 17–21, 2022, Atlanta, GA, USA††price: 15.00††doi:
10.1145/3511808.3557405††isbn: 978-1-4503-9236-5/22/10††ccs: Information
systems Recommender systems

ACM Reference Format:
Jiarui Jin, Xianyu Chen, Weinan Zhang, Yuanbo Chen, Zaifan Jiang, Zekun Zhu,
Zhewen Su, Yong Yu. 2022. Multi-Scale User Behavior Network for Entire Space
Multi-Task Learning. In Proceedings of the 31st ACM International Conference
on Information and Knowledge Management, October 17–21, 2022, Atlanta, GA,
USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3511808.3557405

##  1\. Introduction

Implicit feedbacks from the user’s behaviors are much important data sources
for any personalized online service in information systems (Zhao et al.,
2019). Such information systems often need to optimize multiple objectives at
the same time (Ma et al., 2018b). For example, when recommending videos to a
user, the user is expected to not only click and watch the video but also to
purchase it. Specifically, these behaviors can be categorized into two
classes: (i) engagement behaviors, such as the user’s observations and clicks;
(ii) satisfaction behaviors, such as the user’s watch time and purchases. As
Figure 1 shows, a user would like to click and eventually purchase an observed
item, which indicates a typical behavior path “observation $\rightarrow$ click
$\rightarrow$ conversion” (i.e., “engagement behaviors $\rightarrow$
satisfaction behaviors”) (Wen et al., 2020; Ma et al., 2018a). We call it
entire space behavior path.

11footnotetext: Weinan Zhang is the corresponding author.

Recent researches are mainly developed based on the existing multi-task
learning techniques (Caruana, 1997) to simultaneously learn multiple types of
user behaviors, which can be roughly categorized into two directions. One
direction (Zhao et al., 2015; Ma et al., 2018a; Wen et al., 2020; Meng et al.,
2020) is to leverage the behavior decomposition which constructs the user’s
micro-actions as auxiliary information to promote the CTR and CVR predictions.
Another line (Ma et al., 2018b; Zhao et al., 2019; Wang et al., 2020) is to
design an effective feature sharing strategy among the CTR and CVR prediction
models.

However, when modelling the user’s multiple behaviors, almost all the existing
papers have not well used, even may not be aware of, the following key
characteristics of user behavior patterns:

  * •

There exists _contextual dependence_ among the multiple behaviors in one list:
a user’s behaviors on an item may affect her behaviors on its following items.
This influence would be either excitation or discouragement. One example is
that if a teen has purchased a hat, she then might not be interested in other
hats.

  * •

Different behavior paths trigger with _multiple time scales_. For example, in
Taobao e-commerce
platform111https://tianchi.aliyun.com/datalab/dataSet.html?dataId=408, the
average time interval of clicks is 12.23, while that of purchases is 32.08,
which indicates that the behavior paths “observation $\rightarrow$ click” and
“click $\rightarrow$ conversion” happen with different time scales.

Figure 1. Top diagram shows the pipeline information systems, which consists
of a ranking system and the user multiple behaviors. Bottom figure illustrates
the entire space behavior path “observation $\rightarrow$ click $\rightarrow$
conversion”.

An illustrated example of the above characteristics is provided in Figure 2.
To this end, we propose a novel framework called Hierarchical rEcurrent
Ranking On the Entire Space (HEROES) to formulate the entire space user
behavior path (i.e., “observation $\rightarrow$ click $\rightarrow$
conversion”) in a multi-scale fashion. Concretely, as shown in Figure 4(Left),
HEROES establishes a hierarchical architecture: the lower layer (i.e., CTR
layer) estimates the engagement behaviors (i.e., “observation $\rightarrow$
click”), while the upper layer (i.e., CVR layer) estimates the satisfaction
behaviors (i.e., “click $\rightarrow$ conversion”). We tweak the design of
gates in (Chung et al., 2016) and allocate them both within and between two
layers, which allows HEREOS to automatically learn a suitable time scale for
each layer. Besides the architecture, we then incorporate the neural Hawkes
process (Mei and Eisner, 2016) into the recurrent unit (Chung et al., 2016) to
form a new unit (called HEROES unit, as shown in Figure 4(Right)), which can
not only encode the contextual item features but also model the excitation or
discouragement brought from the user’s past behaviors. By applying HEROES
units into the hierarchical architecture, HEROES is able to simultaneously
model all the top-down contextual contents (including the item features and
the user’s behaviors) and learn appropriate time scales (as Figure 3 depicts)
for the entire space behavior path “observation $\rightarrow$ click
$\rightarrow$ conversion”.

Furthermore, we also show that HEROES can be seamlessly applied to the
unbiased learning-to-rank task by incorporating with the survival analysis
technique (Jin et al., 2020).

The major contributions of this paper can be outlined as follows.

  * •

We propose a novel paradigm named HEROES, where we model the multiple user
behaviors on entire space (i.e., “observation $\rightarrow$ click
$\rightarrow$ conversion”) in a multi-scale manner.

  * •

We design a novel recurrent unit to take both the contextual items and the
user’s previous behaviors into consideration.

  * •

We show that our approach can be seamlessly used for unbiased ranking by
incorporating with survival analysis technique.

We conduct the comprehensive experiments on three industrial datasets, whose
results exhibit that our method can learn an effective ranker over multiple
objectives. To our knowledge, this work is the first work that simultaneously
models the user’s multiple behaviors in a multi-scale fashion.

##  2\. Preliminary

###  2.1. Learning-to-Rank with Multiple Objectives

The fundamental goal of learning-to-rank (LTR) scenarios is to learn a ranker
$f$, which assigns a score to the item according to its feature. Then, the
item list concerning a query $q$ is provided in the descending order of their
scores. Let $\mathcal{D}_{q}$ denote the set of items associated with $q$,
$d_{i}$ denote the $i$-th item in $\mathcal{D}_{q}$ and $\bm{x}_{i}$ denote
the feature vector of $d_{i}$. Let $b_{i}$ represent the score of $d_{i}$. For
simplicity, we only consider the binary score here, i.e., $b_{i}=0$ or
$b_{i}=1$. One can easily extend it to the multi-level scores. We can describe
the risk function as

(1)
$\mathcal{R}(f)=\sum_{q}\sum_{d_{i}\in\mathcal{D}_{q}}L(f(\bm{x}_{i}),b_{i}),$

where $f$ denotes a ranker, and $L(f(\bm{x}_{i}),b_{i})$ denotes a loss
function. The goal of LTR is to find the optimal ranker $f^{*}$ that minimizes
the loss function. In the CTR prediction, the ranker is learned with implicit
feedbacks containing click labeled data (i.e., the score $b_{i}$ of $d_{i}$ in
Eq. (1) is replaced by its click signal $c_{i}$); while in the CVR prediction,
$b_{i}$ of $d_{i}$ in Eq. (1) is replaced by its conversion signal $v_{i}$.
Here $v_{i}$ is a binary value that denotes whether the conversion event
occurs at $d_{i}$.

The goal in this paper is to jointly optimize the CTR and CVR predictions.
Formally, the input is $\mathcal{D}_{q}$ organized as a set of data samples
$\\{(\bm{x},c,v,I)\\}$, where $\bm{x}$ is the item features, $I$ is the length
of the item list, $c$ is the click signal, and $v$ is the conversion signal.
The output is the predictions of click and conversion probabilities of each
item $d_{i}$ (i.e., $P(c_{i}=1|\bm{x}_{i})$ and $P(v_{i}=1|\bm{x}_{i})$).

Figure 2. An illustrated example of the characteristics of the user’s
behaviors: (i) _contextual dependence_ : click occurs when browsing
interesting items, which may encourage the observation, and conversion happens
when finding a favored item, which may discourage the observation; (ii)
_multiple time scales_ : conversions always happen after the user finding a
favorite clicked item, and thus conversions occur less frequently than clicks.

###  2.2. Relevance for Behavior Dependence

Note that the user’s multiple behaviors naturally have dependence among each
other (a.k.a., behavior path “observation $\rightarrow$ click $\rightarrow$
conversion”). Hence, instead of separately modelling the click and conversion,
the current prevailing approaches introduce the post-click relevance (denoted
as $r^{v}_{i}$ for $d_{i}$) (so called the post-view conversion rate (Ma et
al., 2018a; Wen et al., 2020)), which is defined as

(2) $P(r^{v}_{i}=1|\bm{x}_{i})\coloneqq
P(v_{i}=1|c_{i}=1;\bm{x}_{i})=\frac{P(v_{i}=1|\bm{x}_{i})}{P(c_{i}=1|\bm{x}_{i})},$

which allows the model to first separately estimate $P(c_{i}=1|\bm{x}_{i})$
and $P(r^{v}_{i}=1|\bm{x}_{i})$ and then recover $P(v_{i}=1|\bm{x}_{i})$ by
computing the production.

As the post-click relevance (i.e., $r^{v}_{i}$ for $d_{i}$) is built based on
the behavior path “click $\rightarrow$ conversion”, we can similarly define
the prior-click relevance (denoted as $r^{c}_{i}$ for $d_{i}$) as

(3) $P(r^{c}_{i}=1|\bm{x}_{i})\coloneqq
P(c_{i}=1|o_{i}=1;\bm{x}_{i})=\frac{P(c_{i}=1|\bm{x}_{i})}{P(o_{i}=1|\bm{x}_{i})},$

which reflects a simple fact that a user clicks ($c_{i}=1$) the item $d_{i}$
only when it is both observed ($o_{i}=1$) and perceived as prior-click
relevant ($r^{c}_{i}=1$). Eq. (3) has been widely used in unbiased LTR
researches (Joachims et al., 2017; Wang et al., 2016; Wang et al., 2018; Ai et
al., 2018; Jin et al., 2020), whose goal is to discover the prior-click
relevance from the biased click data. In contrast, biased LTR refers to
directly regarding the click as the prior-click relevance, where the prior-
click relevance is defined as

(4) $P(r^{c}_{i}=1|\bm{x}_{i})\coloneqq P(c_{i}=1|\bm{x}_{i}).$

##  3\. HEROES

In this section, we present the HEROES in the context of the biased LTR (i.e.,
using Eq. (4)) where we first describe our architecture design and then
introduce the proposed HEROES unit to construct the architecture. Finally, we
show the loss function for biased LTR.

###  3.1. HEROES Architecture

In HEROES, we first divide the entire space behavior path into two layers:
“observation $\rightarrow$ click” in the CTR layer and “click $\rightarrow$
conversion” in the CVR layer, and then model the user’s multiple behaviors
both within and between two layers.

We begin by introducing the definition of the inherent relevance:

###### Definition 0.

_(Inherent Relevance)_ For each item $d_{i}$, we define a binary variable
$\widetilde{r}_{i}$ as the inherent relevance. Specifically,
$\widetilde{r}_{i}^{c}$ and $\widetilde{r}_{i}^{v}$ are the inherent relevance
to motivate a user to click and purchase $d_{i}$ respectively, both of which
are solely determinated by the item features and are free of the effect from
all the external factors such as the contextual items and the user’s past
behaviors.

In contrast, we call $r^{v}_{i}$ in Eq. (2) and $r^{c}_{i}$ in Eq. (3)
_behavioral relevance_ for conversion and click of $d_{i}$ respectively, which
are affected by the external factors. To estimate $\widetilde{r}_{i}$ and
$r_{i}$222For simplicity, we use $r_{i}$ to denote both $r^{c}_{i}$ and
$r^{v}_{i}$; and similar notations for $\widetilde{r}_{i}$, $h_{i}$,
$f_{\theta}$, $z$. for each item $d_{i}$, we further introduce
$\widetilde{h}_{i}$ and $h_{i}$ which are defined as

(5) $\displaystyle\widetilde{h}^{c}_{i}$ $\displaystyle\coloneqq
P(\widetilde{r}^{c}_{i}=1),\ \widetilde{h}^{v}_{i}\coloneqq
P(\widetilde{r}^{v}_{i}=1);$ $\displaystyle h^{c}_{i}$ $\displaystyle\coloneqq
P(r^{c}_{i}=1),\ h^{v}_{i}\coloneqq P(r^{v}_{i}=1).$

Table 1. A summary of notations regarding the item $d_{i}$.  Notations |  Explanations
---|---
$c_{i},v_{i}$ |  Click, conversion (implicit feedback)
$r^{c}_{i}$ |  Prior-click (behavioral) relevance (see Eq. (3) for unbiased LTR, and Eq. (4) for biased LTR)
$r^{v}_{i}$ |  Post-click (behavioral) relevance (see Eq. (2))
$\widetilde{r}^{c}_{i},\widetilde{r}^{v}_{i}$ |  Inherent relevance (see definition 1)
$h^{c}_{i},h^{v}_{i},\widetilde{h}^{c}_{i},\widetilde{h}^{v}_{i}$ |  Probability of $r^{c}_{i}=1,r^{v}_{i}=1,\widetilde{r}^{c}_{i}=1,\widetilde{r}^{v}_{i}=1$

Intra-Layer Behavior Modelling. In each layer (i.e., behavior path), since the
user’s behaviors on each item $d_{i}$ (i.e., $h_{i}$) can be either excited or
discouraged by the user’s previous behaviors, we apply the Hawkes process
(Hawkes, 1971; Embrechts et al., 2011) to formulate the behavioral relevance
as

(6) $h_{i}:=\widetilde{h}_{i}+\sum_{j\leq
i}\lambda_{j}\exp{(-\delta_{j}(t_{i}-t_{j}))},$

where $\lambda_{j}\in\mathbb{R}$ is the _learnable_ degree to which the user’s
behaviors (e.g., click or conversion) on item $d_{j}$ initially excite (when
$\lambda_{j}>0$) or discourage (when $\lambda_{j}<0$) that on item $d_{i}$;
and $\delta_{j}>0$ is the _learnable_ decay rate of the excitation or
discouragement. In other words, when studying $d_{i}$, as the time interval
$t_{i}-t_{j}$ increases, its behavioral relevance $h_{i}$ might both rise and
fall (conditioned on the effects from the intermediate items, i.e.,
$\\{\lambda_{j}\\}_{j\leq i}$), but eventually approach its inherent relevance
$\widetilde{h}_{i}$, as the influences from previous behaviors on $d_{j}$ will
decay toward 0 at rate $\delta_{j}>0$. Here, $t_{i}$, $t_{j}$ are the behavior
occurrence time in continuous space and can be roughly approximated by $i$ and
$j$ in discrete time space.

Note that Eq. (6) can be regarded as a conceptual formulation, as it can not
guarantee $h_{i},\widetilde{h}_{i}\in[0,1]$. We will later introduce our
HEROES unit design which implements Eq. (6) to the contextual item feature
modelling in Section 3.2.

Inter-Layer Behavior Modelling. As introduced above, for each item $d_{i}$,
there are two behavioral factors: the prior-click relevance $h^{c}_{i}$
representing how likely the user would click $d_{i}$ after observing it in the
CTR layer, and the post-click relevance $h^{v}_{i}$ representing how likely
the user would purchase $d_{i}$ after clicking it in the CVR layer. We
explicitly model the correlations across these layers as followings.

(7) $h^{v}_{i}=\phi(h^{c}_{i}),\ h^{c}_{i+1}=\psi(h^{v}_{i}),$

where $\phi(\cdot)$ and $\psi(\cdot)$ denote parameterized mapping functions,
and we will specify them in later Section 3.2. The intuition behind Eq. (7) is
straightforward: The former equation represents the case where a user may
click on an item because its abstract content (e.g., title) is interesting
(i.e., $h^{c}_{i}$) and purchase it after carefully checking whether its
detailed information (e.g., description) is relevant (i.e., $h^{v}_{i}$). The
latter equation shows the case where a user finds and purchases a favored item
(i.e., $h^{v}_{i}$), and then it is likely for the user to stop browsing and
not click its following items (i.e, $h^{c}_{i+1}$).

Figure 3. An illustrated example of the gate mechanism: $g_{i}=0$ denotes no
click at $d_{i}$ where the contextual information is passed via
“observation$\rightarrow$click” in the CTR layer; and $g_{i}=1$ denotes a
click at $d_{i}$ where the contextual information is summarized in the CTR
layer and is further passed via “click$\rightarrow$conversion” in the CVR
layer.

According to Eq. (7), we introduce a hierarchical architecture, whose
formulation can be written as

(8) $\displaystyle h^{c}_{i},\widetilde{h}^{c}_{i}$
$\displaystyle=f^{c}_{\theta}(h^{c}_{i-1},h^{v}_{i-1},\widetilde{h}^{c}_{i-1},\widetilde{h}^{v}_{i-1};\bm{x}_{q});$
$\displaystyle h^{v}_{i},\widetilde{h}^{v}_{i}$
$\displaystyle=f^{v}_{\theta}(h^{c}_{i},h^{v}_{i-1},\widetilde{h}^{c}_{i},\widetilde{h}^{v}_{i-1};\bm{x}_{q}),$

where $f^{c}_{\theta}$ and $f^{v}_{\theta}$ are the recurrent functions in the
CTR and CVR layers that take the contextual information $\bm{x}_{q}$333For
each recurrent unit for $i$-th document $d_{i}$, we concatenate the one-hot
embedding vector of its position (i.e., $i$) and its document feature (i.e.,
$\bm{x}_{i}$) as the input. Considering that recurrent network will encode all
the contextual information in query $q$ (i.e., $D_{q}$), we use $\bm{x}_{q}$
for simplicity. as the input, and output $h^{c}_{i}$, $\widetilde{h}^{c}_{i}$
and $h^{v}_{i}$, $\widetilde{h}^{v}_{i}$.

Figure 4. An illustrated example of overall architecture (in the left part)
and recurrent unit of HEROES (in the right part). Left: in the lower layer
(i.e., CTR layer) for engagement objective modeling, we model through the
behavior path “observation$\rightarrow$click”; while in the upper layer (i.e.,
CVR layer) for satisfaction objective modeling, we model through the behavior
path “click$\rightarrow$conversion”. Right: in the upper part, we incorporate
the recurrent unit and the Hawkes process to form a HEROES unit, which can
simultaneously model the inherent relevance and behavioral relevance by mining
the contextual item features and the user’s previous behaviors.

Gate Mechanism. Besides the aforementioned intra- and inter-layer behavior
modelling which are proposed to incorporate the contextual information, we
further introduce a gate mechanism to explicitly discover the hidden
structure. Specifically, the CTR layer is expected to model the contextual
information in each segment following the behavior path
“observation$\rightarrow$click” (i.e., starting at a user observing an item
and ending at the user clicking an item). Similarly, the CVR layer is expected
to model the contextual information in each segment following the behavior
path “click$\rightarrow$conversion” (i.e., starting at a user clicking an item
and ending at the user purchasing an item).

To this end, we define $g_{i}$ as a boundary detector as

(9) $g_{i}=\left\\{\begin{aligned} 1,\ &\ \text{if }P(c_{i}=1)>0.5\\\ 0.\ &\
\text{otherwise}\end{aligned}\right.,$

We illustrate how $g_{i}$ influences the hierarchical structure in Figure 3:
If there is no click at $d_{i}$ (i.e., $g_{i}=0$), then the contextual
information of $d_{i}$ should pass through the behavior path
“observation$\rightarrow$click” in the CTR layer; otherwise (i.e., $g_{i}=1$),
the contextual information of $d_{i}$ should be summarized in the CTR layer
and pass through the behavior path “click$\rightarrow$convsersion” in the CVR
layer.

###  3.2. HEROES Unit

For each item $d_{i}$, HEROES unit models model inherent relevance
$\widetilde{h}_{i}$ and behavioral relevance $h_{i}$ in a recurrent fashion
which incorporate the contexts both within (i.e., intra-layer) and across
(i.e., inter-layer) the CTR and CVR layers.

Intra-Layer Mechanism. Let $\widetilde{s}_{i}$ and $s_{i}$ denote the unit
states in each cell which correspond to $\widetilde{h}_{i}$ and $h_{i}$. To
achieve the gate mechanism in Figure 3, we first recursively compute the
states in the CTR layer as:

(10) $\begin{aligned} \widetilde{s}^{c}_{i}&=\left\\{\begin{aligned}
\widetilde{\text{f}}^{c}_{i}\odot\widetilde{s}^{c}_{i-1}+\widetilde{\text{i}}^{c}_{i}\odot\widetilde{\text{g}}^{c}_{i},\
\langle 1\rangle\\\
\widetilde{\text{i}}^{c}_{i}\odot\widetilde{\text{g}}^{c}_{i},\ \langle
2\rangle\end{aligned}\right.;\ s^{c}_{i}=\left\\{\begin{aligned}
\text{f}^{c}_{i}\odot s^{c}(t_{i-1})+\text{i}^{c}_{i}\odot\text{g}^{c}_{i},\
\langle 1\rangle\\\ \text{i}^{c}_{i}\odot\text{g}^{c}_{i},\ \langle
2\rangle\end{aligned}\right.\end{aligned},$

where $\odot$ denotes element-wise product operation. $\langle 1\rangle\langle
2\rangle$ respectively denote the conditions: $g_{i-1}=0,g_{i-1}=1$. Here,
$\text{f}_{i}$ (i.e., $\text{f}_{i}^{c},\text{f}_{i}^{v}$,
$\widetilde{\text{f}}_{i}^{c}$, $\widetilde{\text{f}}_{i}^{v}$);
$\text{i}_{i}$ (i.e., $\text{i}_{i}^{c},\text{i}_{i}^{v}$,
$\widetilde{\text{i}}^{c}$, $\widetilde{\text{i}}^{v}$); $\text{o}_{i}$ (i.e.,
$\text{o}_{i}^{c},\text{o}_{i}^{v}$
$\widetilde{\text{o}}_{i}^{c},\widetilde{\text{o}}_{i}^{v}$) are forget,
input, output gates, and $\text{g}_{i}$ (i.e.,
$\text{g}_{i}^{c},\text{g}_{i}^{v}$
$\widetilde{\text{g}}_{i}^{c},\widetilde{\text{g}}_{i}^{v}$) is a cell
proposal vector, whose calculations will be later introduced in Eq. (15).

We call the operation under $\langle 1\rangle$ _update_ , and it under
$\langle 2\rangle$ _summarize_. When there is no click at $d_{i-1}$, then
there should the contextual information from $d_{i-1}$ passing to $d_{i}$, and
thus _update_ operation will update the states at $d_{i}$. Otherwise (i.e.,
there is a click at $d_{i-1}$), then _summarize_ operation will summarize the
contextual information in the current segment to update it in the CVR layer
(see Eq. (13)) and reinitialize the state for the next segment.

In _update_ operation, we introduce $s^{c}(t)$ to formulate the Hawkes process
in Eq. (6) as

(11)
$s^{c}(t)=\widetilde{s}^{c}_{i+1}+(s^{c}_{i+1}-\widetilde{s}^{c}_{i+1})\exp{(-\delta_{i+1}(t-t_{i}))}\text{
for }t\in(t_{i},t_{i+1}].$

Here, $\delta_{i+1}=f_{\gamma}(\text{MLP}(y_{i+1}||h^{c}_{i+1}))$ where
$y_{i+1}$ is a multi-hot embedding vector representing the user’s behavior on
item $d_{i+1}$, and $||$ denotes the concatenation operation. If $y_{i+1}$ is
not available in some cases, we can directly apply
$\delta_{i+1}=f_{\gamma}(\text{MLP}(h^{c}_{i+1}))$ instead. Here, we follow
(Mei and Eisner, 2016) to define $f_{\gamma}(x)$ as
$f_{\gamma}(x)\coloneqq\gamma\log(1+\exp{(x/\gamma)})$ where $\gamma$ is set
as 5 in our experiment.

Then, the hidden states $\widetilde{h}^{c}_{i}$ and $h^{c}_{i}$ can calculated
by

(12) $\begin{aligned}
\widetilde{h}^{c}_{i}=\widetilde{\text{o}}^{c}_{i}\odot\text{tanh}(\widetilde{s}^{c}_{i}),\
h^{c}_{i}=\text{o}^{c}_{i}\odot\text{tanh}(s^{c}_{i})\end{aligned}.$

Similar as Eq. (10), the states in the CVR layer are recursively updated
following

(13) $\displaystyle\widetilde{s}^{v}_{i}$
$\displaystyle=\left\\{\begin{aligned}
\widetilde{\text{f}}^{v}_{i}\odot\widetilde{s}^{v}_{i-1}+\widetilde{\text{i}}^{v}_{i}\odot\widetilde{\text{g}}^{v}_{i},\
\langle 3\rangle\\\ s^{v}_{i-1},\ \langle 4\rangle\end{aligned}\right.;\
s^{v}_{i}=\left\\{\begin{aligned} \text{f}^{v}_{i}\odot
s^{v}_{i-1}+\text{i}^{v}_{i}\odot\text{g}^{v}_{i},\ \langle 3\rangle\\\
s^{v}_{i-1},\ \langle 4\rangle\end{aligned}\right.,$

where $\langle 3\rangle\langle 4\rangle$ respectively denote the conditions:
$g_{i}=1,g_{i}=0$. We call the operation under $\langle 3\rangle$ _update_ ,
and it under $\langle 4\rangle$ _copy_ : When there is a click at $d_{i}$,
then there should be the contextual information passed from the CTR layer, and
thus _update_ operation will update the states at $d_{i}$. Otherwise (i.e.,
there is no click at $d_{i}$), then _copy_ operation will simply copy the
states and hidden states of the previous timestep without fusing any
information. The hidden states $\widetilde{h}^{v}_{i}$ and $h^{v}_{i}$ can be
computed via

(14) $\begin{aligned} \widetilde{h}^{v}_{i}=\left\\{\begin{aligned}
\widetilde{\text{o}}^{v}_{i}\odot\text{tanh}(\widetilde{s}^{v}_{i}),\ \langle
3\rangle\\\ \widetilde{h}^{v}_{i-1},\ \langle 4\rangle\\\
\end{aligned}\right.;\ h^{v}_{i}=\left\\{\begin{aligned}
\text{o}^{v}_{i}\odot\text{tanh}(s^{v}_{i}),\ \langle 3\rangle\\\
h^{v}_{i-1},\ \langle 4\rangle\\\ \end{aligned}\right.\end{aligned}.$

Inter-Layer Mechanism. $\text{i}_{i}$, $\text{o}_{i}$, $\text{g}_{i}$ are
designed to encode the top-down contextual formation. Formally, for each item
$d_{i}$, their values are updated as follows:

(15) $\displaystyle\text{f}_{i}$ $\displaystyle=\text{sigmoid
}(\text{MLP}(\text{s}_{i})),\ \text{i}_{i}=\text{sigmoid
}(\text{MLP}(\text{s}_{i})),$ $\displaystyle\text{o}_{i}$
$\displaystyle=\text{sigmoid }(\text{MLP}(\text{s}_{i})),\
\text{g}_{i}=\text{tanh }(\text{MLP}(\text{s}_{i})),$

where $\text{s}_{i}$ is the top-down state, computed as

(16) $\displaystyle\text{s}^{c}_{i}$ $\displaystyle=[(1-g_{i-1})\cdot
U^{c}_{i-1}\cdot h^{c}_{i-1}+g_{i-1}\cdot U^{r}_{i-1}\cdot h^{v}_{i-1}],$
$\displaystyle\text{s}^{v}_{i}$ $\displaystyle=[U^{v}_{i-1}\cdot
h^{v}_{i-1}+g_{i}\cdot W^{r}_{i}\cdot h^{c}_{i}].$

where $U^{c}_{i-1}$, $U^{r}_{i-1}$, $U^{v}_{i-1}$ and $W^{r}_{i}$ are
trainable weights.

Note that the above operations in Eqs. (12), (14), (15), (16) implicitly force
the CVR layer to absorb the summary information from the CTR layer according
to the top-down contexts. Also, these operations are not assigned with a fixed
update intervals, and thus can be adaptively adjusted corresponding to
different contexts.

###  3.3. Loss Function

Considering that both click and conversion signals are binary, we adopt binary
cross entropy (BCE) loss as

(17) $L=L_{c}+\alpha\cdot L_{v}\text{ where }$ (18)
$L_{b}=-\sum_{(b,\bm{x}_{q})\in\mathcal{D}_{q}}\left(b\cdot\log
P(b|\bm{x}_{q})+(1-b)\cdot\log(1-P(b|\bm{x}_{q}))\right),$

where $b$ can denote either click $c$ or conversion $v$.

For (biased) LTR, for each item $d_{i}$, the estimation of its click is
$P(c_{i}=1)=P(r^{c}_{i}=1)=h^{c}_{i}$, and its conversion is
$P(v_{i}=1)=P(r^{v}_{i}=1)\cdot P(c_{i}=1)=h^{c}_{i}\cdot h^{v}_{i}$.

##  4\. HEROES for Unbiased LTR

In this section, we extend the HEROES architecture into unbiased LTR (i.e.,
using Eq. (3)). We first describe how to use the HEROES to model the user’s
multiple behavior through the entire space behavior path, and then present the
corresponding loss function.

###  4.1. Behavior Modelling on the Entire Space

Note that the HEROES introduced in Section 3 that does not explicitly model
the behavior path “observation$\rightarrow$click”. However, as stated in
Section 2, unbiased LTR requires the HEROES to use Eq. (3) and learn the
multiple user behaviors through the entire space behavior path
“observation$\rightarrow$click$\rightarrow$conversion”.

To this end, we introduce the survival analysis technique (Jin et al., 2020;
Ren et al., 2019) to simultaneously estimate the user’s observations, clicks,
conversions. The main assumption of the survival analysis technique is that a
patient will keep _survival_ until she _leaves_ the hospital or meets _death_
, which follows the path “survival$\rightarrow$death”.

Analogously, we can use it to formulate the user’s multiple behaviors, which
are similar but hierarchical: In the CTR layer, a user will keep _observing_
until she _leaves_ due to the lost interest or _clicks_ an item to check its
detailed information, which follows the path “observation $\rightarrow$
click”; where in the CVR layer, a user will keep _clicking_ to search for a
worthwhile item until she _leaves_ due to mismatching between the user
requirement and the current item or _purchases_ an item due to success in
finding a favorite item, which follows the path “click $\rightarrow$
conversion”.

Based on the analysis above, we can define the probability density function
(P.D.F.) of a user behavior occurring at $i$-th item $d_{i}$ as

(19) $\displaystyle P(c_{i}=1)=P(z^{c}=i),\ P(v_{i}=1)=P(z^{v}=i),$

where $z^{c}$ and $z^{v}$ represent the click and conversion behaviors
respectively; and $z=i$ means that the behavior occurs in $d_{i}$ and $z\geq
i$ means that the behavior occurs after $d_{i}$.

From the analogy between “survival $\rightarrow$ death” and “observation
$\rightarrow$ click”, “click $\rightarrow$ conversion”, we can find that when
studying the click behavior $z^{c}$ in the CTR layer, the CDF in this case
(i.e., $P(z^{c}\geq i)$) denotes the observation probability, since a user
will keep browsing until she finds an interesting item and clicks to check
details. Similarly, if studying the conversion behavior $z^{v}$ in the CVR
layer, the CDF here (i.e., $P(z^{v}\geq i)$) denotes the click probability
since a user will keep clicking the items until she eventually purchases a
favored one. Thus, we have

(20) $\displaystyle P(o_{i}=1)=P(z^{c}\geq i),\ P(c_{i}=1)=P(z^{v}\geq i).$

We then can derive the prior-click relevance $r^{c}_{i}$ and the post-click
relevance $r^{v}_{i}$ by the conditional click probability $h^{c}_{i}$ and the
conditional conversion probability $h^{v}_{i}$, which can be formulated as

(21) $h^{c}_{i}\coloneqq
P(r^{c}_{i}=1)=\frac{P(c_{i}=1)}{P(o_{i}=1)}=\frac{P(z^{c}=i)}{P(z^{c}\geq
i)},$ (22) $h^{v}_{i}\coloneqq
P(r^{v}_{i}=1)=\frac{P(v_{i}=1)}{P(c_{i}=1)}=\frac{P(z^{v}=i)}{P(z^{v}\geq
i)},$

which also indicates the probability that the click behavior $z^{c}$ (or
conversion behavior $z^{v}$) lies at $d_{i}$ given the condition that $z^{c}$
(or $z^{v}$) is larger than the last observation (or click) boundary. For each
layer, according to Eqs. (20), (21) and (22), we can derive that

(23) $\displaystyle P(z\geq i|\bm{x}_{q};\theta)=P(z\neq 1,z\neq 2,\dots,z\neq
i-1|\bm{x}_{q};\theta)$ $\displaystyle=$
$\displaystyle\prod_{\tau:\tau<i}[1-P(z=\tau|z\geq\tau,\bm{x}_{q};\theta)]=\prod_{\tau:\tau<i}(1-h_{\tau}).$

Here, we use the probability chain rule to calculate the observation
probability $P(i\leq z^{c})$ (or the click probability $P(i\leq z^{v})$) at
$d_{i}$ through multiplying the conditional unclick probability
$1-h^{c}_{\tau}$ (or the conditional unconversion probability
$1-h^{v}_{\tau}$). Combining Eqs. (21), (22) and (23) together, we have

(24) $P(z=i|\bm{x}_{q};\theta)=h_{i}\cdot\prod_{\tau:\tau<i}(1-h_{\tau}).$

Based on above description, we can express all the user’s behaviors including
observation $P(o_{i}=1)$, click $P(c_{i}=1)$, conversion $P(v_{i}=1)$ on
$d_{i}$ by $h_{i}$.

One straightforward approach to estimate $h_{i}$ is directly applying Eq. (8).
However, it might cause _incompatible issue_ : Eq. (8) uses the Hawkes process
to model the user’s multiple behaviors (i.e., Eq. (6)), while Eq. (21) is
derived from the survival analysis. Therefore, another way is to first tweak
the HEROES unit by replacing $s^{c}(t_{i-1})$ by $s^{c}_{i-1}$ in Eq. (10),
and then employ HEROES (i.e., Eq. (8)) to obtain $h_{i}$. We will later
establish an empirical investigation of the performance of these two
approaches.

###  4.2. Loss Function on the Entire Space

The first type of loss is based on the PDF. In the CTR and CVR layers, we aim
to minimize the negative log-likelihood of the clicked or purchased item
$d_{j}$ (i.e., $z=j$) as

(25) $\displaystyle L_{\mathtt{pdf}}$
$\displaystyle=-\text{log}\prod_{(\bm{x}_{q},z)\in\mathcal{D}_{q}}P(z=j|\bm{x}_{q};\theta)$
$\displaystyle=-\sum_{(\bm{x}_{q},z)\in\mathcal{D}_{q}}[\text{log}\
h_{j}+\sum_{\tau:\tau<j}\text{log}(1-h_{\tau})],$

The second type of loss is based on the CDF. Let $I$ denote the length of the
item list. There are two cases: (i) for those lists where there is a click
occurring in the CTR layer or a conversion occurring in the CVR layer (i.e.,
$z\leq I$), we have

(26) $\displaystyle L_{\mathtt{occur}}$
$\displaystyle=-\text{log}\prod_{(\bm{x}_{q},I)\in\mathcal{D}_{\mathtt{occur}}}P(I\geq
z|\bm{x}_{q};\theta)$
$\displaystyle=-\sum_{(\bm{x}_{q},I)\in\mathcal{D}_{\mathtt{occur}}}\text{log}\
[1-\prod_{\tau:\tau\leq I}(1-h_{\tau})],$

where $\mathcal{D}_{\mathtt{occur}}$ is the dataset of the above lists. For
the other lists, there is no click occurring in the CTR layer or no conversion
occurring in the CVR layer (i.e., $z>I$), we have

(27) $\displaystyle L_{\mathtt{non-occur}}$
$\displaystyle=-\text{log}\prod_{(\bm{x}_{q},I)\in\mathcal{D}_{\mathtt{non-
occur}}}P(z>I|\bm{x}_{q};\theta)$
$\displaystyle=-\sum_{(\bm{x}_{q},I)\in\mathcal{D}_{\mathtt{non-
occur}}}\sum_{\tau:\tau\leq I}\text{log}\ (1-h_{\tau}),$

where $\mathcal{D}_{\mathtt{non-occur}}$ is the dataset of the above lists,
and $\mathcal{D}_{q}=\mathcal{D}_{\mathtt{occur}}\cup\mathcal{D}_{\mathtt{non-
occur}}$.

Combining all the objective functions (i.e., C.D.F. and P.D.F. losses), our
goal is to minimize the negative log-likelihood over all the data samples as

(28) $L_{b}=L_{\mathtt{pdf}}+\beta\cdot L_{\mathtt{cdf}}\text{ where
}L_{\mathtt{cdf}}=L_{\mathtt{non-occur}}+L_{\mathtt{occur}},$

where all these losses are computed over the behavior $z^{b}$ (where $b$ can
either denote $c$ or $v$), and the hyper-parameter $\beta$ balances P.D.F. and
C.D.F. losses at the same level to stabilize the model training. The overall
loss $L$ can be derived by applying Eq. (28) into Eq. (17).

For unbiased LTR, for each item $d_{i}$, according to Eq. (24), the estimation
of its click is
$P(c_{i}=1)=P(z^{c}=i)=h^{c}_{i}\cdot\prod_{\tau:\tau<i}(1-h^{c}_{\tau})$, and
it conversion is
$P(v_{i}=1)=P(z^{v}=i)=h^{v}_{i}\cdot\prod_{\tau:\tau<i}(1-h^{v}_{\tau})$.

###  4.3. Overall Algorithm

We show the overall algorithm in Algorithm 1. It is not hard to see that the
main components of HEROES are the hierarchical recurrent neural network. Let
$I_{\text{max}}$ denote the maximal length of document lists. Then, the
calculation of each HEROES unit $f_{\theta}$ will run for maximal
$2I_{\text{max}}$ times. We assume the average case time performance of
$f_{\theta}$ is $O(C)$. The subsequent calculation is to obtain the
multiplication results of $h_{i}$ or $1-h_{i}$ to form the losses, whose
complexity is $O(I_{\text{max}})$. Then, the overall time complexity is
$O(2CI_{\text{max}})+O(I_{\text{max}})=O(2CI_{\text{max}})$.

Note that we use _behavioral relevance_ $r_{i}$ for training to mine its
latent _inherent relevance_ $\widetilde{r}_{i}$ for inference. Take the HEROES
for biased LTR as an example. During training, we compute the loss in Eq. (18)
using $P(c_{i}=1)=P(r^{c}_{i}=1)=h^{c}_{i}$ for the CTR prediction and
$P(v_{i}=1)=P(c_{i}=1)\cdot P(r^{v}=1)=h^{c}_{i}\cdot h^{v}_{i}$ for the CVR
prediction. During inference, we use
$P(c_{i}=1)=P(\widetilde{r}^{c}_{i}=1)=\widetilde{h}^{c}_{i}$ for the CTR
estimation and $P(v_{i}=1)=P(c_{i}=1)\cdot
P(\widetilde{r}^{v}_{i}=1)=\widehat{h}^{c}_{i}\cdot\widetilde{h}^{v}_{i}$ for
the CVR estimation. Comparing to the existing pipeline (e.g., (Zhao et al.,
2019)) using $h^{c}_{i}$ and $h^{c}_{i}\cdot h^{v}_{i}$ for both training and
inference, the proposed pipeline is able to encode the effect from the user’s
previous behaviors and recover the true inherent relevance.

Algorithm 1 HEROES

1: Initialize all parameters.

2: repeat

3: Randomly sample a batch $\mathcal{B}$ from $\mathcal{D}$.

4: for each item $d_{i}$ with $\mathcal{D}_{q}$ in $\mathcal{B}$ do

5: Calculate $h^{c}_{i}$, $h^{v}_{i}$, $\widetilde{h}^{c}_{i}$,
$\widetilde{h}^{v}_{i}$ using Eq. (8).

6: end for

7: Compute loss $L$ using $h^{c}_{i}$, $h^{v}_{i}$ by Eq. (17). {Training}

8: Update parameters $\theta$ by minimizing $L$.

9: Generate ranking according to $\widetilde{h}^{c}_{i}$,
$\widetilde{h}^{v}_{i}$. {Inference}

10: until convergence

Table 2. Comparison of different multi-task models and sequential models on
three industrial datasets. Results of both Click-Through Rate (CTR) and
Conversion Rate (CVR) are reported. Bold values are the best in each column,
while the second best values are underlined. * indicates $p<0.001$ in
significance tests compared to the best baseline.

Ranker | Task | Criteo | Taobao E-Commerce | Diantao Live Broadcast
---|---|---|---|---
AUC | LogLoss | NDCG | AUC | LogLoss | NDCG | AUC | LogLoss | NDCG
DUPN | CVR | 0.9505 | 0.1137 | 0.7348 | 0.6747 | 0.5194 | 0.6843 | 0.8232 | 0.2345 | 0.7522
CTR | 0.7410 | 0.5863 | 0.7526 | 0.5777 | 0.7215 | 0.4576 | 0.7156 | 0.6032 | 0.7009
ESMM | CVR | 0.8750 | 0.4466 | 0.7194 | 0.6443 | 0.6330 | 0.6490 | 0.7046 | 0.2743 | 0.6697
CTR | 0.6476 | 0.6511 | 0.7460 | 0.5410 | 0.7591 | 0.4166 | 0.6664 | 0.6577 | 0.6601
ESM2 | CVR | 0.8798 | 0.4360 | 0.7235 | 0.6453 | 0.6376 | 0.6471 | 0.7039 | 0.2756 | 0.6688
CTR | 0.6740 | 0.6370 | 0.7496 | 0.5437 | 0.7573 | 0.4170 | 0.6742 | 0.6512 | 0.6608
MMoE | CVR | 0.8817 | 0.4420 | 0.7182 | 0.6537 | 0.6267 | 0.6452 | 0.7283 | 0.2731 | 0.6653
CTR | 0.6779 | 0.6343 | 0.7540 | 0.5410 | 0.7463 | 0.4093 | 0.6770 | 0.6513 | 0.6618
DRSR | CVR | 0.9468 | 0.1366 | 0.7644 | 0.6723 | 0.5156 | 0.6892 | 0.8140 | 0.2546 | 0.7697
CTR | 0.7452 | 0.5837 | 0.7687 | 0.5759 | 0.7171 | 0.4578 | 0.6985 | 0.6103 | 0.7053
RRN | CVR | 0.9564 | 0.1169 | 0.7739 | 0.6732 | 0.5061 | 0.6890 | 0.8156 | 0.2698 | 0.7421
CTR | 0.7496 | 0.5797 | 0.7706 | 0.5766 | 0.7075 | 0.4575 | 0.6926 | 0.6019 | 0.6928
NARM | CVR | 0.9524 | 0.1172 | 0.7644 | 0.6733 | 0.5160 | 0.6893 | 0.8234 | 0.2595 | 0.7612
CTR | 0.7511 | 0.5810 | 0.7724 | 0.5764 | 0.7186 | 0.4576 | 0.7082 | 0.5958 | 0.7012
STAMP | CVR | 0.9406 | 0.1209 | 0.8014 | 0.6668 | 0.5210 | 0.6892 | 0.8467 | 0.2465 | 0.7689
CTR | 0.7391 | 0.5929 | 0.7702 | 0.5748 | 0.7235 | 0.4575 | 0.7123 | 0.5940 | 0.7070
Time-LSTM | CVR | 0.9622 | 0.1132 | 0.7979 | 0.6745 | 0.5169 | 0.6889 | 0.8540 | 0.2412 | 0.7787
CTR | 0.7602 | 0.5703 | 0.7738 | 0.5776 | 0.7192 | 0.4576 | 0.7195 | 0.6040 | 0.7124
LSTM | CVR | 0.8429 | 0.4841 | 0.6629 | 0.6721 | 0.4783 | 0.6885 | 0.7124 | 0.2736 | 0.7475
CTR | 0.6032 | 0.6042 | 0.7503 | 0.5749 | 0.7222 | 0.4493 | 0.6633 | 0.6542 | 0.6792
NHP | CVR | 0.9533 | 0.1127 | 0.7682 | 0.6743 | 0.4914 | 0.6893 | 0.8267 | 0.2535 | 0.7622
CTR | 0.7428 | 0.5816 | 0.7656 | 0.5773 | 0.7214 | 0.4576 | 0.7033 | 0.6042 | 0.7068
$\text{HEROES}^{-}_{\mathtt{intra}}$ | CVR | 0.8801 | 0.4270 | 0.7327 | 0.6917 | 0.5209 | 0.6998 | 0.8045 | 0.2675 | 0.7712
CTR | 0.6764 | 0.6612 | 0.7521 | 0.5483 | 0.7174 | 0.4682 | 0.7091 | 0.5976 | 0.7135
$\text{HEROES}^{-}_{\mathtt{inter}}$ | CVR | 0.9682 | 0.1152 | 0.7832 | 0.6932 | 0.4918 | 0.7082 | 0.8346 | 0.2225 | 0.7883
CTR | 0.7632 | 0.5721 | 0.7882 | 0.5927 | 0.7032 | 0.4721 | 0.7138 | 0.6021 | 0.7123
$\text{HEROES}^{-}_{\mathtt{unit}}$ | CVR | 0.9705 | 0.1016 | 0.8348 | 0.7402 | 0.4366 | 0.7106 | 0.8601 | 0.2350 | 0.7810
CTR | 0.7787 | 0.5483 | 0.7832 | 0.5920 | 0.7084 | 0.4701 | 0.7412 | 0.5942 | 0.7111
HEROES | CVR | 0.9759∗ | 0.0975∗ | 0.8551∗ | 0.7503∗ | 0.3519∗ | 0.7137∗ | 0.8649∗ | 0.2203∗ | 0.7893∗
CTR | 0.7870∗ | 0.5400∗ | 0.7913∗ | 0.5953∗ | 0.7024∗ | 0.4727∗ | 0.7492∗ | 0.5893∗ | 0.7166∗

Table 3. Comparison of unbiased LTR and biased LTR version of HEROES under
click generation model PBM.

Ranker | Task | Taobao E-Commerce (PBM)
---|---|---
AUC | LogLoss | NDCG
Relevance Data (HEROES) | CVR | 0.7503 | 0.3519 | 0.7137
CTR | 0.5953 | 0.7024 | 0.4727
HEROES+ | CVR | 0.7442 | 0.3674 | 0.7064
CTR | 0.5735 | 0.7206 | 0.4567
HEROES${}^{+}_{\mathtt{comb}}$ | CVR | 0.7463 | 0.3638 | 0.7110
CTR | 0.5738 | 0.7202 | 0.4521
Click Data (HEROES) | CVR | 0.7412 | 0.3746 | 0.7024
CTR | 0.5643 | 0.7563 | 0.4284

##  5\. Experiment

###  5.1. Dataset Description and Evaluation Flow

We use three large-scale real-world datasets for the evaluations, where the
first two are public benchmark datasets and the last one is created by our
own:

  * •

Criteo dataset444https://ailab.criteo.com/ressources/ is formed of Criteo live
traffic data in a period of 30 days. It consists of more than 5.5 million
impressions with 2.3 million clicks and 438 thousand conversions. Since the
query signal is not available, following (Ren et al., 2018), we incorporate
the user ID and conversion ID to divide the full dataset into several
sequential data. In this way, we can obtain 2.2 million item lists (i.e.,
queries).

  * •

Taobao E-Commerce
dataset555https://tianchi.aliyun.com/datalab/dataSet.html?dataId=408 is
collected from the traffic logs of Taobao’s recommender system. It contains
the logs of 444 thousand users browsing 85 million items under 1,614 thousand
queries. In these queries, there are sequential user behaviors, including
3,317 thousand click labels and 17 thousand conversion signals.

  * •

Diantao Live Broadcast Recommendation dataset is collected from the user
interaction logs of Diantao App which shares the same database of users and
anchors with Taobao App. It contains more than 44 million logs of 905 thousand
users’ browsing histories over 527 thousand items in 9,305 thousand queries.
Features of the user include age, gender, city, etc., and features of the
document include title, time, etc. In each query, we regard the items whose
playtime more than $7$s as the clicked ones. And we further treat the clicked
items that are liked or commented on by the user as the purchased ones.

For each dataset, we split the users’ sequential historical records by queries
into training/validation/test datasets at a ratio of 6:2:2. For fair
comparison, we do not input the user’s behaviors (i.e., $y$) into
$f_{\gamma}(\cdot)$ in Eq. (11). We train each model with training queries and
evaluate its performance with new queries. In order to evaluate the above
methods, for each method, we choose Area under the ROC Curve (AUC), LogLoss,
Normalized Distributed Cumulative Gain (NDCG) as evaluation measures.
Specifically, we calculate NDCG over the whole ranking list. Namely, we
compute NDCG@K where K is the length of the ranking list.

###  5.2. Experimental Configuration

As we conduct the experiments in terms of the CTR and CVR predictions, the
most related existing baselines are those originally proposed for multi-task
learning, including DUPN (Ni et al., 2018), ESMM (Ma et al., 2018a), ESM2 (Wen
et al., 2020), MMoE (Ma et al., 2018b). We further extend the existing methods
originally designed for the CTR prediction into these multi-task setting by
using two individual models independently learning and predicting for the CTR
and CVR tasks respectively. As one of the main contributions in the paper is
to design a novel sequential unit, thus, we mainly include the following
sequential models here: LSTM (Hochreiter and Schmidhuber, 1997), RRN (Wu et
al., 2017), NARM (Li et al., 2017), NARM (Li et al., 2017), STAMP (Liu et al.,
2018), DRSR (Jin et al., 2020), NHP (Mei and Eisner, 2016).

Figure 5. (a) Performance change of HEROES against the ratio of CVR and CTR
loss weights. (b) Performance change of HEROES against click data with
different amounts of training data. (c) Training/inference time comparisons of
HEROES against the sequential models.

Besides these baselines, we introduce several variants of HEROES as ablations.
More specifically, in order to further investigate the effect from each
component, we design the following three variants:

  * •

$\textbf{HEROES}^{-}_{\mathtt{intra}}$: we adopt a MLP, instead of our
sequential model to model the correlations within each layer in Section 3.1.

  * •

$\textbf{HEROES}^{-}_{\mathtt{inter}}$: we train the CTR and CVR layers
independently without modeling the correlations cross two layers in Section
3.1.

  * •

$\textbf{HEROES}^{-}_{\mathtt{unit}}$: we adopt a standard LSTM unit instead
of one introduced in Section 3.2.

For further evaluation in the different settings, we clarify the use of HEROES
in the context of biased LTR and unbiased LTR as

  * •

HEROES is the HEROES for biased LTR setting, where we follow Section 3.3 to
produce the loss and the CTR and CVR predictions.

  * •

$\textbf{HEROES}^{+}_{\mathtt{comb}}$ is the HEROES with unbiased LTR setting,
where we follow Section 4.2 to produce the loss and the CTR and CVR
predictions.

  * •

$\textbf{HEROES}^{+}$ is a variant of HEROES${}^{+}_{\mathtt{comb}}$ where we
tweak the HEROES unit by replacing $s^{c}(t_{i-1})$ with $s^{c}_{i-1}$ in Eq.
(10).

###  5.3. Performance Comparison of CTR and CVR

Table 2 summarizes the results. The major findings from our experiments are
summarized as follows:

  * •

The performance of HEROES is significantly better than the multi-task learning
methods (including DUPN, ESMM, ESM2, MMoE). One explanation is that although
DUPN uses LSTM to encode the sequential data and MMoE incorporates mixture-of-
expert structure, their performances are still limited by treating the
multiple behaviors with the same time scales.

  * •

HEROES significantly outperforms the baselines (including LSTM, RRN, NARM,
STAMP, Time-LSTM, DRSR, NHP). A potential reason is that although NHP uses the
Hawkes process and Time-LSTM incorporates the time intervals into the LSTM,
they do not consider the correlations among the user’s multiple behaviors.

  * •

Note that generally speaking, the conversions are harder to predict than the
clicks, as the conversion signals are usually much sparser in the real-world
scenarios. However, Table 2 shows the opposite. We provide three possible
explanations as follows. (i) For those multi-task learning models (e.g.,
HEREOS, DUPN), the CVR prediction can not only benefit from the conversion
signals but also can promote the predictions of the click signals. (ii) There
are usually less noises in the conversion signals than the click ones. (iii)
For these three datasets, we observe that purchased items are often located at
the end of the sequences, as users are likely to keep browsing until finding
the favorite items.

In order to deeply analyze the model design and its superiority, we conduct
the following ablation studies on the Criteo dataset.

Effect of Architecture. We investigate the effect of our hierarchical
architecture design by comparing our model to
$\text{HEROES}^{-}_{\mathtt{intra}}$ and $\text{HEROES}^{-}_{\mathtt{inter}}$.
As Table 2 shows, MLP is not capable to encode the sequential patterns which
verifies HEROES using sequential module for modelling intra-layer
correlations. Also, we can see that the correlations across the layers can
also benefit the CTR and CVR predictions, which verifies the necessity of
building a up-down channel across the CTR and CVR layers.

Effect of Unit. In order to specific the performance gain from HEROES unit
design, we introduce HEROES${}^{-}_{\mathtt{unit}}$, which keeps the
hierarchical architecture but uses a standard LSTM unit. Results reported in
Table 2 demonstrates the improvements of the unit.

Effect of Loss Function Weight. In order to study the influence of choosing
different weights for CTR and CVR losses, we assign different values to
$\alpha$ in Eq. (17), where $\alpha=\alpha/1$ represents the ratio of CVR and
CTR weights. Result depicted in Figure 5(a) indicates both of these losses
play an important role for the final performance.

Robustness Analysis. We investigate the robustness of HEROES with different
amounts of training data. We first randomly select a subset of training data
(i.e., 20% - 100%) to generate click data and then use these datasets to train
HEROES model. Figure 5(b) shows that HEROES can still work well even with
limited training data.

Complexity Analysis. We study the time complexity of HEROES against baseline
methods LSTM, RRN, Time-LSTM, NHP which are the sequential models. From Figure
5(c), we observe that during training, NHP and HEROES are the most time-
consuming methods as they incorporate the Hawkes process in the recurrent
networks; while their inference time are comparable to the other methods.

###  5.4. Performance Comparison of Unbiased LTR

In order to verify whether HEROES+ can work in the context of the unbiased
LTR, we follow the click data generation process from (Ai et al., 2018; Hu et
al., 2019; Jin et al., 2020) to introduce the position bias, and conduct the
experiment on Taobao E-Commerce data. First, we train a Rank SVM model using
$1\%$ of the training data with relevance labels. Next, we use the trained
model to create an initial ranked list for each query. Then, we simulate the
user browsing processes and sample clicks from the initial list. Position-
based Model (PBM) (Richardson et al., 2007) simulates the user browsing
behavior based on the assumption that the bias of an item only depends on its
position, which can be formulated as $P(o_{i})=\rho_{i}^{\tau}$, where
$\rho_{i}$ represents position bias at position $i$ and $\tau\in[0,+\infty]$
is a parameter controlling the degree of position bias. The position bias
$\rho_{i}$ is obtained from an eye-tracking experiment in (Joachims et al.,
2005) and the parameter $\tau$ is set as one by default. It also assumes that
a user decides to click a item $d_{i}$ according to the probability
$P(c_{i})=P(o_{i})\cdot P(r_{i})$. Since there is no typical investigation of
the position bias on the behavior path “click$\rightarrow$conversion”, we
simply operate on the conversion signals by assigning $v=0$ for those items
with no click (i.e., $c=0$) and keep the original values for those clicked
items. We regard the initial lists as the relevance data (i.e., unbiased data)
and generated lists as the click data (i.e., biased data). As Table 3 shows,
our unbiased version, denoted as HEROES+, outperforms the biased one, denoted
as HEROES, which shows that HEROES+ can mitigate the position bias. HEROES+
and HEROES${}^{+}_{\mathtt{comb}}$ achieve comparable results, which indicates
that incorporating the Hawkes process into the survival analysis process would
not cause much difference.

Visualization Analysis. Here, we investigate whether the performance gain of
HEROES+ indeed owes to the reduction of position bias. We compare the ranking
list given by the debiased ranker against the initial ranker. Specifically, we
first identify the items at each position given by the initial ranker. Then we
calculate the average positions of the items at each original position after
re-ranking. We also calculate their average positions after re-ranking their
relevance labels, which is regarded as the ground truth. Ideally, the average
positions produced by the debiasing methods should be close to the average
position by relevance labels. We summarize the results in Figure 6(a). The
curve of HEROES+ (in red with “$\small\blacksquare$”) is the closest to the
relevance label curve (in purple with mark ‘$\small\bullet$’), indicating that
the performance enhancement of HEROES+ is indeed due to effective debiasing.

Robustness Analysis. We evaluate the robustness of HEROES+ under different
degrees of position bias. In the above experiments, we only test the
performance of HEROES+ with click data generated from click models with a
given degree of position bias, i.e., $\tau=1$ in Taobao E-Commerce (PBM).
Therefore, we set the hyper-parameters for each click generation model to five
values and examine whether HEROES+ is still equally effective. Figure 6(b)
shows the AUC results as the degree of position bias increases; the results in
terms of other measures follow similar trends.

Figure 6. (a) Average position after re-ranking of the item at each original
position. (b) Performance change of HEROES+ against click data with different
degrees of position bias.

##  6\. Related Work

There are a variety of user behaviors such as browsing, clicking (i.e.,
engagement behaviors), and rating, purchasing (i.e., satisfaction behaviors)
(Zhao et al., 2019). There are mainly two directions of existing works on
behavior awareness and multi-task information systems. One line of the
research (Zhao et al., 2015; Ma et al., 2018a; Wen et al., 2020; Meng et al.,
2020; Wen et al., 2021; Xi et al., 2021) is to investigate the behavior
decomposition to learn and estimate multiple types of the user behaviors over
all the samples. For example, Wen et al. (2020) composes the post-click
behaviors and develops a multi-task learning algorithm to combine these
estimations to compute a final utility score for ranking. Wen et al. (2021)
further incorporates the sequential behavior graph to encode the dependence
among the user’s multiple behaviors. The other direction of the research (Ma
et al., 2018b; Zhao et al., 2019; Wang et al., 2020; Hadash et al., 2018; Ni
et al., 2018; Tang et al., 2020; Ding et al., 2021; Dai et al., 2021; Zhao et
al., 2019; Gao et al., 2019) aims to explicitly learn the task relationship
and design an effective feature sharing technique. For instance, Ma et al.
(2018b) and Zhao et al. (2019) adopts the Mixture-of-Experts model (Jacobs et
al., 1991) to tradeoff between task-specific objectives and inter-task
relationships. There is also a recently emerged direction (Wang et al., 2022;
Zhang et al., 2020) studying the CTR and CVR estimations from a causal
perspective. However, all the above previous literature do not explicitly
capture, or even are not aware of, the natural multi-scale characteristics of
the user’s multiple behaviors. Instead, our work establishes a hierarchical
architecture which can incorporate the contextual information to automatically
find a specific time scale for each behavior path (i.e.,
“observation$\rightarrow$click”, “click$\rightarrow$conversion”).

##  7\. Conclusion and Future Work

In this paper, we propose a paradigm named HEROES, which can automatically
discover the user’s multi-scale browsing patterns to model the user’s
engagement and satisfaction behaviors. We design a novel recurrent unit to
consider both excitation and discouragement from the contexts. We further show
that HEROES learn over the entire space behavior path
“observation$\rightarrow$click$\rightarrow$conversion”, and can be extended to
unbiased LTR task. It would be interesting to investigate modelling more
complex user behaviors (e.g., adding the items into the wish list or cart) in
future work.

Acknowledgments. This work was supported by Alibaba Group through Alibaba
Research Intern Program. The Shanghai Jiao Tong University Team is supported
by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102)
and National Natural Science Foundation of China (62076161, 62177033). We
would also like to thank Wu Wen Jun Honorary Doctoral Scholarship from AI
Institute, Shanghai Jiao Tong University.

## References

  * (1)
  * Ai et al. (2018) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018.  Unbiased Learning to Rank with Unbiased Propensity Estimation.  _SIGIR_ (2018).
  * Caruana (1997) Rich Caruana. 1997\.  Multitask learning.  _Machine learning_ 28, 1 (1997), 41–75.
  * Chung et al. (2016) Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2016\.  Hierarchical multiscale recurrent neural networks. In _ICLR_.
  * Dai et al. (2021) Xin Dai, Xiangnan Kong, Tian Guo, and Xinlu He. 2021\.  FiShNet: Fine-Grained Filter Sharing for Resource-Efficient Multi-Task Learning. In _CIKM_.
  * Ding et al. (2021) Ke Ding, Xin Dong, Yong He, Lei Cheng, Chilin Fu, Zhaoxin Huan, Hai Li, Tan Yan, Liang Zhang, Xiaolu Zhang, et al. 2021\.  MSSM: a multiple-level sparse sharing model for efficient multi-task learning. In _SIGIR_.
  * Embrechts et al. (2011) Paul Embrechts, Thomas Liniger, and Lu Lin. 2011.  Multivariate Hawkes processes: an application to financial data.  _Journal of Applied Probability_ 48, A (2011), 367–378.
  * Gao et al. (2019) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin. 2019\.  Neural multi-task recommendation from multi-behavior data. In _ICDE_.
  * Hadash et al. (2018) Guy Hadash, Oren Sar Shalom, and Rita Osadchy. 2018\.  Rank and rate: multi-task learning for recommender systems. In _Recsys_.
  * Hawkes (1971) Alan G Hawkes. 1971\.  Spectra of some self-exciting and mutually exciting point processes.  _Biometrika_ 58, 1 (1971), 83–90.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hu et al. (2019) Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019\.  Unbiased LambdaMART: An unbiased pairwise learning-to-rank algorithm. In _WWW_.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991.  Adaptive mixtures of local experts.  _Neural computation_ 3, 1 (1991), 79–87.
  * Jin et al. (2020) Jiarui Jin, Yuchen Fang, Weinan Zhang, Kan Ren, Guorui Zhou, Jian Xu, Yong Yu, Jun Wang, Xiaoqiang Zhu, and Kun Gai. 2020\.  A Deep Recurrent Survival Model for Unbiased Ranking. In _SIGIR_.
  * Joachims et al. (2005) Thorsten Joachims, Laura A Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005.  Accurately interpreting clickthrough data as implicit feedback. In _SIGIR_.
  * Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017\.  Unbiased learning-to-rank with biased feedback. In _WSDM_.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural attentive session-based recommendation. In _CIKM_.
  * Liu et al. (2018) Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018\.  STAMP: short-term attention/memory priority model for session-based recommendation. In _KDD_.
  * Ma et al. (2018b) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018b.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In _KDD_.
  * Ma et al. (2018a) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018a.  Entire space multi-task model: An effective approach for estimating post-click conversion rate. In _SIGIR_.
  * Mei and Eisner (2016) Hongyuan Mei and Jason Eisner. 2016.  The neural hawkes process: A neurally self-modulating multivariate point process.  _arXiv preprint arXiv:1612.09328_ (2016).
  * Meng et al. (2020) Wenjing Meng, Deqing Yang, and Yanghua Xiao. 2020\.  Incorporating user micro-behaviors and item knowledge into multi-task learning for session-based recommendation. In _SIGIR_.
  * Ni et al. (2018) Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. 2018.  Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. In _KDD_.
  * Ren et al. (2018) Kan Ren, Yuchen Fang, Weinan Zhang, Shuhao Liu, Jiajun Li, Ya Zhang, Yong Yu, and Jun Wang. 2018\.  Learning multi-touch conversion attribution with dual-attention mechanisms for online advertising. In _CIKM_.
  * Ren et al. (2019) Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. 2019.  Deep Recurrent Survival Analysis. In _AAAI_.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _WWW_.
  * Tang et al. (2020) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020\.  Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In _Fourteenth ACM Conference on Recommender Systems_.
  * Wang et al. (2022) Hao Wang, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. 2022\.  ESCM2: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation.  _SIGIR_ (2022).
  * Wang et al. (2020) Menghan Wang, Yujie Lin, Guli Lin, Keping Yang, and Xiao-ming Wu. 2020.  M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systems. In _KDD_.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _SIGIR_.
  * Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018\.  Position bias estimation for unbiased learning to rank in personal search. In _WSDM_.
  * Wen et al. (2021) Hong Wen, Jing Zhang, Fuyu Lv, Wentian Bao, Tianyi Wang, and Zulong Chen. 2021\.  Hierarchically Modeling Micro and Macro Behaviors via Multi-Task Learning for Conversion Rate Prediction.  _SIGIR_ (2021).
  * Wen et al. (2020) Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. 2020.  Entire Space Multi-Task Modeling via Post-Click Behavior Decomposition for Conversion Rate Prediction.  _SIGIR_.
  * Wu et al. (2017) Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing. 2017.  Recurrent recommender networks. In _WSDM_.
  * Xi et al. (2021) Dongbo Xi, Zhen Chen, Peng Yan, Yinger Zhang, Yongchun Zhu, Fuzhen Zhuang, and Yu Chen. 2021.  Modeling the Sequential Dependence among Audience Multi-step Conversions with Multi-task Learning in Targeted Display Advertising.  _KDD_ (2021).
  * Zhang et al. (2020) Wenhao Zhang, Wentian Bao, Xiao-Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. 2020.  Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In _WWW_.
  * Zhao et al. (2015) Zhe Zhao, Zhiyuan Cheng, Lichan Hong, and Ed H Chi. 2015\.  Improving user topic interest profiles by behavior factorization. In _WWW_.
  * Zhao et al. (2019) Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019.  Recommending what video to watch next: a multitask ranking system. In _RecSys_.