# Whole Page Unbiased Learning to Rank

Haitao Mao<EMAIL_ADDRESS>Michigan State UniversityUnited States ,  Lixin
Zou<EMAIL_ADDRESS>Baidu Inc.China ,  Yujia Zheng<EMAIL_ADDRESS>Carnegie Mellon UniversityUnited States ,  Jiliang Tang<EMAIL_ADDRESS>Michigan State UniversityUnited States ,  Xiaokai Chu<EMAIL_ADDRESS>Baidu Inc.China ,  Jiashu Zhao<EMAIL_ADDRESS>Wilfrid Laurier UniversityChina
and  Dawei Yin<EMAIL_ADDRESS>Baidu Inc.China

(2018)

###### Abstract.

The page presentation biases in the information retrieval system, especially
on the click behavior, is a well-known challenge that hinders improving
ranking models’ performance with implicit user feedback. Unbiased Learning to
Rank (ULTR) algorithms are then proposed to learn an unbiased ranking model
with biased click data. However, most existing algorithms are specifically
designed to mitigate position-related bias, e.g., trust bias, without
considering biases induced by other features in search result page
presentation(SERP). For example, the multimedia type may generate attractive
bias. Unfortunately, those biases widely exist in industrial systems and may
lead to an unsatisfactory search experience. Therefore, we introduce a new
problem, i.e., whole-page Unbiased Learning to Rank(WP-ULTR), aiming to handle
biases induced by whole-page SERP features simultaneously. It presents
tremendous challenges. For example, a suitable user behavior model (user
behavior hypothesis) can be hard to find; and complex biases cannot be handled
by existing algorithms. To address the above challenges, we propose a Bias
Agnostic whole-page unbiased Learning to rank algorithm, BAL, to automatically
discover and mitigate the biases from multiple SERP features with no specific
design. Experimental results on a real-world dataset verify the effectiveness
of the BAL.

Information Retrieval, Unbiased Learning to Rank, Deep Learning

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Social
networks††ccs: Information systems Learning to rank

##  1\. Introduction

A key component in modern information retrieval systems is the ranking model
that aims to provide highly relevant documents given the particular user
query. It is of great practical value in multiple scenarios, e.g., e-commerce
and the web search engine. Ideally, the ranking model should be learned with
experts’ annotated relevance labels, which, unfortunately, is both expensive
and labour-intensive. A more affordable alternative is to utilize the implicit
biased user feedback, i.e., click, as training labels, which are easy to
collect and more suitable to meet the large data requirement in deep learning.
However, the click is affected by biases induced by search result presentation
features (SERP) from the whole page, such as trust bias (Agarwal et al., 2019)
induced by position, attractive bias (Yue et al., 2010) induced by abstract,
and vertical bias (Chen et al., 2012; Lagun and Agichtein, 2014; Liu et al.,
2015) induced by multimedia types. To mitigate biases in implicit user
feedback, Unbiased Learning to Rank (ULTR) (Joachims et al., 2017b; Wang et
al., 2016; Wang et al., 2018) algorithms have been introduced. Generally, most
existing ULTR algorithms often consist of two procedures: (1) User behavior
model design (i.e., the user behavior assumption (Chuklin et al., 2013;
Richardson et al., 2007; Mao et al., 2018)) aims to model user click behavior
with SERP features and relevance score, such as the trust position-based model
(Agarwal et al., 2019), and the mobile user behavior model (Mao et al., 2018).
Unfortunately, these user behavior models have been proposed with predefined
assumptions from human knowledge, which are designed for position-related bias
and inapplicable to biases from more complex SERP features. (2) Unbiased
learning focuses on mitigating biases and learning toward an unbiased ranking
model under the defined user behavior model, e.g., the IPW (Rosenbaum and
Rubin, 1983) method. However, existing ULTR algorithms only consider position-
related biases. Such position-based Unbiased Learning to Rank (PB-ULTR) is
insufficient for the modern search engine that often presents more biases on
multiple SERP features. Consequently, unsatisfactory results have been
observed on the real-world dataset (Zou et al., 2022), where PB-ULTR
solutions, including IPW (Joachims et al., 2017b), PairD (Hu et al., 2019a),
and REM (Wang et al., 2018), perform no better than that trained on biased
click data directly.

The challenges mentioned above motivate us to introduce a new research problem
in this work, i.e., whole-page Unbiased Learning to Rank (WP-ULTR). It aims to
simultaneously mitigate all the biases introduced by the features on the whole
search page presentation, such as position, SERP height, and multi-media type.
It is a non-trivial problem. Tremendous challenges exist in user behavior
model design and unbiased learning. For the user behavior model design step,
the heuristic user behavior hypothesis in PB-ULTR is not applicable for the
following reasons: (1) Existing user behavior hypotheses have been designed
specifically for the position, which cannot be directly extended to other SERP
features. (2) Designing the user behavior model under the effects of multiple
SERP features is non-trivial. For example, the web with video usually has a
larger height, often leading to more clicks. However, such bias has not been
touched by existing literature. (3) An universal user behavior hypothesis,
e.g., the examination hypothesis (Richardson et al., 2007), might not exist in
the WP-ULTR. Different scenarios may have distinct SERP features, and each
scenario typically requires a specific user behavior model. For example, a
vertical search engine like Linkedin can have a substantially different whole-
page presentation from a general search engine like Google. Therefore, it is
desired to design an algorithm that can automatically identify the complicated
relationships in the user behavior model instead of heuristic designs. For the
unbiased learning step, the major challenge attributes to the complexity of
the user behavior model. Existing unbiased learning algorithms have usually
been designed for the specific and simple user behavior model on the position.
However, biases of WP-ULTR are much more complicated and beyond the scope of
existing PB-ULTR algorithms. For example, existing algorithms do not consider
the bias caused by confounders (i.e., a variable that causes spurious
association between two target variables by influencing both of them) between
click and the SERP features. In our WP-ULTR scenario, the relevance between
query and document can be the confounder that introduces spurious association
between the click and SERP features (e.g., SERP height, ranking position) by
influencing both of them. Correspondingly, ignoring such bias may lead to
inaccurate bias estimation and unsatisfying performance. Therefore, we need to
design a more general unbiased learning algorithm suitable for general user
behavior models.

In this work, we propose BAL, a Bias Agnostic whole page unbiased Learning to
rank algorithm. It can automatically discover and mitigate biases induced by
whole-page SERP features. For the user behavior model design step, we replace
the heuristic design with the causal discovery algorithm. It can automatically
design the user behavior model by discovering the causal relationship among
click, true relevance, and whole-page SERP features with a certain guarantee.
Therefore, it avoids the difficulties to design the user behavior model with
heuristic hypotheses which could be both inaccurate and labor-intensive. For
the unbiased learning step, we propose a more general unbiased learning
algorithm that is flexible to numerous scenarios. We first propose an
Importance Reweighting (Liu and Tao, 2015) based algorithm to remove the
confounding bias. Then we further mitigate the bias induced by SERP features
by blocking the gradient related to SERP features and only learning the effect
from the relevance score to click directly. The advantages over the existing
algorithms are two-fold. First, it can mitigate the relevance score’s
confounding bias on click and SERP features, i.e., the inaccurate bias
estimation. Second, it can be easily extended to mitigate biases from multiple
SERP features rather than only position. Remarkably, though the BAL does not
rely on any predefined click behavior hypothesis, BAL is still explainable
since it generates the causal graph to indicate how the click is biased on
SERP features. In summary, the main contributions of our work are as follows:

  * •

We introduce a novel problem whole-page Unbiased Learning to Rank (WP-ULTR)
that considers biases induced by SERP features other than position.

  * •

We propose a bias-agnostic whole-page ULTR algorithm BAL that can discover the
user behavior model automatically and mitigate biases from multiple SERP
features simultaneously.

  * •

Extensive experiments on a large-scale real-world dataset indicate the
effectiveness of our algorithm.

Figure 1.  An overall procedure illustration of the BAL algorithm. The biased
observation inputs include the query-document relevance score
$\hat{\mathbf{r}}$, the click $\mathbf{c}$, and SERP features $\mathbf{x}$.
BAL is then applied in two steps: (1) User behavior model design step learns a
causal graph with causal discovery; (2) Unbiased learning steps then mitigate
biases found in the causal graph towards an unbiased ranking model.

##  2\. Preliminary

This section describes the user behavior model and the problem of whole-page
unbiased learning to rank.

###  2.1. User Behavior Model

#### Problem definition

The user behavior model aims to formally model how user feedback is affected
by SERP features and the relevance between the query and its corresponding
document. It provides the potential to mitigate the negative bias induced by
SERP features while preserving the query-document relevance. Suppose that we
have a collection of queries $\mathcal{Q}$, where each query
$\mathbf{q}\in\mathcal{Q}$ is relevant to a number of documents
$\mathcal{D}_{\mathbf{q}}=\left\\{\mathbf{d}_{i}\right\\}_{i=1}^{N}$ retrieved
from all indexed documents $\mathcal{D}$. The relevance between the query and
the corresponding document is usually determined by a ranking model
$\hat{\mathbf{r}}=f(\mathbf{q},\mathbf{d};\Theta)$, where $\Theta$ is the
model parameters. There are also whole-page SERP features
$\mathbf{x}=\left\\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\right\\}$ related to
the document $d$, e.g., visual appearances, multi-media types, and positions.
In the following, we introduce how to utilize the causal graph to describe the
user behavior model. The advantages of adopting the causal graph are as
follows: (1) The causal graph provides a natural way to denote the complex
relationships among click, relevance score, and SERP features; and (2) Causal
discovery techniques, instead of the traditional heuristically designed user
behavior model, can be employed to discover the causal graph automatically.

#### Causal graph

The causal graph $\mathcal{G}=(\mathbf{V},\mathbf{E})$ is utilized to describe
the user behavior model. $\mathbf{V}$ is a set of nodes where each node
corresponds to a variable. In WP-ULTR, variables include those corresponding
to SERP features $x=\\{x_{1},x_{2},\cdots,x_{n}\\}$, user click $c$, and the
query-document relevance score $\hat{r}$ generated by the ranking model.
$\mathbf{E}$ is a set of directed edges where each edge denotes the causal
relationship between cause (starting node) and effect (ending node).
Confounding bias is a general bias due to a systematic distortion, often
overlooked without a causal view. It means that two variables are dependent on
a hidden variable, i.e., a confounding variable. The bias then appears as a
spurious correlation between those two variables. Not surprisingly, there
exists confounding bias in WP-ULTR. An example is illustrated in Fig. 2(a)(1).
The click $c$ and the position $p$ are both dependent on the relevance score
$\hat{r}$. Without explicitly considering these hidden causal relations, the
spurious correlation between position and click could greatly bias ranking
models. Unfortunately, most of the current developments have not considered
this confounding bias. To mitigate such confounding bias, we block the
backdoor path from relevance score $\hat{r}$ to the position as shown in Fig.
2(a)(2). Details can be found in Section 3.

###  2.2. Whole-Page Unbiased Learning to Rank

#### Learning to Rank

The task of a ranking model $\hat{\mathbf{r}}=f(\mathbf{q},\mathbf{d};\Theta)$
is to rank the document with more relevance to the top of the ranking list.
With the estimated relevance score $\hat{\mathbf{r}}$, a ranking list
$\pi_{\mathbf{f},\mathbf{q}}$ is generated by descendingly sorting documents
$\mathcal{D}_{\mathbf{q}}$ according to $\hat{\mathbf{r}}$. Therefore, the
goal of learning a rank model is to maximize the order-consistent evaluation
metric i.e. $\vartheta$ (e.g., DCG (Järvelin and Kekäläinen, 2017), PNR (Zou
et al., 2021), and ERR (Chapelle and Chang, 2011)) as

(1)
$\displaystyle\mathbf{f}^{\ast}=\max_{\mathbf{f}}\mathbb{E}_{\mathbf{q}\in\mathcal{Q}}\vartheta(\mathcal{Y}_{\mathbf{q}},\pi_{\mathbf{f},\mathbf{q}}).$

where
$\mathcal{Y}_{\mathbf{q}}=\\{\mathbf{y}_{\mathbf{d}}\\}_{\mathbf{d}\in\mathcal{D}_{\mathbf{q}}}$
is a set of annotated relevance labels $y_{d}$ corresponding to
$\mathbf{q},\mathbf{d}$. Usually, $\mathbf{y}_{\mathbf{d}}$ is the graded
relevance in 0-4 ratings, which indicates the relevance of document
$\mathbf{d}_{i}$ as $\\{$bad, fair, good, excellent, perfect$\\}$,
respectively. To learn the scoring function $\mathbf{f}$, a loss function is
to approximate $\mathbf{y}_{\mathbf{d}}$ with
$\mathbf{f}(\mathbf{q},\mathbf{d})$ as

(2)
$\ell(\mathbf{f}_{ideal})=\mathbb{E}_{\mathbf{q}\in\mathcal{Q}}\left[\sum_{\mathbf{d}\in\mathcal{D}_{\mathbf{q}}}\Delta(\mathbf{f}(\mathbf{q},\mathbf{d}),\mathbf{y}_{\mathbf{d}})\right],$

where $\Delta$ is a function that computes the individual loss for each
document. However, the above loss is impractical since the annotated relevance
labels acquired from expert judgment are expensive.

#### Whole-Page Unbiased Learning to Rank

Unbiased Learning to Rank is proposed as an alternative and intuitive approach
to utilize the user’s implicit feedback as the training signal, which is much
easier to obtain. For example, by replacing the relevance label $y_{d}$ with
click label $c_{d}$ in Eq. 2, a naive empirical ranking loss is derived as
follows

(3)
$\displaystyle\ell_{naive}(\mathbf{f})=\frac{1}{|\mathcal{Q}_{o}|}\sum_{\mathbf{q}\in\mathcal{Q}_{o}}\left[\sum_{\mathbf{d}\in\mathcal{D}_{\mathbf{q}}}\Delta\left(\mathbf{f}(\mathbf{q},\mathbf{d}),\mathbf{c}_{\mathbf{d}}\right)\right],$

where $\mathcal{Q}_{o}$ is the training query set. $\mathbf{c}_{\mathbf{d}}$,
indicating whether the document $\mathbf{d}$ in the ranked list is clicked, is
utilized as the training signal. However, the above loss function is biased on
the whole-page SERP features
$\mathbf{x}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}\\}$ as
described in the user behavior model. To address this issue, we propose whole-
page Unbiased Learning to rank (WP-ULTR). It aims to find the suitable loss
function to remove the effect of data bias induced by multiple SERP features
$\mathbf{x}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}\\}$.
Remarkably, $\mathbf{x}$ can be with arbitrary data types, e.g., categorical,
continuous, and ordinal features.

##  3\. Bias Agnostic Learning Algorithm

In this section, we first provide an overview of our BAL algorithm shown in
Figure 1. It consists of two major steps: the user behavior model design step
and the unbiased learning step. The former step is to find the complicated
relationship among click $\mathbf{c}$, relevance score $\hat{\mathbf{r}}$, and
SERP features
$\mathbf{x}=\left\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}\right\\}$
by using the logged user behavior data. Specifically, we describe the user
behavior model with a causal graph generated by the causal discovery
technique. In Figure 1, we present an example of the user behavior model in
this step, which considers the bias effect of SERP features, including
multimedia type, SERP height, and ranking position. Moreover, relevance score
$\hat{r}$ shows a typical confounding effect on $p$ and $c$.

The unbiased learning step mitigates the bias effect in two sub-steps: user
behavior model modification and ranking model learning. In the former sub-
step, we mitigate the bias induced by confounding factor $\hat{r}$ by removing
the backdoor path from $\hat{r}$ to $p$ with sampling reweighting. In the
ranking model learning sub-step, we learn an unbiased ranking model with only
the effect of relevance score to click by blocking the gradient from other
SERP features. We provide more details on these key components of BAL in the
following subsections.

(a) The confounder case1 where $\hat{r}$ is the only parent node for $p$. $p$
is the SERP feature related to click $c$

(b) The confounder case2 where both $\hat{r}$ and other SERP feature $m$ are
parent nodes for $p$. $p$ is the SERP feature related to the click $c$.

Figure 2.  Two representative cases on how the relevance score $\hat{r}$
reveals the confounding bias. The unbiased learning algorithm first removes
the confounding bias then learns the click effect on the relevance score.
$\hat{r}$, $r$, $c$, $p$, and $m$ correspond to the relevance score, true
relevance, click, position, and multimedia types, respectively.

###  3.1. User Behavior Model Design

In this subsection, we focus on utilising the causal discovery algorithm to
automatically find the user behavior model with the biased observation data.
Then we can identify biases in the user behavior model and estimate the
influence score on the SERP feature for further measuring and mitigating
biases in the unbiased learning steps.

#### SERP feature preprocessing

The WP-ULTR problem has more SERP features with multiple data types, which PB-
ULTR algorithms cannot handle. In general, there are three SERP feature types:
continuous, ordinal, and categorical. Since deep learning is more suitable for
continuous features, we convert ordinal and categorical features into
continuous features. For the ordinal feature like the ranking position
$\mathbf{p}$, we transform it into a continuous one with a Bradly-Terry model
(Bradley and Terry, 1952; Marden, 1996; Wu et al., 2014; Wu et al., 2018) as
$p_{i,j}=\frac{e^{s_{i}}}{e^{s_{i}}+e^{s_{j}}}$, where $p_{i,j}$ is the
probability that the ranking position $i$ is lower than the position $j$. $s$
is the transformed continuous score, where a larger score represents a higher
rank. Then we can obtain the continuous score $s$ by maximizing the likelihood
optimization objective shown below. With all the position pairs
$(i,j)\in\mathbf{\varpi}$ satisfying position $i$ lower than position $j$, we
maximize the log-likelihood of $\mathbf{\varpi}$ as
$\sum\limits_{(i,j)\in\mathcal{\varpi}}\log{p_{i,j}}$. For example, the score
$s$ on position $\mathbf{p}=2$ should be larger than any score with a higher
position $\mathbf{p}>2$.

For the categorical features, e.g., multimedia type, we utilize the embedding
$\mathbf{E}=\\{\mathbf{e}_{1},\mathbf{e}_{2},\cdots,\mathbf{e}_{n}\\}\in\mathbb{R}^{n\times
d}$ to transform the discrete one into the learnable continuous embedding.

#### Causal discovery

The fundamental challenge in WP-ULTR is that multiple SERP features are newly
introduced. It adds more difficulties to heuristically designing a suitable
user behavior model. To address this challenge, we adopt the causal discovery
technique, which can automatically find the user behavior model with the
training data. It identifies the causal relationship among the click
$\mathbf{c}$, whole-page presentation features
$\mathbf{x}=\left\\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\right\\}$, and the
relevance score $\hat{\mathbf{r}}$ generated by the biased ranking model.
Specifically, we apply the PC algorithm (Spirtes et al., 2000) with Kernel-
based Conditional Independence test (KCI) (Zhang et al., 2012a) to discover
the causal graph with mixed-type data. KCI is a kernel method to provide non-
parametric tests for conditional independence, which has been verified to
apply to general distributions without limitations on functional relations and
data types (Zhang et al., 2012a). Together with PC, we can recover the hidden
structure in the context of real-world learning-to-rank. Moreover, some prior
knowledge in ULTR is predefined on the causal graph as follows: (1) relevance
score $\hat{r}$ must be the parent of click $c$ since users only click the
document when it is relevant to the query. (2) click $c$ cannot be the parent
of SERP features $x$ since users click the document after they observe the
SERP features. (3) SERP features $x$ cannot be the parent of relevance score
$\hat{r}$. The SERP features are designed to optimize the ranking metric, for
example, DCG (Järvelin and Kekäläinen, 2017), after we estimate the relevance
of all candidate documents. Therefore, the SERP features are determined by the
relevance score to some extent, and SERP features are not the parent of the
relevance score.

The causal graph $\hat{\mathcal{G}}$ generated by the causal discovery
algorithm is utilized to describe the user behavior model. It can help
identify biases in user behaviors easily. Based on the aforementioned prior
knowledge, there are two types of potential bias as follows: (1) Confounding
bias: the relevance score $\hat{r}$ is connected with both the SERP feature
$x_{i}$ and the click $c$. Meanwhile, the SERP feature $x_{i}$ is connected
with click $c$. Then the spurious correlation exists from $x_{i}$ to $c_{i}$,
which can lead to the overestimation on the relationship between $x_{i}$ and
$c_{i}$. (2) SERP bias: both SERP features $x_{i}$ and the relevance score
$\hat{r}$ are connected with the click $c$. Then if we learn directly from the
click with the loss in Eq. (3), the model will learn the effects from both the
SERP feature $x_{i}$ and the relevance score $\hat{r}$ to click. However, the
ranking model should only consider the relevance between the query and the
document with no effect from the SERP feature $x_{i}$.

#### Influence score estimation on the SERP feature

After the causal discovery sub-step, we can identify where biases are from.
However, how those biases affect the click behavior for each data instance is
still unclear. To achieve this goal, it is necessary to first estimate the
influence on the SERP feature. The influence score estimation on the SERP
feature is defined as $p(x_{i}|\pi_{x_{i}})$, the conditional probability of
the SERP feature $x_{i}$ given its parent variables $\pi_{x_{i}}$ including
the relevance score and other SERP features connected in the causal graph.
Intuitively speaking, the influence score estimates the sample influence on
the confounding bias, i.e., the change of user behavior model whether the
confounding bias is mitigated or not.

Multiple layer perceptron (MLP) is utilized as the probability estimator. It
is expressive to deal with most parametric families of conditional probability
distributions $P(x_{i}|\pi_{x_{i}})$.

For each SERP feature $\mathbf{x}_{i}$, we learn an MLP parameterized with
$\phi_{i}=\\{\mathbf{W}_{i}^{1},\mathbf{W}_{i}^{2},\cdots,\mathbf{W}_{i}^{L}\\}$,
where $L$ is the number of layers in MLP. We first concatenate click
$\mathbf{c}$, relevance score $\hat{\mathbf{r}}$, and SERP features
$\mathbf{x}$ as the input data,
$\mathbf{Z}=\left[\mathbf{c},\hat{\mathbf{r}},\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\right]\in\mathbb{R}^{d}$.
Then we preprocess each SERP feature $\mathbf{x}_{i}$ to the corresponding
masked input $\mathbf{Z}_{\pi_{x_{i}}}$ by only keeping features corresponding
to the parent nodes of $\mathbf{x}_{i}$ while masking others. Then, we forward
the masked input to the corresponding network as follows:

(4) $\mathbf{H}_{i}=\mathbf{W}_{i}^{L}f(\cdots
f(\mathbf{W}_{i}^{2}f(\mathbf{W}_{i}^{1}\mathbf{Z}_{\phi_{i}})\cdots)$

where $\mathbf{H}_{i}\in\mathbb{R}^{m}$ with $m$ parameters which describe the
desired distribution. For example, if the desired distribution is a Gaussian
distribution, then $m=2$ corresponds to the mean and standard deviation,
respectively.

Finally, the maximum likelihood optimization objective is as follows:

(5) $\max_{\phi}\mathbb{E}_{X\sim P_{X}}\sum_{i=1}^{|\mathcal{G}|}\log
p_{i}\left(\mathbf{X}_{i}\mid\mathbf{X}_{\pi_{i}};\phi_{i}\right)$

Remarkably, the maximum likelihood objective may be different for different
types of SERP features. In this work, we use polynomial distribution for
categorical features and Gaussian distributions for continuous ones,
respectively.

###  3.2. Unbiased learning

The goal of the unbiased learning step is to mitigate biases and learn toward
an unbiased ranking model. In this work, we propose following two sub-steps to
mitigate the biases, i.e., the confounding bias and SERP bias, discussed in
Section 3.1: (1) Unbiased user behavior model modification sub-step aims to
diminish the influence of confounding bias. It modifies the user behavior
model for removing the backdoor path the relevance score to the SERP features
with importance reweighting. (2) Ranking model learning sub-step aims to
mitigate the SERP bias and learning towards a ranking model based on query-
document relevance with no influence by SERP features. To achieve this goal,
we blocks the gradients related to SERP features to avoid learning SERP
features’ affect on click. In the following, we first introduce the backbone
model of WP-ULTR and then present the detail of the above two sub-steps.

#### Ranking model backbone

The large-scale language model BERT architecture(Devlin et al., 2018) is
utilized as the default backbone. Query, document title, and document abstract
are concatenated with [SEP] separator as the input. The ranking model is first
warmed up by pretraining with both Masked Language Model (MLM) loss (Devlin et
al., 2018), and the naive loss in Eq. (3) with no ULTR techniques. Remarkably,
the pre-trained ranking model is biased on SERP features. Then, our BAL
algorithm is proposed to discover and mitigate biases in the pre-trained
ranking model to train toward the unbiased one.

#### Unbiased user behavior model modification

This subsection focuses on the confounding bias induced by the relevance score
on the SERP feature and the click. As shown in Figure 2(a)(1), relevance score
$\hat{\mathbf{r}}$ shows influence on both the position $\mathbf{p}$ and click
$\mathbf{c}$. This bias induced by the confounder $\hat{\mathbf{r}}$ results
in the spurious correlation between the position $\mathbf{p}$ and click
$\mathbf{c}$. This spurious correlation leads to the SERP feature’s over-
estimated influence on the click and relevance’s under-estimated influence on
the click. Ideally, the causal graph structure without confounding effect is
demonstrated in Figure 2(a)(2) where relevance $\hat{\mathbf{r}}$ has no
relationship with the position $\mathbf{p}$. With this user behavior model, we
can further accurately estimate the relevance’s effect on the click and
achieve the unbiased learning.

To remove the backdoor path from the relevance score to the SERP feature, we
create a pseudo population following the unconfounded causal graph through
importance reweighting (Liu and Tao, 2015). Specifically, we explain two
typical scenarios in Figure 2(a) and Figure 2(b). For the formal case, we
identify the suitable reweighting score by making a comparison between the
data distribution of the original causal graph $\hat{\mathcal{G}}$ in Figure
2(a)(1) and the expected causal graph $\mathcal{G}$ in Figure 2(a)(2). The
data distribution for the original causal graph $\hat{\mathcal{G}}$ is:

(6)
$p_{\hat{\mathcal{G}}}(\hat{\mathbf{r}},\mathbf{c},\mathbf{p})=p(\hat{\mathbf{r}})\cdot
p(\mathbf{p}|\hat{\mathbf{r}})\cdot
p(\mathbf{c}|\mathbf{p},\hat{\mathbf{r}}).$

And the data distribution for the expected causal graph $\mathcal{G}$ is:

(7)
$p_{\mathcal{G}}(\hat{\mathbf{r}},\mathbf{c},\mathbf{p})=p(\hat{\mathbf{r}})\cdot
p(\mathbf{p})\cdot p(\mathbf{c}|\mathbf{p},\hat{\mathbf{r}})$

To reweight the data distribution of $\hat{\mathcal{G}}$ towards the data
distribution of $\mathcal{G}$, the reweighting score on the position should be
$\mathbf{w}_{p}=\frac{p(p)}{p(p|\hat{r})}=\frac{p_{\mathcal{G}}(\hat{r},c,p)}{p_{\hat{\mathcal{G}}}(\hat{r},c,p)}$,
where $p(p|\hat{r})$ is the influence score on SERP feature from Section 3.1.

The other scenario is that the SERP feature $p$ is not only dependent on the
click but also on other SERP features $m$ as illustrated in Figure 2(b).
Similarly, the reweighting score is
$\mathbf{w}=\frac{p_{\mathcal{G}}(\mathbf{r},\mathbf{c},\mathbf{p},\mathbf{m})}{p_{\hat{\mathcal{G}}}(\hat{\mathbf{r}},\mathbf{c},\mathbf{p},\mathbf{m})}=\frac{p(\mathbf{p}|\hat{\mathbf{r}},\mathbf{m})}{p(\mathbf{p}|\mathbf{m})}$,
where $p(\mathbf{p}|\hat{\mathbf{r}},\mathbf{m})$ can be obtained from the
SERP propensity score estimator easily. $p(\mathbf{p}|\mathbf{m})$ can be
estimated via data permutation in batch as
$p(\mathbf{p}|\mathbf{m})=\sum_{\mathbf{\hat{r}}^{\prime}\in\mathbf{R}}p(\hat{\mathbf{r}}^{\prime})p(\mathbf{p}|\hat{\mathbf{r}}^{\prime},\mathbf{m})$.
$\mathbf{R}$ is a batch of relevance score data.

Taking the above two scenarios into consideration, we can remove the backdoor
path on each SERP feature. Then, the effect of SERP features and relevance
score on the click can be corrected by maximizing the reweighted maximum
likelihood objective as:

(8) $\max_{\phi}\mathbb{E}_{X\sim P_{X}}\mathbf{w}_{i}\log
p\left(\mathbf{c}\mid\mathbf{Z}_{\pi_{c}},\phi_{c}\right)$

where $\mathbf{w}=\prod_{i}\mathbf{w}_{i}$ is the total reweighting score
defined as the product of the reweighting score $\mathbf{w}_{i}$ on each SERP
feature with confounding bias on the relevance score.
$p\left(\mathbf{c}_{i}\mid\mathbf{Z}_{\pi_{c}};\phi_{c}\right)$ can be
estimated with an MLP parameterized with $\phi_{c}$, similar to the influence
estimator on the SERP feature. Then, we can get a well-learned MLP estimator
on the data distribution of the unconfounded causal graph $\mathcal{G}$.

#### Ranking model learning

After mitigating the confounding bias from the relevance score, the SERP bias
still exists, where the relevance score and the SERP feature can affect the
click user behavior. However, the ranking model should only consider the
relevance between the query and the document without SERP features. To learn a
ranking model without effect from SERP features, we block the gradient related
to SERP features and only learn the relationship from relevance to click,
shown in Figure 2(a)(3) with a red arrow. The item-wise loss in Eq. (8) is
utilized to optimize the ranking model parameters $\Theta$. Note that the
unbiased user behavior model modification sub-step utilizes the same
optimization object but on a different parameter set, $\phi_{c}$, with the
goal of removing the backdoor path. Ultimately, we discover and mitigate
biases from multiple SERP features and learn towards an unbiased ranking
model.

Table 1. Performance comparison of BAL algorithm, its variants, and PB-ULTR
baseline algorithms on the Baidu-ULTR dataset. The best performance is
highlighted in boldface.

| DCG@1 | DCG@3 | DCG@5 | DCG@10 | ERR@1 | ERR@3 | ERR@5 | ERR@10
---|---|---|---|---|---|---|---|---
Naive | 1.235$\pm$0.029 | 2.743$\pm$0.072 | 3.889$\pm$0.087 | 6.170$\pm$0.124 | 0.077$\pm$0.002 | 0.133$\pm$0.003 | 0.156$\pm$0.003 | 0.178$\pm$0.003
IPW | 1.239$\pm$0.038 | 2.742$\pm$0.076 | 3.896$\pm$0.100 | 6.194$\pm$0.115 | 0.077$\pm$0.002 | 0.133$\pm$0.003 | 0.156$\pm$0.004 | 0.178$\pm$0.003
REM | 1.230$\pm$0.042 | 2.740$\pm$0.079 | 3.891$\pm$0.099 | 6.177$\pm$0.126 | 0.077$\pm$0.003 | 0.132$\pm$0.003 | 0.156$\pm$0.004 | 0.178$\pm$0.004
PairD | 1.243$\pm$0.037 | 2.760$\pm$0.078 | 3.910$\pm$0.092 | 6.214$\pm$0.114 | 0.078$\pm$0.002 | 0.133$\pm$0.003 | 0.156$\pm$0.003 | 0.179$\pm$0.003
DLA | 1.293$\pm$0.015 | 2.839$\pm$0.011 | 3.976$\pm$0.007 | 6.236$\pm$0.017 | 0.081$\pm$0.001 | 0.137$\pm$0.001 | 0.160$\pm$0.001 | 0.181$\pm$0.001
BAL | 1.383$\pm$0.018 | 3.086$\pm$0.078 | 4.088$\pm$0.142 | 6.410$\pm$0.117 | 0.098$\pm$0.003 | 0.153$\pm$0.004 | 0.171$\pm$0.004 | 0.192$\pm$0.003

Table 2. Performance comparison of ULTR algorithms on queries with different
frequencies in the Baidu-ULTR dataset. The best performance is highlighted in
boldface.

| DCG@1 | DCG@3 | DCG@5 | DCG@10
---|---|---|---|---
High | Tail | High | Tail | High | Tail | High | Tail
Naive | 1.730$\pm$0.053 | 0.875$\pm$0.044 | 3.726$\pm$0.052 | 2.028$\pm$0.072 | 5.029$\pm$0.093 | 3.060$\pm$0.086 | 8.873$\pm$0.118 | 4.203$\pm$0.107
IPW | 1.743$\pm$0.061 | 0.872$\pm$0.074 | 3.697$\pm$0.174 | 2.047$\pm$0.101 | 5.020$\pm$0.110 | 3.078$\pm$0.065 | 8.872$\pm$0.133 | 4.216$\pm$0.080
REM | 1.734$\pm$0.073 | 0.863$\pm$0.051 | 3.711$\pm$0.119 | 2.034$\pm$0.054 | 5.015$\pm$0.123 | 3.106$\pm$0.079 | 8.869$\pm$0.136 | 4.320$\pm$0.097
PairD | 1.752$\pm$0.068 | 0.873$\pm$0.064 | 3.674$\pm$0.122 | 2.060$\pm$0.084 | 4.938$\pm$0.181 | 3.129$\pm$0.073 | 8.898$\pm$0.141 | 4.261$\pm$0.127
DLA | 1.883$\pm$0.099 | 0.864$\pm$0.053 | 3.791$\pm$0.057 | 2.146$\pm$0.016 | 5.054$\pm$0.033 | 3.332$\pm$0.019 | 9.017$\pm$0.029 | 4.689$\pm$0.013
BAL | 2.100$\pm$0.113 | 0.861$\pm$0.107 | 4.525$\pm$0.083 | 2.039$\pm$0.113 | 5.734$\pm$0.129 | 2.890$\pm$0.132 | 9.247$\pm$0.157 | 4.710$\pm$0.175

##  4\. Experiment

In this section, we design experiments to validate the effectiveness of BAL by
answering the following research questions.

  * •

RQ1: How does the performance of BAL the first algorithm in WP-ULTR, compare
with the state-of-the-art PB-ULTR algorithms in the real-world dataset?

  * •

RQ2: How does BAL perform on queries with different frequencies?

  * •

RQ3: How do different components in BAL affect its effectiveness?

  * •

RQ4: How does the user behavior model (the causal graph) change along with the
ULTR procedure?

###  4.1. Experiment Setting

#### Dataset.

Our experiments are conducted on the largest real-world ULTR dataset, Baidu-
ULTR (Zou et al., 2022), to analyze the effectiveness of our proposed BAL
algorithm. It includes 383,429,526 queries and 1,287,710,306 documents with
click recorded for training. 7,008 queries and 367,262 documents with expert
annotations are utilized for evaluation. The relevance annotation of each
document to the specific query is judged by expert annotators in five levels,
i.e., {bad, fair, good, excellent, perfect}. Baidu-ULTR is the only open-
source ULTR dataset providing the whole-page SERP features, including the
ranking position, the document title, the document abstract, the multimedia
type of the SERP (e.g., video, image, and advertisement), the SERP height (the
vertical pixels), and the maximum SERP height. The title and the abstract of
the document are utilized to generate the query-document relevant score. And
the click feedback is collected from the real-world Baidu search engine,
rather than the synthetic data generated with the heuristic assumption in most
existing ULTR datasets (Chapelle and Chang, 2011; Qin et al., 2010; Dato et
al., 2016).

Remark. Synthetic experimental analysis on the previous PB-ULTR works
(Joachims et al., 2017b; Ai et al., 2018a; Wang et al., 2018; Hu et al.,
2019b) cannot be utilized in the WP-ULTR scenario. The position-based click
model (PBM) (Chapelle and Zhang, 2009) is usually utilized to generate the
synthetic click data. The PBM utilizes the examination assumption (Richardson
et al., 2007) as $p(c)=p(o)\cdot p(r)$, where $p(c)$, $p(o)$, and $p(r)$ are
the probabilities of click, observation, and relevance, respectively. However,
PBM and other click models cannot be extended to the WP-ULTR scenario since
the generated click data is only biased to the position while ignoring other
SERP features. Moreover, we cannot examine how well the algorithm estimates
the bias since we do not know what the true bias is in WP-ULTR.

#### Metric.

The DCG (Discounted Cumulative Gain) (Järvelin and Kekäläinen, 2017) and ERR
(Ghanbari and Shakery, 2019) (Expected Reciprocal Rank) are utilized as
metrics to access the performance of the relevance model. For both metrics, we
report the results at ranks of 1, 3, 5, and 10.

#### Baselines.

We select representative ULTR algorithms as baseline methods including IPW
(Joachims et al., 2017b), DLA (Ai et al., 2018a), REM (Wang et al., 2018), and
PairD (Hu et al., 2019b). Notice that all the algorithms which belong to the
PB-ULTR algorithm only consider position-related biases. Moreover, a naive
algorithm is utilized which trains the ranking model directly with the biased
click data, as stated in Eq. 3. We adopt the same ULTR toolkit (Tran et al.,
2021) utilized in hyperparameter selection to implement baseline methods.
Remarkably, the naive algorithm may not be necessary to show the worst
performance. Other ULTR baseline algorithms may not also perform as expected
if real-world datasets do not meet their predefined user behavior hypotheses.
Notice that, all the implementation details are detailed in the Appendix A

###  4.2. Overall Performance Comparison (RQ1)

The experimental results of all baselines, and our proposed BAL algorithm on
DCG@{1, 3, 5, 10} and ERR@{1, 3, 5, 10} are illustrated in Table 1. We can
observe that our BAL algorithm shows consistently better results than all
baseline algorithms across all the evaluation metrics. The maximum gains on
DCG@N metrics and ERR@N metrics over the best baseline performance are 0.247
and 0.017, respectively. Remarkably, maximum gains on DCG@N and ERR@N are from
$N=1,3$, respectively. It indicates that our algorithm can be more beneficial
on those small-screen devices like mobile phones where users usually only
browse a few documents. For the baseline methods focusing on the PB-ULTR
scenarios, PairD, REM, and IPW show similar performance with the naive
algorithm while the DLA shows limited performance improvement. The existing
PB-ULTR algorithms are not suitable for the real-world dataset with more
complicated biases. It indicates the necessity of introducing the new WP-ULTR
problem. Further fine-grain experimental analysis on the effectiveness of BAL
over existing PB-ULTR algorithms can be found in Section 4.4.

#### Discussion.

There could be two key reasons why PB-ULTR algorithms show no better
performance than the biased naive algorithm. First, position-based hypotheses
are not sufficient in the real-world scenario since there are biases induced
by other SERP features. For example, users are more likely to click on the
document with the video type. Evidence from one empirical observation can
further support this perspective – the training loss of PB-ULTR algorithms can
vary from 100 to 0.5 between two training steps while the naive algorithm
shows a more stable training procedure. This attributes to the unsuitable
reweighting score on the inaccurate position hypothesis in the training
procedure across different data batches. Second, position-based biases have
been extensively studied thus existing PB-ULTR algorithms may have already
been adopted by the existing search engines. That is why applying the same
algorithm to the newly generated click data shows no additional benefit.

###  4.3. The Impact of Query frequency (RQ2)

In the real-world search engine, user search queries follow heavy-tailed
distributions. We further conduct experiments to investigate how algorithms
perform on queries with different frequencies. The evaluation set with expert
annotations provides a frequency identifier for each query. Queries are split
into 10 buckets descendingly according to the query frequency. We illustrate
our performance on the queries with high and tail frequencies, which
correspond to buckets 0, 1, 2, 3, 4, and buckets 5, 6, 7, 8, 9, respectively.

The experimental results are shown in Table 2. One observation is that all
algorithms show much better performance on high-frequency queries than on
tail-frequency queries. It indicates that learning to rank on the tail queries
is still a challenging problem. Our proposed BAL algorithm can consistently
perform significantly better than baseline algorithms for high-frequency
queries. While the performance on the tail-frequency queries is only slightly
better than baseline methods. The main reason is that the statistic
significance-based causal discovery algorithm is more likely to find biases in
high-frequency queries.

Figure 3.  An illustration of how the causal graph changes during the training
procedure on Baidu-ULTR. $r$, $c$, $p$, $m$, $h$, and $mh$ correspond to
relevance score, click, position, multimedia types, SERP height, and maximum
SERP height, respectively.

###  4.4. Effects of BAL and its components (RQ1,3,4)

In this subsection, we conduct further experiments to investigate the
effectiveness of (1) the entire BAL algorithm, (2) the click model design, and
(3) the unbiased learning. The experiment for the effectiveness of the
unbiased learning step shows how the causal graph varies in different training
steps, which also responds to RQ4.

(a) Effectiveness of BAL algorithm

(b) Effectiveness of user behavior model design

Figure 4. (a) Average positions after re-ranking of documents at the top-10
original position by different ULTR methods. (b) Performance comparison on the
naive algorithm, BAL and its variants without user behavior model design.

#### Effectiveness of BAL algorithm

To further support the effectiveness of BAL, we conduct a fine-grained
analysis to see how algorithms perform on each original position. We first
identified the document at each position given by the original ranker, i.e.,
the ranker trained with the naive algorithm. After re-rank by the ULTR
algorithms, we calculate the new document average positions at the original
position. Moreover, re-rank results with ground truth relevance annotation are
also included as the upper bound. The results on Baidu-ULTR are illustrated in
Figure 4(a). The curve and corresponding shadow area stand for the mean value
and the standard deviation, respectively. Note that, we provide the results on
the top 10 documents with the original ranker for better visualization. The
top documents are also the most important for the ranking system. One
observation is that the curves of position-based ULTR algorithms are more
distant from that of true relevance labels. This further indicates position
bias may have already been alleviated in the new click data. Contrastively,
the curve of our BAL is the closest one to the true relevance annotation in
most cases, which further indicates the superiority of our algorithm.

#### Effectiveness of click model design

To verify the effectiveness of our click model design, we propose two variants
of our proposed BAL without the causal discovery algorithm as follows: (1)
Position Based BAL (PB-BAL) replaces the causal discovery algorithm with the
predefined user behavior model frequently utilized in PB-ULTR, with only two
edges, from the position and relevant score to the click. (2) Fully biased BAL
(FB-BAL) recognizes that the click is biased on all SERP features. The causal
graph of the FB-BAL is that the relevance score $r$ will affect all SERP
features, and all SERP features will affect the click. Experimental results
are illustrated in Figure 4(b). We note that both variants show a performance
drop which indicates the importance of the click model design. For the PB-BAL
algorithm, its performance is only comparable with the naive algorithm. This
observation further indicates that only considering position-based biases is
not sufficient for the WP-ULTR problem. For the FB-BAL algorithm, it shows the
worst performance across all algorithms. This observation suggests that if
mitigating the bias improperly, the true relationship may be removed by
mistake, leading to unsatisfied results.

#### Effectiveness of unbiased learning

To verify how well unbiased learning works, we examine how the user behavior
model (causal graph) changes along with the training procedure. This
experiment also provides a closer look at how biases are mitigated in the BAL
training procedure. Results are shown in Figure 3. First, we figure out what
the original causal graph and the expected graph after training are. For the
original causal graph, we observe that multiple biases exist. The click is
biased to both the multimedia type and the position. Remarkably, relationships
also exist among SERP features. The height shows an effect on the maximum
height. For the expected causal graph after training, all backdoor paths from
the relevance score to SERP features are removed. Then we make the following
observations on how the causal graph varies during training as follows: (1)
The backdoor path from the relevance score to the position is mitigated first.
This may further indicate that the position bias has already been alleviated
by the feature process. It is much easier to mitigate position bias. (2) A new
edge from relevance score to height appears. (3) The above new edge disappears
quickly. The main reason for this observation is that the causal discovery
algorithm is conducted on only a small portion of queries. Some relationships
may appear in some particular data. However, it is not a common pattern across
the whole dataset. It will quickly disappear in a few training steps. We
ignore other similar small fluctuations that appeared in the training
procedure. (4) The backdoor path from the relevance score to the position is
removed. The above experimental results verify that the unbiased learning step
of our BAL can successfully mitigate biases found by the click model design.

##  5\. Related Work

#### Unbiased Learning to Rank

Unbiased learning to rank aims to optimize with an unbiased estimation of the
loss function based on biased user feedback. Existing ULTR algorithms only
focus on position-based biases without taking other biases into consideration.
(Wang et al., 2016) mitigates the position bias with two key components: (1)
an estimator on the predefined examination user behavior model (Richardson et
al., 2007) which can estimate the examination propensity score of each
document by result randomization experiments. (2) an unbiased learning
procedure with Inverse Propensity Weighting (IPW) (Rosenbaum and Rubin, 1983).
(Joachims et al., 2017b) further proves that the learning objective with IPW
is an unbiased estimate of the true relevance loss function based on the exam
hypothesis. (Wang et al., 2018) improves the estimator with the regression-
based EM algorithm with no requirement for the results permutation. (Hu et
al., 2019a) improves the unbiased learning procedure from a pointwise view to
a pairwise view, which focuses more on the relative order of documents rather
than the absolute relevant score of documents. Furthermore, (Ai et al., 2018b)
indicates that those two components are actually a dual problem and can be
learned jointly. Despite the position bias, other position-based biases have
also been taken into consideration. (Agarwal et al., 2019) takes the position-
dependent trust bias (Joachims et al., 2017a) into consideration which is more
robust to noise. However, those position-based algorithms cannot be
successfully utilized in the real-world WP-ULTR scenario with biases induced
by multiple SERP features.

#### User behavior model

The user behavior models (Keane and O’Brien, 2006; Richardson et al., 2007;
Joachims et al., 2017a, 2007; Yue et al., 2010; Wang et al., 2013) are
proposed to analyze how click is biased on the user SERP features. (Richardson
et al., 2007) finds the examination model where the model with a higher
position is more likely to be examine. Then both position and examination will
lead to the final click. (Yue et al., 2010) shows that the more attractive
results are more likely to receive clicks. The most frequently utilized one is
the (Joachims et al., 2007) which proposes that the user shows more attention
to the document with a higher position. In the existing ULTR and click model
techniques, only position-related user behavior models are utilized for
further study.

#### Click Model

Instead of the end-to-end training in ULTR, click models (Craswell et al.,
2008; Dupret and Piwowarski, 2008; Chapelle and Zhang, 2009; Xu et al., 2010)
mitigate the bias with the following two steps: (1) extract the true relevance
feedback from the click data (2) use the new signal to train an unbiased
ranking model. (Craswell et al., 2008) is a cascade model designed with
position bias. It assumes that users are more likely to click document in a
higher position since users will read the result page from top to down. User
behavior model (Chapelle and Zhang, 2009) incorporates the examination bias
based on the position and last click.

Remark. Click model is different from the user behavior model. The user
behavior model is a probabilistic graph describing the existing biases in the
search engine. By contrast, the click model and ULTR algorithms are specific
techniques to utilize the assumptions in the user behavior model to learn an
unbiased ranker.

#### Causal discovery

Causal discovery aims to learn the causal relationship among variables from
purely observational data. It serves as one of the key tools for scientific
discovery in various fields, such as Biology (Zhang et al., 2012b). One major
class of methods, namely constraint-based methods, leverage conditional
independence tests to estimate the causal graph (e.g., PC (Spirtes et al.,
2000) and FCI (Spirtes et al., 2013)). Another popular category is score-based
methods, of which the GES (Chickering, 2002) is a representative one. Besides,
much attention has been drawn to weakening the assumptions and extending the
applicability, for instance, relaxing functional or distributional constraints
(Shimizu et al., 2006; Zhang and Hyvarinen, 2012; Hoyer et al., 2008; Zheng et
al., 2022), focusing on mixed data types (Andrews et al., 2019), or improving
the scalability (Ng et al., 2021).

In our setting, we are required to deal with complex real-world data,
suggesting that our method should be able to deal with general distributions
with arbitrary functional relations and data types. By utilizing nonparametric
tests, such as Kernel Conditional Independence test (KCI) (Zhang et al.,
2012a), constraint-based methods (e.g., PC) can be applied in the general
setting. Thus, we adopt PC with KCI for the procedure of structure learning.

##  6\. Conclusion & future work

Learning to rank with expert annotation data can be expensive and labor-
intensive. One solution is to design a suitable user behavior model and learn
an unbiased ranking system from implicit user feedback. The majority of
existing user behavior models only consider the position-based bias. However,
multiple SERP features can introduce biases in addition to the position. In
this paper, we propose a new problem, i.e., Whole-page Unbiased learning to
Rank (WP-ULTR). It takes biases induced by all SERP features into
consideration. In particlar, we propose the BAL algorithm to automatically
find and mitigate biases instead of the heuristic design. Extensive
experiments on a large-scale real-world dataset indicate the effectiveness of
BAL.

One future direction is to extend our WP-ULTR to more user behaviors. As
previous works only consider the click user behavior while other user
behaviors, (e.g., SERP time, SERP count, and slip-off count), are ignored. We
may be able to include more user behaviors to find more complex user behavior
models and mitigate the real-world biases.

## References

  * (1)
  * Agarwal et al. (2019) Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork. 2019.  Addressing trust bias for unbiased learning-to-rank. In _The World Wide Web Conference_. 4–14.
  * Ai et al. (2018a) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018a.  Unbiased Learning to Rank with Unbiased Propensity Estimation. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_ (Ann Arbor, MI, USA) _(SIGIR ’18)_. ACM, New York, NY, USA, 385–394.  https://doi.org/10.1145/3209978.3209986
  * Ai et al. (2018b) Qingyao Ai, Jiaxin Mao, Yiqun Liu, and W. Bruce Croft. 2018b.  Unbiased Learning to Rank: Theory and Practice. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ (Torino, Italy) _(CIKM ’18)_. ACM, New York, NY, USA, 2305–2306.  https://doi.org/10.1145/3269206.3274274
  * Andrews et al. (2019) Bryan Andrews, Joseph Ramsey, and Gregory F. Cooper. 2019\.  Learning High-dimensional Directed Acyclic Graphs with Mixed Data-types.  _Proceedings of machine learning research_ 104 (2019), 4–21.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E. Terry. 1952.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS.  _Biometrika_ (1952).
  * Chapelle and Chang (2011) Olivier Chapelle and Yi Chang. 2011.  Yahoo! learning to rank challenge overview. In _Proceedings of the learning to rank challenge_. PMLR, 1–24.
  * Chapelle and Zhang (2009) Olivier Chapelle and Ya Zhang. 2009.  A dynamic bayesian network click model for web search ranking. In _WWW ’09_.
  * Chen et al. (2012) Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012.  Beyond ten blue links: enabling user click modeling in federated web search. In _WSDM ’12_.
  * Chickering (2002) David Maxwell Chickering. 2002\.  Optimal structure identification with greedy search.  _Journal of machine learning research_ 3, Nov (2002), 507–554.
  * Chuklin et al. (2013) Aleksandr Chuklin, Pavel Serdyukov, and M. de Rijke. 2013\.  Click model-based information retrieval metrics.  _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_ (2013).
  * Craswell et al. (2008) Nick Craswell, Onno Zoeter, Michael J. Taylor, and Bill Ramsey. 2008.  An experimental comparison of click position-bias models. In _WSDM ’08_.
  * Dato et al. (2016) Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016\.  Fast ranking with additive ensembles of oblivious and non-oblivious regression trees.  _ACM Transactions on Information Systems (TOIS)_ 35, 2 (2016), 1–31.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dupret and Piwowarski (2008) Georges Dupret and Benjamin Piwowarski. 2008.  A user browsing model to predict search engine click data from past observations.. In _SIGIR ’08_.
  * Ghanbari and Shakery (2019) Elham Ghanbari and Azadeh Shakery. 2019.  ERR. Rank: An algorithm based on learning to rank for direct optimization of Expected Reciprocal Rank.  _Applied Intelligence_ 49, 3 (2019), 1185–1199.
  * Hoyer et al. (2008) Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. 2008\.  Nonlinear causal discovery with additive noise models.  _Advances in neural information processing systems_ 21 (2008).
  * Hu et al. (2019a) Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019a.  Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm.  _The World Wide Web Conference_ (2019).
  * Hu et al. (2019b) Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019b.  Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_. ACM, 2830–2836.
  * Järvelin and Kekäläinen (2017) K. Järvelin and Jaana Kekäläinen. 2017.  IR evaluation methods for retrieving highly relevant documents. In _SIGIR’17_.
  * Joachims et al. (2017a) Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017a.  Accurately Interpreting Clickthrough Data as Implicit Feedback.  _ACM SIGIR Forum_ 51 (2017), 4 – 11.
  * Joachims et al. (2007) Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search.  _ACM Trans. Inf. Syst._ 25 (2007), 7.
  * Joachims et al. (2017b) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017b.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. 781–789.
  * Keane and O’Brien (2006) Mark T. Keane and Maeve O’Brien. 2006.  Modeling Result-List Searching in the World Wide Web: The Role of Relevance Topologies and Trust Bias.
  * Lagun and Agichtein (2014) Dmitry Lagun and Eugene Agichtein. 2014.  Effects of task and domain on searcher attention.  _Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval_ (2014).
  * Liu and Tao (2015) Tongliang Liu and Dacheng Tao. 2015.  Classification with noisy labels by importance reweighting.  _IEEE Transactions on pattern analysis and machine intelligence_ 38, 3 (2015), 447–461.
  * Liu et al. (2015) Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015.  Influence of Vertical Result in Web Search Examination.  _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2015).
  * Mao et al. (2018) Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018\.  Constructing Click Models for Mobile Search.  _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_ (2018).
  * Marden (1996) John Marden. 1996\.  Analyzing and Modeling Rank Data.
  * Ng et al. (2021) Ignavier Ng, Yujia Zheng, Jiji Zhang, and Kun Zhang. 2021\.  Reliable Causal Discovery with Improved Exact Search and Weaker Assumptions.  _Advances in Neural Information Processing Systems_ 34 (2021), 20308–20320.
  * Qin et al. (2010) Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. 2010\.  LETOR: A benchmark collection for research on learning to rank for information retrieval.  _Information Retrieval_ 13, 4 (2010), 346–374.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert J. Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _WWW ’07_.
  * Rosenbaum and Rubin (1983) Paul R. Rosenbaum and Donald B. Rubin. 1983.  The central role of the propensity score in observational studies for causal effects.  _Biometrika_ 70 (1983), 41–55.
  * Shimizu et al. (2006) Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. 2006\.  A linear non-Gaussian acyclic model for causal discovery.  _Journal of Machine Learning Research_ 7, 10 (2006).
  * Spirtes et al. (2000) Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. 2000.  _Causation, prediction, and search_.  MIT press.
  * Spirtes et al. (2013) Peter L Spirtes, Christopher Meek, and Thomas S Richardson. 2013.  Causal inference in the presence of latent variables and selection bias.  _arXiv preprint arXiv:1302.4983_ (2013).
  * Tran et al. (2021) Anh Tran, Tao Yang, and Qingyao Ai. 2021.  ULTRA: An Unbiased Learning To Rank Algorithm Toolbox. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4613–4622.
  * Wang et al. (2013) Chao Ching Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013.  Incorporating vertical results into search click models.  _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_ (2013).
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 115–124.
  * Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018\.  Position bias estimation for unbiased learning to rank in personal search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 610–618.
  * Wu et al. (2014) Fangzhao Wu, Jun Xu, Hang Li, and Xin Jiang. 2014\.  Ranking Optimization with Constraints.  _Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management_ (2014).
  * Wu et al. (2018) Yongkai Wu, Lu Zhang, and Xintao Wu. 2018.  On Discrimination Discovery and Removal in Ranked Data using Causal Graph.  _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (2018).
  * Xu et al. (2010) Wanhong Xu, Eren Manavoglu, and Erick Cantú-Paz. 2010\.  Temporal click model for sponsored search.  _Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval_ (2010).
  * Yue et al. (2010) Yisong Yue, Rajan Patel, and Hein Roehrig. 2010.  Beyond position bias: examining result attractiveness as a source of presentation bias in clickthrough data. In _WWW ’10_.
  * Zhang and Hyvarinen (2012) Kun Zhang and Aapo Hyvarinen. 2012.  On the identifiability of the post-nonlinear causal model.  _arXiv preprint arXiv:1205.2599_ (2012).
  * Zhang et al. (2012a) Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2012a.  Kernel-based conditional independence test and application in causal discovery.  _arXiv preprint arXiv:1202.3775_ (2012).
  * Zhang et al. (2012b) Xiujun Zhang, Xing-Ming Zhao, Kun He, Le Lu, Yongwei Cao, Jingdong Liu, Jin-Kao Hao, Zhi-Ping Liu, and Luonan Chen. 2012b.  Inferring gene regulatory networks from gene expression data by path consistency algorithm based on conditional mutual information.  _Bioinformatics_ 28, 1 (2012), 98–104.
  * Zheng et al. (2022) Yujia Zheng, Ignavier Ng, and Kun Zhang. 2022.  On the Identifiability of Nonlinear ICA: Sparsity and Beyond.  _Advances in neural information processing systems_ (2022).
  * Zou et al. (2022) Lixin Zou, Haitao Mao, Xiaokai Chu, Jiliang Tang, Wenwen Ye, Shuaiqiang Wang, and Dawei Yin. 2022.  A Large Scale Search Dataset for Unbiased Learning to Rank.
  * Zou et al. (2021) Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained language model based ranking in Baidu search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 4014–4022.

##  Appendix A Experimental Implementation Details

In our experiments, the following ranking model is utilized for approximating
the scoring function $f(\mathbf{q},\mathbf{d})$. The ranking model employs the
same architecture as BERT. It has 12 layers. Each layer has 768 dimensions
with 12 heads. The model is pre-trained both the MLM loss (Devlin et al.,
2018) and the naive loss in Eq. (3). The pre-trained model we utilized is
provided by (Zou et al., 2022). It can be found in the link:
https://github.com/ChuXiaokai/baidu_ultr_dataset. An MLP decoder is built upon
on the [CLS] embedding. The hidden layer dimensions of the 4-layer MLP
decoders are 512-256-128. The hyper-parameters for training the ULTR
algorithms are selected settings from ULTR toolkit111https://github.com/ULTR-
Community/ULTRA_pytorch (Tran et al., 2021). The library causal-
learning222https://github.com/cmu-phil/causal-learn is utilized for the click
model design. Each experiment result is obtained by 5 repeat runs with
different random seeds. The Adam optimizer is utilized for training. The best
performance are selected based on the highest performance on DCG@10. All the
models are trained on the machine with 80G Memory, 4 NVIDIA A100 GPUs, and
128T Disk. Notice that, the Baidu-ULTR dataset still suffers from some
drawbacks. We can not utilize the pre-training model trained on other data
sources since the raw text feature is after desensitization. The MLP model we
utilized for SERP estimator has 2 hidden layer, and each dimension is 10.