# Modeling Relevance Ranking
under the Pre-training and Fine-tuning Paradigm

Lin Bo,1 Liang Pang,3 Gang Wang,4 Jun Xu,2,∗ XiuQiang He,4 Ji-Rong Wen2
1School of Information, Renmin University of China2Gaoling School of
Artificial Intelligence, Renmin University of China3Institute of Computing
Technology, Chinese Academy of Sciences; 4Huawei Noah’s Ark Lab bolin20,
junxu<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

**footnotetext: Corresponding author

Recently, pre-trained language models such as BERT have been applied to
document ranking for information retrieval (IR). These methods usually first
pre-train a general language model on an unlabeled large corpus and then
conduct ranking-specific fine-tuning on expert-labeled relevance datasets.
Though preliminary successes have been observed in a variety of IR tasks, a
lot of room still remains for further improvement. Ideally, an IR system would
model relevance from a user-system dualism: the user’s view and the system’s
view. User’s view judges the relevance based on the activities of “real users”
while the system’s view focuses on the relevance signals from the system side,
e.g., from the experts or algorithms, etc. (Saracevic, 1975; Hjørland, 2010).
Inspired by the user-system relevance views and the success of pre-trained
language models, in this paper we propose a novel ranking framework called
Pre-Rank that takes both user’s view and system’s view into consideration,
under the pre-training and fine-tuning paradigm. Specifically, to model the
user’s view of relevance, Pre-Rank pre-trains the initial query-document
representations based on a large-scale user activities data such as the click
log. To model the system’s view of relevance, Pre-Rank further fine-tunes the
model on expert-labeled relevance data. More importantly, the pre-trained
representations, are fine-tuned together with handcrafted learning-to-rank
features under a wide and deep network architecture. In this way, Pre-Rank can
model the relevance by incorporating the relevant knowledge and signals from
both real search users and the IR experts. To verify the effectiveness of Pre-
Rank, we showed two implementations by using BERT (Devlin et al., 2018) and
SetRank (Pang et al., 2020) as the underlying ranking model, respectively.
Experimental results base on three publicly available benchmarks showed that
in both of the implementations, Pre-Rank can respectively outperform the
underlying ranking models and achieved state-of-the-art performances. The
results demonstrate the effectiveness of Pre-Rank in combining the user-system
views of relevance.

pre-trained IR model; neural information retrieval

##  1\. Introduction

Relevance ranking, whose objective is to provide the right ranking order of a
list of documents for a given query (Liu, 2011; Li, 2011), has played a vital
role in the field of information retrieval (IR). Machine learning models,
especially deep neural networks (Goodfellow et al., 2016) have been applied to
relevance ranking and many ranking techniques have been developed (Guo et al.,
2020a; Xu et al., 2018). One branch of the research formalizes the learning of
ranking models as first pre-training a general language model on large-scale
unlabeled texts and then fine-tuning on the labeled relevance data. For
example, Nogueira and Cho (2019) consider the training of a passage ranking
model as a downstream task in BERT fine-tuning; Chang et al. (2020) propose to
pre-train a BERT using the Inverse Cloze Task (ICT) as the objective, which
aims to teach the model to predict the removed sentence given a context text;
Ma et al. (2021) proposes a new task of representative words prediction (ROP)
to pre-train a BERT model for the ad-hoc retrieval.

Despite improvements that have been observed in many ranking tasks, existing
studies either directly consider the ranking learning as a downstream task
under the pre-training framework (Nogueira and Cho, 2019), or simply adapt
pre-training objectives. The nature and requirements of relevance ranking are
rarely taken into consideration. According to the studies in (Saracevic, 1975;
Hjørland, 2010), the relevance of a document to a query can be considered from
two views: the user’s view and the system’s view. The user’s view prefers that
the relevance assessments should be made by the “real users”. The system’s
view also called the algorithm’s view, emphasizes systems processing
information objects and matching them with queries. Ideally, an IR model would
consider both of these two views when conducting the relevance ranking of
documents. Existing studies, however, usually model the relevance from only
one of the views.

Inspired by the observations and the recent progress of the pre-trained
language models, in this paper we propose modeling both the user’s and the
system’s view of relevance under the pre-training and fine-tuning framework,
called Pre-Rank. In the pre-training stage, it makes use of the real user’s
activities, i.e., the large-scale user click log, rather than the unlabeled
text corpus for training. Since the user activities imply the relevance
judgments of the query-document pairs from the real users, the pre-trained
representations are based on the user’s view of relevance, rather than the
general natural language processing (NLP) knowledge. One problem with the pre-
trained representations is that the user’s activities are usually noisy and
contain strong biases (e.g., position bias, selection bias, etc.). To
alleviate the issue, Pre-Rank fine-tunes the pre-trained model parameters with
query-document pairs with unbiased labels by the experts. To further enhance
the ranking performances and incorporating the expert knowledge, Pre-Rank also
extends a wide branch (Cheng et al., 2016) to the pre-trained network,
resulting in a wide and deep architecture where the wide branch is responsible
to inject the handcrafted learning-to-rank features into the model. Since the
labeled query-document pairs and the handcrafted features were created based
on the expert’s knowledge on the relevance, the fine-tuning stage naturally
adapts the ranking model to reflect the system’s view on the relevance and
alleviate the biases from the users.

Pre-Rank offers several advantages. First of all, compared to existing studies
in which the pre-trained models are based on unsupervised text data, Pre-Rank
makes use of the click log. The pre-trained representations, therefore,
reflect the user’s view on relevance, which makes it easy to conduct the
downstream fine-tune on expert-labeled relevance data. Second, the handcrafted
learning-to-rank features and expert-labeled query-document pairs reflect the
expert knowledge (system’s view) on IR relevance, which is complementary to
relevance information from user activities in the pre-training stage. It is
believed that modeling the two views simultaneously is helpful, especially
when very limited labeled query-document pairs are available for fine-tuning.
Third, existing studies have shown that the user clicks contain biased
information (e.g., position bias, selection bias, etc.) and therefore may hurt
the model performances if being directly used as the training corpus (Chen et
al., 2020). Pre-Rank provides an effective pre-training and fine-tuning
approach to relieving the bias issue.

Pre-Rank is a general framework that can involve several types of neural
ranking models as its underlying ranking model. As examples, we present two
implementations of Pre-Rank based on the state-of-the-art deep ranking models
of BERT and SetRank. In the Pre-Rank implementation with BERT, we start with
BERT${}_{\textrm{BASE}}$ and continue the pre-training with binary cross-
entropy loss on the click log. In the fine-tuning stage, the extended wide and
deep BERT network is fine-tuned with expert-labeled data. In the
implementation with SetRank, the model is pre-trained on the click log with
list-wise cross-entropy loss. After extending with handcrafted features, the
wide and deep SetRank network is fine-tuned with the labeled relevance data.

We conducted experiments to test the effectiveness of Pre-Rank by pre-training
on large-scale ORCAS click log (Craswell et al., 2020a), and fine-tuning on
three publicly available benchmarks of MQ2007, MQ2008 (Qin et al., 2010), and
TREC19 (Craswell et al., 2020b). In our experiments, we found that Pre-Rank
can respectively outperform the underlying neural ranking models of BERT and
SetRank, and achieved state-of-the-art ranking performances on three
benchmarks. The results indicate the effectiveness of introducing click-data
and handcrafted features in the pre-training and fine-tuning. The experimental
analysis also showed that combining the relevance signals from the user’s view
and from the system’s view can help to improve the accuracy of relevance
ranking.

##  2\. RELATED WORK

One of the most fundamental problems in information retrieval is how to
interpret the concept of relevance. Saracevic (1975); Hjørland (2010) present
“the user’s view” and “the system’s view” of relevance. The user’s view
formalize and measure relevance based on “real users”. In practice, commercial
search engines have collected large-scale user activity data (e.g., click log)
which provide implicit relevance assessments from the real users. One benefit
of using click log is it “removes the cost to the experts of examining and
rating the items.” (Fox et al., 2005). A large number of studies have been
conducted to use the click log in search. For example, Joachims (2002) trains
the ranking SVM model based on the document preference pairs constructed based
on the user clicks. Craswell and Szummer (2007) apply a Markov random walk
model to a large click log. Relevant documents that have not yet been clicked
for that query can be retrieved and ranked effectively. Agichtein et al.
(2006) construct ranking features based on user feedback. See also (Dou et
al., 2008). One difficulty of using the activities from real users is that the
log data contains much noise and biases. Joachims et al. (2017a) examined the
reliability of the implicit relevance information derived from the click data
and found that the clicks are informative but biased, including the position
biases, selection bias, etc. Therefore, the user clicks could not be treated
as relevant judgments (Joachims et al., 2017a; Radlinski and Joachims, 2005;
Joachims et al., 2007). Directly using the click log will inherent biases
which is a key difficulty to effetely use it (Joachims et al., 2017b).
Recently, a large number of methods (e.g., counterfactual learning) have been
proposed to remove the bias information in the click log (Yuan et al., 2020;
Yuan et al., 2019).

In contrast, the system’s view of relevance (also called algorithmic
relevance) “describes the relationship between the query (terms) and the
collection of information objects expressed by the retrieved information
object(s)” (Borlund, 2003). As one of the representative approaches,
traditional learning-to-rank models (Liu, 2011; Li, 2011) represent the query-
document pairs with handcrafted features by the experts, and learn the ranking
models based on labeled relevance data by the annotators (Burges et al., 2005;
Burges, 2010; Yue et al., 2007; Xu and Li, 2007). One difficulty of the
approach is the high cost of gathering high quality handcrafted features and
relevant labels. In recent years, with the availability of large-scale
datasets, deep neural networks have also been applied to IR ranking, called
neural information retrieval (Mitra et al., 2018). The neural information
retrieval models can automatically learn the query-document features from the
raw data, which bridge the gap between query and document vocabulary. For
example, some deep approaches (Huang et al., 2013; Guo et al., 2016; Pang et
al., 2016) focus on discovering the interactions between query and documents
and learn complicated interaction patterns. Both traditional handcrafted
features and automatically learned features are crucial for relevance ranking
(Guo et al., 2020b).

More recently, pre-trained language representation models have led to
significant improvements on many NLP tasks (Devlin et al., 2018; Radford et
al., 2018; Yang et al., 2019). These models are pre-trained on a great amount
of unlabeled data with a large neural architecture. As one of the most
representative works, BERT (Devlin et al., 2018) gets the language
representation model by pre-training on large-scale unlabeled data on a
bidirectional transformer-based model. By adding a simple feedforward
classification layer on top of BERT, it can outperform many task-specific
architectures on various tasks, including IR ranking. There have been studies
(Nogueira and Cho, 2019; Nogueira et al., 2020; Dai and Callan, 2019; Zhang et
al., 2021) applied the pre-trained models to the search tasks, by feeding
query-document pair into BERT and compute the relevance score over the multi-
layer perception (MLP) layer of “[CLS]” token. State-of-the-art performances
have been achieved. In (Ma et al., 2021), the authors designed a specific pre-
training task for IR, called representative words prediction, by sampling a
pair of word sets according to the document language model.

The pre-trained models usually employ plain text to learn the initial
representations, which are originally designed from NLP tasks and do not match
well with the task of relevance ranking. In this paper, we try to use a large-
scale user click log to pre-train the initial representations and then fine-
tune the model on expert-labeled data. From the user-system view of relevance,
the work can be explained as a study of combining these two views in one
model. See also the experimental settings in (Dai and Callan, 2019; Qiao et
al., 2019).

##  3\. Pre-Rank: Our Approach

The proposed framework, called Pre-Rank, consists of two stages: pre-training
stage and fine-tuning stage, in which the model learns the relevance from
user’s and system’s view respectively. The overall structure can be seen in
Figure 1.

The goal of the pre-training stage is to make the model incorporate specific
knowledge of the search domain. In this stage, we used click log to generate
supervision signals, because those data can be massively and cheaply obtained,
and can reflect the implicit relevance from the user’s view, although such
type of signal has some biases problem. We considered raw text feature in this
stage because it is a common feature across different ranking datasets.

With a good initialization of the model, the fine-tuning stage aims to
directly optimize the target task. Different from the pre-training stage, the
fine-tuning stage is suitable to consider dataset dependent features and use
expert-labeled corpus as supervision signals. Though it is costly and time-
consuming to construct large-scale handcrafted features and expert-labeled
corpus, the clean relevance signals make them valuable to adjust the biases
and relief the issues of the pre-trained models. Specifically, we used the
combination of the learned features and handcrafted features to represent the
query-document pairs and fine-tune the extended ranking model on the expert-
labeled corpus.

Figure 1. The pre-training (a) and fine-tuning (b) paradigm of Pre-Rank.

###  3.1. Modeling User’s View of Relevance with Pre-train

In the pre-training stage, it is supposed that we are given a set of search
activities from the real users (e.g., large-scale click log):

(1)
$\mathbb{D}^{c}=\left\\{(q^{c},\mathbf{d}^{c}=(d_{1}^{c},d_{2}^{c}\cdots,d_{m}^{c}),\mathbf{c}=(c_{1},c_{2}\cdots,c_{m}))\right\\},$

where $q^{c}=\left\\{w_{1},\cdots,w_{|q^{c}|}\right\\}$ is a query inputted by
real users, where $w_{i}$ denotes the $i$-th word in the query. Given the
query $q^{c}$, an IR system will retrieve a set of documents $\mathbf{d}^{c}$
presented to the users, where $d_{i}^{c}$ denotes the $i$-th document in
$\mathbf{d}^{c}$. $d_{i}^{c}=\left\\{v_{1},\cdots,v_{|d_{i}^{c}|}\right\\}$
also consists of a list of words and $v_{j}$ denotes the $j$-th word in the
document. Let $\mathbf{c}$ be the list of click signals associated with
$\mathbf{d}^{c}$ where $c_{i}\in\\{0,1\\}$ denotes whether the $i$-th document
is clicked by the user where $c_{i}=1$ if the user clicked $d^{c}_{i}$ and 0
otherwise.

The pre-training stage aims to learn a ranking model $M^{pre}$ using a pre-
training algorithm $\mathcal{A}^{pre}$, based on the click log
$\mathbb{D}^{c}$:

$M^{pre}\leftarrow\mathcal{A}^{pre}(\mathbb{D}^{c}).$

Usually, the model $M^{pre}$ contains an encoder layer $E^{pre}(\cdot)$ and a
prediction layer $P^{pre}(\cdot)$. The encoder layer aims to generate a vector
that encodes the interactions between query and documents. The prediction
layer aims to predict the possibility of user clicks on the retrieved
documents given the query.

The training algorithm $\mathcal{A}^{pre}$ consists of two steps: first
optimizes the masked language modeling (MLM) objective and then optimizes the
click prediction objective. The MLM objective, proposed in (Taylor, 1953) for
NLP, aims to build the contextual representations for queries and documents,
the loss denotes as $\mathcal{L}_{MLM}$. Specifically, MLM masks out some
tokens from input and then trains the model to predict the masked ones from
the rest of the sentence. MLM has been proven to build good contextual
representations in many NLP tasks. Following the existing practices (Devlin et
al., 2018), the MLM loss is defined as

(2) $\mathcal{L}_{MLM}=-\sum_{\hat{x}\in m(\mathbf{x})}\log
p\left(\hat{x}\mid\mathbf{x}_{\backslash m(\mathbf{x})}\right),$

where $\mathbf{x}$ is the input sentences, $m(\mathbf{x})$ are the randomly
masked words from $\mathbf{x}$, $\mathbf{x}_{\backslash m(\mathbf{x})}$
represent the rest of words from $\mathbf{x}$, and $p$ is the prediction
probability of the masked word $\hat{x}$ (Devlin et al., 2018).

The click prediction objective models the probability of whether a user will
click on the document from $\textbf{d}^{c}$ when entering query $q^{c}$, it
reflects the relevance between $q^{c}$ and $\textbf{d}^{c}$ from the user’s
perspective. The loss denotes as $\mathcal{L}_{click}$ aims to minimize the
differences between the click predictions and the real user clicks:

(3)
$\mathcal{L}_{click}=\sum_{(q^{c},\mathbf{d}^{c},\mathbf{c})\in\mathbb{D}^{c}}\ell^{pre}(P^{pre}(E^{pre}(q^{c},\mathbf{d}^{c})),\mathbf{c}),$

where $\ell^{pre}$ is the query-level loss at pre-training stage.

Through optimizing the pre-training objectives, the pre-trained model
$M^{pre}$ is able to reflect the implicit relevance between query and
documents as well as the contextual interaction between query and documents.
Noisy and large-scale are two major properties in this stage. Click-through
data contains a lot of noise for the sake of positional bias and explosion
bias. The noise in the inputs and supervision signals makes it hard to model
the relevance, while large-scale data make it possible to involve relevance
related information in the model. In the following fine-tuning stage, the
expert-labeled data are used to alleviate these issues.

###  3.2. Modeling System’s View of Relevance with Fine-tune

In the fine-tuning stage, we are given a set of expert-labeled data
$\mathbb{D}^{l}$ as the training corpus:

(4)
$\mathbb{D}^{l}=\left\\{(q^{l},\mathbf{d}^{l}=(d_{1}^{l},d_{2}^{l}\cdots,d_{m}^{l}),\mathbf{l}=(l_{1},l_{2},\cdots,l_{m}))\right\\},$

where $q^{l}$ is the given query and $\textbf{d}^{l}$ is the document list
associate with query $q^{l}$, where $d_{i}^{l}$ is the $i$-th document in
$\mathbf{d}^{l}$. Let $\mathbf{l}$ be the list of relevance labels associated
with $\mathbf{d}^{l}$ where $l_{i}\in\\{0,1,\cdots,k\\}$ denotes how the
$i$-th document relevance to the query judge by expert and $k$ varies on
different dataset. Moreover,
$q^{l}=\left\\{w_{1},\cdots,w_{|q^{l}|}\right\\}$,$d_{i}^{l}=\left\\{v_{1},\cdots,v_{|d_{i}^{l}|}\right\\}$
representing respectively as the raw text of query and document, $w_{i}$ and
$v_{i}$ denotes the $i$-th word in $q^{l}$ and $d_{i}^{l}$.

During the fine-tuning stage, we aim to fine-tune the model $M^{pre}$ using
algorithm $\mathcal{A}^{fine}$ with the expert-labeled data $\mathbb{D}^{l}$:

$M^{fine}\leftarrow\mathcal{A}^{fine}(\mathbb{D}^{l},M^{pre}).$

Similar to the pre-training model, $M^{fine}$ also contains an encoder layer
$E^{fine}(\cdot)$ and a prediction layer $P^{fine}(\cdot)$. The encoder layer
$E^{fine}$ shares the same structure with $E^{pre}$ and is initialized by the
pre-trained parameters in $E^{pre}$. During fine-tuning, the parameters in
$E^{fine}$ will be further tuned on the labeled data, takes $q^{l}$ and
$\textbf{d}^{l}$ as input and the output is denoted as $\mathbf{h}^{fine}$,
$\mathbf{h}^{fine}=E^{fine}(q^{l},\textbf{d}^{l})$. As for $P^{fine}$, the
network structure is an extension of $P^{pre}$, by adding a wide branch for
involving the handcrafted learning-to-rank features. The parameters in
$P^{fine}$ are randomly initialized and tuned in the fine-tuning stage.

The training algorithm $\mathcal{A}^{fine}$ aims to minimize the loss between
prediction and human label $\mathbf{l}$, to get the final ranking list for
labeled data:

(5)
$\mathcal{L}_{label}=\sum_{(q^{l},\mathbf{d}^{l},\mathbf{l})\in\mathbb{D}^{l}}\ell^{fine}\left(P^{fine}\left([\mathbf{h}^{fine},\psi(q^{l},\mathbf{d}^{l})]\right),\mathbf{l}\right),$

where $\ell^{fine}$ is the query-level loss at the fine-tuning stage,
$\psi(q^{l},\mathbf{d}^{l})$ outputs the handcrafted learning-to-rank features
for the inputted query and document, and $[\cdot,\cdot]$ denotes the
concatenation of the inputted vectors.

During the fine-tuning stage, we use unbiased datasets labeled by experts that
model relevance from the system’s view, which can relieves the noise and bias
issues with the click log. Clean but small-scale are the main properties in
this stage. We use the original features in the benchmark datasets if it is
provided, if not, we handcraft ourselves. With the training algorithm
$\mathcal{A}^{fine}$, model $M^{fine}$ can predict the explicit relevance
between query and documents and also leverage the handcrafted features.

Figure 2. Implementation of Pre-Rank with BERT as the underlying model.

##  4\. Implementations

Pre-Rank is a general ranking framework. In principle, it can be used to
improve different ranking methods, by using the method as its underlying
ranking model. In this section, we penetrate into the details of two Pre-Rank
implementations which use BERT and SetRank as the underlying ranker, denote as
Pre-Rank (BERT) and Pre-Rank (SetRank) respectively.

###  4.1. Implementation with BERT

BERT is a language representation model proposed by Devlin et al. (2018). It
pre-trains deep bidirectional representations on BooksCorpus (Zhu et al.,
2015) and English Wikipedia by using the pre-training objectives of masked
language model (MLM) and next sentence prediction. The first token of every
input sequence to BERT is always a special classification token ([CLS]). To
fine-tune on downstream tasks, the final hidden vector corresponding to the
first input token ([CLS]) is feed into a classification layer to get the final
prediction. In this paper, we use the off-the-shelf BERT to conduct the pre-
training and fine-tuning. The overall architecture of implementing Pre-Rank
with BERT is shown in Figure 2.

####  4.1.1. Pre-training Stage

The pre-training stage is shown in Figure 2(a). During this stage, the model
takes the click log as input. Since BERT is a pair inputted model, for
adaptation, we use $(q_{i}^{c},d_{i}^{c},c_{i})\in\mathbb{D}^{c}$ as one
training instance. As for the encoder layer $E^{pre}$ in BERT, the input is
the concatenation of query tokens and the clicked document tokens, with
special delimiting tokens i.e., $[CLS]+q_{i}^{c}+[SEP]+d_{i}^{c}+[SEP].$ The
tokens go through several layers of transformers to get fully interaction
between query and document. Finally, the output embedding vector of the [CLS]
token, denoted as $\mathbf{h}^{pre}_{i}$, is used as a representation for the
entire query-document pair, which is obtained by

$\mathbf{h}^{pre}_{i}=E^{pre}([CLS]+q_{i}^{c}+[SEP]+d_{i}^{c}+[SEP]).$

$\mathbf{h}^{pre}_{i}$ is then feed into the prediction layer $P^{pre}$, which
is a multi-layer perception (MLP) to predict the possibility of click. Cross-
entropy loss is used here as the learning objective. Therefore, Eq. (3) can be
written as

(6)
$\mathcal{L}_{click}=\sum_{(q^{c}_{i},d_{i}^{c},c_{i})\in\mathbb{D}^{c}_{BERT}}\mathrm{CE}(P^{pre}(E^{pre}(q_{i}^{c},d_{i}^{c})),c_{i}),$

where CE denotes the cross-entropy loss function.

During the pre-training stage, the model is first initialized with the pre-
trained BERT${}_{\textrm{BASE}}$’s parameters to leverage the language model,
while the prediction layer is learned from scratch. The pre-training stage
first optimizes the mask language model loss in Eq. (2), and thereafter
optimizes the click loss in Eq. (6).

####  4.1.2. Fine-tuning Stage

The fine-tuning stage is shown in Figure 2(b). Similar to that of in the pre-
training stage, we use $(q_{i}^{l},d_{i}^{l},l_{i})\in\mathbb{D}^{l}$ as one
training instance. As for the encoder $E^{fine}$ in $M^{fine}$, we initialize
the parameters with the pre-trained encoder $E^{pre}$ from first stage, and
fine-tune it with labeled data. Following the pre-training stage, the input is
the concatenated query and documents tokens:
$[CLS]+q_{i}^{l}+[SEP]+d_{i}^{l}+[SEP]$. The tokens go through several layers
of transformers to get fully interaction, and achieve the representation
corresponding to the [CLS] token:

$\mathbf{h}^{fine}_{i}=E^{fine}([CLS]+q_{i}^{l}+[SEP]+d_{i}^{l}+[SEP]).$

The output embedding of the [CLS] token $\mathbf{h}^{fine}_{i}$ is
concatenated with the handcrafted learning-to-rank features
$\psi(q_{i}^{l},d_{i}^{l})$, and then feed into the predictor layer to predict
the possibility of relevance. Please note that to involve the handcrafted
features, the predictor becomes a wide and deep model architecture and the
extended wide part corresponds to the learning-to-rank features. To learning
the parameters, the cross-entropy loss between predictions and human labels is
constructed and optimized.

(7)
$\mathcal{L}_{label}=\sum_{(q^{l}_{i},d_{i}^{l},l_{i})\in\mathbb{D}^{l}_{BERT}}\mathrm{CE}\left(P^{fine}\left([\mathbf{h}^{fine}_{i},\psi(q_{i}^{l},d_{i}^{l})]\right),l_{i}\right).$

###  4.2. Implementation with SetRank

In this section, we first introduce SetRank and then describe the details of
implementation.

SetRank is a listwise ranking model described by Pang et al. (2020). It is a
permutation-invariant ranking model defined on document sets of any size. The
architecture of SetRank contains three layers, representation layer, encoding
layer and ranking layer. The representation layer generates representations of
query-document pairs separately with its original learning-to-rank features
and ordinal embeddings. The encoding layer jointly processes the documents
with multiple sub-layers of multi-head self-attention blocks (MSAB). The final
ranking layer calculates the scores and sorts the documents.

####  4.2.1. Pre-training Stage

The pre-training stage is shown in Figure 3(a). During the pre-training stage,
the model takes the click log $\mathbb{D}^{c}$ as the training data, as
denoted in Eq. (1). $M^{pre}$ also contains an encoder layer $E^{pre}$ and a
prediction layer $P^{pre}$. As for the encoder layer, we use the contextual
query and document presentations generate by BERT. Given a query $q^{c}$ and
its associated document set
$\mathbf{d}^{c}=(d_{1}^{c},d_{2}^{c}\cdots,d_{m}^{c})$, each of the document
$d_{i}^{c}$ in $\mathbf{d}^{c}$ can be represented as a feature vector, the
output embedding of the [CLS] token:

$\textbf{h}_{i}^{pre}=E^{pre}([CLS]+q^{c}+[SEP]+d_{i}^{c}+[SEP]).$

all the $(q^{c},d_{i}^{c})$ pair in $\mathbf{d}^{c}$’s representation
$\textbf{h}_{i}^{pre}$ form a matrix
$\mathbf{H}^{pre}=\left[\textbf{h}_{1}^{pre},\textbf{h}_{2}^{pre},\cdots,\textbf{h}_{m}^{pre}\right]$,
is then feed into the prediction layer $P^{pre}$, the core architecture of
SetRank, several layers of a Multi-head self attention block (MSAB) and a row-
wise feed-forward network (rFF) to projects each document representation into
one real value as the corresponding click score.

Algorithm $\mathcal{A}^{pre}$ aim to minimize the loss between prediction and
click. Due to SetRank is a list ranking approach, list wise loss is used here
to optimize the parameters. That is, Eq. (3) can be written as:

(8)
$\mathcal{L}_{click}=\sum_{(q^{c},\mathbf{d}^{c},\mathbf{c})\in\mathbb{D}^{c}}\mathrm{CE_{list}}(P^{pre}(E^{pre}(q^{c},\mathbf{d}^{c}),\mathbf{c}),$

where $\mathrm{CE_{list}}$ denotes the list-wise cross-entropy loss function.
The encoder $E^{pre}$ is initialized with a pre-trained BERT model to leverage
the pre-trained language model, while the parameters in $P^{pre}$ are learned
from the scratch.

####  4.2.2. Fine-tuning Stage

As shown in Figure 3(b), in the fine-tuning stage, for each query $q^{l}$ a
list of documents $\mathbf{d}^{l}=(d_{1}^{l},d_{2}^{l}\cdots,d_{m}^{l})$ is
given, as denoted in Eq. (4). Similar to that of in the pre-training stage,
each $(q^{l},d_{i}^{l})$ pair is represent with BERT:

$\textbf{h}^{fine}_{i}=E^{fine}([CLS]+q^{l}+[SEP]+d_{i}^{l}+[SEP]).$

All the $(q^{l},d^{l}_{i})$ pair in $\textbf{d}^{l}$’s representation
$\textbf{h}_{i}^{fine}$ form a matrix on the contextual embedding vectors
$\textbf{H}^{fine}=\left[\textbf{h}_{1}^{fine},\cdots,\textbf{h}_{m}^{fine}\right]$.

Meanwhile, the learning-to-rank feature vectors $\psi(q^{l},d_{i}^{l})$ for
each $(q^{l},d^{l}_{i})$ pair are also respectively generated, forming a
feature matrix
$\mathbf{\Psi}=[\psi(q^{l},d_{1}^{l}),\cdots,\psi(q^{l},d_{m}^{l})]$. Both
$\mathbf{H}^{fine}$ and $\mathbf{\Psi}$ are inputted as the final result of
representation layer: $\mathbf{X}=[\mathbf{H}^{fine},\mathbf{\Psi}],$ where
$[\cdot,\cdot]$ denotes concatenation of $\mathbf{H}^{fine}$ and learning-to-
rank features $\mathbf{\Psi}$. Then $\mathbf{X}$ is feed into SetRank’s MSAB
blocks and rFF to get the final scores. The ranking can be achieved by sorting
the documents according to the scores.

The algorithm $\mathcal{A}^{fine}$ aims to minimize the loss between the list
prediction and human label $\mathbf{l}$ of the query-document list:

(9)
$\mathcal{L}_{label}=\sum_{(q^{l},\mathbf{d}^{l},\mathbf{l})\in\mathbb{D}^{l}}\mathrm{CE_{list}}\left(P^{fine}\left({[\mathbf{H}^{fine},\mathbf{\Psi}]}\right),\mathbf{l}\right).$

Figure 3. Implementation of Pre-Rank using SetRank as the underlying model.

##  5\. EXPERIMENTS

We conducted experiments to verify the effectiveness of Pre-Rank on three
publicly available IR benchmarks.

###  5.1. Experimental Settings

In this section, we describe the experimental settings, includes datasets,
baselines, evaluation metrics, and experimental details.

####  5.1.1. Datasets

In all of the experiments, the ranking models were pre-trained on ORCAS
(Craswell et al., 2020a), a large-scale publicly available click log corpus.
ORCAS contains the real user clicks related to the documents used in the TREC
Deep Learning Track. The whole ORCAS corpus contains 1.4 million TREC DL URLs,
18 million connections to 10 million distinct queries after the aggregation
and filtering (for satisfying the $k$-anonymity requirements).

Table 1. Statistics of the datasets with human labels.  Dataset | Genre | #Queries | #Documents | #Labeled Q-D pairs
---|---|---|---|---
MQ2007 | .gov | 1,692 | 65,323 | 69,623
MQ2008 | .gov | 784 | 14,384 | 15,211
TREC19 | web | 0.37M | 3.2M | 388,464

The pre-trained models were then fine-tuned on downstream retrieval datasets
with human annotated relevance labels. We conducted fine-tune on three
datasets MQ2007, MQ2008, and TREC2019, which are published in LETOR 4.0 (Qin
et al., 2010) and TREC19 (Craswell et al., 2020b). Table 1 lists some
statistics of the three datasets. MQ2007 contains 1,692 queries 65,323
documents and 69,623 expert-labeled query-document pairs. MQ2008 contains 784
queries 14,384 documents and 15,211 expert-labeled query-document pairs. The
number of queries in MQ2008 is relatively small, making it insufficient to
learn deep learning model. In the experiments, we followed the practices in
(Pang et al., 2017) and combined the training set of MQ2007 with that of
MQ2008 while keeping the validation and test sets unchanged. We still denoted
the new dataset as MQ2008. MQ2007 and MQ2008 in total contain 69,623 and
84,834 query documents pairs. The two datasets consist of not only the raw
query and document texts, but also 46 dimensions handcrafted features for each
query-document pair. TREC 2019 Deep Learning Track benchmark is a large-scale
ad-hoc retrieval dataset, which served two tasks: document retrieval and
passage retrieval. Both tasks have large training sets with human relevance
assessments, derived from MS MARCO (Bajaj et al., 2016). The document
retrieval task has a corpus of 3.2 million documents with 367 thousand
training queries, and 388,464 labeled query-document pairs. Please note TREC19
only provides the relevant documents for each query. In this paper, we
conducted experiments on the sub-task of “top-100 reranking” in document
retrieval.

####  5.1.2. Baselines and Evaluation Metrics

Several types of relevance ranking baselines, including the traditional
relevance ranking models, learning-to-rank models, the state-of-the-art neural
IR models, were selected in the experiments for comparisons:

  * •

BM25 (Robertson and Walker, 1994): a classical and widely used model for
relevance ranking;

  * •

AdaRank (Xu and Li, 2007): a traditional learning-to-rank model that aims to
directly optimize the performance measure based on boosting;

  * •

LambdaMart (Burges, 2010): a widely used traditional listwise learning-to-rank
model based on gradient boosting;

  * •

BERT (Devlin et al., 2018): a state-of-the-art pre-trained language model
which uses MLM and next sentence prediction (NSP) to pre-train the contextual
language representation and sentence pair representation. The pre-trained
representations can be fine-tuned to a number of downstream tasks, including
relevance ranking.

  * •

SetRank (Pang et al., 2020): a permutation-invariant neural IR model which has
the ability to model cross-document interactions so as to capture local
context information under a query;

  * •

PROP(Ma et al., 2021): a recently proposed pre-trained IR model that tailored
the training object during pre-training. In practice, PROP was pre-trained on
Wikipedia and MS MARCO Document Ranking dataset, denoted as PROPwiki and
PROPMARCO, respectively.

To evaluate the performances of Pre-Rank and the baselines on MQ2007 and
MQ2008, we followed the original data partitions provided by LETRO4.0 and
conducted 5-fold cross validation. The average results were reported. As for
the evaluation measures, we used the Precision at 10 (P@10), Normalized
Discounted Cumulative Gain at 10 (NDCG@10), and Mean Average Precision (MAP)
(Robertson, 2000).

To evaluate TREC19, we tested the performances of the proposed Pre-Rank and
baselines on the dev set. As for the evaluation measures, we used Recall at 10
(Recall@10) and Mean Reciprocal Rank at 10 (MRR@10).

####  5.1.3. Experimental Details

During the pre-training procedures, to incorporate relevance assessment from
the real users into the ranking model, we performed the pre-training on the
ORCAS dataset. ORCAS collected large-scale real user queries, documents, and
clicks from a search engine. Like the pre-training of BERT, we also performed
two tasks on ORCAS dataset, namely mask language model and click prediction.
The procedure of the mask language model task is identical to that of the
original BERT model. We randomly masked 15% tokens and made the model to
“restore” them. As for the click prediction, the positive instances are
derived from the click data of ORCAS, and the negative instances are randomly
selected from the top 100 documents of the corresponding query if users did
not click them. We combined the two tasks for training the pre-trained model
by first running the mask language model task and then running the click
prediction task.

During the fine-tuning procedures, we used the pre-trained models to re-rank
the candidate documents provided by the datasets.The fine-tuning made use of
the raw text of queries, documents, and the handcrafted features provided in
the dataset. For MQ2007 and MQ2008, the learning rate was tuned between
$10^{-5}$ to $2\times 10^{-5}$. For fine-tuning Pre-Rank(BERT), we randomly
selected 20 documents from the datasets for each query. The input was
truncated to 256. For fine-tuning Pre-Ran(SetRank), the list sizes were set to
20 and the input of raw texts were also truncated to 256 for fair comparisons.
When conducting the tests on the fine-tuned model, all documents are inputted
to get the final ranks. For TREC19 dataset, we randomly select 40 documents
per query from the top100 retrieval results officially provided by this
dataset, and other settings as the same as MQ2007 and MQ2008 datasets.

Pre-Rank utilized handcrafted relevance features at the fine-tuning stage.
These features are combined with the pre-trained representations for making
accurate relevance predictions. On MQ2007 and MQ2008, we directly used the
learning-to-rank features provided by the LETOR4.0 corpus. Each query-document
pair is represented as a 46-dimensional real vector. On TREC19, we extracted
21-dimensional features for each query-document pair. These features reflect
the information of the query-level, document-level, and the interaction
between the query and document, including query length, BM25 score, tf-idf,
query-document matching with bi-grams, tri-grams similarity, word2vec
similarity, etc.

For the traditional IR models and learning-to-rank models (i.e., BM25,
AdaRank, and LambdaMart), we used the implementations shared in (Pang et al.,
2017). The implementations of BERT and Pre-Rank are based on the popular
Transformers library 111https://github.com/huggingface/transformers. As for
BERT, the encoder layer of Pre-Rank(BERT) and the encoder layer of Pre-
Rank(SetRank) are initialized by BERT${}_{\textrm{BASE}}$’s checkpoint release
by Google 222https://github.com/google-research/bert.

###  5.2. Experimental results

Table 2. Ranking performance comparisons between Pre-Rank and the baselines on three benchmarks with human labels. ‘*’ and ‘$\dagger$’ respectively indicate the improvements over BERT and SetRank are statistically significant ($p$-value$<0.05$). Model | MQ2007 | MQ2008 | TREC19
---|---|---|---
P@10 | NDCG@10 | MAP | P@10 | NDCG@10 | MAP | MRR@10 | Recall@10
BM25 | 0.366 | 0.414 | 0.450 | 0.245 | 0.220 | 0.465 | 0.234 | 0.473
AdaRank | 0.373 | 0.439 | 0.460 | 0.247 | 0.222 | 0.468 | 0.271 | 0.533
LambdaMart | 0.384 | 0.446 | 0.468 | 0.251 | 0.231 | 0.478 | 0.273 | 0.529
BERT | 0.418 | 0.495 | 0.500 | 0.252 | 0.247 | 0.502 | 0.370 | 0.632
SetRank | 0.418 | 0.497 | 0.498 | 0.255 | 0.249 | 0.498 | 0.383 | 0.638
PROPwiki | 0.432 | 0.523 | - | 0.267 | 0.262 | - | 0.360 | 0.622
PROPMARCO | 0.430 | 0.522 | - | 0.269 | 0.266 | - | 0.360 | 0.628
Pre-Rank(BERT) | 0.430∗ | 0.520∗ | 0.521∗ | 0.255∗ | 0.252∗ | 0.514∗ | 0.376∗ | 0.644∗
Pre-Rank(SetRank) | 0436† | 0.526† | 0.525† | 0.258† | 0.258† | 0.521† | 0.388† | 0.648†

Table 2 reports the experimental results of the proposed Pre-Rank and all of
the baselines on the downstream datasets of MQ2007, MQ2008, and TREC19. The
baselines’ results mainly follow (Pang et al., 2017). For PROPwiki and
PROPMARCO we used the numbers reported in (Ma et al., 2021) thus have no
results on TREC19 and in terms of MAP on MQ2007 and MQ2008. The results on
TREC19 of PROP are based on the released models in (Ma et al., 2021).

By comparing Pre-Rank (BERT) with BERT and Pre-Rank (SetRank) with SetRank, we
can see that Pre-Rank outperformed the underlying neural IR model of BERT and
SetRank with large margin. We conducted significant tests on the improvements.
The results indicate that all of the improvements over the raw underlying
ranking models are significant ($p$-value $<0.05$). Considering that BERT and
SetRank are already very strong neural IR baselines, the results indicate the
effectiveness of the Pre-Rank framework. It also verified the effectiveness of
Pre-Rank’s approach that involving both the user’s view and the system’s view
of relevance signals in one model using the pre-training and fine-tuning
paradigm. Comparing the two implementations, we found that Pre-Rank (SetRank)
performed better than Pre-Rank (BERT), verified the advantages of the
permutation-invariant ranking model.

From the results, we also found that: (1) the neural IR models of BERT and
SetRank can obtain better results than traditional models such as BM25,
AdaRank and LambdaMART, indicating that automatically learned features can
capture more relevance signals other than the handcrafted learning-to-rank
features;

(2) the pre-trained ranking model PROP (Ma et al., 2021) improved BERT on
MQ2007 and MQ2008 by a huge margin, verified the power of the pre-training and
fine-tuning paradigm; (3) the proposed Pre-Rank (SetRank) outperformed PROP in
most cases, verified the effectiveness and necessity of modeling the relevance
from both user’s view and system’s view.

###  5.3. Discussions

We conducted experiments to show the reasons that our approaches outperformed
the baselines and impacts of different parameter settings.

####  5.3.1. Impact of handcrafted features at the fine-tuning stage

One of the unique characteristics of Pre-Rank is its ability to involve both
the pre-trained representations and handcrafted learning-to-rank features. In
the experiments, we tested the effects of the handcrafted features.
Specifically, we tested the ranking accuracies of Pre-Rank (BERT) and Pre-Rank
(SetRank) when the wide parts (which are responsible for injecting learning-
to-rank features) were removed during the fine-tuning. In the experiments, all
of the inputs were under the same setting for each dataset. Table 3 lists the
experimental results. In the table, the Pre-Rank (BERT) and Pre-Rank (SetRank)
versions with and without the handcrafted features were denoted as “/w ltrFtr”
and “w/o ltrFtr”. From the results, we can see that in most cases combined
with the automatically pre-trained features, the handcrafted features can
further improve the ranking accuracy under the Pre-Rank framework. The results
verified that though the powerfulness of the pre-trained representations, the
expert knowledge encoded in the traditional learning-to-rank features (one
aspect of the system’s view on relevance) is still valuable for relevance
ranking.

Table 3. Impacts of the handcrafted learning-to-rank features at the fine-tuningstage.  |  | MQ2007 | MQ2008 | TREC19
---|---|---|---|---
| Settings | P@10 | NDCG@10 | MAP | P@10 | NDCG@10 | MAP | MRR@10 | Recall@10
Pre-Rank (BERT) | w/o LtrFtr | 0.430 | 0.516 | 0.518 | 0.256 | 0.252 | 0.513 | 0.375 | 0.642
w/ LtrFtr | 0.430 | 0.520 | 0.521 | 0.255 | 0.252 | 0.514 | 0.376 | 0.644
Pre-Rank (SetRank) | w/o LtrFtr | 0.433 | 0.521 | 0.520 | 0.256 | 0.252 | 0.510 | 0.389 | 0.643
w/ LtrFtr | 0.436 | 0.526 | 0.525 | 0.258 | 0.258 | 0.521 | 0.388 | 0.648

####  5.3.2. Selection of the learning objectives at the pre-training stage

During the pre-training stage, Pre-Rank utilized two learning objectives: the
traditional mask language model (MLM) objective based on the raw texts, and
the click prediction objective based on the real user’s click activities. We
conducted experiments to investigate the impacts of these two learning
objectives in Pre-Rank. Specifically, we removed one of the objectives and
pre-trained the initial representations with the remained objective. The fine-
tuning stage is kept unchanged. Table 4 shows the performances of the
variations of Pre-Rank (BERT) and Pre-Rank (SetRank) in which the pre-training
objectives are adjusted. In the table, we denote the experiments pre-training
with MLM objective only, pre-training with click prediction objective only,
and pre-training with MLM and click prediction as “w/MLM”, “w/click”, and
“w/MLM + Click”, respectively. From the results reported in Table 4, we found
that Pre-Rank models pre-trained with click prediction objective only
(“w/Click”) performed similarly to the full version: pre-training with both
MLM and click prediction objectives (“w/MLM + Click”). On the other hand, the
model that pre-trained with MLM objective only (“w/MLM”) performed much worse.
The phenomenon can be observed in both the experiments with Pre-Rank (BERT)
and Pre-Rank (SetRank). The results clearly indicate that: (1) the activities
from the real users (user’s view of relevance) are critical relevance signals
for relevance ranking; (2) through existing studies have proven MLM to be an
effective pre-training objective for many NLP related downstream tasks, it
seems that MLM is not an optimal selection for relevance ranking. Both the
user’s view and the system’s view on relevance also rarely touch the NLP
knowledge of the texts in the queries and documents, especially the knowledge
that can be derived from the mask language modeling. Therefore, it is easy to
understand why very limited improvements can be observed after adding the MLM
objective to the click prediction objective.

Table 4. Impacts of the pre-training objectives in Pre-Rank.  |  | MQ2007 | MQ2008 | TREC19
---|---|---|---|---
| Settings | P@10 | NDCG@10 | MAP | P@10 | NDCG@10 | MAP | MRR@10 | Recall@10
Pre-Rank(BERT) | w/MLM | 0.424 | 0.502 | 0.504 | 0.253 | 0.246 | 0.502 | 0.371 | 0.636
w/MLM+Click | 0.430 | 0.520 | 0.521 | 0.255 | 0.252 | 0.514 | 0.376 | 0.644
w/Click | 0.432 | 0.521 | 0.518 | 0.255 | 0.252 | 0.507 | 0.373 | 0.641
Pre-Rank(SetRank) | w/MLM | 0.418 | 0.487 | 0.496 | 0.256 | 0.253 | 0.515 | 0.387 | 0.644
w/MLM+Click | 0.436 | 0.526 | 0.525 | 0.258 | 0.258 | 0.521 | 0.388 | 0.648
w/Click | 0.436 | 0.526 | 0.525 | 0.259 | 0.258 | 0.523 | 0.387 | 0.645

####  5.3.3. Ability to improve long queries

We also conducted experiments to show on which kinds of queries our approaches
can perform well. Specifically, we categorized the queries of MQ2007 into
different query groups according to the lengths of the queries. For each query
group, we tested the performance improvements of Pre-Rank (BERT) over the
underlying ranking model of BERT, in terms of NDCG@10, and the performance
improvements of Pre-Rank (SetRank) over the underlying ranking model of
SetRank in terms of NDCG@10. Figure 4 showed the NDCG@10 improvements w.r.t.
the query groups. The results are the average values over the 5 folds’ test
set. We can see that there is a trend that Pre-Rank can achieve more
improvements on the query groups with long query lengths (e.g., the group
whose query length is 10). This is particularly obvious in the case of Pre-
Rank (BERT). One reason for the phenomenon is that by pre-training on raw text
from the user’s view, Pre-Rank models more semantic relevance and therefore
improves when the query is longer and requires more semantic information.

Figure 4. NDCG@10 improvements of Pre-Rank (BERT) and Pre-Rank(SetRank) over
the underlying ranking models w.r.t. different query lengths on MQ2007.

####  5.3.4. Effects of combining relevance signals from user’s view and
system’s view

We conducted experiments to test the effects of combining the user’s view and
the system’s view on relevance in learning ranking models. Specifically, we
trained SetRank models based on three settings of the training data: (1)
trained on the ORCAS dataset that only contains clicks from real users
(denoted as “Click-Only”); (2) trained on the TREC19 dataset that only
contains relevance labels by the experts (denoted as “Label-Only”); (3) first
trained on the ORCAS click log, and then continued the training on TREC19
expert-labeled dataset (denoted as “Click + Label”). We tested the
performances on TREC19’s dev set and Figure 5 shows the performance curves of
these three settings w.r.t. training steps in terms of MRR@10. From the
results, we can conclude that (1) the large-scale click log (relevance signals
from the user’s view) is effective in training ranking models in the early
stages. However, as the training goes on, the noise and biases in the click
log hurt the training procedure and hinder the further improvements; (2) the
limited but clean labeled data (relevance signals from the system’s view) is
useful in training and the model’s performance improved steadily until
convergence; (3) the ranking performances can be further improved when the
model trained on ORCAS was continually trained on TREC19 data. This is because
these two views are complementary: the clean and unbiased labeled data in
TREC19 relieves the noise and bias issues with the click log, while the large-
scale click log provides massive relevance signals to overcome the scale
limitation.

Figure 5. Performance curves of SetRank models when respectively trained with
user clicks (ORCAS), with labeled data (TREC19), and first with user clicks
and then with labeled data.

##  6\. CONCLUSION

This paper proposes a novel relevance ranking framework under the pre-training
and fine-tuning paradigm, for modeling the relevance information from both the
user’s view and the system’s view. The framework, called Pre-Rank, first pre-
trains the initial representations on real user’s search activities (e.g.,
click log) and then fine-tunes the ranking model on expert-labeled data with
handcrafted learning-to-rank features. We implemented two versions of Pre-Rank
which used BERT and SetRank as its underlying ranking model, respectively.
Experimental results on publicly available benchmarks showed that after pre-
training on large-scale user activities data, both of the pre-trained Pre-Rank
versions can be well fine-tuned on three benchmarks and significantly enhanced
the ranking performances.

###### Acknowledgements.

This work was funded by the National Key R&D Program of China
(2019YFE0198200), the National Natural Science Foundation of China (No.
61872338, No. 61906180, No. 61832017), Beijing Academy of Artificial
Intelligence (BAAI2019ZD0305), and Beijing Outstanding Young Scientist Program
NO. BJJWZYJH012019100020098.

## References

  * (1)
  * Agichtein et al. (2006) Eugene Agichtein, Eric Brill, and Susan Dumais. 2006\.  Improving web search ranking by incorporating user behavior information. In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_. 19–26.
  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016\.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_ (2016).
  * Borlund (2003) Pia Borlund. 2003\.  The concept of relevance in IR.  _Journal of the American Society for information Science and Technology_ 54, 10 (2003), 913–925.
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent. In _Proceedings of the 22nd international conference on Machine learning_. 89–96.
  * Burges (2010) Christopher JC Burges. 2010\.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ 11, 23-581 (2010), 81.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020.  Pre-training tasks for embedding-based large-scale retrieval.  _arXiv preprint arXiv:2002.03932_ (2020).
  * Chen et al. (2020) Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2020\.  Bias and Debias in Recommender System: A Survey and Future Directions.  _arXiv preprint arXiv:2010.03240_ (2020).
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Craswell et al. (2020a) Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020a.  ORCAS: 20 Million Clicked Query-Document Pairs for Analyzing Search. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2983–2989.
  * Craswell et al. (2020b) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020b.  Overview of the trec 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_ (2020).
  * Craswell and Szummer (2007) Nick Craswell and Martin Szummer. 2007.  Random walks on the click graph. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 239–246.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for IR with contextual neural language modeling. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 985–988.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dou et al. (2008) Zhicheng Dou, Ruihua Song, Xiaojie Yuan, and Ji-Rong Wen. 2008\.  Are click-through data adequate for learning web search rankings?. In _Proceedings of the 17th ACM conference on Information and knowledge management_. 73–82.
  * Fox et al. (2005) Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005.  Evaluating implicit measures to improve web search.  _ACM Transactions on Information Systems (TOIS)_ 23, 2 (2005), 147–168.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016.  _Deep learning_. Vol. 1.  MIT press Cambridge.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016\.  A deep relevance matching model for ad-hoc retrieval. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 55–64.
  * Guo et al. (2020a) Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W Bruce Croft, and Xueqi Cheng. 2020a.  A deep look into neural ranking models for information retrieval.  _Information Processing & Management_ 57, 6 (2020), 102067\.
  * Guo et al. (2020b) Weiwei Guo, Xiaowei Liu, Sida Wang, Huiji Gao, Ananth Sankar, Zimeng Yang, Qi Guo, Liang Zhang, Bo Long, Bee-Chung Chen, et al. 2020b.  Detext: A deep text ranking framework with bert. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2509–2516.
  * Hjørland (2010) Birger Hjørland. 2010\.  The foundation of the concept of relevance.  _Journal of the american society for information science and technology_ 61, 2 (2010), 217–237.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Joachims (2002) Thorsten Joachims. 2002\.  Optimizing search engines using clickthrough data. In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_. 133–142.
  * Joachims et al. (2017a) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017a.  Accurately interpreting clickthrough data as implicit feedback. In _ACM SIGIR Forum_ , Vol. 51. Acm New York, NY, USA, 4–11.
  * Joachims et al. (2007) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007\.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search.  _ACM Transactions on Information Systems (TOIS)_ 25, 2 (2007), 7–es.
  * Joachims et al. (2017b) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017b.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. 781–789.
  * Li (2011) Hang Li. 2011.  Learning to rank for information retrieval and natural language processing.  _Synthesis lectures on human language technologies_ 4, 1 (2011), 1–113.
  * Liu (2011) Tie-Yan Liu. 2011\.  Learning to rank for information retrieval.  (2011).
  * Ma et al. (2021) Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2021\.  PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 283–291.
  * Mitra et al. (2018) Bhaskar Mitra, Nick Craswell, et al. 2018\.  _An introduction to neural information retrieval_.  Now Foundations and Trends.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _arXiv preprint arXiv:1901.04085_ (2019).
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  _arXiv preprint arXiv:2003.06713_ (2020).
  * Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016\.  Text matching as image recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Pang et al. (2017) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. 2017\.  Deeprank: A new deep architecture for relevance ranking in information retrieval. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 257–266.
  * Pang et al. (2020) Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. 2020\.  Setrank: Learning a permutation-invariant ranking model for information retrieval. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 499–508.
  * Qiao et al. (2019) Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019\.  Understanding the Behaviors of BERT in Ranking.  _arXiv preprint arXiv:1904.07531_ (2019).
  * Qin et al. (2010) Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. 2010\.  LETOR: A benchmark collection for research on learning to rank for information retrieval.  _Information Retrieval_ 13, 4 (2010), 346–374.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  (2018).
  * Radlinski and Joachims (2005) Filip Radlinski and Thorsten Joachims. 2005.  Query chains: learning to rank from implicit feedback. In _Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining_. 239–248.
  * Robertson (2000) Stephen Robertson. 2000\.  Evaluation in information retrieval. In _European Summer School on Information Retrieval_. Springer, 81–92.
  * Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR’94_. Springer, 232–241.
  * Saracevic (1975) Tefko Saracevic. 1975\.  Relevance: A review of and a framework for the thinking on the notion in information science.  _Journal of the American Society for information science_ 26, 6 (1975), 321–343.
  * Taylor (1953) Wilson L Taylor. 1953\.  “Cloze procedure”: A new tool for measuring readability.  _Journalism quarterly_ 30, 4 (1953), 415–433.
  * Xu et al. (2018) Jun Xu, Xiangnan He, and Hang Li. 2018.  Deep learning for matching in search and recommendation. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 1365–1368.
  * Xu and Li (2007) Jun Xu and Hang Li. 2007\.  Adarank: a boosting algorithm for information retrieval. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 391–398.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_ (2019).
  * Yuan et al. (2019) Bowen Yuan, Jui-Yang Hsia, Meng-Yuan Yang, Hong Zhu, Chih-Yao Chang, Zhenhua Dong, and Chih-Jen Lin. 2019.  Improving ad click prediction by considering non-displayed events. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 329–338.
  * Yuan et al. (2020) Bowen Yuan, Yaxu Liu, Jui-Yang Hsia, Zhenhua Dong, and Chih-Jen Lin. 2020.  Unbiased Ad click prediction for position-aware advertising systems. In _Fourteenth ACM Conference on Recommender Systems_. 368–377.
  * Yue et al. (2007) Yisong Yue, Thomas Finley, Filip Radlinski, and Thorsten Joachims. 2007.  A support vector method for optimizing average precision. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 271–278.
  * Zhang et al. (2021) Xinyu Zhang, Andrew Yates, and Jimmy Lin. 2021.  Comparing Score Aggregation Approaches for Document Retrieval with Pretrained Transformers. In _Advances in Information Retrieval_ , Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer International Publishing, Cham, 150–163.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_. 19–27.


# Lightweight high-resolution Subject Matting in the Real World

###### Abstract

Existing saliency object detection (SOD) methods struggle to satisfy fast
inference and accurate results simultaneously in high resolution scenes. They
are limited by the quality of public datasets and efficient network modules
for high-resolution images. To alleviate these issues, we propose to construct
a saliency object matting dataset HRSOM and a lightweight network PSUNet.
Considering efficient inference of mobile depolyment framework, we design a
symmetric pixel shuffle module and a lightweight module TRSU. Compared to 13
SOD methods, the proposed PSUNet has the best objective performance on the
high-resolution benchmark dataset. Evaluation results of objective assessment
are superior compared to U2Net that has 10 times of parameter amount of our
network. On Snapdragon 8 Gen 2 Mobile Platform, inference a single
640$\times$640 image only takes 113ms. And on the subjective assessment,
evaluation results are better than the industry benchmark IOS16 (Lift subject
from background).

Index Terms—  Segmentation, Matting, Light-weight network, High-resolution,
Mobile devices

##  1 Introduction

SOD is a technique for automatic identification and localization of the most
salient objects. SOD algorithms can be classified into traditional methods and
deep learning methods. Traditional methods mainly use low-level features such
as color, brightness, orientation, and contrast of an image to compute
saliency maps, but they often fail to deal with complex background and
multiple objects [1, 2, 3]. Deep learning methods utilize CNNs to learn high-
level semantic features to improve performance of SOD. One of the challenges
of SOD is to extract saliency maps with clear boundaries from high-resolution
images while controlling the computational cost. Practical SOD algorithms need
to balance between speed and accuracy, in order to adapt to various
application requirements and hardware conditions. In this regard, Wu et
al.[10], Lee et al.[11], Wang et al.[7] designed lightweight structures to
accelerate inference. Liu et al.[12], Wu et al.[13], Qin et al.[8, 14] used
multi-scale feature fusion to improve the quality of saliency maps. Qin et
al.[15] used coarse-to-fine framework to obtain finer results. Xie et al.[20]
, Kim et al.[16] designed high resolution modules to obtain accurate high
resolution saliency maps.

High-resolution SOD datasets [19, 20, 18] also contribute to the advancement
of SOD algorithms. They contain images of better quality and labels of higher
accuracy, compared to low-resolution SOD datasets. Most existing SOD methods
can achieve good performances on public datasets, but they are unable to apply
on real scenes. Firstly, most SOD datasets are low-resolution, which hinders
the algorithm to process high-resolution images. Secondly, binary labels
define SOD as a binary segmentation task of foreground and background. And
label edges of some datasets have jaggedness, leading to difficulty for the
algorithm to derive fine results. Lastly, saliency labels have large deviation
from subjective perception, which is not friendly for learning.

Our contributions are summarised as following. We propose a lightweight SOD
network PSUNet, which is designed with full considerations of mobile
adaptation to achieve a good balance between high resolution and efficiency.
We propose a symmetric pixel shuffle module to effectively enhance high-
resolution SOD results, at a very low cost of computation and parameter amount
increment. We construct a saliency object matting dataset HRSOM consisting of
13w+ high-quality and high-resolution images, based on almost all available
saliency detection datasets and self-collected datasets. We design a set of
training methods and subjective and objective measures for real-world
datasets, enabling the algorithm to robustly cope with open-world problems.

Fig. 1: Overall structure of PSUNet.

##  2 Method

###  2.1 PSUNet

Our proposed PSUNet adopts the UNet [17] architecture and a symmetric pixel
shuffle module (SPSM) for high-resolution scenarios. It greatly improves the
capability of processing high-resolution images with only tiny increment of
computation and parameter amount. Due to limited computation power on mobile
devices, network width has much larger impact on inference speed than depth.
Hence during the design of base module, we attempt to control the network
depth as much as possible while reducing computation cost of single
convolution. And we design a Tiny Residual Ublock (TRSU). To control the
network width, skip strategy takes the form of summation of feature maps.

####  2.1.1 Symmetrical Pixel Shuffle Module

PixelShuffle [9] is widely used in image super-resolution tasks. It is a very
efficient sampling method to obtain high-resolution feature maps from low-
resolution ones, through convolution and reorganization of multiple channels.
Inverse pixel shuffle process extracts high resolution image features and
preserves global features. For SOD task, global information is the key to
detect salient objects. At the same time, high-resolution input also retains
more detailed information. Compared to bilinear interpolation for
downsampling, inverse pixel shuffle has no loss on details, as demonstrated in
the later section of ablation study. Implementation of Symmetrical Pixel
Shuffle Module is shown in Fig. 1(b). In the upsampling process, a grid of
t$\times$t is constructed for each low-resolution pixel. Feature values at the
corresponding positions in t$\times$t feature maps are utilised to fill into
these small grids according to predefined rules. And downsampling process
conducts reorganization by following the same rules and fills the small grids
divided by each low-resolution pixel. In this process, the model can adjust
weights of the t$\times$t shuffle channels and continuously optimizes the
generated results, which can be expressed as below,

$\displaystyle\small N(C\times t\times t)HW$ $\displaystyle=$ $\displaystyle
NC(H\times t)(W\times t)$ (1)

where $N$ represents number of batches, $C$ represents number of channels, $H$
and $W$ represent height and width of feature map, $t$ represents sampling
ratio.

####  2.1.2 Encoder-Decoder

The overall structure of Encoder-Decoder is similar to U2Net [8] and consists
of the proposed lightweight TRSU (Tiny ReSidual Ublock) module. As shown in
Fig. 1(d), the proposed TRSU module also has the structure of UNet. After a
layer of 3$\times$3 convolution, feature maps are sampled to a smaller size by
a stack of multiple combinations of depth separable convolution and max
pooling. Then after several layers of depth separable convolution, resolution
is restored by the same number of combinations of upsampling and depth
separable convolution. The final output is derived after a layer of 3$\times$3
convolution. Skip connections in TRSU module are in the form of addition to
control the network width. MIDBLock (Fig. 1(c)) stacks pure depth-separable
convolution structures to extract deeper features. For TRSU module in the
decoder, all convolutions in Fig. 1(d) are replaced with depth-separable
convolutions. Finally, feature maps are fed into SPSM after a single layer of
convolution. Skip connections between the encoder and the decoder are also in
the form of addition.

###  2.2 HRSOM Dataset

We integrated RGB SOD low-resolution datasets DUTS-TR, HKU-IS, RGB SOD high-
resolution datasets UHRSD, HRSOD, DIS5K, RGB-D SOD datasets COME15K [6], and
matting datasets AM, P3M. To construct the integrated dataset, we collaborated
with data label experts to establish standards of saliency criteria for
practical applications. And we manually excluded images unsuitable for
labeling, especially for scenarios where multiple datasets behave
inconsistently. Real super-resolution (SR) algorithm Real-ESRGAN [5] was used
to increase resolution to more than 720 pixels on the short side. Bilinear
interpolation for upsampling was used to increase label resolution by the same
rate. Labels of all SOD datasets are refined using the pre-trained large
matting model. Fig. 2 shows the results of low resolution images before and
after processing. It can be seen that the image details are well preserved
after SR, and edge details of label results after matting are significantly
improved. Collected 6w+ data through self-acquisition and purchase are
manually labeled according to the established saliency criteria. Public data
after cleaning and processing and self-collected data after labeling are
integrated into the HRSOM dataset of 13w+ data in total.

Fig. 2: Image and label before and after processing.

###  2.3 Real scenes Training Strategy

####  2.3.1 Hybrid Loss

Considering three granularity levels of pixel, image block and full image, we
propose a hybrid Loss (Eq.4.) consisting of BCE Loss, SSIM loss and IOU loss.
BCE Loss (Eq.2) is a pixel-level loss function that compares the predicted
probability of each pixel with the ground truth label, and computes the
negative log-likelihood as results. It allows the model to learn whether each
pixel belongs to the salient object or not.

$\small\begin{split}\mathcal{L}_{bce}&=-\sum_{i=1}^{H}\sum_{j=1}^{W}[G_{ij}\log{(P_{ij})}+(1-G_{ij})\log{(1-P_{ij})}]\end{split}$
(2)

SSIM loss [21] characterizes structural similarity and is of image block
level. It calculates the differences between two image blocks in terms of
luminance, contrast, and structural similarity. It enables the model to learn
local structural features of salient objects, especially the edges. IOU loss
is an image-level loss function that calculates the degree of overlap between
two saliency maps. It allows the model to learn the overall shape and
positional information of salient object.

$\small\mathcal{L}_{iou}=1-\frac{\sum_{i=1}^{H}\sum_{j=1}^{W}P_{ij}G_{ij}}{\sum_{i=1}^{H}\sum_{j=1}^{W}(P_{ij}+G_{ij}-P_{ij}G_{ij})}$
(3)

Combination of these three loss functions can achieve pixel level supervision
on both global semantics and local details.

$\small\mathcal{L}_{total}=\mathcal{L}_{bce}+\mathcal{L}_{ssim}+\mathcal{L}_{iou}$
(4)

####  2.3.2 Multi-Stage Training

For the open-world scenario, we design a two-stage training strategy. In the
first stage, a traditional SOD training approach is used to improve detection
accuracy of salient objects. This process uses hybrid Loss for training to
derive high maxF Score. Similar to most lightweight SOD methods, resultant
saliency maps have artifacts and normal edges, leading to unsatisfactory
subjective effects. In the second stage, a matting training approach is
adopted to fine-tune the model trained in the first stage. A combination of L1
loss and edge loss is used, and learning rate is reduced to 0.3 times (i.e.
0.0003) of the first stage. After this stage of training, while maintaining
saliency object detection accuracy, quality of edges in our results are
enhanced, artifacts problems are effectively circumvented, and subjective
results are significantly improved.

##  3 Experiments

###  3.1 Implementation Details

Training batch size is set to 16. Adam optimizer [22] is used to train our
network and its hyper parameters are set to the default values. Initial
learning rate is set to 1e-3, betas are set to (0.9, 0.999), epsilon 1e-8,
weight decay set to 0. Models are trained on 8 V100 GPUs until convergence.
For data augmentation, images are resized to 680$\times$680, and then randomly
cropped to 640$\times$640\. In experiments, we add random horizontal flip,
random resize small with padding, and random noise. For test, input images are
resized to 640$\times$640 for inference to obtain saliency maps. Then saliency
maps are resized to the original input image size. Bilinear interpolation is
adopted for resizing. Three common metrics MAE, maxF and S-measure are adopted
to evaluate our model and SOTA models.

####  3.1.1 Datasets

For fair comparison, the PSUNet is trained on DUTS-TR dataset [23]. Validation
is conducted on three datasets, DAVIS-S [19], HRSOD-TE [18], and UHRSD-TE
[20]. DAVIS-S has 92 high-resolution images from the DAVIS dataset. HRSOD
dataset is designed for high-resolution SOD, containing 1610 training images
and 400 test images. UHRSD dataset is the highest resolution SOD dataset
available, containing 5920 4K-8K resolution images, 4932 training images and
988 test images. And we use its downscaled version of 2K resolution. Due to
low resolution of DUTS-TR and our high resolution structure design, training
with only DUTS-TR tend to affect the high resolution image results. Some high
resolution methods also train on HRSOD-TR and UHRSD-TR dataset. We follow the
same training scheme to compare with the corresponding methods. Besides, our
training is also conducted on DUTS-TR after super-resolution.

###  3.2 Comparison experiments

####  3.2.1 Comparison with the SOTAs

We compare the proposed PSUNet with 13 other lightweight SOD methods. For fair
comparison, model parameter amount, computational capacity, and saliency maps
are either from open-source code or provided by the authors. Results in Table
1 show that we outperform most of the lightweight methods on three high-
resolution datasets, with much smaller computation capacity and parameter
amount. The proposed PSUNet contains 4.15M parameters and requires 12.38G MACs
of computation when the input image size is 640$\times$640\. It is much
lighter than the majority of SOD methods. Experiment results show that the
objective indicators are even comparable to U2Net.

Method | MACs | Params | DAVIS-S | HRSOD-TE | UHRSD-TE
---|---|---|---|---|---
maxF↑ | MAE↓ | Sm↑ | maxF↑ | MAE↓ | Sm↑ | maxF↑ | MAE↓ | Sm↑
BASNet | 127.04 | 87.06 | .863 | .039 | .881 | .881 | .038 | .890 | .901 | .053 | .883
PoolNet | 89.00 | 68.21 |  .914 | .022 | .919 |  .888 | .041 | .895 |  .908 | .052 | .888
EGNet | 120.15 | 111.69 |  .913 | .023 | .922 |  .894 | .038 | .899 |  .916 | .049 | .895
SCRN | 15.10 | 25.23 |  .807 | .024 | .912 |  .896 | .034 | .904 |  .918 | .048 | .895
F3Net | 16.43 | 25.54 |  .918 | .020 | .913 |  .894 | .035 | .897 |  .916 | .045 | .891
MINet | 86.92 | 162.38 |  .896 | .024 | .904 |  .903 | .034 | .900 |  .920 | .043 | .894
LDF | 15.51 | 25.15 | .916 | .019 | .922 | .911 | .032 | .905 | .919 | .047 | .890
GateNet | 161.91 | 128.63 |  .917 | .024 | .915 |  .912 | .032 | .911 |  .921 | .048 | .895
PFSNet | 37.39 | 31.18 |  .910 | .019 | .922 | .914 | .032 | .907 |  .920 | .042 | .899
CTDNet | 10.15 | 24.63 |  .917 | .021 | .911 |  .907 | .033 | .898 |  .917 | .050 | .883
ICON | 64.90 | 19.17 | .923 | .023 | .914 | .908 | .041 | .894 | .913 | .050 | .888
U2Netp | 19.80 | 1.13 | .889 | .038 | .891 | .886 | .046 | .883 | .915 | .059 | .878
U2Net | 58.66 | 44.01 | .911 | .026 | .911 | .900 | .039 | .897 | .917 | .049 | .889
Ours-D | 12.38 | 4.15 | .936 | .018 | .926 | .888 | .042 | .887 | .909 | .054 | .884
Ours-DH | 12.38 | 4.15 | .952 | .014 | .938 | .907 | .039 | .898 | .922 | .046 | .897
Ours-HU | 12.38 | 4.15 | .943 | .017 | .934 | .900 | .043 | .888 | .922 | .048 | .891
Ours-DHU | 12.38 | 4.15 | .946 | .016 | .936 | .908 | .038 | .896 | .942 | .035 | .918
Ours-ALL | 12.38 | 4.15 | .958 | .012 | .948 | .922 | .041 | .896 | .948 | .038 | .915

Table 1: Objective comparison with the SOTAs. D represents DUTS-TR, H
represents HRSOD-TR, U represents UHRSD-TR, and ALL represents our integrated
HRSOM dataset. The best results are marked in red, second is in blue.

| 1st percentage↑ | 3rd percentage↓
---|---|---
Dimension | Ours | IOS | Ours | IOS
Subject Completeness | 35% | 33% | 25% | 49%
Edge Accuracy | 36% | 28% | 25% | 55%

Table 2: Subjective comparison with IOS based on user study. Fig. 3: Visual
comparison with IOS

Ablation | Configurations | maxF↑ | MAE↓ | Sm↑
---|---|---|---|---
Arch. | No SPSM | 0.905 | 0.059 | 0.877
SPSM with t=2 | 0.922 | 0.046 | 0.897
SPSM with t=3 | 0.937 | 0.042 | 0.907
SPSM with t=4 | 0.940 | 0.040 | 0.909
Loss | BCE | 0.918 | 0.057 | 0.886
BCE + SSIM | 0.921 | 0.053 | 0.890
BCE + IoU | 0.918 | 0.050 | 0.886
BCE + SSIM + IoU | 0.922 | 0.046 | 0.897

Table 3: Ablation Study

####  3.2.2 Subjective evaluation

We obtain robust open-world lightweight SOD model based on the proposed HRSOM
dataset with two-stage training strategy. After float16 quantization, they are
deployed to mobile devices. Model inference takes only 113ms on the high-
performance Snapdragon 8 Gen 2 Mobile Platform, and 234ms on the middle level
chip platform MediaTek Dimensity 8100-Max. To compare with industrial SOTA
products, we collected 500 open-world images to derive the IOS16 Quick Matting
results and our results. These results were shuffled and removed labels. At
each time, 100 sets of images were randomly selected and present to 93
experienced subjective reviewers for blind evaluation and scoring, according
to subject completeness and edge accuracy. For fairness, one result from our
earlier trained models was added to make up a group of 3 images per set. The
results of the subjective scoring ranking are shown in Table 2, and some hard
case comparison results are shown in Fig. 3.

###  3.3 Ablation study

In this section, we study the effect of different sampling ratio t of SPSM on
objective metrics, and verify the effectiveness of hybrid loss. In Table 3, we
compare the difference in results with and without SPSM, and results at
different values of sampling ratio t in SPSM. Models are trained on HRSOD-TR
and UHRSO-TR datasets. Experiments demonstrate that our SPSM has higher
improvement in high resolution scenes compared to no SPSM. And for scenes with
large resolution, the larger value of t, the more significant improvement. We
also compare effects of different combinations of loss functions. SPSM is set
with t=2 and size of training image 640$\times$640\. Models were trained on
HRSOD-TR and UHRSD-TR dataset and tested on UHRSD-TE dataset. Experiment
results prove that our Hybrid Loss is meaningful.

##  4 Conclusions

In this paper, we propose a lightweight salient object matting network PSUNet
for high-resolution images. Compared with 13 SOD SOTA methods on three high-
resolution SOD datasets, the proposed PSUNet achieves optimal performance for
all objective metrics, while controlling the amount of computation. Based on
the HRSOM dataset and the proposed multi-stage training strategy, we have
achieved the best subjective and objective results in the industry and the
academia.

## References

  * [1] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang,  “Saliency detection via graph-based manifold ranking,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 3166–3173.
  * [2] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun,  “Saliency optimization from robust background detection,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 2814–2821.
  * [3] R Sai Srivatsa and R Venkatesh Babu,  “Salient object detection via objectness measure,”  in 2015 IEEE international conference on image processing (ICIP). IEEE, 2015, pp. 4481–4485.
  * [4] Jianming Zhang, Stan Sclaroff, Zhe Lin, Xiaohui Shen, Brian Price, and Radomir Mech,  “Minimum barrier salient object detection at 80 fps,”  in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1404–1412.
  * [5] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan,  “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, October 2021, pp. 1905–1914.
  * [6] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Xin Yu, Yiran Zhong, Nick Barnes, and Ling Shao,  “Rgb-d saliency detection via cascaded mutual information minimization,”  in International Conference on Computer Vision (ICCV), 2021.
  * [7] Fanyi Wang and Yihui Zhang,  “A de-raining semantic segmentation network for real-time foreground segmentation,”  Journal of Real-Time Image Processing, vol. 18, pp. 873–887, 2021\.
  * [8] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand,  “U2-net: Going deeper with nested u-structure for salient object detection,”  2020, vol. 106, p. 107404.
  * [9] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang,  “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1874–1883.
  * [10] Yu-Huan Wu, Yun Liu, Jun Xu, Jia-Wang Bian, Yu-Chao Gu, and Ming-Ming Cheng,  “Mobilesal: Extremely efficient rgb-d salient object detection,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10261–10269, 2022.
  * [11] Min Seok Lee, WooSeok Shin, and Sung Won Han,  “Tracer: Extreme attention guided salient object tracing network,”  arXiv preprint arXiv:2112.07380, 2021.
  * [12] Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang,  “A simple pooling-based design for real-time salient object detection,”  in IEEE CVPR, 2019.
  * [13] Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng,  “P2t: Pyramid pooling transformer for scene understanding,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–12, 2022.
  * [14] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool,  “Highly accurate dichotomous image segmentation,”  in ECCV, 2022.
  * [15] Xuebin Qin, Deng-Ping Fan, Chenyang Huang, Cyril Diagne, Zichen Zhang, Adrià Cabeza Sant’Anna, Albert Suarez, Martin Jagersand, and Ling Shao,  “Boundary-aware segmentation network for mobile and web applications,”  arXiv preprint arXiv:2101.04704, 2021.
  * [16] Taehun Kim, Kunhee Kim, Joonyeong Lee, Dongmin Cha, Jiho Lee, and Daijin Kim,  “Revisiting image pyramid structure for high resolution salient object detection,” 2022.
  * [17] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,” 2015\.
  * [18] Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu,  “Towards high-resolution salient object detection,” 2019.
  * [19] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung,  “A benchmark dataset and evaluation methodology for video object segmentation,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 724–732.
  * [20] Chenxi Xie, Changqun Xia, Mingcan Ma, Zhirui Zhao, Xiaowu Chen, and Jia Li,  “Pyramid grafting network for one-stage high resolution saliency detection,”  in CVPR, 2022.
  * [21] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli,  “Image quality assessment: from error visibility to structural similarity,”  IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [22] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,” 2017.
  * [23] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan,  “Learning to detect salient objects with image-level supervision,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3796–3805.