# Session-level Normalization and Click-through Data Enhancement for Session-
based Evaluation

Haonan Chen  Gaoling School of Artificial Intelligence, Renmin University of
ChinaBeijingChina<EMAIL_ADDRESS>,  Zhicheng Dou  Gaoling School of
Artificial Intelligence, Renmin University of ChinaBeijingChina<EMAIL_ADDRESS>and  Jiaxin Mao  Gaoling School of Artificial Intelligence, Renmin University
of ChinaBeijingChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

Since a user usually has to issue a sequence of queries and examine multiple
documents to resolve a complex information need in a search session,
researchers have paid much attention to evaluating search systems at the
session level rather than the single-query level. Most existing session-level
metrics evaluate each query separately and then aggregate the query-level
scores using a session-level weighting function. The assumptions behind these
metrics are that all queries in the session should be involved, and their
orders are fixed. However, if a search system could make the user satisfied
with her first few queries, she may not need any subsequent queries. Besides,
in most real-world search scenarios, due to a lack of explicit feedback from
real users, we can only leverage some implicit feedback, such as users’
clicks, as relevance labels for offline evaluation. Such implicit feedback
might be different from the real relevance in a search session as some
documents may be omitted in the previous query but identified in the later
reformulations. To address the above issues, we make two assumptions about
session-based evaluation, which explicitly describe an ideal session-search
system and how to enhance click-through data in computing session-level
evaluation metrics. Based on our assumptions, we design a session-level metric
called Normalized U-Measure (NUM). NUM evaluates a session as a whole and
utilizes an ideal session to normalize the result of the actual session.
Besides, it infers session-level relevance labels based on implicit feedback.
Experiments on two public datasets demonstrate the effectiveness of NUM by
comparing it with existing session-based metrics in terms of correlation with
user satisfaction and intuitiveness. We also conduct ablation studies to
explore whether these assumptions hold.

††copyright: acmcopyright††journalyear: 2023††doi:
xx.xxxx/xxxxxxx.xxxxxxx††conference: WSDM ’24: The 16th ACM International
Conference on Web Search and Data Mining; February 27-March 3, 2024;
Mexico††booktitle: WSDM ’24: The 17th ACM International Conference on Web
Search and Data Mining, February 27-March 3, 2024, Mexico††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06

##  1\. Introduction

With the development of search engines, researchers increasingly focus on
building better evaluation methods. In the early years, the Cranfield paradigm
(Cleverdon et al., 1966) was the dominant approach in evaluating the search
results of a single query. However, when a user is trying to complete a
complex search task, she may issue multiple queries and browse a series of
documents to obtain sufficient information in a _search session_ (Zhu et al.,
2021; Zuo et al., 2022; Chen et al., 2022a). Many works have emerged to design
session-based evaluation metrics. Some of them have already been used in some
evaluation tasks, e.g., Session-based DCG (Järvelin et al., 2008) in the TREC
Session Track (Kanoulas et al., 2010) and Recency-aware Session-based Metric
(Zhang et al., 2020) in the recent NTCIR Session Search (SS) Task (Chen et
al., 2022b). However, there are still some remaining challenges for session-
based evaluation. In this work, we identify two major challenges of session-
based evaluation as some existing session-based metrics are based on
oversimplified or problematic assumptions.

The first challenge is that most existing session-level metrics (e.g., the
metrics used in these tasks) evaluate each query in a session based on an
existing query-level metric, and then aggregate those query-level scores with
some session-level discount factors or weighting schema to evaluate systems at
the session level. For example, sDCG (Järvelin et al., 2008) is based on the
cascade hypothesis, which gives lower-ranked search results and later-issued
queries smaller weights, and RSMs (Zhang et al., 2020) gives larger weights to
the recently issued queries. While previous studies show that these
aggregation metrics correlate well with users’ session-level satisfaction
feedback (Zhang et al., 2020), these metrics all implicitly assume that a
user’s query sequence in a search session will not be altered by the systems.
However, it would be expected that if a user is satisfied with the information
retrieved by the present and past queries, she may not need any subsequent
queries. (Following (Jones and Klinkner, 2008; Wang et al., 2013), we assume
that in the same session, the queries that may represent different sub-topics
serve the same primary information need.) To put it another way, we presume
that an ideal search system would return all relevant documents in a session
before all irrelevant documents, so the user can spend the least effort in
completing the search task. For example, a user issues a query “Java” and
clicks “What is Java Language”. After a minute, she issues another query “Java
Project”, and clicks “Java Projects for Beginners”. We assume the user would
prefer a system if it could predict that she is seeking the second clicked
document (e.g., using a personalized search model) and place it at the
beginning of this session. However, none of the existing session-level
evaluation metrics can fully take this reduction in effort into consideration
and give a maximum score to such an ideal search system. Note that it is true
that a user may learn something in the session which triggers her to seek new
pieces of information. However, we believe that an ideal system can predict
the change of her interest, lead her to discover all information needs as soon
as possible, and rank all the documents that serve her needs high in the
session.

Another challenge that may hinder the computation of session-level evaluation
metrics is that in real-world search scenarios, search engines can only record
implicit user feedback to the documents, i.e., click-through data. Therefore,
most metrics have to assume that the clicked documents are “relevant” to the
query in offline evaluation. However, users may skip some relevant documents
during the search because of position bias. This problem is more common in a
long search session as Price et al. (Price et al., 2007) found out that users
may omit documents in previous queries yet recognize and click them in later
reformulations, i.e., the session’s subsequent queries. For example, a user
issues a query “MacBook” then clicks “MacBook on Amazon”. A minute before, the
search system had already ranked this document among the top ten results when
she searched “Apple”, but she omitted it. We assume that when evaluating this
system, we should also mark ”MacBook on Amazon” relevant to ”Apple” in order
to reward this system for successfully predicting the user’s search intent and
saving her efforts on issuing another query. Thus, we need to consider the
“relevance” of a document at the session level but not at the level of each
separate query. However, existing metrics do not account for these omitted
documents and simply assume the unclicked documents are irrelevant (when only
implicit feedback is available), because these metrics only take clicks as
per-query relevance judgments, rather than considering clicks in the entire
session.

To tackle these challenges, in this paper, we make two assumptions about a
user’s search behaviors in a search session:

Assumption 1: An ideal search system should rank all the documents that the
user requires, i.e., all the relevant documents, before all the irrelevant
documents in the entire session. By doing so, it can save the user’s effort
because she may not even have to reformulate her queries. In other words, we
need to remove the boundary of queries in the session-level evaluation.

Assumption 2: A document that is clicked in a subsequent query but omitted in
a preceding query in the same session is relevant to that preceding query.
Consequently, the first occurrence of this document is assumed to be relevant
to the information need of the current session.

We will refer to Assumption 1 as A1 and Assumption2 as A2.

Most offline evaluation in the industry can only use implicit user feedback
(click-through data) to infer relevance labels because it is costly to get
human relevance labels. Besides, most human-labeled relevance is query-level
relevance, not the session-level relevance that is preferred when evaluating
session search systems. Thus, our assumptions in this paper are based on
actual click-through data (implicit feedback) and user behaviors rather than
manual labels. Under this condition, we describe an ideal session search
system (A1) and try to mine session-level relevance labels from click-through
data (A2).

The common idea implied in these assumptions is that we need to reduce the
impact of query boundaries in session search evaluation. Based on these
assumptions, we design a session-level metric called Normalized session level
U-Measure (NUM) based upon the original U-Measure (Sakai and Dou, 2013). This
metric evaluates a session as a single “virtual query” and employs an ideal
session to normalize the evaluation result according to A1. NUM also converts
the click-through feedback into session-level relevance labels based on A2.
Experiments on two public datasets (TianGong-SS-FSD (Zhang et al., 2020) and
NTCIR-16 Session Search Task (Chen et al., 2022b)) show that NUM is an
intuitive session-level metric that correlates well with user satisfaction.
Furthermore, ablation studies confirm that A1 and A2 are valid.

To summarize, the contributions of the paper are as follows:

(1) We make two assumptions about what we should do in session-based
evaluation. With these assumptions, we describe what an ideal session should
be and discuss how to use the click-through data to derive session-level
relevance labels.

(2) We design a session-level metric called NUM based on U-measure (Sakai and
Dou, 2013). It treats a session as a virtual query and uses the evaluation
score of the ideal session for normalization (session-level normalization).

(3) We show that NUM correlates well with user satisfaction. Further studies
also demonstrate its intuitiveness. In addition, the ablation studies explore
the reasonability of our two assumptions.

Figure 1. The illustration of how U-measure constructs trailtext from a two-
query session. The results clicked by the user are marked with red checkmarks
and the results marked as relevant are filled with color gray. The right part
is the constructed trailtext, where “$s_{i}$” is the $i$-th string of it.

##  2\. Related Work

###  2.1. Session-based Evaluation Metrics

Because in real-world search scenarios, users usually issue a sequence of
queries to complete a complex search task, researchers have gone beyond the
traditional query-level metrics (e.g., MAP, nDCG, and MRR) and designed some
session-level metrics.

Session-based DCG (sDCG) (Järvelin et al., 2008) is a multi-query metric based
on Discounted Cumulative Gain (DCG) (Järvelin and Kekäläinen, 2002). Based on
the cascade hypothesis, sDCG discounts the weights of the lower-ranked results
and later-issued queries. Similarly, Lipani et al. (Lipani et al., 2019)
proposed a session-based version of Rank-Biased Precision (RBP) (Moffat and
Zobel, 2008), which added a new parameter over RBP to balance between query
reformulation behaviors or continuing to examine documents. Yang and Lad (Yang
and Lad, 2009) proposed a utility-based evaluation framework. They evaluated
the Expected Utility of the search system over all possible interaction
patterns. Van Dijk et al. (van Dijk et al., 2019) leverages a Markovian Chain
of users’ behaviors in a session to substitute the fixed discount of
documents. Moffat et al. (Wicaksono and Moffat, 2021), extended the C/W/L
framework to session-based effectiveness evaluation.

Liu et al. (Liu et al., 2018, 2019) showed how the recency effect can affect
users’ session-level satisfaction with user studies. They proposed that the
later-issued queries should receive higher weights, which is in contrast to
the cascade hypothesis. Based on these findings, Zhang et al. (Zhang et al.,
2020) proposed Recency-aware Session-based Metrics (RSMs), which incorporate
the recency effect into session-based evaluation. Their experimental results
showed that RSMs have the strongest correlations with user satisfaction among
these metrics and achieve state-of-the-art performance in estimating user
satisfaction.

Although various session-based evaluation metrics have been proposed, and some
perform quite well in estimating user satisfaction and/or measuring system
effectiveness, most of these session-based metrics evaluate each query
separately and aggregate the evaluation scores using some discount factors or
weighting schema. Therefore, they all implicitly assume that all queries in
the session should be considered in the session-based evaluation and the
orders of the queries are definite. However, as we stated in A1, if a search
system could make the user satisfied with the first few queries, she may not
need any subsequent queries. Few existing session-based metrics have taken
this into consideration.

Besides, most existing metrics are based on the Cranfield/test collection
approach (test collections with explicit relevance judgments). However, as
stated in Section 1, most offline scenarios in the industry can only use
implicit user feedback (click-through data). Therefore, these metrics would
have to directly treat click-based labels as relevance labels. Our work
attempts to bridge this gap by enhancing click-through data based on A2. By
this, we aim to infer session-level relevance labels with implicit feedback.

###  2.2. U-measure

In this part, we will briefly review U-measure (Sakai and Dou, 2013), based on
which we design a session-level metric NUM. U-measure is a framework for
evaluating information access that can be used to evaluate various IR tasks.

Figure 1 is an example in its original paper of how U-measure constructs a
trailtext from a two-query session. When evaluating a session, U-measure
treats the session as a single “virtual query” (as a whole) by building a
trailtext for this session. A trailtext $tt$ is made up of $n$ strings
concatenated together: $tt=s_{1}s_{2}\cdots s_{n}$. Each string $s_{i}$ could
be a snippet or the entire content. Sakai and Dou presume that the trailtext
represents what the user reads in the exact order during an information
search. Besides, they assume that a user reads $F$% (20% in (Sakai and Dou,
2013)) of the content of a document, i.e., only $F$% of a document’s length is
counted in the trailtext. They define $pos(s_{i})=\sum_{j=1}^{i}|s_{j}|$ as
the offset position of $s_{i}$. Specifically, trailtexts are derived from
session data that follows the algorithm in (Sakai and Dou, 2013) (Figure 5).
The general computation of U-measure is:

(1) ${\rm U}=\sum^{|tt|}_{pos=1}{\rm gain}(pos){\rm D}(pos),$

where ${\rm D}(pos)$ is the position-based decay function and ${\rm
gain}(pos)$ is the corresponding gain. Specifically, if $s_{i}$ is not
relevant, ${\rm gain}(pos(s_{i}))$ = 0, and if $s_{i}$ is regarded
$l$-relevant, ${\rm gain}(pos(s_{i}))=gv_{l}$, where $gv_{l}$ is a gain value
for relevance level $l$. ${\rm D}(pos)$ is a linear decay function:

(2) ${\rm D}(pos)={\rm max}(0,1-\frac{pos}{L}),$

where $L$ denotes the maximum amount of text the user may read in a single
session and represents the point at which all information units are considered
worthless.

We choose U-measure as our metric’s backbone because: (1) Rather than
aggregating the evaluation results of all queries in a session, which may
contain queries that the user does not need, U-measure evaluates a session as
a single “virtual query” by building a trailtext. (2) U-measure takes the
length of a document into consideration and has the diminishing return
property, making it more realistic than rank-based metrics.

Note that the original paper of U-measure (Sakai and Dou, 2013) primarily
introduces it as a general evaluation framework for a variety of IR tasks,
rather than specifically for session search. Therefore, many improvements are
required to bring it to session-level evaluation (based on our two
assumptions). Experiments conducted in Section 5.3 demonstrate the
effectiveness and necessity of these modifications.

##  3\. The Proposed Methods

In this section, we will introduce the proposed metric Normalized U-Measure
(NUM). When evaluating a session, instead of aggregating the scores of each
query in the session, it evaluates the session as a “virtual query” by
building a comprehensive trailtext for the whole session. Besides, it uses the
ideal session to normalize the evaluation result in accordance with A1. Based
on A2, it uses click-through data to infer session-level relevance labels.

Figure 2. The illustration of NUM. The upper part is the actual session, and
the lower part is the ideal session. The results clicked by the user are
marked with red checkmarks, and the results marked as relevant are filled with
color gray. Rank 4 of the first query is marked relevant even though it is not
clicked here because it is clicked in the subsequent query. We treat a session
as a virtual query, based on which we build a trailtext to enable session-
level evaluation. We construct the trailtext based on actual user clicks for
the actual session and based on the enhanced session-level labels for the
ideal session.

###  3.1. Normalizing U-Measure

####  3.1.1. Ideal Session

As we stated in A1, we believe that an ideal search system should rank all
relevant documents before all irrelevant documents so that the user may not
even need to reformulate her queries, saving her time and effort. As
illustrated in Figure 2, we place all the relevant documents before all
irrelevant documents to build the ideal trailtext because it allows the user
to get all of the information she needs without reformulation. By doing so, we
manage to explicitly tell the search system what an optimal session-level
ranking is and lead it to limit the number of reformulations, which we believe
will make the user more satisfied. Note that the relevance label we refer to
here is the session-level labels derived from click-through data based on A2.

We notice that there could be different re-ranking strategies of the documents
with the same relevance scores in the ideal session. If the user scans the
ranked documents from top to bottom, she will learn by reading the preceding
document that subsequent documents are relevant. Thus, we maintain the same
order of clicks in the trailtext of the ideal session as the actual session.

####  3.1.2. Session-level Normalization

The value of NUM represents the proximity of an actual session to the ideal
session. Note that when building trailtext for the actual session, based on
(Joachims et al., 2007; Sakai and Dou, 2013), we assume that the user reads
all the documents she clicks and all the snippets above the lowest clicked
document in the session. And for the ideal session, we simply presume the user
reads the documents that are marked session-level relevant (inferred by click-
through data and A2) since all of them are presented at the beginning of the
session and they all support the user’s main information need. The session-
level normalization technique is defined as follows:

(3) ${\rm NUM}=\frac{{\rm U}(S_{\rm actual})}{{\rm U}(S_{\rm ideal})},$

as illustrated in Eq (1) and Eq (2), ${\rm U}(S)$ can be computed as follows:

(4) ${\text{U}}(S)=\sum_{pos=1}^{|{tt}|}{\rm gain}(pos){\rm
max}(1-\frac{pos}{L}),$

where $L$ is the longest Maximal Trailtext Length (MTL) across all conceivable
search sessions. A session’s MTL is the sum of the text lengths of (1) all
snippets above the last click of each query, (2) all documents clicked by the
user in that session, and (3) all reformulation texts (illustrated in Section
3.1.3). As a result, $L$ reflects the most text the user has to read in a
single session.

####  3.1.3. Reformulation Text

As shown in the upper right part of Figure 2, we add an empty text named
reformulation text in the trailtext between two queries. We believe it can
penalize query reformulation behaviors to an adjustable degree, which the
original U-measure does not account for. The intuition here is that U-measure
uses text length to simulate reading time, thus we use empty texts to
represent the user’s reformulation time.

We determine the length of the reformulation text by exploring a Chinese field
study dataset TianGong-SS-FSD (Zhang et al., 2020) to find out how much time
users spend between the exit of a query and the issuance of the following
query. We compute the query reformulation time for each query. Besides, we
eliminate the queries with negative reformulation time (due to multi-tabbing
or logging errors) and 4% of queries with the largest reformulation time
values (extremely long times for unknown reasons). More details are in Section
5.1.

###  3.2. Click-through Data Enhancement

We use the click-through data to infer session-level relevance labels based on
A2. We believe the same document should be marked relevant to this session if
it was clicked in a subsequent query but skipped in a preceding one. As
illustrated in the left part of Figure 2, the document $d_{4}$ at rank 4 is
marked relevant because it is clicked in the subsequent query.

Note that there will be duplicate documents that are considered relevant in
the ideal session after the enhancement, e.g., $d_{4}$ at rank 4 and rank 1’.
Among the existing works, some do not give any special treatment to duplicate
relevant documents (Kanoulas et al., 2011; Sakai and Dou, 2013) and some
choose to discount these documents (Yang and Lad, 2009). We identify three
possible choices of dealing with $d_{4}$: (1) We include it in the trailtext
because we believe that both rank 4 and rank 1’ are informative. Besides,
users may hope to find their clicked documents remain in the top results,
which can facilitate the re-finding behaviors and the trust of search systems
(Kanoulas et al., 2011); (2) We include it in the trailtext but give a
discount to its relevance score; (3) We exclude it in the trailtext because it
is redundant. We will include $s_{6}$ in our instantiation of NUM (choice (1))
and study these three approaches in Section 5.4.

##  4\. Experimental Setup

###  4.1. Datasets

We conduct our experiments on two public datasets: TianGong-SS-FSD (Zhang et
al., 2020) and NTCIR-16 Session Search (SS) Task (Chen et al., 2022b). For
simplicity, we denote the two datasets as $FSD$ and $NST$ in what follows111We
do not use TREC 2014 Session Track because its sessions are relatively old and
we want to study user behaviors on modern search engines..

####  4.1.1. $FSD$

$FSD$ is a dataset collected from field studies. It records the users’
session-level satisfaction rating, which we believe is a good standard for
evaluating the metrics because it reflects the user’s actual feelings about
the search system. We explore the correlation of metrics (NUM and the baseline
metrics) with user satisfaction ratings on this dataset. Following (Zhang et
al., 2020), we filter out the sessions containing more than one SERP. Besides,
to facilitate NUM usage, we filtered out the sessions that do not contain any
clicks and only have one query. We believe this kind of session can be
considered good abandonment because in each of these sessions, the user’s
information need is resolved only by the results pages (e.g., the information
of snippets), without having to click on a result or do any query
reformulations (Li et al., 2009).

Table 1. The statistics of the pre-processed datasets. | TianGong-SS-FSD | NTCIR-16 SS
---|---|---
# Sessions | 994 | 2,000
# Queries | 3,411 | 6,420
Avg. # Query per Session | 3.43 | 3.21
# Results per Query | 10 | 10
Avg. # Clicks per Session | 3.49 | 3.08

####  4.1.2. $NST$

$NST$ is collected from a Chinese search engine. Its full data has 147,154
sessions, and there are 2,000 of them that have human-labeled relevance. We
test the intuitiveness of the metrics on these 2,000 sessions because we may
use these manual labels in future work and the scale of 2,000 sessions are
large enough to draw conclusions (already larger than the dataset used in many
works that are based on Cranfield/test collection (Zhang et al., 2020; Lipani
et al., 2019; Järvelin et al., 2008)). We use the remaining 145,154 sessions
to estimate $L$ and the length of the reformulation text when testing
intuitiveness on $NST$. These sessions do not contain any good abandonment.

The statistics of the pre-processed datasets are shown in Table 1.

###  4.2. Meta-evaluation Approaches

We utilize two meta-evaluation approaches to evaluate and compare NUM with
existing session-based metrics:

####  4.2.1. Correlation with User Satisfaction.

Since user satisfaction in information retrieval can be defined as the
fulfillment of a specific objective (Kelly, 2009) and it assesses users’
actual feelings about a system, it can be considered as the ground truth to
evaluate the evaluation metrics (Al-Maskari et al., 2007; Huffman and
Hochster, 2007). With the session-level user satisfaction feedback in $FSD$,
we can compare the performance of different session-based metrics by computing
the correlation between them and user satisfaction on this dataset.

####  4.2.2. Intuitiveness

The Concordance Test (Sakai, 2012) is proposed to quantify the intuitiveness
of diversity metrics. We believe it can also predict the intuitiveness of
session-based metrics. We will first choose some golden standard measures and
presume them to actually represent intuitiveness. Given a pair of metrics (M1,
M2), the relative intuitiveness of M1(or M2) is computed in terms of
preference agreement with the golden standard measures.

###  4.3. Generating Runs for NST

Since the organizers of NTCIR-16 Session Search (SS) Task have not released
the run data of the participants, we use some ranking models to re-rank $NST$
to generate some runs for the experiments of intuitiveness. The models are
comprised of: (1) ad-hoc ranking models, including KNRM (Xiong et al., 2017),
ARC-II (Hu et al., 2014), Conv-KNRM (Dai et al., 2018), and DUET (Mitra et
al., 2017); (2) session-based ranking models, including HBA-Transformers (Qu
et al., 2020), COCA (Zhu et al., 2021), and RICR (Chen et al., 2022a). The
settings of these models are all the same as in their original papers.

Moreover, since the current session-based ranking models are not advanced
enough to consider the two assumptions we put forward, we artificially
construct two types of runs based on these ranking models: (1) Ideal runs. For
each query of the session, we first add the candidate documents of the
subsequent queries in the same session into its pool. Then we re-rank the
candidates using these models and keep the top ten results. We believe this
can make the re-ranked session closer to the ideal session that we defined.
(2) Diversified runs. For each query of the session, based on the extended
candidate pool described above, we discard the candidate documents that are
already presented (included in the top ten results) in the preceding queries.
We believe this can make the session more “diversified”.

We generate 7 original runs, 7 ideal runs, and 7 diversified runs based on 7
ranking models aforementioned for $NST$.

Table 2. Spearman’s $\rho$ and Kendall’s $\tau$ between session-based metrics and user satisfaction on $FSD$. The best performance is in bold, and the second-best performance is underlined. ”Improv.” reflects the improvements of NUM over RS-RBP. Metric | sDCG | sRBP | sDCG/q | sRBP/q | U-measure | U-measure/q | RS-DCG | RS-RBP | NUM | Improv.
---|---|---|---|---|---|---|---|---|---|---
Spearman’s $\rho$ | 0.0335 | 0.0508 | 0.3136 | 0.3150 | -0.2282 | 0.1008 | 0.3473 | 0.3508 | 0.3611 | 2.94%
Kendall’s $\tau$ | 0.0242 | 0.0382 | 0.2492 | 0.2507 | -0.1800 | 0.0789 | 0.2771 | 0.2795 | 0.2884 | 3.18%
Table 3. Performance of ablated metrics on FSD. | Spearman’s $\rho$ | Kendall’s $\tau$
---|---|---
NUM w/o. SN | -0.1960 -154.28% | -0.1561 -154.13%
NUM w/o. RT | 0.3276 -9.28% | 0.2639 -8.50%
NUM w/o. SE | 0.3522 -2.46% | 0.2789 -3.29%
NUM (Full) |  0.3611 - |  0.2884 -

###  4.4. Baselines

To demonstrate the effectiveness of NUM and verify our two assumptions, we
compare NUM with existing DCG-based and RBP-based metrics sDCG (Järvelin et
al., 2008), sRBP (Lipani et al., 2019), sDCG/q, sRBP/q, RS-DCG (Zhang et al.,
2020) and RS-RBP (Zhang et al., 2020). In addition, to verify the
effectiveness of the session-level normalization introduced in Section 3.1.2,
we also compare our metric with U-measure (Sakai and Dou, 2013) and
U-measure/q.

Following the settings of (Zhang et al., 2020), supposing each query has $N$
documents and each session $S$ has $M$ queries, the computation of sDCG can be
described as follows:

(5) ${\rm sDCG}(S)=\sum_{m=1}^{M}\sum_{n=1}^{N}\frac{{\rm g}(d_{m,n})}{(1+{\rm
log}_{b_{q}}m)(1+{\rm log}_{b_{r}}n)},$

where ${\rm g}(d_{m,n})$ maps the score of the $n$-th document in the $m$-th
query of the session. And sRBP is computed as follows:

(6) ${\rm
sRBP}(S)=(1-p)\sum_{m=1}^{M}\left(\frac{p-bp}{1-bp}\right)^{m-1}\sum_{n=1}^{N}(bp)^{n-1}{\rm
g}(d_{m,n}).$

“/q” in sDCG/q and sRBP/q is a way of normalizing metrics by simply the number
of queries in the session. It is described as: ${\rm Metric/q}={{\rm
Metric}(S)}/{M}$.

For the computations of RS-DCG and RS-RBP, we apply the settings of their
original paper (Zhang et al., 2020):

(7) ${\text{RS-
DCG}}(S)=\sum_{m=1}^{M}e^{-\lambda(M-m)}\sum_{n=1}^{N}\frac{{\rm
g}(d_{m,n})}{(1+{\rm log}_{b_{q}}m)(1+{\rm log}_{b_{r}}n)},$ (8) ${\text{RS-
RBP}(S)}=\sum_{m=1}^{M}e^{-\lambda(M-m)}\left(\frac{p-bp}{1-bp}\right)^{m-1}\sum_{n=1}^{N}(bp)^{n-1}{\rm
g}(d_{m,n}).$

Following (Sakai and Dou, 2013), we calculate the $gain$ value of a
$l$-relevant document as: ${\text{g}}(d)=(2^{l}-1)/{2^{H}}$, where $H$ is the
highest relevance level. Since this paper mainly discusses the scenarios where
we only have implicit feedback, the $gain$ value of a session-level relevant
document is $(2^{1}-1)/2^{1}=0.5$. For NUM, the session-level labels are
inferred from click-through data, according to A2. For the other metrics, we
treat the clicked document directly as relevant ones, and do not use A2 to
enhance the corresponding relevance labels.

For U-measure, we build a trailtext from user clicks and compute U-measure on
it as illustrated in Eq (4). The value of $L$ in (Sakai and Dou, 2013) is
estimated using Microsoft’s Bing (September 7, 2012, US market) data. However,
we have to re-estimate $L$ due to the inconsistency between the language of
Microsoft’s Bing (English) and the datasets we utilize in this study
(Chinese). Furthermore, we believe that we should take the search engine that
the dataset uses into consideration when we estimate $L$. This is because we
believe that a user’s tolerance for “the largest amount of text that the user
may have to read in one session” can be different from one search engine to
another. Thus we estimated $L$ independently for TianGong-SS-FSD (Zhang et
al., 2020) and NTCIR-16 Session Search (SS) Task (Chen et al., 2022b). We
estimated the MTL for each session by assuming that each snippet is 80
characters long (which is a reasonable assumption for Chinese search engines),
and discarded 1% of the sessions with the highest MTL values. Note that we set
$F$ as 20 following the original U-measure paper (Sakai and Dou, 2013).

For the instantiations of the baseline metrics and NUM, we adopt different
approaches with respect to different meta-evaluation techniques and the
corresponding datasets. More details can be found in Section 5.1 and Section
5.2. Besides, we provide an anonymous version of our code for
review.222https://anonymous.4open.science/r/NUM-C228/

##  5\. Results and Analysis

###  5.1. Correlation with User Satisfaction

We first compare the performance of NUM to the baseline metrics by computing
Spearman’s $\rho$ (Spearman, 1961) and Kendall’s $\tau$ (Kendall, 1938) with
user satisfaction on $FSD$.

For the instantiations of the baseline metrics and NUM in this experiment, we
adopt a 5-fold cross-validation method following (Zhang et al., 2020). We
repeat this approach ten times. For each time, we use one fold of data to test
the metrics’ correlation with user satisfaction and use the other four folds
to tune the parameters. We tune the parameters of DCG-based and RBP-based
baselines to fit user satisfaction based on Spearman’s $\rho$. For DCG-based
metrics, $b_{r}$ and $b_{q}$ are searched in range $(1.0,5.0]$ with step of
0.1. For RBP-based metrics, $b$ and $p$ are searched in range $(0,1)$. The
exact values of the parameters can be found in the anonymous version of our
code. For U-measure-based metrics (including NUM), we only estimate (not tune)
their parameters ($L$ and the length of the reformulation text) because these
parameters should be consistent with users’ real reading behavior. Note that
we estimate them on the same folds of data as other baselines for fair
comparisons.

The average estimated $L$ is 19,336 for U-measure-based metrics. Besides, We
found that a query reformulation takes an average of 206 seconds, or 3.43
minutes, which is a relatively high cost. Additionally, native Chinese
speakers can usually read at an average speed of 255±29 words per minute
(Trauzettel-Klosinski et al., 2012). As a result, the average estimated length
of the reformulation text is ($255\times 3.43=875.5$) words (Chinese
characters).

We report the average Spearman’s $\rho$ and Kendall’s $\tau$ between each
metric and user satisfaction across all ten times of 5-fold cross-validation
on $FSD$. The results are shown in Table 2, which demonstrate the
effectiveness of our method333We will notice that the results of some metrics
are lower than those reported in (Zhang et al., 2020). This is because this
paper mainly discusses real-world offline scenarios where we can only obtain
implicit feedback, so we use the click-through data to estimate user
satisfaction.. Furthermore, we can make the following observations:

(1) NUM achieves the best results among all metrics, demonstrating its
effectiveness of estimating user satisfaction. For example, when compared to
the state-of-the-art baseline RS-RBP, our metric has improved Kendall’s $\tau$
by around 3.18%.

(2) Compared to the original U-measure and the simply-normalized U-measure/q,
NUM has a stronger correlation with user satisfaction, which demonstrates the
effectiveness of the proposed session-level normalization. We can observe that
NUM outperforms the original U-measure, indicating that it is necessary to
apply the session-level normalization and the click-through data enhancement
for session-based evaluation. Moreover, NUM performs better than U-measure/q,
which further demonstrates the effectiveness of session-level normalization.

###  5.2. Intuitiveness

Table 4. Intuitiveness based on preference agreement with the proposed golden standard measures(AP and LCD). For each metric combination, the higher score is in bold, and the number of disagreements between these two metrics is stated in the parentheses below. The abbreviation ”UM” stands for ”U-measure”. AP | sRBP | UM | RS-DCG | RS-RBP | NUM
---|---|---|---|---|---
sDCG |  0.89/0.83 |  0.89/0.70 |  0.89/0.65 |  0.84/0.67 |  0.64/0.91
(75,655) | (114,588) | (117,044) | (130,035) | (154,371)
sRBP | - |  0.85/0.71 |  0.86/0.67 |  0.84/0.69 |  0.62/0.92
- | (125,447) | (122,497) | (125,116) | (154,137)
UM | - | - |  0.86/0.79 |  0.79/0.78 |  0.58/0.94
- | - | (96,944) | (115,498) | (172,019)
RS-DCG | - | - | - |  0.82/0.90 |  0.56/0.94
- | - | - | (72,863) | (182,192)
RS-RBP | - | - | - | - |  0.57/0.92
- | - | - | - | (179,640)
LCD | sRBP | UM | RS-DCG | RS-RBP | NUM
sDCG |  0.81/0.86 |  0.93/0.59 |  0.84/0.62 |  0.75/0.68 |  0.56/0.85
(75,655) | (114,588) | (117,044) | (130,035) | (154,371)
sRBP | - |  0.92/0.58 |  0.84/0.61 |  0.77/0.67 |  0.57/0.84
- | (125,447) | (122,497) | (125,116) | (154,137)
UM | - | - |  0.72/0.86 |  0.62/0.88 |  0.45/0.94
- | - | (96,944) | (115,498) | (172,019)
RS-DCG | - | - | - |  0.73/0.95 |  0.48/0.87
- | - | - | (72,863) | (182,192)
RS-RBP | - | - | - | - |  0.53/0.83
- | - | - | - | (179,640)

Since there are few works trying to evaluate the intuitiveness of session-
based metrics, the golden standard measures that represent the intuitiveness
of session search have yet to be discovered. In this work, we suggest two
metrics444For these metrics, we treat the clicked document directly as
relevant ones and do not use A2 to enhance their labels. More details of these
metrics can be found in our code. for the intuitiveness test of session-level
metrics:

(1) Average Precision (AP). We believe this measure is simple but intuitive,
by which we simply compute the precision of each query and average these
values. Note that because MAP depends on the recall base, which can not be
estimated with click data, we thus use average precision instead.

(2) The position of the Last Clicked Document (LCD). This measure records the
position of the last clicked document in the whole session and takes the
reciprocal of this position as the score: ${\text{LCD}}(S)={1}/{Index_{lc}}$,
where $Index_{lc}$ is the position of the last clicked document in the session
$S$. This value depicts the number of snippets a user has to examine in order
to obtain all the information she needs. For example, if each query has 10
candidate documents and the last clicked document of the session is the fourth
document of the second query, then $Index_{lc}=10+4=14,{\text{LCD}}(S)=1/14$.
We believe that the session with a higher LCD value should be preferred
because the user can scan fewer snippets to complete her search task, saving
her time and effort. Thus, we believe LCD also represents the intuitiveness of
session search.

Table 5. The ablation experiments of intuitiveness based on preference agreement with LCD.  Metric |  NUM (Full) |  # disagreements
---|---|---
NUM w/o. SN |  0.46/0.93 |  (157,785)
NUM w/o. RT |  0.78/0.99 |  (89,495)
NUM w/o. SE |  0.75/0.91 |  (96,245)

For the instantiations of the DCG-based and RBP-based metrics in this
experiment, since there are no golden standard labels like user satisfaction
in $NST$, we use the mean of the parameters tuned on each fold in Section 5.1
(their exact values can be found in the anonymous version of our code). For
U-measure-based metrics, we only need to estimate (not tune) $L$ as we
explained in Section 5.1. Thus, we use the remaining 145,154 sessions of $NST$
to estimate them and test the intuitiveness on the other 2,000 sessions. The
estimated $L$ is 12,792. For the length of the reformulation text, since there
is no start and end timestamp of a query in $NST$, we simply use the estimated
one in Section 5.1 (362).

We exclude sDCG/q, sRBP/q, and U-measure/q from this experiment because the
“/q” normalization does not affect the concordance test (two runs have the
same number of queries). The concordance test is performed on all 21 runs (7
original runs + 7 ideal runs + 7 diversified runs, i.e.,
(21$\times$20/2$\times$2000= )420,000 session pairs.

The results are presented in Table 4. For example, the result at the top left
represents that sDCG and sRBP disagree in 75,655 pairs of sessions. Among
these disagreed pairs, sDCG agrees with AP on around 89% of them, while sRBP
agrees on about 83%, which implies sDCG is more intuitive than sRBP in terms
of AP. Furthermore, we can observe that:

(1) In terms of all golden standard measurements, NUM is more intuitive than
all baselines. For example, NUM agrees with AP on about 94% of the 182,192
disagreement session pairs, whereas RS-DCG is only consistent with AP on
around 56%.

(2) Incorporating the recency effect makes the metric less intuitive. After
incorporating the recency effect (RS-DCG and RS-RBP), we can observe that the
intuitiveness of metrics decreases in terms of all golden standard measures.
These findings reveal that, while the recency effect is beneficial for
estimating user satisfaction, it degrades the metrics’ intuitiveness.

###  5.3. Ablation Study

To further explore the reasonability of our two assumptions and the
effectiveness of the improvements we make over the original U-measure, we
design several variants of NUM. Specifically, we conduct the ablation
experiments on $FSD$ and $NST$ as follows:

$\bullet$ NUM w/o. SN. We remove the session-level normalization part (SN,
illustrated in Section 3.1.2). In another word, we only evaluate the actual
session without considering the ideal session.

$\bullet$ NUM w/o. RT. We discard the reformulation text (RT, introduced in
Section 3.1.3). We do not add extra empty texts into the trailtext to penalize
query reformulations.

$\bullet$ NUM w/o. SE. We eliminate the click-through data enhancement, which
states that the same document should be tagged relevant to the preceding query
if it is clicked in a subsequent query but skipped in the previous one, i.e.,
the supplemental enhancement (SE) based on A2.

The results of the ablation experiments are shown in Table 3 and Table 5. From
which we can draw the following conclusions:

(1) Normalizing the evaluation result at the session level is effective. In
A1, we presume that in an ideal session, all relevant documents should be
ranked before all irrelevant documents. We can evaluate the similarity between
this session and the ideal session by a session-level normalization. After
removing this technique, our metric’s intuitiveness and correlation with user
satisfaction both drop. For example, Spearman’s $\rho$ on FSD decreases by
about 154.28%. Furthermore, NUM without session-level normalization agrees
with LCD on 46% of the disagreement pairs, whereas NUM agrees with LCD on 93%.
These declines show that normalizing the actual session’s evaluation score
with the ideal session’s score is effective. It also supports A1 that an ideal
session should rank all relevant documents before all irrelevant documents in
the session.

(2) It is useful to include a reformulation text in the trailtext for each
query reformulation. We propose to add a reformulation text in the trailtext
between every two queries (as illustrated in Section 3.1.3). We believe it can
penalize the query reformulation behavior to an adjustable degree, which the
original U-measure does not take into account. After eliminating the
reformulation texts, the performance of our metric drops. For example, the
performance of estimating user satisfaction decreases by about 8.50% in terms
of Kendall’s $\tau$. Furthermore, NUM agrees with LCD on about 78% of the
disagreement pairs without the reformulation text, whereas with the
reformulation text, NUM agrees with around 99%. This indicates that penalizing
query reformulations by adding empty texts into the trailtext is effective.

(3) The supplemental enhancement (SE) makes our metric more intuitive and
correlate better with user satisfaction. To verify A2, we propose that if a
document is clicked in a subsequent query but skipped in a preceding one, it
should be tagged relevant to the preceding query. After removing SE, the
performance decreases. For example, it causes a decrease of 3.29% in terms of
Kendall’s $\tau$. Moreover, NUM agrees with LCD on about 75% of disagreement
pairs without SE, but the full NUM agrees with LCD on around 91%. These
results demonstrate the effectiveness of SE and verify A2. The reason these
reductions are smaller than those of the preceding two removals is that the
number of documents that require SE is small (approximately 1% of $FSD$ and 3%
of $NST$).

###  5.4. Dealing with Duplicate Documents

As aforementioned in Section 3.2, there will be duplicate documents that are
considered relevant in the ideal session after enhancing the click-through
data based on A2. There are three approaches to deal with the subsequent
duplicate clicked documents: (1) Exclude it from the trailtext; (2) Include it
in the trailtext but give a discount to its relevance score; (3) Include it in
the trailtext and do not give any discount. For approach (2), since this
discount factor is not our work’s main contribution, we will simply set the
discount value as 0.5 and leave more sophisticated discount factors to future
work.

We implement these three approaches on $FSD$ and report the correlation with
user satisfaction. The performances are presented in Table 6. We will notice
that the difference between them is relatively trivial. This is because that
the number of documents that require SE is small (approximately 1% of $FSD$).
We can observe that approach (3) achieves best performance, which demonstrates
that the duplicate documents produced by SE based on A2 should be include in
the trailtext of the ideal session. A possible reason is that the user may
need to re-find information that she had already found or skimmed through
earlier in the session.

Table 6. Performances of different approaches to dealing with duplicate documents on $FSD$. Approach |  Spearman’s $\rho$ |  Kendall’s
---|---|---
Exclude |  0.3600 |  0.2876
Include & Discount |  0.3597 |  0.2871
No Discount |  0.3611 |  0.2884

##  6\. Conclusions and Future Work

In this work, we identify two challenges in session-based evaluation and make
two assumptions about evaluating a session. A1 states that an ideal search
system should rank all relevant documents before all irrelevant documents in
the session. A2 believes that the documents clicked in a subsequent query but
omitted in a preceding query are also relevant to that preceding query. To
verify these assumptions, we design a session-level metric called Normalized
U-Measure (NUM). NUM evaluates a session as a virtual query, uses the score of
an ideal session to normalize the evaluation result (A1) and enhances the
click-through data (A2). Experiments on two public datasets demonstrate that
NUM is intuitive and able to estimate user satisfaction well. In addition,
ablation studies demonstrate the effectiveness of A1 and A2.

Our work may provide some guidance for designing better session-based
evaluation metrics: (1) Future metrics may need to evaluate the session as a
whole rather than aggregating each query’s result. (2) Though we only
incorporate the session-level normalization for U-measure in this study, this
proposed technique can be applied to many session-based metrics. (3) It may be
inappropriate to directly use user clicks as relevance judgment in real-world
search scenarios. (4) Intuitiveness can be a good way to evaluate session-
based metrics.

Nevertheless, our work still has some limitations that we aim to address in
future work: (1) In this work, we only verify our assumptions through some
experiments on two public datasets. Some delicate user studies are needed to
further verify the proposed assumptions and explore more assumptions about
session-based evaluation. (2) We propose two assumptions and NUM under the
condition of offline evaluation with implicit feedback. We will consider
applying NUM to the scenarios where human relevance labels are available. (3)
We propose one approach to infer session-level labels based on click-through
data in real-world search scenarios. There are more click-through data
enhancement techniques to be discovered. For example, the click dwell time and
eye-tracking information can be incorporated. (4) We propose two simple
metrics that we believe can represent the intuitiveness of session search. It
is only a primary step of exploring what metrics can serve as the golden
standard measures for the intuitiveness of session search.

## References

  * (1)
  * Al-Maskari et al. (2007) Azzah Al-Maskari, Mark Sanderson, and Paul D. Clough. 2007\.  The relationship between IR effectiveness measures and user satisfaction. In _SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007_. ACM, 773–774.  https://doi.org/10.1145/1277741.1277902
  * Chen et al. (2022a) Haonan Chen, Zhicheng Dou*, Qiannan Zhu, Xiaochen Zuo, and Ji-Rong Wen. 2022a.  Integrating Representation and Interaction for Context-Aware Document Ranking.  _ACM Trans. Inf. Syst._ (2022).  https://doi.org/10.1145/3529955
  * Chen et al. (2022b) Jia Chen, Weihao Wu, Jiaxin Mao, Beining Wang, Fan Zhang, and Yiqun Liu. 2022b.  Overview of the NTCIR-16 Session Search (SS) Task.  _Proceedings of NTCIR-16. to appear_ (2022).
  * Cleverdon et al. (1966) Cyril W Cleverdon, Jack Mills, and E Michael Keen. 1966\.  Factors determining the performance of indexing systems,(Volume 1: Design).  _Cranfield: College of Aeronautics_ 28 (1966).
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018_ , Yi Chang, Chengxiang Zhai, Yan Liu, and Yoelle Maarek (Eds.). ACM, 126–134.  https://doi.org/10.1145/3159652.3159659
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014\.  Convolutional Neural Network Architectures for Matching Natural Language Sentences. In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_. 2042–2050.
  * Huffman and Hochster (2007) Scott B. Huffman and Michael Hochster. 2007.  How well does result relevance predict session satisfaction?. In _SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007_. ACM, 567–574.  https://doi.org/10.1145/1277741.1277839
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002\.  Cumulated gain-based evaluation of IR techniques.  _ACM Trans. Inf. Syst._ 20, 4 (2002), 422–446.  https://doi.org/10.1145/582415.582418
  * Järvelin et al. (2008) Kalervo Järvelin, Susan L. Price, Lois M. L. Delcambre, and Marianne Lykke Nielsen. 2008\.  Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions. In _Advances in Information Retrieval , 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-April 3, 2008. Proceedings_ , Vol. 4956. 4–15.  https://doi.org/10.1007/978-3-540-78646-7_4
  * Joachims et al. (2007) Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search.  _ACM Trans. Inf. Syst._ 25, 2 (2007), 7\.  https://doi.org/10.1145/1229179.1229181
  * Jones and Klinkner (2008) Rosie Jones and Kristina Lisa Klinkner. 2008.  Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In _Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, Napa Valley, California, USA, October 26-30, 2008_. ACM, 699–708.  https://doi.org/10.1145/1458082.1458176
  * Kanoulas et al. (2011) Evangelos Kanoulas, Ben Carterette, Paul D. Clough, and Mark Sanderson. 2011.  Evaluating multi-query sessions. In _Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011, Beijing, China, July 25-29, 2011_. ACM, 1053–1062.  https://doi.org/10.1145/2009916.2010056
  * Kanoulas et al. (2010) Evangelos Kanoulas, Paul D. Clough, Ben Carterette, and Mark Sanderson. 2010.  Overview of the TREC 2010 Session Track. In _Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithersburg, Maryland, USA, November 16-19, 2010_ _(NIST Special Publication)_ , Vol. 500-294. National Institute of Standards and Technology (NIST).
  * Kelly (2009) Diane Kelly. 2009\.  Methods for Evaluating Interactive Information Retrieval Systems with Users.  _Found. Trends Inf. Retr._ 3, 1-2 (2009), 1–224.  https://doi.org/10.1561/1500000012
  * Kendall (1938) Maurice G Kendall. 1938\.  A new measure of rank correlation.  _Biometrika_ 30, 1/2 (1938), 81–93.
  * Li et al. (2009) Jane Li, Scott B. Huffman, and Akihito Tokuda. 2009\.  Good abandonment in mobile and PC internet search. In _Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009_. ACM, 43–50.  https://doi.org/10.1145/1571941.1571951
  * Lipani et al. (2019) Aldo Lipani, Ben Carterette, and Emine Yilmaz. 2019\.  From a User Model for Query Sessions to Session Rank Biased Precision (sRBP). In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2019, Santa Clara, CA, USA, October 2-5, 2019_. ACM, 109–116.  https://doi.org/10.1145/3341981.3344216
  * Liu et al. (2018) Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, and Shaoping Ma. 2018.  Towards Designing Better Session Search Evaluation Metrics. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_. ACM, 1121–1124.  https://doi.org/10.1145/3209978.3210097
  * Liu et al. (2019) Mengyang Liu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019.  Investigating Cognitive Effects in Session-level Search User Satisfaction. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_. ACM, 923–931.  https://doi.org/10.1145/3292500.3330981
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017\.  Learning to Match using Local and Distributed Representations of Text for Web Search. In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_. ACM, 1291–1299.  https://doi.org/10.1145/3038912.3052579
  * Moffat and Zobel (2008) Alistair Moffat and Justin Zobel. 2008.  Rank-biased precision for measurement of retrieval effectiveness.  _ACM Trans. Inf. Syst._ 27, 1 (2008), 2:1–2:27.  https://doi.org/10.1145/1416950.1416952
  * Price et al. (2007) Susan Price, Marianne Lykke Nielsen, Lois M. L. Delcambre, and Peter Vedsted. 2007\.  Semantic components enhance retrieval of domain-specific documents. In _Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, CIKM 2007, Lisbon, Portugal, November 6-10, 2007_. ACM, 429–438.  https://doi.org/10.1145/1321440.1321502
  * Qu et al. (2020) Chen Qu, Chenyan Xiong, Yizhe Zhang, Corby Rosset, W. Bruce Croft, and Paul Bennett. 2020\.  Contextual Re-Ranking with Behavior Aware Transformers. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_. ACM, 1589–1592.  https://doi.org/10.1145/3397271.3401276
  * Sakai (2012) Tetsuya Sakai. 2012\.  Evaluation with informational and navigational intents. In _Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012_. ACM, 499–508.  https://doi.org/10.1145/2187836.2187904
  * Sakai and Dou (2013) Tetsuya Sakai and Zhicheng Dou. 2013.  Summaries, ranked retrieval and sessions: a unified framework for information access evaluation. In _The 36th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’13, Dublin, Ireland - July 28 - August 01, 2013_. ACM, 473–482.  https://doi.org/10.1145/2484028.2484031
  * Spearman (1961) Charles Spearman. 1961\.  The proof and measurement of association between two things.  (1961).
  * Trauzettel-Klosinski et al. (2012) Susanne Trauzettel-Klosinski, Klaus Dietz, IReST Study Group, et al. 2012\.  Standardized assessment of reading performance: The new international reading speed texts IReST.  _Investigative ophthalmology & visual science_ 53, 9 (2012), 5452–5461.
  * van Dijk et al. (2019) David van Dijk, Marco Ferrante, Nicola Ferro, and Evangelos Kanoulas. 2019.  A Markovian Approach to Evaluate Session-Based IR Systems. In _Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part I_ , Vol. 11437. Springer, 621–635.  https://doi.org/10.1007/978-3-030-15712-8_40
  * Wang et al. (2013) Hongning Wang, Yang Song, Ming Wei Chang, Xiaodong He, Ryen W. White, and Wei Chu. 2013\.  Learning to extract cross-session search tasks.  _WWW 2013 - Proceedings of the 22nd International Conference on World Wide Web_ (2013), 1353–1363.  https://doi.org/10.1145/2488388.2488507
  * Wicaksono and Moffat (2021) Alfan Farizki Wicaksono and Alistair Moffat. 2021.  Modeling search and session effectiveness.  _Inf. Process. Manag._ 58, 4 (2021), 102601.  https://doi.org/10.1016/j.ipm.2021.102601
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_. ACM, 55–64.  https://doi.org/10.1145/3077136.3080809
  * Yang and Lad (2009) Yiming Yang and Abhimanyu Lad. 2009.  Modeling Expected Utility of Multi-session Information Distillation. In _Advances in Information Retrieval Theory, Second International Conference on the Theory of Information Retrieval, ICTIR 2009, Cambridge, UK, September 10-12, 2009, Proceedings_ _(Lecture Notes in Computer Science)_ , Vol. 5766. Springer, 164–175.  https://doi.org/10.1007/978-3-642-04417-5_15
  * Zhang et al. (2020) Fan Zhang, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Min Zhang, and Shaoping Ma. 2020\.  Cascade or Recency: Constructing Better Evaluation Metrics for Session Search. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_. ACM, 389–398.  https://doi.org/10.1145/3397271.3401163
  * Zhu et al. (2021) Yutao Zhu, Jian-Yun Nie, Zhicheng Dou, Zhengyi Ma, Xinyu Zhang, Pan Du, Xiaochen Zuo, and Hao Jiang. 2021\.  Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_. ACM, 2780–2791.  https://doi.org/10.1145/3459637.3482243
  * Zuo et al. (2022) Xiaochen Zuo, Zhicheng Dou, and Ji-Rong Wen. 2022\.  Improving Session Search by Modeling Multi-Granularity Historical Query Change. In _WSDM ’22, The Fifteenth ACM International Conference on Web Search and Data Mining, February 21-25, 2022, Tempe, AZ, USA_. ACM.  https://doi.org/10.1145/3488560.3498415