# Unsupervised Explanation Generation for Machine Reading Comprehension

Yiming Cui1,2, Ting Liu1, Shijin Wang2,3, Guoping Hu2,
1Research Center for SCIR, Harbin Institute of Technology, Harbin, China
2State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China
3iFLYTEK AI Research (Hebei), Langfang, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

With the blooming of various Pre-trained Language Models (PLMs), Machine
Reading Comprehension (MRC) has embraced significant improvements on various
benchmarks and even surpass human performances. However, the existing works
only target on the accuracy of the final predictions and neglect the
importance of the explanations for the prediction, which is a big obstacle
when utilizing these models in real-life applications to convince humans. In
this paper, we propose a self-explainable framework for the machine reading
comprehension task. The main idea is that the proposed system tries to use
less passage information and achieve similar results compared to the system
that uses the whole passage, while the filtered passage will be used as
explanations. We carried out experiments on three multiple-choice MRC
datasets, and found that the proposed system could achieve consistent
improvements over baseline systems. To evaluate the explainability, we
compared our approach with the traditional attention mechanism in human
evaluations and found that the proposed system has a notable advantage over
the latter one.

##  1 Introduction

Machine Reading Comprehension (MRC) is a task that to read and comprehend
passages and answer relevant questions, requiring a deep understanding of the
passage and question as well as their relationships. With the latest pre-
trained language models, some of the systems could surpass human performance.
While massive efforts have been made, most of the previous works were focusing
on designing more sophisticated systems to achieve better evaluation scores
and neglect the importance of the explainability of the system, which is
crucial for the human to understand the decision process inside black-box
artificial intelligent models.

As a result, Explainable Artificial Intelligence (XAI) (Gunning, 2017)
received much more attention in these years than ever, which aims to produce
more explainable machine learning models while preserving a high accuracy of
the model output and let the humans understand its intrinsic mechanism. To
improve the explainability of the AI, one could seek decomposability of the
traditional machine learning model (such as decision trees, rule-based
systems, etc.), or use post-hoc techniques for deep learning models (Barredo
Arrieta et al., 2019).

For the machine reading comprehension task, we could annotate additional
explanations to the questions and let the model simultaneously learn to
predict both the final answer and its explanation. This is similar to HotpotQA
(Yang et al., 2018), which is a multi-hop MRC dataset, where it requires the
machine to predict both the final answer span and its supporting facts.
However, this requires a large amount of labeled data for the machine learning
system to learn a reliable model, which is much time-consuming and expensive.

To both address the explainability of the machine reading comprehension
systems and alleviate human annotations, in this paper, we propose a novel
self-explainable mechanism called Recursive Dynamic Gating (RDG) to the
existing models without relying on additional annotated data. The RDG
mechanism injects into the pre-trained language model and applies a filter in
the input representations of each transformer layer to control the amount of
the information. Moreover, to improve the accuracy of the gating, we apply a
convolutional operation to mimic the N-gram summarization of the local
context. In the inference stage, we extract the gating values to represent the
selections of the input words as explanations. The contributions of this paper
are listed as follows.

  * •

To improve the explainability of the MRC model, we propose a self-explainable
mechanism called Recursive Dynamic Gating (RDG) strategy to control the amount
of information that flows into each transformer layer.

  * •

The results on several multiple-choice machine reading comprehension datasets
show that the proposed method could consistently improve objective benchmarks.

  * •

Through careful human evaluation, the proposed method shows a notable
advantage over the traditional attention mechanism in explainability.

##  2 Related Work

Machine Reading Comprehension (MRC) is similar to the traditional question
answering, but put more emphasis on the comprehension of the passage. Various
datasets and models have been proposed (Dhingra et al., 2017; Kadlec et al.,
2016; Cui et al., 2017) since the emerge of the cloze-style reading
comprehension (Hermann et al., 2015; Hill et al., 2015), which is regarded as
the starting point of the recent MRC research. Later, the release of SQuAD
(Rajpurkar et al., 2016) becomes a milestone in MRC, which is a span-
extraction reading comprehension task, and greatly accelerated the research in
this area. With the blooming of the recent pre-trained language models, such
as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al.,
2019b), ALBERT (Lan et al., 2019), etc., some of the systems could surpass
human performance on several MRC datasets.

However, though the benchmarks have been refreshed to such a high level, the
decision process and the explanations of these artifacts still remains
unclear, raising concerns about the reliability of the artificial intelligence
systems. In this context, Explainable Artificial Intelligence (XAI) becomes
more important than ever. Conventional machine learning approaches, such as
decision tree, could be decomposed into ‘explainable’ components to reveal its
explainability. However, this is not the case for the recent machine learning
approaches, especially in natural language processing, where the models are
usually composed of deep neural networks, such as BERT. To better understand
the internal mechanism of BERT-based architecture, Kovaleva et al. (2019)
discovered that, there are repetitive attention patterns across different
heads in the multi-head attention mechanism indicating its over-
parametrization.

To achieve multi-hop and explainable question answering, HotpotQA (Yang et
al., 2018) is proposed, which requires the machine to not only give the final
answer but also extract the evidence sentences in the passages. Following this
dataset, various models have been proposed (Yang et al., 2018; Shao et al.,
2020) to improve the overall accuracy. Liu et al. (2019a) proposed to build a
generative explanation framework for the text classification with two new
datasets that contain summaries, rating scores, and fine-grained reasons.
However, most of these works rely on human annotations in order to do
supervised learning, which will be a challenge in real-life applications.

Another direction is to seek a self-explainable system that does not require
prior annotations for the explanations. For example, Perez et al. (2019)
proposes to train an evidence agent to select the passage sentences that most
convince the MRC model of a given answer using a subset of the passage. Moon
et al. (2019) propose a Memory Graph Network to enable dynamic expansion of
memory slots through graph traversals to improve explainability in QA
reasoning. Wang et al. (2019) applied distant supervision to generate pseudo
evidence sentence labels and refined them through a deep probabilistic logic
learning framework.

In this paper, we focus on designing a self-explanatory system for the machine
reading comprehension task. Inspired by Information Bottleneck (IB) principle
(Tishby et al., 2000), we propose a novel mechanism called Recursive Dynamic
Gating (RDG) to achieve better explainability for the BERT-like pre-trained
language models without any additional annotations. Unlike previous works, our
model aims to achieve better explainability of the answer while preserving or
even surpass the original system in the objective evaluations. Instead of
using a neural network model to filter a subset of the sentences in the
passage, which could be non-differentiable and requires additional makeup
(such as Gumbel-Softmax reparameterization (Jang et al., 2017)), we seek to
control the amount of passage information in the hidden representations
directly to avoid such circumstances.

Figure 1:  System overview of the Recursive Dynamic Gating (RDG) mechanism for
machine reading comprehension task. Note that, minor operations are omitted
for simplicity.

Unlike the attention mechanism (Bahdanau et al., 2015) in traditional non-
transformer models, the attention information will be separated into several
subsets in multi-head attention (Vaswani et al., 2017), which is the key
component in transformer-based models. Though the multi-head attention has
proved to be effective, it is harder for the human to get a comprehensive view
of which part of the passage that the model attended to. As a remedy, our RDG
mechanism provides a straightforward explanation of the model prediction,
which could be more friendly in the human view.

##  3 The Approach

###  3.1 Overview

In this paper, we focus on the multiple-choice machine reading comprehension
(MC-MRC) task, which requires the machine to read a passage and choose a
correct answer among several candidates for the given question. Unlike SQuAD
(Rajpurkar et al., 2016), the answer for the MC-MRC task is not an exact span
in the passage, where the human needs to know the explanations for the answer
predictions.

We will overview our model on a general basis, to better understand its
mechanism. The overall neural architecture of our model is depicted in Figure
1. The primary idea is to train a two-pass system, including the original
fine-tuning system (teacher) and an explanatory system (student), which are
jointly trained.

  * •

Teacher: uses the original input to learn the relations between the passage,
question, and candidate answers.

  * •

Student: utilizes the RDG mechanism to filter the passage information in the
spirit of using less passage information to achieve similar results to the
teacher system.

In the inference stage, we use the teacher system to predict the final answer
and use the student system to generate explanations.

###  3.2 Formulation for ALBERT

In this paper, we use ALBERT (Lan et al., 2019) as the backbone of our model
for its effectiveness in machine reading comprehension as well as other NLP
tasks. We omit rather extensive full formulations for ALBERT and only shows
the high-level architecture. Given an input sequence $X=\\{x_{1},..,x_{l}\\}$
($l$ is the maximum sequence length), ALBERT model will convert it into a
contextualized representations $H_{L}\in\mathbb{R}^{l*h}$ through an embedding
layer (which consists of word embedding, positional embedding, and token type
embedding), and a consecutive $L$-layer transformer. Note that the weight for
each transformer layer is identical through cross-layer parameter sharing.
Also, the teacher and student ALBERT model shares the same parameters with all
transformer layers initialized by the pre-trained ALBERT weight.

$\displaystyle H_{0}=\mathbf{Embedding}(X)$ (1) $\displaystyle
H_{i}=\mathbf{Transformer}(H_{i-1}),i=\\{1,...,L\\}$ (2)

###  3.3 Recursive Dynamic Gating

Our recursive dynamic gating $R\in\mathbb{R}^{m}$ applies on the input
representation of each transformer layer to dynamically control the
information flow to the next layer, which mimics the process of multi-hop
reading strategy. Specifically, for $i$th transformer layer, we rewrite the
Equation 2 as follows.

$\displaystyle H_{i}=\mathbf{Transformer}(H_{i-1}*R_{i})$ (3)

Given previous transformer layer output representation $H_{i-1}$, we will
construct recursive dynamic gating $R_{i}$ for $i$th layer as follows. We
first feed $H_{i-1}$ into a 1-layer transformer and squeeze the output into a
scalar with respect to each word, indicating their importance, which is
similar to the attention mechanism. Note that, the transformer model for
gating mechanism is also initialized with pre-trained weights but does not
share with other transformer layers in ALBERT model.

$\displaystyle R^{\prime}_{i}={\bf w}^{T}\cdot\mathbf{Transformer}(H_{i-1})$
(4)

To let the resulting gating value aware of its adjacent words, we propose a
smoothing technique based on the convolutional operation. We use a
convolutional filter to dynamically summarize the gating value in a fixed
window. Following Kim (2014), we apply the 1D convolution operation, which is
commonly used in natural language processing tasks. To keep the sequence
length unchanged, we apply the same-length convolution with a kernel size of
$k$. Inspired by Cui et al. (2019), we apply a residual connection to
‘highlight’ original word in place. To keep consistent with original
transformer implementation, we apply Gaussian Error Linear Unit (GELU)
(Hendrycks and Gimpel, 2016) for activation, where we clipped activation
output in the range of $[0,1]$.111While it is natural to use sigmoid or
softmax function, we did not find additional gains by using these functions.

$\displaystyle\hat{R}_{i}=\mathbf{GELU}(conv(R^{\prime}_{i},k)+R^{\prime}_{i})$
(5)

As the gating mechanism will diminish the input representations (in extreme
cases, all representation will be erased, i.e., multiplied by zeros), to avoid
sudden severe changes, we apply a recursive learning approach. In this paper,
we investigate two variants: recursive history (Eq. 6a) and constant history
(Eq. 6b), where $I\in\mathbb{R}^{\mathcal{\\\ }l}$ represents the input mask
(padding positions are 0 and others are 1). The final gating value will be
determined by both historical gating value $R_{i-1}$ and the calculated
current gating value $R^{\prime}_{i}$, with a history decay coefficient
$\gamma\in(0,1)$. We observe that the recursive history (Eq. 6a) is better for
the larger dataset as the gating mechanism will be learned much adequately,
while the constant history (Eq. 6b) is suitable for smaller dataset.

$\displaystyle\tilde{R}_{i}=\gamma^{i}*R_{i-1}+(1-\gamma^{i})*\hat{R}_{i}$
(6a) $\displaystyle\tilde{R}_{i}=\gamma^{i}*I+(1-\gamma^{i})*\hat{R}_{i}$ (6b)

As our aim is to filter the passage, we apply a mask to the $\tilde{R}_{i}$ to
preserve the gating values in the passage and set constant ones in the
question and choice to protect their representations from diminishing. In
practice, this could be achieved by the calculations between the input mask
$I$ and token type ids $T$ (where the positions of the question and choices
are set to 1, and others are 0).

$\displaystyle R_{i}=\tilde{R}_{i}*I*(1-T)+T$ (7)

In this way, we’ve injected recursive dynamic gating into the ALBERT model.
Finally, we apply a linear transformation ${\bf
w_{o}}\in\mathbb{R}^{\mathcal{\\\ }h}$ to get final logits for both teacher
and student model, as shown below, where $H_{T}$ and $H_{S}$ represents their
final output of ALBERT model.

$\displaystyle p_{T}={\bf w_{o}}H_{T},~{}~{}~{}p_{S}={\bf w_{o}}H_{S}$ (8)

System | RACE | DREAM | C3
---|---|---|---
Dev | Test | Dev | Test | Dev | Test
ALBERT-base (Zhu et al., 2020) | - | - | 64.5 | 64.4 | - | -
ALBERT-base + DUMA (Zhu et al., 2020) | - | - | 67.1 | 67.6 | - | -
ALBERT-base + TB-DUMA (Zhu et al., 2020) | - | - | 67.8 | 67.2 | - | -
ALBERT-base (Xu et al., 2020) | - | - | - | - | 60.4 | 59.6
ALBERT-base (Lan et al., 2019) | - | 66.8 | - | - | - | -
ALBERT-base (re-run) | 71.1 (70.8) | 69.7 (69.1) | 68.0 (67.1) | 66.7 (66.1) | 60.3 (60.1) | 60.0 (59.5)
ALBERT-base + RDG (constant) | 70.9 (70.6) | 70.1 (69.6) | 69.1 (67.8) | 68.4 (67.1) | 60.9 (60.2) | 60.6 (60.2)
ALBERT-base + RDG (recursive) | 72.3 (71.4) | 71.4 (70.3) | 68.1 (67.4) | 68.2 (66.9) | 60.5 (59.8) | 60.1 (59.7)
Table 1:  Experimental results on RACE, DREAM, and C3. The average score is
depicted in brackets.

###  3.4 Training Objective

The training objective is constructed in three parts.

$\displaystyle\mathcal{L}=\mathcal{L}_{ce}+\mathcal{L}_{kd}+\mathcal{L}_{cos}$
(9)

First, we will maximize the output for both teacher and student using standard
cross-entropy loss, as shown below, where $p^{\prime}_{T}$ and
$p^{\prime}_{S}$ is the softmaxed values respectively.

$\displaystyle\mathcal{L}_{ce}=-\frac{1}{2N}\sum\limits_{i=1}^{N}(y_{T}\log(p^{\prime}_{T})+y_{S}\log(p^{\prime}_{S}))$
(10)

Compared to the teacher, the student will use a filtered input for each
transformer layer and thus will learn much slower. In this case, we apply an
additional knowledge distillation (KD) loss, specifically MSE loss in this
paper, between the logits of the teacher and student model to accelerate the
learning, as shown below, where the $T$ is the temperature argument.

$\displaystyle\mathcal{L}_{mse}=\frac{1}{N}\sum\limits_{i=1}^{N}(\frac{p_{T}}{T}-\frac{p_{S}}{T})^{2}$
(11)

By only using the first two losses are not enough, as the model will only
optimize for the final accuracy and could result in all-one gating values for
any input, which will be identical to the teacher model and will not be
informative in explainability (see Section 4.3 for further discussion). In
this context, at last, we calculate the cosine similarity between the input
mask $I\in\mathbb{R}^{\mathcal{\\\ }l}$ and the final gating value $R_{L}$ to
encourage the model to achieve better scores (cross-entropy and KD loss) with
less input information (cosine similarity loss).

$\displaystyle\mathcal{L}_{cos}=\frac{1}{N}\sum\limits_{i=1}^{N}cosine(I,R_{L})$
(12)

##  4 Experiments

###  4.1 Experimental Setups

We evaluate our approach on three multiple-choice machine reading
comprehension datasets, including RACE (Lai et al., 2017), DREAM (Sun et al.,
2019), and C3 (Sun et al., 2020), where the last one is a Chinese MRC dataset.
Unlike the span-extraction machine reading comprehension (such as well-known
SQuAD (Rajpurkar et al., 2016)), where the answer is an exact span in the
passage, the multiple-choice machine reading comprehension requires to select
correct answer among several candidates, which is not directly picked from the
original passage and is suitable to test the explainability of our model. The
statistics of these datasets are shown in Table 2. Note that, for RACE and
DREAM datasets, there are always four candidate answers. However, for C3
dataset, the number of the candidate answers ranges from 2 to 4, and we
manually add dummy candidates to those samples that have less than four
candidates. As indicated in the previous section, we use the recursive history
for the RACE and the constant history for the other datasets.

| RACE | DREAM | C3
---|---|---|---
Train # | 87,866 | 6,116 | 8,023
Dev # | 4,887 | 2,040 | 2,674
Test # | 4,934 | 2,041 | 2,672
Choices # per question | 4 | 4 | 2 ~ 4
Gating decay rate | 0.9 | 0.8 | 0.8
Conv kernel size | 3 | 3 | 5
Initial learning rate | 2e-5 | 1e-5 | 2e-5
Training steps | 1,200 | 1,000 | 2,000
Table 2:  Data statistics and hyper-parameter settings.

We also list the most important hyper-parameter settings for each task in
Table 2. Following original ALBERT implementation, for all tasks, we use a
maximum sequence length of 512, and classifier dropout of 0.1, and ALBERT
dropout of 0. The temperature of the MSE loss is set to 8, which is a common
choice in knowledge distillation settings, as suggested in Yang et al. (2020).
We use Adam (Kingma and Ba, 2014) with weight decay optimizer with a warm-up
rate of 0.1. The training batch size is set to 32. We use ALBERT-base-v2
checkpoint for English and ALBERT-base-zh checkpoint for Chinese to initialize
all transformer layers.222https://github.com/google-research/albert To ensure
the stability of the results, for each experiment, we perform ten training
trials with different random seeds and report the maximum and average scores.
Our implementation is based on the TensorFlow (Abadi et al., 2016) version of
run_race.py provided by ALBERT. All models are trained on Cloud TPU v2 with
64GB HBM.

Score |  Description | Attention | RDG
---|---|---|---
5-Excellent |  Sufficient explanations, very few redundant text segments | 1% | 46%
4-Acceptable |  Sufficient explanations, contains a few redundant text segments but not exceed the core explanations | 17% | 34%
3-Marginal |  Almost sufficient explanations, but requires the further information in the original passage | 21% | 10%
2-Trivial |  Only a few explanations with redundant text segments that exceed the core explanations | 29% | 8%
1-Wrong |  No informative explanations provided, or completely unreadable | 32% | 2%
Weighted Average Score | 2.26 | 4.14
Table 3:  Human evaluation on answer explainability over 100 correctly
predicted samples in the RACE test set. Importance: Usability $>$ Redundancy
$>$ Readability

###  4.2 Results

We will evaluate our model with respect to the objective benchmarks to see if
the teacher model could benefit from a student that is good at extracting
explanations. The overall experimental results are shown in Table 1. To ensure
a fair comparison, we re-run the baseline system and yields better results
(except for the development set of C3) than previous state-of-the-art systems
on the ALBERT-base model. On top of our baseline system, as we can see that,
our model yields stable and consistent improvements in all three datasets,
indicating that by simultaneously learning the teacher and student model could
improve the overall prediction accuracy. Compared to the best scores in RACE,
the accuracy of the student output is 67.8/66.8, which is lower than the
teacher system, and thus we stick to use the teacher system to make the final
prediction.333We also tried to combine the output of both teacher and student
system, but did not find additional gains. As the main body system, i.e.,
ALBERT model, is shared between the teacher and student, the learning of the
student will also ‘regularize’ the teacher, and thus the final prediction of
the teacher system will improve against the baseline. Also we have found that
recursive history mechanism (Eq. 6a) is efficient for the larger dataset
(i.e., RACE), which demonstrate that the learning for the history requires
more training data.

###  4.3 Ablation Study

We ablate critical components in our RDG system to separately evaluate their
contributions to the whole system. Specifically, we ablate the RDG system with
recursive history on RACE dataset. Note that, unlike most of the previous
works that only present a single-run in ablations, in this paper, all ablation
experiments are also conducted ten times, and we report their average scores
to ensure the reliability of the results, as shown in Table 4.

| Dev | Test | Avg
---|---|---|---
ALBERT-base+RDG | 71.4 | 70.3 | 70.9
w/o cosine loss | 71.3 | 70.2 | 70.8
w/o KD loss | 71.3 | 69.7 | 70.5
w/o conv smoothing | 71.0 | 70.0 | 70.5
w/o historical gating | 70.9 | 69.8 | 70.4
ALBERT-base | 70.8 | 69.1 | 70.0
Table 4:  Results of model ablations on RACE.

Historical gating and convolutional smoothing are the most important
components in our RDG mechanism, demonstrating that it is necessary to build a
well-designed architecture for the gating mechanism to effectively filter the
information. The KD loss seems to also be effective in our system, indicating
that the KD loss will accelerate the learning of the student system. As a
result, the accuracy of student output will rise from 60.8/60.3 to 67.8/66.8
(ALBERT-base+RDG, as indicated in the previous section), which also verify
this assumption. Eliminating the cosine loss seems to be less effective in
improving the final performance. After careful examination, when removing the
cosine loss, the gating values for most of the words will be one and RDG loses
its explainability, as the student will learn to let all information pass to
the transformer layers to maximize the objective performance instead of
pointing out the key components. This indicates that though the cosine loss
does not improve the overall accuracy, it is essential to achieve better
explainability in our RDG system.

Passage 1: water is very important to living things. without water there can
be no life on the earth. all animals and plants need water. man also needs
water. we need water to drink, to cook our food and to clean ourselves. water
is needed in offices, factories and schools. water is needed everywhere. there
is water in seas, rivers and lakes. water is found almost everywhere. even in
the desert part of the world, there is some water in the air. you can not see
or feel it when it is a part of the air. the water in the seas, rivers and
lakes is a liquid, the water in the air is a gas, and we call it water vapour
. clouds are made of water. they may be made of very small drops of water.
they may also be made of snow crystals . snow crystals are very very small
crystals of ice. ice is frozen water. it is a solid. there can be snow and ice
everywhere in winter. water may be a solid or a liquid or a gas. when it is a
solid, it may be as hard as a stone. when it is a liquid, you can drink it.
when it is a gas, you can not see or feel it.
---
Question |  Candidates |  Attention |  RDG
where can we find water? |  A: we can find water when it turns into vapor B: water is only in seas and rivers C: we can see water in deserts here and there D: water can be found almost everywhere |  water is very important to living things. without water. can be no life on…is needed everywhere. there… feel it. |  … rivers and lakes. water is found almost everywhere…
clouds are made of |  A: seas, rivers and lakes B: blocks of ice C: very small drops of water or snow crystals D: solid, liquid and gas |  water is very important to living things… you can not see or… winter. water may be… you can drink it…feel it. |  …they may be made of very small drops of water. they may also be made of snow…
Passage 2: is it important to have breakfast every day? a short time ago, a
test was given in the united states. people of different ages, from 12 to 83,
were asked to have a test. during the test, these people were given all kinds
of breakfast, and sometimes they got no breakfast at all. scientists wanted to
see how well their bodies worked after eating different kinds of breakfast.
the results show that if a person eats a right breakfast, he or she will work
better than if he or she has no breakfast. if a student has fruit, eggs, bread
and milk before going to school, he or she will learn more quickly and listen
more carefully in class. some people think it will help you lose weight if you
have no breakfast. but the result is opposite to what they think. this is
because people become so hungry at noon that they eat too much for lunch. they
will gain weight instead of losing it.
Question |  Candidates |  Attention |  RDG
what were the people given during the test? |  A: no breakfast at all. B: very rich breakfast. C: little food for breakfast. D: different foods or sometimes none. |  is it important … every day? a short … a test.during the test, these people were given all kinds of… losing it. |  … given all kinds of breakfast, and sometimes they got no breakfast at all…
what do the results show? |  A: breakfast has affected on work and studies. B: breakfast has little to do with a person’s work. C: will work better if he only has fruit and milk. D: girl students should have less for breakfast. |  is it important… every day? a short… of breakfast. the results show that if… no breakfast.if a… result is opposite to what… losing it. |  …show that if a person eats a right breakfast, he or she will work better than if… will learn more quickly and…
Passage 3: traditionally, women have fallen behind men in adoption of internet
technologies, but a study released yesterday by the pew internet & american
life project found that women under age 65 now get ahead of men in internet
usage, though only by a few percentage points. but the survey also noted that
the between women and men on the web is even greater among the 18 to 29 age
group and african americans. the report, ”how women and men use the internet”,
examined use by both sexes, looking at what men and women are doing online as
well as their rate of adopting new web-based technologies. the report found
that 86 percent of women aged 18 to 29 were online, compared with 80 percent
of men in the same age group. among african americans, 60 percent of women are
online, compared with 50 percent of men. in other age groups, the disparity is
only slight, with women outpacing men by 3 percentage points. however, among
the older group, those age 65 and older, 34 percent of men are online,
compared with 21 percent of women. men tend to use the web for information and
entertainment, getting sports scores and stock quotes and downloading music,
while women tend to be heavier users of mapping and direction services, and
communication services such as email.
Question |  Candidates |  Attention |  RDG
among african americans, women |  A: equal men in the use of the internet . B: use the internet less than men . C: use the internet more than men . D: use the internet better than men . |  … age 65 now get ahead of men…, though only by a… 86 percent of women aged. to 29 were online,… women. men tend … e-mail. |  …, 60 percent of women are online, compared with 50 percent of men …
Table 5:  Explainability comparisons of the attention mechanism and our RDG
system on correctly predicted samples in the RACE test set. The ‘…’ indicate
the omitted passage words. Ground truths are depicted in boldface.

##  5 Explainability Evaluation

###  5.1 Human Evaluation

Though our RDG system yields better performance in the objective benchmarks,
our primary goal is to improve the explainability of the MRC systems. Unlike
the explanation system with supervised learning, which could be evaluated
through the comparisons between the prediction and ground truth, in this
paper, we carried out human evaluations to demonstrate the explainability of
the proposed method, as no ground truths are available. We compared our RDG
approach with the traditional in-built multi-head attention mechanism to
evaluate which one provides better clues for the answer. We perform the
following steps for each system. Note that we extract these explanations with
respect to the ground truth candidate.

  * •

Attention: as in-built multi-head attention in BERT-like language models are
in 4D (batch, num_att_heads, seq_length, seq_length_softmax), we perform the
average operation through attention head dimension and sequence length
dimension to get a flattened 2D attention score. We only print out attention
scores in the passage.444We also tried to examine all the attention heads
(12*12=144), but did not find a specific head for this purpose across
different samples.

  * •

RDG: we directly extract the final gating values to represent the importance
of each input word.

After obtaining these scores, in descending order, we select the top 10 words
and their contextual words ($\pm 2$ positions) to demonstrate the
explainability. We randomly select 100 questions among the correctly predicted
samples in the RACE test set. The annotators are instructed to first read the
question and candidate answers, and then read the explanations of two systems.
When the explanations are not readable, the annotators are allowed to see the
original passage to assess their usability.

According to the quality of the explanations for the correct answer, the
annotator will grade two systems, ranging from 1-wrong to 5-excellent. The
results of the human evaluation, as well as the standard for each score, are
listed in Table 3. As we can see that the proposed RDG system could
substantially improve the explainability over traditional in-built attention
mechanism, with an average score of 4.14, which is ‘acceptable’. The RDG
system will extract much more meaningful explanations with 80% for
‘acceptable‘ and ‘excellent’ grades, while the traditional attention mechanism
only takes 18%.

We find the traditional attention mechanism is not that straightforward for
the human to explicitly know the explanations for the answer, as it is hard to
find a ‘conclusive’ head for the overall attention in the multi-head
mechanism. Nonetheless, as the multi-head attention mechanism will implicitly
learn the semantic, dependency, and other linguistic knowledge in each head,
it is interesting to see if we could utilize them to generate much more
meaningful explanations for humans, and we will leave it for future work.

###  5.2 Case Analysis

We select several cases to analyze the linguistic aspect of the generated
explanations, as shown in Table 5. As we can see that the explanations
generated by the RDG system substantially outperform the traditional attention
mechanism in terms of the usability, readability, and variety, where we
conclude our observations as follows.

  * •

Usability: The most important aspect is the usability of the explanations. Our
RDG system could give relatively informative explanations to the answer in
terms of the keywords. For example, in passage 1 question 1, both explanations
could identify the keyword ‘everywhere’ in the option D. However, from the
original passage, we can know that the word ‘everywhere’ in the explanation of
the attention indicate ‘water is needed everywhere’, which is not accordance
with the question, while our RDG system correctly identifies the explanation
‘water is found almost everywhere’. In another case in the passage 2 question
2, our RDG system could also extract clues in discontinued span, where ‘work
and studies’ in the candidate A mapped to two discontinued span ‘work better’
and ‘learn more quickly’ in the original passage.

  * •

Readability: The explanations generated by attention contain more text
segments than the RDG system, and some of them are hard for the human to
understand. On the contrary, the RDG system could generate more fluent
explanations. As the explanations are identified not only by the top-$k$ words
but also their context, it could be inferred that our model could generate
more concentrated explanations rather than scattered ones.

  * •

Variety: The explanations generated by attention show a strong pattern that
the most front and end words are always be chosen, which is the closest term
to [CLS] and [SEP] tokens in ALBERT (or other BERT-like models). For example,
in passage 1, all explanations generated by the attention mechanism include
the first sentence ‘water is very important to living things’. This
observation is similar to the previous study by Kovaleva et al. (2019), that a
majority of attention maps show strong attention to the [CLS] and [SEP] or the
token itself. However, our RDG system will generate varied explanations
according to different questions instead of meaningless special tokens or
repetitive expressions in the human view.

It is worth mentioning that not all cases are explainable by ‘text matching’
or ‘semantic matching’. In passage 3, there is no obvious sentences that
matches the correct candidate answer ‘use the internet more than men’.
However, our RDG system could identify the correct clues for the answer,
mentioning the higher percentage of women than men in the internet usage.
While the attention mechanism failed to identify the correct explanations.

###  5.3 Negative Sample Analysis

In the previous sections, we show and analyze the explanations for the
correctly predicted samples. However, though the explanations for the wrongly
predicted sample is less informative to the human, we also wonder if we could
find clues for these samples. We also manually select 100 wrongly predicted
samples and see whether the explanations corresponding to the ground truth
candidate answer are informative to evaluate their potentials in choosing a
better candidate. The results show that the attention system only has 18%
samples gives reasonable explanations for the correct answer, while our RDG
system has a high percentage of 56%, indicating that our system has a better
correlation between the ground truth and its explanations in the passage.

##  6 Conclusion

In this paper, we aim to improve the explainability for the machine reading
comprehension task, which is different from most of the previous works that
were only striving for better objective evaluation scores. To achieve this
goal, we propose a novel mechanism called Recursive Dynamic Gating (RDG) to
gradually refine the amount of the input information in each layer of the pre-
trained language model. Also, we propose an attention smoothing technique that
will increase the accuracy of the RDG mechanism. Experimental results on three
multiple-choice machine reading comprehension datasets show that the proposed
RDG mechanism could not only improve the objective evaluation scores, but also
show an advantage over the traditional attention mechanism in explainability.

As the explainability will become an important topic in future research for
machine reading comprehension as well as more generalized natural language
processing tasks, in the future, we are going to further utilize the
explanation segments as a source to generate more fluent and comprehensive
explanations.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: A system for large-scale machine learning.  In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_ , pages 265–283.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _Proceedings of ICLR_.
  * Barredo Arrieta et al. (2019) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2019.  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI.  _arXiv e-prints_ , page arXiv:1910.10045.
  * Cui et al. (2017) Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017.  Attention-over-attention neural networks for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 593–602. Association for Computational Linguistics.
  * Cui et al. (2019) Yiming Cui, Wei-Nan Zhang, Wanxiang Che, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2019.  Contextual recurrent units for cloze-style reading comprehension.  _arXiv preprint arXiv:1911.05960_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhingra et al. (2017) Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017.  Gated-attention readers for text comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1832–1846. Association for Computational Linguistics.
  * Gunning (2017) David Gunning. 2017.  Explainable artificial intelligence (xai).  _Defense Advanced Research Projects Agency (DARPA), nd Web_ , 2.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems_ , pages 1684–1692.
  * Hill et al. (2015) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015.  The goldilocks principle: Reading children’s books with explicit memory representations.  _arXiv preprint arXiv:1511.02301_.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparametrization with gumble-softmax.  In _ICLR 2017_.
  * Kadlec et al. (2016) Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016.  Text understanding with the attention sum reader network.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 908–918. Association for Computational Linguistics.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of EMNLP 2014_ , pages 1746–1751. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 796–805. Association for Computational Linguistics.
  * Lan et al. (2019) Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _ArXiv_ , abs/1909.11942.
  * Liu et al. (2019a) Hui Liu, Qingyu Yin, and William Yang Wang. 2019a.  Towards explainable NLP: A generative explanation framework for text classification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5570–5581, Florence, Italy. Association for Computational Linguistics.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Moon et al. (2019) Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019.  Memory graph networks for explainable memory-grounded question answering.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 728–736, Hong Kong, China. Association for Computational Linguistics.
  * Perez et al. (2019) Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. 2019.  Finding generalizable evidence by learning to convince q&a models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2402–2411, Hong Kong, China. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392. Association for Computational Linguistics.
  * Shao et al. (2020) Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020.  Is graph structure necessary for multi-hop reasoning?  _arXiv preprint arXiv:2004.03116_.
  * Sun et al. (2019) Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019.  DREAM: A challenge data set and models for dialogue-based reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 7:217–231.
  * Sun et al. (2020) Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020.  Investigating prior knowledge for challenging chinese machine reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 8:141–155.
  * Tishby et al. (2000) Naftali Tishby, Fernando C Pereira, and William Bialek. 2000.  The information bottleneck method.  _arXiv preprint physics/0004057_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2019) Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019.  Evidence sentence extraction for machine reading comprehension.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 696–707. Association for Computational Linguistics.
  * Xu et al. (2020) Liang Xu, Xuanwei Zhang, Lu Li, Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, Yiming Cui, Cong Yu, Qianqian Dong, Yin Tian, Dian Yu, Bo Shi, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, and Zhenzhong Lan. 2020.  CLUE: A Chinese Language Understanding Evaluation Benchmark.  _arXiv e-prints_ , page arXiv:2004.05986.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yang et al. (2020) Ziqing Yang, Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, and Guoping Hu. 2020.  TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics.
  * Zhu et al. (2020) Pengfei Zhu, Hai Zhao, and Xiaoguang Li. 2020.  Dual Multi-head Co-attention for Multi-choice Reading Comprehension.  _arXiv e-prints_ , page arXiv:2001.09415.


11institutetext: Indian Institute of Science Education and Research Kolkata,
India
11email: {as18ms065, prm18ms118<EMAIL_ADDRESS>
†These authors contributed equally to this work.
∗Corresponding author

# Exploring the Nexus Between Retrievability and
Query Generation Strategies111Accepted at ECIR 2024.

Aman Sinha† Priyanshu Raj Mall† Dwaipayan Roy∗

###### Abstract

Quantifying bias in retrieval functions through document retrievability scores
is vital for assessing recall-oriented retrieval systems. However, many
studies investigating retrieval model bias lack validation of their query
generation methods as accurate representations of retrievability for real
users and their queries. This limitation results from the absence of
established criteria for query generation in retrievability assessments.
Typically, researchers resort to using frequent collocations from document
corpora when no query log is available. In this study, we address the issue of
reproducibility and seek to validate query generation methods by comparing
retrievability scores generated from artificially generated queries to those
derived from query logs. Our findings demonstrate a minimal or negligible
correlation between retrievability scores from artificial queries and those
from query logs. This suggests that artificially generated queries may not
accurately reflect retrievability scores as derived from query logs. We
further explore alternative query generation techniques, uncovering a
variation that exhibits the highest correlation. This alternative approach
holds promise for improving reproducibility when query logs are unavailable.

###### Keywords:

Reproducibility Retrievability Empirical Study.

##  1 Introduction

In the field of Information Retrieval (IR), the study of biases and fairness
of retrieval models together with their effectiveness has been an active area
of research [23, 26]. Among the various metrics and methodologies developed,
_retrievability_ has emerged as a significant way of measuring how accessible
a document is in a collection, given a retrieval system and a set of queries
[8]. It reflects the degree to which the system favors certain documents while
retrieval over others, regardless of their relevance. Retrievability can be
used to evaluate the fairness, diversity, and coverage of retrieval systems,
as well as to identify and analyze the factors that affect the accessibility
of documents [7]. However, calculating the retrievability of a collection of
documents is a complex task comprising several interconnected steps.
Initially, a set of queries is selected based on which the retrievals are
performed. Researchers have used query logs as well as artificially generated
queries in this step. Subsequently, a retrieval model is configured with
optimal parameters and the actual retrieval process follows, producing ranked
document lists. Retrievability scores are then computed based on the rank of
the documents in the retrieved lists. Finally, the computed scores are used to
estimate “retrievability bias” by metrics like the Gini coefficient[27],
Atkinson index [4], Theli index [30] or Palma index [39] borrowed from
economics, to quantify bias and inequality (mainly in wealth) within a
population.

Various query generation techniques employed at the initial stage can yield
distinct query sets, potentially resulting in diverse retrievability scores
for the same collection and retrieval system. Previous research has indicated
that retrievability scores calculated using simulated queries differ from
those obtained using actual queries extracted from query logs [45]. However, a
comprehensive and systematic comparison of retrievability scores computed
using various query simulation methods has yet to be undertaken. The
discrepancy, if any, poses a challenge to the reproducibility of
retrievability experiments as different researchers may employ different query
generation methods, leading to diverse conclusions on the same collection. To
investigate the potential impact on result reproducibility and experimental
validation, this paper conducts a comparative analysis of computed
retrievability scores using the most commonly adopted query generation
techniques in the research community. The main contributions of this paper
include:

  * •

Providing an overview of query generation techniques commonly used in
retrievability experiments documented in the literature.

  * •

Conducting a comprehensive empirical assessment of the reproducibility of
retrievability experiments employing different query generation techniques and
test collections.

Additionally, we address the following research questions in this paper:

  * RQ1 -

While keeping the other parameters constant, does varying the query simulation
techniques lead to differences in retrievability scores, thus impacting the
reproducibility of results?

  * RQ2 -

Do these differences in retrievability scores exhibit consistency across
various collections, thereby influencing the reproducibility of results?

The rest of the paper is organized as follows. In the next section, we provide
essential background information and review previous research related to
retrievability, with a particular emphasis on the prevalent query simulation
techniques in this field. Section 3 delves into the specifics of our
experimental setups before presenting the empirical result on benchmark test
collections in Section 4. We conclude the paper in Section 5 highlighting the
findings considering the issues with reproducing results, and suggesting some
directions to pursue further study. The codes and the simulated queries are
available from the following repository:
https://mallpriyanshu.github.io/retrievability-survey/ecir2024.

##  2 Background and Motivation

###  2.1 Retrievability: A Measure of Accessibility

Retrievability is a measure that evaluates the ease with which a document can
be retrieved in a specific configuration of an IR system. Azzopardi and Vinay
[8] introduced the concept and formally defined the retrievability score for a
document $d$ in a collection $D$ as $r(d)$ with respect to a given IR system.
Typically, the calculation of retrievability involves the execution of the
following sequence of steps [36]: $i)$ query set generation, $ii)$ system
configuration and retrieval, $iii)$ computation of retrievability scores, and
$iv)$ summarising the scores.

For an ideal calculation of retrievability scores, a comprehensive universe of
all possible queries is needed, although obtaining this is practically
unattainable. In practice, researchers commonly select from two alternatives:
either using a sufficiently large query log containing diverse user queries
[42, 45] or simulating a set of queries from the collection that adequately
represents its overall topic distribution [8, 9, 14, 16]. Once an appropriate
set of queries has been chosen, the second step is selecting a retrieval model
with an optimal parameter setting. Prior research has extensively examined the
impact of altering system configurations [8, 15, 50] where the authors
conclude that changing the retrieval model can significantly impact the
retrievability scores of a document collection. The retrievals are then
performed with the set of queries, and the retrievability of a document $d$ is
computed in the third step as _the summation of the number of times $d$ is
retrieved within the top $c$ position_. Mathematically, the retrievability
$r(d)$ of a document $d$ ($d\in D)$ with respect to an IR system is computed
using the formula in Equation 1.

$r(d)=\sum_{q\in\mathsf{Q}}o_{q}\cdot f(k_{dq},c)$ (1)

Here, $o_{q}$ indicates the likelihood of the query (usually considered
uniform across query set) and $k_{dq}$ is the rank of $d$ for query $q$.
Calculating $r(d)$ relies on the choice of a utility function $f(.)$, either
cumulative or gravity-based. Previous studies have shown a high correlation
between these two measures [47, 48], and consequent research predominantly
reports retrievability analysis using the cumulative-based function [13, 21,
45, 49]. Defined as a binary function, the cumulative utility function
$f(k_{dq},c)$ is equal to $1$ if document $d$ is retrieved within the top $c$
documents for query $q$; otherwise, it is considered $0$. This utility
function provides a straightforward interpretation of the retrievability score
for each document. It simply corresponds to the number of times the document
is retrieved within the top $c$ ranks. Documents beyond the top $c$ positions
are disregarded, simulating a user who only examines the first $c$ results.
Therefore, the more queries that retrieve a document within rank $c$, the
higher its retrievability score.

The final step involves summarizing the retrievability scores across the
collection to assess existing bias. This quantification is commonly achieved
using the Gini coefficient, a measure frequently employed in economics and
social sciences to assess inequality within a population [27]. Calculating the
retrievability bias within a document collection, the Gini coefficient is
computed as follows:

$G=\frac{\sum_{i=1}^{N}(2i-N-1)\cdot r(d_{i})}{N\sum_{j=1}^{N}r(d_{j})}$ (2)

where $N$ represents total number of documents in the collection ($|D|$). It
provides a decimal value in the range $[0,1]$. A Gini coefficient of zero
denotes perfect equality, indicating that all documents in the collection have
an equal retrievability score according to $r(d)$. Conversely, a Gini
coefficient of one indicates total inequality, with only one document having
$r(d)=|Q|$ while all other documents have $r(d)=0$. Other inequality metrics,
such as Atkinson index [4], Theli index [30], Palma index [39] etc., are also
explored in [53] where the authors argue that various inequality metrics
generally agree on which system and settings minimize the retrievability bias
and the least inequality indicated by Gini coefficient is consistent with the
other metrics discussed. Hence, the Gini coefficient $G$ is commonly employed
in subsequent works [8, 16, 18] to quantify the amount of bias in the
distribution of the retrievability scores in a collection of documents.

To visually assess the retrievability distribution in a document collection,
one can calculate retrievability scores for each document using Equation 1 and
plot the sorted values to create a _Lorenz curve_. This curve represents the
cumulative score distribution of documents sorted by retrievability scores in
ascending order. If the retrievability scores are evenly distributed, the
Lorenz Curve will be linear; a skewed curve in contrast indicates a greater
level of inequality or bias. The Lorenz curve is a concept commonly used in
economics to depict wealth distribution, where a straight line signifies equal
wealth distribution across a population, while a skewed curve represents
inequality, as shown in Figure 1.

Figure 1: A sample Lorenz Curve. The values belonging to the distribution in
which the disparity is to be observed are arranged in a sorted manner along
the X-axis.

Retrievability, and the underlying theory of retrievability, have found
applications in various domains. For instance, it has been used in the
development of inverted indexes to enhance the efficiency and performance of
retrieval systems by capitalizing on terms that contribute to a document’s
retrievability [41]. Additionally, retrievability has been leveraged to
investigate bias in search engines and retrieval systems on the web [6] and
within patent collections [18], leading to improvements in system efficiency
during pruning processes [54].

The initial and most critical step while calculating the retrievability of a
collection of documents involves query generation, followed by retrieval using
the generated queries, and finally, the computation of retrievability scores
based on the appearance of documents within a rank cutoff. Prior research has
extensively examined the impact of altering system configurations [8, 9, 15,
45, 50], modifying the nature of the function [48], and varying score
summarization techniques [53]. However, there is limited empirical evidence
concerning the influence of modifying query sets on the resultant
retrievability scores. Filling the void, in this paper, we study the effect of
various query-generation techniques on the retrievability scores of documents
in a collection. We discuss some of the well-known techniques popular among
researchers for query generation in the next section.

###  2.2 Query Generation

The initial step in a retrievability computation typically involves obtaining
a set of all possible queries addressable by the collection ($Q$ in Equation
1). As per Equation 1, the most ideal calculation of $r(d)$ would require an
extensive set $Q$. In general, a query log would be the preferred option for
$Q$; however, obtaining such a log can be difficult and often not feasible.
Hence, researchers have inclined towards artificial simulation of the set $Q$
from the collection. Following the inception of the concept of retrievability
in [8], a popular choice among researchers in simulating $Q$ based on term
statistics in the collection itself [14, 15, 16, 17, 18, 19, 21, 38]. The
approach by Azzopardi and Vinay [8] in generating the query set was influenced
by Callan and Connel [24]. The authors generated a set of queries, which
included single-term queries, formed by selecting each term in the vocabulary
that appeared at least 5 times, and bi-term queries, formed by selecting each
bigram in the collection that appeared at least 20 times. The list of bigrams
was truncated at 20 million, and each query was issued to the system to
estimate its retrievability. Azzopardi and Bache later followed a similar
methodology to generate query sets for the AP and WSJ collections by ranking
the most frequent 100,000 bigrams in each collection [5]. The same approach
was also adapted in [9] where the authors applied retrievability to quantify
the accessibility of patent documents. Although the queries used in patent
search are often longer than two words, the authors choose two-word queries
because this means there will be “just one Boolean operator and it thus
affords a comparison between the use of AND and OR within the various IR
models”. In the other studies on retrievability, the prevalent choice among
researchers has been the adoption of a similar approach for query generation
[47, 48, 49, 50, 51, 52, 53].

In their work on patent retrieval [14], the authors created queries using
terms from the _claim section_ of the patent documents. Single-term queries
are formed from each document with a term frequency lower bound of 2. To make
longer queries, two and three-term queries are generated by the combination of
the frequent terms in that document. In their subsequent works [15, 16], the
authors employed a controlled query generation technique proposed in [31] for
patent retrieval. Subsequent works from the same research group introduced a
different approach that involves generating query sets for each document in
the corpus. They select terms that appear more than once within the document,
resulting in four query subsets: single terms, combinations of two terms,
combinations of three terms, and combinations of four terms [17, 18, 19]. Some
further truncation of query set was adapted in [11, 12, 13, 20, 22] where
queries were generated with combinations of terms that appear more than once
in the document. Stopwords and terms having document frequency $>$ 25% of
collection size were removed.

Assessing document retrievability in a collection necessitates a comprehensive
set of queries that cover all aspects of the collection. Ideally, this would
involve an extensive query log spanning a significant time period. However,
acquiring such a log is often costly and unavailable, prompting researchers to
resort to simulated queries. Despite the challenges, some research has been
conducted using real user query logs acquired from certain closed domains. The
authors in [44, 45] use a set of real queries submitted to the National
Library of the Netherlands, between March 2015 to July 2015. Additionally, to
make the results comparable to [8], the authors create a set of simulated
queries: they select the top 2 million unigrams and bigrams respectively as
single term and two term queries. All the queries were finalized after
stemming and removal of stopwords and operators. On comparing the
characteristics of simulated queries to those formed by real user queries, the
authors report significant differences regarding the composition, the number
of unique terms used, and the presence of named entities: real user queries
exhibited a considerably higher fraction of named entities compared to the
simulated query set. Additionally, due to the considerably low number of real
queries in contrast to the vast size of the collection (exceeding 102
million), the authors acknowledged the presence of a varying document-query
ratio compared to those reported in [8]. Authors in [2] employ a transformer-
based generative model docT5query [37] for generating queries and conclude
that BERT-based rankers demonstrate less retrievability bias than user-given
queries. Recently, Roy et al. [42] studied the retrievability distribution in
an integrated retrieval system containing documents in multiple categories
[42, 43]. However, this work does not conduct a direct comparison of the
retrievability scores (and their distributions) when computed using real and
simulated query sets.

The contrasting nature of simulated queries, when compared to real queries, is
briefly reported in [45]. The various techniques used for generating simulated
queries illustrate the diversity in approaches taken at the initial stage of a
retrievability analysis by different groups of researchers. This also
highlights the absence of a well-defined structure for such an analysis paving
a potential issue with reproducing results. It prompts the question: can we
anticipate a similar distribution of retrievability scores when the query set
is altered? Answering that question, in this paper, we present a systematic
comparison of the retrievability scores computed based on several of these
techniques together with real-user query logs. According to the observation in
[45], the real queries contain a significantly higher fraction of named
entities compared to the simulated query set. Inspired by this observation, we
additionally introduce a rule-based query simulation technique for
retrievability assessment, further contributing to the examination of this
diversity in approaches. To maintain the primary narrative of the paper, we
have detailed the method in the Appendix 0.A.

##  3 Experimental Setup

As presented in Section 2, the set of queries employed in computing the
retrievability plays an important role in the computation and our focus of
this paper is to foresee whether alteration in this set can affect the final
score, potentially leading to challenges in reproducing results for different
collections and retrieval models. In this section, we present the datasets and
the query sets that are used in this paper to verify this empirically with
real data.

###  3.1 Datasets

We perform the experiments of the retrievability score distributions on two
benchmark retrieval datasets as well as on a diverse encyclopedia of
documents. Particularly, the TREC ad-hoc Robust collection [46] containing
news articles, WT10g collection [29] with web pages, together with the English
Wikipedia article dump of February, 2023 are employed in this
study222https://dumps.wikimedia.org/enwiki/. All the datasets are preprocessed
with stopword removal following SMART stopword list
333https://tinyurl.com/smart-stopword and stemmed using the Porter Stemmer
before indexing them using Apache Lucene. Overall statistics of the datasets
are presented in Table 1.

Table 1: Basic statistics of datasets used for empirical verification of the variation in the computed retrievability scores when different query sets are employed. Dataset |  # documents |  Collection Type |  # terms
---|---|---|---
TREC Robust | 528,155 |  News articles | 1,502,031
WT10g | 1,692,096 |  Web pages | 9,674,707
Wikipedia | 6,584,626 | Encyclopedia | 18,797,260

###  3.2 Simulated Queries

In order to foresee the variation, if any, in the retrievability distribution
of documents in a collection, we use the following artificial query generation
techniques:

  1. 1.

Simulated Query set 1 ($\mathsf{SQ_{1}}$): Following the suggestion in [8],
the first set of the queries with which the retrievals are performed is formed
by a sampling method proposed in [24]. In this process, the terms undergo
analysis and filtering, including stemming using the Porter stemmer and
removing stopwords. Terms occurring more than 5 times in the collection are
treated as single-term queries. Additionally, two-term queries are formed by
pairing consecutively occurring terms that have a collection frequency of at
least 20 times. These resulting bigrams are then ranked by occurrence, and the
top two million are selected as the final set of two-term queries.

  2. 2.

Simulated Query set 2 ($\mathsf{SQ_{2}}$): The second technique is based on
[13]. Specifically, queries are generated with the combinations of terms that
appear more than once in the document. Stopwords and other terms with document
frequency $>$ 25% of collection size are removed. Queries up to 4 terms are
formed using the boolean AND operator, with duplicate queries being removed.

  3. 3.

Simulated Query set 3 ($\mathsf{SQ_{3}}$): In this approach (based on [31]), a
set of related documents is chosen as the source document set with clustering.
Language models are defined for both the source documents and the corpus, with
relative entropy used for comparison. Terms in the vocabulary are sorted based
on their contribution to relative entropy, serving as a term discrimination
scoring function. The most discriminating term initiates the query
construction process, forming single-term and two-term queries for different
query sets. The process continues by identifying next most informative term
and generating subsequent queries until all terms have been used.

  4. 4.

Rule-based Simulated Query set ($\mathsf{RSQ}$): Building upon the insight in
[45], that real queries contain a notably greater proportion of named entities
compared to simulated queries, we introduce a rule-based query simulation
technique for retrievability assessment. The approach is presented in Appendix
0.A.

Table 2: Number of queries generated for each dataset using the individual query generation techniques. Note that, the simulated queries are generated from each dataset and applied in that particular dataset for the retrievability computation. AOL query set is filtered separately for each dataset as discussed in Section 3.3.  | Query Type | Number of queries
---|---|---
|  Robust04 |  WT10g |  Wikipedia
$\mathsf{SQ_{1}}$ [8] | Simulated | 1061184 | 3902575 | 5320001
$\mathsf{SQ_{2}}$ [13] | Simulated | 6903196 | 18724509 | 84920098
$\mathsf{SQ_{3}}$ [31] | Simulated | 1644533 | 7346091 | 26272419
$\mathsf{RSQ}$ | Simulated | 3900000 | 4500000 | 4550000
$\mathsf{AOL}$ [40] | Real Query | 3885206 | 4302650 | 4550977

###  3.3 Real Queries

A majority of the works done in the domain of retrievability employ queries
generated using simulation techniques. In this work, we use a general web
search query log to make our study conclusive. Specifically, we utilize the
AOL query log [40] that contains actual user query click data spanning from
March 1st, 2006, to May 31st, 2006. It sparked significant controversy within
the media, primarily due to privacy-related concerns [10, 28, 3]. Although the
query log is more than a decade old, it is still widely used in recent
literature [25, 1, 34, 33, 55, 35].

To prepare the real query set for our experiments, the initial step involves
extracting all unique queries from the AOL query log. Queries containing
periods are eliminated to filter out website links. Given the distinct
temporal scopes of our datasets and query sets, a conspicuous challenge arises
in the form of topic and vocabulary mismatch. To mitigate this challenge as
much as possible, we adopt an exclusionary approach: a query is selected if
all of its terms are present in the corpus vocabulary. This strategic
refinement serves to align the queries closely with the content of the target
collection. Note that, the AOL queries are refined for each collection
separately following the above exclusionary approach. Table 2 contains the
individual count of queries finalized for each dataset.

###  3.4 Retrieval Model

In accordance with the insights and recommendations put forth in [8], we only
use BM25 as the model for performing the retrieval. Specifically, we use the
implementation available in Apache Lucene. To fine-tune the BM25 parameters,
we utilize the TREC ad-hoc retrieval topic set [46], which is then applied on
the Robust dataset for the retrievability study. Similarly, the TREC web topic
sets [29] are employed for parameter tuning, and these adjusted parameters are
subsequently applied to the WT10g and Wikipedia datasets.

Table 3: Gini coefficient of Retrievability score distribution when computed using different query generation techniques. The highest and the lowest Gini values for each dataset are marked respectively with a superscript † and ∗.  Dataset | $\mathsf{SQ_{1}}$ | $\mathsf{SQ_{2}}$ | $\mathsf{SQ_{3}}$ | $\mathsf{RSQ}$ | $\mathsf{AOL}$
---|---|---|---|---|---
Robust | $~{}~{}0.3307~{}~{}$ | $~{}~{}0.4556~{}~{}$ | $~{}~{}0.4300~{}~{}$ | $~{}~{}0.3052^{*}~{}~{}$ | $~{}~{}{0.6032}^{\dagger}~{}~{}$
WT10g | 0.5371 | 0.6391 | 0.6359 | $0.5009^{*}$ | $0.6541^{\dagger}$
Wikipedia | 0.5380 | 0.6210 | 0.6290 | $0.4820^{*}$ | $0.6798^{\dagger}$
Figure 2: Gini coefficient for each dataset when the retrievability scores are
calculated using different query sets as discussed in Section 3.

##  4 Comparing Retrievability

###  4.1 Gini Coefficient and Lorenz Curve

We present the Gini coefficient for each dataset utilizing all five query sets
in Table 3. The variation in Gini values is graphically presented in Figure 2.
In the table, the least disparity is observed across all the datasets when
utilizing the rule-based simulation for query generation (denoted as
$\mathsf{RSQ}$ in Table 3). The distributions of retrievability values are
presented using Lorenz curve in Figure 3. Furthermore, the original query
simulation technique proposed by [8] (indicated by $\mathsf{SQ_{1}}$ in Table
3) follows a similar trend. The other two simulation techniques
($\mathsf{SQ_{2}}$ and $\mathsf{SQ_{3}}$) exhibit higher disparity. In
contrast, a significant disparity in retrievability scores across all the
datasets is observed when the real-query log ($\mathsf{AOL}$) is used. These
results lead to the following conclusions:

  * •

The web collection WT10g and the encyclopedia Wikipedia exhibit much more
diverse retrievability scores as compared to the relatively smaller news
collection TREC Robust across all the query sets.

  * •

The approach in [8] selects terms from the collection based on their
frequency, including non-informative queries ($\mathsf{SQ_{1}}$). This method
results in a more evenly distributed query generation covering a wide range of
documents in the collection. As a result, it tends to produce lower Gini
values compared to other query sets.

  * •

Realistic queries generated using the rule-based approach ($\mathsf{RSQ}$)
result in even lower Gini coefficients across all datasets.

  * •

AOL query logs mostly consist of known item searches rather than exploratory
ones leading to a significant disparity in computed retrievability scores,
with Gini coefficients consistently exceeding $0.6$ across all datasets.

Figure 3: Lorenz curve with retrievability computed with various query sets on
different collections. Table 4: Correlation between retrievability scores when
computed using the real query set (AOL) and the four simulated query sets
(discussed in Section 3.2). Upper triangle contains Pearson’s $r$ while lower
triangle reports Kendall’s rank correlation $\tau$. All differences
statistically significant with $p<0.05$. Highest (in bold) and lowest
(superscripts ∗) correlations for each dataset are highlighted.

| Robust04
---|---
|  SQ${{}_{1}~{}~{}~{}~{}}$ |  SQ${{}_{2}~{}~{}~{}~{}}$ |  SQ${{}_{3}~{}~{}~{}~{}}$ |  RSQ |  AOL
SQ1 |  | 0.0923 | -0.0025 |  0.7549 | 0.1250
SQ2 | 0.1758 |  | -0.0004∗ | 0.1461 | 0.0033
SQ3 | 0.0108 | -0.0040∗ |  | 0.0030 | 0.0165
RSQ |  0.5617 | 0.3023 | 0.0233 |  | 0.2076
AOL | 0.1413 | -0.0174 | 0.0321 | 0.1811 |
| WT10g
SQ1 |  | 0.1206 | 0.2094 | 0.3103 | 0.1100
SQ2 | 0.3575 |  |  0.5503 | 0.1931 |  0.0900∗
SQ3 | 0.3223 | 0.3047 |  | 0.2535 | 0.1795
RSQ |  0.5672 | 0.5312 | 0.3189 |  | 0.3135
AOL | 0.3731 |  0.2977∗ | 0.3830 | 0.4126 |
| Wikipedia
SQ1 |  | 0.1586 | 0.2613 | 0.4768 | 0.2379
SQ2 | 0.3508 |  |  0.7019 | 0.1599 | 0.0598
SQ3 | 0.3897 | 0.5116 |  | 0.1048 |  0.0308∗
RSQ |  0.5350 | 0.4867 | 0.3739 |  | 0.3065
AOL | 0.3603 | 0.3074 |  0.2338∗ | 0.4194 |

###  4.2 Comparing Correlation

Table 4 reports the degree of variation in retrievability scores when
comparing the simulation-based query generation techniques ($\mathsf{SQ_{1}}$,
$\mathsf{SQ_{2}}$, $\mathsf{SQ_{3}}$, and $\mathsf{RSQ}$) along with the AOL
query log. We compute the Pearson’s $r$ between the distribution of each pair
of retrievability scores computed using the different query sets and report in
the upper half of Table 4. Kendall’s $\tau$ is presented in the lower half of
the table which is computed using the document rank arranged following their
retrievability scores. The table shows negligible correlations between scores
computed using almost any query set pairs. Interestingly, the highest
correlations, in terms of both Pearson’s $r$ and Kendall’s $\tau$ occur when
using the rule-based technique $\mathsf{RSQ}$ (Appendix 0.A). Further, the
massive difference in the highest and lowest correlations validates that
distinct query sets lead to divergent retrievability scores, affirming both
RQ1 and RQ2.

##  5 Conclusion

The examination of retrievability has predominantly relied on the use of
simulated queries since its inception with few works using real query logs.
However, a comprehensive comparative analysis of the computed values has been
absent in the existing literature. Bridging this gap, in this paper, we
examine the impact of five distinct and diverse query sets, ranging from
simulated queries to actual user query logs, on the computed retrievability
scores when the retrieval is performed using BM25.

Our findings on diverse datasets, including news, web, and Wikipedia
collections expose substantial variations in computed retrievability scores
across query sets, providing an affirmative response to RQ1 and underscoring
the reproducibility challenges associated with retrievability scores. Among
the popular query simulation techniques for retrievability studies, the query
set generated following the method in [8] ($\mathsf{SQ_{1}}$) exhibits the
least disparity in computed retrievability scores. In contrast, the AOL query
log generates the highest disparity. Furthermore, the utilization of a rule-
based simulation technique ($\mathsf{RSQ}$) results in the least inequality in
the retrievability scores among all. These observations reveal a high
sensitivity of these scores with respect to the query generation methods
employed. Our research demonstrates that the selection of a query set directly
impacts computed retrievability scores, regardless of the nature of the
collection. This provides a conclusive response to RQ2. These findings
emphasize the need for standardization of query set construction in
retrievability studies and contribute to a deeper understanding of the nuances
of accessibility within the field of information retrieval. In previous
research on retrievability, various studies utilized multiple retrieval models
for a single query set per collection. As part of our future work, we will
explore the impact of diverse retrieval models on the retrievability scores
with different query sets.

##  Appendix 0.A Appendix - A POS-based Query Generation Technique

Identifying collocations in a text corpus typically involves counting co-
occurring word pairs, revealing words that go beyond their individual
meanings. Relying solely on the most frequent bigrams often yields
uninteresting results, as many of them consist of function words, offering
limited insights. To improve collocation quality, Justeson and Katz [32]
introduced a simple yet effective heuristic. They apply a part-of-speech
filter to candidate phrases, preserving patterns likely to represent genuine
‘phrases’ rather than random word combinations. This approach enhances the
meaningfulness of the collocation identification process.

We apply this approach for query generation improving the relevance and
effectiveness of the generated queries.

  1. 1.

Perform Part-of-Speech (POS) tagging: Initially, we employ POS tagging on all
the documents within the collection. This step assigns appropriate grammatical
tags to each word, facilitating the subsequent identification of N-grams.

  2. 2.

Extract N-grams: N-grams, where N represents the desired length of the word
sequences, are extracted from the POS-tagged documents. In our case, we
consider N to range from 1 to 4, enabling the identification of unigrams,
bigrams, trigrams, and quadgrams.

  3. 3.

Select N-grams with ‘query-like’ POS tag patterns: From the pool of extracted
N-grams, we apply Justeson and Katz’s [32] recommended POS tag patterns to
filter and retain N-grams that exhibit patterns resembling queries. The
specific POS tag patterns for each N-gram type are provided in Table 5. Tag
patterns for Quadgrams are proposed by us heuristically from our observations.

Table 5: POS Tag rules for N-gram query generation

unigram | bigram | trigram | quadgram
---|---|---|---
noun | adj noun | adj adj noun | noun verb adp noun
| noun noun | adj noun noun | adj noun adj noun
|  | noun adj noun | noun adp adj noun
|  | noun noun noun | noun noun adp noun
|  | noun adp noun | noun verb noun noun
|  |  | adv adj noun noun
|  |  | adj noun verb noun
|  |  | noun adj noun noun

Subsequently, the resulting list of N-grams is sorted in descending order
based on their occurrence frequencies. To ensure a manageable and relevant set
of queries, we truncate the list at specific thresholds. These thresholds are
determined by drawing inspiration from the frequency distribution of queries
found in the AOL query set [40]. We aim to maintain a proportional ratio
between the selected N-grams and the query frequencies observed in the AOL
real query set, preserving a close alignment with real-world query usage
patterns.

## References

  * [1] Ahmad, W.U., Chang, K.W., Wang, H.: Context attentive document ranking and query suggestion. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 385–394. SIGIR’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3331184.3331246, https://doi.org/10.1145/3331184.3331246
  * [2] Amin Abolghasemi, Suzan Verberne, A.A., Azzopardi, L.: Retrievability bias estimation using synthetically generated queries. In: Proceedings of the First Workshop on Generative Information Retrieval - GenIR@SIGIR2023 held in conjunction with SIGIR 2023. GenIR@SIGIR2023 (2023), https://coda.io/@sigir/gen-ir/accepted-papers-17
  * [3] Anderson, N.: The ethics of using aol search data https://arstechnica.com/uncategorized/2006/08/7578/
  * [4] Atkinson, A.B.: On the measurement of inequality. Journal of Economic Theory 2(3), 244–263 (1970). https://doi.org/https://doi.org/10.1016/0022-0531(70)90039-6, https://www.sciencedirect.com/science/article/pii/0022053170900396
  * [5] Azzopardi, L., Bache, R.: On the relationship between effectiveness and accessibility. In: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. pp. 889–890 (2010)
  * [6] Azzopardi, L., Owens, C.: Search engine predilection towards news media providers. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. pp. 774–775 (2009)
  * [7] Azzopardi, L., Vinay, V.: Accessibility in information retrieval. In: Advances in Information Retrieval: 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-April 3, 2008. Proceedings 30. pp. 482–489. Springer (2008)
  * [8] Azzopardi, L., Vinay, V.: Retrievability: An evaluation measure for higher order information access tasks. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management. p. 561–570. CIKM ’08, Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1458082.1458157, https://doi.org/10.1145/1458082.1458157
  * [9] Bache, R., Azzopardi, L.: Improving access to large patent corpora. Trans. Large Scale Data Knowl. Centered Syst. 2, 103–121 (2010). https://doi.org/10.1007/978-3-642-16175-9_4, https://doi.org/10.1007/978-3-642-16175-9_4
  * [10] Barbaro, Michael; Zeller Jr, T.: A face is exposed for aol searcher no. 4417749 https://www.nytimes.com/2006/08/09/technology/09aol.html
  * [11] Bashir, S.: Improving retrievability with improved cluster-based pseudo-relevance feedback selection. Expert Syst. Appl. 39(8), 7495–7502 (2012). https://doi.org/10.1016/j.eswa.2012.01.041, https://doi.org/10.1016/j.eswa.2012.01.041
  * [12] Bashir, S.: Estimating retrievability ranks of documents using document features. Neurocomputing 123, 216–232 (2014)
  * [13] Bashir, S., Khattak, A.S.: Producing efficient retrievability ranks of documents using normalized retrievability scoring function. Journal of Intelligent Information Systems 42, 457–484 (2014). https://doi.org/10.1007/s10844-013-0274-3, https://doi.org/10.1007/s10844-013-0274-3
  * [14] Bashir, S., Rauber, A.: Analyzing document retrievability in patent retrieval settings. In: Database and Expert Systems Applications: 20th International Conference, DEXA 2009, Linz, Austria, August 31–September 4, 2009. Proceedings 20. pp. 753–760. Springer (2009)
  * [15] Bashir, S., Rauber, A.: Identification of low/high retrievable patents using content-based features. In: Proceedings of the 2nd international workshop on Patent information retrieval. pp. 9–16 (2009)
  * [16] Bashir, S., Rauber, A.: Improving retrievability of patents with cluster-based pseudo-relevance feedback documents selection. In: Proceedings of the 18th ACM conference on Information and knowledge management. pp. 1863–1866 (2009)
  * [17] Bashir, S., Rauber, A.: Improving retrievability and recall by automatic corpus partitioning. Transactions on large-scale data-and knowledge-centered systems II pp. 122–140 (2010)
  * [18] Bashir, S., Rauber, A.: Improving retrievability of patents in prior-art search. In: Advances in Information Retrieval: 32nd European Conference on IR Research, ECIR 2010, Milton Keynes, UK, March 28-31, 2010. Proceedings 32. pp. 457–470. Springer (2010)
  * [19] Bashir, S., Rauber, A.: On the relationship between query characteristics and ir functions retrieval bias. Journal of the American Society for Information Science and Technology 62(8), 1515–1532 (2011)
  * [20] Bashir, S., Rauber, A.: Automatic ranking of retrieval models using retrievability measure. Knowledge and information systems 41, 189–221 (2014)
  * [21] Bashir, S., Rauber, A.: Retrieval models versus retrievability. Current Challenges in Patent Information Retrieval pp. 185–212 (2017)
  * [22] Bashir, S., Rauber, A.: Retrieval Models Versus Retrievability, pp. 185–212. Springer Berlin Heidelberg, Berlin, Heidelberg (2017). https://doi.org/10.1007/978-3-662-53817-3_7, https://doi.org/10.1007/978-3-662-53817-3_7
  * [23] Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds.): Advances in Bias and Fairness in Information Retrieval. Springer Nature Switzerland (2023). https://doi.org/10.1007/978-3-031-37249-0, https://doi.org/10.1007/978-3-031-37249-0
  * [24] Callan, J., Connell, M.: Query-based sampling of text databases. ACM Transactions on Information Systems (TOIS) 19(2), 97–130 (2001)
  * [25] Dehghani, M., Zamani, H., Severyn, A., Kamps, J., Croft, W.B.: Neural ranking models with weak supervision. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 65–74. SIGIR ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3077136.3080832, https://doi.org/10.1145/3077136.3080832
  * [26] Ekstrand, M.D., Das, A., Burke, R., Diaz, F.: Fairness in information access systems. Foundations and Trends® in Information Retrieval 16(1-2), 1–177 (2022). https://doi.org/10.1561/1500000079, http://dx.doi.org/10.1561/1500000079
  * [27] Gini, C.: On the measure of concentration with special reference to income and statistics. Colorado College Publication, General Series 208(1), 73–79 (1936)
  * [28] Hafner, K.: Tempting data, privacy concerns; researchers yearn to use aol logs, but they hesitate https://www.nytimes.com/2006/08/23/technology/23search.html
  * [29] Hawking, D.: Overview of the TREC-9 web track. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of The Ninth Text REtrieval Conference, TREC 2000, Gaithersburg, Maryland, USA, November 13-16, 2000. NIST Special Publication, vol. 500-249. National Institute of Standards and Technology (NIST) (2000), http://trec.nist.gov/pubs/trec9/papers/web9.pdf
  * [30] Johnston, J.: H. Theil. Economics and Information Theory. The Economic Journal 79(315), 601–602 (09 1969). https://doi.org/10.2307/2230396, https://doi.org/10.2307/2230396
  * [31] Jordan, C., Watters, C., Gao, Q.: Using controlled query generation to evaluate blind relevance feedback algorithms. In: Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries. pp. 286–295 (2006)
  * [32] Justeson, J.S., Katz, S.M.: Co-occurrences of antonymous adjectives and their contexts. Computational Linguistics 17(1), 1–20 (1991), https://aclanthology.org/J91-1001
  * [33] Kang, Y.M., Liu, W., Zhou, Y.: Queryblazer: Efficient query autocompletion framework. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. p. 1020–1028. WSDM ’21, Association for Computing Machinery (2021). https://doi.org/10.1145/3437963.3441725
  * [34] Ma, Z., Dou, Z., Bian, G., Wen, J.R.: Pstie: Time information enhanced personalized search. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. p. 1075–1084. CIKM ’20, Association for Computing Machinery (2020). https://doi.org/10.1145/3340531.3411877
  * [35] MacAvaney, S., Macdonald, C., Ounis, I.: Reproducing personalised session search over the aol query log. In: Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part I. p. 627–640. Springer-Verlag, Berlin, Heidelberg (2022). https://doi.org/10.1007/978-3-030-99736-6_42
  * [36] McLellan, C.: The relationship between retrievability bias and retrieval performance. Ph.D. thesis, University of Glasgow, UK (2019), https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.775857
  * [37] Nogueira, R., Lin, J.: From doc2query to doctttttquery. In: Online preprint 6 (2019), https://github.com/castorini/docTTTTTquery
  * [38] Noor, S., Bashir, S.: Evaluating bias in retrieval systems for recall oriented documents retrieval. International Arab Journal of Information Technology (IAJIT) 12(1) (2015)
  * [39] Palma, J.G.: Homogeneous middles vs. heterogeneous tails, and the end of the ‘inverted-u’: the share of the rich is what it’s all about. Cambridge working papers in economics, Faculty of Economics, University of Cambridge (2011), https://EconPapers.repec.org/RePEc:cam:camdae:1111
  * [40] Pass, G., Chowdhury, A., Torgeson, C.: A picture of search. In: Proceedings of the 1st International Conference on Scalable Information Systems. p. 1–es. InfoScale ’06, Association for Computing Machinery (2006). https://doi.org/10.1145/1146847.1146848
  * [41] Pickens, J., Cooper, M., Golovchinsky, G.: Reverted indexing for feedback and expansion. In: Proceedings of the 19th ACM international conference on Information and knowledge management. pp. 1049–1058 (2010)
  * [42] Roy, D., Carevic, Z., Mayr, P.: Studying retrievability of publications and datasets in an integrated retrieval system. In: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. JCDL ’22, Association for Computing Machinery (2022). https://doi.org/10.1145/3529372.3530931
  * [43] Roy, D., Carevic, Z., Mayr, P.: Retrievability in an integrated retrieval system: an extended study. International Journal on Digital Libraries (Apr 2023). https://doi.org/10.1007/s00799-023-00363-4, https://doi.org/10.1007/s00799-023-00363-4
  * [44] Traub, M.C., Samar, T., van Ossenbruggen, J., Hardman, L.: Impact of crowdsourcing ocr improvements on retrievability bias. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries. p. 29–36. JCDL ’18, Association for Computing Machinery (2018). https://doi.org/10.1145/3197026.3197046
  * [45] Traub, M.C., Samar, T., Van Ossenbruggen, J., He, J., de Vries, A., Hardman, L.: Querylog-based assessment of retrievability bias in a large newspaper corpus. In: 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL). pp. 7–16. IEEE (2016)
  * [46] Voorhees, E.M.: Overview of the TREC 2004 robust track. In: Voorhees, E.M., Buckland, L.P. (eds.) Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, Gaithersburg, Maryland, USA, November 16-19, 2004. NIST Special Publication, vol. 500-261. National Institute of Standards and Technology (NIST) (2004), http://trec.nist.gov/pubs/trec13/papers/ROBUST.OVERVIEW.pdf
  * [47] Wilkie, C., Azzopardi, L.: An initial investigation on the relationship between usage and findability. In: Advances in Information Retrieval: 35th European Conference on IR Research, ECIR 2013, Moscow, Russia, March 24-27, 2013. Proceedings 35. pp. 808–811. Springer (2013)
  * [48] Wilkie, C., Azzopardi, L.: Relating retrievability, performance and length. In: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. pp. 937–940 (2013)
  * [49] Wilkie, C., Azzopardi, L.: Best and fairest: An empirical analysis of retrieval system bias. In: de Rijke, M., Kenter, T., de Vries, A.P., Zhai, C., de Jong, F., Radinsky, K., Hofmann, K. (eds.) Advances in Information Retrieval. pp. 13–25. Springer International Publishing, Cham (2014)
  * [50] Wilkie, C., Azzopardi, L.: Efficiently estimating retrievability bias. In: de Rijke, M., Kenter, T., de Vries, A.P., Zhai, C., de Jong, F., Radinsky, K., Hofmann, K. (eds.) Advances in Information Retrieval. pp. 720–726. Springer International Publishing, Cham (2014)
  * [51] Wilkie, C., Azzopardi, L.: A retrievability analysis: Exploring the relationship between retrieval bias and retrieval performance. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. pp. 81–90 (2014)
  * [52] Wilkie, C., Azzopardi, L.: Query length, retrievability bias and performance. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. p. 1787–1790. CIKM ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2806416.2806604, https://doi.org/10.1145/2806416.2806604
  * [53] Wilkie, C., Azzopardi, L.: Retrievability and retrieval bias: A comparison of inequality measures. In: Advances in Information Retrieval: 37th European Conference on IR Research, ECIR 2015, Vienna, Austria, March 29-April 2, 2015\. Proceedings 37. pp. 209–214. Springer (2015)
  * [54] Zheng, L., Cox, I.J.: Document-oriented pruning of the inverted index in information retrieval systems. In: 2009 International Conference on Advanced Information Networking and Applications Workshops. pp. 697–702. IEEE (2009)
  * [55] Zhu, Y., Nie, J.Y., Dou, Z., Ma, Z., Zhang, X., Du, P., Zuo, X., Jiang, H.: Contrastive learning of user behavior sequence for context-aware document ranking. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. p. 2780–2791. CIKM ’21, Association for Computing Machinery (2021). https://doi.org/10.1145/3459637.3482243