# Improving Content Retrievability in Search with Controllable Query
Generation

Gustavo Penha1, Enrico Palumbo2, Maryam Aziz3, Alice Wang3, Hugues Bouchard4
Spotify1Netherlands, 2Italy, 3USA, 4Spain
<EMAIL_ADDRESS>
(2023)

###### Abstract.

An important goal of online platforms is to enable content discovery, i.e.
allow users to find a catalog entity they were not familiar with. A pre-
requisite to discover an entity, e.g. a book, with a search engine is that the
entity is _retrievable_ , i.e. there are queries for which the system will
surface such entity in the top results. However, machine-learned search
engines have a high retrievability bias, where the majority of the queries
return the same entities. This happens partly due to the predominance of
narrow intent queries, where users create queries using the title of an
already known entity, e.g. in book search “harry potter”. The amount of broad
queries where users want to discover new entities, e.g. in music search “chill
lyrical electronica with an atmospheric feeling to it”, and have a higher
tolerance to what they might find, is small in comparison. We focus here on
two factors that have a negative impact on the retrievability of the entities
(I) the training data used for dense retrieval models and (II) the
distribution of narrow and broad intent queries issued in the system. We
propose CtrlQGen, a method that generates queries for a chosen underlying
intent—narrow or broad. We can use CtrlQGen to improve factor (I) by
generating training data for dense retrieval models comprised of diverse
synthetic queries. CtrlQGen can also be used to deal with factor (II) by
suggesting queries with broader intents to users. Our results on datasets from
the domains of music, podcasts, and books reveal that we can significantly
decrease the retrievability bias of a dense retrieval model when using
CtrlQGen. First, by using the generated queries as training data for dense
models we make 9% of the entities retrievable—go from zero to non-zero
retrievability. Second, by suggesting broader queries to users, we can make
12% of the entities retrievable in the best case.

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
ACM Web Conference 2023; May 1–5, 2023; Austin, TX, USA††booktitle:
Proceedings of the ACM Web Conference 2023 (WWW ’23), May 1–5, 2023, Austin,
TX, USA††price: 15.00††doi: 10.1145/3543507.3583261††isbn:
978-1-4503-9416-1/23/04††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Figure 1. TSNE reduction of queries and entities when embedded with a Bi-
Encoder trained with query logs and clicked entities (left), and synthetic
queries from our proposed method CtrlQGen (right). The left model surfaces the
same four entities for most queries while six entities are never retrieved as
the most similar entity. The right model distributes the queries better, i.e.
has less retrievability bias.

In online content platforms, users can search for catalog entities111Catalog
entities are items from a platform that can be retrieved and/or recommended to
users. For example, the book “The Fellowship of the Ring by J.R.R Tolkien” is
an entity from an online book platform. We refer to such items as _entities_
throughout the paper. that they are already familiar with, for example, they
issue queries with the title of a track to listen to next or of a book they
would like to read. This type of search based on bibliographic data (e.g.
title, artist, author, etc) to find entities (Bainbridge et al., 2003) has
been referred to as _narrow_ intent queries (Hosey et al., 2019). However,
user information needs are diverse and can be more complex depending on their
current mindset (Li et al., 2019).

When users have an exploratory mindset, they have a higher tolerance and are
prone to explore different alternatives through _broad_ queries. Non-focused
information needs are generally complex and require multiple interactions.
Many users solve such information needs outside the search engine of the
platforms, by asking broad queries to other users in forums such as
subreddits222See for example https://www.reddit.com/r/musicsuggestions/ or
/r/booksuggestions/. as existing search systems are ineffective for broader
intents.

Broad intents are an opportunity to surface under-served entities that would
not be discovered otherwise without affecting user satisfaction (Tomasi et
al., 2020). While approaches to promote the discovery of entities have been
studied from the perspective of recommender systems (Aziz et al., 2021;
Mehrotra, 2021), they do not generalize to search engines where there is an
input query. A prerequisite to improving the discoverability of entities
through search is that the entity is retrievable. Azzopardi and Vinay (2008b)
defined the _retrievability of a document as how many queries lead to the
entity being surfaced in the top-k results._

For example, if we assume that the users will only interact with the top-1
ranked entity of the list, the dense retrieval model used to embed queries and
entities in the left Voronoi plot of Figure 1 would make one of the five
leftmost entities appear for every query (they are closest neighbors in the
embedded space). These entities would have a high concentration of
retrievability, i.e. retrievability bias, compared to the remaining entities
which have no query close in the embedding space. Retrievability bias limits
exploration, as it becomes harder to discover new entities through search when
they have low retrievability scores.

In this paper, we study the effect of _generating queries_ on the
retrievability of the system. Although the implications of query generation
techniques for training document/passage dense models have been studied in
detail (Liang et al., 2020; Ma et al., 2020; Wang et al., 2021; Dai et al.,
2022), little attention has been given to generating queries for _entities_
and their impact on the effectiveness and _retrievability bias_. Dense models
have shown promising results for different retrieval tasks (Lin et al., 2021),
requiring a significant amount of in-domain supervision data for training
(Thakur et al., 2021). Query generation approaches have shown to be effective
in generating training data for domains with a scarcity of labeled data (Liang
et al., 2020; Ma et al., 2020; Bonifacio et al., 2022).

Unlike previous approaches to query generation which are agnostic to search
intents, we propose CtrlQGen which controls for the underlying intent. By
generating both narrow and broad queries for an entity we are able to (I)
train the dense retrieval model for both types of intents and (II) suggest
broader and more exploratory queries to users. With the use of weak
supervision through the proposed _weak labeling functions_ , CtrlQGen does not
strictly require any training data to generate synthetic queries for a given
entity.

With our empirical evaluation using three datasets in the domains of music,
podcasts, and books we set out to answer the following research question: _To
what extent can we reduce the retrievability bias of entity search with
automatically generated queries without significant impact in the
effectiveness?_

Considering that the retrievability of an entity depends on (I) the retrieval
model which decides which entities are surfaced for each query and (II) the
set of queries used for the estimation, we generate two retrievability
debiasing hypotheses that focus on modifications to the retrieval model and
the set of queries respectively. Our first hypothesis, H1, is that training
dense retrieval models with CtrlQGen queries will lead to less retrievability
bias compared to training with real queries and their respective clicked
entities. The click data is prone to different biases, for example, many
queries will be issued for the most popular entities, i.e. popularity bias,
and after training the model on such data and this bias will be reinforced in
later interactions with the system. Conversely, with CtrlQGen we can obtain
pairs of query-entity to train the model for any given entity, which can be
randomly sampled from the collection.

Our second hypothesis, H2, is that suggesting broad queries using CtrlQGen
will lead to less retrievability bias. Narrow queries have by definition less
relevant entities than broad queries. By assisting users in formulating their
queries with the suggestion of broad queries we can potentially influence
users’ query behaviors and then have an impact on the query type distribution.

Our main findings and contributions are:

  * •

We introduce CtrlQGen, a novel method to generate queries for a given entity
conditioned on a desired underlying intent (narrow or broad). We demonstrate
two ways of using the generated queries: as training data for dense retrieval
models and as query suggestions.

  * •

We find positive evidence for H1: dense models fine-tuned on synthetic queries
have significantly less retrievability bias than models fine-tuned on click
data. When using the queries from the proposed CtrlQGen we reduce the
retrievability bias by 10% in terms of Gini scores on average when compared to
a model that uses the click data and make 9% of the Tracks collection of
entities retrievable—go from zero to non-zero retrievability score.

  * •

Regarding H2, we show that applying CtrlQGen for generating query suggestions
can reduce the retrievability bias of the system up to 9% percent and increase
the number of entities that have non-zero retrievability 11% for the Tracks
collection when using a Bi-Encoder model that was trained with an unbiased set
of queries.

Next, we describe the related work, followed by the proposed method in Section
3. Section 4 describes the experimental setup used to answer the research
question, followed by our experiments in Section 5. We conclude the paper in
Section 6.

##  2\. Related Work

We first discuss here related work on search for the domains considered here.
Then we look into retrievability followed by a discussion on query generation
techniques and their applications.

###  2.1. Entity Search

A number of studies have explored user behavior when searching for specific
entities such as music tracks, products, and books (Hosey et al., 2019;
Garcia-Gathright et al., 2018; Bainbridge et al., 2003; Laplante, 2008). While
focused searches have the goal of finding a specific entity, non-focused
searches involve broader intents, where the user is in an exploratory mindset
(Li et al., 2019; Su et al., 2018; Sguerra et al., 2022).

The Social Book Search Lab CLEF (Koolen et al., 2016) that ran from 2011 to
2016333The book corpus of the SBS tasks is no longer available. enabled a
number of studies in complex search for the book domain (Ullah and Khusro,
2020; Bogers and Petras, 2017; Bogers and Koolen, 2018; Ullah et al., 2021;
Chaa et al., 2018). A richer document representation for books which contains
for example reviews, tags, and controlled vocabulary was shown to have better
retrieval effectiveness. It has also been shown in the music domain that
multiple sources of data such as metadata, audio features, tags, and lyrics
lead to better effectiveness for downstream tasks (Kim et al., 2020). For
podcast search, the TREC2020 podcasts track (Jones et al., 2021) revealed that
adding the additional information of transcripts also leads to higher
effectiveness when compared to using only the episode title and description.

Another external source of information for entities that was shown to be
useful for downstream tasks (Stamatelatos et al., 2021) is the concept of
lists, where users group together a number of entities that are similar in a
way. In the music domain, this is often referred to as playlists. The creation
of lists with curated entities is also common in the domains of books (Liu et
al., 2014) and movies (Greene and Cunningham, 2013).

Figure 2. Left: Components of the CtrlQGen method. Each entity is serialized
by concatenating the values of each metadata, e.g. title: The Fellowship of
the Ring [SEP] author names: J.R.R. Tolkien […]. (2) Labeled data (entity ;
query ; intent) is not strictly required, due to the use of weak labeling
functions which output a query and intent for a given entity, e.g. (The
Fellowship of the Ring ; fantasy book ; broad). (3) Control over the
underlying intent (narrow or broad) when generating the query via prompting,
e.g. “Generate a query with narrow/broad intent from: ¡serialized_entity¿”.
Right: Different ways of using the proposed CtrlQGen to improve the
retrievability of the search system: modifying the ranker by fine-tuning on
synthetic queries and modifying the set of queries by suggesting broad queries
for the narrow intent queries issued.

###  2.2. Retrievability

To estimate the retrievability of a document (Azzopardi and Vinay, 2008a, b)
proposed to sum the popularity of the queries that retrieve the given document
above a position that the user would actually look at (e.g. in the top-5
documents). Retrievability scores can be used to determine if a retrieval
system has a concentration of retrievability, for example, to verify if
certain types of documents are being surfaced more than others. For example,
(Roy et al., 2022) showed that for a collection with datasets and articles,
the retrievability bias was stronger for datasets when compared with articles.

Even though a system with less retrievability bias does not necessarily mean
that the system is more effective, studies have found a correlation between
the two (Bashir and Rauber, 2017; Wilkie and Azzopardi, 2013, 2014),
suggesting that a measure of retrievability bias can potentially be used to
select better retrieval systems. In order to reduce the retrievability bias of
a system (Bashir and Rauber, 2010) proposed a query expansion technique with a
novel document selection process for pseudo-relevance feedback in the domain
of patent search. Chakraborty et al. (2020) proposed to use retrievability of
a document over a set of query variations to decide which documents to use for
relevance feedback. Finding which queries lead to a document can be also used
to improve search transparency (Li et al., 2022).

###  2.3. Query Generation

Query generation techniques can be broadly categorized based on their input:
documents or queries. For generating known-item queries for a given document,
i.e. queries where the task is to find a previously seen document, techniques
have been proposed that select a number of document terms based on different
sampling methods (Kumar et al., 2011; Azzopardi et al., 2007; Azzopardi and De
Rijke, 2006). Liu et al. (2021) tackled a similar problem with the additional
constraint that the generated queries are also informative. Generating queries
for a given document using a seq2seq model was first proposed by (Nogueira et
al., 2019b). Unlike sampling methods proposed for generating known-item
queries, a seq2seq approach such as docT5query (Nogueira et al., 2019a) is
able to generate queries where its terms do not occur in the input document,
being able to mitigate the vocabulary mismatch problem. Similarly, (Liang et
al., 2020; Ma et al., 2020; Wang et al., 2021) generate queries based on
documents using a transformer encoder-decoder model, but instead of using the
queries for document augmentation, they employ the queries as additional
training data for training bi-encoders, leading to significant gains in
retrieval effectiveness—specially in cross-domain evaluation settings. It has
also been proposed to replace the fine-tuned encoder-decoder model to generate
queries with little supervision by doing in-context learning with models such
as GPT-3 (Dai et al., 2022; Bonifacio et al., 2022). Zhuang et al. (2022) used
generated queries with the goal of improving the effectiveness of the emerging
differentiable search indexes. Another recent direction for query generation
is to incorporate explicit knowledge when generating queries, e.g. with the
use of knowledge graphs (Shen et al., 2022; Cho et al., 2022; Han et al.,
2019).

Generating queries for a given query has also been shown to be useful in IR.
For example by generating query suggestions or reformulations that help users
explore and express their information needs (Cao et al., 2008; Mei et al.,
2008). Another objective is to generate query variants that can be used to
obtain more effective ranking models by combining such variants for the given
query (Belkin et al., 1995; Benham et al., 2019), and also to better evaluate
ranking models (Penha et al., 2022; Zuccon et al., 2016; Bailey et al., 2017).

The closest to our problem is the generation of queries for product search.
Lien et al. (2022) used textual data from the reviews associated with the
documents (products) to generate queries automatically for the following
products: headphones, tents, and conditioners. In the domain of movies,
Bassani and Pasi (2021) generated queries automatically for a document (a
movie) based on a number of predefined semantic components such as genre and
year. We propose here a method to generate queries that can take advantage of
manually created functions as a weak supervision signal, and also employ pre-
trained language models. Unlike previous methods to generate queries, CtrlQGen
does intent-aware generation, where it is possible to control for the
underlying intent of the output query.

##  3\. Controllable Query Generation

In this section, we first describe the three main components of the proposed
CtrlQGen followed by different applications for the generated queries. Figure
2 displays a diagram of the model, as well as two different ways to employ the
synthetic queries. The serialization module is required to obtain a text
representation for a given entity so that text-based models can use that as
input. The second component is called weak labeling, which is able to bypass
the need for a large amount of labeled data. Finally, the last component is
the intent-aware generation, which is able to control for different types of
intent (broad and narrow).

###  3.1. Model Components

####  3.1.1. Serialization

This module takes as input an entity $e$ and outputs a string representation
of the entity: $e_{serialized}=s(e)$. The serialization function $s$
concatenates every metadata column of the entity with their respective values,
using a special token:
$s(e)={\color[rgb]{.75,.5,.25}col_{1}}:val_{1}[SEP]{\color[rgb]{.75,.5,.25}col_{2}}:val_{2}[SEP]...[SEP]{\color[rgb]{.75,.5,.25}col_{n}}:val_{n}$.
So for example the book with the title The Fellowship of the Ring becomes:

title: The Fellowship of the Ring [SEP] series name: The Lord of the Rings #1
[SEP] author names: J.R.R. Tolkien [SEP] publication year: 1954 [SEP]
language: EN [SEP] genres: Fantasy, Classics, Fiction, Adventure, High
Fantasy, …[SEP] description: One Ring to rule them all, One Ring to find them
…[SEP] review: This book is full of wonder and adventure with fantastic
writing … [SEP] lists: fantasy, uk-and-ireland, witches-wizards, fiction,
british, …

####  3.1.2. Weak Labeling

In order to train CtrlQGen we require a dataset
$\mathcal{D}=\\{(e_{i},i_{i},q_{i})\\}_{i=1}^{M}$ with training triplets of
entity, intent, and query, which are the input, control variable, and output
respectively. In each triplet, the query $q$ has the underlying intent $i$
(narrow or broad) when matching with the entity $e$. One option to acquire
such data is to ask annotators to create narrow and broad queries for a given
entity. Alternatively, we can employ weak labeling functions that generate
such data based on heuristics.

We present here two flavors of weak labeling functions. The first is
completely _unsupervised_ (WeakLabeling-Un), and thus is able to generate both
query and intents for any given entity. The second requires queries that are
related to each entity and thus is based on _intent prediction_ of the given
query (WeakLabeling-IP).

Figure 3. Two variants of the weak labeling functions. While WeakLabeling-Un
(top) outputs query and intents for a given entity, WeakLabeling-IP (bottom)
outputs an intent label for a given pair of entity and query.

WeakLabeling-Un. The core intuition is that we can define a set of metatada
columns that are inherently associated with narrow intent queries since they
can identify the entity (narrow-fields), e.g. title and artists, and a set of
metadata columns that capture characteristics of the entity that other
entities might also have, e.g. genres, and thus can be considered to be broad
columns (broad-fields). In order to generate a set of queries and intents for
a given entity we rely on randomly sampling terms from all possible
combinations of the respective fields. So for example to generate a narrow
intent query in the music domain, we could use either the title of a track,
the album, the artist, or a combination of the three. After sampling terms
from such the respective columns for the query, we apply a number of functions
to generate query variations in a stochastic manner: shuffling words, adding
misspellings, and removing prefixes444Since the datasets considered come from
a large-scale online platform with _instant search_ , many log queries are not
complete and are just prefixes of the entity titles. This happens because the
user might stop before the end of the query as the result could be already
found in the list of results..

Specifically, when generating broad queries, we differentiate between metadata
columns that are based on free text (broad-fields-ft), e.g. reviews, and the
ones which are already category-like terms (broad-fields), e.g. genres. For
the free text columns, in order to avoid selecting terms that are
uninformative, we apply a sampling strategy that prioritizes terms with higher
IDF. As another weak labeling function for the free text columns, we apply a
text summarization model to select more informative terms.

WeakLabeling-IP In order to take advantage of existing data of entities and
queries, e.g. query logs with clicked entities, this variant predicts if the
query is broad or narrow based on its narrow and broad columns. If the
similarity555We employ here a transformer sentence representation and cosine
similarity. of the query and the values of the narrow queries is higher than
the similarity of the query with the values of the broad queries then the weak
label will be narrow, otherwise broad. So for example, if the entity is a book
with the title “The Brothers Karamazov”, and the input query is “Karamazov”,
the label would be narrow whereas if the input query is “russian theological
fiction” the label would be broad as it would be more similar to the
categories of the book.

####  3.1.3. Intent-aware Generation

Given the training dataset $\mathcal{D}=\\{(e_{i},i_{i},q_{i})\\}_{i=1}^{M}$,
we train an encoder-decoder model $G$ that receives as input the entity and
the underlying intent to control for, and it outputs the query: $G(e,i)=q$. In
order to achieve that we rely on adding the control variable as part of the
language model prompt. We train the model with the following prompt: “Generate
a query with narrow/broad intent from: ¡serialized_entity¿” and its respective
query as the output. So for example the query “lord of th” with intent narrow
would lead to the following training instance:

Input Generate a narrow query from: title: The Fellowship of the Ring [SEP]
series name: The Lord of the Rings #1 [SEP] author names: J.R.R. Tolkien [SEP]
publication year: 1954 [SEP] language: EN [SEP] genres: Fantasy, Classics,
Fiction, Adventure, High Fantasy, …[SEP] description: One Ring to rule them
all, One Ring to find them …[SEP] review: This book is full of wonder and
adventure with fantastic writing … [SEP] lists: fantasy, uk-and-ireland,
witches-wizards, fiction, brittish, … Output lord of th

###  3.2. Applications

####  3.2.1. Synthetic training data

We can use the generated queries to train Bi-Encoder retrieval models, as
shown on the right top part of Figure 2. For a randomly sampled set of
entities $\mathcal{E^{\prime}}$ from the collection $\mathcal{E}$ we apply
CtrlQGen with both desired intents
$q^{\prime}_{narrow}=G(e,\texttt{narrow}{})$ and
$q^{\prime}_{broad}=G(e,\texttt{broad}{})$ for each $e$ in
$\mathcal{E^{\prime}}$. After that, given a desired weight proportion of broad
queries and narrow queries ($P_{narrow}$, $P_{broad}$) we can sample training
instances from the synthetic generated queries $\mathcal{Q^{\prime}}$ for
training the Bi-Encoder. This gives us a dataset of pairs of synthetic queries
and respective relevant entities that can be used to train Bi-Encoder models,
controlling for the desired proportion of underlying intents.

####  3.2.2. Query suggestion

We can employ the CtrlQGen model to perform query suggestion, as shown on the
bottom right part of Figure 2. Since the majority of the queries for entities
have a narrow intent behind them, one approach to modifying the user’s
behavior is to suggest broader queries. In order to do that, we can employ the
generated queries in the following manner. First, for a given input query $q$,
we can obtain a list $\mathcal{R}_{q}$ with the top-k entities ranked for it
using a ranking model. For each entity in the top-k ranked list, we apply
CtrlQGen to generate a set of broad queries $\mathcal{Q^{\prime}}$ to
recommend: $\mathcal{Q^{\prime}}=\\{G(e_{i},\texttt{broad}{})\\}$ for $e_{i}$
in $\mathcal{R}_{q}$666In our experiments this set of queries
$\mathcal{Q^{\prime}}$ is appended, according to a percentage of acceptance,
to the set of log queries $\mathcal{Q}$ in order to calculate the
retrievability bias.. The complexity of this approach is $O(n^{2}*d*k)$, where
$n$ is the sequence length, $d$ is the number of dimensions of the transformer
model and $k$ is the size of the list considered to generate suggestions.

Table 1. Datasets metadata and statistics. Metadata columns 1–3 are considered to be narrow-fields, whereas 4–9 are broad-fields. In the experiments the broad columns which are free-text (broad-fields-ft) are: Episode & show description and Transcript for Podcasts, and User reviews and Description for Books. |  | Tracks | Podcasts | Books
---|---|---|---|---
Metadata |  | (1)
---
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
| Title
---
Album name
Artist names
Release year
Language
Genres
Descriptors
Lyric
User Playlists
| Title
---
Show name
Host names
Ingested date
Language
Categories
Episode & show description
Transcript
Topics
| Title
---
Series name
Author names
Publication year
Language
Genres
Description
User reviews
User lists
# docs | 682k | 600k | 617k
# queries | 100k | 100k | 100k
|  Click # qrels
---
train/val/test
75.9k/9.5k/9.5k | 14.4k/1.8k/1.8k | 117.5k/14.7k/14.7k
Avg doc len | 55.87 | 80.76 | 161.58
Avg query len | 1.96 | 3.06 | 4.47

##  4\. Experimental Setup

In this section, we first describe the data used to test our hypotheses,
followed by the implementation details of the methods and baselines as well as
how we evaluate different approaches.

###  4.1. Datasets

In order to test our hypothesis and compare different methods to generate
queries we rely on three datasets: Tracks, Podcasts, and Books. For each
dataset, we have a set of entities (¿600k entities), a set of 100k queries,
and a set of relevance judgements. Table 1 describes the statistics of the
datasets and examples of entities.

While the queries and entities from Tracks and Podcasts were extracted from a
large-scale online platform the Books dataset is a subset of the Goodreads
public dataset from (Wan and McAuley, 2018)
777https://github.com/MengtingWan/goodreads. The query sets from Tracks and
Podcasts are a unique subset of randomly sampled entities and queries from the
logs of a large scale online audio platform, where clicks for a given entity
after issuing the query are considered to be the relevance signal in our
experiments.

We also use the number of distinct users for which the query was issued by,
and use them as $o_{q}$ to calculate retrievability scores (see Section 4.3).
Regarding the columns, as seen in Table 1, the ones with numbers 1–3 are
considered to be narrow-fields, whereas 4–9 are broad-fields. The broad
columns which are free-text (broad-fields-ft) are: Episode & show description
and Transcript for Podcasts, and User reviews and Description for Books.

Since the Goodreads dataset does not have any set of queries available, we
generate a set of queries automatically: 75% of the queries are narrow,
generated by sampling words from the narrow-fields, and 25% of them are from
broad-fields and consider that as the relevance labels. This specific split of
narrow and broad queries was chosen to simulate actual user behavior observed
in the other two datasets (Tracks and Podcasts) where narrow queries are the
majority but in a less extreme fashion. We use the number of ratings the
entities from Books have as a proxy for the number of users that would issue
such queries ($o_{q}$).

####  4.1.1. Broad queries datasets

Since the majority of the queries from Tracks, Podcasts and Books are narrow,
we also employ two smaller additional sets of queries and relevance labels
that have an underlying broad intent. They are
$\texttt{Tracks}{}_{\texttt{broad}{}}$ and
$\texttt{Podcasts}{}_{\texttt{broad}{}}$, containing a total of 1309 and 500
queries. The $\texttt{Tracks}{}_{\texttt{broad}{}}$ is a sample of queries
from the logs that have a high predicted probability of being broad based on
the interaction signals the user had after issuing the query. If the user
interacts with entities such as playlists and hubs more than tracks and albums
they are more likely to be issuing a broad query. Based on this set, we get
the clicked entities where the query does not match the title, artist, or
album of the entity, avoiding cases where the query seems broad but is in fact
a narrow interaction, e.g. query “pop” and clicking a track with the title
“POP!”. For $\texttt{Podcasts}{}_{\texttt{broad}{}}$, there is no parallel for
the Tracks playlists so we employ a set of manually curated pairs of broad
queries and entities. Annotators were instructed to write a query relevant to
the podcast episode while avoiding exact matches and matching diverse metadata
fields.

###  4.2. Implementation Details

####  4.2.1. Query generation models

As baselines for generating synthetic queries, we first use QGen, a common
approach to generate queries from documents used in this manner in different
previous work (Nogueira et al., 2019a; Liang et al., 2020; Ma et al., 2020;
Wang et al., 2021). We rely on fine-tuning T5 (Raffel et al., 2019) (t5-base)
on a subset of Click train set with 10k pairs query-entities. The second
baseline for generating queries requires very little supervision signal:
InPars (Bonifacio et al., 2022). The model uses in-context learning, i.e. few
examples in the prompt of the document and expected query, and large language
models. For a fair comparison, we randomly sample examples to use in the
prompt every time we are generating the output queries, this way InPars has
access to the same amount of training pairs of query and entities as
QGen888This was shown to be effective for the validation sets of Podcasts and
Books. For Tracks we did not observe the same, so we used a fixed prompt with
the same two examples randomly selected from the dataset.. We rely on the open
bigscience/bloom-760m999https://bigscience.huggingface.co/blog/bloom release
to do so101010We explore larger GPT-3 models on the appendix and see that the
larger 175B parameter one does not significantly improve over smaller models..
For the CtrlQGen implementation we also rely on the T5 (t5-base) model. When
generating the queries with T5, for both QGen and CtrlQGen we employ
$do\\_sample$=True and $top\\_k$=10.

####  4.2.2. Retrieval models

For BM25 (Robertson and Walker, 1994) we resort to the default hyperparameters
and implementation provided by the PyTerrier toolkit (Macdonald and
Tonellotto, 2020). For the zero-shot Bi-Encoder models, we rely on the
SentenceTransformers (Reimers and Gurevych, 2019) model
releases111111https://www.sbert.net/docs/pretrained_models.html. The library
uses Hugginface transformers for the pre-trained models such as BERT (Devlin
et al., 2018) and MPNet (Song et al., 2020). Specifically, we employ the pre-
trained model all-mpnet-base-v2. When fine-tuning the Bi-Encoder models on the
Click or synthetic datasets, we rely on the MultipleNegativesRankingLoss,
which uses in-batch random negatives to train the model. We fine-tune the
dense models for a total of 10k steps. Thus, all dense models were trained on
the same amount of (synthetic or not) queries. We use a batch size of 8, with
10% of the training steps as warmup steps. The learning rate is 2e-5 and the
weight decay is 0.01. We refer to the Bi-Encoder model trained on Click data
as $\texttt{Bi-Encoder}_{\texttt{Click}{}}$ and a Bi-Encoder model trained on
the queries from CtrlQGen as $\texttt{Bi-Encoder}_{\texttt{CtrlQGen}{}}$.

###  4.3. Evaluation Procedure

To evaluate the effectiveness of the retrieval systems we use the recall at
100, $R@100$. The choice for R@100 is due to the objective of increasing the
retrievability of items considering the first 100 options121212A second stage
re-ranker in this pipeline could be precision-focused if the retriever is able
to find enough relevant and diverse options.. We perform Students t-tests at
the confidence level of 0.95 with Bonferroni correction to compare the
difference between models with statistical significance.

To evaluate how biased the retrieval system is in terms of retrievability, we
first estimate the retrievability of an entity $e$ as defined by (Azzopardi
and Vinay, 2008b): $r(\mathbf{e})=\sum_{\mathbf{q}\in\mathbf{Q}}o_{q}\cdot
f\left(k_{eq},c\right)$, where $\mathbf{Q}$ is the set of queries131313The
size of $\mathbf{Q}$ is 100k for all computations., $o_{q}$ is the weight of
each query—here we use the number of users that issued the query—and
$f\left(k_{eq},c\right)$ is 1 if the entity $e$ is ranked above $c$ by the
search system (in our experiments we set c=100) and 0 otherwise. In order to
get a number that summarizes how concentrated or biased the retrievability
scores are we calculate the Gini score (Gastwirth, 1972):
$G=\frac{\sum_{i=1}^{N}(2*i-N-1)*r\left(\mathbf{e}_{\mathbf{i}}\right)}{N\sum_{j=1}^{N}r\left(\mathbf{e}_{\mathbf{j}}\right)}$,
where G=1 means only one entity concentrates all the retrievability, and G=0
means every entity in the collection has the same retrievability score. In
order to perform statistical testing for the Gini scores we follow (Gamboa et
al., 2010).

Table 2. Retrieval effectiveness (R@100$\uparrow{}$ the higher the better) and retrievability bias (Gini $\downarrow{}$ the lower the better) of dense retrieval models trained on different training data for predominantly narrow queries (Click test sets). Bold indicate the best model for each category with statistical significance and superscripts indicate statistically significant improvements over the respective model using students t-test at 0.95 confidence with Bonferoni correction for multiple comparisons. The values for the Books dataset on row ($c$) are not included as they are already a synthetic set of queries.  | R@100$\uparrow{}$ | Gini$\downarrow{}$
---|---|---
| Zero-shot
---
(no target domain Click training data)
Tracks |  Podcasts |  Books |  Tracks |  Podcasts |  Books
(a) BM25 |  0.182b |  0.436b |  0.721bd |  0.752bh |  0.666bcdefh |  0.779b
(b) Bi-Encoder |  0.142 |  0.323 |  0.415 |  0.818h |  0.765 |  0.836d
(c) Bi-Encoder${}_{\texttt{WeakLabeling-Un}}$ (Ours) |  0.222 abd |  0.465b |  - |  0.748abh |  0.730bh |  -
| Fine-tuned on synthetic data
---
(target domain Click training data to train query generators)
(d) Bi-Encoder${}_{\texttt{InPars}{}}$ (Bonifacio et al., 2022) |  0.202ab |  0.474ab |  0.492b |  0.712abch |  0.677bch |  0.842
(e) Bi-Encoder${}_{\texttt{QGen}{}}$ (Ma et al., 2020) |  0.296abcd |  0.503abc |  0.755abd |  0.701abcdh |  0.674bcdfh |  0.766abgh
(f) Bi-Encoder${}_{\texttt{CtrlQGen}{}}$ (Ours) |  0.333abcde |  0.500abc |  0.770abde |  0.693abcdeh |  0.676bdch |  0.762abegh
| Fine-tuned on target data or in combination with synthetic data
---
(access to target domain Click training data)
(g) Bi-Encoder${}_{\texttt{Click}{}+\texttt{CtrlQGen}{}}$ (Ours) |  0.361abcdef |  0.622abcdef |  0.775abde |  0.817bh |  0.741bh |  0.768 ab
(h) Bi-Encoder${}_{\texttt{Click}{}}$ |  0.366abcdef |  0.634abcdef |  0.769abde |  0.856 |  0.763b |  0.767abg

##  5\. Results

In this section, we first describe the experimental results on H1—training
dense retrieval models on synthetic queries leads to less retrievability bias
than training on real queries and clicked entities—followed by the results for
H2—suggesting broad queries generated by our proposed method CtrlQGen leads to
less retrievability bias when compared to the set of queries from the logs.

###  5.1. H1: Modifying the Ranker with Generated Queries as Training Data

#### Evaluation with narrow intent queries

Table 2 displays R@100 and Gini scores for different retrieval models on the
three datasets which contain mostly narrow intent queries. Zero-shot models do
not have access to any Click relevance labels for training. As expected, a Bi-
Encoder that has no access to the target domain queries and entities does not
perform well, and it has worse effectiveness than BM25 (30% less R@100 on
average, as seen row a vs row b). When using the target training data to fine-
tune the dense retrieval model (Bi-Encoder${}_{\texttt{Click}}{}$) we observe
that it outperforms zero-shot models significantly, with absolute gains of
R@100 up to 158% (row h vs row b). However, both the model trained on the
Click data (row h) and the pre-trained Bi-Encoder (row c) have significantly
more bias than BM25, as seen by the Gini scores increases of 9.2% and 10%
respectively.

When using the synthetic queries created by any of the query generation models
to train the dense retrieval methods (InPars, QGen, CtrlQGen) as described in
Section 3.2.1, we observe significant drops of 10% Gini on average (rows d,e,f
vs row h), _indicating positive evidence for our first hypothesis_ that a
model trained on the synthetic queries lead to less retrievability bias than
the model trained on the Click data. Specifically, with CtrlQGen141414We
employ here CtrlQGennarrow which sets ($P_{narrow}$, $P_{broad}$) as ($100\%$,
$0\%$) and WeakLabeling-IP as found to be optimal in the validation
experiments (c.f. Table 5.1). we show that we can get statistically
significant better effectiveness and retrievability for Tracks and Books than
the query generation baselines with 24% more R@100 and 3% less Gini on average
over all datasets and baselines (row f vs rows d, e). We also show that we
improve the retrievability over the model trained on Click data (row f vs row
h) by 10% Gini. This effectively makes more than 62k (9%) entities in the
Tracks dataset retrievable compared to $\texttt{Bi-
Encoder}_{\texttt{Click}{}}$, i.e. the entity goes from zero to a non-zero
value.

We see also that with a combination of synthetic queries from CtrlQGen and
queries from the Click dataset 151515We set the percentage as the optimal one
in the validation set: 10% synthetic queries and 90% queries and clicks from
Click. (row g) we can achieve similar effectiveness to the model training on
the Click dataset (no statistical difference) while having less retrievability
bias for both Tracks and Podcasts datasets with statistical significance,
being Pareto optimal when considering both objectives.

Table 3. Retrieval effectiveness (R@100$\uparrow{}$ the higher the better) and retrievability bias (Gini $\downarrow{}$ the lower the better) of dense models trained on different training data for a subset of broad queries. Bold indicates the best model for each category with statistical significance and superscripts indicate statistically significant improvements over the respective model using Students t-test at 0.95 confidence with Bonferoni correction. When the set of queries $\mathcal{Q^{\prime}}$ were generated with ($P_{narrow}$, $P_{broad}$) as ($100\%$, $0\%$), we refer to it as $\texttt{CtrlQGen}{}_{narrow}$, with ($0\%$, $100\%$) we call it $\texttt{CtrlQGen}{}_{broad}$ and with ($50\%$, $50\%$) as $\texttt{CtrlQGen}{}_{both}$. Method | R@100 $\uparrow$ | Gini $\downarrow$
---|---|---
| Tracks${}_{\texttt{broad}{}}$ | Podcasts${}_{\texttt{broad}{}}$ | Tracks${}_{\texttt{broad}{}}$ | Podcasts${}_{\texttt{broad}{}}$
(a) Bi-Encoder${}_{\texttt{CtrlQGen}{}_{broad}}$ | 0.074bcdef | 0.800ef | 0.596b | 0.831bf
(b) Bi-Encoder${}_{\texttt{Click}{}}$ | 0.035def | 0.756f | 0.878 | 0.846
(c) Bi-Encoder${}_{\texttt{CtrlQGen}{}_{both}}$ | 0.033def | 0.780f | 0.492 abef | 0.831bf
(d) Bi-Encoder${}_{\texttt{InPars}{}}$ (Bonifacio et al., 2022) | 0.010 | 0.827cef | 0.489abcef | 0.816abcf
(e) Bi-Encoder${}_{\texttt{QGen}{}}$ (Ma et al., 2020) | 0.009 | 0.744f | 0.540ab | 0.820abcf
(f) Bi-Encoder${}_{\texttt{CtrlQGen}{}_{narrow}}$ | 0.003 | 0.609 | 0.517abe | 0.835b

#### Evaluation with broad intent queries

In order to understand how the models perform for exploratory and complex
information needs, we take a closer look at the effectiveness and
retrievability of the models in a set containing only broad intent queries.
Table 3 shows that a model trained on synthetic queries from CtrlQGen gets
significantly better when we include broader queries in the training (going
from 0, 50 and 100% on rows f, c and a).

A model trained only on synthetic broad queries outperforms a model trained on
Click data by 111% of R@100 for Tracks${}_{\texttt{broad}{}}$ with statistical
significance (row a vs row b). We also observe significant drops in the
retrievability bias when we compare models trained with CtrlQGen queries with
models trained with Click data, going from 0.878 to 0.596 and from 0.846 to
0.831 as seen in Table 3 (row a vs row b). Baseline models to generate queries
(rows d and e) also have significantly less retrievability bias than the model
trained on click data (row b) _showing again positive evidence for our first
hypothesis_ that models trained on synthetic queries lead to less
retrievability bias.

Table 4. Ablation study where we add components of the proposed CtrlQGen to
QGen one at a time using Click validation sets which has predominantly narrow
queries. We also report the effectiveness for the broad queries only datasets.
The $\dagger$ superscripts indicate statistically significant improvements
over QGen using students t-test at 0.95 confidence.

|  | Predominantly narrow queries | Broad queries
---|---|---|---
|  | R@100$\uparrow{}$ |  Gini $\downarrow$ | R@100$\uparrow{}$ |  Gini $\downarrow$
|  | Tracks | Podcasts | Books | Tracks | Podcasts | Books | Tracks${}_{\texttt{broad}{}}$ | Podcasts${}_{\texttt{broad}{}}$
QGen (Ma et al., 2020) | 0.289 | 0.512 | 0.756 | 0.701 | 0.674 | 0.766 | 0.009 | 0.744 | 0.540 | 0.820
|  |  |  |  |  |  |  |  |  |
\+ (1) Serialization | 0.312† | 0.509 | 0.761† | 0.694† | 0.666† | 0.761† | 0.006 | 0.711 | 0.528† | 0.824
\+ (2a,3) Intent-aware Generation | 0.305† | 0.505 | 0.761† | 0.688† | 0.669† | 0.756† | 0.008 | 0.751 | 0.528† | 0.820
${}^{\texttt{WeakLabeling-IP}}$ |  |  |  |  |  |  |  |  |  |
\+ (2a,2b,3) Intent-aware Generation | 0.289 | 0.508 | - | 0.704 | 0.704 | - | 0.036† | 0.787† | 0.486† | 0.829
${}^{\texttt{WeakLabeling-IP}+\texttt{WeakLabeling-Un}}$ |  |  |  |  |  |  |  |  |  |
\+ (1,2a,3) CtrlQGen | 0.300† | 0.522† | 0.763† | 0.694† | 0.676 | 0.760† | 0.007 | 0.713 | 0.522† | 0.822
${}^{\texttt{WeakLabeling-IP}}$ |  |  |  |  |  |  |  |  |  |
\+ (1,2a,2b,3) CtrlQGen | 0.283 | 0.490 | - | 0.701 | 0.674 | - | 0.033† | 0.780† | 0.480† | 0.833
${}^{\texttt{WeakLabeling-IP}+\texttt{WeakLabeling-Un}}$ |  |  |  |  |  |  |  |  |  |

#### Contribution of each module of CtrlQGen

When using the set of queries generated by WeakLabeling-Un(no supervision
available) to train the dense retriever, we get statistically significant
improvements over the model that is not fine-tuned (row c vs row b, going from
0.142 to 0.222 R@100 and from 0.323 to 0.465 R@100 for the Tracks and Podcasts
datasets as seen in Table 2. Using the remaining components of CtrlQGen we
obtain significant improvements over using solely WeakLabeling-Un. A natural
question is from which modules the improvements are coming. To answer this
question Table 5.1 displays an ablation study on the components of CtrlQGen.
If we remove all components of CtrlQGen we end up with the baseline QGen
(first line of the table). We incrementally add each component of the model in
the following rows.

The main findings are that (I) the serialization component, where we indicate
which the metadata columns and their respective values as opposed to values
only, is beneficial for both R@100 and Gini for both the narrow evaluation set
of queries and broad set of queries and (II) the WeakLabeling-Un is only
beneficial for the broad set of queries as WeakLabeling-IP cover narrow
queries well (they are the majority of the available existing queries).

###  5.2. H2: Modifying the Set of Queries by Suggesting Generated Queries

In order to test our second hypothesis that suggesting broad queries with
CtrlQGen leads to less retrievability bias compared to the queries found in
the logs, we rely on a simulation where a percentage of suggested queries are
accepted and added into the set of queries as described in Section 3.2.2. For
each entity in the top-5 ranked list for the log queries we create 3 query
suggestions.

Figure 4 displays the results of this simulation. We see that if the Bi-
Encoder is trained on a set of broad queries, the retrievability of the system
drops significantly as higher percentages of suggested broad queries by
CtrlQGen are accepted, with decreases of Gini up to 11% and 7% for Tracks and
Podcasts _showing positive evidence for our second hypothesis_. If we consider
that all queries are accepted by the users a total of 78k (11%) entities for
the Tracks dataset would become retrievable, i.e. retrievability different
than zero, compared to using CtrlQGenboth with the log queries. We also see
that only modifying the set of queries is not enough, as a Bi-Encoder trained
on the Click data does not achieve the same effect, showing that it is also
necessary to employ a model that was trained for both narrow and broad
queries.

Figure 4. Suggesting broad queries with CtrlQGen reduces the retrievability of
a model trained on synthetic data.

##  6\. Conclusion

We propose here CtrlQGen, a new approach to generate synthetic queries for
entities that allows to control for the query intent and that can work in the
absence of annotated data through the use of a weak-labeling function that
leverages content metadata. We study the impact that the generated queries
have on decreasing the retrievability bias and effectiveness, i.e. on helping
the search engine surface more entities while avoiding negative effects on the
relevance of results. Our experimental results in three different domains show
that training dense retrieval models on synthetic queries from CtrlQGen leads
to significant decreases in the retrievability bias of the system with
comparable effectiveness. We also demonstrate how to reduce the retrievability
bias by suggesting queries generated by CtrlQGen.

As future work, we believe important directions to be: (I) taking into account
the interplay between recommendation and search in the measure of the
accessibility of an entity, (II) improving the representation of entities for
which most metadata information is not available and (III) study methods to
reduce the retrievability of a system for re-ranking scenarios (IV) study the
impact of increased content retrievability on content discovery.

## References

  * (1)
  * Aziz et al. (2021) Maryam Aziz, Alice Wang, Aasish Pappu, Hugues Bouchard, Yu Zhao, Benjamin Carterette, and Mounia Lalmas. 2021.  Leveraging Semantic Information to Facilitate the Discovery of Underserved Podcasts. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3707–3716.
  * Azzopardi and De Rijke (2006) Leif Azzopardi and Maarten De Rijke. 2006.  Automatic construction of known-item finding test beds. In _Proceedings of the 29th annual international ACM SIGIR conference on Research and Development in Information Retrieval_. 603–604.
  * Azzopardi et al. (2007) Leif Azzopardi, Maarten De Rijke, and Krisztian Balog. 2007\.  Building simulated queries for known-item topics: an analysis using six european languages. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 455–462.
  * Azzopardi and Vinay (2008a) Leif Azzopardi and Vishwa Vinay. 2008a.  Accessibility in information retrieval. In _European Conference on Information Retrieval_. Springer, 482–489.
  * Azzopardi and Vinay (2008b) Leif Azzopardi and Vishwa Vinay. 2008b.  Retrievability: An evaluation measure for higher order information access tasks. In _Proceedings of the 17th ACM conference on Information and knowledge management_. 561–570.
  * Bailey et al. (2017) Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2017.  Retrieval consistency in the presence of query variations. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 395–404.
  * Bainbridge et al. (2003) David Bainbridge, Sally Jo Cunningham, and J Stephen Downie. 2003\.  How people describe their music information needs: A grounded theory analysis of music queries.  (2003).
  * Bashir and Rauber (2010) Shariq Bashir and Andreas Rauber. 2010.  Improving retrievability of patents in prior-art search. In _European Conference on Information Retrieval_. Springer, 457–470.
  * Bashir and Rauber (2017) Shariq Bashir and Andreas Rauber. 2017.  Retrieval Models Versus Retrievability.  In _Current Challenges in Patent Information Retrieval_. Springer, 185–212.
  * Bassani and Pasi (2021) Elias Bassani and Gabriella Pasi. 2021.  Semantic Query Labeling Through Synthetic Query Generation. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, Canada) _(SIGIR ’21)_. Association for Computing Machinery, New York, NY, USA, 2278–2282.  https://doi.org/10.1145/3404835.3463071
  * Belkin et al. (1995) Nicholas J. Belkin, Paul Kantor, Edward A. Fox, and Joseph A Shaw. 1995.  Combining the evidence of multiple query representations for information retrieval.  _Information Processing & Management_ 31, 3 (1995), 431–448.
  * Benham et al. (2019) Rodger Benham, Joel Mackenzie, Alistair Moffat, and J Shane Culpepper. 2019.  Boosting search performance using query variations.  _ACM Transactions on Information Systems (TOIS)_ 37, 4 (2019), 1–25.
  * Bogers and Koolen (2018) Toine Bogers and Marijn Koolen. 2018.  “I’m looking for something like…”: Combining Narratives and Example Items for Narrative-driven Book Recommendation. In _Knowledge-aware and Conversational Recommender Systems Workshop_. CEUR Workshop Proceedings.
  * Bogers and Petras (2017) Toine Bogers and Vivien Petras. 2017.  Supporting book search: A comprehensive comparison of tags vs. controlled vocabulary metadata.  _Data and Information Management_ 1, 1 (2017), 17–34.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  InPars: Unsupervised Dataset Generation for Information Retrieval. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2387–2392.
  * Cao et al. (2008) Huanhuan Cao, Daxin Jiang, Jian Pei, Qi He, Zhen Liao, Enhong Chen, and Hang Li. 2008.  Context-aware query suggestion by mining click-through and session data. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 875–883.
  * Chaa et al. (2018) Messaoud Chaa, Omar Nouali, and Patrice Bellot. 2018\.  Combining tags and reviews to improve social book search performance. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, 64–75.
  * Chakraborty et al. (2020) Anirban Chakraborty, Debasis Ganguly, and Owen Conlan. 2020\.  Retrievability based document selection for relevance feedback with automatically generated query variants. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 125–134.
  * Cho et al. (2022) Sukmin Cho, Soyeong Jeong, Wonsuk Yang, and Jong C Park. 2022\.  Query Generation with External Knowledge for Dense Retrieval. In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_. 22–32.
  * Dai et al. (2022) Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2022\.  Promptagator: Few-shot Dense Retrieval From 8 Examples.  _arXiv preprint arXiv:2209.11755_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Gamboa et al. (2010) Luis Fernando Gamboa, Andrés García-Suaza, and Jesús Otero. 2010.  Statistical inference for testing Gini coefficients: An application for Colombia.  _Ensayos sobre Politica Economica_ 28, 62 (2010), 226–241.
  * Garcia-Gathright et al. (2018) Jean Garcia-Gathright, Brian St. Thomas, Christine Hosey, Zahra Nazari, and Fernando Diaz. 2018\.  Understanding and evaluating user satisfaction with music discovery. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 55–64.
  * Gastwirth (1972) Joseph L Gastwirth. 1972\.  The estimation of the Lorenz curve and Gini index.  _The review of economics and statistics_ (1972), 306–316.
  * Greene and Cunningham (2013) Derek Greene and Pádraig Cunningham. 2013.  Discovering latent patterns from the analysis of user-curated movie lists.  _arXiv preprint arXiv:1308.5125_ (2013).
  * Han et al. (2019) Fred X Han, Di Niu, Kunfeng Lai, Weidong Guo, Yancheng He, and Yu Xu. 2019\.  Inferring search queries from web documents via a graph-augmented sequence to attention network. In _The World Wide Web Conference_. 2792–2798.
  * Hosey et al. (2019) Christine Hosey, Lara Vujović, Brian St. Thomas, Jean Garcia-Gathright, and Jennifer Thom. 2019\.  Just give me what I want: How people use and evaluate music search. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Jones et al. (2021) Rosie Jones, Ben Carterette, Ann Clifton, Maria Eskevich, Gareth JF Jones, Jussi Karlgren, Aasish Pappu, Sravana Reddy, and Yongze Yu. 2021.  TREC 2020 podcasts track overview.  _arXiv preprint arXiv:2103.15953_ (2021).
  * Kim et al. (2020) Jaehun Kim, Julián Urbano, Cynthia Liem, and Alan Hanjalic. 2020.  One deep music representation to rule them all? A comparative analysis of different representation learning strategies.  _Neural Computing and Applications_ 32, 4 (2020), 1067–1093.
  * Koolen et al. (2016) Marijn Koolen, Toine Bogers, Maria Gäde, Mark Hall, Iris Hendrickx, Hugo Huurdeman, Jaap Kamps, Mette Skov, Suzan Verberne, and David Walsh. 2016\.  Overview of the CLEF 2016 social book search lab. In _International conference of the cross-language evaluation forum for European languages_. Springer, 351–370.
  * Kumar et al. (2011) Ravi Kumar, Silvio Lattanzi, and Prabhakar Raghavan. 2011\.  An algorithmic treatment of strong queries. In _Proceedings of the fourth ACM international conference on Web search and data mining_. 775–784.
  * Laplante (2008) Audrey Laplante. 2008\.  Everyday life music information-seeking behaviour of young adults: an exploratory study.  (2008).
  * Li et al. (2019) Ang Li, Jennifer Thom, Praveen Chandar, Christine Hosey, Brian St Thomas, and Jean Garcia-Gathright. 2019.  Search mindsets: Understanding focused and non-focused information seeking in music search. In _The World Wide Web Conference_. 2971–2977.
  * Li et al. (2022) Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, and Asia J Biega. 2022.  Exposing Query Identification for Search Transparency. In _Proceedings of the ACM Web Conference 2022_. 3662–3672.
  * Liang et al. (2020) Davis Liang, Peng Xu, Siamak Shakeri, Cicero Nogueira dos Santos, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020.  Embedding-based zero-shot retrieval through query generation.  _arXiv preprint arXiv:2009.10270_ (2020).
  * Lien et al. (2022) Yen-Chieh Lien, Rongting Zhang, F Maxwell Harper, Vanessa Murdock, and Chia-Jung Lee. 2022\.  Leveraging Customer Reviews for E-commerce Query Generation. In _European Conference on Information Retrieval_. Springer, 190–198.
  * Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021\.  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ 14, 4 (2021), 1–325.
  * Liu et al. (2021) Binsheng Liu, Xiaolu Lu, and J Shane Culpepper. 2021.  Strong natural language query generation.  _Information Retrieval Journal_ 24, 4 (2021), 322–346.
  * Liu et al. (2014) Yidan Liu, Min Xie, and Laks VS Lakshmanan. 2014.  Recommending user generated item lists. In _Proceedings of the 8th ACM Conference on Recommender systems_. 185–192.
  * Ma et al. (2020) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2020.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.  _arXiv preprint arXiv:2004.14503_ (2020).
  * Macdonald and Tonellotto (2020) Craig Macdonald and Nicola Tonellotto. 2020.  Declarative Experimentation inInformation Retrieval using PyTerrier. In _Proceedings of ICTIR 2020_.
  * Mehrotra (2021) Rishabh Mehrotra. 2021\.  Algorithmic Balancing of Familiarity, Similarity, & Discovery in Music Recommendations. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3996–4005.
  * Mei et al. (2008) Qiaozhu Mei, Dengyong Zhou, and Kenneth Church. 2008\.  Query suggestion using hitting time. In _Proceedings of the 17th ACM conference on Information and knowledge management_. 469–478.
  * Nogueira et al. (2019a) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019a.  From doc2query to docTTTTTquery.  _Online preprint_ 6 (2019).
  * Nogueira et al. (2019b) Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b.  Document expansion by query prediction.  _arXiv preprint arXiv:1904.08375_ (2019).
  * Penha et al. (2022) Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022\.  Evaluating the robustness of retrieval pipelines with query variation generators. In _European Conference on Information Retrieval_. Springer, 397–412.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  https://arxiv.org/abs/1908.10084
  * Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR’94_. Springer, 232–241.
  * Roy et al. (2022) Dwaipayan Roy, Zeljko Carevic, and Philipp Mayr. 2022\.  Studying retrievability of publications and datasets in an integrated retrieval system.  _arXiv preprint arXiv:2205.00937_ (2022).
  * Sguerra et al. (2022) Bruno Sguerra, Marion Baranes, Romain Hennequin, and Manuel Moussallam. 2022.  Navigational, Informational or Punk-Rock? An Exploration of Search Intent in the Musical Domain. In _Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization_. 202–211.
  * Shen et al. (2022) Xinyao Shen, Jiangjie Chen, Jiaze Chen, Chun Zeng, and Yanghua Xiao. 2022.  Diversified query generation guided by knowledge graph. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 897–907.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in Neural Information Processing Systems_ 33 (2020), 16857–16867.
  * Stamatelatos et al. (2021) Giorgos Stamatelatos, George Drosatos, Sotirios Gyftopoulos, Helen Briola, and Pavlos S Efraimidis. 2021\.  Point-of-interest lists and their potential in recommendation systems.  _Information Technology & Tourism_ 23, 2 (2021), 209–239.
  * Su et al. (2018) Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018.  User intent, behaviour, and perceived satisfaction in product search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 547–555.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021\.  BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_ (2021).
  * Tomasi et al. (2020) Federico Tomasi, Rishabh Mehrotra, Aasish Pappu, Judith Bütepage, Brian Brost, Hugo Galvão, and Mounia Lalmas. 2020\.  Query Understanding for Surfacing Under-served Music Content. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2765–2772.
  * Ullah and Khusro (2020) Irfan Ullah and Shah Khusro. 2020.  Social book search: the impact of the social web on book retrieval and recommendation.  _Multimedia Tools and Applications_ 79, 11 (2020), 8011–8060.
  * Ullah et al. (2021) Irfan Ullah, Shah Khusro, and Ibrar Ahmad. 2021.  Improving social book search using structure semantics, bibliographic descriptions and social metadata.  _Multimedia Tools and Applications_ 80, 4 (2021), 5131–5172.
  * Wan and McAuley (2018) Mengting Wan and Julian J. McAuley. 2018.  Item recommendation on monotonic behavior chains. In _Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018_ , Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O’Donovan (Eds.). ACM, 86–94.  https://doi.org/10.1145/3240323.3240369
  * Wang et al. (2021) Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021.  Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval.  _arXiv preprint arXiv:2112.07577_ (2021).
  * Wilkie and Azzopardi (2013) Colin Wilkie and Leif Azzopardi. 2013.  Relating retrievability, performance and length. In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_. 937–940.
  * Wilkie and Azzopardi (2014) Colin Wilkie and Leif Azzopardi. 2014.  A retrievability analysis: Exploring the relationship between retrieval bias and retrieval performance. In _Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management_. 81–90.
  * Zhuang et al. (2022) Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022.  Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation.  _arXiv preprint arXiv:2206.10128_ (2022).
  * Zuccon et al. (2016) Guido Zuccon, Joao Palotti, and Allan Hanbury. 2016\.  Query variations and their effect on comparing information retrieval systems. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 691–700.

##  Appendix A Unsupervised Weak Labeling Functions

In this appendix, we define the functions used in WeakLabeling-Un.

###  A.1. Random Terms Selection

Samples words from the entity $e$, given possible $P$ length percentages for
the query with probability $Pr$. def sample_words(e, P, Pr): words =
tokenize(e) p_words_to_sample = np.random.choice(P, 1, p=Pr) n =
int(len(words) * p_words_to_sample) words = np.random.choice(words, n,
replace=False) return words

###  A.2. Query Variation Ordering

Generates a query variation by shuffling two random words from the query. def
qv_ordering(q): words = tokenize(q) idxs = [i for i in range(0, len(words))]
p1, p2 = np.random.choice(idxs, 2, replace=False) words[p1], words[p2] =
words[p2], words[p1] return ” ”.join(words)

###  A.3. Query Variation Misspelling

Generates a query variation by adding a misspelling error with $P$
probabilities of removing and addition. def qv_misspelling(q, P): t =
np.random.choice([”rem”, ”mdf”], 1, p=P) idxs=[i for i in range(len(query))]
l=string.ascii_letters if t == ”rem”: idx_rem = np.random.choice(idxs, 1)[0]
qv = q[0:idx_rem] + q[idx_rem+1:] elif t == ”mdf”: idx_mdf =
np.random.choice(idxs, 1)[0] char_add = np.random.choice(len(l), 1)[0] qv =
q[0:idx_mdf] + l[char_add] + q[idx_mdf+1:] return qv

###  A.4. Query Variation Prefix

Generates a query variation by removing $P$ percentages of the suffix of the
query with probabilities $Pr$. def qv_prefix_query(q, P, Pr): rem =
np.random.choice(P, 1, p=Pr) return q[:int((1-rem)*len(q))]

###  A.5. Query from Free-Text Column by Summarization

Generates a query by summarizing the value of a free-text column (broad-
fields-ft). For our experiments we rely on the pre-trianed summarizer model
snrspeaks/t5-one-line-summary161616https://huggingface.co/snrspeaks/t5-one-
line-summary. from transformers import pipeline def q_summarizer(ft, m): pipe
= pipeline(”text2text-generation”, model = m) return pipe(”summarize:
”.format(ft))[0]

##  Appendix B Bias mitigation for the Click dataset

In this appendix, we investigate if it is possible to mitigate the biases of
the Click data with a simpler approach.

When fine-tuning the Bi-Encoder with Click data in our experiments we do not
employ the same combination of queries and entities twice, even if that pair
is highly popular in the logs. This is already a way of reducing the bias in
the Click dataset. However, there are still many query variations that lead to
the same entities, i.e. queries with different forms but with the same
underlying information need, which are not removed when we get distinct
queries for training. In order to mitigate this bias from the Click data by
removing multiple queries that lead to the same entity, we randomly select
only one of the queries for each entity to train the Bi-Encoder model on.

The result of this experiment is that such a bias mitigation strategy indeed
improves the retrievability of the system: the Gini scores go from 0.856 to
0.803 for Tracks and from 0.763 to 0.713 for Podcasts. However the mitigated
Click data approach still leads to 30% and 5% more retrievability bias than
CtrlQGen, for Tracks and Podcasts respectively.

##  Appendix C Scaling InPars with GPT-3

In this appendix, we test if increasing the model size of the InPars model
using GPT-3 as the language model has a significant effect on the Bi-Encoder
trained with such synthetic queries.

We see from 5 that this is not the case for both datasets, and the highest
R@100 is reached when using the 1.2B GPT-3 model (babbage-001). Similar
results were found in the InPars paper (Bonifacio et al., 2022).

Figure 5. Scaling InPars baseline using GPT-3.

##  Appendix D Overlap of generated queries

In this appendix, we check if the queries generated by CtrlQGen have a
significant overlap with either the log queries (is CtrlQGen just copying
existing queries?) or with the input entity (is CtrlQGen just copying words
from the input entity?).

###  D.1. With log queries from Click

Out of the 10k narrow queries generated by CtrlQGen to test the first
hypothesis (results from Tables 2), there are only 6% and 12% exact matches
with set of queries from the logs (Click) for the Tracks and Podcasts datasets
respectively. For the second hypothesis, out of the 376k broad queries
generated, there are only 2% and 1% are exact matches with the set of queries
from the logs. This shows the diversity of the generated queries from
CtrlQGen, and that it is not just copying input queries from the log.

###  D.2. With the input entity

Out of the 10k narrow queries generated by CtrlQGen to test the first
hypothesis, 25% and 48% are not a subset of the serialized entity for Tracks
and Podcasts datasets respectively. For the second hypothesis, out of the 376k
broad queries generated, a total of 70% of queries are not subsets of the
serialized entity for both datasets. This shows that while for narrow queries
substrings of the entity are the majority of the cases when generating broad
queries this is not the case. Also, this indicates that for both cases the
model is not always selecting parts of the input as the query.

##  Appendix E Dataset details

For the Books dataset, we take into account the top two most-voted reviews and
use the first 50 tokens. For the Tracks dataset, we use the first lyric line
and the most frequent lyric line and employ a maximum of 25 descriptors. For
the Podcasts dataset, we use the first 50 tokens of the description and of the
transcript. For the Books and Tracks datasets we use a maximum of 25
playlists.