# Improving Retrieval in Theme-specific Applications
using a Corpus Topical Taxonomy

SeongKu Kang  University of Illinois at Urbana-ChampaignILUSA
<EMAIL_ADDRESS>,  Shivam Agarwal  University of Illinois at Urbana-
ChampaignILUSA<EMAIL_ADDRESS>,  Bowen Jin  University of Illinois at
Urbana-ChampaignILUSA<EMAIL_ADDRESS>,  Dongha Lee  Yonsei
UniversitySeoulRepublic of Korea<EMAIL_ADDRESS>,  Hwanjo Yu  Pohang
University of
Science and TechnologyPohangRepublic of Korea<EMAIL_ADDRESS>and
Jiawei Han  University of Illinois at Urbana-ChampaignILUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Document retrieval has greatly benefited from the advancements of large-scale
pre-trained language models (PLMs). However, their effectiveness is often
limited in theme-specific applications for specialized areas or industries,
due to unique terminologies, incomplete contexts of user queries, and
specialized search intents. To capture the theme-specific information and
improve retrieval, we propose to use a corpus topical taxonomy, which outlines
the latent topic structure of the corpus while reflecting user-interested
aspects. We introduce ToTER (Topical Taxonomy Enhanced Retrieval) framework,
which identifies the central topics of queries and documents with the guidance
of the taxonomy, and exploits their topical relatedness to supplement missing
contexts. As a plug-and-play framework, ToTER can be flexibly employed to
enhance various PLM-based retrievers. Through extensive quantitative,
ablative, and exploratory experiments on two real-world datasets, we ascertain
the benefits of using topical taxonomy for retrieval in theme-specific
applications and demonstrate the effectiveness of ToTER.

Document retrieval; Topical taxonomy; Theme-specific application

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
ACM Web Conference 2024; May 13–17, 2024; Singapore, Singapore††booktitle:
Proceedings of the ACM Web Conference 2024 (WWW ’24), May 13–17, 2024,
Singapore, Singapore††doi: 10.1145/3589334.3645512††isbn:
979-8-4007-0171-9/24/05††ccs: Information systems Information retrieval††ccs:
Information systems Specialized information retrieval††ccs: Information
systems Document topic models

##  1\. Introduction

Pre-trained language models (PLMs) have greatly improved document retrieval
(Izacard et al., 2021; Xiong et al., 2021; Khattab and Zaharia, 2020;
Karpukhin et al., 2020). The PLM-based retrieval models are first pre-trained
on the massive textual corpora to grasp language understanding. Subsequently,
they are fine-tuned using vast datasets of annotated query-document pairs,
which enables the models to capture their semantic similarities. While
successful in general domains like web search which consist of a broad user
base, they are often limited in specialized applications with specific themes.

Theme-specific applications are specialized areas or industries where
retrieval tasks are focused on a specific theme (e.g., academic paper search,
product search in e-commerce). Retrieval in theme-specific applications poses
three challenges spanning specialized terminology and niche content (C1),
limited contexts of user query (C2), and specialized user interests and search
intents (C3).

C1: Theme-specific domains often have specialized terminologies, which are not
frequently included in the general text corpus. For example, Table 1(a) shows
that an academic paper includes many technical terms specific to certain
research fields, such as “proof of retrievability” and “cryptographic proof”.
PLM-based retrievers trained on general text corpora often lack an inherent
understanding of domain-specific specialized and niche terminologies (Dong et
al., 2022).

C2: Users familiar with the domain often omit contexts they believe are
naturally implied in their query. For example, in product search, users enter
a query such as “RTX 3090” without adding contexts such as “graphics cards”.
Table 1(a) shows queries from domain experts may skip over general contexts
such as “cryptography” or “computer security”. Omitted terms hinder the
model’s ability to fully comprehend the query, leading to imprecise retrieval
outcomes. Inferring missing contexts is more challenging in theme-specific
applications as it often requires domain-specific knowledge.

Table 1. Examples of retrieval in theme-specific applications. We use
Contriever-MS (retriever) and MiniLM-L-12 (reranker). Contents closely related
to the query are denoted in bold. Details of topic class and core phrase
discovery are provided in section 4.

2.5 (a) Academic domain | (b) Product domain |
---|---|---
Query |  Provable data possession at untrusted stores |  Query |  #1 black natural hair dye without ammonia or peroxide
Document A (label: relevant rank: top-173) |  Pors: proofs of retrievability for large files. In this paper, we define and explore proofs of retrievability (PORs). … A POR may be viewed as a kind of cryptographic proof of knowledge (POK). … We view PORs as an important tool for semi-trusted online archives. Existing cryptographic techniques help users ensure the privacy and integrity of files they retrieve. … |  Document A  (label: relevant rank: top-70) |  ONC NATURALCOLORS (1N Black) 4 fl. oz. (120 mL). Healthier permanent hair dye with certified organic ingredients, ammonia free, vegan friendly, 100 gray coverage. … Cruelty-free and vegan. It is time to make the clean choice.
|  Document B (label: irrelevant rank: top-11) |  Roux Fanci-full Rinse 16 Hidden Honey. Tones and enhances gray and blonde hair. Rinses in and shampoos out. No ammonia or peroxide. … 15 applications per bottle, temporary hair color, 15 ounce bottle.
2.5 ToTER rank:  top-10 |  Topic classes: cryptography, trusted computing, digital content, computer network, computer security, computer science |  ToTER rank:  top-5 (Doc.A) top-32 (Doc.B) |  Topic classes: hair color, hair coloring products, hair care, beauty & personal care
Core phrases: encryption, access control, security, key, server |  Core phrases: dye, permanent, lasting, permanent hair color, ammonia free
2.5 |  |  |

C3: Users in theme-specific applications have more specialized interests and
intents compared to general web searches. For example, researchers may want to
find papers within a specific field of study to discern a particular research
trajectory. In product search, users often filter results based on specific
product attributes. For example, Table 1(b) shows that both documents are
somewhat relevant to the query as both of them are about ammonia-free hair
color products. However, the query targets hair dye with lasting effects,
instead of hair rinse with temporary effects. These specialized search intents
are not effectively captured by models trained on general corpora.

Accumulating ample labeled data can mitigate these challenges to some extent.
However, the creation of such datasets in theme-specific applications is
particularly challenging due to the need for domain expertise (e.g., academic
domain) and the proprietary nature of user logs in specialized applications
(e.g., e-commerce) (Chaudhary et al., 2023; Li et al., 2023). As a result,
PLM-based retrieval models often struggle to accurately capture relevance in
theme-specific applications (Thakur et al., 2021).

To improve retrieval without relying on labeled data, we propose to use a
corpus topical taxonomy (Huang et al., 2020; Lee et al., 2022a; Meng et al.,
2020; Shang et al., 2020; Zhang et al., 2018), which has been extensively
studied for organizing topics in a corpus. A corpus topical taxonomy outlines
the latent topic hierarchy within the corpus as a tree structure, where each
node is a topic class represented by a cluster of semantically coherent terms
describing the topic, as shown in Figure 1. Recent taxonomy construction
studies (Huang et al., 2020; Lee et al., 2022a; Arous et al., 2023) have
effectively reflected user-interested aspects, drawing from a foundational
seed taxonomy rooted in human knowledge of the application (e.g., fields of
study from Mircosoft Academic (Shen et al., 2018)). The constructed taxonomy
can be subsequently employed to provide additional clues to link queries and
documents by discerning their topical relatedness and supplementing the
missing contexts.

We propose Topical Taxonomy Enhanced Retrieval (ToTER) framework, which
systematically leverages the corpus taxonomy to complement the semantic
matching of PLM-based retrieval. The taxonomy provides a high-level topic
hierarchy of the entire corpus. To harness this corpus-level knowledge for
retrieval, we first link it to individual documents. Specifically, ToTER first
conducts topic class relevance learning to discern the relevance of each
document to each topic class node in the taxonomy. We formulate this step as
an unsupervised multi-label classification problem without document-topic
labels. ToTER introduces a new silver label generation strategy along with a
new collective distillation process to produce rich and reliable signals. This
class relevance learning allows ToTER to effectively identify central subjects
of a given text under the guidance of the topical taxonomy reflecting user
interests.

Based on the identified topic class relevance, ToTER leverages the topical
relatedness of a query and documents to complement the semantic matching by
PLM-based retrievers. In Table 1(a), we see that ToTER can improve retrieval
by identifying common topic classes like “cryptography” and “computer
security” for both query and document, given the presence of terms frequently
used for these topic classes (C1). Furthermore, ToTER combines the topical
relatedness with more fine-grained phrase knowledge for each topic class,
helping to distinguish documents having similar topics. In Table 1(b), ToTER
identifies and utilizes core topical phrases such as “dye”, “lasting”, and
“permanent hair color” to enrich the query, enabling more accurate finding of
relevant documents (C2). This entire process is built upon the topical
taxonomy reflecting user-interested aspects (C3). Formally, ToTER introduces
three strategies to complement the PLM-based retrieval: (1) search space
adjustment, (2) class relevance matching, and (3) query enrichment by core
phrases. Our contributions are summarized as follows:

  * •

We present a systematic approach to incorporate topical taxonomy into
retrieval in theme-specific applications, which remains unexplored in the
previous literature.

  * •

We propose ToTER that deliberately discerns and utilizes topical relatedness
of queries and documents. As a plug-and-play framework, it can be integrated
with various PLM-based models.

  * •

We validate the effectiveness of ToTER by extensive experiments. ToTER
consistently improves retrieval accuracy in scenarios with both no labeled
data and limited labels.

##  2\. Problem Formulation

###  2.1. Concept Definition

PLM-based multi-stage retrieval. Most PLM-based retrieval systems leverage
multi-stage retrieve-then-rerank pipeline (Ma et al., 2020; Dong et al., 2022;
Zhang et al., 2022a). Specifically, given a query $q$, a retriever retrieves a
set of candidate documents $\mathcal{D}_{q}$ from a large corpus
$\mathcal{D}$, where $|\mathcal{D}_{q}|\ll|\mathcal{D}|$. Following the first-
stage retrieval, a reranker computes a more fine-grained relevance for each
candidate document, and generates the final ranked list by reordering them.

The first-stage retriever typically adopts a dual-encoder architecture, where
query and documents are encoded separately and the relevance is measured by
the similarity between their embeddings.

(1) $s_{de}(q,d)=sim(f_{\theta}(q),f_{\theta^{\prime}}(d)),$

where $f_{\theta}$ and $f_{\theta^{\prime}}$ are the query and document
encoders, and $sim(\cdot,\cdot)$ is the similarity function such as the cosine
similarity. The document embeddings are pre-computed and efficiently retrieved
via approximate nearest-neighbor (ANN) search techniques (Johnson et al.,
2019).

The second-stage reranker mostly adopts a cross-encoder architecture which
takes the concatenation of a query and a document as input and assesses its
relevance score.

(2) $s_{ce}(q,d)=f_{\phi}(q,d),$

where $f_{\phi}$ denotes the reranker. By fully exploring the interactions
between the query and document, it generates more accurate relevance scores
compared to the dual-encoder (Zhang et al., 2022a).

Topical taxonomy. A corpus topical taxonomy
$\mathcal{T}=(\mathcal{C},\mathcal{R})$ represents a tree structure that
outlines the latent topic hierarchy within the target corpus. Each node
$c_{j}\in\mathcal{C}$ corresponds to a topic class which is described by a
coherent cluster of terms111Each term is regarded as a phrase composed of one
or multiple word tokens, so the terms “phrase” and “term” are used
interchangeably in this paper. describing the topic, denoted by $P_{j}$. The
most salient term (i.e., center term) is utilized as the class name. Each edge
($\in\mathcal{R}$) indicates a theme-specific relationship between a parent
and child node, such as “is a subfield of” or “is a type of”. Figure 1 shows
an example of topical taxonomy.

To construct the topical structure reflecting theme-specific user interests,
taxonomies are built upon a foundational seed taxonomy rooted in human
knowledge of the application (Huang et al., 2020; Lee et al., 2022a; Arous et
al., 2023). An example is the fields of study in the academic domain (Shen et
al., 2018), which embodies researchers’ inclination to organize academic
concepts and studies. Based on this application knowledge, recent methods
(Huang et al., 2020; Lee et al., 2022a; Zhang et al., 2023; Lee et al., 2022b)
have effectively generated taxonomy having remarkable term coherency, topic
coverage, and user-interest alignment.

###  2.2. Problem Definition

We focus on retrieval in theme-specific applications, which are specialized
areas or industries where retrieval tasks are centered around specific themes.
These applications differ from general web search in terms of (1) domain-
specialized terminology and niche content, (2) expert users familiar with the
domain, and (3) specialized search intents. Given a target corpus
$\mathcal{D}$ and its topical taxonomy $\mathcal{T}$222The topical taxonomy
can be obtained using any off-the-shelf taxonomy completion technique. In this
work, we use the recently proposed method (Lee et al., 2022a)., our goal is to
develop a systematic framework that exploits the topic hierarchy knowledge to
improve the existing PLM-based multi-stage retrieval. We focus on scenarios
where labeled data from the target corpus is unavailable. It is worth noting
that we pursue a plug-and-play solution that can be flexibly integrated with
various existing retriever and reranker models.

Figure 1. The overview of Topical Taxonomy-Enhanced Retrieval (ToTER)
framework.

##  3\. Related Work

Topical taxonomy completion. Topical taxonomy represents the latent topic
hierarchy of document collections, providing valuable knowledge of contents in
many applications. Early methods build a corpus topic taxonomy from scratch by
extracting discriminative term clusters from the corpus in a hierarchical
fashion (Zhang et al., 2018). To generate user-interested topic structure,
recent approaches (Shang et al., 2020; Lee et al., 2022a; Huang et al., 2020;
Lee et al., 2022b) have started with a seed taxonomy rooted in human knowledge
of the application and expanded it by discovering novel topics from the target
corpus. Specifically, (Huang et al., 2020) trains classifiers to capture user-
interested relations from parent-child topic pairs, (Lee et al., 2022a)
recursively clusters phrases to identify new subtopics based on the known
topic relations. Very recently, (Lee et al., 2022b) generates topic-
conditioned terms by leveraging hierarchical relations from the seed taxonomy.

PLM-based retrieval models.. PLM-based retrieval models have advanced in both
training and encoding strategies. In terms of training strategy, starting from
in-batch and hard negatives by BM25 (Karpukhin et al., 2020; Luan et al.,
2021), advanced hard negative mining by dynamic sampling (Zhan et al., 2021)
and denoising using a cross-encoder (Qu et al., 2021) have been studied. Many
works have focused on pre-training with unsupervised contrastive learning
(Izacard et al., 2021; Gao et al., 2021; Gao and Callan, 2022), knowledge
graph (Kim et al., 2021; Dong et al., 2022), and synthetic data (Ma et al.,
2020; Chaudhary et al., 2023; Dai et al., 2023) to improve the capability of
models. Recent studies (Ren et al., 2021; Zhang et al., 2022a) also show that
joint training of the retriever and reranker can further improve their
effectiveness. In terms of encoding strategy, single-vector representation
models (Xiong et al., 2021) encode a given text as a single vector, and multi-
vector representation models (Khattab and Zaharia, 2020) use multiple vectors
to improve expressiveness. Recent sparse representation models (Formal et al.,
2021, 2022) use sparse lexical representations based on the logits of the
masked language model layer of PLMs, which enables a natural query and
document expansion.

Despite their effectiveness, they require fine-tuning with massive labeled
data to be adapted to the new domain corpus. As a plug-and-play framework,
ToTER complements the above approaches using a topical taxonomy without
resorting to the labeled data.

Retrieval with auxiliary corpus knowledge. These techniques aim to improve
retrieval by exploiting knowledge of the target corpus. One notable approach
is pseudo-relevance feedback (PRF), which utilizes the top-ranked results from
an initial retrieval to enhance the semantic matching process. The existing
methods have exploited key terms (Wang et al., 2021; Kim et al., 2022), text
segments (Zheng et al., 2020), and documents (Yu et al., 2021) from top-ranked
results as an additional context to complement the query. Recently, (Mao et
al., 2021; Mackie et al., 2023) have directly utilized knowledge stored in the
PLMs for query expansion. Despite their effectiveness in filling missing
contexts, their effectiveness is often limited in theme-specific applications
due to the suboptimal initial retrieval quality and the need for domain-
specific knowledge.

Another approach leverages inter-document similarity via a corpus graph whose
nodes are documents and edges connect most similar documents. Under the
assumption that similar documents tend to be relevant to the same query,
(MacAvaney et al., 2022) adapts the candidate document set for reranker using
the nearest neighbors in the graph. (Kulkarni et al., 2023) first uses lexical
retrieval to obtain seed documents and uses the graph to gradually expand the
search space for retrievers. Lastly, there have been a few attempts to use
topic information for retrieval. (Wei and Croft, 2006; Jian et al., 2016)
combine LDA (Blei et al., 2003) with statistic-based retrieval, and
(Hofstätter et al., 2021; Wang and Zuccon, 2023) use topics for a balanced
batch construction. (Li et al., 2021) incorporates topic information into a
word embedding-based model. However, there has been no attempt to exploit the
high-quality corpus taxonomy.

It is worth noting that there exist a few attempts to use external knowledge
(e.g., knowledge base) (Dong et al., 2022). We focus on exploiting knowledge
of the target corpus, and ToTER can be combined with the external knowledge-
based models as well.

##  4\. METHODOLOGY

We present Topical Taxonomy-Enhanced Retrieval (ToTER) framework. We first
explain how ToTER bridges the given taxonomy with the target corpus in the
training phase (section 4.1), then present how ToTER enhances PLM-based
retrieval in the inference phase (section 4.2). The overview of ToTER is
presented in Figure 1.

###  4.1. Topic Class Relevance Learning

The taxonomy reveals the latent structure of the whole corpus. To exploit it
for retrieval, we first connect the corpus-level knowledge to individual
documents. We formulate this step as an unsupervised multi-label
classification, assessing the relevance of each document to each topic class
without document-topic labels. We first introduce our silver label generation
strategy (section 4.1.1). Then, we propose a new training strategy to produce
rich and reliable training signals (section 4.1.2). Lastly, we summarize the
overall training process (section 4.1.3).

####  4.1.1. Taxonomy-guided silver label generation

As the first step, we seek to identify a small candidate set of relevant
classes for each document, which will be our silver labels for training.
Utilizing the hierarchical structure of topic classes, we introduce a top-down
approach that recursively assigns documents to the child node with the highest
similarity, gradually narrowing down the topic. Given a document, we start
from the “root node” and compute its similarity to each child node. The
document is then assigned to the child node with the highest similarity.333
Although a document can cover multiple topics, we assign it to the most
similar child node to generate reliable silver labels. Indeed, we observe
increasing the assignment number easily leads to degraded training efficacy.
This assignment process recurs until it reaches leaf nodes. Once every
document has been assigned, we apply a filtering step to retain only reliable
labels for each document.

Document-class similarity computation. For each document $d$, we compute its
similarity with a child node $c_{j}$ by considering all phrases related to
$c_{j}$. Let $P^{\mathcal{T}}_{j}$ denote the union of all phrases from the
subtree having $c_{j}$ as a root node. The similarity is computed by
considering both lexical aspect ($\operatorname{sim}_{L}$) based on statistics
and semantic aspect ($\operatorname{sim}_{S}$) based on PLMs, which are
defined as:

(3) $\displaystyle{sim}_{L}(d,c_{j})$
$\displaystyle=\frac{1}{|P^{\mathcal{T}}_{j}|}\sum_{p\in
P^{\mathcal{T}}_{j}}\operatorname{tf}(p,d)\cdot\operatorname{idf}(p)$
$\displaystyle{sim}_{S}(d,c_{j})$
$\displaystyle=\frac{1}{|P^{\mathcal{T}}_{j}|}\sum_{p\in
P^{\mathcal{T}}_{j}}\operatorname{cos}(\mathbf{h}_{p},\mathbf{h}_{d}),$

where $\mathbf{h}_{p}$ and $\mathbf{h}_{d}$ denote representations from PLM
for a phrase $p$ and a document $d$. The lexical and semantic similarity can
reveal complementary aspects; lexical matching has strengths in handling
domain-specific terminologies that rarely exist in the general corpus, and
semantic matching excels in capturing broader contextual meanings while
flexibly handling non-exact matching terms.

To jointly consider the two aspects, we adopt the ensemble score based on the
reciprocal rank (Zhang et al., 2023, 2022b). By ranking all child nodes in
descending order of ${sim}_{L}$ and ${sim}_{S}$, each node will have two rank
positions ${rank}_{L}$ and ${rank}_{S}$, respectively. The overall similarity
is:

(4)
$\displaystyle{sim}_{O}(d,c_{j})=\left(\frac{1}{2}\left(\frac{1}{{rank}_{L}(c_{j})}\right)^{\rho}+\frac{1}{2}\left(\frac{1}{{rank}_{S}(c_{j})}\right)^{\rho}\right)^{1/{\rho}},$

where $0<\rho\leq 1$ is a constant. We set $\rho=0.1$. In this ensemble score,
a high position in one ranking can largely compensate for a lower position in
another, thus allowing for the joint consideration of two distinct aspects.

Filtering step. After the class assignment for all documents, we apply a
filtering step to only retain assignments with high similarity. For each class
node, we keep documents whose similarity exceeds the median similarity of all
documents assigned to the class. If a document is filtered out from a certain
node, the document is also removed from all its child nodes, ensuring
hierarchical consistency.444We apply the filtering to nodes at the second or
deeper levels to ensure every document has at least one class assignment (the
root node has level 0). Finally, for each document $d$, we obtain silver
labels $\mathbf{y}^{s}_{d}\in\\{0,1\\}^{|\mathcal{C}|}$, where $y^{s}_{dj}=1$
if $c_{j}$ is assigned to $d$, otherwise $0$.

####  4.1.2. Class relevance learning

Based on the obtained silver labels, we train a class relevance estimator that
predicts relevance between a document and topic classes. For effective
training, we propose a new collective topic knowledge distillation strategy
designed to complement incomplete silver labels.

Class relevance estimator. As developing a novel architecture is not the focus
of this paper, we employ the existing methods to encode documents and class
nodes. For the text encoder, we use BERT (Devlin et al., 2019) with mean
pooling to obtain $\mathbf{h}_{d}$ for each document $d$. For the topic class
encoder, we adopt graph convolutional networks (GCNs) (Kipf and Welling, 2016)
to incorporate both semantic and structural information. For each class node
$c_{j}$, we first obtain its ego graph that includes its $L$-hop neighboring
nodes and apply GCN layers to propagate node features over the taxonomy
structure. Each node feature is initialized by the BERT representation of its
class name. After stacking $L$ GCN layers, we use the representation of the
ego node, denoted as $\mathbf{c}_{j}$, as the final class representation.

Then, we calculate the topic class-document relevance by bilinear interaction
between their representations, i.e.,
$\hat{y}_{dj}=\sigma(\mathbf{c}^{\top}_{j}\mathbf{M}\mathbf{h}_{d})$, where
$\mathbf{M}$ is a trainable interaction matrix and $\sigma(\cdot)$ is the
sigmoid function. The estimator is trained by the binary cross-entropy loss:

(5)
$\displaystyle\min_{\mathbf{M},\,\theta_{GCN}}\mathcal{L}=-\sum_{d\in\mathcal{D}}\sum_{c_{j}\in\mathcal{C}}y_{dj}\log\hat{y}_{dj}+(1-y_{dj})\log(1-\hat{y}_{dj}).$

In the early stages of training, we use the silver labels
$\mathbf{y}_{d}=\mathbf{y}^{s}_{d}$. As training progresses, we exploit
collective topic labels $\mathbf{y}_{d}=\mathbf{y}^{c}_{d}$ obtained from
similar documents, which we will introduce below.

Collective topic knowledge distillation (CKD). As the silver labels
incompletely reveal true relevance classes, relying solely on them leads to
suboptimal estimation. As a solution, we propose CKD, designed to complement
the incomplete labels. Our core idea is that the topic distribution of a
document can be inferred from semantically similar documents. Specifically,
(1) Using each document $d$ as a query, we retrieve a small subset of
semantically similar documents $\mathcal{D}_{d}$ from the corpus. To
accurately retrieve $\mathcal{D}_{d}$, we consider both semantic similarity
(via a dual-encoder) and topical relatedness (via our estimator). This will be
explained in Sec 4.2.2. (2) We compute the class relevance distributions for
each retrieved document, i.e., $\\{\mathbf{\hat{y}}_{d^{\prime}}\mid
d^{\prime}\in\mathcal{D}_{d}\\}$. (3) By averaging the predicted
distributions, we generate collective relevance labels
$\mathbf{y}^{c}_{d}\in(0,1)^{|C|}$.

Unlike $\mathbf{y}^{s}_{d}$ which consists of binary values,
$\mathbf{y}^{c}_{d}$ reveals the soft probability of a document’s relevance to
each class. Notably, $\mathbf{y}^{c}_{d}$ reveals topic classes that are
highly pertinent to documents similar to $d$, providing rich supervision not
included from $\mathbf{y}^{s}_{d}$. Moreover, this collective knowledge
distills more stable and reliable signals than using individual predictions
for pseudo-labeling, as done in conventional self-training (Xie et al., 2016;
Iscen et al., 2019). It is worth noting that $\mathbf{y}^{c}_{d}$ gets refined
during the training. That is, the topic estimator is improved with collective
knowledge, which again results in more accurate discovery of similar documents
and their topic distributions.

Input : A target corpus $\mathcal{D}$, a corpus topical taxonomy
$\mathcal{T}$,

a retriever $f$, a update period $t$

Output : Trained class relevance estimator $g$

1 Randomly initialize the training parameters of $g$

2 Generate silver labels $\mathbf{y}^{s}_{d}$ for all $d\in\mathcal{D}$ //
section 4.1.1

3 Warm-up the estimator $g$ using only $\mathbf{y}^{s}_{d}$ // Eq.5

/* Collective knowledge distillation */

4 for  _$e=1,...,\text{ }epoch_{max}$ _ do

5 foreach  _$d\in\mathcal{D}$_ do

6 if  _$e\,\%\,t==0$_ then

7 Retrieve $\mathcal{D}_{d}$ using $f$ with CRM // section 4.2.2

8 Compute class relevance as $\\{\mathbf{\hat{y}}_{d^{\prime}}\mid
d^{\prime}\in\mathcal{D}_{d}\\}$ using $g$

9 Obtain collective labels
$\mathbf{y}^{c}_{d}=\operatorname{AVG}(\\{\mathbf{\hat{y}}_{d^{\prime}}\mid
d^{\prime}\in\mathcal{D}_{d}\\})$

10

11 Train the estimator $g$ using $\mathbf{y}^{c}_{d}$ // Eq.5

12

13

Algorithm 1 Training algorithm of ToTER.

####  4.1.3. Training algorithm of ToTER

The training process of ToTER is provided in Algorithm 1. We generate the
silver labels $\mathbf{y}^{s}_{d}$ and use them to warm up $g$. After the
warm-up, we train $g$ with the collective knowledge distillation. In specific,
for each document $d$, we retrieve a set of similar documents
$\mathcal{D}_{d}$ with CRM, and generate the collective labels
$\mathbf{y}^{c}_{d}$ by averaging their class relevance distributions. In
practice, we update the collective topic labels $\mathbf{y}^{c}_{d}$ every $t$
epochs, which makes the training process more efficient and robust. In this
work, we set $|\mathcal{D}_{d}|=10$ and $t=25$.

###  4.2. Topical Taxonomy-Enhanced Retrieval

We present how ToTER improves PLM-based retrieval at the inference phase.
ToTER consists of three strategies to complement the existing retrieve-then-
rerank pipeline. Each strategy is designed to gradually focus on fine-grained
ranking, as shown in Figure 1.

Class relevance estimation. After training, for every document $d$ in the
corpus, we compute its topic class relevance as $\hat{\mathbf{y}}_{d}$.
Considering each document only covers a small subset of topics within the
corpus, we focus on classes with high relevance. To indicate these relevant
classes, we introduce a binary indicator vector
$\hat{\mathbf{b}}_{d}\in\\{0,1\\}^{|\mathcal{C}|}$, where $\hat{b}_{dj}=1$
denotes that $d$ has a certain degree of relevance to $c_{j}$, otherwise
$\hat{b}_{dj}=0$. We recursively retain the top $m\%$ classes for each level
of taxonomy by setting the corresponding elements of $\hat{\mathbf{b}}_{d}$ as
$1$. If a class is not retained at a higher level, all its child classes are
not retained as well, ensuring hierarchical consistency. We set $m=10$. At
test time, for a given query $q$, we obtain $\hat{\mathbf{y}}_{q}$ and
$\hat{\mathbf{b}}_{q}$ in the same way.

####  4.2.1. Search space adjustment (SSA) to reduce search space.

The topic class relevance reveals the central subjects of each document,
providing a snapshot of its main focus. Before applying the PLM-based
retrievers, we seek to filter out a large number of irrelevant documents
having little topic class overlap with the query. This step can benefit
subsequent retrieval by reducing the search space while preserving topically
relevant documents that may otherwise be overlooked by PLM-based retrievers.
For search space reduction, lexical models (e.g., BM25) are mostly considered
due to their high efficiency (Kulkarni et al., 2023; Thakur et al., 2021). We
expect topic-based SSA can have strengths in identifying relevant documents,
compared to using lexical similarity based on word overlap.

As topics are discrete categories, we can efficiently compute the topic
overlap using the binary vectors. In specific, we compute the degree of topic
overlap between the query and each document using bitwise operations:
$\operatorname{Popcount}(\operatorname{AND}(\hat{\mathbf{b}}_{q},\hat{\mathbf{b}}_{d}))$.
Then, we filter out documents with low degrees of overlap, obtaining the
reduced search space $\mathcal{D}^{SSA}_{q}$. The size of this space can be
determined empirically. We continue subsequent retrieval on
$\mathcal{D}^{SSA}_{q}$ instead of $\mathcal{D}$.

Remarks. Compared to using real-valued vectors $\hat{\mathbf{y}}_{*}$, the
proposed SSA is more efficient as it uses bitwise operations of binary
vectors, largely reducing the need for floating-point operations. It can be
further accelerated using multi-index hashing for binary codes (Norouzi et
al., 2013). As search speed acceleration is a distinct research topic, we
focus on the accuracy aspect in this work.

####  4.2.2. Class relevance matching (CRM) for retriever

The first-stage retriever aims to find a set of candidate documents
$\mathcal{D}_{q}$. In this step, we exploit topical relatedness of the query
and document, which is the similarity between distributions of the relevant
topic classes. Topical relatedness focuses on the relevance of the central
subjects of the input texts identified using the class estimator. This can
help to handle lexical mismatches and fill in missing contexts, providing a
complementary aspect to semantic similarity. Formally, we retrieve
$\mathcal{D}_{q}$ based on $s(q,d)$, considering both semantic similarity from
dual-encoder $s_{de}(q,d)$ and topical relatedness $s_{CRM}(q,d)$:

(6) $\displaystyle s(q,d)=combine(s_{de}(q,d),\,s_{CRM}(q,d)),$ $\displaystyle
s_{CRM}(q,d)=sim(\hat{\mathbf{y}}_{q}\odot\hat{\mathbf{b}}_{q},\,\hat{\mathbf{y}}_{d}\odot\hat{\mathbf{b}}_{d}).$

We obtain the relevant class distribution using element-wise multiplication,
denoted as $\odot$, with the binary vector. Then, we compute the similarity
between the query and document distributions using the $sim(\cdot,\cdot)$
function, where we use inner-product. $combine(\cdot,\cdot)$ denotes a
function to consolidate two scores, and we adopt a simple addition with
rescaling via z-score normalization.555A hyperparameter can be also used to
balance two scores of varying magnitudes.  We also explored reflecting the
granularity of topics by exclusively focusing on broad or specific topic
classes in CRM. However, considering the overall topic distribution,
encompassing both broad and specific topics, proved most effective. Please
refer to section 5.3.1 for a detailed study.

####  4.2.3. Query enrichment by core phrases (QEP) for reranker.

In this last stage, a reranker reorders $\mathcal{D}_{q}$ based on their fine-
grained relevance to the query $q$. Since $\mathcal{D}_{q}$ already have
similar relevant classes via CRM, in this step, we delve deeper into each
topic by focusing on class-related phrases. As discussed earlier, users
familiar with a domain often omit contexts in their queries, which makes it
difficult to find accurate relevance. To address this, we use phrase-level
knowledge to enrich queries.

A remaining question is how to identify phrases to complement a given query.
QEP is built upon the relevance model philosophy (Lavrenko and Croft, 2017),
which assumes that both a query and its relevant documents are generated from
a shared underlying relevance model. Although the true relevance model behind
the query is unknown, it can be inferred from the most relevant documents
obtained from retrieval (Yu et al., 2021; Wang et al., 2021; Zheng et al.,
2020). Based on this idea, we identify core phrases to enrich the query using
both the topic class knowledge and top-ranked retrieved documents. From the
set of relevant class phrases $\\{P_{j}|\hat{b}_{qj}=1\\}$, we collect the
top-$k$ core phrases $P^{q}$ that most frequently appear in top-ranked
documents.666The phrases related to each class ($P_{j}$) are provided in the
topical taxonomy (section 2.1). The frequencies of phrases for each document
are pre-computed. Then, we enrich the original query with $P^{q}$ and use the
enriched query for inference:

(7) $s_{ce}(q,d)=f_{\phi}(q^{QEP},d),\quad q^{QEP}=[q;\texttt{TMPLT}(P^{q})],$

where TMPLT denotes the template of the hard prompt. In this work, we use
$\texttt{TMPLT}(P^{q})=``\textit{, relevant topics: }\\{P^{q}\\}$”. We also
tried using just the topic class names (i.e., the most salient phrase) for
query enrichment. However, we found that they are often too coarse-grained,
thus bringing limited information for fine-grained rankings.

##  5\. Experiments

We experiment to answer the following research questions:

  1. RQ1

How does each strategy of ToTER affect retrieval accuracy?

  2. RQ2

How does ToTER compare with other techniques that use auxiliary corpus
knowledge in terms of retrieval accuracy?

  3. RQ3

Is ToTER compatible with a variety of PLM-based retrieval models used for each
retrieval stage?

  4. RQ4

How does each proposed component affect ToTER?

  5. RQ5

How does the labeled data affect the effectiveness of ToTER?

  6. RQ6

How sensitive is ToTER to the taxonomy quality?

###  5.1. Experimental Setup

####  5.1.1. Dataset

We simulate two theme-specific applications: (1) academic paper search using
SCIDOCS dataset (Thakur et al., 2021; Cohan et al., 2020), (2) product search
in e-commerce using Amazon ESCI dataset (Reddy et al.,
2022).777https://github.com/amazon-science/esci-data SCIDOCS dataset is widely
used as a benchmark dataset evaluating the zero-shot prediction capability of
retrieval models (Thakur et al., 2021). Amazon ESCI dataset is adopted from
KDD Cup 2022-Task 1. We use the English (US) data and treat ‘E (exact match)’
as the relevant relation. Each document contains the product title, product
description, and product bullet points. We focus on scenarios with no training
labels from the target corpus in section 5.2 and analyze the impacts of
labeled data in section 5.4.1. Table 2 summarizes the data statistics.

Table 2. Data statistics of two datasets. Avg. D/Q indicates the average
number of relevant documents per query.

| Academic domain | Product domain
---|---|---
#Corpus | 25,657 | 601,354
#Training query | - | 20,888
#Test query | 1,000 | 8,956
Avg. D/Q | 4.9 | 8.83

####  5.1.2. Corpus Topical taxonomy construction

The corpus topical taxonomy is obtained by applying the existing taxonomy
completion technique (Lee et al., 2022a) on a seed taxonomy. For the seed
taxonomy, we utilize the fields of study hierarchy from Microsoft Academic
Graph (Shen et al., 2018) (for the academic domain) and Amazon store
taxonomy888https://www.amazonlistingservice.com/blog/amazon-store-taxonomy-
organization sourced from Amazon.com (for the product domain). Each seed
taxonomy mirrors user interest in each application. The former reflects
researchers’ inclination towards structuring academic concepts and studies,
while the latter embodies customers’ interest in browsing and selecting
products. Table 3 provides the statistics of the constructed taxonomies.

Table 3. Taxonomy statistics of two datasets.

| Academic domain | Product domain
---|---|---
#Topic classes | 4,028 | 14,954
#Edges | 8,445 | 18,360
Depth | 5 | 10

####  5.1.3. Retrieval setup and metrics

Without ToTER, the retrieval process follows the standard (1) retrieval, (2)
reranking pipeline (section 2.1). ToTER has an added SSA step to reduce the
initial search space. Thus, with ToTER, the retrieval process follows (1) SSA,
(2) retrieval (with CRM), and (3) reranking (with QEP).

For each retrieval stage, we employ the metrics and retrieval sizes widely
used in the prior literature (MacAvaney et al., 2022; Kulkarni et al., 2023).
In the first-stage retrieval, which aims to retrieve all potentially relevant
documents, recall is mostly considered, with retrieval sizes typically ranging
from $100$ to $1000$. For the second-stage reranking, more sophisticated
metrics (e.g., NDCG, MAP) that consider the detailed rankings of relevant
documents are widely employed, with smaller retrieval sizes within $100$. For
SSA and the first-stage retrieval, we use Recall (R@$K$). For the second-stage
reranking, we additionally use NDCG (N@$K$) and MAP (MAP@$K$). In our
experiments, we set the size of search space adjustment
$|\mathcal{D}^{SSA}_{q}|=2500$ and the candidate set for reranking
$|\mathcal{D}_{q}|=100$ (MacAvaney et al., 2022).

Figure 2. Search space adjustment performance comparison on the academic and
product domain.

###  5.2. Results and Analysis (RQ1,RQ2, and RQ3)

To validate the effectiveness of each strategy of ToTER, we provide
comparisons with various baselines for each retrieval stage.

####  5.2.1. Initial search space adjustment

We first assess the efficacy of SSA in filtering out irrelevant documents from
the corpus. As this can be seen as a kind of retrieval with a large retrieval
size, we compare various retrieval methods. Note that SSA does not require
backbone retrieval models, as it only uses the class estimator.

Compared methods. (a) Lexical and sparse model: BM25 (Robertson et al., 2009),
docT5query (Nogueira et al., 2019), (b) pre-trained PLMs for each corpus:
SCIBERT (Beltagy et al., 2019) & BERT-base (Devlin et al., 2019), (c)
unsupervised domain adaptation: GenQ (Thakur et al., 2021), (d) PLM-based
retriever: Contriever-MS (Izacard et al., 2021), TAS-B (Hofstätter et al.,
2021), ColBERT (Khattab and Zaharia, 2020), SPLADE++ (Formal et al., 2022).
GenQ uses synthetic data generated for the target corpus999We fine-tune
Contriever-MS using BM25 negatives on the synthetic data., and other
retrievers in (d) are fine-tuned using massive labeled data from the general
domain (Nguyen et al., 2016).

Table 4. Retrieval performance comparison on the academic and product domain.
Significant differences with the baseline (i.e., retrieval without using ToTER
framework) are marked with * (p-value ¡ 0.05 in the one-sample t-test).

| Search space | Method | Contriever-MS | SPLADE++ | ColBERT
---|---|---|---|---|---
| R@100 | R@500 | R@1000 | R@100 | R@500 | R@1000 | R@100 | R@500 | R@1000
Academic domain |  | Retriever | 0.3783 | 0.5498 | 0.6216 | 0.3705 | 0.5294 | 0.6004 | 0.3382 | 0.4864 | 0.5624
Entire corpus $(\mathcal{D})$ | w/ BQE | 0.3846 | 0.5543 | 0.6280 | 0.3911 | 0.5523 | 0.6193 | 0.3484 | 0.4991 | 0.5783
w/ PRF | 0.3815 | 0.5510 | 0.6266 | 0.3852 | 0.5441 | 0.6146 | 0.3484 | 0.5011 | 0.5860
w/ GAR | 0.3848 | 0.5499 | 0.6218 | 0.3772 | 0.5336 | 0.6033 | 0.3486 | 0.4912 | 0.5629
w/ LADR | 0.3806 | 0.5626 | 0.6302 | 0.3763 | 0.5577 | 0.6264 | 0.3501 | 0.5339 | 0.6115
| w/ TopicGQA | 0.3835 | 0.5491 | 0.6261 | 0.3672 | 0.5271 | 0.5995 | 0.3393 | 0.4845 | 0.5581
| Retriever | 0.3887 | 0.5866 | 0.6793 | 0.3913 | 0.5809 | 0.6725 | 0.3586 | 0.5486 | 0.6502
Reduced corpus $(\mathcal{D}^{SSA})$ | w/ BQE | 0.3940 | 0.5861 | 0.6836 | 0.4086 | 0.5942 | 0.6856 | 0.3658 | 0.5557 | 0.6553
w/ PRF | 0.3915 | 0.5862 | 0.6806 | 0.4032 | 0.5896 | 0.6806 | 0.3615 | 0.5547 | 0.6593
w/ GAR | 0.3928 | 0.5862 | 0.6802 | 0.4099 | 0.5966 | 0.6815 | 0.3964 | 0.5887 | 0.6810
w/ TopicGQA | 0.3933 | 0.5873 | 0.6802 | 0.3902 | 0.5772 | 0.6719 | 0.3584 | 0.5420 | 0.6382
| w/ ToTER-CRM | 0.4432* | 0.6399* | 0.7268* | 0.4490* | 0.6364* | 0.7252* | 0.4326* | 0.6346* | 0.7232*
Product domain |  | Retriever | 0.4992 | 0.6962 | 0.8294 | 0.5220 | 0.7129 | 0.7732 | 0.5342 | 0.7091 | 0.7685
Entire corpus $(\mathcal{D})$ | w/ BQE | 0.5256 | 0.7258 | 0.8481 | 0.5582 | 0.7398 | 0.8056 | 0.5603 | 0.7593 | 0.8075
w/ PRF | 0.5097 | 0.7444 | 0.8572 | 0.5244 | 0.7661 | 0.8383 | 0.5350 | 0.7331 | 0.8265
w/ GAR | 0.5158 | 0.7231 | 0.8409 | 0.5455 | 0.7214 | 0.8067 | 0.5515 | 0.7421 | 0.8140
w/ LADR | 0.5157 | 0.7228 | 0.8498 | 0.5377 | 0.7308 | 0.8215 | 0.5435 | 0.7228 | 0.8259
| w/ TopicGQA | 0.5172 | 0.7343 | 0.8298 | 0.5334 | 0.7235 | 0.7924 | 0.5252 | 0.7072 | 0.7818
| Retriever | 0.5009 | 0.7085 | 0.8555 | 0.5231 | 0.7264 | 0.8229 | 0.5303 | 0.7401 | 0.8394
Reduced corpus $(\mathcal{D}^{SSA})$ | w/ BQE | 0.5285 | 0.7411 | 0.8632 | 0.5608 | 0.7467 | 0.8462 | 0.5638 | 0.7602 | 0.8492
w/ PRF | 0.5124 | 0.7462 | 0.8663 | 0.5311 | 0.7779 | 0.8603 | 0.5355 | 0.7649 | 0.8495
w/ GAR | 0.5427 | 0.7593 | 0.8564 | 0.5606 | 0.7540 | 0.8416 | 0.5569 | 0.7634 | 0.8396
w/ TopicGQA | 0.5186 | 0.7381 | 0.8493 | 0.5384 | 0.7286 | 0.8351 | 0.5390 | 0.7459 | 0.8598
| w/ ToTER-CRM | 0.5515* | 0.7899* | 0.8692* | 0.5717* | 0.7856* | 0.8648* | 0.5661 | 0.7997* | 0.8625*

Findings. Figure 2 presents recalls (R@$K$) for varying sizes of retrieved
documents ($K$). SSA consistently achieves the highest recall when $K$ is
large ($\geq 10^{3}$), which shows its efficacy in accurately filtering out
irrelevant documents. Using class relevance learning, ToTER categorizes
documents based on the theme-specific taxonomy. This approach aids in
identifying the central subject of documents, which may not be effectively
captured by the lexical and semantic similarity based on word overlap and
contextual meaning.

Among the competitors, PLM-based retrievers, fine-tuned with vast labeled
data, consistently show high recalls. On the other hand, GenQ, fine-tuned with
synthetic data, shows limited performance. This result also aligns with
(Chaudhary et al., 2023). We find that the generated queries are often trivial
and fail to reflect the domain-specific knowledge, which may lead to
suboptimal results. Lastly, the effectiveness of SSA rapidly declines when $K$
is small ($\leq 5\times 10^{2}$). This outcome is expected, given that SSA
only considers the overlap degree of relevant topic classes.

####  5.2.2. Retrieval

We assess the effectiveness of CRM for the first-stage retrieval. We select
three backbone models, which show competitive performance and also represent
three different encoding strategies: Contreiver-MS, SPLADE++, and ColBERT.
These correspond to single-vector, sparse, and multi-vector representation
models, respectively. We report results for both (a) without SSA (i.e.,
retrieval from $\mathcal{D}$) and (b) with SSA (i.e., retrieval from
$\mathcal{D}^{SSA}$).

Compared methods. We compare CRM with various state-of-the-art methods to
improve backbone retrievers using auxiliary corpus knowledge. The first group
leverages pseudo-relevance feedback.

  * •

BERT-QE (BQE) (Zheng et al., 2020) uses core text segments (or chucks)
obtained from top retrieval results to complement the original query.

  * •

PRF has been separately studied for the single-vector (Yu et al., 2021) and
multi-vector representation models (Wang et al., 2021). They exploit document-
and term-level knowledge for query enrichment, respectively. We apply the
corresponding PRF method for each backbone model.

The second group uses inter-document similarity via a corpus graph.

  * •

GAR (MacAvaney et al., 2022) uses nearest neighbors in the graph to refine the
initial ranking results. It is proposed for the reranking, but we apply it to
the retrieval as well, as it brings consistent improvements.

  * •

LADR (Kulkarni et al., 2023) uses lexical retriever in conjunction with the
corpus graph to gradually expand the search space for PLM-based retriever. We
use LADR-adaptive with no time constraints. As LADR controls the search space,
we apply it solely to $\mathcal{D}$.

As discussed in section 3, using topic knowledge for PLM-based retrievers has
not been studied well. Following a recent approach that uses generative
augmentation (Mao et al., 2021; Mackie et al., 2023), we devise a new baseline
that leverages topic knowledge discovered by PLMs.

  * •

TopicGQA uses generative query augmentation (Mao et al., 2021; Mackie et al.,
2023). Given a query, we extract its topic using PLMs, and enrich it by adding
the predicted topics using the same template to ToTER (Eq.7). We use T0-3B
with the prompt proposed in (Mackie et al., 2023) (Appendix A.2.2).

Findings. Table 4 presents the retrieval results. First, we observe that
retrieval from the filtered corpus via SSA ($\mathcal{D}^{SSA}$) consistently
yields higher recalls than retrieval from the entire corpus ($\mathcal{D}$),
which again shows the effectiveness of our topic-based SSA. Second, methods
that utilize auxiliary corpus knowledge consistently boost the retrieval
performance. While BQE and PRF excel in the product domain, corpus graph
knowledge demonstrates superior effectiveness in the academic domain.
Conversely, TopicGQA, which leverages topic knowledge extracted using PLMs,
shows limited effectiveness and even degrades the performance (e.g., R@1000,
ColBERT in the academic domain). We notice that TopicGQA often fails to
generate contexts reflecting domain knowledge. For example, in Table 1(a), it
generates topics like “data ownership” and “prove”, while relevant they do not
reveal the high-level contexts of the academic paper. Lastly, CRM consistently
shows the highest recall in all setups. Based on the taxonomy, it can identify
topic classes reflecting domain knowledge (e.g., “cryptography”, “computer
security”). The topical relatedness is incorporated with the semantic
similarity, providing complementary knowledge to each other. These
observations collectively show the effectiveness of using corpus taxonomy for
theme-specific retrieval.

Table 5. Reranking performance comparison on the academic and product domain.
Significant differences with the baseline (i.e., reranking without using ToTER
framework) are marked with * (p-value ¡ 0.05 in the one-sample t-test).

|  Candidate set generation | Method | MiniLM-L-12 | MonoT5-base
---|---|---|---|---
N@3 | N@10 | MAP@10 | R@10 | R@50 | N@3 | N@10 | MAP@10 | R@10 | R@50
Academic domain | Retriever | no reranking | 0.1589 | 0.1652 | 0.0966 | 0.1726 | 0.3166 | 0.1589 | 0.1652 | 0.0966 | 0.1726 | 0.3166
Reranker | 0.1695 | 0.1760 | 0.1030 | 0.1827 | 0.3347 | 0.1748 | 0.1835 | 0.1078 | 0.1936 | 0.3368
w/ GAR | 0.1701 | 0.1767 | 0.1036 | 0.1841 | 0.3358 | 0.1752 | 0.1854 | 0.1093 | 0.1972 | 0.3368
w/ TopicGQA | 0.1666 | 0.1753 | 0.1023 | 0.1822 | 0.3301 | 0.1727 | 0.1800 | 0.1060 | 0.1876 | 0.3309
| no reranking | 0.1748 | 0.1838 | 0.1074 | 0.1949 | 0.3633 | 0.1748 | 0.1838 | 0.1074 | 0.1949 | 0.3633
Retriever w/ ToTER-SSA, CRM | Reranker | 0.1780 | 0.1852 | 0.1090 | 0.1953 | 0.3634 | 0.1758 | 0.1868 | 0.1090 | 0.1997 | 0.3663
w/ GAR | 0.1784 | 0.1870 | 0.1101 | 0.1979 | 0.3633 | 0.1794 | 0.1900 | 0.1118 | 0.2013 | 0.3671
w/ TopicGQA | 0.1719 | 0.1829 | 0.1072 | 0.1918 | 0.3632 | 0.1752 | 0.1854 | 0.1087 | 0.1962 | 0.3634
| w/ ToTER-QEP | 0.1821* | 0.1915* | 0.1126* | 0.2026* | 0.3660* | 0.1828 | 0.1930* | 0.1137 | 0.2048 | 0.3732*
Product domain | Retriever | no reranking | 0.2917 | 0.2845 | 0.1592 | 0.2401 | 0.4425 | 0.2917 | 0.2845 | 0.1592 | 0.2401 | 0.4425
Reranker | 0.2972 | 0.2937 | 0.1664 | 0.2513 | 0.4544 | 0.3317 | 0.3214 | 0.1883 | 0.2642 | 0.4965
w/ GAR | 0.2972 | 0.2986 | 0.1697 | 0.2610 | 0.4741 | 0.3317 | 0.3217 | 0.1892 | 0.2642 | 0.5061
w/ TopicGQA | 0.2965 | 0.2952 | 0.1658 | 0.2522 | 0.4621 | 0.3205 | 0.3043 | 0.1720 | 0.2621 | 0.4967
| no reranking | 0.3048 | 0.2856 | 0.1603 | 0.2459 | 0.4627 | 0.3048 | 0.2856 | 0.1603 | 0.2459 | 0.4627
Retriever w/ ToTER-SSA, CRM | Reranker | 0.2903 | 0.2942 | 0.1644 | 0.2601 | 0.4689 | 0.3317 | 0.3215 | 0.1889 | 0.2642 | 0.5048
w/ GAR | 0.3047 | 0.3021 | 0.1714 | 0.2654 | 0.4759 | 0.3318 | 0.3253 | 0.1916 | 0.2699 | 0.5098
w/ TopicGQA | 0.2978 | 0.3039 | 0.1734 | 0.2608 | 0.4785 | 0.3301 | 0.3185 | 0.1887 | 0.2626 | 0.5021
| w/ ToTER-QEP | 0.3189* | 0.3139* | 0.1818* | 0.2701* | 0.4891* | 0.3416 | 0.3304* | 0.1921 | 0.2729* | 0.5227*

####  5.2.3. Reranking

We assess the effectiveness of QEP for the second-stage reranking. Following
(MacAvaney et al., 2022; Thakur et al., 2021), we use two backbone models:
MiniLM-L-12 (Wang et al., 2020), MonoT5-base (Nogueira et al., 2020). We
report results for reranking top-100 results from both (a) retriever and (b)
SSA & CRM.101010We use the retrieval results of Contriever-MS (for the
academic domain) and SPLADE++ (for the product domain), which show the highest
recalls within top-100.

Compared methods. We use GAR (MacAvaney et al., 2022), the state-of-the-art
method proposed for the reranking stage, as our main competitor. We also
compare TopicGQA. Note that QEP and TopicGQA only differ in the way of
generating contexts to enrich queries.

Findings. In Table 5, similar to results in Table 4, TopicGQA shows limited
performance. For the query in Table 1(b), it generates contexts of “dye”,
“peroxide”, and “ammonia”, failing to add new information. The best
performance is consistently achieved by using all three strategies of ToTER.
QEP differs from TopicGQA in that it identifies core phrases using both the
topic class knowledge and top-ranked results, under the idea of relevance
model (Lavrenko and Croft, 2017). These processes are guided by the taxonomy
reflecting user-interested aspects, which may not be effectively revealed from
the corpus graph. Based on the findings in section 5.2, we conclude that each
strategy of ToTER effectively enhances retrieval in each stage (RQ1, RQ2) and
also has great compatibility with PLM-based models (RQ3).

Table 6. Ablation results for class relevance learning of training phase.

| R@100 | R@500 | R@1000
---|---|---|---
ToTER-SSA, CRM | 0.4432 | 0.6399 | 0.7268
$\mathbf{y}^{s}$ only | 0.3809 | 0.6080 | 0.7131
CKD $\rightarrow$ Self-training | 0.4327 | 0.6242 | 0.7226
w/o ToTER | 0.3783 | 0.5498 | 0.6216

Table 7. Ablation results for each strategy of inference phase.

|  | R@2500 | R@5000 | R@10000
---|---|---|---|---
SSA | ToTER-SSA | 0.8439 | 0.9053 | 0.9599
Low-level focus | 0.7832 | 0.8784 | 0.9505
High-level focus | 0.5569 | 0.6964 | 0.8552
|  | R@100 | R@500 | R@1000
CRM | ToTER-SSA, CRM | 0.4432 | 0.6399 | 0.7268
Low-level focus | 0.3864 | 0.5739 | 0.6623
High-level focus | 0.3361 | 0.4606 | 0.5087
w/o ToTER | 0.3783 | 0.5498 | 0.6216
|  | N@10 | R@10 | R@50
QEP | ToTER-SSA, CRM, QEP | 0.1915 | 0.2026 | 0.3660
w/o top-ranked docs. | 0.1759 | 0.1855 | 0.3549
w/o ToTER | 0.1760 | 0.1827 | 0.3347

###  5.3. Study of ToTER (RQ4)

We provide a detailed ablation and hyperparameter study of ToTER. We report
results with Contriever-MS (for retriever) and MiniLM-L-12 (for reranker) on
academic domain.

####  5.3.1. Ablation study

Table 6 and Table 7 present ablation results for the training and inference
phase, respectively.

Training phase. We compare two ablations intended to verify the effectiveness
of our silver label generation and collective topic knowledge distillation
(CKD): (1) $\mathbf{y}^{s}$ only solely uses the generated silver labels for
the class relevance learning without CKD, and (2) CKD $\rightarrow$ Self-
training replaces CKD with the standard self-training. The self-training is a
well-established semi-supervised learning technique used to achieve better
generalization when the given labels are incomplete (Xie et al., 2016). The
core difference between self-training and CKD is that self-training generates
additional training signals using model prediction on individual data
instances, whereas CKD uses collective knowledge of the averaged prediction on
semantically similar documents.

We observe that both silver labels and CKD play important roles in class
relevance learning. First, the class relevance estimator only trained with
$\mathbf{y}^{s}$ consistently improves the retrieval performance. This
supports the effectiveness of our silver label generation strategy. Also, we
observe that exploring relevant but unlabeled classes is highly important.
Both self-training and CKD bring considerable improvements compared to using
only the silver label. However, replacing CKD with self-training consistently
degrades the retrieval effectiveness. We also find that self-training is
rather unstable compared to using CKD.

Table 8. Retrieval performance with varying retaining percent $m$. $m\,(\%)$ | Recall@100 | Recall@500 | Recall@1000
---|---|---|---
5 | 0.4396 | 0.6353 | 0.7216
10 | 0.4432 | 0.6399 | 0.7268
15 | 0.4414 | 0.6402 | 0.7274
20 | 0.4410 | 0.6401 | 0.7278
Table 9. Reranking performance with varying sizes of the number of core phrases $k$. $k$ | NDCG@10 | Recall@10 | Recall@50
---|---|---|---
2 | 0.1779 | 0.1888 | 0.3550
5 | 0.1915 | 0.2026 | 0.3660
7 | 0.1927 | 0.2044 | 0.3633
10 | 0.1866 | 0.1956 | 0.3632
w/o ToTER | 0.1760 | 0.1827 | 0.3347

Inference phase. We compare ablations for SSA, CRM, and QEP. For SSA and CRM,
which utilize the estimated class relevance distributions, we compare two
alternative design choices: (1) low-level focus emphasizes relevance to the
more specific, narrower classes found at the lowermost levels, specifically
the lowest two levels. (2) high-level focus is the opposite choice of the low-
level focus. It targets more general, broader classes closer to the root node,
specifically the top three levels. For QEP, we compare an ablation that
ablates the use of top-ranked documents in the core phrase identification,
denoted as (3) w/o top-ranked docs. Specifically, from the set of relevant
class phrases $\\{P_{j}|\hat{b}_{qj}=1\\}$, we collect the top-$k$ core
phrases $P^{q}$ that most frequently appear across the corpus.

We observe that both low-level and high-level focus results in suboptimal
results, and considering both board and specific topics consistently leads to
the best recalls. In particular, ignoring low-level classes (i.e., high-level
focus) more drastically degrades the retrieval accuracy. Also, we observe that
using the frequency information from top-ranked documents is indeed effective
in finding proper contexts to enrich queries. For example, for the query in
Table 1(a), w/o top-ranked docs identifies core phrases like “key”, “scale”,
“software”, “management”, and “system”, which are relevant but not closely
related to the query compared to the phrases obtained from ToTER. These
results support the validity of our design choice that uses the overall topic
class distributions for SSA and CRM, and our core phrase identification based
on the relevance model for QEP.

####  5.3.2. Hyperparameter study

We provide analyses to guide the hyperparameter selection of ToTER.
Specifically, we investigate the effects of two hyperparameters introduced by
ToTER: (1) $m$ denotes the retaining percent of topic classes (section 4.2),
and (2) $k$ denotes the number of core phrases used to enrich queries in QEP
(section 4.2.3). We report the results in the academic domain. Similar
tendencies are observed in the product domain. First, Table 8 presents the
effects of $m$ on retrieval. We observe stable performance overall with $m$
around $5-20\%$. In this work, we set $m=10\%$. Second, Table 9 presents the
effects of $k$ on reranking. We observe the best performance is achieved with
$k$ around $5-7$. In this work, we set $k=5$.

###  5.4. Exploratory Analysis (RQ5, RQ6)

We now explore the impact of labeled data and the quality of taxonomy on the
effectiveness of ToTER. We report the results with Contriever-MS (for
retriever) and MiniLM-L-12 (for reranker) on the product domain.

####  5.4.1. ToTER with labeled data

While this work focuses on scenarios with no labeled data from the target
corpus, ToTER can use labeled data for its training process (Appendix A.1).
Table 10 presents the performance of ToTER with varying amounts of labeled
data. L0/L1/L2/L3 denote the setups using 0/33/66/100% of the available
training labels, respectively. We observe that the overall performance of the
retriever and reranker is largely improved by fine-tuning with labeled data.
Next, we observe that ToTER effectively enhances the retrieval process in all
setups. In specific, SSA effectively narrows down the initial search space
without hurting recalls of the fine-tuned retriever. Furthermore, CRM and QEP
consistently improve both the retriever and reranker. These results show that
ToTER can effectively leverage the labeled data, yielding a good synergy with
the fine-tuned PLM-based models (RQ4).

####  5.4.2. Impacts of taxonomy quality

The power of ToTER is primarily attributed to the topical taxonomy. While the
methodology for taxonomy completion has been extensively studied and well-
established, it’s crucial to assess the robustness of ToTER regarding the
quality of the given taxonomy. To this end, we consider two aspects measuring
taxonomy quality (Lee et al., 2022a): (1) topic completeness, which assesses
how fully the topic nodes cover the true topics, and (2) term coherence, which
assesses the semantic relatedness of terms (or phrases) within a topic node.
To impair completeness, we apply random pruning, which randomly removes a node
and all its child nodes.111111We apply pruning by controlling the ratio of
removed nodes to the total nodes. To impair coherence, we apply level-wise
node shuffling, which randomly swaps nodes at the same level.121212The root
node has level $0$. We set the shuffling ratio as 10%. Note that such random
shuffling corresponds to an extreme scenario.

In Figure 3, we observe that although both types of noise degrade the
effectiveness, ToTER has a considerable degree of robustness. In particular,
it shows a more stable performance for the pruning. We conjecture this
stability arises because the relevance to the removed nodes can be partially
inferred from the relevance to the remaining topic nodes. For the node
shuffling, noise at deeper levels has higher impacts. This can be due to the
increased node numbers at deeper levels and the decreased number of child
nodes which can help to reduce the impacts of shuffling. Based on the
observations, we conclude that ToTER has a certain degree of robustness to
taxonomy quality and can effectively enhance retrieval using the existing
taxonomy completion methods (RQ5).

Table 10. Performance of ToTER with labeled data. Significant differences with
the baseline are marked with * (p-value ¡ 0.05 in the one-sample (L0) /paired
(L1-L3) t-test).

| Method | L0 | L1 | L2 | L3
---|---|---|---|---|---
R@100 | Retriever | 0.4992 | 0.5168 | 0.5179 | 0.5527
w/ ToTER-SSA | 0.5009 | 0.5191 | 0.5202 | 0.5665
w/ ToTER-SSA, CRM | 0.5515* | 0.5502* | 0.5865* | 0.6098*
R@500 | Retriever | 0.6962 | 0.7592 | 0.7801 | 0.8071
w/ ToTER-SSA | 0.7085 | 0.7697 | 0.7802 | 0.8090
w/ ToTER-SSA, CRM | 0.7899* | 0.7823* | 0.8179* | 0.8434*
R@1K | Retriever | 0.8294 | 0.8333 | 0.8442 | 0.8714
w/ ToTER-SSA | 0.8555 | 0.8353 | 0.8468 | 0.8718
w/ ToTER-SSA, CRM | 0.8692* | 0.8636* | 0.8671* | 0.8906*

L3 | N@3 | N@10 | MAP@10 | R@10 | R@50
---|---|---|---|---|---
Retriever & Reranker | 0.3188 | 0.3107 | 0.1746 | 0.2616 | 0.5072
w/ ToTER | 0.3241 | 0.3219 | 0.1907* | 0.2778* | 0.5249*

Figure 3. Retrieval results with taxonomy impaired in terms of (left) topic
completeness and (right) term coherence.

##  6\. Conclusion

We propose a new ToTER framework to enhance PLM-based retrieval in theme-
specific applications using a corpus topical taxonomy. ToTER identifies the
central topics of queries and documents with the guidance of topical taxonomy
via class relevance learning, and exploits their topical relatedness to
complement semantic matching by PLM-based models. ToTER introduces three
strategies, SSA, CRM, and QEP, which gradually focus on fine-grained ranking
following the retrieve-then-rerank pipeline. Our comprehensive experiments on
two real-world datasets ascertain the benefits of using topical taxonomy and
demonstrate the effectiveness of ToTER. In future work, we plan to explore
diverse approaches to leveraging the taxonomy in training retrievers and
rerankers.

## Acknowledgements

This work was supported IITP grant funded by MSIT (No.2018-0-00584,
No.2019-0-01906), NRF grant funded by the MSIT (No.RS-2023-00217286,
No.2020R1A2B5B03097210). It was also in part by US DARPA KAIROS Program No.
FA8750-19-2-1004 and INCAS Program No. HR001121C0165, National Science
Foundation IIS-19-56151, and the Molecule Maker Lab Institute: An AI Research
Institutes program supported by NSF under Award No. 2019897, and the Institute
for Geospatial Understanding through an Integrative Discovery Environment
(I-GUIDE) by NSF under Award No. 2118329.

## References

  * (1)
  * Arous et al. (2023) Ines Arous, Ljiljana Dolamic, and Philippe Cudré-Mauroux. 2023.  TaxoComplete: Self-Supervised Taxonomy Completion Leveraging Position-Enhanced Semantic Matching. In _WWW_. 2509–2518.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: Pretrained Language Model for Scientific Text. In _EMNLP_.  arXiv:arXiv:1903.10676
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ 3, Jan (2003), 993–1022.
  * Chaudhary et al. (2023) Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, Kazuma Hashimoto, Mike Bendersky, and Marc Najork. 2023.  Exploring the Viability of Synthetic Query Generation for Relevance Prediction.  _arXiv preprint arXiv:2305.11944_ (2023).
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers. In _ACL_.
  * Dai et al. (2023) Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2023.  Promptagator: Few-shot dense retrieval from 8 examples. In _ICLR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT_.
  * Dong et al. (2022) Qian Dong, Yiding Liu, Suqi Cheng, Shuaiqiang Wang, Zhicong Cheng, Shuzi Niu, and Dawei Yin. 2022.  Incorporating explicit knowledge in pre-trained language models for passage re-ranking. In _SIGIR_. 1490–1501.
  * Formal et al. (2022) Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022.  From distillation to hard negative sampling: Making sparse neural ir models more effective. In _SIGIR_. 2353–2359.
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  SPLADE: Sparse lexical and expansion model for first stage ranking. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2288–2292.
  * Gao and Callan (2022) Luyu Gao and Jamie Callan. 2022.  Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. In _ACL_. 2843–2853.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _EMNLP_. 6894–6910.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021.  Efficiently teaching an effective dense retriever with balanced topic aware sampling. In _SIGIR_. 113–122.
  * Huang et al. (2020) Jiaxin Huang, Yiqing Xie, Yu Meng, Yunyi Zhang, and Jiawei Han. 2020.  Corel: Seed-guided topical taxonomy construction by concept learning and relation transferring. In _KDD_. 1928–1936.
  * Iscen et al. (2019) Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. 2019.  Label propagation for deep semi-supervised learning. In _CVPR_. 5070–5079.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_ (2021).
  * Jian et al. (2016) Fanghong Jian, Jimmy Xiangji Huang, Jiashu Zhao, Tingting He, and Po Hu. 2016.  A simple enhancement for ad-hoc information retrieval via topic modelling. In _SIGIR_. 733–736.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ 7, 3 (2019), 535–547.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _EMNLP_. 6769–6781.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert. In _SIGIR_. 39–48.
  * Kim et al. (2021) Bosung Kim, Hyewon Choi, Haeun Yu, and Youngjoong Ko. 2021.  Query reformulation for descriptive queries of jargon words using a knowledge graph based on a dictionary. In _CIKM_. 854–862.
  * Kim et al. (2022) Jihyuk Kim, Minsoo Kim, and Seung-won Hwang. 2022.  Collective Relevance Labeling for Passage Retrieval. In _NAACL-HLT_. 4141–4147.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Kulkarni et al. (2023) Hrishikesh Kulkarni, Sean MacAvaney, Nazli Goharian, and Ophir Frieder. 2023.  Lexically-Accelerated Dense Retrieval. In _SIGIR_. 152–162.
  * Lavrenko and Croft (2017) Victor Lavrenko and W Bruce Croft. 2017.  Relevance-based language models. In _ACM SIGIR Forum_ , Vol. 51. ACM New York, NY, USA, 260–267.
  * Lee et al. (2022a) Dongha Lee, Jiaming Shen, SeongKu Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu. 2022a.  Taxocom: Topic taxonomy completion with hierarchical discovery of novel topic clusters. In _WWW_. 2819–2829.
  * Lee et al. (2022b) Dongha Lee, Jiaming Shen, Seonghyeon Lee, Susik Yoon, Hwanjo Yu, and Jiawei Han. 2022b.  Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation. In _Findings of the Association for Computational Linguistics: EMNLP 2022_. 1687–1700.
  * Li et al. (2023) Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2023.  SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval. In _SIGIR_.
  * Li et al. (2021) Xiangsheng Li, Jiaxin Mao, Weizhi Ma, Yiqun Liu, Min Zhang, Shaoping Ma, Zhaowei Wang, and Xiuqiang He. 2021.  Topic-enhanced knowledge-aware retrieval model for diverse relevance estimation. In _WWW_. 756–767.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 329–345.
  * Ma et al. (2020) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2020.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.  _arXiv preprint arXiv:2004.14503_ (2020).
  * MacAvaney et al. (2022) Sean MacAvaney, Nicola Tonellotto, and Craig Macdonald. 2022.  Adaptive re-ranking with a corpus graph. In _CIKM_. 1491–1500.
  * Mackie et al. (2023) Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023.  Generative Relevance Feedback with Large Language Models. In _SIGIR_.
  * Mao et al. (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021.  Generation-Augmented Retrieval for Open-Domain Question Answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4089–4100.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. 2020.  Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding. In _KDD_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human-generated machine reading comprehension dataset.  (2016).
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document Ranking with a Pretrained Sequence-to-Sequence Model. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 708–718.
  * Nogueira et al. (2019) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019.  From doc2query to docTTTTTquery.  _Online preprint_ 6 (2019), 2.
  * Norouzi et al. (2013) Mohammad Norouzi, Ali Punjani, and David J Fleet. 2013.  Fast exact search in hamming space with multi-index hashing.  _IEEE transactions on pattern analysis and machine intelligence_ 36, 6 (2013), 1107–1119.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In _NAACL-HLT_. 5835–5847.
  * Reddy et al. (2022) Chandan K Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022.  Shopping queries dataset: A large-scale ESCI benchmark for improving product search.  _arXiv preprint arXiv:2206.06588_ (2022).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _EMNLP_.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. In _EMNLP_. 2825–2835.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.
  * Shang et al. (2020) Jingbo Shang, Xinyang Zhang, Liyuan Liu, Sha Li, and Jiawei Han. 2020.  Nettaxo: Automated topic taxonomy construction from text-rich network. In _WWW_. 1908–1919.
  * Shen et al. (2018) Zhihong Shen, Hao Ma, and Kuansan Wang. 2018.  A Web-scale system for scientific knowledge exploration. In _Proceedings of ACL 2018, System Demonstrations_. 87–92.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Wang and Zuccon (2023) Shuai Wang and Guido Zuccon. 2023.  Balanced Topic Aware Sampling for Effective Dense Retriever: A Reproducibility Study. In _SIGIR_. 2542–2551.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _NeurIPS_ 33 (2020), 5776–5788.
  * Wang et al. (2021) Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021.  Pseudo-relevance feedback for multiple representation dense retrieval. In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_. 297–306.
  * Wei and Croft (2006) Xing Wei and W Bruce Croft. 2006.  LDA-based document models for ad-hoc retrieval. In _SIGIRl_. 178–185.
  * Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016.  Unsupervised deep embedding for clustering analysis. In _ICML_. PMLR, 478–487.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _ICLR_.
  * Yu et al. (2021) HongChien Yu, Chenyan Xiong, and Jamie Callan. 2021.  Improving query representations for dense retrieval with pseudo relevance feedback. In _CIKM_. 3592–3596.
  * Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021.  Optimizing dense retrieval model training with hard negatives. In _SIGIR_. 1503–1512.
  * Zhang et al. (2018) Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian Sadler, Michelle Vanni, and Jiawei Han. 2018.  Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering. In _KDD_. 2701–2709.
  * Zhang et al. (2022a) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022a.  Adversarial Retriever-Ranker model for Dense Retrieval. In _ICLR_.
  * Zhang et al. (2022b) Yu Zhang, Yu Meng, Xuan Wang, Sheng Wang, and Jiawei Han. 2022b.  Seed-guided topic discovery with out-of-vocabulary seeds. In _NAACL_.
  * Zhang et al. (2023) Yu Zhang, Yunyi Zhang, Martin Michalski, Yucheng Jiang, Yu Meng, and Jiawei Han. 2023.  Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts. In _WSDM_. 429–437.
  * Zheng et al. (2020) Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020.  BERT-QE: Contextualized Query Expansion for Document Re-ranking. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 4718–4728.

##  Appendix A Appendix

The source code of ToTER is provided through the author’s GitHub
repository.131313https://github.com/SeongKu-Kang/ToTER_WWW24

###  A.1. ToTER with Labeled Data

While ToTER mainly focuses on scenarios without available labeled data from
the target corpus, it can also benefit from harnessing $(q,d)$ labels. First,
ToTER can directly use $q$ for class relevance learning (Eq.5) by treating it
as an additional document. By using the class labels of its relevant document
$d$, we can reflect their relevance into the class estimator. Also, the
enriched queries by ToTER can be directly used to enhance the fine-tuning of
the retriever and reranker. That is, we use $q^{QEP}$ instead of $q$ as model
input, where the fine-tuning follows the standard contrastive learning
(Karpukhin et al., 2020).

###  A.2. Experiment details

####  A.2.1. Corpus Topical taxonomy construction

The corpus topical taxonomy is obtained by applying the existing taxonomy
completion technique (Lee et al., 2022a) on a seed taxonomy. Based on the seed
taxonomy, we conduct taxonomy completion which completes and adjusts the
taxonomy for the target corpus. This is a critical step to ensure the taxonomy
aligns with the target corpus. Note that the seed taxonomy is incomplete; it
contains numerous topics irrelevant to the corpus as well as failing to cover
all topics. For example, in the case of the SCIDOCS dataset, we discovered
that over 95% of topic classes from the seed taxonomy have no (or a very weak)
relevance to documents in the corpus. Also, as it does not cover all specific
topics in the corpus, we need to expand it by identifying new topics not
present in the original seed taxonomy. We use the recently proposed technique
(Lee et al., 2022a) to obtain the corpus topical taxonomy. We use the official
implementation provided by the
authors.141414https://github.com/donalee/taxocom/tree/main A notable change
from the original implementation is that we additionally use PLM knowledge for
more effective topic discovery (Zhang et al., 2023).

####  A.2.2. Implementation details

In our experiments, we use BEIR benchmark
framework151515https://github.com/beir-cellar/beir for evaluating all compared
methods. For BM25, we use Elasticsearch. For docT5query and GenQ, we use T5
models with checkpoints provided by BEIR.161616castorini/doc2query-t5-base-
msmarco, BeIR/query-gen-msmarco-t5-base-v1 For all compared PLM-based
retrieval and reranking models, we use checkpoints that are publicly
available: Contriever-MS171717facebook/contriever-msmarco, TAS-B181818msmarco-
distilbert-base-tas-b,
ColBERT191919https://github.com/terrierteam/pyterrier_colbert,
SPLADE++202020naver/splade-cocondenser-ensembledistil, MiniLM-L-12212121cross-
encoder/ms-marco-MiniLM-L-12-v2, and MonoT5-base222222castorini/monot5-base-
msmarco-10k. For multi-vector representation models, we use ColBERT.v1 solely
for compatibility with the public ColBERT-PRF implementation.

To generate the corpus graph, we use Contriever-MS for the academic domain and
SPLADE++ for the product domain, as they consistently show the highest recalls
within top-100. The number of neighbors in the graph is set as $10$, as it
shows stable results in both GAR (MacAvaney et al., 2022) and LADR (Kulkarni
et al., 2023). For BERT-QE (Zheng et al., 2020), we set both the number of
top-ranked documents and the number of core segments (or chunks) as 10,
following the paper. For PRF of the single and sparse representation model
(i.e., Contriever-MS, SPLADE++), we set the number of documents for query
enrichment as $5$. For PRF for the multi-vector representation model (i.e.,
ColBERT), we use the official implementation and provided values. For other
baseline-specific hyperparameters, we follow the recommended values in the
original papers (Yu et al., 2021; Wang et al., 2021; MacAvaney et al., 2022;
Kulkarni et al., 2023). For TopicGQA, we use T0-3B232323bigscience/T0_3B with
the prompt suggested in (Mackie et al., 2023): ”Based on the query, generate a
bullet-point list of relevant topics present in relevant documents:”.

For fine-tuning with labeled data in the product domain (section 5.4.1), we
use sentence_transformers framework (Reimers and Gurevych, 2019). We continue
fine-tuning for $10$ epochs with a learning rate of $7e^{-5}$. We discovered
that further increasing the training epochs consistently degrades the
retrieval accuracy. For ToTER, we set the retaining percent $m=10\%$, the
number of core phrases $k=5$. Following BERT-QE (Zheng et al., 2020), the
number of top-ranked documents (for collective labels and QEP) is set to $10$.

###  A.3. Supplementary Results

We provide additional case studies of ToTER and generation-based baselines
(i.e., GenQ, TopicGQA). Table 11 presents the generated contexts from GenQ and
TopicGQA for examples in Table 1. We observe that the generated queries and
topics do not effectively reveal the domain-specific contexts. These results
again show the difficulty of retrieval in theme-specific applications and the
importance of proper use of well-structured corpus knowledge.

Lastly, Table A.3 presents additional case studies for each domain. Similar to
the previous case studies in Table 1, ToTER can effectively identify common
topic classes and core topic phrases, guided by the theme-specific taxonomy.
This information complements the semantic matching by PLM-based retrieval
models, allowing the models to more accurately find relevant documents.

Table 11. Case studies of generation-based baselines for examples in Table 1.
For GenQ for the product domain, we present results of Document A.

2.5 |  (a) Academic domain |  (b) Product domain
---|---|---
GenQ |  “what are pors.” |  “what is the name for the hair dyes.”
“what is por.” |  “what is onc hair color.”
TopicGQA |  data ownership, prove, store, proving, possession |  dye, peroxide, ammonia, hair, natural
2.5 |  |

Table 12. Additional case studies of ToTER. We use Contriever-MS (retriever)
and MiniLM-L-12 (reranker). Contents closely related to the query are denoted
in bold.

2.5 (a) Academic domain | (b) Product domain |
---|---|---
Query |  Exploring venue popularity in foursquare |  Query |  0.07 lash extensions not easy fans
Document A (label: relevant,  rank: top-167) |  Event-based social networks: linking the online and offline social worlds. Newly emerged event-based online social services, such as Meetup and Plancast, have experienced increased popularity and rapid growth. From these services, we observed a new type of social network - event-based social network (EBSN). … This paper is the first research to study EBSNs at scale and paves the way for future studies on this new type of social network. |  Document A (label: relevant,  rank: top-57) |  Volume Lash Extensions 0.07 D Curl Mix 8-15mm Eyelash Extensions. … One Second easily grafted eyelash, a special craft that allowing you to create 2D10D fans. Thickness 0.03/0.05/0.07/0.10mm. CurlC/CC/D/DD curl lash extensions. Single length 8mm25mm. Mixed length 8-15mm MIX, 9-16mm MIX, 15-20mm MIX in one tray. The root of the volume lashes extensions will not separate, any flowering and novices can operate. …
2.5 ToTER rank: top-28 |  Topic classes: diffusion, social network, social behavior, world wide web, computer science, economy, economics |  ToTER rank: top-7 |  Topic classes: makeup brushes and tools, false lashes, tools and accessories, eye, beauty and personal care
Core phrases: point of interest, location, check, social network, foursquare |  Core phrases: eyelash, fan, extensions, volume lash extensions, lash
2.5 |  |  |