#  TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text
Classification with Minimal Supervision

Yunyi Zhang  University of Illinois Urbana-Champaign, IL,
<EMAIL_ADDRESS>,  Ruozhen Yang∗ University of Illinois Urbana-
Champaign, IL<EMAIL_ADDRESS>,  Xueqiang Xu∗ University of Illinois
Urbana-Champaign, IL<EMAIL_ADDRESS>,  Jinfeng Xiao  University of
Illinois Urbana-Champaign, IL<EMAIL_ADDRESS>,  Jiaming Shen  Google
Research
NY<EMAIL_ADDRESS>and  Jiawei Han  University of Illinois Urbana-
Champaign, IL<EMAIL_ADDRESS>
###### Abstract.

Hierarchical text classification aims to categorize each document into a set
of classes in a label taxonomy. Most earlier works focus on fully or semi-
supervised methods that require a large amount of human annotated data which
is costly and time-consuming to acquire. To alleviate human efforts, in this
paper, we work on hierarchical text classification with the minimal amount of
supervision: using the sole class name of each node as the only supervision.
Recently, large language models (LLM) show competitive performance on various
tasks through zero-shot prompting, but this method performs poorly in the
hierarchical setting, because it is ineffective to include the large and
structured label space in a prompt. On the other hand, previous weakly-
supervised hierarchical text classification methods only utilize the raw
taxonomy skeleton and ignore the rich information hidden in the text corpus
that can serve as additional class-indicative features. To tackle the above
challenges, we propose TELEClass, Taxonomy Enrichment and LLM-Enhanced weakly-
supervised hierarchical text Classification, which (1) automatically enriches
the label taxonomy with class-indicative topical terms mined from the corpus
to facilitate classifier training and (2) utilizes LLMs for both data
annotation and creation tailored for the hierarchical label space. Experiments
show that TELEClass can outperform previous weakly-supervised hierarchical
text classification methods and LLM-based zero-shot prompting methods on two
public datasets.

Weakly-Supervised Text Classification, Hierarchical Text Classification,
Taxonomy Enrichment, Large Language Model

∗Equal Contribution.

##  1\. Introduction

Hierarchical text classification, aiming to classify documents into one or
multiple classes in a label taxonomy, is an essential and fundamental task in
text mining and natural language processing. Compared with standard text
classification where label space is flat and relatively small (e.g., less than
20 classes), hierarchical text classification is more challenging given the
larger and more structured label space and the existence of fine-grained and
long-tail classes in the taxonomy. Most earlier works tackle this task in
fully supervised (You et al., 2019; Huang et al., 2019; Chen et al., 2021) or
semi-supervised settings (Berthelot et al., 2019; Gururangan et al., 2019),
and different models are proposed to learn from a substantial amount of human-
labeled data. However, acquiring human annotation is often costly, time-
consuming, and not scalable.

Figure 1. An example document tagged with 3 classes. We automatically enrich
each node with class-indicative terms and utilize LLMs to facilitate
classification.

Recently, large language models (LLM) such as GPT-4 (OpenAI, 2023) and Llama2
(Touvron et al., 2023) have demonstrated strong performance in flat text
classification through zero-shot or few-shot prompting (Sun et al., 2023).
However, applying LLMs in hierarchical settings, with large and structured
label spaces, remains challenging. Directly including hundreds of classes in
prompts is ineffective, leading to structural information loss, increased
computational costs, and diminished clarity for LLMs in focusing on critical
information. Along another line of research, Meng et al. (2019) propose to
train a moderate-size text classifier by utilizing a small set of keywords or
labeled documents for each class. However, compiling keyword lists for
hundreds of classes and obtaining representative documents for each specific
and niche category still demand significant human efforts. Shen et al. (2021)
study the hierarchical text classification with _minimal supervision_ , which
takes the class name as the only supervision signal. Specifically, they
introduce TaxoClass, a method that generates pseudo-labels with a textual
entailment model for classifier training. However, this method overlooks
additional class-relevant features in the text and suffers from unreliable
pseudo-label selection due to the entailment model’s domain shift.

In this study, we advance minimally supervised hierarchical text
classification by integrating corpus-based taxonomy enrichment and leveraging
LLMs tailored for the hierarchical label structure. First, we enrich the
existing label taxonomy skeleton with automatically extracted class-specific
topical terms from the corpus, enhancing supervision signals and thus
improving the pseudo-label quality for classifier training. Second, we explore
the application of LLMs in hierarchical text classification from two
perspectives. Specifically, we enhance LLM annotation efficiency and
effectiveness through a taxonomy-guided candidate search and optimize LLM-
generated document accuracy by using taxonomy paths to create more precise
pseudo-data.

Leveraging the above ideas, we introduce TELEClass: Taxonomy Enrichment and
LLM-Enhanced weakly-supervised hierarchical text Classification. TELEClass
consists of four major steps: (1) _LLM-Enhanced Core Class Annotation_ , where
we identify document “core classes” (i.e., fine-grained classes that most
accurately describe the documents) by first using a textual entailment model
for top-down candidate search on the taxonomy and then prompting an LLM to
select the most precise core classes. (2) _Corpus-Based Taxonomy Enrichment_ ,
where we analyze the taxonomy structure and the roughly classified documents
to identify class-indicative topical terms through semantic and statistical
analysis, enriching the label taxonomy with more features. For example, the
“conditioner” class in Figure 1 is enriched with key terms like “moisture” and
“soft hair”, which distinguish it from its sibling classes. (3) _Core Class
Refinement with Enriched Taxonomy_ , where we embed documents and classes
based on the enriched label taxonomy and refine the initially selected core
classes by identifying the most similar classes for each document. (4) _Text
Classifier Training with Path-Based Data Augmentation_ , where we sample label
paths from the taxonomy and guide the LLM to generate pseudo documents most
accurately describing these fine-grained classes. Finally, we train the text
classifier on two types of pseudo-labels, the core classes and the generated
data, with a simple text-matching network and multi-label training strategy.

The contributions of this paper are summarized as follows: (1) We propose
TELEClass, a new method for minimally supervised hierarchical text
classification, which requires only the class names of the label taxonomy as
supervision to train a multi-label text classifier. (2) We propose to enrich
the label taxonomy with class-indicative topical terms mined from the text
corpus, based on which we utilize an embedding-based document-class matching
method to improve the pseudo-label quality. (3) We study two ways of adopting
large language models to the hierarchical text classification setting, which
can improve the pseudo-label quality and solve the data scarcity issue for
fine-grained classes. (4) Extensive experiments on two datasets show that
TELEClass can outperform strong weakly-supervised hierarchical text
classification methods and zero-shot LLM prompting methods. 111Code and
datasets are available upon request now and will later be available online.

##  2\. Problem Definition

Following Shen et al. (2021), we define the minimally-supervised hierarchical
text classification task as to train a text classifier that can categorize
each document into multiple nodes on a label taxonomy by using the name of
each node as the only supervision. For example, in Figure 1, the input
document is classified as “hair care”, “shampoo”, and “scalp treatment”.

Formally, the task input includes an unlabeled text corpus
$\mathcal{D}=\\{d_{1},\dots,d_{|\mathcal{D}|}\\}$ and a directed acyclic graph
$\mathcal{T}=(\mathcal{C},\mathcal{R})$ as the label taxonomy. Each
$c_{i}\in\mathcal{C}$ represents a target class in the taxonomy, coupled with
a unique textual surface name $s_{i}$. Each edge $\langle
c_{i},c_{j}\rangle\in\mathcal{R}$ indicates a hypernymy relation, where class
$c_{j}$ is a sub-class of $c_{i}$. For example, one such edge in Figure 1 is
between $s_{i}=\text{``hair care''}$ and $s_{j}=\text{``shampoo''}$. Then, the
goal of our task is to train a multi-label text classifier $f(\cdot)$ that can
map a document $d$ into a binary encoding of its corresponding classes,
$f(d)=[y_{1},\dots,y_{|\mathcal{C}|}]$, where $y_{i}=1$ represents $d$ belongs
to class $c_{i}$, otherwise $y_{i}=0$.

##  3\. Methodology

In this section, we will introduce TELEClass consisting of the following
modules: (1) LLM-enhanced core class annotation, (2) corpus-based taxonomy
enrichment, (3) core class refinement with enriched taxonomy, and (4) text
classifier training with path-based data augmentation. Figure 2 shows an
overview of TELEClass.

Figure 2. Overview of the TELEClass framework.

###  3.1. LLM-enhanced Core Class Annotation

Inspired by previous studies, we first tag each document with its “core
classes”, which are defined as a set of classes that can describe the document
most accurately (Shen et al., 2021). This process also mimics the process of
human performing hierarchical text classification: first, select a set of most
essential classes for the document and then trace back to their ancestors to
complete the labeling. For example, in Figure 1, by first tagging the document
with “shampoo” and “scalp treatment”, we can easily find its complete set of
classes.

To mine the core classes for each document, existing methods utilize a textual
entailment model (e.g., RoBERTa-large-MNLI) and combine tree-based top-down
candidate searching with corpus-level statistical analysis for core class
selection. Specifically, the similarity score between a document and a class
is defined as the entailment score from the document to a class-specific
hypothesis such as “This example is [Class Name]”. These similarity scores
facilitate a top-down search in the taxonomy to shortlist candidate classes,
which are then evaluated across the corpus to determine the most reliable core
classes.

However, there are two limitations of this method. First, the textual
entailment model, not being directly trained for document classification,
offers inconsistent zero-shot performance, particularly in distinguishing
closely related, fine-grained classes within a hierarchical structure. Second,
given a pair of premise (document) and hypothesis, the entailment model is
trained to predict a distribution over three classes, namely entailment,
neutral, and contradiction. Therefore, it is questionable whether the
generated scores can support corpus-level comparison, because the scores
between different pairs may not be directly comparable. Both limitations will
lead to noisy core class selection and adversely impact the classifier
performance.

In this work, we propose to utilize an LLM to enhance the core class
annotation step. These LLMs are trained on very large textual data and human
preference data and shown to have superior zero-shot ability for the document
classification task with a small and flat label space (Sun et al., 2023).
However, LLMs cannot effectively understand the hierarchical label space,
which is large and structured and cannot be easily represented in a prompt.
Therefore, we precede the LLM annotation with the top-down candidate search
method using a textual entailment model.

Specifically, for each document $d_{i}$, we start from the Root node at level
0 of the taxonomy by adding it into a queue. For each class at level $l$ in
the queue, we select $l+3$ of its children that have the highest similarity to
document $d_{i}$, where the similarity score $sim^{\text{entail}}(c,d_{i})$ is
obtained from a textual entailment model. Then, among all the selected
children at level $l+1$, we keep the top $(l+2)^{2}$ of them with the highest
path-based similarity scores and add to the queue, where the path-based
similarity $sim^{\text{path}}(c,d_{i})$ is defined recursively as,

$\displaystyle sim^{\text{path}}(\text{Root},d)$ $\displaystyle=1,$
$\displaystyle sim^{\text{path}}(c,d)$ $\displaystyle=\max_{c_{p}\in
Par(c)}sim^{\text{path}}(c_{p},d_{i})\cdot sim^{\text{entail}}(c,d_{i}),$

where $Par(c)$ is the set of all parents of $c$. This process finishes when
there is no class in the queue, and all the nodes (other than Root) ever
pushed into the queue are treated as candidate core classes for document
$d_{i}$. To use an LLM to annotate core classes, we first construct an
instruction for each document, which asks the LLM to select the most accurate
classes for the document from its candidate core classes. Then, the document
is fed to the LLM as a query and it will generate a list of core classes
according to the document and candidate core classes.

Our LLM-enhanced core class annotation step will assign an initial set of core
classes (denoted as $\mathbb{C}^{0}_{i}$) for each document
$d_{i}\in\mathcal{D}$. In the next two steps, we introduce how TELEClass
leverages the initial core classes to enrich the label taxonomy with class-
indicative features. This enriched taxonomy is subsequently employed to refine
the initial core classes, utilizing embedding-based similarity scores that are
more comparable across documents.

###  3.2. Corpus-Based Taxonomy Enrichment

Due to the hierarchical nature of taxonomy, fined-grained classes at lower
levels are often hard to distinguish with only class label names. Therefore,
we propose to enrich the weak supervision, i.e. the label taxonomy, with a set
of _class-indicative topical terms_ for each class. These enriched terms can
help better interpret the class and thus facilitate text classification. For
example, in Figure 2, “shampoo” and “conditioner” are two fine-grained classes
that are similar to each other. We can effectively separate the two classes by
identifying a set of class-specific terms such as “flakes” for “shampoo” and
“moisture” for “conditioner”.

In this work, we combine statistical document occurrence features and semantic
similarity and further contrast a class with its siblings to identify a set of
key terms that can distinguish it from its most similar nodes in the taxonomy.
Notice that, because the input taxonomy $\mathcal{T}$ is a directed acyclic
graph rather than a simple tree, a node $c$ can have multiple sets of
siblings, each of which corresponds to one of $c$’s parent nodes. Besides,
because we have already tagged each document with its core classes, we can use
them as rough classification results and form a document cluster for each
class to help the enrichment step.

Formally, given a class $c\in\mathcal{C}$ and its siblings corresponding to
one of its parents $c_{p}$, $Sib(c,c_{p})=\\{c^{\prime}\in\mathcal{C}|\langle
c_{p},c^{\prime}\rangle\in\mathcal{R}\\}$, $\mathbf{c}_{p}\in Par(c)$, we find
a set of _class-indicative topical terms_ of $c$ corresponding to $c_{p}$,
denoted as $T(c,c_{p})=\\{t_{1},t_{2},\dots,t_{k}\\}$. Each term in
$T(c,c_{p})$ can signify the class $c$ and distinguish it from its siblings
under $c_{p}$. To identify these class-indicative terms, we construct a set of
documents $D^{0}_{c}$ for each class $c$, containing all the documents whose
initial core classes contain $c$ or its descendants,

(1) $\small D^{0}_{c}=\left\\{d_{i}\in D|\exists
c^{\prime}\in\mathbb{C}^{0}_{i},c^{\prime}\in\\{c\\}\cup Des(c)\right\\},$

where $Des(c)$ is the set of all descedants of $c$. We consider the following
3 factors for class-indicative term selection: _popularity_ ,
_distinctiveness_ , and _semantic similarity_.

Popularity. First, a class-indicative term of $c$ should be frequently
mentioned by the documents about $c$, denoted as its popularity within $c$. We
calculate the popularity of a term $t$ within a class $c$ as,

(2) $\small pop(t,c)=\log(1+df(t,D^{0}_{c})),$

where the document frequency $df(t,D)$ stands for the number of documents in
$D$ that mention $t$.

Distinctiveness. Second, a class-indicative term $t$ for a class $c$ should
also be infrequent in its siblings so that a document containing $t$ likely
indicates that it is about $c$ instead of $c$’s siblings. We define the
distinctiveness score of a term $t$ for a class $c$, corresponding to its
siblings under $c_{p}$, as,

(3) $\small
dist(t,c,c_{p})=\frac{\exp(BM25(t,D^{0}_{c}))}{1+\sum_{c^{\prime}\in
Sib(c,c_{p})}\exp(BM25(t,D^{0}_{c^{\prime}}))},$

where $BM25(\cdot,\cdot)$ denotes the BM25 relevance function (Robertson and
Zaragoza, 2009).

Semantic Similarity. Third, we also require a class-indicative term to be
semantically similar to the class name. We use a small pretrained language
model (e.g., BERT-base-uncased (Devlin et al., 2019)) to encode each term and
class name. Specifically, for each $w$, either a class name or a term from the
corpus, we feed it into a PLM as $\texttt{[CLS]}\,w\,\texttt{[SEP]}$. Because
$w$ may be tokenized into multiple word pieces by the PLM, we take the average
of the final layer embeddings of all its pieces as the embedding of $w$,
denoted as $\mathbf{t}$ for a term $t$ and $\mathbf{c}$ for a class $c$. Then,
cosine similarity is used to measure the semantic similarity between a term
$t$ and a class $c$.

Finally, we define the affinity score between a term $t$ and a class $c$
corresponding to parent $p$ to be the geometric mean of the above scores, with
a weight parameter $\alpha$,

(4) $\small aff(t,c,c_{p})=pop(t,c)^{\alpha}\cdot
dist(t,c,c_{p})^{1-\alpha}\cdot\cos(\mathbf{t},\mathbf{c}).$

Taxonomy Enrichment. We first apply a phrase mining tool, AutoPhrase (Shang et
al., 2018), to mine quality single-token and multi-token phrases from the
corpus to serve as candidates for taxonomy enrichment. Then, for each class
$c$ and each of its parents $p$, we select the top-$k$ terms with the highest
affinity scores with $c$ corresponding to $p$, denoted as $T(c,c_{p})$. Then,
we aggregate the terms corresponding to each parent of $c$ to get the final
enriched class-indicative topical terms for class $c$ as follows

(5) $\small T_{c}=\bigcup_{c_{p}\in Par(c)}T(c,c_{p}).$

###  3.3. Core Class Refinement with Enriched Taxonomy

With the enriched class-indicative terms for each class, we propose to further
utilize them to refine the initial core classes. In this paper, we adopt an
embedding-based document-class matching method, which has two distinct
advantages in the hierarchical setting. (1) By pre-computing the document and
class representations, calculating their similarity is much more efficient
than the textual entailment-based method, promoting a more comprehensive score
comparison on the entire taxonomy instead of just on a sub-tree. (2) While
textual entailment model is trained to predict an entailment score for each
pair of document and class, which follows an unknown distribution and is
unreliable for corpus-level comparison, the embedding similarity is more
general and easier to compare across classes and documents.

Unlike previous methods in flat text classification (Wang et al., 2021a) that
use keyword-level embeddings to estimate document and class representations,
here, because we already have a set of roughly classified documents for each
class, $D^{0}_{c}$, we can directly define their representations based on
document-level embeddings. To obtain document representations, we utilize a
pretrained Sentence Transformer model (Reimers and Gurevych, 2019) to encode
the entire document, which we denote as $\vec{d}$. Then, for each class $c$,
we identify a subset of its assigned documents which explicitly mention at
least one of the class-indicative keywords and thus most confidently belong to
this class, $D_{c}=\\{d\in D^{0}_{c}|\exists w\in T_{c},w\in d\\}$. Then, we
use the average of their document embeddings as the class representation,
$\vec{c}=\frac{1}{|D_{c}|}\sum_{d\in D_{c}}\vec{d}$. Finally, we compute the
document-class matching score as the cosine similarity between their
representations.

Based on the document-class matching scores, we make an observation that the
true core classes often have much higher matching scores with the document
compared to other classes. Therefore, we use the largest “similarity gap” for
each document to identify its core classes. Specifically, for each document
$d_{i}\in\mathcal{D}$, we first get a ranked list of classes according to the
matching scores, denoted as
$[c^{i}_{1},c^{i}_{2},\dots,c^{i}_{|\mathcal{C}|}]$, where
$\text{diff}^{\,i}(j):=\cos(\vec{d}_{i},\vec{c}^{i}_{j})-\cos(\vec{d}_{i},\vec{c}^{i}_{j+1})>0$
for $j\in\\{1,\dots,|\mathcal{C}|-1\\}$. Then, we find the position $m_{i}$
with the highest similarity difference with its next one in the list. After
that, we treat the classes ranked above this position as this document’s
refined core classes $\mathbb{C}_{i}$, and the corresponding similarity gap as
the confidence estimation $conf_{i}$.

(6)
$\displaystyle\small\begin{split}conf_{i}=\text{diff}^{\,i}(m_{i}),\quad\mathbb{C}_{i}&=\\{c^{i}_{1},c^{i}_{2},\dots,c^{i}_{m_{i}}\\},\\\
m_{i}=\operatorname*{arg\,max}_{j\in\\{1,\dots,|\mathcal{C}|-1\\}}&\text{diff}^{\,i}(j).\end{split}$

Finally, we select the set of documents and their refined core classes with
the top 50% confidence scores $conf_{i}$, to be the final set of refined core
classes for classifier training. Formally,
$\mathcal{D}^{\text{core}}=\\{d_{i}\\}$, where each
$d_{i}\in\mathcal{D}^{\text{core}}$ satisfies
$|\\{d_{j}|conf_{j}<conf_{i}\\}|\geq\frac{1}{2}|\mathcal{D}|$.

###  3.4. Text Classifier Training with Path-Based Data Augmentation

The final step of TELEClass is to train a hierarchical text classifier using
the confident refined core classes. One straightforward way is to directly use
the selected core classes as a complete set of pseudo-labeled documents and
train a text classifier in a common supervised way. However, such a strategy
is ineffective, because the core classes are not comprehensive enough and
cannot cover all the classes in the taxonomy. The reason is two-fold. First,
the core class selection step is document-centric, meaning that it only
focuses on selecting the most accurate core classes for each document, while
ignoring whether each class in the taxonomy is ever selected as a core class.
Second, the label space in the hierarchical text classification task naturally
contains some fine-grained and long-tail classes, so they are often not
guaranteed to be selected as core classes due to their low frequency.
Empirically, for the two datasets we use in our experiments, Amazon and
DBPedia, the missing rates of the selected core classes are 11.6% and 5.4%,
respectively. These missing classes will never be used as positive classes in
the training process, if we only train the classifier with the selected core
classes.

Therefore, to overcome this issue, we further utilize the language generation
ability of LLMs to augment the current pseudo-labels. We propose the idea of
_path-based document generation by LLMs_ to generate augmented documents for
each path from a level-1 node to a leaf node in the taxonomy. By adding the
generated documents to the pseudo-labeled data, we can ensure the coverage of
the pseudo-data. Moreover, we use a path instead of a single class to guide
the LLM generation, because the meaning of lower-level classes is often
conditioned on their parents. For example, in Figure 2, a path “hair care”
$\rightarrow$ “shampoo” can guide the LLM to generate text about hair shampoo
instead of pet shampoo or carpet shampoo that are in different paths.

More specifically, for each path from a level-1 node to a leaf node in the
taxonomy, $p=(c_{1},\dots,c_{l})$, we query an LLM to generate a small number
of documents (e.g., $q=5$) corresponding to the path $p$ and denote the
resulting documents as $d^{p}_{i}$, $i=1,\dots,q$. By generating $q$ documents
for each path and treating all the classes in the query path as the pseudo-
labels for the generated documents, we get a set of generated data
$\mathcal{D}^{\text{gen}}=\\{d^{p}_{i}\\}_{p\in
Path(\mathcal{T}),i=1,\dots,q}$, where $Path(\mathcal{T})$ denotes all such
paths in the taxonomy $\mathcal{T}$, and $\mathbb{C}_{p}$ is the set of
classes in the path $p$. $\mathcal{D}^{\text{gen}}$ can serve as augmented
data to train the text classifier together with the selected refined core
classes $\mathcal{D}^{\text{core}}$. Notice that, here we generate pseudo
documents for _each_ path instead of just generating for the classes that have
no or very few assigned documents. The reason is that, because our core
classes are obtained based on static keyword-based features (i.e., the
enriched class-indicative topical terms), it may induce certain bias of the
matching process into the pseudo-data and affect the learning process (Dong et
al., 2023). Therefore, we use an LLM to generate pseudo documents for each
path in the taxonomy, which can also serve as regularization of the potential
bias.

Now, with two sets of pseudo-data prepared, we are ready to introduce the
classifier architecture and the training process.

Classifier architecture. Because we do not focus on proposing a new model for
the hierarchical text classification task, we choose to use a standard
architecture as our text classifier, including a document encoder, class
embeddings, and a matching network.

  * •

Document encoder: we initialize the document encoder $g(\cdot)$ with a
pretrained BERT-base model (Devlin et al., 2019) and use the hidden
representation of the [CLS] token from the last layer as the document
encoding, $\mathbf{d}_{i}=g(d_{i})$.

  * •

Class embeddings: we initialize the class embeddings with the class surface
name embeddings obtained by a pretrained BERT model (c.f. Section 3.2),
denoted by $\mathbf{c}_{j}$. Notice that, here we detach the embeddings from
the base model, so only the embeddings will be updated during the training
process without back-propagation to the backbone model.

  * •

Matching Network: we utilize a log-bilinear matching network to calculate the
probability of document $d_{i}$ belonging to class $c_{j}$:

$\small
p(c_{j}|d_{i})=\mathcal{P}(y_{j}=1|d_{i})=\sigma(\exp{(\mathbf{c}_{j}^{T}\mathbf{W}\mathbf{d}_{i})}),$

where $\sigma$ stands for the sigmoid function and $\mathbf{W}$ is a learnable
interaction matrix.

Training process. For each document $d_{i}$ tagged with core classes
$\mathbb{C}_{i}$, we construct its positive classes
$\mathbb{C}^{\text{core}}_{i,+}$ as the union of its core classes and their
ancestors in the label taxonomy, and its negative classes
$\mathbb{C}^{\text{core}}_{i,-}$ are the ones that are not positive classes or
descendants of any core class. This is because the ancestors of confident core
classes are also likely to be true labels, and the descendants may not all be
negative given that the automatically generated core classes are not optimal.
Formally,

$\displaystyle\small\mathbb{C}^{\text{core}}_{i,+}$
$\displaystyle=\mathbb{C}_{i}\cup\left(\cup_{c\in\mathbb{C}_{i}}Anc(c)\right),$
$\displaystyle\mathbb{C}^{\text{core}}_{i,-}$
$\displaystyle=\mathcal{C}-\mathbb{C}^{\text{core}}_{i,+}-\cup_{c\in\mathbb{C}_{i}}Des(c),$

where $Anc(c)$ and $Des(c)$ denote the set of ancestors and descendants and
class $c$, respectively. For the LLM-generated documents, we are more
confident in its pseudo labels, so we simply treat all the classes in the
corresponding path as positive classes and all other classes as negative,

$\small\mathbb{C}^{\text{gen}}_{p,+}=\mathbb{C}_{p},\quad\mathbb{C}^{\text{gen}}_{p,-}=\mathcal{C}-\mathbb{C}_{p}.$

Then, we train a multi-label classifier with the binary cross entropy loss
(BCE):

$\displaystyle\mathcal{L}^{\text{core}}=-\sum_{d_{i}\in\mathcal{D}^{\text{core}}}\left(\sum_{c_{j}\in\mathbb{C}^{\text{core}}_{i,+}}\log{p(c_{j}|d_{i})}+\sum_{c_{j}\in\mathbb{C}^{\text{core}}_{i,-}}\log{(1-p(c_{j}|d_{i}))}\right)$
$\displaystyle\mathcal{L}^{\text{gen}}=-\sum_{d^{p}_{i}\in\mathcal{D}^{\text{gen}}}\left(\sum_{c_{j}\in\mathbb{C}^{\text{gen}}_{p,+}}\log{p(c_{j}|d^{p}_{i})}+\sum_{c_{j}\in\mathbb{C}^{\text{gen}}_{p,-}}\log{(1-p(c_{j}|d^{p}_{i}))}\right)$
(7)
$\small\mathcal{L}=\mathcal{L}^{\text{core}}+\frac{|\mathcal{D}^{\text{core}}|}{|\mathcal{D}^{\text{gen}}|}\cdot\mathcal{L}^{\text{gen}},$

where $\frac{|\mathcal{D}^{\text{core}}|}{|\mathcal{D}^{\text{gen}}|}$ serves
as a scaling factor to balance two sets of pseudo-data and can be exactly
calculated as $\frac{|\mathcal{D}|}{2q\cdot|Path(\mathcal{T})|}$.

We train the text-matching model with $\mathcal{L}$ to get the final
classifier. Notice that, we do not continue to train the classifier with self-
training that is commonly used in previous studies (Meng et al., 2019; Shen et
al., 2021), because we want to keep the classifier training part simple. Using
self-training may further improve the model performance, which we leave for
future exploration. Algorithm 1 summarizes TELEClass.

Input: A corpus $\mathcal{D}$, a label taxonomy $\mathcal{T}$, a textual
entailment model $\mathcal{M}$, a sentence encoder $\mathcal{S}$, an LLM
$\mathcal{G}$.

Output: A text classifier $F$ that can classify each document into a set of
classes in $\mathcal{T}$.

1 // LLM-Enhanced Core Class Annotation;

2 for  _$d_{i}\in\mathcal{D}$_ do

3 $\mathbb{C}^{0}_{i}\leftarrow$ use the LLM $\mathcal{G}$ to select core
classes from candidates retrieved by $\mathcal{M}$;

4

5// Corpus-Based Taxonomy Enrichment;

6 for  _$c\in\mathcal{C}$_ do

7 $D^{0}_{c}\leftarrow$ a set of roughly classified documents (Eq. 1);

8 for  _$c_{p}\in Par(c)$_ do

9 $T(c,c_{p})\leftarrow$ top terms ranked by affinity (Eq. 4);

10

11 $T_{c}\leftarrow$ aggregate class-indicative terms with Eq. 5;

12

13// Core Class Refinement with Enriched Taxonomy;

14 $\vec{d}\leftarrow$ document representation $\mathcal{S}(d)$;

15 for  _$c\in\mathcal{C}$_ do

16 $D_{c}\leftarrow$ confident documents by matching $T_{c}$;

17 $\vec{c}\leftarrow$ average document representation in $D_{c}$;

18

19for  _$d_{i}\in\mathcal{D}$_ do

20 $\mathbb{C}_{i},conf_{i}\leftarrow$ refined core classes using
$\cos(\vec{d},\vec{c})$ and Eq. 6;

21

22$\mathcal{D}^{\text{core}}\leftarrow$ confident refined core classes;

23 // Text Classifier Training with Path-Based Data Augmentation;

24 $\mathcal{D}^{\text{gen}}\leftarrow$ generate $q$ documents for each path
using $\mathcal{G}$;

25 $F\leftarrow$ train classifier with $\mathcal{D}^{\text{core}}$,
$\mathcal{D}^{\text{gen}}$, and Eq. 7;

26 Return $F$;

Algorithm 1 TELEClass Table 1. Experiment results on Amazon-531 and
DBPedia-298 datasets, evaluated by Example-F1, P@k, and MRR. The best score
among zero-shot and weakly-supervised methods is boldfaced. “†” indicates the
numbers for these baselines are directly from previous paper (Shen et al.,
2021). “—” means the method cannot generate a ranking of predictions and thus
MRR cannot be calculated.

Supervision Type | Methods | Amazon-531 | DBPedia-298
---|---|---|---
Example-F1 | P@1 | P@3 | MRR | Example-F1 | P@1 | P@3 | MRR
Zero-Shot | Hier-0Shot-TC† | 0.4742 | 0.7144 | 0.4610 | — | 0.6765 | 0.7871 | 0.6765 | —
GPT-3.5-turbo | 0.5164 | 0.6807 | 0.4752 | — | 0.4816 | 0.5328 | 0.4547 | —
Weakly-Supervised | Hier-doc2vec† | 0.3157 | 0.5805 | 0.3115 | — | 0.1443 | 0.2635 | 0.1443 | —
WeSHClass† | 0.2458 | 0.5773 | 0.2517 | — | 0.3047 | 0.5359 | 0.3048 | —
TaxoClass-NoST† | 0.5431 | 0.7918 | 0.5414 | 0.5911 | 0.7712 | 0.8621 | 0.7712 | 0.8221
TaxoClass† | 0.5934 | 0.8120 | 0.5894 | 0.6332 | 0.8156 | 0.8942 | 0.8156 | 0.8762
TELEClass | 0.6330 | 0.8439 | 0.6269 | 0.6664 | 0.8684 | 0.9293 | 0.8684 | 0.8880
Fully-Supervised | 0.8843 | 0.9524 | 0.8758 | 0.9085 | 0.9786 | 0.9945 | 0.9786 | 0.9826

##  4\. Experiments

###  4.1. Experiment Setup

####  4.1.1. Datasets

We use two public datasets in different domains for evaluation. (1) Amazon-531
(McAuley and Leskovec, 2013) consists of Amazon product reviews and a three-
layer label taxonomy of 531 product types, with 29,487 unlabeled training
samples and 19,685 testing samples. (2) DBpedia-298 (Lehmann et al., 2015)
consists of Wikipedia articles with a three-layer label taxonomy of 298
categories, with 196,665 unlabeled training samples and 49,167 testing
samples.

####  4.1.2. Compared Methods

We compare the following methods on the weakly-supervised hierarchical text
classification task.

  * •

Hier-0Shot-TC (Yin et al., 2019) is a zero-shot approach, which utilizes a
pretrained textual entailment model to iterative find the most similar class
at each level for a document.

  * •

GPT-3.5-turbo is a zero-shot approach that queries GPT-3.5-turbo by directly
providing all classes in the prompt.

  * •

Hier-doc2vec (Le and Mikolov, 2014) is a weakly-supervised approach, which
first trains document and class representations in the same embedding space,
and then iteratively selects the most similar class at each level.

  * •

WeSHClass (Meng et al., 2019) is a weakly-supervised approach using a set of
keywords for each class. It first generates pseudo documents to pretrain text
classifiers and then performs self-training.

  * •

TaxoClass and its variant TaxoClass-NoST (Shen et al., 2021) is a weakly-
supervised approach that only uses the class name of each class. It first uses
a textual entailment model with a top-down search and corpus-level comparison
to select core classes, which are then used as pseudo-training data. We
include both its full model, TaxoClass, and its variation TaxoClass-NoST that
does not apply self-training on the trained classifier, which is the same as
ours.

  * •

TELEClass is our newly proposed weakly-supervised approach that only uses the
class name for each class.

  * •

Fully-Supervised is a fully-supervised baseline that uses the entire labeled
training data to train the text-matching network used in TELEClass.

####  4.1.3. Evaluation Metrics

Following previous studies (Shen et al., 2021), we utilize the following
evaluation metrics: Example-F1, Precision at k (P@k) for $k=1,3$, and Mean
Reciprocal Rank (MRR). See Appendix A for definitions of these metrics.

####  4.1.4. Implementation Details

We use RoBERTa-large-MNLI as the textual entailment model for core class
candidate search. We query GPT-3.5-turbo-0613 for both core class annotation
and path-based generation. For taxonomy enrichment, we get the term and class
name embeddings using a pre-trained BERT-base-uncased. The weight parameter
$\alpha$ is set to 0.2, and we select top $k=20$ enriched terms for each class
(c.f. Section 3.2). For core class refinement, we get the document and class
embeddings using Sentence Transformer all-mpnet-base-v2 (c.f. Section 3.3). We
generate $q=5$ documents with path-based generation for each class. The
document encoder in the final classifier is initialized with BERT-base-
uncased. We train the classifier using AdamW optimizer with a learning rate
5e-5, and the batch size is 64. We include prompts we used for LLM annotation
and generation in Appendix B.

###  4.2. Experimental Results

Table 1 shows the evaluation results of all the compared methods. We make the
following observations. (1) Overall, TELEClass achieves the best performance
among the compared strong zero-shot and weakly-supervised baselines. (2) By
comparing with other weakly-supervised methods, we find that TELEClass
significantly outperforms TaxoClass-ST, which is the strongest baseline that
also does not use self-training as TELEClass. This further demonstrates the
effectiveness of TELEClass. (3) Although LLM (e.g., GPT-3.5-turbo) show
superior power in many tasks, naïvely prompting GPT-3.5-turbo in the
hierarchical text classification task does not yield promising performance and
underperforms strong weakly-supervised text classifier by a large margin. We
will conduct a more detailed comparison of LLM prompting for hierarchical text
classification in Section 4.4.

Table 2. Performance of TELEClass and its ablations on Amazon-531 and
DBPedia-298 datasets. The best score is boldfaced.

Methods | Amazon-531 | DBPedia-298
---|---|---
Example-F1 | P@1 | P@3 | MRR | Example-F1 | P@1 | P@3 | MRR
Gen-Only | 0.5151 | 0.7477 | 0.5096 | 0.5357 | 0.7930 | 0.9421 | 0.7930 | 0.8209
TELEClass-NoEnrich | 0.6324 | 0.8375 | 0.6263 | 0.6730 | 0.6868 | 0.7726 | 0.6868 | 0.7182
TELEClass-NoGen | 0.6084 | 0.8238 | 0.6024 | 0.6361 | 0.8364 | 0.9146 | 0.8364 | 0.8599
TELEClass | 0.6330 | 0.8439 | 0.6269 | 0.6664 | 0.8684 | 0.9293 | 0.8684 | 0.8880

Table 3. Performance comparison of zero-shot LLM prompting. We only report
Example-F1 and P@k, because it is not straightforward to get ranking of
classes predicted by LLMs for MRR calculation. We also report estimated cost
in US dollar (as of the pricing in Feb 2024) and running time in minutes for
each method on the entire test set. “‡” indicates that we report the
performance on a 1,000-document subset of test data.

Methods | Amazon-531 | DBPedia-298
---|---|---
Example-F1 | P@1 | P@3 | Est. Cost | Est. Time | Example-F1 | P@1 | P@3 | Est. Cost | Est. Time
GPT-3.5-turbo | 0.5164 | 0.6807 | 0.4752 | $60 | 240 mins | 0.4816 | 0.5328 | 0.4547 | $80 | 400 mins
GPT-3.5-turbo (level) | 0.6621 | 0.8574 | 0.6444 | $20 | 800 mins | 0.6649 | 0.8301 | 0.6488 | $60 | 1,000 mins
GPT-4‡ | 0.6994 | 0.8220 | 0.6890 | $800 | 400 mins | 0.6054 | 0.6520 | 0.5920 | $2,500 | 1,000 mins
TELEClass | 0.6330 | 0.8439 | 0.6269 | <$1 | 3 mins | 0.8684 | 0.9293 | 0.8684 | <$1 | 7 mins

###  4.3. Ablation Studies

To better understand if each component of TELEClass contributes to the entire
method, we additionally conduct ablation studies by removing part of TELEClass
and reporting the performance. Table 2 shows the results of the following
ablations of TELEClass:

  * •

Gen-Only only uses the augmented documents produced by path-based LLM
generation to train the final classifier.

  * •

TELEClass-NoEnrich does not perform taxonomy enrichment, so directly uses the
initial core classes annotated by LLM plus augmented documents to train the
final classifier.

  * •

TELEClass-NoGen does not use the augmented documents produced by path-based
LLM generation, so only uses the refined core classes by taxonomy enrichment
and embedding-based matching to train the final classifier.

We find that the full model TELEClass achieves the overall best performance
among the compared methods, showing the effectiveness of each of its
components. Interestingly, we find that taxonomy enrichment and path-based
data generation make different levels of contribution on two data sets: path-
based data generation brings more improvement on Amazon-531, while taxonomy
enrichment contributes more on DBpedia-298. The reasons could be as follows.
Amazon-531 has a large label space and more skewed class distribution. We find
that around 11.6% of classes are not selected as core classes in Amazon, while
this number for DBpedia is 5.4%. This necessitates data augmentation to
provide training data that covers each class in the taxonomy. On the other
hand, although DBpedia has a smaller label space, its class names are more
subtle to distinguish, which can also be demonstrated by the lower performance
of zero-shot LLM prompting on DBpedia compared to Amazon-531 (c.f.
GPT-3.5-turbo in Table 1). Therefore, by mining class-indicative terms,
TELEClass can enrich each class with more clues to facilitate better
classification.

###  4.4. Comparison with Zero-Shot LLM Prompting

In this section, we further compare TELEClass with zero-shot LLM prompting.
Because it is not straightforward to get ranked predictions by LLMs, we only
report Example-F1 and P@k as performance evaluations. Additionally, we report
the estimated cost and time for each method on the entire test set. The cost
is reported in US dollar based on the pricing in February 2024. The inference
time is reported in minutes, and please be aware that this is just a rough
estimation as the actual running time is also dependent on the server
condition.

We include the following settings to compare with TELEClass:

  * •

GPT-3.5-turbo: We include all the classes in the prompt and ask GPT-3.5-turbo
model to provide 3 most appropriate classes for a given document.

  * •

GPT-3.5-turbo (level): We perform level-by-level prompting using
GPT-3.5-turbo. Starting from the root node, we ask the model to return one
most appropriate class for a given document, and we iteratively prompt the
model with the child nodes of the selected node at each level. Notice that
this method can only generate a path in the taxonomy, but in the actual multi-
label hierarchical classification setting, the true labels may not sit in the
same path in the label taxonomy.

  * •

GPT-4: Similar to the first one, we include all the classes in the prompt and
ask GPT-4 to provide 3 most appropriate classes for a given document. Given
the limited budget, we only test it on a sampled subset of the test split,
which contains 1,000 documents for each dataset.

Table 3 shows the experiment results. We find that TELEClass consistent
outperforms all compared methods on DBpedia, while on Amazon, TELEClass
underperforms GPT-3.5-turbo (level) and GPT-4 but still being comparable. As
for the cost, once trained, TELEClass does not require an additional cost on
inference and also has substantially shorter inference time. Prompting LLMs
takes longer time and can be prohibitively expensive (e.g., using GPT-4), and
the cost will scale up with an increasing size of test data. Also, we find
that the level-by-level version of GPT-3.5-turbo consistently outperforms the
naïve version, demonstrating the necessity of taxonomy structure. It saves the
cost because of the much shorter prompts, but takes longer time due to more
queries made per document.

Table 4. Intermediate results on two documents, including selected core
classes by different methods, true labels, and the corresponding taxonomy
enrichment results. The optimal core class in the true labels is marked with
©.

Dataset | Document | Core Classes by… | True Labels | Corr. Enrichment
---|---|---|---|---
DBPedia |  The Lindenhurst Memorial Library (LML) is located in Lindenhurst, New York, and is one of the fifty six libraries that are part of the Suffolk Cooperative Library System … |  TaxoClass: village TELEClass initial: building TELEClass refined: library |  library©, agent, educational institution |  Class: library Top Enrichment: national library, central library, collection, volumes…
Amazon |  Since mom (89 yrs young) isn’t steady on her feet, we have place these grab bars around the room. It gives her the stability and security she needs. |  TaxoClass: personal care, health personal care, safety TELEClass initial: daily living aids, medical supplies equipment, safety, TELEClass refined: bathroom aids safety |  health personal care, medical supplies equipment, bathroom aids safety© |  Class: bathroom aids safety Top Enrichment: seat, toilet, shower, safety, handles…

###  4.5. Case studies

To better understand the TELEClass framework, we show some intermediate
results of two documents in Table 4, including the core classes selected by
(1) the original TaxoClass (Shen et al., 2021) method, (2) TELEClass’s initial
core classes selected by LLM, and (3) refined core classes of TELEClass.
Besides, we also include the true labels of the documents and the taxonomy
enrichment results of the corresponding core class in the table. Overall, we
can see that TELEClass’s refined core class is the most accurate. For example,
for the first Wikipedia article about a library, TaxoClass selects “village”
as the core class, while TELEClass’s initial core class finds a closer one
“building” thanks to the power of LLMs. Then, with the enriched classes-
indicative features as guidance, TELEClass’s refined core class correctly
identifies the optimal core class, which is “library”. In the other example,
TELEClass also pinpoints the most accurate core class “bathroom aids safety”
while other methods can only find more general or partially relevant classes.

##  5\. Related Work

###  5.1. Weakly-Supervised Text Classification

Weakly-supervised text classification trains a classifier with limited
guidance, aiming to reduce human efforts while maintaining high proficiency.
Various sources of weak supervision have been explored, including distant
supervision (Gabrilovich and Markovitch, 2007; Chang et al., 2008; Song and
Roth, 2014) like knowledge bases, keywords (Agichtein and Gravano, 2000; Tao
et al., 2018; Meng et al., 2018, 2020; Wang et al., 2021a; Zhang et al.,
2021b), and heuristic rules (Ratner et al., 2016; Badene et al., 2019; Shu et
al., 2020). Later, extremely weakly-supervised methods are proposed to solely
rely on class label names for generating pseudo-labels and training
classifiers. LOTClass (Meng et al., 2020) utilizes MLM-based PLM as a
knowledge base for extracting class-indicative keywords. XClass (Wang et al.,
2021a) extracts keywords for creating static class representations through
clustering. PIEClass (Zhang et al., 2023a) employs PLMs’ zero-shot prompting
to obtain pseudo labels with noise-robust iterative ensemble training.
MEGClass (Kargupta et al., 2023) acquires contextualized sentence
representations to capture topical information at the document level. WOTClass
(Wang et al., 2023) focuses on mining and ranking class-indicative keywords
from generated classes and extracting classes with overlapping keywords.

###  5.2. Hierarchical Text Classification

A hierarchy provides a systematic top-to-down structure with inherent semantic
relations that can assist in text classification. Typical hierarchical text
classification can be categorized into two groups: local approaches and global
approaches. Local approaches train multiple classifiers for each node or local
structures (Banerjee et al., 2019; Liu et al., 2005; Wehrmann et al., 2018).
Global approaches, learn hierarchy structure into a single classifier through
recursive regularization (Gopal and Yang, 2013), a graph neural network
(GNN)-based encoder (Peng et al., 2018; Huang et al., 2019; Zhou et al., 2020;
Shen et al., 2021), or a joint document label embedding space (Chen et al.,
2021).

Weak supervision is also studied for hierarchical text classification.
WeSHClass (Meng et al., 2019) uses a few keywords or example documents per
class and pretrains classifiers with pseudo documents followed by self-
training. TaxoClass (Shen et al., 2021) follows the same setting as ours which
uses the sole class name of each class as the only supervision. It identifies
core classes for each document using a textual entailment model, which is then
used to train a multi-label classifier. Additionally, MATCH (Zhang et al.,
2021c) and HiMeCat (Zhang et al., 2021a) study how to integrate associated
metadata into the label hierarchy for document categorization with weak
supervision.

###  5.3. LLMs as Generators and Annotators

Large language models (LLMs) have demonstrated impressive performance in many
downstream tasks and are explored to help low-resource settings by
synthesizing data as generators or annotators (Ding et al., 2023). For data
generation, few-shot examples (Wang et al., 2021b) or class-conditioned
prompts (Meng et al., 2022) are explored for LLM generation and the generated
data can be used as pseudo-training data to further fine-tune a small model as
the final classifier (Ye et al., 2022). Recently, Yu et al. (2023) propose an
attribute-aware topical text classification method that incorporates ChatGPT
to generate topic-dependent attributes and topic-independent attributes to
reduce topic ambiguity and increase topic diversity for generation. For data
annotation, previous works utilize LLMs for unsupervised annotation (Feng et
al., 2021), Chain-of-Thought annotation with explanation generation (He et
al., 2023), and active annotation (Zhang et al., 2023b).

##  6\. Conclusion and Future Work

In this paper, we propose a new method, TELEClass, for the minimally-
supervised hierarchical text classification task with two major contributions.
First, we enrich the input label taxonomy with more class-indicative topical
terms for each class, which can serve as additional features to understand the
classes and facilitate classification. Second, we explore the utilization of
LLMs in the hierarchical text classification in two directions: data
annotation and data creation. On two public datasets, TELEClass can outperform
existing baseline methods substantially, and we further demonstrate its
effectiveness through ablation studies. We also conduct a comparative analysis
on performance and cost for zero-shot LLM prompting for the hierarchical text
classification task. For future works, we plan to generalize TELEClass’s idea
into other low-resource text mining tasks with hierarchical label spaces, such
as fine-grained entity typing. We also plan to explore how to extend TELEClass
into a more complicated label space, such as an extremely large space with
millions of labels.

## References

  * (1)
  * Agichtein and Gravano (2000) Eugene Agichtein and Luis Gravano. 2000.  Snowball: extracting relations from large plain-text collections. In _Digital library_.
  * Badene et al. (2019) Sonia Badene, Kate Thompson, Jean-Pierre Lorré, and Nicholas Asher. 2019.  Data Programming for Learning Discourse Structure. In _ACL_.
  * Banerjee et al. (2019) Siddhartha Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, and Kostas Tsioutsiouliklis. 2019.  Hierarchical Transfer Learning for Multi-label Text Classification. In _ACL_.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver, Nicolas Papernot, and Colin Raffel. 2019.  MixMatch: a holistic approach to semi-supervised learning. In _NeurIPS_.
  * Chang et al. (2008) Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008.  Importance of Semantic Representation: Dataless Classification. In _AAAI_.
  * Chen et al. (2021) Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021.  Hierarchy-aware Label Semantics Matching Network for Hierarchical Text Classification. In _ACL-IJCNLP_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT_.
  * Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023.  Is GPT-3 a Good Data Annotator?. In _ACL_.
  * Dong et al. (2023) Chengyu Dong, Zihan Wang, and Jingbo Shang. 2023.  Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification. In _EMNLP_.
  * Feng et al. (2021) Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021.  Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization. In _ACL-IJCNLP_.
  * Gabrilovich and Markovitch (2007) Evgeniy Gabrilovich and Shaul Markovitch. 2007.  Computing Semantic Relatedness Using Wikipedia-Based Explicit Semantic Analysis. In _IJCAI_.
  * Gopal and Yang (2013) Siddharth Gopal and Yiming Yang. 2013.  Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In _KDD_.
  * Gururangan et al. (2019) Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019.  Variational Pretraining for Semi-supervised Text Classification. In _ACL_.
  * He et al. (2023) Xingwei He, Zheng-Wen Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023.  AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators.  _ArXiv_ abs/2303.16854 (2023).
  * Huang et al. (2019) Wei Huang, Enhong Chen, Qi Liu, Yuying Chen, Zai Huang, Yang Liu, Zhou Zhao, Dan Zhang, and Shijin Wang. 2019.  Hierarchical Multi-label Text Classification: An Attention-based Recurrent Network Approach. In _CIKM_.
  * Kargupta et al. (2023) Priyanka Kargupta, Tanay Komarlu, Susik Yoon, Xuan Wang, and Jiawei Han. 2023.  MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities. In _Findings of EMNLP_.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed Representations of Sentences and Documents. In _ICML_.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, S. Auer, and Christian Bizer. 2015.  DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia.  _Semantic Web_ 6 (2015), 167–195.
  * Liu et al. (2005) Tieyan Liu, Yiming Yang, Hao Wan, Huajun Zeng, Zheng Chen, and Weiying Ma. 2005.  Support vector machines classification with a very large-scale taxonomy. In _KDD_.
  * McAuley and Leskovec (2013) Julian McAuley and Jure Leskovec. 2013.  Hidden factors and hidden topics: understanding rating dimensions with review text. In _RecSys_.
  * Meng et al. (2022) Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022.  Generating Training Data with Language Models: Towards Zero-Shot Language Understanding. In _NeurIPS_.
  * Meng et al. (2018) Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018.  Weakly-Supervised Neural Text Classification. In _CIKM_.
  * Meng et al. (2019) Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2019.  Weakly-supervised hierarchical text classification. In _AAAI_.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020.  Text Classification Using Label Names Only: A Language Model Self-Training Approach. In _EMNLP_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _ArXiv_ abs/2303.08774 (2023).
  * Peng et al. (2018) Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Yangqiu Song, and Qiang Yang. 2018.  Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN. In _WWW_.
  * Ratner et al. (2016) Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016.  Data Programming: Creating Large Training Sets, Quickly. In _NIPS_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _EMNLP_.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Found. Trends Inf. Retr._ 3, 4 (apr 2009), 333–389.
  * Shang et al. (2018) Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R. Voss, and Jiawei Han. 2018.  Automated Phrase Mining from Massive Text Corpora.  _TKDE_ 30, 10 (2018), 1825–1837.
  * Shen et al. (2021) Jiaming Shen, Wenda Qiu, Yu Meng, Jingbo Shang, Xiang Ren, and Jiawei Han. 2021.  TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names. In _NAACL_.
  * Shu et al. (2020) Kai Shu, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan Awadallah, Milad Shokouhi, and Susan Dumais. 2020.  Learning with Weak Supervision for Email Intent Detection. In _SIGIR_.
  * Song and Roth (2014) Yangqiu Song and Dan Roth. 2014.  On Dataless Hierarchical Text Classification. In _AAAI_.
  * Sørensen (1948) T. Sørensen. 1948.  _A Method of Establishing Groups of Equal Amplitude in Plant Sociology Based on Similarity of Species Content and Its Application to Analyses of the Vegetation on Danish Commons_.  Munksgaard in Komm.
  * Sun et al. (2023) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023.  Text Classification via Large Language Models. In _Findings of EMNLP_.
  * Tao et al. (2018) Fangbo Tao, Chao Zhang, Xiusi Chen, Meng Jiang, Tim Hanratty, Lance Kaplan, and Jiawei Han. 2018.  Doc2Cube: Allocating Documents to Text Cube Without Labeled Data. In _ICDM_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _ArXiv_ abs/2307.09288 (2023).
  * Wang et al. (2023) Tianle Wang, Zihan Wang, Weitang Liu, and Jingbo Shang. 2023.  WOT-Class: Weakly Supervised Open-world Text Classification. In _CIKM_.
  * Wang et al. (2021a) Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021a.  X-Class: Text Classification with Extremely Weak Supervision. In _NAACL_.
  * Wang et al. (2021b) Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021b.  Towards Zero-Label Language Learning.  _ArXiv_ abs/2109.09193 (2021).
  * Wehrmann et al. (2018) Jonatas Wehrmann, Ricardo Cerri, and Rodrigo C. Barros. 2018.  Hierarchical Multi-label Classification Networks. In _ICML_.
  * Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022.  ZeroGen: Efficient Zero-shot Learning via Dataset Generation. In _EMNLP_.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. In _EMNLP-IJCNLP_.
  * You et al. (2019) Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019.  AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification. In _NeurIPS_.
  * Yu et al. (2023) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023.  Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. In _NeurIPS_.
  * Zhang et al. (2021b) Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021b.  Weakly-supervised Text Classification Based on Keyword Graph. In _EMNLP_.
  * Zhang et al. (2023b) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023b.  LLMaAA: Making Large Language Models as Active Annotators. In _Findings of EMNLP_.
  * Zhang et al. (2021a) Yu Zhang, Xiusi Chen, Yu Meng, and Jiawei Han. 2021a.  Hierarchical Metadata-Aware Document Categorization under Weak Supervision. In _WSDM_.
  * Zhang et al. (2023a) Yunyi Zhang, Minhao Jiang, Yu Meng, Yu Zhang, and Jiawei Han. 2023a.  PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training. In _EMNLP_.
  * Zhang et al. (2021c) Yu Zhang, Zhihong Shen, Yuxiao Dong, Kuansan Wang, and Jiawei Han. 2021c.  MATCH: Metadata-Aware Text Classification in A Large Hierarchy. In _WWW_.
  * Zhou et al. (2020) Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gongshen Liu. 2020.  Hierarchy-Aware Global Model for Hierarchical Text Classification. In _ACL_.

##  Appendix A Evaluation Metrics

Let $\mathbb{C}^{\text{true}}_{i}$ and $\mathbb{C}^{\text{pred}}_{i}$ denote
the set of true labels and the set of predicted labels for document
$d_{i}\in\mathcal{D}$, respectively. If a method can generate rankings of
classes within $\mathbb{C}^{\text{pred}}_{i}$, we further denote its top-k
predicted labels as
$\mathbb{C}^{\text{pred}}_{i,k}\subset\mathbb{C}^{\text{pred}}_{i}$. The
evaluation metrics are defined as follows:

  * •

Example-F1 (Sørensen, 1948), which is also called micro-Dice coefficient,
evaluates the multi-label classification results without ranking,

(8)
$\text{Example-F1}=\frac{1}{|\mathcal{D}|}\sum_{d_{i}\in\mathcal{D}}\frac{2\cdot|\mathbb{C}_{i}^{\text{true}}\cap\mathbb{C}_{i}^{\text{pred}}|}{|\mathbb{C}_{i}^{\text{true}}|+|\mathbb{C}_{i}^{\text{pred}}|}.$

  * •

Precision at k, or P@k, is a ranking-based metric that evaluates the precision
of top-$k$ predicted classes,

(9)
$\text{P@k}=\frac{1}{k}\sum_{d_{i}\in\mathcal{D}}\frac{|\mathbb{C}_{i}^{\text{true}}\cap\mathbb{C}^{\text{pred}}_{i,k}|}{\min(k,|\mathbb{C}_{i}^{\text{true}}|)}.$

  * •

Mean Reciprocal Rank, or MRR, is another ranking-based metric, which evaluates
the multi-label predictions based on the inverse of true labels’ ranks within
predicted classes,

(10)
$\text{MRR}=\frac{1}{|\mathcal{D}|}\sum_{d_{i}\in\mathcal{D}}\frac{1}{|\mathbb{C}_{i}^{\text{true}}|}\sum_{c_{j}\in\mathbb{C}_{i}^{\text{true}}}\frac{1}{\min\\{k|c_{j}\in\mathbb{C}^{\text{pred}}_{i,k}\\}}.$

##  Appendix B Prompts for LLM

  * •

Core class annotation for Amazon-531

Instruction: You will be provided with an Amazon product review, and your task
is to select its product types from the following categories: [Candidate
Classes]. Just give the category names as shown in the provided list. Query:
[Document]

  * •

Path-based generation for Amazon-531

Instruction: Suppose you are an Amazon Reviewer, please generate 5 various and
reliable passages following the requirements below: 1\. Must generate reviews
following the themes of the taxonomy path: [Path]. 2\. Must be in length about
100 words. 3\. The writing style and format of the text should be a product
review. 4\. Should keep the generated text to be diverse, specific, and
consistent with the given taxonomy path. You should focus on [The Leaf Node on
the Path].

  * •

Core class annotation for DBPedia-298

Instruction: You will be provided with a Wikipedia article describing an
entity at the beginning, and your task is to select its types from the
following categories: [Candidate Classes]. Just give the category names as
shown in the provided list. Query: [Document]

  * •

Path-based generation for DBPedia-298

Instruction: Suppose you are a Wikipedia Contributor, please generate 5
various and reliable passages following the requirements below: 1\. Must
generate reviews following the themes of the taxonomy path: [Path]. 2\. Must
be in length about 100 words. 3\. The writing style and format of the text
should be a Wikipedia page. 4\. Should keep the generated text to be diverse,
specific, and consistent with the given taxonomy path. You should focus on
[The Leaf Node on the Path].