# Label-Guided Prompt for Multi-label Few-shot Aspect Category Detection

ChaoFeng Guan1 YaoHui Zhu2 Yu Bai3&LingYun Wang4 1,3,4Shenyang Aerospace
University
2Beijing Normal University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multi-label few-shot aspect category detection aims at identifying multiple
aspect categories from sentences with a limited number of training instances.
The representation of sentences and categories is a key issue in this task.
Most of current methods extract keywords for the sentence representations and
the category representations. Sentences often contain many category-
independent words, which leads to suboptimal performance of keyword-based
methods. Instead of directly extracting keywords, we propose a label-guided
prompt method to represent sentences and categories. To be specific, we design
label-specific prompts to represent sentences by combining crucial contextual
and semantic information. Further, the label is introduced into a prompt to
obtain category descriptions by utilizing a large language model. This kind of
category descriptions contain the characteristics of the aspect categories,
guiding the construction of discriminative category prototypes. Experimental
results on two public datasets show that our method outperforms current state-
of-the-art methods with a 3.86%$\sim$4.75% improvement in the Macro-F1 score.

##  1 Introduction

Multi-label Aspect Category Detection aims at detecting multiple aspects of
viewpoints mentioned in a sentence, which is an important subtask of aspect-
based sentiment analysis Pontiki et al. (2016). It is difficult to obtain a
sufficient number of annotated comment sentences, for example, more expensive
items usually have only a small number of comments. As a result, some studies
focus on mining multiple aspects of viewpoints with a few comments, named
Multi-label Few-shot Aspect Category Detection (MFACD) Hu et al. (2021), which
has received increasing attention from researchers in the field of artificial
intelligence.

Existing MFACD methods usually use a meta-learning strategy on a large amount
of base categories to acquire transferable knowledge, which contributes to
learn effective representations of category prototypes from a small number of
comment sentences. Then these category prototypes are used to detect aspect
categories of test comment sentences with similarities between test sentences
and category prototypes. In the MFACD task, single comment sentence usually
contains several different aspect categories at the same time (see sentence in
Table 1). Since the category prototype generation is based on the sentence
representations, the multiple categories of a sentence result in inadequate
differentiation among category prototypes.

To achieve representative category prototypes, Hu et al. (2021) extracted
category-relevant keywords for sentence representation using an attention
mechanism. Nevertheless, there are a large number of category-irrelevant words
in comment sentences Zhao et al. (2022) such as “a”, “the”, “my”. These large
numbers of category-irrelevant words make the method based on automatic
keyword extraction ineffective, since the method possibly tends to pay
attention to those irrelevant words. In order to effectively capture category-
relevant keywords, Zhao et al. (2023, 2022); Wang and Iwaihara (2023) utilize
category labels to assist keyword extraction. However, there are similarities
between category labels, resulting in the keywords filtered with the label are
not clearly distinguishable, thus affecting the learning of discriminative
category prototypes.

Aspect Category |  Sentences

---|---
(A) ambiance (B) service (C) food | 1\. The cozy environment, efficient service,
and tasty pizza made our night delightful.
2\. Enjoyed the modern decor, quick staff response,
and delicious burgers.
(G) price (H) location (I) cleanliness | 3\. The competitive prices, prime location in the city center,
and the impeccable cleanliness of the rooms were impressive.
4\. Guests appreciated the fair room rates, spotless environment,
and the easy access to local attractions.

Table 1: Multiple aspects of the sample are linked together, and different
colors mean different categories.

Although the above approaches based on keyword representations demonstrate
some effectiveness, it is still difficult to explicitly obtain keywords
characterizing category prototypes from sentences. Compared to directly
extracting keywords for sentence representations, we argue that prompt
learning in the large language mode (LLM) is able to obtain semantic
information of category from sentences Jiang et al. (2022a). Furthermore,
introducing the label information in the prompt can better capture the
viewpoint categories of sentences.

Based on the above analysis, we propose a label-guided prompt (LGP) method for
MFACD by enhancing sentence representations and their category prototypes. At
the level of the sentence representation, the knowledge of the pre-training
model is exploited via a prompt, and further the category label is introduced
into the prompt to accurately express sentence category information. At the
level of category prototype generation, we utilize another kind of prompt for
a LLM to generate category descriptions, which contain the characteristics of
the aspect categories, guiding the construction of discriminative category
prototypes.

The contributions are summarized below:

  * •

We propose a label-guided prompt (LGP) method to enhance the representations
of sentences for MFACD.

  * •

We leverage large language model to obtain category descriptions, which can
effectively guide discriminative category prototype generation.

  * •

Experimental results on two public datasets show that the proposed method
achieves state-of-the-art performance.

##  2 Related Work

###  2.1 Multi-label Few-shot Aspect Category Detection

Previous research of aspect category detection can be divided into two
categories, unsupervised methods Schouten et al. (2017); Su et al. (2021); Hai
et al. (2011) and supervised methods Jiang et al. (2019); Movahedi et al.
(2019); Ghadery et al. (2019). These methods detect aspect categories from
predefined sets, heavily relying on large amounts of labeled data to train a
discriminative classifier, which fails to generalize well to novel aspect
categories with only a few labeled. To this end, some researches begin to
explore few-shot learning, which allows for rapid adaptation to new categories
by only utilizing a small number of samples. Few-shot learning aims to address
the challenges of limited data and data sparsity, and this task has received
extensive attention in many fields, such as text classification Rios and
Kavuluru (2018); Chalkidis et al. (2019), image recognition Zhu et al. (2020b,
a), intent detection Hou et al. (2021), and relationship classification Gao et
al. (2019).

Meanwhile, the work Proto-AWATT Hu et al. (2021) first attempted to address
aspect category detection in multi-label few-shot scenarios. This approach
based prototypical network utilizes the attention mechanism to alleviate the
noise from irrelevant aspects. In order to filter irrelevant words more
efficiently, Zhao et al. (2022); Wang and Iwaihara (2023) proposed a label-
guided attention strategy. However, the presence of semantically similar
category labels may lead to ineffective filtering of irrelevant words. In this
paper, instead of directly extracting keywords for sentence representations,
we propose a label-guided prompt (LGP) method to enhance sentence
representations and their category prototypes for the MFACD task.

###  2.2 Prompt Learning

Prompt learning is the use of cue message to elicit knowledge from pre-trained
language models. The pre-trained language models contain powerful abilities,
and they can improve many tasks. For example, some works Reimers and Gurevych
(2019); Gao et al. (2021) leveraged the BERT to obtain enhanced sentence
representations. Despite the success of BERT in sentence embedding, the
performance of the original BERT was not satisfactory Li et al. (2020). PPT Gu
et al. (2022) showed that designing effective prompt statements can guide the
learning of the model, and this approach can perform well with a small number
of samples. Meanwhile, the work PromptBERT Jiang et al. (2022b) uses a novel
contrastive learning method for learning better sentence representations by
introducing prompt. Inspired by the PPT and the PromptBERT, we attempt to
apply prompt learning for the MFACD task.

Figure 1: The overview of our proposed LGP framework. The left part describes
the sentences of support set and query set, with different colors representing
different categories. The middle section describes the details of PESR and
PEPG. PESR can enhance sentence representation by the prompt. PEPG guides the
prototype generation through features of category descriptions.

##  3 Methods

The proposed method aims to improve the sentence representation and the
category prototype via the prompt. The pipeline of the proposed method
involves two crucial components Prompt Enhanced Sentence Representation (PESR)
module and Prompt Enhanced Prototype Generation (PEPG) module, as depicted in
Figure 1. The PESR module involves learning sentence representations by using
a prompt with the category label. Based on the above improved sentence
representations, PEPG module enhances the category prototype with the
assistance of the category descriptions. Before elaborating on the details of
each component, we first introduce our problem definition.

###  3.1 Problem Formulation

In tackling the MFACD task Snell et al. (2017), we also adopt the meta-
learning strategy, which performs on multiple $N$-way $K$-shot tasks. Each
task consists of a support set $S$ and a query set $Q$, spanning $N$
categories with $K$ sentences per category in the support set. Namely,
$S=\\{x_{i}^{j}|i=1,...,N;j=1,..,K\\}$, where $x_{i}^{j}$ represents the
$j$-th sentence of $i$-th category. The $Q$ is composed of some query
sentences $x_{q}$. The goal of the MFACD task is to learn a model from $S$ to
recognize aspect categories of $x_{q}$.

###  3.2 Prompt Enhanced Sentence Representation

Most of the current approaches use pre-trained models to obtain word
representations, which are used to represent sentences. But these approaches
leads to discrepancies between the pre-trained target and the downstream task,
thus limiting the performance of the pre-trained language model. In contrast,
prompt-based approaches can transform a sentence embedding task into a masked
language model (MLM) task. These methods effectively utilize the extensive
knowledge of the pre-trained model and avoids the bias inherent in direct
embedding Brown et al. (2020). Therefore, we design specific prompt templates
to enhance the adaptive capacity of pre-trained model on downstream tasks.

Prompt Template. We have crafted two kinds of prompt templates for sentence
representations. One kind is designed for sentences of support sets, and
another is for the query sentences. The prompt template for sentences of
support sets is as follows:

  * •

$P_{s}(x)$ = “About [$x$] Category [MASK] are : [$L_{x}$].”

where $x$ represents a sentence of support set and $L_{x}$ represents the
category label of $x$, and [MASK] contains $m$ learnable tokens. The prompt
template for query sentences is as follows:

  * •

$P_{q}(x_{q})$ = “About [$x_{q}$] Category [MASK].”

where $x_{q}$ represents a query sentence.

Sentence Representation. Given a sentence $x$, we can obtain the prompt with
the above templates. Without loss of generality, we denote the prompt of $x$
as $P(x)$, where $P(x)$ can be $P_{s}(x)$ or $P_{t}(x)$. Subsequently, the
$P(x)$ is tokenized to fed a pre-trained model $E(\cdot)$ to obtain the hidden
layer vector of the [MASK], which is denoted as $h\in R^{m\times d}$:

$\begin{matrix}h=E(P(x))\end{matrix}$ (1)

where $m$ is the number of tokens, and $d$ is the dimension of feature vector.
Multiple tokens are incorporated into a sentence template to enhance the
sentence representation. The final feature representation of $x$ is obtained
by a mean pooling operation of $h$. The formalization is as follows:

$\begin{matrix}v=M_{p}(h)\end{matrix}$ (2)

where $M_{p}(\cdot)$ means the mean pooling operation and $v\in R^{1\times d}$
is the sentence representation.

According to the above method of sentence representations, we can obtain
sentence representations of support set, which can be denoted as
$V=\\{v_{i}^{j}|i=1,...,N;j=1,..,K\\}$, which $v_{i}^{j}$ represents the
$j$-th sentence feature representations of $i$-th category. Similarly, the
query sentence is fed to the encoder with the $P_{q}(\cdot)$ template to
obtain its representations $v_{q}$. The per-trained model effectively extract
highly relevant information via the prompt for the entire sentence, obtaining
precise sentence representations.

###  3.3 Prompt Enhanced Prototype Generation

To tackle the challenge of imprecise category prototypes in few-shot learning
scenarios, we harness the sophisticated capabilities of LLM for meticulous
label description. These descriptions can effectively aid in the precise
formation of category prototypes.

Category Descriptions. To produce elaborate category descriptions, we adopt a
language model prompt template, denoted as $P_{c}(c_{i})$=“Provide a
comprehensive description of [$c_{i}$].” where $c_{i}$ symbolizes the category
label. The purpose of this prompt is to distinctly encapsulate and highlight
the characteristics of category. Subsequently, the crafted prompt
$P_{c}(c_{i})$ is fed into a LLM to generate detailed descriptions about the
$c_{i}$. This process can be formalized as:

$\begin{matrix}D_{i}=G_{M}(P_{c}(c_{i}))\end{matrix}$ (3)

where $D_{i}$ represents category descriptions and $G_{M}$ represents the
large language model. Leveraging the LLM’s proficiency in generating detailed
descriptions, which can extract the profound semantics and core
characteristics inherent to each category. These are some examples of category
descriptions as follows:

  1. 1.

food_food_bread:Bread is a staple food made by baking a dough of flour and
water, typically described based on its type (such as whole wheat, white
bread), size (such as one loaf, one slice), texture (such as soft or hard),
ingredients (such as containing kernels or seeds), and flavor (such as sweet
or salty).

  2. 2.

food_mealtype_breakfast:Breakfast is the first meal of the day, typically
consumed in the morning, consisting of food items such as cereals, eggs,
toast, pancakes, fruits, and beverages like coffee or juice.

  3. 3.

food_food_side_pasta:The food item in question is a side dish consisting of
pasta, which is typically prepared from durum wheat dough and served as an
accompaniment to a main course in various cuisines.

  4. 4.

food_portion:Food portion refers to the specific amount of food provided or
consumed during a meal, usually described by weight (such as grams,
kilograms), volume (such as milliliters, liters), portion size (such as one or
half portions), comparison (such as a football sized watermelon), cutlery unit
(such as a bowl of rice), sensory description (such as a sumptuous dinner), or
appropriate number of people (such as two servings).

It can be observed that these category descriptions contains a wealth of
information about the category. According to Eq. (1) with the prompt template
$P_{s}(\cdot)$ and Eq. (2), we can obtain the representations of $D_{i}$,
denoted as $v_{c_{i}}\in R^{1\times d}$.

Prototype Generation. The $v_{c_{i}}$ is used to provide a guidance for the
prototype generation with sentence representations. The discriminant prototype
$r_{i}$ for the $i^{th}$ class is extracted by calculating importance based on
the category descriptions and the sentence representations $V_{i}\in
R^{K\times d}$. The $V_{i}$ is composed of $\\{v_{i}^{j}|j=1,...,K\\}$. The
common used method of obtaining the prototype is mean pooling of these
sentence representations. In this kind of method, the weight of each sentence
is equal. We think that some sentences possibly contain a mass of noise
information. The same weight for all sentences can not avoid those noise
information. Thus, we use the description representations $v_{c_{i}}$
generated by LLM to reduce the importance of those noise sentence. The
importance of sentence representations $a_{i}\in R^{1\times K}$ is:

$\begin{matrix}a_{i}=\text{softmax}(M_{p}(v_{c_{i}}*V_{i})^{T})\par\end{matrix}$
(4)

where softmax() is the softmax function and $\odot$ is the Hadamard product,
i.e., element-wise multiplication. In this Hadamard product, the $v_{c_{i}}$
extends its size to reach $V_{i}$ by row copy. Using the attention weight of
each sentence $a_{i}$, we can obtain the prototype $r_{i}\in R^{1\times d}$ .

$\begin{matrix}r_{i}=a_{i}*V_{i},\end{matrix}$ (5)

where $*$ is the matrix multiplication. According to the above calculation, we
can obtain all category prototypes $\\{r_{i}|i=1,2,...,N\\}$.

###  3.4 Training and Inference

Training. In noisy scenarios, the query sample may contain more than one
target aspect category. Following the previous works Hu et al. (2021), we
employ query-attention to customize multiple prototype-specific query
representations $v_{i}^{q}\in R^{d}$, which can be obtained by:

$\begin{matrix}v_{i}^{q}=\text{softmax}(\text{tanh}(r_{i})\odot v_{q})\odot
v_{q}\end{matrix}$ (6)

where $\tanh()$ is the tanh function.

The final training loss is:

$\begin{matrix}L=-\frac{1}{N}\sum\limits_{i=1}^{N}y_{i}(\hat{y}_{i}-log{\sum\limits_{i=1}^{N}e^{\hat{y}_{i}}})\end{matrix}$
(7)

where $y_{i}$ is the ground truth label of $i$-th category (i.e., $y_{i}$=0 or
1). $\hat{y}_{i}$ is calculated as follows:

$\begin{matrix}\hat{y}_{i}=\frac{Cos(r_{i},v_{i}^{q})-\mu}{\delta}\end{matrix}$
(8)

where $Cos(r_{i},v_{i}^{q})$ represents the cosine similarity between the
prototype $r_{i}$ and prototype-specific query representations $v_{i}^{q}$,
$\mu$ is the mean of cosine similarity and $\delta$ is the variance of cosine
similarity.

Inference. In multi-label classification, compared with the method of the
fixed threshold, the method of the dynamic threshold adopts a adaptive value
based on label distribution, accurately adapting to different label
characteristics. We use a simple and effective strategy of dynamic thresholds
by combining varies of statistical magnitude. This dynamic threshold
$y_{i}^{d}$ of $i$-th category is calculated as follows:

$\begin{matrix}y_{i}^{d}=\alpha\cdot
m(\hat{y}_{i})+\beta\cdot\sigma(\hat{y}_{i})+\gamma\cdot\max(\hat{y}_{i})+\\\
(1-\gamma)\cdot\min(\hat{y}_{i})\end{matrix}$ (9)

where $m()$ represents the mean value, $\sigma()$ is the standard deviation,
max() is the maximum and min() is the minimum value. $\alpha$, $\beta$, and
$\gamma$ are adjustable parameters.

Dataset | #cls. | #sent./cls. | #sent.
---|---|---|---
FewAsp(multi) | 100 | 400 | 40000
FewAsp | 100 | 630 | 63000
Table 2: Dataset statistics. #cls. and #sent. denote the number of classes and samples, respectively. Models | 5-way 5-shot | 5-way 10-shot | 10-way 5-shot | 10-way 10-shot
---|---|---|---|---
AUC F1 | AUC F1 | AUC F1 | AUC F1
Prototypical Network Snell et al. (2017) | 88.88 66.96 | 91.77 73.27 | 87.35 52.06 | 90.13 59.03
IMPAllen et al. (2019) | 89.95 68.96 | 92.30 74.13 | 88.50 54.14 | 90.81 59.84
Proto-HATT Gao et al. (2019) | 91.54 70.26 | 93.43 75.24 | 90.63 57.26 | 92.86 61.51
Proto-AWATT Hu et al. (2021) | 93.35 75.37 | 95.28 80.16 | 92.06 65.65 | 93.42 69.70
LDF Zhao et al. (2022) | 94.65 78.27 | 95.71 81.87 | 92.74 67.13 | 94.29 71.97
LPN Liu et al. (2022) | 96.45 82.22 | 97.15 84.90 | 95.36 71.42 | 96.55 76.51
FSO Zhao et al. (2023) | 96.92 83.44 | 97.38 85.08 | 95.65 73.78 | 96.28 76.58
LGP(Ours) | 97.37 87.49 | 97.49 87.67 | 96.33 77.92 | 96.69 78.95
Table 3: Comparisons of AUC (%) and Macro-F1 (%) score on FewAsp. The best score is in bold. Models | 5-way 5-shot | 5-way 10-shot | 10-way 5-shot | 10-way 10-shot
---|---|---|---|---
AUC F1 | AUC F1 | AUC F1 | AUC F1
Prototypical Network Snell et al. (2017) | 89.67 67.88 | 91.60 72.32 | 88.01 52.72 | 90.68 58.92
IMP Allen et al. (2019) | 90.12 68.86 | 92.29 73.51 | 88.71 53.96 | 91.10 59.86
Proto-HATT Gao et al. (2019) | 91.10 69.15 | 93.03 73.91 | 90.44 55.34 | 92.38 60.21
Proto-AWATT Hu et al. (2021) | 91.45 71.72 | 93.89 77.19 | 89.80 58.89 | 92.34 66.76
LDF Zhao et al. (2022) | 92.62 73.38 | 94.34 78.81 | 90.87 62.06 | 92.93 68.23
LPNLiu et al. (2022) | 95.66 79.48 | 96.55 82.81 | 94.51 67.28 | 95.66 71.87
FSO Zhao et al. (2023) | 96.01 81.04 | 96.67 82.22 | 94.93 70.26 | 95.71 72.46
LGP(Ours) | 97.67 85.22 | 97.86 86.08 | 95.89 75.01 | 96.35 76.97
Table 4: Comparisons of AUC (%) and Macro-F1 (%) score on FewAsp(multi). The best score is in bold. Models | 5-way 2-shot | 5-way 3-shot | 10-way 2-shot | 10-way 3-shot
---|---|---|---|---
AUC F1 | AUC F1 | AUC F1 | AUC F1
LDF Zhao et al. (2022) | 91.30 70.57 | 91.23 70.51 | 89.69 57.45 | 90.16 59.67
LPN Liu et al. (2022) | 92.90 70.69 | 94.21 74.55 | 91.36 56.11 | 92.67 60.34
FSO Zhao et al. (2023) | 95.14 77.44 | 95.27 78.58 | 93.97 66.41 | 94.32 68.28
LGP(Ours) | 95.44 81.47 | 96.15 83.60 | 94.50 71.72 | 94.89 72.98
Table 5: Comparison of AUC (%) and Macro-F1 (%) score on FewAsp(multi). The
best score is in bold. Figure 2: Feature visualization of category prototype
on FewAsp(Multi). The category prototypes are calculated from 3000 5-way
5-shot tasks. Different colors means different categories.

##  4 Experiments

###  4.1 Datasets and Experimental Setups

Datasets. In order to verify the validity of LGP model, two publicly available
datasets FewAsp (multi) and FewAsp form Hu et al. (2021) are used for
experimental evaluation. The FewAsp(multi) dataset consists of sentences
featuring multiple aspect categories. The FewAsp dataset comprises sentences
with mixed (one or more) category types. The above two datasets have the same
100 aspect categories, of which 64 aspect categories are used for training, 16
aspect categories for validation, and 20 aspect categories for testing.
General information of two datasets is presented in Table 2.

Implementation Details. In our study, we employ BERT-uncased English version
Kenton and Toutanova (2019) as the encoder, with the implementation in
PyTorch. The LLM GPT-3.5-Turbo is chosen for generating category descriptions.
All experiments are conducted on a single NVIDIA A100, utilizing CUDA version
12.2. During training, we use the AdamW optimer Loshchilov and Hutter (2017)
with a learning rate of 5e-5. And we set the hyper-parameters $\alpha$,
$\beta$ and $\sigma$ with 0.3, 0.7 and 0.7, respectively. During each epoch,
800 tasks are randomly sampled for training. In each task, $N\in\\{5,10\\}$,
$K\in\\{2,3,5,10\\}$, and the number of query instances of each category is 5.

Evaluation Metric. Following Hu et al. (2021), we use Macro-F1 and AUC scores
as the evaluation metrics. 600 tasks sampled from test data are used for
evaluation, and the average performance on the 600 tasks is reported for
comparisons.

###  4.2 Experimental Results and Analysis

The comparative experimental results on the FewAsp dataset are shown in Table
3. It can be observed that the proposed LGP demonstrates the superior
performance than the existing methods under all experimental evaluations. It
is worth noting that LGP surpass current state-of-the-art method FSO Zhao et
al. (2023) with 4.05% and 4.14% gains on F1 score under 5-way 5-shot setting
and 10-way 5-shot setting, respectively.

The comparative results on the FewAsp(multi) dataset are shown in Table 4. We
also observe that the proposed LGP achieves state-of-the-art performance under
all experimental evaluations. Compared with FSO, the LGP obtains large
performance gains on F1 score with 4.08%, 3.86%, 4.75% and 4.51% improvements
under 5-way 5-shot, 5-way 10-shot, 10-way 5-shot and 10-way 10-shot settings,
respectively. The LGP obtains 1.66%, 1.21%, 0.96% and 0.64% improvements on
AUC over FSO, under 5-way 5-shot, 5-way 10-shot, 10-way 5-shot and 10-way
10-shot settings, respectively.

In addition, following the work FSO Zhao et al. (2023), we also conduct
experiments on FewAsp(multi) with fewer training samples such as 2-shot and
3-shot. The experimental results are exhibited in Table 5. The proposed method
LGP also achieves the best performance under all experimental settings. And
the relatively large improvements on F1 score also can be found.

To summarize, the proposed method LGP shows performance advantage on the above
two datasets under different experimental settings, exhibiting the
effectiveness and the superiority of the proposed method. It seems to suggest
that enhancing representations of both sentences and category prototype with
the prompt is a promising method to address the MFACD task.

###  4.3 Ablation Study

Effectiveness of Components. The proposed method LGP mainly involves two
components Prompt Enhanced Sentence Representation (PESR) and Prompt Enhanced
Prototype Generation (PEPG). To verify the effectiveness of each component, we
conduct ablation study, and experimental results of are shown in Table 6. The
removal of the PESR module results in a 2.18% decrease in F1 score and a 1.66%
decrease in AUC score. This seems to mean that the absence of PESR undermines
the model’s comprehension of sentence context. Excluding the PEPG module leads
to a 1.49% reduction in F1 scores and a 1.54% reduction in AUC scores,
indicating the vital role of PEPG in forming representative category
prototypes. The combined removal of both PESR and PEPG causes the most
substantial decline in performance, which can explain that their synergy is
crucial to the model.

PESR | PEPG | AUC |  $\delta$ AUC | F1 |  $\delta$ F1
---|---|---|---|---|---
✓ | ✓ | 97.67 |  | 85.22 |
✗ | ✓ | 96.01 | -1.66 | 83.04 | -2.18
✓ | ✗ | 96.13 | -1.54 | 83.73 | -1.49
✗ | ✗ | 95.09 | -2.58 | 80.29 | -4.93
Table 6:  Ablation study on FewAsp(multi) under 5-way 5-shot setting. PESR
means Prompt Enhanced Sentence Representation and PESR means Prompt Enhanced
Prototype Generation.

Moreover, comparing PESR and PEPG, we observe that the performance reduction
is more obvious when the PESR is removed. This indicates the PESR is a more
effective component than the PEPG and the method of sentence representation
exerts a greater influence than the prototype generation.

Visualization. We visualize feature representations of category prototypes
with and without PESR and PEPG. The tool of t-SNE van der Maaten and Hinton
(2008) is used to visualize the feature vectors, and the visualization results
of 20 test categories on FewAsp(Multi) are illustrated in Figure 2.

It can be seen that there are distribution differences in the feature
representations of category prototypes before and after the application of the
PESR and PEPG modules. When we don’t have any prompt to help for prototyping,
namely without both PESR and PEPG module, the distribution of category
prototypes is scattered with blurred boundaries between categories. When only
PESR is applied, there is an improvement in the aggregation of prototypes, but
the distinction between categories remains insufficient. When both PESR and
PEPG are applied, the category prototypes are not only more compact but also
more clearly separated from each other, indicating that the PESR and PEPG
modules can effectively enhance the generation of representative category
prototypes.

###  4.4 Discussions

Effect of Prompt Templates. A crucial issue for prompt-based tasks is finding
the suitable template. This issue has been explored through two distinct types
of templates: hard template and soft template, as detailed in Table 7. Hard
templates are crafted manually, providing a structured and fixed format. In
contrast, soft templates offer a learnable prompt structure, allowing the
model to autonomously adapt and learn the most effective format. According to
Lester et al. (2021), we compare the following soft prompt: “$p_{1}$, [MASK],
…[$x$], $p_{n}$”, where $p_{i}$ means the learnable prompt.

Templates | F1
---|---
Soft |  $p_{1}$, [MASK], …[$x$], $p_{n}$ | 82.87
Hard | In [x], the opinions about [$L_{x}$] are [MASK]. | 84.41
In [x], the opinion that exists are [MASK].
[$L_{x}$] The aspects of [$x$] are [MASK]. | 84.71
The aspects of [$x$] are [MASK].
[$L_{x}$] What are the aspects of [$x$] [MASK]. | 84.43
What are the aspects of [$x$] [MASK].
This [$x$] means : [$L_{x}$] [MASK]. | 84.44
This [$x$] means : [MASK].
About [$x$] Category [MASK] are : [$L_{x}$]. | 85.22
About [$x$] Category [MASK].
Table 7: Comparisons of Macro-F1 (%) for different templates under the 5-way
5-shot setting on FewAsp(multi).

Table 7 showcases the impact of different prompt templates under 5-way 5-shot
setting on FewAsp(multi). Among the templates, “About [$x$] Category [MASK]
are : [$L_{x}$]” stands out with the highest performance with 85.22% F1,
indicating its effectiveness in guiding the model for aspect-based sentiment
analysis. In comparison, the soft prompt scores lower at 82.87% F1. The
inclusion of the category label ([$L_{x}$]) in prompts generally improves
performance, as seen in the superior results of structured prompts over their
underlined variants without [$L_{x}$]. Despite different prompts have
different effects on the model’s performance, on the whole, a prompt-based
approach generally leads to notable enhancements in the model.

Effect of Tokens. We explicitly assess the effect of the token from its
quantity and state (i.e. fixed token or learnable token). When the BETR is
tunable (see the blue line in Figure 3), the performance of both fixed token
and learnable token method is improved at the beginning of the number of token
increasing, after which the performance is not significantly improved. This is
attributed to the improvement of the ability with involving multiple tokens.

Figure 3:  The impact on the number of tokens, token state and encoder
parameters.

When the BETR is frozen (see the green line in Figure 3), the method of the
fixed token obtain performance degrade after the number of tokens exceeds 5.
However, that doesn’t happen when the tokens are learnable. It can be observed
that replacing the fixed tokens with learnable tokens contributes to the
stability of our model.

Effect of Encoder Parameters. To explicitly assess the effect of parameter
learning in the encoder, we explore two experiments frozen BETR (the green
line in Figure 3) and tunable BETR (the blue line in Figure 3). It can be
observed that the proposed method with a tunable BETR is clearly superior to
that with a frozen BETR under all experimental settings. The reason is that
allowing the parameters of BERT to be trainable enables the model to
dynamically adapt current MFACD tasks.

##  5 Conclusions and Further Works

We propose a Label-Guided Prompt (LGP) method for noisy MFACD tasks by
utilizing semantic prompt to obtain better sentence representations and
category prototypes. This method employs crafted prompts integrated with label
information, and it can extract vital contextual and semantic insights from
the sentences. Meanwhile, a LLM is used to generate precise category
descriptions, which effectively guide the generation of more accurate category
prototypes. Extensive experimental results on two datasets demonstrate that
the proposed method achieves state-of-the-art performance.

In the future work, we aim to explore some novel approaches from advanced
prompting mechanisms, context-sensitive learning techniques and co-
evolutionary model-prompt strategies. By pushing the boundaries of aspect
category detection, our research endeavors to enhance the accuracy and
robustness of detecting multiple labels in situations where data availability
is limited.

## References

  * Allen et al. [2019] Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum.  Infinite mixture prototypes for few-shot learning.  In ICML, pages 232–241, 2019.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, pages 1877–1901, 2020.
  * Chalkidis et al. [2019] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos.  Large-scale multi-label text classification on eu legislation.  arXiv preprint arXiv:1906.02192, 2019.
  * Gao et al. [2019] Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun.  Hybrid attention-based prototypical networks for noisy few-shot relation classification.  In AAAI, pages 6407–6414, 2019.
  * Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In EMNLP, pages 6894–6910, 2021.
  * Ghadery et al. [2019] Erfan Ghadery, Sajad Movahedi, Heshaam Faili, and Azadeh Shakery.  Mncn: A multilingual ngram-based convolutional network for aspect category detection in online reviews.  In AAAI, pages 6441–6448, 2019.
  * Gu et al. [2022] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang.  PPT: Pre-trained prompt tuning for few-shot learning.  In ACL, pages 8410–8423, 2022.
  * Hai et al. [2011] Zhen Hai, Kuiyu Chang, and Jung-jae Kim.  Implicit feature identification via co-occurrence association rule mining.  In CICLing, pages 393–404, 2011.
  * Hou et al. [2021] Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu.  Few-shot learning for multi-label intent detection.  In AAAI, pages 13036–13044, 2021.
  * Hu et al. [2021] Mengting Hu, Shiwan Zhao, Honglei Guo, Chao Xue, Hang Gao, Tiegang Gao, Renhong Cheng, and Zhong Su.  Multi-label few-shot learning for aspect category detection.  In ACL-IJCNLP, pages 6330–6340, 2021.
  * Jiang et al. [2019] Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang.  A challenge dataset and effective models for aspect-based sentiment analysis.  In EMNLP-IJCNLP, pages 6280–6285, 2019.
  * Jiang et al. [2022a] Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang.  Promptbert: Improving bert sentence embeddings with prompts.  In EMNLP, pages 8826–8837, 2022.
  * Jiang et al. [2022b] Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang.  PromptBERT: Improving BERT sentence embeddings with prompts.  In EMNLP, pages 8826–8837, 2022.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, pages 4171–4186, 2019.
  * Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * Li et al. [2020] Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li.  On the sentence embeddings from pre-trained language models.  In EMNLP, pages 9119–9130, 2020.
  * Liu et al. [2022] Han Liu, Feng Zhang, Xiaotong Zhang, Siyang Zhao, Junjie Sun, Hong Yu, and Xianchao Zhang.  Label-enhanced prototypical network with contrastive learning for multi-label few-shot aspect category detection.  In SIGKDD, pages 1079–1087, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * Movahedi et al. [2019] Sajad Movahedi, Erfan Ghadery, Heshaam Faili, and Azadeh Shakery.  Aspect category detection via topic-attention network.  arXiv preprint arXiv:1901.01183, 2019.
  * Pontiki et al. [2016] Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammed AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al.  Semeval-2016 task 5: Aspect based sentiment analysis.  In SemEval, pages 19–30, 2016.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In EMNLP-IJCNLP, pages 3982–3992, 2019.
  * Rios and Kavuluru [2018] Anthony Rios and Ramakanth Kavuluru.  Few-shot and zero-shot multi-label learning for structured label spaces.  In EMNLP, pages 3132–3142, 2018.
  * Schouten et al. [2017] Kim Schouten, Onne Van Der Weijde, Flavius Frasincar, and Rommert Dekker.  Supervised and unsupervised aspect category detection for sentiment analysis with co-occurrence data.  IEEE transactions on cybernetics, 48(4):1263–1275, 2017.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, pages 4080–4090, 2017.
  * Su et al. [2021] Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou.  Whitening sentence representations for better semantics and faster retrieval.  arXiv preprint arXiv:2103.15316, 2021.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  * Wang and Iwaihara [2023] Zeyu Wang and Mizuho Iwaihara.  Few-shot multi-label aspect category detection utilizing prototypical network with sentence-level weighting and label augmentation.  In DEXA, pages 363–377, 2023.
  * Zhao et al. [2022] Fei Zhao, Yuchen Shen, Zhen Wu, and Xinyu Dai.  Label-driven denoising framework for multi-label few-shot aspect category detection.  arXiv preprint arXiv:2210.04220, 2022.
  * Zhao et al. [2023] Shiman Zhao, Wei Chen, and Tengjiao Wang.  Learning few-shot sample-set operations for noisy multi-label aspect category detection.  In IJCAI, pages 5306–5313, 2023.
  * Zhu et al. [2020a] Yaohui Zhu, Chenlong Liu, and Shuqiang Jiang.  Multi-attention meta learning for few-shot fine-grained image recognition.  In IJCAI, pages 1090–1096, 2020.
  * Zhu et al. [2020b] Yaohui Zhu, Weiqing Min, and Shuqiang Jiang.  Attribute-guided feature learning for few-shot image recognition.  IEEE Transactions on Multimedia, 23:1200–1209, 2020.