# Instruction Tuning with Lexicons for Zero-Shot Style Classification

Ruohao Guo Wei Xu Alan Ritter
Georgia Institute of Technology
<EMAIL_ADDRESS>{wei.xu<EMAIL_ADDRESS>
###### Abstract

Style is used to convey authors’ intentions and attitudes. Despite the success
of large pre-trained language models on style classification, prior work
relies on fine-tuning with labeled examples. Prompting large language models
to classify style without fine-tuning is challenging because language styles
can be difficult to define. In this study, we investigate the effectiveness of
style lexicons as a means for instructing language models how to identify new
styles that are unseen during training. Our experiments show that lexicon-
based instructions improve transfer zero-shot performance significantly. We
will release our code and data.

##  1 Introduction

People display different language styles in their communications Hovy (1987).
Style is often important for understanding the intention behind an utterance,
and can provide insight into the person’s communication goals Kang and Hovy
(2021). For example, one may use humor to make the conversation more smooth
and more enjoyable, while sarcasm is commonly employed to express disapproval
or contempt. People are also known to adapt the style from the communities
they interact with (Danescu-Niculescu-Mizil et al., 2013b). In some sense,
there are as many language styles as there are speakers, and possibly more
because people can change and adapt their style over time. Styles are more
commonly grouped into coarse-grained and overlapping categories, such as
“Internet English”, “polite conversation”, or “Elizabethan English”.

Style classification is the task of identifying and categorizing the style of
language being used (Figure 1). The open-ended nature of style motivates the
need for zero-shot classification, as it is difficult to collect examples of
every possible style. Recent advances in language modeling (Brown et al.,
2020a; Chowdhery et al., 2022) and instruction fine-tuning Zhong et al.
(2021); Sanh et al. (2021), have enabled surprisingly effective zero-shot
learning. However, as we show in §4.1, using open-source variants of these
models to identify language styles remains a challenge. For example, Flan-T5
Chung et al. (2022) achieved only an F1 score of 35.07, averaged across six
diverse language styles, in a zero-shot style classification experiment.

Figure 1: Cross-style zero-shot classification with LexPT (§2.2) consists of
two steps: (1) instruction fine-tune the model on source styles with LexPT;
(2) transfer the learned model to any target style zero-shot with LexPT. A
LexPT input is composed of a prompt template $T(\cdot)$, classes, lexicons and
an input sentence.

In this paper, we study the question of whether style lexicons can be used to
instruct language models how to classify styles without any text samples or
labels. Before the widespread adoptation of deep learning methods in NLP,
lexicons were often used for text classification (Taboada et al., 2011), and
also to generate features for use in linear models (Mohammad et al., 2013). A
variety of lexicons have been developed for linguistic styles, including
politeness (Danescu-Niculescu-Mizil et al., 2013a), readability (Maddela and
Xu, 2018), and Shakespearean English Xu et al. (2012). We investigate the
effectiveness of lexicons for instructing large language models on how to
identify a style of writing. Extensive experiments demonstrate that including
target style lexicons in instructions used to prompt language models can
improve zero-shot classification, for example increasing the average F1 score
of Flan-T5 from 35.07 to 47.28. Furthermore, we show that fine-tuning models
to better understand lexicon-based instructions yields further improvements. A
version of T5 Raffel et al. (2020) with the same number of parameters, that
was instruction-tuned on seven source styles, achieves an average F1 score of
57.30, an absolute 10 point improvement over Flan-T5.

Our work demonstrates that lexicons can be used in combination with large
language models to perform zero-shot cross-style transfer to classify unseen
language styles, even when no target-style samples are available. Moreover, we
show how models, such as T5 or GPT-J Wang and Komatsuzaki (2021) can be
instruction fine-tuned to identify novel language styles using lexicons.

##  2 Methodology

In this work, we study the task of cross-style zero-shot classification by
introducing Lexicon-based PrompTs (LexPT), a method for instructing language
models to classify style using lexicons. The key idea is to perform meta-
training on existing annotated datasets for high-resource styles, then
incorporate a small lexicon into a natural-language prompt to improve zero-
shot performance on any arbitrary target writing style.

Style Dataset $|C|$ B? #Tra, #Val, #Test Lexicon Sources Age∗ Kang and Hovy
(2021) 2 ✗ 14k, 2k, 2k ChatGPT, Dict Country Kang and Hovy (2021) 2 ✗ 33k, 4k,
4k ChatGPT, Dict Formality Rao and Tetreault (2018) 2 ✓ 209k,10k,5k NLP Wang
et al. (2010), Dict Hate/Offense Davidson et al. (2017) 3 ✗ 22k,1k,1k NLP Ahn
(2005), Dict Humor CrowdTruth (2016) 2 ✓ 40k,2k,2k ChatGPT, Dict Politeness
Danescu-Niculescu-Mizil et al. (2013a) 2 ✓ 10k,0.5k,0.6k NLP Danescu-
Niculescu-Mizil et al. (2013a), Dict Politics Kang and Hovy (2021) 3 ✗ 33k,
4k, 4k NLP Sim et al. (2013), Dict Readability Arase et al. (2022) 2 ✗
7k,1k,1k NLP Maddela and Xu (2018), Dict Romance Kang and Hovy (2021) 2 ✓
2k,0.1k,0.1k ChatGPT, Dict Sarcasm Khodak et al. (2018) 2 ✓ 11k,3k,3k ChatGPT,
Dict Sentiment Socher et al. (2013) 2 ✗ 236k,1k,2k NLP Mohammad (2021), Dict
Shakespeare Xu et al. (2012) 2 ✓ 32k,2k,2k NLP Xu et al. (2012), Dict
Subjectivity Pang and Lee (2004) 2 ✓ 6k,1k,2k NLP Wilson et al. (2005), Dict

Table 1: Statistics of datasets and lexicons. $|C|$ refers to the number of
classes in each style dataset. B? indicates whether or not the class
distribution is balanced. #Train,#Val,#Test lists the number of examples in
train, validation and test sets, respectively. Note that to better compare
across different styles, we transformed the _Age_ dataset into a binary one by
mapping six (i.e., _Under12_ , _12-17_ , _18-24_ , _45-54_ , _55-74_ ,
_75YearsOrOlder_) out of the eight original age classes to two new classes
(i.e., _youthful_ , _mature_). The original age classes _25-34_ and _35-44_
were excluded.

###  2.1 Zero-shot Cross-style Transfer

The goal of zero-shot style classification is to predict whether a given piece
of text $x$ presents one of a fixed set of target styles $y\in Y_{tgt}$ that
are unseen by the model during training time. Meanwhile, we assume the model
has access to training examples from a separate set of high-resource source
styles
$(X_{src},Y_{src})=\bigcup^{|Y_{src}|}_{j=1}\\{(x^{j}_{i},y^{j}_{i})\\}^{|X^{j}_{src}|}_{i=1}$,
where labeled data is more abundantly available. We experiment with pre-
trained language models, which are known to exhibit zero-shot learning
capabilities, to demonstrate the effectiveness of instruction fine-tuning on
source styles.

###  2.2 Lexicon-based Prompting (LexPT)

We investigate the utility of incorporating style lexicons into the language
models’ instructions. Given a text input $x$ and a style with pre-defined
classes $C=\left\\{c_{k}\right\\}^{|C|}_{k=1}$, we design a lexicon-based
prompt (LexPT) as
$\bar{x}=T(c_{1},L_{c_{1}},\cdots,c_{k},L_{c_{k}},\cdots,x)$, where
$L_{c_{k}}$ is a list of words or phrases that are representative of the class
$c_{k}$. For zero-shot learning, a language model is expected to predict one
of the classes $\hat{c}\in C$, given $\bar{x}$. The prompt template $T(\cdot)$
is a function that converts all elements into a specific structure by filling
them in the predefined slots (see templates in the Appendix Table 13).

##### Instruction Fine-Tuning on Source Styles.

To predict held-out target styles without supervised data, LexPT requires the
model to understand the lexicon-based instructions that can transfer across
styles. This is achieved by fine-tuning the model on the source styles using
LexPT. The objective is to optimize the conditional log-likelihood of the gold
class given LexPT inputs. To prevent memorization of styles used during
instruction fine-tuning, we introduce class randomization (§3.4 and §3.5). Our
hypothesis is that if we randomize style-specific identifiers used in the
prompt, such as class names, the model will not be able to depend on them for
prediction; instead, it will need to learn to leverage lexicons included in
the instructions to make accurate predictions. Specifically, one variant of
LexPT (LexPT${}_{\text{I,Rw}}$) replaces each class name $c_{k}$ in the prompt
with random English words. In §4.3, we conduct a detailed investigation into
the impact of randomization by comparing the LexPT variants that use different
approaches to randomization.

##### Zero-Shot Inference on Target Styles.

For a new target style, the model is given an LexPT input $\bar{x}$ that is
constructed using the same lexicon-based instructions as used during fine-
tuning on source styles, except that the lexicons used here are for the target
style. Following T0 Sanh et al. (2021), we use rank classification, for which
we compute the likelihood of each style class and then pick the one with the
highest likelihood as the final prediction.

##  3 Experimental Settings

To evaluate the effectiveness of LexPT in generalizing to various writing
styles, we consider a total of thirteen distinct styles in our experiments
based on the availability of high-quality data. These styles are taken from a
variety of sources, including the XSLUE benchmark Kang and Hovy (2021),
Subjectivity Pang and Lee (2004), Shakespeare Xu et al. (2012) and Readability
Arase et al. (2022) datasets. We compare LexPT with other prompting methods
under three settings: (i) _zero-shot learning_ with LLMs and instruction-tuned
models, (ii) _transfer zero-shot learning_ , and (iii) _few-shot learning_
with T5 and GPT-J. Finally, to validate the potential factors that contribute
to the effectiveness of LexPT, we experiment with its variants to compare
their performance across target styles.

###  3.1 Dataset Split

As summarized in Table 1, all thirteen datasets involve sentence-level
classification, covering a wide range of text sources (e.g., Wikipedia, social
media, web data). See more details in Appendix A. In a transfer zero-shot
setting, a model is fine-tuned on the source styles with one prompting method,
before being evaluated on the target styles. Specifically, we combine the
train and validation sets of all source datasets and shuffle them. For each
source style, the train set is used for fine-tuning and the validation set is
used for model selection Chen and Ritter (2021). For each target style, the
test set is used for evaluation. We ensure that the datasets used as targets
do not share any examples with the source datasets. In the main paper, we
present the results of our largest split, comprising Sentiment, Formality,
Politeness, Hate/Offense, Readability, Politics, and Subjectivity in the
source split, while the remaining six styles are allocated in the target
split. Experiments for other different source/target style splits are provided
in Appendix E.4.

###  3.2 Lexicon Collection

We use stylistic lexicons that have been created by other NLP researchers
wherever possible (listed as “NLP” in Table 1). These high-quality lexicons
were either manually annotated Maddela and Xu (2018) or automatically induced
using corpus-based approaches Danescu-Niculescu-Mizil et al. (2013a); Socher
et al. (2013). For styles where such lexicons are not readily available, we
explore three methods to create lexicons, using either ChatGPT, dictionaries,
or manual construction.

##### ChatGPT-generated Lexicons.

Prior work has used models, such as BERT to generate a class vocabularies for
topic classification Meng et al. (2020). Inspired by this approach, we utilize
the knowledge of large language models (LLMs) by prompting them to generate a
list of words that express the specific class of a style. In a preliminary
study, we experimented with many LLMs, including BERT, GPT-J, GPT-NeoX,
GPT-3.5 and ChatGPT. Among all, ChatGPT performs the best; for example,
generating keywords such as “funny, laugh-out-loud, silly” for the “humorous”
class. The prompts we used for ChatGPT are presented in Appendix B.

##### Dictionary-based Lexicons.

We also considered lexicons generated by extracting the description of each
style from Google Dictionary.111An online dictionary service licensed from
Oxford University Press’s OxfordDictionaries.com.:
https://www.google.com/search?q=Dictionary For example, the words for the
class “humorous” are being comical, amusing, witty.

For manual construction, we simply ask a native English speaker to write out a
list of words for each language style. Appendix C provides more examples and
details of each lexicon source. Depending on the construction method, these
lexicons may vary in quality and size from a few words to thousands.
Nevertheless, we will show the benefits of LexPT with as few as five words
sampled from lexicons to represent each writing style (§5).

###  3.3 Pretrained Language Models

We experiment with the open-sourced conditional generation model T5 Raffel et
al. (2020) and the causal language model GPT-J Wang and Komatsuzaki
(2021).222In preliminary experiments, we compare the performance of fully
fine-tuned, partially fine-tuned, and parameter-efficiently-finetuned GPT-J.
Finetuning GPT-J with LoRA Hu et al. (2021) performs the best, so we use it to
fine-tune GPT-J. We also include experiments with the instruction-tuned models
T0 Sanh et al. (2021), Flan-T5 Chung et al. (2022)333We ensure none of the
target style datasets appear in the fine-tuning tasks of T0 or Flan-T5. and
GPT-3.5 (text-davinci-003), as these models have demonstrated the ability to
effectively respond to instructions and generalize well to unseen tasks Chung
et al. (2022). Implementation details are described in Appendix D.

###  3.4 Compared Methods in Different Settings

To assess the effectiveness and generality of our approach, we compare LexPT
with other prompting methods in three learning settings.

####  3.4.1 Zero-Shot Learning

A model is prompted to predict the target styles without any fine-tuning. In
this setting, we experiment with Flan-T5, T0, GPT-J and GPT-3.5. For each
model, we compare the standard prompts (i.e., No Lexicon) and LexPT without
demonstrations (e.g., example sentences for a target style). Both prompting
methods utilize the same template which is described in Appendix F.1, except
that class names instead of lexicons are used in the No Lexicon prompt.

####  3.4.2 Transfer Zero-Shot Learning

In this setting, a model is first fine-tuned on the source styles before being
evaluated on the unseen target styles under a zero-shot setup. We have
introduced this setting formally in §2.1. We use the same prompt template as
in §3.4.1, and experiment with T5 and GPT-J to compare different prompting
methods that are applicable:

##### No Lexicon.

Both source fine-tuning and target evaluation take the same template to
construct the prompt inputs without using lexicons.

##### LexPT.

For each class (e.g., “polite” or “impolite”) of the style (e.g., politeness),
we randomly select a subset of $m$ words or phrases from the corresponding
lexicon, then incorporate them into the prompts. We use $m=5$ for most
experiments in the paper unless otherwise specified and include an analysis on
using varied values of $m$ in §5.

####  3.4.3 Few-Shot Learning

We also investigate the performance of different prompting methods in the few-
shot setting, where a small number of training examples of the target styles
are included in the prompts.

##### MetaICL.

We adapt MetaICL Min et al. (2022a), which was originally developed for meta
in-context learning on multiple tasks (not related to style classification) to
learning on multiple styles, which is new in this paper. During the fine-
tuning stage, for each iteration, one source style is sampled, and $K$ labeled
examples (i.e., $K$ demonstrations) are randomly selected from the train set
of the chosen style. Each prompt consists of $K$ demonstrations followed by an
input sentence for the model to predict the corresponding class. At inference
time, the prompt is constructed similarly to the fine-tuning stage, except
that the $K$ demonstrations are randomly sampled from the train set of the
target styles instead of the source styles. Recently, Min et al. (2022b) have
shown that gold labels are not required in MetaICL. We re-examine this finding
by conducting experiments in our task using random and gold labels
respectively. We follow Min et al. (2022b) to set $K=4$ and $K=16$. Lexicons
are not used in this method.

##### MetaICL+Lex.

For a more comprehensive comparison between the two sources of supervision
(i.e., demonstrations vs. lexicons), we also modify MetaICL to incorporate
lexical signals. Specifically, we concatenate the name of each class with its
corresponding lexicon words, and prepend this information to each labeled
example to form a modified demonstration. Each prompt consists of $K$ modified
demonstrations followed by an input sentence. Following the approach of
MetaICL Min et al. (2022b), we conduct experiments with both random and gold
labels in the demonstration for MetaICL+Lex.

##### LexPT+K.

To conduct a fair comparison between LexPT and MetaICL, it is necessary to
incorporate supervision from $K$ target-style demonstrations into our
approach. Hence we introduce a modification to LexPT, called LexPT+K, which is
based on LexPT and involves a second step of fine-tuning on $K$ target
demonstrations. Specifically, for each target style, we randomly select $K$
examples from its train set and assign a label to each of them. Similar to
MetaICL and MetaICL+Lex, the assigned labels can be either random or gold.
Next, a model that was previously fine-tuned using LexPT on the source styles
is further fine-tuned on these $K$ demonstrations of the target style.

###  3.5 Variants of LexPT

To gain a better understanding of the key aspects of LexPT, we investigate
multiple variants based on: (i) whether or not the prompt template contains
human-written natural language instructions; (ii) the degree of class name
randomization. We provide new empirical results (§4) in the cross-style
transfer setup to add to the ongoing discussions Min et al. (2022b); Zhao et
al. (2022) of varied design choices in the prompting methods.

All LexPT variants are summarized in Table 2, while Figure 9 shows the example
prompts used in each method. The subscript I means the prompt template
$T(\cdot)$ contains natural language instructions. R# stands for randomizing
class names with numerical indices, and Rw refers to using random words as
class names in the prompt. We simply use the default English word list in
Ubuntu444/usr/share/dict/words for this word-based randomization. The only
difference between Rw- and Rw lies in the size of the fixed set of words used
for randomization (rand. vocab size). In Rw-, the randomized indices are
replaced with a fixed set of English words and the number of words in this
fixed set is equal to the number of indices in R#. The size of the fixed set
is much larger in Rw than in Rw-, which reduces the chance of assigning the
same word to the same class in different examples (i.e., higher
randomization). Since Rw provides more options for randomizing classes, the
same class name is less likely to be re-used across prompts, which we
hypothesize forces the model to learn to use the lexicons to classify styles.

|  no rand. |  rand. indices | rand. words
---|---|---|---
rand. vocab size |  – |  3 |  3 |  99,171
[1pt/1pt]  | w/o instructions
---
LexPT${}_{\text{0}}$ |  LexPT${}_{\text{R\\#}}$ |  LexPT${}_{\text{Rw-}}$ |  LexPT${}_{\text{Rw}}$
[1pt/1pt]  | w/ instructions
---
LexPT${}_{\text{I}}$ |  LexPT${}_{\text{I,Rw\\#}}$ |  LexPT${}_{\text{I,Rw-}}$ |  LexPT${}_{\text{I,Rw}}$

Table 2: Variants of LexPT. rand. vocab is the fixed set of indices or words, from which a class name can be randomly selected. Despite the rand. vocab size of 99,171 in the last column, only about 19% of these words are actually selected through random sampling.  Model | #Params | Prompt | Shakespeare | Romance | Humor | Country | Sarcasm | Age | Avg.
---|---|---|---|---|---|---|---|---|---
_0-Shot (without any fine-tuning)_
Flan-T5 | 250M | No Lexicon | 33.36 | 33.33 | 33.33 | 43.15 | 33.33 | 33.92 | 35.07
LexPT${}_{\text{I,Rw}}$ | 49.95 | 51.30 | 48.66 | 35.34 | 49.40 | 49.02 | 47.28
[1pt/1pt] T0 | 3B | No Lexicon | 33.36 | 50.72 | 44.58 | 49.30 | 33.32 | 45.15 | 42.74
LexPT${}_{\text{I,Rw}}$ | 49.92 | 53.54 | 49.02 | 37.13 | 50.73 | 48.55 | 48.15
[1pt/1pt] GPT-J | 6B | No Lexicon | 33.36 | 33.33 | 33.33 | 49.30 | 33.33 | 35.30 | 36.33
LexPT${}_{\text{I,Rw}}$ | 47.84 | 41.63 | 48.75 | 40.10 | 49.98 | 49.73 | 46.34
[1pt/1pt] GPT-3.5 | 175B | No Lexicon | 71.26 | 99.00 | 50.26 | 17.01 | 50.59 | 35.33 | 53.91
LexPT${}_{\text{I}}$ † | 76.18 | 95.00 | 55.85 | 51.39 | 55.99 | 25.22 | 59.94
_Transfer 0-Shot (fine-tuned only on source styles)_
T5 | 220M | No Lexicon | 33.31 | 43.57 | 36.43 | 19.86 | 33.37 | 35.75 | 33.72
LexPT${}_{\text{I}}$ | 33.31 | 69.62 | 36.82 | 24.68 | 33.33 | 42.31 | 40.01
LexPT${}_{\text{R\\#}}$ | 48.03 | 74.42 | 46.94 | 65.96 | 47.61 | 51.62 | 55.76
LexPT${}_{\text{I,Rw}}$ | 55.10 | 78.98 | 60.56 | 49.09 | 49.25 | 50.80 | 57.30
[1pt/1pt] GPT-J | 6B | No Lexicon | 58.16 | 87.82 | 33.33 | 53.11 | 44.10 | 35.25 | 51.96
LexPT${}_{\text{I}}$ | 45.23 | 94.00 | 33.59 | 50.01 | 51.30 | 35.44 | 51.60
LexPT${}_{\text{R\\#}}$ | 39.54 | 63.11 | 48.60 | 49.04 | 40.60 | 41.51 | 47.07
LexPT${}_{\text{I,Rw}}$ | 56.76 | 83.99 | 55.86 | 44.97 | 48.84 | 47.47 | 56.32
_Oracle Supervised Classifier (fine-tuned directly on target styles)_
BERT | 110M | - | 91.05 | 99.0* | 97.3* | 54.5* | 73.1* | 65.89 | 80.14
T5 | 220M | - | 90.76 | 98.0* | 97.4* | 51.8* | 73.7* | 65.08 | 79.46
Majority Baseline | - | 33.3 | 33.3* | 33.3* | 49.2* | 33.3* | 35.30 | 36.28
Table 3: Zero-shot performance on target styles in F1 scores. In _0-shot_
setting, LexPT${}_{\text{I,Rw}}$ outperforms other prompting methods on all
models except LexPT${}_{\text{I}}$ on GPT-3.5 (†: see more results in Appendix
E.1). In _transfer 0-shot_ , each method trains on the source styles before
being evaluated on the targets. All LexPTs use five lexicon words in each
prompt. To gauge the level of difficulty in learning different styles, we
evaluate the Oracle classifier by fine-tuning a model on an individual target
style (∗: reported by Kang and Hovy (2021)), and the majority classifier by
choosing the majority label in training data. Bold highlights the best method
under each experiment setting.

##  4 Experimental Results

We follow the XSLUE benchmark Kang and Hovy (2021) to report macro-average F1
on style classification tasks. Our experimental results reveal that utilizing
lexicons in the prompts can improve cross-style transfer performance in all
three settings. We observe that class randomization plays a helpful role in
achieving this improvement, especially when source style fine-tuning is
involved.

###  4.1 Zero-shot Learning Results

Table 3 shows both zero-shot and transfer zero-shot learning results for
different methods.

##### LexPT outperforms the No Lexicon prompts in the zero-shot setting
without fine-tuning.

In the zero-shot setup, LexPT${}_{\text{I,Rw}}$ demonstrates a significant
advantage over the standard prompts without lexicon information (i.e., No
Lexicon) across all the experimented models, except for GPT-3.5 where
LexPT${}_{\text{I}}$ outperforms all the other prompts. While GPT-3.5 exhibits
impressive performance in the absence of lexicons, by simply integrating
lexicons into the prompts (i.e., LexPT${}_{\text{I}}$), its performance can be
further enhanced in four out of six styles. Notably, the F1 score improves
from 17.01 to 51.39 for Country (“British” or “American” English).

##### Instruction-tuning on source styles improves performance on unseen
styles when LexPT is used.

Both T5 and GPT-J demonstrate a consistent improvement in the transfer zero-
shot setting over the zero-shot setting (without any fine-tuning) when
comparing models of similar size, such as T5 and Flan-T5, as well as GPT-J.
This suggests the benefits of instruction-tuning on source styles and the
effectiveness of cross-style transfer learning.

Interestingly, T5 and GPT-J exhibit distinct performance patterns across
different prompt designs. For T5, incorporating lexicons into prompts and
randomizing classes can consistently improve transfer learning performance.
The best LexPT${}_{\text{I, Rw}}$ method outperforms No Lexicon prompts by
23.58 F1 points on average. One possible explanation is that, if we fix the
class names during source fine-tuning, the model tends to memorize these names
instead of learning from other signals. This is not ideal as our goal is to
predict the class for unseen styles and thus learning from the lexicon is
important. By randomizing the class names during instruction tuning, the model
is able to focus more on the lexicons and other common information shared
across styles (e.g., instructions) rather than style-specific tokens (e.g.,
class names). This suggests that format transfer, i.e., classification based
on the relevance between each style lexicon and the input sentence, is
crucial. More experiments on class name randomization are discussed in §4.3.
In contrast, the performance of GPT-J is more varied, where the benefit of
LexPT is not as consistent as it has for T5. This is probably because GPT-J
has a different architecture. It is also possible that source-style fine-
tuning has a counterproductive effect on certain styles within the GPT-J
framework.

Shakespeare Romance Humor Country Sarcasm Age Avg. Examples w/ _random_ labels
MetaICL4 44.37±6.99 56.21±26.64 37.82±5.02 41.84±18.46 35.55±2.94 40.96±11.19
42.79 MetaICL4+Lex 39.80±1.47 64.58±18.72 38.59±4.41 49.72±0.44 43.77±6.52
35.30±0.00 45.29 LexPT${}_{\text{I, Rw}}$+4 54.97±0.52 83.63±4.76 58.11±2.81
49.07±0.48 47.98±0.61 46.44±0.97 56.70 MetaICL16 55.49±11.66 66.91±20.48
36.11±4.58 7.74±4.67 33.33±0.00 31.24±0.00 38.47 LexPT${}_{\text{I, Rw}}$+16
56.68±2.71 66.87±17.72 57.69±1.93 51.67±0.76 45.67±3.71 47.81±1.62 54.40
Examples w/ _gold_ labels MetaICL4 64.30±13.01 53.53±27.30 49.79±12.46
49.29±0.01 34.28±1.57 36.21±1.25 47.90 MetaICL4+Lex 43.90±8.06 75.80±6.52
42.78±3.99 49.42±0.36 38.62±3.69 35.30±0.00 47.63 LexPT${}_{\text{I, Rw}}$+4
54.42±1.78 85.48±3.00 58.83±4.93 48.92±0.43 43.11±4.91 45.84±1.94 56.10
MetaICL16 72.93±8.15 95.79±0.84 52.05±8.52 47.90±3.07 33.33±0.00 35.30±0.00
56.22 LexPT${}_{\text{I, Rw}}$+16 60.99±6.75 94.00±1.41 63.26±3.35 51.85±0.41
44.93±4.34 47.42±4.54 60.41

Table 4: Few-shot performance of GPT-J. The subscript of MetaICL represents
the number ($K$) of demonstrations in one prompt. For each method
(MetaICL${}_{\text{K}}$, MetaICL${}_{\text{K}}$+Lex, or LexPT+K), we choose a
set of $K$ examples with five different random seeds. By introducing lexicons
into prompts, the standard deviation of performance across five runs generally
decreases. More results on varying values of $K$ are shown in Appendix E.2.
Figure 2: Performance of different LexPT variants on T5. LexPT with class
Randomization are better than those without. LexPT with natural language
instructions are generally better than those without.

###  4.2 Few-shot Learning Results

We present the performance of different few-shot learning methods in Table 4.

##### Incorporating lexicons in few-shot learning reduces the sensitivity to
example selection.

One interesting observation is that by introducing lexicon into prompts
(LexPT+K and MetaICL${}_{\text{K}}$+Lex), the standard deviation of
performance across five runs generally decreases. For example, MetaICL4
performs extremely unreliably on Romance with a high standard deviation of
27.30 across five runs, while MetaICL4+Lex not only improves the performance
but also stabilizes the inference with the standard deviation dropped to 6.52.
This may suggest that introducing lexicons in prompting can reduce a model’s
dependence on the selected few-shot demonstrations (Liu et al., 2022).

##### Introducing lexicons into prompts can be beneficial when gold labels are
not available.

When the gold examples of target styles are inaccessible (i.e., _random_
labels), introducing lexicon into MetaICL is generally more useful than
increasing the number of examples. For example, the average performance drops
by 4.32 when $K$ increases from 4 to 16, which is in contrast with a
performance improvement of MetaICL4+Lex over MetaICL4. It can also be seen
that LexPT+K generally outperforms the other methods when considering examples
with random labels. These may suggest that lexicons can provide a useful
signal for the prediction of unseen styles when gold labels of few-shot
demonstrations are absent.

On the other hand, when ground-truth labels are known, MetaICL16 shows
superior performance in Shakespeare and Romance, while LexPT${}_{\text{I,
Rw}}$+16 outperforms the other methods in Humor, Country, Sarcasm, Age, and
demonstrates the highest average performance of 60.41 F1 points. This suggests
that different methods may excel in different target styles, and the best
method might vary depending on the specific style targeted.

###  4.3 Comparison of LexPT Variants

We study the impact of instructions and class randomization in LexPT by
comparing the eight variants (§3.5) listed in Table 2. Our experiment results
in Figure 2 show that introducing class randomization can improve the
performance of transfer learning consistently across all six styles. For
example, in the absence of instructions, the average F1 improves from 35.58
(LexPT0) to 50.54 (LexPT${}_{\text{R\\#}}$), while with instructions, the
average F1 improves from 41.72 (LexPT${}_{\text{I}}$) to 51.23
(LexPT${}_{\text{I,R\\#}}$). Moreover, by comparing LexPT${}_{\text{R\\#}}$
with LexPT${}_{\text{Rw-}}$, and LexPT${}_{\text{I,R\\#}}$ with
LexPT${}_{\text{I,Rw-}}$, we find that prompts without instructions usually
work better with numeric index labels, while those with instructions are
usually better with word labels.

Figure 3: Impact of the number ($m$) of lexicon words or phrases used in each
LexPT. src-$m$ is for fine-tuning on source styles and tgt-$m$ for evaluation
on targets.

##  5 Further Analyses

To gain a better understanding, we study the following aspects of LexPT: (i)
the number of lexicon words included in each prompt, (ii) choice of prompt
templates, (iii) choice of lexicon sources, and (iv) model size.

##### Impact of the number of lexicon words.

When predicting a target style, the zero-shot LexPT model only has access to a
subset ($m$) of lexicon words that express or imply the style classes rather
than example sentences. To investigate the model’s dependence on the number of
lexicon words, we take LexPT${}_{\text{I,Rw}}$ and incrementally increase $m$
from $0$ to $30$ in both fine-tuning and inference phases. Figure 3 shows a
general trend that the average F1 of six targets initially increases with
increasing $m$, but then either drops or stabilizes. Figure 7 provides a
detailed view of this trend for each individual style, revealing that
different styles reach their peak performance at different values of $m$. On
average, our method performs the best when $m=5$.

Moreover, we fix the model fine-tuned using LexPT${}_{\text{I,Rw}}$ with
$m=5$, and then gradually increase $m$ while evaluating target styles. A
similar trend is noticed in Figure 3. It can also be seen that when target
styles have no lexicon resources ($m=0$), increasing the number of lexicon
words in each prompt during source fine-tuning might be beneficial. For
instance, src-5, tgt-0 improves the performance of src-0, tgt-0 by an average
of 3.96 F1 points.

##### Impact of prompt templates.

Prior works find that prompting a large LLM on an unseen task is extremely
sensitive to the prompt design, such as the wording of prompts Sanh et al.
(2021). To investigate the sensitivity of LexPT, we experiment with four
different prompt templates t1, t2, t3, t4 (Table 13), each of which contains
natural language instructions. For each template, we fine-tune a model on the
source styles using LexPT. Table 5 shows that LexPT${}_{\text{I}}$ is
sensitive to the choice of prompt templates. However, after introducing class
randomization, LexPT${}_{\text{I, Rw}}$ improves the average F1 across the
explored templates by a substantial margin while also reducing the standard
deviation across them, which indicates that it is more robust to the wording
choices of prompts.

dict nlp+chat class human Avg. SD. w/o rand. (_LexPT ${}_{\text{I}}$_) t1
42.55 43.88 41.99 41.64 42.52 0.99 t2 39.05 41.72 33.71 41.56 39.01 3.74 t3
35.40 40.21 36.13 38.69 37.61 2.24 t4 30.43 36.33 37.02 36.14 34.98 3.06 Avg.
36.86 40.54 37.21 39.51 SD. 5.18 3.18 3.48 2.63 w/ rand. (_LexPT
${}_{\text{I,Rw}}$_) t1 54.20 54.72 53.16 55.15 54.31 0.86 t2 54.74 54.23
50.24 54.83 53.51 2.20 t3 53.24 52.17 51.59 53.85 52.71 1.02 t4 55.83 51.89
55.02 53.91 54.16 1.71 Avg. 54.50 53.25 52.50 54.44 SD. 1.08 1.43 2.06 0.66

Table 5: For each combination of the lexicon source and the prompt template,
class randomization (LexPT${}_{\text{I,Rw}}$) consistently improves the
average scores. t1, t2, t3 and t4 are four different templates detailed in
Appendix Table 13. dict, nlp+chat, class and human are the different lexicon
sources described in §5. Each white cell reports the result averaged over the
six styles in the target split. Light grey cells indicate the average and the
standard deviation scores over four lexicon sources. Dark grey cells represent
the average and the standard deviation over four templates.

##### Impact of lexicon source.

Given the importance of lexicon shown in previous experiments, we study the
impact of lexicon choices on LexPT. We fix all other aspects (e.g., the number
of lexicon words, prompt template, etc.) and consider lexicons from the
following sources: (1) dict: all lexicons are from dictionary; (2) nlp+chat:
for classes that have NLP lexicons, we directly use them, whereas for those
without, we create ones using ChatGPT; (3) class: each class lexicon contains
only its class name, e.g., the humorous lexicon has a single word humorous;
(4) human: we have a native English speaker create a lexicon for each class.

Table 5 shows that without class randomization, the average F1 for nlp+chat
across four templates is higher than that of dict, class and human, at 40.54.
After randomization, dict performs the best with an average score of 54.50.
Moreover, randomizing the classes with words in LexPT consistently improves
the average F1 while reducing the standard deviation across the four lexicon
sources, regardless of the prompt templates employed. Another interesting
observation is that human, the human-created lexicons, is most robust to the
change of prompt templates, both with and without randomization.

Figure 4: Scaling behavior of LexPT${}_{\text{I,Rw}}$.

##### Impact of model scale.

We study how LexPT behaves as model scale increases by varying the size of T5
models: t5-small (60M), t5-base (220M), t5-large (770M) and t5-3b (3B).
Results are reported in Figure 4. Some styles perform better on larger models,
while others prefer smaller models. For example, Humor behaves worse as model
scales up. The unexpected scaling behavior contradicts the common observation
in most NLP tasks where performance improves with larger models.

In order to figure out the possible reasons, we took a closer look at the
training process in which we found that larger models have larger training
loss but better validation scores on source styles. So we tuned the
hyperparameters (e.g., the number of epochs, learning rate) and report the
results in Appendix E.5. Interestingly, the scaling curves on most styles do
not change. Moreover, during tuning t5-large and t5-3b, we observed a trend
where smaller training losses correlate with better validation F1 (on source
styles) but worse test F1 (on target styles).

One possible explanation is that the larger models may overfit to source
styles. During the fine-tuning process, the larger model might prioritize
minimizing loss by fitting features from the training data, rather than format
learning, which is crucial for our task. The model is expected to understand
the format of LexPT (i.e., the use of lexicons) and predict unseen styles
through format transfer. This discrepancy in prioritization could potentially
elucidate the observed performance improvement in the source styles, but a
decline in performance when dealing with unseen styles. Further insights can
be derived from the recent work Wei et al. (2022), which explored eleven
inverse scaling tasks. The study showed that even the same task can have
different scaling curves based on the type of prompting used and the setup
(e.g., zero-shot or few-shot learning). They also found that the inverse
scaling trend may not necessarily hold when the model is sufficiently large.

##  6 Related Work

##### Style classification.

Research in NLP has studied various styles. Kang and Hovy (2021) provided a
benchmark that combines mostly existing datasets for cross-style learning,
such as formality Rao and Tetreault (2018), sarcasm Khodak et al. (2018),
Hate/Offense (i.e., toxicity) Davidson et al. (2017) and sentiment Socher et
al. (2013). Other writing styles include but are not limited to readability
(i.e., simplicity) Arase et al. (2022), Shakespearean English Xu et al.
(2012), subjectivity Pang and Lee (2004), biasedness Pryzant et al. (2020) and
engagingness Jin et al. (2020). Despite an extensive range of style
classification tasks studied in prior research, zero-shot or cross-style
classification is relatively underexplored Puri and Catanzaro (2019); Kang and
Hovy (2021). Furthermore, much of the cross-style research thus far has
predominantly focused on generation tasks, rather than classification Jin et
al. (2022); Zhou et al. (2023). In this study, we aim to address this research
gap by concentrating on zero-shot and few-shot style classification across a
collection of different styles.

##### Language model prompting.

Large language models (LLMs) like GPT-3 Brown et al. (2020b) demonstrated
impressive potential for zero-shot learning by conditioning on an appropriate
textual context, also known as prompts. Since then, how to design appropriate
prompts has become a popular line of research Liu et al. (2022); Sanh et al.
(2021). In this work, we propose to introduce lexicon into prompts. In order
to encourage format transfer to arbitrary target styles, we replace classes
with random words when fine-tuning on source styles. This is similar to Zhao
et al. (2022), which indexes and shuffles the slot description in natural
language for dialogue state tracking. We focus on style classification.
Moreover, LexPT enriches the instructions by including a distinct lexicon for
each style class. In addition, to enhance cross-style transfer, we substitute
the class names with entirely random words. This differs from the practice
observed in previous studies Min et al. (2022b); Zhao et al. (2022); Wei et
al. (2023), where a predefined set of names, precisely equal in size to the
number of labels in the datasets, was used. Examples of such predefined sets
include numeric indices Zhao et al. (2022) or specific labels like
Apple/Orange Min et al. (2022b). Recently, Zhou et al. (2023) specified the
target styles in the instructions as constraints to improve controlled text
generation. LexPT is different as it uses lexicon to define the style.

##  7 Conclusion & Discussions

In this work, we study zero-shot and few-shot style classification using
different prompting methods. Experiments show that conventional prompts often
struggle to transfer knowledge across diverse styles, especially with
moderately large models. However, our analyses show that by incorporating the
appropriate style lexicons and leveraging class randomization in the prompts,
it is possible to instruction-tune a modest-sized model (e.g. t5-base) to
infer unseen styles effectively. It also shows an interesting promise of using
prompt design and fine-tuning to make the language model robust to unseen
domains. Moreover, we speculate that our approach may generalize to other
classification tasks and even generation tasks (e.g., cross-style transfer),
and we would like to dig deeper into this hypothesis in future work.

## Limitations

In our method, we leverage the lexicons we have collected (as detailed in
Table 1). However, it is important to acknowledge that a potential limitation
of our approach lies in the possibility of different performance outcomes when
using lexicons of varying qualities. While we have conducted comparisons
between lexicons from different sources in §5, it is plausible that utilizing
different lexicons could yield different results. Another limitation is that
we only include a limited set of styles for evaluation due to space
constraints. Different styles may behave differently. We leave evaluating on a
more diverse set of styles as future work.

## Acknowledgments

This research is supported in part by the NSF (IIS-2052498, IIS-2144493 and
IIS-2112633), ODNI and IARPA via the HIATUS program (2022-22072200004). The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies, either
expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for governmental
purposes notwithstanding any copyright annotation therein.

## References

  * Ahn (2005) Luis Von Ahn. 2005.  Useful resources: Offensive/profane word list.  https://www.cs.cmu.edu/~biglou/resources/.
  * Arase et al. (2022) Yuki Arase, Satoru Uchida, and Tomoyuki Kajiwara. 2022.  CEFR-based sentence difficulty annotation and assessment.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6206–6219, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a.  Language models are few-shot learners.  _Advances in neural information processing systems_.
  * Brown et al. (2020b) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen and Ritter (2021) Yang Chen and Alan Ritter. 2021.  Model selection for cross-lingual transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  PaLM: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * CrowdTruth (2016) CrowdTruth. 2016.  Short Text Corpus For Humor Detection.  http://github.com/CrowdTruth/Short-Text-Corpus-For-Humor-Detection.  [Online; accessed 1-Oct-2019].
  * Danescu-Niculescu-Mizil et al. (2013a) Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013a.  A computational approach to politeness with application to social factors.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 250–259, Sofia, Bulgaria. Association for Computational Linguistics.
  * Danescu-Niculescu-Mizil et al. (2013b) Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013b.  No country for old members: User lifecycle and linguistic change in online communities.  In _Proceedings of the 22nd international conference on World Wide Web_.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the 11th International AAAI Conference on Web and Social Media_ , ICWSM ’17, pages 512–515.
  * Hovy (1987) Eduard Hovy. 1987.  Generating natural language under pragmatic constraints.  _Journal of Pragmatics_ , 11(6):689–719.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jin et al. (2022) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022.  Deep Learning for Text Style Transfer: A Survey.  _Computational Linguistics_ , 48(1):155–205.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020.  Hooks in the headline: Learning to generate headlines with controlled styles.  _arXiv preprint arXiv:2004.01980_.
  * Kang and Hovy (2021) Dongyeop Kang and Eduard Hovy. 2021.  Style is not a single variable: Case studies for cross-style language understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Khodak et al. (2018) Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018.  A large self-annotated corpus for sarcasm.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Maddela and Xu (2018) Mounica Maddela and Wei Xu. 2018.  A word-complexity lexicon and a neural readability ranking model for lexical simplification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3749–3760, Brussels, Belgium. Association for Computational Linguistics.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020.  Text classification using label names only: A language model self-training approach.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022a.  MetaICL: Learning to learn in context.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2791–2809, Seattle, United States. Association for Computational Linguistics.
  * Min et al. (2022b) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022b.  Rethinking the role of demonstrations: What makes in-context learning work?  In _EMNLP_.
  * Mohammad et al. (2013) Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013.  Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets.  In _Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_.
  * Mohammad (2021) Saif M. Mohammad. 2021.  Sentiment analysis: Automatically detecting valence, emotions, and other affectual states from text.  In Herb Meiselman, editor, _Emotion Measurement (Second Edition)_. Elsevier.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.
  * Pryzant et al. (2020) Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020.  Automatically neutralizing subjective bias in text.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(01):480–489.
  * Puri and Catanzaro (2019) Raul Puri and Bryan Catanzaro. 2019.  Zero-shot text classification with generative language models.  _arXiv preprint arXiv:1912.10165_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018.  Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.  _CoRR_ , abs/2110.08207.
  * Sim et al. (2013) Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013.  Measuring ideological proportions in political speeches.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 91–101, Seattle, Washington, USA. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Taboada et al. (2011) Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011.  Lexicon-based methods for sentiment analysis.  _Computational linguistics_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2010) Tong Wang, Julian Brooke, and Graeme Hirst. 2010.  Inducing lexicons of formality from corpora.
  * Wei et al. (2022) Jason Wei, Yi Tay, and Quoc V. Le. 2022.  Inverse scaling can become u-shaped.  _ArXiv_ , abs/2211.02011.
  * Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023.  Larger language models do in-context learning differently.  _arXiv preprint arXiv:2303.03846_.
  * Wilson et al. (2005) Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005.  Recognizing contextual polarity in phrase-level sentiment analysis.  In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_ , pages 347–354, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2012) Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012.  Paraphrasing for style.  In _COLING_ , pages 2899–2914.
  * Zhao et al. (2022) Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022.  Description-driven task-oriented dialog modeling.
  * Zhong et al. (2021) Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021.  Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Zhou et al. (2023) Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023.  Controlled text generation with natural language instructions.  _arXiv preprint arXiv:2304.14293_.

##  Appendix A More Details About the Datasets

The XSLUE benchmark, designed for exploring cross-style language
understanding, encompasses 15 styles Kang and Hovy (2021). We choose 10
writing styles from XSLUE based on their suitability for our task.
Specifically, we consider the task type (i.e., whether the task is
classification or not), task granularity (e.g., whether the annotated style is
sentence-level or not), expressiveness at both the word and phrase level
(i.e., the possibility of expressing a style with lexicons). For example, the
TroFi dataset for style Metaphor is not used because it is focused on the
literal usage of one specific verb in a sentence. Take the verb drink as an
example, it is a literal expression in the sentence I stayed home and drank
for two years after that, he notes sadly, whereas in So the kids gave Mom a
watch, said a couple of nice things, and drank a retirement toast in her
honor, drink is non-literal.

##  Appendix B Create Lexicons with ChatGPT

For style classes that do not have an available NLP lexicon, we prompt ChatGPT
to generate a list of words or phrases that express or imply that specific
style. Table 6 shows the prompts used. Figure 5 presents some examples of
ChatGPT output.

Style | Class | ChatGPT Prompt
---|---|---
Politeness | impolite |  | Give me 10 words that show impolite style.
---
Give me 20 words or short phrases that people may use when they show impolite
attitude towards others.
Romance | literal |  | What’s the difference between literal text and romantic text?
---
Give me 20 words or short phrases that show the literal style rather than
romantic
style.
Humor | humorous |  | Give me 10 words that show humorous style.
---
Give me 20 words or short phrases that people may use in text to show humor.
literal |  | What’s the difference between literal text and humorous text?
---
Give me 20 words or short phrases that show the literal style rather than
humorous style.
Sarcasm | sarcastic |  | Give me 10 words that show sarcastic style.
---
Give me 20 words or short phrases that people may use in text to show sarcasm.
literal |  | What’s the difference between literal text and sarcastic text?
---
Give me 20 words or short phrases that show the literal style rather than
sarcastic
style.
Age | under12 |  | Give me some words or phrases that an under-12-year-old child might say or write.
---
What words or phrases can a child under 12 say?
Imagine that you are 8 years old, what words or phrases do you often use in
communication and writing?
12-17 |  | Give me some words or phrases that people aged 12-17 might say or write.
---
What words or phrases can a teenager aged 12-17 say?
Imagine that you are 14 years old, what words or phrases do you often use in
communication and writing?
18-24 |  | Give me some words or phrases that people aged 18-24 might say or write.
---
What words or phrases can a person aged 18-24 say?
Imagine that you are 21 years old, what words or phrases do you often use in
communication and writing?
25-34 |  | Give me some words or phrases that people aged 25-34 might say or write.
---
What words or phrases can a person aged 25-34 say?
Imagine that you are 30 years old, what words or phrases do you often use in
communication and writing?
35-44 |  | Give me some words or phrases that people aged 35-44 might say or write.
---
What words or phrases can a person aged 35-44 say?
Imagine that you are 40 years old, what words or phrases do you often use in
communication and writing?
45-54 |  | Give me some words or phrases that people aged 45-54 might say or write.
---
What words or phrases can a person aged 45-54 say?
Imagine that you are 50 years old, what words or phrases do you often use in
communication and writing?
55-74 |  | Give me some words or phrases that people aged 55-74 might say or write.
---
What words or phrases can a person aged 55-74 say?
Imagine that you are 65 years old, what words or phrases do you often use in
communication and writing?
| 75Years
---
OrOlder
| Give me some words or phrases that people aged 75 or older might say or
write.
---
What words or phrases can a person aged 75 or older say?
Imagine that you are 80 years old, what words or phrases do you often use in
communication and writing?
Table 6: Prompts used for ChatGPT to generate lexicon. Since we map the _Age_
dataset to a binary one, we also map the corresponding lexicons of its
original age classes to the new classes. For example, the _youthful_ lexicon
contains the contents of _Under12_ , _12-17_ and _18-24_ lexicons.

(a) impolite

(b) under12

(c) 12-17

Figure 5: Examples of ChatGPT output for different style classes.

##  Appendix C Examples and Details of Lexicons

Recall in §3.2, we discuss the difference sources from which we collected a
lexicon. Table 7 shows more examples of lexicons from different sources.
Statistics of NLP and ChatGPT lexicons are presented in Table 8.

Style Class Lex Src Lex NLP admittedly, albeit, insofar… formal Dict in
accordance with rules of convention or etiquette; official NLP dude, kinda,
sorta, repo… Formality informal Dict having a relaxed, friendly, or unofficial
style ChatGPT funny, laugh-out-loud, silly… humorous Dict being comical,
amusing, witty human chuckle, wisecrack, hilarious… ChatGPT grim, formal,
solemn, dour… Humor literal Dict not humorous; serious human analysis,
scrutinize, enforce…

Table 7: Examples of lexicons. Class represents the category in a style. Each
lexicon contains words or phrases that express or describe the class. Lex Src
indicates how the lexicon is collected (§3.2).

Style Class Lex Src Lex Size (# of words/phrases) Age youthful ChatGPT 98
mature ChatGPT 65 Country U.K ChatGPT 131 U.S.A ChatGPT 127 Formality formal
NLP 330 informal NLP 370 hate NLP 178 Hate/Offense offensive NLP 1403 neither
ChatGPT 5 Humor humorous ChatGPT 21 literal ChatGPT 6 Politeness polite NLP
110 impolite ChatGPT 54 Politics LeftWing NLP 2581 Centrist NLP 1231 RightWing
NLP 2416 Readability simple NLP 10290 complex NLP 4890 Romance romantic
ChatGPT 58 literal ChatGPT 5 Sarcasm sarcastic ChatGPT 34 literal ChatGPT 2
Sentiment positive NLP 204 negative NLP 292 Shakespeare shakespearean NLP 1524
modern NLP 1524 Subjectivity subjective NLP 5569 objective NLP 2653

Table 8: Statistics of lexicons.

##  Appendix D Implementation Details

We use PyTorch Paszke et al. (2019) and Huggingface Transformers Wolf et al.
(2020) in the experiments. In transfer zero-shot experiments, we first fine-
tune a model on the source styles before evaluating it on the target styles.
Hyperparameters are selected following the common practices in previous
research. Table 9 reports the hyperparameters for source style fine-tuning.

Hyperparameter | T5-based Models | GPT-J
---|---|---
optimizer | Adafactor | Adam
learning rate | 1e-4 | 1e-5
batch size | 8 | 4
max encoder/input length | 512 | 512
max decoder/target length | 16 | 16
# epochs |  | Prompts with word randomization: 5
---
Other prompts: 3
1
Table 9: Hyperparameters of source-style fine-tuning. Note that the number of
epochs depends on the model convergence rate. Prompts with word randomization
converge more slowly than the other prompts, so their epoch is longer.

##  Appendix E Additional Results & Analyses

###  E.1 Zero-shot Results of All Prompting Methods

Table 10 presents the performance of all prompts on each model in a zero-shot
setting (i.e., no source style fine-tuning). It is observed that simply adding
lexicon to the prompts ($i.e$ LexPT${}_{\text{I}}$) is not helpful. After
randomizing the classes in the prompt, both LexPT${}_{\text{R\\#}}$ and
LexPT${}_{\text{I,Rw}}$ have shown improvement, with LP${}_{\text{I,Rw}}$
demonstrating a significant advancement. Moreover, for No Lexicon prompt and
LexPT${}_{\text{I}}$, F1 on many styles are around 33.33. Through an
examination of per-class scores for each target dataset, we discovered that
the main cause is the models’ tendency to consistently predict a single class.
In general, the benefits of LexPT are not obvious in the zero-shot setting.
One possible reason is that these LLMs never saw such prompts during their
pre-training. This also emphasizes the importance of source fine-tuning with
LexPT before zero-shot evaluation, which improves format learning and thereby
enhances the ability of LLMs to do format transfer.

Model | Prompt | Shakespeare | Romance | Humor | Country | Sarcasm | Age | Avg.
---|---|---|---|---|---|---|---|---
Flan-T5 | No Lexicon | 33.36 | 33.33 | 33.33 | 43.15 | 33.33 | 33.92 | 35.07
LexPT${}_{\text{I}}$ | 35.12 | 33.33 | 33.33 | 3.35 | 33.33 | 33.92 | 28.73
LexPT${}_{\text{R\\#}}$ | 32.95 | 32.43 | 33.60 | 33.83 | 33.43 | 33.52 | 33.29
LexPT${}_{\text{I,Rw}}$ | 49.95 | 51.30 | 48.66 | 35.34 | 49.40 | 49.02 | 47.28
T0 | No Lexicon | 33.36 | 50.72 | 44.58 | 49.30 | 33.32 | 45.15 | 42.74
LexPT${}_{\text{I}}$ | 33.35 | 33.33 | 48.97 | 49.30 | 33.33 | 37.50 | 39.30
LexPT${}_{\text{R\\#}}$ | 49.83 | 45.01 | 49.11 | 36.24 | 48.11 | 37.01 | 44.22
LexPT${}_{\text{I,Rw}}$ | 49.92 | 53.54 | 49.02 | 37.13 | 50.73 | 48.55 | 48.15
GPT-J | No Lexicon | 33.36 | 33.33 | 33.33 | 49.30 | 33.33 | 35.30 | 36.33
LexPT${}_{\text{I}}$ | 33.36 | 33.33 | 33.73 | 49.30 | 33.33 | 35.30 | 36.39
LexPT${}_{\text{R\\#}}$ | 45.15 | 35.03 | 38.08 | 40.41 | 33.41 | 37.92 | 38.33
LexPT${}_{\text{I,Rw}}$ | 47.84 | 41.63 | 48.75 | 40.10 | 49.98 | 49.73 | 46.34
GPT-3 | No Lexicon | 71.26 | 99.00 | 50.26 | 17.01 | 50.59 | 35.33 | 53.91
LexPT${}_{\text{I}}$ | 76.18 | 95.00 | 55.85 | 51.39 | 55.99 | 25.22 | 59.94
LexPT${}_{\text{I,Rw}}$ | 31.92 | 71.82 | 35.54 | 23.71 | 42.52 | 31.09 | 39.43
Table 10: Zero-shot learning on target styles.

###  E.2 Varying Number of Training Examples (K) used in Few-Shot Learning

We investigate the impact of the number of examples ($K$) that are used in the
few-shot learning methods MetaICL${}_{\text{K}}$ and LexPT+K. Results are
reported in Figure 6. The performance of both methods deteriorates with an
increase in $K$ when using random labels. However, when gold labels are
utilized for the target-style training examples, the performance improves with
larger $K$, particularly showing significant improvement from $K=8$ to $K=16$.
Moreover, as $K$ increases, the performance disparity between utilizing
ground-truth labels and random labels further expands. These observations show
that the ground-truth input-label mapping is important in our case.

Figure 6: Ablation on the number of training examples ($K$) in a few-shot
learning setting.

###  E.3 Varying Number of Lexicon Words (m)

Figure 7 illustrates the pattern of performance shifts associated with an
increase in the number of lexicon words or phrases incorporated into each
prompt, broken down by each target style.

Figure 7: Impact of the number $m$ of lexicon words or phrases used in each
LexPT. The solid lines represent the cases where $m$ is applied to both source
fine-tuning and target evaluation. The dotted lines (i.e., _Style_ -5) show
the scores of target styles when lexicon size 5 is used for source fine-
tuning, while the size of target-style lexicons $m$ is varied for evaluation.

###  E.4 More Experiments on Style Splits

This section presents additional experimental results of our approach
utilizing the different style splits listed in Table 11. Results are presented
in Table 12.

Split | Source Styles
---|---
style${}_{\text{src1}}$ |  | Politeness, Formality, Sentiment
---
style${}_{\text{src2}}$ |  | Politeness, Formality, Sentiment,
---
Hate/Offense
style${}_{\text{src3}}$ |  | Politeness, Formality, Sentiment,
---
Hate/Offense, Politics
style${}_{\text{src4}}$ |  | Politeness, Formality, Sentiment,
---
Hate/Offense, Politics, Readability,
Subjectivity
Table 11: Source styles used in different source-target style splits.

Model #Params Prompt style${}_{\text{src1}}$ style${}_{\text{src2}}$
style${}_{\text{src3}}$ style${}_{\text{src4}}$ No Lexicon 36.72 36.27 30.01
33.72 T5 220M LexPT${}_{\text{I,Rw}}$ 53.30 53.27 54.18 57.30 No Lexicon 50.14
53.64 56.06 51.96 GPT-J 6B LexPT${}_{\text{I,Rw}}$ 54.14 56.15 57.52 56.32

Table 12: Average F1 on the six target styles. Across all source-target
splits, LexPT${}_{\text{I,Rw}}$ improves the average performance on unseen
styles compared to No Lexicon prompts for both T5 and GPT-J.

###  E.5 More Experiments on Model Scaling Ablation

To investigate the possible reason why performance on some styles drops on
larger models, we adjust the number of epochs and learning rate on T5-large
and T5-3b. Results are reported in Figure 8.

Figure 8: Scaling behavior of target styles. If not specified, the number of
epochs is 3 and learning rate is 1e-4.

Prompt Template $T(\cdot)$ Input Output t1 Which style best describes the
sentence “{sentence}”? styles: \- {className1}: {$e_{1},...,e_{k}$} \-
{className2}: {$e_{1},...,e_{k}$} … classNamei t2 In this task, you are given
sentences. The task is to classify a sentence as "{className1}" if the style
of the sentence is similar to the words "{$e_{1},\cdots,e_{k}$}" or as
"{className2}" if the style of the sentence is similar to the words
"{$e_{1},\cdots,e_{k}$}" or as … Here is the sentence: "{sentence}". t3 The
task is to classify styles of sentences. We define the following styles:
"{className1}" is defined by "{$e_{1},\cdots,e_{k}$}"; "{className2}" is
defined by "{$e_{1},\cdots,e_{k}$}"; …. Here is the sentence: "{sentence}",
which is more like t4 Context: "{className1}" is defined by
"{$e_{1},\cdots,e_{k}$}", "{className2}" is defined by
"{$e_{1},\cdots,e_{k}$}" … Sentence: {sentence} Question: which is the correct
style of the sentence? Answer:

Table 13: Prompt templates with instructions.

##  Appendix F Prompting Templates

###  F.1 Template in Main Experiments

In our main experiments (§4), we conduct a comparative analysis between LexPT
and the standard prompt, specifically the No Lexicon prompt. Both utilize the
template t2 in Table 13 except that No Lexicon does not incorporate any
lexicon sampling. Instead, each slot for the lexicon words contains only the
corresponding class name. Here is an example input of No Lexicon prompt on
Politeness: _In this task, you are given sentences. The task is to classify a
sentence as "polite" if the style of the sentence is similar to the words
"polite" or as "impolite" if the style of the sentence is similar to the words
"impolite". Here is the sentence: "I’ve just noticed I wrote… and smooth out
the text?"_. Its output is _polite_.

###  F.2 Ablation on Templates with Instructions

In §5, we discuss how the prompt template $T(\cdot)$ affects LexPT. Here we
provide the four templates used for our experiments in Table 13.

Figure 9: Examples of different LexPT on _Politeness_. Red part is
(randomized) classes, the green part represents the words sampled from each
class lexicon, and yellow stands for the input sentence and the uncolored part
is the prompt template $T(\cdot)$.  Figure 10: MetaICL+Lex input consists of
$K$ demonstrations and an input sentence. Each demonstration contains $m$
lexicon words for each class, followed by an example with its label.

##  Appendix G Prompt Examples

Figure 9 shows the example input and output for all LexPT variants. In
MetaICL${}_{\text{K}}$+Lex, one prompt consists of $K$ demonstrations and an
input sentence. Figure 10 provides an example demonstration.