853 1Rensselaer Polytechnic Institute, Troy, NY, USA
2College of the Holy Cross, Worcester, MA, USA

# LLM-augmented Preference Learning from Natural Language

Inwon Kang1, Sikai Ruan1, Tyler Ho1, Jui-Chien Lin1, Farhad Mohsin2,
Oshani Seneviratne1, Lirong Xia1

###### Abstract.

Finding preferences expressed in natural language is an important but
challenging task. State-of-the-art(SotA) methods leverage transformer-based
models such as BERT, RoBERTa, etc. and graph neural architectures such as
graph attention networks. Since Large Language Models (LLMs) are equipped to
deal with larger context lengths and have much larger model sizes than the
transformer-based model, we investigate their ability to classify comparative
text directly. This work aims to serve as a first step towards using LLMs for
the CPC task. We design and conduct a set of experiments that format the
classification task into an input prompt for the LLM and a methodology to get
a fixed-format response that can be automatically evaluated. Comparing
performances with existing methods, we see that pre-trained LLMs are able to
outperform the previous SotA models with no fine-tuning involved. Our results
show that the LLMs can consistently outperform the SotA when the target text
is large – i.e. composed of multiple sentences –, and are still comparable to
the SotA performance in shorter text. We also find that few-shot learning
yields better performance than zero-shot learning.

###### Key words and phrases:

Preference Learning, Natural Language Processing, Large Language Models

##  1\. Introduction

Group decision making is an important task in multi-agent systems, in which a
group of agents aim to make a collective decision based on their preferences.
With the advent of artificial intelligence and machine learning techniques,
finding better ways of making group decisions efficiently and accurate becomes
an important problem Xia (2022). In particular, state-of-the-art natural
language processing (NLP) techniques have been used to improve preference
elicitation, learning, and aggregation. For example, learning preferences from
group discussions such as those in forums or chat rooms can provide an
unobtrusive way of learning preferences and making a group decision. Learning
preferences from text will also help in making better decisions when there are
a large number of alternatives or when there is uncertainty about the
preferences.

Mohsin et al. (2021) proposed a framework for making group decisions from
natural language and created a dataset for texts with preferences. However,
making group decisions turned out to be a difficult problem because predicting
individual preferences is a difficult task by itself. Predicting preferences
in text can be as simple as using simple grammatical rules. For example, look
for expressions such as: “better than”: the sentence “Tea is better than
coffee” expresses a straightforward preference towards tea. However, human
language is capable of much more nuanced expressions as well. Consider the
sentence “I used to think tea is better than coffee. However, that was a while
ago.”. A glance at the sentence may suggest that the author prefers tea over
coffee but the sentence describes the opposite of the preference. While more
than this example is needed to confuse most English speakers, such nuances in
natural language add to the challenges of preference classification with
machine learning (ML) models. Panchenko et al. (2019) and Ma et al. (2020)
proposed methods predicting the preference between two alternatives from
individual pieces of texts using modern machine learning techniques such as
transformer-based word embeddings Pennington et al. (2014); Devlin et al.
(2019) and graph neural networks. However, the benchmark dataset Panchenko et
al. (2018) considered only simple texts consisting of single sentences, and
all texts contained mention of both alternatives. Haque et al. (2022) and
Mohsin et al. (2023) proposed methods to improve these performances for more
complex but realistic text that included implicit mention of alternatives and
multi-sentence texts. However, unlike tasks like sentiment analysis and stance
detection, where these NLP techniques achieved high accuracy, predicting
preferences is a more difficult task with relatively low accuracy levels.

Large language models (LLMs) and foundation models have brought in a new wave
of improvements in artificial intelligence systems. There is potential to use
these LLMs in various multi-agent scenarios for different problems, including
negotiation, delegation and making group decisions. Quite recently, Meta
released a new feature in their chatting app, that allows an LLM-powered
chatbot to help make group
decisions111https://about.fb.com/news/2023/09/introducing-ai-powered-
assistants-characters-and-creative-tools/. However, preference aggregation,
deliberation, etc. depends on the language model’s ability to identify
preferences expressed in the text. The question is:

Can LLMs identify comparative preferences in texts?

Our Contributions We investigate popular LLM’s ability to identify and predict
preferences in text in this work. For this, we use two benchmark datasets:
First, the Compsent-19 dataset Panchenko et al. (2019) which has single
sentence texts with mentions of both alternatives, and second, the College
Confidential dataset, which contains more complex texts. In particular, we
experiment with two versions of Meta’s LLaMa-2 model Touvron et al. (2023),
the 13B-parameter version and the 4-bit quantized 70B-parameter version. We
also considered OpenAI’s popular GPT-3.5-Turbo model and GPT-4 model OpenAI
(2023). We design and experiment with different prompts that ask the LLMs to
predict the preference expressed in the text to determine what type of prompt
results in the best prediction performance. Our key findings are the
following.

Key finding 1: LLMs can outperform previous state-of-the-art models.

Results from our experiments show that the LLMs are able to outperform
previous state-of-the-art models by just using few examples. The best
performance comes from the largest model (GPT-4), but the results show that
even the smaller models (LLaMa-2-70B) are still able to outperform the state-
of-the-art models. In the College Confidential tasks where the text is longer
and has a more complex grammatical structure, almost all instances of the
prompt/example configuration on both GPT-4 and LLaMa-2-70B outperform the
state-of-the-art performance.

Key finding 2: Few-shot learning outperforms zero-shot learning.

We use the zero-shot and few-shot prompts in combination with different styles
of instruction to arrive at this conclusion. Our prompt design was motivated
by work in LLM literature that indicated that LLMs can behave as zero-shot
Kojima et al. (2022); Wei et al. (2022a) or few-shot Brown et al. (2020)
learners. Our results show that few-shot learning almost always outperforms
zero-shot learning in most models. We also find that smaller models may
struggle with handling both the few-shot examples and longer prompts. But if
the model is powerful enough – e.g. GPT-4 –, providing detailed instruction
with real-life examples yields the best performance.

Key finding 3: LLMs have superior performance in large and complex texts.

The text in College Confidential dataset contains multiple sentences, complex
grammatical structure, and pronouns; many mention the same entity more than
once. All of these make College Confidential a challenging dataset.
Fortunately, we found that LLMs have the ability to handle large and complex
context, especially the LLMs with large number of parameters. The size of the
models proved to have some effect on the performance. In the case of the
smallest model (LLaMa-2-13B), getting it to predict preferences in a well-
formatted way was difficult, and when it did predict preferences, it was
mostly incorrect. On the one hand, for the LLaMa-2-70B model – which is
smaller than GPT-3.5-Turbo–, we developed a retry prompt process that is an
iterative process (Figure 1) that manages to get well-formatted predictions
from the LLM. On the other hand, we did not need this iterative process for
OpenAI’s GPT-3.5-Turbo and GPT-4 as it produced perfectly formatted responses
which allowed for automatic evaluation. Interestingly, both LLaMa-2-70B and
GPT-4 significantly outperformed state-of-the-art methods using transformer-
based embeddings and graph neural networks for both simple and complex texts
(the Compsent-19 and College Confidential datasets). We find that both LLMs
perform similarly for Compsent-19 (the single sentence benchmark). And for
College Confidential, GPT-4 performs better than LLaMa-2-70B.

While these findings are not surprising given the versatility of LLMs, we note
that these performances were a result of non-trivial deliberation. Our
methodology allows the LLM to be used without human intervention. Because the
LLMs do not output in a fixed format – only restricted to English language –,
we design a method to coerce the model to output its responses in a fixed
format which can then be integrated into a larger pipeline in an automated
way.

This work aims to serve as the first step towards integrating LLMs in the CPC
task. While the previous state-of-the-art models have their advantage in some
aspects, our results show that LLMs are able to effectively handle the CPC
task with proper prompting techniques and are able to outperform them, even
without fine-tuning.

###  1.1. Related Works

Preference learning from text. A common approach for learning preferences from
agents when wanting to make a group decision is preference elicitation. This
involves asking interactive questions to efficiently elicit agents’
preferential information that is sufficient for making a group decision (e.g.,
in Boutilier (2002); Conitzer and Sandholm (2002); Mandal et al. (2020); Zhao
et al. (2018)). Xia (2019) gives a good exposition to different preference
learning methods. Mohsin et al. (2021) proposed an unobtrusive framework of
learning preferences from text (such as in chat rooms or forums) to make group
decisions.

Panchenko et al. (2019) built the Compsent-19 dataset with the specific goal
of categorizing comparative sentences and finding expressed preferences.
Panchenko et al. (2019) used pre-trained sentence embeddings (Conneau et al.,
2017; Pennington et al., 2014), among other features, to train for the
classification problem. Since then, other machine learning techniques have
been used for predicting preference. Transformer models (Vaswani et al.,
2017), in particular, have been useful tool in NLP because of their ability to
learn large dimensions of data without overfitting. This has led to newer
embeddings, both at token and sentence levels Devlin et al. (2019); Liu et al.
(2019); Gao et al. (2021). Previous works have used text embeddings and graph
neural networks like GAT(Veličković et al., 2018) for preference
detection/tagging. These models are effective with textual and graph-
structured data. Recent works in preference learning from text (Huang and
Carley, 2019; Ma et al., 2020; Li et al., 2021) have made use of dependency
graphs and graph neural network methods. Li et al. (2021) additionally used
the knowledge transfer technique from the related sentiment analysis task. But
all of these methods were tested on the Compsent-19 benchmark (Panchenko et
al., 2019) and thus dealt with single sentences with explicit mention of
alternatives. On the other hand, Mohsin et al. (2021) and Haque et al. (2022)
introduced new datasets which dealt with implicit preferences, where both
entities might not be mentioned. The College Confidential dataset Mohsin et
al. (2021) in particular contained multi-sentence texts in a discussion
setting. We work with both the simpler Compsent-19 dataset and the more
complex College Confidential dataset.

Large Language Models. The introduction of LLMs and foundation models
(Chowdhery et al., 2022; Radford and Narasimhan, 2018; Touvron et al., 2023;
OpenAI, 2023) such as ChatGPT, LLaMa and Bard have brought forth a new
paradigm in many domains of machine learning. In particular, ChatGPT (OpenAI,
2022) has demonstrated an impressive ability to perform various tasks while
displaying human-like qualities through text. While these models mainly
operate in natural language domain, past works have found that they can be
well suited for other tasks, such as classification or regression (Hegselmann
et al., 2023; Brown et al., 2020; Wei et al., 2022a). However, the input to an
LLM still needs to be encoded into human language, and past works have
experimented with different ways of converting the tasks into natural
language.

Brown et al. (2020) demonstrate that few-shot learning can be utilized to
enhance the performance of LLMs. In few-shot learning, the user includes
correct examples of the given task to help the model’s understanding of the
task. Wei et al. (2022a) show that LLMs can be powerful zero-shot learners
when fine-tuning is applied to the pre-trained weights. Wei et al. (2022b) use
chain-of-thought prompting to guide the LLM towards making a step-by-step
decision and show that this style of prompting can achieve better performance
in tasks such as arithmetic or reasoning-based tasks. In addition, Kojima et
al. (2022) show that LLMs can have decent performance in zero-shot settings by
using the chain-of-thought prompting process. Schick and Schütze (2021)
introduce Pattern Exploiting Training (PET), in which the prompt follows some
pattern to elicit a higher quality of response from the LLM.

LLMs have proven to be proficient in non-textual settings as well. Hegselmann
et al. (2023) explore using LLMs to predict tabular data in a few-shot
setting. The benchmark results show that LLMs are able to outperform previous
state-of-the-art models, and that even zero-shot learning can achieve
nontrivial performance in many instances.

##  2\. Preliminaries

###  2.1. Task Description

Given a text $t$, and a two alternatives $A$ and $B$, the goal is to predict
the preference relation between $A$ and $B$. Ideally, there can be four
possible cases: $A$ is preferred to $B$ ($A\succ B$), $B$ is preferred to $A$
($A\prec B$), both are equally preferred ($A=B$), and there is no preference
relation between the two alternatives (N/A).

This task has sometimes been called comparative preference classification
(CPC). We consider two preference datasets in this work: College Confidential
(Mohsin et al., 2021) and Compsent-19 (Panchenko et al., 2018). The College
Confidential dataset Mohsin et al. (2021) contains comments from a college
admission forum. The authors search for discussion threads where the original
poster asks for opinion comparing multiple colleges and collect the following
posts to build the dataset. The discussion threads discuss more than two
colleges in some cases. The multi-way comparisons are divided into pairwise
comparisons. The resulting dataset consists of 2964 pairwise comparison
instances, with 4 classes – No Preference, $A\succ B$, $A\prec B$, $A=B$.

Compsent-19 (Panchenko et al., 2018) is a binary-comparative dataset that
contains a single comparative sentence between two alternatives. The
alternatives are picked from various domains such as computer science concepts
– programming languages, hardware devices –, or brands. The authors query for
sentences that contain mentions of both alternatives from the Common Crawl
dataset and present a final dataset of 7,199 sentences with 217 unique pairs
of alternatives and 3 classes – $A\succ B$, $A\prec B$, N/A.

Because of the difference in the class representation of the two datasets, we
only consider the 3 class cases in this work – $A\succ B$, $A\prec B$, N/A –,
where N/A refers to both No Preference and $A=B$. The distribution of labels,
along with the average text size for both datasets, is given in Table 1. The
average text size is given in token numbers, where tokens are building blocks
of sentences in NLP. Words, punctuations, and parts of words can all be
individual tokens. From this, we see that the College Confidential dataset has
text that is, on average, more than four times longer than those in
Compsent-19. Tables 2 and 3 show a few example texts, along with labels, from
each dataset.

| Label Distribution | Average Token Length
---|---|---
A $\succ$ B | A $\prec$ B | N/A | Total
College Confidential | 598 | 544 | 1822 | 2964 | 116.12
Compsent-19 | 1364 | 593 | 5242 | 7199 | 26.94
Table 1. Statistics for the datasets Sentence | Label
---|---
If Duke is more expensive then go to UCB. Your parents will thank you for saving them money by going there. And you will have more access to jobs in Calif when you graduate. To me its a no brainer. | $C\succ D$
Daughter graduating from Cal next month. full disclosure. To me, Cal is a no brainer. | $C\succ D$
Both are really good schools. You cannot make a mistake going to either place. | N/A
Table 2. Examples from the College Confidential dataset Sentence | Label
---|---
Golf is easier to pick up than baseball. | $g\succ b$
I’m considering learning Python and more PHP if any of those would be better. | N/A
Table 3. Examples from the Compsent-19 dataset

###  2.2. NLP and ML Terminology

In this section, we discuss the various NLP techniques that we consider in
this work.

Large language models (LLM) usually refer to pre-trained models that use the
attention mechanism. What distinguishes an LLM from a regular language model
is the large amount of data used for pre-training and the number of parameters
in the architecture, which are in the order of billions. We will only refer to
large-scale transformer-based (Vaswani et al., 2017) models that are pre-
trained on massive corpuses, such as GPT (OpenAI, 2022, 2023) and LLaMa
(Touvron et al., 2023) as LLMs to avoid any confusion. In this work, we
consider OpenAI’s GPT-4 (OpenAI, 2023) as the state-of-the-art LLM and use a
fine-tuned version of Meta’s LLaMa-2 (Touvron et al., 2023)’s 70B model222This
was finetuned by Upstage for instructions.
https://huggingface.co/TheBloke/Upstage-Llama-2-70B-instruct-v2-GPTQ and the
original 13B chat model333https://huggingface.co/TheBloke/Llama-2-13B-chat-
GGML as the open-source alternative. Due to the memory constraints, we use the
4-bit quantized version of the 70B model444We use a machine with a single
A6000 with 48G VRAM. We will refer to these models as LLaMa-2-70B and
LLaMa-2-13B for clarity’s sake.

The prompt is what is inputted to the LLM to generate its response. The prompt
consists of three different parts: system message, user message, and assistant
message. The system message sets the context of the interaction with the LLM,
as per the OpenAI official documentation
555https://help.openai.com/en/articles/7042661-chatgpt-api-transition-guide.
The system message is used for instructing the model about the input format of
the data points and ensuring that the output of LLMs conforms to a specific
format. This pattern is also adopted by many open-source LLM models, including
the LLaMa-2 models we consider in this work. For example, both the original
LLaMa-2 weights and fine-tuned version use all three of these message types.

Prompt engineering refers to building a specific prompt that works the best
for the task. Prompt engineering includes designing how to wrap the task into
a prompt and how to break the task down into smaller tasks that the LLM can
handle.

Zero-shot learning. Zero-shot prompts further assume that language models do
not need any examples but understand the concept from training data. So, in
these prompts, no examples are provided, but the LLM is directly prompted to
predict a preference. Kojima et al. (2022) first showed some zero-shot
predictive capabilities of LLMs.

Few-shot learning. Few-shot prompts for language models Brown et al. (2020)
indicate the scenario where no weights are updated but a few examples are
given to the language model to provide context. So, in our task of predicting
preferences expressed in text, examples include a text, the names of the
alternatives, and the expressed preference. Then, the model will receive
instructions to predict preference for a new given text and pair of
alternatives. The number of examples provided is a hyperparameter of the
algorithm. The name few-shot comes from the general concept of few-shot
learning in ML Vinyals et al. (2016).

Model | Cost per Output Token ($\times 10^{-6}$) | Cost per Input Token ($\times 10^{-6}$) |  Architecture Size |  Pre-train Token Amount
---|---|---|---|---
LlaMa-2 70B |  0 |  0 |  70B |  2T
GPT-3.5-Turbo |  2 |  1.5 |  175B* |  unknown
GPT-4 |  60 |  30 |  unknown |  unknown
Table 4.  Statistics of individual models. The cost is measured in U.S.
dollars. Brown et al. (2020); Touvron et al. (2023) * This value is taken from
GPT-3 and it is difficult to confirm whether GPT-3.5-Turbo contains the same
amount of parameters.

##  3\. Experiments

In this work, we seek to assess the potential and limitations of using LLMs
for classification tasks. We use an example from the prompts used for College
Confidential to illustrate the workflow.

All of our code is publicly available on
Github.666https://github.com/inwonakng/llm-preference/

###  3.1. Prompt Structure

For the College Confidential dataset, the system message starts with the
sentences that tell the LLMs that it will be given one comment and two
colleges and that its job is to identify the preference between the two
colleges in this comment. For Compsent-19, which does not have a single
domain, we modify the role to ask it to assume the role of an internet forum
user. Some output rules are added to the system message to make the output
conform to a specific format. For example, we provide rules such as You MUST
respond with “A is preferred over B” if college A is preferred over college B.
Once we have an output that conforms to the desired format, we can assess the
accuracy of LLMs in classification tasks in an automatic manner.

When sending the prompt to the LLM, a conversation is represented as a list of
tuples. The first element of the tuple is the user’s input, and the second is
the LLM’s output. The final response from the model is triggered by sending a
list of those tuples, followed by the last user instruction, to which the
model will respond. This structure allows us to simulate the history of the
interaction between the user and the model before asking for its response.

During our experiments, we find that using clear instructions and capitalizing
the instruction words can lead the LLM to have less inconsistent responses.
For example, instead of saying do … if …, using -you MUST … if … leads to a
more consistent and rule-following output from the LLM. We test two versions
of prompts with this setting. The initial version of the prompt is referred to
as long, which has detailed rules and context in instruction. We also test
with a paraphrased version of this prompt, which we refer to as short.

We also experiment with different hyperparameters available for the LLMs. In
particular, we use different values for temperature and top_p of both LLaMa-2
and GPT-4. These parameters control how the best response is chosen by the
LLM. Temperature is responsible for how the model chooses each token. Lower
temperature values lead the model to choose the tokens with more likelihood.
Top P is used to pick a set of tokens that follow a previously selected token.
Given a selected token, the following set of tokens is chosen such that the
set is the minimum number of tokens whose probability exceeds the P value. We
find that temperature $=1$ and top_p $=0.7$ work the best for the short prompt
and temperature $=0.7$ and top_p $=0.1$ work the best for the long prompt.

> Example of short prompt
> You will be given two colleges A and B, and a comment. Your job is to
> identify the preference between the two given colleges in the comment.
> The names of the two colleges and the comment are delimited with triple
> backticks.
> Here are the rules:
> You MUST NOT use the colleges’ real names.
> You MUST refer to the colleges as A or B.
> You MUST respond with ```No preference```if there is no explicit preference
> in the comment.
> You MUST respond with ```A is preferred over B```if college A is preferred
> over college B.
> You MUST respond with ```B is preferred over A```if college B is preferred
> over college A.
> You MUST respond with ```Equal preference```if colleges A and B are equally
> preferred.
> You MUST respond with ```No preference```, ```A is preferred over B```, ```B
> is preferred over A```, or ```Equal preference```.
> College A: ```{alternative_a}```
> College B: ```{alternative_b}```
> Comment: ```{text}```
>

###  3.2. Few-shot Learning

In few-shot learning, correct examples of the task are added to the prompt to
guide the LLM. For each label, we select one data point with that label in the
dataset as part of the few-shot example and exclude those points from the
testing set. In order to ensure that the examples contain enough content to be
helpful while not increasing the prompt length too much, we select the text
with the minimum length, which has more than 100 words for each label. As for
zero-shot learning, the examples are simply an empty set, i.e., no example
will be used.

An example of the interaction tuple that is added to the chat history is as
the following:

> User:
> ``` Comment: I would prefer Stanford rather than UCB. ```
>
> ``` Option A: Stanford University ```
>
> ``` Option B: UCB ```
> Assistant:
> ``` A is preferred over B ```

Figure 1.  Illustration of the retry prompt process. The incorrect output is
appended to the original prompt, followed by a retry message to remind the
rules.

###  3.3. Retry Prompting

Even if we specify that LLMs should produce specific content for each label,
there is still a possibility that the LLMs may generate content that does not
conform to the format. In some cases, we find that the model is correct in its
classification, but we are not able to programmatically evaluate it due to the
response being malformed. For example, instead of saying A is preferred over
B, the model can respond with … Therefore, I think A is preferred over B. To
overcome cases such as this and allow for an automatic evaluation of the
model’s output, we use what we call a retry prompt. Instead of discarding the
previously correct but malformed response, we re-build a prompt to continue
the conversation and remind the model of the formatting rules again by adding
another user message. Using the tuple structure of the conversation history,
we construct the retry prompt in a way that appears as the continuation of the
task that was incorrectly formatted. We append the tuple of the original task
message and the model’s incorrect response to the list of the conversation
history. The retry message is then sent as the final user input, prompting the
model to have access to its possibly correct but incorrectly formatted
response to fix the format. Using this technique, we are able to find some
sets of rules with which the model’s output was consistent for all the test
inputs. Figure 1 shows an illustration of this process.

Specifically, a retry request is triggered when LLM produces a statement such
as A is better than B in every way, which deviates from the prescribed format.
We use the same prompt rule as the previous short prompt rules in {Rules…},
and the retry prompt is as the following:

> Example of short retry prompt
> You have an incorrect format in your response.
> Here is a reminder of the rules:
> {Rules…}

##  4\. Results

In summary, we deploy the classification experiments in the following settings
and analyze the results on LLaMa-2-70B, GPT-4, GPT-3.5-Turbo with long &
detailed prompt and short & concise prompt with zero-shot and few-shot
settings on College Confidential and Compsent-19.

###  4.1. Prompt Engineering

We experiment with various prompt techniques to convert the CPC task into a
text format. Following best practices suggested by
OpenAI777https://help.openai.com/en/articles/6654000-best-practices-for-
prompt-engineering-with-openai-api and
DeepLearningAI888https://www.deeplearning.ai/short-courses/chatgpt-prompt-
engineering-for-developers/, we develop two sets of final prompts that are
able used as input to the LLM to detect and classify preference in the text
effectively. For example, we set the role for the LLM to assume in the
instruction (system) prompt and explain the rules in a clear bullet list. We
also find that delimiters that are not commonly used in regular English text
were best understood by the LLM. We use the triple-backtick ``` to wrap the
comment and the two alternatives to note that they are separate from the
instructions. While GPT was also able to understand other delimiters, such as
triple-hash (###), this delimiter clashed with the template used by the
instruct fine-tuned LLaMa-2 model used in the work. Because of this reason, we
use the triple-backtick delimiter for both models for consistency’s sake.

When expressing the rules of the task, such as the formatting and the goal of
the task, we find that a simple sentence structure in a commanding tone works
best. Interestingly, we also find that using a more conversational tone – e.g.
instead of “Do XXX”, “Let’s do XXX” – helped the model follow the instructions
more effectively. For this reason, we express the rules themselves in a simple
structure and capitalize the modal verbs – e.g. must, must not – and use a
more conversational tone to end the prompt.

Model | Prompt | Train Mode | F1 Micro | F1 Macro | F1[N/A] | F1[A >B] | F1[A <B]
---|---|---|---|---|---|---|---
LLaMa-2 70B | Short | zero-shot | 0.7287 | 0.6303 | 0.8165 | 0.5445 | 0.5299
few-shot | 0.7381 | 0.6284 | 0.8264 | 0.5359 | 0.523
Long | zero-shot | 0.7274 | 0.6111 | 0.8225 | 0.5265 | 0.4842
few-shot | 0.7247 | 0.5956 | 0.8186 | 0.4989 | 0.4692
GPT-3.5-Turbo | Short | zero-shot | 0.6838 | 0.6165 | 0.7839 | 0.5402 | 0.5255
few-shot | 0.7054 | 0.6374 | 0.7919 | 0.5683 | 0.5522
Long | zero-shot | 0.6393 | 0.4636 | 0.7887 | 0.4160 | 0.1862
few-shot | 0.6841 | 0.4970 | 0.7987 | 0.3408 | 0.3516
GPT-4 | Short | zero-shot | 0.7213 | 0.6815 | 0.7945 | 0.6442 | 0.6058
few-shot | 0.7624 | 0.715 | 0.8276 | 0.6755 | 0.6418
Long | zero-shot | 0.6879 | 0.6524 | 0.7682 | 0.6259 | 0.5629
few-shot | 0.7304 | 0.6860 | 0.8050 | 0.6416 | 0.6113
Table 5.  Comparison of LLM performance on College Confidential dataset.  Model | F1 Micro | F1 Macro | F1[N/A] | F1[A >B] | F1[A <B]
---|---|---|---|---|---
Best LLM | 0.7624 | 0.7150 | 0.8276 | 0.6755 | 0.6418
Best SotA | 0.67* | 0.57† | 0.79* | 0.60† | 0.42†
Table 6. Comparison of best performance of LLM and SotA models on College Confidential dataset. * Results using SimCSEXGBoost as presented by Mohsin et al. (2023). † Results using MultiSentPref-20 as presented by Mohsin et al. (2023).  Model | Prompt | Train Mode | F1 Micro | F1 Macro | F1[N/A] | F1[A >B] | F1[A <B]
---|---|---|---|---|---|---|---
LLaMa-2 70B | Short | zero-shot | 0.7613 | 0.6494 | 0.8432 | 0.6045 | 0.5004
few-shot | 0.8524 | 0.7544 | 0.9063 | 0.7543 | 0.6027
Long | zero-shot | 0.7969 | 0.6981 | 0.8672 | 0.6369 | 0.5902
few-shot | 0.8521 | 0.7470 | 0.9091 | 0.7205 | 0.6114
GPT-3.5-Turbo | Short | zero-shot | 0.5957 | 0.5473 | 0.6674 | 0.5302 | 0.4442
few-shot | 0.8374 | 0.7212 | 0.8977 | 0.7048 | 0.5611
Long | zero-shot | 0.5347 | 0.4084 | 0.6410 | 0.4430 | 0.1413
few-shot | 0.8374 | 0.6857 | 0.9030 | 0.6781 | 0.4759
GPT-4 | Short | zero-shot | 0.8149 | 0.7397 | 0.8739 | 0.7479 | 0.5974
few-shot | 0.853 | 0.7792 | 0.9028 | 0.7939 | 0.6409
Long | zero-shot | 0.7839 | 0.7091 | 0.8493 | 0.7045 | 0.5736
few-shot | 0.8580 | 0.7808 | 0.9083 | 0.7836 | 0.6506
Table 7.  Comparison of LLM performance on Compsent-19 dataset.  Model | F1 Micro | F1 Macro | F1[N/A] | F1[A >B] | F1[A <B]
---|---|---|---|---|---
Best LLM | 0.8580 | 0.7808 | 0.9083 | 0.7939 | 0.6506
Best SotA | 0.8743$\ddagger$ | 0.7578$\ddagger$ | 0.9298$\P$ | 0.7821$\ddagger$ | 0.5872$\ddagger$
Table 8. Comparison of best performance of LLM and SotA models on Compsent-19
dataset. $\ddagger$ Results using EDGATBERT(8) as presented by Ma et al.
(2020). $\P$ Results using EDGATBERT(9) as presented by Ma et al. (2020).

###  4.2. Classification Performance

Tables 7 and 5 show the results from our experiments with LLaMa-2,
GPT-3.5-Turbo, and GPT-4.

The best score for each dataset and model combination is highlighted in bold.
Because of the imbalance of labels in our dataset, we focus on both the Macro
and Micro F1 scores, which calculate the unweighted and weighted averages of
the individual F1 scores.

The results show that few-shot learning with the short prompt outputs the best
performance in most cases. However, more detailed instructions may be helpful
when the input text is shorter. The model’s performance on shorter text – i.e.
Compsent-19 – in Table 7 shows that the zero-shot performance on LLaMa-2 70B
with the long prompt can be better than that with the short prompt. This
suggests that the detailed instructions are helpful, but their effects are
diminished when the other part of the prompt becomes lengthy.

It is also worth noting that the few-shot long prompt in Compsent-19
outperforms the few-shot short prompt more consistently with GPT-4. This
suggests that GPT-4 may be more capable of handling complex/verbose
instructions, especially when the input text itself is short.

While LLaMa-2 outperformed the previous state-of-the-art, the GPT-4
outperforms the LLaMa-2 models in general. However, LLaMa-2 is able to
outperform GPT-3.5-Turbo.

In addition to the single-stage classification experiments, we consider
another variety of prompting to handle long input. While Compsent-19 contains
a single sentence per text, College Confidential dataset’s text can be as long
as multiple paragraphs. We design an experiment where the LLM first summarizes
the input text and uses this summary to run the preference classification.
Specifically, the LLM is prompted to summarize the preference expressed in the
text while ensuring the output contains the names of the two colleges. This
provides the necessary information for the subsequent preference
classification task.

However, we find that the summary method does not perform as well as expected.
When comparing using no summary versus using a summary, we see that no summary
prompt outperforms the summary condition in the overall performance. When
analyzing performance by text length, the summary approach only benefits texts
longer than 400 words. Since posts in the College Confidential dataset tend to
be shorter, the summary task may be unsuitable for this corpus. The poor
performance of the summary may be due to the brevity of the source texts in
College Confidential as well.

###  4.3. Output Consistency

Throughout our experiments, we find that the LLM’s responses tend to be
inconsistent when faced with the same prompts. We observe that this happens
often in the LLaMa-2 models and GPT-3.5-Turbo. We note that GPT-4 was able to
follow the rules much more consistently than LLaMa-2 – while more than half of
the tasks for LLaMa-2 had to be run with the retry prompt, only a handful of
cases were needed for GPT-4. For instance, the predicted label drastically
changes if the same question is asked again, even though the response is
correctly formatted.

We also find that the output from the few-shot is more likely to conform to
the output format than zero-shot. It is likely because the examples show the
correct outputs, and these examples can help LLM to do in-context learning and
understand the output format rules.

The experiments on LLaMa-2 were deployed on the 4-bit quantized version of the
70B model. We also deployed the experiments on the 13B model of LLaMa-2.
However, we observe that the LLaMa-2-13B model could not handle the College
Confidential or Compsent-19 dataset. Specifically, the output consistency is
hard to satisfy on LLaMa-2-13B, especially for zero-shot. On the other hand,
LLaMa-2-13B tends to give fixed answers like “No preference” or “Equal
preference” for most cases, which results in a bad performance.

###  4.4. Detecting Preference

As expected, the F1 scores for cases where preference is present are much
lower than the N/A cases, meaning the models struggled more with the
classification task. It is also worth noting that when the input text is a
single sentence, as in the Compsent-19 dataset, the LLM tends to perform
better in both detecting/classifying the preference. Overall, we again note
that GPT-4 was able to outperform the LLaMa-2 models and GPT-3.5-Turbo in both
detecting and classifying the preference.

###  4.5. Comparison to Previous Work

We compare our results against the state-of-the-art (SotA) results from
previous works that leverage GNN-based models. Ma et al. (2020) present their
results on the Compsent-19 dataset using the ED-GAT architecture, and Mohsin
et al. (2023) present the results from MultiSentPref on the College
Confidential dataset. Tables 6 and 8 show the comparison between the best LLM
and SotA performances.

For College Confidential, we find that the LLM performance from GPT-4
outperforms the MultiSentPref’s best performance by a significant margin. Even
LLaMa-2-70B can outperform every metric considered except for F1[A ¿ B].

We also find that GPT-4 is able to outperform the best performance from ED-GAT
in classifying the performance, albeit by a smaller margin than when compared
to MultiSentPref. It is worth noting here that ED-GAT performs better at
detecting the lack of preference. This observation is also reflected in a
higher F1 micro score as the dataset is unbalanced and contains approximately
$2.7\times$ more no preference rows as preference present rows.

It is interesting to note that MultiSentPref (Mohsin et al., 2023) is an
extension of ED-GAT (Ma et al., 2020), but it does not perform nearly as well
in College Confidential. Notably, while Compsent-19 contains a single sentence
per text, College Confidential’s content can extend to multiple paragraphs;
see Table 1. This disparity suggests that LLM’s superiority over SotA likely
arises from its capacity to manage extensive context, suggesting that it may
be better at handling complex tasks compared to the previous graph-based SotA
models.

Overall, we see an improvement over previous SotA models with both GPT-4 and
LLaMa-2 in most metrics. GPT-4 consistently outperforms both MultiSentPref and
ED-GAT in [preference classification while LLaMa-2 is able to outperform
MultiSentPref on College Confidential but falls short of ED-GAT on
Compsent-19. While ED-GAT outperforms in the preference detection task in
Compsent-19, the difference is insignificant. The large improvement on the
College Confidential dataset indicates that LLMs have significantly improved
over previous SotA models in classifying longer context-length examples
compared to smaller ones.

##  5\. Future Work

Future extensions of this work can be branched into multiple directions. The
first direction would be to improve the LLM’s performance by fine-tuning the
model or improving the prompt. While fine-tuning the model will improve the
performance, the lack of labeled datasets in the CPC domain poses a challenge.
The prompt engineering direction can include a more fine-grained approach to
the problem, such as handling the two parts of CPC separately – detecting the
preference and classifying only if it exists – or adding a summarization task
before the classification to handle longer text better. We considered a
version of the summarization task in our work but were not able to find a set
of prompts that led to a higher performance than the currently presented
single-stage method. Future works could explore more prompt engineering to
find a set of prompts that allow the smaller models to handle large text more
efficiently.

Another direction is to consider an ensemble learning approach of the LLMs. As
seen in the comparison with the SotA models, models from previous work can
outperform the current LLM approach. Thus, combining multiple previous models
and LLMs in an ensemble could lead to better performance. Another possibility
is to consider the LLMs in an ensemble. The LLMs’ predictions can be
aggregated to form an ensemble output, resulting in better performance.

In another direction, we can generate new text datasets using an LLM to
augment the original dataset to remedy the lack of comparative text datasets.
This newly augmented dataset could be used to train smaller models that
usually need more data than currently available. Thus, a knowledge
distillation process could be tried in which the preference predictive
capabilities of LLMs can be distilled into smaller and more efficient models.

In a broader direction, we can explore the capabilities of LLMs in multi-agent
scenarios such as group decision-making, deliberation, iterative decision-
making, etc. We see much ongoing work in this domain, proposing to use LLMs
for social choice Fish et al. (2023) or to facilitate multi-agent
collaborations Gong et al. (2023); Chen et al. (2023). In this same vein, we
can work to create an AI agent that assists in making better group decisions
under uncertainty.

Finally, any application of an LLM for a specific task inherits the bias
present in training the original LLMs. While LLaMa-2 was trained on open-
source texts, this is not true for the GPT models, so we can not even be sure
of the amount of bias in the models. Future work applying LLMs for preference
learning and elicitation should further fine-tune the models to remove any
possible biases since this is an even bigger issue when considering
comparative texts. Also, while the LLM-based methods outperform state-of-the-
art models in most cases, they still depend on black box methods. Thus, these
methods should be applied cautiously in the real world. In the future, we will
look into what steps we can take to make the preference prediction process
more transparent, particularly focusing on getting explanations for the
predictions.

##  6\. Conclusion

In this work, we consider the task of predicting preferences expressed in text
by using LLMs with an automated evaluation scheme. Specifically, we experiment
with four types of commercial and open-source LLMs – OpenAI’s GPT-4 and
GPT-3.5-Turbo, Meta’s LLaMa-2-70B and LLaMa-2-13B. We also test the efficacy
of different kinds of prompting methods to represent the task in a textual
format and find two methods that are able to help the LLMs perform well in the
task. Using these two prompts, we test the zero-shot and few-shot techniques
to wrap the classification task into a conversational format that the LLM can
handle.

Our results show that LLMs are able to outperform the previous SotA approaches
in predicting preferences from text. We also find that using few-shot prompts
by including examples from the dataset as a part of the prompt can further
improve the LLM’s performance. While the smaller models are not able to handle
a lengthy set of instructions and text as efficiently, we find that the larger
model, such as GPT-4, is able to handle both. The comparisons to previous SotA
models show that the LLM is an effective replacement when handling longer
texts. Some SotA approaches can still outperform our results using the LLMs
when the input text is short. The observation that older methods can sometimes
outperform LLMs suggests that we can use a sufficiently large enough LLM can
be used in combination with other existing techniques for even better
performances. To summarize, our main findings are twofold: 1) LLM can be
better than SotA models. 2) few-shot prompts are better than zero-shot ones.

## Ethical Impacts

Any application of LLMs for a specific task inherits the biases of the
original LLM. We consider ethical concerns over this and discuss possible ways
of alleviating some of those concerns in the Future Works section.

## References

  * (1)
  * Boutilier (2002) Craig Boutilier. 2002.  A POMDP formulation of preference elicitation problems. In _Proceedings of the National Conference on Artificial Intelligence (AAAI)_. Edmonton, AB, Canada, 239–246.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models Are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 1877–1901.
  * Chen et al. (2023) Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. 2023\.  Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents.  _arXiv preprint arXiv:2308.10848_ (2023).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  https://doi.org/10.48550/arXiv.2204.02311 arXiv:2204.02311 [cs]
  * Conitzer and Sandholm (2002) Vincent Conitzer and Tuomas Sandholm. 2002.  Vote Elicitation: Complexity and Strategy-Proofness. In _Proceedings of the National Conference on Artificial Intelligence (AAAI)_. Edmonton, AB, Canada, 392–397.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In _Proceedings of EMNLP 2017_. Association for Computational Linguistics, Copenhagen, Denmark, 670–680.  https://www.aclweb.org/anthology/D17-1070
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805 [cs.CL]
  * Fish et al. (2023) Sara Fish, Paul Gölz, David C Parkes, Ariel D Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. 2023.  Generative Social Choice.  _arXiv preprint arXiv:2309.01291_ (2023).
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of EMNLP 2021_.
  * Gong et al. (2023) Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, et al. 2023\.  MindAgent: Emergent Gaming Interaction.  _arXiv preprint arXiv:2309.09971_ (2023).
  * Haque et al. (2022) Amanul Haque, Vaibhav Garg, Hui Guo, and Munindar P Singh. 2022.  Pixie: Preference in Implicit and Explicit Comparisons. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 106–112.
  * Hegselmann et al. (2023) Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023.  TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics_. PMLR, 5549–5581.
  * Huang and Carley (2019) Binxuan Huang and Kathleen Carley. 2019.  Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 5469–5477.  https://doi.org/10.18653/v1/D19-1549
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large Language Models Are Zero-Shot Reasoners.  _Advances in Neural Information Processing Systems_ 35 (Dec. 2022), 22199–22213.
  * Li et al. (2021) Zeyu Li, Yilong Qin, Zihan Liu, and Wei Wang. 2021.  Powering Comparative Classification with Sentiment Analysis via Domain Adaptive Knowledge Transfer. In _Proceedings of the EMNLP 2021_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Ma et al. (2020) Nianzu Ma, Sahisnu Mazumder, Hao Wang, and Bing Liu. 2020.  Entity-aware dependency-based deep graph attention network for comparative preference classification. In _Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL-2020)_.
  * Mandal et al. (2020) Debmalya Mandal, Nisarg Shah, and David P. Woodruff. 2020.  Optimal Communication-Distortion Tradeoff in Voting. In _Proceedings of ACM EC_.
  * Mohsin et al. (2023) Farhad Mohsin, Inwon Kang, Yuxuan Chen, Jingbo Shang, and Lirong Xia. 2023.  Dependency and Coreference-boosted Multi-Sentence Preference model. In _The 9th International Workshop on Deep Learning on Graphs: Method and Applications (DLG-AAAI-23)_.
  * Mohsin et al. (2021) Farhad Mohsin, Lei Luo, Wufei Ma, Inwon Kang, Zhibing Zhao, Ao Liu, Rohit Vaish, and Lirong Xia. 2021.  Making group decisions from natural language-based preferences. In _Proceedings of the 8th International Workshop on Computational Social Choice (COMSOC)_.
  * OpenAI (2022) OpenAI. 2022.  Introducing ChatGPT.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs]
  * Panchenko et al. (2019) Alexander Panchenko, Alexander Bondarenko, Mirco Franzek, Matthias Hagen, and Chris Biemann. 2019.  Categorizing Comparative Sentences. In _Proceedings of the 6th Workshop on Argument Mining_. Association for Computational Linguistics, Florence, Italy, 136–145.  https://doi.org/10.18653/v1/W19-4516
  * Panchenko et al. (2018) Alexander Panchenko, Eugen Ruppert, Stefano Faralli, Simone P. Ponzetto, and Chris Biemann. 2018.  Building a Web-Scale -Parsed Corpus from CommonCrawl. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Miyazaki, Japan.  https://aclanthology.org/L18-1286
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the EMNLP 2014_. 1532–1543.
  * Radford and Narasimhan (2018) Alec Radford and Karthik Narasimhan. 2018.  Improving Language Understanding by Generative Pre-Training.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Association for Computational Linguistics, Online, 255–269.  https://doi.org/10.18653/v1/2021.eacl-main.20
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  https://doi.org/10.48550/arXiv.2302.13971 arXiv:2302.13971 [cs]
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Proceedings of NeurIPS 2017_ (2017).
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _International Conference on Learning Representations_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016\.  Matching networks for one shot learning.  _Advances in neural information processing systems_ 29 (2016).
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022a.  FINETUNED LANGUAGE MODELS ARE ZERO-SHOT LEARNERS.  _International Conference on Learning Representations_ (2022).
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022b.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  _Advances in Neural Information Processing Systems_ 35 (Dec. 2022), 24824–24837.
  * Xia (2019) Lirong Xia. 2019.  Learning and decision-making from rank data.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ 13, 1 (2019), 1–159.
  * Xia (2022) Lirong Xia. 2022.  Group decision making under uncertain preferences: powered by AI, empowered by AI.  _Annals of the New York Academy of Sciences_ 1511, 1 (2022), 22–39.
  * Zhao et al. (2018) Zhibing Zhao, Haoming Li, Junming Wang, Jeffrey Kephart, Nicholas Mattei, Hui Su, and Lirong Xia. 2018.  A Cost-Effective Framework for Preference Elicitation and Aggregation. In _Proceedings of Uncertainty in Artificial Intelligence_.

##  7\. Appendix

###  7.1. Full prompt for long

> Instruction Message
> Pretend that you are a user on college confidential forums.
> Your job is to detect if there exists a preference between two options in a
> comment.
> If there exists a preference, you must detect what the preference is.
> If the author of the comment expresses an explicit preference, you must
> detect it.
> You will be given a comment and two alternatives for each task. The options
> will be denoted by ``` Option A:``` and ``` Option B:``` .
> The comment will be denoted by ``` Comment:``` .
>
>
> Rules: \- You MUST NOT respond with a summary of the comment.
> \- You MUST NOT use the options’ real names.
> \- You MUST refer to the options as A or B.
> \- You MUST respond with ”No preference“‘ if there is no strict preference.
> \- You MUST respond with ``` A is preferred over B``` if option A is
> preferred over option B.
> \- You MUST respond with ``` B is preferred over A``` if option B is
> preferred over option A.
> \- You MUST respond with ``` Equal preference``` if options A and B are
> equally preferred.
> \- You MUST respond using one of the four phrases above.

> Retry Message
> Your response was incorrect.
> Let’s try again.
> Here is a reminder of the rules:
>
>
> \- You MUST ONLY report the preference in the comment.
> \- You MUST respond only using one of the following phrases: ``` No
> preference``` , ``` A is preferred over B``` , ``` B is preferred over A```
> , ``` Equal preference``` . Do not say anything else.
> \- You MUST respond with ``` No preference``` if there is no strict
> preference.
> \- You MUST respond with ``` A is preferred over B``` if option A is
> preferred over option B.
> \- You MUST respond with ``` B is preferred over A``` if option B is
> preferred over option A.
> \- You MUST respond with ``` Equal preference``` if options A and B are
> equally preferred.
> \- You MUST NOT use the options’s real names. \- You MUST ONLY refer to the
> options as ``` A``` or ``` B``` .
> \- You MUST NOT respond with any other details than the preference expressed
> in the comment.
> \- You MUST NOT explain your reasoning behind the response. Only respond
> with the given phrase.
> \- You MUST NOT use any punctuation in the response.
>
>
> Your previous response was not in any of the required responses.
> Try again and respond with a correct response to the previous comment.
> You MUST NOT reply the same response.
>

###  7.2. Full prompt for short

> Instruction Message
> You will be given two colleges A and B, and a comment. Your job is to
> identify the preference between the two given colleges in the comment.
> The names of the two colleges and the comment are delimited with triple
> backticks.
>
>
> Here are the rules: You MUST NOT use the colleges’ real names.
> You MUST refer to the colleges as A or B.
> You MUST respond with ``` No preference``` if there is no explicit
> preference in the comment.
> You MUST respond with ``` A is preferred over B``` if college A is preferred
> over college B.
> You MUST respond with ``` B is preferred over A``` if college B is preferred
> over college A.
> You MUST respond with ``` Equal preference``` if colleges A and B are
> equally preferred.
> You MUST respond with ``` No preference``` , ``` A is preferred over B``` ,
> ``` B is preferred over A``` , or ``` Equal preference``` .

> Retry Message
> You have an incorrect format in your response.
> Here is a reminder of the rules:
> You MUST NOT use the colleges’ real names.
> You MUST refer to the colleges as A or B.
> You MUST respond with ``` No preference``` if there is no explicit
> preference in the comment.
> You MUST respond with ``` A is preferred over B``` if college A is preferred
> over college B.
> You MUST respond with ``` B is preferred over A``` if college B is preferred
> over college A.
> You MUST respond with ``` Equal preference``` if colleges A and B are
> equally preferred.
> You MUST respond with ``` No preference``` , ``` A is preferred over B``` ,
> ``` B is preferred over A``` , or ``` Equal preference``` .