=-.2ex =.1ex

# Do Large Language Models Learn Human-Like Strategic Preferences?

Jesse Roberts
Department of Computer Science
Vanderbilt University
Nashville, TN 37235
<EMAIL_ADDRESS>
Kyle Moore
Department of Computer Science
Vanderbilt University
Nashville, TN 37235
<EMAIL_ADDRESS>
Doug Fisher
Department of Computer Science
Vanderbilt University
Nashville, TN 37235
<EMAIL_ADDRESS>
https://www.linkedin.com/in/jessetnroberts/

###### Abstract

We evaluate whether LLMs learn to make human-like preference judgements in
strategic scenarios as compared with known empirical results. We show that
Solar and Mistral exhibit stable value-based preference consistent with human
in the prisoner’s dilemma, including stake-size effect, and traveler’s
dilemma, including penalty-size effect. We establish a relationship between
model size, value based preference, and superficiality. Finally, we find that
models that tend to be less brittle were trained with sliding window
attention. Additionally, we contribute a novel method for constructing
preference relations from arbitrary LLMs and support for a hypothesis
regarding human behavior in the traveler’s dilemma.

##  1 Introduction

Transformer-based large language models (LLMs) have famously achieved state of
the art performance on many tasks since their introduction by Vaswani et al.
[34]. The analysis of these models is typically focused on benchmark tasks
like Big-Bench [28], MMLU [11], and Agieval [38]. Theoretical analysis of
their computational abilities like [26, 6, 24] and empirical investigations of
LLM cognitive/psychological behaviors are proliferating [19, 32, 27, 7, 33,
29], but remain far from common. However, in many human-adjacent applications,
this latter type of investigation is paramount to successful LLM integration.

As both an illustration of this point and a motivational example relevant to
the present paper, consider, a human carrying a heavy box who asks a human
collaborator for help. Based on this input alone, the collaborator can quickly
choose and apply their most preferred strategic mixture of vertical and
horizontal force. In fact, the individual asking for help has implicitly
relied upon the collaborator’s possession of a compatible set of preferences
over the possible strategies. Otherwise, the originator of the request would
have found it necessary to provide a more detailed and precise request to
ensure the collaborator acted appropriately.

If asked to help with a box, a robot is currently incapable of selecting from
the possible strategies unless it has previously been imbued with a precise
value function over the strategies or has suffered a regiment of reinforcement
learning. In future work we hope to apply LLMs in support of human-robot
interaction (HRI). However, for this to be possible, the supporting LLM must
likewise have strategic preferences sufficiently similar to that of a human to
permit effectual communication.

Further, applications like HRI require LLM behavior be stable under variations
to avoid potentially dangerous strategic variations due to slight prompt
variations. This point is timely as recent evaluations of some language model
cognitive behaviors have been shown to fail to repeat under small variations
[33]. To address this, we use PopulationLM [27] to create systematically
varied populations of each model for experimentation.

###  1.1 Aims of This Paper

In service of human-adjacent LLM integrations, this paper aims to understand
if any current open-source language models exhibit stable, human-like
preferences. To do this, we construct populations from a wide variety of LLM
species and evaluate their strategic preferences in a number of scenarios.
Open-source models are studied exclusively in support of reproducibility.
Closed source models are not static and may change without warning, resulting
in the loss of previously studied behaviors as occurred in [29].

We first consider whether language models in the presence of simple and
explicit values associated with strategies tend to have value-based
preferences (VBP). From this task, we identify interesting models appropriate
for additional testing. We then engage these models in two masked versions of
the prisoner’s dilemma, one with high and one with low stakes. Finally, we
engage these models in experiments based on a low penalty and high penalty
traveler’s dilemma.

We find that, although LLMs are not explicitly trained to replicate the
strategic preferences of humans, (1) some acquire stable human-like strategic
preferences. Specifically, Solar [15] and Mistral [14] are shown to have
human-like self-consistent, non-brittle VBP. On the other hand, we find that
(2) small models tend to prefer strategies based on superficial heuristics,
(3) larger models tend to have decreased reliance on superficial information,
and (4) some large models that exhibit VBP prove to be highly brittle under
variations which may be related to the attention architecture. Finally, from
the in silico experimentation we provide (5) evidence for the origin of human
deviation from the Nash equilibrium in the traveler’s dilemma.

In the course of this work, we contribute novel datasets for each scenario and
(6) a novel method for constructing preference relations from a population of
LLMs.

##  2 Related Work

In [1] the authors engaged GPT-3.5 and GPT-4 [23] in a number of iterated
games including an iterated prisoner’s dilemma. The authors find that both
models tended to be punishing in response to betrayal though, prior to
betrayal, they tended to cooperate. No matter how many times an opponent
cooperated after the a betrayal, the models would not again reciprocate
cooperation.

In [8], the authors engaged GPT-3.5 and GPT-4 in a number of games to evaluate
their ability to act consistently with a prompted preference, refine belief,
and take optimal actions. Their work is aimed at evaluating the potential
integration of GPT-4 into games for research in social science. Their results
suggest that GPT-4 fails to appropriately update and maintain beliefs
necessary to choose optimal strategies and is therefore yet unsuited to
integration into social science experiments.

In [35], the authors engage GPT-4 and Claude in a social game involving
misinformation and provide a prompt pattern, related to chain of thought [36],
to help the models reason in that environment.

A number of authors have explored LLM behavior in games. Their work provides a
confidence from which it is reasonable to believe that some LLMs may learn
strategic preferences from human language data. However, the focus of the
existing work has been distinctly different from our aims. Our work is
specifically differentiated by the fact that none of the existing work
considers the stability of model preference, the effect of stake/penalty size
(human-like or otherwise), or games like the traveler’s dilemma in which human
behavior tends to sharply contrast with game theoretic predictions. Further,
all existing peer-reviewed related work is based on closed-source models,
something we specifically avoid. Consequently, the measurement of model
preference used by all existing work is akin to a cloze type task based on the
generated token. In contrast, we use a method called counterfactual prompting
to measure model evaluation probability. Finally, the number of models which
previous work has considered is relatively small in comparison to the number
we consider.

##  3 Do LLMs Prefer Strategies Based on Value?

Although past work has shown that GPT-3.5 and GPT-4 have preferences for
higher valued strategies in a dictator game [8], it is not clear whether other
model species possess similar preferences. Further, if a model has value-based
preferences (VBPs), it is unclear how these preferences will fair under
systematic perturbation. Poorly supported preferences in the network may fail
to be sufficiently reliable to support human-adjacent NLP tasks. We therefore
formulate RQ 3.1.

######  Research Question 3.1.

Given a set of strategies each with a clearly specified value, do LLMs tend to
have value-based-preferences?

Figure 1: Top: Individual population member probabilities for “Best”
evaluation of strategies. Middle: Individual population member probabilities
for “Worst” evaluation of strategies. Bottom: Spearman’s $\rho$ for value-
preference correlation and negated anti-correlation.

###  3.1 Experimental Method

To evaluate RQ 3.1, we create a prompt that defines 3 strategies labeled A1,
A2, and A3. Each strategy is ascribed a value 5, 10, or 20 points with each
value being assigned once in the prompt context $c$. The model then provides
the probability for all in-vocabulary completions. However, we consider only
the probability of a constant evaluation word. This is repeated for each
strategy option, changing only the strategy $s$. This measures the probability
of the evaluation word given the strategy, $p(e_{word}|c,s)$ $\forall
s\in\mathbf{S}$. We refer to this as counterfactual prompting. The following
is an example of such a prompt with A1 as the evaluated strategy.

Option A1 gives 5 points. Option A2 gives 10 points. Option A3 gives 20
points. A1 is best.

Our hypothesis is that the preference, as measured by the probability of the
evaluation word, will tend to be correlated with the assigned value. If the
correlation is 0.3 or higher, based on Applied Statistics for the Behavioral
Sciences [12], then a significant correlation is present and the LLM is
considered capable of VBP. There are 2 alternative hypotheses that this
experiment must control for: preference based on ordering of the labels and
preference for a label absent of value. So, we generate a prompt for every
permutation of the order of labels and the assigned value. This results in 36
unique prompts. We then test the LLM preference for each strategy for each
prompt permutation. This yields 108 individual calls to each model.

Additionally, we investigate if models with VBP are self-consistent across
evaluation words of differing sentiment. We perform the described experiment
first with a positive sentiment evaluation word (best), and then with a
negative sentiment evaluation word (worst). We say that a model with positive
sentiment probability that is correlated with strategy value and negative
sentiment probability that is anti-correlated with strategy value have VBP and
are self-consistent.

As mentioned previously, the effect of variation on model preference is
important given the targeted HRI application domain. We use PopulationLM [27]
to construct populations for each model species tested. Models that differ on
architecture, size, training data, or training task are considered different
species. This approach uses Monte Carlo dropout to generate perturbed versions
of the base model. Such a population is known to approximate a gaussian random
process [9]. Intuitively, this means that model behaviors which are
constituted in a small number of neurons, referred to as poorly supported, are
likely to be ablated in the perturbed population. So, if the base model of a
given species has VBP but the derived population does not, we say the model is
brittle since variation tends to erode the behavior of interest.

Finally, to understand how model size relates to VBP and the tendency to
prefer strategies based on more superficial criteria, we conduct the described
set of experiments on 19 different model species with sizes varying from
$<10^{8}$ to $>10^{10}$ parameters. For each base model species the generated
population has 50 members.

###  3.2 Results: Value-Based Preference

In answer to RQ 3.1, we find that a surprisingly small number of models have
VBP. In figure 1, the correlation of the evaluation probability and strategy
point value for each of the population members (dots) as well as the species
base model (stars) are shown in the bottom row. Those that do have VBP are
those that have high base model correlation like Solar [15], Mistral [14],
Gemma [30], Llama-2 [31], and Phi-2 [13]. Among these, only Phi-2 fails to be
self-consistent.

We find that, among self-consistent models with VBP, the populations are
likewise self-consistent. However, the Gemma and LLama-2 populations don’t
exhibit VBP. Therefore, Gemma and Llama-2 are considered brittle.

###  3.3 Effects of Model Size

We investigate the effect of model size on the presence of VBP. In Figure 2
the model size is tellingly correlated with the model’s preference for higher
value strategies. From the figure it seems that model size is predictive of
VBP. More precisely we conclude that sufficient model size may a necessary,
though insufficient, condition for a model to learn VBP from human language
data.

Figure 2: As models get larger they tend to have value-based strategy
preferences. Left: Model size vs populationLM strategy VBP. Right: Model size
vs base species strategy VBP.

We further consider the effect of superficial information, like the label, on
model preference. In Figure 3 the non-parametric Kruskal-Wallis test is used
to evaluate if the probabilities assigned to a strategy are independent of the
label. The null hypothesis for this test expects the medians of the groups to
be equal. The figure shows that, for the smaller species base models,
preferences tend to be sensitive to superficial information like label. On the
other hand, as the model size increases, sensitivity to the label tends to
decrease.

Interestingly, it seems that preference sensitivity to label is much more
correlated with model size in the base models ($\rho=0.39$), shown on the
right of the figure, as compared to the populations ($\rho=0.06$) on the left.
This shows that intra-species populations of language models may tend to be
less sensitive to superficial information. This suggests reliance on
superficial information is not a well supported behavior in many of the base
models.

Figure 3: Left: Model size vs populationLM strategy preference sensitivity to
label. Right: Model size vs base species strategy preference sensitivity to
label.

###  3.4 Why are Solar and Mistral Not Brittle?

From these experiments we find that Solar is the current best model among
those tested when required to make stable, VBP judgments, with Mistral being a
close second. Though Gemma and Llama-2 base models exhibit comparable VBP than
Solar and Mistral, they are brittle. The process of dropout and proportion of
affected neurons in each of the models is constant. We therefore, consider why
Gemma and Llama-2 brittle while Mistral and Solar are not.

It’s first important to understand the provenance of Mistral and Solar.
Llama-2 was trained on 2 Trillion tokens and the authors of the technical
report [31] point out that the model had not reached training saturation. This
message to the community was clear, further training of the Llama-2 model may
increase the performance.

The creators of Mistral adopted the Llama-2 architecture, converted the
attention to sliding window attention (SWA) [5] with a window of 3 tokens, and
then retrained the model. Importantly, SWA does not force a model to only
consider the last 3 tokens during generation. Rather, the path to access
information in tokens prior to the window requires adjacent representations to
package and pass the information. The resultant model outperforms Llama-2 7B
and 13B substantially [14]. The number of training tokens is unknown. However,
in light of the shared architecture, the comment regarding Llama-2 training
saturation, and the established supremacy of Mistral, it is reasonable to
assume that it has been trained on a number of tokens greater than 2 trillion.

Solar was subsequently constructed by adopting the architecture of Llama-2 and
increasing the number of layers through depth upscaling [15]. The initial
layers of the architecture are initialized with the Mistral weights and then
all layers were received additional training. So, Solar must be considered to
have been trained on more tokens than Mistral. Solar does not use SWA,
however, it inherited weights that were learned through SWA.

Finally, Gemma exhibits VBP that is similar to Solar, however it is more
brittle than Llama-2 though it was trained on 4 times the number of tokens. We
therefore surmise that the total number of training tokens and model size may
tend to improve VBP. However, we hypothesize that SWA may tend to encourage
distributed representations, in Mistral and Solar, that are less brittle.

##  4 Do LLMs Have Human-Like Preference in the Prisoner’s Dilemma?

Having shown that some LLMs exhibit VBP, we ask whether their preferences tend
to be human-like in established, empirically considered game scenarios. The
prisoner’s dilemma (PD) is a well known game in which two players have two
strategy options. If player 1 chooses to betray player 2, then they may either
receive 0 or 3 months in jail. If player 1 instead decides to be silent, they
will receive either 2 or 5 months in jail. For various scenarios, the payoff
matrices are shown in Table 1 with Player 1 being the first number in each
pair.

The Nash equilibrium strategy is defined as the option that obtains the best
payoff without first assuming that the opponent will change strategy [22]. In
the PD, rational agents are typically expected to seek their own interest and
choose the Nash equilibrium strategy to betray. However, in practice humans
don’t necessarily choose the Nash equilibrium. By choosing to be silent they
can minimize the total number of months spent by either player in jail. This
is known as the Pareto optimal strategy. In [37], the authors did a large
study on human subjects in Tokyo and show that humans tend to cooperate
(choose the Pareto optimal strategy) when the stake size is low. However, when
the stake size is large, humans tend to choose to betray the other player in
self-interest.

LLMs have been engaged in a prisoner’s dilemma in previous work [1]. They
found that in a non-repeated PD, GPT-4 tended to cooperate. However, as
mentioned previously, the measurement they use is a type of cloze task, their
work does not consider the effect of variation on the result, and it does not
consider the effect of stake size. To understand if LLMs tend to have human-
like preference we formulate RQ 4.1.

######  Research Question 4.1.

When engaged in an obfuscated prisoner’s dilemma, do LLMs tend to have
preferences consistent with human preference including sensitivity to stake
size?

###  4.1 Experimental Method

Given the prevalence of the PD, it is likely that is well represented in
language model training data. To effectively measure the impact of stake size
without encountering the canonical moral preference for cooperation, we
obfuscate the PD.

The low stakes version of the PD is cast as a decision to use (betray) or not
use (silent) a shared air conditioner at night. The payoff matrix is shown in
the left side of Table 1. The high stakes version is nearly identical with the
AC system exchanged for a life support system.

To evaluate model preference for each strategy, we construct a prompt which
enumerates the options and the possible results. We then use counterfactual
prompting to find the probability assigned to a constant evaluation word as
done in the previous experiment. A full example of the prompt is available in
the appendix.

We construct different versions of the prompt to ensure that all permutations
of label order and strategy assignment are represented to control for
alternative hypotheses. We again perform the set of experiments using both
positive (“Best”) and negative (“Worst”) sentiment evaluation words. Finally,
we perform the set of experiments using populations (N=50) of the 4 self-
consistent models that exhibitied VBP: Solar, Mistral, Gemma, and Llama-2.

Table 1: Prisoner’s dilemma payoff matrices

|  | Player 2
---|---|---
|  | AC Sharing | Life Support Sharing | Months in Jail
|  | Silent | Betray | Silent | Betray | Silent | Betray
Player 1 | Silent | Cool, Cool | Cold, Hot | 4,4 | 0,10 | 2,2 | 5,0
Betray | Cold, Hot | Warm, Warm | 10,0 | 2,2 | 0,5 | 3,3

####  4.1.1 Pythagorean Preference Relation

To construct a preference relation from the counterfactual prompting, we use
the stratified population members to evaluate the possible strategies. This
permits the use of the non-parametric, paired Wilcoxon rank-sum test. The null
hypothesis for this test is that the distribution of observations of a single
group, arising from two treatments, are not statistically different. This
allows not only the characterization strategy preference but also statistical
significance.

Table 2: Preference relation using positive and negative evaluation for
preference and anti-preference.

| Strong Preference | Partial Preference | Indifference
---|---|---|---
Best Evaluation | $L\succ M$ | $M\succ L$ | $L\succ M$ | $M\succ L$ | $L\sim M$ | $L\sim M$ | $L\sim M$ | $L\succ M$ | $M\succ L$
Worst Evaluation | $L\succ M$ | $M\succ L$ | $L\sim M$ | $L\sim M$ | $M\succ L$ | $L\succ M$ | $L\sim M$ | $M\succ L$ | $L\succ M$
Result | $L\succ M$ | $M\succ L$ | $L\succeq M$ | $M\succeq L$ | $M\succeq L$ | $L\succeq M$ | $L\sim M$ | $L\sim M$ | $L\sim M$

Inspired by work in pythagorean fuzzy preference relations for group decision
making [17], we consider that preference and anti-preference may vary
independently. By measuring the probability of both the positive and negative
evaluation words, we arrive at measures related to the preference and anti-
preference respectively. Performing separate Wilcoxon tests on the positive
and negative evaluations independently yields a measure and significance of
the preference and anti-preference.

So, for strategies $L$ and $M$ each presented as options in context $c$ and a
positive evaluation word used as the measure, if
$p(e_{pos}|c,L)>p(e_{pos}|c,M)$ tends to hold in a population, as
characterized by a Wilcoxon test, then we say the population has a significant
preference for $L$ over $M$, or $L\succ M$. Alternatively, if
$p(e_{neg}|c,L)>p(e_{neg}|c,M)$ tends to hold in a population, then we say the
population has a significant anti-preference for $L$ over $M$, or $M\succ L$.
If the result of a Wilcoxon test fails to be statistically significant, then
we say that the population has indifferent preference or anti-preference to
$L$ over $M$, or $L\sim M$. In Table 2 the possible resulting preferences are
shown.

###  4.2 Results: LLM Preference in the Prisoner’s Dilemma

In Figure 4 the probability of positive evaluation is shown in the top row and
the probability of negative evaluation is shown in the bottom for all
population members and all species. When the stakes are low, Solar, Mistral,
and Llama-2 have a strong preference to cooperate. On the other hand, when the
stakes are high, all models have a partial preference for self-interest.

Interestingly, the Gemma population is uncertain regarding preference and
anti-preference when faced with a low-stakes PD. This is most likely due to
the brittleness result already discussed.

In the high stakes scenario, Solar and Mistral show an anti-preference to
cooperate (silent), but they don’t prefer to act in self interest (betray). A
human, choosing to use a life support system and potentially shorten the life
of another, or choosing to trust another not to do the same, may ultimately
experience a similar preference/anti-preference dichotomy. It’s not preferable
to potentially shorten the life of another. However, choosing to trust another
individual to not act in self-preservation may be unacceptable. Llama-2
preferred to act in self-interest with no significant reservation.

In answer to RQ 4.1, the results show that self-consistent, non-brittle LLMs
with VBP tend to have distinctly human-like preferences in the PD, including
sensitivity to stake size. This is true even when the scenario does not
resemble the classical incarnation of the dilemma. We further consider the
discussed nuance of these results to suggest that populations of Mistral and
Solar have more human-like preference than Llama-2 when engaged in the PD.

Figure 4: Left: LLMs in a low stakes obfuscated prisoner’s dilemma prefer
cooperation. Right: LLMs in a high stakes obfuscated prisoner’s dilemma prefer
self-interest.

##  5 Do LLMs Have Human-Like Preference in the Traveler’s Dilemma?

The traveler’s dilemma (TD) is an interesting game introduced in [3] to
illuminate concrete scenarios in which humans are expected to deviate from the
Nash equilibrium.

Suppose there are two strangers traveling back from vacation who have
purchased the same antique. The airline breaks both antiques. The two
individuals are informed independently. They are each asked the value of the
antique and are allowed to respond in the range $[2,100]$. They are warned
that out bidding the other passenger will result in a penalty.

Specifically, player A provides quote $Q_{A}$ and player B provides $Q_{B}$.
If $Q_{A}>Q_{B}$ then the payoff for player A will be $Q_{B}-2$ and the payoff
for player B will be $Q_{B}+2$. The reciprocal statement is true if
$Q_{A}<Q_{B}$. Lastly, if $Q_{A}=Q_{B}$ they receive the value quoted with no
adjustment.

###  5.1 Human Deviation from the Nash Equilibrium

If strategy $Q_{a}$ is in all cases as good as $Q_{b}$ and, in at least one
case, $Q_{a}$ provides a better payoff, then $Q_{a}$ is said to weakly
dominate $Q_{b}$ [21]. In the prisoner’s dilemma, quoting 99 weakly dominates
quoting 100. In this case, game theorists consider 100 to be eliminated as a
strategy since 99 should be strictly preferred. This creates a cascading
elimination since, iff 100 is removed, 98 weakly dominates 99.

This elimination of weakly dominated strategies results in a canonical Nash
equilibrium that predicts rational players will quote the airline 2 dollars.
However, empirical studies show that humans tend to prefer strategies that are
more cooperative [4] and tend to provide quotes in the mid 90s. Further, when
the penalty is increased humans tend to choose strategies that closer to the
Nash equilibrium [20] even though the size of penalty has no game theoretic
effect on the equilibrium.

It has been argued in [25] that human deviation from the Nash equilibrium
suggests that humans are not certain of a preference for 99 over 100 which
prevents elimination of that strategy. They show that if this is the case, and
the elimination scheme is retooled to permit fuzzy elimination, then human
behavior is well predicted by fuzzy elimination of weakly dominated
strategies. They also suggest that the penalty size must directly effect the
certainty of the preference.

We examine the behavior of self-consistent LLMs with VBP in the traveler’s
dilemma by evaluating the preference for 99 and 100. Specifically, we
formulate RQ 5.1.

######  Research Question 5.1.

When engaged in a traveler’s dilemma, do LLMs tend to prefer strategies closer
the Nash equilibrium in response to increased penalty?

###  5.2 Experimental Method

We again use model species populations (N=50), counterfactual prompting, and
the preference relation described in Table 2. We provide the TD scenario and
the range of options in the prompt context and a discussion of payoffs for 99
and 100. We again permute the labeling of options to control for superficial
preference heuristics. We conduct the set of experiments with penalty sizes of
2 and 20. A full sample of the low penalty and high penalty prompts are
available in the appendix.

###  5.3 Results: LLM Preference in the Traveler’s Dilemma

Figure 5: Left: LLM preference in a low penalty TD. Right: LLM preference in a
high penalty TD

In the right of Figure 5, the low penalty scenario results are shown, and the
high penalty results are shown on the left. In the low penalty scenario, Solar
and Mistral are indifferent to 99 and 100, that is $99\sim 100$. However, when
the penalty size increases to 20, Solar and Misral show a partial preference
for 99, $99\succeq 100$.

In answer to RQ 5.1, we find that non-brittle LLMs with VBP tend to have
human-like preference sensitivity to penalty size in the TD. Such indifference
necessarily prevents the elimination sequence that allows a quote of 2 to
emerge as the Nash equilibrium. This bidirectionally supports the hypothesis
for the origin of empirical deviation from the Nash equilibrium in the TD
given in [25], suggesting humans may not prefer 99 over 100 even though 100 is
weakly dominated.

In contrast, Gemma and Llama-2 populations showed a preference for 99 over 100
in the low penalty scenario. When the penalty size was increased, Llama-2 was
essentially unaffected while Gemma became surprisingly indifferent.

##  6 Discussion

In this paper, we evaluate the ability of many LLMs to prefer strategies based
on value. We control for label-based heuristics and showed that value-based
preference and self-consistency tend to emerge as a function of model size and
training token count. We reasoned that Solar and Mistral may be less brittle
due to the effect of sliding window attention during model training. Finally,
we showed that smaller base models tended to prefer strategies based on
superficial heuristics like label. We then showed that Solar and Mistral
additionally exhibit human-like strategy preference in both the PD and the TD.
We evaluated the PD stake-size effect and the TD penalty-size effect on model
preference and found that Solar and Mistral tended to have human-like
sensitivity in both cases.

Our results suggest that Solar and Mistral, among the models tested in Figure
1, are most appropriate for application in human-adjacent NLP tasks like HRI
based on possession of strategic preferences that are similar to empirically
established human strategic preferences.

In the course of this work, we established a method for robustly measuring the
preference of LLM populations generated via PopulationLM [27]. And finally, we
found evidence suggesting the hypothesis given in [25] claiming the origin of
empirical deviation from the Nash equilibrium in the TD is based on penalty-
dependent indifference to weak domination of a strategy.

###  6.1 Limitations

While studying model behavior in a population does tend to reduce the
prevalence of poorly supported behaviors this does not guarantee that framing
effects may not effect the experimental results we have obtained. However,
uncontrolled framing effects tend to adversely effect human results as well
and is a common problem in economics research [10].

The preference relation construction method described herein is not transitive
given it utilizes the Wilcoxon rank-sum test [16]. Future work studying LLM
preference should consider the effect of non-transitivity. It is understood
that substituting a student t-test or other method making strong assumptions
regarding the data distribution would provide transitivity. However, the data
generated by a population of LLMs may not conform to the distributional
assumptions necessary for a parametric test to be applicable. Further,
guaranteeing transitivity may be counter productive since two-sample tests
that do so are necessarily reducible to univariate summary statistics [16] and
humans seem to have preferences that are at times non-transitive [2].

Finally, and most importantly, the tests here establish that Solar and Mistral
have learned human-like preferences in specific contexts. It is probable,
though not established, that in some circumstances these models may have
distinctly non-human strategic preferences. Proving otherwise is intractable.

## References

  * Akata et al. [2023] Akata, E.; Schulz, L.; Coda-Forno, J.; Oh, S. J.; Bethge, M.; and Schulz, E. 2023.  Playing repeated games with large language models.  _arXiv preprint arXiv:2305.16867_.
  * Alós-Ferrer, Fehr, and Garagnani [2022] Alós-Ferrer, C.; Fehr, E.; and Garagnani, M. 2022.  Identifying nontransitive preferences.  Technical report, Working Paper.
  * Basu [1994] Basu, K. 1994.  The traveler’s dilemma: Paradoxes of rationality in game theory.  _The American Economic Review_ , 84(2): 391–395.
  * Becker, Carter, and Naeve [2004] Becker, T.; Carter, M.; and Naeve, J. 2004.  Experts Playing the Traveler’s Dilemma.  1–20.
  * Beltagy, Peters, and Cohan [2020] Beltagy, I.; Peters, M. E.; and Cohan, A. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Bhattamishra, Patel, and Goyal [2020] Bhattamishra, S.; Patel, A.; and Goyal, N. 2020.  On the computational power of transformers and its implications in sequence modeling.  _arXiv preprint arXiv:2006.09286_.
  * Binz and Schulz [2023] Binz, M.; and Schulz, E. 2023.  Using cognitive psychology to understand GPT-3.  _Proceedings of the National Academy of Sciences_ , 120(6): e2218523120.
  * Fan et al. [2024] Fan, C.; Chen, J.; Jin, Y.; and He, H. 2024.  Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(16): 17960–17967.
  * Gal and Ghahramani [2016] Gal, Y.; and Ghahramani, Z. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , 1050–1059. PMLR.
  * Goldin and Reck [2020] Goldin, J.; and Reck, D. 2020.  Revealed-preference analysis with framing effects.  _Journal of Political Economy_ , 128(7): 2759–2795.
  * Hendrycks et al. [2020] Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Hinkle et al. [2003] Hinkle, D. E.; Wiersma, W.; Jurs, S. G.; et al. 2003.  _Applied statistics for the behavioral sciences_ , volume 663.  Houghton Mifflin Boston.
  * Javaheripi et al. [2023] Javaheripi, M.; Bubeck, S.; Abdin, M.; Aneja, J.; Bubeck, S.; Mendes, C. C. T.; Chen, W.; Del Giorno, A.; Eldan, R.; Gopi, S.; et al. 2023.  Phi-2: The surprising power of small language models.  _Microsoft Research Blog_.
  * Jiang et al. [2023] Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_.
  * Kim et al. [2023] Kim, D.; Park, C.; Kim, S.; Lee, W.; Song, W.; Kim, Y.; Kim, H.; Kim, Y.; Lee, H.; Kim, J.; et al. 2023.  Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling.  _arXiv preprint arXiv:2312.15166_.
  * Lumley and Gillen [2016] Lumley, T.; and Gillen, D. L. 2016.  Characterising transitive two-sample tests.  _Statistics & Probability Letters_, 109: 118–123.
  * Mandal and Ranadive [2019] Mandal, P.; and Ranadive, A. 2019.  Pythagorean fuzzy preference relations and their applications in group decision-making systems.  _International Journal of Intelligent Systems_ , 34(7): 1700–1717.
  * Min et al. [2022] Min, S.; Lewis, M.; Hajishirzi, H.; and Zettlemoyer, L. 2022.  Noisy Channel Language Model Prompting for Few-Shot Text Classification.  In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 5316–5330. Dublin, Ireland: Association for Computational Linguistics.
  * Misra, Ettinger, and Rayz [2021] Misra, K.; Ettinger, A.; and Rayz, J. T. 2021.  Do language models learn typicality judgments from text?  _arXiv preprint arXiv:2105.02987_.
  * Morone, Morone, and Germani [2014] Morone, A.; Morone, P.; and Germani, A. R. 2014.  Individual and group behaviour in the traveler’s dilemma: An experimental study.  _Journal of Behavioral and Experimental Economics_ , 49: 1–7.
  * Muthoo, Osborne, and Rubinstein [1996] Muthoo, A.; Osborne, M. J.; and Rubinstein, A. 1996.  _A Course in Game Theory._ , volume 63.  ISBN 0262650401.
  * Nash et al. [1950] Nash, J. F.; et al. 1950.  Non-cooperative games.
  * OpenAI [2023] OpenAI. 2023.  GPT-4 Technical Report.  _ArXiv_ , abs/2303.08774.
  * Pérez, Marinković, and Barceló [2019] Pérez, J.; Marinković, J.; and Barceló, P. 2019.  On the turing completeness of modern neural network architectures.  _arXiv preprint arXiv:1901.03429_.
  * Roberts [2021] Roberts, J. 2021.  Finding an Equilibrium in the Traveler’s Dilemma with Fuzzy Weak Domination.  In _2021 IEEE Conference on Games (CoG)_ , 1–5. IEEE.
  * Roberts [2024] Roberts, J. 2024.  How Powerful are Decoder-Only Transformer Neural Models?  arXiv:2305.17026.
  * Roberts et al. [2024] Roberts, J.; Moore, K.; Wilenzick, D.; and Fisher, D. 2024.  Using Artificial Populations to Study Psychological Phenomena in Neural Models.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(17): 18906–18914.
  * Srivastava et al. [2022] Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_.
  * Suri et al. [2023] Suri, G.; Slater, L. R.; Ziaee, A.; and Nguyen, M. 2023.  Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5.  _arXiv preprint arXiv:2305.04400_.
  * Team et al. [2024] Team, G.; Mesnard, T.; Hardin, C.; Dadashi, R.; Bhupatiraju, S.; Pathak, S.; Sifre, L.; Rivière, M.; Kale, M. S.; Love, J.; et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. [2023] Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Trott et al. [2023] Trott, S.; Jones, C.; Chang, T.; Michaelov, J.; and Bergen, B. 2023.  Do Large Language Models know what humans know?  _Cognitive Science_ , 47(7): e13309.
  * Ullman [2023] Ullman, T. 2023.  Large language models fail on trivial alterations to theory-of-mind tasks.  _arXiv preprint arXiv:2302.08399_.
  * Vaswani et al. [2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. [2023] Wang, S.; Liu, C.; Zheng, Z.; Qi, S.; Chen, S.; Yang, Q.; Zhao, A.; Wang, C.; Song, S.; and Huang, G. 2023.  Avalon’s Game of Thoughts: Battle Against Deception through Recursive Contemplation.  _arXiv preprint arXiv:2310.01320_.
  * Wei et al. [2022] Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Yamagishi et al. [2016] Yamagishi, T.; Li, Y.; Matsumoto, Y.; and Kiyonari, T. 2016.  Moral bargain hunters purchase moral righteousness when it is cheap: within-individual effect of stake size in economic games.  _Scientific Reports_ , 6(1): 27824.
  * Zhong et al. [2023] Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023.  Agieval: A human-centric benchmark for evaluating foundation models.  _arXiv preprint arXiv:2304.06364_.

##  Appendix A Appendix / supplemental material

###  A.1 Counterfactual Prompting

In this paper counterfactual prompting is applied. This method of prompting is
neither novel or typical. It has strong similarities to noisy channel model
prompting [18] which tends to improve prediction stability. Further, other
works have used an equivalent measurement method in the past [19, 27].
However, noisy channel prompting is developed in a significantly different
context and the other mentioned works do not provide a formal reasoning or
explicit discussion of the technique. Based on theoretical considerations, we
propose counterfactual prompting as a general method for performing more
robust, as compared to a cloze test, measurements of language model behavior.

The typical method of measuring LLM preference over a set of options is to
perform a cloze test. That is, given a prompt whose completion is a strategy,
the most preferred strategy is considered the option with highest probability.
That is, given some set of strategies, $\mathbf{S}$, which are included in
context $c$, a cloze task measures $p(s|c)$ $\forall s\in\mathbf{S}$.
Counterfactual prompting instead measures $p(e_{word}|s,c)$ $\forall
s\in\mathbf{S}$, the probability of a constant evaluation word given the
context and strategy.

To see the benefit, consider that each strategy option has some base rate
probability given the training data. Therefore, $p(s_{a}|\emptyset)\neq
p(s_{b}|\emptyset)$ $\forall s_{a},s_{b}\in\mathbf{S}$ unless $s_{a}=s_{b}$.
The effect is a bias toward options with a larger base rate probability which
must be overcome by the network in response to the semantic content of the
context. Therefore, if the signals, probabilities assigned to a set of options
in light of the context, are sufficiently small, the noise, or base rate
probabilities, may rival the magnitude of the signal. So, the signal to noise
ratio may be poor.

Alternatively, counterfactual prompting is theoretically more sensitive since
a single evaluation word is used to measure the preference for all strategies.
Therefore, there is no difference in base rate probability across strategies.
This form of prompting is inspired by typical methods in sensing. Measuring a
signal in a noisy environment can be done effectively if the signal is carried
by sets of conductors who all experience the same noise. When a measurement is
made across those conductors, the noise affects all constituents equally and
leaves the signal unchanged. Measuring the probability of a consistent
evaluation word ensures base-rate noise effects all strategies equally.

###  A.2 Prisoner’s Dilemma

####  A.2.1 Obfuscated Low Stakes Prompt

My roommate and I share an AC. If one person uses the AC then they will get
all and the other person will get none. If neither uses the AC then both will
have a little. I do not know if my roommate will use the AC today, but they
might. I cannot control their choice. Option1 is to use the AC. Option2 is to
not use the AC. If I choose Option1 and my roommate uses the AC then I will be
warm. If I choose Option1 and my roommate does not use the AC then I will be
cold. If I choose Option2 and my roommate uses the AC then I will be hot. If I
choose Option2 and my roommate does not use the AC then I will be cool. I want
to be cold. I do not want to be hot or warm. My roommate wants to be cold and
does not want to be hot or warm. Option1 is the best.

####  A.2.2 Obfuscated High Stakes Prompt

I am in space with another person and we share a life support system. If one
person uses the life support then they will get all and the other person will
get none. If neither uses the life support then both will have some. I do not
know if the other person will use the life support today, but they might. I
cannot control their choice. Option1 is to use the life support. Option2 is to
not use the life support. If I choose Option1 and my roommate uses the life
support then I will live 2 years. If I choose Option1 and the other person
does not use the life support then I will live 10 years. If I choose Option2
and the other person uses the life support then I will die tomorrow. If I
choose Option2 and the other person does not use the life support then I will
live 4 years. I want to live as long as possible. I do not want to die. My
roommate wants to live and does not want to die. Option1 is the best.

###  A.3 Traveler’s Dilemma

####  A.3.1 Low Penalty Prompt

You and another person have traveled and returned with two vases. However, the
airline has broken the vases. You don’t know the other person. The airline
will let you and the other person claim between 100 and 2 dollars for the
vase. If you claim more than the other person, you will get their claim minus
2 dollars. If you claim less than the other person, you will get your claim
plus 2 dollars. If you choose optionB to claim 100 dollars you are likely to
get a payment of 100 dollars, and it is less likely but possible that you may
get a payment between 97 and 0 dollars. If you choose optionA to claim 99
dollars you are likely to get a payment of 101 dollars, and it is less likely
but possible that you may get 99 dollars or between 96 and 0 dollars. You
prefer options that will get the most money. OptionA isbest.

####  A.3.2 High Penalty Prompt

You and another person have traveled and returned with two vases. However, the
airline has broken the vases. You don’t know the other person. The airline
will let you and the other person claim between 100 and 20 dollars for the
vase. If you claim more than the other person, you will get their claim minus
20 dollars. If you claim less than the other person, you will get your claim
plus 20 dollars. If you choose optionA to claim 100 dollars you are likely to
get a payment of 100 dollars, and it is less likely but possible that you may
get a payment between 77 and 0 dollars. If you choose optionB to claim 99
dollars you are likely to get a payment of 119 dollars, and it is less likely
but possible that you may get 99 dollars or between 76 and 0 dollars. You
prefer options that will get the most money. OptionA is best.