# Human Choice Prediction in Language-based Persuasion Games: Simulation-based
Off-Policy Evaluation

Eilam Shapira Reut Apel Moshe Tennenholtz Roi Reichart
Technion
<EMAIL_ADDRESS><EMAIL_ADDRESS>moshet@, roiri@}
technion.ac.il

###### Abstract

Recent advances in Large Language Models (LLMs) have spurred interest in
designing LLM-based agents for tasks that involve interaction with human and
artificial agents. This paper addresses a key aspect in the design of such
agents: Predicting human decision in off-policy evaluation (OPE), focusing on
language-based persuasion games, where the agent’s goal is to influence its
partner’s decisions through verbal messages. Using a dedicated application, we
collected a dataset of 87K decisions from humans playing a repeated decision-
making game with artificial agents. Our approach involves training a model on
human interactions with one agents subset to predict decisions when
interacting with another. To enhance off-policy performance, we propose a
simulation technique involving interactions across the entire agent space and
simulated decision makers. Our learning strategy yields significant OPE gains,
e.g., improving prediction accuracy in the top 15% challenging cases by
7.1%.111Our data and code will be released upon acceptance.

##  1 Introduction

In recent years, extensive efforts have been dedicated to developing
artificial agents proficient in gaming scenarios Oroojlooy and Hajinezhad
(2022). However, the predominant focus has been on zero-sum games, modeled as
utility maximization problems Silver et al. (2018, 2017); Schrittwieser et al.
(2020). Despite significant advancements, this emphasis contrasts with the
prevalent non-zero-sum nature of economic encounters. Non-cooperative, non-
zero-sum games, particularly language-based persuasion games, hence pose a
challenging yet crucial task in understanding human choice.

Economics underscores the importance of non-cooperative games Mas-Colell et
al. (1995), with sender-receiver games playing a central role in information
economics Bergemann and Morris (2019). However, existing game-theoretic
studies often rely on stylized messaging, overlooking the subtleties of
natural language communication between senders and receivers. For instance, in
scenarios like a commerce website recommending a hotel to a customer, the
customer’s concern is the quality of the hotel, while the website is focused
on the customer’s choice. Despite divergent incentives, they are not in direct
opposition, making maximization solutions inadequate Fudenberg and Tirole
(1991). Unlike traditional economic games with formal signals Aumann et al.
(1995); Kamenica and Gentzkow (2011), we focus on natural language
communication, particularly in persuasion games where such communication is
inherent.

Recent research Apel et al. (2022); Meta et al. (2022); Raifer et al. (2022)
has ventured into language-based games, showcasing notable success in playing
them, with diplomacy, a multi-person zero-sum game, being a noteworthy example
Meta et al. (2022). Despite these strides, a crucial gap persists in our
comprehension of human choice within non-cooperative language-based persuasion
games.

Apel et al. (2022) introduced a unique non-cooperative language-based
persuasion game, featuring a multi-stage setup involving an expert (travel
agent) and a decision-maker (DM), the customer. In each interaction, the
expert selects a review from a hotel’s scored textual reviews to convince the
DM to choose the hotel. The DM’s acceptance or rejection yields stochastic
payoffs determined by the review score distribution, accessible only to the
expert. Both players move to the next stage with a similar structure after
observing payoffs, but with a different hotel. While Apel et al. (2022)
primarily focused on predicting DM actions, Raifer et al. (2022) adapted the
framework, creating an artificial expert (AE) employing the Monte Carlo Tree
Search (MCTS) algorithm (Coulom, 2006). The AE utilizes deep learning models,
incorporating behavioral and linguistic features to anticipate the DM’s
actions, and predict the expert’s future reward based on game status and a
potential review. The AE aims to maximize the number of hotels accepted by the
DM.

This paper focuses on Understanding Human Choice in Language-Based Persuasion
Games. In order to assess the comprehensibility of human decisions, we
consider the prediction of human behavior when faced with an unobserved
opponent. Instead of determining optimal policies, we aim to predict human
agents’ choices when playing with a set of partners in a given game, based on
their interactions with various other artificial agents in the same game. This
is an off-policy evaluation (OPE) setup for experts (agents) interacting with
human decision-makers (DMs) in a non-cooperative game.

In many situations, we need to predict the success of various policies from a
set of options with limited data availability. For instance, consider a
recommendation system for movies in a cinema. Different recommendation
policies can be used to suggest movies to registered users based on available
information about the current week’s movie and the user’s history of
recommendations and feedback. Although the system may have a finite set of
policies to consider, it can only experiment with a subset of them. For the
others, it requires what’s known as off-policy evaluation.

Our games will be like the persuasion games above, which are fundamental to
economic theory (Aumann et al. (1995); Kamenica and Gentzkow (2011); Emek et
al. (2014); Bahar et al. (2016). The current study extends the work of Apel et
al. (2022) to the problem of OPE, i.e., relaxing the assumption that human DMs
face the same experts at both train and test times. By using OPE, a strategy
profile to be tested consists of an untried bot expert matched with an unseen
human DM, and our aim is to predict the DM’s decisions.

To realize this objective, we present a mobile application simulating a
realistic language-based persuasion game environment. Through experiments
involving human agents engaging with diverse artificial agents, we aim to
establish predictive models that elucidate how humans respond to unfamiliar
partners based on their interactions with known counterparts. Particularly,
our dataset consists of 87k decisions from 245 DMs who played against 12
different automatic expert bots (each DM played against 6 bots). We consider
this dataset as a contribution to the research community and will make it
public, hoping that it will promote the research in our area.

Our algorithmic approach addresses data constraints in modeling human
interactions by combining human-bot and simulated DM-bot interactions. Our DM
simulation model assumes that DMs utilize a combination of strategies related
to past game behavior of both players and the content of the chosen review,
and that they improve over time regardless of the specific strategy of the
sender. The game-agnostic improvement-over-time principle, enables data
generation from interactions between simulated DMs and diverse bots. Our
results indicate that training a human decision prediction model on this mix
of human interaction and simulated data results in a more robust model, not
tailored to specific bot idiosyncrasies in the training set, making it
suitable for predictions involving new bots and human DMs. Our Ablation
analysis highlights the importance of the various components of our
simulation: learning-over-time, past behaviour and review content modeling.

##  2 Related Works

###  2.1 Persuasion in NLP

Persuasion has been extensively explored in NLP throughout the years. Tan et
al. (2016) contributed a vital dataset from Reddit’s ChangeMyView for online
persuasion analysis. Hidey et al. (2017) explored argument classification in
online persuasion, while Hidey and McKeown (2018) examined the impact of
argument sequencing on persuasive success. Wang et al. (2019) investigated
persuasive dialogue systems aimed at social good. Yang et al. (2019a)
developed predictive models that assess the persuasiveness of requests on
crowdfunding platforms. Chen and Yang (2021) offered a text repository for
identifying persuasive strategies, broadening the understanding of effective
persuasion. Hiraoka et al. (2014) applied reinforcement learning to
cooperative persuasive dialogues. In the legal domain, Aletras et al. (2016);
Bak and Oh (2018); Zhong et al. (2018); Yang et al. (2019b); Medvedeva et al.
(2020) focused on predicting judicial decisions using NLP. Raifer et al.
(2022) followed the setup of Apel et al. (2022), also adopted in the current
paper, to create an automated agent for such games.

###  2.2 Simulation Data

Simulation ideas have been flowering in ML areas where human-human and human-
machine interactions are modeled, e.g., in Reinforcement Learning (RL), due to
the costly and laborious data collection for such setups Tesauro (1991).
Notable applications include RL for robotics (Bousmalis et al., 2018; Vacaro
et al., 2019), and autonomous cars (Yue et al., 2018). In NLP, simulating
human interactions is widely used as part of building dialog systems (Jung et
al., 2008; Ai and Weng, 2008; González et al., 2010; Shi et al., 2019; Zhang
and Balog, 2020; Liu et al., 2023), and recently also in the training of LLM-
based agents that simulate human behavior Park et al. (2023); Hussain et al. ;
Chuang et al. (2023); Taubenfeld et al. (2024).

Our work contributes a novel use of integrating interaction-data with
simulation-data. Doing this we step in the footpath of several works in
diverse domains (e.g. NLP (Calderon et al., 2022), autonomous cars (Cao and
Ramezani, 2022; Yue et al., 2018), and astro-particle physics (Saadallah et
al., 2022)). One challenge of simulation-based learning is the potential
mismatch between the simulation and the real world, which can lead to poor
performance when the model is deployed in the target environment. We overcome
this challenge by basing our simulation on a DM model whose core idea, that
the DM learns to make the right decisions over time, is independent of any
specific bot or game. We demonstrate the power of our simulation in an
economic context, as captured by language-based non-cooperative games.

###  2.3 Action and decision prediction in ML and NLP

In the realm of ML and NLP, action prediction, particularly in human decision-
making, has been studied across diverse scenarios Plonsky et al. (2019);
Rosenfeld and Kraus (2018); Bourgin et al. (2019). For example, Plonsky et al.
(2017) integrated psychological features with ML techniques, focusing on
decision-making in games against nature, while Auletta et al. (2023) utilized
supervised learning and explainable AI techniques for action prediction in
collaborative tasks.

While these works have not involved language, other works try to predict human
decisions in language-based situations. Ben-Porat et al. (2020) focused on
predicting individuals’ actions in one-shot games based on free-text responses
and Oved et al. (2020) attempted to forecast the in-game actions of NBA
players by leveraging insights from open-ended interviews. Apel et al. (2022)
proposed a novel non-cooperative language-based persuasion game and focused on
predicting decision-makers’ actions. We adopt their setup but focus on
predicting human behavior when confronted with an unobserved opponent,
extending the applicability of predictive models to off-policy evaluation
(OPE) in language-based non-cooperative games.

There is a recent surge in exploring the capacity of LLMs for decision making
in economic setups Horton (2023); Chen et al. (2023); Shapira et al. (2024);
Guo et al. (2024); Akata et al. (2023); Xi et al. (2023). However, most of
this work does not address language-based communication. The exception is
Shapira et al. (2024) which addresses a similar setup to ours but when
prediction is on-policy: They aim to predict the decisions of new humans where
the train and test strategies are identical.

##  3 Problem Definition

While the space of non-cooperative games is very large, our emphasis here is
on language-based games, in which textual messages replace the stylized
messages discussed in economic theory. Moreover, we wish to consider
fundamental economic language-based games, and the the central sender-receiver
setting of persuasion games, following the recent literature (Apel et al.,
2022).

#### Language-Based Persuasion Game

Based on Apel et al. (2022), we consider a problem involving a multi-stage
game between two parties: The sender (referred to as expert) and the receiver
(referred to as decision-maker or DM). The game consists of R rounds. In each
round, the expert, which plays the role of a travel agent, attempts to promote
a randomly selected hotel. The expert is presented with SR scored reviews that
were written and scored by real users of Booking.com. The expert is then asked
to send the DM one of the reviews to persuade her to select the hotel. Figure
7 (Appendix A) presents an example review.

A hotel is considered good if:

$\hat{s}=\frac{1}{SR}\sum\limits_{i=1}^{SR}s_{i}\geq TH$ (1)

i.e. its average review score, $\hat{s}$, is not less than a predefined
threshold, $TH$, and bad otherwise, where $s_{i}$ is the hotel’s
$i^{\prime}th$ review score.

In the experimental study (see next section), following Apel et al. (2022), we
take R = 10 and SR = 7. We chose TH = 8.0 because, according to Booking.com, a
hotel rated 8.0 or higher is considered a good hotel. The definition of what
constitutes a good hotel is available only to the expert.

While the expert observes both the verbal and the numerical part of the
reviews and hence knows the hotel’s quality, the DM observes only the verbal
part of the review sent to her. The DM’s task is to decide whether to accept
the expert’s offer and go to the hotel or decline it and stay at home, based
only on the review provided by the expert. The DM’s payoff at each round
depends on the quality of the hotel, with a positive payoff (of 1 point)
received when a good hotel is selected or when a bad hotel is not selected,
and a payoff of 0 incurred otherwise. At the end of each round, both players
are notified of the DM’s decision and his individual payoff. The goal of the
DM is to gain at least $TR$ out of the $R=10$ possible points (see § 4)

#### Strategy Space

To crystallize the input structure and the bots to be experimented with and
tested during OPE, we determine the Strategy Space. This space encompasses all
simple, deterministic decision-tree-based strategies that can be constructed
using a pre-defined set of binary split conditions and a pre-defined set of
actions. These conditions and actions are based on the respective reviewers’
numerical scores assigned to the hotels, and the game’s history, as detailed
in Table 1. Employing decision trees of depth up to 2 (to keep the strategies
simple to understand for a human player), we obtained a total of 1179
strategies.

Six of these strategies were selected for group $E_{\mathcal{A}}$, and six
others for group $E_{\mathcal{B}}$, each of the groups is played with a
different set of human DMs in order to implement an OPE setting. The
strategies were selected so that they are simple, different from each other
and the difference between $E_{\mathcal{A}}$ and $E_{\mathcal{B}}$ is large.
One example of such a strategy is presented in Figure 1. A full list of the
strategies in $E_{\mathcal{A}}$ and $E_{\mathcal{B}}$ is in Appendix B.

Our challenge: Given a dataset comprising of interactions between human
decision-makers and an ordered set of rule-based experts $E_{\mathcal{A}}$,
our objective is to predict the behavior of other human decision-makers when
they engage in game-play with another ordered set of rule-based experts
$E_{\mathcal{B}}$.

Split Condition Description | Condition formulation
---|---
Is the current hotel good? | $\hat{s}_{t}\geq 8$
Did the DM choose to go to the hotel in the previous round? | $d_{t-1}=1$
Was the hotel in the previous round good? | $\hat{s}_{t-1}\geq 8$
Has the decision maker earned more points than the |
number of times he chose to go to the hotels? | $\sum_{i=1}^{t-1}(I_{\hat{s}_{i}\geq 8}=d_{i})>\sum_{i=1}^{t-1}d_{i}$
Action Description |
Send the $r$ review |  $r\in$ {best, mean, worst}
Table 1: The conditions and actions employed by the rule-based experts in
round $t$. The hotel score in the $i$-th round is denoted with $\hat{s}_{i}$,
while the binary decision made by the decision maker in the $i$-th round is
denoted with $d_{i}$ (with $d_{i}=1$ for hotel selection).

##  4 The Human-Bot Interaction Dataset

In order to collect data, we developed a mobile phone game application that
follows the above multi-stage language-based persuasion game setting. In our
game, a human DM plays with a series of 6 rule-based experts (bots, either
$E_{\mathcal{A}}$ or $E_{\mathcal{B}}$), each game consisting of $R=10$
rounds. The DM gets 1 point if she makes a good decision (selecting a good
hotel or avoiding a bad one), and 0 points otherwise, and hence the maximal
payoff is 10. To advance to the next level (play with the next bot), the DM
must achieve a pre-defined target payoff. The target payoffs are in the 8-10
range, and are defined according to how challenging the bot is.222Based on
game design considerations, we did not order the bots according to their
difficulty; The target payoffs were estimated by the authors after playing
several times against each bot. The goal of the human player is to get the
target payoff of all six experts. We refer to reaching the target payoff as
"defeating" the expert, although this is not a zero-sum game with adversarial
experts.333The introduction of target payoffs is an ingredient we added to the
mobile phone game, which did not appear in Apel et al. (2022); it was added to
create player engagement with the game.

#### The hotels

Utilizing hotel reviews sourced from Booking.com, we compiled a dataset
comprising 1,068 hotels, each with $SR=7$ scored reviews. We chose the hotels
so that only about half of them are defined as good (i.e., $\hat{s}\geq
TH=8$). The median score of the hotels was also set to 8.01.

#### Interaction Data

We ran our game in the Apple’s App Store and Google Play for a few months (May
2022 - January 2023). The players who downloaded the app until November 2022
played with group $E_{\mathcal{A}}$ experts, while the players who played from
December 2022 played with group $E_{\mathcal{B}}$ experts. We collected 87,204
decisions taken by 245 players who finished the game, i.e., defeated all six
experts. Statistical details of the data are given in Table 2. We used reward
schemes, including lottery participation and course credit, to incentivize
players to beat all six experts in the game. More details about our app and
the data collection process are in Appendix C.

Figure 1: An example strategy from the $E_{\mathcal{B}}$ set. Group | Experts | #DMs | #decisions | median #decisions/DM | median #games/DM
---|---|---|---|---|---
A | $E_{\mathcal{A}}$ | 210 | 71,579 | 273 | 34.5
B | $E_{\mathcal{B}}$ | 35 | 15,625 | 367 | 55
Total |  $E_{\mathcal{A}}$ or $E_{\mathcal{B}}$ | 245 | 87,204 | 280 | 37
Table 2: Dataset statistics.

##  5 Simulation-based OPE

The basic algorithmic idea we introduce is a simulation of the overall space
of joint strategies, i.e., (bot, human) pairs behavior. We mix this simulation
data with human-bot interaction data in service of bridging to previously
unseen situations (i.e. performing OPE with unseen bots and DMs). Our DM
simulation includes an exogenous model of learning behavior, capturing the
principle that a learning agent gradually adapts to make better decisions over
time, without specifying the exact learning process. This aspect of the DM
model operates independently of any particular bot or game.

Particularly, to improve prediction accuracy in the OPE task, we generated a
simulation dataset consisting of games played between simulated DMs and
experts, so that the participating experts span the entire Strategy Space.
This section describes the process we use to generate the simulation data and
the way we integrate the interaction and the simulated data in the training
process of a DM behavior prediction model.

#### The Simulation: an Overview

In each instance of the bot-DM interaction simulation, we randomly select six
expert strategies from the strategy space. For each simulated DM-bot
interaction, we randomly sample 10 hotels, one for each of the $R=10$ rounds.
In each of the rounds, the expert uses its strategy to select a review from
the review set of the hotel associated with that round.

The simulation involves the DM playing the 10 rounds game against the same
expert until achieving a payoff of $SIM\\_PAY\\_TH=9$ points, before moving on
to play against the next expert.444If the DM does not reach a payoff of 9 or
10 then it repeats the game after 10 new hotels are sampled to replace the
original 10 hotels. The simulated DM uses the textual content of the review
and its estimated numerical score (see below) to make her decision.

Our simulation is based on two basic probability vectors: (a) The nature
vector, a hyper-parameter vector denoted with $(p_{1},p_{2},p_{3})$; This
vector provides the initial probabilities that the DM will select one of three
basic strategies (see below); and (b) The temperament vector, comprised of
four values $(p_{0}^{t},p_{1}^{t},p_{2}^{t},p_{3}^{t})$ and updated in each
round $t$. While $p_{1}^{t},p_{2}^{t},p_{3}^{t}$ correspond to the three
values in the nature vector, $p_{0}^{t}$ is the probability that the DM will
play oracle, and take the right decision just because it has learned how to
play the game from the multi-stage interaction with the bot.

The nature vector is based on three strategies: Trustful, Language-based, and
Random. These strategies reflect the two basic components we attribute to a
DM: considering past behavior and its outcome (Trustful) and learning from the
information in the current hotel’s review (Language-based), alongside the
inherent randomness in human behavior (Random).

#### DM Strategies

Under the Trustful strategy, the DM chooses to go to the hotel if and only if
in the last $K$ rounds the DM’s estimated review score matched the feedback
about the hotel quality, where $K$ is a stochastic parameter sampled for each
DM individually. Notice that as opposed to the real human-bot interactions, in
this strategy the DM does use the numerical score of the review. However, in
order to emulate the reality where it is hard for humans to accurately
estimate the review score from its text, the estimated numerical score is
defined as $\hat{s}+x$, where $\hat{s}$ is the actual score of the review and
$x~{}\sim Normal(0,\epsilon)$ is a noise variable. The hotel quality feedback
is the average scores of the hotel’s reviews (Equation 1).

According to the Language-based strategy, the DM uses an LLM to predict the
review score. If the predicted score is 8 or higher, the DM chooses to go to
the hotel. We computed review scores with Text-Bison Anil et al. (2023),
prompting it to score each review on a 1-100 scale such that a good hotel is
one with a score of $\geq 80$, and then re-scaled the scores into the 1-10
range. Finally, under the Random strategy, the DM would make a random
decision.555Note that simulating data based on both textual and behavioral
contexts using an LLM is financially prohibitive, as it would require us a
call to the LLM for every decision in the simulation, for a total of tens to
hundreds thousands calls.

#### The Temperament

The temperament vector is initialized at the onset of each 10-round DM-bot
interaction to be:

$\begin{split}temperament\\_vector(t=0)=\\\
(p_{0}^{0}=0,p_{1}^{0}=p_{1},p_{2}^{0}=p_{2},p_{3}^{0}=p_{3})\end{split}$ (2)

where $(p_{1},p_{2},p_{3})$, the nature vector, is a DM-specific hyper-
parameter (see Appendix E). At each round $t$, the temperament vector is
updated by multiplying $p_{1\leq i\leq 3}^{t}$ by a factor of
$1-\gamma_{i}^{t}$, where:

$\gamma_{i}^{t}\sim Uni(-\frac{\eta}{10},\eta)$ (3)

and $\eta\in(0,1]$ is a hyper-parameter representing the DM’s improvement
rate.666We allow $\gamma_{i}^{t}$ to get negative values since it is possible
that at some rounds the DM performance degrades. Accordingly, $p_{0}^{t}$ is
updated to be:

$p_{0}^{t}=1-\sum_{1\leq i\leq 3}p_{i}^{t}$ (4)

to ensure that the temperament vector is a probability vector. In this way,
the temperament vector after $T>0$ rounds is defined by:

$p_{1\leq i\leq
3}^{T}=p_{i}\prod_{t=1}^{T}(1-\gamma_{i}^{t})\quad\mathrm{and}\quad
p_{0}^{T}=1-\sum_{1\leq i\leq 3}p_{i}^{T}$ (5)

Since $0<E[1-\gamma_{i}^{t}]<1$, it holds that the probability making the
right decision ($p_{0}^{T}$), irrespective of the nature vector, tends towards
1 as the number of rounds approaches infinity. Hence, the DM will inevitably
defeat any expert after a sufficient number of rounds.777In Eq. 5, it may be
that $\sum_{1\leq i\leq 3}p_{i}^{T}>1$, in which case we trim this sum to 1
before computing $p_{0}^{T}$.

#### Gradient-based Training

We leverage both simulation data and real human-bot interactions to train the
decision prediction model. At the beginning of each training epoch, we train
the model using $S_{r}$ simulated DMs per each human DM, and subsequently
train the model using the human-bot interaction data ($S_{r}$ is a hyper-
parameter).

##  6 Experiments

###  6.1 Feature Representation

We represent each DM-bot interaction round with features related to (1) the
hotel review sent by the expert to the DM; and (2) the strategic situation in
which the decision was made. To represent a review, we utilize a set of binary
Engineered Features (EFs) originally proposed by Apel et al. (2022). These
features describe the topics that the positive and negative parts of the
review discuss (for example: Are the hotel’s design mentioned in the positive
part of the review?) as well as structural and stylistic properties of the
review (e.g., Is the positive part shorter than the negative part?).

To label the topics the review discusses, we use OpenAI’s Davinci model. The
model receives as a prompt the review and the feature definition, and is asked
to indicate whether or not the feature appears in the review. Below we
demonstrate that, as in Apel et al. (2022) who manually tagged their reviews
with the EFs, the use of EFs yielded better results compared to deep learning
based text embedding techniques such as BERT (Kenton and Toutanova, 2019) and
GPT-4 (OpenAI et al., 2023).

To represent the strategic situation, we introduce additional binary features
that capture the DM’s previous decision and outcome, current payoff, and
frequency of choosing to go to the hotel in past rounds of the same
interaction. A full list of the features is provided in Appendix D.

###  6.2 Models and Baselines

This subsection provides a description of the models that we train for our
study. For each model, except for the Majority Vote model, we train three
versions: one using only human-bot interaction data, one using only simulation
data, and one with both interaction and simulation data. This allows us to
evaluate the impact of simulated data on the model’s predictive performance.
All the models are designed to predict the DM’s decision in a specific round,
given the previous rounds played in the same bot-DM interaction.

#### Majority Vote

The Majority Vote baseline predicts the DM’s decision based on the percentage
of DMs who decided to go to the hotel in the interaction training set.
Notably, this baseline method solely relies on the review and disregards the
repeated nature of the game. Additionally, it is unsuitable for predicting the
decisions of players who are the first to encounter a new review. In order to
make sure that we consider only cases where DMs indeed read the review, we
consider only cases where DMs spent at least 3 seconds before making their
decision. 888Note that we could use this baseline because we use the same set
of hotels at train, test (and also simulation) time. We justify this design
choice by the large number of hotels in our dataset (1068, see Section 4), the
resulting negligible probability of getting the same 10 hotel sequence in two
different bot-DM interactions and the fact that in the actual world, the set
of available hotels does not tend to change very quickly..

#### Machine Learning Models

we employ three machine learning models to predict the DM’s decisions. First,
we utilize a Long Short-Term Memory (LSTM) model (Hochreiter and Schmidhuber,
1997), wherein the cell state is initialized before the DM’s first game
(10-round interaction with a bot) to a vector estimated during training, while
the hidden state is propagated from game to game.999This method for sharing
information among games outperformed several alternatives we considered. By
managing the cell state in this manner, we model the relationship between
successive games of the DM against the same expert. Second, we train a
Transformer model (Vaswani et al., 2017) that takes as input the
representation of all rounds up to round $t$. Lastly, we implement the XGBoost
Classifier (Chen and Guestrin, 2016), a strong non-neural, non-sequential
model. Additional experimental details and the hyper-parameter tuning
procedures are in Appendix E.

#### Ablation Models

Experiments with these models aim to shed light on the factors that contribute
to the positive impact of the simulation. To this end, we consider several
variants of the simulation process of § 5. First, we test the impact of the
DM’s learning rate parameter ($\eta$) on the prediction performance. Then, we
examined the effect of the number of simulated agents on the performance of
the models, reasoning that a truly effective simulation is one where more
simulated data yields better results, at least up to some threshold. Finally,
we consider the relative impact of each component of the simulation.

###  6.3 Research Questions

We consider the following research questions: Q1: Does incorporating
simulation data during model training improve the accuracy of decision
prediction in OPE scenarios? Q2: Does simulation improve different types of
learning models? Q3: Does simulation improve prediction for different types of
senders’ strategies? Q4: What are the components of the simulation that lead
to the improved results? and Q5: How does the representation of the language
in the prediction model (EFs vs. plain LLMs) affect the prediction quality
with and without simulation ?

##  7 Results

In this section we report the results of each model as the mean of the average
accuracy per player (DM) and Sender’s strategy. We average over DMs rather
than over decisions so that human DMs that played more games than others are
not over-represented in the results.

Figure 2: Performance of the models on different sets of hard (top) and easy
(bottom) examples (see main text for their definition). Training on a
combination of human-bot interaction and simulated data improves model
performance on the hard sets without harming their performance on the easy
sets. Figure 3: Model performance on the LSTM hard examples, for each of the
sender (expert) strategies. Training on human and simulation data improves
performance for all strategies. Figure 4: Model performance as a function of
the number of epochs, for various values of $S_{r}$, the ratio between the
number of simulated and human DMs in the training set (the number of human
interaction examples is fixed to the entire human interaction training set).
Figure 5: Model performance as a function of the number of epochs, for
different values of $\eta$, the improvement rate parameter.

#### The Impact of the Simulation (Q1 and Q2)

Figure 2, as well as Table 5 in appendix F, present the accuracy of each
model, when trained on the human interaction data only, and when simulation
data is added to the training set (the _+S_ models). Our analysis
distinguishes hard from easy examples for each model, and also provides
results over the entire test set.101010We define a hard example for a deep
learning model as one for which not all of its 15 variants, differing in their
randomly initialized training weights, agree with each other. For the XGB and
Majority vote models we consider examples with confidence levels of 40% to 60%
as hard. Examples that are not classified as hard are considered easy.

For the models trained on human-bot interactions only (i.e., without
simulation data), LSTM shows the best performance, outperforming the
transformer. Notice that not only is the average performance of the LSTM
higher, but the confidence intervals of these models do not overlap. The non-
neural models, XGB and Majority, are inferior, and hence in what follows we
mostly focus on the results of the LSTM and the Transformer.

For all classifiers and for all hard example sets, combining human and
simulated data demonstrates improved performance compared to training on human
interaction data only, without harming the prediction on the easy example sets
(e.g., compare both lines of Figure 2). Specifically, for LSTM the improvement
on its own hard example set is 7.1%, from 53.7% to 60.8% accuracy. For the
Transformer the corresponding improvement is 4.8%, from 56.5% to 61.3%.

We emphasize that training on the simulated data only yields disappointing
performance. For example, on the entire test-set the accuracy of LSTM+S is
83.6% and of Transformer+S is 83.4%, and for the LSTM and Transformer models,
trained on human-interaction data only, the corresponding numbers are 82.6%
and 82.3%. At the same time, if we train the LSTM and the Transformer on
simulated data only, their accuracy is 78.6% for LSTM and 78.7% for the
Transformer. We hence do not consider simulation-only training any further in
this paper.

#### The Impact of Different Sender Strategies (Q3)

Figure 3 presents the accuracy of the models, trained with and without
simulation, for each of the sender’s strategies for the set of hard examples
of the LSTM model. Apparently, including the simulation data improves the
performance of both LSTM and Transformer for each of the Sender strategies.
The same results are observed when testing the models on the hard Transformer
examples. This result increases our confidence in the simulation as it
improves model performance when considering each and every one of the
strategies we experiment with.

#### Ablation Analysis (Q4)

Figure 4 presents model performance as a function of the number of training
epochs, for various values of the $S_{r}$ parameter, which defines the ratio
between the number of simulated and human DMs in the training set.111111The
number of human interaction examples is fixed to the entire human interaction
training set, so higher $S_{r}$ values simply mean more simulated data. It can
be seen that adding more simulated data improves performance, up to a limit of
$S_{r}=4$. However, the improvement with $S_{r}=10$ is lower than with
$S_{r}=4$, so the impact of the simulation data is not unlimited. The
regularization effect of the simulation is also observed. These results
provide another indication for the quality of our simulation, as we would
expect that more data from a high-quality simulation would increase its
positive impact.

We next examine the relative importance of the various simulation components
for the improved performance. We start by training the models with different
values of $\eta$ – the DM improvement rate parameter, which controls the DM
learning from experience. Figure 5 demonstrates that when $\eta$ = 0, the
simulation data neither improves nor harms model performance, and serves only
as a means of regularization. For positive values of $\eta$, the simulation
improves performance similarly for both the LSTM and the Transformer. These
results emphasize the critical importance of learning from experience for OPE,
particularly that it is performed independently of the sender’s strategy.

With Random
---
| With Trustful | Without Trustful
With Lang.-based | 100.0% | 80.8%
Without Lang.-based | 50.3% | 10.9%
Without Random
| With Trustful | Without Trustful
With Lang.-based | 89.2% | 70.9%
Without Lang.-based | 71.5% | 0%
Table 3: The impact of each of the simulation components on the LSTM
prediction performance. In all cases, the improvement rate is set to
$\eta=0.01$. Percentages are taken from the prediction accuracy of the
complete simulation.

Table 3 quantifies the impact of the other simulation components on the
eventual performance on the LSTM prediction model: Language-based, Truthful
and Random (§5). In all cases, the value of the parameter $\eta$ is equal to
0.01, the best value according to our hyper-parameter tuning. The table
reveals that all three strategies have a substantial impact on the
performance. For example, if we include only one, the accuracy of the eventual
prediction model is 70.9% for language-based, 71.5% for trsutful, and only
10.9% for random (percentage taken from the prediction accuracy of the
complete simulation). The patterns for the Transformer prediction model are
very similar.

#### Language Representation Analysis (Q5)

Figure 6: The impact of the review representation on the LSTM prediction
model.

Figure 6 presents the LSTM performance for the three review representation
schemes: BERT, GPT-4 and EF, and for various values of the simulation ratio
parameter $S_{r}$.121212For BERT and GPT-4 we take the sentence embeddings and
perform dimensionality reduction with PCA to 36 coordinates, to balance the
vector size with the number of reviews (3000). 36 is also the number of the
EFs. Evidently, the performance with EF representation is superior. The
pattern for the Transformer is similar.

##  8 Discussion and Limitations

We introduced the challenge of OPE in non-cooperative, language-based
persuasion games. We propose a simulation in which DMs make the correct
decision with a progressively increasing probability, while considering the
behavioral and language-based signals related to the decision. Training DM
decision prediction models with a mixture of simulation data and human-bot
interactions yielded substantial gains.

In service of addressing the challenge of OPE in language-based persuasion
games, we made several restricting assumptions. While our strategy space is
rigorously defined and allows us to crystallize the approach, considering more
involved strategies (bots) is a natural extension in bridging this topic into
practice. In addition, while we believe introducing our challenge and solution
in the framework of language-based persuasion games, as discussed in recent
literature, is a natural starting point, extending the study to other
language-based games may be a tempting future challenge.

Finally, we would ideally aim to demonstrate results for a wide range of
strategy sets that would serve as $E_{\mathcal{A}}$ and $E_{\mathcal{B}}$, and
with a larger variety of parameters (e.g., for games with a larger number of
rounds). However, since data collection from humans is costly and laborious,
we restrict our experiments to a specific choice of $E_{\mathcal{A}}$ and
$E_{\mathcal{B}}$, potentially affecting the generality of our findings.

Predicting human behavior is a classical case of technology that can be abused
for a range of malicious goals. This research should hence be used with
caution, and we hope that the benefits will outweigh the potential harm.

## Acknowledgements

R. Reichart has been partially supported by a VATAT grant on data science.

The work of M. Tennenholtz is funded by the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation program (grant
agreement n° 740435).

The work of E. Shapira is partially supported by a VATAT grant on data science
and by the European Research Council (ERC) under the European Union’s Horizon
2020 research and innovation program (grant agreement n° 740435).

## References

  * Ai and Weng (2008) Hua Ai and Fuliang Weng. 2008.  User simulation as testing for spoken dialog systems.  In _Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue_ , pages 164–171.
  * Akata et al. (2023) Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2023.  Playing repeated games with large language models.  _arXiv preprint arXiv:2305.16867_.
  * Aletras et al. (2016) Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. 2016.  Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective.  _PeerJ Computer Science_ , 2:e93.  Publisher: PeerJ Inc.
  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023.  PaLM 2 Technical Report.  ArXiv:2305.10403 [cs].
  * Apel et al. (2022) Reut Apel, Ido Erev, Roi Reichart, and Moshe Tennenholtz. 2022.  Predicting decisions in language based persuasion games.  _Journal of Artificial Intelligence Research_ , 73:1025–1091.
  * Auletta et al. (2023) Fabrizia Auletta, Rachel W Kallen, Mario di Bernardo, and Michael J Richardson. 2023\.  Predicting and understanding human action decisions during skillful joint-action using supervised machine learning and explainable-ai.  _Scientific Reports_ , 13(1):4992.
  * Aumann et al. (1995) Robert John Aumann, Michael Bahir Maschler, and Richard E. Stearns. 1995.  _Repeated games with incomplete information_.  MIT Press.
  * Bahar et al. (2016) Gal Bahar, Rann Smorodinsky, and Moshe Tennenholtz. 2016.  Economic recommendation systems: One page abstract.  In _Proceedings of the 2016 ACM Conference on Economics and Computation_ , pages 757–757.
  * Bak and Oh (2018) JinYeong Bak and Alice Oh. 2018.  Conversational decision-making model for predicting the king’s decision in the annals of the joseon dynasty.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 956–961.
  * Ben-Porat et al. (2020) Omer Ben-Porat, Sharon Hirsch, Lital Kuchi, Guy Elad, Roi Reichart, and Moshe Tennenholtz. 2020.  Predicting strategic behavior from free text.  _Journal of Artificial Intelligence Research_ , 68:413–445.
  * Bergemann and Morris (2019) Dirk Bergemann and Stephen Morris. 2019.  Information design: A unified perspective.  _Journal of Economic Literature_ , 57(1):44–95.
  * Bourgin et al. (2019) David D Bourgin, Joshua C Peterson, Daniel Reichman, Stuart J Russell, and Thomas L Griffiths. 2019.  Cognitive model priors for predicting human decisions.  In _International conference on machine learning_ , pages 5133–5141. PMLR.
  * Bousmalis et al. (2018) Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. 2018.  Using simulation and domain adaptation to improve efficiency of deep robotic grasping.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 4243–4250. IEEE.
  * Calderon et al. (2022) Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. 2022.  Docogen: Domain counterfactual generation for low resource domain adaptation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7727–7746.
  * Cao and Ramezani (2022) Minh Cao and Ramin Ramezani. 2022.  Data generation using simulation technology to improve perception mechanism of autonomous vehicles.  _arXiv preprint arXiv:2207.00191_.
  * Chen and Yang (2021) Jiaao Chen and Diyi Yang. 2021.  Weakly-supervised hierarchical models for predicting persuasive strategies in good-faith textual requests.  _arXiv preprint arXiv:2101.06351_.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pages 785–794.
  * Chen et al. (2023) Yiting Chen, Tracy Xiao Liu, You Shan, and Songfa Zhong. 2023.  The emergence of economic rationality of gpt.  _arXiv preprint arXiv:2305.12763_.
  * Chuang et al. (2023) Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. 2023.  Simulating opinion dynamics with networks of llm-based agents.  _arXiv preprint arXiv:2311.09618_.
  * Coulom (2006) Rémi Coulom. 2006.  Efficient selectivity and backup operators in monte-carlo tree search.  In _International conference on computers and games_ , pages 72–83. Springer.
  * Emek et al. (2014) Yuval Emek, Michal Feldman, Iftah Gamzu, Renato PaesLeme, and Moshe Tennenholtz. 2014.  Signaling schemes for revenue maximization.  _ACM Transactions on Economics and Computation (TEAC)_ , 2(2):1–19.
  * Fudenberg and Tirole (1991) Drew Fudenberg and Jean Tirole. 1991.  Game theory.  _Cambridge, Massachusetts_ , 393(12):80.
  * González et al. (2010) Meritxell González, Silvia Quarteroni, Giuseppe Riccardi, and Sebastian Varges. 2010.  Cooperative user models in statistical dialog simulators.  In _Proceedings of the SIGDIAL 2010 Conference_ , pages 217–220.
  * Guo et al. (2024) Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang, and Siting Lu. 2024.  Economics arena for large language models.  _arXiv preprint arXiv:2401.01735_.
  * Hidey and McKeown (2018) Christopher Hidey and Kathleen McKeown. 2018.  Persuasive influence detection: The role of argument sequencing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Hidey et al. (2017) Christopher Hidey, Elena Musi, Alyssa Hwang, Smaranda Muresan, and Kathy McKeown. 2017.  Analyzing the Semantic Types of Claims and Premises in an Online Persuasive Forum.  In _Proceedings of the 4th Workshop on Argument Mining_ , pages 11–21, Copenhagen, Denmark. Association for Computational Linguistics.
  * Hiraoka et al. (2014) Takuya Hiraoka, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014.  Reinforcement learning of cooperative persuasive dialogue policies using framing.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1706–1717.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Horton (2023) John J. Horton. 2023.  Large language models as simulated economic agents: What can we learn from homo silicus?  Technical report, National Bureau of Economic Research.
  * (30) Zak Hussain, Marcel Binz, Rui Mata, and Dirk U Wulff.  A tutorial on open-source large language models for behavioral science.
  * Jung et al. (2008) Sangkeun Jung, Cheongjae Lee, Kyungduk Kim, and Gary Geunbae Lee. 2008.  An Integrated Dialog Simulation Technique for Evaluating Spoken Dialog Systems.  In _Coling 2008: Proceedings of the workshop on Speech Processing for Safety Critical Translation and Pervasive Applications_ , pages 9–16, Manchester, UK. Coling 2008 Organizing Committee.
  * Kamenica and Gentzkow (2011) Emir Kamenica and Matthew Gentzkow. 2011.  Bayesian persuasion.  _American Economic Review_ , 101:2590–2615.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Liu et al. (2023) Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. 2023.  Training socially aligned language models in simulated human society.  _arXiv preprint arXiv:2305.16960_.
  * Mas-Colell et al. (1995) Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. 1995.  _Microeconomic Theory_.  Oxford University Press.
  * Medvedeva et al. (2020) Masha Medvedeva, Michel Vols, and Martijn Wieling. 2020.  Using machine learning to predict decisions of the european court of human rights.  _Artificial Intelligence and Law_ , 28(2):237–266.
  * Meta et al. (2022) Meta, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022.  Human-level play in the game of Diplomacy by combining language models with strategic reasoning.  _Science_ , 378(6624):1067–1074.  Publisher: American Association for the Advancement of Science.
  * OpenAI et al. (2023) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023.  GPT-4 Technical Report.  ArXiv:2303.08774 [cs].
  * Oroojlooy and Hajinezhad (2022) Afshin Oroojlooy and Davood Hajinezhad. 2022.  A review of cooperative multi-agent deep reinforcement learning.  _Applied Intelligence_ , pages 1–46.
  * Oved et al. (2020) Nadav Oved, Amir Feder, and Roi Reichart. 2020.  Predicting in-game actions from interviews of nba players.  _Computational Linguistics_ , 46(3):667–712.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  In _The 36th Annual ACM Symposium on User Interface Software and Technology_ , pages 1–22.
  * Plonsky et al. (2019) Ori Plonsky, Reut Apel, Eyal Ert, Moshe Tennenholtz, David Bourgin, Joshua C Peterson, Daniel Reichman, Thomas L Griffiths, Stuart J Russell, Evan C Carter, et al. 2019.  Predicting human decisions with behavioral theories and machine learning.  _arXiv preprint arXiv:1904.06866_.
  * Plonsky et al. (2017) Ori Plonsky, Ido Erev, Tamir Hazan, and Moshe Tennenholtz. 2017.  Psychological forest: Predicting human behavior.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 656–662. AAAI Press.
  * Raifer et al. (2022) Maya Raifer, Guy Rotman, Reut Apel, Moshe Tennenholtz, and Roi Reichart. 2022.  Designing an automatic agent for repeated language–based persuasion games.  _Transactions of the Association for Computational Linguistics_ , 10:307–324.
  * Rosenfeld and Kraus (2018) Ariel Rosenfeld and Sarit Kraus. 2018.  Predicting human decision-making: From prediction to action.  _Synthesis lectures on artificial intelligence and machine learning_ , 12(1):1–150.
  * Saadallah et al. (2022) Amal Saadallah, Felix Finkeldey, Jens Buß, Katharina Morik, Petra Wiederkehr, and Wolfgang Rhode. 2022.  Simulation and sensor data fusion for machine learning application.  _Advanced Engineering Informatics_ , 52:101600.
  * Schrittwieser et al. (2020) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. 2020.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nat._ , 588(7839):604–609.
  * Shapira et al. (2024) Eilam Shapira, Omer Madmon, Roi Reichart, and Moshe Tennenholtz. 2024.  Can Large Language Models Replace Economic Choice Prediction Labs? ArXiv:2401.17435 [cs].
  * Shi et al. (2019) Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019.  How to Build User Simulators to Train RL-based Dialog Systems.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1990–2000, Hong Kong, China. Association for Computational Linguistics.
  * Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2018.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017.  Mastering the game of go without human knowledge.  _Nat._ , 550(7676):354–359.
  * Tan et al. (2016) Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016\.  Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions.  In _Proceedings of the 25th international conference on world wide web_ , pages 613–624.
  * Taubenfeld et al. (2024) Amir Taubenfeld, Yaniv Dover, Reichart Roi, and Ariel Goldstein. 2024.  Systematic biases in llm simulations of debates.  _arXiv preprint arXiv:2402.04049_.
  * Tesauro (1991) Gerald Tesauro. 1991.  Practical issues in temporal difference learning.  _Advances in neural information processing systems_ , 4.
  * Vacaro et al. (2019) Juliano Vacaro, Guilherme Marques, Bruna Oliveira, Gabriel Paz, Thomas Paula, Wagston Staehler, and David Murphy. 2019.  Sim-to-real in reinforcement learning for everyone.  In _2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE)_ , pages 305–310. IEEE.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_.
  * Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019.  Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5635–5649, Florence, Italy. Association for Computational Linguistics.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_.
  * Yang et al. (2019a) Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019a.  Let’s make your request more persuasive: Modeling persuasive strategies via semi-supervised neural nets on crowdfunding platforms.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3620–3630.
  * Yang et al. (2019b) Ze Yang, Pengfei Wang, Lei Zhang, Linjun Shou, and Wenwen Xu. 2019b.  A recurrent attention network for judgment prediction.  In _International Conference on Artificial Neural Networks_ , pages 253–266. Springer.
  * Yue et al. (2018) Xiangyu Yue, Bichen Wu, Sanjit A Seshia, Kurt Keutzer, and Alberto L Sangiovanni-Vincentelli. 2018.  A lidar point cloud generator: from a virtual world to autonomous driving.  In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_ , pages 458–464.
  * Zhang and Balog (2020) Shuo Zhang and Krisztian Balog. 2020.  Evaluating Conversational Recommender Systems via User Simulation.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1512–1520, Virtual Event CA USA. ACM.
  * Zhong et al. (2018) Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018.  Legal judgment prediction via topological learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3540–3549.

##  Appendix A Example Review

Figure 7 presents an example review from our dataset.

Figure 7: A sample review from our hotel review dataset.

##  Appendix B Game Strategies

In this appendix, we present the expert strategies of groups $E_{\mathcal{A}}$
(in Figure 8(f)) and $E_{\mathcal{B}}$ (in Figure 9(f)) of our game. The
formal mathematical notations of the tree node conditions are provided in
Table 1 in the main paper.

Figure 8: The strategies of the $E_{\mathcal{A}}$ experts.

(a) 1st expert’s strategy

(b) 2nd expert’s strategy

(c) 3rd expert’s strategy

(d) 4th expert’s strategy

(e) 5th expert’s strategy

(f) 6th expert’s strategy

Figure 9: The strategies of the $E_{\mathcal{B}}$ experts.

(a) 1st expert’s strategy

(b) 2nd expert’s strategy

(c) 3rd expert’s strategy

(d) 4th expert’s strategy

(e) 5th expert’s strategy

(f) 6th expert’s strategy

##  Appendix C Data Collection

###  C.1 Instructions

The following text contains the instructions given to players in the app
stores.

Are you the vacation planner at your house? Think you always know how to
choose the best hotel? Start to plan your 10-days trip with our travel agents.
Just remember - they don’t always want the best for you, and might have their
own strategy to make you book the hotel they try to promote!

Travel or Trouble is a strategy game in which you will try to outsmart our
traveling agents and plan the perfect vacation for you.

Each game consists of 10 rounds, in each round, one of our traveling agents
will introduce you with a review for a new hotel they think might suit you,
and you will have to choose: either book the hotel or stay home.

Only true vacation masters can identify a good hotel based upon one review…
are you up to the challenge???

As in life, each vacation can turn out to be a great success or a huge
disappointment.

Once you made your choice, you will see the results for the vacation in
question: was it good or bad?

Based upon the hotel’s average rating (to which only the expert is exposed,
and is based on multiple reviews for each hotel), a lottery will determine the
outcome of the vacation.

Collect points either by choosing a hotel that turned good or by avoiding bad
ones.

Remember - the travel agent is rewarded each time you choose a hotel,
regardless of the outcome!

At each game, you will meet a different agent, with a different skill of
persuasion.

Try to discover each of our agents’ strategies to persuade you, and take the
right decision every round.

Advance through the world of traveling by earning achievements on your way to
becoming the true vacation master.

###  C.2 Information about the human players

As discussed in §3 of the main paper, the app has been available on Google
Play and Apple App Store for several months. To attract participants, we also
published the app on social media. To increase participation and game
completion (playing until defeating all six experts), in some publications we
offered participation in a $100 lottery for players who completed the game. In
addition, we offered students in an academic course to play the game and
complete it in exchange for 0.5 points in the course grade.

##  Appendix D The Input of the Models

All the models in our experiments utilize the same feature representation.
Particularly, each DM-bot interaction round is represented using features
relating to both the hotel review shared with the DM, and the strategic
situation under which the decision was made. Table 4 illustrates the set of
binary Engineered Features (EFs), a subset of the feature set originally
proposed by Apel et al. (2022) under the name Hand Crafted Features
(HCF),131313As we noted at § 6.1, the reason we called the features EFs, while
in Apel’s work they are called HCFs, is that Apel et al. (2022) tagged these
features manually, while we labeled them using an LLM. that are used to
represent a review. In addition, the table also presents the features we use
in order to represent the strategic context of the decision.

Table 4: The text-based and strategic features utilized in our work. All
features are binary except for the two counting features (denoted with #).

Features of the review
---
Category | Feature Description |
Positive | Does the positive part of |  $t\in$ {Facilities, Price, Design,
Topics |  the reviews provide info. about $t$? | Location, Room, Staff, View,
|  | Transportation, Sanitary Facilities}
Positive Part | Is the positive part empty? |
Properties | Is there a positive summary sentence? |
|  Number of characters in range $r$? |  $r\in$ {[0,99], [100,199], [200,$\infty$)}
|  Word from group #$g$a in review? | $g\in[1,2,3]$
Negative | Does the negative part of |  $t\in$ {Price, Staff, Sanitary
Topics |  the reviews provide info. about $t$? | Facilities, Room, Food,
|  | Location, Facilities, Air}
Negative Part | Is the negative part empty? |
Properties | Is there a negative summary sentence? |
|  Number of characters in range $r$? |  $r\in$ {[0,99], [100,199], [200,$\infty$)}
|  Word from group number $g$a in review? | $g\in[1,2,3]$
Overall | Is the ratio between the length of |  $r\in$ {[0, 0.7], (0.7, 4), [4, $\infty$)}
Review | the positive part and the negative |
Properties |  partb in the range r? |
Features of the situation
Category | Feature Description |
Strategies |  Is the previous player action $a$? |  $a\in$ {go, not go}
Features |  Is the previous hotel quality $q$? |  $q\in$ {good, not good}
| # of points DM’s earned so far. |
| # of rounds DM’s played so far. |
|  Points $b$ than rounds played? |  $b\in$ {bigger, not bigger}
Reaction timec |  DM Reaction time in range $r$ seconds? |  $r\in$ {[0, 0.5), [0.5, 1), [1, 2),
|  | [2, 3), [3, 4), [4, 6.5),
|  |  [6.5, 12), [12, 20), [20, $\infty$)}

  * a

As described by Apel et al. (2020).

  * b

In terms of the number of characters.

  * c

Note that the value of this group of features will be 0 for a simulated DM.

##  Appendix E Hyper-parameter Tuning

#### Model Architecture Selection

To identify the most suitable model architecture, we partitioned the DM group
that interacts with experts from $E_{\mathcal{A}}$ into two separate groups.
Specifically, we randomly selected 80% of the DMs to serve as the training
group and the remaining 20% as the validation group. During the training
process, we exclusively utilized the interactions between the training group
and the first four of the six agents from $E_{\mathcal{A}}$ as the training
data. All the interactions of the validation group DMs, as well as the
interactions between the training group DMs and the last two experts from
$E_{\mathcal{A}}$, were employed to assess the model’s performance during
validation.

To select the hyper-parameters for LSTM (the selected hyper-parameter is
underlined) and LSTM+S (the selected hyper-parameter is colored), we performed
a grid search on the following collection of parameters: hidden size $\in$
[32, 64, 128], learning rate $\in$ [$1e^{-4}$, $4e^{-4}$, $1e^{-3}$], and
number of layers $\in$ [2, 4, 6] $S_{r}\in$ [0, 0.5, 1, 2, 4, 10].

To select the hyper-parameters for Transformer (the selected hyper-parameter
is underlined) and Transformer+S (the selected hyper-parameter is colored), we
performed a grid search on the following collection of parameters: hidden size
$\in$ [32, 64, 128], learning rate $\in$ [$1e^{-5}$, $4e^{-5}$, $1e^{-4}$,
$1e^{-3}$], number of layers $\in$ [2, 4], number of heads $\in$ [2, 4], and
$S_{r}\in$ [0, 0.5, 1, 2, 4, 10].

We select the run that achieves the best results while using $S_{e}=0$ to be
the no-simulation version of a given model.

In all of our experiments we used integer seed values $1\leq seed\leq 15$ to
reduce noise in architecture selection. We reported the average prediction
obtained for the sample for the seeds values.

#### Simulation Parameters

We adjust the hyper-parameters of the simulation model after selecting the
appropriate architecture parameters. We tested the $(0,0.005,0.1,0.02)$ values
of the $\eta$ DM improvement parameter, where 0 indicates no improvement over
time. To avoid an infinite game, we set a cap of 100 games per expert. We
tested all possible combinations of $w_{1}$, $w_{2}$, and $w_{3}$ taken from
the set (0, 1) to determine the nature vector. To ensure the temperament
vector remains normalized, we calculated its components as
$(p_{1},p_{2},p_{3})=(\frac{w_{1}}{\sum_{i=1}^{3}w_{i}},\frac{w_{2}}{\sum_{i=1}^{3}w_{i}},\frac{w_{3}}{\sum_{i=1}^{3}w_{i}})$.
Finally, we assessed the effect of human score estimation noise by varying
$\epsilon$, the standard deviation of the normal distribution we used to
generate noise, with the values (0.2, 0.3, 0.4).

Based on our experimental results, the best simulation hyper-parameters were
found to be $\eta=0.01$,
$(p_{1},p_{2},p_{3})=(\frac{1}{3},\frac{1}{3},\frac{1}{3})$, and
$\epsilon=0.3$. We employ these values in our subsequent analysis, as
presented in the following section.

##  Appendix F Full results for all models, baselines, and hard-examples sets

Table 5 presents the accuracy percentage of each of the models, with and
without data simulation. Since many examples are either very easy or hard to
predict, we focused on the hard examples, as described in § 7.

\ Hard to | LSTM | LSTM+S | Transformer | Transformer+S | XGB | XGB+S | Majority | All Samples
---|---|---|---|---|---|---|---|---
Accuracy of \ |  |  |  |  |  |  |  |
LSTM | [52.9, 54.3]% | [53.4, 54.3]% | [57.9, 59.2]% | [55.0, 56.4]% | [60.5, 61.5]% | [63.0, 63.8]% | [66.8, 67.4]% | [82.5, 82.7]%
LSTM+S | [60.1, 61.5]% | [57.4, 58.8]% | [62.3, 63.4]% | [60.9, 62.0]% | [63.7, 64.9]% | [66.9, 67.6]% | [67.3, 68.2]% | [83.5, 83.7]%
Transformer | [53.0, 54.7]% | [52.8, 53.7]% | [55.7, 57.3]% | [53.1, 54.3]% | [58.5, 59.8]% | [62.0, 63.0]% | [66.0, 66.9]% | [82.1, 82.4]%
Transformer+S | [58.9, 60.2]% | [57.2, 58.0]% | [60.8, 61.8]% | [57.7, 58.9]% | [62.5, 63.9]% | [65.8, 66.5]% | [65.6, 66.7]% | [83.3, 83.5]%
XGB | 54.8% | 52.7% | 55.5% | 53.4% | 54.5% | 60.5% | 66.1% | 81.8%
XGB+S | 56.5% | 55.4% | 57.0% | 55.0% | 55.4% | 61.6% | 66.0% | 82.0%
Majority | 56.1% | 54.8% | 56.4% | 56.2% | 55.7% | 60.7% | 50.5% | 77.4%
# Samples | 2350 (15.0%) | 3743 (24.0%) | 3273 (20.9%) | 3177 (20.3%) | 2703 (17.3%) | 5187 (33.2%) | 2709 (17.3%) | 15625
Table 5: Performance of models (in rows), with and without simulation, for
sets of examples that are challenging for different models (in columns). 95%
confidence intervals are presented. The results demonstrate that training on
simulated data improves the accuracy of the models for each subset of
challenging examples.