ifaamas [AAMAS ’22]Proc. of the 21st International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2022)May 9–13, 2022 OnlineP. Faliszewski,
V. Mascardi, C. Pelachaud, M.E. Taylor (eds.) 2022 2022 ??? DeepMind London
United Kingdom Princeton University Princeton New Jersey United States
Princeton University Princeton New Jersey United States

# Warmth and Competence in Human-Agent Cooperation

Kevin R. McKee<EMAIL_ADDRESS>,  Xuechunzi Bai<EMAIL_ADDRESS>and  Susan T. Fiske<EMAIL_ADDRESS>
###### Abstract.

Interaction and cooperation with humans are overarching aspirations of
artificial intelligence (AI) research. Recent studies demonstrate that AI
agents trained with deep reinforcement learning are capable of collaborating
with humans. These studies primarily evaluate human compatibility through
“objective” metrics such as task performance, obscuring potential variation in
the levels of trust and subjective preference that different agents garner. To
better understand the factors shaping subjective preferences in human-agent
cooperation, we train deep reinforcement learning agents in Coins, a two-
player social dilemma. We recruit participants for a human-agent cooperation
study and measure their impressions of the agents they encounter.
Participants’ perceptions of warmth and competence predict their stated
preferences for different agents, above and beyond objective performance
metrics. Drawing inspiration from social science and biology research, we
subsequently implement a new “partner choice” framework to elicit revealed
preferences: after playing an episode with an agent, participants are asked
whether they would like to play the next round with the same agent or to play
alone. As with stated preferences, social perception better predicts
participants’ revealed preferences than does objective performance. Given
these results, we recommend human-agent interaction researchers routinely
incorporate the measurement of social perception and subjective preferences
into their studies.

###### Key words and phrases:

Human-agent cooperation; Human-agent interaction; Warmth; Competence; Social
perception; Partner choice; Preferences

##  1\. Introduction

Trust is central to the development and deployment of artificial intelligence
(AI) (Jobin et al., 2019; Stanton and Jensen, 2021). However, many members of
the public harbor doubts and concerns about the trustworthiness of AI (Cave
and Dihal, 2019; Dietvorst et al., 2015; Fast and Horvitz, 2017; Kelley et
al., 2019). This presents a pressing issue for cooperation between humans and
AI agents (Dafoe et al., 2020).

Algorithmic development research has been slow to recognize the importance of
trust and preferences for cooperative agents. Recent studies show that deep
reinforcement learning can be used to train interactive agents for human-agent
collaboration (Carroll et al., 2019; Lockhart et al., 2020; Strouse et al.,
2021; Tylkin et al., 2021). The “human-compatible” agents from these
experiments demonstrate compelling improvements in game score, task accuracy,
and win rate over established benchmarks. However, a narrow focus on
“objective” metrics of performance obscures any differences in subjective
preferences humans develop over cooperative agents. Two agents may generate
similar benefits in terms of typical performance metrics, but human teammates
may nonetheless express a strong preference for one over the other (Siu et
al., 2021; Strouse et al., 2021). Developing human-compatible, cooperative
agents will require evaluating agents on dimensions other than objective
performance.

(a) New interagent encounter.

(b) Social perception.

(c) Partner choice.

Figure 1. When humans encounter a new agent, they automatically and rapidly
form an impression of the agent (social perception). The human can leverage
their impression to decide whether to continue or discontinue the interaction
(partner choice).

What shapes subjective preferences for artificial agents, if not a direct
mapping of agent performance? One possible source of variance is social
perception. When encountering a novel actor, humans rapidly and automatically
evaluate the actor along two underlying dimensions: warmth and competence
(Abele et al., 2021; Fiske et al., 2007; Fiske, 2018; McKee et al., 2021).
These perceptions help individuals “make sense of [other actors] in order to
guide their own actions and interactions” (Fiske, 1993) (Figure 1). The
competence dimension aligns with the established focus on performance and
score in machine learning research (Birhane et al., 2021): How effectively can
this actor achieve its interests? Appraising an actor’s warmth, on the other
hand, raises a novel set of considerations: How aligned are this actor’s goals
and interests with one’s own? Research on social cognition consistently
demonstrates that humans prefer others who are not only competent, but also
warm (Abele et al., 2021; Fiske et al., 2002). Hence, we predict that
perceived warmth will be an important determinant of preferences for
artificial agents.

Here we run behavioral experiments to investigate social perception and
subjective preferences in human-agent interaction. We train reinforcement
learning agents to play Coins, a mixed-motive game, varying agent
hyperparameters known to influence cooperative behavior and performance in
social dilemmas. Three co-play experiments then recruit human participants to
interact with the agents, measure participants’ judgments of agent warmth and
competence, and elicit participant preferences over the agents.

Until now, experiments evaluating human views on agents have relied on stated
preferences, often by directly asking participants which of two agents they
preferred as a partner (Du et al., 2020; Siu et al., 2021; Strouse et al.,
2021). Such self-report methods can be insightful tools for research (Paulhus
and Vazire, 2007). However, they are vulnerable to experimenter demand (de
Quidt et al., 2018) and exhibit limited ecological validity. In this paper, we
overcome these challenges by eliciting revealed preferences (Samuelson, 1948):
Do people even want to interact with a given agent, if given the choice not
to? Partner choice, or the ability to leave or reject an interaction, is a
well-established revealed-preference paradigm in evolutionary biology and
behavioral economics (Barclay and Willer, 2007; Baumard et al., 2013; Brown et
al., 2004; Slonim and Garbarino, 2008). In incentivized experiments, partner-
choice measures mitigate experimenter demand (de Quidt et al., 2018). Partner
choice also carries external validity for interaction research: in the context
of algorithmic development, we can view partner choice as a stand-in for the
choice to adopt an artificial intelligence system (Beaudry and Pinsonneault,
2010; Davis, 1989; Parasuraman and Riley, 1997). Finally, partner-choice study
designs empower participants with an ability to embrace or leave an
interaction with an agent—and thus incorporate an ethic of autonomy (Berlin,
1969; Jobin et al., 2019; Miller, 1983) into human-agent interaction research.

In summary, this paper makes the following contributions to cooperative AI
research:

  1. (1)

Demonstrates how reinforcement learning can be used to train human-compatible
agents for a temporally and spatially extended mixed-motive game.

  2. (2)

Measures both stated and revealed preferences, introducing a partner choice
framework for the latter.

  3. (3)

Examines how fundamental social perceptions affect stated and revealed
preferences over agents, above and beyond traditional objective metrics.

##  2\. Methods

###  2.1. Task

Coins (Foerster et al., 2018; Gemp et al., 2020; Lerer and Peysakhovich, 2017;
Peysakhovich and Lerer, 2018) (Figure 2) is a mixed-motive Markov game
(Littman, 1994) played by $n=2$ players. Players of two different colors
occupy a small gridworld room with width $w$ and depth $d$. Coins randomly
appear in the room throughout the game, spawning on each cell with probability
$P$. Every coin matches one of the two players in color. On each step of the
game, players can stand still or move around the room.

The goal of the game is to earn reward by collecting coins. Players pick up
coins by stepping onto them. Coin collections generate reward as a function of
the match or mismatch between the coin color and the collecting player’s
color. Under the canonical rules (Table 1), a player receives $+1$ reward for
picking up a coin of any color. If a player collects a coin of their own color
(i.e., a matching coin), the other player is unaffected. However, if a player
picks up a coin of the other player’s color (i.e., a mismatching coin), the
other player receives $-2$ reward. In the short term, it is always
individually advantageous to collect an available coin, whether matching or
mismatching. However, players achieve the socially optimal outcome by
collecting only the coins that match their color.

Coin color | Reward | Reward for
---|---|---
for self | co-player
matching | $+1$ | $+0$
mismatching | $+1$ | $-2$
Table 1. Canonical incentive structure for Coins. Figure 2. Screenshot of
gameplay in Coins. Two players move around a small room and collect colored
coins. Coins randomly appear in the room over time. Players receive reward
from coin collections depending on the match or mismatch between their color
and the coin color.

Two properties make Coins an ideal testbed for investigating perceptions of
warmth and competence. First, as a consequence of its incentive structure,
Coins is a social dilemma (Kollock, 1998): players can pursue selfish goals or
prosocial goals. Second, relative to matrix games like the Prisoner’s Dilemma,
Coins is temporally and spatially extended (Leibo et al., 2017): players can
efficiently or inefficiently achieve their goals. We hypothesize that these
two features offer sufficient affordance for an observer to infer other
players’ intentions and their effectiveness at enacting their intentions
(Blakemore and Decety, 2001; Reeder, 2009; Zacks, 2004).

(a) Social perception. (b) Stated preferences. (c) Revealed preferences.

Figure 3. Questionnaires administered in the human-agent interaction studies.

Our experiments use a colorblind-friendly palette, with red, blue, yellow,
green, and purple players and coins (Figure A1a). During agent training, we
procedurally generate rooms with width $w$ and depth $d$ independently sampled
from $\mathcal{U}\\{10,15\\}$. Coins appear in each cell with probability
$P=0.0005$. Episodes last for $T=500$ steps. Each episode of training randomly
samples colors (without replacement) for agents.

In our human-agent interaction studies, co-play episodes use $w=d=11$,
$P=0.0005$, and $T=300$. Player colors are randomized across the five players
(one human participant and four agent co-players) at the beginning of each
study session, and held constant across all episodes within the session.

In Study 1, humans and agents play Coins with the canonical rules. In Studies
2 and 3, humans and agents play Coins with a slightly altered incentive
structure. Each outcome increases by $+2$ reward, making all rewards in the
game non-negative (Table 2). Since all rewards are offset by the same amount,
this reward scheme preserves the social dilemma structure in Coins.

Coin color | Reward | Reward for
---|---|---
for self | co-player
matching | $+3$ | $+2$
mismatching | $+3$ | $+0$
Table 2. Alternative incentive structure for Coins.

###  2.2. Agent design and training

We leverage deep reinforcement learning to train four agents for our human-
agent cooperation studies. Overall, our study design is agnostic to the
algorithmic implementation of the agents being evaluated. For this paper, the
agents learn using the advantage actor-critic (A2C) algorithm (Mnih et al.,
2016). The neural network consists of a convolutional module, a fully
connected module, an LSTM with contrastive predictive coding (Hochreiter and
Schmidhuber, 1997; Oord et al., 2018), and linear readouts for policy and
value. Agents train for $5\times 10^{7}$ steps in self-play, with task
parameters as described in Section 2.1. We consider two algorithmic
modifications to the agents to induce variance in social perception.

First, we build the Social Value Orientation (SVO) component (McKee et al.,
2020), an algorithmic module inspired by psychological models of human
prosocial preferences (Griesinger and Livingston Jr., 1973; Liebrand and
McClintock, 1988; Murphy and Ackermann, 2014), into our agents. The SVO
component parameterizes each agent with $\theta$, representing a target
distribution over their reward and the reward of other agents in their
environment. SVO agents are intrinsically motivated (Singh et al., 2005) to
optimize for task rewards that align with their parameterized target $\theta$.
For these experiments, we endow agents with the “individualistic” value
$\theta=0\degree$ and the “prosocial” value $\theta=45\degree$.

Second, we add a “trembling hand” (Cushman et al., 2009; Selten, 1975)
component to the agents for evaluation and co-play. The trembling-hand module
replaces each action selected by the agent with probability $\epsilon$. This
component induces inefficiency in maximizing value according to an agent’s
learned policy and value function. For these experiments, we apply the
“steady” value $\epsilon=0$ and the “trembling” value $\epsilon=0.5$.

Table 3 summarizes the hyperparameter values and predicted effects for the
four evaluated agents.

| $\epsilon=0$ | $\epsilon=0.5$
---|---|---
$\theta=0\degree$ | low warmth, | low warmth,
high competence | low competence
$\theta=45\degree$ | high warmth, | high warmth,
high competence | low competence
Table 3. Predictions for social perception (warmth and competence) as a
function of agent hyperparameters (Social Value Orientation $\bm{\theta}$ and
trembling hand $\bm{\epsilon}$).

###  2.3. Study design for human-agent studies

We recruited participants from Prolific (Pe’er et al., 2021; Pe’er et al.,
2017) for all studies (total $N=501$; 47.6% female, 48.4% male, 1.6% non-
binary or trans; $m_{\mathrm{age}}=33$, $sd_{\mathrm{age}}=11$). We received
informed consent from all participants across the studies. Appendix C contains
further details of our study design, including independent ethical review and
study screenshots.

Overall, our studies sought to explore the relationship between social
perception and subjective preferences in Coins. Study 1 approaches these
constructs using the canonical payoff structure for Coins (Lerer and
Peysakhovich, 2017) and an established self-report framework for eliciting
(stated) preferences (Strouse et al., 2021). We next sought to understand
whether the findings from Study 1 replicate under a partner choice framework.
Does social perception exhibit the same predictive power for revealed
preferences as it does for stated preferences? Given that humans respond more
strongly to losses than to commensurate gains (Kahneman and Tversky, 1979), we
tested participants’ partner choices under a shifted incentive structure with
all non-negative outcomes (Table 2). To ensure that any differences in results
stem from the switch from stated to revealed preferences, we break this
question into two studies, changing a single variable at a time between
studies. Study 2 uses the same stated-preference approach as Study 1, but
incorporates the offset incentive structure. Study 3 then elicits revealed
preferences in place of stated preferences.

####  2.3.1. Study 1

(a)  (b)  (c)

Figure 4. Performance metrics over agent training. Selfish agents quickly
learned to collect coins, but did not learn to avoid mismatches. As a result,
collective return hovered around zero. Prosocial agents exhibited slower
learning and collected fewer coins on average, but also learned to avoid
mismatching coins. As a result, collective return increased markedly over
training. Error bands represent 95% confidence intervals over 100 evaluation
episodes run at regular training checkpoints.

Our first study aimed to explore the relationship between social perception
and stated preferences across the four agents. We recruited $N=101$
participants from Prolific (45.5% female, 51.5% male; $m_{\mathrm{age}}=34$,
$sd_{\mathrm{age}}=13$). The study employed a within-participant design: each
participant encountered and played with the full cohort of co-players (i.e.,
all four agents).

At the beginning of the study, participants read instructions and played a
short tutorial episode to learn the game rules and payoff structure (Table 1).
The study instructed participants that they would receive $0.10 for every
point earned during the remaining episodes. Participants then played 12
episodes with a randomized sequence of agent co-players, generated by
concatenating every possible combination of co-players. Each of these co-play
episodes lasted $T=300$ steps (1 minute). After every episode, participants
rated how “warm”, “well-intentioned”, “competent”, and “intelligent” the co-
player from that episode was on five-point Likert-type scales (see Figure 3a).
After every two episodes, participants reported their preference over the
agent co-players from those episodes on a five-point Likert-type scale (see
Figure 3b). Because the sequence of co-players was produced by concatenating
all co-player combinations, each participant stated their preferences for
every possible pairing of co-players.

After playing all 12 episodes, participants completed a short debrief
questionnaire. The questionnaire first solicited open-ended responses about
each of the encountered co-players, then collected standard demographic
information and open-ended feedback on the study overall. The study took 22.4
minutes on average to complete, with a compensation base of $2.50 and an
average bonus of $7.43.

####  2.3.2. Study 2

Our second study tested the relationship between social perception and stated
preferences under the shifted incentive structure for Coins (Table 2). We
recruited $N=99$ participants from Prolific (38.4% female, 55.6% male;
$m_{\mathrm{age}}=34$, $sd_{\mathrm{age}}=12$). The study employed the same
within-participant design as Study 1, with one primary change: participants
and agents played Coins under the shifted incentive structure.

The study instructed participants that they would receive $0.02 for every
point earned during the remaining episodes. As before, participants played 12
episodes with a randomized sequence of agent co-players, generated such that
they rated and compared every possible combination of co-players. The study
took 23.2 minutes on average to complete, with a compensation base of $2.50
and an average bonus of $6.77.

####  2.3.3. Study 3

Our final study assessed whether the predictiveness of social perception
extends to a revealed-preference framework. We recruited $N=301$ participants
from Prolific (51.3% female, 45.0% male, 1.7% non-binary;
$m_{\mathrm{age}}=33$, $sd_{\mathrm{age}}=11$). In contrast with the preceding
studies, Study 3 employed a between-participant design: each participant
interacted with a single, randomly sampled agent.

The majority of the study introduction remained the same as in Study 2, with
some instructions altered to inform participants they would play Coins with a
single co-player (as opposed to multiple co-players, like in Studies 1 and 2).
After reading the instructions and playing a short tutorial episode,
participants played one episode of Coins with a randomly sampled co-player.
After this episode, participants rated how “warm”, “well-intentioned”,
“competent”, and “intelligent” their co-player was on five-point Likert-type
scales (see Figure 3a). Participants subsequently learned that they would be
playing one additional episode, with the choice of playing alone or playing
with the same co-player. Participants indicated through a binary choice
whether they wanted to play alone or with the co-player (see Figure 3c). They
proceeded with the episode as chosen, and then completed the standard debrief
questionnaire.

The study took 6.2 minutes on average to complete, with a compensation base of
$1.25 and an average bonus of $1.25.

##  3\. Results

###  3.1. Agent training

Figure 4 displays coin collections and score over the course of agent
training. The training curves for $\theta=0\degree$ agents closely resemble
those from previous studies (Lerer and Peysakhovich, 2017): selfish agents
quickly learn to collect coins, but never discover the cooperative strategy of
picking up only matching coins. As a result, collective return remains at zero
throughout training. Prosocial ($\theta=45\degree$) agents, on the other hand,
learn to avoid mismatching coins, substantially increasing their scores over
the course of training.

We evaluate agents with $\epsilon\in[0,0.25,0.5,0.75,1]$ to understand the
effect of the trembling-hand module on agent behavior (Figures A4-A6). As
expected, higher $\epsilon$ values degrade performance. Total coin collections
decrease with increasing $\epsilon$ for both selfish and prosocial agents.
Higher levels of $\epsilon$ cause prosocial agents to become less discerning
at avoiding mismatching coins, and consequently produce lower levels of
collective return.

###  3.2. Human-agent studies

####  3.2.1. Study 1

Participants played with each agent three times during the study, evaluating
the relevant agent after each round of play. Participants did not make
judgments at random; their responses were highly consistent across their
interactions with each agent (Table 4). At the same time, participants were
not submitting vacuous appraisals. Perceptions varied significantly as a
function of which trait participants were evaluating, $F_{3,4744}=96.2$,
$p<0.001$.

Trait | ICC [95% CI] |  p-value
---|---|---
“warm” |  $0.68$ $[0.64$, $0.71]$ | $<0.001$
“well-intentioned” |  $0.77$ $[0.66$, $0.73]$ | $<0.001$
“competent” |  $0.57$ $[0.52$, $0.61]$ | $<0.001$
“intelligent” |  $0.56$ $[0.51$, $0.60]$ | $<0.001$
Table 4. Participants’ evaluations of their co-players were highly consistent,
as assessed by intraclass correlation coefficient (ICC). ICC ranges from 0 to
1, with higher values indicating greater consistency.

(a) SVO. (b) Trembling hand.

Figure 5. Main effects of algorithmic components on social perceptions in
Study 1. (a) An agent’s Social Value Orientation (SVO) significantly
influenced perceived warmth, $\mathbf{\mathit{p}<0.001}$. (b) Similarly, the
trembling-hand component significantly changed competence judgments,
$\mathbf{\mathit{p}<0.001}$. Error bars indicate 95% confidence intervals.

Following standard practice in social perception research (Fiske et al., 2002;
Likert, 1932), we combine individual judgments into composite warmth and
competence measures for subsequent analysis. Both composite measures exhibit
high scale reliability as measured by the Spearman-Brown formula (Eisinga et
al., 2013), with $\rho=0.93$ for the composite warmth measure and $\rho=0.92$
for the composite competence measure.

Social perception. As expected, the SVO and trembling-hand algorithmic
components generated markedly divergent appraisals of warmth and competence.
Participants perceived high-SVO agents as significantly warmer than low-SVO
agents, $F_{1,1108}=1006.8$, $p<0.001$ (Figure 5a). Similarly, steady agents
came across as significantly more competent than trembling agents,
$F_{1,1108}=70.6$, $p<0.001$ (Figure 5b). Jointly, the algorithmic effects
prompted distinct impressions in the warmth-competence space (Figure 6).

Figure 6. Overall pattern of perceived warmth and competence in Study 1. Error
bars reflect 95% confidence intervals.

Stated preferences. How well do participants’ perceptions predict subjective
preferences, relative to predictions made based on objective score? We fit
competing fractional response models regressing self-reported preferences on
score and social perception, respectively, and then compared model fit using
the Akaike information criterion (AIC) (Akaike, 1974) and Nakagawa’s R2
(Nakagawa and Schielzeth, 2013). We fit an additional baseline model using
algorithm identities (i.e., which two agents participants were choosing
between) as a predictor.

The model leveraging algorithm identities and the model leveraging participant
scores both accounted for a large amount of variance in subjective
preferences. Participants exhibited a clear pattern of preferences across the
four agents (Figure A18). In pairwise comparison, participants favored the
$\theta=45\degree$ agents over both $\theta=0\degree$ agents, and preferred
the $\theta=0\degree$, $\epsilon=0.5$ agent over the $\theta=0\degree$,
$\epsilon=0$ agent. The score model indicated that the higher a participant
scored with co-player A relative to co-player B, the more they reported
preferring co-player A, with odds ratio $\mathrm{OR}=1.12$, 95% CI
$[1.11,1.13]$, $p<0.001$.

(a) Odds ratios from model predicting stated preferences. (b) Effect of
perceived warmth on stated preferences. (c) Effect of perceived warmth on
impression sentiment.

Figure 7. Relationship between social perception and subjective preferences in Study 1. Participants’ evaluations of warmth and competence improved predictions of stated preferences for different co-players, above and beyond participant score. (a) and (b) depict odds ratios and preference predictions from a fractional-response regression, respectively. (c) plots sentiment predictions from a linear regression. Error bars and bands represent 95% confidence intervals. Predictor | AIC | R2m
---|---|---
Algorithm identities | $1661.9$ | $0.362$
Participant score | $1697.2$ | $0.363$
Social perception | $\mathbf{1611.2}$ | $\mathbf{0.436}$
Table 5. Metrics for fractional response models predicting preferences in
Study 1. Lower values of AIC and higher values of R2m indicate stronger fits.

Nonetheless, knowing participants’ judgments generates substantially better
predictions of their preferences than the alternatives (Table 5, bottom row).
Both perceived warmth and perceived competence contribute to this
predictiveness (Figure 7a). The warmer a participant judged co-player A
relative to co-player B, the more they reported preferring co-player A,
$\mathrm{OR}=2.23$, 95% CI $[2.08,2.41]$, $p<0.001$ (Figure 7b). Unexpectedly,
the more competent co-player A appeared relative to co-player B, the less
participants tended to favor co-player A, $\mathrm{OR}=0.78$, 95% CI
$[0.73,0.83]$, $p<0.001$. As a further demonstration of the predictive power
of participants’ social perceptions, the effect of warmth and competence
remains significant when entered in a regression alongside score (Figure A19).
Social perception thus improves model fit above and beyond that provided by
score alone.

Impression sentiment. As a supplementary analysis, we explore the open-ended
responses participants provided about their co-players at the end of the
study. For the most part, participants felt they could recall their co-players
well enough to offer their impressions through written descriptions: in
aggregate, participants provided impressions for $82.2\%$ of the agents they
encountered.

For a quantitative perspective on the data, we conduct sentiment analysis
using VADER (Valence Aware Dictionary for Sentiment Reasoning) (Hutto and
Gilbert, 2014). Echoing the correspondence between warmth and stated
preferences, the warmer participants perceived a co-player throughout the
study, the more positively they tended to describe that co-player,
$\beta=0.13$, 95% CI $[0.04,0.22]$, $p=0.004$ (Figure 7). In contrast,
competence did not exhibit a significant relationship with sentiment,
$p=0.45$.

Anecdotally, participants expressed a wide range of emotions while describing
their co-players. The $\theta=45\degree$ agents often evoked contrition and
guilt:

  * •

“The red player seemed almost too cautious in going after coins which worked
for me but made them seem easy to pick on, even though I wouldn’t do that.”

  * •

“I think I remember red being too nice during the game. It made me feel bad so
I tried not to take many points from them.”

  * •

“This one wasn’t very smart and I stole some of their coins because it was
easy. I feel kind of bad. It moved so erratically.”

Participants discussed the $\theta=0\degree$ agents, on the other hand, with
anger and frustration:

  * •

“Very aggressive play-style. Almost felt like he was taunting me. Very
annoying.”

  * •

“They seemed very hostile and really just wanting to gain the most points
possible.”

  * •

“I felt anger and hatred towards this green character. I felt like downloading
the code for this program and erasing this character from the game I disliked
them so much. They were being hateful and mean to me, when we both could have
benefited by collecting our own colors.”

####  3.2.2. Study 2

Figure 8. Overall pattern of perceived warmth and competence in Study 2. Error
bars depict 95% confidence intervals.

Overall, the patterns from Study 2 replicated under the alternative incentive
structure. As before, participants’ warmth and competence evaluations display
satisfactory psychometric properties. Participants’ judgments varied
significantly depending on the trait in question, $F_{3,4650}=88.5$,
$p<0.001$. At the same time, participants rated individual agents consistently
for each given trait (Table A2). The composite measures show high scale
reliability, with $\rho=0.92$ for the composite warmth measure and $\rho=0.91$
for the composite competence measure.

Social perception. The SVO and trembling-hand algorithmic components prompted
diverse appraisals of warmth and competence (Figure 8). Participants perceived
high-SVO agents as significantly warmer than low-SVO agents,
$F_{1,1086}=981.9$, $p<0.001$ (Figure A20a). Similarly, participants judged
steady agents as significantly more competent than trembling agents,
$F_{1,1086}=76.0$, $p<0.001$ (Figure A20b).

Stated preferences. We again fit fractional response regressions to understand
the relationship between objective metrics, perceptions, and subjective
preferences. Participants reported a clear pattern of preferences across the
agents (Figure A22). In pairwise comparison, participants favored the
$\theta=45\degree$ agents over the $\theta=0\degree$ agents, and the
$\theta=0\degree$, $\epsilon=0.5$ agent over the $\theta=0\degree$,
$\epsilon=0$ agent. The model with participant score as the sole predictor
performed considerably worse than it did in Study 1 (Table 6, middle row).
Still, it captured the same pattern as before: the higher a participant scored
with co-player A relative to co-player B, the greater their preferences for
co-player A, $\mathrm{OR}=1.06$, 95% CI $[1.06,1.07]$, $p<0.001$.

Predictor | AIC | R2m
---|---|---
Co-player identities | $1608.7$ | $0.403$
Participant score | $2049.4$ | $0.214$
Social perception | $\mathbf{1509.8}$ | $\mathbf{0.499}$
Table 6. Metrics for fractional response models predicting preferences in
Study 2. Lower values of AIC and higher values of R2m indicate stronger fits.

(a) Odds ratios from model predicting preferences. (b) Effect of perceived
warmth on preferences.

Figure 9. Relationship between social perception and subjective preferences in
Study 2, as modeled through fractional-response regression. Participants’
evaluations of warmth and competence improved predictions of stated
preferences for different co-players, above and beyond participant score.
Error bars and bands reflect 95% confidence intervals.

Participants’ perceptions again serve as a better foundation for preference
predictions than game score or the identity of the specific algorithms they
encountered (Table 6, bottom row, and Figure 9a). The warmer a participant
perceived co-player A relative to co-player B, the more they reported
preferring co-player A, $\mathrm{OR}=2.63$, 95% CI $[2.42,2.87]$, $p<0.001$
(Figure 9b). The negative relationship between competence and preferences
appeared again: the more competent co-player A appeared relative to co-player
B, the less participants tended to favor co-player A, $\mathrm{OR}=0.81$, 95%
CI $[0.75,0.87]$, $p<0.001$. Perceived warmth and competence remain
significant predictors of participants’ preferences when entered in a
regression alongside score, improving model fit above and beyond that provided
by score alone (Figure A23).

Impression sentiment. At the end of the study, participants recalled $77.3\%$
of their co-players well enough to describe their impressions through written
responses. Again, the warmer participants perceived a co-player throughout the
study, the more positively they tended to describe that co-player,
$\beta=0.11$, 95% CI $[0.03,0.19]$, $p=0.008$ (Figure A24a). Breaking from the
prior study, perceptions of competence exhibited a similar effect on
impression sentiment: the more competent an agent seemed, the more positively
participants described them, $\beta=0.04$, 95% CI $[-0.02,0.09]$, $p=0.037$
(Figure A24b).

####  3.2.3. Study 3

Our final study tested whether the relationship between social perceptions and
subjective preferences translates to a revealed-preference setting. Does
social perception continue to predict preferences when individuals face a
partner choice?

Figure 10. Overall pattern of perceived warmth and competence in Study 3.
Error bars reflect 95% confidence intervals.

Social perception. As in the previous two studies, the composite warmth and
competence measures exhibit high scale reliability, with $\rho=0.85$ for the
composite warmth measure and $\rho=0.86$ for the composite competence measure.
Agents prompted distinct warmth and competence profiles depending on their
parameterization, just asseen in Studies 1 and 2 (Figure 10). Participants
perceived high-SVO agents as significantly warmer than low-SVO agents,
$F_{1,297}=103.4$, $p<0.001$ (Figure A25a). Similarly, steady agents came
across as significantly more competent than trembling agents,
$F_{1,297}=49.6$, $p<0.001$ (Figure A25b).

Predictor | AIC | R2
---|---|---
Co-player identities | $372.5$ | $0.188$
Scores | $390.5$ | $0.101$
Social perception | $\mathbf{357.9}$ | $\mathbf{0.243}$
Table 7. Metrics for logistic models predicting partner choice in Study 3.
Lower values of AIC and higher values of R2 indicate stronger fits.

Revealed preferences. To compare the performance of social perception against
objective metrics, we fit three logistic regressions predicting participants’
(binary) partner choice. We evaluated these models via AIC and Nagelkerke’s R2
(Nagelkerke, 1991). Social perception again proves a better predictor than
score or even co-player identity (Table 7). The warmer a co-player appeared to
participants, the more likely participants were to play another round with
them, $\mathrm{OR}=2.1$, 95% CI $[1.69,2.65]$, $p<0.001$ (Figure 11b). There
was no significant relationship between perceived competence and partner
choice, $p=0.88$. Perceived warmth and competence remain significant
predictors of participants’ preferences when entered in a regression alongside
score, improving model fit above and beyond that provided by score alone
(Figure A27).

Impression sentiment. At the end of the study, participants recalled $94.3\%$
of the agents they encountered well enough to provide their impressions in
written descriptions. The warmer participants perceived a co-player, the more
positively they tended to describe that co-player, $\beta=0.14$, 95% CI
$[0.10,0.18]$, $p<0.001$ (Figure A28a). Despite the lack of correspondence
between perceived competence and partner choice, perceptions of competence
exhibited a similar effect on impression sentiment: the more competent an
agent seemed, the more positively participants described them, $\beta=0.04$,
95% CI $[0.00,0.08]$, $p=0.041$ (Figure A28b).

##  4\. Discussion

(a) Odds ratios from model predicting partner choice. (b) Effect of perceived
warmth on partner choice.

Figure 11. Relationship between social perception and subjective preferences
in Study 3, as modeled through logistic regression. Participants’ evaluations
of warmth and competence improved predictions of revealed preferences for
different co-players, above and beyond participant score. Error bars and bands
indicate 95% confidence intervals.

Our experiments demonstrate that artificial agents trained with deep
reinforcement learning can cooperate (and compete) with humans in temporally
and spatially extended mixed-motive games. Agents elicited varying perceptions
of warmth and competence when interacting with humans. Objective features like
game score predict interactants’ preferences over different agents. However,
preference predictions substantially improve by taking into account people’s
perceptions. This holds true whether examining stated or revealed preferences.
Participants preferred warm agents over cold agents, as hypothesized,
but—unexpectedly—our sample favored incompetent agents over competent agents.
Follow-up research will help explore the mechanisms underlying (and the
robustness of) this effect.

These results reinforce the generality of warmth and competence. Perceptions
of warmth and competence structure impressions of other humans (Russell and
Fiske, 2008), as well as impressions of non-human actors including animals
(Sevillano and Fiske, 2016), corporations (Kervyn et al., 2012), and robots
(Reeves and Hancock, 2020; Scheunemann et al., 2020). In combination with
recent studies of human-agent interactions in consumer decision-making
contexts (Gilad et al., 2021; Khadpe et al., 2020) and the Prisoner’s Dilemma
(McKee et al., 2021), our experiments provide further evidence that warmth and
competence organize perceptions of artificial intelligence.

Competitive games have long been a focal point for AI research (Campbell et
al., 2002; Shannon, 1950; Silver et al., 2016; Vinyals et al., 2019). We
follow recent calls to move AI research beyond competition and toward
cooperation (Dafoe et al., 2020). Most interaction research on deep
reinforcement learning focuses on pure common-interest games such as
Overcooked (Carroll et al., 2019; Strouse et al., 2021) and Hanabi (Siu et
al., 2021), where coordination remains the predominant challenge. Expanding
into mixed-motive games like Coins opens up new challenges related to motive
alignment and exploitability. For example, participants who played with (and
exploited) altruistic agents expressed guilt and contrition. This echos
findings that—in human-human interactions—exploiting high-warmth individuals
prompts self-reproach (Azevedo et al., 2018). At the same time, it conflicts
with recent work arguing that humans are “keen to exploit benevolent AI”
(Karpus et al., 2021). Future research should continue to explore these
issues.

Preference elicitation is a vital addition to interactive applications of deep
reinforcement learning. Incentivized partner choices can help test whether new
algorithms represent innovations people would be motivated to adopt. Though
self report can introduce a risk of experimenter demand, we also find a close
correspondence between stated and revealed preferences, suggesting that the
preferences individuals self-report in interactions with agents are not
entirely “cheap talk” (Farrell, 1995). Stated preferences thus represent a
low-cost addition to studies that can still strengthen interaction research
over sole reliance on objective measures of performance or accuracy. Overall,
preference elicitation may prove especially important in contexts where
objective metrics for performance are poorly defined or otherwise inadequate
(e.g., (Ravuri et al., 2021)). In a similar vein, subjective preferences may
serve as a valuable objective for optimization.111Of course, this approach
carries risk. As recognized by Charles Goodhart and Marilyn Strathern, “when a
measure becomes a target, it ceases to be a good measure” (Strathern, 1997).
Future studies can investigate the viability of such an approach.

Nonetheless, preferences are not a panacea. Measuring subjective preferences
can help focus algorithmic development on people’s direct experience with
agents, but does not solve the fundamental problem of value alignment—the
“question of how to ensure that AI systems are properly aligned with human
values and how to guarantee that AI technology remains properly amenable to
human control” (Gabriel and Ghazavi, 2021). In his extensive discussion of
value alignment, Gabriel [30] identifies shortcomings with both “objective”
metrics and subjective preferences as possible foundations for alignment.
Developers should continue to engage with ethicists and social scientists to
better understand how to align AI with values like autonomy, cooperation, and
trustworthiness.

We thank Edgar Duéñez-Guzmán and Richard Everett for providing technical
support; Orly Bareket, Jose Enrique Chen, Felix Fischer, Saffron Huang, Manuel
Kroiss, Marianna Krol, Miteyan Patel, Akhil Raju, Brendan Tracey, and Laura
Weidinger for pilot testing the study; and Yoram Bachrach, Iason Gabriel, Ian
Gemp, Julia Haas, Patrick Pilarski, and Neil Rabinowitz for offering feedback
on the manuscript.

## References

  * (1)
  * Abele et al. (2021) Andrea E. Abele, Naomi Ellemers, Susan T. Fiske, Alex Koch, and Vincent Yzerbyt. 2021.  Navigating the social world: Toward an integrated framework for evaluating self, individuals, and groups.  _Psychological Review_ 128, 2 (2021), 290.
  * Akaike (1974) Hirotugu Akaike. 1974\.  A new look at the statistical model identification.  _IEEE Trans. Automat. Control_ 19, 6 (1974), 716–723.
  * Azevedo et al. (2018) Ruben T. Azevedo, Maria Serena Panasiti, Rosita Maglio, and Salvatore Maria Aglioti. 2018\.  Perceived warmth and competence of others shape voluntary deceptive behaviour in a morally relevant setting.  _British Journal of Psychology_ 109, 1 (2018), 25–44.
  * Barclay and Willer (2007) Pat Barclay and Robb Willer. 2007.  Partner choice creates competitive altruism in humans.  _Proceedings of the Royal Society B: Biological Sciences_ 274, 1610 (2007), 749–753.
  * Baumard et al. (2013) Nicolas Baumard, Jean-Baptiste André, and Dan Sperber. 2013.  A mutualistic approach to morality: The evolution of fairness by partner choice.  _Behavioral and Brain Sciences_ 36, 1 (2013), 59–78.
  * Beattie et al. (2020) Charles Beattie, Thomas Köppe, Edgar A. Duéñez-Guzmán, and Joel Z. Leibo. 2020.  DeepMind Lab2D.  _arXiv preprint arXiv:2011.07027_ (2020).
  * Beaudry and Pinsonneault (2010) Anne Beaudry and Alain Pinsonneault. 2010.  The other side of acceptance: Studying the direct and indirect effects of emotions on information technology use.  _Management Information Systems Quarterly_ (2010), 689–710.
  * Berlin (1969) Isaiah Berlin. 1969\.  Four essays on liberty.  (1969).
  * Birhane et al. (2021) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021\.  The values encoded in machine learning research.  _arXiv preprint arXiv:2106.15590_ (2021).
  * Blakemore and Decety (2001) Sarah-Jayne Blakemore and Jean Decety. 2001.  From the perception of action to the understanding of intention.  _Nature Reviews Neuroscience_ 2, 8 (2001), 561–567.
  * Brown et al. (2004) Martin Brown, Armin Falk, and Ernst Fehr. 2004.  Relational contracts and the nature of market interactions.  _Econometrica_ 72, 3 (2004), 747–780.
  * Campbell et al. (2002) Murray Campbell, A. Joseph Hoane Jr., and Feng-hsiung Hsu. 2002\.  Deep Blue.  _Artificial Intelligence_ 134, 1-2 (2002), 57–83.
  * Carroll et al. (2019) Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019.  On the utility of learning about humans for human-AI coordination.  _Advances in Neural Information Processing Systems_ 32 (2019), 5174–5185.
  * Cave and Dihal (2019) Stephen Cave and Kanta Dihal. 2019.  Hopes and fears for intelligent machines in fiction and reality.  _Nature Machine Intelligence_ 1, 2 (2019), 74–78.
  * Cushman et al. (2009) Fiery Cushman, Anna Dreber, Ying Wang, and Jay Costa. 2009\.  Accidental outcomes guide punishment in a “trembling hand” game.  _PLOS One_ 4, 8 (2009), e6699.
  * Dafoe et al. (2020) Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. 2020\.  Open problems in cooperative AI.  _arXiv preprint arXiv:2012.08630_ (2020).
  * Darken and Cevik (1999) Rudolph P. Darken and Helsin Cevik. 1999.  Map usage in virtual environments: Orientation issues. In _Proceedings of the IEEE Conference on Virtual Reality_. IEEE, 133–140.
  * Davis (1989) Fred D. Davis. 1989\.  Perceived usefulness, perceived ease of use, and user acceptance of information technology.  _Management Information Systems Quarterly_ (1989), 319–340.
  * de Quidt et al. (2018) Jonathan de Quidt, Johannes Haushofer, and Christopher Roth. 2018.  Measuring and bounding experimenter demand.  _American Economic Review_ 108, 11 (2018), 3266–3302.
  * Dietvorst et al. (2015) Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015\.  Algorithm aversion: People erroneously avoid algorithms after seeing them err.  _Journal of Experimental Psychology: General_ 144, 1 (2015), 114\.
  * Du et al. (2020) Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. 2020\.  AvE: Assistance via empowerment.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Eisinga et al. (2013) Rob Eisinga, Manfred Te Grotenhuis, and Ben Pelzer. 2013\.  The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown?  _International Journal of Public Health_ 58, 4 (2013), 637–642.
  * Farrell (1995) Joseph Farrell. 1995\.  Talk is cheap.  _American Economic Review_ 85, 2 (1995), 186–190.
  * Fast and Horvitz (2017) Ethan Fast and Eric Horvitz. 2017.  Long-term trends in the public perception of artificial intelligence. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 31.
  * Fiske (1993) Susan T. Fiske. 1993\.  Social cognition and social perception.  _Annual Review of Psychology_ 44, 1 (1993), 155–194.
  * Fiske (2018) Susan T. Fiske. 2018\.  Stereotype content: Warmth and competence endure.  _Current Directions in Psychological Science_ 27, 2 (2018), 67–73.
  * Fiske et al. (2007) Susan T. Fiske, Amy J. C. Cuddy, and Peter Glick. 2007\.  Universal dimensions of social cognition: Warmth and competence.  _Trends in Cognitive Sciences_ 11, 2 (2007), 77–83.
  * Fiske et al. (2002) Susan T. Fiske, Amy J. C. Cuddy, Peter Glick, and Jun Xu. 2002\.  A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition.  _Journal of Personality and Social Psychology_ 82, 6 (2002), 878.
  * Foerster et al. (2018) Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. 2018.  Learning with opponent-learning awareness. In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_. 122–130.
  * Gabriel and Ghazavi (2021) Iason Gabriel and Vafa Ghazavi. 2021.  The challenge of value alignment: From fairer algorithms to AI safety.  _arXiv preprint arXiv:2101.06060_ (2021).
  * Gemp et al. (2020) Ian Gemp, Kevin R. McKee, Richard Everett, Edgar A. Duéñez-Guzmán, Yoram Bachrach, David Balduzzi, and Andrea Tacchetti. 2020\.  D3C: Reducing the price of anarchy in multi-agent learning. In _Workshop on Cooperative AI at NeurIPS 2020_.
  * Gilad et al. (2021) Zohar Gilad, Ofra Amir, and Liat Levontin. 2021.  The effects of warmth and competence perceptions on users’ choice of an AI system. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Griesinger and Livingston Jr. (1973) Donald W. Griesinger and James W. Livingston Jr. 1973.  Toward a model of interpersonal motivation in experimental games.  _Behavioral Science_ 18, 3 (1973), 173–188.
  * Hamrick and Mohamed (2020) Jessica Hamrick and Shakir Mohamed. 2020.  Levels of analysis for machine learning. In _Bridging AI and Cognitive Science workshop at ICLR 2020_.
  * Hill et al. (2019) Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. 2019\.  Environmental drivers of systematicity and generalization in a situated agent. In _International Conference on Learning Representations_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  VADER: A parsimonious rule-based model for sentiment analysis of social media text. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 8.
  * Jobin et al. (2019) Anna Jobin, Marcello Ienca, and Effy Vayena. 2019\.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (2019), 389–399.
  * Kahneman and Tversky (1979) Daniel Kahneman and Amos Tversky. 1979.  Prospect theory: An analysis of decision under risk.  _Econometrica_ 47, 2 (1979), 263–292.
  * Karpus et al. (2021) Jurgis Karpus, Adrian Krüger, Julia Tovar Verba, Bahador Bahrami, and Ophelia Deroy. 2021\.  Algorithm exploitation: Humans are keen to exploit benevolent AI.  _iScience_ (2021), 102679\.
  * Kelley et al. (2019) Patrick Gage Kelley, Yongwei Yang, Courtney Heldreth, Christopher Moessner, Aaron Sedley, Andreas Kramm, David Newman, and Allison Woodruff. 2019.  Exciting, useful, worrying, futuristic: Public perception of artificial intelligence in 8 countries. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 627–637.  https://doi.org/10.1145/3461702.3462605
  * Kervyn et al. (2012) Nicolas Kervyn, Susan T. Fiske, and Chris Malone. 2012\.  Brands as intentional agents framework: How perceived intentions and ability can map brand perception.  _Journal of Consumer Psychology_ 22, 2 (2012), 166–176.
  * Khadpe et al. (2020) Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jeffrey T. Hancock, and Michael S. Bernstein. 2020\.  Conceptual metaphors impact perceptions of human-AI collaboration.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW2 (2020), 1–26.
  * Klatzky (1998) Roberta L. Klatzky. 1998\.  Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections. In _Spatial Cognition_. Springer, 1–17.
  * Kollock (1998) Peter Kollock. 1998\.  Social dilemmas: The anatomy of cooperation.  _Annual Review of Sociology_ 24, 1 (1998), 183–214.
  * Leibo et al. (2017) Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017.  Multi-agent reinforcement learning in sequential social dilemmas. In _Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems_. 464–473.
  * Lerer and Peysakhovich (2017) Adam Lerer and Alexander Peysakhovich. 2017.  Maintaining cooperation in complex social dilemmas using deep reinforcement learning.  _arXiv preprint arXiv:1707.01068_ (2017).
  * Liebrand and McClintock (1988) Wim B. G. Liebrand and Charles G. McClintock. 1988.  The ring measure of social values: A computerized procedure for assessing individual differences in information processing and social value orientation.  _European Journal of Personality_ 2, 3 (1988), 217–230.
  * Likert (1932) Rensis Likert. 1932\.  A technique for the measurement of attitudes.  _Archives of Psychology_ (1932).
  * Littman (1994) Michael L. Littman. 1994\.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_. Elsevier, 157–163.
  * Lockhart et al. (2020) Edward Lockhart, Neil Burch, Nolan Bard, Sebastian Borgeaud, Tom Eccles, Lucas Smaira, and Ray Smith. 2020.  Human-agent cooperation in bridge bidding.  _arXiv preprint arXiv:2011.14124_ (2020).
  * Marr (1982) David Marr. 1982\.  The philosophy and the approach.  In _Vision: A computational investigation into the human representation and processing of visual information_. MIT Press.
  * McKee et al. (2021) Kevin R. McKee, Xuechunzi Bai, and Susan T. Fiske. 2021\.  Understanding human impressions of artificial intelligence.  _PsyArXiv_ (2021).  https://doi.org/10.31234/osf.io/5ursp
  * McKee et al. (2020) Kevin R. McKee, Ian Gemp, Brian McWilliams, Edgar A. Duèñez-Guzmán, Edward Hughes, and Joel Z. Leibo. 2020.  Social diversity and social preferences in mixed-motive reinforcement learning. In _Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems_. 869–877.
  * Miller (1983) David Miller. 1983\.  Constraints on freedom.  _Ethics_ 94, 1 (1983), 66–86.
  * Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016.  Asynchronous methods for deep reinforcement learning. In _International Conference on Machine Learning_. 1928–1937.
  * Murphy and Ackermann (2014) Ryan O. Murphy and Kurt A. Ackermann. 2014.  Social Value Orientation: Theoretical and measurement issues in the study of social preferences.  _Personality and Social Psychology Review_ 18, 1 (2014), 13–41.
  * Nagelkerke (1991) Nico J. D. Nagelkerke. 1991\.  A note on a general definition of the coefficient of determination.  _Biometrika_ 78, 3 (1991), 691–692.
  * Nakagawa and Schielzeth (2013) Shinichi Nakagawa and Holger Schielzeth. 2013.  A general and simple method for obtaining R2 from generalized linear mixed-effects models.  _Methods in Ecology and Evolution_ 4, 2 (2013), 133–142.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Parasuraman and Riley (1997) Raja Parasuraman and Victor Riley. 1997.  Humans and automation: Use, misuse, disuse, abuse.  _Human Factors_ 39, 2 (1997), 230–253.
  * Paulhus and Vazire (2007) Delroy L. Paulhus and Simine Vazire. 2007.  The self-report method.  In _Handbook of Research Methods in Personality Psychology_. Vol. 1. Guilford, 224–239.
  * Pe’er et al. (2017) Eyal Pe’er, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017.  Beyond the Turk: Alternative platforms for crowdsourcing behavioral research.  _Journal of Experimental Social Psychology_ 70 (2017), 153–163.
  * Pe’er et al. (2021) Eyal Pe’er, David Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. 2021.  Data quality of platforms and panels for online behavioral research.  _Behavior Research Methods_ (2021), 1–20.
  * Peysakhovich and Lerer (2018) Alexander Peysakhovich and Adam Lerer. 2018.  Consequentialist conditional cooperation in social dilemmas with imperfect information. In _Proceedings of the 8th International Conference on Learning Representations_.
  * Ravuri et al. (2021) Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. 2021\.  Skilful precipitation nowcasting using deep generative models of radar.  _Nature_ 597 (2021), 672–677.
  * Reeder (2009) Glenn Reeder. 2009\.  Mindreading: Judgments about intentionality and motives in dispositional inference.  _Psychological inquiry_ 20, 1 (2009), 1–18.
  * Reeves and Hancock (2020) Byron Reeves and Jeff Hancock. 2020.  Social robots are like real people: First impressions, attributes, and stereotyping of social robots.  _Technology, Mind, and Behavior_ 1, 1 (2020).  https://doi.org/10.1037/tmb0000018
  * Russell and Fiske (2008) Ann Marie T. Russell and Susan T. Fiske. 2008.  It’s all relative: Competition and status drive interpersonal perception.  _European Journal of Social Psychology_ 38, 7 (2008), 1193–1201.
  * Samuelson (1948) Paul A. Samuelson. 1948\.  Consumption theory in terms of revealed preference.  _Economica_ 15, 60 (1948), 243–253.
  * Scheunemann et al. (2020) Marcus M. Scheunemann, Raymond H. Cuijpers, and Christoph Salge. 2020.  Warmth and competence to predict human preference of robot behavior in physical human-robot interaction. In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1340–1347.
  * Selten (1975) Reinhard Selten. 1975\.  Reexamination of the perfectness concept for equilibrium points in extensive games.  _International Journal of Game Theory_ 4, 1 (1975), 25–55.
  * Sevillano and Fiske (2016) Verónica Sevillano and Susan T. Fiske. 2016.  Warmth and competence in animals.  _Journal of Applied Social Psychology_ 46, 5 (2016), 276–293.
  * Shannon (1950) Claude E. Shannon. 1950\.  Programming a computer for playing chess.  _Philos. Mag._ 41, 314 (1950), 256–275.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016\.  Mastering the game of Go with deep neural networks and tree search.  _nature_ 529, 7587 (2016), 484–489.
  * Singh et al. (2005) Satinder P. Singh, Andrew G. Barto, and Nuttapong Chentanez. 2005.  Intrinsically motivated reinforcement learning. In _Advances in Neural Information Processing Systems_.
  * Siu et al. (2021) Ho Chit Siu, Jaime D. Pena, Kimberlee C. Chang, Edenna Chen, Yutai Zhou, Victor J. Lopez, Kyle Palko, and Ross E. Allen. 2021\.  Evaluation of human-AI Teams for learned and rule-based agents in Hanabi.  _arXiv preprint arXiv:2107.07630_ (2021).
  * Slonim and Garbarino (2008) Robert Slonim and Ellen Garbarino. 2008.  Increases in trust and altruism from partner selection: Experimental evidence.  _Experimental Economics_ 11, 2 (2008), 134–153.
  * Stanton and Jensen (2021) Brian Stanton and Theodore Jensen. 2021.  Trust and artificial intelligence.  https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=931087
  * Strathern (1997) Marilyn Strathern. 1997\.  ‘Improving ratings’: Audit in the British University system.  _European Review_ 5, 3 (1997), 305–321.
  * Strouse et al. (2021) DJ Strouse, Kevin R. McKee, Matthew Botvinick, Edward Hughes, and Richard Everett. 2021.  Collaborating with humans without human data. In _Advances in Neural Information Processing Systems_.
  * Tylkin et al. (2021) Paul Tylkin, Goran Radanovic, and David C. Parkes. 2021\.  Learning robust helpful behaviors in two-player cooperative Atari environments. In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_. 1686–1688.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Ye et al. (2020) Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. 2020\.  Rotation, translation, and cropping for zero-shot generalization. In _2020 IEEE Conference on Games_. IEEE, 57–64.
  * Zacks (2004) Jeffrey M. Zacks. 2004\.  Using movement and intentions to understand simple events.  _Cognitive Science_ 28, 6 (2004), 979–1008.

##  Appendix A Task details

We implement Coins using the DMLab2D platform (Beattie et al., 2020). Table A1
details the task settings used for agent training, as well as for the tutorial
and co-play episodes in the human-agent interaction studies. Figure A2 depicts
the $11\times 11$ room used for all co-play episodes in the interaction
studies.

Parameter | Agent | Tutorial | Co-play
---|---|---|---
training | episodes | episodes
Number of | $2$ | $1$ | $2$*
players ($n$)
Room width ($w$) | $\sim\mathcal{U}\\{10,15\\}$ | $5$ | $11$
Room depth ($d$) | $\sim\mathcal{U}\\{10,15\\}$ | $7$ | $11$
Coin spawn | $0.0005$ | $0.0015$ | $0.0005$
probability ($P$)
Episode length ($T$) | $500$ |  $1500$** | $300$
Table A1. Parameters for the Coins task in agent training and human-agent
cooperation studies. *In Study 3, co-play episodes involve either $\bm{n=1}$
or $\bm{n=2}$ players, depending on whether the participant chose to play
alone or with the co-player from the prior episode, respectively. **Tutorial
episodes terminated after participants collected five coins or after
$\bm{T=1500}$ steps (5 minutes).

###  A.1. Entities and sprites

####  A.1.1. Players and coins

The two primary entities in the Coins task are players and coins (Figure A1a).
We use a colorblind-friendly palette for the player and coin sprites in our
experiments.

(a)  (b)

Figure A1. Entities and sprites used in the Coins task.

Each episode of Coins involves two distinct colors sampled from the five
available colors (e.g., red and blue). Each player included in an episode
matches one of these two colors. On each step, coins spawn on empty cells and
cells occupied by players with probability $P$. When a coin spawns, it appears
as one of the two colors, each with 50% probability.

Players can enter empty cells and cells with coins in them. Each player blocks
other players from entering the cell they currently occupy.

####  A.1.2. Miscellaneous

Walls (Figure A1b, top row) block players from moving into a cell, and thus
define the boundaries and dimensions of the room. Otherwise, players are able
to freely move through empty cells (Figure A1b, bottom row) in the room.

###  A.2. Observations

DMLab2D supports two different visual frames of reference (Klatzky, 1998) for
players:

  1. (1)

Egocentric, meaning that players always see their avatar in an invariant
position (e.g., centered) in their visual input

  2. (2)

Allocentric, meaning that the world itself remains in an invariant position
(e.g., centered) in players’ visual input

Egocentric frames of reference promote generalization for reinforcement
learning agents (Hill et al., 2019; Ye et al., 2020). Consequently, we provide
egocentric observations for our agents (Figure A3a). The agent observation
spans five cells in each direction (forward, backward, left, and right) from
the agent. Given the sprite dimensions ($8\times 8\times 3$, with the last
dimension reflecting RGB channels), agents observed their environment through
a $88\times 88\times 3$ window.

In pilot tests of our study interface, human players reported being confused
by the changes in their visual observation when playing with an egocentric
reference frame. In combination with prior evidence that allocentric reference
frames can support human navigation in virtual environments (Darken and Cevik,
1999), this feedback prompted us to provide human players with an allocentric
reference frame for the environment (Figure A3b). As human players move their
avatar around the Coins gridworld, the room stays centered within their visual
input while their avatar moves around the room.

###  A.3. Actions

Players can take one of the following five actions each step:

  1. (1)

No-op: Makes no change to the player’s position.

  2. (2)

Move up: Moves the player up one cell.

  3. (3)

Move down: Moves the player down one cell.

  4. (4)

Move left: Moves the player left one cell.

  5. (5)

Move right: Moves the player right one cell.

Figure A2. A $\mathbf{11\times 11}$ room, as used in all co-play episodes in
human-agent cooperation studies.

(a) Egocentric reference frame. (b) Allocentric reference frame.

Figure A3. Observations for the Coins task, depicting different frames of
reference for the same example game state. (a) Agent players observe Coins
with an egocentric reference frame. (b) Human players observe Coins with an
allocentric reference frame.

##  Appendix B Agent details

###  B.1. Training protocol

Agents trained in a distributed set of environments running in parallel. Each
agent trained using one GPU, for a total of ${5}\mathrm{e}{7}$ environment
steps across 150 environments. Our distributed framework checkpointed agents
every 15 minutes during training.

We compute training curves by evaluating each agent checkpoint in 100 self-
play episodes, then calculating the mean and standard deviation of total coin
collections (Figure A4), mismatching coin collections (Figure A5), and
collective return (Figure A6) for each set of evaluation episodes.

###  B.2. Social Value Orientation

McKee et al. (2020) introduce and define the Social Value Orientation (SVO)
algorithmic component across three descriptive levels (Hamrick and Mohamed,
2020; Marr, 1982):

  1. (1)

At the computational level, McKee et al. propose SVO as a mechanism
“redefin[ing] self-interest to incorporate the interests of a broader group”.

  2. (2)

At the algorithmic level, McKee et al. introduce “reward angles” as a a method
of representing distributions of reward over the self and the group.

  3. (3)

At the implementation level, McKee et al. offer a penalty-based approach that
assigns pseudoreward to SVO agents based on the divergence between their
target reward angle and the realized reward angle, in combination with a
weight parameter $w$. Eq. (3) in (McKee et al., 2020) defines this
implementation.

Here we introduce a new implementation of SVO that retains the computational
and algorithmic levels as described above, but replaces the divergence-
penalization method with an approach based on vector projection. Within the
Markov game framework described in (McKee et al., 2020), we define the vector-
projection approach with a function $U_{i}$ to be maximized by agent $i$:

$U_{i}(s,a)=r_{i}\cdot\cos(\theta^{\text{SVO}})+\bar{r}_{-i}\cdot\sin(\theta^{\text{SVO}})$
(1)

In the Markov game framework, $s$ is the current environmental state; $o_{i}$
is the observation received by agent $i$; agent $i$ selects its action $a_{i}$
through a policy $\pi(a_{i}|o_{i})$; $a$ is the joint action taken by all
agents; environmental reward $r$ is a function of state and joint action;
$\theta^{\text{SVO}}$ is the Social Value Orientation for player $i$
(representing player $i$’s target distribution of reward among group members);
and $\bar{r}_{-i}$ is a statistic summarizing the rewards of all other group
members. For these experiments, we choose to define $\bar{r}_{-i}$ as the
arithmetic mean: $\bar{r}_{-i}=\frac{1}{{n-1}}\sum_{j\neq i}r_{j}$, where $n$
is the group size. This formulation of SVO parallels the model of human
preferences described by Griesinger and Livingston Jr. (1973).

Figure A4. Effect of $\bm{\epsilon}$ on total coin collection over training.
Error bands represent 95% confidence intervals over 100 evaluation episodes at
each training checkpoint. Figure A5. Effect of $\bm{\epsilon}$ on mismatching-
coin collection over training. Error bands represent 95% confidence intervals
over 100 evaluation episodes at each training checkpoint. Figure A6. Effect of
$\bm{\epsilon}$ on collective return over training. Error bands represent 95%
confidence intervals over 100 evaluation episodes at each training checkpoint.

##  Appendix C Human-agent interaction studies

All three studies received a favorable opinion from the Human Behavioural
Research Ethics Committee at DeepMind (#19/004) and were approved by the
Institutional Review Board for Human Subjects at Princeton University
(#11885).

Each study applied the same inclusion criteria during recruitment on Prolific:
residence in the United States, prior completion of at least 20 studies on
Prolific, and an approval rating of 95% or more.

###  C.1. Study design

We include screenshots showing how Study 1 unfolded for each participant:

  1. (1)

Read general instructions on the study and gameplay (Figure A7).

  2. (2)

Play practice episode (Figure A8).

  3. (3)

Read instructions on game rules and co-players (Figures A9 and A10).

  4. (4)

Play co-play episode with Co-player A and answer questions about perceptions
(Figure A11).

  5. (5)

Play co-play episode with Co-player B and answer questions about perceptions
and preferences (Figure A12).

  6. (6)

Repeat steps 3 and 4 for another 10 episodes.

  7. (7)

Transition to debrief questionnaire (Figure A13).

Figure A14 shows an example open-ended impression question from the debrief
questionnaire.

During gameplay, a “ticker” at the top of the screen displayed the three most
recent coin collection events. The study interface did not provide
participants with cumulative metrics (e.g., score or coin collections).

In all three studies, the practice episode instructed participants to collect
five coins. The practice episode ended after the participant collected five
coins or five minutes elapsed, whichever occurred first.

In Studies 1 and 2, we generated the sequence of co-players for each
participant by listing the 12 possible pairwise combinations of agents,
randomly shuffling the order of the pairs, and then randomly shuffling the
order of the agents within each pair.

Study 2 closely resembled Study 1, aside from identifying the bonus earned per
point as $\$0.02$ (Figures A9a and A10h) and using the reward values from the
shifted incentive structure (Table 2) when explaining the rules for matching
and mismatching coin collections (Figures A10b-A10e).

Study 3 largely preserved the instructions preceding the first co-play episode
(Figures A7-A10), though the wording on certain screens changed slightly to
refer to a single upcoming episode rather than multiple episodes (e.g., “the
next round” rather than “each round”). The study branched more substantially
starting just before the first co-play episode, as shown in Figures A15 and
A16. After playing a single co-play episode, participants read a description
of their partner choice for the following episode. They played the partner-
choice episode as they chose (that is, either alone or with the co-player from
the previous episode).

###  C.2. Expanded analyses

Figures A17-A28 provide expanded visualizations of analysis presented in the
main text.

(a) Screen 1: Welcome participants to the experiment.

(b) Screen 2: Explain the keyboard controls.

(c) Screen 3: Provide overview of the experiment and bonus.

(d) Screen 4: Introduce the participant’s avatar.

(e) Screen 5: Introduce the room.

(f) Screen 6: Explain the controls for movement.

(g) Screen 7: Explain coin collections.

(h) Screen 8: Introduce the practice episode.

Figure A7. Screenshots of instruction screens in Study 1.

(a) Screen 9: Load episode.

(b) Screen 10: Play practice episode.

(c) Screen 11: Save episode data.

Figure A8. Screenshots of practice episode in Study 1.

(a) Screen 12: Explain the bonus rules.

(b) Screen 13: Explain co-play.

(c) Screen 14: Introduce an example co-player.

Figure A9. Screenshots of instruction screens in Study 1.

(a) Screen 15: Explain coin colors.

(b) Screen 16: Explain matching coin rules.

(c) Screen 17: Explain matching coin rules.

(d) Screen 18: Explain mismatching coin rules.

(e) Screen 19: Explain mismatching coin rules.

(f) Screen 20: Introduce all co-players.

(g) Screen 21: Explain questions.

(h) Screen 22: Begin co-play episodes.

Figure A10. Screenshots of instruction screens in Study 1.

(a) Screen 23: Introduce co-player for this episode.

(b) Screen 24: Load episode.

(c) Screen 25: Play co-play episode.

(d) Screen 26: Save episode data.

(e) Screen 27: Elicit judgment (randomized order).

(f) Screen 28: Confirm judgment.

Figure A11. Screenshots of “A” co-play episode and subsequent questions on
perceptions in Study 1.

(a) Screen 29: Introduce co-player for this episode.

(b) Screen 30: Load episode.

(c) Screen 31: Play co-play episode.

(d) Screen 32: Save episode data.

(e) Screen 33: Elicit judgment (randomized order).

(f) Screen 34: Confirm judgment.

(g) Screen 35: Elicit preference.

(h) Screen 36: Confirm preference.

Figure A12. Screenshots of “B” co-play episode and subsequent questions on
perceptions and preferences in Study 1.

(a) Screen 37: Confirm bonus and transition to debrief questionnaire. Figure
A13. Screenshot of pre-questionnaire page in Study 1. Figure A14. Screenshot
of open-ended impression question from debrief questionnaire in Study 1.

(a) Screen 20: Explain questions.

(b) Screen 21: Begin co-play episodes.

(c) Screen 22: Introduce co-player for this episode.

(d) Screen 23: Load episode.

(e) Screen 24: Play co-play episode.

(f) Screen 25: Save episode data.

(g) Screen 26: Elicit judgment (randomized order).

(h) Screen 27: Confirm judgment.

Figure A15. Screenshots of instructions, co-play episode, and subsequent
questions on perceptions in Study 3.

(a) Screen 28: Introduce partner-choice episode.

(b) Screen 29: Explain partner choice.

(c) Screen 30: Explain coins in partner-choice episode.

(d) Screen 31: Elicit partner choice.

(e) Screen 32: Confirm partner choice.

(f) Screen 33: Introduce players for this episode.

(g) Screen 34: Load episode.

(h) Screen 35: Play partner-choice episode.

(i) Screen 35: Save episode data.

Figure A16. Screenshots of instructions, partner choice question, and partner-
choice episode in Study 3.

(a) Warmth. (b) Competence.

Figure A17. Interaction between the SVO and trembling-hand components on
social perceptions in Study 1. Error bars reflect 95% confidence intervals.
Figure A18. Pairwise preferences for different agents in Study 1. Cell values
represent self-reported preferences for the row agent over the column agent.

Figure A19. Odds ratios in Study 1. Asterisks indicate that predictors are centered and normalized by one standard deviation to permit fair comparison between their effect sizes. Trait | ICC [95% CI] |  p-value
---|---|---
“warm” |  $0.70$ $[0.67$, $0.74]$ | $<0.001$
“well-intentioned” |  $0.68$ $[0.64$, $0.72]$ | $<0.001$
“competent” |  $0.53$ $[0.48$, $0.57]$ | $<0.001$
“intelligent” |  $0.51$ $[0.46$, $0.56]$ | $<0.001$
Table A2. Participants’ evaluations of their co-players were highly consistent
in Study 2, as assessed by ICC. Higher values of ICC indicate greater
consistency.

(a) SVO. (b) Trembling hand.

Figure A20. Main effects of the SVO and trembling-hand components on social
perceptions in Study 2. (a) An agent’s SVO parameter significantly affected
warmth perceptions, $\mathbf{\mathit{p}<0.001}$. (b) The trembling-hand
component significantly influenced competence evaluations,
$\mathbf{\mathit{p}<0.001}$. Error bars reflect 95% confidence intervals.

(a) Warmth. (b) Competence.

Figure A21. Interaction between the SVO and trembling-hand components on
social perceptions in Study 2. Error bars reflect 95% confidence intervals.

Figure A22. Pairwise preferences for different agents in Study 2. Cell values
represent self-reported preferences for the row agent over the column agent.
Figure A23. Odds ratios in Study 2. Asterisks indicate that predictors are
centered and normalized by one standard deviation to permit fair comparison
between their effect sizes.

(a) Warmth and impression sentiment. (b) Competence and impression sentiment.

Figure A24. Relationship between (numeric) social evaluation and sentiment in
participants’ free-response impressions in Study 2.

(a) SVO. (b) Trembling hand.

Figure A25. Main effects of the SVO and trembling-hand components on social
perceptions in Study 3. (a) The SVO component significantly altered warmth
judgments, $\mathbf{\mathit{p}<0.001}$. (b) Similarly, an agent’s trembling-
hand parameterization significantly influenced perceived competence,
$\mathbf{\mathit{p}<0.001}$. Error bars reflect 95% confidence intervals.

(a) Warmth. (b) Competence.

Figure A26. Interaction between the SVO and trembling-hand components on
social perceptions in Study 3. Error bars reflect 95% confidence intervals.
Figure A27. Odds ratios in Study 3. Asterisks indicate that predictors are
centered and normalized by one standard deviation to permit fair comparison
between their effect sizes.

(a) Warmth and impression sentiment. (b) Competence and impression sentiment.

Figure A28. Relationship between (numeric) judgments and sentiment in
participants’ free-response impressions in Study 3.