# MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment
Tasks

Allen Nie1∗ Yuhui Zhang1 Atharva Amdekar2
Chris Piech1 Tatsunori Hashimoto1 Tobias Gerstenberg3∗
1Computer Science 2ICME 3Psychology
Stanford University
*{anie<EMAIL_ADDRESS>

###### Abstract

Human commonsense understanding of the physical and social world is organized
around intuitive theories. These theories support making causal and moral
judgments. When something bad happens, we naturally ask: who did what, and
why? A rich literature in cognitive science has studied people’s causal and
moral intuitions. This work has revealed a number of factors that
systematically influence people’s judgments, such as the violation of norms
and whether the harm is avoidable or inevitable. We collected a dataset of
stories from 24 cognitive science papers and developed a system to annotate
each story with the factors they investigated. Using this dataset, we test
whether large language models (LLMs) make causal and moral judgments about
text-based scenarios that align with those of human participants. On the
aggregate level, alignment has improved with more recent LLMs. However, using
statistical analyses, we find that LLMs weigh the different factors quite
differently from human participants. These results show how curated, challenge
datasets combined with insights from cognitive science can help us go beyond
comparisons based merely on aggregate metrics: we uncover LLMs implicit
tendencies and show to what extent these align with human intuitions.

##  1 Introduction

Humans rely on their intuition to understand the world. This intuition helps
us to understand not only physical events (e.g., one ball caused the other to
move) but also complex social situations (e.g., the collapse of Sam Bankman-
Fried’s FTX caused unprecedented turmoil in the cryptocurrency market). Given
a complex set of events, even with a great amount of ambiguity, we can answer
questions such as “What or who caused it?” Our answers to this question
reflect how we intuitively understand events, people, and the world around us
(Sloman & Lagnado, 2015; Pearl & Mackenzie, 2018). How do humans handle this
complexity?

Cognitive scientists have proposed that we do so by organizing our
understanding of the world into intuitive theories (Gerstenberg & Tenenbaum,
2017; Wellman & Gelman, 1992). Accordingly, people have intuitive theories of
the physical and social world with which we reason about how objects and
agents interact with one another (Battaglia et al., 2013; Ullman et al., 2017;
Gerstenberg et al., 2021; Baker et al., 2017; Davis & Marcus, 2015; Lake et
al., 2017). Concepts related to causality and morality form key ingredients of
people’s physical and social theories. Given a story, humans can readily make
causal and moral judgments about the objects and agents involved in the story.

Studying these human intuitions or systematic tendencies when making decisions
or judgments is the central focus of psychological experimentations. What are
these tendencies? How do they influence our judgment? Over the last several
decades, using text-based vignettes, psychologists have disentangled what
factors influence people’s causal and moral judgments. These factors can be
understood as the building blocks of our thought processes for making causal
and moral judgments.

Over the last few years, large language models (LLMs) have become increasingly
successful in emulating certain aspects of human commonsense reasoning ranging
from tasks such as physical reasoning (Tsividis et al., 2021), visual
reasoning (Buch et al., 2022), moral reasoning (Hendrycks et al., 2020), and
text comprehension (Brown et al., 2020; Liu et al., 2019b; Bommasani et al.,
2021). Here, we investigate to what extent pretrained LLMs align with human
intuitions about the role of objects and agents in text-based scenarios, using
judgments about causality and morality as two case studies. We show how
carefully constructed text scenarios from cognitive science can help shed
light on the extent to which humans and LLMs align in a way that goes beyond
mere agreement in aggregate.

Prior work on alignment between LLMs and human intuitions usually collected
evaluation datasets in two stages. In the first phase, participants write
stories with open-ended instructions. In the second phase, another group of
participants labels these participant-generated stories (e.g. Hendrycks et
al., 2020). The upside of this approach is the ease of obtaining a large
number of examples in a short period of time. The downside of this approach is
that the crowd-sourced stories are often not carefully written, and that they
lack experimental control. Here, we take a different approach. Instead of
relying on participant-generated scenarios, we collected two datasets from the
existing literature in cognitive science: one on causal judgments and another
on moral judgments. These scenarios were carefully written by researchers with
the intention of systematically manipulating one or a few factors that have
been theorized to influence people’s judgments. These latent factors naturally
bring structures to the otherwise plain text stories, which aim to explain
human judgments (see Figure 1). Using these scenarios, we can design a series
of experiments that aim to measure the LLMs’ alignment with human intuition
and use the scientific framework around human judgments to analyze where the
differences occur. We have released the code and dataset here:
https://github.com/cicl-stanford/moca

##### Our Contributions

  1. (C1):

We summarize the main experimental findings of 24 cognitive science papers
into factors that have been shown to influence participants’ judgments on
moral and causal stories (see short version in 2(a), full version in Table
A1). From these papers, we create a causal and moral judgment challenge set in
which only a few words are changed, yet they lead to big differences in
people’s judgments. We collected 5150 human responses to our stories and
annotated the latent factors with experts (see Figure 1).

  2. (C2):

We evaluate how a wide range of LLMs align with human judgments. This includes
models of different sizes, models fine-tuned via reinforcement learning using
human feedback (RLHF), and those distilled from instruction-following, like
Alpaca-7B. Our results show that larger models and RLHF techniques lead to
better alignment.

  3. (C3):

We compute the Average Marginal Component Effect (AMCE) to reveal the implicit
tendencies for each factor from human and LLM judgments. Our analysis reveals
a number of implicit tendency differences between models trained within the
same company or trained with the same technique. Our analysis highlights the
importance of using a carefully constructed dataset to understand tendency
alignment.

##  2 Related Work

For causal reasoning, there is an active line of research at the intersection
of natural language processing (NLP) and commonsense reasoning that involves
extracting and representing causal relationships among entities in text. In
some cases, these relationships are based on commonsense knowledge of how
objects behave in the world (Sap et al., 2019; Bosselut et al., 2019; Talmor
et al., 2019). In other cases, models identify scenario-specific causal
relations and outcomes. Sometimes, the causal relationship between entities is
explicitly stated (e.g., those cancers were caused by radiation) (Hendrickx et
al., 2010), while at other times the relationship is left implicit and needs
to be inferred (Mostafazadeh et al., 2016). Causal reasoning is also included
in broad benchmarks for language understanding, such as the Choice of
Plausible Alternatives (COPA) in SuperGLUE (Roemmele et al., 2011; Wang et
al., 2019).

For moral reasoning, tasks have focused on evaluations of agents in narrative-
like text. These tasks and datasets vary in the amount of structure they
provide, ranging from pairs of free-form anecdotes and judgment labels (Lourie
et al., 2021; Hendrycks et al., 2020), to inputs with components separated out
into norms, intention, actions, and consequences (Emelin et al., 2021). Prior
work has also looked into moral dilemmas, particularly around everyday
scenarios (Hendrycks et al., 2020; Jiang et al., 2021). In this work, the
moral scenarios were generated through crowd-sourcing (Emelin et al., 2021;
Ziems et al., 2022), and there is general agreement about what’s the morally
right thing to do in a scenario (i.e. these scenarios weren’t moral dilemmas).
Alternatively, scenarios have been scraped from online communities such as
Reddit (Forbes et al., 2020; Lourie et al., 2021; Roemmele et al., 2011).
While these scenarios have greater external validity, they lack the
experimental control of scenarios from cognitive science research. How LLMs
align with people’s intuitions across a range of moral dilemmas hasn’t
systematically been studied before. Kıcıman et al. (2023) proposed modularized
tests to understand whether LLMs can perform different types of causal
analysis. Our work investigates whether these LLMs exhibit implicit tendencies
similar to those of humans. Scherrer et al. (2023) evaluated LLMs on moral
scenarios but focused on moral rule violation along the axis of moral
“ambiguity”. There are also discussions of moral behaviors with reinforcement
learning agents in game-like environments (Hendrycks et al., 2021; Reinecke et
al., 2023; Pan et al., 2023) and with respect to legal reasoning (Almeida et
al., 2023).

Figure 1: Two examples from our collected dataset. (a) shows a causal judgment
story, and (b) shows a moral judgment story. In (a), a conjunction of two
events was required, an abnormal event occurred, and Jane violated a
prescriptive norm (scenario taken from Knobe & Fraser, 2008). In (b), the
teenager’s death was inevitable; his death is a necessary means to save
others, and bringing about his death requires the use of personal force
(scenario taken from Christensen et al., 2014).

Our work falls under a broad range of research on commonsense reasoning, where
models are encouraged to produce outputs that match human intuitions (Gabriel,
2020; Kenton et al., 2021). In some cases, such as physical commonsense
reasoning, alignment with human behavior is straightforward, while in the case
of social commonsense, the subjectivity and diversity of human intuitions make
alignment more challenging (Davani et al., 2022). Prior analysis of language
models’ reasoning abilities includes measuring their behavior in zero-shot
settings, after task fine-tuning (Jiang et al., 2021; Hendrycks et al., 2020),
or with human-curated support, such as chain-of-thought prompt engineering
(Wei et al., 2022b; Wang et al., 2022). Evaluating LLMs with causal hypotheses
has also been explored by Kosoy et al. (2022). While they focused on the
influence of causal structure, we investigate a variety of factors (including
causal structure). Concurrently, Jin et al. (2022) curated a challenge dataset
to examine and verify LLMs’ ability to conform to three categories of social
norms. We look at five factors that have been shown to influence people’s
moral judgments and analyze whether LLMs respond to these factors similarly to
humans.

##  3 Judgment Tasks

We study the alignment between LLMs and human participants in two case
studies: (1) a causal judgment task about whether someone or something caused
the outcome of interest, and (2) a moral judgment task where the question is
whether what a protagonist did (or failed to do) was morally permissible.
Figure 1 shows an example of each task.

###  3.1 Causal Judgment Task

Deciding what “the” cause of an outcome was can be challenging because there
are often multiple events that contributed to an outcome. Here is an example
story:

A merry-go-round horse can only support one child. Suzy is on the horse. Billy
is not allowed to get on, but he climbs on anyway. The horse broke due to the
two children’s weight. Did Billy cause the horse to break?

Even though both children were involved in breaking the horse, when
participants are asked to assess whether Billy (who wasn’t allowed to get on)
caused it to break, they tend to answer with “yes”. Formal models of causal
selection have been developed that capture choice patterns in scenarios like
this one (Kominsky et al., 2015; Icard et al., 2017; Gerstenberg & Icard,
2020; Quillien & Lucas, 2023). In this example, Billy violated a prescriptive
norm, whereas Suzy didn’t. People often select norm-violating events as the
cause of an outcome (Knobe & Fraser, 2008; Hitchcock & Knobe, 2009; Alicke et
al., 2011; Hilton & Slugoski, 1986; Hart & Honoré, 1959/1985). It is worth
noting that causality is closely related to responsibility and blame. In fact,
formal models of responsibility and blame have been built on top of a causal
selection model (Halpern, 2015; Lagnado et al., 2013).

In addition to norm violations, prior work has identified a number of factors
that systematically influence people’s causal judgments. Here, we focus on the
following six factors: causal structure, agent awareness, norm type, event
normality, action/omission, and time. 2(a) provides brief definitions of the
different factors.

###  3.2 Moral Permissibility Task

Philosophers and cognitive scientists have used moral dilemmas to develop and
evaluate normative theories of ethical behavior, and to investigate what moral
intuitions people have. A popular moral dilemma is the trolley problem (Foot,
1967; Thomson, 1976), where the question is whether it’s morally permissible
for the protagonist to re-route a trolley from the main track (saving some) to
a side track (killing others). Some LLMs already show a certain degree of
alignment with human responses in various versions of the trolley dilemma
(Hendrycks et al., 2020; Jiang et al., 2021). However, only a few factors
varied in these trolley dilemmas, such as the number of people on each track
or personal attributes (age, social status, etc.; Awad et al., 2018).

A large number of factors have been shown to influence people’s moral
intuitions. For example, in Figure 1b, people may consider factors such as
whether personal force was required to bring about the effect (Greene et al.,
2009), whether there was a risk for the protagonist themselves (Bloomfield,
2007), whether the harm was a side effect or a means for bringing about the
less bad outcome (Moore et al., 2008), and whether the harm was inevitable
(Hauser, 2006; Mikhail, 2007). This work has shown that people’s moral
judgments are sensitive to a variety of factors that haven’t been considered
in existing work on the alignment between LLMs and people’s moral intuitions
(Waldmann & Dieterich, 2007; Liao et al., 2012; Christensen et al., 2014;
Kleiman-Weiner et al., 2015). 2(b) provides brief definitions of these
factors.

Table 1: Dataset: We report dataset statistics on the label distribution, average length of each story, and inter-rater agreement between two annotators on the factors and the sentences they highlight. Additionally, we collect a binary response for each story from 25 people. Dataset | # Stories |  Yes ($p>0.6$) |  No ($p<0.4$) | Ambiguous |  | # words per
---
story
| # words per
---
translated story
Causal | 144 | 48 | 50 | 46 | 162 | 82.9
Moral | 62 | 23 | 10 | 29 | 72.5 | 53.5
Total | 206 | 71 | 60 | 75 | 135 | 74.1

###  3.3 Dataset

For each task, we transcribe stories from a number of papers. For the Causal
Judgment Task, we transcribed 144 stories from a selection of 20 papers, and
for the Moral Permissibility Task, we transcribed 62 stories from 4 papers
that covered a wide range of moral dilemmas. High-level descriptions of the
dataset are presented in Table 1. We select these papers relying on expert
domain knowledge – as these papers contain robust scientific findings and
cover different kinds of scenarios. We collect 25 yes/no answers for each
story from a crowd-sourcing platform with IRB approval for our data collection
process. We describe the experimental design and data collection details in
Appendix A.15. We additionally recruited two domain experts to annotate what
factors are present in each story. We report the inter-rater agreement between
the two annotators and describe their annotation process in Appendix A.14.

Table 2: Factors: Factors that influence causal selection judgments (top) and
moral permissibility judgments (bottom). We provide definitions for each
factor in Appendix A.1 and Appendix A.2. See the full version in Table A1.

(a) Causal Factors Factor | Attribute
---|---
Causal Structure | Conjunctive | Disjunctive
Agent Awareness | Aware | Unaware
Norm Type | Prescriptive | Statistical
Event Normality | Normal | Abnormal
Action / Omission | Action | Omission
Time | Early | Late | Same Time

(b) Moral Factors Factor | Attribute
---|---
Causal Role | Means | Side Effect
| Locus of Intervention
---
Instrument | Patient of Harm
Personal Force | Personal | Impersonal
| Counterfactual
---
Evitability
Avoidable | Inevitable
Beneficence | Benefit Self | Benefit Other

##  4 Experiment and Result

###  4.1 Do LLMs make the same judgments as people on these stories?

##### Setup.

We first directly compare the responses of a set of LLMs to those of human
participants across the set of stories in the original cognitive science
experiments. We choose a wide range of language models that have achieved good
performance with fine-tuning on other natural language understanding tasks
(Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019a; Lan et al.,
2019; Clark et al., 2020). We conduct a 3-class comparison to compute
accuracy: a response of “Yes”, “No’, and an additional response of “ambiguous”
when the agreement between the average human responses or the probability of
model output is 50% ± 10%. We normalize the model output probability to
compute $P(\text{``Yes''}|\text{Story + Prompt})$ and
$P(\text{``No''}|\text{Story + Prompt})$. For chat-based APIs, we prompt the
model to output these labels directly. See Appendix A.3 for details about how
we computed the probability for each model. Results should not be compared
between completion and chat models.

We report discrete agreement (Agg) between LLMs and human participants. The
agreement is computed in the same way as accuracy, but since we don’t want to
imply that higher agreement is necessarily better, we use the term “agreement”
instead of “accuracy”. We also report additional measures of agreement, such
as the area under the receiver operating characteristic curve (AUC), the
absolute-mean-squared error (MAE), and cross-entropy (CE) on the probability
of the matched label.

Table 3: Original Story: We ran our experiments and compute the 95%
bootstrapped confidence interval for the result. We report discrete agreement
(Agg), the area under the curve for the unambiguous stories (AUC, higher is
better matched, 0.5 is chance, and below 0.5 is worse matched than chance),
mean absolute error (MAE, lower is better matched), and cross-entropy (CE,
lower is better matched). In 3(b), we only report results from the best model.
Red shows the alignment decreased compared to before. Blue shows the alignment
increased. See Table A2 for a full comparison.

(a) Zero-shot with Different Large Language Models

| Causal Judgment | Moral Permissibility
---|---|---
Models |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$) |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$)
RoBERTa-large |  34.0$\pm$5.6 | 0.50 | 0.50 | 2.6 |  26.6$\pm$7.3 | 0.50 | 0.50 | 2.65
ALBERT-xxlarge |  35.1$\pm$5.6 | 0.51 | 0.38 | 1.37 |  23.4$\pm$8.1 | 0.48 | 0.37 | 1.82
Electra-gen-large |  34.7$\pm$5.9 | 0.53 | 0.44 | 1.94 |  26.6$\pm$8.1 | 0.49 | 0.46 | 1.59
GPT2-XL |  34.4$\pm$5.6 | 0.49 | 0.42 | 1.47 |  26.6$\pm$8.1 | 0.51 | 0.43 | 1.71
GPT3-babbage-v1 |  31.2$\pm$5.9 | 0.47 | 0.33 | 0.74 |  18.5$\pm$7.3 | 0.43 | 0.41 | 1.03
GPT3-curie-v1 |  36.8$\pm$6.2 | 0.51 | 0.37 | 1.14 |  26.6$\pm$8.1 | 0.45 | 0.39 | 1.32
GPT3.5-davinci-v2 |  37.8$\pm$5.6 | 0.61 | 0.31 | 0.72 |  32.3$\pm$8.1 | 0.67 | 0.30 | 0.74
GPT3.5-davinci-v3 |  41.3$\pm$5.6 | 0.67 | 0.41 | 1.39 |  20.2$\pm$7.3 | 0.61 | 0.51 | 2.61
Alpaca-7B |  32.3$\pm$5.6 | 0.51 | 0.28 | 0.65 |  32.3$\pm$8.1 | 0.45 | 0.32 | 0.91
Anthropic-claude-v1 |  36.1$\pm$5.9 | 0.57 | 0.34 | 1.37 |  37.1$\pm$8.9 | 0.59 | 0.29 | 1.29
GPT3.5-turbo |  35.4$\pm$5.6 | 0.51 | 0.45 | 2.11 |  40.3$\pm$8.9 | 0.65 | 0.33 | 1.49
GPT-4 |  43.1$\pm$5.9 | 0.61 | 0.44 | 1.89 |  41.9$\pm$8.9 | 0.74 | 0.31 | 1.28
Delphi | — | — | — | — |  22.6$\pm$11.3 | — | — | —

(b) Zero-shot with Social Simulacra and Prompt Optimization

Models | Causal (GPT3.5-davinci-v3) | Moral (GPT3.5-davinci-v2)
---|---|---
Methods |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$) |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$)
Persona (average) |  39.7$\pm$1.1 | 0.60 | 0.31 | 0.73 |  28.9$\pm$1.6 | 0.55 | 0.33 | 0.86
Persona (best) |  42.7$\pm$5.6 | 0.60 | 0.31 | 0.71 |  37.1$\pm$8.5 | 0.58 | 0.27 | 0.72
Persona (worst) |  37.2$\pm$5.4 | 0.62 | 0.29 | 0.61 |  18.5$\pm$6.5 | 0.58 | 0.46 | 0.71
Auto Prompt Engineer |  40.6$\pm$5.8 | 0.64 | 0.41 | 1.55 |  40.4$\pm$8.8 | 0.46 | 0.28 | 0.63

##### Zero-shot Alignment

We report the result in Table 3. We notice that there is no model that
perfectly aligns with human judgments on these metrics. We first find that
larger models (GPT-3, GPT-3.5, GPT-4, and Claude) generally align with humans
better than smaller models (GPT-2, RoBERTa, etc.). This is most apparent when
one compares the performance of GPT3-curie-v1 (6.7B) to GPT3-davinci-v1
(175B). Both models are trained using the same setup, but on both the Moral
Permissibility Task and the Causal Judgment Task, GPT3.5-davinci-v2 aligns
better than GPT3-curie-v1. While existing literature has shown improved
performances of LLMs on general deterministic tasks (Wei et al., 2022a), we
show here that LLM alignment also improves on ambiguous tasks.

Recently, techniques such as supervised fine-tuning on human instructions
(GPT3.5-davinci-v2; Ouyang et al., 2022), using reinforcement learning to
update LLM weights from human feedback (RLHF; GPT3.5-davinci-v3, GPT3.5-turbo,
Anthropic-claude-v1; Stiennon et al., 2020; Bai et al., 2022a), and multi-
modal joint training on both language and images (GPT-4; OpenAI, 2023) have
improved LLMs on various tasks. Perhaps surprisingly, unlike the monotonic
improvement we find by increased model size, these techniques impact these
models’ aggregate-level alignment differently. Comparing the result between
GPT3.5-davinci-v2 and GPT3.5-davinci-v3, RLHF seems to increase aggregate
alignment on causal judgment but decrease alignment on moral judgment.
Comparing the results between Anthropic-claude-v1 and GPT3.5-davinci-v3, it
seems that even though both models are fine-tuned with RLHF, on aggregate,
their alignment with human judgments differs. In the next section, we will
design a method to examine how different training methods cause subtle shifts
in the model’s implicit tendency.

##### Social Simulacra

Park et al. (2022) proposed a framework to elicit diverse social behaviors
from LLMs by prompting them with different personas. For example, adding the
persona, “Jane Smith is a liberal activist.” before the question. The human
responses we collect on each story are diverse (see Figure A4 for the
distribution of P(Yes)). We shouldn’t expect LLMs to generate a single
response that matches the diversity of human perspectives. Therefore, we
designed an experiment where we sample 5 personas from 5 simulacra defined by
Park et al. (2022) (25 in total). We elicit $P$(“Yes”$|$Persona+Story+Prompt)
and P(“No”$|$Persona+Story+Prompt) from the model.

We report three ways of incorporating personas: Persona (average), where we
average $P$ over all personas. Persona (best), where we find the persona that
gave the most human-aligned responses. Persona (worst), where we use $P$ from
the persona that gave the least human-aligned responses. See how we designed
this experiment in Appendix A.4 and our sampled personas in Table A3. We chose
the GPT3.5-davinci-v2 and GPT-3.5-davinci-v3 for this experiment because their
API endpoints explicitly provide the probability of Yes or No answers, and
they align with humans best.

Personas elicit very different _moral permissibility_ responses – the most and
least-aligned persona have close to 18.6% difference in discrete agreement.
However, _causal judgments_ do not vary much based on different personas. We
show the most and least aligned persona in Appendix A.4.

##### Automatic Prompt Optimization

We additionally explore whether we can use a black box method to automatically
identify a prompt that is maximally aligned with collected human responses.
Zhou et al. (2022) proposed Automatic Prompt Engineer (APE), a method that
uses LLMs to generate prompts and then use the log-probability of a few
training examples to select the best prompts. Note that APE does not currently
have a workflow defined for chat-based models. Our experiment used five
stories as training examples to select prompts. We describe how we searched
the optimal prompt in Appendix A.5. We see that APE is still limited by the
generative capability of LLMs – we are unable to find a better prompt for the
Causal Judgment Task for GPT3.5-davinci-v3. However, for the Moral
Permissibility Task, we found a prompt that substantially improved
GPT3.5-davinci-v2’s discrete agreement by 8.1%. The discovered prompt is
“James is a consequentialist.” On average, we found that prompts for the Moral
Permissibility Task with words like “consequentialist” or “utilitarian” lead
to the highest alignment with humans. However, this observation is nuanced,
and we provide an extended discussion in Appendix A.6. For causality, good
prompts are with the words “a good student”, “a good researcher”, or “a very
logical person”. We report all the generated prompts in Appendix A.5.

###  4.2 Do LLMs have the same implicit tendencies as people on these stories?

In the previous section, we focus on analyzing aggregate metrics such as
agreement over all stories. Such analysis often provides no information beyond
comparing highly complicated systems with a single number. Since each of our
stories is a combination of factors with corresponding attributes (see Figure
1), we can leverage conjoint analysis and compute the Average Marginal
Component Effect (AMCE) for each factor attribute (Hainmueller et al., 2014;
Awad et al., 2018), where AMCE reveals the implicit tendency of the underlying
system when a particular attribute is present.

####  4.2.1 Method

##### Average Marginal Component Effect

We provide an overview of how AMCE is computed for each factor that reveals
the implicit tendency of the system. For $N$ responses, $K$ stories, and $J$
factors. Each factor has 2 attributes/levels. For 3-level factors such as
“Time”, we only pick “Early” and “Late” attributes. Each response is
$Y_{jnk}\in\\{0,1\\}$, denoting $n$-th response, $j$-th factor, and $k$-th
story. For LLMs, $Y_{jnk}=P($Yes$|$Story$)$. We also define a factor profile
matrix $T\in\\{0,1\\}^{J\times K}$, where $T_{jk}$ denotes the attribute level
of the $j$-th factor in $k$-the story. Now, we can use a non-parametric
difference-in-means estimator to compute AMCE:

$\Delta(j)=\frac{\sum_{j=1}^{J}\sum_{n=1}^{N}\sum_{k=1}^{K}\mathbbm{1}\\{T_{kj}=1\\}Y_{jnk}}{\sum_{j=1}^{J}\sum_{n=1}^{N}\sum_{k=1}^{K}\mathbbm{1}\\{T_{kj}=1\\}}-\frac{\sum_{j=1}^{J}\sum_{n=1}^{N}\sum_{k=1}^{K}\mathbbm{1}\\{T_{kj}=0\\}Y_{jnk}}{\sum_{j=1}^{J}\sum_{k=1}^{K}\sum_{n=1}^{N}\mathbbm{1}\\{T_{kj}=0\\}}$
(1)

Note that if there is no preference between the two attributes of the same
factor, $\Delta(j)=0$. Here is an example of how this estimator works. If we
want to know if a system has a stronger tendency to find causing harm to be
more morally permissible when they are “Inevitable” than “Avoidable” for the
factor “Counterfactual Evitability”, we can find all stories where
“Counterfactual Evitability” has the attribute of “Inevitable”, compute the
average of all the binary responses $Y$ for all respondents, then subtract the
average responses $Y$ for all stories with attribute “Avoidable”. We
additionally compute the 95% confidence interval on ACME using Bootstrap and
show these results in Figure A2 and Figure A3. We only plot the average
$\Delta(j)$ in Figure 2 and Figure 3. More details in Appendix A.7.

##### Zero-shot Implicit Tendency

We use the standard radar chart to capture the multi-factor tendency of each
LLM and human. We put the attributes of the same factor (e.g., Abnormal and
Normal) on opposite sides – since the leaning is over one attribute or the
other: $\Delta(j)\in[-1,1]$. Each concentric circle is marked with a
probability (e.g., 0.1), which represents the change in $P$(Yes) if the factor
attribute had been present in the story. The implicit tendency is defined as
the expected change in the probability of the system outputting “Yes” if this
attribute had been present in the story. Intuitively, we can say a model has
an implicit tendency for one attribute when it’s more likely to judge an event
to be the cause, or the impending harm to be more morally permissible, if this
attribute is present.

Figure 2: Causal Implicit Tendencies. Each concentric circle marks the
implicit tendency – the change in probability of responding “Yes” if the
attribute had been present in the story. For example, if “Abnormal" is present
in the story, humans will have a 25% higher probability to respond “Yes” on
average than “Normal” being present. The left figure compares models of
different sizes. The right figure compares models trained or finetuned with
different methods.

####  4.2.2 Results

##### Causality: Sensitivity to Abnormality

People tend to cite abnormal events as the cause of the outcome. In our
stories, the normality of events is clearly stated. However, a model still
needs to first, identify the norm specified in the story and then, determine
which event violated the norm. We find that text-davinci-003 exhibits a strong
tendency for citing abnormal events as causal compared to any other models
(and to humans). We also find that smaller models (text-babbage-001, text-
curie-001, and Alpaca-7B) don’t have a tendency for abnormal over normal
events, which might indicate an inability to perform the required chained
reasoning.

##### Causality: Statistical or Prescriptive Norm

Statistical norms refer to commonly and naturally occurring events, such as
“David usually comes home around 6 pm.” or “When you flip the switch, lights
usually turn on.” Prescriptive norms refer to human-made rules, such as “You
are not allowed to run in the hallway.” or “The speed limit is 60 mph.” People
tend to find events that violate prescriptive norms to be the cause of the
outcome. Surprisingly, although GPT-4, Claude-v1, text-davinci-002, and
smaller models, such as text-curie-001 and Alpaca-7B, have similar tendencies
to humans, text-davinci-003 and its counterpart ChatGPT exhibit a strong
tendency for statistical norms. This shows that LLM implicit tendencies can
change for different training methods.

Figure 3: Moral Implicit Tendencies. Note that the Left and right figures have
different scales.

##### Morality: Benefit Self or Others

The stories in the Moral Permissibility Task often involve scenarios where
participants must make a choice: they are either safe from danger, and their
decision only affects others, or they need to save their own lives along with
other people. Humans are more likely to save themselves (self-beneficial).
Interestingly, smaller models such as text-babbage-001 and text-curie-001 find
actions more morally permissible when they are self-beneficial, while larger
models do not. RLHF makes the models more other-beneficial. For example, while
Claude-v1 seems to prefer more self-beneficial actions, text-davinci-003,
ChatGPT, and GPT-4, all prefer to be other-beneficial. Human and model
tendencies do not align on this factor, but since models are lifeless systems,
we may want models to be more other-beneficial. See Figure A2 for a clearer
comparison.

##### Morality: Inevitable or Avoidable Harm

Some stories involve situations where harm would inevitably happen to a group
of people regardless of the participant’s choice. Harm directed towards
inevitable consequences is often considered less bad. This requires the model
to comprehend the information described in the story about the counterfactual
scenario. We find that ChatGPT, Claude-v1, and GPT-4 find harmful actions
toward inevitable consequences much more morally permissible than any other
model.

##### Morality: Intervening on Agent or Patient of Harm

A hijacked airplane full of passengers is about to hit a building full of
people. The people in the hijacked airplane are referred to as agents of harm,
and the people in the building are the patients of harm. Since we compute the
confidence interval on AMCE, although some models (such as GPT-4) seem to have
a tendency to intervene on the agent of harm, the upper/lower bound of the
confidence interval shows there might not be a clear tendency (see Figure A2).

##### Morality: Personal or Impersonal Force

This factor captures whether or not the outcome was brought about by using
personal force (e.g., pushing a person). Generally, people are less likely to
find actions morally permissible that involve personal force. We find that
only two models, ChatGPT and GPT-4, align with human intuitions about personal
force.

##### Morality: Means or Side Effect

Assessing an agent’s causal role is important for moral judgments. Generally,
people find actions more morally permissible when they lead to harm as side
effects rather than when the harm is necessary to bring out the outcome. This
can be illustrated as a diagram in Figure A1. We find that no model exhibits a
clear preference for harm as a side effect.

We additionally saw the following larger trends across models:

  1. (1):

Non-monotonicity: The alignment to human biases does not necessarily increase
with model size. We speculate that alignment is an area where the inverse
scaling law applies (McKenzie et al., 2023).

  2. (2):

Heterogeneity: Interestingly, but perhaps not surprisingly, models that used
the same training method and fine-tuned for human preferences do not have the
same implicit tendencies – we highlight the difference between Claude-v1 and
GPT3.5-turbo.

  3. (3):

Self-Tendency vs. Other-Preference: Humans are ego-centric and often make
self-beneficial decisions. However, when asked to judge the behaviors of
others, we prefer others to be altruistic. For example, in scenarios involving
self-driving cars, Kallioinen et al. (2019) reported a stark contrast between
what we tend to do versus what we want other people to do. This difference
will make models trained on human preferences through RLHF different from how
humans would tend to act in a given situation. As it is more prevalent to use
LLMs as proxy humans for data labeling or experiments (Dillion et al., 2023;
Gilardi et al., 2023), it is increasingly important to understand and measure
these implicit differences.

####  4.2.3 Other Analyses

##### Same Model with Different Prompting Methods

We also investigate how different prompting methods shift a model’s implicit
tendencies. Prompting methods shifted the model’s implicit tendency
significantly for some factors (e.g., abnormal sensitivity, benefit self or
others) but not much for others (e.g., inevitable or avoidable harm). We
report this in Figure A3.

##### Predicting Factor-Attribute as Few-shot Learning

We asked LLMs to predict which factor attribute each story contains as a few-
shot natural language understanding task. Across all prediction tasks, text-
davinci-v2 achieved 85.5% accuracy on causal factors and 70.9% on moral
factors. We report more details in Appendix A.10. This shows that even though
LLMs and humans do not align, they are very capable of following examples and
instructions to annotate stories with factors that are relevant to human
judgments.

##### Hallucination

Table 4: Model Hallucination: number of stories where the model hallucinated. Model | Causal | Moral
---|---|---
text-curie-001 | 8/10 | 3/10
Claude-v1 | 2/10 | 2/10
GPT3.5-turbo | 2/10 | 0/10
GPT-4 | 0/10 | 0/10

We conduct a small-scale analysis to investigate whether the tendency
difference is due to actual tendency difference or model hallucinations. We
annotated 80 examples by sampling 10 examples where 4 models made mistakes
across 2 tasks. We prompt the model to explain why they made the choice for
the original story. We use the explanation to check if the model hallucinates
and has an accurate grasp of the facts represented in the story. A model
hallucinates when they re-state the core story in a way that’s inconsistent
with the facts provided in the story. For example, if an action is not
performed by character A, and the model thinks it is performed by character A,
we count this as a hallucination. In Table 4, we can clearly see that when a
model makes a mistake, smaller models tend to hallucinate more.

We read the model explanations on examples where no hallucination is found,
and we conclude that larger models indeed have a different tendency compared
to humans in these stories. On moral stories, Claude-v1, GPT3.5-turbo, and
GPT-4 all seem to be bound by pre-entered moral principles and choose the same
action regardless of the story circumstances (potentially due to the influence
of AI constitution; Bai et al., 2022b), while humans tend to consider a few
more factors (i.e., nuances in the story). For example, when faced with
intricate situations, language models tend to take a very passive stance. This
is true even if the action could save a greater number of people by putting a
smaller number of people in danger. However, when self-sacrifice is in
question, the language model is inclined to metaphorically “sacrifice” itself.
Our observation is based on a limited set of examples. We leave more extended
evaluation to future work.

##  5 Conclusion

We summarized the main findings of 24 cognitive science papers around human
intuitions on causal and moral judgments. We collected a dataset of causal and
moral judgment tasks and used the scientific findings to enrich the stories
with structural annotations. Using conjoint analysis, we compute the average
marginal component effect (AMCE) for people’s and the models’ implicit
tendency on the factors important to making causal and moral judgments. We
find subtle differences in what humans and models prefer. Our work illuminates
the importance of using a richly annotated dataset to understand a model’s
systematic tendencies, an important step to understanding where models and
humans align or misalign.

## Acknowledgments

Research reported in this paper was supported in part by a Hoffman-Yee grant
and a HAI seed grant. We would like to thank Paul Henne for his assistance in
curating high-impact causal reasoning literature. We thank Lucy Li for writing
the ethical consideration section for this paper. We would also like to thank
Jon Gauthier, Nicholas Tomlin, Matthew Jorke, Xuechen Li, Sameer Jha, Rose E
Wang, and Dora Demszky for general discussions. An earlier version of the
dataset was submitted to BigBench. We would like to thank the reviewers
Marcelo Menegali, Adrià Garriga-Alonso, and Denis Emelin for their comments
and feedback. We thank May Khoo for data annotations and facilitation of
labeling. We thank Simon Lee Huang for his work to run experiments with
Automated Prompt Engineer and Social Simulacra.

## References

  * Alicke et al. (2011) Mark D Alicke, David Rose, and Dori Bloom.  Causation, norm violation, and culpable control.  _The Journal of Philosophy_ , 108(12):670–696, 2011.
  * Almeida et al. (2023) Guilherme FCF Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo.  Exploring the psychology of gpt-4’s moral and legal reasoning.  _arXiv preprint arXiv:2308.01264_ , 2023.
  * Awad et al. (2018) Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan.  The moral machine experiment.  _Nature_ , 563(7729):59–64, 2018.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Baker et al. (2017) Chris L. Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing.  _Nature Human Behaviour_ , 1(4):0064, 2017.  doi: 10.1038/s41562-017-0064.  URL https://doi.org/10.1038%2Fs41562-017-0064.
  * Baron & Ritov (2004) Jonathan Baron and Ilana Ritov.  Omission bias, individual differences, and normality.  _Organizational Behavior and Human Decision Processes_ , 94(2):74–85, 2004.
  * Battaglia et al. (2013) Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum.  Simulation as an engine of physical scene understanding.  _Proceedings of the National Academy of Sciences_ , 110(45):18327–18332, 2013.
  * Bloomfield (2007) Paul Bloomfield.  _Morality and Self-interest_.  Oxford University Press, 2007.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.  COMET: Commonsense transformers for automatic knowledge graph construction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4762–4779, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1470.  URL https://aclanthology.org/P19-1470.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Buch et al. (2022) Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles.  Revisiting the" video" in video-language understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2917–2927, 2022.
  * Cave et al. (2019) Stephen Cave, Rune Nyrup, Karina Vold, and Adrian Weller.  Motivations and risks of machine ethics.  _Proceedings of the IEEE_ , 107(3):562–574, 2019\.  doi: 10.1109/JPROC.2018.2865996.
  * Christensen et al. (2014) Julia F Christensen, Albert Flexas, Margareta Calabrese, Nadine K Gut, and Antoni Gomila.  Moral judgment reloaded: a moral dilemma validation study.  _Frontiers in psychology_ , 5:607, 2014.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_ , 2020.
  * Clarke et al. (2015) Randolph Clarke, Joshua Shepherd, John Stigall, Robyn Repko Waller, and Chris Zarpentine.  Causation, norms, and omissions: A study of causal judgments.  _Philosophical Psychology_ , 28(2):279–293, 2015\.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ , 10:92–110, 2022.
  * Davis & Marcus (2015) Ernest Davis and Gary Marcus.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  _Communications of the ACM_ , 58(9):92–103, 2015\.
  * DeScioli et al. (2011) Peter DeScioli, Rebecca Bruening, and Robert Kurzban.  The omission effect in moral cognition: Toward a functional explanation.  _Evolution and Human Behavior_ , 32(3):204–215, 2011.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Dillion et al. (2023) Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray.  Can ai language models replace human participants?  _Trends in Cognitive Sciences_ , 2023.
  * Eatherly & Anders (2015) Claude Eatherly and Günther Anders.  _Burning Conscience: The Case Of The Hiroshima Pilot Claude Eatherly_.  Pickle Partners Publishing, 2015.
  * Eberhardt (2020) Jennifer L Eberhardt.  _Biased: Uncovering the hidden prejudice that shapes what we see, think, and do_.  Penguin Books, 2020.
  * Emelin et al. (2021) Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi.  Moral stories: Situated reasoning about norms, intents, actions, and their consequences.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 698–718, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.54.  URL https://aclanthology.org/2021.emnlp-main.54.
  * Foot (1967) Philippa Foot.  The problem of abortion and the doctrine of the double effect.  _Oxford review_ , 5, 1967.
  * Forbes et al. (2020) Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi.  Social chemistry 101: Learning to reason about social and moral norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 653–670, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.48.  URL https://aclanthology.org/2020.emnlp-main.48.
  * Gabriel (2020) Iason Gabriel.  Artificial intelligence, values, and alignment.  _Minds and machines_ , 30(3):411–437, 2020.
  * Gerstenberg & Icard (2020) Tobias Gerstenberg and Thomas F. Icard.  Expectations affect physical causation judgments.  _Journal of Experimental Psychology: General_ , 149(3):599–607, 2020.
  * Gerstenberg & Lagnado (2012) Tobias Gerstenberg and David A. Lagnado.  When contributions make a difference: Explaining order effects in responsibility attributions.  _Psychonomic Bulletin & Review_, 19(4):729–736, 2012.
  * Gerstenberg & Stephan (2021) Tobias Gerstenberg and Simon Stephan.  A counterfactual simulation model of causation by omission.  _Cognition_ , 2021.  URL https://psyarxiv.com/wmh4c/.
  * Gerstenberg & Tenenbaum (2017) Tobias Gerstenberg and Joshua B. Tenenbaum.  Intuitive theories.  In Michael Waldmannn (ed.), _Oxford Handbook of Causal Reasoning_ , pp. 515–548. Oxford University Press, 2017.
  * Gerstenberg et al. (2021) Tobias Gerstenberg, Noah D. Goodman, David A. Lagnado, and Joshua B. Tenenbaum.  A counterfactual simulation model of causal judgments for physical events.  _Psychological Review_ , 128(6):936–975, 2021\.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_ , 2023.
  * Green (2021) Ben Green.  Data science as political action: Grounding data science in a politics of justice.  _Journal of Social Computing_ , 2(3):249–265, 2021.  doi: 10.23919/JSC.2021.0029.
  * Greene et al. (2009) Joshua D Greene, Fiery A Cushman, Lisa E Stewart, Kelly Lowenberg, Leigh E Nystrom, and Jonathan D Cohen.  Pushing moral buttons: The interaction between personal force and intention in moral judgment.  _Cognition_ , 111(3):364–371, 2009.
  * Hainmueller et al. (2014) Jens Hainmueller, Daniel J Hopkins, and Teppei Yamamoto.  Causal inference in conjoint analysis: Understanding multidimensional choices via stated preference experiments.  _Political analysis_ , 22(1):1–30, 2014.
  * Halpern (2015) Joseph Y Halpern.  Cause, responsibility and blame: a structural-model approach.  _Law, probability and risk_ , 14(2):91–118, 2015\.
  * Hart & Honoré (1959/1985) H. L. A. Hart and T. Honoré.  _Causation in the law_.  Oxford University Press, New York, 1959/1985.
  * Hauser (2006) Marc Hauser.  _Moral minds: How nature designed our universal sense of right and wrong._ Ecco/HarperCollins Publishers, 2006.
  * Hendrickx et al. (2010) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz.  SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pp. 33–38, Uppsala, Sweden, July 2010. Association for Computational Linguistics.  URL https://aclanthology.org/S10-1006.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.  Aligning ai with shared human values.  _arXiv preprint arXiv:2008.02275_ , 2020.
  * Hendrycks et al. (2021) Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt.  What would jiminy cricket do? towards agents that behave morally.  _arXiv preprint arXiv:2110.13136_ , 2021.
  * Henne et al. (2017) Paul Henne, Ángel Pinillos, and Felipe De Brigard.  Cause by omission and norm: Not watering plants.  _Australasian Journal of Philosophy_ , 95(2):270–283, 2017.
  * Henne et al. (2019) Paul Henne, Laura Niemi, Ángel Pinillos, Felipe De Brigard, and Joshua Knobe.  A counterfactual explanation for the action effect in causal judgment.  _Cognition_ , 190:157–164, 2019.
  * Henne et al. (2021) Paul Henne, Aleksandra Kulesza, Karla Perez, and Augustana Houcek.  Counterfactual thinking and recency effects in causal judgment.  _Cognition_ , 212:104708, 2021.
  * Hilton & Slugoski (1986) D. J. Hilton and B. R. Slugoski.  Knowledge-based causal attribution: The abnormal conditions focus model.  _Psychological Review_ , 93(1):75–88, 1986.
  * Hilton et al. (2010) D. J. Hilton, J. McClure, and R. M. Sutton.  Selecting explanations from causal chains: Do statistical principles explain preferences for voluntary causes.  _European Journal of Social Psychology_ , 40(3):383–400, 2010.
  * Hitchcock & Knobe (2009) Christopher Hitchcock and Joshua Knobe.  Cause and norm.  _The Journal of Philosophy_ , 106(11):587–612, 2009.
  * Huebner et al. (2011) Bryce Huebner, Marc D Hauser, and Phillip Pettit.  How the source, inevitability and means of bringing about harm interact in folk-moral judgments.  _Mind & Language_, 26(2):210–233, 2011.
  * Icard et al. (2017) Thomas F Icard, Jonathan F Kominsky, and Joshua Knobe.  Normality and actual causal strength.  _Cognition_ , 161:80–93, 2017.
  * Jiang et al. (2021) Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi.  Delphi: Towards machine ethics and norms.  _arXiv preprint arXiv:2110.07574_ , 2021.
  * Jin et al. (2022) Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf.  When to make exceptions: Exploring language models as accounts of human moral judgment.  _arXiv preprint arXiv:2210.01478_ , 2022.
  * Kallioinen et al. (2019) Noa Kallioinen, Maria Pershina, Jannik Zeiser, Farbod Nosrat Nezami, Gordon Pipa, Achim Stephan, and Peter König.  Moral judgements on the actions of self-driving cars and human drivers in dilemma situations from different perspectives.  _Frontiers in psychology_ , 10:2415, 2019.
  * Kenton et al. (2021) Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving.  Alignment of language agents.  _arXiv preprint arXiv:2103.14659_ , 2021.
  * Kıcıman et al. (2023) Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan.  Causal reasoning and large language models: Opening a new frontier for causality.  _arXiv preprint arXiv:2305.00050_ , 2023.
  * Kleiman-Weiner et al. (2015) Max Kleiman-Weiner, Tobias Gerstenberg, Sydney Levine, and Joshua B Tenenbaum.  Inference of intention and permissibility in moral decision making.  In _CogSci_ , 2015.
  * Knobe & Fraser (2008) Joshua Knobe and Ben Fraser.  Causal judgment and moral judgment: Two experiments.  _Moral psychology_ , 2:441–447, 2008.
  * Kominsky & Phillips (2019) Jonathan F Kominsky and Jonathan Phillips.  Immoral professors and malfunctioning tools: Counterfactual relevance accounts explain the effect of norm violations on causal selection.  _Cognitive science_ , 43(11):e12792, 2019.
  * Kominsky et al. (2015) Jonathan F Kominsky, Jonathan Phillips, Tobias Gerstenberg, David Lagnado, and Joshua Knobe.  Causal superseding.  _Cognition_ , 137:196–209, 2015.
  * Kosoy et al. (2022) Eliza Kosoy, David M Chan, Adrian Liu, Jasmine Collins, Bryanna Kaufmann, Sandy Han Huang, Jessica B Hamrick, John Canny, Nan Rosemary Ke, and Alison Gopnik.  Towards understanding how machines can learn causal overhypotheses.  _arXiv preprint arXiv:2206.08353_ , 2022.
  * Lagnado et al. (2013) D. A. Lagnado, T. Gerstenberg, and R. Zultan.  Causal responsibility and counterfactuals.  _Cognitive Science_ , 47:1036–1073, 2013.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40, 2017.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ , 2019.
  * Liao et al. (2012) S Matthew Liao, Alex Wiegmann, Joshua Alexander, and Gerard Vong.  Putting the trolley in order: Experimental philosophy and the loop case.  _Philosophical Psychology_ , 25(5):661–671, 2012\.
  * Liu et al. (2019a) Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, and Zhongjun He.  Robust neural machine translation with joint textual and phonetic embedding.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 3044–3049, Florence, Italy, July 2019a. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1291.  URL https://aclanthology.org/P19-1291.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019b.
  * Lourie et al. (2021) Nicholas Lourie, Ronan Le Bras, and Yejin Choi.  Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 13470–13479, 2021.
  * Mandel (2003) David R Mandel.  Judgment dissociation theory: An analysis of differences in causal, counterfactual and covariational reasoning.  _Journal of Experimental Psychology: General_ , 132(3):419, 2003.
  * McKenzie et al. (2023) Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al.  Inverse scaling: When bigger isn’t better.  _arXiv preprint arXiv:2306.09479_ , 2023.
  * Mikhail (2007) John Mikhail.  Universal moral grammar: Theory, evidence and the future.  _Trends in cognitive sciences_ , 11(4):143–152, 2007.
  * Moore et al. (2008) Adam B Moore, Brian A Clark, and Michael J Kane.  Who shalt not kill? individual differences in working memory capacity, executive control, and moral judgment.  _Psychological science_ , 19(6):549–557, 2008\.
  * Morris et al. (2019) Adam Morris, Jonathan Phillips, Tobias Gerstenberg, and Fiery Cushman.  Quantitative causal selection patterns in token causation.  _PloS one_ , 14(8):e0219704, 2019.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 839–849, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-1098.  URL https://aclanthology.org/N16-1098.
  * N’gbala & Branscombe (1995) Ahogni N’gbala and Nyla R Branscombe.  Mental simulation and causal attribution: When simulating an event does not affect fault assignment.  _Journal of Experimental Social Psychology_ , 31(2):139–162, 1995.
  * O’Neill et al. (2021) Kevin O’Neill, Paul Henne, Paul Bello, John Pearson, and Felipe De Brigard.  Degrading causation.  2021\.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  _arXiv_ , 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Pan et al. (2023) Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks.  Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark.  In _International Conference on Machine Learning_ , pp. 26837–26867. PMLR, 2023.
  * Park et al. (2022) Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein.  Social simulacra: Creating populated prototypes for social computing systems.  In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ , pp. 1–18, 2022.
  * Pearl & Mackenzie (2018) Judea Pearl and Dana Mackenzie.  _The book of why: the new science of cause and effect_.  Basic books, 2018.
  * Quillien & Lucas (2023) Tadeg Quillien and Christopher G Lucas.  Counterfactuals and the logic of causal selection.  _Psychological Review_ , 2023.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Reinecke et al. (2023) Madeline G Reinecke, Yiran Mao, Markus Kunesch, Edgar A Duéñez-Guzmán, Julia Haas, and Joel Z Leibo.  The puzzle of evaluating moral cognition in artificial agents.  _Cognitive Science_ , 47(8):e13315, 2023.
  * Reuter et al. (2014) Kevin Reuter, Lara Kirfel, Raphael Van Riel, and Luca Barlassina.  The good, the bad, and the timely: how temporal order and moral judgment influence causal selection.  _Frontiers in psychology_ , 5:1336, 2014.
  * Ritov & Baron (1992) Ilana Ritov and Jonathan Baron.  Status-quo and omission biases.  _Journal of Risk and Uncertainty_ , 5(1), 1992.  doi: 10.1007/BF00208786.  URL http://dx.doi.org/10.1007/BF00208786.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_ , 2011.
  * Samland et al. (2016) Jana Samland, Marina Josephs, Michael R Waldmann, and Hannes Rakoczy.  The role of prescriptive norms and knowledge in children’s and adults’ causal selection.  _Journal of Experimental Psychology: General_ , 145(2):125, 2016.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33-01, pp. 3027–3035, 2019.
  * Scherrer et al. (2023) Nino Scherrer, Claudia Shi, Amir Feder, and David M Blei.  Evaluating the moral beliefs encoded in llms.  _arXiv preprint arXiv:2307.14324_ , 2023.
  * Sloman & Lagnado (2015) Steven A Sloman and David Lagnado.  Causality in thought.  _Annual review of psychology_ , 66:223–247, 2015.
  * Spellman (1997) B. A. Spellman.  Crediting causality.  _Journal of Experimental Psychology: General_ , 126(4):323–348, 1997.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Sytsma (2021) Justin Sytsma.  Crossed wires: Blaming artifacts for bad outcomes.  2021\.
  * Talat et al. (2021) Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams.  A word on machine ethics: A response to jiang et al. (2021), 2021.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1421.  URL https://aclanthology.org/N19-1421.
  * Thomson (1976) Judith Jarvis Thomson.  Killing, letting die, and the trolley problem.  _The Monist_ , 59(2):204–217, 1976.
  * Tsividis et al. (2021) Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum.  Human-level reinforcement learning through theory-based modeling, exploration, and planning.  _arXiv preprint arXiv:2107.12544_ , 2021.
  * Ullman et al. (2017) Tomer D. Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B. Tenenbaum.  Mind games: Game engines as an architecture for intuitive physics.  _Trends in Cognitive Sciences_ , 21(9):649–665, 2017.  doi: 10.1016/j.tics.2017.05.012.  URL https://doi.org/10.1016%2Fj.tics.2017.05.012.
  * Waldmann & Dieterich (2007) Michael R Waldmann and Jörn H Dieterich.  Throwing a bomb on a person versus throwing a person on a bomb: Intervention myopia in moral intuitions.  _Psychological science_ , 18(3):247–253, 2007\.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_ , 2019.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022b.
  * Wellman & Gelman (1992) H. M. Wellman and S. A. Gelman.  Cognitive development: Foundational theories of core domains.  _Annual Review of Psychology_ , 43(1):337–375, 1992.
  * Wells & Gavanski (1989) Gary L Wells and Igor Gavanski.  Mental simulation of causality.  _Journal of personality and social psychology_ , 56(2):161, 1989.
  * Wiegmann et al. (2012) Alex Wiegmann, Yasmina Okan, and Jonas Nagel.  Order effects in moral judgment.  _Philosophical Psychology_ , 25(6):813–836, 2012\.
  * Wolff (2007) Phillip Wolff.  Representing causation.  _Journal of experimental psychology: General_ , 136(1):82, 2007.
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_ , 2022.
  * Ziems et al. (2022) Caleb Ziems, Jane A Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang.  The moral integrity corpus: A benchmark for ethical dialogue systems.  _arXiv preprint arXiv:2204.03021_ , 2022.

##  Appendix A Appendix

### Ethical considerations

It is imperative that we assess implicit intuitions underlying commonsense
reasoning abilities in LLMs capable of open language generation. This is
especially important for cases related to morality, considering that
algorithmic neutrality is not possible: systems reflect the data they ingest
and the people who build them (Green, 2021). In addition, even if a model is
not explicitly given the responsibility to make moral judgments, these
judgments can appear across many forms of freely generated text. Some may
argue that equipping models with moral reasoning skills can allow them to
behave better in complex environments, but an AI capable of reasoning is not
necessarily bound to be ethically well-aligned, as many humans themselves have
demonstrated (Cave et al., 2019; Talat et al., 2021). Using carefully curated
data, we advocate for the inspection of how choices are made and whether
factors used in human cognitive processes are or can be incorporated by
models. In the case of our moral permissibility task, we would like it to be a
dataset for investigating these underlying factors rather than a flat
benchmark to beat. Whether alignment between LLMs and human participants is a
desired goal in this context is also unclear. For example, there are well-
documented biases that affect people’s moral intuitions, and we wouldn’t want
to replicate these biases in LLMs (Eberhardt, 2020). However, we argue that
despite these challenges and difficulties, our research, which focuses on
utilizing the insights and frameworks proposed by cognitive science and
philosophy to analyze and scaffold LLMs to generate more human-aligned
judgments, is a small step forward in a promising direction.

###  A.1 Factors in Causal Selection Task

##### Causal Structure

In a situation where multiple events are present in a story, and the outcome
is caused by one or more of these events, it’s important to understand the
causal structure of the situation: Is one event sufficient to bring about the
outcome (disjunctive causes) or are all of the considered events required
(conjunctive causes)? (Wells & Gavanski, 1989; Mandel, 2003; Sloman & Lagnado,
2015).

##### Event Normality

People have a general tendency to cite abnormal events (rather than normal
events) as causes (Knobe & Fraser, 2008; Hitchcock & Knobe, 2009; Alicke et
al., 2011; O’Neill et al., 2021; Morris et al., 2019). This effect is present
for both children and adults (Samland et al., 2016). Norms can be broadly
classified into “statistical norm” (norms about what tends to be) or
“prescriptive norm” (norms about what should be) (Icard et al., 2017; N’gbala
& Branscombe, 1995; Kominsky et al., 2015). Event normality applies to
inanimate objects as well. For example, a malfunctioning machine is judged by
more people to be the cause than a machine that functions as it should (often
called the “norm of proper functioning”) (Sytsma, 2021).

##### Agent knowledge

The epistemic state of the agent also plays an important role in how people
judge whether the agent is the cause of the outcome. For example, an agent who
is aware of the potential consequences of their action and knowingly performs
an action that leads to a foreseeable bad outcome is judged more harshly than
an agent who lacks the relevant knowledge. This is often characterized as
knowledge versus ignorance (Samland et al., 2016; Kominsky & Phillips, 2019).

Table A1: Factors that influence causal selection judgments (left) and moral
permissibility judgments (right). We provide a list of simplified definitions
for each tag in the story. We also added references to papers that primarily
investigate these factors. We refer to higher-level concepts (e.g., “Causal
Role”) as factors and the possible values for each factor (e.g., “Means” or
“Side Effect”) as tags.

Causal selection
---
Factors |  Definitions
| Causal
---
Structure
(Wells & Gavanski, 1989; Mandel, 2003; Sloman & Lagnado, 2015)
Conjunctive |  All events must happen in order for the outcome to occur. Each event is a necessary cause for the outcome.
Disjunctive |  Any event will cause the outcome to occur. Each event is a sufficient cause for the outcome.
| Agent
---
Awareness
(Samland et al., 2016; Kominsky & Phillips, 2019)
Aware |  Agent is aware that their action will break the norm/rule or they know their action is “abnormal”.
Unaware |  Agent is unaware or ignorant that their action will break the norm/rule, or they don’t know their action is “abnormal”.
Norm Type |  (Icard et al., 2017; N’gbala & Branscombe, 1995; Kominsky et al., 2015; Sytsma, 2021)
Prescriptive Norm |  A norm about what is supposed to happen.
Statistical Norm |  A norm about what tends to happen.
| Event
---
Normality
(Knobe & Fraser, 2008; Hitchcock & Knobe, 2009; Alicke et al., 2011; O’Neill
et al., 2021; Morris et al., 2019; Samland et al., 2016)
Normal Event |  The event that led to the outcome is considered “normal”.
Abnormal Event |  The event that led to the outcome is considered “abnormal/unexpected”.
| Action
---
or
Omission
(Ritov & Baron, 1992; Baron & Ritov, 2004; Henne et al., 2017, 2019; Clarke et
al., 2015; DeScioli et al., 2011; Gerstenberg & Stephan, 2021)
Action as Cause |  Agent performed an action that led to the outcome.
Omission as Cause |  Agent did not perform the action, and the omission led to the outcome.
Time |  (Reuter et al., 2014; Henne et al., 2021)
Early |  The event happened early.
Late |  The event happened late.
Same Time |  Multiple events happened at the same time.

Moral permissibility
---
Factors |  Definitions
Causal Role |  (Hauser, 2006; Christensen et al., 2014; Wiegmann et al., 2012)
Means |  The harm is instrumental/necessary to produce the outcome.
Side Effect |  The harm happens as a side-effect of the agent’s action, unnecessary to produce the outcome.
Personal Force |  (Christensen et al., 2014; Wiegmann et al., 2012)
Personal |  Agent is directly involved in the production of the harm.
Impersonal |  Agent is only indirectly involved in the process that results in the harm (e.g., using a device).
| Counterfactual
---
Evitability
(Moore et al., 2008; Huebner et al., 2011; Christensen et al., 2014; Wiegmann
et al., 2012)
Avoidable |  The harm would not have occurred if the agent hadn’t acted.
Inevitable |  The harm would have occurred even if the agent hadn’t acted.
Beneficence |  (Bloomfield, 2007; Christensen et al., 2014; Wiegmann et al., 2012)
Self-Beneficial |  The agent themself benefits from their action.
Other-Beneficial |  Only other people benefit from the agent’s action.
| Locus of
---
Intervention
(Waldmann & Dieterich, 2007)
Instrument of Harm |  The intervention is directed at the instrument of harm (e.g., the runaway train, the hijacked airplane).
Patient of Harm |  The intervention is directed at the patient of harm (e.g., the workers on the train track).

##### Action/omission

People tend to select an action as the cause rather than inaction. This
phenomenon is often called the “omission bias” (Ritov & Baron, 1992; Baron &
Ritov, 2004). In a scenario where one person acts, and another one doesn’t,
the person who acted tends to be cited as the cause of the outcome (Henne et
al., 2017, 2019; Clarke et al., 2015; DeScioli et al., 2011; Gerstenberg &
Stephan, 2021).

##### Temporal effect

Causal selections are also affected by the order in which the events occur
(Gerstenberg & Lagnado, 2012). When several events unfolding over time lead to
an outcome, people have a general tendency to select later events rather than
earlier events as the actual cause of the outcome (Reuter et al., 2014).
However, it also depends on how the events are causally related to one another
(Hilton et al., 2010; Spellman, 1997). When earlier events determine the
course of action, these events tend to be selected (Henne et al., 2021).

###  A.2 Factors in Moral Permissibility Task

Figure A1: Causal Role We show the causal diagram of the difference between
means and side effects. A represents action conducted by the agent. H
represents the harm. O represents the outcome. (a) is the diagram that
represents harm as a side effect, and (b) represents harm as a means.

##### Causal Role

Assessing an agent’s causal role is important for moral judgments. Actions are
more likely to be seen as permissible when they didn’t play an important
causal role in how the negative outcome came about. One important distinction
is between an event as a means versus a side effect of the outcome
(illustrated in Figure A1). In Figure A1a, the action (A) leads to a desired
outcome (O) but also has the side effect of harming someone (H). In Figure
A1b, the harm is a means to bringing about the outcome. Generally, people find
actions more morally permissible when they lead to harm as side effects rather
than when the harm is a means for bringing about the outcome.

##### Personal Force

This factor captures whether or not the outcome was brought about by the use
of personal force (e.g., pushing a person). Generally, people are less likely
to find actions morally permissible that involve personal force.

This is also a factor that connects causal judgment and moral judgment – in
causal judgment, a normative framework that describes whether a cause will be
selected by human tendency is through the judgment and observation of force
transfer (Wolff, 2007). Moral philosophy that studied the mass murders during
the Second World War has suggested that killing via a mechanism (e.g., “press
a button”) is much easier than actively involved in the killing acts (Eatherly
& Anders, 2015).

##### Counterfactual Evitability

Could the harm have been avoided? In the story where the teenager was going to
trigger the bomb, which would kill him and the others in the building, in a
counterfactual world, had the agent not killed the teenager, they would have
died from the explosion they caused anyway. Harm directed towards inevitable
consequences is often considered less evil (Hauser, 2006).

##### Beneficence

People are more likely to take actions that benefit themselves (Bloomfield,
2007). In the stories, participants were not asked to choose themselves over
others; instead, the framing often is they are either safe from danger, and
their decision only affects others, or they need to save their own lives along
with other people.

##### Locus of Intervention

People are often asked to choose to intervene on the instrument of harm or
recipient of harm. For example, a hijacked airplane with passengers can crash
into a building full of people. If we use a missile to shoot down the hijacked
airplane, killing all the passengers onboard, then we would be intervening on
the instrument of harm (the plane). If we choose to misdirect the hijacked
airplane’s navigation system to hit another smaller building also filled with
fewer people, then we would be intervening on the patient of harm. Presented
with this type of choice, human tendency is usually to intervene on the
instrument of harm (Waldmann & Dieterich, 2007).

###  A.3 Original Story Experiment Details

We conduct a 3-class comparison, a binarized response of “Yes” or “No”, and an
additional class of “ambiguous” when the agreement between human or model
output is 50% ± 10%. We test alignment using a prompt-based zero-shot task
setup. For each model and scenario, we use the normalized probability and
compare $P(\text{``Yes''}|\text{Story + Prompt})$ and
$P(\text{``No''}|\text{Story + Prompt})$. We apply temperature-based sampling
to generate samples of “Yes” or “No” answers per story for each experimental
run. For models that are trained with a masked language model objective, we
append a mask token after the prompt question. For a generative language model
like GPT2, we directly ask it to produce the next word. We also included two
models for which we didn’t have direct access – GPT-3 (Brown et al., 2020) and
Delphi (Jiang et al., 2021). We used their APIs to evaluate alignment.

For Chat-based APIs (Alpaca-7B, Anthropic-claude-v1, GPT3.5-turbo, GPT-4),
since the probability of each token is not provided, we first experimented
with sampling multiple times from a model with a high temperature. We sampled
10 responses for a few stories to see if we have any variation in generated
responses – to our surprise, we always get the same response. After this
initial exploration, we only generate one response per story and set $P=1$ for
the chosen response. In our prompt, we tell the chat-based models that they
can output “not sure” if they are not certain. When “not sure” is found, we
assign $P=0.5$ for either choice.

###  A.4 Social Simulacra Experiment Details

There are 50 simulacra (community) in total provided by Park et al. (2022);
each simulacra has 200 persona prompts. The persona in each simulacrum roughly
shares a common theme. Some of such themes include “A community for discussing
Yu-Gi-Oh! Master Duel!” or “A community for discussing Disney’s movie,
Encanto.”

Most of the simulacra themes are highly irrelevant to our task domain and
evaluating all simulacra personas are cost prohibitive. Therefore, we choose 5
simulacra that are most relevant to us – they contain personas that usually
describe ideological or political perspectives. We then randomly choose 5
personas out of 200 personas in each simulacrum. We show our selection in
Table A3. You can access the complete list of 200 personas, such as personas
listed in LuxuryLifeHabits here111https://social-
simulacra.herokuapp.com/LuxuryLifeHabits#simulacra.

For both models, the persona that most closely aligned with our collected
responses for Causal Judgment Task is “Emily White is a Republican who got fed
up with the party”, and the least aligned persona is “Angela Campbell is a
woman who likes to party until the sun comes up”. For Moral Permissibility
Task, the most aligned persona for both models is “Allen Lee is a really cool
guy”, and the least aligned persona is “Paul Brown is an anti-vaccine
activist”.

Table A2: The full zero-shot performance of two GPT3.5 models with Social
Simulacra and Prompt Optimization methods.

(a) Zero-shot with Social Simulacra and Prompt Optimization (text-davinci-002)

| Causal Judgment | Moral Permissibility
---|---|---
Methods |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$) |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$)
GPT3.5-davinci-v2 |  37.8$\pm$5.6 | 0.61 | 0.31 | 0.72 |  32.3$\pm$8.1 | 0.67 | 0.30 | 0.74
Persona (average) |  39.7$\pm$1.1 | 0.60 | 0.31 | 0.73 |  28.9$\pm$1.6 | 0.55 | 0.33 | 0.86
Persona (best) |  42.0$\pm$5.9 | 0.60 | 0.30 | 0.71 |  37.1$\pm$8.5 | 0.58 | 0.27 | 0.72
Persona (worst) |  36.8$\pm$5.9 | 0.62 | 0.29 | 0.61 |  18.5$\pm$6.9 | 0.58 | 0.46 | 0.71
Auto Prompt Engineer |  38.5$\pm$5.8 | 0.60 | 0.36 | 0.89 |  40.4$\pm$8.8 | 0.46 | 0.28 | 0.63

(b) Zero-shot with Social Simulacra and Prompt Optimization (text-davinci-003)

| Causal Judgment | Moral Permissibility
---|---|---
Methods |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$) |  Agg ($\uparrow$) |  AUC ($\uparrow$) |  MAE ($\downarrow$) |  CE ($\downarrow$)
GPT3.5-davinci-v3 |  41.3$\pm$5.6 | 0.67 | 0.41 | 1.39 |  20.2$\pm$7.3 | 0.61 | 0.51 | 2.61
Persona (average) |  39.6$\pm$1.1 | 0.60 | 0.31 | 1.56 |  28.8$\pm$1.6 | 0.55 | 0.33 | 0.86
Persona (best) |  42.7$\pm$5.6 | 0.60 | 0.31 | 0.71 |  37.1$\pm$8.5 | 0.57 | 0.28 | 0.72
Persona (worst) |  37.2$\pm$5.4 | 0.62 | 0.29 | 0.61 |  18.5$\pm$6.5 | 0.58 | 0.46 | 0.71
Auto Prompt Engineer |  40.6$\pm$5.8 | 0.64 | 0.41 | 1.55 |  33.3$\pm$8.3 | 0.50 | 0.47 | 1.62

Table A3: List of sampled persona from the Social Simulacra Dataset Category | Persona Prompt
---|---
insaneProtestors | Amy Hsuis a lawyer with a former history working with social movements
Jared Cox is a firefighter who is married and has 3 children
Ryan Andrews is a photographer who likes to go on adventures
Tom Smith is a teacher who is preparing the next generation
Jane Smith is a liberal activist
luxuryLifeHabits | Jake Williams is a person who has been abused
Polly Wangerin is a feminist and a democratic socialist
Jordan Sanders is a woman who supports Medicare for All
Emily White is a Republican who got fed up with the party
Nadia Rasheedis a scientist who is really good at science
newDealAmerica | Nia Ellis is a Bernie Sanders supporter; a woman of color
Kayla James is a person who studied biology
Sarah Lee is a Bernie Sanders supporter who loves dogs
Jon Blair is the Director of the Office of LGBT Affairs
Sam Thompson is a retired union worker and Hillary Clinton supporter
nonPoliticalTwitter | Angela Campbell is a woman who likes to party until the sun comes up
Liz Akins is a mom who wants to know how the plant will help her son
Tristan Wright is a person who likes to make new friends
Anthony Woods is a man who likes to meditate
Allen Lee is a really cool guy
politicsPeopleTwitter | Suzy Han is a doctor trying to give mental health support to the alt-right
Dennis Johnson is a local minister fighting against misinformation
Paul Brown is an anti-vaccine activist
Ann Watson is a law student that has been trying to spread political awareness
Sam Thompson is a student trying to expose right-wing lies

###  A.5 Automatic Prompt Engineer Experiment Details

We use the official implementation of Automatic Prompt
Engineer222https://github.com/keirp/automatic_prompt_engineer. We asked LLMs
to generate 200 prompts for GPT3.5-davinci-v2 and GPT3.5-davinci-v3.

Automatic Prompt Engineer provides us with a few default templates for
engineering prompts. These prompts are fine if the prompt you’re trying to
engineer can be described as an imperative statement, like “produce X” or
“generate Y.” However, in our experiment, these default templates do not lead
to any good prompts. Inspired by the Social Simulacra experiment results, we
switched to a more persona-like template.

> “James gave these answers to the following questions: [EXAMPLES] James is a
> [PROMPT]”

We chose “James” because it is a common first name. We are aware that LLMs
have implicit biases towards gendered names, and we suspect the generated
prompt will be different if the name changes. We leave a systematic
exploration of bias in automatic prompt generation for future work.

In this prompt, [Examples] is replaced with input-output pairs from the
dataset in the format of the default demos template, which is: [Story] Output:
[Answer]. After the prompt has been generated, APE evaluates the quality of
the prompt by the following template:

> “James is a [PROMPT]. James is given this prompt: "[STORY]" James is asked
> to please answer Yes or No. James answers: [Answer]”

We randomly sampled 5 causal and moral stories to use as training examples and
excluded them from the experiment result. These stories are indices 1, 8, 23,
29, 112 for Causal Judgment Task and indices 0, 7, 28, 49, 61 for Moral
Permissibility Task. For evaluation and numbers reported in 3(b), we select
the best prompt for each model and use it to generate responses for all our
stories.

We show the generated prompt and their average log-probability (the higher,
the better):

    Causal
    text-davinci-002
    ----------------
    -0.97:  very honest person, so his answers are all correct.
    -1.01:  good student, and he always gives correct answers.
    -1.02:  good student, and he always does his homework on time.
            However, he is not very good at tests.
            When he took his last math test, he got a C.
    -1.08:  very good student, and he always gives correct answers.
    -1.10:  good researcher, and he is careful to make sure that
            his answers are correct.
    -1.11:  very logical person, and he has thought about
            these questions very carefully.
            He is certain that his answers are correct.
    -1.11:  very good student, and he always gets full marks on his tests.
    -1.12:  good researcher, and he is very methodical. He is very careful
            to make sure that all of his answers are consistent with the
            information given in the question.
    -1.12:  very logical person, and he seems to be very sure
            of his answers. I would trust his answers.
    -1.13:  very literal person. He always answers questions
            with a simple yes or no, regardless of whether
            that is the most accurate answer.

    Causal
    text-davinci-003
    ----------------
    -0.87:  trustworthy source.
    -0.98:  very smart person!
    -1.25:  reliable answerer.
    -1.27:  good listener!
    -1.29:  responsible answer-giver.
    -1.34:  very smart student!
    -1.49:  reliable source of answers.
    -1.51:  reliable answer giver.
    -1.61:  reliable source of answers!
    -1.63:  reliable source of accurate answers.

    Moral
    text-davinci-002
    ----------------
    -1.11:  consequentialist.
    -1.15:  consequentialist, so he would always choose the option
            that leads to the best outcome, even if it means sacrificing
            one life to save many.
    -1.17:  consequentialist, meaning he believes that the morally right
            thing to do is the one that will produce the best consequences.
            In each of the scenarios above, James believes that the best
            thing to do is the one that will save the most lives.
    -1.21:  utilitarian. He believes that the morally right course of action
            is the one that results in the greatest good for the greatest
            number of people. In the first scenario, releasing one
            person from the helicopter will result in the greatest
            good for the greatest number of people
    -1.23:  utilitarian, so he would always choose the option that will result
            in the greatest good for the greatest number of people. In the first
            scenario, he would throw one person off the helicopter to save the
            other eight. In the second scenario, he would drive

    Moral
    text-davinci-003
    ----------------
    -0.00:  utilitarian and believes in maximizing overall happiness. Therefore,
            his answer to all of the questions is yes, as sacrificing the lives of
            one or a few people for the sake of saving more lives is the action that
            will bring the most happiness overall.
    -0.00:  utilitarian, so he believes that the greatest good for the greatest
            number should be pursued. Therefore, his answer to all of the questions
            is yes, because in each case sacrificing a few people will save the
            lives of many more people.
    -0.01:  utilitarian, believing that decisions should be made to produce the
            greatest good for the greatest number of people. His answers of "yes" in
            all of these scenarios reflects his belief that it is best to sacrifice
            a few individuals in order to save the many.
    -0.01:  utilitarian and believes that the most ethical action is the one that
            results in the greatest good for the greatest number of people. In each
            of these scenarios, sacrificing one person for the sake of saving more
            lives is the best option according to utilitarianism, so
    -0.01:  person who believes that the ends justify the means, as his answers
            demonstrate. He believes that sacrificing the lives of a few to save the
            lives of many is justified in certain circumstances.

###  A.6 Additional Discussion on the Utilitarian Principle in Moral Judgments

There is a difference between what prompt aligns LLMs best with humans and the
actual responses produced for each story. The prompt that makes LLMs align
best with human reasoning is “adopt a utilitarian/consequentialist framing”,
but this does not mean that human participants explicitly considered
consequentialist principles when providing judgments for the different
scenarios. Indeed, there are scenarios where human participants make judgments
that go against utilitarian principles. For example, in one of the stories,
participants are asked whether it’s morally permissible to kill 5 children of
a particular race in an orphanage in order to save hundreds of other children
in WWII. Here, participants overwhelmingly refuse to kill even if it means
that this would have resulted in the greatest number of lives saved. So, as
the reviewer suggests, while utilitarian principles certainly seem to play an
important part in human moral reasoning, it’s not the only consideration that
matters to humans.

Different people’s moral intuitions vary substantially. For some individuals,
utilitarian principles matter a lot, whereas, for other individuals,
deontological rules that prescribe what actions are right or wrong (e.g., do
not kill) have a strong impact on their judgments. Currently, our evaluation
of LLM alignment is against the aggregate of human judgments, and here it
looks like prompting the LLM to adopt a utilitarian framing leads to the best
alignment. That said, it’s possible that different prompts would help the
model to align better with different subgroups of human participants. While we
are ultimately interested in better capturing the variance in human moral and
causal intuitions, we chose not to focus on this aspect and leave it for
future work.

###  A.7 Additional Details on Average Marginal Component Effect Experiment

We discuss the three assumptions that the conjoint analysis needs to have in
order to claim the non-parametric estimator we used is an unbiased estimation
of the AMCE. The first assumption is the stability and no carryover effect. In
our context, this means the set of factor attributes from the story fully
determines how the respondents will produce their responses. Since each of our
stories has been used in previous scientific experiments and has validated
scientific hypotheses, we believe these stories were written very carefully to
highlight the factor attributes within the story. The second assumption is no
partial order effect. All the stories are independent, and we randomly sample
the stories to present to the respondents. The second assumption is thus
satisfied. The third and last assumption is the randomization of factors. This
assumption is satisfied in Awad et al. (2018) because the scenarios in their
experiment are randomly generated. We argue by collecting stories from a large
number of papers, and with each paper focusing on investigating different
scientific theories, the factors present in each story should be random if
they are drawn from different studies. However, we acknowledge that because we
are performing a meta-analysis over existing experiment designs, the stories
we collected might be implicitly biased by our selection process. So it is
possible that the AMCE we compute is a biased estimate. We compute confidence
intervals via bootstrapping on AMCE and report them in Figure A2 and Figure
A3. Any ACME effect that crosses over 0.0 should be regarded as our analysis
is inconclusive about whether the given model or human has a clear tendency
given our dataset.

Figure A2: AMCE for Different LLMs: We show the 95% bootstrapped confidence
interval on AMCE. Figure A3: AMCE for LLM with Prompting Methods: We show
that, with different prompting methods, the AMCE could change.

###  A.8 Annotation Guidelines

We provide the annotation guidelines as PDFs in our supplementary material.
The annotator has a master’s degree, and the annotation guidelines were
developed jointly by researchers (authors) affiliated with computer science
and psychology.

###  A.9 Additional Statistics on Data

We provide two figures on the distribution of P(Yes) for the stories in our
dataset. This corroborates our claim that human causal and moral reasoning are
diverse. We show the distribution in Figure A4.

(a)

(b)

Figure A4: The probability distribution of P(Yes) for aggregate human
responses of Yes and No over 25 participants for each story.

###  A.10 Factor Inference Experiment

We test if LLMs like GPT-3 have the ability to infer and assign the correct
tag for the text segments in the story. We manually write a few examples of
high-level instructions for each of the factors in Table A1. We include all
the examples we used in the appendix and provided in our code.

We find that LLMs such as GPT-3 are very good at inferring tags based on a few
examples – especially for factors that can be clearly specified by
instructions and examples. We report the performance of inferring causal and
moral factors in the story in 4(a) and 4(b). Though GPT-3 performed poorly in
the original story, on the causal factor inference task, the lowest accuracy
is 74.7%, and the highest is 97.3%. This means that even though GPT-3 is
unable to directly generate the human-matching answer for causal stories –
they can accurately recognize and assign the right tags for the text segments
with minimal guidance from an expert human.

However, there are also factors that are more difficult to clearly define for
LLMs. Generally, GPT-3 performed much worse on the moral structural inference
task (4(b)), with a performance from 57.9% to 86.0%. For example, what “causal
role” a person’s action played in a scenario is a particularly difficult
category for GPT-3 to correctly annotate.

Table A4: Factor Inference Experiment: We test if GPT-3 models can correctly
analyze a sub-segment of stories (extracted by human annotators). We run
models 5 times to compute an average score and 95% confidence intervals.

(a) Causal Factor Inference

GPT3 Size |  | Weighted
---
Average
| Action
---
Omission
Norm Type | Time |  | Agent
---
Awareness
| Causal
---
Structure
| Event
---
Normality
| (N=497) | (N=112) | (N=88) | (N=67) | (N=43) | (N=98) | (N=89)
Majority | 68.2 | 71.4 | 76.1 | 86.6 | 53.5 | 67.3 | 50.6
GPT3-babbage-v1 |  64.7 $\pm$ 3.5 |  80.5 $\pm$ 2.4 |  68.2 $\pm$ 3.7 |  35.8 $\pm$ 7.1 |  88.8 $\pm$ 3.2 |  58.6 $\pm$ 7.3 |  58.0 $\pm$ 8.2
GPT3-curie-v1 |  82.8 $\pm$ 0.8 |  88.8 $\pm$ 2.3 |  91.4 $\pm$ 1.3 |  81.8 $\pm$ 2.0 |  96.7 $\pm$ 4.8 |  76.1 $\pm$ 4.1 |  68.1 $\pm$ 2.9
GPT3.5-davinci-v2 | 85.5 $\pm$ 1.0 |  88.0 $\pm$ 2.2 | 97.3 $\pm$ 0.8 |  81.5 $\pm$ 2.1 | 97.2 $\pm$ 2.4 |  74.7 $\pm$ 3.4 | 80.0 $\pm$ 1.8

(b) Moral Factor Inference

GPT3 Size |  | Weighted
---
Average
(N=202)
| Personal
---
Force
(N=48)
| Beneficience
---
(N=48)
| Locus of
---
Intervention
(N=48)
| Causal Role
---
(N=48)
| Counterfactual
---
Evitability
(N=48)
Majority | 55.4 | 50.0 | 54.2 | 60.0 | 62.5 | 54.2
GPT3-babbage-v1 |  51.9 $\pm$ 2.5 |  56.7 $\pm$ 5.6 |  50.8 $\pm$ 5.0 |  60.0 $\pm$ 15.2 |  40.0 $\pm$ 7.2 |  58.3 $\pm$ 5.2
GPT3-curie-v1 |  65.2 $\pm$ 2.7 |  67.5 $\pm$ 5.4 |  53.7 $\pm$ 1.2 |  66.0 $\pm$ 14.2 |  59.2 $\pm$ 7.0 | 80.4 $\pm$ 2.9
GPT3.5-davinci-v2 | 70.9 $\pm$ 2.4 | 72.9 $\pm$ 7.3 | 77.5 $\pm$ 2.2 | 86.0 $\pm$ 14.2 |  57.9 $\pm$ 5.0 |  72.1 $\pm$ 5.4

We report the weighted recall and precision for the factor attribute inference
task. We show that with limited instructions, GPT3.5-davinci-v2 can infer the
correct attributes with high precision and recall for the causal factor
inference task.

Table A5: Factor Inference Experiment (Recall):We report weighted recall for
each of the subtask. We boldface results that outperform the majority
baseline.

(a) Causal Factor Inference (Recall)

GPT3 Size |  | Action
---
Omission
Norm Type | Time |  | Agent
---
Awareness
| Causal
---
Structure
| Event
---
Normality
| (N=112) | (N=88) | (N=67) | (N=43) | (N=98) | (N=89)
GPT3-babbage-v1 | 28.6 | 76.1 | 19.4 | 69.8 | 32.7 | 49.4
GPT3-curie-v1 | 32.1 | 23.9 | 7.5 | 46.5 | 67.3 | 49.4
GPT3.5-davinci-v2 | 92.0 | 97.7 | 85.1 | 100.0 | 98.0 | 84.3

(b) Moral Factor Inference (Recall)

GPT3 Size |  | Personal
---
Force
(N=48)
| Beneficience
---
(N=48)
| Locus of
---
Intervention
(N=48)
| Causal Role
---
(N=48)
| Counterfactual
---
Evitability
(N=48)
GPT3-babbage-v1 | 50.0 | 54.2 | 40.0 | 37.5 | 56.2
GPT3-curie-v1 | 50.0 | 50.0 | 40.0 | 62.5 | 54.2
GPT3.5-davinci-v2 | 75.0 | 75.0 | 90.0 | 64.6 | 68.8

Table A6: Factor Inference Tasks (Precision): We report weighted precision for
each of subtask. We boldface results that outperform the majority baseline.

(a) Causal Factor Inference (Precision)

GPT3 Size |  | Action
---
Omission
Norm Type | Time |  | Agent
---
Awareness
| Causal
---
Structure
| Event
---
Normality
| (N=112) | (N=88) | (N=67) | (N=43) | (N=98) | (N=89)
GPT3-babbage-v1 | 8.2 | 58.0 | 72.6 | 76.0 | 10.7 | 24.4
GPT3-curie-v1 | 79.9 | 5.7 | 0.6 | 21.6 | 62.2 | 24.4
GPT3.5-davinci-v2 | 92.8 | 97.8 | 86.4 | 100.0 | 98.0 | 86.1

(b) Moral Factor Inference (Precision)

GPT3 Size |  | Personal
---
Force
(N=48)
| Beneficience
---
(N=48)
| Locus of
---
Intervention
(N=48)
| Causal Role
---
(N=48)
| Counterfactual
---
Evitability
(N=48)
GPT3-babbage-v1 | 25.0 | 29.3 | 16.0 | 14.1 | 57.7
GPT3-curie-v1 | 25.0 | 28.3 | 16.0 | 39.1 | 29.3
GPT3.5-davinci-v2 | 78.1 | 77.5 | 91.4 | 65.3 | 81.4

###  A.11 Prompts for Factor Inference

We also designed the prompts for asking GPT-3 to infer factor attributes.
Unlike the standard setup in few-shot LLM inference tasks, where K samples are
randomly chosen from a pool of training data, we manually write high-level
instructions for GPT-3 or makeup examples that are different from the data.
Also, unlike randomly sampling K examples, we treat our manually written
prompt as “instructions” and keep it the same across examples. Although this
strategy is very simplistic, we find it to work well for our task.

We provide the full set of prompts in the supplement. Here are some examples.
For causal factor inference, we show an example of inferring conjunctive or
disjunctive causal structure. Note how high-level our description is.

> Both A and B need to happen in order for C to happen. Is this conjunctive or
> disjunctive? Answer: Conjunctive.
> Either A or B (or both) need to happen in order for C to happen. Is this
> conjunctive or disjunctive? Answer: Disjunctive.
> [Text Segment to Infer] Is this conjunctive or disjunctive? Answer: [LLM
> Generated Answer]

For moral factor inference, we show an example of inferring personal and
impersonal force:

> In the scenario, I pushed a button to trap another person. Is the action
> personal or impersonal? Answer: Impersonal.
> In the scenario, my action directly affected another person through the use
> of physical force. Is the action personal or impersonal? Answer: Personal.
> [Text Segment to Infer] Is the action personal or impersonal? Answer: [LLM
> Generated Answer]

###  A.12 Disambiguation: Factor vs. Factor-Attributes

In the paper, we use “factor” to refer to high-level concepts that can be
attributed to a text segment. For example, “location” is a higher-level
concept and maps to “factor” in our definition. However, the underlying
specific location, such as “hotel” or “restaurant” – we address as
“attributes”. It is crucial to know that they correspond to two separate
steps: the model should identify the factor for which the text segment
contains information. Then, the model should identify the underlying attribute
for this factor. This is a rather complicated process, and in this paper, we
focus on the second step – given a pre-selected text segment with a known
factor for that segment (by a human), whether LLMs can infer the underlying
attributes accurately. We find that LLMs can accomplish this task relatively
well. We leave the investigation of the full pipeline to future work.

###  A.13 Delphi API Response Labeling Guideline

Delphi has provided a FreeQA API where we can submit the text, and the
returned response falls under three classes: good, discretionary, and bad.
Since we are calculating our accuracy using a 3-class strategy, we treat good
as “Yes”, bad as “No”, and discretionary as “ambiguous”.

###  A.14 Annotation Agreement Calculation

We calculate the agreement between two annotators on both causal and moral
datasets. Each annotator is allowed to assign as many or as few factors per
story as they choose. After assigning the factor, they are asked to select
text inside the story that best supports their decision to assign the factor
to the story.

We calculate the agreement by IoU (Intersection over Union). The factor-
attribute assignment overlap between two annotators on the causal dataset is
0.8920. The moral dataset’s factor annotation is pre-specified by the authors
of the original papers, therefore needing no manual annotation. For text
segments for each factor, we only compute IoU if both annotators agree the
factor is present. Under this condition, the IoU for causal text segments is
0.8145, and for moral text segments, it is 0.8423.

We include our calculation in compute_correlation.py file.

###  A.15 Crowd Sourced Voting Experiment Design and Interface

We recruit participants from Prolific to give responses to our stories. For
each story, we elicit a yes/no answer from a single participant. We pay wages
equivalent to $12/hr, and we only recruit participants from the US and UK,
with English as their first language. Each participant is presented with 5-6
stories either from the causal dataset or the moral dataset. We present the
story and question and then ask them to choose yes or no. The interface is
designed so that the participant is presented with one story at a time.
Participants are allowed to change their decisions to any story before final
submission. We keep track of how long they spend on each story. We do not
collect personal, private information of any sort from the participants except
their yes/no responses. All the raw data from our experiment is provided in
the supplement.

###  A.16 Model API Card

Since our works use many models that can only be accessed through API, we
provide details of these models and access time in Table A7. We obtained early
access to Anthropic’s claude-v1, a chat model. Anthropic has since
discontinued the API access, and we cannot use the same API anymore.

Model | Access Date |  Details
---|---|---
GPT3-babbage-v1 | 2023-04-03 |  GPT3-babbage-001 model that involves supervised fine-tuning on human-written demonstrations.
GPT3-curie-v1 | 2023-04-03 |  GPT3-curie-001 model that involves supervised fine-tuning on human-written demonstrations.
GPT3.5-davinci-v2 | 2023-04-03 |  GPT3.5-davinci-002 model that involves supervised fine-tuning on human-written demonstrations. Derived from code-davinci-002.
GPT3.5-davinci-v3 | 2023-04-03 |  GPT3.5-davinci-003 model that involves reinforcement learning (PPO) with reward models. Derived from GPT3.5-davinci-002.
Anthropic-claude-v1 | 2023-04-03 |  A model trained using reinforcement learning from human feedback.
GPT3.5-turbo | 2023-10-20 |  Sibling model of GPT3.5-davinci-003 is optimized for chat but works well for traditional completions tasks as well. Snapshot from 2023-06-13.
GPT4 | 2023-10-20 |  GPT-4 is a large multimodal model (currently only accepting text inputs and emitting text outputs) that is optimized for chat but works well for traditional completions tasks. Snapshot of gpt-4 from 2023-06-13.
Table A7: List of models that are accessed through API in this work, including
6 OpenAI models and 1 Anthropic model.

###  A.17 FAQs

##### Are there domain experts in moral psychology/philosophy involved in the
design process of this paper?

All the factors mentioned in Table 2 are taken directly from the papers that
conducted the original experiments. A graduate student and a moral/causal
psychology professor were involved in carefully discussing and designing these
factors. A thorough literature review was conducted to make sure these factors
are comprehensive enough.

###  A.18 Data Sheet

####  A.18.1 Motivation

  * •

For what purpose was the dataset created? The dataset was created to evaluate
language models on causal selection problems and moral trolley problems.

  * •

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)? The dataset is
created by a multidisciplinary team at (Anonymous) University with researchers
in the background of Computer Science, Statistics, Natural Language
Processing, and Psychology.

  * •

Who funded the creation of the dataset? This project did not receive external
funding.

  * •

Any other comments? N/A

####  A.18.2 Composition

  * •

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? The instances are all text descriptions of
scenarios, not based on real events or people.

  * •

Does the dataset contain all possible instances, or is it a sample (not
necessarily random) of instances from a larger set? Yes, there is some
filtering – for example, we did not transcribe ALL story snippets that the
causal papers we cited. This is for the purpose of label balancing. We try to
select pairs of stories based on whether there is a significant change in
human judgments that were verified by the paper’s experiments. This helps make
our dataset more balanced. For the moral dilemma dataset, we transcribed all
the texts from each paper, resulting in a large label imbalance.

  * •

What data does each instance consist of? Text.

  * •

Is there a label or target associated with each instance? Yes, for each
instance, we have a binarized Yes/No label.

  * •

Is any information missing from individual instances? No

  * •

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? Multiple examples can come from the same
paper, which is used to measure the same or similar factors (scientific
hypotheses).

  * •

Are there recommended data splits (e.g., training, development/validation,
testing)? The entire dataset should only be used for testing.

  * •

Are there any errors, sources of noise, or redundancies in the dataset? No

  * •

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? The dataset is
self-contained.

  * •

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)? No

  * •

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? No

  * •

Does the dataset relate to people? Yes, we collected human responses on each
story.

  * •

Does the dataset identify any subpopulations (e.g., by age, gender)? No

  * •

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? No

  * •

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers, criminal history)? No

####  A.18.3 Collection Process

  * •

How was the data associated with each instance acquired? The data (text) were
transcribed from research papers. Papers sometimes would display multiple
segments of the text in a table. Therefore, we assemble the text according to
the table and experimental settings. We do have to interpret the experimental
results based on the paper’s table/figures. When papers report an actual mean
response score, we add them to the data file (comment section); when the paper
doesn’t report a mean response score but instead provides a bar plot of
ratings, we try to read the height of the bar. Note that since we binarize the
responses to “Yes” and “No” – we only need to compare the bar height to the
median height, which is relatively easy to do.

  * •

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
Data were collected by two graduate students. Follow-up factor annotation was
conducted on TagTog.

  * •

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)? N/A

  * •

Who was involved in the data collection process (e.g., students, crowd
workers, contractors) and how they were compensated (e.g., how much were crowd
workers paid)? Two graduate students are the data collectors. One of them is
the lead author of the paper, who does not seek compensation. The other one is
compensated at $15/hr. The data collection process took about 70 hours, and
are compensated for $1025 in total. The additional text span annotation took
about 20 hours for $25/hr and is compensated for $500 in total. We
additionally use Prolific to collect 25 human responses for each story for
$12/hr rate. In total, Prolific workers were compensated for $2500 in total
for 205 hours of annotation work.

  * •

Over what timeframe was the data collected? Does this The original papers of
the dataset were published between 1976 to 2021. We transcribed the data
between March 2021 and August 2021. The text segment annotation happened
between March 2022 and May 2022. The Prolific crowdsourcing annotation
happened during the month of September 2022.

  * •

Were any ethical review processes conducted (e.g., by an institutional review
board)? Yes, we obtained IRB approval for our study.

  * •

Does the dataset relate to people? Explained above.

  * •

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)? We accessed the papers
through the paper publisher’s websites. Our dataset does not include a copy of
the original paper, except we do provide a comment that links each data to
which paper it was transcribed from.

  * •

Were the individuals in question notified about the data collection? Yes. We
have a consent and notification form at the beginning of every study.

  * •

Did the individuals in question consent to the collection and use of their
data? Yes. We have a consent form at the beginning of every study.

  * •

If consent was obtained, was the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? Yes, on
the consent form, we provide the email, phone number, and address of the
primary investigator (PI) for the participants to contact.

  * •

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis)been conducted? All our
participants’ data have remained anonymous. We do not collect PPI (Private
Personal Information).

  * •

Any other comments? N/A

####  A.18.4 Preprocessing/cleaning/labeling

  * •

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? We perform quality checks
on our labels by analyzing how long each participant took to label the
example. We rejected participant labels if they didn’t spend an adequate
amount of time. Additionally, for 16 stories we transcribed from “Immoral
Professors and Malfunctioning Tools: Counterfactual Relevance Accounts Explain
the Effect of Norm Violations on Causal Selection (Kominsky, Phillips, 2019)”,
we have to make them shorter because they are longer than some of the language
models can handle. We removed some sentences that only add additional context
to the story but do not contribute to overall causal judgment (i.e., sentences
like “Alex and Benni are very reliable and Tom is satisfied with their
work.”). We still include the raw, unprocessed instances in our dataset. We
did not modify any other instance.

  * •

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? We add the original mean
response score in the comment field of our data file whenever the original
research paper provides the actual score. We also kept the original
transcription text in the data file.

  * •

Is the software used to preprocess/clean/label the instances available? No

  * •

Any other comments? No

####  A.18.5 Use

  * •

Has the dataset been used for any tasks already? No

  * •

Is there a repository that links to any or all papers or systems that use the
dataset? No. We hope future users of this dataset will cite this paper, and
then all the follow-up papers/systems will show up in the Google Scholar
search engine.

  * •

What (other) tasks could the dataset be used for? N/A

  * •

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? MoCa
is not a certification task, i.e., if the language model achieves high
performance on MoCa, it is human-aligned. MoCa is only an evaluation task that
tests if the model’s behavior is similar to humans. Our focus is narrow and
only on certain aspects of alignment with humans. It cannot and should not be
used to make a sweeping and general statement about AI-human alignment.

  * •

Are there tasks for which the dataset should not be used? N/A

  * •

Any other comments? No

####  A.18.6 Distribution

  * •

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? Yes. The link can be found in the paper.

  * •

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Data will be stored in a GitHub repo.

  * •

When will the dataset be distributed? The data are available right now.

  * •

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? The dataset
is under a Creative Commons license (CC BY 4.0).

  * •

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? N/A

  * •

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? N/A

  * •

Any other comments? N/A

####  A.18.7 Meta-Data for Stories

We made detailed comments for each of our collected examples. We include the
experiment conditions, which table/figure from the original paper we derive
the true label, and if possible, we include the average Likert scale score
from the human responses. We provide them as part of the appendix for ease of
reading, but they are also accessible through the data files we provided in
the supplementary material.

We only sub-sampled some stories to put here:

  1. Story

Normality and actual causal strength (Dropbox) (Icard, Kominsky, Knobe, 2017).
Experiment 1, Vignette 1. (Same as Kominsky et al. 2015). Condition: normative
vs. abnormal, conjunctive. M=3.37 vs. M=5.61.

  2. Story

Normality and actual causal strength (Dropbox) (Icard, Kominsky, Knobe, 2017).
Experiment 1, Vignette 1: Motion detector. Condition: normative vs. abnormal,
disjunctive. M=3.25 vs. M=4.18. Prescriptive norm.

  3. Story

Normality and actual causal strength (Dropbox) (Icard, Kominsky, Knobe, 2017).
Experiment 1, Vignette 3: Train. We skip "battery” because it’s a moral
Condition: normative vs. abnormal, conjunctive. Prescriptive norm.

  4. Story

Normality and actual causal strength (Dropbox) (Icard, Kominsky, Knobe, 2017).
Experiment 1, Vignette 3: Train. We skip "battery” because it’s a moral
Condition: normative vs. abnormal, disjunctive. Prescriptive norm.

  5. Story

What you foresee isn’t what you forget: No evidence for the influence of
epistemic states on causal judgments for abnormal negligent behavior. (Murray,
et al., 2021) (Dropbox). Experiment 1: Epistemic advantage is not necessary
for abnormal inflation, Vignette 1 (No knowledge), 2 (Knowledge) x 2
(Normality). Normal vs. abnormal. (M = 7.50, SD = 1.50, n = 85) vs. (M = 3.32,
SD = 2.47, n = 77). Changed (Henne et al. 2017), add whether Kate noticed or
not.

  6. Story

What you foresee isn’t what you forget: No evidence for the influence of
epistemic states on causal judgments for abnormal negligent behavior. (Murray,
et al., 2021) (Dropbox). Experiment 1: Epistemic advantage is not necessary
for abnormal inflation, Vignette 1 (No knowledge), 2 (Knowledge) x 2
(Normality). (M = 7.48, SD = 1.74, n = 85) vs. (M = 4.79, SD = 2.60, n = 101).
Changed (Henne et al. 2017), add whether Kate noticed or not.

  7. Story

What you foresee isn’t what you forget: No evidence for the influence of
epistemic states on causal judgments for abnormal negligent behavior. (Murray,
et al., 2021) (Dropbox). Experiment 2: Outcome expectation is not necessary
for abnormal inflation, 2 (Knowledge) x 2 (Normality), Vignette 1 (No
knowledge). (Mean = approx. 8) vs. (Mean = approx. 3).

  8. Story

What you foresee isn’t what you forget: No evidence for the influence of
epistemic states on causal judgments for abnormal

  9. Story

Degrading causation (O’Neill, et al., 2019). Experiment 2. Conjunctive
Electricity, Number of Causes 3. Normal vs. Abnormal. We only picked out ones
where we can binarize (>0.5). We modified the question from “To what extent
did X cause Y" (“totally vs. “not at all") to “Did X cause Y" (“Yes" vs.
“No"). Testing abnormal inflation.

  10. Story

Crossed Wires: Blaming Artifacts for Bad Outcomes (Sytsma, 2021), Study 1:
Machine Case with Responsibility Attributions. (M=4.89, SD=1.94) vs. (M=2.83,
SD= 1.86)

  11. Story

Crossed Wires: Blaming Artifacts for Bad Outcomes (Sytsma, 2021), Study 3:
Machine Case with Movement. (M=5.03, SD=2.24) vs. (M=2.88, SD=2.07)

  12. Story

Causation, norm violation, and culpable control (Dropbox) (Alicke, et al.,
2011), Study 2, condition 1. Cheat vs. Did not Cheat condition is not so
interesting (moral violation is judged as more causal). However, in Did not
Cheat condition, norm vs. counternorm is more nuanced. Norm vs. Counternorm.
(M=4.12 vs. M=2.62) Figure 5.

  13. Story

Counterfactual thinking and recency effects in causal judgment (Dropbox)
(Henne, et al., 2021). Experiment 1, Vignette 1 (Overdetermination, Early vs.
Late). (M=35.33 vs. M=-3.44)

  14. Story

Counterfactual thinking and recency effects in causal judgment (Dropbox)
(Henne, et al., 2021). Experiment 1, Vignette 1

  15. Story

The good, the bad, and the timely: how temporal order and moral judgment
influence causal selection, (Reuter, et al, 2014), Rule violation. Scenario 1
(no rule violation) vs. Scenario 10 (Zoe violates a rule). These results are
more comparative because the raw experiment has 5 options: Alice, Zoe, Both,
None of the two, Not sure.

Annotations for moral stories (Trolley problems only). Moral Machine Dataset
and Simplified Moral Machine Dataset are synthetic, we documented the
rationale at the beginning of this appendix.

  1. Story

Throwing a Bomb on a Person Versus Throwing a Person on a Bomb Intervention
Myopia in Moral Intuitions (Dropbox) (Waldmann & Dieterich, 2007), Experiment
2 (based on Experiment 1 Trolley version, but with modifications), agent
intervention with harm to two people as a side effect (AI/S), (M = 4.85, SD =
1.01). AI/M (Agent intervention, use people as means) (M=4.74).

  2. Story

Inference of Intention and Permissibility in Moral Decision Making (Dropbox)
(Kleiman-Weiner, et al., 2015). Trial 1 (2v1). Trolley dilemma story based on
Mikhail, J. (2007). Universal moral grammar: Theory, evidence and the future.
Trends in cognitive sciences, 11(4), 143–152. Trial 2 (1vB)

  3. Story

Inference of Intention and Permissibility in Moral Decision Making (Dropbox)
(Kleiman-Weiner, et al., 2015). Trial 1 (2vB). Trolley dilemma story based on
Mikhail, J. (2007). Universal moral grammar: Theory, evidence and the future.
Trends in cognitive sciences, 11(4), 143–152. Trial 2 (Bv2)

  4. Story

Moral judgment reloaded: a moral dilemma validation study (Christensen, et
al., 2014), Personal-Instrumental (1) vs. Impersonal-Accidental (2) – BURNING
BUILDING (a) vs. BURNING BUILDING (b) Mean = 3.3953488372093 vs.
5.27906976744186